[ { "path": ".clang-format", "content": "---\n# See https://releases.llvm.org/14.0.0/tools/clang/docs/ClangFormatStyleOptions.html\nBasedOnStyle: Chromium\nAccessModifierOffset: -4\nAlignAfterOpenBracket: AlwaysBreak\n# AlignArrayOfStructures can cause crashes, see https://github.com/llvm/llvm-project/issues/55269\n#AlignArrayOfStructures: Left\nAllowAllParametersOfDeclarationOnNextLine: false\nAllowShortBlocksOnASingleLine: Empty\nAllowShortFunctionsOnASingleLine: All\nBinPackArguments: false\nBinPackParameters: false\nBreakBeforeBinaryOperators: NonAssignment\nBreakBeforeBraces: Custom\nBraceWrapping:\n # NB: due to https://github.com/llvm/llvm-project/issues/55582 the Multiline setting will not\n # always work (should be fixed in clang-format 15, but that is not available as a python wheel yet\n # due to https://github.com/ssciwr/clang-format-wheel/issues/49)\n AfterControlStatement: MultiLine # makes sure multiline ifs don't run into their bodies\n AfterFunction: true # makes constructors with initialisers much nicer\nBreakBeforeConceptDeclarations: true\nBreakBeforeTernaryOperators: true\nBreakConstructorInitializers: BeforeComma\nBreakStringLiterals: true\nColumnLimit: 100\nCompactNamespaces: true\nConstructorInitializerIndentWidth: 0\nContinuationIndentWidth: 4\nCpp11BracedListStyle: true\nDerivePointerAlignment: false # force use of the PointerAlignment setting\nFixNamespaceComments: true\nIncludeBlocks : Regroup\nIncludeCategories:\n # Aim is:\n # 0. the \"main\" header file (#include \"foo.h\" in foo.cpp) automatically gets priority 0\n # 1. internal headers (#include \"util/helpers.h\"): quotation marks, with a '/'\n # 2. third-party headers (#include ): angle brackets, '/' or .h/.hpp/.h++\n # file ext\n # 3. standard library headers (#include ): angle brackets, no file ext, no '/'\n - Regex: '^\"'\n Priority: 1\n - Regex: '^<.*/'\n Priority: 2\n - Regex: '\\.h>'\n Priority: 2\n - Regex: '\\.hpp>'\n Priority: 2\n - Regex: '\\.h\\+\\+>'\n Priority: 2\nIncludeIsMainRegex: '(_test|_tests|Tests|Test)?$'\n # foo.h will be considered the \"main\" header (and sorted to the top) for all of the following:\n # - foo.cpp\n # - foo_test.cpp\n # - foo_tests.cpp\n # - fooTests.cpp (although this is intended for Foo.h and FooTests.cpp)\n # - fooTest.cpp (although this is intended for Foo.h and FooTest.cpp)\nIndentCaseLabels: false\nIndentWidth: 4\nInsertBraces: true\nPackConstructorInitializers: CurrentLine\nPointerAlignment: Middle\nQualifierAlignment: Right # const east\n# clang 14 *should* know about QualifierOrder (according to its docs) but claims it doesn't\n#QualifierOrder: ['static', 'constexpr', 'inline', 'type', 'const', 'volatile', 'restrict']\nReflowComments: false\nSeparateDefinitionBlocks: Always\nSortIncludes: CaseInsensitive\nSortUsingDeclarations: true\nSpaceAroundPointerQualifiers: Before\nStandard: c++20\n" }, { "path": ".codespellrc", "content": "# Waiting for pyproject.toml support: https://github.com/codespell-project/codespell/issues/2055\n\n[codespell]\n# \"write-changes\" doesn't work with \"ignore-regex\"\n# https://github.com/codespell-project/codespell/issues/2056\n# comma-separated list of built-in dictionaries (default is \"clear,rare\")\nbuiltin = clear,rare,code\n# show the line in which the error occurred\ncontext = 0\n# these options are turned on by specifying them here\ncheck-filenames =\ncheck-hidden =\nenable-colors =\n# split words on underscores\n# e.g. \"foo_bar\" is split into two words (\"foo\", \"bar\") instead of one word (\"foo_bar\")\nignore-regex = _\n# comma-separated list of false positives\nignore-words-list = iff,inout,befores,deque,stdio,O_WRONLY,wronly,sv_lite,lite,creat,arange\n# comma-separated list of globs of files not to check\nskip = .gitignore,.codespellrc\n" }, { "path": ".flake8", "content": "# Waiting for pyproject.toml support: https://gitlab.com/pycqa/flake8/-/issues/428\n\n[flake8]\nextend-ignore = E203, W503\nmax-line-length = 120\nper-file-ignores = __init__.py:F401, __init__.py:F403\n" }, { "path": ".gitattributes", "content": "# Based on https://github.com/alexkaratarakis/gitattributes\n# Auto detect text files and force linux-style line endings\n* text=auto eol=lf\n# Documents\n*.bibtex text diff=bibtex\n*.doc diff=astextplain\n*.DOC diff=astextplain\n*.docx diff=astextplain\n*.DOCX diff=astextplain\n*.dot diff=astextplain\n*.DOT diff=astextplain\n*.pdf diff=astextplain\n*.PDF diff=astextplain\n*.rtf diff=astextplain\n*.RTF diff=astextplain\n*.md text diff=markdown\n*.mdx text diff=markdown\n*.tex text diff=tex\n*.adoc text\n*.textile text\n*.mustache text\n*.csv text\n*.tab text\n*.tsv text\n*.txt text\n*.sql text\n*.epub diff=astextplain\n# Graphics\n*.png binary\n*.jpg binary\n*.jpeg binary\n*.gif binary\n*.tif binary\n*.tiff binary\n*.ico binary\n*.svg binary\n*.eps binary\n*.bash text\n*.fish text\n*.sh text\n*.zsh text\n# These are explicitly windows files and should use crlf\n*.bat text eol=crlf\n*.cmd text eol=crlf\n*.ps1 text eol=crlf\n# Serialisation\n*.json text\n*.toml text\n*.xml text\n*.yaml text\n*.yml text\n# Archives\n*.7z binary\n*.gz binary\n*.tar binary\n*.tgz binary\n*.zip binary\n# Text files where line endings should be preserved\n*.diff -text\n*.patch -text\n# Exclude git(lab)-specific files when making an archive of the source tree\n.gitattributes export-ignore\n.gitignore export-ignore\n.gitkeep export-ignore\n.git-blame-ignore-revs export-ignore\n.gitlab-ci.yml export-ignore\n/ci export-ignore\n# C++ Sources\n*.c text diff=cpp\n*.cc text diff=cpp\n*.cxx text diff=cpp\n*.cpp text diff=cpp\n*.c++ text diff=cpp\n*.hpp text diff=cpp\n*.h text diff=cpp\n*.h++ text diff=cpp\n*.hh text diff=cpp\n# Read formats\n*.pod5 filter=lfs diff=lfs merge=lfs -text\n*.fast5 binary\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.md", "content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n## Issue Description\n\n> Please provide a description of your issue and include any commands used to reproduce the issue.\n\n## Logs\n\n> Please provide any log files. These can be generated by setting the `POD5_DEBUG` environment variable e.g. `POD5_DEBUG=1 pod5 view my.pod5`\n\n## Specifications\n\n- Pod5 Version:\n- Python Version:\n- Platform:\n" }, { "path": ".gitignore", "content": "build*/\n.conan/\ncmake-build*/\nCMakeUserPresets.json\n_build/\n.conan/\n.cache/\ndist/\n.DS_Store\n.pod5\nvenv/\n*.venv/\nuv.lock\ndocs/public/\n.tmp_pod5*\n_version.py\n*egg-info/\nPOD5Version.cmake\n*.swp\ntest_package/CMakeUserPresets.json\n.vscode/\n.devcontainer/\n__pycache__\npython/Python.framework/\n/fuzz/corpus_*\n" }, { "path": ".gitlab-ci.yml", "content": "stages:\n - .pre\n - build\n - test\n - build-conan\n - archive\n - deploy\n\ninclude:\n - local: '/ci/gitlab-ci-common.yml'\n\nvariables:\n GIT_SUBMODULE_STRATEGY: recursive\n STABLE_BRANCH_NAME: master\n DO_UPLOAD: \"yes\" # Always upload in conan upload jobs (only run on tags)\n CONAN_PROFILE_BUILD_TYPE: Release\n CONAN_VENV_PYTHON: \"3.13\"\n CMAKE_VERSION: \"4.2.3\"\nbefore_script:\n - \"\"\n\n# The versions that we build and test.\n.parallel-py-versions:\n parallel:\n matrix:\n - PYTHON_VERSION: [\"3.10\", \"3.11\", \"3.12\", \"3.13\", \"3.14\"]\n\n\n# ======================================\n#\n# Docker\n#\n# ======================================\n\n\n.build-docker-image:\n stage: .pre\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-docker:latest\n before_script:\n - docker login --username ${CI_REGISTRY_USER} --password ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}\n when: manual\n retry:\n max: 2\n when: runner_system_failure\n script:\n - tag=\"${CI_REGISTRY_IMAGE}/${IMAGE_TAG}\"\n - docker image build --pull --target \"${PLATFORM}\"\n --tag \"${tag}\" --file ${DOCKERFILE} ci/docker\n - docker image push ${tag}\n\ndocker base aarch64:\n tags:\n - docker-builder-arm\n extends:\n - .build-docker-image\n variables:\n IMAGE_TAG: \"build-arm64\"\n DOCKERFILE: \"ci/docker/Dockerfile.py39.arm64\"\n\ndocker base x86-64:\n tags:\n - docker-builder\n extends:\n - .build-docker-image\n variables:\n IMAGE_TAG: \"build-x64\"\n DOCKERFILE: \"ci/docker/Dockerfile.py39.x64\"\n\ndocker conda:\n tags:\n - docker-builder\n extends:\n - .build-docker-image\n variables:\n IMAGE_TAG: \"conda\"\n DOCKERFILE: \"ci/docker/Dockerfile.conda\"\n\n\n.docker template:\n stage: docker\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-docker:latest\n before_script:\n - docker login --username ${CI_REGISTRY_USER} --password ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}\n retry:\n max: 2\n when: runner_system_failure\n\n# ======================================\n#\n# Versioning\n#\n# ======================================\n\nprepare_version:\n stage: .pre\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:3.10\n script:\n - git tag -d $(git tag -l \"*a*\")\n - git tag -d $(git tag -l \"*b*\")\n - git tag -d $(git tag -l \"*r*\")\n - git tag -d $(git tag -l \"*c*\")\n - git tag -d $(git tag -l \"*dev*\")\n - if [[ ${CI_COMMIT_TAG/#v/} && -z $( git tag -l \"${CI_COMMIT_TAG/#v/}\" ) ]]; then git tag ${CI_COMMIT_TAG/#v/}; fi\n - pip install --upgrade pip setuptools_scm~=7.1\n - apt update && apt install -y git-lfs\n - git status --porcelain\n - python -m setuptools_scm\n - cat _version.py\n # Show the version that will be used in the pod5/pyproject.toml\n - VERSION=$(grep \"__version__\" _version.py | awk '{print $5}' | tr -d \"'\" | cut -d'+' -f1)\n - echo $VERSION\n - python -m pod5_make_version\n - cat cmake/POD5Version.cmake\n - cat _version.py python/lib_pod5/src/lib_pod5/_version.py\n - cat _version.py python/pod5/src/pod5/_version.py\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - \"cmake/POD5Version.cmake\"\n - \"_version.py\"\n - \"python/lib_pod5/src/lib_pod5/_version.py\"\n - \"python/pod5/src/pod5/_version.py\"\n\n\n# ======================================\n#\n# Pre-Flight Setup / Checks\n#\n# ======================================\n\ntag_version_check:\n stage: .pre\n needs:\n - \"prepare_version\"\n only:\n - tags\n image: ${CI_REGISTRY}/minknow/images/build-x86_64-gcc13:latest\n script:\n - uv venv .venv\n - source .venv/bin/activate\n - uv pip install \"cmake==${CMAKE_VERSION}\"\n - pod5_version=\"$(cmake -P ci/get_tag_version.cmake 2>&1)\"\n - tag_version=\"${CI_COMMIT_TAG/#v/}\"\n - if [[ \"${pod5_version}\" != \"${tag_version}\" ]]; then\n echo \"Tag is for release ${tag_version}, but POD5 version is $pod5_version\";\n exit 1;\n fi\n\napi_lib_version_check:\n stage: .pre\n needs:\n - \"prepare_version\"\n image: ${CI_REGISTRY}/minknow/images/build-x86_64-gcc13:latest\n script:\n - cat _version.py\n - NO_DEV_VERSION=$(grep \"__version__\" _version.py | awk '{print $5}' | tr -d \"'\" | cut -d'+' -f1 | sed 's/\\([0-9]\\+\\.[0-9]\\+\\.[0-9]\\+\\).*$/\\1/')\n - echo $NO_DEV_VERSION\n - cat python/pod5/pyproject.toml\n - echo \"If this jobs fails then we have forgotten to match the api and lib version in the api python/pod5/pyproject.toml\"\n - grep \"lib_pod5\\s*==\\s*$NO_DEV_VERSION\" python/pod5/pyproject.toml\n\n\npre-commit checks:\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:3.10\n stage: .pre\n tags:\n - linux_x86\n - docker\n script:\n - pip install pre-commit\n - if ! pre-commit run --all-files; then\n - cat \"${PRE_COMMIT_HOME}/pre-commit.log\"\n - >-\n if grep -F -q \\\n -e \"InvalidManifestError\" \\\n -e \"error: [Errno 17] File exists: 'build/temp.linux-x86_64-cpython-\" \\\n \"${PRE_COMMIT_HOME}/pre-commit.log\"; then\n - echo \"Bad cache state detected, deleting cache and re-running\"\n - rm -rf \"${PRE_COMMIT_HOME}/\"\n - pre-commit run --all-files\n - else\n - exit 1\n - fi\n - fi\n after_script:\n - cat \"${PRE_COMMIT_HOME}/pre-commit.log\" || true\n variables:\n PRE_COMMIT_HOME: ${CI_PROJECT_DIR}/.cache/pre-commit\n cache:\n paths:\n - ${PRE_COMMIT_HOME}\n\n\n# ======================================\n#\n# Build Lib Standalone\n#\n# ======================================\n\n\nbuild-standalone-ubu22:\n stage: build\n image: external-docker.artifactory.oxfordnanolabs.local/ubuntu:22.04\n needs:\n - \"prepare_version\"\n script:\n - export DEBIAN_FRONTEND=noninteractive\n - apt-get update\n - apt-get install -y -V ca-certificates lsb-release wget\n - wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\n - apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\n - apt-get update\n - apt-get install -y cmake build-essential libzstd-dev libzstd-dev libflatbuffers-dev libarrow-dev=18.0.0-1\n - mkdir -p build\n - cd build\n - cmake\n -D POD5_DISABLE_TESTS=OFF\n -D POD5_BUILD_EXAMPLES=ON\n -D BUILD_PYTHON_WHEEL=OFF\n ..\n - cmake --build . --parallel\n - ctest -C Release -VV\n\n\n# ======================================\n#\n# Build helpers\n#\n# ======================================\n\n\n# Takes CMAKE_ARGS, AUDITWHEEL_PLATFORM, and PYTHON_VERSION.\n.conan-build-and-test:\n - |\n - export TOOLCHAIN_FILE=build/generators/conan_toolchain.cmake\n - pod5_version=\"$(cmake -P ci/get_tag_version.cmake 2>&1)\"\n - mkdir -p build\n - cd build\n - ${conan_exe} install --profile ${CONAN_PROFILE} ${EXTRA_INSTALL_ARGS} ..\n - cmake ${CMAKE_ARGS}\n -D BUILD_SHARED_LIB=ON\n -D CMAKE_BUILD_TYPE=Release\n -D POD5_DISABLE_TESTS=OFF\n -D POD5_BUILD_EXAMPLES=ON\n -D BUILD_PYTHON_WHEEL=OFF\n -D CMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_FILE}\n ..\n - cmake --build . --config Release --parallel\n - ctest -C Release -VV\n - ../ci/install.sh\n - cmake ${CMAKE_ARGS}\n -D BUILD_SHARED_LIB=OFF\n -D CMAKE_BUILD_TYPE=Release\n -D POD5_DISABLE_TESTS=OFF\n -D POD5_BUILD_EXAMPLES=ON\n -D BUILD_PYTHON_WHEEL=ON\n -D PYTHON_VERSION=${PYTHON_VERSION}\n -D CMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_FILE}\n ..\n - cmake --build . --config Release --parallel\n - ctest -C Release -VV\n - ../ci/install.sh STATIC_BUILD\n - ../ci/package.sh ${OUTPUT_SKU} ${AUDITWHEEL_PLATFORM}\n\n\n# ======================================\n#\n# Build Lib Linux\n#\n# ======================================\n\n\n.build-linux:\n stage: build\n needs:\n - \"prepare_version\"\n variables:\n EXTRA_INSTALL_ARGS: \"-o arrow:with_boost=False -o arrow:with_thrift=False -o arrow:parquet=False\"\n before_script:\n - /opt/python/cp310-cp310/bin/pip install -U pip 'conan<2' auditwheel build \"cmake==${CMAKE_VERSION}\"\n - ln -n /opt/python/cp310-cp310/bin/auditwheel /usr/bin/auditwheel\n - ln -n /opt/python/cp310-cp310/bin/conan /usr/bin/conan\n - conan config install --verify-ssl=no ${CONAN_CONFIG_URL}\n - conan_exe=$(which conan)\n script:\n - !reference [\".conan-build-and-test\"]\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - \"lib_pod5*.tar.gz\"\n - \"lib_pod5*.whl\"\n\nlinux-x64-gcc9-release-build:\n image: external-quay.artifactory.oxfordnanolabs.local/pypa/manylinux2014_x86_64\n extends:\n - .build-linux\n - .parallel-py-versions\n tags:\n - linux\n variables:\n CONAN_PROFILE: \"linux-x86_64-gcc9.jinja\"\n CONAN_PROFILE_CPPSTD: 17\n OUTPUT_SKU: \"linux-x64\"\n AUDITWHEEL_PLATFORM: manylinux2014_x86_64\n\nlinux-aarch64-gcc9-release-build:\n image: external-quay.artifactory.oxfordnanolabs.local/pypa/manylinux2014_aarch64\n extends:\n - .build-linux\n - .parallel-py-versions\n tags:\n - linux_aarch64\n - high-cpu\n variables:\n CONAN_PROFILE: \"linux-aarch64-gcc9.jinja\"\n CONAN_PROFILE_CPPSTD: 17\n OUTPUT_SKU: \"linux-arm64\"\n AUDITWHEEL_PLATFORM: manylinux2014_aarch64\n\n\n# ======================================\n#\n# Build Lib OSX\n#\n# ======================================\n\n\n.build-osx-common:\n stage: build\n needs:\n - \"prepare_version\"\n variables:\n EXTRA_INSTALL_ARGS: \"-o arrow:with_boost=False -o arrow:with_thrift=False -o arrow:parquet=False\"\n before_script:\n - uv venv .venv_conan --python ${CONAN_VENV_PYTHON} --seed\n - source .venv_conan/bin/activate\n # Note that cmake 3.31+ do not work properly on macOS 14 (and earlier)\n # Pinning to 3.30 avoid SSL issues when connecting to internal servers\n - uv pip install -U pip 'conan<2' 'cmake==3.30.9'\n - conan config install --verify-ssl=no \"${CONAN_CONFIG_URL}\"\n - conan_exe=$(which conan)\n - uv python install ${PYTHON_VERSION}\n - uv venv --python \"python${PYTHON_VERSION}\" .venv --seed\n - source .venv/bin/activate\n - which python\n - python --version\n script:\n - python3 -c \"import sysconfig; print(sysconfig.get_platform())\"\n - !reference [\".conan-build-and-test\"]\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - \"lib_pod5*.tar.gz\"\n - \"lib_pod5*.whl\"\n\nosx-arm64-clang15-release-build:\n extends:\n - .build-osx-common\n - .parallel-py-versions\n tags:\n - osx_arm64\n - xcode-15.3\n - conan\n variables:\n CONAN_PROFILE: \"macos-aarch64-appleclang-15.0.jinja\"\n CONAN_PROFILE_CPPSTD: 20\n CMAKE_ARGS: \"-DCMAKE_OSX_ARCHITECTURES=arm64\"\n MACOSX_DEPLOYMENT_TARGET: \"14.0\"\n OUTPUT_SKU: \"osx-14.0-arm64\"\n FORCE_PYTHON_PLATFORM: macosx_14_0_arm64\n\n\n# ======================================\n#\n# Build Lib Windows\n#\n# ======================================\n\n\n.build-win-common:\n stage: build\n needs:\n - \"prepare_version\"\n retry: 1\n variables:\n # We need to override arrow's boost 1.85.0 requirement to match the version we use internally.\n EXTRA_INSTALL_ARGS: \"-o arrow:with_thrift=False -o arrow:parquet=False --require=boost/1.86.0@ -o boost:without_locale=True\"\n before_script:\n - uv venv .venv_conan --python ${CONAN_VENV_PYTHON} --seed\n - source .venv_conan/Scripts/activate\n - uv pip install 'conan<2' \"cmake==${CMAKE_VERSION}\"\n - conan config install --verify-ssl=no \"${CONAN_CONFIG_URL}\"\n - conan_exe=$(which conan)\n - uv python install ${PYTHON_VERSION}\n - uv venv --python \"python${PYTHON_VERSION}\" .venv --seed\n - source .venv/Scripts/activate\n script:\n - uv pip install build\n - !reference [\".conan-build-and-test\"]\n after_script:\n # HACK: for some reason, pod5_unit_tests.exe is sticking around; deleting it works, but it\n # doesn't go away immediately (as though something had it open with FILE_SHARE_DELETE, although\n # the Handle utility from SysInternals couldn't find anything).\n # This also appears to be happening for the fuzz targets, so remove and wait for every exe.\n - rm -v build/Release/bin/*.exe\n - date\n - while true; do\n - ls build/Release/bin/*.exe || break\n - sleep 1\n - done\n - date\n\nwin-x64-msvc2019-release-build:\n extends:\n - .build-win-common\n - .parallel-py-versions\n tags:\n - windows\n - VS2019\n - conan\n variables:\n CONAN_PROFILE: \"windows-x86_64-vs2019.jinja\"\n CONAN_PROFILE_CPPSTD: 17\n OUTPUT_SKU: \"win-x64\"\n CMAKE_ARGS: \"-A x64\"\n CMAKE_GENERATOR: \"Visual Studio 16 2019\"\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - \"lib_pod5*.tar.gz\"\n - \"lib_pod5*.whl\"\n\n# ======================================\n#\n# Build Python API\n#\n# ======================================\n\n\nbuild-python-api:\n stage: build\n needs:\n - \"prepare_version\"\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:3.10\n tags:\n - linux\n script:\n - git tag -d $(git tag -l \"*a*\")\n - git tag -d $(git tag -l \"*b*\")\n - git tag -d $(git tag -l \"*r*\")\n - git tag -d $(git tag -l \"*c*\")\n - git tag -d $(git tag -l \"*dev*\")\n - if [[ ${CI_COMMIT_TAG/#v/} && -z $( git tag -l \"${CI_COMMIT_TAG/#v/}\" ) ]]; then git tag ${CI_COMMIT_TAG/#v/}; fi\n - cat _version.py\n - VERSION=$(grep \"__version__\" _version.py | awk '{print $5}' | tr -d \"'\" | cut -d'+' -f1)\n - echo $VERSION\n - cd python/pod5/\n # update the lib_pod5 dependency in pod5/pyproject.toml to match\n - sed -i \"s/.*lib_pod5.*/\\ \\ \\ \\ \\'lib_pod5 == ${VERSION}\\',/\" pyproject.toml\n - cat pyproject.toml\n - pip install -U pip build\n - python -m build --outdir ../../\n - cd ../..\n - ls *.whl *.tar.gz\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - \"pod5*.whl\"\n - \"pod5*.tar.gz\"\n\n\n# ======================================\n#\n# Test Tools\n#\n# ======================================\n\ntools-linux-x64:\n extends:\n - .parallel-py-versions\n stage: test\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:${PYTHON_VERSION}\n tags:\n - linux\n before_script:\n - python${PYTHON_VERSION} -m venv .venv/\n - source .venv/bin/activate\n needs:\n - linux-x64-gcc9-release-build\n - build-python-api\n script:\n - pip install ./lib_pod5*cp${PYTHON_VERSION/./}*.whl pod5-*.whl\n - pod5 convert fast5 ./test_data/ --output ./output_files --one-to-one ./test_data\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./test_data/multi_fast5_zip_v4.pod5\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./test_data/multi_fast5_zip_v3.pod5\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./test_data/multi_fast5_zip_v2.pod5\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./test_data/multi_fast5_zip_v1.pod5\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./test_data/multi_fast5_zip_v0.pod5\n - pod5 convert to_fast5 ./output_files/ --output ./output_files\n - pod5 convert fast5 ./output_files/*.fast5 --output ./output_files_2 --one-to-one ./output_files/\n - python${PYTHON_VERSION} python/pod5/test_utils/check_pod5_files_equal.py ./output_files/multi_fast5_zip.pod5 ./output_files_2/*.pod5\n\n\n# ======================================\n#\n# Pytest\n#\n# ======================================\n\n\n.pytest:\n stage: test\n before_script:\n - python${PYTHON_VERSION} -m venv .venv/\n - source .venv/*/activate\n - python --version\n - python -m pip install --upgrade pip\n script:\n - pip install ./lib_pod5*cp${PYTHON_VERSION/./}*.whl pod5-*.whl\n - pip install pytest pytest-cov pytest-mock psutil\n - pytest\n - POD5_DISABLE_MMAP_OPEN=1 pytest\n\n.pytest-with-uv:\n extends:\n - .pytest\n before_script:\n - uv python install ${PYTHON_VERSION}\n - uv venv --python \"python${PYTHON_VERSION}\" .venv --seed\n - source .venv/*/activate\n\npytest-linux-x64:\n extends:\n - .pytest\n - .parallel-py-versions\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:${PYTHON_VERSION}\n tags:\n - linux\n needs:\n - linux-x64-gcc9-release-build\n - build-python-api\n\npytest-linux-aarch64:\n extends:\n - .pytest\n - .parallel-py-versions\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:${PYTHON_VERSION}\n tags:\n - linux_aarch64\n - high-cpu\n needs:\n - linux-aarch64-gcc9-release-build\n - build-python-api\n\npytest-osx-arm64:\n extends:\n - .pytest-with-uv\n - .parallel-py-versions\n tags:\n - osx_arm64\n needs:\n - osx-arm64-clang15-release-build\n - build-python-api\n\npytest-win-x64:\n retry: 1\n extends:\n - .pytest-with-uv\n - .parallel-py-versions\n tags:\n - windows\n needs:\n - win-x64-msvc2019-release-build\n - build-python-api\n\n\n# ======================================\n#\n# Conda Testing\n#\n# ======================================\n\n\nconda_pytest:\n extends:\n - .pytest\n - .parallel-py-versions\n image: ${CI_REGISTRY}/minknow/pod5-file-format/conda:latest\n tags:\n - linux\n needs:\n - linux-x64-gcc9-release-build\n - build-python-api\n before_script:\n - |\n cat > environment.yml << EOF\n name: pod5_conda_test\n channels:\n - conda-forge\n - bioconda\n dependencies:\n - python=${PYTHON_VERSION}\n - cmake\n - pyarrow\n - pip\n EOF\n - cat environment.yml\n - mamba --version\n - mamba env create -f environment.yml\n - conda env list\n # This is a work around for conda init in gitlab\n - eval \"$(conda shell.bash hook)\"\n - conda activate pod5_conda_test\n\n\n# ======================================\n#\n# Benchmarks\n#\n# ======================================\n\n\n.benchmark:\n stage: test\n before_script:\n - python3 -m venv .venv/\n - source .venv/bin/activate\n script:\n - pip install ./${LIB_WHEEL_GLOB} pod5-*.whl setuptools\n - pip install -r ./benchmarks/image/requirements-benchmarks.txt\n - ./benchmarks/image/install_slow5.sh\n - export PATH=\"$(pwd)/slow5tools-v1.3.0/:$PATH\"\n - ./benchmarks/run_benchmarks.py ./test_data/ ./benchmark-outputs\n\nbenchmark-linux-x64:\n extends: [\".benchmark\"]\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:3.14\n tags:\n - linux\n needs:\n - linux-x64-gcc9-release-build\n - build-python-api\n variables:\n LIB_WHEEL_GLOB: \"lib_pod5*cp314*.whl\"\n\nbenchmark-linux-aarch64:\n extends: [\".benchmark\"]\n image: ${CI_REGISTRY}/traque/ont-docker-base/ont-base-python:3.14\n tags:\n - linux_aarch64\n - high-cpu\n needs:\n - linux-aarch64-gcc9-release-build\n - build-python-api\n variables:\n LIB_WHEEL_GLOB: \"lib_pod5*cp314*.whl\"\n\n\n# ======================================\n#\n# Fuzz tests and coverage reports\n#\n# ======================================\n\n.generic-linux-x64-gcc11-build:\n stage: build\n image: external-docker.artifactory.oxfordnanolabs.local/ubuntu:jammy\n tags:\n - linux\n variables:\n CONAN_PROFILE: \"linux-x86_64-gcc11.jinja\"\n CONAN_PROFILE_CPPSTD: 20\n CMAKE_BUILD_TYPE: Release\n needs:\n - \"prepare_version\"\n script:\n # Install requirements.\n - apt-get update\n - apt-get install -y pip\n - pip install -U pip 'conan<2' auditwheel build \"cmake==${CMAKE_VERSION}\"\n - conan config install --verify-ssl=no ${CONAN_CONFIG_URL}\n # Setup build.\n - pod5_version=\"$(cmake -P ci/get_tag_version.cmake 2>&1)\"\n - mkdir -p build\n - pushd build\n # Tell conan that it's OK to use libstdc++ settings.\n - conan install\n --profile ${CONAN_PROFILE}\n -s compiler.libcxx=libstdc++11\n -s compiler.cppstd=${CONAN_PROFILE_CPPSTD}\n -s build_type=${CMAKE_BUILD_TYPE}\n -o arrow:with_boost=False\n -o arrow:with_thrift=False\n -o arrow:parquet=False\n ..\n - cmake\n -D CMAKE_TOOLCHAIN_FILE=build/generators/conan_toolchain.cmake\n -D CMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}\n -D BUILD_PYTHON_WHEEL=OFF\n ${CMAKE_EXTRA_ARGS}\n ..\n # Do the build\n - cmake --build . --config ${CMAKE_BUILD_TYPE} --parallel\n - popd\n\nlinux-x64-gcc11-fuzz:\n extends: .generic-linux-x64-gcc11-build\n allow_failure: true\n variables:\n CC: clang\n CXX: clang++\n CMAKE_EXTRA_ARGS: \"-D ENABLE_FUZZERS=ON -D FUZZER_RUN_TIME=600\"\n script:\n # We need clang for libFuzzer.\n - apt-get update\n - apt-get install -y clang\n # Do the build.\n - !reference [\".generic-linux-x64-gcc11-build\", \"script\"]\n # Remove the zipped corpora now that we've extracted it, since we\n # don't need it artifacted.\n - rm fuzz/*.zip\n # Run the tests.\n - ctest -C Release --test-dir build -VV -R ${FUZZER_TEST}\n parallel:\n matrix:\n - FUZZER_TEST:\n - \"compress\"\n - \"file\"\n artifacts:\n # Artifact everything in /fuzz so that we can get to any new/failing corpora.\n when: always\n paths:\n - ./fuzz\n\nlinux-x64-gcc11-coverage:\n extends: .generic-linux-x64-gcc11-build\n variables:\n CMAKE_BUILD_TYPE: \"Debug\"\n CMAKE_EXTRA_ARGS: \"-D POD5_DISABLE_TESTS=OFF -D ENABLE_COVERAGE_REPORT=ON\"\n script:\n # We need a venv.\n - apt-get update\n - apt-get install -y python3-venv\n # Do the build.\n - !reference [\".generic-linux-x64-gcc11-build\", \"script\"]\n # Run the coverage report.\n - ./ci/generate_coverage_report.sh build\n coverage: '/^TOTAL\\s+\\d+\\s+\\d+\\s+(\\d+(?:\\.\\d+)?\\%)$/'\n artifacts:\n reports:\n coverage_report:\n coverage_format: cobertura\n path: coverage-report-*.xml\n paths:\n # Artifact the human readable ones too.\n - coverage-report-*.html\n\n\n# ======================================\n#\n# Conan\n#\n# ======================================\n\n.setup-venv:\n - KERNEL=$(uname -s)\n - if [[ ! ${KERNEL} =~ \"Linux\" ]]; then\n # Must use an explicit version here otherwise we get the windows store one.\n # Can be any version since it's only for installing conan.\n - python3.13 -m venv .venv\n - source .venv/*/activate\n - fi\n\n.reset-line-endings:\n # This is needed to enforce LF line-endings in the pybind submodule\n # otherwise conan generates different revisions for windows and unix\n - re='^(MINGW|CYGWIN|MSYS).*'\n - if [[ $(uname -s) =~ $re ]]; then\n - git rm -rf :/\n - git checkout HEAD -- :/\n - fi\n\n.conan-setup-common:\n - !reference [\".reset-line-endings\"]\n - !reference [\".setup-venv\"]\n - pip install 'conan<2'\n - conan --version\n - VERSIONS=\"$(cmake -P ci/get_tag_version.cmake 2>&1)\"\n\n.conan-build-common:\n stage: build-conan\n dependencies:\n - \"prepare_version\"\n before_script:\n - !reference [\".conan-setup-common\"]\n - conan remove \"*\" -f\n - conan config install --verify-ssl=no \"${CONAN_CONFIG_URL}\"\n\n.conan2-common:\n before_script:\n - !reference [\".reset-line-endings\"]\n - !reference [\".setup-venv\"]\n - pip install --upgrade conan\n - conan --version\n - conan remove \"*\" --confirm\n - conan config install --verify-ssl=no \"${CONAN2_CONFIG_URL}\"\n\n.conan2-build:\n extends: .conan2-common\n stage: build-conan\n dependencies:\n - \"prepare_version\"\n script:\n - version=$(cmake -P ci/get_tag_version.cmake 2>&1 | cut -d. -f1-3)\n # set up the correct ref\n - opts=(\"--version=${version}\" --user=nanopore --channel=stable)\n # fail if we can't find dependencies\n - opts+=(\"--build=pod5_file_format/*\")\n # select the build profile\n - opts+=(-pr:a \"${PROFILE_BASE}\")\n # use the arrow packages we have built\n - opts+=('-o:a=arrow/*:with_thrift=False' '-o:a=arrow/*:parquet=False' '-o:a=arrow/*:with_zstd=True' '-o:a=arrow/*:with_boost=False')\n - echo \"Running conan create . ${opts[@]}\"\n - conan create . \"${opts[@]}\"\n - conan cache save \"*/*:*\" --file=conan-${CI_JOB_ID}.tgz\n variables:\n # use an arrow package that doesn't use Boost, even on Windows\n CONAN_MANUAL_OVERRIDES: \"arrow/*:arrow/18.0.0@nanopore/noboost\"\n CONAN_PROFILE_CPPSTD: \"20\"\n artifacts:\n paths:\n - 'conan-*.tgz'\n parallel:\n matrix:\n - CONAN_PROFILE_BUILD_TYPE: [\"Debug\", \"Release\"]\n\n.conan2-upload:\n extends: .conan2-common\n stage: deploy\n #only: [\"tags\"]\n script:\n - for f in conan-*.tgz; do conan cache restore \"$f\"; done\n - conan remote auth ONT-Conan-V2 --force\n - conan upload \"*:*\" --check --confirm --remote=ONT-Conan-V2 --dry-run\n\n.conan-upload:\n extends: .upload-package # from informatics/conan-config\n stage: deploy\n only: [\"tags\"]\n before_script:\n - pip install \"cmake==${CMAKE_VERSION}\"\n - !reference [\".conan-setup-common\"]\n variables:\n EXPECTED_PACKAGE_COUNT: \"4\" # Expect shared and static packages\n\n# Conan: build and upload packages:\nbuild-conan:windows-x86_64-vs2019:\n extends:\n - .profile-windows-x86_64-vs2019\n - .build-package-win\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\", \"win-x64-msvc2019-release-build\"]\nupload-conan:windows-x86_64-vs2019:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:windows-x86_64-vs2019\" ]\n\nbuild-conan2:windows-x86_64-vs2019:\n extends:\n - .conan2-build\n - .profile-windows-x86_64-vs2019-conan2\n needs: [\"prepare_version\", \"win-x64-msvc2019-release-build\"]\nupload-conan2:windows-x86_64-vs2019:\n extends:\n - .conan2-upload\n - .profile-windows-x86_64-vs2019-conan2\n dependencies: [ \"prepare_version\", \"build-conan2:windows-x86_64-vs2019\" ]\n\nbuild-conan:macos-aarch64-appleclang-15.0:\n extends:\n - .profile-macos-aarch64-appleclang-15.0\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\", \"osx-arm64-clang15-release-build\"]\nupload-conan:macos-aarch64-appleclang-15.0:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:macos-aarch64-appleclang-15.0\" ]\n\nbuild-conan:macos-aarch64-appleclang-16.0:\n extends:\n - .profile-macos-aarch64-appleclang-16.0\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:macos-aarch64-appleclang-16.0:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:macos-aarch64-appleclang-16.0\" ]\n\nbuild-conan2:macos-aarch64-appleclang-15.0:\n extends:\n - .profile-macos-aarch64-appleclang-15.0-conan2\n - .conan2-build\n needs: [\"prepare_version\", \"osx-arm64-clang15-release-build\"]\nupload-conan2:macos-aarch64-appleclang-15.0:\n extends:\n - .profile-macos-aarch64-appleclang-15.0-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:macos-aarch64-appleclang-15.0\" ]\n\nbuild-conan2:macos-aarch64-appleclang-16.0:\n extends:\n - .profile-macos-aarch64-appleclang-16.0-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:macos-aarch64-appleclang-16.0:\n extends:\n - .profile-macos-aarch64-appleclang-16.0-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:macos-aarch64-appleclang-16.0\" ]\n\nbuild-conan:linux-x86_64-gcc11:\n extends:\n - .profile-linux-x86_64-gcc11\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc11:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc11\" ]\n\nbuild-conan2:linux-x86_64-gcc11:\n extends:\n - .profile-linux-x86_64-gcc11-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-x86_64-gcc11:\n extends:\n - .profile-linux-x86_64-gcc11-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-x86_64-gcc11\" ]\n\nbuild-conan2:linux-x86_64-gcc11-asan-static:\n extends:\n - .profile-linux-x86_64-gcc11-asan-static-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-x86_64-gcc11-asan-static:\n extends:\n - .profile-linux-x86_64-gcc11-asan-static-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-x86_64-gcc11-asan-static\" ]\n\nbuild-conan2:linux-x86_64-gcc11-usan-static:\n extends:\n - .profile-linux-x86_64-gcc11-usan-static-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-x86_64-gcc11-usan-static:\n extends:\n - .profile-linux-x86_64-gcc11-usan-static-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-x86_64-gcc11-usan-static\" ]\n\nbuild-conan2:linux-x86_64-gcc11-tsan-static:\n extends:\n - .profile-linux-x86_64-gcc11-tsan-static-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-x86_64-gcc11-tsan-static:\n extends:\n - .profile-linux-x86_64-gcc11-tsan-static-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-x86_64-gcc11-tsan-static\" ]\n\nbuild-conan:linux-x86_64-gcc13:\n extends:\n - .profile-linux-x86_64-gcc13\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc13:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc13\" ]\n\nbuild-conan2:linux-x86_64-gcc13:\n extends:\n - .profile-linux-x86_64-gcc13-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-x86_64-gcc13:\n extends:\n - .profile-linux-x86_64-gcc13-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-x86_64-gcc13\" ]\n\nbuild-conan:linux-x86_64-gcc11-asan-static:\n extends:\n - .profile-linux-x86_64-gcc11-asan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc11-asan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc11-asan-static\" ]\n\nbuild-conan:linux-x86_64-gcc13-asan-static:\n extends:\n - .profile-linux-x86_64-gcc13-asan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc13-asan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc13-asan-static\" ]\n\nbuild-conan:linux-x86_64-gcc11-tsan-static:\n extends:\n - .profile-linux-x86_64-gcc11-tsan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc11-tsan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc11-tsan-static\" ]\n\nbuild-conan:linux-x86_64-gcc13-tsan-static:\n extends:\n - .profile-linux-x86_64-gcc13-tsan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc13-tsan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc13-tsan-static\" ]\n\nbuild-conan:linux-x86_64-gcc11-usan-static:\n extends:\n - .profile-linux-x86_64-gcc11-usan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc11-usan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc11-usan-static\" ]\n\nbuild-conan:linux-x86_64-gcc13-usan-static:\n extends:\n - .profile-linux-x86_64-gcc13-usan-static\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-x86_64-gcc13-usan-static:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-x86_64-gcc13-usan-static\" ]\n\nbuild-conan:linux-aarch64-gcc11:\n extends:\n - .profile-linux-aarch64-gcc11\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-aarch64-gcc11:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-aarch64-gcc11\" ]\n\nbuild-conan2:linux-aarch64-gcc11:\n extends:\n - .profile-linux-aarch64-gcc11-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-aarch64-gcc11:\n extends:\n - .profile-linux-aarch64-gcc11-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-aarch64-gcc11\" ]\n\nbuild-conan:linux-aarch64-gcc13:\n extends:\n - .profile-linux-aarch64-gcc13\n - .build-package\n - .conan-build-common\n - .build-shared-and-static\n needs: [\"prepare_version\"]\nupload-conan:linux-aarch64-gcc13:\n extends: .conan-upload\n dependencies: [ \"prepare_version\", \"build-conan:linux-aarch64-gcc13\" ]\n\nbuild-conan2:linux-aarch64-gcc13:\n extends:\n - .profile-linux-aarch64-gcc13-conan2\n - .conan2-build\n needs: [\"prepare_version\"]\nupload-conan2:linux-aarch64-gcc13:\n extends:\n - .profile-linux-aarch64-gcc13-conan2\n - .conan2-upload\n dependencies: [ \"prepare_version\", \"build-conan2:linux-aarch64-gcc13\" ]\n\n# ======================================\n#\n# Archive\n#\n# ======================================\n\n\nbuild-archive:\n stage: archive\n needs:\n - linux-x64-gcc9-release-build\n - linux-aarch64-gcc9-release-build\n - osx-arm64-clang15-release-build\n - win-x64-msvc2019-release-build\n - build-python-api\n script:\n - find .\n artifacts:\n name: \"${CI_JOB_NAME}-artifacts\"\n paths:\n - ./*.tar.gz\n - ./*.whl\n\n\n# ======================================\n#\n# Deploy\n#\n# ======================================\n\n\ninternal_wheel_upload:\n stage: deploy\n image: ${UPLOAD_PYTHON_IMAGE}\n needs:\n - build-archive\n script:\n - ls -lh .\n - pip install twine\n - twine upload *.whl pod5*.tar.gz\n only: [\"tags\"]\n when: manual\n\nexternal_wheel_upload:\n stage: deploy\n image: ${UPLOAD_PYTHON_IMAGE}\n needs:\n - build-archive\n script:\n - ls -lh .\n - pip install twine\n - unset TWINE_REPOSITORY_URL\n - unset TWINE_CERT\n - twine upload lib*.whl -u __token__ -p\"${EXTERNAL_LIB_POD5_PYPI_KEY}\"\n - twine upload pod5*.whl pod5*.tar.gz -u __token__ -p\"${EXTERNAL_POD5_PYPI_KEY}\"\n only: [\"tags\"]\n when: manual\n\n\n# ======================================\n#\n# MLHub Testing\n#\n# ======================================\n\nmlhub:\n stage: deploy\n image: ${MLHUB_TRIGGER_IMAGE}\n needs: [\"build-archive\"]\n variables:\n GIT_STRATEGY: none\n script:\n - |\n curl -i --header \"Content-Type: application/json\" \\\n --request POST \\\n --data '{\n \"key\": \"'${MLHUB_TRIGGER_KEY}'\",\n \"job_name\": \"POD5-CI '${CI_COMMIT_REF_NAME}' - '\"$CI_COMMIT_TITLE\"' \",\n \"script_parameters\": {\n \"mode\":\"artifact\",\n \"source\":\"'${CI_COMMIT_SHA}'\"\n \"python_ver\":\"'${PYTHON_VERSION}'\"\n }\n }' \\\n ${MLHUB_TRIGGER_URL}\n when: manual\n extends:\n - .parallel-py-versions\n" }, { "path": ".gitmodules", "content": "[submodule \"third_party/pybind11\"]\n\tpath = third_party/pybind11\n\turl = https://github.com/pybind/pybind11.git\n\tbranch = stable\n" }, { "path": ".pre-commit-config.yaml", "content": "# See https://pre-commit.com for more information\n# See https://pre-commit.com/hooks.html for more hooks\nrepos:\n - repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v5.0.0\n hooks:\n - id: trailing-whitespace\n - id: end-of-file-fixer\n - id: check-case-conflict\n - id: check-merge-conflict\n - id: check-added-large-files\n - repo: https://github.com/psf/black\n rev: 25.1.0\n hooks:\n - id: black\n - repo: https://github.com/codespell-project/codespell\n rev: v2.4.1\n hooks:\n - id: codespell\n exclude: 'third_party/'\n - repo: https://github.com/PyCQA/flake8\n rev: 7.2.0\n hooks:\n - id: flake8\n exclude: docs/conf.py\n - repo: https://github.com/shellcheck-py/shellcheck-py\n rev: v0.10.0.1\n hooks:\n - id: shellcheck\n - repo: https://github.com/pre-commit/mirrors-clang-format\n rev: 'v20.1.4'\n hooks:\n - id: clang-format\n exclude: 'third_party/'\n - repo: https://github.com/pre-commit/mirrors-mypy\n rev: 'v1.15.0'\n hooks:\n - id: mypy\n files: 'python/pod5/src/'\n args: [ --check-untyped-defs, --ignore-missing-imports ]\n additional_dependencies:\n - types-Deprecated\n - types-setuptools\n - types-pytz\n\n# NB: by default, pre-commit only installs the pre-commit hook (\"commit\" stage),\n# but you can tell `pre-commit install` to install other hooks.\n# This set of default stages ensures we don't slow down or break other git operations\n# even if you install hooks for them.\ndefault_stages:\n - pre-commit\n - pre-merge-commit\n - manual\n\n# vi:et:sw=2:sts=2:\n" }, { "path": ".readthedocs.yaml", "content": "# Read the Docs configuration file\n# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details\n\n# Required\nversion: 2\n\n# Build all formats\nformats: all\n\n# Set the version of Python and other tools you might need\nbuild:\n os: ubuntu-20.04\n tools:\n python: \"3.10\"\n jobs:\n pre_build:\n - python -c \"import pod5; print(pod5.__version__)\"\n# Build documentation in the docs/ directory with Sphinx\nsphinx:\n configuration: docs/conf.py\n\n# If using Sphinx, optionally build your docs in additional formats such as PDF\n# formats:\n# - pdf\n\n# Optionally declare the Python requirements required to build your docs\npython:\n install:\n - requirements: docs/requirements.txt\n" }, { "path": "CHANGELOG.md", "content": "\n\n# Changelog\n\nAll notable changes, updates, and fixes to pod5 will be documented here\n\nThe format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)\nand this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n## [0.3.39]\n\n### Fixed\n\n- Fix python bindings build without Conan.\n\n### Changed\n\n- CI Stabilisation\n\n## [0.3.38]\n\n### Changed\n\n- CI Stabilisation\n\n## [0.3.37]\n\n### Changed\n\n- Use standard file IO to read POD5 header and footer metadata before memory mapping (if not disabled e.g. `POD5_DISABLE_MMAP_OPEN=1`). This should to avoid SIGBUS errors caused by memory mapping file stubs (archive artefacts).\n- Improve file_reader_writer unit-tests robustness\n- Scale number of open input file handles during pod5 subset / filter by the system limits and number of output files.\n\n### Fixed\n\n- Fixed bug where invalid read ids could be passed into pod5 subset via summary table.\n- Fixed bug where invalid read ids in `DatasetReader.reads` selection could return valid read records.\n- Fixed bug in CI where the python venv was not activated resulting in incorrect conan version being used.\n\n## [0.3.36]\n\n### Added\n\n- Added missing licence files.\n\n### Removed\n\n- Removed Python 3.9 and macOS 10.15 support since they're EOL.\n\n## [0.3.35]\n\n### Added\n\n- Python 3.14 support\n\n### Removed\n\n- Removed sphinx-style python docstrings references\n- Removed most documentation as part of migration to \n- Deprecated `--duplicate-ok` argument from pod5 tools - duplicating reads is now always invalid.\n\n### Changed\n\n- Moved filter + subset implementation into C++ for improved performance.\n- Performance improvements to `pod5 view` especially when reading read ids from large files.\n- Updated polars version from \"~=1.20,<1.32\" to \"~= 1.30\"\n- Switch to uv for managing CI python environments\n- Updated to pyarrow 22.0.0\n\n## [0.3.34]\n\n### Added\n\n- `open_pore_level` to `pod5 inspect read`\n\n### Changed\n\n- Fixed migration behaviour on nfs systems, where migrated tables could be left orphaned on disk.\n- Limited polars install version to \"~=1.20,<1.32\" following breaking changes\n- Tidied up how tmp files are named, used a larger set of numbers for naming.\n\n## [0.3.33]\n\n### Added\n\n- Added Conan 2 CI\n\n### Changed\n\n- Reduced virtual memory usage when opening POD5 files by 75%.\n- Python API now memory maps inner tables using the `mmap.mmap` `offset` and `length` arguments directly instead of taking a slice of the whole file.\n\n## [0.3.32]\n\n### Added\n\n- Option to allow users of C++ API to not keep file handles open if required.\n\n### Changed\n\n- Order of `pod5 view` is backwards compatible with 0.3.30, new `open_pore_level` field is at the end of the list.\n\n## [0.3.31]\n\n### Added\n\n- Added new field `open_pore_level`, containing the level of the open pore as tracked by MinKNOW for this channel/well.\n\n### Removed\n\n- Deprecated support for unused read scaling values \"tracked_scaling_scale\", \"tracked_scaling_shift\", \"predicted_scaling_scale\", \"predicted_scaling_shift\", \"num_reads_since_mux_change\" and \"time_since_mux_change\". These will be removed from stored data and writer API in 0.4.0, with accessing API remaining in place until 0.5.0.\n\n## [0.3.30]\n\n### Changed\n\n- Build with sanitization on GCC13\n\n### Removed\n\n- Dropped incorrect sanitized conan jobs.\n\n## [0.3.29]\n\n### Removed\n\n- Dropped support for macOS x86\n\n## [0.3.28]\n\n### Changed\n\n- Additional testing for Linux file access.\n\n## [0.3.27]\n\n### Fixed\n\n- Fixed some crashes when parsing corrupt POD5 files.\n- Fixed missing error handling when the C API is called incorrectly.\n- Fixed and clarified C API thread safety.\n- Fall back to regular IO if direct IO is requested, but file opening fails.\n\n### Removed\n\n- Dropped automated ARM+GCC8 builds.\n\n### Changed\n\n- Bumped polars to next major version (`~= 1.20`).\n\n## [0.3.26]\n\n### Changed\n\n- The read end reason now includes paused - for reads that ended because acquisition was paused.\n\n## [0.3.25]\n\n### Changed\n\n- Python 3.8 wheels are no longer built for Windows or macOS (Python 3.8 is end-of-life).\n- Better error messages and testing of file recovery.\n\n### Added\n\n- Conan pod5 builds with address, thread and undefined behaviour sanitizer support.\n- Added fuzz testing.\n- New option cleanup temporary files after file recovery.\n\n## [0.3.24]\n\n### Changed\n\n- Update to arrow 18 for the cpp library.\n\n### Fixed\n\n- Flush `pod5 view` header to prevent issue on Windows systems where header would not be on top.\n\n## [0.3.23]\n\n### Changed\n\n- Removed use of python `build` when building wheel in cmake.\n\n## [0.3.22]\n\n### Added\n\n- `ArrowTableHandle` `stream` member to store the `BatchFileReader` backend\n- `ArrowTableHandle` `options` argument to pass in `IpcReadOptions`\n- `pod5::default_memory_pool` function which selects an appropriate memory pool even on large page systems.\n\n### Changed\n\n- Refactored Multi-threading in `DatasetReader` to prevent too many open files errors\n- Updated dependency to `pyarrow~=18.0.0` for `python>=3.9`\n- Relaxed h5py python dependency\n\n## [0.3.21]\n\n### Added\n\n- Support for python 3.13.\n\n### Changed\n\n- Removed use of Boost. This does not affect the C interface, but may require changes to\n consumers of the C++ headers.\n\n## [0.3.20]\n\n### Changed\n\n- Refactored directio writing engine to open up async io support.\n- Fixed Boost version compatibility checking in Conan packages.\n\n## [0.3.19]\n\n### Added\n\n- New end reason for reads terminated due to an analysis configuration change.\n\n### Changed\n\n- Reduced allocations when compressing signal.\n\n### Fixed\n\n- Crash when searching empty file for reads.\n\n## [0.3.18]\n\n### Added\n\n- Ability to disable flushing on batch complete\n- Use new LinuxOutputStream to cache allocations and reduce memory when writing many files.\n\n## [0.3.17]\n\n### Changed\n\n- Move svb headers to correct subdirectory in\n\n## [0.3.16]\n\n### Added\n\n- svb16 headers packaged with pod5\n\n### Changed\n\n- Directio output now writes on batch complete without flushing explicitly.\n\n## [0.3.15]\n\n### Added\n\n- Added new end reasons \"api_request\" and \"device_data_error\" to allow for new read end reasons future minknow versions will generate.\n- Allow directio to specify the chunk size directly.\n\n## [0.3.14]\n\n### Added\n\n- gcc8 builds\n\n## [0.3.13]\n\n### Fixed\n\n- Instability when creating a pod5 writer fails.\n- Issue with directio mode where space is over reserved.\n\n## [0.3.12]\n\n### Fixed\n\n- Fixed issues reading signal from uncompressed pod5 files.\n\n## [0.3.11]\n\n### Added\n\n- Typechecking on `Writer.add_reads` to inform users incorrectly passing `ReadRecords`\n- Compatibility with numpy 2.0.\n\n### Fixed\n\n- `DatasetReader` correctly handles string paths\n\n## [0.3.10]\n\n### Added\n\n- Required pypa project metadata.\n\n### Removed\n\n- Dropped support OSX builds for XCode < 14.2.\n\n## [0.3.9]\n\n### Fixed\n\n- `ReadRecord.to_read()` missing fields\n\n## [0.3.8]\n\n### Fixed\n\n- Conan windows upload jobs failure due to using different line endings.\n\n## [0.3.7]\n\n### Fixed\n\n- CI package uploading to PyPi following [API token migration](https://pypi.org/help/#apitoken).\n- Documentation for some functions.\n- Explicitly sized type in `pod5_vbz_decompress_signal()`.\n- CI execution of tests.\n\n### Changed\n\n- Updated `pre-commit` to `clang-format-17`.\n- Updated Arrow to 12.0.0.\n\n## [0.3.6]\n\n### Fixed\n\n- Polars `ColumnNotFoundError: not_set` introduced by `polars==0.20.0`\n\n## [0.3.5]\n\n### Fixed\n\n- Arrow build flags in conanfile are now configured in the configure() fnc rather than being default options.\n\n## [0.3.4]\n\n### Added\n\n- boost_internal_build flag in conanfile.\n- CI now builds with the above flag turned on.\n\n## [0.3.3]\n\n### Added\n\n- CI for appleclang 14\n- cppstd builds\n\n## [0.3.2]\n\n### Added\n\n- Support for Python 3.12\n\n## [0.3.1] 2023-11-10\n\n### Fixed\n\n- Logging no longer calls `basicConfig` which may unintentionally edit users logging configuration\n\n## [0.3.0] 2023-11-07\n\n### Changed\n\n- Transfers dataframes used in subsetting / filter use categorical fields to reduce memory consumption\n- Polars version increased to `~=0.19`\n- Documentation regarding positional arguments\n- Renamed deprecated `polars.groupby` to `polars.group_by`\n\n### Fixed\n\n- Fixed a bug in the build scripts that prevented iOS and Windows Conan packages from being uploaded.\n- Remove exposed artifactory URL env var from gitlab ci config.\n- `convert to_fast5` writes byte encoded read_ids to match Minkow (was `str`)\n\n### Removed\n\n- Removed python3.7 support\n\n## [0.2.9] 2023-11-02\n\n### Fixed\n\n- Corrected the visibility of dependencies when building pod5 as a shared library.\n\n## [0.2.8] 2023-11-01\n\n### Added\n\n- Added compression status to `pod5 inspect summary `\n- Added environment override \"POD5_DISABLE_MMAP_OPEN\" to force non-mmapped opening of files.\n\n### Fixed\n\n- Remove exposed artifactory URL env var from gitlab ci config.\n- `convert to_fast5` writes byte encoded read_ids to match Minkow (was `str`)\n\n## [0.2.7] 2023-09-11\n\n### Added\n\n- `DatasetReader` class for reading collections of pod5 files\n- Return index errors when querying invalid errors from API's\n\n### Changed\n\n- Recursive search for files now traverses symbolic links and ignores hidden files\n- Tweak block size of directio writes to 1MB.\n\n## [0.2.6] 2023-09-04\n\n### Changed\n\n- Write pod5 files using DirectIO on Linux platforms (performance)\n\n## [0.2.5] 2023-08-01\n\n### Added\n\n- Shared builds to conan\n\n### Fixed\n\n- `num_minknow_events` field description from `int8` to `uint64`\n- `ReadRecord.num_minknow_events` return type-hint from `float` to `int`\n\n## [0.2.4] 2023-07-13\n\n### Changed\n\n- Increased `numpy` minimum version to `>= 1.21.0`\n- Improved performance of `subset`, `filter` and `merge` tools.\n- `Repacker.wait` and `Repacker.waiter` parameters\n\n### Deprecated\n\n- `Repacker.wait` and `Repacker.waiter` some parameters are deprecated and issue `FutureWarning`\n\n### Fixed\n\n- `Repacker.is_complete` returning `True` when work is queued.\n\n## [0.2.3] 2023-06-26\n\n### Added\n\n- Add API (pod5_open_file_options) to prevent pod5 from opening a file using mmap, instead using direct file IO.\n- Default field values (empty string) when converting fast5 files with missing fields\n\n### Changed\n\n- Corrected Oxford Nanopore Technologies company name in package metadata to use Public Limited Company (Plc) instead of Limited (Ltd)\n- Limited the number of processes created when specifying `--threads` to the number of cpu cores available `os.cpu_count()`\n- Reduced the default value for `--threads` from 8 to 4 to improve stability on resource constrained systems\n\n## [0.2.2] 2023-06-06\n\n### Fixed\n\n- Add API error when adding reads with invalid end reason, pore type or run info.\n\n## [0.2.1] 2023-05-25\n\n### Changed\n\n- Update internal arrow lib to not export flatbuffers symbols.\n\n## [0.2.0] 2023-05-18\n\n### Added\n\n- `pod5 view` tool to view / inspect pod5 files as tables. Gives a >200x speed improvement compared to `pod5 inspect reads`\n- `pod5 recover` tool to recover data from corrupted / truncated pod5 files\n- `pod5 update` documentation\n- source distributions to pypi\n\n### Changed\n\n- `pod5 subset` and `pod5 filter` uses `polars` to parse inputs\n- `pod5 subset` and `pod5 filter` csv formatting requirements tightened\n- `pod5` tools which use multiple pod5 file inputs now accept directories which can be searched recursively with `-r/--recursive`\n- `pod5 subset` `--read-id-column` argument abbreviateion `-r` change to `-R` to allow `-r/--recursive` to be consistent for all tools\n- `pod5` tools use hyphens in all arguments (e.g. `--force-overwrite` and `--read-id-column`)\n- `pod5 merge` and `pod5 update` uses named `-o/--output` argument instead of positional `output` argument to standardise tools\n- `pod5 update` progress bar and better detection of name conflicts\n- Minimised number of open file handles in tools to prevent `Too many open files` error\n- Logging added to `merge`, `filter` and `subset`. Enabled with `POD5_DEBUG=1`\n\n### Deprecated\n\n- `pod5 inspect reads` deprecated in-favour of `pod5 view`\n\n### Fixed\n\n- Exception raised when calling `pod5` without any arguments\n- Exception raised in `pod5 convert fast5` where closed writers were reopened after being closed by a caught exception\n- Fixed Gitlab 38, pod5_get_end_reason and pod5_get_pore_type ignoring input string length checks.\n\n### Removed\n\n- `pod5 subset` `--json` mapping arguments\n- `pod5 merge` `--chunk-size` argument\n- `ReadTableVersion` replaced with an integer value\n\n## [0.1.21] 2023-04-27\n\n### Fixed\n\n- Repacker `reads_completed` value while copying a selection of reads.\n- Fixed crash when trying to load files with a bad footer.\n\n## [0.1.20] 2023-04-20\n\n### Fixed\n\n- Fixed merging many files running out the size limit of dictionary indices.\n\n## [0.1.19] 2023-04-14\n\n### Changed\n\n- `pod5 convert fast5` now creates logs when `POD5_DEBUG=1` set\n- `pod5 convert fast5` checks multi-read fast5s at conversion time\n\n### Fixed\n\n- Fixed memory usage growth over time as signal was loaded with large pod5 files.\n- Fixed crash loading malicious files (found via fuzz testing)\n- Fixed leaks and UB when running unit tests.\n- Fixed run-away memory consumption during fast5 conversion\n\n## [0.1.17] 2023-04-06\n\n### Changed\n\n- Updated internal arrow version to 8.0.0.3\n\n## [0.1.16] 2023-04-06\n\n### Fixed\n\n- Fixed issue where pod5 would read out of bounds memory when decompressing some reads.\n\n## [0.1.15] 2023-03-31\n\n### Changed\n\n- Refactored `pod5 convert fast5` to use `concurrent.futures` only.\n- Add further info to error message when signal cannot be decompressed by zstd\n- Make merge operation not generate multiple identical run infos.\n\n### Fixed\n\n- Fixed closing uninitialised file handles.\n- Fixed `pod5 inspect reads` repeating header\n- Fixed a crash with certain pod5 search operations.\n\n## [0.1.13] 2023-03-23\n\n### Fixed\n\n- Fix loading large pod5 files on virtual-memory limited systems.\n\n## [0.1.12] 2023-03-20\n\n### Added\n\n- Added `--output` argument to `pod5 convert fast5` and `to_fast5` replacing positional argument of the same name\n- Added `--strict` argument to `pod5 convert fast5` to promptly stop on exceptions\n- Added readthedocs documentation links in README.md\n\n### Changed\n\n- Updated developer installation instructions to use `conan<2`\n- Reworked `pod5 convert fast5` to tolerate runtime exceptions\n- Use same type `run_info_index_t` for `pod5_get_file_run_info_count` and `pod5_get_file_run_info`.\n\n### Fixed\n\n- Fixed file handle leak in repacker\n\n## [0.1.11] 2023-03-13\n\n### Added\n\n- Python API supports python 3.11\n- Added missing python API wheels on windows\n\n### Changed\n\n- Changed python API dependency version `pyarrow~=11.0.0` from `8.0.0` to support python 3.11\n- Changed python API dependency version `hdf5~=8.0.0` from `v7.0.0` to support python 3.11\n\n## [0.1.10] 2023-03-09\n\n### Added\n\n- Added `pod5_get_read_count` to find the count of all reads in file\n- Added `pod5_get_read_ids` to retrieve all read id's in file\n- Added `pod5_get_file_run_info` to retrieve a run info at an absolute index in the file\n- Added `pod5_free_run_info` to free run info's (replaces `pod5_release_run_info`)\n- Added `pod5_get_file_run_info_count` to find the number of run info's in a file\n- Added `pod5 filter` tool to subset pod5 files with simple list of read ids\n- Added `tqdm` progress bar to `pod5 subset` (disable with `POD5_PBAR=0`)\n\n### Changed\n\n- Reworked `pod5 subset` to give better control over resources used\n- `pod5 subset` can now parse csv and tsv tables / summaries\n- `pod5 repack` now repacks all inputs one-to-one\n\n### Deprecated\n\n- Deprecated `pod5_release_run_info` (see `pod5_free_run_info`)\n\n### Removed\n\n- Removed filepath header line from `pod5 inspect reads`\n\n## [0.1.9] 2023-03-07\n\n### Added\n\n- Added version attributes to `lib-pod5`\n\n### Changed\n\n- Versioning now controlled by VCS inspection using `setuptools_scm`\n\n## [0.1.8] 2023-02-23\n\n### Added\n\n- Added more `read_id` getter methods to `Reader`\n- Added support for python 3.8 + 3.10 on windows\n- Added gcc7 linux build of pod5\n\n### Changed\n\n- Update to zlib 1.2.13\n- Update to zstd 1.5.4\n- Pinned `pre-commit=v2.21.0` while supporting `python3.7`\n- Reworked `pod5 convert to_fast5` output filenames to allow for `1-1` mapping\n\n### Fixed\n\n- Fixed `pod5 inspect read`\n- Fixed `pod5 convert to_fast5` creating an empty fast5 output\n- Fixed `pod5 convert to_fast5` ignoring the `--force_overwrite` argument\n- Fixed issue where thread_pool.h wasn't shipped.\n\n## [0.1.5] - 2023-01-20\n\n### Added\n\n- Explicitly re-exported `lib-pod5` public symbols and added `py.typed` marker file to support type-checking.\n\n### Fixed\n\n- Fixed issue where closing many pod5 files in sequence is slow.\n- Fixed incorrect python types and adopted python type-checking.\n\n## [0.1.4] - 2022-12-22\n\n### Added\n\n- Linux python 3.11 wheels\n- ReadTheDocs documentation support\n\n### Fixed\n\n- OSX arm64 wheel naming corrections - works with wider set of python executables\n\n## [0.1.3] - 2022-12-16\n\n### Added\n\n- Added `Reader.__iter__` method.\n\n### Changed\n\n- Renamed `EndReason.name` to `EndReason.reason` to access the inner enum and added\n `EndReason.name` as a property to return the string representation of this enum value.\n- `BaseRead`, `Read`, `CompressedRead`, `Calibration` and `Pore` dataclasses are now mutable.\n\n### Removed\n\n- Removed deprecated `Writer` functions.\n\n### Fixed\n\n- Fixed osx arm64 wheel compatibility for older python versions.\n- Fixed EndReason type errors.\n- Fixed EndReason in pod5 to fast5 conversion.\n\n## [0.1.2] - 2022-12-06\n\n### Changed\n\n- Optimised the file writing utilities\n\n## [0.1.1] - 2022-12-06\n\n### Changed\n\n- Restricted exported boost dependencies of conan package to just the boost::headers component.\n\n## [0.1] - 2022-12-02\n\n### Changed\n\n- Documentation edits\n- `Writer.add_reads` now handles both `Read` and `CompressedRead`.\n\n### Deprecated\n\n- Deprecated `Writer` methods `add_read_object` and `add_read_objects` for `add_read` and `add_reads` respectively.\n\n### Removed\n\n- Removed direct pod5 tool scripts.\n\n### Fixed\n\n- Fixed name of internal utils - \"pad_file\".\n- Fixed spelling of various internal variables.\n- Fixed `pod5 convert to_fast5`\n\n## [0.0.43]\n\n### Changed\n\n- Reformat c++ code with more consistent format file.\n\n## [0.0.42]\n\n### Added\n\n- Added `pod5` tools entry-point\n- Added api to query file version information as written on disk.\n\n### Changed\n\n- Fixed signal_chunk_size type error in convert-from-fast5\n- Replaced `ont_fast5_api` dependency with `vbz_h5py_plugin`\n- Restructured Python packaging to include `lib_pod5_format` which contains the native bindings build from pybind11.\n- `pod5_format` and `pod5_format_tools` are now pure python packages which depend on `lib_pod5_format`\n- Python packages `pod5_format` and `pod5_format_tools` have been merged into single `pod5` pure-python package.\n- `pod5-convert-from-fast5` `--output-one-to-one` reworked so that output files maintain the input structure making this argument more flexible and avoid filename clobbering.\n- Added missing `lib_pod5.update_file` function to pyi.\n- `pod5-convert-from-fast5` `output` now takes existing directories and\n writes `output.pod5` (current behaviour) or creates a new file with the given name if it doesn't exist.\n- Renamed arguments in tools relating to multi-processing / multi-threading from `-p/--processes` to the mode common `-t/--threads`.\n\n## [0.0.41] - 2022-10-27\n\n### Changed\n\n- Fixed pod5-inspect erroring when loading data.\n- Fixed issue where some files in between 0.34 - 0.38 wouldn't load correctly.\n\n## [0.0.40] - 2022-10-27\n\n### Changed\n\n- Fixed migrating of large files from older versions.\n\n## [0.0.39] - 2022-10-18\n\n### Changed\n\n- Fixed building against the c++ api - previously missing include files.\n\n## [0.0.38] - 2022-10-18\n\n### Changed\n\n- All data in the read table that was previously contained in dictionaries of structs is now stored in the read table, or a new \"run info\" table.\n This change simplifies data access into the pod5 files, and helps users who want to convert the pod5 data to pandas or other arrow-compatible reader formats.\n Old data is migrated on load, and will continue to work, data can be permanently migrated using the tool `pod5-migrate`\n\n### Removed\n\n- Support for opening and writing \"split\" pod5 files. All API's now expect and return combined pod5 files.\n\n## [0.0.37] - 2022-10-18\n\n### Changed\n\n- Updated Conan recipe to support building without specifying C++ standard version.\n\n## [0.0.36] - 2022-10-07\n\n### Changed\n\n- Bump the Boost and Arrow versions to pick up latest changes.\n\n## [0.0.35] - 2022-10-07\n\n### Changed\n\n- Support C++17 + C++20 with the conan package pod5 generates.\n\n## [0.0.34] - 2022-10-05\n\n### Changed\n\n- Modified `pod5_format_tools/pod5_convert_to_fast5.py` to separate `pod5_convert_to_fast5_argparser()` and `convert_from_fast5()` out from `pod5_convert_from_fast5.main()`.\n\n## [0.0.33] - 2022-10-05\n\n### Added\n\n- Added `num_samples` field to read table, containing the total number of samples a read contains. The field is filled in by API if it doesn't exist.\n\n### Changed\n\n- File version is now V2, due to the addition of `num_samples`.\n\n## [0.0.32] - 2022-10-03\n\n### Fixed\n\n- Fixed an issue where multi-threaded access to a single batch could cause a crash discovered by dorado testing.\n- Fixed help text in convert to fast5 script.\n" }, { "path": "CMakeLists.txt", "content": "cmake_minimum_required(VERSION 3.18.0)\n\nproject(POD5)\n\ninclude(${PROJECT_SOURCE_DIR}/cmake/POD5Version.cmake)\n\nset(CMAKE_PROJECT_VERSION ${POD5_NUMERIC_VERSION})\n\nset(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} \"${PROJECT_SOURCE_DIR}/cmake\")\n\n# use compiler cache if available\noption(DISABLE_CCACHE \"Do not try to use ccache to speed compilation\" NO)\nif (NOT DISABLE_CCACHE)\n find_program(CCACHE_EXECUTABLE ccache\n HINTS \"C:/Program\\ Files/ccache/\"\n )\n if (CCACHE_EXECUTABLE)\n message(STATUS \"Using ccache: ${CCACHE_EXECUTABLE}\")\n set(CMAKE_CXX_COMPILER_LAUNCHER \"${CCACHE_EXECUTABLE}\")\n set(CMAKE_C_COMPILER_LAUNCHER \"${CCACHE_EXECUTABLE}\")\n endif()\nendif()\n\nif (NOT DEFINED ENABLE_CONAN)\n option(ENABLE_CONAN \"Enable conan for dependency installation\" OFF)\nendif()\n\nif (NOT DEFINED CONAN2)\n option(CONAN2 \"Temp flag until we fully migrate to conan2\" OFF)\nendif()\n\noption(BUILD_SHARED_LIB \"Build a shared library\" OFF)\n\noption(POD5_DISABLE_TESTS \"Disable building all tests\" ON)\noption(POD5_BUILD_EXAMPLES \"Enable building all examples\" OFF)\n\noption(ENABLE_ADDRESS_SANITIZER \"Enable address sanitizer\" OFF)\n\nif (NOT DEFINED ENABLE_POD5_PACKAGING)\n option(ENABLE_POD5_PACKAGING \"Enable packaging support\" ON)\nendif()\n\noption(BUILD_PYTHON_WHEEL \"Build a python wheel for pod5\" OFF)\n\n# debug symbols don't depend on the build type, only on this option\noption(DISABLE_DEBUG_SYMBOLS \"Force debug symbols to be disabled\" OFF)\nif (NOT DISABLE_DEBUG_SYMBOLS)\n if (MSVC)\n # Z7 embeds deubgging info into .obj files, which is easier to manage for\n # build accelerators (note that a .pdb will still be generated for libs)\n # https://docs.microsoft.com/en-us/cpp/build/reference/z7-zi-zi-debug-information-format\n add_compile_options(/Z7)\n # this will use fastlink in the IDE and full link from the command link\n # https://docs.microsoft.com/en-us/cpp/build/reference/debug-generate-debug-info\n set(CMAKE_EXE_LINKER_FLAGS \"${CMAKE_EXE_LINKER_FLAGS} /DEBUG\")\n set(CMAKE_MODULE_LINKER_FLAGS \"${CMAKE_EXE_LINKER_FLAGS} /DEBUG\")\n set(CMAKE_SHARED_LINKER_FLAGS \"${CMAKE_EXE_LINKER_FLAGS} /DEBUG\")\n # /DEBUG option is not recognised for STATIC lib linking\n elseif (CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES \"Clang\")\n add_compile_options(-g)\n endif()\nendif()\n\noption(ENABLE_FUZZERS \"Build the fuzzers that can be used to catch new issues\" OFF)\nif (ENABLE_FUZZERS)\n include(pod5_fuzz)\nendif()\nadd_compile_definitions(POD5_ENABLE_FUZZERS=$)\n\noption(ENABLE_COVERAGE_REPORT \"Executables emit coverage reports\" OFF)\nif (ENABLE_COVERAGE_REPORT)\n if (DISABLE_DEBUG_SYMBOLS)\n message(FATAL_ERROR \"Debug symbols are required for coverage reports to work\")\n elseif (NOT CMAKE_BUILD_TYPE STREQUAL \"Debug\")\n message(FATAL_ERROR \"Only unoptimised builds give reliable coverage reports\")\n elseif (CMAKE_CXX_COMPILER_ID MATCHES \"(GNU|Clang)\")\n add_compile_options(--coverage)\n add_link_options(--coverage)\n else()\n message(FATAL_ERROR \"Cannot enable coverage on unknown compiler\")\n endif()\nendif()\n\n\n# FIXME: DISABLE CONDITIONAL TO WORK ON BIONIC\nif (ENABLE_CONAN AND CMAKE_COMPILER_IS_GNUCXX AND\n CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL \"9.0\" AND\n CMAKE_CXX_COMPILER_VERSION VERSION_LESS \"10.0\")\n # We build POD5 on CentOS 7 in CI, where we have GCC 9 but only the pre-C++11 ABI\n # See https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html\n # This forces GCC 9 on other platforms (eg: Ubuntu Focal) to use the same ABI.\n # The main gain here is being able to use the same conan packages.\n add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=0)\nendif()\n\nif(ENABLE_ADDRESS_SANITIZER)\n add_compile_options(\"-fsanitize=address\")\n add_link_options(\"-fsanitize=address\")\nendif()\n\ninclude_directories(\"third_party/include\")\n\nforeach (config \"Release\" \"Debug\")\n string(TOUPPER \"${config}\" config_upper)\n set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY_${config_upper} ${CMAKE_BINARY_DIR}/${config}/lib)\n set(CMAKE_LIBRARY_OUTPUT_DIRECTORY_${config_upper} ${CMAKE_BINARY_DIR}/${config}/lib)\n set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_${config_upper} ${CMAKE_BINARY_DIR}/${config}/bin)\nendforeach()\n\nset(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME \"archive\")\n\ninclude(GenerateExportHeader)\n\nenable_testing()\n\nif (BUILD_PYTHON_WHEEL)\n find_package(Python ${PYTHON_VERSION} EXACT COMPONENTS Interpreter Development)\n set(PYBIND11_FINDPYTHON ON)\n add_subdirectory(third_party/pybind11)\n install(\n FILES third_party/pybind11/LICENSE\n DESTINATION licenses\n RENAME pybind11.txt\n )\nendif()\n\nadd_subdirectory(c++)\n\n# The fuzz directory contains both the fuzzers and the regression runners,\n# the latter of which can be built as ordinary tests.\nif (ENABLE_FUZZERS OR NOT POD5_DISABLE_TESTS)\n add_subdirectory(fuzz)\nendif()\n\n# Install licenses.\ninstall(\n DIRECTORY ${CMAKE_BINARY_DIR}/pod5_conan_licenses/\n DESTINATION licenses\n)\ninstall(\n FILES\n LICENSE.md\n third_party/licenses/gsl-lite.txt\n DESTINATION\n licenses\n)\n\nif (ENABLE_POD5_PACKAGING)\n include(pod5_packaging)\nendif()\n" }, { "path": "CMakePresets.json", "content": "{\n \"version\": 4,\n \"include\": [\n \"cmake/presets/conan-provider.json\",\n \"cmake/presets/conan-build-options.json\",\n \"cmake/presets/conan-profiles.json\"\n ],\n \"configurePresets\": [\n {\n \"name\": \"conan2-linux-gcc9-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc9-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-x86_64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc9-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc9-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-x86_64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc9-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc13-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-x86_64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc13-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc13-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-x86_64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc13-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-x86_64-cppstd20-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-x86_64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-asan-static-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-asan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-asan-static-x86_64-cppstd20-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-asan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-usan-static-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-usan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-usan-static-x86_64-cppstd20-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-usan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-tsan-static-x86_64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-tsan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-tsan-static-x86_64-cppstd20-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-tsan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-asan-static-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-asan-static-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-asan-static-x86_64-cppstd17-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-asan-static-x86_64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-usan-static-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-usan-static-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-usan-static-x86_64-cppstd17-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-usan-static-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-tsan-static-x86_64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-tsan-static-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-tsan-static-x86_64-cppstd17-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-tsan-static-x86_64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-aarch64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc13-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-aarch64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc13-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-aarch64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc13-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc13-aarch64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc13-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-aarch64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-aarch64-cppstd20-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-aarch64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc11-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc11-aarch64-cppstd17-release\",\n \"hidden\": false,\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc11-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-aarch64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc9-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-aarch64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc9-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-aarch64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-gcc9-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-linux-gcc9-aarch64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-gcc9-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-15.0-aarch64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-appleclang-15.0-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-15.0-aarch64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-appleclang-15.0-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-15.0-aarch64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-appleclang-15.0-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-15.0-aarch64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-appleclang-15.0-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-16.0-aarch64-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-appleclang-16.0-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-16.0-aarch64-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-appleclang-16.0-aarch64-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-16.0-aarch64-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-appleclang-16.0-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-macos-appleclang-16.0-aarch64-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-appleclang-16.0-aarch64-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-windows-x86_64-vs2019-cppstd17-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-windows-x86_64-vs2019-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-windows-x86_64-vs2019-cppstd17-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-windows-x86_64-vs2019-profile\",\n \"conan2-cppstd17\"\n ]\n },\n {\n \"name\": \"conan2-windows-x86_64-vs2019-cppstd20-debug\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-debug\",\n \"conan2-windows-x86_64-vs2019-profile\",\n \"conan2-cppstd20\"\n ]\n },\n {\n \"name\": \"conan2-windows-x86_64-vs2019-cppstd20-release\",\n \"inherits\": [\n \"conan2-provider\",\n \"conan2-release\",\n \"conan2-windows-x86_64-vs2019-profile\",\n \"conan2-cppstd20\"\n ]\n }\n ]\n}\n" }, { "path": "DEV.md", "content": "Development\n===========\n\nIf you want to contribute to pod5_file_format, or our pre-built binaries do not meet your platform requirements, you can build pod5 from source using the instructions in `docs/install.rst`.\n\n### Developing\n\nBuilding the project requires several tools and libraries are available:\n\n- CMake\n- Arrow\n- Zstd\n- Flatbuffers\n- Python\n- setuptools_scm\n\n```bash\n# Docs on installing arrow from here: https://arrow.apache.org/install/\n> sudo apt install -y -V ca-certificates lsb-release wget\n> wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\n> sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb\n> sudo apt update\n# Now install the rest of the dependencies:\n> sudo apt install cmake libzstd-dev libzstd-dev libflatbuffers-dev libarrow-dev=12.0.1-1\n> pip install setuptools_scm~=7.1\n# Finally start build of POD5:\n> git clone https://github.com/nanoporetech/pod5-file-format.git\n> cd pod5-file-format\n> git submodule update --init --recursive\n> python -m setuptools_scm\n> python ./pod5_make_version.py\n> mkdir build\n> cd build\n> cmake ..\n> make -j\n```\n\n### Pre commit\n\nThe project uses pre-commit to ensure code is consistently formatted, you can set this up using pip:\n\n```bash\n> pip install pre-commit==v2.21.0\n# Install pre-commit hooks in your pod5-file-format repo:\n> cd pod5-file-format\n> pre-commit install\n# Run hooks on all files:\n> pre-commit run --all-files\n```\n\nPython Development\n==================\n\nAfter completing the required build stages above, to create a Python virtual environment for development follow the instructions below .\n\n```bash\n\n> cd python\n> make install\n\n```\n" }, { "path": "LICENSE.md", "content": "This Source Code Form is subject to the terms of the Mozilla Public\nLicense, v. 2.0. If a copy of the MPL was not distributed with this\nfile, You can obtain one at https://mozilla.org/MPL/2.0/.\n\n©2021 Oxford Nanopore Technologies PLC.\n\n\nMozilla Public License Version 2.0\n==================================\n\n### 1. Definitions\n\n**1.1. “Contributor”**\n means each individual or legal entity that creates, contributes to\n the creation of, or owns Covered Software.\n\n**1.2. “Contributor Version”**\n means the combination of the Contributions of others (if any) used\n by a Contributor and that particular Contributor's Contribution.\n\n**1.3. “Contribution”**\n means Covered Software of a particular Contributor.\n\n**1.4. “Covered Software”**\n means Source Code Form to which the initial Contributor has attached\n the notice in Exhibit A, the Executable Form of such Source Code\n Form, and Modifications of such Source Code Form, in each case\n including portions thereof.\n\n**1.5. “Incompatible With Secondary Licenses”**\n means\n\n* **(a)** that the initial Contributor has attached the notice described\n in Exhibit B to the Covered Software; or\n* **(b)** that the Covered Software was made available under the terms of\n version 1.1 or earlier of the License, but not also under the\n terms of a Secondary License.\n\n**1.6. “Executable Form”**\n means any form of the work other than Source Code Form.\n\n**1.7. “Larger Work”**\n means a work that combines Covered Software with other material, in\n a separate file or files, that is not Covered Software.\n\n**1.8. “License”**\n means this document.\n\n**1.9. “Licensable”**\n means having the right to grant, to the maximum extent possible,\n whether at the time of the initial grant or subsequently, any and\n all of the rights conveyed by this License.\n\n**1.10. “Modifications”**\n means any of the following:\n\n* **(a)** any file in Source Code Form that results from an addition to,\n deletion from, or modification of the contents of Covered\n Software; or\n* **(b)** any new file in Source Code Form that contains any Covered\n Software.\n\n**1.11. “Patent Claims” of a Contributor**\n means any patent claim(s), including without limitation, method,\n process, and apparatus claims, in any patent Licensable by such\n Contributor that would be infringed, but for the grant of the\n License, by the making, using, selling, offering for sale, having\n made, import, or transfer of either its Contributions or its\n Contributor Version.\n\n**1.12. “Secondary License”**\n means either the GNU General Public License, Version 2.0, the GNU\n Lesser General Public License, Version 2.1, the GNU Affero General\n Public License, Version 3.0, or any later versions of those\n licenses.\n\n**1.13. “Source Code Form”**\n means the form of the work preferred for making modifications.\n\n**1.14. “You” (or “Your”)**\n means an individual or a legal entity exercising rights under this\n License. For legal entities, “You” includes any entity that\n controls, is controlled by, or is under common control with You. For\n purposes of this definition, “control” means **(a)** the power, direct\n or indirect, to cause the direction or management of such entity,\n whether by contract or otherwise, or **(b)** ownership of more than\n fifty percent (50%) of the outstanding shares or beneficial\n ownership of such entity.\n\n\n### 2. License Grants and Conditions\n\n#### 2.1. Grants\n\nEach Contributor hereby grants You a world-wide, royalty-free,\nnon-exclusive license:\n\n* **(a)** under intellectual property rights (other than patent or trademark)\n Licensable by such Contributor to use, reproduce, make available,\n modify, display, perform, distribute, and otherwise exploit its\n Contributions, either on an unmodified basis, with Modifications, or\n as part of a Larger Work; and\n* **(b)** under Patent Claims of such Contributor to make, use, sell, offer\n for sale, have made, import, and otherwise transfer either its\n Contributions or its Contributor Version.\n\n#### 2.2. Effective Date\n\nThe licenses granted in Section 2.1 with respect to any Contribution\nbecome effective for each Contribution on the date the Contributor first\ndistributes such Contribution.\n\n#### 2.3. Limitations on Grant Scope\n\nThe licenses granted in this Section 2 are the only rights granted under\nthis License. No additional rights or licenses will be implied from the\ndistribution or licensing of Covered Software under this License.\nNotwithstanding Section 2.1(b) above, no patent license is granted by a\nContributor:\n\n* **(a)** for any code that a Contributor has removed from Covered Software;\n or\n* **(b)** for infringements caused by: **(i)** Your and any other third party's\n modifications of Covered Software, or **(ii)** the combination of its\n Contributions with other software (except as part of its Contributor\n Version); or\n* **(c)** under Patent Claims infringed by Covered Software in the absence of\n its Contributions.\n\nThis License does not grant any rights in the trademarks, service marks,\nor logos of any Contributor (except as may be necessary to comply with\nthe notice requirements in Section 3.4).\n\n#### 2.4. Subsequent Licenses\n\nNo Contributor makes additional grants as a result of Your choice to\ndistribute the Covered Software under a subsequent version of this\nLicense (see Section 10.2) or under the terms of a Secondary License (if\npermitted under the terms of Section 3.3).\n\n#### 2.5. Representation\n\nEach Contributor represents that the Contributor believes its\nContributions are its original creation(s) or it has sufficient rights\nto grant the rights to its Contributions conveyed by this License.\n\n#### 2.6. Fair Use\n\nThis License is not intended to limit any rights You have under\napplicable copyright doctrines of fair use, fair dealing, or other\nequivalents.\n\n#### 2.7. Conditions\n\nSections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted\nin Section 2.1.\n\n\n### 3. Responsibilities\n\n#### 3.1. Distribution of Source Form\n\nAll distribution of Covered Software in Source Code Form, including any\nModifications that You create or to which You contribute, must be under\nthe terms of this License. You must inform recipients that the Source\nCode Form of the Covered Software is governed by the terms of this\nLicense, and how they can obtain a copy of this License. You may not\nattempt to alter or restrict the recipients' rights in the Source Code\nForm.\n\n#### 3.2. Distribution of Executable Form\n\nIf You distribute Covered Software in Executable Form then:\n\n* **(a)** such Covered Software must also be made available in Source Code\n Form, as described in Section 3.1, and You must inform recipients of\n the Executable Form how they can obtain a copy of such Source Code\n Form by reasonable means in a timely manner, at a charge no more\n than the cost of distribution to the recipient; and\n\n* **(b)** You may distribute such Executable Form under the terms of this\n License, or sublicense it under different terms, provided that the\n license for the Executable Form does not attempt to limit or alter\n the recipients' rights in the Source Code Form under this License.\n\n#### 3.3. Distribution of a Larger Work\n\nYou may create and distribute a Larger Work under terms of Your choice,\nprovided that You also comply with the requirements of this License for\nthe Covered Software. If the Larger Work is a combination of Covered\nSoftware with a work governed by one or more Secondary Licenses, and the\nCovered Software is not Incompatible With Secondary Licenses, this\nLicense permits You to additionally distribute such Covered Software\nunder the terms of such Secondary License(s), so that the recipient of\nthe Larger Work may, at their option, further distribute the Covered\nSoftware under the terms of either this License or such Secondary\nLicense(s).\n\n#### 3.4. Notices\n\nYou may not remove or alter the substance of any license notices\n(including copyright notices, patent notices, disclaimers of warranty,\nor limitations of liability) contained within the Source Code Form of\nthe Covered Software, except that You may alter any license notices to\nthe extent required to remedy known factual inaccuracies.\n\n#### 3.5. Application of Additional Terms\n\nYou may choose to offer, and to charge a fee for, warranty, support,\nindemnity or liability obligations to one or more recipients of Covered\nSoftware. However, You may do so only on Your own behalf, and not on\nbehalf of any Contributor. You must make it absolutely clear that any\nsuch warranty, support, indemnity, or liability obligation is offered by\nYou alone, and You hereby agree to indemnify every Contributor for any\nliability incurred by such Contributor as a result of warranty, support,\nindemnity or liability terms You offer. You may include additional\ndisclaimers of warranty and limitations of liability specific to any\njurisdiction.\n\n\n### 4. Inability to Comply Due to Statute or Regulation\n\nIf it is impossible for You to comply with any of the terms of this\nLicense with respect to some or all of the Covered Software due to\nstatute, judicial order, or regulation then You must: **(a)** comply with\nthe terms of this License to the maximum extent possible; and **(b)**\ndescribe the limitations and the code they affect. Such description must\nbe placed in a text file included with all distributions of the Covered\nSoftware under this License. Except to the extent prohibited by statute\nor regulation, such description must be sufficiently detailed for a\nrecipient of ordinary skill to be able to understand it.\n\n\n### 5. Termination\n\n**5.1.** The rights granted under this License will terminate automatically\nif You fail to comply with any of its terms. However, if You become\ncompliant, then the rights granted under this License from a particular\nContributor are reinstated **(a)** provisionally, unless and until such\nContributor explicitly and finally terminates Your grants, and **(b)** on an\nongoing basis, if such Contributor fails to notify You of the\nnon-compliance by some reasonable means prior to 60 days after You have\ncome back into compliance. Moreover, Your grants from a particular\nContributor are reinstated on an ongoing basis if such Contributor\nnotifies You of the non-compliance by some reasonable means, this is the\nfirst time You have received notice of non-compliance with this License\nfrom such Contributor, and You become compliant prior to 30 days after\nYour receipt of the notice.\n\n**5.2.** If You initiate litigation against any entity by asserting a patent\ninfringement claim (excluding declaratory judgment actions,\ncounter-claims, and cross-claims) alleging that a Contributor Version\ndirectly or indirectly infringes any patent, then the rights granted to\nYou by any and all Contributors for the Covered Software under Section\n2.1 of this License shall terminate.\n\n**5.3.** In the event of termination under Sections 5.1 or 5.2 above, all\nend user license agreements (excluding distributors and resellers) which\nhave been validly granted by You or Your distributors under this License\nprior to termination shall survive termination.\n\n\n### 6. Disclaimer of Warranty\n\n> Covered Software is provided under this License on an “as is”\n> basis, without warranty of any kind, either expressed, implied, or\n> statutory, including, without limitation, warranties that the\n> Covered Software is free of defects, merchantable, fit for a\n> particular purpose or non-infringing. The entire risk as to the\n> quality and performance of the Covered Software is with You.\n> Should any Covered Software prove defective in any respect, You\n> (not any Contributor) assume the cost of any necessary servicing,\n> repair, or correction. This disclaimer of warranty constitutes an\n> essential part of this License. No use of any Covered Software is\n> authorized under this License except under this disclaimer.\n\n### 7. Limitation of Liability\n\n> Under no circumstances and under no legal theory, whether tort\n> (including negligence), contract, or otherwise, shall any\n> Contributor, or anyone who distributes Covered Software as\n> permitted above, be liable to You for any direct, indirect,\n> special, incidental, or consequential damages of any character\n> including, without limitation, damages for lost profits, loss of\n> goodwill, work stoppage, computer failure or malfunction, or any\n> and all other commercial damages or losses, even if such party\n> shall have been informed of the possibility of such damages. This\n> limitation of liability shall not apply to liability for death or\n> personal injury resulting from such party's negligence to the\n> extent applicable law prohibits such limitation. Some\n> jurisdictions do not allow the exclusion or limitation of\n> incidental or consequential damages, so this exclusion and\n> limitation may not apply to You.\n\n\n### 8. Litigation\n\nAny litigation relating to this License may be brought only in the\ncourts of a jurisdiction where the defendant maintains its principal\nplace of business and such litigation shall be governed by laws of that\njurisdiction, without reference to its conflict-of-law provisions.\nNothing in this Section shall prevent a party's ability to bring\ncross-claims or counter-claims.\n\n\n### 9. Miscellaneous\n\nThis License represents the complete agreement concerning the subject\nmatter hereof. If any provision of this License is held to be\nunenforceable, such provision shall be reformed only to the extent\nnecessary to make it enforceable. Any law or regulation which provides\nthat the language of a contract shall be construed against the drafter\nshall not be used to construe this License against a Contributor.\n\n\n### 10. Versions of the License\n\n#### 10.1. New Versions\n\nMozilla Foundation is the license steward. Except as provided in Section\n10.3, no one other than the license steward has the right to modify or\npublish new versions of this License. Each version will be given a\ndistinguishing version number.\n\n#### 10.2. Effect of New Versions\n\nYou may distribute the Covered Software under the terms of the version\nof the License under which You originally received the Covered Software,\nor under the terms of any subsequent version published by the license\nsteward.\n\n#### 10.3. Modified Versions\n\nIf you create software not governed by this License, and you want to\ncreate a new license for such software, you may create and use a\nmodified version of this License if you rename the license and remove\nany references to the name of the license steward (except to note that\nsuch modified license differs from this License).\n\n#### 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses\n\nIf You choose to distribute Source Code Form that is Incompatible With\nSecondary Licenses under the terms of this version of the License, the\nnotice described in Exhibit B of this License must be attached.\n\n## Exhibit A - Source Code Form License Notice\n\n This Source Code Form is subject to the terms of the Mozilla Public\n License, v. 2.0. If a copy of the MPL was not distributed with this\n file, You can obtain one at https://mozilla.org/MPL/2.0/.\n\nIf it is not possible or desirable to put the notice in a particular\nfile, then You may include the notice in a location (such as a LICENSE\nfile in a relevant directory) where a recipient would be likely to look\nfor such a notice.\n\nYou may add additional accurate notices of copyright ownership.\n\n## Exhibit B - “Incompatible With Secondary Licenses” Notice\n\n This Source Code Form is \"Incompatible With Secondary Licenses\", as\n defined by the Mozilla Public License, v. 2.0.\n" }, { "path": "README.md", "content": "[![Documentation Status](https://readthedocs.org/projects/pod5-file-format/badge/?version=latest)](https://pod5-file-format.readthedocs.io/)\n\nPOD5 File Format\n================\n\nPOD5 File Format\n================\n\nPOD5 is a file format for storing nanopore dna data in an easily accessible way.\nThe format is able to be written in a streaming manner which allows a sequencing\ninstrument to directly write the format.\n\nData in POD5 is stored using [Apache Arrow](https://github.com/apache/arrow), allowing\nusers to consume data in many languages using standard tools.\n\nWhat does this project contain\n------------------------------\n\nThis project contains a core library for reading and writing POD5 data, and a toolkit for\naccessing this data in other languages.\n\nDocumentation\n-------------\n\nFull documentation is found at https://pod5-file-format.readthedocs.io/\n\n\nUsage\n-----\n\nPOD5 is also bundled as a python module for easy use in scripts, a user can install using:\n\n```bash\n> pip install pod5\n```\n\nThis python module provides the python library to write custom scripts against.\n\nPlease see [examples](./python/pod5/examples) for documentation on using the library.\n\nThe `pod5` package also provides [a selection of tools](./python/pod5/README.md).\n\n\nDesign\n------\n\nFor information about the design of POD5, see the [docs](./docs/README.md).\n\nDevelopment\n-----------\n\nIf you want to contribute to pod5_file_format, or our pre-built binaries do not meet your platform requirements, you can build pod5 from source using the instructions in [DEV.md](DEV.md)\n" }, { "path": "benchmarks/.gitignore", "content": "*/outputs/\nimage/*.whl\n" }, { "path": "benchmarks/README.md", "content": "POD5 Benchmarks\n==============\n\nBuilding the benchmark environment\n----------------------------------\n\nTo run benchmarks you first have to build the docker environment to run them:\n\n```bash\n> ./build.sh\n```\n\n\nRunning a benchmark\n-------------------\n\nTo run a benchmark, use the helper script to start the docker image:\n\n```bash\n> ./run_benchmark.sh convert ./path-to-source-files/\n```\n\n\nBenchmarking Results\n--------------------\n\n Note preliminary results\n\n Results run on:\n 0.0.16 POD5\n pyslow5 dev branch (commit 2643310a)\n\n Benchmark numbers are produced using a GridION.\n\n Note the benchmarks are run using python APIs, more work is required on C benchmarks.\n\n\n## PCR Dataset\n\nOn dataset a PCR Zymo dataset PAM50264, on 10.4.1 e8.2 data (`pcr_zymo/20220419_1706_2E_PAM50264_3c6f33f1`):\n\n### File sizes\n\n| pod5 | blow5 | fast5 |\n|--------|---------|---------|\n| 37G | 37G | 52G |\n\n### Timings\n\n| | pod5 | blow5 | fast5 |\n|-------------------------------------|------------|------------|------------|\n| convert | 197.5 secs | 241.4 secs | Not Run |\n| find all read ids | 10.1 secs | 1.8 secs | 5.2 secs |\n| find all samples | 22.3 secs | 82.5 secs | 520.6 secs |\n| find selected read ids read number | 1.1 secs | 5.8 secs | 387.1 secs |\n| find selected read ids sample count | 1.5 secs | 5.7 secs | 417.8 secs |\n| find selected read ids samples | 5.3 secs | 6.4 secs | 465.6 secs |\n\n```* Note blow5 convert times include the index + merge operation```\n\n\n## InterARTIC Dataset\n\nDataset available at:\nhttps://github.com/Psy-Fer/interARTIC\n\n### File sizes\n\n| pod5 | blow5 | fast5 |\n|--------|---------|---------|\n| 3.3G | 3.4G | 6.9G |\n\n### Timings\n\n| | pod5 | blow5 | fast5 |\n|-------------------------------------|-----------|-----------|-----------|\n| convert | 28.6 secs | 21.0 secs | Not Run |\n| find all read ids | 0.5 secs | 0.5 secs | 0.7 secs |\n| find all samples | 3.0 secs | 8.0 secs | 73.5 secs |\n| find selected read ids read number | 0.4 secs | 1.3 secs | 29.3 secs |\n| find selected read ids sample count | 0.6 secs | 1.3 secs | 30.4 secs |\n| find selected read ids samples | 1.4 secs | 1.3 secs | 37.8 secs |\n\n```* Note blow5 convert times include the index + merge operation```\n" }, { "path": "benchmarks/build.sh", "content": "#!/bin/bash\n\nset -o errexit\nset -o pipefail\nset -o nounset\n# set -o xtrace\n\nscript_dir=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\ncd \"${script_dir}\"\n\ncd image/\ndocker build -t pod5-benchmark-base -f Dockerfile.base .\n" }, { "path": "benchmarks/convert/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\n./tools/fast5_to_single_blow5.sh \"$input_dir\" \"$output_dir\"\n" }, { "path": "benchmarks/convert/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\npod5 convert fast5 \"$input_dir\" --output \"$output_dir\"\n" }, { "path": "benchmarks/find_all_read_ids/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\ntools/pyslow5_tests.py \"${input_dir}\"/blow5/*.blow5 \"${output_dir}\" get_all_read_ids\n" }, { "path": "benchmarks/find_all_read_ids/run_fast5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\ntools/find_and_get_fast5.py \"${input_dir}/fast5\" \"${output_dir}\"\n" }, { "path": "benchmarks/find_all_read_ids/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\n./tools/find_and_get_pod5.py \"${input_dir}/pod5\" \"${output_dir}\"\n" }, { "path": "benchmarks/find_all_samples/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\ntools/pyslow5_tests.py \"${input_dir}\"/blow5/*.blow5 \"${output_dir}\" all_values --get-column samples\n" }, { "path": "benchmarks/find_all_samples/run_fast5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\ntools/find_and_get_fast5.py \"${input_dir}/fast5\" \"${output_dir}\" --get-column samples\n" }, { "path": "benchmarks/find_all_samples/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\n./tools/find_and_get_pod5.py \"${input_dir}/pod5\" \"${output_dir}\" --get-column samples\n" }, { "path": "benchmarks/find_selected_read_ids_read_number/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/pyslow5_tests.py \"${input_dir}\"/blow5/*.blow5 \"${output_dir}\" sample_values --get-column read_number --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_read_number/run_fast5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/find_and_get_fast5.py \"${input_dir}/fast5\" \"${output_dir}\" --get-column read_number --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_read_number/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\ntype_output_dir=$2\nfull_output_dir=$3\n\n./tools/find_and_get_pod5.py \"${input_dir}/pod5\" \"${type_output_dir}\" --get-column read_number --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_sample_count/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/pyslow5_tests.py \"${input_dir}\"/blow5/*.blow5 \"${output_dir}\" sample_values --get-column sample_count --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_sample_count/run_fast5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/find_and_get_fast5.py \"${input_dir}/fast5\" \"${output_dir}\" --get-column sample_count --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_sample_count/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\ntype_output_dir=$2\nfull_output_dir=$3\n\n./tools/find_and_get_pod5.py \"${input_dir}/pod5\" \"${type_output_dir}\" --get-column sample_count --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_samples/run_blow5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/pyslow5_tests.py \"${input_dir}\"/blow5/*.blow5 \"${output_dir}\" sample_values --get-column samples --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_samples/run_fast5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\nfull_output_dir=$3\n\ntools/find_and_get_fast5.py \"${input_dir}/fast5\" \"${output_dir}\" --get-column samples --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/find_selected_read_ids_samples/run_pod5.sh", "content": "#!/bin/bash\n\ninput_dir=$1\ntype_output_dir=$2\nfull_output_dir=$3\n\n./tools/find_and_get_pod5.py \"${input_dir}/pod5\" \"${type_output_dir}\" --get-column samples --select-ids \"${full_output_dir}/selected_read_ids.csv\"\n" }, { "path": "benchmarks/image/Dockerfile.base", "content": "FROM ubuntu:20.04\n\nRUN apt update && apt install -y wget python3 python3-pip git libzstd-dev\n\nRUN wget https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.1/ont-vbz-hdf-plugin_1.0.1-1.focal_amd64.deb && apt install -y ./ont-vbz-hdf-plugin_1.0.1-1.focal_amd64.deb && rm ont-vbz-hdf-plugin_1.0.1-1.focal_amd64.deb\n\nCOPY ./requirements-benchmarks.txt /\nRUN pip install -r /requirements-benchmarks.txt\n\nCOPY ./install_slow5.sh /\nRUN /install_slow5.sh\nENV PATH=\"/slow5tools-v0.4.0/:$PATH\"\n\nRUN pip install numpy\n\nCOPY ./pod5*.whl /\nRUN pip install *.whl && rm *.whl\n" }, { "path": "benchmarks/image/install_slow5.sh", "content": "#!/bin/bash\n\nset -e\n\n: \"${SLOW_5_TOOLS_VERSION:=v1.3.0}\"\n: \"${SLOW_5_LIB_VERSION:=v1.3.1}\"\n\napt update\napt install -y libzstd-dev libhdf5-dev\n\nwget \"https://github.com/hasindu2008/slow5tools/releases/download/${SLOW_5_TOOLS_VERSION}/slow5tools-${SLOW_5_TOOLS_VERSION}-release.tar.gz\"\ntar xvf \"slow5tools-${SLOW_5_TOOLS_VERSION}-release.tar.gz\"\nrm \"slow5tools-${SLOW_5_TOOLS_VERSION}-release.tar.gz\"\n(\n cd \"slow5tools-${SLOW_5_TOOLS_VERSION}\"\n ./configure\n make zstd=1 -j \"$(nproc)\"\n)\n\n# pyslow5 must be built with zstd support for fair comparison (otherwise default zlib is slower than zstd)\ngit clone -b \"${SLOW_5_LIB_VERSION}\" https://github.com/hasindu2008/slow5lib\n\n(\n cd slow5lib/\n\n echo \"Installing numpy\"\n pip install numpy\n\n make pyslow5 -j \"$(nproc)\" 2> build_log.txt || (cat build_log.txt && exit)\n echo \"Installing pyslow5\"\n PYSLOW5_ZSTD=1 pip install dist/*.tar.gz\n\n # adding slow5 C API benchmarks\n make zstd=1 -j \"$(nproc)\" && test/bench/build.sh\n)\n" }, { "path": "benchmarks/image/requirements-benchmarks.txt", "content": "h5py\nnumpy\npandas\ntabulate\n" }, { "path": "benchmarks/run_benchmarks.py", "content": "#!/usr/bin/env python3\n\"\"\"\nExample usage:\n```\n> taskset -c 0-10 ./benchmarks/run_benchmarks.py ./input_files/ \\\n ./benchmark-outputs/ --skip-to-benchmark find_all_samples\n```\n\"\"\"\n\nimport argparse\nimport json\nimport shutil\nimport subprocess\nimport time\nfrom collections import namedtuple\nfrom pathlib import Path\n\nimport tabulate\n\nBenchmark = namedtuple(\n \"Benchmark\",\n [\"name\", \"file_types\", \"checks\", \"input_benchmark\", \"pre_run\", \"post_run_fixup\"],\n)\n\nBENCHMARK_ROOT = Path(__file__).resolve().parent\n\nPOD5_FILE_TYPE = \"pod5\"\nBLOW5_FILE_TYPE = \"blow5\"\nFAST5_FILE_TYPE = \"fast5\"\nALL_FILE_TYPES = [POD5_FILE_TYPE, BLOW5_FILE_TYPE, FAST5_FILE_TYPE]\n\n\ndef du(path):\n \"\"\"disk usage in human readable format (e.g. '2,1GB')\"\"\"\n return subprocess.check_output([\"du\", \"-sh\", path]).split()[0].decode(\"utf-8\")\n\n\ndef generate_report(input_dir, output_dir, timing_results):\n report = \"\"\n\n skipped_benchmarks = len(ALL_BENCHMARKS) - len(timing_results)\n\n skipped = \"\"\n if skipped_benchmarks != 0:\n skipped = f\", skipped {skipped_benchmarks}\"\n report += f\"Ran {len(timing_results)} benchmarks{skipped}\\n\\n\"\n report += f\"Input data was {input_dir}\\n\\n\"\n\n report += \"File sizes\\n\"\n report += \"----------\\n\\n\"\n\n convert_output_dir = output_dir / \"convert\"\n sizes = []\n for file_type in ALL_FILE_TYPES:\n file_type_dir = convert_output_dir / file_type\n if file_type_dir.exists:\n sizes.append(du(file_type_dir))\n else:\n sizes.append(\"Not Run\")\n report += (\n tabulate.tabulate([sizes], headers=ALL_FILE_TYPES, tablefmt=\"github\") + \"\\n\\n\"\n )\n\n report += \"Timings\\n\"\n report += \"-------\\n\\n\"\n results = []\n for benchmark in ALL_BENCHMARKS:\n row = [benchmark.name.replace(\"_\", \" \")]\n results.append(row)\n\n if benchmark.name in timing_results:\n timings = timing_results[benchmark.name]\n for file_type in ALL_FILE_TYPES:\n if file_type in timings:\n row.append(f\"{timings[file_type]:.1f} secs\")\n else:\n row.append(\"Not Run\")\n else:\n for file_type in ALL_FILE_TYPES:\n row.append(\"Not Run\")\n\n results_headers = [\"\"] + ALL_FILE_TYPES\n report += (\n tabulate.tabulate(results, headers=results_headers, tablefmt=\"github\") + \"\\n\"\n )\n return report\n\n\ndef check_read_ids(benchmark, file_types, output_dir, only_format):\n if only_format is not None:\n print(\"Not checking read ids - only one format executed\")\n return\n\n csv_check_files = []\n for file_type in file_types:\n csv_check_files.append(output_dir / file_type / \"read_ids.csv\")\n\n for a, b in zip(csv_check_files[1:], csv_check_files):\n subprocess.run(\n [BENCHMARK_ROOT / \"tools\" / \"check_csvs_consistent.py\", a, b], check=True\n )\n\n\ndef check_file_sizes(benchmark, file_types, output_dir, only_format):\n print(\"File sizes for output dir\")\n subprocess.run([\"du\", \"-sh\"] + list(output_dir.glob(\"*\")), check=True)\n\n\ndef copy_fast5_files(benchmark, input_dir, output_dir):\n shutil.copytree(input_dir, output_dir / \"fast5\")\n\n\ndef randomly_select_read_ids(benchmark, input_dir, output_dir):\n print(\"Randomly selecting read ids for benchmark\")\n\n subprocess.run(\n [\n BENCHMARK_ROOT / \"tools\" / \"find_and_get_pod5.py\",\n input_dir / \"pod5\",\n output_dir,\n ],\n check=True,\n )\n subprocess.run(\n [\n BENCHMARK_ROOT / \"tools\" / \"select-random-ids.py\",\n output_dir / \"read_ids.csv\",\n output_dir / \"selected_read_ids.csv\",\n \"--select-ratio\",\n \"0.1\",\n ],\n check=True,\n )\n\n\nALL_BENCHMARKS = [\n Benchmark(\n \"convert\",\n [POD5_FILE_TYPE, BLOW5_FILE_TYPE],\n [check_file_sizes],\n input_benchmark=None,\n post_run_fixup=copy_fast5_files,\n pre_run=None,\n ),\n Benchmark(\n \"find_all_read_ids\",\n ALL_FILE_TYPES,\n [check_read_ids],\n input_benchmark=\"convert\",\n post_run_fixup=None,\n pre_run=None,\n ),\n Benchmark(\n \"find_all_samples\",\n ALL_FILE_TYPES,\n [check_read_ids],\n input_benchmark=\"convert\",\n post_run_fixup=None,\n pre_run=None,\n ),\n Benchmark(\n \"find_selected_read_ids_read_number\",\n ALL_FILE_TYPES,\n [check_read_ids],\n input_benchmark=\"convert\",\n post_run_fixup=None,\n pre_run=randomly_select_read_ids,\n ),\n Benchmark(\n \"find_selected_read_ids_sample_count\",\n ALL_FILE_TYPES,\n [check_read_ids],\n input_benchmark=\"convert\",\n post_run_fixup=None,\n pre_run=randomly_select_read_ids,\n ),\n Benchmark(\n \"find_selected_read_ids_samples\",\n ALL_FILE_TYPES,\n [check_read_ids],\n input_benchmark=\"convert\",\n post_run_fixup=None,\n pre_run=randomly_select_read_ids,\n ),\n]\n\n\ndef run_benchmark(benchmark, input_dir, output_dir, only_format=None):\n if output_dir.exists():\n print(\"Removing old output dir\")\n shutil.rmtree(output_dir)\n\n file_types = benchmark.file_types if benchmark.file_types else ALL_FILE_TYPES\n\n time_results = {}\n\n if benchmark.pre_run:\n benchmark.pre_run(benchmark, input_dir, output_dir)\n\n for file_type in file_types:\n if only_format is not None and only_format != file_type:\n print(f\"## Skipping for file type {file_type}:\")\n continue\n print(f\"## Running for file type {file_type}:\")\n file_type_output_dir = output_dir / file_type\n file_type_output_dir.mkdir(exist_ok=True, parents=True)\n start = time.time()\n subprocess.run(\n [\n BENCHMARK_ROOT / benchmark.name / f\"run_{file_type}.sh\",\n input_dir,\n file_type_output_dir,\n output_dir,\n ],\n check=True,\n cwd=BENCHMARK_ROOT,\n )\n end = time.time()\n duration_secs = end - start\n time_results[file_type] = duration_secs\n print(f\"## Took {duration_secs:.2f} seconds\")\n\n if benchmark.post_run_fixup:\n benchmark.post_run_fixup(benchmark, input_dir, output_dir)\n\n if benchmark.checks:\n print(\"## Running checks\")\n for check in benchmark.checks:\n check(benchmark, file_types, output_dir, only_format)\n\n return time_results\n\n\ndef run_benchmarks(args):\n timing_results = {}\n\n input_dir = args.input_dir.resolve()\n output_dir = args.output_dir.resolve()\n\n skip_list = []\n if args.skip_to_benchmark:\n found = False\n for benchmark in ALL_BENCHMARKS:\n if benchmark.name == args.skip_to_benchmark:\n found = True\n\n if not found:\n skip_list.append(benchmark.name)\n\n for benchmark in ALL_BENCHMARKS:\n if benchmark.name in skip_list:\n print(f\"# Skipping benchmark {benchmark.name}\")\n continue\n print(f\"# Running benchmark {benchmark.name}:\")\n benchmark_input_dir = input_dir\n if benchmark.input_benchmark:\n benchmark_input_dir = output_dir / benchmark.input_benchmark\n timing_results[benchmark.name] = run_benchmark(\n benchmark,\n benchmark_input_dir,\n output_dir / benchmark.name,\n args.only_format,\n )\n\n report = generate_report(input_dir, output_dir, timing_results)\n print(report)\n\n with open(args.output_dir / \"timings.json\", \"w\") as f:\n f.write(json.dumps(timing_results, indent=2))\n with open(args.output_dir / \"report.md\", \"w\") as f:\n f.write(report)\n\n\ndef main():\n parser = argparse.ArgumentParser(\"Run Benchmarks for POD5 format\")\n parser.add_argument(\"input_dir\", type=Path)\n parser.add_argument(\"output_dir\", type=Path)\n parser.add_argument(\n \"--skip-to-benchmark\",\n type=str,\n help=\"Start benchmarking from a named benchmark\",\n )\n parser.add_argument(\n \"--only-format\",\n type=str,\n help=\"Only run benchmarks for a single format\",\n )\n\n args = parser.parse_args()\n\n run_benchmarks(args)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/run_benchmarks_in_docker.sh", "content": "#!/bin/bash\n\nset -e\n\ninput_dir=$(readlink -f \"$1\")\noutput_dir=\"$(pwd)/pod5-benchmark-outputs\"\nmkdir -p \"${output_dir}\"\n\nscript_dir=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\n\necho \"Running benchmark on input '${input_dir}'\"\n\ndocker run --rm -it -v\"${input_dir}\":/input -v\"${output_dir}\":/outputs -v\"${script_dir}\"/:/benchmarks pod5-benchmark-base /benchmarks/tools/run_benchmarks_docker_entry.sh\n" }, { "path": "benchmarks/tools/check_csvs_consistent.py", "content": "#!/usr/bin/env python3\n\nimport argparse\nimport sys\n\nimport pandas as pd\nfrom pandas.testing import assert_frame_equal\n\n\ndef check_consistency(df1, df2):\n df1 = df1.sort_values(\"read_id\", ignore_index=True)\n df2 = df2.sort_values(\"read_id\", ignore_index=True)\n\n assert_frame_equal(df1, df2)\n\n print(\"Data frames are consistent\")\n sys.exit(0)\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"input_a\")\n parser.add_argument(\"input_b\")\n\n args = parser.parse_args()\n\n a = pd.read_csv(args.input_a)\n b = pd.read_csv(args.input_b)\n\n print(f\"Check consistency of files {args.input_a} and {args.input_b}\")\n check_consistency(a, b)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/tools/fast5_to_single_blow5.sh", "content": "#!/bin/bash\n\ninput_path=$1\noutput_path=$2\n\nmkdir -p \"$output_path\"\n\ntemp_dir=\"${output_path}/tmp\"\nmkdir -p \"$temp_dir\"\n\n# specific options (-c zstd -s svb-zd) must be provided to slow5tools to create compression comparable to vbz\n# also number of processes/threads must be set to 10 to match with default value in pod5_convert\n# however, the svb-zd stream variable byte + zig-zag delta implementation in slow5 mirrors\n# ONT's previous 32 bit zigzag delta, where as pod5 is using a newer 16 bit zigzag delta with SIMD optimisations\n# so pod5 has the added performance benefit of using the newer zigzag delta\n# slow5 compression methods are modular, so we can easily add the new one iff necessary\nslow5tools f2s \"$input_path\" -d \"$temp_dir\" -p 10 -c zstd -s svb-zd\n\n# Most comparable to have one file for both formats:\nslow5tools cat \"$temp_dir -o $output_path/file.blow5\" || slow5tools merge \"$temp_dir\" -o \"$output_path/file.blow5\" -t 10 -c zstd -s svb-zd\n#if the files are from the same run ID, slow5tools cat can be used, which is significantly faster\n#slow5tools cat $temp_dir -o $output_path/file.blow5\n\nrm -r \"$temp_dir\"\n\n# Index will get generated on first test anyway, we should do it now to give best results later:\n# current slow5tools implementation decompresses the whole record for indexing and is not efficient\n# the specification supports partial decompress of the record (also signal chunking if necessary)\nslow5tools index \"$output_path/file.blow5\"\n" }, { "path": "benchmarks/tools/find_and_get_fast5.py", "content": "#!/usr/bin/env python3\n\nimport argparse\nfrom pathlib import Path\n\nimport h5py\nimport numpy\nimport pandas as pd\n\n\ndef select_reads(file, selection):\n if selection is not None:\n for read in selection:\n path = f\"/read_{read}\"\n if path not in file:\n continue\n yield read, path\n else:\n for key in file.keys():\n if key.startswith(\"read_\"):\n yield key[5:], key\n\n\ndef run(input_dir, output, select_read_ids=None, get_columns=[]):\n output.mkdir(parents=True, exist_ok=True)\n\n if select_read_ids is not None:\n print(f\"Selecting {len(select_read_ids)} specific read ids\")\n if get_columns is not None:\n print(f\"Selecting columns: {get_columns}\")\n\n read_ids = []\n extracted_columns = {\"read_id\": read_ids}\n print(f\"Search for input files in {input_dir}\")\n for file in input_dir.glob(\"*.fast5\"):\n print(f\"Searching for reads in {file}\")\n\n file = h5py.File(file, \"r\")\n\n for read_id, read_path in select_reads(file, select_read_ids):\n read_ids.append(read_id)\n\n for c in get_columns:\n if c not in extracted_columns:\n extracted_columns[c] = []\n col = extracted_columns[c]\n\n if c == \"read_number\":\n col.append(file[f\"{read_path}/Raw\"].attrs[\"read_number\"])\n elif c == \"sample_count\":\n col.append(len(file[f\"{read_path}/Raw\"][\"Signal\"]))\n elif c == \"samples\":\n col.append(numpy.sum(file[f\"{read_path}/Raw\"][\"Signal\"]))\n\n df = pd.DataFrame(extracted_columns)\n print(f\"Selected {len(read_ids)} items\")\n df.to_csv(output / \"read_ids.csv\", index=False)\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"input\", type=Path)\n parser.add_argument(\"output\", type=Path)\n parser.add_argument(\n \"--select-ids\",\n type=str,\n help=\"CSV file with a read_id column, listing ids to find in input files\",\n )\n parser.add_argument(\n \"--get-column\",\n default=[],\n nargs=\"+\",\n type=str,\n help=\"Add columns that should be extracted\",\n )\n\n args = parser.parse_args()\n\n select_read_ids = None\n if args.select_ids:\n select_read_ids = pd.read_csv(args.select_ids)[\"read_id\"]\n\n run(\n args.input,\n args.output,\n select_read_ids=select_read_ids,\n get_columns=args.get_column,\n )\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/tools/find_and_get_pod5.py", "content": "#!/usr/bin/env python3\n\nimport argparse\nimport multiprocessing as mp\nimport tempfile\nfrom collections import namedtuple\nfrom pathlib import Path\nfrom queue import Empty\n\nimport numpy\nimport pandas as pd\n\nimport pod5 as p5\n\nSelectReadIdsData = namedtuple(\n \"SelectReadIdsData\", [\"path\", \"slice_start\", \"slice_end\", \"shape\"]\n)\n\n\ndef load_mapped_ids(select_read_ids_data):\n \"\"\"Load a set of read ids from a mmapped file on disk\"\"\"\n select_read_ids_all = numpy.memmap(\n select_read_ids_data.path,\n dtype=numpy.uint8,\n mode=\"r+\",\n shape=select_read_ids_data.shape,\n )\n return select_read_ids_all[\n select_read_ids_data.slice_start : select_read_ids_data.slice_end\n ]\n\n\ndef do_batch_work(filename, batches, column, mode, result_q):\n \"\"\"\n Per process worker to do loading of data from a set of batches\n \"\"\"\n\n read_ids = []\n vals = []\n extracted_columns = {\"read_id\": read_ids, column: vals}\n\n if column == \"samples\":\n file = p5.Reader(filename)\n for batch in file.read_batches(batch_selection=batches, preload={\"samples\"}):\n read_ids.extend(p5.format_read_ids(batch.read_id_column))\n\n for read in batch.reads():\n vals.append(numpy.sum(read.signal))\n else:\n print(f\"Unknown column {column}\")\n result_q.put(pd.DataFrame(extracted_columns))\n\n\ndef do_search_work(files, select_read_ids_data, column, mode, result_q):\n \"\"\"\n Per process worker to do loading of data from a number of read ids\n \"\"\"\n select_read_ids = load_mapped_ids(select_read_ids_data)\n\n read_ids = []\n vals = []\n extracted_columns = {\"read_id\": read_ids, column: vals}\n\n if column == \"samples\":\n for filename in files:\n file = p5.Reader(filename)\n for batch in file.read_batches(select_read_ids, preload={\"samples\"}):\n read_ids.extend(p5.format_read_ids(batch.read_id_column))\n vals.extend([numpy.sum(s) for s in batch.cached_samples_column])\n else:\n print(f\"Unknown column {column}\")\n result_q.put(pd.DataFrame(extracted_columns))\n\n\ndef run_multiprocess(files, output, select_read_ids=None, column=None, mode=None):\n \"\"\"\n Do work across a number of python multiprocesses\n \"\"\"\n mp.set_start_method(\"spawn\")\n\n if select_read_ids is not None:\n print(\"Placing select read id data on disk for mmapping:\")\n numpy_select_read_ids = p5.pack_read_ids(select_read_ids)\n\n # Copy data to memory-map\n fp = tempfile.NamedTemporaryFile()\n fp.close()\n mapped_select_read_ids = numpy.memmap(\n fp.name, dtype=numpy.uint8, mode=\"w+\", shape=numpy_select_read_ids.shape\n )\n numpy.copyto(mapped_select_read_ids, numpy_select_read_ids)\n select_read_ids_mmap_path = Path(fp.name)\n\n result_queue = mp.Queue()\n runners = 10\n\n processes = []\n if select_read_ids is not None:\n approx_chunk_size = max(1, len(select_read_ids) // runners)\n start_index = 0\n while start_index < len(select_read_ids):\n select_read_ids_data = SelectReadIdsData(\n select_read_ids_mmap_path,\n start_index,\n start_index + approx_chunk_size,\n numpy_select_read_ids.shape,\n )\n\n p = mp.Process(\n target=do_search_work,\n args=(files, select_read_ids_data, column, mode, result_queue),\n )\n p.start()\n processes.append(p)\n start_index += approx_chunk_size\n else:\n for filename in files:\n file = p5.Reader(filename)\n batches = list(range(file.batch_count))\n approx_chunk_size = max(1, len(batches) // runners)\n start_index = 0\n while start_index < len(batches):\n select_batches = batches[start_index : start_index + approx_chunk_size]\n p = mp.Process(\n target=do_batch_work,\n args=(filename, select_batches, column, mode, result_queue),\n )\n p.start()\n processes.append(p)\n start_index += len(select_batches)\n\n print(\"Wait for processes...\")\n\n items = []\n while len(items) < len(processes):\n try:\n item = result_queue.get(timeout=0.5)\n items.append(item)\n except Empty:\n continue\n\n for p in processes:\n p.join()\n\n return pd.concat(items)\n\n if select_read_ids is not None:\n select_read_ids_mmap_path.unlink()\n\n\ndef run_get_read_ids(files):\n \"\"\"\n Load all read ids from the file.\n \"\"\"\n read_ids = []\n for filename in files:\n file = p5.Reader(filename)\n for batch in file.read_batches():\n read_ids.extend(p5.format_read_ids(batch.read_id_column))\n return pd.DataFrame({\"read_id\": read_ids})\n\n\ndef run_select(files, select_read_ids, column):\n \"\"\"\n Load column from a specific set of read ids\n \"\"\"\n read_ids = []\n vals = []\n extracted_columns = {\"read_id\": read_ids, column: vals}\n\n for filename in files:\n file = p5.Reader(filename)\n if column == \"sample_count\":\n for batch in file.read_batches(select_read_ids, preload={\"sample_count\"}):\n read_id_selection = batch.read_id_column\n read_ids.extend(p5.format_read_ids(read_id_selection))\n vals.extend(batch.cached_sample_count_column)\n else:\n col_name = f\"{column}_column\"\n for batch in file.read_batches(select_read_ids):\n read_id_selection = batch.read_id_column\n read_ids.extend(p5.format_read_ids(read_id_selection))\n\n read_number_selection = getattr(batch, col_name)\n vals.extend(read_number_selection)\n\n return pd.DataFrame(extracted_columns)\n\n\ndef run_batched(files, column):\n \"\"\"\n Load column from a all reads\n \"\"\"\n read_ids = []\n vals = []\n extracted_columns = {\"read_id\": read_ids, column: vals}\n\n for filename in files:\n file = p5.Reader(filename)\n if column == \"sample_count\":\n for batch in file.read_batches(preload={\"sample_count\"}):\n read_ids.extend(p5.format_read_ids(batch.read_id_column))\n vals.extend(batch.cached_sample_count_column)\n else:\n col_name = f\"{column}_column\"\n for batch in file.read_batches():\n read_ids.extend(p5.format_read_ids(batch.read_id_column))\n vals.extend(getattr(batch, col_name).to_numpy())\n\n return pd.DataFrame(extracted_columns)\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"input\", type=Path)\n parser.add_argument(\"output\", type=Path)\n parser.add_argument(\n \"--select-ids\",\n type=str,\n help=\"CSV file with a read_id column, listing ids to find in input files\",\n )\n parser.add_argument(\n \"--get-column\",\n default=None,\n type=str,\n help=\"Add column that should be extracted\",\n )\n args = parser.parse_args()\n\n select_read_ids = None\n if args.select_ids:\n select_read_ids = pd.read_csv(args.select_ids)[\"read_id\"]\n\n if select_read_ids is not None:\n print(f\"Selecting {len(select_read_ids)} specific read ids\")\n if args.get_column is not None:\n print(f\"Selecting column: {args.get_column}\")\n\n mode = None\n\n print(f\"Search for input files in {args.input}\")\n files = list(args.input.glob(\"*.pod5\"))\n print(f\"Searching in {[str(f) for f in files]}\")\n\n # Run benchmark using most appropriate method:\n if args.get_column is None:\n df = run_get_read_ids(files)\n elif args.get_column == \"samples\":\n # Because we the \"samples\" column to be the sum\n # of all samples in input data, it is quicker to use\n # python multiprocessing to split the summing work:\n df = run_multiprocess(\n files,\n args.output,\n select_read_ids=select_read_ids,\n column=args.get_column,\n mode=mode,\n )\n elif args.select_ids:\n df = run_select(\n files,\n select_read_ids=select_read_ids,\n column=args.get_column,\n )\n else:\n df = run_batched(\n files,\n column=args.get_column,\n )\n\n print(f\"Selected {len(df)} items\")\n args.output.mkdir(parents=True, exist_ok=True)\n df.to_csv(args.output / \"read_ids.csv\", index=False)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/tools/pyslow5_tests.py", "content": "#!/usr/bin/env python3\n\nimport argparse\nimport multiprocessing as mp\nfrom pathlib import Path\nfrom queue import Empty\n\nimport numpy\nimport pandas as pd\nimport pyslow5\n\n\ndef random_access(s5_file, read_list, col, result_q):\n file = pyslow5.Open(str(s5_file), \"r\")\n print(\"processing \", s5_file)\n read_ids = []\n extracted_columns = {\"read_id\": read_ids}\n extracted_columns[col] = []\n vals = extracted_columns[col]\n if col == \"samples\":\n for read in file.get_read_list_multi(read_list, threads=10, batchsize=5000):\n read_ids.append(read[\"read_id\"])\n vals.append(numpy.sum(read[\"signal\"]))\n elif col == \"sample_count\":\n for read in file.get_read_list_multi(read_list, threads=10, batchsize=5000):\n read_ids.append(read[\"read_id\"])\n vals.append(read[\"len_raw_signal\"])\n else:\n for read in file.get_read_list_multi(\n read_list, threads=10, batchsize=5000, pA=False, aux=col\n ):\n read_ids.append(read[\"read_id\"])\n vals.append(read[col])\n result_q.put(pd.DataFrame(extracted_columns))\n\n\ndef run(s5_file, benchmark, select_read_ids, col):\n if benchmark == \"get_all_read_ids\":\n read_ids = []\n extracted_columns = {\"read_id\": read_ids}\n file = pyslow5.Open(str(s5_file), \"r\")\n print(\"processing \", s5_file)\n read_ids, num_reads = file.get_read_ids()\n extracted_columns = {\"read_id\": read_ids}\n\n elif benchmark == \"sample_values\":\n mp.set_start_method(\"spawn\")\n result_queue = mp.Queue()\n runners = 10\n processes = []\n approx_chunk_size = max(1, len(select_read_ids) // runners)\n select_ids = []\n for i in range(0, len(select_read_ids), approx_chunk_size):\n for j in range(i, min(len(select_read_ids), i + approx_chunk_size)):\n select_ids.append(select_read_ids[j])\n p = mp.Process(\n target=random_access, args=(s5_file, select_ids, col, result_queue)\n )\n p.start()\n processes.append(p)\n select_ids = []\n\n print(\"Wait for processes...\")\n items = []\n while len(items) < len(processes):\n try:\n item = result_queue.get(timeout=0.5)\n items.append(item)\n except Empty:\n continue\n\n for p in processes:\n p.join()\n\n df = pd.concat(items)\n return df\n\n elif benchmark == \"all_values\":\n read_ids = []\n extracted_columns = {\"read_id\": read_ids}\n file = pyslow5.Open(str(s5_file), \"r\")\n print(\"processing \", s5_file)\n extracted_columns[col] = []\n vals = extracted_columns[col]\n if col == \"samples\":\n for read in file.seq_reads_multi(threads=10, batchsize=5000):\n read_ids.append(read[\"read_id\"])\n vals.append(numpy.sum(read[\"signal\"]))\n elif col == \"sample_count\":\n for read in file.seq_reads_multi(threads=10, batchsize=5000):\n read_ids.append(read[\"read_id\"])\n vals.append(read[\"len_raw_signal\"])\n else:\n for read in file.seq_reads_multi(\n threads=10, batchsize=5000, pA=False, aux=col\n ):\n read_ids.append(read[\"read_id\"])\n vals.append(read[col])\n\n return pd.DataFrame(extracted_columns)\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"input\", type=Path)\n parser.add_argument(\"output\", type=Path)\n parser.add_argument(\n \"benchmark\",\n type=str,\n choices=[\"get_all_read_ids\", \"sample_values\", \"all_values\"],\n help=\"which benchmark to run\",\n )\n parser.add_argument(\n \"--select-ids\",\n type=str,\n help=\"CSV file with a read_id column, listing ids to find in input files\",\n )\n parser.add_argument(\n \"--get-column\",\n default=None,\n type=str,\n help=\"Add columns that should be extracted\",\n )\n\n args = parser.parse_args()\n\n args.output.mkdir(parents=True, exist_ok=True)\n select_read_ids = None\n select_reads = []\n if args.select_ids:\n select_read_ids = pd.read_csv(args.select_ids)[\"read_id\"]\n for i in select_read_ids:\n select_reads.append(i)\n\n print(f\"Num of select_reads: {len(select_reads)}\")\n\n df = run(\n args.input,\n args.benchmark,\n select_read_ids=select_reads,\n col=args.get_column,\n )\n print(f\"Selected {len(df)} items\")\n df.to_csv(args.output / \"read_ids.csv\", index=False)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "benchmarks/tools/run_benchmarks_docker_entry.sh", "content": "#!/bin/bash\n\n# Use taskset to limit benchmarks to specific cores, ensuring a fair test of limited resources:\ntaskset -c 0-10 /benchmarks/run_benchmarks.py /input /outputs\n" }, { "path": "benchmarks/tools/select-random-ids.py", "content": "#!/usr/bin/env python3\n\nimport argparse\nfrom pathlib import Path\n\nimport pandas as pd\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"input_csv\", type=Path)\n parser.add_argument(\"output_csv\", type=Path)\n parser.add_argument(\"--select-ratio\", type=float)\n\n args = parser.parse_args()\n\n df = pd.read_csv(args.input_csv)\n\n selected_rows_df = df.sample(frac=args.select_ratio)\n\n args.output_csv.parent.mkdir(parents=True, exist_ok=True)\n selected_rows_df.to_csv(args.output_csv)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "c++/CMakeLists.txt", "content": "\n\nif (ENABLE_CONAN)\n find_package(Arrow REQUIRED CONFIG)\n find_package(Flatbuffers REQUIRED CONFIG)\n find_package(zstd REQUIRED CONFIG)\n find_package(ZLIB REQUIRED CONFIG)\n\n if (${CMAKE_SYSTEM_NAME} STREQUAL \"Linux\")\n find_package(jemalloc REQUIRED CONFIG)\n endif()\nelse()\n find_package(Arrow REQUIRED)\n find_package(Flatbuffers REQUIRED)\n find_package(zstd REQUIRED)\n find_package(ZLIB REQUIRED)\n\n # Our non-conan ubuntu CI build has a different name for this target\n if (NOT CONAN2)\n add_library(arrow::arrow INTERFACE IMPORTED)\n target_link_libraries(arrow::arrow INTERFACE Arrow::arrow_shared)\n endif()\nendif()\n\nfind_package(Threads REQUIRED)\n\nfind_program(\n FLATBUFFERS_FLATC_EXECUTABLE\n flatc\n)\ninclude(BuildFlatBuffers)\n\nconfigure_file(\n pod5_format/version.h.in\n pod5_format/version.h\n)\n\nset(pod5_library_type STATIC)\nif (BUILD_SHARED_LIB)\n set(pod5_library_type SHARED)\nendif()\n\nadd_library(pod5_format ${pod5_library_type}\n pod5_format/file_recovery.h\n pod5_format/file_writer.cpp\n pod5_format/file_writer.h\n pod5_format/file_reader.cpp\n pod5_format/file_reader.h\n pod5_format/file_updater.cpp\n pod5_format/file_updater.h\n\n pod5_format/async_signal_loader.cpp\n pod5_format/async_signal_loader.h\n\n pod5_format/schema_metadata.cpp\n pod5_format/table_reader.h\n pod5_format/schema_field_builder.h\n\n pod5_format/read_table_reader.cpp\n pod5_format/read_table_reader.h\n pod5_format/read_table_schema.cpp\n pod5_format/read_table_schema.h\n pod5_format/read_table_writer.cpp\n pod5_format/read_table_writer.h\n pod5_format/read_table_writer_utils.cpp\n pod5_format/read_table_writer_utils.h\n pod5_format/read_table_utils.cpp\n pod5_format/read_table_utils.h\n\n pod5_format/run_info_table_reader.cpp\n pod5_format/run_info_table_reader.h\n pod5_format/run_info_table_schema.cpp\n pod5_format/run_info_table_schema.h\n pod5_format/run_info_table_writer.cpp\n pod5_format/run_info_table_writer.h\n\n pod5_format/signal_compression.cpp\n pod5_format/signal_compression.h\n pod5_format/signal_table_reader.cpp\n pod5_format/signal_table_reader.h\n pod5_format/signal_table_schema.cpp\n pod5_format/signal_table_schema.h\n pod5_format/signal_table_writer.cpp\n pod5_format/signal_table_writer.h\n pod5_format/signal_table_utils.h\n pod5_format/signal_builder.h\n\n pod5_format/c_api.cpp\n pod5_format/c_api.h\n\n pod5_format/expandable_buffer.h\n pod5_format/io_manager.cpp\n pod5_format/io_manager.h\n pod5_format/memory_pool.cpp\n pod5_format/memory_pool.h\n pod5_format/result.h\n pod5_format/schema_utils.cpp\n pod5_format/schema_utils.h\n pod5_format/table_reader.cpp\n pod5_format/table_reader.h\n pod5_format/thread_pool.cpp\n pod5_format/thread_pool.h\n pod5_format/tuple_utils.h\n pod5_format/types.cpp\n pod5_format/types.h\n pod5_format/uuid.h\n\n pod5_format/migration/migration.cpp\n pod5_format/migration/migration.h\n pod5_format/migration/migration_utils.h\n pod5_format/migration/v0_to_v1.cpp\n pod5_format/migration/v1_to_v2.cpp\n pod5_format/migration/v2_to_v3.cpp\n pod5_format/migration/v3_to_v4.cpp\n\n pod5_format/internal/async_output_stream.h\n pod5_format/internal/combined_file_utils.h\n pod5_format/internal/linux_output_stream.h\n\n pod5_format/svb16/common.hpp\n pod5_format/svb16/decode.hpp\n pod5_format/svb16/decode_scalar.hpp\n pod5_format/svb16/decode_x64.hpp\n pod5_format/svb16/encode.hpp\n pod5_format/svb16/encode_scalar.hpp\n pod5_format/svb16/encode_x64.hpp\n pod5_format/svb16/intrinsics.hpp\n pod5_format/svb16/shuffle_tables.hpp\n pod5_format/svb16/simd_detect_x64.hpp\n)\n\nset(public_headers)\nlist(APPEND public_headers\n pod5_format/file_writer.h\n pod5_format/file_reader.h\n\n pod5_format/schema_metadata.h\n\n pod5_format/read_table_reader.h\n pod5_format/read_table_schema.h\n pod5_format/read_table_writer.h\n pod5_format/read_table_writer_utils.h\n pod5_format/read_table_utils.h\n\n pod5_format/run_info_table_writer.h\n pod5_format/run_info_table_reader.h\n pod5_format/run_info_table_schema.h\n\n pod5_format/signal_compression.h\n pod5_format/signal_table_reader.h\n pod5_format/signal_table_schema.h\n pod5_format/signal_table_writer.h\n pod5_format/signal_table_utils.h\n pod5_format/signal_builder.h\n pod5_format/uuid.h\n\n pod5_format/c_api.h\n\n pod5_format/expandable_buffer.h\n pod5_format/file_output_stream.h\n pod5_format/io_manager.h\n pod5_format/memory_pool.h\n pod5_format/result.h\n pod5_format/dictionary_writer.h\n pod5_format/schema_field_builder.h\n pod5_format/schema_utils.h\n pod5_format/table_reader.h\n pod5_format/thread_pool.h\n pod5_format/tuple_utils.h\n pod5_format/types.h\n\n\n ${CMAKE_CURRENT_BINARY_DIR}/pod5_format/pod5_format_export.h\n)\n\nset(svb16_headers\n pod5_format/svb16/svb16.h\n pod5_format/svb16/common.hpp\n pod5_format/svb16/decode.hpp\n pod5_format/svb16/decode_scalar.hpp\n pod5_format/svb16/decode_x64.hpp\n pod5_format/svb16/encode.hpp\n pod5_format/svb16/encode_scalar.hpp\n pod5_format/svb16/encode_x64.hpp\n pod5_format/svb16/intrinsics.hpp\n pod5_format/svb16/shuffle_tables.hpp\n pod5_format/svb16/simd_detect_x64.hpp\n)\n\nset_target_properties(pod5_format\n PROPERTIES\n POSITION_INDEPENDENT_CODE 1\n CXX_STANDARD 20\n PUBLIC_HEADER \"${public_headers}\"\n)\n\n# Link these libraries publicly when doing a static lib build\nset(maybe_public_libs\n arrow::arrow\n flatbuffers::flatbuffers\n)\n\nif (BUILD_SHARED_LIB)\n target_link_libraries(pod5_format PRIVATE ${maybe_public_libs})\nelse()\n target_link_libraries(pod5_format PUBLIC ${maybe_public_libs})\nendif()\n\ntarget_link_libraries(pod5_format\n PRIVATE\n pod5_flatbuffers\n zstd::zstd\n ZLIB::ZLIB\n Threads::Threads\n)\n\nif(APPLE)\n find_library(CORE_FOUNDATION CoreFoundation)\n target_link_libraries(pod5_format PRIVATE ${CORE_FOUNDATION})\nendif()\n\ntarget_include_directories(pod5_format\n PUBLIC\n ${CMAKE_CURRENT_SOURCE_DIR}\n ${CMAKE_CURRENT_BINARY_DIR}\n)\n\nflatbuffers_generate_headers(\n TARGET pod5_flatbuffers\n SCHEMAS\n pod5_format/flatbuffers/footer.fbs\n INCLUDE_PREFIX \"\"\n FLAGS --cpp\n)\n\nif (NOT MSVC)\n set(pod5_warning_options -Werror -Wall -Wno-comment -Wno-error=deprecated-declarations -Wno-deprecated-declarations)\n target_compile_options(pod5_format PRIVATE ${pod5_warning_options})\nendif()\ngenerate_export_header(pod5_format EXPORT_FILE_NAME pod5_format/pod5_format_export.h)\n\ninstall(\n TARGETS pod5_format\n PUBLIC_HEADER DESTINATION \"include/pod5_format\"\n)\n\ninstall(\n FILES ${svb16_headers}\n DESTINATION \"include/pod5_format/svb16\"\n)\n\nif (POD5_BUILD_EXAMPLES)\n add_subdirectory(examples)\nendif()\nif (NOT POD5_DISABLE_TESTS)\n add_subdirectory(test)\nendif()\n\nif (BUILD_PYTHON_WHEEL)\n add_subdirectory(pod5_format_pybind)\nendif()\n" }, { "path": "c++/examples/CMakeLists.txt", "content": "add_executable(find_all_read_ids\n find_all_read_ids.cpp\n)\n\ntarget_link_libraries(find_all_read_ids\n pod5_format\n)\n# Needs C++17 to use pod5_format/uuid.h\nset_target_properties(find_all_read_ids PROPERTIES CXX_STANDARD 17)\n\nadd_executable(find_specific_read_ids\n find_specific_read_ids.cpp\n)\n\ntarget_link_libraries(find_specific_read_ids\n pod5_format\n)\n# Needs C++17 to use pod5_format/uuid.h\nset_target_properties(find_specific_read_ids PROPERTIES CXX_STANDARD 17)\n\nadd_executable(find_all_read_data\n find_all_read_data.cpp\n)\n\ntarget_link_libraries(find_all_read_data\n pod5_format\n)\n# Needs C++17 to use pod5_format/uuid.h\nset_target_properties(find_all_read_data PROPERTIES CXX_STANDARD 17)\n\nadd_executable(find_specific_read_ids_with_signal\n find_specific_read_ids_with_signal.cpp\n)\n\ntarget_link_libraries(find_specific_read_ids_with_signal\n pod5_format\n)\n# Needs C++17 to use pod5_format/uuid.h\nset_target_properties(find_specific_read_ids_with_signal PROPERTIES CXX_STANDARD 17)\n" }, { "path": "c++/examples/README.md", "content": "C++ Examples\n============\n\nThese examples use the POD5 C API to read file data, they are written using C++.\n\nfind_all_read_ids\n-----------------\n\nFind all the read ids in a given pod5 file, and save their read id to a text file.\n\nfind_specific_read_ids\n----------------------\n\nFind specific read ids in a given pod5 file, and save their read number to a text file.\n" }, { "path": "c++/examples/find_all_read_data.cpp", "content": "#include \"pod5_format/c_api.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n#include \n\nint main(int argc, char ** argv)\n{\n if (argc != 2) {\n std::cerr << \"Expected one argument - an pod5 file to search\\n\";\n return EXIT_FAILURE;\n }\n\n // Initialise the POD5 library:\n pod5_init();\n\n // Open the file ready for walking:\n Pod5FileReader_t * file = pod5_open_file(argv[1]);\n if (!file) {\n std::cerr << \"Failed to open file \" << argv[1] << \": \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_count = 0;\n if (pod5_get_read_batch_count(&batch_count, file) != POD5_OK) {\n std::cerr << \"Failed to query batch count: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t read_count = 0;\n\n for (std::size_t batch_index = 0; batch_index < batch_count; ++batch_index) {\n std::cout << \"batch_index: \" << batch_index + 1 << \"/\" << batch_count << \"\\n\";\n\n Pod5ReadRecordBatch_t * batch = nullptr;\n if (pod5_get_read_batch(&batch, file, batch_index) != POD5_OK) {\n std::cerr << \"Failed to get batch: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_row_count = 0;\n if (pod5_get_read_batch_row_count(&batch_row_count, batch) != POD5_OK) {\n std::cerr << \"Failed to get batch row count\\n\";\n return EXIT_FAILURE;\n }\n\n for (std::size_t row = 0; row < batch_row_count; ++row) {\n uint16_t read_table_version = 0;\n ReadBatchRowInfo_t read_data;\n if (pod5_get_read_batch_row_info_data(\n batch, row, READ_BATCH_ROW_INFO_VERSION, &read_data, &read_table_version)\n != POD5_OK)\n {\n std::cerr << \"Failed to get read \" << row << \": \" << pod5_get_error_string()\n << \"\\n\";\n return EXIT_FAILURE;\n }\n\n read_count += 1;\n\n std::size_t sample_count = 0;\n pod5_get_read_complete_sample_count(file, batch, row, &sample_count);\n\n std::vector samples;\n samples.resize(sample_count);\n pod5_get_read_complete_signal(file, batch, row, samples.size(), samples.data());\n\n // Run info\n RunInfoDictData_t * run_info = nullptr;\n if (pod5_get_run_info(batch, read_data.run_info, &run_info) != POD5_OK) {\n std::cerr << \"Failed to get run info \" + std::to_string(read_data.run_info) + \" : \"\n + pod5_get_error_string()\n << \"\\n\";\n return EXIT_FAILURE;\n }\n\n pod5_free_run_info(run_info);\n }\n\n if (pod5_free_read_batch(batch) != POD5_OK) {\n std::cerr << \"Failed to release batch\\n\";\n return EXIT_FAILURE;\n }\n }\n\n std::cout << \"Extracted \" << read_count << \" reads \"\n << \"\\n\";\n\n // Close the reader\n if (pod5_close_and_free_reader(file) != POD5_OK) {\n std::cerr << \"Failed to close reader: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n // Cleanup the library\n pod5_terminate();\n}\n" }, { "path": "c++/examples/find_all_read_ids.cpp", "content": "#include \"pod5_format/c_api.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n#include \n\nint main(int argc, char ** argv)\n{\n if (argc != 2) {\n std::cerr << \"Expected one argument - an pod5 file to search\\n\";\n return EXIT_FAILURE;\n }\n\n // Initialise the POD5 library:\n pod5_init();\n\n // Open the file ready for walking:\n Pod5FileReader_t * file = pod5_open_file(argv[1]);\n if (!file) {\n std::cerr << \"Failed to open file \" << argv[1] << \": \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_count = 0;\n if (pod5_get_read_batch_count(&batch_count, file) != POD5_OK) {\n std::cerr << \"Failed to query batch count: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::string output_path(\"read_ids.txt\");\n std::cout << \"Writing read ids to \" << output_path << \"\\n\";\n std::ofstream output_stream(output_path);\n\n std::size_t read_count = 0;\n\n for (std::size_t batch_index = 0; batch_index < batch_count; ++batch_index) {\n Pod5ReadRecordBatch_t * batch = nullptr;\n if (pod5_get_read_batch(&batch, file, batch_index) != POD5_OK) {\n std::cerr << \"Failed to get batch: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_row_count = 0;\n if (pod5_get_read_batch_row_count(&batch_row_count, batch) != POD5_OK) {\n std::cerr << \"Failed to get batch row count\\n\";\n return EXIT_FAILURE;\n }\n\n for (std::size_t row = 0; row < batch_row_count; ++row) {\n uint16_t read_table_version = 0;\n ReadBatchRowInfo_t read_data;\n if (pod5_get_read_batch_row_info_data(\n batch, row, READ_BATCH_ROW_INFO_VERSION, &read_data, &read_table_version)\n != POD5_OK)\n {\n std::cerr << \"Failed to get read \" << row << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::array formatted_read_id;\n pod5_format_read_id(read_data.read_id, formatted_read_id.data());\n output_stream << formatted_read_id.data() << \"\\n\";\n read_count += 1;\n\n std::size_t sample_count = 0;\n pod5_get_read_complete_sample_count(file, batch, row, &sample_count);\n\n std::vector samples;\n samples.resize(sample_count);\n pod5_get_read_complete_signal(file, batch, row, samples.size(), samples.data());\n }\n\n if (pod5_free_read_batch(batch) != POD5_OK) {\n std::cerr << \"Failed to release batch\\n\";\n return EXIT_FAILURE;\n }\n }\n\n std::cout << \"Extracted \" << read_count << \" read ids into \" << output_path << \"\\n\";\n\n // Close the reader\n if (pod5_close_and_free_reader(file) != POD5_OK) {\n std::cerr << \"Failed to close reader: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n // Cleanup the library\n pod5_terminate();\n}\n" }, { "path": "c++/examples/find_specific_read_ids.cpp", "content": "#include \"pod5_format/c_api.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n\nint main(int argc, char ** argv)\n{\n if (argc != 3) {\n std::cerr << \"Expected two arguments:\\n\"\n << \" - an pod5 file to search\\n\"\n << \" - a file containing newline separated of read ids\\n\";\n return EXIT_FAILURE;\n }\n\n // Initialise the POD5 library:\n pod5_init();\n\n // Open the file ready for walking:\n Pod5FileReader_t * file = pod5_open_file(argv[1]);\n if (!file) {\n std::cerr << \"Failed to open file \" << argv[1] << \": \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_count = 0;\n if (pod5_get_read_batch_count(&batch_count, file) != POD5_OK) {\n std::cerr << \"Failed to query batch count: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::vector search_uuids;\n std::string input_path(argv[2]);\n try {\n std::cout << \"Reading input read ids from \" << input_path << \"\\n\";\n std::string line;\n std::ifstream input_stream(input_path);\n while (std::getline(input_stream, line)) {\n auto const uuid = pod5::Uuid::from_string(line);\n if (!uuid) {\n std::cerr << '\"' << line << \"\\\" is not a valid UUID, ignoring it\\n\";\n } else {\n search_uuids.push_back(*uuid);\n }\n }\n std::cout << \" Read \" << search_uuids.size() << \" ids from the text file\\n\";\n } catch (std::exception const & e) {\n std::cerr << \"Failed to parse UUID values from \" << input_path << \": \" << e.what() << \"\\n\";\n }\n\n std::string output_path(\"read_ids.txt\");\n std::cout << \"Writing selected read numbers to \" << output_path << \"\\n\";\n std::ofstream output_stream(output_path);\n\n // Plan the most efficient route through the file for the required read ids:\n std::vector traversal_batch_counts(batch_count);\n std::vector traversal_row_indices(search_uuids.size());\n std::size_t find_success_count = 0;\n if (pod5_plan_traversal(\n file,\n (uint8_t *)search_uuids.data(),\n search_uuids.size(),\n traversal_batch_counts.data(),\n traversal_row_indices.data(),\n &find_success_count)\n != POD5_OK)\n {\n std::cerr << \"Failed to plan traversal of file: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n if (find_success_count != search_uuids.size()) {\n std::cerr << \"Failed to find \" << (search_uuids.size() - find_success_count) << \" reads\\n\";\n }\n\n std::size_t read_count = 0;\n std::size_t row_offset = 0;\n\n // Walk the suggested traversal route, storing read data.\n for (std::size_t batch_index = 0; batch_index < batch_count; ++batch_index) {\n Pod5ReadRecordBatch_t * batch = nullptr;\n if (pod5_get_read_batch(&batch, file, batch_index) != POD5_OK) {\n std::cerr << \"Failed to get batch: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::cout << \"Processing batch \" << (batch_index + 1) << \" of \" << batch_count << \"\\n\";\n for (std::size_t row_index = 0; row_index < traversal_batch_counts[batch_index];\n ++row_index)\n {\n std::uint32_t batch_row = traversal_row_indices[row_index + row_offset];\n\n uint16_t read_table_version = 0;\n ReadBatchRowInfo_t read_data;\n if (pod5_get_read_batch_row_info_data(\n batch, batch_row, READ_BATCH_ROW_INFO_VERSION, &read_data, &read_table_version)\n != POD5_OK)\n {\n std::cerr << \"Failed to get read \" << batch_row << \"\\n\";\n return EXIT_FAILURE;\n }\n\n output_stream << read_data.read_number << \"\\n\";\n read_count += 1;\n }\n row_offset += traversal_batch_counts[batch_index];\n\n if (pod5_free_read_batch(batch) != POD5_OK) {\n std::cerr << \"Failed to release batch\\n\";\n return EXIT_FAILURE;\n }\n }\n\n std::cout << \"Extracted \" << read_count << \" read numbers into \" << output_path << \"\\n\";\n\n // Close the reader\n if (pod5_close_and_free_reader(file) != POD5_OK) {\n std::cerr << \"Failed to close reader: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n // Cleanup the library\n pod5_terminate();\n}\n" }, { "path": "c++/examples/find_specific_read_ids_with_signal.cpp", "content": "#include \"pod5_format/c_api.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n\nint main(int argc, char ** argv)\n{\n if (argc != 3) {\n std::cerr << \"Expected two arguments:\\n\"\n << \" - an pod5 file to search\\n\"\n << \" - a file containing newline separated of read ids\\n\";\n return EXIT_FAILURE;\n }\n\n // Initialise the POD5 library:\n pod5_init();\n\n // Open the file ready for walking:\n Pod5FileReader_t * file = pod5_open_file(argv[1]);\n if (!file) {\n std::cerr << \"Failed to open file \" << argv[1] << \": \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t batch_count = 0;\n if (pod5_get_read_batch_count(&batch_count, file) != POD5_OK) {\n std::cerr << \"Failed to query batch count: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::vector search_uuids;\n std::string input_path(argv[2]);\n try {\n std::cout << \"Reading input read ids from \" << input_path << \"\\n\";\n std::string line;\n std::ifstream input_stream(input_path);\n while (std::getline(input_stream, line)) {\n auto const uuid = pod5::Uuid::from_string(line);\n if (!uuid) {\n std::cerr << '\"' << line << \"\\\" is not a valid UUID, ignoring it\\n\";\n } else {\n search_uuids.push_back(*uuid);\n }\n }\n std::cout << \" Read \" << search_uuids.size() << \" ids from the text file\\n\";\n } catch (std::exception const & e) {\n std::cerr << \"Failed to parse UUID values from \" << input_path << \": \" << e.what() << \"\\n\";\n }\n\n std::string output_path(\"read_ids.txt\");\n std::cout << \"Writing selected read numbers to \" << output_path << \"\\n\";\n std::ofstream output_stream(output_path);\n\n // Plan the most efficient route through the file for the required read ids:\n std::vector traversal_batch_counts(batch_count);\n std::vector traversal_row_indices(search_uuids.size());\n std::size_t find_success_count = 0;\n if (pod5_plan_traversal(\n file,\n (uint8_t *)search_uuids.data(),\n search_uuids.size(),\n traversal_batch_counts.data(),\n traversal_row_indices.data(),\n &find_success_count)\n != POD5_OK)\n {\n std::cerr << \"Failed to plan traversal of file: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n if (find_success_count != search_uuids.size()) {\n std::cerr << \"Failed to find \" << (search_uuids.size() - find_success_count) << \" reads\\n\";\n }\n\n std::size_t read_count = 0;\n std::size_t samples_read = 0;\n\n std::size_t row_offset = 0;\n\n // Walk the suggested traversal route, storing read data.\n for (std::size_t batch_index = 0; batch_index < batch_count; ++batch_index) {\n Pod5ReadRecordBatch_t * batch = nullptr;\n if (pod5_get_read_batch(&batch, file, batch_index) != POD5_OK) {\n std::cerr << \"Failed to get batch: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::cout << \"Processing batch \" << (batch_index + 1) << \" of \" << batch_count << \"\\n\";\n for (std::size_t row_index = 0; row_index < traversal_batch_counts[batch_index];\n ++row_index)\n {\n std::uint32_t batch_row = traversal_row_indices[row_index + row_offset];\n\n uint16_t read_table_version = 0;\n ReadBatchRowInfo_t read_data;\n if (pod5_get_read_batch_row_info_data(\n batch, batch_row, READ_BATCH_ROW_INFO_VERSION, &read_data, &read_table_version)\n != POD5_OK)\n {\n std::cerr << \"Failed to get read \" << batch_row << \"\\n\";\n return EXIT_FAILURE;\n }\n\n std::size_t sample_count = 0;\n pod5_get_read_complete_sample_count(file, batch, batch_row, &sample_count);\n\n std::vector samples;\n samples.resize(sample_count);\n pod5_get_read_complete_signal(file, batch, batch_row, samples.size(), samples.data());\n\n std::int64_t samples_sum = 0;\n for (std::size_t i = 0; i < samples.size(); ++i) {\n samples_sum += samples[i];\n }\n\n output_stream << read_data.calibration_offset << \" \" << read_data.calibration_scale\n << \" \" << samples_sum << \"\\n\";\n read_count += 1;\n samples_read += samples.size();\n }\n row_offset += traversal_batch_counts[batch_index];\n\n if (pod5_free_read_batch(batch) != POD5_OK) {\n std::cerr << \"Failed to release batch\\n\";\n return EXIT_FAILURE;\n }\n }\n\n std::cout << \"Extracted \" << read_count << \" reads and \" << samples_read << \" samples into \"\n << output_path << \"\\n\";\n\n // Close the reader\n if (pod5_close_and_free_reader(file) != POD5_OK) {\n std::cerr << \"Failed to close reader: \" << pod5_get_error_string() << \"\\n\";\n return EXIT_FAILURE;\n }\n\n // Cleanup the library\n pod5_terminate();\n}\n" }, { "path": "c++/pod5_format/async_signal_loader.cpp", "content": "#include \"pod5_format/async_signal_loader.h\"\n\nnamespace pod5 {\n\nstd::size_t const AsyncSignalLoader::MINIMUM_JOB_SIZE = 50;\n\nAsyncSignalLoader::AsyncSignalLoader(\n std::shared_ptr const & reader,\n SamplesMode samples_mode,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows,\n std::size_t worker_count,\n std::size_t max_pending_batches)\n: m_reader(reader)\n, m_samples_mode(samples_mode)\n, m_max_pending_batches(max_pending_batches)\n, m_reads_batch_count(m_reader->num_read_record_batches())\n, m_batch_counts(batch_counts)\n, m_total_batch_count_so_far(0)\n, m_batch_rows(batch_rows)\n, m_worker_job_size(\n std::max(\n MINIMUM_JOB_SIZE,\n m_batch_rows.size() / (m_reads_batch_count * worker_count * 2)))\n, m_current_batch(0)\n, m_finished(false)\n, m_has_error(false)\n, m_batches_size(0)\n{\n // Setup first batch:\n {\n std::unique_lock l(m_worker_sync);\n auto setup_result = setup_next_in_progress_batch(l);\n if (!setup_result.ok()) {\n set_error(setup_result);\n }\n }\n\n // Kick off workers on jobs:\n for (std::size_t i = 0; i < worker_count; ++i) {\n m_workers.emplace_back([&] { run_worker(); });\n }\n}\n\nAsyncSignalLoader::~AsyncSignalLoader()\n{\n m_finished = true;\n // Wait for all workers to complete:\n for (std::size_t i = 0; i < m_workers.size(); ++i) {\n m_workers[i].join();\n }\n}\n\nResult> AsyncSignalLoader::release_next_batch(\n std::optional timeout)\n{\n std::shared_ptr batch;\n\n // Return any error, if one has occurred:\n if (m_has_error) {\n return error();\n }\n\n // First wait until there is a batch available:\n do {\n std::unique_lock l(m_batches_sync);\n // Wait until there is a batch available:\n m_batch_done.wait_until(\n l, timeout.value_or(std::chrono::steady_clock::now() + std::chrono::seconds(5)), [&] {\n return m_batches.size() || m_finished || m_has_error;\n });\n\n // Grab a batch if one exists (note error or user destroying us might have happened instead):\n if (!m_batches.empty()) {\n batch = std::move(m_batches.front());\n assert(batch);\n m_batches.pop_front();\n m_batches_size -= 1;\n break;\n }\n\n if (timeout && std::chrono::steady_clock::now() > *timeout) {\n return nullptr;\n }\n } while (!m_finished && !m_has_error);\n\n // Return any error, if one has occurred during our wait:\n if (m_has_error) {\n return error();\n }\n\n // If we got a batch, wait for all work to be finished, then return it:\n if (batch) {\n // Wait if we are ahead of the loader:\n while (!batch->is_complete()) {\n std::this_thread::sleep_for(std::chrono::milliseconds(1));\n }\n\n return batch->release_data();\n }\n\n // No more data - return null.\n return nullptr;\n}\n\nvoid AsyncSignalLoader::set_error(pod5::Status status)\n{\n assert(!status.ok());\n {\n std::lock_guard l{m_error_mutex};\n m_error = std::move(status);\n }\n m_has_error = true;\n}\n\npod5::Status AsyncSignalLoader::error() const\n{\n std::lock_guard l{m_error_mutex};\n return m_error;\n}\n\nvoid AsyncSignalLoader::run_worker()\n{\n // Continue to work while there is work to do, and no error has occurred\n while (!m_finished && !m_has_error) {\n std::shared_ptr batch;\n std::uint32_t row_start = 0;\n\n // Try to secure some new work:\n {\n std::unique_lock l(m_worker_sync);\n // If we have run out of batches to process, release anything in progress and return:\n if (m_current_batch >= m_reads_batch_count) {\n release_in_progress_batch();\n break;\n }\n\n // If we have more batches than asked for complete that have\n // not been queried, wait for it to get taken:\n if (m_batches_size > m_max_pending_batches) {\n l.unlock();\n std::this_thread::sleep_for(std::chrono::milliseconds(10));\n continue;\n }\n\n // Now, if we have no work left in the current batch, release that:\n if (!m_in_progress_batch->has_work_left()) {\n if (!m_batch_counts.empty()) {\n m_total_batch_count_so_far += m_batch_counts[m_current_batch];\n }\n\n // Release the current batch:\n release_in_progress_batch();\n\n // Then try to setup the next batch, if one exists:\n m_current_batch += 1;\n if (m_current_batch >= m_reads_batch_count) {\n // No more work to do.\n m_finished = true;\n break;\n }\n\n auto setup_result = setup_next_in_progress_batch(l);\n if (!setup_result.ok()) {\n set_error(setup_result);\n return;\n }\n }\n\n // Finally, tell the work package we have secured we are starting to do some work:\n batch = m_in_progress_batch;\n row_start = m_in_progress_batch->start_rows(l, m_worker_job_size);\n }\n\n // Now execute the work, for all the rows we said we would:\n std::uint32_t const row_end =\n std::min(row_start + m_worker_job_size, batch->job_row_count());\n\n do_work(batch, row_start, row_end);\n\n // And report the work completed for anyone waiting:\n batch->complete_rows(m_worker_job_size);\n }\n}\n\nvoid AsyncSignalLoader::do_work(\n std::shared_ptr const & batch,\n std::uint32_t row_start,\n std::uint32_t row_end)\n{\n // First secure the sample counts column for the batch we are processing:\n auto signal_column = batch->read_batch().signal_column();\n\n // And record where we are starting in the batch rows array, if it exists:\n for (std::uint32_t i = row_start; i < row_end; ++i) {\n // Find the actual batch row to query - we may be working on a subset of batch data:\n auto const actual_batch_row = batch->get_batch_row_to_query(i);\n // Get the signal row data for the read:\n auto const signal_rows = std::static_pointer_cast(\n signal_column->value_slice(actual_batch_row));\n auto const signal_rows_span =\n gsl::make_span(signal_rows->raw_values(), signal_rows->length());\n\n // Find the sample count for these rows:\n auto sample_count_result = m_reader->extract_sample_count(signal_rows_span);\n if (!sample_count_result.ok()) {\n set_error(sample_count_result.status());\n return;\n }\n std::uint64_t sample_count = *sample_count_result;\n\n // And query the samples if that has been requested:\n std::vector samples;\n if (m_samples_mode == SamplesMode::Samples) {\n samples.resize(sample_count);\n auto samples_result =\n m_reader->extract_samples(signal_rows_span, gsl::make_span(samples));\n if (!samples_result.ok()) {\n set_error(std::move(samples_result));\n return;\n }\n sample_count = samples.size();\n }\n\n // Store the queried data into the batch:\n batch->set_samples(i, sample_count, std::move(samples));\n }\n}\n\nStatus AsyncSignalLoader::setup_next_in_progress_batch(std::unique_lock & lock)\n{\n assert(!m_in_progress_batch);\n ARROW_ASSIGN_OR_RAISE(auto read_batch, m_reader->read_read_record_batch(m_current_batch));\n std::size_t row_count = read_batch.num_rows();\n\n gsl::span next_specific_batch_rows;\n if (!m_batch_counts.empty()) {\n row_count = m_batch_counts[m_current_batch];\n if (!m_batch_rows.empty()) {\n next_specific_batch_rows = m_batch_rows.subspan(m_total_batch_count_so_far, row_count);\n }\n }\n\n m_in_progress_batch = std::make_shared(\n m_current_batch, row_count, next_specific_batch_rows, std::move(read_batch));\n return Status::OK();\n}\n\nvoid AsyncSignalLoader::release_in_progress_batch()\n{\n if (m_in_progress_batch) {\n assert(!m_in_progress_batch->has_work_left());\n std::lock_guard l(m_batches_sync);\n m_batches.emplace_back(std::move(m_in_progress_batch));\n m_batches_size += 1;\n m_batch_done.notify_all();\n }\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/async_signal_loader.h", "content": "#pragma once\n\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/signal_table_reader.h\"\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nclass POD5_FORMAT_EXPORT CachedBatchSignalData {\npublic:\n CachedBatchSignalData(std::uint32_t batch_index, std::size_t entry_count)\n : m_batch_index(batch_index)\n , m_sample_counts(entry_count)\n , m_samples(entry_count)\n {\n }\n\n std::uint32_t batch_index() const { return m_batch_index; }\n\n /// Find a list of sample counts for all requested batch rows.\n std::vector const & sample_count() const { return m_sample_counts; }\n\n /// Find a list of signal samples counts for all requested batch rows.\n std::vector> const & samples() const { return m_samples; }\n\n void\n set_samples(std::size_t row, std::uint64_t sample_count, std::vector && samples)\n {\n m_sample_counts[row] = sample_count;\n m_samples[row] = std::move(samples);\n }\n\nprivate:\n std::uint32_t m_batch_index;\n std::vector m_sample_counts;\n std::vector> m_samples;\n};\n\nclass POD5_FORMAT_EXPORT SignalCacheWorkPackage {\npublic:\n SignalCacheWorkPackage(\n std::uint32_t batch_index,\n std::size_t job_row_count,\n gsl::span const & specific_job_rows,\n pod5::ReadTableRecordBatch && read_batch)\n : m_job_row_count(job_row_count)\n , m_specific_job_rows(specific_job_rows)\n , m_next_row_to_start(0)\n , m_completed_rows(0)\n , m_cached_data(std::make_unique(batch_index, m_job_row_count))\n , m_read_batch(std::move(read_batch))\n {\n }\n\n std::uint32_t job_row_count() const { return m_job_row_count; }\n\n void\n set_samples(std::size_t row, std::uint64_t sample_count, std::vector && samples)\n {\n m_cached_data->set_samples(row, sample_count, std::move(samples));\n }\n\n std::unique_ptr release_data() { return std::move(m_cached_data); }\n\n pod5::ReadTableRecordBatch const & read_batch() const { return m_read_batch; }\n\n // Find the actual batch row to query, for a given job row index.\n std::uint32_t get_batch_row_to_query(std::uint32_t job_row_index) const\n {\n // We allow the caller to specify a subset of batch rows to iterate:\n if (!m_specific_job_rows.empty()) {\n return m_specific_job_rows[job_row_index];\n }\n\n return job_row_index;\n }\n\n std::uint32_t start_rows(std::unique_lock & l, std::size_t row_count)\n {\n auto row = m_next_row_to_start;\n m_next_row_to_start += row_count;\n return row;\n }\n\n void complete_rows(std::uint32_t row_count) { m_completed_rows += row_count; }\n\n bool has_work_left() const { return m_next_row_to_start < m_job_row_count; }\n\n bool is_complete() const { return m_completed_rows.load() >= m_job_row_count; }\n\nprivate:\n std::size_t m_job_row_count;\n gsl::span m_specific_job_rows;\n\n std::uint32_t m_next_row_to_start;\n std::atomic m_completed_rows;\n\n std::unique_ptr m_cached_data;\n pod5::ReadTableRecordBatch m_read_batch;\n};\n\nclass POD5_FORMAT_EXPORT AsyncSignalLoader {\npublic:\n // Minimum number of tasks one thread will do in a batch.\n static std::size_t const MINIMUM_JOB_SIZE;\n enum class SamplesMode {\n NoSamples,\n Samples,\n };\n\n AsyncSignalLoader(\n std::shared_ptr const & reader,\n SamplesMode samples_mode,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows,\n std::size_t worker_count = std::thread::hardware_concurrency(),\n std::size_t max_pending_batches = 10);\n\n ~AsyncSignalLoader();\n\n /// Find if all work is complete in the loader.\n bool is_finished() const { return m_finished; }\n\n /// Get the next batch of loaded signal, always returns the consecutive next signal batch\n /// \\note Returns nullptr when timeoout occurs, or if all data is exhausted.\n Result> release_next_batch(\n std::optional timeout = std::nullopt);\n\nprivate:\n /// Set an error code that will stop all async loading and return an error to the caller.\n void set_error(pod5::Status status);\n pod5::Status error() const;\n\n void run_worker();\n void do_work(\n std::shared_ptr const & batch,\n std::uint32_t row_start,\n std::uint32_t row_end);\n\n /// Setup a new batch for in progress work to contain.\n /// \\param lock A lock held on m_worker_sync.\n /// \\note There must not be a batch already in progress.\n /// \\note m_current_batch is used as the index of the next batch to begin.\n Status setup_next_in_progress_batch(std::unique_lock & lock);\n\n /// Release the currently in progress batch to readers, if it exists.\n /// \\note This call locks m_batches_sync internally.\n /// \\note The batch must not have any work remaining to start, but can be completing already started work.\n /// \\note This call notifys the condition variable to alert readers that new data is available.\n void release_in_progress_batch();\n\n std::shared_ptr m_reader;\n SamplesMode m_samples_mode;\n std::size_t m_max_pending_batches;\n std::size_t m_reads_batch_count;\n gsl::span m_batch_counts;\n std::size_t m_total_batch_count_so_far;\n gsl::span m_batch_rows;\n\n std::uint32_t const m_worker_job_size;\n\n std::mutex m_worker_sync;\n std::condition_variable m_batch_done;\n std::uint32_t m_current_batch;\n\n std::atomic m_finished;\n std::atomic m_has_error;\n mutable std::mutex m_error_mutex;\n pod5::Status m_error;\n std::shared_ptr m_in_progress_batch;\n\n std::mutex m_batches_sync;\n std::atomic m_batches_size;\n std::deque> m_batches;\n\n std::vector m_workers;\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/c_api.cpp", "content": "#include \"pod5_format/c_api.h\"\n\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/file_writer.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/signal_compression.h\"\n#include \"pod5_format/signal_table_reader.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n//---------------------------------------------------------------------------------------------------------------------\nstruct Pod5FileReader {\n Pod5FileReader(std::shared_ptr && reader_) : reader(std::move(reader_)) {}\n\n std::shared_ptr reader;\n};\n\nstruct Pod5FileWriter {\n Pod5FileWriter(std::unique_ptr && writer_) : writer(std::move(writer_)) {}\n\n std::unique_ptr writer;\n};\n\nstruct Pod5ReadRecordBatch {\n Pod5ReadRecordBatch(\n pod5::ReadTableRecordBatch && batch_,\n std::shared_ptr reader)\n : batch(std::move(batch_))\n , reader(std::move(reader))\n {\n }\n\n pod5::ReadTableRecordBatch const batch;\n std::shared_ptr reader;\n};\n\nnamespace {\n//---------------------------------------------------------------------------------------------------------------------\nthread_local pod5_error_t g_pod5_error_no;\nthread_local std::string g_pod5_error_string;\n\nvoid pod5_set_error(arrow::Status status)\n{\n g_pod5_error_no = (pod5_error_t)status.code();\n g_pod5_error_string = status.ToString();\n}\n\nvoid pod5_reset_error()\n{\n g_pod5_error_no = pod5_error_t::POD5_OK;\n g_pod5_error_string.clear();\n}\n\n#define POD5_C_RETURN_NOT_OK(result) \\\n do { \\\n ::arrow::Status __s = (result); \\\n if (!__s.ok()) { \\\n pod5_set_error(__s); \\\n return g_pod5_error_no; \\\n } \\\n } while (0)\n\n#define POD5_C_ASSIGN_OR_RAISE_IMPL(result_name, lhs, rexpr) \\\n auto && result_name = (rexpr); \\\n if (!(result_name).ok()) { \\\n pod5_set_error((result_name).status()); \\\n return g_pod5_error_no; \\\n } \\\n lhs = std::move(result_name).ValueUnsafe();\n\n#define POD5_C_ASSIGN_OR_RAISE(lhs, rexpr) \\\n POD5_C_ASSIGN_OR_RAISE_IMPL( \\\n ARROW_ASSIGN_OR_RAISE_NAME(_error_or_value, __COUNTER__), lhs, rexpr);\n\n//---------------------------------------------------------------------------------------------------------------------\nbool check_string_not_empty(char const * str)\n{\n if (!str) {\n pod5_set_error(arrow::Status::Invalid(\"null string passed to C API\"));\n return false;\n }\n\n if (strlen(str) == 0) {\n pod5_set_error(arrow::Status::Invalid(\"empty string passed to C API\"));\n return false;\n }\n\n return true;\n}\n\nbool check_not_null(void const * ptr)\n{\n if (!ptr) {\n pod5_set_error(arrow::Status::Invalid(\"null passed to C API\"));\n return false;\n }\n return true;\n}\n\nbool check_file_not_null(void const * file)\n{\n if (!file) {\n pod5_set_error(arrow::Status::Invalid(\"null file passed to C API\"));\n return false;\n }\n return true;\n}\n\nbool check_output_pointer_not_null(void const * output)\n{\n if (!output) {\n pod5_set_error(arrow::Status::Invalid(\"null output parameter passed to C API\"));\n return false;\n }\n return true;\n}\n\n//---------------------------------------------------------------------------------------------------------------------\npod5::FileWriterOptions make_internal_writer_options(Pod5WriterOptions const * options)\n{\n pod5::FileWriterOptions internal_options;\n if (options) {\n if (options->max_signal_chunk_size != 0) {\n internal_options.set_max_signal_chunk_size(options->max_signal_chunk_size);\n }\n\n if (options->signal_compression_type == UNCOMPRESSED_SIGNAL) {\n internal_options.set_signal_type(pod5::SignalType::UncompressedSignal);\n }\n\n if (options->signal_table_batch_size != 0) {\n internal_options.set_signal_table_batch_size(options->signal_table_batch_size);\n }\n if (options->read_table_batch_size != 0) {\n internal_options.set_read_table_batch_size(options->read_table_batch_size);\n }\n }\n return internal_options;\n}\n\npod5::FileReaderOptions make_internal_reader_options(Pod5ReaderOptions const & options)\n{\n pod5::FileReaderOptions internal_options;\n internal_options.set_force_disable_file_mapping(options.force_disable_file_mapping);\n return internal_options;\n}\n\n} // namespace\n\nextern \"C\" {\n\n//---------------------------------------------------------------------------------------------------------------------\npod5_error_t pod5_init()\n{\n pod5_reset_error();\n POD5_C_RETURN_NOT_OK(pod5::register_extension_types());\n return POD5_OK;\n}\n\npod5_error_t pod5_terminate()\n{\n pod5_reset_error();\n POD5_C_RETURN_NOT_OK(pod5::unregister_extension_types());\n return POD5_OK;\n}\n\npod5_error_t pod5_get_error_no() { return g_pod5_error_no; }\n\nchar const * pod5_get_error_string() { return g_pod5_error_string.c_str(); }\n\n//---------------------------------------------------------------------------------------------------------------------\nPod5FileReader * pod5_open_file(char const * filename)\n{\n Pod5ReaderOptions_t options{};\n return pod5_open_file_options(filename, &options);\n}\n\nPod5FileReader * pod5_open_file_options(char const * filename, Pod5ReaderOptions_t const * options)\n{\n pod5_reset_error();\n\n if (!check_string_not_empty(filename) || !check_not_null(options)) {\n return nullptr;\n }\n\n auto internal_reader = pod5::open_file_reader(filename, make_internal_reader_options(*options));\n if (!internal_reader.ok()) {\n pod5_set_error(internal_reader.status());\n return nullptr;\n }\n\n auto reader = std::make_unique(std::move(*internal_reader));\n return reader.release();\n}\n\npod5_error_t pod5_close_and_free_reader(Pod5FileReader * file)\n{\n pod5_reset_error();\n\n std::unique_ptr ptr{file};\n ptr.reset();\n return POD5_OK;\n}\n\npod5_error_t pod5_get_file_info(Pod5FileReader_t const * reader, FileInfo * file_info)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(file_info)) {\n return g_pod5_error_no;\n }\n\n auto const metadata = reader->reader->schema_metadata();\n metadata.file_identifier.to_c_array(file_info->file_identifier);\n\n file_info->version.major = metadata.writing_pod5_version.major_version();\n file_info->version.minor = metadata.writing_pod5_version.minor_version();\n file_info->version.revision = metadata.writing_pod5_version.revision_version();\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_file_read_table_location(\n Pod5FileReader_t const * reader,\n EmbeddedFileData_t * file_data)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(file_data)) {\n return g_pod5_error_no;\n }\n auto const & read_table_location = reader->reader->read_table_location();\n\n file_data->file_name = read_table_location.file_path.c_str();\n file_data->offset = read_table_location.offset;\n file_data->length = read_table_location.size;\n return POD5_OK;\n}\n\npod5_error_t pod5_get_file_signal_table_location(\n Pod5FileReader_t const * reader,\n EmbeddedFileData_t * file_data)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(file_data)) {\n return g_pod5_error_no;\n }\n auto const & signal_table_location = reader->reader->signal_table_location();\n\n file_data->file_name = signal_table_location.file_path.c_str();\n file_data->offset = signal_table_location.offset;\n file_data->length = signal_table_location.size;\n return POD5_OK;\n}\n\npod5_error_t pod5_get_file_run_info_table_location(\n Pod5FileReader_t const * reader,\n EmbeddedFileData_t * file_data)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(file_data)) {\n return g_pod5_error_no;\n }\n auto const & run_info_table_location = reader->reader->run_info_table_location();\n\n file_data->file_name = run_info_table_location.file_path.c_str();\n file_data->offset = run_info_table_location.offset;\n file_data->length = run_info_table_location.size;\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_count(Pod5FileReader_t const * reader, size_t * count)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(count)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(*count, reader->reader->read_count());\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_ids(Pod5FileReader_t const * reader, size_t count, read_id_t * read_ids)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(read_ids)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto read_count, reader->reader->read_count());\n if (count < read_count) {\n pod5_set_error(arrow::Status::Invalid(\"array to short to receive read ids\"));\n return g_pod5_error_no;\n }\n\n std::size_t count_so_far = 0;\n for (std::size_t i = 0; i < reader->reader->num_read_record_batches(); ++i) {\n POD5_C_ASSIGN_OR_RAISE(auto const batch, reader->reader->read_read_record_batch(i));\n\n auto const read_id_column = batch.read_id_column();\n auto raw_data = reinterpret_cast(read_id_column->raw_values());\n std::copy(\n raw_data,\n raw_data + (read_id_column->length() * sizeof(read_id_t)),\n reinterpret_cast(read_ids + count_so_far));\n count_so_far += read_id_column->length();\n }\n\n return POD5_OK;\n}\n\npod5_error_t pod5_plan_traversal(\n Pod5FileReader_t const * reader,\n uint8_t const * read_id_array,\n size_t read_id_count,\n uint32_t * batch_counts,\n uint32_t * batch_rows,\n size_t * find_success_count_out)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_not_null(read_id_array)\n || !check_output_pointer_not_null(batch_counts)\n || !check_output_pointer_not_null(batch_rows)\n || !check_output_pointer_not_null(find_success_count_out))\n {\n return g_pod5_error_no;\n }\n\n auto search_input = pod5::ReadIdSearchInput(\n gsl::make_span(reinterpret_cast(read_id_array), read_id_count));\n\n POD5_C_ASSIGN_OR_RAISE(\n auto find_success_count,\n reader->reader->search_for_read_ids(\n search_input,\n gsl::make_span(batch_counts, reader->reader->num_read_record_batches()),\n gsl::make_span(batch_rows, read_id_count)));\n\n // TODO: on MAJOR_VERSION bump drop this out param and do the check internally.\n *find_success_count_out = find_success_count;\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_batch_count(size_t * count, Pod5FileReader const * reader)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(count)) {\n return g_pod5_error_no;\n }\n\n *count = reader->reader->num_read_record_batches();\n return POD5_OK;\n}\n\npod5_error_t\npod5_get_read_batch(Pod5ReadRecordBatch ** batch, Pod5FileReader const * reader, size_t index)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_output_pointer_not_null(batch)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto internal_batch, reader->reader->read_read_record_batch(index));\n\n auto wrapped_batch =\n std::make_unique(std::move(internal_batch), reader->reader);\n\n *batch = wrapped_batch.release();\n return POD5_OK;\n}\n\npod5_error_t pod5_free_read_batch(Pod5ReadRecordBatch * batch)\n{\n pod5_reset_error();\n\n std::unique_ptr ptr{batch};\n ptr.reset();\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_batch_row_count(size_t * count, Pod5ReadRecordBatch const * batch)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(count)) {\n return g_pod5_error_no;\n }\n\n *count = batch->batch.num_rows();\n return POD5_OK;\n}\n\nstatic pod5_error_t check_row_index_and_set_error(size_t row, int64_t batch_size)\n{\n if (row > static_cast(std::numeric_limits::max())\n || static_cast(row) >= batch_size)\n {\n pod5_set_error(\n arrow::Status::IndexError(\n \"Invalid index into batch. Index \", row, \" with batch size \", batch_size));\n return g_pod5_error_no;\n }\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_batch_row_info_data(\n Pod5ReadRecordBatch_t const * batch,\n size_t row,\n uint16_t struct_version,\n void * row_data,\n uint16_t * read_table_version)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(row_data)\n || !check_output_pointer_not_null(read_table_version))\n {\n return g_pod5_error_no;\n }\n\n static_assert(\n READ_BATCH_ROW_INFO_VERSION == READ_BATCH_ROW_INFO_VERSION_4,\n \"New versions must be explicitly loaded\");\n\n auto load_common_v3_v4_fields = [](pod5::ReadTableRecordColumns const & cols,\n std::size_t row,\n auto * typed_row_data) {\n // Inform the caller of the version of the input table.\n\n if (check_row_index_and_set_error(row, cols.read_id->length()) != POD5_OK) {\n return g_pod5_error_no;\n }\n\n auto read_id_val = cols.read_id->Value(row);\n read_id_val.to_c_array(typed_row_data->read_id);\n\n typed_row_data->read_number = cols.read_number->Value(row);\n typed_row_data->start_sample = cols.start_sample->Value(row);\n typed_row_data->median_before = cols.median_before->Value(row);\n typed_row_data->channel = cols.channel->Value(row);\n typed_row_data->well = cols.well->Value(row);\n auto const & pore_type_col = cols.pore_type->indices();\n typed_row_data->pore_type =\n static_cast(*pore_type_col).Value(row);\n typed_row_data->calibration_offset = cols.calibration_offset->Value(row);\n typed_row_data->calibration_scale = cols.calibration_scale->Value(row);\n auto const & end_reason_col = cols.end_reason->indices();\n typed_row_data->end_reason =\n static_cast(*end_reason_col).Value(row);\n typed_row_data->end_reason_forced = cols.end_reason_forced->Value(row);\n auto const & run_info_col = cols.run_info->indices();\n typed_row_data->run_info = static_cast(*run_info_col).Value(row);\n typed_row_data->num_minknow_events = cols.num_minknow_events->Value(row);\n typed_row_data->tracked_scaling_scale = cols.tracked_scaling_scale->Value(row);\n typed_row_data->tracked_scaling_shift = cols.tracked_scaling_shift->Value(row);\n typed_row_data->predicted_scaling_scale = cols.predicted_scaling_scale->Value(row);\n typed_row_data->predicted_scaling_shift = cols.predicted_scaling_shift->Value(row);\n typed_row_data->num_reads_since_mux_change = cols.num_reads_since_mux_change->Value(row);\n typed_row_data->time_since_mux_change = cols.time_since_mux_change->Value(row);\n\n typed_row_data->signal_row_count = cols.signal->value_length(row);\n typed_row_data->num_samples = cols.num_samples->Value(row);\n return POD5_OK;\n };\n\n if (struct_version == READ_BATCH_ROW_INFO_VERSION_3) {\n auto typed_row_data = static_cast(row_data);\n\n POD5_C_ASSIGN_OR_RAISE(auto cols, batch->batch.columns());\n *read_table_version = cols.table_version.as_int();\n\n auto result = load_common_v3_v4_fields(cols, row, typed_row_data);\n if (result != POD5_OK) {\n return result;\n }\n } else if (struct_version == READ_BATCH_ROW_INFO_VERSION_4) {\n auto typed_row_data = static_cast(row_data);\n\n POD5_C_ASSIGN_OR_RAISE(auto cols, batch->batch.columns());\n *read_table_version = cols.table_version.as_int();\n\n auto result = load_common_v3_v4_fields(cols, row, typed_row_data);\n if (result != POD5_OK) {\n return result;\n }\n\n // This is the only difference between v3 and v4.\n typed_row_data->open_pore_level = cols.open_pore_level->Value(row);\n } else {\n pod5_set_error(\n arrow::Status::Invalid(\"Invalid struct version '\", struct_version, \"' passed\"));\n return g_pod5_error_no;\n }\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_signal_row_indices(\n Pod5ReadRecordBatch const * batch,\n size_t row,\n int64_t signal_row_indices_count,\n uint64_t * signal_row_indices)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(signal_row_indices)) {\n return g_pod5_error_no;\n }\n\n auto const signal_col = batch->batch.signal_column();\n if (check_row_index_and_set_error(row, signal_col->length()) != POD5_OK) {\n return g_pod5_error_no;\n }\n\n auto const row_data =\n std::static_pointer_cast(signal_col->value_slice(row));\n\n if (signal_row_indices_count != row_data->length()) {\n pod5_set_error(\n pod5::Status::Invalid(\n \"Incorrect number of signal indices, expected \",\n row_data->length(),\n \" received \",\n signal_row_indices_count));\n return g_pod5_error_no;\n }\n\n for (std::int64_t i = 0; i < signal_row_indices_count; ++i) {\n signal_row_indices[i] = row_data->Value(i);\n }\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_calibration_extra_info(\n Pod5ReadRecordBatch_t const * batch,\n size_t row,\n CalibrationExtraData_t * calibration_extra_data)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(calibration_extra_data)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto cols, batch->batch.columns());\n\n if (check_row_index_and_set_error(row, cols.calibration_scale->length()) != POD5_OK) {\n return g_pod5_error_no;\n }\n\n auto scale = cols.calibration_scale->Value(row);\n auto const & run_info_indices = cols.run_info->indices();\n auto const run_info_dict_index =\n static_cast(*run_info_indices).Value(row);\n\n POD5_C_ASSIGN_OR_RAISE(\n auto const acquisition_id, batch->batch.get_run_info(run_info_dict_index));\n POD5_C_ASSIGN_OR_RAISE(auto run_info_data, batch->reader->find_run_info(acquisition_id));\n\n calibration_extra_data->digitisation = run_info_data->adc_max - run_info_data->adc_min + 1;\n calibration_extra_data->range = scale * calibration_extra_data->digitisation;\n\n return POD5_OK;\n}\n\nnamespace {\n\nstruct RunInfoDataCHelper : public RunInfoDictData {\n struct InternalMapHelper {\n std::vector keys;\n std::vector values;\n };\n\n RunInfoDataCHelper(std::shared_ptr && internal_data_)\n : internal_data(std::move(internal_data_))\n {\n acquisition_id = internal_data->acquisition_id.c_str();\n acquisition_start_time_ms = internal_data->acquisition_start_time;\n adc_max = internal_data->adc_max;\n adc_min = internal_data->adc_min;\n context_tags = map_to_c(internal_data->context_tags, context_tags_helper);\n experiment_name = internal_data->experiment_name.c_str();\n flow_cell_id = internal_data->flow_cell_id.c_str();\n flow_cell_product_code = internal_data->flow_cell_product_code.c_str();\n protocol_name = internal_data->protocol_name.c_str();\n protocol_run_id = internal_data->protocol_run_id.c_str();\n protocol_start_time_ms = internal_data->protocol_start_time;\n sample_id = internal_data->sample_id.c_str();\n sample_rate = internal_data->sample_rate;\n sequencing_kit = internal_data->sequencing_kit.c_str();\n sequencer_position = internal_data->sequencer_position.c_str();\n sequencer_position_type = internal_data->sequencer_position_type.c_str();\n software = internal_data->software.c_str();\n system_name = internal_data->system_name.c_str();\n system_type = internal_data->system_type.c_str();\n tracking_id = map_to_c(internal_data->tracking_id, tracking_id_helper);\n }\n\n KeyValueData map_to_c(pod5::RunInfoData::MapType const & map, InternalMapHelper & helper)\n {\n helper.keys.reserve(map.size());\n helper.values.reserve(map.size());\n for (auto const & item : map) {\n helper.keys.push_back(item.first.c_str());\n helper.values.push_back(item.second.c_str());\n }\n\n KeyValueData result;\n result.size = helper.keys.size();\n result.keys = helper.keys.data();\n result.values = helper.values.data();\n return result;\n }\n\n std::shared_ptr internal_data;\n InternalMapHelper context_tags_helper;\n InternalMapHelper tracking_id_helper;\n};\n\n} // namespace\n\npod5_error_t pod5_get_run_info(\n Pod5ReadRecordBatch const * batch,\n int16_t run_info,\n RunInfoDictData ** run_info_data)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(run_info_data)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto const acquisition_id, batch->batch.get_run_info(run_info));\n POD5_C_ASSIGN_OR_RAISE(auto internal_data, batch->reader->find_run_info(acquisition_id));\n\n auto data = std::make_unique(std::move(internal_data));\n *run_info_data = data.release();\n return POD5_OK;\n}\n\npod5_error_t pod5_get_file_run_info(\n Pod5FileReader_t const * file,\n run_info_index_t run_info_index,\n RunInfoDictData_t ** run_info_data)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(file) || !check_output_pointer_not_null(run_info_data)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto internal_data, file->reader->get_run_info(run_info_index));\n\n auto data = std::make_unique(std::move(internal_data));\n *run_info_data = data.release();\n return POD5_OK;\n}\n\npod5_error_t pod5_free_run_info(RunInfoDictData_t * run_info_data)\n{\n pod5_reset_error();\n\n std::unique_ptr helper(static_cast(run_info_data));\n helper.reset();\n\n return POD5_OK;\n}\n\npod5_error_t pod5_release_run_info(RunInfoDictData * run_info_data)\n{\n return pod5_free_run_info(run_info_data);\n}\n\npod5_error_t pod5_get_file_run_info_count(\n Pod5FileReader_t const * file,\n run_info_index_t * run_info_count)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(file) || !check_output_pointer_not_null(run_info_count)) {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(*run_info_count, file->reader->get_run_info_count());\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_end_reason(\n Pod5ReadRecordBatch_t const * batch,\n int16_t end_reason,\n pod5_end_reason * end_reason_value,\n char * end_reason_string_value,\n size_t * end_reason_string_value_size)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(end_reason_value)\n || !check_output_pointer_not_null(end_reason_string_value)\n || !check_output_pointer_not_null(end_reason_string_value_size))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto const end_reason_val, batch->batch.get_end_reason(end_reason));\n auto const input_buffer_len = *end_reason_string_value_size;\n *end_reason_string_value_size = end_reason_val.second.size() + 1;\n if (end_reason_val.second.size() >= input_buffer_len) {\n return POD5_ERROR_STRING_NOT_LONG_ENOUGH;\n }\n\n *end_reason_value = POD5_END_REASON_UNKNOWN;\n switch (end_reason_val.first) {\n case pod5::ReadEndReason::mux_change:\n *end_reason_value = POD5_END_REASON_MUX_CHANGE;\n break;\n case pod5::ReadEndReason::unblock_mux_change:\n *end_reason_value = POD5_END_REASON_UNBLOCK_MUX_CHANGE;\n break;\n case pod5::ReadEndReason::data_service_unblock_mux_change:\n *end_reason_value = POD5_END_REASON_DATA_SERVICE_UNBLOCK_MUX_CHANGE;\n break;\n case pod5::ReadEndReason::signal_positive:\n *end_reason_value = POD5_END_REASON_SIGNAL_POSITIVE;\n break;\n case pod5::ReadEndReason::signal_negative:\n *end_reason_value = POD5_END_REASON_SIGNAL_NEGATIVE;\n break;\n case pod5::ReadEndReason::api_request:\n *end_reason_value = POD5_END_REASON_API_REQUEST;\n break;\n case pod5::ReadEndReason::device_data_error:\n *end_reason_value = POD5_END_REASON_DEVICE_DATA_ERROR;\n break;\n case pod5::ReadEndReason::analysis_config_change:\n *end_reason_value = POD5_END_REASON_ANALYSIS_CONFIG_CHANGE;\n break;\n case pod5::ReadEndReason::paused:\n *end_reason_value = POD5_END_REASON_PAUSED;\n break;\n case pod5::ReadEndReason::unknown:\n *end_reason_value = POD5_END_REASON_UNKNOWN;\n break;\n }\n\n std::copy(end_reason_val.second.begin(), end_reason_val.second.end(), end_reason_string_value);\n end_reason_string_value[*end_reason_string_value_size] = '\\0';\n return POD5_OK;\n}\n\npod5_error_t pod5_get_pore_type(\n Pod5ReadRecordBatch_t const * batch,\n int16_t pore_type,\n char * pore_type_string_value,\n size_t * pore_type_string_value_size)\n{\n pod5_reset_error();\n\n if (!check_not_null(batch) || !check_output_pointer_not_null(pore_type_string_value)\n || !check_output_pointer_not_null(pore_type_string_value_size))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto const pore_type_str, batch->batch.get_pore_type(pore_type));\n auto const input_buffer_len = *pore_type_string_value_size;\n *pore_type_string_value_size = pore_type_str.size() + 1;\n if (pore_type_str.size() >= input_buffer_len) {\n return POD5_ERROR_STRING_NOT_LONG_ENOUGH;\n }\n\n std::copy(pore_type_str.begin(), pore_type_str.end(), pore_type_string_value);\n pore_type_string_value[*pore_type_string_value_size] = '\\0';\n return POD5_OK;\n}\n\nnamespace {\n\nclass SignalRowInfoCHelper : public SignalRowInfo {\npublic:\n SignalRowInfoCHelper(pod5::SignalTableRecordBatch && b) : batch(std::move(b)) {}\n\n pod5::SignalTableRecordBatch const batch;\n};\n\n} // namespace\n\npod5_error_t pod5_get_signal_row_info(\n Pod5FileReader const * reader,\n size_t signal_rows_count,\n uint64_t const * signal_rows,\n SignalRowInfo ** signal_row_info)\n{\n pod5_reset_error();\n\n // Check for a valid reader.\n if (!check_file_not_null(reader)) {\n return g_pod5_error_no;\n }\n\n // Check for valid inputs.\n if (signal_rows_count == 0) {\n // Nothing to do.\n return POD5_OK;\n } else if (!check_not_null(signal_rows) || !check_output_pointer_not_null(signal_row_info)) {\n return g_pod5_error_no;\n }\n\n // Sort all rows first, in order to make searching faster.\n std::vector signal_rows_sorted{signal_rows, signal_rows + signal_rows_count};\n std::sort(signal_rows_sorted.begin(), signal_rows_sorted.end());\n\n // Store allocations to a temporary buffer so that we don't leak them on failure.\n std::vector> row_infos(signal_rows_count);\n\n // Then loop all rows, forward.\n for (std::size_t completed_rows = 0; completed_rows < signal_rows_sorted.size();\n completed_rows++)\n {\n auto const start_row = signal_rows_sorted[completed_rows];\n\n std::size_t batch_row = 0;\n POD5_C_ASSIGN_OR_RAISE(\n std::size_t row_batch,\n (reader->reader->signal_batch_for_row_id(start_row, &batch_row)));\n POD5_C_ASSIGN_OR_RAISE(auto batch, reader->reader->read_signal_record_batch(row_batch));\n\n auto output = std::make_unique(std::move(batch));\n\n output->batch_index = start_row;\n output->batch_row_index = batch_row;\n\n auto samples = output->batch.samples_column();\n output->stored_sample_count = samples->Value(batch_row);\n POD5_C_ASSIGN_OR_RAISE(\n output->stored_byte_count, output->batch.samples_byte_count(batch_row));\n\n row_infos[completed_rows] = std::move(output);\n }\n\n // Pass ownership of the info back to the caller.\n for (std::size_t row_idx = 0; row_idx < signal_rows_count; row_idx++) {\n signal_row_info[row_idx] = row_infos[row_idx].release();\n }\n return POD5_OK;\n}\n\npod5_error_t pod5_free_signal_row_info(size_t signal_rows_count, SignalRowInfo_t ** signal_row_info)\n{\n pod5_reset_error();\n\n if (signal_rows_count > 0 && !check_not_null(signal_row_info)) {\n return g_pod5_error_no;\n }\n\n for (std::size_t i = 0; i < signal_rows_count; ++i) {\n std::unique_ptr helper(\n static_cast(signal_row_info[i]));\n helper.reset();\n }\n return POD5_OK;\n}\n\npod5_error_t pod5_get_signal(\n Pod5FileReader const * reader,\n SignalRowInfo_t const * row_info,\n size_t sample_count,\n int16_t * sample_data)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_not_null(row_info)\n || !check_output_pointer_not_null(sample_data))\n {\n return g_pod5_error_no;\n }\n\n auto * row_info_data = static_cast(row_info);\n\n POD5_C_RETURN_NOT_OK(row_info_data->batch.extract_signal_row(\n row_info->batch_row_index, gsl::make_span(sample_data, sample_count)));\n\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_complete_sample_count(\n Pod5FileReader_t const * reader,\n Pod5ReadRecordBatch_t const * batch,\n size_t batch_row,\n size_t * sample_count)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_not_null(batch)\n || !check_output_pointer_not_null(sample_count))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto const & signal_rows, batch->batch.get_signal_rows(batch_row));\n\n POD5_C_ASSIGN_OR_RAISE(\n *sample_count,\n reader->reader->extract_sample_count(\n gsl::make_span(signal_rows->raw_values(), signal_rows->length())));\n return POD5_OK;\n}\n\npod5_error_t pod5_get_read_complete_signal(\n Pod5FileReader_t const * reader,\n Pod5ReadRecordBatch_t const * batch,\n size_t batch_row,\n size_t sample_count,\n int16_t * signal)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(reader) || !check_not_null(batch)\n || !check_output_pointer_not_null(signal))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(auto const & signal_rows, batch->batch.get_signal_rows(batch_row));\n\n POD5_C_RETURN_NOT_OK(reader->reader->extract_samples(\n gsl::make_span(signal_rows->raw_values(), signal_rows->length()),\n gsl::make_span(signal, sample_count)));\n return POD5_OK;\n}\n\n//---------------------------------------------------------------------------------------------------------------------\nPod5FileWriter *\npod5_create_file(char const * filename, char const * writer_name, Pod5WriterOptions const * options)\n{\n pod5_reset_error();\n\n if (!check_string_not_empty(filename) || !check_string_not_empty(writer_name)) {\n return nullptr;\n }\n\n auto internal_writer =\n pod5::create_file_writer(filename, writer_name, make_internal_writer_options(options));\n if (!internal_writer.ok()) {\n pod5_set_error(internal_writer.status());\n return nullptr;\n }\n\n auto writer = std::make_unique(std::move(*internal_writer));\n return writer.release();\n}\n\npod5_error_t pod5_close_and_free_writer(Pod5FileWriter * file)\n{\n pod5_reset_error();\n\n std::unique_ptr ptr{file};\n if (ptr) {\n POD5_C_RETURN_NOT_OK(ptr->writer->close());\n }\n\n ptr.reset();\n return POD5_OK;\n}\n\npod5_error_t pod5_add_pore(int16_t * pore_index, Pod5FileWriter * file, char const * pore_type)\n{\n pod5_reset_error();\n\n if (!check_string_not_empty(pore_type) || !check_file_not_null(file)\n || !check_output_pointer_not_null(pore_index))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(*pore_index, file->writer->add_pore_type(pore_type));\n return POD5_OK;\n}\n\npod5_error_t pod5_add_run_info(\n int16_t * run_info_index,\n Pod5FileWriter * file,\n char const * acquisition_id,\n int64_t acquisition_start_time_ms,\n int16_t adc_max,\n int16_t adc_min,\n size_t context_tags_count,\n char const ** context_tags_keys,\n char const ** context_tags_values,\n char const * experiment_name,\n char const * flow_cell_id,\n char const * flow_cell_product_code,\n char const * protocol_name,\n char const * protocol_run_id,\n int64_t protocol_start_time_ms,\n char const * sample_id,\n uint16_t sample_rate,\n char const * sequencing_kit,\n char const * sequencer_position,\n char const * sequencer_position_type,\n char const * software,\n char const * system_name,\n char const * system_type,\n size_t tracking_id_count,\n char const ** tracking_id_keys,\n char const ** tracking_id_values)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(file) || !check_not_null(run_info_index)\n || !check_string_not_empty(acquisition_id) || !check_string_not_empty(experiment_name)\n || !check_string_not_empty(flow_cell_id) || !check_string_not_empty(flow_cell_product_code)\n || !check_string_not_empty(protocol_name) || !check_string_not_empty(protocol_run_id)\n || !check_string_not_empty(sample_id) || !check_string_not_empty(sequencing_kit)\n || !check_string_not_empty(sequencer_position)\n || !check_string_not_empty(sequencer_position_type) || !check_string_not_empty(software)\n || !check_string_not_empty(system_name) || !check_string_not_empty(system_type))\n {\n return g_pod5_error_no;\n }\n\n auto const parse_map =\n [](std::size_t tracking_id_count,\n char const ** tracking_id_keys,\n char const ** tracking_id_values) -> pod5::Result {\n if (!check_not_null(tracking_id_keys) || !check_not_null(tracking_id_values)) {\n return arrow::Status::Invalid(g_pod5_error_string);\n }\n\n pod5::RunInfoData::MapType result;\n for (std::size_t i = 0; i < tracking_id_count; ++i) {\n auto key = tracking_id_keys[i];\n auto value = tracking_id_values[i];\n if (!check_string_not_empty(key) || !check_string_not_empty(value)) {\n return arrow::Status::Invalid(g_pod5_error_string);\n }\n\n result.emplace_back(key, value);\n }\n return result;\n };\n\n POD5_C_ASSIGN_OR_RAISE(\n auto const context_tags,\n parse_map(context_tags_count, context_tags_keys, context_tags_values));\n POD5_C_ASSIGN_OR_RAISE(\n auto const tracking_id, parse_map(tracking_id_count, tracking_id_keys, tracking_id_values));\n\n POD5_C_ASSIGN_OR_RAISE(\n *run_info_index,\n file->writer->add_run_info(\n pod5::RunInfoData(\n acquisition_id,\n acquisition_start_time_ms,\n adc_max,\n adc_min,\n context_tags,\n experiment_name,\n flow_cell_id,\n flow_cell_product_code,\n protocol_name,\n protocol_run_id,\n protocol_start_time_ms,\n sample_id,\n sample_rate,\n sequencing_kit,\n sequencer_position,\n sequencer_position_type,\n software,\n system_name,\n system_type,\n tracking_id)));\n\n return POD5_OK;\n}\n\nstatic bool check_read_data_struct(std::uint16_t struct_version, void const * row_data)\n{\n static_assert(\n READ_BATCH_ROW_INFO_VERSION == READ_BATCH_ROW_INFO_VERSION_4,\n \"New versions must be explicitly loaded\");\n\n if (!check_not_null(row_data)) {\n return false;\n }\n\n if (struct_version < READ_BATCH_ROW_INFO_VERSION_3) {\n pod5_set_error(arrow::Status::Invalid(\"Unable to write V1 + V2 reads, update to V3 API.\"));\n return false;\n }\n\n auto check_common_v3_v4_fields = [](auto typed_row_data) -> bool {\n return check_not_null(typed_row_data->read_id)\n && check_not_null(typed_row_data->read_number)\n && check_not_null(typed_row_data->start_sample)\n && check_not_null(typed_row_data->median_before)\n && check_not_null(typed_row_data->channel) && check_not_null(typed_row_data->well)\n && check_not_null(typed_row_data->pore_type)\n && check_not_null(typed_row_data->calibration_offset)\n && check_not_null(typed_row_data->calibration_scale)\n && check_not_null(typed_row_data->end_reason)\n && check_not_null(typed_row_data->end_reason_forced)\n && check_not_null(typed_row_data->run_info_id)\n && check_not_null(typed_row_data->num_minknow_events)\n && check_not_null(typed_row_data->tracked_scaling_scale)\n && check_not_null(typed_row_data->tracked_scaling_shift)\n && check_not_null(typed_row_data->predicted_scaling_scale)\n && check_not_null(typed_row_data->predicted_scaling_shift)\n && check_not_null(typed_row_data->num_reads_since_mux_change)\n && check_not_null(typed_row_data->time_since_mux_change);\n };\n\n if (struct_version == READ_BATCH_ROW_INFO_VERSION_3) {\n auto const * typed_row_data = static_cast(row_data);\n\n if (!check_common_v3_v4_fields(typed_row_data)) {\n return false;\n }\n }\n\n if (struct_version == READ_BATCH_ROW_INFO_VERSION_4) {\n auto const * typed_row_data = static_cast(row_data);\n\n if (!check_common_v3_v4_fields(typed_row_data)\n || !check_not_null(typed_row_data->open_pore_level))\n {\n return false;\n }\n }\n\n return true;\n}\n\nstatic bool load_struct_row_into_read_data(\n std::unique_ptr const & writer,\n pod5::ReadData & read_data,\n std::uint16_t struct_version,\n void const * row_data,\n std::uint32_t row_id)\n{\n static_assert(\n READ_BATCH_ROW_INFO_VERSION == READ_BATCH_ROW_INFO_VERSION_4,\n \"New versions must be explicitly loaded\");\n\n auto load_common_v3_v4_fields = [](std::unique_ptr const & writer,\n auto const * typed_row_data,\n std::uint32_t row_id,\n pod5::ReadData & read_data) {\n pod5::Uuid read_id_uuid{typed_row_data->read_id[row_id]};\n\n std::optional end_reason_internal;\n switch (typed_row_data->end_reason[row_id]) {\n case POD5_END_REASON_UNKNOWN:\n end_reason_internal = pod5::ReadEndReason::unknown;\n break;\n case POD5_END_REASON_MUX_CHANGE:\n end_reason_internal = pod5::ReadEndReason::mux_change;\n break;\n case POD5_END_REASON_UNBLOCK_MUX_CHANGE:\n end_reason_internal = pod5::ReadEndReason::unblock_mux_change;\n break;\n case POD5_END_REASON_DATA_SERVICE_UNBLOCK_MUX_CHANGE:\n end_reason_internal = pod5::ReadEndReason::data_service_unblock_mux_change;\n break;\n case POD5_END_REASON_SIGNAL_POSITIVE:\n end_reason_internal = pod5::ReadEndReason::signal_positive;\n break;\n case POD5_END_REASON_SIGNAL_NEGATIVE:\n end_reason_internal = pod5::ReadEndReason::signal_negative;\n break;\n case POD5_END_REASON_API_REQUEST:\n end_reason_internal = pod5::ReadEndReason::api_request;\n break;\n case POD5_END_REASON_DEVICE_DATA_ERROR:\n end_reason_internal = pod5::ReadEndReason::device_data_error;\n break;\n case POD5_END_REASON_ANALYSIS_CONFIG_CHANGE:\n end_reason_internal = pod5::ReadEndReason::analysis_config_change;\n break;\n case POD5_END_REASON_PAUSED:\n end_reason_internal = pod5::ReadEndReason::paused;\n break;\n }\n if (!end_reason_internal.has_value()) {\n pod5_set_error(\n arrow::Status::Invalid(\n \"out of range end reason \",\n typed_row_data->end_reason[row_id],\n \" passed to add read\"));\n return false;\n }\n\n auto const end_reason_index = writer->lookup_end_reason(*end_reason_internal);\n if (!end_reason_index.ok()) {\n pod5_set_error(end_reason_index.status());\n return false;\n }\n\n read_data = pod5::ReadData{\n read_id_uuid,\n typed_row_data->read_number[row_id],\n typed_row_data->start_sample[row_id],\n typed_row_data->channel[row_id],\n typed_row_data->well[row_id],\n typed_row_data->pore_type[row_id],\n typed_row_data->calibration_offset[row_id],\n typed_row_data->calibration_scale[row_id],\n typed_row_data->median_before[row_id],\n *end_reason_index,\n typed_row_data->end_reason_forced[row_id] != 0,\n typed_row_data->run_info_id[row_id],\n typed_row_data->num_minknow_events[row_id],\n typed_row_data->tracked_scaling_scale[row_id],\n typed_row_data->tracked_scaling_shift[row_id],\n typed_row_data->predicted_scaling_scale[row_id],\n typed_row_data->predicted_scaling_shift[row_id],\n typed_row_data->num_reads_since_mux_change[row_id],\n typed_row_data->time_since_mux_change[row_id],\n // open_pore_level is only present in v4.\n std::numeric_limits::quiet_NaN()};\n return true;\n };\n\n // Version 0-2 are no longer supported for writing.\n if (struct_version == READ_BATCH_ROW_INFO_VERSION_4) {\n auto const * typed_row_data = static_cast(row_data);\n\n if (!load_common_v3_v4_fields(writer, typed_row_data, row_id, read_data)) {\n return false;\n }\n read_data.open_pore_level = typed_row_data->open_pore_level[row_id];\n } else if (struct_version == READ_BATCH_ROW_INFO_VERSION_3) {\n auto const * typed_row_data = static_cast(row_data);\n\n if (!load_common_v3_v4_fields(writer, typed_row_data, row_id, read_data)) {\n return false;\n }\n } else {\n pod5_set_error(\n arrow::Status::Invalid(\"Invalid writer struct version '\", struct_version, \"' passed\"));\n return false;\n }\n return true;\n};\n\npod5_error_t pod5_add_reads_data(\n Pod5FileWriter_t * file,\n uint32_t read_count,\n uint16_t struct_version,\n void const * row_data,\n int16_t const ** signal,\n uint32_t const * signal_size)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(file)) {\n return g_pod5_error_no;\n }\n if (read_count == 0) {\n return POD5_OK;\n }\n if (!check_read_data_struct(struct_version, row_data) || !check_not_null(signal)\n || !check_not_null(signal_size))\n {\n return g_pod5_error_no;\n }\n for (std::uint32_t read = 0; read < read_count; ++read) {\n if (!check_not_null(signal[read])) {\n return g_pod5_error_no;\n }\n }\n\n for (std::uint32_t read = 0; read < read_count; ++read) {\n pod5::ReadData read_data;\n if (!load_struct_row_into_read_data(\n file->writer, read_data, struct_version, row_data, read))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_RETURN_NOT_OK(file->writer->add_complete_read(\n read_data, gsl::make_span(signal[read], signal_size[read])));\n }\n\n return POD5_OK;\n}\n\npod5_error_t pod5_add_reads_data_pre_compressed(\n Pod5FileWriter_t * file,\n uint32_t read_count,\n uint16_t struct_version,\n void const * row_data,\n char const *** compressed_signal,\n size_t const ** compressed_signal_size,\n uint32_t const ** sample_counts,\n size_t const * signal_chunk_count)\n{\n pod5_reset_error();\n\n if (!check_file_not_null(file)) {\n return g_pod5_error_no;\n }\n if (read_count == 0) {\n return POD5_OK;\n }\n if (!check_read_data_struct(struct_version, row_data) || !check_not_null(compressed_signal)\n || !check_not_null(compressed_signal_size) || !check_not_null(sample_counts)\n || !check_not_null(signal_chunk_count))\n {\n return g_pod5_error_no;\n }\n for (std::uint32_t read = 0; read < read_count; ++read) {\n if (!check_not_null(compressed_signal[read])\n || !check_not_null(compressed_signal_size[read])\n || !check_not_null(sample_counts[read]))\n {\n return g_pod5_error_no;\n }\n }\n\n for (std::uint32_t read = 0; read < read_count; ++read) {\n pod5::ReadData read_data;\n if (!load_struct_row_into_read_data(\n file->writer, read_data, struct_version, row_data, read))\n {\n return g_pod5_error_no;\n }\n\n std::uint64_t total_sample_count = 0;\n std::vector signal_rows;\n for (std::size_t i = 0; i < signal_chunk_count[read]; ++i) {\n auto signal = compressed_signal[read][i];\n auto signal_size = compressed_signal_size[read][i];\n auto sample_count = sample_counts[read][i];\n total_sample_count += sample_count;\n POD5_C_ASSIGN_OR_RAISE(\n auto row_id,\n file->writer->add_pre_compressed_signal(\n read_data.read_id,\n gsl::make_span(signal, signal_size).as_span(),\n sample_count));\n signal_rows.push_back(row_id);\n }\n\n POD5_C_RETURN_NOT_OK(file->writer->add_complete_read(\n read_data, gsl::make_span(signal_rows), total_sample_count));\n }\n return POD5_OK;\n}\n\nsize_t pod5_vbz_compressed_signal_max_size(size_t sample_count)\n{\n pod5_reset_error();\n\n auto const compressed_size = pod5::compressed_signal_max_size(sample_count);\n if (!compressed_size.ok()) {\n // TODO: on MAJOR_VERSION bump change this to return an error code.\n pod5_set_error(compressed_size.status());\n return 0;\n } else {\n return compressed_size.ValueUnsafe();\n }\n}\n\npod5_error_t pod5_vbz_compress_signal(\n int16_t const * signal,\n size_t signal_size,\n char * compressed_signal_out,\n size_t * compressed_signal_size)\n{\n pod5_reset_error();\n\n if (!check_not_null(signal) || !check_output_pointer_not_null(compressed_signal_out)\n || !check_output_pointer_not_null(compressed_signal_size))\n {\n return g_pod5_error_no;\n }\n\n POD5_C_ASSIGN_OR_RAISE(\n auto buffer,\n pod5::compress_signal(gsl::make_span(signal, signal_size), arrow::system_memory_pool()));\n\n if ((std::size_t)buffer->size() > *compressed_signal_size) {\n pod5_set_error(\n pod5::Status::Invalid(\n \"Compressed signal size (\",\n buffer->size(),\n \") is greater than provided buffer size (\",\n compressed_signal_size,\n \")\"));\n return g_pod5_error_no;\n }\n\n std::copy(buffer->data(), buffer->data() + buffer->size(), compressed_signal_out);\n *compressed_signal_size = buffer->size();\n\n return POD5_OK;\n}\n\npod5_error_t pod5_vbz_decompress_signal(\n char const * compressed_signal,\n size_t compressed_signal_size,\n size_t sample_count,\n int16_t * signal_out)\n{\n pod5_reset_error();\n\n if (!check_not_null(compressed_signal) || !check_output_pointer_not_null(signal_out)) {\n return g_pod5_error_no;\n }\n\n auto const in_span =\n gsl::make_span(compressed_signal, compressed_signal_size).as_span();\n auto out_span = gsl::make_span(signal_out, sample_count);\n POD5_C_RETURN_NOT_OK(pod5::decompress_signal(in_span, arrow::system_memory_pool(), out_span));\n\n return POD5_OK;\n}\n\npod5_error_t pod5_format_read_id(read_id_t const read_id, char * read_id_string)\n{\n pod5_reset_error();\n\n if (!check_not_null(read_id) || !check_output_pointer_not_null(read_id_string)) {\n return g_pod5_error_no;\n }\n\n auto * uuid_data = reinterpret_cast(read_id);\n uuid_data->write_to(read_id_string);\n read_id_string[36] = '\\0';\n\n return POD5_OK;\n}\n}\n" }, { "path": "c++/pod5_format/c_api.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n\n#include \n#include \n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n#ifndef _WIN32\n#define POD5_DEPRECATED __attribute__((deprecated))\n#elif (__STDC_VERSION__ >= 202000)\n#define POD5_DEPRECATED [[deprecated]]\n#else\n#define POD5_DEPRECATED\n#endif\n\n/// All functions are thread safe unless otherwise stated. Types may be used by multiple\n/// threads as long as the functions being called only take them by const pointer.\n\nstruct Pod5FileReader;\ntypedef struct Pod5FileReader Pod5FileReader_t;\nstruct Pod5FileWriter;\ntypedef struct Pod5FileWriter Pod5FileWriter_t;\nstruct Pod5ReadRecordBatch;\ntypedef struct Pod5ReadRecordBatch Pod5ReadRecordBatch_t;\n\n//---------------------------------------------------------------------------------------------------------------------\n// Error management\n//---------------------------------------------------------------------------------------------------------------------\n\n/// \\brief Integer error codes.\n/// \\note Taken from the arrow status enum.\nenum pod5_error {\n POD5_OK = 0,\n POD5_ERROR_OUTOFMEMORY = 1,\n POD5_ERROR_KEYERROR = 2,\n POD5_ERROR_TYPEERROR = 3,\n POD5_ERROR_INVALID = 4,\n POD5_ERROR_IOERROR = 5,\n POD5_ERROR_CAPACITYERROR = 6,\n POD5_ERROR_INDEXERROR = 7,\n POD5_ERROR_CANCELLED = 8,\n POD5_ERROR_UNKNOWNERROR = 9,\n POD5_ERROR_NOTIMPLEMENTED = 10,\n POD5_ERROR_SERIALIZATIONERROR = 11,\n POD5_ERROR_STRING_NOT_LONG_ENOUGH = 12,\n};\ntypedef enum pod5_error pod5_error_t;\n\n/// \\brief Get the most recent error number from all pod5 api's on the current thread.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_error_no();\n/// \\brief Get the most recent error description string from all pod5 api's on the current thread.\n/// \\note The string's lifetime is internally managed, a caller should not free it.\nPOD5_FORMAT_EXPORT char const * pod5_get_error_string();\n\n//---------------------------------------------------------------------------------------------------------------------\n// Global state\n//---------------------------------------------------------------------------------------------------------------------\n\n/// \\brief Initialise and register global pod5 types\nPOD5_FORMAT_EXPORT pod5_error_t pod5_init();\n/// \\brief Terminate global pod5 types\nPOD5_FORMAT_EXPORT pod5_error_t pod5_terminate();\n\n//---------------------------------------------------------------------------------------------------------------------\n// Shared Structures\n//---------------------------------------------------------------------------------------------------------------------\n\nenum pod5_end_reason {\n POD5_END_REASON_UNKNOWN = 0,\n POD5_END_REASON_MUX_CHANGE = 1,\n POD5_END_REASON_UNBLOCK_MUX_CHANGE = 2,\n POD5_END_REASON_DATA_SERVICE_UNBLOCK_MUX_CHANGE = 3,\n POD5_END_REASON_SIGNAL_POSITIVE = 4,\n POD5_END_REASON_SIGNAL_NEGATIVE = 5,\n POD5_END_REASON_API_REQUEST = 6,\n POD5_END_REASON_DEVICE_DATA_ERROR = 7,\n POD5_END_REASON_ANALYSIS_CONFIG_CHANGE = 8,\n POD5_END_REASON_PAUSED = 9\n};\ntypedef enum pod5_end_reason pod5_end_reason_t;\n\ntypedef uint16_t run_info_index_t;\n\ntypedef uint8_t read_id_t[16];\ntypedef uint8_t run_id_t[16];\n\n// Single entry of read data:\nstruct ReadBatchRowInfoV3 {\n // The read id data, in binary form.\n read_id_t read_id;\n\n // Read number for the read.\n uint32_t read_number;\n // Start sample for the read.\n uint64_t start_sample;\n // Median before level.\n float median_before;\n\n // Channel for the read.\n uint16_t channel;\n // Well for the read.\n uint8_t well;\n // Dictionary index for the pore type.\n int16_t pore_type;\n // Calibration offset type for the read.\n float calibration_offset;\n // Palibration type for the read.\n float calibration_scale;\n // End reason index for the read.\n int16_t end_reason;\n // Was the end reason for the read forced (0 for false, 1 for true).\n uint8_t end_reason_forced;\n // Dictionary index for run id for the read, can be used to look up run info.\n int16_t run_info;\n\n // Number of minknow events that the read contains\n uint64_t num_minknow_events;\n\n // Scale/Shift for tracked read scaling values (based on previous reads)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float tracked_scaling_scale;\n POD5_DEPRECATED float tracked_scaling_shift;\n\n // Scale/Shift for predicted read scaling values (based on this read's raw signal)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float predicted_scaling_scale;\n POD5_DEPRECATED float predicted_scaling_shift;\n\n // How many reads have been selected prior to this read on the channel-well since it was made active.\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED uint32_t num_reads_since_mux_change;\n // How many seconds have passed since the channel-well was made active\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float time_since_mux_change;\n\n // Number of signal row entries for the read.\n int64_t signal_row_count;\n\n // The length of the read in samples.\n uint64_t num_samples;\n};\n\n// Single entry of read data:\nstruct ReadBatchRowInfoV4 {\n // The read id data, in binary form.\n read_id_t read_id;\n\n // Read number for the read.\n uint32_t read_number;\n // Start sample for the read.\n uint64_t start_sample;\n // Median before level.\n float median_before;\n\n // Channel for the read.\n uint16_t channel;\n // Well for the read.\n uint8_t well;\n // Dictionary index for the pore type.\n int16_t pore_type;\n // Calibration offset type for the read.\n float calibration_offset;\n // Palibration type for the read.\n float calibration_scale;\n // End reason index for the read.\n int16_t end_reason;\n // Was the end reason for the read forced (0 for false, 1 for true).\n uint8_t end_reason_forced;\n // Dictionary index for run id for the read, can be used to look up run info.\n int16_t run_info;\n\n // Number of minknow events that the read contains\n uint64_t num_minknow_events;\n\n // Scale/Shift for tracked read scaling values (based on previous reads)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float tracked_scaling_scale;\n POD5_DEPRECATED float tracked_scaling_shift;\n\n // Scale/Shift for predicted read scaling values (based on this read's raw signal)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float predicted_scaling_scale;\n POD5_DEPRECATED float predicted_scaling_shift;\n\n // How many reads have been selected prior to this read on the channel-well since it was made active.\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED uint32_t num_reads_since_mux_change;\n // How many seconds have passed since the channel-well was made active\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float time_since_mux_change;\n\n // Number of signal row entries for the read.\n int64_t signal_row_count;\n\n // The length of the read in samples.\n uint64_t num_samples;\n\n // The level of the pore.\n float open_pore_level;\n};\n\n// Typedef for latest batch row info structure.\ntypedef struct ReadBatchRowInfoV4 ReadBatchRowInfo_t;\n\nstruct POD5_DEPRECATED ReadBatchRowInfoArrayV3 {\n // The read id data, in binary form.\n read_id_t const * read_id;\n\n // Read number for the read.\n uint32_t const * read_number;\n // Start sample for the read.\n uint64_t const * start_sample;\n // Median before level.\n float const * median_before;\n\n // Channel for the read.\n uint16_t const * channel;\n // Well for the read.\n uint8_t const * well;\n // Pore type for the read.\n int16_t const * pore_type;\n // Calibration offset type for the read.\n float const * calibration_offset;\n // Palibration type for the read.\n float const * calibration_scale;\n // End reason type for the read.\n pod5_end_reason_t const * end_reason;\n // Was the end reason for the read forced (0 for false, 1 for true).\n uint8_t const * end_reason_forced;\n // Run info type for the read.\n int16_t const * run_info_id;\n\n // Number of minknow events that the read contains\n uint64_t const * num_minknow_events;\n\n // Scale/Shift for tracked read scaling values (based on previous reads)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * tracked_scaling_scale;\n POD5_DEPRECATED float const * tracked_scaling_shift;\n\n // Scale/Shift for predicted read scaling values (based on this read's raw signal)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * predicted_scaling_scale;\n POD5_DEPRECATED float const * predicted_scaling_shift;\n\n // How many reads have been selected prior to this read on the channel-well since it was made active.\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED uint32_t const * num_reads_since_mux_change;\n // How many seconds have passed since the channel-well was made active\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * time_since_mux_change;\n};\n\n// Array of read data:\nstruct ReadBatchRowInfoArrayV4 {\n // The read id data, in binary form.\n read_id_t const * read_id;\n\n // Read number for the read.\n uint32_t const * read_number;\n // Start sample for the read.\n uint64_t const * start_sample;\n // Median before level.\n float const * median_before;\n\n // Channel for the read.\n uint16_t const * channel;\n // Well for the read.\n uint8_t const * well;\n // Pore type for the read.\n int16_t const * pore_type;\n // Calibration offset type for the read.\n float const * calibration_offset;\n // Palibration type for the read.\n float const * calibration_scale;\n // End reason type for the read.\n pod5_end_reason_t const * end_reason;\n // Was the end reason for the read forced (0 for false, 1 for true).\n uint8_t const * end_reason_forced;\n // Run info type for the read.\n int16_t const * run_info_id;\n\n // Number of minknow events that the read contains\n uint64_t const * num_minknow_events;\n\n // Scale/Shift for tracked read scaling values (based on previous reads)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * tracked_scaling_scale;\n POD5_DEPRECATED float const * tracked_scaling_shift;\n\n // Scale/Shift for predicted read scaling values (based on this read's raw signal)\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * predicted_scaling_scale;\n POD5_DEPRECATED float const * predicted_scaling_shift;\n\n // How many reads have been selected prior to this read on the channel-well since it was made active.\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED uint32_t const * num_reads_since_mux_change;\n // How many seconds have passed since the channel-well was made active\n // DEPRECATED: will be removed in 0.4.0\n POD5_DEPRECATED float const * time_since_mux_change;\n\n // The level of the pore.\n float const * open_pore_level;\n};\n\n// Typedef for latest batch row info structure.\ntypedef struct ReadBatchRowInfoArrayV4 ReadBatchRowInfoArray_t;\n\n#define READ_BATCH_ROW_INFO_VERSION_0 0\n// Addition of num_minknow_events fields, scaling fields.\n#define READ_BATCH_ROW_INFO_VERSION_1 1\n// Addition of num_samples fields.\n#define READ_BATCH_ROW_INFO_VERSION_2 2\n// Flattening of read structures.\n#define READ_BATCH_ROW_INFO_VERSION_3 3\n// Introduction of new open_pore_level field.\n#define READ_BATCH_ROW_INFO_VERSION_4 4\n// Latest available version.\n#define READ_BATCH_ROW_INFO_VERSION READ_BATCH_ROW_INFO_VERSION_4\n\n//---------------------------------------------------------------------------------------------------------------------\n// Reading files\n//---------------------------------------------------------------------------------------------------------------------\n\n// Options to control how a file is written.\nstruct Pod5ReaderOptions {\n /// \\brief Disable file mapping into memory. Reduces memory usage of pod5 files, at the expense\n /// of the underlying arrow file loading into memory on demand.\n char force_disable_file_mapping;\n};\ntypedef struct Pod5ReaderOptions Pod5ReaderOptions_t;\n\n/// \\brief Open a file reader with default options.\n/// \\param filename The filename of the pod5 file.\n/// \\see pod5_open_file_options\nPOD5_FORMAT_EXPORT Pod5FileReader_t * pod5_open_file(char const * filename);\n\n/// \\brief Open a file reader\n/// \\param filename The filename of the pod5 file.\n/// \\param options The options to use when opening the file.\n/// \\return A reader, or null on error. The reason for the error can be queried with\n/// pod5_get_error_no() and pod5_get_error_string().\nPOD5_FORMAT_EXPORT Pod5FileReader_t * pod5_open_file_options(\n char const * filename,\n Pod5ReaderOptions_t const * options);\n\n/// \\brief Close a file reader, releasing all memory held by the reader.\n/// \\param file A previously opened reader.\n/// \\note Any references to \\a file or its components are no longer valid after this call.\n/// \\note It is safe to call this with a null \\a file.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_close_and_free_reader(Pod5FileReader_t * file);\n\nstruct FileInfo {\n read_id_t file_identifier;\n\n struct Version {\n uint16_t major;\n uint16_t minor;\n uint16_t revision;\n } version;\n};\ntypedef struct FileInfo FileInfo_t;\n\n/// \\brief Find info about a file.\n/// \\param[in] file The file to be queried.\n/// \\param[out] file_info The info read from the file.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_file_info(Pod5FileReader_t const * file, FileInfo_t * file_info);\n\nstruct EmbeddedFileData {\n // The embedded file name - note this may not be the original file name, if the file has been migrated.\n // This pointer will remain valid until the next pod5 api call on the associated reader.\n char const * file_name;\n size_t offset;\n size_t length;\n};\ntypedef struct EmbeddedFileData EmbeddedFileData_t;\n\n/// \\brief Find the location of the read table data\n/// \\param[in] file The file to be queried.\n/// \\param[out] file_data The output read table file data.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_file_read_table_location(Pod5FileReader_t const * file, EmbeddedFileData_t * file_data);\n\n/// \\brief Find the location of the signal table data\n/// \\param[in] file The file to be queried.\n/// \\param[out] file_data The output signal table file data.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_file_signal_table_location(Pod5FileReader_t const * file, EmbeddedFileData_t * file_data);\n\n/// \\brief Find the location of the run info table data\n/// \\param[in] file The file to be queried.\n/// \\param[out] file_data The output signal table file data.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_file_run_info_table_location(\n Pod5FileReader_t const * file,\n EmbeddedFileData_t * file_data);\n\n/// \\brief Find the number of reads in the file.\n/// \\param[in] reader The file reader to read from\n/// \\param[out] count The number of reads in the file\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_read_count(Pod5FileReader_t const * reader, size_t * count);\n\n/// \\brief Grab the read_id's from the file.\n/// \\param[in] reader The file reader to read from.\n/// \\param count The number of read_id's allocated in [read_ids], an error is raised if the count is not greater or equal to pod5_get_read_count.\n/// \\param[out] read_ids The read id's written in a contiguous array.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_read_ids(Pod5FileReader_t const * reader, size_t count, read_id_t * read_ids);\n\n/// \\brief Plan the most efficient route through the data for the given read ids\n/// \\param[in] file The file to be queried.\n/// \\param[in] read_id_array The read id array (contiguous array, 16 bytes per id).\n/// \\param read_id_count The number of read ids.\n/// \\param[out] batch_counts The number of rows per batch that need to be visited (rows listed in batch_rows),\n/// input array length should be the number of read table batches.\n/// \\param[out] batch_rows Rows to visit per batch, packed into one array. Offsets into this array from\n/// [batch_counts] provide the per-batch row data. Input array length should\n/// equal read_id_count.\n/// \\param[out] find_success_count The number of requested read ids that were found.\n/// \\note The output arrays are sorted in file storage order, to improve read efficiency.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_plan_traversal(\n Pod5FileReader_t const * file,\n uint8_t const * read_id_array,\n size_t read_id_count,\n uint32_t * batch_counts,\n uint32_t * batch_rows,\n size_t * find_success_count);\n\n/// \\brief Find the number of read batches in the file.\n/// \\param[out] count The number of read batches in the file\n/// \\param[in] reader The file reader to read from\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_read_batch_count(size_t * count, Pod5FileReader_t const * reader);\n\n/// \\brief Get a read batch from the file.\n/// \\param[out] batch The extracted batch.\n/// \\param[in] reader The file reader to read from\n/// \\param index The index of the batch to read.\n/// \\note Batches returned from this API must be freed using #pod5_free_read_batch\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_read_batch(Pod5ReadRecordBatch_t ** batch, Pod5FileReader_t const * reader, size_t index);\n\n/// \\brief Release a read batch when it is not longer used.\n/// \\param batch The batch to release.\n/// \\note Any references to \\a batch or its components are no longer valid after this call.\n/// \\note It is safe to call this with a null \\a batch.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_free_read_batch(Pod5ReadRecordBatch_t * batch);\n\n/// \\brief Find the number of rows in a batch.\n/// \\param[out] count The number of rows in the batch.\n/// \\param[in] batch The batch to query the number of rows for.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_read_batch_row_count(size_t * count, Pod5ReadRecordBatch_t const * batch);\n\n/// \\brief Find the info for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param row The row index to query.\n/// \\param struct_version The version of the struct being passed in, calling code\n/// should use [READ_BATCH_ROW_INFO_VERSION].\n/// \\param[out] row_data The data for reading into, should be a pointer to ReadBatchRowInfo_t.\n/// \\param[out] read_table_version The table version read from the file, will indicate which fields should be available.\n/// See READ_BATCH_ROW_INFO_VERSION and ReadBatchRowInfo_t above for corresponding fields.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_read_batch_row_info_data(\n Pod5ReadRecordBatch_t const * batch,\n size_t row,\n uint16_t struct_version,\n void * row_data,\n uint16_t * read_table_version);\n\n/// \\brief Find the signal indices for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param row The row index to query.\n/// \\param signal_row_indices_count Number of entries in the signal_row_indices array.\n/// \\param[out] signal_row_indices The signal row indices read out of the read row.\n/// \\note signal_row_indices_count Must equal signal_row_count returned from pod5_get_read_batch_row_info_data\n/// or an error is generated.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_signal_row_indices(\n Pod5ReadRecordBatch_t const * batch,\n size_t row,\n int64_t signal_row_indices_count,\n uint64_t * signal_row_indices);\n\nstruct CalibrationExtraData {\n // The digitisation value used by the sequencer, equal to:\n //\n // adc_max - adc_min + 1\n uint16_t digitisation;\n // The range of the calibrated channel in pA.\n float range;\n};\ntypedef struct CalibrationExtraData CalibrationExtraData_t;\n\n/// \\brief Find the extra calibration info for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param row The read row index.\n/// \\param[out] calibration_extra_data Output location for the calibration data.\n/// \\note The values are computed from data held in the file, and written directly to the address provided, there is no need to release any data.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_calibration_extra_info(\n Pod5ReadRecordBatch_t const * batch,\n size_t row,\n CalibrationExtraData_t * calibration_extra_data);\n\nstruct KeyValueData {\n size_t size;\n char const ** keys;\n char const ** values;\n};\n\nstruct RunInfoDictData {\n char const * acquisition_id;\n int64_t acquisition_start_time_ms;\n int16_t adc_max;\n int16_t adc_min;\n struct KeyValueData context_tags;\n char const * experiment_name;\n char const * flow_cell_id;\n char const * flow_cell_product_code;\n char const * protocol_name;\n char const * protocol_run_id;\n int64_t protocol_start_time_ms;\n char const * sample_id;\n uint16_t sample_rate;\n char const * sequencing_kit;\n char const * sequencer_position;\n char const * sequencer_position_type;\n char const * software;\n char const * system_name;\n char const * system_type;\n struct KeyValueData tracking_id;\n};\ntypedef struct RunInfoDictData RunInfoDictData_t;\n\n/// \\brief Find the run info for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param run_info The run info index to query from the passed batch.\n/// \\param[out] run_info_data Output location for the run info data.\n/// \\note The returned run_info value should be released using pod5_free_run_info when it is no longer used.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_run_info(\n Pod5ReadRecordBatch_t const * batch,\n int16_t run_info,\n RunInfoDictData_t ** run_info_data);\n\n/// \\brief Find the run info for a row in a file.\n/// \\param[in] file The file to query.\n/// \\param run_info_index The run info index to query from the passed file.\n/// \\param[out] run_info_data Output location for the run info data.\n/// \\note The returned run_info value should be released using pod5_free_run_info when it is no longer used.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_file_run_info(\n Pod5FileReader_t const * file,\n run_info_index_t run_info_index,\n RunInfoDictData_t ** run_info_data);\n\n/// \\brief Release a RunInfoDictData struct after use.\n/// \\param run_info_data The run info to release.\n/// \\note Any references to \\a run_info_data or its components are no longer valid after this call.\n/// \\note It is safe to call this with a null \\a run_info_data.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_free_run_info(RunInfoDictData_t * run_info_data);\n\n/// \\brief Release a RunInfoDictData struct after use.\n/// \\deprecated\nPOD5_FORMAT_EXPORT POD5_DEPRECATED pod5_error_t\npod5_release_run_info(RunInfoDictData_t * run_info_data);\n\n/// \\brief Find the run info for a row in a read file.\n/// \\param[in] file The file to query.\n/// \\param[out] run_info_count The number of run info's that are present in they queried file\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_get_file_run_info_count(Pod5FileReader_t const * file, run_info_index_t * run_info_count);\n\n/// \\brief Find the end reason for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param end_reason The end reason index to query from the passed batch.\n/// \\param[out] end_reason_value The enum value for end reason.\n/// \\param[out] end_reason_string_value Output location for the string value for the end reason.\n/// \\param[in,out] end_reason_string_value_size Size of [end_reason_string_value], the number of characters written (including 1 for null character) is placed in this value on return.\n/// \\note If the string input is not long enough POD5_ERROR_STRING_NOT_LONG_ENOUGH is returned.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_end_reason(\n Pod5ReadRecordBatch_t const * batch,\n int16_t end_reason,\n pod5_end_reason_t * end_reason_value,\n char * end_reason_string_value,\n size_t * end_reason_string_value_size);\n\n/// \\brief Find the pore type for a row in a read batch.\n/// \\param[in] batch The read batch to query.\n/// \\param pore_type The pore type index to query from the passed batch.\n/// \\param[out] pore_type_string_value Output location for the string value for the pore type.\n/// \\param[in,out] pore_type_string_value_size Size of [pore_type_string_value], the number of characters written (including 1 for null character) is placed in this value on return.\n/// \\note If the string input is not long enough POD5_ERROR_STRING_NOT_LONG_ENOUGH is returned.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_pore_type(\n Pod5ReadRecordBatch_t const * batch,\n int16_t pore_type,\n char * pore_type_string_value,\n size_t * pore_type_string_value_size);\n\nstruct SignalRowInfo {\n size_t batch_index;\n size_t batch_row_index;\n uint32_t stored_sample_count;\n size_t stored_byte_count;\n};\ntypedef struct SignalRowInfo SignalRowInfo_t;\n\n/// \\brief Find the info for a signal row in a reader.\n/// \\param[in] reader The reader to query.\n/// \\param signal_rows_count The number of signal rows to query.\n/// \\param[in] signal_rows The signal rows to query.\n/// \\param[out] signal_row_info Pointers to the output signal row information (must be an array of size signal_rows_count)\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_signal_row_info(\n Pod5FileReader_t const * reader,\n size_t signal_rows_count,\n uint64_t const * signal_rows,\n SignalRowInfo_t ** signal_row_info);\n\n/// \\brief Release a list of signal row infos allocated by [pod5_get_signal_row_info].\n/// \\param signal_rows_count The number of signal rows to release.\n/// \\param signal_row_info The signal row infos to release.\n/// \\note Calls to pod5_free_signal_row_info must be 1:1 with [pod5_get_signal_row_info], you cannot free part of the returned data.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_free_signal_row_info(size_t signal_rows_count, SignalRowInfo_t ** signal_row_info);\n\n/// \\brief Find the info for a signal row in a reader.\n/// \\param[in] reader The reader to query.\n/// \\param[in] row_info The signal row info batch index to query data for.\n/// \\param sample_count The number of samples allocated in [sample_data] (must equal the length of signal data in the row).\n/// \\param[out] sample_data The output location for the queried samples.\n/// \\note The signal data is allocated by the caller and should be released as appropriate by the caller.\n/// \\todo MAJOR_VERSION Rename to include \"chunk\" or \"row\" or similar to indicate this gets only part of read signal.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_signal(\n Pod5FileReader_t const * reader,\n SignalRowInfo_t const * row_info,\n size_t sample_count,\n int16_t * sample_data);\n\n/// \\brief Find the sample count for a full read.\n/// \\param[in] reader The reader to query.\n/// \\param[in] batch The read batch to query.\n/// \\param batch_row The read row to query data for.\n/// \\param[out] sample_count The number of samples in the read - including all chunks of raw data.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_read_complete_sample_count(\n Pod5FileReader_t const * reader,\n Pod5ReadRecordBatch_t const * batch,\n size_t batch_row,\n size_t * sample_count);\n\n/// \\brief Find the signal for a full read.\n/// \\param[in] reader The reader to query.\n/// \\param[in] batch The read batch to query.\n/// \\param batch_row The read row to query data for.\n/// \\param sample_count The number of samples allocated in [signal] (must equal the length of signal data in the queryied read row).\n/// \\param[out] signal The output location for the queried samples.\n/// \\note The signal data is allocated by the caller and should be released as appropriate by the caller.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_get_read_complete_signal(\n Pod5FileReader_t const * reader,\n Pod5ReadRecordBatch_t const * batch,\n size_t batch_row,\n size_t sample_count,\n int16_t * signal);\n\n//---------------------------------------------------------------------------------------------------------------------\n// Writing files\n//---------------------------------------------------------------------------------------------------------------------\n\n// Signal compression options.\nenum CompressionOption {\n /// \\brief Use the default signal compression option.\n DEFAULT_SIGNAL_COMPRESSION = 0,\n /// \\brief Use vbz to compress read signals in tables.\n VBZ_SIGNAL_COMPRESSION = 1,\n /// \\brief Write signals uncompressed to tables.\n UNCOMPRESSED_SIGNAL = 2,\n};\n\n// Options to control how a file is written.\nstruct Pod5WriterOptions {\n /// \\brief Maximum number of samples to place in one signal record in the signals table.\n /// \\note Use zero to use default value.\n uint32_t max_signal_chunk_size;\n /// \\brief Signal type to write to the signals table.\n /// \\note Use 'DEFAULT_SIGNAL_COMPRESSION' to use default value.\n int8_t signal_compression_type;\n\n /// \\brief The size of each batch written for the signal table (zero for default).\n size_t signal_table_batch_size;\n /// \\brief The size of each batch written for the reads table (zero for default).\n size_t read_table_batch_size;\n};\ntypedef struct Pod5WriterOptions Pod5WriterOptions_t;\n\n/// \\brief Create a new pod5 file using specified filenames and options.\n/// \\param filename The filename of the pod5 file.\n/// \\param writer_name A descriptive string for the user software writing this file.\n/// \\param options Options controlling how the file will be written.\n/// \\return A writer, or null on error. The reason for the error can be queried with\n/// pod5_get_error_no() and pod5_get_error_string().\nPOD5_FORMAT_EXPORT Pod5FileWriter_t * pod5_create_file(\n char const * filename,\n char const * writer_name,\n Pod5WriterOptions_t const * options);\n\n/// \\brief Close a file writer, releasing all memory held by the writer.\n/// \\param file A previously opened writer.\n/// \\note Any references to \\a file or its components are no longer valid after this call.\n/// \\note It is safe to call this with a null \\a file.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_close_and_free_writer(Pod5FileWriter_t * file);\n\n/// \\brief Add a new pore type to the file.\n/// \\param[out] pore_index The index of the added pore.\n/// \\param file The file to add the new pore type to.\n/// \\param pore_type The pore type string for the pore.\nPOD5_FORMAT_EXPORT pod5_error_t\npod5_add_pore(int16_t * pore_index, Pod5FileWriter_t * file, char const * pore_type);\n\n/// \\brief Add a new run info to the file, containing tracking information about a sequencing run.\n/// \\param[out] run_info_index The index of the added run_info.\n/// \\param file The file to add the new pore type to.\n/// \\param acquisition_id The offset parameter for the calibration.\n/// \\param acquisition_start_time_ms Milliseconds after unix epoch when the acquisition was started.\n/// \\param adc_max Maximum ADC value supported by this hardware.\n/// \\param adc_min Minimum ADC value supported by this hardware.\n/// \\param context_tags_count Number of entries in the context tags map.\n/// \\param context_tags_keys Array of strings used as keys into the context tags map (must have context_tags_count entries).\n/// \\param context_tags_values Array of strings used as values in the context tags map (must have context_tags_count entries).\n/// \\param experiment_name Name given by the user to the group including this protocol.\n/// \\param flow_cell_id Id for the flow cell used in the run.\n/// \\param flow_cell_product_code Product code for the flow cell used in the run.\n/// \\param protocol_name Name given by the user to the protocol.\n/// \\param protocol_run_id Run id for the protocol.\n/// \\param protocol_start_time_ms Milliseconds after unix epoch when the protocol was started.\n/// \\param sample_id Name given by the user for sample id.\n/// \\param sample_rate Sample rate of the run.\n/// \\param sequencing_kit Sequencing kit used in the run.\n/// \\param sequencer_position Sequencer position used in the run.\n/// \\param sequencer_position_type Sequencer position type used in the run.\n/// \\param software Name of the software used to produce the run.\n/// \\param system_name Name of the system used to produce the run.\n/// \\param system_type Type of the system used to produce the run.\n/// \\param tracking_id_count Number of entries in the tracking id map.\n/// \\param tracking_id_keys Array of strings used as keys into the tracking id map (must have tracking_id_count entries).\n/// \\param tracking_id_values Array of strings used as values in the tracking id map (must have tracking_id_count entries).\nPOD5_FORMAT_EXPORT pod5_error_t pod5_add_run_info(\n int16_t * run_info_index,\n Pod5FileWriter_t * file,\n char const * acquisition_id,\n int64_t acquisition_start_time_ms,\n int16_t adc_max,\n int16_t adc_min,\n size_t context_tags_count,\n char const ** context_tags_keys,\n char const ** context_tags_values,\n char const * experiment_name,\n char const * flow_cell_id,\n char const * flow_cell_product_code,\n char const * protocol_name,\n char const * protocol_run_id,\n int64_t protocol_start_time_ms,\n char const * sample_id,\n uint16_t sample_rate,\n char const * sequencing_kit,\n char const * sequencer_position,\n char const * sequencer_position_type,\n char const * software,\n char const * system_name,\n char const * system_type,\n size_t tracking_id_count,\n char const ** tracking_id_keys,\n char const ** tracking_id_values);\n\n/// \\brief Add a read to the file.\n///\n/// For each read `r`, where `0 <= r < read_count`:\n/// - `row_data->field[r]` describes a field of the read metadata\n/// - `signal[r]` is the raw signal data for the read\n/// - `signal_size[r]` is the length of `signal[r]` (in samples, not in bytes)\n///\n/// \\param file The file to add the reads to.\n/// \\param read_count The number of reads to add with this call.\n/// \\param struct_version The version of the struct of [row_data] being filled, use READ_BATCH_ROW_INFO_VERSION.\n/// \\param row_data The array data for injecting into the file, should be ReadBatchRowInfoArray_t.\n/// All fields of the array must have length [read_count].\n/// \\param signal The signal data for the reads.\n/// \\param signal_size The number of samples in the signal data.\n/// This must be an array of length [read_count].\nPOD5_FORMAT_EXPORT pod5_error_t pod5_add_reads_data(\n Pod5FileWriter_t * file,\n uint32_t read_count,\n uint16_t struct_version,\n void const * row_data,\n int16_t const ** signal,\n uint32_t const * signal_size);\n\n/// \\brief Add a read to the file, with pre compressed signal chunk sections.\n///\n/// Consider using the simpler [pod5_add_reads_data] unless you have performance requirements that demand\n/// more control over compression and chunking.\n///\n/// Data should be compressed using [pod5_vbz_compress_signal].\n///\n/// For each read `r`, where `0 <= r < read_count`:\n/// - `row_data->field[r]` describes a field of the read metadata\n/// - `signal_chunk_count[r]` is the number of signal chunks\n/// - for each signal chunk `i` where `0 <= i < signal_chunk_count[r]`:\n/// - `sample_counts[r][i]` is the number of samples in the chunk (ie: the size of the uncompressed data in\n/// samples, not in bytes)\n/// - `compressed_signal[r][i]` is the compressed data\n/// - `compressed_signal_size[r][i]` is the length of the compressed data at `compressed_signal[r][i]`\n///\n/// \\param file The file to add the read to.\n/// \\param read_count The number of reads to add with this call.\n/// \\param struct_version The version of the struct of [row_data] being filled, use READ_BATCH_ROW_INFO_VERSION.\n/// \\param row_data The array data for injecting into the file, should be ReadBatchRowInfoArray_t.\n/// All fields of the array must have length [read_count].\n/// \\param compressed_signal The signal chunks data for the read.\n/// \\param compressed_signal_size The sizes (in bytes) of each signal chunk.\n/// \\param sample_counts The number of samples of each signal chunk. In other words, it is the *uncompressed* size of the\n/// corresponding [compressed_signal] array, in samples (not bytes!).\n/// \\param signal_chunk_count The number of sections of compressed signal.\n/// This must be an array of length [read_count].\nPOD5_FORMAT_EXPORT pod5_error_t pod5_add_reads_data_pre_compressed(\n Pod5FileWriter_t * file,\n uint32_t read_count,\n uint16_t struct_version,\n void const * row_data,\n char const *** compressed_signal,\n size_t const ** compressed_signal_size,\n uint32_t const ** sample_counts,\n size_t const * signal_chunk_count);\n\n/// \\brief Find the max size of a compressed array of samples.\n/// \\param sample_count The number of samples in the source signal.\n/// \\return The max number of bytes required for the compressed signal, or 0 on error.\n/// The reason for the error can be queried with pod5_get_error_no() and\n/// pod5_get_error_string().\nPOD5_FORMAT_EXPORT size_t pod5_vbz_compressed_signal_max_size(size_t sample_count);\n\n/// \\brief VBZ compress an array of samples.\n/// \\param signal The signal to compress.\n/// \\param signal_size The number of samples to compress.\n/// \\param[out] compressed_signal_out The compressed signal.\n/// \\param[in,out] compressed_signal_size The number of compressed bytes, should be set to the size of compressed_signal_out on call.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_vbz_compress_signal(\n int16_t const * signal,\n size_t signal_size,\n char * compressed_signal_out,\n size_t * compressed_signal_size);\n\n/// \\brief VBZ decompress an array of samples.\n/// \\param compressed_signal The signal to decompress.\n/// \\param compressed_signal_size The number of compressed bytes, ie the size of compressed_signal in bytes.\n/// \\param sample_count The number of samples to decompress, ie the size of signal_out in samples.\n/// \\param[out] signal_out The decompressed signal.\nPOD5_FORMAT_EXPORT pod5_error_t pod5_vbz_decompress_signal(\n char const * compressed_signal,\n size_t compressed_signal_size,\n size_t sample_count,\n int16_t * signal_out);\n\n//---------------------------------------------------------------------------------------------------------------------\n// Global state\n//---------------------------------------------------------------------------------------------------------------------\n\n/// \\brief Format a packed binary read id as a readable read id string:\n/// \\param read_id A 16 byte binary formatted UUID.\n/// \\param[out] read_id_string Output string containing the string formatted UUID (expects a string of at least 37 bytes, one null byte is written.)\nPOD5_FORMAT_EXPORT pod5_error_t pod5_format_read_id(read_id_t const read_id, char * read_id_string);\n\n#ifdef __cplusplus\n}\n#endif\n" }, { "path": "c++/pod5_format/dictionary_writer.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n\nnamespace arrow {\nclass Array;\n}\n\nnamespace pod5 {\n\nclass POD5_FORMAT_EXPORT DictionaryWriter {\npublic:\n virtual ~DictionaryWriter() = default;\n\n pod5::Result> build_dictionary_array(\n std::shared_ptr const & indices);\n virtual pod5::Result> get_value_array() = 0;\n virtual std::size_t item_count() = 0;\n\n bool is_valid(std::size_t value) { return value < item_count(); }\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/expandable_buffer.h", "content": "#pragma once\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\ntemplate \nclass ExpandableBuffer {\npublic:\n static constexpr int EXPANSION_FACTOR = 2;\n\n ExpandableBuffer(arrow::MemoryPool * pool = nullptr) { m_pool = pool; }\n\n arrow::Status init_buffer(arrow::MemoryPool * pool)\n {\n m_pool = pool;\n return clear();\n }\n\n std::size_t size() const\n {\n if (!m_buffer) {\n return 0;\n }\n return m_buffer->size() / sizeof(T);\n }\n\n std::uint8_t * mutable_data() { return m_buffer->mutable_data(); }\n\n std::shared_ptr get_buffer() const { return m_buffer; }\n\n arrow::Status clear()\n {\n if (!m_buffer || m_buffer.use_count() > 1) {\n ARROW_ASSIGN_OR_RAISE(m_buffer, arrow::AllocateResizableBuffer(0, m_pool));\n return arrow::Status::OK();\n } else {\n return m_buffer->Resize(0, false);\n }\n }\n\n gsl::span get_data_span() const\n {\n if (!m_buffer) {\n return {};\n }\n\n return gsl::make_span(m_buffer->data(), m_buffer->size()).template as_span();\n }\n\n /// \\brief Append an object where you don't know the size up front.\n /// \\param max_size The maximum possible size of the object to append.\n /// \\param append_fn A function that appends the object to the buffer.\n template \n arrow::Status append(std::size_t max_size, Callable append_fn)\n {\n auto const old_size = m_buffer->size();\n ARROW_RETURN_NOT_OK(reserve(old_size + max_size));\n auto const potential_buffer = gsl::make_span(m_buffer->mutable_data() + old_size, max_size);\n ARROW_ASSIGN_OR_RAISE(auto final_size, append_fn(potential_buffer));\n assert(final_size < max_size);\n return resize(old_size + final_size);\n }\n\n arrow::Status append(T const & new_value)\n {\n auto const bytes_span =\n gsl::make_span(&new_value, 1).template as_span();\n\n return append_bytes(bytes_span);\n }\n\n arrow::Status append_array(gsl::span const & new_value_span)\n {\n auto const bytes_span = new_value_span.template as_span();\n\n return append_bytes(bytes_span);\n }\n\n arrow::Status resize(std::int64_t new_size)\n {\n ARROW_RETURN_NOT_OK(reserve(new_size));\n return m_buffer->Resize(new_size, false);\n }\n\n arrow::Status reserve(std::int64_t new_capacity)\n {\n assert(m_buffer);\n auto const old_size = m_buffer->size();\n if (m_buffer.use_count() > 1) {\n std::shared_ptr buffer;\n ARROW_ASSIGN_OR_RAISE(buffer, arrow::AllocateResizableBuffer(old_size, m_pool));\n\n std::copy(m_buffer->data(), m_buffer->data() + old_size, buffer->mutable_data());\n std::swap(m_buffer, buffer);\n }\n\n if (new_capacity > m_buffer->capacity()) {\n ARROW_RETURN_NOT_OK(m_buffer->Reserve(new_capacity * EXPANSION_FACTOR));\n }\n return arrow::Status::OK();\n }\n\nprivate:\n arrow::Status append_bytes(gsl::span const & bytes_span)\n {\n auto old_size = 0;\n if (!m_buffer) {\n ARROW_ASSIGN_OR_RAISE(\n m_buffer, arrow::AllocateResizableBuffer(bytes_span.size(), m_pool));\n } else {\n old_size = m_buffer->size();\n }\n auto const new_size = old_size + bytes_span.size();\n ARROW_RETURN_NOT_OK(reserve(new_size));\n\n ARROW_RETURN_NOT_OK(m_buffer->Resize(new_size, false));\n std::copy(bytes_span.begin(), bytes_span.end(), m_buffer->mutable_data() + old_size);\n return arrow::Status::OK();\n }\n\n std::shared_ptr m_buffer;\n arrow::MemoryPool * m_pool = nullptr;\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_output_stream.h", "content": "#pragma once\n\n#include \n#include \n\nnamespace pod5 {\n\nclass FileOutputStream : public arrow::io::OutputStream {\npublic:\n virtual arrow::Status batch_complete() { return arrow::Status::OK(); }\n\n virtual void set_file_start_offset(std::size_t val) {}\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_reader.cpp", "content": "#include \"pod5_format/file_reader.h\"\n\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/memory_pool.h\"\n#include \"pod5_format/migration/migration.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/run_info_table_reader.h\"\n#include \"pod5_format/signal_table_reader.h\"\n\n#include \n#include \n#include \n\n#include \n#include \n#if defined(__APPLE__) || defined(__linux__) || defined(__unix__)\n#include \n#endif\n\nnamespace pod5 {\n\nnamespace {\n#if defined(__APPLE__) || defined(__linux__) || defined(__unix__)\nconstexpr std::uint64_t kStatBlockBytes = S_BLKSIZE;\n// Issue warning if the majority of the file is missing to avoid false positives\nconstexpr double kMinMissingFractionThreshold = 0.8;\n\nvoid warn_if_stat_size_and_blocks_differ_significantly(std::string const & path)\n{\n struct stat file_stat = {};\n if (::stat(path.c_str(), &file_stat) != 0 || file_stat.st_size <= 0 || file_stat.st_blocks < 0)\n {\n return;\n }\n\n auto const logical_size_bytes = static_cast(file_stat.st_size);\n auto const allocated_size_bytes =\n static_cast(file_stat.st_blocks) * kStatBlockBytes;\n\n if (allocated_size_bytes >= logical_size_bytes) {\n return;\n }\n\n auto const missing_bytes = logical_size_bytes - allocated_size_bytes;\n auto const missing_fraction =\n static_cast(missing_bytes) / static_cast(logical_size_bytes);\n if (missing_fraction < kMinMissingFractionThreshold) {\n return;\n }\n\n std::cerr << \"Warning: POD5 file '\" << path << \"' has st_size=\" << logical_size_bytes\n << \" bytes but only approximately \" << allocated_size_bytes\n << \" bytes allocated via st_blocks. The file may be sparse or offloaded and \"\n \"open/read operations may fail.\"\n << std::endl;\n}\n#else\nvoid warn_if_stat_size_and_blocks_differ_significantly(std::string const &) {}\n#endif\n} // namespace\n\nFileReaderOptions::FileReaderOptions()\n: m_memory_pool(pod5::default_memory_pool())\n, m_max_cached_signal_table_batches(DEFAULT_MAX_CACHED_SIGNAL_TABLE_BATCHES)\n{\n}\n\nvoid FileReaderOptions::set_max_cached_signal_table_batches(\n std::size_t max_cached_signal_table_batches)\n{\n m_max_cached_signal_table_batches = max_cached_signal_table_batches;\n}\n\ninline FileLocation make_file_locaton(combined_file_utils::ParsedFileInfo const & parsed_file_info)\n{\n return FileLocation{\n parsed_file_info.file_path,\n std::size_t(parsed_file_info.file_start_offset),\n std::size_t(parsed_file_info.file_length)};\n}\n\nclass FileReaderImpl : public FileReader {\npublic:\n FileReaderImpl(\n Version const & file_version_pre_migration,\n MigrationResult && migration_result,\n RunInfoTableReader && run_info_table_reader,\n ReadTableReader && read_table_reader,\n SignalTableReader && signal_table_reader)\n : m_file_version_pre_migration(file_version_pre_migration)\n , m_migration_result(std::move(migration_result))\n , m_run_info_table_location(make_file_locaton(m_migration_result.footer().run_info_table))\n , m_read_table_location(make_file_locaton(m_migration_result.footer().reads_table))\n , m_signal_table_location(make_file_locaton(m_migration_result.footer().signal_table))\n , m_run_info_table_reader(std::move(run_info_table_reader))\n , m_read_table_reader(std::move(read_table_reader))\n , m_signal_table_reader(std::move(signal_table_reader))\n {\n }\n\n SchemaMetadataDescription schema_metadata() const override\n {\n return m_read_table_reader.schema_metadata();\n }\n\n Result run_info_count() const override\n {\n return m_run_info_table_reader.CountRows();\n }\n\n virtual Result read_count() const override\n {\n auto const batch_count = num_read_record_batches();\n if (batch_count == 0) {\n return 0;\n }\n\n ARROW_ASSIGN_OR_RAISE(auto const first_batch, read_read_record_batch(0));\n ARROW_ASSIGN_OR_RAISE(auto const last_batch, read_read_record_batch(batch_count - 1));\n\n return (batch_count - 1) * first_batch.num_rows() + last_batch.num_rows();\n }\n\n Result read_read_record_batch(std::size_t i) const override\n {\n return m_read_table_reader.read_record_batch(i);\n }\n\n std::size_t num_read_record_batches() const override\n {\n return m_read_table_reader.num_record_batches();\n }\n\n Result search_for_read_ids(\n ReadIdSearchInput const & search_input,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows) const override\n {\n return m_read_table_reader.search_for_read_ids(search_input, batch_counts, batch_rows);\n }\n\n Result read_signal_record_batch(std::size_t i) const override\n {\n return m_signal_table_reader.read_record_batch(i);\n }\n\n std::size_t num_signal_record_batches() const override\n {\n return m_signal_table_reader.num_record_batches();\n }\n\n Result signal_batch_for_row_id(std::size_t row, std::size_t * batch_row)\n const override\n {\n return m_signal_table_reader.signal_batch_for_row_id(row, batch_row);\n }\n\n Result extract_sample_count(\n gsl::span const & row_indices) const override\n {\n return m_signal_table_reader.extract_sample_count(row_indices);\n }\n\n Status extract_samples(\n gsl::span const & row_indices,\n gsl::span const & output_samples) const override\n {\n return m_signal_table_reader.extract_samples(row_indices, output_samples);\n }\n\n Result>> extract_samples_inplace(\n gsl::span const & row_indices,\n std::vector & sample_count) const override\n {\n return m_signal_table_reader.extract_samples_inplace(row_indices, sample_count);\n }\n\n FileLocation const & run_info_table_location() const override\n {\n return m_run_info_table_location;\n }\n\n FileLocation const & read_table_location() const override { return m_read_table_location; }\n\n FileLocation const & signal_table_location() const override { return m_signal_table_location; }\n\n Version file_version_pre_migration() const override { return m_file_version_pre_migration; }\n\n SignalType signal_type() const override { return m_signal_table_reader.signal_type(); }\n\n Result> find_run_info(\n std::string const & acquisition_id) const override\n {\n return m_run_info_table_reader.find_run_info(acquisition_id);\n }\n\n Result> get_run_info(std::size_t index) const override\n {\n return m_run_info_table_reader.get_run_info(index);\n }\n\n Result get_run_info_count() const override\n {\n return m_run_info_table_reader.get_run_info_count();\n }\n\nprivate:\n Version m_file_version_pre_migration;\n MigrationResult m_migration_result;\n FileLocation m_run_info_table_location;\n FileLocation m_read_table_location;\n FileLocation m_signal_table_location;\n RunInfoTableReader m_run_info_table_reader;\n ReadTableReader m_read_table_reader;\n SignalTableReader m_signal_table_reader;\n};\n\npod5::Result> open_file_reader(\n std::string const & path,\n FileReaderOptions const & options)\n{\n auto pool = options.memory_pool();\n if (!pool) {\n return Status::Invalid(\"Invalid memory pool specified for file writer\");\n }\n\n // Issue warning if the file appears to be a stub\n warn_if_stat_size_and_blocks_differ_significantly(path);\n\n // \"Preflight\" file reads are done via standard file I/O first to prevent SIGBUS errors\n // if the file is not resident (e.g. stub remains when file is archived).\n // If mmap succeeds afterwards, use it for normal table reads otherwise continue\n // using this preflight file handle.\n ARROW_ASSIGN_OR_RAISE(auto preflight_file, arrow::io::ReadableFile::Open(path, pool));\n ARROW_ASSIGN_OR_RAISE(\n auto original_footer_metadata, combined_file_utils::read_footer(path, preflight_file));\n\n std::shared_ptr file = preflight_file;\n if (!options.force_disable_file_mapping() && getenv(\"POD5_DISABLE_MMAP_OPEN\") == nullptr) {\n auto file_opt = arrow::io::MemoryMappedFile::Open(path, arrow::io::FileMode::READ);\n if (file_opt.ok()) {\n file = *file_opt;\n // Downstream handles are extracted from the `footer.{table}.file`, update them\n // to use the mmap handle.\n combined_file_utils::bind_footer_file(original_footer_metadata, file);\n }\n }\n\n ARROW_ASSIGN_OR_RAISE(\n auto const original_writer_version,\n parse_version_number(original_footer_metadata.writer_pod5_version));\n ARROW_ASSIGN_OR_RAISE(\n auto migration_result,\n migrate_if_required(original_writer_version, original_footer_metadata, file, pool));\n\n // Files are written standalone, and so needs to be treated with a file offset - it wants to seek around as if the reads file is standalone:\n\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_sub_file, open_sub_file(migration_result.footer().run_info_table));\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_table_reader, make_run_info_table_reader(run_info_sub_file, pool));\n\n ARROW_ASSIGN_OR_RAISE(\n auto reads_sub_file, open_sub_file(migration_result.footer().reads_table));\n ARROW_ASSIGN_OR_RAISE(auto read_table_reader, make_read_table_reader(reads_sub_file, pool));\n\n ARROW_ASSIGN_OR_RAISE(\n auto signal_sub_file, open_sub_file(migration_result.footer().signal_table));\n ARROW_ASSIGN_OR_RAISE(\n auto signal_table_reader,\n make_signal_table_reader(signal_sub_file, options.max_cached_signal_table_batches(), pool));\n\n auto signal_metadata = signal_table_reader.schema_metadata();\n auto reads_metadata = read_table_reader.schema_metadata();\n if (signal_metadata.file_identifier != reads_metadata.file_identifier) {\n return Status::Invalid(\n \"Invalid read and signal file pair signal identifier: \",\n signal_metadata.file_identifier,\n \", reads identifier: \",\n reads_metadata.file_identifier);\n }\n\n return std::make_shared(\n original_writer_version,\n std::move(migration_result),\n std::move(run_info_table_reader),\n std::move(read_table_reader),\n std::move(signal_table_reader));\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_reader.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/signal_table_utils.h\"\n\n#include \n#include \n\nnamespace arrow {\nclass Array;\nclass Buffer;\nclass MemoryPool;\n} // namespace arrow\n\nnamespace pod5 {\n\nclass Version;\nstruct SchemaMetadataDescription;\n\nclass POD5_FORMAT_EXPORT FileReaderOptions {\npublic:\n static constexpr std::uint32_t DEFAULT_MAX_CACHED_SIGNAL_TABLE_BATCHES = 5;\n\n FileReaderOptions();\n\n void memory_pool(arrow::MemoryPool * memory_pool) { m_memory_pool = memory_pool; }\n\n arrow::MemoryPool * memory_pool() const { return m_memory_pool; }\n\n std::size_t max_cached_signal_table_batches() const\n {\n return m_max_cached_signal_table_batches;\n }\n\n // Set how many signal table batches can be cached in memory,\n // Note: 0 here implies no limit.\n void set_max_cached_signal_table_batches(std::size_t max_cached_signal_table_batches);\n\n void set_force_disable_file_mapping(bool force_disable_file_mapping)\n {\n m_force_disable_file_mapping = force_disable_file_mapping;\n }\n\n bool force_disable_file_mapping() const { return m_force_disable_file_mapping; }\n\nprivate:\n arrow::MemoryPool * m_memory_pool;\n std::size_t m_max_cached_signal_table_batches;\n bool m_force_disable_file_mapping = false;\n};\n\nclass POD5_FORMAT_EXPORT FileLocation {\npublic:\n FileLocation(std::string const & file_path_, std::size_t offset_, std::size_t size_)\n : file_path(file_path_)\n , offset(offset_)\n , size(size_)\n {\n }\n\n std::string file_path;\n std::size_t offset;\n std::size_t size;\n};\n\nclass ReadTableRecordBatch;\nclass SignalTableRecordBatch;\n\nclass POD5_FORMAT_EXPORT FileReader {\npublic:\n virtual ~FileReader() = default;\n\n /// \\brief Find the read schema metadata for this file.\n virtual SchemaMetadataDescription schema_metadata() const = 0;\n\n virtual Result run_info_count() const = 0;\n virtual Result read_count() const = 0;\n\n virtual Result read_read_record_batch(std::size_t i) const = 0;\n virtual std::size_t num_read_record_batches() const = 0;\n\n virtual Result search_for_read_ids(\n ReadIdSearchInput const & search_input,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows) const = 0;\n\n virtual Result read_signal_record_batch(std::size_t i) const = 0;\n virtual std::size_t num_signal_record_batches() const = 0;\n virtual Result signal_batch_for_row_id(std::size_t row, std::size_t * batch_row)\n const = 0;\n /// \\brief Find the number of samples in a given list of rows.\n /// \\param row_indices The rows to query for sample ount.\n /// \\returns The sum of all sample counts on input rows.\n virtual Result extract_sample_count(\n gsl::span const & row_indices) const = 0;\n\n /// \\brief Extract the samples for a list of rows.\n /// \\param row_indices The rows to query for samples.\n /// \\param output_samples The output samples from the rows.\n virtual Status extract_samples(\n gsl::span const & row_indices,\n gsl::span const & output_samples) const = 0;\n\n /// \\brief Extract the samples as written in the arrow table for a list of rows.\n /// \\param row_indices The rows to query for samples.\n /// \\param sample_count The output samples from the rows.\n virtual Result>> extract_samples_inplace(\n gsl::span const & row_indices,\n std::vector & sample_count) const = 0;\n\n virtual FileLocation const & run_info_table_location() const = 0;\n virtual FileLocation const & read_table_location() const = 0;\n virtual FileLocation const & signal_table_location() const = 0;\n\n virtual Version file_version_pre_migration() const = 0;\n\n virtual SignalType signal_type() const = 0;\n\n virtual Result> find_run_info(\n std::string const & acquisition_id) const = 0;\n\n virtual Result> get_run_info(std::size_t index) const = 0;\n virtual Result get_run_info_count() const = 0;\n};\n\nPOD5_FORMAT_EXPORT pod5::Result> open_file_reader(\n std::string const & path,\n FileReaderOptions const & options = {});\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_recovery.h", "content": "#pragma once\n\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/schema_metadata.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nstatic constexpr char const * kArrowMagicBytes = \"ARROW1\";\n\nstruct RecoveredData {\n // Metadata from the original file:\n SchemaMetadataDescription metadata;\n\n std::size_t recovered_batches = 0;\n arrow::Status failed_batch_status;\n std::size_t recovered_rows = 0;\n};\n\ntemplate \narrow::Result recover_arrow_file(\n std::shared_ptr const & file_to_recover,\n DestFileType const & destination_file)\n{\n // Check for arrow start file:\n int32_t const magic_size = static_cast(::strlen(kArrowMagicBytes));\n ARROW_ASSIGN_OR_RAISE(auto buffer, file_to_recover->ReadAt(0, magic_size));\n if (buffer->size() < magic_size || memcmp(buffer->data(), kArrowMagicBytes, magic_size)) {\n return arrow::Status::Invalid(\"Not an Arrow file\");\n }\n\n // Open the stream format within the ipc file:\n ARROW_ASSIGN_OR_RAISE(\n auto input_stream, combined_file_utils::open_sub_file(file_to_recover, 8));\n ARROW_ASSIGN_OR_RAISE(\n auto opened_stream, arrow::ipc::RecordBatchStreamReader::Open(input_stream));\n\n auto const & expected_schema = destination_file->schema();\n auto schema = opened_stream->schema();\n if (!schema->Equals(*expected_schema, false)) {\n return arrow::Status::Invalid(\n \"Recovered file Schema does not match expected schema, version mismatch?\");\n }\n\n RecoveredData recovered_data;\n ARROW_ASSIGN_OR_RAISE(\n recovered_data.metadata, read_schema_key_value_metadata(schema->metadata()));\n while (true) {\n auto result_opt = opened_stream->Next();\n // Check if the batch failed to load:\n if (!result_opt.ok()) {\n recovered_data.failed_batch_status = result_opt.status();\n return recovered_data;\n }\n\n auto & result = *result_opt;\n if (!result) {\n break;\n }\n\n recovered_data.recovered_batches += 1;\n recovered_data.recovered_rows += result->num_rows();\n ARROW_RETURN_NOT_OK(destination_file->write_batch(*result));\n }\n\n return recovered_data;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_updater.cpp", "content": "#include \"pod5_format/file_updater.h\"\n\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n\nnamespace pod5 {\n\npod5::Status update_file(\n arrow::MemoryPool * pool,\n std::shared_ptr const & source,\n std::string destination)\n{\n ARROW_ASSIGN_OR_RAISE(auto main_file, arrow::io::FileOutputStream::Open(destination, false));\n\n std::random_device gen;\n auto uuid_gen = BasicUuidRandomGenerator{gen};\n auto const section_marker = uuid_gen();\n\n auto metadata = source->schema_metadata();\n\n // Write the initial header to the combined file:\n ARROW_RETURN_NOT_OK(combined_file_utils::write_combined_header(main_file, section_marker));\n\n ARROW_ASSIGN_OR_RAISE(\n auto signal_info_table,\n combined_file_utils::write_file_and_marker(\n pool,\n main_file,\n source->signal_table_location(),\n combined_file_utils::SubFileCleanup::LeaveOrignalFile,\n section_marker));\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_info_table,\n combined_file_utils::write_file_and_marker(\n pool,\n main_file,\n source->run_info_table_location(),\n combined_file_utils::SubFileCleanup::LeaveOrignalFile,\n section_marker));\n ARROW_ASSIGN_OR_RAISE(\n auto reads_info_table,\n combined_file_utils::write_file_and_marker(\n pool,\n main_file,\n source->read_table_location(),\n combined_file_utils::SubFileCleanup::LeaveOrignalFile,\n section_marker));\n\n // Write full file footer:\n ARROW_RETURN_NOT_OK(\n combined_file_utils::write_footer(\n main_file,\n section_marker,\n metadata.file_identifier,\n metadata.writing_software,\n signal_info_table,\n run_info_info_table,\n reads_info_table));\n\n return main_file->Close();\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_updater.h", "content": "#pragma once\n\n#include \"pod5_format/result.h\"\n\n#include \n\nnamespace arrow {\nclass MemoryPool;\n}\n\nnamespace pod5 {\n\nclass FileReader;\n\n/// \\brief Write the path [destination] with any migrated data from [source].\n/// \\param source The source file data to write updated.\n/// \\param destination The destination path to write the data to.\n/// \\note The destination path should not be the same file that was opened for input.\npod5::Status update_file(\n arrow::MemoryPool * pool,\n std::shared_ptr const & source,\n std::string destination);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_writer.cpp", "content": "#include \"pod5_format/file_writer.h\"\n\n#include \"pod5_format/file_recovery.h\"\n#include \"pod5_format/internal/async_output_stream.h\"\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/io_manager.h\"\n#include \"pod5_format/memory_pool.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/read_table_writer.h\"\n#include \"pod5_format/read_table_writer_utils.h\"\n#include \"pod5_format/run_info_table_writer.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/signal_table_reader.h\"\n#include \"pod5_format/signal_table_writer.h\"\n#include \"pod5_format/thread_pool.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n\n#ifdef __linux__\n#include \"pod5_format/internal/linux_output_stream.h\"\n#endif\n\nnamespace {\nstruct CachedFileValues {\n std::shared_ptr io_manager;\n std::shared_ptr thread_pool;\n};\n\nenum class FlushMode { Default, ForceFlushOnBatchComplete };\n\narrow::Result> make_file_stream(\n std::string const & path,\n pod5::FileWriterOptions const & options,\n CachedFileValues & cached_values,\n bool keep_file_open,\n FlushMode flush_mode = FlushMode::Default)\n{\n auto const flush_on_batch_complete =\n flush_mode == FlushMode::ForceFlushOnBatchComplete || options.flush_on_batch_complete();\n#ifdef __linux__\n if (options.use_directio() || options.use_sync_io()) {\n if (!cached_values.io_manager) {\n if (options.io_manager()) {\n cached_values.io_manager = options.io_manager();\n } else {\n ARROW_ASSIGN_OR_RAISE(\n cached_values.io_manager, pod5::make_sync_io_manager(options.memory_pool()));\n }\n }\n auto const ret = pod5::LinuxOutputStream::make(\n path,\n cached_values.io_manager,\n options.write_chunk_size(),\n options.use_directio(),\n options.use_sync_io(),\n flush_on_batch_complete,\n keep_file_open);\n\n // Failure could be due to direct IO used by LinuxOutputStream not being\n // supported. On error drop-through to make a regular AsyncOutputStream.\n if (ret.ok()) {\n return ret;\n }\n }\n\n#endif\n if (!cached_values.thread_pool) {\n if (options.thread_pool()) {\n cached_values.thread_pool = options.thread_pool();\n } else {\n cached_values.thread_pool = pod5::make_thread_pool(1);\n }\n }\n\n return pod5::AsyncOutputStream::make(\n path,\n cached_values.thread_pool,\n flush_on_batch_complete,\n options.memory_pool(),\n keep_file_open);\n}\n} // namespace\n\nnamespace pod5 {\n\nFileWriterOptions::FileWriterOptions()\n: m_max_signal_chunk_size(DEFAULT_SIGNAL_CHUNK_SIZE)\n, m_memory_pool(pod5::default_memory_pool())\n, m_signal_type(DEFAULT_SIGNAL_TYPE)\n, m_signal_table_batch_size(DEFAULT_SIGNAL_TABLE_BATCH_SIZE)\n, m_read_table_batch_size(DEFAULT_READ_TABLE_BATCH_SIZE)\n, m_run_info_table_batch_size(DEFAULT_RUN_INFO_TABLE_BATCH_SIZE)\n, m_use_directio{DEFAULT_USE_DIRECTIO}\n, m_write_chunk_size(DEFAULT_WRITE_CHUNK_SIZE)\n, m_use_sync_io(DEFAULT_USE_SYNC_IO)\n, m_flush_on_batch_complete(DEFAULT_FLUSH_ON_BATCH_COMPLETE)\n, m_keep_signal_file_open(DEFAULT_KEEP_FILES_OPEN)\n, m_keep_run_info_file_open(DEFAULT_KEEP_FILES_OPEN)\n, m_keep_read_table_file_open(DEFAULT_KEEP_FILES_OPEN)\n{\n}\n\nclass FileWriterImpl {\npublic:\n class WriterTypeImpl;\n\n struct DictionaryWriters {\n std::shared_ptr end_reason_writer;\n std::shared_ptr pore_writer;\n std::shared_ptr run_info_writer;\n };\n\n FileWriterImpl(\n DictionaryWriters && read_table_dict_writers,\n RunInfoTableWriter && run_info_table_writer,\n ReadTableWriter && read_table_writer,\n SignalTableWriter && signal_table_writer,\n std::uint32_t signal_chunk_size,\n arrow::MemoryPool * pool)\n : m_read_table_dict_writers(std::move(read_table_dict_writers))\n , m_run_info_table_writer(std::move(run_info_table_writer))\n , m_read_table_writer(std::move(read_table_writer))\n , m_signal_table_writer(std::move(signal_table_writer))\n , m_signal_chunk_size(signal_chunk_size)\n , m_pool(pool)\n {\n }\n\n virtual ~FileWriterImpl() = default;\n\n virtual std::string path() const = 0;\n\n pod5::Result lookup_end_reason(ReadEndReason end_reason)\n {\n return m_read_table_dict_writers.end_reason_writer->lookup(end_reason);\n }\n\n pod5::Result add_pore_type(std::string const & pore_type_data)\n {\n return m_read_table_dict_writers.pore_writer->add(pore_type_data);\n }\n\n pod5::Result add_run_info(RunInfoData const & run_info_data)\n {\n ARROW_RETURN_NOT_OK(m_run_info_table_writer->add_run_info(run_info_data));\n return m_read_table_dict_writers.run_info_writer->add(run_info_data.acquisition_id);\n }\n\n pod5::Status add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal)\n {\n if (!m_signal_table_writer || !m_read_table_writer) {\n return arrow::Status::Invalid(\"File writer closed, cannot write further data\");\n }\n\n ARROW_RETURN_NOT_OK(check_read(read_data));\n\n ARROW_ASSIGN_OR_RAISE(\n std::vector signal_rows, add_signal(read_data.read_id, signal));\n\n // Write read data and signal row entries:\n auto read_table_row = m_read_table_writer->add_read(\n read_data, gsl::make_span(signal_rows.data(), signal_rows.size()), signal.size());\n return read_table_row.status();\n }\n\n pod5::Status add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal_rows,\n std::uint64_t signal_duration)\n {\n if (!m_signal_table_writer || !m_read_table_writer) {\n return arrow::Status::Invalid(\"File writer closed, cannot write further data\");\n }\n\n ARROW_RETURN_NOT_OK(check_read(read_data));\n\n // Write read data and signal row entries:\n auto read_table_row =\n m_read_table_writer->add_read(read_data, signal_rows, signal_duration);\n return read_table_row.status();\n }\n\n arrow::Status check_read(ReadData const & read_data)\n {\n if (!m_read_table_dict_writers.run_info_writer->is_valid(read_data.run_info)) {\n return arrow::Status::Invalid(\"Invalid run info passed to add_read\");\n }\n\n if (!m_read_table_dict_writers.pore_writer->is_valid(read_data.pore_type)) {\n return arrow::Status::Invalid(\"Invalid pore type passed to add_read\");\n }\n\n if (!m_read_table_dict_writers.end_reason_writer->is_valid(read_data.end_reason)) {\n return arrow::Status::Invalid(\"Invalid end reason passed to add_read\");\n }\n\n return arrow::Status::OK();\n }\n\n pod5::Result> add_signal(\n Uuid const & read_id,\n gsl::span const & signal)\n {\n if (!m_signal_table_writer || !m_read_table_writer) {\n return arrow::Status::Invalid(\"File writer closed, cannot write further data\");\n }\n\n std::vector signal_rows;\n signal_rows.reserve((signal.size() / m_signal_chunk_size) + 1);\n\n // Chunk and write each piece of signal to the file:\n for (std::size_t chunk_start = 0; chunk_start < signal.size();\n chunk_start += m_signal_chunk_size)\n {\n std::size_t chunk_size =\n std::min(signal.size() - chunk_start, m_signal_chunk_size);\n\n auto const chunk_span = signal.subspan(chunk_start, chunk_size);\n\n ARROW_ASSIGN_OR_RAISE(\n auto row_index, m_signal_table_writer->add_signal(read_id, chunk_span));\n signal_rows.push_back(row_index);\n }\n return signal_rows;\n }\n\n pod5::Result add_pre_compressed_signal(\n Uuid const & read_id,\n gsl::span const & signal_bytes,\n std::uint32_t sample_count)\n {\n if (!m_signal_table_writer || !m_read_table_writer) {\n return arrow::Status::Invalid(\"File writer closed, cannot write further data\");\n }\n\n return m_signal_table_writer->add_pre_compressed_signal(\n read_id, signal_bytes, sample_count);\n }\n\n pod5::Result> add_signal_batch(\n std::size_t row_count,\n std::vector> && columns,\n bool final_batch)\n {\n if (!m_signal_table_writer || !m_read_table_writer) {\n return arrow::Status::Invalid(\"File writer closed, cannot write further data\");\n }\n\n return m_signal_table_writer->add_signal_batch(row_count, std::move(columns), final_batch);\n }\n\n SignalType signal_type() const { return m_signal_table_writer->signal_type(); }\n\n std::size_t signal_table_batch_size() const\n {\n return m_signal_table_writer->table_batch_size();\n }\n\n pod5::Status close_run_info_table_writer()\n {\n if (m_run_info_table_writer) {\n ARROW_RETURN_NOT_OK(m_run_info_table_writer->close());\n m_run_info_table_writer = std::nullopt;\n }\n return pod5::Status::OK();\n }\n\n pod5::Status close_read_table_writer()\n {\n if (m_read_table_writer) {\n ARROW_RETURN_NOT_OK(m_read_table_writer->close());\n m_read_table_writer = std::nullopt;\n }\n return pod5::Status::OK();\n }\n\n pod5::Status close_signal_table_writer()\n {\n if (m_signal_table_writer) {\n ARROW_RETURN_NOT_OK(m_signal_table_writer->close());\n m_signal_table_writer = std::nullopt;\n }\n return pod5::Status::OK();\n }\n\n virtual arrow::Status close() = 0;\n\n bool is_closed() const\n {\n assert(!!m_read_table_writer == !!m_signal_table_writer);\n return !m_signal_table_writer;\n }\n\n arrow::MemoryPool * pool() const { return m_pool; }\n\n RunInfoTableWriter * run_info_table_writer()\n {\n if (is_closed() || !m_run_info_table_writer.has_value()) {\n return nullptr;\n }\n return &m_run_info_table_writer.value();\n }\n\n ReadTableWriter * read_table_writer()\n {\n if (is_closed() || !m_read_table_writer.has_value()) {\n return nullptr;\n }\n return &m_read_table_writer.value();\n }\n\n SignalTableWriter * signal_table_writer()\n {\n if (is_closed() || !m_signal_table_writer.has_value()) {\n return nullptr;\n }\n return &m_signal_table_writer.value();\n }\n\nprivate:\n DictionaryWriters m_read_table_dict_writers;\n std::optional m_run_info_table_writer;\n std::optional m_read_table_writer;\n std::optional m_signal_table_writer;\n std::uint32_t m_signal_chunk_size;\n arrow::MemoryPool * m_pool;\n};\n\nclass CombinedFileWriterImpl : public FileWriterImpl {\npublic:\n CombinedFileWriterImpl(\n std::string const & path,\n std::string const & run_info_tmp_path,\n std::string const & reads_tmp_path,\n std::int64_t signal_file_start_offset,\n Uuid const & section_marker,\n Uuid const & file_identifier,\n std::string const & software_name,\n DictionaryWriters && dict_writers,\n RunInfoTableWriter && run_info_table_writer,\n ReadTableWriter && read_table_writer,\n SignalTableWriter && signal_table_writer,\n std::uint32_t signal_chunk_size,\n arrow::MemoryPool * pool)\n : FileWriterImpl(\n std::move(dict_writers),\n std::move(run_info_table_writer),\n std::move(read_table_writer),\n std::move(signal_table_writer),\n signal_chunk_size,\n pool)\n , m_path(path)\n , m_run_info_tmp_path(run_info_tmp_path)\n , m_reads_tmp_path(reads_tmp_path)\n , m_signal_file_start_offset(signal_file_start_offset)\n , m_section_marker(section_marker)\n , m_file_identifier(file_identifier)\n , m_software_name(software_name)\n {\n }\n\n std::string path() const override { return m_path; }\n\n arrow::Status close() override\n {\n if (is_closed()) {\n return arrow::Status::OK();\n }\n ARROW_RETURN_NOT_OK(close_run_info_table_writer());\n ARROW_RETURN_NOT_OK(close_read_table_writer());\n ARROW_RETURN_NOT_OK(close_signal_table_writer());\n\n // Open main path with append set:\n ARROW_ASSIGN_OR_RAISE(auto file, arrow::io::FileOutputStream::Open(m_path, true));\n\n // Record signal table length:\n combined_file_utils::FileInfo signal_table;\n signal_table.file_start_offset = m_signal_file_start_offset;\n ARROW_ASSIGN_OR_RAISE(signal_table.file_length, file->Tell());\n signal_table.file_length -= signal_table.file_start_offset;\n\n // pad file to 8 bytes and mark section:\n ARROW_RETURN_NOT_OK(combined_file_utils::pad_file(file, 8));\n ARROW_RETURN_NOT_OK(combined_file_utils::write_section_marker(file, m_section_marker));\n\n auto file_location_for_full_file =\n [&](std::string const & filename) -> arrow::Result {\n ARROW_ASSIGN_OR_RAISE(auto file, arrow::io::ReadableFile::Open(filename, pool()));\n ARROW_ASSIGN_OR_RAISE(auto size, file->GetSize());\n return FileLocation{filename, 0, std::size_t(size)};\n };\n\n // Write in run_info table:\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_location, file_location_for_full_file(m_run_info_tmp_path));\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_info_table,\n combined_file_utils::write_file_and_marker(\n pool(),\n file,\n run_info_location,\n combined_file_utils::SubFileCleanup::CleanupOriginalFile,\n m_section_marker));\n\n // Write in read table:\n ARROW_ASSIGN_OR_RAISE(auto reads_location, file_location_for_full_file(m_reads_tmp_path));\n ARROW_ASSIGN_OR_RAISE(\n auto reads_info_table,\n combined_file_utils::write_file_and_marker(\n pool(),\n file,\n reads_location,\n combined_file_utils::SubFileCleanup::CleanupOriginalFile,\n m_section_marker));\n\n // Write full file footer:\n ARROW_RETURN_NOT_OK(\n combined_file_utils::write_footer(\n file,\n m_section_marker,\n m_file_identifier,\n m_software_name,\n signal_table,\n run_info_info_table,\n reads_info_table));\n return arrow::Status::OK();\n }\n\nprivate:\n std::string m_path;\n std::string m_run_info_tmp_path;\n std::string m_reads_tmp_path;\n std::int64_t m_signal_file_start_offset;\n Uuid m_section_marker;\n Uuid m_file_identifier;\n std::string m_software_name;\n};\n\nFileWriter::FileWriter(std::unique_ptr && impl) : m_impl(std::move(impl)) {}\n\nFileWriter::~FileWriter() { (void)close(); }\n\nstd::string FileWriter::path() const { return m_impl->path(); }\n\narrow::Status FileWriter::close() { return m_impl->close(); }\n\narrow::Status FileWriter::add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal)\n{\n return m_impl->add_complete_read(read_data, signal);\n}\n\narrow::Status FileWriter::add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal_rows,\n std::uint64_t signal_duration)\n{\n return m_impl->add_complete_read(read_data, signal_rows, signal_duration);\n}\n\npod5::Result> FileWriter::add_signal(\n Uuid const & read_id,\n gsl::span const & signal)\n{\n return m_impl->add_signal(read_id, signal);\n}\n\npod5::Result FileWriter::add_pre_compressed_signal(\n Uuid const & read_id,\n gsl::span const & signal_bytes,\n std::uint32_t sample_count)\n{\n return m_impl->add_pre_compressed_signal(read_id, signal_bytes, sample_count);\n}\n\npod5::Result> FileWriter::add_signal_batch(\n std::size_t row_count,\n std::vector> && columns,\n bool final_batch)\n{\n return m_impl->add_signal_batch(row_count, std::move(columns), final_batch);\n}\n\npod5::Result FileWriter::lookup_end_reason(ReadEndReason end_reason) const\n{\n return m_impl->lookup_end_reason(end_reason);\n}\n\npod5::Result FileWriter::add_pore_type(std::string const & pore_type_data)\n{\n return m_impl->add_pore_type(pore_type_data);\n}\n\npod5::Result FileWriter::add_run_info(RunInfoData const & run_info_data)\n{\n return m_impl->add_run_info(run_info_data);\n}\n\nSignalType FileWriter::signal_type() const { return m_impl->signal_type(); }\n\nstd::size_t FileWriter::signal_table_batch_size() const\n{\n return m_impl->signal_table_batch_size();\n}\n\npod5::Result make_dictionary_writers(arrow::MemoryPool * pool)\n{\n FileWriterImpl::DictionaryWriters writers;\n\n ARROW_ASSIGN_OR_RAISE(writers.end_reason_writer, pod5::make_end_reason_writer(pool));\n ARROW_ASSIGN_OR_RAISE(writers.pore_writer, pod5::make_pore_writer(pool));\n ARROW_ASSIGN_OR_RAISE(writers.run_info_writer, pod5::make_run_info_writer(pool));\n\n return writers;\n}\n\nstd::string make_reads_tmp_path(\n ::arrow::internal::PlatformFilename const & arrow_path,\n Uuid const & file_identifier)\n{\n return arrow_path.Parent().ToString() + \"/\" + (\".\" + to_string(file_identifier) + \".tmp-reads\");\n}\n\nstd::string make_run_info_tmp_path(\n ::arrow::internal::PlatformFilename const & arrow_path,\n Uuid const & file_identifier)\n{\n return arrow_path.Parent().ToString() + \"/\"\n + (\".\" + to_string(file_identifier) + \".tmp-run-info\");\n}\n\npod5::Result> create_file_writer(\n std::string const & path,\n std::string const & writing_software_name,\n FileWriterOptions const & options)\n{\n auto pool = options.memory_pool();\n if (!pool) {\n return Status::Invalid(\"Invalid memory pool specified for file writer\");\n }\n\n ARROW_ASSIGN_OR_RAISE(auto arrow_path, ::arrow::internal::PlatformFilename::FromString(path));\n ARROW_ASSIGN_OR_RAISE(bool file_exists, arrow::internal::FileExists(arrow_path));\n if (file_exists) {\n return Status::Invalid(\"Unable to create new file '\", path, \"', already exists\");\n }\n\n // Open dictionary writers:\n ARROW_ASSIGN_OR_RAISE(auto dict_writers, make_dictionary_writers(pool));\n\n // Prep file metadata:\n std::random_device gen;\n auto uuid_gen = BasicUuidRandomGenerator{gen};\n auto const section_marker = uuid_gen();\n auto const file_identifier = uuid_gen();\n\n ARROW_ASSIGN_OR_RAISE(auto current_version, parse_version_number(Pod5Version));\n ARROW_ASSIGN_OR_RAISE(\n auto file_schema_metadata,\n make_schema_key_value_metadata({file_identifier, writing_software_name, current_version}));\n\n auto reads_tmp_path = make_reads_tmp_path(arrow_path, file_identifier);\n auto run_info_tmp_path = make_run_info_tmp_path(arrow_path, file_identifier);\n\n CachedFileValues cached_values;\n\n // Prepare the temporary reads file:\n ARROW_ASSIGN_OR_RAISE(\n auto read_table_file_async,\n make_file_stream(\n reads_tmp_path, options, cached_values, options.keep_read_table_file_open()));\n ARROW_ASSIGN_OR_RAISE(\n auto read_table_tmp_writer,\n make_read_table_writer(\n read_table_file_async,\n file_schema_metadata,\n options.read_table_batch_size(),\n dict_writers.pore_writer,\n dict_writers.end_reason_writer,\n dict_writers.run_info_writer,\n pool));\n\n // Prepare the temporary run_info file:\n //\n // Run info is normally global, if we don't flush on batch complete we can\n // lose a large number of reads in a crash.\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_table_file_async,\n make_file_stream(\n run_info_tmp_path,\n options,\n cached_values,\n options.keep_run_info_file_open(),\n FlushMode::ForceFlushOnBatchComplete));\n\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_table_tmp_writer,\n make_run_info_table_writer(\n run_info_table_file_async,\n file_schema_metadata,\n options.run_info_table_batch_size(),\n pool));\n\n // Prepare the main file - and set up the signal table to write here:\n ARROW_ASSIGN_OR_RAISE(\n auto signal_file,\n make_file_stream(path, options, cached_values, options.keep_signal_file_open()));\n\n // Write the initial header to the combined file:\n ARROW_RETURN_NOT_OK(combined_file_utils::write_combined_header(signal_file, section_marker));\n\n ARROW_ASSIGN_OR_RAISE(size_t const signal_table_start, signal_file->Tell());\n\n static_cast(signal_file.get())->set_file_start_offset(signal_table_start);\n\n // Then place the signal file directly after that:\n ARROW_ASSIGN_OR_RAISE(\n auto signal_table_writer,\n make_signal_table_writer(\n signal_file,\n file_schema_metadata,\n options.signal_table_batch_size(),\n options.signal_type(),\n pool));\n\n // Throw it all together into a writer object:\n return std::make_unique(std::make_unique(\n path,\n run_info_tmp_path,\n reads_tmp_path,\n signal_table_start,\n section_marker,\n file_identifier,\n writing_software_name,\n std::move(dict_writers),\n std::move(run_info_table_tmp_writer),\n std::move(read_table_tmp_writer),\n std::move(signal_table_writer),\n options.max_signal_chunk_size(),\n pool));\n}\n\nstatic Status add_recovery_failure_context(\n Status status,\n std::string const & tmp_path,\n std::string const & description)\n{\n assert(!status.ok());\n std::string const error_context =\n \"Failed whilst attempting to recover \" + description + \" from file - \" + tmp_path;\n if (status.detail()) {\n return status.WithMessage(error_context);\n }\n return arrow::Status::FromArgs(status.code(), error_context + \". Detail: \" + status.message());\n}\n\ntemplate \nstatic Status append_recovered_file(\n std::string const & tmp_path,\n writer_type const & destination_writer,\n std::string const & description,\n arrow::MemoryPool * const pool)\n{\n arrow::Status inner_status = [&] {\n ARROW_ASSIGN_OR_RAISE(auto file, arrow::io::ReadableFile::Open(tmp_path, pool));\n ARROW_ASSIGN_OR_RAISE(auto size, file->GetSize());\n if (size == 0) {\n return arrow::Status::Invalid(\"File is empty/zero bytes long.\");\n }\n ARROW_ASSIGN_OR_RAISE(\n RecoveredData const recovered_raw_data, recover_arrow_file(file, destination_writer));\n return arrow::Status::OK();\n }();\n if (!inner_status.ok()) {\n return add_recovery_failure_context(inner_status, tmp_path, description);\n }\n return inner_status;\n}\n\nnamespace {\nstruct TemporaryFilePaths {\n std::string run_info;\n std::string reads;\n};\n} // namespace\n\nstatic pod5::Status recover_file(\n std::string const & src_path,\n std::string const & dest_path,\n FileWriterOptions const & options,\n TemporaryFilePaths & temporary_file_paths)\n{\n if (!check_extension_types_registered()) {\n return arrow::Status::Invalid(\"POD5 library is not correctly initialised.\");\n }\n\n // Create a file to push recovered data into:\n ARROW_ASSIGN_OR_RAISE(\n auto dest_file, create_file_writer(dest_path, \"pod5_file_recovery\", options));\n\n auto pool = arrow::default_memory_pool();\n ARROW_ASSIGN_OR_RAISE(\n auto arrow_path, ::arrow::internal::PlatformFilename::FromString(src_path));\n ARROW_ASSIGN_OR_RAISE(auto file, arrow::io::ReadableFile::Open(src_path, pool));\n\n // Signature should be right at 0:\n ARROW_RETURN_NOT_OK(combined_file_utils::check_signature(file, 0));\n\n // Recover the signal data into [dest_file]:\n arrow::Result recovered_raw_data;\n {\n ARROW_ASSIGN_OR_RAISE(\n auto raw_sub_file,\n combined_file_utils::open_sub_file(file, combined_file_utils::header_size));\n recovered_raw_data =\n recover_arrow_file(raw_sub_file, dest_file->impl()->signal_table_writer());\n }\n if (!recovered_raw_data.ok()) {\n return add_recovery_failure_context(\n recovered_raw_data.status(), arrow_path.ToString(), \"signal data sub file\");\n }\n\n auto file_identifier = recovered_raw_data->metadata.file_identifier;\n temporary_file_paths.run_info = make_run_info_tmp_path(arrow_path, file_identifier);\n temporary_file_paths.reads = make_reads_tmp_path(arrow_path, file_identifier);\n\n // Recover the run info data into [dest_file]:\n auto run_info_writer = dest_file->impl()->run_info_table_writer();\n ARROW_RETURN_NOT_OK(append_recovered_file(\n temporary_file_paths.run_info, run_info_writer, \"run information\", pool));\n\n // Recover the read data into [dest_file]:\n auto read_writer = dest_file->impl()->read_table_writer();\n ARROW_RETURN_NOT_OK(\n append_recovered_file(temporary_file_paths.reads, read_writer, \"reads\", pool));\n\n return dest_file->close();\n}\n\n/// This is a thorough count of all rows. Doing it this way ensures that all rows can be read.\nstatic pod5::Result count_recovered_rows(\n std::filesystem::path const & recovered_path)\n{\n ARROW_ASSIGN_OR_RAISE(\n std::shared_ptr recovered,\n pod5::open_file_reader(recovered_path.string()));\n RecoveredRowCounts counts;\n\n std::size_t const signal_batches = recovered->num_signal_record_batches();\n for (std::size_t index = 0; index < signal_batches; ++index) {\n ARROW_ASSIGN_OR_RAISE(auto const signal_batch, recovered->read_signal_record_batch(index));\n counts.signal += signal_batch.num_rows();\n }\n\n ARROW_ASSIGN_OR_RAISE(counts.run_info, recovered->run_info_count());\n\n std::size_t const read_batches = recovered->num_read_record_batches();\n for (std::size_t index = 0; index < read_batches; ++index) {\n ARROW_ASSIGN_OR_RAISE(auto const record_batch, recovered->read_read_record_batch(index));\n counts.reads += record_batch.num_rows();\n }\n return counts;\n}\n\n/// \\brief File is considered useless for recovery if it is 0 bytes long\n/// or if all bytes have value 0.\nstatic bool is_useless(std::filesystem::path const & file_path)\n{\n if (file_size(file_path) == 0) {\n return true;\n }\n std::ifstream file{file_path, std::ios::in | std::ios::binary};\n if (!file.is_open()) {\n // If we can't open the file, assume there is data, just in case.\n return false;\n }\n while (true) {\n std::uint8_t byte;\n file >> byte;\n if (file.eof()) {\n return true;\n }\n if (byte != 0) {\n return false;\n }\n }\n}\n\nstatic void remove_if_useless(std::filesystem::path const & file_path)\n{\n if (exists(file_path) && is_useless(file_path)) {\n remove(file_path);\n }\n}\n\nstatic std::optional try_remove(std::string const & file_path)\n{\n try {\n std::filesystem::remove(file_path);\n return {};\n } catch (std::filesystem::filesystem_error const & error) {\n return CleanupError{.file_path = file_path, .description = error.what()};\n }\n}\n\nstatic Status add_clean_up_error(Status status, std::filesystem::filesystem_error const & exception)\n{\n return arrow::Status::FromArgs(status.code(), status.message(), exception.what());\n}\n\npod5::Result recover_file(\n std::string const & src_path,\n std::string const & dest_path,\n RecoverFileOptions const & options)\n{\n TemporaryFilePaths temp_file_paths;\n auto const result = [&]() -> pod5::Result {\n ARROW_RETURN_NOT_OK(\n recover_file(src_path, dest_path, options.file_writer_options, temp_file_paths));\n auto const row_count_result = count_recovered_rows(dest_path);\n if (!row_count_result.ok()) {\n return add_recovery_failure_context(row_count_result.status(), dest_path, \"row counts\");\n }\n return row_count_result;\n }();\n if (!options.cleanup) {\n auto const to_recovery_details = [](RecoveredRowCounts counts) {\n return RecoveryDetails{counts};\n };\n return result.Map(to_recovery_details);\n }\n if (!result.ok()) {\n try {\n if (std::filesystem::exists(dest_path)) {\n std::filesystem::remove(dest_path);\n }\n remove_if_useless(temp_file_paths.reads);\n remove_if_useless(temp_file_paths.run_info);\n remove_if_useless(src_path);\n } catch (std::filesystem::filesystem_error const & error) {\n return add_clean_up_error(result.status(), error);\n }\n return result.status();\n }\n\n RecoveryDetails details{.row_counts = *result};\n if (auto const error = try_remove(src_path)) {\n details.cleanup_errors.push_back(*error);\n }\n if (auto const error = try_remove(temp_file_paths.reads)) {\n details.cleanup_errors.push_back(*error);\n }\n if (auto const error = try_remove(temp_file_paths.run_info)) {\n details.cleanup_errors.push_back(*error);\n }\n return details;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/file_writer.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/signal_table_utils.h\"\n\n#include \n#include \n\nnamespace arrow {\nclass Array;\nclass MemoryPool;\n} // namespace arrow\n\nnamespace pod5 {\n\nclass IOManager;\nclass ThreadPool;\n\nclass POD5_FORMAT_EXPORT FileWriterOptions {\npublic:\n /// \\brief Default chunk size for signal table entries\n static constexpr std::uint32_t DEFAULT_SIGNAL_CHUNK_SIZE = 102'400;\n static constexpr std::uint32_t DEFAULT_SIGNAL_TABLE_BATCH_SIZE = 100;\n static constexpr std::uint32_t DEFAULT_READ_TABLE_BATCH_SIZE = 1000;\n static constexpr std::uint32_t DEFAULT_RUN_INFO_TABLE_BATCH_SIZE = 1;\n static constexpr SignalType DEFAULT_SIGNAL_TYPE = SignalType::VbzSignal;\n static constexpr bool DEFAULT_USE_DIRECTIO = false;\n static constexpr bool DEFAULT_USE_SYNC_IO = false;\n static constexpr bool DEFAULT_FLUSH_ON_BATCH_COMPLETE = true;\n static constexpr std::size_t DEFAULT_WRITE_CHUNK_SIZE = 2 * 1024 * 1024;\n static constexpr std::size_t DEFAULT_KEEP_FILES_OPEN = true;\n\n FileWriterOptions();\n\n void set_max_signal_chunk_size(std::uint32_t chunk_size)\n {\n m_max_signal_chunk_size = chunk_size;\n }\n\n std::uint32_t max_signal_chunk_size() const { return m_max_signal_chunk_size; }\n\n void set_memory_pool(arrow::MemoryPool * memory_pool) { m_memory_pool = memory_pool; }\n\n arrow::MemoryPool * memory_pool() const { return m_memory_pool; }\n\n void set_signal_type(SignalType signal_type) { m_signal_type = signal_type; }\n\n SignalType signal_type() const { return m_signal_type; }\n\n void set_signal_table_batch_size(std::size_t batch_size)\n {\n m_signal_table_batch_size = batch_size;\n }\n\n std::size_t signal_table_batch_size() const { return m_signal_table_batch_size; }\n\n void set_read_table_batch_size(std::size_t batch_size) { m_read_table_batch_size = batch_size; }\n\n std::size_t read_table_batch_size() const { return m_read_table_batch_size; }\n\n void set_run_info_table_batch_size(std::size_t batch_size)\n {\n m_run_info_table_batch_size = batch_size;\n }\n\n std::size_t run_info_table_batch_size() const { return m_run_info_table_batch_size; }\n\n void set_io_manager(std::shared_ptr const & io_manager)\n {\n m_io_manager = io_manager;\n }\n\n std::shared_ptr io_manager() const { return m_io_manager; }\n\n void set_thread_pool(std::shared_ptr const & writer_thread_pool)\n {\n m_writer_thread_pool = writer_thread_pool;\n }\n\n std::shared_ptr thread_pool() const { return m_writer_thread_pool; }\n\n void set_use_directio(bool use_directio) { m_use_directio = use_directio; }\n\n bool use_directio() const { return m_use_directio; }\n\n void set_write_chunk_size(std::size_t chunk_size) { m_write_chunk_size = chunk_size; }\n\n std::size_t write_chunk_size() const { return m_write_chunk_size; }\n\n void set_use_sync_io(bool use_sync_io) { m_use_sync_io = use_sync_io; }\n\n bool use_sync_io() const { return m_use_sync_io; }\n\n void set_flush_on_batch_complete(bool flush_on_batch_complete)\n {\n m_flush_on_batch_complete = flush_on_batch_complete;\n }\n\n bool flush_on_batch_complete() const { return m_flush_on_batch_complete; }\n\n bool keep_signal_file_open() const { return m_keep_signal_file_open; }\n\n void set_keep_signal_file_open(bool keep_signal_file_open)\n {\n m_keep_signal_file_open = keep_signal_file_open;\n }\n\n bool keep_run_info_file_open() const { return m_keep_run_info_file_open; }\n\n void set_keep_run_info_file_open(bool keep_run_info_file_open)\n {\n m_keep_run_info_file_open = keep_run_info_file_open;\n }\n\n bool keep_read_table_file_open() const { return m_keep_read_table_file_open; }\n\n void set_keep_read_table_file_open(bool keep_read_table_file_open)\n {\n m_keep_read_table_file_open = keep_read_table_file_open;\n }\n\nprivate:\n std::shared_ptr m_writer_thread_pool;\n std::shared_ptr m_io_manager;\n std::uint32_t m_max_signal_chunk_size;\n arrow::MemoryPool * m_memory_pool;\n SignalType m_signal_type;\n std::size_t m_signal_table_batch_size;\n std::size_t m_read_table_batch_size;\n std::size_t m_run_info_table_batch_size;\n bool m_use_directio;\n std::size_t m_write_chunk_size;\n bool m_use_sync_io;\n bool m_flush_on_batch_complete;\n bool m_keep_signal_file_open;\n bool m_keep_run_info_file_open;\n bool m_keep_read_table_file_open;\n};\n\nclass FileWriterImpl;\n\nclass POD5_FORMAT_EXPORT FileWriter {\npublic:\n FileWriter(std::unique_ptr && impl);\n ~FileWriter();\n\n std::string path() const;\n\n pod5::Status close();\n\n pod5::Status add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal);\n\n /// \\brief Add a complete with rows already pre appended.\n pod5::Status add_complete_read(\n ReadData const & read_data,\n gsl::span const & signal_rows,\n std::uint64_t signal_duration);\n\n pod5::Result> add_signal(\n Uuid const & read_id,\n gsl::span const & signal);\n\n pod5::Result add_pre_compressed_signal(\n Uuid const & read_id,\n gsl::span const & signal_bytes,\n std::uint32_t sample_count);\n\n pod5::Result> add_signal_batch(\n std::size_t row_count,\n std::vector> && columns,\n bool final_batch);\n\n // Find or create an end reason index representing this read end reason.\n pod5::Result lookup_end_reason(ReadEndReason end_reason) const;\n pod5::Result add_pore_type(std::string const & pore_type_data);\n pod5::Result add_run_info(RunInfoData const & run_info_data);\n\n SignalType signal_type() const;\n std::size_t signal_table_batch_size() const;\n\n FileWriterImpl * impl() const { return m_impl.get(); };\n\nprivate:\n std::unique_ptr m_impl;\n};\n\nPOD5_FORMAT_EXPORT pod5::Result> create_file_writer(\n std::string const & path,\n std::string const & writing_software_name,\n FileWriterOptions const & options = {});\n\nstruct POD5_FORMAT_EXPORT RecoverFileOptions {\n FileWriterOptions file_writer_options = {};\n\n /// If this is set to true, recover_file will remove the following files\n /// * Temp files which we have successfully recovered data from.\n /// * Temp files which we have failed to recover data from and which hold no data.\n /// * Output file created during failed recovery.\n bool cleanup = false;\n};\n\nstruct POD5_FORMAT_EXPORT RecoveredRowCounts final {\n std::size_t signal = 0;\n std::size_t run_info = 0;\n std::size_t reads = 0;\n};\n\nstruct POD5_FORMAT_EXPORT CleanupError final {\n std::string file_path;\n std::string description;\n};\n\nstruct POD5_FORMAT_EXPORT RecoveryDetails final {\n RecoveredRowCounts row_counts;\n std::vector cleanup_errors;\n};\n\nPOD5_FORMAT_EXPORT pod5::Result recover_file(\n std::string const & src_path,\n std::string const & dest_path,\n RecoverFileOptions const & options = {});\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/flatbuffers/footer.fbs", "content": "namespace Minknow.ReadsFormat;\n\nenum ContentType:short {\n // The Reads table (an Arrow table)\n ReadsTable,\n // The Signal table (an Arrow table)\n SignalTable,\n // An index for looking up data in the ReadsTable by read_id\n ReadIdIndex,\n // An index based on other columns and/or tables (it will need to be opened to find out what it indexes)\n OtherIndex,\n // The Run Info table (an Arrow table)\n RunInfoTable,\n}\n\nenum Format:short {\n // The Apache Feather V2 format, also known as the Apache Arrow IPC File format.\n FeatherV2,\n}\n\n// Describes an embedded file.\ntable EmbeddedFile {\n // The start of the embedded file\n offset: int64;\n // The length of the embedded file (excluding any padding)\n length: int64;\n // The format of the file\n format: Format;\n // What contents should be expected in the file\n content_type: ContentType;\n}\n\ntable Footer {\n // Must match the \"MINKNOW:file_identifier\" custom metadata entry in the schemas of the bundled tables.\n file_identifier: string;\n // A free-form description of the software that wrote the file, intended to help pin down the source of files that violate the specification.\n software: string;\n // The version of this specification that the table schemas are based on (1.0.0).\n pod5_version: string;\n // The Apache Arrow tables stored in the file.\n contents: [ EmbeddedFile ];\n}\n" }, { "path": "c++/pod5_format/internal/async_output_stream.h", "content": "#pragma once\n\n#include \"pod5_format/file_output_stream.h\"\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"pod5_format/thread_pool.h\"\n\n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nclass AsyncOutputStream : public FileOutputStream {\n struct PrivateDummy {};\n\npublic:\n static arrow::Result> make(\n std::string const & file_path,\n std::shared_ptr const & thread_pool,\n bool flush_on_batch_complete,\n arrow::MemoryPool * memory_pool = arrow::default_memory_pool(),\n bool keep_file_open = true)\n {\n return std::make_shared(\n file_path,\n thread_pool,\n flush_on_batch_complete,\n memory_pool,\n keep_file_open,\n PrivateDummy{});\n }\n\n ~AsyncOutputStream() { (void)Close(); }\n\n arrow::Status Close() override\n {\n // flush all output\n ARROW_RETURN_NOT_OK(Flush());\n\n // and close stream\n std::lock_guard l{m_file_handle_mutex};\n if (m_file_handle) {\n fclose(m_file_handle);\n m_file_handle = nullptr;\n }\n return arrow::Status::OK();\n }\n\n arrow::Future<> CloseAsync() override\n {\n ARROW_RETURN_NOT_OK(Close());\n return FileOutputStream::CloseAsync();\n }\n\n arrow::Status Abort() override\n {\n std::lock_guard l{m_file_handle_mutex};\n if (m_file_handle) {\n fclose(m_file_handle);\n m_file_handle = nullptr;\n }\n return arrow::Status::OK();\n }\n\n arrow::Result Tell() const override\n {\n return m_actual_bytes_written - m_file_start_offset;\n }\n\n bool closed() const override { return m_file_handle == nullptr; }\n\n arrow::Status Write(void const * data, int64_t nbytes) override\n {\n POD5_TRACE_FUNCTION();\n ARROW_ASSIGN_OR_RAISE(\n std::shared_ptr buffer, arrow::AllocateBuffer(nbytes, m_memory_pool));\n auto const char_data = static_cast(data);\n std::copy(char_data, char_data + nbytes, buffer->mutable_data());\n return Write(buffer);\n }\n\n arrow::Status Write(std::shared_ptr const & data) override\n {\n POD5_TRACE_FUNCTION();\n if (m_has_error) {\n return error();\n }\n\n std::size_t const BUFFER_SIZE = 10 * 1024 * 1024; // 10mb pending writes max\n while ((m_submitted_byte_writes - m_completed_byte_writes) > BUFFER_SIZE) {\n std::this_thread::sleep_for(std::chrono::milliseconds(5));\n }\n\n m_submitted_byte_writes += data->size();\n m_actual_bytes_written += data->size();\n\n m_submitted_writes += 1;\n m_strand->post([&, data] {\n POD5_TRACE_FUNCTION();\n if (m_has_error) {\n return;\n }\n\n std::lock_guard l{m_file_handle_mutex};\n auto file_handle = get_or_open_file_handle(l);\n if (!file_handle) {\n set_error(arrow::Status::IOError(\"Failed to open file handle for writing\"));\n return;\n }\n if (fwrite(data->data(), 1, (std::size_t)data->size(), file_handle)\n != (std::size_t)data->size())\n {\n set_error(arrow::Status::IOError(\"Failed to write data to file\"));\n return;\n }\n m_completed_byte_writes += data->size();\n\n // Ensure we do this after editing all the other members, in order to prevent `Flush`\n // returning until we are done.\n m_completed_writes += 1;\n\n // Close the file handle if we do not have further writes pending:\n if (m_submitted_writes == m_completed_writes) {\n close_file_handle(l);\n }\n });\n\n return arrow::Status::OK();\n }\n\n arrow::Status Flush() override\n {\n POD5_TRACE_FUNCTION();\n // Wait for our completed writes to match our submitted writes,\n // this guarantees our async operations are finished.\n auto wait_for_write_count = m_submitted_writes.load();\n while (m_completed_writes.load() < wait_for_write_count && !m_has_error) {\n std::this_thread::sleep_for(std::chrono::microseconds(10));\n }\n\n if (m_has_error) {\n return error();\n }\n\n // No file handle so nothing to flush\n std::lock_guard l{m_file_handle_mutex};\n if (!m_file_handle) {\n return arrow::Status::OK();\n }\n\n if (fflush(m_file_handle) != 0) {\n return arrow::Status::IOError(\"Error flushing file\");\n }\n return arrow::Status::OK();\n }\n\n void set_file_start_offset(std::size_t val) override { m_file_start_offset = val; }\n\n arrow::Status batch_complete() override\n {\n if (m_flush_on_batch_complete) {\n return Flush();\n }\n return arrow::Status::OK();\n }\n\n AsyncOutputStream(\n std::string const & file_path,\n std::shared_ptr const & thread_pool,\n bool flush_on_batch_complete,\n arrow::MemoryPool * memory_pool,\n bool keep_file_open,\n PrivateDummy)\n : m_has_error{false}\n , m_submitted_writes{0}\n , m_completed_writes{0}\n , m_submitted_byte_writes{0}\n , m_completed_byte_writes{0}\n , m_actual_bytes_written{0}\n , m_flush_on_batch_complete(flush_on_batch_complete)\n , m_file_path(file_path)\n , m_keep_file_open(keep_file_open)\n , m_file_start_offset{0}\n , m_strand{thread_pool->create_strand()}\n , m_memory_pool(memory_pool)\n {\n m_file_handle = fopen(m_file_path.c_str(), \"wb\");\n if (!m_file_handle) {\n set_error(arrow::Status::IOError(\"Failed to open file for writing: \", errno));\n }\n if (!m_keep_file_open) {\n fclose(m_file_handle);\n m_file_handle = nullptr;\n }\n }\n\nprivate:\n arrow::MemoryPool * memory_pool() { return m_memory_pool; }\n\n FILE * get_or_open_file_handle([[maybe_unused]] std::lock_guard & lock)\n {\n if (m_file_handle) {\n return m_file_handle;\n }\n\n m_file_handle = fopen(m_file_path.c_str(), \"ab\");\n return m_file_handle;\n }\n\n void close_file_handle([[maybe_unused]] std::lock_guard & lock)\n {\n if (m_file_handle && !m_keep_file_open) {\n fclose(m_file_handle);\n m_file_handle = nullptr;\n }\n }\n\n void set_error(arrow::Status status)\n {\n assert(!status.ok());\n {\n std::lock_guard l{m_error_mutex};\n m_error = std::move(status);\n }\n m_has_error = true;\n }\n\n arrow::Status error() const\n {\n std::lock_guard l{m_error_mutex};\n return m_error;\n }\n\n std::atomic m_has_error;\n\n std::atomic m_submitted_writes;\n std::atomic m_completed_writes;\n std::atomic m_submitted_byte_writes;\n std::atomic m_completed_byte_writes;\n // this represents the number of data bytes written (excluding any padding for alignment)\n // used for truncating the file for instance\n std::int64_t m_actual_bytes_written;\n\n bool m_flush_on_batch_complete;\n\n std::string m_file_path;\n std::mutex m_file_handle_mutex;\n FILE * m_file_handle{nullptr};\n bool m_keep_file_open{false};\n\n mutable std::mutex m_error_mutex;\n arrow::Status m_error;\n\n std::size_t m_file_start_offset;\n std::shared_ptr m_strand;\n arrow::MemoryPool * m_memory_pool;\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/internal/combined_file_utils.h", "content": "#pragma once\n\n#include \"pod5_flatbuffers/footer_generated.h\"\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 { namespace combined_file_utils {\n\nstatic constexpr std::array\n FILE_SIGNATURE{'\\213', 'P', 'O', 'D', '\\r', '\\n', '\\032', '\\n'};\n\nstatic constexpr std::size_t header_size = 24; // signature 8 bytes, section marker 16 bytes\n\ninline pod5::Status pad_file(\n std::shared_ptr const & sink,\n std::uint32_t pad_to_size)\n{\n ARROW_ASSIGN_OR_RAISE(auto const current_byte_location, sink->Tell());\n auto const bytes_to_write = pad_to_size - (current_byte_location % pad_to_size);\n if (bytes_to_write == pad_to_size) {\n return pod5::Status::OK();\n }\n\n std::array zeroes{};\n return sink->Write(zeroes.data(), bytes_to_write);\n}\n\ninline pod5::Status write_file_signature(std::shared_ptr const & sink)\n{\n return sink->Write(FILE_SIGNATURE.data(), FILE_SIGNATURE.size());\n}\n\ninline pod5::Status write_section_marker(\n std::shared_ptr const & sink,\n Uuid const & section_marker)\n{\n return sink->Write(section_marker.data(), section_marker.size());\n}\n\ninline pod5::Status write_combined_header(\n std::shared_ptr const & sink,\n Uuid const & section_marker)\n{\n ARROW_RETURN_NOT_OK(write_file_signature(sink));\n return write_section_marker(sink, section_marker);\n}\n\ninline pod5::Status write_footer_magic(std::shared_ptr const & sink)\n{\n return sink->Write(\"FOOTER\\0\\0\", 8);\n}\n\nstruct FileInfo {\n std::int64_t file_start_offset = 0;\n std::int64_t file_length = 0;\n};\n\nstruct ParsedFileInfo : FileInfo {\n std::string file_path;\n std::shared_ptr file;\n\n arrow::Status from_full_file(std::string in_file_path)\n {\n file_path = in_file_path;\n ARROW_ASSIGN_OR_RAISE(\n file, arrow::io::MemoryMappedFile::Open(in_file_path, arrow::io::FileMode::READ));\n file_start_offset = 0;\n ARROW_ASSIGN_OR_RAISE(file_length, file->GetSize());\n return arrow::Status::OK();\n }\n};\n\ninline pod5::Result write_footer_flatbuffer(\n std::shared_ptr const & sink,\n Uuid const & file_identifier,\n std::string const & software_name,\n FileInfo const & signal_table,\n FileInfo const & run_info_table,\n FileInfo const & reads_table)\n{\n flatbuffers::FlatBufferBuilder builder(1024);\n\n auto signal_file = Minknow::ReadsFormat::CreateEmbeddedFile(\n builder,\n signal_table.file_start_offset,\n signal_table.file_length,\n Minknow::ReadsFormat::Format_FeatherV2,\n Minknow::ReadsFormat::ContentType_SignalTable);\n\n auto run_info_file = Minknow::ReadsFormat::CreateEmbeddedFile(\n builder,\n run_info_table.file_start_offset,\n run_info_table.file_length,\n Minknow::ReadsFormat::Format_FeatherV2,\n Minknow::ReadsFormat::ContentType_RunInfoTable);\n\n auto reads_file = Minknow::ReadsFormat::CreateEmbeddedFile(\n builder,\n reads_table.file_start_offset,\n reads_table.file_length,\n Minknow::ReadsFormat::Format_FeatherV2,\n Minknow::ReadsFormat::ContentType_ReadsTable);\n\n std::vector> const files{\n signal_file, run_info_file, reads_file};\n auto footer = Minknow::ReadsFormat::CreateFooterDirect(\n builder,\n to_string(file_identifier).c_str(),\n software_name.c_str(),\n Pod5Version.c_str(),\n &files);\n\n builder.Finish(footer);\n ARROW_RETURN_NOT_OK(sink->Write(builder.GetBufferPointer(), builder.GetSize()));\n return builder.GetSize();\n}\n\ninline pod5::Status write_footer(\n std::shared_ptr const & sink,\n Uuid const & section_marker,\n Uuid const & file_identifier,\n std::string const & software_name,\n FileInfo const & signal_table,\n FileInfo const & run_info_table,\n FileInfo const & reads_table)\n{\n ARROW_RETURN_NOT_OK(write_footer_magic(sink));\n ARROW_ASSIGN_OR_RAISE(\n std::int64_t length,\n write_footer_flatbuffer(\n sink, file_identifier, software_name, signal_table, run_info_table, reads_table));\n ARROW_RETURN_NOT_OK(pad_file(sink, 8));\n\n std::int64_t paded_flatbuffer_size = arrow::bit_util::ToLittleEndian(length);\n ARROW_RETURN_NOT_OK(sink->Write(&paded_flatbuffer_size, sizeof(paded_flatbuffer_size)));\n\n ARROW_RETURN_NOT_OK(write_section_marker(sink, section_marker));\n return write_file_signature(sink);\n}\n\nstruct ParsedFooter {\n Uuid file_identifier;\n std::string software_name;\n std::string writer_pod5_version;\n\n ParsedFileInfo run_info_table;\n ParsedFileInfo reads_table;\n ParsedFileInfo signal_table;\n};\n\ninline void bind_footer_file(\n ParsedFooter & footer,\n std::shared_ptr const & file)\n{\n footer.reads_table.file = file;\n footer.run_info_table.file = file;\n footer.signal_table.file = file;\n}\n\ninline pod5::Status check_signature(\n std::shared_ptr const & file,\n std::int64_t offset_in_file)\n{\n std::array read_signature;\n ARROW_ASSIGN_OR_RAISE(\n auto read_bytes,\n file->ReadAt(offset_in_file, read_signature.size(), read_signature.data()));\n if (read_bytes != (std::int16_t)read_signature.size() || read_signature != FILE_SIGNATURE) {\n return arrow::Status::IOError(\"Invalid signature in file\");\n }\n\n return arrow::Status::OK();\n}\n\ninline pod5::Result read_footer_flatbuffer(\n std::vector const & footer_data)\n{\n auto verifier = flatbuffers::Verifier(footer_data.data(), footer_data.size());\n if (!verifier.VerifyBuffer()) {\n return arrow::Status::IOError(\"Invalid footer found in file\");\n }\n return flatbuffers::GetRoot(footer_data.data());\n}\n\ninline pod5::Result read_footer(\n std::string const & file_path,\n std::shared_ptr const & file)\n{\n // Verify signature at start and end of file:\n ARROW_RETURN_NOT_OK(check_signature(file, 0));\n ARROW_ASSIGN_OR_RAISE(auto const file_size, file->GetSize());\n ARROW_RETURN_NOT_OK(check_signature(file, file_size - FILE_SIGNATURE.size()));\n\n auto footer_length_data_end = file_size;\n footer_length_data_end -= FILE_SIGNATURE.size();\n footer_length_data_end -= sizeof(Uuid);\n\n std::int64_t footer_length = 0;\n ARROW_RETURN_NOT_OK(file->ReadAt(\n footer_length_data_end - sizeof(footer_length), sizeof(footer_length), &footer_length));\n footer_length = arrow::bit_util::FromLittleEndian(footer_length);\n if (footer_length < 0\n || static_cast(footer_length) > footer_length_data_end - sizeof(footer_length))\n {\n return arrow::Status::IOError(\"Invalid footer length\");\n }\n\n std::vector footer_data;\n footer_data.resize(footer_length);\n ARROW_ASSIGN_OR_RAISE(\n auto read_bytes,\n file->ReadAt(\n footer_length_data_end - sizeof(footer_length) - footer_length,\n footer_length,\n footer_data.data()));\n if (read_bytes != footer_length) {\n return arrow::Status::IOError(\"Failed to read footer data\");\n }\n ARROW_ASSIGN_OR_RAISE(auto fb_footer, read_footer_flatbuffer(footer_data));\n\n ParsedFooter footer;\n if (!fb_footer->file_identifier()) {\n return arrow::Status::IOError(\"Invalid footer file_identifier\");\n }\n auto const identifier = Uuid::from_string(fb_footer->file_identifier()->str());\n if (!identifier) {\n return Status::IOError(\n \"Invalid file_identifier in file: '\", fb_footer->file_identifier()->str(), \"'\");\n }\n footer.file_identifier = *identifier;\n\n if (!fb_footer->software()) {\n return arrow::Status::IOError(\"Invalid footer software\");\n }\n footer.software_name = fb_footer->software()->str();\n\n if (!fb_footer->pod5_version()) {\n return arrow::Status::IOError(\"Invalid footer pod5_version\");\n }\n footer.writer_pod5_version = fb_footer->pod5_version()->str();\n\n if (!fb_footer->contents()) {\n return arrow::Status::IOError(\"Invalid footer contents\");\n }\n for (auto const embedded_file : *fb_footer->contents()) {\n if (embedded_file->format() != Minknow::ReadsFormat::Format_FeatherV2) {\n return arrow::Status::IOError(\"Invalid embedded file format\");\n }\n switch (embedded_file->content_type()) {\n case Minknow::ReadsFormat::ContentType_RunInfoTable:\n footer.run_info_table.file_start_offset = embedded_file->offset();\n footer.run_info_table.file_length = embedded_file->length();\n footer.run_info_table.file = file;\n footer.run_info_table.file_path = file_path;\n break;\n case Minknow::ReadsFormat::ContentType_ReadsTable:\n footer.reads_table.file_start_offset = embedded_file->offset();\n footer.reads_table.file_length = embedded_file->length();\n footer.reads_table.file = file;\n footer.reads_table.file_path = file_path;\n break;\n case Minknow::ReadsFormat::ContentType_SignalTable:\n footer.signal_table.file_start_offset = embedded_file->offset();\n footer.signal_table.file_length = embedded_file->length();\n footer.signal_table.file = file;\n footer.signal_table.file_path = file_path;\n break;\n\n default:\n return arrow::Status::IOError(\"Unknown embedded file type\");\n }\n }\n\n return footer;\n}\n\nclass SubFile : public arrow::io::internal::RandomAccessFileConcurrencyWrapper {\npublic:\n SubFile(\n std::shared_ptr main_file,\n std::int64_t sub_file_offset,\n std::int64_t sub_file_length)\n : m_file(std::move(main_file))\n , m_sub_file_offset(sub_file_offset)\n , m_sub_file_length(sub_file_length)\n {\n }\n\nprotected:\n arrow::Status DoClose() { return m_file->Close(); }\n\n bool closed() const override { return m_file->closed(); }\n\n arrow::Result DoTell() const\n {\n ARROW_ASSIGN_OR_RAISE(auto t, m_file->Tell());\n return t - m_sub_file_offset;\n }\n\n arrow::Status DoSeek(int64_t offset)\n {\n if (offset < 0 || offset > m_sub_file_length) {\n return arrow::Status::IOError(\"Invalid offset into SubFile\");\n }\n offset += m_sub_file_offset;\n return m_file->Seek(offset);\n }\n\n arrow::Result DoRead(int64_t length, void * data)\n {\n ARROW_ASSIGN_OR_RAISE(auto pos, m_file->Tell());\n int64_t const remaining = m_sub_file_offset + m_sub_file_length - pos;\n length = std::min(remaining, length);\n return m_file->Read(length, data);\n }\n\n arrow::Result> DoRead(int64_t length)\n {\n ARROW_ASSIGN_OR_RAISE(auto pos, m_file->Tell());\n int64_t const remaining = m_sub_file_offset + m_sub_file_length - pos;\n length = std::min(remaining, length);\n return m_file->Read(length);\n }\n\n Result DoReadAt(int64_t position, int64_t nbytes, void * out)\n {\n if (position < 0 || position > m_sub_file_length) {\n return arrow::Status::IOError(\"Invalid offset into SubFile\");\n }\n int64_t const remaining = m_sub_file_length - position;\n nbytes = std::min(nbytes, remaining);\n return m_file->ReadAt(position + m_sub_file_offset, nbytes, out);\n }\n\n Result> DoReadAt(int64_t position, int64_t nbytes)\n {\n if (position < 0 || position > m_sub_file_length) {\n return arrow::Status::IOError(\"Invalid offset into SubFile\");\n }\n int64_t const remaining = m_sub_file_length - position;\n nbytes = std::min(nbytes, remaining);\n return m_file->ReadAt(position + m_sub_file_offset, nbytes);\n }\n\n arrow::Result DoGetSize() { return m_sub_file_length; }\n\nprivate:\n friend RandomAccessFileConcurrencyWrapper;\n\n std::shared_ptr m_file;\n std::int64_t m_sub_file_offset;\n std::int64_t m_sub_file_length;\n};\n\ninline arrow::Result> open_sub_file(ParsedFileInfo file_info)\n{\n if (!file_info.file) {\n return arrow::Status::Invalid(\"Failed to open file from footer\");\n }\n ARROW_ASSIGN_OR_RAISE(auto file_size, file_info.file->GetSize());\n if (file_info.file_length < 0 || file_info.file_length > file_size\n || file_info.file_start_offset > file_size - file_info.file_length)\n {\n return arrow::Status::Invalid(\"Bad footer info\");\n }\n // Restrict our open file to just the run info section:\n auto sub_file = std::make_shared(\n file_info.file, file_info.file_start_offset, file_info.file_length);\n ARROW_RETURN_NOT_OK(sub_file->Seek(0));\n return sub_file;\n}\n\ninline arrow::Result> open_sub_file(\n std::shared_ptr const & file,\n std::size_t offset)\n{\n if (!file) {\n return arrow::Status::Invalid(\"Failed to open file from footer\");\n }\n\n ARROW_ASSIGN_OR_RAISE(auto file_size, file->GetSize());\n\n // Restrict our open file to just the run info section:\n auto sub_file = std::make_shared(file, offset, file_size - offset);\n ARROW_RETURN_NOT_OK(sub_file->Seek(0));\n return sub_file;\n}\n\nenum class SubFileCleanup { CleanupOriginalFile, LeaveOrignalFile };\n\ninline arrow::Result write_file(\n arrow::MemoryPool * pool,\n std::shared_ptr const & file,\n FileLocation const & file_location,\n SubFileCleanup cleanup_mode)\n{\n combined_file_utils::FileInfo table_data;\n // Record file start location in bytes within the main file:\n ARROW_ASSIGN_OR_RAISE(table_data.file_start_offset, file->Tell());\n\n {\n // Stream out the reads table into the main file:\n ARROW_ASSIGN_OR_RAISE(\n auto reads_table_file_in, arrow::io::ReadableFile::Open(file_location.file_path, pool));\n ARROW_RETURN_NOT_OK(reads_table_file_in->Seek(file_location.offset));\n std::int64_t copied_bytes = 0;\n std::int64_t target_chunk_size = 10 * 1024 * 1024; // Read in 10MB of data at a time\n while (copied_bytes < std::int64_t(file_location.size)) {\n std::size_t const to_read =\n std::min(file_location.size - copied_bytes, target_chunk_size);\n ARROW_ASSIGN_OR_RAISE(auto const read_buffer, reads_table_file_in->Read(to_read));\n copied_bytes += read_buffer->size();\n ARROW_RETURN_NOT_OK(file->Write(read_buffer));\n }\n\n // Store the reads file length for later reading:\n ARROW_ASSIGN_OR_RAISE(table_data.file_length, file->Tell());\n table_data.file_length -= table_data.file_start_offset;\n }\n\n if (cleanup_mode == SubFileCleanup::CleanupOriginalFile) {\n // Clean up the tmp read path:\n ARROW_ASSIGN_OR_RAISE(\n auto arrow_path,\n ::arrow::internal::PlatformFilename::FromString(file_location.file_path));\n ARROW_RETURN_NOT_OK(arrow::internal::DeleteFile(arrow_path));\n }\n\n return table_data;\n}\n\ninline arrow::Result write_file_and_marker(\n arrow::MemoryPool * pool,\n std::shared_ptr const & file,\n FileLocation const & file_location,\n SubFileCleanup cleanup_mode,\n Uuid const & section_marker)\n{\n ARROW_ASSIGN_OR_RAISE(auto file_info, write_file(pool, file, file_location, cleanup_mode));\n // Pad file to 8 bytes and mark section:\n ARROW_RETURN_NOT_OK(combined_file_utils::pad_file(file, 8));\n ARROW_RETURN_NOT_OK(combined_file_utils::write_section_marker(file, section_marker));\n return file_info;\n}\n\n}} // namespace pod5::combined_file_utils\n" }, { "path": "c++/pod5_format/internal/linux_output_stream.h", "content": "#pragma once\n\n#include \"pod5_format/file_output_stream.h\"\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"pod5_format/io_manager.h\"\n\n#include \n#include \n#include \n\n#include \n#include \n\n#ifdef __linux__\n#include \n#include \n#endif\n\nnamespace pod5 {\n\nnamespace {\nconstexpr size_t fallocate_chunk = 50 * 256 * IOManager::Alignment; // 50MB\n} // namespace\n\n#ifdef __linux__\nclass LinuxOutputStream : public FileOutputStream {\n struct PrivateDummy {};\n\npublic:\n static arrow::Result> make(\n std::string const & file_path,\n std::shared_ptr const & io_manager,\n std::size_t write_chunk_size,\n bool use_directio,\n bool use_syncio,\n bool flush_on_batch_complete,\n bool keep_file_open = true)\n {\n auto flags = O_RDWR | O_CREAT;\n if (use_directio) {\n flags |= O_DIRECT;\n }\n\n if (use_syncio) {\n flags |= O_SYNC;\n }\n\n auto const initial_file_descriptor = open(file_path.c_str(), flags, 0644);\n if (initial_file_descriptor < 0) {\n return arrow::Status::Invalid(\"Failed to open file\");\n }\n\n return std::make_shared(\n file_path,\n initial_file_descriptor,\n flags,\n io_manager,\n write_chunk_size,\n keep_file_open,\n flush_on_batch_complete,\n PrivateDummy{});\n }\n\n ~LinuxOutputStream() { (void)Close(); }\n\n arrow::Status Close() override\n {\n // flush all output\n ARROW_RETURN_NOT_OK(Flush());\n\n while (!m_queued_writes.empty()) {\n ARROW_RETURN_NOT_OK(process_queued_writes());\n\n if (!m_queued_writes.empty()) {\n ARROW_RETURN_NOT_OK(m_io_manager->wait_for_event(std::chrono::seconds(1)));\n }\n }\n\n std::lock_guard l{m_file_handle_mutex};\n ARROW_ASSIGN_OR_RAISE(auto const file_descriptor, get_or_open_fd(l));\n\n // truncate excess data\n if (::ftruncate(file_descriptor, m_bytes_written) < 0) {\n return arrow::Status::IOError(\"Failed to truncate file\");\n }\n\n // and close stream\n return close_fd(l, true);\n }\n\n arrow::Future<> CloseAsync() override { return Close(); }\n\n arrow::Status Abort() override\n {\n std::lock_guard l{m_file_handle_mutex};\n return close_fd(l, true);\n }\n\n arrow::Result Tell() const override { return m_bytes_written - m_file_start_offset; }\n\n bool closed() const override { return m_file_descriptor == -1; }\n\n arrow::Status Write(void const * data, int64_t nbytes) override\n {\n ARROW_RETURN_NOT_OK(allocate_file_space(nbytes));\n\n auto remaining_data = gsl::make_span(reinterpret_cast(data), nbytes);\n while (!remaining_data.empty()) {\n ARROW_ASSIGN_OR_RAISE(\n remaining_data, m_aligned_buffer.consume_until_full(remaining_data));\n\n if (m_aligned_buffer.is_full()) {\n ARROW_RETURN_NOT_OK(flush_writes(FlushMode::AlignedWrites));\n }\n }\n\n m_bytes_written += nbytes;\n\n return arrow::Status::OK();\n }\n\n arrow::Status Write(std::shared_ptr const & data) override\n {\n ARROW_RETURN_NOT_OK(allocate_file_space(data->size()));\n\n auto remaining_data = gsl::make_span(data->data(), data->size());\n while (!remaining_data.empty()) {\n ARROW_ASSIGN_OR_RAISE(\n remaining_data, m_aligned_buffer.consume_until_full(remaining_data));\n\n if (m_aligned_buffer.is_full()) {\n ARROW_RETURN_NOT_OK(flush_writes(FlushMode::AlignedWrites));\n }\n }\n\n m_bytes_written += data->size();\n\n return arrow::Status::OK();\n }\n\n arrow::Status batch_complete() override\n {\n if (m_flush_on_batch_complete) {\n return flush_writes(FlushMode::AllWrites);\n }\n return arrow::Status::OK();\n }\n\n arrow::Status Flush() override\n {\n ARROW_RETURN_NOT_OK(flush_writes(FlushMode::AllWrites));\n\n std::lock_guard l{m_file_handle_mutex};\n if (m_file_descriptor < 0) {\n return arrow::Status::OK();\n }\n\n if (fsync(m_file_descriptor) < 0) {\n return arrow::Status::IOError(\"Error flushing file\");\n }\n\n return arrow::Status::OK();\n }\n\n void set_file_start_offset(std::size_t val) override { m_file_start_offset = val; }\n\n void set_flush_on_batch_complete(bool flush_on_batch_complete)\n {\n m_flush_on_batch_complete = flush_on_batch_complete;\n }\n\n LinuxOutputStream(\n std::string const & file_path,\n int initial_file_descriptor,\n int flags,\n std::shared_ptr const & io_manager,\n std::size_t write_chunk_size,\n bool keep_file_open,\n bool flush_on_batch_complete,\n PrivateDummy)\n : m_file_path(file_path)\n , m_flags(flags)\n , m_file_descriptor(initial_file_descriptor)\n , m_keep_file_open(keep_file_open)\n , m_aligned_buffer(write_chunk_size, io_manager)\n , m_io_manager(io_manager)\n , m_flush_on_batch_complete(flush_on_batch_complete)\n {\n if (!m_keep_file_open) {\n close(m_file_descriptor);\n m_file_descriptor = -1;\n }\n }\n\nprotected:\n enum class FlushMode { AllWrites, AlignedWrites };\n\n class AlignedBuffer {\n public:\n AlignedBuffer(std::size_t capacity, std::shared_ptr const & io_manager)\n : m_io_manager(io_manager)\n , m_capacity(capacity)\n {\n }\n\n // Copy input span to the end of the buffer until this buffer is full.\n //\n // Return any remaining buffer.\n arrow::Result> consume_until_full(\n gsl::span input)\n {\n ARROW_RETURN_NOT_OK(ensure_next_write());\n\n auto & buffer = m_next_write->get_buffer();\n assert((std::size_t)buffer.capacity() >= m_capacity);\n auto const remaining_buffer_bytes = buffer.capacity() - buffer.size();\n auto const to_copy = std::min(input.size(), (std::size_t)remaining_buffer_bytes);\n\n std::copy(\n input.begin(), input.begin() + to_copy, buffer.mutable_data() + buffer.size());\n ARROW_RETURN_NOT_OK(buffer.Resize(buffer.size() + to_copy, false));\n\n return input.subspan(to_copy);\n }\n\n // Find if the buffer is full (m_size == m_capacity)\n bool is_full() const\n {\n if (!m_next_write) {\n return false;\n }\n\n return (std::size_t)m_next_write->get_buffer().size() == m_capacity;\n }\n\n arrow::Result> release_all_writes_and_align(\n std::size_t * out_aligned_write_size)\n {\n ARROW_RETURN_NOT_OK(ensure_next_write());\n\n *out_aligned_write_size = aligned_write_size(m_next_write->get_buffer().size());\n\n auto result_write = std::move(m_next_write);\n auto & result_write_buffer = result_write->get_buffer();\n ARROW_ASSIGN_OR_RAISE(m_next_write, m_io_manager->allocate_new_write(m_capacity));\n auto & next_write_buffer = m_next_write->get_buffer();\n std::copy(\n result_write_buffer.data() + *out_aligned_write_size,\n result_write_buffer.data() + result_write_buffer.size(),\n next_write_buffer.mutable_data());\n ARROW_RETURN_NOT_OK(next_write_buffer.Resize(\n result_write_buffer.size() - *out_aligned_write_size, false));\n\n // Ensure the result write buffer is aligned to our write alignment.\n auto const result_unaligned_size = result_write_buffer.size();\n auto result_aligned_size =\n result_unaligned_size + (-result_unaligned_size & (IOManager::Alignment - 1));\n ARROW_RETURN_NOT_OK(result_write_buffer.Resize(result_aligned_size, false));\n assert(result_write_buffer.size() % IOManager::Alignment == 0);\n\n result_write->set_state(QueuedWrite::WriteState::ReadyForWrite);\n return result_write;\n }\n\n arrow::Result> release_aligned_writes()\n {\n ARROW_RETURN_NOT_OK(ensure_next_write());\n\n auto result_write = std::move(m_next_write);\n auto & result_write_buffer = result_write->get_buffer();\n ARROW_ASSIGN_OR_RAISE(m_next_write, m_io_manager->allocate_new_write(m_capacity));\n auto & next_write_buffer = m_next_write->get_buffer();\n\n auto aligned_size = aligned_write_size(result_write_buffer.size());\n std::copy(\n result_write_buffer.data() + aligned_size,\n result_write_buffer.data() + result_write_buffer.size(),\n next_write_buffer.mutable_data());\n ARROW_RETURN_NOT_OK(result_write_buffer.Resize(aligned_size, false));\n ARROW_RETURN_NOT_OK(\n next_write_buffer.Resize(result_write_buffer.size() - aligned_size, false));\n\n result_write->set_state(QueuedWrite::WriteState::ReadyForWrite);\n return result_write;\n }\n\n private:\n std::size_t aligned_write_size(std::size_t input_size) const\n {\n return (input_size / IOManager::Alignment) * IOManager::Alignment;\n }\n\n arrow::Status ensure_next_write()\n {\n if (m_next_write) {\n return arrow::Status::OK();\n }\n\n ARROW_ASSIGN_OR_RAISE(m_next_write, m_io_manager->allocate_new_write(m_capacity));\n assert((std::size_t)m_next_write->get_buffer().capacity() >= m_capacity);\n return arrow::Status::OK();\n }\n\n std::shared_ptr m_next_write;\n std::shared_ptr m_io_manager;\n std::size_t m_capacity;\n };\n\n arrow::Result get_or_open_fd([[maybe_unused]] std::lock_guard & lock)\n {\n if (m_file_descriptor >= 0) {\n return m_file_descriptor;\n }\n\n m_file_descriptor = open(m_file_path.c_str(), m_flags, 0644);\n if (m_file_descriptor < 0) {\n return arrow::Status::IOError(\"Failed to open file for writing\");\n }\n return m_file_descriptor;\n }\n\n arrow::Status close_fd([[maybe_unused]] std::lock_guard & lock, bool force = false)\n {\n if (m_keep_file_open && !force) {\n return arrow::Status::OK();\n }\n\n if (close(m_file_descriptor) != 0) {\n return arrow::Status::IOError(\"Error closing file\");\n }\n m_file_descriptor = -1;\n return arrow::Status::OK();\n }\n\n arrow::Status flush_writes(FlushMode flush_mode)\n {\n std::size_t write_offset{};\n std::shared_ptr released_data;\n\n if (flush_mode == FlushMode::AllWrites) {\n std::size_t aligned_write_size = 0;\n ARROW_ASSIGN_OR_RAISE(\n released_data, m_aligned_buffer.release_all_writes_and_align(&aligned_write_size));\n write_offset = m_bytes_submitted_to_manager;\n m_bytes_submitted_to_manager += aligned_write_size;\n } else if (flush_mode == FlushMode::AlignedWrites) {\n ARROW_ASSIGN_OR_RAISE(released_data, m_aligned_buffer.release_aligned_writes());\n write_offset = m_bytes_submitted_to_manager;\n m_bytes_submitted_to_manager += released_data->get_buffer().size();\n } else {\n assert(false);\n return arrow::Status::Invalid(\"Invalid FlushMode Passed.\");\n }\n\n assert(released_data->get_buffer().size() % IOManager::Alignment == 0);\n\n if (released_data->get_buffer().size() == 0) {\n return arrow::Status::OK();\n }\n\n std::lock_guard lock(m_file_handle_mutex);\n ARROW_ASSIGN_OR_RAISE(auto const file_descriptor, get_or_open_fd(lock));\n released_data->prepare_for_write(file_descriptor, write_offset);\n\n m_queued_writes.emplace_back(released_data);\n ARROW_RETURN_NOT_OK(m_io_manager->write_buffer(std::move(released_data)));\n\n ARROW_RETURN_NOT_OK(process_queued_writes());\n\n if (m_queued_writes.empty()) {\n return close_fd(lock);\n } else {\n // If we have queued writes, we keep the file open.\n return arrow::Status::OK();\n }\n }\n\n arrow::Status process_queued_writes()\n {\n for (auto it = m_queued_writes.begin(); it != m_queued_writes.end();) {\n if ((*it)->state() == QueuedWrite::WriteState::Completed) {\n ARROW_RETURN_NOT_OK(m_io_manager->return_used_write(std::move(*it)));\n it = m_queued_writes.erase(it);\n } else {\n ++it;\n }\n }\n\n return arrow::Status::OK();\n }\n\n arrow::Status allocate_file_space(std::size_t new_write_size)\n {\n auto new_total_size = m_bytes_written + new_write_size;\n if (new_total_size > m_fallocate_offset) {\n // reserve more space before continuing\n m_fallocate_offset += fallocate_chunk;\n\n std::lock_guard lock(m_file_handle_mutex);\n ARROW_ASSIGN_OR_RAISE(auto const file_descriptor, get_or_open_fd(lock));\n\n // If this fails, we will just write less optimially, so we ignore the result.\n ::fallocate(file_descriptor, 0, m_fallocate_offset, fallocate_chunk);\n }\n\n return arrow::Status::OK();\n }\n\n std::string m_file_path;\n int m_flags;\n std::mutex m_file_handle_mutex;\n int m_file_descriptor;\n bool m_keep_file_open{false};\n\n AlignedBuffer m_aligned_buffer;\n std::vector> m_queued_writes;\n std::shared_ptr m_io_manager;\n std::size_t m_fallocate_offset{0};\n std::size_t m_file_start_offset{0};\n std::size_t m_bytes_written{0};\n std::size_t m_bytes_submitted_to_manager{0};\n bool m_flush_on_batch_complete;\n};\n#endif\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/internal/tracing/tracing.h", "content": "#pragma once\n\n#define POD5_TRACE_FUNCTION()\n" }, { "path": "c++/pod5_format/io_manager.cpp", "content": "#include \"pod5_format/io_manager.h\"\n\n#ifdef __linux__\n#include \n#endif\n\nnamespace pod5 {\n\n#ifdef __linux__\nclass IOManagerSyncImpl : public IOManager {\npublic:\n IOManagerSyncImpl(arrow::MemoryPool * memory_pool) : m_memory_pool(memory_pool) {}\n\n arrow::Result> allocate_new_write(std::size_t capacity) override\n {\n if (m_queued_writes.size()) {\n auto new_write = m_queued_writes.back();\n m_queued_writes.pop_back();\n ARROW_RETURN_NOT_OK(new_write->reset_queued_write());\n ARROW_RETURN_NOT_OK(new_write->get_buffer().Reserve(capacity));\n assert((std::size_t)new_write->get_buffer().capacity() >= capacity);\n return new_write;\n }\n\n ARROW_ASSIGN_OR_RAISE(\n std::unique_ptr buffer,\n arrow::AllocateResizableBuffer(capacity, IOManager::Alignment, m_memory_pool));\n ARROW_RETURN_NOT_OK(buffer->Resize(0, false));\n assert((std::size_t)buffer->capacity() >= capacity);\n assert((std::size_t)buffer->size() == 0);\n return std::make_shared(std::move(buffer));\n }\n\n arrow::Status return_used_write(std::shared_ptr && used_write) override\n {\n if (m_queued_writes.size() < CachedBufferCount) {\n m_queued_writes.push_back(std::move(used_write));\n }\n used_write.reset();\n return arrow::Status::OK();\n }\n\n arrow::Status write_buffer(std::shared_ptr && data) override\n {\n auto result = lseek(data->file_descriptor(), data->file_offset(), SEEK_SET);\n if (result < 0) {\n return arrow::Status::IOError(\"Error seeking in file\");\n }\n\n result =\n write(data->file_descriptor(), data->get_buffer().data(), data->get_buffer().size());\n if (result < 0) {\n return arrow::Status::IOError(\n \"Error writing to file: \",\n errno,\n \" desc: \",\n data->file_descriptor(),\n \" offset: \",\n data->file_offset(),\n \" size: \",\n data->get_buffer().size());\n }\n\n data->set_state(QueuedWrite::WriteState::Completed);\n\n return {};\n }\n\nprivate:\n arrow::MemoryPool * m_memory_pool;\n std::vector> m_queued_writes;\n};\n\narrow::Result> make_sync_io_manager(arrow::MemoryPool * memory_pool)\n{\n return std::make_shared(memory_pool);\n}\n#endif\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/io_manager.h", "content": "#pragma once\n\n#include \n#include \n#include \n\n#ifdef __linux__\n#include \n#endif\n\n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\n#ifdef __linux__\nclass QueuedWrite {\npublic:\n QueuedWrite() = default;\n\n QueuedWrite(std::unique_ptr && buffer) : m_buffer(std::move(buffer)) {}\n\n arrow::Status reset_queued_write()\n {\n assert(m_state != WriteState::ReadyForWrite);\n assert(m_state != WriteState::InFlight);\n m_iovec = {};\n m_state = WriteState::Empty;\n m_file_offset = -1;\n m_file_descriptor = -1;\n return m_buffer->Resize(0, false);\n }\n\n void prepare_for_write(int file_descriptor, std::uint64_t offset)\n {\n m_file_descriptor = file_descriptor;\n m_file_offset = offset;\n m_iovec = {.iov_base = m_buffer->mutable_data(), .iov_len = (std::size_t)m_buffer->size()};\n set_state(WriteState::ReadyForWrite);\n }\n\n arrow::ResizableBuffer & get_buffer() { return *m_buffer; }\n\n arrow::Buffer const & get_buffer() const { return *m_buffer; }\n\n int file_descriptor() const { return m_file_descriptor; }\n\n std::uint64_t file_offset() const { return m_file_offset; }\n\n iovec * get_iovec_for_buffer() { return &m_iovec; }\n\n enum class WriteState { Empty, ReadyForWrite, InFlight, Completed };\n\n WriteState state() const { return m_state; }\n\n void set_state(WriteState state) { m_state = state; }\n\nprivate:\n std::unique_ptr m_buffer;\n std::uint64_t m_file_offset{(std::uint64_t)-1};\n iovec m_iovec{};\n int m_file_descriptor{-1};\n WriteState m_state{WriteState::Empty};\n};\n#endif\n\nclass IOManager {\npublic:\n constexpr static size_t Alignment = 4096; // buffer alignment (for block devices)\n constexpr static size_t CachedBufferCount = 5;\n\n virtual ~IOManager() = default;\n\n#ifdef __linux__\n virtual arrow::Result> allocate_new_write(\n std::size_t capacity) = 0;\n virtual arrow::Status return_used_write(std::shared_ptr && used_write) = 0;\n\n virtual arrow::Status write_buffer(std::shared_ptr && data) = 0;\n\n virtual arrow::Status wait_for_event(std::chrono::nanoseconds timeout) { return {}; }\n#endif\n};\n\n#ifdef __linux__\narrow::Result> make_sync_io_manager(\n arrow::MemoryPool * memory_pool = arrow::default_memory_pool());\n#endif\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/memory_pool.cpp", "content": "#include \"memory_pool.h\"\n\n#ifdef _WIN32\n#include \n#elif !defined(__FreeBSD__)\n#include \n#endif\n\nnamespace {\n\n// Referenced from the jemalloc source:\n// https://github.com/jemalloc/jemalloc/blob/b82333fdec6e5833f88780fcf1fc50b799268e1b/src/pages.c#L596C1-L616C2\nsize_t os_page_detect(void)\n{\n#ifdef _WIN32\n SYSTEM_INFO si;\n GetSystemInfo(&si);\n return si.dwPageSize;\n#elif defined(__FreeBSD__)\n /*\n\t * This returns the value obtained from\n\t * the auxv vector, avoiding a syscall.\n\t */\n return getpagesize();\n#else\n long result = sysconf(_SC_PAGESIZE);\n if (result == -1) {\n return 4095 * 16; // Default to safe, large page size\n }\n return (size_t)result;\n#endif\n}\n\n} // namespace\n\nnamespace pod5 {\n\narrow::MemoryPool * default_memory_pool()\n{\n // Default to system memory pool for systems with large pages:\n if (os_page_detect() > 4096) {\n return arrow::system_memory_pool();\n }\n return arrow::default_memory_pool();\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/memory_pool.h", "content": "#pragma once\n#include \n\nnamespace pod5 {\n\n/// \\brief Find a memory pool that should be used by default when opening or creating a pod5 file.\n/// \\note This function differs from the arrow equivalent by not using jemalloc on systems with large\n/// pages, which jemalloc does not support.\narrow::MemoryPool * default_memory_pool();\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/migration.cpp", "content": "#include \"pod5_format/migration/migration.h\"\n\n#include \n\nnamespace pod5 {\n\nstatic bool registered_delete_at_exit_called = false;\nstd::vector registered_delete_at_exit_paths;\n\nvoid register_delete_at_exit(arrow::internal::PlatformFilename const & path)\n{\n registered_delete_at_exit_paths.push_back(path);\n\n if (!registered_delete_at_exit_called) {\n std::atexit([] {\n std::size_t delete_failed = 0;\n for (auto const & path : registered_delete_at_exit_paths) {\n auto result = ::arrow::internal::DeleteDirTree(path);\n if (!result.ok()) {\n delete_failed += 1;\n }\n }\n\n if (delete_failed > 0) {\n std::cerr << \"Warning: Failed to remove \" << delete_failed\n << \" temporary migration directories at exit.\\n\";\n }\n });\n registered_delete_at_exit_called = true;\n }\n}\n\nResult> MakeTmpDir(char const * suffix)\n{\n std::default_random_engine gen(\n static_cast(arrow::internal::GetRandomSeed()));\n\n for (std::uint32_t counter = 0; counter < 5; ++counter) {\n std::string tmp_path = std::string{\".tmp_\"} + suffix;\n\n tmp_path += \"_\" + std::to_string(gen());\n\n ARROW_ASSIGN_OR_RAISE(\n auto filename, arrow::internal::PlatformFilename::FromString(tmp_path));\n ARROW_ASSIGN_OR_RAISE(auto created, CreateDir(filename));\n if (created) {\n return std::make_unique(std::move(filename));\n }\n }\n\n return arrow::Status::Invalid(\"Failed to make temporary directory\");\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/migration.h", "content": "#pragma once\n\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_utils.h\"\n\n#include \n\n#include \n\nnamespace pod5 {\n\nvoid register_delete_at_exit(arrow::internal::PlatformFilename const & path);\n\nclass TemporaryDir {\npublic:\n TemporaryDir(arrow::internal::PlatformFilename && path) : m_path(path) {}\n\n ~TemporaryDir() { cleanup(); }\n\n arrow::internal::PlatformFilename const & path() { return m_path; };\n\n void cleanup()\n {\n if (m_path.ToString().empty()) {\n return;\n }\n\n auto result = ::arrow::internal::DeleteDirTree(m_path);\n if (!result.ok()) {\n // Push the delete of this directory off to exit, when all open file handles should be closed.\n register_delete_at_exit(m_path);\n } else {\n m_path = {};\n }\n }\n\nprivate:\n arrow::internal::PlatformFilename m_path;\n};\n\nResult> MakeTmpDir(char const * suffix);\n\nclass MigrationResult {\npublic:\n MigrationResult(combined_file_utils::ParsedFooter const & footer) : m_footer(footer) {}\n\n MigrationResult(MigrationResult &&) = default;\n MigrationResult & operator=(MigrationResult &&) = default;\n MigrationResult(MigrationResult const &) = delete;\n MigrationResult & operator=(MigrationResult const &) = delete;\n\n combined_file_utils::ParsedFooter & footer() { return m_footer; }\n\n combined_file_utils::ParsedFooter const & footer() const { return m_footer; }\n\n void add_temp_dir(std::unique_ptr && temp_dir)\n {\n m_temp_dirs.emplace_back(std::move(temp_dir));\n }\n\nprivate:\n // This is first so we clean it up last, after the\n // footer and any open files it contains is destroyed.\n std::vector> m_temp_dirs;\n combined_file_utils::ParsedFooter m_footer;\n};\n\narrow::Result migrate_v0_to_v1(\n MigrationResult && v0_input,\n arrow::MemoryPool * pool);\narrow::Result migrate_v1_to_v2(\n MigrationResult && v1_input,\n arrow::MemoryPool * pool);\narrow::Result migrate_v2_to_v3(\n MigrationResult && v2_input,\n arrow::MemoryPool * pool);\narrow::Result migrate_v3_to_v4(\n MigrationResult && v2_input,\n arrow::MemoryPool * pool);\n\ninline arrow::Result migrate_if_required(\n Version writer_version,\n combined_file_utils::ParsedFooter const & read_footer,\n std::shared_ptr const & source,\n arrow::MemoryPool * pool)\n{\n MigrationResult result{read_footer};\n\n if (writer_version < Version(0, 0, 24)) {\n // Added fields for read scaling\n ARROW_ASSIGN_OR_RAISE(result, migrate_v0_to_v1(std::move(result), pool));\n }\n\n if (writer_version < Version(0, 0, 32)) {\n // Added num samples field\n ARROW_ASSIGN_OR_RAISE(result, migrate_v1_to_v2(std::move(result), pool));\n }\n\n if (writer_version < Version(0, 0, 38)) {\n // Flattening fields\n ARROW_ASSIGN_OR_RAISE(result, migrate_v2_to_v3(std::move(result), pool));\n }\n if (writer_version < Version(0, 3, 30)) {\n // Flattening fields\n ARROW_ASSIGN_OR_RAISE(result, migrate_v3_to_v4(std::move(result), pool));\n }\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/migration_utils.h", "content": "#pragma once\n\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_metadata.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\ntemplate \narrow::Result>\nmake_filled_array(arrow::MemoryPool * pool, std::size_t row_count, U default_value)\n{\n // Minimal iterator to repeat the same value N times.\n // TODO: replace with std::views::repeat in C++23\n struct RepeatIter {\n // These are necessary to make |std::distance| and |std::copy| fast.\n using iterator_category [[maybe_unused]] = std::random_access_iterator_tag;\n using value_type = U const;\n using difference_type = std::int64_t;\n using pointer [[maybe_unused]] = value_type *;\n using reference = value_type &;\n\n std::size_t m_idx;\n value_type m_value;\n\n constexpr RepeatIter & operator++()\n {\n m_idx++;\n return *this;\n }\n\n constexpr RepeatIter operator++(int)\n {\n RepeatIter retval = *this;\n m_idx++;\n return retval;\n }\n\n constexpr bool operator==(RepeatIter const & other) const { return m_idx == other.m_idx; }\n\n constexpr bool operator!=(RepeatIter const & other) const { return !operator==(other); }\n\n constexpr difference_type operator-(RepeatIter const & other) const\n {\n return static_cast(m_idx) - static_cast(other.m_idx);\n }\n\n constexpr reference operator*() const { return m_value; }\n };\n\n RepeatIter iter_begin{0, default_value};\n RepeatIter iter_end{row_count, default_value};\n\n T builder(pool);\n ARROW_RETURN_NOT_OK(builder.AppendValues(iter_begin, iter_end));\n return builder.Finish();\n}\n\ninline arrow::Status set_column(\n std::shared_ptr const & schema,\n std::vector> & columns,\n char const * field_name,\n arrow::Result> const & array)\n{\n auto field_index = schema->GetFieldIndex(field_name);\n if (field_index == -1) {\n return arrow::Status::Invalid(\"Failed to find field '\", field_name, \"' during migration.\");\n }\n\n if (field_index >= (std::int64_t)columns.size()) {\n columns.resize(field_index + 1);\n }\n\n ARROW_ASSIGN_OR_RAISE(columns[field_index], array);\n\n return arrow::Status::OK();\n}\n\ninline arrow::Status copy_column(\n std::shared_ptr const & schema_a,\n std::vector> & columns_a,\n char const * field_name,\n std::shared_ptr const & schema_b,\n std::vector> & columns_b)\n{\n auto field_index_a = schema_a->GetFieldIndex(field_name);\n if (field_index_a == -1 || field_index_a >= (std::int64_t)columns_a.size()) {\n return arrow::Status::Invalid(\"Failed to find field '\", field_name, \"' during migration.\");\n }\n\n auto source_column = columns_a[field_index_a];\n\n auto field_index_b = schema_b->GetFieldIndex(field_name);\n if (field_index_b >= (std::int64_t)columns_b.size()) {\n columns_b.resize(field_index_b + 1);\n }\n\n columns_b[field_index_b] = source_column;\n\n return arrow::Status::OK();\n}\n\nstruct Pod5BatchRecordReader {\n std::shared_ptr reader;\n std::shared_ptr schema;\n std::shared_ptr metadata;\n};\n\nstruct Pod5BatchRecordWriter {\n std::shared_ptr writer;\n std::shared_ptr schema;\n\n arrow::Status write_batch(\n std::size_t num_rows,\n std::vector> const & columns)\n {\n auto const record_batch = arrow::RecordBatch::Make(schema, num_rows, std::move(columns));\n return writer->WriteRecordBatch(*record_batch);\n }\n};\n\ninline pod5::Result open_record_batch_reader(\n arrow::MemoryPool * pool,\n combined_file_utils::ParsedFileInfo file_info)\n{\n Pod5BatchRecordReader result;\n ARROW_ASSIGN_OR_RAISE(auto file, open_sub_file(file_info));\n\n arrow::ipc::IpcReadOptions read_options;\n read_options.memory_pool = pool;\n ARROW_ASSIGN_OR_RAISE(\n result.reader, arrow::ipc::RecordBatchFileReader::Open(file, read_options));\n\n result.schema = result.reader->schema();\n result.metadata = result.schema->metadata();\n if (!result.metadata) {\n return Status::IOError(\"Missing metadata on read table schema\");\n }\n\n return result;\n}\n\ninline pod5::Result> update_metadata(\n std::shared_ptr original_metadata,\n Version version_to_write)\n{\n auto result = original_metadata->Copy();\n // Update the reader for the new version:\n ARROW_RETURN_NOT_OK(result->Set(\"MINKNOW:pod5_version\", version_to_write.to_string()));\n return result;\n}\n\ninline pod5::Result make_record_batch_writer(\n arrow::MemoryPool * pool,\n std::string path,\n std::shared_ptr schema,\n std::shared_ptr metadata)\n{\n ARROW_ASSIGN_OR_RAISE(auto file, arrow::io::FileOutputStream::Open(path, false));\n arrow::ipc::IpcWriteOptions write_options;\n write_options.memory_pool = pool;\n write_options.emit_dictionary_deltas = true;\n\n Pod5BatchRecordWriter result;\n ARROW_ASSIGN_OR_RAISE(\n result.writer, arrow::ipc::MakeFileWriter(file, schema, write_options, metadata));\n result.schema = schema;\n\n return result;\n}\n\ninline pod5::Status check_columns(\n std::shared_ptr const & schema,\n std::vector> const & columns)\n{\n for (std::size_t i = 0; i < columns.size(); ++i) {\n auto const & column = columns[i];\n auto const & schema_field = schema->field(i);\n\n if (auto list = std::dynamic_pointer_cast(column)) {\n auto last_value = list->value_offset(0);\n for (int i = 1; i <= list->length(); ++i) {\n if (list->value_offset(i) < last_value) {\n return arrow::Status::Invalid(\n \"Field content for field `\",\n schema_field->name(),\n \"`, list offsets are invalid\"\n \" at row index \",\n i,\n \" (\",\n list->value_offset(i),\n \" < \",\n last_value,\n \")\");\n }\n last_value = list->value_offset(i);\n }\n } else if (auto dict = std::dynamic_pointer_cast(column)) {\n auto dict_values = dict->dictionary();\n auto string_dictionary_values =\n std::dynamic_pointer_cast(dict_values);\n if (string_dictionary_values) {\n auto const value_offsets = string_dictionary_values->value_offsets();\n std::int64_t const value_offsets_length =\n value_offsets->size() / sizeof(arrow::StringArray::offset_type);\n if (value_offsets_length != (1 + dict_values->length()))\n { // We expect N+1 offsets for the final element length\n return arrow::Status::Invalid(\n \"Dictionary length for field `\",\n schema_field->name(),\n \"`, dictionary length is \",\n dict_values->length(),\n \" but value offsets is length \",\n value_offsets_length);\n }\n }\n\n auto indices = std::dynamic_pointer_cast(dict->indices());\n if (!indices) {\n return arrow::Status::Invalid(\n \"Field content for field `\",\n schema_field->name(),\n \"`, dictionary indexes are missing\");\n }\n for (int i = 0; i < indices->length(); ++i) {\n if (indices->Value(i) >= dict_values->length()) {\n return arrow::Status::Invalid(\n \"Field content for field `\",\n schema_field->name(),\n \"`, dictionary indexes are invalid\"\n \" at row index \",\n i,\n \" (\",\n indices->Value(i),\n \" >= \",\n dict_values->length(),\n \")\");\n }\n }\n }\n }\n\n return {};\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/v0_to_v1.cpp", "content": "#include \"pod5_format/migration/migration.h\"\n#include \"pod5_format/migration/migration_utils.h\"\n#include \"pod5_format/table_reader.h\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\narrow::Result migrate_v0_to_v1(\n MigrationResult && v0_input,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(auto temp_dir, MakeTmpDir(\"pod5_v0_v1_migration\"));\n ARROW_ASSIGN_OR_RAISE(auto v1_reads_table_path, temp_dir->path().Join(\"reads_table.arrow\"));\n\n {\n ARROW_ASSIGN_OR_RAISE(\n auto v0_reader, open_record_batch_reader(pool, v0_input.footer().reads_table));\n\n auto v1_new_schama = arrow::schema(\n {arrow::field(\"num_minknow_events\", arrow::uint64()),\n arrow::field(\"tracked_scaling_scale\", arrow::float32()),\n arrow::field(\"tracked_scaling_shift\", arrow::float32()),\n arrow::field(\"predicted_scaling_scale\", arrow::float32()),\n arrow::field(\"predicted_scaling_shift\", arrow::float32()),\n arrow::field(\"num_reads_since_mux_change\", arrow::uint32()),\n arrow::field(\"time_since_mux_change\", arrow::float32())});\n\n ARROW_ASSIGN_OR_RAISE(\n auto v1_schema, arrow::UnifySchemas({v0_reader.schema, v1_new_schama}));\n\n ARROW_ASSIGN_OR_RAISE(\n auto new_metadata, update_metadata(v0_reader.metadata, Version(0, 0, 24)));\n ARROW_ASSIGN_OR_RAISE(\n auto v1_writer,\n make_record_batch_writer(\n pool, v1_reads_table_path.ToString(), v1_schema, new_metadata));\n\n for (std::int64_t batch_idx = 0; batch_idx < v0_reader.reader->num_record_batches();\n ++batch_idx)\n {\n // Read V0 data:\n ARROW_ASSIGN_OR_RAISE(\n auto v0_batch, ReadRecordBatchAndValidate(*v0_reader.reader, batch_idx));\n ARROW_RETURN_NOT_OK(v0_batch->ValidateFull());\n auto const num_rows = v0_batch->num_rows();\n\n if (num_rows < 0) {\n return arrow::Status::Invalid(\"Invalid number of rows\");\n } else if (POD5_ENABLE_FUZZERS && num_rows > 1'000'000) {\n return arrow::Status::Invalid(\"Skipping huge sizes when fuzzing\");\n }\n\n // Extend with V1 data:\n std::vector> columns = v0_batch->columns();\n ARROW_RETURN_NOT_OK(check_columns(v0_reader.schema, columns));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"num_minknow_events\",\n make_filled_array(pool, num_rows, 0)));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"tracked_scaling_scale\",\n make_filled_array(\n pool, num_rows, std::numeric_limits::quiet_NaN())));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"tracked_scaling_shift\",\n make_filled_array(\n pool, num_rows, std::numeric_limits::quiet_NaN())));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"predicted_scaling_scale\",\n make_filled_array(\n pool, num_rows, std::numeric_limits::quiet_NaN())));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"predicted_scaling_shift\",\n make_filled_array(\n pool, num_rows, std::numeric_limits::quiet_NaN())));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"num_reads_since_mux_change\",\n make_filled_array(pool, num_rows, 0)));\n ARROW_RETURN_NOT_OK(set_column(\n v1_schema,\n columns,\n \"time_since_mux_change\",\n make_filled_array(pool, num_rows, 0.0f)));\n ARROW_RETURN_NOT_OK(v1_writer.write_batch(num_rows, std::move(columns)));\n }\n\n ARROW_RETURN_NOT_OK(v1_writer.writer->Close());\n }\n\n // Set up migrated data to point at our new table:\n MigrationResult result = std::move(v0_input);\n ARROW_RETURN_NOT_OK(result.footer().reads_table.from_full_file(v1_reads_table_path.ToString()));\n result.add_temp_dir(std::move(temp_dir));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/v1_to_v2.cpp", "content": "#include \"pod5_format/migration/migration.h\"\n#include \"pod5_format/migration/migration_utils.h\"\n#include \"pod5_format/table_reader.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\narrow::Result get_num_samples(\n std::shared_ptr const & signal_col,\n std::size_t row_idx,\n std::vector> const & signal_batches)\n{\n if (signal_batches.empty()) {\n return 0;\n }\n\n std::size_t signal_batch_size = signal_batches[0]->num_rows();\n std::size_t num_samples = 0;\n\n auto values = std::dynamic_pointer_cast(signal_col->values());\n if (!values) {\n return arrow::Status::Invalid(\"Invalid signal column, potentially corrupt file.\");\n }\n\n auto offset = signal_col->value_offset(row_idx);\n for (std::int64_t index = 0; index < signal_col->value_length(row_idx); ++index) {\n auto const abs_index = offset + index;\n if (abs_index < 0 || abs_index >= values->length()) {\n return arrow::Status::Invalid(\"Invalid signal column, potentially corrupt file.\");\n }\n\n auto const abs_row = values->Value(abs_index);\n\n auto const batch_idx = abs_row / signal_batch_size;\n auto const batch_row = abs_row - (batch_idx * signal_batch_size);\n\n if (batch_idx >= signal_batches.size()) {\n return arrow::Status::Invalid(\n \"Invalid signal row \", abs_row, \", cannot find signal batch \", batch_idx);\n }\n\n auto batch = signal_batches[batch_idx];\n\n auto samples_column =\n std::dynamic_pointer_cast(batch->GetColumnByName(\"samples\"));\n if (!samples_column) {\n return arrow::Status::Invalid(\"`samples` column is missing from file\");\n }\n if (batch_row >= (std::size_t)samples_column->length()) {\n return arrow::Status::Invalid(\n \"Invalid signal batch row \", batch_row, \", length is \", samples_column->length());\n }\n num_samples += samples_column->Value(batch_row);\n }\n\n return num_samples;\n}\n\narrow::Result migrate_v1_to_v2(\n MigrationResult && v1_input,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(auto temp_dir, MakeTmpDir(\"pod5_v1_v2_migration\"));\n ARROW_ASSIGN_OR_RAISE(auto v2_reads_table_path, temp_dir->path().Join(\"reads_table.arrow\"));\n\n {\n ARROW_ASSIGN_OR_RAISE(\n auto v1_reader, open_record_batch_reader(pool, v1_input.footer().reads_table));\n ARROW_ASSIGN_OR_RAISE(\n auto v1_signal_reader, open_record_batch_reader(pool, v1_input.footer().signal_table));\n std::vector> signal_batches(\n v1_signal_reader.reader->num_record_batches());\n for (std::size_t batch_idx = 0;\n batch_idx < (std::size_t)v1_signal_reader.reader->num_record_batches();\n ++batch_idx)\n {\n ARROW_ASSIGN_OR_RAISE(\n signal_batches[batch_idx],\n ReadRecordBatchAndValidate(*v1_signal_reader.reader, batch_idx));\n ARROW_RETURN_NOT_OK(signal_batches[batch_idx]->ValidateFull());\n }\n\n auto v2_new_schama = arrow::schema({arrow::field(\"num_samples\", arrow::uint64())});\n ARROW_ASSIGN_OR_RAISE(\n auto new_metadata, update_metadata(v1_reader.metadata, Version(0, 0, 32)));\n ARROW_ASSIGN_OR_RAISE(\n auto v2_schema, arrow::UnifySchemas({v1_reader.schema, v2_new_schama}));\n ARROW_ASSIGN_OR_RAISE(\n auto v2_writer,\n make_record_batch_writer(\n pool, v2_reads_table_path.ToString(), v2_schema, new_metadata));\n\n for (std::int64_t batch_idx = 0; batch_idx < v1_reader.reader->num_record_batches();\n ++batch_idx)\n {\n // Read V1 data:\n ARROW_ASSIGN_OR_RAISE(\n auto v1_batch, ReadRecordBatchAndValidate(*v1_reader.reader, batch_idx));\n ARROW_RETURN_NOT_OK(v1_batch->ValidateFull());\n auto const num_rows = v1_batch->num_rows();\n\n // Extend with V2 data:\n std::vector> columns = v1_batch->columns();\n\n auto signal_column =\n std::dynamic_pointer_cast(v1_batch->GetColumnByName(\"signal\"));\n if (!signal_column) {\n return arrow::Status::Invalid(\"`signal` column is missing from file\");\n }\n ARROW_RETURN_NOT_OK(signal_column->ValidateFull());\n\n arrow::UInt64Builder num_samples_builder;\n for (std::int64_t row = 0; row < num_rows; ++row) {\n ARROW_ASSIGN_OR_RAISE(\n auto num_samples, get_num_samples(signal_column, row, signal_batches));\n ARROW_RETURN_NOT_OK(num_samples_builder.Append(num_samples));\n }\n ARROW_RETURN_NOT_OK(\n set_column(v2_schema, columns, \"num_samples\", num_samples_builder.Finish()));\n ARROW_RETURN_NOT_OK(v2_writer.write_batch(num_rows, std::move(columns)));\n }\n\n ARROW_RETURN_NOT_OK(v2_writer.writer->Close());\n }\n\n // Set up migrated data to point at our new table:\n MigrationResult result = std::move(v1_input);\n ARROW_RETURN_NOT_OK(result.footer().reads_table.from_full_file(v2_reads_table_path.ToString()));\n result.add_temp_dir(std::move(temp_dir));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/v2_to_v3.cpp", "content": "#include \"pod5_format/migration/migration.h\"\n#include \"pod5_format/migration/migration_utils.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nstruct StructRow {\n std::int64_t dict_item_index;\n std::shared_ptr data;\n};\n\nstruct StringDictBuilder {\n arrow::Int16Builder indices;\n arrow::StringBuilder items;\n\n arrow::Result> finish()\n {\n ARROW_ASSIGN_OR_RAISE(auto finished_indices, indices.Finish());\n ARROW_ASSIGN_OR_RAISE(auto finished_items, items.Finish());\n\n auto const & finished_items_val = static_cast(*finished_items);\n\n // Re append the finished items to the now blank list\n for (std::int64_t i = 0; i < finished_items_val.length(); ++i) {\n ARROW_RETURN_NOT_OK(items.Append(finished_items_val.GetView(i)));\n }\n\n return arrow::DictionaryArray::FromArrays(finished_indices, finished_items);\n }\n\n std::unordered_map lookup;\n};\n\narrow::Result get_dict_struct(\n std::shared_ptr const & batch,\n std::size_t row,\n char const * field_name)\n{\n auto column = batch->GetColumnByName(field_name);\n if (!column) {\n return Status::Invalid(\"Failed to find column \", field_name);\n }\n\n auto dict_column = std::dynamic_pointer_cast(column);\n if (!dict_column) {\n return Status::Invalid(\"Found column \", field_name, \" is not a dictionary as expected\");\n }\n\n auto dict_items = std::dynamic_pointer_cast(dict_column->dictionary());\n if (!dict_items) {\n return Status::Invalid(\"Dictionary column is not a struct as expected\");\n }\n\n return StructRow{dict_column->GetValueIndex(row), dict_items};\n}\n\ntemplate \narrow::Status\nappend_struct_row(StructRow const & struct_row, char const * field_name, Builder & builder)\n{\n auto field_array = struct_row.data->GetFieldByName(field_name);\n if (!field_array) {\n return Status::Invalid(\"Struct is missing \", field_name, \" field\");\n }\n\n auto typed_field_array = std::dynamic_pointer_cast(field_array);\n if (!typed_field_array) {\n return Status::Invalid(field_name, \" field is the wrong type\");\n }\n\n if (struct_row.dict_item_index < 0 || struct_row.dict_item_index >= field_array->length()) {\n return Status::Invalid(\"Dictionary index is out of range\");\n }\n return builder.Append(typed_field_array->Value(struct_row.dict_item_index));\n}\n\narrow::Status append_struct_row_to_dict(\n StructRow const & struct_row,\n char const * field_name,\n StringDictBuilder & builder)\n{\n auto field_array = struct_row.data->GetFieldByName(field_name);\n if (!field_array) {\n return Status::Invalid(\"Struct is missing \", field_name, \" field\");\n }\n\n auto typed_field_array = std::dynamic_pointer_cast(field_array);\n if (!typed_field_array) {\n return Status::Invalid(field_name, \" field is the wrong type\");\n }\n\n if (struct_row.dict_item_index < 0 || struct_row.dict_item_index >= field_array->length()) {\n return Status::Invalid(\"Dictionary index is out of range\");\n }\n\n auto str_value = typed_field_array->GetString(struct_row.dict_item_index);\n auto it = builder.lookup.find(str_value);\n if (it != builder.lookup.end()) {\n return builder.indices.Append(it->second);\n }\n\n auto index = builder.items.length();\n ARROW_RETURN_NOT_OK(builder.items.Append(str_value));\n builder.lookup[str_value] = index;\n return builder.indices.Append(index);\n}\n\narrow::Result migrate_v2_to_v3(\n MigrationResult && v2_input,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(auto temp_dir, MakeTmpDir(\"pod5_v2_v3_migration\"));\n ARROW_ASSIGN_OR_RAISE(auto v3_reads_table_path, temp_dir->path().Join(\"reads_table.arrow\"));\n ARROW_ASSIGN_OR_RAISE(\n auto v3_run_info_table_path, temp_dir->path().Join(\"run_info_table.arrow\"));\n\n {\n ARROW_ASSIGN_OR_RAISE(\n auto v2_reader, open_record_batch_reader(pool, v2_input.footer().reads_table));\n ARROW_ASSIGN_OR_RAISE(\n auto new_metadata, update_metadata(v2_reader.metadata, Version(0, 0, 35)));\n\n auto const num_record_batches = v2_reader.reader->num_record_batches();\n\n {\n auto v3_reads_schema = arrow::schema(\n {arrow::field(\"read_id\", uuid()),\n arrow::field(\"signal\", arrow::list(arrow::uint64())),\n arrow::field(\"read_number\", arrow::uint32()),\n arrow::field(\"start\", arrow::uint64()),\n arrow::field(\"median_before\", arrow::float32()),\n arrow::field(\"num_minknow_events\", arrow::uint64()),\n arrow::field(\"tracked_scaling_scale\", arrow::float32()),\n arrow::field(\"tracked_scaling_shift\", arrow::float32()),\n arrow::field(\"predicted_scaling_scale\", arrow::float32()),\n arrow::field(\"predicted_scaling_shift\", arrow::float32()),\n arrow::field(\"num_reads_since_mux_change\", arrow::uint32()),\n arrow::field(\"time_since_mux_change\", arrow::float32()),\n arrow::field(\"num_samples\", arrow::uint64()),\n arrow::field(\"channel\", arrow::uint16()),\n arrow::field(\"well\", arrow::uint8()),\n arrow::field(\"pore_type\", arrow::dictionary(arrow::int16(), arrow::utf8())),\n arrow::field(\"calibration_offset\", arrow::float32()),\n arrow::field(\"calibration_scale\", arrow::float32()),\n arrow::field(\"end_reason\", arrow::dictionary(arrow::int16(), arrow::utf8())),\n arrow::field(\"end_reason_forced\", arrow::boolean()),\n arrow::field(\"run_info\", arrow::dictionary(arrow::int16(), arrow::utf8()))},\n new_metadata);\n ARROW_ASSIGN_OR_RAISE(\n auto v3_reads_writer,\n make_record_batch_writer(\n pool, v3_reads_table_path.ToString(), v3_reads_schema, new_metadata));\n\n std::vector const columns_to_copy{\n \"read_id\",\n \"signal\",\n \"read_number\",\n \"start\",\n \"median_before\",\n \"num_minknow_events\",\n \"tracked_scaling_scale\",\n \"tracked_scaling_shift\",\n \"predicted_scaling_scale\",\n \"predicted_scaling_shift\",\n \"num_reads_since_mux_change\",\n \"time_since_mux_change\",\n \"num_samples\"};\n\n // Builders for dict columns\n StringDictBuilder pore_type;\n StringDictBuilder end_reason;\n StringDictBuilder run_info;\n for (std::int64_t batch_idx = 0; batch_idx < num_record_batches; ++batch_idx) {\n // Read V2 data:\n ARROW_ASSIGN_OR_RAISE(auto v2_batch, v2_reader.reader->ReadRecordBatch(batch_idx));\n ARROW_RETURN_NOT_OK(v2_batch->ValidateFull());\n auto const num_rows = v2_batch->num_rows();\n\n std::vector> v3_columns;\n\n // Write V3 data:\n std::vector> v2_columns = v2_batch->columns();\n for (auto const & col_name : columns_to_copy) {\n ARROW_RETURN_NOT_OK(copy_column(\n v2_reader.schema,\n v2_columns,\n col_name.data(),\n v3_reads_schema,\n v3_columns));\n }\n\n arrow::UInt16Builder channel;\n arrow::UInt8Builder well;\n arrow::FloatBuilder calibration_offset;\n arrow::FloatBuilder calibration_scale;\n arrow::BooleanBuilder end_reason_forced;\n for (std::int64_t row = 0; row < num_rows; ++row) {\n ARROW_ASSIGN_OR_RAISE(\n auto calibration_data, get_dict_struct(v2_batch, row, \"calibration\"));\n ARROW_RETURN_NOT_OK(\n append_struct_row(\n calibration_data, \"offset\", calibration_offset));\n ARROW_RETURN_NOT_OK(\n append_struct_row(\n calibration_data, \"scale\", calibration_scale));\n\n ARROW_ASSIGN_OR_RAISE(auto pore_data, get_dict_struct(v2_batch, row, \"pore\"));\n ARROW_RETURN_NOT_OK(\n append_struct_row(pore_data, \"channel\", channel));\n ARROW_RETURN_NOT_OK(\n append_struct_row(pore_data, \"well\", well));\n ARROW_RETURN_NOT_OK(\n append_struct_row_to_dict(pore_data, \"pore_type\", pore_type));\n\n ARROW_ASSIGN_OR_RAISE(\n auto end_reason_data, get_dict_struct(v2_batch, row, \"end_reason\"));\n ARROW_RETURN_NOT_OK(\n append_struct_row_to_dict(end_reason_data, \"name\", end_reason));\n ARROW_RETURN_NOT_OK(\n append_struct_row(\n end_reason_data, \"forced\", end_reason_forced));\n\n ARROW_ASSIGN_OR_RAISE(\n auto run_info_data, get_dict_struct(v2_batch, row, \"run_info\"));\n ARROW_RETURN_NOT_OK(\n append_struct_row_to_dict(run_info_data, \"acquisition_id\", run_info));\n }\n ARROW_RETURN_NOT_OK(set_column(\n v3_reads_schema,\n v3_columns,\n \"calibration_offset\",\n calibration_offset.Finish()));\n ARROW_RETURN_NOT_OK(set_column(\n v3_reads_schema, v3_columns, \"calibration_scale\", calibration_scale.Finish()));\n ARROW_RETURN_NOT_OK(\n set_column(v3_reads_schema, v3_columns, \"channel\", channel.Finish()));\n ARROW_RETURN_NOT_OK(set_column(v3_reads_schema, v3_columns, \"well\", well.Finish()));\n ARROW_RETURN_NOT_OK(\n set_column(v3_reads_schema, v3_columns, \"pore_type\", pore_type.finish()));\n ARROW_RETURN_NOT_OK(\n set_column(v3_reads_schema, v3_columns, \"end_reason\", end_reason.finish()));\n ARROW_RETURN_NOT_OK(set_column(\n v3_reads_schema, v3_columns, \"end_reason_forced\", end_reason_forced.Finish()));\n ARROW_RETURN_NOT_OK(\n set_column(v3_reads_schema, v3_columns, \"run_info\", run_info.finish()));\n\n ARROW_RETURN_NOT_OK(v3_reads_writer.write_batch(num_rows, std::move(v3_columns)));\n }\n ARROW_RETURN_NOT_OK(v3_reads_writer.writer->Close());\n }\n\n if (num_record_batches > 0) {\n ARROW_ASSIGN_OR_RAISE(\n auto v2_last_batch, v2_reader.reader->ReadRecordBatch(num_record_batches - 1));\n auto run_info_column = std::dynamic_pointer_cast(\n v2_last_batch->GetColumnByName(\"run_info\"));\n if (!run_info_column) {\n return arrow::Status::Invalid(\"Failed to find the run info column\");\n }\n auto run_info_dict_type =\n std::dynamic_pointer_cast(run_info_column->type());\n if (!run_info_dict_type) {\n return arrow::Status::Invalid(\"Failed to find a run info of the right type\");\n }\n auto run_info_items =\n std::dynamic_pointer_cast(run_info_column->dictionary());\n if (!run_info_items) {\n return arrow::Status::Invalid(\"Failed to find a run info items array\");\n }\n auto run_info_items_type =\n std::dynamic_pointer_cast(run_info_items->type());\n if (!run_info_items_type) {\n return arrow::Status::Invalid(\n \"Failed to find a run info items array of the right type\");\n }\n\n // Append all the run info dict-struct data to the new table:\n auto v3_run_info_schema = arrow::schema(run_info_items_type->fields(), new_metadata);\n ARROW_ASSIGN_OR_RAISE(\n auto v3_run_info_writer,\n make_record_batch_writer(\n pool, v3_run_info_table_path.ToString(), v3_run_info_schema, new_metadata));\n\n auto const & fields = run_info_items->fields();\n std::vector> v3_columns(\n v3_run_info_schema->fields().size());\n for (std::size_t col = 0; col < v3_columns.size(); ++col) {\n v3_columns[col] = fields[col];\n }\n\n ARROW_RETURN_NOT_OK(\n v3_run_info_writer.write_batch(run_info_items->length(), std::move(v3_columns)));\n ARROW_RETURN_NOT_OK(v3_run_info_writer.writer->Close());\n }\n }\n\n // Set up migrated data to point at our new table:\n MigrationResult result = std::move(v2_input);\n ARROW_RETURN_NOT_OK(result.footer().reads_table.from_full_file(v3_reads_table_path.ToString()));\n ARROW_RETURN_NOT_OK(\n result.footer().run_info_table.from_full_file(v3_run_info_table_path.ToString()));\n result.add_temp_dir(std::move(temp_dir));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/migration/v3_to_v4.cpp", "content": "#include \"pod5_format/migration/migration.h\"\n#include \"pod5_format/migration/migration_utils.h\"\n#include \"pod5_format/table_reader.h\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\narrow::Result migrate_v3_to_v4(\n MigrationResult && v3_input,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(auto temp_dir, MakeTmpDir(\"pod5_v3_v4_migration\"));\n ARROW_ASSIGN_OR_RAISE(auto v4_reads_table_path, temp_dir->path().Join(\"reads_table.arrow\"));\n\n {\n ARROW_ASSIGN_OR_RAISE(\n auto v3_reader, open_record_batch_reader(pool, v3_input.footer().reads_table));\n\n auto v4_new_schama = arrow::schema({arrow::field(\"open_pore_level\", arrow::float32())});\n\n ARROW_ASSIGN_OR_RAISE(\n auto v4_schema, arrow::UnifySchemas({v3_reader.schema, v4_new_schama}));\n\n ARROW_ASSIGN_OR_RAISE(\n auto new_metadata, update_metadata(v3_reader.metadata, Version(0, 3, 30)));\n ARROW_ASSIGN_OR_RAISE(\n auto v4_writer,\n make_record_batch_writer(\n pool, v4_reads_table_path.ToString(), v4_schema, new_metadata));\n\n for (std::int64_t batch_idx = 0; batch_idx < v3_reader.reader->num_record_batches();\n ++batch_idx)\n {\n // Read V0 data:\n ARROW_ASSIGN_OR_RAISE(\n auto v3_batch, ReadRecordBatchAndValidate(*v3_reader.reader, batch_idx));\n ARROW_RETURN_NOT_OK(v3_batch->ValidateFull());\n auto const num_rows = v3_batch->num_rows();\n\n if (num_rows < 0) {\n return arrow::Status::Invalid(\"Invalid number of rows\");\n } else if (POD5_ENABLE_FUZZERS && num_rows > 1'000'000) {\n return arrow::Status::Invalid(\"Skipping huge sizes when fuzzing\");\n }\n\n // Extend with V4 data:\n std::vector> columns = v3_batch->columns();\n ARROW_RETURN_NOT_OK(check_columns(v3_reader.schema, columns));\n ARROW_RETURN_NOT_OK(set_column(\n v4_schema,\n columns,\n \"open_pore_level\",\n make_filled_array(\n pool, num_rows, std::numeric_limits::quiet_NaN())));\n ARROW_RETURN_NOT_OK(v4_writer.write_batch(num_rows, std::move(columns)));\n }\n\n ARROW_RETURN_NOT_OK(v4_writer.writer->Close());\n }\n\n // Set up migrated data to point at our new table:\n MigrationResult result = std::move(v3_input);\n ARROW_RETURN_NOT_OK(result.footer().reads_table.from_full_file(v4_reads_table_path.ToString()));\n result.add_temp_dir(std::move(temp_dir));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_reader.cpp", "content": "#include \"pod5_format/read_table_reader.h\"\n\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/schema_utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nReadTableRecordBatch::ReadTableRecordBatch(\n std::shared_ptr && batch,\n std::shared_ptr const & field_locations)\n: TableRecordBatch(std::move(batch))\n, m_field_locations(field_locations)\n{\n}\n\nReadTableRecordBatch::ReadTableRecordBatch(ReadTableRecordBatch && other)\n: TableRecordBatch(std::move(other))\n, m_field_locations(std::move(other.m_field_locations))\n{\n}\n\nstd::shared_ptr ReadTableRecordBatch::read_id_column() const\n{\n return find_column(batch(), m_field_locations->read_id);\n}\n\nstd::shared_ptr ReadTableRecordBatch::signal_column() const\n{\n return find_column(batch(), m_field_locations->signal);\n}\n\nResult ReadTableRecordBatch::columns() const\n{\n ReadTableRecordColumns result;\n result.table_version = m_field_locations->table_version();\n\n auto const & bat = batch();\n\n // V0 fields:\n result.read_id = find_column(bat, m_field_locations->read_id);\n result.signal = find_column(bat, m_field_locations->signal);\n result.read_number = find_column(bat, m_field_locations->read_number);\n result.start_sample = find_column(bat, m_field_locations->start);\n result.median_before = find_column(bat, m_field_locations->median_before);\n\n // V1 fields:\n if (result.table_version >= ReadTableSpecVersion::v1()) {\n result.num_minknow_events = find_column(bat, m_field_locations->num_minknow_events);\n\n result.tracked_scaling_scale = find_column(bat, m_field_locations->tracked_scaling_scale);\n result.tracked_scaling_shift = find_column(bat, m_field_locations->tracked_scaling_shift);\n result.predicted_scaling_scale =\n find_column(bat, m_field_locations->predicted_scaling_scale);\n result.predicted_scaling_shift =\n find_column(bat, m_field_locations->predicted_scaling_shift);\n result.num_reads_since_mux_change =\n find_column(bat, m_field_locations->num_reads_since_mux_change);\n result.time_since_mux_change = find_column(bat, m_field_locations->time_since_mux_change);\n }\n\n // V2 fields:\n if (result.table_version >= ReadTableSpecVersion::v2()) {\n result.num_samples = find_column(bat, m_field_locations->num_samples);\n }\n\n // V3 fields:\n if (result.table_version >= ReadTableSpecVersion::v3()) {\n result.channel = find_column(bat, m_field_locations->channel);\n result.well = find_column(bat, m_field_locations->well);\n result.pore_type = find_column(bat, m_field_locations->pore_type);\n result.calibration_offset = find_column(bat, m_field_locations->calibration_offset);\n result.calibration_scale = find_column(bat, m_field_locations->calibration_scale);\n result.end_reason = find_column(bat, m_field_locations->end_reason);\n result.end_reason_forced = find_column(bat, m_field_locations->end_reason_forced);\n result.run_info = find_column(bat, m_field_locations->run_info);\n }\n\n if (result.table_version >= ReadTableSpecVersion::v4()) {\n result.open_pore_level = find_column(bat, m_field_locations->open_pore_level);\n }\n\n return result;\n}\n\nResult> ReadTableRecordBatch::get_signal_rows(\n std::int64_t batch_row) const\n{\n auto signal_col = signal_column();\n\n auto const & values = signal_col->values();\n\n auto const offset = signal_col->value_offset(batch_row);\n if (offset >= values->length()) {\n return arrow::Status::Invalid(\n \"Invalid signal row offset '\", offset, \"' is outside the size of the values array.\");\n }\n\n auto const length = signal_col->value_length(batch_row);\n if (length > values->length() - offset) {\n return arrow::Status::Invalid(\n \"Invalid signal row length '\", length, \"' is outside the size of the values array.\");\n }\n\n return std::static_pointer_cast(values->Slice(offset, length));\n}\n\ntemplate \nauto & ReadTableRecordBatch::get_dictionary(\n std::shared_ptr const & array) const\n{\n auto & initialised = m_dictionary_initialised[static_cast(which)];\n if (initialised.load(std::memory_order_acquire)) {\n return array->dictionary();\n }\n\n std::lock_guard lock(m_dictionary_access_lock);\n auto & dict = array->dictionary();\n initialised.store(true, std::memory_order_release);\n return dict;\n}\n\nResult ReadTableRecordBatch::get_pore_type(std::int16_t pore_index) const\n{\n if (!m_field_locations->pore_type.found_field()) {\n return arrow::Status::Invalid(\"pore field is not present in the file\");\n }\n\n auto pore_column = find_column(batch(), m_field_locations->pore_type);\n auto const & pore_dict = get_dictionary(pore_column);\n auto const & pore_data = static_cast(*pore_dict);\n if (pore_index < 0 || pore_index >= pore_data.length()) {\n return arrow::Status::IndexError(\n \"Invalid index \", pore_index, \" for pore array of length \", pore_data.length());\n }\n\n return pore_data.GetString(pore_index);\n}\n\nResult> ReadTableRecordBatch::get_end_reason(\n std::int16_t end_reason_index) const\n{\n if (!m_field_locations->end_reason.found_field()) {\n return arrow::Status::Invalid(\"end_reason field is not present in the file\");\n }\n\n auto end_reason_column = find_column(batch(), m_field_locations->end_reason);\n auto const & end_reason_dict = get_dictionary(end_reason_column);\n auto const & end_reason_data = static_cast(*end_reason_dict);\n if (end_reason_index >= end_reason_data.length()) {\n return arrow::Status::IndexError(\n \"Invalid index \",\n end_reason_index,\n \" for end reason array of length \",\n end_reason_data.length());\n }\n\n auto str_value = end_reason_data.GetString(end_reason_index);\n auto reason = end_reason_from_string(str_value);\n\n return std::make_pair(reason, std::move(str_value));\n}\n\nResult ReadTableRecordBatch::get_run_info(std::int16_t run_info_index) const\n{\n if (!m_field_locations->run_info.found_field()) {\n return arrow::Status::Invalid(\"end_reason field is not present in the file\");\n }\n\n auto run_info_column = find_column(batch(), m_field_locations->run_info);\n auto const & run_info_dict = get_dictionary(run_info_column);\n auto const & run_info_data = static_cast(*run_info_dict);\n if (run_info_index < 0 || run_info_index >= run_info_data.length()) {\n return arrow::Status::IndexError(\n \"Invalid index \",\n run_info_index,\n \" for run info array of length \",\n run_info_data.length());\n }\n\n return run_info_data.GetString(run_info_index);\n}\n\n//---------------------------------------------------------------------------------------------------------------------\n\nReadTableReader::ReadTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n std::shared_ptr const & field_locations,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool)\n: TableReader(std::move(input_source), std::move(reader), std::move(schema_metadata), pool)\n, m_field_locations(field_locations)\n{\n}\n\nReadTableReader::ReadTableReader(ReadTableReader && other)\n: TableReader(std::move(other))\n, m_sorted_file_read_ids(std::move(other.m_sorted_file_read_ids))\n, m_field_locations(std::move(other.m_field_locations))\n{\n}\n\nReadTableReader & ReadTableReader::operator=(ReadTableReader && other)\n{\n static_cast(*this) = std::move(static_cast(*this));\n m_field_locations = std::move(other.m_field_locations);\n m_sorted_file_read_ids = std::move(other.m_sorted_file_read_ids);\n return *this;\n}\n\nResult ReadTableReader::read_record_batch(std::size_t i) const\n{\n std::lock_guard l(m_batch_get_mutex);\n ARROW_ASSIGN_OR_RAISE(auto record_batch, TableReader::ReadRecordBatch(i));\n return ReadTableRecordBatch{std::move(record_batch), m_field_locations};\n}\n\nStatus ReadTableReader::build_read_id_lookup() const\n{\n std::lock_guard lock(m_sorted_file_read_ids_mutex);\n\n if (!m_sorted_file_read_ids.empty()) {\n return Status::OK();\n }\n\n std::vector file_read_ids;\n\n auto const batch_count = num_record_batches();\n std::size_t abs_row_count = 0;\n\n // Loop each batch and copy read ids out into the index:\n for (std::size_t i = 0; i < batch_count; ++i) {\n ARROW_ASSIGN_OR_RAISE(auto batch, read_record_batch(i));\n\n if (file_read_ids.empty()) {\n file_read_ids.reserve(batch.num_rows() * batch_count);\n }\n file_read_ids.resize(file_read_ids.size() + batch.num_rows());\n\n auto read_id_col = batch.read_id_column();\n auto raw_read_id_values = read_id_col->raw_values();\n for (std::size_t row = 0; row < (std::size_t)read_id_col->length(); ++row) {\n // Record the id, and its location within the file:\n file_read_ids[abs_row_count].id = raw_read_id_values[row];\n file_read_ids[abs_row_count].batch = i;\n file_read_ids[abs_row_count].batch_row = row;\n abs_row_count += 1;\n }\n }\n\n // Sort by read id for searching later:\n std::sort(file_read_ids.begin(), file_read_ids.end(), [](auto const & a, auto const & b) {\n return a.id < b.id;\n });\n\n // Move data out now we successfully build the index:\n m_sorted_file_read_ids = std::move(file_read_ids);\n\n return Status::OK();\n}\n\nResult ReadTableReader::search_for_read_ids(\n ReadIdSearchInput const & search_input,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows) const\n{\n ARROW_RETURN_NOT_OK(build_read_id_lookup());\n\n if (m_sorted_file_read_ids.empty()) {\n return 0;\n }\n\n std::size_t successes = 0;\n\n std::vector> batch_data(batch_counts.size());\n auto const initial_reserve_size = search_input.read_id_count() / batch_counts.size();\n for (auto & br : batch_data) {\n br.reserve(initial_reserve_size);\n }\n\n auto file_ids_current_it = m_sorted_file_read_ids.begin();\n auto const file_ids_end = m_sorted_file_read_ids.end();\n for (std::size_t i = 0; i < search_input.read_id_count(); ++i) {\n auto const & search_item = search_input[i];\n\n // Increment file pointer while less than the search term:\n while (file_ids_current_it != file_ids_end && file_ids_current_it->id < search_item.id) {\n ++file_ids_current_it;\n }\n\n // No more ids to search, both lists are sorted and we haven't found this one, we won't find any others.\n if (file_ids_current_it == file_ids_end) {\n break;\n }\n\n // If we found it record the location:\n if (file_ids_current_it->id == search_item.id) {\n batch_data[file_ids_current_it->batch].push_back(file_ids_current_it->batch_row);\n successes += 1;\n }\n }\n\n std::size_t full_size_so_far = 0;\n for (std::size_t i = 0; i < batch_data.size(); ++i) {\n auto & data = batch_data[i];\n batch_counts[i] = data.size();\n\n // Ensure the batch indices within the batch are sorted:\n std::sort(data.begin(), data.end());\n\n // Copy the row indices into the packed vector:\n std::copy(data.begin(), data.end(), batch_rows.begin() + full_size_so_far);\n\n full_size_so_far += data.size();\n }\n\n return successes;\n}\n\n//---------------------------------------------------------------------------------------------------------------------\n\nResult make_read_table_reader(\n std::shared_ptr const & input,\n arrow::MemoryPool * pool)\n{\n arrow::ipc::IpcReadOptions options;\n options.memory_pool = pool;\n\n ARROW_ASSIGN_OR_RAISE(auto reader, arrow::ipc::RecordBatchFileReader::Open(input, options));\n\n auto read_metadata_key_values = reader->schema()->metadata();\n if (!read_metadata_key_values) {\n return Status::IOError(\"Missing metadata on read table schema\");\n }\n ARROW_ASSIGN_OR_RAISE(\n auto read_metadata, read_schema_key_value_metadata(read_metadata_key_values));\n ARROW_ASSIGN_OR_RAISE(\n auto field_locations, read_read_table_schema(read_metadata, reader->schema()));\n\n return ReadTableReader(\n {input}, std::move(reader), field_locations, std::move(read_metadata), pool);\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_reader.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_schema.h\"\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/table_reader.h\"\n#include \"pod5_format/types.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n\n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace io {\nclass RandomAccessFile;\n}\n\nnamespace ipc {\nclass RecordBatchFileReader;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nclass CalibrationData;\nclass EndReasonData;\nclass PoreData;\nclass RunInfoData;\nclass ReadIdSearchInput;\n\nstruct ReadTableRecordColumns {\n std::shared_ptr read_id;\n std::shared_ptr signal;\n std::shared_ptr read_number;\n std::shared_ptr start_sample;\n std::shared_ptr median_before;\n\n std::shared_ptr num_minknow_events;\n\n [[deprecated]] std::shared_ptr tracked_scaling_scale;\n [[deprecated]] std::shared_ptr tracked_scaling_shift;\n [[deprecated]] std::shared_ptr predicted_scaling_scale;\n [[deprecated]] std::shared_ptr predicted_scaling_shift;\n [[deprecated]] std::shared_ptr num_reads_since_mux_change;\n [[deprecated]] std::shared_ptr time_since_mux_change;\n\n std::shared_ptr num_samples;\n\n std::shared_ptr channel;\n std::shared_ptr well;\n std::shared_ptr pore_type;\n std::shared_ptr calibration_offset;\n std::shared_ptr calibration_scale;\n std::shared_ptr end_reason;\n std::shared_ptr end_reason_forced;\n std::shared_ptr run_info;\n\n std::shared_ptr open_pore_level;\n\n TableSpecVersion table_version;\n};\n\nclass POD5_FORMAT_EXPORT ReadTableRecordBatch : public TableRecordBatch {\npublic:\n ReadTableRecordBatch(\n std::shared_ptr && batch,\n std::shared_ptr const & field_locations);\n\n ReadTableRecordBatch(ReadTableRecordBatch &&);\n\n std::shared_ptr read_id_column() const;\n std::shared_ptr signal_column() const;\n\n Result get_pore_type(std::int16_t pore_dict_index) const;\n Result> get_end_reason(\n std::int16_t end_reason_dict_index) const;\n Result get_run_info(std::int16_t run_info_dict_index) const;\n\n Result columns() const;\n\n Result> get_signal_rows(std::int64_t batch_row) const;\n\nprivate:\n std::shared_ptr m_field_locations;\n\n enum class Dict : std::size_t { Pore, EndReason, RunInfo, Max };\n mutable std::atomic_bool m_dictionary_initialised[static_cast(Dict::Max)]{};\n mutable std::mutex m_dictionary_access_lock;\n template \n auto & get_dictionary(std::shared_ptr const & array) const;\n};\n\nclass POD5_FORMAT_EXPORT ReadTableReader : public TableReader {\npublic:\n ReadTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n std::shared_ptr const & field_locations,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool);\n\n ReadTableReader(ReadTableReader && other);\n ReadTableReader & operator=(ReadTableReader && other);\n\n Result read_record_batch(std::size_t i) const;\n\n Result search_for_read_ids(\n ReadIdSearchInput const & search_input,\n gsl::span const & batch_counts,\n gsl::span const & batch_rows) const;\n\nprivate:\n struct IndexData {\n Uuid id;\n std::size_t batch;\n std::size_t batch_row;\n };\n\n Status build_read_id_lookup() const;\n mutable std::vector m_sorted_file_read_ids;\n mutable std::mutex m_sorted_file_read_ids_mutex;\n\nprivate:\n std::shared_ptr m_field_locations;\n\n mutable std::mutex m_batch_get_mutex;\n};\n\nPOD5_FORMAT_EXPORT Result make_read_table_reader(\n std::shared_ptr const & sink,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_schema.cpp", "content": "#include \"pod5_format/read_table_schema.h\"\n\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/types.h\"\n\nnamespace pod5 {\n\nReadTableSchemaDescription::ReadTableSchemaDescription()\n: SchemaDescriptionBase(ReadTableSpecVersion::latest())\n// V0 Fields\n, read_id(this, \"read_id\", uuid(), ReadTableSpecVersion::v0())\n, signal(this, \"signal\", arrow::list(arrow::uint64()), ReadTableSpecVersion::v0())\n, read_number(this, \"read_number\", arrow::uint32(), ReadTableSpecVersion::v0())\n, start(this, \"start\", arrow::uint64(), ReadTableSpecVersion::v0())\n, median_before(this, \"median_before\", arrow::float32(), ReadTableSpecVersion::v0())\n,\n// V1 Fields\nnum_minknow_events(this, \"num_minknow_events\", arrow::uint64(), ReadTableSpecVersion::v1())\n, tracked_scaling_scale(this, \"tracked_scaling_scale\", arrow::float32(), ReadTableSpecVersion::v1())\n, tracked_scaling_shift(this, \"tracked_scaling_shift\", arrow::float32(), ReadTableSpecVersion::v1())\n, predicted_scaling_scale(\n this,\n \"predicted_scaling_scale\",\n arrow::float32(),\n ReadTableSpecVersion::v1())\n, predicted_scaling_shift(\n this,\n \"predicted_scaling_shift\",\n arrow::float32(),\n ReadTableSpecVersion::v1())\n, num_reads_since_mux_change(\n this,\n \"num_reads_since_mux_change\",\n arrow::uint32(),\n ReadTableSpecVersion::v1())\n, time_since_mux_change(this, \"time_since_mux_change\", arrow::float32(), ReadTableSpecVersion::v1())\n,\n// V2 Fields\nnum_samples(this, \"num_samples\", arrow::uint64(), ReadTableSpecVersion::v2())\n,\n// V3 Fields\nchannel(this, \"channel\", arrow::uint16(), ReadTableSpecVersion::v3())\n, well(this, \"well\", arrow::uint8(), ReadTableSpecVersion::v3())\n, pore_type(\n this,\n \"pore_type\",\n arrow::dictionary(arrow::int16(), arrow::utf8()),\n ReadTableSpecVersion::v3())\n, calibration_offset(this, \"calibration_offset\", arrow::float32(), ReadTableSpecVersion::v3())\n, calibration_scale(this, \"calibration_scale\", arrow::float32(), ReadTableSpecVersion::v3())\n, end_reason(\n this,\n \"end_reason\",\n arrow::dictionary(arrow::int16(), arrow::utf8()),\n ReadTableSpecVersion::v3())\n, end_reason_forced(this, \"end_reason_forced\", arrow::boolean(), ReadTableSpecVersion::v3())\n, run_info(\n this,\n \"run_info\",\n arrow::dictionary(arrow::int16(), arrow::utf8()),\n ReadTableSpecVersion::v3())\n, open_pore_level(this, \"open_pore_level\", arrow::float32(), ReadTableSpecVersion::v4())\n{\n}\n\nTableSpecVersion ReadTableSchemaDescription::table_version_from_file_version(\n Version file_version) const\n{\n return ReadTableSpecVersion::latest();\n}\n\nResult> read_read_table_schema(\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const & schema)\n{\n auto result = std::make_shared();\n ARROW_RETURN_NOT_OK(ReadTableSchemaDescription::read_schema(result, schema_metadata, schema));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_schema.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_utils.h\"\n#include \"pod5_format/tuple_utils.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n\nnamespace arrow {\nclass KeyValueMetadata;\nclass Schema;\nclass DataType;\nclass StructType;\n} // namespace arrow\n\nnamespace pod5 {\n\nstruct SchemaMetadataDescription;\n\nclass ReadTableSpecVersion {\npublic:\n static TableSpecVersion v0() { return TableSpecVersion::first_version(); }\n\n static TableSpecVersion v1()\n {\n // Addition of num_minknow_events and scaling parameters\n return TableSpecVersion::at_version(1);\n }\n\n static TableSpecVersion v2()\n {\n // Addition of num_samples parameters\n return TableSpecVersion::at_version(2);\n }\n\n static TableSpecVersion v3()\n {\n // Flattening of dictionaries into separate table.\n return TableSpecVersion::at_version(3);\n }\n\n static TableSpecVersion v4()\n {\n // Flattening of dictionaries into separate table.\n return TableSpecVersion::at_version(4);\n }\n\n static TableSpecVersion latest() { return v4(); }\n};\n\nclass ReadTableSchemaDescription : public SchemaDescriptionBase {\npublic:\n ReadTableSchemaDescription();\n\n ReadTableSchemaDescription(ReadTableSchemaDescription const &) = delete;\n ReadTableSchemaDescription & operator=(ReadTableSchemaDescription const &) = delete;\n\n TableSpecVersion table_version_from_file_version(Version file_version) const override;\n\n // V0 fields\n Field<0, UuidArray> read_id;\n ListField<1, arrow::ListArray, arrow::UInt64Array> signal;\n Field<2, arrow::UInt32Array> read_number;\n Field<3, arrow::UInt64Array> start;\n Field<4, arrow::FloatArray> median_before;\n\n // V1 fields\n Field<5, arrow::UInt64Array> num_minknow_events;\n [[deprecated]] Field<6, arrow::FloatArray> tracked_scaling_scale;\n [[deprecated]] Field<7, arrow::FloatArray> tracked_scaling_shift;\n [[deprecated]] Field<8, arrow::FloatArray> predicted_scaling_scale;\n [[deprecated]] Field<9, arrow::FloatArray> predicted_scaling_shift;\n [[deprecated]] Field<10, arrow::UInt32Array> num_reads_since_mux_change;\n [[deprecated]] Field<11, arrow::FloatArray> time_since_mux_change;\n\n // V2 fields\n Field<12, arrow::UInt64Array> num_samples;\n\n // V3 fields\n Field<13, arrow::UInt16Array> channel;\n Field<14, arrow::UInt8Array> well;\n Field<15, arrow::DictionaryArray> pore_type;\n Field<16, arrow::FloatArray> calibration_offset;\n Field<17, arrow::FloatArray> calibration_scale;\n Field<18, arrow::DictionaryArray> end_reason;\n Field<19, arrow::BooleanArray> end_reason_forced;\n Field<20, arrow::DictionaryArray> run_info;\n\n // V4 fields\n Field<21, arrow::FloatArray> open_pore_level;\n\n // Field Builders only for fields we write in newly generated files.\n // Should not include fields which are removed in the latest version:\n using FieldBuilders = FieldBuilder<\n // V0 fields\n decltype(read_id),\n decltype(signal),\n decltype(read_number),\n decltype(start),\n decltype(median_before),\n\n // V1 fields\n decltype(num_minknow_events),\n decltype(tracked_scaling_scale),\n decltype(tracked_scaling_shift),\n decltype(predicted_scaling_scale),\n decltype(predicted_scaling_shift),\n decltype(num_reads_since_mux_change),\n decltype(time_since_mux_change),\n\n // V2 fields\n decltype(num_samples),\n\n // V3 fields\n decltype(channel),\n decltype(well),\n decltype(pore_type),\n decltype(calibration_offset),\n decltype(calibration_scale),\n decltype(end_reason),\n decltype(end_reason_forced),\n decltype(run_info),\n\n // V4 fields\n decltype(open_pore_level)>;\n};\n\nPOD5_FORMAT_EXPORT Result> read_read_table_schema(\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const &);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_utils.cpp", "content": "#include \"pod5_format/read_table_utils.h\"\n\n#include \n\nnamespace pod5 {\n\nReadIdSearchInput::ReadIdSearchInput(gsl::span const & input_ids)\n: m_search_read_ids(input_ids.size())\n{\n // Copy in search input:\n for (std::size_t i = 0; i < input_ids.size(); ++i) {\n m_search_read_ids[i].id = input_ids[i];\n m_search_read_ids[i].index = i;\n }\n\n // Sort input based on read id:\n std::sort(\n m_search_read_ids.begin(), m_search_read_ids.end(), [](auto const & a, auto const & b) {\n return a.id < b.id;\n });\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_utils.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n\n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nusing PoreDictionaryIndex = std::int16_t;\nusing EndReasonDictionaryIndex = std::int16_t;\nusing RunInfoDictionaryIndex = std::int16_t;\n\nclass ReadData {\npublic:\n ReadData() = default;\n\n /// \\brief Create a new read data structure to add to a read.\n /// \\param read_id The read id for the read entry.\n /// \\param read_number Read number for this read.\n /// \\param start_sample The sample which this read starts at.\n /// \\param median_before The median of the read chunk prior to the start of this read.\n /// \\param end_reason The dictionary index of the end reason name which caused this read to complete.\n /// \\param end_reason_forced Boolean value indicating if the read end was forced.\n /// \\param run_info The dictionary index of the run info for this read.\n /// \\param num_minknow_events The number of minknow events in the read.\n ReadData(\n Uuid const & read_id,\n std::uint32_t read_number,\n std::uint64_t start_sample,\n std::uint16_t channel,\n std::uint8_t well,\n PoreDictionaryIndex pore_type,\n float calibration_offset,\n float calibration_scale,\n float median_before,\n EndReasonDictionaryIndex end_reason,\n bool end_reason_forced,\n RunInfoDictionaryIndex run_info,\n std::uint64_t num_minknow_events,\n float tracked_scaling_scale,\n float tracked_scaling_shift,\n float predicted_scaling_scale,\n float predicted_scaling_shift,\n std::uint32_t num_reads_since_mux_change,\n float time_since_mux_change,\n float open_pore_level)\n : read_id(read_id)\n , read_number(read_number)\n , start_sample(start_sample)\n , median_before(median_before)\n , end_reason(end_reason)\n , end_reason_forced(end_reason_forced)\n , run_info(run_info)\n , num_minknow_events(num_minknow_events)\n , tracked_scaling_scale(tracked_scaling_scale)\n , tracked_scaling_shift(tracked_scaling_shift)\n , predicted_scaling_scale(predicted_scaling_scale)\n , predicted_scaling_shift(predicted_scaling_shift)\n , num_reads_since_mux_change(num_reads_since_mux_change)\n , time_since_mux_change(time_since_mux_change)\n , channel(channel)\n , well(well)\n , pore_type(pore_type)\n , calibration_offset(calibration_offset)\n , calibration_scale(calibration_scale)\n , open_pore_level(open_pore_level)\n {\n }\n\n // V1 Fields\n Uuid read_id;\n std::uint32_t read_number;\n std::uint64_t start_sample;\n float median_before;\n EndReasonDictionaryIndex end_reason;\n bool end_reason_forced;\n RunInfoDictionaryIndex run_info;\n\n // V2 Fields\n std::uint64_t num_minknow_events;\n [[deprecated]] float tracked_scaling_scale;\n [[deprecated]] float tracked_scaling_shift;\n [[deprecated]] float predicted_scaling_scale;\n [[deprecated]] float predicted_scaling_shift;\n [[deprecated]] std::uint32_t num_reads_since_mux_change;\n [[deprecated]] float time_since_mux_change;\n\n // V3 Fields\n std::uint16_t channel;\n std::uint8_t well;\n PoreDictionaryIndex pore_type;\n float calibration_offset;\n float calibration_scale;\n\n // V4 Fields\n float open_pore_level;\n};\n\ninline bool operator==(ReadData const & a, ReadData const & b)\n{\n return a.read_id == b.read_id && a.read_number == b.read_number\n && a.start_sample == b.start_sample && a.median_before == b.median_before\n && a.end_reason == b.end_reason && a.end_reason_forced == b.end_reason_forced\n && a.run_info == b.run_info && a.num_minknow_events == b.num_minknow_events\n && a.tracked_scaling_scale == b.tracked_scaling_scale\n && a.tracked_scaling_shift == b.tracked_scaling_shift\n && a.predicted_scaling_scale == b.predicted_scaling_scale\n && a.predicted_scaling_shift == b.predicted_scaling_shift\n && a.num_reads_since_mux_change == b.num_reads_since_mux_change\n && a.time_since_mux_change == b.time_since_mux_change && a.channel == b.channel\n && a.well == b.well && a.pore_type == b.pore_type\n && a.calibration_offset == b.calibration_offset\n && a.calibration_scale == b.calibration_scale && a.open_pore_level == b.open_pore_level;\n}\n\nclass RunInfoData {\npublic:\n using MapType = std::vector>;\n\n RunInfoData(\n std::string acquisition_id,\n std::int64_t acquisition_start_time,\n std::int16_t adc_max,\n std::int16_t adc_min,\n MapType context_tags,\n std::string experiment_name,\n std::string flow_cell_id,\n std::string flow_cell_product_code,\n std::string protocol_name,\n std::string protocol_run_id,\n std::int64_t protocol_start_time,\n std::string sample_id,\n std::uint16_t sample_rate,\n std::string sequencing_kit,\n std::string sequencer_position,\n std::string sequencer_position_type,\n std::string software,\n std::string system_name,\n std::string system_type,\n MapType tracking_id)\n : acquisition_id(std::move(acquisition_id))\n , acquisition_start_time(std::move(acquisition_start_time))\n , adc_max(std::move(adc_max))\n , adc_min(std::move(adc_min))\n , context_tags(std::move(context_tags))\n , experiment_name(std::move(experiment_name))\n , flow_cell_id(std::move(flow_cell_id))\n , flow_cell_product_code(std::move(flow_cell_product_code))\n , protocol_name(std::move(protocol_name))\n , protocol_run_id(std::move(protocol_run_id))\n , protocol_start_time(std::move(protocol_start_time))\n , sample_id(std::move(sample_id))\n , sample_rate(std::move(sample_rate))\n , sequencing_kit(std::move(sequencing_kit))\n , sequencer_position(std::move(sequencer_position))\n , sequencer_position_type(std::move(sequencer_position_type))\n , software(std::move(software))\n , system_name(std::move(system_name))\n , system_type(std::move(system_type))\n , tracking_id(std::move(tracking_id))\n {\n }\n\n static std::int64_t convert_from_system_clock(std::chrono::system_clock::time_point value)\n {\n return value.time_since_epoch() / std::chrono::milliseconds(1);\n }\n\n static std::chrono::system_clock::time_point convert_to_system_clock(\n std::int64_t since_epoch_ms)\n {\n return std::chrono::system_clock::time_point() + std::chrono::milliseconds(since_epoch_ms);\n }\n\n std::string acquisition_id;\n std::int64_t acquisition_start_time;\n std::int16_t adc_max;\n std::int16_t adc_min;\n MapType context_tags;\n std::string experiment_name;\n std::string flow_cell_id;\n std::string flow_cell_product_code;\n std::string protocol_name;\n std::string protocol_run_id;\n std::int64_t protocol_start_time;\n std::string sample_id;\n std::uint16_t sample_rate;\n std::string sequencing_kit;\n std::string sequencer_position;\n std::string sequencer_position_type;\n std::string software;\n std::string system_name;\n std::string system_type;\n MapType tracking_id;\n};\n\ninline bool operator==(RunInfoData const & a, RunInfoData const & b)\n{\n return a.acquisition_id == b.acquisition_id\n && a.acquisition_start_time == b.acquisition_start_time && a.adc_max == b.adc_max\n && a.adc_min == b.adc_min && a.context_tags == b.context_tags\n && a.experiment_name == b.experiment_name && a.flow_cell_id == b.flow_cell_id\n && a.flow_cell_product_code == b.flow_cell_product_code\n && a.protocol_name == b.protocol_name && a.protocol_run_id == b.protocol_run_id\n && a.protocol_start_time == b.protocol_start_time && a.sample_id == b.sample_id\n && a.sample_rate == b.sample_rate && a.sequencing_kit == b.sequencing_kit\n && a.sequencer_position == b.sequencer_position\n && a.sequencer_position_type == b.sequencer_position_type && a.software == b.software\n && a.system_name == b.system_name && a.system_type == b.system_type\n && a.tracking_id == b.tracking_id;\n}\n\nenum class ReadEndReason : std::uint8_t {\n unknown,\n mux_change,\n unblock_mux_change,\n data_service_unblock_mux_change,\n signal_positive,\n signal_negative,\n api_request,\n device_data_error,\n analysis_config_change,\n paused,\n\n last_end_reason = paused\n};\n\ninline char const * end_reason_as_string(ReadEndReason reason)\n{\n static_assert(\n ReadEndReason::last_end_reason == ReadEndReason::paused,\n \"Need to add new end reason to this function\");\n switch (reason) {\n case ReadEndReason::mux_change:\n return \"mux_change\";\n case ReadEndReason::unblock_mux_change:\n return \"unblock_mux_change\";\n case ReadEndReason::data_service_unblock_mux_change:\n return \"data_service_unblock_mux_change\";\n case ReadEndReason::signal_positive:\n return \"signal_positive\";\n case ReadEndReason::signal_negative:\n return \"signal_negative\";\n case ReadEndReason::api_request:\n return \"api_request\";\n case ReadEndReason::device_data_error:\n return \"device_data_error\";\n case ReadEndReason::analysis_config_change:\n return \"analysis_config_change\";\n case ReadEndReason::paused:\n return \"paused\";\n case ReadEndReason::unknown:\n break;\n }\n return \"unknown\";\n}\n\ninline ReadEndReason end_reason_from_string(std::string const & reason)\n{\n static_assert(\n ReadEndReason::last_end_reason == ReadEndReason::paused,\n \"Need to add new end reason to this function\");\n if (reason == \"unknown\") {\n return ReadEndReason::unknown;\n } else if (reason == \"mux_change\") {\n return ReadEndReason::mux_change;\n } else if (reason == \"unblock_mux_change\") {\n return ReadEndReason::unblock_mux_change;\n } else if (reason == \"data_service_unblock_mux_change\") {\n return ReadEndReason::data_service_unblock_mux_change;\n } else if (reason == \"signal_positive\") {\n return ReadEndReason::signal_positive;\n } else if (reason == \"signal_negative\") {\n return ReadEndReason::signal_negative;\n } else if (reason == \"api_request\") {\n return ReadEndReason::api_request;\n } else if (reason == \"device_data_error\") {\n return ReadEndReason::device_data_error;\n } else if (reason == \"analysis_config_change\") {\n return ReadEndReason::analysis_config_change;\n } else if (reason == \"paused\") {\n return ReadEndReason::paused;\n }\n\n return ReadEndReason::unknown;\n}\n\n/// \\brief Input query to a search for a number of read ids in a file:\nclass POD5_FORMAT_EXPORT ReadIdSearchInput {\npublic:\n struct InputId {\n Uuid id;\n std::size_t index;\n };\n\n ReadIdSearchInput(gsl::span const & input_ids);\n\n std::size_t read_id_count() const { return m_search_read_ids.size(); }\n\n InputId const & operator[](std::size_t i) const { return m_search_read_ids[i]; }\n\nprivate:\n std::vector m_search_read_ids;\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_writer.cpp", "content": "#include \"pod5_format/read_table_writer.h\"\n\n#include \"pod5_format/file_output_stream.h\"\n#include \"pod5_format/internal/tracing/tracing.h\"\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nReadTableWriter::ReadTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n std::shared_ptr const & field_locations,\n std::size_t table_batch_size,\n std::shared_ptr const & pore_writer,\n std::shared_ptr const & end_reason_writer,\n std::shared_ptr const & run_info_writer,\n std::shared_ptr const & output_stream,\n arrow::MemoryPool * pool)\n: m_schema(schema)\n, m_field_locations(field_locations)\n, m_table_batch_size(table_batch_size)\n, m_writer(std::move(writer))\n, m_field_builders(m_field_locations, pool)\n, m_output_stream{output_stream}\n{\n m_field_builders.get_builder(m_field_locations->pore_type).set_dict_writer(pore_writer);\n m_field_builders.get_builder(m_field_locations->end_reason).set_dict_writer(end_reason_writer);\n m_field_builders.get_builder(m_field_locations->run_info).set_dict_writer(run_info_writer);\n}\n\nReadTableWriter::ReadTableWriter(ReadTableWriter && other) = default;\nReadTableWriter & ReadTableWriter::operator=(ReadTableWriter &&) = default;\n\nReadTableWriter::~ReadTableWriter()\n{\n if (m_writer) {\n (void)close();\n }\n}\n\nResult ReadTableWriter::add_read(\n ReadData const & read_data,\n gsl::span const & signal,\n std::uint64_t signal_duration)\n{\n POD5_TRACE_FUNCTION();\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_RETURN_NOT_OK(reserve_rows());\n\n auto row_id = m_written_batched_row_count + m_current_batch_row_count;\n ARROW_RETURN_NOT_OK(m_field_builders.append(\n // V0 Fields\n read_data.read_id,\n signal,\n read_data.read_number,\n read_data.start_sample,\n read_data.median_before,\n\n // V1 Fields\n read_data.num_minknow_events,\n read_data.tracked_scaling_scale,\n read_data.tracked_scaling_shift,\n read_data.predicted_scaling_scale,\n read_data.predicted_scaling_shift,\n read_data.num_reads_since_mux_change,\n read_data.time_since_mux_change,\n\n // V2 Fields\n signal_duration,\n\n // V3 Fields\n read_data.channel,\n read_data.well,\n read_data.pore_type,\n read_data.calibration_offset,\n read_data.calibration_scale,\n read_data.end_reason,\n read_data.end_reason_forced,\n read_data.run_info,\n\n // V4 Fields\n read_data.open_pore_level));\n\n ++m_current_batch_row_count;\n\n if (m_current_batch_row_count >= m_table_batch_size) {\n ARROW_RETURN_NOT_OK(write_batch());\n }\n return row_id;\n}\n\nStatus ReadTableWriter::close()\n{\n // Check for already closed\n if (!m_writer) {\n return Status::OK();\n }\n\n ARROW_RETURN_NOT_OK(write_batch());\n ARROW_RETURN_NOT_OK(m_writer->Close());\n m_writer = nullptr;\n return Status::OK();\n}\n\nStatus ReadTableWriter::write_batch(arrow::RecordBatch const & record_batch)\n{\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus ReadTableWriter::write_batch()\n{\n POD5_TRACE_FUNCTION();\n if (m_current_batch_row_count == 0) {\n return Status::OK();\n }\n\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_ASSIGN_OR_RAISE(auto columns, m_field_builders.finish_columns());\n\n auto const record_batch =\n arrow::RecordBatch::Make(m_schema, m_current_batch_row_count, std::move(columns));\n\n m_written_batched_row_count += m_current_batch_row_count;\n m_current_batch_row_count = 0;\n\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(*record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus ReadTableWriter::reserve_rows()\n{\n // Only reserve if we have not already reserved (at the start of a batch)\n if (m_current_batch_row_count > 0) {\n return arrow::Status::OK();\n }\n\n return m_field_builders.reserve(m_table_batch_size);\n}\n\nResult make_read_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n std::shared_ptr const & pore_writer,\n std::shared_ptr const & end_reason_writer,\n std::shared_ptr const & run_info_writer,\n arrow::MemoryPool * pool)\n{\n auto field_locations = std::make_shared();\n auto schema = field_locations->make_writer_schema(metadata);\n\n arrow::ipc::IpcWriteOptions options;\n options.memory_pool = pool;\n options.emit_dictionary_deltas = true;\n // todo... consider:\n //ARROW_ASSIGN_OR_RAISE(options.codec, arrow::util::Codec::Create(arrow::Compression::LZ4_FRAME));\n\n ARROW_ASSIGN_OR_RAISE(auto writer, arrow::ipc::MakeFileWriter(sink, schema, options, metadata));\n\n auto read_table_writer = ReadTableWriter(\n std::move(writer),\n std::move(schema),\n field_locations,\n table_batch_size,\n pore_writer,\n end_reason_writer,\n run_info_writer,\n sink,\n pool);\n\n return read_table_writer;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_writer.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_schema.h\"\n#include \"pod5_format/read_table_writer_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_field_builder.h\"\n#include \"pod5_format/signal_table_utils.h\"\n\n#include \n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace ipc {\nclass RecordBatchWriter;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nclass FileOutputStream;\n\nclass POD5_FORMAT_EXPORT ReadTableWriter {\npublic:\n ReadTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n std::shared_ptr const & field_locations,\n std::size_t table_batch_size,\n std::shared_ptr const & pore_writer,\n std::shared_ptr const & end_reason_writer,\n std::shared_ptr const & run_info_writer,\n std::shared_ptr const & output_stream,\n arrow::MemoryPool * pool);\n ReadTableWriter(ReadTableWriter &&);\n ReadTableWriter & operator=(ReadTableWriter &&);\n ReadTableWriter(ReadTableWriter const &) = delete;\n ReadTableWriter & operator=(ReadTableWriter const &) = delete;\n ~ReadTableWriter();\n\n /// \\brief Add a read to the read table, adding to the current batch.\n /// \\param read_data The data to add as a read.\n /// \\param signal List of signal table row indices that belong to this read.\n /// \\param signal_duration The length of the read in samples.\n /// \\returns The row index of the inserted read, or a status on failure.\n Result add_read(\n ReadData const & read_data,\n gsl::span const & signal,\n std::uint64_t signal_duration);\n\n /// \\brief Close this writer, signaling no further data will be written to the writer.\n Status close();\n\n /// \\brief Reserve space for future row writes, called automatically when a flush occurs.\n Status reserve_rows();\n\n /// \\brief Find the schema for the table\n std::shared_ptr const & schema() const { return m_schema; }\n\n /// \\brief Flush passed data into the writer as a record batch.\n Status write_batch(arrow::RecordBatch const &);\n\nprivate:\n /// \\brief Flush buffered data into the writer as a record batch.\n Status write_batch();\n\n std::shared_ptr m_schema;\n std::shared_ptr m_field_locations;\n std::size_t m_table_batch_size;\n\n std::shared_ptr m_writer;\n\n ReadTableSchemaDescription::FieldBuilders m_field_builders;\n\n std::size_t m_written_batched_row_count = 0;\n std::size_t m_current_batch_row_count = 0;\n std::shared_ptr m_output_stream;\n};\n\n/// \\brief Make a new writer for a read table.\n/// \\param sink Sink to be used for output of the table.\n/// \\param metadata Metadata to be applied to the table schema.\n/// \\param table_batch_size The size of each batch written for the table.\n/// \\param pool Pool to be used for building table in memory.\n/// \\returns The writer for the new table.\nPOD5_FORMAT_EXPORT Result make_read_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n std::shared_ptr const & pore_writer,\n std::shared_ptr const & end_reason_writer,\n std::shared_ptr const & run_info_writer,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_writer_utils.cpp", "content": "#include \"pod5_format/read_table_writer_utils.h\"\n\n#include \"pod5_format/read_table_schema.h\"\n\n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nnamespace detail {\n\narrow::Result> get_array_data(\n std::shared_ptr const & type,\n StringDictionaryKeyBuilder const & builder,\n std::size_t expected_length)\n{\n auto const value_data = builder.get_string_data();\n if (!value_data) {\n return Status::Invalid(\"Missing array value data for dictionary\");\n }\n\n arrow::TypedBufferBuilder offset_builder;\n auto const & offset_data = builder.get_typed_offset_data();\n if (offset_data.size() != expected_length) {\n return Status::Invalid(\"Invalid size for field in struct\");\n }\n ARROW_RETURN_NOT_OK(offset_builder.Append(offset_data.data(), offset_data.size()));\n // Append final offset - size of value data.\n ARROW_RETURN_NOT_OK(offset_builder.Append(value_data->size()));\n\n std::shared_ptr offsets;\n ARROW_RETURN_NOT_OK(offset_builder.Finish(&offsets));\n\n return arrow::ArrayData::Make(type, expected_length, {nullptr, offsets, value_data}, 0, 0);\n}\n\n} // namespace detail\n\narrow::Result> make_pore_writer(arrow::MemoryPool * pool)\n{\n return std::make_shared(pool);\n}\n\narrow::Result> make_end_reason_writer(arrow::MemoryPool * pool)\n{\n std::shared_ptr end_reasons;\n {\n arrow::StringBuilder builder(pool);\n for (int end_reason = 0; end_reason <= (int)ReadEndReason::last_end_reason; ++end_reason) {\n ARROW_RETURN_NOT_OK(builder.Append(end_reason_as_string((ReadEndReason)end_reason)));\n }\n\n ARROW_RETURN_NOT_OK(builder.Finish(&end_reasons));\n }\n\n return std::make_shared(end_reasons);\n}\n\narrow::Result> make_run_info_writer(arrow::MemoryPool * pool)\n{\n return std::make_shared(pool);\n}\n\npod5::Result> DictionaryWriter::build_dictionary_array(\n std::shared_ptr const & indices)\n{\n ARROW_ASSIGN_OR_RAISE(auto res, get_value_array());\n return arrow::DictionaryArray::FromArrays(indices, res);\n}\n\nPoreWriter::PoreWriter(arrow::MemoryPool * pool) : m_builder(pool) {}\n\npod5::Result> PoreWriter::get_value_array()\n{\n ARROW_ASSIGN_OR_RAISE(auto array_data, get_array_data(arrow::utf8(), m_builder, item_count()));\n return std::make_shared(array_data);\n}\n\nstd::size_t PoreWriter::item_count() { return m_builder.length(); }\n\nEndReasonWriter::EndReasonWriter(std::shared_ptr const & end_reasons)\n: m_end_reasons(end_reasons)\n{\n}\n\npod5::Result> EndReasonWriter::get_value_array()\n{\n return m_end_reasons;\n}\n\nstd::size_t EndReasonWriter::item_count() { return m_end_reasons->length(); }\n\nRunInfoWriter::RunInfoWriter(arrow::MemoryPool * pool) : m_builder(pool) {}\n\npod5::Result> RunInfoWriter::get_value_array()\n{\n ARROW_ASSIGN_OR_RAISE(auto array_data, get_array_data(arrow::utf8(), m_builder, item_count()));\n return std::make_shared(array_data);\n}\n\nstd::size_t RunInfoWriter::item_count() { return m_builder.length(); }\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/read_table_writer_utils.h", "content": "#pragma once\n\n#include \"pod5_format/dictionary_writer.h\"\n#include \"pod5_format/expandable_buffer.h\"\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/tuple_utils.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n\nnamespace pod5 {\n\nnamespace detail {\n\nclass StringDictionaryKeyBuilder {\npublic:\n StringDictionaryKeyBuilder(arrow::MemoryPool * pool = nullptr)\n : m_offset_values(pool)\n , m_string_values(pool)\n {\n }\n\n arrow::Status init_buffer(arrow::MemoryPool * pool)\n {\n ARROW_RETURN_NOT_OK(m_offset_values.init_buffer(pool));\n return m_string_values.init_buffer(pool);\n }\n\n arrow::Status append(std::string const & value)\n {\n ARROW_RETURN_NOT_OK(m_offset_values.append(m_string_values.size()));\n return m_string_values.append_array(\n gsl::make_span(value.data(), value.size()).as_span());\n }\n\n std::size_t length() const { return m_offset_values.size(); }\n\n std::shared_ptr get_string_data() const { return m_string_values.get_buffer(); }\n\n gsl::span get_typed_offset_data() const\n {\n return m_offset_values.get_data_span();\n }\n\nprivate:\n ExpandableBuffer m_offset_values;\n ExpandableBuffer m_string_values;\n};\n\n} // namespace detail\n\nclass POD5_FORMAT_EXPORT PoreWriter : public DictionaryWriter {\npublic:\n PoreWriter(arrow::MemoryPool * pool);\n\n pod5::Result add(std::string const & pore_type)\n {\n auto const index = item_count();\n\n if (index >= std::size_t(std::numeric_limits::max())) {\n return arrow::Status::Invalid(\n \"Failed to add pore to dictionary, too many indices in file\");\n }\n\n ARROW_RETURN_NOT_OK(m_builder.append(pore_type));\n return index;\n }\n\n pod5::Result> get_value_array() override;\n std::size_t item_count() override;\n\nprivate:\n detail::StringDictionaryKeyBuilder m_builder;\n};\n\nclass POD5_FORMAT_EXPORT EndReasonWriter : public DictionaryWriter {\npublic:\n EndReasonWriter(std::shared_ptr const & end_reasons);\n\n pod5::Result lookup(ReadEndReason end_reason) const\n {\n if (end_reason > ReadEndReason::last_end_reason) {\n return pod5::Status::Invalid(\"Invalid read end reason requested\");\n }\n return EndReasonDictionaryIndex(end_reason);\n }\n\n pod5::Result> get_value_array() override;\n std::size_t item_count() override;\n\nprivate:\n std::shared_ptr m_end_reasons;\n};\n\nclass POD5_FORMAT_EXPORT RunInfoWriter : public DictionaryWriter {\npublic:\n RunInfoWriter(arrow::MemoryPool * pool);\n\n pod5::Result add(std::string const & acquisition_id)\n {\n auto const index = item_count();\n\n if (index >= std::size_t(std::numeric_limits::max())) {\n return arrow::Status::Invalid(\n \"Failed to add run info to dictionary, too many indices in file\");\n }\n\n ARROW_RETURN_NOT_OK(m_builder.append(acquisition_id));\n return index;\n }\n\n pod5::Result> get_value_array() override;\n std::size_t item_count() override;\n\nprivate:\n detail::StringDictionaryKeyBuilder m_builder;\n};\n\nPOD5_FORMAT_EXPORT arrow::Result> make_pore_writer(\n arrow::MemoryPool * pool);\n\nPOD5_FORMAT_EXPORT arrow::Result> make_end_reason_writer(\n arrow::MemoryPool * pool);\n\nPOD5_FORMAT_EXPORT arrow::Result> make_run_info_writer(\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/result.h", "content": "#pragma once\n\n#include \n\nnamespace pod5 {\n\n/// pod5::Result is just an Arrow Result right now.\ntemplate \nusing Result = arrow::Result;\nusing Status = arrow::Status;\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_reader.cpp", "content": "#include \"pod5_format/run_info_table_reader.h\"\n\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/schema_utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\ninline std::vector> value_for_map(\n std::shared_ptr const & map_array,\n std::size_t row_index)\n{\n std::size_t offset = map_array->value_offset(row_index);\n std::size_t length = map_array->value_length(row_index);\n\n auto const & keys = std::dynamic_pointer_cast(map_array->keys());\n auto const & items = std::dynamic_pointer_cast(map_array->items());\n\n std::vector> result;\n for (std::size_t i = offset; i < offset + length; ++i) {\n result.push_back(std::make_pair(keys->GetString(i), items->GetString(i)));\n }\n return result;\n}\n\nRunInfoTableRecordBatch::RunInfoTableRecordBatch(\n std::shared_ptr && batch,\n std::shared_ptr const & field_locations)\n: TableRecordBatch(std::move(batch))\n, m_field_locations(field_locations)\n{\n}\n\nRunInfoTableRecordBatch::RunInfoTableRecordBatch(RunInfoTableRecordBatch && other)\n: TableRecordBatch(std::move(other))\n{\n m_field_locations = std::move(other.m_field_locations);\n}\n\nRunInfoTableRecordBatch & RunInfoTableRecordBatch::operator=(RunInfoTableRecordBatch && other)\n{\n TableRecordBatch & base = *this;\n base = other;\n\n m_field_locations = std::move(other.m_field_locations);\n return *this;\n}\n\nResult RunInfoTableRecordBatch::columns() const\n{\n RunInfoTableRecordColumns result;\n result.table_version = m_field_locations->table_version();\n\n auto const & bat = batch();\n\n // V0 fields:\n result.acquisition_id = find_column(bat, m_field_locations->acquisition_id);\n result.acquisition_start_time = find_column(bat, m_field_locations->acquisition_start_time);\n result.adc_max = find_column(bat, m_field_locations->adc_max);\n result.adc_min = find_column(bat, m_field_locations->adc_min);\n result.context_tags = find_column(bat, m_field_locations->context_tags);\n result.experiment_name = find_column(bat, m_field_locations->experiment_name);\n result.flow_cell_id = find_column(bat, m_field_locations->flow_cell_id);\n result.flow_cell_product_code = find_column(bat, m_field_locations->flow_cell_product_code);\n result.protocol_name = find_column(bat, m_field_locations->protocol_name);\n result.protocol_run_id = find_column(bat, m_field_locations->protocol_run_id);\n result.protocol_start_time = find_column(bat, m_field_locations->protocol_start_time);\n result.sample_id = find_column(bat, m_field_locations->sample_id);\n result.sample_rate = find_column(bat, m_field_locations->sample_rate);\n result.sequencing_kit = find_column(bat, m_field_locations->sequencing_kit);\n result.sequencer_position = find_column(bat, m_field_locations->sequencer_position);\n result.sequencer_position_type = find_column(bat, m_field_locations->sequencer_position_type);\n result.software = find_column(bat, m_field_locations->software);\n result.system_name = find_column(bat, m_field_locations->system_name);\n result.system_type = find_column(bat, m_field_locations->system_type);\n result.tracking_id = find_column(bat, m_field_locations->tracking_id);\n\n return result;\n}\n\n//---------------------------------------------------------------------------------------------------------------------\n\nRunInfoTableReader::RunInfoTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n std::shared_ptr const & field_locations,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool)\n: TableReader(std::move(input_source), std::move(reader), std::move(schema_metadata), pool)\n, m_field_locations(field_locations)\n{\n}\n\nRunInfoTableReader::RunInfoTableReader(RunInfoTableReader && other)\n: TableReader(std::move(other))\n, m_field_locations(std::move(other.m_field_locations))\n{\n}\n\nRunInfoTableReader & RunInfoTableReader::operator=(RunInfoTableReader && other)\n{\n static_cast(*this) = std::move(static_cast(*this));\n m_field_locations = std::move(other.m_field_locations);\n return *this;\n}\n\nResult RunInfoTableReader::read_record_batch(std::size_t i) const\n{\n std::lock_guard l(m_batch_get_mutex);\n ARROW_ASSIGN_OR_RAISE(auto record_batch, TableReader::ReadRecordBatch(i));\n return RunInfoTableRecordBatch{std::move(record_batch), m_field_locations};\n}\n\nResult> RunInfoTableReader::find_run_info(\n std::string const & acquisition_id) const\n{\n std::lock_guard l(m_run_info_lookup_mutex);\n auto it = m_run_info_lookup.find(acquisition_id);\n if (it != m_run_info_lookup.end()) {\n return it->second;\n }\n\n ARROW_RETURN_NOT_OK(prepare_run_infos_vector());\n\n std::shared_ptr run_info = nullptr;\n std::size_t glb_run_info_index = 0;\n for (std::size_t i = 0; i < num_record_batches(); ++i) {\n ARROW_ASSIGN_OR_RAISE(auto batch, read_record_batch(i));\n auto acq_id = find_column(batch.batch(), m_field_locations->acquisition_id);\n\n for (std::size_t j = 0; j < batch.num_rows(); ++j) {\n if (acq_id->Value(j) == acquisition_id) {\n ARROW_ASSIGN_OR_RAISE(\n run_info, load_run_info_from_batch(batch, j, glb_run_info_index++));\n break;\n }\n }\n\n if (run_info) {\n break;\n }\n }\n\n if (!run_info) {\n return arrow::Status::Invalid(\n \"Failed to find acquisition id '\", acquisition_id, \"' in run info table\");\n }\n\n return run_info;\n}\n\nResult> RunInfoTableReader::get_run_info(std::size_t index) const\n{\n ARROW_RETURN_NOT_OK(prepare_run_infos_vector());\n\n if (index < 0 || index >= m_run_infos.size()) {\n return arrow::Status::IndexError(\n \"Invalid index into run infos (expected \", index, \" < \", m_run_infos.size(), \")\");\n }\n\n if (m_run_infos[index]) {\n return m_run_infos[index];\n }\n\n ARROW_ASSIGN_OR_RAISE(auto first_batch, read_record_batch(0));\n auto const batch_size = first_batch.num_rows();\n\n auto const batch_idx = index / batch_size;\n auto const batch_row = index - (batch_idx * batch_size);\n\n if (batch_idx >= num_record_batches()) {\n return Status::Invalid(\"Row outside batch bounds\");\n }\n\n ARROW_ASSIGN_OR_RAISE(auto batch, read_record_batch(batch_idx));\n\n return load_run_info_from_batch(batch, batch_row, index);\n}\n\nResult RunInfoTableReader::get_run_info_count() const\n{\n auto batch_count = num_record_batches();\n if (batch_count == 0) {\n return 0;\n }\n\n ARROW_ASSIGN_OR_RAISE(auto first_batch, read_record_batch(0));\n ARROW_ASSIGN_OR_RAISE(auto last_batch, read_record_batch(batch_count - 1));\n\n return (batch_count - 1) * first_batch.num_rows() + last_batch.num_rows();\n}\n\nResult> RunInfoTableReader::load_run_info_from_batch(\n RunInfoTableRecordBatch const & batch,\n std::size_t batch_index,\n std::size_t global_index) const\n{\n ARROW_ASSIGN_OR_RAISE(auto columns, batch.columns());\n\n auto acquisition_id = columns.acquisition_id->GetString(batch_index);\n auto run_info = std::make_shared(\n acquisition_id,\n columns.acquisition_start_time->Value(batch_index),\n columns.adc_max->Value(batch_index),\n columns.adc_min->Value(batch_index),\n value_for_map(columns.context_tags, batch_index),\n columns.experiment_name->GetString(batch_index),\n columns.flow_cell_id->GetString(batch_index),\n columns.flow_cell_product_code->GetString(batch_index),\n columns.protocol_name->GetString(batch_index),\n columns.protocol_run_id->GetString(batch_index),\n columns.protocol_start_time->Value(batch_index),\n columns.sample_id->GetString(batch_index),\n columns.sample_rate->Value(batch_index),\n columns.sequencing_kit->GetString(batch_index),\n columns.sequencer_position->GetString(batch_index),\n columns.sequencer_position_type->GetString(batch_index),\n columns.software->GetString(batch_index),\n columns.system_name->GetString(batch_index),\n columns.system_type->GetString(batch_index),\n value_for_map(columns.tracking_id, batch_index));\n\n // Cache run info for later retrieval by index:\n m_run_infos[global_index] = run_info;\n m_run_info_lookup[acquisition_id] = run_info;\n return run_info;\n}\n\narrow::Status RunInfoTableReader::prepare_run_infos_vector() const\n{\n if (m_run_infos.empty()) {\n ARROW_ASSIGN_OR_RAISE(auto row_count, get_run_info_count())\n m_run_infos.resize(row_count);\n }\n\n return Status::OK();\n}\n\n//---------------------------------------------------------------------------------------------------------------------\n\nResult make_run_info_table_reader(\n std::shared_ptr const & input,\n arrow::MemoryPool * pool)\n{\n arrow::ipc::IpcReadOptions options;\n options.memory_pool = pool;\n\n ARROW_ASSIGN_OR_RAISE(auto reader, arrow::ipc::RecordBatchFileReader::Open(input, options));\n\n auto read_metadata_key_values = reader->schema()->metadata();\n if (!read_metadata_key_values) {\n return Status::IOError(\"Missing metadata on run info table schema\");\n }\n ARROW_ASSIGN_OR_RAISE(\n auto read_metadata, read_schema_key_value_metadata(read_metadata_key_values));\n ARROW_ASSIGN_OR_RAISE(\n auto field_locations, read_run_info_table_schema(read_metadata, reader->schema()));\n\n return RunInfoTableReader(\n {input}, std::move(reader), field_locations, std::move(read_metadata), pool);\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_reader.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/read_table_utils.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/run_info_table_schema.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/table_reader.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n\n#include \n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace io {\nclass RandomAccessFile;\n}\n\nnamespace ipc {\nclass RecordBatchFileReader;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nstruct RunInfoTableRecordColumns {\n // V0 Fields\n std::shared_ptr acquisition_id;\n std::shared_ptr acquisition_start_time;\n std::shared_ptr adc_max;\n std::shared_ptr adc_min;\n std::shared_ptr context_tags;\n std::shared_ptr experiment_name;\n std::shared_ptr flow_cell_id;\n std::shared_ptr flow_cell_product_code;\n std::shared_ptr protocol_name;\n std::shared_ptr protocol_run_id;\n std::shared_ptr protocol_start_time;\n std::shared_ptr sample_id;\n std::shared_ptr sample_rate;\n std::shared_ptr sequencing_kit;\n std::shared_ptr sequencer_position;\n std::shared_ptr sequencer_position_type;\n std::shared_ptr software;\n std::shared_ptr system_name;\n std::shared_ptr system_type;\n std::shared_ptr tracking_id;\n\n TableSpecVersion table_version;\n};\n\nclass POD5_FORMAT_EXPORT RunInfoTableRecordBatch : public TableRecordBatch {\npublic:\n RunInfoTableRecordBatch(\n std::shared_ptr && batch,\n std::shared_ptr const & field_locations);\n RunInfoTableRecordBatch(RunInfoTableRecordBatch &&);\n RunInfoTableRecordBatch & operator=(RunInfoTableRecordBatch &&);\n\n Result columns() const;\n\nprivate:\n std::shared_ptr m_field_locations;\n};\n\nclass POD5_FORMAT_EXPORT RunInfoTableReader : public TableReader {\npublic:\n RunInfoTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n std::shared_ptr const & field_locations,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool);\n\n RunInfoTableReader(RunInfoTableReader && other);\n RunInfoTableReader & operator=(RunInfoTableReader && other);\n\n Result read_record_batch(std::size_t i) const;\n\n Result> find_run_info(\n std::string const & acquisition_id) const;\n\n Result> get_run_info(std::size_t index) const;\n Result get_run_info_count() const;\n\nprivate:\n Result> load_run_info_from_batch(\n RunInfoTableRecordBatch const & batch,\n std::size_t batch_index,\n std::size_t global_index) const;\n arrow::Status prepare_run_infos_vector() const;\n\n std::shared_ptr m_field_locations;\n mutable std::mutex m_batch_get_mutex;\n mutable std::unordered_map> m_run_info_lookup;\n mutable std::vector> m_run_infos;\n mutable std::mutex m_run_info_lookup_mutex;\n};\n\nPOD5_FORMAT_EXPORT Result make_run_info_table_reader(\n std::shared_ptr const & sink,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_schema.cpp", "content": "#include \"pod5_format/run_info_table_schema.h\"\n\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/types.h\"\n\nnamespace pod5 {\n\nRunInfoTableSchemaDescription::RunInfoTableSchemaDescription()\n: SchemaDescriptionBase(RunInfoTableSpecVersion::latest())\n// V0 Fields\n, acquisition_id(this, \"acquisition_id\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, acquisition_start_time(\n this,\n \"acquisition_start_time\",\n arrow::timestamp(arrow::TimeUnit::MILLI, \"UTC\"),\n RunInfoTableSpecVersion::v0())\n, adc_max(this, \"adc_max\", arrow::int16(), RunInfoTableSpecVersion::v0())\n, adc_min(this, \"adc_min\", arrow::int16(), RunInfoTableSpecVersion::v0())\n, context_tags(\n this,\n \"context_tags\",\n arrow::map(arrow::utf8(), arrow::utf8()),\n RunInfoTableSpecVersion::v0())\n, experiment_name(this, \"experiment_name\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, flow_cell_id(this, \"flow_cell_id\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, flow_cell_product_code(\n this,\n \"flow_cell_product_code\",\n arrow::utf8(),\n RunInfoTableSpecVersion::v0())\n, protocol_name(this, \"protocol_name\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, protocol_run_id(this, \"protocol_run_id\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, protocol_start_time(\n this,\n \"protocol_start_time\",\n arrow::timestamp(arrow::TimeUnit::MILLI, \"UTC\"),\n RunInfoTableSpecVersion::v0())\n, sample_id(this, \"sample_id\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, sample_rate(this, \"sample_rate\", arrow::uint16(), RunInfoTableSpecVersion::v0())\n, sequencing_kit(this, \"sequencing_kit\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, sequencer_position(this, \"sequencer_position\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, sequencer_position_type(\n this,\n \"sequencer_position_type\",\n arrow::utf8(),\n RunInfoTableSpecVersion::v0())\n, software(this, \"software\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, system_name(this, \"system_name\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, system_type(this, \"system_type\", arrow::utf8(), RunInfoTableSpecVersion::v0())\n, tracking_id(\n this,\n \"tracking_id\",\n arrow::map(arrow::utf8(), arrow::utf8()),\n RunInfoTableSpecVersion::v0())\n{\n}\n\nTableSpecVersion RunInfoTableSchemaDescription::table_version_from_file_version(\n Version file_version) const\n{\n return RunInfoTableSpecVersion::latest();\n}\n\nResult> read_run_info_table_schema(\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const & schema)\n{\n auto result = std::make_shared();\n ARROW_RETURN_NOT_OK(\n RunInfoTableSchemaDescription::read_schema(result, schema_metadata, schema));\n\n return result;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_schema.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_utils.h\"\n#include \"pod5_format/tuple_utils.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n\nnamespace arrow {\nclass KeyValueMetadata;\nclass Schema;\nclass DataType;\nclass StructType;\n} // namespace arrow\n\nnamespace pod5 {\n\nclass RunInfoTableSpecVersion {\npublic:\n static TableSpecVersion v0() { return TableSpecVersion::first_version(); }\n\n static TableSpecVersion latest() { return v0(); }\n};\n\nclass RunInfoTableSchemaDescription : public SchemaDescriptionBase {\npublic:\n RunInfoTableSchemaDescription();\n\n RunInfoTableSchemaDescription(RunInfoTableSchemaDescription const &) = delete;\n RunInfoTableSchemaDescription & operator=(RunInfoTableSchemaDescription const &) = delete;\n\n TableSpecVersion table_version_from_file_version(Version file_version) const override;\n\n Field<0, arrow::StringArray> acquisition_id;\n Field<1, arrow::TimestampArray> acquisition_start_time;\n Field<2, arrow::Int16Array> adc_max;\n Field<3, arrow::Int16Array> adc_min;\n Field<4, arrow::MapArray> context_tags;\n Field<5, arrow::StringArray> experiment_name;\n Field<6, arrow::StringArray> flow_cell_id;\n Field<7, arrow::StringArray> flow_cell_product_code;\n Field<8, arrow::StringArray> protocol_name;\n Field<9, arrow::StringArray> protocol_run_id;\n Field<10, arrow::TimestampArray> protocol_start_time;\n Field<11, arrow::StringArray> sample_id;\n Field<12, arrow::UInt16Array> sample_rate;\n Field<13, arrow::StringArray> sequencing_kit;\n Field<14, arrow::StringArray> sequencer_position;\n Field<15, arrow::StringArray> sequencer_position_type;\n Field<16, arrow::StringArray> software;\n Field<17, arrow::StringArray> system_name;\n Field<18, arrow::StringArray> system_type;\n Field<19, arrow::MapArray> tracking_id;\n\n // Field Builders only for fields we write in newly generated files.\n // Should not include fields which are removed in the latest version:\n using FieldBuilders = FieldBuilder<\n // V0 fields\n decltype(acquisition_id),\n decltype(acquisition_start_time),\n decltype(adc_max),\n decltype(adc_min),\n decltype(context_tags),\n decltype(experiment_name),\n decltype(flow_cell_id),\n decltype(flow_cell_product_code),\n decltype(protocol_name),\n decltype(protocol_run_id),\n decltype(protocol_start_time),\n decltype(sample_id),\n decltype(sample_rate),\n decltype(sequencing_kit),\n decltype(sequencer_position),\n decltype(sequencer_position_type),\n decltype(software),\n decltype(system_name),\n decltype(system_type),\n decltype(tracking_id)>;\n};\n\nPOD5_FORMAT_EXPORT Result>\nread_run_info_table_schema(\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const &);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_writer.cpp", "content": "#include \"pod5_format/run_info_table_writer.h\"\n\n#include \"pod5_format/file_output_stream.h\"\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"pod5_format/read_table_utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nRunInfoTableWriter::RunInfoTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n std::shared_ptr const & field_locations,\n std::shared_ptr const & output_stream,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool)\n: m_schema(schema)\n, m_field_locations(field_locations)\n, m_output_stream{output_stream}\n, m_table_batch_size(table_batch_size)\n, m_writer(std::move(writer))\n, m_field_builders(m_field_locations, pool)\n{\n}\n\nRunInfoTableWriter::RunInfoTableWriter(RunInfoTableWriter && other) = default;\nRunInfoTableWriter & RunInfoTableWriter::operator=(RunInfoTableWriter &&) = default;\n\nRunInfoTableWriter::~RunInfoTableWriter()\n{\n if (m_writer) {\n (void)close();\n }\n}\n\nResult RunInfoTableWriter::add_run_info(RunInfoData const & run_info_data)\n{\n POD5_TRACE_FUNCTION();\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_RETURN_NOT_OK(reserve_rows());\n\n auto row_id = m_written_batched_row_count + m_current_batch_row_count;\n ARROW_RETURN_NOT_OK(m_field_builders.append(\n // V0 Fields\n run_info_data.acquisition_id,\n run_info_data.acquisition_start_time,\n run_info_data.adc_max,\n run_info_data.adc_min,\n run_info_data.context_tags,\n run_info_data.experiment_name,\n run_info_data.flow_cell_id,\n run_info_data.flow_cell_product_code,\n run_info_data.protocol_name,\n run_info_data.protocol_run_id,\n run_info_data.protocol_start_time,\n run_info_data.sample_id,\n run_info_data.sample_rate,\n run_info_data.sequencing_kit,\n run_info_data.sequencer_position,\n run_info_data.sequencer_position_type,\n run_info_data.software,\n run_info_data.system_name,\n run_info_data.system_type,\n run_info_data.tracking_id));\n\n ++m_current_batch_row_count;\n\n if (m_current_batch_row_count >= m_table_batch_size) {\n ARROW_RETURN_NOT_OK(write_batch());\n }\n return row_id;\n}\n\nStatus RunInfoTableWriter::close()\n{\n // Check for already closed\n if (!m_writer) {\n return Status::OK();\n }\n\n ARROW_RETURN_NOT_OK(write_batch());\n ARROW_RETURN_NOT_OK(m_writer->Close());\n m_writer = nullptr;\n return Status::OK();\n}\n\nStatus RunInfoTableWriter::write_batch(arrow::RecordBatch const & record_batch)\n{\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus RunInfoTableWriter::write_batch()\n{\n POD5_TRACE_FUNCTION();\n if (m_current_batch_row_count == 0) {\n return Status::OK();\n }\n\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_ASSIGN_OR_RAISE(auto columns, m_field_builders.finish_columns());\n\n auto const record_batch =\n arrow::RecordBatch::Make(m_schema, m_current_batch_row_count, std::move(columns));\n\n m_written_batched_row_count += m_current_batch_row_count;\n m_current_batch_row_count = 0;\n\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(*record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus RunInfoTableWriter::reserve_rows()\n{\n // Only reserve if we have not already reserved (at the start of a batch)\n if (m_current_batch_row_count > 0) {\n return arrow::Status::OK();\n }\n\n return m_field_builders.reserve(m_table_batch_size);\n}\n\nResult make_run_info_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool)\n{\n auto field_locations = std::make_shared();\n auto schema = field_locations->make_writer_schema(metadata);\n\n arrow::ipc::IpcWriteOptions options;\n options.memory_pool = pool;\n\n ARROW_ASSIGN_OR_RAISE(auto writer, arrow::ipc::MakeFileWriter(sink, schema, options, metadata));\n\n auto run_info_table_writer = RunInfoTableWriter(\n std::move(writer), std::move(schema), field_locations, sink, table_batch_size, pool);\n\n return run_info_table_writer;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/run_info_table_writer.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/run_info_table_schema.h\"\n#include \"pod5_format/schema_field_builder.h\"\n\n#include \n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace ipc {\nclass RecordBatchWriter;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nclass FileOutputStream;\nclass RunInfoData;\n\nclass POD5_FORMAT_EXPORT RunInfoTableWriter {\npublic:\n RunInfoTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n std::shared_ptr const & field_locations,\n std::shared_ptr const & output_stream,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool);\n RunInfoTableWriter(RunInfoTableWriter &&);\n RunInfoTableWriter & operator=(RunInfoTableWriter &&);\n RunInfoTableWriter(RunInfoTableWriter const &) = delete;\n RunInfoTableWriter & operator=(RunInfoTableWriter const &) = delete;\n ~RunInfoTableWriter();\n\n /// \\brief Add a run info to the table, adding to the current batch.\n /// \\param run_info_data The run info data to add.\n /// \\returns The row index of the inserted read, or a status on failure.\n Result add_run_info(RunInfoData const & run_info_data);\n\n /// \\brief Close this writer, signaling no further data will be written to the writer.\n Status close();\n\n /// \\brief Reserve space for future row writes, called automatically when a flush occurs.\n Status reserve_rows();\n\n /// \\brief Find the schema for the table\n std::shared_ptr const & schema() const { return m_schema; }\n\n /// \\brief Flush passed data into the writer as a record batch.\n Status write_batch(arrow::RecordBatch const &);\n\nprivate:\n /// \\brief Flush buffered data into the writer as a record batch.\n Status write_batch();\n\n std::shared_ptr m_schema;\n std::shared_ptr m_field_locations;\n std::shared_ptr m_output_stream;\n std::size_t m_table_batch_size;\n\n std::shared_ptr m_writer;\n\n RunInfoTableSchemaDescription::FieldBuilders m_field_builders;\n\n std::size_t m_written_batched_row_count = 0;\n std::size_t m_current_batch_row_count = 0;\n};\n\n/// \\brief Make a new writer for a read table.\n/// \\param sink Sink to be used for output of the table.\n/// \\param metadata Metadata to be applied to the table schema.\n/// \\param table_batch_size The size of each batch written for the table.\n/// \\param pool Pool to be used for building table in memory.\n/// \\returns The writer for the new table.\nPOD5_FORMAT_EXPORT Result make_run_info_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/schema_field_builder.h", "content": "#pragma once\n\n#include \"pod5_format/dictionary_writer.h\"\n#include \"pod5_format/read_table_schema.h\"\n\n#include \n#include \n#include \n\nnamespace pod5 {\n\nclass DictionaryWriter;\n\nnamespace detail {\ntemplate \nclass BuilderHelper;\ntemplate \nclass ListBuilderHelper;\n\ntemplate <>\nclass BuilderHelper : public arrow::FixedSizeBinaryBuilder {\npublic:\n BuilderHelper(std::shared_ptr const & uuid_type, arrow::MemoryPool * pool)\n : arrow::FixedSizeBinaryBuilder(find_storage_type(uuid_type), pool)\n {\n assert(byte_width() == 16);\n }\n\n static std::shared_ptr find_storage_type(\n std::shared_ptr const & uuid_type)\n {\n assert(uuid_type->id() == arrow::Type::EXTENSION);\n auto const & uuid_extension = static_cast(*uuid_type);\n return uuid_extension.storage_type();\n }\n\n arrow::Status Append(Uuid const & uuid)\n {\n return static_cast(this)->Append(uuid.data());\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::FloatBuilder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::FloatBuilder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::UInt8Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::UInt8Builder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::UInt16Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::UInt16Builder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::Int16Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::Int16Builder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::UInt32Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::UInt32Builder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::UInt64Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::UInt64Builder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::BooleanBuilder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::BooleanBuilder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper> : public arrow::TimestampBuilder {\npublic:\n BuilderHelper(std::shared_ptr const & type, arrow::MemoryPool * pool)\n : arrow::TimestampBuilder(type, pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::StringBuilder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::StringBuilder(pool)\n {\n }\n};\n\ntemplate <>\nclass BuilderHelper {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : m_key_builder(std::make_shared(pool))\n , m_item_builder(std::make_shared(pool))\n , m_map_builder(pool, m_key_builder, m_item_builder)\n {\n }\n\n arrow::Status Finish(std::shared_ptr * dest)\n {\n return m_map_builder.Finish(dest);\n }\n\n arrow::Status Reserve(std::size_t rows) { return m_map_builder.Reserve(rows); }\n\n arrow::Status Append(std::vector> const & items)\n {\n ARROW_RETURN_NOT_OK(m_map_builder.Append()); // start new slot\n for (auto const & pair : items) {\n ARROW_RETURN_NOT_OK(m_key_builder->Append(pair.first));\n ARROW_RETURN_NOT_OK(m_item_builder->Append(pair.second));\n }\n return arrow::Status::OK();\n }\n\nprivate:\n std::shared_ptr m_key_builder;\n std::shared_ptr m_item_builder;\n arrow::MapBuilder m_map_builder;\n};\n\ntemplate <>\nclass BuilderHelper : public arrow::Int16Builder {\npublic:\n BuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : arrow::Int16Builder(pool)\n {\n }\n\n void set_dict_writer(std::shared_ptr const & writer)\n {\n m_dict_writer = writer;\n }\n\n arrow::Status Finish(std::shared_ptr * dest)\n {\n arrow::Int16Builder * index_builder = this;\n ARROW_ASSIGN_OR_RAISE(auto indices, index_builder->Finish());\n ARROW_ASSIGN_OR_RAISE(*dest, m_dict_writer->build_dictionary_array(indices));\n return arrow::Status::OK();\n }\n\nprivate:\n std::shared_ptr m_dict_writer;\n};\n\ntemplate \nclass ListBuilderHelper {\npublic:\n ListBuilderHelper(std::shared_ptr const &, arrow::MemoryPool * pool)\n : m_array_builder(std::make_shared>(nullptr, pool))\n , m_builder(std::make_unique(pool, m_array_builder))\n {\n }\n\n arrow::Status Reserve(std::size_t rows)\n {\n ARROW_RETURN_NOT_OK(m_builder->Reserve(rows));\n return m_array_builder->Reserve(rows);\n }\n\n arrow::Status Finish(std::shared_ptr * dest) { return m_builder->Finish(dest); }\n\n template \n arrow::Status Append(Items const & items)\n {\n ARROW_RETURN_NOT_OK(m_builder->Append()); // start new slot\n return m_array_builder->AppendValues(items.data(), items.size());\n }\n\nprivate:\n std::shared_ptr> m_array_builder;\n std::unique_ptr m_builder;\n};\n\n} // namespace detail\n\ntemplate \nclass FieldBuilder {\npublic:\n using BuilderTuple = std::tuple;\n\n template \n FieldBuilder(std::shared_ptr const & desc_base, arrow::MemoryPool * pool)\n : m_builders(\n typename Args::BuilderType(\n desc_base->fields()[Args::WriteIndex::value]->datatype(),\n pool)...)\n {\n }\n\n template \n std::tuple_element_t & get_builder(FieldType)\n {\n return std::get(m_builders);\n }\n\n arrow::Result>> finish_columns()\n {\n arrow::Status result;\n std::vector> columns;\n columns.resize(std::tuple_size::value);\n\n detail::for_each_in_tuple(m_builders, [&](auto & element, std::size_t index) {\n if (result.ok()) {\n result = element.Finish(&columns[index]);\n assert(columns[index] || !result.ok());\n }\n });\n\n if (!result.ok()) {\n return result;\n }\n\n return columns;\n }\n\n arrow::Status reserve(std::size_t row_count)\n {\n arrow::Status result;\n detail::for_each_in_tuple(m_builders, [&](auto & element, std::size_t _) {\n if (result.ok()) {\n result = element.Reserve(row_count);\n }\n });\n return result;\n }\n\n template \n arrow::Status append(AppendArgs const &... args)\n {\n auto args_list = std::forward_as_tuple(args...);\n\n arrow::Status result;\n for_each_in_tuple_zipped(\n m_builders, args_list, [&](auto & builder, auto & item, std::size_t _) {\n if (result.ok()) {\n result = builder.Append(item);\n }\n });\n return result;\n }\n\nprivate:\n BuilderTuple m_builders;\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/schema_metadata.cpp", "content": "#include \"pod5_format/schema_metadata.h\"\n\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n\n#include \n\nnamespace pod5 {\n\nResult parse_version_number(std::string const & ver)\n{\n std::uint16_t components[3];\n std::size_t component_index = 0;\n std::size_t last_char_index = 0;\n std::size_t char_index = 0;\n\n auto parse_component = [&](std::size_t last_char_index, std::size_t char_index) {\n auto const component_str =\n std::string(ver.data() + last_char_index, ver.data() + char_index);\n\n std::size_t pos = 0;\n int val = std::stoi(component_str, &pos);\n\n if (pos != (char_index - last_char_index)) {\n throw std::runtime_error(\"Invalid remaining characters after version number\");\n }\n\n return val;\n };\n\n try {\n while (char_index < ver.size()) {\n if (ver[char_index] == '.') {\n if (component_index > 3) {\n return Status::Invalid(\"Invalid component count\");\n }\n components[component_index] = parse_component(last_char_index, char_index);\n\n last_char_index = char_index + 1;\n component_index += 1;\n }\n char_index += 1;\n }\n\n // extract the final component\n if (component_index != 2) {\n return Status::Invalid(\"Invalid component count\");\n }\n components[2] = parse_component(last_char_index, char_index);\n } catch (std::exception const & e) {\n return Status::Invalid(e.what());\n }\n\n return Version{components[0], components[1], components[2]};\n}\n\nVersion current_build_version_number()\n{\n return Version(Pod5MajorVersion, Pod5MinorVersion, Pod5RevVersion);\n}\n\nResult> make_schema_key_value_metadata(\n SchemaMetadataDescription const & schema_metadata)\n{\n if (schema_metadata.writing_software.empty()) {\n return Status::Invalid(\"Expected writing_software to be specified for metadata\");\n }\n\n if (schema_metadata.writing_pod5_version == Version{}) {\n return Status::Invalid(\"Expected writing_pod5_version to be specified for metadata\");\n }\n\n if (schema_metadata.file_identifier == Uuid{}) {\n return Status::Invalid(\"Expected file_identifier to be specified for metadata\");\n }\n\n return arrow::KeyValueMetadata::Make(\n {\"MINKNOW:file_identifier\", \"MINKNOW:software\", \"MINKNOW:pod5_version\"},\n {to_string(schema_metadata.file_identifier),\n schema_metadata.writing_software,\n schema_metadata.writing_pod5_version.to_string()});\n}\n\nResult read_schema_key_value_metadata(\n std::shared_ptr const & key_value_metadata)\n{\n ARROW_ASSIGN_OR_RAISE(\n auto file_identifier_str, key_value_metadata->Get(\"MINKNOW:file_identifier\"));\n ARROW_ASSIGN_OR_RAISE(auto software_str, key_value_metadata->Get(\"MINKNOW:software\"));\n ARROW_ASSIGN_OR_RAISE(auto pod5_version_str, key_value_metadata->Get(\"MINKNOW:pod5_version\"));\n ARROW_ASSIGN_OR_RAISE(auto pod5_version, parse_version_number(pod5_version_str));\n\n auto const file_identifier = Uuid::from_string(file_identifier_str);\n if (!file_identifier) {\n return Status::IOError(\n \"Schema file_identifier metadata not uuid form: '\", file_identifier_str, \"'\");\n }\n\n return SchemaMetadataDescription{*file_identifier, software_str, pod5_version};\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/schema_metadata.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n\nnamespace arrow {\nclass KeyValueMetadata;\n}\n\nnamespace pod5 {\n\nclass Version {\npublic:\n Version() : m_version(0, 0, 0) {}\n\n Version(std::uint16_t major, std::uint16_t minor, std::uint16_t revision)\n : m_version(major, minor, revision)\n {\n }\n\n bool operator<(Version const & in) const { return m_version < in.m_version; }\n\n bool operator>(Version const & in) const { return m_version > in.m_version; }\n\n bool operator==(Version const & in) const { return m_version == in.m_version; }\n\n bool operator!=(Version const & in) const { return m_version != in.m_version; }\n\n std::string to_string() const\n {\n return std::to_string(std::get<0>(m_version)) + \".\" + std::to_string(std::get<1>(m_version))\n + \".\" + std::to_string(std::get<2>(m_version));\n }\n\n std::uint16_t major_version() const { return std::get<0>(m_version); }\n\n std::uint16_t minor_version() const { return std::get<1>(m_version); }\n\n std::uint16_t revision_version() const { return std::get<2>(m_version); }\n\nprivate:\n std::tuple m_version;\n};\n\nPOD5_FORMAT_EXPORT Result parse_version_number(std::string const & ver);\nPOD5_FORMAT_EXPORT Version current_build_version_number();\n\nstruct SchemaMetadataDescription {\n Uuid file_identifier;\n std::string writing_software;\n Version writing_pod5_version;\n};\n\nPOD5_FORMAT_EXPORT Result>\nmake_schema_key_value_metadata(SchemaMetadataDescription const & schema_metadata);\n\nPOD5_FORMAT_EXPORT Result read_schema_key_value_metadata(\n std::shared_ptr const & key_value_metadata);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/schema_utils.cpp", "content": "#include \"pod5_format/schema_utils.h\"\n\nnamespace pod5 {\n\n/// \\brief Make a new schema for a read table to be written (will only contain fields which are written in the latest version).\n/// \\param metadata Metadata to be applied to the schema.\n/// \\returns The schema for a read table.\nstd::shared_ptr SchemaDescriptionBase::make_writer_schema(\n std::shared_ptr const & metadata) const\n{\n auto const latest_version = latest_table_version();\n arrow::FieldVector writer_fields;\n for (auto & field : fields()) {\n if (field->removed_table_spec_version() > latest_version) {\n writer_fields.emplace_back(arrow::field(field->name(), field->datatype()));\n }\n }\n return arrow::schema(writer_fields, metadata);\n}\n\nStatus SchemaDescriptionBase::read_schema(\n std::shared_ptr dest_schema,\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const & schema)\n{\n dest_schema->m_table_spec_version =\n dest_schema->table_version_from_file_version(schema_metadata.writing_pod5_version);\n\n for (auto & field : dest_schema->fields()) {\n if (dest_schema->table_version() < field->added_table_spec_version()\n || dest_schema->table_version() >= field->removed_table_spec_version())\n {\n continue;\n }\n\n auto const & datatype = field->datatype();\n int field_index = 0;\n if (datatype->id() == arrow::Type::DICTIONARY) {\n auto const & dict_type = static_cast(*datatype);\n if (dict_type.value_type()->id() == arrow::Type::STRUCT) {\n std::shared_ptr value_type;\n ARROW_ASSIGN_OR_RAISE(\n field_index,\n find_dict_field(schema, field->name().c_str(), arrow::int16(), &value_type));\n } else {\n std::shared_ptr value_type;\n ARROW_ASSIGN_OR_RAISE(\n field_index,\n find_dict_field(schema, field->name().c_str(), arrow::int16(), &value_type));\n }\n } else {\n ARROW_ASSIGN_OR_RAISE(field_index, find_field(schema, field->name().c_str(), datatype));\n }\n field->set_field_index(field_index);\n }\n\n return arrow::Status::OK();\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/schema_utils.h", "content": "#pragma once\n\n#include \"pod5_format/schema_metadata.h\"\n\n#include \n#include \n\nnamespace pod5 {\n\ninline arrow::Result find_field_untyped(\n std::shared_ptr const & schema,\n char const * name)\n{\n auto const field_idx = schema->GetFieldIndex(name);\n if (field_idx == -1) {\n return Status::TypeError(\"Schema missing field '\", name, \"'\");\n }\n\n return field_idx;\n}\n\ninline arrow::Result find_field(\n std::shared_ptr const & schema,\n char const * name,\n std::shared_ptr const & expected_data_type)\n{\n ARROW_ASSIGN_OR_RAISE(auto field_idx, find_field_untyped(schema, name));\n\n auto const field = schema->field(field_idx);\n auto const type = field->type();\n\n if (!type->Equals(expected_data_type)) {\n return Status::TypeError(\n \"Schema field '\", name, \"' is incorrect type: '\", type->name(), \"'\");\n }\n\n return field_idx;\n}\n\ntemplate \ninline arrow::Result find_dict_field(\n std::shared_ptr const & schema,\n char const * name,\n std::shared_ptr const & index_type,\n std::shared_ptr * value_type)\n{\n ARROW_ASSIGN_OR_RAISE(auto field_idx, find_field_untyped(schema, name));\n\n auto const field = schema->field(field_idx);\n auto const type = std::dynamic_pointer_cast(field->type());\n if (!type) {\n return Status::TypeError(\"Dictionary field was unexpected type: \", field->type()->name());\n }\n\n if (!type->index_type()->Equals(index_type)) {\n return Status::TypeError(\n \"Schema field '\", name, \"' is incorrect type: '\", type->name(), \"'\");\n }\n\n *value_type = std::dynamic_pointer_cast(type->value_type());\n if (!*value_type) {\n return Status::TypeError(\n \"Dictionary value was unexpected type: \", type->value_type()->name());\n }\n return field_idx;\n}\n\ntemplate \nstd::shared_ptr find_column(\n std::shared_ptr const & batch,\n FieldType const & field)\n{\n auto field_base = batch->column(field.field_index());\n return std::static_pointer_cast(std::move(field_base));\n}\n\nclass FieldBase;\n\nenum class SpecialFieldValues : int {\n InvalidField = -1,\n};\n\nclass TableSpecVersion {\npublic:\n using UnderlyingType = std::uint8_t;\n\n TableSpecVersion() : m_version(std::numeric_limits::max()) {}\n\n static TableSpecVersion first_version() { return TableSpecVersion(0); }\n\n static TableSpecVersion unknown_version() { return TableSpecVersion(); }\n\n static TableSpecVersion at_version(UnderlyingType version) { return TableSpecVersion(version); }\n\n UnderlyingType as_int() const { return m_version; }\n\n bool operator<(TableSpecVersion const & other) const { return m_version < other.m_version; }\n\n bool operator>(TableSpecVersion const & other) const { return m_version > other.m_version; }\n\n bool operator<=(TableSpecVersion const & other) const { return m_version <= other.m_version; }\n\n bool operator>=(TableSpecVersion const & other) const { return m_version >= other.m_version; }\n\nprivate:\n TableSpecVersion(UnderlyingType version) : m_version(version) {}\n\n UnderlyingType m_version;\n};\n\nclass SchemaDescriptionBase {\npublic:\n SchemaDescriptionBase(TableSpecVersion version) : m_table_spec_version(version) {}\n\n virtual ~SchemaDescriptionBase() = default;\n\n void add_field(FieldBase * field) { m_fields.push_back(field); }\n\n std::vector const & fields() { return m_fields; }\n\n std::vector const & fields() const\n {\n return reinterpret_cast const &>(m_fields);\n }\n\n TableSpecVersion latest_table_version() const\n {\n return table_version_from_file_version(current_build_version_number());\n }\n\n virtual TableSpecVersion table_version_from_file_version(Version file_version) const = 0;\n\n TableSpecVersion table_version() const { return m_table_spec_version; }\n\n /// \\brief Make a new schema for a read table to be written (will only contain fields which are written in the latest version).\n /// \\param metadata Metadata to be applied to the schema.\n /// \\returns The schema for a read table.\n std::shared_ptr make_writer_schema(\n std::shared_ptr const & metadata) const;\n\n static Status read_schema(\n std::shared_ptr dest_schema,\n SchemaMetadataDescription const & schema_metadata,\n std::shared_ptr const & schema);\n\nprivate:\n std::vector m_fields;\n TableSpecVersion m_table_spec_version;\n};\n\nnamespace detail {\ntemplate \nclass BuilderHelper;\ntemplate \nclass ListBuilderHelper;\n} // namespace detail\n\nclass FieldBase {\npublic:\n FieldBase(\n SchemaDescriptionBase * owner,\n int field_index,\n std::string name,\n std::shared_ptr const & datatype,\n TableSpecVersion added_table_spec_version = TableSpecVersion::first_version(),\n TableSpecVersion removed_table_spec_version = TableSpecVersion::unknown_version())\n : m_name(name)\n , m_datatype(datatype)\n , m_field_index(field_index)\n , m_added_table_spec_version(added_table_spec_version)\n , m_removed_table_spec_version(removed_table_spec_version)\n {\n owner->add_field(this);\n }\n\n std::string const & name() const { return m_name; }\n\n std::shared_ptr const & datatype() const { return m_datatype; }\n\n int field_index() const { return m_field_index; }\n\n TableSpecVersion added_table_spec_version() const { return m_added_table_spec_version; }\n\n TableSpecVersion removed_table_spec_version() const { return m_removed_table_spec_version; }\n\n void set_field_index(int index) { m_field_index = index; }\n\n bool found_field() const { return m_field_index != (int)SpecialFieldValues::InvalidField; }\n\nprivate:\n std::string m_name;\n std::shared_ptr m_datatype;\n int m_field_index = (int)SpecialFieldValues::InvalidField;\n TableSpecVersion m_added_table_spec_version;\n TableSpecVersion m_removed_table_spec_version;\n};\n\ntemplate \nstruct Field : public FieldBase {\n using WriteIndex = std::integral_constant;\n using ArrayType = ArrayType_;\n using BuilderType = detail::BuilderHelper;\n\n Field(\n SchemaDescriptionBase * owner,\n std::string name,\n std::shared_ptr const & datatype,\n TableSpecVersion added_table_spec_version = TableSpecVersion::first_version(),\n TableSpecVersion removed_table_spec_version = TableSpecVersion::unknown_version())\n : FieldBase(\n owner,\n WriteIndex::value,\n name,\n datatype,\n added_table_spec_version,\n removed_table_spec_version)\n {\n }\n};\n\ntemplate \nstruct ListField : public Field {\n using ElementType = ElementType_;\n using BuilderType = detail::ListBuilderHelper;\n\n ListField(\n SchemaDescriptionBase * owner,\n std::string name,\n std::shared_ptr const & datatype,\n TableSpecVersion added_table_spec_version = TableSpecVersion::first_version(),\n TableSpecVersion removed_table_spec_version = TableSpecVersion::unknown_version())\n : Field(\n owner,\n name,\n datatype,\n added_table_spec_version,\n removed_table_spec_version)\n {\n }\n};\n\ntemplate \nclass FieldBuilder;\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_builder.h", "content": "#pragma once\n\n#include \"pod5_format/expandable_buffer.h\"\n#include \"pod5_format/signal_compression.h\"\n#include \"pod5_format/signal_table_utils.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nstruct UncompressedSignalBuilder {\n std::shared_ptr signal_data_builder;\n std::unique_ptr signal_builder;\n};\n\nstruct VbzSignalBuilder {\n ExpandableBuffer offset_values;\n ExpandableBuffer data_values;\n};\n\nusing SignalBuilderVariant = std::variant;\n\ninline arrow::Result make_signal_builder(\n SignalType compression_type,\n arrow::MemoryPool * pool)\n{\n if (compression_type == SignalType::UncompressedSignal) {\n auto signal_array_builder = std::make_shared(pool);\n return UncompressedSignalBuilder{\n signal_array_builder,\n std::make_unique(pool, signal_array_builder),\n };\n } else {\n VbzSignalBuilder vbz_builder;\n ARROW_RETURN_NOT_OK(vbz_builder.offset_values.init_buffer(pool));\n ARROW_RETURN_NOT_OK(vbz_builder.data_values.init_buffer(pool));\n return vbz_builder;\n }\n}\n\nnamespace visitors {\nclass reserve_rows {\npublic:\n reserve_rows(std::size_t row_count, std::size_t approx_read_samples)\n : m_row_count(row_count)\n , m_approx_read_samples(approx_read_samples)\n {\n }\n\n Status operator()(UncompressedSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.signal_builder->Reserve(m_row_count));\n return builder.signal_data_builder->Reserve(m_row_count * m_approx_read_samples);\n }\n\n Status operator()(VbzSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.offset_values.reserve(m_row_count + 1));\n return builder.data_values.reserve(m_row_count * m_approx_read_samples);\n }\n\n std::size_t m_row_count;\n std::size_t m_approx_read_samples;\n};\n\nclass append_pre_compressed_signal {\npublic:\n append_pre_compressed_signal(gsl::span const & signal) : m_signal(signal) {}\n\n Status operator()(UncompressedSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.signal_builder->Append()); // start new slot\n\n auto as_uncompressed = m_signal.as_span();\n return builder.signal_data_builder->AppendValues(\n as_uncompressed.data(), as_uncompressed.size());\n }\n\n Status operator()(VbzSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.offset_values.append(builder.data_values.size()));\n return builder.data_values.append_array(m_signal);\n }\n\n gsl::span m_signal;\n};\n\nclass append_signal {\npublic:\n append_signal(gsl::span const & signal, arrow::MemoryPool * pool)\n : m_signal(signal)\n , m_pool(pool)\n {\n }\n\n Status operator()(UncompressedSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.signal_builder->Append()); // start new slot\n return builder.signal_data_builder->AppendValues(m_signal.data(), m_signal.size());\n }\n\n Status operator()(VbzSignalBuilder & builder) const\n {\n ARROW_RETURN_NOT_OK(builder.offset_values.append(builder.data_values.size()));\n\n ARROW_ASSIGN_OR_RAISE(auto const max_size, compressed_signal_max_size(m_signal.size()));\n\n // Compress the signal in place into our buffer.\n return builder.data_values.append(\n max_size, [&](gsl::span buffer) -> arrow::Result {\n return compress_signal(m_signal, m_pool, buffer);\n });\n }\n\n gsl::span m_signal;\n arrow::MemoryPool * m_pool;\n};\n\nclass finish_column {\npublic:\n finish_column(std::shared_ptr * dest) : m_dest(dest) {}\n\n Status operator()(UncompressedSignalBuilder & builder) const\n {\n return builder.signal_builder->Finish(m_dest);\n }\n\n Status operator()(VbzSignalBuilder & builder) const\n {\n auto offsets_copy = builder.offset_values;\n ARROW_RETURN_NOT_OK(builder.offset_values.clear());\n\n auto const value_data = builder.data_values.get_buffer();\n ARROW_RETURN_NOT_OK(builder.data_values.clear());\n\n auto const length = offsets_copy.size();\n\n // Write final offset (values length)\n ARROW_RETURN_NOT_OK(offsets_copy.append(value_data->size()));\n auto const offsets = offsets_copy.get_buffer();\n\n std::shared_ptr null_bitmap;\n\n *m_dest = arrow::MakeArray(\n arrow::ArrayData::Make(vbz_signal(), length, {null_bitmap, offsets, value_data}, 0, 0));\n\n return arrow::Status::OK();\n }\n\n std::shared_ptr * m_dest;\n};\n\n} // namespace visitors\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_compression.cpp", "content": "#include \"pod5_format/signal_compression.h\"\n\n#include \"pod5_format/svb16/decode.hpp\"\n#include \"pod5_format/svb16/encode.hpp\"\n\n#include \n#include \n#include \n\n#include \n#include \n\nnamespace pod5 {\n\nnamespace {\n\n// SVB is designed around 32 bit sizes, so that's the maximum uncompressed samples allowed.\nconstexpr std::size_t max_uncompressed_samples = std::numeric_limits::max();\n\nclass DecompressContext {\n struct DCtxDeleter {\n void operator()(ZSTD_DCtx * ctx) { ZSTD_freeDCtx(ctx); }\n };\n\n std::unique_ptr m_context;\n\npublic:\n DecompressContext() { m_context.reset(ZSTD_createDCtx()); }\n\n ZSTD_DCtx * get() { return m_context.get(); }\n\n explicit operator bool() const { return static_cast(m_context); }\n};\n\n} // namespace\n\narrow::Result compressed_signal_max_size(std::size_t sample_count)\n{\n if (sample_count > max_uncompressed_samples) {\n return arrow::Status::Invalid(\n sample_count, \" samples exceeds max of \", max_uncompressed_samples);\n }\n\n auto const max_svb_size = svb16_max_encoded_length(sample_count);\n auto const zstd_compressed_max_size = ZSTD_compressBound(max_svb_size);\n if (ZSTD_isError(zstd_compressed_max_size)) {\n return pod5::Status::Invalid(\n sample_count,\n \" samples exceeds zstd limit: (\",\n zstd_compressed_max_size,\n \" \",\n ZSTD_getErrorName(zstd_compressed_max_size),\n \")\");\n }\n\n return zstd_compressed_max_size;\n}\n\narrow::Result compress_signal(\n gsl::span samples,\n arrow::MemoryPool * pool,\n gsl::span destination)\n{\n std::size_t const sample_count = samples.size();\n if (sample_count > max_uncompressed_samples) {\n return arrow::Status::Invalid(\n sample_count, \" samples exceeds max of \", max_uncompressed_samples);\n }\n\n // First compress the data using svb:\n auto const max_size = svb16_max_encoded_length(sample_count);\n ARROW_ASSIGN_OR_RAISE(auto intermediate, arrow::AllocateResizableBuffer(max_size, pool));\n\n static constexpr bool UseDelta = true;\n static constexpr bool UseZigzag = true;\n auto const encoded_count = svb16::encode(\n samples.data(), intermediate->mutable_data(), sample_count);\n ARROW_RETURN_NOT_OK(intermediate->Resize(encoded_count));\n\n // Now compress the svb data using zstd:\n size_t const zstd_compressed_max_size = ZSTD_compressBound(intermediate->size());\n if (ZSTD_isError(zstd_compressed_max_size)) {\n return pod5::Status::Invalid(\n \"Failed to find zstd max size for data: (\",\n zstd_compressed_max_size,\n \" \",\n ZSTD_getErrorName(zstd_compressed_max_size),\n \")\");\n }\n\n /* Compress.\n * If you are doing many compressions, you may want to reuse the context.\n * See the multiple_simple_compression.c example.\n */\n size_t const compressed_size = ZSTD_compress(\n destination.data(), destination.size(), intermediate->data(), intermediate->size(), 1);\n if (ZSTD_isError(compressed_size)) {\n return pod5::Status::Invalid(\n \"Failed to compress data: (\",\n compressed_size,\n \" \",\n ZSTD_getErrorName(compressed_size),\n \")\");\n }\n return compressed_size;\n}\n\narrow::Result> compress_signal(\n gsl::span samples,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(\n std::size_t const sample_count, compressed_signal_max_size(samples.size()));\n\n ARROW_ASSIGN_OR_RAISE(\n std::shared_ptr out,\n arrow::AllocateResizableBuffer(sample_count, pool));\n\n ARROW_ASSIGN_OR_RAISE(\n auto final_size,\n compress_signal(samples, pool, gsl::make_span(out->mutable_data(), out->size())));\n\n ARROW_RETURN_NOT_OK(out->Resize(final_size));\n return out;\n}\n\narrow::Status decompress_signal(\n gsl::span compressed_bytes,\n arrow::MemoryPool * pool,\n gsl::span destination)\n{\n // Check that we could have compressed this size.\n ARROW_ASSIGN_OR_RAISE(\n std::size_t const max_compressed_size, compressed_signal_max_size(destination.size()));\n if (compressed_bytes.size() > max_compressed_size) {\n return pod5::Status::Invalid(\n \"Input data corrupt: compressed input size (\",\n compressed_bytes.size(),\n \") exceeds max compressed output size (\",\n max_compressed_size,\n \")\");\n }\n\n // Find out how big zstd thinks the data is.\n unsigned long long const decompressed_zstd_size =\n ZSTD_getFrameContentSize(compressed_bytes.data(), compressed_bytes.size());\n if (ZSTD_isError(decompressed_zstd_size)) {\n return pod5::Status::Invalid(\n \"Input data not compressed by zstd: (\",\n decompressed_zstd_size,\n \" \",\n ZSTD_getErrorName(decompressed_zstd_size),\n \")\");\n }\n\n // Documentation of |ZSTD_getFrameContentSize| explicitly states that we should bounds check this:\n // * note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.\n // * Always ensure return value fits within application's authorized limits.\n // * Each application can set its own limits.\n std::size_t const max_svb16_compressed_size = svb16_max_encoded_length(destination.size());\n if (decompressed_zstd_size > max_svb16_compressed_size) {\n return arrow::Status::Invalid(\n \"Input data corrupt: claimed size (\",\n decompressed_zstd_size,\n \") exceeds max compressed output size (\",\n max_svb16_compressed_size,\n \")\");\n }\n\n // Check that we have enough memory to decompress.\n // Note: this will return 0 on unsupported platforms, so we skip it there.\n std::int64_t const system_memory = arrow::internal::GetTotalMemoryBytes();\n assert(system_memory > 0);\n if (system_memory > 0 && decompressed_zstd_size >= static_cast(system_memory)) {\n return arrow::Status::OutOfMemory(\n \"Not enough system memory (\",\n system_memory,\n \") to decompress file (\",\n decompressed_zstd_size,\n \")\");\n }\n\n if (POD5_ENABLE_FUZZERS && decompressed_zstd_size > 1'000'000) {\n return arrow::Status::Invalid(\"Skipping huge sizes when fuzzing\");\n }\n\n thread_local DecompressContext decompress_context;\n if (!decompress_context) {\n return arrow::Status::OutOfMemory(\"Failed to create zstd decompress context\");\n }\n\n // Decompress the data using zstd.\n auto const allocation_padding = svb16::decode_input_buffer_padding_byte_count();\n ARROW_ASSIGN_OR_RAISE(\n auto intermediate,\n arrow::AllocateResizableBuffer(decompressed_zstd_size + allocation_padding, pool));\n size_t const decompress_res = ZSTD_decompressDCtx(\n decompress_context.get(),\n intermediate->mutable_data(),\n intermediate->size(),\n compressed_bytes.data(),\n compressed_bytes.size());\n if (ZSTD_isError(decompress_res)) {\n return pod5::Status::Invalid(\n \"Input data failed to decompress using zstd: (\",\n decompress_res,\n \" \",\n ZSTD_getErrorName(decompress_res),\n \")\");\n }\n\n auto const svb16_compressed_data_with_padding =\n gsl::make_span(intermediate->data(), intermediate->size());\n auto const svb16_compressed_data_no_padding =\n svb16_compressed_data_with_padding.subspan(0, decompressed_zstd_size);\n\n // Validate the data.\n if (!svb16::validate(svb16_compressed_data_no_padding, destination.size())) {\n return pod5::Status::Invalid(\"Compressed signal data is corrupt\");\n }\n\n // Now decompress the data using svb:\n static constexpr bool UseDelta = true;\n static constexpr bool UseZigzag = true;\n auto consumed_count = svb16::decode(\n destination, svb16_compressed_data_with_padding);\n if (consumed_count != decompressed_zstd_size) {\n return pod5::Status::Invalid(\"Remaining data at end of signal buffer\");\n }\n\n return pod5::Status::OK();\n}\n\narrow::Result> decompress_signal(\n gsl::span compressed_bytes,\n std::uint32_t samples_count,\n arrow::MemoryPool * pool)\n{\n ARROW_ASSIGN_OR_RAISE(\n std::shared_ptr out,\n arrow::AllocateResizableBuffer(samples_count * sizeof(SampleType), pool));\n\n auto signal_span = gsl::make_span(out->mutable_data(), out->size()).as_span();\n\n ARROW_RETURN_NOT_OK(decompress_signal(compressed_bytes, pool, signal_span));\n return out;\n}\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_compression.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n\n#include \n\nnamespace arrow {\nclass MemoryPool;\nclass Buffer;\n} // namespace arrow\n\nnamespace pod5 {\n\nusing SampleType = std::int16_t;\n\nPOD5_FORMAT_EXPORT arrow::Result compressed_signal_max_size(std::size_t sample_count);\n\nPOD5_FORMAT_EXPORT arrow::Result compress_signal(\n gsl::span samples,\n arrow::MemoryPool * pool,\n gsl::span destination);\n\nPOD5_FORMAT_EXPORT arrow::Result> compress_signal(\n gsl::span samples,\n arrow::MemoryPool * pool);\n\nPOD5_FORMAT_EXPORT arrow::Result> decompress_signal(\n gsl::span compressed_bytes,\n std::uint32_t samples_count,\n arrow::MemoryPool * pool);\n\nPOD5_FORMAT_EXPORT arrow::Status decompress_signal(\n gsl::span compressed_bytes,\n arrow::MemoryPool * pool,\n gsl::span destination);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_reader.cpp", "content": "#include \"pod5_format/signal_table_reader.h\"\n\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/signal_compression.h\"\n#include \"pod5_format/table_reader.h\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nstruct SignalTableReaderCacheCleaner {\n static void make_space_in_table_batches(\n std::unordered_map & cached_batches)\n {\n std::vector> access_ordered_data;\n access_ordered_data.reserve(cached_batches.size());\n\n for (auto item : cached_batches) {\n access_ordered_data.emplace_back(\n std::make_pair(item.first, item.second.last_access_index));\n }\n std::sort(\n access_ordered_data.begin(),\n access_ordered_data.end(),\n [](auto const & a, auto const & b) { return a.second < b.second; });\n\n // Clear about 20% of the cache to make space for further growth:\n auto const to_clear = std::max(1, cached_batches.size() * 0.2f);\n for (std::size_t i = 0; i < to_clear; ++i) {\n auto const index_to_remove = access_ordered_data[i].first;\n cached_batches.erase(index_to_remove);\n }\n }\n};\n\nSignalTableRecordBatch::SignalTableRecordBatch(\n std::shared_ptr const & batch,\n SignalTableSchemaDescription field_locations,\n arrow::MemoryPool * pool)\n: TableRecordBatch(batch)\n, m_field_locations(field_locations)\n, m_pool(pool)\n{\n}\n\nstd::shared_ptr SignalTableRecordBatch::read_id_column() const\n{\n return std::static_pointer_cast(batch()->column(m_field_locations.read_id));\n}\n\nstd::shared_ptr SignalTableRecordBatch::uncompressed_signal_column() const\n{\n return std::static_pointer_cast(\n batch()->column(m_field_locations.signal));\n}\n\nstd::shared_ptr SignalTableRecordBatch::vbz_signal_column() const\n{\n return std::static_pointer_cast(batch()->column(m_field_locations.signal));\n}\n\nstd::shared_ptr SignalTableRecordBatch::samples_column() const\n{\n return std::static_pointer_cast(batch()->column(m_field_locations.samples));\n}\n\nResult SignalTableRecordBatch::samples_byte_count(std::size_t row_index) const\n{\n switch (m_field_locations.signal_type) {\n case SignalType::UncompressedSignal: {\n auto signal_column = uncompressed_signal_column();\n auto signal = signal_column->value_slice(row_index);\n return signal->length() * sizeof(std::int16_t);\n }\n case SignalType::VbzSignal: {\n auto signal_column = vbz_signal_column();\n auto signal_compressed = signal_column->Value(row_index);\n return signal_compressed.size();\n }\n }\n\n return pod5::Status::Invalid(\"Unknown signal type\");\n}\n\nStatus SignalTableRecordBatch::extract_signal_row(\n std::size_t row_index,\n gsl::span samples) const\n{\n if (row_index >= num_rows()) {\n return pod5::Status::Invalid(\n \"Queried signal row \",\n row_index,\n \" is outside the available rows (\",\n num_rows(),\n \" in batch)\");\n }\n\n auto sample_count = samples_column();\n auto samples_in_row = sample_count->Value(row_index);\n if (samples_in_row != samples.size()) {\n return pod5::Status::Invalid(\n \"Unexpected size for sample array \", samples.size(), \" expected \", samples_in_row);\n }\n\n switch (m_field_locations.signal_type) {\n case SignalType::UncompressedSignal: {\n auto signal_column = uncompressed_signal_column();\n auto signal =\n std::static_pointer_cast(signal_column->value_slice(row_index));\n std::copy(signal->raw_values(), signal->raw_values() + signal->length(), samples.begin());\n return Status::OK();\n }\n case SignalType::VbzSignal: {\n auto signal_column = vbz_signal_column();\n auto signal_compressed = signal_column->Value(row_index);\n return pod5::decompress_signal(signal_compressed, m_pool, samples);\n }\n }\n\n return pod5::Status::Invalid(\"Unknown signal type\");\n}\n\nResult> SignalTableRecordBatch::extract_signal_row_inplace(\n std::size_t row_index) const\n{\n if (row_index >= num_rows()) {\n return pod5::Status::Invalid(\n \"Queried signal row \",\n row_index,\n \" is outside the available rows (\",\n num_rows(),\n \" in batch)\");\n }\n\n switch (m_field_locations.signal_type) {\n case SignalType::UncompressedSignal: {\n auto signal_column = uncompressed_signal_column();\n auto const value_slice =\n std::static_pointer_cast(signal_column->value_slice(row_index));\n\n auto const element_size =\n sizeof(std::remove_reference::type::TypeClass);\n\n auto const values = value_slice->values();\n auto offset = signal_column->value_offset(row_index);\n auto length = signal_column->value_length(row_index);\n return arrow::SliceBuffer(values, offset * element_size, length * element_size);\n }\n case SignalType::VbzSignal: {\n auto signal_column = vbz_signal_column();\n return signal_column->ValueAsBuffer(row_index);\n }\n }\n\n return pod5::Status::Invalid(\"Unknown signal type\");\n}\n\n//---------------------------------------------------------------------------------------------------------------------\n\nSignalTableReader::SignalTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n SignalTableSchemaDescription field_locations,\n SchemaMetadataDescription && schema_metadata,\n std::size_t num_record_batches,\n std::size_t batch_size,\n std::size_t max_cached_table_batches,\n arrow::MemoryPool * pool)\n: TableReader(std::move(input_source), std::move(reader), std::move(schema_metadata), pool)\n, m_field_locations(field_locations)\n, m_pool(pool)\n, m_max_cached_table_batches(max_cached_table_batches)\n, m_table_batches(num_record_batches)\n, m_batch_size(batch_size)\n{\n}\n\nSignalTableReader::SignalTableReader(SignalTableReader && other)\n: TableReader(std::move(other))\n, m_field_locations(std::move(other.m_field_locations))\n, m_pool(other.m_pool)\n, m_max_cached_table_batches(other.m_max_cached_table_batches)\n, m_table_batches(std::move(other.m_table_batches))\n, m_batch_size(other.m_batch_size)\n{\n}\n\nSignalTableReader & SignalTableReader::operator=(SignalTableReader && other)\n{\n m_field_locations = std::move(other.m_field_locations);\n m_pool = other.m_pool;\n m_max_cached_table_batches = other.m_max_cached_table_batches;\n m_batch_size = other.m_batch_size;\n m_table_batches = std::move(other.m_table_batches);\n static_cast(*this) = std::move(static_cast(other));\n return *this;\n}\n\nResult SignalTableReader::read_record_batch(std::size_t i) const\n{\n std::lock_guard l(m_batch_get_mutex);\n if (m_last_read_record_batch_index == i) {\n return pod5::SignalTableRecordBatch{m_last_read_record_batch, m_field_locations, m_pool};\n }\n\n auto it = m_table_batches.find(i);\n if (it != m_table_batches.end()) {\n it->second.last_access_index = m_last_access_index++;\n return it->second.item;\n }\n\n // If limited in cached batches, then ensure we apply limit:\n if (m_max_cached_table_batches != 0 && m_table_batches.size() >= m_max_cached_table_batches) {\n SignalTableReaderCacheCleaner::make_space_in_table_batches(m_table_batches);\n assert(m_table_batches.size() < m_max_cached_table_batches);\n }\n\n ARROW_ASSIGN_OR_RAISE(m_last_read_record_batch, TableReader::ReadRecordBatch(i));\n m_last_read_record_batch_index = i;\n auto inserted = m_table_batches.emplace(\n i,\n CachedItem{\n pod5::SignalTableRecordBatch{m_last_read_record_batch, m_field_locations, m_pool},\n m_last_access_index++});\n return inserted.first->second.item;\n}\n\nResult SignalTableReader::signal_batch_for_row_id(\n std::uint64_t row,\n std::size_t * batch_row) const\n{\n if (m_batch_size == 0) {\n return Status::Invalid(\"Invalid row '\", row, \"' for file with zero signal rows.\");\n }\n\n auto batch = row / m_batch_size;\n\n if (batch_row) {\n *batch_row = row - (batch * m_batch_size);\n }\n\n if (batch >= num_record_batches()) {\n return Status::Invalid(\"Row outside batch bounds\");\n }\n\n return batch;\n}\n\nResult SignalTableReader::extract_sample_count(\n gsl::span const & row_indices) const\n{\n std::size_t sample_count = 0;\n for (auto const & signal_row : row_indices) {\n std::size_t batch_row = 0;\n ARROW_ASSIGN_OR_RAISE(\n auto const signal_batch_index, signal_batch_for_row_id(signal_row, &batch_row));\n\n ARROW_ASSIGN_OR_RAISE(auto const & signal_batch, read_record_batch(signal_batch_index));\n auto const & samples_column = signal_batch.samples_column();\n sample_count += samples_column->Value(batch_row);\n }\n return sample_count;\n}\n\nStatus SignalTableReader::extract_samples(\n gsl::span const & row_indices,\n gsl::span const & output_samples) const\n{\n std::size_t sample_count = 0;\n\n for (auto const & signal_row : row_indices) {\n std::size_t batch_row = 0;\n ARROW_ASSIGN_OR_RAISE(\n auto const signal_batch_index, signal_batch_for_row_id(signal_row, &batch_row));\n\n ARROW_ASSIGN_OR_RAISE(auto const & signal_batch, read_record_batch(signal_batch_index));\n auto const & samples_column = signal_batch.samples_column();\n auto const row_samples_count = samples_column->Value(batch_row);\n std::size_t const sample_start = sample_count;\n sample_count += row_samples_count;\n if (sample_count > output_samples.size()) {\n return Status::Invalid(\"Too few samples in input samples array\");\n }\n\n ARROW_RETURN_NOT_OK(signal_batch.extract_signal_row(\n batch_row, output_samples.subspan(sample_start, row_samples_count)));\n }\n return Status::OK();\n}\n\nResult>> SignalTableReader::extract_samples_inplace(\n gsl::span const & row_indices,\n std::vector & sample_count) const\n{\n std::vector> sample_buffers;\n\n for (auto const & signal_row : row_indices) {\n std::size_t batch_row = 0;\n ARROW_ASSIGN_OR_RAISE(\n auto const signal_batch_index, signal_batch_for_row_id(signal_row, &batch_row));\n\n ARROW_ASSIGN_OR_RAISE(auto const & signal_batch, read_record_batch(signal_batch_index));\n\n ARROW_ASSIGN_OR_RAISE(auto signal_data, signal_batch.extract_signal_row_inplace(batch_row));\n sample_buffers.emplace_back(std::move(signal_data));\n\n auto const & samples_column = signal_batch.samples_column();\n sample_count.push_back(samples_column->Value(batch_row));\n }\n return sample_buffers;\n}\n\nSignalType SignalTableReader::signal_type() const { return m_field_locations.signal_type; }\n\n//---------------------------------------------------------------------------------------------------------------------\nResult make_signal_table_reader(\n std::shared_ptr const & input,\n std::size_t max_cached_table_batches,\n arrow::MemoryPool * pool)\n{\n arrow::ipc::IpcReadOptions options;\n options.memory_pool = pool;\n\n ARROW_ASSIGN_OR_RAISE(auto reader, arrow::ipc::RecordBatchFileReader::Open(input, options));\n\n auto read_metadata_key_values = reader->schema()->metadata();\n if (!read_metadata_key_values) {\n return Status::IOError(\"Missing metadata on signal table schema\");\n }\n ARROW_ASSIGN_OR_RAISE(\n auto read_metadata, read_schema_key_value_metadata(read_metadata_key_values));\n ARROW_ASSIGN_OR_RAISE(auto field_locations, read_signal_table_schema(reader->schema()));\n\n std::size_t const num_record_batches = reader->num_record_batches();\n std::size_t batch_size = 0;\n if (num_record_batches > 0) {\n ARROW_ASSIGN_OR_RAISE(auto const batch_zero, ReadRecordBatchAndValidate(*reader, 0));\n batch_size = batch_zero->num_rows();\n }\n\n return SignalTableReader(\n {input},\n std::move(reader),\n field_locations,\n std::move(read_metadata),\n num_record_batches,\n batch_size,\n max_cached_table_batches,\n pool);\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_reader.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/signal_table_schema.h\"\n#include \"pod5_format/table_reader.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n\n#include \n#include \n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace io {\nclass RandomAccessFile;\n}\n\nnamespace ipc {\nclass RecordBatchFileReader;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nstruct SignalTableReaderCacheCleaner;\n\nclass POD5_FORMAT_EXPORT SignalTableRecordBatch : public TableRecordBatch {\npublic:\n SignalTableRecordBatch(\n std::shared_ptr const & batch,\n SignalTableSchemaDescription field_locations,\n arrow::MemoryPool * pool);\n\n std::shared_ptr read_id_column() const;\n std::shared_ptr uncompressed_signal_column() const;\n std::shared_ptr vbz_signal_column() const;\n std::shared_ptr samples_column() const;\n\n Result samples_byte_count(std::size_t row_index) const;\n\n /// \\brief Extract a row of sample data into [samples], decompressing if required.\n Status extract_signal_row(std::size_t row_index, gsl::span samples) const;\n Result> extract_signal_row_inplace(std::size_t row_index) const;\n\nprivate:\n SignalTableSchemaDescription m_field_locations;\n arrow::MemoryPool * m_pool;\n};\n\nclass POD5_FORMAT_EXPORT SignalTableReader : public TableReader {\npublic:\n SignalTableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n SignalTableSchemaDescription field_locations,\n SchemaMetadataDescription && schema_metadata,\n std::size_t num_record_batches,\n std::size_t batch_size,\n std::size_t max_cached_table_batches,\n arrow::MemoryPool * pool);\n\n SignalTableReader(SignalTableReader &&);\n SignalTableReader & operator=(SignalTableReader &&);\n\n Result read_record_batch(std::size_t i) const;\n\n Result signal_batch_for_row_id(std::uint64_t row, std::size_t * batch_row) const;\n\n /// \\brief Find the number of samples in a given list of rows.\n /// \\param row_indices The rows to query for sample ount.\n /// \\returns The sum of all sample counts on input rows.\n Result extract_sample_count(\n gsl::span const & row_indices) const;\n\n /// \\brief Extract the samples for a list of rows.\n /// \\param row_indices The rows to query for samples.\n /// \\param output_samples The output samples from the rows. Data in the vector is cleared before appending.\n Status extract_samples(\n gsl::span const & row_indices,\n gsl::span const & output_samples) const;\n\n /// \\brief Extract the samples as written in the arrow table for a list of rows.\n /// \\param row_indices The rows to query for samples.\n Result>> extract_samples_inplace(\n gsl::span const & row_indices,\n std::vector & sample_count) const;\n\n /// \\brief Find the signal type of this writer\n SignalType signal_type() const;\n\nprivate:\n SignalTableSchemaDescription m_field_locations;\n arrow::MemoryPool * m_pool;\n std::size_t m_max_cached_table_batches;\n\n mutable std::size_t m_last_read_record_batch_index = -1;\n mutable std::shared_ptr m_last_read_record_batch;\n\n mutable std::mutex m_batch_get_mutex;\n using AccessIndex = std::uint64_t;\n\n struct CachedItem {\n pod5::SignalTableRecordBatch item;\n AccessIndex last_access_index;\n };\n\n mutable std::unordered_map m_table_batches;\n\n mutable AccessIndex m_last_access_index = 0;\n\n std::size_t m_batch_size;\n\n friend struct SignalTableReaderCacheCleaner;\n};\n\nPOD5_FORMAT_EXPORT Result make_signal_table_reader(\n std::shared_ptr const & sink,\n std::size_t max_cached_table_batches,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_schema.cpp", "content": "#include \"pod5_format/signal_table_schema.h\"\n\n#include \"pod5_format/schema_utils.h\"\n#include \"pod5_format/types.h\"\n\n#include \n\nnamespace pod5 {\n\nstd::shared_ptr make_signal_table_schema(\n SignalType signal_type,\n std::shared_ptr const & metadata,\n SignalTableSchemaDescription * field_locations)\n{\n auto const uuid_type = uuid();\n\n if (field_locations) {\n *field_locations = {};\n field_locations->signal_type = signal_type;\n }\n\n std::shared_ptr signal_schema_type;\n switch (signal_type) {\n case SignalType::UncompressedSignal:\n signal_schema_type = arrow::large_list(arrow::int16());\n break;\n case SignalType::VbzSignal:\n signal_schema_type = vbz_signal();\n break;\n }\n\n return arrow::schema(\n {\n arrow::field(\"read_id\", uuid_type),\n arrow::field(\"signal\", signal_schema_type),\n arrow::field(\"samples\", arrow::uint32()),\n },\n metadata);\n}\n\nResult read_signal_table_schema(\n std::shared_ptr const & schema)\n{\n ARROW_ASSIGN_OR_RAISE(auto read_id_field_idx, find_field(schema, \"read_id\", uuid()));\n ARROW_ASSIGN_OR_RAISE(auto samples_field_idx, find_field(schema, \"samples\", arrow::uint32()));\n\n ARROW_ASSIGN_OR_RAISE(auto signal_field_idx, find_field_untyped(schema, \"signal\"));\n SignalType signal_type = SignalType::UncompressedSignal;\n {\n auto const signal_field = schema->field(signal_field_idx);\n\n auto const signal_arrow_type = signal_field->type();\n if (signal_arrow_type->id() == arrow::Type::LARGE_LIST) {\n auto const & signal_list_field =\n static_cast(*signal_arrow_type);\n if (signal_list_field.value_type()->id() != arrow::Type::INT16) {\n return Status::TypeError(\"Schema field 'signal' list value type is incorrect type\");\n }\n } else if (signal_arrow_type->Equals(vbz_signal())) {\n signal_type = SignalType::VbzSignal;\n } else {\n return Status::TypeError(\n \"Schema field 'signal' is incorrect type: '\", signal_arrow_type->name(), \"'\");\n }\n }\n\n return SignalTableSchemaDescription{\n signal_type, read_id_field_idx, signal_field_idx, samples_field_idx};\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_schema.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/signal_table_utils.h\"\n\n#include \n\nnamespace arrow {\nclass KeyValueMetadata;\nclass Schema;\n} // namespace arrow\n\nnamespace pod5 {\n\nstruct SignalTableSchemaDescription {\n SignalType signal_type;\n\n int read_id = 0;\n int signal = 1;\n int samples = 2;\n};\n\n/// \\brief Make a new schema for a signal table.\n/// \\param signal_type The type of signal to use.\n/// \\param metadata Metadata to be applied to the schema.\n/// \\param field_locations [optional] The signal table field locations, for use when writing to the table.\n/// \\returns The schema for a signal table.\nPOD5_FORMAT_EXPORT std::shared_ptr make_signal_table_schema(\n SignalType signal_type,\n std::shared_ptr const & metadata,\n SignalTableSchemaDescription * field_locations);\n\nPOD5_FORMAT_EXPORT Result read_signal_table_schema(\n std::shared_ptr const &);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_utils.h", "content": "#pragma once\n\nnamespace pod5 {\n\nusing SignalTableRowIndex = std::uint64_t;\n\nenum class SignalType {\n UncompressedSignal,\n VbzSignal,\n};\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_writer.cpp", "content": "#include \"pod5_format/signal_table_writer.h\"\n\n#include \"pod5_format/file_output_stream.h\"\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nSignalTableWriter::SignalTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n SignalBuilderVariant && signal_builder,\n SignalTableSchemaDescription const & field_locations,\n std::shared_ptr const & output_stream,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool)\n: m_pool(pool)\n, m_schema(schema)\n, m_field_locations(field_locations)\n, m_output_stream{output_stream}\n, m_table_batch_size(table_batch_size)\n, m_writer(std::move(writer))\n, m_signal_builder(std::move(signal_builder))\n{\n m_read_id_builder = make_read_id_builder(m_pool);\n m_samples_builder = std::make_unique(m_pool);\n}\n\nSignalTableWriter::SignalTableWriter(SignalTableWriter && other) = default;\nSignalTableWriter & SignalTableWriter::operator=(SignalTableWriter &&) = default;\n\nSignalTableWriter::~SignalTableWriter()\n{\n if (m_writer) {\n (void)close();\n }\n}\n\nResult SignalTableWriter::add_signal(\n Uuid const & read_id,\n gsl::span const & signal)\n{\n POD5_TRACE_FUNCTION();\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_RETURN_NOT_OK(reserve_rows());\n\n auto row_id = m_written_batched_row_count + m_current_batch_row_count;\n ARROW_RETURN_NOT_OK(m_read_id_builder->Append(read_id.data()));\n\n ARROW_RETURN_NOT_OK(std::visit(visitors::append_signal{signal, m_pool}, m_signal_builder));\n\n ARROW_RETURN_NOT_OK(m_samples_builder->Append(signal.size()));\n ++m_current_batch_row_count;\n\n if (m_current_batch_row_count >= m_table_batch_size) {\n ARROW_RETURN_NOT_OK(write_batch());\n }\n\n return row_id;\n}\n\nResult SignalTableWriter::add_pre_compressed_signal(\n Uuid const & read_id,\n gsl::span const & signal,\n std::uint32_t sample_count)\n{\n POD5_TRACE_FUNCTION();\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n ARROW_RETURN_NOT_OK(reserve_rows());\n\n auto row_id = m_written_batched_row_count + m_current_batch_row_count;\n ARROW_RETURN_NOT_OK(m_read_id_builder->Append(read_id.data()));\n\n ARROW_RETURN_NOT_OK(\n std::visit(visitors::append_pre_compressed_signal{signal}, m_signal_builder));\n\n ARROW_RETURN_NOT_OK(m_samples_builder->Append(sample_count));\n ++m_current_batch_row_count;\n\n if (m_current_batch_row_count >= m_table_batch_size) {\n ARROW_RETURN_NOT_OK(write_batch());\n }\n\n return row_id;\n}\n\npod5::Result>\nSignalTableWriter::add_signal_batch(\n std::size_t row_count,\n std::vector> && columns,\n bool final_batch)\n{\n POD5_TRACE_FUNCTION();\n if (!m_writer) {\n return Status::Invalid(\"Unable to write batches, writer is closed.\");\n }\n\n if (m_current_batch_row_count != 0) {\n return Status::Invalid(\"Unable to write batches directly and using per read methods\");\n }\n\n if (!final_batch && row_count != m_table_batch_size) {\n return Status::Invalid(\"Unable to write invalid sized signal batch to signal table\");\n }\n\n auto const record_batch = arrow::RecordBatch::Make(m_schema, row_count, std::move(columns));\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(*record_batch));\n if (final_batch) {\n ARROW_RETURN_NOT_OK(close());\n }\n\n auto first_row_id = m_written_batched_row_count;\n m_written_batched_row_count += row_count;\n return std::make_pair(first_row_id, m_written_batched_row_count);\n}\n\nStatus SignalTableWriter::close()\n{\n // Check for already closed\n if (!m_writer) {\n return Status::OK();\n }\n\n ARROW_RETURN_NOT_OK(write_batch());\n\n ARROW_RETURN_NOT_OK(m_writer->Close());\n m_writer = nullptr;\n return Status::OK();\n}\n\nSignalType SignalTableWriter::signal_type() const { return m_field_locations.signal_type; }\n\nStatus SignalTableWriter::write_batch(arrow::RecordBatch const & record_batch)\n{\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus SignalTableWriter::write_batch()\n{\n POD5_TRACE_FUNCTION();\n if (m_current_batch_row_count == 0) {\n return Status::OK();\n }\n\n if (!m_writer) {\n return Status::IOError(\"Writer terminated\");\n }\n\n std::vector> columns{nullptr, nullptr, nullptr};\n ARROW_RETURN_NOT_OK(m_read_id_builder->Finish(&columns[m_field_locations.read_id]));\n\n ARROW_RETURN_NOT_OK(\n std::visit(visitors::finish_column{&columns[m_field_locations.signal]}, m_signal_builder));\n\n ARROW_RETURN_NOT_OK(m_samples_builder->Finish(&columns[m_field_locations.samples]));\n\n auto const record_batch =\n arrow::RecordBatch::Make(m_schema, m_current_batch_row_count, std::move(columns));\n m_written_batched_row_count += m_current_batch_row_count;\n m_current_batch_row_count = 0;\n\n ARROW_RETURN_NOT_OK(m_writer->WriteRecordBatch(*record_batch));\n return m_output_stream->batch_complete();\n}\n\nStatus SignalTableWriter::reserve_rows()\n{\n // Only reserve if we have not already reserved (at the start of a batch)\n if (m_current_batch_row_count > 0) {\n return arrow::Status::OK();\n }\n\n ARROW_RETURN_NOT_OK(m_read_id_builder->Reserve(m_table_batch_size));\n ARROW_RETURN_NOT_OK(m_samples_builder->Reserve(m_table_batch_size));\n\n static constexpr std::uint32_t APPROX_READ_SIZE = 102'400;\n\n return std::visit(\n visitors::reserve_rows{m_table_batch_size, APPROX_READ_SIZE}, m_signal_builder);\n}\n\nResult make_signal_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n SignalType compression_type,\n arrow::MemoryPool * pool)\n{\n SignalTableSchemaDescription field_locations;\n auto schema = make_signal_table_schema(compression_type, metadata, &field_locations);\n\n arrow::ipc::IpcWriteOptions options;\n options.memory_pool = pool;\n\n ARROW_ASSIGN_OR_RAISE(auto writer, arrow::ipc::MakeFileWriter(sink, schema, options, metadata));\n\n ARROW_ASSIGN_OR_RAISE(auto signal_builder, make_signal_builder(compression_type, pool));\n\n auto signal_table_writer = SignalTableWriter(\n std::move(writer),\n std::move(schema),\n std::move(signal_builder),\n field_locations,\n sink,\n table_batch_size,\n pool);\n\n return signal_table_writer;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/signal_table_writer.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/signal_builder.h\"\n#include \"pod5_format/signal_table_schema.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n\nnamespace arrow {\nclass Schema;\n\nnamespace ipc {\nclass RecordBatchWriter;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nclass FileOutputStream;\n\nclass POD5_FORMAT_EXPORT SignalTableWriter {\npublic:\n SignalTableWriter(\n std::shared_ptr && writer,\n std::shared_ptr && schema,\n SignalBuilderVariant && signal_builder,\n SignalTableSchemaDescription const & field_locations,\n std::shared_ptr const & output_stream,\n std::size_t table_batch_size,\n arrow::MemoryPool * pool);\n SignalTableWriter(SignalTableWriter &&);\n SignalTableWriter & operator=(SignalTableWriter &&);\n SignalTableWriter(SignalTableWriter const &) = delete;\n SignalTableWriter & operator=(SignalTableWriter const &) = delete;\n ~SignalTableWriter();\n\n /// \\brief Find the size of table batches for the signal table writer.\n std::size_t table_batch_size() const { return m_table_batch_size; }\n\n /// \\brief Add a read to the signal table, adding to the current batch.\n /// \\param read_id The read id for the read entry\n /// \\param signal The signal for the read entry\n /// \\returns The row index of the inserted signal, or a status on failure.\n Result add_signal(\n Uuid const & read_id,\n gsl::span const & signal);\n\n /// \\brief Add a pre-compressed read to the signal table, adding to the current batch.\n /// The batch is not flushed to disk until #flush is called.\n ///\n /// The user should call #compress_signal on *this* writer to compress the signal prior\n /// to calling this method, to ensure the signal is compressed correctly for the table.\n ///\n /// \\param read_id The read id for the read entry\n /// \\param signal The signal for the read entry\n /// \\returns The row index of the inserted signal, or a status on failure.\n Result add_pre_compressed_signal(\n Uuid const & read_id,\n gsl::span const & signal,\n std::uint32_t sample_count);\n\n pod5::Result> add_signal_batch(\n std::size_t row_count,\n std::vector> && columns,\n bool final_batch);\n\n /// \\brief Close this writer, signaling no further data will be written to the writer.\n Status close();\n\n /// \\brief Find the signal type of this writer\n SignalType signal_type() const;\n\n /// \\brief Reserve space for future row writes, called automatically when a flush occurs.\n Status reserve_rows();\n\n /// \\brief Find the schema for the signal table\n std::shared_ptr const & schema() const { return m_schema; }\n\n /// \\brief Flush passed data into the writer as a record batch.\n Status write_batch(arrow::RecordBatch const &);\n\nprivate:\n /// \\brief Flush buffered data into the writer as a record batch.\n Status write_batch();\n\n arrow::MemoryPool * m_pool = nullptr;\n std::shared_ptr m_schema;\n SignalTableSchemaDescription m_field_locations;\n std::shared_ptr m_output_stream;\n std::size_t m_table_batch_size;\n\n std::shared_ptr m_writer;\n\n std::unique_ptr m_read_id_builder;\n SignalBuilderVariant m_signal_builder;\n std::unique_ptr m_samples_builder;\n\n std::size_t m_written_batched_row_count = 0;\n std::size_t m_current_batch_row_count = 0;\n};\n\n/// \\brief Make a new writer for a signal table.\n/// \\param sink Sink to be used for output of the table.\n/// \\param metadata Metadata to be applied to the table schema.\n/// \\param table_batch_size The size of each batch written for the table.\n/// \\param pool Pool to be used for building table in memory.\n/// \\returns The writer for the new table.\nPOD5_FORMAT_EXPORT Result make_signal_table_writer(\n std::shared_ptr const & sink,\n std::shared_ptr const & metadata,\n std::size_t table_batch_size,\n SignalType compression_type,\n arrow::MemoryPool * pool);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/svb16/common.hpp", "content": "#pragma once\n\n#if __cplusplus >= 201703L\n#define SVB16_IF_CONSTEXPR if constexpr\n#else\n#define SVB16_IF_CONSTEXPR if\n#endif\n\n#ifdef _MSC_VER\n#define SVB_RESTRICT __restrict\n#else\n#define SVB_RESTRICT __restrict__\n#endif\n\n#if defined(__x86_64__) || defined(_M_AMD64) // x64\n#define SVB16_X64\n#elif defined(__arm__) || defined(__aarch64__)\n#define SVB16_ARM\n#endif\n\n#ifndef __has_builtin\n#define __has_builtin(x) 0\n#endif\n\n#if __has_builtin(__builtin_popcount)\n// likely to be a single instruction (POPCNT) on x86_64\n#define svb16_popcount __builtin_popcount\n#else\n// optimising compilers can often convert this pattern to POPCNT on x86_64\ninline int svb16_popcount(unsigned int i)\n{\n i = i - ((i >> 1) & 0x55555555); // add pairs of bits\n i = (i & 0x33333333) + ((i >> 2) & 0x33333333); // quads\n i = (i + (i >> 4)) & 0x0F0F0F0F; // groups of 8\n return (i * 0x01010101) >> 24; // horizontal sum of bytes\n}\n#endif\n" }, { "path": "c++/pod5_format/svb16/decode.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n#include \"decode_scalar.hpp\"\n#include \"svb16.h\" // svb16_key_length\n\n#include \n\n#ifdef SVB16_X64\n#include \"decode_x64.hpp\"\n#include \"simd_detect_x64.hpp\"\n#endif\n\nnamespace svb16 {\n\n// Required extra space after readable buffers passed in.\n//\n// Require 1 128 bit buffer beyond the end of all input readable buffers.\ninline std::size_t decode_input_buffer_padding_byte_count()\n{\n#ifdef SVB16_X64\n return sizeof(__m128i);\n#else\n return 0;\n#endif\n}\n\ntemplate \nsize_t decode(gsl::span out, gsl::span in, Int16T prev = 0)\n{\n auto keys_length = ::svb16_key_length(out.size());\n auto const keys = in.subspan(0, keys_length);\n auto const data = in.subspan(keys_length);\n#ifdef SVB16_X64\n if (has_sse4_1()) {\n return decode_sse(out, keys, data, prev) - in.begin();\n }\n#endif\n return decode_scalar(out, keys, data, prev) - in.begin();\n}\n\ninline bool validate(gsl::span compressed_input, std::size_t out_size)\n{\n auto const keys_length = ::svb16_key_length(out_size);\n if (keys_length > compressed_input.size()) {\n return false;\n }\n\n // Pull out the parts of the input data.\n auto const keys_span = compressed_input.subspan(0, keys_length);\n auto const data_span = compressed_input.subspan(keys_length);\n auto keys_ptr = keys_span.begin();\n\n // Accumulate the key sizes in a wider type to avoid overflow.\n using Accumulator = std::\n conditional_t= sizeof(std::uint64_t), std::size_t, std::uint64_t>;\n Accumulator encoded_size = 0;\n\n // Give the compiler a hint that it can avoid branches in the inner loop.\n for (std::size_t c = 0; c < out_size / 8; c++) {\n uint8_t const key_byte = *keys_ptr++;\n for (uint8_t shift = 0; shift < 8; shift++) {\n uint8_t const code = (key_byte >> shift) & 0x01;\n encoded_size += code + 1;\n }\n }\n out_size &= 7;\n\n // Process the remainder one at a time.\n uint8_t shift = 0;\n uint8_t key_byte = *keys_ptr++;\n for (std::size_t c = 0; c < out_size; c++) {\n if (shift == 8) {\n shift = 0;\n key_byte = *keys_ptr++;\n }\n uint8_t const code = (key_byte >> shift) & 0x01;\n encoded_size += code + 1;\n shift++;\n }\n\n return encoded_size == data_span.size();\n}\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/decode_scalar.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n\n#include \n\n#include \n#include \n#include \n#include \n\nnamespace svb16 {\nnamespace detail {\ninline uint16_t zigzag_decode(uint16_t val)\n{\n return (val >> 1) ^ static_cast(0 - (val & 1));\n}\n\ninline uint16_t decode_data(gsl::span::iterator & dataPtr, uint8_t code)\n{\n uint16_t val;\n\n if (code == 0) { // 1 byte\n val = (uint16_t)*dataPtr;\n dataPtr += 1;\n } else { // 2 bytes\n val = 0;\n memcpy(&val, dataPtr, 2); // assumes little endian\n dataPtr += 2;\n }\n\n return val;\n}\n} // namespace detail\n\ntemplate \nuint8_t const * decode_scalar(\n gsl::span out_span,\n gsl::span keys_span,\n gsl::span data_span,\n Int16T prev = 0)\n{\n auto const count = out_span.size();\n if (count == 0) {\n return data_span.begin();\n }\n\n auto out = out_span.begin();\n auto keys = keys_span.begin();\n auto data = data_span.begin();\n\n uint8_t shift = 0; // cycles 0 through 7 then resets\n uint8_t key_byte = *keys++;\n // need to do the arithmetic in unsigned space so it wraps\n auto u_prev = static_cast(prev);\n for (uint32_t c = 0; c < count; c++, shift++) {\n if (shift == 8) {\n shift = 0;\n key_byte = *keys++;\n }\n uint16_t value = detail::decode_data(data, (key_byte >> shift) & 0x01);\n SVB16_IF_CONSTEXPR(UseZigzag) { value = detail::zigzag_decode(value); }\n SVB16_IF_CONSTEXPR(UseDelta)\n {\n value += u_prev;\n u_prev = value;\n }\n *out++ = static_cast(value);\n }\n\n assert(out == out_span.end());\n assert(keys == keys_span.end());\n assert(data <= data_span.end());\n return data;\n}\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/decode_x64.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n#include \"decode_scalar.hpp\"\n#include \"intrinsics.hpp\"\n#include \"shuffle_tables.hpp\"\n#include \"svb16.h\" // svb16_key_length\n\n#include \n\n#include \n#include \n\n#ifdef SVB16_X64\n\nnamespace svb16 {\nnamespace detail {\n[[gnu::target(\"ssse3\")]] inline __m128i zigzag_decode(__m128i val)\n{\n return _mm_xor_si128(\n // N >> 1\n _mm_srli_epi16(val, 1),\n // 0xFFFF if N & 1 else 0x0000\n _mm_srai_epi16(_mm_slli_epi16(val, 15), 15)\n // alternative: _mm_sign_epi16(ones, _mm_slli_epi16(buf, 15))\n );\n}\n\n[[gnu::target(\"ssse3\")]] inline __m128i unpack(uint32_t key, uint8_t const * SVB_RESTRICT * data)\n{\n auto const len = static_cast(8 + svb16_popcount(key));\n __m128i data_reg = _mm_loadu_si128(reinterpret_cast<__m128i const *>(*data));\n __m128i const shuffle = *reinterpret_cast<__m128i const *>(&g_decode_shuffle_table[key]);\n\n data_reg = _mm_shuffle_epi8(data_reg, shuffle);\n *data += len;\n\n return data_reg;\n}\n\ntemplate \n[[gnu::target(\"ssse3\")]] inline void store_8(Int16T * to, __m128i value, __m128i * prev)\n{\n SVB16_IF_CONSTEXPR(UseZigzag) { value = zigzag_decode(value); }\n\n SVB16_IF_CONSTEXPR(UseDelta)\n {\n auto const broadcast_last_16 =\n m128i_from_bytes(14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15);\n // value == [A B C D E F G H] (16 bit values)\n __m128i add = _mm_slli_si128(value, 2);\n // add == [- A B C D E F G]\n *prev = _mm_shuffle_epi8(*prev, broadcast_last_16);\n // *prev == [P P P P P P P P]\n value = _mm_add_epi16(value, add);\n // value == [A AB BC CD DE FG GH]\n add = _mm_slli_si128(value, 4);\n // add == [- - A AB BC CD DE EF]\n value = _mm_add_epi16(value, add);\n // value == [A AB ABC ABCD BCDE CDEF DEFG EFGH]\n add = _mm_slli_si128(value, 8);\n // add == [- - - - A AB ABC ABCD]\n value = _mm_add_epi16(value, add);\n // value == [A AB ABC ABCD ABCDE ABCDEF ABCDEFG ABCDEFGH]\n value = _mm_add_epi16(value, *prev);\n // value == [PA PAB PABC PABCD PABCDE PABCDEF PABCDEFG PABCDEFGH]\n *prev = value;\n }\n\n _mm_storeu_si128(reinterpret_cast<__m128i *>(to), value);\n}\n} // namespace detail\n\ntemplate \n[[gnu::target(\"sse4.1\")]] uint8_t const * decode_sse(\n gsl::span out_span,\n gsl::span keys_span,\n gsl::span data_span,\n Int16T prev = 0)\n{\n auto store_8 = [](Int16T * to, __m128i value, __m128i * prev) {\n detail::store_8(to, value, prev);\n };\n // this code treats all input as uint16_t (except the zigzag code, which treats it as int16_t)\n // this isn't a problem, as the scalar code does the same\n\n auto out = out_span.begin();\n auto const count = out_span.size();\n auto keys_it = keys_span.begin();\n auto data = data_span.begin();\n\n // handle blocks of 32 values\n if (count >= 64) {\n size_t const key_bytes = count / 8;\n\n __m128i prev_reg;\n SVB16_IF_CONSTEXPR(UseDelta) { prev_reg = _mm_set1_epi16(prev); }\n\n int64_t offset = -static_cast(key_bytes) / 8 + 1; // 8 -> 4?\n uint64_t const * keyPtr64 = reinterpret_cast(keys_it) - offset;\n uint64_t nextkeys;\n memcpy(&nextkeys, keyPtr64 + offset, sizeof(nextkeys));\n\n __m128i data_reg;\n\n for (; offset != 0; ++offset) {\n uint64_t keys = nextkeys;\n memcpy(&nextkeys, keyPtr64 + offset + 1, sizeof(nextkeys));\n // faster 16-bit delta since we only have 8-bit values\n if (!keys) { // 64 1-byte ints in a row\n\n // _mm_cvtepu8_epi16: SSE4.1\n data_reg =\n _mm_cvtepu8_epi16(_mm_lddqu_si128(reinterpret_cast<__m128i const *>(data)));\n store_8(out, data_reg, &prev_reg);\n data_reg =\n _mm_cvtepu8_epi16(_mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 8)));\n store_8(out + 8, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 16)));\n store_8(out + 16, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 24)));\n store_8(out + 24, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 32)));\n store_8(out + 32, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + +40)));\n store_8(out + 40, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 48)));\n store_8(out + 48, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 56)));\n store_8(out + 56, data_reg, &prev_reg);\n out += 64;\n data += 64;\n continue;\n }\n\n data_reg = detail::unpack(keys & 0x00FF, &data);\n store_8(out, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 8, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 16, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 24, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 32, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 40, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 48, data_reg, &prev_reg);\n\n // Note we load at least sizeof(__m128i) bytes from the end of data\n // here, need to ensure that is available to read.\n //\n // But we might not use it all depending on the unpacking.\n //\n // This is ok due to `decode_input_buffer_padding_byte_count` enuring\n // extra space on the input buffer.\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 56, data_reg, &prev_reg);\n\n out += 64;\n }\n {\n uint64_t keys = nextkeys;\n // faster 16-bit delta since we only have 8-bit values\n if (!keys) { // 64 1-byte ints in a row\n data_reg =\n _mm_cvtepu8_epi16(_mm_lddqu_si128(reinterpret_cast<__m128i const *>(data)));\n store_8(out, data_reg, &prev_reg);\n data_reg =\n _mm_cvtepu8_epi16(_mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 8)));\n store_8(out + 8, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 16)));\n store_8(out + 16, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 24)));\n store_8(out + 24, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 32)));\n store_8(out + 32, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + +40)));\n store_8(out + 40, data_reg, &prev_reg);\n data_reg = _mm_cvtepu8_epi16(\n _mm_lddqu_si128(reinterpret_cast<__m128i const *>(data + 48)));\n store_8(out + 48, data_reg, &prev_reg);\n // Only load the first 8 bytes here, otherwise we may run off the end of the buffer\n data_reg = _mm_cvtepu8_epi16(\n _mm_loadl_epi64(reinterpret_cast<__m128i const *>(data + 56)));\n store_8(out + 56, data_reg, &prev_reg);\n out += 64;\n data += 64;\n\n } else {\n data_reg = detail::unpack(keys & 0x00FF, &data);\n store_8(out, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 8, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 16, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 24, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 32, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 40, data_reg, &prev_reg);\n\n keys >>= 16;\n data_reg = detail::unpack((keys & 0x00FF), &data);\n store_8(out + 48, data_reg, &prev_reg);\n data_reg = detail::unpack((keys & 0xFF00) >> 8, &data);\n store_8(out + 56, data_reg, &prev_reg);\n\n out += 64;\n }\n }\n prev = out[-1];\n\n keys_it += key_bytes - (key_bytes & 7);\n }\n\n assert(out <= out_span.end());\n assert(keys_it <= keys_span.end());\n assert(data <= data_span.end());\n\n auto out_scalar_span = gsl::make_span(out, out_span.end());\n assert(out_scalar_span.size() == (count & 63));\n\n auto keys_scalar_span = gsl::make_span(keys_it, keys_span.end());\n auto data_scalar_span = gsl::make_span(data, data_span.end());\n\n return decode_scalar(\n out_scalar_span, keys_scalar_span, data_scalar_span, prev);\n}\n\n#endif // SVB16_X64\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/encode.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n#include \"encode_scalar.hpp\"\n#include \"svb16.h\" // svb16_key_length\n#ifdef SVB16_X64\n#include \"encode_x64.hpp\"\n#include \"simd_detect_x64.hpp\"\n#endif\n\nnamespace svb16 {\n\ntemplate \nsize_t encode(Int16T const * in, uint8_t * SVB_RESTRICT out, uint32_t count, Int16T prev = 0)\n{\n auto const keys = out;\n auto const data = keys + ::svb16_key_length(count);\n#ifdef SVB16_X64\n if (has_ssse3()) {\n return encode_sse(in, keys, data, count, prev) - out;\n }\n#endif\n return encode_scalar(in, keys, data, count, prev) - out;\n}\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/encode_scalar.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n\n#include \n#include \n#include \n\nnamespace svb16 {\nnamespace detail {\ninline uint16_t zigzag_encode(uint16_t val)\n{\n return (val + val) ^ static_cast(static_cast(val) >> 15);\n}\n} // namespace detail\n\ntemplate \nuint8_t * encode_scalar(\n Int16T const * in,\n uint8_t * SVB_RESTRICT keys,\n uint8_t * SVB_RESTRICT data,\n uint32_t count,\n Int16T prev = 0)\n{\n if (count == 0) {\n return data;\n }\n\n uint8_t shift = 0; // cycles 0 through 7 then resets\n uint8_t key_byte = 0;\n for (uint32_t c = 0; c < count; c++) {\n if (shift == 8) {\n shift = 0;\n *keys++ = key_byte;\n key_byte = 0;\n }\n uint16_t value;\n SVB16_IF_CONSTEXPR(UseDelta)\n {\n // need to do the arithmetic in unsigned space so it wraps\n value = static_cast(in[c]) - static_cast(prev);\n SVB16_IF_CONSTEXPR(UseZigzag) { value = detail::zigzag_encode(value); }\n prev = in[c];\n }\n else SVB16_IF_CONSTEXPR(UseZigzag) {\n value = detail::zigzag_encode(static_cast(in[c]));\n }\n else {\n value = static_cast(in[c]);\n }\n\n if (value < (1 << 8)) { // 1 byte\n *data = static_cast(value);\n ++data;\n } else { // 2 bytes\n std::memcpy(data, &value, 2); // assumes little endian\n data += 2;\n key_byte |= 1 << shift;\n }\n\n shift += 1;\n }\n\n *keys = key_byte; // write last key (no increment needed)\n return data;\n}\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/encode_x64.hpp", "content": "#pragma once\n\n#include \"common.hpp\"\n#include \"encode_scalar.hpp\"\n#include \"intrinsics.hpp\"\n#include \"shuffle_tables.hpp\"\n#include \"svb16.h\" // svb16_key_length\n\n#include \n#include \n\n#ifdef SVB16_X64\n\nnamespace svb16 {\nnamespace detail {\n[[gnu::target(\"ssse3\")]] inline __m128i delta(__m128i curr, __m128i prev)\n{\n return _mm_sub_epi16(curr, _mm_alignr_epi8(curr, prev, 14));\n}\n\n[[gnu::target(\"ssse3\")]] inline __m128i zigzag_encode(__m128i val)\n{\n return _mm_xor_si128(_mm_add_epi16(val, val), _mm_srai_epi16(val, 16));\n}\n\ntemplate \n[[gnu::target(\"ssse3\")]] inline __m128i load_8(Int16T const * from, __m128i * prev)\n{\n auto const loaded = _mm_loadu_si128(reinterpret_cast<__m128i const *>(from));\n SVB16_IF_CONSTEXPR(UseDelta && UseZigzag)\n {\n auto const result = delta(loaded, *prev);\n *prev = loaded;\n return zigzag_encode(result);\n }\n else SVB16_IF_CONSTEXPR(UseDelta) {\n auto const result = delta(loaded, *prev);\n *prev = loaded;\n return result;\n }\n else SVB16_IF_CONSTEXPR(UseZigzag) {\n return zigzag_encode(loaded);\n }\n else {\n return loaded;\n }\n}\n} // namespace detail\n\ntemplate \n[[gnu::target(\"ssse3\")]] uint8_t * encode_sse(\n Int16T const * in,\n uint8_t * SVB_RESTRICT keys_dest,\n uint8_t * SVB_RESTRICT data_dest,\n uint32_t count,\n Int16T prev = 0)\n{\n // this code treats all input as uint16_t (except the zigzag code, which treats it as int16_t)\n // this isn't a problem, as the scalar code does the same\n __m128i prev_reg;\n SVB16_IF_CONSTEXPR(UseDelta) { prev_reg = _mm_set1_epi16(prev); }\n //auto const key_len = svb16_key_length(count);\n auto const mask_01 = detail::m128i_from_bytes(\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01,\n 0x01);\n for (Int16T const * end = &in [(count & ~15)]; in != end; in += 16) {\n // load up 16 values into r0 and r1\n auto r0 = detail::load_8(in, &prev_reg);\n auto r1 = detail::load_8(in + 8, &prev_reg);\n\n // 1 byte per input byte: 1 if the byte is set, 0 if not\n auto r2 = _mm_min_epu8(mask_01, r0);\n auto r3 = _mm_min_epu8(mask_01, r1);\n // 1 byte per input Int16T: FF if the MSB is set, 00 or 01 if not\n // (us = unsigned saturation)\n r2 = _mm_packus_epi16(r2, r3);\n // 1 bit per input Int16T: 1 if the MSB is set, 0 if not\n // only the low 16 bits are set\n auto const keys = static_cast(_mm_movemask_epi8(r2));\n\n // use the shuffle table to discard the MSB if the corresponidng key bit is not set\n r2 = _mm_loadu_si128((__m128i *)&g_encode_shuffle_table[(keys << 4) & 0x07F0]);\n r3 = _mm_loadu_si128((__m128i *)&g_encode_shuffle_table[(keys >> 4) & 0x07F0]);\n r0 = _mm_shuffle_epi8(r0, r2);\n r1 = _mm_shuffle_epi8(r1, r3);\n\n // store the data to data_dest (note that we often end up with overlapping writes)\n _mm_storeu_si128(reinterpret_cast<__m128i *>(data_dest), r0);\n data_dest += 8 + svb16_popcount(keys & 0xFF);\n _mm_storeu_si128(reinterpret_cast<__m128i *>(data_dest), r1);\n data_dest += 8 + svb16_popcount(keys >> 8);\n\n *reinterpret_cast(keys_dest) = keys;\n keys_dest += 2;\n }\n\n SVB16_IF_CONSTEXPR(UseDelta) { prev = _mm_extract_epi16(prev_reg, 7); }\n // max two control bytes (16 values) left, use the scalar function\n count &= 15;\n return encode_scalar(in, keys_dest, data_dest, count, prev);\n}\n\n#endif // SVB16_X64\n\n} // namespace svb16\n" }, { "path": "c++/pod5_format/svb16/generate_shuffle_tables.py", "content": "def encode_table_row(control):\n table = []\n for i in range(7):\n offset = i * 2\n # first byte\n table.append(offset)\n if (control >> i) & 1:\n table.append(offset + 1)\n final_offset = 14\n for j in range(2):\n table.append(final_offset + j)\n for i in range(16 - len(table)):\n table.append(0xFF)\n return table\n\n\ndef decode_table_row(control):\n table = []\n offset = 0\n for i in range(8):\n table.append(offset)\n offset += 1\n if (control >> i) & 1:\n table.append(offset)\n offset += 1\n else:\n table.append(0xFF)\n return table\n\n\ndef print_x64_encode_table():\n print(\"static constexpr uint8_t g_encode_shuffle_table[128*16] = {\")\n for i in range(128):\n table = encode_table_row(i)\n print(\"\\t\", \", \".join(f\"0x{v:02X}\" for v in table), \",\", sep=\"\")\n print(\"};\\n\\n\")\n\n\ndef print_x64_decode_table():\n print(\"static const uint8_t g_decode_shuffle_table[256][16] = {\")\n for i in range(256):\n table = decode_table_row(i)\n print(\"\\t{ \", \", \".join(f\"0x{v:02X}\" for v in table), \"},\", sep=\"\")\n print(\"};\\n\\n\")\n\n\nif __name__ == \"__main__\":\n print(\"#pragma once\")\n print('#include \"common.hpp\" // arch macros')\n print(\"#include \")\n print()\n print(\"#ifdef SVB16_X64\")\n print_x64_encode_table()\n print_x64_decode_table()\n print(\"#endif\")\n" }, { "path": "c++/pod5_format/svb16/intrinsics.hpp", "content": "#pragma once\n\n#include \"common.hpp\" // architecture macros\n\n#if defined(_MSC_VER)\n#include \n#elif defined(__GNUC__) && defined(SVB16_X64)\n#include \n#elif defined(__GNUC__) && defined(__ARM_NEON__)\n#include \n#endif\n\n#include \n\nnamespace svb16 { namespace detail {\n[[gnu::target(\"sse2\")]] inline constexpr __m128i m128i_from_bytes(\n uint8_t a,\n uint8_t b,\n uint8_t c,\n uint8_t d,\n uint8_t e,\n uint8_t f,\n uint8_t g,\n uint8_t h,\n uint8_t i,\n uint8_t j,\n uint8_t k,\n uint8_t l,\n uint8_t m,\n uint8_t n,\n uint8_t o,\n uint8_t p)\n{\n#ifdef _MSC_VER\n return __m128i{\n (char)a,\n (char)b,\n (char)c,\n (char)d,\n (char)e,\n (char)f,\n (char)g,\n (char)h,\n (char)i,\n (char)j,\n (char)k,\n (char)l,\n (char)m,\n (char)n,\n (char)o,\n (char)p};\n#else\n return __m128i{\n static_cast(static_cast(h) << 56) + (static_cast(g) << 48)\n + (static_cast(f) << 40) + (static_cast(e) << 32)\n + (static_cast(d) << 24) + (static_cast(c) << 16)\n + (static_cast(b) << 8) + static_cast(a),\n static_cast(static_cast(h) << 56) + (static_cast(g) << 48)\n + (static_cast(f) << 40) + (static_cast(e) << 32)\n + (static_cast(d) << 24) + (static_cast(c) << 16)\n + (static_cast(b) << 8) + static_cast(a)};\n#endif\n}\n}} // namespace svb16::detail\n" }, { "path": "c++/pod5_format/svb16/shuffle_tables.hpp", "content": "#pragma once\n#include \"common.hpp\" // arch macros\n\n#include \n\n#ifdef SVB16_X64\nstatic constexpr uint8_t g_encode_shuffle_table[128 * 16] = {\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF,\n 0x00, 0x01, 0x02, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF,\n 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F,\n};\n\nstatic uint8_t const g_decode_shuffle_table[256][16] = {\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0xFF},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E,\n 0xFF},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0xFF,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0xFF,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0xFF,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0xFF,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0xFF,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0xFF,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0xFF,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0xFF,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0xFF,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0xFF,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0xFF,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D},\n {0x00,\n 0x01,\n 0x02,\n 0xFF,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0xFF,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E},\n {0x00,\n 0x01,\n 0x02,\n 0x03,\n 0x04,\n 0x05,\n 0x06,\n 0x07,\n 0x08,\n 0x09,\n 0x0A,\n 0x0B,\n 0x0C,\n 0x0D,\n 0x0E,\n 0x0F},\n};\n\n#endif\n" }, { "path": "c++/pod5_format/svb16/simd_detect_x64.hpp", "content": "#pragma once\n\n#include \"common.hpp\" // architecture macros\n\n#if defined(SVB16_X64)\n\n#ifdef _MSC_VER\n#include \n#endif\n\n// __AVX__ is documented for MSVC, but __SSE4_1__ isn't\n#if defined(__AVX__) || defined(__SSE4_1__)\n\ninline constexpr bool has_ssse3() { return true; }\n\ninline constexpr bool has_sse4_1() { return true; }\n\n#else\n\nstruct CpuidResult {\n unsigned int eax;\n unsigned int ebx;\n unsigned int ecx;\n unsigned int edx;\n};\n\ninline CpuidResult cpuid(unsigned int leaf, unsigned int subleaf)\n{\n#ifdef _MSC_VER\n int info[4];\n __cpuidex(info, static_cast(leaf), static_cast(subleaf));\n return CpuidResult{\n static_cast(info[0]),\n static_cast(info[1]),\n static_cast(info[2]),\n static_cast(info[3]),\n };\n#else\n CpuidResult info;\n asm(\"cpuid\\n\\t\"\n : \"=a\"(info.eax), \"=b\"(info.ebx), \"=c\"(info.ecx), \"=d\"(info.edx)\n : \"0\"(leaf), \"2\"(subleaf));\n return info;\n#endif\n}\n\ninline unsigned int cpuid_leaf1_ecx()\n{\n // using C++11 atomic static variables\n static unsigned int const ecx = cpuid(1, 0).ecx;\n return ecx;\n}\n\n#if defined(__SSSE3__)\ninline constexpr bool has_ssse3() { return true; }\n#else\ninline bool has_ssse3() { return (cpuid_leaf1_ecx() & (1 << 9)) != 0; }\n#endif\n\ninline bool has_sse4_1() { return (cpuid_leaf1_ecx() & (1 << 19)) != 0; }\n\n#endif // defined(__SSE4_1__)\n#endif // defined(SVB16_X64)\n" }, { "path": "c++/pod5_format/svb16/streamvbytedelta_decode_16.c", "content": "#include \"streamvbyte_isadetection.h\"\n#include \"streamvbytedelta.h\"\n\n#include // for memcpy\n\nstatic inline uint16_t zigzag_decode_16(uint16_t val)\n{\n return (val >> 1) ^ (uint16_t)(0 - (val & 1));\n}\n\nstatic inline uint16_t _decode_data(uint8_t const ** dataPtrPtr, uint8_t code)\n{\n uint8_t const * dataPtr = *dataPtrPtr;\n uint16_t val;\n\n if (code == 0) { // 1 byte\n val = (uint16_t)*dataPtr;\n dataPtr += 1;\n } else { // 2 bytes\n val = 0;\n memcpy(&val, dataPtr, 2); // assumes little endian\n dataPtr += 2;\n }\n\n *dataPtrPtr = dataPtr;\n return val;\n}\n\nstatic uint8_t const * svb_decode_scalar_d1_init(\n uint16_t * outPtr,\n uint8_t const * keyPtr,\n uint8_t const * dataPtr,\n uint32_t count,\n uint16_t prev)\n{\n if (count == 0) {\n return dataPtr; // no reads or writes if no data\n }\n\n uint8_t shift = 0;\n uint16_t key = *keyPtr++;\n\n for (uint32_t c = 0; c < count; c++) {\n if (shift == 8) {\n shift = 0;\n key = *keyPtr++;\n }\n uint16_t val = zigzag_decode_16(_decode_data(&dataPtr, (key >> shift) & 0x1));\n //uint16_t val = _decode_data(&dataPtr, (key >> shift) & 0x1);\n val += prev;\n *outPtr++ = val;\n prev = val;\n shift += 1;\n }\n\n return dataPtr; // pointer to first unused byte after end\n}\n\n#ifdef STREAMVBYTE_X64\n#include \"streamvbytedelta_x64_decode_16.c\"\n#endif\n\nsize_t streamvbyte_zigzag_delta_decode_16(\n uint8_t const * in,\n uint16_t * out,\n uint32_t count,\n uint16_t prev)\n{\n // keyLen = ceil(count / 8), without overflowing (1 bit per input value):\n uint32_t keyLen = (count >> 3) + (((count & 7) + 7) >> 3);\n uint8_t const * keyPtr = in;\n uint8_t const * dataPtr = keyPtr + keyLen; // data starts at end of keys\n#ifdef STREAMVBYTE_X64\n if (streamvbyte_ssse3()) {\n return svb_decode_avx_d1_init(out, keyPtr, dataPtr, count, prev) - in;\n }\n#endif\n return svb_decode_scalar_d1_init(out, keyPtr, dataPtr, count, prev) - in;\n}\n" }, { "path": "c++/pod5_format/svb16/streamvbytedelta_encode_16.c", "content": "#include \"streamvbyte_isadetection.h\"\n#include \"streamvbytedelta.h\"\n\n#include \n#include // for memcpy\n\n#ifdef STREAMVBYTE_X64\n#include \"streamvbytedelta_x64_encode_16.c\"\n#endif\n\nstatic inline uint16_t _zigzag_encode_16(uint16_t val)\n{\n return (val + val) ^ ((int16_t)val >> 15);\n}\n\nstatic uint8_t _encode_data(uint16_t val, uint8_t * __restrict__ * dataPtrPtr)\n{\n uint8_t * dataPtr = *dataPtrPtr;\n uint8_t code;\n\n if (val < (1 << 8)) { // 1 byte\n *dataPtr = (uint8_t)(val);\n *dataPtrPtr += 1;\n code = 0;\n } else { // 2 bytes\n memcpy(dataPtr, &val, 2); // assumes little endian\n *dataPtrPtr += 2;\n code = 1;\n }\n\n return code;\n}\n\nstatic uint8_t * svb_encode_scalar_d1_init(\n uint16_t const * in,\n uint8_t * __restrict__ keyPtr,\n uint8_t * __restrict__ dataPtr,\n uint32_t count,\n uint16_t prev)\n{\n if (count == 0) {\n return dataPtr; // exit immediately if no data\n }\n\n uint8_t shift = 0; // cycles 0 through 7 then resets\n uint8_t key = 0;\n for (uint32_t c = 0; c < count; c++) {\n if (shift == 8) {\n shift = 0;\n *keyPtr++ = key;\n key = 0;\n }\n uint16_t val = _zigzag_encode_16((uint16_t)(in[c] - prev));\n //uint16_t val = in[c] - prev;\n prev = in[c];\n uint8_t code = _encode_data(val, &dataPtr);\n key |= code << shift;\n shift += 1;\n }\n\n *keyPtr = key; // write last key (no increment needed)\n return dataPtr; // pointer to first unused data byte\n}\n\nsize_t streamvbyte_zigzag_delta_encode_16(\n uint16_t const * in,\n uint32_t count,\n uint8_t * out,\n uint16_t prev)\n{\n#ifdef STREAMVBYTE_X64\n if (streamvbyte_ssse3()) {\n return streamvbyte_zigzag_delta_encode_SSSE3_d1_init(in, count, out, prev);\n }\n#endif\n uint8_t * keyPtr = out; // keys come at start\n // keyLen = ceil(count / 8), without overflowing (1 bit per input value):\n uint32_t keyLen = (count >> 3) + (((count & 7) + 7) >> 3);\n uint8_t * dataPtr = keyPtr + keyLen; // variable byte data after all keys\n return svb_encode_scalar_d1_init(in, keyPtr, dataPtr, count, prev) - out;\n}\n" }, { "path": "c++/pod5_format/svb16/streamvbytedelta_x64_decode_16.c", "content": "#include \"streamvbyte_isadetection.h\"\n#include \"streamvbyte_shuffle_tables_decode_16.h\"\n\n#include // for memcpy\n#ifdef STREAMVBYTE_X64\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic __m128i undo_zigzag_16(__m128i buf)\n{\n return _mm_xor_si128(\n // N >> 1\n _mm_srli_epi16(buf, 1),\n // 0xFFFF if N & 1 else 0x0000\n _mm_srai_epi16(_mm_slli_epi16(buf, 15), 15)\n // alternative: _mm_sign_epi16(ones, _mm_slli_epi16(buf, 15))\n );\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic inline __m128i _decode_avx(uint32_t key, uint8_t const * __restrict__ * dataPtrPtr)\n{\n uint8_t len = 8 + popcount(key);\n __m128i Data = _mm_loadu_si128((__m128i *)*dataPtrPtr);\n __m128i Shuf = *(__m128i *)&shuffleTable[key];\n\n Data = _mm_shuffle_epi8(Data, Shuf);\n *dataPtrPtr += len;\n\n return Data;\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic inline void _write_avx(uint16_t * out, __m128i Vec)\n{\n _mm_storeu_si128((__m128i *)out, Vec);\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic inline __m128i _write_16bit_avx_d1(uint16_t * out, __m128i Vec, __m128i Prev)\n{\n#ifndef _MSC_VER\n __m128i BroadcastLast16 = {0x0F0E0F0E0F0E0F0E, 0x0F0E0F0E0F0E0F0E};\n#else\n __m128i BroadcastLast16 = {14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15};\n#endif\n Vec = undo_zigzag_16(Vec);\n // vec == [A B C D E F G H] (16 bit values)\n __m128i Add = _mm_slli_si128(Vec, 2); // [- A B C D E F G]\n Prev = _mm_shuffle_epi8(Prev, BroadcastLast16); // [P P P P P P P P]\n Vec = _mm_add_epi16(Vec, Add); // [A AB BC CD DE FG GH]\n Add = _mm_slli_si128(Vec, 4); // [- - A AB BC CD DE EF]\n Vec = _mm_add_epi16(Vec, Add); // [A AB ABC ABCD BCDE CDEF DEFG EFGH]\n Add = _mm_slli_si128(Vec, 8); // [- - - - A AB ABC ABCD]\n Vec = _mm_add_epi16(Vec, Add); // [A AB ABC ABCD ABCDE ABCDEF ABCDEFG ABCDEFGH]\n Vec = _mm_add_epi16(Vec, Prev); // [PA PAB PABC PABCD PABCDE PABCDEF PABCDEFG PABCDEFGH]\n _write_avx(out, Vec);\n return Vec;\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic uint8_t const * svb_decode_avx_d1_init(\n uint16_t * out,\n uint8_t const * __restrict__ keyPtr,\n uint8_t const * __restrict__ dataPtr,\n uint64_t count,\n uint16_t prev)\n{\n uint64_t keybytes = count / 4; // number of key bytes\n if (keybytes >= 8) {\n __m128i Prev = _mm_set1_epi16(prev);\n __m128i Data;\n\n int64_t Offset = -(int64_t)keybytes / 8 + 1;\n\n uint64_t const * keyPtr64 = (uint64_t const *)keyPtr - Offset;\n uint64_t nextkeys;\n memcpy(&nextkeys, keyPtr64 + Offset, sizeof(nextkeys));\n for (; Offset != 0; ++Offset) {\n uint64_t keys = nextkeys;\n memcpy(&nextkeys, keyPtr64 + Offset + 1, sizeof(nextkeys));\n // faster 16-bit delta since we only have 8-bit values\n if (!keys) { // 32 1-byte ints in a row\n\n // _mm_cvtepu8_epi16: SSE4.1\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr)));\n Prev = _write_16bit_avx_d1(out, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr + 8)));\n Prev = _write_16bit_avx_d1(out + 8, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr + 16)));\n Prev = _write_16bit_avx_d1(out + 16, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr + 24)));\n Prev = _write_16bit_avx_d1(out + 24, Data, Prev);\n out += 32;\n dataPtr += 32;\n continue;\n }\n\n Data = _decode_avx(keys & 0x00FF, &dataPtr);\n Prev = _write_16bit_avx_d1(out, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 4, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 8, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 12, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 16, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 20, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 24, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 28, Data, Prev);\n\n out += 32;\n }\n {\n uint64_t keys = nextkeys;\n // faster 16-bit delta since we only have 8-bit values\n if (!keys) { // 32 1-byte ints in a row\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr)));\n Prev = _write_16bit_avx_d1(out, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr + 8)));\n Prev = _write_16bit_avx_d1(out + 8, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_lddqu_si128((__m128i *)(dataPtr + 16)));\n Prev = _write_16bit_avx_d1(out + 16, Data, Prev);\n Data = _mm_cvtepu8_epi16(_mm_loadl_epi64((__m128i *)(dataPtr + 24)));\n Prev = _write_16bit_avx_d1(out + 24, Data, Prev);\n out += 32;\n dataPtr += 32;\n\n } else {\n Data = _decode_avx(keys & 0x00FF, &dataPtr);\n Prev = _write_16bit_avx_d1(out, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 4, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 8, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 12, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 16, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 20, Data, Prev);\n\n keys >>= 16;\n Data = _decode_avx((keys & 0x00FF), &dataPtr);\n Prev = _write_16bit_avx_d1(out + 24, Data, Prev);\n Data = _decode_avx((keys & 0xFF00) >> 8, &dataPtr);\n Prev = _write_16bit_avx_d1(out + 28, Data, Prev);\n\n out += 32;\n }\n }\n prev = out[-1];\n }\n uint64_t consumedkeys = keybytes - (keybytes & 7);\n return svb_decode_scalar_d1_init(out, keyPtr + consumedkeys, dataPtr, count & 31, prev);\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n#endif\n" }, { "path": "c++/pod5_format/svb16/streamvbytedelta_x64_encode_16.c", "content": "\n#include \"streamvbyte_isadetection.h\"\n#include \"streamvbyte_shuffle_tables_encode_16.h\"\n\n#include \n#include \n\n#ifdef STREAMVBYTE_X64\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic __m128i Delta(__m128i curr, __m128i prev)\n{\n // _mm_alignr_epi8: SSSE3\n return _mm_sub_epi16(curr, _mm_alignr_epi8(curr, prev, 14));\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\nSTREAMVBYTE_TARGET_SSSE3\nstatic __m128i zigzag_16(__m128i buf)\n{\n return _mm_xor_si128(_mm_add_epi16(buf, buf), _mm_srai_epi16(buf, 16));\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n\n// based on code by aqrit (streamvbyte_encode_SSSE3)\nSTREAMVBYTE_TARGET_SSSE3\nsize_t streamvbyte_zigzag_delta_encode_SSSE3_d1_init(\n uint16_t const * in,\n uint32_t count,\n uint8_t * out,\n uint16_t prev)\n{\n __m128i Prev = _mm_set1_epi16(prev);\n uint32_t keyLen = (count >> 3) + (((count & 7) + 7) >> 3); // 1-bit rounded to full byte\n uint8_t * restrict keyPtr = &out[0];\n uint8_t * restrict dataPtr = &out[keyLen]; // variable length data after keys\n\n __m128i const mask_01 = _mm_set1_epi8(0x01);\n\n for (uint16_t const * end = &in [(count & ~15)]; in != end; in += 16) {\n __m128i rawr0, r0, rawr1, r1, r2, r3;\n size_t keys;\n\n rawr0 = _mm_loadu_si128((__m128i *)&in[0]);\n r0 = zigzag_16(Delta(rawr0, Prev));\n Prev = rawr0;\n rawr1 = _mm_loadu_si128((__m128i *)&in[8]);\n r1 = zigzag_16(Delta(rawr1, Prev));\n Prev = rawr1;\n\n // 1 if the byte is set, 0 if not\n r2 = _mm_min_epu8(mask_01, r0);\n r3 = _mm_min_epu8(mask_01, r1);\n // for each (u)int16, FF if the MSB is set, 00 or 01 if not (us = unsigned saturation)\n r2 = _mm_packus_epi16(r2, r3);\n // for each byte, store a bit: 1 if FF, 0 if 00 or 01 (so 1 if MSB is set, 0 if not)\n keys = (size_t)_mm_movemask_epi8(r2);\n\n r2 = _mm_loadu_si128((__m128i *)&shuf_lut[(keys << 4) & 0x07F0]);\n r3 = _mm_loadu_si128((__m128i *)&shuf_lut[(keys >> 4) & 0x07F0]);\n // _mm_shuffle_epi8: SSSE3\n r0 = _mm_shuffle_epi8(r0, r2);\n r1 = _mm_shuffle_epi8(r1, r3);\n\n _mm_storeu_si128((__m128i *)dataPtr, r0);\n dataPtr += 8 + popcount(keys & 0xFF);\n _mm_storeu_si128((__m128i *)dataPtr, r1);\n dataPtr += 8 + popcount(keys >> 8);\n\n *((uint16_t *)keyPtr) = (uint16_t)keys;\n keyPtr += 2;\n }\n prev = _mm_extract_epi16(Prev, 7);\n\n // do remaining - max two control bytes left\n uint16_t key = 0;\n for (size_t i = 0; i < (count & 15); i++) {\n // TODO: can we factor this out to reuse the non-intrinsic code?\n uint16_t dw = in[i] - prev;\n prev = in[i];\n uint16_t zz = (dw + dw) ^ ((int16_t)dw >> 15);\n uint16_t symbol = (zz > 0x00FF);\n key |= symbol << (i + i);\n *((uint16_t *)dataPtr) = zz;\n dataPtr += 1 + symbol;\n }\n memcpy(keyPtr, &key, ((count & 15) + 5) >> 3);\n\n return dataPtr - out;\n}\n\nSTREAMVBYTE_UNTARGET_REGION\n#endif\n" }, { "path": "c++/pod5_format/svb16/svb16.c", "content": "" }, { "path": "c++/pod5_format/svb16/svb16.h", "content": "#ifndef SVB16_H\n#define SVB16_H\n\n#include \n\n#if defined(__cplusplus)\nextern \"C\" {\n#endif\n\n/// Get the number of key bytes required to encode a given number of 16-bit integers.\ninline uint32_t svb16_key_length(uint32_t count)\n{\n // ceil(count / 8.0), without overflowing or using fp arithmetic\n return (count >> 3) + (((count & 7) + 7) >> 3);\n}\n\n/// Get the maximum number of bytes required to encode a given number of 16-bit integers.\ninline uint32_t svb16_max_encoded_length(uint32_t count)\n{\n return svb16_key_length(count) + (2 * count);\n}\n\n#if defined(__cplusplus)\n};\n#endif\n\n#endif // SVB16_H\n" }, { "path": "c++/pod5_format/table_reader.cpp", "content": "#include \"pod5_format/table_reader.h\"\n\n#include \n#include \n#include \n\nnamespace pod5 {\n\nTableRecordBatch::TableRecordBatch(std::shared_ptr const & batch)\n: m_batch(batch)\n{\n}\n\nTableRecordBatch::TableRecordBatch(std::shared_ptr && batch)\n: m_batch(std::move(batch))\n{\n}\n\nTableRecordBatch::TableRecordBatch(TableRecordBatch const &) = default;\nTableRecordBatch & TableRecordBatch::operator=(TableRecordBatch const &) = default;\nTableRecordBatch::TableRecordBatch(TableRecordBatch &&) = default;\nTableRecordBatch & TableRecordBatch::operator=(TableRecordBatch &&) = default;\nTableRecordBatch::~TableRecordBatch() = default;\n\nstd::size_t TableRecordBatch::num_rows() const { return m_batch->num_rows(); }\n\n//---------------------------------------------------------------------------------------------------------------------\n\nTableReader::TableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool)\n: m_input_source(std::move(input_source))\n, m_reader(std::move(reader))\n, m_schema_metadata(std::move(schema_metadata))\n{\n}\n\nTableReader::TableReader(TableReader &&) = default;\nTableReader & TableReader::operator=(TableReader &&) = default;\nTableReader::~TableReader() = default;\n\nstd::size_t TableReader::num_record_batches() const { return m_reader->num_record_batches(); }\n\nResult TableReader::CountRows() const { return m_reader->CountRows(); }\n\nResult> TableReader::ReadRecordBatch(int i) const\n{\n return ReadRecordBatchAndValidate(*m_reader, i);\n}\n\nResult> ReadRecordBatchAndValidate(\n arrow::ipc::RecordBatchFileReader & reader,\n int i)\n{\n ARROW_ASSIGN_OR_RAISE(auto batch, reader.ReadRecordBatch(i));\n ARROW_RETURN_NOT_OK(batch->ValidateFull());\n\n // Check that the data buffers are aligned.\n std::vector unaligned_columns;\n unaligned_columns.reserve(batch->num_columns());\n if (!arrow::util::CheckAlignment(*batch, arrow::util::kValueAlignment, &unaligned_columns)) {\n return Status::Invalid(\"Column data alignment check failed\");\n }\n\n return batch;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/table_reader.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/schema_metadata.h\"\n\n#include \n\nnamespace arrow {\nclass MemoryPool;\nclass RecordBatch;\n\nnamespace ipc {\nclass RecordBatchFileReader;\n}\n} // namespace arrow\n\nnamespace pod5 {\n\nclass POD5_FORMAT_EXPORT TableRecordBatch {\npublic:\n TableRecordBatch(std::shared_ptr const & batch);\n TableRecordBatch(std::shared_ptr && batch);\n\n TableRecordBatch(TableRecordBatch &&);\n TableRecordBatch & operator=(TableRecordBatch &&);\n TableRecordBatch(TableRecordBatch const &);\n TableRecordBatch & operator=(TableRecordBatch const &);\n ~TableRecordBatch();\n\n std::size_t num_rows() const;\n\n std::shared_ptr const & batch() const { return m_batch; }\n\nprivate:\n std::shared_ptr m_batch;\n};\n\nclass POD5_FORMAT_EXPORT TableReader {\npublic:\n TableReader(\n std::shared_ptr && input_source,\n std::shared_ptr && reader,\n SchemaMetadataDescription && schema_metadata,\n arrow::MemoryPool * pool);\n TableReader(TableReader &&);\n TableReader & operator=(TableReader &&);\n TableReader(TableReader const &) = delete;\n TableReader & operator=(TableReader const &) = delete;\n ~TableReader();\n\n SchemaMetadataDescription const & schema_metadata() const { return m_schema_metadata; }\n\n std::size_t num_record_batches() const;\n\n Result CountRows() const;\n\n Result> ReadRecordBatch(int i) const;\n\nprivate:\n std::shared_ptr m_input_source;\n std::shared_ptr m_reader;\n SchemaMetadataDescription m_schema_metadata;\n};\n\n// Same as RecordBatchFileReader::ReadRecordBatch() but validates the contents.\nResult> ReadRecordBatchAndValidate(\n arrow::ipc::RecordBatchFileReader & reader,\n int i);\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/thread_pool.cpp", "content": "#include \"pod5_format/thread_pool.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nnamespace {\n\nclass ThreadPoolImpl : public ThreadPool, public std::enable_shared_from_this {\npublic:\n ThreadPoolImpl(std::size_t worker_count)\n {\n assert(worker_count > 0);\n for (std::size_t i = 0; i < std::max(1, worker_count); ++i) {\n m_threads.emplace_back([&] { run_thread(); });\n }\n }\n\n ~ThreadPoolImpl() { stop_and_drain(); }\n\n void run_thread()\n {\n bool keep_alive = true;\n std::optional work;\n\n while (keep_alive) {\n {\n std::unique_lock lock{m_work_mutex};\n\n if (work) {\n if (work->strand_id != NO_STRAND) {\n m_busy_strands[work->strand_id] = false;\n }\n work = std::nullopt;\n }\n\n // find the first piece of work whose strand isn't already busy\n for (auto it = m_work.begin(); it != m_work.end(); ++it) {\n if (it->strand_id == NO_STRAND || !m_busy_strands.at(it->strand_id)) {\n if (it->strand_id != NO_STRAND) {\n m_busy_strands[it->strand_id] = true;\n }\n work = std::move(*it);\n m_work.erase(it);\n break;\n }\n }\n\n if (!work) {\n if (m_keep_alive) {\n m_work_ready.wait(lock);\n keep_alive = m_keep_alive || !m_work.empty();\n } else {\n // If there wasn't any work for us to pick up, any remaining work must be\n // for strands with running tasks (in which case the workers handling those\n // tasks will pick them up. This will work because once a task finishes, the\n // worker will check for work *before* checking m_keep_alive. Thus it's safe\n // for *this* worker to exit.\n keep_alive = false;\n }\n continue;\n }\n }\n assert(work);\n\n if (work->callback) {\n work->callback();\n }\n }\n }\n\n void post(std::function callback) override\n {\n {\n std::lock_guard l{m_work_mutex};\n if (!m_keep_alive) {\n throw std::logic_error{\"ThreadPool: post() called after stop_and_drain()\"};\n }\n m_work.emplace_back(WorkItem{std::move(callback), NO_STRAND});\n }\n\n m_work_ready.notify_one();\n }\n\n void post(std::function callback, uint64_t const strand_id)\n {\n assert(strand_id != NO_STRAND);\n\n std::lock_guard l{m_work_mutex};\n if (!m_keep_alive) {\n throw std::logic_error{\"ThreadPool: post() called after stop_and_drain()\"};\n }\n if (m_busy_strands.size() <= strand_id) {\n m_busy_strands.resize(strand_id + 1);\n }\n m_work.emplace_back(WorkItem{std::move(callback), strand_id});\n\n // only send a wakeup if the strand isn't already busy - it's generally more efficient to do\n // this outside the lock, but the conditional makes that hard to reason about\n if (!m_busy_strands.at(strand_id)) {\n m_work_ready.notify_one();\n }\n }\n\n void stop_and_drain() override\n {\n {\n std::lock_guard lock{m_work_mutex};\n m_keep_alive = false;\n }\n m_work_ready.notify_all();\n for (auto & thread : m_threads) {\n if (thread.joinable()) {\n thread.join();\n }\n }\n\n assert(m_work.empty());\n }\n\n void wait_for_drain() override\n {\n auto const drained = [&]() {\n std::lock_guard lock{m_work_mutex};\n return m_work.empty();\n };\n while (!drained()) {\n m_work_ready.notify_all();\n std::this_thread::sleep_for(std::chrono::milliseconds{10});\n }\n }\n\n std::shared_ptr create_strand() override;\n\nprivate:\n struct WorkItem {\n std::function callback;\n uint64_t strand_id;\n\n explicit operator bool() const { return !!callback; }\n };\n\n static constexpr uint64_t NO_STRAND = UINT64_MAX;\n\n std::mutex m_work_mutex;\n bool m_keep_alive{true};\n std::condition_variable m_work_ready;\n std::deque m_work;\n std::vector m_busy_strands;\n\n std::atomic m_next_strand_id{0};\n std::vector m_threads;\n};\n\nclass StrandImpl : public ThreadPoolStrand {\npublic:\n StrandImpl(std::shared_ptr pool, uint64_t const strand_id)\n : m_pool(std::move(pool))\n , m_strand_id(strand_id)\n {\n }\n\n void post(std::function callback) override\n {\n m_pool->post(std::move(callback), m_strand_id);\n }\n\n std::shared_ptr m_pool;\n uint64_t m_strand_id;\n};\n\nstd::shared_ptr ThreadPoolImpl::create_strand()\n{\n uint64_t strand_id;\n {\n std::lock_guard l{m_work_mutex};\n if (!m_keep_alive) {\n throw std::logic_error{\"ThreadPool: create_strand() called after stop_and_drain()\"};\n }\n strand_id = m_next_strand_id++;\n }\n return std::make_shared(shared_from_this(), strand_id);\n}\n} // namespace\n\nstd::shared_ptr make_thread_pool(std::size_t worker_threads)\n{\n return std::make_shared(worker_threads);\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/thread_pool.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n\n#include \n#include \n\nnamespace pod5 {\n\nclass POD5_FORMAT_EXPORT ThreadPoolStrand {\npublic:\n virtual ~ThreadPoolStrand() = default;\n virtual void post(std::function callback) = 0;\n};\n\nclass POD5_FORMAT_EXPORT ThreadPool {\npublic:\n virtual ~ThreadPool() = default;\n virtual std::shared_ptr create_strand() = 0;\n virtual void post(std::function callback) = 0;\n /// Stops the thread pool and drains all active work.\n ///\n /// Further calls to create_strand() or post() (including on an existing strand created from\n /// this pool) will throw.\n virtual void stop_and_drain() = 0;\n\n /// Waits for the threads to process all posted work.\n virtual void wait_for_drain() = 0;\n};\n\nPOD5_FORMAT_EXPORT std::shared_ptr make_thread_pool(std::size_t worker_threads);\n} // namespace pod5\n" }, { "path": "c++/pod5_format/tuple_utils.h", "content": "#pragma once\n\n#include \n#include \n\nnamespace pod5 { namespace detail {\n\ntemplate \nvoid for_each(T && t, F f, std::integer_sequence)\n{\n auto l = {(f(std::get(t), Is), 0)...};\n (void)l;\n}\n\ntemplate \nvoid for_each_in_tuple(std::tuple & t, F f)\n{\n detail::for_each(t, f, std::make_integer_sequence());\n}\n\ntemplate \nvoid for_each_zipped(T1 && t1, T2 && t2, F f, std::integer_sequence)\n{\n auto l = {(f(std::get(t1), std::get(t2), Is), 0)...};\n (void)l;\n}\n\ntemplate \nvoid for_each_in_tuple_zipped(T1 & t1, T2 & t2, F f)\n{\n static_assert(\n std::tuple_size::value == std::tuple_size::value, \"Tuples must be same size\");\n detail::for_each_zipped(\n t1, t2, f, std::make_integer_sequence::value>());\n}\n\n}} // namespace pod5::detail\n" }, { "path": "c++/pod5_format/types.cpp", "content": "#include \"pod5_format/types.h\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace pod5 {\n\nUuid const * UuidArray::raw_values() const\n{\n auto const & array = static_cast(*storage_);\n return reinterpret_cast(array.GetValue(0));\n}\n\nUuid UuidArray::Value(int64_t i) const\n{\n auto const & array = static_cast(*storage_);\n return *reinterpret_cast(array.GetValue(i));\n}\n\nbool UuidType::ExtensionEquals(ExtensionType const & other) const\n{\n // no parameters to consider\n return other.extension_name() == extension_name();\n}\n\nstd::shared_ptr UuidType::MakeArray(std::shared_ptr data) const\n{\n DCHECK_EQ(data->type->id(), arrow::Type::EXTENSION);\n DCHECK_EQ(\n static_cast(*data->type).extension_name(), extension_name());\n return std::make_shared(data);\n}\n\nstd::string UuidType::Serialize() const { return \"\"; }\n\narrow::Result> UuidType::Deserialize(\n std::shared_ptr storage_type,\n std::string const & serialized_data) const\n{\n if (serialized_data != \"\") {\n return arrow::Status::Invalid(\"Unexpected type metadata: '\", serialized_data, \"'\");\n }\n if (!storage_type->Equals(*arrow::fixed_size_binary(16))) {\n return arrow::Status::Invalid(\n \"Incorrect storage for UuidType: '\", storage_type->ToString(), \"'\");\n }\n return std::make_shared();\n}\n\ngsl::span VbzSignalArray::Value(int64_t i) const\n{\n auto const & array = static_cast(*storage_);\n\n arrow::LargeBinaryArray::offset_type value_length = 0;\n auto value_ptr = array.GetValue(i, &value_length);\n return gsl::make_span(value_ptr, value_length);\n}\n\nstd::shared_ptr VbzSignalArray::ValueAsBuffer(int64_t i) const\n{\n auto const & array = static_cast(*storage_);\n\n auto offset = array.value_offset(i);\n auto length = array.value_length(i);\n auto const value_data = array.value_data();\n\n return arrow::SliceBuffer(value_data, offset, length);\n}\n\nbool VbzSignalType::ExtensionEquals(ExtensionType const & other) const\n{\n // no parameters to consider\n return other.extension_name() == extension_name();\n}\n\nstd::shared_ptr VbzSignalType::MakeArray(std::shared_ptr data) const\n{\n DCHECK_EQ(data->type->id(), arrow::Type::EXTENSION);\n DCHECK_EQ(\n static_cast(*data->type).extension_name(), extension_name());\n return std::make_shared(data);\n}\n\nstd::string VbzSignalType::Serialize() const { return \"\"; }\n\narrow::Result> VbzSignalType::Deserialize(\n std::shared_ptr storage_type,\n std::string const & serialized_data) const\n{\n if (serialized_data != \"\") {\n return arrow::Status::Invalid(\"Unexpected type metadata: '\", serialized_data, \"'\");\n }\n if (!storage_type->Equals(*arrow::large_binary())) {\n return arrow::Status::Invalid(\n \"Incorrect storage for VbzSignalType: '\", storage_type->ToString(), \"'\");\n }\n return std::make_shared();\n}\n\nstd::unique_ptr make_read_id_builder(arrow::MemoryPool * pool)\n{\n auto const & uuid_type = uuid();\n assert(uuid_type->id() == arrow::Type::EXTENSION);\n auto result = std::make_unique(uuid_type->storage_type(), pool);\n assert(result->byte_width() == 16);\n return result;\n}\n\nstd::shared_ptr const & vbz_signal()\n{\n static auto vbz_signal = std::make_shared();\n return vbz_signal;\n}\n\nstd::shared_ptr const & uuid()\n{\n static auto uuid = std::make_shared();\n return uuid;\n}\n\nnamespace {\n\nstd::mutex & get_pod5_register_mutex()\n{\n // Heap allocated so that it's safe for user code to call during static init\n // and destruction, not that they should.\n static std::mutex & m = *new std::mutex{};\n return m;\n}\n\nstd::size_t s_pod5_register_count(0);\n\n} // namespace\n\npod5::Status register_extension_types()\n{\n std::lock_guard lock(get_pod5_register_mutex());\n if (++s_pod5_register_count == 1) {\n ARROW_RETURN_NOT_OK(arrow::RegisterExtensionType(uuid()));\n ARROW_RETURN_NOT_OK(arrow::RegisterExtensionType(vbz_signal()));\n }\n return pod5::Status::OK();\n}\n\npod5::Status unregister_extension_types()\n{\n std::lock_guard lock(get_pod5_register_mutex());\n auto register_count = --s_pod5_register_count;\n if (register_count == 0) {\n if (arrow::GetExtensionType(\"minknow.uuid\")) {\n ARROW_RETURN_NOT_OK(arrow::UnregisterExtensionType(\"minknow.uuid\"));\n }\n if (arrow::GetExtensionType(\"minknow.vbz\")) {\n ARROW_RETURN_NOT_OK(arrow::UnregisterExtensionType(\"minknow.vbz\"));\n }\n }\n return pod5::Status::OK();\n}\n\nbool check_extension_types_registered()\n{\n std::lock_guard lock(get_pod5_register_mutex());\n return s_pod5_register_count > 0;\n}\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/types.h", "content": "#pragma once\n\n#include \"pod5_format/pod5_format_export.h\"\n#include \"pod5_format/result.h\"\n#include \"pod5_format/uuid.h\"\n\n#include \n#include \n#include \n\nnamespace pod5 {\n\nclass POD5_FORMAT_EXPORT UuidArray : public arrow::ExtensionArray {\npublic:\n using IteratorType = arrow::stl::ArrayIterator;\n\n using ExtensionArray::ExtensionArray;\n\n Uuid const * raw_values() const;\n\n Uuid Value(int64_t i) const;\n\n // this isn't actually a view - it copies the data - but\n // (a) it's only 16 bytes, which is what a view (pointer + size) would require anyway\n // (b) arrow::std::ArrayIterator hard-codes the name of this method (even though it is supposed\n // to be configurable via the ValueAccessor template parameter)\n Uuid GetView(int64_t i) const { return Value(i); }\n\n std::optional operator[](int64_t i) const { return *IteratorType(*this, i); }\n\n IteratorType begin() const { return IteratorType(*this); }\n\n IteratorType end() const { return IteratorType(*this, length()); }\n};\n\nclass POD5_FORMAT_EXPORT UuidType : public arrow::ExtensionType {\npublic:\n UuidType() : ExtensionType(arrow::fixed_size_binary(16)) {}\n\n std::string extension_name() const override { return \"minknow.uuid\"; }\n\n bool ExtensionEquals(ExtensionType const & other) const override;\n std::shared_ptr MakeArray(std::shared_ptr data) const override;\n std::string Serialize() const override;\n arrow::Result> Deserialize(\n std::shared_ptr storage_type,\n std::string const & serialized_data) const override;\n};\n\nclass POD5_FORMAT_EXPORT VbzSignalArray : public arrow::ExtensionArray {\npublic:\n using IteratorType = arrow::stl::ArrayIterator;\n\n gsl::span Value(int64_t i) const;\n std::shared_ptr ValueAsBuffer(int64_t i) const;\n\n using ExtensionArray::ExtensionArray;\n};\n\nclass POD5_FORMAT_EXPORT VbzSignalType : public arrow::ExtensionType {\npublic:\n VbzSignalType() : ExtensionType(arrow::large_binary()) {}\n\n std::string extension_name() const override { return \"minknow.vbz\"; }\n\n bool ExtensionEquals(ExtensionType const & other) const override;\n std::shared_ptr MakeArray(std::shared_ptr data) const override;\n std::string Serialize() const override;\n arrow::Result> Deserialize(\n std::shared_ptr storage_type,\n std::string const & serialized_data) const override;\n};\n\nstd::unique_ptr make_read_id_builder(arrow::MemoryPool * pool);\n\nstd::shared_ptr const & vbz_signal();\nstd::shared_ptr const & uuid();\n\n/// \\brief Register all required extension types.\nPOD5_FORMAT_EXPORT pod5::Status register_extension_types();\n\n/// \\brief Unregister all required extension types.\nPOD5_FORMAT_EXPORT pod5::Status unregister_extension_types();\n\n/// \\brief Returns true iff the required extension types are registered.\n/// \\details The caller can expect the extension types to be registered if the number of calls to\n/// `register_extension_types` exceeds the number of calls to `unregister_extension_types`.\nbool check_extension_types_registered();\n\n} // namespace pod5\n" }, { "path": "c++/pod5_format/uuid.h", "content": "#pragma once\n\n// This file contains code from https://github.com/mariusbancila/stduuid/ which has the following\n// license:\n//\n// MIT License\n//\n// Copyright (c) 2017\n//\n// Permission is hereby granted, free of charge, to any person obtaining a copy\n// of this software and associated documentation files (the \"Software\"), to deal\n// in the Software without restriction, including without limitation the rights\n// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n// copies of the Software, and to permit persons to whom the Software is\n// furnished to do so, subject to the following conditions:\n//\n// The above copyright notice and this permission notice shall be included in all\n// copies or substantial portions of the Software.\n//\n// THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n// SOFTWARE.\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace pod5 {\n\nnamespace uuid_detail {\n\ntemplate \n[[nodiscard]] constexpr inline unsigned char hex2char(TChar const ch) noexcept\n{\n if (ch >= static_cast('0') && ch <= static_cast('9')) {\n return static_cast(ch - static_cast('0'));\n }\n if (ch >= static_cast('a') && ch <= static_cast('f')) {\n return static_cast(10 + ch - static_cast('a'));\n }\n if (ch >= static_cast('A') && ch <= static_cast('F')) {\n return static_cast(10 + ch - static_cast('A'));\n }\n return 0;\n}\n\ntemplate \n[[nodiscard]] constexpr inline bool is_hex(TChar const ch) noexcept\n{\n return (ch >= static_cast('0') && ch <= static_cast('9'))\n || (ch >= static_cast('a') && ch <= static_cast('f'))\n || (ch >= static_cast('A') && ch <= static_cast('F'));\n}\n\ntemplate \n[[nodiscard]] constexpr std::basic_string_view to_string_view(TChar const * str) noexcept\n{\n if (str) {\n return str;\n }\n return {};\n}\n\ntemplate \n[[nodiscard]] constexpr std::\n basic_string_view\n to_string_view(StringType const & str) noexcept\n{\n return str;\n}\n\ntemplate \ninline constexpr CharT empty_guid[37] = \"00000000-0000-0000-0000-000000000000\";\n\ntemplate <>\ninline constexpr wchar_t empty_guid[37] = L\"00000000-0000-0000-0000-000000000000\";\n\ntemplate \ninline constexpr CharT guid_encoder[17] = \"0123456789abcdef\";\n\ntemplate <>\ninline constexpr wchar_t guid_encoder[17] = L\"0123456789abcdef\";\n\n} // namespace uuid_detail\n\n// Forward declare uuid & to_string so that we can declare to_string as a friend later.\nclass Uuid;\ntemplate <\n class CharT = char,\n class Traits = std::char_traits,\n class Allocator = std::allocator>\nstd::basic_string to_string(Uuid const & id);\n\n/// A representation of a Universally Unique IDentifier.\n///\n/// This code implements part of RFC 4122. It does not aim to be a complete implementation of UUIDs,\n/// but provides enough for the uses in the POD5 library.\nclass Uuid {\npublic:\n using value_type = uint8_t;\n\n constexpr Uuid() noexcept = default;\n\n Uuid(value_type const (&arr)[16]) noexcept\n {\n std::copy(std::cbegin(arr), std::cend(arr), std::begin(m_data));\n }\n\n constexpr Uuid(std::array const & arr) noexcept : m_data{arr} {}\n\n template \n explicit Uuid(ForwardIterator first, ForwardIterator last)\n {\n if (std::distance(first, last) == 16) {\n std::copy(first, last, std::begin(m_data));\n }\n }\n\n [[nodiscard]] constexpr bool is_nil() const noexcept\n {\n for (size_t i = 0; i < m_data.size(); ++i) {\n if (m_data[i] != 0) {\n return false;\n }\n }\n return true;\n }\n\n void swap(Uuid & other) noexcept { m_data.swap(other.m_data); }\n\n uint8_t const * data() const noexcept { return m_data.data(); }\n\n size_t size() const noexcept { return m_data.size(); }\n\n void to_c_array(value_type (&arr)[16]) const noexcept\n {\n std::copy(std::cbegin(m_data), std::cend(m_data), std::begin(arr));\n }\n\n // Note: uustr must be at least 36 characters\n template \n void write_to(CharT * uustr) const noexcept\n {\n for (size_t i = 0, index = 0; i < 36; ++i) {\n if (i == 8 || i == 13 || i == 18 || i == 23) {\n uustr[i] = uuid_detail::empty_guid[i];\n continue;\n }\n uustr[i] = uuid_detail::guid_encoder[m_data[index] >> 4 & 0x0f];\n uustr[++i] = uuid_detail::guid_encoder[m_data[index] & 0x0f];\n index++;\n }\n }\n\n template \n [[nodiscard]] constexpr static std::optional from_string(\n StringType const & in_str) noexcept\n {\n auto str = uuid_detail::to_string_view(in_str);\n bool firstDigit = true;\n size_t hasBraces = 0;\n size_t index = 0;\n\n std::array data{{0}};\n\n if (str.empty()) {\n return {};\n }\n\n if (str.front() == '{') {\n hasBraces = 1;\n }\n if (hasBraces && str.back() != '}') {\n return {};\n }\n\n for (size_t i = hasBraces; i < str.size() - hasBraces; ++i) {\n if (str[i] == '-') {\n continue;\n }\n\n if (index >= 16 || !uuid_detail::is_hex(str[i])) {\n return {};\n }\n\n if (firstDigit) {\n data[index] = static_cast(uuid_detail::hex2char(str[i]) << 4);\n firstDigit = false;\n } else {\n data[index] = static_cast(data[index] | uuid_detail::hex2char(str[i]));\n index++;\n firstDigit = true;\n }\n }\n\n if (index < 16) {\n return {};\n }\n\n return Uuid{data};\n }\n\nprivate:\n std::array m_data{{0}};\n\n friend bool operator==(Uuid const & lhs, Uuid const & rhs) noexcept;\n friend bool operator<(Uuid const & lhs, Uuid const & rhs) noexcept;\n\n template \n friend std::basic_ostream & operator<<(\n std::basic_ostream & s,\n Uuid const & id);\n\n template \n friend std::basic_string to_string(Uuid const & id);\n\n friend std::hash;\n};\n\n[[nodiscard]] inline bool operator==(Uuid const & lhs, Uuid const & rhs) noexcept\n{\n return lhs.m_data == rhs.m_data;\n}\n\n[[nodiscard]] inline bool operator!=(Uuid const & lhs, Uuid const & rhs) noexcept\n{\n return !(lhs == rhs);\n}\n\n[[nodiscard]] inline bool operator<(Uuid const & lhs, Uuid const & rhs) noexcept\n{\n return lhs.m_data < rhs.m_data;\n}\n\ntemplate \n[[nodiscard]] inline std::basic_string to_string(Uuid const & id)\n{\n std::basic_string uustr{uuid_detail::empty_guid};\n id.write_to(uustr.data());\n return uustr;\n}\n\ntemplate \nstd::basic_ostream & operator<<(std::basic_ostream & s, Uuid const & id)\n{\n s << to_string(id);\n return s;\n}\n\ninline void swap(Uuid & lhs, Uuid & rhs) noexcept { lhs.swap(rhs); }\n\ntemplate \nclass BasicUuidRandomGenerator {\npublic:\n using engine_type = UniformRandomNumberGenerator;\n\n explicit BasicUuidRandomGenerator(engine_type & gen) : generator(&gen) {}\n\n explicit BasicUuidRandomGenerator(engine_type * gen) : generator(gen) {}\n\n [[nodiscard]] Uuid operator()()\n {\n alignas(uint32_t) uint8_t bytes[16];\n for (int i = 0; i < 16; i += 4) {\n *reinterpret_cast(bytes + i) = distribution(*generator);\n }\n\n // variant must be 10xxxxxx\n bytes[8] &= 0xBF;\n bytes[8] |= 0x80;\n\n // version must be 0100xxxx\n bytes[6] &= 0x4F;\n bytes[6] |= 0x40;\n\n return Uuid{std::begin(bytes), std::end(bytes)};\n }\n\nprivate:\n std::uniform_int_distribution distribution;\n UniformRandomNumberGenerator * generator;\n};\n\nusing UuidRandomGenerator = BasicUuidRandomGenerator;\n\n} // namespace pod5\n\nnamespace std {\ntemplate <>\nstruct hash {\n using argument_type = pod5::Uuid;\n using result_type = std::size_t;\n\n [[nodiscard]] result_type operator()(argument_type const & uuid) const\n {\n uint64_t l = static_cast(uuid.m_data[0]) << 56\n | static_cast(uuid.m_data[1]) << 48\n | static_cast(uuid.m_data[2]) << 40\n | static_cast(uuid.m_data[3]) << 32\n | static_cast(uuid.m_data[4]) << 24\n | static_cast(uuid.m_data[5]) << 16\n | static_cast(uuid.m_data[6]) << 8\n | static_cast(uuid.m_data[7]);\n uint64_t h = static_cast(uuid.m_data[8]) << 56\n | static_cast(uuid.m_data[9]) << 48\n | static_cast(uuid.m_data[10]) << 40\n | static_cast(uuid.m_data[11]) << 32\n | static_cast(uuid.m_data[12]) << 24\n | static_cast(uuid.m_data[13]) << 16\n | static_cast(uuid.m_data[14]) << 8\n | static_cast(uuid.m_data[15]);\n\n if constexpr (sizeof(result_type) > 4) {\n return result_type(l ^ h);\n } else {\n uint64_t hash64 = l ^ h;\n return result_type(uint32_t(hash64 >> 32) ^ uint32_t(hash64));\n }\n }\n};\n} // namespace std\n" }, { "path": "c++/pod5_format/version.h.in", "content": "#pragma once\n\n#include \n\nnamespace pod5 {\n\nstd::uint16_t const Pod5MajorVersion = @POD5_VERSION_MAJOR@;\nstd::uint16_t const Pod5MinorVersion = @POD5_VERSION_MINOR@;\nstd::uint16_t const Pod5RevVersion = @POD5_VERSION_REV@;\n\nstd::string const Pod5Version = \"@POD5_NUMERIC_VERSION@\";\n\n}\n" }, { "path": "c++/pod5_format_pybind/CMakeLists.txt", "content": "\npybind11_add_module(pod5_format_pybind\n api.h\n bindings.cpp\n utils.h\n subset.cpp\n subset.h\n\n repack/repack_functions.h\n repack/repack_states.h\n repack/repack_utils.h\n repack/repack_output.cpp\n repack/repack_output.h\n repack/repacker.cpp\n repack/repacker.h\n)\n\ntarget_link_libraries(pod5_format_pybind\n PRIVATE\n pod5_format\n)\n\nif (NOT MSVC)\n target_compile_options(pod5_format_pybind PRIVATE ${pod5_warning_options})\nendif()\n\nset_target_properties(pod5_format_pybind\n PROPERTIES\n POSITION_INDEPENDENT_CODE 1\n CXX_STANDARD 20\n)\n\n# Non-conan license files to copy.\nset(pod5_cxx_licenses_src\n \"${CMAKE_SOURCE_DIR}/LICENSE.md\"\n \"${CMAKE_SOURCE_DIR}/third_party/licenses/gsl-lite.txt\"\n \"${CMAKE_SOURCE_DIR}/third_party/pybind11/LICENSE\"\n)\n# Destination name for the above files.\nset(pod5_cxx_licenses_dst\n \"LICENSE.md\"\n \"gsl-lite.txt\"\n \"pybind11.txt\"\n)\n\nset(python_project_root \"${CMAKE_SOURCE_DIR}/python/lib_pod5/\")\n\nconfigure_file(\n ${CMAKE_CURRENT_SOURCE_DIR}/_version.py.in\n ${python_project_root}/src/lib_pod5/_version.py\n)\n\nset(wheel_output_stub \"${CMAKE_CURRENT_BINARY_DIR}/wheel.touch\")\n\nset(wheel_output_dir \"${CMAKE_CURRENT_BINARY_DIR}/wheel_${POD5_FULL_VERSION}\")\nfile(MAKE_DIRECTORY ${wheel_output_dir})\n\nadd_custom_command(\n OUTPUT \"${wheel_output_stub}\"\n COMMAND ${CMAKE_COMMAND}\n ARGS\n -D \"PYTHON_EXECUTABLE=${Python_EXECUTABLE}\"\n -D \"PYTHON_PROJECT_DIR=${python_project_root}\"\n -D \"PYBIND_INPUT_LIB=$\"\n -D \"WHEEL_OUTPUT_DIR=${wheel_output_dir}\"\n -D \"POD5_CONAN_LICENSES=${CMAKE_BINARY_DIR}/pod5_conan_licenses\"\n -D \"POD5_CXX_LICENSES_SRC=${pod5_cxx_licenses_src}\"\n -D \"POD5_CXX_LICENSES_DST=${pod5_cxx_licenses_dst}\"\n -P \"${CMAKE_CURRENT_SOURCE_DIR}/build_wheel.cmake\"\n DEPENDS\n pod5_format_pybind\n VERBATIM\n)\n\nadd_custom_target(lib_pod5_python_wheel ALL\n SOURCES\n build_wheel.cmake\n DEPENDS\n \"${wheel_output_stub}\"\n)\n\ninstall(\n DIRECTORY \"${wheel_output_dir}/\"\n DESTINATION \".\"\n COMPONENT wheel\n FILES_MATCHING PATTERN \"lib_pod5*.whl\"\n)\n" }, { "path": "c++/pod5_format_pybind/_version.py.in", "content": "# This file is auto generated by cmake during compilation\n__version__ = version = \"@POD5_FULL_VERSION@\"\n__version_tuple__ = version_tuple = (@POD5_VERSION_MAJOR@, @POD5_VERSION_MINOR@, @POD5_VERSION_REV@)\n" }, { "path": "c++/pod5_format_pybind/api.h", "content": "#pragma once\n\n#include \"pod5_format/async_signal_loader.h\"\n#include \"pod5_format/c_api.h\"\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/file_updater.h\"\n#include \"pod5_format/file_writer.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/signal_compression.h\"\n#include \"pod5_format/signal_table_reader.h\"\n#include \"pod5_format/thread_pool.h\"\n#include \"pod5_format/uuid.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n\nnamespace py = pybind11;\n\ninline std::shared_ptr create_file(\n char const * path,\n std::string const & writer_name,\n pod5::FileWriterOptions const * options)\n{\n pod5::FileWriterOptions dummy;\n POD5_PYTHON_ASSIGN_OR_RAISE(\n auto writer,\n pod5::create_file_writer(\n path, writer_name, options ? *options : pod5::FileWriterOptions{}));\n return writer;\n}\n\ninline pod5::RecoveryDetails recover_file(\n char const * src_filename,\n char const * dest_filename,\n pod5::RecoverFileOptions const * const options)\n{\n POD5_PYTHON_ASSIGN_OR_RAISE(\n pod5::RecoveryDetails details,\n pod5::recover_file(\n src_filename, dest_filename, options ? *options : pod5::RecoverFileOptions{}));\n return details;\n}\n\nclass Pod5SignalCacheBatch {\npublic:\n Pod5SignalCacheBatch(\n pod5::AsyncSignalLoader::SamplesMode samples_mode,\n pod5::CachedBatchSignalData && cached_data)\n : m_samples_mode(samples_mode)\n , m_cached_data(std::move(cached_data))\n {\n }\n\n py::array_t sample_count() const\n {\n return py::array_t(\n m_cached_data.sample_count().size(), m_cached_data.sample_count().data());\n }\n\n py::list samples() const\n {\n py::list py_samples;\n if (m_samples_mode != pod5::AsyncSignalLoader::SamplesMode::Samples) {\n return py_samples;\n }\n for (auto const & row_samples : m_cached_data.samples()) {\n py_samples.append(\n py::array_t(\n {row_samples.size()}, {sizeof(std::int16_t)}, row_samples.data()));\n }\n\n return py_samples;\n }\n\n std::uint32_t batch_index() const { return m_cached_data.batch_index(); }\n\nprivate:\n pod5::AsyncSignalLoader::SamplesMode m_samples_mode;\n pod5::CachedBatchSignalData m_cached_data;\n};\n\nclass Pod5AsyncSignalLoader {\npublic:\n // Make an async loader for all reads in the file\n Pod5AsyncSignalLoader(\n std::shared_ptr const & reader,\n pod5::AsyncSignalLoader::SamplesMode samples_mode,\n std::size_t worker_count = std::thread::hardware_concurrency(),\n std::size_t max_pending_batches = 10)\n : m_samples_mode(samples_mode)\n , m_batch_counts_ref({})\n , m_batch_rows_ref({})\n , m_async_loader(reader, samples_mode, {}, {}, worker_count, max_pending_batches)\n {\n }\n\n // Make an async loader for specific batches\n Pod5AsyncSignalLoader(\n std::shared_ptr const & reader,\n pod5::AsyncSignalLoader::SamplesMode samples_mode,\n py::array_t && batches,\n std::size_t worker_count = std::thread::hardware_concurrency(),\n std::size_t max_pending_batches = 10)\n : m_samples_mode(samples_mode)\n , m_batch_sizes(make_batch_counts(reader, batches))\n , m_async_loader(\n reader,\n samples_mode,\n gsl::make_span(m_batch_sizes),\n {},\n worker_count,\n max_pending_batches)\n {\n }\n\n // Make an async loader for specific reads in specific batches\n Pod5AsyncSignalLoader(\n std::shared_ptr const & reader,\n pod5::AsyncSignalLoader::SamplesMode samples_mode,\n py::array_t && batch_counts,\n py::array_t && batch_rows,\n std::size_t worker_count = std::thread::hardware_concurrency(),\n std::size_t max_pending_batches = 10)\n : m_samples_mode(samples_mode)\n , m_batch_counts_ref(std::move(batch_counts))\n , m_batch_rows_ref(std::move(batch_rows))\n , m_async_loader(\n reader,\n samples_mode,\n gsl::make_span(m_batch_counts_ref.data(), m_batch_counts_ref.size()),\n gsl::make_span(m_batch_rows_ref.data(), m_batch_rows_ref.size()),\n worker_count,\n max_pending_batches)\n {\n }\n\n std::shared_ptr release_next_batch()\n {\n auto batch = m_async_loader.release_next_batch();\n if (!batch.ok()) {\n throw std::runtime_error(batch.status().ToString());\n }\n\n if (!*batch) {\n assert(m_async_loader.is_finished());\n throw pybind11::stop_iteration();\n }\n\n return std::make_shared(m_samples_mode, std::move(**batch));\n }\n\n std::vector make_batch_counts(\n std::shared_ptr const & reader,\n py::array_t const & batches)\n {\n std::vector batch_counts(reader->num_read_record_batches(), 0);\n for (auto const & batch_idx : gsl::make_span(batches.data(), batches.shape(0))) {\n auto read_batch = reader->read_read_record_batch(batch_idx);\n if (!read_batch.ok()) {\n throw std::runtime_error(\n \"Failed to query read batch count: \" + read_batch.status().ToString());\n }\n\n batch_counts[batch_idx] = read_batch->num_rows();\n }\n return batch_counts;\n }\n\n pod5::AsyncSignalLoader::SamplesMode m_samples_mode;\n std::vector m_batch_sizes;\n py::array_t m_batch_counts_ref;\n py::array_t m_batch_rows_ref;\n pod5::AsyncSignalLoader m_async_loader;\n};\n\nstruct Pod5FileReaderPtr {\n std::shared_ptr reader = nullptr;\n\n Pod5FileReaderPtr(std::shared_ptr && reader_) : reader(std::move(reader_)) {}\n\n pod5::FileLocation get_file_run_info_table_location() const\n {\n return reader->run_info_table_location();\n }\n\n pod5::FileLocation get_file_read_table_location() const\n {\n return reader->read_table_location();\n }\n\n pod5::FileLocation get_file_signal_table_location() const\n {\n return reader->signal_table_location();\n }\n\n std::string get_file_version_pre_migration() const\n {\n return reader->file_version_pre_migration().to_string();\n }\n\n void close() { reader = nullptr; }\n\n std::size_t plan_traversal(\n py::array_t const & read_id_data,\n py::array_t & batch_counts,\n py::array_t & batch_rows)\n {\n auto const read_id_count = read_id_data.shape(0);\n auto search_input = pod5::ReadIdSearchInput(\n gsl::make_span(\n reinterpret_cast(read_id_data.data()), read_id_count));\n\n POD5_PYTHON_ASSIGN_OR_RAISE(\n auto find_success_count,\n reader->search_for_read_ids(\n search_input,\n gsl::make_span(batch_counts.mutable_data(), reader->num_read_record_batches()),\n gsl::make_span(batch_rows.mutable_data(), read_id_count)));\n\n return find_success_count;\n }\n\n std::shared_ptr batch_get_signal(bool get_samples, bool get_sample_count)\n {\n return std::make_shared(\n reader,\n get_samples ? pod5::AsyncSignalLoader::SamplesMode::Samples\n : pod5::AsyncSignalLoader::SamplesMode::NoSamples);\n }\n\n std::shared_ptr batch_get_signal_batches(\n bool get_samples,\n bool get_sample_count,\n py::array_t && batches)\n {\n return std::make_shared(\n reader,\n get_samples ? pod5::AsyncSignalLoader::SamplesMode::Samples\n : pod5::AsyncSignalLoader::SamplesMode::NoSamples,\n std::move(batches));\n }\n\n std::shared_ptr batch_get_signal_selection(\n bool get_samples,\n bool get_sample_count,\n py::array_t && batch_counts,\n py::array_t && batch_rows)\n {\n return std::make_shared(\n reader,\n get_samples ? pod5::AsyncSignalLoader::SamplesMode::Samples\n : pod5::AsyncSignalLoader::SamplesMode::NoSamples,\n std::move(batch_counts),\n std::move(batch_rows));\n }\n};\n\ninline Pod5FileReaderPtr open_file(char const * filename)\n{\n POD5_PYTHON_ASSIGN_OR_RAISE(auto reader, pod5::open_file_reader(filename, {}));\n return Pod5FileReaderPtr(std::move(reader));\n}\n\ninline void write_updated_file_to_dest(Pod5FileReaderPtr source, char const * dest_filename)\n{\n POD5_PYTHON_RETURN_NOT_OK(\n pod5::update_file(arrow::default_memory_pool(), source.reader, dest_filename));\n}\n\ninline pod5::RunInfoDictionaryIndex FileWriter_add_run_info(\n pod5::FileWriter & w,\n std::string & acquisition_id,\n std::int64_t acquisition_start_time,\n std::int16_t adc_max,\n std::int16_t adc_min,\n std::vector> && context_tags,\n std::string & experiment_name,\n std::string & flow_cell_id,\n std::string & flow_cell_product_code,\n std::string & protocol_name,\n std::string & protocol_run_id,\n std::int64_t protocol_start_time,\n std::string & sample_id,\n std::uint16_t sample_rate,\n std::string & sequencing_kit,\n std::string & sequencer_position,\n std::string & sequencer_position_type,\n std::string & software,\n std::string & system_name,\n std::string & system_type,\n std::vector> && tracking_id)\n{\n return throw_on_error(w.add_run_info(\n {std::move(acquisition_id),\n acquisition_start_time,\n adc_max,\n adc_min,\n std::move(context_tags),\n std::move(experiment_name),\n std::move(flow_cell_id),\n std::move(flow_cell_product_code),\n std::move(protocol_name),\n std::move(protocol_run_id),\n std::move(protocol_start_time),\n std::move(sample_id),\n sample_rate,\n std::move(sequencing_kit),\n std::move(sequencer_position),\n std::move(sequencer_position_type),\n std::move(software),\n std::move(system_name),\n std::move(system_type),\n std::move(tracking_id)}));\n}\n\ninline pod5::ReadData make_read_data(\n std::size_t row_id,\n py::array_t const & read_id_data,\n py::array_t const & read_numbers,\n py::array_t const & start_samples,\n py::array_t const & channels,\n py::array_t const & wells,\n py::array_t const & pore_types,\n py::array_t const & calibration_offsets,\n py::array_t const & calibration_scales,\n py::array_t const & median_befores,\n py::array_t const & end_reasons,\n py::array_t const & end_reason_forceds,\n py::array_t const & run_infos,\n py::array_t const &\n num_minknow_events,\n py::array_t const & tracked_scaling_scale,\n py::array_t const & tracked_scaling_shift,\n py::array_t const & predicted_scaling_scale,\n py::array_t const & predicted_scaling_shift,\n py::array_t const &\n num_reads_since_mux_change,\n py::array_t const & time_since_mux_change,\n py::array_t const & open_pore_level)\n{\n auto read_ids = reinterpret_cast(read_id_data.data(0));\n return pod5::ReadData{\n read_ids[row_id],\n *read_numbers.data(row_id),\n *start_samples.data(row_id),\n *channels.data(row_id),\n *wells.data(row_id),\n *pore_types.data(row_id),\n *calibration_offsets.data(row_id),\n *calibration_scales.data(row_id),\n *median_befores.data(row_id),\n *end_reasons.data(row_id),\n *end_reason_forceds.data(row_id),\n *run_infos.data(row_id),\n *num_minknow_events.data(row_id),\n *tracked_scaling_scale.data(row_id),\n *tracked_scaling_shift.data(row_id),\n *predicted_scaling_scale.data(row_id),\n *predicted_scaling_shift.data(row_id),\n *num_reads_since_mux_change.data(row_id),\n *time_since_mux_change.data(row_id),\n *open_pore_level.data(row_id)};\n}\n\ninline void FileWriter_add_reads(\n pod5::FileWriter & w,\n std::size_t count,\n py::array_t const & read_id_data,\n py::array_t const & read_numbers,\n py::array_t const & start_samples,\n py::array_t const & channels,\n py::array_t const & wells,\n py::array_t const & pore_types,\n py::array_t const & calibration_offsets,\n py::array_t const & calibration_scales,\n py::array_t const & median_befores,\n py::array_t const & end_reasons,\n py::array_t const & end_reason_forceds,\n py::array_t const & run_infos,\n py::array_t const &\n num_minknow_events,\n py::array_t const & tracked_scaling_scales,\n py::array_t const & tracked_scaling_shifts,\n py::array_t const & predicted_scaling_scales,\n py::array_t const & predicted_scaling_shifts,\n py::array_t const &\n num_reads_since_mux_changes,\n py::array_t const & time_since_mux_changes,\n py::array_t const & open_pore_levels,\n py::list signal_ptrs)\n{\n if (read_id_data.shape(1) != 16) {\n throw std::runtime_error(\"Read id array is of unexpected size\");\n }\n\n auto signal_it = signal_ptrs.begin();\n for (std::size_t i = 0; i < count; ++i, ++signal_it) {\n if (signal_it == signal_ptrs.end()) {\n throw std::runtime_error(\"Missing signal data\");\n }\n auto signal =\n signal_it->cast>();\n auto signal_span = gsl::make_span(signal.data(), signal.size());\n\n auto read_data = make_read_data(\n i,\n read_id_data,\n read_numbers,\n start_samples,\n channels,\n wells,\n pore_types,\n calibration_offsets,\n calibration_scales,\n median_befores,\n end_reasons,\n end_reason_forceds,\n run_infos,\n num_minknow_events,\n tracked_scaling_scales,\n tracked_scaling_shifts,\n predicted_scaling_scales,\n predicted_scaling_shifts,\n num_reads_since_mux_changes,\n time_since_mux_changes,\n open_pore_levels);\n\n throw_on_error(w.add_complete_read(read_data, signal_span));\n }\n}\n\ninline void FileWriter_add_reads_pre_compressed(\n pod5::FileWriter & w,\n std::size_t count,\n py::array_t const & read_id_data,\n py::array_t const & read_numbers,\n py::array_t const & start_samples,\n py::array_t const & channels,\n py::array_t const & wells,\n py::array_t const & pore_types,\n py::array_t const & calibration_offsets,\n py::array_t const & calibration_scales,\n py::array_t const & median_befores,\n py::array_t const & end_reasons,\n py::array_t const & end_reason_forceds,\n py::array_t const & run_infos,\n py::array_t const &\n num_minknow_events,\n py::array_t const & tracked_scaling_scales,\n py::array_t const & tracked_scaling_shifts,\n py::array_t const & predicted_scaling_scales,\n py::array_t const & predicted_scaling_shifts,\n py::array_t const &\n num_reads_since_mux_changes,\n py::array_t const & time_since_mux_changes,\n py::array_t const & open_pore_levels,\n py::list compressed_signal_ptrs,\n py::array_t const & sample_counts,\n py::array_t const &\n signal_chunk_counts)\n{\n if (read_id_data.shape(1) != 16) {\n throw std::runtime_error(\"Read id array is of unexpected size\");\n }\n\n auto read_ids = reinterpret_cast(read_id_data.data(0));\n auto compressed_signal_it = compressed_signal_ptrs.begin();\n auto sample_counts_it = sample_counts.data();\n for (std::size_t i = 0; i < count; ++i) {\n auto const read_id = read_ids[i];\n\n auto const signal_chunk_count = *signal_chunk_counts.data(i);\n std::uint64_t signal_duration_count = 0;\n std::vector signal_rows(signal_chunk_count);\n for (std::size_t signal_chunk_idx = 0; signal_chunk_idx < signal_chunk_count;\n ++signal_chunk_idx)\n {\n if (compressed_signal_it == compressed_signal_ptrs.end()) {\n throw std::runtime_error(\"Missing signal data\");\n }\n auto compressed_signal =\n compressed_signal_it\n ->cast>();\n auto compressed_signal_span =\n gsl::make_span(compressed_signal.data(), compressed_signal.size());\n\n auto signal_row = throw_on_error(\n w.add_pre_compressed_signal(read_id, compressed_signal_span, *sample_counts_it));\n signal_rows[signal_chunk_idx] = signal_row;\n\n signal_duration_count += *sample_counts_it;\n ++compressed_signal_it;\n ++sample_counts_it;\n }\n\n auto read_data = make_read_data(\n i,\n read_id_data,\n read_numbers,\n start_samples,\n channels,\n wells,\n pore_types,\n calibration_offsets,\n calibration_scales,\n median_befores,\n end_reasons,\n end_reason_forceds,\n run_infos,\n num_minknow_events,\n tracked_scaling_scales,\n tracked_scaling_shifts,\n predicted_scaling_scales,\n predicted_scaling_shifts,\n num_reads_since_mux_changes,\n time_since_mux_changes,\n open_pore_levels);\n\n throw_on_error(w.add_complete_read(read_data, signal_rows, signal_duration_count));\n }\n}\n\ninline void decompress_signal_wrapper(\n py::array_t const & compressed_signal,\n py::array_t & signal_out)\n{\n throw_on_error(\n pod5::decompress_signal(\n gsl::make_span(compressed_signal.data(0), compressed_signal.shape(0)),\n arrow::system_memory_pool(),\n gsl::make_span(signal_out.mutable_data(0), signal_out.shape(0))));\n}\n\ninline std::size_t compress_signal_wrapper(\n py::array_t const & signal,\n py::array_t & compressed_signal_out)\n{\n auto size = throw_on_error(\n pod5::compress_signal(\n gsl::make_span(signal.data(), signal.shape(0)),\n arrow::system_memory_pool(),\n gsl::make_span(compressed_signal_out.mutable_data(), compressed_signal_out.shape(0))));\n\n return size;\n}\n\ninline std::size_t vbz_compressed_signal_max_size(std::size_t sample_count)\n{\n POD5_PYTHON_ASSIGN_OR_RAISE(\n std::size_t const max_size, pod5::compressed_signal_max_size(sample_count));\n return max_size;\n}\n\ninline std::size_t load_read_id_iterable(\n py::iterable const & read_ids_str,\n py::array_t & read_id_data_out)\n{\n std::size_t out_idx = 0;\n auto read_ids = reinterpret_cast(read_id_data_out.mutable_data());\n auto read_ids_out_len = read_id_data_out.shape(0);\n\n std::string temp_uuid;\n for (auto & read_id : read_ids_str) {\n if (out_idx >= (std::size_t)read_ids_out_len) {\n throw std::runtime_error(\"Too many input uuids for output container\");\n }\n\n temp_uuid = read_id.cast();\n if (auto const found_uuid = pod5::Uuid::from_string(temp_uuid)) {\n read_ids[out_idx++] = *found_uuid;\n }\n // if it's invalid, ignore it - we will return one fewer read ids than expected and the caller can deal with it.\n }\n\n return out_idx;\n}\n\ninline py::list format_read_id_to_str(\n py::array_t & read_id_data_out)\n{\n if (read_id_data_out.size() % 16 != 0) {\n throw std::runtime_error(\n \"Unexpected amount of data for read id - expected data to align to 16 bytes.\");\n }\n\n py::list result;\n\n std::array str_data;\n std::size_t const count = read_id_data_out.size() / 16;\n for (std::size_t i = 0; i < count; ++i) {\n auto read_id_data = read_id_data_out.data() + (i * 16);\n\n pod5_format_read_id(read_id_data, str_data.data());\n result.append(py::str(str_data.data(), str_data.size() - 1));\n }\n\n return result;\n}\n" }, { "path": "c++/pod5_format_pybind/bindings.cpp", "content": "#include \"api.h\"\n#include \"pod5_format/c_api.h\"\n#include \"repack/repack_output.h\"\n#include \"repack/repacker.h\"\n#include \"subset.h\"\n\nPYBIND11_MODULE(pod5_format_pybind, m)\n{\n using namespace pod5;\n pod5_init();\n\n m.doc() = \"POD5 Format Raw Bindings\";\n\n py::enum_(m, \"SignalType\", py::arithmetic(), \"SignalType enum\")\n .value(\"UncompressedSignal\", SignalType::UncompressedSignal, \"Signal is not compressed\")\n .value(\"VbzSignal\", SignalType::VbzSignal, \"Signal is compressed using vbz\")\n .export_values();\n\n py::class_(m, \"FileWriterOptions\")\n .def(py::init<>())\n .def_property(\n \"max_signal_chunk_size\",\n &FileWriterOptions::max_signal_chunk_size,\n &FileWriterOptions::set_max_signal_chunk_size)\n .def_property(\n \"signal_table_batch_size\",\n &FileWriterOptions::signal_table_batch_size,\n &FileWriterOptions::set_signal_table_batch_size)\n .def_property(\n \"read_table_batch_size\",\n &FileWriterOptions::read_table_batch_size,\n &FileWriterOptions::set_read_table_batch_size)\n .def_property(\n \"signal_compression_type\",\n &FileWriterOptions::signal_type,\n &FileWriterOptions::set_signal_type);\n\n py::class_>(m, \"FileWriter\")\n .def(\"close\", [](pod5::FileWriter & w) { throw_on_error(w.close()); })\n .def(\n \"add_pore\",\n [](pod5::FileWriter & w, std::string pore_type) {\n return throw_on_error(w.add_pore_type(std::move(pore_type)));\n })\n .def(\n \"add_end_reason\",\n [](pod5::FileWriter & w, int name) {\n return throw_on_error(w.lookup_end_reason((pod5::ReadEndReason)name));\n })\n .def(\"add_run_info\", FileWriter_add_run_info)\n .def(\"add_reads\", FileWriter_add_reads)\n .def(\"add_reads_pre_compressed\", FileWriter_add_reads_pre_compressed);\n\n py::class_(m, \"EmbeddedFileData\")\n .def_readonly(\"file_path\", &pod5::FileLocation::file_path)\n .def_readonly(\"offset\", &pod5::FileLocation::offset)\n .def_readonly(\"length\", &pod5::FileLocation::size);\n\n py::class_>(\n m, \"Pod5AsyncSignalLoader\")\n .def(\"release_next_batch\", &Pod5AsyncSignalLoader::release_next_batch);\n\n py::class_>(\n m, \"Pod5SignalCacheBatch\")\n .def_property_readonly(\"batch_index\", &Pod5SignalCacheBatch::batch_index)\n .def_property_readonly(\"sample_count\", &Pod5SignalCacheBatch::sample_count)\n .def_property_readonly(\"samples\", &Pod5SignalCacheBatch::samples);\n\n py::class_(m, \"Pod5FileReader\")\n .def(\n \"get_file_run_info_table_location\",\n &Pod5FileReaderPtr::get_file_run_info_table_location)\n .def(\"get_file_read_table_location\", &Pod5FileReaderPtr::get_file_read_table_location)\n .def(\"get_file_signal_table_location\", &Pod5FileReaderPtr::get_file_signal_table_location)\n .def(\"get_file_version_pre_migration\", &Pod5FileReaderPtr::get_file_version_pre_migration)\n .def(\"plan_traversal\", &Pod5FileReaderPtr::plan_traversal)\n .def(\"batch_get_signal\", &Pod5FileReaderPtr::batch_get_signal)\n .def(\"batch_get_signal_selection\", &Pod5FileReaderPtr::batch_get_signal_selection)\n .def(\"batch_get_signal_batches\", &Pod5FileReaderPtr::batch_get_signal_batches)\n .def(\"close\", &Pod5FileReaderPtr::close);\n\n // Errors API\n m.def(\"get_error_string\", &pod5_get_error_string, \"Get the most recent error as a string\");\n\n // Creating files\n m.def(\n \"create_file\",\n &create_file,\n \"Create a POD5 file for writing\",\n py::arg(\"filename\"),\n py::arg(\"writer_name\"),\n py::arg(\"options\") = nullptr);\n\n // Opening files\n m.def(\"open_file\", &open_file, \"Open a POD5 file for reading\");\n\n // Recovering files\n py::class_(m, \"RecoverFileOptions\")\n .def(py::init<>())\n .def_readwrite(\"file_writer_options\", &RecoverFileOptions::file_writer_options)\n .def_readwrite(\"cleanup\", &RecoverFileOptions::cleanup);\n py::class_(m, \"RecoveredRowCounts\")\n .def(py::init<>())\n .def_readwrite(\"signal\", &RecoveredRowCounts::signal)\n .def_readwrite(\"run_info\", &RecoveredRowCounts::run_info)\n .def_readwrite(\"reads\", &RecoveredRowCounts::reads);\n py::class_(m, \"CleanupError\")\n .def(py::init<>())\n .def_readwrite(\"file_path\", &CleanupError::file_path)\n .def_readwrite(\"description\", &CleanupError::description);\n py::class_(m, \"RecoveryDetails\")\n .def(py::init<>())\n .def_readwrite(\"row_counts\", &RecoveryDetails::row_counts)\n .def_readwrite(\"cleanup_errors\", &RecoveryDetails::cleanup_errors);\n m.def(\n \"recover_file\",\n &::recover_file,\n \"Recover a POD5 file which was not closed correctly\",\n py::arg(\"src_filename\"),\n py::arg(\"dest_filename\"),\n py::arg(\"options\") = nullptr);\n\n m.def(\n \"update_file\",\n &write_updated_file_to_dest,\n \"Update a POD5 file to the latest writer format\");\n\n // Signal API\n m.def(\"decompress_signal\", &decompress_signal_wrapper, \"Decompress a numpy array of signal\");\n m.def(\"compress_signal\", &compress_signal_wrapper, \"Compress a numpy array of signal\");\n m.def(\"vbz_compressed_signal_max_size\", &vbz_compressed_signal_max_size);\n\n // Repacker API\n py::class_>(\n m, \"Pod5RepackerOutput\");\n\n py::class_>(m, \"Repacker\")\n .def(py::init<>())\n .def(\"add_output\", &repack::Pod5Repacker::add_output)\n .def(\"set_output_finished\", &repack::Pod5Repacker::set_output_finished)\n .def(\"add_all_reads_to_output\", &repack::Pod5Repacker::add_all_reads_to_output)\n .def(\"add_selected_reads_to_output\", &repack::Pod5Repacker::py_add_selected_reads_to_output)\n .def(\"finish\", &repack::Pod5Repacker::finish)\n .def_property_readonly(\"is_complete\", &repack::Pod5Repacker::is_complete)\n .def_property_readonly(\n \"currently_open_file_reader_count\",\n &repack::Pod5Repacker::currently_open_file_reader_count)\n .def_property_readonly(\"reads_completed\", &repack::Pod5Repacker::reads_completed);\n\n // Util API\n m.def(\n \"load_read_id_iterable\",\n &load_read_id_iterable,\n \"Load an iterable of read ids into a numpy array of data\");\n m.def(\"format_read_id_to_str\", &format_read_id_to_str, \"Format an array of read ids to string\");\n\n m.def(\n \"subset_pod5s_with_mapping\",\n &subset_pod5s_with_mapping,\n \"Subset pod5 files given a mapping\");\n}\n" }, { "path": "c++/pod5_format_pybind/build_wheel.cmake", "content": "\nmessage(\"Building python lib-pod5 wheel using ${PYTHON_EXECUTABLE}\")\nmessage(\" project dir ${PYTHON_PROJECT_DIR}\")\nmessage(\" with lib ${PYBIND_INPUT_LIB}\")\nmessage(\" with conan licenses ${POD5_CONAN_LICENSES}\")\nmessage(\" with c++ licenses ${POD5_CXX_LICENSES_SRC}\")\nmessage(\" into ${WHEEL_OUTPUT_DIR}\")\nmessage(\" using: ${PYTHON_EXECUTABLE} -m pip wheel . --wheel-dir ${WHEEL_OUTPUT_DIR}\")\n\n# Copy the prebuilt lib into the wheel src.\nfile(COPY \"${PYBIND_INPUT_LIB}\" DESTINATION \"${PYTHON_PROJECT_DIR}/src/lib_pod5\")\n\n# Copy the licenses into the wheel src.\n# Note: the trailing / on src is important since it tells cmake to copy only the contents.\nif(EXISTS \"${POD5_CONAN_LICENSES}\")\n file(INSTALL \"${POD5_CONAN_LICENSES}/\" DESTINATION \"${PYTHON_PROJECT_DIR}/licenses\")\nendif()\n\nforeach(license_src license_dst IN ZIP_LISTS POD5_CXX_LICENSES_SRC POD5_CXX_LICENSES_DST)\n file(COPY_FILE \"${license_src}\" \"${PYTHON_PROJECT_DIR}/licenses/${license_dst}\")\nendforeach()\n\nexecute_process(\n COMMAND ${PYTHON_EXECUTABLE} -m pip wheel . --wheel-dir ${WHEEL_OUTPUT_DIR}\n WORKING_DIRECTORY \"${PYTHON_PROJECT_DIR}/\"\n RESULT_VARIABLE exit_code\n OUTPUT_VARIABLE output\n ERROR_VARIABLE output\n)\n\nif (NOT exit_code EQUAL 0)\n message(FATAL_ERROR \"Could not generate wheel: ${output}\")\nendif()\n\nfile(GLOB pod5_wheel_names \"${WHEEL_OUTPUT_DIR}/*.whl\")\nforeach(wheel ${pod5_wheel_names})\n message(\"Built wheel ${wheel}\")\nendforeach()\n" }, { "path": "c++/pod5_format_pybind/repack/repack_functions.h", "content": "#pragma once\n\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/signal_builder.h\"\n#include \"pod5_format/signal_table_schema.h\"\n#include \"pod5_format/uuid.h\"\n#include \"repack_utils.h\"\n\n#include \n#include \n#include \n\n#include \n#include \n\nnamespace repack {\n\nstruct ReadReadData {\n std::shared_ptr input;\n std::vector reads;\n std::vector signal_durations;\n std::vector signal_row_sizes;\n\n std::vector signal_rows_read_ids;\n std::vector signal_rows;\n};\n\narrow::Result read_read_data(\n ReadsTableDictionaryThreadCache & reads_table_cache,\n states::unread_read_table_rows && in_batch)\n{\n POD5_TRACE_FUNCTION();\n\n auto const & source_file = in_batch.input;\n ARROW_ASSIGN_OR_RAISE(\n auto source_read_table_batch, source_file->read_read_record_batch(in_batch.batch_index));\n\n ARROW_ASSIGN_OR_RAISE(auto columns, source_read_table_batch.columns());\n\n auto const & pore_type_columns = columns.pore_type->indices();\n auto const & source_reads_pore_type_column =\n static_cast(*pore_type_columns);\n auto const & end_reason_columns = columns.end_reason->indices();\n auto const & source_reads_end_reason_column =\n static_cast(*end_reason_columns);\n auto const & run_info_columns = columns.run_info->indices();\n auto const & source_reads_run_info_column =\n static_cast(*run_info_columns);\n auto source_reads_signal_column = source_read_table_batch.signal_column();\n\n auto batch_rows = std::move(in_batch.batch_rows);\n if (batch_rows.empty()) {\n auto const source_batch_row_count = source_read_table_batch.num_rows();\n batch_rows.resize(source_batch_row_count);\n std::iota(batch_rows.begin(), batch_rows.end(), 0);\n }\n\n ReadReadData result;\n result.input = source_file;\n result.reads.reserve(batch_rows.size());\n result.signal_rows.reserve(batch_rows.size());\n result.signal_row_sizes.reserve(batch_rows.size());\n for (std::size_t batch_row_index = 0; batch_row_index < batch_rows.size(); ++batch_row_index) {\n auto batch_row = batch_rows[batch_row_index];\n // Find the read params\n auto const & read_id = columns.read_id->Value(batch_row);\n auto const & read_number = columns.read_number->Value(batch_row);\n auto const & start_sample = columns.start_sample->Value(batch_row);\n auto const & channel = columns.channel->Value(batch_row);\n auto const & well = columns.well->Value(batch_row);\n auto const & calibration_offset = columns.calibration_offset->Value(batch_row);\n auto const & calibration_scale = columns.calibration_scale->Value(batch_row);\n auto const & median_before = columns.median_before->Value(batch_row);\n auto const & end_reason_forced = columns.end_reason_forced->Value(batch_row);\n auto const & num_minknow_events = columns.num_minknow_events->Value(batch_row);\n auto const & tracked_scaling_scale = columns.tracked_scaling_scale->Value(batch_row);\n auto const & tracked_scaling_shift = columns.tracked_scaling_shift->Value(batch_row);\n auto const & predicted_scaling_scale = columns.predicted_scaling_scale->Value(batch_row);\n auto const & predicted_scaling_shift = columns.predicted_scaling_shift->Value(batch_row);\n auto const & num_reads_since_mux_change =\n columns.num_reads_since_mux_change->Value(batch_row);\n auto const & time_since_mux_change = columns.time_since_mux_change->Value(batch_row);\n auto const & open_pore_level = columns.open_pore_level->Value(batch_row);\n auto const & num_samples = columns.num_samples->Value(batch_row);\n\n auto const & pore_type_index = source_reads_pore_type_column.Value(batch_row);\n auto const & end_reason_index = source_reads_end_reason_column.Value(batch_row);\n auto const & run_info_index = source_reads_run_info_column.Value(batch_row);\n\n ARROW_ASSIGN_OR_RAISE(\n auto dest_pore_index,\n reads_table_cache.find_pore_index(\n source_file, source_read_table_batch, pore_type_index));\n ARROW_ASSIGN_OR_RAISE(\n auto dest_run_info_index,\n reads_table_cache.find_run_info_index(\n source_file, source_read_table_batch, run_info_index));\n\n result.reads.emplace_back(\n read_id,\n read_number,\n start_sample,\n channel,\n well,\n dest_pore_index,\n calibration_offset,\n calibration_scale,\n median_before,\n end_reason_index,\n end_reason_forced,\n dest_run_info_index,\n num_minknow_events,\n tracked_scaling_scale,\n tracked_scaling_shift,\n predicted_scaling_scale,\n predicted_scaling_shift,\n num_reads_since_mux_change,\n time_since_mux_change,\n open_pore_level);\n result.signal_durations.emplace_back(num_samples);\n\n auto const signal_rows = std::static_pointer_cast(\n source_reads_signal_column->value_slice(batch_row));\n auto const signal_rows_span =\n gsl::make_span(signal_rows->raw_values(), signal_rows->length());\n\n result.signal_rows.insert(\n result.signal_rows.end(), signal_rows_span.begin(), signal_rows_span.end());\n for (std::size_t i = 0; i < signal_rows_span.size(); ++i) {\n result.signal_rows_read_ids.emplace_back(read_id);\n }\n result.signal_row_sizes.emplace_back(signal_rows_span.size());\n }\n return result;\n}\n\narrow::Status read_signal(\n std::shared_ptr const & source_file,\n pod5::SignalType input_compression_type,\n std::uint64_t abs_signal_row,\n pod5::Uuid read_id,\n pod5::SignalType output_compression_type,\n arrow::FixedSizeBinaryBuilder & read_id_builder,\n pod5::SignalBuilderVariant & signal_builder,\n arrow::UInt32Builder & samples_builder,\n arrow::MemoryPool * pool)\n{\n auto signal_rows_span = gsl::make_span(&abs_signal_row, 1);\n\n // If were using the same compression type in both files, just copy compressed:\n if (input_compression_type == output_compression_type\n && output_compression_type == pod5::SignalType::VbzSignal)\n {\n std::vector sample_counts;\n ARROW_ASSIGN_OR_RAISE(\n auto extracted_signal,\n source_file->extract_samples_inplace(signal_rows_span, sample_counts));\n\n assert(1 == extracted_signal.size());\n assert(sample_counts.size() == extracted_signal.size());\n auto signal_span =\n gsl::make_span(extracted_signal.front()->data(), extracted_signal.front()->size());\n\n ARROW_RETURN_NOT_OK(read_id_builder.Append(read_id.data()));\n ARROW_RETURN_NOT_OK(\n std::visit(pod5::visitors::append_pre_compressed_signal{signal_span}, signal_builder));\n ARROW_RETURN_NOT_OK(samples_builder.Append(sample_counts.front()));\n } else {\n // Find the sample count of the complete read:\n ARROW_ASSIGN_OR_RAISE(\n auto sample_count, source_file->extract_sample_count(signal_rows_span));\n\n std::vector signal(sample_count);\n auto signal_buffer_span = gsl::make_span(signal);\n ARROW_RETURN_NOT_OK(source_file->extract_samples(signal_rows_span, signal_buffer_span));\n\n ARROW_RETURN_NOT_OK(read_id_builder.Append(read_id.data()));\n ARROW_RETURN_NOT_OK(\n std::visit(pod5::visitors::append_signal{signal_buffer_span, pool}, signal_builder));\n ARROW_RETURN_NOT_OK(samples_builder.Append(sample_count));\n }\n return arrow::Status::OK();\n}\n\nstruct RequestedSignalReads {\n std::vector complete_requests;\n std::shared_ptr partial_request;\n};\n\narrow::Result request_signal_reads(\n std::shared_ptr const & source_file,\n pod5::SignalType output_compression_type,\n std::size_t signal_batch_size,\n std::vector read_ids,\n std::vector signal_rows,\n std::shared_ptr const & partial_request,\n std::shared_ptr const & dest_read_table_rows,\n arrow::MemoryPool * pool)\n{\n POD5_TRACE_FUNCTION();\n\n auto const input_signal_type = source_file->signal_type();\n\n assert(read_ids.size() == signal_rows.size());\n\n RequestedSignalReads result;\n auto next_request = partial_request;\n\n assert(signal_rows.size() == dest_read_table_rows->signal_row_indices.size());\n\n std::size_t signal_rows_position = 0;\n while (signal_rows_position < signal_rows.size()) {\n if (!next_request) {\n ARROW_ASSIGN_OR_RAISE(\n auto signal_builder, pod5::make_signal_builder(output_compression_type, pool));\n next_request = std::make_shared(\n std::move(signal_builder), pool);\n next_request->patch_rows.reserve(signal_batch_size);\n }\n auto to_write = std::min(\n signal_rows.size() - signal_rows_position,\n signal_batch_size - next_request->patch_rows.size());\n\n for (std::size_t i = 0; i < to_write; ++i) {\n auto const dest_batch_row_index = signal_rows_position + i;\n assert(dest_batch_row_index < signal_rows.size());\n assert(dest_batch_row_index < dest_read_table_rows->signal_row_indices.size());\n\n ARROW_RETURN_NOT_OK(read_signal(\n source_file,\n input_signal_type,\n signal_rows[signal_rows_position + i],\n read_ids[signal_rows_position + i],\n output_compression_type,\n *next_request->read_id_builder,\n next_request->signal_builder,\n next_request->samples_builder,\n pool));\n\n next_request->patch_rows.emplace_back(dest_read_table_rows, dest_batch_row_index);\n }\n signal_rows_position += to_write;\n\n assert(next_request->row_count() <= signal_batch_size);\n assert(next_request->row_count() <= signal_batch_size);\n if (next_request->row_count() >= signal_batch_size) {\n result.complete_requests.emplace_back(std::move(next_request));\n next_request.reset();\n }\n }\n\n result.partial_request = next_request;\n return result;\n}\n\nstruct ReadSignal {\n std::size_t row_count;\n bool final_batch;\n std::vector> columns;\n};\n\narrow::Result read_signal_data(states::read_split_signal_table_batch_rows & signal_rows)\n{\n POD5_TRACE_FUNCTION();\n\n ReadSignal result;\n\n pod5::SignalTableSchemaDescription field_locations;\n result.final_batch = signal_rows.final_batch;\n result.row_count = signal_rows.row_count();\n result.columns = {nullptr, nullptr, nullptr};\n ARROW_RETURN_NOT_OK(\n signal_rows.read_id_builder->Finish(&result.columns[field_locations.read_id]));\n ARROW_RETURN_NOT_OK(\n std::visit(\n pod5::visitors::finish_column{&result.columns[field_locations.signal]},\n signal_rows.signal_builder));\n ARROW_RETURN_NOT_OK(\n signal_rows.samples_builder.Finish(&result.columns[field_locations.samples]));\n return result;\n}\n\narrow::Status write_reads(\n std::shared_ptr const & output,\n std::vector const & reads,\n std::vector const & signal_durations,\n std::vector const & signal_row_sizes,\n std::vector const & signal_row_indices)\n{\n POD5_TRACE_FUNCTION();\n std::size_t signal_position = 0;\n auto signal_indices_span = gsl::make_span(signal_row_indices);\n for (std::size_t i = 0; i < reads.size(); ++i) {\n auto signal_rows = signal_indices_span.subspan(signal_position, signal_row_sizes[i]);\n signal_position += signal_row_sizes[i];\n\n ARROW_RETURN_NOT_OK(output->add_complete_read(reads[i], signal_rows, signal_durations[i]));\n }\n\n return arrow::Status::OK();\n}\n\narrow::Status check_duplicate_read_ids(\n std::unordered_set & output_read_ids,\n std::vector const & new_reads)\n{\n for (auto const & read : new_reads) {\n auto result = output_read_ids.insert(read.read_id);\n if (!result.second) {\n return arrow::Status::Invalid(\n \"Duplicate read id \", to_string(read.read_id), \" found in file\");\n }\n }\n\n return arrow::Status::OK();\n}\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/repack/repack_output.cpp", "content": "#include \"repack_output.h\"\n\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"repack_functions.h\"\n\n#include \n#include \n#include \n\nnamespace repack {\n\nnamespace {\nstruct is_not_nullptr {\n template \n bool operator()(T const & t) const\n {\n return t != nullptr;\n }\n};\n\n#if 0\nstruct get_name{\n template \n std::string operator()(T const& t) const {\n return typeid(typename T::element_type).name();\n }\n};\n\ntemplate \nvoid dump_queued_items(T const& queued) {\n std::map items;\n\n for (auto const& item : queued) {\n items[std::visit(get_name{}, item)] += 1;\n }\n\n std::cout << \"Queued items:\\n\";\n for (auto pr : items) {\n std::cout << \" \" << pr.first << \": \" << pr.second << \"\\n\";\n }\n}\n#endif\n\n} // namespace\n\nstruct Pod5RepackerOutputThreadState {\n Pod5RepackerOutputThreadState(std::shared_ptr const & dict_manager)\n : dict_cache(dict_manager)\n {\n }\n\n ReadsTableDictionaryThreadCache dict_cache;\n};\n\nstruct Pod5RepackerOutputState {\n Pod5RepackerOutputState(\n std::shared_ptr const & _output_file,\n bool _check_duplicate_read_ids,\n arrow::MemoryPool * _memory_pool)\n : output_file(_output_file)\n , check_duplicate_read_ids(_check_duplicate_read_ids)\n , memory_pool(_memory_pool)\n , dict_manager(\n std::make_shared(_output_file, read_table_writer_mutex))\n {\n }\n\n Pod5RepackerOutputThreadState * get_thread_state()\n {\n std::lock_guard l{thread_states_mutex};\n auto it = thread_states.find(std::this_thread::get_id());\n if (it == thread_states.end()) {\n it = thread_states.emplace(std::this_thread::get_id(), dict_manager).first;\n }\n return &it->second;\n }\n\n std::shared_ptr output_file;\n bool check_duplicate_read_ids;\n arrow::MemoryPool * memory_pool;\n std::mutex read_table_writer_mutex;\n std::mutex signal_table_writer_mutex;\n std::shared_ptr dict_manager;\n std::mutex partial_signal_batch_mutex;\n std::shared_ptr partial_signal_batch;\n std::atomic reads_completed{0};\n\n std::mutex thread_states_mutex;\n std::unordered_map thread_states;\n\n std::mutex output_read_ids_mutex;\n std::unordered_set output_read_ids;\n};\n\nnamespace {\n\nstruct StateProgressResult {\n StateProgressResult() = default;\n\n StateProgressResult(std::vector && _new_states)\n : new_states(_new_states)\n {\n }\n\n std::vector new_states;\n};\n\nstruct StateOperator {\n StateOperator(Pod5RepackerOutputState * _progress_state) : progress_state(_progress_state) {}\n\n arrow::Result operator()(\n std::shared_ptr & batch) const\n {\n POD5_TRACE_FUNCTION();\n\n // Read out the read table data from the source file\n ARROW_ASSIGN_OR_RAISE(\n auto read_result,\n read_read_data(progress_state->get_thread_state()->dict_cache, std::move(*batch)));\n batch.reset();\n\n auto read_table_rows = std::make_shared();\n read_table_rows->reads = std::move(read_result.reads);\n read_table_rows->signal_durations = std::move(read_result.signal_durations);\n read_table_rows->signal_row_sizes = std::move(read_result.signal_row_sizes);\n read_table_rows->signal_row_indices.resize(read_result.signal_rows.size());\n\n if (progress_state->check_duplicate_read_ids) {\n std::lock_guard l{progress_state->output_read_ids_mutex};\n ARROW_RETURN_NOT_OK(\n check_duplicate_read_ids(progress_state->output_read_ids, read_table_rows->reads));\n }\n\n // Split the read table rows into new signal table batches:\n {\n std::lock_guard l{progress_state->partial_signal_batch_mutex};\n ARROW_ASSIGN_OR_RAISE(\n auto signal_request_result,\n request_signal_reads(\n read_result.input,\n progress_state->output_file->signal_type(),\n progress_state->output_file->signal_table_batch_size(),\n read_result.signal_rows_read_ids,\n read_result.signal_rows,\n progress_state->partial_signal_batch,\n read_table_rows,\n progress_state->memory_pool));\n\n progress_state->partial_signal_batch = signal_request_result.partial_request;\n return StateProgressResult{std::move(signal_request_result.complete_requests)};\n }\n }\n\n arrow::Result operator()(\n std::shared_ptr & batch) const\n {\n POD5_TRACE_FUNCTION();\n\n ARROW_ASSIGN_OR_RAISE(auto read_signal_result, read_signal_data(*batch));\n\n std::pair inserted_signal_rows;\n {\n std::lock_guard l(progress_state->signal_table_writer_mutex);\n ARROW_ASSIGN_OR_RAISE(\n inserted_signal_rows,\n progress_state->output_file->add_signal_batch(\n read_signal_result.row_count,\n std::move(read_signal_result.columns),\n read_signal_result.final_batch));\n }\n\n std::vector result_new_states;\n\n for (std::size_t i = 0; i < batch->patch_rows.size(); ++i) {\n auto const & row = batch->patch_rows[i];\n\n auto const & dest_read_table = row.dest_read_table;\n assert(dest_read_table);\n assert(row.dest_batch_row_index < dest_read_table->signal_row_indices.size());\n dest_read_table->signal_row_indices[row.dest_batch_row_index] =\n inserted_signal_rows.first + i;\n dest_read_table->written_row_indices += 1;\n\n // Check if this read table is completed!\n if (dest_read_table->written_row_indices > dest_read_table->signal_row_indices.size()) {\n assert(false);\n }\n if (dest_read_table->written_row_indices == dest_read_table->signal_row_indices.size())\n {\n result_new_states.push_back(dest_read_table);\n }\n }\n\n return StateProgressResult{std::move(result_new_states)};\n }\n\n arrow::Result operator()(\n std::shared_ptr & batch) const\n {\n POD5_TRACE_FUNCTION();\n assert(batch->written_row_indices == batch->signal_row_indices.size());\n\n std::lock_guard l(progress_state->read_table_writer_mutex);\n ARROW_RETURN_NOT_OK(write_reads(\n progress_state->output_file,\n batch->reads,\n batch->signal_durations,\n batch->signal_row_sizes,\n batch->signal_row_indices));\n progress_state->reads_completed += batch->reads.size();\n\n return StateProgressResult{{}};\n }\n\n arrow::Result operator()(std::shared_ptr & batch) const\n {\n POD5_TRACE_FUNCTION();\n\n std::vector final_states;\n // No further reads expected, flush all partial state:\n std::lock_guard l{progress_state->partial_signal_batch_mutex};\n if (progress_state->partial_signal_batch) {\n progress_state->partial_signal_batch->final_batch = true;\n\n final_states.emplace_back(std::move(progress_state->partial_signal_batch));\n progress_state->partial_signal_batch.reset();\n }\n\n return StateProgressResult{std::move(final_states)};\n }\n\n Pod5RepackerOutputState * progress_state;\n};\n\n} // namespace\n\nPod5RepackerOutput::Pod5RepackerOutput(\n std::shared_ptr const & repacker,\n std::shared_ptr thread_pool,\n std::shared_ptr const & output,\n bool check_duplicate_read_ids)\n: m_repacker(repacker)\n, m_thread_pool(thread_pool)\n, m_output(output)\n, m_progress_state(\n std::make_unique(\n output,\n check_duplicate_read_ids,\n arrow::default_memory_pool()))\n{\n}\n\nPod5RepackerOutput::~Pod5RepackerOutput() {}\n\nbool Pod5RepackerOutput::has_tasks() const\n{\n if (m_in_flight > 0) {\n return true;\n }\n std::lock_guard l{m_active_read_table_states_mutex};\n return m_active_read_table_states.size() > 0;\n}\n\nvoid Pod5RepackerOutput::set_finished()\n{\n if (!m_finished) {\n // Wait for all other tasks to flush through the output.\n while (!m_has_error) {\n if (!has_tasks()) {\n break;\n }\n\n std::this_thread::sleep_for(std::chrono::milliseconds(1));\n }\n\n {\n std::lock_guard l{m_active_read_table_states_mutex};\n m_active_read_table_states.emplace_front(std::make_shared());\n }\n post_try_work();\n\n m_finished = true;\n }\n}\n\nbool Pod5RepackerOutput::is_complete() const\n{\n if (!m_finished) {\n return false;\n }\n std::lock_guard l{m_active_read_table_states_mutex};\n return m_active_read_table_states.empty();\n}\n\nstd::size_t Pod5RepackerOutput::reads_completed() const\n{\n return m_progress_state->reads_completed;\n}\n\nvoid Pod5RepackerOutput::register_new_reads(\n std::shared_ptr const & input,\n std::size_t batch_index,\n std::vector && batch_rows)\n{\n if (m_finished) {\n throw std::runtime_error(\"Failed to add reads to finished output\");\n }\n\n {\n std::lock_guard l{m_active_read_table_states_mutex};\n m_active_read_table_states.emplace_front(\n std::make_shared(\n input, batch_index, std::move(batch_rows)));\n }\n\n post_try_work();\n}\n\nvoid Pod5RepackerOutput::post_try_work()\n{\n m_thread_pool->post([&]() {\n POD5_TRACE_FUNCTION();\n\n auto get_next_work = [](auto & locked_states) -> states::shared_variant {\n if (locked_states.empty()) {\n return {};\n }\n\n auto work = locked_states.back();\n locked_states.pop_back();\n return work;\n };\n\n StateOperator state_operator{m_progress_state.get()};\n\n states::shared_variant next_work;\n while (!m_has_error) {\n m_in_flight += 1;\n // Its important we don't release this until any new states\n // are in `m_active_read_table_states`\n auto remove_in_flight = gsl::finally([&] { m_in_flight -= 1; });\n\n if (!std::visit(is_not_nullptr{}, next_work)) {\n std::lock_guard l{m_active_read_table_states_mutex};\n next_work = get_next_work(m_active_read_table_states);\n if (!std::visit(is_not_nullptr{}, next_work)) {\n return;\n }\n }\n\n auto result = std::visit(state_operator, next_work);\n if (!result.ok()) {\n set_error(result.status());\n return;\n }\n next_work = {};\n\n {\n std::lock_guard l{m_active_read_table_states_mutex};\n auto && states = m_active_read_table_states;\n states.insert(states.end(), result->new_states.begin(), result->new_states.end());\n\n next_work = get_next_work(states);\n }\n }\n });\n}\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/repack/repack_output.h", "content": "#pragma once\n\n#include \"pod5_format/file_writer.h\"\n#include \"pod5_format/thread_pool.h\"\n#include \"repack_states.h\"\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace repack {\n\nclass Pod5Repacker;\n\nstruct Pod5RepackerOutputState;\n\nclass Pod5RepackerOutput {\npublic:\n Pod5RepackerOutput(\n std::shared_ptr const & repacker,\n std::shared_ptr thread_pool,\n std::shared_ptr const & output,\n bool check_duplicate_read_ids);\n ~Pod5RepackerOutput();\n\n std::string path() const { return m_output->path(); }\n\n std::shared_ptr const & repacker() const { return m_repacker; }\n\n bool has_tasks() const;\n\n arrow::Status error()\n {\n std::lock_guard l{m_error_mutex};\n return m_error;\n }\n\n bool has_error() const { return m_has_error.load(); }\n\n // Inform the output no further reads will be added\n void set_finished();\n\n // Check if the output has completed all writes\n bool is_complete() const;\n\n // Number of reads completed\n std::size_t reads_completed() const;\n\n // Register new writes to the output, should not be called after #set_reads_finished\n void register_new_reads(\n std::shared_ptr const & input,\n std::size_t batch_index,\n std::vector && batch_rows = {} // All rows by default\n );\n\nprivate:\n void post_try_work();\n\n void set_error(arrow::Status error)\n {\n assert(!error.ok());\n {\n std::lock_guard l{m_error_mutex};\n m_error = std::move(error);\n }\n m_has_error = true;\n }\n\n std::shared_ptr m_repacker;\n std::shared_ptr m_thread_pool;\n std::shared_ptr m_output;\n std::atomic m_finished{false};\n\n std::atomic m_has_error{false};\n std::mutex m_error_mutex;\n arrow::Status m_error;\n\n std::atomic m_in_flight{0};\n mutable std::mutex m_active_read_table_states_mutex;\n std::deque m_active_read_table_states;\n\n std::unique_ptr m_progress_state;\n};\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/repack/repack_states.h", "content": "#pragma once\n\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/signal_builder.h\"\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n\nnamespace repack { namespace states {\n\nclass unread_read_table_rows {\npublic:\n unread_read_table_rows(\n std::shared_ptr const & _input,\n std::size_t _batch_index,\n std::vector && _batch_rows)\n : input(_input)\n , batch_index(_batch_index)\n , batch_rows(std::move(_batch_rows))\n {\n }\n\n std::shared_ptr input;\n std::size_t batch_index;\n std::vector batch_rows;\n};\n\nclass read_read_table_rows_no_signal {\npublic:\n std::vector reads;\n std::vector signal_durations;\n std::vector signal_row_sizes;\n\n std::atomic written_row_indices{0};\n std::vector signal_row_indices;\n};\n\nclass read_split_signal_table_batch_rows {\npublic:\n struct PatchRecord {\n PatchRecord(\n std::shared_ptr dest_read_table,\n std::uint64_t dest_batch_row_index)\n : dest_read_table(dest_read_table)\n , dest_batch_row_index(dest_batch_row_index)\n {\n }\n\n std::shared_ptr dest_read_table;\n std::uint64_t dest_batch_row_index;\n };\n\n read_split_signal_table_batch_rows(\n pod5::SignalBuilderVariant && signal_builder,\n arrow::MemoryPool * pool)\n : read_id_builder(pod5::make_read_id_builder(pool))\n , signal_builder(std::move(signal_builder))\n , samples_builder(pool)\n {\n }\n\n std::unique_ptr read_id_builder;\n pod5::SignalBuilderVariant signal_builder;\n arrow::UInt32Builder samples_builder;\n\n std::vector patch_rows;\n bool final_batch = false;\n\n std::size_t row_count() const { return patch_rows.size(); }\n};\n\nstruct finished {};\n\nusing shared_variant = std::variant<\n std::shared_ptr,\n std::shared_ptr,\n std::shared_ptr,\n std::shared_ptr>;\n\n}} // namespace repack::states\n" }, { "path": "c++/pod5_format_pybind/repack/repack_utils.h", "content": "#pragma once\n\n#include \"pod5_format/read_table_reader.h\"\n\n#include \n\n#include \n#include \n\nnamespace repack {\n\nstruct pair_hasher {\n template \n std::size_t operator()(std::pair const & pair) const\n {\n return std::hash{}(pair.first) ^ std::hash{}(pair.second);\n }\n};\n\nstruct run_info_hasher {\n std::size_t operator()(pod5::RunInfoData const & run_info) const\n {\n return std::hash{}(run_info.acquisition_id);\n }\n};\n\nclass ReadsTableDictionaryManager {\npublic:\n ReadsTableDictionaryManager(\n std::shared_ptr const & output_file,\n std::mutex & writer_mutex)\n : m_output_file(output_file)\n , m_writer_mutex(writer_mutex)\n {\n }\n\n // Find or create a pore index in the output file - expects to run on strand.\n arrow::Result find_pore_index(\n std::shared_ptr const & source_file,\n pod5::ReadTableRecordBatch const & source_batch,\n pod5::PoreDictionaryIndex source_index)\n {\n std::lock_guard l(m_writer_mutex);\n\n ARROW_ASSIGN_OR_RAISE(auto source_data, source_batch.get_pore_type(source_index));\n pod5::PoreDictionaryIndex dest_index = 0;\n\n // See if we have the same run info by value stored in the file:\n auto data_lookup_it = m_pore_data_indexes.find(source_data);\n if (data_lookup_it != m_pore_data_indexes.end()) {\n dest_index = data_lookup_it->second;\n } else {\n ARROW_ASSIGN_OR_RAISE(dest_index, m_output_file->add_pore_type(source_data));\n }\n\n m_pore_data_indexes[source_data] = dest_index;\n return dest_index;\n }\n\n // Find or create a run_info index in the output file - expects to run on strand.\n arrow::Result find_run_info_index(\n std::shared_ptr const & source_file,\n pod5::ReadTableRecordBatch const & source_batch,\n pod5::RunInfoDictionaryIndex source_index)\n {\n std::lock_guard l(m_writer_mutex);\n\n ARROW_ASSIGN_OR_RAISE(auto source_data, source_batch.get_run_info(source_index));\n ARROW_ASSIGN_OR_RAISE(auto const run_info, source_file->find_run_info(source_data));\n pod5::RunInfoDictionaryIndex dest_index = 0;\n\n // See if we have the same run info by value stored in the file:\n auto data_lookup_it = m_run_info_data_indexes.find(*run_info);\n if (data_lookup_it != m_run_info_data_indexes.end()) {\n dest_index = data_lookup_it->second;\n } else {\n ARROW_ASSIGN_OR_RAISE(dest_index, m_output_file->add_run_info(*run_info));\n }\n\n m_run_info_data_indexes[*run_info] = dest_index;\n return dest_index;\n }\n\nprivate:\n std::shared_ptr m_output_file;\n\n std::mutex & m_writer_mutex;\n\n std::unordered_map m_pore_data_indexes;\n std::unordered_map\n m_run_info_data_indexes;\n};\n\nclass ReadsTableDictionaryThreadCache {\npublic:\n ReadsTableDictionaryThreadCache(std::shared_ptr const & main_cache)\n : m_main_cache(main_cache)\n {\n }\n\n // Find or create a pore index in the output file - expects to run on strand.\n arrow::Result find_pore_index(\n std::shared_ptr const & source_file,\n pod5::ReadTableRecordBatch const & source_batch,\n pod5::PoreDictionaryIndex source_index)\n {\n auto const key = std::make_pair(make_file_key(source_file), source_index);\n auto const it = m_pore_indexes.find(key);\n if (it != m_pore_indexes.end()) {\n return it->second;\n }\n\n ARROW_ASSIGN_OR_RAISE(\n auto dest_index,\n m_main_cache->find_pore_index(source_file, source_batch, source_index));\n m_pore_indexes[key] = dest_index;\n return dest_index;\n }\n\n // Find or create a run_info index in the output file - expects to run on strand.\n arrow::Result find_run_info_index(\n std::shared_ptr const & source_file,\n pod5::ReadTableRecordBatch const & source_batch,\n pod5::RunInfoDictionaryIndex source_index)\n {\n auto const key = std::make_pair(make_file_key(source_file), source_index);\n auto const it = m_run_info_indexes.find(key);\n if (it != m_run_info_indexes.end()) {\n return it->second;\n }\n\n ARROW_ASSIGN_OR_RAISE(\n auto dest_index,\n m_main_cache->find_run_info_index(source_file, source_batch, source_index));\n m_run_info_indexes[key] = dest_index;\n return dest_index;\n }\n\nprivate:\n using FileKey = std::uint64_t;\n\n FileKey make_file_key(std::shared_ptr const & file)\n {\n return reinterpret_cast(file.get());\n }\n\n template \n using DictionaryLookup =\n std::unordered_map, IndexType, pair_hasher>;\n\n std::shared_ptr m_main_cache;\n\n DictionaryLookup m_pore_indexes;\n DictionaryLookup m_run_info_indexes;\n};\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/repack/repacker.cpp", "content": "#include \"repacker.h\"\n\n#include \"pod5_format/internal/tracing/tracing.h\"\n#include \"repack_output.h\"\n#include \"repack_states.h\"\n\nnamespace repack {\n\nnamespace {\n\nvoid repacker_add_reads_preconditions(\n std::shared_ptr const & repacker,\n std::shared_ptr const & output,\n Pod5FileReaderPtr const & input)\n{\n if (output->repacker() != repacker) {\n throw std::runtime_error(\"Invalid repacker output passed, created by another repacker\");\n }\n\n if (!input.reader) {\n throw std::runtime_error(\"Invalid input passed to repacker, no reader\");\n }\n}\n\n} // namespace\n\nPod5Repacker::Pod5Repacker() : m_thread_pool{pod5::make_thread_pool(10)} {}\n\nPod5Repacker::~Pod5Repacker() { finish(); }\n\nvoid Pod5Repacker::finish()\n{\n POD5_TRACE_FUNCTION();\n for (auto & output : m_outputs) {\n output->set_finished();\n }\n\n check_for_error();\n\n m_thread_pool->stop_and_drain();\n\n for (auto & output : m_outputs) {\n m_reads_complete_deleted_outputs += output->reads_completed();\n }\n m_outputs.clear();\n}\n\nstd::shared_ptr Pod5Repacker::add_output(\n std::shared_ptr const & output,\n bool check_duplicate_read_ids)\n{\n POD5_TRACE_FUNCTION();\n auto repacker_output = std::make_shared(\n shared_from_this(), m_thread_pool, output, check_duplicate_read_ids);\n m_outputs.push_back(repacker_output);\n return repacker_output;\n}\n\nvoid Pod5Repacker::set_output_finished(std::shared_ptr const & output)\n{\n if (output->repacker() != shared_from_this()) {\n throw std::runtime_error(\"Invalid repacker output passed, created by another repacker\");\n }\n\n output->set_finished();\n}\n\nvoid Pod5Repacker::add_all_reads_to_output(\n std::shared_ptr const & output,\n Pod5FileReaderPtr const & input)\n{\n POD5_TRACE_FUNCTION();\n repacker_add_reads_preconditions(shared_from_this(), output, input);\n\n for (std::size_t i = 0; i < input.reader->num_read_record_batches(); ++i) {\n output->register_new_reads(input.reader, i);\n }\n\n register_submitted_reader(input.reader);\n}\n\nvoid Pod5Repacker::py_add_selected_reads_to_output(\n std::shared_ptr const & output,\n Pod5FileReaderPtr const & input,\n py::array_t && batch_counts,\n py::array_t && all_batch_rows)\n{\n repacker_add_reads_preconditions(shared_from_this(), output, input);\n\n auto batch_counts_span = gsl::make_span(batch_counts.data(), batch_counts.size());\n auto all_batch_rows_span = gsl::make_span(all_batch_rows.data(), all_batch_rows.size());\n\n add_selected_reads_to_output(output, input.reader, batch_counts_span, all_batch_rows_span);\n}\n\nvoid Pod5Repacker::add_selected_reads_to_output(\n std::shared_ptr const & output,\n std::shared_ptr const & input,\n gsl::span batch_counts_span,\n gsl::span all_batch_rows_span)\n{\n POD5_TRACE_FUNCTION();\n\n std::size_t current_start_point = 0;\n for (std::size_t i = 0; i < input->num_read_record_batches(); ++i) {\n std::vector batch_rows;\n auto const batch_rows_span =\n all_batch_rows_span.subspan(current_start_point, batch_counts_span[i]);\n\n // If this batch has no selected\n if (batch_rows_span.empty()) {\n continue;\n }\n\n batch_rows.insert(batch_rows.end(), batch_rows_span.begin(), batch_rows_span.end());\n current_start_point += batch_counts_span[i];\n\n output->register_new_reads(input, i, std::move(batch_rows));\n }\n\n register_submitted_reader(input);\n}\n\nvoid Pod5Repacker::check_for_error() const\n{\n for (auto const & output : m_outputs) {\n if (output->has_error()) {\n throw std::runtime_error(output->error().ToString());\n }\n }\n}\n\nbool Pod5Repacker::is_complete() const\n{\n POD5_TRACE_FUNCTION();\n check_for_error();\n\n for (auto const & output : m_outputs) {\n if (!output->is_complete()) {\n return false;\n }\n }\n\n return true;\n}\n\nstd::size_t Pod5Repacker::reads_completed() const\n{\n POD5_TRACE_FUNCTION();\n check_for_error();\n\n std::size_t reads_complete = 0;\n for (auto const & output : m_outputs) {\n reads_complete += output->reads_completed();\n }\n\n return reads_complete + m_reads_complete_deleted_outputs;\n}\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/repack/repacker.h", "content": "#pragma once\n\n#include \"pod5_format_pybind/api.h\"\n\n#include \n\n#include \n#include \n#include \n\nnamespace repack {\n\nclass Pod5RepackerOutput;\n\nclass Pod5Repacker : public std::enable_shared_from_this {\npublic:\n Pod5Repacker();\n ~Pod5Repacker();\n\n void finish();\n\n std::shared_ptr add_output(\n std::shared_ptr const & output,\n bool check_duplicate_read_ids);\n void set_output_finished(std::shared_ptr const & output);\n\n void add_all_reads_to_output(\n std::shared_ptr const & output,\n Pod5FileReaderPtr const & input);\n\n void py_add_selected_reads_to_output(\n std::shared_ptr const & output,\n Pod5FileReaderPtr const & input,\n py::array_t && batch_counts,\n py::array_t && all_batch_rows);\n\n void add_selected_reads_to_output(\n std::shared_ptr const & output,\n std::shared_ptr const & input,\n gsl::span batch_counts,\n gsl::span all_batch_rows);\n\n bool is_complete() const;\n std::size_t reads_completed() const;\n\n std::size_t currently_open_file_reader_count()\n {\n check_for_error();\n cleanup_submitted_readers();\n return m_file_readers.size();\n }\n\nprivate:\n void check_for_error() const;\n\n void cleanup_submitted_readers()\n {\n std::erase_if(m_file_readers, [](auto const & ptr) { return ptr.expired(); });\n }\n\n void register_submitted_reader(std::shared_ptr const & input)\n {\n cleanup_submitted_readers();\n m_file_readers.insert(input);\n }\n\n std::shared_ptr m_thread_pool;\n std::set, std::owner_less<>> m_file_readers;\n std::vector> m_outputs;\n\n std::size_t m_reads_complete_deleted_outputs{0};\n};\n\n} // namespace repack\n" }, { "path": "c++/pod5_format_pybind/subset.cpp", "content": "#include \"subset.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#ifndef _WIN32\n#include \n#include \n#else\n#include \n\n#include // _getmaxstdio\n#endif\n\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/file_writer.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"repack/repacker.h\"\n\n#include \n\nnamespace io_limits {\n\n// Balance the number of open inputs by the output-side handle usage.\n// Prefer outputs over inputs to reduce the number of output\n// batches which iterate over all inputs.\nconstexpr std::float_t kOutputsBias = 0.7f;\nconstexpr std::size_t kMinHandles = 1;\nconstexpr std::size_t kBaseReserve = 16;\n\nstd::size_t clamp_open_inputs(std::size_t soft_limit, std::size_t output_files)\n{\n constexpr std::size_t kMaxInHandles = 256;\n std::size_t const reserve = kBaseReserve + output_files;\n if (soft_limit <= reserve + kMinHandles) {\n return kMinHandles;\n }\n return std::clamp(soft_limit - reserve, kMinHandles, kMaxInHandles);\n}\n\nstd::size_t clamp_open_outputs(std::size_t soft_limit)\n{\n constexpr std::size_t kMaxOutHandles = 4096;\n std::size_t const reserve = kBaseReserve + kMinHandles;\n if (soft_limit <= reserve + kMinHandles) {\n return kMinHandles;\n }\n std::size_t soft_upper = (std::size_t)(soft_limit * kOutputsBias);\n if (soft_upper > 32) {\n soft_upper = (soft_upper / 16) * 16;\n }\n\n return std::clamp(std::min(soft_limit - reserve, soft_upper), kMinHandles, kMaxOutHandles);\n}\n\nstd::size_t detect_soft_limit()\n{\n //\n constexpr std::size_t kSoftLimitFallback = 1024;\n\n#ifndef _WIN32\n // Attempt to get the resource limits (if any)\n struct rlimit rl{};\n if (getrlimit(RLIMIT_NOFILE, &rl) == 0 && rl.rlim_cur != RLIM_INFINITY) {\n return static_cast(rl.rlim_cur);\n }\n long sc = sysconf(_SC_OPEN_MAX);\n return sc > 0 ? static_cast(sc) : kSoftLimitFallback;\n#else\n // Only stdio stream limit, not a true OS handle limit.\n int n = _getmaxstdio();\n return n > 0 ? static_cast(n) : kSoftLimitFallback;\n#endif\n}\n} // namespace io_limits\n\n/// \\brief Simple progress bar for console applications\nclass ProgressBar {\npublic:\n static constexpr int PB_WIDTH = 60;\n\n ProgressBar() {}\n\n ~ProgressBar() { std::fputs(\"\\n\", stdout); }\n\n void set_task(std::string const & task_name)\n {\n m_task = task_name;\n print_progress();\n }\n\n void update_max_steps(std::size_t max_steps) { this->m_max_steps = max_steps; }\n\n void update(std::size_t current_step)\n {\n if (current_step == m_current_step) {\n return;\n }\n m_current_step = current_step;\n print_progress();\n }\n\n void print_progress()\n {\n float complete_ratio = static_cast(m_current_step) / static_cast(m_max_steps);\n int complete_length = static_cast(complete_ratio * PB_WIDTH);\n std::string complete_string{\"\\r[\"};\n for (int i = 0; i < PB_WIDTH; ++i) {\n if (i < complete_length) {\n complete_string += \"=\";\n } else {\n complete_string += \" \";\n }\n }\n complete_string += \"] (\" + std::to_string(m_current_step) + \"/\"\n + std::to_string(m_max_steps) + \") \" + m_task;\n m_max_printed_width = std::max(m_max_printed_width, complete_string.size());\n // Pad to max width to overwrite previous longer lines\n complete_string.resize(m_max_printed_width, ' ');\n std::cout << complete_string.c_str() << std::flush;\n }\n\nprivate:\n std::string m_task;\n std::size_t m_max_steps{0};\n std::size_t m_current_step{0};\n std::size_t m_max_printed_width{0};\n};\n\nvoid subset_pod5s_with_mapping(\n std::vector inputs,\n std::filesystem::path output,\n std::map> read_id_to_dest,\n bool missing_ok,\n bool duplicate_ok,\n bool force_overwrite)\n{\n auto next_interrupt_check = std::chrono::steady_clock::now();\n auto poll_python_interrupt = [&]() {\n auto const now = std::chrono::steady_clock::now();\n if (now < next_interrupt_check) {\n return;\n }\n next_interrupt_check = now + std::chrono::milliseconds(500);\n\n pybind11::gil_scoped_acquire gil;\n if (PyErr_CheckSignals() != 0) {\n throw pybind11::error_already_set();\n }\n };\n\n struct OutputInfo {\n OutputInfo(std::shared_ptr && repacker_output_)\n : repacker_output(std::move(repacker_output_))\n {\n }\n\n std::shared_ptr repacker_output;\n\n void clear_per_input_working_data()\n {\n batch_counts.clear();\n all_batch_rows.clear();\n batch_counts.reserve(32);\n all_batch_rows.reserve(128);\n }\n\n void add_row(std::uint32_t row_index)\n {\n all_batch_rows.push_back(row_index);\n next_batch_size += 1;\n }\n\n void finish_batch()\n {\n batch_counts.push_back(next_batch_size);\n next_batch_size = 0;\n }\n\n // Per file working vectors:\n std::uint32_t next_batch_size = 0;\n std::vector batch_counts;\n std::vector all_batch_rows;\n };\n\n pod5::FileWriterOptions output_options{};\n output_options.set_keep_signal_file_open(false);\n output_options.set_keep_read_table_file_open(false);\n output_options.set_keep_run_info_file_open(false);\n pod5::FileReaderOptions input_options{};\n input_options.set_force_disable_file_mapping(true);\n\n // Process inputs in deterministic lexical path order.\n std::sort(inputs.begin(), inputs.end());\n\n std::vector created_output_files;\n auto cleanup = gsl::finally([&]() {\n for (auto const & path : created_output_files) {\n std::error_code ec;\n std::filesystem::remove(path, ec);\n }\n });\n\n bool issued_migration_warning = false;\n std::size_t const io_soft_limit = io_limits::detect_soft_limit();\n std::size_t const max_out_size = io_limits::clamp_open_outputs(io_soft_limit);\n\n // Create indexable view of the map iterators so we can conveniently index in batches.\n std::vector>::const_iterator> read_id_dest_iters;\n read_id_dest_iters.reserve(read_id_to_dest.size());\n std::size_t total_requested_read_ids = 0;\n for (auto it = read_id_to_dest.begin(); it != read_id_to_dest.end(); ++it) {\n poll_python_interrupt();\n\n // Check we're not unintentionally overwriting files\n auto const output_path = output / it->first;\n if (std::filesystem::exists(output_path)) {\n if (!force_overwrite) {\n throw std::runtime_error(\n \"Output files already exists and --force-overwrite not set. \");\n } else {\n std::filesystem::remove(output_path);\n }\n }\n\n // Index the map iterator and tally total reads\n read_id_dest_iters.push_back(it);\n total_requested_read_ids += it->second.size();\n }\n\n std::size_t found_read_count = 0;\n std::size_t total_reads_completed = 0;\n\n std::size_t const total_output_batches =\n (read_id_dest_iters.size() + max_out_size - 1) / max_out_size;\n\n if (total_output_batches > 1) {\n std::cerr << \"Subsetting inputs into \" << std::to_string(read_id_dest_iters.size())\n << \" files in \" << std::to_string(total_output_batches) << \" batches of at most \"\n << max_out_size << \" outputs. IO limit: \" << std::to_string(io_soft_limit)\n << std::endl;\n }\n\n ProgressBar progress_bar;\n progress_bar.set_task(\"Starting...\");\n progress_bar.update_max_steps(total_requested_read_ids);\n\n // Iterate over outputs in batches\n for (std::size_t out_st = 0; out_st < read_id_dest_iters.size(); out_st += max_out_size) {\n poll_python_interrupt();\n std::size_t const output_batch_index = (out_st / max_out_size) + 1;\n std::size_t const out_end = std::min(out_st + max_out_size, read_id_dest_iters.size());\n std::string const batch_prefix = \"Batch [\" + std::to_string(output_batch_index) + \"/\"\n + std::to_string(total_output_batches) + \"]: \";\n\n auto repacker = std::make_shared();\n std::unordered_multimap read_id_lookup;\n std::vector dest_to_output;\n dest_to_output.reserve(out_end - out_st);\n\n // For each output in this batch\n for (std::size_t out_idx = out_st; out_idx < out_end; ++out_idx) {\n poll_python_interrupt();\n auto const & read_id_dest = *read_id_dest_iters[out_idx];\n auto const output_path = output / read_id_dest.first;\n\n // Create the output file\n auto writer =\n pod5::create_file_writer(output_path.string(), \"pod5_subset\", output_options);\n if (!writer.ok()) {\n std::cerr << \"Failed to create output file: \" << output_path << std::endl;\n throw std::runtime_error(\"Failed to create output POD5 file\");\n }\n\n // Add the output file writer to the repacker\n created_output_files.push_back(output_path);\n auto repacker_output_file = repacker->add_output(std::move(*writer), !duplicate_ok);\n std::size_t const repacker_output_idx = dest_to_output.size();\n dest_to_output.emplace_back(std::move(repacker_output_file));\n\n // Associate the requested read_ids to this output\n for (auto const & read_id : read_id_dest.second) {\n auto read_id_uuid = pod5::Uuid::from_string(read_id);\n if (!read_id_uuid) {\n std::cerr << \"Invalid read id uuid: \" << read_id << std::endl;\n throw std::runtime_error(\"Invalid read id uuid in mapping\");\n }\n read_id_lookup.insert(std::make_pair(*read_id_uuid, repacker_output_idx));\n }\n }\n\n // Scale the max open input files by current output handle usage and system limits.\n std::size_t const max_open_input_files =\n io_limits::clamp_open_inputs(io_soft_limit, dest_to_output.size());\n std::size_t const max_in_size = std::max(1, max_open_input_files);\n\n // Wait for the number of open readers in the repacker to go below `limit`\n auto wait_for_open_readers_below = [&](std::size_t limit) {\n auto last_update = std::chrono::steady_clock::now();\n while (repacker->currently_open_file_reader_count() >= limit) {\n std::this_thread::sleep_for(std::chrono::milliseconds(100));\n poll_python_interrupt();\n\n auto const now = std::chrono::steady_clock::now();\n if (now - last_update >= std::chrono::milliseconds(2000)) {\n progress_bar.update(total_reads_completed + repacker->reads_completed());\n progress_bar.set_task(\n batch_prefix + \"Waiting for queued writes to complete from \"\n + std::to_string(repacker->currently_open_file_reader_count())\n + \"files...\");\n last_update = now;\n }\n }\n };\n\n // Wait for the repacker to finish with it's currently open readers\n auto wait_for_open_readers_zero = [&]() {\n auto last_update = std::chrono::steady_clock::now();\n while (repacker->currently_open_file_reader_count() > 0) {\n std::this_thread::sleep_for(std::chrono::milliseconds(100));\n poll_python_interrupt();\n\n auto const now = std::chrono::steady_clock::now();\n if (now - last_update >= std::chrono::milliseconds(2000)) {\n progress_bar.update(total_reads_completed + repacker->reads_completed());\n progress_bar.set_task(batch_prefix + \"Waiting for batch IO to complete...\");\n last_update = now;\n }\n }\n };\n\n // Walk each input file in chunks for this output batch.\n for (std::size_t in_st = 0; in_st < inputs.size(); in_st += max_in_size) {\n poll_python_interrupt();\n std::size_t const in_end = std::min(in_st + max_in_size, inputs.size());\n\n // Add an input in this chunk\n for (std::size_t in_idx = in_st; in_idx < in_end; ++in_idx) {\n poll_python_interrupt();\n auto const & input_path = inputs[in_idx];\n\n // Keep in-flight readers below chunk limit.\n wait_for_open_readers_below(max_in_size);\n\n // Clear previous row selections from a previous input file.\n for (auto & output_file : dest_to_output) {\n output_file.clear_per_input_working_data();\n }\n\n // \"Batch [i/N]: Subsetting {input}\"\n progress_bar.set_task(\n batch_prefix + \"Subsetting \" + input_path.filename().string());\n\n // Open the input file\n auto input_reader_opt = pod5::open_file_reader(input_path.string(), input_options);\n if (!input_reader_opt.ok()) {\n std::cerr << \"Failed to open input file: \" << input_path << std::endl;\n throw std::runtime_error(\"Failed to open input POD5 file\");\n }\n auto const & input_reader = *input_reader_opt;\n if (!issued_migration_warning && out_st == 0) {\n auto const pre_migration_version = input_reader->file_version_pre_migration();\n auto const post_migration_version =\n input_reader->schema_metadata().writing_pod5_version;\n if (pre_migration_version != post_migration_version) {\n std::cerr << \"Warning: Migrated an input from POD5 version \"\n << pre_migration_version.to_string() << \" to \"\n << post_migration_version.to_string()\n << \" while subsetting. This can affect performance \"\n \"significantly. Consider updating input files.\"\n << std::endl;\n }\n issued_migration_warning = true;\n }\n\n // Walk the input file batches:\n for (std::size_t i = 0; i < input_reader->num_read_record_batches(); ++i) {\n poll_python_interrupt();\n auto batch = input_reader->read_read_record_batch(i);\n if (!batch.ok()) {\n std::cerr << \"Failed to read batch \" << i\n << \" from input file: \" << input_path << std::endl;\n throw std::runtime_error(\"Failed to read read record batch from POD5 file\");\n }\n\n // Test each read id in the batch to see if we want it:\n auto const & read_id_column = batch->read_id_column();\n for (std::int64_t row = 0; row < read_id_column->length(); ++row) {\n if ((row & 0x3FF) == 0) {\n poll_python_interrupt();\n }\n auto const found = read_id_lookup.equal_range(read_id_column->Value(row));\n for (auto it = found.first; it != found.second; ++it) {\n dest_to_output[it->second].add_row(row);\n found_read_count += 1;\n }\n }\n\n // Store how many rows in this batch were selected:\n for (auto & output_file : dest_to_output) {\n output_file.finish_batch();\n }\n\n progress_bar.update(total_reads_completed + repacker->reads_completed());\n }\n\n // Submit selected reads to each output:\n for (auto & output_file : dest_to_output) {\n repacker->add_selected_reads_to_output(\n output_file.repacker_output,\n input_reader,\n gsl::make_span(output_file.batch_counts),\n gsl::make_span(output_file.all_batch_rows));\n }\n }\n\n // Batch drain barrier for inputs in this output batch.\n wait_for_open_readers_zero();\n }\n\n // Set this output batch to finished:\n std::thread finisher([&] {\n for (auto & output_file : dest_to_output) {\n repacker->set_output_finished(output_file.repacker_output);\n }\n });\n auto join_finisher = gsl::finally([&] {\n if (finisher.joinable()) {\n finisher.join();\n }\n });\n\n // Wait for this batch to complete:\n progress_bar.set_task(batch_prefix + \"Waiting for batch IO to complete...\");\n try {\n while (!repacker->is_complete()) {\n std::this_thread::sleep_for(std::chrono::milliseconds(100));\n poll_python_interrupt();\n progress_bar.update(total_reads_completed + repacker->reads_completed());\n }\n } catch (pybind11::error_already_set const &) {\n throw;\n } catch (std::exception const & e) {\n std::cout << \"\\nError during repacking: \" << e.what() << std::endl;\n }\n\n if (finisher.joinable()) {\n finisher.join();\n }\n\n repacker->finish();\n total_reads_completed += repacker->reads_completed();\n }\n progress_bar.set_task(\"Finished\");\n\n if (found_read_count < total_requested_read_ids && !missing_ok) {\n throw std::runtime_error(\"Missing read_ids from inputs but --missing-ok not set\");\n }\n\n // Clear created output files from cleanup since we succeeded:\n created_output_files.clear();\n}\n" }, { "path": "c++/pod5_format_pybind/subset.h", "content": "#include \n\n#include \n#include \n#include \n#include \n\nvoid subset_pod5s_with_mapping(\n std::vector inputs,\n std::filesystem::path output,\n std::map> read_id_to_dest,\n bool missing_ok,\n bool duplicate_ok,\n bool force_overwrite);\n" }, { "path": "c++/pod5_format_pybind/utils.h", "content": "#pragma once\n\n#include \"pod5_format/result.h\"\n\ninline void raise_error(arrow::Status const & status)\n{\n throw std::runtime_error(status.ToString());\n}\n\ntemplate \ninline void raise_error(arrow::Result const & result)\n{\n throw std::runtime_error(result.status().ToString());\n}\n\n#define POD5_PYTHON_RETURN_NOT_OK(statement) \\\n do { \\\n auto const _res = (statement); \\\n if (!_res.ok()) { \\\n raise_error(_res); \\\n } \\\n } while (false)\n\n#define POD5_PYTHON_ASSIGN_OR_RAISE_IMPL(result_name, lhs, rexpr) \\\n auto && result_name = (rexpr); \\\n if (!(result_name).ok()) { \\\n raise_error(result_name); \\\n } \\\n lhs = std::move(result_name).ValueUnsafe();\n\n#define POD5_PYTHON_ASSIGN_OR_RAISE(lhs, rexpr) \\\n POD5_PYTHON_ASSIGN_OR_RAISE_IMPL( \\\n ARROW_ASSIGN_OR_RAISE_NAME(_error_or_value, __COUNTER__), lhs, rexpr);\n\ninline void throw_on_error(pod5::Status const & s)\n{\n if (!s.ok()) {\n throw std::runtime_error(s.ToString());\n }\n}\n\ntemplate \ninline T throw_on_error(pod5::Result const & s)\n{\n if (!s.ok()) {\n throw std::runtime_error(s.status().ToString());\n }\n return *s;\n}\n" }, { "path": "c++/test/CMakeLists.txt", "content": "\nadd_executable(pod5_unit_tests\n main.cpp\n c_api_null_input.cpp\n c_api_test_utils.h\n c_api_tests.cpp\n c_api_build_test.c\n file_reader_writer_tests.cpp\n output_stream_tests.cpp\n read_table_writer_utils_tests.cpp\n read_table_tests.cpp\n run_info_table_tests.cpp\n schema_tests.cpp\n signal_compression_tests.cpp\n signal_table_tests.cpp\n svb16_scalar_tests.cpp\n svb16_x64_tests.cpp\n test_utils.h\n thread_pool_tests.cpp\n utils.h\n uuid_tests.cpp\n)\n\nif (${CMAKE_CXX_COMPILER_ID} MATCHES \"Clang\")\n set_source_files_properties(c_api_build_test.c PROPERTIES COMPILE_OPTIONS \"-Wdocumentation\")\nendif()\n\ntarget_link_libraries(pod5_unit_tests\n PUBLIC\n pod5_format\n ${maybe_public_libs}\n)\n\nset_property(TARGET pod5_unit_tests PROPERTY CXX_STANDARD 20)\nif (NOT MSVC)\n target_compile_options(pod5_unit_tests PRIVATE ${pod5_warning_options})\nendif()\n\nadd_test(\n NAME pod5_unit_tests\n COMMAND pod5_unit_tests\n)\n" }, { "path": "c++/test/TemporaryDirectory.h", "content": "#pragma once\n\n#include \"pod5_format/uuid.h\"\n\n#include \n\nnamespace ont { namespace testutils {\n\nstatic std::string make_unique_name()\n{\n std::random_device gen;\n auto uuid_gen = pod5::BasicUuidRandomGenerator{gen};\n return to_string(uuid_gen());\n}\n\n/// A scoped directory with a fixed name.\nclass TemporaryDirectory {\npublic:\n /// Where to create the directory.\n enum class Location { CurrentDir, TempDir };\n enum class DeleteBehaviour { AfterOnly, BeforeAndAfter };\n\n /// Creates a random temporary directory\n TemporaryDirectory()\n : TemporaryDirectory(make_unique_name(), Location::TempDir, DeleteBehaviour::AfterOnly)\n {\n }\n\n /// Create a directory.\n explicit TemporaryDirectory(std::filesystem::path path, DeleteBehaviour delete_behaviour)\n : TemporaryDirectory(std::move(path), Location::CurrentDir, delete_behaviour)\n {\n }\n\n /// Create a directory.\n explicit TemporaryDirectory(\n std::filesystem::path path,\n Location location = Location::CurrentDir,\n DeleteBehaviour delete_behaviour = DeleteBehaviour::AfterOnly)\n {\n if (!path.is_absolute()) {\n if (location == Location::CurrentDir) {\n path = std::filesystem::absolute(path);\n } else {\n path = std::filesystem::temp_directory_path() / path;\n }\n }\n if (delete_behaviour == DeleteBehaviour::BeforeAndAfter) {\n std::filesystem::remove_all(m_path);\n }\n std::filesystem::create_directories(path);\n m_path = path;\n }\n\n TemporaryDirectory(TemporaryDirectory const &) = delete;\n TemporaryDirectory & operator=(TemporaryDirectory const &) = delete;\n\n TemporaryDirectory(TemporaryDirectory &&) = default;\n TemporaryDirectory & operator=(TemporaryDirectory &&) = default;\n\n /// Remove the referenced directory.\n ///\n /// Does nothing if this is not a valid object.\n ~TemporaryDirectory()\n {\n if (!m_path.empty()) {\n std::error_code error;\n std::filesystem::remove_all(m_path, error);\n }\n }\n\n /// Path to the directory.\n std::filesystem::path const & path() const { return m_path; }\n\n explicit operator bool() const { return !m_path.empty(); }\n\nprivate:\n std::filesystem::path m_path;\n};\n\ntemplate \nstd::basic_ostream & operator<<(\n std::basic_ostream & os,\n TemporaryDirectory const & td)\n{\n return os << \"TemporaryDirectory{ \" << td.path() << \" }\";\n}\n\n}} // namespace ont::testutils\n" }, { "path": "c++/test/c_api_build_test.c", "content": "#include \"pod5_format/c_api.h\"\n\n// Build check to verify a c file can include the c_api\n" }, { "path": "c++/test/c_api_null_input.cpp", "content": "#include \"c_api_test_utils.h\"\n#include \"pod5_format/c_api.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n\nnamespace {\n\nvoid pod5_reset_error()\n{\n pod5_vbz_compressed_signal_max_size(1);\n REQUIRE_POD5_OK(pod5_get_error_no());\n REQUIRE(pod5_get_error_string() == std::string_view{});\n}\n\nnamespace detail {\n\ntemplate \nconstexpr std::size_t ptr_idx_to_arg_idx()\n{\n // Count how many pointers we've seen at each arg.\n std::size_t ptr_count[]{static_cast(std::is_pointer_v)...};\n std::partial_sum(std::begin(ptr_count), std::end(ptr_count), std::begin(ptr_count));\n\n // Find which arg matches our index.\n for (std::size_t arg_i = 0; arg_i < std::size(ptr_count); arg_i++) {\n if (ptr_count[arg_i] == PtrIdx + 1) {\n return arg_i;\n }\n }\n\n throw \"Cannot find arg for ptr\";\n}\n\ntemplate \nvoid make_ptr_null_impl(std::tuple & args, std::uint64_t valid_ptr_bitset)\n{\n // Grab the arg that we'll be modifying.\n constexpr std::size_t ArgIdx = ptr_idx_to_arg_idx();\n auto & arg = std::get(args);\n using ArgT = std::remove_reference_t;\n static_assert(std::is_pointer_v);\n\n // If the arg isn't a valid one then replace it with a nullptr.\n auto const valid = (valid_ptr_bitset >> PtrIdx) & 1;\n if (!valid) {\n arg = nullptr;\n }\n}\n\ntemplate \nvoid make_ptrs_null(\n std::tuple & args,\n std::uint64_t valid_ptr_bitset,\n std::index_sequence)\n{\n (make_ptr_null_impl(args, valid_ptr_bitset), ...);\n}\n\ntemplate \nauto unpack_and_call(\n pod5_error_t (*func)(Args...),\n std::tuple args,\n std::index_sequence)\n{\n return func(std::get(args)...);\n}\n\n} // namespace detail\n\ntemplate \nvoid call_with_nulls(pod5_error_t (*func)(Args...), Args... args)\n{\n auto const valid_inputs = std::make_tuple(args...);\n\n constexpr std::size_t num_args = sizeof...(Args);\n static_assert(num_args <= 64, \"uint64_t isn't big enough for a bitmask\");\n constexpr std::size_t num_pointers = (std::is_pointer_v + ...);\n\n constexpr auto ArgIdxs = std::make_index_sequence();\n constexpr auto PtrIdxs = std::make_index_sequence();\n\n // Try every combination of NULL for the pointers.\n for (std::uint64_t valid_ptr_bitset = 0; std::popcount(valid_ptr_bitset) != num_pointers;\n valid_ptr_bitset++)\n {\n CAPTURE(valid_ptr_bitset);\n\n // Replace some args with nulls.\n auto inputs = valid_inputs;\n detail::make_ptrs_null(inputs, valid_ptr_bitset, PtrIdxs);\n\n // Make the call.\n pod5_reset_error();\n pod5_error_t const result = detail::unpack_and_call(func, inputs, ArgIdxs);\n\n // Check that it was an error.\n // TODO: We could improve this to check that the first invalid arg matches the error that's\n // reported (ie null string, null file, etc...), but this is already overengineered enough.\n //int const first_ptr = std::countr_zero(~valid_ptr_bitset); // codespell:ignore\n CHECK_POD5_NOT_OK(result);\n CHECK_THAT(pod5_get_error_string(), Catch::Matchers::Contains(\"null\"));\n }\n}\n\nTEST_CASE(\"NULL input doesn't crash\")\n{\n using Catch::Matchers::Contains;\n\n pod5_init();\n auto cleanup = gsl::finally([] { pod5_terminate(); });\n\n // Make a temporary file for the read API to use.\n static constexpr char const temporary_filename[] = \"./foo_c_api.pod5\";\n {\n REQUIRE(remove_file_if_exists(temporary_filename).ok());\n Pod5FileWriter_t * writer = pod5_create_file(temporary_filename, \"c_software\", nullptr);\n REQUIRE_POD5_OK(pod5_get_error_no());\n REQUIRE(writer);\n\n std::int16_t pore_type_id{};\n REQUIRE_POD5_OK(pod5_add_pore(&pore_type_id, writer, \"pore_type\"));\n\n std::int16_t run_info_id{};\n size_t const num_kv_pairs = 1;\n char const * keys[]{\"key\"};\n char const * values[]{\"value\"};\n REQUIRE_POD5_OK(pod5_add_run_info(\n &run_info_id,\n writer,\n \"acquisition_id\",\n 1,\n 1,\n -1,\n num_kv_pairs,\n keys,\n values,\n \"experiment_name\",\n \"flow_cell_id\",\n \"flow_cell_product_code\",\n \"protocol_name\",\n \"protocol_run_id\",\n 1,\n \"sample_id\",\n 1,\n \"sequencing_kit\",\n \"sequencer_position\",\n \"sequencer_position_type\",\n \"software\",\n \"system_name\",\n \"system_type\",\n num_kv_pairs,\n keys,\n values));\n\n read_id_t const read_id{};\n uint32_t const read_number{};\n uint64_t const start_sample{};\n float const median_before{};\n uint16_t const channel{};\n uint8_t const well{};\n float const calibration_offset{};\n float const calibration_scale{};\n pod5_end_reason_t const end_reason{};\n uint8_t const end_reason_forced{};\n uint64_t const num_minknow_events{};\n float const tracked_scaling_scale{};\n float const tracked_scaling_shift{};\n float const predicted_scaling_scale{};\n float const predicted_scaling_shift{};\n uint32_t const num_reads_since_mux_change{};\n float const time_since_mux_change{};\n float const open_pore_level{};\n\n ReadBatchRowInfoArrayV3 const row_data_v3{\n &read_id,\n &read_number,\n &start_sample,\n &median_before,\n &channel,\n &well,\n &pore_type_id,\n &calibration_offset,\n &calibration_scale,\n &end_reason,\n &end_reason_forced,\n &run_info_id,\n &num_minknow_events,\n &tracked_scaling_scale,\n &tracked_scaling_shift,\n &predicted_scaling_scale,\n &predicted_scaling_shift,\n &num_reads_since_mux_change,\n &time_since_mux_change};\n\n int16_t const signal_data[]{1, 2, 3, 4, 5};\n uint32_t const signal_size = std::size(signal_data);\n auto * signal_data_ptr = signal_data;\n\n REQUIRE_POD5_OK(pod5_add_reads_data(\n writer,\n 1,\n READ_BATCH_ROW_INFO_VERSION_3,\n &row_data_v3,\n &signal_data_ptr,\n &signal_size));\n\n ReadBatchRowInfoArrayV4 const row_data_v4{\n &read_id,\n &read_number,\n &start_sample,\n &median_before,\n &channel,\n &well,\n &pore_type_id,\n &calibration_offset,\n &calibration_scale,\n &end_reason,\n &end_reason_forced,\n &run_info_id,\n &num_minknow_events,\n &tracked_scaling_scale,\n &tracked_scaling_shift,\n &predicted_scaling_scale,\n &predicted_scaling_shift,\n &num_reads_since_mux_change,\n &time_since_mux_change,\n &open_pore_level};\n\n REQUIRE_POD5_OK(pod5_add_reads_data(\n writer,\n 1,\n READ_BATCH_ROW_INFO_VERSION_4,\n &row_data_v4,\n &signal_data_ptr,\n &signal_size));\n\n REQUIRE_POD5_OK(pod5_close_and_free_writer(writer));\n }\n\n SECTION(\"Reader API\")\n {\n {\n INFO(\"pod5_open_file\")\n\n pod5_reset_error();\n CHECK(pod5_open_file(nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n }\n\n {\n INFO(\"pod5_open_file_options\")\n\n Pod5ReaderOptions_t options{};\n\n pod5_reset_error();\n CHECK(pod5_open_file_options(nullptr, nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n\n pod5_reset_error();\n CHECK(pod5_open_file_options(temporary_filename, nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null passed\"));\n\n pod5_reset_error();\n CHECK(pod5_open_file_options(nullptr, &options) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n }\n\n {\n INFO(\"pod5_close_and_free_reader\")\n\n pod5_reset_error();\n CHECK_POD5_OK(pod5_close_and_free_reader(nullptr));\n }\n\n // The rest of these functions require a reader.\n Pod5FileReader_t * mutable_reader = pod5_open_file(temporary_filename);\n REQUIRE(mutable_reader);\n auto close_reader =\n gsl::finally([&mutable_reader] { pod5_close_and_free_reader(mutable_reader); });\n Pod5FileReader_t const * reader = mutable_reader;\n\n {\n INFO(\"pod5_get_file_info\")\n\n FileInfo file_info{};\n call_with_nulls(pod5_get_file_info, reader, &file_info);\n }\n\n {\n INFO(\"pod5_get_file_read_table_location\")\n\n EmbeddedFileData_t file_data{};\n call_with_nulls(pod5_get_file_read_table_location, reader, &file_data);\n }\n\n {\n INFO(\"pod5_get_file_signal_table_location\")\n\n EmbeddedFileData_t file_data{};\n call_with_nulls(pod5_get_file_signal_table_location, reader, &file_data);\n }\n\n {\n INFO(\"pod5_get_file_run_info_table_location\")\n\n EmbeddedFileData_t file_data{};\n call_with_nulls(pod5_get_file_run_info_table_location, reader, &file_data);\n }\n\n {\n INFO(\"pod5_get_read_count\")\n\n size_t count{};\n call_with_nulls(pod5_get_read_count, reader, &count);\n }\n\n {\n INFO(\"pod5_get_read_ids\")\n\n std::array read_ids{};\n call_with_nulls(pod5_get_read_ids, reader, read_ids.size(), read_ids.data());\n }\n\n {\n INFO(\"pod5_plan_traversal\")\n\n constexpr std::size_t read_id_count = 1;\n uint8_t const read_id_array[read_id_count * 16]{};\n uint32_t batch_counts{};\n uint32_t batch_rows{};\n size_t find_success_count_out{};\n call_with_nulls(\n pod5_plan_traversal,\n reader,\n read_id_array,\n read_id_count,\n &batch_counts,\n &batch_rows,\n &find_success_count_out);\n }\n\n {\n INFO(\"pod5_get_read_batch_count\")\n\n size_t count{};\n call_with_nulls(pod5_get_read_batch_count, &count, reader);\n }\n\n {\n INFO(\"pod5_get_read_batch\")\n\n Pod5ReadRecordBatch_t * batch = nullptr;\n size_t index{};\n call_with_nulls(pod5_get_read_batch, &batch, reader, index);\n }\n\n {\n INFO(\"pod5_free_read_batch\")\n\n pod5_reset_error();\n CHECK_POD5_OK(pod5_free_read_batch(nullptr));\n }\n\n // The rest of these functions require a batch.\n Pod5ReadRecordBatch_t * mutable_batch = nullptr;\n CHECK_POD5_OK(pod5_get_read_batch(&mutable_batch, reader, 0));\n REQUIRE(mutable_batch);\n auto free_batch = gsl::finally([&mutable_batch] { pod5_free_read_batch(mutable_batch); });\n Pod5ReadRecordBatch_t const * batch = mutable_batch;\n\n {\n INFO(\"pod5_get_read_batch_row_count\")\n\n size_t count{};\n call_with_nulls(pod5_get_read_batch_row_count, &count, batch);\n }\n\n {\n INFO(\"pod5_get_read_batch_row_info_data\")\n\n ReadBatchRowInfoV4 row_info{};\n size_t row = 0;\n uint16_t struct_version = READ_BATCH_ROW_INFO_VERSION;\n uint16_t read_table_version{};\n call_with_nulls(\n pod5_get_read_batch_row_info_data,\n batch,\n row,\n struct_version,\n static_cast(&row_info),\n &read_table_version);\n }\n\n {\n INFO(\"pod5_get_signal_row_indices\")\n\n size_t row = 0;\n uint64_t indices[1];\n call_with_nulls(\n pod5_get_signal_row_indices,\n batch,\n row,\n static_cast(std::size(indices)),\n indices);\n }\n\n {\n INFO(\"pod5_get_calibration_extra_info\")\n\n size_t row = 0;\n CalibrationExtraData_t data{};\n call_with_nulls(pod5_get_calibration_extra_info, batch, row, &data);\n }\n\n {\n INFO(\"pod5_get_run_info\")\n\n int16_t index = 0;\n RunInfoDictData_t * data = nullptr;\n call_with_nulls(pod5_get_run_info, batch, index, &data);\n }\n\n {\n INFO(\"pod5_get_file_run_info\")\n\n run_info_index_t run_info_index = 0;\n RunInfoDictData_t * run_info_data = nullptr;\n call_with_nulls(pod5_get_file_run_info, reader, run_info_index, &run_info_data);\n }\n\n {\n INFO(\"pod5_free_run_info\")\n\n pod5_reset_error();\n CHECK_POD5_OK(pod5_free_run_info(nullptr));\n }\n\n {\n INFO(\"pod5_release_run_info\")\n\n pod5_reset_error();\n#ifndef _WIN32\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n#endif\n CHECK_POD5_OK(pod5_release_run_info(nullptr));\n#ifndef _WIN32\n#pragma GCC diagnostic pop\n#endif\n }\n\n {\n INFO(\"pod5_get_file_run_info_count\")\n\n run_info_index_t count{};\n call_with_nulls(pod5_get_file_run_info_count, reader, &count);\n }\n\n {\n INFO(\"pod5_get_end_reason\")\n\n int16_t index = 0;\n pod5_end_reason end_reason{};\n std::array string{};\n size_t string_len = string.size();\n call_with_nulls(\n pod5_get_end_reason, batch, index, &end_reason, string.data(), &string_len);\n }\n\n {\n INFO(\"pod5_get_pore_type\")\n\n int16_t index = 0;\n std::array string{};\n size_t string_len = string.size();\n call_with_nulls(pod5_get_pore_type, batch, index, string.data(), &string_len);\n }\n\n {\n INFO(\"pod5_get_signal_row_info\")\n\n std::array const signal_rows{};\n SignalRowInfo * signal_row_info = nullptr;\n call_with_nulls(\n pod5_get_signal_row_info,\n reader,\n signal_rows.size(),\n signal_rows.data(),\n &signal_row_info);\n }\n\n {\n INFO(\"pod5_free_signal_row_info\")\n\n pod5_reset_error();\n CHECK_POD5_OK(pod5_free_signal_row_info(0, nullptr));\n CHECK_POD5_NOT_OK(pod5_free_signal_row_info(1, nullptr));\n }\n\n {\n INFO(\"pod5_get_signal\")\n\n // We need a signal row info.\n uint64_t const signal_row_index = 0;\n SignalRowInfo_t * signal_row_info = nullptr;\n CHECK_POD5_OK(pod5_get_signal_row_info(reader, 1, &signal_row_index, &signal_row_info));\n REQUIRE(signal_row_info);\n auto free_signal_row_info = gsl::finally(\n [&signal_row_info] { pod5_free_signal_row_info(1, &signal_row_info); });\n\n std::array samples{};\n call_with_nulls(\n pod5_get_signal,\n reader,\n static_cast(signal_row_info),\n samples.size(),\n samples.data());\n }\n\n {\n INFO(\"pod5_get_read_complete_sample_count\")\n\n size_t row = 0;\n size_t count{};\n call_with_nulls(pod5_get_read_complete_sample_count, reader, batch, row, &count);\n }\n\n {\n INFO(\"pod5_get_read_complete_signal\")\n\n size_t row = 1;\n std::array samples{};\n call_with_nulls(\n pod5_get_read_complete_signal, reader, batch, row, samples.size(), samples.data());\n }\n }\n\n SECTION(\"Writer API\")\n {\n {\n INFO(\"pod5_create_file\")\n\n pod5_reset_error();\n CHECK(pod5_create_file(nullptr, nullptr, nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n\n pod5_reset_error();\n CHECK(pod5_create_file(temporary_filename, nullptr, nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n\n pod5_reset_error();\n CHECK(pod5_create_file(nullptr, temporary_filename, nullptr) == nullptr);\n CHECK_THAT(pod5_get_error_string(), Contains(\"null string passed\"));\n }\n\n {\n INFO(\"pod5_close_and_free_writer\")\n\n pod5_reset_error();\n CHECK_POD5_OK(pod5_close_and_free_writer(nullptr));\n }\n\n // The rest of these functions require a writer.\n REQUIRE(remove_file_if_exists(temporary_filename).ok());\n Pod5FileWriter_t * writer = pod5_create_file(temporary_filename, \"c_software\", nullptr);\n REQUIRE(writer);\n auto close_writer = gsl::finally([&writer] { pod5_close_and_free_writer(writer); });\n\n {\n INFO(\"pod5_add_pore\")\n\n int16_t pore_index{};\n char const pore_type[] = \"test\";\n call_with_nulls(pod5_add_pore, &pore_index, writer, pore_type);\n }\n\n {\n INFO(\"pod5_add_run_info\")\n\n int16_t run_info_index{};\n\n char const dummy_string[] = \"test\";\n\n char const * acquisition_id = dummy_string;\n int64_t acquisition_start_time_ms = 1;\n int16_t adc_max = 1;\n int16_t adc_min = -1;\n char const * experiment_name = dummy_string;\n char const * flow_cell_id = dummy_string;\n char const * flow_cell_product_code = dummy_string;\n char const * protocol_name = dummy_string;\n char const * protocol_run_id = dummy_string;\n int64_t protocol_start_time_ms = 1;\n char const * sample_id = dummy_string;\n uint16_t sample_rate = 1;\n char const * sequencing_kit = dummy_string;\n char const * sequencer_position = dummy_string;\n char const * sequencer_position_type = dummy_string;\n char const * software = dummy_string;\n char const * system_name = dummy_string;\n char const * system_type = dummy_string;\n\n size_t context_tags_count = 1;\n char const * context_tags_keys[]{dummy_string};\n char const * context_tags_values[]{dummy_string};\n\n size_t tracking_id_count = 1;\n char const * tracking_id_keys[]{dummy_string};\n char const * tracking_id_values[]{dummy_string};\n\n call_with_nulls(\n pod5_add_run_info,\n &run_info_index,\n writer,\n acquisition_id,\n acquisition_start_time_ms,\n adc_max,\n adc_min,\n context_tags_count,\n context_tags_keys,\n context_tags_values,\n experiment_name,\n flow_cell_id,\n flow_cell_product_code,\n protocol_name,\n protocol_run_id,\n protocol_start_time_ms,\n sample_id,\n sample_rate,\n sequencing_kit,\n sequencer_position,\n sequencer_position_type,\n software,\n system_name,\n system_type,\n tracking_id_count,\n tracking_id_keys,\n tracking_id_values);\n }\n\n {\n INFO(\"pod5_add_reads_data\")\n\n uint32_t count = 1;\n uint16_t version = READ_BATCH_ROW_INFO_VERSION;\n ReadBatchRowInfoArray_t row_info{};\n int16_t const signal[]{1, 2, 3, 4, 5};\n int16_t const * signals[]{signal};\n uint32_t const signal_size = std::size(signal);\n\n call_with_nulls(\n pod5_add_reads_data,\n writer,\n count,\n version,\n static_cast(&row_info),\n signals,\n &signal_size);\n }\n\n {\n INFO(\"pod5_add_reads_data_pre_compressed\")\n\n uint32_t count = 1;\n uint16_t version = READ_BATCH_ROW_INFO_VERSION;\n ReadBatchRowInfoArray_t row_info{};\n\n char const read0_compressed_signal_chunk0[]{1, 2, 3, 4, 5};\n char const * read0_compressed_signal[]{read0_compressed_signal_chunk0};\n size_t const read0_compressed_signal_sizes[]{std::size(read0_compressed_signal_chunk0)};\n uint32_t const read0_sample_counts[]{3};\n size_t const read0_signal_chunk_count = std::size(read0_compressed_signal);\n\n char const ** compressed_signals[]{read0_compressed_signal};\n size_t const * compressed_signal_sizes[]{read0_compressed_signal_sizes};\n uint32_t const * sample_counts[]{read0_sample_counts};\n size_t const signal_chunk_counts[]{read0_signal_chunk_count};\n\n call_with_nulls(\n pod5_add_reads_data_pre_compressed,\n writer,\n count,\n version,\n static_cast(&row_info),\n compressed_signals,\n compressed_signal_sizes,\n sample_counts,\n signal_chunk_counts);\n }\n }\n\n SECTION(\"VBZ API\")\n {\n {\n INFO(\"pod5_vbz_compress_signal\")\n\n std::array const signal{};\n std::array compressed{};\n size_t compressed_size = compressed.size();\n call_with_nulls(\n pod5_vbz_compress_signal,\n signal.data(),\n signal.size(),\n compressed.data(),\n &compressed_size);\n }\n\n {\n INFO(\"pod5_vbz_decompress_signal\")\n\n std::array const compressed{};\n std::array signal{};\n call_with_nulls(\n pod5_vbz_decompress_signal,\n compressed.data(),\n compressed.size(),\n signal.size(),\n signal.data());\n }\n }\n\n SECTION(\"Misc API\")\n {\n {\n INFO(\"pod5_format_read_id\")\n\n read_id_t const read_id{};\n char * string = nullptr;\n call_with_nulls(pod5_format_read_id, read_id, string);\n }\n }\n}\n\n} // namespace\n" }, { "path": "c++/test/c_api_test_utils.h", "content": "#pragma once\n\n#include \"pod5_format/c_api.h\"\n\n#include \n\n#include \n\n#define CHECK_POD5_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n CHECK_THAT(testutils::Pod5C_Result::capture(_res), testutils::IsPod5COk()); \\\n } while (false)\n\n#define REQUIRE_POD5_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n REQUIRE_THAT(testutils::Pod5C_Result::capture(_res), testutils::IsPod5COk()); \\\n } while (false)\n\n#define CHECK_POD5_NOT_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n CHECK_THAT(testutils::Pod5C_Result::capture(_res), !testutils::IsPod5COk()); \\\n } while (false)\n\n#define REQUIRE_POD5_NOT_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n REQUIRE_THAT(testutils::Pod5C_Result::capture(_res), !testutils::IsPod5COk()); \\\n } while (false)\n\nnamespace testutils {\n\nstruct Pod5C_Result {\n static Pod5C_Result capture(pod5_error_t err_num)\n {\n return Pod5C_Result{err_num, pod5_get_error_string()};\n }\n\n pod5_error_t error_code;\n std::string error_string;\n};\n\nclass IsPod5COk : public Catch::MatcherBase {\npublic:\n IsPod5COk() = default;\n\n bool match(Pod5C_Result const & result) const override { return result.error_code == POD5_OK; }\n\n virtual std::string describe() const override { return \"== POD5_OK\"; }\n};\n\n} // namespace testutils\n\ntemplate <>\nstruct Catch::StringMaker {\n static std::string convert(testutils::Pod5C_Result const & value)\n {\n return \"{ code: \" + std::to_string(value.error_code) + \"| \" + value.error_string + \" }\";\n }\n};\n" }, { "path": "c++/test/c_api_tests.cpp", "content": "#include \"pod5_format/c_api.h\"\n\n#include \"c_api_test_utils.h\"\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n#include \"utils.h\"\n\n#include \n#include \n\n#include \n#include \n#include \n\nstruct Pod5ReadId {\n Pod5ReadId() = default;\n\n Pod5ReadId(pod5::Uuid const & uid) { uid.to_c_array(read_id); }\n\n pod5::Uuid as_uuid() const { return pod5::Uuid{read_id}; }\n\n bool operator==(Pod5ReadId const & other) const { return as_uuid() == other.as_uuid(); }\n\n read_id_t read_id;\n};\n\nstd::ostream & operator<<(std::ostream & str, Pod5ReadId rid) { return str << rid.as_uuid(); }\n\nSCENARIO(\"C API Reads\")\n{\n static constexpr char const * filename = \"./foo_c_api.pod5\";\n\n pod5_init();\n auto fin = gsl::finally([] { pod5_terminate(); });\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n auto input_read_id = uuid_gen();\n auto input_read_id_2 = uuid_gen();\n std::vector signal_1(10);\n std::iota(signal_1.begin(), signal_1.end(), -20000);\n\n std::vector signal_2(20);\n std::iota(signal_2.begin(), signal_2.end(), 0);\n\n std::int16_t adc_min = -4096;\n std::int16_t adc_max = 4095;\n\n float calibration_offset = 54.0f;\n float calibration_scale = 100.0f;\n\n float predicted_scale = 2.3f;\n float predicted_shift = 10.0f;\n float tracked_scale = 4.3f;\n float tracked_shift = 15.0f;\n std::uint32_t num_reads_since_mux_change = 1234;\n float time_since_mux_change = 2.4f;\n float open_pore_level = 123.0f;\n std::uint64_t num_minknow_events = 104;\n\n // Write the file:\n {\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK_FALSE(pod5_create_file(NULL, \"c_software\", NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n CHECK_FALSE(pod5_create_file(\"\", \"c_software\", NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n CHECK_FALSE(pod5_create_file(\"\", NULL, NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n\n REQUIRE(remove_file_if_exists(filename).ok());\n\n auto file = pod5_create_file(filename, \"c_software\", NULL);\n REQUIRE(file);\n CHECK_POD5_OK(pod5_get_error_no());\n\n std::int16_t pore_type_id = -1;\n CHECK_POD5_OK(pod5_add_pore(&pore_type_id, file, \"pore_type\"));\n CHECK(pore_type_id == 0);\n\n std::vector context_tags_keys{\"thing\", \"foo\"};\n std::vector context_tags_values{\"thing_val\", \"foo_val\"};\n std::vector tracking_id_keys{\"baz\", \"other\"};\n std::vector tracking_id_values{\"baz_val\", \"other_val\"};\n\n std::uint32_t read_number = 12;\n std::uint64_t start_sample = 10245;\n float median_before = 200.0f;\n std::uint16_t channel = 43;\n std::uint8_t well = 4;\n pod5_end_reason_t end_reason = POD5_END_REASON_MUX_CHANGE;\n uint8_t end_reason_forced = false;\n auto read_id_array = (read_id_t const *)input_read_id.data();\n\n std::int16_t run_info_id = 0;\n ReadBatchRowInfoArrayV4 row_data{\n read_id_array,\n &read_number,\n &start_sample,\n &median_before,\n &channel,\n &well,\n &pore_type_id,\n &calibration_offset,\n &calibration_scale,\n &end_reason,\n &end_reason_forced,\n &run_info_id,\n &num_minknow_events,\n &tracked_scale,\n &tracked_shift,\n &predicted_scale,\n &predicted_shift,\n &num_reads_since_mux_change,\n &time_since_mux_change,\n &open_pore_level};\n\n std::int16_t const * signal_arr[] = {signal_1.data()};\n std::uint32_t signal_size[] = {(std::uint32_t)signal_1.size()};\n\n // Referencing a non-existent run id should fail:\n CHECK(\n pod5_add_reads_data(\n file, 1, READ_BATCH_ROW_INFO_VERSION_4, &row_data, signal_arr, signal_size)\n == POD5_ERROR_INVALID);\n\n // Now actually add the run info:\n CHECK_POD5_OK(pod5_add_run_info(\n &run_info_id,\n file,\n \"acquisition_id\",\n 15400,\n adc_max,\n adc_min,\n context_tags_keys.size(),\n context_tags_keys.data(),\n context_tags_values.data(),\n \"experiment_name\",\n \"flow_cell_id\",\n \"flow_cell_product_code\",\n \"protocol_name\",\n \"protocol_run_id\",\n 200000,\n \"sample_id\",\n 4000,\n \"sequencing_kit\",\n \"sequencer_position\",\n \"sequencer_position_type\",\n \"software\",\n \"system_name\",\n \"system_type\",\n tracking_id_keys.size(),\n tracking_id_keys.data(),\n tracking_id_values.data()));\n CHECK(run_info_id == 0);\n\n CHECK_POD5_OK(pod5_add_reads_data(\n file, 1, READ_BATCH_ROW_INFO_VERSION_4, &row_data, signal_arr, signal_size));\n\n {\n auto compressed_read_max_size = pod5_vbz_compressed_signal_max_size(signal_2.size());\n std::vector compressed_signal(compressed_read_max_size);\n char const * compressed_data[] = {compressed_signal.data()};\n char const ** compressed_data_ptr = compressed_data;\n std::size_t compressed_size[] = {compressed_signal.size()};\n std::size_t const * compressed_size_ptr = compressed_size;\n std::uint32_t signal_size[] = {(std::uint32_t)signal_2.size()};\n std::uint32_t const * signal_size_ptr = signal_size;\n pod5_vbz_compress_signal(\n signal_2.data(), signal_2.size(), compressed_signal.data(), compressed_size);\n\n std::size_t signal_counts = 1;\n\n auto read_id_array = (read_id_t const *)input_read_id_2.data();\n ReadBatchRowInfoArrayV3 row_data_v3{\n read_id_array,\n &read_number,\n &start_sample,\n &median_before,\n &channel,\n &well,\n &pore_type_id,\n &calibration_offset,\n &calibration_scale,\n &end_reason,\n &end_reason_forced,\n &run_info_id,\n &num_minknow_events,\n &tracked_scale,\n &tracked_shift,\n &predicted_scale,\n &predicted_shift,\n &num_reads_since_mux_change,\n &time_since_mux_change};\n\n CHECK_POD5_OK(pod5_add_reads_data_pre_compressed(\n file,\n 1,\n READ_BATCH_ROW_INFO_VERSION_3,\n &row_data_v3,\n &compressed_data_ptr,\n &compressed_size_ptr,\n &signal_size_ptr,\n &signal_counts));\n }\n\n CHECK_POD5_OK(pod5_close_and_free_writer(file));\n CHECK_POD5_OK(pod5_get_error_no());\n }\n\n // Read the file back:\n {\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK_FALSE(pod5_open_file(NULL));\n auto file = pod5_open_file(filename);\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK(file);\n\n FileInfo_t file_info;\n CHECK_POD5_OK(pod5_get_file_info(file, &file_info));\n CHECK(file_info.version.major == pod5::Pod5MajorVersion);\n CHECK(file_info.version.minor == pod5::Pod5MinorVersion);\n CHECK(file_info.version.revision == pod5::Pod5RevVersion);\n {\n auto reader = pod5::open_file_reader(filename);\n pod5::Uuid file_identifier{file_info.file_identifier};\n CHECK(file_identifier == (*reader)->schema_metadata().file_identifier);\n }\n\n std::size_t read_count = 0;\n CHECK_POD5_OK(pod5_get_read_count(file, &read_count));\n REQUIRE(read_count == 2);\n\n std::vector read_ids(2);\n CHECK(pod5_get_read_ids(file, 1, (read_id_t *)read_ids.data()) != POD5_OK);\n CHECK_POD5_OK(pod5_get_read_ids(file, read_ids.size(), (read_id_t *)read_ids.data()));\n std::vector expected_read_ids{input_read_id, input_read_id_2};\n CHECK(read_ids == expected_read_ids);\n\n std::size_t batch_count = 0;\n CHECK_POD5_OK(pod5_get_read_batch_count(&batch_count, file));\n REQUIRE(batch_count == 1);\n\n Pod5ReadRecordBatch * batch_0 = nullptr;\n CHECK_POD5_OK(pod5_get_read_batch(&batch_0, file, 0));\n REQUIRE(batch_0);\n\n std::size_t row_count = 0;\n CHECK_POD5_OK(pod5_get_read_batch_row_count(&row_count, batch_0));\n REQUIRE(row_count == 2);\n\n // Check out of bounds accesses get errors\n {\n ReadBatchRowInfoV4 v3_struct;\n uint16_t input_version = 0;\n CHECK(\n pod5_get_read_batch_row_info_data(\n batch_0, row_count, READ_BATCH_ROW_INFO_VERSION, &v3_struct, &input_version)\n == POD5_ERROR_INDEXERROR);\n\n std::vector signal_row_indices{1};\n CHECK(\n pod5_get_signal_row_indices(\n batch_0, row_count, signal_row_indices.size(), signal_row_indices.data())\n == POD5_ERROR_INDEXERROR);\n\n CalibrationExtraData calibration_extra_data{};\n CHECK(\n pod5_get_calibration_extra_info(batch_0, row_count, &calibration_extra_data)\n == POD5_ERROR_INDEXERROR);\n }\n\n for (std::size_t row = 0; row < row_count; ++row) {\n CAPTURE(row);\n auto signal = signal_1;\n if (row == 1) {\n signal = signal_2;\n }\n\n static_assert(\n std::is_same::value,\n \"Update this if new structs added\");\n\n ReadBatchRowInfoV3 v3_struct;\n ReadBatchRowInfoV4 v4_struct;\n uint16_t input_version = 0;\n CHECK_POD5_OK(pod5_get_read_batch_row_info_data(\n batch_0, row, READ_BATCH_ROW_INFO_VERSION_3, &v3_struct, &input_version));\n CHECK(\n input_version\n == 4); // We're reading from a v4 file, even if the input struct is v3.\n CHECK_POD5_OK(pod5_get_read_batch_row_info_data(\n batch_0, row, READ_BATCH_ROW_INFO_VERSION_4, &v4_struct, &input_version));\n CHECK(input_version == 4);\n\n auto check_v3_or_v4 = [&](auto name, auto const & input_struct) {\n CAPTURE(name);\n std::string formatted_uuid(36, '\\0');\n CHECK_POD5_OK(pod5_format_read_id(input_struct.read_id, &formatted_uuid[0]));\n CHECK(\n formatted_uuid\n == to_string(*reinterpret_cast(input_struct.read_id)));\n\n CHECK(input_struct.read_number == 12);\n CHECK(input_struct.start_sample == 10245);\n CHECK(input_struct.median_before == 200.0f);\n CHECK(input_struct.channel == 43);\n CHECK(input_struct.well == 4);\n CHECK(input_struct.pore_type == 0);\n CHECK(input_struct.calibration_offset == calibration_offset);\n CHECK(input_struct.calibration_scale == calibration_scale);\n CHECK(input_struct.end_reason == 1);\n CHECK(input_struct.end_reason_forced == uint8_t{false});\n CHECK(input_struct.run_info == 0);\n CHECK(input_struct.num_minknow_events == num_minknow_events);\n CHECK(input_struct.tracked_scaling_scale == tracked_scale);\n CHECK(input_struct.tracked_scaling_shift == tracked_shift);\n CHECK(input_struct.predicted_scaling_scale == predicted_scale);\n CHECK(input_struct.predicted_scaling_shift == predicted_shift);\n CHECK(input_struct.num_reads_since_mux_change == num_reads_since_mux_change);\n CHECK(input_struct.time_since_mux_change == time_since_mux_change);\n CHECK(input_struct.signal_row_count == 1);\n CHECK(input_struct.num_samples == signal.size());\n };\n\n check_v3_or_v4(\"v3\", v3_struct);\n check_v3_or_v4(\"v4\", v4_struct);\n if (row == 0) {\n CHECK(v4_struct.open_pore_level == open_pore_level);\n } else {\n CHECK(std::isnan(v4_struct.open_pore_level));\n }\n\n std::vector signal_row_indices(v3_struct.signal_row_count);\n CHECK_POD5_OK(pod5_get_signal_row_indices(\n batch_0, row, signal_row_indices.size(), signal_row_indices.data()));\n\n std::vector signal_row_info(v3_struct.signal_row_count);\n CHECK_POD5_OK(pod5_get_signal_row_info(\n file,\n signal_row_indices.size(),\n signal_row_indices.data(),\n signal_row_info.data()));\n\n std::vector read_signal(signal_row_info.front()->stored_sample_count);\n REQUIRE(signal_row_info.front()->stored_sample_count == signal.size());\n CHECK_POD5_OK(pod5_get_signal(\n file,\n signal_row_info.front(),\n signal_row_info.front()->stored_sample_count,\n read_signal.data()));\n CHECK(read_signal == signal);\n\n std::size_t sample_count = 0;\n CHECK_POD5_OK(pod5_get_read_complete_sample_count(file, batch_0, row, &sample_count));\n CHECK(sample_count == signal_row_info.front()->stored_sample_count);\n CHECK_POD5_OK(pod5_get_read_complete_signal(\n file, batch_0, row, sample_count, read_signal.data()));\n CHECK(read_signal == signal);\n\n CHECK_POD5_OK(\n pod5_free_signal_row_info(signal_row_indices.size(), signal_row_info.data()));\n\n std::string expected_pore_type{\"pore_type\"};\n std::array char_buffer{};\n std::size_t returned_size = 2; // deliberately too short!\n {\n CHECK(\n pod5_get_pore_type(\n batch_0, v3_struct.pore_type, char_buffer.data(), &returned_size)\n == POD5_ERROR_STRING_NOT_LONG_ENOUGH);\n CHECK(returned_size == expected_pore_type.size() + 1);\n }\n {\n returned_size = char_buffer.size();\n CHECK_POD5_OK(pod5_get_pore_type(\n batch_0, v3_struct.pore_type, char_buffer.data(), &returned_size));\n CHECK(returned_size == expected_pore_type.size() + 1);\n CHECK(std::string{char_buffer.data()} == expected_pore_type);\n }\n {\n returned_size = char_buffer.size();\n CHECK(\n pod5_get_pore_type(batch_0, -1, char_buffer.data(), &returned_size)\n == POD5_ERROR_INDEXERROR);\n CHECK(returned_size == char_buffer.size());\n }\n\n std::string expected_end_reason{\"mux_change\"};\n {\n returned_size = 2; // deliberately too short!\n pod5_end_reason end_reason = POD5_END_REASON_UNKNOWN;\n CHECK(\n pod5_get_end_reason(\n batch_0,\n v3_struct.end_reason,\n &end_reason,\n char_buffer.data(),\n &returned_size)\n == POD5_ERROR_STRING_NOT_LONG_ENOUGH);\n CHECK(returned_size == expected_end_reason.size() + 1);\n }\n {\n returned_size = char_buffer.size();\n pod5_end_reason end_reason = POD5_END_REASON_UNKNOWN;\n CHECK_POD5_OK(pod5_get_end_reason(\n batch_0,\n v3_struct.end_reason,\n &end_reason,\n char_buffer.data(),\n &returned_size));\n CHECK(returned_size == expected_end_reason.size() + 1);\n CHECK(end_reason == POD5_END_REASON_MUX_CHANGE);\n CHECK(std::string{char_buffer.data()} == expected_end_reason);\n }\n // Check getting with an invalid input end reason index:\n {\n returned_size = char_buffer.size();\n pod5_end_reason end_reason = POD5_END_REASON_UNKNOWN;\n CHECK(\n pod5_get_end_reason(\n batch_0,\n v3_struct.end_reason + 100,\n &end_reason,\n char_buffer.data(),\n &returned_size)\n == POD5_ERROR_INDEXERROR);\n CHECK(returned_size == char_buffer.size());\n CHECK(end_reason == POD5_END_REASON_UNKNOWN);\n }\n\n CalibrationExtraData calibration_extra_data{};\n CHECK_POD5_OK(pod5_get_calibration_extra_info(batch_0, row, &calibration_extra_data));\n CHECK(calibration_extra_data.digitisation == adc_max - adc_min + 1);\n CHECK(calibration_extra_data.range == 8192 * calibration_scale);\n }\n\n SECTION(\"Embedded files\")\n {\n for (auto [get_file_location, name] : {\n std::make_tuple(\n pod5_get_file_read_table_location, \"pod5_get_file_read_table_location\"),\n std::make_tuple(\n pod5_get_file_signal_table_location,\n \"pod5_get_file_signal_table_location\"),\n std::make_tuple(\n pod5_get_file_run_info_table_location,\n \"pod5_get_file_run_info_table_location\"),\n })\n {\n CAPTURE(name);\n EmbeddedFileData_t embedded_file_data{};\n CHECK_POD5_OK(get_file_location(file, &embedded_file_data));\n REQUIRE(embedded_file_data.file_name != nullptr);\n CHECK(embedded_file_data.file_name == std::string_view{filename});\n CHECK(embedded_file_data.offset > 0);\n CHECK(embedded_file_data.length > 0);\n }\n }\n\n run_info_index_t run_info_count = 0;\n CHECK_POD5_OK(pod5_get_file_run_info_count(file, &run_info_count));\n REQUIRE(run_info_count == 1);\n\n // Check getting invalid run info indexes fails correctly.\n RunInfoDictData * run_info_error = nullptr;\n CHECK(pod5_get_run_info(batch_0, -1, &run_info_error) == POD5_ERROR_INDEXERROR);\n CHECK_FALSE(run_info_error);\n CHECK(pod5_get_run_info(batch_0, run_info_count, &run_info_error) == POD5_ERROR_INDEXERROR);\n CHECK_FALSE(run_info_error);\n CHECK(pod5_get_file_run_info(file, -1, &run_info_error) == POD5_ERROR_INDEXERROR);\n CHECK_FALSE(run_info_error);\n CHECK(\n pod5_get_file_run_info(file, run_info_count, &run_info_error) == POD5_ERROR_INDEXERROR);\n CHECK_FALSE(run_info_error);\n\n auto check_run_info = [](RunInfoDictData * run_info) {\n REQUIRE(run_info);\n CHECK(run_info->tracking_id.size == 2);\n CHECK(run_info->tracking_id.keys[0] == std::string(\"baz\"));\n CHECK(run_info->tracking_id.keys[1] == std::string(\"other\"));\n CHECK(run_info->tracking_id.values[0] == std::string(\"baz_val\"));\n CHECK(run_info->tracking_id.values[1] == std::string(\"other_val\"));\n CHECK(run_info->context_tags.size == 2);\n CHECK(run_info->context_tags.keys[0] == std::string(\"thing\"));\n CHECK(run_info->context_tags.keys[1] == std::string(\"foo\"));\n CHECK(run_info->context_tags.values[0] == std::string(\"thing_val\"));\n CHECK(run_info->context_tags.values[1] == std::string(\"foo_val\"));\n };\n\n RunInfoDictData * run_info_data_out_1 = nullptr;\n CHECK_POD5_OK(pod5_get_file_run_info(file, 0, &run_info_data_out_1));\n check_run_info(run_info_data_out_1);\n pod5_free_run_info(run_info_data_out_1);\n\n RunInfoDictData * run_info_data_out_2 = nullptr;\n CHECK_POD5_OK(pod5_get_run_info(batch_0, 0, &run_info_data_out_2));\n check_run_info(run_info_data_out_2);\n pod5_free_run_info(run_info_data_out_2);\n\n pod5_free_read_batch(batch_0);\n\n pod5_close_and_free_reader(file);\n CHECK_POD5_OK(pod5_get_error_no());\n }\n}\n\nSCENARIO(\"C API Many Reads\")\n{\n static constexpr char const * filename = \"./foo_c_api.pod5\";\n\n pod5_init();\n auto fin = gsl::finally([] { pod5_terminate(); });\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n std::vector signal_1(10);\n std::iota(signal_1.begin(), signal_1.end(), -20000);\n\n std::vector signal_2(20);\n std::iota(signal_2.begin(), signal_2.end(), 0);\n\n std::size_t const read_count = 10037;\n\n std::int16_t const adc_min = -4096;\n std::int16_t const adc_max = 4095;\n\n std::vector read_id_array(read_count);\n std::generate(read_id_array.begin(), read_id_array.end(), uuid_gen);\n\n // Write the file:\n {\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK_FALSE(pod5_create_file(NULL, \"c_software\", NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n CHECK_FALSE(pod5_create_file(\"\", \"c_software\", NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n CHECK_FALSE(pod5_create_file(\"\", NULL, NULL));\n CHECK(pod5_get_error_no() == POD5_ERROR_INVALID);\n\n REQUIRE(remove_file_if_exists(filename).ok());\n\n auto file = pod5_create_file(filename, \"c_software\", NULL);\n REQUIRE(file);\n CHECK_POD5_OK(pod5_get_error_no());\n\n std::int16_t pore_type_id = -1;\n CHECK_POD5_OK(pod5_add_pore(&pore_type_id, file, \"pore_type\"));\n CHECK(pore_type_id == 0);\n\n std::vector context_tags_keys{\"thing\", \"foo\"};\n std::vector context_tags_values{\"thing_val\", \"foo_val\"};\n std::vector tracking_id_keys{\"baz\", \"other\"};\n std::vector tracking_id_values{\"baz_val\", \"other_val\"};\n\n std::int16_t run_info_id = -1;\n CHECK_POD5_OK(pod5_add_run_info(\n &run_info_id,\n file,\n \"acquisition_id\",\n 15400,\n adc_max,\n adc_min,\n context_tags_keys.size(),\n context_tags_keys.data(),\n context_tags_values.data(),\n \"experiment_name\",\n \"flow_cell_id\",\n \"flow_cell_product_code\",\n \"protocol_name\",\n \"protocol_run_id\",\n 200000,\n \"sample_id\",\n 4000,\n \"sequencing_kit\",\n \"sequencer_position\",\n \"sequencer_position_type\",\n \"software\",\n \"system_name\",\n \"system_type\",\n tracking_id_keys.size(),\n tracking_id_keys.data(),\n tracking_id_values.data()));\n CHECK(run_info_id == 0);\n\n std::vector read_number(read_count, 12);\n std::vector start_sample(read_count, 10245);\n std::vector median_before(read_count, 200.0f);\n std::vector channel(read_count, 43);\n std::vector well(read_count, 4);\n std::vector end_reason(read_count, POD5_END_REASON_MUX_CHANGE);\n std::vector end_reason_forced(read_count, false);\n\n std::vector calibration_offset(read_count, 54.0f);\n std::vector calibration_scale(read_count, 100.0f);\n\n std::vector predicted_scale(read_count, 2.3f);\n std::vector predicted_shift(read_count, 10.0f);\n std::vector tracked_scale(read_count, 4.3f);\n std::vector tracked_shift(read_count, 15.0f);\n std::vector num_reads_since_mux_change(read_count, 1234);\n std::vector time_since_mux_change(read_count, 2.4f);\n std::vector open_pore_level(read_count, 123.0f);\n std::vector num_minknow_events(read_count, 104);\n\n std::vector pore_type_ids(read_count, pore_type_id);\n std::vector run_info_ids(read_count, run_info_id);\n\n std::vector signal_arr;\n std::vector signal_size;\n ReadBatchRowInfoArrayV4 row_data{\n (read_id_t *)read_id_array.data(),\n read_number.data(),\n start_sample.data(),\n median_before.data(),\n channel.data(),\n well.data(),\n pore_type_ids.data(),\n calibration_offset.data(),\n calibration_scale.data(),\n end_reason.data(),\n end_reason_forced.data(),\n run_info_ids.data(),\n num_minknow_events.data(),\n tracked_scale.data(),\n tracked_shift.data(),\n predicted_scale.data(),\n predicted_shift.data(),\n num_reads_since_mux_change.data(),\n time_since_mux_change.data(),\n open_pore_level.data()};\n\n for (std::size_t i = 0; i < read_count; ++i) {\n signal_arr.push_back(signal_1.data());\n signal_size.push_back((std::uint32_t)signal_1.size());\n }\n\n CHECK_POD5_OK(pod5_add_reads_data(\n file,\n read_count,\n READ_BATCH_ROW_INFO_VERSION_3,\n &row_data,\n signal_arr.data(),\n signal_size.data()));\n\n CHECK_POD5_OK(pod5_close_and_free_writer(file));\n CHECK_POD5_OK(pod5_get_error_no());\n }\n\n // Read the file back:\n {\n Pod5ReaderOptions_t options{};\n options.force_disable_file_mapping = true;\n\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK_FALSE(pod5_open_file_options(NULL, &options));\n CHECK_FALSE(pod5_open_file_options(filename, NULL));\n auto file = pod5_open_file_options(filename, &options);\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK(file);\n\n FileInfo_t file_info;\n CHECK_POD5_OK(pod5_get_file_info(file, &file_info));\n CHECK(file_info.version.major == pod5::Pod5MajorVersion);\n CHECK(file_info.version.minor == pod5::Pod5MinorVersion);\n CHECK(file_info.version.revision == pod5::Pod5RevVersion);\n {\n auto reader = pod5::open_file_reader(filename);\n pod5::Uuid file_identifier{file_info.file_identifier};\n CHECK(file_identifier == (*reader)->schema_metadata().file_identifier);\n }\n\n std::size_t read_count_returned = 0;\n CHECK_POD5_OK(pod5_get_read_count(file, &read_count_returned));\n REQUIRE(read_count_returned == read_count);\n\n // Randomise the order of the read IDs and then try and plan a path through them.\n std::shuffle(read_id_array.begin(), read_id_array.end(), gen);\n std::vector batch_counts(read_count);\n std::vector batch_rows(read_count);\n std::size_t find_success_count = 0;\n CHECK_POD5_OK(pod5_plan_traversal(\n file,\n reinterpret_cast(read_id_array.data()),\n read_count,\n batch_counts.data(),\n batch_rows.data(),\n &find_success_count));\n REQUIRE(find_success_count == read_count);\n\n CHECK_POD5_OK(pod5_close_and_free_reader(file));\n }\n}\n\nSCENARIO(\"C API Run Info\")\n{\n static constexpr char const * filename = \"./foo_c_api.pod5\";\n\n pod5_init();\n auto fin = gsl::finally([] { pod5_terminate(); });\n\n std::int16_t adc_min = -4096;\n std::int16_t adc_max = 4095;\n\n auto expected_acq_id = [](std::size_t index) {\n std::string acquisition_id{\"acquisition_id_\"};\n acquisition_id += std::to_string(index);\n return acquisition_id;\n };\n\n // Write the file:\n {\n REQUIRE(remove_file_if_exists(filename).ok());\n\n auto file = pod5_create_file(filename, \"c_software\", NULL);\n REQUIRE(file);\n CHECK_POD5_OK(pod5_get_error_no());\n\n std::vector context_tags_keys{\"thing\", \"foo\"};\n std::vector context_tags_values{\"thing_val\", \"foo_val\"};\n std::vector tracking_id_keys{\"baz\", \"other\"};\n std::vector tracking_id_values{\"baz_val\", \"other_val\"};\n\n for (std::size_t i = 0; i < 10; ++i) {\n std::int16_t run_info_id = -1;\n CHECK_POD5_OK(pod5_add_run_info(\n &run_info_id,\n file,\n expected_acq_id(i).c_str(),\n 15400,\n adc_max,\n adc_min,\n context_tags_keys.size(),\n context_tags_keys.data(),\n context_tags_values.data(),\n \"experiment_name\",\n \"flow_cell_id\",\n \"flow_cell_product_code\",\n \"protocol_name\",\n \"protocol_run_id\",\n 200000,\n \"sample_id\",\n 4000,\n \"sequencing_kit\",\n \"sequencer_position\",\n \"sequencer_position_type\",\n \"software\",\n \"system_name\",\n \"system_type\",\n tracking_id_keys.size(),\n tracking_id_keys.data(),\n tracking_id_values.data()));\n CHECK(run_info_id == static_cast(i));\n }\n CHECK_POD5_OK(pod5_close_and_free_writer(file));\n }\n\n // Read the file back:\n {\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK_FALSE(pod5_open_file(NULL));\n auto file = pod5_open_file(filename);\n CHECK_POD5_OK(pod5_get_error_no());\n CHECK(pod5_get_error_string() == std::string{\"\"});\n REQUIRE(file);\n\n run_info_index_t run_info_count = 0;\n CHECK_POD5_OK(pod5_get_file_run_info_count(file, &run_info_count));\n REQUIRE(run_info_count == 10);\n\n for (run_info_index_t i = 0; i < 10; ++i) {\n RunInfoDictData * run_info_data_out = nullptr;\n CHECK_POD5_OK(pod5_get_file_run_info(file, i, &run_info_data_out));\n CHECK(run_info_data_out->acquisition_id == expected_acq_id(i));\n pod5_free_run_info(run_info_data_out);\n }\n\n CHECK_POD5_OK(pod5_close_and_free_reader(file));\n }\n}\n\nTEST_CASE(\"Missing file passed to pod5_open_file\")\n{\n pod5_init();\n auto cleanup = gsl::finally([] { pod5_terminate(); });\n\n static constexpr char const temporary_filename[] = \"./foo_c_api.pod5\";\n REQUIRE(remove_file_if_exists(temporary_filename).ok());\n\n CHECK(pod5_open_file(temporary_filename) == nullptr);\n}\n\nTEST_CASE(\"Existing file passed to pod5_create_file\")\n{\n pod5_init();\n auto cleanup = gsl::finally([] { pod5_terminate(); });\n\n static constexpr char const temporary_filename[] = \"./foo_c_api.pod5\";\n REQUIRE(remove_file_if_exists(temporary_filename).ok());\n\n // Create it once.\n Pod5FileWriter_t * writer = pod5_create_file(temporary_filename, \"c_software\", nullptr);\n REQUIRE_POD5_OK(pod5_get_error_no());\n REQUIRE(writer != nullptr);\n REQUIRE_POD5_OK(pod5_close_and_free_writer(writer));\n\n // File already exists so this should fail.\n CHECK(pod5_create_file(temporary_filename, \"c_software\", nullptr) == nullptr);\n}\n\nTEST_CASE(\"pod5_create_file with options\")\n{\n pod5_init();\n auto cleanup = gsl::finally([] { pod5_terminate(); });\n\n static constexpr char const temporary_filename[] = \"./foo_c_api.pod5\";\n REQUIRE(remove_file_if_exists(temporary_filename).ok());\n\n Pod5WriterOptions_t test_options{};\n Pod5WriterOptions_t const * options = nullptr;\n bool const with_options = GENERATE(false, true);\n if (with_options) {\n options = &test_options;\n } else {\n test_options.max_signal_chunk_size = GENERATE(0, 1, 2);\n test_options.signal_compression_type = GENERATE(\n CompressionOption::DEFAULT_SIGNAL_COMPRESSION,\n CompressionOption::VBZ_SIGNAL_COMPRESSION,\n CompressionOption::UNCOMPRESSED_SIGNAL);\n test_options.signal_table_batch_size = GENERATE(0, 1, 2);\n test_options.read_table_batch_size = GENERATE(0, 1, 2);\n }\n\n CAPTURE(\n with_options,\n test_options.max_signal_chunk_size,\n test_options.signal_compression_type,\n test_options.signal_table_batch_size,\n test_options.read_table_batch_size);\n\n Pod5FileWriter_t * writer = pod5_create_file(temporary_filename, \"c_software\", options);\n REQUIRE_POD5_OK(pod5_get_error_no());\n REQUIRE(writer != nullptr);\n REQUIRE_POD5_OK(pod5_close_and_free_writer(writer));\n}\n\nTEST_CASE(\"VBZ compression\")\n{\n pod5_init();\n auto cleanup = gsl::finally([] { pod5_terminate(); });\n\n std::size_t const sample_count = 20;\n std::vector input_signal(sample_count);\n std::iota(input_signal.begin(), input_signal.end(), -sample_count / 2);\n\n // Determine max size.\n std::size_t const compressed_read_max_size =\n pod5_vbz_compressed_signal_max_size(input_signal.size());\n REQUIRE(compressed_read_max_size > 0);\n\n // Compress it.\n std::vector compressed_signal(compressed_read_max_size);\n std::size_t compressed_size = compressed_read_max_size;\n REQUIRE_POD5_OK(pod5_vbz_compress_signal(\n input_signal.data(), input_signal.size(), compressed_signal.data(), &compressed_size));\n REQUIRE(compressed_size <= compressed_read_max_size);\n compressed_signal.resize(compressed_size);\n\n // Decompress it.\n std::vector output_signal(sample_count);\n REQUIRE_POD5_OK(pod5_vbz_decompress_signal(\n compressed_signal.data(), compressed_signal.size(), sample_count, output_signal.data()));\n REQUIRE(input_signal == output_signal);\n\n // Providing incorrect buffer sizes should fail rather than crash.\n CHECK_POD5_NOT_OK(pod5_vbz_decompress_signal(\n compressed_signal.data(),\n compressed_signal.size(),\n sample_count * 2,\n output_signal.data()));\n std::size_t bad_compressed_size = compressed_size / 2;\n CHECK_POD5_NOT_OK(pod5_vbz_compress_signal(\n input_signal.data(), input_signal.size(), compressed_signal.data(), &bad_compressed_size));\n\n // Going over the maximum size should produce an error.\n size_t const max_size_error = pod5_vbz_compressed_signal_max_size(std::uint64_t{1} << 48);\n CHECK(max_size_error == 0);\n CHECK_POD5_NOT_OK(pod5_get_error_no());\n}\n" }, { "path": "c++/test/file_reader_writer_tests.cpp", "content": "#include \"pod5_format/async_signal_loader.h\"\n#include \"pod5_format/file_reader.h\"\n#include \"pod5_format/file_writer.h\"\n#include \"pod5_format/internal/combined_file_utils.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/signal_table_reader.h\"\n#include \"pod5_format/thread_pool.h\"\n#include \"pod5_format/uuid.h\"\n#include \"TemporaryDirectory.h\"\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n\nvoid run_file_reader_writer_tests(\n char const * file,\n pod5::FileWriterOptions const & extra_options = {})\n{\n REQUIRE_ARROW_STATUS_OK(remove_file_if_exists(file));\n (void)pod5::register_extension_types();\n auto fin = gsl::finally([] { (void)pod5::unregister_extension_types(); });\n\n auto const run_info_data = get_test_run_info_data(\"_run_info\");\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n auto read_id_1 = uuid_gen();\n\n std::uint16_t channel = 25;\n std::uint8_t well = 3;\n std::uint32_t read_number = 1234;\n std::uint64_t start_sample = 12340;\n std::uint64_t num_minknow_events = 27;\n float median_before = 224.0f;\n float calib_offset = 22.5f;\n float calib_scale = 1.2f;\n float tracked_scaling_scale = 2.3f;\n float tracked_scaling_shift = 100.0f;\n float predicted_scaling_scale = 1.5f;\n float predicted_scaling_shift = 50.0f;\n std::uint32_t num_reads_since_mux_change = 3;\n float time_since_mux_change = 200.0f;\n float open_pore_level = 150.0f;\n\n std::vector signal_1(100'000);\n std::iota(signal_1.begin(), signal_1.end(), 0);\n\n // Write a file:\n {\n pod5::FileWriterOptions options = extra_options;\n options.set_max_signal_chunk_size(20'480);\n options.set_read_table_batch_size(1);\n options.set_signal_table_batch_size(5);\n\n auto writer = pod5::create_file_writer(file, \"test_software\", options);\n REQUIRE_ARROW_STATUS_OK(writer);\n\n auto run_info = (*writer)->add_run_info(run_info_data);\n auto end_reason = (*writer)->lookup_end_reason(pod5::ReadEndReason::signal_negative);\n bool end_reason_forced = true;\n auto pore_type = (*writer)->add_pore_type(\"Pore_type\");\n\n for (std::size_t i = 0; i < 10; ++i) {\n CHECK_ARROW_STATUS_OK((*writer)->add_complete_read(\n {read_id_1,\n read_number,\n start_sample,\n channel,\n well,\n *pore_type,\n calib_offset,\n calib_scale,\n median_before,\n *end_reason,\n end_reason_forced,\n *run_info,\n num_minknow_events,\n tracked_scaling_scale,\n tracked_scaling_shift,\n predicted_scaling_scale,\n predicted_scaling_shift,\n num_reads_since_mux_change,\n time_since_mux_change,\n open_pore_level},\n gsl::make_span(signal_1)));\n }\n }\n\n // Open the file for reading:\n // Write a file:\n {\n auto reader = pod5::open_file_reader(file, {});\n REQUIRE_ARROW_STATUS_OK(reader);\n\n REQUIRE((*reader)->num_read_record_batches() == 10);\n for (std::size_t i = 0; i < 10; ++i) {\n auto read_batch = (*reader)->read_read_record_batch(i);\n REQUIRE_ARROW_STATUS_OK(read_batch);\n\n auto read_id_array = read_batch->read_id_column();\n CHECK(read_id_array->length() == 1);\n CHECK(read_id_array->Value(0) == read_id_1);\n\n auto columns = *read_batch->columns();\n auto const run_info_dict_index =\n std::dynamic_pointer_cast(columns.run_info->indices())->Value(0);\n CHECK(run_info_dict_index == 0);\n auto const run_info_id = read_batch->get_run_info(run_info_dict_index);\n CHECK(*run_info_id == run_info_data.acquisition_id);\n auto const run_info = (*reader)->find_run_info(*run_info_id);\n CHECK(**run_info == run_info_data);\n\n REQUIRE((*reader)->num_signal_record_batches() == 10);\n auto signal_batch = (*reader)->read_signal_record_batch(i);\n REQUIRE_ARROW_STATUS_OK(signal_batch);\n\n auto signal_read_id_array = signal_batch->read_id_column();\n CHECK(signal_read_id_array->length() == 5);\n CHECK(signal_read_id_array->Value(0) == read_id_1);\n CHECK(signal_read_id_array->Value(1) == read_id_1);\n CHECK(signal_read_id_array->Value(2) == read_id_1);\n CHECK(signal_read_id_array->Value(3) == read_id_1);\n CHECK(signal_read_id_array->Value(4) == read_id_1);\n\n auto vbz_signal_array = signal_batch->vbz_signal_column();\n CHECK(vbz_signal_array->length() == 5);\n\n auto samples_array = signal_batch->samples_column();\n CHECK(samples_array->Value(0) == 20'480);\n CHECK(samples_array->Value(1) == 20'480);\n CHECK(samples_array->Value(2) == 20'480);\n CHECK(samples_array->Value(3) == 20'480);\n CHECK(samples_array->Value(4) == 18'080);\n }\n\n auto const samples_mode = GENERATE(\n pod5::AsyncSignalLoader::SamplesMode::NoSamples,\n pod5::AsyncSignalLoader::SamplesMode::Samples);\n\n pod5::AsyncSignalLoader async_no_samples_loader(\n *reader,\n samples_mode,\n {}, // Read all the batches\n {} // No specific rows within batches)\n );\n\n for (std::size_t i = 0; i < 10; ++i) {\n CAPTURE(i);\n auto first_batch_res = async_no_samples_loader.release_next_batch();\n REQUIRE_ARROW_STATUS_OK(first_batch_res);\n auto first_batch = std::move(*first_batch_res);\n CHECK(first_batch->batch_index() == i);\n\n CHECK(first_batch->sample_count().size() == 1);\n CHECK(first_batch->sample_count()[0] == signal_1.size());\n\n CHECK(first_batch->samples().size() == 1);\n if (samples_mode == pod5::AsyncSignalLoader::SamplesMode::Samples) {\n CHECK(first_batch->samples()[0] == signal_1);\n } else {\n CHECK(first_batch->samples()[0].size() == 0);\n }\n }\n }\n}\n\nSCENARIO(\"File Reader Writer Tests\") { run_file_reader_writer_tests(\"./foo.pod5\"); }\n\n#ifdef __linux__\nTEST_CASE(\"Additional make_file_stream() tests\")\n{\n // When the user filesystem doesn't support direct-io, but it is requested then\n // make_file_stream() should fallback to a \"regular\" FileOutputStream\n\n // This because of the disk mounting, this test can only be run by someone or something that\n // is effectively a root user.\n if (::geteuid() != 0) {\n WARN(\"SKIPPING TEST: Need root privileges to mount a test drive.\");\n return;\n }\n\n std::filesystem::path const dir_path = \"./ramdisk_\" + std::to_string(std::time(nullptr));\n\n // Create and mount tmpfs drive.\n try {\n std::filesystem::create_directory(dir_path);\n auto const mount_cmd = std::string{\"mount -o size=500M -t tmpfs none \"} + dir_path.string();\n auto const mount_return = std::system(mount_cmd.c_str());\n if (mount_return != 0) {\n // we have seen this fail in CI where the test thinks it can mount\n // but CI fails to mount, just skip the test\n WARN(\"SKIPPING TEST: Need root privileges to mount a test drive.\");\n return;\n }\n } catch (std::exception const & e) {\n FAIL(\"Failed to create and mount a tmpfs drive: \" << e.what());\n }\n\n auto const umount_cmd = std::string{\"umount \"} + dir_path.string();\n auto remove_directory = gsl::finally([&] { std::filesystem::remove(dir_path); });\n auto remove_mount = gsl::finally([&] { std::ignore = std::system(umount_cmd.c_str()); });\n\n pod5::FileWriterOptions options_for_direct_io;\n options_for_direct_io.set_use_directio(true);\n options_for_direct_io.set_use_sync_io(true);\n options_for_direct_io.set_write_chunk_size(524288);\n\n try {\n auto const test_file_path = dir_path / \"bar.pod5\";\n run_file_reader_writer_tests(test_file_path.c_str(), options_for_direct_io);\n } catch (std::exception const & e) {\n FAIL(\"Failed to run file reader/writer tests: \" << e.what());\n }\n}\n#endif\n\nSCENARIO(\"Opening older files\")\n{\n (void)pod5::register_extension_types();\n auto fin = gsl::finally([] { (void)pod5::unregister_extension_types(); });\n\n auto uuid_from_string = [](char const * val) -> pod5::Uuid {\n auto result = pod5::Uuid::from_string(val);\n REQUIRE(result);\n return *result;\n };\n\n struct ReadData {\n pod5::Uuid read_id;\n std::uint32_t read_number;\n float calibration_offset;\n float calibration_scale;\n std::string end_reason;\n std::string pore_type;\n std::string run_info_id;\n };\n\n std::vector test_read_data{\n {{uuid_from_string(\"0000173c-bf67-44e7-9a9c-1ad0bc728e74\")},\n 1093,\n 21.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"002fde30-9e23-4125-9eae-d112c18a81a7\")},\n 75,\n 4.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"006d1319-2877-4b34-85df-34de7250a47b\")},\n 1053,\n 6.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"00728efb-2120-4224-87d8-580fbb0bd4b2\")},\n 657,\n 2.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"007cc97e-6de2-4ff6-a0fd-1c1eca816425\")},\n 1625,\n 23.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"008468c3-e477-46c4-a6e2-7d021a4ebf0b\")},\n 411,\n 4.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"008ed3dc-86c2-452f-b107-6877a473d177\")},\n 513,\n 5.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"00919556-e519-4960-8aa5-c2dfa020980c\")},\n 56,\n 2.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"00925f34-6baf-47fc-b40c-22591e27fb5c\")},\n 930,\n 37.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {{uuid_from_string(\"009dc9bd-c5f4-487b-ba4c-b9ce7e3a711e\")},\n 195,\n 14.0f,\n 0.1755f,\n \"unknown\",\n \"not_set\",\n \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n };\n\n auto repo_root =\n ::arrow::internal::PlatformFilename::FromString(__FILE__)->Parent().Parent().Parent();\n auto path = GENERATE_COPY(\n *repo_root.Join(\"test_data/multi_fast5_zip_v0.pod5\"),\n *repo_root.Join(\"test_data/multi_fast5_zip_v1.pod5\"),\n *repo_root.Join(\"test_data/multi_fast5_zip_v2.pod5\"),\n *repo_root.Join(\"test_data/multi_fast5_zip_v3.pod5\"),\n *repo_root.Join(\"test_data/multi_fast5_zip_v4.pod5\"));\n auto reader = pod5::open_file_reader(path.ToString(), {});\n CHECK_ARROW_STATUS_OK(reader);\n\n auto metadata = (*reader)->schema_metadata();\n CHECK(metadata.writing_software == \"Python API\");\n\n std::size_t abs_row = 0;\n\n for (std::size_t i = 0; i < (*reader)->num_read_record_batches(); ++i) {\n auto batch = (*reader)->read_read_record_batch(i);\n\n auto columns = batch->columns();\n REQUIRE_ARROW_STATUS_OK(columns);\n\n for (std::size_t row = 0; row < batch->num_rows(); ++row) {\n CAPTURE(abs_row);\n auto read_data = test_read_data[row];\n CHECK(columns->read_id->Value(row) == read_data.read_id);\n CHECK(columns->read_number->Value(row) == read_data.read_number);\n CHECK(columns->calibration_offset->Value(row) == read_data.calibration_offset);\n CHECK(columns->calibration_scale->Value(row) == Approx(read_data.calibration_scale));\n auto end_reason = *batch->get_end_reason(columns->end_reason->GetValueIndex(row));\n CHECK(end_reason.first == pod5::end_reason_from_string(read_data.end_reason));\n CHECK(end_reason.second == read_data.end_reason);\n auto pore_type = batch->get_pore_type(columns->pore_type->GetValueIndex(row));\n CHECK(*pore_type == read_data.pore_type);\n auto run_info_id = batch->get_run_info(columns->run_info->GetValueIndex(row));\n CHECK(*run_info_id == read_data.run_info_id);\n\n ++abs_row;\n }\n }\n CHECK(abs_row == test_read_data.size());\n\n auto run_info = (*reader)->find_run_info(test_read_data[0].run_info_id);\n REQUIRE_ARROW_STATUS_OK(run_info);\n CHECK((*run_info)->acquisition_id == test_read_data[0].run_info_id);\n CHECK((*run_info)->adc_min == -4096);\n CHECK((*run_info)->adc_max == 4095);\n CHECK((*run_info)->protocol_run_id == \"df049455-3552-438c-8176-d4a5b1dd9fc5\");\n CHECK((*run_info)->software == \"python-pod5-converter\");\n CHECK(\n (*run_info)->tracking_id\n == pod5::RunInfoData::MapType{\n {\"asic_id\", \"131070\"},\n {\"asic_id_eeprom\", \"0\"},\n {\"asic_temp\", \"35.043102\"},\n {\"asic_version\", \"IA02C\"},\n {\"auto_update\", \"0\"},\n {\"auto_update_source\", \"https://mirror.oxfordnanoportal.com/software/MinKNOW/\"},\n {\"bream_is_standard\", \"0\"},\n {\"device_id\", \"MS00000\"},\n {\"device_type\", \"minion\"},\n {\"distribution_status\", \"modified\"},\n {\"distribution_version\", \"unknown\"},\n {\"exp_script_name\", \"c449127e3461a521e0865fe6a88716f6f6b0b30c\"},\n {\"exp_script_purpose\", \"sequencing_run\"},\n {\"exp_start_time\", \"2019-05-13T11:11:43Z\"},\n {\"flow_cell_id\", \"\"},\n {\"guppy_version\", \"3.0.3+7e7b7d0\"},\n {\"heatsink_temp\", \"35.000000\"},\n {\"hostname\", \"happy_fish\"},\n {\"installation_type\", \"prod\"},\n {\"local_firmware_file\", \"1\"},\n {\"operating_system\", \"ubuntu 16.04\"},\n {\"protocol_group_id\", \"TEST_EXPERIMENT\"},\n {\"protocol_run_id\", \"df049455-3552-438c-8176-d4a5b1dd9fc5\"},\n {\"protocols_version\", \"4.0.6\"},\n {\"run_id\", \"a08e850aaa44c8b56765eee10b386fc3e516a62b\"},\n {\"sample_id\", \"TEST_SAMPLE\"},\n {\"usb_config\", \"MinION_fx3_1.1.1_ONT#MinION_fpga_1.1.0#ctrl#Auto\"},\n {\"version\", \"3.4.0-rc3\"},\n });\n}\n\n/// Create empty file at \\p path.\nstatic void touch(std::filesystem::path const & path) { std::ofstream const ofs(path); }\n\n/// Create file containing bytes of value zero at \\p path.\nstatic void write_zeros(std::filesystem::path const & path)\n{\n std::ofstream file_stream(path, std::ios::binary);\n for (int i = 0; i < 1000000; ++i) {\n file_stream.put('\\0');\n }\n}\n\n/// Returns true iff the file exists and contains non-null data.\nstatic bool file_writing_started(std::filesystem::path const & file_path)\n{\n if (!exists(file_path)) {\n return false;\n }\n if (!is_regular_file(file_path)) {\n return false;\n }\n // This should be enough for the check as unwritten files are usually\n // empty or populated with nulls if writing has not been done.\n auto const MINIMUM_BYTES_WRITTEN = 3;\n if (file_size(file_path) < 3) {\n return false;\n }\n std::ifstream file{file_path, std::ios::in | std::ios::binary};\n for (auto byte_index = 0; byte_index < MINIMUM_BYTES_WRITTEN; ++byte_index) {\n std::uint8_t byte;\n file >> byte;\n if (byte == 0) {\n return false;\n }\n }\n return true;\n}\n\nstatic bool files_ready_to_recover(std::filesystem::path const & directory_path)\n{\n using directory_iterator = std::filesystem::directory_iterator;\n // The directory should contain 3 files for recovery. A `pod5.tmp` with the signal data\n // a `.tmp-reads` for the reads and a `.tmp-run-info` for the run information.\n return std::count_if(\n directory_iterator(directory_path), directory_iterator{}, file_writing_started)\n >= 3;\n}\n\nstatic void wait_for_files_to_recover(std::filesystem::path const & directory_path)\n{\n using clock = std::chrono::steady_clock;\n auto const begin_waiting = clock::now();\n auto const time_waited = [&]() {\n return std::chrono::duration_cast(clock::now() - begin_waiting);\n };\n\n while (!files_ready_to_recover(directory_path)) {\n REQUIRE(time_waited() < std::chrono::milliseconds{100000});\n // Give any asynchronous file writing threads a chance to write to disk, before we continue.\n std::this_thread::sleep_for(std::chrono::milliseconds{100});\n }\n}\n\nstatic std::filesystem::path create_files_for_recovery(\n std::string const & file_name,\n pod5::Uuid read_id_1,\n ont::testutils::TemporaryDirectory & recovery_directory)\n{\n auto const run_info_data = get_test_run_info_data(\"_run_info\");\n\n std::uint16_t channel = 25;\n std::uint8_t well = 3;\n std::uint32_t read_number = 1234;\n std::uint64_t start_sample = 12340;\n std::uint64_t num_minknow_events = 27;\n float median_before = 224.0f;\n float calib_offset = 22.5f;\n float calib_scale = 1.2f;\n float tracked_scaling_scale = 2.3f;\n float tracked_scaling_shift = 100.0f;\n float predicted_scaling_scale = 1.5f;\n float predicted_scaling_shift = 50.0f;\n std::uint32_t num_reads_since_mux_change = 3;\n float time_since_mux_change = 200.0f;\n float open_pore_level = 150.0f;\n\n std::vector signal_1(100'000);\n std::iota(signal_1.begin(), signal_1.end(), 0);\n\n ont::testutils::TemporaryDirectory data_writing_directory;\n auto file = data_writing_directory.path() / file_name;\n\n pod5::FileWriterOptions options;\n options.set_max_signal_chunk_size(20'480);\n options.set_read_table_batch_size(1);\n options.set_signal_table_batch_size(5);\n options.set_use_sync_io(true);\n auto thread_pool = pod5::make_thread_pool(4);\n options.set_thread_pool(thread_pool);\n\n auto writer_result = pod5::create_file_writer(file.string(), \"test_software\", options);\n REQUIRE_ARROW_STATUS_OK(writer_result);\n std::unique_ptr writer = std::move(*writer_result);\n\n auto run_info = writer->add_run_info(run_info_data);\n auto end_reason = writer->lookup_end_reason(pod5::ReadEndReason::signal_negative);\n bool end_reason_forced = true;\n auto pore_type = writer->add_pore_type(\"Pore_type\");\n\n for (std::size_t i = 0; i < 10; ++i) {\n CHECK_ARROW_STATUS_OK(writer->add_complete_read(\n {read_id_1,\n read_number,\n start_sample,\n channel,\n well,\n *pore_type,\n calib_offset,\n calib_scale,\n median_before,\n *end_reason,\n end_reason_forced,\n *run_info,\n num_minknow_events,\n tracked_scaling_scale,\n tracked_scaling_shift,\n predicted_scaling_scale,\n predicted_scaling_shift,\n num_reads_since_mux_change,\n time_since_mux_change,\n open_pore_level},\n gsl::make_span(signal_1)));\n }\n\n wait_for_files_to_recover(data_writing_directory.path());\n\n // Intermittent failures were seen on Windows, where the file was in the middle of being\n // written when we copied it. This ensures that the file writing threads are done before\n // we take the files.\n thread_pool->wait_for_drain();\n\n // The files are deliberately copied here before they can be properly finalised\n // by the destructor of the FileWriter.\n std::filesystem::copy(data_writing_directory.path(), recovery_directory.path());\n\n REQUIRE(files_ready_to_recover(recovery_directory.path()));\n\n return recovery_directory.path() / file_name;\n}\n\n/// This is equivalent to the C++20 `std::string::ends_with` function. It should be replaced with\n/// the standard library function once we move to the C++20 standard and drop support for building\n/// with GCC 8.\nstatic bool ends_with(std::string const & search_in, std::string const & suffix)\n{\n if (suffix.size() > search_in.size()) {\n return false;\n }\n return search_in.compare(search_in.size() - suffix.size(), std::string::npos, suffix) == 0;\n}\n\nTEST_CASE(\"Check custom rolled ends_with works\", \"[string_utilities]\")\n{\n CHECK(ends_with(\"abc\", \"abc\"));\n CHECK(ends_with(\"abcdef\", \"def\"));\n CHECK_FALSE(ends_with(\"abcdef\", \"abc\"));\n CHECK_FALSE(ends_with(\"def\", \"abcdef\"));\n CHECK_FALSE(ends_with(\"abc\", \"def\"));\n}\n\nstatic std::string escape_for_regex(std::string const & input)\n{\n std::string output;\n for (auto const & character : input) {\n switch (character) {\n case '\\\\':\n case '/':\n case '.':\n case '[':\n case ']':\n case '(':\n case ')':\n output += std::string(\"\\\\\");\n default:;\n }\n output += character;\n }\n return output;\n}\n\nTEST_CASE(\"Recovering .pod5.tmp files\", \"[recovery]\")\n{\n std::string const file_name = \"foo.pod5.tmp\";\n ont::testutils::TemporaryDirectory recovery_directory;\n auto const registration_status = pod5::register_extension_types();\n REQUIRE(registration_status.ok());\n auto const unregister = [] { (void)pod5::unregister_extension_types(); };\n auto fin = std::make_unique>(unregister);\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n std::filesystem::path const path_to_recover =\n create_files_for_recovery(file_name, uuid_gen(), recovery_directory);\n\n REQUIRE(exists(path_to_recover));\n std::filesystem::path reads_path, run_path;\n for (auto const & directory_entry :\n std::filesystem::directory_iterator{recovery_directory.path()})\n {\n if (!directory_entry.is_regular_file()) {\n continue;\n }\n if (ends_with(directory_entry.path().filename().string(), (\".tmp-reads\"))) {\n reads_path = directory_entry.path();\n }\n if (ends_with(directory_entry.path().filename().string(), (\".tmp-run-info\"))) {\n run_path = directory_entry.path();\n }\n }\n REQUIRE(exists(reads_path));\n REQUIRE(exists(run_path));\n auto const recovered_file_path = recovery_directory.path() / (file_name + \"-recovered.pod5\");\n // Confirm that no recovered file is left over from previous test runs.\n REQUIRE_FALSE(exists(recovered_file_path));\n\n // Paths are implicitly convertible to the kind of strings used for paths\n // on the current platform. On Windows this is an `std::wstring`, but the\n // recover_file_writer takes a `std::string`, so we need the explicit\n // conversion to make the build work on that platform.\n // `generic_string()` is used rather than `native()` because Arrow paths\n // always use `/` as a separator, even on Windows.\n std::string const to_recover = path_to_recover.generic_string();\n std::string const recovered = recovered_file_path.generic_string();\n\n bool const cleanup = GENERATE(true, false);\n pod5::RecoverFileOptions const options{.cleanup = cleanup};\n\n CAPTURE(to_recover, recovered, cleanup);\n\n SECTION(\"Recovering basic set of .tmp files.\")\n {\n auto const recovery_details = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_ARROW_STATUS_OK(recovery_details);\n CHECK(exists(recovered_file_path));\n CHECK(recovery_details->row_counts.run_info == 1);\n CHECK(recovery_details->row_counts.signal == 50);\n CHECK(recovery_details->row_counts.reads == 10);\n CHECK(recovery_details->cleanup_errors.empty());\n if (cleanup) {\n CHECK_FALSE(exists(path_to_recover));\n CHECK_FALSE(exists(reads_path));\n CHECK_FALSE(exists(run_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n\n SECTION(\"Recovering whilst extensions are not registered.\")\n {\n fin = {};\n auto recover_result2 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result2.ok());\n REQUIRE(\n recover_result2.status().ToString()\n == \"Invalid: POD5 library is not correctly initialised.\");\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n\n SECTION(\"Recovering without run information.\")\n {\n remove(run_path);\n std::string const run_info_string = run_path.generic_string();\n CAPTURE(run_info_string);\n\n SECTION(\"Recovering set of .tmp files with run info file missing.\")\n {\n auto recover_result3 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result3.ok());\n auto const result_message3 = recover_result3.status().ToString();\n auto const expected_regex3 =\n \"IOError: Failed whilst attempting to recover run information from file - \"\n + escape_for_regex(run_info_string) + R\"(\\. Detail: \\[(errno|Windows error) 2\\] )\"\n + R\"((No such file or directory|The system cannot find the file specified)[.\\n\\r]*)\";\n REQUIRE_THAT(result_message3, Catch::Matchers::Matches(expected_regex3));\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK_FALSE(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n }\n }\n\n SECTION(\"Recovering set of .tmp files with run info file empty.\")\n {\n touch(run_path);\n auto recover_result4 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result4.ok());\n REQUIRE(\n recover_result4.status().ToString()\n == \"Invalid: Failed whilst attempting to recover run information from file - \"\n + run_info_string + \". Detail: File is empty/zero bytes long.\");\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK_FALSE(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n\n SECTION(\"Recovering set of .tmp files with run info file zeroed.\")\n {\n write_zeros(run_path);\n auto recover_result5 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result5.ok());\n REQUIRE(\n recover_result5.status().ToString()\n == \"Invalid: Failed whilst attempting to recover run information from file - \"\n + run_info_string + \". Detail: Not an Arrow file\");\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK_FALSE(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n }\n\n SECTION(\"Recovering without read information.\")\n {\n remove(reads_path);\n std::string const reads_string = reads_path.generic_string();\n CAPTURE(reads_string);\n\n SECTION(\"Recovering set of .tmp files with reads file missing.\")\n {\n auto recover_result6 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result6.ok());\n auto const result_message6 = recover_result6.status().ToString();\n auto const expected_regex6 =\n \"IOError: Failed whilst attempting to recover reads from file - \"\n + escape_for_regex(reads_string) + R\"(\\. Detail: \\[(errno|Windows error) 2\\] )\"\n + R\"((No such file or directory|The system cannot find the file specified)[.\\n\\r]*)\";\n REQUIRE_THAT(result_message6, Catch::Matchers::Matches(expected_regex6));\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK_FALSE(exists(reads_path));\n CHECK(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(run_path));\n }\n }\n\n SECTION(\"Recovering set of .tmp files with reads file empty.\")\n {\n touch(reads_path);\n auto recover_result7 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result7.ok());\n REQUIRE(\n recover_result7.status().ToString()\n == \"Invalid: Failed whilst attempting to recover reads from file - \" + reads_string\n + \". Detail: File is empty/zero bytes long.\");\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK_FALSE(exists(reads_path));\n CHECK(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n\n SECTION(\"Recovering set of .tmp files with reads file zeroed.\")\n {\n write_zeros(reads_path);\n auto recover_result7 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result7.ok());\n REQUIRE(\n recover_result7.status().ToString()\n == \"Invalid: Failed whilst attempting to recover reads from file - \" + reads_string\n + \". Detail: Not an Arrow file\");\n if (cleanup) {\n CHECK(exists(path_to_recover));\n CHECK_FALSE(exists(reads_path));\n CHECK(exists(run_path));\n CHECK_FALSE(exists(recovered_file_path));\n } else {\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n }\n\n SECTION(\"Error messages for problems with combined .pod5.tmp file.\")\n {\n remove(path_to_recover);\n\n SECTION(\"Recovering set of .tmp files with .pod5.tmp file missing.\")\n {\n auto recover_result8 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result8.ok());\n auto const result_message = recover_result8.status().ToString();\n auto const expected_regex =\n \"IOError: Failed to open local file '\" + escape_for_regex(to_recover)\n + R\"('\\. Detail: \\[(errno|Windows error) 2\\] )\"\n + R\"((No such file or directory|The system cannot find the file specified)[.\\n\\r]*)\";\n CAPTURE(result_message, expected_regex);\n REQUIRE_THAT(result_message, Catch::Matchers::Matches(expected_regex));\n if (cleanup) {\n CHECK_FALSE(exists(recovered_file_path));\n }\n CHECK_FALSE(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n\n SECTION(\"Recovering set of .tmp files with .pod5.tmp file empty.\")\n {\n touch(path_to_recover);\n auto recover_result9 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result9.ok());\n REQUIRE(recover_result9.status().ToString() == \"IOError: Invalid signature in file\");\n if (cleanup) {\n CHECK_FALSE(exists(recovered_file_path));\n CHECK_FALSE(exists(path_to_recover));\n } else {\n CHECK(exists(path_to_recover));\n }\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n\n SECTION(\"Recovering set of .tmp files with .pod5.tmp file zeroed.\")\n {\n write_zeros(path_to_recover);\n auto recover_result10 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result10.ok());\n REQUIRE(recover_result10.status().ToString() == \"IOError: Invalid signature in file\");\n if (cleanup) {\n CHECK_FALSE(exists(recovered_file_path));\n CHECK_FALSE(exists(path_to_recover));\n } else {\n CHECK(exists(path_to_recover));\n }\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n\n arrow::Result> result_tmp_file =\n arrow::io::FileOutputStream::Open(to_recover, false);\n REQUIRE_ARROW_STATUS_OK(result_tmp_file);\n std::shared_ptr tmp_file = std::move(*result_tmp_file);\n REQUIRE_ARROW_STATUS_OK(pod5::combined_file_utils::write_file_signature(tmp_file));\n\n SECTION(\"Recover .pod5.tmp missing section marker after signature.\")\n {\n tmp_file = {};\n auto recover_result11 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result11.ok());\n REQUIRE(recover_result11.status().ToString() == \"IOError: Invalid offset into SubFile\");\n if (cleanup) {\n CHECK_FALSE(exists(recovered_file_path));\n }\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n\n SECTION(\"Recover .pod5.tmp missing signal sub file.\")\n {\n pod5::Uuid section_id = uuid_gen();\n REQUIRE_ARROW_STATUS_OK(tmp_file->Write(section_id.data(), section_id.size()));\n tmp_file = {};\n auto recover_result12 = pod5::recover_file(to_recover, recovered, options);\n REQUIRE_FALSE(recover_result12.ok());\n REQUIRE(\n recover_result12.status().ToString()\n == \"Invalid: Failed whilst attempting to recover signal data sub file from file - \"\n + to_recover + \". Detail: Not an Arrow file\");\n if (cleanup) {\n CHECK_FALSE(exists(recovered_file_path));\n }\n CHECK(exists(path_to_recover));\n CHECK(exists(reads_path));\n CHECK(exists(run_path));\n }\n }\n}\n" }, { "path": "c++/test/main.cpp", "content": "#define CATCH_CONFIG_MAIN\n#include \n" }, { "path": "c++/test/output_stream_tests.cpp", "content": "#include \"pod5_format/internal/async_output_stream.h\"\n#include \"pod5_format/internal/linux_output_stream.h\"\n#include \"test_utils.h\"\n\n#include \n#include \n\n#include \n#include \n\nnamespace {\nstatic constexpr std::size_t TestDataSize = 1024 * 1024 * 100;\n\nstd::shared_ptr get_test_data()\n{\n static std::shared_ptr const data = [] {\n auto result = *arrow::AllocateResizableBuffer(TestDataSize);\n for (std::size_t i = 0; i < TestDataSize; ++i) {\n result->mutable_data()[i] = i % 256;\n }\n return result;\n }();\n\n return data;\n}\n\nstd::vector read_file(char const * filename)\n{\n std::ifstream fin(filename, std::ios::binary);\n\n return std::vector(std::istreambuf_iterator(fin), std::istreambuf_iterator());\n}\n\nvoid check_file_contents(char const * filename)\n{\n auto contents = read_file(filename);\n auto expected_contents = get_test_data();\n auto expected_contents_span = expected_contents->span_as();\n\n REQUIRE(contents.size() == expected_contents_span.size());\n\n for (std::size_t i = 0; i < expected_contents_span.size(); i += 1) {\n CHECK(contents[i] == expected_contents_span[i]);\n }\n}\n} // namespace\n\nvoid run_output_stream_test(std::shared_ptr output_stream)\n{\n auto const data = get_test_data();\n\n std::size_t small_writes_bytes_consumed = 0;\n {\n CHECK_ARROW_STATUS_OK(output_stream->Write(data->data() + small_writes_bytes_consumed, 1));\n small_writes_bytes_consumed += 1;\n CHECK_ARROW_STATUS_OK(output_stream->Write(data->data() + small_writes_bytes_consumed, 2));\n small_writes_bytes_consumed += 2;\n CHECK_ARROW_STATUS_OK(output_stream->Write(data->data() + small_writes_bytes_consumed, 4));\n small_writes_bytes_consumed += 4;\n CHECK_ARROW_STATUS_OK(output_stream->Write(data->data() + small_writes_bytes_consumed, 8));\n small_writes_bytes_consumed += 8;\n\n CHECK_ARROW_STATUS_OK(output_stream->Flush());\n }\n\n auto remaining_data_buffer = arrow::SliceBuffer(data, small_writes_bytes_consumed);\n\n {\n auto chunk_1 = arrow::SliceBuffer(remaining_data_buffer, 0, 1024);\n auto chunk_2 = arrow::SliceBuffer(remaining_data_buffer, 1024, 63);\n remaining_data_buffer = arrow::SliceBuffer(remaining_data_buffer, 1024 + 63);\n CHECK_ARROW_STATUS_OK(output_stream->Write(chunk_1));\n CHECK_ARROW_STATUS_OK(output_stream->Write(chunk_2));\n CHECK_ARROW_STATUS_OK(output_stream->Flush());\n }\n\n {\n auto chunk_1 = arrow::SliceBuffer(remaining_data_buffer, 0, 1024 * 1024);\n auto chunk_2 = arrow::SliceBuffer(remaining_data_buffer, 1024 * 1024, 1023);\n remaining_data_buffer = arrow::SliceBuffer(remaining_data_buffer, 1024 * 1024 + 1023);\n CHECK_ARROW_STATUS_OK(output_stream->Write(chunk_1));\n CHECK_ARROW_STATUS_OK(output_stream->Write(chunk_2));\n CHECK_ARROW_STATUS_OK(output_stream->Flush());\n }\n\n CHECK_ARROW_STATUS_OK(output_stream->Write(remaining_data_buffer));\n CHECK_ARROW_STATUS_OK(output_stream->Flush());\n}\n\nTEST_CASE(\"AsyncOutputStream\", \"[OutputStream]\")\n{\n using namespace pod5;\n\n auto const filename = \"./test_file.bin\";\n {\n std::ofstream f(filename, std::ios_base::trunc);\n }\n {\n bool keep_file_open = GENERATE(true, false);\n\n auto thread_pool = make_thread_pool(1);\n auto stream = *AsyncOutputStream::make(\n filename, thread_pool, true, arrow::default_memory_pool(), keep_file_open);\n\n run_output_stream_test(stream);\n }\n check_file_contents(filename);\n}\n\n#ifdef __linux__\nTEST_CASE(\"LinuxOutputStream IOManagerSyncImpl\", \"[OutputStream]\")\n{\n using namespace pod5;\n\n bool keep_file_open = GENERATE(true, false);\n CAPTURE(keep_file_open);\n\n auto filename = \"./test_file.bin\";\n {\n std::ofstream f(filename, std::ios_base::trunc);\n }\n {\n auto io_manager = pod5::make_sync_io_manager();\n REQUIRE_ARROW_STATUS_OK(io_manager);\n auto stream = *LinuxOutputStream::make(\n filename, *io_manager, 10 * 1024 * 1024, true, false, true, keep_file_open);\n\n run_output_stream_test(stream);\n }\n check_file_contents(filename);\n}\n\n#endif\n" }, { "path": "c++/test/read_table_tests.cpp", "content": "#include \"pod5_format/internal/async_output_stream.h\"\n#include \"pod5_format/read_table_reader.h\"\n#include \"pod5_format/read_table_writer.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/types.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nbool operator==(\n std::shared_ptr const & array,\n std::vector const & vec)\n{\n auto const length = static_cast(array->length());\n if (length != vec.size()) {\n return false;\n }\n\n for (std::size_t i = 0; i < length; ++i) {\n if ((*array)[i] != vec[i]) {\n return false;\n }\n }\n return true;\n}\n\nSCENARIO(\"Read table Tests\")\n{\n using namespace pod5;\n\n (void)pod5::register_extension_types();\n auto fin = gsl::finally([] { (void)pod5::unregister_extension_types(); });\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n\n auto file_identifier = uuid_gen();\n\n auto data_for_index = [&](std::size_t index) {\n std::array uuid_source{};\n Uuid read_id{uuid_source};\n\n return std::make_tuple(\n pod5::ReadData{\n read_id,\n std::uint32_t(index * 2),\n std::uint64_t(index * 10),\n std::uint16_t(index + 1),\n std::uint8_t(index + 2),\n 0,\n index * 0.1f,\n index * 0.2f,\n index * 100.0f,\n 0,\n true,\n 0,\n std::uint64_t(index * 150),\n index * 0.4f,\n index * 0.3f,\n index * 0.6f,\n index * 0.5f,\n std::uint32_t(index + 10),\n index * 50.0f,\n index * 0.7f},\n std::vector{index + 2, index + 3});\n };\n\n GIVEN(\"A read table writer\")\n {\n auto filename = \"./foo.pod5\";\n auto pool = arrow::system_memory_pool();\n\n auto const record_batch_count = GENERATE(as{}, 1, 2, 5, 10);\n auto const read_count = GENERATE(1, 2);\n\n {\n auto file_out =\n *pod5::AsyncOutputStream::make(filename, pod5::make_thread_pool(1), true);\n auto schema_metadata = make_schema_key_value_metadata(\n {file_identifier, \"test_software\", *parse_version_number(Pod5Version)});\n REQUIRE_ARROW_STATUS_OK(schema_metadata);\n\n auto pore_writer = pod5::make_pore_writer(pool);\n REQUIRE_ARROW_STATUS_OK(pore_writer);\n auto end_reason_writer = pod5::make_end_reason_writer(pool);\n REQUIRE_ARROW_STATUS_OK(end_reason_writer);\n auto run_info_writer = pod5::make_run_info_writer(pool);\n REQUIRE_ARROW_STATUS_OK(run_info_writer);\n\n auto writer = pod5::make_read_table_writer(\n file_out,\n *schema_metadata,\n read_count,\n *pore_writer,\n *end_reason_writer,\n *run_info_writer,\n pool);\n REQUIRE_ARROW_STATUS_OK(writer);\n\n auto const pore_1 = (*pore_writer)->add(\"Well Type\");\n REQUIRE_ARROW_STATUS_OK(pore_1);\n auto const end_reason_1 = (*end_reason_writer)->lookup(pod5::ReadEndReason::mux_change);\n REQUIRE_ARROW_STATUS_OK(end_reason_1);\n auto const run_info_1 = (*run_info_writer)->add(\"acq_id_1\");\n REQUIRE_ARROW_STATUS_OK(run_info_1);\n auto const run_info_2 = (*run_info_writer)->add(\"acq_id_2\");\n REQUIRE_ARROW_STATUS_OK(run_info_2);\n\n for (std::size_t i = 0; i < record_batch_count; ++i) {\n for (std::size_t j = 0; j < static_cast(read_count); ++j) {\n auto const idx = j + i * read_count;\n\n pod5::ReadData read_data;\n std::vector signal;\n std::tie(read_data, signal) = data_for_index(idx);\n auto row = writer->add_read(read_data, signal, signal.size());\n\n REQUIRE_ARROW_STATUS_OK(row);\n CHECK(*row == idx);\n }\n }\n REQUIRE_ARROW_STATUS_OK(writer->close());\n }\n\n auto file_in = arrow::io::ReadableFile::Open(filename, pool);\n {\n REQUIRE_ARROW_STATUS_OK(file_in);\n\n auto reader = pod5::make_read_table_reader(*file_in, pool);\n REQUIRE_ARROW_STATUS_OK(reader);\n\n auto metadata = reader->schema_metadata();\n CHECK(metadata.file_identifier == file_identifier);\n CHECK(metadata.writing_software == \"test_software\");\n CHECK(metadata.writing_pod5_version == *parse_version_number(Pod5Version));\n\n REQUIRE(reader->num_record_batches() == record_batch_count);\n for (std::size_t i = 0; i < record_batch_count; ++i) {\n auto const record_batch = reader->read_record_batch(i);\n REQUIRE_ARROW_STATUS_OK(record_batch);\n REQUIRE(record_batch->num_rows() == static_cast(read_count));\n\n auto columns = record_batch->columns();\n\n CHECK(columns->read_id->length() == read_count);\n CHECK(columns->signal->length() == read_count);\n CHECK(columns->channel->length() == read_count);\n CHECK(columns->well->length() == read_count);\n CHECK(columns->pore_type->length() == read_count);\n CHECK(columns->calibration_offset->length() == read_count);\n CHECK(columns->calibration_scale->length() == read_count);\n CHECK(columns->read_number->length() == read_count);\n CHECK(columns->start_sample->length() == read_count);\n CHECK(columns->median_before->length() == read_count);\n CHECK(columns->num_samples->length() == read_count);\n CHECK(columns->end_reason->length() == read_count);\n CHECK(columns->end_reason_forced->length() == read_count);\n CHECK(columns->run_info->length() == read_count);\n\n auto pore_indices =\n std::static_pointer_cast(columns->pore_type->indices());\n auto end_reason_indices =\n std::static_pointer_cast(columns->end_reason->indices());\n auto run_info_indices =\n std::static_pointer_cast(columns->run_info->indices());\n for (auto j = 0; j < read_count; ++j) {\n auto idx = j + i * read_count;\n\n pod5::ReadData read_data;\n std::vector expected_signal;\n std::tie(read_data, expected_signal) = data_for_index(idx);\n\n CHECK(columns->read_id->Value(j) == read_data.read_id);\n\n auto signal_data = std::static_pointer_cast(\n columns->signal->value_slice(j));\n CHECK(\n gsl::make_span(signal_data->raw_values(), signal_data->length())\n == gsl::make_span(expected_signal));\n\n CHECK(columns->read_number->Value(j) == read_data.read_number);\n CHECK(columns->start_sample->Value(j) == read_data.start_sample);\n CHECK(columns->median_before->Value(j) == read_data.median_before);\n CHECK(columns->num_samples->Value(j) == expected_signal.size());\n CHECK(columns->calibration_offset->Value(j) == read_data.calibration_offset);\n CHECK(columns->calibration_scale->Value(j) == read_data.calibration_scale);\n CHECK(columns->channel->Value(j) == read_data.channel);\n CHECK(columns->well->Value(j) == read_data.well);\n\n CHECK(end_reason_indices->Value(j) == read_data.end_reason);\n CHECK(pore_indices->Value(j) == read_data.pore_type);\n CHECK(run_info_indices->Value(j) == read_data.run_info);\n }\n\n auto pore_data = record_batch->get_pore_type(0);\n REQUIRE_ARROW_STATUS_OK(pore_data);\n CHECK(*pore_data == \"Well Type\");\n\n auto end_reason_data = record_batch->get_end_reason(1);\n REQUIRE_ARROW_STATUS_OK(end_reason_data);\n CHECK(end_reason_data->first == pod5::ReadEndReason::mux_change);\n CHECK(end_reason_data->second == \"mux_change\");\n\n auto run_info_data = record_batch->get_run_info(0);\n REQUIRE_ARROW_STATUS_OK(run_info_data);\n CHECK(*run_info_data == \"acq_id_1\");\n\n run_info_data = record_batch->get_run_info(1);\n REQUIRE_ARROW_STATUS_OK(run_info_data);\n CHECK(*run_info_data == \"acq_id_2\");\n }\n }\n }\n}\n" }, { "path": "c++/test/read_table_writer_utils_tests.cpp", "content": "#include \"pod5_format/read_table_writer_utils.h\"\n\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n\nTEST_CASE(\"Run Info Writer Tests\")\n{\n auto pool = arrow::system_memory_pool();\n auto run_info_writer = pod5::make_run_info_writer(pool);\n REQUIRE_ARROW_STATUS_OK(run_info_writer);\n\n auto index = (*run_info_writer)->add(\"acq_id_1\");\n CHECK(*index == 0);\n CHECK((*run_info_writer)->item_count() == 1);\n\n // Important to always call this so we test calling it twice\n auto const value_array = (*run_info_writer)->get_value_array();\n\n WHEN(\"Checking the first row\")\n {\n REQUIRE_ARROW_STATUS_OK(value_array);\n\n auto string_value_array = std::dynamic_pointer_cast(*value_array);\n REQUIRE(string_value_array);\n\n CHECK(string_value_array->length() == 1);\n CHECK(string_value_array->Value(0) == \"acq_id_1\");\n }\n\n index = (*run_info_writer)->add(\"acq_id_2\");\n CHECK(*index == 1);\n CHECK((*run_info_writer)->item_count() == 2);\n\n WHEN(\"Checking the rows after a second append\")\n {\n auto value_array = (*run_info_writer)->get_value_array();\n REQUIRE_ARROW_STATUS_OK(value_array);\n\n auto string_value_array = std::dynamic_pointer_cast(*value_array);\n REQUIRE(string_value_array);\n\n CHECK(string_value_array->length() == 2);\n CHECK(string_value_array->Value(0) == \"acq_id_1\");\n CHECK(string_value_array->Value(1) == \"acq_id_2\");\n }\n}\n" }, { "path": "c++/test/run_info_table_tests.cpp", "content": "#include \"pod5_format/internal/async_output_stream.h\"\n#include \"pod5_format/run_info_table_reader.h\"\n#include \"pod5_format/run_info_table_writer.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/types.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nSCENARIO(\"Run Info table Tests\")\n{\n using namespace pod5;\n\n (void)pod5::register_extension_types();\n auto fin = gsl::finally([] { (void)pod5::unregister_extension_types(); });\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n\n auto file_identifier = uuid_gen();\n\n GIVEN(\"A read table writer\")\n {\n auto filename = \"./foo.pod5\";\n auto pool = arrow::system_memory_pool();\n\n auto run_info_data_0 = get_test_run_info_data();\n auto run_info_data_1 = get_test_run_info_data(\"_2\");\n\n {\n auto file_out =\n *pod5::AsyncOutputStream::make(filename, pod5::make_thread_pool(1), true);\n auto schema_metadata = make_schema_key_value_metadata(\n {file_identifier, \"test_software\", *parse_version_number(Pod5Version)});\n REQUIRE_ARROW_STATUS_OK(schema_metadata);\n\n std::size_t run_info_per_batch = 2;\n\n auto writer = pod5::make_run_info_table_writer(\n file_out, *schema_metadata, run_info_per_batch, pool);\n REQUIRE_ARROW_STATUS_OK(writer);\n\n REQUIRE_ARROW_STATUS_OK(writer->add_run_info(run_info_data_0));\n REQUIRE_ARROW_STATUS_OK(writer->add_run_info(run_info_data_1));\n }\n\n auto file_in = arrow::io::ReadableFile::Open(filename, pool);\n {\n REQUIRE_ARROW_STATUS_OK(file_in);\n\n auto reader = pod5::make_run_info_table_reader(*file_in, pool);\n REQUIRE_ARROW_STATUS_OK(reader);\n\n auto metadata = reader->schema_metadata();\n CHECK(metadata.file_identifier == file_identifier);\n CHECK(metadata.writing_software == \"test_software\");\n CHECK(metadata.writing_pod5_version == *parse_version_number(Pod5Version));\n\n REQUIRE(reader->num_record_batches() == 1);\n auto const record_batch = reader->read_record_batch(0);\n REQUIRE_ARROW_STATUS_OK(record_batch);\n REQUIRE(record_batch->num_rows() == 2);\n\n auto columns = record_batch->columns();\n REQUIRE_ARROW_STATUS_OK(columns);\n\n auto check_run_info = [](auto & columns,\n std::size_t index,\n pod5::RunInfoData const & run_info_data) {\n CHECK(columns.acquisition_id->Value(index) == run_info_data.acquisition_id);\n\n CHECK(\n columns.acquisition_start_time->Value(index)\n == run_info_data.acquisition_start_time);\n CHECK(columns.adc_max->Value(index) == run_info_data.adc_max);\n CHECK(columns.adc_min->Value(index) == run_info_data.adc_min);\n CHECK(columns.experiment_name->Value(index) == run_info_data.experiment_name);\n CHECK(columns.flow_cell_id->Value(index) == run_info_data.flow_cell_id);\n CHECK(\n columns.flow_cell_product_code->Value(index)\n == run_info_data.flow_cell_product_code);\n CHECK(columns.protocol_name->Value(index) == run_info_data.protocol_name);\n CHECK(columns.protocol_run_id->Value(index) == run_info_data.protocol_run_id);\n CHECK(\n columns.protocol_start_time->Value(index) == run_info_data.protocol_start_time);\n CHECK(columns.sample_id->Value(index) == run_info_data.sample_id);\n CHECK(columns.sample_rate->Value(index) == run_info_data.sample_rate);\n CHECK(columns.sequencing_kit->Value(index) == run_info_data.sequencing_kit);\n CHECK(columns.sequencer_position->Value(index) == run_info_data.sequencer_position);\n CHECK(\n columns.sequencer_position_type->Value(index)\n == run_info_data.sequencer_position_type);\n CHECK(columns.software->Value(index) == run_info_data.software);\n CHECK(columns.system_name->Value(index) == run_info_data.system_name);\n CHECK(columns.system_type->Value(index) == run_info_data.system_type);\n };\n\n check_run_info(*columns, 0, run_info_data_0);\n check_run_info(*columns, 1, run_info_data_1);\n\n auto found_run_info_0 = reader->find_run_info(run_info_data_0.acquisition_id);\n CHECK_ARROW_STATUS_OK(found_run_info_0);\n CHECK(**found_run_info_0 == run_info_data_0);\n\n auto found_run_info_1 = reader->find_run_info(run_info_data_1.acquisition_id);\n CHECK_ARROW_STATUS_OK(found_run_info_1);\n CHECK(**found_run_info_1 == run_info_data_1);\n }\n }\n}\n" }, { "path": "c++/test/schema_tests.cpp", "content": "#include \"pod5_format/schema_metadata.h\"\n#include \"test_utils.h\"\n\n#include \n\nSCENARIO(\"Version Tests\")\n{\n using namespace pod5;\n\n CHECK(Version(1, 2, 3) < Version(3, 2, 1));\n CHECK(Version(1, 2, 3) < Version(1, 3, 3));\n CHECK(Version(1, 2, 3) < Version(1, 2, 4));\n\n CHECK(Version(3, 2, 1) > Version(1, 2, 3));\n CHECK(Version(1, 3, 3) > Version(1, 2, 3));\n CHECK(Version(1, 2, 4) > Version(1, 2, 3));\n\n CHECK(Version(1, 2, 3) == Version(1, 2, 3));\n\n CHECK(Version(1, 2, 3) != Version(2, 2, 3));\n CHECK(Version(1, 2, 3) != Version(1, 3, 3));\n CHECK(Version(1, 2, 3) != Version(1, 2, 4));\n\n CHECK_ARROW_STATUS_NOT_OK(parse_version_number(\"1.2.3.4\"));\n CHECK_ARROW_STATUS_NOT_OK(parse_version_number(\"1.2.3-pre\"));\n\n auto const parsed_version = parse_version_number(\"10.200.3\");\n REQUIRE_ARROW_STATUS_OK(parsed_version);\n CHECK(Version(10, 200, 3) == *parsed_version);\n CHECK(parsed_version->major_version() == 10);\n CHECK(parsed_version->minor_version() == 200);\n CHECK(parsed_version->revision_version() == 3);\n\n CHECK(Version(1, 200, 30).to_string() == \"1.200.30\");\n}\n" }, { "path": "c++/test/signal_compression_tests.cpp", "content": "#include \"pod5_format/signal_compression.h\"\n\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nSCENARIO(\"Signal compression Tests\")\n{\n auto pool = arrow::system_memory_pool();\n\n std::vector signal(100'00);\n std::iota(signal.begin(), signal.end(), 0);\n\n auto compressed = pod5::compress_signal(gsl::make_span(signal), pool);\n REQUIRE_ARROW_STATUS_OK(compressed);\n auto compressed_span = gsl::make_span((*compressed)->data(), (*compressed)->size());\n\n auto decompressed = pod5::decompress_signal(compressed_span, signal.size(), pool);\n REQUIRE_ARROW_STATUS_OK(decompressed);\n auto decompressed_span = gsl::make_span((*decompressed)->data(), (*decompressed)->size())\n .as_span();\n\n CHECK(gsl::make_span(signal) == decompressed_span);\n}\n" }, { "path": "c++/test/signal_table_tests.cpp", "content": "#include \"pod5_format/internal/async_output_stream.h\"\n#include \"pod5_format/schema_metadata.h\"\n#include \"pod5_format/signal_compression.h\"\n#include \"pod5_format/signal_table_reader.h\"\n#include \"pod5_format/signal_table_writer.h\"\n#include \"pod5_format/types.h\"\n#include \"pod5_format/uuid.h\"\n#include \"pod5_format/version.h\"\n#include \"test_utils.h\"\n#include \"utils.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\nSCENARIO(\"Signal table Tests\")\n{\n using namespace pod5;\n\n (void)pod5::register_extension_types();\n auto fin = gsl::finally([] { (void)pod5::unregister_extension_types(); });\n\n std::mt19937 gen{Catch::rngSeed()};\n auto uuid_gen = pod5::UuidRandomGenerator{gen};\n\n auto file_identifier = uuid_gen();\n\n auto read_id_1 = uuid_gen();\n auto read_id_2 = uuid_gen();\n std::vector signal_1(100'000);\n std::iota(signal_1.begin(), signal_1.end(), 0);\n std::vector signal_2(10'000, 1);\n\n GIVEN(\"A signal table writer\")\n {\n auto filename = \"./foo.pod5\";\n auto pool = arrow::system_memory_pool();\n\n auto signal_type = GENERATE(SignalType::UncompressedSignal, SignalType::VbzSignal);\n\n {\n auto file_out =\n *pod5::AsyncOutputStream::make(filename, pod5::make_thread_pool(1), true);\n auto schema_metadata = make_schema_key_value_metadata(\n {file_identifier, \"test_software\", *parse_version_number(Pod5Version)});\n REQUIRE_ARROW_STATUS_OK(schema_metadata);\n\n auto writer =\n pod5::make_signal_table_writer(file_out, *schema_metadata, 100, signal_type, pool);\n REQUIRE_ARROW_STATUS_OK(writer);\n\n WHEN(\"Writing a read\")\n {\n auto row_1 = writer->add_signal(read_id_1, gsl::make_span(signal_1));\n\n auto row_2 = writer->add_signal(read_id_2, gsl::make_span(signal_2));\n\n REQUIRE_ARROW_STATUS_OK(writer->close());\n\n THEN(\"Read row ids are correct\")\n {\n REQUIRE_ARROW_STATUS_OK(row_1);\n REQUIRE_ARROW_STATUS_OK(row_2);\n CHECK(*row_1 == 0);\n CHECK(*row_2 == 1);\n }\n }\n }\n\n auto file_in = arrow::io::ReadableFile::Open(filename, pool);\n {\n REQUIRE_ARROW_STATUS_OK(file_in);\n\n auto reader = pod5::make_signal_table_reader(*file_in, 20, pool);\n CAPTURE(reader);\n REQUIRE_ARROW_STATUS_OK(reader);\n\n auto metadata = reader->schema_metadata();\n CHECK(metadata.file_identifier == file_identifier);\n CHECK(metadata.writing_software == \"test_software\");\n CHECK(metadata.writing_pod5_version == *parse_version_number(Pod5Version));\n\n REQUIRE(reader->num_record_batches() == 1);\n auto const record_batch_0 = reader->read_record_batch(0);\n REQUIRE_ARROW_STATUS_OK(record_batch_0);\n REQUIRE(record_batch_0->num_rows() == 2);\n\n auto read_id = record_batch_0->read_id_column();\n CHECK(read_id->length() == 2);\n CHECK(read_id->Value(0) == read_id_1);\n CHECK(read_id->Value(1) == read_id_2);\n\n if (signal_type == SignalType::VbzSignal) {\n auto signal = record_batch_0->vbz_signal_column();\n CHECK(signal->length() == 2);\n\n auto compare_compressed_signal =\n [&](gsl::span compressed_actual,\n std::vector const & expected) {\n auto decompressed =\n pod5::decompress_signal(compressed_actual, expected.size(), pool);\n REQUIRE_ARROW_STATUS_OK(decompressed);\n\n auto actual =\n gsl::make_span((*decompressed)->data(), (*decompressed)->size())\n .as_span();\n CHECK(actual == gsl::make_span(expected));\n };\n\n auto signal_typed = std::static_pointer_cast(signal);\n compare_compressed_signal(signal_typed->Value(0), signal_1);\n compare_compressed_signal(signal_typed->Value(1), signal_2);\n } else if (signal_type == SignalType::UncompressedSignal) {\n auto signal = record_batch_0->uncompressed_signal_column();\n CHECK(signal->length() == 2);\n\n auto signal_1_read =\n std::static_pointer_cast(signal->value_slice(0));\n std::vector stored_values_1(\n signal_1_read->raw_values(),\n signal_1_read->raw_values() + signal_1_read->length());\n CHECK(stored_values_1 == signal_1);\n auto signal_2_read =\n std::static_pointer_cast(signal->value_slice(1));\n std::vector stored_values_2(\n signal_2_read->raw_values(),\n signal_2_read->raw_values() + signal_2_read->length());\n CHECK(stored_values_2 == signal_2);\n } else {\n FAIL(\"Unknown signal type\");\n }\n\n auto samples = record_batch_0->samples_column();\n CHECK(samples->length() == 2);\n CHECK(samples->Value(0) == signal_1.size());\n CHECK(samples->Value(1) == signal_2.size());\n }\n }\n}\n" }, { "path": "c++/test/svb16_scalar_tests.cpp", "content": "#include \"pod5_format/svb16/decode.hpp\"\n#include \"pod5_format/svb16/encode.hpp\"\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n\nusing Catch::Matchers::Equals;\n\ntemplate \nvoid test_scalar_encode_scalar_decode()\n{\n static constexpr uint32_t DATA_COUNT = 1024;\n std::minstd_rand rng;\n\n std::vector data(DATA_COUNT);\n std::uniform_int_distribution dist{\n std::numeric_limits::min(), std::numeric_limits::max()};\n std::generate(data.begin(), data.end(), [&] { return dist(rng); });\n\n std::vector encoded(svb16_max_encoded_length(data.size()));\n auto const encoded_count =\n svb16::encode_scalar(\n data.data(), encoded.data(), encoded.data() + svb16_key_length(data.size()), DATA_COUNT)\n - encoded.data();\n\n CHECK(encoded_count <= svb16_max_encoded_length(data.size()));\n\n std::vector decoded(DATA_COUNT);\n auto const encoded_span = gsl::make_span(encoded);\n auto const key_length = svb16_key_length(data.size());\n auto const consumed = svb16::decode_scalar(\n gsl::make_span(decoded),\n encoded_span.subspan(0, key_length),\n encoded_span.subspan(key_length))\n - encoded.data();\n\n CHECK(consumed == encoded_count);\n\n CHECK_THAT(decoded, Equals(data));\n}\n\nTEST_CASE(\"Scalar decode is inverse of scalar encode\", \"[scalar]\")\n{\n SECTION(\"Unsigned, no delta, no zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Signed, no delta, no zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Unsigned, delta, no zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Signed, delta, no zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Unsigned, delta, zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Signed, delta, zig-zag\") { test_scalar_encode_scalar_decode(); }\n SECTION(\"Unsigned, no delta, zig-zag\")\n {\n // this scenario doesn't really make sense, but it's possible, so let's test it\n test_scalar_encode_scalar_decode();\n }\n SECTION(\"Signed, no delta, zig-zag\")\n {\n test_scalar_encode_scalar_decode();\n }\n}\n" }, { "path": "c++/test/svb16_x64_tests.cpp", "content": "#include \"pod5_format/svb16/decode.hpp\"\n#include \"pod5_format/svb16/encode.hpp\"\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#ifdef SVB16_X64\n\nusing Catch::Matchers::Equals;\n\ntemplate \nvoid test_sse_encode_scalar_decode()\n{\n uint32_t const DATA_COUNT = GENERATE(\n 1000,\n 20000); // Deliberately not aligned to 64 so we test the scalar tidy up code at the end.\n std::minstd_rand rng;\n std::vector data(DATA_COUNT);\n std::uniform_int_distribution dist{\n std::numeric_limits::min(), std::numeric_limits::max()};\n std::generate(data.begin(), data.end(), [&] { return dist(rng); });\n\n std::vector encoded(svb16_max_encoded_length(data.size()));\n auto const encoded_count =\n svb16::encode_sse(\n data.data(), encoded.data(), encoded.data() + svb16_key_length(data.size()), DATA_COUNT)\n - encoded.data();\n\n CHECK(encoded_count <= svb16_max_encoded_length(data.size()));\n\n std::vector encoded_scalar(svb16_max_encoded_length(data.size()));\n auto const scalar_encoded_count = svb16::encode_scalar(\n data.data(),\n encoded_scalar.data(),\n encoded_scalar.data() + svb16_key_length(data.size()),\n DATA_COUNT)\n - encoded_scalar.data();\n CHECK(scalar_encoded_count == encoded_count);\n CHECK(encoded == encoded_scalar);\n\n std::vector decoded(DATA_COUNT);\n auto const encoded_span = gsl::make_span(encoded);\n auto const key_length = svb16_key_length(data.size());\n auto const consumed = svb16::decode_sse(\n gsl::make_span(decoded),\n encoded_span.subspan(0, key_length),\n encoded_span.subspan(key_length))\n - encoded.data();\n\n CHECK(consumed == encoded_count);\n\n CHECK_THAT(decoded, Equals(data));\n}\n\nTEST_CASE(\"SSE decode is inverse of scalar encode\", \"[scalar]\")\n{\n SECTION(\"Unsigned, no delta, no zig-zag\")\n {\n test_sse_encode_scalar_decode();\n }\n SECTION(\"Signed, no delta, no zig-zag\")\n {\n test_sse_encode_scalar_decode();\n }\n SECTION(\"Unsigned, delta, no zig-zag\")\n {\n test_sse_encode_scalar_decode();\n }\n SECTION(\"Signed, delta, no zig-zag\") { test_sse_encode_scalar_decode(); }\n SECTION(\"Unsigned, delta, zig-zag\") { test_sse_encode_scalar_decode(); }\n SECTION(\"Signed, delta, zig-zag\") { test_sse_encode_scalar_decode(); }\n SECTION(\"Unsigned, no delta, zig-zag\")\n {\n // this scenario doesn't really make sense, but it's possible, so let's test it\n test_sse_encode_scalar_decode();\n }\n SECTION(\"Signed, no delta, zig-zag\") { test_sse_encode_scalar_decode(); }\n}\n\n#endif\n" }, { "path": "c++/test/test_utils.h", "content": "#pragma once\n\n#include \n#include \n#include \n\ntemplate \nstruct Catch::StringMaker> {\n static std::string convert(arrow::Result const & value) { return value.status().ToString(); }\n};\n\ntemplate \nclass IsStatusOk : public Catch::MatcherBase {\npublic:\n IsStatusOk() = default;\n\n bool match(arrow::Status const & status) const override { return status.ok() == CheckOk; }\n\n virtual std::string describe() const override { return \"== arrow::Status::OK()\"; }\n};\n\ntemplate \nclass IsResultOk : public Catch::MatcherBase> {\npublic:\n IsResultOk() = default;\n\n bool match(arrow::Result const & status) const override { return status.ok() == CheckOk; }\n\n virtual std::string describe() const override { return \"== arrow::Status::OK()\"; }\n};\n\ntemplate \ninline IsResultOk _is_arrow_ok(arrow::Result const &)\n{\n return IsResultOk();\n}\n\ninline IsStatusOk _is_arrow_ok(arrow::Status const &) { return IsStatusOk(); }\n\ntemplate \ninline IsResultOk _is_arrow_not_ok(arrow::Result const &)\n{\n return IsResultOk();\n}\n\ninline IsStatusOk _is_arrow_not_ok(arrow::Status const &) { return IsStatusOk(); }\n\n#define CHECK_ARROW_STATUS_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n CHECK_THAT(_res, _is_arrow_ok(_res)); \\\n } while (false)\n\n#define REQUIRE_ARROW_STATUS_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n REQUIRE_THAT(_res, _is_arrow_ok(_res)); \\\n } while (false)\n\n#define CHECK_ARROW_STATUS_NOT_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n CHECK_THAT(_res, _is_arrow_not_ok(_res)); \\\n } while (false)\n\n#define REQUIRE_ARROW_STATUS_NOT_OK(statement) \\\n do { \\\n auto const & _res = (statement); \\\n REQUIRE_THAT(_res, _is_arrow_not_ok(_res)); \\\n } while (false)\n" }, { "path": "c++/test/thread_pool_tests.cpp", "content": "#include \"pod5_format/thread_pool.h\"\n\n#include \n\n#include \n#include \n#include \n#include \n\nTEST_CASE(\"Thread pool runs tasks concurrently\", \"[thread_pool]\")\n{\n using namespace std::chrono_literals;\n\n auto const explicit_stop = GENERATE(true, false);\n CAPTURE(explicit_stop);\n\n auto const use_strands = GENERATE(true, false);\n CAPTURE(use_strands);\n\n // semaphores only in std lib in c++20, so fake them\n std::mutex sem_mutex;\n int sem1 = 2;\n std::condition_variable cv1;\n int sem2 = 2;\n std::condition_variable cv2;\n\n auto const create_task = [&]() -> std::function {\n return [&] {\n std::unique_lock l{sem_mutex};\n sem1--;\n if (sem1 > 0) {\n cv1.wait(l, [&] { return sem1 == 0; });\n } else {\n l.unlock();\n cv1.notify_all();\n std::this_thread::sleep_for(1ms);\n l.lock();\n }\n\n sem2--;\n if (sem2 > 0) {\n cv2.wait(l, [&] { return sem2 == 0; });\n } else {\n l.unlock();\n cv2.notify_all();\n }\n };\n };\n\n auto thread_pool = pod5::make_thread_pool(2);\n std::shared_ptr strands[2];\n if (use_strands) {\n for (unsigned i = 0; i < 2; ++i) {\n strands[i] = thread_pool->create_strand();\n strands[i]->post(create_task());\n }\n } else {\n thread_pool->post(create_task());\n thread_pool->post(create_task());\n }\n\n if (explicit_stop) {\n thread_pool->stop_and_drain();\n } else {\n thread_pool.reset();\n for (unsigned i = 0; i < 2; ++i) {\n strands[i].reset();\n }\n }\n\n REQUIRE(sem1 == 0);\n REQUIRE(sem2 == 0);\n}\n\nTEST_CASE(\"Tasks on the same strand are serialised\", \"[thread_pool]\")\n{\n using namespace std::chrono_literals;\n\n auto const explicit_stop = GENERATE(true, false);\n CAPTURE(explicit_stop);\n\n std::mutex seq_mutex;\n std::vector seq;\n seq.reserve(4);\n\n auto const create_task = [&](int const num) -> std::function {\n return [&, num] {\n {\n std::lock_guard l{seq_mutex};\n seq.push_back(num);\n }\n std::this_thread::sleep_for(50ms);\n {\n std::lock_guard l{seq_mutex};\n seq.push_back(num);\n }\n };\n };\n\n auto thread_pool = pod5::make_thread_pool(2);\n auto strand = thread_pool->create_strand();\n strand->post(create_task(0));\n strand->post(create_task(1));\n\n if (explicit_stop) {\n thread_pool->stop_and_drain();\n } else {\n thread_pool.reset();\n strand.reset();\n }\n\n REQUIRE(seq.size() == 4);\n if (seq[0] == 0) {\n REQUIRE(seq == (std::vector{0, 0, 1, 1}));\n } else {\n REQUIRE(seq == (std::vector{1, 1, 0, 0}));\n }\n}\n" }, { "path": "c++/test/utils.h", "content": "#pragma once\n\n#include \"pod5_format/read_table_utils.h\"\n#include \"test_utils.h\"\n\n#include \n#include \n\ninline pod5::RunInfoData get_test_run_info_data(\n std::string suffix = \"\",\n std::int16_t adc_center_offset = 0,\n std::int16_t sample_rate = 4000)\n{\n return pod5::RunInfoData(\n \"acquisition_id\" + suffix,\n 1005,\n 4095 + adc_center_offset,\n -4096 + adc_center_offset,\n {{\"context\" + suffix, \"tags\" + suffix},\n {\"other\" + suffix, \"tagz\" + suffix},\n {\"third\" + suffix, \"thing\" + suffix}},\n \"experiment_name\" + suffix,\n \"flow_cell_id\" + suffix,\n \"flow_cell_product_code\" + suffix,\n \"protocol_name\" + suffix,\n \"protocol_run_id\" + suffix,\n 200005,\n \"sample_id\" + suffix,\n sample_rate,\n \"sequencing_kit\" + suffix,\n \"sequencer_position\" + suffix,\n \"sequencer_position_type\" + suffix,\n \"software\" + suffix,\n \"system_name\" + suffix,\n \"system_type\" + suffix,\n {{\"tracking\" + suffix, \"id\" + suffix}});\n}\n\ninline arrow::Status remove_file_if_exists(std::string const & file)\n{\n ARROW_ASSIGN_OR_RAISE(\n auto arrow_reads_path, ::arrow::internal::PlatformFilename::FromString(file));\n ARROW_ASSIGN_OR_RAISE(bool file_exists, arrow::internal::FileExists(arrow_reads_path));\n if (file_exists) {\n ARROW_RETURN_NOT_OK(arrow::internal::DeleteFile(arrow_reads_path));\n }\n return arrow::Status::OK();\n}\n" }, { "path": "c++/test/uuid_tests.cpp", "content": "// This file contains code from https://github.com/mariusbancila/stduuid/ which has the following\n// license:\n//\n// MIT License\n//\n// Copyright (c) 2017\n//\n// Permission is hereby granted, free of charge, to any person obtaining a copy\n// of this software and associated documentation files (the \"Software\"), to deal\n// in the Software without restriction, including without limitation the rights\n// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n// copies of the Software, and to permit persons to whom the Software is\n// furnished to do so, subject to the following conditions:\n//\n// The above copyright notice and this permission notice shall be included in all\n// copies or substantial portions of the Software.\n//\n// THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n// SOFTWARE.\n\n#include \"pod5_format/uuid.h\"\n\n#include \n\n#include \n#include \n\nTEST_CASE(\"Default constructor returns nil UUID\", \"[pod5::Uuid]\")\n{\n pod5::Uuid nil;\n REQUIRE(nil.is_nil());\n}\n\nTEST_CASE(\"Default constructor produces all-zero string\", \"[pod5::Uuid]\")\n{\n pod5::Uuid nil;\n REQUIRE(to_string(nil) == \"00000000-0000-0000-0000-000000000000\");\n REQUIRE(pod5::to_string(nil) == L\"00000000-0000-0000-0000-000000000000\");\n}\n\nTEST_CASE(\"Parsing the nil UUID is nil\", \"[pod5::Uuid]\")\n{\n auto const no_braces = pod5::Uuid::from_string(\"00000000-0000-0000-0000-000000000000\");\n auto const braces = pod5::Uuid::from_string(\"{00000000-0000-0000-0000-000000000000}\");\n auto const no_braces_w = pod5::Uuid::from_string(L\"00000000-0000-0000-0000-000000000000\");\n auto const braces_w = pod5::Uuid::from_string(L\"{00000000-0000-0000-0000-000000000000}\");\n\n REQUIRE(no_braces);\n REQUIRE(no_braces->is_nil());\n REQUIRE(braces);\n REQUIRE(braces->is_nil());\n REQUIRE(no_braces_w);\n REQUIRE(no_braces_w->is_nil());\n REQUIRE(braces_w);\n REQUIRE(braces_w->is_nil());\n}\n\nTEST_CASE(\"Parsing produces the same value with or without braces\", \"[pod5::Uuid]\")\n{\n auto const no_braces = pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n auto const braces = pod5::Uuid::from_string(\"{1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7}\");\n auto const no_braces_w = pod5::Uuid::from_string(L\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n auto const braces_w = pod5::Uuid::from_string(L\"{1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7}\");\n\n REQUIRE(no_braces == braces);\n REQUIRE(no_braces_w == braces_w);\n}\n\nTEST_CASE(\"Parsing produces the same value from char or wchar_t\", \"[pod5::Uuid]\")\n{\n auto const no_braces = pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n auto const braces = pod5::Uuid::from_string(\"{1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7}\");\n auto const no_braces_w = pod5::Uuid::from_string(L\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n auto const braces_w = pod5::Uuid::from_string(L\"{1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7}\");\n\n REQUIRE(no_braces == no_braces_w);\n REQUIRE(braces == braces_w);\n}\n\nTEST_CASE(\"A parsed UUID prints the same value\", \"[pod5::Uuid]\")\n{\n auto const guid = pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n REQUIRE(guid);\n REQUIRE(to_string(*guid) == \"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n REQUIRE(pod5::to_string(*guid) == L\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\");\n}\n\nTEST_CASE(\"Invalid UUIDs cannot be parsed\", \"[pod5::Uuid]\")\n{\n REQUIRE_FALSE(pod5::Uuid::from_string(\"\"));\n REQUIRE_FALSE(pod5::Uuid::from_string(\"{}\"));\n // mismatched braces\n REQUIRE_FALSE(pod5::Uuid::from_string(\"{1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7\"));\n REQUIRE_FALSE(pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c7}\"));\n // missing a char\n REQUIRE_FALSE(pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c\"));\n // too many chars\n REQUIRE_FALSE(pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96c77\"));\n // invalid characters\n REQUIRE_FALSE(pod5::Uuid::from_string(\"1d5a3dd9-2d50-4f2b-a0fb-a3a749eb96cg\"));\n}\n\nTEST_CASE(\"Construction from iterators\", \"[pod5::Uuid]\")\n{\n using namespace std::string_literals;\n\n {\n std::array const arr{\n {0x47,\n 0x18,\n 0x38,\n 0x23,\n 0x25,\n 0x74,\n 0x4b,\n 0xfd,\n 0xb4,\n 0x11,\n 0x99,\n 0xed,\n 0x17,\n 0x7d,\n 0x3e,\n 0x43}};\n\n pod5::Uuid guid(arr.begin(), arr.end());\n REQUIRE(to_string(guid) == \"47183823-2574-4bfd-b411-99ed177d3e43\"s);\n }\n\n {\n uint8_t const arr[16] = {\n 0x47,\n 0x18,\n 0x38,\n 0x23,\n 0x25,\n 0x74,\n 0x4b,\n 0xfd,\n 0xb4,\n 0x11,\n 0x99,\n 0xed,\n 0x17,\n 0x7d,\n 0x3e,\n 0x43};\n\n pod5::Uuid guid(std::begin(arr), std::end(arr));\n REQUIRE(to_string(guid) == \"47183823-2574-4bfd-b411-99ed177d3e43\"s);\n }\n}\n\nTEST_CASE(\"Construction from arrays\", \"[pod5::Uuid]\")\n{\n using namespace std::string_literals;\n\n {\n pod5::Uuid guid{\n {0x47,\n 0x18,\n 0x38,\n 0x23,\n 0x25,\n 0x74,\n 0x4b,\n 0xfd,\n 0xb4,\n 0x11,\n 0x99,\n 0xed,\n 0x17,\n 0x7d,\n 0x3e,\n 0x43}};\n\n REQUIRE(to_string(guid) == \"47183823-2574-4bfd-b411-99ed177d3e43\"s);\n }\n\n {\n std::array const arr{\n {0x47,\n 0x18,\n 0x38,\n 0x23,\n 0x25,\n 0x74,\n 0x4b,\n 0xfd,\n 0xb4,\n 0x11,\n 0x99,\n 0xed,\n 0x17,\n 0x7d,\n 0x3e,\n 0x43}};\n\n pod5::Uuid guid(arr);\n REQUIRE(to_string(guid) == \"47183823-2574-4bfd-b411-99ed177d3e43\"s);\n }\n\n {\n uint8_t const arr[16] = {\n 0x47,\n 0x18,\n 0x38,\n 0x23,\n 0x25,\n 0x74,\n 0x4b,\n 0xfd,\n 0xb4,\n 0x11,\n 0x99,\n 0xed,\n 0x17,\n 0x7d,\n 0x3e,\n 0x43};\n\n pod5::Uuid guid(arr);\n REQUIRE(to_string(guid) == \"47183823-2574-4bfd-b411-99ed177d3e43\"s);\n }\n}\n\nTEST_CASE(\"Test equality\", \"[operators]\")\n{\n pod5::Uuid empty;\n\n auto engine = pod5::UuidRandomGenerator::engine_type{Catch::rngSeed()};\n pod5::Uuid guid = pod5::UuidRandomGenerator{engine}();\n\n REQUIRE(empty == empty);\n REQUIRE(guid == guid);\n REQUIRE(empty != guid);\n}\n\nTEST_CASE(\"Test comparison\", \"[operators]\")\n{\n auto empty = pod5::Uuid{};\n\n auto engine = pod5::UuidRandomGenerator::engine_type{Catch::rngSeed()};\n\n pod5::UuidRandomGenerator gen{engine};\n auto id = gen();\n\n REQUIRE(empty < id);\n\n std::set ids{pod5::Uuid{}, gen(), gen(), gen(), gen()};\n\n REQUIRE(ids.size() == 5);\n REQUIRE(ids.find(pod5::Uuid{}) != ids.end());\n}\n\nTEST_CASE(\"Test hashing\", \"[ops]\")\n{\n using namespace std::string_literals;\n auto str = \"47183823-2574-4bfd-b411-99ed177d3e43\"s;\n auto guid = pod5::Uuid::from_string(str).value();\n\n auto h1 = std::hash{};\n auto h2 = std::hash{};\n#ifdef UUID_HASH_STRING_BASED\n REQUIRE(h1(str) == h2(guid));\n#else\n REQUIRE(h1(str) != h2(guid));\n#endif\n\n auto engine = pod5::UuidRandomGenerator::engine_type{Catch::rngSeed()};\n pod5::UuidRandomGenerator gen{engine};\n\n std::unordered_set ids{pod5::Uuid{}, gen(), gen(), gen(), gen()};\n\n REQUIRE(ids.size() == 5);\n REQUIRE(ids.find(pod5::Uuid{}) != ids.end());\n}\n\nTEST_CASE(\"Test swap\", \"[ops]\")\n{\n pod5::Uuid empty;\n\n auto engine = pod5::UuidRandomGenerator::engine_type{Catch::rngSeed()};\n pod5::Uuid guid = pod5::UuidRandomGenerator{engine}();\n\n REQUIRE(empty.is_nil());\n REQUIRE_FALSE(guid.is_nil());\n\n std::swap(empty, guid);\n\n REQUIRE_FALSE(empty.is_nil());\n REQUIRE(guid.is_nil());\n\n empty.swap(guid);\n\n REQUIRE(empty.is_nil());\n REQUIRE_FALSE(guid.is_nil());\n}\n\nTEST_CASE(\"Test constexpr\", \"[const]\")\n{\n constexpr pod5::Uuid empty;\n static_assert(empty.is_nil());\n}\n\nTEST_CASE(\"Test size\", \"[operators]\") { REQUIRE(sizeof(pod5::Uuid) == 16); }\n" }, { "path": "ci/docker/Dockerfile.conda", "content": "FROM condaforge/mambaforge:latest\nWORKDIR /\n" }, { "path": "ci/docker/Dockerfile.py39.arm64", "content": "from git.oxfordnanolabs.local:4567/minknow/images/build-aarch64-gcc9\n\nRUN yum groupinstall \"Development Tools\" -y\nRUN yum install wget openssl-devel libffi-devel bzip2-devel -y\nRUN wget https://www.python.org/ftp/python/3.9.10/Python-3.9.10.tgz\nRUN tar xvf Python-*\nWORKDIR Python-3.9.10/\nRUN ./configure --enable-optimizations\nRUN make altinstall\nRUN rm /usr/bin/python3 && ln -s /usr/local/bin/python3.9 /usr/bin/python3\n\nWORKDIR /\n" }, { "path": "ci/docker/Dockerfile.py39.x64", "content": "from git.oxfordnanolabs.local:4567/minknow/images/build-x86_64-gcc9\n\nRUN yum groupinstall \"Development Tools\" -y\nRUN yum install wget openssl-devel libffi-devel bzip2-devel -y\nRUN wget https://www.python.org/ftp/python/3.9.10/Python-3.9.10.tgz\nRUN tar xvf Python-*\nWORKDIR Python-3.9.10/\nRUN ./configure --enable-optimizations\nRUN make altinstall\nRUN rm /usr/bin/python3 && ln -s /usr/local/bin/python3.9 /usr/bin/python3\n\nWORKDIR /\n" }, { "path": "ci/generate_coverage_report.sh", "content": "#!/bin/bash -e\n\n# Parse args.\nif [ $# -ne 1 ]; then\n echo \"Usage: $0 build_dir\"\n exit 1\nfi\nbuild_dir=$(realpath \"$1\")\n\n# Set up the venv.\necho \"Setting up venv\"\nif [ ! -e .coverage_venv ]; then\n python3 -m venv .coverage_venv\nfi\n# shellcheck disable=SC1091 # \"Not following: .coverage_venv/bin/activate was not specified as input\"\nsource .coverage_venv/bin/activate\n# --cobertura support added in 5.1.\npip install -U 'gcovr>=5.1'\n\n# Determine the root of the project.\n# Note: shellcheck wants these split up into separate lines.\nproject_root=$(realpath \"$0\")\nproject_root=$(dirname \"${project_root}\")\nproject_root=$(dirname \"${project_root}\")\ncd \"${project_root}\"\n\ngcovr_args=(\n # work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68080\n --gcov-ignore-parse-errors=negative_hits.warn\n --filter \"${project_root}/c\\+\\+\"\n)\n\nfunction generate_coverage {\n test_name=$1\n regex=$2\n\n echo \"Generating coverage report for ${test_name}\"\n\n # Clear out old coverage info.\n find \"${project_root}\" -name \"*.gcda\" -delete\n\n # Run the test.\n # shellcheck disable=SC2086 # the regex is intentionally split\n ctest --test-dir \"${build_dir}\" ${regex}\n\n # Generate the coverage report for this test.\n gcovr \"${gcovr_args[@]}\" --cobertura \"${project_root}/coverage-report-${test_name}.xml\"\n gcovr \"${gcovr_args[@]}\" --html-single-page --html-details \"${project_root}/coverage-report-${test_name}.html\"\n}\n\n# Generate a report for each test.\nfor test_name in $(ctest --test-dir \"${build_dir}\" -N | sed -rn 's/^ +Test +#[0-9]+: +(.*)$/\\1/p'); do\n generate_coverage \"${test_name}\" \"-R ^${test_name}\\$\"\ndone\n\n# Generate a full coverage report too.\ngenerate_coverage \"all\" \"\"\n\n# CI wants to see a TOTAL line in order to report coverage, so give it the one from all tests.\n# gcovr only has a resolution of 1%, so do the calculation ourselves.\ngcovr \"${gcovr_args[@]}\" | grep TOTAL | awk '{print $1, $2, $3, 100 * $3 / $2 \"%\"}'\n" }, { "path": "ci/get_tag_version.cmake", "content": "set(CANONICAL_TAG_BUILD TRUE)\ninclude(\"${CMAKE_CURRENT_LIST_DIR}/../cmake/POD5Version.cmake\")\nmessage(\"${POD5_FULL_VERSION}\")\n" }, { "path": "ci/gitlab-ci-common.yml", "content": "variables:\n CONAN_USER: nanopore\n CONAN_CHANNEL: stable\n CONAN_REFERENCE: '.'\n # Location of the .conan dir: having it in $CI_PROJECT_DIR makes it easy to grab the packages as\n # artifacts, and putting it in a job-specific subdir allows multiple packages to be unpacked\n # into a single upload job (otherwise the metadata.json files would overwrite each other)\n CONAN_USER_HOME: '${CI_PROJECT_DIR}/${CI_JOB_ID}'\n PACKAGES_PER_VERSION: 2\n # can set this instead for the total number:\n #EXPECTED_PACKAGE_COUNT: 2\n\nstages:\n - build\n - upload\n\nbefore_script:\n - conan config install --verify-ssl=no \"${CONAN_CONFIG_URL}\"\n\n#\n# use the extends keyword to inherit the job templates defined below\n#\n\n\n.parallel-cppstd:\n # A matrix definition to allow conan builds with different cppstd\n parallel:\n matrix:\n - CONAN_PROFILE_CPPSTD: [17, 20]\n\n.tarball-package: &tarball-package\n # gitlab-runner on Windows silently fails to archive files whose full path is longer than 260\n # characters; the MSYS `tar` command is not subject to this limitation (providing Windows has\n # been configured to allow long paths), so we tar up packages in the build job and untar them in\n # the upload job.\n #\n # This also allows us to only archive the package we just built, and not any of its dependencies\n # (because we can use `conan inspect` to find the name of the right packages).\n - PACKAGE_DIR=\"${CONAN_USER_HOME#${PWD}/}/.conan/data/$(conan inspect --raw name ${CONAN_REFERENCE})\"\n - echo \"Packing from $PACKAGE_DIR\"\n - tar -czvf \"conan-${CI_JOB_ID}.tar.gz\" \"$PACKAGE_DIR\"/*/${CONAN_USER}/${CONAN_CHANNEL}/{package,metadata.json}\n - rm -rf \"${CONAN_USER_HOME}/.conan\"\n\n.profile-variables: &profile-variables\n # The caller (an individual package) should have set up either PROFILE_BASE or PROFILE_BASE_HOST\n # and PROFILE_BASE_BUILD. We set variables so that both PROFILE_BASE_HOST and PROFILE_BASE_BUILD\n # are defined correctly after this call, or exit.\n - if [[ -n ${PROFILE_BASE} && ( -n ${PROFILE_BASE_HOST} || -n ${PROFILE_BASE_BUILD} ) ]]; then\n - echo \"Only one of PROFILE_BASE or (PROFILE_BASE_HOST and PROFILE_BASE_BUILD) should be defined\"\n - exit 1\n - fi\n - if [[ -n ${PROFILE_BASE} ]]; then\n - PROFILE_BASE_HOST=${PROFILE_BASE}\n - PROFILE_BASE_BUILD=${PROFILE_BASE}\n - fi\n - if [[ -z ${PROFILE_BASE_HOST} || -z ${PROFILE_BASE_BUILD} ]]; then\n - echo \"Both PROFILE_BASE_HOST and PROFILE_BASE_BUILD variables need to be defined\"\n - exit 1\n - fi\n\n.build-package:\n # The script builds all required conan packages. The caller needs to set up:\n # Either PROFILE_BASE or both PROFILE_BASE_HOST and PROFILE_BASE_BUILD\n # VERSIONS as an array if one or more version numbers are wanted.\n # EXTRA_CREATE_ARGS is passed to conan unchanged, if present.\n #\n # EXTRA_CREATE_ARGS is only used by libcurl, which builds the libcurl in parallel with c_ares\n # set to True and to False.\n #\n #\n # The after_script removes unneeded builds and sources and packages everything into a tarball,\n # artifacts defines the name and path for build artifacts.\n stage: build\n variables:\n # For Linux we need to tell arrow to not use boost.\n EXTRA_CREATE_ARGS: \"-o arrow:with_boost=False -o arrow:with_thrift=False -o arrow:parquet=False -o arrow:with_zstd=True\"\n script:\n - *profile-variables\n - |\n if [[ -n ${VERSIONS} ]]; then\n for version in ${VERSIONS}; do\n export CONAN_PROFILE_BUILD_TYPE=Debug\n conan create --profile:build ${PROFILE_BASE_BUILD} --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE} ${version}@${CONAN_USER}/${CONAN_CHANNEL} ${EXTRA_CREATE_ARGS}\n export CONAN_PROFILE_BUILD_TYPE=Release\n conan create --profile:build ${PROFILE_BASE_BUILD} --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE} ${version}@${CONAN_USER}/${CONAN_CHANNEL} ${EXTRA_CREATE_ARGS}\n done\n else\n export CONAN_PROFILE_BUILD_TYPE=Debug\n conan create --profile:build ${PROFILE_BASE_BUILD} --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE} ${CONAN_USER}/${CONAN_CHANNEL} ${EXTRA_CREATE_ARGS}\n export CONAN_PROFILE_BUILD_TYPE=Release\n conan create --profile:build ${PROFILE_BASE_BUILD} --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE} ${CONAN_USER}/${CONAN_CHANNEL} ${EXTRA_CREATE_ARGS}\n fi\n after_script:\n # Re-load the venv if it exists\n - if ls .venv/*/activate >/dev/null 2>&1; then source .venv/*/activate; fi\n - conan --version\n # Avoid storing things on the CI node unnecessarily\n - conan remove \"*\" --builds --src --force\n - *tarball-package\n artifacts:\n name: \"${CI_PROJECT_NAME}-${CI_JOB_ID}\"\n paths:\n - 'conan-*.tar.gz'\n\n.build-package-win:\n # Almost the same as build-package. Sets two additional variables CONAN_USER_HOME_SHORT and\n # CONAN_USE_ALWAYS_SHORT_PATHS. \"script\" is exactly the same as for build-package. \"after_script\"\n # does some additional processing needed for Windows between removing conan builds and sources,\n # and creating tarballs.\n extends: .build-package\n variables:\n # avoid interfering with the standard conan short-path directory\n CONAN_USER_HOME_SHORT: 'c:\\.conan-tmp'\n # we're nesting conan's data dir pretty deep, so build systems that would normally be ok can\n # fail if we don't use short paths\n CONAN_USE_ALWAYS_SHORT_PATHS: '1'\n # We need to override arrow's boost 1.85.0 requirement to match the version we use internally.\n EXTRA_CREATE_ARGS: \"-o arrow:with_thrift=False -o arrow:parquet=False --require=boost/1.86.0@ -o boost:without_locale=True\"\n\n after_script:\n # Avoid storing things on the CI node unnecessarily\n - conan remove \"*\" --builds --src --force\n # CONAN_USE_ALWAYS_SHORT_PATHS links paths deep in the data dir to dirs in c:\\.conan\n # Resolve package links (so they can be gathered into artifacts):\n - shopt -s nullglob # allow there to be nothing, eg: if CONAN_USE_ALWAYS_SHORT_PATHS is off\n # MOVE_COMMAND can be set to, say, \"cp -r\" if necessary. Moving has been seen to fail for\n # packages with executables (especially if those executables are run as part of the test\n # package), such as protobuf.\n - for link in ${CONAN_USER_HOME}/.conan/data/*/*/$CONAN_USER/$CONAN_CHANNEL/package/*/.conan_link; do\n source=$(cat $link) && ${MOVE_COMMAND:-mv} $(cygpath \"$source\")/* $(dirname $link) && rm $link;\n done\n # Clean up the short_paths folder (even on failure):\n - rm -rf \"/c/.conan-tmp\"\n - *tarball-package\n\n# This can be used to override the script stage to build both static and shared versions of a\n# library. The \"conan create\" commands are duplicates with either -o ${PACKAGE}:shared=False or\n# -o ${PACKAGE}:shared=True added. Since this doesn't use \"extends\" the caller has to extend\n# either build-package or build-package-win as well.\n.build-shared-and-static:\n script:\n - *profile-variables\n - PACKAGE=\"$(conan inspect --raw name .)\"\n - if [[ -n ${VERSIONS} ]]; then\n - for version in ${VERSIONS}; do\n - export CONAN_PROFILE_BUILD_TYPE=Debug\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${version}@${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=False\n ${EXTRA_CREATE_ARGS}\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${version}@${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=True\n ${EXTRA_CREATE_ARGS}\n - export CONAN_PROFILE_BUILD_TYPE=Release\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${version}@${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=False\n ${EXTRA_CREATE_ARGS}\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${version}@${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=True\n ${EXTRA_CREATE_ARGS}\n - done\n - else\n - export CONAN_PROFILE_BUILD_TYPE=Debug\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=False\n ${EXTRA_CREATE_ARGS}\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=True\n ${EXTRA_CREATE_ARGS}\n - export CONAN_PROFILE_BUILD_TYPE=Release\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=False\n ${EXTRA_CREATE_ARGS}\n - conan create --profile:build ${PROFILE_BASE_BUILD}\n --profile:host ${PROFILE_BASE_HOST} ${CONAN_REFERENCE}\n ${CONAN_USER}/${CONAN_CHANNEL}\n -o ${PACKAGE}:shared=True\n ${EXTRA_CREATE_ARGS}\n - fi\n\n.upload-package:\n stage: upload\n image: git.oxfordnanolabs.local:4567/traque/ont-docker-base/ont-base-python:3.8\n tags:\n - linux\n - docker\n before_script:\n - echo -e \"\\e[0Ksection_start:`date +%s`:install_conan[collapsed=true]\\r\\e[0KInstalling conan\"\n - pip install 'conan<2'\n - echo -e \"\\e[0Ksection_end:`date +%s`:install_conan\\r\\e[0K\"\n script:\n # BSD tar (on macOS) puts some extra optional information into the tarballs that GNU tar\n # complains about. --warning=no-unknown-keyword suppresses this.\n - for tarball in conan-*.tar.gz; do tar --warning=no-unknown-keyword -xf \"$tarball\"; done\n - for conan_dir in ./*/.conan; do\n - job_dir=\"$(dirname \"$conan_dir\")\"\n - echo -e \"\\e[0Ksection_start:`date +%s`:upload_package\\r\\e[0KUploading from $job_dir\"\n - export CONAN_USER_HOME=\"$PWD/$job_dir\"\n - conan config install --verify-ssl=no \"${CONAN_CONFIG_URL}\"\n - if [[ -n ${VERSIONS} ]]; then\n - expected_recipe_count=$(echo ${VERSIONS} | wc -w)\n - for version in ${VERSIONS}; do\n - conan export ${CONAN_REFERENCE} ${version}@${CONAN_USER}/${CONAN_CHANNEL}\n - done\n - else\n - expected_recipe_count=1\n - conan export ${CONAN_REFERENCE} ${CONAN_USER}/${CONAN_CHANNEL}\n - fi\n - PACKAGE=\"$(conan inspect --raw name ${CONAN_REFERENCE})\"\n - recipes=\"$(conan search --raw \"${PACKAGE}/*@${CONAN_USER}/${CONAN_CHANNEL}\")\"\n - recipe_count=\"$(echo $recipes | wc -w)\"\n - package_count=0\n - for recipe in $recipes; do\n - echo \"${recipe}:\"\n - conan search \"$recipe\"\n - package_count=$(($package_count + $(conan search \"$recipe\" | grep \"Package_ID:\" | wc -l)))\n - done\n - if [[ -z $EXPECTED_PACKAGE_COUNT ]]; then\n - EXPECTED_PACKAGE_COUNT=$((PACKAGES_PER_VERSION * expected_recipe_count))\n - fi\n - if [[ $recipe_count -ne $expected_recipe_count ]] || [[ $package_count -ne $EXPECTED_PACKAGE_COUNT ]]; then\n - echo \"Expected $expected_recipe_count recipe(s) with $EXPECTED_PACKAGE_COUNT package(s), got $recipe_count recipe(s) with $package_count package(s)\"\n - exit 1\n - fi\n # conan claims it should pick this information up automatically, given the variable names,\n # but it doesn't seem to work if you don't do this:\n - conan user -r ont-artifactory -p \"${CONAN_PASSWORD}\" \"${CONAN_LOGIN_USERNAME}\"\n - EXTRA_ARGS=\n - if [[ -z $DO_UPLOAD ]]; then\n - DO_UPLOAD=no\n - if [[ $CI_COMMIT_REF_NAME == stable/* ]] || [[ $CI_COMMIT_REF_NAME == release/* ]] || [[ $CI_COMMIT_REF_NAME == $STABLE_BRANCH_NAME ]]; then\n - DO_UPLOAD=yes\n - fi\n - fi\n - if [[ $DO_UPLOAD == \"yes\" ]]; then\n - EXTRA_ARGS=--force\n - else\n - 'echo \"WARNING: NOT uploading to artifactory for this branch\"'\n - EXTRA_ARGS=--skip-upload\n - fi\n - for recipe in $recipes; do\n - conan upload -r ont-artifactory --all --check --confirm ${EXTRA_ARGS} \"$recipe\"\n - done\n - echo -e \"\\e[0Ksection_end:`date +%s`:upload_package\\r\\e[0K\"\n - done # for conan_dir\n\n#\n# Various setup methods. Each sets a number of relevant tags, and one or two variables: For\n# non-cross compiling one variable PROFILE_BASE is set with the name of a profile which will be\n# adapted by adding \"\" or \"\". For cross compiling two variables PROFILE_BASE_BUILD\n# for the profile of the build machine and PROFILE_BASE_HOST for the host machine are set.\n#\n\n#\n# Set up for Windows versions\n#\n.profile-windows-x86_64-vs2019:\n # Set up for Windows x86 using VS 2019, using conan and the profile windows-x86_64-vs2019,\n # adapted for debug and release. To be called from individual packages by using \"extends\".\n tags:\n - windows\n - cmake\n - VS2019\n - conan\n variables:\n PROFILE_BASE: windows-x86_64-vs2019.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-windows-x86_64-vs2019-conan2:\n # Set up for Windows x86 using VS 2019, using conan and the profile windows-x86_64-vs2019,\n # adapted for debug and release. To be called from individual packages by using \"extends\".\n tags:\n - windows\n - cmake\n - VS2019\n - conan\n variables:\n CMAKE_GENERATOR: \"Visual Studio 16 2019\"\n PROFILE_BASE: windows-x86_64-vs2019.jinja\n CMAKE_PRESET: \"conan2-windows-x86_64-vs2019-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n#\n# Set up for MacOS versions\n#\n.profile-macos-aarch64-appleclang-15.0:\n # Set up for MacOS arm 64 using clang 15.0, using conan and the profile\n # macos-aarch64-appleclang-15.0, adapted for debug and release. To be called from individual\n # packages by using \"extends\".\n tags:\n - osx_arm64\n - xcode-15.3\n - conan\n variables:\n PROFILE_BASE: macos-aarch64-appleclang-15.0.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-macos-aarch64-appleclang-16.0:\n # Set up for MacOS arm 64 using clang 16.0, using conan and the profile\n # macos-aarch64-appleclang-16.0, adapted for debug and release. To be called from individual\n # packages by using \"extends\".\n tags:\n - osx_arm64\n - xcode-16.1\n - conan\n variables:\n PROFILE_BASE: macos-aarch64-appleclang-16.0.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-macos-aarch64-appleclang-15.0-conan2:\n # Set up for MacOS arm 64 using clang 15.0, using conan and the profile\n # macos-aarch64-appleclang-15.0, adapted for debug and release. To be called from individual\n # packages by using \"extends\".\n tags:\n - osx_arm64\n - xcode-15.3\n - conan\n variables:\n PROFILE_BASE: macos-aarch64-appleclang-15.0.jinja\n CMAKE_PRESET: \"conan2-macos-appleclang-15.0-aarch64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-macos-aarch64-appleclang-16.0-conan2:\n # Set up for MacOS arm 64 using clang 16.0, using conan and the profile\n # macos-aarch64-appleclang-16.0, adapted for debug and release. To be called from individual\n # packages by using \"extends\".\n tags:\n - osx_arm64\n - xcode-16.1\n - conan\n variables:\n PROFILE_BASE: macos-aarch64-appleclang-16.0.jinja\n CMAKE_PRESET: \"conan2-macos-appleclang-16.0-aarch64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n#\n# Set up for linux versions\n#\n.profile-linux-x86_64-gcc9:\n # Set up for linux x86 using gcc9, using docker and the profile linux-x86_64-gcc9, adapted\n # for debug and release. To be called from individual packages by using \"extends\".\n #\n # The docker image builds on CentOS 7 using devtoolset-9, for maximum compatibility. This means\n # the compiled code will work on any Ubuntu distro from Xenial onwards (and most other\n # still-supported Linux distros). Differences between GCC 9's libstdc++ and GCC 4.8's libstdc++\n # are handled by a static library, so no special handling of libstdc++ is required.\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc9:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc9.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc9-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc9:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc9.jinja\n CMAKE_PRESET: \"conan2-linux-gcc9-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n\n.profile-linux-x86_64-gcc11:\n # Set up for linux x86 using gcc11, using docker and the profile linux-aarch64-gcc11, adapted\n # for debug and release. To be called from individual packages by using \"extends\".\n #\n # Note that the docker image uses a GCC 11 backport to Ubuntu Bionic. Compiled artifacts will\n # be mostly compatible with Ubuntu Bionic and later, except that they will need the correct\n # libstdc++ to be available. This can be achieved by installing libstdc++6 from the GCC 11\n # backport (available in the ~ubuntu-toolchain-r/test PPA), or by otherwise shipping that\n # version of libstdc++6 in a way that the software can find it.\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc11-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11.jinja\n CMAKE_PRESET: \"conan2-linux-gcc11-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-x86_64-gcc11-asan-static-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-asan-static.jinja\n CMAKE_PRESET: \"conan2-linux-gcc11-asan-static-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-x86_64-gcc11-usan-static-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-usan-static.jinja\n CMAKE_PRESET: \"conan2-linux-gcc11-usan-static-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-x86_64-gcc11-tsan-static-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-tsan-static.jinja\n CMAKE_PRESET: \"conan2-linux-gcc11-tsan-static-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-x86_64-gcc11-asan-static:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-asan-static.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc11-ausan-static:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-ausan-static.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc11-tsan-static:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-tsan-static.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc11-usan-static:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc11:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc11-usan-static.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc13:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc13:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc13.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-x86_64-gcc13-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-x86_64-gcc13:latest\n tags:\n - linux_x86\n - docker\n variables:\n PROFILE_BASE: linux-x86_64-gcc13.jinja\n CMAKE_PRESET: \"conan2-linux-gcc13-x86_64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-x86_64-gcc13-asan-static:\n extends: .profile-linux-x86_64-gcc13\n variables:\n PROFILE_BASE: linux-x86_64-gcc13-asan-static.jinja\n\n.profile-linux-x86_64-gcc13-tsan-static:\n extends: .profile-linux-x86_64-gcc13\n variables:\n PROFILE_BASE: linux-x86_64-gcc13-tsan-static.jinja\n\n.profile-linux-x86_64-gcc13-usan-static:\n extends: .profile-linux-x86_64-gcc13\n variables:\n PROFILE_BASE: linux-x86_64-gcc13-usan-static.jinja\n\n\n.profile-linux-aarch64-gcc9:\n # Set up for linux arm64 using gcc9, using docker and the profile linux-aarch64-gcc9, adapted\n # for debug and release. To be called from individual packages by using \"extends\".\n #\n # The docker image builds on CentOS 7 using devtoolset-9, for maximum compatibility. This means\n # the compiled code will work on any Ubuntu distro from Xenial onwards (and most other\n # still-supported Linux distros). Differences between GCC 9's libstdc++ and GCC 4.8's libstdc++\n # are handled by a static library, so no special handling of libstdc++ is required.\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc9:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc9.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-aarch64-gcc9-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc9:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc9.jinja\n CMAKE_PRESET: \"conan2-linux-gcc9-aarch64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-aarch64-gcc11:\n # Set up for linux arm64 using gcc11, using docker and the profile linux-aarch64-gcc11, adapted\n # for debug and release. To be called from individual packages by using \"extends\".\n #\n # Note that the docker image uses a GCC 11 backport to Ubuntu Bionic. Compiled artifacts will\n # be mostly compatible with Ubuntu Bionic and later, except that they will need the correct\n # libstdc++ to be available. This can be achieved by installing libstdc++6 from the GCC 11\n # backport (available in the ~ubuntu-toolchain-r/test PPA), or by otherwise shipping that\n # version of libstdc++6 in a way that the software can find it.\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc11:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc11.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-aarch64-gcc11-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc11:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc11.jinja\n CMAKE_PRESET: \"conan2-linux-gcc11-aarch64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n\n.profile-linux-aarch64-gcc13:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc13:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc13.jinja\n parallel: !reference [.parallel-cppstd,parallel]\n\n.profile-linux-aarch64-gcc13-conan2:\n image: git.oxfordnanolabs.local:4567/informatics/conan-config/linux-aarch64-gcc13:latest\n tags:\n - linux_aarch64\n - docker\n variables:\n PROFILE_BASE: linux-aarch64-gcc13.jinja\n CMAKE_PRESET: \"conan2-linux-gcc13-aarch64-cppstd${CONAN_PROFILE_CPPSTD}-release\"\n" }, { "path": "ci/install.sh", "content": "#!/bin/bash\n\nset -o errexit\nset -o pipefail\nset -o nounset\n# set -o xtrace\n\n# Tar up the archive build:\n(\n cmake -DCMAKE_INSTALL_PREFIX=\"archive\" -DBUILD_TYPE=\"Release\" -DCOMPONENT=\"archive\" -P \"cmake_install.cmake\"\n if [ \"$#\" -ge 1 ] && [ \"$1\" == \"STATIC_BUILD\" ]; then\n if [[ \"$OSTYPE\" == \"linux-gnu\"* ]] && [[ -e \"archive/lib64\" ]]; then\n cp \"../build/third_party/libs\"/* \"archive/lib64\"\n else\n cp \"../build/third_party/libs\"/* \"archive/lib\"\n fi\n fi\n)\n\n# Find the wheel:\n(\n cmake -DCMAKE_INSTALL_PREFIX=\"wheel\" -DBUILD_TYPE=\"Release\" -DCOMPONENT=\"wheel\" -P \"cmake_install.cmake\"\n)\n" }, { "path": "ci/package.sh", "content": "#!/bin/bash\n\nset -o errexit\nset -o pipefail\nset -o nounset\n# set -o xtrace\n\noutput_sku=$1\nauditwheel_platform=\nif [ $# -gt 1 ]; then\n auditwheel_platform=\"${2}\"\nfi\n\nCURRENT_DIR=$(pwd)\n\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\nREPO_ROOT=\"${SCRIPT_DIR}/../\"\n\ncd \"${REPO_ROOT}\"\npod5_version=\"$(cmake -P ci/get_tag_version.cmake 2>&1)\"\n\ncd \"${CURRENT_DIR}\"\n\n# Tar up the archive build:\n(\n cd ./archive\n tar -cvzf \"${REPO_ROOT}/lib_pod5-${pod5_version}-${output_sku}.tar.gz\" .\n)\n\n# Find the wheel:\n(\n # Wheels are optional:\n if [ -d \"wheel/\" ] ; then\n cd wheel/\n if [ -z \"${auditwheel_platform}\" ]; then\n mv ./*.whl \"${REPO_ROOT}/\"\n else\n echo \"Running audit wheel\"\n pwd\n ls\n auditwheel repair ./*.whl --plat \"${auditwheel_platform}\" -w \"${REPO_ROOT}/\"\n fi\n fi\n)\n" }, { "path": "ci/unpack_libs_for_python.sh", "content": "#!/bin/bash\n\ninput_dir=$1\noutput_dir=$2\n\necho \"Unpacking builds from $input_dir to $output_dir\"\n\nfile_regex=\".*/lib_pod5-[0-9\\.]*-(.*).tar.gz\"\nfor i in \"${input_dir}\"/lib_pod5*.tar.gz; do\n\n if [[ $i =~ $file_regex ]]\n then\n sku=\"${BASH_REMATCH[1]}\"\n echo \"Extracting for SKU: $sku\"\n else\n echo \"$i doesn't match expected file pattern\" >&2\n exit 1\n fi\n\n sku_out_dir=\"$output_dir/$sku/\"\n mkdir -p \"${sku_out_dir}\"\n\n tmp_dir=\"$output_dir/tmp\"\n mkdir -p \"$tmp_dir\"\n tar -xzf \"$i\" --directory \"$output_dir/tmp\"\n\n mv \"${tmp_dir}\"/lib/* \"${sku_out_dir}\"\n\n rm -r \"$tmp_dir\"\ndone\n\necho \"unpacked skus:\"\nls \"${output_dir}/\"\necho \"contents:\"\nls \"${output_dir}\"/*\n" }, { "path": "cmake/BuildFlatBuffers.cmake", "content": "# Copyright 2015 Google Inc. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# General function to create FlatBuffer build rules for the given list of\n# schemas.\n#\n# flatbuffers_schemas: A list of flatbuffer schema files to process.\n#\n# schema_include_dirs: A list of schema file include directories, which will be\n# passed to flatc via the -I parameter.\n#\n# custom_target_name: The generated files will be added as dependencies for a\n# new custom target with this name. You should add that target as a dependency\n# for your main target to ensure these files are built. You can also retrieve\n# various properties from this target, such as GENERATED_INCLUDES_DIR,\n# BINARY_SCHEMAS_DIR, and COPY_TEXT_SCHEMAS_DIR.\n#\n# additional_dependencies: A list of additional dependencies that you'd like\n# all generated files to depend on. Pass in a blank string if you have none.\n#\n# generated_includes_dir: Where to generate the C++ header files for these\n# schemas. The generated includes directory will automatically be added to\n# CMake's include directories, and will be where generated header files are\n# placed. This parameter is optional; pass in empty string if you don't want to\n# generate include files for these schemas.\n#\n# binary_schemas_dir: If you specify an optional binary schema directory, binary\n# schemas will be generated for these schemas as well, and placed into the given\n# directory.\n#\n# copy_text_schemas_dir: If you want all text schemas (including schemas from\n# all schema include directories) copied into a directory (for example, if you\n# need them within your project to build JSON files), you can specify that\n# folder here. All text schemas will be copied to that folder.\n#\n# IMPORTANT: Make sure you quote all list arguments you pass to this function!\n# Otherwise CMake will only pass in the first element.\n# Example: build_flatbuffers(\"${fb_files}\" \"${include_dirs}\" target_name ...)\nfunction(build_flatbuffers flatbuffers_schemas\n schema_include_dirs\n custom_target_name\n additional_dependencies\n generated_includes_dir\n binary_schemas_dir\n copy_text_schemas_dir)\n\n # Test if including from FindFlatBuffers\n if(FLATBUFFERS_FLATC_EXECUTABLE)\n set(FLATC_TARGET \"\")\n set(FLATC ${FLATBUFFERS_FLATC_EXECUTABLE})\n else()\n set(FLATC_TARGET flatc)\n set(FLATC flatc)\n endif()\n set(FLATC_SCHEMA_ARGS --gen-mutable)\n if(FLATBUFFERS_FLATC_SCHEMA_EXTRA_ARGS)\n set(FLATC_SCHEMA_ARGS\n ${FLATBUFFERS_FLATC_SCHEMA_EXTRA_ARGS}\n ${FLATC_SCHEMA_ARGS}\n )\n endif()\n\n set(working_dir \"${CMAKE_CURRENT_SOURCE_DIR}\")\n\n set(schema_glob \"*.fbs\")\n # Generate the include files parameters.\n set(include_params \"\")\n set(all_generated_files \"\")\n foreach (include_dir ${schema_include_dirs})\n set(include_params -I ${include_dir} ${include_params})\n if (NOT ${copy_text_schemas_dir} STREQUAL \"\")\n # Copy text schemas from dependent folders.\n file(GLOB_RECURSE dependent_schemas ${include_dir}/${schema_glob})\n foreach (dependent_schema ${dependent_schemas})\n file(COPY ${dependent_schema} DESTINATION ${copy_text_schemas_dir})\n endforeach()\n endif()\n endforeach()\n\n foreach(schema ${flatbuffers_schemas})\n get_filename_component(filename ${schema} NAME_WE)\n # For each schema, do the things we requested.\n if (NOT ${generated_includes_dir} STREQUAL \"\")\n set(generated_include ${generated_includes_dir}/${filename}_generated.h)\n add_custom_command(\n OUTPUT ${generated_include}\n COMMAND ${FLATC} ${FLATC_SCHEMA_ARGS}\n -o ${generated_includes_dir}\n ${include_params}\n -c ${schema}\n DEPENDS ${FLATC_TARGET} ${schema} ${additional_dependencies}\n WORKING_DIRECTORY \"${working_dir}\")\n list(APPEND all_generated_files ${generated_include})\n endif()\n\n if (NOT ${binary_schemas_dir} STREQUAL \"\")\n set(binary_schema ${binary_schemas_dir}/${filename}.bfbs)\n add_custom_command(\n OUTPUT ${binary_schema}\n COMMAND ${FLATC} -b --schema\n -o ${binary_schemas_dir}\n ${include_params}\n ${schema}\n DEPENDS ${FLATC_TARGET} ${schema} ${additional_dependencies}\n WORKING_DIRECTORY \"${working_dir}\")\n list(APPEND all_generated_files ${binary_schema})\n endif()\n\n if (NOT ${copy_text_schemas_dir} STREQUAL \"\")\n file(COPY ${schema} DESTINATION ${copy_text_schemas_dir})\n endif()\n endforeach()\n\n # Create a custom target that depends on all the generated files.\n # This is the target that you can depend on to trigger all these\n # to be built.\n add_custom_target(${custom_target_name}\n DEPENDS ${all_generated_files} ${additional_dependencies})\n\n # Register the include directory we are using.\n if (NOT ${generated_includes_dir} STREQUAL \"\")\n include_directories(${generated_includes_dir})\n set_property(TARGET ${custom_target_name}\n PROPERTY GENERATED_INCLUDES_DIR\n ${generated_includes_dir})\n endif()\n\n # Register the binary schemas dir we are using.\n if (NOT ${binary_schemas_dir} STREQUAL \"\")\n set_property(TARGET ${custom_target_name}\n PROPERTY BINARY_SCHEMAS_DIR\n ${binary_schemas_dir})\n endif()\n\n # Register the text schema copy dir we are using.\n if (NOT ${copy_text_schemas_dir} STREQUAL \"\")\n set_property(TARGET ${custom_target_name}\n PROPERTY COPY_TEXT_SCHEMAS_DIR\n ${copy_text_schemas_dir})\n endif()\nendfunction()\n\n# Creates a target that can be linked against that generates flatbuffer headers.\n#\n# This function takes a target name and a list of schemas. You can also specify\n# other flagc flags using the FLAGS option to change the behavior of the flatc\n# tool.\n#\n# Arguments:\n# TARGET: The name of the target to generate.\n# SCHEMAS: The list of schema files to generate code for.\n# BINARY_SCHEMAS_DIR: Optional. The directory in which to generate binary\n# schemas. Binary schemas will only be generated if a path is provided.\n# INCLUDE: Optional. Search for includes in the specified paths. (Use this\n# instead of \"-I \" and the FLAGS option so that CMake is aware of\n# the directories that need to be searched).\n# INCLUDE_PREFIX: Optional. The directory in which to place the generated\n# files. Use this instead of the --include-prefix option.\n# FLAGS: Optional. A list of any additional flags that you would like to pass\n# to flatc.\n#\n# Example:\n#\n# flatbuffers_generate_headers(\n# TARGET my_generated_headers_target\n# INCLUDE_PREFIX ${MY_INCLUDE_PREFIX}\"\n# SCHEMAS ${MY_SCHEMA_FILES}\n# BINARY_SCHEMAS_DIR \"${MY_BINARY_SCHEMA_DIRECTORY}\"\n# FLAGS --gen-object-api)\n#\n# target_link_libraries(MyExecutableTarget\n# PRIVATE my_generated_headers_target\n# )\nfunction(flatbuffers_generate_headers)\n # Parse function arguments.\n set(options)\n set(one_value_args\n \"TARGET\"\n \"INCLUDE_PREFIX\"\n \"BINARY_SCHEMAS_DIR\")\n set(multi_value_args\n \"SCHEMAS\"\n \"INCLUDE\"\n \"FLAGS\")\n cmake_parse_arguments(\n PARSE_ARGV 0\n FLATBUFFERS_GENERATE_HEADERS\n \"${options}\"\n \"${one_value_args}\"\n \"${multi_value_args}\")\n\n # Test if including from FindFlatBuffers\n if(FLATBUFFERS_FLATC_EXECUTABLE)\n set(FLATC_TARGET \"\")\n set(FLATC ${FLATBUFFERS_FLATC_EXECUTABLE})\n else()\n set(FLATC_TARGET flatc)\n set(FLATC flatc)\n endif()\n\n set(working_dir \"${CMAKE_CURRENT_SOURCE_DIR}\")\n\n # Generate the include files parameters.\n set(include_params \"\")\n foreach (include_dir ${FLATBUFFERS_GENERATE_HEADERS_INCLUDE})\n set(include_params -I ${include_dir} ${include_params})\n endforeach()\n\n # Create a directory to place the generated code.\n set(generated_target_dir \"${CMAKE_CURRENT_BINARY_DIR}/${FLATBUFFERS_GENERATE_HEADERS_TARGET}\")\n set(generated_include_dir \"${generated_target_dir}\")\n if (NOT ${FLATBUFFERS_GENERATE_HEADERS_INCLUDE_PREFIX} STREQUAL \"\")\n set(generated_include_dir \"${generated_include_dir}/${FLATBUFFERS_GENERATE_HEADERS_INCLUDE_PREFIX}\")\n list(APPEND FLATBUFFERS_GENERATE_HEADERS_FLAGS\n \"--include-prefix\" ${FLATBUFFERS_GENERATE_HEADERS_INCLUDE_PREFIX})\n endif()\n\n # Create rules to generate the code for each schema.\n foreach(schema ${FLATBUFFERS_GENERATE_HEADERS_SCHEMAS})\n get_filename_component(filename ${schema} NAME_WE)\n set(generated_include \"${generated_include_dir}/${filename}_generated.h\")\n\n # Generate files for grpc if needed\n set(generated_source_file)\n if(\"${FLATBUFFERS_GENERATE_HEADERS_FLAGS}\" MATCHES \"--grpc\")\n # Check if schema file contain a rpc_service definition\n file(STRINGS ${schema} has_grpc REGEX \"rpc_service\")\n if(has_grpc)\n list(APPEND generated_include \"${generated_include_dir}/${filename}.grpc.fb.h\")\n set(generated_source_file \"${generated_include_dir}/${filename}.grpc.fb.cc\")\n endif()\n endif()\n\n add_custom_command(\n OUTPUT ${generated_include} ${generated_source_file}\n COMMAND ${FLATC} ${FLATC_ARGS}\n -o ${generated_include_dir}\n ${include_params}\n -c ${schema}\n ${FLATBUFFERS_GENERATE_HEADERS_FLAGS}\n DEPENDS ${FLATC_TARGET} ${schema}\n WORKING_DIRECTORY \"${working_dir}\"\n COMMENT \"Building ${schema} flatbuffers...\")\n list(APPEND all_generated_header_files ${generated_include})\n list(APPEND all_generated_source_files ${generated_source_file})\n\n # Generate the binary flatbuffers schemas if instructed to.\n if (NOT ${FLATBUFFERS_GENERATE_HEADERS_BINARY_SCHEMAS_DIR} STREQUAL \"\")\n set(binary_schema\n \"${FLATBUFFERS_GENERATE_HEADERS_BINARY_SCHEMAS_DIR}/${filename}.bfbs\")\n add_custom_command(\n OUTPUT ${binary_schema}\n COMMAND ${FLATC} -b --schema\n -o ${FLATBUFFERS_GENERATE_HEADERS_BINARY_SCHEMAS_DIR}\n ${include_params}\n ${schema}\n DEPENDS ${FLATC_TARGET} ${schema}\n WORKING_DIRECTORY \"${working_dir}\")\n list(APPEND all_generated_binary_files ${binary_schema})\n endif()\n endforeach()\n\n # Set up interface library\n add_library(${FLATBUFFERS_GENERATE_HEADERS_TARGET} INTERFACE)\n target_sources(\n ${FLATBUFFERS_GENERATE_HEADERS_TARGET}\n INTERFACE\n ${all_generated_header_files}\n ${all_generated_binary_files}\n ${all_generated_source_files}\n ${FLATBUFFERS_GENERATE_HEADERS_SCHEMAS})\n add_dependencies(\n ${FLATBUFFERS_GENERATE_HEADERS_TARGET}\n ${FLATC}\n ${FLATBUFFERS_GENERATE_HEADERS_SCHEMAS})\n target_include_directories(\n ${FLATBUFFERS_GENERATE_HEADERS_TARGET}\n INTERFACE ${generated_target_dir})\n\n # Organize file layout for IDEs.\n source_group(\n TREE \"${generated_target_dir}\"\n PREFIX \"Flatbuffers/Generated/Headers Files\"\n FILES ${all_generated_header_files})\n source_group(\n TREE \"${generated_target_dir}\"\n PREFIX \"Flatbuffers/Generated/Source Files\"\n FILES ${all_generated_source_files})\n source_group(\n TREE ${working_dir}\n PREFIX \"Flatbuffers/Schemas\"\n FILES ${FLATBUFFERS_GENERATE_HEADERS_SCHEMAS})\n if (NOT ${FLATBUFFERS_GENERATE_HEADERS_BINARY_SCHEMAS_DIR} STREQUAL \"\")\n source_group(\n TREE \"${FLATBUFFERS_GENERATE_HEADERS_BINARY_SCHEMAS_DIR}\"\n PREFIX \"Flatbuffers/Generated/Binary Schemas\"\n FILES ${all_generated_binary_files})\n endif()\nendfunction()\n\n# Creates a target that can be linked against that generates flatbuffer binaries\n# from json files.\n#\n# This function takes a target name and a list of schemas and Json files. You\n# can also specify other flagc flags and options to change the behavior of the\n# flatc compiler.\n#\n# Adding this target to your executable ensurses that the flatbuffer binaries\n# are compiled before your executable is run.\n#\n# Arguments:\n# TARGET: The name of the target to generate.\n# JSON_FILES: The list of json files to compile to flatbuffers binaries.\n# SCHEMA: The flatbuffers schema of the Json files to be compiled.\n# INCLUDE: Optional. Search for includes in the specified paths. (Use this\n# instead of \"-I \" and the FLAGS option so that CMake is aware of\n# the directories that need to be searched).\n# OUTPUT_DIR: The directly where the generated flatbuffers binaries should be\n# placed.\n# FLAGS: Optional. A list of any additional flags that you would like to pass\n# to flatc.\n#\n# Example:\n#\n# flatbuffers_generate_binary_files(\n# TARGET my_binary_data\n# SCHEMA \"${MY_SCHEMA_DIR}/my_example_schema.fbs\"\n# JSON_FILES ${MY_JSON_FILES}\n# OUTPUT_DIR \"${MY_BINARY_DATA_DIRECTORY}\"\n# FLAGS --strict-json)\n#\n# target_link_libraries(MyExecutableTarget\n# PRIVATE my_binary_data\n# )\nfunction(flatbuffers_generate_binary_files)\n # Parse function arguments.\n set(options)\n set(one_value_args\n \"TARGET\"\n \"SCHEMA\"\n \"OUTPUT_DIR\")\n set(multi_value_args\n \"JSON_FILES\"\n \"INCLUDE\"\n \"FLAGS\")\n cmake_parse_arguments(\n PARSE_ARGV 0\n FLATBUFFERS_GENERATE_BINARY_FILES\n \"${options}\"\n \"${one_value_args}\"\n \"${multi_value_args}\")\n\n # Test if including from FindFlatBuffers\n if(FLATBUFFERS_FLATC_EXECUTABLE)\n set(FLATC_TARGET \"\")\n set(FLATC ${FLATBUFFERS_FLATC_EXECUTABLE})\n else()\n set(FLATC_TARGET flatc)\n set(FLATC flatc)\n endif()\n\n set(working_dir \"${CMAKE_CURRENT_SOURCE_DIR}\")\n\n # Generate the include files parameters.\n set(include_params \"\")\n foreach (include_dir ${FLATBUFFERS_GENERATE_BINARY_FILES_INCLUDE})\n set(include_params -I ${include_dir} ${include_params})\n endforeach()\n\n # Create rules to generate the flatbuffers binary for each json file.\n foreach(json_file ${FLATBUFFERS_GENERATE_BINARY_FILES_JSON_FILES})\n get_filename_component(filename ${json_file} NAME_WE)\n set(generated_binary_file \"${FLATBUFFERS_GENERATE_BINARY_FILES_OUTPUT_DIR}/${filename}.bin\")\n add_custom_command(\n OUTPUT ${generated_binary_file}\n COMMAND ${FLATC} ${FLATC_ARGS}\n -o ${FLATBUFFERS_GENERATE_BINARY_FILES_OUTPUT_DIR}\n ${include_params}\n -b ${FLATBUFFERS_GENERATE_BINARY_FILES_SCHEMA} ${json_file}\n ${FLATBUFFERS_GENERATE_BINARY_FILES_FLAGS}\n DEPENDS ${FLATC_TARGET} ${json_file}\n WORKING_DIRECTORY \"${working_dir}\"\n COMMENT \"Building ${json_file} binary flatbuffers...\")\n list(APPEND all_generated_binary_files ${generated_binary_file})\n endforeach()\n\n # Set up interface library\n add_library(${FLATBUFFERS_GENERATE_BINARY_FILES_TARGET} INTERFACE)\n target_sources(\n ${FLATBUFFERS_GENERATE_BINARY_FILES_TARGET}\n INTERFACE\n ${all_generated_binary_files}\n ${FLATBUFFERS_GENERATE_BINARY_FILES_JSON_FILES}\n ${FLATBUFFERS_GENERATE_BINARY_FILES_SCHEMA})\n add_dependencies(\n ${FLATBUFFERS_GENERATE_BINARY_FILES_TARGET}\n ${FLATC})\n\n # Organize file layout for IDEs.\n source_group(\n TREE ${working_dir}\n PREFIX \"Flatbuffers/JSON Files\"\n FILES ${FLATBUFFERS_GENERATE_BINARY_FILES_JSON_FILES})\n source_group(\n TREE ${working_dir}\n PREFIX \"Flatbuffers/Schemas\"\n FILES ${FLATBUFFERS_GENERATE_BINARY_FILES_SCHEMA})\n source_group(\n TREE ${FLATBUFFERS_GENERATE_BINARY_FILES_OUTPUT_DIR}\n PREFIX \"Flatbuffers/Generated/Binary Files\"\n FILES ${all_generated_binary_files})\nendfunction()\n" }, { "path": "cmake/Findzstd.cmake", "content": "find_path(ZSTD_INCLUDE_DIR\n NAMES zstd.h\n PATHS\n ${CONAN_INCLUDE_DIRS_RELEASE}\n ${CONAN_INCLUDE_DIRS_DEBUG}\n)\n\nset(ZSTD_NAMES zstd zstd_static)\nset(ZSTD_NAMES_DEBUG zstdd zstd_staticd)\n\nfind_library(ZSTD_LIBRARY_RELEASE\n NAMES ${ZSTD_NAMES}\n PATHS ${CONAN_LIB_DIRS_RELEASE}\n)\nfind_library(ZSTD_LIBRARY_DEBUG\n NAMES\n ${ZSTD_NAMES_DEBUG}\n ${ZSTD_NAMES}\n PATHS ${CONAN_LIB_DIRS_DEBUG}\n)\n\ninclude(SelectLibraryConfigurations)\nselect_library_configurations(ZSTD)\n\nif(ZSTD_INCLUDE_DIR AND EXISTS \"${ZSTD_INCLUDE_DIR}/zstd.h\")\n file(STRINGS \"${ZSTD_INCLUDE_DIR}/zstd.h\" ZSTD_VERSION_MAJOR_LINE REGEX \"^#define ZSTD_VERSION_MAJOR.*$\")\n file(STRINGS \"${ZSTD_INCLUDE_DIR}/zstd.h\" ZSTD_VERSION_MINOR_LINE REGEX \"^#define ZSTD_VERSION_MINOR.*$\")\n file(STRINGS \"${ZSTD_INCLUDE_DIR}/zstd.h\" ZSTD_VERSION_RELEASE_LINE REGEX \"^#define ZSTD_VERSION_RELEASE.*$\")\n\n string(REGEX REPLACE \"^.*ZSTD_VERSION_MAJOR *([0-9]+)$\" \"\\\\1\" ZSTD_VERSION_MAJOR \"${ZSTD_VERSION_MAJOR_LINE}\")\n string(REGEX REPLACE \"^.*ZSTD_VERSION_MINOR *([0-9]+)$\" \"\\\\1\" ZSTD_VERSION_MINOR \"${ZSTD_VERSION_MINOR_LINE}\")\n string(REGEX REPLACE \"^.*ZSTD_VERSION_RELEASE *([0-9]+)$\" \"\\\\1\" ZSTD_VERSION_RELEASE \"${ZSTD_VERSION_RELEASE_LINE}\")\n\n set(ZSTD_VERSION_STRING \"${ZSTD_VERSION_MAJOR}.${ZSTD_VERSION_MINOR}.${ZSTD_VERSION_RELEASE}\")\nendif()\n\n# handle the QUIETLY and REQUIRED arguments and set ZLIB_FOUND to TRUE if\n# all listed variables are TRUE\ninclude(FindPackageHandleStandardArgs)\nfind_package_handle_standard_args(zstd REQUIRED_VARS ZSTD_LIBRARY ZSTD_INCLUDE_DIR\n VERSION_VAR ZSTD_VERSION_STRING)\n\n\nif (ZSTD_FOUND)\n set(ZSTD_INCLUDE_DIRS ${ZSTD_INCLUDE_DIR})\n\n if (NOT ZSTD_LIBRARIES)\n set(ZSTD_LIBRARIES ${ZSTD_LIBRARY})\n endif()\n\n if (NOT TARGET zstd::zstd)\n add_library(zstd::zstd UNKNOWN IMPORTED)\n set_target_properties(zstd::zstd PROPERTIES\n INTERFACE_INCLUDE_DIRECTORIES \"${ZSTD_INCLUDE_DIRS}\")\n\n if(ZSTD_LIBRARY_RELEASE)\n set_property(TARGET zstd::zstd APPEND PROPERTY\n IMPORTED_CONFIGURATIONS RELEASE)\n set_target_properties(zstd::zstd PROPERTIES\n IMPORTED_LOCATION_RELEASE \"${ZSTD_LIBRARY_RELEASE}\")\n endif()\n\n if(ZSTD_LIBRARY_DEBUG)\n set_property(TARGET zstd::zstd APPEND PROPERTY\n IMPORTED_CONFIGURATIONS DEBUG)\n set_target_properties(zstd::zstd PROPERTIES\n IMPORTED_LOCATION_DEBUG \"${ZSTD_LIBRARY_DEBUG}\")\n endif()\n\n if(NOT ZSTD_LIBRARY_RELEASE AND NOT ZSTD_LIBRARY_DEBUG)\n set_property(TARGET zstd::zstd APPEND PROPERTY\n IMPORTED_LOCATION \"${ZSTD_LIBRARY}\")\n endif()\n endif()\nendif()\n" }, { "path": "cmake/conan_provider.cmake", "content": "# The MIT License (MIT)\n#\n# Copyright (c) 2024 JFrog\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nset(CONAN_MINIMUM_VERSION 2.0.5)\n\n# Create a new policy scope and set the minimum required cmake version so the\n# features behind a policy setting like if(... IN_LIST ...) behaves as expected\n# even if the parent project does not specify a minimum cmake version or a minimum\n# version less than this module requires (e.g. 3.0) before the first project() call.\n# (see: https://cmake.org/cmake/help/latest/variable/CMAKE_PROJECT_TOP_LEVEL_INCLUDES.html)\n#\n# The policy-affecting calls like cmake_policy(SET...) or `cmake_minimum_required` only\n# affects the current policy scope, i.e. between the PUSH and POP in this case.\n#\n# https://cmake.org/cmake/help/book/mastering-cmake/chapter/Policies.html#the-policy-stack\ncmake_policy(PUSH)\ncmake_minimum_required(VERSION 3.24)\n\n\nfunction(detect_os os os_api_level os_sdk os_subsystem os_version)\n # it could be cross compilation\n message(STATUS \"CMake-Conan: cmake_system_name=${CMAKE_SYSTEM_NAME}\")\n if(CMAKE_SYSTEM_NAME AND NOT CMAKE_SYSTEM_NAME STREQUAL \"Generic\")\n if(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n set(${os} Macos PARENT_SCOPE)\n elseif(CMAKE_SYSTEM_NAME STREQUAL \"QNX\")\n set(${os} Neutrino PARENT_SCOPE)\n elseif(CMAKE_SYSTEM_NAME STREQUAL \"CYGWIN\")\n set(${os} Windows PARENT_SCOPE)\n set(${os_subsystem} cygwin PARENT_SCOPE)\n elseif(CMAKE_SYSTEM_NAME MATCHES \"^MSYS\")\n set(${os} Windows PARENT_SCOPE)\n set(${os_subsystem} msys2 PARENT_SCOPE)\n elseif(CMAKE_SYSTEM_NAME STREQUAL \"Emscripten\")\n # https://github.com/emscripten-core/emscripten/blob/4.0.6/cmake/Modules/Platform/Emscripten.cmake#L17C1-L17C34\n set(${os} Emscripten PARENT_SCOPE)\n else()\n set(${os} ${CMAKE_SYSTEM_NAME} PARENT_SCOPE)\n endif()\n if(CMAKE_SYSTEM_NAME STREQUAL \"Android\")\n if(DEFINED ANDROID_PLATFORM)\n string(REGEX MATCH \"[0-9]+\" _os_api_level ${ANDROID_PLATFORM})\n elseif(DEFINED CMAKE_SYSTEM_VERSION)\n set(_os_api_level ${CMAKE_SYSTEM_VERSION})\n endif()\n message(STATUS \"CMake-Conan: android api level=${_os_api_level}\")\n set(${os_api_level} ${_os_api_level} PARENT_SCOPE)\n endif()\n if(CMAKE_SYSTEM_NAME MATCHES \"Darwin|iOS|tvOS|watchOS\")\n # CMAKE_OSX_SYSROOT contains the full path to the SDK for MakeFile/Ninja\n # generators, but just has the original input string for Xcode.\n if(NOT IS_DIRECTORY ${CMAKE_OSX_SYSROOT})\n set(_os_sdk ${CMAKE_OSX_SYSROOT})\n else()\n if(CMAKE_OSX_SYSROOT MATCHES Simulator)\n set(apple_platform_suffix simulator)\n else()\n set(apple_platform_suffix os)\n endif()\n if(CMAKE_OSX_SYSROOT MATCHES AppleTV)\n set(_os_sdk \"appletv${apple_platform_suffix}\")\n elseif(CMAKE_OSX_SYSROOT MATCHES iPhone)\n set(_os_sdk \"iphone${apple_platform_suffix}\")\n elseif(CMAKE_OSX_SYSROOT MATCHES Watch)\n set(_os_sdk \"watch${apple_platform_suffix}\")\n endif()\n endif()\n if(DEFINED os_sdk)\n message(STATUS \"CMake-Conan: cmake_osx_sysroot=${CMAKE_OSX_SYSROOT}\")\n set(${os_sdk} ${_os_sdk} PARENT_SCOPE)\n endif()\n if(DEFINED CMAKE_OSX_DEPLOYMENT_TARGET)\n message(STATUS \"CMake-Conan: cmake_osx_deployment_target=${CMAKE_OSX_DEPLOYMENT_TARGET}\")\n set(${os_version} ${CMAKE_OSX_DEPLOYMENT_TARGET} PARENT_SCOPE)\n endif()\n endif()\n endif()\nendfunction()\n\n\nfunction(detect_arch arch)\n # CMAKE_OSX_ARCHITECTURES can contain multiple architectures, but Conan only supports one.\n # Therefore this code only finds one. If the recipes support multiple architectures, the\n # build will work. Otherwise, there will be a linker error for the missing architecture(s).\n if(DEFINED CMAKE_OSX_ARCHITECTURES)\n string(REPLACE \" \" \";\" apple_arch_list \"${CMAKE_OSX_ARCHITECTURES}\")\n list(LENGTH apple_arch_list apple_arch_count)\n if(apple_arch_count GREATER 1)\n message(WARNING \"CMake-Conan: Multiple architectures detected, this will only work if Conan recipe(s) produce fat binaries.\")\n endif()\n endif()\n if(CMAKE_SYSTEM_NAME MATCHES \"Darwin|iOS|tvOS|watchOS\" AND NOT CMAKE_OSX_ARCHITECTURES STREQUAL \"\")\n set(host_arch ${CMAKE_OSX_ARCHITECTURES})\n elseif(MSVC)\n set(host_arch ${CMAKE_CXX_COMPILER_ARCHITECTURE_ID})\n else()\n set(host_arch ${CMAKE_SYSTEM_PROCESSOR})\n endif()\n if(host_arch MATCHES \"aarch64|arm64|ARM64\")\n set(_arch armv8)\n elseif(host_arch MATCHES \"armv7|armv7-a|armv7l|ARMV7\")\n set(_arch armv7)\n elseif(host_arch MATCHES armv7s)\n set(_arch armv7s)\n elseif(host_arch MATCHES \"i686|i386|X86\")\n set(_arch x86)\n elseif(host_arch MATCHES \"AMD64|amd64|x86_64|x64\")\n set(_arch x86_64)\n endif()\n if(EMSCRIPTEN)\n # https://github.com/emscripten-core/emscripten/blob/4.0.6/cmake/Modules/Platform/Emscripten.cmake#L294C1-L294C80\n set(_arch wasm)\n endif()\n message(STATUS \"CMake-Conan: cmake_system_processor=${_arch}\")\n set(${arch} ${_arch} PARENT_SCOPE)\nendfunction()\n\n\nfunction(detect_cxx_standard cxx_standard)\n set(${cxx_standard} ${CMAKE_CXX_STANDARD} PARENT_SCOPE)\n if(CMAKE_CXX_EXTENSIONS)\n set(${cxx_standard} \"gnu${CMAKE_CXX_STANDARD}\" PARENT_SCOPE)\n endif()\nendfunction()\n\n\nmacro(detect_gnu_libstdcxx)\n # _conan_is_gnu_libstdcxx true if GNU libstdc++\n check_cxx_source_compiles(\"\n #include \n #if !defined(__GLIBCXX__) && !defined(__GLIBCPP__)\n static_assert(false);\n #endif\n int main(){}\" _conan_is_gnu_libstdcxx)\n\n # _conan_gnu_libstdcxx_is_cxx11_abi true if C++11 ABI\n check_cxx_source_compiles(\"\n #include \n static_assert(sizeof(std::string) != sizeof(void*), \\\"using libstdc++\\\");\n int main () {}\" _conan_gnu_libstdcxx_is_cxx11_abi)\n\n set(_conan_gnu_libstdcxx_suffix \"\")\n if(_conan_gnu_libstdcxx_is_cxx11_abi)\n set(_conan_gnu_libstdcxx_suffix \"11\")\n endif()\n unset (_conan_gnu_libstdcxx_is_cxx11_abi)\nendmacro()\n\n\nmacro(detect_libcxx)\n # _conan_is_libcxx true if LLVM libc++\n check_cxx_source_compiles(\"\n #include \n #if !defined(_LIBCPP_VERSION)\n static_assert(false);\n #endif\n int main(){}\" _conan_is_libcxx)\nendmacro()\n\n\nfunction(detect_lib_cxx lib_cxx)\n if(CMAKE_SYSTEM_NAME STREQUAL \"Android\")\n message(STATUS \"CMake-Conan: android_stl=${CMAKE_ANDROID_STL_TYPE}\")\n set(${lib_cxx} ${CMAKE_ANDROID_STL_TYPE} PARENT_SCOPE)\n return()\n endif()\n\n include(CheckCXXSourceCompiles)\n\n if(CMAKE_CXX_COMPILER_ID MATCHES \"GNU\")\n detect_gnu_libstdcxx()\n set(${lib_cxx} \"libstdc++${_conan_gnu_libstdcxx_suffix}\" PARENT_SCOPE)\n elseif(CMAKE_CXX_COMPILER_ID MATCHES \"AppleClang\")\n set(${lib_cxx} \"libc++\" PARENT_SCOPE)\n elseif(CMAKE_CXX_COMPILER_ID MATCHES \"Clang\" AND NOT CMAKE_SYSTEM_NAME MATCHES \"Windows\")\n # Check for libc++\n detect_libcxx()\n if(_conan_is_libcxx)\n set(${lib_cxx} \"libc++\" PARENT_SCOPE)\n return()\n endif()\n\n # Check for libstdc++\n detect_gnu_libstdcxx()\n if(_conan_is_gnu_libstdcxx)\n set(${lib_cxx} \"libstdc++${_conan_gnu_libstdcxx_suffix}\" PARENT_SCOPE)\n return()\n endif()\n\n # TODO: it would be an error if we reach this point\n elseif(CMAKE_CXX_COMPILER_ID MATCHES \"MSVC\")\n # Do nothing - compiler.runtime and compiler.runtime_type\n # should be handled separately: https://github.com/conan-io/cmake-conan/pull/516\n return()\n else()\n # TODO: unable to determine, ask user to provide a full profile file instead\n endif()\nendfunction()\n\n\nfunction(detect_compiler compiler compiler_version compiler_runtime compiler_runtime_type)\n if(DEFINED CMAKE_CXX_COMPILER_ID)\n set(_compiler ${CMAKE_CXX_COMPILER_ID})\n set(_compiler_version ${CMAKE_CXX_COMPILER_VERSION})\n else()\n if(NOT DEFINED CMAKE_C_COMPILER_ID)\n message(FATAL_ERROR \"C or C++ compiler not defined\")\n endif()\n set(_compiler ${CMAKE_C_COMPILER_ID})\n set(_compiler_version ${CMAKE_C_COMPILER_VERSION})\n endif()\n\n message(STATUS \"CMake-Conan: CMake compiler=${_compiler}\")\n message(STATUS \"CMake-Conan: CMake compiler version=${_compiler_version}\")\n\n if(_compiler MATCHES MSVC)\n set(_compiler \"msvc\")\n string(SUBSTRING ${MSVC_VERSION} 0 3 _compiler_version)\n # Configure compiler.runtime and compiler.runtime_type settings for MSVC\n if(CMAKE_MSVC_RUNTIME_LIBRARY)\n set(_msvc_runtime_library ${CMAKE_MSVC_RUNTIME_LIBRARY})\n else()\n set(_msvc_runtime_library MultiThreaded$<$:Debug>DLL) # default value documented by CMake\n endif()\n\n set(_KNOWN_MSVC_RUNTIME_VALUES \"\")\n list(APPEND _KNOWN_MSVC_RUNTIME_VALUES MultiThreaded MultiThreadedDLL)\n list(APPEND _KNOWN_MSVC_RUNTIME_VALUES MultiThreadedDebug MultiThreadedDebugDLL)\n list(APPEND _KNOWN_MSVC_RUNTIME_VALUES MultiThreaded$<$:Debug> MultiThreaded$<$:Debug>DLL)\n\n # only accept the 6 possible values, otherwise we don't don't know to map this\n if(NOT _msvc_runtime_library IN_LIST _KNOWN_MSVC_RUNTIME_VALUES)\n message(FATAL_ERROR \"CMake-Conan: unable to map MSVC runtime: ${_msvc_runtime_library} to Conan settings\")\n endif()\n\n # Runtime is \"dynamic\" in all cases if it ends in DLL\n if(_msvc_runtime_library MATCHES \".*DLL$\")\n set(_compiler_runtime \"dynamic\")\n else()\n set(_compiler_runtime \"static\")\n endif()\n message(STATUS \"CMake-Conan: CMake compiler.runtime=${_compiler_runtime}\")\n\n # Only define compiler.runtime_type when explicitly requested\n # If a generator expression is used, let Conan handle it conditional on build_type\n if(NOT _msvc_runtime_library MATCHES \":Debug>\")\n if(_msvc_runtime_library MATCHES \"Debug\")\n set(_compiler_runtime_type \"Debug\")\n else()\n set(_compiler_runtime_type \"Release\")\n endif()\n message(STATUS \"CMake-Conan: CMake compiler.runtime_type=${_compiler_runtime_type}\")\n endif()\n\n unset(_KNOWN_MSVC_RUNTIME_VALUES)\n\n elseif(_compiler MATCHES AppleClang)\n set(_compiler \"apple-clang\")\n string(REPLACE \".\" \";\" VERSION_LIST ${_compiler_version})\n list(GET VERSION_LIST 0 _compiler_version)\n elseif(_compiler MATCHES Clang)\n set(_compiler \"clang\")\n string(REPLACE \".\" \";\" VERSION_LIST ${_compiler_version})\n list(GET VERSION_LIST 0 _compiler_version)\n elseif(_compiler MATCHES GNU)\n set(_compiler \"gcc\")\n string(REPLACE \".\" \";\" VERSION_LIST ${_compiler_version})\n list(GET VERSION_LIST 0 _compiler_version)\n endif()\n\n message(STATUS \"CMake-Conan: [settings] compiler=${_compiler}\")\n message(STATUS \"CMake-Conan: [settings] compiler.version=${_compiler_version}\")\n if (_compiler_runtime)\n message(STATUS \"CMake-Conan: [settings] compiler.runtime=${_compiler_runtime}\")\n endif()\n if (_compiler_runtime_type)\n message(STATUS \"CMake-Conan: [settings] compiler.runtime_type=${_compiler_runtime_type}\")\n endif()\n\n set(${compiler} ${_compiler} PARENT_SCOPE)\n set(${compiler_version} ${_compiler_version} PARENT_SCOPE)\n set(${compiler_runtime} ${_compiler_runtime} PARENT_SCOPE)\n set(${compiler_runtime_type} ${_compiler_runtime_type} PARENT_SCOPE)\nendfunction()\n\n\nfunction(detect_build_type build_type)\n get_property(multiconfig_generator GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)\n if(NOT multiconfig_generator)\n # Only set when we know we are in a single-configuration generator\n # Note: we may want to fail early if `CMAKE_BUILD_TYPE` is not defined\n set(${build_type} ${CMAKE_BUILD_TYPE} PARENT_SCOPE)\n endif()\nendfunction()\n\n\nmacro(set_conan_compiler_if_appleclang lang command output_variable)\n if(CMAKE_${lang}_COMPILER_ID STREQUAL \"AppleClang\")\n execute_process(COMMAND xcrun --find ${command}\n OUTPUT_VARIABLE _xcrun_out OUTPUT_STRIP_TRAILING_WHITESPACE)\n cmake_path(GET _xcrun_out PARENT_PATH _xcrun_toolchain_path)\n cmake_path(GET CMAKE_${lang}_COMPILER PARENT_PATH _compiler_parent_path)\n if (\"${_xcrun_toolchain_path}\" STREQUAL \"${_compiler_parent_path}\")\n set(${output_variable} \"\")\n endif()\n unset(_xcrun_out)\n unset(_xcrun_toolchain_path)\n unset(_compiler_parent_path)\n endif()\nendmacro()\n\n\nmacro(append_compiler_executables_configuration)\n set(_conan_c_compiler \"\")\n set(_conan_cpp_compiler \"\")\n set(_conan_rc_compiler \"\")\n set(_conan_compilers_list \"\")\n if(CMAKE_C_COMPILER)\n set(_conan_c_compiler \"\\\"c\\\":\\\"${CMAKE_C_COMPILER}\\\"\")\n set_conan_compiler_if_appleclang(C cc _conan_c_compiler)\n list(APPEND _conan_compilers_list ${_conan_c_compiler})\n else()\n message(WARNING \"CMake-Conan: The C compiler is not defined. \"\n \"Please define CMAKE_C_COMPILER or enable the C language.\")\n endif()\n if(CMAKE_CXX_COMPILER)\n set(_conan_cpp_compiler \"\\\"cpp\\\":\\\"${CMAKE_CXX_COMPILER}\\\"\")\n set_conan_compiler_if_appleclang(CXX c++ _conan_cpp_compiler)\n list(APPEND _conan_compilers_list ${_conan_cpp_compiler})\n else()\n message(WARNING \"CMake-Conan: The C++ compiler is not defined. \"\n \"Please define CMAKE_CXX_COMPILER or enable the C++ language.\")\n endif()\n if(CMAKE_RC_COMPILER)\n set(_conan_rc_compiler \"\\\"rc\\\":\\\"${CMAKE_RC_COMPILER}\\\"\")\n list(APPEND _conan_compilers_list ${_conan_rc_compiler})\n # Not necessary to warn if RC not defined\n endif()\n if(NOT \"x${_conan_compilers_list}\" STREQUAL \"x\")\n string(REPLACE \";\" \",\" _conan_compilers_list \"${_conan_compilers_list}\")\n string(APPEND profile \"tools.build:compiler_executables={${_conan_compilers_list}}\\n\")\n endif()\n unset(_conan_c_compiler)\n unset(_conan_cpp_compiler)\n unset(_conan_rc_compiler)\n unset(_conan_compilers_list)\nendmacro()\n\n\nfunction(detect_host_profile output_file)\n detect_os(os os_api_level os_sdk os_subsystem os_version)\n detect_arch(arch)\n detect_compiler(compiler compiler_version compiler_runtime compiler_runtime_type)\n detect_cxx_standard(compiler_cppstd)\n detect_lib_cxx(compiler_libcxx)\n detect_build_type(build_type)\n\n set(profile \"\")\n string(APPEND profile \"[settings]\\n\")\n if(arch)\n string(APPEND profile arch=${arch} \"\\n\")\n endif()\n if(os)\n string(APPEND profile os=${os} \"\\n\")\n endif()\n if(os_api_level)\n string(APPEND profile os.api_level=${os_api_level} \"\\n\")\n endif()\n if(os_version)\n string(APPEND profile os.version=${os_version} \"\\n\")\n endif()\n if(os_sdk)\n string(APPEND profile os.sdk=${os_sdk} \"\\n\")\n endif()\n if(os_subsystem)\n string(APPEND profile os.subsystem=${os_subsystem} \"\\n\")\n endif()\n if(compiler)\n string(APPEND profile compiler=${compiler} \"\\n\")\n endif()\n if(compiler_version)\n string(APPEND profile compiler.version=${compiler_version} \"\\n\")\n endif()\n if(compiler_runtime)\n string(APPEND profile compiler.runtime=${compiler_runtime} \"\\n\")\n endif()\n if(compiler_runtime_type)\n string(APPEND profile compiler.runtime_type=${compiler_runtime_type} \"\\n\")\n endif()\n if(compiler_cppstd)\n string(APPEND profile compiler.cppstd=${compiler_cppstd} \"\\n\")\n endif()\n if(compiler_libcxx)\n string(APPEND profile compiler.libcxx=${compiler_libcxx} \"\\n\")\n endif()\n if(build_type)\n string(APPEND profile \"build_type=${build_type}\\n\")\n endif()\n\n if(NOT DEFINED output_file)\n set(file_name \"${CMAKE_BINARY_DIR}/profile\")\n else()\n set(file_name ${output_file})\n endif()\n\n string(APPEND profile \"[conf]\\n\")\n string(APPEND profile \"tools.cmake.cmaketoolchain:generator=${CMAKE_GENERATOR}\\n\")\n\n # propagate compilers via profile\n append_compiler_executables_configuration()\n\n if(os STREQUAL \"Android\")\n string(APPEND profile \"tools.android:ndk_path=${CMAKE_ANDROID_NDK}\\n\")\n endif()\n\n message(STATUS \"CMake-Conan: Creating profile ${file_name}\")\n file(WRITE ${file_name} ${profile})\n message(STATUS \"CMake-Conan: Profile: \\n${profile}\")\nendfunction()\n\n\nfunction(conan_profile_detect_default)\n message(STATUS \"CMake-Conan: Checking if a default profile exists\")\n execute_process(COMMAND ${CONAN_COMMAND} profile path default\n RESULT_VARIABLE return_code\n OUTPUT_VARIABLE conan_stdout\n ERROR_VARIABLE conan_stderr\n ECHO_ERROR_VARIABLE # show the text output regardless\n ECHO_OUTPUT_VARIABLE\n WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n if(NOT ${return_code} EQUAL \"0\")\n message(STATUS \"CMake-Conan: The default profile doesn't exist, detecting it.\")\n execute_process(COMMAND ${CONAN_COMMAND} profile detect\n RESULT_VARIABLE return_code\n OUTPUT_VARIABLE conan_stdout\n ERROR_VARIABLE conan_stderr\n ECHO_ERROR_VARIABLE # show the text output regardless\n ECHO_OUTPUT_VARIABLE\n WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n endif()\nendfunction()\n\n\nfunction(conan_install)\n cmake_parse_arguments(ARGS conan_args ${ARGN})\n set(conan_output_folder ${CMAKE_BINARY_DIR}/conan)\n # Invoke \"conan install\" with the provided arguments\n set(conan_args ${conan_args} -of=${conan_output_folder})\n message(STATUS \"CMake-Conan: conan install ${CMAKE_SOURCE_DIR} ${conan_args} ${ARGN}\")\n\n\n # In case there was not a valid cmake executable in the PATH, we inject the\n # same we used to invoke the provider to the PATH\n if(DEFINED PATH_TO_CMAKE_BIN)\n set(old_path $ENV{PATH})\n set(ENV{PATH} \"$ENV{PATH}:${PATH_TO_CMAKE_BIN}\")\n endif()\n\n execute_process(COMMAND ${CONAN_COMMAND} install ${CMAKE_SOURCE_DIR} ${conan_args} ${ARGN} --format=json\n RESULT_VARIABLE return_code\n OUTPUT_VARIABLE conan_stdout\n ERROR_VARIABLE conan_stderr\n ECHO_ERROR_VARIABLE # show the text output regardless\n WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n\n if(DEFINED PATH_TO_CMAKE_BIN)\n set(ENV{PATH} \"${old_path}\")\n endif()\n\n if(NOT \"${return_code}\" STREQUAL \"0\")\n message(FATAL_ERROR \"Conan install failed='${return_code}'\")\n endif()\n\n # the files are generated in a folder that depends on the layout used, if\n # one is specified, but we don't know a priori where this is.\n # TODO: this can be made more robust if Conan can provide this in the json output\n string(JSON conan_generators_folder GET \"${conan_stdout}\" graph nodes 0 generators_folder)\n cmake_path(CONVERT ${conan_generators_folder} TO_CMAKE_PATH_LIST conan_generators_folder)\n\n message(STATUS \"CMake-Conan: CONAN_GENERATORS_FOLDER=${conan_generators_folder}\")\n set_property(GLOBAL PROPERTY CONAN_GENERATORS_FOLDER \"${conan_generators_folder}\")\n # reconfigure on conanfile changes\n string(JSON conanfile GET \"${conan_stdout}\" graph nodes 0 label)\n message(STATUS \"CMake-Conan: CONANFILE=${CMAKE_SOURCE_DIR}/${conanfile}\")\n set_property(DIRECTORY ${CMAKE_SOURCE_DIR} APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS \"${CMAKE_SOURCE_DIR}/${conanfile}\")\n # success\n set_property(GLOBAL PROPERTY CONAN_INSTALL_SUCCESS TRUE)\n\nendfunction()\n\n\nfunction(conan_get_version conan_command conan_current_version)\n execute_process(\n COMMAND ${conan_command} --version\n OUTPUT_VARIABLE conan_output\n RESULT_VARIABLE conan_result\n OUTPUT_STRIP_TRAILING_WHITESPACE\n )\n if(conan_result)\n message(FATAL_ERROR \"CMake-Conan: Error when trying to run Conan\")\n endif()\n\n string(REGEX MATCH \"[0-9]+\\\\.[0-9]+\\\\.[0-9]+\" conan_version ${conan_output})\n set(${conan_current_version} ${conan_version} PARENT_SCOPE)\nendfunction()\n\n\nfunction(conan_version_check)\n set(options )\n set(one_value_args MINIMUM CURRENT)\n set(multi_value_args )\n cmake_parse_arguments(conan_version_check\n \"${options}\" \"${one_value_args}\" \"${multi_value_args}\" ${ARGN})\n\n if(NOT conan_version_check_MINIMUM)\n message(FATAL_ERROR \"CMake-Conan: Required parameter MINIMUM not set!\")\n endif()\n if(NOT conan_version_check_CURRENT)\n message(FATAL_ERROR \"CMake-Conan: Required parameter CURRENT not set!\")\n endif()\n\n if(conan_version_check_CURRENT VERSION_LESS conan_version_check_MINIMUM)\n message(FATAL_ERROR \"CMake-Conan: Conan version must be ${conan_version_check_MINIMUM} or later\")\n endif()\nendfunction()\n\n\nmacro(construct_profile_argument argument_variable profile_list)\n set(${argument_variable} \"\")\n if(\"${profile_list}\" STREQUAL \"CONAN_HOST_PROFILE\")\n set(_arg_flag \"--profile:host=\")\n elseif(\"${profile_list}\" STREQUAL \"CONAN_BUILD_PROFILE\")\n set(_arg_flag \"--profile:build=\")\n endif()\n\n set(_profile_list \"${${profile_list}}\")\n list(TRANSFORM _profile_list REPLACE \"auto-cmake\" \"${CMAKE_BINARY_DIR}/conan_host_profile\")\n list(TRANSFORM _profile_list PREPEND ${_arg_flag})\n set(${argument_variable} ${_profile_list})\n\n unset(_arg_flag)\n unset(_profile_list)\nendmacro()\n\n\nmacro(conan_provide_dependency method package_name)\n set_property(GLOBAL PROPERTY CONAN_PROVIDE_DEPENDENCY_INVOKED TRUE)\n get_property(_conan_install_success GLOBAL PROPERTY CONAN_INSTALL_SUCCESS)\n if(NOT _conan_install_success)\n find_program(CONAN_COMMAND \"conan\" REQUIRED)\n conan_get_version(${CONAN_COMMAND} CONAN_CURRENT_VERSION)\n conan_version_check(MINIMUM ${CONAN_MINIMUM_VERSION} CURRENT ${CONAN_CURRENT_VERSION})\n message(STATUS \"CMake-Conan: first find_package() found. Installing dependencies with Conan\")\n if(\"default\" IN_LIST CONAN_HOST_PROFILE OR \"default\" IN_LIST CONAN_BUILD_PROFILE)\n conan_profile_detect_default()\n endif()\n if(\"auto-cmake\" IN_LIST CONAN_HOST_PROFILE)\n detect_host_profile(${CMAKE_BINARY_DIR}/conan_host_profile)\n endif()\n construct_profile_argument(_host_profile_flags CONAN_HOST_PROFILE)\n construct_profile_argument(_build_profile_flags CONAN_BUILD_PROFILE)\n if(EXISTS \"${CMAKE_SOURCE_DIR}/conanfile.py\")\n file(READ \"${CMAKE_SOURCE_DIR}/conanfile.py\" outfile)\n if(NOT \"${outfile}\" MATCHES \".*CMakeDeps.*\")\n message(WARNING \"Cmake-conan: CMakeDeps generator was not defined in the conanfile\")\n endif()\n set(generator \"\")\n elseif (EXISTS \"${CMAKE_SOURCE_DIR}/conanfile.txt\")\n file(READ \"${CMAKE_SOURCE_DIR}/conanfile.txt\" outfile)\n if(NOT \"${outfile}\" MATCHES \".*CMakeDeps.*\")\n message(WARNING \"Cmake-conan: CMakeDeps generator was not defined in the conanfile. \"\n \"Please define the generator as it will be mandatory in the future\")\n endif()\n set(generator \"-g;CMakeDeps\")\n endif()\n get_property(_multiconfig_generator GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)\n if(NOT _multiconfig_generator)\n message(STATUS \"CMake-Conan: Installing single configuration ${CMAKE_BUILD_TYPE}\")\n conan_install(${_host_profile_flags} ${_build_profile_flags} ${CONAN_INSTALL_ARGS} ${generator})\n else()\n message(STATUS \"CMake-Conan: Installing both Debug and Release\")\n conan_install(${_host_profile_flags} ${_build_profile_flags} -s build_type=Release ${CONAN_INSTALL_ARGS} ${generator})\n conan_install(${_host_profile_flags} ${_build_profile_flags} -s build_type=Debug ${CONAN_INSTALL_ARGS} ${generator})\n endif()\n unset(_host_profile_flags)\n unset(_build_profile_flags)\n unset(_multiconfig_generator)\n unset(_conan_install_success)\n else()\n message(STATUS \"CMake-Conan: find_package(${ARGV1}) found, 'conan install' already ran\")\n unset(_conan_install_success)\n endif()\n\n get_property(_conan_generators_folder GLOBAL PROPERTY CONAN_GENERATORS_FOLDER)\n\n # Ensure that we consider Conan-provided packages ahead of any other,\n # irrespective of other settings that modify the search order or search paths\n # This follows the guidelines from the find_package documentation\n # (https://cmake.org/cmake/help/latest/command/find_package.html):\n # find_package ( PATHS paths... NO_DEFAULT_PATH)\n # find_package ()\n\n # Filter out `REQUIRED` from the argument list, as the first call may fail\n set(_find_args_${package_name} \"${ARGN}\")\n list(REMOVE_ITEM _find_args_${package_name} \"REQUIRED\")\n if(NOT \"MODULE\" IN_LIST _find_args_${package_name})\n find_package(${package_name} ${_find_args_${package_name}} BYPASS_PROVIDER PATHS \"${_conan_generators_folder}\" NO_DEFAULT_PATH NO_CMAKE_FIND_ROOT_PATH)\n unset(_find_args_${package_name})\n endif()\n\n # Invoke find_package a second time - if the first call succeeded,\n # this will simply reuse the result. If not, fall back to CMake default search\n # behaviour, also allowing modules to be searched.\n if(NOT ${package_name}_FOUND)\n list(FIND CMAKE_MODULE_PATH \"${_conan_generators_folder}\" _index)\n if(_index EQUAL -1)\n list(PREPEND CMAKE_MODULE_PATH \"${_conan_generators_folder}\")\n endif()\n unset(_index)\n find_package(${package_name} ${ARGN} BYPASS_PROVIDER)\n list(REMOVE_ITEM CMAKE_MODULE_PATH \"${_conan_generators_folder}\")\n endif()\nendmacro()\n\n\ncmake_language(\n SET_DEPENDENCY_PROVIDER conan_provide_dependency\n SUPPORTED_METHODS FIND_PACKAGE\n)\n\n\nmacro(conan_provide_dependency_check)\n set(_conan_provide_dependency_invoked FALSE)\n get_property(_conan_provide_dependency_invoked GLOBAL PROPERTY CONAN_PROVIDE_DEPENDENCY_INVOKED)\n if(NOT _conan_provide_dependency_invoked)\n message(WARNING \"Conan is correctly configured as dependency provider, \"\n \"but Conan has not been invoked. Please add at least one \"\n \"call to `find_package()`.\")\n if(DEFINED CONAN_COMMAND)\n # suppress warning in case `CONAN_COMMAND` was specified but unused.\n set(_conan_command ${CONAN_COMMAND})\n unset(_conan_command)\n endif()\n endif()\n unset(_conan_provide_dependency_invoked)\nendmacro()\n\n\n# Add a deferred call at the end of processing the top-level directory\n# to check if the dependency provider was invoked at all.\ncmake_language(DEFER DIRECTORY \"${CMAKE_SOURCE_DIR}\" CALL conan_provide_dependency_check)\n\n# Configurable variables for Conan profiles\nset(CONAN_HOST_PROFILE \"default;auto-cmake\" CACHE STRING \"Conan host profile\")\nset(CONAN_BUILD_PROFILE \"default\" CACHE STRING \"Conan build profile\")\nset(CONAN_INSTALL_ARGS \"--build=missing\" CACHE STRING \"Command line arguments for conan install\")\n\nfind_program(_cmake_program NAMES cmake NO_PACKAGE_ROOT_PATH NO_CMAKE_PATH NO_CMAKE_ENVIRONMENT_PATH NO_CMAKE_SYSTEM_PATH NO_CMAKE_FIND_ROOT_PATH)\nif(NOT _cmake_program)\n get_filename_component(PATH_TO_CMAKE_BIN \"${CMAKE_COMMAND}\" DIRECTORY)\n set(PATH_TO_CMAKE_BIN \"${PATH_TO_CMAKE_BIN}\" CACHE INTERNAL \"Path where the CMake executable is\")\nendif()\n\ncmake_policy(POP)\n" }, { "path": "cmake/pod5_fuzz.cmake", "content": "if (NOT CMAKE_CXX_COMPILER_ID MATCHES \"Clang\")\n message(FATAL_ERROR\n \"Only LLVM based compilers are supported for fuzzing. Assuming that \"\n \"'clang' is install, it can be picked by setting the environment \"\n \"variables 'CC=clang' and 'CXX=clang++' before invoking cmake.\"\n )\nendif()\n\n# Build everything with fuzzing instrumentation and sanitizers\nset(POD5_SANITIZER_FLAGS -fsanitize=address,undefined,fuzzer-no-link)\nadd_compile_options(-g ${POD5_SANITIZER_FLAGS} -UNDEBUG -O1)\nadd_link_options(${POD5_SANITIZER_FLAGS})\n" }, { "path": "cmake/pod5_packaging.cmake", "content": "\nset(CPACK_PACKAGE_NAME \"lib_pod5\")\nset(CPACK_PACKAGE_VENDOR \"Oxford Nanopore\")\nset(CPACK_VERBATIM_VARIABLES true)\nset(CPACK_PACKAGE_VERSION_MAJOR ${POD5_VERSION_MAJOR})\nset(CPACK_PACKAGE_VERSION_MINOR ${POD5_VERSION_MINOR})\nset(CPACK_PACKAGE_VERSION_PATCH ${POD5_VERSION_REV})\nset(CPACK_RESOURCE_FILE_LICENSE \"${CMAKE_SOURCE_DIR}/LICENSE.md\")\n\ninclude(CPack)\n" }, { "path": "cmake/presets/conan-build-options.json", "content": "{\n \"version\": 4,\n \"configurePresets\": [\n {\n \"name\": \"conan2-debug\",\n \"binaryDir\": \"${sourceDir}/build\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CMAKE_BUILD_TYPE\": \"Debug\"\n },\n \"environment\": {\n \"CONAN_PROFILE_BUILD_TYPE\": \"Debug\"\n }\n },\n {\n \"name\": \"conan2-release\",\n \"binaryDir\": \"${sourceDir}/build\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CMAKE_BUILD_TYPE\": \"Release\"\n },\n \"environment\": {\n \"CONAN_PROFILE_BUILD_TYPE\": \"Release\"\n }\n },\n {\n \"name\": \"conan2-cppstd20\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CMAKE_CXX_STANDARD\": \"20\"\n },\n \"environment\": {\n \"CONAN_PROFILE_CPPSTD\": \"20\"\n }\n },\n {\n \"name\": \"conan2-cppstd17\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CMAKE_CXX_STANDARD\": \"17\"\n },\n \"environment\": {\n \"CONAN_PROFILE_CPPSTD\": \"17\"\n }\n }\n ]\n}\n" }, { "path": "cmake/presets/conan-profiles.json", "content": "{\n \"version\": 4,\n \"configurePresets\": [\n {\n \"name\": \"conan2-gcc9-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc9.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc9.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc11-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc11.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc11.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc11-asan-static-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc11-asan-static.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc11-asan-static.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc11-usan-static-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc11-usan-static.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc11-usan-static.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc11-tsan-static-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc11-tsan-static.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc11-tsan-static.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc13-x86_64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-x86_64-gcc13.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-x86_64-gcc13.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc13-aarch64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-aarch64-gcc13.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-aarch64-gcc13.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc11-aarch64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-aarch64-gcc11.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-aarch64-gcc11.jinja\"\n }\n },\n {\n \"name\": \"conan2-gcc9-aarch64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"linux-aarch64-gcc9.jinja\",\n \"CONAN_BUILD_PROFILE\": \"linux-aarch64-gcc9.jinja\"\n }\n },\n {\n \"name\": \"conan2-appleclang-15.0-aarch64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"macos-aarch64-appleclang-15.0.jinja\",\n \"CONAN_BUILD_PROFILE\": \"macos-aarch64-appleclang-15.0.jinja\"\n }\n },\n {\n \"name\": \"conan2-appleclang-16.0-aarch64-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"macos-aarch64-appleclang-16.0.jinja\",\n \"CONAN_BUILD_PROFILE\": \"macos-aarch64-appleclang-16.0.jinja\"\n }\n },\n {\n \"name\": \"conan2-windows-x86_64-vs2019-profile\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CONAN_HOST_PROFILE\": \"windows-x86_64-vs2019.jinja\",\n \"CONAN_BUILD_PROFILE\": \"windows-x86_64-vs2019.jinja\"\n },\n \"environment\": {\n \"CMAKE_GENERATOR\": \"Visual Studio 16 2019\"\n }\n }\n ]\n}\n" }, { "path": "cmake/presets/conan-provider.json", "content": "{\n \"version\": 4,\n \"configurePresets\": [\n {\n \"name\": \"conan2-provider\",\n \"hidden\": true,\n \"cacheVariables\": {\n \"CMAKE_PROJECT_TOP_LEVEL_INCLUDES\": \"${sourceDir}/cmake/conan_provider.cmake\",\n \"CONAN2\": \"ON\",\n \"CONAN_INSTALL_ARGS\":\n \"--build=never;-o:a=arrow/*:with_boost=False;-o:a=arrow/*:with_thrift=False;-o:a=arrow/*:parquet=False;-o:a=arrow/*:with_zstd=True;\"\n }\n }\n ]\n}\n" }, { "path": "conanfile.py", "content": "from conan import ConanFile\nfrom conan.tools.cmake import CMakeToolchain, CMake, CMakeDeps\nfrom conan.tools.files import collect_libs, copy\nfrom conan.tools.build import cross_building\nfrom conan.tools.cmake import cmake_layout\nimport os\n\n\nclass Pod5Conan(ConanFile):\n name = \"pod5_file_format\"\n license = \"MPL 2.0\"\n url = \"https://github.com/nanoporetech/pod5-file-format\"\n description = \"POD5 File format\"\n topics = \"nanopore\", \"sequencing\", \"genomic\", \"dna\", \"arrow\"\n settings = \"os\", \"compiler\", \"build_type\", \"arch\"\n options = {\"shared\": [True, False]}\n default_options = {\n \"shared\": False,\n }\n exports_sources = [\n \"c++/*\",\n \"cmake/*\",\n \"python/*\",\n \"third_party/*\",\n \"CMakeLists.txt\",\n \"LICENSE.md\",\n ]\n\n \"\"\"\n When building a static library, we need to pack arrow, zstd and if on linux jemalloc,\n alongside pod5 static lib to avoid linking errors. This function copies those libs to\n a folder called third_party in the build directory. The ci/install.sh ensures they end\n up in the correct location to be deployed, if install is done via cmake.\n \"\"\"\n\n def _setup_third_party_deps_packaging(self):\n deps_to_pack = (\n [\"arrow\", \"zstd\", \"jemalloc\"]\n if self.settings.os == \"Linux\"\n else [\"arrow\", \"zstd\"]\n )\n static_lib_ext_wildcard = \"*.a\" if self.settings.os != \"Windows\" else \"*.lib\"\n for dep in deps_to_pack:\n if dep == \"jemalloc\":\n static_lib_ext_wildcard = (\n \"*_pic.a\" if self.settings.os != \"Windows\" else \"*_pic.lib\"\n )\n dep_object = self.dependencies[dep]\n copy(\n self,\n static_lib_ext_wildcard,\n dep_object.cpp_info.libdir,\n f\"{self.build_folder}/third_party/libs\",\n )\n\n def _licenses_path(self):\n # This needs to match the install step inside CMake.\n return os.path.join(self.build_folder, \"pod5_conan_licenses\")\n\n def _copy_licenses(self):\n # Copy each dependency's licenses.\n for require, dependency in self.dependencies.items():\n # package_folder will be None if this dependency isn't used.\n if dependency.package_folder is not None:\n copy(\n self,\n \"license*\",\n dependency.package_folder,\n os.path.join(self._licenses_path(), dependency.ref.name),\n ignore_case=True,\n )\n\n def layout(self):\n cmake_layout(self, \"Ninja Multi-Config\")\n\n def requirements(self):\n self.requires(\"arrow/18.0.0\")\n self.requires(\"flatbuffers/2.0.0\")\n self.requires(\"zstd/[>=1.4.8 <=2.0.0]\")\n self.requires(\"zlib/[>=1.2.11 <=2.0.0]\")\n if not (\n self.settings.os == \"Windows\"\n or self.settings.os == \"Macos\"\n or self.settings.os == \"iOS\"\n ):\n self.requires(\"jemalloc/5.2.1\")\n\n \"\"\"\n When cross compiling we need pre compiled flatbuffers for flatc to run on the build machine\n which is not the target.\n The flatbuffers version is most likely available already; it is on the master branch and\n quite likely already built on the development branch. However, it seems that conan\n doesn't realise this since it is the same package that it tries to build, even though it\n is a different revision, flatbuffers on the other hand is downloaded.\n \"\"\"\n\n def build_requirements(self):\n if hasattr(self, \"settings_build\") and cross_building(self):\n # We are using an older version of flatbuffers not available on CCI.\n # @TODO: Update to a version that exists in CCI\n # When this line changes a corresponding change in .gitlab-ci.yml is required where this\n # package is uninstalled.\n self.tool_requires(\"flatbuffers/2.0.0\")\n\n def generate(self):\n if not self.options.shared:\n self._setup_third_party_deps_packaging()\n\n self._copy_licenses()\n\n tc = CMakeToolchain(self)\n tc.variables[\"ENABLE_CONAN\"] = \"ON\"\n tc.variables[\"BUILD_PYTHON_WHEEL\"] = \"OFF\"\n tc.variables[\"POD5_DISABLE_TESTS\"] = \"ON\"\n tc.variables[\"POD5_BUILD_EXAMPLES\"] = \"OFF\"\n tc.variables[\"BUILD_SHARED_LIB\"] = \"ON\" if self.options.shared else \"OFF\"\n\n tc.generate()\n\n deps = CMakeDeps(self)\n deps.check_components_exist = True\n\n # This ensures that target names in cmake would be in the form of libname::libname\n deps.set_property(\"zstd\", \"cmake_target_name\", None)\n deps.generate()\n\n def build(self):\n cmake = CMake(self)\n cmake.configure()\n cmake.build()\n\n def package(self):\n cmake = CMake(self)\n cmake.install()\n\n # Copy the license files\n copy(\n self,\n \"*\",\n self._licenses_path(),\n os.path.join(self.package_folder, \"licenses\"),\n )\n\n # Package the required third party libs after installing pod5 static\n if not self.options.shared:\n src = f\"{self.build_folder}/third_party/libs/\"\n dst = f\"{self.build_folder}/lib/\"\n copy(self, \"*\", src, dst)\n\n def package_info(self):\n # Note: package_info collects information in self.cpp_info. It is called from the Conan\n # application.\n #\n # This call is made immediately after the pre_package_info hook and before the\n # post_package_info hook. To get more information, we can \"import traceback\" and \"import inspect\",\n # then call traceback.print_stack() to print the complete call stack, or examine\n # inspect.stack().\n #\n # The caller has created self.cpp_info with the name set to the name of self, with a rootpath,\n # version and description from self, env_info and user_info set with default values,\n # public_deps set to an array with the names of public requirements in conanfile.requires.items.\n\n # Additions for this package. Note that everything in requirements needs to be mentioned\n # here. Except for Windows and Macos, jemalloc is also needed.\n\n self.cpp_info.libs = collect_libs(self)\n self.cpp_info.requires = [\n \"arrow::arrow\",\n \"flatbuffers::flatbuffers\",\n \"zstd::zstd\",\n \"zlib::zlib\",\n ]\n\n # Workaround for broken Arrow package - ensure transitive includes are available\n # Since our headers include Arrow headers, we need Arrow's includes to be transitively available\n try:\n arrow_dep = self.dependencies[\"arrow\"]\n arrow_include_path = os.path.join(arrow_dep.package_folder, \"include\")\n if os.path.exists(arrow_include_path):\n self.cpp_info.includedirs.append(arrow_include_path)\n except Exception:\n # Arrow dependency not found or other issue - let it fail naturally\n pass\n\n # self.cpp\n if self.settings.os == \"Linux\":\n self.cpp_info.requires.append(\"jemalloc::jemalloc\")\n\n if self.settings.os in [\"iOS\", \"Macos\"]:\n self.cpp_info.frameworks = [\"CoreFoundation\"]\n" }, { "path": "docs/DESIGN.md", "content": "POD5 File Format Design Details\n==============================\n\n## Summary\n\nThis file format has the following design goals (roughly in priority order):\n\n- Good write performance for MinKNOW\n- Recoverable if the writing process crashes\n- Good read performance for downstream tools, including basecall model generation\n- Efficient use of space\n- Straightforward to implement and maintain\n- Extensibility\n\nNote that trade-offs have been made between these goals, but we have mostly aimed to make those run-time decisions.\n\nWe have also chosen not to optimise for editing existing files.\n\n\n### Write performance\n\nThe aspects of this format that are designed to maximise write performance are:\n\n- Data can be written sequentially\n - The sequential access pattern makes it easy to use efficient operating system APIs (such as io_uring on Linux)\n - The sequential access pattern helps the operating system's I/O scheduler maximise throughput\n- Signal data from different reads can be interleaved, and data streams can be safely abandoned (at the cost of using more space than necessary)\n - This allows MinKNOW to write out data as it arrives, potentially avoiding the need have an intermediate caching format (this file format can be used for the cache and the final output)\n- Support for space- and CPU-efficient compression routines (VBZ)\n - This reduces the amount of data that needs to be written, which reduces I/O load\n\n### Recovery\n\nThe aspects of this format that are designed to allow for recovery if the writing process crashes are:\n\n- A way to indicate that a file is actually complete as intended (complete files end with a recognisable footer)\n- The Apache Feather format can be assembled by reading it sequentially, without using the footer\n- The data file format is append-only, which means that once data is recorded it cannot be corrupted by later updates\n\n### Read performance\n\nThe aspects of this format that are designed to maximise read performance are:\n\n- The Apache Feather format can be memory mapped and used directly\n- Apache Arrow has significant existing engineering work geared around efficient access to data, from the layout of the data itself to the library tooling\n- Storing direct information about signal data locations with the row table\n - This allows quick access to a read's data without scanning the data file\n- It is possible to only decode part of a long read, due to read data being stored in chunks\n - This is useful for model training\n- Read access does not require locking or otherwise modifying the file\n - This allows multi-threaded and multi-process access to a file for reading\n\n### Efficient use of space\n\nThe aspects of this format that are designed to maximise use of space are:\n\n- Support for efficient compression routines (VBZ)\n- Apache Arrow's support for dictionary encoding\n- Apache Arrow's support for compressing buffers with standard compression routines\n\n### Ease of implementation\n\nThe aspects of this format that are designed to make the format easy to implement are:\n\n- Relying on an existing, widely-used format (Apache Arrow)\n\n### Extensibility\n\nThe aspects of this format that are designed to make the format extensible are:\n\n- Apache Arrow uses a self-describing schema with named columns, so it is straightforward to write code that is resilient in the face of things like additional columns being added.\n" }, { "path": "docs/README.md", "content": "Design documentation for POD5\n============================\n\nThe POD5 file format has been specifically designed to be suitable for Nanopore read data, we had some specific design goals:\n\nDesign Goals\n------------\n\nThe primary purpose of this file format is store reads produced by Oxford Nanopore sequencing, and in particular the signal data from those reads (which can then be basecalled or processed in other ways).\n\nThis file format has the following design goals:\n\n- Good write performance for MinKNOW\n- Recoverable if the writing process crashes\n- Good read performance for downstream tools, including basecall model generation\n- Efficient use of space\n- Straightforward to implement and maintain\n- Extensibility\n\nNote that trade-offs have been made between these goals, but we have mostly aimed to make those run-time decisions.\n\nWe have also chosen not to optimise for editing existing files.\n\nMore detailed information around general format goals can be found in [DESIGN](./DESIGN.md), more detailed format specification is available in [SPECIFICATION](./SPECIFICATION.md).\n" }, { "path": "docs/SPECIFICATION.md", "content": "POD5 Format Specification\n=========================\n\n## Overview\n\nThe file format is, at its core, a collection of Apache Arrow tables, stored in the Apache Feather 2\n(also know as Apache Arrow IPC File) format, and bundled into a container format. The container file\nhas the extension `.pod5`.\n\n### Table Schemas\n\nPOD5 files are a custom wrapper format around arrow that contain several [arrow\ntables](https://arrow.apache.org/docs/python/data.html#tables).\n\nAll the tables should have the following `custom_metadata` fields set on them:\n\n| Name | Example Value | Notes |\n|-------------------------|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|\n| MINKNOW:pod5_version | 1.0.0 | The version of this specification that the schema was based on. |\n| MINKNOW:software | MinNOW Core 5.2.3 | A free-form description of the software that wrote the file, intended to help pin down the source of files that violate the specification. |\n| MINKNOW:file_identifier | cbf91180-0684-4a39-bf56-41eaf437de9e | Must be identical across all tables. Allows checking that the files correspond to each other. |\n\n### Extension Types\n\nSeveral fields in the table schemas use [custom arrow\ntypes](https://arrow.apache.org/docs/python/data.html#custom-schema-and-field-metadata).\n\n#### minknow.uuid\n\nThe schemas make extensive use of UUIDs to identify reads. This is stored using an extension type,\nwith the following properties:\n\n Name: \"minknow.uuid\"\n Physical storage: FixedBinary(16)\n\n#### minknow.vbz\n\nStorage for VBZ-encoded data:\n\n Name: \"minknow.vbz\"\n Physical storage: LargeBinary\n\n### Tables\n\nThe Reads, Signal and Run Info tables must all be present in a POD5 file. Note that some very early\nPOD5 files produced by pre-0.1 versions of the pod5 library did not include a Run Info table,\ninstead including that information in the Reads table.\n\n#### Reads Table\n\nThe Reads table contains a single row per read, and describes the metadata for each read. The\n`signal` column of the read links to the Signal table, and allows a reads signal to be retrieved.\nThe `run_info` column links to the the Run Info table, providing more context for the read and\navoiding duplicating data that is common to many or all reads in the file.\n\nSome fields of the Reads table are\n[dictionaries](https://arrow.apache.org/docs/python/data.html#dictionary-arrays): the contents of\nthe table are stored in a lookup written prior to each batch of read rows and the read row itself\nthen contains an integer index. This allows space savings on fields that would otherwise be\nrepeated. Only simple types are stored in dictionaries as third party tools have limited support for\ndictionaries of structs.\n\n[tables/reads.toml] contains specific information about fields in the reads table.\n\n#### Signal Table\n\nThe signal table contains the (optionally compressed) signal data where one row contains sequence of\nsample data, and some information about the sample data origin.\n\n[tables/signal.toml] contains specific information about fields in the signal table.\n\n#### Run Info Table\n\nThe run info table contains a single row per MinKNOW run that any read in the file came from.\n\nSeveral fields of the Reads table are\n[dictionaries](https://arrow.apache.org/docs/python/data.html#dictionary-arrays), the contents of\nthe table are stored in a lookup written prior to each batch of read rows, the read row itself then\ncontains an integer index. This allows space savings on fields that would otherwise be repeated.\n\n[tables/run_info.toml] contains specific information about fields in the reads table.\n\n### Combined file Layout\n\n#### Layout\n\n```\n\n
\n
\n...\n
\n