Repository: libfuse/pyfuse3 Branch: main Commit: 96c287b4fd1a Files: 66 Total size: 828.6 KB Directory structure: gitextract_s87k40ee/ ├── .github/ │ └── workflows/ │ ├── codespell.yml │ └── test.yml ├── .gitignore ├── .readthedocs.yaml ├── Changes.rst ├── Include/ │ ├── fuse_common.pxd │ ├── fuse_lowlevel.pxd │ ├── fuse_opt.pxd │ └── libc_extra.pxd ├── LICENSE ├── MANIFEST.in ├── README.rst ├── developer-notes/ │ ├── FUSEError Performance.html │ ├── FUSEError Performance.ipynb │ ├── Namedtuple.html │ ├── Namedtuple.ipynb │ ├── lookup_counts.rst │ ├── release_process.rst │ ├── setup.md │ └── valgrind.md ├── doc/ │ └── .placeholder ├── examples/ │ ├── hello.py │ ├── hello_asyncio.py │ ├── passthroughfs.py │ └── tmpfs.py ├── pyproject.toml ├── rst/ │ ├── _static/ │ │ └── .placeholder │ ├── _templates/ │ │ └── localtoc.html │ ├── about.rst │ ├── asyncio.rst │ ├── changes.rst │ ├── conf.py │ ├── data.rst │ ├── example.rst │ ├── fuse_api.rst │ ├── general.rst │ ├── gotchas.rst │ ├── index.rst │ ├── install.rst │ ├── operations.rst │ └── util.rst ├── src/ │ └── pyfuse3/ │ ├── __init__.pyi │ ├── __init__.pyx │ ├── _pyfuse3.py │ ├── asyncio.py │ ├── darwin_compat.c │ ├── darwin_compat.h │ ├── gettime.h │ ├── handlers.pxi │ ├── internal.pxi │ ├── macros.c │ ├── macros.pxd │ ├── py.typed │ ├── pyfuse3.h │ └── xattr.h ├── test/ │ ├── conftest.py │ ├── pytest.ini │ ├── pytest_checklogs.py │ ├── test_api.py │ ├── test_examples.py │ ├── test_fs.py │ ├── test_rounding.py │ └── util.py └── util/ ├── build_backend.py ├── sdist-sign └── upload-pypi ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/workflows/codespell.yml ================================================ # Codespell configuration is within pyproject.toml --- name: Codespell on: push: branches: [main] pull_request: branches: [main] permissions: contents: read jobs: codespell: name: Check for spelling errors runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Codespell uses: codespell-project/actions-codespell@v2 ================================================ FILE: .github/workflows/test.yml ================================================ name: Test on: [push] env: FORCE_COLOR: 1 jobs: build: runs-on: ${{ matrix.os }} strategy: fail-fast: true matrix: python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] os: [ubuntu-24.04] steps: - uses: actions/checkout@v4 with: # Retrieve some git history, which will hopefully include the last release tag # and enable us to generate better version numbers. fetch-depth: 256 - name: Install uv uses: astral-sh/setup-uv@v6 with: enable-cache: true python-version: ${{ matrix.python-version }} - name: Install Linux dependencies run: | sudo apt-get update sudo apt-get install -y libattr1-dev libfuse3-dev fuse3 pkg-config gcc - name: Build run: uv sync --locked - name: Lint (ruff) run: uv run ruff check - name: Lint (format) run: uv run ruff format --diff - name: Type check (mypy) run: uv run mypy . - name: Type check (pyright) run: uv run pyright - name: Run tests run: uv run pytest -v -rs test/ - name: Build docs run: uv run sphinx-build -b html rst doc/html ================================================ FILE: .gitignore ================================================ MANIFEST build/ dist/ doc/html/ doc/doctrees/ src/pyfuse3/__init__.c src/pyfuse3/__init__*.so src/*.so src/pyfuse3.c test/.cache/ __pycache__ test/.pytest_cache/ *.egg-info *.pyc ================================================ FILE: .readthedocs.yaml ================================================ # .readthedocs.yaml - Read the Docs configuration file. # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details. version: 2 build: os: ubuntu-22.04 tools: python: "3.11" jobs: post_checkout: # Retrieve some git history, which will hopefully include the last release tag # and enable us to generate better version numbers. - git fetch --depth=256 apt_packages: - build-essential - pkg-config - libfuse3-dev python: install: - method: pip path: . sphinx: configuration: rst/conf.py ================================================ FILE: Changes.rst ================================================ =========== Changelog =========== .. currentmodule:: pyfuse3 pyfuse 3.4.2 (2026-01-06) ========================= * Removed the `pyfuse3_asyncio` module. This has been renamed to `pyfuse3.asyncio` for quite some time. * Fixed a test failure in test_examples.py * Many internal changes, modernizing build process and adding more type annotations. Release 3.4.1 (2025-12-22) ========================== * Cythonized with latest Cython 3.2.3. * CI: also test on Python 3.14, on Ubuntu 24.04. * asyncio: - use .run and .get_running_loop, #106. - replace deprecated Future() with create_future; fix lazy trio.Lock init. * use SPDX license identifier, #107 * fix LICENSE text: LGPL v2.1, #102 Release 3.4.0 (2024-08-28) ========================== * Cythonized with latest Cython 3.0.11 to support Python 3.13. * CI: also test python 3.13, run mypy. * Move ``_pyfuse3`` to ``pyfuse3._pyfuse3`` and add a compatibility wrapper for the old name. * Move ``pyfuse3_asyncio`` to ``pyfuse3.asyncio`` and add a compatibility wrapper for the old name. * Add `bytes` subclass `XAttrNameT` as the type of extended attribute names. * Various fixes to type annotations. * Add ``py.typed`` marker to enable external use of type annotations. Release 3.3.0 (2023-08-06) ========================== * Note: This is the first pyfuse3 release compatible with Cython 3.0.0 release. Cython 0.29.x is also still supported. * Cythonized with latest Cython 3.0.0. * Drop Python 3.6 and 3.7 support and testing, #71. * CI: also test python 3.12. test on cython 0.29 and cython 3.0. * Tell Cython that callbacks may raise exceptions, #80. * Fix lookup in examples/hello.py, similar to #16. * Misc. CI, testing, build and sphinx related fixes. Release 3.2.3 (2023-05-09) ========================== * cythonize with latest Cython 0.29.34 (brings Python 3.12 support) * add a minimal pyproject.toml, require setuptools * tests: fix integer overflow on 32-bit arches, fixes #47 * test: Use shutil.which() instead of external which(1) program * setup.py: catch more generic OSError when searching Cython, fixes #63 * setup.py: require Cython >= 0.29 * fix basedir computation in setup.py (fix pip install -e .) * use sphinx < 6.0 due to compatibility issues with more recent versions Release 3.2.2 (2022-09-28) ========================== * remove support for python 3.5 (broken, out of support by python devs) * cythonize with latest Cython 0.29.x (brings Python 3.11 support) * use github actions for CI, remove travis-ci * update README: minimal maintenance, not developed * update setup.py with tested python versions * examples/tmpfs.py: work around strange kernel behaviour (calling SETATTR after UNLINK of a (not open) file): respond with ENOENT instead of crashing. Release 3.2.1 (2021-09-17) ========================== * Add type annotations * Passing a XATTR_CREATE or XATTR_REPLACE to `setxattr` is now working correctly. Release 3.2.0 (2020-12-30) ========================== * Fix long-standing rounding error in file date handling when the nanosecond part of file dates were > 999999500. * There is a new `pyfuse3.terminate()` function to gracefully end the main loop. Release 3.1.1 (2020-10-06) ========================== * No source changes. Regenerated Cython files with Cython 0.29.21 for Python 3.9 compatibility. Release 3.1.0 (2020-05-31) ========================== * Made compatible with newest Trio module. Release 3.0.0 (2020-05-08) ========================== * Changed `~Operations.create` handler to return a `FileInfo` struct to allow for modification of certain kernel file attributes, e.g. ``direct_io``. Note that this change breaks backwards compatibility, code that depends on the old behavior needs to be changed. Release 2.0.0 ============= * Changed `~Operations.open` handler to return the new `FileInfo` struct to allow for modification of certain kernel file attributes, e.g. ``direct_io``. Note that this change breaks backwards compatibility, code that depends on the old behavior needs to be changed. Release 1.3.1 (2019-07-17) ========================== * Fixed a bug in the :file:`hello_asyncio.py` example. Release 1.3 (2019-06-02) ======================== * Fixed a bug in the :file:`tmpfs.py` and :file:`passthroughfs.py` example file systems (so rename operations no longer fail). Release 1.2 (2018-12-22) ======================== * Clarified that `invalidate_inode` may block in some circumstances. * Added support for using the asyncio module instead of Trio. Release 1.1 (2018-11-02) ======================== * Fixed :file:`examples/passthroughfs.py` - was not handling readdir() correctly. * `invalidate_entry_async` now accepts an additional *ignore_enoent* parameter. When this is set, no errors are logged if the kernel is not actually aware of the entry that should have been removed. Release 1.0 (2018-10-08) ======================== * Added a new `syncfs` function. Release 0.9 (2018-09-27) ======================== * First release * pyfuse3 was forked from python-llfuse - thanks for all the work! * If you need compatibility with Python 2.x or libfuse 2.x, you may want to take a look at python-llfuse instead. ================================================ FILE: Include/fuse_common.pxd ================================================ ''' fuse_common.pxd This file contains Cython definitions for fuse_common.h Copyright © 2010 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' from fuse_opt cimport * from posix.types cimport off_t from libc.stdint cimport uint64_t # Based on fuse sources, revision tag fuse_2_9_4 cdef extern from * nogil: # fuse_common.h should not be included struct fuse_file_info: int flags unsigned int direct_io unsigned int keep_cache unsigned int nonseekable uint64_t fh uint64_t lock_owner struct fuse_conn_info: unsigned proto_major unsigned proto_minor unsigned max_write unsigned max_read unsigned max_readahead unsigned capable unsigned want unsigned max_background unsigned congestion_threshold unsigned time_gran struct fuse_session: pass struct fuse_chan: pass struct fuse_pollhandle: pass void fuse_pollhandle_destroy(fuse_pollhandle *ph) struct fuse_loop_config: int clone_fd unsigned max_idle_threads # Capability bits for fuse_conn_info.{capable,want} enum: FUSE_CAP_ASYNC_READ FUSE_CAP_POSIX_LOCKS FUSE_CAP_ATOMIC_O_TRUNC FUSE_CAP_EXPORT_SUPPORT FUSE_CAP_DONT_MASK FUSE_CAP_SPLICE_WRITE FUSE_CAP_SPLICE_MOVE FUSE_CAP_SPLICE_READ FUSE_CAP_FLOCK_LOCKS FUSE_CAP_IOCTL_DIR FUSE_CAP_AUTO_INVAL_DATA FUSE_CAP_READDIRPLUS FUSE_CAP_READDIRPLUS_AUTO FUSE_CAP_ASYNC_DIO FUSE_CAP_WRITEBACK_CACHE FUSE_CAP_NO_OPEN_SUPPORT FUSE_CAP_PARALLEL_DIROPS FUSE_CAP_POSIX_ACL FUSE_CAP_HANDLE_KILLPRIV int fuse_set_signal_handlers(fuse_session *se) void fuse_remove_signal_handlers(fuse_session *se) # fuse_common.h declares these as enums, but they are # actually flags (i.e., FUSE_BUF_IS_FD|FUSE_BUF_FD_SEEK) # is a valid variable. Therefore, we declare the type # as integer instead. ctypedef int fuse_buf_flags enum: FUSE_BUF_IS_FD FUSE_BUF_FD_SEEK FUSE_BUF_FD_RETRY ctypedef int fuse_buf_copy_flags enum: FUSE_BUF_NO_SPLICE FUSE_BUF_FORCE_SPLICE FUSE_BUF_SPLICE_MOVE FUSE_BUF_SPLICE_NONBLOCK struct fuse_buf: size_t size fuse_buf_flags flags void *mem int fd off_t pos struct fuse_bufvec: size_t count size_t idx size_t off fuse_buf buf[1] size_t fuse_buf_size(fuse_bufvec *bufv) ssize_t fuse_buf_copy(fuse_bufvec *dst, fuse_bufvec *src, fuse_buf_copy_flags flags) ================================================ FILE: Include/fuse_lowlevel.pxd ================================================ ''' fuse_lowlevel.pxd This file contains Cython definitions for fuse_lowlevel.h Copyright © 2010 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' from fuse_common cimport * from posix.stat cimport * from posix.types cimport * from libc_extra cimport statvfs from libc.stdlib cimport const_char from libc.stdint cimport uint32_t # Based on fuse sources, revision tag fuse-3.2.6 cdef extern from "" nogil: enum: FUSE_ROOT_ID ctypedef unsigned fuse_ino_t ctypedef struct fuse_req: pass ctypedef fuse_req* fuse_req_t struct fuse_entry_param: fuse_ino_t ino uint64_t generation struct_stat attr double attr_timeout double entry_timeout struct fuse_ctx: uid_t uid gid_t gid pid_t pid mode_t umask struct fuse_forget_data: fuse_ino_t ino uint64_t nlookup ctypedef fuse_ctx const_fuse_ctx "const struct fuse_ctx" int FUSE_SET_ATTR_MODE int FUSE_SET_ATTR_UID int FUSE_SET_ATTR_GID int FUSE_SET_ATTR_SIZE int FUSE_SET_ATTR_ATIME int FUSE_SET_ATTR_MTIME int FUSE_SET_ATTR_ATIME_NOW int FUSE_SET_ATTR_MTIME_NOW int FUSE_SET_ATTR_CTIME # Request handlers # We allow these functions to raise exceptions because we will catch them # when checking exception status on return from fuse_session_process_buf(). struct fuse_lowlevel_ops: void (*init) (void *userdata, fuse_conn_info *conn) except * void (*destroy) (void *userdata) except * void (*lookup) (fuse_req_t req, fuse_ino_t parent, const_char *name) except * void (*forget) (fuse_req_t req, fuse_ino_t ino, uint64_t nlookup) except * void (*getattr) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*setattr) (fuse_req_t req, fuse_ino_t ino, struct_stat *attr, int to_set, fuse_file_info *fi) except * void (*readlink) (fuse_req_t req, fuse_ino_t ino) except * void (*mknod) (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode, dev_t rdev) except * void (*mkdir) (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode) except * void (*unlink) (fuse_req_t req, fuse_ino_t parent, const_char *name) except * void (*rmdir) (fuse_req_t req, fuse_ino_t parent, const_char *name) except * void (*symlink) (fuse_req_t req, const_char *link, fuse_ino_t parent, const_char *name) except * void (*rename) (fuse_req_t req, fuse_ino_t parent, const_char *name, fuse_ino_t newparent, const_char *newname, unsigned flags) except * void (*link) (fuse_req_t req, fuse_ino_t ino, fuse_ino_t newparent, const_char *newname) except * void (*open) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*read) (fuse_req_t req, fuse_ino_t ino, size_t size, off_t off, fuse_file_info *fi) except * void (*write) (fuse_req_t req, fuse_ino_t ino, const_char *buf, size_t size, off_t off, fuse_file_info *fi) except * void (*flush) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*release) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*fsync) (fuse_req_t req, fuse_ino_t ino, int datasync, fuse_file_info *fi) except * void (*opendir) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*readdir) (fuse_req_t req, fuse_ino_t ino, size_t size, off_t off, fuse_file_info *fi) except * void (*releasedir) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi) except * void (*fsyncdir) (fuse_req_t req, fuse_ino_t ino, int datasync, fuse_file_info *fi) except * void (*statfs) (fuse_req_t req, fuse_ino_t ino) except * void (*setxattr) (fuse_req_t req, fuse_ino_t ino, const_char *name, const_char *value, size_t size, int flags) except * void (*getxattr) (fuse_req_t req, fuse_ino_t ino, const_char *name, size_t size) except * void (*listxattr) (fuse_req_t req, fuse_ino_t ino, size_t size) except * void (*removexattr) (fuse_req_t req, fuse_ino_t ino, const_char *name) except * void (*access) (fuse_req_t req, fuse_ino_t ino, int mask) except * void (*create) (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode, fuse_file_info *fi) except * void (*write_buf) (fuse_req_t req, fuse_ino_t ino, fuse_bufvec *bufv, off_t off, fuse_file_info *fi) except * void (*retrieve_reply) (fuse_req_t req, void *cookie, fuse_ino_t ino, off_t offset, fuse_bufvec *bufv) except * void (*forget_multi) (fuse_req_t req, size_t count, fuse_forget_data *forgets) except * void (*fallocate) (fuse_req_t req, fuse_ino_t ino, int mode, off_t offset, off_t length, fuse_file_info *fi) except * void (*readdirplus) (fuse_req_t req, fuse_ino_t ino, size_t size, off_t off, fuse_file_info *fi) except * void (*poll) (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi, fuse_pollhandle *ph) except * # Reply functions int fuse_reply_err(fuse_req_t req, int err) void fuse_reply_none(fuse_req_t req) int fuse_reply_entry(fuse_req_t req, fuse_entry_param *e) int fuse_reply_create(fuse_req_t req, fuse_entry_param *e, fuse_file_info *fi) int fuse_reply_attr(fuse_req_t req, struct_stat *attr, double attr_timeout) int fuse_reply_readlink(fuse_req_t req, const_char *link) int fuse_reply_open(fuse_req_t req, fuse_file_info *fi) int fuse_reply_write(fuse_req_t req, size_t count) int fuse_reply_buf(fuse_req_t req, const_char *buf, size_t size) int fuse_reply_data(fuse_req_t req, fuse_bufvec *bufv, fuse_buf_copy_flags flags) int fuse_reply_statfs(fuse_req_t req, statvfs *stbuf) int fuse_reply_xattr(fuse_req_t req, size_t count) int fuse_reply_poll(fuse_req_t req, unsigned revents) size_t fuse_add_direntry(fuse_req_t req, const_char *buf, size_t bufsize, const_char *name, struct_stat *stbuf, off_t off) size_t fuse_add_direntry_plus(fuse_req_t req, char *buf, size_t bufsize, char *name, fuse_entry_param *e, off_t off) # Notification int fuse_lowlevel_notify_inval_inode(fuse_session *se, fuse_ino_t ino, off_t off, off_t len) int fuse_lowlevel_notify_inval_entry(fuse_session *se, fuse_ino_t parent, const_char *name, size_t namelen) int fuse_lowlevel_notify_delete(fuse_session *se, fuse_ino_t parent, fuse_ino_t child, const_char *name, size_t namelen) int fuse_lowlevel_notify_store(fuse_session *se, fuse_ino_t ino, off_t offset, fuse_bufvec *bufv, fuse_buf_copy_flags flags) int fuse_lowlevel_notify_retrieve(fuse_session *se, fuse_ino_t ino, size_t size, off_t offset, void *cookie) int fuse_lowlevel_notify_poll(fuse_pollhandle *ph) # Utility functions void *fuse_req_userdata(fuse_req_t req) fuse_ctx *fuse_req_ctx(fuse_req_t req) int fuse_req_getgroups(fuse_req_t req, size_t size, gid_t list[]) # Inquiry functions void fuse_lowlevel_version() void fuse_lowlevel_help() # Filesystem setup & teardown fuse_session *fuse_session_new(fuse_args *args, fuse_lowlevel_ops *op, size_t op_size, void *userdata) int fuse_session_mount(fuse_session *se, char *mountpoint) int fuse_session_loop(fuse_session *se) int fuse_session_loop_mt(fuse_session *se, fuse_loop_config *config); void fuse_session_exit(fuse_session *se) void fuse_session_reset(fuse_session *se) bint fuse_session_exited(fuse_session *se) void fuse_session_unmount(fuse_session *se) void fuse_session_destroy(fuse_session *se) # Custom event loop support int fuse_session_fd(fuse_session *se) int fuse_session_receive_buf(fuse_session *se, fuse_buf *buf) void fuse_session_process_buf(fuse_session *se, fuse_buf *buf) except * ================================================ FILE: Include/fuse_opt.pxd ================================================ ''' fuse_opt.pxd This file contains Cython definitions for fuse_opt.h Copyright © 2010 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' # Based on fuse sources, revision tag fuse_2_8_3 cdef extern from "" nogil: struct fuse_args: int argc char **argv int allocated ================================================ FILE: Include/libc_extra.pxd ================================================ ''' libc_extra.pxd This file contains Cython definitions libc functions that are not included in the pxd files shipped with Cython. Copyright © 2010 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' from posix.time cimport timespec cdef extern from "" nogil: ctypedef struct DIR: pass cdef struct dirent: char* d_name dirent* readdir(DIR* dirp) int readdir_r(DIR *dirp, dirent *entry, dirent **result) cdef extern from "" nogil: DIR *opendir(char *name) int closedir(DIR* dirp) cdef extern from "" nogil: ctypedef int fsblkcnt_t ctypedef int fsfilcnt_t struct statvfs: unsigned long f_bsize unsigned long f_frsize fsblkcnt_t f_blocks fsblkcnt_t f_bfree fsblkcnt_t f_bavail fsfilcnt_t f_files fsfilcnt_t f_ffree fsfilcnt_t f_favail unsigned long f_namemax cdef extern from "xattr.h" nogil: int setxattr_p (char *path, char *name, void *value, int size, int namespace) ssize_t getxattr_p (char *path, char *name, void *value, int size, int namespace) enum: EXTATTR_NAMESPACE_SYSTEM EXTATTR_NAMESPACE_USER XATTR_CREATE XATTR_REPLACE XATTR_NOFOLLOW XATTR_NODEFAULT XATTR_NOSECURITY cdef extern from "gettime.h" nogil: int gettime_realtime(timespec *tp) cdef extern from "" nogil: int syncfs(int fd) ================================================ FILE: LICENSE ================================================ GNU LESSER GENERAL PUBLIC LICENSE Version 2.1, February 1999 Copyright (C) 1991, 1999 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. [This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library Public License, version 2, hence the version number 2.1.] Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. We call this license the "Lesser" General Public License because it does Less to protect the user's freedom than the ordinary General Public License. It also provides other free software developers Less of an advantage over competing non-free programs. These disadvantages are the reason we use the ordinary General Public License for many libraries. However, the Lesser license provides advantages in certain special circumstances. For example, on rare occasions, there may be a special need to encourage the widest possible use of a certain library, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use the library. A more frequent case is that a free library does the same job as widely used non-free libraries. In this case, there is little to gain by limiting the free library to free software only, so we use the Lesser General Public License. In other cases, permission to use a particular library in non-free programs enables a greater number of people to use a large body of free software. For example, permission to use the GNU C Library in non-free programs enables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linux operating system. Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that the user of a program that is linked with the Library has the freedom and the wherewithal to run that program using a modified version of the Library. The precise terms and conditions for copying, distribution and modification follow. Pay close attention to the difference between a "work based on the library" and a "work that uses the library". The former contains code derived from the library, whereas the latter must be combined with the library in order to run. GNU LESSER GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any software library or other program which contains a notice placed by the copyright holder or other authorized party saying it may be distributed under the terms of this Lesser General Public License (also called "this License"). Each licensee is addressed as "you". A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application programs (which use some of those functions and data) to form executables. The "Library", below, refers to any such software library or work which has been distributed under these terms. A "work based on the Library" means either the Library or any derivative work under copyright law: that is to say, a work containing the Library or a portion of it, either verbatim or with modifications and/or translated straightforwardly into another language. (Hereinafter, translation is included without limitation in the term "modification".) "Source code" for a work means the preferred form of the work for making modifications to it. For a library, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the library. Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running a program using the Library is not restricted, and output from such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. 1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) The modified work must itself be a software library. b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change. c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License. d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful. (For example, a function in a library to compute square roots has a purpose that is entirely well-defined independent of the application. Therefore, Subsection 2d requires that any application-supplied function or table used by this function must be optional: if the application does not supply it, the square root function must still compute square roots.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Library, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Library, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Library. In addition, mere aggregation of another work not based on the Library with the Library (or with a work based on the Library) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices. Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General Public License applies to all subsequent copies and derivative works made from that copy. This option is useful when you wish to copy part of the code of the Library into a program that is not a library. 4. You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange. If distribution of object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place satisfies the requirement to distribute the source code, even though third parties are not compelled to copy the source along with the object code. 5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. However, linking a "work that uses the Library" with the Library creates an executable that is a derivative of the Library (because it contains portions of the Library), rather than a "work that uses the library". The executable is therefore covered by this License. Section 6 states terms for distribution of such executables. When a "work that uses the Library" uses material from a header file that is part of the Library, the object code for the work may be a derivative work of the Library even though the source code is not. Whether this is true is especially significant if the work can be linked without the Library, or if the work is itself a library. The threshold for this to be true is not precisely defined by law. If such an object file uses only numerical parameters, data structure layouts and accessors, and small macros and small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardless of whether it is legally a derivative work. (Executables containing this object code plus portions of the Library will still fall under Section 6.) Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under the terms of Section 6. Any executables containing that work also fall under Section 6, whether or not they are linked directly with the Library itself. 6. As an exception to the Sections above, you may also combine or link a "work that uses the Library" with the Library to produce a work containing portions of the Library, and distribute that work under terms of your choice, provided that the terms permit modification of the work for the customer's own use and reverse engineering for debugging such modifications. You must give prominent notice with each copy of the work that the Library is used in it and that the Library and its use are covered by this License. You must supply a copy of this License. If the work during execution displays copyright notices, you must include the copyright notice for the Library among them, as well as a reference directing the user to the copy of this License. Also, you must do one of these things: a) Accompany the work with the complete corresponding machine-readable source code for the Library including whatever changes were used in the work (which must be distributed under Sections 1 and 2 above); and, if the work is an executable linked with the Library, with the complete machine-readable "work that uses the Library", as object code and/or source code, so that the user can modify the Library and then relink to produce a modified executable containing the modified Library. (It is understood that the user who changes the contents of definitions files in the Library will not necessarily be able to recompile the application to use the modified definitions.) b) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (1) uses at run time a copy of the library already present on the user's computer system, rather than copying library functions into the executable, and (2) will operate properly with a modified version of the library, if the user installs one, as long as the modified version is interface-compatible with the version that the work was made with. c) Accompany the work with a written offer, valid for at least three years, to give the same user the materials specified in Subsection 6a, above, for a charge no more than the cost of performing this distribution. d) If distribution of the work is made by offering access to copy from a designated place, offer equivalent access to copy the above specified materials from the same place. e) Verify that the user has already received a copy of these materials or that you have already sent this user a copy. For an executable, the required form of the "work that uses the Library" must include any data and utility programs needed for reproducing the executable from it. However, as a special exception, the materials to be distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute. 7. You may place library facilities that are a work based on the Library side-by-side in a single library together with other library facilities not covered by this License, and distribute such a combined library, provided that the separate distribution of the work based on the Library and of the other library facilities is otherwise permitted, and provided that you do these two things: a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities. This must be distributed under the terms of the Sections above. b) Give prominent notice with the combined library of the fact that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. 8. You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 9. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Library or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Library (or any work based on the Library), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Library or works based on it. 10. Each time you redistribute the Library (or any work based on the Library), the recipient automatically receives a license from the original licensor to copy, distribute, link with or modify the Library subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties with this License. 11. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Library at all. For example, if a patent license would not permit royalty-free redistribution of the Library by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Library. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply, and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 12. If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Library under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 13. The Free Software Foundation may publish revised and/or new versions of the Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Library specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Library does not specify a license version number, you may choose any version ever published by the Free Software Foundation. 14. If you wish to incorporate parts of the Library into other free programs whose distribution conditions are incompatible with these, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Libraries If you develop a new library, and you want it to be of the greatest possible use to the public, we recommend making it free software that everyone can redistribute and change. You can do so by permitting redistribution under these terms (or, alternatively, under the terms of the ordinary General Public License). To apply these terms, attach the following notices to the library. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, see . Also add information on how to contact you by electronic and paper mail. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the library, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the library `Frob' (a library for tweaking knobs) written by James Random Hacker. , 1 April 1990 Moe Ghoul, President of Vice That's all there is to it! GNU LESSER GENERAL PUBLIC LICENSE Version 2.1, February 1999 Copyright (C) 1991, 1999 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. [This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library Public License, version 2, hence the version number 2.1.] Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. We call this license the "Lesser" General Public License because it does Less to protect the user's freedom than the ordinary General Public License. It also provides other free software developers Less of an advantage over competing non-free programs. These disadvantages are the reason we use the ordinary General Public License for many libraries. However, the Lesser license provides advantages in certain special circumstances. For example, on rare occasions, there may be a special need to encourage the widest possible use of a certain library, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use the library. A more frequent case is that a free library does the same job as widely used non-free libraries. In this case, there is little to gain by limiting the free library to free software only, so we use the Lesser General Public License. In other cases, permission to use a particular library in non-free programs enables a greater number of people to use a large body of free software. For example, permission to use the GNU C Library in non-free programs enables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linux operating system. Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that the user of a program that is linked with the Library has the freedom and the wherewithal to run that program using a modified version of the Library. The precise terms and conditions for copying, distribution and modification follow. Pay close attention to the difference between a "work based on the library" and a "work that uses the library". The former contains code derived from the library, whereas the latter must be combined with the library in order to run. GNU LESSER GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any software library or other program which contains a notice placed by the copyright holder or other authorized party saying it may be distributed under the terms of this Lesser General Public License (also called "this License"). Each licensee is addressed as "you". A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application programs (which use some of those functions and data) to form executables. The "Library", below, refers to any such software library or work which has been distributed under these terms. A "work based on the Library" means either the Library or any derivative work under copyright law: that is to say, a work containing the Library or a portion of it, either verbatim or with modifications and/or translated straightforwardly into another language. (Hereinafter, translation is included without limitation in the term "modification".) "Source code" for a work means the preferred form of the work for making modifications to it. For a library, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the library. Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running a program using the Library is not restricted, and output from such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. 1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) The modified work must itself be a software library. b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change. c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License. d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful. (For example, a function in a library to compute square roots has a purpose that is entirely well-defined independent of the application. Therefore, Subsection 2d requires that any application-supplied function or table used by this function must be optional: if the application does not supply it, the square root function must still compute square roots.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Library, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Library, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Library. In addition, mere aggregation of another work not based on the Library with the Library (or with a work based on the Library) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices. Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General Public License applies to all subsequent copies and derivative works made from that copy. This option is useful when you wish to copy part of the code of the Library into a program that is not a library. 4. You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange. If distribution of object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place satisfies the requirement to distribute the source code, even though third parties are not compelled to copy the source along with the object code. 5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. However, linking a "work that uses the Library" with the Library creates an executable that is a derivative of the Library (because it contains portions of the Library), rather than a "work that uses the library". The executable is therefore covered by this License. Section 6 states terms for distribution of such executables. When a "work that uses the Library" uses material from a header file that is part of the Library, the object code for the work may be a derivative work of the Library even though the source code is not. Whether this is true is especially significant if the work can be linked without the Library, or if the work is itself a library. The threshold for this to be true is not precisely defined by law. If such an object file uses only numerical parameters, data structure layouts and accessors, and small macros and small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardless of whether it is legally a derivative work. (Executables containing this object code plus portions of the Library will still fall under Section 6.) Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under the terms of Section 6. Any executables containing that work also fall under Section 6, whether or not they are linked directly with the Library itself. 6. As an exception to the Sections above, you may also combine or link a "work that uses the Library" with the Library to produce a work containing portions of the Library, and distribute that work under terms of your choice, provided that the terms permit modification of the work for the customer's own use and reverse engineering for debugging such modifications. You must give prominent notice with each copy of the work that the Library is used in it and that the Library and its use are covered by this License. You must supply a copy of this License. If the work during execution displays copyright notices, you must include the copyright notice for the Library among them, as well as a reference directing the user to the copy of this License. Also, you must do one of these things: a) Accompany the work with the complete corresponding machine-readable source code for the Library including whatever changes were used in the work (which must be distributed under Sections 1 and 2 above); and, if the work is an executable linked with the Library, with the complete machine-readable "work that uses the Library", as object code and/or source code, so that the user can modify the Library and then relink to produce a modified executable containing the modified Library. (It is understood that the user who changes the contents of definitions files in the Library will not necessarily be able to recompile the application to use the modified definitions.) b) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (1) uses at run time a copy of the library already present on the user's computer system, rather than copying library functions into the executable, and (2) will operate properly with a modified version of the library, if the user installs one, as long as the modified version is interface-compatible with the version that the work was made with. c) Accompany the work with a written offer, valid for at least three years, to give the same user the materials specified in Subsection 6a, above, for a charge no more than the cost of performing this distribution. d) If distribution of the work is made by offering access to copy from a designated place, offer equivalent access to copy the above specified materials from the same place. e) Verify that the user has already received a copy of these materials or that you have already sent this user a copy. For an executable, the required form of the "work that uses the Library" must include any data and utility programs needed for reproducing the executable from it. However, as a special exception, the materials to be distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute. 7. You may place library facilities that are a work based on the Library side-by-side in a single library together with other library facilities not covered by this License, and distribute such a combined library, provided that the separate distribution of the work based on the Library and of the other library facilities is otherwise permitted, and provided that you do these two things: a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities. This must be distributed under the terms of the Sections above. b) Give prominent notice with the combined library of the fact that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. 8. You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 9. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Library or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Library (or any work based on the Library), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Library or works based on it. 10. Each time you redistribute the Library (or any work based on the Library), the recipient automatically receives a license from the original licensor to copy, distribute, link with or modify the Library subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties with this License. 11. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Library at all. For example, if a patent license would not permit royalty-free redistribution of the Library by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Library. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply, and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 12. If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Library under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 13. The Free Software Foundation may publish revised and/or new versions of the Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Library specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Library does not specify a license version number, you may choose any version ever published by the Free Software Foundation. 14. If you wish to incorporate parts of the Library into other free programs whose distribution conditions are incompatible with these, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Libraries If you develop a new library, and you want it to be of the greatest possible use to the public, we recommend making it free software that everyone can redistribute and change. You can do so by permitting redistribution under these terms (or, alternatively, under the terms of the ordinary General Public License). To apply these terms, attach the following notices to the library. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, see . Also add information on how to contact you by electronic and paper mail. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the library, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the library `Frob' (a library for tweaking knobs) written by James Random Hacker. , 1 April 1990 Moe Ghoul, President of Vice That's all there is to it! ================================================ FILE: MANIFEST.in ================================================ include Changes.rst include LICENSE graft doc/html graft Include graft examples graft rst graft util graft test prune test/.cache prune .github exclude MANIFEST.in exclude .git* exclude .readthedocs.yaml recursive-include src *.pyx *.pyi *.py *.pxi *.pxd *.c *.h global-exclude *.pyc ================================================ FILE: README.rst ================================================ .. NOTE: We cannot use sophisticated ReST syntax (like e.g. :file:`foo`) here because this isn't rendered correctly by PyPi. The pyfuse3 Module ================== .. start-intro pyfuse3 is a set of Python 3 bindings for `libfuse 3`_. It provides an asynchronous API compatible with Trio_ and asyncio_, and enables you to easily write a full-featured Linux filesystem in Python. pyfuse3 releases can be downloaded from PyPi_. The documentation can be `read online`__ and is also included in the ``doc/html`` directory of the pyfuse3 tarball. Getting Help ------------ Please report any bugs on the `issue tracker`_. For discussion and questions, please use the general `FUSE mailing list`_ or `GitHub Discussions `_. Development Status ------------------ pyfuse3 is stable when used with Trio. The current maintainers ensure that bugs are addressed and pyfuse3 continues to work with new Python and libfuse versions. There is no plan to add new features or other non-bugfix work. However, pull requests for new features or other improvements may be accepted. Using pyfuse3 with asyncio (rather than Trio) support is less well tested, there may be bugs, and some of them may not be easily fixable. If you need a synchronous (non async) implementation, `mfusepy `_ is a maintained alternative. Contributing ------------ The pyfuse3 source code is available on GitHub_. .. __: https://pyfuse3.readthedocs.io/ .. _`libfuse 3`: http://github.com/libfuse/libfuse .. _FUSE mailing list: https://lists.sourceforge.net/lists/listinfo/fuse-devel .. _issue tracker: https://github.com/libfuse/pyfuse3/issues .. _mailing list archive: http://dir.gmane.org/gmane.comp.file-systems.fuse.devel .. _Gmane: http://www.gmane.org/ .. _PyPi: https://pypi.python.org/pypi/pyfuse3/ .. _GitHub: https://github.com/libfuse/pyfuse3 .. _Trio: https://github.com/python-trio/trio .. _asyncio: https://docs.python.org/3/library/asyncio.html ================================================ FILE: developer-notes/FUSEError Performance.html ================================================ FUSEError Performance
In [1]:
%load_ext cython

Regular class vs extensions class

In [12]:
%%cython
cimport cython

cdef class FUSEErrorExt(Exception):
    '''
    This exception may be raised by request handlers to indicate that
    the requested operation could not be carried out. The system call
    that resulted in the request (if any) will then fail with error
    code *errno_*.
    '''

    # If we call this variable "errno", we will get syntax errors
    # during C compilation (maybe something else declares errno as
    # a macro?)
    cdef int errno_

    property errno:
        '''Error code to return to client process'''
        def __get__(self):
            return self.errno_
        def __set__(self, val):
            self.errno_ = val

    def __init__(self, errno):
        self.errno_ = errno

warning: /home/nikratio/.cache/ipython/cython/_cython_magic_f3365cad4f189403d0322b37c637671e.pyx:8:5: freelists cannot be used on subtypes, only the base class can manage them
In [4]:
class FUSEErrorInt(Exception):
    def __init__(self, errno):
        self.errno = errno
In [5]:
def test_ext():
    a = 0
    for i in range(100):
        try:
            raise FUSEErrorExt(i)
        except FUSEErrorExt as exc:
            a += exc.errno
        except:
            print('This should not happen')
    return a

def test_int():
    a = 0
    for i in range(100):
        try:
            raise FUSEErrorInt(i)
        except FUSEErrorInt as exc:
            a += exc.errno
        except:
            print('This should not happen')
    return a
In [6]:
assert test_ext() == test_int()
%timeit test_ext()
%timeit test_int()

The slowest run took 8.54 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 36 µs per loop
10000 loops, best of 3: 57.9 µs per loop

Instantiation vs Factory Function with Cache

(Unfortunately we cannot use @cython.freelist for derived classes)

In [7]:
cache = dict()
def getError(errno):
    try:
        return cache[errno]
    except KeyError:
        cache[errno] = FUSEErrorExt(errno)
        return cache[errno]
    
def test_ext_cached():
    a = 0
    for i in range(100):
        try:
            raise getError(i)
        except FUSEErrorExt as exc:
            a += exc.errno
        except:
            print('This should not happen')
    return a
In [8]:
assert test_ext() == test_ext_cached()
%timeit test_ext()
%timeit test_ext_cached()

10000 loops, best of 3: 32.7 µs per loop
10000 loops, best of 3: 32.4 µs per loop

Catching Exception vs Ordinary Return

In [9]:
def handler(i):
    return getError(i)

def test_ext_direct():
    a = 0
    for i in range(100):
        res = handler(i)
        if isinstance(res, FUSEErrorExt):
            a += res.errno
    return a
In [10]:
assert test_ext_cached() == test_ext_direct()
%timeit test_ext_cached()
%timeit test_ext_direct()

The slowest run took 5.52 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 32.4 µs per loop
10000 loops, best of 3: 28.4 µs per loop
In [ ]:
================================================ FILE: developer-notes/FUSEError Performance.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%load_ext cython" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Regular class vs extensions class" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "warning: /home/nikratio/.cache/ipython/cython/_cython_magic_f3365cad4f189403d0322b37c637671e.pyx:8:5: freelists cannot be used on subtypes, only the base class can manage them\n" ] } ], "source": [ "%%cython\n", "cimport cython\n", "\n", "cdef class FUSEErrorExt(Exception):\n", " '''\n", " This exception may be raised by request handlers to indicate that\n", " the requested operation could not be carried out. The system call\n", " that resulted in the request (if any) will then fail with error\n", " code *errno_*.\n", " '''\n", "\n", " # If we call this variable \"errno\", we will get syntax errors\n", " # during C compilation (maybe something else declares errno as\n", " # a macro?)\n", " cdef int errno_\n", "\n", " property errno:\n", " '''Error code to return to client process'''\n", " def __get__(self):\n", " return self.errno_\n", " def __set__(self, val):\n", " self.errno_ = val\n", "\n", " def __init__(self, errno):\n", " self.errno_ = errno" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "class FUSEErrorInt(Exception):\n", " def __init__(self, errno):\n", " self.errno = errno" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def test_ext():\n", " a = 0\n", " for i in range(100):\n", " try:\n", " raise FUSEErrorExt(i)\n", " except FUSEErrorExt as exc:\n", " a += exc.errno\n", " except:\n", " print(\"This should not happen\")\n", " return a\n", "\n", "\n", "def test_int():\n", " a = 0\n", " for i in range(100):\n", " try:\n", " raise FUSEErrorInt(i)\n", " except FUSEErrorInt as exc:\n", " a += exc.errno\n", " except:\n", " print(\"This should not happen\")\n", " return a" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The slowest run took 8.54 times longer than the fastest. This could mean that an intermediate result is being cached.\n", "10000 loops, best of 3: 36 µs per loop\n", "10000 loops, best of 3: 57.9 µs per loop\n" ] } ], "source": [ "assert test_ext() == test_int()\n", "%timeit test_ext()\n", "%timeit test_int()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Instantiation vs Factory Function with Cache" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(Unfortunately we cannot use @cython.freelist for derived classes)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cache = dict()\n", "\n", "\n", "def getError(errno):\n", " try:\n", " return cache[errno]\n", " except KeyError:\n", " cache[errno] = FUSEErrorExt(errno)\n", " return cache[errno]\n", "\n", "\n", "def test_ext_cached():\n", " a = 0\n", " for i in range(100):\n", " try:\n", " raise getError(i)\n", " except FUSEErrorExt as exc:\n", " a += exc.errno\n", " except:\n", " print(\"This should not happen\")\n", " return a" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10000 loops, best of 3: 32.7 µs per loop\n", "10000 loops, best of 3: 32.4 µs per loop\n" ] } ], "source": [ "assert test_ext() == test_ext_cached()\n", "%timeit test_ext()\n", "%timeit test_ext_cached()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Catching Exception vs Ordinary Return" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def handler(i):\n", " return getError(i)\n", "\n", "\n", "def test_ext_direct():\n", " a = 0\n", " for i in range(100):\n", " res = handler(i)\n", " if isinstance(res, FUSEErrorExt):\n", " a += res.errno\n", " return a" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The slowest run took 5.52 times longer than the fastest. This could mean that an intermediate result is being cached.\n", "10000 loops, best of 3: 32.4 µs per loop\n", "10000 loops, best of 3: 28.4 µs per loop\n" ] } ], "source": [ "assert test_ext_cached() == test_ext_direct()\n", "%timeit test_ext_cached()\n", "%timeit test_ext_direct()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.3" } }, "nbformat": 4, "nbformat_minor": 0 } ================================================ FILE: developer-notes/Namedtuple.html ================================================ Namedtuple

Namedtuple vs Extension Class

In [1]:
%load_ext Cython
In [2]:
%%cython

cdef class InvalRequestExt:
    cdef int ino
    cdef char attr_only
    
    def __cinit__(self, ino, attr_only):
        self.ino = ino
        self.attr_only = bool(attr_only)
In [3]:
from collections import namedtuple
InvalRequestTup = namedtuple('InvalRequestTup', [ 'inode', 'attr_only' ])
In [4]:
def test(cls):
    inst = []
    for i in range(500):
        inst.append(cls(i, False))
    return inst
In [5]:
assert len(test(InvalRequestExt)) == len(test(InvalRequestTup))
In [6]:
%timeit test(InvalRequestExt)
%timeit test(InvalRequestTup)

10000 loops, best of 3: 63.3 µs per loop
1000 loops, best of 3: 204 µs per loop
In [ ]:
================================================ FILE: developer-notes/Namedtuple.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Namedtuple vs Extension Class" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%load_ext Cython" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%cython\n", "\n", "cdef class InvalRequestExt:\n", " cdef int ino\n", " cdef char attr_only\n", " \n", " def __cinit__(self, ino, attr_only):\n", " self.ino = ino\n", " self.attr_only = bool(attr_only)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from collections import namedtuple\n", "\n", "InvalRequestTup = namedtuple(\"InvalRequestTup\", [\"inode\", \"attr_only\"])" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def test(cls):\n", " inst = []\n", " for i in range(500):\n", " inst.append(cls(i, False))\n", " return inst" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "assert len(test(InvalRequestExt)) == len(test(InvalRequestTup))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10000 loops, best of 3: 63.3 µs per loop\n", "1000 loops, best of 3: 204 µs per loop\n" ] } ], "source": [ "%timeit test(InvalRequestExt)\n", "%timeit test(InvalRequestTup)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.3" } }, "nbformat": 4, "nbformat_minor": 0 } ================================================ FILE: developer-notes/lookup_counts.rst ================================================ =========================================== Move Lookup Count Management into PYFUSE3? =========================================== It would be nice if PYFUSE3 could keep track of the lookup count management. That way its users wouldn't need to worry about which handlers increase the lookup count, and `forget` would only be called when the lookup count reaches zero. Unfortunately, this is only possible when serializing all Python request handlers. The reason is the following: If an application wants to distinguish between "active" and forgotten inodes it generally wants to establish some internal lstate that survives as long as the corresponding inode is active. However, in order to maintain that state, it has to be protected by the same lock as the lookup count. This makes it impossible to update the lookup count in PYFUSE3 after the python handler method has returned. Example:: class WontWork: def lookup(self, name): inode = get_inode(name) lookup_count[inode] += 1 # for simplicity, assume this is atomic cache[inode] = get_state(inode) def forget(self, inode): lookup_count[inode] -= 1 # for simplicity, assume this is atomic if lookup_count[inode] == 0: del cache[inode] def open(self, inode): # This works, because lookup() must have returned before # open() can be called. assert lookup_count[inode] > 0 # This won't work, because forget() may have been interrupted by lookup() # between `if` and `del` assert cache[inode] class WouldWork: def lookup(self, name): inode = get_inode(name) with lock(inode): lookup_count[inode] += 1 cache[inode] = get_state(inode) def forget(self, inode): with lock(inode): lookup_count[inode] -= 1 if lookup_count[inode] == 0: del cache[inode] def open(self, inode): assert lookup_count[inode] > 0 assert cache[inode] A slightly less complex situation arises if the application does not want to keep state, but is just using lookup counts to postpone inode removal until `forget`. In this case, one correct implementation is:: class SimpleOps: def lookup(self, name): inode = get_inode(name) with lookup_lock: lookup_count[inode] += 1 def forget(self, inode): with lookup_lock: lookup_count[inode] -= 1 if lookup_count[inode] > 0: return del lookup_count[inode] self.maybe_remove_inode(inode) def maybe_remove_inode(self, inode): with lock(inode): if refcount_of(inode) > 0: return if inode in lookup_count: # may have been looked up before refcount became zero return # Inode is not referenced by any directory entries (so it cannot be # looked up), and it is not known to the kernel (so it cannot be # passed to any other handlers). The lock on inode is required not # just because increment/decrement of the reference count may not be # atomic, but also because an `unlink` handler may have already # decreased the reference count, but still want to do something with # the inode. delete_inode(inode) def unlink_entry(self, name): delete_name(name) inode = get_inode(name) with lock(inode): decr_refcount_for(inode) return inode def unlink(self, name): inode = self.unlink_entry(name) if inode not in lookup_count: self.maybe_remove_inode(inode) Here, the operations that modify lookup_count as well as the complete forget() function could be moved into pyfuse3. The price of this is that the application can no longer tell for sure if an inode is known to the kernel. This is a problem if e.g. inode numbers are generated dynamically - without forget(), how does the file system know when it can re-use an inode? Therefore, I've decided not to implement this feature. Applications have to keep track of the lookup count manually. ================================================ FILE: developer-notes/release_process.rst ================================================ Steps for Releasing a New Version --------------------------------- * `export NEWVER=XX.YY.Z` * Add release date and version to `Changes.rst` * `git commit --all -m "Released $NEWVER"` * `git tag v$NEWVER` * `uv sync --locked` * `uv run sphinx-build -b html rst doc/html` * `uv build --sdist` * `gpg --detach-sign --armor --output dist/pyfuse3-$NEWVER.tar.gz.asc dist/pyfuse3-$NEWVER.targ.nz` (or `util/sdist-sign $NEWVER`) * `uv run twine upload dist/pyfuse3-$NEWVER.tar.gz` (or `util/upload-pypi $NEWVER`) * Create release on GitHub (https://github.com/libfuse/pyfuse3/releases/) * Send announcement to mailing list * Get contributors: `git log --pretty="format:%an <%aE>" "${PREV_TAG}..v${NEWVER}" | sort -u` Announcement template: ---------------------- Dear all, I'm happy to announce a new release of pyfuse3, version . pyfuse3 is a set of Python 3 bindings for `libfuse 3`_. It provides an asynchronous API compatible with Trio_ and asyncio_, and enables you to easily write a full-featured Linux filesystem in Python. From the changelog: The following people have contributed code to this release: [PASTE HERE] As usual, the newest release can be downloaded from PyPi at https://pypi.python.org/pypi/pyfuse3/. Please report any bugs on the issue tracker at https://github.com/libfuse/pyfuse3/issues. For discussion and questions, please use the general FUSE mailing list (i.e., this list) or the GitHub discussion forum at https://github.com/libfuse/pyfuse3/discussions. ================================================ FILE: developer-notes/setup.md ================================================ # How to run/develop pyfuse3 from Git To run unit tests, build the documentation, and make changes to pyfuse3, the recommended procedure is to create a virtual environment and install pyfuse3, build dependencies, and development tools into this environment. You can do this using a tool like [uv](https://docs.astral.sh/uv/getting-started/installation/) or by hand as follows: ```sh $ python3 -m venv .venv # create the venv $ . .venv/bin/activate # activate it $ pip install --upgrade pip # upgrade pip $ pip install ".[dev]" # install build dependencies $ pip install --no-build-isolation --editable . # install pyfuse3 in editable mode ``` As long as the venv is active, you can run tests with ```sh $ pytest test/ ``` and build the HTML documentation and manpages with: ```sh $ sphinx-build -b html rst doc/html ``` ================================================ FILE: developer-notes/valgrind.md ================================================ To run tests under valgrind: - Build python `--with-valgrind --with-pydebug`. - Run `valgrind --trace-children=yes "--trace-children-skip=*mount*" python-dbg -m pytest test/` ================================================ FILE: doc/.placeholder ================================================ ================================================ FILE: examples/hello.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' hello.py - Example file system for pyfuse3. This program presents a static file system containing a single file. Copyright © 2015 Nikolaus Rath Copyright © 2015 Gerion Entrup. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' import errno import logging import os import stat from argparse import ArgumentParser, Namespace from typing import cast import trio import pyfuse3 from pyfuse3 import EntryAttributes, FileHandleT, FileInfo, InodeT, ReaddirToken, RequestContext try: import faulthandler except ImportError: pass else: faulthandler.enable() log = logging.getLogger(__name__) class TestFs(pyfuse3.Operations): def __init__(self) -> None: super(TestFs, self).__init__() self.hello_name = b"message" self.hello_inode = cast(InodeT, pyfuse3.ROOT_INODE + 1) self.hello_data = b"hello world\n" async def getattr(self, inode: InodeT, ctx: RequestContext | None = None) -> EntryAttributes: entry = EntryAttributes() if inode == pyfuse3.ROOT_INODE: entry.st_mode = stat.S_IFDIR | 0o755 entry.st_size = 0 elif inode == self.hello_inode: entry.st_mode = stat.S_IFREG | 0o644 entry.st_size = len(self.hello_data) else: raise pyfuse3.FUSEError(errno.ENOENT) stamp = int(1438467123.985654 * 1e9) entry.st_atime_ns = stamp entry.st_ctime_ns = stamp entry.st_mtime_ns = stamp entry.st_gid = os.getgid() entry.st_uid = os.getuid() entry.st_ino = inode return entry async def lookup( self, parent_inode: InodeT, name: bytes, ctx: RequestContext ) -> EntryAttributes: if parent_inode != pyfuse3.ROOT_INODE or name != self.hello_name: raise pyfuse3.FUSEError(errno.ENOENT) return await self.getattr(self.hello_inode, ctx) async def opendir(self, inode: InodeT, ctx: RequestContext) -> FileHandleT: if inode != pyfuse3.ROOT_INODE: raise pyfuse3.FUSEError(errno.ENOENT) # For simplicity, we use the inode as file handle return FileHandleT(inode) async def readdir(self, fh: FileHandleT, start_id: int, token: ReaddirToken) -> None: assert fh == pyfuse3.ROOT_INODE # only one entry if start_id == 0: pyfuse3.readdir_reply(token, self.hello_name, await self.getattr(self.hello_inode), 1) return async def open(self, inode: InodeT, flags: int, ctx: RequestContext) -> FileInfo: if inode != self.hello_inode: raise pyfuse3.FUSEError(errno.ENOENT) if flags & os.O_RDWR or flags & os.O_WRONLY: raise pyfuse3.FUSEError(errno.EACCES) # For simplicity, we use the inode as file handle return FileInfo(fh=FileHandleT(inode)) async def read(self, fh: FileHandleT, off: int, size: int) -> bytes: assert fh == self.hello_inode return self.hello_data[off : off + size] def init_logging(debug: bool = False) -> None: formatter = logging.Formatter( '%(asctime)s.%(msecs)03d %(threadName)s: [%(name)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S", ) handler = logging.StreamHandler() handler.setFormatter(formatter) root_logger = logging.getLogger() if debug: handler.setLevel(logging.DEBUG) root_logger.setLevel(logging.DEBUG) else: handler.setLevel(logging.INFO) root_logger.setLevel(logging.INFO) root_logger.addHandler(handler) def parse_args() -> Namespace: '''Parse command line''' parser = ArgumentParser() parser.add_argument('mountpoint', type=str, help='Where to mount the file system') parser.add_argument( '--debug', action='store_true', default=False, help='Enable debugging output' ) parser.add_argument( '--debug-fuse', action='store_true', default=False, help='Enable FUSE debugging output' ) return parser.parse_args() def main() -> None: options = parse_args() init_logging(options.debug) testfs = TestFs() fuse_options = set(pyfuse3.default_options) fuse_options.add('fsname=hello') if options.debug_fuse: fuse_options.add('debug') pyfuse3.init(testfs, options.mountpoint, fuse_options) try: trio.run(pyfuse3.main) except: pyfuse3.close(unmount=False) raise pyfuse3.close() if __name__ == '__main__': main() ================================================ FILE: examples/hello_asyncio.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' hello_asyncio.py - Example file system for pyfuse3 using asyncio. This program presents a static file system containing a single file. Copyright © 2015 Nikolaus Rath Copyright © 2015 Gerion Entrup. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' import asyncio import errno import logging import os import stat from argparse import ArgumentParser, Namespace from typing import cast import pyfuse3 import pyfuse3.asyncio from pyfuse3 import EntryAttributes, FileHandleT, FileInfo, InodeT, ReaddirToken, RequestContext try: import faulthandler except ImportError: pass else: faulthandler.enable() log = logging.getLogger(__name__) pyfuse3.asyncio.enable() class TestFs(pyfuse3.Operations): def __init__(self) -> None: super(TestFs, self).__init__() self.hello_name = b"message" self.hello_inode = cast(InodeT, pyfuse3.ROOT_INODE + 1) self.hello_data = b"hello world\n" async def getattr(self, inode: InodeT, ctx: RequestContext | None = None) -> EntryAttributes: entry = EntryAttributes() if inode == pyfuse3.ROOT_INODE: entry.st_mode = stat.S_IFDIR | 0o755 entry.st_size = 0 elif inode == self.hello_inode: entry.st_mode = stat.S_IFREG | 0o644 entry.st_size = len(self.hello_data) else: raise pyfuse3.FUSEError(errno.ENOENT) stamp = int(1438467123.985654 * 1e9) entry.st_atime_ns = stamp entry.st_ctime_ns = stamp entry.st_mtime_ns = stamp entry.st_gid = os.getgid() entry.st_uid = os.getuid() entry.st_ino = inode return entry async def lookup( self, parent_inode: InodeT, name: bytes, ctx: RequestContext ) -> EntryAttributes: if parent_inode != pyfuse3.ROOT_INODE or name != self.hello_name: raise pyfuse3.FUSEError(errno.ENOENT) return await self.getattr(self.hello_inode, ctx) async def opendir(self, inode: InodeT, ctx: RequestContext) -> FileHandleT: if inode != pyfuse3.ROOT_INODE: raise pyfuse3.FUSEError(errno.ENOENT) # For simplicity, we use the inode as file handle return FileHandleT(inode) async def readdir(self, fh: FileHandleT, start_id: int, token: ReaddirToken) -> None: assert fh == pyfuse3.ROOT_INODE # only one entry if start_id == 0: pyfuse3.readdir_reply(token, self.hello_name, await self.getattr(self.hello_inode), 1) return async def setxattr(self, inode: InodeT, name: bytes, value: bytes, ctx: RequestContext) -> None: if inode != pyfuse3.ROOT_INODE or name != b'command': raise pyfuse3.FUSEError(errno.ENOTSUP) if value == b'terminate': pyfuse3.terminate() else: raise pyfuse3.FUSEError(errno.EINVAL) async def open(self, inode: InodeT, flags: int, ctx: RequestContext) -> FileInfo: if inode != self.hello_inode: raise pyfuse3.FUSEError(errno.ENOENT) if flags & os.O_RDWR or flags & os.O_WRONLY: raise pyfuse3.FUSEError(errno.EACCES) # For simplicity, we use the inode as file handle return FileInfo(fh=FileHandleT(inode)) async def read(self, fh: FileHandleT, off: int, size: int) -> bytes: assert fh == self.hello_inode return self.hello_data[off : off + size] def init_logging(debug: bool = False) -> None: formatter = logging.Formatter( '%(asctime)s.%(msecs)03d %(threadName)s: [%(name)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S", ) handler = logging.StreamHandler() handler.setFormatter(formatter) root_logger = logging.getLogger() if debug: handler.setLevel(logging.DEBUG) root_logger.setLevel(logging.DEBUG) else: handler.setLevel(logging.INFO) root_logger.setLevel(logging.INFO) root_logger.addHandler(handler) def parse_args() -> Namespace: '''Parse command line''' parser = ArgumentParser() parser.add_argument('mountpoint', type=str, help='Where to mount the file system') parser.add_argument( '--debug', action='store_true', default=False, help='Enable debugging output' ) parser.add_argument( '--debug-fuse', action='store_true', default=False, help='Enable FUSE debugging output' ) return parser.parse_args() def main() -> None: options = parse_args() init_logging(options.debug) testfs = TestFs() fuse_options = set(pyfuse3.default_options) fuse_options.add('fsname=hello_asyncio') if options.debug_fuse: fuse_options.add('debug') pyfuse3.init(testfs, options.mountpoint, fuse_options) try: asyncio.run(pyfuse3.main()) except: pyfuse3.close(unmount=False) raise pyfuse3.close() if __name__ == '__main__': main() ================================================ FILE: examples/passthroughfs.py ================================================ #!/usr/bin/env python3 ''' passthroughfs.py - Example file system for pyfuse3 This file system mirrors the contents of a specified directory tree. Caveats: * Inode generation numbers are not passed through but set to zero. * Block size (st_blksize) and number of allocated blocks (st_blocks) are not passed through. * Performance for large directories is not good, because the directory is always read completely. * There may be a way to break-out of the directory tree. * The readdir implementation is not fully POSIX compliant. If a directory contains hardlinks and is modified during a readdir call, readdir() may return some of the hardlinked files twice or omit them completely. * If you delete or rename files in the underlying file system, the passthrough file system will get confused. Copyright © Nikolaus Rath Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' import errno import faulthandler import logging import os import stat as stat_m import sys from argparse import ArgumentParser, Namespace from collections import defaultdict from collections.abc import Sequence from os import fsdecode, fsencode from typing import cast import trio import pyfuse3 from pyfuse3 import ( EntryAttributes, FileHandleT, FileInfo, FUSEError, InodeT, ReaddirToken, RequestContext, SetattrFields, StatvfsData, ) faulthandler.enable() log = logging.getLogger(__name__) class Operations(pyfuse3.Operations): def __init__(self, source: str, enable_writeback_cache: bool = False) -> None: super().__init__() self.enable_writeback_cache = enable_writeback_cache self._inode_path_map: dict[InodeT, str | set[str]] = {pyfuse3.ROOT_INODE: source} self._lookup_cnt: defaultdict[InodeT, int] = defaultdict(lambda: 0) self._fd_inode_map: dict[int, InodeT] = dict() self._inode_fd_map: dict[InodeT, int] = dict() self._fd_open_count: dict[int, int] = dict() def _inode_to_path(self, inode: InodeT) -> str: try: val = self._inode_path_map[inode] except KeyError: raise FUSEError(errno.ENOENT) if isinstance(val, set): # In case of hardlinks, pick any path val = next(iter(val)) return val def _add_path(self, inode: InodeT, path: str) -> None: log.debug('_add_path for %d, %s', inode, path) self._lookup_cnt[inode] += 1 # With hardlinks, one inode may map to multiple paths. if inode not in self._inode_path_map: self._inode_path_map[inode] = path return val = self._inode_path_map[inode] if isinstance(val, set): val.add(path) elif val != path: self._inode_path_map[inode] = {path, val} async def forget(self, inode_list: Sequence[tuple[InodeT, int]]) -> None: for inode, nlookup in inode_list: if self._lookup_cnt[inode] > nlookup: self._lookup_cnt[inode] -= nlookup continue log.debug('forgetting about inode %d', inode) assert inode not in self._inode_fd_map del self._lookup_cnt[inode] try: del self._inode_path_map[inode] except KeyError: # may have been deleted pass async def lookup( self, parent_inode: InodeT, name: bytes, ctx: RequestContext ) -> EntryAttributes: name_str = fsdecode(name) log.debug('lookup for %s in %d', name_str, parent_inode) path = os.path.join(self._inode_to_path(parent_inode), name_str) attr = self._getattr(path=path) if name_str != '.' and name_str != '..': self._add_path(InodeT(attr.st_ino), path) return attr async def getattr(self, inode: InodeT, ctx: RequestContext | None = None) -> EntryAttributes: if inode in self._inode_fd_map: return self._getattr(fd=self._inode_fd_map[inode]) else: return self._getattr(path=self._inode_to_path(inode)) def _getattr(self, path: str | None = None, fd: int | None = None) -> EntryAttributes: assert fd is None or path is None assert not (fd is None and path is None) try: if fd is None: assert path is not None stat = os.lstat(path) else: stat = os.fstat(fd) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) entry = EntryAttributes() for attr in ( 'st_ino', 'st_mode', 'st_nlink', 'st_uid', 'st_gid', 'st_rdev', 'st_size', 'st_atime_ns', 'st_mtime_ns', 'st_ctime_ns', ): setattr(entry, attr, getattr(stat, attr)) entry.generation = 0 entry.entry_timeout = 0 entry.attr_timeout = 0 entry.st_blksize = 512 entry.st_blocks = (entry.st_size + entry.st_blksize - 1) // entry.st_blksize return entry async def readlink(self, inode: InodeT, ctx: RequestContext) -> bytes: path = self._inode_to_path(inode) try: target = os.readlink(path) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) return fsencode(target) async def opendir(self, inode: InodeT, ctx: RequestContext) -> FileHandleT: # For simplicity, we use the inode as file handle return FileHandleT(inode) async def readdir(self, fh: FileHandleT, start_id: int, token: ReaddirToken) -> None: path = self._inode_to_path(InodeT(fh)) log.debug('reading %s', path) entries: list[tuple[InodeT, str, EntryAttributes]] = [] for name in os.listdir(path): if name == '.' or name == '..': continue attr = self._getattr(path=os.path.join(path, name)) entries.append((InodeT(attr.st_ino), name, attr)) log.debug('read %d entries, starting at %d', len(entries), start_id) # This is not fully posix compatible. If there are hardlinks # (two names with the same inode), we don't have a unique # offset to start in between them. Note that we cannot simply # count entries, because then we would skip over entries # (or return them more than once) if the number of directory # entries changes between two calls to readdir(). for ino, name, attr in sorted(entries): if ino <= start_id: continue if not pyfuse3.readdir_reply(token, fsencode(name), attr, ino): break self._add_path(attr.st_ino, os.path.join(path, name)) async def unlink(self, parent_inode: InodeT, name: bytes, ctx: RequestContext) -> None: name_str = fsdecode(name) parent = self._inode_to_path(parent_inode) path = os.path.join(parent, name_str) try: inode = os.lstat(path).st_ino os.unlink(path) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) if inode in self._lookup_cnt: self._forget_path(InodeT(inode), path) async def rmdir(self, parent_inode: InodeT, name: bytes, ctx: RequestContext) -> None: name_str = fsdecode(name) parent = self._inode_to_path(parent_inode) path = os.path.join(parent, name_str) try: inode = os.lstat(path).st_ino os.rmdir(path) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) if inode in self._lookup_cnt: self._forget_path(InodeT(inode), path) def _forget_path(self, inode: InodeT, path: str) -> None: log.debug('forget %s for %d', path, inode) val = self._inode_path_map[inode] if isinstance(val, set): val.remove(path) if len(val) == 1: self._inode_path_map[inode] = next(iter(val)) else: del self._inode_path_map[inode] async def symlink( self, parent_inode: InodeT, name: bytes, target: bytes, ctx: RequestContext ) -> EntryAttributes: name_str = fsdecode(name) target_str = fsdecode(target) parent = self._inode_to_path(parent_inode) path = os.path.join(parent, name_str) try: os.symlink(target_str, path) os.lchown(path, ctx.uid, ctx.gid) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) inode = InodeT(os.lstat(path).st_ino) self._add_path(inode, path) return await self.getattr(inode, ctx) async def rename( self, parent_inode_old: InodeT, name_old: bytes, parent_inode_new: InodeT, name_new: bytes, flags: int, ctx: RequestContext, ) -> None: if flags != 0: raise FUSEError(errno.EINVAL) name_old_str = fsdecode(name_old) name_new_str = fsdecode(name_new) parent_old = self._inode_to_path(parent_inode_old) parent_new = self._inode_to_path(parent_inode_new) path_old = os.path.join(parent_old, name_old_str) path_new = os.path.join(parent_new, name_new_str) try: os.rename(path_old, path_new) inode = cast(InodeT, os.lstat(path_new).st_ino) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) if inode not in self._lookup_cnt: return val = self._inode_path_map[inode] if isinstance(val, set): assert len(val) > 1 val.add(path_new) val.remove(path_old) else: assert val == path_old self._inode_path_map[inode] = path_new async def link( self, inode: InodeT, new_parent_inode: InodeT, new_name: bytes, ctx: RequestContext ) -> EntryAttributes: new_name_str = fsdecode(new_name) parent = self._inode_to_path(new_parent_inode) path = os.path.join(parent, new_name_str) try: os.link(self._inode_to_path(inode), path, follow_symlinks=False) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) self._add_path(inode, path) return await self.getattr(inode, ctx) async def setattr( self, inode: InodeT, attr: EntryAttributes, fields: SetattrFields, fh: FileHandleT | None, ctx: RequestContext, ) -> EntryAttributes: try: if fields.update_size: if fh is None: os.truncate(self._inode_to_path(inode), attr.st_size) else: os.ftruncate(fh, attr.st_size) if fields.update_mode: # Under Linux, chmod always resolves symlinks so we should # actually never get a setattr() request for a symbolic # link. assert not stat_m.S_ISLNK(attr.st_mode) if fh is None: os.chmod(self._inode_to_path(inode), stat_m.S_IMODE(attr.st_mode)) else: os.fchmod(fh, stat_m.S_IMODE(attr.st_mode)) if fields.update_uid and fields.update_gid: if fh is None: os.chown( self._inode_to_path(inode), attr.st_uid, attr.st_gid, follow_symlinks=False ) else: os.fchown(fh, attr.st_uid, attr.st_gid) elif fields.update_uid: if fh is None: os.chown(self._inode_to_path(inode), attr.st_uid, -1, follow_symlinks=False) else: os.fchown(fh, attr.st_uid, -1) elif fields.update_gid: if fh is None: os.chown(self._inode_to_path(inode), -1, attr.st_gid, follow_symlinks=False) else: os.fchown(fh, -1, attr.st_gid) if fields.update_atime and fields.update_mtime: if fh is None: os.utime( self._inode_to_path(inode), None, follow_symlinks=False, ns=(attr.st_atime_ns, attr.st_mtime_ns), ) else: os.utime(fh, None, ns=(attr.st_atime_ns, attr.st_mtime_ns)) elif fields.update_atime or fields.update_mtime: # We can only set both values, so we first need to retrieve the # one that we shouldn't be changing. if fh is None: path = self._inode_to_path(inode) oldstat = os.stat(path, follow_symlinks=False) else: oldstat = os.fstat(fh) if not fields.update_atime: attr.st_atime_ns = oldstat.st_atime_ns else: attr.st_mtime_ns = oldstat.st_mtime_ns if fh is None: os.utime( path, # pyright: ignore[reportPossiblyUnboundVariable] None, follow_symlinks=False, ns=(attr.st_atime_ns, attr.st_mtime_ns), ) else: os.utime(fh, None, ns=(attr.st_atime_ns, attr.st_mtime_ns)) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) return await self.getattr(inode, ctx) async def mknod( self, parent_inode: InodeT, name: bytes, mode: int, rdev: int, ctx: RequestContext ) -> EntryAttributes: path = os.path.join(self._inode_to_path(parent_inode), fsdecode(name)) try: os.mknod(path, mode=(mode & ~ctx.umask), device=rdev) os.chown(path, ctx.uid, ctx.gid) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) attr = self._getattr(path=path) self._add_path(attr.st_ino, path) return attr async def mkdir( self, parent_inode: InodeT, name: bytes, mode: int, ctx: RequestContext ) -> EntryAttributes: path = os.path.join(self._inode_to_path(parent_inode), fsdecode(name)) try: os.mkdir(path, mode=(mode & ~ctx.umask)) os.chown(path, ctx.uid, ctx.gid) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) attr = self._getattr(path=path) self._add_path(attr.st_ino, path) return attr async def statfs(self, ctx: RequestContext) -> StatvfsData: root = self._inode_path_map[pyfuse3.ROOT_INODE] assert isinstance(root, str) stat_ = StatvfsData() try: statfs = os.statvfs(root) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) for attr in ( 'f_bsize', 'f_frsize', 'f_blocks', 'f_bfree', 'f_bavail', 'f_files', 'f_ffree', 'f_favail', ): setattr(stat_, attr, getattr(statfs, attr)) stat_.f_namemax = statfs.f_namemax - (len(root) + 1) return stat_ async def open(self, inode: InodeT, flags: int, ctx: RequestContext) -> FileInfo: if inode in self._inode_fd_map: fd = self._inode_fd_map[inode] self._fd_open_count[fd] += 1 return FileInfo(fh=FileHandleT(fd)) assert flags & os.O_CREAT == 0 try: fd = os.open(self._inode_to_path(inode), flags) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) self._inode_fd_map[inode] = fd self._fd_inode_map[fd] = inode self._fd_open_count[fd] = 1 return FileInfo(fh=cast(FileHandleT, fd)) async def create( self, parent_inode: InodeT, name: bytes, mode: int, flags: int, ctx: RequestContext ) -> tuple[FileInfo, EntryAttributes]: path = os.path.join(self._inode_to_path(parent_inode), fsdecode(name)) try: fd = os.open(path, flags | os.O_CREAT | os.O_TRUNC) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) attr = self._getattr(fd=fd) self._add_path(attr.st_ino, path) self._inode_fd_map[attr.st_ino] = fd self._fd_inode_map[fd] = attr.st_ino self._fd_open_count[fd] = 1 return (FileInfo(fh=cast(FileHandleT, fd)), attr) async def read(self, fh: FileHandleT, off: int, size: int) -> bytes: os.lseek(fh, off, os.SEEK_SET) return os.read(fh, size) async def write(self, fh: FileHandleT, off: int, buf: bytes) -> int: os.lseek(fh, off, os.SEEK_SET) return os.write(fh, buf) async def release(self, fh: FileHandleT) -> None: if self._fd_open_count[fh] > 1: self._fd_open_count[fh] -= 1 return del self._fd_open_count[fh] inode = self._fd_inode_map[fh] del self._inode_fd_map[inode] del self._fd_inode_map[fh] try: os.close(fh) except OSError as exc: assert exc.errno is not None raise FUSEError(exc.errno) def init_logging(debug: bool = False) -> None: formatter = logging.Formatter( '%(asctime)s.%(msecs)03d %(threadName)s: [%(name)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S", ) handler = logging.StreamHandler() handler.setFormatter(formatter) root_logger = logging.getLogger() if debug: handler.setLevel(logging.DEBUG) root_logger.setLevel(logging.DEBUG) else: handler.setLevel(logging.INFO) root_logger.setLevel(logging.INFO) root_logger.addHandler(handler) def parse_args(args: list[str]) -> Namespace: '''Parse command line''' parser = ArgumentParser() parser.add_argument('source', type=str, help='Directory tree to mirror') parser.add_argument('mountpoint', type=str, help='Where to mount the file system') parser.add_argument( '--debug', action='store_true', default=False, help='Enable debugging output' ) parser.add_argument( '--debug-fuse', action='store_true', default=False, help='Enable FUSE debugging output' ) parser.add_argument( '--enable-writeback-cache', action='store_true', default=False, help='Enable writeback cache (default: disabled)', ) return parser.parse_args(args) def main() -> None: options = parse_args(sys.argv[1:]) init_logging(options.debug) operations = Operations(options.source, enable_writeback_cache=options.enable_writeback_cache) log.debug('Mounting...') fuse_options = set(pyfuse3.default_options) fuse_options.add('fsname=passthroughfs') if options.debug_fuse: fuse_options.add('debug') pyfuse3.init(operations, options.mountpoint, fuse_options) try: log.debug('Entering main loop..') trio.run(pyfuse3.main) except: pyfuse3.close(unmount=False) raise log.debug('Unmounting..') pyfuse3.close() if __name__ == '__main__': main() ================================================ FILE: examples/tmpfs.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' tmpfs.py - Example file system for pyfuse3. This file system stores all data in memory. Copyright © 2013 Nikolaus Rath Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' import errno import logging import os import sqlite3 import stat from argparse import ArgumentParser, Namespace from collections import defaultdict from time import time from typing import Any, cast import trio import pyfuse3 from pyfuse3 import ( EntryAttributes, FileHandleT, FileInfo, FUSEError, InodeT, ReaddirToken, RequestContext, SetattrFields, StatvfsData, ) try: import faulthandler except ImportError: pass else: faulthandler.enable() log = logging.getLogger() class Operations(pyfuse3.Operations): '''An example filesystem that stores all data in memory This is a very simple implementation with terrible performance. Don't try to store significant amounts of data. Also, there are some other flaws that have not been fixed to keep the code easier to understand: * atime, mtime and ctime are not updated * generation numbers are not supported * lookup counts are not maintained ''' enable_writeback_cache = True def __init__(self) -> None: super(Operations, self).__init__() self.db: sqlite3.Connection = sqlite3.connect(':memory:') self.db.text_factory = str self.db.row_factory = sqlite3.Row self.cursor: sqlite3.Cursor = self.db.cursor() self.inode_open_count: defaultdict[InodeT, int] = defaultdict(int) self.init_tables() def init_tables(self) -> None: '''Initialize file system tables''' self.cursor.execute(""" CREATE TABLE inodes ( id INTEGER PRIMARY KEY, uid INT NOT NULL, gid INT NOT NULL, mode INT NOT NULL, mtime_ns INT NOT NULL, atime_ns INT NOT NULL, ctime_ns INT NOT NULL, target BLOB(256) , size INT NOT NULL DEFAULT 0, rdev INT NOT NULL DEFAULT 0, data BLOB ) """) self.cursor.execute(""" CREATE TABLE contents ( rowid INTEGER PRIMARY KEY AUTOINCREMENT, name BLOB(256) NOT NULL, inode INT NOT NULL REFERENCES inodes(id), parent_inode INT NOT NULL REFERENCES inodes(id), UNIQUE (name, parent_inode) )""") # Insert root directory now_ns = int(time() * 1e9) self.cursor.execute( "INSERT INTO inodes (id,mode,uid,gid,mtime_ns,atime_ns,ctime_ns) " "VALUES (?,?,?,?,?,?,?)", ( pyfuse3.ROOT_INODE, stat.S_IFDIR | stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH, os.getuid(), os.getgid(), now_ns, now_ns, now_ns, ), ) self.cursor.execute( "INSERT INTO contents (name, parent_inode, inode) VALUES (?,?,?)", (b'..', pyfuse3.ROOT_INODE, pyfuse3.ROOT_INODE), ) def get_row(self, *a: Any, **kw: Any) -> sqlite3.Row: self.cursor.execute(*a, **kw) try: row = next(self.cursor) except StopIteration: raise NoSuchRowError() try: next(self.cursor) except StopIteration: pass else: raise NoUniqueValueError() return row async def lookup( self, parent_inode: InodeT, name: bytes, ctx: RequestContext ) -> EntryAttributes: if name == b'.': inode = parent_inode elif name == b'..': inode = self.get_row("SELECT * FROM contents WHERE inode=?", (parent_inode,))[ 'parent_inode' ] else: try: inode = self.get_row( "SELECT * FROM contents WHERE name=? AND parent_inode=?", (name, parent_inode) )['inode'] except NoSuchRowError: raise (pyfuse3.FUSEError(errno.ENOENT)) return await self.getattr(InodeT(inode), ctx) async def getattr(self, inode: InodeT, ctx: RequestContext | None = None) -> EntryAttributes: try: row = self.get_row("SELECT * FROM inodes WHERE id=?", (inode,)) except NoSuchRowError: raise (pyfuse3.FUSEError(errno.ENOENT)) entry = EntryAttributes() entry.st_ino = inode entry.generation = 0 entry.entry_timeout = 300 entry.attr_timeout = 300 entry.st_mode = row['mode'] entry.st_nlink = self.get_row("SELECT COUNT(inode) FROM contents WHERE inode=?", (inode,))[ 0 ] entry.st_uid = row['uid'] entry.st_gid = row['gid'] entry.st_rdev = row['rdev'] entry.st_size = row['size'] entry.st_blksize = 512 entry.st_blocks = 1 entry.st_atime_ns = row['atime_ns'] entry.st_mtime_ns = row['mtime_ns'] entry.st_ctime_ns = row['ctime_ns'] return entry async def readlink(self, inode: InodeT, ctx: RequestContext) -> bytes: return self.get_row('SELECT * FROM inodes WHERE id=?', (inode,))['target'] async def opendir(self, inode: InodeT, ctx: RequestContext) -> FileHandleT: # For simplicity, we use the inode as file handle return FileHandleT(inode) async def readdir(self, fh: FileHandleT, start_id: int, token: ReaddirToken) -> None: if start_id == 0: off = -1 else: off = start_id cursor2 = self.db.cursor() cursor2.execute( "SELECT * FROM contents WHERE parent_inode=? AND rowid > ? ORDER BY rowid", (fh, off) ) for row in cursor2: pyfuse3.readdir_reply( token, row['name'], await self.getattr(InodeT(row['inode'])), row['rowid'] ) async def unlink(self, parent_inode: InodeT, name: bytes, ctx: RequestContext) -> None: entry = await self.lookup(parent_inode, name, ctx) if stat.S_ISDIR(entry.st_mode): raise pyfuse3.FUSEError(errno.EISDIR) self._remove(parent_inode, name, entry) async def rmdir(self, parent_inode: InodeT, name: bytes, ctx: RequestContext) -> None: entry = await self.lookup(parent_inode, name, ctx) if not stat.S_ISDIR(entry.st_mode): raise pyfuse3.FUSEError(errno.ENOTDIR) self._remove(parent_inode, name, entry) def _remove(self, parent_inode: InodeT, name: bytes, entry: EntryAttributes) -> None: if ( self.get_row("SELECT COUNT(inode) FROM contents WHERE parent_inode=?", (entry.st_ino,))[ 0 ] > 0 ): raise pyfuse3.FUSEError(errno.ENOTEMPTY) self.cursor.execute( "DELETE FROM contents WHERE name=? AND parent_inode=?", (name, parent_inode) ) if entry.st_nlink == 1 and entry.st_ino not in self.inode_open_count: self.cursor.execute("DELETE FROM inodes WHERE id=?", (entry.st_ino,)) async def symlink( self, parent_inode: InodeT, name: bytes, target: bytes, ctx: RequestContext ) -> EntryAttributes: mode = ( stat.S_IFLNK | stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IWOTH | stat.S_IXOTH ) return await self._create(parent_inode, name, mode, ctx, target=target) async def rename( self, parent_inode_old: InodeT, name_old: bytes, parent_inode_new: InodeT, name_new: bytes, flags: int, ctx: RequestContext, ) -> None: if flags != 0: raise FUSEError(errno.EINVAL) entry_old = await self.lookup(parent_inode_old, name_old, ctx) entry_new = None try: entry_new = await self.lookup( parent_inode_new, name_new if isinstance(name_new, bytes) else name_new.encode(), ctx, ) except pyfuse3.FUSEError as exc: if exc.errno != errno.ENOENT: raise if entry_new is not None: self._replace( parent_inode_old, name_old, parent_inode_new, name_new, entry_old, entry_new ) else: self.cursor.execute( "UPDATE contents SET name=?, parent_inode=? WHERE name=? AND parent_inode=?", (name_new, parent_inode_new, name_old, parent_inode_old), ) def _replace( self, parent_inode_old: InodeT, name_old: bytes, parent_inode_new: InodeT, name_new: bytes, entry_old: EntryAttributes, entry_new: EntryAttributes, ) -> None: if ( self.get_row( "SELECT COUNT(inode) FROM contents WHERE parent_inode=?", (entry_new.st_ino,) )[0] > 0 ): raise pyfuse3.FUSEError(errno.ENOTEMPTY) self.cursor.execute( "UPDATE contents SET inode=? WHERE name=? AND parent_inode=?", (entry_old.st_ino, name_new, parent_inode_new), ) self.db.execute( 'DELETE FROM contents WHERE name=? AND parent_inode=?', (name_old, parent_inode_old) ) if entry_new.st_nlink == 1 and entry_new.st_ino not in self.inode_open_count: self.cursor.execute("DELETE FROM inodes WHERE id=?", (entry_new.st_ino,)) async def link( self, inode: InodeT, new_parent_inode: InodeT, new_name: bytes, ctx: RequestContext ) -> EntryAttributes: entry_p = await self.getattr(new_parent_inode, ctx) if entry_p.st_nlink == 0: log.warning( 'Attempted to create entry %s with unlinked parent %d', new_name, new_parent_inode ) raise FUSEError(errno.EINVAL) self.cursor.execute( "INSERT INTO contents (name, inode, parent_inode) VALUES(?,?,?)", (new_name, inode, new_parent_inode), ) return await self.getattr(inode, ctx) async def setattr( self, inode: InodeT, attr: EntryAttributes, fields: SetattrFields, fh: FileHandleT | None, ctx: RequestContext, ) -> EntryAttributes: if fields.update_size: data = self.get_row('SELECT data FROM inodes WHERE id=?', (inode,))[0] if data is None: data = b'' if len(data) < attr.st_size: data = data + b'\0' * (attr.st_size - len(data)) else: data = data[: attr.st_size] self.cursor.execute( 'UPDATE inodes SET data=?, size=? WHERE id=?', (memoryview(data), attr.st_size, inode), ) if fields.update_mode: self.cursor.execute('UPDATE inodes SET mode=? WHERE id=?', (attr.st_mode, inode)) if fields.update_uid: self.cursor.execute('UPDATE inodes SET uid=? WHERE id=?', (attr.st_uid, inode)) if fields.update_gid: self.cursor.execute('UPDATE inodes SET gid=? WHERE id=?', (attr.st_gid, inode)) if fields.update_atime: self.cursor.execute( 'UPDATE inodes SET atime_ns=? WHERE id=?', (attr.st_atime_ns, inode) ) if fields.update_mtime: self.cursor.execute( 'UPDATE inodes SET mtime_ns=? WHERE id=?', (attr.st_mtime_ns, inode) ) if fields.update_ctime: self.cursor.execute( 'UPDATE inodes SET ctime_ns=? WHERE id=?', (attr.st_ctime_ns, inode) ) else: self.cursor.execute( 'UPDATE inodes SET ctime_ns=? WHERE id=?', (int(time() * 1e9), inode) ) return await self.getattr(inode, ctx) async def mknod( self, parent_inode: InodeT, name: bytes, mode: int, rdev: int, ctx: RequestContext ) -> EntryAttributes: return await self._create(parent_inode, name, mode, ctx, rdev=rdev) async def mkdir( self, parent_inode: InodeT, name: bytes, mode: int, ctx: RequestContext ) -> EntryAttributes: return await self._create(parent_inode, name, mode, ctx) async def statfs(self, ctx: RequestContext) -> StatvfsData: stat_ = StatvfsData() stat_.f_bsize = 512 stat_.f_frsize = 512 size = self.get_row('SELECT SUM(size) FROM inodes')[0] stat_.f_blocks = size // stat_.f_frsize stat_.f_bfree = max(size // stat_.f_frsize, 1024) stat_.f_bavail = stat_.f_bfree inodes = self.get_row('SELECT COUNT(id) FROM inodes')[0] stat_.f_files = inodes stat_.f_ffree = max(inodes, 100) stat_.f_favail = stat_.f_ffree return stat_ async def open(self, inode: InodeT, flags: int, ctx: RequestContext) -> FileInfo: self.inode_open_count[inode] += 1 # For simplicity, we use the inode as file handle return FileInfo(fh=FileHandleT(inode)) async def access(self, inode: InodeT, mode: int, ctx: RequestContext) -> bool: # Yeah, could be a function and has unused arguments # pylint: disable=R0201,W0613 return True async def create( self, parent_inode: InodeT, name: bytes, mode: int, flags: int, ctx: RequestContext ) -> tuple[FileInfo, EntryAttributes]: # pylint: disable=W0612 entry = await self._create(parent_inode, name, mode, ctx) self.inode_open_count[entry.st_ino] += 1 # For simplicity, we use the inode as file handle return (FileInfo(fh=FileHandleT(entry.st_ino)), entry) async def _create( self, parent_inode: InodeT, name: bytes, mode: int, ctx: RequestContext, rdev: int = 0, target: bytes | None = None, ) -> EntryAttributes: if (await self.getattr(parent_inode, ctx)).st_nlink == 0: log.warning('Attempted to create entry %s with unlinked parent %d', name, parent_inode) raise FUSEError(errno.EINVAL) now_ns = int(time() * 1e9) self.cursor.execute( 'INSERT INTO inodes (uid, gid, mode, mtime_ns, atime_ns, ' 'ctime_ns, target, rdev) VALUES(?, ?, ?, ?, ?, ?, ?, ?)', (ctx.uid, ctx.gid, mode, now_ns, now_ns, now_ns, target, rdev), ) inode = cast(InodeT, self.cursor.lastrowid) self.db.execute( "INSERT INTO contents(name, inode, parent_inode) VALUES(?,?,?)", (name, inode, parent_inode), ) return await self.getattr(inode, ctx) async def read(self, fh: FileHandleT, off: int, size: int) -> bytes: data = self.get_row('SELECT data FROM inodes WHERE id=?', (fh,))[0] if data is None: data = b'' return data[off : off + size] async def write(self, fh: FileHandleT, off: int, buf: bytes) -> int: data = self.get_row('SELECT data FROM inodes WHERE id=?', (fh,))[0] if data is None: data = b'' data = data[:off] + buf + data[off + len(buf) :] self.cursor.execute( 'UPDATE inodes SET data=?, size=? WHERE id=?', (memoryview(data), len(data), fh) ) return len(buf) async def release(self, fh: FileHandleT) -> None: inode = cast(InodeT, fh) self.inode_open_count[inode] -= 1 if self.inode_open_count[inode] == 0: del self.inode_open_count[inode] if (await self.getattr(inode)).st_nlink == 0: self.cursor.execute("DELETE FROM inodes WHERE id=?", (inode,)) class NoUniqueValueError(Exception): def __str__(self) -> str: return 'Query generated more than 1 result row' class NoSuchRowError(Exception): def __str__(self) -> str: return 'Query produced 0 result rows' def init_logging(debug: bool = False) -> None: formatter = logging.Formatter( '%(asctime)s.%(msecs)03d %(threadName)s: [%(name)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S", ) handler = logging.StreamHandler() handler.setFormatter(formatter) root_logger = logging.getLogger() if debug: handler.setLevel(logging.DEBUG) root_logger.setLevel(logging.DEBUG) else: handler.setLevel(logging.INFO) root_logger.setLevel(logging.INFO) root_logger.addHandler(handler) def parse_args() -> Namespace: '''Parse command line''' parser = ArgumentParser() parser.add_argument('mountpoint', type=str, help='Where to mount the file system') parser.add_argument( '--debug', action='store_true', default=False, help='Enable debugging output' ) parser.add_argument( '--debug-fuse', action='store_true', default=False, help='Enable FUSE debugging output' ) return parser.parse_args() if __name__ == '__main__': options = parse_args() init_logging(options.debug) operations = Operations() fuse_options = set(pyfuse3.default_options) fuse_options.add('fsname=tmpfs') fuse_options.discard('default_permissions') if options.debug_fuse: fuse_options.add('debug') pyfuse3.init(operations, options.mountpoint, fuse_options) try: trio.run(pyfuse3.main) except: pyfuse3.close(unmount=False) raise pyfuse3.close() ================================================ FILE: pyproject.toml ================================================ [build-system] requires = ["setuptools>=78.1.1", "setuptools_scm>=8.0", "Cython"] build-backend = "build_backend" backend-path = ["util"] [project] name = "pyfuse3" dynamic = ["version"] description = "Python 3 bindings for libfuse 3 with async I/O support" readme = "README.rst" requires-python = ">=3.10" license = "LGPL-2.1-or-later" authors = [ {name = "Nikolaus Rath", email = "Nikolaus@rath.org"} ] keywords = ["FUSE", "python"] classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Programming Language :: Python", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Filesystems", "Operating System :: POSIX :: Linux", "Typing :: Typed", ] dependencies = [ "trio >= 0.15", ] [project.urls] Homepage = "https://github.com/libfuse/pyfuse3" [dependency-groups] dev = [ "pyright>=1.1.407", "mypy>=1.19.1", "pytest >= 3.4.0", "pytest-trio", "ruff>=0.14.10", "sphinx", "twine", ] [tool.setuptools_scm] [tool.setuptools] include-package-data = true [tool.setuptools.packages.find] where = ["src"] [tool.setuptools.package-data] pyfuse3 = ["py.typed"] [tool.ruff] line-length = 100 extend-exclude = [ "developer-notes/", ] [tool.ruff.lint.isort] combine-as-imports = true case-sensitive = false [tool.ruff.lint] ignore = [ "E731", # Do not assign a lambda expression, use a def ] extend-select = [ 'RUF100', # Warn about unused suppressions 'I', # Import ordering ] [tool.mypy] exclude = [ "^util/.+", "^rst/conf.py$" ] warn_unused_configs = true disallow_untyped_defs = false check_untyped_defs = true warn_redundant_casts = true warn_unused_ignores = true [tool.pyright] typeCheckingMode = "standard" exclude = [ "**/__pycache__", "**/.*", "util/", "rst/conf.py" ] # Need for pyright to resolve tests importing tests/util.py (when pytest runs the # test, it adds the tests/ directory to sys.path) extraPaths = ['test'] [tool.codespell] skip = '.git,*.html,developer-notes/' ignore-words-list = 're-use,re-used' [tool.ruff.format] quote-style = "preserve" ================================================ FILE: rst/_static/.placeholder ================================================ ================================================ FILE: rst/_templates/localtoc.html ================================================

{{ _('Table Of Contents') }}

{{ toctree() }} ================================================ FILE: rst/about.rst ================================================ ======= About ======= .. include:: ../README.rst :start-after: start-intro ================================================ FILE: rst/asyncio.rst ================================================ .. _asyncio: ================= asyncio Support ================= By default, pyfuse3 uses asynchronous I/O using Trio_ (and most of the documentation assumes that you are using Trio). If you'd rather use asyncio, import the *pyfuse3.asyncio* module and call its *enable()* function before using *pyfuse3*. For example:: import pyfuse3 import pyfuse3.asyncio pyfuse3.asyncio.enable() # Use pyfuse3 as usual from here on. .. _Trio: https://github.com/python-trio/trio ================================================ FILE: rst/changes.rst ================================================ .. include:: ../Changes.rst ================================================ FILE: rst/conf.py ================================================ # -*- coding: utf-8 -*- # # pyfuse3 documentation build configuration file, created by # sphinx-quickstart on Sat Oct 16 14:14:40 2010. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # -- General configuration ----------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx'] # Link to Python standard library intersphinx_mapping = { 'python': ('https://docs.python.org/3/', None), 'trio': ('https://trio.readthedocs.io/en/stable/', None), } # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' autodoc_docstring_signature = True # The encoding of source files. source_encoding = 'utf-8' # The master toctree document. master_doc = 'index' # Warn about unresolved references nitpicky = True # General information about the project. project = 'pyfuse3' copyright = '2010-2025, Nikolaus Rath' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. try: from importlib.metadata import version version = version('pyfuse3') except Exception: # Fallback version if package is not installed version = 'dev' # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: # today = '' # Else, today_fmt is used as the format for a strftime call. # today_fmt = '%B %d, %Y' # List of documents that shouldn't be included in the build. unused_docs = [] # List of directories, relative to source directory, that shouldn't be searched # for source files. exclude_trees = [] # The reST default role (used for this markup: `text`) to use for all documents. default_role = 'py:obj' primary_domain = 'py' # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # pygments_style = 'colorful' highlight_language = 'python' # A list of ignored prefixes for module index sorting. # modindex_common_prefix = [] def setup(app): # Mangle NewTypes re-exported from pyfuse3._pyfuse3 so they appear to # come from their canonical location at the top of the package import pyfuse3 for name in ('FileHandleT', 'InodeT'): getattr(pyfuse3, name).__module__ = 'pyfuse3' # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. # html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". # html_title = None # A shorter title for the navigation bar. Default is the same as html_title. # html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. # html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. # html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. # html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. # html_use_smartypants = True # Custom sidebar templates, maps document names to template names. # html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. # html_additional_pages = {} # If false, no module index is generated. html_use_modindex = False # If false, no index is generated. html_use_index = True # If true, the index is split into individual pages for each letter. html_split_index = False # If true, links to the reST sources are added to the pages. html_show_sourcelink = False # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. # html_use_opensearch = '' # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). # html_file_suffix = '' # Output file base name for HTML help builder. htmlhelp_basename = 'pyfuse3doc' ================================================ FILE: rst/data.rst ================================================ ================= Data Structures ================= .. currentmodule:: pyfuse3 .. py:data:: ENOATTR This errorcode is unfortunately missing in the `errno` module, so it is provided by pyfuse3 instead. .. py:data:: ROOT_INODE The inode of the root directory, i.e. the mount point of the file system. .. py:data:: RENAME_EXCHANGE A flag that may be passed to the `~Operations.rename` handler. When passed, the handler must atomically exchange the two paths (which must both exist). .. py:data:: RENAME_NOREPLACE A flag that may be passed to the `~Operations.rename` handler. When passed, the handler must not replace an existing target. .. py:data:: default_options This is a recommended set of options that should be passed to `pyfuse3.init` to get reasonable behavior and performance. pyfuse3 is compatible with any other combination of options as well, but you should only deviate from the defaults with good reason. (The :samp:`fsname=` option is guaranteed never to be included in the default options, so you can always safely add it to the set). The default options are: * ``default_permissions`` enables permission checking by kernel. Without this any umask (or uid/gid) would not have an effect. .. autoexception:: FUSEError .. autoclass currently doesn't work for NewTypes .. https://github.com/sphinx-doc/sphinx/issues/11552 .. class:: FileHandleT A subclass of `int`, representing an integer file handle produced by a `~Operations.create`, `~Operations.open`, or `~Operations.opendir` call. .. class:: FileNameT A subclass of `bytes`, representing a file name, with no embedded zero-bytes (``\0``). .. class:: FlagT A subclass of `int`, representing flags modifying the behavior of an operation. .. class:: InodeT A subclass of `int`, representing an inode number. .. class:: ModeT A subclass of `int`, representing a file mode. .. class:: XAttrNameT A subclass of `bytes`, representing an extended attribute name, with no embedded zero-bytes (``\0``). .. autoclass:: RequestContext .. attribute:: pid .. attribute:: uid .. attribute:: gid .. attribute:: umask .. autoclass:: StatvfsData .. attribute:: f_bsize .. attribute:: f_frsize .. attribute:: f_blocks .. attribute:: f_bfree .. attribute:: f_bavail .. attribute:: f_files .. attribute:: f_ffree .. attribute:: f_favail .. attribute:: f_namemax .. autoclass:: EntryAttributes .. autoattribute:: st_ino .. autoattribute:: generation .. autoattribute:: entry_timeout .. autoattribute:: attr_timeout .. autoattribute:: st_mode .. autoattribute:: st_nlink .. autoattribute:: st_uid .. autoattribute:: st_gid .. autoattribute:: st_rdev .. autoattribute:: st_size .. autoattribute:: st_blksize .. autoattribute:: st_blocks .. autoattribute:: st_atime_ns .. autoattribute:: st_ctime_ns .. autoattribute:: st_mtime_ns .. autoclass:: FileInfo .. autoattribute:: fh This attribute must be set to the file handle to be returned from `Operations.open`. .. autoattribute:: direct_io If true, signals to the kernel that this file should not be cached or buffered. .. autoattribute:: keep_cache If true, signals to the kernel that previously cached data for this inode is still valid, and should not be invalidated. .. autoattribute:: nonseekable If true, indicates that the file does not support seeking. .. autoclass:: SetattrFields .. attribute:: update_atime If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_atime_ns` field contains an updated value. .. attribute:: update_mtime If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_mtime_ns` field contains an updated value. .. attribute:: update_mode If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_mode` field contains an updated value. .. attribute:: update_uid If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_uid` field contains an updated value. .. attribute:: update_gid If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_gid` field contains an updated value. .. attribute:: update_size If this attribute is true, it signals the `Operations.setattr` method that the `~EntryAttributes.st_size` field contains an updated value. .. autoclass:: ReaddirToken An identifier for a particular `~Operations.readdir` invocation. ================================================ FILE: rst/example.rst ================================================ .. _example file system: ====================== Example File Systems ====================== pyfuse3 comes with several example file systems in the :file:`examples` directory of the release tarball. For completeness, these examples are also included here. Single-file, Read-only File System ================================== (shipped as :file:`examples/lltest.py`) .. literalinclude:: ../examples/hello.py :linenos: :language: python In-memory File System ===================== (shipped as :file:`examples/tmpfs.py`) .. literalinclude:: ../examples/tmpfs.py :linenos: :language: python Passthrough / Overlay File System ================================= (shipped as :file:`examples/passthroughfs.py`) .. literalinclude:: ../examples/passthroughfs.py :linenos: :language: python ================================================ FILE: rst/fuse_api.rst ================================================ ==================== FUSE API Functions ==================== .. currentmodule:: pyfuse3 .. autofunction:: init .. autofunction:: main .. autofunction:: terminate .. autofunction:: close .. autofunction:: invalidate_inode .. autofunction:: invalidate_entry .. autofunction:: invalidate_entry_async .. autofunction:: notify_store .. autofunction:: readdir_reply .. py:data:: trio_token Set to the value returned by `trio.lowlevel.current_trio_token` while `main` is running. Can be used by other threads to run code in the main loop through `trio.from_thread.run`. ================================================ FILE: rst/general.rst ================================================ ===================== General Information ===================== .. currentmodule:: pyfuse3 .. _getting_started: Getting started =============== A file system is implemented by subclassing the `pyfuse3.Operations` class and implementing the various request handlers. The handlers respond to requests received from the FUSE kernel module and perform functions like looking up the inode given a file name, looking up attributes of an inode, opening a (file) inode for reading or writing or listing the contents of a (directory) inode. By default, pyfuse3 uses asynchronous I/O using Trio_, and most of the documentation assumes that you are using Trio. If you'd rather use asyncio, take a look at :ref:`asyncio Support `. If you would like to use Trio (which is recommended) but you have not yet used Trio before, please read the `Trio tutorial`_ first. An instance of the operations class is passed to `pyfuse3.init` to mount the file system. To enter the request handling loop, run `pyfuse3.main` in a trio event loop. This function will return when the file system should be unmounted again, which is done by calling `pyfuse3.close`. All character data (directory entry names, extended attribute names and values, symbolic link targets etc) are passed as `bytes` and must be returned as `bytes`. For easier debugging, it is strongly recommended that applications using pyfuse3 also make use of the faulthandler_ module. .. _faulthandler: http://docs.python.org/3/library/faulthandler.html .. _Trio tutorial: https://trio.readthedocs.io/en/latest/tutorial.html .. _Trio: https://github.com/python-trio/trio Lookup Counts ============= Most file systems need to keep track which inodes are currently known to the kernel. This is, for example, necessary to correctly implement the *unlink* system call: when unlinking a directory entry whose associated inode is currently opened, the file system must defer removal of the inode (and thus the file contents) until it is no longer in use by any process. FUSE file systems achieve this by using "lookup counts". A lookup count is a number that's associated with an inode. An inode with a lookup count of zero is currently not known to the kernel. This means that if there are no directory entries referring to such an inode it can be safely removed, or (if a file system implements dynamic inode numbers), the inode number can be safely recycled. The lookup count of an inode is increased by every operation that could make the inode "known" to the kernel. This includes e.g. `~Operations.lookup`, `~Operations.create` and `~Operations.readdir` (to determine if a given request handler affects the lookup count, please refer to its description in the `Operations` class). The lookup count is decreased by calls to the `~Operations.forget` handler. FUSE and VFS Locking ==================== FUSE and the kernel's VFS layer provide some basic locking that FUSE file systems automatically take advantage of. Specifically: * Calls to `~Operations.rename`, `~Operations.create`, `~Operations.symlink`, `~Operations.mknod`, `~Operations.link` and `~Operations.mkdir` acquire a write-lock on the inode of the directory in which the respective operation takes place (two in case of rename). * Calls to `~Operations.lookup` acquire a read-lock on the inode of the parent directory (meaning that lookups in the same directory may run concurrently, but never at the same time as e.g. a rename or mkdir operation). * Unless writeback caching is enabled, calls to `~Operations.write` for the same inode are automatically serialized (i.e., there are never concurrent calls for the same inode even when multithreading is enabled). ================================================ FILE: rst/gotchas.rst ================================================ ================ Common Gotchas ================ .. currentmodule:: pyfuse3 This chapter lists some common gotchas that should be avoided. Removing inodes in unlink handler ================================= If your file system is mounted at :file:`mnt`, the following code should complete without errors:: with open('mnt/file_one', 'w+') as fh1: fh1.write('foo') fh1.flush() with open('mnt/file_one', 'a') as fh2: os.unlink('mnt/file_one') assert 'file_one' not in os.listdir('mnt') fh2.write('bar') os.close(os.dup(fh1.fileno())) fh1.seek(0) assert fh1.read() == 'foobar' If you're getting an error, then you probably did a mistake when implementing the `~Operations.unlink` handler and are removing the file contents when you should be deferring removal to the `~Operations.forget` handler. ================================================ FILE: rst/index.rst ================================================ ============================= pyfuse3 Documentation ============================= Table of Contents ----------------- .. module:: pyfuse3 .. toctree:: :maxdepth: 1 about.rst install.rst general.rst asyncio.rst fuse_api.rst data.rst operations.rst util.rst gotchas.rst example.rst changes.rst Indices and tables ------------------ * :ref:`genindex` * :ref:`search` ================================================ FILE: rst/install.rst ================================================ ============== Installation ============== .. highlight:: sh Dependencies ============ As usual, Python dependencies are specified in :file:`pyproject.toml`. However, to build pyfuse3 you also need the following additional dependencies installed on your system: * libfuse_, version 3.3.0 or newer, including development headers (typically distributions provide them in a *libfuse3-devel* or *libfuse3-dev* package). * the `pkg-config`_ tool * the `attr`_ library * Python development headers (for example, the *python3-dev* or *python3-devel* package on many Linux distributions) * A C compiler Stable releases =============== To install a stable pyfuse3 release, ensure you have the non-Python dependencies installed and then use your favorite Python package manager to install pyfuse3 from PyPI (e.g. ``pip install pyfuse3``). Installing from Git / Developing pyfuse3 ======================================== Clone the pyfuse3_ repository and take a look at :file:`developer_notes/setup.md`. .. _libfuse: http://github.com/libfuse/libfuse .. _attr: http://savannah.nongnu.org/projects/attr/ .. _`pkg-config`: http://www.freedesktop.org/wiki/Software/pkg-config .. _pyfuse3: https://github.com/libfuse/pyfuse3/ ================================================ FILE: rst/operations.rst ================================================ ================== Request Handlers ================== (You can use the :ref:`genindex` to directly jump to a specific handler). .. currentmodule:: pyfuse3 .. autoclass:: Operations :members: .. attribute:: supports_dot_lookup = True If set, indicates that the filesystem supports lookup of the ``.`` and ``..`` entries. This is required if the file system will be shared over NFS. .. attribute:: enable_writeback_cache = True Enables write-caching in the kernel if available. This means that individual write request may be buffered and merged in the kernel before they are send to the filesystem. .. attribute:: enable_acl = False Enable ACL support. When enabled, the kernel will cache and have responsibility for enforcing ACLs. ACL will be stored as xattrs and passed to userspace, which is responsible for updating the ACLs in the filesystem, keeping the file mode in sync with the ACL, and ensuring inheritance of default ACLs when new filesystem nodes are created. Note that this requires that the file system is able to parse and interpret the xattr representation of ACLs. Enabling this feature implicitly turns on the ``default_permissions`` option. ================================================ FILE: rst/util.rst ================================================ ==================== Utility Functions ==================== The following functions do not necessarily translate to calls to the FUSE library. They are provided because they're potentially useful when implementing file systems in Python. .. currentmodule:: pyfuse3 .. autofunction:: setxattr .. autofunction:: getxattr .. autofunction:: listdir .. autofunction:: get_sup_groups .. autofunction:: syncfs ================================================ FILE: src/pyfuse3/__init__.pyi ================================================ ''' __init__.pyi Type annotation stubs for the external API in __init__.pyx. Copyright © 2021 Oliver Galvin This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' from typing import List, Literal, Mapping, Optional, Union from trio.lowlevel import TrioToken # Need to explicitly import with the same name to prevent mypy from # complaining that these types do not exist. from ._pyfuse3 import ( FileHandleT as FileHandleT, FileNameT as FileNameT, FlagT as FlagT, InodeT as InodeT, ModeT as ModeT, Operations as Operations, XAttrNameT as XAttrNameT, ) ENOATTR: int RENAME_EXCHANGE: FlagT RENAME_NOREPLACE: FlagT ROOT_INODE: InodeT trio_token: Optional[TrioToken] __version__: str _NANOS_PER_SEC: int NamespaceT = Literal["system", "user"] StatDict = Mapping[str, int] default_options: frozenset[str] class ReaddirToken: pass class RequestContext: @property def uid(self) -> int: ... @property def pid(self) -> int: ... @property def gid(self) -> int: ... @property def umask(self) -> int: ... def __getstate__(self) -> None: ... class SetattrFields: @property def update_atime(self) -> bool: ... @property def update_mtime(self) -> bool: ... @property def update_ctime(self) -> bool: ... @property def update_mode(self) -> bool: ... @property def update_uid(self) -> bool: ... @property def update_gid(self) -> bool: ... @property def update_size(self) -> bool: ... def __init__(self) -> None: ... def __getstate__(self) -> None: ... class EntryAttributes: st_ino: InodeT generation: int entry_timeout: Union[float, int] attr_timeout: Union[float, int] st_mode: ModeT st_nlink: int st_uid: int st_gid: int st_rdev: int st_size: int st_blksize: int st_blocks: int st_atime_ns: int st_ctime_ns: int st_mtime_ns: int st_birthtime_ns: int def __init__(self) -> None: ... def __getstate__(self) -> StatDict: ... def __setstate__(self, state: StatDict) -> None: ... class FileInfo: fh: FileHandleT direct_io: bool keep_cache: bool nonseekable: bool def __init__( self, fh: FileHandleT = ..., direct_io: bool = ..., keep_cache: bool = ..., nonseekable: bool = ..., ) -> None: ... class StatvfsData: f_bsize: int f_frsize: int f_blocks: int f_bfree: int f_bavail: int f_files: int f_ffree: int f_favail: int f_namemax: int def __init__(self) -> None: ... def __getstate__(self) -> StatDict: ... def __setstate__(self, state: StatDict) -> None: ... class FUSEError(Exception): @property def errno(self) -> int: ... @property def errno_(self) -> int: ... def __init__(self, errno: int) -> None: ... def __str__(self) -> str: ... class PollHandle: def __getstate__(self) -> None: ... def notify(self) -> None: ... def listdir(path: str) -> List[str]: ... def syncfs(path: str) -> str: ... def setxattr(path: str, name: str, value: bytes, namespace: NamespaceT = ...) -> None: ... def getxattr(path: str, name: str, size_guess: int = ..., namespace: NamespaceT = ...) -> bytes: ... def init(ops: Operations, mountpoint: str, options: set[str] = ...) -> None: ... async def main(min_tasks: int = ..., max_tasks: int = ...) -> None: ... def terminate() -> None: ... def close(unmount: bool = ...) -> None: ... def invalidate_inode(inode: InodeT, attr_only: bool = ...) -> None: ... def invalidate_entry(inode_p: InodeT, name: FileNameT, deleted: InodeT = ...) -> None: ... def invalidate_entry_async( inode_p: InodeT, name: FileNameT, deleted: InodeT = ..., ignore_enoent: bool = ... ) -> None: ... def notify_store(inode: InodeT, offset: int, data: bytes) -> None: ... def get_sup_groups(pid: int) -> set[int]: ... def readdir_reply( token: ReaddirToken, name: FileNameT, attr: EntryAttributes, next_id: int ) -> bool: ... ================================================ FILE: src/pyfuse3/__init__.pyx ================================================ ''' __init__.pyx Copyright © 2013 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' cdef extern from "pyfuse3.h": int PLATFORM enum: PLATFORM_LINUX PLATFORM_BSD PLATFORM_DARWIN ########### # C IMPORTS ########### from fuse_lowlevel cimport * from .macros cimport * from posix.stat cimport struct_stat, S_IFMT, S_IFDIR, S_IFREG from posix.types cimport mode_t, dev_t, off_t from libc.stdint cimport uint32_t from libc.stdlib cimport const_char from libc cimport stdlib, string, errno from posix cimport unistd from libc.errno cimport EACCES, ETIMEDOUT, EPROTO, EINVAL, ENOMSG, ENOATTR from posix.unistd cimport getpid from posix.time cimport timespec from cpython.bytes cimport (PyBytes_AsStringAndSize, PyBytes_FromStringAndSize, PyBytes_AsString, PyBytes_FromString, PyBytes_AS_STRING) from cpython.buffer cimport (PyObject_GetBuffer, PyBuffer_Release, PyBUF_CONTIG_RO, PyBUF_CONTIG) cimport cpython.exc cimport cython cimport libc_extra ######################## # EXTERNAL DEFINITIONS # ######################## cdef extern from "" nogil: enum: RENAME_EXCHANGE RENAME_NOREPLACE cdef extern from "Python.h" nogil: int PY_SSIZE_T_MAX ################ # PYTHON IMPORTS ################ from pickle import PicklingError from queue import Queue import logging import os import os.path import sys import trio import threading import typing from . import _pyfuse3 _pyfuse3.FUSEError = FUSEError from ._pyfuse3 import (Operations, async_wrapper, FileHandleT, FileNameT, FlagT, InodeT, ModeT, XAttrNameT) ################## # GLOBAL VARIABLES ################## log = logging.getLogger("pyfuse3") fse = sys.getfilesystemencoding() cdef object operations cdef object mountpoint_b cdef fuse_session* session = NULL cdef fuse_lowlevel_ops fuse_ops cdef int session_fd cdef object py_retval cdef object _notify_queue = None ROOT_INODE = FUSE_ROOT_ID __version__ = _pyfuse3.__version__ _NANOS_PER_SEC = 1000000000 # In the Cython source, we want the names to refer to the # C constants. Therefore, we assign through globals(). g = globals() g['ENOATTR'] = ENOATTR g['RENAME_EXCHANGE'] = RENAME_EXCHANGE g['RENAME_NOREPLACE'] = RENAME_NOREPLACE trio_token = None ####################### # FUSE REQUEST HANDLERS ####################### include "handlers.pxi" ######################################## # INTERNAL FUNCTIONS & DATA STRUCTURES # ######################################## include "internal.pxi" ###################### # EXTERNAL API # ###################### @cython.freelist(10) cdef class RequestContext: ''' Instances of this class are passed to some `Operations` methods to provide information about the caller of the syscall that initiated the request. ''' cdef readonly uid_t uid cdef readonly pid_t pid cdef readonly gid_t gid cdef readonly mode_t umask def __getstate__(self): raise PicklingError("RequestContext instances can't be pickled") @cython.freelist(10) cdef class SetattrFields: ''' `SetattrFields` instances are passed to the `~Operations.setattr` handler to specify which attributes should be updated. ''' cdef readonly object update_atime cdef readonly object update_mtime cdef readonly object update_ctime cdef readonly object update_mode cdef readonly object update_uid cdef readonly object update_gid cdef readonly object update_size def __cinit__(self): self.update_atime = False self.update_mtime = False self.update_ctime = False self.update_mode = False self.update_uid = False self.update_gid = False self.update_size = False def __getstate__(self): raise PicklingError("SetattrFields instances can't be pickled") @cython.freelist(30) cdef class EntryAttributes: ''' Instances of this class store attributes of directory entries. Most of the attributes correspond to the elements of the ``stat`` C struct as returned by e.g. ``fstat`` and should be self-explanatory. ''' # Attributes are documented in rst/data.rst cdef fuse_entry_param fuse_param cdef struct_stat *attr def __cinit__(self): string.memset(&self.fuse_param, 0, sizeof(fuse_entry_param)) self.attr = &self.fuse_param.attr self.fuse_param.generation = 0 self.fuse_param.entry_timeout = 300 self.fuse_param.attr_timeout = 300 self.attr.st_mode = S_IFREG self.attr.st_blksize = 4096 self.attr.st_nlink = 1 @property def st_ino(self): return self.fuse_param.ino @st_ino.setter def st_ino(self, val): self.fuse_param.ino = val self.attr.st_ino = val @property def generation(self): '''The inode generation number''' return self.fuse_param.generation @generation.setter def generation(self, val): self.fuse_param.generation = val @property def attr_timeout(self): '''Validity timeout for the attributes of the directory entry Floating point numbers may be used. Units are seconds. ''' return self.fuse_param.attr_timeout @attr_timeout.setter def attr_timeout(self, val): self.fuse_param.attr_timeout = val @property def entry_timeout(self): '''Validity timeout for the name/existence of the directory entry Floating point numbers may be used. Units are seconds. ''' return self.fuse_param.entry_timeout @entry_timeout.setter def entry_timeout(self, val): self.fuse_param.entry_timeout = val @property def st_mode(self): return self.attr.st_mode @st_mode.setter def st_mode(self, val): self.attr.st_mode = val @property def st_nlink(self): return self.attr.st_nlink @st_nlink.setter def st_nlink(self, val): self.attr.st_nlink = val @property def st_uid(self): return self.attr.st_uid @st_uid.setter def st_uid(self, val): self.attr.st_uid = val @property def st_gid(self): return self.attr.st_gid @st_gid.setter def st_gid(self, val): self.attr.st_gid = val @property def st_rdev(self): return self.attr.st_rdev @st_rdev.setter def st_rdev(self, val): self.attr.st_rdev = val @property def st_size(self): return self.attr.st_size @st_size.setter def st_size(self, val): self.attr.st_size = val @property def st_blocks(self): return self.attr.st_blocks @st_blocks.setter def st_blocks(self, val): self.attr.st_blocks = val @property def st_blksize(self): return self.attr.st_blksize @st_blksize.setter def st_blksize(self, val): self.attr.st_blksize = val @property def st_atime_ns(self): '''Time of last access in (integer) nanoseconds''' return (int(self.attr.st_atime) * _NANOS_PER_SEC + GET_ATIME_NS(self.attr)) @st_atime_ns.setter def st_atime_ns(self, val): self.attr.st_atime = val // _NANOS_PER_SEC SET_ATIME_NS(self.attr, val % _NANOS_PER_SEC) @property def st_mtime_ns(self): '''Time of last modification in (integer) nanoseconds''' return (int(self.attr.st_mtime) * _NANOS_PER_SEC + GET_MTIME_NS(self.attr)) @st_mtime_ns.setter def st_mtime_ns(self, val): self.attr.st_mtime = val // _NANOS_PER_SEC SET_MTIME_NS(self.attr, val % _NANOS_PER_SEC) @property def st_ctime_ns(self): '''Time of last inode modification in (integer) nanoseconds''' return (int(self.attr.st_ctime) * _NANOS_PER_SEC + GET_CTIME_NS(self.attr)) @st_ctime_ns.setter def st_ctime_ns(self, val): self.attr.st_ctime = val // _NANOS_PER_SEC SET_CTIME_NS(self.attr, val % _NANOS_PER_SEC) @property def st_birthtime_ns(self): '''Time of inode creation in (integer) nanoseconds. Only available under BSD and OS X. Will be zero on Linux. ''' # Use C macro to prevent compiler error on Linux # (where st_birthtime does not exist) return int(GET_BIRTHTIME(self.attr) * _NANOS_PER_SEC + GET_BIRTHTIME_NS(self.attr)) @st_birthtime_ns.setter def st_birthtime_ns(self, val): # Use C macro to prevent compiler error on Linux # (where st_birthtime does not exist) SET_BIRTHTIME(self.attr, val // _NANOS_PER_SEC) SET_BIRTHTIME_NS(self.attr, val % _NANOS_PER_SEC) # Pickling and copy support def __getstate__(self): state = dict() for k in ('st_ino', 'generation', 'entry_timeout', 'attr_timeout', 'st_mode', 'st_nlink', 'st_uid', 'st_gid', 'st_rdev', 'st_size', 'st_blksize', 'st_blocks', 'st_atime_ns', 'st_ctime_ns', 'st_mtime_ns', 'st_birthtime_ns'): state[k] = getattr(self, k) return state def __setstate__(self, state): for (k,v) in state.items(): setattr(self, k, v) @cython.freelist(10) cdef class FileInfo: ''' Instances of this class store options and data that `Operations.open` returns. The attributes correspond to the elements of the ``fuse_file_info`` struct that are relevant to the `Operations.open` function. ''' cdef public uint64_t fh cdef public bint direct_io cdef public bint keep_cache cdef public bint nonseekable def __cinit__(self, fh=0, direct_io=0, keep_cache=1, nonseekable=0): self.fh = fh self.direct_io = direct_io self.keep_cache = keep_cache self.nonseekable = nonseekable cdef _copy_to_fuse(self, fuse_file_info *out): out.fh = self.fh # Due to how Cython generates its C code, GCC will complain about # assigning to the bitfields in the fuse_file_info struct. # This is the workaround. if self.direct_io: out.direct_io = 1 else: out.direct_io = 0 if self.keep_cache: out.keep_cache = 1 else: out.keep_cache = 0 if self.nonseekable: out.nonseekable = 1 else: out.nonseekable = 0 @cython.freelist(1) cdef class StatvfsData: ''' Instances of this class store information about the file system. The attributes correspond to the elements of the ``statvfs`` struct, see :manpage:`statvfs(2)` for details. ''' cdef statvfs stat def __cinit__(self): string.memset(&self.stat, 0, sizeof(statvfs)) @property def f_bsize(self): return self.stat.f_bsize @f_bsize.setter def f_bsize(self, val): self.stat.f_bsize = val @property def f_frsize(self): return self.stat.f_frsize @f_frsize.setter def f_frsize(self, val): self.stat.f_frsize = val @property def f_blocks(self): return self.stat.f_blocks @f_blocks.setter def f_blocks(self, val): self.stat.f_blocks = val @property def f_bfree(self): return self.stat.f_bfree @f_bfree.setter def f_bfree(self, val): self.stat.f_bfree = val @property def f_bavail(self): return self.stat.f_bavail @f_bavail.setter def f_bavail(self, val): self.stat.f_bavail = val @property def f_files(self): return self.stat.f_files @f_files.setter def f_files(self, val): self.stat.f_files = val @property def f_ffree(self): return self.stat.f_ffree @f_ffree.setter def f_ffree(self, val): self.stat.f_ffree = val @property def f_favail(self): return self.stat.f_favail @f_favail.setter def f_favail(self, val): self.stat.f_favail = val @property def f_namemax(self): return self.stat.f_namemax @f_namemax.setter def f_namemax(self, val): self.stat.f_namemax = val # Pickling and copy support def __getstate__(self): state = dict() for k in ('f_bsize', 'f_frsize', 'f_blocks', 'f_bfree', 'f_bavail', 'f_files', 'f_ffree', 'f_favail', 'f_namemax'): state[k] = getattr(self, k) return state def __setstate__(self, state): for (k,v) in state.items(): setattr(self, k, v) # As of Cython 0.28.1, @cython.freelist cannot be used for # classes that derive from a builtin type. cdef class FUSEError(Exception): ''' This exception may be raised by request handlers to indicate that the requested operation could not be carried out. The system call that resulted in the request (if any) will then fail with error code *errno_*. ''' # If we call this variable "errno", we will get syntax errors # during C compilation (maybe something else declares errno as # a macro?) cdef readonly int errno_ @property def errno(self): '''Error code to return to client process''' return self.errno_ def __cinit__(self, errno): self.errno_ = errno def __str__(self): return strerror(self.errno_) @cython.freelist(10) cdef class PollHandle: ''' Opaque handle for delivering poll(2) readiness notifications. Instances of this class are created by pyfuse3 and passed to `Operations.poll`. The filesystem may keep a reference and later call `PollHandle.notify` on the handle to wake up any process currently blocked in :manpage:`poll(2)`, :manpage:`select(2)` or :manpage:`epoll_wait(2)` for the corresponding file descriptor. A single notification is sufficient to clear all pending waiters; filesystems should normally discard the handle after notifying. The underlying ``fuse_pollhandle`` is automatically destroyed when the Python object is garbage collected, so filesystems should simply drop the reference when the notification is no longer needed. ''' cdef fuse_pollhandle *_ph def __cinit__(self): self._ph = NULL def __init__(self): raise TypeError('PollHandle cannot be instantiated directly') @staticmethod cdef PollHandle from_ptr(fuse_pollhandle *ph): cdef PollHandle self if ph == NULL: raise ValueError('NULL fuse_pollhandle') self = PollHandle.__new__(PollHandle) self._ph = ph return self def __dealloc__(self): if self._ph is not NULL: fuse_pollhandle_destroy(self._ph) self._ph = NULL def __getstate__(self): raise PicklingError("PollHandle instances can't be pickled") def notify(self): ''' Notify IO readiness for this poll handle. After this returns, any process waiting in :manpage:`poll(2)`, :manpage:`select(2)` or :manpage:`epoll_wait(2)` on the corresponding file descriptor will be woken so it can re-poll the filesystem for the current readiness mask. Each `PollHandle` is intended for a single notification. After a successful call, the filesystem should not call `notify_poll` again on the same handle and should discard it. ''' cdef int ret if self._ph == NULL: raise RuntimeError('PollHandle is no longer valid') with nogil: ret = fuse_lowlevel_notify_poll(self._ph) if ret != 0: raise OSError(-ret, 'fuse_lowlevel_notify_poll returned: ' + strerror(-ret)) def listdir(path): '''Like `os.listdir`, but releases the GIL. This function returns an iterator over the directory entries in *path*. The returned values are of type :ref:`str `. Surrogate escape coding (cf. `PEP 383 `_) is used for directory names that do not have a string representation. ''' if not isinstance(path, str): raise TypeError('*path* argument must be of type str') cdef libc_extra.DIR* dirp cdef libc_extra.dirent* res cdef char* buf path_b = str2bytes(path) buf = path_b with nogil: dirp = libc_extra.opendir(buf) if dirp == NULL: raise OSError(errno.errno, strerror(errno.errno), path) names = list() while True: errno.errno = 0 with nogil: res = libc_extra.readdir(dirp) if res is NULL: if errno.errno != 0: raise OSError(errno.errno, strerror(errno.errno), path) else: break if string.strcmp(res.d_name, b'.') == 0 or \ string.strcmp(res.d_name, b'..') == 0: continue names.append(bytes2str(PyBytes_FromString(res.d_name))) with nogil: libc_extra.closedir(dirp) return names def syncfs(path): '''Sync filesystem mounted at *path* This is a Python interface to the syncfs(2) system call. There is no particular relation to libfuse, it is provided by pyfuse3 as a convenience. ''' cdef int ret fd = os.open(path, flags=os.O_DIRECTORY) try: ret = libc_extra.syncfs(fd) if ret != 0: raise OSError(errno.errno, strerror(errno.errno), path) finally: os.close(fd) def setxattr(path, name, bytes value, namespace='user'): '''Set extended attribute *path* and *name* have to be of type `str`. In Python 3.x, they may contain surrogates. *value* has to be of type `bytes`. Under FreeBSD, the *namespace* parameter may be set to *system* or *user* to select the namespace for the extended attribute. For other platforms, this parameter is ignored. In contrast to the `os.setxattr` function from the standard library, the method provided by pyfuse3 is also available for non-Linux systems. ''' if not isinstance(path, str): raise TypeError('*path* argument must be of type str') if not isinstance(name, str): raise TypeError('*name* argument must be of type str') if namespace not in ('system', 'user'): raise ValueError('*namespace* parameter must be "system" or "user", not %s' % namespace) cdef int ret cdef Py_ssize_t len_ cdef char *cvalue cdef char *cpath cdef char *cname cdef int cnamespace if namespace == 'system': cnamespace = libc_extra.EXTATTR_NAMESPACE_SYSTEM else: cnamespace = libc_extra.EXTATTR_NAMESPACE_USER path_b = str2bytes(path) name_b = str2bytes(name) PyBytes_AsStringAndSize(value, &cvalue, &len_) cpath = path_b cname = name_b with nogil: # len_ is guaranteed positive ret = libc_extra.setxattr_p( cpath, cname, cvalue, len_, cnamespace) if ret != 0: raise OSError(errno.errno, strerror(errno.errno), path) def getxattr(path, name, size_t size_guess=128, namespace='user'): '''Get extended attribute *path* and *name* have to be of type `str`. In Python 3.x, they may contain surrogates. Returns a value of type `bytes`. If the caller knows the approximate size of the attribute value, it should be supplied in *size_guess*. If the guess turns out to be wrong, the system call has to be carried out three times (the first call will fail, the second determines the size and the third finally gets the value). Under FreeBSD, the *namespace* parameter may be set to *system* or *user* to select the namespace for the extended attribute. For other platforms, this parameter is ignored. In contrast to the `os.getxattr` function from the standard library, the method provided by pyfuse3 is also available for non-Linux systems. ''' if not isinstance(path, str): raise TypeError('*path* argument must be of type str') if not isinstance(name, str): raise TypeError('*name* argument must be of type str') if namespace not in ('system', 'user'): raise ValueError('*namespace* parameter must be "system" or "user", not %s' % namespace) cdef ssize_t ret cdef char *buf cdef char *cpath cdef char *cname cdef size_t bufsize cdef int cnamespace if namespace == 'system': cnamespace = libc_extra.EXTATTR_NAMESPACE_SYSTEM else: cnamespace = libc_extra.EXTATTR_NAMESPACE_USER path_b = str2bytes(path) name_b = str2bytes(name) cpath = path_b cname = name_b bufsize = size_guess buf = stdlib.malloc(bufsize * sizeof(char)) if buf is NULL: cpython.exc.PyErr_NoMemory() try: with nogil: ret = libc_extra.getxattr_p(cpath, cname, buf, bufsize, cnamespace) if ret < 0 and errno.errno == errno.ERANGE: with nogil: ret = libc_extra.getxattr_p(cpath, cname, NULL, 0, cnamespace) if ret < 0: raise OSError(errno.errno, strerror(errno.errno), path) bufsize = ret stdlib.free(buf) buf = stdlib.malloc(bufsize * sizeof(char)) if buf is NULL: cpython.exc.PyErr_NoMemory() with nogil: ret = libc_extra.getxattr_p(cpath, cname, buf, bufsize, cnamespace) if ret < 0: raise OSError(errno.errno, strerror(errno.errno), path) return PyBytes_FromStringAndSize(buf, ret) finally: stdlib.free(buf) default_options = frozenset(('default_permissions',)) def init(ops, mountpoint, options=default_options): '''Initialize and mount FUSE file system *ops* has to be an instance of the `Operations` class (or another class defining the same methods). *args* has to be a set of strings. `default_options` provides some reasonable defaults. It is recommended to use these options as a basis and add or remove options as necessary. For example:: my_opts = set(pyfuse3.default_options) my_opts.add('allow_other') my_opts.discard('default_permissions') pyfuse3.init(ops, mountpoint, my_opts) Valid options are listed under ``struct fuse_opt fuse_mount_opts[]`` (in `mount.c `_) and ``struct fuse_opt fuse_ll_opts[]`` (in `fuse_lowlevel_c `_). ''' log.debug('Initializing pyfuse3') cdef fuse_args f_args cdef int res if not isinstance(mountpoint, str): raise TypeError('*mountpoint_* argument must be of type str') global operations global fuse_ops global mountpoint_b global session global session_fd global worker_data worker_data = _WorkerData() mountpoint_b = str2bytes(os.path.abspath(mountpoint)) operations = ops make_fuse_args(options, &f_args) log.debug('Calling fuse_session_new') init_fuse_ops() session = fuse_session_new(&f_args, &fuse_ops, sizeof(fuse_ops), NULL) if not session: raise RuntimeError("fuse_session_new() failed") log.debug('Calling fuse_session_mount') res = fuse_session_mount(session, mountpoint_b) if res != 0: raise RuntimeError('fuse_session_mount failed') session_fd = fuse_session_fd(session) @async_wrapper async def main(int min_tasks=1, int max_tasks=99): '''Run FUSE main loop''' if session == NULL: raise RuntimeError('Need to call init() before main()') global trio_token trio_token = trio.lowlevel.current_trio_token() try: async with trio.open_nursery() as nursery: worker_data.task_count = 1 worker_data.task_serial = 1 nursery.start_soon(_session_loop, nursery, min_tasks, max_tasks, name=worker_data.get_name()) finally: trio_token = None if _notify_queue is not None: _notify_queue.put(None) def terminate(): '''Terminate FUSE main loop. This function gracefully terminates the FUSE main loop (resulting in the call to main() to return). When called by a thread different from the one that runs the main loop, the call must be wrapped with `trio.from_thread.run_sync`. The necessary *trio_token* argument can (for convenience) be retrieved from the `trio_token` module attribute. ''' fuse_session_exit(session) trio.lowlevel.notify_closing(session_fd) def close(unmount=True): '''Clean up and ensure filesystem is unmounted If *unmount* is False, only clean up operations are performed, but the file system is not explicitly unmounted. Normally, the filesystem is unmounted by the user calling umount(8) or fusermount(1), which then terminates the FUSE main loop. However, the loop may also terminate as a result of an exception or a signal. In this case the filesystem remains mounted, but any attempt to access it will block (while the filesystem process is still running) or (after the filesystem process has terminated) return an error. If *unmount* is True, this function will ensure that the filesystem is properly unmounted. Note: if the connection to the kernel is terminated via the ``/sys/fs/fuse/connections/`` interface, this function will *not* unmount the filesystem even if *unmount* is True. ''' global mountpoint_b global session if unmount: log.debug('Calling fuse_session_unmount') fuse_session_unmount(session) log.debug('Calling fuse_session_destroy') fuse_session_destroy(session) mountpoint_b = None session = NULL def invalidate_inode(fuse_ino_t inode, attr_only=False): '''Invalidate cache for *inode* Instructs the FUSE kernel module to forget cached attributes and data (unless *attr_only* is True) for *inode*. **This operation may block** if writeback caching is active and there is dirty data for the inode that is to be invalidated. Unfortunately there is no way to return control to the event loop until writeback is complete (leading to a deadlock if the necessary write() requests cannot be processed by the filesystem). Unless writeback caching is disabled, this function should therefore be called from a separate thread. If the operation is not supported by the kernel, raises `OSError` with errno ENOSYS. ''' cdef int ret if attr_only: with nogil: ret = fuse_lowlevel_notify_inval_inode(session, inode, -1, 0) else: with nogil: ret = fuse_lowlevel_notify_inval_inode(session, inode, 0, 0) if ret != 0: raise OSError(-ret, 'fuse_lowlevel_notify_inval_inode returned: ' + strerror(-ret)) def invalidate_entry(fuse_ino_t inode_p, bytes name, fuse_ino_t deleted=0): '''Invalidate directory entry Instructs the FUSE kernel module to forget about the directory entry *name* in the directory with inode *inode_p*. If the inode passed as *deleted* matches the inode that is currently associated with *name* by the kernel, any inotify watchers of this inode are informed that the entry has been deleted. If there is a pending filesystem operation that is related to the parent directory or directory entry, this function will block until that operation has completed. Therefore, to avoid a deadlock this function must not be called while handling a related request, nor while holding a lock that could be needed for handling such a request. As for kernel 4.18, a "related operation" is a `~Operations.lookup`, `~Operations.symlink`, `~Operations.mknod`, `~Operations.mkdir`, `~Operations.unlink`, `~Operations.rename`, `~Operations.link` or `~Operations.create` request for the parent, and a `~Operations.setattr`, `~Operations.unlink`, `~Operations.rmdir`, `~Operations.rename`, `~Operations.setxattr`, `~Operations.removexattr` or `~Operations.readdir` request for the inode itself. For technical reasons, this function can also not return control to the main event loop but will actually block. To return control to the event loop while this function is running, call it in a separate thread using `trio.run_sync_in_worker_thread `_. A less complicated alternative is to use the `invalidate_entry_async` function instead. ''' cdef char *cname cdef ssize_t slen cdef size_t len_ cdef int ret PyBytes_AsStringAndSize(name, &cname, &slen) # len_ is guaranteed positive len_ = slen if deleted: with nogil: # might block! ret = fuse_lowlevel_notify_delete(session, inode_p, deleted, cname, len_) if ret != 0: raise OSError(-ret, 'fuse_lowlevel_notify_delete returned: ' + strerror(-ret)) else: with nogil: # might block! ret = fuse_lowlevel_notify_inval_entry(session, inode_p, cname, len_) if ret != 0: raise OSError(-ret, 'fuse_lowlevel_notify_inval_entry returned: ' + strerror(-ret)) def invalidate_entry_async(inode_p, name, deleted=0, ignore_enoent=False): '''Asynchronously invalidate directory entry This function performs the same operation as `invalidate_entry`, but does so asynchronously in a separate thread. This avoids the deadlocks that may occur when using `invalidate_entry` from within a request handler, but means that the function generally returns before the kernel has actually invalidated the entry, and that no errors can be reported (they will be logged though). The directory entries that are to be invalidated are put in an unbounded queue which is processed by a single thread. This means that if the entry at the beginning of the queue cannot be invalidated yet because a related file system operation is still in progress, none of the other entries will be processed and repeated calls to this function will result in continued growth of the queue. If there are errors, an exception is logged using the `logging` module. If *ignore_enoent* is True, ignore ENOENT errors (which occur if the kernel doesn't actually have knowledge of the entry that is to be removed). ''' global _notify_queue if _notify_queue is None: log.debug('Starting notify worker.') _notify_queue = Queue() t = threading.Thread(target=_notify_loop) t.daemon = True t.start() _notify_queue.put((inode_p, name, deleted, ignore_enoent)) def notify_store(inode, offset, data): '''Store data in kernel page cache Sends *data* for the kernel to store it in the page cache for *inode* at *offset*. If this provides data beyond the current file size, the file is automatically extended. If this function raises an exception, the store may still have completed partially. If the operation is not supported by the kernel, raises `OSError` with errno ENOSYS. ''' # This should not block, but the kernel may need to do some work so release # the GIL to give other threads a chance to run. cdef int ret cdef fuse_ino_t ino cdef off_t off cdef Py_buffer pybuf cdef fuse_bufvec bufvec cdef fuse_buf *buf PyObject_GetBuffer(data, &pybuf, PyBUF_CONTIG_RO) bufvec.count = 1 bufvec.idx = 0 bufvec.off = 0 buf = bufvec.buf buf[0].flags = 0 buf[0].mem = pybuf.buf buf[0].size = pybuf.len # guaranteed positive ino = inode off = offset with nogil: ret = fuse_lowlevel_notify_store(session, ino, off, &bufvec, 0) PyBuffer_Release(&pybuf) if ret != 0: raise OSError(-ret, 'fuse_lowlevel_notify_store returned: ' + strerror(-ret)) def get_sup_groups(pid): '''Return supplementary group ids of *pid* This function is relatively expensive because it has to read the group ids from ``/proc/[pid]/status``. For the same reason, it will also not work on systems that do not provide a ``/proc`` file system. Returns a set. ''' with open('/proc/%d/status' % pid, 'r') as fh: for line in fh: if line.startswith('Groups:'): break else: raise RuntimeError("Unable to parse %s" % fh.name) gids = set() for x in line.split()[1:]: gids.add(int(x)) return gids def readdir_reply(ReaddirToken token, name, EntryAttributes attr, off_t next_id): '''Report a directory entry in response to a `~Operations.readdir` request. This function should be called by the `~Operations.readdir` handler to provide the list of directory entries. The function should be called once for each directory entry, until it returns False. *token* must be the token received by the `~Operations.readdir` handler. *name* and must be the name of the directory entry and *attr* an `EntryAttributes` instance holding its attributes. *next_id* must be a 64-bit integer value that uniquely identifies the current position in the list of directory entries. It may be passed back to a later `~Operations.readdir` call to start another listing at the right position. This value should be robust in the presence of file removals and creations, i.e. if files are created or removed after a call to `~Operations.readdir` and `~Operations.readdir` is called again with *start_id* set to any previously supplied *next_id* values, under no circumstances must any file be reported twice or skipped over. ''' cdef char *cname if token.buf_start == NULL: token.buf_start = calloc_or_raise(token.size, sizeof(char)) token.buf = token.buf_start cname = PyBytes_AsString(name) len_ = fuse_add_direntry_plus(token.req, token.buf, token.size, cname, &attr.fuse_param, next_id) if len_ > token.size: return False token.size -= len_ token.buf = &token.buf[len_] return True ================================================ FILE: src/pyfuse3/_pyfuse3.py ================================================ ''' _pyfuse3.py Pure-Python components of pyfuse3. Copyright © 2018 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' import errno import functools import logging from importlib.metadata import PackageNotFoundError, version as package_version from typing import TYPE_CHECKING, Any, Callable, NewType, Optional, Sequence, Tuple # Version information try: __version__ = package_version('pyfuse3') except PackageNotFoundError: __version__ = 'unknown' # These types are specific instances of builtin types: FileHandleT = NewType("FileHandleT", int) FileNameT = bytes FlagT = int InodeT = NewType("InodeT", int) ModeT = int XAttrNameT = bytes if TYPE_CHECKING: # These types are defined elsewhere in the C code from pyfuse3 import ( EntryAttributes, FileInfo, FUSEError, PollHandle, ReaddirToken, RequestContext, SetattrFields, StatvfsData, ) else: # Will be injected by pyfuse3 extension module FUSEError = None __all__ = ['Operations', 'async_wrapper'] log = logging.getLogger(__name__) # Any top level trio coroutines (i.e., coroutines that are passed # to trio.run) must be pure-Python. This wrapper ensures that this # is the case for Cython-defined async functions. def async_wrapper(fn: Callable[..., Any]) -> Callable[..., Any]: @functools.wraps(fn) async def wrapper(*args, **kwargs): await fn(*args, **kwargs) return wrapper class Operations: ''' This class defines the request handler methods that an pyfuse3 file system may implement. If a particular request handler has not been implemented, it must raise `FUSEError` with an errorcode of `errno.ENOSYS`. Further requests of this type will then be handled directly by the FUSE kernel module without calling the handler again. The only exception that request handlers are allowed to raise is `FUSEError`. This will cause the specified errno to be returned by the syscall that is being handled. It is recommended that file systems are derived from this class and only overwrite the handlers that they actually implement. (The methods defined in this class all just raise ``FUSEError(ENOSYS)`` or do nothing). ''' supports_dot_lookup: bool = True enable_writeback_cache: bool = False enable_acl: bool = False def init(self) -> None: '''Initialize operations. This method will be called just before the file system starts handling requests. It must not raise any exceptions (not even `FUSEError`), since it is not handling a particular client request. ''' pass async def lookup( self, parent_inode: InodeT, name: FileNameT, ctx: "RequestContext" ) -> "EntryAttributes": '''Look up a directory entry by name and get its attributes. This method should return an `EntryAttributes` instance for the directory entry *name* in the directory with inode *parent_inode*. If there is no such entry, the method should either return an `EntryAttributes` instance with zero ``st_ino`` value (in which case the negative lookup will be cached as specified by ``entry_timeout``), or it should raise `FUSEError` with an errno of `errno.ENOENT` (in this case the negative result will not be cached). *ctx* will be a `RequestContext` instance. The file system must be able to handle lookups for :file:`.` and :file:`..`, no matter if these entries are returned by `readdir` or not. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) async def forget(self, inode_list: Sequence[Tuple[InodeT, int]]) -> None: '''Decrease lookup counts for inodes in *inode_list*. *inode_list* is a list of ``(inode, nlookup)`` tuples. This method should reduce the lookup count for each *inode* by *nlookup*. If the lookup count reaches zero, the inode is currently not known to the kernel. In this case, the file system will typically check if there are still directory entries referring to this inode and, if not, remove the inode. If the file system is unmounted, it may not have received `forget` calls to bring all lookup counts to zero. The filesystem needs to take care to clean up inodes that at that point still have non-zero lookup count (e.g. by explicitly calling `forget` with the current lookup count for every such inode after `main` has returned). This method must not raise any exceptions (not even `FUSEError`), since it is not handling a particular client request. ''' pass async def getattr(self, inode: InodeT, ctx: "RequestContext") -> "EntryAttributes": '''Get attributes for *inode*. *ctx* will be a `RequestContext` instance. This method should return an `EntryAttributes` instance with the attributes of *inode*. The `~EntryAttributes.entry_timeout` attribute is ignored in this context. ''' raise FUSEError(errno.ENOSYS) async def setattr( self, inode: InodeT, attr: "EntryAttributes", fields: "SetattrFields", fh: Optional[FileHandleT], ctx: "RequestContext", ) -> "EntryAttributes": '''Change attributes of *inode*. *fields* will be an `SetattrFields` instance that specifies which attributes are to be updated. *attr* will be an `EntryAttributes` instance for *inode* that contains the new values for changed attributes, and undefined values for all other attributes. Most file systems will additionally set the `~EntryAttributes.st_ctime_ns` attribute to the current time (to indicate that the inode metadata was changed). If the syscall that is being processed received a file descriptor argument (like e.g. :manpage:`ftruncate(2)` or :manpage:`fchmod(2)`), *fh* will be the file handle returned by the corresponding call to the `open` handler. If the syscall was path based (like e.g. :manpage:`truncate(2)` or :manpage:`chmod(2)`), *fh* will be `None`. *ctx* will be a `RequestContext` instance. The method should return an `EntryAttributes` instance (containing both the changed and unchanged values). ''' raise FUSEError(errno.ENOSYS) async def readlink(self, inode: InodeT, ctx: "RequestContext") -> FileNameT: '''Return target of symbolic link *inode*. *ctx* will be a `RequestContext` instance. ''' raise FUSEError(errno.ENOSYS) async def mknod( self, parent_inode: InodeT, name: FileNameT, mode: ModeT, rdev: int, ctx: "RequestContext", ) -> "EntryAttributes": '''Create (possibly special) file. This method must create a (special or regular) file *name* in the directory with inode *parent_inode*. Whether the file is special or regular is determined by its *mode*. If the file is neither a block nor character device, *rdev* can be ignored. *ctx* will be a `RequestContext` instance. The method must return an `EntryAttributes` instance with the attributes of the newly created directory entry. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) async def mkdir( self, parent_inode: InodeT, name: FileNameT, mode: ModeT, ctx: "RequestContext" ) -> "EntryAttributes": '''Create a directory. This method must create a new directory *name* with mode *mode* in the directory with inode *parent_inode*. *ctx* will be a `RequestContext` instance. This method must return an `EntryAttributes` instance with the attributes of the newly created directory entry. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) async def unlink(self, parent_inode: InodeT, name: FileNameT, ctx: "RequestContext") -> None: '''Remove a (possibly special) file. This method must remove the (special or regular) file *name* from the directory with inode *parent_inode*. *ctx* will be a `RequestContext` instance. If the inode associated with *file* (i.e., not the *parent_inode*) has a non-zero lookup count, or if there are still other directory entries referring to this inode (due to hardlinks), the file system must remove only the directory entry (so that future calls to `readdir` for *parent_inode* will no longer include *name*, but e.g. calls to `getattr` for *file*'s inode still succeed). (Potential) removal of the associated inode with the file contents and metadata must be deferred to the `forget` method to be carried out when the lookup count reaches zero (and of course only if at that point there are no more directory entries associated with the inode either). ''' raise FUSEError(errno.ENOSYS) async def rmdir(self, parent_inode: InodeT, name: FileNameT, ctx: "RequestContext") -> None: '''Remove directory *name*. This method must remove the directory *name* from the directory with inode *parent_inode*. *ctx* will be a `RequestContext` instance. If there are still entries in the directory, the method should raise ``FUSEError(errno.ENOTEMPTY)``. If the inode associated with *name* (i.e., not the *parent_inode*) has a non-zero lookup count, the file system must remove only the directory entry (so that future calls to `readdir` for *parent_inode* will no longer include *name*, but e.g. calls to `getattr` for *file*'s inode still succeed). Removal of the associated inode holding the directory contents and metadata must be deferred to the `forget` method to be carried out when the lookup count reaches zero. (Since hard links to directories are not allowed by POSIX, this method is not required to check if there are still other directory entries referring to the same inode. This conveniently avoids the ambiguities associated with the ``.`` and ``..`` entries). ''' raise FUSEError(errno.ENOSYS) async def symlink( self, parent_inode: InodeT, name: FileNameT, target: FileNameT, ctx: "RequestContext", ) -> "EntryAttributes": '''Create a symbolic link. This method must create a symbolink link named *name* in the directory with inode *parent_inode*, pointing to *target*. *ctx* will be a `RequestContext` instance. The method must return an `EntryAttributes` instance with the attributes of the newly created directory entry. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) async def rename( self, parent_inode_old: InodeT, name_old: FileNameT, parent_inode_new: InodeT, name_new: FileNameT, flags: FlagT, ctx: "RequestContext", ) -> None: '''Rename a directory entry. This method must rename *name_old* in the directory with inode *parent_inode_old* to *name_new* in the directory with inode *parent_inode_new*. If *name_new* already exists, it should be overwritten. *flags* may be `RENAME_EXCHANGE` or `RENAME_NOREPLACE`. If `RENAME_NOREPLACE` is specified, the filesystem must not overwrite *name_new* if it exists and return an error instead. If `RENAME_EXCHANGE` is specified, the filesystem must atomically exchange the two files, i.e. both must exist and neither may be deleted. *ctx* will be a `RequestContext` instance. Let the inode associated with *name_old* in *parent_inode_old* be *inode_moved*, and the inode associated with *name_new* in *parent_inode_new* (if it exists) be called *inode_deref*. If *inode_deref* exists and has a non-zero lookup count, or if there are other directory entries referring to *inode_deref*), the file system must update only the directory entry for *name_new* to point to *inode_moved* instead of *inode_deref*. (Potential) removal of *inode_deref* (containing the previous contents of *name_new*) must be deferred to the `forget` method to be carried out when the lookup count reaches zero (and of course only if at that point there are no more directory entries associated with *inode_deref* either). ''' raise FUSEError(errno.ENOSYS) async def link( self, inode: InodeT, new_parent_inode: InodeT, new_name: FileNameT, ctx: "RequestContext", ) -> "EntryAttributes": '''Create directory entry *name* in *parent_inode* referring to *inode*. *ctx* will be a `RequestContext` instance. The method must return an `EntryAttributes` instance with the attributes of the newly created directory entry. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) async def open(self, inode: InodeT, flags: FlagT, ctx: "RequestContext") -> "FileInfo": '''Open a inode *inode* with *flags*. *ctx* will be a `RequestContext` instance. *flags* will be a bitwise or of the open flags described in the :manpage:`open(2)` manpage and defined in the `os` module (with the exception of ``O_CREAT``, ``O_EXCL``, ``O_NOCTTY`` and ``O_TRUNC``) This method must return a `FileInfo` instance. The `FileInfo.fh` field must contain an integer file handle, which will be passed to the `read`, `write`, `flush`, `fsync` and `release` methods to identify the open file. The `FileInfo` instance may also have relevant configuration attributes set; see the `FileInfo` documentation for more information. ''' raise FUSEError(errno.ENOSYS) async def read(self, fh: FileHandleT, off: int, size: int) -> bytes: '''Read *size* bytes from *fh* at position *off*. *fh* will be an integer filehandle returned by a prior `open` or `create` call. This function should return exactly the number of bytes requested except on EOF or error, otherwise the rest of the data will be substituted with zeroes. ''' raise FUSEError(errno.ENOSYS) async def write(self, fh: FileHandleT, off: int, buf: bytes) -> int: '''Write *buf* into *fh* at *off*. *fh* will be an integer filehandle returned by a prior `open` or `create` call. This method must return the number of bytes written. However, unless the file system has been mounted with the ``direct_io`` option, the file system *must* always write *all* the provided data (i.e., return ``len(buf)``). ''' raise FUSEError(errno.ENOSYS) async def flush(self, fh: FileHandleT) -> None: '''Handle close() syscall. *fh* will be an integer filehandle returned by a prior `open` or `create` call. This method is called whenever a file descriptor is closed. It may be called multiple times for the same open file (e.g. if the file handle has been duplicated). ''' raise FUSEError(errno.ENOSYS) async def release(self, fh: FileHandleT) -> None: '''Release open file. This method will be called when the last file descriptor of *fh* has been closed, i.e. when the file is no longer opened by any client process. *fh* will be an integer filehandle returned by a prior `open` or `create` call. Once `release` has been called, no future requests for *fh* will be received (until the value is re-used in the return value of another `open` or `create` call). This method may return an error by raising `FUSEError`, but the error will be discarded because there is no corresponding client request. ''' raise FUSEError(errno.ENOSYS) async def fsync(self, fh: FileHandleT, datasync: bool) -> None: '''Flush buffers for open file *fh*. If *datasync* is true, only the file contents should be flushed (in contrast to the metadata about the file). *fh* will be an integer filehandle returned by a prior `open` or `create` call. ''' raise FUSEError(errno.ENOSYS) async def poll( self, inode: InodeT, fh: FileHandleT, poll_handle: Optional["PollHandle"], ctx: "RequestContext", ) -> int: '''Check IO readiness on an open file. This method is called when a process performs :manpage:`poll(2)`, :manpage:`select(2)` or :manpage:`epoll_wait(2)` on a file descriptor backed by *fh* (returned by a prior `open` or `create` call). *inode* identifies the inode that *fh* refers to. The method will return the bitwise-or of the currently active poll events, for example `select.POLLIN`, `select.POLLOUT` or `select.POLLPRI`. If no events are currently ready, it will return `0`. If *poll_handle* is `None`, the kernel has not provided a notification handle for this request. The filesystem should only return the current readiness mask and must not attempt to store a handle or arrange a later `PollHandle.notify` call for this poll request. If *poll_handle* is not `None`, the kernel has provided a notification handle that may be used to wake waiters if readiness changes after this method returns. The filesystem may store the handle and later call `PollHandle.notify` when a relevant event becomes available. Each `~Operations.poll` call produces a fresh handle; storing a new handle should replace any previously held one, allowing the old handle to be destroyed. If this method raises `FUSEError(errno.ENOSYS)` (the default), the kernel will fall back to a default poll implementation and will not call this handler again for the lifetime of the mount. ''' raise FUSEError(errno.ENOSYS) async def opendir(self, inode: InodeT, ctx: "RequestContext") -> FileHandleT: '''Open the directory with inode *inode*. *ctx* will be a `RequestContext` instance. This method should return an integer file handle. The file handle will be passed to the `readdir`, `fsyncdir` and `releasedir` methods to identify the directory. ''' raise FUSEError(errno.ENOSYS) async def readdir(self, fh: FileHandleT, start_id: int, token: "ReaddirToken") -> None: '''Read entries in open directory *fh*. This method should list the contents of directory *fh* (as returned by a prior `opendir` call), starting at the entry identified by *start_id*. Instead of returning the directory entries directly, the method must call `readdir_reply` for each directory entry. If `readdir_reply` returns True, the file system must increase the lookup count for the provided directory entry by one and call `readdir_reply` again for the next entry (if any). If `readdir_reply` returns False, the lookup count must *not* be increased and the method should return without further calls to `readdir_reply`. The *start_id* parameter will be either zero (in which case listing should begin with the first entry) or it will correspond to a value that was previously passed by the file system to the `readdir_reply` function in the *next_id* parameter. If entries are added or removed during a `readdir` cycle, they may or may not be returned. However, they must not cause other entries to be skipped or returned more than once. :file:`.` and :file:`..` entries may be included but are not required. However, if they are reported the filesystem *must not* increase the lookup count for the corresponding inodes (even if `readdir_reply` returns True). ''' raise FUSEError(errno.ENOSYS) async def releasedir(self, fh: FileHandleT) -> None: '''Release open directory. This method will be called exactly once for each `opendir` call. After *fh* has been released, no further `readdir` requests will be received for it (until it is opened again with `opendir`). ''' raise FUSEError(errno.ENOSYS) async def fsyncdir(self, fh: FileHandleT, datasync: bool) -> None: '''Flush buffers for open directory *fh*. If *datasync* is true, only the directory contents should be flushed (in contrast to metadata about the directory itself). ''' raise FUSEError(errno.ENOSYS) async def statfs(self, ctx: "RequestContext") -> "StatvfsData": '''Get file system statistics. *ctx* will be a `RequestContext` instance. The method must return an appropriately filled `StatvfsData` instance. ''' raise FUSEError(errno.ENOSYS) def stacktrace(self) -> None: '''Asynchronous debugging. This method will be called when the ``fuse_stacktrace`` extended attribute is set on the mountpoint. The default implementation logs the current stack trace of every running Python thread. This can be quite useful to debug file system deadlocks. ''' import sys import traceback from os.path import basename code = list() for threadId, frame in sys._current_frames().items(): code.append(f"\n# ThreadID: {threadId}") for filename, lineno, name, line in traceback.extract_stack(frame): code.append(f'{basename(filename)}:{lineno}, in {name}') if line: code.append(f" {line.strip()}") log.error("\n".join(code)) async def setxattr( self, inode: InodeT, name: XAttrNameT, value: bytes, ctx: "RequestContext" ) -> None: '''Set extended attribute *name* of *inode* to *value*. *ctx* will be a `RequestContext` instance. The attribute may or may not exist already. Both *name* and *value* will be of type `bytes`. *name* is guaranteed not to contain zero-bytes (``\\0``). ''' raise FUSEError(errno.ENOSYS) async def getxattr(self, inode: InodeT, name: XAttrNameT, ctx: "RequestContext") -> bytes: '''Return extended attribute *name* of *inode*. *ctx* will be a `RequestContext` instance. If the attribute does not exist, the method must raise `FUSEError` with an error code of `ENOATTR`. *name* will be of type `bytes`, but is guaranteed not to contain zero-bytes (``\\0``). ''' raise FUSEError(errno.ENOSYS) async def listxattr(self, inode: InodeT, ctx: "RequestContext") -> Sequence[XAttrNameT]: '''Get list of extended attributes for *inode*. *ctx* will be a `RequestContext` instance. This method must return a sequence of `bytes` objects. The objects must not include zero-bytes (``\\0``). ''' raise FUSEError(errno.ENOSYS) async def removexattr(self, inode: InodeT, name: XAttrNameT, ctx: "RequestContext") -> None: '''Remove extended attribute *name* of *inode*. *ctx* will be a `RequestContext` instance. If the attribute does not exist, the method must raise `FUSEError` with an error code of `ENOATTR`. *name* will be of type `bytes`, but is guaranteed not to contain zero-bytes (``\\0``). ''' raise FUSEError(errno.ENOSYS) async def access(self, inode: InodeT, mode: ModeT, ctx: "RequestContext") -> bool: '''Check if requesting process has *mode* rights on *inode*. *ctx* will be a `RequestContext` instance. The method must return a boolean value. If the ``default_permissions`` mount option is given, this method is not called. When implementing this method, the `get_sup_groups` function may be useful. ''' raise FUSEError(errno.ENOSYS) async def create( self, parent_inode: InodeT, name: FileNameT, mode: ModeT, flags: FlagT, ctx: "RequestContext", ) -> Tuple["FileInfo", "EntryAttributes"]: '''Create a file with permissions *mode* and open it with *flags*. *ctx* will be a `RequestContext` instance. The method must return a tuple of the form *(fi, attr)*, where *fi* is a FileInfo instance handle like the one returned by `open` and *attr* is an `EntryAttributes` instance with the attributes of the newly created directory entry. (Successful) execution of this handler increases the lookup count for the returned inode by one. ''' raise FUSEError(errno.ENOSYS) ================================================ FILE: src/pyfuse3/asyncio.py ================================================ ''' asyncio.py asyncio compatibility layer for pyfuse3 Copyright © 2018 Nikolaus Rath Copyright © 2018 JustAnotherArchivist This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' import asyncio import collections import sys from typing import Any, Callable, Iterable, Optional, Set, Type import pyfuse3 from ._pyfuse3 import FileHandleT Lock = asyncio.Lock def enable() -> None: '''Switch pyfuse3 to asyncio mode.''' fake_trio = sys.modules['pyfuse3.asyncio'] fake_trio.lowlevel = fake_trio # type: ignore fake_trio.from_thread = fake_trio # type: ignore pyfuse3.trio = fake_trio # type: ignore def disable() -> None: '''Switch pyfuse3 to default (trio) mode.''' pyfuse3.trio = sys.modules['trio'] # type: ignore def current_trio_token() -> str: return 'asyncio' _read_futures = collections.defaultdict(set) async def wait_readable(fd: FileHandleT) -> None: loop = asyncio.get_running_loop() future: 'asyncio.Future[Any]' = loop.create_future() _read_futures[fd].add(future) try: loop.add_reader(fd, future.set_result, None) future.add_done_callback(lambda f: loop.remove_reader(fd)) await future finally: _read_futures[fd].remove(future) if not _read_futures[fd]: del _read_futures[fd] def notify_closing(fd: FileHandleT) -> None: for f in _read_futures[fd]: f.set_exception(ClosedResourceError()) class ClosedResourceError(Exception): pass def current_task() -> 'Optional[asyncio.Task[Any]]': if sys.version_info < (3, 7): return asyncio.Task.current_task() else: return asyncio.current_task() class _Nursery: async def __aenter__(self) -> "_Nursery": self.tasks: 'Set[asyncio.Task[Any]]' = set() return self def start_soon( self, func: Callable[..., Any], *args: Iterable[Any], name: Optional[str] = None ) -> None: if sys.version_info < (3, 7): task = asyncio.ensure_future(func(*args)) else: task = asyncio.create_task(func(*args)) task.name = name # type: ignore self.tasks.add(task) async def __aexit__( self, exc_type: Optional[Type[BaseException]], exc_value: Optional[BaseException], traceback: Optional[Any], ) -> None: # Wait for tasks to finish while len(self.tasks): # Create a copy of the task list to ensure that it's not a problem # when self.tasks is modified done, pending = await asyncio.wait(tuple(self.tasks)) for task in done: self.tasks.discard(task) # We waited for ALL_COMPLETED (default value of 'when' arg to # asyncio.wait), so all tasks should be completed. If that's not the # case, something's seriously wrong. assert len(pending) == 0 def open_nursery() -> _Nursery: return _Nursery() ================================================ FILE: src/pyfuse3/darwin_compat.c ================================================ /* * Copyright (c) 2006-2008 Amit Singh/Google Inc. * Copyright (c) 2012 Anatol Pomozov * Copyright (c) 2011-2013 Benjamin Fleischer */ #include "darwin_compat.h" #include #include #include /* * Semaphore implementation based on: * * Copyright (C) 2000,02 Free Software Foundation, Inc. * This file is part of the GNU C Library. * Written by Gal Le Mignot * * The GNU C Library is free software; you can redistribute it and/or * modify it under the terms of the GNU Library General Public License as * published by the Free Software Foundation; either version 2 of the * License, or (at your option) any later version. * * The GNU C Library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Library General Public License for more details. * * You should have received a copy of the GNU Library General Public * License along with the GNU C Library; see the file COPYING.LIB. If not, * write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, * Boston, MA 02111-1307, USA. */ /* Semaphores */ #define __SEM_ID_NONE ((int)0x0) #define __SEM_ID_LOCAL ((int)0xcafef00d) /* http://www.opengroup.org/onlinepubs/007908799/xsh/sem_init.html */ int darwin_sem_init(darwin_sem_t *sem, int pshared, unsigned int value) { if (pshared) { errno = ENOSYS; return -1; } sem->id = __SEM_ID_NONE; if (pthread_cond_init(&sem->__data.local.count_cond, NULL)) { goto cond_init_fail; } if (pthread_mutex_init(&sem->__data.local.count_lock, NULL)) { goto mutex_init_fail; } sem->__data.local.count = value; sem->id = __SEM_ID_LOCAL; return 0; mutex_init_fail: pthread_cond_destroy(&sem->__data.local.count_cond); cond_init_fail: return -1; } /* http://www.opengroup.org/onlinepubs/007908799/xsh/sem_destroy.html */ int darwin_sem_destroy(darwin_sem_t *sem) { int res = 0; pthread_mutex_lock(&sem->__data.local.count_lock); sem->id = __SEM_ID_NONE; pthread_cond_broadcast(&sem->__data.local.count_cond); if (pthread_cond_destroy(&sem->__data.local.count_cond)) { res = -1; } pthread_mutex_unlock(&sem->__data.local.count_lock); if (pthread_mutex_destroy(&sem->__data.local.count_lock)) { res = -1; } return res; } int darwin_sem_getvalue(darwin_sem_t *sem, unsigned int *sval) { int res = 0; pthread_mutex_lock(&sem->__data.local.count_lock); if (sem->id != __SEM_ID_LOCAL) { res = -1; errno = EINVAL; } else { *sval = sem->__data.local.count; } pthread_mutex_unlock(&sem->__data.local.count_lock); return res; } /* http://www.opengroup.org/onlinepubs/007908799/xsh/sem_post.html */ int darwin_sem_post(darwin_sem_t *sem) { int res = 0; pthread_mutex_lock(&sem->__data.local.count_lock); if (sem->id != __SEM_ID_LOCAL) { res = -1; errno = EINVAL; } else if (sem->__data.local.count < DARWIN_SEM_VALUE_MAX) { sem->__data.local.count++; if (sem->__data.local.count == 1) { pthread_cond_signal(&sem->__data.local.count_cond); } } else { errno = ERANGE; res = -1; } pthread_mutex_unlock(&sem->__data.local.count_lock); return res; } /* http://www.opengroup.org/onlinepubs/009695399/functions/sem_timedwait.html */ int darwin_sem_timedwait(darwin_sem_t *sem, const struct timespec *abs_timeout) { int res = 0; if (abs_timeout && (abs_timeout->tv_nsec < 0 || abs_timeout->tv_nsec >= 1000000000)) { errno = EINVAL; return -1; } pthread_cleanup_push((void(*)(void*))&pthread_mutex_unlock, &sem->__data.local.count_lock); pthread_mutex_lock(&sem->__data.local.count_lock); if (sem->id != __SEM_ID_LOCAL) { errno = EINVAL; res = -1; } else { if (!sem->__data.local.count) { res = pthread_cond_timedwait(&sem->__data.local.count_cond, &sem->__data.local.count_lock, abs_timeout); } if (res) { assert(res == ETIMEDOUT); res = -1; errno = ETIMEDOUT; } else if (sem->id != __SEM_ID_LOCAL) { res = -1; errno = EINVAL; } else { sem->__data.local.count--; } } pthread_cleanup_pop(1); return res; } /* http://www.opengroup.org/onlinepubs/007908799/xsh/sem_trywait.html */ int darwin_sem_trywait(darwin_sem_t *sem) { int res = 0; pthread_mutex_lock(&sem->__data.local.count_lock); if (sem->id != __SEM_ID_LOCAL) { res = -1; errno = EINVAL; } else if (sem->__data.local.count) { sem->__data.local.count--; } else { res = -1; errno = EAGAIN; } pthread_mutex_unlock (&sem->__data.local.count_lock); return res; } /* http://www.opengroup.org/onlinepubs/007908799/xsh/sem_wait.html */ int darwin_sem_wait(darwin_sem_t *sem) { /* Must be volatile or will be clobbered by longjmp */ volatile int res = 0; pthread_cleanup_push((void(*)(void*))&pthread_mutex_unlock, &sem->__data.local.count_lock); pthread_mutex_lock(&sem->__data.local.count_lock); if (sem->id != __SEM_ID_LOCAL) { errno = EINVAL; res = -1; } else { if (!sem->__data.local.count) { pthread_cond_wait(&sem->__data.local.count_cond, &sem->__data.local.count_lock); if (!sem->__data.local.count) { /* spurious wakeup, assume it is an interruption */ res = -1; errno = EINTR; goto out; } } if (sem->id != __SEM_ID_LOCAL) { res = -1; errno = EINVAL; } else { sem->__data.local.count--; } } out: pthread_cleanup_pop(1); return res; } ================================================ FILE: src/pyfuse3/darwin_compat.h ================================================ /* * Copyright (c) 2006-2008 Amit Singh/Google Inc. * Copyright (c) 2011-2013 Benjamin Fleischer */ #ifndef _DARWIN_COMPAT_ #define _DARWIN_COMPAT_ #include /* Semaphores */ typedef struct darwin_sem { int id; union { struct { unsigned int count; pthread_mutex_t count_lock; pthread_cond_t count_cond; } local; } __data; } darwin_sem_t; #define DARWIN_SEM_VALUE_MAX ((int32_t)32767) int darwin_sem_init(darwin_sem_t *sem, int pshared, unsigned int value); int darwin_sem_destroy(darwin_sem_t *sem); int darwin_sem_getvalue(darwin_sem_t *sem, unsigned int *value); int darwin_sem_post(darwin_sem_t *sem); int darwin_sem_timedwait(darwin_sem_t *sem, const struct timespec *abs_timeout); int darwin_sem_trywait(darwin_sem_t *sem); int darwin_sem_wait(darwin_sem_t *sem); /* Caller must not include */ typedef darwin_sem_t sem_t; #define sem_init(s, p, v) darwin_sem_init(s, p, v) #define sem_destroy(s) darwin_sem_destroy(s) #define sem_getvalue(s, v) darwin_sem_getvalue(s, v) #define sem_post(s) darwin_sem_post(s) #define sem_timedwait(s, t) darwin_sem_timedwait(s, t) #define sem_trywait(s) darwin_sem_trywait(s) #define sem_wait(s) darwin_sem_wait(s) #define SEM_VALUE_MAX DARWIN_SEM_VALUE_MAX #endif /* _DARWIN_COMPAT_ */ ================================================ FILE: src/pyfuse3/gettime.h ================================================ /* * gettime.h * * Platform-independent interface to system clock * * Copyright © 2015 Nikolaus Rath * * This file is part of pyfuse3. This work may be distributed under the * terms of the GNU LGPL. */ /* * Linux */ #if PLATFORM == PLATFORM_LINUX #include static int gettime_realtime(struct timespec *tp) { return clock_gettime(CLOCK_REALTIME, tp); } /* * FreeBSD & NetBSD */ #elif PLATFORM == PLATFORM_BSD #include static int gettime_realtime(struct timespec *tp) { return clock_gettime(CLOCK_REALTIME, tp); } /* * Darwin */ #elif PLATFORM == PLATFORM_DARWIN #include static int gettime_realtime(struct timespec *tp) { struct timeval tv; int res; res = gettimeofday(&tv, NULL); if(res != 0) return -1; tp->tv_sec = tv.tv_sec; tp->tv_nsec = tv.tv_usec * 1000; return 0; } /* * Unknown system */ #else #error This should not happen #endif ================================================ FILE: src/pyfuse3/handlers.pxi ================================================ ''' handlers.pxi This file defines the FUSE request handlers. It is included by __init__.pyx. Copyright © 2013 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' @cython.freelist(60) cdef class _Container: """For internal use by pyfuse3 only.""" # This serves as a generic container to pass C variables # through Python. Which fields have valid data depends on # context. cdef dev_t rdev cdef fuse_file_info fi cdef fuse_ino_t ino cdef fuse_ino_t parent cdef fuse_req_t req cdef int flags cdef mode_t mode cdef off_t off cdef size_t size cdef struct_stat stat cdef uint64_t fh cdef void fuse_init (void *userdata, fuse_conn_info *conn): if not conn.capable & FUSE_CAP_READDIRPLUS: raise RuntimeError('Kernel too old, pyfuse3 requires kernel 3.9 or newer!') conn.want &= ~( FUSE_CAP_READDIRPLUS_AUTO) if (operations.supports_dot_lookup and conn.capable & FUSE_CAP_EXPORT_SUPPORT): conn.want |= FUSE_CAP_EXPORT_SUPPORT if (operations.enable_writeback_cache and conn.capable & FUSE_CAP_WRITEBACK_CACHE): conn.want |= FUSE_CAP_WRITEBACK_CACHE if (operations.enable_acl and conn.capable & FUSE_CAP_POSIX_ACL): conn.want |= FUSE_CAP_POSIX_ACL # Blocking rather than async, in case we decide to let the # init handler modify `conn` in the future. operations.init() cdef void fuse_lookup (fuse_req_t req, fuse_ino_t parent, const_char *name): cdef _Container c = _Container() c.req = req c.parent = parent save_retval(fuse_lookup_async(c, PyBytes_FromString(name))) async def fuse_lookup_async (_Container c, name): cdef EntryAttributes entry cdef int ret ctx = get_request_context(c.req) try: entry = await operations.lookup( c.parent, name, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_entry(c.req, &entry.fuse_param) if ret != 0: log.error('fuse_lookup(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_forget (fuse_req_t req, fuse_ino_t ino, uint64_t nlookup): save_retval(operations.forget([(ino, nlookup)])) fuse_reply_none(req) cdef void fuse_forget_multi(fuse_req_t req, size_t count, fuse_forget_data *forgets): forget_list = list() for el in forgets[:count]: forget_list.append((el.ino, el.nlookup)) save_retval(operations.forget(forget_list)) fuse_reply_none(req) cdef void fuse_getattr (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.ino = ino save_retval(fuse_getattr_async(c)) async def fuse_getattr_async (_Container c): cdef int ret cdef EntryAttributes entry ctx = get_request_context(c.req) try: entry = await operations.getattr(c.ino, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_attr(c.req, entry.attr, entry.fuse_param.attr_timeout) if ret != 0: log.error('fuse_getattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_setattr (fuse_req_t req, fuse_ino_t ino, struct_stat *stat, int to_set, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.ino = ino c.stat = stat[0] c.flags = to_set if fi is NULL: fh = None else: fh = fi.fh save_retval(fuse_setattr_async(c, fh)) async def fuse_setattr_async (_Container c, fh): cdef int ret cdef timespec now cdef EntryAttributes entry cdef SetattrFields fields cdef struct_stat *attr cdef int to_set = c.flags ctx = get_request_context(c.req) entry = EntryAttributes() fields = SetattrFields.__new__(SetattrFields) string.memcpy(entry.attr, &c.stat, sizeof(struct_stat)) attr = entry.attr if to_set & (FUSE_SET_ATTR_ATIME_NOW | FUSE_SET_ATTR_MTIME_NOW): ret = libc_extra.gettime_realtime(&now) if ret != 0: log.error('fuse_setattr(): clock_gettime(CLOCK_REALTIME) failed with %s', strerror(errno.errno)) if to_set & FUSE_SET_ATTR_ATIME: fields.update_atime = True elif to_set & FUSE_SET_ATTR_ATIME_NOW: fields.update_atime = True attr.st_atime = now.tv_sec SET_ATIME_NS(attr, now.tv_nsec) if to_set & FUSE_SET_ATTR_MTIME: fields.update_mtime = True elif to_set & FUSE_SET_ATTR_MTIME_NOW: fields.update_mtime = True attr.st_mtime = now.tv_sec SET_MTIME_NS(attr, now.tv_nsec) fields.update_ctime = bool(to_set & FUSE_SET_ATTR_CTIME) fields.update_mode = bool(to_set & FUSE_SET_ATTR_MODE) fields.update_uid = bool(to_set & FUSE_SET_ATTR_UID) fields.update_gid = bool(to_set & FUSE_SET_ATTR_GID) fields.update_size = bool(to_set & FUSE_SET_ATTR_SIZE) try: entry = await operations.setattr(c.ino, entry, fields, fh, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_attr(c.req, entry.attr, entry.fuse_param.attr_timeout) if ret != 0: log.error('fuse_setattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_readlink (fuse_req_t req, fuse_ino_t ino): cdef _Container c = _Container() c.req = req c.ino = ino save_retval(fuse_readlink_async(c)) async def fuse_readlink_async (_Container c): cdef int ret cdef char* name ctx = get_request_context(c.req) try: target = await operations.readlink(c.ino, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: name = PyBytes_AsString(target) ret = fuse_reply_readlink(c.req, name) if ret != 0: log.error('fuse_readlink(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_mknod (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode, dev_t rdev): cdef _Container c = _Container() c.req = req c.parent = parent c.mode = mode c.rdev = rdev save_retval(fuse_mknod_async(c, PyBytes_FromString(name))) async def fuse_mknod_async (_Container c, name): cdef int ret cdef EntryAttributes entry ctx = get_request_context(c.req) try: entry = await operations.mknod( c.parent, name, c.mode, c.rdev, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_entry(c.req, &entry.fuse_param) if ret != 0: log.error('fuse_mknod(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_mkdir (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode): cdef _Container c = _Container() c.req = req c.parent = parent c.mode = mode save_retval(fuse_mkdir_async(c, PyBytes_FromString(name))) async def fuse_mkdir_async (_Container c, name): cdef int ret cdef EntryAttributes entry # Force the entry type to directory. We need to explicitly cast, # because on BSD the S_* are not of type mode_t. c.mode = (c.mode & ~ S_IFMT) | S_IFDIR ctx = get_request_context(c.req) try: entry = await operations.mkdir( c.parent, name, c.mode, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_entry(c.req, &entry.fuse_param) if ret != 0: log.error('fuse_mkdir(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_unlink (fuse_req_t req, fuse_ino_t parent, const_char *name): cdef _Container c = _Container() c.req = req c.parent = parent save_retval(fuse_unlink_async(c, PyBytes_FromString(name))) async def fuse_unlink_async (_Container c, name): cdef int ret ctx = get_request_context(c.req) try: await operations.unlink(c.parent, name, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_unlink(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_rmdir (fuse_req_t req, fuse_ino_t parent, const_char *name): cdef _Container c = _Container() c.req = req c.parent = parent save_retval(fuse_rmdir_async(c, PyBytes_FromString(name))) async def fuse_rmdir_async (_Container c, name): cdef int ret ctx = get_request_context(c.req) try: await operations.rmdir(c.parent, name, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_rmdir(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_symlink (fuse_req_t req, const_char *link, fuse_ino_t parent, const_char *name): cdef _Container c = _Container() c.req = req c.parent = parent save_retval(fuse_symlink_async( c, PyBytes_FromString(name), PyBytes_FromString(link))) async def fuse_symlink_async (_Container c, name, link): cdef int ret cdef EntryAttributes entry ctx = get_request_context(c.req) try: entry = await operations.symlink( c.parent, name, link, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_entry(c.req, &entry.fuse_param) if ret != 0: log.error('fuse_symlink(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_rename (fuse_req_t req, fuse_ino_t parent, const_char *name, fuse_ino_t newparent, const_char *newname, unsigned flags): cdef _Container c = _Container() c.req = req c.parent = parent c.ino = newparent c.flags = flags save_retval(fuse_rename_async( c, PyBytes_FromString(name), PyBytes_FromString(newname))) async def fuse_rename_async (_Container c, name, newname): cdef int ret cdef unsigned flags = c.flags cdef fuse_ino_t newparent = c.ino ctx = get_request_context(c.req) try: await operations.rename(c.parent, name, newparent, newname, flags, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_rename(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_link (fuse_req_t req, fuse_ino_t ino, fuse_ino_t newparent, const_char *newname): cdef _Container c = _Container() c.req = req c.ino = ino c.parent = newparent save_retval(fuse_link_async(c, PyBytes_FromString(newname))) async def fuse_link_async (_Container c, newname): cdef int ret cdef EntryAttributes entry ctx = get_request_context(c.req) try: entry = await operations.link( c.ino, c.parent, newname, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_entry(c.req, &entry.fuse_param) if ret != 0: log.error('fuse_link(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_open (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.ino = ino c.fi = fi[0] save_retval(fuse_open_async(c)) async def fuse_open_async (_Container c): cdef int ret cdef FileInfo fi ctx = get_request_context(c.req) try: fi = await operations.open(c.ino, c.fi.flags, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: fi._copy_to_fuse(&c.fi) ret = fuse_reply_open(c.req, &c.fi) if ret != 0: log.error('fuse_link(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_read (fuse_req_t req, fuse_ino_t ino, size_t size, off_t off, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.size = size c.off = off c.fh = fi.fh save_retval(fuse_read_async(c)) async def fuse_read_async (_Container c): cdef int ret cdef Py_buffer pybuf try: buf = await operations.read(c.fh, c.off, c.size) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: PyObject_GetBuffer(buf, &pybuf, PyBUF_CONTIG_RO) ret = fuse_reply_buf(c.req, pybuf.buf, pybuf.len) PyBuffer_Release(&pybuf) if ret != 0: log.error('fuse_read(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_write (fuse_req_t req, fuse_ino_t ino, const_char *buf, size_t size, off_t off, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.size = size c.off = off c.fh = fi.fh if size > PY_SSIZE_T_MAX: raise OverflowError('Value too long to convert to Python') pbuf = PyBytes_FromStringAndSize(buf, size) save_retval(fuse_write_async(c, pbuf)) async def fuse_write_async (_Container c, pbuf): cdef int ret cdef size_t len_ try: len_ = await operations.write(c.fh, c.off, pbuf) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_write(c.req, len_) if ret != 0: log.error('fuse_write(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_write_buf(fuse_req_t req, fuse_ino_t ino, fuse_bufvec *bufv, off_t off, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.off = off c.fh = fi.fh buf = PyBytes_from_bufvec(bufv) save_retval(fuse_write_buf_async(c, buf)) async def fuse_write_buf_async (_Container c, buf): cdef int ret cdef size_t len_ try: len_ = await operations.write(c.fh, c.off, buf) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_write(c.req, len_) if ret != 0: log.error('fuse_write_buf(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_flush (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.fh = fi.fh save_retval(fuse_flush_async(c)) async def fuse_flush_async (_Container c): cdef int ret try: await operations.flush(c.fh) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_flush(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_release (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.fh = fi.fh save_retval(fuse_release_async(c)) async def fuse_release_async (_Container c): cdef int ret try: await operations.release(c.fh) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_release(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_fsync (fuse_req_t req, fuse_ino_t ino, int datasync, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.flags = datasync c.fh = fi.fh save_retval(fuse_fsync_async(c)) async def fuse_fsync_async (_Container c): cdef int ret try: await operations.fsync(c.fh, c.flags != 0) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_fsync(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_opendir (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.ino = ino c.fi = fi[0] save_retval(fuse_opendir_async(c)) async def fuse_opendir_async (_Container c): cdef int ret ctx = get_request_context(c.req) try: c.fi.fh = await operations.opendir(c.ino, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_open(c.req, &c.fi) if ret != 0: log.error('fuse_opendir(): fuse_reply_* failed with %s', strerror(-ret)) @cython.freelist(10) cdef class ReaddirToken: cdef fuse_req_t req cdef char *buf_start cdef char *buf cdef size_t size cdef void fuse_readdirplus (fuse_req_t req, fuse_ino_t ino, size_t size, off_t off, fuse_file_info *fi): global py_retval cdef _Container c = _Container() c.req = req c.size = size c.off = off c.fh = fi.fh save_retval(fuse_readdirplus_async(c)) async def fuse_readdirplus_async (_Container c): cdef int ret cdef ReaddirToken token = ReaddirToken() token.buf_start = NULL token.size = c.size token.req = c.req try: await operations.readdir(c.fh, c.off, token) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: if token.buf_start == NULL: ret = fuse_reply_buf(c.req, NULL, 0) else: ret = fuse_reply_buf(c.req, token.buf_start, c.size - token.size) finally: stdlib.free(token.buf_start) if ret != 0: log.error('fuse_readdirplus(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_releasedir (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.fh = fi.fh save_retval(fuse_releasedir_async(c)) async def fuse_releasedir_async (_Container c): cdef int ret try: await operations.releasedir(c.fh) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_releasedir(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_fsyncdir (fuse_req_t req, fuse_ino_t ino, int datasync, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.flags = datasync c.fh = fi.fh save_retval(fuse_fsyncdir_async(c)) async def fuse_fsyncdir_async (_Container c): cdef int ret try: await operations.fsyncdir(c.fh, c.flags != 0) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_fsyncdir(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_statfs (fuse_req_t req, fuse_ino_t ino): cdef _Container c = _Container() c.req = req save_retval(fuse_statfs_async(c)) async def fuse_statfs_async (_Container c): cdef int ret cdef StatvfsData stats ctx = get_request_context(c.req) try: stats = await operations.statfs(ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_statfs(c.req, &stats.stat) if ret != 0: log.error('fuse_statfs(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_setxattr (fuse_req_t req, fuse_ino_t ino, const_char *cname, const_char *cvalue, size_t size, int flags): cdef _Container c = _Container() c.req = req c.ino = ino c.size = size c.flags = flags name = PyBytes_FromString(cname) if c.size > PY_SSIZE_T_MAX: raise OverflowError('Value too long to convert to Python') value = PyBytes_FromStringAndSize(cvalue, c.size) save_retval(fuse_setxattr_async(c, name, value)) async def fuse_setxattr_async (_Container c, name, value): cdef int ret # Special case for deadlock debugging if c.ino == FUSE_ROOT_ID and name == 'fuse_stacktrace': operations.stacktrace() fuse_reply_err(c.req, 0) return # Make sure we know all the flags if c.flags & ~(libc_extra.XATTR_CREATE | libc_extra.XATTR_REPLACE): raise ValueError('unknown flag(s): %o' % c.flags) ctx = get_request_context(c.req) try: if c.flags & libc_extra.XATTR_CREATE: # Attribute must not exist try: await operations.getxattr(c.ino, name, ctx) except FUSEError as e: if e.errno != ENOATTR: raise else: raise FUSEError(errno.EEXIST) elif c.flags & libc_extra.XATTR_REPLACE: # Attribute must exist await operations.getxattr(c.ino, name, ctx) await operations.setxattr(c.ino, name, value, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_setxattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_getxattr (fuse_req_t req, fuse_ino_t ino, const_char *name, size_t size): cdef _Container c = _Container() c.req = req c.ino = ino c.size = size save_retval(fuse_getxattr_async(c, PyBytes_FromString(name))) async def fuse_getxattr_async (_Container c, name): cdef int ret cdef ssize_t len_s cdef size_t len_ cdef char *cbuf ctx = get_request_context(c.req) try: buf = await operations.getxattr(c.ino, name, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: PyBytes_AsStringAndSize(buf, &cbuf, &len_s) len_ = len_s # guaranteed positive if c.size == 0: ret = fuse_reply_xattr(c.req, len_) elif len_ <= c.size: ret = fuse_reply_buf(c.req, cbuf, len_) else: ret = fuse_reply_err(c.req, errno.ERANGE) if ret != 0: log.error('fuse_getxattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_listxattr (fuse_req_t req, fuse_ino_t ino, size_t size): cdef _Container c = _Container() c.req = req c.ino = ino c.size = size save_retval(fuse_listxattr_async(c)) async def fuse_listxattr_async (_Container c): cdef int ret cdef ssize_t len_s cdef size_t len_ cdef char *cbuf ctx = get_request_context(c.req) try: res = await operations.listxattr(c.ino, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: buf = b'\0'.join(res) + b'\0' PyBytes_AsStringAndSize(buf, &cbuf, &len_s) len_ = len_s # guaranteed positive if len_ == 1: # No attributes len_ = 0 if c.size == 0: ret = fuse_reply_xattr(c.req, len_) elif len_ <= c.size: ret = fuse_reply_buf(c.req, cbuf, len_) else: ret = fuse_reply_err(c.req, errno.ERANGE) if ret != 0: log.error('fuse_listxattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_removexattr (fuse_req_t req, fuse_ino_t ino, const_char *name): cdef _Container c = _Container() c.req = req c.ino = ino save_retval(fuse_removexattr_async(c, PyBytes_FromString(name))) async def fuse_removexattr_async (_Container c, name): cdef int ret ctx = get_request_context(c.req) try: await operations.removexattr(c.ino, name, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: ret = fuse_reply_err(c.req, 0) if ret != 0: log.error('fuse_removexattr(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_access (fuse_req_t req, fuse_ino_t ino, int mask): cdef _Container c = _Container() c.req = req c.ino = ino c.flags = mask save_retval(fuse_access_async(c)) async def fuse_access_async (_Container c): cdef int ret cdef int mask = c.flags ctx = get_request_context(c.req) try: allowed = await operations.access(c.ino, mask, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: if allowed: ret = fuse_reply_err(c.req, 0) else: ret = fuse_reply_err(c.req, EACCES) if ret != 0: log.error('fuse_access(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_poll (fuse_req_t req, fuse_ino_t ino, fuse_file_info *fi, fuse_pollhandle *ph): cdef _Container c = _Container() cdef object py_ph c.req = req c.ino = ino if fi is NULL: c.fh = 0 else: c.fh = fi.fh if ph == NULL: py_ph = None else: py_ph = PollHandle.from_ptr(ph) save_retval(fuse_poll_async(c, py_ph)) async def fuse_poll_async (_Container c, object py_ph): cdef int ret cdef unsigned revents ctx = get_request_context(c.req) try: result = await operations.poll(c.ino, c.fh, py_ph, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: revents = (result if result is not None else 0) ret = fuse_reply_poll(c.req, revents) if ret != 0: log.error('fuse_poll(): fuse_reply_* failed with %s', strerror(-ret)) cdef void fuse_create (fuse_req_t req, fuse_ino_t parent, const_char *name, mode_t mode, fuse_file_info *fi): cdef _Container c = _Container() c.req = req c.parent = parent c.mode = mode c.fi = fi[0] save_retval(fuse_create_async(c, PyBytes_FromString(name))) async def fuse_create_async (_Container c, name): cdef int ret cdef EntryAttributes entry cdef FileInfo fi ctx = get_request_context(c.req) try: tmp = await operations.create(c.parent, name, c.mode, c.fi.flags, ctx) except FUSEError as e: ret = fuse_reply_err(c.req, e.errno) else: fi = tmp[0] entry = tmp[1] fi._copy_to_fuse(&c.fi) ret = fuse_reply_create(c.req, &entry.fuse_param, &c.fi) if ret != 0: log.error('fuse_create(): fuse_reply_* failed with %s', strerror(-ret)) ================================================ FILE: src/pyfuse3/internal.pxi ================================================ ''' internal.pxi This file defines functions and data structures that are used internally by pyfuse3. It is included by __init__.pyx. Copyright © 2013 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' cdef void save_retval(object val): global py_retval if py_retval is not None and val is not None: log.error('py_retval was not awaited - please report a bug at ' 'https://github.com/libfuse/pyfuse3/issues!') py_retval = val cdef object get_request_context(fuse_req_t req): '''Get RequestContext() object''' cdef const_fuse_ctx* context cdef RequestContext ctx context = fuse_req_ctx(req) ctx = RequestContext.__new__(RequestContext) ctx.pid = context.pid ctx.uid = context.uid ctx.gid = context.gid ctx.umask = context.umask return ctx cdef void init_fuse_ops(): '''Initialize fuse_lowlevel_ops structure''' string.memset(&fuse_ops, 0, sizeof(fuse_lowlevel_ops)) fuse_ops.init = fuse_init fuse_ops.lookup = fuse_lookup fuse_ops.forget = fuse_forget fuse_ops.getattr = fuse_getattr fuse_ops.setattr = fuse_setattr fuse_ops.readlink = fuse_readlink fuse_ops.mknod = fuse_mknod fuse_ops.mkdir = fuse_mkdir fuse_ops.unlink = fuse_unlink fuse_ops.rmdir = fuse_rmdir fuse_ops.symlink = fuse_symlink fuse_ops.rename = fuse_rename fuse_ops.link = fuse_link fuse_ops.open = fuse_open fuse_ops.read = fuse_read fuse_ops.write = fuse_write fuse_ops.flush = fuse_flush fuse_ops.release = fuse_release fuse_ops.fsync = fuse_fsync fuse_ops.opendir = fuse_opendir fuse_ops.readdirplus = fuse_readdirplus fuse_ops.releasedir = fuse_releasedir fuse_ops.fsyncdir = fuse_fsyncdir fuse_ops.statfs = fuse_statfs ASSIGN_NOT_DARWIN(fuse_ops.setxattr, &fuse_setxattr) ASSIGN_NOT_DARWIN(fuse_ops.getxattr, &fuse_getxattr) fuse_ops.listxattr = fuse_listxattr fuse_ops.removexattr = fuse_removexattr fuse_ops.access = fuse_access fuse_ops.create = fuse_create fuse_ops.forget_multi = fuse_forget_multi fuse_ops.write_buf = fuse_write_buf fuse_ops.poll = fuse_poll cdef make_fuse_args(args, fuse_args* f_args): cdef char* arg cdef int i cdef ssize_t size_s cdef size_t size args_new = [ b'pyfuse3' ] for el in args: args_new.append(b'-o') args_new.append(el.encode('us-ascii')) args = args_new f_args.argc = len(args) if f_args.argc == 0: f_args.argv = NULL return f_args.allocated = 1 f_args.argv = stdlib.calloc( f_args.argc, sizeof(char*)) if f_args.argv is NULL: cpython.exc.PyErr_NoMemory() try: for (i, el) in enumerate(args): PyBytes_AsStringAndSize(el, &arg, &size_s) size = size_s # guaranteed positive f_args.argv[i] = stdlib.malloc((size+1)*sizeof(char)) if f_args.argv[i] is NULL: cpython.exc.PyErr_NoMemory() string.strncpy(f_args.argv[i], arg, size+1) except: for i in range(f_args.argc): # Freeing a NULL pointer (if this element has not been allocated # yet) is fine. stdlib.free(f_args.argv[i]) stdlib.free(f_args.argv) raise def _notify_loop(): '''Process async invalidate_entry calls.''' while True: req = _notify_queue.get() if req is None: log.debug('terminating notify thread') break (inode_p, name, deleted, ignore_enoent) = req try: invalidate_entry(inode_p, name, deleted) except Exception as exc: if ignore_enoent and isinstance(exc, FileNotFoundError): pass else: log.exception('Failed to submit invalidate_entry request for ' 'parent inode %d, name %s', req[0], req[1]) cdef str2bytes(s): '''Convert *s* to bytes''' return s.encode(fse, 'surrogateescape') cdef bytes2str(s): '''Convert *s* to str''' return s.decode(fse, 'surrogateescape') cdef strerror(int errno): try: return os.strerror(errno) except ValueError: return 'errno: %d' % errno cdef PyBytes_from_bufvec(fuse_bufvec *src): cdef fuse_bufvec dst cdef size_t len_ cdef ssize_t res len_ = fuse_buf_size(src) - src.off if len_ > PY_SSIZE_T_MAX: raise OverflowError('Value too long to convert to Python') buf = PyBytes_FromStringAndSize(NULL, len_) dst.count = 1 dst.idx = 0 dst.off = 0 dst.buf[0].mem = PyBytes_AS_STRING(buf) dst.buf[0].size = len_ dst.buf[0].flags = 0 res = fuse_buf_copy(&dst, src, 0) if res < 0: raise OSError(errno.errno, 'fuse_buf_copy failed with ' + strerror(errno.errno)) elif res < len_: # This is expected to be rare return buf[:res] else: return buf cdef void* calloc_or_raise(size_t nmemb, size_t size) except NULL: cdef void* mem mem = stdlib.calloc(nmemb, size) if mem is NULL: raise MemoryError() return mem cdef class _WorkerData: """For internal use by pyfuse3 only.""" cdef int task_count cdef int task_serial cdef object read_lock cdef int active_readers def __init__(self): self.read_lock = None self.active_readers = 0 cdef get_name(self): self.task_serial += 1 return 'pyfuse-%02d' % self.task_serial # Delay initialization so that pyfuse3.asyncio can replace # the trio module. cdef _WorkerData worker_data async def _wait_fuse_readable(): '''Wait for FUSE fd to become readable Return True if the fd is readable, or False if the main loop should terminate. ''' if worker_data.read_lock is None: worker_data.read_lock = trio.Lock() #name = trio.lowlevel.current_task().name worker_data.active_readers += 1 try: #log.debug('%s: Waiting for read lock...', name) async with worker_data.read_lock: #log.debug('%s: Waiting for fuse fd to become readable...', name) if fuse_session_exited(session): log.debug('FUSE session exit flag set while waiting for FUSE fd ' 'to become readable.') return False await trio.lowlevel.wait_readable(session_fd) #log.debug('%s: fuse fd readable, unparking next task.', name) except trio.ClosedResourceError: log.debug('FUSE fd about to be closed.') return False finally: worker_data.active_readers -= 1 return True @async_wrapper async def _session_loop(nursery, int min_tasks, int max_tasks): cdef int res cdef fuse_buf buf name = trio.lowlevel.current_task().name buf.mem = NULL buf.size = 0 buf.pos = 0 buf.flags = 0 while not fuse_session_exited(session): if worker_data.active_readers > min_tasks: log.debug('%s: too many idle tasks (%d total, %d waiting), terminating.', name, worker_data.task_count, worker_data.active_readers) break if not await _wait_fuse_readable(): break res = fuse_session_receive_buf(session, &buf) if not worker_data.active_readers and worker_data.task_count < max_tasks: worker_data.task_count += 1 log.debug('%s: No tasks waiting, starting another worker (now %d total).', name, worker_data.task_count) nursery.start_soon(_session_loop, nursery, min_tasks, max_tasks, name=worker_data.get_name()) if res == -errno.EINTR: continue elif res < 0: raise OSError(-res, 'fuse_session_receive_buf failed with ' + strerror(-res)) elif res == 0: break # When fuse_session_process_buf() calls back into one of our handler # methods, the handler will start a co-routine and store it in # py_retval. #log.debug('%s: processing request...', name) save_retval(None) fuse_session_process_buf(session, &buf) if py_retval is not None: await py_retval #log.debug('%s: processing complete.', name) log.debug('%s: terminated', name) stdlib.free(buf.mem) worker_data.task_count -= 1 ================================================ FILE: src/pyfuse3/macros.c ================================================ /* macros.c - Pre-processor macros Copyright © 2013 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. */ /* * Macros to access the nanosecond attributes in struct stat in a * platform independent way. Stolen from fuse_misc.h. */ #if PLATFORM == PLATFORM_LINUX #define GET_ATIME_NS(stbuf) ((stbuf)->st_atim.tv_nsec) #define GET_CTIME_NS(stbuf) ((stbuf)->st_ctim.tv_nsec) #define GET_MTIME_NS(stbuf) ((stbuf)->st_mtim.tv_nsec) #define SET_ATIME_NS(stbuf, val) (stbuf)->st_atim.tv_nsec = (val) #define SET_CTIME_NS(stbuf, val) (stbuf)->st_ctim.tv_nsec = (val) #define SET_MTIME_NS(stbuf, val) (stbuf)->st_mtim.tv_nsec = (val) #define GET_BIRTHTIME_NS(stbuf) (0) #define GET_BIRTHTIME(stbuf) (0) #define SET_BIRTHTIME_NS(stbuf, val) do {} while (0) #define SET_BIRTHTIME(stbuf, val) do {} while (0) /* BSD and OS-X */ #else #define GET_BIRTHTIME(stbuf) ((stbuf)->st_birthtime) #define SET_BIRTHTIME(stbuf, val) ((stbuf)->st_birthtime = (val)) #define GET_ATIME_NS(stbuf) ((stbuf)->st_atimespec.tv_nsec) #define GET_CTIME_NS(stbuf) ((stbuf)->st_ctimespec.tv_nsec) #define GET_MTIME_NS(stbuf) ((stbuf)->st_mtimespec.tv_nsec) #define GET_BIRTHTIME_NS(stbuf) ((stbuf)->st_birthtimespec.tv_nsec) #define SET_ATIME_NS(stbuf, val) ((stbuf)->st_atimespec.tv_nsec = (val)) #define SET_CTIME_NS(stbuf, val) ((stbuf)->st_ctimespec.tv_nsec = (val)) #define SET_MTIME_NS(stbuf, val) ((stbuf)->st_mtimespec.tv_nsec = (val)) #define SET_BIRTHTIME_NS(stbuf, val) ((stbuf)->st_birthtimespec.tv_nsec = (val)) #endif #if PLATFORM == PLATFORM_LINUX || PLATFORM == PLATFORM_BSD #define ASSIGN_DARWIN(x,y) #define ASSIGN_NOT_DARWIN(x,y) ((x) = (y)) #elif PLATFORM == PLATFORM_DARWIN #define ASSIGN_DARWIN(x,y) ((x) = (y)) #define ASSIGN_NOT_DARWIN(x,y) #else #error This should not happen #endif ================================================ FILE: src/pyfuse3/macros.pxd ================================================ ''' macros.pxd Cython definitions for macros.c Copyright © 2018 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' from posix.stat cimport struct_stat cdef extern from "macros.c" nogil: long GET_BIRTHTIME(struct_stat* buf) long GET_ATIME_NS(struct_stat* buf) long GET_CTIME_NS(struct_stat* buf) long GET_MTIME_NS(struct_stat* buf) long GET_BIRTHTIME_NS(struct_stat* buf) void SET_BIRTHTIME(struct_stat* buf, long val) void SET_ATIME_NS(struct_stat* buf, long val) void SET_CTIME_NS(struct_stat* buf, long val) void SET_MTIME_NS(struct_stat* buf, long val) void SET_BIRTHTIME_NS(struct_stat* buf, long val) void ASSIGN_DARWIN(void*, void*) void ASSIGN_NOT_DARWIN(void*, void*) ================================================ FILE: src/pyfuse3/py.typed ================================================ ================================================ FILE: src/pyfuse3/pyfuse3.h ================================================ /* pyfuse3.h Copyright © 2013 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. */ #define PLATFORM_LINUX 1 #define PLATFORM_BSD 2 #define PLATFORM_DARWIN 3 #ifdef __linux__ #define PLATFORM PLATFORM_LINUX #elif __FreeBSD_kernel__ && __GLIBC__ #define PLATFORM PLATFORM_LINUX #elif __FreeBSD__ #define PLATFORM PLATFORM_BSD #elif __NetBSD__ #define PLATFORM PLATFORM_BSD #elif __APPLE__ && __MACH__ #define PLATFORM PLATFORM_DARWIN #else #error "Unable to determine system (Linux/FreeBSD/NetBSD/Darwin)" #endif #if PLATFORM == PLATFORM_DARWIN #include "darwin_compat.h" #else /* See also: Include/pthreads.pxd */ #include #endif #include #if FUSE_VERSION < 32 #error FUSE version too old, 3.2.0 or newer required #endif ================================================ FILE: src/pyfuse3/xattr.h ================================================ /* * xattr.h * * Platform-independent interface to extended attributes * * Copyright © 2015 Nikolaus Rath * * This file is part of pyfuse3. This work may be distributed under the * terms of the GNU LGPL. */ #ifndef UNUSED # if defined(__GNUC__) # if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4)) # define UNUSED __attribute__ ((__unused__)) # else # define UNUSED # endif # else # define UNUSED # endif #endif /* * Linux */ #if PLATFORM == PLATFORM_LINUX #include /* * Newer versions of attr deprecate attr/xattr.h which defines ENOATTR as a * synonym for ENODATA. To keep compatibility with the old style and the new, * define this ourselves. */ #ifndef ENOATTR #define ENOATTR ENODATA #endif #define EXTATTR_NAMESPACE_USER 0 #define EXTATTR_NAMESPACE_SYSTEM 0 #define XATTR_NOFOLLOW 0 #define XATTR_NODEFAULT 0 #define XATTR_NOSECURITY 0 static ssize_t getxattr_p (char *path, char *name, void *value, size_t size, UNUSED int namespace) { return getxattr(path, name, value, size); } static int setxattr_p (char *path, char *name, void *value, size_t size, UNUSED int namespace) { return setxattr(path, name, value, size, 0); } /* * FreeBSD & NetBSD */ #elif PLATFORM == PLATFORM_BSD #include #include #include #include #define XATTR_NOFOLLOW 0 #define XATTR_NODEFAULT 0 #define XATTR_NOSECURITY 0 /* FreeBSD doesn't have on operation to only set the attribute if it already exists (XATTR_REPLACE), or only if it does not yet exist (XATTR_CREATE). Setting these values to zero ensures that we can never test positively for them */ #define XATTR_CREATE 0 #define XATTR_REPLACE 0 static ssize_t getxattr_p (char *path, char *name, void *value, size_t size, int namespace) { /* If size > SSIZE_MAX, we cannot determine if we got all the data (because the return value doesn't fit into ssize_t) */ if (size >= SSIZE_MAX) { errno = EINVAL; return -1; } ssize_t ret; ret = extattr_get_file(path, namespace, name, value, size); if (ret > 0 && (size_t) ret == size) { errno = ERANGE; return -1; } return ret; } static int setxattr_p (char *path, char *name, void *value, size_t size, int namespace) { if (size >= SSIZE_MAX) { errno = EINVAL; return -1; } ssize_t ret; ret = extattr_set_file(path, namespace, name, value, size); if (ret < 0) { /* Errno values really ought to fit into int, but better safe than sorry */ if (ret < INT_MIN) return -EOVERFLOW; else return (int) ret; } if ((size_t)ret != size) { errno = ENOSPC; return -1; } return 0; } /* * Darwin */ #elif PLATFORM == PLATFORM_DARWIN #include #define EXTATTR_NAMESPACE_USER 0 #define EXTATTR_NAMESPACE_SYSTEM 0 static ssize_t getxattr_p (char *path, char *name, void *value, size_t size, UNUSED int namespace) { return getxattr(path, name, value, size, 0, 0); } static int setxattr_p (char *path, char *name, void *value, size_t size, UNUSED int namespace) { return setxattr(path, name, value, size, 0, 0); } /* * Unknown system */ #else #error This should not happen #endif ================================================ FILE: test/conftest.py ================================================ import gc import logging import os.path import sys import time import pytest # Enable output checks pytest_plugins = 'pytest_checklogs' # Register false positives @pytest.fixture(autouse=True) def register_false_checklog_pos(reg_output): # DeprecationWarnings are unfortunately quite often a result of indirect # imports via third party modules, so we can't actually fix them. reg_output(r'(Pending)?DeprecationWarning', count=0) # Valgrind output reg_output(r'^==\d+== Memcheck, a memory error detector$') reg_output(r'^==\d+== For counts of detected and suppressed errors, rerun with: -v') reg_output(r'^==\d+== ERROR SUMMARY: 0 errors from 0 contexts') def pytest_addoption(parser): group = parser.getgroup("general") group._addoption( "--installed", action="store_true", default=False, help="Test the installed package." ) group = parser.getgroup("terminal reporting") group._addoption( "--logdebug", action="append", metavar='', help="Activate debugging output from for tests. Use `all` " "to get debug messages from all modules. This option can be " "specified multiple times.", ) # If a test fails, wait a moment before retrieving the captured # stdout/stderr. When using a server process, this makes sure that we capture # any potential output of the server that comes *after* a test has failed. For # example, if a request handler raises an exception, the server first signals an # error to FUSE (causing the test to fail), and then logs the exception. Without # the extra delay, the exception will go into nowhere. @pytest.hookimpl(hookwrapper=True) def pytest_pyfunc_call(pyfuncitem): outcome = yield failed = outcome.excinfo is not None if failed: time.sleep(1) def pytest_configure(config): # If we are running from the source directory, make sure that we load # modules from here basedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) if not config.getoption('installed'): pyfuse3_path = os.path.join(basedir, 'src') if os.path.exists(os.path.join(basedir, 'setup.py')) and os.path.exists( os.path.join(basedir, 'src', 'pyfuse3', '__init__.pyx') ): sys.path.insert(0, pyfuse3_path) # Make sure that called processes use the same path pp = os.environ.get('PYTHONPATH', None) if pp: pp = '%s:%s' % (pyfuse3_path, pp) else: pp = pyfuse3_path os.environ['PYTHONPATH'] = pp try: import faulthandler except ImportError: pass else: faulthandler.enable() # When running from VCS repo, enable all warnings if os.path.exists(os.path.join(basedir, 'MANIFEST.in')): import warnings warnings.resetwarnings() warnings.simplefilter('default') # Configure logging. We don't set a default handler but rely on # the catchlog pytest plugin. logdebug = config.getoption('logdebug') root_logger = logging.getLogger() if logdebug is not None: logging.disable(logging.NOTSET) if 'all' in logdebug: root_logger.setLevel(logging.DEBUG) else: for module in logdebug: logging.getLogger(module).setLevel(logging.DEBUG) else: root_logger.setLevel(logging.INFO) logging.disable(logging.DEBUG) logging.captureWarnings(capture=True) # Run gc.collect() at the end of every test, so that we get ResourceWarnings # as early as possible. def pytest_runtest_teardown(item, nextitem): gc.collect() ================================================ FILE: test/pytest.ini ================================================ [pytest] addopts = --verbose --assert=rewrite --tb=native -x markers = uses_fuse ================================================ FILE: test/pytest_checklogs.py ================================================ #!/usr/bin/env python3 ''' pytest_checklogs.py - this file is part of S3QL. Copyright (C) 2008 Nikolaus Rath This work can be distributed under the terms of the GNU GPLv3. py.test plugin to look for suspicious phrases in messages emitted on stdout/stderr or via the logging module. False positives can be registered via a new `reg_output` fixture (for messages to stdout/stderr), and a `assert_logs` function (for logging messages). ''' import functools import logging import re from collections.abc import Generator from contextlib import contextmanager import pytest class CountMessagesHandler(logging.Handler): def __init__(self, level: int = logging.NOTSET) -> None: super().__init__(level) self.count: int = 0 def emit(self, record: logging.LogRecord) -> None: self.count += 1 @contextmanager def assert_logs( pattern: str, level: int = logging.WARNING, count: int | None = None ) -> Generator[None, None, None]: '''Assert that suite emits specified log message *pattern* is matched against the *unformatted* log message, i.e. before any arguments are merged. If *count* is not None, raise an exception unless exactly *count* matching messages are caught. Matched log records will also be flagged so that the caplog fixture does not generate exceptions for them (no matter their severity). ''' def filter(record: logging.LogRecord) -> bool: if record.levelno == level and re.search(pattern, record.msg): record.checklogs_ignore = True return True return False handler = CountMessagesHandler() handler.setLevel(level) handler.addFilter(filter) logger = logging.getLogger() logger.addHandler(handler) try: yield finally: logger.removeHandler(handler) if count is not None and handler.count != count: pytest.fail( 'Expected to catch %d %r messages, but got only %d' % (count, pattern, handler.count) ) def check_test_output(capfd, item): (stdout, stderr) = capfd.readouterr() # Strip out false positives try: false_pos = item.checklogs_fp except AttributeError: false_pos = () for pattern, flags, count in false_pos: cp = re.compile(pattern, flags) (stdout, cnt) = cp.subn('', stdout, count=count) if count == 0 or count - cnt > 0: stderr = cp.sub('', stderr, count=count - cnt) for pattern in ( 'exception', 'error', 'warning', 'fatal', 'traceback', 'fault', 'crash(?:ed)?', 'abort(?:ed)', 'fishy', ): cp = re.compile(r'\b{}\b'.format(pattern), re.IGNORECASE | re.MULTILINE) hit = cp.search(stderr) if hit: pytest.fail('Suspicious output to stderr (matched "%s")' % hit.group(0)) hit = cp.search(stdout) if hit: pytest.fail('Suspicious output to stdout (matched "%s")' % hit.group(0)) def register_output(item, pattern: str, count: int = 1, flags: int = re.MULTILINE) -> None: '''Register *pattern* as false positive for output checking This prevents the test from failing because the output otherwise appears suspicious. ''' item.checklogs_fp.append((pattern, flags, count)) @pytest.fixture() def reg_output(request): assert not hasattr(request.node, 'checklogs_fp') request.node.checklogs_fp = [] return functools.partial(register_output, request.node) # Autouse fixtures are instantiated before explicitly used fixtures, this should also # catch log messages emitted when e.g. initializing resources in other fixtures. @pytest.fixture(autouse=True) def check_output(caplog, capfd, request): yield for when in ("setup", "call", "teardown"): for record in caplog.get_records(when): if record.levelno >= logging.WARNING and not getattr(record, 'checklogs_ignore', False): pytest.fail('Logger received warning messages.') check_test_output(capfd, request.node) ================================================ FILE: test/test_api.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' test_api.py - Unit tests for pyfuse3. Copyright © 2015 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' if __name__ == '__main__': import sys import pytest sys.exit(pytest.main([__file__] + sys.argv[1:])) import errno import os import tempfile from copy import copy from pickle import PicklingError import pytest import pyfuse3 from pyfuse3 import EntryAttributes, RequestContext, SetattrFields, StatvfsData def test_listdir(): # There is a race-condition here if /usr/bin is modified while the test # runs - but hopefully this is sufficiently rare. list1 = set(os.listdir('/usr/bin')) list2 = set(pyfuse3.listdir('/usr/bin')) assert list1 == list2 def test_sup_groups(): gids = pyfuse3.get_sup_groups(os.getpid()) gids2 = set(os.getgroups()) assert gids == gids2 def test_syncfs(): pyfuse3.syncfs('.') def _getxattr_helper(path, name): errno = None try: value = pyfuse3.getxattr(path, name) except OSError as exc: errno = exc.errno value = None if not hasattr(os, 'getxattr'): return value try: value2 = os.getxattr(path, name) except OSError as exc: assert exc.errno == errno else: assert value2 is not None assert value2 == value return value def test_entry_res(): a = EntryAttributes() val = 1000.2735 a.st_atime_ns = int(val * 1e9) assert a.st_atime_ns / 1e9 == val def test_xattr(): with tempfile.NamedTemporaryFile() as fh: key = 'user.new_attribute' assert _getxattr_helper(fh.name, key) is None value = b'a nice little bytestring' try: pyfuse3.setxattr(fh.name, key, value) except OSError as exc: if exc.errno == errno.ENOTSUP: pytest.skip('xattrs not supported for %s' % fh.name) raise assert _getxattr_helper(fh.name, key) == value if not hasattr(os, 'setxattr'): return key = 'user.another_new_attribute' assert _getxattr_helper(fh.name, key) is None value = b'a nice little bytestring, but slightly modified' os.setxattr(fh.name, key, value) assert _getxattr_helper(fh.name, key) == value def test_copy(): for obj in (SetattrFields(), RequestContext()): pytest.raises(PicklingError, copy, obj) for inst, attr in ((EntryAttributes(), 'st_mode'), (StatvfsData(), 'f_files')): setattr(inst, attr, 42) inst_copy = copy(inst) assert getattr(inst, attr) == getattr(inst_copy, attr) exc = pyfuse3.FUSEError(10) assert exc.errno == copy(exc).errno ================================================ FILE: test/test_examples.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' test_examples.py - Unit tests for pyfuse3. Copyright © 2015 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' if __name__ == '__main__': import sys import pytest sys.exit(pytest.main([__file__] + sys.argv[1:])) import errno import filecmp import os import shutil import stat import subprocess import sys from tempfile import NamedTemporaryFile import pytest from pyfuse3 import _NANOS_PER_SEC from util import cleanup, fuse_test_marker, umount, wait_for_mount basename = os.path.join(os.path.dirname(__file__), '..') TEST_FILE = __file__ pytestmark = fuse_test_marker() with open(TEST_FILE, 'rb') as fh: TEST_DATA = fh.read() def name_generator(__ctr=[0]): __ctr[0] += 1 return 'testfile_%d' % __ctr[0] @pytest.mark.parametrize('filename', ('hello.py', 'hello_asyncio.py')) def test_hello(tmpdir, filename): mnt_dir = str(tmpdir) cmdline = [sys.executable, os.path.join(basename, 'examples', filename), mnt_dir] mount_process = subprocess.Popen(cmdline, stdin=subprocess.DEVNULL, universal_newlines=True) try: wait_for_mount(mount_process, mnt_dir) assert os.listdir(mnt_dir) == ['message'] filename = os.path.join(mnt_dir, 'message') with open(filename, 'r') as fh: assert fh.read() == 'hello world\n' with pytest.raises(IOError) as exc_info: open(filename, 'r+') assert exc_info.value.errno == errno.EACCES with pytest.raises(IOError) as exc_info: open(filename + 'does-not-exist', 'r+') assert exc_info.value.errno == errno.ENOENT except: cleanup(mount_process, mnt_dir) raise else: umount(mount_process, mnt_dir) def test_tmpfs(tmpdir): mnt_dir = str(tmpdir) cmdline = [sys.executable, os.path.join(basename, 'examples', 'tmpfs.py'), mnt_dir] mount_process = subprocess.Popen(cmdline, stdin=subprocess.DEVNULL, universal_newlines=True) try: wait_for_mount(mount_process, mnt_dir) tst_write(mnt_dir) tst_mkdir(mnt_dir) tst_symlink(mnt_dir) tst_mknod(mnt_dir) tst_chown(mnt_dir) tst_chmod(mnt_dir) tst_utimens(mnt_dir) tst_rounding(mnt_dir) tst_link(mnt_dir) tst_rename(mnt_dir) tst_readdir(mnt_dir) tst_statvfs(mnt_dir) tst_truncate_path(mnt_dir) tst_truncate_fd(mnt_dir) tst_unlink(mnt_dir) except: cleanup(mount_process, mnt_dir) raise else: umount(mount_process, mnt_dir) @pytest.mark.parametrize('enable_writeback_cache', (True, False)) def test_passthroughfs(tmpdir, enable_writeback_cache): mnt_dir = str(tmpdir.mkdir('mnt')) src_dir = str(tmpdir.mkdir('src')) cmdline = [ sys.executable, os.path.join(basename, 'examples', 'passthroughfs.py'), src_dir, mnt_dir, ] if enable_writeback_cache: cmdline.append('--enable-writeback-cache') mount_process = subprocess.Popen(cmdline, stdin=subprocess.DEVNULL, universal_newlines=True) try: wait_for_mount(mount_process, mnt_dir) tst_write(mnt_dir) tst_mkdir(mnt_dir) tst_symlink(mnt_dir) tst_mknod(mnt_dir) if os.getuid() == 0: tst_chown(mnt_dir) tst_chmod(mnt_dir) # Underlying fs may not have full nanosecond resolution tst_utimens(mnt_dir, ns_tol=1000) tst_rounding(mnt_dir) tst_link(mnt_dir) tst_rename(mnt_dir) tst_readdir(mnt_dir) tst_statvfs(mnt_dir) tst_truncate_path(mnt_dir) tst_truncate_fd(mnt_dir) tst_unlink(mnt_dir) tst_passthrough(src_dir, mnt_dir, enable_writeback_cache=enable_writeback_cache) except: cleanup(mount_process, mnt_dir) raise else: umount(mount_process, mnt_dir) def checked_unlink(filename, path, isdir=False): fullname = os.path.join(path, filename) if isdir: os.rmdir(fullname) else: os.unlink(fullname) with pytest.raises(OSError) as exc_info: os.stat(fullname) assert exc_info.value.errno == errno.ENOENT assert filename not in os.listdir(path) def tst_mkdir(mnt_dir): dirname = name_generator() fullname = mnt_dir + "/" + dirname os.mkdir(fullname) fstat = os.stat(fullname) assert stat.S_ISDIR(fstat.st_mode) assert os.listdir(fullname) == [] assert fstat.st_nlink in (1, 2) assert dirname in os.listdir(mnt_dir) checked_unlink(dirname, mnt_dir, isdir=True) def tst_symlink(mnt_dir): linkname = name_generator() fullname = mnt_dir + "/" + linkname os.symlink("/imaginary/dest", fullname) fstat = os.lstat(fullname) assert stat.S_ISLNK(fstat.st_mode) assert os.readlink(fullname) == "/imaginary/dest" assert fstat.st_nlink == 1 assert linkname in os.listdir(mnt_dir) checked_unlink(linkname, mnt_dir) def tst_mknod(mnt_dir): filename = os.path.join(mnt_dir, name_generator()) shutil.copyfile(TEST_FILE, filename) fstat = os.lstat(filename) assert stat.S_ISREG(fstat.st_mode) assert fstat.st_nlink == 1 assert os.path.basename(filename) in os.listdir(mnt_dir) assert filecmp.cmp(TEST_FILE, filename, False) checked_unlink(filename, mnt_dir) def tst_chown(mnt_dir): filename = os.path.join(mnt_dir, name_generator()) os.mkdir(filename) fstat = os.lstat(filename) uid = fstat.st_uid gid = fstat.st_gid uid_new = uid + 1 os.chown(filename, uid_new, -1) fstat = os.lstat(filename) assert fstat.st_uid == uid_new assert fstat.st_gid == gid gid_new = gid + 1 os.chown(filename, -1, gid_new) fstat = os.lstat(filename) assert fstat.st_uid == uid_new assert fstat.st_gid == gid_new checked_unlink(filename, mnt_dir, isdir=True) def tst_chmod(mnt_dir): filename = os.path.join(mnt_dir, name_generator()) os.mkdir(filename) fstat = os.lstat(filename) mode = stat.S_IMODE(fstat.st_mode) mode_new = 0o640 assert mode != mode_new os.chmod(filename, mode_new) fstat = os.lstat(filename) assert stat.S_IMODE(fstat.st_mode) == mode_new checked_unlink(filename, mnt_dir, isdir=True) def tst_write(mnt_dir): name = os.path.join(mnt_dir, name_generator()) shutil.copyfile(TEST_FILE, name) assert filecmp.cmp(name, TEST_FILE, False) checked_unlink(name, mnt_dir) def tst_unlink(mnt_dir): name = os.path.join(mnt_dir, name_generator()) data1 = b'foo' data2 = b'bar' with open(os.path.join(mnt_dir, name), 'wb+', buffering=0) as fh: fh.write(data1) checked_unlink(name, mnt_dir) fh.write(data2) fh.seek(0) assert fh.read() == data1 + data2 def tst_statvfs(mnt_dir): os.statvfs(mnt_dir) def tst_link(mnt_dir): name1 = os.path.join(mnt_dir, name_generator()) name2 = os.path.join(mnt_dir, name_generator()) shutil.copyfile(TEST_FILE, name1) assert filecmp.cmp(name1, TEST_FILE, False) os.link(name1, name2) fstat1 = os.lstat(name1) fstat2 = os.lstat(name2) assert fstat1 == fstat2 assert fstat1.st_nlink == 2 assert os.path.basename(name2) in os.listdir(mnt_dir) assert filecmp.cmp(name1, name2, False) os.unlink(name2) fstat1 = os.lstat(name1) assert fstat1.st_nlink == 1 os.unlink(name1) def tst_rename(mnt_dir): name1 = os.path.join(mnt_dir, name_generator()) name2 = os.path.join(mnt_dir, name_generator()) shutil.copyfile(TEST_FILE, name1) assert os.path.basename(name1) in os.listdir(mnt_dir) assert os.path.basename(name2) not in os.listdir(mnt_dir) assert filecmp.cmp(name1, TEST_FILE, False) fstat1 = os.lstat(name1) os.rename(name1, name2) fstat2 = os.lstat(name2) assert fstat1 == fstat2 assert filecmp.cmp(name2, TEST_FILE, False) assert os.path.basename(name1) not in os.listdir(mnt_dir) assert os.path.basename(name2) in os.listdir(mnt_dir) os.unlink(name2) def tst_readdir(mnt_dir): dir_ = os.path.join(mnt_dir, name_generator()) file_ = dir_ + "/" + name_generator() subdir = dir_ + "/" + name_generator() subfile = subdir + "/" + name_generator() os.mkdir(dir_) shutil.copyfile(TEST_FILE, file_) os.mkdir(subdir) shutil.copyfile(TEST_FILE, subfile) listdir_is = os.listdir(dir_) listdir_is.sort() listdir_should = [os.path.basename(file_), os.path.basename(subdir)] listdir_should.sort() assert listdir_is == listdir_should os.unlink(file_) os.unlink(subfile) os.rmdir(subdir) os.rmdir(dir_) def tst_truncate_path(mnt_dir): assert len(TEST_DATA) > 1024 filename = os.path.join(mnt_dir, name_generator()) with open(filename, 'wb') as fh: fh.write(TEST_DATA) fstat = os.stat(filename) size = fstat.st_size assert size == len(TEST_DATA) # Add zeros at the end os.truncate(filename, size + 1024) assert os.stat(filename).st_size == size + 1024 with open(filename, 'rb') as fh: assert fh.read(size) == TEST_DATA assert fh.read(1025) == b'\0' * 1024 # Truncate data os.truncate(filename, size - 1024) assert os.stat(filename).st_size == size - 1024 with open(filename, 'rb') as fh: assert fh.read(size) == TEST_DATA[: size - 1024] os.unlink(filename) def tst_truncate_fd(mnt_dir): assert len(TEST_DATA) > 1024 with NamedTemporaryFile('w+b', 0, dir=mnt_dir) as fh: fd = fh.fileno() fh.write(TEST_DATA) fstat = os.fstat(fd) size = fstat.st_size assert size == len(TEST_DATA) # Add zeros at the end os.ftruncate(fd, size + 1024) assert os.fstat(fd).st_size == size + 1024 fh.seek(0) assert fh.read(size) == TEST_DATA assert fh.read(1025) == b'\0' * 1024 # Truncate data os.ftruncate(fd, size - 1024) assert os.fstat(fd).st_size == size - 1024 fh.seek(0) assert fh.read(size) == TEST_DATA[: size - 1024] def tst_utimens(mnt_dir, ns_tol=0): filename = os.path.join(mnt_dir, name_generator()) os.mkdir(filename) fstat = os.lstat(filename) atime = fstat.st_atime + 42.28 mtime = fstat.st_mtime - 42.23 atime_ns = fstat.st_atime_ns + int(42.28 * 1e9) mtime_ns = fstat.st_mtime_ns - int(42.23 * 1e9) os.utime(filename, None, ns=(atime_ns, mtime_ns)) fstat = os.lstat(filename) assert abs(fstat.st_atime - atime) < 1e-3 assert abs(fstat.st_mtime - mtime) < 1e-3 assert abs(fstat.st_atime_ns - atime_ns) <= ns_tol assert abs(fstat.st_mtime_ns - mtime_ns) <= ns_tol checked_unlink(filename, mnt_dir, isdir=True) def tst_rounding(mnt_dir, ns_tol=0): filename = os.path.join(mnt_dir, name_generator()) os.mkdir(filename) fstat = os.lstat(filename) # Approximately 67 years, ending in 999. # Note: 67 years were chosen to avoid y2038 issues (1970 + 67 = 2037). # Testing these is **not** in scope of this test. secs = 67 * 365 * 24 * 3600 + 999 # Max nanos nanos = _NANOS_PER_SEC - 1 # seconds+ns and ns_tol as a float in seconds secs_f = secs + nanos / _NANOS_PER_SEC secs_tol = ns_tol / _NANOS_PER_SEC atime_ns = secs * _NANOS_PER_SEC + nanos mtime_ns = atime_ns os.utime(filename, None, ns=(atime_ns, mtime_ns)) fstat = os.lstat(filename) assert abs(fstat.st_atime - secs_f) <= secs_tol assert abs(fstat.st_mtime - secs_f) <= secs_tol assert abs(fstat.st_atime_ns - atime_ns) <= ns_tol assert abs(fstat.st_mtime_ns - mtime_ns) <= ns_tol checked_unlink(filename, mnt_dir, isdir=True) def tst_passthrough(src_dir, mnt_dir, enable_writeback_cache: bool = False): # Test propagation from source to mirror name = name_generator() src_name = os.path.join(src_dir, name) mnt_name = os.path.join(mnt_dir, name) assert name not in os.listdir(src_dir) assert name not in os.listdir(mnt_dir) with open(src_name, 'w') as fh: fh.write('Hello, world') assert name in os.listdir(src_dir) assert name in os.listdir(mnt_dir) assert_same_stats(src_name, mnt_name, check_times=not enable_writeback_cache) # Test propagation from mirror to source name = name_generator() src_name = os.path.join(src_dir, name) mnt_name = os.path.join(mnt_dir, name) assert name not in os.listdir(src_dir) assert name not in os.listdir(mnt_dir) with open(mnt_name, 'w') as fh: fh.write('Hello, world') assert name in os.listdir(src_dir) assert name in os.listdir(mnt_dir) assert_same_stats(src_name, mnt_name, check_times=not enable_writeback_cache) # Test propagation inside subdirectory name = name_generator() src_dir = os.path.join(src_dir, 'subdir') mnt_dir = os.path.join(mnt_dir, 'subdir') os.mkdir(src_dir) src_name = os.path.join(src_dir, name) mnt_name = os.path.join(mnt_dir, name) assert name not in os.listdir(src_dir) assert name not in os.listdir(mnt_dir) with open(mnt_name, 'w') as fh: fh.write('Hello, world') assert name in os.listdir(src_dir) assert name in os.listdir(mnt_dir) assert_same_stats(src_name, mnt_name, check_times=not enable_writeback_cache) def assert_same_stats(name1, name2, check_times: bool = True): stat1 = os.stat(name1) stat2 = os.stat(name2) for name in ( 'st_atime_ns', 'st_mtime_ns', 'st_ctime_ns', 'st_mode', 'st_ino', 'st_nlink', 'st_uid', 'st_gid', 'st_size', ): v1 = getattr(stat1, name) v2 = getattr(stat2, name) # Known bug, cf. https://github.com/libfuse/pyfuse3/issues/57 if name.endswith('_ns') and os.getenv('CI') == 'true': continue # When FUSE writeback cache is enabled, the kernel maintains mtime/ctime # internally and only flushes them to the underlying filesystem on close. # Until then, the timestamps reported for the passthrough mount and the # backing directory may legitimately differ, so skip strict time checks. if name.endswith('_ns') and not check_times: continue assert v1 == v2, 'Attribute {} differs by {} ({} vs {})'.format(name, v1 - v2, v1, v2) ================================================ FILE: test/test_fs.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' test_fs.py - Unit tests for pyfuse3. Copyright © 2015 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' import sys from typing import cast import pytest if __name__ == '__main__': sys.exit(pytest.main([__file__] + sys.argv[1:])) import errno import logging import multiprocessing import os import select import stat import threading import time import trio import pyfuse3 from pyfuse3 import ( EntryAttributes, FileHandleT, FileInfo, FUSEError, InodeT, PollHandle, ReaddirToken, RequestContext, ) from util import cleanup, fuse_test_marker, umount, wait_for_mount pytestmark = fuse_test_marker() def get_mp(): # We can't use forkserver because we have to make sure # that the server inherits the per-test stdout/stderr file # descriptors. if hasattr(multiprocessing, 'get_context'): mp = multiprocessing.get_context('fork') else: # Older versions only support *fork* anyway mp = multiprocessing # type: ignore[assignment] if threading.active_count() != 1: raise RuntimeError("Multi-threaded test running is not supported") return mp @pytest.fixture() def testfs(tmpdir): yield from _mount_fs(tmpdir, Fs) @pytest.fixture() def pollfs(tmpdir): yield from _mount_fs(tmpdir, PollTestFs) def _mount_fs(tmpdir, fs_class): mnt_dir = str(tmpdir) mp = get_mp() with mp.Manager() as mgr: cross_process = mgr.Namespace() mount_process = mp.Process(target=run_fs, args=(mnt_dir, cross_process, fs_class)) mount_process.start() try: wait_for_mount(mount_process, mnt_dir) yield (mnt_dir, cross_process) except: cleanup(mount_process, mnt_dir) raise else: umount(mount_process, mnt_dir) def test_invalidate_entry(testfs): (mnt_dir, fs_state) = testfs path = os.path.join(mnt_dir, 'message') os.stat(path) assert fs_state.lookup_called fs_state.lookup_called = False os.stat(path) assert not fs_state.lookup_called # Hardcoded sleeptimes - sorry! Needed because of the special semantics of # invalidate_entry() pyfuse3.setxattr(mnt_dir, 'command', b'forget_entry') time.sleep(1.1) os.stat(path) assert fs_state.lookup_called def test_invalidate_inode(testfs): (mnt_dir, fs_state) = testfs with open(os.path.join(mnt_dir, 'message'), 'r') as fh: assert fh.read() == 'hello world\n' assert fs_state.read_called fs_state.read_called = False fh.seek(0) assert fh.read() == 'hello world\n' assert not fs_state.read_called pyfuse3.setxattr(mnt_dir, 'command', b'forget_inode') fh.seek(0) assert fh.read() == 'hello world\n' assert fs_state.read_called def test_notify_store(testfs): (mnt_dir, fs_state) = testfs with open(os.path.join(mnt_dir, 'message'), 'r') as fh: pyfuse3.setxattr(mnt_dir, 'command', b'store') fs_state.read_called = False assert fh.read() == 'hello world\n' assert not fs_state.read_called def test_notify_poll(pollfs): (mnt_dir, fs_state) = pollfs path = os.path.join(mnt_dir, 'message') with open(path, 'rb', buffering=0) as fh: poller = select.poll() poller.register(fh.fileno(), select.POLLPRI) events = [] def poll_wait(): events.extend(poller.poll(5000)) thread = threading.Thread(target=poll_wait) thread.start() deadline = time.monotonic() + 5 while time.monotonic() < deadline and not fs_state.poll_handle_received: time.sleep(0.01) assert fs_state.poll_called assert fs_state.poll_handle_received assert not events pyfuse3.setxattr(path, 'command', b'poll_ready') thread.join(5) assert not thread.is_alive() assert events assert events[0][0] == fh.fileno() assert events[0][1] & select.POLLPRI def test_entry_timeout(testfs): (mnt_dir, fs_state) = testfs fs_state.entry_timeout = 1 path = os.path.join(mnt_dir, 'message') os.stat(path) assert fs_state.lookup_called fs_state.lookup_called = False os.stat(path) assert not fs_state.lookup_called time.sleep(fs_state.entry_timeout * 1.1) fs_state.lookup_called = False os.stat(path) assert fs_state.lookup_called def test_attr_timeout(testfs): (mnt_dir, fs_state) = testfs fs_state.attr_timeout = 1 with open(os.path.join(mnt_dir, 'message'), 'r') as fh: os.fstat(fh.fileno()) assert fs_state.getattr_called fs_state.getattr_called = False os.fstat(fh.fileno()) assert not fs_state.getattr_called time.sleep(fs_state.attr_timeout * 1.1) fs_state.getattr_called = False os.fstat(fh.fileno()) assert fs_state.getattr_called def test_terminate(tmpdir): mnt_dir = str(tmpdir) mp = get_mp() with mp.Manager() as mgr: fs_state = mgr.Namespace() mount_process = mp.Process(target=run_fs, args=(mnt_dir, fs_state)) mount_process.start() try: wait_for_mount(mount_process, mnt_dir) pyfuse3.setxattr(mnt_dir, 'command', b'terminate') mount_process.join(5) assert mount_process.exitcode is not None except: cleanup(mount_process, mnt_dir) raise class Fs(pyfuse3.Operations): def __init__(self, cross_process): super(Fs, self).__init__() self.hello_name = b"message" self.hello_inode = cast(InodeT, pyfuse3.ROOT_INODE + 1) self.hello_data = b"hello world\n" self.status = cross_process self.lookup_cnt = 0 self.status.getattr_called = False self.status.lookup_called = False self.status.read_called = False self.status.entry_timeout = 99999 self.status.attr_timeout = 99999 async def getattr(self, inode: InodeT, ctx: RequestContext | None = None) -> EntryAttributes: entry = EntryAttributes() if inode == pyfuse3.ROOT_INODE: entry.st_mode = stat.S_IFDIR | 0o755 entry.st_size = 0 elif inode == self.hello_inode: entry.st_mode = stat.S_IFREG | 0o644 entry.st_size = len(self.hello_data) else: raise pyfuse3.FUSEError(errno.ENOENT) stamp = int(1438467123.985654 * 1e9) entry.st_atime_ns = stamp entry.st_ctime_ns = stamp entry.st_mtime_ns = stamp entry.st_gid = os.getgid() entry.st_uid = os.getuid() entry.st_ino = inode entry.entry_timeout = self.status.entry_timeout entry.attr_timeout = self.status.attr_timeout self.status.getattr_called = True return entry async def forget(self, inode_list): for inode, cnt in inode_list: if inode == self.hello_inode: self.lookup_cnt -= 1 assert self.lookup_cnt >= 0 else: assert inode == pyfuse3.ROOT_INODE async def lookup( self, parent_inode: InodeT, name: bytes, ctx: RequestContext ) -> EntryAttributes: if parent_inode != pyfuse3.ROOT_INODE or name != self.hello_name: raise pyfuse3.FUSEError(errno.ENOENT) self.lookup_cnt += 1 self.status.lookup_called = True return await self.getattr(self.hello_inode, ctx) async def opendir(self, inode, ctx): if inode != pyfuse3.ROOT_INODE: raise pyfuse3.FUSEError(errno.ENOENT) # For simplicity, we use the inode as file handle return FileHandleT(inode) async def readdir(self, fh: FileHandleT, start_id: int, token: ReaddirToken) -> None: assert fh == pyfuse3.ROOT_INODE if start_id == 0: pyfuse3.readdir_reply(token, self.hello_name, await self.getattr(self.hello_inode), 1) return async def open(self, inode, flags, ctx): if inode != self.hello_inode: raise pyfuse3.FUSEError(errno.ENOENT) if flags & os.O_RDWR or flags & os.O_WRONLY: raise pyfuse3.FUSEError(errno.EACCES) # For simplicity, we use the inode as file handle return FileInfo(fh=FileHandleT(inode)) async def read(self, fh, off, size): assert fh == self.hello_inode self.status.read_called = True return self.hello_data[off : off + size] async def setxattr(self, inode, name, value, ctx): if inode != pyfuse3.ROOT_INODE or name != b'command': raise FUSEError(errno.ENOTSUP) if value == b'forget_entry': pyfuse3.invalidate_entry_async(pyfuse3.ROOT_INODE, self.hello_name) # Make sure that the request is pending before we return await trio.sleep(0.1) elif value == b'forget_inode': pyfuse3.invalidate_inode(self.hello_inode) elif value == b'store': pyfuse3.notify_store(self.hello_inode, offset=0, data=self.hello_data) elif value == b'terminate': pyfuse3.terminate() else: raise FUSEError(errno.EINVAL) class PollTestFs(Fs): def __init__(self, cross_process): super().__init__(cross_process) self.poll_handle: PollHandle | None = None self.status.poll_called = False self.status.poll_handle_received = False self.status.poll_ready = False async def poll( self, inode: InodeT, fh: FileHandleT, poll_handle: PollHandle | None, ctx: RequestContext, ) -> int: assert inode == self.hello_inode assert fh == self.hello_inode self.status.poll_called = True if poll_handle is not None: self.poll_handle = poll_handle self.status.poll_handle_received = True if self.status.poll_ready: return select.POLLPRI return 0 async def setxattr(self, inode, name, value, ctx): if value != b"poll_ready": return await super().setxattr(inode, name, value, ctx) if inode != self.hello_inode or name != b"command": raise FUSEError(errno.ENOTSUP) self.status.poll_ready = True if self.poll_handle is None: raise FUSEError(errno.EINVAL) self.poll_handle.notify() self.poll_handle = None def run_fs(mountpoint, cross_process, fs_class=Fs): # Logging (note that we run in a new process, so we can't # rely on direct log capture and instead print to stdout) root_logger = logging.getLogger() formatter = logging.Formatter( '%(asctime)s.%(msecs)03d %(levelname)s %(funcName)s(%(threadName)s): %(message)s', datefmt="%M:%S", ) handler = logging.StreamHandler(sys.stdout) handler.setLevel(logging.DEBUG) handler.setFormatter(formatter) root_logger.addHandler(handler) root_logger.setLevel(logging.DEBUG) testfs = fs_class(cross_process) fuse_options = set(pyfuse3.default_options) fuse_options.add('fsname=pyfuse3_testfs') pyfuse3.init(testfs, mountpoint, fuse_options) try: trio.run(pyfuse3.main) finally: pyfuse3.close() ================================================ FILE: test/test_rounding.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' test_rounding.py - Unit tests for pyfuse3. Copyright © 2020 Philip Warner This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' if __name__ == '__main__': import sys import pytest sys.exit(pytest.main([__file__] + sys.argv[1:])) from pyfuse3 import _NANOS_PER_SEC, EntryAttributes def test_rounding(): # Incorrect division previously resulted in rounding errors for # all dates. entry = EntryAttributes() # Approximately 67 years, ending in 999. # Note: 67 years were chosen to avoid y2038 issues (1970 + 67 = 2037). # Testing these is **not** in scope of this test. secs = 67 * 365 * 24 * 3600 + 999 nanos = _NANOS_PER_SEC - 1 total = secs * _NANOS_PER_SEC + nanos entry.st_atime_ns = total entry.st_ctime_ns = total entry.st_mtime_ns = total # Birthtime skipped -- only valid under BSD and OSX # entry.st_birthtime_ns = total assert entry.st_atime_ns == total assert entry.st_ctime_ns == total assert entry.st_mtime_ns == total # Birthtime skipped -- only valid under BSD and OSX # assert entry.st_birthtime_ns == total ================================================ FILE: test/util.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- ''' util.py - Utility functions for pyfuse3 unit tests. Copyright © 2015 Nikolaus Rath This file is part of pyfuse3. This work may be distributed under the terms of the GNU LGPL. ''' import multiprocessing import os import platform import shutil import stat import subprocess import time from collections.abc import Callable from multiprocessing.context import ForkProcess from typing import TypeVar import pytest Process = subprocess.Popen | multiprocessing.Process | ForkProcess def fuse_test_marker(): '''Return a pytest.marker that indicates FUSE availability If system/user/environment does not support FUSE, return a `pytest.mark.skip` object with more details. If FUSE is supported, return `pytest.mark.uses_fuse()`. ''' if platform.system() == 'Darwin': # No working autodetection, just assume it will work. return skip = lambda x: pytest.mark.skip(reason=x) fusermount_path = shutil.which('fusermount') if fusermount_path is None: return skip("Can't find fusermount executable") if not os.path.exists('/dev/fuse'): return skip("FUSE kernel module does not seem to be loaded") if os.getuid() == 0: return pytest.mark.uses_fuse() mode = os.stat(fusermount_path).st_mode if mode & stat.S_ISUID == 0: return skip('fusermount executable not setuid, and we are not root.') try: fd = os.open('/dev/fuse', os.O_RDWR) except OSError as exc: return skip('Unable to open /dev/fuse: %s' % exc.strerror) else: os.close(fd) return pytest.mark.uses_fuse() def exitcode(process: Process) -> int | None: if isinstance(process, subprocess.Popen): return process.poll() else: if process.is_alive(): return None else: return process.exitcode T = TypeVar('T') def wait_for(callable: Callable[[], T], timeout: float = 10, interval: float = 0.1) -> T | None: '''Wait until *callable* returns something True and return it If *timeout* expires, return None ''' waited = 0.0 while True: ret = callable() if ret: return ret if waited > timeout: return None waited += interval time.sleep(interval) def wait_for_mount(mount_process: Process, mnt_dir: str) -> bool: elapsed = 0.0 while elapsed < 30: if os.path.ismount(mnt_dir): return True if exitcode(mount_process) is not None: pytest.fail('file system process terminated prematurely') time.sleep(0.1) elapsed += 0.1 pytest.fail("mountpoint failed to come up") def cleanup(mount_process: Process, mnt_dir: str) -> None: if platform.system() == 'Darwin': subprocess.call( ['umount', '-l', mnt_dir], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT ) else: subprocess.call( ['fusermount', '-z', '-u', mnt_dir], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT ) mount_process.terminate() if isinstance(mount_process, subprocess.Popen): try: mount_process.wait(1) except subprocess.TimeoutExpired: mount_process.kill() else: mount_process.join(5) if mount_process.exitcode is None: mount_process.kill() def umount(mount_process: Process, mnt_dir: str) -> None: if platform.system() == 'Darwin': subprocess.check_call(['umount', '-l', mnt_dir]) else: subprocess.check_call(['fusermount', '-z', '-u', mnt_dir]) assert not os.path.ismount(mnt_dir) if isinstance(mount_process, subprocess.Popen): try: code = mount_process.wait(5) if code == 0: return pytest.fail('file system process terminated with code %s' % (code,)) except subprocess.TimeoutExpired: mount_process.terminate() try: mount_process.wait(1) except subprocess.TimeoutExpired: mount_process.kill() else: mount_process.join(5) if mount_process.exitcode == 0: return elif mount_process.exitcode is None: mount_process.terminate() mount_process.join(1) else: pytest.fail('file system process terminated with code %s' % (mount_process.exitcode,)) pytest.fail('mount process did not terminate') ================================================ FILE: util/build_backend.py ================================================ """ Custom build backend for pyfuse3. This wraps setuptools.build_meta and dynamically configures the Cython extension based on pkg-config output and platform detection. """ import os import subprocess from setuptools import Extension from setuptools.build_meta import * # noqa: F403 def pkg_config(pkg, cflags=True, ldflags=False, min_ver=None): """Frontend to pkg-config""" if min_ver: cmd = ['pkg-config', pkg, '--atleast-version', min_ver] if subprocess.call(cmd) != 0: cmd = ['pkg-config', '--modversion', pkg] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE) version = proc.communicate()[0].strip() if not version: raise SystemExit(2) # pkg-config generates error message already else: raise SystemExit( '%s version too old (found: %s, required: %s)' % (pkg, version, min_ver) ) cmd = ['pkg-config', pkg] if cflags: cmd.append('--cflags') if ldflags: cmd.append('--libs') proc = subprocess.Popen(cmd, stdout=subprocess.PIPE) cflags = proc.stdout.readline().rstrip() proc.stdout.close() if proc.wait() != 0: raise SystemExit(2) # pkg-config generates error message already return cflags.decode('us-ascii').split() def get_extension_modules(): """Build the Cython extension with platform-specific configuration.""" # Get fuse3 flags from pkg-config compile_args = pkg_config('fuse3', cflags=True, ldflags=False, min_ver='3.2.0') compile_args += [ '-DFUSE_USE_VERSION=32', '-Wall', '-Wextra', '-Wconversion', '-Wsign-compare', '-Wno-unused-function', '-Wno-implicit-fallthrough', '-Wno-unused-parameter', ] link_args = pkg_config('fuse3', cflags=False, ldflags=True, min_ver='3.2.0') link_args.append('-lpthread') # Determine source files based on platform c_sources = ['src/pyfuse3/__init__.pyx'] if os.uname()[0] in ('Linux', 'GNU/kFreeBSD'): link_args.append('-lrt') elif os.uname()[0] == 'Darwin': c_sources.append('src/pyfuse3/darwin_compat.c') return [ Extension( 'pyfuse3.__init__', c_sources, extra_compile_args=compile_args, extra_link_args=link_args, include_dirs=['Include'], ) ] # Override get_requires_for_build_wheel to ensure we have pkg-config available def get_requires_for_build_wheel(config_settings=None): """Return build requirements.""" from setuptools.build_meta import get_requires_for_build_wheel as orig return orig(config_settings) # Hook into the build process _orig_build_wheel = build_wheel # noqa: F405 _orig_build_editable = build_editable if 'build_editable' in dir() else None # noqa: F405 _orig_build_sdist = build_sdist # noqa: F405 def build_wheel(wheel_directory, config_settings=None, metadata_directory=None): """Build wheel with dynamic extension configuration.""" # Inject our extension modules into the distribution from setuptools import Distribution # Monkey-patch Distribution to include our extensions orig_init = Distribution.__init__ def patched_init(self, attrs=None): if attrs is None: attrs = {} # Add our dynamically configured extension modules if 'ext_modules' not in attrs: attrs['ext_modules'] = get_extension_modules() orig_init(self, attrs) Distribution.__init__ = patched_init try: return _orig_build_wheel(wheel_directory, config_settings, metadata_directory) finally: Distribution.__init__ = orig_init def build_editable(wheel_directory, config_settings=None, metadata_directory=None): """Build editable wheel with dynamic extension configuration.""" if _orig_build_editable is None: raise NotImplementedError("build_editable not available") from setuptools import Distribution # Monkey-patch Distribution to include our extensions orig_init = Distribution.__init__ def patched_init(self, attrs=None): if attrs is None: attrs = {} # Add our dynamically configured extension modules if 'ext_modules' not in attrs: attrs['ext_modules'] = get_extension_modules() orig_init(self, attrs) Distribution.__init__ = patched_init try: return _orig_build_editable(wheel_directory, config_settings, metadata_directory) finally: Distribution.__init__ = orig_init def build_sdist(sdist_directory, config_settings=None): """Build source distribution.""" # For sdist, we don't need to configure extensions return _orig_build_sdist(sdist_directory, config_settings) ================================================ FILE: util/sdist-sign ================================================ #!/bin/bash R=$1 if [ "$R" = "" ]; then echo "Usage: sdist-sign 1.2.3" exit fi if [ "$QUBES_GPG_DOMAIN" = "" ]; then GPG=gpg else GPG=qubes-gpg-client-wrapper fi python setup.py sdist D=dist/pyfuse3-$R.tar.gz $GPG --detach-sign --local-user "Thomas Waldmann" --armor --output $D.asc $D ================================================ FILE: util/upload-pypi ================================================ #!/bin/bash R=$1 if [ "$R" = "" ]; then echo "Usage: upload-pypi 1.2.3 [test]" exit fi if [ "$2" = "test" ]; then export TWINE_REPOSITORY=testpyfuse3 else export TWINE_REPOSITORY=pyfuse3 fi D=dist/pyfuse3-$R.tar.gz twine upload $D