Repository: jessegrosjean/jwalk
Branch: main
Commit: 891125eec76b
Files: 50
Total size: 156.6 KB
Directory structure:
gitextract_cyuqkkuz/
├── .github/
│ └── workflows/
│ └── ci.yml
├── .gitignore
├── CHANGELOG.md
├── Cargo.toml
├── LICENSE
├── README.md
├── benches/
│ ├── benchmarks.md
│ └── walk_benchmark.rs
├── examples/
│ ├── dc.rs
│ ├── du.rs
│ └── shared/
│ └── mod.rs
├── src/
│ ├── core/
│ │ ├── dir_entry.rs
│ │ ├── dir_entry_iter.rs
│ │ ├── error.rs
│ │ ├── index_path.rs
│ │ ├── mod.rs
│ │ ├── ordered.rs
│ │ ├── ordered_queue.rs
│ │ ├── read_children.rs
│ │ ├── read_dir.rs
│ │ ├── read_dir_iter.rs
│ │ ├── read_dir_spec.rs
│ │ └── run_context.rs
│ └── lib.rs
└── tests/
├── assets/
│ └── test_dir/
│ ├── a.txt
│ ├── b.txt
│ ├── c.txt
│ ├── group 1/
│ │ └── d.txt
│ └── group 2/
│ ├── .hidden_file.txt
│ └── e.txt
├── detect_deadlock.rs
├── fixtures/
│ ├── make-basic.sh
│ ├── make-depth-test-3.sh
│ ├── make-depth-test.sh
│ ├── make-many-dirs.sh
│ ├── make-many-files.sh
│ ├── make-many-mixed.sh
│ ├── make-nested.sh
│ ├── make-one-dir.sh
│ ├── make-one-file.sh
│ ├── make-siblings.sh
│ ├── make-sort-test.sh
│ ├── make-symlink-loop.sh
│ ├── make-symlink-root-dir.sh
│ ├── make-symlink-root-file.sh
│ ├── make-symlink-self-loop.sh
│ └── make-symlinks.sh
├── integration.rs
├── relative_paths.rs
└── util/
└── mod.rs
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/ci.yml
================================================
name: Rust
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: Swatinem/rust-cache@v2
- name: tests
run: cargo test
- name: docs
run: cargo doc
- name: bench
run: JWALK_BENCHMARK_DIR=~/ cargo bench
================================================
FILE: .gitignore
================================================
# Rust build artifacts
/target
# Benchmark assets (large files, regenerate with benchmark setup)
/benches/assets/linux_checkout
# Lock file (not committed for libraries)
Cargo.lock
# Generated test fixtures - archives and extracted platform-specific contents
tests/fixtures/generated-archives/
tests/fixtures/generated-do-not-edit/
================================================
FILE: CHANGELOG.md
================================================
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## 0.8.1 (2022-12-15)
### New Features
- <csr-id-b49e157b539150a44b761e43d8b09621367e760c/> re-export `rayon` in the crate root.
This makes creating a `ThreadPool` easier as it doesn't force us to
maintain our own `rayon` dependency.
### Commit Statistics
<csr-read-only-do-not-edit/>
- 1 commit contributed to the release.
- 1 commit was understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- re-export `rayon` in the crate root. ([`b49e157`](https://github.com/Byron/jwalk/commit/b49e157b539150a44b761e43d8b09621367e760c))
</details>
## 0.8.0 (2022-12-15)
### New Features (BREAKING)
- <csr-id-3a717219411a7478b90c4d694d57e28d8941dde1/> `Parallelism::RayonExistingPool::busy_timeout` is now optional.
That way we can indicate that no waiting should be done as we know the
given threadpool has enough resources.
### Commit Statistics
<csr-read-only-do-not-edit/>
- 2 commits contributed to the release.
- 1 commit was understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Release jwalk v0.8.0 ([`be0bd21`](https://github.com/Byron/jwalk/commit/be0bd21bd5213033ac55b90ec7753d6e72b4bd84))
- `Parallelism::RayonExistingPool::busy_timeout` is now optional. ([`3a71721`](https://github.com/Byron/jwalk/commit/3a717219411a7478b90c4d694d57e28d8941dde1))
</details>
## 0.7.0 (2022-12-15)
This release makes iterator creation fallible to avoid potential hangs when there is no available thread to process
any of the iterator work.
### New Features
- <csr-id-7d5b8b870bfca2b1b68de1427fbdc0ec1a1bff2b/> `WalkDirGeneric::try_into_iter()` for early error handling.
If we can't instantiate the iterator due to a busy thread-pool,
we can now abort early instead of yielding a fake-entry just to
show an error occurred. This is the preferred way to instantiate
a `jwalk` iterator.
### New Features (BREAKING)
- <csr-id-3bf1bc226571869e4a5c357d4f6e40ad0a28f3ff/> Detect possible deadlocks when instantiating a parallel iterator.
Deadlocks can happen if the producer for results doesn't start as there
is no free thread on the rayon pool, and the only way for it to become free
is if the iterator produces results.
We now offer a `busy_timeout` in the relevant variants of the
`Parallelism` enumeration to allow controlling how long we will wait
until we abort with an error.
### Commit Statistics
<csr-read-only-do-not-edit/>
- 7 commits contributed to the release.
- 1 day passed between releases.
- 2 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Thanks Clippy
<csr-read-only-do-not-edit/>
[Clippy](https://github.com/rust-lang/rust-clippy) helped 1 time to make code idiomatic.
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Release jwalk v0.7.0 ([`c265744`](https://github.com/Byron/jwalk/commit/c265744d74b3ea231eacee6332a99ee292f3018a))
- prepare changelog prior to release ([`67364f9`](https://github.com/Byron/jwalk/commit/67364f910f1ba1ffeb5c30d0f75709431b212e2b))
- refactor ([`a94d14b`](https://github.com/Byron/jwalk/commit/a94d14b34980e5dd53ea6dde9c5676f44c80a7fa))
- thanks clippy ([`7e300c6`](https://github.com/Byron/jwalk/commit/7e300c68691f462ebb0848db915d7798b49dfccc))
- `WalkDirGeneric::try_into_iter()` for early error handling. ([`7d5b8b8`](https://github.com/Byron/jwalk/commit/7d5b8b870bfca2b1b68de1427fbdc0ec1a1bff2b))
- Detect possible deadlocks when instantiating a parallel iterator. ([`3bf1bc2`](https://github.com/Byron/jwalk/commit/3bf1bc226571869e4a5c357d4f6e40ad0a28f3ff))
- fix various IDE warnings ([`cc0009f`](https://github.com/Byron/jwalk/commit/cc0009f626ac86f92e20ce3d4e7c2a8f00d979a0))
</details>
## 0.6.2 (2022-12-13)
### Bug Fixes
- <csr-id-bd3e88017ea29c3b89b518f3a721ba35577b7666/> stalling issue when threadpool is used is no more.
The issue seems to have been that `install` blocks whereas `spawn`
properly releases the main thread.
This seems to have been changed subtly due to changes in `rayon`,
which breaks an assumption on how the code is executed.
Replacing `install()` with `spawn` calls resolved the issue.
### Commit Statistics
<csr-read-only-do-not-edit/>
- 5 commits contributed to the release.
- 1 commit was understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Release jwalk v0.6.2 ([`2d1b2fb`](https://github.com/Byron/jwalk/commit/2d1b2fbe59a0ebb0413b54a8b8b0bba713e4d0e3))
- Merge branch 'stalling-issue' ([`7bd2f35`](https://github.com/Byron/jwalk/commit/7bd2f35d4fd106edbafa187ef4481333bb60da7d))
- stalling issue when threadpool is used is no more. ([`bd3e880`](https://github.com/Byron/jwalk/commit/bd3e88017ea29c3b89b518f3a721ba35577b7666))
- refactor ([`1032308`](https://github.com/Byron/jwalk/commit/10323089dbf00e01a0280a35f826ca269b6eeea6))
- print each path seen during iteration ([`5e83ad5`](https://github.com/Byron/jwalk/commit/5e83ad5f09852a6449f63b0c954eec81413de1c2))
</details>
## 0.6.1 (2022-12-13)
The first release under new ownership with no user-facing changes.
- The project uses GitHub CI and `cargo smart-release` for releases.
- some code cleanup based on `cargo clippy`.
### Changing project ownership to @Byron
- Thanks and good luck! (By https://github.com/jessegrosjean)
- Thank you, my pleasure (By https://github.com/Byron)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 30 commits contributed to the release over the course of 705 calendar days.
- 705 days passed between releases.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Thanks Clippy
<csr-read-only-do-not-edit/>
[Clippy](https://github.com/rust-lang/rust-clippy) helped 1 time to make code idiomatic.
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Release jwalk v0.6.1 ([`6a2781c`](https://github.com/Byron/jwalk/commit/6a2781c6211a6db777c08bcdeb60f0317c00bc3e))
- prepare changelog prior to release ([`c772967`](https://github.com/Byron/jwalk/commit/c77296707df392193dc47bcd1f465fa813215b82))
- another round of link adjustments ([`7c12dc3`](https://github.com/Byron/jwalk/commit/7c12dc333d5086ed41228ee410653def9ff5adf7))
- thanks clippy ([`51e2b0d`](https://github.com/Byron/jwalk/commit/51e2b0d0330b264972422d44ca25affcced981d5))
- run benchmarks on CI ([`cc0fd74`](https://github.com/Byron/jwalk/commit/cc0fd74d439fb24b063d3d6d4d05f3370d41bf65))
- set version back to what's current, change URLs and add myself as author ([`aa5b24d`](https://github.com/Byron/jwalk/commit/aa5b24dff23b4fd3b48e6a3e4c59a504a380beda))
- cleanup tests ([`d8f0756`](https://github.com/Byron/jwalk/commit/d8f07566eb2487f6f73bd06edc8286d0bf3ee015))
- rename 'master' to 'main' ([`362d03c`](https://github.com/Byron/jwalk/commit/362d03c1059658f88a77b7b9db7d14a2ccc2e6b5))
- enable CI ([`a115e7a`](https://github.com/Byron/jwalk/commit/a115e7acbf6c147c2bfeba3981c6bc5d587c2aef))
- Moved to example/crash ([`1a09da5`](https://github.com/Byron/jwalk/commit/1a09da59cda994758a856c30f931afe4ee0208d8))
- Changing project ownership ([`cd5d1ae`](https://github.com/Byron/jwalk/commit/cd5d1aed268ad5a0692f698a419bdea835aa71a5))
- Add crash example ([`c0b262b`](https://github.com/Byron/jwalk/commit/c0b262bc4b4bbb0134c7987cbb6995a46a005070))
- More unneeded code removal ([`69c00ec`](https://github.com/Byron/jwalk/commit/69c00ec6f60b0010c4a99db7ea67093f401d8260))
- Added failing combine with rayon test ([`5eccf5e`](https://github.com/Byron/jwalk/commit/5eccf5efe09c1a24e839157b6098a9bc1c8948cc))
- Remove some unneeded rayon calls ([`f61a535`](https://github.com/Byron/jwalk/commit/f61a53585514f50f772a4cb81c57150ac1fd4450))
- Update to rayon 1.6.1, remove jwalk_par_bridge ([`94c0385`](https://github.com/Byron/jwalk/commit/94c0385959c4004aff3ec2c60e729fa654579141))
- Merge pull request #33 from bootandy/patch-2 ([`9575132`](https://github.com/Byron/jwalk/commit/9575132b76513bea64405a7cc5a98708d31d5743))
- Fix typo ([`9ae51a5`](https://github.com/Byron/jwalk/commit/9ae51a5823a2c530871b2ba7fbce5096a8d37339))
- Merge pull request #32 from Byron/master ([`5c857d4`](https://github.com/Byron/jwalk/commit/5c857d4e7fa2d587617e442d7a81e070c2c55175))
- Don't ignore hidden files in `du` example ([`0786beb`](https://github.com/Byron/jwalk/commit/0786bebf3962e862c56577da389d9b14dfb3b5f1))
- Remove unused imports in example ([`80a6d2e`](https://github.com/Byron/jwalk/commit/80a6d2e3054e84b36ae6c45791b5b62d579dbea7))
- Update readme ([`6f9ebf5`](https://github.com/Byron/jwalk/commit/6f9ebf54dcfcc561c1e0afe0b797fcef8dc65b51))
- Update to from latest rayon src/iter/par_bridge.rs ([`e3a46c1`](https://github.com/Byron/jwalk/commit/e3a46c1b02111725b7d3c7929a6b34079692f154))
- Add simple du example ([`ae905c6`](https://github.com/Byron/jwalk/commit/ae905c60762c72aab9230717c8ae7fd6e9fcf720))
- Rename new bench to "rayon" ([`5ee29f5`](https://github.com/Byron/jwalk/commit/5ee29f5755f7ca21c2f7d2d07b54d195c08acc72))
- Merge pull request #31 from Byron/master ([`4817c1a`](https://github.com/Byron/jwalk/commit/4817c1a6817f659f175bce271701bfbbad5b233b))
- reduce the recursion to its core, keep no state ([`247cf38`](https://github.com/Byron/jwalk/commit/247cf387210231ae699c5a247a024fb7724846ec))
- Support for file metadata to approximate typical tool usage ([`a6afcfe`](https://github.com/Byron/jwalk/commit/a6afcfe535eee18fc9ce216a6facee35b58d6ed1))
- Add simple recursive way of building a file tree for reference ([`9c66ef6`](https://github.com/Byron/jwalk/commit/9c66ef6009c47283b43b44cf08eefbdc25765737))
- (cargo-release) start next development iteration 0.6.0 ([`a1d5209`](https://github.com/Byron/jwalk/commit/a1d5209c32723e81ed7c76e0c0ee8e3de2a07748))
</details>
## v0.6.0 (2021-01-06)
Added depth and path being read to params to ProcessReadDirFunction callback.
Allow setting initial root_read_dir_state (ReadDirState) instead of always
getting ::default() value.
### Commit Statistics
<csr-read-only-do-not-edit/>
- 2 commits contributed to the release over the course of 9 calendar days.
- 9 days passed between releases.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Change release to 0.6 because of breaking changes ([`d74cc13`](https://github.com/Byron/jwalk/commit/d74cc130c506d5b2744bb9cfda8078ac2da0208f))
- (cargo-release) start next development iteration 0.5.2 ([`6e4ba03`](https://github.com/Byron/jwalk/commit/6e4ba039756cccff449a317576705c4979bb8fbc))
</details>
## v0.5.2 (2020-12-28)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 12 commits contributed to the release over the course of 289 calendar days.
- 289 days passed between releases.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.5.2 ([`85ef009`](https://github.com/Byron/jwalk/commit/85ef009dfe12ee23c1dacda54955281a441ec97a))
- update dependencies ([`ea66ef8`](https://github.com/Byron/jwalk/commit/ea66ef8b8a957566d994d59048be285a23f29462))
- Add more capability to ProcessReadDirFunction ([`2459776`](https://github.com/Byron/jwalk/commit/24597762a6bec16dcc5f421bdd0f3dee960f68cd))
- Add test processing jwalk entries with rayon par_bridge() ([`57860ff`](https://github.com/Byron/jwalk/commit/57860ff3cfd69dc172d35666d4986055b5ba2e05))
- cargo fmt ([`a431561`](https://github.com/Byron/jwalk/commit/a431561f11d59defcf0eed3707b46683d1f89655))
- Merge pull request #25 from brmmm3/fix_warnings ([`fe12f26`](https://github.com/Byron/jwalk/commit/fe12f2666db66cc3e056267438b7b7b9f946f3b7))
- Fix warnings ([`64ddb05`](https://github.com/Byron/jwalk/commit/64ddb05cbc130b865cc872d1e56c4ef71ad1a9bd))
- Merge branch 'master' of https://github.com/jessegrosjean/walk ([`32e46c6`](https://github.com/Byron/jwalk/commit/32e46c68fb3933a2fd6fc9a329b7a72305f5eb64))
- Note preload_metadata removal ([`3bd6618`](https://github.com/Byron/jwalk/commit/3bd6618d2a2dc689837abf97c1c9fc1f86475164))
- Merge pull request #23 from bootandy/patch-1 ([`b4776b9`](https://github.com/Byron/jwalk/commit/b4776b9c4d1ce6f194f542d92a795653c7960408))
- fix typo ([`d630f80`](https://github.com/Byron/jwalk/commit/d630f80335edcec2abdd6b0fb6a525565b9adda9))
- (cargo-release) start next development iteration 0.5.1 ([`6d93359`](https://github.com/Byron/jwalk/commit/6d93359836bd7c4e1ddefa4462eb8083f45a82b1))
</details>
## v0.5.1 (2020-03-13)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 5 commits contributed to the release.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.5.1 ([`1621bee`](https://github.com/Byron/jwalk/commit/1621bee2077484836558f41ec6e90be40643dcd6))
- More tests for relative paths ([`246c997`](https://github.com/Byron/jwalk/commit/246c99701e03a671274b2a17cd3660f2388fc9c4))
- Use path for root dir_entry.file_name if path has now filename of own ([`80f6f7a`](https://github.com/Byron/jwalk/commit/80f6f7ae7699651ec4684510eb70e4289e0ac28c))
- Merge pull request #19 from brmmm3/simplify_and_then ([`ff234bb`](https://github.com/Byron/jwalk/commit/ff234bbbcf8999585e6c3d7e4ba8c2b44f1b8d17))
- (cargo-release) start next development iteration 0.5.0 ([`5e97e11`](https://github.com/Byron/jwalk/commit/5e97e1159843b3ce39617a0a3ca2fea3e535e629))
</details>
## v0.5.0 (2020-03-13)
<csr-id-11fa0bc9e8541af333aafb41cc89218435474df4/>
First major change is that API and behavior are now closer to [`walkdir`] and
jwalk now runs the majority of `walkdir`s tests.
Second major change is the walk can now be parameterized with a client state
type. This state can be manipulated from the `process_read_dir` callback and
then is passed down when reading descendens with the `process_read_dir`
callback.
Part of this second change is that `preload_metadata` option is removed. That
means `DirEntry.metadata()` is never a cached value. Instead you want to read
metadata you should do it in the `process_entries` callback and store whatever
values you need as `client_state`. See this [benchmark] as an example.
### Chore
- <csr-id-11fa0bc9e8541af333aafb41cc89218435474df4/> Update criterion to 0.3
### Commit Statistics
<csr-read-only-do-not-edit/>
- 26 commits contributed to the release over the course of 294 calendar days.
- 294 days passed between releases.
- 1 commit was understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.5.0 ([`cf9d248`](https://github.com/Byron/jwalk/commit/cf9d248d13932b4afa531f704f8f3eb8e2d55ec3))
- Merge pull request #21 from jessegrosjean/symlinks ([`6410407`](https://github.com/Byron/jwalk/commit/641040798c71ab68fdba8874b6221f8e9d6295d7))
- Update dependencies ([`85dff6e`](https://github.com/Byron/jwalk/commit/85dff6e02f5c52b887b429d12411f9faf1db2a5c))
- Merge branch 'master' of https://github.com/jessegrosjean/walk into symlinks ([`0dfefa9`](https://github.com/Byron/jwalk/commit/0dfefa9e0e068a6f1902ffe0da5c99511bb88792))
- Get follow links working and passing tests ([`797f76f`](https://github.com/Byron/jwalk/commit/797f76f8c2cfcaf8b02c0891794d9310e2bf30f6))
- Clean ([`ff8f491`](https://github.com/Byron/jwalk/commit/ff8f491a1945ef2a148adffe9bb999b49ca2dc73))
- Merge pull request #18 from brmmm3/remove_unnecessary_clone ([`898ff91`](https://github.com/Byron/jwalk/commit/898ff914ae0c9d6f1f49e2fbe64b21742a93b524))
- Simplify and_then ([`b330914`](https://github.com/Byron/jwalk/commit/b3309145eee88c8b48daf1453d68e5d6d98d3ce7))
- Remove unnecessary clone. ([`0f8b1c8`](https://github.com/Byron/jwalk/commit/0f8b1c8350c4d31e6de88fddbc7fd61ecd990065))
- Merge pull request #12 from ignatenkobrain/patch-1 ([`f9a144b`](https://github.com/Byron/jwalk/commit/f9a144bd504027d44d0fd86c1aeef077ce3e543e))
- Update criterion to 0.3 ([`11fa0bc`](https://github.com/Byron/jwalk/commit/11fa0bc9e8541af333aafb41cc89218435474df4))
- Make Parallelism param actually have effect. Fix some related bugs. ([`3b90d8e`](https://github.com/Byron/jwalk/commit/3b90d8e40ba39d378077bb2f20da789af6addda0))
- use walkdir error struct ([`850954f`](https://github.com/Byron/jwalk/commit/850954f6b3756bbca66e2bccfde9f5f9297f2bb3))
- test/benches back to working ([`45529ef`](https://github.com/Byron/jwalk/commit/45529efd5496a4ab1694aeb5a82d8eab0f683e36))
- in progress more closesly follow walkdir ([`66d46ac`](https://github.com/Byron/jwalk/commit/66d46acfb2a375e30d1a904e1ab17a9b5855f31e))
- symlink work in progress ([`7ee0cb3`](https://github.com/Byron/jwalk/commit/7ee0cb3e72eb074f9081b5fd3cbc475e61fb9a43))
- Merge branch 'master' of https://github.com/jessegrosjean/walk ([`17ddb91`](https://github.com/Byron/jwalk/commit/17ddb917a12d3e9787b6cbb2ace28758a712a95e))
- Walk is now parameterized with client_state type ([`4e4218f`](https://github.com/Byron/jwalk/commit/4e4218f7dcd187ace10bd793515822976dd0259c))
- Merge pull request #8 from vks/cleanup ([`1074566`](https://github.com/Byron/jwalk/commit/10745666a4c6620337d8d1c1c099ddc2a90d02c8))
- Fix compiler warnings ([`064ee60`](https://github.com/Byron/jwalk/commit/064ee604277c94b38ea9768447a88070fcb641d8))
- Remove Cargo.lock ([`8683484`](https://github.com/Byron/jwalk/commit/86834849ce473fd2a910caf5d890c3c6067dcac8))
- fix table formatting ([`0ef72d1`](https://github.com/Byron/jwalk/commit/0ef72d1838e732771cafcd05ef04a70280628017))
- Update benchmarks ([`fd9af20`](https://github.com/Byron/jwalk/commit/fd9af20e6e523449f07f3810fd8eb2987812ec13))
- Merge pull request #5 from spacekookie/patch-1 ([`c10dbaa`](https://github.com/Byron/jwalk/commit/c10dbaaaf2130de162c163c86f6d23bb1ace506a))
- Reformatting benchmarks table ([`99678be`](https://github.com/Byron/jwalk/commit/99678be587f1da6b869fe5afdd95c3a14b7666ef))
- (cargo-release) start next development iteration 0.4.0 ([`29b6b1e`](https://github.com/Byron/jwalk/commit/29b6b1ecedc85a2baa01fddee3b6e75dbc230b54))
</details>
## v0.4.0 (2019-05-24)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 3 commits contributed to the release over the course of 91 calendar days.
- 91 days passed between releases.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.4.0 ([`5d68189`](https://github.com/Byron/jwalk/commit/5d681896e37c7518c9c31690467412f49fa3a418))
- Added content spec error reporting for root DirEntry ([`9643ce0`](https://github.com/Byron/jwalk/commit/9643ce091daa437b7dce082a25664688bb1ada90))
- (cargo-release) start next development iteration 0.3.0 ([`956b1a5`](https://github.com/Byron/jwalk/commit/956b1a5d615a7dda34238626bc5097afc4be0d12))
</details>
## v0.3.0 (2019-02-21)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 8 commits contributed to the release over the course of 9 calendar days.
- 9 days passed between releases.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.3.0 ([`09ebe45`](https://github.com/Byron/jwalk/commit/09ebe4508697551cf411664047e69220b8bd7ddb))
- Update dependencies ([`f47607d`](https://github.com/Byron/jwalk/commit/f47607d959e497b2577d5b7ceb1b911405ba07a1))
- Spelling ([`b8e9aea`](https://github.com/Byron/jwalk/commit/b8e9aea52d1e3e0f643b42cf17d80ac467f775bf))
- Fix bug when max_depth was set to 0 ([`29c035e`](https://github.com/Byron/jwalk/commit/29c035e70990e9a442bb4f79735f1bd406688c03))
- Revert "Simplify, stop tracking depth in read dir specs." ([`331a896`](https://github.com/Byron/jwalk/commit/331a8964286b121f5bfe55140d2f4d7f0d08e28f))
- Simplify, stop tracking depth in read dir specs. ([`c1bffdd`](https://github.com/Byron/jwalk/commit/c1bffdd2bc56ddec23bac58f93c8bc52d8150015))
- More badges! ([`e0d3e3e`](https://github.com/Byron/jwalk/commit/e0d3e3e67bcd2602433a84cb8e729f093cf38637))
- (cargo-release) start next development iteration 0.2.1 ([`133d168`](https://github.com/Byron/jwalk/commit/133d168a4586313a4e5eceb2ea70c348f387a5b4))
</details>
## v0.2.1 (2019-02-11)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 4 commits contributed to the release.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- (cargo-release) version 0.2.1 ([`1dee5e1`](https://github.com/Byron/jwalk/commit/1dee5e12ec7f07d2d177e116d3eceac0ec5e2bbf))
- Fix usage documentation. Other documentation updates. ([`2631dbd`](https://github.com/Byron/jwalk/commit/2631dbd01c92eaa50ca877c819151ef129fcb6ef))
- Update usage ([`04f897f`](https://github.com/Byron/jwalk/commit/04f897f2be26b576c06af2ecb327ecb4c8ddfcda))
- (cargo-release) start next development iteration 0.2.0 ([`da74631`](https://github.com/Byron/jwalk/commit/da746315a12af5a9bbe4ece4897303f938d7d469))
</details>
## v0.2.0 (2019-02-11)
### Commit Statistics
<csr-read-only-do-not-edit/>
- 37 commits contributed to the release over the course of 13 calendar days.
- 0 commits were understood as [conventional](https://www.conventionalcommits.org).
- 0 issues like '(#ID)' were seen in commit messages
### Commit Details
<csr-read-only-do-not-edit/>
<details><summary>view details</summary>
* **Uncategorized**
- Expose DirEntry fields for easy destructure. ([`ad9404e`](https://github.com/Byron/jwalk/commit/ad9404e19e78b2714b2db3da4ab85a91a8dbc481))
- Clippy suggestions and more doc updates ([`8fd5ae1`](https://github.com/Byron/jwalk/commit/8fd5ae172783d0e2ce3a696540e12d2576e4db9c))
- Rename "children" to content. Simplify sort to boolean. ([`f432c4f`](https://github.com/Byron/jwalk/commit/f432c4f806b01cfcaa16f4573a1d89d6817ab0ca))
- More docs for ReadDirSpec ([`347bc92`](https://github.com/Byron/jwalk/commit/347bc92cec8181b6a25cc80a663345cd3d87a3bb))
- More docs on how DirEntry is implemented ([`0bb13ee`](https://github.com/Byron/jwalk/commit/0bb13eecfa2f3f5b6448b5a7c4701add75d857cf))
- Fixing too many keywords! ([`ff5fba1`](https://github.com/Byron/jwalk/commit/ff5fba1ebe53d0240d64fb6f5e16e0dc1a11d2f8))
- More readme tweaks ([`0b23848`](https://github.com/Byron/jwalk/commit/0b238487c71a600dad31f05d57ca7f49d337d40d))
- Shorter readme page. ([`0ee3b3c`](https://github.com/Byron/jwalk/commit/0ee3b3c58af31b95099d79446dc67de88967115c))
- More doc updates ([`b3ac65c`](https://github.com/Byron/jwalk/commit/b3ac65cdca26fc1ca9e936d35256df517025fbca))
- Remove unused dependency ([`301e2a5`](https://github.com/Byron/jwalk/commit/301e2a5c053675dee50960dbea319ebcea901f28))
- Add badge ([`ab07f7a`](https://github.com/Byron/jwalk/commit/ab07f7ae7890a4fc18f509d9bea30f6a73e78f78))
- Create .travis.yml ([`66828d7`](https://github.com/Byron/jwalk/commit/66828d74f5cb3119bda9d99a7993ad8373bb5166))
- Add usage ([`58c7707`](https://github.com/Byron/jwalk/commit/58c770758cfd9f2076893b463490ef2e94dd3659))
- split code up into more files ([`566da8f`](https://github.com/Byron/jwalk/commit/566da8f0ac14f470dc181471e3c98a940ea96641))
- More docs cleanup ([`ade977d`](https://github.com/Byron/jwalk/commit/ade977dc29c59850b8ceb6f00089080c8e3cee18))
- Fix readme headings ([`738d9f6`](https://github.com/Byron/jwalk/commit/738d9f616005f22285985134258e49b800ff92b6))
- Fix README example ([`1f42a3e`](https://github.com/Byron/jwalk/commit/1f42a3eee18e602d65d420a04a8aace5cb87a20a))
- tests and cleanup ([`5fc2859`](https://github.com/Byron/jwalk/commit/5fc2859b730992aeb8cd74ae7398393ea61e2ff6))
- Fix DirEntry depth ([`5cc9db7`](https://github.com/Byron/jwalk/commit/5cc9db710664d51948f29fbad42e4a1be19b7390))
- Much cleaner, ready for real testing now. ([`2e58da7`](https://github.com/Byron/jwalk/commit/2e58da7524859f571b0206d480a471b17c7e7e75))
- Preping to box instead of template client function. ([`5cd8e04`](https://github.com/Byron/jwalk/commit/5cd8e04b8cd0cf2bdd8ce8a329a081316b5d06af))
- Add more walk options and tests ([`c6f8385`](https://github.com/Byron/jwalk/commit/c6f8385f4d43a66d2c1b2a4bfc29fec1fcde42c1))
- Add hidden file for tests ([`6372a1a`](https://github.com/Byron/jwalk/commit/6372a1aebbcd49aead0833e18c874f0609d09a63))
- Add fts to bench ([`32a527d`](https://github.com/Byron/jwalk/commit/32a527d4e39d853234396289b7cb7e1d3c479232))
- Cleanup ([`d3899aa`](https://github.com/Byron/jwalk/commit/d3899aacebfde722a1a37b4ecbecb81ce701df8e))
- Always return ReadDirResutls, no longer an option ([`32d6b86`](https://github.com/Byron/jwalk/commit/32d6b865ea6b967bb8a7e66710c82039bcf25b80))
- Merge pull request #2 from jessegrosjean/ordered ([`5b4745e`](https://github.com/Byron/jwalk/commit/5b4745e47937fdcccc413f522dfea1a47125b483))
- Merge branch 'master' into ordered ([`3ecc5b6`](https://github.com/Byron/jwalk/commit/3ecc5b62bd86430f5ee4380f113288bdfc7df346))
- Ready for feedback? ([`5d9ee63`](https://github.com/Byron/jwalk/commit/5d9ee635e30697cf1beab1b224b918cbff214ec9))
- more work on work_tree ([`4ced894`](https://github.com/Byron/jwalk/commit/4ced894c71e82a96f4f3c644fe7ec1dd51ad124f))
- working on more generic "work tree" ([`ebf9789`](https://github.com/Byron/jwalk/commit/ebf9789a8799b5e1c9d12d52f303fcb6ca59e0b9))
- Merge pull request #1 from jessegrosjean/ordered ([`36b2d2f`](https://github.com/Byron/jwalk/commit/36b2d2fba8ff5b224f961b9bcbe486e196a90943))
- Cleaned up ([`6b0df4a`](https://github.com/Byron/jwalk/commit/6b0df4aa401541481da9a86d40bb574ef95b9ca6))
- parameterized walk ([`115a5fe`](https://github.com/Byron/jwalk/commit/115a5fefb83968e283ae48bd68c599547735fae6))
- In progress ordered walk ([`ba02dda`](https://github.com/Byron/jwalk/commit/ba02ddaa4cf48b3e2546ae0a2ca4859f8151df50))
- Adding ordered version ([`b63f810`](https://github.com/Byron/jwalk/commit/b63f810eb7ca3ad3b93ea4746ea99a8ce4b16ea6))
- init commit ([`8f64d5a`](https://github.com/Byron/jwalk/commit/8f64d5ae22a3221058f50c17bc97b3c90afac3e0))
</details>
================================================
FILE: Cargo.toml
================================================
[package]
name = "jwalk"
version = "0.9.0"
authors = ["Jesse Grosjean <jesse@hogbaysoftware.com>", "Sebastian Thiel <byronimo@gmail.com>"]
description = "Filesystem walk performed in parallel with streamed and sorted results."
documentation = "https://docs.rs/jwalk"
homepage = "https://github.com/Byron/jwalk"
repository = "https://github.com/Byron/jwalk"
readme = "README.md"
keywords = ["directory", "recursive", "walk", "iterator", "parallel"]
categories = ["filesystem", "concurrency"]
license = "MIT"
edition = "2021"
[dependencies]
rayon = "1.5"
crossbeam = "0.8"
[dev-dependencies]
criterion = "0.5.1"
fs_extra = "1.2"
walkdir = "2.3"
ignore = "0.4"
tempfile = "3.1"
num_cpus = "1.12"
lazy_static = "1.4"
gix-testtools = "0.16.1"
serial_test = "3"
# For examples
clap = { version = "4.4.13", features = ["derive"] }
bytesize = "1.3.0"
[[bench]]
name = "walk_benchmark"
harness = false
serial_test = "3"
================================================
FILE: LICENSE
================================================
The MIT License (MIT)
Copyright (c) 2019 Jesse Grosjean
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
================================================
FILE: README.md
================================================
jwalk
=======
Filesystem walk.
- Performed in parallel using Rayon
- Entries streamed in sorted order
- Custom sort/filter/skip/state
[](https://github.com/Byron/jwalk/actions)
[](https://crates.io/crates/jwalk)
### Usage
Add this to your `Cargo.toml`:
```toml
[dependencies]
jwalk = "0.6"
```
Lean More: [docs.rs/jwalk](https://docs.rs/jwalk)
### Example
Recursively iterate over the "foo" directory sorting by name:
```rust
use jwalk::{WalkDir};
for entry in WalkDir::new("foo").sort(true) {
println!("{}", entry?.path().display());
}
```
### Inspiration
This crate is inspired by both [`walkdir`](https://crates.io/crates/walkdir) and
[`ignore`](https://crates.io/crates/ignore). It attempts to combine the
parallelism of `ignore` with `walkdir`'s streaming iterator API. Some code and
comments are copied directly from `walkdir`.
### Why use this crate?
This crate is particularly good when you want streamed sorted results. In my
tests it's about 4x `walkdir` speed for sorted results with metadata. Also this
crate's `process_read_dir` callback allows you to arbitrarily
sort/filter/skip/state entries before they are yielded.
### Why not use this crate?
Directory traversal is already pretty fast. If you don't need this crate's speed
then `walkdir` provides a smaller and more tested single threaded
implementation.
This crate's parallelism happens at the directory level. It will help when
walking deep file systems with many directories. It wont help when reading a
single directory with many files.
### Benchmarks
[Benchmarks](https://github.com/Byron/jwalk/blob/main/benches/benchmarks.md)
comparing this crate with `walkdir` and `ignore`.
================================================
FILE: benches/benchmarks.md
================================================
Time to walk Linux's source tree on iMac (Retina 5K, 27-inch, Late 2015):
| | threads | jwalk | ignore | walkdir |
|--------------------|----------|------------|------------|--------------|
| unsorted | 8 | 54.631 ms | 70.848 ms | - |
| sorted | 8 | 56.133 ms | 93.345 ms | - |
| sorted, metadata | 8 | 86.985 ms | 122.08 ms | - |
| sorted, first 100 | 8 | 8.9931 ms | - | - |
| unsorted | 2 | 88.416 ms | 108.97 ms | - |
| unsorted | 1 | 141.66 ms | - | 134.28 ms |
| sorted | 1 | 150.89 ms | - | 170.24 ms |
| sorted, metadata | 1 | 313.91 ms | - | 310.26 ms |
## Notes
Comparing the performance of `jwalk`, `ignore`, and `walkdir` and how well they
can use multiple threads.
Options:
- "unsorted" means entries are returned in `read_dir` order.
- "sorted" means entries are returned sorted by name.
- "metadata" means filesystem metadata is loaded for each entry.
- "first 100" means only first 100 entries are taken.
================================================
FILE: benches/walk_benchmark.rs
================================================
#![allow(dead_code)]
#![allow(unused_imports)]
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use ignore::WalkBuilder;
use jwalk::{Error, Parallelism, WalkDir, WalkDirGeneric};
use num_cpus;
use rayon::prelude::*;
use std::cmp;
use std::fs::Metadata;
use std::path::Path;
use std::path::PathBuf;
use std::process::Command;
use std::sync::mpsc;
use walkdir;
fn big_dir() -> PathBuf {
std::env::var_os("JWALK_BENCHMARK_DIR")
.expect(
"the JWALK_BENCHMARK_DIR must be set to the directory to traverse for the benchmark",
)
.into()
}
fn checkout_linux_if_needed() {
let linux_dir = big_dir();
if !linux_dir.exists() {
println!("will git clone linux...");
let output = Command::new("git")
.arg("clone")
.arg("https://github.com/BurntSushi/linux.git")
.arg(&linux_dir)
.output()
.expect("failed to git clone linux");
println!("did git clone linux...{:?}", output);
}
}
fn walk_benches(c: &mut Criterion) {
checkout_linux_if_needed();
c.bench_function("rayon (unsorted, n threads)", |b| {
b.iter(|| black_box(rayon_recursive_descent(big_dir(), None, false)))
});
c.bench_function("rayon (unsorted, metadata, n threads)", |b| {
b.iter(|| black_box(rayon_recursive_descent(big_dir(), None, true)))
});
c.bench_function("jwalk (unsorted, n threads)", |b| {
b.iter(|| for _ in WalkDir::new(big_dir()) {})
});
c.bench_function("jwalk (sorted, n threads)", |b| {
b.iter(|| for _ in WalkDir::new(big_dir()).sort(true) {})
});
c.bench_function("jwalk (sorted, metadata, n threads)", |b| {
b.iter(|| {
for _ in WalkDirGeneric::<((), Option<Result<Metadata, Error>>)>::new(big_dir())
.sort(true)
.process_read_dir(|_, _, _, dir_entry_results| {
dir_entry_results.iter_mut().for_each(|dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
dir_entry.client_state = Some(dir_entry.metadata());
}
})
})
{}
})
});
c.bench_function("jwalk (sorted, n threads, first 100)", |b| {
b.iter(
|| {
for _ in WalkDir::new(big_dir()).sort(true).into_iter().take(100) {}
},
)
});
c.bench_function("jwalk (unsorted, 2 threads)", |b| {
b.iter(
|| {
for _ in WalkDir::new(big_dir()).parallelism(Parallelism::RayonNewPool(2)) {}
},
)
});
c.bench_function("jwalk (unsorted, 1 thread)", |b| {
b.iter(
|| {
for _ in WalkDir::new(big_dir()).parallelism(Parallelism::Serial) {}
},
)
});
c.bench_function("jwalk (sorted, 1 thread)", |b| {
b.iter(|| {
for _ in WalkDir::new(big_dir())
.sort(true)
.parallelism(Parallelism::Serial)
{}
})
});
c.bench_function("jwalk (sorted, metadata, 1 thread)", |b| {
b.iter(|| {
for _ in WalkDirGeneric::<((), Option<Result<Metadata, Error>>)>::new(big_dir())
.sort(true)
.parallelism(Parallelism::Serial)
.process_read_dir(|_, _, _, dir_entry_results| {
dir_entry_results.iter_mut().for_each(|dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
dir_entry.client_state = Some(dir_entry.metadata());
}
})
})
{}
})
});
c.bench_function("ignore (unsorted, n threads)", move |b| {
b.iter(|| {
WalkBuilder::new(big_dir())
.hidden(false)
.standard_filters(false)
.threads(cmp::min(12, num_cpus::get()))
.build_parallel()
.run(move || Box::new(move |_| ignore::WalkState::Continue));
})
});
c.bench_function("ignore (sorted, n threads)", move |b| {
b.iter(|| {
let (tx, rx) = mpsc::channel();
WalkBuilder::new(big_dir())
.hidden(false)
.standard_filters(false)
.threads(cmp::min(12, num_cpus::get()))
.build_parallel()
.run(move || {
let tx = tx.clone();
Box::new(move |dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
tx.send(dir_entry.file_name().to_owned()).unwrap();
}
ignore::WalkState::Continue
})
});
let mut metadatas: Vec<_> = rx.into_iter().collect();
metadatas.sort_by(|a, b| a.len().cmp(&b.len()))
})
});
c.bench_function("ignore (sorted, metadata, n threads)", move |b| {
b.iter(|| {
let (tx, rx) = mpsc::channel();
WalkBuilder::new(big_dir())
.hidden(false)
.standard_filters(false)
.threads(cmp::min(12, num_cpus::get()))
.build_parallel()
.run(move || {
let tx = tx.clone();
Box::new(move |dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
let _ = dir_entry.metadata();
tx.send(dir_entry.file_name().to_owned()).unwrap();
}
ignore::WalkState::Continue
})
});
let mut metadatas: Vec<_> = rx.into_iter().collect();
metadatas.sort_by(|a, b| a.len().cmp(&b.len()))
})
});
c.bench_function("ignore (unsorted, 2 threads)", move |b| {
b.iter(|| {
WalkBuilder::new(big_dir())
.hidden(false)
.standard_filters(false)
.threads(cmp::min(2, num_cpus::get()))
.build_parallel()
.run(move || Box::new(move |_| ignore::WalkState::Continue));
})
});
c.bench_function("walkdir (unsorted, 1 thread)", move |b| {
b.iter(|| for _ in walkdir::WalkDir::new(big_dir()) {})
});
c.bench_function("walkdir (sorted, 1 thread)", move |b| {
b.iter(|| {
for _ in
walkdir::WalkDir::new(big_dir()).sort_by(|a, b| a.file_name().cmp(b.file_name()))
{
}
})
});
c.bench_function("walkdir (sorted, metadata, 1 thread)", move |b| {
b.iter(|| {
for each in
walkdir::WalkDir::new(big_dir()).sort_by(|a, b| a.file_name().cmp(b.file_name()))
{
let _ = each.unwrap().metadata();
}
})
});
}
fn rayon_recursive_descent(
root: impl AsRef<Path>,
file_type: Option<std::fs::FileType>,
get_file_metadata: bool,
) {
let root = root.as_ref();
let (_metadata, is_dir) = file_type
.map(|ft| {
(
if !ft.is_dir() && get_file_metadata {
std::fs::symlink_metadata(root).ok()
} else {
None
},
ft.is_dir(),
)
})
.or_else(|| {
std::fs::symlink_metadata(root)
.map(|m| {
let is_dir = m.file_type().is_dir();
(Some(m), is_dir)
})
.ok()
})
.unwrap_or((None, false));
if is_dir {
std::fs::read_dir(root)
.map(|iter| {
iter.filter_map(Result::ok)
.collect::<Vec<_>>()
.into_par_iter()
.map(|entry| {
rayon_recursive_descent(
entry.path(),
entry.file_type().ok(),
get_file_metadata,
)
})
.for_each(|_| {})
})
.unwrap_or_default()
};
}
criterion_group! {
name = benches;
config = Criterion::default().sample_size(10);
targets = walk_benches
}
criterion_main!(benches);
================================================
FILE: examples/dc.rs
================================================
//! Collect the amount of directories and files as fast as possible.
mod shared;
use clap::Parser;
use jwalk::WalkDirGeneric;
use walkdir::WalkDir;
#[derive(clap::Parser)]
pub struct OurArgs {
/// Use the `walkdir` crate for walking instead.
#[arg(long)]
use_walkdir: bool,
#[clap(flatten)]
inner: shared::Args,
}
fn main() {
let args = OurArgs::parse();
let parallelism = args.inner.parallelism();
let threads = args.inner.threads();
let path = args.inner.root.unwrap_or_else(|| ".".into());
let (mut dirs, mut files, mut symlinks) = (0, 0, 0);
if args.use_walkdir {
for dir_entry_result in WalkDir::new(&path)
.follow_links(false)
.follow_root_links(true)
{
match dir_entry_result {
Ok(dir_entry) => {
if dir_entry.file_type().is_dir() {
dirs += 1;
} else if dir_entry.file_type().is_file() {
files += 1;
} else if dir_entry.file_type().is_symlink() {
symlinks += 1
}
}
Err(error) => {
println!("Read dir_entry error: {}", error);
}
}
}
} else {
for dir_entry_result in WalkDirGeneric::<((), Option<u64>)>::new(&path)
.skip_hidden(false)
.follow_links(false)
.parallelism(parallelism)
{
match dir_entry_result {
Ok(dir_entry) => {
if dir_entry.file_type.is_dir() {
dirs += 1;
} else if dir_entry.file_type.is_file() {
files += 1;
} else if dir_entry.file_type.is_symlink() {
symlinks += 1
}
}
Err(error) => {
println!("Read dir_entry error: {}", error);
}
}
}
}
println!(
"dirs: {dirs}, files: {files}, symlinks: {symlinks} ({})",
if args.use_walkdir {
"walkdir single-threaded".to_string()
} else {
format!("threads: {threads}")
}
);
}
================================================
FILE: examples/du.rs
================================================
mod shared;
use bytesize::ByteSize;
use clap::Parser;
use jwalk::WalkDirGeneric;
fn main() {
let args = shared::Args::parse();
let mut total: u64 = 0;
let parallelism = args.parallelism();
let path = args.root.unwrap_or_else(|| ".".into());
for dir_entry_result in WalkDirGeneric::<((), Option<u64>)>::new(&path)
.skip_hidden(false)
.parallelism(parallelism)
.process_read_dir(|_, _, _, dir_entry_results| {
dir_entry_results.iter_mut().for_each(|dir_entry_result| {
if let Ok(dir_entry) = dir_entry_result {
if !dir_entry.file_type.is_dir() {
dir_entry.client_state =
Some(dir_entry.metadata().map(|m| m.len()).unwrap_or_default());
}
}
})
})
{
match dir_entry_result {
Ok(dir_entry) => {
if let Some(len) = &dir_entry.client_state {
eprintln!("counting {:?}", dir_entry.path());
total += len;
}
}
Err(error) => {
println!("Read dir_entry error: {}", error);
}
}
}
println!("path: {:?} total bytes: {}", path, ByteSize(total));
}
================================================
FILE: examples/shared/mod.rs
================================================
use jwalk::Parallelism;
use std::num::NonZeroUsize;
use std::path::PathBuf;
#[derive(clap::Parser)]
pub struct Args {
/// The amount of threads to use. Default to 4 on MacOS and `available_parallelism` on other platforms.
/// Set to 1 for single-threaded operation.
#[arg(long, short = 't')]
pub threads: Option<NonZeroUsize>,
/// The path from which to start the operation, or `.` if unset
pub root: Option<PathBuf>,
}
impl Args {
pub fn threads(&self) -> usize {
self.threads
.unwrap_or_else(|| {
if cfg!(target_vendor = "apple") {
NonZeroUsize::new(4).unwrap()
} else {
std::thread::available_parallelism().unwrap_or(NonZeroUsize::new(1).unwrap())
}
})
.get()
}
pub fn parallelism(&self) -> Parallelism {
match self.threads() {
1 => Parallelism::Serial,
n => Parallelism::RayonNewPool(n),
}
}
}
================================================
FILE: src/core/dir_entry.rs
================================================
use std::ffi::{OsStr, OsString};
use std::fmt;
use std::fs::{self, FileType};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::{ClientState, Error, ReadChildren, ReadDirSpec, Result};
/// Representation of a file or directory.
///
/// This representation does not wrap a `std::fs::DirEntry`. Instead it copies
/// `file_name`, `file_type`, and optionally `metadata` out of the underlying
/// `std::fs::DirEntry`. This allows it to quickly drop the underlying file
/// descriptor.
pub struct DirEntry<C: ClientState> {
/// Depth of this entry relative to the root directory where the walk
/// started.
pub depth: usize,
/// File name of this entry without leading path component.
pub file_name: OsString,
/// File type for the file/directory that this entry points at.
pub file_type: FileType,
/// Field where clients can store state from within the The
/// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir)
/// callback.
pub client_state: C::DirEntryState,
/// Path used by this entry's parent to read this entry.
pub parent_path: Arc<Path>,
/// Describes how to recurse from this DirEntry.
/// The [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir)
/// callback may set this field to `None` to skip reading the
/// contents of a particular directory.
pub read_children: Option<ReadChildren<C>>,
// True if [`follow_links`] is `true` AND was created from a symlink path.
follow_link: bool,
// Origins of symlinks followed to get to this entry.
follow_link_ancestors: Arc<Vec<Arc<Path>>>,
}
impl<C: ClientState> DirEntry<C> {
pub(crate) fn from_entry(
depth: usize,
parent_path: Arc<Path>,
fs_dir_entry: &fs::DirEntry,
follow_link_ancestors: Arc<Vec<Arc<Path>>>,
) -> Result<Self> {
let file_type = fs_dir_entry
.file_type()
.map_err(|err| Error::from_path(depth, fs_dir_entry.path(), err))?;
let file_name = fs_dir_entry.file_name();
let read_children: Option<ReadChildren<C>> = if file_type.is_dir() {
Some(ReadChildren {
path: Arc::from(parent_path.join(&file_name)),
error: None,
client_read_state: None,
})
} else {
None
};
Ok(DirEntry {
depth,
file_name,
file_type,
parent_path,
read_children,
client_state: C::DirEntryState::default(),
follow_link: false,
follow_link_ancestors,
})
}
// Only used for root and when following links.
pub(crate) fn from_path(
depth: usize,
path: &Path,
follow_link: bool,
follow_link_ancestors: Arc<Vec<Arc<Path>>>,
) -> Result<Self> {
let metadata = if follow_link {
fs::metadata(path).map_err(|err| Error::from_path(depth, path.to_owned(), err))?
} else {
fs::symlink_metadata(path)
.map_err(|err| Error::from_path(depth, path.to_owned(), err))?
};
let root_name = path.file_name().unwrap_or(path.as_os_str());
let read_children = if metadata.file_type().is_dir() {
Some(ReadChildren {
path: Arc::from(path),
error: None,
client_read_state: None,
})
} else {
None
};
Ok(DirEntry {
depth,
file_name: root_name.to_owned(),
file_type: metadata.file_type(),
parent_path: Arc::from(path.parent().map(Path::to_path_buf).unwrap_or_default()),
read_children,
client_state: C::DirEntryState::default(),
follow_link,
follow_link_ancestors,
})
}
/// Return the file type for the file that this entry points to.
///
/// If this is a symbolic link and [`follow_links`] is `true`, then this
/// returns the type of the target.
///
/// This never makes any system calls.
///
/// [`follow_links`]: struct.WalkDir.html#method.follow_links
pub fn file_type(&self) -> FileType {
self.file_type
}
/// Return the file name of this entry.
///
/// If this entry has no file name (e.g., `/`), then the full path is
/// returned.
pub fn file_name(&self) -> &OsStr {
&self.file_name
}
/// Returns the depth at which this entry was created relative to the root.
///
/// The smallest depth is `0` and always corresponds to the path given
/// to the `new` function on `WalkDir`. Its direct descendants have depth
/// `1`, and their descendants have depth `2`, and so on.
pub fn depth(&self) -> usize {
self.depth
}
/// Path to the file/directory represented by this entry.
///
/// The path is created by joining `parent_path` with `file_name`.
pub fn path(&self) -> PathBuf {
self.parent_path.join(&self.file_name)
}
/// Returns `true` if and only if this entry was created from a symbolic
/// link. This is unaffected by the [`follow_links`] setting.
///
/// When `true`, the value returned by the [`path`] method is a
/// symbolic link name. To get the full target path, you must call
/// [`std::fs::read_link(entry.path())`].
///
/// [`path`]: struct.DirEntry.html#method.path
/// [`follow_links`]: struct.WalkDir.html#method.follow_links
/// [`std::fs::read_link(entry.path())`]: https://doc.rust-lang.org/stable/std/fs/fn.read_link.html
pub fn path_is_symlink(&self) -> bool {
self.file_type.is_symlink() || self.follow_link
}
/// Return the metadata for the file that this entry points to.
///
/// This will follow symbolic links if and only if the [`WalkDir`] value
/// has [`follow_links`] enabled.
///
/// # Platform behavior
///
/// This always calls [`std::fs::symlink_metadata`].
///
/// If this entry is a symbolic link and [`follow_links`] is enabled, then
/// [`std::fs::metadata`] is called instead.
///
/// # Errors
///
/// Similar to [`std::fs::metadata`], returns errors for path values that
/// the program does not have permissions to access or if the path does not
/// exist.
///
/// [`WalkDir`]: struct.WalkDir.html
/// [`follow_links`]: struct.WalkDir.html#method.follow_links
/// [`std::fs::metadata`]: https://doc.rust-lang.org/std/fs/fn.metadata.html
/// [`std::fs::symlink_metadata`]: https://doc.rust-lang.org/stable/std/fs/fn.symlink_metadata.html
pub fn metadata(&self) -> Result<fs::Metadata> {
if self.follow_link {
fs::metadata(self.path())
} else {
fs::symlink_metadata(self.path())
}
.map_err(|err| Error::from_entry(self, err))
}
/// Reference to the path of the directory containing this entry.
pub fn parent_path(&self) -> &Path {
&self.parent_path
}
pub(crate) fn read_children_spec(
&self,
client_read_state: C::ReadDirState,
) -> Option<ReadDirSpec<C>> {
self.read_children
.as_ref()
.map(|read_children| ReadDirSpec {
depth: self.depth,
client_read_state: read_children
.client_read_state
.clone()
.unwrap_or(client_read_state),
path: read_children.path.clone(),
follow_link_ancestors: self.follow_link_ancestors.clone(),
})
}
pub(crate) fn follow_symlink(&self) -> Result<Self> {
let path = self.path();
let origins = self.follow_link_ancestors.clone();
let dir_entry = DirEntry::from_path(self.depth, &path, true, origins)?;
if dir_entry.file_type.is_dir() {
let target = fs::read_link(&path).map_err(|err| Error::from_io(self.depth, err))?;
for ancestor in self.follow_link_ancestors.iter().rev() {
if target.as_path() == ancestor.as_ref() {
return Err(Error::from_loop(
self.depth,
ancestor.as_ref(),
path.as_ref(),
));
}
}
}
Ok(dir_entry)
}
}
impl<C: ClientState> fmt::Debug for DirEntry<C> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "DirEntry({:?})", self.path())
}
}
================================================
FILE: src/core/dir_entry_iter.rs
================================================
use std::iter::Peekable;
use super::*;
use crate::Result;
/// DirEntry iterator from `WalkDir.into_iter()`.
///
/// Yields entries from recursive traversal of filesystem.
pub struct DirEntryIter<C: ClientState> {
min_depth: usize,
// iterator yielding next ReadDir results when needed
pub(crate) read_dir_iter: Option<Peekable<ReadDirIter<C>>>,
// stack of ReadDir results, track location in filesystem traversal
read_dir_results_stack: Vec<vec::IntoIter<Result<DirEntry<C>>>>,
}
impl<C: ClientState> DirEntryIter<C> {
pub(crate) fn new(
root_entry_results: Vec<Result<DirEntry<C>>>,
parallelism: Parallelism,
min_depth: usize,
root_read_dir_state: C::ReadDirState,
core_read_dir_callback: Arc<ReadDirCallback<C>>,
) -> DirEntryIter<C> {
// 1. Gather read_dir_specs from root level
let read_dir_specs: Vec<_> = root_entry_results
.iter()
.flat_map(|dir_entry_result| {
dir_entry_result
.as_ref()
.ok()?
.read_children_spec(root_read_dir_state.clone())
})
.collect();
// 2. Init new read_dir_iter from those specs
let read_dir_iter =
ReadDirIter::try_new(read_dir_specs, parallelism, core_read_dir_callback)
.map(|iter| iter.peekable());
// 3. Return DirEntryIter that will return initial root entries and then
// fill and process read_dir_iter until complete
DirEntryIter {
min_depth,
read_dir_iter,
read_dir_results_stack: vec![root_entry_results.into_iter()],
}
}
fn push_next_read_dir_results(
iter: &mut Peekable<ReadDirIter<C>>,
results: &mut Vec<vec::IntoIter<Result<DirEntry<C>>>>,
) -> Result<()> {
// Push next read dir results or return error if read failed
let read_dir_result = iter.next().unwrap();
let read_dir = match read_dir_result {
Ok(read_dir) => read_dir,
Err(err) => return Err(err),
};
let ReadDir { results_list, .. } = read_dir;
results.push(results_list.into_iter());
Ok(())
}
}
impl<C: ClientState> Iterator for DirEntryIter<C> {
type Item = Result<DirEntry<C>>;
fn next(&mut self) -> Option<Self::Item> {
loop {
// 1. Get current read dir results iter from top of stack
let top_read_dir_results = self.read_dir_results_stack.last_mut()?;
// 2. If more results in current read dir then process
if let Some(dir_entry_result) = top_read_dir_results.next() {
// 2.1 Handle error case
let mut dir_entry = match dir_entry_result {
Ok(dir_entry) => dir_entry,
Err(err) => return Some(Err(err)),
};
// 2.2 If dir_entry has a read_children means we need to read a new
// directory and push those results onto read_dir_results_stack
if let Some(ref mut read_children) = dir_entry.read_children {
let iter = match self.read_dir_iter.as_mut().ok_or_else(Error::busy) {
Ok(iter) => iter,
Err(err) => return Some(Err(err)),
};
if let Err(err) =
Self::push_next_read_dir_results(iter, &mut self.read_dir_results_stack)
{
read_children.error = Some(err);
}
}
if dir_entry.depth >= self.min_depth {
// 2.3 Finished, return dir_entry
return Some(Ok(dir_entry));
}
} else {
// If no more results in current then pop stack
self.read_dir_results_stack.pop();
}
}
}
}
================================================
FILE: src/core/error.rs
================================================
use std::error;
use std::fmt;
use std::io;
use std::path::{Path, PathBuf};
use crate::{ClientState, DirEntry};
/// An error produced by recursively walking a directory.
///
/// This error type is a light wrapper around [`std::io::Error`]. In
/// particular, it adds the following information:
///
/// * The depth at which the error occurred in the file tree, relative to the
/// root.
/// * The path, if any, associated with the IO error.
/// * An indication that a loop occurred when following symbolic links. In this
/// case, there is no underlying IO error.
///
/// To maintain good ergonomics, this type has a
/// [`impl From<Error> for std::io::Error`][impl] defined which preserves the original context.
/// This allows you to use an [`io::Result`] with methods in this crate if you don't care about
/// accessing the underlying error data in a structured form.
///
/// [`std::io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html
/// [`io::Result`]: https://doc.rust-lang.org/stable/std/io/type.Result.html
/// [impl]: struct.Error.html#impl-From%3CError%3E
#[derive(Debug)]
pub struct Error {
depth: usize,
inner: ErrorInner,
}
#[derive(Debug)]
enum ErrorInner {
Io {
path: Option<PathBuf>,
err: io::Error,
},
Loop {
ancestor: PathBuf,
child: PathBuf,
},
ThreadpoolBusy,
}
impl Error {
/// Returns the path associated with this error if one exists.
///
/// For example, if an error occurred while opening a directory handle,
/// the error will include the path passed to [`std::fs::read_dir`].
///
/// [`std::fs::read_dir`]: https://doc.rust-lang.org/stable/std/fs/fn.read_dir.html
pub fn path(&self) -> Option<&Path> {
match self.inner {
ErrorInner::ThreadpoolBusy => None,
ErrorInner::Io { path: None, .. } => None,
ErrorInner::Io {
path: Some(ref path),
..
} => Some(path),
ErrorInner::Loop { ref child, .. } => Some(child),
}
}
/// Returns the path at which a cycle was detected.
///
/// If no cycle was detected, [`None`] is returned.
///
/// A cycle is detected when a directory entry is equivalent to one of
/// its ancestors.
///
/// To get the path to the child directory entry in the cycle, use the
/// [`path`] method.
///
/// [`None`]: https://doc.rust-lang.org/stable/std/option/enum.Option.html#variant.None
/// [`path`]: struct.Error.html#path
pub fn loop_ancestor(&self) -> Option<&Path> {
match self.inner {
ErrorInner::Loop { ref ancestor, .. } => Some(ancestor),
_ => None,
}
}
/// Returns the depth at which this error occurred relative to the root.
///
/// The smallest depth is `0` and always corresponds to the path given to
/// the [`new`] function on [`WalkDir`]. Its direct descendants have depth
/// `1`, and their descendants have depth `2`, and so on.
///
/// [`new`]: struct.WalkDir.html#method.new
/// [`WalkDir`]: struct.WalkDir.html
pub fn depth(&self) -> usize {
self.depth
}
/// Inspect the original [`io::Error`] if there is one.
///
/// [`None`] is returned if the [`Error`] doesn't correspond to an
/// [`io::Error`]. This might happen, for example, when the error was
/// produced because a cycle was found in the directory tree while
/// following symbolic links.
///
/// This method returns a borrowed value that is bound to the lifetime of the [`Error`]. To
/// obtain an owned value, the [`into_io_error`] can be used instead.
///
/// > This is the original [`io::Error`] and is _not_ the same as
/// > [`impl From<Error> for std::io::Error`][impl] which contains additional context about the
/// error.
///
/// # Example
///
/// ```rust,no_run
/// use std::io;
/// use std::path::Path;
///
/// use walkdir::WalkDir;
///
/// for entry in WalkDir::new("foo") {
/// match entry {
/// Ok(entry) => println!("{}", entry.path().display()),
/// Err(err) => {
/// let path = err.path().unwrap_or(Path::new("")).display();
/// println!("failed to access entry {}", path);
/// if let Some(inner) = err.io_error() {
/// match inner.kind() {
/// io::ErrorKind::InvalidData => {
/// println!(
/// "entry contains invalid data: {}",
/// inner)
/// }
/// io::ErrorKind::PermissionDenied => {
/// println!(
/// "Missing permission to read entry: {}",
/// inner)
/// }
/// _ => {
/// println!(
/// "Unexpected error occurred: {}",
/// inner)
/// }
/// }
/// }
/// }
/// }
/// }
/// ```
///
/// [`None`]: https://doc.rust-lang.org/stable/std/option/enum.Option.html#variant.None
/// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html
/// [`From`]: https://doc.rust-lang.org/stable/std/convert/trait.From.html
/// [`Error`]: struct.Error.html
/// [`into_io_error`]: struct.Error.html#method.into_io_error
/// [impl]: struct.Error.html#impl-From%3CError%3E
pub fn io_error(&self) -> Option<&io::Error> {
match self.inner {
ErrorInner::Io { ref err, .. } => Some(err),
_ => None,
}
}
/// Returns true if this error is due to a busy thread-pool that prevented its effective use.
///
/// Note that business detection is timeout based, and we don't know if it would have been a deadlock or not.
pub fn is_busy(&self) -> bool {
matches!(self.inner, ErrorInner::ThreadpoolBusy)
}
/// Similar to [`io_error`] except consumes self to convert to the original
/// [`io::Error`] if one exists.
///
/// [`io_error`]: struct.Error.html#method.io_error
/// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html
pub fn into_io_error(self) -> Option<io::Error> {
match self.inner {
ErrorInner::Io { err, .. } => Some(err),
_ => None,
}
}
pub(crate) fn busy() -> Self {
Error {
depth: 0,
inner: ErrorInner::ThreadpoolBusy,
}
}
pub(crate) fn from_path(depth: usize, pb: PathBuf, err: io::Error) -> Self {
Error {
depth,
inner: ErrorInner::Io {
path: Some(pb),
err,
},
}
}
pub(crate) fn from_entry<C: ClientState>(dent: &DirEntry<C>, err: io::Error) -> Self {
Error {
depth: dent.depth(),
inner: ErrorInner::Io {
path: Some(dent.path()),
err,
},
}
}
pub(crate) fn from_io(depth: usize, err: io::Error) -> Self {
Error {
depth,
inner: ErrorInner::Io { path: None, err },
}
}
pub(crate) fn from_loop(depth: usize, ancestor: &Path, child: &Path) -> Self {
Error {
depth,
inner: ErrorInner::Loop {
ancestor: ancestor.to_path_buf(),
child: child.to_path_buf(),
},
}
}
}
impl error::Error for Error {
fn source(&self) -> Option<&(dyn error::Error + 'static)> {
match self.inner {
ErrorInner::Io { ref err, .. } => Some(err),
ErrorInner::Loop { .. } | ErrorInner::ThreadpoolBusy => None,
}
}
#[allow(deprecated)]
fn description(&self) -> &str {
match self.inner {
ErrorInner::Io { ref err, .. } => err.description(),
ErrorInner::Loop { .. } => "file system loop found",
ErrorInner::ThreadpoolBusy => "thread-pool busy",
}
}
fn cause(&self) -> Option<&dyn error::Error> {
self.source()
}
}
impl fmt::Display for Error {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self.inner {
ErrorInner::ThreadpoolBusy => f.write_str("rayon thread-pool too busy or dependency loop detected - aborting before possibility of deadlock"),
ErrorInner::Io {
path: None,
ref err,
} => err.fmt(f),
ErrorInner::Io {
path: Some(ref path),
ref err,
} => write!(f, "IO error for operation on {}: {}", path.display(), err),
ErrorInner::Loop {
ref ancestor,
ref child,
} => write!(
f,
"File system loop found: \
{} points to an ancestor {}",
child.display(),
ancestor.display()
),
}
}
}
impl From<Error> for io::Error {
/// Convert the [`Error`] to an [`io::Error`], preserving the original
/// [`Error`] as the ["inner error"]. Note that this also makes the display
/// of the error include the context.
///
/// This is different from [`into_io_error`] which returns the original
/// [`io::Error`].
///
/// [`Error`]: struct.Error.html
/// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html
/// ["inner error"]: https://doc.rust-lang.org/std/io/struct.Error.html#method.into_inner
/// [`into_io_error`]: struct.WalkDir.html#method.into_io_error
fn from(walk_err: Error) -> io::Error {
let kind = match walk_err {
Error {
inner: ErrorInner::Io { ref err, .. },
..
} => err.kind(),
Error {
inner: ErrorInner::Loop { .. },
..
} => io::ErrorKind::Other,
Error {
inner: ErrorInner::ThreadpoolBusy,
..
} => io::ErrorKind::Other,
};
io::Error::new(kind, walk_err)
}
}
================================================
FILE: src/core/index_path.rs
================================================
use std::cmp::Ordering;
#[derive(Clone, Debug)]
pub struct IndexPath {
pub indices: Vec<usize>,
}
impl IndexPath {
pub fn new(indices: Vec<usize>) -> IndexPath {
IndexPath { indices }
}
pub fn adding(&self, index: usize) -> IndexPath {
let mut indices = self.indices.clone();
indices.push(index);
IndexPath::new(indices)
}
pub fn push(&mut self, index: usize) {
self.indices.push(index);
}
pub fn increment_last(&mut self) {
*self.indices.last_mut().unwrap() += 1;
}
pub fn pop(&mut self) -> Option<usize> {
self.indices.pop()
}
pub fn is_empty(&self) -> bool {
self.indices.is_empty()
}
}
impl PartialEq for IndexPath {
fn eq(&self, o: &Self) -> bool {
self.indices.eq(&o.indices)
}
}
impl Eq for IndexPath {}
impl PartialOrd for IndexPath {
fn partial_cmp(&self, o: &Self) -> Option<Ordering> {
Some(o.indices.cmp(&self.indices))
}
}
impl Ord for IndexPath {
fn cmp(&self, o: &Self) -> Ordering {
o.indices.cmp(&self.indices)
}
}
================================================
FILE: src/core/mod.rs
================================================
mod dir_entry;
mod dir_entry_iter;
mod error;
mod index_path;
mod ordered;
mod ordered_queue;
mod read_children;
mod read_dir;
mod read_dir_iter;
mod read_dir_spec;
mod run_context;
use rayon::prelude::*;
use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use std::vec;
use index_path::*;
use ordered::*;
use ordered_queue::*;
use read_dir_iter::*;
use run_context::*;
pub use dir_entry::DirEntry;
pub use dir_entry_iter::DirEntryIter;
pub use error::Error;
pub use read_children::ReadChildren;
pub use read_dir::ReadDir;
pub use read_dir_spec::ReadDirSpec;
use crate::{ClientState, Parallelism};
================================================
FILE: src/core/ordered.rs
================================================
use std::cmp::Ordering;
use super::index_path::IndexPath;
pub struct Ordered<T> {
pub value: T,
pub index_path: IndexPath,
pub(crate) child_count: usize,
}
impl<T> Ordered<T> {
pub fn new(value: T, index_path: IndexPath, child_count: usize) -> Ordered<T> {
Ordered {
value,
index_path,
child_count,
}
}
}
impl<T> PartialEq for Ordered<T> {
fn eq(&self, o: &Self) -> bool {
self.index_path.eq(&o.index_path)
}
}
impl<T> Eq for Ordered<T> {}
impl<T> PartialOrd for Ordered<T> {
fn partial_cmp(&self, o: &Self) -> Option<Ordering> {
Some(self.index_path.cmp(&o.index_path))
}
}
impl<T> Ord for Ordered<T> {
fn cmp(&self, o: &Self) -> Ordering {
self.index_path.cmp(&o.index_path)
}
}
================================================
FILE: src/core/ordered_queue.rs
================================================
//! Ordered queue backed by a channel.
use crossbeam::channel::{self, Receiver, SendError, Sender, TryRecvError};
use std::collections::BinaryHeap;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering as AtomicOrdering};
use std::sync::Arc;
use std::thread;
use super::*;
pub(crate) struct OrderedQueue<T>
where
T: Send,
{
sender: Sender<Ordered<T>>,
pending_count: Arc<AtomicUsize>,
stop: Arc<AtomicBool>,
}
pub enum Ordering {
Relaxed,
Strict,
}
pub struct OrderedQueueIter<T>
where
T: Send,
{
ordering: Ordering,
stop: Arc<AtomicBool>,
receiver: Receiver<Ordered<T>>,
receive_buffer: BinaryHeap<Ordered<T>>,
pending_count: Arc<AtomicUsize>,
ordered_matcher: OrderedMatcher,
}
struct OrderedMatcher {
looking_for: IndexPath,
child_count_stack: Vec<usize>,
}
pub(crate) fn new_ordered_queue<T>(
stop: Arc<AtomicBool>,
ordering: Ordering,
) -> (OrderedQueue<T>, OrderedQueueIter<T>)
where
T: Send,
{
let pending_count = Arc::new(AtomicUsize::new(0));
let (sender, receiver) = channel::unbounded();
(
OrderedQueue {
sender,
pending_count: pending_count.clone(),
stop: stop.clone(),
},
OrderedQueueIter {
ordering,
receiver,
ordered_matcher: OrderedMatcher::default(),
receive_buffer: BinaryHeap::new(),
pending_count,
stop,
},
)
}
impl<T> OrderedQueue<T>
where
T: Send,
{
pub fn push(&self, ordered: Ordered<T>) -> Result<(), SendError<Ordered<T>>> {
self.pending_count.fetch_add(1, AtomicOrdering::SeqCst);
self.sender.send(ordered)
}
pub fn complete_item(&self) {
self.pending_count.fetch_sub(1, AtomicOrdering::SeqCst);
}
}
impl<T> Clone for OrderedQueue<T>
where
T: Send,
{
fn clone(&self) -> Self {
OrderedQueue {
sender: self.sender.clone(),
pending_count: self.pending_count.clone(),
stop: self.stop.clone(),
}
}
}
impl<T> OrderedQueueIter<T>
where
T: Send,
{
fn pending_count(&self) -> usize {
self.pending_count.load(AtomicOrdering::SeqCst)
}
fn is_stop(&self) -> bool {
self.stop.load(AtomicOrdering::SeqCst)
}
fn try_next_relaxed(&mut self) -> Result<Ordered<T>, TryRecvError> {
if self.is_stop() {
return Err(TryRecvError::Disconnected);
}
while let Ok(ordered_work) = self.receiver.try_recv() {
self.receive_buffer.push(ordered_work)
}
if let Some(ordered_work) = self.receive_buffer.pop() {
Ok(ordered_work)
} else if self.pending_count() == 0 {
Err(TryRecvError::Disconnected)
} else {
Err(TryRecvError::Empty)
}
}
fn try_next_strict(&mut self) -> Result<Ordered<T>, TryRecvError> {
let looking_for = &self.ordered_matcher.looking_for;
loop {
if self.is_stop() {
return Err(TryRecvError::Disconnected);
}
let top_ordered = self.receive_buffer.peek();
if let Some(top_ordered) = top_ordered {
if top_ordered.index_path.eq(looking_for) {
break;
}
}
if self.ordered_matcher.is_none() {
return Err(TryRecvError::Disconnected);
}
match self.receiver.try_recv() {
Ok(ordered) => {
self.receive_buffer.push(ordered);
}
Err(err) => match err {
TryRecvError::Empty => thread::yield_now(),
TryRecvError::Disconnected => break,
},
}
}
let ordered = self.receive_buffer.pop().unwrap();
self.ordered_matcher.advance_past(&ordered);
Ok(ordered)
}
}
impl<T> Iterator for OrderedQueueIter<T>
where
T: Send,
{
type Item = Ordered<T>;
fn next(&mut self) -> Option<Ordered<T>> {
loop {
let try_next = match self.ordering {
Ordering::Relaxed => self.try_next_relaxed(),
Ordering::Strict => self.try_next_strict(),
};
match try_next {
Ok(next) => {
return Some(next);
}
Err(err) => match err {
TryRecvError::Empty => thread::yield_now(),
TryRecvError::Disconnected => return None,
},
}
}
}
}
impl OrderedMatcher {
fn is_none(&self) -> bool {
self.looking_for.is_empty()
}
fn decrement_remaining_children(&mut self) {
*self.child_count_stack.last_mut().unwrap() -= 1;
}
fn advance_past<T>(&mut self, ordered: &Ordered<T>) {
self.decrement_remaining_children();
if ordered.child_count > 0 {
self.looking_for.push(0);
self.child_count_stack.push(ordered.child_count);
} else {
self.looking_for.increment_last();
while !self.child_count_stack.is_empty() && *self.child_count_stack.last().unwrap() == 0
{
self.looking_for.pop();
self.child_count_stack.pop();
if !self.looking_for.is_empty() {
self.looking_for.increment_last();
}
}
}
}
}
impl Default for OrderedMatcher {
fn default() -> OrderedMatcher {
OrderedMatcher {
looking_for: IndexPath::new(vec![0]),
child_count_stack: vec![1],
}
}
}
================================================
FILE: src/core/read_children.rs
================================================
use std::path::Path;
use std::sync::Arc;
use crate::{ClientState, Error};
/// A reduced, but public, version of ReadDirSpec
pub struct ReadChildren<C: ClientState> {
/// Path that will be used to read child entries. This is
/// automatically set for directories.
pub(crate) path: Arc<Path>,
/// If the resulting `fs::read_dir` generates an error
/// then that error is stored here.
pub(crate) error: Option<Error>,
/// Use this to customize the ReadDirState passed to the next
/// process_read_dir.
/// If None, will clone the previous ReadDirState after the parent
/// call to process_read_dir.
pub client_read_state: Option<C::ReadDirState>,
}
impl<C: ClientState> ReadChildren<C> {
pub(crate) fn new(path: &Path) -> Self {
Self {
path: Arc::from(path),
error: None,
client_read_state: None,
}
}
/// Return the error stored that occurred when reading the directory.
pub fn error(&self) -> Option<&Error> {
self.error.as_ref()
}
}
================================================
FILE: src/core/read_dir.rs
================================================
use super::{ClientState, DirEntry, IndexPath, Ordered, ReadDirSpec};
use crate::Result;
/// Results of successfully reading a directory.
#[derive(Debug)]
pub struct ReadDir<C: ClientState> {
pub(crate) read_dir_state: C::ReadDirState,
pub(crate) results_list: Vec<Result<DirEntry<C>>>,
}
impl<C: ClientState> ReadDir<C> {
pub fn new(
read_dir_state: C::ReadDirState,
results_list: Vec<Result<DirEntry<C>>>,
) -> ReadDir<C> {
ReadDir {
read_dir_state,
results_list,
}
}
pub fn read_children_specs(&self) -> impl Iterator<Item = ReadDirSpec<C>> + '_ {
self.results_list.iter().filter_map(move |each| {
each.as_ref()
.ok()?
.read_children_spec(self.read_dir_state.clone())
})
}
pub fn ordered_read_children_specs(
&self,
index_path: &IndexPath,
) -> Vec<Ordered<ReadDirSpec<C>>> {
self.read_children_specs()
.enumerate()
.map(|(i, spec)| Ordered::new(spec, index_path.adding(i), 0))
.collect()
}
}
================================================
FILE: src/core/read_dir_iter.rs
================================================
use std::sync::Arc;
use super::*;
use crate::Result;
/// Client's read dir function.
pub(crate) type ReadDirCallback<C> =
dyn Fn(ReadDirSpec<C>) -> Result<ReadDir<C>> + Send + Sync + 'static;
/// Result<ReadDir> Iterator.
///
/// Yields ReadDirs (results of fs::read_dir) in order required for recursive
/// directory traversal. Depending on Walk/ParWalk state these reads might be
/// computed in parallel.
pub enum ReadDirIter<C: ClientState> {
Walk {
read_dir_spec_stack: Vec<ReadDirSpec<C>>,
core_read_dir_callback: Arc<ReadDirCallback<C>>,
},
ParWalk {
read_dir_result_iter: OrderedQueueIter<Result<ReadDir<C>>>,
},
}
impl<C: ClientState> ReadDirIter<C> {
pub(crate) fn try_new(
read_dir_specs: Vec<ReadDirSpec<C>>,
parallelism: Parallelism,
core_read_dir_callback: Arc<ReadDirCallback<C>>,
) -> Option<Self> {
if let Parallelism::Serial = parallelism {
ReadDirIter::Walk {
read_dir_spec_stack: read_dir_specs,
core_read_dir_callback,
}
} else {
let stop = Arc::new(AtomicBool::new(false));
let read_dir_result_queue = new_ordered_queue(stop.clone(), Ordering::Strict);
let (read_dir_result_queue, read_dir_result_iter) = read_dir_result_queue;
let read_dir_spec_queue = new_ordered_queue(stop.clone(), Ordering::Relaxed);
let (read_dir_spec_queue, read_dir_spec_iter) = read_dir_spec_queue;
for (i, read_dir_spec) in read_dir_specs.into_iter().enumerate() {
read_dir_spec_queue
.push(Ordered::new(read_dir_spec, IndexPath::new(vec![0]), i))
.unwrap();
}
let run_context = RunContext {
stop,
read_dir_spec_queue,
read_dir_result_queue,
core_read_dir_callback,
};
let (startup_tx, startup_rx) = parallelism
.timeout()
.map(|duration| {
let (tx, rx) = crossbeam::channel::unbounded();
(Some(tx), Some((rx, duration)))
})
.unwrap_or((None, None));
parallelism.spawn(move || {
if let Some(tx) = startup_tx {
if tx.send(()).is_err() {
// rayon didn't install this function in time so the listener exited. Do the same.
return;
}
}
read_dir_spec_iter.par_bridge().for_each_with(
run_context,
|run_context, ordered_read_dir_spec| {
multi_threaded_walk_dir(ordered_read_dir_spec, run_context);
},
);
});
if startup_rx.map_or(false, |(rx, duration)| rx.recv_timeout(duration).is_err()) {
return None;
}
ReadDirIter::ParWalk {
read_dir_result_iter,
}
}
.into()
}
}
impl<C: ClientState> Iterator for ReadDirIter<C> {
type Item = Result<ReadDir<C>>;
fn next(&mut self) -> Option<Self::Item> {
match self {
ReadDirIter::Walk {
read_dir_spec_stack,
core_read_dir_callback,
} => {
let read_dir_spec = read_dir_spec_stack.pop()?;
let read_dir_result = core_read_dir_callback(read_dir_spec);
if let Ok(read_dir) = read_dir_result.as_ref() {
for each_spec in read_dir
.read_children_specs()
.collect::<Vec<_>>()
.into_iter()
.rev()
{
read_dir_spec_stack.push(each_spec);
}
}
Some(read_dir_result)
}
ReadDirIter::ParWalk {
read_dir_result_iter,
} => read_dir_result_iter
.next()
.map(|read_dir_result| read_dir_result.value),
}
}
}
fn multi_threaded_walk_dir<C: ClientState>(
ordered_read_dir_spec: Ordered<ReadDirSpec<C>>,
run_context: &mut RunContext<C>,
) {
let Ordered {
value: read_dir_spec,
index_path,
..
} = ordered_read_dir_spec;
let read_dir_result = (run_context.core_read_dir_callback)(read_dir_spec);
let ordered_read_children_specs = read_dir_result
.as_ref()
.ok()
.map(|read_dir| read_dir.ordered_read_children_specs(&index_path));
let ordered_read_dir_result = Ordered::new(
read_dir_result,
index_path,
ordered_read_children_specs.as_ref().map_or(0, Vec::len),
);
if !run_context.send_read_dir_result(ordered_read_dir_result) {
run_context.stop();
return;
}
if let Some(ordered_read_children_specs) = ordered_read_children_specs {
for each in ordered_read_children_specs {
if !run_context.schedule_read_dir_spec(each) {
run_context.stop();
return;
}
}
}
run_context.complete_item();
}
================================================
FILE: src/core/read_dir_spec.rs
================================================
use std::path::Path;
use std::sync::Arc;
use crate::ClientState;
/// Specification for reading a directory.
///
/// When a directory is read a new `ReadDirSpec` is created for each folder
/// found in that directory. These specs are then sent to a work queue that is
/// used to schedule future directory reads. Use
/// [`max_depth`](struct.WalkDir.html#method.max_depth) and
/// [`process_read_dir`](struct.WalkDir.html#method.process_read_dir) to change
/// this default behavior.
#[derive(Debug)]
pub struct ReadDirSpec<C: ClientState> {
/// Depth of the directory to read relative to root of walk.
pub(crate) depth: usize,
/// Path of the the directory to read.
pub path: Arc<Path>,
/// Client branch state that was set in the
/// [`process_read_dir`](struct.WalkDir.html#method.process_read_dir) callback
/// when reading this directory's parent. One intended use case is to store
/// `.gitignore` state to filter entries during the walk.
pub client_read_state: C::ReadDirState,
// Origins of symlinks followed to get to this entry.
pub(crate) follow_link_ancestors: Arc<Vec<Arc<Path>>>,
}
================================================
FILE: src/core/run_context.rs
================================================
use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering};
use std::sync::Arc;
use super::{ClientState, Ordered, OrderedQueue, ReadDir, ReadDirCallback, ReadDirSpec};
use crate::Result;
pub(crate) struct RunContext<C: ClientState> {
pub(crate) stop: Arc<AtomicBool>,
pub(crate) read_dir_spec_queue: OrderedQueue<ReadDirSpec<C>>,
pub(crate) read_dir_result_queue: OrderedQueue<Result<ReadDir<C>>>,
pub(crate) core_read_dir_callback: Arc<ReadDirCallback<C>>,
}
impl<C: ClientState> RunContext<C> {
pub(crate) fn stop(&self) {
self.stop.store(true, AtomicOrdering::SeqCst);
}
pub(crate) fn schedule_read_dir_spec(&self, ordered_read_dir: Ordered<ReadDirSpec<C>>) -> bool {
self.read_dir_spec_queue.push(ordered_read_dir).is_ok()
}
pub(crate) fn send_read_dir_result(
&self,
read_dir_result: Ordered<Result<ReadDir<C>>>,
) -> bool {
self.read_dir_result_queue.push(read_dir_result).is_ok()
}
pub(crate) fn complete_item(&self) {
self.read_dir_spec_queue.complete_item()
}
}
impl<C: ClientState> Clone for RunContext<C> {
fn clone(&self) -> Self {
RunContext {
stop: self.stop.clone(),
read_dir_spec_queue: self.read_dir_spec_queue.clone(),
read_dir_result_queue: self.read_dir_result_queue.clone(),
core_read_dir_callback: self.core_read_dir_callback.clone(),
}
}
}
================================================
FILE: src/lib.rs
================================================
#![warn(clippy::all)]
#![deny(rust_2018_idioms, missing_docs)]
//! Filesystem walk.
//!
//! - Performed in parallel using rayon
//! - Entries streamed in sorted order
//! - Custom sort/filter/skip/state
//!
//! # Example
//!
//! Recursively iterate over the "foo" directory sorting by name:
//!
//! ```no_run
//! # use std::io::Error;
//! use jwalk::{WalkDir};
//!
//! # fn try_main() -> Result<(), Error> {
//! for entry in WalkDir::new("foo").sort(true) {
//! println!("{}", entry?.path().display());
//! }
//! # Ok(())
//! # }
//! ```
//! # Extended Example
//!
//! This example uses the
//! [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir)
//! callback for custom:
//! 1. **Sort** Entries by name
//! 2. **Filter** Errors and hidden files
//! 3. **Skip** Content of directories at depth 2
//! 4. **State** Track depth `read_dir_state`. Mark first entry in each
//! directory with [`client_state`](struct.DirEntry.html#field.client_state)
//! `= true`.
//!
//! ```no_run
//! # use std::io::Error;
//! use std::cmp::Ordering;
//! use jwalk::{ WalkDirGeneric };
//!
//! # fn try_main() -> Result<(), Error> {
//! let walk_dir = WalkDirGeneric::<((usize),(bool))>::new("foo")
//! .process_read_dir(|depth, path, read_dir_state, children| {
//! // 1. Custom sort
//! children.sort_by(|a, b| match (a, b) {
//! (Ok(a), Ok(b)) => a.file_name.cmp(&b.file_name),
//! (Ok(_), Err(_)) => Ordering::Less,
//! (Err(_), Ok(_)) => Ordering::Greater,
//! (Err(_), Err(_)) => Ordering::Equal,
//! });
//! // 2. Custom filter
//! children.retain(|dir_entry_result| {
//! dir_entry_result.as_ref().map(|dir_entry| {
//! dir_entry.file_name
//! .to_str()
//! .map(|s| s.starts_with('.'))
//! .unwrap_or(false)
//! }).unwrap_or(false)
//! });
//! // 3. Custom skip
//! children.iter_mut().for_each(|dir_entry_result| {
//! if let Ok(dir_entry) = dir_entry_result {
//! if dir_entry.depth == 2 {
//! dir_entry.read_children = None;
//! }
//! }
//! });
//! // 4. Custom state
//! *read_dir_state += 1;
//! children.first_mut().map(|dir_entry_result| {
//! if let Ok(dir_entry) = dir_entry_result {
//! dir_entry.client_state = true;
//! }
//! });
//! });
//!
//! for entry in walk_dir {
//! println!("{}", entry?.path().display());
//! }
//! # Ok(())
//! # }
//! ```
//! # Inspiration
//!
//! This crate is inspired by both [`walkdir`](https://crates.io/crates/walkdir)
//! and [`ignore`](https://crates.io/crates/ignore). It attempts to combine the
//! parallelism of `ignore` with `walkdir`'s streaming iterator API. Some code,
//! comments, and test are copied directly from `walkdir`.
//!
//! # Implementation
//!
//! The following structures are central to the implementation:
//!
//! ## `ReadDirSpec`
//!
//! Specification of a future `read_dir` operation. These are stored in the
//! `read_dir_spec_queue` in depth first order. When a rayon thread is ready for
//! work it pulls the first availible `ReadDirSpec` from this queue.
//!
//! ## `ReadDir`
//!
//! Result of a `read_dir` operation generated by rayon thread. These results
//! are stored in the `read_dir_result_queue`, also depth first ordered.
//!
//! ## `ReadDirIter`
//!
//! Pulls `ReadDir` results from the `read_dir_result_queue`. This iterator is
//! driven by calling thread. Results are returned in strict depth first order.
//!
//! ## `DirEntryIter`
//!
//! Wraps a `ReadDirIter` and yields individual `DirEntry` results in strict
//! depth first order.
mod core;
use rayon::{ThreadPool, ThreadPoolBuilder};
use std::cmp::Ordering;
use std::default::Default;
use std::ffi::OsStr;
use std::fmt::Debug;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::core::{ReadDir, ReadDirSpec};
pub use crate::core::{DirEntry, DirEntryIter, ReadChildren, Error};
pub use rayon;
/// Builder for walking a directory.
pub type WalkDir = WalkDirGeneric<((), ())>;
/// A specialized Result type for WalkDir.
pub type Result<T> = std::result::Result<T, Error>;
/// Client state maintained while performing walk.
///
/// for state stored in DirEntry's
/// [`client_state`](struct.DirEntry.html#field.client_state) field.
///
/// Client state can be stored from within the
/// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) callback.
/// The type of ClientState is determined by WalkDirGeneric type parameter.
pub trait ClientState: Send + Default + Debug + 'static {
/// The state held on directory level.
type ReadDirState: Clone + Send + Default + Debug + 'static;
/// The state held for each entry of the directory.
type DirEntryState: Send + Default + Debug + 'static;
}
/// Generic builder for walking a directory.
///
/// [`ClientState`](trait.ClientState.html) type parameter allows you to specify
/// state to be stored with each DirEntry from within the
/// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir)
/// callback.
///
/// Use [`WalkDir`](type.WalkDir.html) if you don't need to store client state
/// into yielded DirEntries.
pub struct WalkDirGeneric<C: ClientState> {
root: PathBuf,
options: WalkDirOptions<C>,
}
type ProcessReadDirFunction<C> = dyn Fn(Option<usize>, &Path, &mut <C as ClientState>::ReadDirState, &mut Vec<Result<DirEntry<C>>>)
+ Send
+ Sync
+ 'static;
/// Degree of parallelism to use when performing walk.
///
/// Parallelism happens at the directory level. It will help when walking deep
/// filesystems with many directories. It wont help when reading a single
/// directory with many files.
///
/// If you plan to perform lots of per file processing you might want to use Rayon to
#[derive(Clone)]
pub enum Parallelism {
/// Run on calling thread, similar to what happens in the `walkdir` crate.
Serial,
/// Run in default rayon thread pool.
RayonDefaultPool {
/// Define when we consider the rayon default pool too busy to serve our iteration and abort the iteration, defaulting to 1s.
///
/// This can happen if `jwalk` is launched from within a par-iter on a pool that only has a single thread,
/// or if there are many parallel `jwalk` invocations that all use the same threadpool, rendering it too busy
/// to respond within this duration.
busy_timeout: std::time::Duration,
},
/// Run in existing rayon thread pool
RayonExistingPool {
/// The pool to spawn our work onto.
pool: Arc<ThreadPool>,
/// Similar to [`Parallelism::RayonDefaultPool::busy_timeout`] if `Some`, but can be `None` to skip the deadlock check
/// in case you know that there is at least one free thread available on the pool.
busy_timeout: Option<std::time::Duration>,
},
/// Run in new rayon thread pool with # threads
RayonNewPool(usize),
}
struct WalkDirOptions<C: ClientState> {
sort: bool,
min_depth: usize,
max_depth: usize,
skip_hidden: bool,
follow_links: bool,
parallelism: Parallelism,
root_read_dir_state: C::ReadDirState,
process_read_dir: Option<Arc<ProcessReadDirFunction<C>>>,
}
impl<C: ClientState> WalkDirGeneric<C> {
/// Create a builder for a recursive directory iterator starting at the file
/// path root. If root is a directory, then it is the first item yielded by
/// the iterator. If root is a file, then it is the first and only item
/// yielded by the iterator.
///
/// Note that his iterator can fail on the first element if `into_iter()` is used as it
/// has to be infallible. Use [`try_into_iter()`][WalkDirGeneric::try_into_iter()]
/// instead for error handling.
pub fn new<P: AsRef<Path>>(root: P) -> Self {
WalkDirGeneric {
root: root.as_ref().to_path_buf(),
options: WalkDirOptions {
sort: false,
min_depth: 0,
max_depth: ::std::usize::MAX,
skip_hidden: true,
follow_links: false,
parallelism: Parallelism::RayonDefaultPool {
busy_timeout: std::time::Duration::from_secs(1),
},
root_read_dir_state: C::ReadDirState::default(),
process_read_dir: None,
},
}
}
/// Try to create an iterator or fail if the rayon threadpool (in any configuration) is busy.
pub fn try_into_iter(self) -> Result<DirEntryIter<C>> {
let iter = self.into_iter();
if iter.read_dir_iter.is_none() {
Err(Error::busy())
} else {
Ok(iter)
}
}
/// Root path of the walk.
pub fn root(&self) -> &Path {
&self.root
}
/// Sort entries by `file_name` per directory. Defaults to `false`. Use
/// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) for custom
/// sorting or filtering.
pub fn sort(mut self, sort: bool) -> Self {
self.options.sort = sort;
self
}
/// Skip hidden entries. Enabled by default.
pub fn skip_hidden(mut self, skip_hidden: bool) -> Self {
self.options.skip_hidden = skip_hidden;
self
}
/// Follow symbolic links. By default, this is disabled.
///
/// When `yes` is `true`, symbolic links are followed as if they were normal
/// directories and files. If a symbolic link is broken or is involved in a
/// loop, an error is yielded.
///
/// When enabled, the yielded [`DirEntry`] values represent the target of
/// the link while the path corresponds to the link. See the [`DirEntry`]
/// type for more details.
///
/// [`DirEntry`]: struct.DirEntry.html
pub fn follow_links(mut self, follow_links: bool) -> Self {
self.options.follow_links = follow_links;
self
}
/// Set the minimum depth of entries yielded by the iterator.
///
/// The smallest depth is `0` and always corresponds to the path given
/// to the `new` function on this type. Its direct descendents have depth
/// `1`, and their descendents have depth `2`, and so on.
pub fn min_depth(mut self, depth: usize) -> Self {
self.options.min_depth = depth;
if self.options.min_depth > self.options.max_depth {
self.options.min_depth = self.options.max_depth;
}
self
}
/// Set the maximum depth of entries yield by the iterator.
///
/// The smallest depth is `0` and always corresponds to the path given
/// to the `new` function on this type. Its direct descendents have depth
/// `1`, and their descendents have depth `2`, and so on.
///
/// A depth < 2 will automatically change `parallelism` to
/// `Parallelism::Serial`. Parrallelism happens at the `fs::read_dir` level.
/// It only makes sense to use multiple threads when reading more then one
/// directory.
///
/// Note that this will not simply filter the entries of the iterator, but
/// it will actually avoid descending into directories when the depth is
/// exceeded.
pub fn max_depth(mut self, depth: usize) -> Self {
self.options.max_depth = depth;
if self.options.max_depth < self.options.min_depth {
self.options.max_depth = self.options.min_depth;
}
if self.options.max_depth < 2 {
self.options.parallelism = Parallelism::Serial;
}
self
}
/// Degree of parallelism to use when performing walk. Defaults to
/// [`Parallelism::RayonDefaultPool`](enum.Parallelism.html#variant.RayonDefaultPool).
pub fn parallelism(mut self, parallelism: Parallelism) -> Self {
self.options.parallelism = parallelism;
self
}
/// Initial ClientState::ReadDirState that is passed to
/// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir)
/// when processing root. Defaults to ClientState::ReadDirState::default().
pub fn root_read_dir_state(mut self, read_dir_state: C::ReadDirState) -> Self {
self.options.root_read_dir_state = read_dir_state;
self
}
/// A callback function to process (sort/filter/skip/state) each directory
/// of entries before they are yielded. Modify the given array to
/// sort/filter entries. Use [`entry.read_children =
/// None`](struct.DirEntry.html#field.read_children) to yield a
/// directory entry but skip reading its contents. Use
/// [`entry.client_state`](struct.DirEntry.html#field.client_state)
/// to store custom state with an entry.
pub fn process_read_dir<F>(mut self, process_by: F) -> Self
where
F: Fn(Option<usize>, &Path, &mut C::ReadDirState, &mut Vec<Result<DirEntry<C>>>)
+ Send
+ Sync
+ 'static,
{
self.options.process_read_dir = Some(Arc::new(process_by));
self
}
}
fn process_dir_entry_result<C: ClientState>(
dir_entry_result: Result<DirEntry<C>>,
follow_links: bool,
) -> Result<DirEntry<C>> {
match dir_entry_result {
Ok(mut dir_entry) => {
if follow_links && dir_entry.file_type.is_symlink() {
dir_entry = dir_entry.follow_symlink()?;
}
if dir_entry.depth == 0 && dir_entry.file_type.is_symlink() {
// As a special case, if we are processing a root entry, then we
// always follow it even if it's a symlink and follow_links is
// false. We are careful to not let this change the semantics of
// the DirEntry however. Namely, the DirEntry should still
// respect the follow_links setting. When it's disabled, it
// should report itself as a symlink. When it's enabled, it
// should always report itself as the target.
let metadata = fs::metadata(dir_entry.path())
.map_err(|err| Error::from_path(0, dir_entry.path(), err))?;
if metadata.file_type().is_dir() {
dir_entry.read_children = Some(ReadChildren::new(&dir_entry.path()));
}
}
Ok(dir_entry)
}
Err(err) => Err(err),
}
}
impl<C: ClientState> IntoIterator for WalkDirGeneric<C> {
type Item = Result<DirEntry<C>>;
type IntoIter = DirEntryIter<C>;
fn into_iter(self) -> DirEntryIter<C> {
let sort = self.options.sort;
let max_depth = self.options.max_depth;
let min_depth = self.options.min_depth;
let parallelism = self.options.parallelism;
let skip_hidden = self.options.skip_hidden;
let follow_links = self.options.follow_links;
let process_read_dir = self.options.process_read_dir.clone();
let mut root_read_dir_state = self.options.root_read_dir_state;
let follow_link_ancestors = if follow_links {
Arc::new(vec![Arc::from(self.root.clone()) as Arc<Path>])
} else {
Arc::new(vec![])
};
let root_entry = DirEntry::from_path(0, &self.root, false, follow_link_ancestors);
let root_parent_path = root_entry
.as_ref()
.map(|root| root.parent_path().to_owned())
.unwrap_or_default();
let mut root_entry_results = vec![process_dir_entry_result(root_entry, follow_links)];
if let Some(process_read_dir) = process_read_dir.as_ref() {
process_read_dir(
None,
&root_parent_path,
&mut root_read_dir_state,
&mut root_entry_results,
);
}
DirEntryIter::new(
root_entry_results,
parallelism,
min_depth,
root_read_dir_state,
Arc::new(move |read_dir_spec| {
let ReadDirSpec {
path,
depth,
mut client_read_state,
mut follow_link_ancestors,
} = read_dir_spec;
let read_dir_depth = depth;
let read_dir_contents_depth = depth + 1;
if read_dir_contents_depth > max_depth {
return Ok(ReadDir::new(client_read_state, Vec::new()));
}
follow_link_ancestors = if follow_links {
let mut ancestors = Vec::with_capacity(follow_link_ancestors.len() + 1);
ancestors.extend(follow_link_ancestors.iter().cloned());
ancestors.push(path.clone());
Arc::new(ancestors)
} else {
follow_link_ancestors
};
let mut dir_entry_results: Vec<_> = fs::read_dir(path.as_ref())
.map_err(|err| Error::from_path(0, path.to_path_buf(), err))?
.filter_map(|dir_entry_result| {
let fs_dir_entry = match dir_entry_result {
Ok(fs_dir_entry) => fs_dir_entry,
Err(err) => {
return Some(Err(Error::from_io(read_dir_contents_depth, err)))
}
};
let dir_entry = match DirEntry::from_entry(
read_dir_contents_depth,
path.clone(),
&fs_dir_entry,
follow_link_ancestors.clone(),
) {
Ok(dir_entry) => dir_entry,
Err(err) => return Some(Err(err)),
};
if skip_hidden && is_hidden(&dir_entry.file_name) {
return None;
}
Some(process_dir_entry_result(Ok(dir_entry), follow_links))
})
.collect();
if sort {
dir_entry_results.sort_by(|a, b| match (a, b) {
(Ok(a), Ok(b)) => a.file_name.cmp(&b.file_name),
(Ok(_), Err(_)) => Ordering::Less,
(Err(_), Ok(_)) => Ordering::Greater,
(Err(_), Err(_)) => Ordering::Equal,
});
}
if let Some(process_read_dir) = process_read_dir.as_ref() {
process_read_dir(
Some(read_dir_depth),
path.as_ref(),
&mut client_read_state,
&mut dir_entry_results,
);
}
Ok(ReadDir::new(client_read_state, dir_entry_results))
}),
)
}
}
impl<C: ClientState> Clone for WalkDirOptions<C> {
fn clone(&self) -> WalkDirOptions<C> {
WalkDirOptions {
sort: false,
min_depth: self.min_depth,
max_depth: self.max_depth,
skip_hidden: self.skip_hidden,
follow_links: self.follow_links,
parallelism: self.parallelism.clone(),
root_read_dir_state: self.root_read_dir_state.clone(),
process_read_dir: self.process_read_dir.clone(),
}
}
}
impl Parallelism {
pub(crate) fn spawn<OP>(&self, op: OP)
where
OP: FnOnce() + Send + 'static,
{
match self {
Parallelism::Serial => op(),
Parallelism::RayonDefaultPool { .. } => rayon::spawn(op),
Parallelism::RayonNewPool(num_threads) => {
let mut thread_pool = ThreadPoolBuilder::new();
if *num_threads > 0 {
thread_pool = thread_pool.num_threads(*num_threads);
}
if let Ok(thread_pool) = thread_pool.build() {
thread_pool.spawn(op);
} else {
rayon::spawn(op);
}
}
Parallelism::RayonExistingPool { pool, .. } => pool.spawn(op),
}
}
pub(crate) fn timeout(&self) -> Option<std::time::Duration> {
match self {
Parallelism::Serial | Parallelism::RayonNewPool(_) => None,
Parallelism::RayonDefaultPool { busy_timeout } => Some(*busy_timeout),
Parallelism::RayonExistingPool { busy_timeout, .. } => *busy_timeout,
}
}
}
fn is_hidden(file_name: &OsStr) -> bool {
file_name
.to_str()
.map(|s| s.starts_with('.'))
.unwrap_or(false)
}
impl<B, E> ClientState for (B, E)
where
B: Clone + Send + Default + Debug + 'static,
E: Send + Default + Debug + 'static,
{
type ReadDirState = B;
type DirEntryState = E;
}
================================================
FILE: tests/assets/test_dir/a.txt
================================================
a
================================================
FILE: tests/assets/test_dir/b.txt
================================================
b
find me
================================================
FILE: tests/assets/test_dir/c.txt
================================================
c
================================================
FILE: tests/assets/test_dir/group 1/d.txt
================================================
find me
================================================
FILE: tests/assets/test_dir/group 2/.hidden_file.txt
================================================
================================================
FILE: tests/assets/test_dir/group 2/e.txt
================================================
find me
================================================
FILE: tests/detect_deadlock.rs
================================================
use jwalk::WalkDir;
use rayon::prelude::*;
#[test]
fn works() {
rayon::ThreadPoolBuilder::new()
.num_threads(1)
.build_global()
.expect("Failed to initialize worker thread pool");
// Does not finish if jwalk uses shared pool with 1 thread, but we can detect this issue and signal this with an error.
(0..=1)
.collect::<Vec<usize>>()
.par_iter()
.for_each(|round| {
let generic = WalkDir::new(".").parallelism(jwalk::Parallelism::RayonDefaultPool {
busy_timeout: std::time::Duration::from_millis(10),
});
if *round == 0 {
for entry in generic {
match entry {
Ok(_) => panic!("Must detect deadlock"),
Err(err) if err.is_busy() => {}
Err(err) => panic!("Unexpected error: {:?}", err),
}
}
} else {
assert!(matches!(generic.try_into_iter(), Err(err) if err.is_busy()));
}
});
}
================================================
FILE: tests/fixtures/make-basic.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Basic directory structure for simple tests
# Creates: dir/a, dir/foo, dir/foo/a
# One dir with one file inside
mkdir -p foo
touch foo/a
================================================
FILE: tests/fixtures/make-depth-test-3.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# 3-level structure for min/max depth tests
mkdir -p a/b/c
================================================
FILE: tests/fixtures/make-depth-test.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Simple 2-level structure for depth tests
mkdir -p a/b
================================================
FILE: tests/fixtures/make-many-dirs.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Many directories in one directory
mkdir -p foo/a foo/b foo/c
================================================
FILE: tests/fixtures/make-many-files.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Many files in one directory
mkdir -p foo
touch foo/a foo/b foo/c
================================================
FILE: tests/fixtures/make-many-mixed.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Mixed files and directories
mkdir -p foo/a foo/c foo/e
touch foo/b foo/d foo/f
================================================
FILE: tests/fixtures/make-nested.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Deeply nested directory structure
mkdir -p a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z
touch a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z/A
================================================
FILE: tests/fixtures/make-one-dir.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Single directory
mkdir -p a
================================================
FILE: tests/fixtures/make-one-file.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Single file
touch a
================================================
FILE: tests/fixtures/make-siblings.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Sibling directories with files
mkdir -p foo bar
touch foo/a foo/b bar/a bar/b
================================================
FILE: tests/fixtures/make-sort-test.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Directory structure for sort testing
mkdir -p foo/bar/baz/abc quux
================================================
FILE: tests/fixtures/make-symlink-loop.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Symlink that creates a loop
mkdir -p a/b/c
ln -s ../../../a a/b/c/a-link
================================================
FILE: tests/fixtures/make-symlink-root-dir.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Directory symlink at root level
mkdir -p a
touch a/zzz
ln -s a a-link
================================================
FILE: tests/fixtures/make-symlink-root-file.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# File symlink at root level
touch a
ln -s a a-link
================================================
FILE: tests/fixtures/make-symlink-self-loop.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# Self-referencing symlink
ln -s a a
================================================
FILE: tests/fixtures/make-symlinks.sh
================================================
#!/usr/bin/env bash
set -eu -o pipefail
# File and directory symlinks
mkdir -p a
touch a_file
ln -s a_file a_file_link
ln -s a a_dir_link
touch a/zzz
================================================
FILE: tests/integration.rs
================================================
use rayon::iter::ParallelIterator;
use rayon::prelude::*;
use std::env;
use std::fs;
use std::path::PathBuf;
mod util;
use jwalk::*;
use util::{parallelism_options, test_dir, Dir, ReadOnlyDir};
#[test]
fn empty() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
let wd = WalkDir::new(dir.path()).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
assert_eq!(1, r.ents().len());
let ent = &r.ents()[0];
assert!(ent.file_type().is_dir());
assert!(!ent.path_is_symlink());
assert_eq!(0, ent.depth());
assert_eq!(dir.path(), ent.path());
assert_eq!(dir.path().file_name().unwrap(), ent.file_name());
}
}
#[test]
fn empty_follow() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
let wd = WalkDir::new(dir.path()).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
assert_eq!(1, r.ents().len());
let ent = &r.ents()[0];
assert!(ent.file_type().is_dir());
assert!(!ent.path_is_symlink());
assert_eq!(0, ent.depth());
assert_eq!(dir.path(), ent.path());
assert_eq!(dir.path().file_name().unwrap(), ent.file_name());
}
}
#[test]
fn empty_file() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
dir.touch("a");
let wd = WalkDir::new(dir.path().join("a")).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
assert_eq!(1, r.ents().len());
let ent = &r.ents()[0];
assert!(ent.file_type().is_file());
assert!(!ent.path_is_symlink());
assert_eq!(0, ent.depth());
assert_eq!(dir.join("a"), ent.path());
assert_eq!("a", ent.file_name());
}
}
#[test]
fn empty_file_follow() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
dir.touch("a");
let wd = WalkDir::new(dir.path().join("a")).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
assert_eq!(1, r.ents().len());
let ent = &r.ents()[0];
assert!(ent.file_type().is_file());
assert!(!ent.path_is_symlink());
assert_eq!(0, ent.depth());
assert_eq!(dir.join("a"), ent.path());
assert_eq!("a", ent.file_name());
}
}
#[test]
fn one_dir() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-one-dir.sh");
let wd = WalkDir::new(dir.path()).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(2, ents.len());
let ent = &ents[1];
assert_eq!(dir.join("a"), ent.path());
assert_eq!(1, ent.depth());
assert_eq!("a", ent.file_name());
assert!(ent.file_type().is_dir());
}
}
#[test]
fn one_file() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-one-file.sh");
let wd = WalkDir::new(dir.path()).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(2, ents.len());
let ent = &ents[1];
assert_eq!(dir.join("a"), ent.path());
assert_eq!(1, ent.depth());
assert_eq!("a", ent.file_name());
assert!(ent.file_type().is_file());
}
}
#[test]
fn one_dir_one_file() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-basic.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("foo"),
dir.join("foo").join("a"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn many_files() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-many-files.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("foo"),
dir.join("foo").join("a"),
dir.join("foo").join("b"),
dir.join("foo").join("c"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn many_dirs() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-many-dirs.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("foo"),
dir.join("foo").join("a"),
dir.join("foo").join("b"),
dir.join("foo").join("c"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn many_mixed() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-many-mixed.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("foo"),
dir.join("foo").join("a"),
dir.join("foo").join("b"),
dir.join("foo").join("c"),
dir.join("foo").join("d"),
dir.join("foo").join("e"),
dir.join("foo").join("f"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn nested() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-nested.sh");
let nested = PathBuf::from("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("a"),
dir.join("a/b"),
dir.join("a/b/c"),
dir.join("a/b/c/d"),
dir.join("a/b/c/d/e"),
dir.join("a/b/c/d/e/f"),
dir.join("a/b/c/d/e/f/g"),
dir.join("a/b/c/d/e/f/g/h"),
dir.join("a/b/c/d/e/f/g/h/i"),
dir.join("a/b/c/d/e/f/g/h/i/j"),
dir.join("a/b/c/d/e/f/g/h/i/j/k"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y"),
dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z"),
dir.join(&nested).join("A"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn siblings() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-siblings.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("bar"),
dir.join("bar").join("a"),
dir.join("bar").join("b"),
dir.join("foo"),
dir.join("foo").join("a"),
dir.join("foo").join("b"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn sym_root_file_nofollow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-file.sh");
let wd = WalkDir::new(dir.join("a-link")).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(1, ents.len());
let link = &ents[0];
assert_eq!(dir.join("a-link"), link.path());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(0, link.depth());
assert!(link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
}
}
#[test]
fn sym_root_file_follow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-file.sh");
let wd = WalkDir::new(dir.join("a-link"))
.sort(true)
.follow_links(true)
.parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
let link = &ents[0];
assert_eq!(dir.join("a-link"), link.path());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(0, link.depth());
assert!(!link.file_type().is_symlink());
assert!(link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(!link.metadata().unwrap().file_type().is_symlink());
assert!(link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
}
}
#[test]
fn sym_root_dir_nofollow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-dir.sh");
let wd = WalkDir::new(dir.join("a-link")).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(2, ents.len());
let link = &ents[0];
assert_eq!(dir.join("a-link"), link.path());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(0, link.depth());
assert!(link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
let link_zzz = &ents[1];
assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path());
assert!(!link_zzz.path_is_symlink());
}
}
#[test]
fn sym_root_dir_follow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-dir.sh");
let wd = WalkDir::new(dir.join("a-link"))
.sort(true)
.follow_links(true)
.parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(2, ents.len());
let link = &ents[0];
assert_eq!(dir.join("a-link"), link.path());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(0, link.depth());
assert!(!link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(link.file_type().is_dir());
assert!(!link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(link.metadata().unwrap().is_dir());
let link_zzz = &ents[1];
assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path());
assert!(!link_zzz.path_is_symlink());
}
}
#[test]
fn sym_file_nofollow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-file.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(3, ents.len());
let (src, link) = (&ents[1], &ents[2]);
assert_eq!(dir.join("a"), src.path());
assert_eq!(dir.join("a-link"), link.path());
assert!(!src.path_is_symlink());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(1, src.depth());
assert_eq!(1, link.depth());
assert!(src.file_type().is_file());
assert!(link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(src.metadata().unwrap().is_file());
assert!(link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
}
}
#[test]
fn sym_file_follow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-file.sh");
let wd = WalkDir::new(dir.path()).sort(true).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(3, ents.len());
let (src, link) = (&ents[1], &ents[2]);
assert_eq!(dir.join("a"), src.path());
assert_eq!(dir.join("a-link"), link.path());
assert!(!src.path_is_symlink());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(1, src.depth());
assert_eq!(1, link.depth());
assert!(src.file_type().is_file());
assert!(!link.file_type().is_symlink());
assert!(link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(src.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().file_type().is_symlink());
assert!(link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
}
}
#[test]
fn sym_dir_nofollow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-dir.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(4, ents.len());
let (src, link) = (&ents[1], &ents[3]);
assert_eq!(dir.join("a"), src.path());
assert_eq!(dir.join("a-link"), link.path());
assert!(!src.path_is_symlink());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(1, src.depth());
assert_eq!(1, link.depth());
assert!(src.file_type().is_dir());
assert!(link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(!link.file_type().is_dir());
assert!(src.metadata().unwrap().is_dir());
assert!(link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(!link.metadata().unwrap().is_dir());
}
}
#[test]
fn sym_dir_follow() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-root-dir.sh");
let wd = WalkDir::new(dir.path()).follow_links(true).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let ents = r.ents();
assert_eq!(5, ents.len());
let (src, link) = (&ents[1], &ents[3]);
assert_eq!(dir.join("a"), src.path());
assert_eq!(dir.join("a-link"), link.path());
assert!(!src.path_is_symlink());
assert!(link.path_is_symlink());
assert_eq!(PathBuf::from("a"), fs::read_link(link.path()).unwrap());
assert_eq!(1, src.depth());
assert_eq!(1, link.depth());
assert!(src.file_type().is_dir());
assert!(!link.file_type().is_symlink());
assert!(!link.file_type().is_file());
assert!(link.file_type().is_dir());
assert!(src.metadata().unwrap().is_dir());
assert!(!link.metadata().unwrap().file_type().is_symlink());
assert!(!link.metadata().unwrap().is_file());
assert!(link.metadata().unwrap().is_dir());
let (src_zzz, link_zzz) = (&ents[2], &ents[4]);
assert_eq!(dir.join("a").join("zzz"), src_zzz.path());
assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path());
assert!(!src_zzz.path_is_symlink());
assert!(!link_zzz.path_is_symlink());
}
}
#[test]
fn sym_noloop() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
dir.mkdirp("a/b/c");
dir.symlink_dir("a", "a/b/c/a-link");
let wd = WalkDir::new(dir.path()).parallelism(parallelism);
let r = dir.run_recursive(wd);
// There's no loop if we aren't following symlinks.
r.assert_no_errors();
assert_eq!(5, r.ents().len());
}
}
#[test]
fn sym_loop_detect() {
for parallelism in parallelism_options() {
let dir = Dir::tmp();
dir.mkdirp("a/b/c");
dir.symlink_dir("a", "a/b/c/a-link");
let wd = WalkDir::new(dir.path()).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
let (ents, errs) = (r.ents(), r.errs());
assert_eq!(4, ents.len());
assert_eq!(1, errs.len());
let err = &errs[0];
let expected = dir.join("a/b/c/a-link");
assert_eq!(Some(&*expected), err.path());
let expected = dir.join("a");
assert_eq!(Some(&*expected), err.loop_ancestor());
assert_eq!(4, err.depth());
assert!(err.io_error().is_none());
}
}
#[test]
fn sym_self_loop_no_error() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-self-loop.sh");
let wd = WalkDir::new(dir.path()).parallelism(parallelism);
let r = dir.run_recursive(wd);
// No errors occur because even though the symlink points to nowhere, it
// is never followed, and thus no error occurs.
r.assert_no_errors();
assert_eq!(2, r.ents().len());
let ent = &r.ents()[1];
assert_eq!(dir.join("a"), ent.path());
assert!(ent.path_is_symlink());
assert!(ent.file_type().is_symlink());
assert!(!ent.file_type().is_file());
assert!(!ent.file_type().is_dir());
assert!(ent.metadata().unwrap().file_type().is_symlink());
assert!(!ent.metadata().unwrap().file_type().is_file());
assert!(!ent.metadata().unwrap().file_type().is_dir());
}
}
#[test]
fn sym_file_self_loop_io_error() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-self-loop.sh");
let wd = WalkDir::new(dir.path()).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
let (ents, errs) = (r.ents(), r.errs());
assert_eq!(1, ents.len());
assert_eq!(1, errs.len());
let err = &errs[0];
let expected = dir.join("a");
assert_eq!(Some(&*expected), err.path());
assert_eq!(1, err.depth());
assert!(err.loop_ancestor().is_none());
assert!(err.io_error().is_some());
}
}
#[test]
fn sym_dir_self_loop_io_error() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-symlink-self-loop.sh");
let wd = WalkDir::new(dir.path()).follow_links(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
let (ents, errs) = (r.ents(), r.errs());
assert_eq!(1, ents.len());
assert_eq!(1, errs.len());
let err = &errs[0];
let expected = dir.join("a");
assert_eq!(Some(&*expected), err.path());
assert_eq!(1, err.depth());
assert!(err.loop_ancestor().is_none());
assert!(err.io_error().is_some());
}
}
#[test]
fn min_depth_1() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test.sh");
let wd = WalkDir::new(dir.path()).min_depth(1).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.join("a"), dir.join("a").join("b")];
assert_eq!(expected, r.paths());
}
}
#[test]
fn min_depth_2() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test.sh");
let wd = WalkDir::new(dir.path()).min_depth(2).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.join("a").join("b")];
assert_eq!(expected, r.paths());
}
}
#[test]
fn max_depth_0() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test.sh");
let wd = WalkDir::new(dir.path()).max_depth(0).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.path().to_path_buf()];
assert_eq!(expected, r.paths());
}
}
#[test]
fn max_depth_1() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test.sh");
let wd = WalkDir::new(dir.path()).max_depth(1).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.path().to_path_buf(), dir.join("a")];
assert_eq!(expected, r.paths());
}
}
#[test]
fn max_depth_2() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test.sh");
let wd = WalkDir::new(dir.path()).max_depth(2).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("a"),
dir.join("a").join("b"),
];
assert_eq!(expected, r.paths());
}
}
#[test]
fn min_max_depth_diff_0() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test-3.sh");
let wd = WalkDir::new(dir.path())
.min_depth(2)
.max_depth(2)
.sort(true)
.parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.join("a").join("b")];
assert_eq!(expected, r.paths());
}
}
#[test]
fn min_max_depth_diff_1() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-depth-test-3.sh");
let wd = WalkDir::new(dir.path())
.min_depth(1)
.max_depth(2)
.sort(true)
.parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![dir.join("a"), dir.join("a").join("b")];
assert_eq!(expected, r.paths());
}
}
#[test]
fn sort() {
for parallelism in parallelism_options() {
let dir = ReadOnlyDir::from_fixture("make-sort-test.sh");
let wd = WalkDir::new(dir.path()).sort(true).parallelism(parallelism);
let r = dir.run_recursive(wd);
r.assert_no_errors();
let expected = vec![
dir.path().to_path_buf(),
dir.join("foo"),
dir.join("foo").join("bar"),
dir.join("foo").join("bar").join("baz"),
dir.join("foo").join("bar").join("baz").join("abc"),
dir.join("quux"),
];
assert_eq!(expected, r.paths());
}
}
fn local_paths(walk_dir: WalkDir) -> Vec<String> {
let root = walk_dir.root().to_owned();
walk_dir
.into_iter()
.map(|each_result| {
let each_entry = each_result.unwrap();
if let Some(err) = each_entry.read_children.as_ref().and_then(|rc| rc.error()) {
panic!("should not encounter any child errors :{:?}", err);
}
let path = each_entry.path();
let path = path.strip_prefix(&root).unwrap().to_path_buf();
let mut path_string = path.to_str().unwrap().to_string();
path_string.push_str(&format!(" ({})", each_entry.depth));
path_string
})
.collect()
}
#[test]
fn walk_serial() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(
WalkDir::new(test_dir)
.parallelism(Parallelism::Serial)
.sort(true),
);
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 1/d.txt (2)",
"group 2 (1)",
"group 2/e.txt (2)",
]
);
}
#[test]
fn sort_by_name_rayon_custom_2_threads() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(
WalkDir::new(test_dir)
.parallelism(Parallelism::RayonNewPool(2))
.sort(true),
);
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 1/d.txt (2)",
"group 2 (1)",
"group 2/e.txt (2)",
]
);
}
#[test]
fn walk_rayon_global() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(WalkDir::new(test_dir).sort(true));
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 1/d.txt (2)",
"group 2 (1)",
"group 2/e.txt (2)",
]
);
}
#[test]
fn walk_rayon_no_lockup() {
// Without jwalk_par_bridge this locks (pre rayon 1.6.1)
// This test now passes without needing jwalk_par_bridge
// and that code has been removed from jwalk.
let pool = std::sync::Arc::new(
rayon::ThreadPoolBuilder::new()
.num_threads(1)
.build()
.unwrap(),
);
let _: Vec<_> = WalkDir::new(PathBuf::from(env!("CARGO_MANIFEST_DIR")))
.parallelism(Parallelism::RayonExistingPool {
pool,
busy_timeout: std::time::Duration::from_millis(500).into(),
})
.process_read_dir(|_, _, _, dir_entry_results| {
for dir_entry_result in dir_entry_results {
let _ = dir_entry_result
.as_ref()
.map(|dir_entry| dir_entry.metadata());
}
})
.sort(true)
.into_iter()
.collect();
}
#[test]
fn combine_with_rayon_no_lockup_1() {
// only run this test if linux_checkout present
let linux_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("benches/assets/linux_checkout");
if linux_dir.exists() {
rayon::scope(|_| {
eprintln!("WalkDir…");
for _entry in WalkDir::new(linux_dir) {}
eprintln!("WalkDir completed");
});
}
}
#[test]
fn combine_with_rayon_no_lockup_2() {
WalkDir::new(PathBuf::from(env!("CARGO_MANIFEST_DIR")))
.sort(true)
.into_iter()
.par_bridge()
.filter_map(|dir_entry_result| {
let dir_entry = dir_entry_result.ok()?;
if dir_entry.file_type().is_file() {
let path = dir_entry.path();
let text = std::fs::read_to_string(path).ok()?;
if text.contains("hello world") {
return Some(true);
}
}
None
})
.count();
}
#[test]
fn see_hidden_files() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(WalkDir::new(test_dir).skip_hidden(false).sort(true).parallelism(parallelism));
assert!(paths.contains(&"group 2/.hidden_file.txt (2)".to_string()));
}
}
#[test]
fn walk_file() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
let walk_dir = WalkDir::new(test_dir.join("a.txt")).parallelism(parallelism);
let mut iter = walk_dir.into_iter();
assert_eq!(
iter.next().unwrap().unwrap().file_name.to_str().unwrap(),
"a.txt"
);
assert!(iter.next().is_none());
}
}
#[test]
fn walk_file_serial() {
let (test_dir, _temp_dir) = test_dir();
let walk_dir = WalkDir::new(test_dir.join("a.txt")).parallelism(Parallelism::Serial);
let mut iter = walk_dir.into_iter();
assert_eq!(
iter.next().unwrap().unwrap().file_name.to_str().unwrap(),
"a.txt"
);
assert!(iter.next().is_none());
}
#[test]
fn error_when_path_does_not_exist() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
let walk_dir = WalkDir::new(test_dir.join("path_does_not_exist")).parallelism(parallelism);
let mut iter = walk_dir.into_iter();
assert!(iter.next().unwrap().is_err());
assert!(iter.next().is_none());
}
}
#[test]
fn error_when_path_removed_durring_iteration() {
let (test_dir, _temp_dir) = test_dir();
let walk_dir = WalkDir::new(&test_dir)
.parallelism(Parallelism::Serial)
.sort(true);
let mut iter = walk_dir.into_iter();
// Read root. read_dir for root is also called since single thread mode.
let _ = iter.next().unwrap().is_ok(); // " (0)",
// Remove group 2 dir from disk
fs_extra::remove_items(&[test_dir.join("group 2")]).unwrap();
let _ = iter.next().unwrap().is_ok(); // "a.txt (1)",
let _ = iter.next().unwrap().is_ok(); // "b.txt (1)",
let _ = iter.next().unwrap().is_ok(); // "c.txt (1)",
let _ = iter.next().unwrap().is_ok(); // "group 1 (1)",
let _ = iter.next().unwrap().is_ok(); // "group 1/d.txt (2)",
// group 2 is read correctly, since it was read before path removed.
let group_2 = iter.next().unwrap().unwrap();
// group 2 content error IS set, since path is removed when try read_dir for
// group 2 path.
assert!(group_2
.read_children
.as_ref()
.and_then(|rc| rc.error())
.is_some());
// done!
assert!(iter.next().is_none());
}
#[test]
fn walk_root() {
for parallelism in parallelism_options() {
let paths: Vec<_> = WalkDir::new("/")
.max_depth(1)
.sort(true)
.parallelism(parallelism)
.into_iter()
.filter_map(|each| Some(each.ok()?.path()))
.collect();
assert_eq!(paths.first().unwrap().to_str().unwrap(), "/");
}
}
#[test]
fn filter_groups_with_process_read_dir() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(
WalkDir::new(test_dir)
.sort(true)
.parallelism(parallelism)
// Filter groups out manually
.process_read_dir(|_depth, _path, _parent, children| {
children.retain(|each_result| {
each_result
.as_ref()
.map(|dir_entry| {
!dir_entry.file_name.to_string_lossy().starts_with("group")
})
.unwrap_or(true)
});
}),
);
assert_eq!(paths, vec![" (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)",]);
}
}
#[test]
fn filter_group_children_with_process_read_dir() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
let paths = local_paths(
WalkDir::new(test_dir)
.sort(true)
.parallelism(parallelism)
// Filter group children
.process_read_dir(|_depth, _path, _parent, children| {
children.iter_mut().for_each(|each_result| {
if let Ok(each) = each_result {
if each.file_name.to_string_lossy().starts_with("group") {
each.read_children = None;
}
}
});
}),
);
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 2 (1)",
]
);
}
}
#[test]
fn pass_readdir_state_with_process_read_dir() {
let (test_dir, _temp_dir) = test_dir();
let iter = WalkDirGeneric::<(Option<String>, ())>::new(&test_dir)
.sort(true)
.process_read_dir(|_depth, path, state, children| {
if let Some(state) = state.as_deref() {
let fname = path.file_name().and_then(|fname| fname.to_str()).unwrap();
assert_eq!(state, fname);
}
children.retain_mut(|each_result| {
if let Ok(each) = each_result {
let fname = each.file_name().to_str().unwrap().to_owned();
if let Some(ref mut rc) = each.read_children {
rc.client_read_state = Some(Some(fname));
return true;
}
}
false
});
})
.into_iter();
assert_eq!(
iter.map(|e| {
let e = e.unwrap();
format!(
"{} ({})",
e.path().strip_prefix(&test_dir).unwrap().to_str().unwrap(),
e.depth()
)
})
.collect::<Vec<_>>(),
vec![" (0)", "group 1 (1)", "group 2 (1)",]
);
}
#[test]
fn test_read_linux() {
for parallelism in parallelism_options() {
// only run this test if linux_checkout present
let linux_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("benches/assets/linux_checkout");
if linux_dir.exists() {
for each in WalkDir::new(&linux_dir).parallelism(parallelism) {
let path = each.unwrap().path();
assert!(path.exists(), "{:?}", path);
}
}
}
}
================================================
FILE: tests/relative_paths.rs
================================================
use serial_test::serial;
use std::env;
use std::fs;
use std::path::PathBuf;
mod util;
use jwalk::*;
use util::{parallelism_options, test_dir};
fn local_paths(walk_dir: WalkDir) -> Vec<String> {
let root = walk_dir.root().to_owned();
walk_dir
.into_iter()
.map(|each_result| {
let each_entry = each_result.unwrap();
if let Some(err) = each_entry.read_children.as_ref().and_then(|c| c.error()) {
panic!("should not encounter any child errors :{:?}", err);
}
let path = each_entry.path();
let path = path.strip_prefix(&root).unwrap().to_path_buf();
let mut path_string = path.to_str().unwrap().to_string();
path_string.push_str(&format!(" ({})", each_entry.depth));
path_string
})
.collect()
}
#[test]
#[serial]
fn walk_relative_1() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
env::set_current_dir(test_dir).unwrap();
let paths = local_paths(
WalkDir::new(".")
.sort(true)
.parallelism(parallelism.clone()),
);
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 1/d.txt (2)",
"group 2 (1)",
"group 2/e.txt (2)",
]
);
let root_dir_entry = WalkDir::new("..")
.parallelism(parallelism)
.into_iter()
.next()
.unwrap()
.unwrap();
assert_eq!(&root_dir_entry.file_name, "..");
}
}
#[test]
#[serial]
fn walk_relative_2() {
for parallelism in parallelism_options() {
let (test_dir, _temp_dir) = test_dir();
env::set_current_dir(test_dir.join("group 1")).unwrap();
let paths = local_paths(
WalkDir::new("..")
.sort(true)
.parallelism(parallelism.clone()),
);
assert_eq!(
paths,
vec![
" (0)",
"a.txt (1)",
"b.txt (1)",
"c.txt (1)",
"group 1 (1)",
"group 1/d.txt (2)",
"group 2 (1)",
"group 2/e.txt (2)",
]
);
let root_dir_entry = WalkDir::new(".")
.parallelism(parallelism)
.into_iter()
.next()
.unwrap()
.unwrap();
assert_eq!(&root_dir_entry.file_name, ".");
}
}
================================================
FILE: tests/util/mod.rs
================================================
use std::env;
use std::error;
use std::fs::{self, File};
use std::io::{self};
use std::path::{Path, PathBuf};
use std::result;
use jwalk::*;
/// Helper to create a read-only fixture from a script
pub fn fixture(script_name: &str) -> PathBuf {
gix_testtools::scripted_fixture_read_only(script_name).unwrap()
}
/// Returns representative parallelism options for testing
pub fn parallelism_options() -> Vec<Parallelism> {
vec![
Parallelism::Serial,
Parallelism::RayonDefaultPool {
busy_timeout: std::time::Duration::from_secs(1),
},
Parallelism::RayonNewPool(2),
]
}
/// Creates a temporary test directory with the contents of tests/assets/test_dir
pub fn test_dir() -> (PathBuf, tempfile::TempDir) {
let template = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/assets/test_dir");
let temp_dir = tempfile::tempdir().unwrap();
let options = fs_extra::dir::CopyOptions::new();
fs_extra::dir::copy(&template, &temp_dir, &options).unwrap();
let mut test_dir = temp_dir.path().to_path_buf();
test_dir.push(template.file_name().unwrap());
(test_dir, temp_dir)
}
/// A helper for managing a read-only directory created from a fixture script.
#[derive(Debug, Clone)]
pub struct ReadOnlyDir {
path: PathBuf,
}
impl ReadOnlyDir {
/// Create a read-only directory from a fixture script.
pub fn from_fixture(script_name: &str) -> ReadOnlyDir {
let path = gix_testtools::scripted_fixture_read_only(script_name).unwrap();
// Convert to absolute path to match Dir::tmp() behavior
let path = env::current_dir().unwrap().join(path);
ReadOnlyDir { path }
}
/// Return the path to this directory.
pub fn path(&self) -> &Path {
&self.path
}
/// Return a path joined to the path to this directory.
pub fn join<P: AsRef<Path>>(&self, path: P) -> PathBuf {
self.path.join(path)
}
/// Run the given iterator and return the result as a distinct collection
/// of directory entries and errors.
pub fn run_recursive<C, I>(&self, it: I) -> RecursiveResults<C>
where
C: ClientState,
I: IntoIterator<Item = result::Result<DirEntry<C>, Error>>,
{
let mut results = RecursiveResults {
ents: vec![],
errs: vec![],
};
for result in it {
match result {
Ok(ent) => results.ents.push(ent),
Err(err) => results.errs.push(err),
}
}
results
}
}
/// Create an error from a format!-like syntax.
#[macro_export]
macro_rules! err {
($($tt:tt)*) => {
Box::<dyn error::Error + Send + Sync>::from(format!($($tt)*))
}
}
/// A convenient result type alias.
pub type Result<T> = result::Result<T, Box<dyn error::Error + Send + Sync>>;
/// The result of running a recursive directory iterator on a single directory.
#[derive(Debug)]
pub struct RecursiveResults<C: ClientState> {
ents: Vec<DirEntry<C>>,
errs: Vec<Error>,
}
impl<C: ClientState> RecursiveResults<C> {
/// Return all of the errors encountered during traversal.
pub fn errs(&self) -> &[Error] {
&self.errs
}
/// Assert that no errors have occurred.
pub fn assert_no_errors(&self) {
assert!(
self.errs.is_empty(),
"expected to find no errors, but found: {:?}",
self.errs
);
}
/// Return all the successfully retrieved directory entries in the order
/// in which they were retrieved.
pub fn ents(&self) -> &[DirEntry<C>] {
&self.ents
}
/// Return all paths from all successfully retrieved directory entries.
///
/// This does not include paths that correspond to an error.
pub fn paths(&self) -> Vec<PathBuf> {
self.ents.iter().map(|d| d.path()).collect()
}
/*
/// Return all the successfully retrieved directory entries, sorted
/// lexicographically by their full file path.
pub fn sorted_ents(&self) -> Vec<DirEntry<C>> {
let mut ents = self.ents.clone();
ents.sort_by(|e1, e2| e1.path().cmp(e2.path()));
ents
}
/// Return all paths from all successfully retrieved directory entries,
/// sorted lexicographically.
///
/// This does not include paths that correspond to an error.
pub fn sorted_paths(&self) -> Vec<PathBuf> {
self.sorted_ents().into_iter().map(|d| d.into_path()).collect()
}*/
}
/// A helper for managing a directory in which to run tests.
///
/// When manipulating paths within this directory, paths are interpreted
/// relative to this directory.
#[derive(Debug)]
pub struct Dir {
dir: TempDir,
}
impl Dir {
/// Create a new empty temporary directory.
pub fn tmp() -> Dir {
let dir = TempDir::new().unwrap();
Dir { dir }
}
/// Return the path to this directory.
pub fn path(&self) -> &Path {
self.dir.path()
}
/// Return a path joined to the path to this directory.
pub fn join<P: AsRef<Path>>(&self, path: P) -> PathBuf {
self.path().join(path)
}
/// Run the given iterator and return the result as a distinct collection
/// of directory entries and errors.
pub fn run_recursive<C, I>(&self, it: I) -> RecursiveResults<C>
where
C: ClientState,
I: IntoIterator<Item = result::Result<DirEntry<C>, Error>>,
{
let mut results = RecursiveResults {
ents: vec![],
errs: vec![],
};
for result in it {
match result {
Ok(ent) => results.ents.push(ent),
Err(err) => results.errs.push(err),
}
}
results
}
/// Create a directory at the given path, while creating all intermediate
/// directories as needed.
pub fn mkdirp<P: AsRef<Path>>(&self, path: P) {
let full = self.join(path);
fs::create_dir_all(&full)
.map_err(|e| err!("failed to create directory {}: {}", full.display(), e))
.unwrap();
}
/// Create an empty file at the given path. All ancestor directories must
/// already exists.
pub fn touch<P: AsRef<Path>>(&self, path: P) {
let full = self.join(path);
File::create(&full)
.map_err(|e| err!("failed to create file {}: {}", full.display(), e))
.unwrap();
}
/// Create empty files at the given paths. All ancestor directories must
/// already exists.
pub fn touch_all<P: AsRef<Path>>(&self, paths: &[P]) {
for p in paths {
self.touch(p);
}
}
/// Create a file symlink to the given src with the given link name.
pub fn symlink_file<P1: AsRef<Path>, P2: AsRef<Path>>(&self, src: P1, link_name: P2) {
#[cfg(windows)]
fn imp(src: &Path, link_name: &Path) -> io::Result<()> {
use std::os::windows::fs::symlink_file;
symlink_file(src, link_name)
}
#[cfg(unix)]
fn imp(src: &Path, link_name: &Path) -> io::Result<()> {
use std::os::unix::fs::symlink;
symlink(src, link_name)
}
let (src, link_name) = (self.join(src), self.join(link_name));
imp(&src, &link_name)
.map_err(|e| {
err!(
"failed to symlink file {} with target {}: {}",
src.display(),
link_name.display(),
e
)
})
.unwrap()
}
/// Create a directory symlink to the given src with the given link name.
pub fn symlink_dir<P1: AsRef<Path>, P2: AsRef<Path>>(&self, src: P1, link_name: P2) {
#[cfg(windows)]
fn imp(src: &Path, link_name: &Path) -> io::Result<()> {
use std::os::windows::fs::symlink_dir;
symlink_dir(src, link_name)
}
#[cfg(unix)]
fn imp(src: &Path, link_name: &Path) -> io::Result<()> {
use std::os::unix::fs::symlink;
symlink(src, link_name)
}
let (src, link_name) = (self.join(src), self.join(link_name));
imp(&src, &link_name)
.map_err(|e| {
err!(
"failed to symlink directory {} with target {}: {}",
src.display(),
link_name.display(),
e
)
})
.unwrap()
}
}
/// A simple wrapper for creating a temporary directory that is automatically
/// deleted when it's dropped.
///
/// We use this in lieu of tempfile because tempfile brings in too many
/// dependencies.
#[derive(Debug)]
pub struct TempDir(PathBuf);
impl Drop for TempDir {
fn drop(&mut self) {
fs::remove_dir_all(&self.0).unwrap();
}
}
impl TempDir {
/// Create a new empty temporary directory under the system's configured
/// temporary directory.
pub fn new() -> Result<TempDir> {
#[allow(deprecated)]
use std::sync::atomic::{AtomicUsize, Ordering, ATOMIC_USIZE_INIT};
static TRIES: usize = 100;
#[allow(deprecated)]
static COUNTER: AtomicUsize = ATOMIC_USIZE_INIT;
let tmpdir = env::temp_dir();
for _ in 0..TRIES {
let count = COUNTER.fetch_add(1, Ordering::SeqCst);
let path = tmpdir.join("rust-walkdir").join(count.to_string());
if path.is_dir() {
continue;
}
fs::create_dir_all(&path)
.map_err(|e| err!("failed to create {}: {}", path.display(), e))?;
return Ok(TempDir(path));
}
Err(err!("failed to create temp dir after {} tries", TRIES))
}
/// Return the underlying path to this temporary directory.
pub fn path(&self) -> &Path {
&self.0
}
}
gitextract_cyuqkkuz/
├── .github/
│ └── workflows/
│ └── ci.yml
├── .gitignore
├── CHANGELOG.md
├── Cargo.toml
├── LICENSE
├── README.md
├── benches/
│ ├── benchmarks.md
│ └── walk_benchmark.rs
├── examples/
│ ├── dc.rs
│ ├── du.rs
│ └── shared/
│ └── mod.rs
├── src/
│ ├── core/
│ │ ├── dir_entry.rs
│ │ ├── dir_entry_iter.rs
│ │ ├── error.rs
│ │ ├── index_path.rs
│ │ ├── mod.rs
│ │ ├── ordered.rs
│ │ ├── ordered_queue.rs
│ │ ├── read_children.rs
│ │ ├── read_dir.rs
│ │ ├── read_dir_iter.rs
│ │ ├── read_dir_spec.rs
│ │ └── run_context.rs
│ └── lib.rs
└── tests/
├── assets/
│ └── test_dir/
│ ├── a.txt
│ ├── b.txt
│ ├── c.txt
│ ├── group 1/
│ │ └── d.txt
│ └── group 2/
│ ├── .hidden_file.txt
│ └── e.txt
├── detect_deadlock.rs
├── fixtures/
│ ├── make-basic.sh
│ ├── make-depth-test-3.sh
│ ├── make-depth-test.sh
│ ├── make-many-dirs.sh
│ ├── make-many-files.sh
│ ├── make-many-mixed.sh
│ ├── make-nested.sh
│ ├── make-one-dir.sh
│ ├── make-one-file.sh
│ ├── make-siblings.sh
│ ├── make-sort-test.sh
│ ├── make-symlink-loop.sh
│ ├── make-symlink-root-dir.sh
│ ├── make-symlink-root-file.sh
│ ├── make-symlink-self-loop.sh
│ └── make-symlinks.sh
├── integration.rs
├── relative_paths.rs
└── util/
└── mod.rs
SYMBOL INDEX (209 symbols across 20 files)
FILE: benches/walk_benchmark.rs
function big_dir (line 17) | fn big_dir() -> PathBuf {
function checkout_linux_if_needed (line 25) | fn checkout_linux_if_needed() {
function walk_benches (line 39) | fn walk_benches(c: &mut Criterion) {
function rayon_recursive_descent (line 213) | fn rayon_recursive_descent(
FILE: examples/dc.rs
type OurArgs (line 9) | pub struct OurArgs {
function main (line 18) | fn main() {
FILE: examples/du.rs
function main (line 7) | fn main() {
FILE: examples/shared/mod.rs
type Args (line 6) | pub struct Args {
method threads (line 16) | pub fn threads(&self) -> usize {
method parallelism (line 28) | pub fn parallelism(&self) -> Parallelism {
FILE: src/core/dir_entry.rs
type DirEntry (line 15) | pub struct DirEntry<C: ClientState> {
function from_entry (line 41) | pub(crate) fn from_entry(
function from_path (line 74) | pub(crate) fn from_path(
function file_type (line 119) | pub fn file_type(&self) -> FileType {
function file_name (line 127) | pub fn file_name(&self) -> &OsStr {
function depth (line 136) | pub fn depth(&self) -> usize {
function path (line 143) | pub fn path(&self) -> PathBuf {
function path_is_symlink (line 157) | pub fn path_is_symlink(&self) -> bool {
function metadata (line 183) | pub fn metadata(&self) -> Result<fs::Metadata> {
function parent_path (line 193) | pub fn parent_path(&self) -> &Path {
function read_children_spec (line 197) | pub(crate) fn read_children_spec(
function follow_symlink (line 214) | pub(crate) fn follow_symlink(&self) -> Result<Self> {
function fmt (line 237) | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
FILE: src/core/dir_entry_iter.rs
type DirEntryIter (line 9) | pub struct DirEntryIter<C: ClientState> {
function new (line 18) | pub(crate) fn new(
function push_next_read_dir_results (line 50) | fn push_next_read_dir_results(
type Item (line 69) | type Item = Result<DirEntry<C>>;
method next (line 70) | fn next(&mut self) -> Option<Self::Item> {
FILE: src/core/error.rs
type Error (line 28) | pub struct Error {
method path (line 53) | pub fn path(&self) -> Option<&Path> {
method loop_ancestor (line 77) | pub fn loop_ancestor(&self) -> Option<&Path> {
method depth (line 92) | pub fn depth(&self) -> usize {
method io_error (line 154) | pub fn io_error(&self) -> Option<&io::Error> {
method is_busy (line 164) | pub fn is_busy(&self) -> bool {
method into_io_error (line 173) | pub fn into_io_error(self) -> Option<io::Error> {
method busy (line 180) | pub(crate) fn busy() -> Self {
method from_path (line 186) | pub(crate) fn from_path(depth: usize, pb: PathBuf, err: io::Error) -> ...
method from_entry (line 196) | pub(crate) fn from_entry<C: ClientState>(dent: &DirEntry<C>, err: io::...
method from_io (line 206) | pub(crate) fn from_io(depth: usize, err: io::Error) -> Self {
method from_loop (line 213) | pub(crate) fn from_loop(depth: usize, ancestor: &Path, child: &Path) -...
method source (line 225) | fn source(&self) -> Option<&(dyn error::Error + 'static)> {
method description (line 233) | fn description(&self) -> &str {
method cause (line 241) | fn cause(&self) -> Option<&dyn error::Error> {
method fmt (line 247) | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
type ErrorInner (line 34) | enum ErrorInner {
function from (line 284) | fn from(walk_err: Error) -> io::Error {
FILE: src/core/index_path.rs
type IndexPath (line 4) | pub struct IndexPath {
method new (line 9) | pub fn new(indices: Vec<usize>) -> IndexPath {
method adding (line 13) | pub fn adding(&self, index: usize) -> IndexPath {
method push (line 19) | pub fn push(&mut self, index: usize) {
method increment_last (line 23) | pub fn increment_last(&mut self) {
method pop (line 27) | pub fn pop(&mut self) -> Option<usize> {
method is_empty (line 31) | pub fn is_empty(&self) -> bool {
method eq (line 37) | fn eq(&self, o: &Self) -> bool {
method partial_cmp (line 45) | fn partial_cmp(&self, o: &Self) -> Option<Ordering> {
method cmp (line 51) | fn cmp(&self, o: &Self) -> Ordering {
FILE: src/core/ordered.rs
type Ordered (line 5) | pub struct Ordered<T> {
function new (line 12) | pub fn new(value: T, index_path: IndexPath, child_count: usize) -> Order...
method eq (line 22) | fn eq(&self, o: &Self) -> bool {
method partial_cmp (line 30) | fn partial_cmp(&self, o: &Self) -> Option<Ordering> {
method cmp (line 36) | fn cmp(&self, o: &Self) -> Ordering {
FILE: src/core/ordered_queue.rs
type OrderedQueue (line 11) | pub(crate) struct OrderedQueue<T>
type Ordering (line 20) | pub enum Ordering {
type OrderedQueueIter (line 25) | pub struct OrderedQueueIter<T>
type OrderedMatcher (line 37) | struct OrderedMatcher {
method is_none (line 186) | fn is_none(&self) -> bool {
method decrement_remaining_children (line 190) | fn decrement_remaining_children(&mut self) {
method advance_past (line 194) | fn advance_past<T>(&mut self, ordered: &Ordered<T>) {
function new_ordered_queue (line 42) | pub(crate) fn new_ordered_queue<T>(
function push (line 72) | pub fn push(&self, ordered: Ordered<T>) -> Result<(), SendError<Ordered<...
function complete_item (line 77) | pub fn complete_item(&self) {
method clone (line 86) | fn clone(&self) -> Self {
function pending_count (line 99) | fn pending_count(&self) -> usize {
function is_stop (line 103) | fn is_stop(&self) -> bool {
function try_next_relaxed (line 107) | fn try_next_relaxed(&mut self) -> Result<Ordered<T>, TryRecvError> {
function try_next_strict (line 125) | fn try_next_strict(&mut self) -> Result<Ordered<T>, TryRecvError> {
type Item (line 165) | type Item = Ordered<T>;
method next (line 166) | fn next(&mut self) -> Option<Ordered<T>> {
method default (line 215) | fn default() -> OrderedMatcher {
FILE: src/core/read_children.rs
type ReadChildren (line 7) | pub struct ReadChildren<C: ClientState> {
function new (line 22) | pub(crate) fn new(path: &Path) -> Self {
function error (line 31) | pub fn error(&self) -> Option<&Error> {
FILE: src/core/read_dir.rs
type ReadDir (line 6) | pub struct ReadDir<C: ClientState> {
function new (line 12) | pub fn new(
function read_children_specs (line 22) | pub fn read_children_specs(&self) -> impl Iterator<Item = ReadDirSpec<C>...
function ordered_read_children_specs (line 30) | pub fn ordered_read_children_specs(
FILE: src/core/read_dir_iter.rs
type ReadDirCallback (line 7) | pub(crate) type ReadDirCallback<C> =
type ReadDirIter (line 15) | pub enum ReadDirIter<C: ClientState> {
function try_new (line 26) | pub(crate) fn try_new(
type Item (line 89) | type Item = Result<ReadDir<C>>;
method next (line 90) | fn next(&mut self) -> Option<Self::Item> {
function multi_threaded_walk_dir (line 122) | fn multi_threaded_walk_dir<C: ClientState>(
FILE: src/core/read_dir_spec.rs
type ReadDirSpec (line 15) | pub struct ReadDirSpec<C: ClientState> {
FILE: src/core/run_context.rs
type RunContext (line 7) | pub(crate) struct RunContext<C: ClientState> {
function stop (line 15) | pub(crate) fn stop(&self) {
function schedule_read_dir_spec (line 19) | pub(crate) fn schedule_read_dir_spec(&self, ordered_read_dir: Ordered<Re...
function send_read_dir_result (line 23) | pub(crate) fn send_read_dir_result(
function complete_item (line 30) | pub(crate) fn complete_item(&self) {
method clone (line 36) | fn clone(&self) -> Self {
FILE: src/lib.rs
type WalkDir (line 133) | pub type WalkDir = WalkDirGeneric<((), ())>;
type Result (line 136) | pub type Result<T> = std::result::Result<T, Error>;
type ClientState (line 146) | pub trait ClientState: Send + Default + Debug + 'static {
type ReadDirState (line 564) | type ReadDirState = B;
type DirEntryState (line 565) | type DirEntryState = E;
type WalkDirGeneric (line 162) | pub struct WalkDirGeneric<C: ClientState> {
type ProcessReadDirFunction (line 167) | type ProcessReadDirFunction<C> = dyn Fn(Option<usize>, &Path, &mut <C as...
type Parallelism (line 180) | pub enum Parallelism {
method spawn (line 521) | pub(crate) fn spawn<OP>(&self, op: OP)
method timeout (line 543) | pub(crate) fn timeout(&self) -> Option<std::time::Duration> {
type WalkDirOptions (line 204) | struct WalkDirOptions<C: ClientState> {
function new (line 224) | pub fn new<P: AsRef<Path>>(root: P) -> Self {
function try_into_iter (line 243) | pub fn try_into_iter(self) -> Result<DirEntryIter<C>> {
function root (line 253) | pub fn root(&self) -> &Path {
function sort (line 260) | pub fn sort(mut self, sort: bool) -> Self {
function skip_hidden (line 266) | pub fn skip_hidden(mut self, skip_hidden: bool) -> Self {
function follow_links (line 282) | pub fn follow_links(mut self, follow_links: bool) -> Self {
function min_depth (line 292) | pub fn min_depth(mut self, depth: usize) -> Self {
function max_depth (line 314) | pub fn max_depth(mut self, depth: usize) -> Self {
function parallelism (line 327) | pub fn parallelism(mut self, parallelism: Parallelism) -> Self {
function root_read_dir_state (line 335) | pub fn root_read_dir_state(mut self, read_dir_state: C::ReadDirState) ->...
function process_read_dir (line 347) | pub fn process_read_dir<F>(mut self, process_by: F) -> Self
function process_dir_entry_result (line 359) | fn process_dir_entry_result<C: ClientState>(
type Item (line 391) | type Item = Result<DirEntry<C>>;
type IntoIter (line 392) | type IntoIter = DirEntryIter<C>;
method into_iter (line 394) | fn into_iter(self) -> DirEntryIter<C> {
method clone (line 506) | fn clone(&self) -> WalkDirOptions<C> {
function is_hidden (line 552) | fn is_hidden(file_name: &OsStr) -> bool {
FILE: tests/detect_deadlock.rs
function works (line 5) | fn works() {
FILE: tests/integration.rs
function empty (line 13) | fn empty() {
function empty_follow (line 31) | fn empty_follow() {
function empty_file (line 49) | fn empty_file() {
function empty_file_follow (line 69) | fn empty_file_follow() {
function one_dir (line 89) | fn one_dir() {
function one_file (line 108) | fn one_file() {
function one_dir_one_file (line 127) | fn one_dir_one_file() {
function many_files (line 145) | fn many_files() {
function many_dirs (line 165) | fn many_dirs() {
function many_mixed (line 185) | fn many_mixed() {
function nested (line 208) | fn nested() {
function siblings (line 252) | fn siblings() {
function sym_root_file_nofollow (line 274) | fn sym_root_file_nofollow() {
function sym_root_file_follow (line 305) | fn sym_root_file_follow() {
function sym_root_dir_nofollow (line 338) | fn sym_root_dir_nofollow() {
function sym_root_dir_follow (line 373) | fn sym_root_dir_follow() {
function sym_file_nofollow (line 411) | fn sym_file_nofollow() {
function sym_file_follow (line 447) | fn sym_file_follow() {
function sym_dir_nofollow (line 483) | fn sym_dir_nofollow() {
function sym_dir_follow (line 519) | fn sym_dir_follow() {
function sym_noloop (line 561) | fn sym_noloop() {
function sym_loop_detect (line 577) | fn sym_loop_detect() {
function sym_self_loop_no_error (line 604) | fn sym_self_loop_no_error() {
function sym_file_self_loop_io_error (line 630) | fn sym_file_self_loop_io_error() {
function sym_dir_self_loop_io_error (line 652) | fn sym_dir_self_loop_io_error() {
function min_depth_1 (line 674) | fn min_depth_1() {
function min_depth_2 (line 688) | fn min_depth_2() {
function max_depth_0 (line 702) | fn max_depth_0() {
function max_depth_1 (line 716) | fn max_depth_1() {
function max_depth_2 (line 730) | fn max_depth_2() {
function min_max_depth_diff_0 (line 748) | fn min_max_depth_diff_0() {
function min_max_depth_diff_1 (line 766) | fn min_max_depth_diff_1() {
function sort (line 784) | fn sort() {
function local_paths (line 804) | fn local_paths(walk_dir: WalkDir) -> Vec<String> {
function walk_serial (line 823) | fn walk_serial() {
function sort_by_name_rayon_custom_2_threads (line 847) | fn sort_by_name_rayon_custom_2_threads() {
function walk_rayon_global (line 870) | fn walk_rayon_global() {
function walk_rayon_no_lockup (line 889) | fn walk_rayon_no_lockup() {
function combine_with_rayon_no_lockup_1 (line 917) | fn combine_with_rayon_no_lockup_1() {
function combine_with_rayon_no_lockup_2 (line 930) | fn combine_with_rayon_no_lockup_2() {
function see_hidden_files (line 950) | fn see_hidden_files() {
function walk_file (line 959) | fn walk_file() {
function walk_file_serial (line 973) | fn walk_file_serial() {
function error_when_path_does_not_exist (line 985) | fn error_when_path_does_not_exist() {
function error_when_path_removed_durring_iteration (line 996) | fn error_when_path_removed_durring_iteration() {
function walk_root (line 1031) | fn walk_root() {
function filter_groups_with_process_read_dir (line 1045) | fn filter_groups_with_process_read_dir() {
function filter_group_children_with_process_read_dir (line 1069) | fn filter_group_children_with_process_read_dir() {
function pass_readdir_state_with_process_read_dir (line 1102) | fn pass_readdir_state_with_process_read_dir() {
function test_read_linux (line 1138) | fn test_read_linux() {
FILE: tests/relative_paths.rs
function local_paths (line 11) | fn local_paths(walk_dir: WalkDir) -> Vec<String> {
function walk_relative_1 (line 31) | fn walk_relative_1() {
function walk_relative_2 (line 69) | fn walk_relative_2() {
FILE: tests/util/mod.rs
function fixture (line 11) | pub fn fixture(script_name: &str) -> PathBuf {
function parallelism_options (line 16) | pub fn parallelism_options() -> Vec<Parallelism> {
function test_dir (line 27) | pub fn test_dir() -> (PathBuf, tempfile::TempDir) {
type ReadOnlyDir (line 39) | pub struct ReadOnlyDir {
method from_fixture (line 45) | pub fn from_fixture(script_name: &str) -> ReadOnlyDir {
method path (line 53) | pub fn path(&self) -> &Path {
method join (line 58) | pub fn join<P: AsRef<Path>>(&self, path: P) -> PathBuf {
method run_recursive (line 64) | pub fn run_recursive<C, I>(&self, it: I) -> RecursiveResults<C>
type Result (line 92) | pub type Result<T> = result::Result<T, Box<dyn error::Error + Send + Syn...
type RecursiveResults (line 96) | pub struct RecursiveResults<C: ClientState> {
function errs (line 103) | pub fn errs(&self) -> &[Error] {
function assert_no_errors (line 108) | pub fn assert_no_errors(&self) {
function ents (line 118) | pub fn ents(&self) -> &[DirEntry<C>] {
function paths (line 125) | pub fn paths(&self) -> Vec<PathBuf> {
type Dir (line 152) | pub struct Dir {
method tmp (line 158) | pub fn tmp() -> Dir {
method path (line 164) | pub fn path(&self) -> &Path {
method join (line 169) | pub fn join<P: AsRef<Path>>(&self, path: P) -> PathBuf {
method run_recursive (line 175) | pub fn run_recursive<C, I>(&self, it: I) -> RecursiveResults<C>
method mkdirp (line 195) | pub fn mkdirp<P: AsRef<Path>>(&self, path: P) {
method touch (line 204) | pub fn touch<P: AsRef<Path>>(&self, path: P) {
method touch_all (line 213) | pub fn touch_all<P: AsRef<Path>>(&self, paths: &[P]) {
method symlink_file (line 220) | pub fn symlink_file<P1: AsRef<Path>, P2: AsRef<Path>>(&self, src: P1, ...
method symlink_dir (line 247) | pub fn symlink_dir<P1: AsRef<Path>, P2: AsRef<Path>>(&self, src: P1, l...
type TempDir (line 280) | pub struct TempDir(PathBuf);
method new (line 291) | pub fn new() -> Result<TempDir> {
method path (line 314) | pub fn path(&self) -> &Path {
method drop (line 283) | fn drop(&mut self) {
Condensed preview — 50 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (168K chars).
[
{
"path": ".github/workflows/ci.yml",
"chars": 358,
"preview": "name: Rust\n\non:\n push:\n branches: [ main ]\n pull_request:\n branches: [ main ]\n\njobs:\n build-and-test:\n runs-"
},
{
"path": ".gitignore",
"chars": 335,
"preview": "# Rust build artifacts\n/target\n\n# Benchmark assets (large files, regenerate with benchmark setup)\n/benches/assets/linux_"
},
{
"path": "CHANGELOG.md",
"chars": 28274,
"preview": "All notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changelog](https://"
},
{
"path": "Cargo.toml",
"chars": 915,
"preview": "[package]\nname = \"jwalk\"\nversion = \"0.9.0\"\nauthors = [\"Jesse Grosjean <jesse@hogbaysoftware.com>\", \"Sebastian Thiel <byr"
},
{
"path": "LICENSE",
"chars": 1080,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2019 Jesse Grosjean\n\nPermission is hereby granted, free of charge, to any person ob"
},
{
"path": "README.md",
"chars": 1807,
"preview": "jwalk\n=======\n\nFilesystem walk.\n\n- Performed in parallel using Rayon\n- Entries streamed in sorted order \n- Custom sort/f"
},
{
"path": "benches/benchmarks.md",
"chars": 1186,
"preview": "Time to walk Linux's source tree on iMac (Retina 5K, 27-inch, Late 2015):\n\n| | threads | jwalk "
},
{
"path": "benches/walk_benchmark.rs",
"chars": 8504,
"preview": "#![allow(dead_code)]\n#![allow(unused_imports)]\n\nuse criterion::{black_box, criterion_group, criterion_main, Criterion};\n"
},
{
"path": "examples/dc.rs",
"chars": 2289,
"preview": "//! Collect the amount of directories and files as fast as possible.\nmod shared;\n\nuse clap::Parser;\nuse jwalk::WalkDirGe"
},
{
"path": "examples/du.rs",
"chars": 1299,
"preview": "mod shared;\n\nuse bytesize::ByteSize;\nuse clap::Parser;\nuse jwalk::WalkDirGeneric;\n\nfn main() {\n let args = shared::Ar"
},
{
"path": "examples/shared/mod.rs",
"chars": 1013,
"preview": "use jwalk::Parallelism;\nuse std::num::NonZeroUsize;\nuse std::path::PathBuf;\n\n#[derive(clap::Parser)]\npub struct Args {\n "
},
{
"path": "src/core/dir_entry.rs",
"chars": 8586,
"preview": "use std::ffi::{OsStr, OsString};\nuse std::fmt;\nuse std::fs::{self, FileType};\nuse std::path::{Path, PathBuf};\nuse std::s"
},
{
"path": "src/core/dir_entry_iter.rs",
"chars": 3972,
"preview": "use std::iter::Peekable;\n\nuse super::*;\nuse crate::Result;\n\n/// DirEntry iterator from `WalkDir.into_iter()`.\n///\n/// Yi"
},
{
"path": "src/core/error.rs",
"chars": 10391,
"preview": "use std::error;\nuse std::fmt;\nuse std::io;\nuse std::path::{Path, PathBuf};\n\nuse crate::{ClientState, DirEntry};\n\n/// An "
},
{
"path": "src/core/index_path.rs",
"chars": 1107,
"preview": "use std::cmp::Ordering;\n\n#[derive(Clone, Debug)]\npub struct IndexPath {\n pub indices: Vec<usize>,\n}\n\nimpl IndexPath {"
},
{
"path": "src/core/mod.rs",
"chars": 605,
"preview": "mod dir_entry;\nmod dir_entry_iter;\nmod error;\nmod index_path;\nmod ordered;\nmod ordered_queue;\nmod read_children;\nmod rea"
},
{
"path": "src/core/ordered.rs",
"chars": 808,
"preview": "use std::cmp::Ordering;\n\nuse super::index_path::IndexPath;\n\npub struct Ordered<T> {\n pub value: T,\n pub index_path"
},
{
"path": "src/core/ordered_queue.rs",
"chars": 5702,
"preview": "//! Ordered queue backed by a channel.\n\nuse crossbeam::channel::{self, Receiver, SendError, Sender, TryRecvError};\nuse s"
},
{
"path": "src/core/read_children.rs",
"chars": 1060,
"preview": "use std::path::Path;\nuse std::sync::Arc;\n\nuse crate::{ClientState, Error};\n\n/// A reduced, but public, version of ReadDi"
},
{
"path": "src/core/read_dir.rs",
"chars": 1115,
"preview": "use super::{ClientState, DirEntry, IndexPath, Ordered, ReadDirSpec};\nuse crate::Result;\n\n/// Results of successfully rea"
},
{
"path": "src/core/read_dir_iter.rs",
"chars": 5297,
"preview": "use std::sync::Arc;\n\nuse super::*;\nuse crate::Result;\n\n/// Client's read dir function.\npub(crate) type ReadDirCallback<C"
},
{
"path": "src/core/read_dir_spec.rs",
"chars": 1141,
"preview": "use std::path::Path;\nuse std::sync::Arc;\n\nuse crate::ClientState;\n\n/// Specification for reading a directory.\n///\n/// Wh"
},
{
"path": "src/core/run_context.rs",
"chars": 1447,
"preview": "use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering};\nuse std::sync::Arc;\n\nuse super::{ClientState, Ordered, "
},
{
"path": "src/lib.rs",
"chars": 21058,
"preview": "#![warn(clippy::all)]\n#![deny(rust_2018_idioms, missing_docs)]\n\n//! Filesystem walk.\n//!\n//! - Performed in parallel usi"
},
{
"path": "tests/assets/test_dir/a.txt",
"chars": 2,
"preview": "a\n"
},
{
"path": "tests/assets/test_dir/b.txt",
"chars": 9,
"preview": "b\nfind me"
},
{
"path": "tests/assets/test_dir/c.txt",
"chars": 2,
"preview": "c\n"
},
{
"path": "tests/assets/test_dir/group 1/d.txt",
"chars": 7,
"preview": "find me"
},
{
"path": "tests/assets/test_dir/group 2/.hidden_file.txt",
"chars": 0,
"preview": ""
},
{
"path": "tests/assets/test_dir/group 2/e.txt",
"chars": 7,
"preview": "find me"
},
{
"path": "tests/detect_deadlock.rs",
"chars": 1079,
"preview": "use jwalk::WalkDir;\nuse rayon::prelude::*;\n\n#[test]\nfn works() {\n rayon::ThreadPoolBuilder::new()\n .num_thread"
},
{
"path": "tests/fixtures/make-basic.sh",
"chars": 179,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Basic directory structure for simple tests\n# Creates: dir/a, dir/foo, dir/foo"
},
{
"path": "tests/fixtures/make-depth-test-3.sh",
"chars": 100,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# 3-level structure for min/max depth tests\nmkdir -p a/b/c\n"
},
{
"path": "tests/fixtures/make-depth-test.sh",
"chars": 97,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Simple 2-level structure for depth tests\nmkdir -p a/b\n"
},
{
"path": "tests/fixtures/make-many-dirs.sh",
"chars": 104,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Many directories in one directory\nmkdir -p foo/a foo/b foo/c\n"
},
{
"path": "tests/fixtures/make-many-files.sh",
"chars": 108,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Many files in one directory\nmkdir -p foo\ntouch foo/a foo/b foo/c\n"
},
{
"path": "tests/fixtures/make-many-mixed.sh",
"chars": 122,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Mixed files and directories\nmkdir -p foo/a foo/c foo/e\ntouch foo/b foo/d foo/"
},
{
"path": "tests/fixtures/make-nested.sh",
"chars": 198,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Deeply nested directory structure\nmkdir -p a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/"
},
{
"path": "tests/fixtures/make-one-dir.sh",
"chars": 71,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Single directory\nmkdir -p a\n"
},
{
"path": "tests/fixtures/make-one-file.sh",
"chars": 63,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Single file\ntouch a\n"
},
{
"path": "tests/fixtures/make-siblings.sh",
"chars": 121,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Sibling directories with files\nmkdir -p foo bar\ntouch foo/a foo/b bar/a bar/b"
},
{
"path": "tests/fixtures/make-sort-test.sh",
"chars": 110,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Directory structure for sort testing\nmkdir -p foo/bar/baz/abc quux\n"
},
{
"path": "tests/fixtures/make-symlink-loop.sh",
"chars": 116,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Symlink that creates a loop\nmkdir -p a/b/c\nln -s ../../../a a/b/c/a-link\n"
},
{
"path": "tests/fixtures/make-symlink-root-dir.sh",
"chars": 113,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Directory symlink at root level\nmkdir -p a\ntouch a/zzz\nln -s a a-link\n"
},
{
"path": "tests/fixtures/make-symlink-root-file.sh",
"chars": 93,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# File symlink at root level\ntouch a\nln -s a a-link\n"
},
{
"path": "tests/fixtures/make-symlink-self-loop.sh",
"chars": 78,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# Self-referencing symlink\nln -s a a\n"
},
{
"path": "tests/fixtures/make-symlinks.sh",
"chars": 151,
"preview": "#!/usr/bin/env bash\nset -eu -o pipefail\n\n# File and directory symlinks\nmkdir -p a\ntouch a_file\nln -s a_file a_file_link\n"
},
{
"path": "tests/integration.rs",
"chars": 35514,
"preview": "use rayon::iter::ParallelIterator;\nuse rayon::prelude::*;\nuse std::env;\nuse std::fs;\nuse std::path::PathBuf;\n\nmod util;\n"
},
{
"path": "tests/relative_paths.rs",
"chars": 2679,
"preview": "use serial_test::serial;\nuse std::env;\nuse std::fs;\nuse std::path::PathBuf;\n\nmod util;\n\nuse jwalk::*;\nuse util::{paralle"
},
{
"path": "tests/util/mod.rs",
"chars": 9875,
"preview": "use std::env;\nuse std::error;\nuse std::fs::{self, File};\nuse std::io::{self};\nuse std::path::{Path, PathBuf};\nuse std::r"
}
]
About this extraction
This page contains the full source code of the jessegrosjean/jwalk GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 50 files (156.6 KB), approximately 42.6k tokens, and a symbol index with 209 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.