Repository: DocJade/fluster_rs Branch: master Commit: 36150f2f440a Files: 133 Total size: 678.4 KB Directory structure: gitextract_wp7ph85n/ ├── .gitignore ├── .vscode/ │ └── settings.json ├── Cargo.toml ├── LICENSE.txt ├── build.rs ├── readme.md ├── src/ │ ├── error_types/ │ │ ├── block.rs │ │ ├── conversions.rs │ │ ├── critical.rs │ │ ├── drive.rs │ │ ├── filesystem.rs │ │ ├── header.rs │ │ └── mod.rs │ ├── filesystem/ │ │ ├── disk_backup/ │ │ │ ├── mod.rs │ │ │ ├── restore.rs │ │ │ └── update.rs │ │ ├── file_attributes/ │ │ │ ├── conversion.rs │ │ │ └── mod.rs │ │ ├── file_handle/ │ │ │ ├── file_handle_methods.rs │ │ │ ├── file_handle_struct.rs │ │ │ └── mod.rs │ │ ├── filesystem_struct.rs │ │ ├── fuse_filesystem_methods.rs │ │ ├── internal_filesystem_methods.rs │ │ ├── item_flag/ │ │ │ ├── flag_struct.rs │ │ │ └── mod.rs │ │ └── mod.rs │ ├── filesystem_design/ │ │ ├── allocation_spec.md │ │ ├── dense_disk.md │ │ ├── design_choices.md │ │ ├── disk_header.md │ │ ├── disk_layout.md │ │ ├── inode_format.md │ │ ├── pool_header.md │ │ ├── pool_layout.md │ │ └── possible_speed_improvments.md │ ├── helpers/ │ │ ├── hex_view.rs │ │ └── mod.rs │ ├── lib.rs │ ├── main.rs │ ├── pool/ │ │ ├── disk/ │ │ │ ├── blank_disk/ │ │ │ │ ├── blank_disk_methods.rs │ │ │ │ ├── blank_disk_struct.rs │ │ │ │ └── mod.rs │ │ │ ├── drive_methods.rs │ │ │ ├── drive_struct.rs │ │ │ ├── generic/ │ │ │ │ ├── block/ │ │ │ │ │ ├── allocate/ │ │ │ │ │ │ ├── block_allocation.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ ├── block_structs.rs │ │ │ │ │ ├── crc.rs │ │ │ │ │ └── mod.rs │ │ │ │ ├── disk_trait.rs │ │ │ │ ├── generic_structs/ │ │ │ │ │ ├── find_space.rs │ │ │ │ │ ├── mod.rs │ │ │ │ │ └── pointer_struct.rs │ │ │ │ ├── io/ │ │ │ │ │ ├── cache/ │ │ │ │ │ │ ├── cache_implementation.rs │ │ │ │ │ │ ├── cache_io.rs │ │ │ │ │ │ ├── cached_allocation.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── statistics.rs │ │ │ │ │ ├── checked_io.rs │ │ │ │ │ ├── mod.rs │ │ │ │ │ ├── read.rs │ │ │ │ │ ├── wipe.rs │ │ │ │ │ └── write.rs │ │ │ │ └── mod.rs │ │ │ ├── mod.rs │ │ │ ├── pool_disk/ │ │ │ │ ├── block/ │ │ │ │ │ ├── header/ │ │ │ │ │ │ ├── header_methods.rs │ │ │ │ │ │ ├── header_struct.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ └── mod.rs │ │ │ │ ├── mod.rs │ │ │ │ ├── pool_disk_methods.rs │ │ │ │ └── pool_disk_struct.rs │ │ │ ├── standard_disk/ │ │ │ │ ├── block/ │ │ │ │ │ ├── directory/ │ │ │ │ │ │ ├── directory_methods.rs │ │ │ │ │ │ ├── directory_struct.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ ├── file_extents/ │ │ │ │ │ │ ├── file_extents_methods.rs │ │ │ │ │ │ ├── file_extents_struct.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ ├── header/ │ │ │ │ │ │ ├── header_methods.rs │ │ │ │ │ │ ├── header_struct.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ ├── inode/ │ │ │ │ │ │ ├── inode_methods.rs │ │ │ │ │ │ ├── inode_struct.rs │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ └── tests.rs │ │ │ │ │ ├── io/ │ │ │ │ │ │ ├── directory/ │ │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ │ ├── movement.rs │ │ │ │ │ │ │ ├── read.rs │ │ │ │ │ │ │ ├── tests.rs │ │ │ │ │ │ │ ├── types.rs │ │ │ │ │ │ │ └── write.rs │ │ │ │ │ │ ├── file/ │ │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ │ ├── movement.rs │ │ │ │ │ │ │ ├── read.rs │ │ │ │ │ │ │ ├── tests.rs │ │ │ │ │ │ │ └── write.rs │ │ │ │ │ │ ├── inode/ │ │ │ │ │ │ │ ├── mod.rs │ │ │ │ │ │ │ ├── read.rs │ │ │ │ │ │ │ ├── tests.rs │ │ │ │ │ │ │ └── write.rs │ │ │ │ │ │ └── mod.rs │ │ │ │ │ └── mod.rs │ │ │ │ ├── mod.rs │ │ │ │ ├── standard_disk_methods.rs │ │ │ │ └── standard_disk_struct.rs │ │ │ └── unknown_disk/ │ │ │ ├── mod.rs │ │ │ ├── unknown_disk_methods.rs │ │ │ └── unknown_disk_struct.rs │ │ ├── io/ │ │ │ ├── allocate.rs │ │ │ └── mod.rs │ │ ├── mod.rs │ │ └── pool_actions/ │ │ ├── mod.rs │ │ ├── pool_methods.rs │ │ └── pool_struct.rs │ └── tui/ │ ├── layout.rs │ ├── mod.rs │ ├── notify.rs │ ├── prompts.rs │ ├── state.rs │ ├── tasks.rs │ └── tui_struct.rs ├── tests/ │ ├── directory.rs │ ├── file.rs │ ├── mount_filesystem.rs │ ├── start_filesystem.rs │ └── test_common.rs └── windows.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ /target /temp_disks .vscode/launch.json ================================================ FILE: .vscode/settings.json ================================================ { "rust-analyzer.runnables.extraEnv": { "RUST_LOG": "info,fluster_fs=debug" // debug level logs in tests }, "rust-analyzer.runnables.cargo.args": [ "--no-fail-fast", "--", "--nocapture" ], "cSpell.words": [ "uncatagorized" ] } ================================================ FILE: Cargo.toml ================================================ [package] name = "fluster_fs" version = "0.1.0" edition = "2024" [dependencies] bitflags = "2.9.1" clap = { version = "4.5.41", features = ["derive"] } crc32c = "0.6.8" ctrlc = "3.4.7" enum_dispatch = "0.3.13" env_logger = "0.11.8" fuse_mt = "0.6.1" lazy_static = "1.5.0" libc = "0.2.174" log = "0.4.27" log-panics = { version = "2.1.0", features = ["with-backtrace"] } once_cell = "1.21.3" oneshot = "0.1.11" rand = "0.9.1" ratatui = "0.29.0" rprompt = "2.2.0" tempfile = "3.20.0" test-log = "0.2.18" thiserror = "2.0.12" tui-logger = "0.17.3" tui-textarea = "0.7.0" # Floppy builds should be small enough to fit on a single floppy. # Because thats funny. [profile.floppy] inherits = "release" strip = true # Automatically strip symbols from the binary. opt-level = "z" # Optimize for size. lto = true codegen-units = 1 # panic = "abort" # We will NOT use abort. Abort is gross. ================================================ FILE: LICENSE.txt ================================================ Attribution 4.0 International ======================================================================= Creative Commons Corporation ("Creative Commons") is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an "as-is" basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. Using Creative Commons Public Licenses Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC- licensed material, or material used under an exception or limitation to copyright. More considerations for licensors: wiki.creativecommons.org/Considerations_for_licensors Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor's permission is not necessary for any reason--for example, because of any applicable exception or limitation to copyright--then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. More considerations for the public: wiki.creativecommons.org/Considerations_for_licensees ======================================================================= Creative Commons Attribution 4.0 International Public License By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. Section 1 -- Definitions. a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. c. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. d. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. e. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. f. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License. g. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. h. Licensor means the individual(s) or entity(ies) granting rights under this Public License. i. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. j. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. k. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. Section 2 -- Scope. a. License grant. 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: a. reproduce and Share the Licensed Material, in whole or in part; and b. produce, reproduce, and Share Adapted Material. 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 3. Term. The term of this Public License is specified in Section 6(a). 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a) (4) never produces Adapted Material. 5. Downstream recipients. a. Offer from the Licensor -- Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. b. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). b. Other rights. 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 2. Patent and trademark rights are not licensed under this Public License. 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties. Section 3 -- License Conditions. Your exercise of the Licensed Rights is expressly made subject to the following conditions. a. Attribution. 1. If You Share the Licensed Material (including in modified form), You must: a. retain the following if it is supplied by the Licensor with the Licensed Material: i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); ii. a copyright notice; iii. a notice that refers to this Public License; iv. a notice that refers to the disclaimer of warranties; v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; b. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and c. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License. Section 4 -- Sui Generis Database Rights. Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database; b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. Section 5 -- Disclaimer of Warranties and Limitation of Liability. a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. Section 6 -- Term and Termination. a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 2. upon express reinstatement by the Licensor. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. c. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. d. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. Section 7 -- Other Terms and Conditions. a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. Section 8 -- Interpretation. a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. ======================================================================= Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” The text of the Creative Commons public licenses is dedicated to the public domain under the CC0 Public Domain Dedication. Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark "Creative Commons" or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. Creative Commons may be contacted at creativecommons.org. ================================================ FILE: build.rs ================================================ fn main() { #[cfg(target_os = "windows")] panic!( "The `fuser` crate cannot be built on windows. You must build and use fluster_fs through WSL." ); } ================================================ FILE: readme.md ================================================


Fluster
Fluster

A futuristic filesystem that's stuck in the past.

Blazingly fast! Support me on Ko-fi! Gluten free!

Features • How To Use • Credits • License

Fluster playing \

## Features * Multi-disk * Spans a single filesystem across as many floppy disks as are required to store the data, treating them as a single pool. * Disk failure detection * Automatically detects and troubleshoot drive and disk issues. * Automatic backups * Floppy disks are unreliable, so every block operation is backed up to `/var/fluster` in case disk recovery is required. * Tiered caching * Triple tiered, in-memory cache to minimize disk swapping, while only using 2 floppy disks worth of memory. * Error checking * Every 512 byte block has a 4 byte CRC to detect corruption or bad reads, and disk operations will automatically retry if the CRC fails. * FUSE based * Built on [FUSE](https://github.com/libfuse/libfuse), which makes Fluster! mountable on any UNIX or UNIX-like system that supports FUSE. ## How To Use To clone and run Fluster!, you'll need [Rust](www.rust-lang.org), a FUSE implementation, a floppy drive, and at least two floppy disks. ### Prerequisites #### For Linux & macOS - **Install:** - **Rust:** Follow the official installation [guide](https://www.rust-lang.org/). - **FUSE:** On most Linux distributions, libfuse is available through your package manager (e.g., sudo apt-get install libfuse-dev). - On macOS, you may need [macFUSE](https://macfuse.github.io/), although I have not tested Fluster! on MacOS at all, since you should use a real operating system. #### For Windows Users - If you're running Fluster! on Windows, please [read this guide](https://github.com/DocJade/fluster_rs/blob/master/windows.md). ### Building and running #### Build Fluster!: ```bash # Clone the repository git clone https://github.com/DocJade/fluster_rs # Go into the repository cd fluster_fs # Build with the recommended 'floppy' profile cargo build --profile floppy ``` #### Run Fluster!: ```bash # Example usage: # Create a directory to mount the filesystem mkdir ~/fluster_mount_point # Run the app (requires root privileges for mounting) sudo ./target/floppy/fluster_fs --block-device-path "/dev/sdX" --mount-point "~/fluster_mount_point" ``` - Replace /dev/sdX with the actual path to your floppy drive. #### Unmounting Fluster!: ```bash fusermount -u ~/fluster_mount_point ``` - Do note that unmounting Fluster! does not immediately shut down fluster, you will still need to swap disks to flush the cache to disk. ## Credits - [DocJade](https://docjade.com/) (That's me!) - [Rust](https://www.rust-lang.org/) ([Not the bad Rust](https://rust.facepunch.com/)) - [The Rippingtons](https://www.rippingtons.com/), who kept me from going insane while writing this. - [Femtanyl](https://femtanyl.bandcamp.com/), who helped me go insane while writing this. ## Notes: Originally I planned to keep an up-to-date implementation spec of Fluster! in `./filesystem_design`, but this slowly became more and more out of date, as is now very un-representative of the final product. Some information in there such as Inode blocks and how they are constructed should be mostly up to date. Dense disks aren't real, and they cannot hurt you. If you're wondering "Where is the EXE so I can just run it myself!" Please understand that there isn't one, and for good reason. If you don't know how to build and run this project yourself, chances are you would use it improperly, and possibly wipe your C: drive. ## See Fluster! in action [YouTube link](https://www.youtube.com/watch?v=cTPBGZcTRqo) ## You may also like... - Pornography, search "boobs" on google for more info. ## License Fluster! © 2025 by DocJade is licensed under Creative Commons Attribution 4.0 International --- > [I should just come up with a cool name and the rest will like as itself.](https://www.youtube.com/watch?v=dmzk5y_mrlg) ================================================ FILE: src/error_types/block.rs ================================================ // Blocks usually return similar types of errors. use thiserror::Error; #[derive(Debug, Error, PartialEq)] /// Errors related to block manipulation. Not disk level modification, but our custom block types. pub enum BlockManipulationError { #[error("Adding content to this block failed, due to the block not having enough capacity for the new content.")] OutOfRoom, #[error("This method can only be called on the final block in a chain of this type of block.")] NotFinalBlockInChain, #[error("The arguments given for this operation are out of bounds, or otherwise not supported.")] Impossible, #[error("The data that was attempted to be retrieved from this block did not exist.")] NotPresent } ================================================ FILE: src/error_types/conversions.rs ================================================ // Conversions between all of the lower types. // // Imports // use std::io::ErrorKind; use std::time::Duration; use log::debug; use log::error; use log::warn; use thiserror::Error; use crate::error_types::critical::CriticalError; use crate::error_types::drive::DriveError; use crate::error_types::drive::DriveIOError; use crate::error_types::drive::InvalidDriveReason; use crate::error_types::drive::WrappedIOError; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::tui::notify::NotifyTui; use crate::tui::tasks::TaskType; // Not error type can just be converted upwards willy-nilly, that led to the old // and horrible FloppyDiskError type which everything ended up returning. Not good. // We do not allow string errors. This is RUST damn it, not python! // We also have a custom conversion error type, so lower level callers can get more info // about what they need to do to be able to perform the cast to a higher error type. #[derive(Debug, Clone, Copy, Error, PartialEq)] /// Errors related to IO on the inserted floppy disk. pub enum CannotConvertError { #[error("You must retry this operation. If retrying repeatedly fails, throw a Critical error.")] MustRetry, } // // Drive errors // impl TryFrom for DriveError { type Error = CannotConvertError; fn try_from(value: DriveIOError) -> Result { match value { DriveIOError::Retry => { // Operation must be retried, cant cast that upwards. Err(CannotConvertError::MustRetry) }, } } } // // std::io:Error wrapping // impl WrappedIOError { pub(crate) fn wrap(io_error: std::io::Error, error_origin: DiskPointer) -> Self { WrappedIOError { io_error, error_origin, } } } // // WrappedIOError to DriveIOError // impl TryFrom for DriveIOError { type Error = CannotConvertError; fn try_from(value: WrappedIOError) -> Result { // Sleep for a tad just in case we're doing a retry std::thread::sleep(Duration::from_secs(1)); // Log where we were trying to do IO at when the error occurred. debug!("IO error occured while trying to access disk {} block {}", value.error_origin.disk, value.error_origin.block); debug!("Error type: {:#?}", value.io_error); match value.io_error.kind() { ErrorKind::NotFound => { // The floppy drive path is not there. CriticalError::DriveInaccessible(InvalidDriveReason::NotFound).handle(); // If handling worked, can retry. Err(CannotConvertError::MustRetry) }, ErrorKind::PermissionDenied => { // Dont have permission to perform IO on the drive. // Nothing we can do. CriticalError::DriveInaccessible(InvalidDriveReason::PermissionDenied).handle(); // If handling worked, can retry. Err(CannotConvertError::MustRetry) }, ErrorKind::ConnectionRefused | ErrorKind::ConnectionReset | ErrorKind::HostUnreachable | ErrorKind::NetworkUnreachable | ErrorKind::ConnectionAborted | ErrorKind::NotConnected | ErrorKind::AddrInUse | ErrorKind::AddrNotAvailable | ErrorKind::NetworkDown | ErrorKind::StaleNetworkFileHandle => { // Okay you should not be using fluster over the network dawg. // 100% your fault CriticalError::DriveInaccessible(InvalidDriveReason::Networking).handle(); // We cant recover from that unreachable!("Networked floppy drive??? Really??? gtfo"); }, ErrorKind::BrokenPipe => { // What // I doubt you could even make fluster start with pipes. unreachable!("Broken pipe with fluster, why are you using pipes in the first place???"); }, ErrorKind::AlreadyExists => { // Fluster does not create files during IO operations, only in backups. // Therefore this should not happen. // Especially since we always open the backups if they already exist. unreachable!("Fluster tried to create a file that already existed somehow. This should be impossible."); }, ErrorKind::WouldBlock => { // Fluster does not ask for blocking IO. // In theory this can just be retried. Err(CannotConvertError::MustRetry) }, ErrorKind::NotADirectory => { // This should never happen, since we always try to write to a file, not a directory. unreachable!("Fluster does not open directories, this is impossible."); }, ErrorKind::IsADirectory => { // User has passed in a directory for the floppy disk drive instead of a file for it. CriticalError::DriveInaccessible(InvalidDriveReason::NotAFile).handle(); // We cant recover from that, but pretend we can Err(CannotConvertError::MustRetry) }, ErrorKind::DirectoryNotEmpty => { // Fluster does not try to delete directories. unreachable!("Fluster does not delete directories, this should be impossible."); }, ErrorKind::ReadOnlyFilesystem => { // Cant use fluster on read-only floppy for obvious reasons. CriticalError::DriveInaccessible(InvalidDriveReason::ReadOnly).handle(); // If it was just the write-protect notch, we can recover. Err(CannotConvertError::MustRetry) }, ErrorKind::InvalidInput => { // The paramaters given for the IO action were bad, chances are, retrying this wont // do anything. We're cooked. // But hopefully this shouldn't happen because I'm epic sauce :D unreachable!("Invalid input parameters into IO action.") }, ErrorKind::InvalidData => { // See above, blah blah blah epic sauce unreachable!("Data not valid for the operation showed up in IO action.") }, ErrorKind::TimedOut => { // The IO took too long, we should be able to try again. Err(CannotConvertError::MustRetry) }, ErrorKind::WriteZero => { // Writing a complete bytestream failed. // Maybe the operation was canceled and needs to be retried? // Not sure if the floppy drive requires minimum write sizes, but 512 aught to be enough. // We dont cast this up, we make the caller retry the write. Err(CannotConvertError::MustRetry) }, ErrorKind::StorageFull => { // Fluster does not use a filesystem when doing writes to the disk. // Maybe this could happen when attempting to write past the end of the disk? // But we have bounds checking for that. warn!("Floppy drive claims to be full, we dont care."); Err(CannotConvertError::MustRetry) }, ErrorKind::NotSeekable => { // We must be able to seek files to read and write from them, this is a // configuration issue. CriticalError::DriveInaccessible(InvalidDriveReason::NotSeekable).handle(); Err(CannotConvertError::MustRetry) }, ErrorKind::QuotaExceeded => { // Not sure what other quotas other than size are possible, the man page // quota(1) doesn't specify any other quota types. // Plus, this shouldn't happen for raw IO, right? unreachable!("Floppy drives shouldn't have a quota."); }, ErrorKind::FileTooLarge => { // Fluster does not use an underlying filesystem. // Very funny since the biggest files we deal with are in the low MBs unreachable!("Somehow a write was too large, even though we dont use a filesystem directly."); }, ErrorKind::ResourceBusy => { // Disk is busy, we can retry though. // Force caller to retry. Err(CannotConvertError::MustRetry) }, ErrorKind::ExecutableFileBusy => { // If you're somehow running the floppy drive as an executable, // you have bigger issues. unreachable!("How are you running the floppy drive as an executable?"); }, ErrorKind::Deadlock => { // File locking deadlock, not much we can do here except try again. // Force caller to retry Err(CannotConvertError::MustRetry) }, ErrorKind::CrossesDevices => { // Fluster does not do renames on the floppy disk path. unreachable!("Fluster does not support rename the file paths, this should never happen."); }, ErrorKind::TooManyLinks => { // We do not create links. unreachable!("Fluster does not support links, no idea how we got here."); }, ErrorKind::InvalidFilename => { // The path to the disk is invalid somehow. CriticalError::DriveInaccessible(InvalidDriveReason::InvalidPath).handle(); // We cant recover from that, but in case we can, just try again. Err(CannotConvertError::MustRetry) }, ErrorKind::ArgumentListTooLong => { // Fluster does not call programs unreachable!("Fluster wasn't able to call an external program. Wait, we don't do that? Huh?"); }, ErrorKind::Interrupted => { // "Interrupted operations can typically be retried." // Force caller to retry Err(CannotConvertError::MustRetry) }, ErrorKind::Unsupported => { // Whatever operation we're trying to do, its not possible. // Not really much we can do here either. CriticalError::DriveInaccessible(InvalidDriveReason::UnsupportedOS).handle(); // We cant recover from that, so this will never be returned. Err(CannotConvertError::MustRetry) }, ErrorKind::UnexpectedEof => { // This would happen if we read past the end of the floppy disk, // which should be protected by guard conditions. // Maybe someone's trying to run fluster with 8" disks? // We'll just retry the operation, since this should be guarded anyways. // Force caller to retry Err(CannotConvertError::MustRetry) }, ErrorKind::OutOfMemory => { // Bro what // Nothing we can really do. panic!("Please visit https://downloadmoreram.com/ then re-run Fluster."); }, ErrorKind::Other => { // "This ErrorKind is not used by the standard library." // This is impossible to reach. unreachable!("Somehow got an `other` error kind, this is impossible as far as i can tell."); }, _ => { // This error is newer than the rust version fluster was originally written for. // GLHF! // Is the floppy drive empty? // code: 123, // message: "No medium found", if let Some(raw) = value.io_error.raw_os_error() && raw == 123_i32 { // No disk is in the drive. // This can happen even if there is a disk in the drive, so we keep // trying. debug!("Is no disk inserted?"); // Just keep retrying, if there is an issue with the floppy drive, we need to // eventually end up in the panic handler. // Show user that we're waiting for the drive to spin up // We wait 5 seconds. That's usually fast enough. let handle = NotifyTui::start_task(TaskType::WaitingForDriveSpinUp, 5*5); for _ in 0..5*5 { NotifyTui::complete_task_step(&handle); std::thread::sleep(Duration::from_millis(100)); } NotifyTui::finish_task(handle); return Err(CannotConvertError::MustRetry) } // Well, we'll just pretend we can retry any unknown error... warn!("UNKNOWN ERROR KIND:"); warn!("{value:#?}"); warn!("Ignoring, pretending we can retry..."); Ok(DriveIOError::Retry) }, } } } ================================================ FILE: src/error_types/critical.rs ================================================ // Critical errors are errors that we cannot recover from without some sort of higher intervention. // Returning this error type means you've done all you possibly can, and need saving at a higher level, or // we are in a unrecoverable state. use std::{ fs::OpenOptions, os::unix::fs::FileExt, path::PathBuf, process::exit }; use thiserror::Error; use log::error; use crate::{ error_types::drive::InvalidDriveReason, filesystem::{ disk_backup::restore::restore_disk, filesystem_struct::FLOPPY_PATH }, tui::prompts::TuiPrompt }; #[derive(Debug, Clone, Copy, Error, PartialEq)] /// Use this error type if an error happens that you are unable to /// recover from without intervention. /// /// Creating critical errors is a last resort. Whatever error that was causing /// your failure must be passed in. pub enum CriticalError { #[error("The floppy drive is inaccessible for some reason.")] DriveInaccessible(InvalidDriveReason), #[error("We've retried an operation too many times. Something must be wrong.")] OutOfRetries(RetryCapError) // Keep track of where we ran out of retries. } #[derive(Debug, Clone, Copy, PartialEq)] /// When you run out of retries on an operation, its useful to know what kind of issue was occurring. pub enum RetryCapError { /// Opening the disk is repeatedly failing OpenDisk, /// Attempting to write a block is repeatedly failing. WriteBlock, /// Attempting to read a block is repeatedly failing. ReadBlock, } // // ========= // Attempt to recover // ========= // impl CriticalError { /// Try to recover from a critical error. /// /// Returns nothing, since if recovery fails, fluster has shut down. /// If this function completes successfully, you can re-attempt the operation that resulted in the critical error. /// This should only be called once per operation, if you are consistently calling attempt_recovery, there is a deeper /// issue that you must address. pub(crate) fn handle(self) { go_handle_critical(self) } } fn go_handle_critical(error: CriticalError) { // Critical recovery is not allowed in tests. if cfg!(test) { panic!("Tried to recover from a critical error! {error:#?}"); } let mitgated = match error { CriticalError::DriveInaccessible(invalid_drive_reason) => handle_drive_inaccessible(invalid_drive_reason), CriticalError::OutOfRetries(reason) => handle_out_of_retries(reason), }; // If that worked, the caller that caused this critical to be thrown should be able to // complete whatever operation they need. if mitgated { return } // None of that worked. We must give up. // .o7 println!("Critical error recovery has failed."); println!("{error:#?}"); // Disk/drive are in unknown state, but by god we still must try to flush via panicking. panic!("Fluster! has encountered an unrecoverable error, and must shut down.\nGoodbye."); } // // Sub-type handlers // // Returns true if mitigation succeeded. fn handle_drive_inaccessible(reason: InvalidDriveReason) -> bool { match reason { InvalidDriveReason::NotAFile => { // A non-file cannot be used as a floppy disk inform_improper_floppy_drive() }, InvalidDriveReason::PermissionDenied => { // Need to be able to do IO obviously. inform_improper_floppy_drive() }, InvalidDriveReason::Networking => { // Cant use network drives inform_improper_floppy_drive() }, InvalidDriveReason::ReadOnly => { // Did they leave the write protect notch on? inform_improper_floppy_drive() }, InvalidDriveReason::NotSeekable => { // Floppy drives must be seekable. inform_improper_floppy_drive() }, InvalidDriveReason::InvalidPath => { // Need to be able to get to the drive inform_improper_floppy_drive() }, InvalidDriveReason::UnsupportedOS => { // Homebrew OS maybe? We don't use that many // file operations, certainly not many unusual ones, thus // this shouldn't happen on normal platforms. error!("Simple file-based IO is marked as unsupported by your operating system."); error!("I'm assuming you're using a non-standard Rust build target / OS destination."); error!("Obviously I cannot support that. If you really want to use Fluster (why?), you'll have to"); error!("update Fluster to make it compatible with your system/setup. Good luck!"); // We shouldn't've even finished a single write yet, there shouldn't be anything to flush. exit(-1); }, InvalidDriveReason::NotFound => { // Maybe the drive is tweaking? // Ask the user if they wanna do troubleshooting. loop { let response = TuiPrompt::prompt_input( "Floppy drive error.".to_string(), "The floppy drive was not found, would you like to retry, or start troubleshooting?\n (R)etry / (T)roubleshoot".to_string(), true ); if response.starts_with('r') { // User just wants to the retry. // retrun true, since we've "done all we can" return true; } else if response.starts_with('t') { // Since the drive is not found, we will first return troubleshooter(); } } }, } } /// Returns true if mitigation succeeded. /// /// yes this is the same as the other handler, but whatever fn handle_out_of_retries(reason:RetryCapError) -> bool { match reason { RetryCapError::OpenDisk => { // Run the troubleshooter troubleshooter() }, RetryCapError::WriteBlock => troubleshooter(), RetryCapError::ReadBlock => troubleshooter(), } } // // User guided troubleshooting // /// Returns true if we were able to pinpoint the issue and resolve it. fn troubleshooter() -> bool { // Inform the user that the troubleshooter is running. println!("Fluster is troubleshooting, please wait..."); // Do the easiest things first, preferably ones that do not involve interaction. // Run the disk checker. // If that passes, we now know: // - Every block on the disk is readable // - Every block on the disk is writable. // - The drive is connected properly and is working. // If all of that is working, troubleshooting is done, since we did not find any issues. // But this is suspicious. Why did the troubleshooter get called when everything is working? if check_disk() { TuiPrompt::prompt_enter( "Strange...".to_string(), "Troubleshooter unexpectedly found nothing wrong.\n Suggestion: You should cancel all file operations and unmount Fluster to flush everything to disk, just in case.\n If you are already in the process of unmounting, good luck!".to_string(), true ); return true; } // Something is wrong with the disk or the drive. We will now walk through the // fastest and easiest options first. // Ask the user to re-seat the disk. TuiPrompt::prompt_enter( "Troubleshooting: Re-seat floppy.".to_string(), "Please eject the floppy disk, then re-insert it.\n If the disk is currently spinning, please wait a moment to see if it will stop spinning before performing the ejection. If the disk continues to spin regardless, proceed with ejection.".to_string(), true ); // Maybe re-seating was all we needed? if check_disk() { // Neat. troubleshooter_finished(); return true } // Now we know for sure that either the disk is dead, or the drive is not working. // Let's try another disk, that'll let us narrow it down if the disk was bad. TuiPrompt::prompt_enter( "Troubleshooting: Different disk.".to_string(), "Please swap disks to any known good disk. Remember which disk was removed.".to_string(), true ); // Run the check then have the user put the possibly bad disk back in for continuity. let disk_bad = check_disk(); TuiPrompt::prompt_enter( "Troubleshooting: Return disk.".to_string(), "Please swap back to the disk you previously removed.".to_string(), true ); // Now, if the known good disk passed the disk check, we know that it's the drive that is having issues. // Otherwise, the disk is bad. if disk_bad { // Bummer, we need to replace this disk. do_disk_restore(); // Disk has been restored. return true; } // Disk wasn't bad, so the drive must be the issue. // Try un-plugging and plugging it back in lmao. loop { do_remount(); if check_disk() { // Remounting fixed it. troubleshooter_finished(); return true; }; // That failed. Retry? let prompted = TuiPrompt::prompt_input( "Troubleshooting: Try again?".to_string(), "Check disk is still failing.\n This is our last troubleshooting step before completely giving up.\n You can also forcibly restore a disk if needed.\n Would you like to try re-mounting again, or throw in the towel?\n (Y)es/(D)isk restore/(G)ive up".to_string(), true ); if prompted.to_ascii_lowercase().contains('g') { // user gives up. break } else if prompted.to_ascii_lowercase().contains('d') { do_disk_restore(); } } // Remounting did not work, and the user has given up. TuiPrompt::prompt_enter( "Troubleshooting failed. :(".to_string(), "Troubleshooting has failed. No fix that was attempted worked.\n All of the disks are backed up to the backup directory. No data should be lost, although there might be partially written data.\n Worst comes to worst, you can re-image all of your disks from backups.\n Before restoring those disks though, make sure to back-up the backups, since they might be slightly corrupt.".to_string(), false ); false } // // Troubleshooting actions // /// Read every block on the disk to determine if the disk is bad. /// /// This may take a while. /// /// Returns true if every block was read and written correctly. fn check_disk() -> bool { println!("Checking if the disk and drive are working..."); // Just loop over all of the blocks and try reading them. // We need to do it manually ourselves since we dont want // to throw another critical error while handling another one. // Open the disk currently in the drive. // If we cannot lock, we obviously cant use the drive. Thus returns false. let disk_path = if let Ok(guard) = FLOPPY_PATH.try_lock() { guard.clone() } else { // The lock is poisoned, which means we died somewhere else. // So since we're already in the troubleshooter, we'll just clear the lock :clueless: then // return false. FLOPPY_PATH.clear_poison(); return false }; // Read the entire thing in one go println!("Open floppy drive..."); let disk_file = match OpenOptions::new().read(true).write(true).open(&disk_path) { Ok(ok) => ok, Err(error) => { // There is something wrong with reading in the drive, which would imply that // the drive is inaccessible or something. We cannot resolve here. println!("Failed to open drive."); println!("{error:#?}"); return false; }, }; println!("Ok."); // Now read in the entire disk. println!("Reading entire disk..."); let mut whole_disk: Vec = vec![0; 512*2880]; let _ = disk_file.sync_all(); let read_result = disk_file.read_exact_at(&mut whole_disk, 0); let _ = disk_file.sync_all(); // If that failed at all, checking the disk is bad either due to the drive, or the disk. if let Err(error) = read_result { // Read failed. Something is up. println!("Fail."); println!("{error:#?}"); return false; }; println!("Ok."); // Now we write the entire disk back again to see if every block accepts writes. println!("Writing entire disk..."); let _ = disk_file.sync_all(); let write_result = disk_file.write_all_at(&whole_disk, 0); let _ = disk_file.sync_all(); // Did the write work? if let Err(error) = write_result { // nope println!("Fail."); println!("{error:#?}"); return false; }; println!("Ok."); println!("Disk and drive appear to be working correctly."); true } /// Some actions might change the path to the floppy disk drive, we need to let the user update that /// if they need. fn update_drive_path() { // what's the new path let new_path: std::path::PathBuf; loop { let possible = TuiPrompt::prompt_input( "Troubleshooting: Path change.".to_string(), "If the path to the floppy drive has changed due to the re-mount, please enter the new path. Otherwise hit enter.".to_string(), false ); // If nothing was entered, the path has not changed if possible.is_empty() { // no change. return } let could_be = PathBuf::from(possible); let maybe = match could_be.canonicalize() { Ok(ok) => ok, Err(err) => { // what TuiPrompt::prompt_enter( "Invalid path.".to_string(), format!("Unable to canonicalize path. Please provide a valid path.\n\n{err:#?}"), false ); continue; }, }; if std::fs::exists(&maybe).unwrap_or(false) { // Good. new_path = maybe; break } else { TuiPrompt::prompt_enter( "Invalid path.".to_string(), "Unable to either open path, or confirm it exists. Please provide a valid path.".to_string(), false ); continue; } } // Set that new path if let Ok(mut something) = FLOPPY_PATH.try_lock() { *something = new_path; } else { // The lock is poisoned, we'll just uhhh... ignore that. // The woes of a non-transactional filesystem. // TODO: kill PastJade for not thinking that far ahead. FLOPPY_PATH.clear_poison(); if let Ok(mut something_the_sequel) = FLOPPY_PATH.try_lock() { *something_the_sequel = new_path; } else { // ???? // Already VERY cooked, might as well panic for fun! panic!("Tried to clear poison on the floppy path but that didn't work! Giving up."); } } } // // User actions // /// Ask the user to remount the floppy drive. fn do_remount() { TuiPrompt::prompt_enter( "Troubleshooting: Remount drive.".to_string(), "Please re-mount the floppy drive.\n You can find more information about remounting in the README.\n Press enter after you have finished re-mounting the drive.".to_string(), false ); // This might have changed the path to the floppy drive. update_drive_path(); } /// Inform the user that the disk needs to be re-created. /// /// Make sure to put the bad disk back in the drive beforehand so the user /// knows what disk to discard. fn do_disk_restore() { TuiPrompt::prompt_enter( "Troubleshooting: Bad disk.".to_string(), "The troubleshooter has determined that the disk currently within the drive is bad.\n This disk will need to be re-created.".to_string(), false ); // Now start the restore. let mut failure = false; loop { if failure { // We tried restoring the disk, but the restore failed. // Tell the user and ask if they want to attempt to restore to // the same disk again, or have them put in a new disk. TuiPrompt::prompt_enter( "Restoration: Failure.".to_string(), "Restoring disk has failed. Restoring can be retried though.\n If you would like to attempt restoring to the same disk that you inserted previously, leave it in the drive, and ignore the message about swapping disks.\n You can re-try as many times as you would like, but if the new disk continues to fail, you should try using another disk to restore onto.\n If you just cannot seem to restore to a new disk, idk man you're cooked lmao good luck bozo.".to_string(), false ); } // Pull out the bad one, disk restore needs an empty drive. // We need to know what disk it was. let disk_number: u16; loop { let to_convert = TuiPrompt::prompt_input( "Restoration: New disk.".to_string(), "Please remove the bad disk currently inserted in the drive, then enter it's disk number.".to_string(), false ); if let Ok(number) = to_convert.parse::() { disk_number = number; break; } TuiPrompt::prompt_enter( "Restoration: Bad number.".to_string(), "Parsing error, please try again. Only enter the number of the disk.".to_string(), false ); } // Now restore that disk. if restore_disk(disk_number) { // restore worked! return } // Restoration failed, it can be retried. failure = true; } } // // User information // // fn inform_improper_mount_point() -> ! { // TuiPrompt::prompt_enter( // "Bad mount point.".to_string(), // "The point where you have tried to mount fluster is invalid for some reason.\n // Please re-confirm that the mount point is valid, then re-run fluster. Good luck!".to_string(), // true // ); // exit(-1) // Exiting, no floppy drive to flush with. // } fn inform_improper_floppy_drive() -> bool { // We cannot use this floppy drive. // First check if the user has inserted a write-protected disk loop { let prompted: String = TuiPrompt::prompt_input( "Troubleshooting: Write protected.".to_string(), "Please remove the floppy disk from the drive, and confirm that it is not set to read-only.\n\n Was the disk set to read-only? \"yes\"/\"no\"".to_string(), false ); if prompted.contains('y') { // Whoops! TuiPrompt::prompt_enter( "Troubleshooting finished.".to_string(), "Cool! That means we do not need to shut down. Please set the disk to read/write, and insert it back into the drive.\n Please make sure you do not insert write protected disks in the future, or the troubleshooter will start again.".to_string(), false ); return true; } else if prompted.contains('n') { // Well crap. break }; } // Disk was not write protected. Drive is bad. TuiPrompt::prompt_enter( "Troubleshooting failed.".to_string(), "Fluster is unable to access the floppy disk from your floppy drive.\n Please make sure that the path you provided for the drive is:\n - Valid\n - A file, and not a directory\n - Accessible by your current user\n - Is not over the network\n - Is not mounted as read-only\n\n Fluster will now exit, since operating without a drive is not possible.".to_string(), true ); exit(-1); // Exiting, since we cant even flush the cache due to no floppy drive. } // Helper just do dedupe fn troubleshooter_finished() { TuiPrompt::prompt_enter( "Troubleshooting succeeded!".to_string(), "Troubleshooting finished successfully.".to_string(), false ); } ================================================ FILE: src/error_types/drive.rs ================================================ // Error types pertaining to the floppy drive itself. // We do not allow string errors. This is RUST damn it, not python! use thiserror::Error; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; #[derive(Debug, Error, PartialEq)] /// Super-error about the floppy drive itself. /// /// We are unable to handle read errors at this level. All IO related errors /// are within the DriveIOError type. pub enum DriveError { #[error("No disk is currently inserted.")] DriveEmpty, #[error("The operation failed for non-critical reasons, but no corruption occurred, and the operation can be retried with the same arguments.")] Retry, #[error("An operation on this disk is taking too long..")] TakingTooLong } #[derive(Debug, Clone, Copy, Error, PartialEq)] /// Errors related to IO on the inserted floppy disk. /// /// This is only used at the lowest of levels on actual IO operations. pub enum DriveIOError { #[error("The operation failed for non-critical reasons, but no corruption occurred, and the operation can be retried with the same arguments.")] Retry, } #[derive(Debug)] pub struct WrappedIOError { /// The io error that you are trying to handle pub(super) io_error: std::io::Error, /// The DiskPointer to where this issue occurred. pub(super) error_origin: DiskPointer } #[derive(Debug, PartialEq, Clone, Copy)] /// Reasons why we cannot use the provided floppy disk path pub enum InvalidDriveReason { /// Pointed at a folder instead of a file. NotAFile, /// We dont have permission to access the path provided PermissionDenied, /// We do not support using fluster over the network. Networking, /// Disk must be read and write. ReadOnly, /// File that refers to the floppy drive is not seekable. NotSeekable, /// The path is invalid in some way. InvalidPath, /// The filesystem (or operating system) that you're running fluster on /// does not support basic disk IO. UnsupportedOS, /// Generic "not found" NotFound } ================================================ FILE: src/error_types/filesystem.rs ================================================ use libc::c_int; use log::error; use crate::error_types::drive::DriveError; // // // ====== // C Error values // ====== // // // Errors gleamed from // https://man7.org/linux/man-pages/man3/errno.3.html // https://man7.org/linux/man-pages/man2/openat.2.html /// Bro thinks he's Shakespeare. pub(in super::super) const FILE_NAME_TOO_LONG: c_int = libc::ENAMETOOLONG; /// Tried to modify a non-empty directory in a way that required it to be empty. pub(in super::super) const DIRECTORY_NOT_EMPTY: c_int = libc::ENOTEMPTY; /// This seat's taken. pub(in super::super) const ITEM_ALREADY_EXISTS: c_int = libc::EEXIST; /// Tried to do directory stuff to a file. pub(in super::super) const NOT_A_DIRECTORY: c_int = libc::ENOTDIR; /// Ad hominem pub(in super::super) const INVALID_ARGUMENT: c_int = libc::EINVAL; /// Tried to do things to a directory that it does not support. pub(in super::super) const IS_A_DIRECTORY: c_int = libc::EISDIR; // /// Function not implemented. // pub(in super::super) const UNIMPLEMENTED: c_int = libc::ENOSYS; /// This operation is not supported in this filesystem. // pub(in super::super) const UNSUPPORTED: c_int = libc::ENOTSUP; /// Access denied / files does not exist. pub(in super::super) const NO_SUCH_ITEM: c_int = libc::ENOENT; /// Tried to seek to an invalid file position. // pub(in super::super) const INVALID_SEEK: c_int = libc::ESPIPE; /// Tried to use a filehandle that is stale. New one is required. pub(in super::super) const STALE_HANDLE: c_int = libc::ESTALE; // Generic IO error. The dreaded OS(5) Input/Output error. pub(in super::super) const GENERIC_FAILURE: c_int = libc::EIO; /// You are insane. // pub(in super::super) const FILE_TOO_BIG: c_int = libc::EFBIG; /// Operation was interrupted for some reason, but can be retried. pub(in super::super) const TRY_AGAIN: c_int = libc::ERESTART; /// Device / filesystem is busy, try again later. /// /// Should never happen in fluster due to being single threaded. pub(in super::super) const BUSY: c_int = libc::EBUSY; impl From for c_int { fn from(value: DriveError) -> Self { match value { DriveError::DriveEmpty => { // The drive empty error should never get this high error!("Drive empty error should never make it to the filesystem level!"); error!("Telling file system that we are busy..."); BUSY }, DriveError::Retry => TRY_AGAIN, DriveError::TakingTooLong => BUSY, } } } ================================================ FILE: src/error_types/header.rs ================================================ // Errors for header conversions. use thiserror::Error; #[derive(Debug, Error, PartialEq)] /// Errors related to block manipulation. Not disk level modification, but our custom block types. pub enum HeaderError { #[error("This is not a header of the requested type.")] Invalid, #[error("The block that was requested to turn into a header is completely blank.")] Blank, } ================================================ FILE: src/error_types/mod.rs ================================================ pub(crate) mod drive; pub(crate) mod conversions; pub(crate) mod block; pub(crate) mod filesystem; pub(crate) mod critical; pub(crate) mod header; ================================================ FILE: src/filesystem/disk_backup/mod.rs ================================================ pub mod restore; pub mod update; ================================================ FILE: src/filesystem/disk_backup/restore.rs ================================================ // Restore a disk from a backup. use std::{fs::File, io::{ Read, Seek }, os::unix::fs::FileExt}; use log::{debug, error, warn}; use crate::{filesystem::filesystem_struct::FLOPPY_PATH, tui::{notify::NotifyTui, prompts::TuiPrompt, tasks::TaskType}}; /// Returns true if the entire disk was re-created successfully. /// /// Assumes the drive is empty when called. pub fn restore_disk(number: u16) -> bool { debug!("Beginning restore of disk `{number}`."); // Get a new blank disk. TuiPrompt::prompt_enter( "Insert blank disk.".to_string(), format!("Please insert a brand new, blank disk that will become the new disk {number}, then press enter.\n WARNING: Disk will NOT be checked for blankness, this WILL destroy data if a non-blank disk is inserted!"), false ); let handle = NotifyTui::start_task(TaskType::RestoreDisk, 6); // Find the disk in the backup folder // If it't not in there, you're cooked. // Try opening the backup file at most 5 times. let mut tries = 1_u8; let mut backed_up: std::fs::File; debug!("Opening backup file..."); loop { match std::fs::OpenOptions::new() .read(true) .open(format!("/var/fluster/disk_{number}.fluster_backup")) { Ok(ok) => { backed_up = ok; break; }, Err(_) => { if tries == 5 { // ruh roh warn!("Fail. Out of retries."); return false; } else { warn!("Fail, trying again..."); tries += 1; continue; } }, }; }; NotifyTui::complete_task_step(&handle); // Now read in the entire floppy backup. // Again, at most 5 tries. let mut bytes: Vec = Vec::with_capacity(2880*512); let mut tries = 1_u8; debug!("Reading backup data..."); loop { if backed_up.rewind().is_ok() && backed_up.read_to_end(&mut bytes).is_ok() { // All good. break } if tries == 5 { // cooked. error!("Fail. Out of retries."); return false; } debug!("Fail, trying again..."); bytes.clear(); tries += 1; continue; }; NotifyTui::complete_task_step(&handle); // Copy the entire contents of that backup to the new disk. // We'll just dump the entire file to the block device without using our floppy handler, // since we cant really trust my logic hehe // Get the path to the floppy drive block device. // We'll pre-clear poison, just in case. FLOPPY_PATH.clear_poison(); let block_path = if let Ok(path) = FLOPPY_PATH.lock() { path.clone() } else { // well... cooked error!("Floppy path is poisoned!"); return false }; NotifyTui::complete_task_step(&handle); // Open the block device as a file let block_file: File = if let Ok(opened) = File::options().read(true).write(true).open(block_path) { opened } else { // Well, we couldn't open the floppy path. Return false. return false; }; NotifyTui::complete_task_step(&handle); // Now write all that in let mut write_worked = false; for _ in 0..5 { let result = block_file.write_all_at(&bytes, 0); match result { Ok(_) => { // write finished! write_worked = true; break }, Err(err) => { // That didn't work! error!("Writing to the drive failed!"); error!("{err:#?}"); continue; }, } } NotifyTui::complete_task_step(&handle); // Now sync the disk to pause until the data all actually hits the disk. // Not a big issue if this doesnt work, we're still gonna wait for the disk to spin down // before swapping disks. let _ = block_file.sync_all(); NotifyTui::complete_task_step(&handle); // Did that work? if write_worked { // Disk written! NotifyTui::finish_task(handle); debug!("Disk restored!"); true } else { // Well shoot. NotifyTui::cancel_task(handle); error!("Disk restore failed!"); false } } ================================================ FILE: src/filesystem/disk_backup/update.rs ================================================ // Update the backup disk with new contents. // The path that the disk backups go in /var/fluster // Porting fluster? Then you've gotta update this for sure. use std::os::unix::fs::FileExt; use log::error; use crate::{filesystem::filesystem_struct::WRITE_BACKUPS, pool::disk::generic::{block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer}}; pub(crate) fn update_backup(block: &RawBlock) { // Ignore backups if needed. if let Some(ian_the_bool) = WRITE_BACKUPS.get() { if !ian_the_bool { // Skip, backups are disabled. return } } else { // The backups flag hasn't been set up, which should be impossible, but we'll just return return } // Make the backup folder if it does not exist yet if std::fs::create_dir_all("/var/fluster").is_err() { // Unable to create the folders and such. error!("Fluster needs to be able to create/use /var/fluster for disk backups."); error!("We cannot continue without backups. Shutting down. If you are unable to use"); error!("backups, set the flag."); panic!("Unable to update backups!"); // we panic here, since we still want to flush the disks. } // Open or create the backup file for the disk let disk_path: String = format!("/var/fluster/disk_{}.fluster_backup", block.block_origin.disk); let backup_file = if let Ok(file) = std::fs::OpenOptions::new().create(true).truncate(false).write(true).open(disk_path) { file } else { // Cannot open the backup file, we're cooked. error!("Fluster was unable to create or open one of its backup files, if you see this spamming your logs, its probably chronic."); error!("Fix your permissions, or backups wont work!"); // pretend we did the backup, crashing is worse. return }; // Write in that block. We will try twice at maximum. for _ in 0..2 { if backup_file.write_all_at(&block.data, block.block_origin.block as u64 * 512).is_err() { // That did not work. Crap. // We'll try again. } else { // That worked! return } } // We couldn't update the file. error!("Fluster failed to write to a backup for one of the disks, if you see this spamming your logs, its probably chronic."); error!("You should investigate!"); } pub(crate) fn large_update_backup(start: DiskPointer, data: &[u8]) { // Ignore backups if needed. if let Some(ian_the_bool) = WRITE_BACKUPS.get() { if !ian_the_bool { // Skip, backups are disabled. return } } else { // The backups flag hasn't been set up, which should be impossible, but we'll just return return } // Open or create the backup file for the disk let disk_path: String = format!("/var/fluster/disk_{}.fluster_backup", start.disk); let backup_file = if let Ok(file) = std::fs::OpenOptions::new().create(true).truncate(false).write(true).open(disk_path) { file } else { // Cannot open the backup file, we're cooked. error!("Fluster was unable to create or open one of its backup files, if you see this spamming your logs, its probably chronic."); error!("Fix your permissions, or backups wont work!"); // pretend we did the backup, crashing is worse. return }; // Write in that block. We will try twice at maximum. for _ in 0..2 { if backup_file.write_all_at(data, start.block as u64 * 512).is_err() { // That did not work. Crap. // We'll try again. } else { // That worked! return } } // We couldn't update the file. error!("Fluster failed to write to a backup for one of the disks, if you see this spamming your logs, its probably chronic."); error!("You should investigate!"); } ================================================ FILE: src/filesystem/file_attributes/conversion.rs ================================================ use fuse_mt::{FileAttr, FileType}; use libc::c_int; use log::debug; use std::time::SystemTime; use crate::{ error_types::drive::DriveError, filesystem::file_handle::file_handle_struct::FileHandle, pool::disk::standard_disk::block::directory::directory_struct::{ DirectoryItem, DirectoryItemFlags }, tui::{notify::NotifyTui, tasks::TaskType} }; // Take in a file handle and spit out its attributes. impl TryFrom for FileAttr { type Error = c_int; fn try_from(value: FileHandle) -> Result { debug!("Retrieving file metadata from handle..."); // Get the directory item let item: DirectoryItem = value.get_directory_item()?; Ok(go_get_metadata(item)?) } } // You can also call this on DirectoryItem impl TryFrom for FileAttr { type Error = DriveError; fn try_from(value: DirectoryItem) -> Result { go_get_metadata(value) } } fn go_get_metadata(item: DirectoryItem) -> Result { debug!("Extracting metadata from item `{}`...", item.name); let handle = NotifyTui::start_task(TaskType::GetMetadata(item.name.clone()), 3); // Now for ease of implementation, we (very stupidly) ignore all file access permissions, // owner information, and group owner information. // Root owns all files (user id 0) // Owner is in the superuser group (group id 0) // All permission bits are set (very scary!) go execute a jpeg, i dont even care anymore. // Due to this, we also do not check any permissions on reads or writes! :D // How big is it debug!("Getting size..."); let size: u64 = item.get_size()?; NotifyTui::complete_task_step(&handle); // extract the times debug!("Created at..."); let creation_time: SystemTime = item.get_created_time()?.into(); debug!("Modified at..."); let modified_time: SystemTime = item.get_modified_time()?.into(); NotifyTui::complete_task_step(&handle); // "What kind of item is this?" // https://www.tiktok.com/@ki2myyysc6/video/7524954406438161694 let file_kind: FileType = if item.flags.contains(DirectoryItemFlags::IsDirectory) { // "This is a directory, used for holding items in a filesystem, such as files or other directories." debug!("Is a directory..."); FileType::Directory } else { // "This is a file, used to store arbitrary data, it is very useful!" debug!("Is a file..."); FileType::RegularFile }; NotifyTui::complete_task_step(&handle); debug!("Metadata done."); NotifyTui::finish_task(handle); // Put it all together Ok(FileAttr { // Size of item in bytes. size, // Bytes div_ceil 512 blocks: size.div_ceil(512), // We dont support access times. atime: SystemTime::UNIX_EPOCH, // modification time mtime: modified_time, // metadata change, not supported ctime: SystemTime::UNIX_EPOCH, // creation time crtime: creation_time, // file type kind: file_kind, // File permissions, not supported perm: 0o777, // All permission bits // links not supported nlink: 2, // This has to be set to 2 or things get angry, idk. // owner id, always root uid: 0, // owner group, always root gid: 0, // special id, not supported rdev: 0, // macos flags, who gaf? not me. use a real operating system /bait flags: 0, }) } ================================================ FILE: src/filesystem/file_attributes/mod.rs ================================================ pub mod conversion; ================================================ FILE: src/filesystem/file_handle/file_handle_methods.rs ================================================ // Make the handle do things. use std::{collections::HashMap, sync::{Arc, Mutex}}; use lazy_static::lazy_static; use libc::c_int; use log::{debug, error}; // // Global info about open files // struct LoveHandles { /// Hashmap of the currently allocated handles allocated: HashMap, /// Highest allocated number (is kept up to date internally) highest: u64, /// Recently freed handles (ie open space in the hashmap) free: Vec } impl LoveHandles { /// Make a new one, should only be called once. fn new() -> Self { // Empty LoveHandles { allocated: HashMap::new(), highest: 0, free: Vec::new(), } } /// Make a new handle fn make_handle(&mut self, item: FileHandle) -> u64 { // Get a number let num = self.next_free(); // Put it in the hashmap. // We also assert that we have not already used this number. assert!(self.allocated.insert(num, item).is_none(), "We already used this handle number, even though we thought it was free!"); // All done. num } /// Get the handle back fn read_handle(&self, number: u64) -> FileHandle { // Handles are not read after freeing, doing so is undefined behavior. if let Some(handle) = self.allocated.get(&number) { // Cool, it's there. handle.clone() } else { // We are cooked. error!("Tried to read a handle that was not allocated!"); panic!("Use after free on handle."); } } /// Get the next free handle (internal abstraction) fn next_free(&mut self) -> u64 { // Prefer vec items if self.free.is_empty() { // Time for a new number then. let give = self.highest; self.highest += 1; return give; } // There is a vec item. self.free.pop().expect("Guarded.") } /// You need to let go... fn release_handle(&mut self, number: u64) { // Handles are only ever freed once. Freeing an empty handle is undefined behavior, thus we // cant do anything but give up. if self.allocated.remove(&number).is_none() { // Bad! error!("Tried to free a handle that was not allocated!"); panic!("Double free on handle."); }; // Is this number right below the current highest? if number == self.highest - 1 { // Yep! Reduce highest. self.highest -= 1; } } } lazy_static! { static ref LOANED_HANDLES: Arc> = Arc::new(Mutex::new(LoveHandles::new())); } // // The actual handles // use crate::{ error_types::filesystem::*, filesystem::file_handle::file_handle_struct::FileHandle, pool::disk::{ standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryItem }, io::directory::types::NamedItem } } }; impl FileHandle { /// The name of the file/folder, if it exists. /// This will return None on the root. pub fn name(&self) -> &str { // Get the name, if it exists. if let Some(name) = self.path.file_name() { name.to_str().expect("Should be valid UTF8") } else { // No name, this must be the root. "" } } /// Allocate the file handle for tracking. /// /// Will block. /// /// Does not create a new ItemHandle, only stores it. pub fn allocate(self) -> u64 { // This is blocking. let read_handles = &mut LOANED_HANDLES.lock().expect("Other mutex holders should not panic."); // Add it read_handles.make_handle(self) } /// Get contents of handle. /// /// Will block. pub fn read(handle: u64) -> Self { // This is blocking let read_handles = LOANED_HANDLES.lock().expect("Other mutex holders should not panic."); read_handles.read_handle(handle) } /// Release a handle. /// /// Will block. pub fn drop_handle(handle: u64) { // This is blocking let read_handles = &mut LOANED_HANDLES.lock().expect("Other mutex holders should not panic."); read_handles.release_handle(handle); } /// Check if this handle is a file or a directory by attempting to read it from disk, otherwise /// deducing the type from it's path string. pub fn is_file(&self) -> Result, c_int> { // Annoyingly, rust's PathBuf type doesn't have a way to test if itself is a directory // without reading from disk, which makes it completely useless for deducing if the passed argument // is a file or folder. Very very annoying. // // You can't just check for file extensions, since files do not _need_ an extension... // // The approach i'll take is to see if the path ends with a delimiter. good luck lmao // But before we fallback to the path based deduction, we can attempt to load the file from disk if it exists. // If it does, we have our sure answer about what type it is, otherwise we will use the crappy string logic. // I don't particularly enjoy needing to access the disk here, but chances are if you're trying to find what type // something is, you'll be modifying it soon anyways. let name: String = self.name().to_string(); debug!("Attempting to deduce if `{}` is a file or directory...", self.path.display()); // Does the parent exist? debug!("Checking if it already exists..."); if let Some(parent) = DirectoryBlock::try_find_directory(self.path.parent())? { // Parent does exist, is this item there in either form? let file: NamedItem = NamedItem::File(name.clone()); let directory: NamedItem = NamedItem::Directory(name.clone()); let maybe_file = parent.find_item(&file)?; if maybe_file.is_some() { // It was a file debug!("Yes, and it's a file."); return Ok(Some(true)); } let maybe_directory = parent.find_item(&directory)?; if maybe_directory.is_some() { // It was a directory. debug!("Yes, and it's a directory."); return Ok(Some(false)); } } debug!("Item did not exist!"); // Rather than guess, we'll just return that the file did not exist, which should not the be the case for // file handles, but maybe the caller just had a stale handle, or is spoofing a new file? Ok(None) } /// Loads in and returns the directory item if it exists. pub fn get_directory_item(&self) -> Result { // Open the containing folder let block = match DirectoryBlock::try_find_directory(self.path.parent())? { Some(ok) => ok, None => { // Containing block did not exist. return Err(NO_SUCH_ITEM); }, }; let named_item = if let Some(bool) = self.get_named_item()? { // There was an item with this name. bool } else { // We are trying to deduce the name of an item that does not exist. return Err(NO_SUCH_ITEM); }; // Find the item if let Some(exists) = block.find_item(&named_item)? { // File existed. Ok(exists) } else { // No such item. Err(NO_SUCH_ITEM) } } /// Get a named item from this handle. pub(crate) fn get_named_item(&self) -> Result, c_int> { // Get a name let name: String = self.name().to_string(); let file_check = self.is_file()?; // If this is none, the caller is trying to extract a named item from // an invalid handle or a spoofed file. We return None, since the item did not // exist in either form. if let Some(is_file) = file_check { // An item was there. // Deduce the type if is_file { // yeah its a file return Ok(Some(NamedItem::File(name))); } else { // dir return Ok(Some(NamedItem::Directory(name))); } } // There was no item. Ok(None) } } ================================================ FILE: src/filesystem/file_handle/file_handle_struct.rs ================================================ // // // ====== // Handle type // ====== // // // We are in charge of our own file handle management. Fun! (lie) // So we need a way to hand out and retrieve them. /// Handle for any type of item (file or directory). #[derive(Debug, Clone)] pub(crate) struct FileHandle { /// The path of this file/folder. pub path: Box, // Non-static size, thus boxed. } ================================================ FILE: src/filesystem/file_handle/mod.rs ================================================ pub(super) mod file_handle_methods; pub(super) mod file_handle_struct; ================================================ FILE: src/filesystem/filesystem_struct.rs ================================================ // This is where the fun begins // Imports use crate::pool::pool_actions::pool_struct::Pool; use std::{ path::PathBuf, sync::{Arc, Mutex, OnceLock}, }; // Structs, Enums, Flags pub struct FlusterFS { #[allow(dead_code)] // it's lying. pub(crate) pool: Arc>, } use lazy_static::lazy_static; // Global varibles // We need to access the path quite deep down into the disk functions, passing it all the way down there would be silly. // Same with the virtual disk flag. lazy_static! { /// Use virtual disks instead of actually mounting the provided floppy drive path. pub(crate) static ref USE_VIRTUAL_DISKS: Mutex> = Mutex::new(None); /// The full path to the floppy drive. pub(crate) static ref FLOPPY_PATH: Mutex = Mutex::new(PathBuf::new()); } // Backups cannot be disabled at runtime, so we use a once lock for them /// Enable and disable backing up disks to /var/fluster pub(crate) static WRITE_BACKUPS: OnceLock = OnceLock::new(); // TUI cannot be disabled mid run. pub(crate) static USE_TUI: OnceLock = OnceLock::new(); /// Options availble at time of pool creation / filesystem load pub struct FilesystemOptions { /// Use virtual disks in a temp folder instead of accessing the floppy drive. /// This option is used for testing. #[allow(dead_code)] // it's lying. pub(super) use_virtual_disks: Option, /// The location of the floppy drive block device #[allow(dead_code)] // it's lying. pub(super) floppy_drive: PathBuf, /// Enable backing up disks to /var/fluster #[allow(dead_code)] // it's lying. pub(super) enable_backup: bool, /// Enable the TUI #[allow(dead_code)] // it's lying. pub(super) enable_tui: bool } ================================================ FILE: src/filesystem/fuse_filesystem_methods.rs ================================================ // The actual FUSE filesystem layer. // // // ====== // Imports // ====== // // use std::{ffi::OsStr, path::Path, time::Duration}; use fuse_mt::{DirectoryEntry, FileAttr, FileType, FilesystemMT, Statfs}; use log::{debug, error, info, warn}; use rand::Rng; use crate::{ filesystem::{ filesystem_struct::FlusterFS, item_flag::flag_struct::ItemFlag }, pool::{disk::{ generic::io::cache::cache_io::CachedBlockIO, standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryItem, DirectoryItemFlags }, io::directory::types::NamedItem } }, pool_actions::pool_struct::{Pool, GLOBAL_POOL}}, tui::{notify::NotifyTui, prompts::TuiPrompt, tasks::TaskType} }; use super::file_handle::file_handle_struct::FileHandle; use crate::error_types::filesystem::*; use fuse_mt::CreatedEntry; // // // ====== // Constants // ====== // // // You should probably be able to set your own custom TTL on mount, but // guess what? You should also probably be able to chown files. but that ain't happening either. // Hard coded to one year, see issue #51 const HANDLE_TIME_TO_LIVE: Duration = Duration::from_secs(365*24*60*60); // // // ====== // The actual fuse layer // ====== // // There's a lot of stuff in here we technically dont need. And I'm going to assume the information on this page is correct // https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html // I have archived this page on internet archive. // Thanks Geoff! I hope your life is going well, 16 years later. // // In theory, some calls like truncate() should be handled by the os before other operations here. We will still check for those flags, just in case. // // Also, in theory again, we should NEVER modify the flags before returning them. Linux VFS depends on this (in theory) impl FilesystemMT for FlusterFS { // The most British function in Fluster fn init(&self, _req: fuse_mt::RequestInfo) -> fuse_mt::ResultEmpty { // To speed up reads / reduce swaps, we will read the entire directory tree structure into memory // on startup. // So we will list the root directory. // None means we get the root let root = DirectoryBlock::try_find_directory(None).expect("There should be a root directory before init").expect("ditto"); // Keep listing each subdirectory until we're done, and if its not a directory, get the size of the file to // load in its info as well. let mut all_items: Vec = root.list().expect("should be there"); while let Some(item) = all_items.pop() { if item.flags.contains(DirectoryItemFlags::IsDirectory) { let block = item.get_directory_block().expect("If this doesn't work we're cooked anyways"); all_items.extend(block.list().expect("Ditto").into_iter()); } else { // This is a file, just get the size // We do this twice to make sure it get promoted. let _ = item.get_size().expect("cooked"); let _ = item.get_size().expect("cooked"); } } Ok(()) } // Called when filesystem is unmounted. Should flush all data to disk. fn destroy(&self) { // Inform user the filesystem is shutting down TuiPrompt::prompt_enter("Fluster! is shutting down.".to_string(), "Cache will now be flushed to disk".to_string(), true ); info!("Shutting down filesystem..."); // Flush all of the tiers of cache. info!("Flushing cache..."); // We dont retry flushing the cache, since if the flushing fails, that means it went all the way through // the troubleshooter and everything. If we retry here, chances are the cache has already dropped some blocks, // so we wouldn't even be able to properly finish it at this point. CachedBlockIO::flush().expect("I sure hope cache flushing works!"); // Now flush pool information info!("Flushing pool info..."); // Same story here. Pool::flush().expect("I sure hope pool flushing works!"); info!("Goodbye! .o/"); } // Get file attributes of an item. fn getattr( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, fh: Option, ) -> fuse_mt::ResultEntry { debug!("Getting attributes of `{}`...", path.display()); // I already wrote a method for this yay // but that assumes we have a handle. if let Some(handle) = fh { debug!("Handle was provided, getting and returning attributes."); // Handle exists, easy path. return Ok( ( HANDLE_TIME_TO_LIVE, FileHandle::read(handle).try_into()? ) ) } // No handle, dang. // Go find that sucker // Making a temporary handle (doesn't need to be allocated) lets us call some easier methods. // We cant just use the TryInto FileAttr since we dont know for sure if the item exists yet. let temp_handle: FileHandle = FileHandle { path: path.into(), }; // Go get the item. // This will automatically throw the correct error if the file/folder did not exist. let found_item = temp_handle.get_directory_item()?; // Get the attributes debug!("Getting attributes of item..."); let found_attributes: FileAttr = found_item.try_into()?; debug!("Done! Returning."); Ok( ( HANDLE_TIME_TO_LIVE, found_attributes ) ) } // We dont support file permissions. // fn chmod( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _fh: Option, // _mode: u32, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // We dont support file permissions. // fn chown( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _fh: Option, // _uid: Option, // _gid: Option, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // File truncation is supported. // Does not always truncate file to 0 bytes long. fn truncate( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, fh: Option, size: u64, ) -> fuse_mt::ResultEmpty { debug!("Truncating `{}` to be `{}` bytes long...", path.display(), size); let task_handle = NotifyTui::start_task( TaskType::FilesystemTruncateFile( path.file_name().unwrap_or(OsStr::new("?")).display().to_string() ), 2 ); // Get a file handle let handle: FileHandle = if let Some(exists) = fh { debug!("File handle was passed in, using that..."); // Got a handle from the call, no fancy work. // Read it in FileHandle::read(exists) } else { debug!("No handle provided, spoofing..."); // Temp handle that we will not allocate. FileHandle { path: path.into(), } }; debug!("Handle obtained."); // Go load the file to truncate. // Will return properly if item does not exist. let found_item = handle.get_directory_item()?; NotifyTui::complete_task_step(&task_handle); // Now with the directory item, we can run the truncation. debug!("Starting truncation..."); found_item.truncate(size)?; NotifyTui::complete_task_step(&task_handle); debug!("Truncation finished."); NotifyTui::finish_task(task_handle); // All done. Ok(()) } // We do not support manually updating timestamps. // fn utimens( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _fh: Option, // _atime: Option, // _mtime: Option, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // We do not support manually updating timestamps. // fn utimens_macos( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _fh: Option, // _crtime: Option, // _chgtime: Option, // _bkuptime: Option, // _flags: Option, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // We do not support symbolic links. // fn readlink(&self, _req: fuse_mt::RequestInfo, _path: &std::path::Path) -> fuse_mt::ResultData { // Err(libc::ENOSYS) // } // "This function is rarely needed, since it's uncommon to make these objects inside special-purpose filesystems." // This is for fancy things like block objects i believe, we do not support this. // fn mknod( // &self, // _req: fuse_mt::RequestInfo, // _parent: &std::path::Path, // _name: &std::ffi::OsStr, // _mode: u32, // _rdev: u32, // ) -> fuse_mt::ResultEntry { // Err(libc::ENOSYS) // } // Create a new directory if it does not already exist. // Returns file attributes about the new directory fn mkdir( &self, _req: fuse_mt::RequestInfo, parent: &std::path::Path, name: &std::ffi::OsStr, _mode: u32, // Permission bit related. Do not need. ) -> fuse_mt::ResultEntry { debug!("Creating new directory in `{}` named `{}`.", parent.display(), name.display()); let handle = NotifyTui::start_task(TaskType::FilesystemMakeDirectory(name.display().to_string()), 3); // Make sure the name isn't too long if name.len() > 255 { debug!("Name is too long."); return Err(FILE_NAME_TOO_LONG); } // the new directory let new_dir: DirectoryItem; let the_name: String = name.to_str().expect("Should be valid utf8").to_string(); // Open parent if let Some(mut parent) = DirectoryBlock::try_find_directory(Some(parent))? { NotifyTui::complete_task_step(&handle); debug!("Checking if directory exists..."); if parent.find_item(&NamedItem::Directory(the_name.clone()))?.is_some() { // Directory already exists. debug!("Directory already exists."); NotifyTui::cancel_task(handle); return Err(ITEM_ALREADY_EXISTS) } // Make the directory debug!("It did not, creating directory..."); new_dir = parent.make_directory(the_name)?; NotifyTui::complete_task_step(&handle); debug!("Directory created."); } else { // No such parent debug!("Parent did not exist."); NotifyTui::cancel_task(handle); return Err(NO_SUCH_ITEM); } // Now we need attribute information about it. debug!("Getting attribute info..."); let attributes: FileAttr = new_dir.try_into()?; NotifyTui::complete_task_step(&handle); debug!("Done."); // All done! debug!("Directory created successfully."); NotifyTui::finish_task(handle); Ok( ( HANDLE_TIME_TO_LIVE, attributes ) ) } // Deletes a file. fn unlink( &self, _req: fuse_mt::RequestInfo, parent: &std::path::Path, name: &std::ffi::OsStr, ) -> fuse_mt::ResultEmpty { debug!("Deleting file `{}` from directory `{}`...", name.display(), parent.display()); let handle = NotifyTui::start_task(TaskType::FilesystemDeleteFile(name.display().to_string()), 3); // Make a fake handle to lookup the file we are looking for let temp_handle: FileHandle = FileHandle { path: parent.join(name).into(), }; // This will return properly if the item did not exist. let file = temp_handle.get_directory_item()?; NotifyTui::complete_task_step(&handle); // Make sure it's a file if file.flags.contains(DirectoryItemFlags::IsDirectory) { // Cannot unlink directories. debug!("A directory was provided, not a file."); NotifyTui::cancel_task(handle); return Err(NOT_A_DIRECTORY); }; // Now we need the parent directory block to perform the removal debug!("Looking for file..."); if let Some(mut parent_dir) = DirectoryBlock::try_find_directory(Some(parent))? { NotifyTui::complete_task_step(&handle); // Delete the file if parent_dir.delete_file(file.into())?.is_some() { NotifyTui::complete_task_step(&handle); // All done. debug!("File deleted."); NotifyTui::finish_task(handle); Ok(()) } else { // Weird, we checked that the directory was there, but when we went to delete it, it wasnt??? warn!("We found the directory to delete, but when we tried to delete it, it was missing."); // this should not happen lmao, but whatever. NotifyTui::cancel_task(handle); Err(NO_SUCH_ITEM) } } else { // Should be impossible to get here, since we were already able to get the item from the parent // a few lines ago. // But we'll still gracefully handle it just in case. debug!("Parent folder does not exist."); NotifyTui::cancel_task(handle); Err(NO_SUCH_ITEM) } } // Deletes a directory. // Should fail if the directory is not empty. fn rmdir( &self, _req: fuse_mt::RequestInfo, parent: &std::path::Path, name: &std::ffi::OsStr, ) -> fuse_mt::ResultEmpty { debug!("Attempting to remove directory `{}` from `{}`...", name.display(), parent.display()); let handle = NotifyTui::start_task(TaskType::FilesystemRemoveDirectory(name.display().to_string()), 4); let string_name: String = name.to_str().expect("Should be valid utf8").to_string(); // Open the parent directory if let Some(parent_dir) = DirectoryBlock::try_find_directory(Some(parent))? { NotifyTui::complete_task_step(&handle); // Parent exists, get the child if let Some(child_dir) = parent_dir.find_item(&NamedItem::Directory(string_name))? { NotifyTui::complete_task_step(&handle); // Directory exists. // Make sure this is actually a directory if !child_dir.flags.contains(DirectoryItemFlags::IsDirectory) { // Not a dir debug!("Provided item is not a directory."); NotifyTui::cancel_task(handle); return Err(NOT_A_DIRECTORY); } // Get the block let block_to_delete = child_dir.get_directory_block()?; NotifyTui::complete_task_step(&handle); // Make sure it's empty if !block_to_delete.is_empty()? { // Nope. debug!("Directory is not empty, cannot delete."); NotifyTui::cancel_task(handle); return Err(DIRECTORY_NOT_EMPTY); } // Run the deletion. debug!("Deleting directory..."); block_to_delete.delete_self(child_dir)?; NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); debug!("Done."); Ok(()) } else { // child directory did not exist. debug!("The directory we wanted to delete does not exist."); NotifyTui::cancel_task(handle); Err(NO_SUCH_ITEM) } } else { // parent dir went to get milk debug!("Parent directory does not exist."); NotifyTui::cancel_task(handle); Err(NO_SUCH_ITEM) } } // We do not support symlinks. // fn symlink( // &self, // _req: fuse_mt::RequestInfo, // _parent: &std::path::Path, // _name: &std::ffi::OsStr, // _target: &std::path::Path, // ) -> fuse_mt::ResultEntry { // Err(libc::ENOSYS) // } // Renames / moves item. // Complicated error logic due to https://man7.org/linux/man-pages/man2/rename.2.html fn rename( &self, _req: fuse_mt::RequestInfo, parent: &std::path::Path, name: &std::ffi::OsStr, newparent: &std::path::Path, newname: &std::ffi::OsStr, ) -> fuse_mt::ResultEmpty { debug!("Renaming a item from `{}` to `{}`,", name.display(), newname.display()); debug!("and moving from `{}` to `{}`.", parent.display(), newparent.display()); // According to the man pages, we should get some flags here. but we dont. // I assume things like RENAME_NOREPLACE are being handled for us then. // Any case: // EISDIR newpath is an existing directory, but oldpath is not a directory. // EINVAL The new pathname contained a path prefix of the old, or, more generally, an attempt was made to make a directory a // subdirectory of itself. (recursion moment) // ENOENT The link named by oldpath does not exist; or, a directory component in newpath does not exist; or, oldpath or newpath // is an empty string. (TLDR this is a catch all) // If we are moving a directory: // The new path must not exist, or be empty. // ENOTEMPTY or EEXIST newpath is a nonempty directory // ENOTDIR A component used as a directory in oldpath or newpath is not, in fact, a directory. Or, oldpath is a directory, and // newpath exists but is not a directory. (Cant move non directories into directories, cant move directory into non directory.) // "If newpath exists but the operation fails for some reason, // rename() guarantees to leave an instance of newpath in place." // why word it like this lmao // if the destination already exists, but the move fails, keep what was already at the destination. // Also in theory, we should be checking if anyone is reading this item and if they are, return busy. // but there isnt any infra for that yet, and with the one year timeouts, you would have to wait a while. // fun. we will ignore it until something explodes. // This is gonna be REALLY complicated. lol. // Make sure the new name isn't too long if newname.len() > 255 { // too long warn!("New item name was too long"); return Err(FILE_NAME_TOO_LONG); } // Now, the "easy" cases. // Where we're coming from (including name of file/folder) let source_full_temp_handle: FileHandle = FileHandle { path: parent.join(name).into(), }; // Where we want to go // Where we're coming from (including name of file/folder) let destination_full_temp_handle: FileHandle = FileHandle { path: newparent.join(newname).into(), }; // If they are the same, we dont need to do anything at all. if source_full_temp_handle.path == destination_full_temp_handle.path { // why... debug!("Source and destination are the same, skipping."); return Ok(()); } // Make sure the two are the same underlying type let source_is_file = match source_full_temp_handle.is_file()? { Some(bool) => bool, None => { // The source item does not exist. return Err(NO_SUCH_ITEM); }, }; let destination_is_file = match destination_full_temp_handle.is_file()? { Some(bool) => bool, None => { // The destination item does not exist, which means we're not going to replace a pre-existing item, thus we // can just copy what the other file type is, since we will be creating this later. source_is_file }, }; debug!("Making sure the two are of the same type..."); if source_is_file == destination_is_file { debug!( "Types are the same, both are {}.", if source_is_file { "files" } else { "directories" } ) // They are both the same. } else { // Types are different, we cannot do that warn!("Types are different, we cannot perform this rename/move."); return Err(NOT_A_DIRECTORY); } // Now that we know the two types are the same, // Grab the parents and the item we are attempting to move depending on type. // For both types of rename/move operations, we must have: // - The parent folder of the source, and destination // - The item we are renaming // But we do not need the destination item to exist. // Any logic around that is handled differently depending on if this was a file or not. // I wrote directory handling first, then came back for file movement. Standing on the shoulders of myself, i know that // directories always return a Err(NO_SUCH_ITEM) if either of the parents do not exist. This is also true of files, so we // can perform that check out here. debug!("Trying to obtain the parents of the source and destination, and the directory item for the item we are trying to move."); let source_item_name: String = name.to_str().expect("Should be valid utf8").to_string(); let destination_item_name: String = newname.to_str().expect("Should be valid utf8").to_string(); debug!("Checking if parents existed..."); // Try to get the source parent directory, then try to get the item refering to the dir we are moving. let mut source_parent_dir: DirectoryBlock = if let Some(exist) = DirectoryBlock::try_find_directory(Some(parent))? { // Good. debug!("Source parent exists."); exist } else { // missing warn!("Source parent did not exist. Cannot continue."); return Err(NO_SUCH_ITEM); }; let mut destination_parent_dir: DirectoryBlock = if let Some(exist) = DirectoryBlock::try_find_directory(Some(newparent))? { // Good. debug!("Destination parent exists."); exist } else { // missing warn!("Destination parent did not exist. Cannot continue."); return Err(NO_SUCH_ITEM); }; // Item logic must be handled lower down, but we can at least abstract the calls out at this point to work with options later. // We know the kind here so we can abstract this away as well. let maybe_source_directory_item: Option; let maybe_destination_directory_item: Option; if source_is_file { // both files. maybe_source_directory_item = source_parent_dir.find_item(&NamedItem::File(source_item_name.clone()))?; maybe_destination_directory_item = destination_parent_dir.find_item(&NamedItem::File(destination_item_name.clone()))?; } else { // both directories. maybe_source_directory_item = source_parent_dir.find_item(&NamedItem::Directory(source_item_name.clone()))?; maybe_destination_directory_item = destination_parent_dir.find_item(&NamedItem::Directory(destination_item_name.clone()))?; }; // The following complicated move logic requires that the two parent directories be different. If the source and // destination directories are the same, we can just rename the inode, skipping all of the fancier operations. if parent == newparent { // Sweet! // We dont even need the destination info drop(maybe_destination_directory_item); drop(destination_full_temp_handle); drop(destination_parent_dir); // Source must exist. let source = match maybe_source_directory_item { Some(ok) => ok, None => { // Can't rename nothing. return Err(NO_SUCH_ITEM); }, }; // Type does not matter, we can just update the name in the directory, since inodes do not hold that info. // Explicitly check the error, silently returning when this fails is bad. let rename_result = match source_parent_dir.try_rename_item(&source.into(), destination_item_name) { Ok(ok) => ok, Err(err) => { // Renaming failed for a lower level issue! warn!("Item rename failed! Why?"); warn!("`{err:#?}`"); // Bail out return Err(err.into()); }, }; if rename_result { // rename worked. debug!("Item renamed successfully."); return Ok(()) } else { // Somehow the item we wanted to rename disapeared, we'll return a generic error. // Since panicing at this high of a level would be stupid. Even though this branch is // unlikely. warn!("Item to rename disappeared!"); return Err(GENERIC_FAILURE); } } // This rename moves the item between directories. // we branch depending on if it was a file or directory, handling is slightly different if source_is_file { // // File movement. // debug!("Starting file movement..."); // If the new item exists, it will be replaced. (see rename(2) manpage) // The source item must exist. debug!("Checking that source item exists..."); let source_item = if let Some(existed) = maybe_source_directory_item { // good debug!("Yes it does."); existed } else { // cant move nothing. warn!("Source item did not exist. Cannot perform rename/move."); return Err(NO_SUCH_ITEM); }; debug!("Checking if destination file already existed..."); // Check if the destination file already exists, that will change our behavior on failure. if maybe_destination_directory_item.is_some() { // Destination item exists, we will be overwriting this, but we will hold onto it just in case. // In theory we should try to put this back if the rename fails. // Since the item exists already, we will extract it. To delete the old file we need to have the file in the block. Which // complicates things a bit. So we pull it out, and when we are ready to delete it, we rename it, put it back in, and delete it. let extracted_old: DirectoryItem = match destination_parent_dir.find_and_extract_item(&NamedItem::File(destination_item_name.clone())) { Ok(ok) => match ok { Some(ok) => ok, None => { // We tried to extract it, but it was no longer there? // No data has been modified (by us here at least), entire operation can be retried. warn!("Destination item was no longer there when extraction was attempted. Non-fatal, but weird."); return Err(NO_SUCH_ITEM) }, }, Err(error) => { // The extraction failed at _some_ point, unknown where. Chances are the destination item is now gone. But we have failed. warn!("Extracting the old item failed for a low level reason, we have to bail."); return Err(error.into()) }, }; // Now we have that copy for later, we can stick a copy of the source item into the destination debug!("Inserting copy of the source file into the destination directory..."); // We must give it the new name first: let mut renamed_source_item = source_item; renamed_source_item.name = destination_item_name.clone(); renamed_source_item.name_length = destination_item_name.len() as u8; // Already checked name length. match destination_parent_dir.add_item(&renamed_source_item) { Ok(_) => (), Err(error) => { // Failed to add item to destination warn!("Failed to put copy into the destination for low level reason."); // Try to put back the file that was here before. warn!("Attempting to restore previous item if possible..."); // Chances are if the previous one failed, this will to, but whatever. if destination_parent_dir.add_item(&extracted_old).is_ok() { warn!("Previous file restored."); } else { // oof warn!("Failed to restore previous file. File has been permanently lost."); } // As good as it'll get. Return the error that caused us to fail. return Err(error.into()) }, } debug!("Done."); debug!("Removing old item from origin directory..."); // Now go to the parent of the source and tell em to slime tf outta the old item match source_parent_dir.find_and_extract_item(&NamedItem::File(source_item_name.clone())) { Ok(ok) => { match ok { Some(_found) => { // Removal worked debug!("Source item removed.") }, None => { // Item we tried to remove was not there. // Weird, but if we got this far, it must have at least made it into the destination. warn!("The source item we tried to remove was not there. Which is fine, since that was the goal anyways."); }, } }, Err(err) => { // We have now failed to remove the previous item, but the new one is in place. This is good enough. // If the item is still in that other block, it is now a duplicate reference of the underlying file. // Not great... but not much I can do here. No transactions means no safe actions! warn!("Failed to remove previous item, it may still be there. Good enough."); // The move has now technically finished, even though we have errored at this point, so guess what? debug!("Removal failed due to: {err:#?}"); return Ok(()); }, }; // Now that the file is no longer where it used to live, we can delete the item that used to live in the destination that // we overwrote. // If this fails, the item is no longer in the file, but will still occupy blocks on disk, which isn't great, but at least // you cant get to them. // Rename the item so we dont collide with the newly moved in file let mut renamed: DirectoryItem = extracted_old; renamed.name.push_str(".fluster_old"); // Is this too long now? if renamed.name.len() > 255 { // Shoot, go with a random name instead. let mut random: rand::prelude::ThreadRng = rand::rng(); let random_name: u128 = random.random(); let mut random_name_string = random_name.to_string(); random_name_string.truncate(128); renamed.name = random_name_string; renamed.name_length = renamed.name.len() as u8; // will fit } // Hold onto the new name for later let deletion_name: String = renamed.name.clone(); // Put the renamed item in there. debug!("Re-inserting old file with a new name to delete it..."); match destination_parent_dir.add_item(&renamed) { Ok(_) => { // Adding worked. debug!("Inserted.") }, Err(err) => { // Damn. We will just leak the blocks this took up. warn!("Insertion failed."); warn!("Just to keep going, we will leak the blocks that the old file references."); warn!("Rename \"finished\"."); debug!("Failed due to: {err:#?}"); // Rename still worked overall. return Ok(()); }, } // Now kill it debug!("Deleting the old file..."); match destination_parent_dir.delete_file(NamedItem::File(deletion_name)) { Ok(ok) => match ok { Some(_) => { // File was deleted. debug!("Old file deleted.") }, None => { // The file did not exist?!? warn!("Somehow, we added the file, and when we went to delete it, it no longer existed."); warn!("It probably leaked, but there is nothing we can do."); // Good enough! warn!("Good enough. Rename finished."); return Ok(()); }, }, Err(err) => { // Yet another leak scenario. warn!("Deletion failed."); warn!("Just to keep going, we will leak the blocks that the old file references."); warn!("Rename \"finished\"."); debug!("Failed due to: {err:#?}"); // Rename still worked overall. return Ok(()); }, } // File has been deleted. Cleanup is now finished. // Done moving file, which replaced a pre-existing file. } else { // We are not trying to overwrite a pre-existing file, this makes our lives easier. debug!("Destination item does not exist yet. We will create it."); // We must give it the new name first: let mut renamed_source_item = source_item; renamed_source_item.name = destination_item_name.clone(); renamed_source_item.name_length = destination_item_name.len() as u8; // Already checked name length. // Put the file into the destination debug!("Adding source file to destination directory..."); match destination_parent_dir.add_item(&renamed_source_item) { Ok(_) => { // That worked }, Err(err) => { // Failed to add to destination // Drive level issue. warn!("Failed at a level lower than us. Unknown state."); debug!("Failed due to: {err:#?}"); return Err(err.into()) }, } debug!("File added."); // Now we need to remove the old item. match source_parent_dir.delete_file(NamedItem::File(source_item_name)) { Ok(ok) => { // if ok is none, the item disappeared, which should not happen. // Yes its weird that the item dissapeared, but its better to let the caller // re-list the directory and try the rename again if needed. Otherwise we would // just crash, which is, sub-par. if ok.is_none() { warn!("The item we were renaming disappeared!"); return Err(GENERIC_FAILURE) } }, Err(err) => { // The file made it to the destination, but removing the original failed. // The old item may still be there, or it leaked blocks due to failed cleanup. warn!("Failed to delete source item, it may still be there."); warn!("Blocks were probably leaked."); warn!("Non-critical deletion failure: {err:#?}") // Good enough. }, } } // All done. debug!("File moved successfully."); Ok(()) } else { // // Directory movement. // debug!("Starting directory movement..."); // Make sure we aren't trying to make a self referential folder debug!("Checking for recursion..."); if destination_full_temp_handle.path.starts_with(&source_full_temp_handle.path) { // Destination contains the source, therefore this is self referential. warn!("Cannot move directory inside of itself."); return Err(INVALID_ARGUMENT); } debug!("No recursion."); // To fulfill the requirement of the destination directory being empty, we will check if the directory exists in the // parent. If it does, make sure its empty, if it is, then we will update the DirectoryItem to point at the block for // the start of the DirectoryBlocks of the source directory. // If the directory does not exist, we will create it and still do the same pointer swap. // if the directory is not empty, we have to cancel the move. // Do both directories exist? debug!("Checking if the parents contain directory item we want to move..."); // source let source_directory_item: DirectoryItem = if let Some(item) = maybe_source_directory_item { // Source exists item } else { // Item was not there. Cant copy nothing. debug!("Source directory missing, cannot rename/move folder."); return Err(NO_SUCH_ITEM); }; debug!("Source good."); // Check that the destination folder exists and is empty. if let Some(item) = maybe_destination_directory_item { // Destination has to be empty debug!("Destination already existed, making sure its empty..."); if item.clone().get_directory_block()?.is_empty()? { // All good, we will delete the directory since we are going to replace it. debug!("It's empty, it will be deleted soon"); } else { // Directory was not empty, we cannot continue warn!("Destination directory was not empty, cannot rename/move folder."); return Err(DIRECTORY_NOT_EMPTY); } } else { // no destination, we will make it debug!("Destination did not exist, creating..."); // Annoying clone. let _ = destination_parent_dir.make_directory(destination_item_name.clone())?; // yes we are creating it to just remove it again, but we are supposed to (in theory if i remember correctly idk its 1am) leave // a folder at the destination even if the move fails. }; debug!("Destination good."); // Now for the fun part, we can extract the DirectoryItem from the first directory, and swap it into // the second one, thus the new folder points at the contents without moving the underlying files in the folder. debug!("Swapping DirectoryItems..."); // Remove the destination folder // "Inshallah he will be grounded into a fine dust" // We have to tread lightly at this point. If the swap fails, we would lose data. // 🤓 erm actually the data would still be there, just not referenced- SHUT UP // Extract, and delete. Extraction cleans up anything this used to point to for us. debug!("Extracting destination..."); let _extracted_dest = match destination_parent_dir.find_and_extract_item(&NamedItem::Directory(destination_item_name.clone())) { Ok(ok) => { // Directory had to've been there, right? if let Some(worked) = ok { // Found and extracted the item, we will hold onto it incase we need to recreate it. worked } else { // What warn!("Tried to delete the destination directory to prepare for swap, but it was no longer there!"); // This should be impossible. // But just in case, er, say the operation failed. return Err(GENERIC_FAILURE); } // We will hold onto it just in case, even though it's empty. }, Err(err) => { // This should fail tests. if cfg!(test) { panic!("{err:#?}"); } // Drive level issue. warn!("Failed at a level lower than us. Unknown state."); return Err(err.into()) }, }; debug!("Destination extracted"); // Now get a COPY of the source, we wont remove the source until we know for sure we have properly moved the folder. // Wait, we already have it, in `source_directory_item`, duh // Attempt to add the directory to the new parent debug!("Inserting copy of source directory..."); // Make sure we rename dat mf fr let mut renamed_source_dir_item = source_directory_item; renamed_source_dir_item.name = destination_item_name.clone(); renamed_source_dir_item.name_length = destination_item_name.len() as u8; // Length checked above match destination_parent_dir.add_item(&renamed_source_dir_item) { Ok(_) => { // that worked // No need to do anything }, Err(err) => { // Drive level issue. warn!("Failed at a level lower than us. Unknown state."); // Attempt to uphold POSIX standard (like hell the rest of fluster is compliant) by // at least attempting to put the original directory back again. // We dont actually need to though, since it hasn't been extracted yet. // This should fail tests. if cfg!(test) { panic!("{err:#?}"); } return Err(err.into()) }, } debug!("Insertion succeeded."); // Now that the data has been safely pointed at from the new location, we will remove the old reference to it. debug!("Removing old source..."); let ashes = match source_parent_dir.find_and_extract_item(&NamedItem::Directory(source_item_name.clone())) { Ok(ash) => ash, Err(err) => { // Drive level issue. warn!("Failed at a level lower than us. Unknown state."); // So now we have properly moved the source into the destination, but the source might still be there. // This wouldn't be too bad, if not for the fact that now both DirectoryBlocks contain the same DirectoryItem which // points to the same directory. Thus they have become hard-linked. Not great. // Good luck lmao. warn!("There isn't really anything we can do at this point, a hard link has been created due to this."); warn!("We consider this good enough. Done moving directory."); // shoulda thought ahead and made fluster more transactional, oh well. // We have to ignore the error, but might as well print it. debug!("{err:#?}"); // This should fail tests. if cfg!(test) { panic!("{err:#?}"); } return Ok(()); }, }; if let Some(_to_ashes) = ashes { // It was there, and removed. // We're all done. debug!("Done.") } else { // ????? DIRECTORY IS NOT THERE (GONE) (STOLEM) warn!("Somehow, we copied the source directory across correctly, but now the source is missing so we cant remove it."); // I mean like, this still worked, no? // goals: // put source in destination: check // source is no longer there: check // sooooooo // GOOD ENOUGH! warn!("...but that's close enough."); }; // All done! debug!("Rename finished, directories renamed/moved."); Ok(()) } // this comment is unreachable, all cases are covered. } // We do not support hard links. // fn link( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _newparent: &std::path::Path, // _newname: &std::ffi::OsStr, // ) -> fuse_mt::ResultEntry { // Err(libc::ENOSYS) // } // Open a file and get a handle that will be used to access it. // Does not create files. fn open( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, flags: u32, ) -> fuse_mt::ResultOpen { debug!("Opening item at path `{}`...", path.display()); let task_handle = NotifyTui::start_task(TaskType::FilesystemOpenFile( path.file_name().unwrap_or(OsStr::new("?")).display().to_string() ), 4); // Deduce the open permissions. debug!("Deducing flags..."); let converted_flag: ItemFlag = ItemFlag::deduce_flag(flags)?; debug!("Ok."); // We require at least one of the read/write flags. // ...or, more correctly: we would require them if we used them. // We dont. Everything is read/write. // We ignore any flags that are not valid for this method, such as // truncation or creation flags. // open() always returns a brand new file handle, regardless if that file was // already open somewhere else. let mut handle: FileHandle = FileHandle { path: path.into(), }; // We do not allocate the file handle until we are sure we will use it. // Make sure the name of the file is not too long. if handle.name().len() > 255 { warn!("File name is too long."); // File name was too long. NotifyTui::cancel_task(task_handle); return Err(FILE_NAME_TOO_LONG) } // If this is the dot directory, we need to go up a level to read ourselves. if handle.name() == "." { // Go up a path. // If this returns none, all is well handle.path = handle.path.parent().unwrap_or(Path::new("")).into(); } // Load in info about where the file should be. // This will bail if a low level floppy issue happens. debug!("Attempting to load in the parent directory..."); let containing_dir_block: DirectoryBlock = match DirectoryBlock::try_find_directory(handle.path.parent())? { Some(ok) => ok, None => { // Cannot load files from directories that do not exist. warn!("Directory that the item was supposed to be contained within does not exist."); NotifyTui::cancel_task(task_handle); return Err(NO_SUCH_ITEM) }, }; debug!("Directory loaded."); NotifyTui::complete_task_step(&task_handle); // At this point. We need to know if we are looking for a directory or a file. debug!("Deducing request item type..."); let extracted_name = handle.name(); let item_to_find: NamedItem = match handle.get_named_item()? { Some(item) => item, None => { // No such item exists. NotifyTui::cancel_task(task_handle); return Err(NO_SUCH_ITEM); }, }; if item_to_find.is_file() { debug!("Looking for a file..."); } else { debug!("Looking for a directory..."); } debug!("Named `{extracted_name}`."); // Hold onto the item until we need it let found_item: DirectoryItem; // Now load in the directory item. debug!("Attempting to find the item..."); if let Some(exists) = containing_dir_block.find_item(&item_to_find)? { debug!("Item exists."); found_item = exists; } else { // No item debug!("Item does not exist."); NotifyTui::cancel_task(task_handle); return Err(NO_SUCH_ITEM); } NotifyTui::complete_task_step(&task_handle); // We have now loaded in the directory item, or bailed out if needed. // Assert that this is a directory if required. // In theory we could check this earlier, but it's good to ensure that the underlying // item agrees. if converted_flag.contains(ItemFlag::ASSERT_DIRECTORY) { debug!("Caller wants to ensure they are opening a directory."); if !found_item.flags.contains(DirectoryItemFlags::IsDirectory) { debug!("This is not a directory."); NotifyTui::cancel_task(task_handle); return Err(NOT_A_DIRECTORY) } debug!("This is a directory."); } // We are done creating/loading the file, its time to get a handle. debug!("Getting a handle on things..."); let new_handle: u64 = handle.allocate(); NotifyTui::complete_task_step(&task_handle); // Done! debug!("Opening finished."); Ok((new_handle, converted_flag.into())) } // Read file data from a file handle. // "Note that it is not an error for this call to request to read past the end of the file, // and you should only return data up to the end of the file // (i.e. the number of bytes returned will be fewer than requested; possibly even zero). // Do not extend the file in this case." // // Uses callbacks, wacky, not sure how that works. fn read( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, fh: u64, offset: u64, size: u32, callback: impl FnOnce(fuse_mt::ResultSlice<'_>) -> fuse_mt::CallbackResult, ) -> fuse_mt::CallbackResult { debug!("Reading `{}` bytes from file `{}`", size, path.display()); let task_handle = NotifyTui::start_task( TaskType::FilesystemReadFile( path.file_name() .unwrap_or(OsStr::new("?")).display().to_string() ), 3 ); // Open the file handle let got_handle = FileHandle::read(fh); // Still not sure if we need to check this, but whatever. if got_handle.path != path.into() { // They aren't the same? not sure what to do with that error!("readdir() tried to read a path, but provided a handle to a different path."); error!("fh: `{}` | path: `{}`", got_handle.path.display(), path.display()); error!("Not sure what to do here, giving up."); NotifyTui::cancel_task(task_handle); return callback(Err(GENERIC_FAILURE)); } // Try finding the directory item let file = match got_handle.get_directory_item() { Ok(ok) => ok, Err(err) => { // Getting the item failed, maybe it wasn't there. NotifyTui::cancel_task(task_handle); return callback(Err(err)) }, }; NotifyTui::complete_task_step(&task_handle); // Make sure that it's a file. if file.flags.contains(DirectoryItemFlags::IsDirectory) { // Can't read a directory! warn!("Tried to read a directory as a file. Ignoring..."); return callback(Err(IS_A_DIRECTORY)); } // Found a file! // We need to bound our read by the size of the file, since the read() filesystem call can // try to read past the end. let file_size = match file.get_size() { Ok(ok) => ok, Err(error) => { // Lower level error NotifyTui::cancel_task(task_handle); warn!("Failed to get size of file! Giving up..."); return callback(Err(error.into())) }, }; NotifyTui::complete_task_step(&task_handle); // Subtract the offset to idk man why am i explaining this im sure you understand. // Reads are limited to 4GB long, which should be way above our max read size anyways. // If a read bigger than that comes in, we'll ignore it. let checking_size = std::cmp::min(size as u64, file_size - offset); let bounded_read_length = if checking_size > u32::MAX.into() { // Cant do that. // Also if you're here, that means the file you're trying to read is actually >4GB. // You are insane. warn!("Tried to read more than 4GB at once!"); NotifyTui::cancel_task(task_handle); return callback(Err(INVALID_ARGUMENT)); } else { // Size checked, this cast is safe. checking_size as u32 }; if bounded_read_length != size { // size did change. debug!("Read was too large, truncated to `{bounded_read_length}` bytes."); } // Do the read. // This vec might be HUGE, this is why we need to limit the read size on the filesystem. debug!("Starting read..."); let read_buffer: Vec = match file.read_file(offset, bounded_read_length) { Ok(ok) => ok, Err(error) => { // Lower level error warn!("Failed while reading the file! Giving up..."); NotifyTui::cancel_task(task_handle); return callback(Err(error.into())) }, }; NotifyTui::complete_task_step(&task_handle); debug!("Read finished."); NotifyTui::finish_task(task_handle); // All done! callback(Ok(&read_buffer)) } // Write data to a file using a file handle. fn write( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, fh: u64, offset: u64, data: Vec, _flags: u32, // hehe ) -> fuse_mt::ResultWrite { debug!("Writing `{}` bytes to file `{}`...", data.len(), path.display()); let task_handle = NotifyTui::start_task( TaskType::FilesystemWriteFile( path.file_name() .unwrap_or(OsStr::new("?")).display().to_string() ), 2 ); // Open the file handle let got_handle = FileHandle::read(fh); // Still not sure if we need to check this, but whatever. if got_handle.path != path.into() { // They aren't the same? not sure what to do with that error!("readdir() tried to read a path, but provided a handle to a different path."); error!("fh: `{}` | path: `{}`", got_handle.path.display(), path.display()); error!("Not sure what to do here, giving up."); NotifyTui::cancel_task(task_handle); return Err(GENERIC_FAILURE); } // Try finding the directory item let file = got_handle.get_directory_item()?; NotifyTui::complete_task_step(&task_handle); // man page: // If count is zero and fd refers to a regular file, then write() may // return a failure status if one of the errors below is detected. // If no errors are detected, or error detection is not performed, 0 // is returned without causing any other effect. If count is zero // and fd refers to a file other than a regular file, the results are // not specified. // // So if we want to write zero bytes, do nothing if data.is_empty() { // uh ok then debug!("Caller wanted to write 0 bytes. Skipping write."); NotifyTui::cancel_task(task_handle); return Ok(0); } // Now write to the file! debug!("Starting write..."); let bytes_written = file.write_file(&data, offset)?; NotifyTui::complete_task_step(&task_handle); debug!("Write completed."); // According to spec, we just tell the caller how many bytes we wrote. // If we didn't write everything, its on them to keep going. // https://man7.org/linux/man-pages/man2/write.2.html // Return the number of bytes written. NotifyTui::finish_task(task_handle); Ok(bytes_written) } // Flushing does not do anything, since we manually handle our caching. fn flush( &self, _req: fuse_mt::RequestInfo, _path: &std::path::Path, _fh: u64, _lock_owner: u64, ) -> fuse_mt::ResultEmpty { // We dont want the OS to be able to flush the cache to disk, this could happen randomly for no reason. // We are responsible for tracking how stale the cache is. // The only time we will flush the cache from this level is on shutdown. Ok(()) } // Releasing a file handle. fn release( &self, _req: fuse_mt::RequestInfo, _path: &std::path::Path, fh: u64, _flags: u32, _lock_owner: u64, _flush: bool, ) -> fuse_mt::ResultEmpty { FileHandle::drop_handle(fh); Ok(()) } // See flush() fn fsync( &self, _req: fuse_mt::RequestInfo, _path: &std::path::Path, _fh: u64, _datasync: bool, ) -> fuse_mt::ResultEmpty { Ok(()) } // Open a directory and get a handle to it fn opendir( &self, req: fuse_mt::RequestInfo, path: &std::path::Path, flags: u32, ) -> fuse_mt::ResultOpen { // This just gets pushed over to open(), since // we already handle directories over there. // // Should we handle files and directories both in open? maybe not. self.open(req, path, flags) } // List the contents of a directory. // "Return one or more directory entries (struct dirent) to the caller." // "This is one of the most complex FUSE functions." Oof. // "The readdir function is somewhat like read, in that it starts at a // given offset and returns results in a caller-supplied buffer." // "However, the offset not a byte offset" What the hell // "...and the results are a series of struct dirents rather than being uninterpreted bytes" those are just words Geoffery // // Luckily we are working at a level way above that! fn readdir( &self, _req: fuse_mt::RequestInfo, path: &std::path::Path, fh: u64, ) -> fuse_mt::ResultReaddir { debug!("Getting contents of directory `{}`...", path.display()); let task_handle = NotifyTui::start_task(TaskType::FilesystemReadDirectory( path.file_name().unwrap_or(OsStr::new("?")).display().to_string() ), 4 ); // Make sure the file handle and the incoming path are the same. I assume they should be, but // cant hurt to check. let got_handle = FileHandle::read(fh); if got_handle.path != path.into() { // They aren't the same? not sure what to do with that error!("readdir() tried to read a path, but provided a handle to a different path."); error!("fh: `{}` | path: `{}`", got_handle.path.display(), path.display()); error!("Not sure what to do here, giving up."); NotifyTui::cancel_task(task_handle); return Err(GENERIC_FAILURE); } // Since we have a handle, getting the directory is easy. debug!("Getting the directory item from handle..."); let dir_item: DirectoryItem = if let Ok(exists) = got_handle.get_directory_item() { // good exists } else { // Tried to read in a directory item that did not exist, yet we have a handle to it? // Guess the handle must be stale? // Yes, get_directory_item() returns its own error, but we should get rid of the invalid handle. warn!("Tried to read in a directory item from a handle, but the item was not there. Returning stale."); NotifyTui::cancel_task(task_handle); return Err(STALE_HANDLE) }; NotifyTui::complete_task_step(&task_handle); // Double check that this is a file. if !dir_item.flags.contains(DirectoryItemFlags::IsDirectory) { // No. warn!("Tried to call readdir on a file!"); NotifyTui::cancel_task(task_handle); return Err(NOT_A_DIRECTORY); } debug!("Getting directory block..."); let dir_block = dir_item.get_directory_block()?; NotifyTui::complete_task_step(&task_handle); // List the files off debug!("Listing items..."); let items = dir_block.list()?; NotifyTui::complete_task_step(&task_handle); // Now pull out the names and types let mut listed_items: Vec = items.iter().map(|item| { let kind = if item.flags.contains(DirectoryItemFlags::IsDirectory) { FileType::Directory } else { FileType::RegularFile }; DirectoryEntry { name: item.name.clone().into(), kind, } }).collect(); NotifyTui::complete_task_step(&task_handle); // Now add the unix `.` item. listed_items.push( DirectoryEntry { name: std::ffi::OsStr::new(".").into(), kind: FileType::Directory, } ); NotifyTui::finish_task(task_handle); // All done! debug!("Done. Directory contained `{}` items.", listed_items.len()); Ok(listed_items) } // See release() fn releasedir( &self, _req: fuse_mt::RequestInfo, _path: &std::path::Path, fh: u64, _flags: u32, ) -> fuse_mt::ResultEmpty { FileHandle::drop_handle(fh); Ok(()) } // See flush() fn fsyncdir( &self, _req: fuse_mt::RequestInfo, _path: &std::path::Path, _fh: u64, _datasync: bool, ) -> fuse_mt::ResultEmpty { Ok(()) } // Get file system statistics. // Seemingly contains information about the file system, like optimal block size and max file name length fn statfs(&self, _req: fuse_mt::RequestInfo, _path: &std::path::Path) -> fuse_mt::ResultStatfs { // This does not appear to be required, but we can implement it anyways. // Get the pool header, we need it for disk counts and such. // If this doesnt work, just tell the caller to try again later. let pool = if let Ok(pool_inner) = GLOBAL_POOL.get().expect("Global pool should be created at this stage!").try_lock() { pool_inner.header } else { // Lock failed. debug!("Tried to get stats about the pool, but locking the pool failed. Skipping..."); return Err(TRY_AGAIN) }; // Number of blocks on the device is just the highest disk*2880 let blocks: u64 = pool.highest_known_disk as u64 * 2880; // We know how many blocks are free. // Not sure how Linux reacts to disks that grow on the fly, but // it seems like a likely enough use-case that this should be fine... let bfree: u64 = pool.pool_standard_blocks_free.into(); // The number of available blocks is the same as the number of free blocks. let bavail = bfree; // Knowing how many files exist is a process that would take a VERY long time to figure out // typically. This seems like a bad idea to actually look up on the fly. // Just say there's a few. let files: u64 = 985; // We don't have a limit to file inodes, so we always have a lot free. let ffree: u64 = 12345; // Blocks are 512 bytes, technically, but innards of those blocks are limited... // Not that huge of a deal for transfers to be aligned, so we'll just say 512 still. let bsize: u32 = 512; // Max filename length is 255 characters. let namelen: u32 = 255; // Fragments are the min size that can be allocated to a file, which in our case is roughly a block. let frsize: u32 = 512; let stat: Statfs = Statfs { blocks, bfree, bavail, files, ffree, bsize, namelen, frsize, }; Ok(stat) } // Extended attributes are not supported. // fn setxattr( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _name: &std::ffi::OsStr, // _value: &[u8], // _flags: u32, // _position: u32, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // Extended attributes are not supported. // fn getxattr( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _name: &std::ffi::OsStr, // _size: u32, // ) -> fuse_mt::ResultXattr { // Err(libc::ENOSYS) // } // Extended attributes are not supported. // fn listxattr( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _size: u32, // ) -> fuse_mt::ResultXattr { // Err(libc::ENOSYS) // } // Extended attributes are not supported. // fn removexattr( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _name: &std::ffi::OsStr, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // "This call is not required but is highly recommended." Okay then we wont do it muhahaha // fn access( // &self, // _req: fuse_mt::RequestInfo, // _path: &std::path::Path, // _mask: u32, // ) -> fuse_mt::ResultEmpty { // Err(libc::ENOSYS) // } // Creates and opens a new file, returns a file handle. fn create( &self, req: fuse_mt::RequestInfo, parent: &std::path::Path, name: &std::ffi::OsStr, _mode: u32, flags: u32, ) -> fuse_mt::ResultCreate { debug!("Creating new file named `{}` in `{}`...", name.display(), parent.display()); let task_handle = NotifyTui::start_task(TaskType::FilesystemCreateFile(name.display().to_string()), 5); // Extract the flags // Will bail if needed. let deduced_flags: ItemFlag = ItemFlag::deduce_flag(flags)?; // Is the name too long? if name.len() > 255 { debug!("File name is too long. Bailing."); NotifyTui::cancel_task(task_handle); return Err(FILE_NAME_TOO_LONG) } // Try and load in the parent directory // This will bail if a low level floppy issue happens. debug!("Attempting to load in the parent directory..."); let mut containing_dir_block: DirectoryBlock = match DirectoryBlock::try_find_directory(Some(parent))? { Some(ok) => ok, None => { // Nope, no parent. warn!("Cannot create files in directories that do not exist."); NotifyTui::cancel_task(task_handle); return Err(NO_SUCH_ITEM) }, }; debug!("Directory loaded."); NotifyTui::complete_task_step(&task_handle); // Make sure the file does not already exist. debug!("Checking if file already exists..."); let converted_name: String = name.to_str().expect("Should be valid UTF8.").to_string(); // Will bail if needed. if let Some(exists) = containing_dir_block.find_item(&NamedItem::File(converted_name.clone()))? { NotifyTui::complete_task_step(&task_handle); debug!("File already exists."); // But do we care? if deduced_flags.contains(ItemFlag::CREATE_EXCLUSIVE) { // Yes we do, this is a failure. debug!("Caller wanted to create this file, not open it. Bailing."); NotifyTui::cancel_task(task_handle); return Err(ITEM_ALREADY_EXISTS) } // Since the file already exists we can skip the creation process. // just load it in as usual. // Full item path let constructed_path: &Path = &parent.join(name); // Dont care about the returned flags, they wont change anyways. let (file_handle, _): (u64, u32) = self.open(req, constructed_path, flags)?; NotifyTui::complete_task_step(&task_handle); // Get the innards of the handle let handle_inner: FileHandle = FileHandle::read(file_handle); NotifyTui::complete_task_step(&task_handle); // Truncate if needed (open(2) syscall) // Must be a file if deduced_flags.contains(ItemFlag::TRUNCATE) && !exists.flags.contains(DirectoryItemFlags::IsDirectory) { self.truncate(req, constructed_path, Some(file_handle), 0)?; // Truncate to 0 } // Get the metadata from that debug!("Getting file attributes..."); let facebook_data: FileAttr = handle_inner.try_into()?; NotifyTui::complete_task_step(&task_handle); // Put it all together // No idea what the TTL should be set to. I'm assuming that's how long the handles last? // I will never drop handles on my side, the OS has to drop em. debug!("Done reading in file, returning."); NotifyTui::finish_task(task_handle); return Ok(CreatedEntry { ttl: HANDLE_TIME_TO_LIVE, attr: facebook_data, fh: file_handle, flags, // We use the same flags we came in with. Not the one from the loaded file. // Is that a bad idea? No idea. Seems to work just fine. }) } NotifyTui::complete_task_step(&task_handle); // File did not exist, actually creating it... debug!("Creating file..."); let resulting_item: DirectoryItem = containing_dir_block.new_file(converted_name)?; NotifyTui::complete_task_step(&task_handle); debug!("Created file."); // Full item path let constructed_path: &Path = &parent.join(name); // Construct and return the handle to the new file let new_handle: FileHandle = FileHandle { path: constructed_path.into(), }; // We can get attributes directly from the directory item we just made let attributes: FileAttr = resulting_item.try_into()?; NotifyTui::complete_task_step(&task_handle); // Allocate the handle for it let handle_num: u64 = new_handle.allocate(); NotifyTui::complete_task_step(&task_handle); // Assemble it, and we're done! NotifyTui::finish_task(task_handle); debug!("Done creating file."); Ok(CreatedEntry { ttl: HANDLE_TIME_TO_LIVE, attr: attributes, fh: handle_num, flags, }) } } ================================================ FILE: src/filesystem/internal_filesystem_methods.rs ================================================ // For stuff like initialization and options. // // // ====== // Imports // ====== // // use std::path::PathBuf; use log::debug; use crate::filesystem::filesystem_struct::USE_TUI; use crate::filesystem::filesystem_struct::WRITE_BACKUPS; use crate::pool::pool_actions::pool_struct::Pool; use crate::filesystem::filesystem_struct::FilesystemOptions; use crate::filesystem::filesystem_struct::FlusterFS; use crate::filesystem::filesystem_struct::FLOPPY_PATH; use crate::filesystem::filesystem_struct::USE_VIRTUAL_DISKS; // // // ====== // Implementations // ====== // // // Filesystem option setup. Does not start filesystem. impl FilesystemOptions { /// Initializes options for the filesystem, also configures the virtual disks if needed. pub fn new(use_virtual_disks: Option, floppy_drive: PathBuf, backup: Option, enable_tui: bool) -> Self { debug!("Configuring file system options..."); // Set the globals // set the floppy disk path debug!("Setting the floppy path..."); debug!("Locking FLOPPY_PATH..."); // There's no way anyone else has a lock on this or its poisoned at this point. *FLOPPY_PATH .try_lock() .expect("Fluster! Is single threaded.") = floppy_drive.clone(); debug!("Done."); // Set the virtual disk flag if needed if let Some(path) = use_virtual_disks.clone() { debug!("Setting up virtual disks..."); // Sanity checks // Make sure this is a directory, and that the directory already exists if !path.is_dir() || !path.exists() { // Why must you do this panic!("Virtual disk argument must be a valid path to a pre-existing directory."); } debug!("Locking USE_VIRTUAL_DISKS..."); *USE_VIRTUAL_DISKS .try_lock() .expect("Fluster! Is single threaded.") = Some(path.to_path_buf()); debug!("Done."); }; // Disable backups if needed. // Backups default to being enabled. let enable_backup = backup.unwrap_or(true); debug!("Setting WRITE_BACKUPS..."); WRITE_BACKUPS.set(enable_backup).expect("This should only ever be called once."); debug!("Done."); // Disable tui // TUI is enabled by default debug!("Setting USE_TUI..."); USE_TUI.set(enable_tui).expect("This should only ever be called once."); debug!("Done."); debug!("Done configuring."); Self { use_virtual_disks, floppy_drive, enable_backup, enable_tui, } } } // Starting the filesystem. impl FlusterFS { /// Create new filesystem handle, this will kick off the whole process of loading in information about the pool. /// Takes in options to configure the new pool. pub fn start(options: &FilesystemOptions) -> Self { debug!("Starting file system..."); // Right now we dont use the options for anything, but they do initialize the globals we need, so we still need to pass it in. #[allow(dead_code)] #[allow(unused_variables)] let unused = options; let fs = FlusterFS { pool: Pool::load() }; debug!("Done starting filesystem."); fs } } ================================================ FILE: src/filesystem/item_flag/flag_struct.rs ================================================ use bitflags::bitflags; use libc::c_int; use log::warn; // // // ====== // Flag type // ====== // // // Flags are handled with bare u32 integers, // hence we have a bitflag type to make dealing with them easier. // Open documentation: // https://man7.org/linux/man-pages/man2/openat.2.html // The flags are in libc:: // When it says "Has no effect", I mean on the fluster side. Fluster just does not care // about this flag being set or unset. // I'm pretty sure that the read/write flags do not overlap. If they do I will split this into multiple types. // All of the c flags are i32 for reasons unknown to me, so we have to cast all of them lol // Not sure why the fuse_mt crate uses u32... bitflags! { /// Flags that items have. #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(crate) struct ItemFlag: u32 { /// The file is opened in append mode. /// Before each write, the file offset is positioned at the end of the file. /// The modification of the file offset and write is done as one atomic step. const APPEND = libc::O_APPEND as u32; /// Async, fluster does not support this. Thus we will not /// add this bit to the flags. /// /// Has no effect. const O_ASYNC = libc::O_ASYNC as u32; /// Has to do with closing when executing, ignoring, good luck. /// /// Has no effect const O_CLOEXEC = libc::O_CLOEXEC as u32; /// If the path does not exist, create it as a regular file. const CREATE = libc::O_CREAT as u32; /// Has to do with direct IO. We don't really care, since we have no special /// handling for this kinda thing. /// /// Has no effect. const O_DIRECT = libc::O_DIRECT as u32; /// Fail the open if the path is not a directory. const ASSERT_DIRECTORY = libc::O_DIRECTORY as u32; /// Has to do with data syncing. We do not care. /// /// Has no effect const O_DSYNC = libc::O_DSYNC as u32; /// Ensure that call creates the file. if this is set and O_CREAT is also set, we're /// supposed to turn a EEXIST on open if path already exists. /// /// O_EXCL is undefined if used without O_CREAT (unless pointing at block devices which fluster is not.) const CREATE_EXCLUSIVE = libc::O_EXCL as u32; /// Deals with filesizes with offsets that can be greated than off_t (I think that's 32 bit) /// /// If you need files that big, fluster is not the tool for you. /// Has no effect. const O_LARGEFILE = libc::O_LARGEFILE as u32; /// Do not update file access time. /// /// Cool, we don't support that anyways. /// /// Has no effect. const O_NOATIME = libc::O_NOATIME as u32; /// If path is a terminal device, do not control it or whatever. /// /// Fluster! does not have terminal devices. /// Has no effect. const O_NOCTTY = libc::O_NOCTTY as u32; /// Symbolic link related. We do not support links. // const O_NOFOLLOW = libc::O_NOFOLLOW; /// Open in non-blocking mode. /// Fluster is single threaded. EVERYTHING blocks dawg. /// /// Has no effect. const O_NONBLOCK = libc::O_NONBLOCK as u32; const O_NDELAY = libc::O_NDELAY as u32; // Alternate name for same flag /// Dont follow symlinks. /// /// Fluster does not have symlinks. /// Has no effect. const O_NOFOLLOW = libc::O_NOFOLLOW as u32; // Alternate name for same flag /// Gets file descriptor for this path but not the actual file. /// /// Guess what buddy? you'll just get the whole file regardless. /// /// Has no effect. const O_PATH = libc::O_PATH as u32; /// Do synchronized file I/O. /// /// This is supposed to force sync to disk, but we are silly and don't care :) /// /// Has no effect. const O_SYNC = libc::O_SYNC as u32; /// Creates unnamed tempfiles. /// /// We do not support this. /// Has no effect. const O_TMPFILE = libc::O_TMPFILE as u32; /// If the file already exists, truncate it. /// /// There is already a truncate method on the filesystem, but this may get called elsewhere /// so we still need to care elsewhere. const TRUNCATE = libc::O_TRUNC as u32; } } /// Convert a flag to a u32 for use in returning. impl From for u32 { fn from(value: ItemFlag) -> Self { value.bits() } } /// Tried to convert a u32 into a valid flag, returns an `Unsupported` error if a non-existent flag is set. impl ItemFlag { pub fn deduce_flag(value: u32) -> Result { // All bits must be used. We need to know what they all are. if let Some(valid) = ItemFlag::from_bits(value) { // All good. Ok(valid) } else { // Has invalid bits set. // We will print some information to deduce the unused bits. warn!("Incoming flag bits had unused bits set. This operation is unimplemented."); warn!("Listing known and unknown flags:"); warn!("Known:"); let known = ItemFlag::from_bits_truncate(value); for name in known.iter_names() { warn!("`{}` with value `{}` (binary: `{:0>32b}`)", name.0, name.1.bits(), name.1.bits()) } warn!("Unknown:"); let unknown_bits = value & !ItemFlag::all().bits(); warn!("{unknown_bits:0>32b}"); // now print out the values of those bits warn!("Values for those bits:"); for i in 0..32 { // shift over to mask out the bit let mask: u32 = 1 << i as u32; // if the bit is set, print the value if mask & unknown_bits != 0 { warn!("{mask} (hex {mask:X})") } } // I've spent several hours trying to track down what 0x8000 is supposed to be a flag for, no luck. // I'm just going to assume its some internal flag at this point. We will ignore all flag bits that we dont know. warn!("Continuing anyways, but ignoring those bits may have side effects."); Ok(known) } } } ================================================ FILE: src/filesystem/item_flag/mod.rs ================================================ pub(super) mod flag_struct; ================================================ FILE: src/filesystem/mod.rs ================================================ mod fuse_filesystem_methods; mod internal_filesystem_methods; pub mod filesystem_struct; mod file_handle; mod item_flag; mod file_attributes; pub mod disk_backup; ================================================ FILE: src/filesystem_design/allocation_spec.md ================================================ # Find and allocate blocks We take in a number (u16) of blocks that the caller wishes to reserve. We return a `Result, AllocationError>` `AllocationError` contains types for block situations such as `NotEnoughSpace`, which requires the caller to add more disks to the pool. ### Note: This operation does not flag blocks as used, this section is read only. The caller is responsible for updating the allocation tables in the headers of disks they write to. Finding the next block follows these steps: ## Scanning: Before we can write any data, we need to ensure we have all the room for it. This is the discovery phase, no data is written. Terminology: - `Start Disk` - - The disk where the allocation of blocks begins. (Allocations of blocks always go upwards away from the pool disk). - `Allocation length` - - The number of blocks we wish to allocate. - `Note` - - Copy this value into memory off of the disk for later use. ### Process: - Insert the pool disk, `Note`: `highest_known_disk`, `pool_blocks_free`, and `disk_with_next_free_block`. - - If `disk_with_next_free_block` == u16::MAX: - - - There are no more free blocks. We need another disk. Return `NotEnoughSpace` - - If `pool_blocks_free` < `Allocation length`: - - - There aren't enough free blocks. We need another disk. Return `NotEnoughSpace` - Insert `pool_blocks_free`, hereafter referred to as the `Start disk` - Goto `Find Blocks` - If not enough space is found: - - There aren't enough free blocks in the entire pool. - - This should have been caught by `pool_blocks_free`. - - An assertion will go here. We should never hit this branch. - Update `pool_blocks_free` with how many blocks were allocated - Update `disk_with_next_free_block`: - - `Note`: Disk and Block numbers of the final allocated block - - If the block number is the final block on the disk: - - - Set `disk_with_next_free_block` to u16::MAX. - - - Otherwise, set `disk_with_next_free_block` to the Disk of the final allocated block. ## Find Blocks Incoming arguments: - `Start Disk` - - Disk number to start our search from. - `Allocation length` - - The number of blocks we wish to allocate. - `End Disk` - - The disk number of the final disk in the pool. Returns: - `Vec` Terminology: - Variable `Index` - - The current disk we are examining. (Lies within range `Start Disk`..=`End Disk`) - - Starts at `Start Disk` - Variable `Free blocks seen` - - Keeps track of how many free blocks we have seen across all disks up to this point. - Variable `Block pointers` - - A Vec of ``s to each free block we are considering. - Variable `Blocks remaining` - - A count of how many more blocks we need to allocate ### Note: This section does not flag blocks as used, this section is read only. ### Process: - Insert disk `Index` - Count number of blocks free in allocation table. - If the number of blocks free is >= `Blocks remaining` - - Copy as many disk pointers into `Block pointers` as there are `Blocks remaining`, and return the pointers. - Copy pointers to all of the free blocks into `Block pointers`, decrement `Blocks remaining` accordingly. - If `Index` >= `End Disk` - - There were not enough free blocks. - - This should not be possible. Caller must guarantee that there is enough free space. - - Assertion goes here. - Increment `Index` - Loop ================================================ FILE: src/filesystem_design/dense_disk.md ================================================ # Dense disk Sometimes, you've got a really big file. And I mean REALLY big. If a file is over the size of a full floppy, find out how many full floppies it can span and reserve those the rest of the file will go in data blocks as usual # Disk header format The disk header lives on block 0 of every disk. Header format: | offset | length | Field | | ------ | ------ | ----------------------------------------------------- | | 0 | 8 | Magic number for identifying a fluster drive.Fluster! | | 8 | 1 | Bitflags | | 9 | 2 | Disk number | | - | - | Reserved | | 148 | 360 | Block usage bitplane | | 509 | 4 | CRC | Bitflags: | bit | flag | | --- | ---------------------------------------------- | | 0 | Reserved | | 1 | Reserved | | 2 | Reserved | | 3 | Reserved | | 4 | Reserved | | 5 | Reserved | | 7 | Reserved for Standard disks. Must never be set.| | 6 | Marker bit, Must always be set. | | 8 | Reserved for Pool headers. Must never be set. | ================================================ FILE: src/filesystem_design/design_choices.md ================================================ # Reasons for certain things # Why 4 byte CRCs? After 20MB of read-write with random head seeking, I only got 1 failed byte. A 4 byte crc on our 512 byte block gives us a hamming distance of 6, which is probably even overkill unless the floppy drive is actively being shaken by a pit bull who mistook it for a toddler. # Why little endian? Stack exchange said it was cool. # What order are the bitflags in the documentation? flag 0 is the least significant bit # Why are some reads bigger than they need to be? I was having an issue reading just 8 bytes into a buffer. Turns out Windows wont let you read directly from a floppy disk into a buffer smaller than 512 bytes. This took an annoyingly long time to figure out. # Why do file extent blocks have a `bytes_free` field even though they arent dynamically allocated? Ease of use. # A lot of stuff seems wasteful cpu wise... Think of it this way, 99% of the time we will be waiting for data from disk, so it evens out! # Why is an entire disk dedicated to information about the pool? Chances are, if you are using this filesystem, you are storing many files across many floppies. Finding a file is a slow and tedious process. We have to start from the first disk and search, possibly swapping between many disks before finding the file we are seeking. Most of this overhead comes from looking up the location of the file inode, not loading the file itself. Dedicating an entire disk to pool information lets us keep a cache of file locations, skipping the entire search process. This will result in fewer disk swaps, and a massive speedup in search time. # Why is the project laid out like that? Originally, I didn't want to accidentally give access to private functions used for subsystems, but I ended up repeatedly dividing everything up until I was left with Pool::Disk::(Some disk type) then each disk implements its own innards, or uses generic functions from Pool::Disk. Organizationally, I feel like it makes sense, but the amount of nesting is pretty wild. This is my first time trying to keep a project organized in a sensible way, so... lol. # Why are there so many comments? I've had too many hobby projects in the pass where I've thought to myself, "The code is self documenting". Sure, that might be the case, but the amount of mental effort it takes to understand what's going on 3 days after I wrote something is a LOT higher than if I just left some comments. I'd prefer too many comments over a headache trying to reverse-engineer what I was thinking previously. Also it lowers the bar of entry for the casual viewer :D # Why do some of the higher level abstractions use so many generic traits? While writing this project, I learned more and more about traits, so I started using them because they're cool! ================================================ FILE: src/filesystem_design/disk_header.md ================================================ # Block layout 512 bytes in size A floppy disk can hold 2880 blocks of this size. # Header update situations: New disk is created: - Highest known disk has to be updated on the root disk (disk 0) # Disk header format The disk header lives on block 0 of every disk. Header format: | offset | length | Field | | ------ | ------ | ----------------------------------------------------- | | 0 | 8 | Magic number for identifying a fluster drive.Fluster! | | 8 | 1 | Bitflags | | 9 | 2 | Disk number (u16) | | - | - | Reserved | | 148 | 360 | Block usage bitplane | | 509 | 4 | CRC | Bitflags: | bit | flag | | --- | --------------------------------------------- | | 0 | Reserved | | 1 | Reserved | | 2 | Reserved | | 3 | Reserved | | 4 | Reserved | | 5 | Reserved | | 6 | Marker bit, Must always be set. | | 7 | Reserved for Dense disks. Must never be set. | | 8 | Reserved for Pool headers. Must never be set. | 8 bytes: 1 byte: bitflags 2 bytes: Disk number 138 bytes: Reserved 360 bytes: Block usage bitplane Final 4 byte: crc ================================================ FILE: src/filesystem_design/disk_layout.md ================================================ # Disk Layout Block 0: Disk header // Only required on the origin disk Block 1: Inode block Block 2: Directory block Unless its a dense disk, dense disks only have the header. Remaining blocks: any inode, directory, or data. # Block types Header (See `disk_header`) Inode Directory Data File Extents Data # Data block 1 byte: bitflags 0: Reserved for future use 1: Reserved for future use 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Reserved for future use remaining bytes: raw data final 4 bytes: CRC # Directory block Items on the directory block don't need to be in any specific order, we do not index directly into these blocks. 1 byte: bitflags 0: This is the last directory block on the disk. 1: Reserved for future use 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Reserved for future use 2 bytes: number of free bytes 4 bytes: next directory block (disk pointer, we have no idea where the next directory could be.) - If u16:MAX then this is the end of the directory chain remaining bytes: directory data final 4 bytes: CRC # File Extents block 1 byte: bitflags 0: Reserved for future use 1: Reserved for future use 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Reserved for future use 2 bytes: number of free bytes 4 bytes: Next block - 2 Bytes: Disk number - 2 Bytes: Block on disk - if all 4 bytes are full 1's, this is the final block remaining bytes: extent data final 4 bytes: CRC # Inode block 1 byte: bitflags 0: This is the last inode block on the disk. 1: Reserved for future use 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Reserved for future use 2 bytes: number of free bytes 4 bytes: next directory block (disk pointer, we have no idea where the next directory could be.) - If u16:MAX then this is the end of the directory chain remaining bytes: inode data final 4 bytes: CRC If you are on the final inode disk and realize you need to make another inode block, you update the bitflag and reserve another block. If you are out of blocks on that disk, go to the next disk if bit 1 is set. If bit 1 is not set, then you can simply go to the next disk indicated. Otherwise you must find a _NEW_ disk to put the next inode block on and update flags accordingly. (New disk inodes must be in the default position) On disk 0, the first inode in the block MUST be a directory referencing `/` aka the root. ================================================ FILE: src/filesystem_design/inode_format.md ================================================ # Example Traversal Lets find `/foo/bar.txt` ### Start at Root Inode The location of the root inode is fixed (Disk 0, Block 1, Slot 0). Read the Inode at this address. It's a Directory type. ### Find Root's Directory Data From the root Inode, get the pointer to its Directory block. ### Scan Root's Directory Data for `foo` Read that DirectoryDataBlock. Search its children map for the key `foo`. If found, you get the InodeAddress for `foo`. If not found and there's a next_block pointer, follow the chain and repeat the search. ### Find `foo`'s Directory Data Read the Inode at `foo`'s address. It's also a Directory type. From this Inode, get the pointer to its Directory block. ### Scan `foo`'s Directory Data for `bar.txt` Read the DirectoryDataBlock for foo. Search its children map for the key `bar.txt`. If found, you get the InodeAddress for `bar.txt`. ### Find `bar.txt`'s Extents Read the Inode at `bar.txt`'s address. It's a File type. From this Inode, get the pointer to its first_extent_block. ### Read File Data Read that FileExtentBlock to get the list of FileExtents, which finally tell you which blocks on which disks hold the actual file data. Extents in this file are in order. # Inode format 1 byte: bitflags - 0: File type (0 directory, 1 file) - 1: Reserved for future use - 2: Reserved for future use - 3: Reserved for future use - 4: Reserved for future use - 5: Reserved for future use - 6: Reserved for future use - 7: Marker bit (Always set) 4-12 bytes: Inode data * File: - 8 bytes for size - 4 bytes for pointer to the File Extents block - 2 Bytes: Disk number - 2 Bytes: Block on disk * Directory: - 4 bytes for pointer to Directory Data block - 2 Bytes: Disk number - 2 Bytes: Block on disk 12 bytes: Created timestamp - 8 bytes: Seconds since epoch - 4 bytes: nanosecond offset 12 bytes: Modified timestamp - 8 bytes: Seconds since epoch - 4 bytes: nanosecond offset # Inode block see `disk_layout` # Directory block see `disk_layout` # Directory item format 1 byte: bitflags 0: Inode is on this disk 1: Reserved for future use 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Marker bit (Always set) 1 byte: length of item name ? bytes: item name 3-5 bytes: inode location - 2 Bytes: Disk number (Not included if flag set) - 2 Bytes: Block on disk - 1 Byte: Index into inode block # File Extents block see `disk_layout` # Extent format bitflag considerations: - You cannot have a local dense-disk. 1 byte: bitflags 0: This extent is a dense-disk 1: The block is on this disk 2: Reserved for future use 3: Reserved for future use 4: Reserved for future use 5: Reserved for future use 6: Reserved for future use 7: Marker bit (Always set) 2-5 Bytes: extent information - 2 Bytes: Disk number (Not included if block is local) - 2 Bytes: Start block (Not included if dense) - 1 Byte: Length (Not included if dense) ================================================ FILE: src/filesystem_design/pool_header.md ================================================ # Pool header The root disk only holds information about the pool. Blocks cannot be stored to this disk. | Offset | Length | Field | | ------ | ------ | ---------------------------------------------------------------------------------------------- | | 0 | 8 | Magic number for idenifying a fluster drive `Fluster!` | | 8 | 1 | Bitflags | | 9 | 2 | Highest known disk number. | | 11 | 2 | Disk with the next free block in the pool.
Set to u16::MAX if the final disk has no room. | | 13 | 4 | Number of blocks free across all disks in the pool. | | - | - | Reserved | | 148 | 360 | Block usage bitplane | | 509 | 4 | Block CRC | Bitflags: | bit | flag | | --- | ----------------------------------------- | | 0 | Reserved | | 1 | Reserved | | 2 | Reserved | | 3 | Reserved | | 4 | Reserved | | 5 | Reserved | | 6 | Reserved | | 7 | Reserved | | 8 | Marks this as a pool header. Must be set. | ================================================ FILE: src/filesystem_design/pool_layout.md ================================================ # The pool The pool will be our highest level of abstraction on top of the disks, every action against the underlying disks should be done through the pool. ================================================ FILE: src/filesystem_design/possible_speed_improvments.md ================================================ # Speed improvement ideas # Disk pre-seek If we know we are about to change disks, is it possible to pre-align the head of the drive to the next block we will read on the next disk while the user swaps? # In-memory inode cache This might be practically mandatory to get any usability out of the file system unless we want to be swapping tons of disks for every read operation. You should be able to enable or disable this on the fly if you want. ================================================ FILE: src/helpers/hex_view.rs ================================================ // Take in a vec of bytes and return a hex view of it pub fn hex_view(bytes: Vec) -> String { let mut offset = 0; let bytes_length = bytes.len(); let mut screen_string = String::new(); // push the header screen_string.push_str(" Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F\n"); while offset < bytes_length { // make the line let mut line = String::new(); // first goes the offset, padded so its 10 characters long line.push_str(&format!("{offset:0>10X} ")); // now for all the numbers for i in 0..16 { // skip if we are outside of range if offset + i >= bytes_length { line.push_str(" "); } else { let byte = bytes[offset + i]; let byte_component = format!("{byte:02X} "); line.push_str(&byte_component); } } // now for the text version line.push(' '); for i in 0..16 { let mut character: char; if offset + i >= bytes_length { character = ' '; } else { // convert let byte = bytes[offset + i]; character = char::from_u32(byte as u32).unwrap_or('?'); // unless: if !character.is_ascii() || character.is_ascii_control() { character = '.'; } } line.push(character); } // line is done. Add it to the screen screen_string.push_str(&line); screen_string.push('\n'); // Now increment the offset offset += 16; } // done! screen_string } ================================================ FILE: src/helpers/mod.rs ================================================ pub mod hex_view; ================================================ FILE: src/lib.rs ================================================ // The library/filesystem cannot use unwraps. #![deny(clippy::unwrap_used)] // Asserts need to have a reason. #![deny(clippy::missing_assert_message)] // Gotta use all the results. #![deny(unused_results)] // I need to force some methods to only be used in special places. // Doing the publicity for it would be a pain, so we just piggyback on // depreciated #![deny(deprecated)] // Only use the filesystem in main.rs // We only support 64 bit systems. since we expect usize to be that size. #[cfg(target_pointer_width = "64")] pub mod filesystem; // Within the crate, we can use: mod helpers; mod error_types; mod pool; pub mod tui; ================================================ FILE: src/main.rs ================================================ use std::{ ffi::OsStr, path::PathBuf, sync::{ atomic::{ AtomicBool, Ordering }, Arc }, thread, time::Duration }; use clap::Parser; use fluster_fs::{ filesystem::filesystem_struct::{ FilesystemOptions, FlusterFS }, tui::notify::TUI_MANAGER }; // Logging use env_logger::Env; use ratatui::{ crossterm::{ execute, terminal::{ disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen } }, prelude::CrosstermBackend, Terminal }; use tui_logger::TuiLoggerFile; #[derive(Parser)] struct Cli { /// Path to the floppy block device. #[arg(long)] block_device_path: String, /// The mount point to mount the Fluster pool. #[arg(long)] mount_point: String, /// Run with virtual floppy disks for testing. Path to put tempfiles in. #[arg(long)] use_virtual_disks: Option, /// Make backups of disks in /var/fluster. Disabling this VERY unsafe, you should /// leave this on unless you are doing testing or don't care that much about your data. #[arg(long)] enable_disk_backup: Option, /// Disable the TUI interface. #[arg(long)] disable_tui: Option, } fn main() { // Get cli arguments let cli = Cli::parse(); // get the mount point let mount_point = PathBuf::from(cli.mount_point); // Start the logger // If we are using the tui, we need to use the TUI logger instead of env. if !cli.disable_tui.unwrap_or(false) { // use the tui tui_logger::init_logger(log::LevelFilter::Debug).unwrap(); let log_level = std::env::var("RUST_LOG") .map(|s| match s.to_lowercase().as_str() { "error" => log::LevelFilter::Error, "warn" => log::LevelFilter::Warn, "info" => log::LevelFilter::Info, "debug" => log::LevelFilter::Debug, "trace" => log::LevelFilter::Trace, _ => log::LevelFilter::Debug, }) .unwrap_or(log::LevelFilter::Debug); tui_logger::set_default_level(log_level); // Also write out to a file. tui_logger::set_log_file(TuiLoggerFile::new( &mount_point.parent() .expect("You really shouldn't be mounting fluster as `/` lmao.") .join("flusterlog.txt") .display() .to_string() )); } else { // normal logger env_logger::Builder::from_env(Env::default().default_filter_or("debug")).init(); } // Log panics log_panics::Config::default().backtrace_mode(log_panics::BacktraceMode::Resolved).install_panic_hook(); // Set up Ctrl+C handler ctrlc::set_handler(move || { println!("Fluster cannot be closed with ctrl+c. You need to unmount the filesystem with `fusermount -u (path)`."); println!("Busy? Close everything that may be looking at the filesystem."); println!("Still busy? Too bad, wait it out. Or suffer data loss. Your choice."); println!("Ignoring..."); }) .unwrap(); // Check if the mount point is valid std::fs::create_dir_all(&mount_point).unwrap(); // Assemble the options let use_virtual_disks: Option = cli.use_virtual_disks.map(PathBuf::from); let backup: Option = cli.enable_disk_backup; let enable_tui = !cli.disable_tui.unwrap_or(false); let options: FilesystemOptions = FilesystemOptions::new(use_virtual_disks, cli.block_device_path.into(), backup, enable_tui); // Now before starting the filesystem, we need to start the TUI if needed. // We also need to be able to tell the TUI to shut down when we unmount. let shutdown_tui = Arc::new(AtomicBool::new(false)); let tui_thread_handle = if enable_tui { let signal = Arc::clone(&shutdown_tui); // Spawn a thread that handles the TUI Some(thread::spawn(move || { // Set up the terminal window let mut terminal = Terminal::new(CrosstermBackend::new(std::io::stdout())).unwrap(); // Swap to another screen to preserve old terminal content execute!(terminal.backend_mut(), EnterAlternateScreen).unwrap(); // Raw mode for tui stuff enable_raw_mode().unwrap(); // Wait a bit for Fluster! to start up and initialize things thread::sleep(Duration::from_millis(100)); // Rendering loop, we do terminal cleanup when told to. while !signal.load(Ordering::Relaxed) { // Try and lock the TUI, if we cant, we'll just skip this frame. if let Ok(mut tui) = TUI_MANAGER.try_lock() { // Draw it! // We ignore if the drawing fails, we'll just try it again. let _ = terminal.draw(|frame| tui.draw(frame)); } // Now wait before drawing the next frame. thread::sleep(Duration::from_millis(16)); // just above 60fps } // We've broken from the loop, time to shut down the TUI. execute!(terminal.backend_mut(), LeaveAlternateScreen).unwrap(); disable_raw_mode().unwrap(); })) } else { None }; let filesystem: FlusterFS = FlusterFS::start(&options); // Now for the fuse mount options let fuse_options = [ OsStr::new("-onodev"), // Disable dev devices OsStr::new("-onoatime"), // No access times OsStr::new("-onosuid"), // Ignore file/folder permissions (lol) OsStr::new("-orw"), // Read/Write OsStr::new("-oexec"), // Files are executable OsStr::new("-osync"), // No async. OsStr::new("-odirsync"), // No async OsStr::new("-oallow_other"), // Allow other users to open the mount point (ie windows outisde of WSL) OsStr::new("-ofsname=fluster"), // Set the name of the fuse mount ]; // Mount it // Internal fuse_mt startup stuff i think, no comments on the function implementation. // takes in the filesystem, and the number of threads the filesystem will use // Fluster! Is single threaded, so we actually set it to 0 threads. Weirdly. let mt_thing = fuse_mt::FuseMT::new(filesystem, 0); match fuse_mt::mount(mt_thing, &mount_point, &fuse_options) { Ok(_) => { // Filesystem was unmounted successfully. println!("Fluster! has been unmounted."); }, Err(err) => { // rhut row println!("Fluster is dead and you killed them. {err:#?}"); }, } // Clean up the TUI. if let Some(handle) = tui_thread_handle { shutdown_tui.store(true, Ordering::Relaxed); handle.join().expect("TUI rendering thread failed to join on shutdown! But Fluster otherwise shut down successfully."); } } ================================================ FILE: src/pool/disk/blank_disk/blank_disk_methods.rs ================================================ // Yep. use std::fs::File; use log::error; use crate::{ error_types::drive::DriveError, pool::disk::{ blank_disk::blank_disk_struct::BlankDisk, generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::write::write_block_direct } } }; impl GenericDiskMethods for BlankDisk { #[doc = " Read a block"] #[doc = " Cannot bypass CRC."] fn unchecked_read_block(&self, _block_number: u16) -> Result { // We should NEVER read a block from a blank disk, why would we do that? unreachable!("Attempted to read a block from a blank disk! Not allowed! You need to turn it into another type first!") } #[doc = " Write a block"] fn unchecked_write_block(&mut self, block: &RawBlock) -> Result<(), DriveError> { // This is the first call, we have not recursed. write_block_direct(&self.disk_file, block, false) } #[doc = " Get the inner file used for IO operations"] fn disk_file(self) -> File { self.disk_file } #[doc = " Get the number of the floppy disk."] fn get_disk_number(&self) -> u16 { // Why are we getting the disk number of a blank floppy? error!("Attempted to get the disk number of a blank disk! Not allowed!"); // We will ignore the action and return a nonsensical number, this prevents fluster // from crashing if you have a disk blank disk in the drive after finishing troubleshooting. u16::MAX } #[doc = " Set the number of this disk."] fn set_disk_number(&mut self, _disk_number: u16) -> () { // You cannot set the number of a blank disk. // Trying to set the disk number is doomed to fail, because at this point it thinks its an initialized disk, which it is not. unreachable!("Attempted to set the disk number of a blank disk! Not allowed!") } #[doc = " Get the inner file used for write operations"] fn disk_file_mut(&mut self) -> &mut File { &mut self.disk_file } #[doc = " Sync all in-memory information to disk"] fn flush(&mut self) -> Result<(), DriveError> { // There is no in-memory information for this disk. // So we can safely ignore this. Ok(()) } #[doc = " Write chunked data, starting at a block."] fn unchecked_write_large(&mut self, data:Vec, start_block:DiskPointer) -> Result<(), DriveError> { crate::pool::disk::generic::io::write::write_large_direct(&self.disk_file, &data, start_block) } #[doc = " Read multiple blocks"] #[doc = " Does not check CRC!"] fn unchecked_read_multiple_blocks(&self, _block_number: u16, _num_block_to_read: u16) -> Result,DriveError> { unreachable!("Attempted to read a block from a blank disk! Not allowed! You need to turn it into another type first!") } } // Occasionally we need a new blank disk impl BlankDisk { pub fn new(file: File) -> Self { Self { disk_file: file } } } impl BlockAllocation for BlankDisk { #[doc = " Get the block allocation table"] fn get_allocation_table(&self) -> &[u8] { unreachable!("Block allocation is not supported on blank disks.") } #[doc = " Update and flush the allocation table to disk."] fn set_allocation_table(&mut self, _new_table: &[u8]) -> Result<(), DriveError> { unreachable!("Block allocation is not supported on blank disks.") } } ================================================ FILE: src/pool/disk/blank_disk/blank_disk_struct.rs ================================================ // Need for type constraints #[derive(Debug)] pub struct BlankDisk { /// Every disk type needs a file! pub(in super::super) disk_file: std::fs::File, } ================================================ FILE: src/pool/disk/blank_disk/mod.rs ================================================ pub mod blank_disk_methods; pub mod blank_disk_struct; ================================================ FILE: src/pool/disk/drive_methods.rs ================================================ // Methods that are generic across all types of disk. // Using the floppy drive interface should work like this: // Request a disk, get back a DiskType that matches the number provided. // Imports use log::error; use log::trace; use log::warn; use crate::error_types::conversions::CannotConvertError; use crate::error_types::critical::CriticalError; use crate::error_types::critical::RetryCapError; use crate::error_types::drive::DriveError; use crate::error_types::drive::DriveIOError; use crate::error_types::drive::WrappedIOError; use crate::helpers::hex_view::hex_view; use crate::pool::disk::blank_disk::blank_disk_struct::BlankDisk; use crate::pool::disk::drive_struct::DiskBootstrap; use crate::pool::disk::generic::block::block_structs::RawBlock; use crate::pool::disk::generic::disk_trait::GenericDiskMethods; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; // The cache is NOT allowed in here at all, since any writes happen through the cache regardless. // Thus if we are loading in a disk, this is a real swap. // use crate::pool::disk::generic::io::cache::cache_io::CachedBlockIO; use crate::pool::disk::generic::io::read::read_block_direct; use crate::pool::disk::generic::io::wipe::destroy_disk; use crate::pool::disk::standard_disk::standard_disk_struct::StandardDisk; use crate::pool::disk::pool_disk::pool_disk_struct::PoolDisk; use crate::filesystem::filesystem_struct::FLOPPY_PATH; use crate::filesystem::filesystem_struct::USE_VIRTUAL_DISKS; use crate::pool::disk::unknown_disk::unknown_disk_struct::UnknownDisk; use crate::tui::notify::NotifyTui; use crate::tui::prompts::TuiPrompt; use super::drive_struct::DiskType; use super::drive_struct::FloppyDrive; use std::fs::File; use std::fs::OpenOptions; use std::sync::atomic::AtomicU16; use std::sync::atomic::Ordering; // Disk tracking global. // To better count disk swaps, we need to know what the most recently opened disk was static CURRENT_DISK_IN_DRIVE: AtomicU16 = AtomicU16::new(u16::MAX); // Implementations /// Various operations on the underlying Disk. /// This is meant to be high level, just enough to get to the disk type below. impl FloppyDrive { /// Open the disk currently in the drive, regardless of disk type. /// This should only be used when initializing the pool. Use open() instead. pub fn open_direct(disk_number: u16) -> Result { // This function does not create disks. open_and_deduce_disk(disk_number, false) } /// Opens a specific disk, or waits until the user inserts that disk. #[deprecated ="You should be using the cache! Unless you are using this in the cache."] pub fn open(disk_number: u16) -> Result { prompt_for_disk(disk_number) } /// Prompts the user for a blank floppy disk. pub fn get_blank_disk(disk_number: u16) -> Result { prompt_for_blank_disk(disk_number) } /// Find out what disk is currently in the drive. pub fn currently_inserted_disk_number() -> u16 { CURRENT_DISK_IN_DRIVE.load(Ordering::Relaxed) } } // Functions for implementations fn open_and_deduce_disk(disk_number: u16, new_disk: bool) -> Result { trace!("Opening and deducing disk disk {disk_number}..."); trace!("Is it a new disk? : {new_disk}"); // First, we need the file to read from let disk_file: File = get_floppy_drive_file(disk_number, new_disk)?; // Now we must get the 0th block // We need to read a block before we have an actual disk, so we need // to call this function directly as a workaround. // This also must be called directly, since we cannot use the cache here. // The cache expects to not be accessed while doing flushes, which requires all // calls that load information about disks to not access the cache. // We must ignore the CRC here, since we know nothing about the disk. trace!("Reading in the header at block 0..."); let header_block: RawBlock = read_block_direct(&disk_file, disk_number, 0, true, false)?; // Now we check for the magic trace!("Checking for magic..."); if !check_for_magic(&header_block.data) { trace!("No magic, checking if its blank..."); // The magic is missing, check if the block is empty if header_block.data.iter().all(|byte| *byte == 0) { // Block is completely blank. trace!("Disk is blank, returning."); return Ok(DiskType::Blank(BlankDisk::new(disk_file))); } // Otherwise, we dont know what kind of disk this is. // Its probably not a fluster disk. trace!("Disk was not blank, returning unknown disk..."); return Ok(DiskType::Unknown(UnknownDisk::new(disk_file))); } // Magic exists, time to figure out what kind of disk this is. trace!("Disk has magic, deducing type..."); // Bitflags will tell us. // Pool disk. // The header reads should check the CRC of the block. if header_block.data[8] & 0b10000000 != 0 { trace!("Head is for a pool disk, returning."); return Ok(DiskType::Pool(PoolDisk::from_header( header_block, disk_file, ))); } // Standard disk. if header_block.data[8] & 0b00100000 != 0 { trace!("Head is for a standard disk, returning."); return Ok(DiskType::Standard(StandardDisk::from_header( header_block, disk_file, ))); } // it should be impossible to get here error!("Hexdump:\n{}", hex_view(header_block.data.to_vec())); error!("We cannot continue with an un-deducible disk!"); panic!("Header of disk did not match any known disk type!"); } /// Get the path of the floppy drive fn get_floppy_drive_file(disk_number: u16, new_disk: bool) -> Result { // If we are running with virtual disks enabled, we are going to use a temp folder instead of the actual disk to speed up // development, waiting for disk seeks is slow and loud lol. if let Ok(maybe_path) = USE_VIRTUAL_DISKS.try_lock() { if let Some(virtual_disk_path) = maybe_path.clone() { // Virtual disks are enabled. trace!("Attempting to access virtual disk {disk_number}..."); trace!("Are we creating this disk? : {new_disk}"); // Get the tempfile. // These files do not delete themselves. // if disk 0 is missing, we need to make it, // because the pool cannot create disk 0 without first loading itself... from disk 0. // This is for virtual disks, so if this fails its on the user. // If using virtual disks fails, we immediately bail. if OpenOptions::new() .read(true) .write(true) .create(true) .truncate(false) .open(virtual_disk_path.join("disk0.fsr")).is_err() { // No good. panic!("You are in-charge of making virtual disks work."); }; // If the tempfile does not exist, that means `create` was never called, which is an issue. // This will create the disk if the correct argument is passed. trace!("Opening the temp disk with read/write privileges..."); let temp_disk_file = if let Ok(file) = OpenOptions::new() .read(true) .write(true) .create(new_disk) // We will panic if the disk does not exist, unless told to create it. .truncate(false) .open(virtual_disk_path.join(format!("disk{disk_number}.fsr"))) { file } else { // Failed. panic!("Disks should be created before read."); }; // Make sure the file is one floppy big, should have no effect on pre-existing files, since // they will already be this size. trace!("Attempting to resize the temporary file to floppy size..."); // This is for virtual disks, so if this fails its on the user. if temp_disk_file.set_len(512 * 2880).is_err() { panic!("If you're using virtual disks, you should be able to resize the virtual disks."); } trace!("Returning virtual disk."); return Ok(temp_disk_file); } } // Get the global path to the floppy disk drive let disk_path = if let Ok(path) = FLOPPY_PATH.try_lock() { path.clone() } else { // Poison? In MY drive method? // _It's more common than you think!_ // I REALLY hope we're shutting down at this point. // We need the disk path to be able to flush contents to disk upon panic, so we have to clean this up. error!("FLOPPY_PATH is poisoned! Clearing, but you REALLY need to shut down immediately."); FLOPPY_PATH.clear_poison(); if let Ok(round_two) = FLOPPY_PATH.try_lock() { round_two.clone() } else { // Err... panic!("Failed to clear poison!"); } }; // Open the disk, or return an error from it. // Before that though, make sure the block device we are trying to // access is actually mounted if !std::fs::exists(&disk_path).unwrap_or(false) { // Bound to fail. return Err(DriveError::DriveEmpty) } // We will try 10 times. // If we fail to open the floppy drive file, there's a bigger issue than this function can deal with. for _ in 0..10 { // Open the file. let open_attempt = OpenOptions::new().read(true).write(true).open(&disk_path); // If it opened, return, otherwise we need to handle the IO error. let io_error = match open_attempt { Ok(ok) => return Ok(ok), Err(err) => err, }; // That did not work, see if we can cast up the error let pointer = DiskPointer { disk: disk_number, block: 0, }; // Try converting that up to a DriveError let wrapped: WrappedIOError = WrappedIOError::wrap(io_error, pointer); let drive_io_error: Result = DriveIOError::try_from(wrapped); // Did that work? let converted = match drive_io_error { Ok(ok) => ok, Err(err) => { // Looks like we need to handle this ourselves. match err { CannotConvertError::MustRetry => { // Look's like we're trying again! continue; }, } }, }; // The conversion worked, can we get it up to a DriveError? let drive_error: Result = DriveError::try_from(converted); // Did that also work? If so, throw it! match drive_error { Ok(ok) => return Err(ok), // lol Err(err) => { match err { CannotConvertError::MustRetry => { continue; }, } }, }; }; drop(disk_path); // We've failed 10 times. Nothing we can do. // We can probably recover for this assuming the critical handler can either rebuild the disk // or somehow make it writable again CriticalError::OutOfRetries(RetryCapError::OpenDisk).handle(); // If that works, recurse, we should be able to get the file now. get_floppy_drive_file(disk_number, new_disk) } /// Look for the magic "Fluster!" string. pub fn check_for_magic(block_bytes: &[u8]) -> bool { // is the "Fluster!" magic present? block_bytes[0..8] == *"Fluster!".as_bytes() } /// Prompt user to insert the disk we want. /// If the disk is already in the drive, no prompt will happen. /// Will error out for non-wrong disk related issues. /// This function does not disable the CRC check, you must use open() if you are ignoring CRC. fn prompt_for_disk(disk_number: u16) -> Result { trace!("Prompting for disk {disk_number}..."); let mut is_user_an_idiot: bool = false; // Did the user put in the wrong disk when asked? let mut disk: DiskType; loop { // Try opening the current disk. // We do not create disks here. disk = open_and_deduce_disk(disk_number, false)?; // Obviously, a blank disk cannot be the right disk, since it has // no disk number. if let DiskType::Blank(_) = disk { // what TuiPrompt::prompt_enter( "Wrong disk".to_string(), "This disk is blank. Try again.".to_string(), true ); continue; } // Is this the correct disk? let new_disk_number = disk.get_disk_number(); // Update the current disk if needed let previous_disk = CURRENT_DISK_IN_DRIVE.load(Ordering::Relaxed); if new_disk_number != previous_disk { // We have swapped disks. // Inform the TUI. It's in charge of tracking swaps. NotifyTui::disk_swapped(new_disk_number); CURRENT_DISK_IN_DRIVE.store(new_disk_number, Ordering::Relaxed); } // Check if this is the right disk number if disk_number == new_disk_number { // Thats the right disk! trace!("Got the correct disk."); return Ok(disk); } warn!("Wrong disk received. Got disk {}", disk.get_disk_number()); // This was not the right disk. // We should ALWAYS get the correct disk when testing. #[cfg(test)] if cfg!(test) { error!("Got an invalid disk during a test!"); panic!("Test received an invalid disk!"); } // Prompt user to swap disks. // But we don't prompt if the read failed, since we want to silently retry it. if is_user_an_idiot { println!("Wrong disk. Try again."); } else { is_user_an_idiot = true; } // If this prompt fails, either there's an issue with the disk, or the user didn't respond in time let result = TuiPrompt::prompt_wait_for_disk_swap( "Please swap disks.".to_string(), format!("Please remove disk {previous_disk}, and insert disk {disk_number}"), true ); match result { Ok(ok) => ok, Err(_) => { // We'll launch the troubleshooter regardless of error type, // since we really need swapping disks to work. CriticalError::OutOfRetries(RetryCapError::OpenDisk).handle(); // Then we just try the swap again. }, } } } // get a blank disk fn prompt_for_blank_disk(disk_number: u16) -> Result { // Pester user for a blank disk let mut try_again: bool = false; // If we are on virtual disks, skip the initial prompt let use_virtual: bool = if let Ok(locked) = USE_VIRTUAL_DISKS.try_lock() { locked.is_some() } else { // Poisoned. We should not be adding new disks after being poisoned. We should be shutting down. // Just give up, if we're trying to do that, chances are we just cannot shut down. panic!("Attempted to get a new, blank disk for a poisoned pool! Not allowed!"); }; if !use_virtual { TuiPrompt::prompt_wait_for_disk_swap( "New disk.".to_string(), format!("Creating a new disk, please insert a blank disk that will become disk {disk_number}."), true )?; } loop { if try_again { let action = TuiPrompt::prompt_input( "Disk is not blank.".to_string(), "That disk is not blank. Please insert a blank disk, then hit enter. Or type \"wipe\" to forcibly wipe this disk.".to_string(), false ); if action.contains("wipe") { // go wipe that disk let mut wipe_me = open_and_deduce_disk(disk_number, false)?; destroy_disk(wipe_me.disk_file_mut())?; drop(wipe_me); } } // we are making a new disk, so we must specify as such. let mut disk = open_and_deduce_disk(disk_number, true)?; match disk { // if its blank, all done DiskType::Blank(blank_disk) => return Ok(blank_disk), _ => { // But if the disk is not blank, display_info_and_ask_wipe(&mut disk)?; // try again try_again = true; continue; } } } } /// Takes in a non-blank disk and displays info about it, then asks the user if they would like to wipe the disk. /// Wipes the disk if the user asks, returns nothing. /// Will also return nothing if the user does not wipe the disk. pub fn display_info_and_ask_wipe(disk: &mut DiskType) -> Result<(), DriveError> { // This isn't a very friendly interface, but it'll do for now. // Display the disk type let answer = TuiPrompt::prompt_input( "Disk is not blank.".to_string(), format!("The disk inserted is not blank. It is of type `{disk:?}`.\nWould you like to wipe this disk?\n\"yes\"/\"no\":"), false ).to_ascii_lowercase().contains("yes"); if answer { // Wipe time! // If this fails, inform user. if destroy_disk(disk.disk_file_mut()).is_err() { TuiPrompt::prompt_enter( "Wipe failed!".to_string(), "Failed to wipe that disk! It's probably bad.".to_string(), true ); } } else { // No wipe. TuiPrompt::prompt_enter( "Wipe canceled.".to_string(), "Okay, this disk will not be wiped.".to_string(), false ); } Ok(()) } ================================================ FILE: src/pool/disk/drive_struct.rs ================================================ // I think I slipped a disk. // Imports use crate::{error_types::drive::DriveError, pool::disk::{ blank_disk::blank_disk_struct::BlankDisk, unknown_disk::unknown_disk_struct::UnknownDisk, }}; use std::fs::File; use enum_dispatch::enum_dispatch; use crate::pool::disk::{ generic::block::block_structs::RawBlock, pool_disk::pool_disk_struct::PoolDisk, standard_disk::standard_disk_struct::StandardDisk, }; // Structs, Enums, Flags /// The floppy drive /// The FloppyDrive type doesn't contain anything itself, its just an interface for /// retrieving the various types of disk. pub struct FloppyDrive { // Nothing! This type is just for methods. } /// The different types of disks contained within a pool. /// This contains disk info. #[enum_dispatch] #[derive(Debug)] pub enum DiskType { Pool(PoolDisk), Standard(StandardDisk), Unknown(UnknownDisk), Blank(BlankDisk), } // /// We also have another type that does not contain the disk info. // #[derive(Debug, Clone, Copy, PartialEq, Eq)] // pub enum JustDiskType { // Pool, // Standard, // Unknown, // Blank, // } // Let us match the two // impl PartialEq for DiskType { // fn eq(&self, other: &JustDiskType) -> bool { // match self { // DiskType::Pool(_) => matches!(other, JustDiskType::Pool), // DiskType::Standard(_) => matches!(other, JustDiskType::Standard), // DiskType::Unknown(_) => matches!(other, JustDiskType::Unknown), // DiskType::Blank(_) => matches!(other, JustDiskType::Blank), // } // } // } /// All disk types need to be able to create themselves from a raw block. /// Or, be able to create themselves from a blank disk. /// We also need to create fake disks to allow creating disks (confusing eh?) pub trait DiskBootstrap { /// Create brand new disk. /// This takes in a blank floppy disk, and does all the needed setup on the disk, /// such as writing the header, and other block setup. fn bootstrap(file: File, disk_number: u16) -> Result where Self: std::marker::Sized; /// Create self from incoming header block and file. fn from_header(block: RawBlock, file: File) -> Self; } ================================================ FILE: src/pool/disk/generic/block/allocate/block_allocation.rs ================================================ // Find, reserve, or even free blocks! // We do not allow these operations to be misused, if invalid state is provided, we panic. // We will not: // free bytes that are already free // allocate bytes that are already allocated // allocate past the end of the table use std::cmp::min; use enum_dispatch::enum_dispatch; use crate::{ error_types::drive::DriveError, pool::disk::drive_struct::DiskType }; use crate::pool::disk::pool_disk::pool_disk_struct::PoolDisk; use crate::pool::disk::standard_disk::standard_disk_struct::StandardDisk; use crate::pool::disk::unknown_disk::unknown_disk_struct::UnknownDisk; use crate::pool::disk::blank_disk::blank_disk_struct::BlankDisk; use log::{ debug, trace }; // To be able to allocate blocks, we need a couple things #[enum_dispatch(DiskType)] pub trait BlockAllocation { /// Get the block allocation table fn get_allocation_table(&self) -> &[u8]; /// Update and flush the allocation table to disk. fn set_allocation_table(&mut self, new_table: &[u8]) -> Result<(), DriveError>; /// Attempts to find free blocks on the disk. /// Returns indexes for the found blocks, or returns the number of blocks free if there is not enough space. fn find_free_blocks(&self, blocks: u16) -> Result, u16> { go_find_free_blocks(self, blocks) } /// Allocates the requested blocks. /// Will panic if fed invalid data. fn allocate_blocks(&mut self, blocks: &Vec) -> Result { go_allocate_or_free_blocks(self, blocks, true) } /// Frees the requested blocks. /// Will panic if fed invalid data. fn free_blocks(&mut self, blocks: &Vec) -> Result { go_allocate_or_free_blocks(self, blocks, false) } /// Check if a specific block is allocated fn is_block_allocated(&self, block_number: u16) -> bool { go_check_block_allocated(self, block_number) } } fn go_find_free_blocks( caller: &T, blocks_requested: u16, ) -> Result, u16> { // The allocation table is a stream of bits, the first bit is the 0th block. // Vector of free block locations. // Pre-allocated, expecting that we get all of blocks. But we obviously could not find // more than 2880 free blocks per disk. let mut free: Vec = Vec::with_capacity(min(2880_u16, blocks_requested) as usize); // Now loop through the table looking for free slots. for (byte_index, byte) in caller.get_allocation_table().iter().enumerate() { // loop over the bits for sub_bit in 0..8 { // check if the furthest left bit is free. // we shift over to the bit we want, then we AND it to check if the highest bit is set. // Since we know the bit on one side of the AND is always set, the result will be 0 if the bit is unset. // Thus, the result of the if statement will be `0` if the block is free. // Could this be done cleaner? Maybe, I'm not very experienced with bitwise operations. if (byte << sub_bit) & 0b10000000 == 0 { // bit isn't set, the block is free! free.push((byte_index as u16 * 8) + sub_bit); // Do we have enough blocks now? if free.len() == blocks_requested.into() { // Yep! return Ok(free); } } } } // We've ran out of bytes. We must not have enough free room. Err(free.len() as u16) } /// allocate false frees the provided bytes. fn go_allocate_or_free_blocks( caller: &mut T, blocks: &Vec, allocate: bool, ) -> Result { debug!( "Attempting to {} {} blocks on the current disk...", if allocate { "Allocate" } else { "free" }, blocks.len() ); // If the user provides a vec with a duplicate item, we will panic from double free / double allocate // Vec ordering does not matter, as we calculate the offset from each item // The user must allocate/free at least one block, and that block cannot be past the end of the table. assert!(*blocks.last().expect("Should allocate at least 1 block.") < 2880, "Tried to free a block past the end of the disk!"); // Table to edit // 2880 blocks / 8 blocks per bit = 360 let mut new_allocation_table: [u8; 360] = [0u8; 360]; new_allocation_table.copy_from_slice(caller.get_allocation_table()); trace!("Updating blocks..."); for block in blocks { // Get the bit // Integer division rounds towards zero, so this is fine. let byte: usize = (block / 8) as usize; let test_bit: u8 = 0b00000001 << (7 - (block % 8)); // check the bit if new_allocation_table[byte] & test_bit == 0 { // block is free. if allocate { // Good! Send it back new_allocation_table[byte] |= test_bit; continue; } else { // We are trying to free a freed block panic!("Cannot free block that is already free!") } } else { // Block is not free if allocate { // Trying to allocate used block. panic!("Cannot allocate block that is already allocated!") } else { // Good! Free the block new_allocation_table[byte] ^= test_bit; continue; } } } trace!("Done updating blocks."); // All operations are done, write back the new table trace!("Writing back new allocation table..."); caller.set_allocation_table(&new_allocation_table)?; debug!("Done."); Ok(blocks.len() as u16) } fn go_check_block_allocated(caller: &T, block_number: u16) -> bool { assert!(block_number < 2880, "Tried to free block {block_number}. That is past the end of the disk."); // Integer division rounds towards zero, so this is fine. let byte: usize = (block_number / 8) as usize; let test_bit: u8 = 0b00000001 << (7 - (block_number % 8)); // check the bit caller.get_allocation_table()[byte] & test_bit != 0 } ================================================ FILE: src/pool/disk/generic/block/allocate/mod.rs ================================================ pub mod block_allocation; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/generic/block/allocate/tests.rs ================================================ // I allocate development time to testing. // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] use test_log::test; // We want to see logs while testing. use rand::{ Rng, rngs::ThreadRng }; use crate::error_types::drive::DriveError; use super::block_allocation::BlockAllocation; #[test] /// Allocate a single block from an empty table, make sure the allocated block is in the right spot. fn allocate_and_free_one_block() { let mut table = TestTable::new(); let open_block = table .find_free_blocks(1) .expect("Should have > 1 block free."); assert_eq!(open_block.len(), 1); // We only asked for 1 block assert_eq!(*open_block.first().expect("Guarded"), 0_u16); // the first block should be free let blocks_allocated = table.allocate_blocks(&open_block).unwrap(); assert_eq!(blocks_allocated, 1); // Should have allocated 1 block assert_eq!(table.block_usage_map[0], 0b10000000); // First block got set. let blocks_freed = table.free_blocks(&[0_u16].to_vec()).unwrap(); // free the first block assert_eq!(blocks_freed, 1); // Should have freed the block assert_eq!(table.block_usage_map[0], 0b00000000); // First block got freed } #[test] /// Attempt to allocate more blocks than there are on a disk /// This is a valid use-case, mass allocations like this will be used for /// putting as much data as we can fit onto a disk. fn oversized_allocation() { let table = TestTable::new(); let open_block = table .find_free_blocks(5000) .expect_err("There shouldn't be enough room."); assert_eq!(open_block, 2880); } /// Fill a table with free gaps in it #[test] fn saturate_table() { for _ in 0..1000 { let mut random: ThreadRng = rand::rng(); let mut table = TestTable::new(); // Fill with random bytes let mut random_table = [0u8; 360]; for byte in random_table.iter_mut() { let new_byte: u8 = random.random(); *byte = new_byte; } table.block_usage_map = random_table; // Make sure that the table knows how many block are actually allocated already. let blocks_pre_set: u32 = random_table.iter().map(|byte| byte.count_ones()).sum(); // Now fill up the table let free_blocks = table .find_free_blocks(5000) .expect_err("There shouldn't be enough room."); // make sure the table is reporting the correct amount of free blocks. assert_eq!(2880 - free_blocks as u32, blocks_pre_set); let blocks_to_allocate = table .find_free_blocks(free_blocks) .expect("Self reported max capacity."); let blocks_allocated: u16 = table.allocate_blocks(&blocks_to_allocate).unwrap(); assert_eq!(blocks_allocated, free_blocks); // Is it actually full tho? let num_unset_bits: u32 = table .block_usage_map .iter() .map(|byte| byte.count_zeros()) .sum(); assert_eq!(num_unset_bits, 0); } } /// Allocate random blocks and make sure they got marked #[test] fn marking() { for _ in 0..1000 { let mut random: ThreadRng = rand::rng(); let mut table = TestTable::new(); // the table is empty, so we should be able to reserve any block we want. let random_block: u16 = random.random_range(0..2880); let allocated_count = table.allocate_blocks(&[random_block].to_vec()).unwrap(); assert_eq!(allocated_count, 1); // Check that it got set let is_allocated = table.is_block_allocated(random_block); assert!(is_allocated) } } // We need a struct that implements the allocation methods for testing struct TestTable { pub block_usage_map: [u8; 360], } impl TestTable { fn new() -> Self { Self { block_usage_map: [0u8; 360], } } } impl BlockAllocation for TestTable { fn get_allocation_table(&self) -> &[u8] { &self.block_usage_map } fn set_allocation_table(&mut self, new_table: &[u8]) -> Result<(), DriveError> { self.block_usage_map = new_table .try_into() .expect("New table should be the same size as old table."); // We dont need to flush, since this table is all in memory for testing Ok(()) } } ================================================ FILE: src/pool/disk/generic/block/block_structs.rs ================================================ // Structs that can be deduced from a block // Imports use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; // Structs, Enums, Flags /// A raw data block /// This should only be used internally, interfacing into this should /// be abstracted away into other types (For example DiskHeader) #[derive(Debug)] pub struct RawBlock { /// Which block on the disk this is pub block_origin: DiskPointer, /// The block in its entirety. pub data: [u8; 512], } ================================================ FILE: src/pool/disk/generic/block/crc.rs ================================================ // CRC check // Check whether the CRC matches the block or not // returns true if crc matches the block correctly. pub fn check_crc(block: [u8; 512]) -> bool { let existing: [u8; 4] = block[508..512].try_into().expect("4 = 4"); let computed: [u8; 4] = compute_crc(&block[0..508]); existing == computed } // Takes in the data and calculates the CRC pub fn compute_crc(bytes: &[u8]) -> [u8; 4] { let checksum: u32 = crc32c::crc32c(bytes); checksum.to_le_bytes() } /// Every block will always have a CRC in its last 4 bytes, regardless of block type. pub(crate) fn add_crc_to_block(block: &mut [u8; 512]) { let crc = compute_crc(&block[0..508]); block[508..].copy_from_slice(&crc); } // Correcting detected errors could in theory be done with trying to flip every bit, but realistically, // we're better off just re-reading it, or restoring it from backup. ================================================ FILE: src/pool/disk/generic/block/mod.rs ================================================ pub mod allocate; pub mod block_structs; pub mod crc; ================================================ FILE: src/pool/disk/generic/disk_trait.rs ================================================ // All types of disk MUST implement this. // Enforced by traits. use std::fs::File; use enum_dispatch::enum_dispatch; use crate::{ error_types::drive::DriveError, pool::disk::{ drive_struct::DiskType, generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer }, } }; // Generic disks must also have disk numbers, and be able to retrieve their inner File. #[enum_dispatch(DiskType)] // Force every disk type to implement these methods. pub trait GenericDiskMethods { /// Read a block /// Cannot bypass CRC. fn unchecked_read_block(&self, block_number: u16) -> Result; /// Read multiple blocks /// Does not check CRC! fn unchecked_read_multiple_blocks(&self, block_number: u16, num_block_to_read: u16) -> Result, DriveError>; /// Write a block. fn unchecked_write_block(&mut self, block: &RawBlock) -> Result<(), DriveError>; /// Write chunked data, starting at a block. fn unchecked_write_large(&mut self, data: Vec, start_block: DiskPointer) -> Result<(), DriveError>; /// Get the inner file. fn disk_file(self) -> File; /// Get the inner file for write operations. fn disk_file_mut(&mut self) -> &mut File; /// Get the number of the floppy disk. fn get_disk_number(&self) -> u16; /// Set the number of this disk. fn set_disk_number(&mut self, disk_number: u16); /// Sync all in-memory information to disk /// Headers and such. fn flush(&mut self) -> Result<(), DriveError>; } ================================================ FILE: src/pool/disk/generic/generic_structs/find_space.rs ================================================ // A cool function that finds free space in a slice of bytes // Trait constraint that all input types must meet pub trait BytePingPong { /// Converts a type to its byte representation fn to_bytes(&self) -> Vec; /// Converts bytes into itself, and /// will discard extra trailing bytes. fn from_bytes(bytes: &[u8]) -> Self; } /// Find a contiguous space of x bytes in the input slice. /// Returns an index into the input data where the next `requested_space` bytes are empty. /// We assume the caller already checked if there is enough room, so if we do not find /// a space big enough, we will return None. pub fn find_free_space(data: &[u8], requested_space: usize) -> Option { // Assumptions: // All incoming types will have bitflags // - All incoming bitflags will have a marker bit in position 7 // Free space will be all 0's // We will: // - Look at a byte and check for the marker bit // - - If there is no bit, this must be the start of unused space // - - If there is a bit, get the length of this item, and jump ahead that far, start over // - Check if the next `requested_space` bytes are empty: // - - Yes? Return the current index // - - No? Find which byte had data, and set the index to that byte. Start over. // How far we are indexed into the data let mut index: usize = 0; let data_length = data.len(); // Sanity check, are we requesting more bytes than there is room possibly for bytes? assert!( requested_space <= data_length, "We cant find `{requested_space}` bytes free space in a slice of `{data_length}` size.." ); // We wont search bytes that are too far into the block to have enough space after them // for the incoming data. while index <= data_length - requested_space { // Check for the marker bit if data[index] & 0b10000000 != 0 { // The bit is set. We need to seek forwards. // To find out how far we need to seek, we will convert the bytes starting at offset // to type , then convert that type back to bytes again, and get the length of that // This might be tad wasteful, but it is simple lol. // Don't like it? Make a pull request! :D index += T::from_bytes(&data[index..]).to_bytes().len(); continue; } // The bit is not set. Check if there's room let enough_space: bool = data[index..index + requested_space] .iter() .all(|&byte| byte == 0); if enough_space { // Found space! return Some(index); } // There was a byte in the way, find which byte caused it let non_empty_byte_offset: usize = data[index..index + requested_space] .iter() .position(|&byte| byte != 0) .expect("There has to be a byte in the way."); // Move that far forward, then try again. // The index we are already on MUST be either zero, or the start of a // Since we already know we arent at the start of , we will always jump at least // one byte forwards. index += non_empty_byte_offset; continue; } // If we made it out of the while loop, that must mean there is not an open space. None } ================================================ FILE: src/pool/disk/generic/generic_structs/mod.rs ================================================ pub mod find_space; pub mod pointer_struct; ================================================ FILE: src/pool/disk/generic/generic_structs/pointer_struct.rs ================================================ /// Points to a specific block on a disk #[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)] pub(crate) struct DiskPointer { pub(crate) disk: u16, pub(crate) block: u16, } impl DiskPointer { pub(crate) fn to_bytes(self) -> [u8; 4] { let mut buffer: [u8; 4] = [0u8; 4]; buffer[..2].copy_from_slice(&self.disk.to_le_bytes()); buffer[2..].copy_from_slice(&self.block.to_le_bytes()); buffer } pub(crate) fn from_bytes(bytes: [u8; 4]) -> Self { Self { disk: u16::from_le_bytes(bytes[..2].try_into().expect("2 = 2")), block: u16::from_le_bytes(bytes[2..].try_into().expect("2 = 2")), } } // Random pointers for testing #[cfg(test)] pub(crate) fn get_random() -> Self { use rand::Rng; let mut random = rand::rng(); Self { disk: random.random(), block: random.random(), } } /// Creates a new disk pointer with no destination. pub(crate) fn new_final_pointer() -> Self { Self { disk: u16::MAX, block: u16::MAX, } } /// Check if this pointer doesn't go anywhere pub(crate) fn no_destination(&self) -> bool { self.disk == u16::MAX || self.block == u16::MAX } } ================================================ FILE: src/pool/disk/generic/io/cache/cache_implementation.rs ================================================ // Non-public cache construction // Some details about the cache: // The lowest tier, 0, is completely emptied when it's full. Since we // assume that the data within there is of very low quality. If it was // worth keeping around, it would have been promoted already // Tier 1 pushes it's best cached item to tier 2 when it's full. // Tier 2 discards its least valuable cache item when it's full. // Within tiers, items are promoted to a higher position whenever a read // successfully hits them. The only exception to this is tier 0, where // successful reads promote an item up to tier 1. // When a new item is added to a tier, it starts in the highest position, as it // is the most fresh. It is expected that if this item is weaker than pre-existing // items, that the newly added top item will quickly slide down in rank. // The lower cache tiers are inherently more volatile, so they need to be // larger to support more opportunities for items to promote before being // thrashed out of the cache. Thus we will split the cache into: // 0: 1/2 of total allowed cache size // 1: 1/4th of total allowed cache size // 2: 1/4th of total allowed cache size // It may seem weird to make the highest tier the same size as the one below it, // but items that reach this tier are now such a high quality that they would be // very quickly replaced if they became stale, since the constant read hits that are // expected of these items would move stale items to the lowest positions very quickly. // Promotion within tiers always moves the item from whatever index it's currently at, to // the very top of the tier. This should ensure that the hottest items stay close to the // top, previously I used bubble sort, which could lead to slightly less used items to // not promote away from the bottom of the queue fast enough. use std::{ collections::{ HashMap, VecDeque }, sync::Mutex }; use lazy_static::lazy_static; use log::debug; use crate::{ error_types::drive::DriveError, pool::disk::{ drive_struct::{ DiskType, FloppyDrive, }, generic::{ block::block_structs::RawBlock, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::{ cache::{ cache_io::CachedBlockIO, statistics::BlockCacheStatistics } } }, standard_disk::standard_disk_struct::StandardDisk }, tui::{notify::NotifyTui, tasks::TaskType} }; // // ========= // GLOBAL? LOCAL? IDK // ========= // // The maximum amount of blocks all caches can store #[cfg(test)] // Small cache on test is faster. const CACHE_SIZE: usize = 2880 * 2; #[cfg(not(test))] const CACHE_SIZE: usize = 2880 * 16; // The actual cached data lazy_static! { static ref CASHEW: Mutex = Mutex::new(BlockCache::new()); } // // ========= // STRUCTS // ========= // /// The wrapper around all the cache tiers /// Only avalible within the cache folder, /// all public interfaces are built on top of CachedBlockIO. pub(super) struct BlockCache { // The different levels of cache. // All of the internals are private. /// Highest quality, items in this level came from the highest spot from the tier below when /// it was completely full. IE filled with the best of level_1. tier_2: TieredCache, /// Might be useful, promoted from level 0 after being read at least once. tier_1: TieredCache, /// Unproven items, might as well be garbage. tier_0: TieredCache, } /// The actual caches #[derive(Clone)] struct TieredCache { /// How big this cache is. size: usize, /// The items currently in the cache, hashmap pair items_map: HashMap, /// Keep track of the order of items in the cache order: VecDeque } /// The cached blocks /// Available in the cache folder to provide conversion methods. #[derive(Debug, Clone)] pub(super) struct CachedBlock { /// Where this block came from. block_origin: DiskPointer, /// The content of the block. data: Vec, /// Whether or not this block needs to be flushed. /// /// Blocks that are read but never written do not need to be flushed. pub(super) requires_flush: bool } // // ========= // Implementations // ========= // // The entire cache // These functions are public to the cache folder, since we need these for read/write impl BlockCache { /// Create a new empty cache fn new() -> Self { // Get the max size of the cache let size: usize = CACHE_SIZE; // Need the 3 tiers // Division rounds down, so this is fine. let tier_0: TieredCache = TieredCache::new(size/2); let tier_1: TieredCache = TieredCache::new(size/4); let tier_2: TieredCache = TieredCache::new(size/4); // All done Self { tier_0, tier_1, tier_2, } } /// Retrieves an item from the cache if it exists. /// /// Updates the underlying caches to promote the read item. pub(super) fn try_find(pointer: DiskPointer) -> Option { go_try_find_cache(pointer, false) } /// Retrieves an item from the cache if it exists, but does not promote the item. pub(super) fn try_find_silent(pointer: DiskPointer) -> Option { go_try_find_cache(pointer, true) } /// Add an item to the cache, or update it if the item is already present. /// /// If the item is new, it will be placed in the lowest tier in the cache. /// /// Make sure you properly set wether the block needs flushing or not. pub(super) fn add_or_update_item(item: CachedBlock) -> Result<(), DriveError> { go_add_or_update_item_cache(item) } /// Get the hit-rate of the cache pub(super) fn get_hit_rate() -> f64 { BlockCacheStatistics::get_hit_rate() } /// Get the pressure of tier 0. /// /// Must drop cache before calling. pub(super) fn get_pressure() -> f64 { go_get_cache_pressure() } // Promotes a tier 0 cache item upwards. fn promote_item(&mut self, item: CachedBlock) { go_promote_item_cache(self, item) } /// Removes an item from the cache if it exists. /// /// You must flush this item to disk yourself (if needed), or you will lose data! /// /// Returns nothing. pub(super) fn remove_item(pointer: &DiskPointer) { go_remove_item_cache(pointer) } // /// Reserve a block on a disk, skipping the disk if possible. // /// // /// Panics if block was already allocated. // pub(super) fn cached_block_allocation(raw_block: &RawBlock) -> Result<(), DriveError> { // let mut cache_disk: CachedAllocationDisk = CachedAllocationDisk::open(raw_block.block_origin.disk)?; // let _ = cache_disk.allocate_blocks(&vec![raw_block.block_origin.block])?; // // Shouldn't even need to check if it allocated one block, no way it could allocate more. // Ok(()) // } /// Flushes all information in a tier to disk. /// /// Caller must drop all references to cache before calling this. pub(super) fn flush(tier_number: usize) -> Result<(), DriveError> { go_flush_tier(tier_number) } /// Drops items from this cache tier that have not been updated, and thus don't need to be written to disk. /// /// You should really only call this on tier 0, since items in the higher tiers are usually very read heavy, thus /// are usually not updated. Cleaning up those higher tiers would almost certainly discard valuable blocks. /// /// Caller must drop all references to cache before calling this. /// /// Returns how many blocks were discarded, or None if the tier was already empty. pub(super) fn cleanup_tier(tier_number: usize) -> Option { go_cleanup_tier(tier_number) } /// Flushes any low-importance pending writes on a selected disk. /// /// This should be called when you know you are about to swap disks, since /// otherwise you might swap disks for a read, then immediately need to swap back /// again because the cache filled up. /// /// Returns how many blocks were freed from the cache. /// /// Caller must drop all references to the cache before calling this. pub(super) fn flush_a_disk(disk_number: u16) -> Result { go_flush_disk_from_cache(disk_number) } /// Find what disk is the most common in the lowest cache tier. /// /// Returns the disk with the most blocks on it (or picks the first one if it is a tie) and how many /// blocks are from that disk. (disk, blocks) /// /// Panics if the tier is empty. /// /// You should clean-up the cache before calling this, as to get a count of only /// dirty blocks. /// /// Caller must drop all references to the cache before calling this. pub(super) fn most_common_disk() -> (u16, u16) { go_find_most_common_disk() } /// Find out how much free space is in a tier /// /// Returns number of empty spaces in the tier pub(super) fn get_tier_space(tier_number: usize) -> usize { go_get_tier_free_space(tier_number) } } // Cache tiers impl TieredCache { /// Create a new, empty tier of a set size fn new(size: usize) -> Self { go_make_new_tier(size) } /// Check if an item is in this tier. /// /// Adds a hit to the tier statistics if found, otherwise /// leaves the statistics alone. /// /// Returns the index of the item if it exists. /// /// Does not update tier order. fn find_item(&self, pointer: &DiskPointer) -> Option { go_find_tier_item(self, pointer) } /// Retrieves an item from this tier at the given index. /// /// Will promote the item within this tier if not silent. /// /// Updates tier order. /// /// Returns None if there is no item at the index. fn get_item(&mut self, index: usize, silent: bool) -> Option<&CachedBlock> { go_get_tier_item(self, index, silent) } /// Extracts an item at an index, removing it from the tier. /// /// Returns None if there is no item at the index. fn extract_item(&mut self, index: usize) -> Option { go_extract_tier_item(self, index) } /// Adds an item to this tier. Will be the new highest item in the tier. /// /// Will panic if tier is already full. fn add_item(&mut self, item: CachedBlock) { go_add_tier_item(self, item) } /// Updates / replaces an item at a given index. /// /// Updates order. /// /// Will panic if index is empty / out of bounds. fn update_item(&mut self, index: usize, new_item: CachedBlock) { go_update_tier_item(self, index, new_item) } /// Pops the best item of the tier. /// /// Returns None if the tier is empty fn get_best(&mut self) -> Option { go_get_tier_best(self) } /// Pops the worst item of the tier. /// /// Returns None if the tier is empty fn get_worst(&mut self) -> Option { go_get_tier_worst(self) } /// Check if this tier is full fn is_full(&self) -> bool { go_check_tier_full(self) } } // Nice to haves for the CachedBlocks impl CachedBlock { /// Turn a CachedBlock into a RawBlock pub(super) fn into_raw(self) -> RawBlock { RawBlock { block_origin: self.block_origin, data: self.data.try_into().expect("Should be 512 bytes."), } } /// Turn a RawBlock into a CachedBlock /// /// Expects the raw block to already have a disk set. pub(super) fn from_raw(block: &RawBlock, requires_flush: bool) -> Self { Self { block_origin: block.block_origin, data: block.data.to_vec(), requires_flush } } } // // ========= // BlockCache Functions // ========= // fn go_try_find_cache(pointer: DiskPointer, silent: bool) -> Option { // Make sure this is a valid disk pointer, otherwise something is horribly wrong. assert!(!pointer.no_destination(), "Tried to find the no_destination pointer in the block cache!"); // To prevent callers from having to lock the global themselves, we will grab it here ourselves // and pass it downwards into any functions that require it. let cache = &mut CASHEW.try_lock().expect("Single threaded."); // Try from highest to lowest // Tier 2 if let Some(found) = cache.tier_2.find_item(&pointer) { // In the highest rank! BlockCacheStatistics::record_hit(); // Grab it, which will also update the order. return cache.tier_2.get_item(found, silent).cloned() } // Tier 1 if let Some(found) = cache.tier_1.find_item(&pointer) { // Somewhat common it seems. BlockCacheStatistics::record_hit(); // Grab it, which will also update the order. return cache.tier_1.get_item(found, silent).cloned() } // Tier 0 if let Some(found) = cache.tier_0.find_item(&pointer) { // Scraping the barrel, but at least it was there! BlockCacheStatistics::record_hit(); // Since this is the lowest tier, we need to immediately promote this if needed. if !silent { let item = cache.tier_0.extract_item(found).expect("Just checked."); cache.promote_item(item.clone()); return Some(item); } else { // Dont need to promote. let read = cache.tier_0.items_map.get(&pointer).expect("Already checked"); return Some(read.clone()); } } // It wasn't in the cache. Record the miss if needed. if !silent { BlockCacheStatistics::record_miss(); } // All done. None } fn go_promote_item_cache(cache: &mut BlockCache, t0_item: CachedBlock) { // This is where the magic happens. // Since tiers only change size or have new items added to them when tier 0 has a good read, // we only have to implement a cache-wide promotion scheme for tier 0. // See if there is room in tier 1 if !cache.tier_1.is_full() { // There was room. cache.tier_1.add_item(t0_item); return } // There was not room, we need to move an item upwards. let t1_best: CachedBlock = cache.tier_1.get_best().expect("How are we empty and full?"); if !cache.tier_2.is_full() { // not full, directly add it. cache.tier_2.add_item(t1_best); } else { // The best cache is full. // We will have to move the worst tier 2 item to tier 0. If we discarded it // outright, the block it contains would never get flushed to disk. let worst_of_2 = cache.tier_2.get_worst().expect("How are we empty and full?"); // Since we popped an item from t0 to call this function, it must now have at least // one slot open, so we can add to it. cache.tier_0.add_item(worst_of_2); // Now put that tier 1 item in tier 2 to make room for the new tier 1 item from tier 0. // Confused yet? cache.tier_2.add_item(t1_best); } // Now that tier 1 has had room made, add the t0 to t1 cache.tier_1.add_item(t0_item); // All done! } fn go_add_or_update_item_cache(block: CachedBlock) -> Result<(), DriveError> { // Make sure the block has a valid location assert!(!block.block_origin.no_destination(), "Attempted to add a block to the cache with a location of no_destination !"); // We don't update the cache statistics in here, since a hit while updating makes no sense. // To prevent callers from having to lock the global themselves, we will grab it here ourselves // and pass it downwards into any functions that require it. let mut cache = CASHEW.try_lock().expect("Single threaded."); // Since we search for the item in every tier before adding, this prevents duplicates. // Top to bottom. if let Some(index) = cache.tier_2.find_item(&block.block_origin) { // Fancy block! cache.tier_2.update_item(index, block); return Ok(()) } if let Some(index) = cache.tier_1.find_item(&block.block_origin) { // Useful! cache.tier_1.update_item(index, block); return Ok(()) } // Annoyingly, we still have to update the garbage, since reading presumes that stuff in tier 0 is up to date. if let Some(index) = cache.tier_0.find_item(&block.block_origin) { // Polished garbage. cache.tier_0.update_item(index, block); return Ok(()) } // It wasn't in any of the tiers, so we will add it to tier 0. // Make sure we have room first // Hold onto the size of the tier let tier_0_size = cache.tier_0.size; if cache.tier_0.is_full() { debug!("Tried adding new block to cache, but cache is full. Cleaning up tier 0..."); // We don't have room, so we need to flush out tier 0 of the cache. // But first we can try dropping items that do not require flushing drop(cache); if BlockCache::cleanup_tier(0).is_none() { // Nothing to cleanup, need to write data. Try the current disk first. debug!("Cleanup wasn't enough, flushing current disk..."); // We want to flush at least a quarter of the cache teir, otherwise we start thrashing // the cache, wasting time. let blocks_required = tier_0_size as u64 / 4; let blocks_freed = BlockCache::flush_a_disk(FloppyDrive::currently_inserted_disk_number())?; if blocks_freed < blocks_required { // Didn't flush enough from the first disk, pick the best disk and flush that next. let (most_common_disk, blocks_for_common) = BlockCache::most_common_disk(); // Would that free enough space? if blocks_for_common as u64 + blocks_freed < blocks_required { // That still wouldn't be enough. Do a full flush. BlockCache::flush(0)?; } else { // That will make enough room, flush that common disk. let _ = BlockCache::flush_a_disk(most_common_disk)?; } } } let cache: &mut std::sync::MutexGuard<'_, BlockCache> = &mut CASHEW.try_lock().expect("Single threaded."); cache.tier_0.add_item(block); return Ok(()); } // Put it in cache.tier_0.add_item(block); drop(cache); // Update the hit rate NotifyTui::set_cache_hit_rate(BlockCache::get_hit_rate()); // Update the cache pressure NotifyTui::set_cache_pressure(BlockCache::get_pressure()); Ok(()) } fn go_remove_item_cache(pointer: &DiskPointer) { // If we just find and extract on every tier, that works // Slow? Maybe... // To prevent callers from having to lock the global themselves, we will grab it here ourselves // and pass it downwards into any functions that require it. let cache = &mut CASHEW.try_lock().expect("Single threaded."); // Since we are clearing just one item, not a whole disk, we only need to check each tier once, since there // cant be any duplicates, and we can return as soon as we see a matching item. if let Some(index) = cache.tier_2.find_item(pointer) { // We discard the removed item. We assume the caller already // grabbed their own copy if they needed it. let _ = cache.tier_2.extract_item(index); return } if let Some(index) = cache.tier_1.find_item(pointer) { let _ = cache.tier_1.extract_item(index); return } if let Some(index) = cache.tier_0.find_item(pointer) { let _ = cache.tier_0.extract_item(index); } } // // ========= // TieredCache Functions // ========= // fn go_make_new_tier(size: usize) -> TieredCache { // New tiers are obviously empty. let mut new_hashmap: HashMap = HashMap::with_capacity(size); new_hashmap.shrink_to(size); let mut new_order: VecDeque = VecDeque::with_capacity(size); new_order.shrink_to(size); TieredCache { size, items_map: new_hashmap, order: new_order } } fn go_find_tier_item(tier: &TieredCache, pointer: &DiskPointer) -> Option { // Does not update order // Just see if it exists. // Skip if the tier is empty if tier.order.is_empty() { return None; } // We check the order, because we care about index here, not the actual block. tier.order.iter().position(|x| x == pointer) } fn go_get_tier_item(tier: &mut TieredCache, index: usize, silent: bool) -> Option<&CachedBlock> { // Updates order if non-silent if !silent { // Find what item the index refers to let wanted_block_pointer: DiskPointer = tier.order.remove(index)?; // Now get that item let the_block = tier.items_map.get(&wanted_block_pointer)?; // Now move the item to the front of the tier tier.order.push_front(wanted_block_pointer); Some(the_block) } else { // Silent operation, we just need to read it. let wanted_pointer = tier.order.get(index)?; let wanted_block = tier.items_map.get(wanted_pointer)?; Some(wanted_block) } } fn go_extract_tier_item(tier: &mut TieredCache, index: usize) -> Option { // Pops an item from any index, preserves order of other items // Find the item let wanted_block_pointer: DiskPointer = tier.order.remove(index)?; // Go get it tier.items_map.remove(&wanted_block_pointer) } fn go_add_tier_item(tier: &mut TieredCache, item: CachedBlock) { // New tier items go at the front, since they are the freshest. assert!(!tier.is_full(), "Tried to add an item to a tier that is already full!"); // Put the pointer into the ordering tier.order.push_front(item.block_origin); // Add to the hashmap let already_existed = tier.items_map.insert(item.block_origin, item); // Make sure that did not already exist assert!(already_existed.is_none(), "Item added to the tier was a duplicate!"); } fn go_update_tier_item(tier: &mut TieredCache, index: usize, new_item: CachedBlock) { // Replace the item, IE the contents of the block have changed. // If the contents have changed, the new item MUST have the flush bool set. assert!(new_item.requires_flush, "Incoming update item for tier did not have the flush bit set!"); // Updating is an access after all... so we will promote it. // Update the order let to_move = tier.order.remove(index).expect("Provided index into the tier should be valid."); tier.order.push_front(to_move); // Now replace the item in the hashmap at the index. let replaced = tier.items_map.insert(to_move, new_item); // Make sure we actually replaced it. Not adding here! assert!(replaced.is_some(), "Tier item we were trying to update wasn't there!"); } fn go_get_tier_best(tier: &mut TieredCache) -> Option { // Best is at the front // Get the pointer let front_pointer = tier.order.pop_front()?; // Get the block // This will return an option, its the callers fault if this item does not exist. tier.items_map.remove(&front_pointer) } fn go_get_tier_worst(tier: &mut TieredCache) -> Option { // The worst item is at the end of the vec // Get the pointer let front_pointer = tier.order.pop_back()?; // Get the block // This will return an option, its the callers fault if this item does not exist. tier.items_map.remove(&front_pointer) } fn go_flush_tier(tier_number: usize) -> Result<(), DriveError> { debug!("Flushing tier {tier_number} of the cache..."); let handle = NotifyTui::start_task(TaskType::FlushTier, 2); // We will be flushing all data from this tier of the cache to disk. // This can be used on any tier, but will usually be called on tier 0. // Run tier cleanup first to remove anything that doesn't need to be written. // Don't care how many blocks are cleaned up. let _ = go_cleanup_tier(tier_number); NotifyTui::complete_task_step(&handle); // We will extract all of the cache items at once, leaving the tier empty. let items_map_to_flush: HashMap; let items_order_to_flush: VecDeque; // We only get the order just to discard it. // Keep the cache locked within just this area. { // Get the block cache let mut cache = CASHEW.try_lock().expect("Single threaded."); // find the tier we need to flush let tier_to_flush: &mut TieredCache = match tier_number { 0 => &mut cache.tier_0, 1 => &mut cache.tier_1, 2 => &mut cache.tier_2, _ => panic!("Tried to access a non-existent cache tier!"), }; // If the tier is empty, there's nothing to do. if tier_to_flush.order.is_empty() { return Ok(()); } // Move all items from the tier into our local variable, // leaving the cache's tier empty. // In theory, if the flush fails, we would now lose data... // just dont fail lol, good luck items_map_to_flush = std::mem::take(&mut tier_to_flush.items_map); items_order_to_flush = std::mem::take(&mut tier_to_flush.order); } let _ = items_order_to_flush; // Cache is now unlocked NotifyTui::complete_task_step(&handle); // first we grab all of the items and sort them by disk, low to high, and also sort the blocks // within those disks to be in order. Since if the blocks are in order, the head doesn't have to move around // the disk as much. // Get the items from the hashmap let mut items: Vec = items_map_to_flush.into_values().collect(); // Before sorting, we can toss any blocks that do not have flush set, since // they were never updated and thus don't need to be written back to disk. items.retain(|block| block.requires_flush); // If we ended up with no items, that means the tier was completely filled with items // that did not need to be flushed, and we can exit early. if items.is_empty() { // Cool return Ok(()); } // There are still items in here, we have work to do. // Sort the blocks we will actually be writing to put the same disks in order, then by block order. items.sort_unstable_by_key(|item| (item.block_origin.disk, item.block_origin.block)); // Now to reduce head movement even further, we don't want to check the allocation table // while making our writes. Since that would require seeking to block 0 after each write. // You might be thinking, "Why can't we use the cache for the allocation tables?", darn good idea, // but we cannot access the cache from down here, since that would require locking the entire cache // a second time. Also we might be out of room in the cache for the read required to get the table, // which would cause us to flush the tier again, which we are already doing. Bad news. // But there are some assumptions we can make about the items we are flushing: // - We assume the items within the cache are valid. (A given, but can't hurt to mention) // - If an item is contained within a cache tier, the block it came from must // be allocated, and moreover, unchanged since the last time we flushed to it. // - We currently have full control over the floppy disk. Since all high-level // IO happens on the cache itself, we can swap disks and even finish on a // completely different disk without worrying about other callers. // - - Furthermore, since we have full control over the disk, the allocation tables // cannot be changing. // - When an item is removed from the cache manually, it must have been flushed to disk. // - Invalidated items on cache levels higher than 0 will put their invalidated item into // tier zero, thus they will be flushed to disk when it is cleared. // Basically, we don't have to care about the allocation table AT ALL down here. If // we have a block, we know it is allocated. When a block is freed, it must be removed // from the cache entirely. // Therefore, we can make all of our writes in one pass per disk, and never have to look at // the allocation table at all! // To properly allow lazy-loading disks into the drive, we allow the disk loading routine to use cached blocks // if they exist. // The problem is, this causes the disk check to always return true if the header is in the cache, meaning // in theory, an incorrect disk can be in the drive. // To solve this, down here we must grab the header from the cache if it is there, then // we hold onto that, load the disk (which now has to do a proper block read to check if its the right disk), then // update the disk if its the correct one. // This is the only place that actual disk writes ever happen in normal operation outside of disk initialization. // Open the first disk to write to // Now we can chunk together the blocks into larger continuous writes for speed. // First chunk by disk let chunked_by_disk: Vec> = items .chunk_by(|a, b| b.block_origin.disk == a.block_origin.disk) .map(|block| block.to_vec()).collect(); NotifyTui::add_steps_to_task(&handle, chunked_by_disk.len() as u64); // Now we can loop over the disks for disk_chunk in chunked_by_disk { // open the disk let mut current_disk: StandardDisk = disk_load_header_invalidation(disk_chunk[0].block_origin.disk)?; // Now chunk together the blocks. // Comparison adds instead of subtracts to prevent overflow. let chunked_by_block: Vec> = disk_chunk .chunk_by(|a, b| b.block_origin.block == a.block_origin.block + 1) .map(|block| block.to_vec()).collect(); NotifyTui::add_steps_to_task(&handle, chunked_by_block.len() as u64); // Now loop over those. for block_chunk in chunked_by_block { // If this chunk only has one item in it, do a normal write. if block_chunk.len() == 1 { // Unchecked due to cached headers. current_disk.unchecked_write_block(&block_chunk[0].clone().into_raw())?; NotifyTui::complete_task_step(&handle); continue; } // There are multiple blocks in a row to update, we need to stitch their bytes together. let bytes_to_write: Vec = block_chunk.iter().flat_map(|block| block.data.clone()).collect(); // Now do the large write. // Unchecked since the headers for the disk may still be in the cache. current_disk.unchecked_write_large(bytes_to_write, block_chunk[0].block_origin)?; NotifyTui::complete_task_step(&handle); } NotifyTui::complete_task_step(&handle); } // All done, don't need to do any cleanup for previously stated reasons debug!("Done flushing tier {tier_number} of the cache."); // Let the TUI know NotifyTui::cache_flushed(); NotifyTui::finish_task(handle); Ok(()) } // Returns an option on if any blocks were freed, and how many. fn go_cleanup_tier(tier_number: usize) -> Option { // Discard all items in this tier that don't need to be written back to disk. debug!("Cleaning up tier {tier_number} of the cache..."); // Usually I would scope the cache, but we'll be doing these operations without touching the disk. // Get the block cache let mut cache = CASHEW.try_lock().expect("Single threaded."); // find the tier we need to flush let tier_to_flush: &mut TieredCache = match tier_number { 0 => &mut cache.tier_0, 1 => &mut cache.tier_1, 2 => &mut cache.tier_2, _ => panic!("Tried to access a non-existent cache tier!"), }; // If the tier is empty, there's nothing to do. if tier_to_flush.order.is_empty() { return None; } // Now go through all the tier items and check if we can discard them. let mut blocks_discarded: u64 = 0; let blocks_to_cleanup_map = &mut tier_to_flush.items_map; let blocks_to_cleanup_order = &mut tier_to_flush.order; // To be clever, we can use retain, and only retain the items that do need to be written, otherwise discarding // the blocks we dont need as we come across them. blocks_to_cleanup_order.retain(|pointer| { // Get the block from the hashmap let block = blocks_to_cleanup_map.get(pointer).expect("If there's a key in, there should be a block."); if block.requires_flush { // This needs to be flushed, so we return true to hold onto this block. return true; // Weird that return works in here, never seen that before. } // Block does not need to be flushed! Discard it. let _ = blocks_to_cleanup_map.remove(pointer); // Increment the discard count blocks_discarded += 1; // Return false to discard this pointer from the order vec false }); // Unneeded blocks have now been discarded. // If we weren't able to free anything, we still need to return None here. if blocks_discarded == 0 { debug!("All blocks in tier require flushing to disk."); return None; } debug!("Dropped {blocks_discarded} un-needed blocks from the tier."); // Now is a good time to update the hit rate of the TUI, since the hit rate must have decreased NotifyTui::set_cache_hit_rate(BlockCache::get_hit_rate()); Some(blocks_discarded) } /// Flush all blocks in tier 0 that correspond to a certain disk. /// /// This should be called before disk swaps to prevent needing to immediately swap back to /// flush the cache. fn go_flush_disk_from_cache(disk_number: u16) -> Result { // Pull out the tier items we need. let handle = NotifyTui::start_task(TaskType::FlushCurrentDisk, 1); debug!("Flushing cached content of disk {disk_number}..."); // Get the block cache let mut cache = CASHEW.try_lock().expect("Single threaded."); // get tier 0 let tier_0: &mut TieredCache = &mut cache.tier_0; // If the tier is already empty, there's nothing to do. if tier_0.order.is_empty() { NotifyTui::cancel_task(handle); return Ok(0); } // Now work our way through the cache, grabbing anything related to the current disk. // Extract it if it refers to the correct disk, // Ignore the block if it does not require flushing. // - We discard it ourselves here since those reads might still be useful, so cleaning up here // Might be too early. let clone_to_flush: HashMap = tier_0.items_map.clone() .extract_if(|pointer, block| pointer.disk == disk_number && block.requires_flush) .collect(); // Split that into pointers and blocks. // We take a clone of them, since we wanna actually delete them later, in case the flush fails. let cloned_pointers_to_discard: Vec = clone_to_flush.clone().keys().cloned().collect(); let mut cloned_blocks_to_flush: Vec = clone_to_flush.clone().into_values().collect(); // We're done working with the cache. let _ = tier_0; drop(cache); // Exit early if we dont have anything if cloned_blocks_to_flush.is_empty() { debug!("Nothing to flush from this disk."); return Ok(0); } // Debug how many blocks we're about to flush debug!("Writing {} blocks to disk...", cloned_blocks_to_flush.len()); // Sort the blocks cloned_blocks_to_flush.sort_unstable_by_key(|block| block.block_origin.block); // Chunk the blocks for faster writes let chunked_blocks: Vec> = cloned_blocks_to_flush .chunk_by(|a, b| b.block_origin.block == a.block_origin.block + 1) .map(|block| block.to_vec()).collect(); // open the disk we're writing to let mut disk: StandardDisk = disk_load_header_invalidation(disk_number)?; // Now loop over those. NotifyTui::add_steps_to_task(&handle, chunked_blocks.len() as u64); NotifyTui::complete_task_step(&handle); for block_chunk in chunked_blocks { // If this chunk only has one item in it, do a normal write. if block_chunk.len() == 1 { disk.unchecked_write_block(&block_chunk[0].clone().into_raw())?; NotifyTui::complete_task_step(&handle); continue; } // There are multiple blocks in a row to update, we need to stitch their bytes together. let bytes_to_write: Vec = block_chunk.iter().flat_map(|block| block.data.clone()).collect(); // Now do the large write. // Unchecked since the headers for the disk may still be in the cache. disk.unchecked_write_large(bytes_to_write, block_chunk[0].block_origin)?; NotifyTui::complete_task_step(&handle); } debug!("Flushing disk from cache complete."); NotifyTui::finish_task(handle); // Now that the writes are done, actually remove the blocks from the cache. If we removed them earlier // and any of these operations failed, we would lose data. // Get the cache/tier back let mut cache = CASHEW.try_lock().expect("Single threaded."); let tier_0: &mut TieredCache = &mut cache.tier_0; // Toss the blocks let _: HashMap = tier_0.items_map .extract_if(|pointer, block| pointer.disk == disk_number && block.requires_flush) .collect(); // Then discard all of the sorting information about those blocks tier_0.order.retain(|order| !cloned_pointers_to_discard.contains(order)); // Done with cache. let _ = tier_0; drop(cache); // Update the hit rate of the cache, might as well. NotifyTui::set_cache_hit_rate(BlockCache::get_hit_rate()); // All done. Ok(cloned_blocks_to_flush.len() as u64) } fn go_check_tier_full(tier: &TieredCache) -> bool { tier.order.len() == tier.size } fn go_find_most_common_disk() -> (u16, u16) { // Hash map to make counting the disks easier, since there can be holes let mut disks: HashMap = HashMap::new(); // Get the block cache let cache = CASHEW.try_lock().expect("Single threaded."); // get tier 0 let tier_0: &TieredCache = &cache.tier_0; // Tally up the blocks for i in &tier_0.order { if let Some(block_count) = disks.get_mut(&i.disk) { // Increment *block_count += 1; } else { // disk is not in hashmap yet let _ = disks.insert(i.disk, 1); } } // Now get the best disk disks.drain().max_by_key(|pair| pair.1).expect("Should only be called on non-empty tiers.") } fn go_get_cache_pressure() -> f64 { // Get the block cache let cache = CASHEW.try_lock().expect("Single threaded."); cache.tier_0.order.len() as f64 / cache.tier_0.size as f64 } fn go_get_tier_free_space(tier_number: usize) -> usize { // Open that tier let cache = CASHEW.try_lock().expect("Single threaded."); let tier_to_check: &TieredCache = match tier_number { 0 => &cache.tier_0, 1 => &cache.tier_1, 2 => &cache.tier_2, _ => panic!("Tried to access a non-existent cache tier!"), }; tier_to_check.size - tier_to_check.items_map.len() } /// Function for handling the possibility of cached disk headers. /// This can only be used in the cache. /// /// This should be used in place of direct disk opening to ensure headers are up to date. pub(in super::super::cache) fn disk_load_header_invalidation(disk_number: u16) -> Result { // Try to find the header for this disk in the cache let header_pointer: DiskPointer = DiskPointer { disk: disk_number, block: 0, }; // If the header is already cached, and is not dirty, we don't need to update the underlying disk. if let Some(is_dirty) = CachedBlockIO::status_of_cached_block(header_pointer) { if is_dirty { // Header needs to be written to the disk real quick // Grab the header from the cache. let header_block = CachedBlockIO::read_block(header_pointer)?; // Remove it CachedBlockIO::remove_block(&header_pointer); // Now write that to the disk # [allow(deprecated)] // This is being used for the cache. let mut disk: StandardDisk = match FloppyDrive::open(disk_number)? { DiskType::Standard(standard_disk) => DiskType::Standard(standard_disk), _ => unreachable!("Cache cannot be used for pool disks."), }.try_into().expect("Must be standard."); disk.unchecked_write_block(&header_block)?; // Disk is now out of date, we will toss it, then it will be opened again below. drop(disk); } } // Header is not cached, or is not dirty. Or we have now written the updated header back to disk. #[allow(deprecated)] // This is being used for the cache. let outgoing = match FloppyDrive::open(disk_number)? { DiskType::Standard(standard_disk) => DiskType::Standard(standard_disk), _ => unreachable!("Cache cannot be used for pool disks."), }; Ok(outgoing.try_into().expect("Must be standard")) } ================================================ FILE: src/pool/disk/generic/io/cache/cache_io.rs ================================================ // External interaction with the block cache use crate::{ error_types::drive::DriveError, pool::disk::{ drive_struct::FloppyDrive, generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::cache::{ cache_implementation::{ BlockCache, CachedBlock }, cached_allocation::CachedAllocationDisk } }, standard_disk::standard_disk_struct::StandardDisk }, tui::notify::NotifyTui}; // // ========= // Structs // ========= // /// Struct for implementing cache methods on. /// Holds no information, this is just for calling. pub struct CachedBlockIO { // m tea } // Cache methods impl CachedBlockIO { // /// Sometimes you need to forcibly write a disk during initialization procedures, so we need a bypass. // /// // /// This will ensure the correct disk is in the drive, and the header is properly up to date before // /// writing anything. // /// // /// !! == DANGER == !! // /// // /// This function should ONLY be used when initializing disks, since this does not properly update the cache. // /// The information written with this function will not be written to cache, nor will the information about this // /// disk be flushed from the cache. // /// // /// This function also does not update the allocation table. // /// // /// You better know what you're doing. // /// // /// !! == DANGER == !! // pub fn forcibly_write_a_block(raw_block: &RawBlock) -> Result<(), DriveError> { // go_force_write_block(raw_block) // } // /// Attempts to read a block from the cache, does not load from disk if not present. // /// // /// Returns the block if present, or None if absent. // pub fn try_read(block_origin: DiskPointer) -> Option { // if let Some(cached) = BlockCache::try_find(block_origin) { // // Was there! // // Tell the TUI // NotifyTui::read_cached(); // return Some(cached.into_raw()) // } // // Missing. // None // } /// Check if a block is in the cache, and if it is dirty or not. /// /// Returns Some(true) if the block is dirty, false if clean, or None if the block is absent. pub fn status_of_cached_block(block_origin: DiskPointer) -> Option { if let Some(cached) = BlockCache::try_find(block_origin) { return Some(cached.requires_flush) } // Missing. None } /// Reads in a block from disk, attempts to read it from the cache first. /// /// Block must already be allocated on origin disk. /// /// Only works on standard disks. pub fn read_block(block_origin: DiskPointer) -> Result { go_read_cached_block(block_origin) } // /// Writes a block to disk. Adds newly written block to cache. // /// // /// Block must not be allocated on destination disk, will allocate on write. // /// // /// Only works on standard disks. // pub fn write_block(raw_block: &RawBlock) -> Result<(), DriveError> { // go_write_cached_block(raw_block) // } /// Updates pre-existing block on disk, updates cache. /// /// Block must be already allocated on the destination disk. /// /// Only works on standard disks. pub fn update_block(raw_block: &RawBlock) -> Result<(), DriveError> { go_update_cached_block(raw_block) } /// Sometimes you just need to remove a block from the cache, not even set it to zeros. /// /// You MUST flush the block you are passing in before calling this function (if needed), or you WILL lose data! pub fn remove_block(block_origin: &DiskPointer) { BlockCache::remove_item(block_origin) } /// Flush the entire cache to disk. pub fn flush() -> Result<(), DriveError> { // There are currently 3 tiers of cache. // ! If that changes, this must be updated ! // ! or there will be unflushed data still ! BlockCache::flush(0)?; BlockCache::flush(1)?; BlockCache::flush(2) } } // // ========= // CachedBlockIO functions // ========= // // This function also updates the block order after the read. fn go_read_cached_block(block_location: DiskPointer) -> Result { // Grab the block from the cache if it exists. // Block must be allocated. // Unless it is a header, which are always allocated. // If we check for header allocation, we would try to open the header for the allocation check, to check if the header is allocated, // which would recurse and overflow the stack. if block_location.block != 0 { // This isn't a header. let is_allocated = CachedAllocationDisk::open(block_location.disk)?.is_block_allocated(block_location.block); assert!(is_allocated, "Tried to use the cache to read a block that was not allocated!"); } let disk_in_drive = FloppyDrive::currently_inserted_disk_number(); if let Some(found_block) = BlockCache::try_find(block_location) { // It was in the cache! Return the block... // Notify the TUI NotifyTui::read_cached(); // If we would've swapped disks, also increment that if disk_in_drive != block_location.disk { NotifyTui::swap_saved(); } return Ok(found_block.into_raw()); } // The block was not in the cache, we need to go get it old-school style. // If we are about to swap disks, we will flush tier 0 of the disk. if disk_in_drive != block_location.disk { // About to swap, do the flush. // Dont care how many blocks this flushes. let _ = BlockCache::flush_a_disk(disk_in_drive)?; }; // Now that the cache was flushed (if needed), do the read. let disk: StandardDisk = super::cache_implementation::disk_load_header_invalidation(block_location.disk)?; // We prefer to read at least 96 blocks, if the extra blocks dont fit, we just discard them. // If that fails somehow, we will try just a standard single block read as a fallback. // We also check to make sure we got something back, otherwise we have to fall back to the other read style. // But if we don't have room for 96 blocks, we will read as many as we can fit. let tier_free_space = BlockCache::get_tier_space(0); let to_read = std::cmp::min(tier_free_space, 96); if let Ok(blocks) = &disk.unchecked_read_multiple_blocks(block_location.block, to_read as u16) && !blocks.is_empty() { for block in blocks { // If the block is already in the cache, we skip adding it, since // it may have been updated already. // Silent, or we would be randomly promoting blocks. if BlockCache::try_find_silent(block.block_origin).is_some() { // Skip continue; } // Add it to the cache, since the block doesn't exist yet. BlockCache::add_or_update_item(CachedBlock::from_raw(block, false))?; } // We have to cast back and forth to clone it. Lol. let silly: RawBlock = CachedBlock::from_raw(&blocks[0], false).into_raw(); return Ok(silly) } // Need to do a singular read. // Already checked if it was allocated. let read_block = disk.unchecked_read_block(block_location.block)?; // Add it to the cache. // This is a block read from disk, so we do not set the flush flag. BlockCache::add_or_update_item(CachedBlock::from_raw(&read_block, false))?; // Return the block. Ok(read_block) } // fn go_write_cached_block(raw_block: &RawBlock) -> Result<(), DriveError> { // // Write a block to the disk, also updating the cache with the block (or adding it if it does not yet exist.) // // // The cache expects the block's destination to be allocated already, so we will allocate it here. // // We want to use the cache for this allocation if at all possible. // BlockCache::cached_block_allocation(raw_block)?; // // // Update the cache with the updated block. // // This is a write, so this will need to be flushed. // BlockCache::add_or_update_item(CachedBlock::from_raw(raw_block, true))?; // // // We don't need to write, since the cache will do it for us. // // // Notify the TUI // NotifyTui::write_cached(); // // Ok(()) // } fn go_update_cached_block(raw_block: &RawBlock) -> Result<(), DriveError> { // Update like windows, but better idk this joke sucks lmao // We have to skip the allocation check if we are attempting to update the header, otherwise // this will recuse and overflow the stack if raw_block.block_origin.block != 0 { // This is not a header. // Make sure block is currently allocated. let is_allocated = CachedAllocationDisk::open(raw_block.block_origin.disk)?.is_block_allocated(raw_block.block_origin.block); assert!(is_allocated, "Tried to use the cache to update a block that was not allocated!"); } // Update the cache with the updated block. // This is an update, so it must be flushed, since the block has changed. BlockCache::add_or_update_item(CachedBlock::from_raw(raw_block, true))?; // Notify the TUI NotifyTui::write_cached(); // We don't need to write, since the cache will do it for us on flush. Ok(()) } // /// Forcibly writes a block to disk immediately, bypasses the cache. // fn go_force_write_block(raw_block: &RawBlock) -> Result<(), DriveError> { // // Load in the disk to write to, ensuring that the header is up to date. // let mut disk: StandardDisk = super::cache_implementation::disk_load_header_invalidation(raw_block.block_origin.disk)?; // disk.unchecked_write_block(raw_block) // } ================================================ FILE: src/pool/disk/generic/io/cache/cached_allocation.rs ================================================ // Sidestep the disk if possible when marking a block as allocated. use log::error; use crate::{ error_types::drive::DriveError, pool::disk::{ generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock }, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::header::header_struct::StandardDiskHeader } }; // To not require a rewrite of pool block allocation logic, we will make fake disks for it to use. pub(crate) struct CachedAllocationDisk { /// The header of the disk we are imitating imitated_header: StandardDiskHeader, /// If anything has changed about the disk while we were using it, it /// needs to be updated in the cache. Otherwise, we would always update /// it even for simple non-mutating checks against the header. was_updated: bool, } impl CachedAllocationDisk { /// Attempt to create a new cached disk for allocation. /// /// This only works if the header for this disk is currently in the cache. /// /// To flush the new allocation table to the cache, this needs to be dropped. /// Thus, if you allocate then immediately write, you need to drop this before the write. pub(crate) fn open(disk_number: u16) -> Result { // Go get the header for this disk. Usually this is cached, but // will fall through if needed. let header_pointer: DiskPointer = DiskPointer { disk: disk_number, block: 0, }; let read: RawBlock = CachedBlockIO::read_block(header_pointer)?; let imitated_header: StandardDiskHeader = StandardDiskHeader::from_block(&read); Ok( Self { imitated_header, was_updated: false, // Haven't done anything yet. } ) } } // We need to support all of the allocation methods that disks normally use. impl BlockAllocation for CachedAllocationDisk { #[doc = " Get the block allocation table"] fn get_allocation_table(&self) -> &[u8] { &self.imitated_header.block_usage_map } #[doc = " Update and flush the allocation table to disk."] fn set_allocation_table(&mut self,new_table: &[u8]) -> Result<(), DriveError> { self.imitated_header.block_usage_map = new_table .try_into() .expect("Incoming table size should be the same as outgoing."); // Since the allocation table has changed, we need to now update the cached block // for this disk later. self.was_updated = true; Ok(()) } } // When these fake disks are dropped, their updated (if updated) blocks need to go into the cache impl Drop for CachedAllocationDisk { fn drop(&mut self) { // If we didn't update the allocation table, we don't need to // do anything, since the cached header is still up to date. if !self.was_updated { return; } // Put our fake header in the cache. let updated = self.imitated_header.to_block(); // If this fails we are major cooked, we will try 10 times. for i in (1..=10).rev() { let result = CachedBlockIO::update_block(&updated); if let Err(bad) = result { // UH OH error!("Attempting to flush a CachedAllocationDisk is failing!"); error!("{bad:#?}"); // If we are out of attempts, we must die. let remaining_attempts = i - 1; if remaining_attempts == 0 { // Cooked. error!("Well.. Shit!"); error!("Filesystem is in an unrecoverable state!"); error!("Giving up."); panic!("Failed to flush CachedAllocationDisk!") // bye bye! } error!("{remaining_attempts} attempts remaining!") } else { // Worked! All done. break } } // Block has been put in the cache. } } ================================================ FILE: src/pool/disk/generic/io/cache/mod.rs ================================================ mod cache_implementation; pub(crate) mod cache_io; mod statistics; pub(crate) mod cached_allocation; ================================================ FILE: src/pool/disk/generic/io/cache/statistics.rs ================================================ // Statistics about the cache use std::{collections::VecDeque, sync::Mutex}; use lazy_static::lazy_static; // Holds the cache const HIT_MEMORY: usize = 10_000; // How many of the last reads we keep track of to calculate hit rate. lazy_static! { // Where the stats are stored static ref CACHE_STATISTICS: Mutex = Mutex::new(BlockCacheStatistics::new()); } // // ========= // Structs // ========= // /// Statistic information about the cache pub(super) struct BlockCacheStatistics { /// Stats for calculating cache hit rates hits_and_misses: VecDeque, // How many disk swaps we've prevented // swaps_saved: u64 } // // ========= // BlockCacheStatistics functions // ========= // // The hit-rate and recoding is public, since its the cache_io that updates and reads these. impl BlockCacheStatistics { /// New stats yay fn new() -> Self { Self { hits_and_misses: VecDeque::with_capacity(HIT_MEMORY), // swaps_saved: 0, } } pub(super) fn get_hit_rate() -> f64 { // Get ourselves let stats = CACHE_STATISTICS.lock().expect("Single threaded"); if stats.hits_and_misses.is_empty() { return 0.0 } // rate is hits / total requests let hits = stats.hits_and_misses.iter().filter(|&&hit| hit).count(); hits as f64 / stats.hits_and_misses.len() as f64 } /// Record a cache hit. /// /// Two functions to avoid confusion. pub(super) fn record_hit() { // Get ourselves let stats = &mut CACHE_STATISTICS.lock().expect("Single threaded"); // Need to pop the oldest hit if we're out of room. if stats.hits_and_misses.len() >= HIT_MEMORY { let _ = stats.hits_and_misses.pop_back(); } stats.hits_and_misses.push_front(true); } /// Record a cache hit. /// /// Two functions to avoid confusion. pub(super) fn record_miss() { // Get ourselves let stats = &mut CACHE_STATISTICS.lock().expect("Single threaded"); // Need to pop the oldest hit if we're out of room. if stats.hits_and_misses.len() >= HIT_MEMORY { let _ = stats.hits_and_misses.pop_back(); } stats.hits_and_misses.push_front(false); } } ================================================ FILE: src/pool/disk/generic/io/checked_io.rs ================================================ // IO operations that ensure allocations are properly set. // We panic in here if we try to read/write in an invalid way, since that indicates a logic error elsewhere. use log::trace; use crate::{ error_types::drive::DriveError, pool::{ disk::generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock, }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, }, pool_actions::pool_struct::GLOBAL_POOL, } }; // A fancy new trait thats built out of other traits! // Automatically add it to all types that implement the subtypes we need. impl CheckedIO for T {} pub(super) trait CheckedIO: BlockAllocation + GenericDiskMethods { /// Read a block from the disk, ensuring it has already been allocated, as to not read junk. /// Panics if block was not allocated. /// /// This should ONLY be used in the cache implementation. If you are dealing with disks directly, you are /// doing it wrong. fn checked_read(&self, block_number: u16) -> Result { trace!("Performing checked read on block {block_number}...",); // Block must be allocated assert!(self.is_block_allocated(block_number), "Tried to read an unallocated block!"); // This unchecked read is safe, because we've now checked it. let result = self.unchecked_read_block(block_number)?; trace!("Block read successfully."); Ok(result) } /// Write a block to the disk, ensuring it has not already been allocated, as to not overwrite data. /// /// Sets the block as used after writing. /// /// Panics if block was not free. fn checked_write(&mut self, block: &RawBlock) -> Result<(), DriveError> { trace!("Performing checked write on block {}...", block.block_origin.block); // Make sure block is free assert!(!self.is_block_allocated(block.block_origin.block), "Tried to write to an non-free block!"); trace!("Block was not already allocated, writing..."); self.unchecked_write_block(block)?; // Now mark the block as allocated. trace!("Marking block as allocated..."); let blocks_allocated = self.allocate_blocks(&[block.block_origin.block].to_vec())?; // Make sure it was actually allocated. assert_eq!(blocks_allocated, 1, "Failed to mark the block as allocated after write!"); // Now decrement the pool header trace!("Updating the pool's free block count..."); trace!("Locking GLOBAL_POOL..."); // If the pool is shutting down, this may be poisoned. If it is, we just have to ignore it since // we dont want to panic during a panic-caused shutdown. if let Ok(mut update) = GLOBAL_POOL.get().expect("Pool must exist for CheckedIO to be performed.").try_lock() { update.header.pool_standard_blocks_free -= 1; }; trace!("Block written successfully."); Ok(()) } /// Updates an underlying block with new information. /// This overwrites the data in the block. (Obviously) /// Panics if block was not previously allocated. fn checked_update(&mut self, block: &RawBlock) -> Result<(), DriveError> { trace!( "Performing checked update on block {}...", block.block_origin.block ); // Make sure block is allocated already assert!(self.is_block_allocated(block.block_origin.block), "Tried to update an unallocated block."); self.unchecked_write_block(block)?; trace!("Block updated successfully."); Ok(()) } /// Updates several blocks starting at start_block with data. Blocks must already be allocated. /// This overwrites the data in the block. (Obviously) /// Panics if any of the blocks were not previously allocated. fn checked_large_update(&mut self, data: Vec, start_block: DiskPointer) -> Result<(), DriveError> { trace!( "Performing checked large update starting on block {}...", start_block.block ); // Make sure all of the blocks this refers to are already allocated. for block in start_block.block..start_block.block + data.len().div_ceil(512) as u16 { assert!(self.is_block_allocated(block), "One of the blocks in a large write was not allocated!"); } self.unchecked_write_large(data, start_block)?; trace!("Blocks updated successfully."); Ok(()) } } ================================================ FILE: src/pool/disk/generic/io/mod.rs ================================================ // pub mod checked_io; pub mod read; pub mod wipe; pub mod write; pub mod cache; ================================================ FILE: src/pool/disk/generic/io/read.rs ================================================ // Reading! // Safety #![deny(clippy::unwrap_used)] #![deny(clippy::expect_used)] // Imports use log::{ error, warn }; use crate::{ error_types::{ conversions::CannotConvertError, critical::{ CriticalError, RetryCapError }, drive::{ DriveError, DriveIOError, WrappedIOError } }, pool::disk::{drive_struct::FloppyDrive, generic::generic_structs::pointer_struct::DiskPointer}, tui::{ notify::NotifyTui, tasks::TaskType } }; use super::super::block::block_structs::RawBlock; use super::super::block::crc::check_crc; use std::{ fs::File, os::unix::fs::FileExt }; // Implementations // DONT USE THE CACHE DOWN HERE! // We rely on this call to _actually_ read from the disk, not just parrot back what's in the cache. // The cache calls this when an item isn't found. Checking again down here is pointless. If it was // in the cache, we wouldn't be here. /// Read a block on the currently inserted disk in the floppy drive /// ONLY FOR LOWER LEVEL USE, USE CHECKED_READ()! pub(crate) fn read_block_direct( disk_file: &File, originating_disk: u16, block_index: u16, ignore_crc: bool, has_recursed: bool, ) -> Result { let handle = NotifyTui::start_task(TaskType::DiskReadBlock, 1); // Bounds checking if block_index >= 2880 { // This block is impossible to access. panic!("Impossible read offset `{block_index}`!") } let pointer: DiskPointer = DiskPointer { disk: originating_disk, block: block_index, }; // allocate space for the block let mut read_buffer: [u8; 512] = [0u8; 512]; // Calculate the offset into the disk let read_offset: u64 = block_index as u64 * 512; // Enter a loop to retry reading the block 10 times at most. // If we try 3 times without success, we are cooked. for _ in 0..3 { // Seek to the requested block and read 512 bytes from it let read_result = disk_file.read_exact_at(&mut read_buffer, read_offset); if let Err(error) = read_result { // That did not work. // Try converting it into a DriveIOError let wrapped: WrappedIOError = WrappedIOError::wrap(error, pointer); let converted: Result = wrapped.try_into(); if let Ok(bail) = converted { // We don't need to / can't handle this error, up we go. // But we might still need to retry this if let Ok(actually_bail) = DriveError::try_from(bail) { // Something is up that we cant handle here. // We don't bail on missing disks though, sometimes the drive is just being // a bit silly and needs a few tries to realize the disk is in there. if actually_bail == DriveError::DriveEmpty { // Try again. continue; } return Err(actually_bail) } } // We must handle the error. Down here that just means trying the write again. continue; } // Read worked. // Check the CRC, unless the user disabled it on this call. // CRC checks should only be disabled when absolutely needed, such as // when reading in unknown blocks from unknown disks to check headers. if !ignore_crc && !check_crc(read_buffer) { // CRC check failed, we have to try again. warn!("CRC check failed, retrying..."); continue; } // Read successful. NotifyTui::complete_task_step(&handle); // send it. let block_origin: DiskPointer = DiskPointer { disk: originating_disk, block: block_index, }; // Inform TUI NotifyTui::finish_task(handle); NotifyTui::block_read(1); return Ok(RawBlock { block_origin, data: read_buffer, }); } // If we've recursed, critical cleanup has failed. if has_recursed { return Err(DriveError::Retry) } // We've made it out of the loop without a good read. We are doomed. error!("Read failure, requires assistance."); // Do the error cleanup CriticalError::OutOfRetries(RetryCapError::ReadBlock).handle(); // Re-grab the file, since the drive path may have changed after disk cleanup // After recovery, the path to the disk may have changed. This is a little naughty, but // we'll re-grab the disk file. let re_open = FloppyDrive::open_direct(originating_disk)?; // The type doesn't matter, as long as we get the disk file out. // If its blank or unknown, we can still write to it, reading would just be bad. let new_file = match re_open { crate::pool::disk::drive_struct::DiskType::Pool(pool_disk) => { pool_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Standard(standard_disk) => { standard_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Unknown(unknown_disk) => { unknown_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Blank(blank_disk) => { blank_disk.disk_file }, }; read_block_direct(&new_file, originating_disk, block_index, ignore_crc, true) } /// Automatically truncate reads if it would go off of the end of the disk. /// /// Returns a Vec of RawBlock. May not be the full length of requested blocks. pub(crate) fn read_multiple_blocks_direct( disk_file: &File, originating_disk: u16, block_index: u16, num_to_read: u16, has_recursed: bool, ) -> Result, DriveError> { // Bounds checking if block_index >= 2880 { // This block is impossible to access. panic!("Impossible read offset `{block_index}`!") } // Figure out how many blocks we can read let checked_num_to_read = std::cmp::min(num_to_read, 2880 - block_index); // Start the read task. let handle = NotifyTui::start_task(TaskType::DiskReadBlock, 1); // The start point of the read let pointer: DiskPointer = DiskPointer { disk: originating_disk, block: block_index, }; // allocate space for the blocks we want to read let mut read_buffer: Vec = vec![0; checked_num_to_read as usize * 512]; // Calculate the offset into the disk let read_offset: u64 = block_index as u64 * 512; // Try to read the entire chunk in. // If we try 3 times without success, we are cooked. for _ in 0..3 { // Seek to the requested block and read as many bytes as we need. let read_result = disk_file.read_exact_at(&mut read_buffer, read_offset); if let Err(error) = read_result { // That did not work. // Try converting it into a DriveIOError let wrapped: WrappedIOError = WrappedIOError::wrap(error, pointer); let converted: Result = wrapped.try_into(); if let Ok(bail) = converted { // We don't need to / can't handle this error, up we go. // But we might still need to retry this if let Ok(actually_bail) = DriveError::try_from(bail) { // Something is up that we cant handle here. // We don't bail on missing disks though, sometimes the drive is just being // a bit silly and needs a few tries to realize the disk is in there. if actually_bail == DriveError::DriveEmpty { // Try again. continue; } return Err(actually_bail) } } // We must handle the error. Down here that just means trying the write again. continue; } // Read worked. // Read successful. NotifyTui::complete_task_step(&handle); // Split it back out into blocks let mut output_blocks: Vec = Vec::with_capacity(checked_num_to_read.into()); let block_chunks = read_buffer.chunks_exact(512); for (index, block) in block_chunks.enumerate() { // Cast the block slice into a known size, this should always work let data = match block.try_into() { Ok(ok) => ok, Err(_) => unreachable!("How was the chunk size of 512 not 512 bytes?"), }; output_blocks.push( RawBlock { block_origin: DiskPointer { disk: originating_disk, block: block_index + index as u16 }, data } ); } // Inform TUI NotifyTui::finish_task(handle); NotifyTui::block_read(checked_num_to_read); return Ok(output_blocks); } // If we've recursed, critical cleanup has failed. if has_recursed { return Err(DriveError::Retry) } // We've made it out of the loop without a good read. We are doomed. error!("Read failure, requires assistance."); // Do the error cleanup CriticalError::OutOfRetries(RetryCapError::ReadBlock).handle(); // Re-grab the file, since the drive path may have changed after disk cleanup // After recovery, the path to the disk may have changed. This is a little naughty, but // we'll re-grab the disk file. let re_open = FloppyDrive::open_direct(originating_disk)?; // The type doesn't matter, as long as we get the disk file out. // If its blank or unknown, we can still write to it, reading would just be bad. let new_file = match re_open { crate::pool::disk::drive_struct::DiskType::Pool(pool_disk) => { pool_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Standard(standard_disk) => { standard_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Unknown(unknown_disk) => { unknown_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Blank(blank_disk) => { blank_disk.disk_file }, }; // recurse. read_multiple_blocks_direct(&new_file, originating_disk, block_index, num_to_read, true) } ================================================ FILE: src/pool/disk/generic/io/wipe.rs ================================================ // Squeaky clean! use std::{fs::File, time::Duration}; use log::debug; use crate::{error_types::drive::DriveError, pool::disk::generic::generic_structs::pointer_struct::DiskPointer, tui::{notify::NotifyTui, tasks::TaskType}}; /// Wipes ALL data on ALL blocks on the disk. pub(crate) fn destroy_disk(disk: &mut File) -> Result<(), DriveError> { // Bye bye! let chunk_size: usize = 64; debug!("Wiping currently inserted disk..."); let ten_blank_blocks: Vec = vec![0; 512 * chunk_size]; // Make a new task to track disk wiping progress. let task_handle = NotifyTui::start_task(TaskType::WipeDisk, (2880/chunk_size) as u64); // Write in large chunks for speed. for i in 0..2880/chunk_size { let pointer: DiskPointer = DiskPointer { disk: 42069_u16, block: (i * chunk_size) as u16, }; // We will keep track of how long this is taking, since if a single chunk of blocks // takes weirdly long, chances are the disk is bad. let now = std::time::Instant::now(); super::write::write_large_direct(disk, &ten_blank_blocks, pointer)?; if now.elapsed() > Duration::from_secs(10) { // Took too long, this disk is no good. NotifyTui::cancel_task(task_handle); return Err(DriveError::TakingTooLong) } let percent = (((i + 1) * chunk_size) as f32 / 2880_f32) * 100.0; debug!("{percent:.1}%..."); NotifyTui::complete_task_step(&task_handle); } debug!("Wipe complete."); NotifyTui::finish_task(task_handle); Ok(()) } ================================================ FILE: src/pool/disk/generic/io/write.rs ================================================ // Writing! // Safety #![deny(clippy::unwrap_used)] #![deny(clippy::expect_used)] // Imports use log::{ trace, error }; use crate::error_types::conversions::CannotConvertError; use crate::error_types::critical::{CriticalError, RetryCapError}; use crate::error_types::drive::{DriveError, DriveIOError, WrappedIOError}; use crate::filesystem::filesystem_struct::WRITE_BACKUPS; use crate::pool::disk::drive_struct::FloppyDrive; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::tui::notify::NotifyTui; use crate::tui::tasks::TaskType; use super::super::block::block_structs::RawBlock; use std::ops::Rem; use std::{ fs::File, os::unix::fs::FileExt }; // Implementations /// Write a block to the currently inserted disk in the floppy drive /// ONLY FOR LOWER LEVEL USE, USE CHECKED_WRITE()! pub(crate) fn write_block_direct(disk_file: &File, block: &RawBlock, has_recursed: bool) -> Result<(), DriveError> { let handle = NotifyTui::start_task(TaskType::DiskWriteBlock, 1); trace!( "Directly writing block {} to currently inserted disk...", block.block_origin.block ); // Bounds checking if block.block_origin.block >= 2880 { // This block is impossible to access. panic!("Impossible write offset `{}`!", block.block_origin.block) } // Update the disk backup crate::filesystem::disk_backup::update::update_backup(block); let pointer: DiskPointer = block.block_origin; // Calculate the offset into the disk let write_offset: u64 = block.block_origin.block as u64 * 512; for _ in 0..3 { // Write the data. let write_result = disk_file.write_all_at(&block.data, write_offset); if let Err(error) = write_result { // That did not work. // Try converting it into a DriveIOError let wrapped: WrappedIOError = WrappedIOError::wrap(error, pointer); let converted: Result = wrapped.try_into(); if let Ok(bail) = converted { // We don't need to / can't handle this error, up we go. // But we might still need to retry this if let Ok(actually_bail) = DriveError::try_from(bail) { // Something is up that we cant handle here. // We don't bail on missing disks though, sometimes the drive is just being // a bit silly and needs a few tries to realize the disk is in there. if actually_bail == DriveError::DriveEmpty { // Try again. continue; } } } // We must handle the error. Down here that just means trying the write again. continue; } // Writing worked! all done. NotifyTui::complete_task_step(&handle); trace!("Block written successfully."); // Attempt to sync the write, we only do this if backups are turned on, since we dont // wanna slow down tests. if let Some(enabled) = WRITE_BACKUPS.get() { if *enabled { // if this fails, oh well. let _ = disk_file.sync_all(); } } // Notify the TUI NotifyTui::block_written(1); NotifyTui::finish_task(handle); return Ok(()); }; // We've made it outside of the loop. The error is unrecoverable. // The recursion failed, so the previous error handling failed. We will bail. if has_recursed { // Rough. return Err(DriveError::Retry); } error!("Write failure, requires assistance."); // Do the error cleanup, if that works, the disk should be working now, and we can recurse, since we // should now be able to complete the operation successfully. CriticalError::OutOfRetries(RetryCapError::WriteBlock).handle(); // After recovery, the path to the disk may have changed. This is a little naughty, but // we'll re-grab the disk file. let re_open = FloppyDrive::open_direct(block.block_origin.disk)?; // The type doesn't matter, as long as we get the disk file out. // If its blank or unknown, we can still write to it, reading would just be bad. let new_file = match re_open { crate::pool::disk::drive_struct::DiskType::Pool(pool_disk) => { pool_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Standard(standard_disk) => { standard_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Unknown(unknown_disk) => { unknown_disk.disk_file }, crate::pool::disk::drive_struct::DiskType::Blank(blank_disk) => { blank_disk.disk_file }, }; // Now recurse. write_block_direct(&new_file, block, true) } /// Write a vec of bytes starting at offset to the currently inserted disk in the floppy drive. /// ONLY FOR LOWER LEVEL USE, USE CHECKED_WRITE()! pub(crate) fn write_large_direct(disk_file: &File, data: &[u8], start_block: DiskPointer) -> Result<(), DriveError> { let handle = NotifyTui::start_task(TaskType::DiskWriteLarge, 1); // Bounds checking if start_block.block >= 2880 { // This block is impossible to access. panic!("Impossible write offset `{}`!", start_block.block) } let pointer: DiskPointer = start_block; // Must write full blocks (512 byte chunks) assert!(data.len().rem(512) == 0, "Large writes must be a multiple of 512!"); // Make sure we don't run off the end of the disk assert!(start_block.block + ((data.len().div_ceil(512) - 1) as u16) < 2880_u16, "Write would go off the end of the disk!"); trace!( "Directly writing {} blocks worth of bytes starting at block {} to currently inserted disk...", data.len().div_ceil(512), start_block.block ); // Update the disk backup crate::filesystem::disk_backup::update::large_update_backup(start_block, data); // Calculate the offset into the disk let write_offset: u64 = start_block.block as u64 * 512; // Pre-sync the disk just in case its already writing. if let Some(enabled) = WRITE_BACKUPS.get() { if *enabled { // if this fails, oh well. let _ = disk_file.sync_all(); } } // Now enter a loop so we can attempt the write at most 10 times, in case it fails. for _ in 0..3 { // Write the data. let write_result = disk_file.write_all_at(data, write_offset); if let Err(error) = write_result { // That did not work. // Try converting it into a DriveIOError let wrapped: WrappedIOError = WrappedIOError::wrap(error, pointer); let converted: Result = wrapped.try_into(); if let Ok(bail) = converted { // We don't need to / can't handle this error, up we go. // But we might still need to retry this if let Ok(actually_bail) = DriveError::try_from(bail) { // Something is up that we cant handle here. // We don't bail on missing disks though, sometimes the drive is just being // a bit silly and needs a few tries to realize the disk is in there. if actually_bail == DriveError::DriveEmpty { // Try again. continue; } } } // We must handle the error. Down here that just means trying the write again. continue; } // Writing worked! all done. trace!("Several blocks written successfully."); NotifyTui::complete_task_step(&handle); // Attempt to sync the write, we only do this if backups are turned on, since we dont // wanna slow down tests. if let Some(enabled) = WRITE_BACKUPS.get() { if *enabled { // if this fails, oh well. let _ = disk_file.sync_all(); } } // Notify the TUI NotifyTui::finish_task(handle); NotifyTui::block_written((data.len()/512) as u16); return Ok(()); }; // We've made it outside of the loop. The error is unrecoverable. error!("Write failure, requires assistance."); // Do the error cleanup, if that works, the disk should be working now, we should now be able to write // to the disk, but instead of recursing, we call the fallback to try to be a bit safer with the failure, and // to prevent infinite recursion. CriticalError::OutOfRetries(RetryCapError::WriteBlock).handle(); large_write_fallback(disk_file, data, start_block) } /// If large writes are continually failing, maybe we'll have better luck with singular writes. fn large_write_fallback(disk_file: &File, data: &[u8], start_block: DiskPointer) -> Result<(), DriveError> { // Extract the vec of data into blocks. // Pointer that'll be incremented as we're creating blocks let mut new_pointer = start_block; // Chunk the data into block sized chunks let chunked = data.chunks(512); // Now this isn't a great idea, but this is the fallback anyways. // If any of the chunks (only the last one could have this happen) is // less than 512 bytes in size, we'll pad it to 512 with zeros, which yeah, stuff // might absolutely explode, but at least we can attempt to keep going lmao // Loop over the chunks and construct the blocks let blocks: Vec = chunked.into_iter().map(|chunk| { // Pad the slice if needed let mut padded: Vec = Vec::with_capacity(512); padded.extend(chunk); let difference = 512 - padded.len(); if difference != 0 { // Add zeros for padding let padding: Vec = vec![0; difference]; padded.extend(padding); } // Now turn that into a slice let sliced: [u8; 512] = if let Ok(got) = padded[0..512].try_into() { got } else { // 512 is not 512 today apparently. unreachable!("512 is 512") }; // Make a block let blocked: RawBlock = RawBlock { block_origin: new_pointer, data: sliced, }; // Increment the block pointer new_pointer.block += 1; // Out goes this block blocked } ).collect(); // Now that we have all of the blocks, write them to the disk for block in blocks { // This is the first call, we have not recursed. write_block_direct(disk_file, &block, false)?; } Ok(()) } ================================================ FILE: src/pool/disk/generic/mod.rs ================================================ pub mod block; pub mod disk_trait; pub mod generic_structs; pub mod io; ================================================ FILE: src/pool/disk/mod.rs ================================================ pub mod blank_disk; mod drive_methods; pub mod drive_struct; pub mod generic; pub mod pool_disk; pub mod standard_disk; pub mod unknown_disk; ================================================ FILE: src/pool/disk/pool_disk/block/header/header_methods.rs ================================================ // So no head? // Imports use std::process::exit; use log::debug; use log::warn; use crate::error_types::drive::DriveError; use crate::error_types::header::HeaderError; use crate::filesystem::disk_backup::restore::restore_disk; use crate::filesystem::filesystem_struct::USE_VIRTUAL_DISKS; use crate::pool::disk::blank_disk::blank_disk_struct::BlankDisk; use crate::pool::disk::drive_methods::check_for_magic; use crate::pool::disk::drive_methods::display_info_and_ask_wipe; use crate::pool::disk::drive_struct::DiskType; use crate::pool::disk::drive_struct::FloppyDrive; use crate::pool::disk::generic::block::block_structs::RawBlock; use crate::pool::disk::generic::block::crc::add_crc_to_block; use crate::pool::disk::generic::disk_trait::GenericDiskMethods; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::pool::disk::generic::io::wipe::destroy_disk; use crate::pool::disk::pool_disk::block::header::header_struct::PoolHeaderFlags; use crate::pool::disk::pool_disk::pool_disk_struct::PoolDisk; use crate::tui::prompts::TuiPrompt; use super::header_struct::PoolDiskHeader; // Implementations impl PoolDiskHeader { /// Reterive the header from the pool disk pub fn read() -> Result { read_pool_header_from_disk() } /// Overwrite the current header stored on the pool disk. pub fn write(&self) -> Result<(), DriveError> { write_pool_header_to_disk(self) } /// Convert the pool header into a RawBlock pub fn to_block(self) -> RawBlock { pool_header_to_raw_block(self) } /// Try and convert a raw block into a pool header pub fn from_block(block: &RawBlock) -> Result { pool_header_from_raw_block(block) } } /// This function bypasses the usual disk types. /// I dont want to rewrite this right now. It'll do. fn read_pool_header_from_disk() -> Result { // Get the header block from the pool disk (disk 0) // If the header is missing, and there is no fluster magic, ask if we are creating a new pool. // This function will return an error if reading in the pool is determined to be impossible due to // factors outside of our control. // Get block 0 of disk 0 // We will contain all of our logic within a loop, so if the user inserts the incorrect disk we can ask for another, etc // This is messy. Sorry. loop { // if we are running with virtual disks, we skip the prompt. if !USE_VIRTUAL_DISKS .try_lock() .expect("Fluster is single threaded.") .is_some() { // Not using virtual disks, prompt the user... let result = TuiPrompt::prompt_input( "Insert pool disk.".to_string(), "Please insert the pool root disk (Disk 0), then press enter. Or type \"wipe\" or \"restore\".".to_string(), false ); // This is the only chance the user gets to enter disk wiping mode. // Why are we doing this in pool/header_methods ? idk. if result.contains("wipe") { disk_wiper_mode() } // Or restore if result.contains("restore") { disk_restore_mode() } } // User wants to open this disk for the pool. // Attempt to extract the header let some_disk = FloppyDrive::open_direct(0)?; // We've now read in either the PoolDisk, or some other type of disk. // Find out what it is. match some_disk { crate::pool::disk::drive_struct::DiskType::Pool(pool_disk) => { // This is what we want! return Ok(pool_disk.header); } crate::pool::disk::drive_struct::DiskType::Standard(standard_disk) => { // For any disk type other than Blank, we will ask if user wants to wipe it. display_info_and_ask_wipe(&mut DiskType::Standard(standard_disk))?; // Start the loop over, if they wiped the disk, the outcome will change. continue; } crate::pool::disk::drive_struct::DiskType::Unknown(file) => { display_info_and_ask_wipe(&mut DiskType::Unknown(file))?; continue; } crate::pool::disk::drive_struct::DiskType::Blank(disk) => { // The disk is blank, we will ask if the user wants to create a new pool. prompt_for_new_pool(disk)?; // The user either created a new pool, or didnt, so we just continue and run through this again. continue; } } } } /// Ask the user if they want to create a new pool with the currently inserted disk. /// If so, we blank out the disk fn prompt_for_new_pool(disk: BlankDisk) -> Result<(), DriveError> { // if we are running with virtual disks, we skip the prompt. debug!("Locking USE_VIRTUAL_DISKS..."); if USE_VIRTUAL_DISKS .try_lock() .expect("Fluster is single threaded.") .is_some() { debug!("We are running with virtual disks, skipping the new pool prompt."); // Using virtual disks, we are going to create the pool immediately. return create_new_pool_disk(disk); } else { // If we are running a test, we should never be asking for user input, thus we should always // be using virtual disks. assert!(!cfg!(test), "Asked for user input during a test!"); } // Ask the user if they want to create a new pool starting on this disk (hereafer disk 0 / root disk) let reply = TuiPrompt::prompt_input( "Create new pool?".to_string(), "This disk is blank.\nDo you wish to create a new pool?\n\"yes\"/\"no\"".to_string(), false ); if !reply.to_lowercase().contains("yes") { // They dont wanna make a new one. return Ok(()); } // User said yes. Make the disk. create_new_pool_disk(disk) } fn pool_header_from_raw_block(block: &RawBlock) -> Result { // As usual, check for the magic if !check_for_magic(&block.data) { // There is no magic, thus this cannot be a header // The disk may be blank though. if block.data.iter().all(|byte| *byte == 0) { // Disk is blank return Err(HeaderError::Blank); } // Something else is wrong. return Err(HeaderError::Invalid); } // Easier alignment let mut offset: usize = 8; // Pool headers always have bit 7 set in the flags, other headers are forbidden from writing this bit. let flags: PoolHeaderFlags = match PoolHeaderFlags::from_bits(block.data[offset]) { Some(ok) => ok, None => { // extra bits in the flags were set, either this isn't a pool header, or it is corrupted in some way. return Err(HeaderError::Invalid); } }; // Make sure the pool header bit is indeed set. if !flags.contains(PoolHeaderFlags::RequiredHeaderBit) { // The header must have missed the joke, since it didn't quite get the bit. // Not a pool header. return Err(HeaderError::Invalid); } offset += 1; // Now we can actually start extracting the header. // Highest disk let highest_known_disk: u16 = u16::from_le_bytes(block.data[offset..offset + 2].try_into().expect("2 bytes = 2 bytes")); offset += 2; // Disk with next free block let disk_with_next_free_block: u16 = u16::from_le_bytes(block.data[offset..offset + 2].try_into().expect("2 bytes = 2 bytes")); offset += 2; // Blocks free in pool let pool_standard_blocks_free: u32 = u32::from_le_bytes(block.data[offset..offset + 4].try_into().expect("2 bytes = 2 bytes")); // Block allocation map // Stop using the offset since this is always at the end. let block_usage_map: [u8; 360] = block.data[148..148 + 360].try_into().expect("2 bytes = 2 bytes"); // The latest inode write is not persisted between launches, so we point at the root inode. let latest_inode_write: DiskPointer = DiskPointer { disk: 1, block: 1 }; Ok(PoolDiskHeader { flags, highest_known_disk, disk_with_next_free_block, pool_standard_blocks_free, latest_inode_write, // This is not persisted between launches. block_usage_map, }) } fn pool_header_to_raw_block(header: PoolDiskHeader) -> RawBlock { // Deconstruct / discombobulate #[deny(unused_variables)] // You need to write ALL of them. let PoolDiskHeader { flags, highest_known_disk, disk_with_next_free_block, pool_standard_blocks_free, latest_inode_write, block_usage_map, } = header; // Create buffer for the header let mut buffer: [u8; 512] = [0u8; 512]; // The magic buffer[0..8].copy_from_slice("Fluster!".as_bytes()); // offset for easier alignment let mut offset: usize = 8; // Flags buffer[offset] = flags.bits(); offset += 1; // Highest known disk buffer[offset..offset + 2].copy_from_slice(&highest_known_disk.to_le_bytes()); offset += 2; // Disk with next free block buffer[offset..offset + 2].copy_from_slice(&disk_with_next_free_block.to_le_bytes()); offset += 2; // Free blocks buffer[offset..offset + 4].copy_from_slice(&pool_standard_blocks_free.to_le_bytes()); // We do not save the inode write disk information. let _ = latest_inode_write; // Block usage map // Doesn't use offset, static location. buffer[148..148 + 360].copy_from_slice(&block_usage_map); // Add the CRC add_crc_to_block(&mut buffer); // This needs to always go at block 0 let block_origin: DiskPointer = DiskPointer { disk: 0, // Pool disk is always disk 0 block: 0, // Header is always at block 0 }; RawBlock { block_origin, data: buffer, } } fn create_new_pool_disk(mut disk: BlankDisk) -> Result<(), DriveError> { // Time for a brand new pool! debug!("A new pool disk was created."); // We will create a brand new header, and write that header to the disk. let new_header = new_pool_header(); // Now we need to write that let writeable_block: RawBlock = new_header.to_block(); // Write it to the disk! // This is unchecked because the disk header does not exist yet, we cannot allocate space // for the header without the header. // We dont use cached IO here, since pool disks cannot have any caching on them. disk.unchecked_write_block(&writeable_block)?; // Done! Ok(()) } // Brand new pool header fn new_pool_header() -> PoolDiskHeader { // Default pool header // Flags let mut flags: PoolHeaderFlags = PoolHeaderFlags::empty(); // Needs the required bit flags.insert(PoolHeaderFlags::RequiredHeaderBit); // The highest known disk for a brand new pool is the root disk itself, zero. let highest_known_disk: u16 = 0; // The disk with the next free block. // Starts at one to skip the pool disk. let disk_with_next_free_block: u16 = 1; // How many pool blocks are free? None! We only have the root disk! let pool_standard_blocks_free: u32 = 0; // What blocks are free on the pool disk? Not the first one! let mut block_usage_map: [u8; 360] = [0u8; 360]; block_usage_map[0] = 0b10000000; // Everything is empty, so the latest write is just gonna be the root inode. let latest_inode_write: DiskPointer = DiskPointer { disk: 1, block: 1 }; PoolDiskHeader { flags, highest_known_disk, disk_with_next_free_block, pool_standard_blocks_free, latest_inode_write, // This is not persisted on disk. block_usage_map, } } /// Put that pool away fn write_pool_header_to_disk(header: &PoolDiskHeader) -> Result<(), DriveError> { // Make a block let header_block = header.to_block(); // Get the pool disk #[allow(deprecated)] // Pool disks cannot use the cache. let mut disk: PoolDisk = match FloppyDrive::open(0)? { DiskType::Pool(pool_disk) => pool_disk, _ => unreachable!("Disk 0 should NEVER be assigned to a non-pool disk!"), }; // Replace the header disk.header = *header; // Write it. // We cant use the usual disk.flush() since that usually uses cache methods. Pool disk blocks are not cached. disk.unchecked_write_block(&header_block)?; // All done. Ok(()) } /// Disk wiper mode fn disk_wiper_mode() -> ! { // Time to wipe some disks! println!("Welcome to disk wiper mode!"); loop { let are_we_done_yet = TuiPrompt::prompt_input( "Wipe finished.".to_string(), "Please insert the next disk you would like to wipe, then hit enter. Or type \"exit\".".to_string(), true ).contains("exit"); if are_we_done_yet { // User is bored. break; } // Wiping another disk. println!("Wiping the inserted disk, please wait..."); // Get the disk from the drive. Disk numbers do not matter here. let mut disk = match FloppyDrive::open_direct(0) { Ok(ok) => ok, Err(err) => { // Uh oh // If there just isn't a disk in the drive, we can continue if err == DriveError::DriveEmpty { println!("Cant wipe nothing bozo. Put a disk in!"); continue; } println!("Opening the disk failed. Here's why:"); println!("{err:#?}"); break; // cannot go further if the drive is angry. }, }; // Wipe the disk match destroy_disk(disk.disk_file_mut()) { Ok(_) => {}, Err(err) => { // Uh oh println!("Wiping the disk failed. Here's why:"); println!("{err:#?}"); match err { DriveError::DriveEmpty => {}, DriveError::Retry => {}, DriveError::TakingTooLong => { println!("That disk is responding VERY slowly to writes, its probably bad.") }, } break; }, } println!("Disk wiped."); } // Done wiping disks, user must restart Fluster. println!("Exiting disk wiping mode. You must restart fluster."); exit(0); } /// Disk restoration mode fn disk_restore_mode() -> ! { loop { let prompt_response = TuiPrompt::prompt_input( "Disk restore mode".to_string(), "Please insert a disk to overwrite, then enter the number of the disk you would like to restore. Or type \"exit\".".to_string(), true ); if prompt_response.contains("exit") { // All done. break } // now what number do we want let disk_to_restore: u16 = match prompt_response.parse() { Ok(ok) => ok, Err(err) => { warn!("Failed to parse number!"); warn!("{err:#?}"); TuiPrompt::prompt_enter( "bruh".to_string(), "Thats not a real disk number, try again.".to_string(), false ); continue; }, }; // Call the restor-er-inator let worked = restore_disk(disk_to_restore); if worked { // Do nothing, since it worked. Just dump em back to the beginning again debug!("Restoration worked!"); continue; } // Didn't work TuiPrompt::prompt_enter( "no workie".to_string(), "That restore did not work, you should try another disk.".to_string(), false ); continue; } // Done restoring disks, user must restart Fluster. println!("Exiting disk restore mode. You must restart fluster."); exit(0); } ================================================ FILE: src/pool/disk/pool_disk/block/header/header_struct.rs ================================================ // Header for the pool disk // Imports use bitflags::bitflags; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; // Structs, Enums, Flags /// The header of the pool disk #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct PoolDiskHeader { /// Flags about the pool. pub flags: PoolHeaderFlags, /// The highest disk number that we have created pub highest_known_disk: u16, /// The next disk with a free block on it, or u16::MAX pub disk_with_next_free_block: u16, /// The number of free standard blocks across all disks pub pool_standard_blocks_free: u32, /// The disk with the most recent inode write. /// Used for speeding up inode additions. pub latest_inode_write: DiskPointer, /// Map of used blocks on this disk pub block_usage_map: [u8; 360], } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct PoolHeaderFlags: u8 { // All Pool headers MUST have this bit set. const RequiredHeaderBit = 0b10000000; } } ================================================ FILE: src/pool/disk/pool_disk/block/header/mod.rs ================================================ mod header_methods; pub mod header_struct; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/pool_disk/block/header/tests.rs ================================================ // Head in the pool? Preposterous! // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] // Imports use rand::Rng; use rand::rngs::ThreadRng; use crate::pool::disk::generic::block::block_structs::RawBlock; use crate::pool::disk::pool_disk::block::header::header_struct::PoolDiskHeader; use crate::pool::disk::pool_disk::block::header::header_struct::PoolHeaderFlags; use test_log::test; // We want to see logs while testing. // Tests // Ensure we can encode and decode a block #[test] fn block_ping_pong() { for _ in 0..1000 { let new_block = PoolDiskHeader::random(); // Wizard, CAST! let raw_block: RawBlock = new_block.to_block(); // Again! let banach_tarski: PoolDiskHeader = PoolDiskHeader::from_block(&raw_block).unwrap(); assert_eq!(new_block, banach_tarski) } } #[cfg(test)] impl PoolDiskHeader { fn random() -> Self { use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; let mut random: ThreadRng = rand::rng(); // latest inode write isnt persisted to the pool on deserialization so we dont care here either let latest_inode_write: DiskPointer = DiskPointer { disk: 1, block: 1 }; Self { flags: PoolHeaderFlags::random(), highest_known_disk: random.random(), disk_with_next_free_block: random.random(), pool_standard_blocks_free: random.random(), block_usage_map: random_allocations(), latest_inode_write, // This does not get saved to disk. } } } #[cfg(test)] impl PoolHeaderFlags { fn random() -> Self { // Currently we only have the marker bit. PoolHeaderFlags::RequiredHeaderBit } } fn random_allocations() -> [u8; 360] { let mut random: ThreadRng = rand::rng(); let mut buffer = [0u8; 360]; for byte in buffer.iter_mut() { *byte = random.random() } buffer } ================================================ FILE: src/pool/disk/pool_disk/block/mod.rs ================================================ pub mod header; ================================================ FILE: src/pool/disk/pool_disk/mod.rs ================================================ pub mod block; pub mod pool_disk_methods; pub mod pool_disk_struct; ================================================ FILE: src/pool/disk/pool_disk/pool_disk_methods.rs ================================================ // Pool disk //Imports use std::fs::File; use log::error; use crate::{ error_types::drive::DriveError, pool::disk::{ drive_struct::DiskBootstrap, generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock, crc::check_crc, }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::{read::read_block_direct, write::write_block_direct} }, pool_disk::block::header::header_struct::PoolDiskHeader, } }; use super::pool_disk_struct::PoolDisk; // Implementations impl PoolDisk { // there are no pool disk methods } // Bootstrapping impl DiskBootstrap for PoolDisk { fn bootstrap(_file: File, _disk_number: u16) -> Result { // Annoyingly, we do bootstrapping of the pool disk from elsewhere, so this has to be here just // to fill criteria for DiskBootstrap unreachable!("Pool disks are not bootstrapped.") } /// This method will panic if fed an invalid block. fn from_header(block: RawBlock, file: File) -> Self { // Immediately check the CRC of the incoming block, we don't know what state it's in if !check_crc(block.data) { // CRC failed! // If the CRC on the pool header has failed, we're cooked. error!("Someday we should be able to recover from crc checks... that is not today."); panic!("Bad CRC on pool header!"); }; // CRC is good, construct the disk... // Expect is fine, dying this early isnt a big deal. // Plus we should be only called after already verifying that the CRC is good, and that the block is not empty. let header = PoolDiskHeader::from_block(&block).expect("Pool header block needs to be good at this point."); Self { number: 0, // The pool disk is always disk 0 header, disk_file: file, } } } // Block allocator // This disk has block level allocations impl BlockAllocation for PoolDisk { fn get_allocation_table(&self) -> &[u8] { &self.header.block_usage_map } fn set_allocation_table(&mut self, new_table: &[u8]) -> Result<(), DriveError> { self.header.block_usage_map = new_table .try_into() .expect("Incoming table size should be the same as outgoing."); self.flush() } } // Generic impl GenericDiskMethods for PoolDisk { #[doc = " Read a block"] #[doc = " Cannot bypass CRC."] fn unchecked_read_block(&self, block_number: u16) -> Result { // This is the first call, we have not recursed. read_block_direct(&self.disk_file, self.number, block_number, false, false) } #[doc = " Write a block"] fn unchecked_write_block(&mut self, block: &RawBlock) -> Result<(), DriveError> { // This is the first call, we have not recursed. write_block_direct(&self.disk_file, block, false) } #[doc = " Get the inner file used for IO operations"] fn disk_file(self) -> File { self.disk_file } #[doc = " Get the number of the floppy disk."] fn get_disk_number(&self) -> u16 { self.number } #[doc = " Set the number of this disk."] fn set_disk_number(&mut self, disk_number: u16) { self.number = disk_number } #[doc = " Get the inner file used for write operations"] fn disk_file_mut(&mut self) -> &mut File { &mut self.disk_file } #[doc = " Sync all in-memory information to disk"] fn flush(&mut self) -> Result<(), DriveError> { error!("You cannot call flush on a pool disk header."); error!("This must be handled manually via a disk unchecked write."); panic!("Tried to flush a pool header with .flush() !"); } #[doc = " Write chunked data, starting at a block."] fn unchecked_write_large(&mut self, _data:Vec, _start_block: DiskPointer) -> Result<(), DriveError> { // We do not allow large writes to the pool disk. // Man the pool disk really ended up useless didn't it? panic!("No large writes on pool disks."); } #[doc = " Read multiple blocks"] #[doc = " Does not check CRC!"] fn unchecked_read_multiple_blocks(&self, _block_number: u16, _num_block_to_read: u16) -> Result, DriveError> { panic!("No large reads on pool disks."); } } ================================================ FILE: src/pool/disk/pool_disk/pool_disk_struct.rs ================================================ // poooool // Imports use crate::pool::disk::pool_disk::block::header::header_struct::PoolDiskHeader; // Structs, Enums, Flags #[derive(Debug)] pub struct PoolDisk { /// Disk number pub number: u16, /// The disk's header pub header: PoolDiskHeader, // The disk's file pub(in super::super) disk_file: std::fs::File, } ================================================ FILE: src/pool/disk/standard_disk/block/directory/directory_methods.rs ================================================ // Directory? Is that come kind of surgery? // imports // Implementations use log::debug; use crate::{error_types::{block::BlockManipulationError, drive::DriveError}, pool::disk::{ generic::{ block::{ block_structs::RawBlock, crc::add_crc_to_block }, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO, }, standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryBlockFlags, DirectoryItem, DirectoryItemFlags }, inode::inode_struct::{ Inode, InodeDirectory, InodeLocation, InodeTimestamp }, } }, tui::{notify::NotifyTui, tasks::TaskType}}; // We can convert from a raw block to a directory bock, but not the other way around. impl From for DirectoryBlock { fn from(block: RawBlock) -> Self { Self::from_block(&block) } } impl DirectoryBlock { /// This assumes that you are writing this block back to the same /// location you got it from. If that is not the case, you need to swap out /// the origin BEFORE using this method. pub fn to_block(&self) -> RawBlock { directory_block_to_bytes(self) } pub fn from_block(block: &RawBlock) -> Self { directory_block_from_bytes(block) } /// Try to add an DirectoryItem to this block. /// /// Returns nothing. pub fn try_add_item(&mut self, item: &DirectoryItem) -> Result<(), BlockManipulationError> { directory_block_try_add_item(self, item) } /// Try to remove a item from a directory. /// The item on the directory must match the item provided exactly. /// /// Returns nothing. pub(in super::super) fn try_remove_item( &mut self, item: &DirectoryItem, ) -> Result<(), BlockManipulationError> { directory_block_try_remove_item(self, item) } /// Create a new directory block. /// /// Requires the location/destination of this block. /// /// New directory blocks are the new final block on the disk. /// New directory blocks do not point to the next block (as none exists). /// Caller is responsible with updating previous block to point to this new block if needed. pub fn new(origin: DiskPointer) -> Self { new_directory_block(origin) } /// Get the items located within this block. /// This function is just to obscure the items by default, so higher up callers /// use higher abstractions pub fn get_items(&self) -> Vec { self.directory_items.clone() } /// Check if this block is empty pub fn is_empty(&self) -> Result { Ok(self.list()?.len() == 0) } } // funtions for those impls fn directory_block_try_remove_item( block: &mut DirectoryBlock, incoming_item: &DirectoryItem, ) -> Result<(), BlockManipulationError> { // Attempt to remove an item // attempt the removal if let Some(index) = block .directory_items .iter() .position(|item| item == incoming_item) { // Item exists. // update the free bytes counter block.bytes_free += incoming_item.to_bytes(block.block_origin.disk).len() as u16; // We can use swap_remove here since the ordering of items does not matter. let _ = block.directory_items.swap_remove(index); Ok(()) } else { Err(BlockManipulationError::NotPresent) } } fn directory_block_try_add_item( block: &mut DirectoryBlock, item: &DirectoryItem, ) -> Result<(), BlockManipulationError> { // Attempt to add a new item to the directory. // check if we have room let new_item_bytes: Vec = item.to_bytes(block.block_origin.disk); let new_item_length: usize = new_item_bytes.len(); if new_item_length > block.bytes_free.into() { // We don't have room for this inode. The caller will have to use another block. return Err(BlockManipulationError::OutOfRoom); } // luckily since directory blocks dont require any ordering, we can just append it to the vec and update // the amount of free space remaining, since writing the actual data will just happen at the deserialization stage. block.directory_items.push(item.clone()); // Update free space // This cast is fine, item lengths could never hit > 2^16 block.bytes_free -= new_item_length as u16; // Done! Ok(()) } fn new_directory_block(origin: DiskPointer) -> DirectoryBlock { // New block! // Flags // New blocks are assumed to be the last in the chain. let flags: DirectoryBlockFlags = DirectoryBlockFlags::empty(); // Currently unused. // Bytes free // An empty block has 501 bytes free. let bytes_free: u16 = 501; // Next block // New blocks assume we are the final block in the chain. let next_block: DiskPointer = DiskPointer::new_final_pointer(); // Items // New blocks have no items. duh. // If this is the root disk, the caller needs to add the root directory. // Not pre-allocated, since we have no idea what's going in here yet. let directory_items: Vec = Vec::new(); // All done. DirectoryBlock { flags, bytes_free, next_block, block_origin: origin, directory_items, } } /// We assume this is being written to the same place as it originated. fn directory_block_to_bytes(block: &DirectoryBlock) -> RawBlock { // Deconstruct the bock let DirectoryBlock { flags, bytes_free, next_block, #[allow(unused_variables)] // The items are extracted in a different way directory_items, block_origin, } = block; let mut buffer: [u8; 512] = [0u8; 512]; // flags buffer[0] = flags.bits(); // free bytes buffer[1..1 + 2].copy_from_slice(&bytes_free.to_le_bytes()); // next block buffer[3..3 + 4].copy_from_slice(&next_block.to_bytes()); // Directory items buffer[7..7 + 501].copy_from_slice(&block.item_bytes_from_vec(block_origin.disk)); // add the CRC add_crc_to_block(&mut buffer); // All done! // This block is going to be written, thus does not need disk information. RawBlock { block_origin: *block_origin, data: buffer, } } fn directory_block_from_bytes(block: &RawBlock) -> DirectoryBlock { // Flags let flags: DirectoryBlockFlags = DirectoryBlockFlags::from_bits_retain(block.data[0]); // Free bytes, come and get 'em let bytes_free: u16 = u16::from_le_bytes(block.data[1..1 + 2].try_into().expect("2 = 2")); // Next block let next_block: DiskPointer = DiskPointer::from_bytes(block.data[3..3 + 4].try_into().expect("2 = 2")); // The directory items let directory_items: Vec = DirectoryBlock::item_vec_from_bytes(&block.data[7..7 + 501], block.block_origin.disk); let block_origin = block.block_origin; // All done DirectoryBlock { flags, bytes_free, next_block, block_origin, directory_items, } } // Conversions for the Vec of items impl DirectoryBlock { fn item_bytes_from_vec(&self, destination_disk: u16) -> [u8; 501] { let mut index: usize = 0; let mut buffer: [u8; 501] = [0u8; 501]; // Iterate over the items for i in &self.directory_items { // Cast item to bytes for byte in i.to_bytes(destination_disk) { // Put bytes in the buffer. buffer[index] = byte; index += 1; } } buffer } fn item_vec_from_bytes(bytes: &[u8], origin_disk: u16) -> Vec { let mut items: Vec = Vec::with_capacity(83); // Theoretical limit let mut index: usize = 0; loop { // Are we out of bytes? if index >= bytes.len() { break; } // Get the flags let flags: DirectoryItemFlags = DirectoryItemFlags::from_bits(bytes[index]) .expect("Flags should only have used bits set."); // Check for marker bit if !flags.contains(DirectoryItemFlags::MarkerBit) { // No more items. break; } // Do the conversion let (item_size, item) = DirectoryItem::from_bytes(&bytes[index..], origin_disk); // increment index index += item_size as usize; // Done with this one items.push(item) } // All done items } } // Conversions for the Vec of items impl DirectoryItem { /// Turn an item into bytes. Requires the destination disk. pub(super) fn to_bytes(&self, destination_disk: u16) -> Vec { let mut vec: Vec = Vec::with_capacity(262); // Theoretical limit // Flags vec.push(self.flags.bits()); // Item name length vec.push(self.name_length); // The name of the item vec.extend(self.name.as_bytes()); // location of the inode vec.extend(self.location.as_bytes(destination_disk)); // All done vec } // Returns self, and how many bytes it took to construct this. pub(super) fn from_bytes(bytes: &[u8], origin_disk: u16) -> (u8, Self) { let mut index: usize = 0; // Flags let flags: DirectoryItemFlags = DirectoryItemFlags::from_bits(bytes[index]).expect("Flags should only have used bits set."); index += 1; // Make sure the flag is set assert!(flags.contains(DirectoryItemFlags::MarkerBit), "Directory items must have the marker bit set."); // Item name length let name_length: u8 = bytes[index]; index += 1; // Item name let name: String = String::from_utf8(bytes[index..index + name_length as usize].to_vec()) .expect("File names should be valid UTF-8"); index += name_length as usize; // Inode location let (location_size, location) = InodeLocation::from_bytes(&bytes[index..], origin_disk); index += location_size as usize; let done = Self { flags, name_length, name, location, }; (index as u8, done) } /// Get the size of the item. Regardless of type. pub(crate) fn get_size(&self) -> Result { let handle = NotifyTui::start_task(TaskType::GetSize, 1); debug!("Getting size of `{}`...", self.name); // Grab the inode to work with let inode: Inode = self.get_inode()?; // If this is a file, it's easy if let Some(file) = inode.extract_file() { debug!("Item is a file, getting size directly..."); NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); return Ok(file.get_size()) } // More work to do NotifyTui::add_steps_to_task(&handle, 2); // Otherwise, this must be a directory, so we need the directory block debug!("Item is a directory..."); let inode_directory: InodeDirectory = inode.extract_directory().expect("Not a file, so its a directory."); NotifyTui::complete_task_step(&handle); // Load the block debug!("Getting origin block..."); let raw_block: RawBlock = CachedBlockIO::read_block(inode_directory.pointer)?; let directory: DirectoryBlock = DirectoryBlock::from_block(&raw_block); NotifyTui::complete_task_step(&handle); // Now we can call the size method. debug!("Calling `get_size` on loaded DirectoryBlock..."); let found_size = directory.get_size()?; NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); Ok(found_size) } /// Get when the inode / item was created. pub(crate) fn get_created_time(&self) -> Result { // get the inode let inode = self.get_inode()?; Ok(inode.created) } /// Get when the inode / item was modified. pub(crate) fn get_modified_time(&self) -> Result { // get the inode let inode = self.get_inode()?; Ok(inode.modified) } // /// All item types point to a block that holds their information. // /// You can see what block they point to, but you REALLY should not be doing reads like this. // fn get_items_pointer(&self) -> Result { // Ok(self.get_inode()?.get_pointer()) // } /// Turn a directory type DirectoryItem into a DirectoryBlock. /// /// Panics if fed a file. pub(crate) fn get_directory_block(&self) -> Result { // Grab the inode to work with let inode: Inode = self.get_inode()?; if let Some(dir) = inode.extract_directory() { // Get the directory block let raw_block: RawBlock = CachedBlockIO::read_block(dir.pointer)?; Ok(DirectoryBlock::from_block(&raw_block)) } else { // This was not a file. panic!("Attempted to turn a DirectoryItem of File type into a DirectoryBlock!") } } } ================================================ FILE: src/pool/disk/standard_disk/block/directory/directory_struct.rs ================================================ // Directory struct! // Imports use bitflags::bitflags; use crate::pool::disk::{ generic::generic_structs::pointer_struct::DiskPointer, standard_disk::block::inode::inode_struct::InodeLocation, }; // Structs / Enums / Flags #[derive(Debug, PartialEq, Eq, Clone)] pub struct DirectoryItem { pub flags: DirectoryItemFlags, pub name_length: u8, pub name: String, pub location: InodeLocation, } // This type is not clone, since you could end up with a block that is out of sync due to // changes made on a copy/clone of it. #[derive(Debug, PartialEq, Eq)] pub(crate) struct DirectoryBlock { pub(super) flags: DirectoryBlockFlags, pub(super) bytes_free: u16, // Points to the next directory block. // Directories are separate from each other, you cannot get from one directory to another by just following // the next block pointer. This pointer represents a _continuation_ of the current directory. pub next_block: DiskPointer, // At runtime its useful to know where this block came from. // This doesn't need to get written to disk. pub block_origin: DiskPointer, // This MUST be set. it cannot point nowhere. pub(crate) directory_items: Vec, } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct DirectoryItemFlags: u8 { const IsDirectory = 0b00000010; // Set if directory const MarkerBit = 0b10000000; } } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct DirectoryBlockFlags: u8 { // Currently unused. } } ================================================ FILE: src/pool/disk/standard_disk/block/directory/mod.rs ================================================ mod directory_methods; pub(crate) mod directory_struct; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/standard_disk/block/directory/tests.rs ================================================ // Directory tests // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] // Imports use rand::{self, Rng}; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryBlock; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryItemFlags; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryItem; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeLocation; use test_log::test; // We want to see logs while testing. // Tests #[test] fn blank_directory_block_serialization() { // We need a origin for the block, even if nonsensical. let block_origin = DiskPointer { disk: 420, block: 69, }; let test_block: DirectoryBlock = DirectoryBlock::new(block_origin); let serialized = test_block.to_block(); let deserialized = DirectoryBlock::from_block(&serialized); assert_eq!(test_block, deserialized) } #[test] fn directory_item_serialization() { for _ in 0..1000 { // Needs a fake disk where this block was read from. let fake_disk: u16 = 21; // you stupid let test_item = DirectoryItem::get_random(); let serialized = test_item.to_bytes(fake_disk); let (_, deserialized) = DirectoryItem::from_bytes(&serialized, fake_disk); assert_eq!(test_item, deserialized) } } // Should be covered by other tests, would need to be rewritten because serializing random data is no longer valid. // #[test] // fn filled_directory_block_serialization() { // for _ in 0..1000 { // // We need a origin for the block, even if nonsensical. // let block_origin = DiskPointer { // disk: 420, // block: 69, // }; // let mut test_block: DirectoryBlock = DirectoryBlock::new(block_origin); // // Fill with random inodes until we run out of room. // loop { // match test_block.try_add_item(&DirectoryItem::get_random()) { // Ok(_) => continue, // Err(err) => match err { // DirectoryBlockError::NotEnoughSpace => { // // Done filling it up // break; // } // _ => panic!("Got an error while adding item!"), // }, // } // } // // // Check serialization // let serialized = test_block.to_block(); // let deserialized = DirectoryBlock::from_block(&serialized); // assert_eq!(test_block, deserialized) // } // } #[test] fn add_and_remove_to_directory_block() { for _ in 0..1000 { // We need a origin for the block, even if nonsensical. let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block: DirectoryBlock = DirectoryBlock::new(block_origin); // Fill with random inodes until we run out of room. let random_item: DirectoryItem = DirectoryItem::get_random(); test_block.try_add_item(&random_item.clone()).unwrap(); // Make sure that went in assert!(!test_block.directory_items.is_empty()); test_block.try_remove_item(&random_item).unwrap(); // Make sure it was removed assert!(test_block.directory_items.is_empty()); } } #[test] fn adding_and_removing_updates_size() { for _ in 0..1000 { // We need a origin for the block, even if nonsensical. let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block: DirectoryBlock = DirectoryBlock::new(block_origin); let random_item: DirectoryItem = DirectoryItem::get_random(); let new_free = test_block.bytes_free; test_block.try_add_item(&random_item).unwrap(); let added_free = test_block.bytes_free; test_block.try_remove_item(&random_item).unwrap(); let removed_free = test_block.bytes_free; // Added should have less space assert!(added_free < new_free); // removed should have more space assert!(added_free < removed_free); // The block should be empty again assert!(new_free == removed_free); } } // Impl for going gorilla mode, absolutely ape shit, etc #[cfg(test)] impl DirectoryItemFlags { fn new() -> Self { // We always need the marker bit set DirectoryItemFlags::MarkerBit } } #[cfg(test)] impl DirectoryItem { fn get_random() -> Self { let name: String = get_random_name(); let name_length: u8 = name.len().try_into().unwrap(); assert_eq!(name_length as usize, name.len()); let location = InodeLocation::get_random(); let flags = DirectoryItemFlags::new(); DirectoryItem { flags, name_length, name, location, } } } #[cfg(test)] fn get_random_name() -> String { // make a random string of at most 255 characters, and at least 1 character use rand::distr::{Alphanumeric, SampleString}; use std::cmp::max; let mut random = rand::rng(); let random_length: u8 = max(random.random(), 1); // at least one character // make the string Alphanumeric.sample_string(&mut random, random_length as usize) } ================================================ FILE: src/pool/disk/standard_disk/block/file_extents/file_extents_methods.rs ================================================ // Method acting, for extents. // Consts // This may change if I decide to get rid of the flags on data blocks, so here's a const. pub(crate) const DATA_BLOCK_OVERHEAD: u64 = 5; // 1 flag, 4 checksum. // Imports use crate::{error_types::block::BlockManipulationError, pool::disk::{ generic::{ block::{ block_structs::RawBlock, crc::add_crc_to_block }, generic_structs::pointer_struct::DiskPointer }, standard_disk::block::file_extents::file_extents_struct::{ ExtentFlags, FileExtent, FileExtentBlock, FileExtentBlockFlags, }, }}; // Implementations // Impl the conversion from RawBlock impl From for FileExtentBlock { fn from(value: RawBlock) -> FileExtentBlock { from_bytes(&value) } } // impl the extent vec to byte conversion impl FileExtentBlock { pub(super) fn extents_to_bytes(&self, destination_disk_number: u16) -> Vec { extents_to_bytes(&self.extents, destination_disk_number) } pub(crate) fn from_block(block: &RawBlock) -> Self { from_bytes(block) } /// Byte me! /// /// This assumes you will be writing this block back to where you got it from. If this /// is not the case, you need to update the block origin before calling. pub(crate) fn to_block(&self) -> RawBlock { to_block(self) } /// Attempts to add a file extent to this block. /// /// Does not write new block to disk. Caller must write it. /// /// Returns nothing pub(crate) fn add_extent(&mut self, extent: FileExtent) -> Result<(), BlockManipulationError> { extent_block_add_extent(self, extent) } /// Create a new extent block. /// /// Requires a destination for the block. /// /// New Extent blocks are the new final block on the disk. /// New Extent blocks do not point to the next block (as none exists). /// Caller is responsible with updating previous block to point to this new block. pub(crate) fn new(block_origin: DiskPointer) -> Self { FileExtentBlock { flags: FileExtentBlockFlags::default(), bytes_free: 501, // new blocks have 501 free bytes next_block: DiskPointer::new_final_pointer(), extents: Vec::new(), // Not pre-allocated, no idea how much will end up in here. block_origin, } } /// Retrieves all extents within this _block_. NOT THE ENTIRE FILE. /// /// If you want all of the extents that a file contains, you should be calling /// methods on the InodeFile itself. /// /// Returned extents may not contain the disk component of their pointers. pub(crate) fn get_extents(&self) -> Vec { // Just a layer of abstraction to prevent direct access. self.extents.clone() } // /// Helper function that calculates how many blocks an input amount of data will require. // /// Does not take into account the sizes of FileExtent blocks or such, just the DataBlock size. // /// We are assuming you aren't going to write more than 32MB at a time. // pub fn size_to_blocks(size_in_bytes: u64) -> u16 { // // This calculation never changes, since the overhead of block is always the same. // // A block holds 512 bytes, but we reserve 1 bytes for the flags (Currently unused), // // and 4 more bytes for the checksum. // // // We will always need to round up on this division. // let mut blocks: u64; // blocks = size_in_bytes / (512 - DATA_BLOCK_OVERHEAD); // // If there is a remainder, we also need to add an additional block. // if size_in_bytes % (512 - DATA_BLOCK_OVERHEAD) != 0 { // // One more. // blocks += 1; // } // // This truncates the value. // // if you are somehow about to write a buffer of >22 floppy disks in one go, you have bigger issues. // blocks as u16 // } /// Forcibly replace all extents in a FileExtentBlock. /// /// This will also canonicalize the incoming extents. IE, if the disk in the extent matches the /// disk this block comes from, we will remove the disk and update flags. /// /// You must ensure that the provided extents will fit. Otherwise this will panic. /// If you aren't sure that the new items will fit, /// you should NOT be calling this method. /// /// This can only be called on the last extent in the chain. /// /// Will automatically recalculate size. pub(in super::super::super::block) fn force_replace_all_extents(&mut self, new_extents: Vec) { // Since outside callers cannot manually drain the extents from a block, this lets us make sure // that if you NEED to update extents, you can do that safely, and recalculate the size automatically. // Pull the extents in so we can modify them as needed. let mut to_add = new_extents; // Where are we? let our_disk = self.block_origin.disk; // Empty ourselves self.extents = Vec::with_capacity(to_add.len()); // Yes this is a silly way to see what the default capacity of an extent block is, but im sure // the compiler will just optimize all of it away. let default_free = FileExtentBlock::new(DiskPointer::new_final_pointer()).bytes_free; self.bytes_free = default_free; // Now add the new extents, fixing the disk numbers as needed. for new in &mut to_add { // if the disk is the same as the block origin, we will set the local flag and such. if new.start_block.disk == our_disk { // Disk matched, Give the extent the local flag. // We don't need to update the disk number, since that'll toss itself on write. new.flags.insert(ExtentFlags::LocalExtent); } else { // Remove the local flag, just in case. new.flags.remove(ExtentFlags::LocalExtent); } // Add it self.add_extent(*new).expect("Should be last extent, and new items shouldn't be too big.") } // All done. } } // // Functions // /// Add an extent to a block. Returns false if extent could not fit. fn extent_block_add_extent( block: &mut FileExtentBlock, extent: FileExtent, ) -> Result<(), BlockManipulationError> { // Try and add an extent to the block // Yes this causes a lot of extent.to_bytes() calls, but // we need to be able to toss the disk number. // Its fast enough. // What block this is going into let destination_disk_number: u16 = block.block_origin.disk; // Since new blocks always have to go at the end of the inode chain, if there // is a block after this, the block needs to immediately fail. if !block.next_block.no_destination() { // Keep goin dawg, not this block. return Err(BlockManipulationError::NotFinalBlockInChain) } // figure out how big the extent is. // This always less than 2^16 bytes, truncation is fine. let extent_size: u16 = extent.to_bytes(destination_disk_number).len() as u16; // will it fit? if extent_size > block.bytes_free { // Nope! return Err(BlockManipulationError::OutOfRoom); } // It'll fit! Add it to the Vec. block.extents.push(extent); // we are using that space now. block.bytes_free -= extent_size; Ok(()) } fn from_bytes(block: &RawBlock) -> FileExtentBlock { // flags let flags: FileExtentBlockFlags = FileExtentBlockFlags::from_bits_retain(block.data[0]); // What block this came from let origin_disk = block.block_origin.disk; // bytes free let bytes_free: u16 = u16::from_le_bytes(block.data[1..1 + 2].try_into().expect("2 = 2")); // Next block let next_block: DiskPointer = DiskPointer::from_bytes(block.data[3..3 + 4].try_into().expect("4 = 4")); // Extract the extents in this block let extent_data = &block.data[7..7 + 501]; let extents: Vec = bytes_to_extents(extent_data, origin_disk); FileExtentBlock { flags, bytes_free, next_block, extents, block_origin: block.block_origin, } } fn to_block(extent_block: &FileExtentBlock) -> RawBlock { let FileExtentBlock { flags, next_block, bytes_free, #[allow(unused_variables)] // The extents are extracted in a different way extents, block_origin: origin // We assume the block will be written back to the same spot it came from. } = extent_block; let mut buffer: [u8; 512] = [0u8; 512]; let mut index: usize = 0; // bitflags buffer[index] = flags.bits(); index += 1; // free bytes buffer[index..index + 2].copy_from_slice(&bytes_free.to_le_bytes()); index += 2; // Next block buffer[index..index + 4].copy_from_slice(&next_block.to_bytes()); index += 4; // Extents buffer[index..index + 501].copy_from_slice(&extent_block.extents_to_bytes(origin.disk)); // add the CRC add_crc_to_block(&mut buffer); let finished_block: RawBlock = RawBlock { block_origin: extent_block.block_origin, data: buffer, }; finished_block } // Convert the extents to a properly sized array of bytes fn extents_to_bytes(extents: &[FileExtent], destination_disk_number: u16) -> Vec { // I couldn't think of a nicer way to do this conversion let mut index: usize = 0; let mut buffer: [u8; 501] = [0u8; 501]; for i in extents { for byte in i.to_bytes(destination_disk_number) { buffer[index] = byte; index += 1; } } buffer.to_vec() } // Takes in bytes and makes extents, automatically determines when to stop. fn bytes_to_extents(bytes: &[u8], origin_disk_number: u16) -> Vec { let mut offset: usize = 0; // As stated in `to_bytes` file extents are at most 6 bytes, so we will pre-allocate // room for a totally full extent block, which right now is 501 bytes. let mut extent_vec: Vec = Vec::with_capacity(501_usize.div_ceil(6)); loop { // make sure we dont go off the deep end if offset >= bytes.len() { // cant be more. break; } // check for the marker let flag = ExtentFlags::from_bits_retain(bytes[offset]); if !flag.contains(ExtentFlags::MarkerBit) { // no more extents to read. break; } // read in an extent let (bytes_used, new_extent) = FileExtent::from_bytes(&bytes[offset..], origin_disk_number); extent_vec.push(new_extent); // increment offset offset += bytes_used as usize; } // Done! extent_vec } // Welcome to subtype impl hell impl FileExtent { /// Must provide what disk the FileExtentBlock that contains this FileExtent will end up on. /// /// Ignores incoming Local flag, will update flags automatically. pub(super) fn to_bytes(mut self, destination_disk_number: u16) -> Vec { let mut vec: Vec = Vec::with_capacity(6); // At most 6 bytes. // If the disk number is the same, we set the local flag. if self.start_block.disk == destination_disk_number { self.flags.insert(ExtentFlags::LocalExtent); } else { // Otherwise we wont. duh self.flags.remove(ExtentFlags::LocalExtent); } // flags vec.push(self.flags.bits()); if !self.flags.contains(ExtentFlags::LocalExtent) { // Disk number vec.extend_from_slice( &self .start_block.disk .to_le_bytes(), ); } // Start block vec.extend_from_slice( &self .start_block .block .to_le_bytes(), ); // Length vec.push( self.length ); vec } /// You can feed feed this too many bytes, but as long as the flag is in the right spot, it will work correctly. /// /// Also returns how many bytes the read extent was made of. pub(super) fn from_bytes(bytes: &[u8], origin_disk_number: u16) -> (u8, FileExtent) { let mut offset: usize = 0; // Extents are always <=6 bytes, so we cast this later let flags: ExtentFlags = ExtentFlags::from_bits(bytes[0]).expect("Unused bits should not be set."); offset += 1; let disk_number: u16; // Disk number if flags.contains(ExtentFlags::LocalExtent) { // Use the provided disk number. disk_number = origin_disk_number; } else { disk_number = u16::from_le_bytes(bytes[offset..offset + 2].try_into().expect("2 = 2 "),); offset += 2; } // Start block let start_block: u16 = u16::from_le_bytes(bytes[offset..offset + 2].try_into().expect("2 = 2 ")); offset += 2; // Length let length: u8 = bytes[offset]; // Final offset increment, since we are also using this to track size. offset += 1; // Construct a pointer for the start block let start_block: DiskPointer = DiskPointer { disk: disk_number, block: start_block, }; // Return the number of bytes this was constructed from, and the extent let the_extent_of_it = FileExtent { flags, start_block, length, }; (offset as u8, the_extent_of_it) } /// Helper function that extracts all of the blocks that this extent refers to. /// /// Only gets info about this specific extent, does no traversal. /// /// Needs to know what disk this FileExtent came from. pub(crate) fn get_pointers(&self) -> Vec { // Each block that the extent references let mut pointers: Vec = Vec::with_capacity(self.length.into()); for n in 0..self.length { pointers.push(DiskPointer { disk: self.start_block.disk, block: self.start_block.block + n as u16 }); }; pointers } /// Make a new file extent pub(crate) fn new(start_block: DiskPointer, length: u8) -> Self { Self { // These flags will be calculated on write. flags: ExtentFlags::MarkerBit, start_block, length, } } } // Default bitflags impl FileExtentBlockFlags { pub fn default() -> Self { // We aren't using any bits right now. FileExtentBlockFlags::empty() } } ================================================ FILE: src/pool/disk/standard_disk/block/file_extents/file_extents_struct.rs ================================================ // File extents // Imports use bitflags::bitflags; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; // Structs, Enums, Flags #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct FileExtent { /// Callers should never have to care about the flags. pub(super) flags: ExtentFlags, /// Points to the first block of the extent. Inclusive. pub(crate) start_block: DiskPointer, /// How many blocks in a row starting from the start block /// are data blocks for this file. /// /// Never traverses disks. pub(crate) length: u8, } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct ExtentFlags: u8 { // While the returned extents will always have their disk number set, at a lower level // we save bytes by tossing the disk bytes if the extent is local. The disk number is // then reconstructed on read. const LocalExtent = 0b00000001; const MarkerBit = 0b10000000; } } // Extents block #[derive(Debug, Clone, PartialEq, Eq)] pub struct FileExtentBlock { pub(super) flags: FileExtentBlockFlags, pub(super) bytes_free: u16, pub(crate) next_block: DiskPointer, // At runtime its useful to know where this block came from. // This doesn't need to get written to disk. pub block_origin: DiskPointer, // This MUST be set. it cannot point nowhere. pub(super) extents: Vec, } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct FileExtentBlockFlags: u8 { // Currently unused. } } ================================================ FILE: src/pool/disk/standard_disk/block/file_extents/mod.rs ================================================ pub mod file_extents_methods; pub mod file_extents_struct; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/standard_disk/block/file_extents/tests.rs ================================================ // Tests are cool. // Imports use crate::error_types::block::BlockManipulationError; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use super::file_extents_struct::ExtentFlags; use super::file_extents_struct::FileExtent; use super::file_extents_struct::FileExtentBlock; use rand::Rng; use rand::rngs::ThreadRng; use test_log::test; // We want to see logs while testing. // Tests #[test] fn random_extents_serialization() { // Make some random extents and de/serialize them for _ in 0..1000 { // Need a test disk number for the serialization to happen on let origin_disk: u16 = 42; let test_extent = FileExtent::random(); let serialized = test_extent.to_bytes(origin_disk); let (de_len, deserialized) = FileExtent::from_bytes(&serialized, origin_disk); let re_serialized = deserialized.to_bytes(origin_disk); let (re_de_len, re_deserialized) = FileExtent::from_bytes(&re_serialized, origin_disk); assert_eq!(deserialized, re_deserialized); assert_eq!(de_len, re_de_len); } } #[test] fn empty_extent_block_serialization() { let block_origin = DiskPointer { disk: 420, block: 69, }; let test_block = FileExtentBlock::new(block_origin); let serialized = test_block.to_block(); let deserialized = FileExtentBlock::from_block(&serialized); assert_eq!(test_block, deserialized); } #[test] fn full_extent_block() { let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block = FileExtentBlock::new(block_origin); let mut extents: Vec = Vec::new(); loop { let new_extent: FileExtent = FileExtent::random(); match test_block.add_extent(new_extent) { Ok(_) => { // keep track of the extents we put in extents.push(new_extent); // keep going } Err(err) => match err { BlockManipulationError::OutOfRoom => break, // full _ => panic!("This only happens on one block, how is this not the final block?") }, } } // Make sure all of the extents stored correctly let retrieved_extents: Vec = test_block.get_extents(); assert!(extents.iter().all(|item| retrieved_extents.contains(item))); } #[test] fn random_block_serialization() { for _ in 0..1000 { // We need a origin for the block, even if nonsensical. let block_origin = DiskPointer { disk: 420, block: 69, }; let test_block = FileExtentBlock::get_random(block_origin); let serialized = test_block.to_block(); let deserialized = FileExtentBlock::from_block(&serialized); assert_eq!(test_block, deserialized) } } // Helper functions #[cfg(test)] impl FileExtentBlock { fn get_random(block_origin: DiskPointer) -> Self { let mut test_block = FileExtentBlock::new(block_origin); let mut random: ThreadRng = rand::rng(); // Fill with a random amount of items. loop { // consider stopping early if random.random_bool(0.50) { break; } let new_extent: FileExtent = FileExtent::random(); match test_block.add_extent(new_extent) { Ok(_) => {} Err(err) => match err { BlockManipulationError::OutOfRoom => break, // full _ => panic!("This only happens on one block, how is this not the final block?") }, } } test_block } } #[cfg(test)] impl FileExtent { fn random() -> Self { let mut random: ThreadRng = rand::rng(); // Flags do not matter, they are auto deduced. let flags = ExtentFlags::new(); let length: u8 = random.random(); let start_block: DiskPointer = DiskPointer::get_random(); // All done. FileExtent { flags, start_block, length, } } } #[cfg(test)] impl ExtentFlags { fn new() -> Self { // always need the marker bit. let mut flag = ExtentFlags::empty(); flag.insert(ExtentFlags::MarkerBit); flag } } ================================================ FILE: src/pool/disk/standard_disk/block/header/header_methods.rs ================================================ // Imports use crate::pool::disk::{ generic::{block::{block_structs::RawBlock, crc::add_crc_to_block}, generic_structs::pointer_struct::DiskPointer}, standard_disk::block::header::header_struct::{StandardDiskHeader, StandardHeaderFlags}, }; // Implementations impl StandardDiskHeader { pub fn from_block(raw_block: &RawBlock) -> StandardDiskHeader { extract_header(raw_block) } pub fn to_block(&self) -> RawBlock { to_disk_block(self) } } // Impl the conversion from a RawBlock to a DiskHeader impl From for StandardDiskHeader { fn from(value: RawBlock) -> Self { extract_header(&value) } } // Functions /// Extract header info from a disk fn extract_header(raw_block: &RawBlock) -> StandardDiskHeader { // Time to pull apart the header! // Bit flags let flags: StandardHeaderFlags = StandardHeaderFlags::from_bits_retain(raw_block.data[8]); // The disk number let disk_number: u16 = u16::from_le_bytes(raw_block.data[9..9 + 2].try_into().expect("Impossible")); // block usage bitplane let block_usage_map: [u8; 360] = raw_block.data[148..148 + 360] .try_into() .expect("Impossible."); StandardDiskHeader { flags, disk_number, block_usage_map, } } /// Converts the header type into its equivalent 512 byte block fn to_disk_block(header: &StandardDiskHeader) -> RawBlock { // Now, this might seem stupid to reconstruct the struct immediately, but // doing this ensures that if the struct is updated, we have to look at this function // as well. let StandardDiskHeader { flags, disk_number, block_usage_map, } = header; // Create the buffer for the header let mut buffer: [u8; 512] = [0u8; 512]; // Magic numbers! buffer[0..8].copy_from_slice("Fluster!".as_bytes()); // Now the flags buffer[8] = flags.bits(); // The disk number buffer[9..9 + 2].copy_from_slice(&disk_number.to_le_bytes()); // The block map buffer[148..148 + 360].copy_from_slice(block_usage_map); // Now CRC it add_crc_to_block(&mut buffer); // Make the RawBlock // Disk headers are always block 0. let block_origin = DiskPointer { disk: header.disk_number, block: 0, }; let finished_block: RawBlock = RawBlock { block_origin, data: buffer, }; // All done! finished_block } ================================================ FILE: src/pool/disk/standard_disk/block/header/header_struct.rs ================================================ // Imports use bitflags::bitflags; // Structs, Enums, Flags /// The header of a disk #[derive(Debug, PartialEq, Eq, Clone)] pub struct StandardDiskHeader { pub flags: StandardHeaderFlags, pub disk_number: u16, pub block_usage_map: [u8; 360], // not to be indexed directly, use a method to check. } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct StandardHeaderFlags: u8 { const Marker = 0b00100000; // Must be set. // 0b01000000; // Reserved for dense disk // 0b10000000; // Reserved for pool disk } } ================================================ FILE: src/pool/disk/standard_disk/block/header/mod.rs ================================================ mod header_methods; pub mod header_struct; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/standard_disk/block/header/tests.rs ================================================ // You need to test head? You can try on me, I guess... ================================================ FILE: src/pool/disk/standard_disk/block/inode/inode_methods.rs ================================================ // Inode a block, then he moved away. // Imports // Implementations use std::time::Duration; use std::time::SystemTime; use std::time::UNIX_EPOCH; use super::inode_struct::Inode; use super::inode_struct::InodeBlock; use super::inode_struct::InodeBlockFlags; use super::inode_struct::InodeFlags; use crate::error_types::block::BlockManipulationError; use crate::error_types::drive::DriveError; use crate::pool::disk::generic::block::crc::add_crc_to_block; use crate::pool::disk::generic::generic_structs::find_space::find_free_space; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::pool::disk::generic::io::cache::cache_io::CachedBlockIO; use crate::pool::disk::generic::{ block::block_structs::RawBlock, generic_structs::find_space::BytePingPong, }; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeLocation; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeOffsetPacking; use crate::pool::disk::standard_disk::block::inode::inode_struct::PackedInodeLocationFlags; use crate::pool::disk::standard_disk::block::inode::inode_struct::{ InodeDirectory, InodeFile, InodeTimestamp, }; impl From for InodeBlock { fn from(value: RawBlock) -> Self { from_raw_block(&value) } } // Add ability for inodes to have space searched for them impl BytePingPong for Inode { fn to_bytes(&self) -> Vec { self.as_bytes() } fn from_bytes(bytes: &[u8]) -> Self { Self::from_bytes(bytes) } } impl InodeBlock { pub fn to_block(&self) -> RawBlock { to_raw_bytes(self) } pub fn from_block(block: &RawBlock) -> Self { from_raw_block(block) } /// Create a new inode block /// /// New Inode blocks are the new final block on the disk. /// New Inode blocks do not point to the next block (as none exists). /// Caller is responsible with updating previous block to point to this new block. pub fn new(block_origin: DiskPointer) -> Self { new_inode_block(block_origin) } /// Try to add an Inode to this block. /// Updates the byte usage counter. /// /// This does NOT automatically flush information to the disk. /// /// Returns the offset of the added inode pub fn try_add_inode(&mut self, inode: Inode) -> Result { inode_block_try_add_inode(self, inode) } /// Removes inodes based off of the offset into the block. /// Updates the byte usage counter. /// This does not remove the data the inode points to. The caller is responsible for propagation. /// /// Does not flush to disk. /// /// Returns nothing. pub fn try_remove_inode(&mut self, inode_offset: u16) -> Result<(), BlockManipulationError> { inode_block_try_remove_inode(self, inode_offset) } /// Try and read an inode from the block. /// /// Returns Inode. pub fn try_read_inode(&self, inode_offset: u16) -> Result { inode_block_try_read_inode(self, inode_offset) } /// Set a new destination on a block. /// /// Does not flush the new destination to disk, only updates it. /// /// May swap disks. Will not return to original disk. pub fn new_destination(&mut self, pointer: DiskPointer) { // dont feel like splitting this into a function rn, sue me. self.next_inode_block = pointer; } /// Underlying inode information (File size for example) may change, so we need to be able to /// update our insides. /// /// Updates internal data, flushes to disk. /// /// Incoming inode data must be the same size as the pre-existing inode. /// Will panic if incoming inode is of the wrong size. /// Will panic if there is not an inode at the location you are trying to overwrite. /// /// May swap disks, does not return to caller disk. Ends up wherever the inode block originally came from. /// /// Returns nothing, pub fn update_inode(&mut self, inode_offset: u16, updated_inode: Inode,) -> Result<(), DriveError> { // get the item at the current offset let old = self.try_read_inode(inode_offset).expect("Caller should provide valid offset"); // Find out how big that is let old_size = old.as_bytes().len(); // Make sure the new one is the right size let new_size = updated_inode.as_bytes().len(); assert_eq!(old_size, new_size, "To update an inode, the new inode must be the same size as the old one."); // Now that we know we can safely perform this operation, we will directly edit ourselves. self.inodes_data[inode_offset as usize..inode_offset as usize + old_size].copy_from_slice(&updated_inode.as_bytes()); // Now we need to flush these updates to disk let raw = self.to_block(); CachedBlockIO::update_block(&raw)?; // All done! Ok(()) } } // // Functions // fn inode_block_try_read_inode(block: &InodeBlock, offset: u16) -> Result { // Attempt to read in the inode at this location // extract function at bottom of file // Bounds checking if offset as usize > block.inodes_data.len() { // We cannot read past the end of the end of the data! return Err(BlockManipulationError::Impossible); } // get a slice with that inode and deserialize it Ok(Inode::from_bytes(&block.inodes_data[offset as usize..])) } fn inode_block_try_remove_inode( block: &mut InodeBlock, inode_offset: u16, ) -> Result<(), BlockManipulationError> { // Attempt to remove an inode from the block // Assumption: // Caller gave us a valid offset. // There isn't a great way to check this besides scanning through the entire block to find all of the // inodes, but we can at least check the marker bit. // Additionally, if there are extra unused bits set in the flags, this is almost certainly an invalid offset. let flags = match InodeFlags::from_bits(block.inodes_data[inode_offset as usize]) { Some(ok) => ok, None => { // Unused bits are set. This cannot be the start of an inode. return Err(BlockManipulationError::Impossible); } }; if !flags.contains(InodeFlags::MarkerBit) { // Missing flag. // This cannot be the beginning of an inode. return Err(BlockManipulationError::Impossible); }; // Assumption: There is a valid inode at the provided offset // Yes the cast back and forth is silly, but at least its easy. let inode_to_remove_length: usize = Inode::from_bytes(&block.inodes_data[inode_offset as usize..]) .as_bytes() .len(); // Blank out those bytes // This range is inclusive because we are removing the last byte of the item as well, not just up to the last byte. block.inodes_data[inode_offset as usize..inode_offset as usize + inode_to_remove_length] .iter_mut() .for_each(|byte| *byte = 0); // sanity check, bytes are now empty #[cfg(test)] { for i in 0..inode_to_remove_length { assert_eq!(block.inodes_data[inode_offset as usize + i], 0, "Zeroed out an Inode, but it didn't end up blank!") } } // update how many bytes are free block.bytes_free += inode_to_remove_length as u16; // Done Ok(()) } fn inode_block_try_add_inode( inode_block: &mut InodeBlock, new_inode: Inode, ) -> Result { // Attempt to add an inode to the block. // Check if we have room for the new inode. let new_inode_bytes: Vec = new_inode.as_bytes(); let new_inode_length: usize = new_inode_bytes.len(); if new_inode_length > inode_block.bytes_free.into() { // We don't have room for this inode. The caller will have to use another block. return Err(BlockManipulationError::OutOfRoom); } // find a spot to put our new Inode let offset = match find_free_space::(&inode_block.inodes_data, new_inode_length) { Some(ok) => ok, None => { // couldn't find enough space, block must be fragmented. // Defrag is hard with only pointers that go in one direction... So no defrag. return Err(BlockManipulationError::OutOfRoom); } }; // Put in the Inode inode_block.inodes_data[offset..offset + new_inode_length].copy_from_slice(&new_inode_bytes); // Subtract the new space we've taken up // Cast from usize to u16 should be fine in all cases, // how would an inode be more than 2^16 bytes? lol. inode_block.bytes_free -= new_inode_length as u16; // Return that offset, we're done. // Cast: max of 501 is < u16. Safe. Ok(offset as u16) } fn new_inode_block(block_origin: DiskPointer) -> InodeBlock { // Create the flags // No default flags are required. let flags: InodeBlockFlags = InodeBlockFlags::empty(); // An inode block with no content has 501 bytes free. let bytes_free: u16 = 501; // Since this is the final block on the disk, and we obviously cant // point to the next disk, since we dont know if it even exists. // Thus, this is the end of the Inode chain. let next_inode_block: DiskPointer = DiskPointer::new_final_pointer(); // A new inode block has no inodes in it. // Special care must be taken by the caller to // ensure to put the root inode into the root disk. let inodes_data: [u8; 501] = [0u8; 501]; // all done InodeBlock { flags, bytes_free, next_inode_block, inodes_data, block_origin, } } fn from_raw_block(block: &RawBlock) -> InodeBlock { // Flags let flags: InodeBlockFlags = InodeBlockFlags::from_bits_retain(block.data[0]); // Bytes free let bytes_free: u16 = u16::from_le_bytes(block.data[1..1 + 2].try_into().expect("2 into 2")); // Next inode block let next_inode_block: DiskPointer = DiskPointer::from_bytes(block.data[3..3 + 4].try_into().expect("4 into 4")); // Inodes let inodes_data: [u8; 501] = block.data[7..7 + 501].try_into().expect("501 into 501"); // From dust we came let block_origin: DiskPointer = block.block_origin; // All done InodeBlock { flags, bytes_free, next_inode_block, inodes_data, block_origin, } } fn to_raw_bytes(block: &InodeBlock) -> RawBlock { let InodeBlock { flags, bytes_free, next_inode_block, inodes_data, block_origin, // And to dust we shall return. } = block; let mut buffer: [u8; 512] = [0u8; 512]; // Flags buffer[0] = flags.bits(); // Bytes free buffer[1..1 + 2].copy_from_slice(&bytes_free.to_le_bytes()); // next inode block buffer[3..3 + 4].copy_from_slice(&next_inode_block.to_bytes()); // inodes buffer[7..7 + 501].copy_from_slice(inodes_data); // crc add_crc_to_block(&mut buffer); // Make the block let final_block: RawBlock = RawBlock { block_origin: *block_origin, data: buffer, }; final_block } // // impl for subtypes // impl Inode { pub(super) fn as_bytes(&self) -> Vec { let mut vec: Vec = Vec::with_capacity(37); // max size of an inode // flags vec.push(self.flags.bits()); // Inode data // There should never be both a file and a directory in an inode. if let Some(directory) = self.directory { vec.extend(directory.as_bytes()); } if let Some(file) = self.file { vec.extend(file.as_bytes()); } // Timestamps // Created vec.extend(self.created.to_bytes()); // Modified vec.extend(self.modified.to_bytes()); // All done. vec } /// Will only read the first inode in provided slice. /// No validation is done to check if this is a valid inode! /// Caller MUST ensure this is a valid slice that contains an inode starting /// at bit zero, otherwise no guarantees can be made about the returned inode. pub(super) fn from_bytes(bytes: &[u8]) -> Self { let mut timestamp_offset: usize = 0; // Flags let flags: InodeFlags = InodeFlags::from_bits(bytes[0]).expect("Flags should only have used bits set."); timestamp_offset += 1; // We must have the marker bit. assert!(flags.contains(InodeFlags::MarkerBit), "Inodes must contain the marker bit."); // File or directory let file: Option = if flags.contains(InodeFlags::FileType) { timestamp_offset += 12; Some(InodeFile::from_bytes( bytes[1..1 + 12].try_into().expect("12 = 12"), )) } else { None }; let directory: Option = if !flags.contains(InodeFlags::FileType) { timestamp_offset += 4; Some(InodeDirectory::from_bytes( bytes[1..1 + 4].try_into().expect("4 = 4"), )) } else { None }; // Timestamps // Created let created: InodeTimestamp = InodeTimestamp::from_bytes( bytes[timestamp_offset..timestamp_offset + 12] .try_into() .expect("12 = 12"), ); // Created timestamp is 12 bytes. timestamp_offset += 12; // Modified let modified: InodeTimestamp = InodeTimestamp::from_bytes( bytes[timestamp_offset..timestamp_offset + 12] .try_into() .expect("12 = 12"), ); // Done. Self { flags, file, directory, created, modified, } } } impl InodeFile { fn as_bytes(&self) -> [u8; 12] { let mut buffer: [u8; 12] = [0u8; 12]; buffer[..8].copy_from_slice(&self.size.to_le_bytes()); buffer[8..].copy_from_slice(&self.pointer.to_bytes()); buffer } fn from_bytes(bytes: [u8; 12]) -> Self { Self { size: u64::from_le_bytes(bytes[..8].try_into().expect("8 = 8")), pointer: DiskPointer::from_bytes(bytes[8..].try_into().expect("4 = 4")), } } } impl InodeDirectory { fn as_bytes(&self) -> [u8; 4] { self.pointer.to_bytes() } fn from_bytes(bytes: [u8; 4]) -> Self { Self { pointer: DiskPointer::from_bytes(bytes), } } // Converts a disk pointer into a directory pub fn from_disk_pointer(pointer: DiskPointer) -> Self { Self { pointer } } } impl InodeTimestamp { pub(super) fn to_bytes(self) -> [u8; 12] { let mut buffer: [u8; 12] = [0u8; 12]; buffer[..8].copy_from_slice(&self.seconds.to_le_bytes()); buffer[8..].copy_from_slice(&self.nanos.to_le_bytes()); buffer } pub(super) fn from_bytes(bytes: [u8; 12]) -> Self { Self { seconds: u64::from_le_bytes(bytes[..8].try_into().expect("8 = 8")), nanos: u32::from_le_bytes(bytes[8..].try_into().expect("4 = 4")), } } // Create a timestamp that refers to the current moment in time. pub fn now() -> Self { // Get the time let now = SystemTime::now(); let duration_since_epoch = now .duration_since(UNIX_EPOCH) .expect("You shouldn't be using fluster in the 1960s. If you are, email me."); Self { seconds: duration_since_epoch.as_secs(), nanos: duration_since_epoch.subsec_nanos(), } } } // impl InodeFlags { // pub fn new() -> Self { // // We need the marker bit. // InodeFlags::MarkerBit // } // } impl InodeLocation { pub fn as_bytes(&self, destination_disk: u16) -> Vec { let mut vec: Vec = Vec::with_capacity(6); // Max size of this type // We can ignore the offset contained within self, since we dont write that value, we // write it with packed instead. // Calculate the flags let (mut flags, offset) = self.packed.extract(); // Can we make this a diskless location? if destination_disk == self.pointer.disk { // Yes flags.insert(PackedInodeLocationFlags::RequiresDisk); } else { // No flags.remove(PackedInodeLocationFlags::RequiresDisk); } // Pack that back down to update the flags. let packed = InodeOffsetPacking::new(flags, offset); // The packed offset stuffs vec.extend_from_slice(&packed.inner.to_le_bytes()); // Disk number, if required. if !flags.contains(PackedInodeLocationFlags::RequiresDisk) { vec.extend_from_slice(&self.pointer.disk.to_le_bytes()); } // Block on disk vec.extend_from_slice(&self.pointer.block.to_le_bytes()); vec } /// Turn the location into bytes. Takes in as many bytes as needed, and no more. /// /// Returns how many bytes it took to create this. /// /// Assumes its fed valid bytes, will panic if not. /// /// Must pass in the disk this came from. pub fn from_bytes(bytes: &[u8], origin_disk: u16) -> (u8, Self) { let mut index: usize = 0; // Extract the flags let packed_number = u16::from_le_bytes(bytes[index..index + 2].try_into().expect("2 = 2")); let packed = InodeOffsetPacking::from_u16(packed_number); let (flags, offset) = packed.extract(); index += 2; // Make sure this is a valid InodeLocation assert!(flags.contains(PackedInodeLocationFlags::MarkerBit), "Inodes must contain the flag bit."); // Disk number // Only need to grab if flag is set let disk: u16 = if flags.contains(PackedInodeLocationFlags::RequiresDisk) { origin_disk } else { // Go get it let tmp = u16::from_le_bytes(bytes[index..index + 2].try_into().expect("2 = 2")); index += 2; tmp }; // The block let block: u16 = u16::from_le_bytes(bytes[index..index + 2].try_into().expect("2 = 2")); index += 2; // Make the pointer let pointer: DiskPointer = DiskPointer { disk, block, }; // Re-assemble packed, we dont use any flags besides the marker bit outside of here. let fit_girl_repacked = InodeOffsetPacking::new(PackedInodeLocationFlags::MarkerBit, offset); // All done. let done = InodeLocation { packed: fit_girl_repacked, pointer, offset }; (index as u8, done) } /// Make a new one! pub(crate) fn new(pointer: DiskPointer, offset: u16) -> Self { // In theory the flags are not read, they are calculated on write. // Just set the marker bit. Self { packed: InodeOffsetPacking::new(PackedInodeLocationFlags::MarkerBit, offset), pointer, offset, } } } impl InodeBlock { /// Find the next block in the inode chain, if it exists. pub fn next_block(&self) -> Option { // First check if we have anywhere to go if self.next_inode_block.no_destination() { // Nowhere to go. None } else { Some(self.next_inode_block) } } } impl InodeFile { /// New empty inode files /// Pointer points to the first extent block. pub fn new(disk_pointer: DiskPointer) -> Self { Self { size: 0, // New files are empty pointer: disk_pointer, } } /// Get size of a file pub fn get_size(&self) -> u64 { self.size } /// Set the size of the file /// /// Does not flush change to disk. pub fn set_size(&mut self, size: u64) { self.size = size; } } // Extract an inode into its inner type impl Inode { pub fn extract_file(&self) -> Option { self.file } pub fn extract_directory(&self) -> Option { self.directory } // /// All inodes point somewhere. // pub fn get_pointer(&self) -> DiskPointer { // if let Some(dir) = self.extract_directory() { // dir.pointer // } else { // self.extract_file().expect("Not a directory, so it must be a file.").pointer // } // } } // convert an inode timestamp into SystemTime impl From for SystemTime { fn from(value: InodeTimestamp) -> Self { // All of our measurements are relative to the Unix Epoch // https://xkcd.com/376/ let epoch: SystemTime = SystemTime::UNIX_EPOCH; // Get the offset let duration: Duration = Duration::new(value.seconds, value.nanos); // 88MPH epoch.checked_add(duration).expect("This will break in 2038. Until then, hi dad!") } } // Deconstruct packed InodeLocation information to get the flags and inode offset impl InodeOffsetPacking { /// Extract the flags and an offset. pub(super) fn extract(&self) -> (PackedInodeLocationFlags, u16) { // First we get the flags, the flags expect 8 bits, so we nab those. // Shift right 8 times to bring the flags in line. let flag_bits: u8 = (self.inner >> 8) as u8; // This can leave a trailing bit at the end, but truncating will remove it. let flags: PackedInodeLocationFlags = PackedInodeLocationFlags::from_bits_truncate(flag_bits); // Now we need the number, which we can get by laying the flags back on top of the inner value. // get the flag bits, invert them, shift them back over, then mask em out // The only remaining bits must be the offset value. let truncated_flag_bits: u16 = !((flags.bits() as u16) << 8); let offset: u16 = self.inner & truncated_flag_bits; // All done. (flags, offset) } /// Make a new one /// This can only be used down here, you should be using InodeLocation::new(). fn new(flags: PackedInodeLocationFlags, offset: u16) -> Self { // See extract() for more docs about what is goin on here. let inner: u16 = offset | ((flags.bits() as u16) << 8); Self { inner, } } /// From a u16. /// /// Yes this is silly. pub(super) fn from_u16(incoming: u16) -> Self { Self { inner: incoming } } } ================================================ FILE: src/pool/disk/standard_disk/block/inode/inode_struct.rs ================================================ // Inode layout // Imports use bitflags::bitflags; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; // Structs, Enums, Flags #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(crate) struct Inode { pub flags: InodeFlags, pub file: Option, pub directory: Option, pub created: InodeTimestamp, pub modified: InodeTimestamp, } #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct InodeFile { /// The size of the pointed to file in bytes. pub(super) size: u64, /// Points to the first extent block in the chain for this file. pub(crate) pointer: DiskPointer, } #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(crate) struct InodeDirectory { /// Points to a DirectoryBlock. pub(crate) pointer: DiskPointer, } #[derive(Debug, PartialEq, Eq, Clone, Copy)] /// Relative to Unix Epoch pub struct InodeTimestamp { pub(crate) seconds: u64, pub(crate) nanos: u32, } // Points to a specific inode globally #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct InodeLocation { /// The InodeBlock that this offset goes into can only hold 501 bytes, thus /// many of the bits in the u16 are unused. (only 9 out of 16 bits are used). /// Thus the upper 7 bits are free for other uses. We can pack flags in here too. /// When reconstructing disk pointers, we need to know if this is local or not, so /// we will use the second highest bit on the offset to denote if we need a disk number to /// reconstruct the pointer. /// /// We will use the highest bit as a marker bit for reading. /// /// For ordering sake, it'll also go at the front of the type. /// /// Must also be private, since you cannot get the usual offset without a method call now. pub(super) packed: InodeOffsetPacking, /// Disk component automatically gets tossed and added on write/read. pub(crate) pointer: DiskPointer, /// This offset is extracted from packed during read pub(crate) offset: u16, } #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(super) struct InodeOffsetPacking { pub(super) inner: u16, // Combination of flags and the offset. } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct InodeFlags: u8 { const FileType = 0b00000001; // Set if this is a file const MarkerBit = 0b10000000; // Always set } } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub(super) struct PackedInodeLocationFlags: u8 { const MarkerBit = 0b10000000; // Always set const RequiresDisk = 0b01000000; // Does this InodeLocation require a disk to be reconstructed? // 5 unused bits. } } // The block #[derive(Debug, PartialEq, Eq, Clone)] pub struct InodeBlock { pub(super) flags: InodeBlockFlags, // Manipulating Inodes must be done through methods on the struct pub(super) bytes_free: u16, pub(super) next_inode_block: DiskPointer, // At runtime its useful to know where this block came from. // This doesn't need to get written to disk. pub block_origin: DiskPointer, // This MUST be set. it cannot point nowhere. pub(super) inodes_data: [u8; 501], } bitflags! { #[derive(Debug, PartialEq, Eq, Clone, Copy)] pub struct InodeBlockFlags: u8 { // Currently unused. } } ================================================ FILE: src/pool/disk/standard_disk/block/inode/mod.rs ================================================ mod inode_methods; pub mod inode_struct; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/standard_disk/block/inode/tests.rs ================================================ // inode the tests. // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] // Imports // Tests use crate::error_types::block::BlockManipulationError; use crate::pool::disk::standard_disk::block::inode::inode_struct::Inode; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeFile; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeBlock; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeFlags; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeLocation; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeDirectory; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeTimestamp; use rand::Rng; use test_log::test; // We want to see logs while testing. #[test] fn blank_inode_block_serialization() { // Just like the directory blocks, we must spoof the disk read. let block_origin = DiskPointer { disk: 420, block: 69, }; let test_block: InodeBlock = InodeBlock::new(block_origin); let serialized = test_block.to_block(); let deserialized = InodeBlock::from_block(&serialized); assert_eq!(test_block, deserialized) } #[test] fn fill_inode_block() { let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block: InodeBlock = InodeBlock::new(block_origin); let mut added_inodes: Vec = Vec::new(); let mut inode_offsets: Vec = Vec::new(); loop { let inode: Inode = Inode::get_random(); let add_result = test_block.try_add_inode(inode); if add_result.is_err() { // It must be full, its impossible to fragment without removing items. assert_eq!(add_result.err().unwrap(), BlockManipulationError::OutOfRoom); break; } // Keep track of all added inodes so we can validate them. added_inodes.push(inode); inode_offsets.push(add_result.unwrap()); } // Ensure all the inodes are present and valid. for i in 0..added_inodes.len() { // read it let read_inode: Inode = test_block.try_read_inode(inode_offsets[i]).unwrap(); // Make sure they're the same. assert_eq!(added_inodes[i], read_inode); } } #[test] fn filled_inode_block_serialization() { for _ in 0..1000 { let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block: InodeBlock = InodeBlock::new(block_origin); // Fill with random inodes until we run out of room. loop { let add_result = test_block.try_add_inode(Inode::get_random()); if add_result.is_err() { // It must be full, its impossible to fragment without removing items. assert_eq!(add_result.err().unwrap(), BlockManipulationError::OutOfRoom); break; } } // Check serialization let serialized = test_block.to_block(); let deserialized = InodeBlock::from_block(&serialized); assert_eq!(test_block, deserialized) } } #[test] fn add_and_read_inode() { for _ in 0..1000 { let block_origin = DiskPointer { disk: 420, block: 69, }; let mut test_block: InodeBlock = InodeBlock::new(block_origin); let inode: Inode = Inode::get_random(); let offset = test_block.try_add_inode(inode).unwrap(); let read_inode = test_block.try_read_inode(offset).unwrap(); assert_eq!(inode, read_inode); } } #[test] // Make sure the offsets are working correctly fn inode_location_consistency() { for _ in 0..1000 { let new: InodeLocation = InodeLocation::get_random(); let disk_number: u16 = new.pointer.disk; let frosted_flaked = new.as_bytes(disk_number); let (_, we_have_the_technology) = InodeLocation::from_bytes(&frosted_flaked, disk_number); assert_eq!(new, we_have_the_technology); } } #[test] // Ensure inodes are the correct size for their subtype fn inode_correct_sizes() { for _ in 0..1000 { let test_inode: Inode = Inode::get_random(); if test_inode.file.is_some() { // A file inode should be 37 bytes long assert_eq!(test_inode.as_bytes().len(), 37) } else { // A directory inode should be 29 bytes long assert_eq!(test_inode.as_bytes().len(), 29) } } } #[test] // Inodes should be the same size when re/deserializing them fn inode_consistent_serialization() { for _ in 0..1000 { let inode: Inode = Inode::get_random(); let serial = inode.as_bytes(); let deserial = Inode::from_bytes(&serial); let re_serial = deserial.as_bytes(); let re_deserial = Inode::from_bytes(&re_serial); // Original Inode survived assert_eq!(inode, re_deserial); // Intermediate did not change assert_eq!(deserial, re_deserial); // byte versions are the same assert_eq!(serial, re_serial); } } #[test] fn timestamp_consistent_serialization() { for _ in 0..1000 { let inode: InodeTimestamp = InodeTimestamp::get_random(); let serial = inode.to_bytes(); let deserial = InodeTimestamp::from_bytes(serial); let re_serial = deserial.to_bytes(); let re_deserial = InodeTimestamp::from_bytes(re_serial); // Original InodeTimestamp survived assert_eq!(inode, re_deserial); // Intermediate did not change assert_eq!(deserial, re_deserial); // byte versions are the same assert_eq!(serial, re_serial); } } // Impl to make randoms #[cfg(test)] impl Inode { pub(crate) fn get_random() -> Self { use rand::random_bool; if random_bool(0.5) { // A file let mut flags = InodeFlags::MarkerBit; flags.insert(InodeFlags::FileType); Inode { flags, file: Some(InodeFile::get_random()), directory: None, created: InodeTimestamp::get_random(), modified: InodeTimestamp::get_random(), } } else { // A directory Inode { flags: InodeFlags::MarkerBit, file: None, directory: Some(InodeDirectory::get_random()), created: InodeTimestamp::get_random(), modified: InodeTimestamp::get_random(), } } } } #[cfg(test)] impl InodeFile { pub(crate) fn get_random() -> Self { let mut random = rand::rng(); InodeFile { size: random.random(), pointer: DiskPointer::get_random(), } } } #[cfg(test)] impl InodeTimestamp { pub(crate) fn get_random() -> Self { let mut random = rand::rng(); InodeTimestamp { seconds: random.random(), nanos: random.random(), } } } #[cfg(test)] impl InodeLocation { #[cfg(test)] pub(crate) fn get_random() -> Self { let mut random = rand::rng(); let pointer: DiskPointer = DiskPointer { disk: random.random(), block: random.random(), }; // random offset // Not testing the entire range but whatever let offset: u16 = random.random_range(0..250); InodeLocation::new(pointer, offset) } } #[cfg(test)] impl InodeDirectory { pub(crate) fn get_random() -> Self { InodeDirectory { pointer: DiskPointer::get_random(), } } } ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/mod.rs ================================================ pub mod movement; pub mod read; #[cfg(test)] pub mod tests; pub(crate) mod types; pub mod write; ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/movement.rs ================================================ // Helpers to move between directories use std::path::{ Component, Path }; use log::debug; use crate::{error_types::drive::DriveError, pool::{ disk::{ generic::{ generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::{ directory::directory_struct::DirectoryBlock, inode::inode_struct::InodeBlock, io::directory::types::NamedItem, }, }, pool_actions::pool_struct::Pool }, tui::{ notify::NotifyTui, tasks::TaskType } }; impl DirectoryBlock { /// Attempts to open a directory in the current directory block. /// This will check if the directory already exists, if it doesn't, /// Ok(None) will be returned, because there was no directory to open. /// /// May swap disks, will end up on whatever disk the new directory is located on, unless /// you specify a return location. /// /// If there is no new directory, this will end up wherever the end of the input directory was, unless /// you set the return disk. pub fn change_directory( self, directory_name: String, ) -> Result, DriveError> { let handle = NotifyTui::start_task(TaskType::ChangingDirectory(directory_name.clone()), 3); debug!("Attempting to CD to `{directory_name}`"); // Get all items in this directory let found_dir = self.find_item(&NamedItem::Directory(directory_name))?; NotifyTui::complete_task_step(&handle); // Return if the dir did not exist, or keep goin let wanted = match found_dir { Some(found) => found, None => { // The directory did not exist. NotifyTui::complete_multiple_task_steps(&handle, 2); NotifyTui::finish_task(handle); debug!("Directory did not exist."); return Ok(None) }, }; // Directory exists, time to open that bad boy // Extract the location let final_destination = &wanted.location; debug!( "Directory claims to live at: disk {} block {} offset {}", final_destination.pointer.disk, final_destination.pointer.block, final_destination.offset ); // Since we got these items from self.list, all of these inode locations MUST have a disk destination // already set for us. So we dont have to check. // Load! // Now this doesn't point to the next directory block, it points to the next _Inode_ block // that points to it. let pointer: DiskPointer = final_destination.pointer; let inode_block = InodeBlock::from_block(&CachedBlockIO::read_block(pointer)?); NotifyTui::complete_task_step(&handle); // Now read in the inode let inode = inode_block .try_read_inode(final_destination.offset) .expect("Directories in a DirectoryBlock should point to a valid inode!"); // Where is the block? let actual_next_block = inode .directory .expect("Should point to a directory inode, not a file.") .pointer; // Just in case... assert!(!actual_next_block.no_destination(), "Tried to open a new directory block, but pointer had no destination"); // Go go go! let new_dir_block: DirectoryBlock = DirectoryBlock::from_block(&CachedBlockIO::read_block(actual_next_block)?); NotifyTui::complete_task_step(&handle); // All done! Enjoy the new block. Ok(Some(new_dir_block)) } /// Attempts to open any directory in the pool. /// /// Will automatically grab the root directory. pub(crate) fn try_find_directory(maybe_path: Option<&Path>) -> Result, DriveError> { debug!("Attempting to find and open a directory..."); // Pretty simple loop, bail if the directory does not exist at any level. let mut current_directory: DirectoryBlock; // Load in the root directory current_directory = Pool::get_root_directory()?; // If no path was supplied, this is the root directory. let path = match maybe_path { Some(ok) => ok, None => { // This must be root. debug!("No path was provided to find, it is assumed the caller wants the root directory."); return Ok(Some(current_directory)) }, }; debug!("Looking for `{}`...", path.display()); // Easy way out, if the incoming path is empty, that means its the root directory itself. if path.iter().count() == 0 { // There are no paths above the root, we are trying to load the root. return Ok(Some(current_directory)) } // Split the path into folder names for folder in path.components() { // Is this the start? if folder == Component::RootDir { // Skip continue; } // Try to move into the folder if let Some(new_dir) = current_directory.change_directory(folder.as_os_str().to_str().expect("Should be valid utf8").to_string())? { // Directory exists. Move in. current_directory = new_dir; continue; } else { // No such directory return Ok(None) } } // Now that we're out of the for loop, we must be in the correct directory. Ok(Some(current_directory)) } } ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/read.rs ================================================ // Higher level abstractions for reading directories. use log::{debug, error, warn}; use crate::{error_types::drive::DriveError, pool::{ disk::{ generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryItem, DirectoryItemFlags }, io::directory::types::NamedItem, } }, pool_actions::pool_struct::Pool }, tui::{notify::NotifyTui, tasks::TaskType}}; impl DirectoryBlock { /// Check if this directory contains an item with the provided name and type. /// This checks the entire directory, not just the current block. /// /// Returns Option if it exists. /// /// May swap disks. pub fn find_item( &self, item_to_find: &NamedItem, ) -> Result, DriveError> { let extracted_debug = item_to_find.debug_strings(); debug!( "Checking if a directory contains the {} `{}`...", extracted_debug.0, extracted_debug.1 ); // Special case if we are trying to find the root directory. if self.is_root() { // This is the root directory, are we trying to find a nameless directory? if *item_to_find == NamedItem::Directory("".to_string()) { // This was a lookup on the root directory. debug!("Caller was looking for the root directory item, skipping lookup..."); return Ok(Some(Pool::get_root_directory_item())); } } // No need to have a task if its the root dir, since that's nearly instant. let handle = NotifyTui::start_task(TaskType::FindItemInDirectory(extracted_debug.1.to_string()), 2); // Get items let items: Vec = self.list()?; NotifyTui::complete_task_step(&handle); // Look for the requested item in the new vec, the index into this vec will be the same // as the index into the og items vec if let Some(item) = item_to_find.find_in(&items) { // It's in there! NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); debug!("Yes it did."); Ok(Some(item)) } else { // The item wasn't in there. NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); debug!("No it didn't."); Ok(None) } } /// Returns an Vec of all items in this directory ordered alphabetically descending. /// /// Returned DirectoryItem(s) will have their InodeLocation's disk set. /// /// May swap disks. pub fn list(&self) -> Result, DriveError> { go_list_directory(self) } /// Get the size of a directory by totalling all of the items contained within it. /// /// Does not recurse into sub-directories. (Seems to be standard behavior in ls -l) /// /// Returns the size in bytes. pub fn get_size(&self) -> Result { debug!("Getting size of a directory..."); // get all the items debug!("Listing items..."); let items = self.list()?; debug!("Totaling up item sizes..."); let mut total_size: u64 = 0; for item in items { // Ignore if this is a directory. // We don't recurse into the next directory, we only get the size of the items // directly contained within this directory. if item.flags.contains(DirectoryItemFlags::IsDirectory) { continue; } // Get the size of this file let inode = item.get_inode()?; let file = inode.extract_file().expect("The inode the directory item points at should be a file."); total_size += file.get_size() } // All done debug!("Size obtained. `{total_size}` bytes."); Ok(total_size) } /// Check if this DirectoryBlock is the head of the root directory. /// /// This will return false on any other block than the head block. fn is_root(&self) -> bool { // Lives in a static place. static ROOT_BLOCK_LOCATION: DiskPointer = DiskPointer { disk: 1, block: 2, }; self.block_origin == ROOT_BLOCK_LOCATION } /// Extracts an item from a directory block, blanking out the space it used to occupy. /// /// This looks for the item in the entire directory, not just the block this was called on. /// Due to this, we assume this is being called on the head of the DirectoryBlock chain. /// /// Automatically flushes changes to disk if required. /// /// If you just want to get the item for reading or minor modifications, use find_item() /// /// Updates the passed in directory block. /// /// Returns nothing if the item did not exist. pub(crate) fn find_and_extract_item(&mut self, item_to_find: &NamedItem) -> Result, DriveError> { // Go find the item. // Nice struct to make dealing with this a bit nicer struct ItemFound { /// The item item: DirectoryItem, /// This is set if the removal of that item caused the block to be fully emptied. /// /// Thus, if this is set, this block needs to be deallocated, and have the block before it /// set to point to this new pointer, which points to the block _after_ the block that the item was found in. /// /// Slightly confusing. empty_thus_new_pointer: Option, /// Which block in the list we were contained within, indexed from front to back. origin_index: usize, } // Get the blocks let mut blocks: Vec = get_blocks(self.block_origin)?; // Find the item, and deduce what block it's in. // Index, the item, maybe pointer to the next block let mut find: Option = None; for (index, block) in blocks.iter_mut().enumerate() { // Is it in here? if let Some(found) = block.block_extract_item(item_to_find)? { // Cool! find = Some(ItemFound { item: found.0, empty_thus_new_pointer: found.1, origin_index: index, }); break } }; // Did we find the item? let found = match find { Some(ok) => ok, None => { // Item did not exist. return Ok(None); }, }; // If we didn't get a pointer, there is no required cleanup, since no blocks were emptied. let new_pointer = match found.empty_thus_new_pointer { Some(ok) => ok, None => { // No cleanup required! return Ok(Some(found.item)); }, }; // We got a pointer, thus a block was emptied. // If the block that was emptied was the first one in the list, don't need to do anything. // Sure, we could shuffle the head forwards, but adding things to directories searches front to back anyways, so // switching the pointers around would be needlessly complicated. if found.origin_index == 0 { // Cool! return Ok(Some(found.item)); } // We have emptied a block in the middle of the chain. We need to update the pointer behind us to point // past us. // This operation is independent to the block in front of us, so no update is required there. // If this was the last block in the chain, this will just point to the no_destination pointer, which just marks the // new end of the chain. let previous_block = &mut blocks[found.origin_index - 1]; // Now update that block with the new pointer previous_block.next_block = new_pointer; // Update it. let raw_ed = previous_block.to_block(); CachedBlockIO::update_block(&raw_ed)?; // Now we will free that block that was emptied. // Now delete the block that we emptied by freeing it. let release_me = blocks[found.origin_index].block_origin; let freed = Pool::free_pool_block_from_disk(&[release_me])?; // this should ALWAYS be 1 assert_eq!(freed, 1, "We should always free one block when removing an empty directory block in a chain."); // All done! // Update the incoming block head, in case we changed it. // Since we need to own this, we'll just pull it out of the vec. // The updated block order does not matter, since we're immediately dropping this afterwards. *self = blocks.swap_remove(0); Ok(Some(found.item)) } /// Extract an item from this directory block, if it exists. /// /// Will flush self to disk if block is updated. /// /// If the block is now empty, will also return Some() pointer it's next block, regardless /// if that block exists or not (will return a final pointer on the last block). /// /// Not a public function, use `find_and_extract_item`. fn block_extract_item(&mut self, item_to_find: &NamedItem) -> Result)>, DriveError> { // Do we have the requested item? if let Some(found) = item_to_find.find_in(&self.directory_items) { // Found the item! // Remove it from ourselves. self.try_remove_item(&found).expect("Guard, we already know its in there."); // Now flush ourselves to disk let raw_block = self.to_block(); CachedBlockIO::update_block(&raw_block)?; // If we are now empty, also return a pointer to the next block let maybe_pointer: Option = if self.get_items().is_empty() { // Yep Some(self.next_block) } else { None }; // Now return the item, and the possible pointer to the next block return Ok(Some((found, maybe_pointer))) } // Not in here. Ok(None) } /// Rename an item in place. /// /// Searches entire directory for the item. /// /// Assumes that the passed in directory block is the head. /// /// Returns true if the item existed and was renamed. /// /// Flushes change to disk. pub(crate) fn try_rename_item(&mut self, to_rename: &NamedItem, new_name: String) -> Result { // Since the size of the item might change (name length change) we cant just update the name directly, we have to // extract the item and re-add it. // This may move the item across disks, thus if its set to local, we must add the disk number. // If the disk number is no longer required after its written down, `add_item` will make it local again, // We also take in the directory item instead of the named item, since you shouldn't be holding onto it after this. // Make sure the name is valid. assert!(new_name.len() <= 255, "Name is too long."); // Get the item if let Some(mut exists) = self.find_and_extract_item(to_rename)? { // Copy it, just in case... let copy = exists.clone(); // Now rename it and put it back exists.name_length = new_name.len() as u8; exists.name = new_name; // If this doesn't work, the item is now gone forever lol, thus // we will check the result of this operation and try to put the item back if we can. let add_result = self.add_item(&exists); if add_result.is_ok() { // All good. Ok(true) } else { // Addition failed! warn!("Adding item during rename failed."); warn!("Attempting to restore non-renamed item..."); if self.add_item(©).is_ok() { // That worked warn!("Old item restored.") } else { error!("Failed to restore old item during rename failure! Item has been lost!"); // Well shit. Not much we can do. println!("Fluster has just lost your file/folder named `{}`, sorry!", copy.name); // we have to give up. panic!("File lost during rename."); } // We need to fail tests even if the item was restored. if cfg!(test) { panic!("Rename failure. Addition failed.") } // Now we are... fine? The item is still there, it just // wasn't renamed. Err(DriveError::Retry) } } else { // No such item. Ok(false) } } } // Functions fn go_list_directory( block: &DirectoryBlock, ) -> Result, DriveError> { let handle = NotifyTui::start_task(TaskType::ListingDirectory, 2); debug!("Listing a directory..."); // We need to iterate over the entire directory and get every single item. // We assume we are handed the first directory in the chain. // Get the blocks debug!("Getting blocks..."); let blocks = get_blocks(block.block_origin)?; debug!("This directory is made of {} blocks.", blocks.len()); NotifyTui::complete_task_step(&handle); // Get the items out of them debug!("Getting items..."); let mut items_found: Vec = blocks.into_iter().flat_map(move |block| { block.get_items() }).collect(); NotifyTui::complete_task_step(&handle); // Sort all of the items by name, not sure what internal order it is, but it will be // sorted by whatever comparison function String uses. debug!("Sorting..."); items_found.sort_by_key(|item| item.name.to_lowercase()); NotifyTui::finish_task(handle); debug!("Directory listing finished."); Ok(items_found) } /// Starting on the head block of a DirectoryBlock, return every block in the chain, in order. /// /// Does not take in a directory block, since we would need to consume it. /// /// Includes the head block. fn get_blocks(start_block_location: DiskPointer) -> Result, DriveError> { // Needing to consume the incoming block would be stinky. But since cloning is not allowed, and we // need to return the head block, we have to go get it ourselves. // This must be a valid block assert!(!start_block_location.no_destination(), "Provided head directory block does not exist!"); let raw_read: RawBlock = CachedBlockIO::read_block(start_block_location)?; let start_block: DirectoryBlock = DirectoryBlock::from_block(&raw_read); // We assume we are handed the first directory in the chain. // Cannot pre-allocate the vec, since we dont know how many blocks there will be. let mut blocks: Vec = Vec::new(); let mut current_dir_block: DirectoryBlock = start_block; // Big 'ol loop, we will break when we hit the end of the directory chain. loop { // Remember where the next block is let next_block: DiskPointer = current_dir_block.next_block; // Add the current block to the Vec blocks.push(current_dir_block); // I want to get off Mr. Bone's wild ride if next_block.no_destination() { // We're done! break; } // Load in the next block. let next_block_reader = CachedBlockIO::read_block(next_block)?; current_dir_block = DirectoryBlock::from_block(&next_block_reader); // Onwards! continue; } Ok(blocks) } ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/tests.rs ================================================ // Files, direct to thee. // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] use std::path::PathBuf; use log::debug; use rand::{rngs::ThreadRng, seq::{IndexedRandom, SliceRandom}, Rng}; use tempfile::{TempDir, tempdir}; use crate::{ filesystem::filesystem_struct::{FilesystemOptions, FlusterFS}, pool::{ disk::standard_disk::block::{directory::directory_struct::DirectoryItem, io::directory::types::NamedItem}, pool_actions::pool_struct::Pool, }, }; use test_log::test; // We want to see logs while testing. // Since these tests touch global state, they need to be forked, otherwise they will collide. #[test] fn add_directory() { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); // Now try adding a directory to the pool let mut block = Pool::get_root_directory().unwrap(); let _ = block.make_directory("test".to_string()).unwrap(); // We dont even check if its there, we just want to know if writing it failed. } #[test] // Make sure creating a file only makes one entry. fn creating_only_makes_one_directory() { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); // Now try adding a directory to the pool let mut block = Pool::get_root_directory().unwrap(); let result = block.make_directory("test".to_string()).unwrap(); let listed = block.list().unwrap(); // There should only be one directory. assert_eq!(listed.len(), 1); // The returned item should be the same assert_eq!(listed[0], result); } #[test] fn add_and_delete_directory() { let _fs = get_filesystem(); let mut block = Pool::get_root_directory().unwrap(); let _ = block.make_directory("test".to_string()).unwrap(); // Now delete that directory // Extract it let test_dir_item = block.find_and_extract_item(&NamedItem::Directory("test".to_string())).unwrap().unwrap(); // Call delete on it test_dir_item.get_directory_block().unwrap().delete_self(test_dir_item).unwrap(); // Directory should now be empty assert!(block.list().unwrap().is_empty()); } #[test] // Make sure directories shrink when items are removed. fn deletion_shrinks() { let _fs = get_filesystem(); let mut block = Pool::get_root_directory().unwrap(); // make a bunch of directories in here with large names to quickly expand the block for i in 0..200 { let _ = block.make_directory(format!("test_this_is_a_long_name_to_use_more_space_lol_{i}")).unwrap(); } // Remove all of them. for i in 0..200 { let delete_me = block.find_and_extract_item(&NamedItem::Directory(format!("test_this_is_a_long_name_to_use_more_space_lol_{i}"))).unwrap().unwrap(); delete_me.get_directory_block().unwrap().delete_self(delete_me).unwrap() } // Now make sure the empty block is only 1 block large. assert!(block.next_block.no_destination()); // Should also contain nothing assert!(block.list().unwrap().is_empty()); } #[test] // Try renaming some items fn rename_items() { let _fs = get_filesystem(); let mut block = Pool::get_root_directory().unwrap(); // A lot of directories let mut directories: Vec = Vec::new(); for i in 0..100 { directories.push(block.make_directory(format!("dir_{i}")).unwrap()); } // A lot of files let mut files: Vec = Vec::new(); for i in 0..100 { files.push(block.new_file(format!("file_{i}.txt")).unwrap()); } // Shuffle for fun let mut all_items: Vec = Vec::new(); all_items.extend(directories); all_items.extend(files); let mut random: ThreadRng = rand::rng(); all_items.shuffle(&mut random); // How many do we have let number_made: usize = all_items.len(); // Go rename all of them for item in all_items { // need a new name... hmmmmm let new_name: String = format!("new_{}", item.name); let renamed = block.try_rename_item(&item.into(), new_name).unwrap(); assert!(renamed) } // Make sure the directory still contains the correct number of items. (ie we didn't duplicate anything.) let list = block.list().unwrap(); assert_eq!(number_made, list.len()); } #[test] fn add_directory_and_list() { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); // Now try adding a directory to the pool let mut block = Pool::get_root_directory().unwrap(); let _ = block.make_directory("test".to_string(),).unwrap(); // try to find it again let new_block = Pool::get_root_directory().unwrap(); assert!( new_block .find_item(&NamedItem::Directory("test".to_string()),) .unwrap() .is_some() ); } #[test] fn nested_directory_hell() { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); let mut random: ThreadRng = rand::rng(); let mut name_number: usize = 0; // Create random directories at random places. for _ in 0..10_000 { // Load in the root let mut where_are_we = Pool::get_root_directory().unwrap(); // We will open random directories a few times, if they exist. loop { // List the current directory let square_holes = where_are_we.list().unwrap(); // If there is no directories at this level, we're done. if square_holes.is_empty() { break; } // Random chance to not go any deeper. if random.random_bool(0.2) { // Incentivize deep nesting. // not going any further. break; } // Random chance to go back to the root for more chaos. if random.random_bool(0.05) { // Back to root! where_are_we = Pool::get_root_directory().unwrap(); continue; } // Looks like we're entering a new directory. let destination = square_holes .choose(&mut random) .expect("Already checked if it was empty.") .name .clone(); // Go forth! where_are_we = where_are_we .change_directory(destination) .unwrap() .unwrap(); continue; } // Now that we've picked a directory, lets make a new one in here. // To make sure we dont end up with duplicate directory names, we just use a counter. let _ = where_are_we .make_directory(name_number.to_string()) .unwrap(); name_number += 1; } } #[test] /// Ensure that directories eventually start reporting other disks besides disk 1 by /// writing way too many directories. fn directories_switch_disks() -> Result<(), ()> { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); for i in 0..3000 { // There's only 2880 blocks on the first disk, assuming no overhead. let mut root_dir = Pool::get_root_directory().unwrap(); let _ = root_dir.make_directory(i.to_string()).unwrap(); } // Now make sure we actually have directories that claim to live on another disk let root_dir_done = Pool::get_root_directory().unwrap(); for dir in root_dir_done.list().unwrap() { if dir.location.pointer.disk != 1 { // Made it to another disk. debug!( "Made it onto another disk! Disk: {}", dir.location.pointer.disk ); return Ok(()); } } // They were all disk 1! panic!("All directories are on disk 1!"); } // We need a filesystem to run directory tests on. pub fn get_filesystem() -> FlusterFS { let temp_dir = get_new_temp_dir(); let floppy_drive: PathBuf = PathBuf::new(); // This is never read since we are using temporary disks. let fs_options = FilesystemOptions::new(Some(temp_dir.path().to_path_buf()), floppy_drive, Some(false), false); FlusterFS::start(&fs_options) // We don't actually have to mount it for non-integration testing. } // Temporary directories for virtual disks pub fn get_new_temp_dir() -> TempDir { let mut dir = tempdir().unwrap(); dir.disable_cleanup(true); debug!( "Created a temp directory at {}, it will not be deleted on exit.", dir.path().to_string_lossy() ); dir } ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/types.rs ================================================ // Helper types. use crate::pool::disk::standard_disk::block::directory::directory_struct::{ DirectoryItemFlags, DirectoryItem, }; // Need a way to search for either a file or a directory #[derive(Ord, PartialEq, Eq, PartialOrd)] pub(crate) enum NamedItem { File(String), Directory(String), } /// Specific types for named items. impl NamedItem { /// Extracts the type's name, and the name of that type. (ie "file", "test.txt") pub fn debug_strings(&self) -> (&'static str, &String) { match self { NamedItem::File(name) => ("file", name), NamedItem::Directory(name) => ("directory", name), } } /// Search a Vec for a NamedItem /// Returns the item if it exists. pub fn find_in(&self, to_search: &[DirectoryItem]) -> Option { // Searching with this function only does the minimum amount of clones // to deduce if the item is present or not, instead of needing to clone the // entire Vec to construct the new type. let item_found: Option<&DirectoryItem> = to_search.iter().find(|item: &&DirectoryItem| { let convert: NamedItem = NamedItem::from((*item).clone()); convert == *self }); item_found.cloned() } /// Helper function to figure out if this is a file pub fn is_file(&self) -> bool { matches!(self, NamedItem::File(_)) } // /// Helper function to figure out if this is a directory // pub fn is_directory(&self) -> bool { // matches!(self, NamedItem::Directory(_)) // } } /// Helper to turn DirectoryItem(s) into NamedItem(s) impl From for NamedItem { fn from(value: DirectoryItem) -> Self { if value.flags.contains(DirectoryItemFlags::IsDirectory) { NamedItem::Directory(value.name) } else { NamedItem::File(value.name) } } } ================================================ FILE: src/pool/disk/standard_disk/block/io/directory/write.rs ================================================ // Write a new directory into a directory block use log::{debug, error}; use crate::{error_types::drive::DriveError, pool::{ disk::{ generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO, }, standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryItem, DirectoryItemFlags }, inode::inode_struct::{ Inode, InodeBlock, InodeDirectory, InodeFlags, InodeTimestamp }, io::directory::types::NamedItem, }, }, pool_actions::pool_struct::Pool, }, tui::{notify::NotifyTui, tasks::TaskType}}; impl DirectoryBlock { /// Add a new item to this block, extending this block if needed. /// Updated blocks are written to disk. /// /// Updates the block that was passed in, since the contents of the block may have changed. /// /// Returns nothing. pub fn add_item( &mut self, item: &DirectoryItem, ) -> Result<(), DriveError> { go_add_item(self, item) } /// Creates a new directory block, and adds its location to the input block. /// Blocks are created and updated as needed. /// /// Updates the directory block that was passed in. /// /// The name of the new directory must be less than 256 characters long. /// Attempting to recreate an already existing directory will panic. /// /// Returns the created directory as a DirectoryItem. pub fn make_directory( &mut self, name: String, ) -> Result { go_make_directory(self, name) } /// Remove a the given directory. Removes all blocks that contained information about this directory, and updates /// all other blocks to remove references to this directory. /// /// You must also pass in the DirectoryItem that refers to this directory. You should extract it from the parent /// directory. /// /// The directory block must be empty of all items. /// /// Consumes the incoming block, since it will no longer exist. /// /// May swap disks. /// /// Returns nothing on success. pub fn delete_self(self, self_item: DirectoryItem) -> Result<(), DriveError> { // In theory, as long as the caller used an extracted directory item to call // this method, even if this call fails, all references to it will now be gone on // a directory level. So even if the inode or the block wasn't freed, its still // "deleted", but just leaked its blocks. Which is unfortunate, but fine. // Make sure the directory is empty. // Caller must check. if !self.list()?.is_empty() { panic!("Cannot delete an non-empty directory!"); } // Directories should shrink when items are removed. An empty // directory should only be 1 block in size. // Thus, we only have to deallocate ourselves. // Remove our inode. // We need to find it manually, since we will be updating the // inode block. let read: RawBlock = CachedBlockIO::read_block(self_item.location.pointer)?; let mut inode_block: InodeBlock = InodeBlock::from_block(&read); if let Err(error) = inode_block.try_remove_inode(self_item.location.offset) { // Not good. Something was wrong with the inode pointer. // This is a very very very bad thing. // The inode blocks may be corrupted. // We cannot recover. panic!("Tried to remove an invalid inode. Unrecoverable. {error:#?}") } // Write back the updated inode block CachedBlockIO::update_block(&inode_block.to_block())?; // Now we can free the block that the directory occupied. let _ = Pool::free_pool_block_from_disk(&[self.block_origin])?; // All done, directory deleted. drop(self); // So long drop(self_item); // Space Cowboy Ok(()) } } fn go_make_directory( directory: &mut DirectoryBlock, name: String, ) -> Result { debug!("Attempting to create a new directory with name `{name}`..."); // Check to make sure this block does not already contain the directory we are trying to add. // We dont care if listing the directory puts us somewhere else, because we're immediately going to // go get a new directory block, which would possibly just swap disks again, and our final update // to the original directory block has its origin already specified with block_origin. debug!("Checking if a directory with that name already exists..."); if directory .find_item(&NamedItem::Directory(name.clone()))? .is_some() { // We are attempting to create a duplicate item. error!("ATTEMPTED TO CREATE A DUPLICATE DIRECTORY! PANICKING!"); panic!("Attempted to create duplicate directory!") } debug!("Name is free."); // And make sure the name isn't too long. assert!(name.len() < 256, "Can't make a directory with this name, it's too long."); // Reserve a spot for the new directory debug!("Getting a new directory block..."); let new_directory_location = go_make_new_directory_block()?; // Now that we've made the directory, we need an inode that points to it. // Since this is a brand new directory, this inode will have a creation and modified time of right now let now = InodeTimestamp::now(); let inode: Inode = Inode { flags: InodeFlags::MarkerBit, // No file bit, since this is a directory file: None, directory: Some(InodeDirectory::from_disk_pointer(new_directory_location)), created: now, modified: now, }; // Go put it somewhere. debug!("Adding the inode for the new directory..."); let inode_result = Pool::fast_add_inode(inode)?; // Now we add this newly created directory to the calling directory. let mut flags: DirectoryItemFlags = DirectoryItemFlags::MarkerBit; // We also must mark it as a directory, not a normal file. flags.insert(DirectoryItemFlags::IsDirectory); // Put it all together let final_directory_item = DirectoryItem { flags, name_length: name.len() as u8, name, location: inode_result, }; // Put it into the caller directory! // We dont need to pass in a return disk, since we will return ourselves next if needed. debug!("Adding the new directory to the caller..."); directory.add_item(&final_directory_item)?; // All done! debug!("Done creating directory."); Ok(final_directory_item) } /// Allocates space for and writes a new directory block. /// /// Returns where the new block is. /// /// May swap disks, does not return to original disk. fn go_make_new_directory_block() -> Result { // Ask the pool for a new block // No crc, will overwrite. let new_directory_location = Pool::find_and_allocate_pool_blocks(1, false)?[0]; // Open the new block and write that bastard let new_directory_block: RawBlock = DirectoryBlock::new(new_directory_location).to_block(); CachedBlockIO::update_block(&new_directory_block)?; // All done! Ok(new_directory_location) } // Add an item to a directory fn go_add_item( directory: &mut DirectoryBlock, item: &DirectoryItem, ) -> Result<(), DriveError> { let handle = NotifyTui::start_task(TaskType::CreateDirectoryItem, 2); debug!("Adding new item to directory..."); // Added items must have their flag set. assert!(item.flags.contains(DirectoryItemFlags::MarkerBit), "New directory items must have the marker bit set."); // Added items must have a valid location assert!(!item.location.pointer.no_destination(), "New directory items must have a proper location."); // Persistent vars // We may load in other blocks, so these may change let mut new_block_origin: DiskPointer; let mut current_directory: &mut DirectoryBlock = directory; // If we swap disks, we need to update the item to not be on the local disk anymore. // We clone here so higher up we can keep directory items that are added to directories instead of consuming them on write. let item_to_add: DirectoryItem = item.clone(); // Need to hold this out here or the borrow will be dropped. let mut next_directory: DirectoryBlock; // Now for the loop loop { // Try adding the item to the current block if current_directory.try_add_item(&item_to_add).is_ok() { // Cool! We found a spot! break; } // There was not enough room in that block, we need to find the next one. new_block_origin = go_find_next_or_extend_block(current_directory)?; // Load the new directory let read_block: RawBlock = CachedBlockIO::read_block(new_block_origin)?; next_directory = DirectoryBlock::from_block(&read_block); current_directory = &mut next_directory; // Time to try again! continue; } NotifyTui::complete_task_step(&handle); // Now that the loop has ended, we need to write the block that we just updated. // We assume the block has already been reserved, we are simply updating it. let to_write: RawBlock = current_directory.to_block(); CachedBlockIO::update_block(&to_write)?; NotifyTui::complete_task_step(&handle); NotifyTui::finish_task(handle); debug!("Item added."); // Done! Ok(()) } /// Finds the next section of this directory, or extends it if there is none. /// /// Needs a mutable reference, since the pointer may change. /// /// May swap disks, will return to original disk. fn go_find_next_or_extend_block( directory: &mut DirectoryBlock, ) -> Result { let mut block_to_load: DiskPointer = directory.next_block; // Make sure we actually have somewhere to go. if !directory.next_block.no_destination() { // Already have another block to go to. return Ok(block_to_load); } // Looks like we need a new block // Get the block in question. block_to_load = go_make_new_directory_block()?; // Now we must update the previous block to point to this new one. directory.next_block = block_to_load; // Write back the updated destination let raw_block: RawBlock = directory.to_block(); CachedBlockIO::update_block(&raw_block)?; // All done. Ok(block_to_load) } ================================================ FILE: src/pool/disk/standard_disk/block/io/file/mod.rs ================================================ pub mod write; pub mod read; pub mod movement; #[cfg(test)] mod tests; ================================================ FILE: src/pool/disk/standard_disk/block/io/file/movement.rs ================================================ // We need to go to seek points and such. use log::debug; use crate::{error_types::drive::DriveError, pool::disk::{ generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::{ directory::directory_struct::DirectoryItem, file_extents::file_extents_methods::DATA_BLOCK_OVERHEAD, inode::inode_struct::{ Inode, InodeBlock, InodeFile } } }}; impl InodeFile { /// Find where a seek lands. /// Returns (index, offset), index is the index into the input blocks array, /// offset is the offset within that block, skipping the flag byte already. pub(super) fn byte_finder(byte_offset: u64) -> (usize, u16) { // Assumptions: // We aren't attempting to find a byte offset that is outside of the file. let block_capacity = 512 - DATA_BLOCK_OVERHEAD; // We can divide the incoming offset by the block capacity to figure out which block it's in. // This gives the index into the `blocks` slice directly. let block_index = (byte_offset / block_capacity) as usize; // Now within that block we can find which byte it is by taking the modulo. // But we do need to move forwards one byte into the block to skip the flag. let offset_in_block = (byte_offset % block_capacity) as u16 + 1; // All done! (block_index, offset_in_block) } } impl DirectoryItem { /// Retrieve the inode that refers to this block. pub(crate) fn get_inode(&self) -> Result { debug!("Extracting inode from DirectoryItem..."); // read in that inode block let pointer: DiskPointer = self.location.pointer; debug!("Reading in InodeBlock at (disk {} block {})...", pointer.disk, pointer.block); let raw_block: RawBlock = CachedBlockIO::read_block(pointer)?; let block: InodeBlock = InodeBlock::from_block(&raw_block); // return the inode let inode_good = block.try_read_inode(self.location.offset).expect("Invalid inode offset provided!"); debug!("Inode found."); Ok(inode_good) } } ================================================ FILE: src/pool/disk/standard_disk/block/io/file/read.rs ================================================ // Reading a block is way easier than writing it. // Must use cached IO, does not touch disk directly. use log::{ debug, trace }; use crate::{error_types::drive::DriveError, pool::disk::{ generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::{ directory::directory_struct::{ DirectoryItem, DirectoryItemFlags }, file_extents::{ file_extents_methods::DATA_BLOCK_OVERHEAD, file_extents_struct::{ FileExtent, FileExtentBlock } }, inode::inode_struct::{ InodeBlock, InodeFile } } }, tui::{notify::NotifyTui, tasks::TaskType}}; impl InodeFile { // Local functions /// Extract all of the extents and spit out a list of all of the blocks. pub(super) fn as_pointers(&self) -> Result, DriveError> { go_to_pointers(self) } /// Extract all of the extents. pub(super) fn as_extents(&self) -> Result, DriveError> { let root = self.get_root_block()?; go_to_extents(&root) } /// Goes and gets the FileExtentBlock this refers to. fn get_root_block(&self) -> Result { go_get_root_block(self) } /// Read a file fn read(&self, seek_point: u64, size: u32) -> Result, DriveError> { go_read_file(self, seek_point, size) } } // We dont want to call read/write on the inodes, we should do it up here so we // we can automatically update the information on the file, and the directory if needed. impl DirectoryItem { /// Read a file. /// /// Assumptions: /// - This is an FILE, not a DIRECTORY. /// - The location of this directory item has it's disk set. /// - The inode that the item points at does exist, and is valid. /// /// Reads in a file at a starting offset, and returns `x` bytes after that offset. /// /// Optionally returns to a specified disk. pub fn read_file(&self, seek_point: u64, size: u32) -> Result, DriveError> { // Is this a file? if self.flags.contains(DirectoryItemFlags::IsDirectory) { // Uh, no it isn't why did you give me a dir? panic!("Tried to read a directory as a file!"); } // Extract out the file let location = &self.location; // Get the inode block let pointer: DiskPointer = location.pointer; let raw_block: RawBlock = CachedBlockIO::read_block(pointer)?; let inode_block: InodeBlock = InodeBlock::from_block(&raw_block); // Get the actual file let inode_file = inode_block.try_read_inode(location.offset).expect("Already checked if it was a file."); let file = inode_file.extract_file().expect("File flag means a file inode should exist."); // Now we can read in the file let read_bytes = file.read(seek_point, size,)?; // Now we have the bytes. If we were writing, we would have to flush info about the file to disk, but we don't // need to for a read. We are all done Ok(read_bytes) } } fn go_to_pointers(location: &InodeFile) -> Result, DriveError> { // get extents let extents = location.as_extents()?; // Extract all the blocks. // Pre-allocating this vec isn't really possible, but we at least know that // every extent will contain at least one block. let mut blocks: Vec = Vec::with_capacity(extents.len()); // For each extent for e in extents { // each block that the extent references for n in 0..e.length { blocks.push(DiskPointer { disk: e.start_block.disk, block: e.start_block.block + n as u16 }); } } Ok(blocks) } // Functions fn go_to_extents( block: &FileExtentBlock, ) -> Result, DriveError> { // Totally didn't just lift the directory logic and tweak it, no sir. debug!("Extracting extents for a file..."); // We need to iterate over the entire ExtentBlock chain and get every single item. // We assume we are handed the first ExtentBlock in the chain. // Cannot pre-allocate here, since we have no idea how many extents there will be. let mut extents_found: Vec = Vec::new(); let mut current_dir_block: FileExtentBlock = block.clone(); // Big 'ol loop, we will break when we hit the end of the directory chain. loop { // Add all of the contents of the current directory to the total. let new_items = current_dir_block.get_extents(); extents_found.extend_from_slice(&new_items); // I want to get off Mr. Bone's wild ride if current_dir_block.next_block.no_destination() { // We're done! trace!("Done getting FileExtent(s)."); break; } trace!("Need to continue on the next block."); // Time to load in the next block. let next_block = current_dir_block.next_block; let raw_block: RawBlock = CachedBlockIO::read_block(next_block)?; current_dir_block = FileExtentBlock::from_block(&raw_block); // Onwards! continue; } debug!("Extents retrieved."); Ok(extents_found) } fn go_get_root_block(file: &InodeFile) -> Result { // Make sure this actually goes somewhere assert!(!file.pointer.no_destination(), "Pointer with no destination!"); let raw_block: RawBlock = CachedBlockIO::read_block(file.pointer)?; let block = FileExtentBlock::from_block(&raw_block); Ok(block) } fn go_read_file(file: &InodeFile, seek_point: u64, size: u32) -> Result, DriveError> { let handle = NotifyTui::start_task(TaskType::FileReadBytes, size.into()); // Make sure the file is big enough assert!(file.get_size()>= seek_point + size as u64, "Not enough bytes in this file to satisfy the read!"); // Find the start point let (block_index, mut byte_index) = InodeFile::byte_finder( seek_point); // The byte_finder already skips the flag, so it ends up adding one, we need to subtract that. // This is a bandaid fix. this logic is ugly. // Not gonna refactor it tho, hehe. byte_index -= 1; let blocks = file.as_pointers()?; let mut bytes_remaining: u32 = size; let mut current_block: usize = block_index; // Since we will be writing into this vec, we need to pre-fill it with zeros to allow for indexing. // Doing it like this also avoids needing to grow the vec with additional data. let mut collected_bytes: Vec = vec![0_u8; size as usize]; // We dont need to deal with the disk at all at this level, we will use // the cache for all IO loop { // Are we done reading? if bytes_remaining == 0 { // All done! break } // Get where the next bytes need to go let append_point = (size - bytes_remaining) as usize; // Read into the buffer let bytes_read = read_bytes_from_block(&mut collected_bytes, append_point, blocks[current_block], byte_index, bytes_remaining)?; // After the first read, we are now aligned to the start of blocks byte_index = 0; // Update how many bytes we've read bytes_remaining -= bytes_read as u32; NotifyTui::complete_multiple_task_steps(&handle, bytes_read.into()); // Keep going! current_block += 1; continue; } NotifyTui::finish_task(handle); // All done! Ok(collected_bytes) } /// Read as many bytes as we can from this block. /// /// Buffer must have enough room for our write. MUST pre-allocate it. /// /// buffer_offset is how far into the provided buffer to append the newly read bytes. /// /// Places read bytes into the provided buffer. /// /// Returns number of bytes read. fn read_bytes_from_block(buffer: &mut [u8], buffer_offset: usize, block: DiskPointer, internal_block_offset: u16, bytes_to_read: u32) -> Result { // How much data a block can hold let data_capacity = 512 - DATA_BLOCK_OVERHEAD as usize; let offset = internal_block_offset as usize; // Check for impossible offsets assert!(offset < data_capacity, "Tried to read outside of the capacity of a block."); // Calculate bytes to write based on REMAINING space. let remaining_space = data_capacity - offset; let bytes_to_read = std::cmp::min(bytes_to_read, remaining_space as u32); // We also don't support 0 byte reads // Since that would be a failure mode of the caller, in theory could be // stuck in an infinite loop type shi. // Why panic? It won't if you fix the caller! :D assert_ne!(bytes_to_read, 0, "Tried to read 0 bytes from a block!"); // load the block let block_copy: RawBlock = CachedBlockIO::read_block(block)?; // Read that sucker // Skip the first byte with the flag let start = offset + 1; let end = start + bytes_to_read as usize; let amount_read: usize = end - start; // Put the bytes into the buffer that was passed in. // Create slices for the buffer and the data, zero cost abstraction i think, just makes // code prettier. let destination_slice = &mut buffer[buffer_offset..buffer_offset + amount_read]; let block_data = &block_copy.data[start..end]; destination_slice.copy_from_slice(block_data); // Return the bytes we read. // No way to read more than 512 bytes, so u16 is fine Ok(amount_read as u16) } ================================================ FILE: src/pool/disk/standard_disk/block/io/file/tests.rs ================================================ // Files, direct to thee. // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] use rand::{rngs::ThreadRng, Rng, RngCore}; use test_log::test; use crate::pool::{disk::{standard_disk::block::{directory::directory_struct::DirectoryItem, io::directory::{tests::get_filesystem, types::NamedItem}}}, pool_actions::pool_struct::Pool}; // We want to see logs while testing. /// Can we make a new file? #[test] fn create_blank() { // Make a blank file let _fs = get_filesystem(); let mut root_block = Pool::get_root_directory().unwrap(); let new_item = root_block.new_file("test123.txt".to_string()).unwrap(); let new_file = new_item.get_inode().unwrap().extract_file().unwrap(); assert_eq!(new_file.get_size(), 0); // Brand new files should be empty. } /// Can we make a new small file? #[test] fn write_small_file() { // Make a blank file let fs = get_filesystem(); let mut root_block = Pool::get_root_directory().unwrap(); let new_item = root_block.new_file("test123.txt".to_string()).unwrap(); // Bytes to write let mut random: ThreadRng = rand::rng(); let mut bytes: [u8; 512] = [0_u8; 512]; random.fill_bytes(&mut bytes); // Write to that file // We will write from the start. let seek_point: u64 = 0; let bytes_written = new_item.write_file(&bytes, seek_point).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); // This should fit on one disk. assert_eq!(fs.pool.lock().expect("testing").header.highest_known_disk, 1); } /// Can we make a new small file? #[test] fn write_big_file() { // Make a blank file let fs = get_filesystem(); let mut root_block = Pool::get_root_directory().unwrap(); let new_file = root_block.new_file("test123.txt".to_string()).unwrap(); // lol, how about 4 MB // This is 4x what fluster will normally handle const FOUR_MEG: usize = 4 * 1024 * 1024; let mut random: ThreadRng = rand::rng(); let mut bytes: Vec = Vec::with_capacity(FOUR_MEG); bytes.resize_with(FOUR_MEG, || 0); random.fill_bytes(&mut bytes); // Write to that file // We will write from the start. let seek_point: u64 = 0; let bytes_written = new_file.write_file(&bytes, seek_point).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); // Make sure this actually used multiple disks assert!(fs.pool.lock().expect("testing").header.highest_known_disk > 1); } /// Make a lot of empty random files. #[test] fn make_lots_of_files() { // Make a blank file let _fs = get_filesystem(); for (current_filename_number, _) in (0..1000).enumerate() { let mut root_block = Pool::get_root_directory().unwrap(); let new_name: String = format!("{current_filename_number}.txt"); let _new_file = root_block.new_file(new_name).unwrap(); // we wont write anything. } } /// Make a lot of filled random size files. #[test] #[ignore = "Very slow."] fn make_lots_of_filled_files() { let _fs = get_filesystem(); let mut random: ThreadRng = rand::rng(); for (current_filename_number, _) in (0..1000).enumerate() { let mut root_block = Pool::get_root_directory().unwrap(); let new_name: String = format!("{current_filename_number}.txt"); let new_file = root_block.new_file(new_name).unwrap(); // Random size between 1 byte and 1 MB let file_size: usize = random.random_range(1..1024*1024); let mut bytes: Vec = Vec::with_capacity(file_size); bytes.resize_with(file_size, || 0); random.fill_bytes(&mut bytes); // Write to that file let bytes_written = new_file.write_file(&bytes, 0).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); } } /// Can we properly read and write a test file? #[test] fn write_and_read_small() { // Make a blank file let _fs = get_filesystem(); let mut root_block = Pool::get_root_directory().unwrap(); let new_file = root_block.new_file("test123.txt".to_string()).unwrap(); // Bytes to write let mut random: ThreadRng = rand::rng(); let mut bytes: [u8; 512] = [0_u8; 512]; random.fill_bytes(&mut bytes); // Write to that file // We will write from the start. let seek_point: u64 = 0; let bytes_written = new_file.write_file(&bytes, seek_point).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); // Read back in that file // We will find the file by its file name, to ensure disk access works correctly. let root_block = Pool::get_root_directory().unwrap(); // Go fetch let named: NamedItem = NamedItem::File("test123.txt".to_string()); let read_me = root_block.find_item(&named).unwrap().expect("We just made it"); // Read the contained data let read_data = read_me.read_file(0, 512).unwrap(); // Does it match? assert_eq!(read_data.len(), bytes.len()); assert_eq!(read_data, bytes); } /// Can we properly read and write a test file? #[test] fn write_and_read_large() { // Make a blank file let _fs = get_filesystem(); let mut root_block = Pool::get_root_directory().unwrap(); let new_file = root_block.new_file("test123.txt".to_string()).unwrap(); // Bytes to write const FOUR_MEG: usize = 4 * 1024 * 1024; let mut random: ThreadRng = rand::rng(); let mut bytes: Vec = Vec::with_capacity(FOUR_MEG); bytes.resize_with(FOUR_MEG, || 0); random.fill_bytes(&mut bytes); // Write to that file // We will write from the start. let seek_point: u64 = 0; let bytes_written = new_file.write_file(&bytes, seek_point).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); // Read back in that file // We will find the file by its file name, to ensure disk access works correctly. let root_block = Pool::get_root_directory().unwrap(); // Go fetch let named: NamedItem = NamedItem::File("test123.txt".to_string()); let read_me = root_block.find_item(&named).unwrap().expect("We just made it"); // Read the contained data let read_data = read_me.read_file(0, FOUR_MEG.try_into().unwrap()).unwrap(); // Does it match? assert_eq!(read_data.len(), bytes.len()); assert_eq!(read_data, bytes); } /// Read and write a lot of random files #[test] #[ignore = "Very slow."] fn read_and_write_random_files() { let _ = get_filesystem(); let mut random: ThreadRng = rand::rng(); let mut random_files: Vec> = Vec::new(); const TEST_LENGTH: usize = 1000; const MAX_FILE_SIZE: usize = 1024 * 1024; // Currently one meg for (current_filename_number, _) in (0..TEST_LENGTH).enumerate() { let mut root_block = Pool::get_root_directory().unwrap(); let new_name: String = format!("{current_filename_number}.txt"); let new_file = root_block.new_file(new_name).unwrap(); // Random size between 1 byte and 1 MB let file_size: usize = random.random_range(1..MAX_FILE_SIZE); let mut bytes: Vec = Vec::with_capacity(file_size); bytes.resize_with(file_size, || 0); random.fill_bytes(&mut bytes); // Keep track of this file random_files.push(bytes.clone()); // Write to that file let bytes_written = new_file.write_file(&bytes, 0).unwrap(); // Make sure we actually wrote all the bytes assert_eq!(bytes_written, bytes.len() as u32); } // Now we need to read all of the files back out let root_block = Pool::get_root_directory().unwrap(); for (current_file, _) in (0..TEST_LENGTH).enumerate() { let named: NamedItem = NamedItem::File(format!("{current_file}.txt")); let found: DirectoryItem = root_block.find_item(&named).unwrap().unwrap(); // read it let file_size: u64 = found.get_inode().unwrap().extract_file().unwrap().get_size(); let read = found.read_file(0, file_size.try_into().unwrap()).unwrap(); // Compare check_byte_vec_equality(&read, &random_files[current_file]); } } /// Test helper for tracking down file corruption. /// /// Returns nothing, panics if bytes are not the same. fn check_byte_vec_equality(a: &[u8], b: &[u8]) { // Make sure they are the same length assert_eq!(a.len(), b.len()); // Check the bytes // The following line is the current Guinness World Records // holder for "Worlds most unreadable iterator chain" for (index, (byte_a, byte_b)) in a.iter().zip(b.iter()).enumerate() { assert_eq!(byte_a, byte_b, "Byte mismatch at index `{index}`! a: `{byte_a}`, b: `{byte_b}`"); // We cant use enumerate here (i think?) so manual index tracking. } } ================================================ FILE: src/pool/disk/standard_disk/block/io/file/write.rs ================================================ // Writing files. // We will take in InodeFile(s) instead of Extent related types, since we need info about how big files are so they are easier to extend. // Creating files is handles on the directory side, since new files just have a name and location. use std::{ cmp::max, ops::{ Div, Rem } }; use log::{debug, warn}; use log::error; use crate::{error_types::drive::DriveError, pool::{ disk::{ generic::{ block::{ block_structs::RawBlock, crc::add_crc_to_block }, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO }, standard_disk::block::{ directory::directory_struct::{ DirectoryBlock, DirectoryItem, DirectoryItemFlags }, file_extents::{ file_extents_methods::DATA_BLOCK_OVERHEAD, file_extents_struct::{ FileExtent, FileExtentBlock } }, inode::inode_struct::{ Inode, InodeBlock, InodeFile, InodeFlags, InodeTimestamp }, io::directory::types::NamedItem } }, pool_actions::pool_struct::Pool }, tui::{notify::NotifyTui, tasks::TaskType}}; impl InodeFile { /// Update the contents of a file starting at the provided seek point. /// Will automatically grow file if needed. /// /// !! This does not flush information to disk! You must /// write back the new size of the file to disk! !! /// /// Optionally returns to a provided disk when done. /// /// Returns number of bytes written, but also updates the incoming file's size automatically fn write(&mut self, bytes: &[u8], seek_point: u64) -> Result { go_write(self, bytes, seek_point) } } impl DirectoryBlock { /// Create a new empty file in this directory with a name. /// /// Adds the file to the directory, flushes it to disk. /// /// Requires a mutable borrow, since this may update the block. /// /// Returns the created file's directory item. Will contain disk info. /// /// Should include extension iirc? pub fn new_file(&mut self, name: String) -> Result { go_make_new_file(self, name) } /// Deletes an file by deallocating every block the file used to take up, and removing it /// from the directory. /// /// If you are looking to truncate a file, you need to call truncate() on the actual directory item. /// /// Returns `None` if the file did not exist. /// /// Panics if fed a directory. Use remove_directory() ! pub fn delete_file(&mut self, file: NamedItem) -> Result, DriveError> { // We only handle files here if !file.is_file() { // Why panic!("Cannot delete_file a non-file!"); } // Extract the item let extracted_item: DirectoryItem; if let Some(exists) = self.find_and_extract_item(&file)? { // Item was there extracted_item = exists } else { // Tried to delete a file that does not exist in this directory. return Ok(None) }; // Delete it. truncate_or_delete_file(&extracted_item, true, None)?; // Since the extraction function already handles pulling out the item from the directory blocks, we are done. Ok(Some(())) } } // We dont want to call read/write on the inodes, we should do it up here so we // we can automatically update the information on the file, and the directory if needed. impl DirectoryItem { /// Write data to a file. /// /// Assumptions: /// - This is an FILE, not a DIRECTORY. /// - The location of this directory item has it's disk set. /// - The inode that the item points at does exist, and is valid. /// /// Write to a file at a starting offset, and sets `x` bytes after that offset. /// /// Does not consume the directory item, since the data lower down was updated, not the /// directory item itself. /// /// Returns how many bytes were written. /// /// Optionally returns to a specified disk. pub fn write_file(&self, bytes: &[u8], seek_point: u64) -> Result { // We only handle files here if self.flags.contains(DirectoryItemFlags::IsDirectory) { // Why panic!("Cannot delete_file a non-file!"); } // Extract out the file let location = &self.location; // Get the inode block let the_pointer_in_question: DiskPointer = location.pointer; let read: RawBlock = CachedBlockIO::read_block(the_pointer_in_question)?; let mut inode_block: InodeBlock = InodeBlock::from_block(&read); // Get the actual file. // The inode MUST exist. // A smarter filesystem would return the fact that the inode is missing and either try to rebuild it or // have the caller discard the item. But er, I dont have time. let inode_with_file = if let Ok(inode) = inode_block.try_read_inode(location.offset) { inode } else { // No inode... // Maybe some day, propagate that... // If this fails in the video, I'll write it lmao. panic!("No inode exists for this DirectoryItem's file. We cannot read it."); }; // We already checked that this is a file. let mut file = if let Some(the_file) = inode_with_file.extract_file() { the_file } else { // ??? panic!("File is a file, but not a file. Nice."); }; // Write to the file // This automatically updates the underlying file with the new size. let num_bytes_written = file.write(bytes, seek_point)?; // Now that the bytes are written, the size of the file may have changed, so we need to flush this new information to disk. // Reconstruct the inode let mut updated_inode: Inode = inode_with_file; // Replace the inner file with the new updated one updated_inode.file = Some(file); // We also need to update the modify timestamp. updated_inode.modified = InodeTimestamp::now(); // Now update the inode in the block. This also flushes to disk for us. inode_block.update_inode(location.offset, updated_inode)?; // All done. Return the number of bytes we wrote. Ok(num_bytes_written) } /// Truncates a file to a specified byte length. /// /// No action needs to be taken after this method. /// /// Panics if fed a directory. pub fn truncate(&self, new_size: u64) -> Result<(), DriveError> { truncate_or_delete_file(self, false, Some(new_size)) } } fn go_write(inode_file: &mut InodeFile, bytes: &[u8], seek_point: u64) -> Result { let handle = NotifyTui::start_task(TaskType::FileWriteBytes, bytes.len() as u64); // Decompose the file into its pointers // No return location, we don't care where this puts us. let mut blocks = inode_file.as_pointers()?; // get the seek point let (block_index, mut byte_index) = InodeFile::byte_finder( seek_point); // The byte_finder already skips the flag, so it ends up adding one, we need to subtract that. byte_index -= 1; // Make sure we actually have a block at that offset. We cannot start writing from unallocated space. if block_index > blocks.len() { // Being told to write to a point we do not have. panic!("Attempted to write to unallocated space!"); } // Now we can calculate where the final byte of this write will end up. // Minus 1, since we are writing to the byte we start the seek from // IE: if we write 1 byte from out offset, we don't actually move forwards into the next byte. let (mut final_block_index, _) = InodeFile::byte_finder(seek_point + bytes.len() as u64 - 1); // If final block index is 0, we still need at least 1 block, since block 0 is the first block. // Thus we must always add 1. final_block_index += 1; // Special case, if we have 0 blocks, everything up till now works fine, but we always need at least 1 block to write into. // To make my life easier, we will do it here instead of relying on the caller. // Now, if our final block index is larger than how many blocks we currently have, that means we need to pre-allocate the new room. if final_block_index > blocks.len() || blocks.is_empty() { // We need to get more room. // Special case will always allocate at least one block. let needed_blocks = max(final_block_index - blocks.len(), 1); // This should always be <= the max write size set in // filesystem_methods.rs. Which should ALWAYS be quite far away // from u16::MAX, but we check anyways. // We hard cap it to u16 block though, just in case. if needed_blocks > u16::MAX.into() { // Crazy. panic!("Tried to write 2^16 blocks worth of data in one go! Not allowed!"); } // Add that many more blocks to this file. // Since we know its already less than u16::MAX this cast is fine. let new_pointers = expand_file(*inode_file, needed_blocks as u16)?; // The new pointers are already in order for us, and we will add them onto the end of the // pointers we grabbed earlier from the file. blocks.extend(new_pointers.iter()); } // Now we know we have enough space for this write, let's get started. let mut bytes_written: usize = 0; // Since the write can start un-aligned, we need to use an offset until its aligned again. let mut byte_write_index: u16 = byte_index; // Now we will loop through the blocks starting at the current index for block in &blocks[block_index..] { // are we out of bytes to write? if bytes_written == bytes.len() { // All done! break } // Update the block let written = update_block(*block, &bytes[bytes_written..], byte_write_index)?; // After the first write, the offset should be fixed now, since we've either written all of our bytes, in // which case we would be done, or we ran out of room in the block, thus the next block's offset would be 0. byte_write_index = 0; // Update how many bytes we've written bytes_written += written; NotifyTui::complete_multiple_task_steps(&handle, written as u64); // Keep going! continue; } // Done writing bytes! // Update the file size, only if we wrote past the end. let write_end = seek_point + bytes_written as u64; if write_end > inode_file.get_size() { inode_file.set_size(write_end); } NotifyTui::finish_task(handle); // Return how many bytes we wrote! Ok(bytes_written as u32) } /// Updates a block with new content, overwriting previous content at an offset. /// /// You can feed in as many bytes as you like, but it will only write as many as it can. /// /// Offset is the first data byte, not the first byte of the block! /// /// Returns number of bytes written. fn update_block(block: DiskPointer, bytes: &[u8], offset: u16) -> Result { // How much data a block can hold let data_capacity = 512 - DATA_BLOCK_OVERHEAD as usize; let offset = offset as usize; // Check for impossible offsets if offset >= data_capacity { panic!("Tried to write outside of the capacity of a block."); } // Calculate bytes to write based on REMAINING space. let remaining_space = data_capacity - offset; let bytes_to_write = std::cmp::min(bytes.len(), remaining_space); // We also don't support 0 byte writes. // Since that would be a failure mode of the caller, in theory could be // stuck in an infinite loop type shi. // Why exit? It won't if you fix the caller! :D if bytes_to_write == 0 { panic!("Tried to write 0 bytes to a block!"); } // Now, if we are about to completely fill a block (ie every byte in the block will change) // we dont need to actually "update" the block, we can just replace it entirely. let mut block_copy = if bytes_to_write == data_capacity && offset == 0 { // Full block replacement, make fake block. // The flag byte never ended up being used, so we dont have to worry about it. RawBlock { block_origin: block, data: [0u8; 512], // Start with a blank slate } } else { // Partial update, still need to read in the old block. CachedBlockIO::read_block(block)? }; // Modify that sucker // Skip the first byte with the flag let start = offset + 1; let end = start + bytes_to_write; block_copy.data[start..end].copy_from_slice(&bytes[..bytes_to_write]); // Update the crc add_crc_to_block(&mut block_copy.data); // Write that sucker CachedBlockIO::update_block(&block_copy)?; // Return the number of bytes we wrote. Ok(bytes_to_write) } /// Expands a file by adding `x` new blocks to the extents. Returns disk pointers for the new extents. /// Updates underlying ExtentBlock(s) for this file. /// /// May swap disks, does not return to any start disk. fn expand_file(inode_file: InodeFile, blocks: u16) -> Result, DriveError> { debug!("Expanding a file by {blocks} blocks..."); // Go grabby some new blocks. // These will be already reserved for us. // We also need to write the CRC for later. let reserved_blocks = Pool::find_and_allocate_pool_blocks(blocks, true)?; // Make some extents from that let new_extents = pointers_into_extents(&reserved_blocks); // Add the extents. expanding_add_extents(inode_file, &new_extents)?; // Return the pointers to those new extents. Ok(reserved_blocks) } /// Expands an ExtentBlockBlock. /// Always extends by one block. /// /// Will swap disks to the location of the new block. Will not return to the disk the caller started on. /// /// Sets the new destination in incoming block. fn expand_extent_block(block: &mut FileExtentBlock) -> Result<(), DriveError> { // Get a new block from the pool. // No need for crc, we will immediately write over it. let new_block_location = Pool::find_and_allocate_pool_blocks(1, false)?[0]; // Put the a block there let new_block: RawBlock = FileExtentBlock::new(new_block_location).to_block(); // Write, since we looked for a free block, didn't reserve it yet. CachedBlockIO::update_block(&new_block)?; // Now update the block we came in here with block.next_block = new_block_location; // Updated! All done. Ok(()) } /// Will automatically deduce runs of blocks from a slice of disk pointers, then add those extents to the block, /// expanding the block if needed. /// /// We assume the incoming pointers are already sorted by disk, block. (1, 1), (1, 2), (2, 1) etc. /// /// Does not check if blocks are already allocated, caller _MUST_ provide marked blocks. fn expanding_add_extents(file: InodeFile, extents: &[FileExtent]) -> Result<(), DriveError> { // We will reverse the extents vec so we can pop them off the back // for easier adding, avoiding an index. let mut new_extents: Vec = extents.to_vec(); new_extents.reverse(); // Go get the extent block to add to. // We need the final one in the chain. let mut current_extent_block: FileExtentBlock; // Read in the initial block let raw_read: RawBlock = CachedBlockIO::read_block(file.pointer)?; current_extent_block = FileExtentBlock::from_block(&raw_read); loop { // Is this the final block? if !current_extent_block.next_block.no_destination() { // No it isn't. We need to load the next block. // Get the block. let reader_mc_deeder: RawBlock = CachedBlockIO::read_block(current_extent_block.next_block)?; current_extent_block = FileExtentBlock::from_block(&reader_mc_deeder); // Try again. continue; } // This is the final block. // Try adding extents while let Some(last) = new_extents.last() { // Try adding a new extent. let added_result = current_extent_block.add_extent(*last); // if that worked, that means we added the extent successfully. if added_result.is_ok() { // Good! Keep going // Pop off the extent since we're done with it. let _ = new_extents.pop(); continue; } // Otherwise we either ran out of room, or this isn't the last block in the chain. // We already checked the latter, so... break } // There are two cases that cause us to break out of that loop. if new_extents.is_empty() { // We ran out of file extents to add. we are done. // Flush it flush_to_disk(¤t_extent_block)?; // bye break } // We must've ran out of room. // Expand the block please. expand_extent_block(&mut current_extent_block)?; // Now we must write that extended block to disk. flush_to_disk(¤t_extent_block)?; // The block has a new destination now, and has been flushed to disk. There is nothing // else left for us to do before just looping again, since the next loop // will see the new destination. continue; } // All done adding extents! We've already flushed the blocks as well. // There is nothing left to do. Ok(()) } /// Automatically groups incoming pointers into a new vec of file extents. /// assumes all of the incoming pointers are already sorted. fn pointers_into_extents(pointers: &[DiskPointer]) -> Vec { // I feel like there is 100% a better way to do this, but i dont know it. so too bad! // Cant pre-allocate room in the vec, since we dont know how many extents we'll be creating, // and estimating it is hard. let mut new_extents: Vec = Vec::new(); // Loop over the pointers and create extents. for pointer in pointers { // Check if we need to make a new extent. // We start at 0 since we increment at the end of the loop. let new: FileExtent = FileExtent::new(*pointer, 0); // We need a new one if: // - There are no extents // - The disk number is different // - The length is maxed out // - The next block is not contiguous. (ie last block was 1, new block != 2) // yes this is ugly, at least it doesnt have to check for local disks anymore if let Some(extent) = new_extents.last() { if extent.start_block.disk != pointer.disk || // Is the disk number different? extent.length == u8::MAX || // Is this extent out of room? extent.start_block.block + extent.length as u16 != pointer.block // Non contiguous? { // Need a new one. new_extents.push(new); } // All checks pass, fall out. } else { // There isn't any extents yet, make the first one new_extents.push(new); } // This pointer extends the previous (or new) extent block, add one to the length. match new_extents.last_mut() { Some(last) => { last.length += 1; }, None => { // I guess we never got any pointers. // Do nothing. }, } } // All done! new_extents } /// Create a new file. fn go_make_new_file(directory_block: &mut DirectoryBlock, name: String) -> Result { // Directory blocks already have a method to add a new item to them, so we just need // to create that item to add. // New files must have a filename that is <= u8::MAX. // Caller is in charge of checking this before giving it to us. if name.len() > u8::MAX.into() { panic!("File name was too long!"); } // Timestamp for file creation let right_now: InodeTimestamp = InodeTimestamp::now(); // No need for CRC, we will be writing over it. let in_progress = Pool::find_and_allocate_pool_blocks(1, false)?; let reserved_block: DiskPointer = in_progress[0]; // Now that we have the new block we need a FileExtentBlock to write into it. let new_block: FileExtentBlock = FileExtentBlock::new(reserved_block); // No need to set the marker bit since this is a file ofc. // Now let's write that new block let raw: RawBlock = new_block.to_block(); // Block is not marked as reserved, so this is a write. CachedBlockIO::update_block(&raw)?; // Construct the file that we'll be returning. let finished_new_file: InodeFile = InodeFile::new(reserved_block); // Now that the block has been written, put that sucker into the directory // We do need an inode location tho, so get one let new_inode: Inode = Inode { flags: { // We need to set the marker bit and the inode type (file) let mut inner = InodeFlags::MarkerBit; inner.insert(InodeFlags::FileType); inner }, file: Some(finished_new_file), directory: None, created: right_now, modified: right_now, }; let new_inode_location = Pool::fast_add_inode(new_inode)?; // Wrap it all up in a little bow to put into the directory // Now we need to set up the flags for the new directory item // Flag let flags: DirectoryItemFlags = DirectoryItemFlags::MarkerBit; // Thats it. Lol. // Construct the new file let new_file: DirectoryItem = DirectoryItem { flags, name_length: name.len() as u8, name, location: new_inode_location, }; directory_block.add_item(&new_file)?; // If we're here, that worked. We are all done adding the item to the directory. // All done. // Dont need to swap disks, already did that on the item add. Ok(new_file) } // One hell of a function. /// Will only truncate if delete is false. fn truncate_or_delete_file(item: &DirectoryItem, delete: bool, new_size: Option) -> Result<(), DriveError> { // Is this a file? if item.flags.contains(DirectoryItemFlags::IsDirectory) { // Uh, no it isn't why did you give me a dir? panic!("Tried to truncate or delete a directory as if it was a file!"); } // Load the size of the directory item let file_size: u64 = item.get_size()?; // If we aren't deleting, and the size is the same as the current size, we can skip truncation. if let Some(the_new_size) = new_size { // Make sure delete flag is not set. // In theory, None means delete is set. if !delete && the_new_size == file_size { // Skip return Ok(()); } } // Extract out the file let file_inode_location = &item.location; // Get the inode block let the_pointer_in_question: DiskPointer = file_inode_location.pointer; let read: RawBlock = CachedBlockIO::read_block(the_pointer_in_question)?; let mut inode_block: InodeBlock = InodeBlock::from_block(&read); // Get the actual file let mut inode_with_file: Inode = if let Ok(inode) = inode_block.try_read_inode(file_inode_location.offset) { inode } else { // The inode for this file does not exist. panic!("Cannot truncate or delete files that do not have an inode!"); }; // We already checked that this is a file. let mut file: InodeFile = if let Some(the_file) = inode_with_file.extract_file() { the_file } else { // ? panic!("Flag for file set, but no file."); }; // If the truncation is just growing the file, we can do this directly on the file itself without messing with the extents. // Truncation can also grow files, check if the truncation is larger than the current size. // Growing cannot happen at the same time as deletion. if let Some(extracted_new_size) = new_size && file_size < extracted_new_size && !delete { // We are just growing. // Growing is easy, we just write zeros to make it the new size. // The difference let grow_size: usize = (extracted_new_size - file_size) as usize; // We need to do this in a loop, since would be consuming as much ram as the write is big, which isn't great. // So we will do it in 1MB chunks, but this may change in the future if its too slow. const CHUNK_SIZE: usize = 1024*1024; let zero_chunk: Vec = vec![0; CHUNK_SIZE]; // Write to the end of the file with the zeros. let mut seek_point = file_size; for _ in 0..grow_size.div(CHUNK_SIZE) { // Yeah... Keep eating... let _ = item.write_file(&zero_chunk, seek_point)?; seek_point += CHUNK_SIZE as u64; } // Final write if there are any remaining bytes. let remainder = grow_size.rem(CHUNK_SIZE); if remainder != 0 { let final_zeros: Vec = vec![0; remainder]; let _ = item.write_file(&final_zeros, seek_point)?; } // Make sure the new size is correct let gotten_size = item.get_size()?; if extracted_new_size != gotten_size { // Truncation did not work properly. error!("Truncated to the wrong size! Expected {extracted_new_size} got {gotten_size} !") }; // All done. return Ok(()); } // If we are here, we must be shrinking or deleting. // If we are deleting, we can skip the more complicated extent logic. if delete { // We are deleting all of the blocks, so just get all of them. let mut used_blocks = file.as_pointers()?; // We also need to free the extent blocks themselves, not just where they point. let mut extent_block_pointer = file.pointer; while !extent_block_pointer.no_destination() { used_blocks.push(extent_block_pointer); // This work has already been done on `to_pointers`, maybe there should be another // method that returns the pointers, and the pointers to the extent blocks at the same time. // Not gonna write that tho, this is fine. let read = CachedBlockIO::read_block(extent_block_pointer)?; let extent_block: FileExtentBlock = FileExtentBlock::from_block(&read); extent_block_pointer = extent_block.next_block; } // We dont have to worry about updating the underlying block, since the deletion call // will discard the item automagically. // Sort the blocks to reduce swap used_blocks.sort_unstable_by_key(|block| (block.disk, block.block)); // Chunk by disk. let chunked = used_blocks.chunk_by(|a, b| a.disk == b.disk); // Delete all the blocks by freeing all of them. for chunk in chunked { let _ = Pool::free_pool_block_from_disk(chunk)?; } // All done! return Ok(()); } // Since we didn't delete, we must be shrinking, since we already checked for growing // This should be guarded. let new_size = new_size.expect("Cannot truncate a file without a size to truncate to."); // To truncate, several things need to happen: // - We need to update the data that is contained within the final data block to write in zeros // - past the new ending size. // - We need to remove all of the data blocks that are no longer used. // - We need to remove all of the file extent blocks that are no longer used. // - We need to update the pointer on the new final extent block (if needed) // - - This will never point at a new block, since its the new final, and we cannot // - - remove extents from the middle of a file. // - We need to remove all extents past this one in the new final file extent block. // - Update the file size. // I considered extracting out all of the extents to consolidate them again, but there would be // no space savings from doing that, since extents are always added at the end of the chain, and must be in order. // Thus, if an extent could have fit in the previous block, it would have already been there. You cannot truncate in // the middle of the file. // Thus, removing extents does not create gaps, and is already as efficient as possible (in Fluster's implementation at least, lol) // Steps: // Deduce what data block is the new final block. // - The FileExtentBlock that contained that block is the new final FileExtentBlock in the chain. // - The Extent that the data block was contained within is now the final Extent within the FileExtentBlock // Update the new final data block. // - Write zeros to the end of the block after the new file end point. // - Do not write it yet, it must be written after the ExtentBlock has been updated, otherwise we // - would be changing file data without changing the file ending, corrupting the underlying file. // Remove all extents past this one in the containing extent block. // - Just pop off all ones after this one, and hold onto the pointers for them // - so we can free those blocks later. // Update the pointer on the new final ExtentBlock // - Hold onto where it used to go, we need it for cleanup later // - Point to nowhere. // All at once: // - Flush the updated final extent to disk, then the updated data block. // - - Make sure to set the extents to be local if needed, since the disk gets set during the read process. // - - After this point, the file has been properly truncated from the filesystem perspective. // - - Even if the rest of the deletion fails after this point, we at least wont be pointing at the data blocks // - - that were supposed to be freed. We'll have leaked them though, which stinks, but whatever. // - Update the file size // - Update the file modify time // Collect all of the DiskPointers to the remaining ExtentBlocks. // - Cant immediately remove them, we still need the DiskPointers to the data blocks // Collect all of the DiskPointers to the data blocks within the extents in the remaining extent blocks. // - Loop over the collected disk pointers in the previous step, open the FileExtentBlock, extract the // - DiskPointers from all of the contained Extents // Free all of the blocks we've collected // Is this a lot of documentation? yes. // Is this the third time i've rewritten this function from near-scratch today? Also yes. // == Deduce what data block is the new final block. == // We need to find the block offset. let (new_final_block_index, new_final_block_byte_index) = InodeFile::byte_finder(new_size); // Now we can loop through the FileExtentBlocks, tracking how many data blocks we've seen so far // Get the first extent block let read: RawBlock = CachedBlockIO::read_block(file.pointer)?; let first_extent_block = FileExtentBlock::from_block(&read); // Go find the block, also keep track of what extent caused us to be full, since // that'll be our new final extent. let mut new_final_extent_block: FileExtentBlock = first_extent_block; // Dummy values that will be overwritten. let mut pointer_to_new_final_data_block: DiskPointer = DiskPointer::new_final_pointer(); let mut offset_in_final_extent: usize = 0; let mut final_extent_index: usize = 0; let mut blocks_seen: usize = 0; let mut done: bool = false; // At this point, we know the truncation HAS to be less than the current size of the file, thus // we do not need to check if we run out of blocks, since they must be there. // If they aren't there, its a more fundamental issue, not our problem. loop { // Get the extents from the ExtentBlock let extents = new_final_extent_block.get_extents(); // Now search through those extents, incrementing how many // blocks we've seen and keeping track of what extent this is for (index, extent) in extents.iter().enumerate() { // How many blocks are in here? let pointers: Vec = extent.get_pointers(); // Add all of those pointers to the count blocks_seen += pointers.len(); // Have we seen enough blocks? if blocks_seen >= new_final_block_index { // The last extent we opened is the final one! // We need to deduce which block it was, since we want to know the index of it. // Sure, we could have looped over the pointers instead of adding the length and just captured the disk pointer // by itself, but we still need to know the index as well, so we can remove the items after it later. // "erm what about iter().enumerate()" sybau ts pmo... // How many blocks we found - the number of blocks we wanted = how many extra blocks we read. // therefore, pointers.len() - extra = index into pointers where the block is // The number of blocks a extent can hold is at most 256, which fits into an i16 when negative. let offset = pointers.len() - (blocks_seen - new_final_block_index); offset_in_final_extent = offset; // Now we can get the pointer to the final data block pointer_to_new_final_data_block = pointers[offset]; final_extent_index = index; done = true; break } } // Done? if done { break } // Need to keep going, get the next block. // Dont need to check if this is a final pointer, since we would crash if it was, and it shouldn't be. let read: RawBlock = CachedBlockIO::read_block(new_final_extent_block.next_block)?; let next = FileExtentBlock::from_block(&read); new_final_extent_block = next; } // == Update the new final data block. == // == - Write zeros to the end of the block after the new file end point. == // == - Do not write it yet, it must be written after the ExtentBlock has been updated, otherwise we == // == - would be changing file data without changing the file ending, corrupting the underlying file. == // easy // Find the index into the block where everything past it will be blanked out... // jk, we already know that hehe, its in new_final_block_byte_index // Now load in the old block so we can update it let mut updated_final_data_block: RawBlock = CachedBlockIO::read_block(pointer_to_new_final_data_block)?; // Now blank it out. // Currently, the last 4 bytes of the block are the checksum. but since we're going to be updating the block anyways, we can write // over it, since we'll have to re-checksum it anyways. updated_final_data_block.data[new_final_block_byte_index as usize..].fill(0_u8); // Put the checksum back on add_crc_to_block(&mut updated_final_data_block.data); // Dont write the block yet, we'll hold onto it until _after_ we do the FileExtentBlock update. // == Remove all extents past this one in the containing extent block. == // == - Just pop off all ones after this one, and hold onto the pointers for them == // == - so we can free those blocks later. == // Start blocks that we need to free. // Cannot pre-allocate, since there's no way to know how many blocks we'll be freeing // at this point. Guesstimations could be made, but oh well. let mut blocks_to_free: Vec = Vec::new(); // We also need to grab the extra disk pointers from the extent that holds the new // final data block, if there are any. // Update the extents for the final block. let mut updated_extents = new_final_extent_block.get_extents(); for (index, extent) in updated_extents.iter_mut().enumerate() { // skip if this is before the final extent if index < final_extent_index { // skip continue; } // is this the final extent if index == final_extent_index { // Now we need to remove the extra disk pointers if there are any. let pointers = extent.get_pointers(); // Split the vec to remove anything after the final data block pointer. // Splitting keeps `start..split`, but we need `start..=split` so we will need to // increment the index. let split_point: usize = offset_in_final_extent + 1; // Splitting will panic if this is past the end of the array. Which would be the case // if we found exactly as many blocks as we needed in the final extent. if split_point > pointers.len() { // We dont need to update this extent. continue; } // Do the splits, keeping the second section so we can get the pointers from it let to_free = pointers.split_at(split_point).1; // Now to update the extent, we just need to remove all items past the split point. // since extents are encoded as a start + length, we can just subtract the number of extra items. extent.length -= to_free.len() as u8; // Now add the extra pointers to the trash pile blocks_to_free.extend(to_free); // All done continue; } // This is after the last extent we care about, trash everything. let pointers = extent.get_pointers(); blocks_to_free.extend(pointers); } // Now that we've collected all the extents we care about, and trashed everything else, we can drop any extra // extents from the block if they exist. // truncate is `..end` not `..=end` so we add one. let truncate_point: usize = final_extent_index + 1; updated_extents.truncate(truncate_point); // Now we only have the extents we care about in `updated_extents`. // == Update the pointer on the new final ExtentBlock == // == - Hold onto where it used to go, we need it for cleanup later == // == - Point to nowhere. == // We have to point to nowhere before we can put in the new extents. // We will also hold onto it let unreferenced_extent_block_chain_start: DiskPointer = new_final_extent_block.next_block; new_final_extent_block.next_block = DiskPointer::new_final_pointer(); // == All at once: == // == - Flush the updated final extent to disk, then the updated data block. == // == - - Make sure to set the extents to be local if needed, since the disk gets set during the read process. == // == - - After this point, the file has been properly truncated from the filesystem perspective. == // == - - Even if the rest of the deletion fails after this point, we at least wont be pointing at the data blocks == // == - - that were supposed to be freed. We'll have leaked them though, which stinks, but whatever. == // == - Update the file size == // == - Update the file modify time == // Add the extents // This does not flush to disk. new_final_extent_block.force_replace_all_extents(updated_extents); // Now for the scary part. // This write must complete, followed by the update to the data block, otherwise data // will corrupt. let finished_extent_block: RawBlock = new_final_extent_block.to_block(); debug!("Writing updated file extent block after size decrease..."); debug!("If this fails, data corruption WILL occur.."); // Oh boy. // We will do all 3 steps at once, even if the two writes fail, as long as the file size change works, the file // will _possibly_ be in a usable state, since bounds checks are done to make sure we dont read past the end. // ...Until something tries to extend the file, new blocks will pointlessly be added, and writing may skip over blocks, // resulting in the next read containing old data from pre-truncation. let extent_block_result = CachedBlockIO::update_block(&finished_extent_block); let data_block_result = CachedBlockIO::update_block(&updated_final_data_block); // Update the file file.set_size(new_size); inode_with_file.file = Some(file); // might as well set the time here too inode_with_file.modified = InodeTimestamp::now(); // Write it back to the block it came from let inode_update_result = inode_block.update_inode(file_inode_location.offset, inode_with_file); // Now, did that all work? let all_worked: bool = extent_block_result.is_ok() && data_block_result.is_ok() && inode_update_result.is_ok(); let all_failed: bool = extent_block_result.is_err() && data_block_result.is_err() && inode_update_result.is_err(); if !all_worked && !all_failed { // at least one fail, but not all of them. // DAMNIT! // "why are you printing so much out here" // Well if the filesystem fails during my factorio run, this would // add extra drama hahaha... god i hope it doesn't fail... error!("TRUNCATION FAILURE!"); error!("Listing what failed:"); error!("==-==-==-==-==-==-==-=="); if extent_block_result.is_err() { error!("- Extent block write."); } if data_block_result.is_err() { error!("- Data block write."); } if inode_update_result.is_err() { error!("- Inode update."); } error!("==-==-==-==-==-==-==-=="); error!("We have to keep going. But this file may now be VERY unstable."); error!("Any further calls on this file will almost certainly do unexpected things."); // We have now leaked all of the blocks that we were intending to free. error!("At least `{}` blocks have now been leaked.", blocks_to_free.len()); error!("That does not include:"); error!("FileExtent blocks past the new final extent block in the extent chain."); error!("Data storing blocks that those FileExtent blocks pointed to."); // We can at least estimate it. // yes i know that %512 does not account for the flags and such in the data blocks, // but we arent counting the extent block overhead either so let leak_estimate: usize = (file_size - new_size).div_ceil(512) as usize; error!("Rough estimate: `{leak_estimate}` additional blocks leaked."); error!("Godspeed."); // If we are testing, this should panic as well. if cfg!(test) { panic!("Truncation fail."); } // No error, since we need to just ignore the error if we dont wanna completely give up return Ok(()); // Too late to turn back, unless all 3 failed? but what are the odds? } else if all_failed { // Woah. // All three operations failed, which, unusually, is a good thing in this case. warn!("TRUNCATION FAILURE!"); warn!("All of the operations failed, which means nothing was changed."); warn!("Scary, but we are actually fine in this case."); warn!("We can continue like nothing happened, because nothing did."); // This should also fail tests. if cfg!(test) { panic!("Truncation fail: No update."); } // If the disk is busy, we retry, so... return Err(DriveError::Retry); } else { // Everything worked! debug!("Content update finished successfully. Phew.") } // Cool! Now we can go free blocks we don't need anymore. // Failure after this point will leak blocks. so we keep it locked up to report a leak. debug!("Running truncate cleanup..."); let cleanup_result = truncate_cleanup(blocks_to_free, unreferenced_extent_block_chain_start); // Did that all sail smoothly match cleanup_result { Ok(free_count) => { // All good! All garbage has been freed! debug!("Truncate cleanup finished successfully."); debug!("Truncation freed {free_count} blocks."); Ok(()) }, Err(err) => { // Oh well. error!("Truncate cleanup did not finish!"); error!("We have leaked blocks!"); error!("Unknown how many leaked. But we can still safely continue."); error!("Failue: {err:#?}"); // panic in tests. if cfg!(test) { panic!("Truncation cleanup fail."); } // We can keep going without caring, too bad about them wasted blocks tho, eh? Ok(()) } } } /// Returns how many blocks were freed. fn truncate_cleanup(pre_collected: Vec, next_extent_block: DiskPointer) -> Result { // The rest of the cleanup happens in this function so we can easily check if any of it fails. debug!("Starting truncation cleanup, started with {} pre-collected blocks...", pre_collected.len()); // Dump those pointers into a new local pile let mut to_free: Vec = pre_collected; // == Collect all of the DiskPointers to the remaining ExtentBlocks. == // == - Cant immediately remove them, we still need the DiskPointers to the data blocks == // Make sure the pointer goes somewhere, if it doesnt, we can skip this step. // Shadow it so we can update it for the loop let mut next_extent_block = next_extent_block; // Also collect the extents, gotta remember where the're from too // Cant pre-allocate this, no idea how many extent blocks there will be. let mut extents: Vec<(u16, FileExtent)> = Vec::new(); debug!("Collecting blocks referred to by extents..."); while !next_extent_block.no_destination() { // Open the extent let raw: RawBlock = CachedBlockIO::read_block(next_extent_block)?; let read: FileExtentBlock = FileExtentBlock::from_block(&raw); // get the extents let tents: Vec<(u16, FileExtent)> = read.get_extents().into_iter().map(|i|{ (next_extent_block.disk, i) }).collect(); extents.extend(tents); // Add this extent block to the pile to_free.push(next_extent_block); // set the next block next_extent_block = read.next_block; } debug!("Done."); // == Collect all of the DiskPointers to the data blocks within the extents in the remaining extent blocks. == // == - Loop over the collected disk pointers in the previous step, open the FileExtentBlock, extract the == // == - DiskPointers from all of the contained Extents == // To save reads i grabbed the extents as i read the blocks, we just need the pointers debug!("Extracting pointers..."); for (_, tent) in extents { // I used to be a tent, but I got too old and cant pitch them anymore. let pointers = tent.get_pointers(); to_free.extend(pointers); } debug!("Done."); // == Free all of the blocks we've collected == // Sort the blocks to reduce the amount of head seeking. This also groups together the disks. to_free.sort_unstable_by_key(|block| (block.disk, block.block)); // Make sure there are no duplicates. // Yes we shouldn't be getting them in the first place, but if we somehow do, this will // crash due to double free. let pre_dedup = to_free.len(); to_free.dedup(); let post_dedup = to_free.len(); if pre_dedup != post_dedup { // Sizes are different, cooked. panic!("There was duplicate blocks during truncation cleanup. Cannot continue."); } // Hold onto how many blocks we're freeing for returning. let amount_freed = to_free.len(); // Split into sections based on when the disk changes. let chunked = to_free.chunk_by(|a, b| a.disk == b.disk); // Now go free all of those blocks. // This will zero out the blocks, and remove them from the cache for us. debug!("Freeing blocks..."); for chunk in chunked { let _ = Pool::free_pool_block_from_disk(chunk)?; } debug!("Done."); debug!("All done cleaning up truncation."); // All done! Return how many blocks we freed. Ok(amount_freed) } /// Just flushes the current FileExtentBlock to disk, nice helper function fn flush_to_disk(block: &FileExtentBlock) -> Result<(), DriveError> { // Raw it let raw = block.to_block(); // Write it. CachedBlockIO::update_block(&raw)?; Ok(()) } ================================================ FILE: src/pool/disk/standard_disk/block/io/inode/mod.rs ================================================ pub mod read; #[cfg(test)] mod tests; pub mod write; ================================================ FILE: src/pool/disk/standard_disk/block/io/inode/read.rs ================================================ // Functions from other files are used for reading inodes at this level. // If you need to touch an inode, you're probably doing some lower level stuff, so // you need to use lower level tools. // // // bonus joke: // what did the blank floppy say to the drive? // // ================================================ FILE: src/pool/disk/standard_disk/block/io/inode/tests.rs ================================================ // Inode tests. // Unwrapping is okay here, since we want unexpected outcomes to fail tests. #![allow(clippy::unwrap_used)] use std::path::PathBuf; use log::debug; use tempfile::{TempDir, tempdir}; use crate::{ filesystem::filesystem_struct::{FilesystemOptions, FlusterFS}, pool::{ disk::standard_disk::block::inode::inode_struct::Inode, pool_actions::pool_struct::Pool, }, }; use test_log::test; // We want to see logs while testing. // In the inode test file, we already create some functions that we can reuse in here. // Since these tests touch global state, they need to be isolated... somehow #[test] fn add_inode() { // Use the filesystem starter to get everything in the right spots let _fs = get_filesystem(); // Now try adding a directory to the pool let _ = Pool::add_inode(Inode::get_random()).unwrap(); } #[test] fn add_many_inode() { let _fs = get_filesystem(); for _ in 0..1000 { let _ = Pool::add_inode(Inode::get_random()).unwrap(); } } // We need a filesystem to run directory tests on. fn get_filesystem() -> FlusterFS { let temp_dir = get_new_temp_dir(); let floppy_drive: PathBuf = PathBuf::new(); // This is never read since we are using temporary disks. let fs_options = FilesystemOptions::new(Some(temp_dir.path().to_path_buf()), floppy_drive, Some(false), false); FlusterFS::start(&fs_options) } // // Helper functions // // Temporary directories for virtual disks fn get_new_temp_dir() -> TempDir { let mut dir = tempdir().unwrap(); dir.disable_cleanup(true); debug!( "Created a temp directory at {}, it will not be deleted on exit.", dir.path().to_string_lossy() ); dir } ================================================ FILE: src/pool/disk/standard_disk/block/io/inode/write.rs ================================================ // Inode this was going somewhere... // Imports // Implementations // Functions use log::trace; use crate::{ error_types::{ block::BlockManipulationError, drive::DriveError }, pool::{ disk::{ generic::{ block::block_structs::RawBlock, generic_structs::pointer_struct::DiskPointer, io::cache::cache_io::CachedBlockIO, }, standard_disk::block::inode::inode_struct::{ Inode, InodeBlock, InodeLocation }, }, pool_actions::pool_struct::{ Pool, GLOBAL_POOL }, } }; // The pool MUST exist for inodes to be created. macro_rules! get_pool { () => { if let Ok(innards) = GLOBAL_POOL.get().expect("There has to be a global pool at this point.").try_lock() { innards } else { // Cannot do inode stuff with dying pool, dying pools need to just shut down immediately. panic!("A poisoned pool cannot have inode operations performed against it!"); } }; } // For the pool implementations, we do not use Self, as we might try to double mut it if the inode // addition routine adds a new disk. impl Pool { /// Add an inode to the never ending inode chain. /// /// This method will look at the pool, and attempt adding an inode /// to the lowest disk with the most recent successful inode write. /// (Ignoring manual writes done outside of this method.) /// /// This function will traverse to the next blocks in the chain, /// and create new disks if needed. /// /// Returns where the inode ended up. pub fn fast_add_inode(inode: Inode) -> Result { trace!("Fast adding inode..."); // Get the pool's latest inode disk let start_pointer: DiskPointer = get_pool!().header.latest_inode_write; // load in that block let da_reader: RawBlock = CachedBlockIO::read_block(start_pointer)?; let start_block: InodeBlock = InodeBlock::from_block(&da_reader); let result = go_add_inode(inode, start_block)?; // Where that ended up needs to be known let success_write_pointer: DiskPointer = result.pointer; // Update the pool with new successful write. { let mut pool = get_pool!(); pool.header.latest_inode_write = success_write_pointer; } // all done Ok(result) } /// Add an inode to the never ending inode chain. /// /// This method adds an inode to the pool, starting from /// the origin inode block on the origin disk. /// This may take a long time, but will ensure that the first /// available spot within the entire inode pool is used. /// /// This function will traverse to the next blocks in the chain, /// and create new disks if needed. /// /// Returns where the inode ended up. pub fn add_inode(inode: Inode) -> Result { trace!("Adding inode, starting from disk 1..."); // Start from the origin. let start_pointer: DiskPointer = DiskPointer { disk: 1, block: 1 }; let da_reader: RawBlock = CachedBlockIO::read_block(start_pointer)?; let start_block: InodeBlock = InodeBlock::from_block(&da_reader); let result = go_add_inode(inode, start_block)?; // Where that ended up needs to be known let success_write_pointer: DiskPointer = result.pointer; // Update the pool with new successful write. { let mut pool = get_pool!(); pool.header.latest_inode_write = success_write_pointer; } // all done Ok(result) } } fn go_add_inode(inode: Inode, start_block: InodeBlock) -> Result { // We will start from the provided block. // For when we switch disks (abstracted away but hehe!) let mut current_block_number: u16 = start_block.block_origin.block; let mut current_disk: u16 = start_block.block_origin.disk; let mut current_block: InodeBlock = start_block; // For when we eventually find a spot. let inode_offset: u16; // Now we loop, looking for room. loop { // Does the current block have room? match current_block.try_add_inode(inode) { Ok(ok) => { // We've got a spot! inode_offset = ok; break; } Err(error) => match error { BlockManipulationError::OutOfRoom => { // Not enough room on this block, go fish. } BlockManipulationError::Impossible => { // Impossible offsets should never happen. // Cant keep going with logic errors. panic!("Impossible inode offset. {inode:#?}"); }, BlockManipulationError::NotFinalBlockInChain | BlockManipulationError::NotPresent => { // Not possible here, inodes dont care about being the final block, // and adding doesnt check if an item is present. panic!("Apparently inodes DO care!"); }, }, } // There wasn't enough room, proceed to the next block. let pointer_to_next_block: DiskPointer = get_next_block(current_block.clone())?; // Go there // We always re-open the disk to get the freshest allocation table. current_disk = pointer_to_next_block.disk; let reader: RawBlock = CachedBlockIO::read_block(pointer_to_next_block)?; current_block = InodeBlock::from_block(&reader); current_block_number = pointer_to_next_block.block; // start over! continue; } // The inode has now been added to the block, we must write this to disk before continuing. let block_to_write: RawBlock = current_block.to_block(); // We are updating, because how would we be writing back to a block that was not allocated when we read it? CachedBlockIO::update_block(&block_to_write)?; // All done! Now we can return where that inode eventually ended up let pointer: DiskPointer = DiskPointer { disk: current_disk, block: current_block_number, }; let new_location: InodeLocation = InodeLocation::new(pointer, inode_offset); Ok(new_location) } fn get_next_block(current_block: InodeBlock) -> Result { // Extract the pointer if let Some(ok) = current_block.next_block() { // Sweet, it exists already. return Ok(ok); } // Time to make a new inode block // Why is there a second function for that? idk i felt like it // This writes the new block. let new_block_location: DiskPointer = make_new_inode_block()?; // Now we just need to update the block we were called on. let mut the_cooler_inode = current_block; the_cooler_inode.new_destination(new_block_location); let please_let_me_hit = the_cooler_inode.to_block(); // Update that block G CachedBlockIO::update_block(&please_let_me_hit)?; // return the pointer to the next block. Ok(new_block_location) } /// We need a new inode block, we will reach upwards and get a new block made for us. fn make_new_inode_block() -> Result { // Ask the pool for a new block pwease // No need for crc since we'll be overwriting it immediately. let ask_nicely = Pool::find_and_allocate_pool_blocks(1, false)?; // And you shall receive. let new_block_location = ask_nicely[0]; // New block to throw there let new_block: InodeBlock = InodeBlock::new(new_block_location); let but_raw: RawBlock = new_block.to_block(); // Write it Ralph! // I'm gonna write it! // New block, so standard write. CachedBlockIO::update_block(&but_raw)?; // Now throw it back, I mean the pointer Ok(new_block_location) } ================================================ FILE: src/pool/disk/standard_disk/block/io/mod.rs ================================================ pub(crate) mod directory; mod file; mod inode; ================================================ FILE: src/pool/disk/standard_disk/block/mod.rs ================================================ pub mod directory; mod file_extents; pub mod header; pub mod inode; pub mod io; ================================================ FILE: src/pool/disk/standard_disk/mod.rs ================================================ pub mod block; mod standard_disk_methods; pub mod standard_disk_struct; ================================================ FILE: src/pool/disk/standard_disk/standard_disk_methods.rs ================================================ // Imports use std::fs::File; use log::debug; use crate::{ error_types::drive::DriveError, pool::{ disk::{ drive_struct::{ DiskBootstrap, DiskType, FloppyDrive }, generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock, }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::{ cache::cache_io::CachedBlockIO, read::{ read_block_direct, read_multiple_blocks_direct }, write::{ write_block_direct, write_large_direct } }, }, standard_disk::{ block::{ directory::directory_struct::DirectoryBlock, header::header_struct::{ StandardDiskHeader, StandardHeaderFlags }, inode::inode_struct::{ Inode, InodeBlock, InodeDirectory, InodeFlags, InodeTimestamp, }, }, standard_disk_struct::StandardDisk, }, }, pool_actions::pool_struct::{ Pool, GLOBAL_POOL }, } }; // Implementations // !! Only numbered options should be public! !! impl DiskBootstrap for StandardDisk { fn bootstrap(file: File, disk_number: u16) -> Result { debug!("Boostrapping a standard disk..."); // Update how many blocks are free in the pool // New standard disks have only the header allocated. // 2880 - 1 = 2879 // But, the disk setup process will automatically decrement this count for us. { let arc = GLOBAL_POOL.get().expect("Shouldn't be allocating disks without a pool"); let mut pool = if let Ok(innards) = arc.try_lock() { innards } else { // Poisoned! We can't make new disks while poisoned. panic!("Attempted to bootstrap a standard disk while pool is poisoned!"); }; pool.header.pool_standard_blocks_free += 2880; } // Make the disk debug!("Running create..."); let mut disk = create(file, disk_number)?; // Now that we have a disk, we can use the safe IO. // if this is disk 1 then we need to add: // Inode block // Directory block // the root directory to that block if disk_number != 1 { // Dont need to do anything. debug!("Done bootstrapping standard disk."); return Ok(disk); } debug!("This is the origin standard disk, doing a bit more..."); // the new origin disk needs to have blocks 1 and 2 allocated as well for the first inode/directory blocks. let _ = disk.allocate_blocks(&vec![1,2])?; // Ignoring resulting value, since it will always be 2. // Which means we also need to update the pool block count again. // Write the inode block debug!("Writing inode block..."); let inode_block_origin: DiskPointer = DiskPointer { disk: disk_number, block: 1, }; let inode_block = InodeBlock::new(inode_block_origin); let inode_writer = inode_block.to_block(); disk.unchecked_write_block(&inode_writer)?; // Create the directory block // Write that to the disk. It goes in block 2. debug!("Writing root directory block..."); let directory_block_origin: DiskPointer = DiskPointer { disk: disk_number, block: 2, }; let directory_block: DirectoryBlock = DirectoryBlock::new(directory_block_origin); let the_directory_block: RawBlock = directory_block.to_block(); disk.unchecked_write_block(&the_directory_block)?; // Now we need to manually add the inode that points to it. Because the inode at the 0 index // of block 1 is the inode that points to the root directory // Add the root inode debug!("Writing root directory inode..."); let pointer_to_dat_mf: DiskPointer = DiskPointer { disk: 1, block: 2, // The root directory is at block 2. }; let root_directory_inode = InodeDirectory::from_disk_pointer(pointer_to_dat_mf); let right_now = InodeTimestamp::now(); let the_actual_inode: Inode = Inode { flags: InodeFlags::MarkerBit, // Not a file, so only the marker. file: None, directory: Some(root_directory_inode), created: right_now, modified: right_now, }; // Adding inodes automatically makes more room if needed, and since this is disk 1, this is actually // the first inode we EVER create, so not having room is preposterous! let inode_result = Pool::add_inode(the_actual_inode).expect("Brand new disk is full????"); // Make sure that actually ended up at the right spot. if inode_result.pointer != inode_block_origin || inode_result.offset != 0 { // Making the inode block is cooked. Too much depends on this being in the correct spot. panic!("Inode ended up at the wrong spot during disk creation! Cannot continue!"); } // All done! debug!("Done bootstrapping standard disk."); // Since we wrote information to it, we need to read in that disk again before returning it #[allow(deprecated)] // We do not use the cache while bootstrapping. let finished_disk: StandardDisk = match FloppyDrive::open(disk_number)? { DiskType::Standard(standard_disk) => standard_disk, _ => { // User managed to swap out disk 1 at the exact moment we finished putting it together. Shame on them. panic!("Disk 1 was replaced in the drive during bootstrapping re-load."); }, }; Ok(finished_disk) } fn from_header(block: RawBlock, file: File) -> Self { // load in the header. // We assume the caller has passed in the freshest version of the header. let header: StandardDiskHeader = StandardDiskHeader::from_block(&block); StandardDisk { number: header.disk_number, disk_file: file, header, } } } // This disk has block level allocations impl BlockAllocation for StandardDisk { fn get_allocation_table(&self) -> &[u8] { &self.header.block_usage_map } fn set_allocation_table(&mut self, new_table: &[u8]) -> Result<(), DriveError> { // Gotta use type annotations here, inline methods on the result dont seem to work on // declaration/assignments. let cast = if let Ok(casted) = TryInto::<[u8; 360]>::try_into(new_table) { casted } else { // Incoming bytes were the wrong length. panic!("Allocation table length is wrong!"); }; self.header.block_usage_map = cast; self.flush() } } /// Ocasionally, we need to create fake headers during disk loading. impl StandardDiskHeader { fn spoof() -> Self { Self { flags: StandardHeaderFlags::from_bits_retain(0b00100000), // Gotta set that marker bit. disk_number: u16::MAX, block_usage_map: [1u8; 360], } } } // // Public functions // /// Initializes a disk by writing header data. /// Returns the newly created disk. /// /// This will only work on a disk that is blank / header-less. /// This will create a disk of any disk number, it is up to the caller to ensure that /// duplicate disks are not created, and to track the creation of this new disk. fn create(file: File, disk_number: u16) -> Result { debug!("Creating new standard disk {disk_number}"); debug!("Creating spoofed disk..."); // Spoof the header, since we're about to give it a new one. let mut disk: StandardDisk = StandardDisk { number: disk_number, header: StandardDiskHeader::spoof(), disk_file: file, }; // Now give it some head er // This function checks if the disk is blank for us. debug!("Initializing the disk from spoof..."); initialize_numbered(&mut disk, disk_number)?; // done debug!("Done creating disk."); Ok(disk) } // // Private functions // /// Initialize a normal disk for usage. (NOT DATA, NOT POOL) /// Expects a disk without a header. /// Will wipe the rest of the disk, /// /// Errors if provided with a disk that has a header. fn initialize_numbered(disk: &mut StandardDisk, disk_number: u16) -> Result<(), DriveError> { debug!("Initializing a new standard disk..."); // A new, fresh disk! // Time to write in all of the header data. // Construct the new header // New disks have no flags set, besides the required marker bit let mut flags: StandardHeaderFlags = StandardHeaderFlags::empty(); flags.insert(StandardHeaderFlags::Marker); // New disks do have a few pre-allocated blocks, namely the header and the first inode block // But they will be allocated during the creation process. let mut block_usage_map: [u8; 360] = [0u8; 360]; // We must mark the first block used, because we cant run the allocated without being able to update // the header block (which needs to be allocated for the update) block_usage_map[0] = 0b10000000; let header = StandardDiskHeader { flags, disk_number, block_usage_map, }; // Now serialize that, and write it let header_block = &header.to_block(); // Update the header on the provided disk, since it's currently spoofed. disk.header = header; // Since this is a brand new disk without proper header information finalized, we have to do a direct write here debug!("Writing header..."); disk.unchecked_write_block(header_block)?; debug!("Header written."); // All done! debug!("Done initializing."); Ok(()) } // Generic disk operations impl GenericDiskMethods for StandardDisk { #[doc = " Read a block"] #[doc = " Cannot bypass CRC."] fn unchecked_read_block(&self, block_number: u16) -> Result { // This is the first call, we have not recursed. read_block_direct(&self.disk_file, self.number, block_number, false, false) } #[doc = " Write a block"] fn unchecked_write_block(&mut self, block: &RawBlock) -> Result<(), DriveError> { // This is the first call, we have not recursed. write_block_direct(&self.disk_file, block, false) } #[doc = " Write chunked data, starting at a block."] fn unchecked_write_large(&mut self, data:Vec, start_block:DiskPointer) -> Result<(), DriveError> { write_large_direct(&self.disk_file, &data, start_block) } #[doc = " Get the inner file used for IO operations"] fn disk_file(self) -> File { self.disk_file } #[doc = " Get the number of the floppy disk."] fn get_disk_number(&self) -> u16 { self.number } #[doc = " Set the number of this disk."] fn set_disk_number(&mut self, disk_number: u16) { self.number = disk_number } #[doc = " Get the inner file used for write operations"] fn disk_file_mut(&mut self) -> &mut File { &mut self.disk_file } #[doc = " Sync all in-memory information to disk"] #[doc = " Headers and such."] fn flush(&mut self) -> Result<(), DriveError> { // Not really to disk, but if nobody is looking... CachedBlockIO::update_block(&self.header.to_block()) } #[doc = " Read multiple blocks"] #[doc = " Does not check CRC!"] fn unchecked_read_multiple_blocks(&self, block_number: u16, num_block_to_read: u16) -> Result, DriveError> { // This is the first call, we have not recursed. read_multiple_blocks_direct(&self.disk_file, self.number, block_number, num_block_to_read, false) } } ================================================ FILE: src/pool/disk/standard_disk/standard_disk_struct.rs ================================================ // Information about a standard disk // Imports use super::block::header::header_struct::StandardDiskHeader; // Structs, Enums, Flags #[derive(Debug)] pub struct StandardDisk { /// Which disk is this? pub number: u16, /// The disk header pub header: StandardDiskHeader, /// The file that refers to this disk pub(in super::super) disk_file: std::fs::File, } ================================================ FILE: src/pool/disk/unknown_disk/mod.rs ================================================ pub mod unknown_disk_methods; pub mod unknown_disk_struct; ================================================ FILE: src/pool/disk/unknown_disk/unknown_disk_methods.rs ================================================ use std::fs::File; use crate::{ error_types::drive::DriveError, pool::disk::{ generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock }, disk_trait::GenericDiskMethods, generic_structs::pointer_struct::DiskPointer, io::write::write_block_direct }, unknown_disk::unknown_disk_struct::UnknownDisk } }; impl GenericDiskMethods for UnknownDisk { #[doc = " Read a block"] #[doc = " Cannot bypass CRC."] fn unchecked_read_block(&self, _block_number: u16) -> Result { // We cant read from generic disks. panic!("Attempted to read blocks from a disk we know nothing about, we cannot do that."); } #[doc = " Write a block"] fn unchecked_write_block(&mut self, block: &RawBlock) -> Result<(), DriveError> { // This is the first call, we have not recursed. write_block_direct(&self.disk_file, block, false) } #[doc = " Get the inner file used for IO operations"] fn disk_file(self) -> File { self.disk_file } #[doc = " Get the number of the floppy disk."] fn get_disk_number(&self) -> u16 { // Unknown disks have no number. panic!("Attempted to get the disk number of a disk we know nothing about! Not allowed!"); } #[doc = " Set the number of this disk."] fn set_disk_number(&mut self, _disk_number: u16) { // You cannot set the disk number of an unknown disk. panic!("Attempted to set the disk number of a disk we know nothing about! Not allowed!"); } #[doc = " Get the inner file used for write operations"] fn disk_file_mut(&mut self) -> &mut File { &mut self.disk_file } #[doc = " Sync all in-memory information to disk"] fn flush(&mut self) -> Result<(), DriveError> { // There is no in-memory information for this disk. // So just don't do anything. Ok(()) } #[doc = " Write chunked data, starting at a block."] fn unchecked_write_large(&mut self, _data: Vec, _start_block: DiskPointer) -> Result<(), DriveError> { panic!("Cannot do large writes to unknown disks!"); } #[doc = " Read multiple blocks"] #[doc = " Does not check CRC!"] fn unchecked_read_multiple_blocks(&self, _block_number: u16, _num_block_to_read: u16) -> Result,DriveError> { panic!("Cannot do large reads from unknown disks!"); } } impl UnknownDisk { pub fn new(file: File) -> Self { Self { disk_file: file } } } impl BlockAllocation for UnknownDisk { #[doc = " Get the block allocation table"] fn get_allocation_table(&self) -> &[u8] { panic!("Unknown disks do not support allocations."); } #[doc = " Update and flush the allocation table to disk."] fn set_allocation_table(&mut self, _new_table: &[u8]) -> Result<(), DriveError> { panic!("Unknown disks do not support allocations."); } } ================================================ FILE: src/pool/disk/unknown_disk/unknown_disk_struct.rs ================================================ // Sometimes we know nothing about a disk, but we still need a type for it // so we can satisfy type constraints on DiskType #[derive(Debug)] pub struct UnknownDisk { /// Every disk needs a file pub(in super::super) disk_file: std::fs::File, } ================================================ FILE: src/pool/io/allocate.rs ================================================ // Pool level block allocations use log::{debug, error}; use crate::{ error_types::drive::DriveError, pool::{ disk::{ generic::{ block::{ allocate::block_allocation::BlockAllocation, block_structs::RawBlock, crc::add_crc_to_block }, generic_structs::pointer_struct::DiskPointer, io::cache::{ cache_io::CachedBlockIO, cached_allocation::CachedAllocationDisk } }, standard_disk::standard_disk_struct::StandardDisk, }, pool_actions::pool_struct::{ Pool, GLOBAL_POOL }, }, tui::{ notify::NotifyTui, tasks::TaskType } }; // We need to be able to get at the innards of the Global Pool quite often. // If they dont exist at this stage, we're cooked regardless and must exit. macro_rules! get_pool { () => { if let Ok(innards) = GLOBAL_POOL.get().expect("Global pool should be created at this stage!").try_lock() { innards } else { // Somebody peed in the pool. // Usually we would try to do something about this, but the pool should be shutting down if // poisoning has occurred. Which in that case, we should NOT be allocating new blocks! // No recovery will be attempted. panic!("A poisoned pool cannot be allocated against!"); } }; } impl Pool { /// Finds blocks across the entire pool. /// /// Can only allocate a maximum of 32MB worth of blocks in one go. /// If you are asking for that many, something is 100% wrong. /// Writes should be limited to 1MB so this should never happen. /// /// If this fails, ur kinda cooked ngl, since now a bunch of random blocks have been /// allocated for no reason. /// /// Searches standard disks, yada yada. /// /// This will mark the blocks as allocated. /// /// You can optionally also set the CRC on the new empty blocks, which is useful for /// new file blocks. /// /// Will add new disks if needed. /// /// May swap disks, will not return to where it started. /// /// Returns disk pointers for the newly reserved blocks, or a disk error. pub fn find_and_allocate_pool_blocks(blocks: u16, add_crc: bool) -> Result, DriveError> { // This is just an abstraction to force a different function name, even though // the function it calls is the same as find_free_pool_blocks() go_find_free_pool_blocks(blocks, add_crc) } /// Frees a block from a disk in the pool. /// /// It's not required, but you should sort the order of the blocks to /// reduce drive seeking. /// /// All blocks must come from same disk. /// /// Returns how many blocks were freed. /// /// Will destroy any data currently in that block. pub fn free_pool_block_from_disk(blocks: &[DiskPointer]) -> Result { go_deallocate_pool_block(blocks) } } fn go_find_free_pool_blocks(blocks: u16, add_crc: bool) -> Result, DriveError> { let handle = NotifyTui::start_task(TaskType::PoolAllocateBlocks(blocks), blocks.into()); debug!("Attempting to allocate {blocks} blocks across the pool..."); // Get where we're most likely to find the next free block, and hold onto the highest // known disk as well, in-case we need to grow the pool. // Scoped to drop the mutex. let (probable_disk, highest_disk) = { let header = get_pool!().header; (header.disk_with_next_free_block, header.highest_known_disk) }; // First we open up the disk with the most recent successful pool allocation let mut disk_to_check = probable_disk; let mut new_highest_disk = highest_disk; // We may create new disks during the process. // We will assume that we will be getting as many blocks as we need, so we can pre-allocate the // vec. let mut free_blocks: Vec = Vec::with_capacity(blocks.into()); // Make sure the highest disks and disk to check are valid. // Getting here is a bad sign, allowing this continue is a worse one. // We will just give up if such an event transpires. if disk_to_check == u16::MAX || new_highest_disk < 1 { // Either we're trying to allocate on an invalid disk, or the pool // doesnt have enough disks? error!("Tried to allocate on u16::MAX or we didn't have enough disks!"); error!("We cannot continue!"); panic!("Impossible pool-level allocation!"); } // Now we loop until we find enough free blocks. loop { // Since we use a mix of real and fake disks in here, we need to have a type that we can use for our allocation // methods. So we will box it up. Yes this is kinda evil. let mut disk: Box; // Check if the disk we are about to load is out of range if disk_to_check > new_highest_disk { debug!("Ran out of room, creating new disk..."); // We need to make this disk before trying to allocate blocks on it. let new_disk: StandardDisk = Pool::new_disk::()?; disk = Box::new(new_disk); // increment the highest known disk new_highest_disk += 1; } else { // We are loading a pre-existing disk, hit up the cache for it. let new_disk: CachedAllocationDisk = CachedAllocationDisk::open(disk_to_check)?; disk = Box::new(new_disk); }; // Check if this disk has enough room. // we will grab all we can. match disk.find_free_blocks(blocks - free_blocks.len() as u16) { Ok(ok) => { debug!("Found the last {} blocks we needed on disk {}!", ok.len(), disk_to_check); // We were able to allocate all of the blocks we asked for! // We're done! free_blocks.append(&mut block_indexes_to_pointers(&ok, disk_to_check)); // Allocate those blocks. let _ = disk.allocate_blocks(&ok)?; // Now we drop the disk to make it flush the new allocation table. drop(disk); // We also need to update the global pool to say these were marked as used, otherwise we would never know. // Trust me I found out the hard way. debug!("Updating the pool's free block count..."); { let mut header = get_pool!().header; header.pool_standard_blocks_free -= ok.len() as u32; } // Add crc to blocks if requested. if add_crc { debug!("CRC requested, adding..."); write_empty_crc(&ok, disk_to_check)?; } NotifyTui::complete_multiple_task_steps(&handle, ok.len() as u64); break; } Err(amount) => { // There wasn't enough blocks free on this disk, but we can allocate at least `amount` debug!("Still need more blocks, disk {disk_to_check} only had {amount} blocks free."); // Bail early if there's zero blocks if amount == 0 { disk_to_check += 1; continue; } let blockie_doos = disk.find_free_blocks(amount).expect("Disk should not change its mind about how many blocks it has free"); free_blocks.append(&mut block_indexes_to_pointers(&blockie_doos, disk_to_check)); // Allocate those blocks if needed. let _ = disk.allocate_blocks(&blockie_doos)?; // Now we drop the disk to make it flush the new allocation table. drop(disk); // We also need to update the global pool to say these were marked as used, otherwise we would never know. // Trust me I found out the hard way. debug!("Updating the pool's free block count..."); { let mut header = get_pool!().header; header.pool_standard_blocks_free -= blockie_doos.len() as u32; } // Add crc to blocks if requested if add_crc { debug!("CRC requested, adding..."); write_empty_crc(&blockie_doos, disk_to_check)?; } NotifyTui::complete_multiple_task_steps(&handle, blockie_doos.len() as u64); // Waiter! Waiter! More disks please! disk_to_check += 1; continue; } } } // Now that we have allocated, the most probable disk is the last disk we got blocks from. { let header = &mut get_pool!().header; header.disk_with_next_free_block = disk_to_check; } // We will sort the resulting vector to make to group the disks together, this will // reduce swapping. free_blocks.sort_unstable_by_key(|pointer| (pointer.disk, pointer.block)); debug!("Allocation complete."); NotifyTui::finish_task(handle); Ok(free_blocks) } // helper fn block_indexes_to_pointers(blocks: &Vec, disk: u16) -> Vec { // We will have as many pointers as we got blocks in. let mut result: Vec = Vec::with_capacity(blocks.len()); for block in blocks { result.push(DiskPointer { disk, block: *block }); } result } /// Sometimes we need a new block that is empty, but we still need to have the crc set. /// Assumes block are already marked. /// /// This method only works on standard disks fn write_empty_crc(blocks: &[u16], disk: u16) -> Result<(), DriveError> { debug!("Adding crc to {} blocks on disk {disk}...", blocks.len()); let handle = NotifyTui::start_task(TaskType::WriteCRC, blocks.len() as u64); // These new blocks do not have their CRC set, we need to just write empty blocks to them to set the crc. let mut empty_data: [u8; 512] = [0_u8; 512]; // CRC that sucker add_crc_to_block(&mut empty_data); // Make block to write, must update inside of loop. for block in blocks { let block_origin: DiskPointer = DiskPointer { disk, block: *block, }; let empty_raw_block: RawBlock = RawBlock { block_origin, data: empty_data, }; CachedBlockIO::update_block(&empty_raw_block)?; NotifyTui::complete_task_step(&handle); } NotifyTui::finish_task(handle); // All of the blocks now have a empty block with a crc on it. Ok(()) } fn go_deallocate_pool_block(blocks: &[DiskPointer]) -> Result { debug!("Deallocating some blocks from the pool..."); let handle = NotifyTui::start_task(TaskType::DiskDeallocateBlocks(blocks.len() as u16), blocks.len() as u64); // We assume the blocks are pre-sorted to reduce disk seeking, but even // if they aren't, the cache will re-sort them on write. // We have to be fed at least one block. if blocks.is_empty() { // Freeing 0 blocks does nothing, so... return Ok(0) } // All of the disk numbers must be the same, otherwise this is a mallformed free call. let starter = blocks[0]; // Already checked if the array is empty. if !blocks.iter().all(|pointer| pointer.disk == starter.disk) { // At least one of the blocks is for a different disk, we can't do that. panic!("Pool deallocation attempted to free blocks from multiple different disks at once! Not allowed!"); } let mut extracted_blocks: Vec = Vec::with_capacity(blocks.len()); for block in blocks { // Hold onto the block number, need it for disk call. extracted_blocks.push(block.block); } // Go zero out the blocks on the disk, but all of the IO will be cached. Thus if // we are trying to free blocks on a disk that isn't in the drive right now, we // might need to swap, also this will auto-chunk the writes, which is faster. debug!("Zeroing out blocks..."); for block in blocks { let empty: RawBlock = RawBlock { block_origin: *block, data: [0_u8; 512], }; // Zero em out with the cache. CachedBlockIO::update_block(&empty)?; NotifyTui::complete_task_step(&handle); } // Now go to and free the blocks from the allocation table. debug!("Freeing blocks from the disk header..."); let mut disk: CachedAllocationDisk = CachedAllocationDisk::open(starter.disk)?; let blocks_freed = disk.free_blocks(&extracted_blocks)?; // Drop it to flush the updated header to cache. drop(disk); // If the current disk in the pool marked with free blocks is higher than the blocks we just freed, // we need to move back the search start for finding new free blocks. // // Then update the free count, since new blocks are available. { let header = &mut get_pool!().header; if header.disk_with_next_free_block > starter.disk { // It's higher, we need to move the pool back. header.disk_with_next_free_block = starter.disk; } header.pool_standard_blocks_free += blocks_freed as u32; } // Return the number of blocks freed. debug!("Done. {blocks_freed} pool blocks freed."); NotifyTui::finish_task(handle); Ok(blocks_freed) } ================================================ FILE: src/pool/io/mod.rs ================================================ pub mod allocate; ================================================ FILE: src/pool/mod.rs ================================================ pub(crate) mod disk; pub mod io; pub mod pool_actions; ================================================ FILE: src/pool/pool_actions/mod.rs ================================================ pub mod pool_methods; pub mod pool_struct; ================================================ FILE: src/pool/pool_actions/pool_methods.rs ================================================ // Interacting with the pool // Imports use super::pool_struct::GLOBAL_POOL; use super::pool_struct::Pool; use crate::error_types::drive::DriveError; use crate::pool::disk::blank_disk::blank_disk_struct::BlankDisk; use crate::pool::disk::drive_struct::DiskBootstrap; use crate::pool::disk::drive_struct::FloppyDrive; use crate::pool::disk::generic::block::block_structs::RawBlock; use crate::pool::disk::generic::disk_trait::GenericDiskMethods; use crate::pool::disk::generic::generic_structs::pointer_struct::DiskPointer; use crate::pool::disk::generic::io::cache::cache_io::CachedBlockIO; use crate::pool::disk::pool_disk::block::header::header_struct::PoolDiskHeader; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryBlock; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryItemFlags; use crate::pool::disk::standard_disk::block::directory::directory_struct::DirectoryItem; use crate::pool::disk::standard_disk::block::inode::inode_struct::InodeLocation; use crate::pool::disk::standard_disk::standard_disk_struct::StandardDisk; use crate::tui::notify::NotifyTui; use crate::tui::tasks::TaskType; use log::debug; use log::error; use std::sync::Arc; use std::sync::Mutex; // Implementations impl Pool { /// Flush all info about the pool to the pool disk. pub fn flush() -> Result<(), DriveError> { flush_pool() } /// Read in pool information from disk /// Returns a handle/pointer/whatever pub fn load() -> Arc> { load() } /// Create a new disk of type and add it to the pool /// Returns that new disk. pub fn new_disk() -> Result { add_disk::() } /// Brand new pools need to run some setup functions to get everything in a ready to use state. fn initalize() -> Result<(), DriveError> { initalize_pool() } /// Get the root inode block /// /// May swap disks, but you should be working with enough abstractions to not care. pub fn get_root_directory() -> Result { pool_get_root_directory() } /// Get a DirectoryItem that has details about the root directory. pub fn get_root_directory_item() -> DirectoryItem { pool_get_root_directory_item() } } /// Sync information about the pool to disk pub(super) fn flush_pool() -> Result<(), DriveError> { debug!("Flushing pool info to disk..."); // Grab the pool let global_pool = GLOBAL_POOL .get() .expect("The pool has to exist, otherwise we couldn't shut it down."); // Now, since we're flushing info, if we're shutting down after a panic, we need to be able to // flush the pool even if it got poisoned. global_pool.clear_poison(); let pool_header:PoolDiskHeader = global_pool.try_lock() .expect("Single threaded, already cleared poison.") .header; // Now write that back to disk. pool_header.write()?; debug!("Pool flushed."); Ok(()) } /// Read in pool information from disk. /// Will prompt to make new pools if needed. /// Returns a pointer thingy to to the global. pub(super) fn load() -> Arc> { debug!("Loading in pool information..."); // Read in the header. If this fails, we cannot start the filesystem. // We try at most 10 times. let mut header: Option = None; for _ in 0..10 { match PoolDiskHeader::read() { Ok(ok) =>{ header = Some(ok); break }, Err(error) => { // If no disk was inserted, yell at the user and try again if error == DriveError::DriveEmpty { // Dumbass. println!("Yo. The drive is empty. Actually put in the disk."); continue; } } }; }; // Did we get it? let header: PoolDiskHeader = if let Some(read) = header { // All good. read } else { // Failed to load in the disk header. error!("Failed to acquire pool header after 10 tries! Giving up!"); error!("Fluster has failed to load the pool header."); error!("Fluster will now exit."); panic!("Failed to get pool header!"); }; let pool = Pool { header, }; // Wrap it for sharing. let shared_pool = Arc::new(Mutex::new(pool)); // Set the global static. This will only work the first time. // Since this is only called on fluster startup, this should only ever be called once. // If we somehow hit here again, we'll just exit if GLOBAL_POOL.set(shared_pool.clone()).is_err() { // wow! panic!("Somehow we've loaded the pool twice!"); } // All operations after this point use the global pool. // We're the only place that could possibly have access to the pool right now. // So if this lock fails, cooked. let highest_known: u16 = if let Some(global_pool) = GLOBAL_POOL.get() { if let Ok(innards) = global_pool.try_lock() { innards.header.highest_known_disk } else { panic!("Locking the global pool immediately after creation failed!"); } } else { // ?????????? panic!("The global pool does not exist, even though we JUST made it?"); }; // Check if this is a brand new pool if highest_known == 0 { // This is a brand new pool, we need to initialize it. match Pool::initalize() { Ok(ok) => ok, Err(error) => { // Initializing the pool failed. This cannot continue. error!("Failed to load the pool."); error!("Reason: {error}"); error!("Fluster will now exit."); panic!("Failed to initalize pool! {error}"); } }; }; // All done shared_pool } /// Set up stuff for a brand new pool fn initalize_pool() -> Result<(), DriveError> { debug!("Doing first time pool setup..."); // Things a pool needs: // A second disk to start storing inodes on. // A root directory. // Lets get that second disk going // First we need to make a standard disk debug!("Creating the standard disk (disk 1)..."); let _ = add_disk::()?; // The root directory is set up on the disk side, so we're done. debug!("Finished first time pool setup."); Ok(()) } /// Add a new disk of Type to the pool. /// Takes the next available disk number. /// Returns the newly created disk of type T. fn add_disk() -> Result { let handle = NotifyTui::start_task(TaskType::CreateNewDisk, 2); debug!( "Attempting to add new disk to the pool of type: {}", std::any::type_name::() ); let highest_known: u16 = GLOBAL_POOL .get() .expect("Pool must exist at to add disks to it.") .try_lock() .expect("Cannot add disks to poisoned pool. Also single threaded, so should not block.") .header .highest_known_disk; let next_open_disk = highest_known + 1; // First, we need a blank disk in the drive. // For virtual disk reasons, we still need to pass in the disk number that // we wish to create. debug!("Getting a new blank disk..."); // We loop until there is a disk in the drive, just in case. // try 10 times let blank_disk: BlankDisk; let mut tries: u8 = 0; loop { if let Ok(disk) = FloppyDrive::get_blank_disk(next_open_disk){ blank_disk = disk; break }; // We need that blank mf. if tries == 10 { // shiet error!("Couldnt get a blank disk! cooked!"); return Err(DriveError::DriveEmpty); // I mean, good enough. } tries += 1; } NotifyTui::complete_task_step(&handle); // Now we need to create a disk to put in there from the supplied generic debug!("Bootstrapping the new disk..."); let bootstrapped = T::bootstrap(blank_disk.disk_file(), next_open_disk)?; NotifyTui::complete_task_step(&handle); // The disk has now bootstrapped itself, we are done here. // We already locked earlier, so this can't be poisoned, unless maybe making the disks also panicked? if let Ok(mut inner) = GLOBAL_POOL.get().expect("Pool has to be set up before we can make disks.").try_lock() { inner.header.highest_known_disk += 1; } else { // Poisoned again! We're probably in really bad shape. Just give up. // In theory this cant even get poisoned at this point due to bootstrapping happening // in the same thread but whatever, compiler doesn't know ig. panic!("Poisoned on disk bootstrapping!"); } debug!("Done adding new disk."); NotifyTui::finish_task(handle); Ok(bootstrapped) } /// Grabs the root inode block fn pool_get_root_directory() -> Result { // Root directory should always be at disk 1 block 2. We just assume that to be the case. // Why do we have a root inode that points to the root directory when its always in a static location? // Beats me, I forgot why I did that. let root_pointer: DiskPointer = DiskPointer { disk: 1, block: 2, }; // Get the root directory block let block_reader: RawBlock = CachedBlockIO::read_block(root_pointer)?; let block = DirectoryBlock::from_block(&block_reader); Ok(block) } /// Grabs the root inode location, duh fn pool_get_root_inode_location() -> InodeLocation { let pointer = DiskPointer { disk: 1, block: 1, }; InodeLocation::new(pointer, 0) } /// Constructs a directory item with info about the root. fn pool_get_root_directory_item() -> DirectoryItem { // The name of the root directory entry is always the delimiter. static DELIMITER: char = std::path::MAIN_SEPARATOR; DirectoryItem { flags: DirectoryItemFlags::IsDirectory, name_length: 0, name: DELIMITER.into(), location: pool_get_root_inode_location(), } } ================================================ FILE: src/pool/pool_actions/pool_struct.rs ================================================ // Did you know, if lightning struct a pool, everyone dies? // Imports use crate::pool::disk::pool_disk::block::header::header_struct::PoolDiskHeader; use lazy_static::lazy_static; use once_cell::sync::OnceCell; use std::sync::{Arc, Mutex}; // The global access to the pool. // It was either have a globally accessible pool, or put a reference to the pool in every method... No thanks. // Know a cleaner way? Make a pull request :D // This is done with a OnceCell so I dont have to spoof a fake pool into here before actually loading one up. lazy_static! { pub(crate) static ref GLOBAL_POOL: OnceCell>> = OnceCell::new(); } // Structs, Enums, Flags // All of the information we need about a pool to do our job. #[derive(Debug)] pub struct Pool { pub(crate) header: PoolDiskHeader, } ================================================ FILE: src/tui/layout.rs ================================================ // how da tui looks. use std::{ time::{ Duration, Instant } }; use ratatui::{ crossterm, layout::{ Alignment, Constraint, Direction, Layout, Rect }, style::{ Color, Style, Stylize }, text::Text, widgets::{ Block, Borders, Clear, Gauge, List }, Frame }; use tui_logger::{ TuiLoggerLevelOutput, TuiLoggerWidget }; use tui_textarea::{ Input, Key }; use crate::tui::{ prompts::TuiPrompt, state::FlusterTUIState, tasks::TaskInfo }; /// The TUI interface of fluster. Call methods on this to update the interface as often as you'd like. pub struct FlusterTUI<'a> { /// The actual internal state pub(super) state: FlusterTUIState, /// The last time the interface was updated pub(super) last_update: Instant, /// When fluster was first launched. pub(super) started: Instant, /// User prompt, if any. pub(super) user_prompt: Option>, } // Drawing time! impl FlusterTUI<'_> { /// Draw the TUI interface for Fluster /// /// Takes in a frame from the terminal. pub fn draw(&mut self, frame: &mut Frame) { // Split the window into sections let layout = Layout::default().margin(0).direction(Direction::Vertical).constraints([ // Progress bars Constraint::Ratio(3, 6), // Statistics Constraint::Ratio(2, 6), // Logging Constraint::Ratio(1, 6), ]).split(frame.area()); // Draw the progress bars if needed. let progress_area = layout[0]; if let Some(task) = &self.state.task { // Collect information about the sub tasks if there are any let tasks = task.get_tasks_info(); // Now we need to put all of the progress bars we'll be creating into a // block, then we will put the block into the layout at the end. let container_block: Block = Block::bordered().title("Running tasks:"); // Now based on the number of tasks we have, and how much room we get // from layout constraints, we will draw the progress bars. // If we have more bars than can be drawn into the space we have, we will // truncate the list from the front, effectively scrolling it upwards. // Gauges are 3 characters tall, top border, bar, and bottom border. // Get the amount of vertical space we have inside of our containing block let inner_height = &container_block.inner(progress_area).height; // Now, how many bars will fit? let max_tasks = inner_height / 3; // if we cant render any tasks, the window is too small. if max_tasks == 0 { return // just skip rendering entirely, lazy solution. } // Since we wouldn't be able to render them anyways, we can discard any tasks // we dont need. // Skip as many as we cant render. Saturating subtraction, because we can // have more room than tasks. let incoming_tasks = tasks.len(); let visible_tasks: Vec = tasks.into_iter().skip(incoming_tasks.saturating_sub(max_tasks.into())).collect(); // Render the container block, not sure why this needs to happen now, but im writing practically the entire // tui before testing it so... YOLO frame.render_widget(&container_block, progress_area); // Now we need constraints for each of the gauges to render into. let bar_layout = Layout::vertical( vec![Constraint::Length(3); visible_tasks.len()] ).split(container_block.inner(progress_area)); // Now for each of the task progress bars that are visible, we render // them into their constraints. // Man, UI logic is confusing. // Zipping so i can iterate in pairs // Also for fun factor, each bar will have a different color. // We'll cycle through them. // Would do rainbow, but orange is missing! let colors: Vec = vec![ Color::LightRed, Color::LightYellow, Color::LightCyan, Color::LightGreen, Color::LightMagenta ]; let mut color_index = 0; for (task, area) in visible_tasks.iter().zip(bar_layout.iter()) { // Make the gauge let gauge = Gauge::default() // Surround the gauge with a border, and put the title of the task on it .block( Block::default() .borders( Borders::ALL ) .title(task.name()) // Add a note for how long this has been running .title_bottom(format!("{} | Guesstimated time remaining: {}",task.time_passed(), task.time_remaining())) ) // Cycling colors .gauge_style(Style::default().fg(colors[color_index])) // Add the percentage .ratio(task.progress()); // Render the gauge into its area frame.render_widget(gauge, *area); // Increment color, looping around color_index = (color_index + 1) % colors.len(); } } else { // No tasks to display. let container_block: Block = Block::bordered().title("Idle, waiting for tasks..."); frame.render_widget(&container_block, progress_area); } // Done with progress bars // Now for the statistics. let statistics_area = layout[1]; // Fancy box for the statistics let stat_box: Block = Block::bordered().title("Statistics:"); // We will split the statistics window in two horizontally. // One side is for disk related information, the other is for cache. // Get the size of the stat box let stat_box_size = &stat_box.inner(statistics_area); // Split it in half let stat_layout = Layout::default().margin(0).direction(Direction::Horizontal).constraints([ // Left Constraint::Ratio(1, 2), // Right Constraint::Ratio(1, 2), ]).split(*stat_box_size); let stat_left: Rect = stat_layout[0]; let stat_right: Rect = stat_layout[1]; // Render the box frame.render_widget(&stat_box, statistics_area); // // Disk statistics // let disk_box: Block = Block::bordered().title("Disk:"); // We'll use a list for statistics. // Turn all the info into strings. let mut disk_strings: Vec = Vec::with_capacity(3); disk_strings.push(format!("Disks swapped: {}", self.state.disk_swap_count)); disk_strings.push(format!("Blocks read: {}", self.state.disk_blocks_read)); disk_strings.push(format!("Blocks written: {}", self.state.disk_blocks_written)); // Now a newline for spacing disk_strings.push("".to_string()); // And the current disk in the drive disk_strings.push(format!("Current disk in drive: {}", self.state.current_disk_in_drive)); disk_strings.push("".to_string()); // How long fluster has been running let seconds = self.started.elapsed().as_secs(); let minutes = seconds / 60; let hours = minutes / 60; disk_strings.push(format!("Uptime: [{:0>2}:{:0>2}:{:0>2}]", hours, minutes%60, seconds%60)); // Listify it. // We'll also surround it with our block let disk_list: List = List::new(disk_strings).block(disk_box); // Render that into the left side frame.render_widget(disk_list, stat_left); // // Disk statistics // let cache_box: Block = Block::bordered().title("Cache:"); let cache_box_size = cache_box.inner(stat_right); frame.render_widget(&cache_box, stat_right); // The cache is a bit more complicated, since we need to throw a gauge in there to // visualize the current cache hit rate. // So we split our space again. let cache_split = Layout::default().margin(0).direction(Direction::Vertical).constraints([ // 3 lines for each gauge Constraint::Min(3), Constraint::Min(3), // The rest of the room is for other stats. Constraint::Percentage(100), ]).split(cache_box_size); let hit_gauge_space = cache_split[0]; let pressure_gauge_space = cache_split[1]; let cache_text = cache_split[2]; // Make the hit gauge let hit_gauge: Gauge = Gauge::default() .block(Block::bordered() .title("Cache hit rate:")) .ratio(self.state.cache_hit_rate); // Render it in frame.render_widget(hit_gauge, hit_gauge_space); // Make the pressure gauge let hit_gauge: Gauge = Gauge::default() .block(Block::bordered() .title("Cache pressure:")) .ratio(self.state.cache_pressure); frame.render_widget(hit_gauge, pressure_gauge_space); // Now for the boring text. let mut cache_strings: Vec = Vec::with_capacity(3); cache_strings.push(format!("Swaps saved: {}", self.state.cache_swaps_saved)); cache_strings.push(format!("Reads cached: {}", self.state.cache_blocks_read)); cache_strings.push(format!("Writes cached: {}", self.state.cache_blocks_written)); cache_strings.push(format!("Cache flushes: {}", self.state.cache_flushes)); let cache_list: List = List::new(cache_strings); // Render that into the remaining stats space frame.render_widget(cache_list, cache_text); // Now finally, for the logs let logs_area = layout[2]; // There's a widget for this! let logs = TuiLoggerWidget::default() .block( Block::bordered().title("Logs:") ) .style_error(Style::default().red()) .style_warn(Style::default().yellow()) .style_info(Style::default().white()) .style_debug(Style::default().green()) .style_trace(Style::default().light_blue()) .output_level(Some(TuiLoggerLevelOutput::Abbreviated)); // Render it in! frame.render_widget(logs, logs_area); // Finally, handle pop-ups if needed. pop_up_handler(frame, &mut self.user_prompt); // Update the render time self.last_update = Instant::now(); // Done! } } // Display pop-ups and prompt for input. fn pop_up_handler(frame: &mut Frame, incoming_pop_up: &mut Option) { // Grab the popup if it exists. let pop_up = if let Some(up) = incoming_pop_up { up } else { // No pop up, nothing to do. return }; // We'll be putting a box in the middle of the screen. // Annoyingly we have to create a grid then just pull the middle out let row = Layout::default().direction(Direction::Vertical).constraints([ Constraint::Ratio(1, 10), Constraint::Ratio(8, 10), Constraint::Ratio(1, 10), ]).split(frame.area())[1]; let popup_layout = Layout::default().direction(Direction::Horizontal).constraints([ Constraint::Ratio(1, 10), Constraint::Ratio(8, 10), Constraint::Ratio(1, 10), ]).split(row)[1]; // Darken everything by dimming everything let frame_size = frame.area(); frame.buffer_mut().set_style(frame_size, Style::new().dim()); // Clear out the area that the pop-up is about to draw into frame.render_widget(Clear, popup_layout); // Now make the prompting window. let pop_up_block = Block::bordered() .title(pop_up.title.clone()) .title_alignment(Alignment::Center) // Blinking! .border_style(if pop_up.flash {Style::new().rapid_blink().red().on_black()} else {Style::new().white().on_black()}) .border_set(ratatui::symbols::border::FULL) // Make the inside of the pop-up white on cyan. .style(Style::new().on_cyan().white()); // Get the side of the inside of that let popup_inside = pop_up_block.inner(popup_layout); // Render the pop up window frame.render_widget(pop_up_block, popup_layout); // Now for the inside of the window. // Split into 2 parts, the top for the message, and the bottom for the input. let top_bottom = Layout::default().direction(Direction::Vertical).constraints([ Constraint::Ratio(7, 10), Constraint::Ratio(3, 10), ]).split(popup_inside); let top = top_bottom[0]; let bottom = top_bottom[1]; // Assemble the top half of the window, which is just text. let text = Text::from(pop_up.content.clone()).centered(); // Render it frame.render_widget(text, top); // And finally, for the part the user interacts with. // If the caller wants a string back, we'll return it. Otherwise we'll just discard it. // Render either the text box, or the continue prompt. // If in manual mode, dont add anything. if pop_up.expects_string { frame.render_widget(&pop_up.text_area, bottom); } else if !pop_up.manual_close { let prompt = Text::from("Press enter to continue.").centered(); frame.render_widget(prompt, bottom); } else { // Manual mode, widget } // Now we check for input, if it's enter, we will return the string if needed, and // remove the prompt. // But we don't block here, we just see if a key has been pressed. // If the polling fails, we just move on, and hope it gets hit next time. let event_waiting = crossterm::event::poll(Duration::from_millis(0)).unwrap_or(false); if !event_waiting { // No events, so we're done. return } // Handle the event. let read = if let Ok(event) = crossterm::event::read() { event } else { // Tried to read, got nothing. Hopefully we'll catch this on the next time around. return }; match read.into() { Input { key: Key::Esc | Key::Enter, .. } => { // The user has exited the prompt, we're done! // Unless the caller is manually handling things. if pop_up.manual_close { // Those keys have no power here. return } // To respond, we need to pull out the TuiPrompt and swap a None into its place. // Higher up we already checked that the pop-up exists, so we can safely extract it. let extracted = incoming_pop_up.take().expect("Guarded, already checked that the pop-up exists."); // We always send the string back, caller will toss it if they dont need it. // It would be really strange for the caller to have died, since they should just be blocking until hearing a result, so // if they dont eat the result, there's not really much we can do. let _ = extracted.callback.send(extracted.text_area.into_lines().concat()); }, input => { // User typed // This still updates the invisible text area, even if we // dont expect input. let _ = &pop_up.text_area.input(input); } } } ================================================ FILE: src/tui/mod.rs ================================================ pub(crate) mod layout; pub(crate) mod state; pub(crate) mod tui_struct; pub mod notify; pub(crate) mod tasks; pub(crate) mod prompts; ================================================ FILE: src/tui/notify.rs ================================================ // Notify the TUI about changes in Fluster! // This is the only place we lock the TUI state. use std::sync::Mutex; use lazy_static::lazy_static; use log::error; use crate::{filesystem::filesystem_struct::USE_TUI, tui::{layout::FlusterTUI, tasks::{ProgressableTask, TaskHandle, TaskType}}}; // Global TUI state lazy_static! { pub static ref TUI_MANAGER: Mutex> = Mutex::new(FlusterTUI::new()); } // We only run some logic if the TUI is enabled. // Plus this is a great excuse to learn macros! // This just sticks a return into the function if the TUI is enabled. macro_rules! skip_if_tui_disabled { () => { if let Some(got) = USE_TUI.get() { if !got { // TUI is not enabled. return } } else { // The flag is not currently set, we'll just do all of the logic behind the scenes just // in case. } }; } // Since we expect things to be single threaded when interacting with the TUI, we have a lot of // locks and expects, thus we will just abstract that out. If the lock fails, the task state would become // desynced, which would almost guarantee a crash. Thus we will exit out if that happens. // // Karen, because it gets the manager macro_rules! karen { () => { if let Ok(got) = TUI_MANAGER.lock() { got } else { // Manager is out, which mean's its poisoned due to a panic or such. // Not great. I really doubt we could recover from that. error!("Couldn't get the TUI manager, giving up!"); error!("{}", std::backtrace::Backtrace::force_capture()); panic!("TUI manager is poisoned!"); } }; } // Now we have a bunch of public functions for updating the TUI. // Its on a struct, just because I like how that looks. pub(crate) struct NotifyTui { // dummy } impl NotifyTui { // // Disk // /// A _real_ disk swap has occurred, you must provide the disk that is now in the drive. pub(crate) fn disk_swapped(new_disk: u16) { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.disk_swap_count += 1; manager.state.current_disk_in_drive = new_disk; } /// A block, or multiple blocks have been read from disk. pub(crate) fn block_read(number: u16) { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.disk_blocks_read += number as u64; } /// Block(s) has been written to disk. pub(crate) fn block_written(amount: u16) { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.disk_blocks_written += amount as u64; } // // Cache // /// The cache saved a swap. pub(crate) fn swap_saved() { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_swaps_saved += 1; } /// A tier of the cache was flushed to disk. pub(crate) fn cache_flushed() { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_flushes += 1; } /// A read was cached instead of read from disk. pub(crate) fn read_cached() { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_blocks_read += 1; } /// A write was cached instead of written to disk. pub(crate) fn write_cached() { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_blocks_written += 1; } /// Update the cache hit-rate pub(crate) fn set_cache_hit_rate(rate: f64) { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_hit_rate = rate; } /// Update the cache pressure pub(crate) fn set_cache_pressure(pressure: f64) { skip_if_tui_disabled!(); let mut manager = karen!(); manager.state.cache_pressure = pressure; } // // Task // /// Start a new task. /// /// You must keep the TaskHandle to be able to update this task. #[must_use] // Cant ignore the handle! pub(crate) fn start_task(task_type: TaskType, steps: u64) -> TaskHandle { // Return a dummy handle if TUI is disabled. if let Some(flag) = USE_TUI.get() { if !flag { return TaskHandle::new(); } } else { // USE_TUI is not set, this really should be set already... // If we give out a fake handle: // - Caller cancels/finishes a task, but now USE_TUI is set, causing // either nothing to happen, or canceling an unrelated task. BAD! // If we give out a TUI handle in non-TUI mode: // - Handle would be given out, added to the TUI state, now USE_TUI is set, // if its set to true, all is good. If its false, still okay, since it just will chill // in the tui manager without ever being cleaned up. Which is fine. // Thus we will give out a handle that is already marked as finished, so when the deconstruction on it runs, // It'll just drop silently. Hopefully? In theory we can only give out one handle at a time, so if this is the // only handle that's out, and the flag is not set, chances are this is the first handle _ever_ so ignoring it is fine, // because the TUI would just try to clean up nothing, which should finish. // Also before adding this if let some, there was an expect here, and i never saw that get hit, so chances are // this will never run anyways. let mut pre_finished = TaskHandle::new(); pre_finished.task_was_finished_or_canceled = true; return pre_finished; } // Create the task and make a new handle let new_task = ProgressableTask::new(task_type, steps); let handle: TaskHandle = TaskHandle::new(); let mut manager = karen!(); let state = &mut manager.state; // If we already have a task, append it if let Some(ref mut task) = state.task { task.add_sub_task(new_task); return handle; } // Currently don't have any tasks, add it directly. state.task = Some(new_task); handle } /// Complete one step of a task /// /// Handle required to ensure you actually have a task you're working on pub(crate) fn complete_task_step(_handle: &TaskHandle) { skip_if_tui_disabled!(); let mut manager = karen!(); if let Some(task) = manager.state.task.as_mut() { task.finish_steps(1); } else { // No task to work on! Out of sync! // Cooked for sure. error!("{}", std::backtrace::Backtrace::force_capture()); panic!("Task state desync! Expected task, got none!"); } } /// Complete multiple steps of a task. /// /// Handle required to ensure you actually have a task you're working on pub(crate) fn complete_multiple_task_steps(_handle: &TaskHandle, steps: u64) { skip_if_tui_disabled!(); let mut manager = karen!(); if let Some(task) = manager.state.task.as_mut() { task.finish_steps(steps); } else { // No task to work on! Out of sync! // Cooked for sure. error!("{}", std::backtrace::Backtrace::force_capture()); panic!("Task state desync! Expected task, got none!"); } } /// Add more steps to the current task. /// /// Handle required to ensure you actually have a task you're working on pub(crate) fn add_steps_to_task(_handle: &TaskHandle, steps: u64) { skip_if_tui_disabled!(); let mut manager = karen!(); if let Some(task) = manager.state.task.as_mut() { task.add_work(steps); } else { // No task to work on! Out of sync! // Cooked for sure. error!("{}", std::backtrace::Backtrace::force_capture()); panic!("Task state desync! Expected task, got none!"); } } /// Finish a task. /// /// Handle required to ensure you actually have a task you're working on pub(crate) fn finish_task(mut handle: TaskHandle) { skip_if_tui_disabled!(); let mut manager = karen!(); let stored_task = &mut manager .state .task; if let Some(took) = stored_task.take() { *stored_task = took.finish_task(); } else { // We have a handle, but no task? // I mean, there's nothing to finish, but removing nothing shouldn't be an issue. // Good enough? } // Update and drop handle by letting it fall out of scope. handle.task_was_finished_or_canceled = true; } /// Cancel a task. /// /// Handle required to ensure you actually have a task you're working on pub(crate) fn cancel_task(mut handle: TaskHandle) { skip_if_tui_disabled!(); let mut manager = karen!(); let stored_task = &mut manager .state .task; if let Some(took) = stored_task.take() { *stored_task = took.cancel_task(); } else { // We have a handle, but no task? // I mean, there's nothing to finish, but removing nothing shouldn't be an issue. // Good enough? } // Update and drop handle by letting it fall out of scope. handle.task_was_finished_or_canceled = true; } /// Forcibly cancel a task without a handle. /// Only used for dropping pub(super) fn force_cancel_task() { skip_if_tui_disabled!(); let mut manager = karen!(); let stored_task = &mut manager .state .task; // Only try to cancel if there's actually a task if let Some(task) = stored_task.take() { *stored_task = task.cancel_task(); } } } ================================================ FILE: src/tui/prompts.rs ================================================ // Gotta talk to people sometimes. use std::path::Path; use std::path::PathBuf; use std::time::Instant; use log::debug; use log::error; use log::info; use ratatui::style::{Style, Stylize}; use tui_textarea::TextArea; use crate::error_types::drive::DriveError; use crate::filesystem::filesystem_struct::FLOPPY_PATH; use crate::{filesystem::filesystem_struct::USE_TUI, tui::notify::TUI_MANAGER}; #[derive(Debug)] pub(crate) struct TuiPrompt<'a> { /// Title of the prompt pub(super) title: String, /// What the prompt is telling the user pub(super) content: String, /// Do we expect to get a string back from this prompt? pub(super) expects_string: bool, /// Where we send the response to. Even if the prompt /// doesn't require a response, we still use the oneshot to more /// easily block the caller of the prompt. /// /// I'm calling this a callback because i feel like it. pub(super) callback: oneshot::Sender, /// Should the window flash to get the user's attention? pub(super) flash: bool, /// The persistent text entry field, persists between /// frames so we don't have to extract input handling out /// to main.rs pub(super) text_area: tui_textarea::TextArea<'a>, /// Allow the caller to manually manipulate the pop-up, IE when enter is /// pressed, the layout code will not close the window. It's up to the caller to close /// the pop up by swapping in a none. pub(super) manual_close: bool, } // if the TUI is disabled, we still need to be able to prompt for input. impl TuiPrompt<'_> { /// Make a new prompt for pressing enter. /// /// This will block until the user presses enter. pub(crate) fn prompt_enter(title: String, content: String, flash: bool) { // We need the channel even if we arent getting a string back, since we wanna // block the caller thread without having to spin in a loop lockin stuff. let (response_tx, response_rx) = oneshot::channel(); // Assemble and start the prompt. let prompt = TuiPrompt { title, content, expects_string: false, callback: response_tx, flash, text_area: TextArea::default(), // Not actually used. manual_close: false }; if let Some(flag) = USE_TUI.get() { if !flag { // Tui is disabled. return disabled_prompt_enter(prompt); } } else { // USE_TUI is not set yet, it really should be at this point. // But since it isn't, we'll just fall back to disabled mode. return disabled_prompt_enter(prompt); } // Run the prompt loop { if let Ok(mut lock) = TUI_MANAGER.lock() { lock.user_prompt = Some(prompt); break } } // Wait for prompt to close let _ = response_rx.recv(); // All done. } /// Make a new prompt for text input. /// /// This will block until the user responds. pub(crate) fn prompt_input(title: String, content: String, flash: bool) -> String { // Get the channel for communicating the result of the prompt let (response_tx, response_rx) = oneshot::channel(); // Assemble and start the prompt. // green text box let mut text_area = TextArea::default(); text_area.set_style(Style::reset().on_black().green()); let prompt = TuiPrompt { title, content, expects_string: true, callback: response_tx, flash, text_area, manual_close: false }; if let Some(flag) = USE_TUI.get() { if !flag { // Legacy mode return disabled_prompt_input(prompt); } } else { // USE_TUI is not set yet, it really should be at this point. // But since it isn't, we'll just fall back to legacy prompting return disabled_prompt_input(prompt); } // Run the prompt loop { if let Ok(mut lock) = TUI_MANAGER.lock() { lock.user_prompt = Some(prompt); break } } // Wait for a response, and return it. // If we got no response for some reason, safest bet is to return nothing. response_rx.recv().unwrap_or_default() } /// Does not actually check what disk was swapped to, just waits for a swap to happen. /// /// Returns an error if we were unable to detect disk swaps. /// /// This function also assumes that the floppy drive is a block device, and checks agains /sys/ to deduce that. /// /// Will this work on MacOS? Probably not. Def wont work on windows. pub(crate) fn prompt_wait_for_disk_swap(title: String, content: String, flash: bool) -> Result<(), DriveError> { // We dont actually care about this response at all, nothing ever gets sent // I'm just too lazy to remove the requirement for it right now. let (response_tx, _response_rx) = oneshot::channel(); // Assemble let prompt = TuiPrompt { title, content, expects_string: false, callback: response_tx, flash, text_area: TextArea::default(), // Not actually used. manual_close: true }; if let Some(flag) = USE_TUI.get() { if !flag { // Legacy mode is not supported at all here, // since test cases should never hit the disk swap prompt. panic!("Cannot swap disks without a tui!"); } } else { // USE_TUI really must be set at this point, this is a dead path as far as i can tell unreachable!("Shouldn't be able to swap disks without ever setting USE_TUI"); } // Get the disk path. let disk_path = if let Ok(guard) = FLOPPY_PATH.try_lock() { guard.clone() } else { // Cant lock it, chances are its poisoned. // Return a retry, since the caller needs to clear the poison (troubleshooter does that) return Err(DriveError::Retry); }; // Now we put the prompt into the TUI, since if we put it in before checking the poison, // it would never close. loop { if let Ok(mut lock) = TUI_MANAGER.lock() { lock.user_prompt = Some(prompt); break } } // Time to wait for a disk swap. // Now we need to wait for the state of the drive to transition to being empty, then full again. // There's probably a smarter way to do this, but I'm not smarter than the compiler anyways. let mut disk_in_drive: bool = get_block_device_size(&disk_path)? > 0; let mut previous_state: bool = disk_in_drive; let mut changes: u8 = 0; // If the drive was already empty when we checked (weird) then we need to increment changes, since // we're waiting for the state to change once (disk inserted) instead of twice (removal, insertion) if !disk_in_drive { changes += 1; } // Now loop till the disk has been swapped. // We keep track of how long this has been going, if over 2 minutes pass with no input, we'll bail out, // just in case we get stuck here, since we'd need to escape out to get to the troubleshooter. let started_waiting: Instant = Instant::now(); while changes < 2 { // Check if there is a disk disk_in_drive = get_block_device_size(&disk_path)? > 0; // Did the state change? if disk_in_drive != previous_state { debug!("Drive state has changed."); // Change! changes += 1; previous_state = !previous_state; // Keep going, the while loop will end itself automagically. continue; } // No change, wait // half second pauses, as far as i can tell, the drive doesn't instantly know when it has a disk in it, // so that should be granular enough. std::thread::sleep(std::time::Duration::from_millis(500)); // If more than 2 minutes have passed, bail if started_waiting.elapsed().as_secs() > 120 { debug!("Disk swap did not occur within 2 minutes, bailing."); // New prompt on screen, this will replace the old prompt. let (timeout_response_tx, timeout_response_rx) = oneshot::channel(); let timeout_prompt = TuiPrompt { title: "Anybody home?".to_string(), content: "No disk was swapped within 2 minutes of requesting. Bailing out.".to_string(), expects_string: false, callback: timeout_response_tx, flash: false, text_area: TextArea::default(), manual_close: false }; // Send the timeout prompt loop { if let Ok(mut lock) = TUI_MANAGER.lock() { lock.user_prompt = Some(timeout_prompt); break } } // wait let _ = timeout_response_rx.recv().unwrap_or_default(); // Error out. return Err(DriveError::TakingTooLong) } }; // Disk has been swapped, remove the prompt loop { if let Ok(mut lock) = TUI_MANAGER.lock() { lock.user_prompt = None; break } } // All done. Ok(()) } } /// Abstraction for getting the size of a block device. /// /// Returns how many blocks the block device has. /// /// This assumes the mount point is `/dev/sd*`, if its not, we cant do anything. /// /// Yes this is hacky. I'm not happy about it either, but std::fs::metadata always returns `0` on /// block devices it seems, and I just need to get this working. /// /// We look in /sys/block/sd*/size to get the size. /// /// That file contains how many blocks are on that device total, and i see dmesg entries reporting /// a block size change whenever the disk is removed, so this should work. /// /// This also requires Fluster! to be ran as root/sudo lmao /// /// Returns an error if the path does not exist (ie the mount point is not there.) fn get_block_device_size(path: &Path) -> Result { // inspired by http://syhpoon.ca/posts/how-to-get-block-device-size-on-linux-with-rust/ // But I dont wanna use unsafe. // This code path also assumes that your floppy drives report a block size of 512. // Mine does at least. lol. // Make sure the path actually exists if !path.exists() { // Nope. // Just tell it to retry, which should hit the retry max count elsewhere and hit the troubleshooter up return Err(DriveError::Retry); } // Make sure the floppy path makes sense assert!(path.has_root(), "Floppy paths must be fully qualified."); // Must be in /dev/ assert!(path.parent() .expect("Floppy path must have have a parent, since its in /dev/") .to_str() .expect("path has to be utf8 compat") .contains("dev"), "Path to the floppy drive must have a parent." ); // Not gonna check if its a block device, its your problem if you dont point at one, and the expect should catch it. // Now we just need the end of the path. IE `/dev/sdf` -> `sdf` let block_device_name = path.file_name() .expect("There must be a file, cant use a folder as a floppy") .to_str() .expect("Should be valid UTF8"); // Now we look into /sys/block/ to get how many blocks are on that device let path_to_info = PathBuf::from(format!("/sys/block/{block_device_name}/size")); let blocks_count_string = std::fs::read_to_string(&path_to_info).expect("If we cant access this file, fluster is doomed to fail."); // Now parse that into a number let block_count: u64 = blocks_count_string.trim().parse().expect("The size file should not contain non-number info.\n"); // This is also a nice spot to make sure this is not bigger than a floppy drive. If it is, // chances are the user pointed at another disk and is gonna wipe their drive lmao, we'll save them. if block_count > 3000 { // A little wiggle room just in-case the drive mounts weird. // This is most likely not a floppy drive. error!("The provided block device that's supposed to be a floppy disk is too big."); error!("Chances are, you accidentally passed in a different drive."); error!("Fluster will now exit, as we don't want to accidentally wipe one of your drives."); error!("If you're trying to use another device as a floppy drive for some reason (unlikely), you're probably"); error!("smart enough to disable this check."); panic!("Floppy drive is too big! See logs!"); } // All done Ok(block_count) } // User input only works with the TUI enabled. fn disabled_prompt_enter(prompt: TuiPrompt) { info!("Skipping prompt..."); info!("Enter prompt: [{}]: {}", prompt.title, prompt.content); } fn disabled_prompt_input(_prompt: TuiPrompt) -> String { error!("You might not like TUI's, but this setting is secretly just for test cases."); panic!("You need to use the TUI to use fluster."); } ================================================ FILE: src/tui/state.rs ================================================ // Stuff for tracking the state of the TUI and Fluster. // Need to be careful in here not to lock at the same time // as Fluster locking other things. use crate::tui::tasks::ProgressableTask; /// Struct for holding information about Fluster's current state. pub(super) struct FlusterTUIState { // // Disk stats // // All stats are from the perspective of the current run. Fluster does not // store statistics on shutdown. /// How many times total the disk has been swapped pub(super) disk_swap_count: u64, /// The total number of blocks that have been read from the physical disk, note that /// this is seperate from cache reads, since we pub(super) disk_blocks_read: u64, /// The total number of blocks that have been written to the physical disk. pub(super) disk_blocks_written: u64, /// What disk is currently in the drive, is updated _after_ disk switches. pub(super) current_disk_in_drive: u16, // // Cache stats // /// The current hit rate of the cache (Only needs to be updated on /// disk swap.) pub(super) cache_hit_rate: f64, /// The current pressure of the cache, aka how full tier 0 is. pub(super) cache_pressure: f64, /// Number of times we went to read a block, but got it from /// the cache instead of hitting the disk pub(super) cache_blocks_read: u64, /// Number of times we went to write a block, but were able to /// temporarily store it in the cache. pub(super) cache_blocks_written: u64, /// Number of times we've flushed tier 0 of the cache to disk. pub(super) cache_flushes: u64, /// Number of times we avoided swapping disks by doing a cached operation pub(super) cache_swaps_saved: u64, // // Tasks // // Current in-progress task, if there is one. pub(super) task: Option, } impl FlusterTUIState { /// Brand new state, used for init pub(super) fn new() -> Self { Self { disk_swap_count: 0, disk_blocks_read: 0, disk_blocks_written: 0, cache_hit_rate: 0.0, cache_pressure: 0.0, cache_blocks_read: 0, cache_blocks_written: 0, cache_flushes: 0, cache_swaps_saved: 0, task: None, current_disk_in_drive: 420, // no idea yet. } } } ================================================ FILE: src/tui/tasks.rs ================================================ // Keeping track of what we're working on use std::time::Instant; use crate::tui::notify::NotifyTui; /// Progress of events. /// /// Updating an event's progress is an indirect action, you can say you've completed a "steps" of a /// task, or add more "steps" to the task. Thus you don't have to keep track of the percentages yourself, /// just if you need to add more work to your task, or if you've completed some of it. /// /// Tasks can have sub-tasks. /// /// Keep in mind that steps only increment, you cannot go backwards. /// /// All actions on a ProgressableTask implicitly apply to the final task in the chain of /// sub-tasks. IE, if you have a->b->c, calling finish_step() will affect c. #[derive(Debug)] pub(crate) struct ProgressableTask { /// What the task is task: TaskInfo, /// A sub task, if any. sub_task: Option> } /// Task information is stored in a second struct for ease of use, since /// everything besides sub-tasks can be Copy-ed. #[derive(Clone, Debug)] pub(super) struct TaskInfo { /// The type of task being performed. task_type: TaskType, /// How many "steps" of the task are needed to finish this task. steps: u64, /// How many "steps" have been completed so far. steps_finished: u64, /// When this task was started start_time: Instant, } /// Every kind of task that can indicate its progress. #[derive(Clone, Debug)] pub(crate) enum TaskType { DiskWriteBlock, DiskWriteLarge, WaitingForDriveSpinUp, DiskReadBlock, /// Includes the name of the file. FilesystemReadFile(String), // man strings are annoying, no more copy /// Includes the name of the file. FilesystemWriteFile(String), /// Includes the name of the file. FilesystemOpenFile(String), /// Includes the name of the file. FilesystemTruncateFile(String), /// Includes name of new file duh FilesystemCreateFile(String), /// Includes the name of the new directory. FilesystemMakeDirectory(String), /// Includes the name of directory to list. FilesystemReadDirectory(String), /// Includes the name of the directory that is about to be removed. FilesystemRemoveDirectory(String), /// Includes the name of the file that is kill. FilesystemDeleteFile(String), /// Includes the name of the file / folder. GetMetadata(String), GetSize, WipeDisk, RestoreDisk, FlushCurrentDisk, FlushTier, FileReadBytes, FileWriteBytes, /// Includes number of requested blocks. PoolAllocateBlocks(u16), /// Includes number of blocks being deallocated. DiskDeallocateBlocks(u16), CreateNewDisk, WriteCRC, /// Includes the name of the directory we're trying to open ChangingDirectory(String), ListingDirectory, /// Includes the name of the item we're looking for FindItemInDirectory(String), CreateDirectoryItem, } /// When we start a task, we are promising to finish it. We need a way to know /// if the task never finished. Thus we hand out a little struct that you need to /// hold onto. If the struct gets dropped, we assume the lowest task in the chain /// was canceled, or has finished. /// /// DO NOT implement clone or copy. pub(crate) struct TaskHandle { /// This is set if the the task this handle was dispatched for /// completed _or_ was canceled, indicating that cleanup is not needed. /// /// Defaults to false pub(super) task_was_finished_or_canceled: bool, } // // Implementations // impl Drop for TaskHandle { fn drop(&mut self) { // Are we done? if self.task_was_finished_or_canceled { // Cool! Don't need to do anything. } else { // The task was never finished or canceled, we must cancel it ourselves // We cant actual use ourselves here, since we need to pass in an owned // value that will be dropped. So we just make a new one. weird, i know NotifyTui::force_cancel_task() } } } impl TaskHandle { pub(super) fn new() -> Self { Self { task_was_finished_or_canceled: false, } } } impl TaskInfo { // yeah fn new(task_type: TaskType, steps: u64) -> TaskInfo { Self { task_type, steps, steps_finished: 0, start_time: Instant::now(), } } /// Get the string name of this task pub(super) fn name(&self) -> String { match &self.task_type { TaskType::WaitingForDriveSpinUp => "Waiting for floppy drive to spin up...".to_string(), TaskType::DiskWriteBlock => "Writing a block...".to_string(), TaskType::DiskReadBlock => "Reading a block...".to_string(), TaskType::WipeDisk => "Wiping disk...".to_string(), TaskType::DiskWriteLarge => "Writing several blocks...".to_string(), TaskType::CreateNewDisk => "Creating new disk...".to_string(), TaskType::WriteCRC => "Writing CRC to block...".to_string(), TaskType::GetSize => "Getting the size of an item...".to_string(), TaskType::ListingDirectory => "Listing a directory...".to_string(), TaskType::CreateDirectoryItem => "Creating new directory item...".to_string(), TaskType::FlushCurrentDisk => "Flushing current disk...".to_string(), TaskType::FlushTier => "Flushing a tier of cache...".to_string(), TaskType::FilesystemReadFile(name) => { format!("Reading from file \"{name}\"...") }, TaskType::FilesystemWriteFile(name) => { format!("Writing to file \"{name}\"...") }, TaskType::FilesystemTruncateFile(name) => { format!("Truncating \"{name}\"...") }, TaskType::PoolAllocateBlocks(number) => { format!("Allocating {number} blocks across disk pool...") }, TaskType::DiskDeallocateBlocks(number) => { format!("Freeing {number} blocks from current disk...") }, TaskType::GetMetadata(name) => { format!("Getting {name}'s metadata...") }, TaskType::ChangingDirectory(name) => { format!("Trying to open directory {name}...") }, TaskType::FindItemInDirectory(name) => { format!("Looking for {name} in current directory...") }, TaskType::FilesystemMakeDirectory(name) => { format!("Making new directory {name}...") }, TaskType::FilesystemOpenFile(name) => { format!("Opening file {name}...") }, TaskType::FilesystemCreateFile(name) => { format!("Creating file {name}...") }, TaskType::FilesystemReadDirectory(name) => { format!("Reading directory {name}...") }, TaskType::FilesystemRemoveDirectory(name) => { format!("Removing directory {name}...") }, TaskType::FilesystemDeleteFile(name) => { format!("Deleting file {name}...") }, TaskType::FileReadBytes => "Reading bytes from file...".to_string(), TaskType::FileWriteBytes => "Writing bytes to file...".to_string(), TaskType::RestoreDisk => "Restoring a disk from backup...".to_string(), } } /// Get a float for how finished the task is pub(super) fn progress(&self) -> f64 { self.steps_finished as f64 / self.steps as f64 } /// Returns a string `[hh:mm:ss]` of how long the task has been running pub(super) fn time_passed(&self) -> String { let elapsed_seconds = self.start_time.elapsed().as_secs(); format!("[{:0>2}:{:0>2}:{:0>2}]", elapsed_seconds/(60*60), (elapsed_seconds/60)%60, elapsed_seconds%60) } /// Returns a string `[hh:mm:ss]` which guesstimates how long the task is going to take /// /// Barely useful, but fun! pub(super) fn time_remaining(&self) -> String { let elapsed_seconds = self.start_time.elapsed().as_secs(); // Now based on how far we've come, estimate how much longer it'll take let mut estimated_seconds = if self.steps_finished > 0 { (elapsed_seconds as f64 * (self.steps as f64 / self.steps_finished as f64)) as u64 } else { // div zero moment, just return all nines lmao return "[99:99:99]".to_string(); }; estimated_seconds = estimated_seconds.saturating_sub(elapsed_seconds); format!("[{:0>2}:{:0>2}:{:0>2}]", estimated_seconds/(60*60), (estimated_seconds/60)%60, estimated_seconds%60) } } impl ProgressableTask { /// Create a new task. /// /// New tasks cannot have a sub task pre-attached. pub(super) fn new(task_type: TaskType, steps: u64) -> ProgressableTask { ProgressableTask { task: TaskInfo::new(task_type, steps), sub_task: None, } } /// Add more steps to a currently in-progress task. /// /// Updates how many steps _need_ to be taken, not how many _have_ been taken. pub(super) fn add_work(&mut self, steps_to_add: u64) { // Recurse if there are sub-tasks if let Some(sub_task) = &mut self.sub_task { return sub_task.add_work(steps_to_add) } self.task.steps += steps_to_add; } /// Indicate that a step has been completed. /// /// You can only finish one step at a time. pub(super) fn finish_steps(&mut self, steps: u64) { // Recurse if there are sub-tasks if let Some(sub_task) = &mut self.sub_task { return sub_task.finish_steps(steps) } self.task.steps_finished += steps; // You cannot finish more steps than you need to complete. assert!(self.task.steps_finished <= self.task.steps, "Tasks cannot complete more steps than they contain!"); } /// Add a sub-task. /// /// New task will be added to the last task in the chain. pub(super) fn add_sub_task(&mut self, new_sub_task: ProgressableTask) { // Recurse if there are sub-tasks if let Some(sub_task) = &mut self.sub_task { return sub_task.add_sub_task(new_sub_task) } self.sub_task = Some(Box::new(new_sub_task)); } /// Finishes the task (or subtask) currently in progress. /// /// Does not return the finished task, it only removes it. /// /// For a task to end, it must have a step count equal to its finished step count. IE /// you cannot finish a task without completing all of the work you promised you would complete. /// /// If you need to end a task due to a failure, you need to use cancel_task() /// /// Will return None if this is the final task in the chain. pub(super) fn finish_task(mut self) -> Option { // Recurse if there are sub-tasks if let Some(sub_task) = self.sub_task { let new_subtask = sub_task.finish_task(); self.sub_task = new_subtask.map(Box::new); return Some(self) } // This is the final task in the chain. // Make sure we finished all of the steps we set out to do. assert!(self.task.steps == self.task.steps_finished, "Task was not finished! {:#?}", self.task); // Remove the task. None } /// Cancels an in-progress task (or subtask). /// /// This ignores the requirement for tasks to have all of their steps completed. /// /// This is an explicit function, as you should do proper handling for canceling tasks. pub(super) fn cancel_task(mut self) -> Option { // Recurse if there are sub-tasks if let Some(sub_task) = self.sub_task { let new_subtask = sub_task.cancel_task(); self.sub_task = new_subtask.map(Box::new); return Some(self) } // This is the last task, remove it. None } /// Extract a Vec from this task. /// /// The head task is the first in the vec. pub(super) fn get_tasks_info(&self) -> Vec { // Recurse downwards, adding a copy of each task info as we go down. let mut tasks: Vec = Vec::new(); tasks.push(self.task.clone()); let mut sub = &self.sub_task; while let Some(new_sub) = sub.as_deref() { // Get the task out tasks.push(new_sub.task.clone()); // go deeper sub = &new_sub.sub_task; } // All tasks have been collected. tasks } } ================================================ FILE: src/tui/tui_struct.rs ================================================ // The struct that holds everything needed to render the tui use std::time::Instant; use crate::tui::{ layout::FlusterTUI, state::FlusterTUIState }; impl FlusterTUI<'_> { /// Brand new state, only for initialization. pub(super) fn new() -> Self { Self { state: FlusterTUIState::new(), last_update: Instant::now(), started: Instant::now(), user_prompt: None, } } } ================================================ FILE: tests/directory.rs ================================================ use std::{ error::Error, ffi::OsStr, thread, time::Duration }; use log::{ error, info }; use rand::{ rngs::ThreadRng, seq::SliceRandom }; // We want to see logs while testing. use test_log::test; use crate::test_common::test_mount_options; pub mod test_common; #[test] // Try creating a directory fn create_directory() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mounted_fs_path = mount_point.path().to_path_buf(); // wait for it to start... thread::sleep(Duration::from_millis(500)); // make a new dir let mut new_dir = mounted_fs_path.clone(); new_dir.push("testdir"); info!("Attempting to create a new directory..."); info!("It will go at `{}`.", new_dir.display()); let creation_result = std::fs::create_dir(new_dir); info!("Finished attempting creation."); // See if it's there info!("Checking if directory exists..."); let find_result = std::fs::read_dir(mounted_fs_path); let mut file_found: bool = false; if let Ok(items) = find_result { info!("Directory read succeeded, checking for the test dir..."); for i in items { // Check the results if let Ok(good) = i { // is this testdir? let item_name = good.file_name(); info!("found {}", item_name.display()); if item_name == "testdir" { // It exists! file_found = true; } // Ignore anything that isnt the directory we are looking for. } else { // Item was an error. uh oh let extracted_error = i.unwrap(); error!("Error directory item: {extracted_error:#?}"); } } } else { error!("Reading the directory failed."); let read_error = find_result.unwrap_err(); if let Some(src) = read_error.source() { error!("source: {src}"); } else { error!("source not marked."); } error!("error: {read_error}"); error!("kind: {}", read_error.kind()); } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Do the unwrap after unmounting, so we unmount even if it failed. // Did the creation fail? if let Err(error) = creation_result { // why? error!("Folder creation failed."); if let Some(src) = error.source() { error!("source: {src}"); } else { error!("source not marked."); } error!("error: {error}"); error!("kind: {}", error.kind()); panic!() } // Was the folder there? assert!(file_found, "Directory was not created, or did not show up when listed."); } #[test] // make dir, and some test items. list items. fn enter_and_list_directory() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mut mounted_fs_path = mount_point.path().to_path_buf(); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test folder mounted_fs_path.push("test"); std::fs::create_dir(&mounted_fs_path).unwrap(); // Make a directory to look for. let mut hidden_folder_path = mounted_fs_path.clone(); hidden_folder_path.push("hidden"); std::fs::create_dir(hidden_folder_path).unwrap(); // Read test folder let result = std::fs::read_dir(&mounted_fs_path).unwrap(); let mut found: bool = false; for i in result { if i.unwrap().file_name() == OsStr::new("hidden") { found = true; } } // cleanup thread::sleep(Duration::from_millis(500)); test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. assert!(found); } #[test] // Make sure the dot directory exists and refers to the parent when listing. fn check_for_dot() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mut mounted_fs_path = mount_point.path().to_path_buf(); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test folder mounted_fs_path.push("test"); std::fs::create_dir(&mounted_fs_path).unwrap(); // Make marker folder let mut mark_applier = mounted_fs_path.clone(); mark_applier.push("hello_everybody"); std::fs::create_dir(mark_applier).unwrap(); // Check listing the dot is the same as listing the parent. let mut dotted = mounted_fs_path.clone(); dotted.push("."); let dot_result = std::fs::read_dir(&dotted).unwrap(); let parent_result = std::fs::read_dir(&mounted_fs_path).unwrap(); // Do they match? // This is the grossest thing ever let mut any_different: bool = false; for i in dot_result.into_iter().zip(parent_result.into_iter()) { let (a, b) = i; if a.unwrap().path() != b.unwrap().path() { any_different = true; } } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. assert!(!any_different); } #[test] // Make a dir and move it to see if rename is working. fn move_empty_directory() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test"); let mut moved_dir = mount_point.path().to_path_buf(); moved_dir.push("moved"); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test folder std::fs::create_dir(&test_dir).unwrap(); // move it std::fs::rename(&test_dir, &moved_dir).unwrap(); // Does it exist? let moved: bool = std::fs::exists(&moved_dir).unwrap(); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it moved. assert!(moved); } #[test] // Try removing a directory fn directory_creation_and_removal() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test"); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test folder std::fs::create_dir(&test_dir).unwrap(); // move it let deleted = std::fs::remove_dir(&test_dir); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it was deleted assert!(deleted.is_ok()); } #[test] #[ignore = "Directory rename bug is too illusive, cant fix rn."] // Higher level renaming test fn rename_lots_of_items() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Hilariously we will just collect all the results we get so we can check // them all at the end after unmount. let mut results: Vec> = Vec::new(); // A lot of directories let mut directories: Vec = Vec::new(); for i in 0..1000 { let mut new_dir = mount_point.path().to_path_buf(); let dir_name = format!("dir_{i}"); new_dir.push(dir_name.clone()); directories.push(dir_name); let go = std::fs::create_dir(&new_dir); results.push(go); } // A lot of files let mut files: Vec = Vec::new(); for i in 0..1000 { let mut new_file = mount_point.path().to_path_buf(); let dir_name = format!("file_{i}.txt"); new_file.push(dir_name.clone()); files.push(dir_name); // Doesn't need any particular content. let go = std::fs::write(new_file, [1,2,3,4]); results.push(go); } // Shuffle for fun let mut all_items: Vec = Vec::new(); all_items.extend(directories); all_items.extend(files); let mut random: ThreadRng = rand::rng(); all_items.shuffle(&mut random); // How many do we have let number_made: usize = all_items.len(); // Go rename all of them for name in all_items { let new_name: String = format!("new_{name}"); let mut previous = mount_point.path().to_path_buf(); let mut new = mount_point.path().to_path_buf(); previous.push(name); new.push(new_name); let go = std::fs::rename(previous, new); results.push(go); } // Make sure the directory still contains the correct number of items. (ie we didn't duplicate anything.) // Listing also returns `.`, but rust (or linux?) drops it here if the directory is not empty. (in theory) // We also cant error out before the unmount, so we get a lil goofy here. let mut list_count: usize = 0; let mut list_names: Vec = Vec::new(); let list = std::fs::read_dir(&mount_point); if let Ok(listed) = list { // listing was ok for i in listed { if let Err(error) = i { // Something is amiss about this entry results.push(Err(error)); } else { // i is good let entry = i.unwrap(); // Skip if this is the `.` item if entry.file_name().to_string_lossy().into_owned() == "." { // skip continue; } list_count += 1; list_names.push(entry.file_name().to_string_lossy().into_owned()); } }; } else { // Listing failed results.push(Err(std::io::Error::other("Listing failed."))); } // Now, because somebody thought this was a good idea (it probably is overall, just not great for FS work) the // returned Result<> from filesystem operations seems to be holding references to the currently open filesystem. // So we have to extract everything. We will turn the errors into strings if they exist. // Loop over them, and only on the errors, make strings // Tried doing this with iter, but couldn't finish writing it bc rust analyzer kept crashing lmao let mut error_strings: Vec = Vec::new(); for result in &results { if let Err(error) = result { // Make a string from that let strung = error.to_string(); error_strings.push(strung); } } // now drop the old results, we cant hold them past the unmount drop(results); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Check all the results if !error_strings.is_empty() { // Something failed for string in error_strings { error!("{string}"); } panic!(); }; // Make sure there are no duplicates list_names.sort_unstable(); let old_list_len: usize = list_names.len(); list_names.dedup(); assert_eq!(old_list_len, list_names.len()); // Every item should contain the word `new` // Every item should only contain `new` once. for name in &list_names { // Check if it contains `new`, if it doesnt, we would like more info for testing if !name.contains("new") { // Not great. error!("`{name}` does not contain new."); // See if there is at least a new version of the directory. if list_names.contains(&format!("new_{name}")) { // It did! So the move did not remove the old directory, but did make the new one. error!("`new_{name}` does exist though.") } panic!("New issue") } assert!(name.matches("new").count() == 1, "`{name}` has new in it more than once."); } // Make sure we have the correct number of items. assert_eq!(number_made, list_count); } // Renaming burn in test #[test] #[ignore = "Slow."] fn rename_burn_in() { for _ in 0..1000 { rename_lots_of_items() } } ================================================ FILE: tests/file.rs ================================================ use std::{thread, time::Duration}; use rand::{rng, rngs::ThreadRng, Rng, RngCore}; // We want to see logs while testing. use test_log::test; use crate::test_common::test_mount_options; pub mod test_common; #[test] // Make a small file (512 bytes) fn make_file_small() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // Make some content let mut random: ThreadRng = rng(); let mut bytes: [u8; 512] = [0_u8; 512]; random.fill(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, bytes); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it was written assert!(write_result.is_ok()); } #[test] // Make a large file (8MB) fn make_file_large() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // The buffer needs to be big enough to store everything. // Arrays cannot be used here, you will overflow the stack. let data_size = 1024*1024*8; let mut bytes: Vec = vec![0u8; data_size as usize]; // Fill-er up! let mut random: ThreadRng = rng(); random.fill_bytes(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, bytes); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it was written assert!(write_result.is_ok()); } #[test] // Make and read file, make sure the contents match. (512 bytes) fn make_and_read_file_small() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // Make some content let mut random: ThreadRng = rng(); let mut bytes: [u8; 512] = [0_u8; 512]; random.fill(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, bytes); // Now read it back in let read_result = std::fs::read(&test_dir); // Does it match? let matched: bool; if let Ok(ref read) = read_result { matched = *read == bytes.to_vec(); } else { matched = false; } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it matched. assert!(write_result.is_ok()); assert!(read_result.is_ok()); assert!(matched); } #[test] // Make and read file, make sure the contents match. (8MB) fn make_and_read_file_large() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // The buffer needs to be big enough to store everything. // Arrays cannot be used here, you will overflow the stack. let data_size = 1024*1024*8; let mut bytes: Vec = vec![0u8; data_size as usize]; // Fill-er up! let mut random: ThreadRng = rng(); random.fill_bytes(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, &bytes); // Now read it back in let read_result = std::fs::read(&test_dir); // Does it match? let matched: bool; if let Ok(ref read) = read_result { matched = *read == bytes.to_vec(); } else { matched = false; } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it matched. assert!(write_result.is_ok()); assert!(read_result.is_ok()); assert!(matched); } #[test] // Make a file and rename it fn move_file() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); let mut moved_dir = mount_point.path().to_path_buf(); moved_dir.push("moved.txt"); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let content: [u8; 512] = [0_u8; 512]; let write_result = std::fs::write(&test_dir, content); // Rename it let rename_result = std::fs::rename(&test_dir, &moved_dir); // Does it exist? let moved: bool = std::fs::exists(&moved_dir).unwrap(); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it moved. assert!(write_result.is_ok()); assert!(rename_result.is_ok()); assert!(moved); } #[test] // Delete a file fn delete_file() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); let mut test_file = mount_point.path().to_path_buf(); test_file.push("test.txt"); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let content: [u8; 512] = [0_u8; 512]; let write_result = std::fs::write(&test_file, content); // Delete it let delete_result = std::fs::remove_file(&test_file); // Does it exist? let removed: bool = std::fs::exists(&test_file).unwrap(); // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Make sure it moved. assert!(write_result.is_ok()); assert!(delete_result.is_ok()); assert!(!removed); } #[test] // Make and read file, make sure the contents match, then do it again to the same file. (512 bytes, 1MB) fn update_file_small() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // The buffer needs to be big enough to store everything. // Arrays cannot be used here, you will overflow the stack. let data_size = 512; let mut bytes: Vec = vec![0u8; data_size as usize]; // Fill-er up! let mut random: ThreadRng = rng(); random.fill_bytes(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, &bytes); // Now read it back in let read_result = std::fs::read(&test_dir); // Does it match? let first_matched: bool; if let Ok(ref read) = read_result { first_matched = *read == bytes.to_vec(); } else { first_matched = false; } // // Now we will update the file // // New bytes will also be twice as large let mut new_bytes: Vec = vec![0u8; data_size as usize * 2]; // Fill-er up! random.fill_bytes(&mut new_bytes); // write again let second_write_result = std::fs::write(&test_dir, &new_bytes); // read again let second_read_result = std::fs::read(&test_dir); let second_matched: bool; if let Ok(ref read) = second_read_result { second_matched = *read == new_bytes.to_vec(); } else { second_matched = false; } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Did all that work? assert!(write_result.is_ok()); assert!(read_result.is_ok()); assert!(first_matched); assert!(second_write_result.is_ok()); assert!(second_read_result.is_ok()); assert_eq!(new_bytes.to_vec(), second_read_result.unwrap()); assert!(second_matched); } #[test] // Make and read file, make sure the contents match, then do it again to the same file. (8MB, 16MB) fn update_file_large() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let thread_mount_path = mount_point.path().to_path_buf(); let mount_options = test_mount_options(); // fs needs to be mounted in another thread bc it blocks let mount_thread = thread::spawn(move || { thread::sleep(Duration::from_millis(100)); // Pause to let the debugger see the thread // If we dont pause, breakpoints dont work. // This blocks until the unmount happens. fuse_mt::mount(fs, &thread_mount_path, &mount_options) }); // Test dir let mut test_dir = mount_point.path().to_path_buf(); test_dir.push("test.txt"); // The buffer needs to be big enough to store everything. // Arrays cannot be used here, you will overflow the stack. let data_size = 1024*1024*8; let mut bytes: Vec = vec![0u8; data_size as usize]; // Fill-er up! let mut random: ThreadRng = rng(); random.fill_bytes(&mut bytes); // wait for it to start... thread::sleep(Duration::from_millis(500)); // Make test file let write_result = std::fs::write(&test_dir, &bytes); // Now read it back in let read_result = std::fs::read(&test_dir); // Does it match? let first_matched: bool; if let Ok(ref read) = read_result { first_matched = *read == bytes.to_vec(); } else { first_matched = false; } // // Now we will update the file // // New bytes will also be twice as large let mut new_bytes: Vec = vec![0u8; data_size as usize * 2]; // Fill-er up! random.fill_bytes(&mut new_bytes); // write again let second_write_result = std::fs::write(&test_dir, &new_bytes); // read again let second_read_result = std::fs::read(&test_dir); let second_matched: bool; if let Ok(ref read) = second_read_result { second_matched = *read == new_bytes.to_vec(); } else { second_matched = false; } // cleanup test_common::unmount(mount_point.path().to_path_buf()); let unmount_result = mount_thread.join(); unmount_result.unwrap().unwrap(); // Unmounting the fs should not fail. // Did all that work? assert!(write_result.is_ok()); assert!(read_result.is_ok()); assert!(first_matched); assert!(second_write_result.is_ok()); assert!(second_read_result.is_ok()); assert!(second_matched); } ================================================ FILE: tests/mount_filesystem.rs ================================================ use std::{thread, time::Duration}; use test_log::test; // We want to see logs while testing. use crate::test_common::test_mount_options; pub mod test_common; #[test] // Try starting up the filesystem fn mount_filesystem() { let fs = test_common::start_filesystem(); let mount_point = test_common::get_actually_temp_dir(); let mount_path = mount_point.path().to_path_buf(); // fs needs to be mounted in another thread bc it blocks let mount_thread_result = thread::spawn(move || { // This blocks this thread until the unmount happens. fuse_mt::mount(fs, &mount_path, &test_mount_options()) }); // wait for it to start... thread::sleep(Duration::from_millis(1000)); // Immediately unmount. // The mounted fs makes/lives in a folder named `fluster_test`, but we just unmount everything // in the folder that containers `fluster_test` test_common::unmount(mount_point.path().to_path_buf()); // Make sure the mount actually happened. // Two unwraps, one for the join, one for the result of fuse_mf::mount mount_thread_result.join().unwrap().unwrap(); } ================================================ FILE: tests/start_filesystem.rs ================================================ use test_log::test; // We want to see logs while testing. pub mod test_common; #[test] // Try starting up the filesystem fn filesystem_starts() { test_common::start_filesystem(); } ================================================ FILE: tests/test_common.rs ================================================ use std::{ffi::OsStr, path::PathBuf}; use fluster_fs::filesystem::filesystem_struct::{FilesystemOptions, FlusterFS}; use fuse_mt::FuseMT; use log::{debug, info}; use tempfile::{TempDir, tempdir}; // // Helper functions // // Temporary directories for virtual disks pub fn get_new_temp_dir() -> TempDir { info!("Getting a persistent temp dir for testing..."); let mut dir = tempdir().unwrap(); dir.disable_cleanup(true); debug!( "Created a temp directory at {}, it will not be deleted on exit.", dir.path().to_string_lossy() ); dir } // Temporary directories for virtual disks pub fn get_actually_temp_dir() -> TempDir { info!("Getting a non-persistent temp dir for testing..."); tempdir().unwrap() } // Create a temporary filesystem, and returns the mt thing used to do the mount. pub fn start_filesystem() -> FuseMT { info!("Starting temp test filesystem..."); let temp_dir = get_new_temp_dir(); let floppy_drive: PathBuf = PathBuf::new(); // This is never read since we are using temporary disks. // Disable backups, since we don't use those in tests for obvious reasons. let fs_options = FilesystemOptions::new(Some(temp_dir.path().to_path_buf()), floppy_drive, Some(false), false); let started = FlusterFS::start(&fs_options); // MT thing that is actually used for mounting. // Zero threads for fully sync. fuse_mt::FuseMT::new(started, 0) } pub fn unmount(mount_point: PathBuf) { info!("Unmounting filesystem...."); let result = std::process::Command::new("fusermount") .arg("-u") .arg(mount_point) .status().unwrap(); assert!(result.success()); } pub fn test_mount_options() -> Vec<&'static OsStr> { [ // No spaces after `-o` or it does not work lol. OsStr::new("-onodev"), // Disable dev devices OsStr::new("-onoatime"), // No access times OsStr::new("-onosuid"), // Ignore file/folder permissions (lol) OsStr::new("-orw"), // Read/Write OsStr::new("-oexec"), // Files are executable OsStr::new("-osync"), // No async. OsStr::new("-odirsync"), // No async // Set the name of the mount point. This should create a // directory in the temp folder with this same name. OsStr::new("-ofsname=fluster_test"), ].to_vec() } ================================================ FILE: windows.md ================================================ This is my punishment for not using Linux in the year of the Linux desktop smh. Special thanks to [Chris Harringon](https://chris.harrington.mn/)! # How to use fluster on windows You cannot build fluster on windows due to the `fuser` crate linking to unix stuff in libc, even if you install a compatibility library like WinFsp. Trust me, I tried. You need to build/run from inside of WSL. # Step 0: Usb storage module The default WSL kernels do not have the usb_storage module enabled, we will need to build our own. Follow [this](https://chris.harrington.mn/project/2022/07/30/wsl2-usb-storage.html) up until the `Windows USB-IP` section. `sudo modprobe usb-storage` should not fail. You can check if it was actually loaded with `lsmod`. # Step 1: USB floppy passthrough. You will also need to pass through the floppy drive to WSL. This will not persist through a reboot, or restarting WSL. You cannot mount a floppy drive normally with `wsl --mount`, so we will have to pass through the USB device. This obviously assumes you have a USB floppy drive. If you have an internal drive you want to use, I cannot help you. Good luck! On windows you will need the `usbipd-win` package `winget install --interactive --exact dorssel.usbipd-win` Then find your drive with `usbipd list` For reference, my drive shows up like so: ``` > usbipd list Connected: BUSID VID:PID DEVICE STATE ... 6-11 0644:0000 TEAC USB Floppy Not shared ``` Now we will bind the floppy to WSL using the bus id from the above command. (This requires an elevated command prompt / powershell window) ``` usbipd bind --busid X-X ``` There will be no output. Now we attatch it to WSL ``` usbipd attach --wsl --busid X-X ``` You will see some information about it finding your WSL distribution, and probably hear a USB discconect sound. The floppy drive will also (probably) spin up, assuming you have a disk inserted. Double check that the floppy is there ``` $ lsblk sde 8:64 1 1.4M 0 disk ``` Take note of which `sdX` device it is. Mine happens to be `sde`. This will be the path that you use when starting Fluster. ***A grain of salt*** This is SO unstable, you might also have to disable, then uninstall the floppy drive in device manager first. If your disk is spinning forever without seeming to do anything, you need to wait it out. Pulling out the disk (at least on my drive) in that state makes USBIPD give up and unmount the drive. # Step 2.5: You need to allow `other user` mounting in fuse to let you access Fluster! from windows through WSL. `sudo nano /etc/fuse.conf` Uncomment `#user_allow_other` # Step 3: Building and running: Required dependancies (non exhaustive): - build-essential - libfuse3-dev - rust (duh) Open the Fluster! source code directory (the one that contains cargo.toml). Build with `cargo build --release` (You can also build with `--profile floppy` for a smaller binary. Using `upx --best` on it should make it fit on a floppy if you really wanna have fun with it. Currently shrinks to under 800kb!) Pick a folder to mount to, for this example I will mount in `~/mounted/`. Remember your floppy drive path (The SD one). For this example, my drive is at `sde`. ### YOU BETTER MAKE DAMN SURE YOU'RE PASSING IN THE FLOPPY DRIVE ### READ THIS Fluster! WILL overwrite data on whatever block device you pass it. If you do not know what "block device" even means, this is your final warning. Do NOT continue further. Since we will be writing directly to the floppy disk without a pre-existing filesystem (we are the file system!) we unfortunately need to escalate permissions. So we'll just run it at sudo. Great idea I know. There is probably a safer way to do this (Such as using udev rules), but I got bored reading the documentation. Run fluster: - Go to the output directory for the binary you built (./target/release/) Run it as sudo with: `sudo ./fluster_fs --block-device-path "/dev/sdX" --mount-point "/home/(your username here)/mounted/fluster"` If you don't want to run as sudo/root, you're smart. Smart enough to figure out another solution. Good luck! Be warned, this will consume the terminal window you started the program with. Try using `tmux` to split your window if needed. (Must be installed ofc, Google it.) You can also run with debug info by pre-pending `RUST_LOG=debug` to the command if you're a nerd. This does add some performance overhead, which would only really matter if you're using the secret `--use-virtual-disks` option.