Repository: abraunegg/onedrive Branch: master Commit: db42574a0195 Files: 91 Total size: 2.5 MB Directory structure: gitextract_uo92plxx/ ├── LICENSE ├── Makefile.in ├── aclocal.m4 ├── changelog.md ├── config ├── configure ├── configure.ac ├── contrib/ │ ├── completions/ │ │ ├── complete.bash │ │ ├── complete.fish │ │ └── complete.zsh │ ├── docker/ │ │ ├── Dockerfile │ │ ├── Dockerfile-alpine │ │ ├── Dockerfile-debian │ │ ├── entrypoint.sh │ │ └── hooks/ │ │ └── post_push │ ├── init.d/ │ │ ├── onedrive.init │ │ └── onedrive_service.sh │ ├── logrotate/ │ │ └── onedrive.logrotate │ ├── pacman/ │ │ └── PKGBUILD.in │ ├── spec/ │ │ └── onedrive.spec.in │ └── systemd/ │ ├── onedrive.service.in │ └── onedrive@.service.in ├── docs/ │ ├── advanced-usage.md │ ├── application-config-options.md │ ├── application-security.md │ ├── build-rpm-howto.md │ ├── business-shared-items.md │ ├── client-architecture.md │ ├── contributing.md │ ├── docker.md │ ├── install.md │ ├── known-issues.md │ ├── national-cloud-deployments.md │ ├── podman.md │ ├── privacy-policy.md │ ├── puml/ │ │ ├── applyPotentiallyChangedItem.puml │ │ ├── applyPotentiallyNewLocalItem.puml │ │ ├── client_side_filtering_processing_order.puml │ │ ├── client_side_filtering_rules.puml │ │ ├── client_use_of_libcurl.puml │ │ ├── code_functional_component_relationships.puml │ │ ├── conflict_handling_default.puml │ │ ├── conflict_handling_default_resync.puml │ │ ├── conflict_handling_local-first_default.puml │ │ ├── conflict_handling_local-first_resync.puml │ │ ├── database_schema.puml │ │ ├── default_sync_flow.puml │ │ ├── downloadFile.puml │ │ ├── high_level_operational_process.puml │ │ ├── is_item_in_sync.puml │ │ ├── local_first_sync_process.puml │ │ ├── main_activity_flows.puml │ │ ├── onedrive_linux_authentication.puml │ │ ├── onedrive_windows_ad_authentication.puml │ │ ├── onedrive_windows_authentication.puml │ │ ├── uploadFile.puml │ │ ├── uploadModifiedFile.puml │ │ └── webhooks.puml │ ├── server-side-filtering-limitations.md │ ├── sharepoint-libraries.md │ ├── terms-of-service.md │ ├── ubuntu-package-install.md │ ├── usage.md │ └── webhooks.md ├── install-sh ├── onedrive.1.in ├── readme.md ├── src/ │ ├── arsd/ │ │ ├── README.md │ │ └── cgi.d │ ├── clientSideFiltering.d │ ├── config.d │ ├── curlEngine.d │ ├── curlWebsockets.d │ ├── intune.d │ ├── itemdb.d │ ├── log.d │ ├── main.d │ ├── monitor.d │ ├── notifications/ │ │ ├── README │ │ ├── dnotify.d │ │ └── notify.d │ ├── onedrive.d │ ├── qxor.d │ ├── socketio.d │ ├── sqlite.d │ ├── sync.d │ ├── util.d │ ├── webhook.d │ └── xattr.d └── tests/ ├── bad-file-name.tar.xz └── makefiles.sh ================================================ FILE CONTENTS ================================================ ================================================ FILE: LICENSE ================================================ GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . ================================================ FILE: Makefile.in ================================================ package = @PACKAGE_NAME@ version = @PACKAGE_VERSION@ prefix = @prefix@ # we don't use @exec_prefix@ because it usually contains '${prefix}' literally # but we use @prefix@/bin/onedrive in the systemd unit files which are generated # from the configure script. # Thus, set exec_prefix unconditionally to prefix # Alternative approach would be add dep on sed, and do manual generation in the Makefile. # exec_prefix = @exec_prefix@ exec_prefix = @prefix@ datarootdir = @datarootdir@ datadir = @datadir@ srcdir = @srcdir@ bindir = @bindir@ mandir = @mandir@ sysconfdir = @sysconfdir@ docdir = $(datadir)/doc/$(package) VPATH = @srcdir@ INSTALL = @INSTALL@ # Icon install locations (system-wide hicolor theme) ICON_THEMEDIR = $(datadir)/icons/hicolor ICON_PLACES_DIR = $(ICON_THEMEDIR)/scalable/places ICON_SOURCE_SVG = contrib/images/onedrive.svg ICON_TARGET_SVG = onedrive.svg NOTIFICATIONS = @NOTIFICATIONS@ HAVE_SYSTEMD = @HAVE_SYSTEMD@ systemduserunitdir = @systemduserunitdir@ systemdsystemunitdir = @systemdsystemunitdir@ all_libs = @curl_LIBS@ @sqlite_LIBS@ @dbus_LIBS@ @notify_LIBS@ @bsd_inotify_LIBS@ @dynamic_linker_LIBS@ COMPLETIONS = @COMPLETIONS@ BASH_COMPLETION_DIR = @BASH_COMPLETION_DIR@ ZSH_COMPLETION_DIR = @ZSH_COMPLETION_DIR@ FISH_COMPLETION_DIR = @FISH_COMPLETION_DIR@ DEBUG = @DEBUG@ DC = @DC@ DCFLAGS = @DCFLAGS@ DEBUG_DCFLAGS = @DEBUG_DCFLAGS@ RELEASE_DCFLAGS = @RELEASE_DCFLAGS@ VERSION_DCFLAG = @VERSION_DCFLAG@ LINKER_DCFLAG = @LINKER_DCFLAG@ OUTPUT_DCFLAG = @OUTPUT_DCFLAG@ WERROR_DCFLAG = @WERROR_DCFLAG@ DCFLAGS += $(WERROR_DCFLAG) ifeq ($(DEBUG),yes) DCFLAGS += $(DEBUG_DCFLAGS) else DCFLAGS += $(RELEASE_DCFLAGS) endif ifeq ($(NOTIFICATIONS),yes) GUI_NOTIFICATIONS = $(addprefix $(VERSION_DCFLAG)=,NoPragma NoGdk Notifications) endif system_unit_files = contrib/systemd/onedrive@.service user_unit_files = contrib/systemd/onedrive.service DOCFILES = readme.md config LICENSE changelog.md docs/advanced-usage.md docs/application-config-options.md docs/application-security.md docs/business-shared-items.md docs/client-architecture.md docs/contributing.md docs/docker.md docs/install.md docs/national-cloud-deployments.md docs/podman.md docs/privacy-policy.md docs/sharepoint-libraries.md docs/terms-of-service.md docs/ubuntu-package-install.md docs/usage.md docs/known-issues.md docs/webhooks.md docs/server-side-filtering-limitations.md ifneq ("$(wildcard /etc/redhat-release)","") RHEL = $(shell cat /etc/redhat-release | grep -E "(Red Hat Enterprise Linux|CentOS|AlmaLinux)" | wc -l) RHEL_VERSION = $(shell rpm --eval "%{rhel}") else RHEL = 0 RHEL_VERSION = 0 endif SOURCES = \ src/main.d \ src/config.d \ src/log.d \ src/util.d \ src/qxor.d \ src/curlEngine.d \ src/onedrive.d \ src/webhook.d \ src/sync.d \ src/itemdb.d \ src/sqlite.d \ src/clientSideFiltering.d \ src/monitor.d \ src/arsd/cgi.d \ src/xattr.d \ src/intune.d \ src/socketio.d \ src/curlWebsockets.d ifeq ($(NOTIFICATIONS),yes) SOURCES += src/notifications/notify.d src/notifications/dnotify.d endif all: onedrive clean: rm -f onedrive onedrive.o version rm -rf autom4te.cache rm -f config.log config.status # Remove files generated via ./configure distclean: clean rm -f Makefile contrib/pacman/PKGBUILD contrib/spec/onedrive.spec onedrive.1 $(system_unit_files) $(user_unit_files) onedrive: $(SOURCES) if [ -f .git/HEAD ] ; then \ git describe --tags > version ; \ else \ echo $(version) > version ; \ fi $(DC) -J. $(GUI_NOTIFICATIONS) $(DCFLAGS) $^ $(addprefix $(LINKER_DCFLAG),$(all_libs)) $(OUTPUT_DCFLAG)$@ install: all mkdir -p $(DESTDIR)$(bindir) $(INSTALL) onedrive $(DESTDIR)$(bindir)/onedrive mkdir -p $(DESTDIR)$(mandir)/man1 $(INSTALL) -m 0644 onedrive.1 $(DESTDIR)$(mandir)/man1/onedrive.1 mkdir -p $(DESTDIR)$(sysconfdir)/logrotate.d $(INSTALL) -m 0644 contrib/logrotate/onedrive.logrotate $(DESTDIR)$(sysconfdir)/logrotate.d/onedrive mkdir -p $(DESTDIR)$(docdir) for file in $(DOCFILES); do \ $(INSTALL) -m 0644 $$file $(DESTDIR)$(docdir); \ done ifeq ($(HAVE_SYSTEMD),yes) mkdir -p $(DESTDIR)$(systemduserunitdir) mkdir -p $(DESTDIR)$(systemdsystemunitdir) ifeq ($(RHEL),1) $(INSTALL) -m 0644 $(system_unit_files) $(DESTDIR)$(systemdsystemunitdir) $(INSTALL) -m 0644 $(user_unit_files) $(DESTDIR)$(systemdsystemunitdir) else $(INSTALL) -m 0644 $(system_unit_files) $(DESTDIR)$(systemdsystemunitdir) $(INSTALL) -m 0644 $(user_unit_files) $(DESTDIR)$(systemduserunitdir) endif else ifeq ($(RHEL_VERSION),6) $(INSTALL) contrib/init.d/onedrive.init $(DESTDIR)/etc/init.d/onedrive $(INSTALL) contrib/init.d/onedrive_service.sh $(DESTDIR)$(bindir)/onedrive_service.sh endif endif ifeq ($(COMPLETIONS),yes) mkdir -p $(DESTDIR)$(ZSH_COMPLETION_DIR) $(INSTALL) -m 0644 contrib/completions/complete.zsh $(DESTDIR)$(ZSH_COMPLETION_DIR)/_onedrive mkdir -p $(DESTDIR)$(BASH_COMPLETION_DIR) $(INSTALL) -m 0644 contrib/completions/complete.bash $(DESTDIR)$(BASH_COMPLETION_DIR)/onedrive mkdir -p $(DESTDIR)$(FISH_COMPLETION_DIR) $(INSTALL) -m 0644 contrib/completions/complete.fish $(DESTDIR)$(FISH_COMPLETION_DIR)/onedrive.fish endif # --- OneDrive folder icon (hicolor) --- mkdir -p $(DESTDIR)$(ICON_PLACES_DIR) $(INSTALL) -m 0644 $(ICON_SOURCE_SVG) $(DESTDIR)$(ICON_PLACES_DIR)/$(ICON_TARGET_SVG) # Refresh icon cache only when installing to the live system (not during staged DESTDIR installs) # and only if the theme directory is a proper theme (has index.theme) if [ -z "$(DESTDIR)" ] && command -v gtk-update-icon-cache >/dev/null 2>&1 \ && [ -f "$(ICON_THEMEDIR)/index.theme" ]; then \ gtk-update-icon-cache -q "$(ICON_THEMEDIR)"; \ fi uninstall: rm -f $(DESTDIR)$(bindir)/onedrive rm -f $(DESTDIR)$(mandir)/man1/onedrive.1 rm -f $(DESTDIR)$(sysconfdir)/logrotate.d/onedrive ifeq ($(HAVE_SYSTEMD),yes) ifeq ($(RHEL),1) rm -f $(DESTDIR)$(systemdsystemunitdir)/onedrive*.service else rm -f $(DESTDIR)$(systemdsystemunitdir)/onedrive*.service rm -f $(DESTDIR)$(systemduserunitdir)/onedrive*.service endif else ifeq ($(RHEL_VERSION),6) rm -f $(DESTDIR)/etc/init.d/onedrive rm -f $(DESTDIR)$(bindir)/onedrive_service.sh endif endif for i in $(DOCFILES) ; do rm -f $(DESTDIR)$(docdir)/$$i ; done ifeq ($(COMPLETIONS),yes) rm -f $(DESTDIR)$(ZSH_COMPLETION_DIR)/_onedrive rm -f $(DESTDIR)$(BASH_COMPLETION_DIR)/onedrive rm -f $(DESTDIR)$(FISH_COMPLETION_DIR)/onedrive.fish endif # --- OneDrive folder icon (hicolor) --- rm -f $(DESTDIR)$(ICON_PLACES_DIR)/$(ICON_TARGET_SVG) # Refresh icon cache if removing from the live system and index.theme exists if [ -z "$(DESTDIR)" ] && command -v gtk-update-icon-cache >/dev/null 2>&1 \ && [ -f "$(ICON_THEMEDIR)/index.theme" ]; then \ gtk-update-icon-cache -q "$(ICON_THEMEDIR)"; \ fi ================================================ FILE: aclocal.m4 ================================================ # generated automatically by aclocal 1.16.1 -*- Autoconf -*- # Copyright (C) 1996-2018 Free Software Foundation, Inc. # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, # with or without modifications, as long as this notice is preserved. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY, to the extent permitted by law; without # even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. m4_ifndef([AC_CONFIG_MACRO_DIRS], [m4_defun([_AM_CONFIG_MACRO_DIRS], [])m4_defun([AC_CONFIG_MACRO_DIRS], [_AM_CONFIG_MACRO_DIRS($@)])]) dnl pkg.m4 - Macros to locate and utilise pkg-config. -*- Autoconf -*- dnl serial 11 (pkg-config-0.29) dnl dnl Copyright © 2004 Scott James Remnant . dnl Copyright © 2012-2015 Dan Nicholson dnl dnl This program is free software; you can redistribute it and/or modify dnl it under the terms of the GNU General Public License as published by dnl the Free Software Foundation; either version 2 of the License, or dnl (at your option) any later version. dnl dnl This program is distributed in the hope that it will be useful, but dnl WITHOUT ANY WARRANTY; without even the implied warranty of dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU dnl General Public License for more details. dnl dnl You should have received a copy of the GNU General Public License dnl along with this program; if not, write to the Free Software dnl Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA dnl 02111-1307, USA. dnl dnl As a special exception to the GNU General Public License, if you dnl distribute this file as part of a program that contains a dnl configuration script generated by Autoconf, you may include it under dnl the same distribution terms that you use for the rest of that dnl program. dnl PKG_PREREQ(MIN-VERSION) dnl ----------------------- dnl Since: 0.29 dnl dnl Verify that the version of the pkg-config macros are at least dnl MIN-VERSION. Unlike PKG_PROG_PKG_CONFIG, which checks the user's dnl installed version of pkg-config, this checks the developer's version dnl of pkg.m4 when generating configure. dnl dnl To ensure that this macro is defined, also add: dnl m4_ifndef([PKG_PREREQ], dnl [m4_fatal([must install pkg-config 0.29 or later before running autoconf/autogen])]) dnl dnl See the "Since" comment for each macro you use to see what version dnl of the macros you require. m4_defun([PKG_PREREQ], [m4_define([PKG_MACROS_VERSION], [0.29]) m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1, [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])]) ])dnl PKG_PREREQ dnl PKG_PROG_PKG_CONFIG([MIN-VERSION]) dnl ---------------------------------- dnl Since: 0.16 dnl dnl Search for the pkg-config tool and set the PKG_CONFIG variable to dnl first found in the path. Checks that the version of pkg-config found dnl is at least MIN-VERSION. If MIN-VERSION is not specified, 0.9.0 is dnl used since that's the first version where most current features of dnl pkg-config existed. AC_DEFUN([PKG_PROG_PKG_CONFIG], [m4_pattern_forbid([^_?PKG_[A-Z_]+$]) m4_pattern_allow([^PKG_CONFIG(_(PATH|LIBDIR|SYSROOT_DIR|ALLOW_SYSTEM_(CFLAGS|LIBS)))?$]) m4_pattern_allow([^PKG_CONFIG_(DISABLE_UNINSTALLED|TOP_BUILD_DIR|DEBUG_SPEW)$]) AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility]) AC_ARG_VAR([PKG_CONFIG_PATH], [directories to add to pkg-config's search path]) AC_ARG_VAR([PKG_CONFIG_LIBDIR], [path overriding pkg-config's built-in search path]) if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) fi if test -n "$PKG_CONFIG"; then _pkg_min_version=m4_default([$1], [0.9.0]) AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then AC_MSG_RESULT([yes]) else AC_MSG_RESULT([no]) PKG_CONFIG="" fi fi[]dnl ])dnl PKG_PROG_PKG_CONFIG dnl PKG_CHECK_EXISTS(MODULES, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) dnl ------------------------------------------------------------------- dnl Since: 0.18 dnl dnl Check to see whether a particular set of modules exists. Similar to dnl PKG_CHECK_MODULES(), but does not set variables or print errors. dnl dnl Please remember that m4 expands AC_REQUIRE([PKG_PROG_PKG_CONFIG]) dnl only at the first occurence in configure.ac, so if the first place dnl it's called might be skipped (such as if it is within an "if", you dnl have to call PKG_CHECK_EXISTS manually AC_DEFUN([PKG_CHECK_EXISTS], [AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl if test -n "$PKG_CONFIG" && \ AC_RUN_LOG([$PKG_CONFIG --exists --print-errors "$1"]); then m4_default([$2], [:]) m4_ifvaln([$3], [else $3])dnl fi]) dnl _PKG_CONFIG([VARIABLE], [COMMAND], [MODULES]) dnl --------------------------------------------- dnl Internal wrapper calling pkg-config via PKG_CONFIG and setting dnl pkg_failed based on the result. m4_define([_PKG_CONFIG], [if test -n "$$1"; then pkg_cv_[]$1="$$1" elif test -n "$PKG_CONFIG"; then PKG_CHECK_EXISTS([$3], [pkg_cv_[]$1=`$PKG_CONFIG --[]$2 "$3" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes ], [pkg_failed=yes]) else pkg_failed=untried fi[]dnl ])dnl _PKG_CONFIG dnl _PKG_SHORT_ERRORS_SUPPORTED dnl --------------------------- dnl Internal check to see if pkg-config supports short errors. AC_DEFUN([_PKG_SHORT_ERRORS_SUPPORTED], [AC_REQUIRE([PKG_PROG_PKG_CONFIG]) if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then _pkg_short_errors_supported=yes else _pkg_short_errors_supported=no fi[]dnl ])dnl _PKG_SHORT_ERRORS_SUPPORTED dnl PKG_CHECK_MODULES(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], dnl [ACTION-IF-NOT-FOUND]) dnl -------------------------------------------------------------- dnl Since: 0.4.0 dnl dnl Note that if there is a possibility the first call to dnl PKG_CHECK_MODULES might not happen, you should be sure to include an dnl explicit call to PKG_PROG_PKG_CONFIG in your configure.ac AC_DEFUN([PKG_CHECK_MODULES], [AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl pkg_failed=no AC_MSG_CHECKING([for $1]) _PKG_CONFIG([$1][_CFLAGS], [cflags], [$2]) _PKG_CONFIG([$1][_LIBS], [libs], [$2]) m4_define([_PKG_TEXT], [Alternatively, you may set the environment variables $1[]_CFLAGS and $1[]_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details.]) if test $pkg_failed = yes; then AC_MSG_RESULT([no]) _PKG_SHORT_ERRORS_SUPPORTED if test $_pkg_short_errors_supported = yes; then $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1` else $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1` fi # Put the nasty error message in config.log where it belongs echo "$$1[]_PKG_ERRORS" >&AS_MESSAGE_LOG_FD m4_default([$4], [AC_MSG_ERROR( [Package requirements ($2) were not met: $$1_PKG_ERRORS Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. _PKG_TEXT])[]dnl ]) elif test $pkg_failed = untried; then AC_MSG_RESULT([no]) m4_default([$4], [AC_MSG_FAILURE( [The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config. _PKG_TEXT To get pkg-config, see .])[]dnl ]) else $1[]_CFLAGS=$pkg_cv_[]$1[]_CFLAGS $1[]_LIBS=$pkg_cv_[]$1[]_LIBS AC_MSG_RESULT([yes]) $3 fi[]dnl ])dnl PKG_CHECK_MODULES dnl PKG_CHECK_MODULES_STATIC(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], dnl [ACTION-IF-NOT-FOUND]) dnl --------------------------------------------------------------------- dnl Since: 0.29 dnl dnl Checks for existence of MODULES and gathers its build flags with dnl static libraries enabled. Sets VARIABLE-PREFIX_CFLAGS from --cflags dnl and VARIABLE-PREFIX_LIBS from --libs. dnl dnl Note that if there is a possibility the first call to dnl PKG_CHECK_MODULES_STATIC might not happen, you should be sure to dnl include an explicit call to PKG_PROG_PKG_CONFIG in your dnl configure.ac. AC_DEFUN([PKG_CHECK_MODULES_STATIC], [AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl _save_PKG_CONFIG=$PKG_CONFIG PKG_CONFIG="$PKG_CONFIG --static" PKG_CHECK_MODULES($@) PKG_CONFIG=$_save_PKG_CONFIG[]dnl ])dnl PKG_CHECK_MODULES_STATIC dnl PKG_INSTALLDIR([DIRECTORY]) dnl ------------------------- dnl Since: 0.27 dnl dnl Substitutes the variable pkgconfigdir as the location where a module dnl should install pkg-config .pc files. By default the directory is dnl $libdir/pkgconfig, but the default can be changed by passing dnl DIRECTORY. The user can override through the --with-pkgconfigdir dnl parameter. AC_DEFUN([PKG_INSTALLDIR], [m4_pushdef([pkg_default], [m4_default([$1], ['${libdir}/pkgconfig'])]) m4_pushdef([pkg_description], [pkg-config installation directory @<:@]pkg_default[@:>@]) AC_ARG_WITH([pkgconfigdir], [AS_HELP_STRING([--with-pkgconfigdir], pkg_description)],, [with_pkgconfigdir=]pkg_default) AC_SUBST([pkgconfigdir], [$with_pkgconfigdir]) m4_popdef([pkg_default]) m4_popdef([pkg_description]) ])dnl PKG_INSTALLDIR dnl PKG_NOARCH_INSTALLDIR([DIRECTORY]) dnl -------------------------------- dnl Since: 0.27 dnl dnl Substitutes the variable noarch_pkgconfigdir as the location where a dnl module should install arch-independent pkg-config .pc files. By dnl default the directory is $datadir/pkgconfig, but the default can be dnl changed by passing DIRECTORY. The user can override through the dnl --with-noarch-pkgconfigdir parameter. AC_DEFUN([PKG_NOARCH_INSTALLDIR], [m4_pushdef([pkg_default], [m4_default([$1], ['${datadir}/pkgconfig'])]) m4_pushdef([pkg_description], [pkg-config arch-independent installation directory @<:@]pkg_default[@:>@]) AC_ARG_WITH([noarch-pkgconfigdir], [AS_HELP_STRING([--with-noarch-pkgconfigdir], pkg_description)],, [with_noarch_pkgconfigdir=]pkg_default) AC_SUBST([noarch_pkgconfigdir], [$with_noarch_pkgconfigdir]) m4_popdef([pkg_default]) m4_popdef([pkg_description]) ])dnl PKG_NOARCH_INSTALLDIR dnl PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE, dnl [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) dnl ------------------------------------------- dnl Since: 0.28 dnl dnl Retrieves the value of the pkg-config variable for the given module. AC_DEFUN([PKG_CHECK_VAR], [AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl _PKG_CONFIG([$1], [variable="][$3]["], [$2]) AS_VAR_COPY([$1], [pkg_cv_][$1]) AS_VAR_IF([$1], [""], [$5], [$4])dnl ])dnl PKG_CHECK_VAR ================================================ FILE: changelog.md ================================================ # Changelog The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## 2.5.10 - 2026-01-30 ### Added * Implement Feature Request: Add configuration option 'disable_version_check' (#3530) * Implement Feature Request: Add automatic debug logging output redaction (#3549) ### Changed * Improve --resync warning prompt for clarity and safer operation (#3562) * Updated Dockerfiles to support newer distributions and associated components (#3565) * FreeBSD: Select inotify type (libc or libnotify) based on FreeBSD version (#3579) * Update that --force and --force-sync cannot be used with --resync (#3593) ### Fixed * Fix Bug: Fix timestamp and hash evaluation to avoid unnecessary file version creation online (#3526) * Fix Bug: Fix that websocket do not work with Sharepoint libraries (#3533) * Fix Bug: Fix that large files fail to download due operational timeout being exceeded (#3541) * Fix Bug: Fix hash functions read efficiency to support 'on-demand' development work (#3544) * Fix Bug: Fix that safeBackup crashes when attempting backing up a non-existent local path (#3545) * Fix Bug: Fix to that the application only performs safeBackup() on deleted items only when a hash change is detected (#3546) * Fix Bug: Prevent mis-configuration where 'recycle_bin_path' is inside 'sync_dir' (#3552) * Fix Bug: Harden logging initialisation: fall back to home directory when log_dir is not writeable (#3555) * Fix Bug: Ensure mkdirRecurse() is correctly wrapped in try block (#3566) * Fix Bug: Fix that 'remove_source_files' does not remove the source file when the file already exists in OneDrive (#3572) * Fix Bug: Enhance displayFileSystemErrorMessage() to include details of the actual path (#3574) * Fix Bug: Enhance downloadFileItem() to ensure greater clarity on download failures (#3575) * Fix Bug: Prevent malformed 'skip_dir' / 'skip_file' rules when using multiple config entries (#3576) * Fix Bug: Detect and prevent 'skip_dir' / 'skip_file' rules shadowing 'sync_list' inclusions (#3577) * Fix Bug: Fix 'skip_dir' and 'skip_file' shadow detection for rooted 'sync_list' paths (#3578) * Fix Bug: Fix 'skip_dir' directory exclusion by normalising input paths before matching (#3580) * Fix Bug: Fix testInternetReachability() function to ensure same curl options used in a consistent manner (#3581) * Fix Bug: Fix performPermanentDelete() to ensure zero content length is set (#3585) * Fix Bug: Fix safeRemove() to retry on EINTR / EBUSY filesystem responses to support 'on-demand' development work (#3586) * Fix Bug: Fix safeRename() to retry on EINTR / EBUSY filesystem responses to support 'on-demand' development work (#3587) * Fix Bug: Fix safeBackup() to retry on EINTR / EBUSY filesystem responses to support 'on-demand' development work (#3589) * Fix Bug: Fix WebSocket reconnect cleanup to prevent GC finalisation crash (#3582) * Fix Bug: Fix setLocalPathTimestamp() to retry on EINTR / EBUSY filesystem responses to support 'on-demand' development work (#3591) * Fix Bug: Fix incorrect handling of failed safeRename() operations to support 'on-demand' development work (#3592) * Fix Bug: Fix Docker entrypoint handling for non-root --user execution (#3602) * Fix Bug: Fix getRemainingFreeSpaceOnline() and correctly handle zero data traversal events for quota tracking (#3618) * Fix Bug: Fix getRemainingFreeSpaceOnline() for Business and SharePoint Accounts (#3621) * Fix Bug: Fix OAuth authorisation code parsing and encoding during token redemption (#3625) * Fix Bug: Fix Graph search(q=…) escaping for apostrophes (#3624) * Fix Bug: Fix handling of 204 No Content responses for Microsoft Graph PATCH requests (#3620) ### Updated * Updated completion files to align to application functionality * Updated documentation ## 2.5.9 - 2025-11-06 ### Fixed * Fix Bug: Fix very high CPU & memory utilisation with 2.5.8 when using --upload-only (#3515) (CRITICAL BUGFIX) * Fix Bug: Unexpected deletion of empty nested folders during first sync with 'sync_list' and --resync (#3513) (CRITICAL BUGFIX) ### Updated * Updated documentation ## 2.5.8 - 2025-11-05 ### Added * Implement Feature Request: Add that dotfiles in sync_list should be synced even when skip_dotfiles = "true" (#3456) * Implement Feature Request: Add websocket notification support (#3413) * Implement Feature Request: Add --download-file feature (#3459) * Implement Feature Request: Add option to remove source folders when using --upload-only --remove-source-files (#3473) * Implement Feature Request: Add support for AlmaLinux (#3485) * Implement Feature Request: Add ONEDRIVE_THREADS Docker option (#3494) * Implement Feature Request: Implement Desktop Manager Integration for GNOME and KDE (#3500) ### Changed * Changed how the file path is computed when there are 'skip_dir' entries to be consistent (#3484) * Changed checkJSONAgainstClientSideFiltering() to avoid multiple calls to computeItemPath() (#3489) ### Fixed * Fix Bug: Ensure driveId target is cached for modified file uploads (#3454) * Fix Bug: Ensure that 'use_intune_sso' and 'use_device_auth' cannot be used together (#3453) * Fix Bug: Force DNS Timeout when forcing a libcurl fresh connection (#3468) * Fix Bug: Fix WebSocket connection failure on libcurl 8.12.x by forcing HTTP/1.1 and disabling ALPN/NPN (#3482) * Fix Bug: Fix application crash after deleting file locally (#3481) * Fix Bug: Fix missing user information when syncing shared files (#3483) * Fix Bug: Fix Shared Folder data being deleted due to 'skip_dir' entry of '.*' (#3476) * Fix Bug: Fix that if using 'sync_list' only add new JSON items early to allow applyPotentiallyChangedItem() to operate as expected (#3505) * Fix Bug: When using --dry-run use tracked renamed directories to avoid falsely indicating local data is new and uploading as new data (#3503) * Fix Bug: Fix the fetching of maximum open files to be more POSIX compliant (#3508) * Fix Bug: Fix the Handling of WebSocket 'echo' from a local change (#3509) ### Updated * Updated documentation ## 2.5.7 - 2025-09-23 ### Added * Implement Feature Request: Show GUI notification when sync suspends due to 'large delete' threshold (#3388) * Implement Feature Request: Implement resumable downloads (#3354) ### Changed * Removed the auto configuration of using a larger fragment size (#3370) * Removed the OpenSSL Test (#3420) ### Fixed * Fix Bug: Catch unhandled OneDriveError exception due to libcurl failing to access the system CA certificate bundle (#3322) * Fix Bug: 'items-dryrun.sqlite3' gets erroneously created when running a 'no sync' operation (#3325) * Fix Bug: Handle online folder deletion|creation with same name that causes 'id' to change (#3332) * Fix Bug: Reduce I/O pressure on SQLite DB Operations (#3334) * Fix Bug: Handle a 409 online folder creation response with a re-query of the API (#3335) * Fix Bug: Fix systemd issue with ExecStartPre statement to be more OS independent (#3348) * Fix Bug: When using --upload-only do not try and update the local file timestamp post upload (#3349) * Fix Bug: Add missing 'config' options to --display-config (#3353) * Fix Bug: Fix that a failed file download can lead to online deletion (#3351) * Fix Bug: Update searchDriveItemForFile() to handle specific 404 response when file cannot be found (#3365) * Fix Bug: Fix that resync state remains true post first successful full sync (#3368) * Fix Bug: Fix that long running big upload (250GB+) fails because of an expired access token (#3361) * Fix Bug: Handle inconsistent OneDrive Personal driveId casing across multiple Microsoft Graph API Endpoints (#3347) * Fix Bug: Update Microsoft OneNote handling for 'OneNote_RecycleBin' objects (#3350) * Fix Bug: Handle invalid JSON response when querying parental details (#3379) * Fix Bug: Fix foreign key issue when performing a --resync due to a missed conversion of driveId to lowercase values and path is covered by 'sync_list' entries (#3383) * Fix Bug: Ensure 'sync_list' inclusion rules are correctly evaluated (#3381) * Fix Bug: Fix issue of trying to create the root folder online (#3403) * Fix Bug: Fix resumable downloads so that the curl engine offset point is reset post successful download (#3406) * Fix Bug: Fix application crash when a file is created and deleted quickly (#3405) * Fix Bug: Fix the support of relocated shared folders for OneDrive Personal (#3411) * Fix Bug: Fix infinite loop after a failed network connection due to changed curl messaging (#3412) * Fix Bug: Fix computePath() to track the parental path anchor when a Shared Folder is relocated with a deeper path (#3417) * Fix Bug: Fix SharePoint Shared Library DB Tie creation (#3419) * Fix Bug: Update safeBackup() function to ensure that the 'safeBackup' path addition is only added once and ignore directories (#3445) ### Updated * Updated OAuth2 Interactive Authorisation Flow prompts to remove any ambiguity on what actions a user needs to take (#3323) * Updated onedrive.spec.in to correct missing dependencies (#3329) * Updated minimum compiler version details (#3330) * Updated documentation and function for how 'threads' is used (#3352) * Updated logging output for upsert() function (#3333) * Updated curl 8.13.x and 8.14.0 to known bad curl versions (#3356) * Updated logging output when processing online deletion events (#3373) * Updated logging output and use of grandparent identifiers when using --dry-run (#3377) * Updated GitHub Action versions for building Docker containers (#3378) * Updated how the ETA values are calculated to avoid negative values (#3386) * Updated Debian Dockerfile to use upstream gosu (#3402) * Updated Debian Dockerfile to use 'bookworm' (#3402) * Updated documentation ## 2.5.6 - 2025-06-05 ### Added * Enhancement: Add gdc support to enable Gentoo compilation * Enhancement: Add a notification to user regarding number of objects received from OneDrive API * Enhancement: Update 'skip_file' documentation and option validation * Enhancement: Add a new configuration option 'force_session_upload' to support editors and applications using atomic save operations * Enhancement: Added 2 functions to check for the presence of required remoteItem elements to create a Shared Folder DB entries * Implement Feature Request: Add local recycle bin or trash folder option * Implement Feature Request: Add configurable upload delay to support Obsidian * Implement Feature Request: Add validation of bools in config file * Implement Feature Request: Add native support for authentication via Intune dbus interface * Implement Feature Request: Implement OAuth2 Device Authorisation Flow ### Changed * Change logging output level for JSON elements that contain URL encoding * Change 'configure.ac' to use a static date value as Debian 'reproducible' build process forces a future date to rebuild any code to determine reproducibility ### Fixed * Fix Regression: Fixed regression in handling Microsoft OneNote package folders being created in error * Fix Regression: Fix OneNote file MimeType detection * Fix Regression: Fix supporting Personal Shared Folders that have been renamed * Fix Bug: Correct the logging output for 'skip_file' exclusions * Fix Bug: Validate raw JSON from Graph API for 15 character driveId API bug * Fix Bug: Fix JSON exception on webhook subscription renewal due to 308 redirect * Fix Bug: Update 'sync_list' line parsing to correctly escape characters for regex parsing * Fix Bug: Fix that an empty folder or folder with Microsoft OneNote files are deleted online when content is shared from a SharePoint Library Document Root * Fix Bug: Fix that empty 'skip_file' forces resync indefinitely * Fix Bug: Fix that 'sync_list' rule segment|depth check fails in some scenarios and implement a better applicable mechanism check * Fix Bug: Resolve crash when getpwuid() breaks when there is a glibc version mismatch * Fix Bug: Resolve crash when opening file fails when computing file hash * Fix Bug: Add check for invalid exclusion 'sync_list' exclusion rules * Fix Bug: Fix uploading of modified files when using --upload-only & --remove-source-files * Fix Bug: Fix local path calculation for Relocated OneDrive Business Shared Folders * Fix Bug: Fix 'sync_list' anywhere rule online directory creation * Fix Bug: Fix online path creation to ensure parental path structure is created in a consistent manner * Fix Bug: Fix handling of POSIX check for existing online items * Fix Bug: Fix args printing in dockerfile entrypoint * Fix Bug: Fix the testing of parental structure for 'sync_list' inclusion when adding inotify watches * Fix Bug: Fix failure to handle API 403 response when file fragment upload fails * Fix Bug: Fix application notification output to be consistent when skipping integrity checks * Fix Bug: Fix how local timestamps are modified * Fix Bug: Fix how online remaining free space is calculated and consumed internally for free space tracking * Fix Bug: Fix logic of determining if a file has valid integrity when using --disable-upload-validation * Fix Bug: Format the OneDrive change into a consumable object for the database earlier to use values in application logging * Fix Bug: Fix upload session offset handling to prevent desynchronisation on large files * Fix Bug: Fix implementation of 'write_xattr_data' to support FreeBSD * Fix Bug: Update hash functions to ensure file is closed if opened * Fix Bug: Dont blindly run safeBackup() if the online timestamp is newer * Fix Bug: Only set xattr values when not using --dry-run * Fix Bug: Fix UTC conversion for existing file timestamp post file download * Fix Bug: Fix that 'check_nosync' and 'skip_size' configuration options when changed, were not triggering a --resync correctly * Fix Bug: Ensure file is closed before renaming to improve compatibility with GCS buckets and network filesystems * Fix Bug: If a file fails to download, path fails to exist. Check path existence before setting xattr values ### Updated * Updated .gitignore to ignore files created during configure to be consistent with other files generated from .in templates * Updated bash,fish and zsh completion files to align with application options * Updated 'config' file to align to application options with applicable descriptions * Updated testbuild runner * Updated Fedora Docker OS version to Fedora 42 * Updated Ubuntu 24.10 curl version 8.9.1 to known bad curl versions and document the bugs associated with it * Updated Makefile to pass libraries after source files in compiler invocation * Updated 'configure.ac' to support more basename formats for DC * Update how threads are set based on available CPUs * Update setLocalPathTimestamp logging output * Update when to perform thread check and set as early as possible * Updated documentation ## 2.5.5 - 2025-03-17 ### Added * Implement Feature Request: Implement 'transfer_order' configuration option to allow the user to determine what order files are transferred in * Implement Feature Request: Implement 'disable_permission_set' configuration option to not set directory and file permissions * Implement Feature Request: Implement 'write_xattr_data' configuration option to add information about file creator/last editor as extended file attributes * Enhancement: Add support for --share-password option when --create-share-link is called * Enhancement: Add support 'localizedMessage' error messages in application output if this is provided in the JSON response from Microsoft Graph API ### Changed * Changed curl debug logging to --debug-https as this is more relevant * Comprehensively overhauled how OneDrive Personal Shared Folders are handled due to major OneDrive API backend platform user migration and major differences in API response output * Comprehensively changed OneDrive Personal 'driveId' value checking due to major OneDrive API backend platform user migration and major differences in API response output ### Fixed * Fix Bug: Fix path calculation for Client Side Filtering evaluations for Personal Accounts * Fix Bug: Fix path calculation for Client Side Filtering evaluations for Business Accounts * Fix Bug: Only perform path calculation if this is actually required * Fix Bug: Fix check for 'globbing' and 'wildcard' rules, that the number of segments before the first wildcard character need to match before the actual rule can be applied * Fix Bug: When using 'sync_list' , ignore specific exclusion to scan that path for new data, which may be actually included by an include rule, but the parent path is excluded * Fix Bug: When removing a OneDrive Personal Shared Folder, remove the actual link, not the remote user folder * Fix Bug: Fix 'Unsupported platform' for inotify watches by using the correct predefined version definition for Linux. ### Updated * Updated Fedora Docker OS version to Fedora 41 * Updated Alpine Docker OS version to Alpine 3.21 * Updated documentation ## 2.5.4 - 2025-02-03 ### Added * Implement Feature Request: Support Permanent Delete on OneDrive * Implement Feature Request: Support the moving of Shared Folder Links to other folders (Business Accounts only) * Enhancement: Added due to ongoing Ubuntu issues with 'curl' and 'libcurl', updated the documentation to include all relevant curl bugs and affected versions * Enhancement: Added quota status messages for nearing | critical | exceeded based on OneDrive Account API response * Enhancement: Added Docker variable to implement a sync once option * Enhancement: Added configuration option 'create_new_file_version' to force create new versions if that is the desire * Enhancement: Added support for adding SharePoint Libraries as Shared Folder Links * Enhancement: Added code and documentation changes to support FreeBSD * Enhancement: Added a check for the 'sea8cc6beffdb43d7976fbc7da445c639' string in the Microsoft OneDrive Personal Account Root ID response that denotes that the account cannot access Microsoft OneDrive at this point in time * Enhancement: Added './' sync_list rule check as this does not align to the documentation and these rules will not get matched correctly. ### Changed * Changed how debug logging outputs HTTP response headers and when this occurs * Changed when the check for no --sync | --monitor occurs so that this fails faster to avoid setting up all the other components * Changed isValidUTF8 function to use 'validate' rather than individual character checking and enhance checks including length constraints * Changed --dry-run authentication message to remove ambiguity that --dry-run cannot be used to authenticate the application ### Fixed * Fix Regression: Fixed regression that sync_list does not traverse shared directories * Fix Regression: Fixed regression of --display-config use after fast failing if --sync or --monitor has not been used * Fix Regression: Fixed regression from v2.4.x in handling uploading new and modified content to OneDrive Business and SharePoint to not create new versions of files post upload which adds to user quota * Fix Regression: Add back file transfer metrics which was available in v2.4.x * Fix Regression: Add code to support using 'display_processing_time' for functional performance which was available in v2.4.x * Fix Bug: Fixed build issue for OpenBSD (however support for OpenBSD itself is still a work-in-progress) * Fix Bug: Fixed issue regarding parsing OpenSSL and when unable to be parsed, do not force the application to exit * Fix Bug: Fixed the import of 'sync_list' rules due to OneDriveGUI creating a blank empty file by default * Fix Bug: Fixed the display of 'sync_list' rules due to OneDriveGUI creating a blank empty file by default * Fix Bug: Fixed that Business Shared Items shortcuts are skipped as being incorrectly detected as Microsoft OneNote Notebook items * Fix Bug: Fixed space calculations due to using ulong variable type to ensure that if calculation is negative, value is negative * Fix Bug: Fixed issue when downloading a file, and this fails due to an API error (400, 401, 5xx), online file is then not deleted * Fix Bug: Fixed skip_dir logic when reverse traversing folder structure * Fix Bug: Fixed issue that when using 'sync_list' if a file is moved to a newly created online folder, whilst the folder is created database wise, ensure this folder exists on local disk * Fix Bug: Fixed path got deleted in handling of move & close_write event when using 'vim'. * Fix Bug: Fixed that the root Personal Shared Folder is not handled due to missing API data European Data Centres * Fix Bug: Fixed the the local timestamp is not set when using --disable-download-validation * Fix Bug: Fixed Upload|Download Loop for AIP Protected File in Monitor Mode * Fix Bug: Fixed --single-directory Shared Folder DB entry creation * Fix Bug: Fixed API Bug to ensure that OneDrive Personal Drive ID and Remote Drive ID values are 16 characters, padded by leading zeros if the provided JSON data has dropped these leading zeros * Fix Bug: Fixed testInternetReachability function so that this always returns a boolean value and not throw an exception ### Updated * Updated documentation ## 2.5.3 - 2024-11-16 ### Added * Implement Feature Request: Implement Docker ENV variable for --cleanup-local-files * Enhancement: Setup a specific SIGPIPE Signal handler for curl/openssl generated signals * Enhancement: Add Check Spelling GitHub Action * Enhancement: Add passive database checkpoints to optimise database operations * Enhancement: Ensure application notifies user of curl versions that contain HTTP/2 bugs that impact the operation of this client * Enhancement: Add OpenSSL version warning * Enhancement: Improve performance with reduced execution time and lower CPU/system resource usage ### Changed * Specifically use a 'mutex' to perform the lock on database actions * Update safeBackup to use a new filename format for easier identification: filename-hostname-safeBackup-number.file_extension * Allow no-sync operations to complete online account checks ### Fixed * Fix Regression: Fix regression for Docker 'sync_dir' use * Fix Bug: Fix that a 'sync_list' entry of '/' will cause a index [0] is out of bounds * Fix Bug: Fix that when creating a new folder online the application generates an exception if it is in a Shared Online Folder * Fix Bug: Fix application crash when session upload files contain zero data or are corrupt * Fix Bug: Fix that curl generates a SIGPIPE that causes application to exit due to upstream device killing idle TCP connection * Fix Bug: Fix that skip_dir is not flagging directories correctly causing deletion if parental path structure needs to be created for sync_list handling * Fix Bug: Fix application crash caused by unable to drop table * Fix Bug: Fix that skip_file in config does not override defaults * Fix Bug: Handle DB upgrades from v2.4.x without causing application crash * Fix Bug: Fix a database statement execution error occurred: NOT NULL constraint failed: item.type due to Microsoft OneNote items * Fix Bug: Fix Operation not permitted FileException Error when attempting to use setTimes() function * Fix Bug: Fix that files with no mime type cause sync to crash * Fix Bug: Fix that bypass_data_preservation operates as intended ### Updated * Fixed spelling errors across all documentation and code * Update Dockerfile-debian to fix that libcurl4 does not get applied despite being pulled in. Explicitly install it from Debian 12 Backports * Add Ubuntu 24.10 OpenSuSE Build Service details * Update Dockerfile-alpine - revert to Alpine 3.19 as application fails to run on Alpine 3.20 * Updated documentation ## 2.5.2 - 2024-09-29 ### Added * Added 15 second sleep to systemd services to allow d-bus daemon to start and be available if present ### Fixed * Fix Bug: Application crash unable to correctly process a timestamp that has fractional seconds * Fix Bug: Fixed application logging output of Personal Shared Folder incorrectly advising there is no free space ### Updated * Updated documentation ## 2.5.1 - 2024-09-27 (DO NOT USE. CONTAINS A MAJOR TIMESTAMP ISSUE BUG) ### Special Thankyou A special thankyou to @phlibi for assistance with diagnosing and troubleshooting the database timestamp issue ### Added * Implement Feature Request: Don't print the d-bus WARNING if disable_notifications is set on cmd line or in config ### Changed * Add --enable-debug to Docker files when building client application to allow for better diagnostics when issues occur * Update Debian Dockerfile to use 'curl' from backports so a more modern curl version is used ### Fixed * Fix Regression: Fix regression of extra quotation marks when using ONEDRIVE_SINGLE_DIRECTORY with Docker * Fix Regression: Fix regression that real-time synchronization is not occurring when using --monitor and sync_list * Fix Regression: Fix regression that --remove-source-files doesn’t work * Fix Bug: Application crash when run synchronize due to negative free space online * Fix Bug: Application crash when performing a URL decode * Fix Bug: Application crash when using sync_list and Personal Shared Folders the root folder fails to present the item id * Fix Bug: Application crash when attempting to read timestamp from database as invalid data was written ### Updated * Updated documentation (various) ## 2.5.0 - 2024-09-16 ### Special Thankyou A special thankyou to all those who helped with testing and providing feedback during the development of this major release. A big thankyou to: * @JC-comp * @Lyncredible * @rrodrigueznt * @bpozdena * @hskrieg * @robertschulze * @aothmane-control * @mozram * @LunCh-CECNL * @pkolmann * @tdcockers * @undefiened * @cyb3rko ### Notable Changes * This version introduces significant changes regarding how the integrity and validation of your data is determined and is not backwards compatible with v2.4.x. * OneDrive Business Shared Folder Sync has been 100% re-written in v2.5.0. If you are using this feature, please read the new documentation carefully. * The application function --download-only no longer automatically deletes local files. Please read the new documentation regarding this feature. ### Added * Implement Feature Request: Multi-threaded uploading/downloading of files * Implement Feature Request: Renaming/Relocation of OneDrive Business shared folders * Implement Feature Request: Support the syncing of individual business shared files * Implement Feature Request: Implement application output to detail upload|download failures at the end of a sync process * Implement Feature Request: Log when manual Authorization is required when using --auth-files * Implement Feature Request: Add cmdline parameter to display (human readable) quota status * Implement Feature Request: Add capability to disable 'fullscan_frequency' * Implement Feature Request: Ability to set --disable-download-validation from Docker environment variable * Implement Feature Request: Ability to set --sync-shared-files from Docker environment variable * Implement Feature Request: file sync (upload/download/delete) notifications ### Changed * Renamed various documentation files to align with document content * Implement buffered logging so that all logging from all upload & download activities are handled correctly * Replace polling monitor loop with blocking wait * Update how the application utilises curl to fix socket reuse * Various performance enhancements * Implement refactored OneDrive API logic * Enforcement of operational conflicts * Enforcement of application configuration defaults and minimums * Utilise threadsafe sqlite DB access methods * Various bugs and other issues identified during development and testing * Various code cleanup and optimisations ### Fixed * Fix Bug: Upload only not working with Business shared folders * Fix Bug: Business shared folders with same basename get merged * Fix Bug: --dry-run prevents authorization * Fix Bug: Log timestamps lacking trailing zeros, leading to poor log file output alignment * Fix Bug: Subscription ID already exists when using webhooks * Fix Bug: Not all files being downloaded when API data includes HTML ASCII Control Sequences * Fix Bug: --display-sync-status does not work when OneNote sections (.one files) are in your OneDrive * Fix Bug: vim backups when editing files cause edited file to be deleted rather than the edited file being uploaded * Fix Bug: skip_dir does not always work as intended for all directory entries * Fix Bug: Online date being changed in download-only mode * Fix Bug: Resolve that download_only = "true" and cleanup_local_files = "true" also deletes files present online * Fix Bug: Resolve that upload session are not canceled with resync option * Fix Bug: Local files should be safely backed up when the item is not in sync locally to prevent data loss when they are deleted online * Fix Bug: Files with newer timestamp are not chosen as version to be kept * Fix Bug: Synced file is removed when updated on the remote while being processed by onedrive * Fix Bug: Cannot select/filter within Personal Shared Folders * Fix Bug: HTML encoding requires to add filter entries twice * Fix Bug: Uploading files using fragments stuck at 0% * Fix Bug: Implement safeguard when sync_dir is missing and is re-created data is not deleted online * Fix Bug: Fix that --get-sharepoint-drive-id does not handle a SharePoint site with more than 200 entries * Fix Bug: Fix that 'sync_list' does not include files that should be included, when specified just as *.ext_type * Fix Bug: Fix 'sync_list' processing so that '.folder_name' is excluded but 'folder_name' is included ### Updated * Overhauled all documentation ## 2.4.25 - 2023-06-21 ### Fixed * Fixed that the application was reporting as v2.2.24 when in fact it was v2.4.24 (release tagging issue) * Fixed that the running version obsolete flag (due to above issue) was causing a false flag as being obsolete * Fixed that zero-byte files do not have a hash as reported by the OneDrive API thus should not generate an error message ### Updated * Update to Debian Docker file to resolve Docker image Operating System reported vulnerabilities * Update to Alpine Docker file to resolve Docker image Operating System reported vulnerabilities * Update to Fedora Docker file to resolve Docker image Operating System reported vulnerabilities * Updated documentation (various) ## 2.4.24 - 2023-06-20 ### Fixed * Fix for extra encoded quotation marks surrounding Docker environment variables * Fix webhook subscription creation for SharePoint Libraries * Fix that a HTTP 504 - Gateway Timeout causes local files to be deleted when using --download-only & --cleanup-local-files mode * Fix that folders are renamed despite using --dry-run * Fix deprecation warnings with dmd 2.103.0 * Fix error that the application is unable to perform a database vacuum: out of memory when exiting ### Removed * Remove sha1 from being used by the client as this is being deprecated by Microsoft in July 2023 * Complete the removal of crc32 elements ### Added * Added ONEDRIVE_SINGLE_DIRECTORY configuration capability to Docker * Added --get-file-link shell completion * Added configuration to allow HTTP session timeout(s) tuning via config (taken from v2.5.x) ### Updated * Update to Debian Docker file to resolve Docker image Operating System reported vulnerabilities * Update to Alpine Docker file to resolve Docker image Operating System reported vulnerabilities * Update to Fedora Docker file to resolve Docker image Operating System reported vulnerabilities * Updated cgi.d to commit 680003a - last upstream change before requiring `core.d` dependency requirement * Updated documentation (various) ## 2.4.23 - 2023-01-06 ### Fixed * Fixed RHEL7, RHEL8 and RHEL9 Makefile and SPEC file compatibility ### Removed * Disable systemd 'PrivateUsers' due to issues with systemd running processes when option is enabled, causes local file deletes on RHEL based systems ### Updated * Update --get-O365-drive-id error handling to display a more a more appropriate error message if the API cannot be found * Update the GitHub version check to utilise the date a release was done, to allow 1 month grace period before generating obsolete version message * Update Alpine Dockerfile to use Alpine 3.17 and Golang 1.19 * Update handling of --source-directory and --destination-directory if one is empty or missing and if used with --synchronize or --monitor * Updated documentation (various) ## 2.4.22 - 2022-12-06 ### Fixed * Fix application crash when local file is changed to a symbolic link with non-existent target * Fix build error with dmd-2.101.0 * Fix build error with LDC 1.28.1 on Alpine * Fix issue of silent exit when unable to delete local files when using --cleanup-local-files * Fix application crash due to access permissions on configured path for sync_dir * Fix potential application crash when exiting due to failure state and unable to cleanly shutdown the database * Fix creation of parent empty directories when parent is excluded by sync_list ### Added * Added performance output details for key functions ### Changed * Switch Docker 'latest' to point at Debian builds rather than Fedora due to ongoing Fedora build failures * Align application logging events to actual application defaults for --monitor operations * Performance Improvement: Avoid duplicate costly path calculations and DB operations if not required * Disable non-working remaining sandboxing options within systemd service files * Performance Improvement: Only check 'sync_list' if this has been enabled and configured * Display 'Sync with OneDrive is complete' when using --synchronize * Change the order of processing between Microsoft OneDrive restrictions and limitations check and skip_file|skip_dir check ### Removed * Remove building Fedora ARMv7 builds due to ongoing build failures ### Updated * Update config change detection handling * Updated documentation (various) ## 2.4.21 - 2022-09-27 ### Fixed * Fix that the download progress bar doesn't always reach 100% when rate_limit is set * Fix --resync handling of database file removal * Fix Makefile to be consistent with permissions that are being used * Fix that logging output for skipped uploaded files is missing * Fix to allow non-sync tasks while sync is running * Fix where --resync is enforced for non-sync operations * Fix to resolve segfault when running 'onedrive --display-sync-status' when run as 2nd process * Fix DMD 2.100.2 depreciation warning ### Added * Add GitHub Action Test Build Workflow (replacing Travis CI) * Add option --display-running-config to display the running configuration as used at application startup * Add 'config' option to request readonly access in oauth authorization step * Add option --cleanup-local-files to cleanup local files regardless of sync state when using --download-only * Add option --with-editing-perms to create a read-write shareable link when used with --create-share-link ### Changed * Change the exit code of the application to 126 when a --resync is required ### Updated * Updated --get-O365-drive-id implementation for data access * Update what application options require an argument * Update application logging output for error messages to remove certain \n prefix when logging to a file * Update onedrive.spec.in to fix error building RPM * Update GUI notification handling for specific skipped scenarios * Updated documentation (various) ## 2.4.20 - 2022-07-20 ### Fixed * Fix 'foreign key constraint failed' when using OneDrive Business Shared Folders due to change to using /delta query * Fix various little spelling errors (checked with lintian during Debian packaging) * Fix handling of a custom configuration directory when using --confdir * Fix to ensure that any active http instance is shutdown before any application exit * Fix to enforce that --confdir must be a directory ### Added * Added 'force_http_11' configuration option to allow forcing HTTP/1.1 operations ### Changed * Increased thread sleep for better process I/O wait handling * Removed 'force_http_2' configuration option ### Updated * Update OneDrive API response handling for National Cloud Deployments * Updated to switch to using curl defaults for HTTP/2 operations * Updated documentation (various) ## 2.4.19 - 2022-06-15 ### Fixed * Update Business Shared Folders to use a /delta query * Update when DB is updated by OneDrive API data and update when file hash is required to be generated ### Added * Added ONEDRIVE_UPLOADONLY flag for Docker ### Updated * Updated GitHub workflows * Updated documentation (various) ## 2.4.18 - 2022-06-02 ### Fixed * Fixed various database related access issues stemming from running multiple instances of the application at the same time using the same configuration data * Fixed --display-config being impacted by --resync flag * Fixed installation permissions for onedrive man-pages file * Fixed that in some situations that users try --upload-only and --download-only together which is not possible * Fixed application crash if unable to read required hash files ### Added * Added Feature Request to add an override for skip_dir|skip_file through flag to force sync * Added a check to validate local filesystem available space before attempting file download * Added GitHub Actions to build Docker containers and push to DockerHub ### Updated * Updated all Docker build files to current distributions, using updated distribution LDC version * Updated logging output to logfiles when an actual sync process is occurring * Updated output of --display-config to be more relevant * Updated manpage to align with application configuration * Updated documentation and Docker files based on minimum compiler versions to dmd-2.088.0 and ldc-1.18.0 * Updated documentation (various) ## 2.4.17 - 2022-04-30 ### Fixed * Fix docker build, by add missing git package for Fedora builds * Fix application crash when attempting to sync a broken symbolic link * Fix Internet connect disruption retry handling and logging output * Fix local folder creation timestamp with timestamp from OneDrive * Fix logging output when download failed ### Added * Add additional logging specifically for delete event to denote in log output the source of a deletion event when running in --monitor mode ### Changed * Improve when the local database integrity check is performed and on what frequency the database integrity check is performed ### Updated * Remove application output ambiguity on how to access 'help' for the client * Update logging output when running in --monitor --verbose mode in regards to the inotify events * Updated documentation (various) ## 2.4.16 - 2022-03-10 ### Fixed * Update application file logging error handling * Explicitly set libcurl options * Fix that when a sync_list exclusion is matched, the item needs to be excluded when using --resync * Fix so that application can be compiled correctly on Android hosts * Fix the handling of 429 and 5xx responses when they are generated by OneDrive in a self-referencing circular pattern * Fix applying permissions to volume directories when running in rootless podman * Fix unhandled errors from OneDrive when initialising subscriptions fail ### Added * Enable GitHub Sponsors * Implement --resync-auth to enable CLI passing in of --rsync approval * Add function to check client version vs latest GitHub release * Add --reauth to allow easy re-authentication of the client * Implement --modified-by to display who last modified a file and when the modification was done * Implement feature request to mark partially-downloaded files as .partial during download * Add documentation for Podman support ### Changed * Document risk regarding using --resync and force user acceptance of usage risk to proceed * Use YAML for Bug Reports and Feature Requests * Update Dockerfiles to use more modern base Linux distribution ### Updated * Updated documentation (various) ## 2.4.15 - 2021-12-31 ### Fixed * Fix unable to upload to OneDrive Business Shared Folders due to OneDrive API restricting quota information * Update fixing edge case with OneDrive Personal Shared Folders and --resync --upload-only ### Added * Add SystemD hardening * Add --operation-timeout argument ### Changed * Updated minimum compiler versions to dmd-2.087.0 and ldc-1.17.0 ### Updated * Updated Dockerfile-alpine to use Alpine 3.14 * Updated documentation (various) ## 2.4.14 - 2021-11-24 ### Fixed * Support DMD 2.097.0 as compiler for Docker Builds * Fix getPathDetailsByDriveId query when using --dry-run and a nested path with --single-directory * Fix edge case when syncing OneDrive Personal Shared Folders * Catch unhandled API response errors when querying OneDrive Business Shared Folders * Catch unhandled API response errors when listing OneDrive Business Shared Folders * Fix error 'Key not found: remaining' with Business Shared Folders (OneDrive API change) * Fix overwriting local files with older versions from OneDrive when items.sqlite3 does not exist and --resync is not used ### Added * Added operation_timeout as a new configuration to assist in cases where operations take longer that 1h to complete * Add Real-Time syncing of remote updates via webhooks * Add --auth-response option and expose through entrypoint.sh for Docker * Add --disable-download-validation ### Changed * Always prompt for credentials for authentication rather than re-using cached browser details * Do not re-auth on --logout ### Updated * Updated documentation (various) ## 2.4.13 - 2021-7-14 ### Fixed * Support DMD 2.097.0 as compiler * Fix to handle OneDrive API Bad Request response when querying if file exists * Fix application crash and incorrect handling of --single-directory when syncing a OneDrive Business Shared Folder due to using 'Add Shortcut to My Files' * Fix application crash due to invalid UTF-8 sequence in the pathname for the application configuration * Fix error message when deleting a large number of files * Fix Docker build process to source GOSU keys from updated GPG key location * Fix application crash due to a conversion overflow when calculating file offset for session uploads * Fix Docker Alpine build failing due to filesystem permissions issue due to Docker build system and Alpine Linux 3.14 incompatibility * Fix that Business Shared Folders with parentheses are ignored ### Updated * Updated Lock Bot to run daily * Updated documentation (various) ## 2.4.12 - 2021-5-28 ### Fixed * Fix an unhandled Error 412 when uploading modified files to OneDrive Business Accounts * Fix 'sync_list' handling of inclusions when name is included in another folders name * Fix that options --upload-only & --remove-source-files are ignored on an upload session restore * Fix to add file check when adding item to database if using --upload-only --remove-source-files * Fix application crash when SharePoint displayName is being withheld ### Updated * Updated Lock Bot to use GitHub Actions * Updated documentation (various) ## 2.4.11 - 2021-4-07 ### Fixed * Fix support for '/*' regardless of location within sync_list file * Fix 429 response handling correctly check for 'retry-after' response header and use set value * Fix 'sync_list' path handling for sub item matching, so that items in parent are not implicitly matched when there is no wildcard present * Fix --get-O365-drive-id to use 'nextLink' value if present when searching for specific SharePoint site names * Fix OneDrive Business Shared Folder existing name conflict check * Fix incorrect error message 'Item cannot be deleted from OneDrive because it was not found in the local database' when item is actually present * Fix application crash when unable to rename folder structure due to unhandled file-system issue * Fix uploading documents to Shared Business Folders when the shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files * Fix that a file record is kept in database when using --no-remote-delete & --remove-source-files ### Added * Added support in --get-O365-drive-id to provide the 'drive_id' for multiple 'document libraries' within a single Shared Library Site ### Removed * Removed the deprecated config option 'force_http_11' which was flagged as deprecated by PR #549 in v2.3.6 (June 2019) ### Updated * Updated error output of --get-O365-drive-id to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match * Updated Docker build files for Raspberry Pi to dedicated armhf & aarch64 Dockerfiles * Updated logging output when in --monitor mode, avoid outputting misleading logging when the new or modified item is a file, not a directory * Updated documentation (various) ## 2.4.10 - 2021-2-19 ### Fixed * Catch database assertion when item path cannot be calculated * Fix alpine Docker build so it uses the same golang alpine version * Search all distinct drive id's rather than just default drive id for --get-file-link * Use correct driveId value to query for changes when using --single-directory * Improve upload handling of files for SharePoint sites and detecting when SharePoint modifies the file post upload * Correctly handle '~' when present in 'log_dir' configuration option * Fix logging output when handing downloaded new files * Fix to use correct path offset for sync_list exclusion matching ### Added * Add upload speed metrics when files are uploaded and clarify that 'data to transfer' is what is needed to be downloaded from OneDrive * Add new config option to rate limit connection to OneDrive * Support new file maximum upload size of 250GB * Support sync_list matching full path root wildcard with exclusions to simplify sync_list configuration ### Updated * Rename Office365.md --> SharePoint-Shared-Libraries.md which better describes this document * Updated Dockerfile config for arm64 * Updated documentation (various) ## 2.4.9 - 2020-12-27 ### Fixed * Fix to handle case where API provided deltaLink generates a further API error * Fix application crash when unable to read a local file due to local file permissions * Fix application crash when calculating the path length due to invalid UTF characters in local path * Fix Docker build on Alpine due missing symbols due to using the edge version of ldc and ldc-runtime * Fix application crash with --get-O365-drive-id when API response is restricted ### Added * Add debug log output of the configured URL's which will be used throughout the application to remove any ambiguity as to using incorrect URL's when making API calls * Improve application startup when using --monitor when there is no network connection to the OneDrive API and only initialise application once OneDrive API is reachable * Add Docker environment variable to allow --logout for re-authentication ### Updated * Remove duplicate code for error output functions and enhance error logging output * Updated documentation ## 2.4.8 - 2020-11-30 ### Fixed * Fix to use config set option for 'remove_source_files' and 'skip_dir_strict_match' rather than ignore if set * Fix download failure and crash due to incorrect local filesystem permissions when using mounted external devices * Fix to not change permissions on pre-existing local directories * Fix logging output when authentication authorisation fails to not say authorisation was successful * Fix to check application_id before setting redirect URL when using specific Azure endpoints * Fix application crash in --monitor mode due to 'Failed to stat file' when setgid is used on a directory and data cannot be read ### Added * Added advanced-usage.md to document advanced client usage such as multi account configurations and Windows dual-boot ### Updated * Updated --verbose logging output for config options when set * Updated documentation (man page, USAGE.md, Office365.md, BusinessSharedFolders.md) ## 2.4.7 - 2020-11-09 ### Fixed * Fix debugging output for /delta changes available queries * Fix logging output for modification comparison source data * Fix Business Shared Folder handling to process only Shared Folders, not individually shared files * Fix cleanup dryrun shm and wal files if they exist * Fix --list-shared-folders to only show folders * Fix to check for the presence of .nosync when processing DB entries * Fix skip_dir matching when using --resync * Fix uploading data to shared business folders when using --upload-only * Fix to merge contents of SQLite WAL file into main database file on sync completion * Fix to check if localModifiedTime is >= than item.mtime to avoid re-upload for equal modified time * Fix to correctly set config directory permissions at first start ### Added * Added environment variable to allow easy HTTPS debug in docker * Added environment variable to allow download-only mode in Docker * Implement Feature: Allow config to specify a tenant id for non-multi-tenant applications * Implement Feature: Adding support for authentication with single tenant custom applications * Implement Feature: Configure specific File and Folder Permissions ### Updated * Updated documentation (readme.md, install.md, usage.md, bug_report.md) ## 2.4.6 - 2020-10-04 ### Fixed * Fix flagging of remaining free space when value is being restricted * Fix --single-directory path handling when path does not exist locally * Fix checking for 'Icon' path as no longer listed by Microsoft as an invalid file or folder name * Fix removing child items on OneDrive when parent item responds with access denied * Fix to handle deletion events for files when inotify events are missing * Fix uninitialised value error as reported by valgrind * Fix to handle deletion events for directories when inotify events are missing ### Added * Implement Feature: Create shareable link * Implement Feature: Support wildcard within sync_list entries * Implement Feature: Support negative patterns in sync_list for fine grained exclusions * Implement Feature: Multiple skip_dir & skip_file configuration rules * Add GUI notification to advise users when the client needs to be reauthenticated ### Updated * Updated documentation (readme.md, install.md, usage.md, bug_report.md) ## 2.4.5 - 2020-08-13 ### Fixed * Fixed fish auto completions installation destination ## 2.4.4 - 2020-08-11 ### Fixed * Fix 'skip_dir' & 'skip_file' pattern matching to ensure correct matching is performed * Fix 'skip_dir' & 'skip_file' so that each directive is only used against directories or files as required in --monitor * Fix client hand when attempting to sync a Unix pipe file * Fix --single-directory & 'sync_list' performance * Fix erroneous 'return' statements which could prematurely end processing all changes returned from OneDrive * Fix segfault when attempting to perform a comparison on an inotify event when determining if event path is directory or file * Fix handling of Shared Folders to ensure these are checked against 'skip_dir' entries * Fix 'Skipping uploading this new file as parent path is not in the database' when uploading to a Personal Shared Folder * Fix how available free space is tracked when uploading files to OneDrive and Shared Folders * Fix --single-directory handling of parent path matching if path is being seen for first time ### Added * Added Fish auto completions ### Updated * Increase maximum individual file size to 100GB due to Microsoft file limit increase * Update Docker build files and align version of compiler across all Docker builds * Update Docker documentation * Update NixOS build information * Update the 'Processing XXXX' output to display the full path * Update logging output when a sync starts and completes when using --monitor * Update Office 365 / SharePoint site search query and response if query return zero match ## 2.4.3 - 2020-06-29 ### Fixed * Check if symbolic link is relative to location path * When using output logfile, fix inconsistent output spacing * Perform initial sync at startup in monitor mode * Handle a 'race' condition to process inotify events generated whilst performing DB or filesystem walk * Fix segfault when moving folder outside the sync directory when using --monitor on Arch Linux ### Added * Added additional inotify event debugging * Added support for loading system configs if there's no user config * Added Ubuntu installation details to include installing the client from a PPA * Added openSUSE installation details to include installing the client from a package * Added support for comments in sync_list file * Implement recursive deletion when Retention Policy is enabled on OneDrive Business Accounts * Implement support for National cloud deployments * Implement OneDrive Business Shared Folders Support ### Updated * Updated documentation files (various) * Updated log output messaging when a full scan has been set or triggered * Updated buildNormalizedPath complexity to simplify code * Updated to only process OneDrive Personal Shared Folders only if account type is 'personal' ## 2.4.2 - 2020-05-27 ### Fixed * Fixed the catching of an unhandled exception when inotify throws an error * Fixed an uncaught '100 Continue' response when files are being uploaded * Fixed progress bar for uploads to be more accurate regarding percentage complete * Fixed handling of database query enforcement if item is from a shared folder * Fixed compiler depreciation of std.digest.digest * Fixed checking & loading of configuration file sequence * Fixed multiple issues reported by Valgrind * Fixed double scan at application startup when using --monitor & --resync together * Fixed when renaming a file locally, ensure that the target filename is valid before attempting to upload to OneDrive * Fixed so that if a file is modified locally and --resync is used, rename the local file for data preservation to prevent local data loss ### Added * Implement 'bypass_data_preservation' enhancement ### Changed * Changed the monitor interval default to 300 seconds ### Updated * Updated the handling of out-of-space message when OneDrive is out of space * Updated debug logging for retry wait times ## 2.4.1 - 2020-05-02 ### Fixed * Fixed the handling of renaming files to a name starting with a dot when skip_dotfiles = true * Fixed the handling of parentheses from path or file names, when doing comparison with regex * Fixed the handling of renaming dotfiles to another dotfile when skip_dotfile=true in monitor mode * Fixed the handling of --dry-run and --resync together correctly as current database may be corrupt * Fixed building on Alpine Linux under Docker * Fixed the handling of --single-directory for --dry-run and --resync scenarios * Fixed the handling of .nosync directive when downloading new files into existing directories that is (was) in sync * Fixed the handling of zero-byte modified files for OneDrive Business * Fixed skip_dotfiles handling of .folders when in monitor mode to prevent monitoring * Fixed the handling of '.folder' -> 'folder' move when skip_dotfiles is enabled * Fixed the handling of folders that cannot be read (permission error) if parent should be skipped * Fixed the handling of moving folders from skipped directory to non-skipped directory via OneDrive web interface * Fixed building on CentOS Linux under Docker * Fixed Codacy reported issues: double quote to prevent globbing and word splitting * Fixed an assertion when attempting to compute complex path comparison from shared folders * Fixed the handling of .folders when being skipped via skip_dir ### Added * Implement Feature: Implement the ability to set --resync as a config option, default is false ### Updated * Update error logging to be consistent when initialising fails * Update error logging output to handle HTML error response reasoning if present * Update link to new Microsoft documentation * Update logging output to differentiate between OneNote objects and other unsupported objects * Update RHEL/CentOS spec file example * Update known-issues.md regarding 'SSL_ERROR_SYSCALL, errno 104' * Update progress bar to be more accurate when downloading large files * Updated #658 and #865 handling of when to trigger a directory walk when changes occur on OneDrive * Updated handling of when a full scan is required due to utilising sync_list * Updated handling of when OneDrive service throws a 429 or 504 response to retry original request after a delay ## 2.4.0 - 2020-03-22 ### Fixed * Fixed how the application handles 429 response codes from OneDrive (critical update) * Fixed building on Alpine Linux under Docker * Fixed how the 'username' is determined from the running process for logfile naming * Fixed file handling when a failed download has occurred due to exiting via CTRL-C * Fixed an unhandled exception when OneDrive throws an error response on initialising * Fixed the handling of moving files into a skipped .folder when skip_dotfiles = true * Fixed the regex parsing of response URI to avoid potentially generating a bad request to OneDrive, leading to a 'AADSTS9002313: Invalid request. Request is malformed or invalid.' response. ### Added * Added a Dockerfile for building on Raspberry Pi / ARM platforms * Implement Feature: warning on big deletes to safeguard data on OneDrive * Implement Feature: delete local files after sync * Implement Feature: perform skip_dir explicit match only * Implement Feature: provide config file option for specifying the Client Identifier ### Changed * Updated the 'Client Identifier' to a new Application ID ### Updated * Updated relevant documentation (README.md, USAGE.md) to add new feature details and clarify existing information * Update completions to include the --force-http-2 option * Update to always log when a file is skipped due to the item being invalid * Update application output when just authorising application to make information clearer * Update logging output when using sync_list to be clearer as to what is actually being processed and why ## 2.3.13 - 2019-12-31 ### Fixed * Change the sync list override flag to false as default when not using sync_list * Fix --dry-run output when using --upload-only & --no-remote-delete and deleting local files ### Added * Add a verbose log entry when a monitor sync loop with OneDrive starts & completes ### Changed * Remove logAndNotify for 'processing X changes' as it is excessive for each change bundle to inform the desktop of the number of changes the client is processing ### Updated * Updated INSTALL.md with Ubuntu 16.x i386 build instructions to reflect working configuration on legacy hardware * Updated INSTALL.md with details of Linux packages * Updated INSTALL.md build instructions for CentOS platforms ## 2.3.12 - 2019-12-04 ### Fixed * Retry session upload fragment when transient errors occur to prevent silent upload failure * Update Microsoft restriction and limitations about windows naming files to include '~' for folder names * Docker guide fixes, add multiple account setup instructions * Check database for excluded sync_list items previously in scope * Catch DNS resolution error * Fix where an item now out of scope should be flagged for local delete * Fix rebuilding of onedrive, but ensure version is properly updated * Update Ubuntu i386 build instructions to use DMD using preferred method ### Added * Add debug message to when a message is sent to dbus or notification daemon * Add i386 instructions for legacy low memory platforms using LDC ## 2.3.11 - 2019-11-05 ### Fixed * Fix typo in the documentation regarding invalid config when upgrading from 'skilion' codebase * Fix handling of skip_dir, skip_file & sync_list config options * Fix typo in the documentation regarding sync_list * Fix log output to be consistent with sync_list exclusion * Fix 'Processing X changes' output to be more reflective of actual activity when using sync_list * Remove unused and unexported SED variable in Makefile.in * Handle curl exceptions and timeouts better with backoff/retry logic * Update skip_dir pattern matching when using wildcards * Fix when a full rescan is performed when using sync_list * Fix 'Key not found: name' when computing skip_dir path * Fix call from --monitor to observe --no-remote-delete * Fix unhandled exception when monitor initialisation failure occurs due to too many open local files * Fix unhandled 412 error response from OneDrive API when moving files right after upload * Fix --monitor when used with --download-only. This fixes a regression introduced in 12947d1. * Fix if --single-directory is being used, and we are using --monitor, only set inotify watches on the single directory ### Changed * Move JSON logging output from error messages to debug output ## 2.3.10 - 2019-10-01 ### Fixed * Fix searching for 'name' when deleting a synced item, if the OneDrive API does not return the expected details in the API call * Fix abnormal termination when no Internet connection * Fix downloading of files from OneDrive Personal Shared Folders when the OneDrive API responds with unexpected additional path data * Fix logging of 'initialisation' of client to actually when the attempt to initialise is performed * Fix when using a sync_list file, using deltaLink will actually 'miss' changes (moves & deletes) on OneDrive as using sync_list discards changes * Fix OneDrive API status code 500 handling when uploading files as error message is not correct * Fix crash when resume_upload file is not a valid JSON * Fix crash when a file system exception is generated when attempting to update the file date & time and this fails ### Added * If there is a case-insensitive match error, also return the remote name from the response * Make user-agent string a configuration option & add to config file * Set default User-Agent to 'OneDrive Client for Linux v{version}' ### Changed * Make verbose logging output optional on Docker * Enable --resync & debug client output via environment variables on Docker ## 2.3.9 - 2019-09-01 ### Fixed * Catch a 403 Forbidden exception when querying Sharepoint Library Names * Fix unhandled error exceptions that cause application to exit / crash when uploading files * Fix JSON object validation for queries made against OneDrive where a JSON response is expected and where that response is to be used and expected to be valid * Fix handling of 5xx responses from OneDrive when uploading via a session ### Added * Detect the need for --resync when config changes either via config file or cli override ### Changed * Change minimum required version of LDC to v1.12.0 ### Removed * Remove redundant logging output due to change in how errors are reported from OneDrive ## 2.3.8 - 2019-08-04 ### Fixed * Fix unable to download all files when OneDrive fails to return file level details used to validate file integrity * Included the flag "-m" to create the home directory when creating the user * Fix entrypoint.sh to work with "sudo docker run" * Fix docker build error on stretch * Fix hidden directories in 'root' from having prefix removed * Fix Sharepoint Document Library handling for .txt & .csv files * Fix logging for init.d service * Fix OneDrive response missing required 'id' element when uploading images * Fix 'Unexpected character '<'. (Line 1:1)' when OneDrive has an exception error * Fix error when creating the sync dir fails when there is no permission to create the sync dir ### Added * Add explicit check for hashes to be returned in cases where OneDrive API fails to provide them despite requested to do so * Add comparison with sha1 if OneDrive provides that rather than quickXor * Add selinux configuration details for a sync folder outside of the home folder * Add date tag on docker.hub * Add back CentOS 6 install & uninstall to Makefile * Add a check to handle moving items out of sync_list sync scope & delete locally if true * Implement --get-file-link which will return the weburl of a file which has been synced to OneDrive ### Changed * Change unauthorized-api exit code to 3 * Update LDC to v1.16.0 for Travis CI testing * Use replace function for modified Sharepoint Document Library files rather than delete and upload as new file, preserving file history * Update Sharepoint modified file handling for files > 4Mb in size ### Removed * Remove -d shorthand for --download-only to avoid confusion with other GNU applications where -d stands for 'debug' ## 2.3.7 - 2019-07-03 ### Fixed * Fix not all files being downloaded due to OneDrive query failure * False DB update which potentially could had lead to false data loss on OneDrive ## 2.3.6 - 2019-07-03 (DO NOT USE) ### Fixed * Fix JSONValue object validation * Fix building without git being available * Fix some spelling/grammatical errors * Fix OneDrive error response on creating upload session ### Added * Add download size & hash check to ensure downloaded files are valid and not corrupt * Added --force-http-2 to use HTTP/2 if desired ### Changed * Deprecated --force-http-1.1 (enabled by default) due to OneDrive inconsistent behavior with HTTP/2 protocol ## 2.3.5 - 2019-06-19 ### Fixed * Handle a directory in the sync_dir when no permission to access * Get rid of forced root necessity during installation * Fix broken autoconf code for --enable-XXX options * Fix so that skip_size check should only be used if configured * Fix a OneDrive Internal Error exception occurring before attempting to download a file ### Added * Check for supported version of D compiler ## 2.3.4 - 2019-06-13 ### Fixed * Fix 'Local files not deleted' when using bad 'skip_file' entry * Fix --dry-run logging output for faking downloading new files * Fix install unit files to correct location on RHEL/CentOS 7 * Fix up unit file removal on all platforms * Fix setting times on a file by adding a check to see if the file was actually downloaded before attempting to set the times on the file * Fix an unhandled curl exception when OneDrive throws an internal timeout error * Check timestamp to ensure that latest timestamp is used when comparing OneDrive changes * Fix handling responses where cTag JSON elements are missing * Fix Docker entrypoint.sh failures when GID is defined but not UID ### Added * Add autoconf based build system * Add an encoding validation check before any path length checks are performed as if the path contains any invalid UTF-8 sequences * Implement --sync-root-files to sync all files in the OneDrive root when using a sync_list file that would normally exclude these files from being synced * Implement skip_size feature request * Implement feature request to support file based OneDrive authorization (request | response) ### Updated * Better handle initialisation issues when OneDrive / MS Graph is experiencing problems that generate 401 & 5xx error codes * Enhance error message when unable to connect to Microsoft OneDrive service when the local CA SSL certificate(s) have issues * Update Dockerfile to correctly build on Docker Hub * Rework directory layout and re-factor MD files for readability ## 2.3.3 - 2019-04-16 ### Fixed * Fix --upload-only check for Sharepoint uploads * Fix check to ensure item root we flag as 'root' actually is OneDrive account 'root' * Handle object error response from OneDrive when uploading to OneDrive Business * Fix handling of some OneDrive accounts not providing 'quota' details * Fix 'resume_upload' handling in the event of bad OneDrive response ### Added * Add debugging for --get-O365-drive-id function * Add shell (bash,zsh) completion support * Add config options for command line switches to allow for better config handling in docker containers ### Updated * Implement more meaningful 5xx error responses * Update onedrive.logrotate indentations and comments * Update 'min_notif_changes' to 'min_notify_changes' ## 2.3.2 - 2019-04-02 ### Fixed * Reduce scanning the entire local system in monitor mode for local changes * Resolve file creation loop when working directly in the synced folder and Microsoft Sharepoint ### Added * Add 'monitor_fullscan_frequency' config option to set the frequency of performing a full disk scan when in monitor mode ### Updated * Update default 'skip_file' to include tmp and lock files generated by LibreOffice * Update database version due to changing defaults of 'skip_file' which will force a rebuild and use of new skip_file default regex ## 2.3.1 - 2019-03-26 ### Fixed * Resolve 'make install' issue where rebuild of application would occur due to 'version' being flagged as .PHONY * Update readme build instructions to include 'make clean;' before build to ensure that 'version' is cleanly removed and can be updated correctly * Update Debian Travis CI build URL's ## 2.3.0 - 2019-03-25 ### Fixed * Resolve application crash if no 'size' value is returned when uploading a new file * Resolve application crash if a 5xx error is returned when uploading a new file * Resolve not 'refreshing' version file when rebuilding * Resolve unexpected application processing by preventing use of --synchronize & --monitor together * Resolve high CPU usage when performing DB reads * Update error logging around directory case-insensitive match * Update Travis CI and ARM dependencies for LDC 1.14.0 * Update Makefile due to build failure if building from release archive file * Update logging as to why a OneDrive object was skipped ### Added * Implement config option 'skip_dir' ## 2.2.6 - 2019-03-12 ### Fixed * Resolve application crash when unable to delete remote folders when business retention policies are enabled * Resolve deprecation warning: loop index implicitly converted from size_t to int * Resolve warnings regarding 'bashisms' * Resolve handling of notification failure is dbus server has not started or available * Resolve handling of response JSON to ensure that 'id' key element is always checked for * Resolve excessive & needless logging in monitor mode * Resolve compiling with LDC on Alpine as musl lacks some standard interfaces * Resolve notification issues when offline and cannot act on changes * Resolve Docker entrypoint.sh to accept command line arguments * Resolve to create a new upload session on reinit * Resolve where on OneDrive query failure, default root and drive id is used if a response is not returned * Resolve Key not found: nextExpectedRanges when attempting session uploads and incorrect response is returned * Resolve application crash when re-using an authentication URI twice after previous --logout * Resolve creating a folder on a shared personal folder appears successful but returns a JSON error * Resolve to treat mv of new file as upload of mv target * Update Debian i386 build dependencies * Update handling of --get-O365-drive-id to print out all 'site names' that match the explicit search entry rather than just the last match * Update Docker readme & documentation * Update handling of validating local file permissions for new file uploads ### Added * Add support for install & uninstall on RHEL / CentOS 6.x * Add support for when notifications are enabled, display the number of OneDrive changes to process if any are found * Add 'config' option 'min_notif_changes' for minimum number of changes to notify on, default = 5 * Add additional Docker container builds utilising a smaller OS footprint * Add configurable interval of logging in monitor mode * Implement new CLI option --skip-dot-files to skip .files and .folders if option is used * Implement new CLI option --check-for-nosync to ignore folder when special file (.nosync) present * Implement new CLI option --dry-run ## 2.2.5 - 2019-01-16 ### Fixed * Update handling of HTTP 412 - Precondition Failed errors * Update --display-config to display sync_list if configured * Add a check for 'id' key on metadata update to prevent 'std.json.JSONException@std/json.d(494): Key not found: id' * Update handling of 'remote' folder designation as 'root' items * Ensure that remote deletes are handled correctly * Handle 'Item not found' exception when unable to query OneDrive 'root' for changes * Add handling for JSON response error when OneDrive API returns a 404 due to OneDrive API regression * Fix items highlighted by codacy review ### Added * Add --force-http-1.1 flag to downgrade any HTTP/2 curl operations to HTTP 1.1 protocol * Support building with ldc2 and usage of pkg-config for lib finding ## 2.2.4 - 2018-12-28 ### Fixed * Resolve JSONException when supplying --get-O365-drive-id option with a string containing spaces * Resolve 'sync_dir' not read from 'config' file when run in Docker container * Resolve logic where potentially a 'default' ~/OneDrive sync_dir could be set despite 'config' file configured for an alternate * Make sure sqlite checkpointing works by properly finalizing statements * Update logic handling of --single-directory to prevent inadvertent local data loss * Resolve signal handling and database shutdown on SIGINT and SIGTERM * Update man page * Implement better help output formatting ### Added * Add debug handling for sync_dir operations * Add debug handling for homePath calculation * Add debug handling for configDirBase calculation * Add debug handling if syncDir is created * Implement Feature Request: Add status command or switch ## 2.2.3 - 2018-12-20 ### Fixed * Fix syncdir option is ignored ## 2.2.2 - 2018-12-20 ### Fixed * Handle short lived files in monitor mode * Provide better log messages, less noise on temporary timeouts * Deal with items that disappear during upload * Deal with deleted move targets * Reinitialize sync engine after three failed attempts * Fix activation of dmd for docker builds * Fix to check displayName rather than description for --get-O365-drive-id * Fix checking of config file keys for validity * Fix exception handling when missing parameter from usage option ### Added * Notification support via libnotify * Add very verbose (debug) mode by double -v -v * Implement option --display-config ## 2.2.1 - 2018-12-04 ### Fixed * Gracefully handle connection errors in monitor mode * Fix renaming of files when syncing * Installation of doc files, addition of man page * Adjust timeout values for libcurl * Continue in monitor mode when sync timed out * Fix unreachable statements * Update Makefile to better support packaging * Allow starting offline in monitor mode ### Added * Implement --get-O365-drive-id to get correct SharePoint Shared Library (#248) * Docker buildfiles for onedrive service (#262) ## 2.2.0 - 2018-11-24 ### Fixed * Updated client to output additional logging when debugging * Resolve database assertion failure due to authentication * Resolve unable to create folders on shared OneDrive Personal accounts ### Added * Implement feature request to Sync from Microsoft SharePoint * Implement feature request to specify a logging directory if logging is enabled ### Changed * Change '--download' to '--download-only' to align with '--upload-only' * Change logging so that logging to a separate file is no longer the default ## 2.1.6 - 2018-11-15 ### Fixed * Updated HTTP/2 transport handling when using curl 7.62.0 for session uploads ### Added * Added PKGBUILD for makepkg for building packages under Arch Linux ## 2.1.5 - 2018-11-11 ### Fixed * Resolve 'Key not found: path' when syncing from some shared folders due to OneDrive API change * Resolve to only upload changes on remote folder if the item is in the database - dont assert if false * Resolve files will not download or upload when using curl 7.62.0 due to HTTP/2 being set as default for all curl operations * Resolve to handle HTTP request returned status code 412 (Precondition Failed) for session uploads to OneDrive Personal Accounts * Resolve unable to remove '~/.config/onedrive/resume_upload: No such file or directory' if there is a session upload error and the resume file does not get created * Resolve handling of response codes when using 2 different systems when using '--upload-only' but the same OneDrive account and uploading the same filename to the same location ### Updated * Updated Travis CI building on LDC v1.11.0 for ARMHF builds * Updated Makefile to use 'install -D -m 644' rather than 'cp -raf' * Updated default config to be aligned to code defaults ## 2.1.4 - 2018-10-10 ### Fixed * Resolve syncing of OneDrive Personal Shared Folders due to OneDrive API change * Resolve incorrect systemd installation location(s) in Makefile ## 2.1.3 - 2018-10-04 ### Fixed * Resolve File download fails if the file is marked as malware in OneDrive * Resolve high CPU usage when running in monitor mode * Resolve how default path is set when running under systemd on headless systems * Resolve incorrectly nested configDir in X11 systems * Resolve Key not found: driveType * Resolve to validate filename length before download to conform with Linux FS limits * Resolve file handling to look for HTML ASCII codes which will cause uploads to fail * Resolve Key not found: expirationDateTime on session resume ### Added * Update Travis CI building to test build on ARM64 ## 2.1.2 - 2018-08-27 ### Fixed * Resolve skipping of symlinks in monitor mode * Resolve Gateway Timeout - JSONValue is not an object * Resolve systemd/user is not supported on CentOS / RHEL * Resolve HTTP request returned status code 429 (Too Many Requests) * Resolve handling of maximum path length calculation * Resolve 'The parent item is not in the local database' * Resolve Correctly handle file case sensitivity issues in same folder * Update unit files documentation link ## 2.1.1 - 2018-08-14 ### Fixed * Fix handling no remote delete of remote directories when using --no-remote-delete * Fix handling of no permission to access a local file / corrupt local file * Fix application crash when unable to access login.microsoft.com upon application startup ### Added * Build instructions for openSUSE Leap 15.0 ## 2.1.0 - 2018-08-10 ### Fixed * Fix handling of database exit scenarios when there is zero disk space left on drive where the items database resides * Fix handling of incorrect database permissions * Fix handling of different database versions to automatically re-create tables if version mis-match * Fix handling timeout when accessing the Microsoft OneDrive Service * Fix localFileModifiedTime to not use fraction seconds ### Added * Implement Feature: Add a progress bar for large uploads & downloads * Implement Feature: Make checkinterval for monitor configurable * Implement Feature: Upload Only Option that does not perform remote delete * Implement Feature: Add ability to skip symlinks * Add dependency, ebuild and build instructions for Gentoo distributions ### Changed * Build instructions for x86, x86_64 and ARM32 platforms * Travis CI files to automate building on x32, x64 and ARM32 architectures * Travis CI files to test built application against valid, invalid and problem files from previous issues ## 2.0.2 - 2018-07-18 ### Fixed * Fix systemd service install for builds with DESTDIR defined * Fix 'HTTP 412 - Precondition Failed' error handling * Gracefully handle OneDrive account password change * Update logic handling of --upload-only and --local-first ## 2.0.1 - 2018-07-11 ### Fixed * Resolve computeQuickXorHash generates a different hash when files are > 64Kb ## 2.0.0 - 2018-07-10 ### Fixed * Resolve conflict resolution issue during syncing - the client does not handle conflicts very well & keeps on adding the hostname to files * Resolve skilion #356 by adding additional check for 409 response from OneDrive * Resolve multiple versions of file shown on website after single upload * Resolve to gracefully fail when 'onedrive' process cannot get exclusive database lock * Resolve 'Key not found: fileSystemInfo' when then item is a remote item (OneDrive Personal) * Resolve skip_file config entry needs to be checked for any characters to escape * Resolve Microsoft Naming Convention not being followed correctly * Resolve Error when trying to upload a file with weird non printable characters present * Resolve Crash if file is locked by online editing (status code 423) * Resolve compilation issue with dmd-2.081.0 * Resolve skip_file configuration doesn't handle spaces or specified directory paths ### Added * Implement Feature: Add a flag to detect when the sync-folder is missing * Implement Travis CI for code testing ### Changed * Update Makefile to use DESTDIR variables * Update OneDrive Business maximum path length from 256 to 400 * Update OneDrive Business allowed characters for files and folders * Update sync_dir handling to use the absolute path for setting parameter to something other than ~/OneDrive via config file or command line * Update Fedora build instructions ## 1.1.2 - 2018-05-17 ### Fixed * Fix 4xx errors including (412 pre-condition, 409 conflict) * Fix Key not found: lastModifiedDateTime (OneDrive API change) * Fix configuration directory not found when run via init.d * Fix skilion Issues #73, #121, #132, #224, #257, #294, #295, #297, #298, #300, #306, #315, #320, #329, #334, #337, #341 ### Added * Add logging - log client activities to a file (/var/log/onedrive/%username%.onedrive.log or ~/onedrive.log) * Add https debugging as a flag * Add `--synchronize` to prevent from syncing when just blindly running the application * Add individual folder sync * Add sync from local directory first rather than download first then upload * Add upload long path check * Add upload only * Add check for max upload file size before attempting upload * Add systemd unit files for single & multi user configuration * Add init.d file for older init.d based services * Add Microsoft naming conventions and namespace validation for items that will be uploaded * Add remaining free space counter at client initialisation to avoid out of space upload issue * Add large file upload size check to align to OneDrive file size limitations * Add upload file size validation & retry if does not match * Add graceful handling of some fatal errors (OneDrive 5xx error handling) ## Unreleased - 2018-02-19 ### Fixed * Crash when the delta link is expired ### Changed * Disabled buffering on stdout ## 1.1.1 - 2018-01-20 ### Fixed * Wrong regex for parsing authentication uri ## 1.1.0 - 2018-01-19 ### Added * Support for shared folders (OneDrive Personal only) * `--download` option to only download changes * `DC` variable in Makefile to chose the compiler ### Changed * Print logs on stdout instead of stderr * Improve log messages ## 1.0.1 - 2017-08-01 ### Added * `--syncdir` option ### Changed * `--version` output simplified * Updated README ### Fixed * Fix crash caused by remotely deleted and recreated directories ## 1.0.0 - 2017-07-14 ### Added * `--version` option ================================================ FILE: config ================================================ # Configuration for OneDrive Linux Client # This file contains the list of supported configuration fields with their default values. # All values need to be enclosed in quotes # When changing a config option below, remove the '#' from the start of the config line # For a more detailed explanation of all config options below see docs/application-config-options.md or the man page. ## This is the config option for application id that used to identify itself to Microsoft OneDrive. #application_id = "d50ca740-c83f-4d1b-b616-12c519384f0c" ## This is the config option to change the Microsoft Azure Authentication Endpoint that the client uses to conform with data and security requirements that requires data to reside within the geographic borders of that country. #azure_ad_endpoint = "" ## This config option allows the locking of the client to a specific single tenant and will configure your client to use the specified tenant id in its Azure AD and Graph endpoint URIs, instead of "common". #azure_tenant_id = "" ## This config option allows the disabling of preserving local data by renaming the local file in the event of data conflict. If this is enabled, you will experience data loss on your local data as the local file will be over-written with data from OneDrive online. Use with care and caution. #bypass_data_preservation = "false" ## This config option is useful to prevent application startup & ongoing use in 'Monitor Mode' if the configured 'sync_dir' is a separate disk that is being mounted by your system. #check_nomount = "false" ## This config option is useful to prevent the sync of a *local* directory to Microsoft OneDrive. It will *not* check for this file online to prevent the download of directories to your local system. #check_nosync = "false" ## This config option defines the number of children in a path that is locally removed which will be classified as a 'big data delete' to safeguard large data removals - which are typically accidental local delete events. #classify_as_big_delete = "1000" ## This config option provides the capability to cleanup local files and folders if they are removed online. #cleanup_local_files = "false" ## This configuration setting manages the TCP connection timeout duration in seconds for HTTPS connections to Microsoft OneDrive when using the curl library. #connect_timeout = "10" ## This setting controls how the application handles the Microsoft SharePoint feature which modifies all PDF, MS Office & HTML files post upload, effectively breaking the integrity of your data online. #create_new_file_version = "false" ## This setting controls the timeout duration, in seconds, for when data is not received on an active connection to Microsoft OneDrive over HTTPS. #data_timeout = "60" ## This setting controls whether the curl library is configured to output additional data to assist with diagnosing HTTPS issues and problems. #debug_https = "false" ## This setting controls whether 'inotify' events should be delayed or not. #delay_inotify_processing = "false" ## This option determines whether the client will conduct integrity validation on files downloaded from Microsoft OneDrive. #disable_download_validation = "false" ## This setting controls whether GUI notifications are sent from the client to your display manager session. #disable_notifications = "false" ## This setting controls whether the application will set the permissions on files and directories using the values of 'sync_dir_permissions' and 'sync_file_permissions'. #disable_permission_set = "false" ## This option determines whether the client will conduct integrity validation on files uploaded to Microsoft OneDrive. #disable_upload_validation = "false" ## This option will include the running config of the application at application startup. #display_running_config = "false" ## This option will display file transfer metrics when enabled. #display_transfer_metrics = "false" ## This setting controls the libcurl DNS cache value. #dns_timeout = "60" ## This setting forces the client to only download data from Microsoft OneDrive and replicate that data locally. #download_only = "false" ## This setting controls the specific drive identifier the client will use when syncing with Microsoft OneDrive. #drive_id = "" ## This setting controls the application capability to test your application configuration without actually performing any real activity. #dry_run = "false" ## This setting controls the application logging all actions to a separate file. #enable_logging = "false" ## This setting controls the file fragment size when uploading large files to Microsoft OneDrive. #file_fragment_size = "10" ## This setting controls the application HTTP protocol version, downgrading to HTTP/1.1 when enabled. #force_http_11 = "false" ## This option, when enabled, forces the client to use a 'session' upload, which, when the 'file' is uploaded by the session, this includes the local timestamp of the file #force_session_upload = "false" ## This setting controls the application IP protocol used when communicating with Microsoft OneDrive. #ip_protocol_version = "0" ## This setting controls what the application considers the 'source of truth' for your data. #local_first = "false" ## This setting controls the custom application log path when 'enable_logging' has been enabled. #log_dir = "" ## This configuration option controls the number of seconds a cURL engine is considered stale and destroyed after last use. #max_curl_idle = "120" ## This configuration option controls how often a full scan of your data is performed in monitor mode. #monitor_fullscan_frequency = "12" ## This setting determines how often the sync loop runs in --monitor mode. #monitor_interval = "300" ## This configuration option controls suppression of frequent monitor log messages. #monitor_log_frequency = "12" ## This configuration option controls whether local deletes are replicated to OneDrive when using --upload-only. #no_remote_delete = "false" ## This setting controls whether the client logs GUI notifications when file actions occur. #notify_file_actions = "false" ## This configuration controls the maximum amount of time a file operation is allowed to take. #operation_timeout = "3600" ## Permanently delete online items when removed locally. Bypasses OneDrive recycle bin. #permanent_delete = "false" ## This setting limits the per-thread bandwidth used by the client. #rate_limit = "0" ## This configuration option controls whether the client operates in read-only mode. #read_only_auth_scope = "false" ## This configuration option allows you to specify the 'Recycle Bin' path for the application. This is only used if 'use_recycle_bin' is enabled. #recycle_bin_path = "/path/to/desired/location/" ## This option removes the local file after a successful upload to OneDrive. #remove_source_files = "false" ## This configuration controls whether a full resync is performed at application startup. #resync = "false" ## This option approves use of --resync, useful in automated environments. #resync_auth = "false" ## This option controls which directories are excluded from sync. #skip_dir = "" ## When enabled, skip_dir matches must be strict, full path matches only. #skip_dir_strict_match = "false" ## When enabled, skip dotfiles and dot folders from sync. #skip_dotfiles = "false" ## This setting controls which files are skipped during sync. #skip_file = "~*|.~*|*.tmp|*.swp|*.partial" ## Skip syncing files larger than this size in MB. #skip_size = "0" ## Skip symbolic links during sync. #skip_symlinks = "false" ## Reserve this much free disk space (in MB) to avoid disk full issues. #space_reservation = "50" ## Sync OneDrive Business shared folders that are shortcuts in 'My Files'. These will be stored in a local folder called 'Files Shared With Me'. #sync_business_shared_items = "false" ## Local directory to sync with OneDrive. #sync_dir = "~/OneDrive" ## Permissions to apply to created local directories. #sync_dir_permissions = "700" ## Permissions to apply to created local files. #sync_file_permissions = "600" ## Sync all root files in sync_dir when using sync_list. #sync_root_files = "false" ## Number of threads to use for upload/download. #threads = "8" ## File transfer ordering between client and OneDrive. #transfer_order = "default" ## Only upload changes to OneDrive, do not download from cloud. #upload_only = "false" ## Authenticate using the Microsoft OAuth2 Device Authorisation Flow #use_device_auth = "true" ## Single Sign-On (SSO) via Intune using the Microsoft Identity Device Broker #use_intune_sso = "true" ## This configuration option controls the application function to move online deleted files to a 'Recycle Bin' on your system. #use_recycle_bin = "false" ## Custom User-Agent string for requests to OneDrive. If you change this, you will get throttled by the Microsoft Graph API. Change with caution. #user_agent = "ISV|abraunegg|OneDrive Client for Linux/vX.Y.Z-A-bcdefghi" ## Enable webhook-based remote update notifications in monitor mode. #webhook_enabled = "false" ## Time in seconds before webhook subscription expires. #webhook_expiration_interval = "600" ## IP address to listen on for incoming webhook updates. #webhook_listening_host = "0.0.0.0" ## TCP port to listen on for incoming webhook updates. #webhook_listening_port = "8888" ## Public webhook URL for Microsoft to send notifications to. #webhook_public_url = "" ## Frequency (in seconds) to renew webhook subscription. #webhook_renewal_interval = "300" ## Frequency (in seconds) to retry a failed webhook subscription renewal. #webhook_retry_interval = "60" ## Write xattr metadata fields (createdBy, lastModifiedBy) to synced files. #write_xattr_data = "false" ================================================ FILE: configure ================================================ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. # Generated by GNU Autoconf 2.69 for onedrive v2.5.10. # # Report bugs to . # # # Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc. # # # This configure script is free software; the Free Software Foundation # gives unlimited permission to copy, distribute and modify it. ## -------------------- ## ## M4sh Initialization. ## ## -------------------- ## # Be more Bourne compatible DUALCASE=1; export DUALCASE # for MKS sh if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on ${1+"$@"}, which # is contrary to our usage. Disable this feature. alias -g '${1+"$@"}'='"$@"' setopt NO_GLOB_SUBST else case `(set -o) 2>/dev/null` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi as_nl=' ' export as_nl # Printing a long string crashes Solaris 7 /usr/bin/printf. as_echo='\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo$as_echo # Prefer a ksh shell builtin over an external printf program on Solaris, # but without wasting forks for bash or zsh. if test -z "$BASH_VERSION$ZSH_VERSION" \ && (test "X`print -r -- $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='print -r --' as_echo_n='print -rn --' elif (test "X`printf %s $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='printf %s\n' as_echo_n='printf %s' else if test "X`(/usr/ucb/echo -n -n $as_echo) 2>/dev/null`" = "X-n $as_echo"; then as_echo_body='eval /usr/ucb/echo -n "$1$as_nl"' as_echo_n='/usr/ucb/echo -n' else as_echo_body='eval expr "X$1" : "X\\(.*\\)"' as_echo_n_body='eval arg=$1; case $arg in #( *"$as_nl"*) expr "X$arg" : "X\\(.*\\)$as_nl"; arg=`expr "X$arg" : ".*$as_nl\\(.*\\)"`;; esac; expr "X$arg" : "X\\(.*\\)" | tr -d "$as_nl" ' export as_echo_n_body as_echo_n='sh -c $as_echo_n_body as_echo' fi export as_echo_body as_echo='sh -c $as_echo_body as_echo' fi # The user is always right. if test "${PATH_SEPARATOR+set}" != set; then PATH_SEPARATOR=: (PATH='/bin;/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 && { (PATH='/bin:/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 || PATH_SEPARATOR=';' } fi # IFS # We need space, tab and new line, in precisely that order. Quoting is # there to prevent editors from complaining about space-tab. # (If _AS_PATH_WALK were called with IFS unset, it would disable word # splitting by setting IFS to empty value.) IFS=" "" $as_nl" # Find who we are. Look in the path if we contain no directory separator. as_myself= case $0 in #(( *[\\/]* ) as_myself=$0 ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. test -r "$as_dir/$0" && as_myself=$as_dir/$0 && break done IFS=$as_save_IFS ;; esac # We did not find ourselves, most probably we were run as `sh COMMAND' # in which case we are not to be found in the path. if test "x$as_myself" = x; then as_myself=$0 fi if test ! -f "$as_myself"; then $as_echo "$as_myself: error: cannot find myself; rerun with an absolute file name" >&2 exit 1 fi # Unset variables that we do not need and which cause bugs (e.g. in # pre-3.0 UWIN ksh). But do not cause bugs in bash 2.01; the "|| exit 1" # suppresses any "Segmentation fault" message there. '((' could # trigger a bug in pdksh 5.2.14. for as_var in BASH_ENV ENV MAIL MAILPATH do eval test x\${$as_var+set} = xset \ && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || : done PS1='$ ' PS2='> ' PS4='+ ' # NLS nuisances. LC_ALL=C export LC_ALL LANGUAGE=C export LANGUAGE # CDPATH. (unset CDPATH) >/dev/null 2>&1 && unset CDPATH # Use a proper internal environment variable to ensure we don't fall # into an infinite loop, continuously re-executing ourselves. if test x"${_as_can_reexec}" != xno && test "x$CONFIG_SHELL" != x; then _as_can_reexec=no; export _as_can_reexec; # We cannot yet assume a decent shell, so we have to provide a # neutralization value for shells without unset; and this also # works around shells that cannot unset nonexistent variables. # Preserve -v and -x to the replacement shell. BASH_ENV=/dev/null ENV=/dev/null (unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV case $- in # (((( *v*x* | *x*v* ) as_opts=-vx ;; *v* ) as_opts=-v ;; *x* ) as_opts=-x ;; * ) as_opts= ;; esac exec $CONFIG_SHELL $as_opts "$as_myself" ${1+"$@"} # Admittedly, this is quite paranoid, since all the known shells bail # out after a failed `exec'. $as_echo "$0: could not re-execute with $CONFIG_SHELL" >&2 as_fn_exit 255 fi # We don't want this to propagate to other subprocesses. { _as_can_reexec=; unset _as_can_reexec;} if test "x$CONFIG_SHELL" = x; then as_bourne_compatible="if test -n \"\${ZSH_VERSION+set}\" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on \${1+\"\$@\"}, which # is contrary to our usage. Disable this feature. alias -g '\${1+\"\$@\"}'='\"\$@\"' setopt NO_GLOB_SUBST else case \`(set -o) 2>/dev/null\` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi " as_required="as_fn_return () { (exit \$1); } as_fn_success () { as_fn_return 0; } as_fn_failure () { as_fn_return 1; } as_fn_ret_success () { return 0; } as_fn_ret_failure () { return 1; } exitcode=0 as_fn_success || { exitcode=1; echo as_fn_success failed.; } as_fn_failure && { exitcode=1; echo as_fn_failure succeeded.; } as_fn_ret_success || { exitcode=1; echo as_fn_ret_success failed.; } as_fn_ret_failure && { exitcode=1; echo as_fn_ret_failure succeeded.; } if ( set x; as_fn_ret_success y && test x = \"\$1\" ); then : else exitcode=1; echo positional parameters were not saved. fi test x\$exitcode = x0 || exit 1 test -x / || exit 1" as_suggested=" as_lineno_1=";as_suggested=$as_suggested$LINENO;as_suggested=$as_suggested" as_lineno_1a=\$LINENO as_lineno_2=";as_suggested=$as_suggested$LINENO;as_suggested=$as_suggested" as_lineno_2a=\$LINENO eval 'test \"x\$as_lineno_1'\$as_run'\" != \"x\$as_lineno_2'\$as_run'\" && test \"x\`expr \$as_lineno_1'\$as_run' + 1\`\" = \"x\$as_lineno_2'\$as_run'\"' || exit 1" if (eval "$as_required") 2>/dev/null; then : as_have_required=yes else as_have_required=no fi if test x$as_have_required = xyes && (eval "$as_suggested") 2>/dev/null; then : else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR as_found=false for as_dir in /bin$PATH_SEPARATOR/usr/bin$PATH_SEPARATOR$PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. as_found=: case $as_dir in #( /*) for as_base in sh bash ksh sh5; do # Try only shells that exist, to save several forks. as_shell=$as_dir/$as_base if { test -f "$as_shell" || test -f "$as_shell.exe"; } && { $as_echo "$as_bourne_compatible""$as_required" | as_run=a "$as_shell"; } 2>/dev/null; then : CONFIG_SHELL=$as_shell as_have_required=yes if { $as_echo "$as_bourne_compatible""$as_suggested" | as_run=a "$as_shell"; } 2>/dev/null; then : break 2 fi fi done;; esac as_found=false done $as_found || { if { test -f "$SHELL" || test -f "$SHELL.exe"; } && { $as_echo "$as_bourne_compatible""$as_required" | as_run=a "$SHELL"; } 2>/dev/null; then : CONFIG_SHELL=$SHELL as_have_required=yes fi; } IFS=$as_save_IFS if test "x$CONFIG_SHELL" != x; then : export CONFIG_SHELL # We cannot yet assume a decent shell, so we have to provide a # neutralization value for shells without unset; and this also # works around shells that cannot unset nonexistent variables. # Preserve -v and -x to the replacement shell. BASH_ENV=/dev/null ENV=/dev/null (unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV case $- in # (((( *v*x* | *x*v* ) as_opts=-vx ;; *v* ) as_opts=-v ;; *x* ) as_opts=-x ;; * ) as_opts= ;; esac exec $CONFIG_SHELL $as_opts "$as_myself" ${1+"$@"} # Admittedly, this is quite paranoid, since all the known shells bail # out after a failed `exec'. $as_echo "$0: could not re-execute with $CONFIG_SHELL" >&2 exit 255 fi if test x$as_have_required = xno; then : $as_echo "$0: This script requires a shell more modern than all" $as_echo "$0: the shells that I found on your system." if test x${ZSH_VERSION+set} = xset ; then $as_echo "$0: In particular, zsh $ZSH_VERSION has bugs and should" $as_echo "$0: be upgraded to zsh 4.3.4 or later." else $as_echo "$0: Please tell bug-autoconf@gnu.org and $0: https://github.com/abraunegg/onedrive about your $0: system, including any error possibly output before this $0: message. Then install a modern shell, or manually run $0: the script under such a shell if you do have one." fi exit 1 fi fi fi SHELL=${CONFIG_SHELL-/bin/sh} export SHELL # Unset more variables known to interfere with behavior of common tools. CLICOLOR_FORCE= GREP_OPTIONS= unset CLICOLOR_FORCE GREP_OPTIONS ## --------------------- ## ## M4sh Shell Functions. ## ## --------------------- ## # as_fn_unset VAR # --------------- # Portably unset VAR. as_fn_unset () { { eval $1=; unset $1;} } as_unset=as_fn_unset # as_fn_set_status STATUS # ----------------------- # Set $? to STATUS, without forking. as_fn_set_status () { return $1 } # as_fn_set_status # as_fn_exit STATUS # ----------------- # Exit the shell with STATUS, even in a "trap 0" or "set -e" context. as_fn_exit () { set +e as_fn_set_status $1 exit $1 } # as_fn_exit # as_fn_mkdir_p # ------------- # Create "$as_dir" as a directory, including parents if necessary. as_fn_mkdir_p () { case $as_dir in #( -*) as_dir=./$as_dir;; esac test -d "$as_dir" || eval $as_mkdir_p || { as_dirs= while :; do case $as_dir in #( *\'*) as_qdir=`$as_echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #'( *) as_qdir=$as_dir;; esac as_dirs="'$as_qdir' $as_dirs" as_dir=`$as_dirname -- "$as_dir" || $as_expr X"$as_dir" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_dir" : 'X\(//\)[^/]' \| \ X"$as_dir" : 'X\(//\)$' \| \ X"$as_dir" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_dir" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` test -d "$as_dir" && break done test -z "$as_dirs" || eval "mkdir $as_dirs" } || test -d "$as_dir" || as_fn_error $? "cannot create directory $as_dir" } # as_fn_mkdir_p # as_fn_executable_p FILE # ----------------------- # Test if FILE is an executable regular file. as_fn_executable_p () { test -f "$1" && test -x "$1" } # as_fn_executable_p # as_fn_append VAR VALUE # ---------------------- # Append the text in VALUE to the end of the definition contained in VAR. Take # advantage of any shell optimizations that allow amortized linear growth over # repeated appends, instead of the typical quadratic growth present in naive # implementations. if (eval "as_var=1; as_var+=2; test x\$as_var = x12") 2>/dev/null; then : eval 'as_fn_append () { eval $1+=\$2 }' else as_fn_append () { eval $1=\$$1\$2 } fi # as_fn_append # as_fn_arith ARG... # ------------------ # Perform arithmetic evaluation on the ARGs, and store the result in the # global $as_val. Take advantage of shells that can avoid forks. The arguments # must be portable across $(()) and expr. if (eval "test \$(( 1 + 1 )) = 2") 2>/dev/null; then : eval 'as_fn_arith () { as_val=$(( $* )) }' else as_fn_arith () { as_val=`expr "$@" || test $? -eq 1` } fi # as_fn_arith # as_fn_error STATUS ERROR [LINENO LOG_FD] # ---------------------------------------- # Output "`basename $0`: error: ERROR" to stderr. If LINENO and LOG_FD are # provided, also output the error to LOG_FD, referencing LINENO. Then exit the # script with STATUS, using 1 if that was 0. as_fn_error () { as_status=$1; test $as_status -eq 0 && as_status=1 if test "$4"; then as_lineno=${as_lineno-"$3"} as_lineno_stack=as_lineno_stack=$as_lineno_stack $as_echo "$as_me:${as_lineno-$LINENO}: error: $2" >&$4 fi $as_echo "$as_me: error: $2" >&2 as_fn_exit $as_status } # as_fn_error if expr a : '\(a\)' >/dev/null 2>&1 && test "X`expr 00001 : '.*\(...\)'`" = X001; then as_expr=expr else as_expr=false fi if (basename -- /) >/dev/null 2>&1 && test "X`basename -- / 2>&1`" = "X/"; then as_basename=basename else as_basename=false fi if (as_dir=`dirname -- /` && test "X$as_dir" = X/) >/dev/null 2>&1; then as_dirname=dirname else as_dirname=false fi as_me=`$as_basename -- "$0" || $as_expr X/"$0" : '.*/\([^/][^/]*\)/*$' \| \ X"$0" : 'X\(//\)$' \| \ X"$0" : 'X\(/\)' \| . 2>/dev/null || $as_echo X/"$0" | sed '/^.*\/\([^/][^/]*\)\/*$/{ s//\1/ q } /^X\/\(\/\/\)$/{ s//\1/ q } /^X\/\(\/\).*/{ s//\1/ q } s/.*/./; q'` # Avoid depending upon Character Ranges. as_cr_letters='abcdefghijklmnopqrstuvwxyz' as_cr_LETTERS='ABCDEFGHIJKLMNOPQRSTUVWXYZ' as_cr_Letters=$as_cr_letters$as_cr_LETTERS as_cr_digits='0123456789' as_cr_alnum=$as_cr_Letters$as_cr_digits as_lineno_1=$LINENO as_lineno_1a=$LINENO as_lineno_2=$LINENO as_lineno_2a=$LINENO eval 'test "x$as_lineno_1'$as_run'" != "x$as_lineno_2'$as_run'" && test "x`expr $as_lineno_1'$as_run' + 1`" = "x$as_lineno_2'$as_run'"' || { # Blame Lee E. McMahon (1931-1989) for sed's syntax. :-) sed -n ' p /[$]LINENO/= ' <$as_myself | sed ' s/[$]LINENO.*/&-/ t lineno b :lineno N :loop s/[$]LINENO\([^'$as_cr_alnum'_].*\n\)\(.*\)/\2\1\2/ t loop s/-\n.*// ' >$as_me.lineno && chmod +x "$as_me.lineno" || { $as_echo "$as_me: error: cannot create $as_me.lineno; rerun with a POSIX shell" >&2; as_fn_exit 1; } # If we had to re-execute with $CONFIG_SHELL, we're ensured to have # already done that, so ensure we don't try to do so again and fall # in an infinite loop. This has already happened in practice. _as_can_reexec=no; export _as_can_reexec # Don't try to exec as it changes $[0], causing all sort of problems # (the dirname of $[0] is not the place where we might find the # original and so on. Autoconf is especially sensitive to this). . "./$as_me.lineno" # Exit status is that of the last command. exit } ECHO_C= ECHO_N= ECHO_T= case `echo -n x` in #((((( -n*) case `echo 'xy\c'` in *c*) ECHO_T=' ';; # ECHO_T is single tab character. xy) ECHO_C='\c';; *) echo `echo ksh88 bug on AIX 6.1` > /dev/null ECHO_T=' ';; esac;; *) ECHO_N='-n';; esac rm -f conf$$ conf$$.exe conf$$.file if test -d conf$$.dir; then rm -f conf$$.dir/conf$$.file else rm -f conf$$.dir mkdir conf$$.dir 2>/dev/null fi if (echo >conf$$.file) 2>/dev/null; then if ln -s conf$$.file conf$$ 2>/dev/null; then as_ln_s='ln -s' # ... but there are two gotchas: # 1) On MSYS, both `ln -s file dir' and `ln file dir' fail. # 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable. # In both cases, we have to default to `cp -pR'. ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe || as_ln_s='cp -pR' elif ln conf$$.file conf$$ 2>/dev/null; then as_ln_s=ln else as_ln_s='cp -pR' fi else as_ln_s='cp -pR' fi rm -f conf$$ conf$$.exe conf$$.dir/conf$$.file conf$$.file rmdir conf$$.dir 2>/dev/null if mkdir -p . 2>/dev/null; then as_mkdir_p='mkdir -p "$as_dir"' else test -d ./-p && rmdir ./-p as_mkdir_p=false fi as_test_x='test -x' as_executable_p=as_fn_executable_p # Sed expression to map a string onto a valid CPP name. as_tr_cpp="eval sed 'y%*$as_cr_letters%P$as_cr_LETTERS%;s%[^_$as_cr_alnum]%_%g'" # Sed expression to map a string onto a valid variable name. as_tr_sh="eval sed 'y%*+%pp%;s%[^_$as_cr_alnum]%_%g'" test -n "$DJDIR" || exec 7<&0 &1 # Name of the host. # hostname on some systems (SVR3.2, old GNU/Linux) returns a bogus exit status, # so uname gets run too. ac_hostname=`(hostname || uname -n) 2>/dev/null | sed 1q` # # Initializations. # ac_default_prefix=/usr/local ac_clean_files= ac_config_libobj_dir=. LIBOBJS= cross_compiling=no subdirs= MFLAGS= MAKEFLAGS= # Identity of this package. PACKAGE_NAME='onedrive' PACKAGE_TARNAME='onedrive' PACKAGE_VERSION='v2.5.10' PACKAGE_STRING='onedrive v2.5.10' PACKAGE_BUGREPORT='https://github.com/abraunegg/onedrive' PACKAGE_URL='' ac_unique_file="src/main.d" ac_subst_vars='LTLIBOBJS LIBOBJS DEBUG FISH_COMPLETION_DIR ZSH_COMPLETION_DIR BASH_COMPLETION_DIR bashcompdir COMPLETIONS dynamic_linker_LIBS bsd_inotify_LIBS NOTIFICATIONS notify_LIBS notify_CFLAGS HAVE_SYSTEMD systemduserunitdir systemdsystemunitdir enable_dbus dbus_LIBS dbus_CFLAGS sqlite_LIBS sqlite_CFLAGS curl_LIBS curl_CFLAGS WERROR_DCFLAG OUTPUT_DCFLAG LINKER_DCFLAG VERSION_DCFLAG RELEASE_DCFLAGS DEBUG_DCFLAGS PACKAGE_DATE PKG_CONFIG_LIBDIR PKG_CONFIG_PATH PKG_CONFIG INSTALL_DATA INSTALL_SCRIPT INSTALL_PROGRAM DCFLAGS DC target_alias host_alias build_alias LIBS ECHO_T ECHO_N ECHO_C DEFS mandir localedir libdir psdir pdfdir dvidir htmldir infodir docdir oldincludedir includedir localstatedir sharedstatedir sysconfdir datadir datarootdir libexecdir sbindir bindir program_transform_name prefix exec_prefix PACKAGE_URL PACKAGE_BUGREPORT PACKAGE_STRING PACKAGE_VERSION PACKAGE_TARNAME PACKAGE_NAME PATH_SEPARATOR SHELL' ac_subst_files='' ac_user_opts=' enable_option_checking enable_version_check with_systemdsystemunitdir with_systemduserunitdir enable_notifications enable_completions with_bash_completion_dir with_zsh_completion_dir with_fish_completion_dir enable_debug ' ac_precious_vars='build_alias host_alias target_alias DC DCFLAGS PKG_CONFIG PKG_CONFIG_PATH PKG_CONFIG_LIBDIR curl_CFLAGS curl_LIBS sqlite_CFLAGS sqlite_LIBS dbus_CFLAGS dbus_LIBS notify_CFLAGS notify_LIBS bashcompdir' # Initialize some variables set by options. ac_init_help= ac_init_version=false ac_unrecognized_opts= ac_unrecognized_sep= # The variables have the same names as the options, with # dashes changed to underlines. cache_file=/dev/null exec_prefix=NONE no_create= no_recursion= prefix=NONE program_prefix=NONE program_suffix=NONE program_transform_name=s,x,x, silent= site= srcdir= verbose= x_includes=NONE x_libraries=NONE # Installation directory options. # These are left unexpanded so users can "make install exec_prefix=/foo" # and all the variables that are supposed to be based on exec_prefix # by default will actually change. # Use braces instead of parens because sh, perl, etc. also accept them. # (The list follows the same order as the GNU Coding Standards.) bindir='${exec_prefix}/bin' sbindir='${exec_prefix}/sbin' libexecdir='${exec_prefix}/libexec' datarootdir='${prefix}/share' datadir='${datarootdir}' sysconfdir='${prefix}/etc' sharedstatedir='${prefix}/com' localstatedir='${prefix}/var' includedir='${prefix}/include' oldincludedir='/usr/include' docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' infodir='${datarootdir}/info' htmldir='${docdir}' dvidir='${docdir}' pdfdir='${docdir}' psdir='${docdir}' libdir='${exec_prefix}/lib' localedir='${datarootdir}/locale' mandir='${datarootdir}/man' ac_prev= ac_dashdash= for ac_option do # If the previous option needs an argument, assign it. if test -n "$ac_prev"; then eval $ac_prev=\$ac_option ac_prev= continue fi case $ac_option in *=?*) ac_optarg=`expr "X$ac_option" : '[^=]*=\(.*\)'` ;; *=) ac_optarg= ;; *) ac_optarg=yes ;; esac # Accept the important Cygnus configure options, so we can diagnose typos. case $ac_dashdash$ac_option in --) ac_dashdash=yes ;; -bindir | --bindir | --bindi | --bind | --bin | --bi) ac_prev=bindir ;; -bindir=* | --bindir=* | --bindi=* | --bind=* | --bin=* | --bi=*) bindir=$ac_optarg ;; -build | --build | --buil | --bui | --bu) ac_prev=build_alias ;; -build=* | --build=* | --buil=* | --bui=* | --bu=*) build_alias=$ac_optarg ;; -cache-file | --cache-file | --cache-fil | --cache-fi \ | --cache-f | --cache- | --cache | --cach | --cac | --ca | --c) ac_prev=cache_file ;; -cache-file=* | --cache-file=* | --cache-fil=* | --cache-fi=* \ | --cache-f=* | --cache-=* | --cache=* | --cach=* | --cac=* | --ca=* | --c=*) cache_file=$ac_optarg ;; --config-cache | -C) cache_file=config.cache ;; -datadir | --datadir | --datadi | --datad) ac_prev=datadir ;; -datadir=* | --datadir=* | --datadi=* | --datad=*) datadir=$ac_optarg ;; -datarootdir | --datarootdir | --datarootdi | --datarootd | --dataroot \ | --dataroo | --dataro | --datar) ac_prev=datarootdir ;; -datarootdir=* | --datarootdir=* | --datarootdi=* | --datarootd=* \ | --dataroot=* | --dataroo=* | --dataro=* | --datar=*) datarootdir=$ac_optarg ;; -disable-* | --disable-*) ac_useropt=`expr "x$ac_option" : 'x-*disable-\(.*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid feature name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "enable_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--disable-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval enable_$ac_useropt=no ;; -docdir | --docdir | --docdi | --doc | --do) ac_prev=docdir ;; -docdir=* | --docdir=* | --docdi=* | --doc=* | --do=*) docdir=$ac_optarg ;; -dvidir | --dvidir | --dvidi | --dvid | --dvi | --dv) ac_prev=dvidir ;; -dvidir=* | --dvidir=* | --dvidi=* | --dvid=* | --dvi=* | --dv=*) dvidir=$ac_optarg ;; -enable-* | --enable-*) ac_useropt=`expr "x$ac_option" : 'x-*enable-\([^=]*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid feature name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "enable_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--enable-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval enable_$ac_useropt=\$ac_optarg ;; -exec-prefix | --exec_prefix | --exec-prefix | --exec-prefi \ | --exec-pref | --exec-pre | --exec-pr | --exec-p | --exec- \ | --exec | --exe | --ex) ac_prev=exec_prefix ;; -exec-prefix=* | --exec_prefix=* | --exec-prefix=* | --exec-prefi=* \ | --exec-pref=* | --exec-pre=* | --exec-pr=* | --exec-p=* | --exec-=* \ | --exec=* | --exe=* | --ex=*) exec_prefix=$ac_optarg ;; -gas | --gas | --ga | --g) # Obsolete; use --with-gas. with_gas=yes ;; -help | --help | --hel | --he | -h) ac_init_help=long ;; -help=r* | --help=r* | --hel=r* | --he=r* | -hr*) ac_init_help=recursive ;; -help=s* | --help=s* | --hel=s* | --he=s* | -hs*) ac_init_help=short ;; -host | --host | --hos | --ho) ac_prev=host_alias ;; -host=* | --host=* | --hos=* | --ho=*) host_alias=$ac_optarg ;; -htmldir | --htmldir | --htmldi | --htmld | --html | --htm | --ht) ac_prev=htmldir ;; -htmldir=* | --htmldir=* | --htmldi=* | --htmld=* | --html=* | --htm=* \ | --ht=*) htmldir=$ac_optarg ;; -includedir | --includedir | --includedi | --included | --include \ | --includ | --inclu | --incl | --inc) ac_prev=includedir ;; -includedir=* | --includedir=* | --includedi=* | --included=* | --include=* \ | --includ=* | --inclu=* | --incl=* | --inc=*) includedir=$ac_optarg ;; -infodir | --infodir | --infodi | --infod | --info | --inf) ac_prev=infodir ;; -infodir=* | --infodir=* | --infodi=* | --infod=* | --info=* | --inf=*) infodir=$ac_optarg ;; -libdir | --libdir | --libdi | --libd) ac_prev=libdir ;; -libdir=* | --libdir=* | --libdi=* | --libd=*) libdir=$ac_optarg ;; -libexecdir | --libexecdir | --libexecdi | --libexecd | --libexec \ | --libexe | --libex | --libe) ac_prev=libexecdir ;; -libexecdir=* | --libexecdir=* | --libexecdi=* | --libexecd=* | --libexec=* \ | --libexe=* | --libex=* | --libe=*) libexecdir=$ac_optarg ;; -localedir | --localedir | --localedi | --localed | --locale) ac_prev=localedir ;; -localedir=* | --localedir=* | --localedi=* | --localed=* | --locale=*) localedir=$ac_optarg ;; -localstatedir | --localstatedir | --localstatedi | --localstated \ | --localstate | --localstat | --localsta | --localst | --locals) ac_prev=localstatedir ;; -localstatedir=* | --localstatedir=* | --localstatedi=* | --localstated=* \ | --localstate=* | --localstat=* | --localsta=* | --localst=* | --locals=*) localstatedir=$ac_optarg ;; -mandir | --mandir | --mandi | --mand | --man | --ma | --m) ac_prev=mandir ;; -mandir=* | --mandir=* | --mandi=* | --mand=* | --man=* | --ma=* | --m=*) mandir=$ac_optarg ;; -nfp | --nfp | --nf) # Obsolete; use --without-fp. with_fp=no ;; -no-create | --no-create | --no-creat | --no-crea | --no-cre \ | --no-cr | --no-c | -n) no_create=yes ;; -no-recursion | --no-recursion | --no-recursio | --no-recursi \ | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r) no_recursion=yes ;; -oldincludedir | --oldincludedir | --oldincludedi | --oldincluded \ | --oldinclude | --oldinclud | --oldinclu | --oldincl | --oldinc \ | --oldin | --oldi | --old | --ol | --o) ac_prev=oldincludedir ;; -oldincludedir=* | --oldincludedir=* | --oldincludedi=* | --oldincluded=* \ | --oldinclude=* | --oldinclud=* | --oldinclu=* | --oldincl=* | --oldinc=* \ | --oldin=* | --oldi=* | --old=* | --ol=* | --o=*) oldincludedir=$ac_optarg ;; -prefix | --prefix | --prefi | --pref | --pre | --pr | --p) ac_prev=prefix ;; -prefix=* | --prefix=* | --prefi=* | --pref=* | --pre=* | --pr=* | --p=*) prefix=$ac_optarg ;; -program-prefix | --program-prefix | --program-prefi | --program-pref \ | --program-pre | --program-pr | --program-p) ac_prev=program_prefix ;; -program-prefix=* | --program-prefix=* | --program-prefi=* \ | --program-pref=* | --program-pre=* | --program-pr=* | --program-p=*) program_prefix=$ac_optarg ;; -program-suffix | --program-suffix | --program-suffi | --program-suff \ | --program-suf | --program-su | --program-s) ac_prev=program_suffix ;; -program-suffix=* | --program-suffix=* | --program-suffi=* \ | --program-suff=* | --program-suf=* | --program-su=* | --program-s=*) program_suffix=$ac_optarg ;; -program-transform-name | --program-transform-name \ | --program-transform-nam | --program-transform-na \ | --program-transform-n | --program-transform- \ | --program-transform | --program-transfor \ | --program-transfo | --program-transf \ | --program-trans | --program-tran \ | --progr-tra | --program-tr | --program-t) ac_prev=program_transform_name ;; -program-transform-name=* | --program-transform-name=* \ | --program-transform-nam=* | --program-transform-na=* \ | --program-transform-n=* | --program-transform-=* \ | --program-transform=* | --program-transfor=* \ | --program-transfo=* | --program-transf=* \ | --program-trans=* | --program-tran=* \ | --progr-tra=* | --program-tr=* | --program-t=*) program_transform_name=$ac_optarg ;; -pdfdir | --pdfdir | --pdfdi | --pdfd | --pdf | --pd) ac_prev=pdfdir ;; -pdfdir=* | --pdfdir=* | --pdfdi=* | --pdfd=* | --pdf=* | --pd=*) pdfdir=$ac_optarg ;; -psdir | --psdir | --psdi | --psd | --ps) ac_prev=psdir ;; -psdir=* | --psdir=* | --psdi=* | --psd=* | --ps=*) psdir=$ac_optarg ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil) silent=yes ;; -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) ac_prev=sbindir ;; -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ | --sbi=* | --sb=*) sbindir=$ac_optarg ;; -sharedstatedir | --sharedstatedir | --sharedstatedi \ | --sharedstated | --sharedstate | --sharedstat | --sharedsta \ | --sharedst | --shareds | --shared | --share | --shar \ | --sha | --sh) ac_prev=sharedstatedir ;; -sharedstatedir=* | --sharedstatedir=* | --sharedstatedi=* \ | --sharedstated=* | --sharedstate=* | --sharedstat=* | --sharedsta=* \ | --sharedst=* | --shareds=* | --shared=* | --share=* | --shar=* \ | --sha=* | --sh=*) sharedstatedir=$ac_optarg ;; -site | --site | --sit) ac_prev=site ;; -site=* | --site=* | --sit=*) site=$ac_optarg ;; -srcdir | --srcdir | --srcdi | --srcd | --src | --sr) ac_prev=srcdir ;; -srcdir=* | --srcdir=* | --srcdi=* | --srcd=* | --src=* | --sr=*) srcdir=$ac_optarg ;; -sysconfdir | --sysconfdir | --sysconfdi | --sysconfd | --sysconf \ | --syscon | --sysco | --sysc | --sys | --sy) ac_prev=sysconfdir ;; -sysconfdir=* | --sysconfdir=* | --sysconfdi=* | --sysconfd=* | --sysconf=* \ | --syscon=* | --sysco=* | --sysc=* | --sys=* | --sy=*) sysconfdir=$ac_optarg ;; -target | --target | --targe | --targ | --tar | --ta | --t) ac_prev=target_alias ;; -target=* | --target=* | --targe=* | --targ=* | --tar=* | --ta=* | --t=*) target_alias=$ac_optarg ;; -v | -verbose | --verbose | --verbos | --verbo | --verb) verbose=yes ;; -version | --version | --versio | --versi | --vers | -V) ac_init_version=: ;; -with-* | --with-*) ac_useropt=`expr "x$ac_option" : 'x-*with-\([^=]*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid package name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "with_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--with-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval with_$ac_useropt=\$ac_optarg ;; -without-* | --without-*) ac_useropt=`expr "x$ac_option" : 'x-*without-\(.*\)'` # Reject names that are not valid shell variable names. expr "x$ac_useropt" : ".*[^-+._$as_cr_alnum]" >/dev/null && as_fn_error $? "invalid package name: $ac_useropt" ac_useropt_orig=$ac_useropt ac_useropt=`$as_echo "$ac_useropt" | sed 's/[-+.]/_/g'` case $ac_user_opts in *" "with_$ac_useropt" "*) ;; *) ac_unrecognized_opts="$ac_unrecognized_opts$ac_unrecognized_sep--without-$ac_useropt_orig" ac_unrecognized_sep=', ';; esac eval with_$ac_useropt=no ;; --x) # Obsolete; use --with-x. with_x=yes ;; -x-includes | --x-includes | --x-include | --x-includ | --x-inclu \ | --x-incl | --x-inc | --x-in | --x-i) ac_prev=x_includes ;; -x-includes=* | --x-includes=* | --x-include=* | --x-includ=* | --x-inclu=* \ | --x-incl=* | --x-inc=* | --x-in=* | --x-i=*) x_includes=$ac_optarg ;; -x-libraries | --x-libraries | --x-librarie | --x-librari \ | --x-librar | --x-libra | --x-libr | --x-lib | --x-li | --x-l) ac_prev=x_libraries ;; -x-libraries=* | --x-libraries=* | --x-librarie=* | --x-librari=* \ | --x-librar=* | --x-libra=* | --x-libr=* | --x-lib=* | --x-li=* | --x-l=*) x_libraries=$ac_optarg ;; -*) as_fn_error $? "unrecognized option: \`$ac_option' Try \`$0 --help' for more information" ;; *=*) ac_envvar=`expr "x$ac_option" : 'x\([^=]*\)='` # Reject names that are not valid shell variable names. case $ac_envvar in #( '' | [0-9]* | *[!_$as_cr_alnum]* ) as_fn_error $? "invalid variable name: \`$ac_envvar'" ;; esac eval $ac_envvar=\$ac_optarg export $ac_envvar ;; *) # FIXME: should be removed in autoconf 3.0. $as_echo "$as_me: WARNING: you should use --build, --host, --target" >&2 expr "x$ac_option" : ".*[^-._$as_cr_alnum]" >/dev/null && $as_echo "$as_me: WARNING: invalid host type: $ac_option" >&2 : "${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}" ;; esac done if test -n "$ac_prev"; then ac_option=--`echo $ac_prev | sed 's/_/-/g'` as_fn_error $? "missing argument to $ac_option" fi if test -n "$ac_unrecognized_opts"; then case $enable_option_checking in no) ;; fatal) as_fn_error $? "unrecognized options: $ac_unrecognized_opts" ;; *) $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2 ;; esac fi # Check all directory arguments for consistency. for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ datadir sysconfdir sharedstatedir localstatedir includedir \ oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ libdir localedir mandir do eval ac_val=\$$ac_var # Remove trailing slashes. case $ac_val in */ ) ac_val=`expr "X$ac_val" : 'X\(.*[^/]\)' \| "X$ac_val" : 'X\(.*\)'` eval $ac_var=\$ac_val;; esac # Be sure to have absolute directory names. case $ac_val in [\\/$]* | ?:[\\/]* ) continue;; NONE | '' ) case $ac_var in *prefix ) continue;; esac;; esac as_fn_error $? "expected an absolute directory name for --$ac_var: $ac_val" done # There might be people who depend on the old broken behavior: `$host' # used to hold the argument of --host etc. # FIXME: To remove some day. build=$build_alias host=$host_alias target=$target_alias # FIXME: To remove some day. if test "x$host_alias" != x; then if test "x$build_alias" = x; then cross_compiling=maybe elif test "x$build_alias" != "x$host_alias"; then cross_compiling=yes fi fi ac_tool_prefix= test -n "$host_alias" && ac_tool_prefix=$host_alias- test "$silent" = yes && exec 6>/dev/null ac_pwd=`pwd` && test -n "$ac_pwd" && ac_ls_di=`ls -di .` && ac_pwd_ls_di=`cd "$ac_pwd" && ls -di .` || as_fn_error $? "working directory cannot be determined" test "X$ac_ls_di" = "X$ac_pwd_ls_di" || as_fn_error $? "pwd does not report name of working directory" # Find the source files, if location was not specified. if test -z "$srcdir"; then ac_srcdir_defaulted=yes # Try the directory containing this script, then the parent directory. ac_confdir=`$as_dirname -- "$as_myself" || $as_expr X"$as_myself" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_myself" : 'X\(//\)[^/]' \| \ X"$as_myself" : 'X\(//\)$' \| \ X"$as_myself" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_myself" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` srcdir=$ac_confdir if test ! -r "$srcdir/$ac_unique_file"; then srcdir=.. fi else ac_srcdir_defaulted=no fi if test ! -r "$srcdir/$ac_unique_file"; then test "$ac_srcdir_defaulted" = yes && srcdir="$ac_confdir or .." as_fn_error $? "cannot find sources ($ac_unique_file) in $srcdir" fi ac_msg="sources are in $srcdir, but \`cd $srcdir' does not work" ac_abs_confdir=`( cd "$srcdir" && test -r "./$ac_unique_file" || as_fn_error $? "$ac_msg" pwd)` # When building in place, set srcdir=. if test "$ac_abs_confdir" = "$ac_pwd"; then srcdir=. fi # Remove unnecessary trailing slashes from srcdir. # Double slashes in file names in object file debugging info # mess up M-x gdb in Emacs. case $srcdir in */) srcdir=`expr "X$srcdir" : 'X\(.*[^/]\)' \| "X$srcdir" : 'X\(.*\)'`;; esac for ac_var in $ac_precious_vars; do eval ac_env_${ac_var}_set=\${${ac_var}+set} eval ac_env_${ac_var}_value=\$${ac_var} eval ac_cv_env_${ac_var}_set=\${${ac_var}+set} eval ac_cv_env_${ac_var}_value=\$${ac_var} done # # Report the --help message. # if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF \`configure' configures onedrive v2.5.10 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables. Defaults for the options are specified in brackets. Configuration: -h, --help display this help and exit --help=short display options specific to this package --help=recursive display the short help of all the included packages -V, --version display version information and exit -q, --quiet, --silent do not print \`checking ...' messages --cache-file=FILE cache test results in FILE [disabled] -C, --config-cache alias for \`--cache-file=config.cache' -n, --no-create do not create output files --srcdir=DIR find the sources in DIR [configure dir or \`..'] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [$ac_default_prefix] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, \`make install' will install all the files in \`$ac_default_prefix/bin', \`$ac_default_prefix/lib' etc. You can specify an installation prefix other than \`$ac_default_prefix' using \`--prefix', for instance \`--prefix=\$HOME'. For better control, use the options below. Fine tuning of the installation directories: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/onedrive] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] _ACEOF cat <<\_ACEOF _ACEOF fi if test -n "$ac_init_help"; then case $ac_init_help in short | recursive ) echo "Configuration of onedrive v2.5.10:";; esac cat <<\_ACEOF Optional Features: --disable-option-checking ignore unrecognized --enable/--with options --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --disable-version-check Disable checks of compiler version during configure time --enable-notifications Enable desktop notifications via libnotify --enable-completions Install shell completions for bash, zsh, and fish --enable-debug Pass debug option to the compiler Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-systemdsystemunitdir=DIR Directory for systemd system service files --with-systemduserunitdir=DIR Directory for systemd user service files --with-bash-completion-dir=DIR Directory for bash completion files --with-zsh-completion-dir=DIR Directory for zsh completion files --with-fish-completion-dir=DIR Directory for fish completion files Some influential environment variables: DC D compiler executable DCFLAGS flags for D compiler PKG_CONFIG path to pkg-config utility PKG_CONFIG_PATH directories to add to pkg-config's search path PKG_CONFIG_LIBDIR path overriding pkg-config's built-in search path curl_CFLAGS C compiler flags for curl, overriding pkg-config curl_LIBS linker flags for curl, overriding pkg-config sqlite_CFLAGS C compiler flags for sqlite, overriding pkg-config sqlite_LIBS linker flags for sqlite, overriding pkg-config dbus_CFLAGS C compiler flags for dbus, overriding pkg-config dbus_LIBS linker flags for dbus, overriding pkg-config notify_CFLAGS C compiler flags for notify, overriding pkg-config notify_LIBS linker flags for notify, overriding pkg-config bashcompdir value of completionsdir for bash-completion, overriding pkg-config Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. Report bugs to . _ACEOF ac_status=$? fi if test "$ac_init_help" = "recursive"; then # If there are subdirs, report their specific --help. for ac_dir in : $ac_subdirs_all; do test "x$ac_dir" = x: && continue test -d "$ac_dir" || { cd "$srcdir" && ac_pwd=`pwd` && srcdir=. && test -d "$ac_dir"; } || continue ac_builddir=. case "$ac_dir" in .) ac_dir_suffix= ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_dir_suffix=/`$as_echo "$ac_dir" | sed 's|^\.[\\/]||'` # A ".." for each directory in $ac_dir_suffix. ac_top_builddir_sub=`$as_echo "$ac_dir_suffix" | sed 's|/[^\\/]*|/..|g;s|/||'` case $ac_top_builddir_sub in "") ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_top_build_prefix=$ac_top_builddir_sub/ ;; esac ;; esac ac_abs_top_builddir=$ac_pwd ac_abs_builddir=$ac_pwd$ac_dir_suffix # for backward compatibility: ac_top_builddir=$ac_top_build_prefix case $srcdir in .) # We are building in place. ac_srcdir=. ac_top_srcdir=$ac_top_builddir_sub ac_abs_top_srcdir=$ac_pwd ;; [\\/]* | ?:[\\/]* ) # Absolute name. ac_srcdir=$srcdir$ac_dir_suffix; ac_top_srcdir=$srcdir ac_abs_top_srcdir=$srcdir ;; *) # Relative name. ac_srcdir=$ac_top_build_prefix$srcdir$ac_dir_suffix ac_top_srcdir=$ac_top_build_prefix$srcdir ac_abs_top_srcdir=$ac_pwd/$srcdir ;; esac ac_abs_srcdir=$ac_abs_top_srcdir$ac_dir_suffix cd "$ac_dir" || { ac_status=$?; continue; } # Check for guested configure. if test -f "$ac_srcdir/configure.gnu"; then echo && $SHELL "$ac_srcdir/configure.gnu" --help=recursive elif test -f "$ac_srcdir/configure"; then echo && $SHELL "$ac_srcdir/configure" --help=recursive else $as_echo "$as_me: WARNING: no configuration information is in $ac_dir" >&2 fi || ac_status=$? cd "$ac_pwd" || { ac_status=$?; break; } done fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF onedrive configure v2.5.10 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. This configure script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it. _ACEOF exit fi ## ------------------------ ## ## Autoconf initialization. ## ## ------------------------ ## cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. It was created by onedrive $as_me v2.5.10, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ _ACEOF exec 5>>config.log { cat <<_ASUNAME ## --------- ## ## Platform. ## ## --------- ## hostname = `(hostname || uname -n) 2>/dev/null | sed 1q` uname -m = `(uname -m) 2>/dev/null || echo unknown` uname -r = `(uname -r) 2>/dev/null || echo unknown` uname -s = `(uname -s) 2>/dev/null || echo unknown` uname -v = `(uname -v) 2>/dev/null || echo unknown` /usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null || echo unknown` /bin/uname -X = `(/bin/uname -X) 2>/dev/null || echo unknown` /bin/arch = `(/bin/arch) 2>/dev/null || echo unknown` /usr/bin/arch -k = `(/usr/bin/arch -k) 2>/dev/null || echo unknown` /usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null || echo unknown` /usr/bin/hostinfo = `(/usr/bin/hostinfo) 2>/dev/null || echo unknown` /bin/machine = `(/bin/machine) 2>/dev/null || echo unknown` /usr/bin/oslevel = `(/usr/bin/oslevel) 2>/dev/null || echo unknown` /bin/universe = `(/bin/universe) 2>/dev/null || echo unknown` _ASUNAME as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. $as_echo "PATH: $as_dir" done IFS=$as_save_IFS } >&5 cat >&5 <<_ACEOF ## ----------- ## ## Core tests. ## ## ----------- ## _ACEOF # Keep a trace of the command line. # Strip out --no-create and --no-recursion so they do not pile up. # Strip out --silent because we don't want to record it for future runs. # Also quote any args containing shell meta-characters. # Make two passes to allow for proper duplicate-argument suppression. ac_configure_args= ac_configure_args0= ac_configure_args1= ac_must_keep_next=false for ac_pass in 1 2 do for ac_arg do case $ac_arg in -no-create | --no-c* | -n | -no-recursion | --no-r*) continue ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil) continue ;; *\'*) ac_arg=`$as_echo "$ac_arg" | sed "s/'/'\\\\\\\\''/g"` ;; esac case $ac_pass in 1) as_fn_append ac_configure_args0 " '$ac_arg'" ;; 2) as_fn_append ac_configure_args1 " '$ac_arg'" if test $ac_must_keep_next = true; then ac_must_keep_next=false # Got value, back to normal. else case $ac_arg in *=* | --config-cache | -C | -disable-* | --disable-* \ | -enable-* | --enable-* | -gas | --g* | -nfp | --nf* \ | -q | -quiet | --q* | -silent | --sil* | -v | -verb* \ | -with-* | --with-* | -without-* | --without-* | --x) case "$ac_configure_args0 " in "$ac_configure_args1"*" '$ac_arg' "* ) continue ;; esac ;; -* ) ac_must_keep_next=true ;; esac fi as_fn_append ac_configure_args " '$ac_arg'" ;; esac done done { ac_configure_args0=; unset ac_configure_args0;} { ac_configure_args1=; unset ac_configure_args1;} # When interrupted or exit'd, cleanup temporary files, and complete # config.log. We remove comments because anyway the quotes in there # would cause problems or look ugly. # WARNING: Use '\'' to represent an apostrophe within the trap. # WARNING: Do not start the trap code with a newline, due to a FreeBSD 4.0 bug. trap 'exit_status=$? # Save into config.log some information that might help in debugging. { echo $as_echo "## ---------------- ## ## Cache variables. ## ## ---------------- ##" echo # The following way of writing the cache mishandles newlines in values, ( for ac_var in `(set) 2>&1 | sed -n '\''s/^\([a-zA-Z_][a-zA-Z0-9_]*\)=.*/\1/p'\''`; do eval ac_val=\$$ac_var case $ac_val in #( *${as_nl}*) case $ac_var in #( *_cv_*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: cache variable $ac_var contains a newline" >&5 $as_echo "$as_me: WARNING: cache variable $ac_var contains a newline" >&2;} ;; esac case $ac_var in #( _ | IFS | as_nl) ;; #( BASH_ARGV | BASH_SOURCE) eval $ac_var= ;; #( *) { eval $ac_var=; unset $ac_var;} ;; esac ;; esac done (set) 2>&1 | case $as_nl`(ac_space='\'' '\''; set) 2>&1` in #( *${as_nl}ac_space=\ *) sed -n \ "s/'\''/'\''\\\\'\'''\''/g; s/^\\([_$as_cr_alnum]*_cv_[_$as_cr_alnum]*\\)=\\(.*\\)/\\1='\''\\2'\''/p" ;; #( *) sed -n "/^[_$as_cr_alnum]*_cv_[_$as_cr_alnum]*=/p" ;; esac | sort ) echo $as_echo "## ----------------- ## ## Output variables. ## ## ----------------- ##" echo for ac_var in $ac_subst_vars do eval ac_val=\$$ac_var case $ac_val in *\'\''*) ac_val=`$as_echo "$ac_val" | sed "s/'\''/'\''\\\\\\\\'\'''\''/g"`;; esac $as_echo "$ac_var='\''$ac_val'\''" done | sort echo if test -n "$ac_subst_files"; then $as_echo "## ------------------- ## ## File substitutions. ## ## ------------------- ##" echo for ac_var in $ac_subst_files do eval ac_val=\$$ac_var case $ac_val in *\'\''*) ac_val=`$as_echo "$ac_val" | sed "s/'\''/'\''\\\\\\\\'\'''\''/g"`;; esac $as_echo "$ac_var='\''$ac_val'\''" done | sort echo fi if test -s confdefs.h; then $as_echo "## ----------- ## ## confdefs.h. ## ## ----------- ##" echo cat confdefs.h echo fi test "$ac_signal" != 0 && $as_echo "$as_me: caught signal $ac_signal" $as_echo "$as_me: exit $exit_status" } >&5 rm -f core *.core core.conftest.* && rm -f -r conftest* confdefs* conf$$* $ac_clean_files && exit $exit_status ' 0 for ac_signal in 1 2 13 15; do trap 'ac_signal='$ac_signal'; as_fn_exit 1' $ac_signal done ac_signal=0 # confdefs.h avoids OS command line length limits that DEFS can exceed. rm -f -r conftest* confdefs.h $as_echo "/* confdefs.h */" > confdefs.h # Predefined preprocessor variables. cat >>confdefs.h <<_ACEOF #define PACKAGE_NAME "$PACKAGE_NAME" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_TARNAME "$PACKAGE_TARNAME" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_VERSION "$PACKAGE_VERSION" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_STRING "$PACKAGE_STRING" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_BUGREPORT "$PACKAGE_BUGREPORT" _ACEOF cat >>confdefs.h <<_ACEOF #define PACKAGE_URL "$PACKAGE_URL" _ACEOF # Let the site file select an alternate cache file if it wants to. # Prefer an explicitly selected file to automatically selected ones. ac_site_file1=NONE ac_site_file2=NONE if test -n "$CONFIG_SITE"; then # We do not want a PATH search for config.site. case $CONFIG_SITE in #(( -*) ac_site_file1=./$CONFIG_SITE;; */*) ac_site_file1=$CONFIG_SITE;; *) ac_site_file1=./$CONFIG_SITE;; esac elif test "x$prefix" != xNONE; then ac_site_file1=$prefix/share/config.site ac_site_file2=$prefix/etc/config.site else ac_site_file1=$ac_default_prefix/share/config.site ac_site_file2=$ac_default_prefix/etc/config.site fi for ac_site_file in "$ac_site_file1" "$ac_site_file2" do test "x$ac_site_file" = xNONE && continue if test /dev/null != "$ac_site_file" && test -r "$ac_site_file"; then { $as_echo "$as_me:${as_lineno-$LINENO}: loading site script $ac_site_file" >&5 $as_echo "$as_me: loading site script $ac_site_file" >&6;} sed 's/^/| /' "$ac_site_file" >&5 . "$ac_site_file" \ || { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "failed to load site script $ac_site_file See \`config.log' for more details" "$LINENO" 5; } fi done if test -r "$cache_file"; then # Some versions of bash will fail to source /dev/null (special files # actually), so we avoid doing that. DJGPP emulates it as a regular file. if test /dev/null != "$cache_file" && test -f "$cache_file"; then { $as_echo "$as_me:${as_lineno-$LINENO}: loading cache $cache_file" >&5 $as_echo "$as_me: loading cache $cache_file" >&6;} case $cache_file in [\\/]* | ?:[\\/]* ) . "$cache_file";; *) . "./$cache_file";; esac fi else { $as_echo "$as_me:${as_lineno-$LINENO}: creating cache $cache_file" >&5 $as_echo "$as_me: creating cache $cache_file" >&6;} >$cache_file fi # Check that the precious variables saved in the cache have kept the same # value. ac_cache_corrupted=false for ac_var in $ac_precious_vars; do eval ac_old_set=\$ac_cv_env_${ac_var}_set eval ac_new_set=\$ac_env_${ac_var}_set eval ac_old_val=\$ac_cv_env_${ac_var}_value eval ac_new_val=\$ac_env_${ac_var}_value case $ac_old_set,$ac_new_set in set,) { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&5 $as_echo "$as_me: error: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&2;} ac_cache_corrupted=: ;; ,set) { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' was not set in the previous run" >&5 $as_echo "$as_me: error: \`$ac_var' was not set in the previous run" >&2;} ac_cache_corrupted=: ;; ,);; *) if test "x$ac_old_val" != "x$ac_new_val"; then # differences in whitespace do not lead to failure. ac_old_val_w=`echo x $ac_old_val` ac_new_val_w=`echo x $ac_new_val` if test "$ac_old_val_w" != "$ac_new_val_w"; then { $as_echo "$as_me:${as_lineno-$LINENO}: error: \`$ac_var' has changed since the previous run:" >&5 $as_echo "$as_me: error: \`$ac_var' has changed since the previous run:" >&2;} ac_cache_corrupted=: else { $as_echo "$as_me:${as_lineno-$LINENO}: warning: ignoring whitespace changes in \`$ac_var' since the previous run:" >&5 $as_echo "$as_me: warning: ignoring whitespace changes in \`$ac_var' since the previous run:" >&2;} eval $ac_var=\$ac_old_val fi { $as_echo "$as_me:${as_lineno-$LINENO}: former value: \`$ac_old_val'" >&5 $as_echo "$as_me: former value: \`$ac_old_val'" >&2;} { $as_echo "$as_me:${as_lineno-$LINENO}: current value: \`$ac_new_val'" >&5 $as_echo "$as_me: current value: \`$ac_new_val'" >&2;} fi;; esac # Pass precious variables to config.status. if test "$ac_new_set" = set; then case $ac_new_val in *\'*) ac_arg=$ac_var=`$as_echo "$ac_new_val" | sed "s/'/'\\\\\\\\''/g"` ;; *) ac_arg=$ac_var=$ac_new_val ;; esac case " $ac_configure_args " in *" '$ac_arg' "*) ;; # Avoid dups. Use of quotes ensures accuracy. *) as_fn_append ac_configure_args " '$ac_arg'" ;; esac fi done if $ac_cache_corrupted; then { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} { $as_echo "$as_me:${as_lineno-$LINENO}: error: changes in the environment can compromise the build" >&5 $as_echo "$as_me: error: changes in the environment can compromise the build" >&2;} as_fn_error $? "run \`make distclean' and/or \`rm $cache_file' and start over" "$LINENO" 5 fi ## -------------------- ## ## Main body of script. ## ## -------------------- ## ac_ext=c ac_cpp='$CPP $CPPFLAGS' ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5' ac_compiler_gnu=$ac_cv_c_compiler_gnu ac_aux_dir= for ac_dir in "$srcdir" "$srcdir/.." "$srcdir/../.."; do if test -f "$ac_dir/install-sh"; then ac_aux_dir=$ac_dir ac_install_sh="$ac_aux_dir/install-sh -c" break elif test -f "$ac_dir/install.sh"; then ac_aux_dir=$ac_dir ac_install_sh="$ac_aux_dir/install.sh -c" break elif test -f "$ac_dir/shtool"; then ac_aux_dir=$ac_dir ac_install_sh="$ac_aux_dir/shtool install -c" break fi done if test -z "$ac_aux_dir"; then as_fn_error $? "cannot find install-sh, install.sh, or shtool in \"$srcdir\" \"$srcdir/..\" \"$srcdir/../..\"" "$LINENO" 5 fi # These three variables are undocumented and unsupported, # and are intended to be withdrawn in a future Autoconf release. # They can cause serious problems if a builder's source tree is in a directory # whose full name contains unusual characters. ac_config_guess="$SHELL $ac_aux_dir/config.guess" # Please don't use this var. ac_config_sub="$SHELL $ac_aux_dir/config.sub" # Please don't use this var. ac_configure="$SHELL $ac_aux_dir/configure" # Please don't use this var. # Find a good install program. We prefer a C program (faster), # so one script is as good as another. But avoid the broken or # incompatible versions: # SysV /etc/install, /usr/sbin/install # SunOS /usr/etc/install # IRIX /sbin/install # AIX /bin/install # AmigaOS /C/install, which installs bootblocks on floppy discs # AIX 4 /usr/bin/installbsd, which doesn't work without a -g flag # AFS /usr/afsws/bin/install, which mishandles nonexistent args # SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff" # OS/2's system install, which has a completely different semantic # ./install, which can be erroneously created by make from ./install.sh. # Reject install programs that cannot install multiple files. { $as_echo "$as_me:${as_lineno-$LINENO}: checking for a BSD-compatible install" >&5 $as_echo_n "checking for a BSD-compatible install... " >&6; } if test -z "$INSTALL"; then if ${ac_cv_path_install+:} false; then : $as_echo_n "(cached) " >&6 else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. # Account for people who put trailing slashes in PATH elements. case $as_dir/ in #(( ./ | .// | /[cC]/* | \ /etc/* | /usr/sbin/* | /usr/etc/* | /sbin/* | /usr/afsws/bin/* | \ ?:[\\/]os2[\\/]install[\\/]* | ?:[\\/]OS2[\\/]INSTALL[\\/]* | \ /usr/ucb/* ) ;; *) # OSF1 and SCO ODT 3.0 have their own names for install. # Don't use installbsd from OSF since it installs stuff as root # by default. for ac_prog in ginstall scoinst install; do for ac_exec_ext in '' $ac_executable_extensions; do if as_fn_executable_p "$as_dir/$ac_prog$ac_exec_ext"; then if test $ac_prog = install && grep dspmsg "$as_dir/$ac_prog$ac_exec_ext" >/dev/null 2>&1; then # AIX install. It has an incompatible calling convention. : elif test $ac_prog = install && grep pwplus "$as_dir/$ac_prog$ac_exec_ext" >/dev/null 2>&1; then # program-specific install script used by HP pwplus--don't use. : else rm -rf conftest.one conftest.two conftest.dir echo one > conftest.one echo two > conftest.two mkdir conftest.dir if "$as_dir/$ac_prog$ac_exec_ext" -c conftest.one conftest.two "`pwd`/conftest.dir" && test -s conftest.one && test -s conftest.two && test -s conftest.dir/conftest.one && test -s conftest.dir/conftest.two then ac_cv_path_install="$as_dir/$ac_prog$ac_exec_ext -c" break 3 fi fi fi done done ;; esac done IFS=$as_save_IFS rm -rf conftest.one conftest.two conftest.dir fi if test "${ac_cv_path_install+set}" = set; then INSTALL=$ac_cv_path_install else # As a last resort, use the slow shell script. Don't cache a # value for INSTALL within a source directory, because that will # break other packages using the cache if that directory is # removed, or if the value is a relative name. INSTALL=$ac_install_sh fi fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $INSTALL" >&5 $as_echo "$INSTALL" >&6; } # Use test -z because SunOS4 sh mishandles braces in ${var-val}. # It thinks the first close brace ends the variable substitution. test -z "$INSTALL_PROGRAM" && INSTALL_PROGRAM='${INSTALL}' test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL}' test -z "$INSTALL_DATA" && INSTALL_DATA='${INSTALL} -m 644' if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then if test -n "$ac_tool_prefix"; then # Extract the first word of "${ac_tool_prefix}pkg-config", so it can be a program name with args. set dummy ${ac_tool_prefix}pkg-config; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if ${ac_cv_path_PKG_CONFIG+:} false; then : $as_echo_n "(cached) " >&6 else case $PKG_CONFIG in [\\/]* | ?:[\\/]*) ac_cv_path_PKG_CONFIG="$PKG_CONFIG" # Let the user override the test with a path. ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then ac_cv_path_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS ;; esac fi PKG_CONFIG=$ac_cv_path_PKG_CONFIG if test -n "$PKG_CONFIG"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PKG_CONFIG" >&5 $as_echo "$PKG_CONFIG" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi fi if test -z "$ac_cv_path_PKG_CONFIG"; then ac_pt_PKG_CONFIG=$PKG_CONFIG # Extract the first word of "pkg-config", so it can be a program name with args. set dummy pkg-config; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then : $as_echo_n "(cached) " >&6 else case $ac_pt_PKG_CONFIG in [\\/]* | ?:[\\/]*) ac_cv_path_ac_pt_PKG_CONFIG="$ac_pt_PKG_CONFIG" # Let the user override the test with a path. ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then ac_cv_path_ac_pt_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS ;; esac fi ac_pt_PKG_CONFIG=$ac_cv_path_ac_pt_PKG_CONFIG if test -n "$ac_pt_PKG_CONFIG"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_pt_PKG_CONFIG" >&5 $as_echo "$ac_pt_PKG_CONFIG" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi if test "x$ac_pt_PKG_CONFIG" = x; then PKG_CONFIG="" else case $cross_compiling:$ac_tool_warned in yes:) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 $as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} ac_tool_warned=yes ;; esac PKG_CONFIG=$ac_pt_PKG_CONFIG fi else PKG_CONFIG="$ac_cv_path_PKG_CONFIG" fi fi if test -n "$PKG_CONFIG"; then _pkg_min_version=0.9.0 { $as_echo "$as_me:${as_lineno-$LINENO}: checking pkg-config is at least version $_pkg_min_version" >&5 $as_echo_n "checking pkg-config is at least version $_pkg_min_version... " >&6; } if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } PKG_CONFIG="" fi fi PACKAGE_DATE="January 2026" for ac_prog in dmd ldmd2 ldc2 gdmd gdc do # Extract the first word of "$ac_prog", so it can be a program name with args. set dummy $ac_prog; ac_word=$2 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 $as_echo_n "checking for $ac_word... " >&6; } if ${ac_cv_prog_DC+:} false; then : $as_echo_n "(cached) " >&6 else if test -n "$DC"; then ac_cv_prog_DC="$DC" # Let the user override the test. else as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. for ac_exec_ext in '' $ac_executable_extensions; do if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then ac_cv_prog_DC="$ac_prog" $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 break 2 fi done done IFS=$as_save_IFS fi fi DC=$ac_cv_prog_DC if test -n "$DC"; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: $DC" >&5 $as_echo "$DC" >&6; } else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } fi test -n "$DC" && break done test -n "$DC" || DC="NOT_FOUND" DC_TYPE= case $(basename $DC) in *ldc2*) DC_TYPE=ldc ;; *gdc*) DC_TYPE=gdc ;; *dmd*) DC_TYPE=dmd ;; NOT_FOUND) as_fn_error 1 "Could not find any compatible D compiler" "$LINENO" 5 esac vercomp () { IFS=. read -r a0 a1 a2 aa <' $bb then return 1 else return 0 fi fi fi fi } DO_VERSION_CHECK=1 # Check whether --enable-version-check was given. if test "${enable_version_check+set}" = set; then : enableval=$enable_version_check; fi if test "x$enable_version_check" = "xno"; then : DO_VERSION_CHECK=0 fi if test "$DO_VERSION_CHECK" = "1"; then : { $as_echo "$as_me:${as_lineno-$LINENO}: checking version of D compiler" >&5 $as_echo_n "checking version of D compiler... " >&6; } # check for valid versions case $(basename $DC) in *ldmd2*|*ldc2*) # LDC - the LLVM D compiler (1.12.0): ... VERSION=`$DC --version` # remove everything up to first ( VERSION=${VERSION#* (} # remove everything after ): VERSION=${VERSION%%):*} # now version should be something like L.M.N MINVERSION=1.20.1 ;; *gdmd*|*gdc*) # Both gdmd and gdc print the same version information VERSION=`${DC} --version | head -n1` # Some examples of output: # gdc (Gentoo 14.2.1_p20250301 p8) 14.2.1 20250301 # gcc (GCC) 14.2.1 20250207 # Arch # gdc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7) # gdc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 VERSION=${VERSION#gdc } # VERSION=(...) VER DATE ... VERSION=${VERSION#*) } # VERSION=VER DATE ... VERSION=${VERSION%% *} MINVERSION=15 ;; *dmd*) # DMD64 D Compiler v2.085.1\n... VERSION=`$DC --version | tr '\n' ' '` VERSION=${VERSION#*Compiler v} VERSION=${VERSION%% *} # now version should be something like L.M.N MINVERSION=2.091.1 ;; esac { $as_echo "$as_me:${as_lineno-$LINENO}: result: $VERSION" >&5 $as_echo "$VERSION" >&6; } vercomp $MINVERSION $VERSION if test $? = 1 then as_fn_error 1 "Compiler version insufficient, current compiler version $VERSION, minimum version $MINVERSION" "$LINENO" 5 fi #echo "MINVERSION=$MINVERSION VERSION=$VERSION" fi case "$DC_TYPE" in dmd) DEBUG_DCFLAGS="-g -debug -gs" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-version LINKER_DCFLAG=-L OUTPUT_DCFLAG=-of WERROR_DCFLAG=-w ;; ldc) DEBUG_DCFLAGS="-g -d-debug -gc" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-d-version LINKER_DCFLAG=-L OUTPUT_DCFLAG=-of WERROR_DCFLAG=-w ;; gdc) DEBUG_DCFLAGS="-g -fdebug" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-fversion LINKER_DCFLAG=-Wl, OUTPUT_DCFLAG=-o WERROR_DCFLAG=-Werror ;; esac pkg_failed=no { $as_echo "$as_me:${as_lineno-$LINENO}: checking for curl" >&5 $as_echo_n "checking for curl... " >&6; } if test -n "$curl_CFLAGS"; then pkg_cv_curl_CFLAGS="$curl_CFLAGS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libcurl\""; } >&5 ($PKG_CONFIG --exists --print-errors "libcurl") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_curl_CFLAGS=`$PKG_CONFIG --cflags "libcurl" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test -n "$curl_LIBS"; then pkg_cv_curl_LIBS="$curl_LIBS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libcurl\""; } >&5 ($PKG_CONFIG --exists --print-errors "libcurl") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_curl_LIBS=`$PKG_CONFIG --libs "libcurl" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test $pkg_failed = yes; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then _pkg_short_errors_supported=yes else _pkg_short_errors_supported=no fi if test $_pkg_short_errors_supported = yes; then curl_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libcurl" 2>&1` else curl_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libcurl" 2>&1` fi # Put the nasty error message in config.log where it belongs echo "$curl_PKG_ERRORS" >&5 as_fn_error $? "Package requirements (libcurl) were not met: $curl_PKG_ERRORS Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables curl_CFLAGS and curl_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details." "$LINENO" 5 elif test $pkg_failed = untried; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config. Alternatively, you may set the environment variables curl_CFLAGS and curl_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. To get pkg-config, see . See \`config.log' for more details" "$LINENO" 5; } else curl_CFLAGS=$pkg_cv_curl_CFLAGS curl_LIBS=$pkg_cv_curl_LIBS { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } fi pkg_failed=no { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite" >&5 $as_echo_n "checking for sqlite... " >&6; } if test -n "$sqlite_CFLAGS"; then pkg_cv_sqlite_CFLAGS="$sqlite_CFLAGS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"sqlite3\""; } >&5 ($PKG_CONFIG --exists --print-errors "sqlite3") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_sqlite_CFLAGS=`$PKG_CONFIG --cflags "sqlite3" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test -n "$sqlite_LIBS"; then pkg_cv_sqlite_LIBS="$sqlite_LIBS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"sqlite3\""; } >&5 ($PKG_CONFIG --exists --print-errors "sqlite3") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_sqlite_LIBS=`$PKG_CONFIG --libs "sqlite3" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test $pkg_failed = yes; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then _pkg_short_errors_supported=yes else _pkg_short_errors_supported=no fi if test $_pkg_short_errors_supported = yes; then sqlite_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "sqlite3" 2>&1` else sqlite_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "sqlite3" 2>&1` fi # Put the nasty error message in config.log where it belongs echo "$sqlite_PKG_ERRORS" >&5 as_fn_error $? "Package requirements (sqlite3) were not met: $sqlite_PKG_ERRORS Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables sqlite_CFLAGS and sqlite_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details." "$LINENO" 5 elif test $pkg_failed = untried; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;} as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config. Alternatively, you may set the environment variables sqlite_CFLAGS and sqlite_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. To get pkg-config, see . See \`config.log' for more details" "$LINENO" 5; } else sqlite_CFLAGS=$pkg_cv_sqlite_CFLAGS sqlite_LIBS=$pkg_cv_sqlite_LIBS { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } fi { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to enable dbus support" >&5 $as_echo_n "checking whether to enable dbus support... " >&6; } case "$(uname -s)" in Linux) enable_dbus=yes { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes (on Linux)" >&5 $as_echo "yes (on Linux)" >&6; } pkg_failed=no { $as_echo "$as_me:${as_lineno-$LINENO}: checking for dbus" >&5 $as_echo_n "checking for dbus... " >&6; } if test -n "$dbus_CFLAGS"; then pkg_cv_dbus_CFLAGS="$dbus_CFLAGS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"dbus-1 >= 1.0\""; } >&5 ($PKG_CONFIG --exists --print-errors "dbus-1 >= 1.0") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_dbus_CFLAGS=`$PKG_CONFIG --cflags "dbus-1 >= 1.0" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test -n "$dbus_LIBS"; then pkg_cv_dbus_LIBS="$dbus_LIBS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"dbus-1 >= 1.0\""; } >&5 ($PKG_CONFIG --exists --print-errors "dbus-1 >= 1.0") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_dbus_LIBS=`$PKG_CONFIG --libs "dbus-1 >= 1.0" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test $pkg_failed = yes; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then _pkg_short_errors_supported=yes else _pkg_short_errors_supported=no fi if test $_pkg_short_errors_supported = yes; then dbus_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "dbus-1 >= 1.0" 2>&1` else dbus_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "dbus-1 >= 1.0" 2>&1` fi # Put the nasty error message in config.log where it belongs echo "$dbus_PKG_ERRORS" >&5 as_fn_error $? "dbus-1 development files not found. Please install dbus-devel (Red Hat), libdbus-1-dev (Debian) or dbus (Arch | Manjaro)" "$LINENO" 5 elif test $pkg_failed = untried; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } as_fn_error $? "dbus-1 development files not found. Please install dbus-devel (Red Hat), libdbus-1-dev (Debian) or dbus (Arch | Manjaro)" "$LINENO" 5 else dbus_CFLAGS=$pkg_cv_dbus_CFLAGS dbus_LIBS=$pkg_cv_dbus_LIBS { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } $as_echo "#define HAVE_DBUS 1" >>confdefs.h fi ;; *) enable_dbus=no { $as_echo "$as_me:${as_lineno-$LINENO}: result: no (not on Linux)" >&5 $as_echo "no (not on Linux)" >&6; } ;; esac # Check whether --with-systemdsystemunitdir was given. if test "${with_systemdsystemunitdir+set}" = set; then : withval=$with_systemdsystemunitdir; else with_systemdsystemunitdir=auto fi if test "x$with_systemdsystemunitdir" = "xyes" -o "x$with_systemdsystemunitdir" = "xauto"; then : def_systemdsystemunitdir=$($PKG_CONFIG --variable=systemdsystemunitdir systemd) if test "x$def_systemdsystemunitdir" = "x"; then : if test "x$with_systemdsystemunitdir" = "xyes"; then : as_fn_error $? "systemd support requested but pkg-config unable to query systemd package" "$LINENO" 5 fi with_systemdsystemunitdir=no else with_systemdsystemunitdir="$def_systemdsystemunitdir" fi fi if test "x$with_systemdsystemunitdir" != "xno"; then : systemdsystemunitdir=$with_systemdsystemunitdir fi # Check whether --with-systemduserunitdir was given. if test "${with_systemduserunitdir+set}" = set; then : withval=$with_systemduserunitdir; else with_systemduserunitdir=auto fi if test "x$with_systemduserunitdir" = "xyes" -o "x$with_systemduserunitdir" = "xauto"; then : def_systemduserunitdir=$($PKG_CONFIG --variable=systemduserunitdir systemd) if test "x$def_systemduserunitdir" = "x"; then : if test "x$with_systemduserunitdir" = "xyes"; then : as_fn_error $? "systemd support requested but pkg-config unable to query systemd package" "$LINENO" 5 fi with_systemduserunitdir=no else with_systemduserunitdir="$def_systemduserunitdir" fi fi if test "x$with_systemduserunitdir" != "xno"; then : systemduserunitdir=$with_systemduserunitdir fi if test "x$with_systemduserunitdir" != "xno" -a "x$with_systemdsystemunitdir" != "xno"; then : havesystemd=yes else havesystemd=no fi HAVE_SYSTEMD=$havesystemd # Check whether --enable-notifications was given. if test "${enable_notifications+set}" = set; then : enableval=$enable_notifications; fi if test "x$enable_notifications" = "xyes"; then : enable_notifications=yes else enable_notifications=no fi if test "x$enable_notifications" = "xyes"; then : pkg_failed=no { $as_echo "$as_me:${as_lineno-$LINENO}: checking for notify" >&5 $as_echo_n "checking for notify... " >&6; } if test -n "$notify_CFLAGS"; then pkg_cv_notify_CFLAGS="$notify_CFLAGS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libnotify\""; } >&5 ($PKG_CONFIG --exists --print-errors "libnotify") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_notify_CFLAGS=`$PKG_CONFIG --cflags "libnotify" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test -n "$notify_LIBS"; then pkg_cv_notify_LIBS="$notify_LIBS" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libnotify\""; } >&5 ($PKG_CONFIG --exists --print-errors "libnotify") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_notify_LIBS=`$PKG_CONFIG --libs "libnotify" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi if test $pkg_failed = yes; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then _pkg_short_errors_supported=yes else _pkg_short_errors_supported=no fi if test $_pkg_short_errors_supported = yes; then notify_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libnotify" 2>&1` else notify_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libnotify" 2>&1` fi # Put the nasty error message in config.log where it belongs echo "$notify_PKG_ERRORS" >&5 enable_notifications=no elif test $pkg_failed = untried; then { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; } enable_notifications=no else notify_CFLAGS=$pkg_cv_notify_CFLAGS notify_LIBS=$pkg_cv_notify_LIBS { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; } fi else notify_LIBS="" fi NOTIFICATIONS=$enable_notifications # Conditionally set bsd_inotify_LIBS based on the platform case "$(uname -s)" in Linux) bsd_inotify_LIBS="" ;; FreeBSD) if test "$(uname -U)" -gt 1500060; then : bsd_inotify_LIBS="" else bsd_inotify_LIBS="-L/usr/local/lib -linotify" fi ;; OpenBSD) bsd_inotify_LIBS="-L/usr/local/lib/inotify -linotify" ;; *) bsd_inotify_LIBS="" ;; esac # Conditionally set dynamic_linker_LIBS based on the platform case "$(uname -s)" in Linux) dynamic_linker_LIBS="-ldl" ;; *) dynamic_linker_LIBS="" ;; esac # Check whether --enable-completions was given. if test "${enable_completions+set}" = set; then : enableval=$enable_completions; fi if test "x$enable_completions" = "xyes"; then : enable_completions=yes else enable_completions=no fi COMPLETIONS=$enable_completions if test "x$enable_completions" = "xyes"; then : # Check whether --with-bash-completion-dir was given. if test "${with_bash_completion_dir+set}" = set; then : withval=$with_bash_completion_dir; else with_bash_completion_dir=auto fi if test "x$with_bash_completion_dir" = "xyes" -o "x$with_bash_completion_dir" = "xauto"; then : if test -n "$bashcompdir"; then pkg_cv_bashcompdir="$bashcompdir" elif test -n "$PKG_CONFIG"; then if test -n "$PKG_CONFIG" && \ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"bash-completion\""; } >&5 ($PKG_CONFIG --exists --print-errors "bash-completion") 2>&5 ac_status=$? $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 test $ac_status = 0; }; then pkg_cv_bashcompdir=`$PKG_CONFIG --variable="completionsdir" "bash-completion" 2>/dev/null` test "x$?" != "x0" && pkg_failed=yes else pkg_failed=yes fi else pkg_failed=untried fi bashcompdir=$pkg_cv_bashcompdir if test "x$bashcompdir" = x""; then : bashcompdir="${sysconfdir}/bash_completion.d" fi with_bash_completion_dir=$bashcompdir fi BASH_COMPLETION_DIR=$with_bash_completion_dir # Check whether --with-zsh-completion-dir was given. if test "${with_zsh_completion_dir+set}" = set; then : withval=$with_zsh_completion_dir; else with_zsh_completion_dir=auto fi if test "x$with_zsh_completion_dir" = "xyes" -o "x$with_zsh_completion_dir" = "xauto"; then : with_zsh_completion_dir="/usr/local/share/zsh/site-functions" fi ZSH_COMPLETION_DIR=$with_zsh_completion_dir # Check whether --with-fish-completion-dir was given. if test "${with_fish_completion_dir+set}" = set; then : withval=$with_fish_completion_dir; else with_fish_completion_dir=auto fi if test "x$with_fish_completion_dir" = "xyes" -o "x$with_fish_completion_dir" = "xauto"; then : with_fish_completion_dir="/usr/local/share/fish/completions" fi FISH_COMPLETION_DIR=$with_fish_completion_dir fi # Check whether --enable-debug was given. if test "${enable_debug+set}" = set; then : enableval=$enable_debug; fi if test "x$enable_debug" = "xyes"; then : DEBUG=yes else DEBUG=no fi ac_config_files="$ac_config_files Makefile contrib/pacman/PKGBUILD contrib/spec/onedrive.spec onedrive.1 contrib/systemd/onedrive.service contrib/systemd/onedrive@.service" cat >confcache <<\_ACEOF # This file is a shell script that caches the results of configure # tests run on this system so they can be shared between configure # scripts and configure runs, see configure's option --config-cache. # It is not useful on other systems. If it contains results you don't # want to keep, you may remove or edit it. # # config.status only pays attention to the cache file if you give it # the --recheck option to rerun configure. # # `ac_cv_env_foo' variables (set or unset) will be overridden when # loading this file, other *unset* `ac_cv_foo' will be assigned the # following values. _ACEOF # The following way of writing the cache mishandles newlines in values, # but we know of no workaround that is simple, portable, and efficient. # So, we kill variables containing newlines. # Ultrix sh set writes to stderr and can't be redirected directly, # and sets the high bit in the cache file unless we assign to the vars. ( for ac_var in `(set) 2>&1 | sed -n 's/^\([a-zA-Z_][a-zA-Z0-9_]*\)=.*/\1/p'`; do eval ac_val=\$$ac_var case $ac_val in #( *${as_nl}*) case $ac_var in #( *_cv_*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: cache variable $ac_var contains a newline" >&5 $as_echo "$as_me: WARNING: cache variable $ac_var contains a newline" >&2;} ;; esac case $ac_var in #( _ | IFS | as_nl) ;; #( BASH_ARGV | BASH_SOURCE) eval $ac_var= ;; #( *) { eval $ac_var=; unset $ac_var;} ;; esac ;; esac done (set) 2>&1 | case $as_nl`(ac_space=' '; set) 2>&1` in #( *${as_nl}ac_space=\ *) # `set' does not quote correctly, so add quotes: double-quote # substitution turns \\\\ into \\, and sed turns \\ into \. sed -n \ "s/'/'\\\\''/g; s/^\\([_$as_cr_alnum]*_cv_[_$as_cr_alnum]*\\)=\\(.*\\)/\\1='\\2'/p" ;; #( *) # `set' quotes correctly as required by POSIX, so do not add quotes. sed -n "/^[_$as_cr_alnum]*_cv_[_$as_cr_alnum]*=/p" ;; esac | sort ) | sed ' /^ac_cv_env_/b end t clear :clear s/^\([^=]*\)=\(.*[{}].*\)$/test "${\1+set}" = set || &/ t end s/^\([^=]*\)=\(.*\)$/\1=${\1=\2}/ :end' >>confcache if diff "$cache_file" confcache >/dev/null 2>&1; then :; else if test -w "$cache_file"; then if test "x$cache_file" != "x/dev/null"; then { $as_echo "$as_me:${as_lineno-$LINENO}: updating cache $cache_file" >&5 $as_echo "$as_me: updating cache $cache_file" >&6;} if test ! -f "$cache_file" || test -h "$cache_file"; then cat confcache >"$cache_file" else case $cache_file in #( */* | ?:*) mv -f confcache "$cache_file"$$ && mv -f "$cache_file"$$ "$cache_file" ;; #( *) mv -f confcache "$cache_file" ;; esac fi fi else { $as_echo "$as_me:${as_lineno-$LINENO}: not updating unwritable cache $cache_file" >&5 $as_echo "$as_me: not updating unwritable cache $cache_file" >&6;} fi fi rm -f confcache test "x$prefix" = xNONE && prefix=$ac_default_prefix # Let make expand exec_prefix. test "x$exec_prefix" = xNONE && exec_prefix='${prefix}' # Transform confdefs.h into DEFS. # Protect against shell expansion while executing Makefile rules. # Protect against Makefile macro expansion. # # If the first sed substitution is executed (which looks for macros that # take arguments), then branch to the quote section. Otherwise, # look for a macro that doesn't take arguments. ac_script=' :mline /\\$/{ N s,\\\n,, b mline } t clear :clear s/^[ ]*#[ ]*define[ ][ ]*\([^ (][^ (]*([^)]*)\)[ ]*\(.*\)/-D\1=\2/g t quote s/^[ ]*#[ ]*define[ ][ ]*\([^ ][^ ]*\)[ ]*\(.*\)/-D\1=\2/g t quote b any :quote s/[ `~#$^&*(){}\\|;'\''"<>?]/\\&/g s/\[/\\&/g s/\]/\\&/g s/\$/$$/g H :any ${ g s/^\n// s/\n/ /g p } ' DEFS=`sed -n "$ac_script" confdefs.h` ac_libobjs= ac_ltlibobjs= U= for ac_i in : $LIBOBJS; do test "x$ac_i" = x: && continue # 1. Remove the extension, and $U if already installed. ac_script='s/\$U\././;s/\.o$//;s/\.obj$//' ac_i=`$as_echo "$ac_i" | sed "$ac_script"` # 2. Prepend LIBOBJDIR. When used with automake>=1.10 LIBOBJDIR # will be set to the directory where LIBOBJS objects are built. as_fn_append ac_libobjs " \${LIBOBJDIR}$ac_i\$U.$ac_objext" as_fn_append ac_ltlibobjs " \${LIBOBJDIR}$ac_i"'$U.lo' done LIBOBJS=$ac_libobjs LTLIBOBJS=$ac_ltlibobjs : "${CONFIG_STATUS=./config.status}" ac_write_fail=0 ac_clean_files_save=$ac_clean_files ac_clean_files="$ac_clean_files $CONFIG_STATUS" { $as_echo "$as_me:${as_lineno-$LINENO}: creating $CONFIG_STATUS" >&5 $as_echo "$as_me: creating $CONFIG_STATUS" >&6;} as_write_fail=0 cat >$CONFIG_STATUS <<_ASEOF || as_write_fail=1 #! $SHELL # Generated by $as_me. # Run this file to recreate the current configuration. # Compiler output produced by configure, useful for debugging # configure, is in config.log if it exists. debug=false ac_cs_recheck=false ac_cs_silent=false SHELL=\${CONFIG_SHELL-$SHELL} export SHELL _ASEOF cat >>$CONFIG_STATUS <<\_ASEOF || as_write_fail=1 ## -------------------- ## ## M4sh Initialization. ## ## -------------------- ## # Be more Bourne compatible DUALCASE=1; export DUALCASE # for MKS sh if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then : emulate sh NULLCMD=: # Pre-4.2 versions of Zsh do word splitting on ${1+"$@"}, which # is contrary to our usage. Disable this feature. alias -g '${1+"$@"}'='"$@"' setopt NO_GLOB_SUBST else case `(set -o) 2>/dev/null` in #( *posix*) : set -o posix ;; #( *) : ;; esac fi as_nl=' ' export as_nl # Printing a long string crashes Solaris 7 /usr/bin/printf. as_echo='\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo as_echo=$as_echo$as_echo$as_echo$as_echo$as_echo$as_echo # Prefer a ksh shell builtin over an external printf program on Solaris, # but without wasting forks for bash or zsh. if test -z "$BASH_VERSION$ZSH_VERSION" \ && (test "X`print -r -- $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='print -r --' as_echo_n='print -rn --' elif (test "X`printf %s $as_echo`" = "X$as_echo") 2>/dev/null; then as_echo='printf %s\n' as_echo_n='printf %s' else if test "X`(/usr/ucb/echo -n -n $as_echo) 2>/dev/null`" = "X-n $as_echo"; then as_echo_body='eval /usr/ucb/echo -n "$1$as_nl"' as_echo_n='/usr/ucb/echo -n' else as_echo_body='eval expr "X$1" : "X\\(.*\\)"' as_echo_n_body='eval arg=$1; case $arg in #( *"$as_nl"*) expr "X$arg" : "X\\(.*\\)$as_nl"; arg=`expr "X$arg" : ".*$as_nl\\(.*\\)"`;; esac; expr "X$arg" : "X\\(.*\\)" | tr -d "$as_nl" ' export as_echo_n_body as_echo_n='sh -c $as_echo_n_body as_echo' fi export as_echo_body as_echo='sh -c $as_echo_body as_echo' fi # The user is always right. if test "${PATH_SEPARATOR+set}" != set; then PATH_SEPARATOR=: (PATH='/bin;/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 && { (PATH='/bin:/bin'; FPATH=$PATH; sh -c :) >/dev/null 2>&1 || PATH_SEPARATOR=';' } fi # IFS # We need space, tab and new line, in precisely that order. Quoting is # there to prevent editors from complaining about space-tab. # (If _AS_PATH_WALK were called with IFS unset, it would disable word # splitting by setting IFS to empty value.) IFS=" "" $as_nl" # Find who we are. Look in the path if we contain no directory separator. as_myself= case $0 in #(( *[\\/]* ) as_myself=$0 ;; *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR for as_dir in $PATH do IFS=$as_save_IFS test -z "$as_dir" && as_dir=. test -r "$as_dir/$0" && as_myself=$as_dir/$0 && break done IFS=$as_save_IFS ;; esac # We did not find ourselves, most probably we were run as `sh COMMAND' # in which case we are not to be found in the path. if test "x$as_myself" = x; then as_myself=$0 fi if test ! -f "$as_myself"; then $as_echo "$as_myself: error: cannot find myself; rerun with an absolute file name" >&2 exit 1 fi # Unset variables that we do not need and which cause bugs (e.g. in # pre-3.0 UWIN ksh). But do not cause bugs in bash 2.01; the "|| exit 1" # suppresses any "Segmentation fault" message there. '((' could # trigger a bug in pdksh 5.2.14. for as_var in BASH_ENV ENV MAIL MAILPATH do eval test x\${$as_var+set} = xset \ && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || : done PS1='$ ' PS2='> ' PS4='+ ' # NLS nuisances. LC_ALL=C export LC_ALL LANGUAGE=C export LANGUAGE # CDPATH. (unset CDPATH) >/dev/null 2>&1 && unset CDPATH # as_fn_error STATUS ERROR [LINENO LOG_FD] # ---------------------------------------- # Output "`basename $0`: error: ERROR" to stderr. If LINENO and LOG_FD are # provided, also output the error to LOG_FD, referencing LINENO. Then exit the # script with STATUS, using 1 if that was 0. as_fn_error () { as_status=$1; test $as_status -eq 0 && as_status=1 if test "$4"; then as_lineno=${as_lineno-"$3"} as_lineno_stack=as_lineno_stack=$as_lineno_stack $as_echo "$as_me:${as_lineno-$LINENO}: error: $2" >&$4 fi $as_echo "$as_me: error: $2" >&2 as_fn_exit $as_status } # as_fn_error # as_fn_set_status STATUS # ----------------------- # Set $? to STATUS, without forking. as_fn_set_status () { return $1 } # as_fn_set_status # as_fn_exit STATUS # ----------------- # Exit the shell with STATUS, even in a "trap 0" or "set -e" context. as_fn_exit () { set +e as_fn_set_status $1 exit $1 } # as_fn_exit # as_fn_unset VAR # --------------- # Portably unset VAR. as_fn_unset () { { eval $1=; unset $1;} } as_unset=as_fn_unset # as_fn_append VAR VALUE # ---------------------- # Append the text in VALUE to the end of the definition contained in VAR. Take # advantage of any shell optimizations that allow amortized linear growth over # repeated appends, instead of the typical quadratic growth present in naive # implementations. if (eval "as_var=1; as_var+=2; test x\$as_var = x12") 2>/dev/null; then : eval 'as_fn_append () { eval $1+=\$2 }' else as_fn_append () { eval $1=\$$1\$2 } fi # as_fn_append # as_fn_arith ARG... # ------------------ # Perform arithmetic evaluation on the ARGs, and store the result in the # global $as_val. Take advantage of shells that can avoid forks. The arguments # must be portable across $(()) and expr. if (eval "test \$(( 1 + 1 )) = 2") 2>/dev/null; then : eval 'as_fn_arith () { as_val=$(( $* )) }' else as_fn_arith () { as_val=`expr "$@" || test $? -eq 1` } fi # as_fn_arith if expr a : '\(a\)' >/dev/null 2>&1 && test "X`expr 00001 : '.*\(...\)'`" = X001; then as_expr=expr else as_expr=false fi if (basename -- /) >/dev/null 2>&1 && test "X`basename -- / 2>&1`" = "X/"; then as_basename=basename else as_basename=false fi if (as_dir=`dirname -- /` && test "X$as_dir" = X/) >/dev/null 2>&1; then as_dirname=dirname else as_dirname=false fi as_me=`$as_basename -- "$0" || $as_expr X/"$0" : '.*/\([^/][^/]*\)/*$' \| \ X"$0" : 'X\(//\)$' \| \ X"$0" : 'X\(/\)' \| . 2>/dev/null || $as_echo X/"$0" | sed '/^.*\/\([^/][^/]*\)\/*$/{ s//\1/ q } /^X\/\(\/\/\)$/{ s//\1/ q } /^X\/\(\/\).*/{ s//\1/ q } s/.*/./; q'` # Avoid depending upon Character Ranges. as_cr_letters='abcdefghijklmnopqrstuvwxyz' as_cr_LETTERS='ABCDEFGHIJKLMNOPQRSTUVWXYZ' as_cr_Letters=$as_cr_letters$as_cr_LETTERS as_cr_digits='0123456789' as_cr_alnum=$as_cr_Letters$as_cr_digits ECHO_C= ECHO_N= ECHO_T= case `echo -n x` in #((((( -n*) case `echo 'xy\c'` in *c*) ECHO_T=' ';; # ECHO_T is single tab character. xy) ECHO_C='\c';; *) echo `echo ksh88 bug on AIX 6.1` > /dev/null ECHO_T=' ';; esac;; *) ECHO_N='-n';; esac rm -f conf$$ conf$$.exe conf$$.file if test -d conf$$.dir; then rm -f conf$$.dir/conf$$.file else rm -f conf$$.dir mkdir conf$$.dir 2>/dev/null fi if (echo >conf$$.file) 2>/dev/null; then if ln -s conf$$.file conf$$ 2>/dev/null; then as_ln_s='ln -s' # ... but there are two gotchas: # 1) On MSYS, both `ln -s file dir' and `ln file dir' fail. # 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable. # In both cases, we have to default to `cp -pR'. ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe || as_ln_s='cp -pR' elif ln conf$$.file conf$$ 2>/dev/null; then as_ln_s=ln else as_ln_s='cp -pR' fi else as_ln_s='cp -pR' fi rm -f conf$$ conf$$.exe conf$$.dir/conf$$.file conf$$.file rmdir conf$$.dir 2>/dev/null # as_fn_mkdir_p # ------------- # Create "$as_dir" as a directory, including parents if necessary. as_fn_mkdir_p () { case $as_dir in #( -*) as_dir=./$as_dir;; esac test -d "$as_dir" || eval $as_mkdir_p || { as_dirs= while :; do case $as_dir in #( *\'*) as_qdir=`$as_echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #'( *) as_qdir=$as_dir;; esac as_dirs="'$as_qdir' $as_dirs" as_dir=`$as_dirname -- "$as_dir" || $as_expr X"$as_dir" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$as_dir" : 'X\(//\)[^/]' \| \ X"$as_dir" : 'X\(//\)$' \| \ X"$as_dir" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$as_dir" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` test -d "$as_dir" && break done test -z "$as_dirs" || eval "mkdir $as_dirs" } || test -d "$as_dir" || as_fn_error $? "cannot create directory $as_dir" } # as_fn_mkdir_p if mkdir -p . 2>/dev/null; then as_mkdir_p='mkdir -p "$as_dir"' else test -d ./-p && rmdir ./-p as_mkdir_p=false fi # as_fn_executable_p FILE # ----------------------- # Test if FILE is an executable regular file. as_fn_executable_p () { test -f "$1" && test -x "$1" } # as_fn_executable_p as_test_x='test -x' as_executable_p=as_fn_executable_p # Sed expression to map a string onto a valid CPP name. as_tr_cpp="eval sed 'y%*$as_cr_letters%P$as_cr_LETTERS%;s%[^_$as_cr_alnum]%_%g'" # Sed expression to map a string onto a valid variable name. as_tr_sh="eval sed 'y%*+%pp%;s%[^_$as_cr_alnum]%_%g'" exec 6>&1 ## ----------------------------------- ## ## Main body of $CONFIG_STATUS script. ## ## ----------------------------------- ## _ASEOF test $as_write_fail = 0 && chmod +x $CONFIG_STATUS || ac_write_fail=1 cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # Save the log message, to keep $0 and so on meaningful, and to # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" This file was extended by onedrive $as_me v2.5.10, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES CONFIG_HEADERS = $CONFIG_HEADERS CONFIG_LINKS = $CONFIG_LINKS CONFIG_COMMANDS = $CONFIG_COMMANDS $ $0 $@ on `(hostname || uname -n) 2>/dev/null | sed 1q` " _ACEOF case $ac_config_files in *" "*) set x $ac_config_files; shift; ac_config_files=$*;; esac cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 # Files that config.status was made for. config_files="$ac_config_files" _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 ac_cs_usage="\ \`$as_me' instantiates files and other configuration actions from templates according to the current configuration. Unless the files and actions are specified as TAGs, all are instantiated by default. Usage: $0 [OPTION]... [TAG]... -h, --help print this help, then exit -V, --version print version number and configuration settings, then exit --config print configuration, then exit -q, --quiet, --silent do not print progress messages -d, --debug don't remove temporary files --recheck update $as_me by reconfiguring in the same conditions --file=FILE[:TEMPLATE] instantiate the configuration file FILE Configuration files: $config_files Report bugs to ." _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ onedrive config.status v2.5.10 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" Copyright (C) 2012 Free Software Foundation, Inc. This config.status script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it." ac_pwd='$ac_pwd' srcdir='$srcdir' INSTALL='$INSTALL' test -n "\$AWK" || AWK=awk _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # The default lists apply if the user does not specify any file. ac_need_defaults=: while test $# != 0 do case $1 in --*=?*) ac_option=`expr "X$1" : 'X\([^=]*\)='` ac_optarg=`expr "X$1" : 'X[^=]*=\(.*\)'` ac_shift=: ;; --*=) ac_option=`expr "X$1" : 'X\([^=]*\)='` ac_optarg= ac_shift=: ;; *) ac_option=$1 ac_optarg=$2 ac_shift=shift ;; esac case $ac_option in # Handling of the options. -recheck | --recheck | --rechec | --reche | --rech | --rec | --re | --r) ac_cs_recheck=: ;; --version | --versio | --versi | --vers | --ver | --ve | --v | -V ) $as_echo "$ac_cs_version"; exit ;; --config | --confi | --conf | --con | --co | --c ) $as_echo "$ac_cs_config"; exit ;; --debug | --debu | --deb | --de | --d | -d ) debug=: ;; --file | --fil | --fi | --f ) $ac_shift case $ac_optarg in *\'*) ac_optarg=`$as_echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"` ;; '') as_fn_error $? "missing file argument" ;; esac as_fn_append CONFIG_FILES " '$ac_optarg'" ac_need_defaults=false;; --he | --h | --help | --hel | -h ) $as_echo "$ac_cs_usage"; exit ;; -q | -quiet | --quiet | --quie | --qui | --qu | --q \ | -silent | --silent | --silen | --sile | --sil | --si | --s) ac_cs_silent=: ;; # This is an error. -*) as_fn_error $? "unrecognized option: \`$1' Try \`$0 --help' for more information." ;; *) as_fn_append ac_config_targets " $1" ac_need_defaults=false ;; esac shift done ac_configure_extra_args= if $ac_cs_silent; then exec 6>/dev/null ac_configure_extra_args="$ac_configure_extra_args --silent" fi _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 if \$ac_cs_recheck; then set X $SHELL '$0' $ac_configure_args \$ac_configure_extra_args --no-create --no-recursion shift \$as_echo "running CONFIG_SHELL=$SHELL \$*" >&6 CONFIG_SHELL='$SHELL' export CONFIG_SHELL exec "\$@" fi _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 exec 5>>config.log { echo sed 'h;s/./-/g;s/^.../## /;s/...$/ ##/;p;x;p;x' <<_ASBOX ## Running $as_me. ## _ASBOX $as_echo "$ac_log" } >&5 _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # Handling of arguments. for ac_config_target in $ac_config_targets do case $ac_config_target in "Makefile") CONFIG_FILES="$CONFIG_FILES Makefile" ;; "contrib/pacman/PKGBUILD") CONFIG_FILES="$CONFIG_FILES contrib/pacman/PKGBUILD" ;; "contrib/spec/onedrive.spec") CONFIG_FILES="$CONFIG_FILES contrib/spec/onedrive.spec" ;; "onedrive.1") CONFIG_FILES="$CONFIG_FILES onedrive.1" ;; "contrib/systemd/onedrive.service") CONFIG_FILES="$CONFIG_FILES contrib/systemd/onedrive.service" ;; "contrib/systemd/onedrive@.service") CONFIG_FILES="$CONFIG_FILES contrib/systemd/onedrive@.service" ;; *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5;; esac done # If the user did not use the arguments to specify the items to instantiate, # then the envvar interface is used. Set only those that are not. # We use the long form for the default assignment because of an extremely # bizarre bug on SunOS 4.1.3. if $ac_need_defaults; then test "${CONFIG_FILES+set}" = set || CONFIG_FILES=$config_files fi # Have a temporary directory for convenience. Make it in the build tree # simply because there is no reason against having it here, and in addition, # creating and moving files from /tmp can sometimes cause problems. # Hook for its removal unless debugging. # Note that there is a small window in which the directory will not be cleaned: # after its creation but before its name has been assigned to `$tmp'. $debug || { tmp= ac_tmp= trap 'exit_status=$? : "${ac_tmp:=$tmp}" { test ! -d "$ac_tmp" || rm -fr "$ac_tmp"; } && exit $exit_status ' 0 trap 'as_fn_exit 1' 1 2 13 15 } # Create a (secure) tmp directory for tmp files. { tmp=`(umask 077 && mktemp -d "./confXXXXXX") 2>/dev/null` && test -d "$tmp" } || { tmp=./conf$$-$RANDOM (umask 077 && mkdir "$tmp") } || as_fn_error $? "cannot create a temporary directory in ." "$LINENO" 5 ac_tmp=$tmp # Set up the scripts for CONFIG_FILES section. # No need to generate them if there are no CONFIG_FILES. # This happens for instance with `./config.status config.h'. if test -n "$CONFIG_FILES"; then ac_cr=`echo X | tr X '\015'` # On cygwin, bash can eat \r inside `` if the user requested igncr. # But we know of no other shell where ac_cr would be empty at this # point, so we can use a bashism as a fallback. if test "x$ac_cr" = x; then eval ac_cr=\$\'\\r\' fi ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' /dev/null` if test "$ac_cs_awk_cr" = "a${ac_cr}b"; then ac_cs_awk_cr='\\r' else ac_cs_awk_cr=$ac_cr fi echo 'BEGIN {' >"$ac_tmp/subs1.awk" && _ACEOF { echo "cat >conf$$subs.awk <<_ACEOF" && echo "$ac_subst_vars" | sed 's/.*/&!$&$ac_delim/' && echo "_ACEOF" } >conf$$subs.sh || as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 ac_delim_num=`echo "$ac_subst_vars" | grep -c '^'` ac_delim='%!_!# ' for ac_last_try in false false false false false :; do . ./conf$$subs.sh || as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 ac_delim_n=`sed -n "s/.*$ac_delim\$/X/p" conf$$subs.awk | grep -c X` if test $ac_delim_n = $ac_delim_num; then break elif $ac_last_try; then as_fn_error $? "could not make $CONFIG_STATUS" "$LINENO" 5 else ac_delim="$ac_delim!$ac_delim _$ac_delim!! " fi done rm -f conf$$subs.sh cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>"\$ac_tmp/subs1.awk" <<\\_ACAWK && _ACEOF sed -n ' h s/^/S["/; s/!.*/"]=/ p g s/^[^!]*!// :repl t repl s/'"$ac_delim"'$// t delim :nl h s/\(.\{148\}\)..*/\1/ t more1 s/["\\]/\\&/g; s/^/"/; s/$/\\n"\\/ p n b repl :more1 s/["\\]/\\&/g; s/^/"/; s/$/"\\/ p g s/.\{148\}// t nl :delim h s/\(.\{148\}\)..*/\1/ t more2 s/["\\]/\\&/g; s/^/"/; s/$/"/ p b :more2 s/["\\]/\\&/g; s/^/"/; s/$/"\\/ p g s/.\{148\}// t delim ' >$CONFIG_STATUS || ac_write_fail=1 rm -f conf$$subs.awk cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 _ACAWK cat >>"\$ac_tmp/subs1.awk" <<_ACAWK && for (key in S) S_is_set[key] = 1 FS = "" } { line = $ 0 nfields = split(line, field, "@") substed = 0 len = length(field[1]) for (i = 2; i < nfields; i++) { key = field[i] keylen = length(key) if (S_is_set[key]) { value = S[key] line = substr(line, 1, len) "" value "" substr(line, len + keylen + 3) len += length(value) + length(field[++i]) substed = 1 } else len += 1 + keylen } print line } _ACAWK _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 if sed "s/$ac_cr//" < /dev/null > /dev/null 2>&1; then sed "s/$ac_cr\$//; s/$ac_cr/$ac_cs_awk_cr/g" else cat fi < "$ac_tmp/subs1.awk" > "$ac_tmp/subs.awk" \ || as_fn_error $? "could not setup config files machinery" "$LINENO" 5 _ACEOF # VPATH may cause trouble with some makes, so we remove sole $(srcdir), # ${srcdir} and @srcdir@ entries from VPATH if srcdir is ".", strip leading and # trailing colons and then remove the whole line if VPATH becomes empty # (actually we leave an empty line to preserve line numbers). if test "x$srcdir" = x.; then ac_vpsub='/^[ ]*VPATH[ ]*=[ ]*/{ h s/// s/^/:/ s/[ ]*$/:/ s/:\$(srcdir):/:/g s/:\${srcdir}:/:/g s/:@srcdir@:/:/g s/^:*// s/:*$// x s/\(=[ ]*\).*/\1/ G s/\n// s/^[^=]*=[ ]*$// }' fi cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 fi # test -n "$CONFIG_FILES" eval set X " :F $CONFIG_FILES " shift for ac_tag do case $ac_tag in :[FHLC]) ac_mode=$ac_tag; continue;; esac case $ac_mode$ac_tag in :[FHL]*:*);; :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5;; :[FH]-) ac_tag=-:-;; :[FH]*) ac_tag=$ac_tag:$ac_tag.in;; esac ac_save_IFS=$IFS IFS=: set x $ac_tag IFS=$ac_save_IFS shift ac_file=$1 shift case $ac_mode in :L) ac_source=$1;; :[FH]) ac_file_inputs= for ac_f do case $ac_f in -) ac_f="$ac_tmp/stdin";; *) # Look for the file first in the build tree, then in the source tree # (if the path is not absolute). The absolute path cannot be DOS-style, # because $ac_f cannot contain `:'. test -f "$ac_f" || case $ac_f in [\\/$]*) false;; *) test -f "$srcdir/$ac_f" && ac_f="$srcdir/$ac_f";; esac || as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5;; esac case $ac_f in *\'*) ac_f=`$as_echo "$ac_f" | sed "s/'/'\\\\\\\\''/g"`;; esac as_fn_append ac_file_inputs " '$ac_f'" done # Let's still pretend it is `configure' which instantiates (i.e., don't # use $as_me), people would be surprised to read: # /* config.h. Generated by config.status. */ configure_input='Generated from '` $as_echo "$*" | sed 's|^[^:]*/||;s|:[^:]*/|, |g' `' by configure.' if test x"$ac_file" != x-; then configure_input="$ac_file. $configure_input" { $as_echo "$as_me:${as_lineno-$LINENO}: creating $ac_file" >&5 $as_echo "$as_me: creating $ac_file" >&6;} fi # Neutralize special characters interpreted by sed in replacement strings. case $configure_input in #( *\&* | *\|* | *\\* ) ac_sed_conf_input=`$as_echo "$configure_input" | sed 's/[\\\\&|]/\\\\&/g'`;; #( *) ac_sed_conf_input=$configure_input;; esac case $ac_tag in *:-:* | *:-) cat >"$ac_tmp/stdin" \ || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;; esac ;; esac ac_dir=`$as_dirname -- "$ac_file" || $as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ X"$ac_file" : 'X\(//\)[^/]' \| \ X"$ac_file" : 'X\(//\)$' \| \ X"$ac_file" : 'X\(/\)' \| . 2>/dev/null || $as_echo X"$ac_file" | sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/ q } /^X\(\/\/\)[^/].*/{ s//\1/ q } /^X\(\/\/\)$/{ s//\1/ q } /^X\(\/\).*/{ s//\1/ q } s/.*/./; q'` as_dir="$ac_dir"; as_fn_mkdir_p ac_builddir=. case "$ac_dir" in .) ac_dir_suffix= ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_dir_suffix=/`$as_echo "$ac_dir" | sed 's|^\.[\\/]||'` # A ".." for each directory in $ac_dir_suffix. ac_top_builddir_sub=`$as_echo "$ac_dir_suffix" | sed 's|/[^\\/]*|/..|g;s|/||'` case $ac_top_builddir_sub in "") ac_top_builddir_sub=. ac_top_build_prefix= ;; *) ac_top_build_prefix=$ac_top_builddir_sub/ ;; esac ;; esac ac_abs_top_builddir=$ac_pwd ac_abs_builddir=$ac_pwd$ac_dir_suffix # for backward compatibility: ac_top_builddir=$ac_top_build_prefix case $srcdir in .) # We are building in place. ac_srcdir=. ac_top_srcdir=$ac_top_builddir_sub ac_abs_top_srcdir=$ac_pwd ;; [\\/]* | ?:[\\/]* ) # Absolute name. ac_srcdir=$srcdir$ac_dir_suffix; ac_top_srcdir=$srcdir ac_abs_top_srcdir=$srcdir ;; *) # Relative name. ac_srcdir=$ac_top_build_prefix$srcdir$ac_dir_suffix ac_top_srcdir=$ac_top_build_prefix$srcdir ac_abs_top_srcdir=$ac_pwd/$srcdir ;; esac ac_abs_srcdir=$ac_abs_top_srcdir$ac_dir_suffix case $ac_mode in :F) # # CONFIG_FILE # case $INSTALL in [\\/$]* | ?:[\\/]* ) ac_INSTALL=$INSTALL ;; *) ac_INSTALL=$ac_top_build_prefix$INSTALL ;; esac _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # If the template does not know about datarootdir, expand it. # FIXME: This hack should be removed a few years after 2.60. ac_datarootdir_hack=; ac_datarootdir_seen= ac_sed_dataroot=' /datarootdir/ { p q } /@datadir@/p /@docdir@/p /@infodir@/p /@localedir@/p /@mandir@/p' case `eval "sed -n \"\$ac_sed_dataroot\" $ac_file_inputs"` in *datarootdir*) ac_datarootdir_seen=yes;; *@datadir@*|*@docdir@*|*@infodir@*|*@localedir@*|*@mandir@*) { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file_inputs seems to ignore the --datarootdir setting" >&5 $as_echo "$as_me: WARNING: $ac_file_inputs seems to ignore the --datarootdir setting" >&2;} _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_datarootdir_hack=' s&@datadir@&$datadir&g s&@docdir@&$docdir&g s&@infodir@&$infodir&g s&@localedir@&$localedir&g s&@mandir@&$mandir&g s&\\\${datarootdir}&$datarootdir&g' ;; esac _ACEOF # Neutralize VPATH when `$srcdir' = `.'. # Shell code in configure.ac might set extrasub. # FIXME: do we really want to maintain this feature? cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_sed_extra="$ac_vpsub $extrasub _ACEOF cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 :t /@[a-zA-Z_][a-zA-Z_0-9]*@/!b s|@configure_input@|$ac_sed_conf_input|;t t s&@top_builddir@&$ac_top_builddir_sub&;t t s&@top_build_prefix@&$ac_top_build_prefix&;t t s&@srcdir@&$ac_srcdir&;t t s&@abs_srcdir@&$ac_abs_srcdir&;t t s&@top_srcdir@&$ac_top_srcdir&;t t s&@abs_top_srcdir@&$ac_abs_top_srcdir&;t t s&@builddir@&$ac_builddir&;t t s&@abs_builddir@&$ac_abs_builddir&;t t s&@abs_top_builddir@&$ac_abs_top_builddir&;t t s&@INSTALL@&$ac_INSTALL&;t t $ac_datarootdir_hack " eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$ac_tmp/subs.awk" \ >$ac_tmp/out || as_fn_error $? "could not create $ac_file" "$LINENO" 5 test -z "$ac_datarootdir_hack$ac_datarootdir_seen" && { ac_out=`sed -n '/\${datarootdir}/p' "$ac_tmp/out"`; test -n "$ac_out"; } && { ac_out=`sed -n '/^[ ]*datarootdir[ ]*:*=/p' \ "$ac_tmp/out"`; test -z "$ac_out"; } && { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file contains a reference to the variable \`datarootdir' which seems to be undefined. Please make sure it is defined" >&5 $as_echo "$as_me: WARNING: $ac_file contains a reference to the variable \`datarootdir' which seems to be undefined. Please make sure it is defined" >&2;} rm -f "$ac_tmp/stdin" case $ac_file in -) cat "$ac_tmp/out" && rm -f "$ac_tmp/out";; *) rm -f "$ac_file" && mv "$ac_tmp/out" "$ac_file";; esac \ || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;; esac done # for ac_tag as_fn_exit 0 _ACEOF ac_clean_files=$ac_clean_files_save test $ac_write_fail = 0 || as_fn_error $? "write failure creating $CONFIG_STATUS" "$LINENO" 5 # configure is writing to config.log, and then calls config.status. # config.status does its own redirection, appending to config.log. # Unfortunately, on DOS this fails, as config.log is still kept open # by configure, so config.status won't be able to write to it; its # output is simply discarded. So we exec the FD to /dev/null, # effectively closing config.log, so it can be properly (re)opened and # appended to by config.status. When coming back to configure, we # need to make the FD available again. if test "$no_create" != yes; then ac_cs_success=: ac_config_status_args= test "$silent" = yes && ac_config_status_args="$ac_config_status_args --quiet" exec 5>/dev/null $SHELL $CONFIG_STATUS $ac_config_status_args || ac_cs_success=false exec 5>>config.log # Use ||, not &&, to avoid exiting from the if with $? = 1, which # would make configure fail if this is the last instruction. $ac_cs_success || as_fn_exit 1 fi if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: unrecognized options: $ac_unrecognized_opts" >&5 $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;} fi ================================================ FILE: configure.ac ================================================ dnl configure.ac for OneDrive Linux Client dnl Copyright 2019 Norbert Preining dnl Licensed GPL v3 or later dnl How to make a release dnl - increase the version number in the AC_INIT call below dnl - change PACKAGE_DATE to 'Month YYYY' to ensure man page has the correct date dnl - run autoconf which generates configure dnl - commit the changed files (configure.ac, configure) dnl - tag the release AC_PREREQ([2.69]) AC_INIT([onedrive],[v2.5.10], [https://github.com/abraunegg/onedrive], [onedrive]) AC_CONFIG_SRCDIR([src/main.d]) AC_ARG_VAR([DC], [D compiler executable]) AC_ARG_VAR([DCFLAGS], [flags for D compiler]) dnl necessary programs: install, pkg-config AC_PROG_INSTALL PKG_PROG_PKG_CONFIG PACKAGE_DATE="January 2026" AC_SUBST([PACKAGE_DATE]) dnl Determine D compiler dnl we check for dmd, dmd2, and ldc2 in this order dnl furthermore, we set DC_TYPE to either dmd or ldc and export this into the dnl Makefile so that we can adjust command line arguments AC_CHECK_PROGS([DC], [dmd ldmd2 ldc2 gdmd gdc], NOT_FOUND) DC_TYPE= case $(basename $DC) in *ldc2*) DC_TYPE=ldc ;; *gdc*) DC_TYPE=gdc ;; *dmd*) DC_TYPE=dmd ;; NOT_FOUND) AC_MSG_ERROR(Could not find any compatible D compiler, 1) esac dnl dash/POSIX version of version comparison vercomp () { IFS=. read -r a0 a1 a2 aa <' $bb then return 1 else return 0 fi fi fi fi } DO_VERSION_CHECK=1 AC_ARG_ENABLE(version-check, AS_HELP_STRING([--disable-version-check], [Disable checks of compiler version during configure time])) AS_IF([test "x$enable_version_check" = "xno"], DO_VERSION_CHECK=0,) AS_IF([test "$DO_VERSION_CHECK" = "1"], [ dnl do the version check AC_MSG_CHECKING([version of D compiler]) # check for valid versions case $(basename $DC) in *ldmd2*|*ldc2*) # LDC - the LLVM D compiler (1.12.0): ... VERSION=`$DC --version` # remove everything up to first ( VERSION=${VERSION#* (} # remove everything after ): VERSION=${VERSION%%):*} # now version should be something like L.M.N MINVERSION=1.20.1 ;; *gdmd*|*gdc*) # Both gdmd and gdc print the same version information VERSION=`${DC} --version | head -n1` # Some examples of output: # gdc (Gentoo 14.2.1_p20250301 p8) 14.2.1 20250301 # gcc (GCC) 14.2.1 20250207 # Arch # gdc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7) # gdc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 VERSION=${VERSION#gdc } # VERSION=(...) VER DATE ... VERSION=${VERSION#*) } # VERSION=VER DATE ... VERSION=${VERSION%% *} MINVERSION=15 ;; *dmd*) # DMD64 D Compiler v2.085.1\n... VERSION=`$DC --version | tr '\n' ' '` VERSION=${VERSION#*Compiler v} VERSION=${VERSION%% *} # now version should be something like L.M.N MINVERSION=2.091.1 ;; esac AC_MSG_RESULT([$VERSION]) vercomp $MINVERSION $VERSION if test $? = 1 then AC_MSG_ERROR([Compiler version insufficient, current compiler version $VERSION, minimum version $MINVERSION], 1) fi #echo "MINVERSION=$MINVERSION VERSION=$VERSION" ]) dnl In case the environment variable DCFLAGS is set, we export it to the dnl generated Makefile at configure run: AC_SUBST([DCFLAGS]) dnl Default flags for each compiler case "$DC_TYPE" in dmd) DEBUG_DCFLAGS="-g -debug -gs" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-version LINKER_DCFLAG=-L OUTPUT_DCFLAG=-of WERROR_DCFLAG=-w ;; ldc) DEBUG_DCFLAGS="-g -d-debug -gc" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-d-version LINKER_DCFLAG=-L OUTPUT_DCFLAG=-of WERROR_DCFLAG=-w ;; gdc) DEBUG_DCFLAGS="-g -fdebug" RELEASE_DCFLAGS=-O VERSION_DCFLAG=-fversion LINKER_DCFLAG=-Wl, OUTPUT_DCFLAG=-o WERROR_DCFLAG=-Werror ;; esac AC_SUBST([DEBUG_DCFLAGS]) AC_SUBST([RELEASE_DCFLAGS]) AC_SUBST([VERSION_DCFLAG]) AC_SUBST([LINKER_DCFLAG]) AC_SUBST([OUTPUT_DCFLAG]) AC_SUBST([WERROR_DCFLAG]) dnl Check for required modules: curl, sqlite and dbus if required PKG_CHECK_MODULES([curl],[libcurl]) PKG_CHECK_MODULES([sqlite],[sqlite3]) AC_MSG_CHECKING([whether to enable dbus support]) case "$(uname -s)" in Linux) enable_dbus=yes AC_MSG_RESULT([yes (on Linux)]) PKG_CHECK_MODULES([dbus], [dbus-1 >= 1.0], [AC_DEFINE([HAVE_DBUS], [1], [Define if you have dbus-1])] , [AC_MSG_ERROR([dbus-1 development files not found. Please install dbus-devel (Red Hat), libdbus-1-dev (Debian) or dbus (Arch | Manjaro)])] ) ;; *) enable_dbus=no AC_MSG_RESULT([no (not on Linux)]) ;; esac AC_SUBST([enable_dbus]) dnl dnl systemd and unit file directories dnl This is a bit tricky, because we want to allow for dnl --with-systemdsystemunitdir=auto dnl as well as =/path/to/dir dnl The first step is that we check whether the --with options is passed to configure run dnl if yes, we don't do anything (the ,, at the end of the next line), and if not, we dnl set with_systemdsystemunitdir=auto, meaning we will try pkg-config to find the correct dnl value. AC_ARG_WITH([systemdsystemunitdir], [AS_HELP_STRING([--with-systemdsystemunitdir=DIR], [Directory for systemd system service files])],, [with_systemdsystemunitdir=auto]) dnl If no value is passed in (or auto/yes is passed in), then we try to find the correct dnl value via pkg-config and put it into $def_systemdsystemunitdir AS_IF([test "x$with_systemdsystemunitdir" = "xyes" -o "x$with_systemdsystemunitdir" = "xauto"], [ dnl true part, so try to determine with pkg-config def_systemdsystemunitdir=$($PKG_CONFIG --variable=systemdsystemunitdir systemd) dnl if we cannot find it via pkg-config, *and* the user explicitly passed it in with, dnl we warn, and in all cases we unset (set to no) the respective variable AS_IF([test "x$def_systemdsystemunitdir" = "x"], [ dnl we couldn't find the default value via pkg-config AS_IF([test "x$with_systemdsystemunitdir" = "xyes"], [AC_MSG_ERROR([systemd support requested but pkg-config unable to query systemd package])]) with_systemdsystemunitdir=no ], [ dnl pkg-config found the value, use it with_systemdsystemunitdir="$def_systemdsystemunitdir" ] ) ] ) dnl finally, if we found a value, put it into the generated Makefile AS_IF([test "x$with_systemdsystemunitdir" != "xno"], [AC_SUBST([systemdsystemunitdir], [$with_systemdsystemunitdir])]) dnl Now do the same as above for systemduserunitdir! AC_ARG_WITH([systemduserunitdir], [AS_HELP_STRING([--with-systemduserunitdir=DIR], [Directory for systemd user service files])],, [with_systemduserunitdir=auto]) AS_IF([test "x$with_systemduserunitdir" = "xyes" -o "x$with_systemduserunitdir" = "xauto"], [ def_systemduserunitdir=$($PKG_CONFIG --variable=systemduserunitdir systemd) AS_IF([test "x$def_systemduserunitdir" = "x"], [ AS_IF([test "x$with_systemduserunitdir" = "xyes"], [AC_MSG_ERROR([systemd support requested but pkg-config unable to query systemd package])]) with_systemduserunitdir=no ], [ with_systemduserunitdir="$def_systemduserunitdir" ] ) ] ) AS_IF([test "x$with_systemduserunitdir" != "xno"], [AC_SUBST([systemduserunitdir], [$with_systemduserunitdir])]) dnl We enable systemd integration only if we have found both user/system unit dirs AS_IF([test "x$with_systemduserunitdir" != "xno" -a "x$with_systemdsystemunitdir" != "xno"], [havesystemd=yes], [havesystemd=no]) AC_SUBST([HAVE_SYSTEMD], $havesystemd) dnl dnl Notification support dnl only check for libnotify if --enable-notifications is given AC_ARG_ENABLE(notifications, AS_HELP_STRING([--enable-notifications], [Enable desktop notifications via libnotify])) AS_IF([test "x$enable_notifications" = "xyes"], [enable_notifications=yes], [enable_notifications=no]) dnl if --enable-notifications was given, check for libnotify, and disable if not found dnl otherwise substitute the notifu AS_IF([test "x$enable_notifications" = "xyes"], [PKG_CHECK_MODULES(notify,libnotify,,enable_notifications=no)], [AC_SUBST([notify_LIBS],"")]) AC_SUBST([NOTIFICATIONS],$enable_notifications) dnl dnl iNotify Support # Conditionally set bsd_inotify_LIBS based on the platform case "$(uname -s)" in Linux) bsd_inotify_LIBS="" ;; FreeBSD) AS_IF([test "$(uname -U)" -gt 1500060], [bsd_inotify_LIBS=""], [bsd_inotify_LIBS="-L/usr/local/lib -linotify"] ) ;; OpenBSD) bsd_inotify_LIBS="-L/usr/local/lib/inotify -linotify" ;; *) bsd_inotify_LIBS="" ;; esac AC_SUBST([bsd_inotify_LIBS]) dnl dnl Dynamic Linker Support # Conditionally set dynamic_linker_LIBS based on the platform case "$(uname -s)" in Linux) dynamic_linker_LIBS="-ldl" ;; *) dynamic_linker_LIBS="" ;; esac AC_SUBST([dynamic_linker_LIBS]) dnl dnl Completion support dnl First determine whether completions are requested, pass that to Makefile AC_ARG_ENABLE([completions], AS_HELP_STRING([--enable-completions], [Install shell completions for bash, zsh, and fish])) AS_IF([test "x$enable_completions" = "xyes"], [enable_completions=yes], [enable_completions=no]) AC_SUBST([COMPLETIONS],$enable_completions) dnl if completions are enabled, search for the bash/zsh completion directory in the dnl similar way as we did for the systemd directories AS_IF([test "x$enable_completions" = "xyes"],[ AC_ARG_WITH([bash-completion-dir], [AS_HELP_STRING([--with-bash-completion-dir=DIR], [Directory for bash completion files])], , [with_bash_completion_dir=auto]) AS_IF([test "x$with_bash_completion_dir" = "xyes" -o "x$with_bash_completion_dir" = "xauto"], [ PKG_CHECK_VAR(bashcompdir, [bash-completion], [completionsdir], , bashcompdir="${sysconfdir}/bash_completion.d") with_bash_completion_dir=$bashcompdir ]) AC_SUBST([BASH_COMPLETION_DIR], $with_bash_completion_dir) AC_ARG_WITH([zsh-completion-dir], [AS_HELP_STRING([--with-zsh-completion-dir=DIR], [Directory for zsh completion files])],, [with_zsh_completion_dir=auto]) AS_IF([test "x$with_zsh_completion_dir" = "xyes" -o "x$with_zsh_completion_dir" = "xauto"], [ with_zsh_completion_dir="/usr/local/share/zsh/site-functions" ]) AC_SUBST([ZSH_COMPLETION_DIR], $with_zsh_completion_dir) AC_ARG_WITH([fish-completion-dir], [AS_HELP_STRING([--with-fish-completion-dir=DIR], [Directory for fish completion files])],, [with_fish_completion_dir=auto]) AS_IF([test "x$with_fish_completion_dir" = "xyes" -o "x$with_fish_completion_dir" = "xauto"], [ with_fish_completion_dir="/usr/local/share/fish/completions" ]) AC_SUBST([FISH_COMPLETION_DIR], $with_fish_completion_dir) ]) dnl dnl Debug support AC_ARG_ENABLE(debug, AS_HELP_STRING([--enable-debug], [Pass debug option to the compiler])) AS_IF([test "x$enable_debug" = "xyes"], AC_SUBST([DEBUG],yes), AC_SUBST([DEBUG],no)) dnl generate necessary files AC_CONFIG_FILES([ Makefile contrib/pacman/PKGBUILD contrib/spec/onedrive.spec onedrive.1 contrib/systemd/onedrive.service contrib/systemd/onedrive@.service ]) AC_OUTPUT ================================================ FILE: contrib/completions/complete.bash ================================================ # BASH completion code for OneDrive Linux Client # (c) 2019 Norbert Preining # License: GPLv3+ (as with the rest of the OneDrive Linux client project) _onedrive() { local cur prev COMPREPLY=() cur=${COMP_WORDS[COMP_CWORD]} prev=${COMP_WORDS[COMP_CWORD-1]} options='--check-for-nomount --check-for-nosync --cleanup-local-files --debug-https --disable-notifications --display-config --display-quota --display-sync-status --disable-download-validation --disable-upload-validation --display-running-config --download-only --dry-run --enable-logging --force --force-http-11 --force-sync --list-shared-items --local-first --logout -m --monitor --no-remote-delete --print-access-token --reauth --remove-source-files --remove-source-folders --resync --resync-auth --skip-dir-strict-match --skip-dot-files --skip-symlinks -s --sync --sync-root-files --sync-shared-files --upload-only -v+ --verbose --version -h --help --with-editing-perms' argopts='--auth-files --auth-response --classify-as-big-delete --confdir --create-directory --create-share-link --destination-directory --download-file --file-fragment-size --get-O365-drive-id --get-file-link --get-sharepoint-drive-id --log-dir --modified-by --monitor-fullscan-frequency --monitor-interval --monitor-log-frequency --remove-directory --share-password --single-directory --skip-dir --skip-file --skip-size --source-directory --space-reservation --syncdir --threads' # Loop on the arguments to manage conflicting options for (( i=0; i < ${#COMP_WORDS[@]}-1; i++ )); do #exclude some mutually exclusive options [[ ${COMP_WORDS[i]} == '--sync' ]] && options=${options/--monitor} [[ ${COMP_WORDS[i]} == '--monitor' ]] && options=${options/--sync} done case "$prev" in --confdir|--syncdir) _filedir return 0 ;; --get-file-link) if command -v sed &> /dev/null; then pushd "$(onedrive --display-config | sed -n "/sync_dir/s/.*= //p")" &> /dev/null _filedir popd &> /dev/null fi return 0 ;; --create-directory|--get-O365-drive-id|--remove-directory|--single-directory|--source-directory) return 0 ;; *) COMPREPLY=( $( compgen -W "$options $argopts" -- "$cur")) return 0 ;; esac # notreached return 0 } complete -F _onedrive onedrive ================================================ FILE: contrib/completions/complete.fish ================================================ # FISH completions for OneDrive Linux Client # License: GPLv3+ (as with the rest of the OneDrive Linux client project) complete -c onedrive -f complete -c onedrive -l auth-files -d "Authenticate using input/output files" complete -c onedrive -l auth-response -d "Authenticate using the response URL" complete -c onedrive -l check-for-nomount -d "Skip sync if .nosync found in sync dir root" complete -c onedrive -l check-for-nosync -d "Skip directories containing .nosync" complete -c onedrive -l classify-as-big-delete -d "Classify as big delete when children exceed number" complete -c onedrive -l cleanup-local-files -d "Cleanup local files when using --download-only" complete -c onedrive -l confdir -d "Directory for configuration files" complete -c onedrive -l create-directory -d "Create directory on OneDrive" complete -c onedrive -l create-share-link -d "Create a shareable link for a file" complete -c onedrive -l debug-https -d "Debug HTTPS communication" complete -c onedrive -l destination-directory -d "Target directory for move/rename operations" complete -c onedrive -l disable-download-validation -d "Disable validation of downloaded files" complete -c onedrive -l disable-notifications -d "Disable desktop notifications in monitor mode" complete -c onedrive -l disable-upload-validation -d "Disable validation of uploaded files" complete -c onedrive -l display-config -d "Display current config" complete -c onedrive -l display-quota -d "Display OneDrive quota" complete -c onedrive -l display-running-config -d "Display config used at startup" complete -c onedrive -l display-sync-status -d "Show current sync status" complete -c onedrive -l download-file -d "Download a single file from Microsoft OneDrive" complete -c onedrive -l download-only -d "Only download remote changes" complete -c onedrive -l dry-run -d "Simulate sync without making changes" complete -c onedrive -l enable-logging -d "Enable logging to a file" complete -c onedrive -l file-fragment-size -d "Specify the file fragment size for large file uploads (in MB)" complete -c onedrive -l force -d "Force delete on big delete detection" complete -c onedrive -l force-http-11 -d "Force HTTP 1.1 usage" complete -c onedrive -l force-sync -d "Force sync of specified folder" complete -c onedrive -l get-file-link -d "Get shareable link for a file" complete -c onedrive -l get-O365-drive-id -d "Get Drive ID for O365 SharePoint (deprecated)" complete -c onedrive -l get-sharepoint-drive-id -d "Get Drive ID for SharePoint" complete -c onedrive -l help -d "Show help message" complete -c onedrive -l list-shared-items -d "List shared OneDrive items" complete -c onedrive -l local-first -d "Prefer local changes during sync" complete -c onedrive -l log-dir -d "Directory for logs" complete -c onedrive -l logout -d "Logout current session" complete -c onedrive -l modified-by -d "Show who last modified a file" complete -c onedrive -l monitor -d "Run in monitor mode" complete -c onedrive -l monitor-fullscan-frequency -d "Full scan every N runs" complete -c onedrive -l monitor-interval -d "Sync interval in monitor mode" complete -c onedrive -l monitor-log-frequency -d "Log status every N seconds in monitor mode" complete -c onedrive -l no-remote-delete -d "Don't delete remote files in --upload-only" complete -c onedrive -l print-access-token -d "Show access token" complete -c onedrive -l reauth -d "Reauthenticate client" complete -c onedrive -l remove-directory -d "Delete remote directory" complete -c onedrive -l remove-source-files -d "Remove uploaded local files" complete -c onedrive -l remove-source-folders -d "Remove the local directory structure post successful file transfer" complete -c onedrive -l resync -d "Perform full resync" complete -c onedrive -l resync-auth -d "Confirm resync action" complete -c onedrive -l share-password -d "Password-protect shared link" complete -c onedrive -l single-directory -d "Sync a single local directory" complete -c onedrive -l skip-dir -d "Skip matching directories" complete -c onedrive -l skip-dir-strict-match -d "Strict matching for skipped dirs" complete -c onedrive -l skip-dot-files -d "Skip hidden files and folders" complete -c onedrive -l skip-file -d "Skip matching files" complete -c onedrive -l skip-size -d "Skip files above given size" complete -c onedrive -l skip-symlinks -d "Ignore symlinks" complete -c onedrive -l source-directory -d "Source path for move/rename" complete -c onedrive -l space-reservation -d "Reserve disk space (MB)" complete -c onedrive -l sync -d "Start sync operation" complete -c onedrive -l syncdir -d "Local sync directory" complete -c onedrive -l synchronize -d "Deprecated alias for --sync" complete -c onedrive -l sync-root-files -d "Sync root files with sync_list" complete -c onedrive -l sync-shared-files -d "Sync shared business files" complete -c onedrive -l threads -d "Specify a value for the number of worker threads used for parallel upload and download operations" complete -c onedrive -l upload-only -d "Only upload local changes" complete -c onedrive -l verbose -d "Increase verbosity" complete -c onedrive -l version -d "Show version" complete -c onedrive -l with-editing-perms -d "Create read-write shared link" ================================================ FILE: contrib/completions/complete.zsh ================================================ #compdef onedrive # # ZSH completion code for OneDrive Linux Client # (c) 2019 Norbert Preining # License: GPLv3+ (as with the rest of the OneDrive Linux client project) local -a all_opts all_opts=( '--auth-files[Perform authentication via file exchange]:auth files:' '--auth-response[Perform authentication via response URL]:auth response:' '--check-for-nomount[Check for the presence of .nosync in the syncdir root. If found, do not perform sync.]' '--check-for-nosync[Check for the presence of .nosync in each directory. If found, skip directory from sync.]' '--classify-as-big-delete[Number of children removed to trigger big delete logic]:threshold:' '--cleanup-local-files[Remove local files when using --download-only]' '--confdir[Set the directory used to store the configuration files]:config directory:_files -/' '--create-directory[Create a directory on OneDrive - no sync will be performed.]:directory name:' '--create-share-link[Create a shareable link for a file]:file name:' '--debug-https[Debug OneDrive HTTPS communication.]' '--destination-directory[Destination directory for renamed or move on OneDrive - no sync will be performed.]:directory name:' '--disable-download-validation[Disable download validation when downloading from OneDrive]' '--disable-notifications[Do not use desktop notifications in monitor mode.]' '--disable-upload-validation[Disable upload validation when uploading to OneDrive]' '--display-config[Display what options the client will use as currently configured - no sync will be performed.]' '--display-quota[Display the quota status of the client - no sync will be performed.]' '--display-running-config[Display options configured on application startup.]' '--display-sync-status[Display the sync status of the client - no sync will be performed.]' '--download-file[Download a single file from Microsoft OneDrive]:file name:' '--download-only[Only download remote changes]' '--dry-run[Perform a trial sync with no changes made]' '--enable-logging[Enable client activity to a separate log file]' '--file-fragment-size[Specify the file fragment size for large file uploads (in MB)]:MB:' '--force[Force the deletion of data when a '\''big delete'\'' is detected]' '--force-http-11[Force the use of HTTP 1.1 for all operations]' '--force-sync[Force a synchronization of a specific folder]' '--get-O365-drive-id[Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library]:site URL:' '--get-file-link[Display the file link of a synced file]:file name:' '--get-sharepoint-drive-id[Query and return the SharePoint Drive ID]:site URL:' '--help[Show this help screen]' '--list-shared-items[List OneDrive Business Shared Items]' '--local-first[Synchronize from the local directory source first, before downloading changes from OneDrive.]' '--log-dir[Directory where logging output is saved]:log directory:_files -/' '--logout[Logout the current user]' '--modified-by[Display the last modified-by details]:file or directory:' '--monitor[Keep monitoring for local and remote changes]' '--monitor-fullscan-frequency[Sync runs before full local scan]:N:' '--monitor-interval[Seconds between syncs when idle in monitor mode]:seconds:' '--monitor-log-frequency[Frequency of logging in monitor mode]:seconds:' '--no-remote-delete[Do not delete remote files when using --upload-only]' '--print-access-token[Print the access token, useful for debugging]' '--reauth[Reauthenticate the client with OneDrive]' '--remove-directory[Remove a directory on OneDrive - no sync will be performed.]:directory name:' '--remove-source-files[Remove source file after upload when using --upload-only]' '--remove-source-folders[Remove the local directory structure post successful file transfer when using --upload-only --remove-source-files]' '--resync[Forget the last saved state, perform a full sync]' '--resync-auth[Approve the use of performing a --resync action]' '--share-password[Password to protect share link]:password:' '--single-directory[Sync a single local directory within the OneDrive root]:source directory:_files -/' '--skip-dir[Skip any directories matching this pattern]:pattern:' '--skip-dir-strict-match[Strict matching for --skip-dir]' '--skip-dot-files[Skip dot files and folders from syncing]' '--skip-file[Skip any files matching this pattern]:pattern:' '--skip-size[Skip new files larger than this size (in MB)]:MB:' '--skip-symlinks[Skip syncing of symlinks]' '--source-directory[Source directory to rename or move on OneDrive]:source directory:' '--space-reservation[Disk space (MB) to reserve]:MB:' '--sync[Perform a synchronisation with Microsoft OneDrive]' '--sync-root-files[Sync all files in sync_dir root when using sync_list.]' '--sync-shared-files[Sync OneDrive Business Shared Files to the local filesystem]' '--syncdir[Specify the local directory used for synchronisation to OneDrive]:sync directory:_files -/' '--synchronize[Perform a synchronisation (deprecated)]' '--threads[Number of threads to use for multi-threaded transfers]:N:' '--upload-only[Only upload to OneDrive, do not sync changes from OneDrive locally]' '--verbose[Print more details, useful for debugging (repeat for extra debugging)]' '--version[Print the version and exit]' '--with-editing-perms[Create a read-write shareable link for a file]' ) _arguments -S "$all_opts[@]" && return 0 ================================================ FILE: contrib/docker/Dockerfile ================================================ # -*-Dockerfile-*- ARG FEDORA_VERSION=43 ARG DEBIAN_VERSION=bullseye ARG GO_VERSION=1.23 ARG GOSU_VERSION=1.17 FROM golang:${GO_VERSION}-${DEBIAN_VERSION} AS builder-gosu ARG GOSU_VERSION RUN go install -ldflags "-s -w" github.com/tianon/gosu@${GOSU_VERSION} FROM fedora:${FEDORA_VERSION} AS builder-onedrive RUN dnf install -y ldc pkgconf libcurl-devel sqlite-devel dbus-devel git awk ENV PKG_CONFIG=/usr/bin/pkgconf COPY . /usr/src/onedrive WORKDIR /usr/src/onedrive RUN ./configure --enable-debug\ && make clean \ && make \ && make install FROM fedora:${FEDORA_VERSION} RUN dnf clean all \ && dnf -y update RUN dnf install -y libcurl sqlite ldc-libs dbus-libs \ && dnf clean all \ && mkdir -p /onedrive/conf /onedrive/data COPY --from=builder-gosu /go/bin/gosu /usr/local/bin/ COPY --from=builder-onedrive /usr/local/bin/onedrive /usr/local/bin/ COPY contrib/docker/entrypoint.sh / RUN chmod +x /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] ================================================ FILE: contrib/docker/Dockerfile-alpine ================================================ # -*-Dockerfile-*- ARG ALPINE_VERSION=3.23 ARG GO_VERSION=1.25 ARG GOSU_VERSION=1.17 FROM golang:${GO_VERSION}-alpine${ALPINE_VERSION} AS builder-gosu ARG GOSU_VERSION RUN go install -ldflags "-s -w" github.com/tianon/gosu@${GOSU_VERSION} FROM alpine:${ALPINE_VERSION} AS builder-onedrive RUN apk add --update --no-cache alpine-sdk gnupg xz curl-dev sqlite-dev dbus-dev binutils-gold autoconf automake ldc COPY . /usr/src/onedrive WORKDIR /usr/src/onedrive RUN autoreconf -fiv \ && ./configure --enable-debug\ && make clean \ && make \ && make install FROM alpine:${ALPINE_VERSION} RUN apk add --upgrade apk-tools \ && apk upgrade --available RUN apk add --update --no-cache bash libcurl libgcc shadow sqlite-libs ldc-runtime dbus-libs \ && mkdir -p /onedrive/conf /onedrive/data COPY --from=builder-gosu /go/bin/gosu /usr/local/bin/ COPY --from=builder-onedrive /usr/local/bin/onedrive /usr/local/bin/ COPY contrib/docker/entrypoint.sh / RUN chmod +x /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] ================================================ FILE: contrib/docker/Dockerfile-debian ================================================ # -*-Dockerfile-*- ARG DEBIAN_VERSION=trixie FROM debian:${DEBIAN_VERSION} AS builder-onedrive ARG DEBIAN_VERSION # Add backports repository and update before initial DEBIAN_FRONTEND installation RUN apt-get clean \ && echo "deb http://deb.debian.org/debian ${DEBIAN_VERSION}-backports main" > /etc/apt/sources.list.d/debian-${DEBIAN_VERSION}-backports.list \ && apt-get update \ && apt-get upgrade -y \ && apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends build-essential curl ca-certificates libcurl4-openssl-dev libsqlite3-dev libxml2-dev libdbus-1-dev pkg-config git ldc \ # Install|update curl from backports && apt-get install -t ${DEBIAN_VERSION}-backports -y curl \ && rm -rf /var/lib/apt/lists/* COPY . /usr/src/onedrive WORKDIR /usr/src/onedrive RUN ./configure --enable-debug\ && make clean \ && make \ && make install FROM debian:${DEBIAN_VERSION}-slim ARG DEBIAN_VERSION # Add backports repository and update after DEBIAN_FRONTEND installation RUN apt-get clean \ && echo "deb http://deb.debian.org/debian ${DEBIAN_VERSION}-backports main" > /etc/apt/sources.list.d/debian-${DEBIAN_VERSION}-backports.list \ && apt-get update \ && apt-get upgrade -y \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends libsqlite3-0 ca-certificates libphobos2-ldc-shared110 libdbus-1-3 \ # Install|update curl and libcurl4t64 from backports to get the latest version && apt-get install -t ${DEBIAN_VERSION}-backports -y curl libcurl4t64 \ && rm -rf /var/lib/apt/lists/* \ # Fix bug with ssl on armhf: https://serverfault.com/a/1045189 && /usr/bin/c_rehash \ && mkdir -p /onedrive/conf /onedrive/data # Install gosu v1.17 from trusted upstream source (built against Go 1.18.2) RUN set -eux; \ arch="$(dpkg --print-architecture)"; \ curl -fsSL -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.17/gosu-${arch}"; \ chmod +x /usr/local/bin/gosu; \ gosu nobody true COPY --from=builder-onedrive /usr/local/bin/onedrive /usr/local/bin/ COPY contrib/docker/entrypoint.sh / RUN chmod +x /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] ================================================ FILE: contrib/docker/entrypoint.sh ================================================ #!/bin/bash -eu set +H -euo pipefail # ---------------------------------------------------------------------- # Determine how the container is being started: # - If started as non-root (e.g. --user 1000:1000), we must NOT attempt # user/group management or chown, as those require root. # - If started as root, we can create/align the user and switch via gosu. # ---------------------------------------------------------------------- CONTAINER_UID="$(id -u)" CONTAINER_GID="$(id -g)" # Default ONEDRIVE_UID/GID: # - When running as non-root: default to the current UID/GID (the values Docker/Podman set) # - When running as root: keep existing behaviour (infer from /onedrive/data unless explicitly provided) if [ "${CONTAINER_UID}" -ne 0 ]; then : "${ONEDRIVE_UID:=${CONTAINER_UID}}" : "${ONEDRIVE_GID:=${CONTAINER_GID}}" else : "${ONEDRIVE_UID:=$(stat /onedrive/data -c '%u')}" : "${ONEDRIVE_GID:=$(stat /onedrive/data -c '%g')}" fi # ---------------------------------------------------------------------- # Root privilege handling # ---------------------------------------------------------------------- if [ "${CONTAINER_UID}" -eq 0 ]; then # Containers should not run the onedrive client as root by default. if [ "${ONEDRIVE_RUNAS_ROOT:=0}" == "1" ]; then echo "# Running container as root due to environment variable override" oduser='root' odgroup='root' else # Root container start is fine, but we will drop privileges to a non-root user. echo "# Container started as root; will drop privileges to UID:GID ${ONEDRIVE_UID}:${ONEDRIVE_GID}" fi # If we are not forcing root runtime, ensure a non-root user exists for ONEDRIVE_UID/GID if [ "${ONEDRIVE_RUNAS_ROOT:=0}" != "1" ]; then # Create / select group for target GID if ! odgroup="$(getent group "${ONEDRIVE_GID}")"; then odgroup='onedrive' groupadd "${odgroup}" -g "${ONEDRIVE_GID}" else odgroup="${odgroup%%:*}" fi # Create / select user for target UID if ! oduser="$(getent passwd "${ONEDRIVE_UID}")"; then oduser='onedrive' useradd -m "${oduser}" -u "${ONEDRIVE_UID}" -g "${ONEDRIVE_GID}" else oduser="${oduser%%:*}" usermod -g "${odgroup}" "${oduser}" fi echo "# Running container as user: ${oduser} (UID:GID ${ONEDRIVE_UID}:${ONEDRIVE_GID})" fi else # Non-root start (e.g. --user). Do not attempt account management or chown. if [ "${ONEDRIVE_RUNAS_ROOT:=0}" == "1" ]; then echo "# NOTE: ONEDRIVE_RUNAS_ROOT=1 requested, but container is not running as root; ignoring." fi echo "# Container started as non-root UID:GID ${CONTAINER_UID}:${CONTAINER_GID}" echo "# Using ONEDRIVE_UID:GID ${ONEDRIVE_UID}:${ONEDRIVE_GID} (no user/group creation performed)" fi # ---------------------------------------------------------------------- # Default parameters # ---------------------------------------------------------------------- ARGS=(--confdir /onedrive/conf --syncdir /onedrive/data) echo "# Base Args: ${ARGS[@]}" # Tell client to use Standalone Mode, based on an environment variable. Otherwise Monitor Mode is used. if [ "${ONEDRIVE_SYNC_ONCE:=0}" == "1" ]; then echo "# We run in Standalone Mode" echo "# Adding --sync" ARGS=(--sync ${ARGS[@]}) else echo "# We run in Monitor Mode" echo "# Adding --monitor" ARGS=(--monitor ${ARGS[@]}) fi # Make Verbose output optional, based on an environment variable if [ "${ONEDRIVE_VERBOSE:=0}" == "1" ]; then echo "# We are being verbose" echo "# Adding --verbose" ARGS=(--verbose ${ARGS[@]}) fi # Tell client to perform debug output, based on an environment variable if [ "${ONEDRIVE_DEBUG:=0}" == "1" ]; then echo "# We are performing debug output" echo "# Adding --verbose --verbose" ARGS=(--verbose --verbose ${ARGS[@]}) fi # Tell client to perform HTTPS debug output, based on an environment variable if [ "${ONEDRIVE_DEBUG_HTTPS:=0}" == "1" ]; then echo "# We are performing HTTPS debug output" echo "# Adding --debug-https" ARGS=(--debug-https ${ARGS[@]}) fi # Tell client to perform a resync based on environment variable if [ "${ONEDRIVE_RESYNC:=0}" == "1" ]; then echo "# We are performing a --resync" echo "# Adding --resync --resync-auth" ARGS=(--resync --resync-auth ${ARGS[@]}) fi # Tell client to sync in download-only mode based on environment variable if [ "${ONEDRIVE_DOWNLOADONLY:=0}" == "1" ]; then echo "# We are synchronising in download-only mode" echo "# Adding --download-only" ARGS=(--download-only ${ARGS[@]}) fi # Tell client to clean up local files when in download-only mode based on environment variable if [ "${ONEDRIVE_CLEANUPLOCAL:=0}" == "1" ]; then echo "# We are cleaning up local files that are not present online" echo "# Adding --cleanup-local-files" ARGS=(--cleanup-local-files ${ARGS[@]}) fi # Tell client to sync in upload-only mode based on environment variable if [ "${ONEDRIVE_UPLOADONLY:=0}" == "1" ]; then echo "# We are synchronising in upload-only mode" echo "# Adding --upload-only" ARGS=(--upload-only ${ARGS[@]}) fi # Tell client to sync in no-remote-delete mode based on environment variable if [ "${ONEDRIVE_NOREMOTEDELETE:=0}" == "1" ]; then echo "# We are synchronising in no-remote-delete mode" echo "# Adding --no-remote-delete" ARGS=(--no-remote-delete ${ARGS[@]}) fi # Tell client to logout based on environment variable if [ "${ONEDRIVE_LOGOUT:=0}" == "1" ]; then echo "# We are logging out" echo "# Adding --logout" ARGS=(--logout ${ARGS[@]}) fi # Tell client to re-authenticate based on environment variable if [ "${ONEDRIVE_REAUTH:=0}" == "1" ]; then echo "# We are logging out to perform a reauthentication" echo "# Adding --reauth" ARGS=(--reauth ${ARGS[@]}) fi # Tell client to utilise auth files at the provided locations based on environment variable if [ -n "${ONEDRIVE_AUTHFILES:=""}" ]; then echo "# We are using auth files to perform authentication" echo "# Adding --auth-files ARG" ARGS=(--auth-files ${ONEDRIVE_AUTHFILES} ${ARGS[@]}) fi # Tell client to utilise provided auth response based on environment variable if [ -n "${ONEDRIVE_AUTHRESPONSE:=""}" ]; then echo "# We are providing the auth response directly to perform authentication" echo "# Adding --auth-response ARG" ARGS=(--auth-response \"${ONEDRIVE_AUTHRESPONSE}\" ${ARGS[@]}) fi # Tell client to print the running configuration at application startup if [ "${ONEDRIVE_DISPLAY_CONFIG:=0}" == "1" ]; then echo "# We are printing the application running configuration at application startup" echo "# Adding --display-running-config" ARGS=(--display-running-config ${ARGS[@]}) fi # Tell client to use sync single dir option if [ -n "${ONEDRIVE_SINGLE_DIRECTORY:=""}" ]; then echo "# We are synchronising in single-directory mode" echo "# Adding --single-directory ARG" ARGS=(--single-directory \"${ONEDRIVE_SINGLE_DIRECTORY}\" ${ARGS[@]}) fi # Tell client run in dry-run mode if [ "${ONEDRIVE_DRYRUN:=0}" == "1" ]; then echo "# We are running in dry-run mode" echo "# Adding --dry-run" ARGS=(--dry-run ${ARGS[@]}) fi # Tell client to disable download validation if [ "${ONEDRIVE_DISABLE_DOWNLOAD_VALIDATION:=0}" == "1" ]; then echo "# We are disabling the download integrity checks performed by this client" echo "# Adding --disable-download-validation" ARGS=(--disable-download-validation ${ARGS[@]}) fi # Tell client to disable upload validation if [ "${ONEDRIVE_DISABLE_UPLOAD_VALIDATION:=0}" == "1" ]; then echo "# We are disabling the upload integrity checks performed by this client" echo "# Adding --disable-upload-validation" ARGS=(--disable-upload-validation ${ARGS[@]}) fi # Tell client to download OneDrive Business Shared Files if 'sync_business_shared_items' option has been enabled in the configuration files if [ "${ONEDRIVE_SYNC_SHARED_FILES:=0}" == "1" ]; then echo "# We are attempting to sync OneDrive Business Shared Files if 'sync_business_shared_items' has been enabled in the config file" echo "# Adding --sync-shared-files" ARGS=(--sync-shared-files ${ARGS[@]}) fi # Tell client to use a different value for file fragment size for large file uploads if [ -n "${ONEDRIVE_FILE_FRAGMENT_SIZE:=""}" ]; then echo "# We are specifying the file fragment size for large file uploads (in MB)" echo "# Adding --file-fragment-size ARG" ARGS=(--file-fragment-size ${ONEDRIVE_FILE_FRAGMENT_SIZE} ${ARGS[@]}) fi # Tell client to use a specific threads value for parallel operations if [ -n "${ONEDRIVE_THREADS:=""}" ]; then echo "# We are specifying a thread value for the number of worker threads used for parallel upload and download operations" echo "# Adding --threads ARG" ARGS=(--threads ${ONEDRIVE_THREADS} ${ARGS[@]}) fi # Allow override of args if command-line parameters are provided if [ ${#} -gt 0 ]; then ARGS=("${@}") fi # ---------------------------------------------------------------------- # Launch # ---------------------------------------------------------------------- # If started non-root, just run directly (no gosu, no chown). if [ "${CONTAINER_UID}" -ne 0 ]; then echo "# Launching 'onedrive' as UID:GID ${CONTAINER_UID}:${CONTAINER_GID}" exec /usr/local/bin/onedrive "${ARGS[@]}" fi # Started as root: # - If ONEDRIVE_RUNAS_ROOT=1: run directly as root. # - Otherwise: chown writable dirs and drop to oduser via gosu. if [ "${ONEDRIVE_RUNAS_ROOT:=0}" == "1" ]; then echo "# Launching 'onedrive' as root" exec /usr/local/bin/onedrive "${ARGS[@]}" else echo "# Changing ownership permissions on /onedrive/data and /onedrive/conf to ${oduser}:${odgroup}" chown "${oduser}:${odgroup}" /onedrive/data /onedrive/conf echo "# Launching 'onedrive' as ${oduser} via gosu" exec gosu "${oduser}" /usr/local/bin/onedrive "${ARGS[@]}" fi ================================================ FILE: contrib/docker/hooks/post_push ================================================ #!/bin/bash BUILD_DATE=`date "+%Y%m%d%H%M"` docker tag ${IMAGE_NAME} "${IMAGE_NAME}-${BUILD_DATE}" docker push "${IMAGE_NAME}-${BUILD_DATE}" ================================================ FILE: contrib/init.d/onedrive.init ================================================ #!/bin/sh # # chkconfig: 2345 20 80 # description: Starts and stops OneDrive Client for Linux # # Source function library. if [ -f /etc/init.d/functions ] ; then . /etc/init.d/functions elif [ -f /etc/rc.d/init.d/functions ] ; then . /etc/rc.d/init.d/functions else exit 1 fi # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ ${NETWORKING} = "no" ] && exit 1 APP_NAME="OneDrive Client for Linux" STOP_TIMEOUT=${STOP_TIMEOUT-5} RETVAL=0 start() { export PATH=/usr/local/bin/:$PATH echo -n "Starting $APP_NAME: " daemon --user root onedrive_service.sh RETVAL=$? echo [ $RETVAL -eq 0 ] && touch /var/lock/subsys/onedrive || \ RETVAL=1 return $RETVAL } stop() { echo -n "Shutting down $APP_NAME: " killproc onedrive RETVAL=$? echo [ $RETVAL = 0 ] && rm -f /var/lock/subsys/onedrive ${pidfile} } restart() { stop start } rhstatus() { status onedrive return $? } # Allow status as non-root. if [ "$1" = status ]; then rhstatus exit $? fi case "$1" in start) start ;; stop) stop ;; restart) restart ;; reload) reload ;; status) rhstatus ;; *) echo "Usage: $0 {start|stop|restart|reload|status}" exit 2 esac exit $? ================================================ FILE: contrib/init.d/onedrive_service.sh ================================================ #!/bin/bash # This script is to assist in starting the onedrive client when using init.d APP_OPTIONS="--monitor --verbose --enable-logging" onedrive "$APP_OPTIONS" > /dev/null 2>&1 & exit 0 ================================================ FILE: contrib/logrotate/onedrive.logrotate ================================================ # Any OneDrive Client logs configured for here /var/log/onedrive/*log { # What user / group should logrotate use? # Logrotate 3.8.9 or greater required otherwise: # "unknown option 'su' -- ignoring line" is generated su root users # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file dateext # compress the log files compress # missing files OK missingok } ================================================ FILE: contrib/pacman/PKGBUILD.in ================================================ pkgname=onedrive pkgver=@PACKAGE_VERSION@ pkgrel=1 # Patch-level (increment this when a patch is applied) pkgdesc="OneDrive Client for Linux" license=("GPL3") url="https://github.com/abraunegg/onedrive/" arch=("i686" "x86_64") depends=("curl" "gcc-libs" "glibc" "sqlite") makedepends=("dmd" "git" "tar" "make") source=("https://github.com/abraunegg/onedrive/archive/v$pkgver.tar.gz") sha256sums=('SKIP') # Use SKIP or actual checksum prepare() { cd "$srcdir" tar -xzf "$pkgname-$pkgver.tar.gz" --one-top-level="$pkgname-$pkgver" --strip-components 1 } build() { cd "$srcdir/$pkgname-$pkgver" git init git add . git commit --allow-empty-message -m "" git tag "v$pkgver" make PREFIX=/usr onedrive } package() { cd "$srcdir/$pkgname-$pkgver" make PREFIX=/usr DESTDIR="$pkgdir" install } ================================================ FILE: contrib/spec/onedrive.spec.in ================================================ # Platform-specific default compiler selection %if 0%{?fedora} || 0%{?rhel} || 0%{?centos} %global default_dcompiler ldc %else %global default_dcompiler dmd %endif # Allow manual override: rpmbuild --define 'dcompiler dmd' %{!?dcompiler: %global dcompiler %{default_dcompiler}} # Compiler version constraints %global dmd_minver 2.091.1 %global ldc_minver 1.20.1 # Conditional BuildRequires %if "%{dcompiler}" == "dmd" BuildRequires: dmd >= %{dmd_minver} %else %if "%{dcompiler}" == "ldc" BuildRequires: ldc >= %{ldc_minver} %else %error Unsupported D compiler selected: %{dcompiler} %endif %endif # Systemd logic %if 0%{?fedora} || 0%{?rhel} >= 7 %global with_systemd 1 %else %global with_systemd 0 %endif %if 0%{?rhel} >= 7 %global rhel_unitdir 1 %else %global rhel_unitdir 0 %endif Name: onedrive Version: 2.5.10 Release: 1%{?dist} Summary: OneDrive Client for Linux Group: System Environment/Network License: GPLv3 URL: https://github.com/abraunegg/onedrive Source0: v%{version}.tar.gz BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildRequires: sqlite-devel >= 3.7.15 BuildRequires: libcurl-devel BuildRequires: dbus-devel Requires: sqlite >= 3.7.15 Requires: libcurl Requires: dbus %if 0%{?with_systemd} Requires(post): systemd Requires(preun): systemd Requires(postun): systemd %else Requires(post): chkconfig Requires(preun): chkconfig Requires(preun): initscripts Requires(postun): initscripts %endif %define debug_package %{nil} %description Free client for Microsoft OneDrive on Linux. Supports personal, business, SharePoint, and shared folders. Built-in client-side filtering, delta sync, webhook support, and more. %prep %setup -q %build %configure --enable-debug --enable-notifications make %install %make_install PREFIX="%{buildroot}" %if 0%{?with_systemd} %if 0%{?rhel_unitdir} # RHEL/CentOS: system unit only install -D -m 0644 contrib/systemd/onedrive.service %{buildroot}%{_unitdir}/onedrive.service install -D -m 0644 contrib/systemd/onedrive@.service %{buildroot}%{_unitdir}/onedrive@.service %else # Fedora: install both system and user units install -D -m 0644 contrib/systemd/onedrive@.service %{buildroot}%{_unitdir}/onedrive@.service install -D -m 0644 contrib/systemd/onedrive.service %{buildroot}%{_userunitdir}/onedrive.service %endif %endif %clean %files %doc readme.md LICENSE changelog.md docs/*.md config %config %{_sysconfdir}/logrotate.d/onedrive %{_mandir}/man1/%{name}.1.gz %{_bindir}/%{name} %if 0%{?with_systemd} %if 0%{?rhel_unitdir} %{_unitdir}/%{name}.service %{_unitdir}/%{name}@.service %else %{_unitdir}/%{name}@.service %{_userunitdir}/%{name}.service %endif %else %{_bindir}/onedrive_service.sh /etc/init.d/onedrive %endif %changelog * Fri Jan 20 2026 - 2.5.10-1 - Release v2.5.10 with new features, bug fixes, and enhancements * Thu Nov 06 2025 - 2.5.9-1 - Release v2.5.9 with new features, bug fixes, and enhancements * Wed Nov 05 2025 - 2.5.8-1 - Release v2.5.8 with new features, bug fixes, and enhancements * Tue Sep 23 2025 - 2.5.7-1 - Release v2.5.7 with new features, bug fixes, and enhancements * Thu Jun 05 2025 - 2.5.6-1 - Release v2.5.6 with new features, bug fixes, and enhancements * Mon Mar 17 2025 - 2.5.5-1 - Release v2.5.5 with new features, bug fixes, and enhancements * Mon Feb 03 2025 - 2.5.4-1 - Release v2.5.4 with new features, bug fixes, and enhancements * Sat Nov 16 2024 - 2.5.3-1 - Release v2.5.3 with new features, bug fixes, and enhancements * Sun Sep 29 2024 - 2.5.2-1 - Release v2.5.2 with new features, bug fixes, and enhancements * Fri Sep 27 2024 - 2.5.1-1 - Release v2.5.1 with new features, bug fixes, and enhancements * Mon Sep 16 2024 - 2.5.0-1 - Release v2.5.0 with new features, bug fixes, and enhancements * Wed Jun 21 2023 - 2.4.25-1 - Release v2.4.25 with new features, bug fixes, and enhancements * Tue Jun 20 2023 - 2.4.24-1 - Release v2.4.24 with new features, bug fixes, and enhancements * Fri Jan 06 2023 - 2.4.23-1 - Release v2.4.23 with new features, bug fixes, and enhancements * Tue Dec 06 2022 - 2.4.22-1 - Release v2.4.22 with new features, bug fixes, and enhancements * Tue Sep 27 2022 - 2.4.21-1 - Release v2.4.21 with new features, bug fixes, and enhancements * Wed Jul 20 2022 - 2.4.20-1 - Release v2.4.20 with new features, bug fixes, and enhancements * Wed Jun 15 2022 - 2.4.19-1 - Release v2.4.19 with new features, bug fixes, and enhancements * Thu Jun 02 2022 - 2.4.18-1 - Release v2.4.18 with new features, bug fixes, and enhancements * Sat Apr 30 2022 - 2.4.17-1 - Release v2.4.17 with new features, bug fixes, and enhancements * Thu Mar 10 2022 - 2.4.16-1 - Release v2.4.16 with new features, bug fixes, and enhancements * Fri Dec 31 2021 - 2.4.15-1 - Release v2.4.15 with new features, bug fixes, and enhancements * Wed Nov 24 2021 - 2.4.14-1 - Release v2.4.14 with new features, bug fixes, and enhancements * Sun Dec 27 2020 - 2.4.9-1 - Release v2.4.9 with new features, bug fixes, and enhancements * Mon Nov 30 2020 - 2.4.8-1 - Release v2.4.8 with new features, bug fixes, and enhancements * Mon Nov 09 2020 - 2.4.7-1 - Release v2.4.7 with new features, bug fixes, and enhancements * Sun Oct 04 2020 - 2.4.6-1 - Release v2.4.6 with new features, bug fixes, and enhancements * Thu Aug 13 2020 - 2.4.5-1 - Release v2.4.5 with new features, bug fixes, and enhancements * Tue Aug 11 2020 - 2.4.4-1 - Release v2.4.4 with new features, bug fixes, and enhancements * Mon Jun 29 2020 - 2.4.3-1 - Release v2.4.3 with new features, bug fixes, and enhancements * Wed May 27 2020 - 2.4.2-1 - Release v2.4.2 with new features, bug fixes, and enhancements * Sat May 02 2020 - 2.4.1-1 - Release v2.4.1 with new features, bug fixes, and enhancements * Sun Mar 22 2020 - 2.4.0-1 - Release v2.4.0 with new features, bug fixes, and enhancements * Tue Dec 31 2019 - 2.3.13-1 - Release v2.3.13 with new features, bug fixes, and enhancements * Wed Dec 04 2019 - 2.3.12-1 - Release v2.3.12 with new features, bug fixes, and enhancements * Tue Nov 05 2019 - 2.3.11-1 - Release v2.3.11 with new features, bug fixes, and enhancements * Tue Oct 01 2019 - 2.3.10-1 - Release v2.3.10 with new features, bug fixes, and enhancements * Sun Sep 01 2019 - 2.3.9-1 - Release v2.3.9 with new features, bug fixes, and enhancements * Sun Aug 04 2019 - 2.3.8-1 - Release v2.3.8 with new features, bug fixes, and enhancements * Wed Jul 03 2019 - 2.3.7-1 - Release v2.3.7 with new features, bug fixes, and enhancements * Wed Jul 03 2019 - 2.3.6-1 - Release v2.3.6 with new features, bug fixes, and enhancements * Wed Jun 19 2019 - 2.3.5-1 - Release v2.3.5 with new features, bug fixes, and enhancements * Thu Jun 13 2019 - 2.3.4-1 - Release v2.3.4 with new features, bug fixes, and enhancements * Tue Apr 16 2019 - 2.3.3-1 - Release v2.3.3 with new features, bug fixes, and enhancements * Tue Apr 02 2019 - 2.3.2-1 - Release v2.3.2 with new features, bug fixes, and enhancements * Tue Mar 26 2019 - 2.3.1-1 - Release v2.3.1 with new features, bug fixes, and enhancements * Mon Mar 25 2019 - 2.3.0-1 - Release v2.3.0 with new features, bug fixes, and enhancements * Tue Mar 12 2019 - 2.2.6-1 - Release v2.2.6 with new features, bug fixes, and enhancements * Wed Jan 16 2019 - 2.2.5-1 - Release v2.2.5 with new features, bug fixes, and enhancements * Fri Dec 28 2018 - 2.2.4-1 - Release v2.2.4 with new features, bug fixes, and enhancements * Thu Dec 20 2018 - 2.2.3-1 - Release v2.2.3 with new features, bug fixes, and enhancements * Thu Dec 20 2018 - 2.2.2-1 - Release v2.2.2 with new features, bug fixes, and enhancements * Tue Dec 04 2018 - 2.2.1-1 - Release v2.2.1 with new features, bug fixes, and enhancements * Sat Nov 24 2018 - 2.2.0-1 - Release v2.2.0 with new features, bug fixes, and enhancements * Thu Nov 15 2018 - 2.1.6-1 - Release v2.1.6 with new features, bug fixes, and enhancements * Sun Nov 11 2018 - 2.1.5-1 - Release v2.1.5 with new features, bug fixes, and enhancements * Wed Oct 10 2018 - 2.1.4-1 - Release v2.1.4 with new features, bug fixes, and enhancements * Thu Oct 04 2018 - 2.1.3-1 - Release v2.1.3 with new features, bug fixes, and enhancements * Mon Aug 27 2018 - 2.1.2-1 - Release v2.1.2 with new features, bug fixes, and enhancements * Tue Aug 14 2018 - 2.1.1-1 - Release v2.1.1 with new features, bug fixes, and enhancements * Fri Aug 10 2018 - 2.1.0-1 - Release v2.1.0 with new features, bug fixes, and enhancements * Wed Jul 18 2018 - 2.0.2-1 - Release v2.0.2 with new features, bug fixes, and enhancements * Wed Jul 11 2018 - 2.0.1-1 - Release v2.0.1 with new features, bug fixes, and enhancements * Tue Jul 10 2018 - 2.0.0-1 - Release v2.0.0 with new features, bug fixes, and enhancements * Thu May 17 2018 - 1.1.2-1 - Release v1.1.2 with new features, bug fixes, and enhancements * Sat Jan 20 2018 - 1.1.1-1 - Release v1.1.1 with new features, bug fixes, and enhancements * Fri Jan 19 2018 - 1.1.0-1 - Release v1.1.0 with new features, bug fixes, and enhancements * Tue Aug 01 2017 - 1.0.1-1 - Release v1.0.1 with new features, bug fixes, and enhancements * Fri Jul 14 2017 - 1.0.0-1 - Release v1.0.0 with new features, bug fixes, and enhancements ================================================ FILE: contrib/systemd/onedrive.service.in ================================================ [Unit] Description=OneDrive Client for Linux Documentation=https://github.com/abraunegg/onedrive After=network-online.target Wants=network-online.target [Service] # Commented out hardenings are disabled because they may not work out of the box on your distribution # If you know what you are doing please try to enable them. ProtectSystem=full #PrivateUsers=true #PrivateDevices=true ProtectHostname=true #ProtectClock=true ProtectKernelTunables=true #ProtectKernelModules=true #ProtectKernelLogs=true ProtectControlGroups=true RestrictRealtime=true ExecStartPre=/bin/sh -c 'sleep 15' ExecStart=@prefix@/bin/onedrive --monitor Restart=on-failure RestartSec=3 # Do not restart the service if a --resync is required which is done via a 126 exit code RestartPreventExitStatus=126 # Time to wait for the service to stop gracefully before forcefully terminating it TimeoutStopSec=90 [Install] WantedBy=default.target ================================================ FILE: contrib/systemd/onedrive@.service.in ================================================ [Unit] Description=OneDrive Client for Linux running for %i Documentation=https://github.com/abraunegg/onedrive After=network-online.target Wants=network-online.target [Service] # Commented out hardenings are disabled because they may not work out of the box on your distribution # If you know what you are doing please try to enable them. ProtectSystem=full #PrivateDevices=true ProtectHostname=true #ProtectClock=true ProtectKernelTunables=true #ProtectKernelModules=true #ProtectKernelLogs=true ProtectControlGroups=true RestrictRealtime=true ExecStartPre=/bin/sh -c 'sleep 15' ExecStart=@prefix@/bin/onedrive --monitor --confdir=/home/%i/.config/onedrive User=%i Group=users Restart=on-failure RestartSec=3 # Do not restart the service if a --resync is required which is done via a 126 exit code RestartPreventExitStatus=126 # Time to wait for the service to stop gracefully before forcefully terminating it TimeoutStopSec=90 [Install] WantedBy=multi-user.target ================================================ FILE: docs/advanced-usage.md ================================================ # Advanced Configuration of the OneDrive Client for Linux This document covers the following scenarios: * [Configuring the client to use multiple OneDrive accounts / configurations](#configuring-the-client-to-use-multiple-onedrive-accounts--configurations) * [Configuring the client to use multiple OneDrive accounts / configurations using Docker](#configuring-the-client-to-use-multiple-onedrive-accounts--configurations-using-docker) * [Configuring the client for use in dual-boot (Windows / Linux) situations](#configuring-the-client-for-use-in-dual-boot-windows--linux-situations) * [Configuring the client for use when 'sync_dir' is a mounted directory](#configuring-the-client-for-use-when-sync_dir-is-a-mounted-directory) * [Upload data from the local ~/OneDrive folder to a specific location on OneDrive](#upload-data-from-the-local-onedrive-folder-to-a-specific-location-on-onedrive) ## Configuring the client to use multiple OneDrive accounts / configurations Essentially, each OneDrive account or SharePoint Shared Library which you require to be synced needs to have its own and unique configuration, local sync directory and service files. To do this, the following steps are needed: 1. Create a unique configuration folder for each onedrive client configuration that you need 2. Copy to this folder a copy of the default configuration file 3. Update the default configuration file as required, changing the required minimum config options and any additional options as needed to support your multi-account configuration 4. Authenticate the client using the new configuration directory 5. Test the configuration using '--display-config' and '--dry-run' 6. Sync the OneDrive account data as required using `--synchronize` or `--monitor` 7. Configure a unique systemd service file for this account configuration ### 1. Create a unique configuration folder for each onedrive client configuration that you need Make the configuration folder as required for this new configuration, for example: ```text mkdir ~/.config/my-new-config ``` ### 2. Copy to this folder a copy of the default configuration file Copy to this folder a copy of the default configuration file by downloading this file from GitHub and saving this file in the directory created above: ```text wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/my-new-config/config ``` ### 3. Update the default configuration file The following config options *must* be updated to ensure that individual account data is not cross populated with other OneDrive accounts or other configurations: * sync_dir Other options that may require to be updated, depending on the OneDrive account that is being configured: * drive_id * application_id * sync_business_shared_folders * skip_dir * skip_file * Creation of a 'sync_list' file if required * Creation of a 'business_shared_folders' file if required ### 4. Authenticate the client Authenticate the client using the specific configuration file: ```text onedrive --confdir="~/.config/my-new-config" ``` You will be asked to open a specific URL by using your web browser where you will have to login into your Microsoft Account and give the application the permission to access your files. After giving permission to the application, you will be redirected to a blank page. Copy the URI of the blank page into the application. ```text [user@hostname ~]$ onedrive --confdir="~/.config/my-new-config" Configuration file successfully loaded Configuring Global Azure AD Endpoints Authorize this app visiting: https://..... Enter the response uri: ``` ### 5. Display and Test the configuration Test the configuration using '--display-config' and '--dry-run'. By doing so, this allows you to test any configuration that you have currently made, enabling you to fix this configuration before using the configuration. #### Display the configuration ```text onedrive --confdir="~/.config/my-new-config" --display-config ``` #### Test the configuration by performing a dry-run ```text onedrive --confdir="~/.config/my-new-config" --synchronize --verbose --dry-run ``` If both of these operate as per your expectation, the configuration of this client setup is complete and validated. If not, amend your configuration as required. ### 6. Sync the OneDrive account data as required Sync the data for the new account configuration as required: ```text onedrive --confdir="~/.config/my-new-config" --synchronize --verbose ``` or ```text onedrive --confdir="~/.config/my-new-config" --monitor --verbose ``` * `--synchronize` does a one-time sync * `--monitor` keeps the application running and monitoring for changes both local and remote ### 7. Automatic syncing of new OneDrive configuration In order to automatically start syncing your OneDrive accounts, you will need to create a service file for each account. From the applicable 'systemd folder' where the applicable systemd service file exists: * RHEL / CentOS: `/usr/lib/systemd/system` * Others: `/usr/lib/systemd/user` and `/lib/systemd/system` ### Step1: Create a new systemd service file #### Red Hat Enterprise Linux, CentOS Linux Copy the required service file to a new name: ```text sudo cp /usr/lib/systemd/system/onedrive.service /usr/lib/systemd/system/onedrive-my-new-config ``` or ```text sudo cp /usr/lib/systemd/system/onedrive@.service /usr/lib/systemd/system/onedrive-my-new-config@.service ``` #### Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora Copy the required service file to a new name: ```text sudo cp /usr/lib/systemd/user/onedrive.service /usr/lib/systemd/user/onedrive-my-new-config.service ``` or ```text sudo cp /lib/systemd/system/onedrive@.service /lib/systemd/system/onedrive-my-new-config@.service ``` ### Step 2: Edit new systemd service file Edit the new systemd file, updating the line beginning with `ExecStart` so that the confdir mirrors the one you used above: ```text ExecStart=/usr/local/bin/onedrive --monitor --confdir="/full/path/to/config/dir" ``` Example: ```text ExecStart=/usr/local/bin/onedrive --monitor --confdir="/home/myusername/.config/my-new-config" ``` > [!IMPORTANT] > When running the client manually, `--confdir="~/.config/......` is acceptable. In a systemd configuration file, the full path must be used. The `~` must be manually expanded when editing your systemd file. ### Step 3: Enable the new systemd service Once the file is correctly edited, you can enable the new systemd service using the following commands. #### Red Hat Enterprise Linux, CentOS Linux ```text systemctl enable onedrive-my-new-config systemctl start onedrive-my-new-config ``` #### Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text systemctl --user enable onedrive-my-new-config systemctl --user start onedrive-my-new-config ``` or ```text systemctl --user enable onedrive-my-new-config@myusername.service systemctl --user start onedrive-my-new-config@myusername.service ``` ### Step 4: Viewing systemd status and logs for the custom service #### Viewing systemd service status - Red Hat Enterprise Linux, CentOS Linux ```text systemctl status onedrive-my-new-config ``` #### Viewing systemd service status - Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text systemctl --user status onedrive-my-new-config ``` #### Viewing journalctl systemd logs - Red Hat Enterprise Linux, CentOS Linux ```text journalctl --unit=onedrive-my-new-config -f ``` #### Viewing journalctl systemd logs - Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text journalctl --user --unit=onedrive-my-new-config -f ``` ### Step 5: (Optional) Run custom systemd service at boot without user login In some cases it may be desirable for the systemd service to start without having to login as your 'user' All the systemd steps above that utilise the `--user` option, will run the systemd service as your particular user. As such, the systemd service will not start unless you actually login to your system. To avoid this issue, you need to reconfigure your 'user' account so that the systemd services you have created will startup without you having to login to your system: ```text loginctl enable-linger ``` Example: ```text alex@ubuntu-headless:~$ loginctl enable-linger alex ``` Repeat these steps for each OneDrive new account that you wish to use. ## Configuring the client to use multiple OneDrive accounts / configurations using Docker In some situations it may be desirable to run multiple Docker containers at the same time, each with their own configuration. To run the Docker container successfully, it needs two unique Docker volumes to operate: * Your configuration Docker volumes * Your data Docker volume When running multiple Docker containers, this is no different - each Docker container must have it's own configuration and data volume. ### High level steps: 1. Create the required unique Docker volumes for the configuration volume 2. Create the required unique local path used for the Docker data volume 3. Start the multiple Docker containers with the required configuration for each container #### Create the required unique Docker volumes for the configuration volume Create the required unique Docker volumes for the configuration volume(s): ```text docker volume create onedrive_conf_sharepoint_site1 docker volume create onedrive_conf_sharepoint_site2 docker volume create onedrive_conf_sharepoint_site3 ... docker volume create onedrive_conf_sharepoint_site50 ``` #### Create the required unique local path used for the Docker data volume Create the required unique local path used for the Docker data volume ```text mkdir -p /use/full/local/path/no/tilde/SharePointSite1 mkdir -p /use/full/local/path/no/tilde/SharePointSite2 mkdir -p /use/full/local/path/no/tilde/SharePointSite3 ... mkdir -p /use/full/local/path/no/tilde/SharePointSite50 ``` #### Start the Docker container with the required configuration (example) ```text docker run -it --name onedrive -v onedrive_conf_sharepoint_site1:/onedrive/conf -v "/use/full/local/path/no/tilde/SharePointSite1:/onedrive/data" driveone/onedrive:latest docker run -it --name onedrive -v onedrive_conf_sharepoint_site2:/onedrive/conf -v "/use/full/local/path/no/tilde/SharePointSite2:/onedrive/data" driveone/onedrive:latest docker run -it --name onedrive -v onedrive_conf_sharepoint_site3:/onedrive/conf -v "/use/full/local/path/no/tilde/SharePointSite3:/onedrive/data" driveone/onedrive:latest ... docker run -it --name onedrive -v onedrive_conf_sharepoint_site50:/onedrive/conf -v "/use/full/local/path/no/tilde/SharePointSite50:/onedrive/data" driveone/onedrive:latest ``` > [!TIP] > To avoid 're-authenticating' and 'authorising' each individual Docker container, if all the Docker containers are using the 'same' OneDrive credentials, you can reuse the 'refresh_token' from one Docker container to another by copying this file to the configuration Docker volume of each Docker container. > > If the account credentials are different .. you will need to re-authenticate each Docker container individually. ## Configuring the client for use in dual-boot (Windows / Linux) situations When dual booting Windows and Linux, depending on the Windows OneDrive account configuration, the 'Files On-Demand' option may be enabled when running OneDrive within your Windows environment. When this option is enabled in Windows, if you are sharing this location between your Windows and Linux systems, all files will be a 0 byte link, and cannot be used under Linux. To fix the problem of windows turning all files (that should be kept offline) into links, you have to uncheck a specific option in the onedrive settings window. The option in question is `Save space and download files as you use them`. To find this setting, open the onedrive pop-up window from the taskbar, click "Help & Settings" > "Settings". This opens a new window. Go to the tab "Settings" and look for the section "Files On-Demand". After unchecking the option and clicking "OK", the Windows OneDrive client should restart itself and start actually downloading your files so they will truly be available on your disk when offline. These files will then be fully accessible under Linux and the Linux OneDrive client. | OneDrive Personal | Onedrive Business
SharePoint | |---|---| | ![Uncheck-Personal](./images/personal-files-on-demand.png) | ![Uncheck-Business](./images/business-files-on-demand.png) | ### Accessing Windows OneDrive Files from Linux (Dual-Boot Setup) When dual-booting between Windows and Linux, accessing OneDrive-synced folders stored on an NTFS partition can be problematic. This is primarily due to Microsoft OneDrive's use of reparse points when the Files On-Demand feature is enabled in Windows. These reparse points can render files inaccessible from Linux, even after disabling Files On-Demand, because the reparse metadata may persist. #### Solution: Use the ntfs-3g-onedrive Plugin The ['ntfs-3g-onedrive'](https://github.com/gbrielgustavo/ntfs-3g-onedrive) plugin is designed to address this issue. It modifies the behavior of the ntfs-3g driver to correctly handle OneDrive's reparse points, allowing you to access your OneDrive files from Linux. > [!IMPORTANT] > The configuration and installation of the 'ntfs-3g-onedrive' driver update on your platform is beyond the scope of this documentation and repository. > > For assistance please seek support via the ['ntfs-3g'](https://github.com/tuxera/ntfs-3g) GitHub project. ## Configuring the client for use when 'sync_dir' is a mounted directory In some environments, your setup might be that your configured 'sync_dir' is pointing to another mounted file system - a NFS|CIFS location, an external drive (USB stick, eSATA etc). As such, you configure your 'sync_dir' as follows: ```text sync_dir = "/path/to/mountpoint/OneDrive" ``` The issue here is - how does the client react if the mount point gets removed - network loss, device removal? The client has zero knowledge of any event that causes a mountpoint to become unavailable, thus, the client (if you are running as a service) will assume that you deleted the files, thus, will go ahead and delete all your files on OneDrive. This is most certainly an undesirable action. There are a few options here which you can configure in your 'config' file to assist you to prevent this sort of item from occurring: 1. classify_as_big_delete 2. check_nomount 3. check_nosync > [!NOTE] > Before making any change to your configuration, stop any sync process & stop any onedrive systemd service from running. ### classify_as_big_delete By default, this uses a value of 1000 files|folders. An undesirable unmount if you have more than 1000 files, this default level will prevent the client from executing the online delete. Modify this value up or down as desired ### check_nomount & check_nosync When configuring the OneDrive client to use a directory on a mounted volume (e.g., external disk, USB device, network share), it is essential to guard against accidental sync deletion if the mount point becomes unavailable. If a mount is lost or not yet available at the time of sync, the 'sync_dir' may appear empty, leading the client to delete the corresponding online content. To safely prevent this, enable the following configuration options: ``` check_nomount = "true" check_nosync = "true" ``` These settings instruct the client to: * Check for the presence of a `.nosync` file in the 'sync_dir' before syncing * Halt syncing immediately if the file is detected, assuming the mount has failed or not available #### How the `.nosync` file works 1. The `.nosync` file is placed on the local filesystem, in the exact directory that will later be covered by the mounted volume. 2. Once the external device is mounted, that directory (and the `.nosync` file) becomes hidden by the mount. 3. If the mount disappears or fails, the `.nosync` file becomes visible again. 4. The OneDrive client detects this and stops syncing, preventing accidental deletions due to the mount being unavailable. #### Scenario 1: 'sync_dir' points directly to a mounted path ``` sync_dir = "/mnt/external/path/to/users/data/location/OneDrive" check_nomount = "true" check_nosync = "true" ``` **Step 1:** Before mounting the device, prepare the `.nosync` file ``` sudo mkdir -p /mnt/external/path/to/users/data/location/OneDrive sudo touch /mnt/external/path/to/users/data/location/OneDrive/.nosync ``` **Step 2:** Test the 'onedrive' Client ``` onedrive -s ``` with the output ``` ... Configuring Global Azure AD Endpoints ERROR: .nosync file found in directory mount point. Aborting application startup process to safeguard data. Attempting to perform a database vacuum to optimise database ... ``` **Step 3:** Mount your device (e.g., via systemd, fstab, or manually) ``` sudo mount /dev/sdX1 /mnt/external ``` **Result:** The OneDrive client will now treat `/mnt/external/path/to/users/data/location/OneDrive` as the sync_dir. If the mount is ever lost, the `.nosync` file becomes visible again, and syncing is halted. #### Scenario 2: 'sync_dir' is a symbolic link to a mounted directory ``` sync_dir = "~/OneDrive" check_nomount = "true" check_nosync = "true" ``` and ``` $ ls -l ~/OneDrive lrwxrwxrwx 1 user user 29 Jul 25 14:44 OneDrive -> /mnt/external/path/to/users/data/location/OneDrive ``` **Step 1:** Before mounting the device, prepare the `.nosync` file ``` sudo mkdir -p /mnt/external/path/to/users/data/location/OneDrive sudo touch /mnt/external/path/to/users/data/location/OneDrive/.nosync ``` **Step 2:** Test the 'onedrive' Client ``` onedrive -s ``` with the output ``` ... Configuring Global Azure AD Endpoints ERROR: .nosync file found in directory mount point. Aborting application startup process to safeguard data. Attempting to perform a database vacuum to optimise database ... ``` **Step 3:** Mount your device (e.g., via systemd, fstab, or manually) ``` sudo mount /dev/sdX1 /mnt/external ``` **Result:** Your symlinked `~/OneDrive` path will now point into the mounted filesystem. If the mount goes missing, the `.nosync` file reappears via the symlink, and the client halts syncing automatically. ## Upload data from the local ~/OneDrive folder to a specific location on OneDrive In some environments, you may not want your local ~/OneDrive folder to be uploaded directly to the root of your OneDrive account online. Unfortunately, the OneDrive API lacks any facility to perform a re-direction of data during upload. The workaround for this is to structure your local filesystem and reconfigure your client to achieve the desired goal. ### High level steps: 1. Create a new folder, for example `/opt/OneDrive` 2. Configure your application config 'sync_dir' to look at this folder 3. Inside `/opt/OneDrive` create the folder you wish to sync the data online to, for example: `/opt/OneDrive/RemoteOnlineDestination` 4. Configure the application to only sync `/opt/OneDrive/RemoteDestination` via 'sync_list' 5. Symbolically link `~/OneDrive` -> `/opt/OneDrive/RemoteOnlineDestination` ### Outcome: * Your `~/OneDrive` will look / feel as per normal * The data will be stored online under `/RemoteOnlineDestination` ### Testing: * Validate your configuration with `onedrive --display-config` * Test your configuration with `onedrive --dry-run` ================================================ FILE: docs/application-config-options.md ================================================ # Application Configuration Options for the OneDrive Client for Linux ## Application Version Before reading this document, please ensure you are running application version [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) or greater. Use `onedrive --version` to determine what application version you are using and upgrade your client if required. ## Table of Contents - [Configuration File Options](#configuration-file-options) - [application_id](#application_id) - [azure_ad_endpoint](#azure_ad_endpoint) - [azure_tenant_id](#azure_tenant_id) - [bypass_data_preservation](#bypass_data_preservation) - [check_nomount](#check_nomount) - [check_nosync](#check_nosync) - [classify_as_big_delete](#classify_as_big_delete) - [cleanup_local_files](#cleanup_local_files) - [connect_timeout](#connect_timeout) - [create_new_file_version](#create_new_file_version) - [data_timeout](#data_timeout) - [debug_https](#debug_https) - [delay_inotify_processing](#delay_inotify_processing) - [disable_download_validation](#disable_download_validation) - [disable_notifications](#disable_notifications) - [disable_permission_set](#disable_permission_set) - [disable_upload_validation](#disable_upload_validation) - [disable_version_check](#disable_version_check) - [disable_websocket_support](#disable_websocket_support) - [display_manager_integration](#display_manager_integration) - [display_running_config](#display_running_config) - [display_transfer_metrics](#display_transfer_metrics) - [dns_timeout](#dns_timeout) - [download_only](#download_only) - [drive_id](#drive_id) - [dry_run](#dry_run) - [enable_logging](#enable_logging) - [file_fragment_size](#file_fragment_size) - [force_http_11](#force_http_11) - [force_session_upload](#force_session_upload) - [inotify_delay](#inotify_delay) - [ip_protocol_version](#ip_protocol_version) - [local_first](#local_first) - [log_dir](#log_dir) - [max_curl_idle](#max_curl_idle) - [monitor_fullscan_frequency](#monitor_fullscan_frequency) - [monitor_interval](#monitor_interval) - [monitor_log_frequency](#monitor_log_frequency) - [no_remote_delete](#no_remote_delete) - [notify_file_actions](#notify_file_actions) - [operation_timeout](#operation_timeout) - [permanent_delete](#permanent_delete) - [rate_limit](#rate_limit) - [read_only_auth_scope](#read_only_auth_scope) - [recycle_bin_path](#recycle_bin_path) - [remove_source_files](#remove_source_files) - [resync](#resync) - [resync_auth](#resync_auth) - [skip_dir](#skip_dir) - [skip_dir_strict_match](#skip_dir_strict_match) - [skip_dotfiles](#skip_dotfiles) - [skip_file](#skip_file) - [skip_size](#skip_size) - [skip_symlinks](#skip_symlinks) - [space_reservation](#space_reservation) - [sync_business_shared_items](#sync_business_shared_items) - [sync_dir](#sync_dir) - [sync_dir_permissions](#sync_dir_permissions) - [sync_file_permissions](#sync_file_permissions) - [sync_root_files](#sync_root_files) - [threads](#threads) - [transfer_order](#transfer_order) - [upload_only](#upload_only) - [use_device_auth](#use_device_auth) - [use_intune_sso](#use_intune_sso) - [use_recycle_bin](#use_recycle_bin) - [user_agent](#user_agent) - [webhook_enabled](#webhook_enabled) - [webhook_expiration_interval](#webhook_expiration_interval) - [webhook_listening_host](#webhook_listening_host) - [webhook_listening_port](#webhook_listening_port) - [webhook_public_url](#webhook_public_url) - [webhook_renewal_interval](#webhook_renewal_interval) - [write_xattr_data](#write_xattr_data) - [Command Line Interface (CLI) Only Options](#command-line-interface-cli-only-options) - [CLI Option: --auth-files](#cli-option---auth-files) - [CLI Option: --auth-response](#cli-option---auth-response) - [CLI Option: --confdir](#cli-option---confdir) - [CLI Option: --create-directory](#cli-option---create-directory) - [CLI Option: --create-share-link](#cli-option---create-share-link) - [CLI Option: --destination-directory](#cli-option---destination-directory) - [CLI Option: --display-config](#cli-option---display-config) - [CLI Option: --display-sync-status](#cli-option---display-sync-status) - [CLI Option: --display-quota](#cli-option---display-quota) - [CLI Option: --download-file](#cli-option---download-file) - [CLI Option: --force](#cli-option---force) - [CLI Option: --force-sync](#cli-option---force-sync) - [CLI Option: --get-file-link](#cli-option---get-file-link) - [CLI Option: --get-sharepoint-drive-id](#cli-option---get-sharepoint-drive-id) - [CLI Option: --list-shared-items](#cli-option---list-shared-items) - [CLI Option: --logout](#cli-option---logout) - [CLI Option: --modified-by](#cli-option---modified-by) - [CLI Option: --monitor | -m](#cli-option---monitor--m) - [CLI Option: --print-access-token](#cli-option---print-access-token) - [CLI Option: --reauth](#cli-option---reauth) - [CLI Option: --remove-directory](#cli-option---remove-directory) - [CLI Option: --share-password](#cli-option---share-password) - [CLI Option: --single-directory](#cli-option---single-directory) - [CLI Option: --source-directory](#cli-option---source-directory) - [CLI Option: --sync | -s](#cli-option---sync--s) - [CLI Option: --sync-shared-files](#cli-option---sync-shared-files) - [CLI Option: --verbose | -v+](#cli-option---verbose--v) - [CLI Option: --with-editing-perms](#cli-option---with-editing-perms) - [Deprecated Configuration File and CLI Options](#deprecated-configuration-file-and-cli-options) - [force_http_2](#force_http_2) - [min_notify_changes](#min_notify_changes) - [CLI Option: --synchronize](#cli-option---synchronize) ## Configuration File Options ### application_id _**Description:**_ This is the config option for application id that used to identify itself to Microsoft OneDrive. In some circumstances, it may be desirable to use your own application id. To do this, you must register a new application with Microsoft Azure via https://portal.azure.com/, then use your new application id with this config option. You can find instructions for configuring your own app registration in [national-cloud-deployments.md](national-cloud-deployments.md) even if you don't necessarily configure it for a national cloud environment. _**Value Type:**_ String _**Default Value:**_ d50ca740-c83f-4d1b-b616-12c519384f0c _**Config Example:**_ `application_id = "d50ca740-c83f-4d1b-b616-12c519384f0c"` ### azure_ad_endpoint _**Description:**_ This is the config option to change the Microsoft Azure Authentication Endpoint that the client uses to conform with data and security requirements that requires data to reside within the geographic borders of that country. _**Value Type:**_ String _**Default Value:**_ *Empty* - not required for normal operation _**Valid Values:**_ USL4, USL5, DE, CN _**Config Example:**_ `azure_ad_endpoint = "DE"` ### azure_tenant_id _**Description:**_ This config option allows the locking of the client to a specific single tenant and will configure your client to use the specified tenant id in its Azure AD and Graph endpoint URIs, instead of "common". The tenant id may be the GUID Directory ID or the fully qualified tenant name. _**Value Type:**_ String _**Default Value:**_ *Empty* - not required for normal operation _**Config Example:**_ `azure_tenant_id = "example.onmicrosoft.us"` or `azure_tenant_id = "0c4be462-a1ab-499b-99e0-da08ce52a2cc"` > [!IMPORTANT] > Must be configured if 'azure_ad_endpoint' is configured. ### bypass_data_preservation _**Description:**_ This config option allows the disabling of preserving local data by renaming the local file in the event of data conflict. If this is enabled, you will experience data loss on your local data as the local file will be over-written with data from OneDrive online. Use with care and caution. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `bypass_data_preservation = "false"` or `bypass_data_preservation = "true"` ### check_nomount _**Description:**_ This config option is useful to prevent application startup & ongoing use in 'Monitor Mode' if the configured 'sync_dir' is a separate disk that is being mounted by your system. This option will check for the presence of a `.nosync` file in your mount point, and if present, abort any sync process to preserve data. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `check_nomount = "false"` or `check_nomount = "true"` _**CLI Option:**_ `--check-for-nomount` > [!TIP] > Create a `.nosync` file in your mount point *before* you mount your disk so that this `.nosync` file visible, in your mount point if your disk is unmounted at any point to preserve your data when you enable this option. ### check_nosync _**Description:**_ This config option is useful to prevent the sync of a *local* directory to Microsoft OneDrive. It will *not* check for this file online to prevent the download of directories to your local system. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `check_nosync = "false"` or `check_nosync = "true"` _**CLI Option Use:**_ `--check-for-nosync` > [!IMPORTANT] > Create a `.nosync` file in any *local* directory that you wish to not sync to Microsoft OneDrive when you enable this option. ### classify_as_big_delete _**Description:**_ This config option defines the number of children in a path that is locally removed which will be classified as a 'big data delete' to safeguard large data removals - which are typically accidental local delete events. _**Value Type:**_ Integer _**Default Value:**_ 1000 _**Config Example:**_ `classify_as_big_delete = "2000"` _**CLI Option Use:**_ `--classify-as-big-delete 2000` > [!NOTE] > If this option is triggered, you will need to add `--force` to force a sync to occur. ### cleanup_local_files _**Description:**_ This config option provides the capability to cleanup local files and folders if they are removed online. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `cleanup_local_files = "false"` or `cleanup_local_files = "true"` _**CLI Option Use:**_ `--cleanup-local-files` > [!IMPORTANT] > This configuration option can only be used with `--download-only`. It cannot be used with any other application option. ### connect_timeout _**Description:**_ This configuration setting manages the TCP connection timeout duration in seconds for HTTPS connections to Microsoft OneDrive when using the curl library (CURLOPT_CONNECTTIMEOUT). _**Value Type:**_ Integer _**Default Value:**_ 10 _**Config Example:**_ `connect_timeout = "15"` ### create_new_file_version _**Description:**_ This setting controls how the application handles the Microsoft SharePoint *feature* which modifies all PDF, MS Office & HTML files post upload, effectively breaking the integrity of your data online. By default, when the application determines that this *feature* has modified your file post upload, the now online modified file will be downloaded. When this option is enabled, rather than downloading the file, a new online file version is created which negates the download of the modified file. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `create_new_file_version = "false"` or `create_new_file_version = "true"` _**CLI Option Use:**_ *None - this is a config file option only* > [!IMPORTANT] > If you enable 'disable_upload_validation' via `disable_upload_validation = "true"` there is zero facility to determine if a file was modified post upload. As such, the application will default to the state that the upload integrity check has failed. When `create_new_file_version = "false"` your uploaded file will be downloaded *regardless* of the online modification state. > [!WARNING] > When this option is set to 'true', new file versions will be created online which will count towards your Microsoft OneDrive Quota. ### data_timeout _**Description:**_ This setting controls the timeout duration, in seconds, for when data is not received on an active connection to Microsoft OneDrive over HTTPS when using the curl library, before that connection is timeout out. _**Value Type:**_ Integer _**Default Value:**_ 60 _**Config Example:**_ `data_timeout = "300"` ### debug_https _**Description:**_ This setting controls whether the curl library is configured to output additional data to assist with diagnosing HTTPS issues and problems. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `debug_https = "false"` or `debug_https = "true"` _**CLI Option Use:**_ `--debug-https` > [!WARNING] > Whilst this option can be used at any time, it is advisable that you only use this option when advised as this will output your `Authorization: bearer` - which is your authentication token to Microsoft OneDrive. ### delay_inotify_processing _**Description:**_ This setting controls whether 'inotify' events should be delayed or not. This option should only ever be enabled when attempting to reduce the impact of editors like Obsidian which constantly write change to disk in an atomic fashion. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `delay_inotify_processing = "false"` or `delay_inotify_processing = "true"` > [!NOTE] > If you enable this option you *must* also enable 'force_session_upload' to ensure that your data uploads are done in a manner that editors, like Obsidian expect. ### disable_download_validation _**Description:**_ This option determines whether the client will conduct integrity validation on files downloaded from Microsoft OneDrive. Sometimes, when downloading files, particularly from SharePoint, there is a discrepancy between the file size reported by the OneDrive API and the byte count received from the SharePoint HTTP Server for the same file. Enable this option to disable the integrity checks performed by this client. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_download_validation = "false"` or `disable_download_validation = "true"` _**CLI Option Use:**_ `--disable-download-validation` > [!CAUTION] > If you're downloading data from SharePoint or OneDrive Business Shared Folders, you might find it necessary to activate this option. It's important to note that any issues encountered aren't due to a problem with this client; instead, they should be regarded as issues with the Microsoft OneDrive technology stack. Enabling this option disables all download integrity checks. > [!CAUTION] > If you are using OneDrive Business Accounts and your organisation implements Azure Information Protection, these AIP files will report as one size & hash online, but when downloaded, will report a totally different size and hash. > > By default these files will fail integrity checking and be deleted, meaning that AIP files will not reside on your platform. > > When you enable this option, the AIP files will download to your platform, however, if there are any other genuine download failures where the size and hash are different, these too will be retained locally meaning you may experience data integrity loss. Use this option with extreme caution. ### disable_notifications _**Description:**_ This setting controls whether GUI notifications are sent from the client to your display manager session. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_notifications = "false"` or `disable_notifications = "true"` _**CLI Option Use:**_ `--disable-notifications` ### disable_permission_set _**Description:**_ This setting controls whether the application will set the permissions on files and directories using the values of 'sync_dir_permissions' and 'sync_file_permissions'. When this option is enabled, file system permission inheritance will be used to assign the permissions for your data. This option may be useful if the file system configured does not allow setting of POSIX permissions. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_permission_set = "false"` or `disable_permission_set = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### disable_upload_validation _**Description:**_ This option determines whether the client will conduct integrity validation on files uploaded to Microsoft OneDrive. Sometimes, when uploading files, particularly to SharePoint, SharePoint will modify your file post upload by adding new data to your file which breaks the integrity checking of the upload performed by this client. Enable this option to disable the integrity checks performed by this client. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_upload_validation = "false"` or `disable_upload_validation = "true"` _**CLI Option Use:**_ `--disable-upload-validation` > [!CAUTION] > If you're uploading data to SharePoint or OneDrive Business Shared Folders, you might find it necessary to activate this option. It's important to note that any issues encountered aren't due to a problem with this client; instead, they should be regarded as issues with the Microsoft OneDrive technology stack. Enabling this option disables all upload integrity checks. ### disable_version_check _**Description:**_ This option determines whether the client will check the GitHub API for the current application version and grace period of running older application versions _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_version_check = "false"` or `disable_version_check = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### disable_websocket_support _**Description:**_ This option disables the built-in WebSocket support that leverages RFC6455 to communicate with the Microsoft Graph API Service, providing near real-time notifications of online changes. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `disable_websocket_support = "false"` or `disable_websocket_support = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### display_manager_integration _**Description:**_ Controls whether the client integrates the configured 'sync_dir' with the desktop’s file manager (e.g. Nautilus for GNOME, Dolphin for KDE), adding it as a “special place” in the sidebar and setting a custom OneDrive folder icon where supported. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `display_manager_integration = "false"` or `display_manager_integration = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### display_running_config _**Description:**_ This option will include the running config of the application at application startup. This may be desirable to enable when running in containerised environments so that any application logging that is occurring, will have the application configuration being consumed at startup, written out to any applicable log file. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `display_running_config = "false"` or `display_running_config = "true"` _**CLI Option Use:**_ `--display-running-config` ### display_transfer_metrics _**Description:**_ This option will display file transfer metrics when enabled. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `display_transfer_metrics = "false"` or `display_transfer_metrics = "true"` _**Output Example:**_ `Transfer Metrics - File: path/to/file.data | Size: 35768 Bytes | Duration: 2.27 Seconds | Speed: 0.02 Mbps (approx)` _**CLI Option Use:**_ *None - this is a config file option only* ### dns_timeout _**Description:**_ This setting controls the libcurl DNS cache value. By default, libcurl caches this info for 60 seconds. This libcurl DNS cache timeout is entirely speculative that a name resolves to the same address for a small amount of time into the future as libcurl does not use DNS TTL properties. We recommend users not to tamper with this option unless strictly necessary. _**Value Type:**_ Integer _**Default Value:**_ 60 _**Config Example:**_ `dns_timeout = "90"` ### download_only _**Description:**_ This setting forces the client to only download data from Microsoft OneDrive and replicate that data locally. No changes made locally will be uploaded to Microsoft OneDrive when using this option. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `download_only = "false"` or `download_only = "true"` _**CLI Option Use:**_ `--download-only` > [!IMPORTANT] > When using this option, the default mode of operation is to not clean up local files that have been deleted online. This ensures that the local data is an *archive* of what was stored online. To cleanup local files use `--cleanup-local-files`. ### drive_id _**Description:**_ This setting controls the specific drive identifier the client will use when syncing with Microsoft OneDrive. _**Value Type:**_ String _**Default Value:**_ *None* _**Config Example:**_ `drive_id = "b!bO8V6s9SSk9R7mWhpIjUrotN73WlW3tEv3OxP_QfIdQimEdOHR-1So6CqeG1MfDB"` > [!NOTE] > This option is typically only used when configuring the client to sync a specific SharePoint Library. If this configuration option is specified in your config file, a value must be specified otherwise the application will exit citing a fatal error has occurred. ### dry_run _**Description:**_ This setting controls the application capability to test your application configuration without actually performing any actual activity (download, upload, move, delete, folder creation). _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `dry_run = "false"` or `dry_run = "true"` _**CLI Option Use:**_ `--dry-run` ### enable_logging _**Description:**_ This setting controls the application logging all actions to a separate file. By default, all log files will be written to `/var/log/onedrive`, however this can changed by using the 'log_dir' config option _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `enable_logging = "false"` or `enable_logging = "true"` _**CLI Option Use:**_ `--enable-logging` > [!IMPORTANT] > Additional configuration is potentially required to configure the default log directory. Refer to the [Enabling the Client Activity Log](./usage.md#enabling-the-client-activity-log) section in usage.md for details ### file_fragment_size _**Description:**_ This option controls the fragment size when uploading large files to Microsoft OneDrive. The value specified is in MB. _**Value Type:**_ Integer _**Default Value:**_ 10 _**Minimum Value:**_ 10 _**Maximum Value:**_ 60 _**Config Example:**_ `file_fragment_size = "25"` _**CLI Option Use:**_ `--file-fragment-size = '25'` > [!NOTE] > Microsoft OneDrive requires that the file fragment size be an exact multiple of 320 KiB. The default value is an exact multiple of this required value. Additional exact multiple options are: > 15, 20, 25, 30, 35, 40, 45, 50, 55 ### force_http_11 _**Description:**_ This setting controls the application HTTP protocol version. By default, the application will use libcurl defaults for which HTTP protocol version will be used to interact with Microsoft OneDrive. Use this setting to downgrade libcurl to only use HTTP/1.1. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `force_http_11 = "false"` or `force_http_11 = "true"` _**CLI Option Use:**_ `--force-http-11` ### force_session_upload _**Description:**_ This option, when enabled, forces the client to use a 'session' upload, which, when the 'file' is uploaded by the session, this includes the local timestamp of the file. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `force_session_upload = "false"` or `force_session_upload = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### inotify_delay _**Description:**_ This option specifies the number of seconds 'inotify' events are paused before they are processed by this client. This value is used to overcome aggressive write applications such as Obsidian which write each keystroke in an atomic manner to the local disk. Due to this atomic write, each 'save' causes the existing file to be deleted and replaced with a new file, which this client sees as multiple constant 'inotify' events. _**Value Type:**_ Integer _**Default Value:**_ 5 _**Maximum Value:**_ 15 _**Config Example:**_ `inotify_delay = "10"` _**CLI Option Use:**_ *None - this is a config file option only* > [!NOTE] > This option is only used if 'delay_inotify_processing' is enabled, otherwise this option is ignored. ### ip_protocol_version _**Description:**_ This setting controls the application IP protocol that should be used when communicating with Microsoft OneDrive. The default is to use IPv4 and IPv6 networks for communicating to Microsoft OneDrive. _**Value Type:**_ Integer _**Default Value:**_ 0 _**Valid Values:**_ 0 = IPv4 + IPv6, 1 = IPv4 Only, 2 = IPv6 Only _**Config Example:**_ `ip_protocol_version = "0"` or `ip_protocol_version = "1"` or `ip_protocol_version = "2"` > [!IMPORTANT] > In some environments where IPv4 and IPv6 are configured at the same time, this causes resolution and routing issues to Microsoft OneDrive. If this is the case, it is advisable to change 'ip_protocol_version' to match your environment. ### local_first _**Description:**_ This setting controls what the application considers the 'source of truth' for your data. By default, what is stored online will be considered as the 'source of truth' when syncing to your local machine. When using this option, your local data will be considered the 'source of truth'. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `local_first = "false"` or `local_first = "true"` _**CLI Option Use:**_ `--local-first` ### log_dir _**Description:**_ This setting controls the custom application log path when 'enable_logging' has been enabled. By default, all log files will be written to `/var/log/onedrive`. _**Value Type:**_ String _**Default Value:**_ *None* _**Config Example:**_ `log_dir = "~/logs/"` _**CLI Option Use:**_ `--log-dir "~/logs/"` ### max_curl_idle _**Description:**_ This configuration option controls the number of seconds that elapse after a cURL engine was last used before it is considered stale and destroyed. Evidence suggests that some upstream network devices ignore the cURL keep-alive setting and forcibly close the active TCP connection when idle. _**Value Type:**_ Integer _**Default Value:**_ 120 _**Config Example:**_ `max_curl_idle = "120"` _**CLI Option Use:**_ *None - this is a config file option only* > [!IMPORTANT] > It is strongly recommended not to modify this setting without conducting thorough network testing. Changing this option may lead to unexpected behaviour or connectivity issues, especially if upstream network devices handle idle connections in non-standard ways. ### monitor_fullscan_frequency _**Description:**_ This configuration option controls the number of 'monitor_interval' iterations between when a full scan of your data is performed to ensure data integrity and consistency. _**Value Type:**_ Integer _**Default Value:**_ 12 _**Config Example:**_ `monitor_fullscan_frequency = "24"` _**CLI Option Use:**_ `--monitor-fullscan-frequency '24'` > [!NOTE] > By default without configuration, 'monitor_fullscan_frequency' is set to 12. In this default state, this means that a full scan is performed every 'monitor_interval' x 'monitor_fullscan_frequency' = 3600 seconds. This setting is only applicable when running in `--monitor` mode. Setting this configuration option to '0' will *disable* the full scan of your data online. ### monitor_interval _**Description:**_ This configuration setting determines how often the synchronisation loops run in --monitor mode, measured in seconds. When this time period elapses, the client will check for online changes in Microsoft OneDrive, conduct integrity checks on local data and scan the local 'sync_dir' to identify any new content that hasn't been uploaded yet. _**Value Type:**_ Integer _**Default Value:**_ 300 _**Config Example:**_ `monitor_interval = "600"` _**CLI Option Use:**_ `--monitor-interval '600'` > [!NOTE] > A minimum value of 300 is enforced for this configuration setting. ### monitor_log_frequency _**Description:**_ This configuration option controls the suppression of frequently printed log items to the system console when using `--monitor` mode. The aim of this configuration item is to reduce the log output when near zero sync activity is occurring. _**Value Type:**_ Integer _**Default Value:**_ 12 _**Config Example:**_ `monitor_log_frequency = "24"` _**CLI Option Use:**_ `--monitor-log-frequency '24'` _**Usage Example:**_ By default, at application start-up when using `--monitor` mode, the following will be logged to indicate that the application has correctly started and has performed all the initial processing steps: ```text Reading configuration file: /home/user/.config/onedrive/config Configuration file successfully loaded Configuring Global Azure AD Endpoints Sync Engine Initialised with new Onedrive API instance All application operations will be performed in: /home/user/OneDrive OneDrive synchronisation interval (seconds): 300 Initialising filesystem inotify monitoring ... Performing initial synchronisation to ensure consistent local state ... Starting a sync with Microsoft OneDrive Fetching items from the OneDrive API for Drive ID: b!bO8V6s9SSk9R7mWhpIjUrotN73WlW3tEv3OxP_QfIdQimEdOHR-1So6CqeG1MfDB .. Processing changes and items received from Microsoft OneDrive ... Performing a database consistency and integrity check on locally stored data ... Scanning the local file system '~/OneDrive' for new data to upload ... Performing a final true-up scan of online data from Microsoft OneDrive Fetching items from the OneDrive API for Drive ID: b!bO8V6s9SSk9R7mWhpIjUrotN73WlW3tEv3OxP_QfIdQimEdOHR-1So6CqeG1MfDB .. Processing changes and items received from Microsoft OneDrive ... Sync with Microsoft OneDrive is complete ``` Then, based on 'monitor_log_frequency', the following output will be logged until the suppression loop value is reached: ```text Starting a sync with Microsoft OneDrive Syncing changes from Microsoft OneDrive ... Sync with Microsoft OneDrive is complete ``` > [!NOTE] > The additional log output `Performing a database consistency and integrity check on locally stored data ...` will only be displayed when this activity is occurring which is triggered by 'monitor_fullscan_frequency'. > [!NOTE] > If verbose application output is being used (`--verbose`), then this configuration setting has zero effect, as application verbose output takes priority over application output suppression. ### no_remote_delete _**Description:**_ This configuration option controls whether local file and folder deletes are actioned on Microsoft OneDrive. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `local_first = "false"` or `local_first = "true"` _**CLI Option Use:**_ `--no-remote-delete` > [!IMPORTANT] > This configuration option can *only* be used in conjunction with `--upload-only` ### notify_file_actions _**Description:**_ This configuration option controls whether the client will log via GUI notifications successful actions that the client performs. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `notify_file_actions = "true"` > [!NOTE] > GUI Notification Support must be compiled in first, otherwise this option will have zero effect and will not be used. ### operation_timeout _**Description:**_ This configuration option controls the maximum total time (in seconds) that any network operation is allowed to take. This limit applies to the *entire* request, including DNS resolution, connection setup, TLS negotiation, and data transfer. This option maps directly to libcurl’s `CURLOPT_TIMEOUT`. _**Value Type:**_ Integer _**Default Value:**_ 0 (no timeout) _**Config Example:**_ `operation_timeout = "3600"` > [!IMPORTANT] > Setting a non-zero value will cause libcurl to abort the operation once the specified time has elapsed — even if data is still flowing normally. > For large file downloads, particularly on slower connections, enabling a finite timeout may cause transfers to be terminated prematurely. > > It is strongly recommend to leave this option at its default of `0` unless you specifically require a hard global time limit. ### permanent_delete _**Description:**_ Permanently delete an item online when it is removed locally. When using this method, they're permanently removed and aren't sent to the Microsoft OneDrive Recycle Bin. Therefore, permanently deleted drive items can't be restored afterward. Online data loss MAY occur in this scenario. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `permanent_delete = "true"` _**CLI Option Use:**_ *None - this is a config file option only* > [!IMPORTANT] > The Microsoft OneDrive API for this capability is also very narrow: > | Account Type | Config Option is Supported | > |:-------------|:----------------:| > | Personal | ❌ | > | Business | ✔ | > | SharePoint | ✔ | > | Microsoft Cloud Germany | ✔ | > | Microsoft Cloud for US Government | ❌ | > | Azure and Office365 operated by VNET in China | ❌ | > > When using this config option against an unsupported Personal Accounts the following message will be generated: > ``` > WARNING: The application is configured to permanently delete files online; however, this action is not supported by Microsoft OneDrive Personal Accounts. > ``` > > When using this config option against a supported account the following message will be generated: > ``` > WARNING: Application has been configured to permanently remove files online rather than send to the recycle bin. Permanently deleted items can't be restored. > WARNING: Online data loss MAY occur in this scenario. > ``` > ### rate_limit _**Description:**_ This configuration option controls the bandwidth used by the application, per thread, when interacting with Microsoft OneDrive. _**Value Type:**_ Integer _**Default Value:**_ 0 (unlimited, use available bandwidth per thread) _**Valid Values:**_ Valid tested values for this configuration option are as follows: * 131072 = 128 KB/s - absolute minimum for basic application operations to prevent timeouts * 262144 = 256 KB/s * 524288 = 512 KB/s * 1048576 = 1 MB/s * 10485760 = 10 MB/s * 104857600 = 100 MB/s _**Config Example:**_ `rate_limit = "131072"` ### read_only_auth_scope _**Description:**_ This configuration option controls whether the OneDrive Client for Linux operates in a totally in read-only operation. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `read_only_auth_scope = "false"` or `read_only_auth_scope = "true"` > [!IMPORTANT] > When using 'read_only_auth_scope' you also will need to remove your existing application access consent otherwise old authentication consent will be valid and will be used. This will mean the application will technically have the consent to upload data until you revoke this consent. ### recycle_bin_path _**Description:**_ This configuration option allows you to specify the 'Recycle Bin' path for the application. _**Value Type:**_ String _**Default Value:**_ *None* however the application will use `~/.local/share/Trash` as the pre-defined default so that files will be placed in the correct location for your user profile. _**CLI Option Use:**_ *None - this is a config file option only* _**Config Example:**_ `recycle_bin_path = "/path/to/desired/location/"` ### remove_source_files _**Description:**_ This configuration option controls whether the OneDrive Client for Linux removes the local file post successful transfer to Microsoft OneDrive. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `remove_source_files = "false"` or `remove_source_files = "true"` _**CLI Option Use:**_ `--remove-source-files` > [!IMPORTANT] > This configuration option can *only* be used in conjunction with `--upload-only` ### remove_source_folders _**Description:**_ This configuration option controls whether the OneDrive Client for Linux removes the local directory structure post successful file transfer to Microsoft OneDrive. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `remove_source_folders = "false"` or `remove_source_folders = "true"` _**CLI Option Use:**_ `--remove-source-folders` > [!IMPORTANT] > This configuration option can *only* be used in conjunction with `--upload-only --remove-source-files` > [!IMPORTANT] > The directory structure will only be removed if it is empty. ### resync _**Description:**_ This configuration option controls whether the known local sync state with Microsoft OneDrive is removed at application startup. When this option is used, a full scan of your data online is performed to ensure that the local sync state is correctly built back up. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `resync = "false"` or `resync = "true"` _**CLI Option Use:**_ `--resync` > [!CAUTION] > It's highly recommended to use this option only if the application prompts you to do so. Don't blindly use this option as a default option. If you alter any of the subsequent configuration items, you will be required to execute a `--resync` to make sure your client is syncing your data with the updated configuration: > * drive_id > * sync_dir > * skip_file > * skip_dir > * skip_dotfiles > * skip_symlinks > * sync_business_shared_items > * Creating, Modifying or Deleting the 'sync_list' file > [!IMPORTANT] > The increased activity against the Microsoft Graph API when using this option may trigger HTTP 429 (throttling) responses during the synchronisation process. ### resync_auth _**Description:**_ This configuration option controls the approval of performing a 'resync' which can be beneficial in automated environments. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `resync_auth = "false"` or `resync_auth = "true"` _**CLI Option Use:**_ `--resync-auth` > [!TIP] > In certain automated environments (assuming you know what you're doing due to using automation), to avoid the 'proceed with acknowledgement' resync requirement, this option allows you to automatically acknowledge the resync prompt. ### skip_dir _**Description:**_ This configuration option controls whether the application skips certain directories from being synced. Directories can be specified in 2 ways: * As a single entry. This will search the respective path for this entry and skip all instances where this directory is present, where ever it may exist. * As a full path entry. This will skip the explicit path as set. > [!IMPORTANT] > Entries for 'skip_dir' are *relative* to your 'sync_dir' path. _**Value Type:**_ String _**Default Value:**_ *Empty* - not required for normal operation _**Config Example:**_ Patterns are case insensitive. `*` and `?` [wildcards characters](https://technet.microsoft.com/en-us/library/bb490639.aspx) are supported. Use `|` to separate multiple patterns. ```text skip_dir = "Desktop|Documents/IISExpress|Documents/SQL Server Management Studio|Documents/Visual Studio*|Documents/WindowsPowerShell|.Rproj-user" ``` The 'skip_dir' option can also be specified multiple times within your config file, for example: ```text skip_dir = "SkipThisDirectoryAnywhere" skip_dir = ".SkipThisOtherDirectoryAnywhere" skip_dir = "/Explicit/Path/To/A/Directory" skip_dir = "/Another/Explicit/Path/To/Different/Directory" ``` This will be interpreted the same as: ```text skip_dir = "SkipThisDirectoryAnywhere|.SkipThisOtherDirectoryAnywhere|/Explicit/Path/To/A/Directory|/Another/Explicit/Path/To/Different/Directory" ``` _**CLI Option Use:**_ `--skip-dir 'SkipThisDirectoryAnywhere|.SkipThisOtherDirectoryAnywhere|/Explicit/Path/To/A/Directory|/Another/Explicit/Path/To/Different/Directory'` > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. If using the config file and CLI option is used, the CLI option will *replace* the config file entries. After changing or modifying this option, you will be required to perform a resync. ### skip_dir_strict_match _**Description:**_ This configuration option controls whether the application performs strict directory matching when checking 'skip_dir' items. When enabled, the 'skip_dir' item must be a full path match to the path to be skipped. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `skip_dir_strict_match = "false"` or `skip_dir_strict_match = "true"` _**CLI Option Use:**_ `--skip-dir-strict-match` ### skip_dotfiles _**Description:**_ This configuration option controls whether the application will skip all .files and .folders when performing sync operations. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `skip_dotfiles = "false"` or `skip_dotfiles = "true"` _**CLI Option Use:**_ `--skip-dot-files` > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. After changing this option, you will be required to perform a resync. ### skip_file _**Description:**_ This configuration option controls whether the application skips certain files from being synced. _**Value Type:**_ String _**Default Value:**_ `~*|.~*|*.tmp|*.swp|*.partial` By default, the following files will be skipped: | Skip File Pattern | Meaning | Why this should be skipped | |:------------------|:---------------------------|:---------------------------| | `~*` | Files that start with `~` | Temporary or backup files. Typically auto-created by various programs during editing sessions. These are not intended to be saved permanently. Example: Emacs, Vim, and others create such files. | | `.~*` | Files that start with `.~` | Hidden lock or temp files, especially from LibreOffice and OpenOffice. (E.g., `.~lock.MyFile.docx#`) These are only used to prevent multiple users editing the same file simultaneously. | | `*.tmp` | Files ending in `.tmp` | Generic temporary files created by applications like browsers, editors, installers. They represent intermediate data and are usually auto-deleted after a session. | | `*.swp` | Files ending in `.swp` | Vim (and vi) swap files. Created to protect against crash recovery during text editing. Should not be synced because they are transient. | | `*.partial` | Files ending in `.partial` | Partially downloaded files. Common in browsers (like Firefox `.partial` download files), background downloaders and this client. Incomplete by nature. Syncing them causes broken files online. | The following suggested skip file patterns are not included in the default configuration but could also be considered for skipping: | Skip File Pattern | Meaning | Why this should be skipped | |:------------------|:---------------------------|:---------------------------| | `*.bak` | Files ending in `.bak` | Backup files created by many text editors, IDEs, or applications. These are automatic backups made to preserve earlier versions of files before editing changes are saved. They are not intended for syncing — they are redundant copies of existing or previous files. | > [!IMPORTANT] > If you define your own 'skip_file' configuration, the default settings listed above will be *overridden*. It is strongly recommended that you explicitly include the default 'skip_file' rules alongside your custom entries to ensure temporary and/or transient files are still correctly skipped. _**Config Example:**_ Patterns are case insensitive. `*` and `?` [wildcards characters](https://technet.microsoft.com/en-us/library/bb490639.aspx) are supported. Use `|` to separate multiple patterns. Files can be skipped in the following fashion: * Specify a wildcard, eg: '*.txt' (skip all txt files) * Explicitly specify the filename and it's full path relative to your sync_dir, eg: '/path/to/file/filename.ext' * Explicitly specify the filename only and skip every instance of this filename, eg: 'filename.ext' ```text skip_file = "~*|/Documents/OneNote*|/Documents/config.xlaunch|myfile.ext|/Documents/keepass.kdbx" ``` > [!IMPORTANT] > Entries for 'skip_file' are *relative* to your 'sync_dir' path. The 'skip_file' option can be specified multiple times within your config file, for example: ```text # Defaults - always keep skip_file = "~*|.~*|*.tmp|*.swp|*.partial" # Custom 'skip_file' additions skip_file = "*.blah" skip_file = "never_sync.file" skip_file = "/Documents/keepass.kdbx" ``` This will be interpreted the same as: ```text skip_file = "~*|.~*|*.tmp|*.swp|*.partial|*.blah|never_sync.file|/Documents/keepass.kdbx" ``` _**CLI Option Use:**_ `--skip-file '~*|.~*|*.tmp|*.swp|*.partial|*.blah|never_sync.file|/Documents/keepass.kdbx'` > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. If using the config file and CLI option is used, the CLI option will *replace* the config file entries. After changing or modifying this option, you will be required to perform a resync. ### skip_size _**Description:**_ This configuration option controls whether the application skips syncing certain files larger than the specified size. The value specified is in MB. _**Value Type:**_ Integer _**Default Value:**_ 0 (all files, regardless of size, are synced) _**Config Example:**_ `skip_size = "50"` _**CLI Option Use:**_ `--skip-size '50'` > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. After changing this option, you will be required to perform a resync. ### skip_symlinks _**Description:**_ This configuration option controls whether the application will skip all symbolic links when performing sync operations. Microsoft OneDrive has no concept or understanding of symbolic links, and attempting to upload a symbolic link to Microsoft OneDrive generates a platform API error. All data (files and folders) that are uploaded to OneDrive must be whole files or actual directories. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `skip_symlinks = "false"` or `skip_symlinks = "true"` _**CLI Option Use:**_ `--skip-symlinks` > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. After changing this option, you will be required to perform a resync. ### space_reservation _**Description:**_ This configuration option controls how much local disk space should be reserved, to prevent the application from filling up your entire disk due to misconfiguration _**Value Type:**_ Integer _**Default Value:**_ 50 MB (expressed as Bytes when using `--display-config`) _**Config Example:**_ `space_reservation = "100"` _**CLI Option Use:**_ `--space-reservation '100'` ### sync_business_shared_items _**Description:**_ This configuration option controls whether OneDrive Business | Office 365 Shared Folders, when added as a 'shortcut' to your 'My Files', will be synced to your local system. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `sync_business_shared_items = "false"` or `sync_business_shared_items = "true"` _**CLI Option Use:**_ *None - this is a config file option only* > [!NOTE] > This option is considered a 'Client Side Filtering Rule' and if configured, is utilised for all sync operations. After changing this option, you will be required to perform a resync. > [!CAUTION] > This option is *not* backwards compatible with any v2.4.x application version. If you are enabling this option on *any* system running v2.5.x application version, all your application versions being used *everywhere* must be v2.5.x codebase. ### sync_dir _**Description:**_ This configuration option determines the location on your local filesystem where your data from Microsoft OneDrive will be saved. _**Value Type:**_ String _**Default Value:**_ `~/OneDrive` _**Config Example:**_ `sync_dir = "~/MyDirToSync"` _**CLI Option Use:**_ `--syncdir '~/MyDirToSync'` > [!CAUTION] > After changing this option, you will be required to perform a resync. Do not change or modify this option without fully understanding the implications of doing so. ### sync_dir_permissions _**Description:**_ This configuration option defines the directory permissions applied when a new directory is created locally during the process of syncing your data from Microsoft OneDrive. _**Value Type:**_ Integer _**Default Value:**_ `700` - This provides the following permissions: `drwx------` _**Config Example:**_ `sync_dir_permissions = "700"` > [!IMPORTANT] > Use the [Unix Permissions Calculator](https://chmod-calculator.com/) to help you determine the necessary new permissions. You will need to manually update all existing directory permissions if you modify this value. ### sync_file_permissions _**Description:**_ This configuration option defines the file permissions applied when a new file is created locally during the process of syncing your data from Microsoft OneDrive. _**Value Type:**_ Integer _**Default Value:**_ `600` - This provides the following permissions: `-rw-------` _**Config Example:**_ `sync_file_permissions = "600"` > [!IMPORTANT] > Use the [Unix Permissions Calculator](https://chmod-calculator.com/) to help you determine the necessary new permissions. You will need to manually update all existing directory permissions if you modify this value. ### sync_root_files _**Description:**_ This configuration option manages the synchronisation of files located in the 'sync_dir' root when using a 'sync_list.' It enables you to sync all these files by default, eliminating the need to repeatedly modify your 'sync_list' and initiate resynchronisation. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `sync_root_files = "false"` or `sync_root_files = "true"` _**CLI Option Use:**_ `--sync-root-files` > [!IMPORTANT] > Although it's not mandatory, it's recommended that after enabling this option, you perform a `--resync`. This ensures that any previously excluded content is now included in your sync process. ### threads _**Description:**_ This configuration option controls the number of worker threads used for parallel upload and download operations when transferring files between your local system and Microsoft OneDrive. Each thread handles a discrete portion of the workload, improving performance when used appropriately. All non-transfer operations, such as folder listings (`/children`), delta queries (`/delta`), and metadata requests are processed serially on a single thread. _**Value Type:**_ Integer _**Default Value:**_ `8` _**Maximum Value:**_ `16` _**Config Example:**_ `threads = "16"` _**CLI Option Use:**_ `--threads '16'` > [!NOTE] > The default value of `8` threads is based on the average number of physical CPU cores found in consumer and workstation-grade Intel and AMD processors released from approximately 2012 through 2025. This includes laptops, desktops, and server-grade CPUs where 4–8 physical cores are typical. > > In extensive testing, configuring the application with more than `16` threads — regardless of available physical CPU cores — frequently caused the Microsoft OneDrive service to become blocked due to excessive API request volume. > [!NOTE] > The threads setting only affects file transfer operations. All API operations outside of upload/download operations are single-threaded. > > This option allows the alignment to Microsoft’s [Graph API guidance](https://learn.microsoft.com/en-us/graph/throttling) which recommends limiting concurrent requests to 5–10. The default of `8` provides a safe and performant baseline. > [!IMPORTANT] > For optimal performance and application stability, the number of threads should not exceed the number of **physical CPU cores** available to the system. Setting the thread count too high can result in **CPU contention**, increased **context switching**, and **reduced throughput** due to over-scheduling. > > If running inside a container or virtual machine, ensure that the container/VM has sufficient allocated CPU cores before increasing this setting. > [!IMPORTANT] > If the configured `threads` value (default or manual) exceeds the number of available CPU cores, the application will issue a warning similar to the following: > > ``` > WARNING: Configured 'threads = 8' exceeds available CPU cores (CPU_COUNT). > This may lead to reduced performance, CPU contention, and instability. For best results, set 'threads' no higher than the number of physical CPU cores. > ``` > > If this warning message appears during application startup, you **must** review and adjust your threads setting to match the number of physical CPU cores on your system to avoid degraded performance or instability. > [!IMPORTANT] > The application fully implements Microsoft’s throttling requirements for handling 429 and 503 response codes by: > * Handles 429 and 503 responses using exponential backoff > * Respects Retry-After headers provided by the API for the required back off period > * Limits concurrency to the recommended limits > > If you receive this application output: >``` >Handling a Microsoft Graph API HTTP 429 Response Code (Too Many Requests) - Internal Thread ID: AbCdEfGhIjKlMnOp >``` > Reduce your configured 'threads' value or raise a support ticket with Microsoft > [!WARNING] > Increasing or keeping the thread count beyond the default or available physical CPU cores will also result in higher **system resource utilisation**, particularly in terms of CPU load and local TCP port consumption. On lower-spec systems or in constrained environments, this may lead to **network saturation**, **unpredictable behaviour**, **increase in throttling behaviour by Microsoft** or **application crashes** due to resource exhaustion. ### transfer_order _**Description:**_ This configuration option controls the transfer order of files between your local system and Microsoft OneDrive. _**Value Type:**_ String _**Default Value:**_ `default` _**Config Example:**_ #### Transfer by size, smallest first ``` transfer_order = "size_asc" ``` #### Transfer by size, largest first ``` transfer_order = "size_dsc" ``` #### Transfer by file name sorted A to Z ``` transfer_order = "name_asc" ``` #### Transfer by file name sorted Z to A ``` transfer_order = "name_dsc" ``` ### upload_only _**Description:**_ This setting forces the client to only upload data to Microsoft OneDrive and replicate the locate state online. By default, this will also remove content online, that has been removed locally. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `upload_only = "false"` or `upload_only = "true"` _**CLI Option Use:**_ `--upload-only` > [!IMPORTANT] > To ensure that data deleted locally remains accessible online, you can use the 'no_remote_delete' option. If you want to delete the data from your local storage after a successful upload to Microsoft OneDrive, you can use the 'remove_source_files' option. ### use_device_auth _**Description:**_ Enable this option to authenticate using the Microsoft OAuth2 Device Authorisation Flow (`device_code` grant). This flow allows the client to initiate a sign-in process without launching a web browser directly — ideal for headless systems or remote sessions. A short code and URL will be provided for the user to complete authentication via a separate browser-enabled device. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `use_device_auth = "false"` or `use_device_auth = "true"` _**CLI Option Use:**_ *None - this is a config file option only* > [!IMPORTANT] > This option is fully supported for Microsoft Entra ID (Work/School) accounts. For personal Microsoft accounts (e.g., @outlook.com or @hotmail.com), this method of authentication is not supported. Please use the interactive interactive authentication method (default) to authenticate this application. ### use_intune_sso _**Description:**_ Enable this option to authenticate using Intune Single Sign-On (SSO) via the Microsoft Identity Device Broker over D-Bus. This method is suitable for environments where the system is Intune-enrolled and allows seamless token retrieval without requiring browser interaction. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `use_intune_sso = "false"` or `use_intune_sso = "true"` _**CLI Option Use:**_ *None - this is a config file option only* > [!NOTE] > The installation and configuration of Intune for your platform is beyond the scope of this documentation. ### use_recycle_bin _**Description:**_ This configuration option controls the application function to move online deleted files to a 'Recycle Bin' on your system. This allows you to review online deleted data manually before this is purged from your actual system. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `use_recycle_bin = "false"` or `use_recycle_bin = "true"` _**CLI Option Use:**_ *None - this is a config file option only* ### user_agent _**Description:**_ This configuration option controls the 'User-Agent' request header that is presented to Microsoft Graph API when accessing the Microsoft OneDrive service. This string lets servers and network peers identify the application, operating system, vendor, and/or version of the application making the request. We recommend users not to tamper with this option unless strictly necessary. _**Value Type:**_ String _**Default Value:**_ `ISV|abraunegg|OneDrive Client for Linux/vX.Y.Z-A-bcdefghi` _**Config Example:**_ `user_agent = "ISV|CompanyName|AppName/Version"` > [!IMPORTANT] > The default 'user_agent' value conforms to specific Microsoft requirements to identify as an ISV that complies with OneDrive traffic decoration requirements. Changing this value potentially will impact how Microsoft see's your client, thus your traffic may get throttled. For further information please read: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online ### webhook_enabled _**Description:**_ This configuration option controls the application feature 'webhooks' to allow you to subscribe to remote updates as published by Microsoft OneDrive. This option only operates when the client is using 'Monitor Mode'. _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ The following is the minimum working example that needs to be added to your 'config' file to enable 'webhooks' successfully: ```text webhook_enabled = "true" webhook_public_url = "https:///webhooks/onedrive" ``` > [!NOTE] > Setting `webhook_enabled = "true"` enables the webhook feature in 'monitor' mode. The onedrive process will listen for incoming updates at a configurable endpoint, which defaults to `0.0.0.0:8888`. > [!IMPORTANT] > A valid HTTPS certificate is required for your public-facing URL if using nginx. Self signed certificates will be rejected. Consider using https://letsencrypt.org/ to utilise free SSL certificates for your public-facing URL. > [!TIP] > If you receive this application error: `Subscription validation request failed. Response must exactly match validationToken query parameter.` the most likely cause for this error will be your nginx configuration. > > To resolve this configuration issue, potentially investigate adding the following 'proxy' configuration options to your nginx configuration file: > ```text > server { > listen 443; > server_name ; > location /webhooks/onedrive { > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Original-Request-URI $request_uri; > proxy_read_timeout 300s; > proxy_connect_timeout 75s; > proxy_buffering off; > proxy_http_version 1.1; > proxy_pass http://127.0.0.1:8888; > } > } > ``` > For any further nginx configuration assistance, please refer to: https://docs.nginx.com/ ### webhook_expiration_interval _**Description:**_ This configuration option controls the frequency at which an existing Microsoft OneDrive webhook subscription expires. The value is expressed in the number of seconds before expiry. _**Value Type:**_ Integer _**Default Value:**_ 600 _**Config Example:**_ `webhook_expiration_interval = "1200"` ### webhook_listening_host _**Description:**_ This configuration option controls the host address that this client binds to, when the webhook feature is enabled. _**Value Type:**_ String _**Default Value:**_ 0.0.0.0 _**Config Example:**_ `webhook_listening_host = ""` - this will use the default value. `webhook_listening_host = "192.168.3.4"` - this will bind the client to use the IP address 192.168.3.4. > [!NOTE] > Use in conjunction with 'webhook_listening_port' to change the webhook listening endpoint. ### webhook_listening_port _**Description:**_ This configuration option controls the TCP port that this client listens on, when the webhook feature is enabled. _**Value Type:**_ Integer _**Default Value:**_ 8888 _**Config Example:**_ `webhook_listening_port = "9999"` > [!NOTE] > Use in conjunction with 'webhook_listening_host' to change the webhook listening endpoint. ### webhook_public_url _**Description:**_ This configuration option controls the URL that Microsoft will send subscription notifications to. This must be a valid Internet accessible URL. _**Value Type:**_ String _**Default Value:**_ *empty* _**Config Example:**_ ```text webhook_public_url = "https:///webhooks/onedrive" ``` ### webhook_renewal_interval _**Description:**_ This configuration option controls the frequency at which an existing Microsoft OneDrive webhook subscription is renewed. The value is expressed in the number of seconds before renewal. _**Value Type:**_ Integer _**Default Value:**_ 300 _**Config Example:**_ `webhook_renewal_interval = "600"` ### webhook_retry_interval _**Description:**_ This configuration option controls the frequency at which an existing Microsoft OneDrive webhook subscription is retried when creating or renewing a subscription failed. The value is expressed in the number of seconds before retry. _**Value Type:**_ Integer _**Default Value:**_ 60 _**Config Example:**_ `webhook_retry_interval = "120"` ### write_xattr_data _**Description:**_ This setting enables writing xattr values detailing the 'createdBy' and 'lastModifiedBy' information provided by the OneDrive API _**Value Type:**_ Boolean _**Default Value:**_ False _**Config Example:**_ `write_xattr_data = "false"` or `write_xattr_data = "true"` _**CLI Option Use:**_ *None - this is a config file option only* _**xattr Data Example:**_ ``` user.onedrive.createdBy="Account Display Name" user.onedrive.lastModifiedBy="Account Display Name" ``` ## Command Line Interface (CLI) Only Options ### CLI Option: --auth-files _**Description:**_ This CLI option allows the user to perform application authentication not via an interactive dialog but via specific files that the application uses to read the authentication data from. _**Usage Example:**_ `onedrive --auth-files authUrl:responseUrl` > [!IMPORTANT] > The authorisation URL is written to the specified 'authUrl' file, then onedrive waits for the file 'responseUrl' to be present, and reads the authentication response from that file. Example: > > ```text > onedrive --auth-files '~/onedrive-auth-url:~/onedrive-response-url' > Reading configuration file: /home/alex/.config/onedrive/config > Configuration file successfully loaded > Configuring Global Azure AD Endpoints > Client requires authentication before proceeding. Waiting for --auth-files elements to be available. > ``` > At this point, the client has written the file `~/onedrive-auth-url` which contains the authentication URL that needs to be visited to perform the authentication process. The client will now wait and watch for the presence of the file `~/onedrive-response-url`. > > Visit the authentication URL, and then create a new file called `~/onedrive-response-url` with the response URI. Once this has been done, the application will acknowledge the presence of this file, read the contents, and authenticate the application. > ```text > Sync Engine Initialised with new Onedrive API instance > > --sync or --monitor switches missing from your command line input. Please add one (not both) of these switches to your command line or use 'onedrive --help' for further assistance. > > No OneDrive sync will be performed without one of these two arguments being present. > ``` ### CLI Option: --auth-response _**Description:**_ This CLI option allows the user to perform application authentication not via an interactive dialog but via providing the authentication response URI directly. _**Usage Example:**_ `onedrive --auth-response https://login.microsoftonline.com/common/oauth2/nativeclient?code=` > [!TIP] > Typically, unless the application client identifier has been modified, authentication scopes are being modified or a specific Azure Tenant is being specified, the authentication URL will most likely be as follows: > ```text > https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=d50ca740-c83f-4d1b-b616-12c519384f0c&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient > ``` > With this URL being known, it is possible ahead of time to request an authentication token by visiting this URL, and performing the authentication access request. ### CLI Option: --confdir _**Description:**_ This CLI option allows the user to specify where all the application configuration and relevant components are stored. _**Usage Example:**_ `onedrive --confdir '~/.config/onedrive-business/'` > [!IMPORTANT] > If using this option, it must be specified each and every time the application is used. If this is omitted, the application default configuration directory will be used. ### CLI Option: --create-directory _**Description:**_ This CLI option allows the user to create the specified directory path on Microsoft OneDrive without performing a sync. _**Usage Example:**_ `onedrive --create-directory 'path/of/new/folder/structure/to/create/'` > [!IMPORTANT] > The specified path to create is relative to your configured 'sync_dir'. ### CLI Option: --create-share-link _**Description:**_ This CLI option enables the creation of a shareable file link that can be provided to users to access the file that is stored on Microsoft OneDrive. By default, the permissions for the file will be 'read-only'. _**Usage Example:**_ `onedrive --create-share-link 'relative/path/to/your/file.txt'` > [!IMPORTANT] > If writable access to the file is required, you must add `--with-editing-perms` to your command. See below for details. ### CLI Option: --destination-directory _**Description:**_ This CLI option specifies the 'destination' portion of moving a file or folder online, without performing a sync operation. _**Usage Example:**_ `onedrive --source-directory 'path/as/source/' --destination-directory 'path/as/destination'` > [!IMPORTANT] > All specified paths are relative to your configured 'sync_dir'. ### CLI Option: --display-config _**Description:**_ This CLI option will display the effective application configuration _**Usage Example:**_ `onedrive --display-config` ### CLI Option: --display-sync-status _**Description:**_ This CLI option will display the sync status of the configured 'sync_dir' _**Usage Example:**_ `onedrive --display-sync-status` > [!TIP] > This option can also use the `--single-directory` option to determine the sync status of a specific directory within the configured 'sync_dir' ### CLI Option: ---display-quota _**Description:**_ This CLI option will display the quota status of the account drive id or the configured 'drive_id' value _**Usage Example:**_ `onedrive --display-quota` ### CLI Option: --download-file _**Description:**_ This CLI option will download a single file based on the online path. No sync will be performed. _**Usage Example:**_ `onedrive --download-file 'path/to/your/file/online'` ### CLI Option: --force _**Description:**_ This CLI option enables the force the deletion of data when a 'big delete' is detected. _**Usage Example:**_ `onedrive --sync --verbose --force` > [!IMPORTANT] > This option should only be used exclusively in cases where you've initiated a 'big delete' and genuinely intend to remove all the data that is set to be deleted online. ### CLI Option: --force-sync _**Description:**_ This CLI option enables the syncing of a specific directory, using the Client Side Filtering application defaults, overriding any user application configuration. _**Usage Example:**_ `onedrive --sync --verbose --force-sync --single-directory 'Data'` > [!NOTE] > When this option is used, you will be presented with the following warning and risk acceptance: > ```text > WARNING: Overriding application configuration to use application defaults for skip_dir and skip_file due to --synch --single-directory --force-sync being used > > The use of --force-sync will reconfigure the application to use defaults. This may have untold and unknown future impacts. > By proceeding in using this option you accept any impacts including any data loss that may occur as a result of using --force-sync. > > Are you sure you wish to proceed with --force-sync [Y/N] > ``` > To proceed with this sync task, you must risk accept the actions you are taking. If you have any concerns, first use `--dry-run` and evaluate the outcome before proceeding with the actual action. ### CLI Option: --get-file-link _**Description:**_ This CLI option queries the OneDrive API and return's the WebURL for the given local file. _**Usage Example:**_ `onedrive --get-file-link 'relative/path/to/your/file.txt'` > [!IMPORTANT] > The path that you should use *must* be relative to your 'sync_dir' ### CLI Option: --get-sharepoint-drive-id _**Description:**_ This CLI option queries the OneDrive API and return's the Office 365 Drive ID for a given Office 365 SharePoint Shared Library that can then be used with 'drive_id' to sync a specific SharePoint Library. _**Usage Example:**_ `onedrive --get-sharepoint-drive-id '*'` or `onedrive --get-sharepoint-drive-id 'PointPublishing Hub Site'` ### CLI Option: --list-shared-items _**Description:**_ This CLI option lists all OneDrive Business Shared items with your account. The resulting list shows shared files and folders that you can configure this client to sync. _**Usage Example:**_ `onedrive --list-shared-items` _**Example Output:**_ ``` ... Listing available OneDrive Business Shared Items: ----------------------------------------------------------------------------------- Shared File: large_document_shared.docx Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: no_download_access.docx Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: online_access_only.txt Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: read_only.txt Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: qewrqwerwqer.txt Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: dummy_file_to_share.docx Shared By: testuser2 testuser2 (testuser2@domain.tld) ----------------------------------------------------------------------------------- Shared Folder: Sub Folder 2 Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared File: file to share.docx Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- Shared Folder: Top Folder Shared By: test user (testuser@domain.tld) ----------------------------------------------------------------------------------- ... ``` ### CLI Option: --logout _**Description:**_ This CLI option removes this clients authentication status with Microsoft OneDrive. Any further application use will require the application to be re-authenticated with Microsoft OneDrive. _**Usage Example:**_ `onedrive --logout` ### CLI Option: --modified-by _**Description:**_ This CLI option queries the OneDrive API and return's the last modified details for the given local file. _**Usage Example:**_ `onedrive --modified-by 'relative/path/to/your/file.txt'` > [!IMPORTANT] > The path that you should use *must* be relative to your 'sync_dir' ### CLI Option: --monitor | -m _**Description:**_ This CLI option controls the 'Monitor Mode' operational aspect of the client. When this option is used, the client will perform on-going syncs of data between Microsoft OneDrive and your local system. Local changes will be uploaded in near-realtime, whilst online changes will be downloaded on the next sync process. The frequency of these checks is governed by the 'monitor_interval' value. _**Usage Example:**_ `onedrive --monitor` or `onedrive -m` ### CLI Option: --print-access-token _**Description:**_ Print the current access token being used to access Microsoft OneDrive. _**Usage Example:**_ `onedrive --verbose --verbose --debug-https --print-access-token` > [!CAUTION] > Do not use this option if you do not know why you are wanting to use it. Be highly cautious of exposing this object. Change your password if you feel that you have inadvertently exposed this token. ### CLI Option: --reauth _**Description:**_ This CLI option controls the ability to re-authenticate your client with Microsoft OneDrive. _**Usage Example:**_ `onedrive --reauth` ### CLI Option: --remove-directory _**Description:**_ This CLI option allows the user to remove the specified directory path on Microsoft OneDrive without performing a sync. _**Usage Example:**_ `onedrive --remove-directory 'path/of/new/folder/structure/to/remove/'` > [!IMPORTANT] > The specified path to remove is relative to your configured 'sync_dir'. ### CLI Option: --share-password _**Description:**_ This CLI option enables the creation of a shareable file link that can only be accessed by providing the valid password. This option can only be used in conjunction with `--create-share-link` _**Usage Example:**_ `onedrive --create-share-link 'relative/path/to/your/file.txt' --share-password 'valid password'` ### CLI Option: --single-directory _**Description:**_ This CLI option controls the applications ability to sync a specific single directory. _**Usage Example:**_ `onedrive --sync --single-directory 'Data'` > [!IMPORTANT] > The path specified is relative to your configured 'sync_dir' path. If the physical local path 'Folder' to sync is `~/OneDrive/Data/Folder` then the command would be `--single-directory 'Data/Folder'`. ### CLI Option: --source-directory _**Description:**_ This CLI option specifies the 'source' portion of moving a file or folder online, without performing a sync operation. _**Usage Example:**_ `onedrive --source-directory 'path/as/source/' --destination-directory 'path/as/destination'` > [!IMPORTANT] > All specified paths are relative to your configured 'sync_dir'. ### CLI Option: --sync | -s _**Description:**_ This CLI option controls the 'Standalone Mode' operational aspect of the client. When this option is used, the client will perform a one-time sync of data between Microsoft OneDrive and your local system. _**Usage Example:**_ `onedrive --sync` or `onedrive -s` ### CLI Option: --sync-shared-files _**Description:**_ Sync OneDrive Business Shared Files to the local filesystem. _**Usage Example:**_ `onedrive --sync --sync-shared-files` > [!IMPORTANT] > To use this option you must first enable 'sync_business_shared_items' within your application configuration. Please read 'business-shared-items.md' for more information regarding this option. ### CLI Option: --verbose | -v+ _**Description:**_ This CLI option controls the verbosity of the application output. Use the option once, to have normal verbose output, use twice to have debug level application output. _**Usage Example:**_ `onedrive --sync --verbose` or `onedrive --monitor --verbose` ### CLI Option: --with-editing-perms _**Description:**_ This CLI option enables the creation of a writable shareable file link that can be provided to users to access the file that is stored on Microsoft OneDrive. This option can only be used in conjunction with `--create-share-link` _**Usage Example:**_ `onedrive --create-share-link 'relative/path/to/your/file.txt' --with-editing-perms` > [!IMPORTANT] > Placement of `--with-editing-perms` is critical. It *must* be placed after the file path as per the example above. ## Deprecated Configuration File and CLI Options The following configuration options are no longer supported: ### force_http_2 _**Description:**_ Force the use of HTTP/2 for all operations where applicable _**Deprecated Config Example:**_ `force_http_2 = "true"` _**Deprecated CLI Option:**_ `--force-http-2` _**Reason for depreciation:**_ HTTP/2 will be used by default where possible, when the OneDrive API platform does not downgrade the connection to HTTP/1.1, thus this configuration option is no longer required. ### min_notify_changes _**Description:**_ Minimum number of pending incoming changes necessary to trigger a GUI desktop notification. _**Deprecated Config Example:**_ `min_notify_changes = "50"` _**Deprecated CLI Option:**_ `--min-notify-changes '50'` _**Reason for depreciation:**_ Application has been totally re-written. When this item was introduced, it was done so to reduce spamming of all events to the GUI desktop. ### CLI Option: --synchronize _**Description:**_ Perform a synchronisation with Microsoft OneDrive _**Deprecated CLI Option:**_ `--synchronize` _**Reason for depreciation:**_ `--synchronize` has been deprecated in favour of `--sync` or `-s` ================================================ FILE: docs/application-security.md ================================================ # OneDrive Client for Linux Application Security This document details the following information: * Why is this application an 'unverified publisher'? * Application Security and Permission Scopes * How to change Permission Scopes * How to review your existing application access consent ## Why is this application an 'unverified publisher'? Publisher Verification, as per the Microsoft [process](https://learn.microsoft.com/en-us/azure/active-directory/develop/publisher-verification-overview) has actually been configured, and, actually has been verified! ### Verified Publisher Configuration Evidence As per the image below, the Azure portal shows that the 'Publisher Domain' has actually been verified: ![confirmed_verified_publisher](./images/confirmed_verified_publisher.jpg) * The 'Publisher Domain' is: https://abraunegg.github.io/ * The required 'Microsoft Identity Association' is: https://abraunegg.github.io/.well-known/microsoft-identity-association.json ## Application Security and Permission Scopes There are 2 main components regarding security for this application: * Azure Application Permissions * User Authentication Permissions Keeping this in mind, security options should follow the security principal of 'least privilege': > The principle that a security architecture should be designed so that each entity > is granted the minimum system resources and authorizations that the entity needs > to perform its function. Reference: [https://csrc.nist.gov/glossary/term/least_privilege](https://csrc.nist.gov/glossary/term/least_privilege) As such, the following API permissions are used by default: ### Default Azure Application Permissions | API / Permissions name | Type | Description | Admin consent required | |---|---|---|---| | Files.Read | Delegated | Have read-only access to user files | No | | Files.Read.All | Delegated | Have read-only access to all files user can access | No | | Sites.Read.All | Delegated | Have read-only access to all items in all site collections | No | | offline_access | Delegated | Maintain access to data you have given it access to | No | ![default_authentication_scopes](./images/default_authentication_scopes.jpg) ### Default User Authentication Permissions When a user authenticates with Microsoft OneDrive, additional account permissions are provided by service to give the user specific access to their data. These are delegated permissions provided by the platform: | API / Permissions name | Type | Description | Admin consent required | |---|---|---|---| | Files.ReadWrite | Delegated | Have full access to user files | No | | Files.ReadWrite.All | Delegated | Have full access to all files user can access | No | | Sites.ReadWrite.All | Delegated | Have full access to all items in all site collections | No | | offline_access | Delegated | Maintain access to data you have given it access to | No | When these delegated API permissions are combined, these provide the effective authentication scope for the OneDrive Client for Linux to access your data. The resulting effective 'default' permissions will be: | API / Permissions name | Type | Description | Admin consent required | |---|---|---|---| | Files.ReadWrite | Delegated | Have full access to user files | No | | Files.ReadWrite.All | Delegated | Have full access to all files user can access | No | | Sites.ReadWrite.All | Delegated | Have full access to all items in all site collections | No | | offline_access | Delegated | Maintain access to data you have given it access to | No | These 'default' permissions will allow the OneDrive Client for Linux to read, write and delete data associated with your OneDrive Account. ## How are the Authentication Scopes used? When using the OneDrive Client for Linux, the above authentication scopes will be presented to the Microsoft Authentication Service (login.microsoftonline.com), where the service will validate the request and provide an applicable token to access Microsoft OneDrive with. This can be illustrated as the following: ![Linux Authentication to Microsoft OneDrive](./puml/onedrive_linux_authentication.png) This is similar to the Microsoft Windows OneDrive Client: ![Windows Authentication to Microsoft OneDrive](./puml/onedrive_windows_authentication.png) In a business setting, IT staff who need to authorise the use of the OneDrive Client for Linux in their environment can be assured of its safety. The primary concern for IT staff should be securing the device running the OneDrive Client for Linux. Unlike in a corporate environment where Windows devices are secured through Active Directory and Group Policy Objects (GPOs) to protect corporate data on the device, it is beyond the responsibility of this client to manage security on Linux devices. ## Configuring read-only access to your OneDrive data In some situations, it may be desirable to configure the OneDrive Client for Linux totally in read-only operation. To change the application to 'read-only' access, add the following to your configuration file: ```text read_only_auth_scope = "true" ``` This will change the user authentication scope request to use read-only access. > [!IMPORTANT] > When changing this value, you *must* re-authenticate the client using the `--reauth` option to utilise the change in authentication scopes. When using read-only authentication scopes, the uploading of any data or local change to OneDrive will fail with the following error: ``` 2022-Aug-06 13:16:45.3349625 ERROR: Microsoft OneDrive API returned an error with the following message: 2022-Aug-06 13:16:45.3351661 Error Message: HTTP request returned status code 403 (Forbidden) 2022-Aug-06 13:16:45.3352467 Error Reason: Access denied 2022-Aug-06 13:16:45.3352838 Error Timestamp: 2022-06-12T13:16:45 2022-Aug-06 13:16:45.3353171 API Request ID: ``` As such, it is also advisable for you to add the following to your configuration file so that 'uploads' are prevented: ```text download_only = "true" ``` > [!IMPORTANT] > Additionally when using 'read_only_auth_scope' you also will need to remove your existing application access consent otherwise old authentication consent will be valid and will be used. This will mean the application will technically have the consent to upload data. See below on how to remove your prior application consent. ## Reviewing your existing application access consent To review your existing application access consent, you need to access the following URL: https://account.live.com/consent/Manage From here, you are able to review what applications have been given what access to your data, and remove application access as required. ================================================ FILE: docs/build-rpm-howto.md ================================================ # RPM Package Build Process The instructions below have been tested on the following systems: * CentOS Stream release 9 These instructions should also be applicable for RedHat & Fedora platforms, or any other RedHat RPM based distribution. ## Prepare Package Development Environment ### Install Development Dependencies Install the following dependencies on your build system: ```text sudo yum groupinstall -y 'Development Tools' sudo yum install -y libcurl-devel sudo yum install -y sqlite-devel sudo yum install -y libnotify-devel sudo yum install -y dbus-devel sudo yum install -y wget mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} ``` ### Install DMD Compiler for Linux Install the latest DMD Compiler for Linux from https://dlang.org/download.html using the Fedora/CentOS x86_64 link. Illustrated below is the installation using the minimum supported compiler. You should always install the latest version of the compiler for your platform when manually building an RPM. ```text sudo yum install -y https://downloads.dlang.org/releases/2.x/2.091.1/dmd-2.091.1-0.fedora.x86_64.rpm ``` ## Build RPM from spec file using the DMD Compiler Build the RPM from the provided spec file: ```text wget https://github.com/abraunegg/onedrive/archive/refs/tags/v2.5.6.tar.gz -O ~/rpmbuild/SOURCES/v2.5.6.tar.gz wget https://raw.githubusercontent.com/abraunegg/onedrive/master/contrib/spec/onedrive.spec.in -O ~/rpmbuild/SPECS/onedrive.spec rpmbuild -ba ~/rpmbuild/SPECS/onedrive.spec --define 'dcompiler dmd' ``` ### RPM Build Example Results Below are example output results of building, installing and running the RPM package on the respective platforms: #### CentOS Stream release 9 RPM Build Process ```text setting SOURCE_DATE_EPOCH=1749081600 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.ZhVuOR + umask 022 + cd /home/alex/rpmbuild/BUILD + cd /home/alex/rpmbuild/BUILD + rm -rf onedrive-2.5.6 + /usr/bin/tar -xof - + /usr/bin/gzip -dc /home/alex/rpmbuild/SOURCES/v2.5.6.tar.gz + STATUS=0 + '[' 0 -ne 0 ']' + cd onedrive-2.5.6 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + RPM_EC=0 ++ jobs -p + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.b9tkxJ + umask 022 + cd /home/alex/rpmbuild/BUILD + cd onedrive-2.5.6 + CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' + export CFLAGS + CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' + export CXXFLAGS + FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules' + export FFLAGS + FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules' + export FCFLAGS + LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' + export LDFLAGS + LT_SYS_LIBRARY_PATH=/usr/lib64: + export LT_SYS_LIBRARY_PATH + CC=gcc + export CC + CXX=g++ + export CXX + '[' '-flto=auto -ffat-lto-objectsx' '!=' x ']' ++ find . -type f -name configure -print + for file in $(find . -type f -name configure -print) + /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\) = /__attribute__ ((used)) char (*f) () = /g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\);/__attribute__ ((used)) char (*f) ();/g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed -r --in-place=.backup 's/^char \$2 \(\);/__attribute__ ((used)) char \$2 ();/g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed --in-place=.backup '1{$!N;$!N};$!N;s/int x = 1;\nint y = 0;\nint z;\nint nan;/volatile int x = 1; volatile int y = 0; volatile int z, nan;/;P;D' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed --in-place=.backup 's#^lt_cv_sys_global_symbol_to_cdecl=.*#lt_cv_sys_global_symbol_to_cdecl="sed -n -e '\''s/^T .* \\(.*\\)$/extern int \\1();/p'\'' -e '\''s/^$symcode* .* \\(.*\\)$/extern char \\1;/p'\''"#' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + '[' 1 = 1 ']' +++ dirname ./configure ++ find . -name config.guess -o -name config.sub + '[' 1 = 1 ']' + '[' x '!=' 'x-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' ']' ++ find . -name ltmain.sh + ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-debug --enable-notifications configure: WARNING: unrecognized options: --disable-dependency-tracking checking for a BSD-compatible install... /usr/bin/install -c checking for x86_64-redhat-linux-gnu-pkg-config... /usr/bin/x86_64-redhat-linux-gnu-pkg-config checking pkg-config is at least version 0.9.0... yes checking for dmd... dmd checking version of D compiler... 2.091.1 checking for curl... yes checking for sqlite... yes checking whether to enable dbus support... yes (on Linux) checking for dbus... yes checking for notify... yes configure: creating ./config.status config.status: creating Makefile config.status: creating contrib/pacman/PKGBUILD config.status: creating contrib/spec/onedrive.spec config.status: creating onedrive.1 config.status: creating contrib/systemd/onedrive.service config.status: creating contrib/systemd/onedrive@.service configure: WARNING: unrecognized options: --disable-dependency-tracking + make if [ -f .git/HEAD ] ; then \ git describe --tags > version ; \ else \ echo v2.5.6 > version ; \ fi dmd -J. -version=NoPragma -version=NoGdk -version=Notifications -w -g -debug -gs src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/webhook.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/monitor.d src/arsd/cgi.d src/xattr.d src/intune.d src/notifications/notify.d src/notifications/dnotify.d -L-lcurl -L-lsqlite3 -L-ldbus-1 -L-lnotify -L-lgdk_pixbuf-2.0 -L-lgio-2.0 -L-lgobject-2.0 -L-lglib-2.0 -L-ldl -ofonedrive + RPM_EC=0 ++ jobs -p + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.Pwy2mS + umask 022 + cd /home/alex/rpmbuild/BUILD + '[' /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 '!=' / ']' + rm -rf /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 ++ dirname /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 + mkdir -p /home/alex/rpmbuild/BUILDROOT + mkdir /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 + cd onedrive-2.5.6 + /usr/bin/make install DESTDIR=/home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 'INSTALL=/usr/bin/install -p' PREFIX=/home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/bin /usr/bin/install -p onedrive /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/bin/onedrive mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/man/man1 /usr/bin/install -p -m 0644 onedrive.1 /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/man/man1/onedrive.1 mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/etc/logrotate.d /usr/bin/install -p -m 0644 contrib/logrotate/onedrive.logrotate /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/etc/logrotate.d/onedrive mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive for file in readme.md config LICENSE changelog.md docs/advanced-usage.md docs/application-config-options.md docs/application-security.md docs/business-shared-items.md docs/client-architecture.md docs/contributing.md docs/docker.md docs/install.md docs/national-cloud-deployments.md docs/podman.md docs/privacy-policy.md docs/sharepoint-libraries.md docs/terms-of-service.md docs/ubuntu-package-install.md docs/usage.md docs/known-issues.md docs/webhooks.md; do \ /usr/bin/install -p -m 0644 $file /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive; \ done mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/user mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/system /usr/bin/install -p -m 0644 contrib/systemd/onedrive@.service /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/system /usr/bin/install -p -m 0644 contrib/systemd/onedrive.service /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/system + install -D -m 0644 contrib/systemd/onedrive.service /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/system/onedrive.service + install -D -m 0644 contrib/systemd/onedrive@.service /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/lib/systemd/system/onedrive@.service + /usr/lib/rpm/check-buildroot + /usr/lib/rpm/redhat/brp-ldconfig + /usr/lib/rpm/brp-compress + /usr/lib/rpm/brp-strip /usr/bin/strip + /usr/lib/rpm/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip + /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0 + /usr/lib/rpm/brp-python-hardlink + /usr/lib/rpm/redhat/brp-mangle-shebangs Processing files: onedrive-2.5.6-1.el9.x86_64 Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.2YAn9k + umask 022 + cd /home/alex/rpmbuild/BUILD + cd onedrive-2.5.6 + DOCDIR=/home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + export LC_ALL=C + LC_ALL=C + export DOCDIR + /usr/bin/mkdir -p /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + cp -pr readme.md /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + cp -pr LICENSE /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + cp -pr changelog.md /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + cp -pr docs/advanced-usage.md docs/application-config-options.md docs/application-security.md docs/build-rpm-howto.md docs/business-shared-items.md docs/client-architecture.md docs/contributing.md docs/docker.md docs/install.md docs/known-issues.md docs/national-cloud-deployments.md docs/podman.md docs/privacy-policy.md docs/sharepoint-libraries.md docs/terms-of-service.md docs/ubuntu-package-install.md docs/usage.md docs/webhooks.md /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + cp -pr config /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64/usr/share/doc/onedrive + RPM_EC=0 ++ jobs -p + exit 0 Provides: config(onedrive) = 2.5.6-1.el9 onedrive = 2.5.6-1.el9 onedrive(x86-64) = 2.5.6-1.el9 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Requires(post): systemd Requires(preun): systemd Requires(postun): systemd Requires: ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.15)(64bit) libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) libc.so.6(GLIBC_2.3.4)(64bit) libc.so.6(GLIBC_2.32)(64bit) libc.so.6(GLIBC_2.33)(64bit) libc.so.6(GLIBC_2.34)(64bit) libc.so.6(GLIBC_2.4)(64bit) libc.so.6(GLIBC_2.6)(64bit) libc.so.6(GLIBC_2.7)(64bit) libc.so.6(GLIBC_2.8)(64bit) libc.so.6(GLIBC_2.9)(64bit) libcurl.so.4()(64bit) libdbus-1.so.3()(64bit) libdbus-1.so.3(LIBDBUS_1_3)(64bit) libgcc_s.so.1()(64bit) libgcc_s.so.1(GCC_3.0)(64bit) libgcc_s.so.1(GCC_4.2.0)(64bit) libgdk_pixbuf-2.0.so.0()(64bit) libgio-2.0.so.0()(64bit) libglib-2.0.so.0()(64bit) libgobject-2.0.so.0()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libnotify.so.4()(64bit) libsqlite3.so.0()(64bit) rtld(GNU_HASH) Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/alex/rpmbuild/BUILDROOT/onedrive-2.5.6-1.el9.x86_64 Wrote: /home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm Wrote: /home/alex/rpmbuild/RPMS/x86_64/onedrive-2.5.6-1.el9.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.tGKXPN + umask 022 + cd /home/alex/rpmbuild/BUILD + cd onedrive-2.5.6 + RPM_EC=0 ++ jobs -p + exit 0 ``` #### CentOS Stream release 9 RPM Package Install Process ```text [alex@centos9stream ~]$ sudo yum -y install /home/alex/rpmbuild/RPMS/x86_64/onedrive-2.5.6-1.el9.x86_64.rpm [sudo] password for alex: Last metadata expiration check: 1:21:53 ago on Tue 10 Jun 2025 06:41:27. Dependencies resolved. ========================================================================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================================================================== Installing: onedrive x86_64 2.5.6-1.el9 @commandline 1.6 M Transaction Summary ========================================================================================================================================================================================== Install 1 Package Total size: 1.6 M Installed size: 8.3 M Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : onedrive-2.5.6-1.el9.x86_64 1/1 Running scriptlet: onedrive-2.5.6-1.el9.x86_64 1/1 Verifying : onedrive-2.5.6-1.el9.x86_64 1/1 Installed: onedrive-2.5.6-1.el9.x86_64 Complete! [alex@centos9stream ~]$ [alex@centos9stream ~]$ onedrive --version onedrive v2.5.6 [alex@centos9stream ~]$ onedrive --display-config WARNING: Configured 'threads = 8' exceeds available CPU cores (1). Capping to 'threads' to 1. Application version = onedrive v2.5.6 Compiled with = DMD 2091 Curl version = libcurl/7.76.1 OpenSSL/3.5.0 zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.10.4/openssl/zlib nghttp2/1.43.0 User Application Config path = /home/alex/.config/onedrive System Application Config path = /etc/onedrive Applicable Application 'config' location = /home/alex/.config/onedrive/config Configuration file found in config location = false - using application defaults Applicable 'sync_list' location = /home/alex/.config/onedrive/sync_list Applicable 'items.sqlite3' location = /home/alex/.config/onedrive/items.sqlite3 Config option 'drive_id' = Config option 'sync_dir' = ~/OneDrive Config option 'use_intune_sso' = false Config option 'use_device_auth' = false Config option 'enable_logging' = false Config option 'log_dir' = /var/log/onedrive Config option 'disable_notifications' = false Config option 'skip_dir' = Config option 'skip_dir_strict_match' = false Config option 'skip_file' = ~*|.~*|*.tmp|*.swp|*.partial Config option 'skip_dotfiles' = false Config option 'skip_symlinks' = false Config option 'monitor_interval' = 300 Config option 'monitor_log_frequency' = 12 Config option 'monitor_fullscan_frequency' = 12 Config option 'read_only_auth_scope' = false Config option 'dry_run' = false Config option 'upload_only' = false Config option 'download_only' = false Config option 'local_first' = false Config option 'check_nosync' = false Config option 'check_nomount' = false Config option 'resync' = false Config option 'resync_auth' = false Config option 'cleanup_local_files' = false Config option 'disable_permission_set' = false Config option 'transfer_order' = default Config option 'classify_as_big_delete' = 1000 Config option 'disable_upload_validation' = false Config option 'disable_download_validation' = false Config option 'bypass_data_preservation' = false Config option 'no_remote_delete' = false Config option 'remove_source_files' = false Config option 'sync_dir_permissions' = 700 Config option 'sync_file_permissions' = 600 Config option 'space_reservation' = 52428800 Config option 'permanent_delete' = false Config option 'write_xattr_data' = false Config option 'application_id' = d50ca740-c83f-4d1b-b616-12c519384f0c Config option 'azure_ad_endpoint' = Config option 'azure_tenant_id' = Config option 'user_agent' = ISV|abraunegg|OneDrive Client for Linux/v2.5.6 Config option 'force_http_11' = false Config option 'debug_https' = false Config option 'rate_limit' = 0 Config option 'operation_timeout' = 3600 Config option 'dns_timeout' = 60 Config option 'connect_timeout' = 10 Config option 'data_timeout' = 60 Config option 'ip_protocol_version' = 0 Config option 'threads' = 1 Config option 'max_curl_idle' = 120 Environment var 'XDG_RUNTIME_DIR' = true Environment var 'DBUS_SESSION_BUS_ADDRESS' = true Config option 'notify_file_actions' = false Config option 'use_recycle_bin' = false Config option 'recycle_bin_path' = /home/alex/.local/share/Trash/ Selective sync 'sync_list' configured = false Config option 'sync_business_shared_items' = false Config option 'webhook_enabled' = false ``` ## Build RPM from SRPM using mock ### Install mock on your platform Use the following installation instructions to install 'mock' on your platform: ```text sudo yum install epel-release sudo yum install mock sudo yum install -y wget mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} ``` ### Configure mock Add your user to the mock group: ```text sudo usermod -a -G mock $USER ``` > [!NOTE] > Log out and back in for the group membership changes to take effect. ### Build a Source RPM (SRPM) file Build the SRPM from the provided spec file: ```text wget https://github.com/abraunegg/onedrive/archive/refs/tags/v2.5.6.tar.gz -O ~/rpmbuild/SOURCES/v2.5.6.tar.gz wget https://raw.githubusercontent.com/abraunegg/onedrive/master/contrib/spec/onedrive.spec.in -O ~/rpmbuild/SPECS/onedrive.spec rpmbuild -bs ~/rpmbuild/SPECS/onedrive.spec ``` > [!NOTE] > This will build a SRPM to the following location: `/home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm` > > This SRPM will be used in the examples below: ### Build Fedora 42 RPM using mock ```text [alex@centos9stream ~]$ mock -r fedora-42-x86_64 /home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm INFO: mock.py version 6.2 starting (python version = 3.9.21, NVR = mock-6.2-1.el9), args: /usr/libexec/mock/mock -r fedora-42-x86_64 /home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm Start(bootstrap): init plugins INFO: selinux enabled Finish(bootstrap): init plugins Start: init plugins INFO: selinux enabled Finish: init plugins INFO: Signal handler active Start: run INFO: Start(/home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm) Config(fedora-42-x86_64) Start: clean chroot Finish: clean chroot Mock Version: 6.2 INFO: Mock Version: 6.2 Start(bootstrap): chroot init INFO: calling preinit hooks INFO: enabled root cache INFO: enabled package manager cache Start(bootstrap): cleaning package manager metadata Finish(bootstrap): cleaning package manager metadata INFO: Package manager dnf5 detected and used (fallback) Finish(bootstrap): chroot init Start: chroot init INFO: calling preinit hooks INFO: enabled root cache Start: unpacking root cache Finish: unpacking root cache INFO: enabled package manager cache Start: cleaning package manager metadata Finish: cleaning package manager metadata INFO: enabled HW Info plugin INFO: Package manager dnf5 detected and used (direct choice) INFO: Buildroot is handled by package management downloaded with a bootstrap image: rpm-4.20.1-1.fc42.x86_64 rpm-sequoia-1.7.0-5.fc42.x86_64 dnf5-5.2.13.1-1.fc42.x86_64 dnf5-plugins-5.2.13.1-1.fc42.x86_64 Start: dnf5 update Updating and loading repositories: updates 100% | 5.5 KiB/s | 5.6 KiB | 00m01s fedora 100% | 5.8 KiB/s | 4.2 KiB | 00m01s Repositories loaded. Nothing to do. Finish: dnf5 update Finish: chroot init Start: build phase for onedrive-2.5.6-1.el9.src.rpm Start: build setup for onedrive-2.5.6-1.el9.src.rpm Building target platforms: x86_64 Building for target x86_64 setting SOURCE_DATE_EPOCH=1749081600 Wrote: /builddir/build/SRPMS/onedrive-2.5.6-1.fc42.src.rpm Updating and loading repositories: updates 100% | 16.5 KiB/s | 5.6 KiB | 00m00s fedora 100% | 8.3 KiB/s | 4.2 KiB | 00m01s Repositories loaded. Package Arch Version Repository Size Installing: dbus-devel x86_64 1:1.16.0-3.fc42 fedora 131.7 KiB ldc x86_64 1:1.40.0-3.fc42 fedora 27.3 MiB libcurl-devel x86_64 8.11.1-4.fc42 fedora 1.3 MiB sqlite-devel x86_64 3.47.2-2.fc42 fedora 673.4 KiB Installing dependencies: annobin-docs noarch 12.94-1.fc42 updates 98.9 KiB annobin-plugin-gcc x86_64 12.94-1.fc42 updates 993.5 KiB brotli x86_64 1.1.0-6.fc42 fedora 31.6 KiB brotli-devel x86_64 1.1.0-6.fc42 fedora 65.6 KiB cmake-filesystem x86_64 3.31.6-2.fc42 fedora 0.0 B cpp x86_64 15.1.1-2.fc42 updates 37.9 MiB dbus-libs x86_64 1:1.16.0-3.fc42 fedora 349.5 KiB gcc x86_64 15.1.1-2.fc42 updates 111.1 MiB gcc-plugin-annobin x86_64 15.1.1-2.fc42 updates 57.1 KiB glibc-devel x86_64 2.41-5.fc42 updates 2.3 MiB kernel-headers x86_64 6.14.3-300.fc42 updates 6.5 MiB keyutils-libs-devel x86_64 1.6.3-5.fc42 fedora 48.2 KiB krb5-devel x86_64 1.21.3-6.fc42 updates 705.9 KiB ldc-libs x86_64 1:1.40.0-3.fc42 fedora 11.6 MiB libcom_err-devel x86_64 1.47.2-3.fc42 fedora 16.7 KiB libedit x86_64 3.1-55.20250104cvs.fc42 fedora 244.1 KiB libidn2-devel x86_64 2.3.8-1.fc42 fedora 149.1 KiB libkadm5 x86_64 1.21.3-6.fc42 updates 213.9 KiB libmpc x86_64 1.3.1-7.fc42 fedora 164.5 KiB libnghttp2-devel x86_64 1.64.0-3.fc42 fedora 295.4 KiB libpsl-devel x86_64 0.21.5-5.fc42 fedora 110.3 KiB libselinux-devel x86_64 3.8-2.fc42 updates 126.8 KiB libsepol-devel x86_64 3.8-1.fc42 fedora 120.8 KiB libssh-devel x86_64 0.11.1-4.fc42 fedora 178.0 KiB libverto-devel x86_64 0.3.2-10.fc42 fedora 25.7 KiB libxcrypt-devel x86_64 4.4.38-7.fc42 updates 30.8 KiB llvm19-filesystem x86_64 19.1.7-13.fc42 updates 0.0 B llvm19-libs x86_64 19.1.7-13.fc42 updates 124.0 MiB make x86_64 1:4.4.1-10.fc42 fedora 1.8 MiB openssl-devel x86_64 1:3.2.4-3.fc42 fedora 4.3 MiB pcre2-devel x86_64 10.45-1.fc42 fedora 2.1 MiB pcre2-utf16 x86_64 10.45-1.fc42 fedora 626.3 KiB pcre2-utf32 x86_64 10.45-1.fc42 fedora 598.2 KiB publicsuffix-list noarch 20250116-1.fc42 fedora 329.8 KiB sqlite x86_64 3.47.2-2.fc42 fedora 1.8 MiB systemd-devel x86_64 257.6-1.fc42 updates 612.3 KiB systemd-rpm-macros noarch 257.6-1.fc42 updates 10.7 KiB xml-common noarch 0.6.3-66.fc42 fedora 78.4 KiB zlib-ng-compat-devel x86_64 2.2.4-3.fc42 fedora 107.0 KiB Transaction Summary: Installing: 43 packages Total size of inbound packages is 103 MiB. Need to download 0 B. After this operation, 339 MiB extra will be used (install 339 MiB, remove 0 B). [ 1/43] ldc-1:1.40.0-3.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 2/43] dbus-devel-1:1.16.0-3.fc42.x86_ 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 3/43] libcurl-devel-0:8.11.1-4.fc42.x 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 4/43] sqlite-devel-0:3.47.2-2.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 5/43] ldc-libs-1:1.40.0-3.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 6/43] cmake-filesystem-0:3.31.6-2.fc4 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 7/43] dbus-libs-1:1.16.0-3.fc42.x86_6 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 8/43] xml-common-0:0.6.3-66.fc42.noar 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [ 9/43] sqlite-0:3.47.2-2.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [10/43] krb5-devel-0:1.21.3-6.fc42.x86_ 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [11/43] libkadm5-0:1.21.3-6.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [12/43] brotli-devel-0:1.1.0-6.fc42.x86 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [13/43] brotli-0:1.1.0-6.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [14/43] libidn2-devel-0:2.3.8-1.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [15/43] libnghttp2-devel-0:1.64.0-3.fc4 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [16/43] libpsl-devel-0:0.21.5-5.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [17/43] publicsuffix-list-0:20250116-1. 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [18/43] libssh-devel-0:0.11.1-4.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [19/43] openssl-devel-1:3.2.4-3.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [20/43] zlib-ng-compat-devel-0:2.2.4-3. 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [21/43] gcc-0:15.1.1-2.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [22/43] cpp-0:15.1.1-2.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [23/43] libmpc-0:1.3.1-7.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [24/43] make-1:4.4.1-10.fc42.x86_64 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [25/43] llvm19-libs-0:19.1.7-13.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [26/43] llvm19-filesystem-0:19.1.7-13.f 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [27/43] libedit-0:3.1-55.20250104cvs.fc 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [28/43] systemd-devel-0:257.6-1.fc42.x8 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [29/43] libselinux-devel-0:3.8-2.fc42.x 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [30/43] libsepol-devel-0:3.8-1.fc42.x86 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [31/43] keyutils-libs-devel-0:1.6.3-5.f 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [32/43] libcom_err-devel-0:1.47.2-3.fc4 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [33/43] libverto-devel-0:0.3.2-10.fc42. 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [34/43] glibc-devel-0:2.41-5.fc42.x86_6 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [35/43] pcre2-devel-0:10.45-1.fc42.x86_ 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [36/43] pcre2-utf16-0:10.45-1.fc42.x86_ 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [37/43] pcre2-utf32-0:10.45-1.fc42.x86_ 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [38/43] kernel-headers-0:6.14.3-300.fc4 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [39/43] libxcrypt-devel-0:4.4.38-7.fc42 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [40/43] gcc-plugin-annobin-0:15.1.1-2.f 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [41/43] systemd-rpm-macros-0:257.6-1.fc 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [42/43] annobin-plugin-gcc-0:12.94-1.fc 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded [43/43] annobin-docs-0:12.94-1.fc42.noa 100% | 0.0 B/s | 0.0 B | 00m00s >>> Already downloaded -------------------------------------------------------------------------------- [43/43] Total 100% | 0.0 B/s | 0.0 B | 00m00s Running transaction [ 1/45] Verify package files 100% | 29.0 B/s | 43.0 B | 00m01s [ 2/45] Prepare transaction 100% | 154.0 B/s | 43.0 B | 00m00s [ 3/45] Installing cmake-filesystem-0:3 100% | 583.8 KiB/s | 7.6 KiB | 00m00s [ 4/45] Installing libmpc-0:1.3.1-7.fc4 100% | 23.2 MiB/s | 166.1 KiB | 00m00s [ 5/45] Installing cpp-0:15.1.1-2.fc42. 100% | 120.6 MiB/s | 37.9 MiB | 00m00s [ 6/45] Installing libssh-devel-0:0.11. 100% | 19.6 MiB/s | 180.5 KiB | 00m00s [ 7/45] Installing zlib-ng-compat-devel 100% | 15.1 MiB/s | 108.5 KiB | 00m00s [ 8/45] Installing annobin-docs-0:12.94 100% | 10.9 MiB/s | 100.0 KiB | 00m00s [ 9/45] Installing kernel-headers-0:6.1 100% | 36.6 MiB/s | 6.7 MiB | 00m00s [10/45] Installing libxcrypt-devel-0:4. 100% | 2.9 MiB/s | 33.1 KiB | 00m00s [11/45] Installing glibc-devel-0:2.41-5 100% | 15.3 MiB/s | 2.3 MiB | 00m00s [12/45] Installing pcre2-utf32-0:10.45- 100% | 18.3 MiB/s | 599.1 KiB | 00m00s [13/45] Installing pcre2-utf16-0:10.45- 100% | 30.6 MiB/s | 627.1 KiB | 00m00s [14/45] Installing pcre2-devel-0:10.45- 100% | 33.8 MiB/s | 2.1 MiB | 00m00s [15/45] Installing libverto-devel-0:0.3 100% | 5.1 MiB/s | 26.4 KiB | 00m00s [16/45] Installing libcom_err-devel-0:1 100% | 761.4 KiB/s | 18.3 KiB | 00m00s [17/45] Installing keyutils-libs-devel- 100% | 5.4 MiB/s | 55.2 KiB | 00m00s [18/45] Installing libsepol-devel-0:3.8 100% | 9.6 MiB/s | 128.3 KiB | 00m00s [19/45] Installing libselinux-devel-0:3 100% | 4.2 MiB/s | 161.6 KiB | 00m00s [20/45] Installing systemd-devel-0:257. 100% | 6.2 MiB/s | 744.1 KiB | 00m00s [21/45] Installing libedit-0:3.1-55.202 100% | 30.0 MiB/s | 245.8 KiB | 00m00s [22/45] Installing llvm19-filesystem-0: 100% | 264.6 KiB/s | 1.1 KiB | 00m00s [23/45] Installing llvm19-libs-0:19.1.7 100% | 137.8 MiB/s | 124.0 MiB | 00m01s [24/45] Installing make-1:4.4.1-10.fc42 100% | 37.5 MiB/s | 1.8 MiB | 00m00s [25/45] Installing gcc-0:15.1.1-2.fc42. 100% | 131.7 MiB/s | 111.2 MiB | 00m01s [26/45] Installing openssl-devel-1:3.2. 100% | 9.0 MiB/s | 5.2 MiB | 00m01s [27/45] Installing publicsuffix-list-0: 100% | 53.8 MiB/s | 330.8 KiB | 00m00s [28/45] Installing libpsl-devel-0:0.21. 100% | 13.9 MiB/s | 113.6 KiB | 00m00s [29/45] Installing libnghttp2-devel-0:1 100% | 48.3 MiB/s | 296.5 KiB | 00m00s [30/45] Installing libidn2-devel-0:2.3. 100% | 11.8 MiB/s | 156.7 KiB | 00m00s [31/45] Installing brotli-0:1.1.0-6.fc4 100% | 1.3 MiB/s | 32.3 KiB | 00m00s [32/45] Installing brotli-devel-0:1.1.0 100% | 8.3 MiB/s | 68.0 KiB | 00m00s [33/45] Installing libkadm5-0:1.21.3-6. 100% | 26.4 MiB/s | 215.9 KiB | 00m00s [34/45] Installing krb5-devel-0:1.21.3- 100% | 18.4 MiB/s | 715.2 KiB | 00m00s [35/45] Installing sqlite-0:3.47.2-2.fc 100% | 41.5 MiB/s | 1.8 MiB | 00m00s [36/45] Installing xml-common-0:0.6.3-6 100% | 9.9 MiB/s | 81.1 KiB | 00m00s [37/45] Installing dbus-libs-1:1.16.0-3 100% | 42.8 MiB/s | 350.6 KiB | 00m00s [38/45] Installing ldc-libs-1:1.40.0-3. 100% | 85.7 MiB/s | 11.6 MiB | 00m00s [39/45] Installing ldc-1:1.40.0-3.fc42. 100% | 83.0 MiB/s | 27.5 MiB | 00m00s [40/45] Installing dbus-devel-1:1.16.0- 100% | 13.3 MiB/s | 136.5 KiB | 00m00s [41/45] Installing sqlite-devel-0:3.47. 100% | 54.9 MiB/s | 674.1 KiB | 00m00s [42/45] Installing libcurl-devel-0:8.11 100% | 3.2 MiB/s | 1.4 MiB | 00m00s [43/45] Installing gcc-plugin-annobin-0 100% | 1.1 MiB/s | 58.8 KiB | 00m00s [44/45] Installing annobin-plugin-gcc-0 100% | 14.1 MiB/s | 995.1 KiB | 00m00s [45/45] Installing systemd-rpm-macros-0 100% | 2.9 KiB/s | 11.3 KiB | 00m04s Complete! Finish: build setup for onedrive-2.5.6-1.el9.src.rpm Start: rpmbuild onedrive-2.5.6-1.el9.src.rpm Start: Outputting list of installed packages Finish: Outputting list of installed packages Building target platforms: x86_64 Building for target x86_64 setting SOURCE_DATE_EPOCH=1749081600 Executing(%mkbuilddir): /bin/sh -e /var/tmp/rpm-tmp.ApSQdT Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.u4DE7z + umask 022 + cd /builddir/build/BUILD/onedrive-2.5.6-build + cd /builddir/build/BUILD/onedrive-2.5.6-build + rm -rf onedrive-2.5.6 + /usr/lib/rpm/rpmuncompress -x /builddir/build/SOURCES/v2.5.6.tar.gz + STATUS=0 + '[' 0 -ne 0 ']' + cd onedrive-2.5.6 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + RPM_EC=0 ++ jobs -p + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.XgQE0g + umask 022 + cd /builddir/build/BUILD/onedrive-2.5.6-build + CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CFLAGS + CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CXXFLAGS + FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FFLAGS + FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FCFLAGS + VALAFLAGS=-g + export VALAFLAGS + RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes --cap-lints=warn' + export RUSTFLAGS + LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes ' + export LDFLAGS + LT_SYS_LIBRARY_PATH=/usr/lib64: + export LT_SYS_LIBRARY_PATH + CC=gcc + export CC + CXX=g++ + export CXX + cd onedrive-2.5.6 + CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CFLAGS + CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CXXFLAGS + FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FFLAGS + FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FCFLAGS + VALAFLAGS=-g + export VALAFLAGS + RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes --cap-lints=warn' + export RUSTFLAGS + LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes ' + export LDFLAGS + LT_SYS_LIBRARY_PATH=/usr/lib64: + export LT_SYS_LIBRARY_PATH + CC=gcc + export CC + CXX=g++ + export CXX + '[' '-flto=auto -ffat-lto-objectsx' '!=' x ']' ++ find . -type f -name configure -print + for file in $(find . -type f -name configure -print) + /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\) = /__attribute__ ((used)) char (*f) () = /g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed -r --in-place=.backup 's/^char \(\*f\) \(\);/__attribute__ ((used)) char (*f) ();/g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed -r --in-place=.backup 's/^char \$2 \(\);/__attribute__ ((used)) char \$2 ();/g' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed --in-place=.backup '1{$!N;$!N};$!N;s/int x = 1;\nint y = 0;\nint z;\nint nan;/volatile int x = 1; volatile int y = 0; volatile int z, nan;/;P;D' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + /usr/bin/sed -r --in-place=.backup '/lt_cv_sys_global_symbol_to_cdecl=/s#(".*"|'\''.*'\'')#"sed -n -e '\''s/^T .* \\(.*\\)$/extern int \\1();/p'\'' -e '\''s/^$symcode* .* \\(.*\\)$/extern char \\1;/p'\''"#' ./configure + diff -u ./configure.backup ./configure + mv ./configure.backup ./configure + '[' 1 = 1 ']' +++ dirname ./configure ++ find . -name config.guess -o -name config.sub + '[' 1 = 1 ']' + '[' x '!=' 'x-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' ']' ++ find . -name ltmain.sh ++ grep -q runstatedir=DIR ./configure + ./configure --build=x86_64-redhat-linux --host=x86_64-redhat-linux --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/bin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-debug --enable-notifications configure: WARNING: unrecognized options: --disable-dependency-tracking checking for a BSD-compatible install... /usr/bin/install -c checking for x86_64-redhat-linux-pkg-config... no checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for dmd... no checking for ldmd2... ldmd2 checking version of D compiler... 1.40.0 checking for curl... yes checking for sqlite... yes checking whether to enable dbus support... yes (on Linux) checking for dbus... yes checking for notify... no configure: creating ./config.status config.status: creating Makefile config.status: creating contrib/pacman/PKGBUILD config.status: creating contrib/spec/onedrive.spec config.status: creating onedrive.1 config.status: creating contrib/systemd/onedrive.service config.status: creating contrib/systemd/onedrive@.service configure: WARNING: unrecognized options: --disable-dependency-tracking + make if [ -f .git/HEAD ] ; then \ git describe --tags > version ; \ else \ echo v2.5.6 > version ; \ fi ldmd2 -J. -w -g -debug -gs src/main.d src/config.d src/log.d src/util.d src/qxor.d src/curlEngine.d src/onedrive.d src/webhook.d src/sync.d src/itemdb.d src/sqlite.d src/clientSideFiltering.d src/monitor.d src/arsd/cgi.d src/xattr.d src/intune.d -L-lcurl -L-lsqlite3 -L-L/usr/lib64/pkgconfig/../../lib64 -L-ldbus-1 -L-ldl -ofonedrive + RPM_EC=0 ++ jobs -p + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jDHAO4 + umask 022 + cd /builddir/build/BUILD/onedrive-2.5.6-build + '[' /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT '!=' / ']' + rm -rf /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT ++ dirname /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT + mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build + mkdir /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT + CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CFLAGS + CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' + export CXXFLAGS + FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FFLAGS + FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -I/usr/lib64/gfortran/modules ' + export FCFLAGS + VALAFLAGS=-g + export VALAFLAGS + RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes --cap-lints=warn' + export RUSTFLAGS + LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes ' + export LDFLAGS + LT_SYS_LIBRARY_PATH=/usr/lib64: + export LT_SYS_LIBRARY_PATH + CC=gcc + export CC + CXX=g++ + export CXX + cd onedrive-2.5.6 + /usr/bin/make install DESTDIR=/builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT 'INSTALL=/usr/bin/install -p' PREFIX=/builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/bin /usr/bin/install -p onedrive /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/bin/onedrive mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/man/man1 /usr/bin/install -p -m 0644 onedrive.1 /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/man/man1/onedrive.1 mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/etc/logrotate.d /usr/bin/install -p -m 0644 contrib/logrotate/onedrive.logrotate /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/etc/logrotate.d/onedrive mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive for file in readme.md config LICENSE changelog.md docs/advanced-usage.md docs/application-config-options.md docs/application-security.md docs/business-shared-items.md docs/client-architecture.md docs/contributing.md docs/docker.md docs/install.md docs/national-cloud-deployments.md docs/podman.md docs/privacy-policy.md docs/sharepoint-libraries.md docs/terms-of-service.md docs/ubuntu-package-install.md docs/usage.md docs/known-issues.md docs/webhooks.md; do \ /usr/bin/install -p -m 0644 $file /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive; \ done + install -D -m 0644 contrib/systemd/onedrive@.service /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/lib/systemd/system/onedrive@.service + install -D -m 0644 contrib/systemd/onedrive.service /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/lib/systemd/user/onedrive.service + /usr/lib/rpm/check-buildroot + /usr/lib/rpm/redhat/brp-ldconfig + /usr/lib/rpm/brp-compress + /usr/lib/rpm/brp-strip /usr/bin/strip + /usr/lib/rpm/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip + /usr/lib/rpm/check-rpaths + /usr/lib/rpm/redhat/brp-mangle-shebangs + /usr/lib/rpm/brp-remove-la-files + env /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0 -j1 + /usr/lib/rpm/redhat/brp-python-hardlink + /usr/bin/add-determinism --brp -j1 /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT Scanned 14 directories and 26 files, processed 1 inodes, 0 modified (0 replaced + 0 rewritten), 0 unsupported format, 0 errors Reading /builddir/build/BUILD/onedrive-2.5.6-build/SPECPARTS/rpm-debuginfo.specpart Processing files: onedrive-2.5.6-1.fc42.x86_64 Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.2lS8Ty + umask 022 + cd /builddir/build/BUILD/onedrive-2.5.6-build + cd onedrive-2.5.6 + DOCDIR=/builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + export LC_ALL=C.UTF-8 + LC_ALL=C.UTF-8 + export DOCDIR + /usr/bin/mkdir -p /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/readme.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/LICENSE /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/changelog.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/advanced-usage.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/application-config-options.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/application-security.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/build-rpm-howto.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/business-shared-items.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/client-architecture.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/contributing.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/docker.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/install.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/known-issues.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/national-cloud-deployments.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/podman.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/privacy-policy.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/sharepoint-libraries.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/terms-of-service.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/ubuntu-package-install.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/usage.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/docs/webhooks.md /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + cp -pr /builddir/build/BUILD/onedrive-2.5.6-build/onedrive-2.5.6/config /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT/usr/share/doc/onedrive + RPM_EC=0 ++ jobs -p + exit 0 Provides: config(onedrive) = 2.5.6-1.fc42 onedrive = 2.5.6-1.fc42 onedrive(x86-64) = 2.5.6-1.fc42 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Requires(post): systemd Requires(preun): systemd Requires(postun): systemd Requires: ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.15)(64bit) libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) libc.so.6(GLIBC_2.33)(64bit) libc.so.6(GLIBC_2.34)(64bit) libc.so.6(GLIBC_2.4)(64bit) libc.so.6(GLIBC_2.7)(64bit) libc.so.6(GLIBC_2.8)(64bit) libcurl.so.4()(64bit) libdbus-1.so.3()(64bit) libdbus-1.so.3(LIBDBUS_1_3)(64bit) libdruntime-ldc-shared.so.110()(64bit) libgcc_s.so.1()(64bit) libgcc_s.so.1(GCC_3.0)(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libphobos2-ldc-shared.so.110()(64bit) libsqlite3.so.0()(64bit) rtld(GNU_HASH) Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILD/onedrive-2.5.6-build/BUILDROOT Wrote: /builddir/build/RPMS/onedrive-2.5.6-1.fc42.x86_64.rpm Finish: rpmbuild onedrive-2.5.6-1.el9.src.rpm Finish: build phase for onedrive-2.5.6-1.el9.src.rpm INFO: Done(/home/alex/rpmbuild/SRPMS/onedrive-2.5.6-1.el9.src.rpm) Config(fedora-42-x86_64) 0 minutes 54 seconds INFO: Results and/or logs in: /var/lib/mock/fedora-42-x86_64/result Finish: run ``` ================================================ FILE: docs/business-shared-items.md ================================================ # How to sync OneDrive Business Shared Items > [!CAUTION] > Before reading this document, please ensure you are running application version [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) or greater. Use `onedrive --version` to determine what application version you are using and upgrade your client if required. > [!CAUTION] > This feature has been 100% re-written from v2.5.0 onwards and is not backwards compatible with v2.4.x client versions. If enabling this feature, you must upgrade to v2.5.0 or above on all systems that are running this client. > > An additional pre-requisite before using this capability in v2.5.0 and above is for you to revert any v2.4.x Shared Business Folder configuration you may be currently using, including, but not limited to: > * Removing `sync_business_shared_folders = "true|false"` from your 'config' file > * Removing the 'business_shared_folders' file > * Removing any local data | shared folder data from your configured 'sync_dir' to ensure that there are no conflicts or issues. > * Removing any configuration online that might be related to using this feature prior to v2.5.0 ## Process Overview Syncing OneDrive Business Shared Folders requires additional configuration for your 'onedrive' client: 1. From the OneDrive web interface, review the 'Shared' objects that have been shared with you. 2. Select the applicable folder, and click the 'Add shortcut to My files', which will then add this to your 'My files' folder 3. Update your OneDrive Client for Linux 'config' file to enable the feature by adding `sync_business_shared_items = "true"`. Adding this option will trigger a `--resync` requirement. 4. Test the configuration using '--dry-run' 5. Remove the use of '--dry-run' and sync the OneDrive Business Shared folders as required ### Enable syncing of OneDrive Business Shared Items via config file ```text sync_business_shared_items = "true" ``` ### Disable syncing of OneDrive Business Shared Items via config file ```text sync_business_shared_items = "false" ``` ## Syncing OneDrive Business Shared Folders Use the following steps to add a OneDrive Business Shared Folder to your account: 1. Login to Microsoft OneDrive online, and navigate to 'Shared' from the left hand side pane ![objects_shared_with_me](./images/objects_shared_with_me.png) 2. Select the respective folder you wish to sync, and click the 'Add shortcut to My files' at the top of the page ![add_shared_folder](./images/add_shared_folder.png) 3. The final result online will look like this: ![shared_folder_added](./images/shared_folder_added.png) When using Microsoft Windows, this shared folder will appear as the following: ![windows_view_shared_folders](./images/windows_view_shared_folders.png) 4. Sync your data using `onedrive --sync --verbose`. If you have just enabled the `sync_business_shared_items = "true"` configuration option, you will be required to perform a resync. During the sync, the selected shared folder will be downloaded: ``` ... Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 4 Finished processing /delta JSON response from the OneDrive API Processing 3 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Creating local directory: ./my_shared_folder Quota information is restricted or not available for this drive. Syncing this OneDrive Business Shared Folder: my_shared_folder Fetching /delta response from the OneDrive API for Drive ID: b!BhWyqa7K_kqXqHtSIlsqjR5iJogxpWxDradnpVGTU2VxBOJh82Y6S4he4rdnGPBT Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 6 Finished processing /delta JSON response from the OneDrive API Processing 6 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Creating local directory: ./my_shared_folder/asdf Creating local directory: ./my_shared_folder/original_data Number of items to download from OneDrive: 3 Downloading file: my_shared_folder/my_folder/file_one.txt ... done Downloading file: my_shared_folder/my_folder/file_two.txt ... done Downloading file: my_shared_folder/original_data/file1.data ... done Performing a database consistency and integrity check on locally stored data ... ``` When this is viewed locally, on Linux, this shared folder is seen as the following: ![linux_shared_folder_view](./images/linux_shared_folder_view.png) Any shared folder you add can utilise any 'client side filtering' rules that you have created. ## Syncing OneDrive Business Shared Files There are two methods to support the syncing OneDrive Business Shared Files with the OneDrive Application 1. Add a 'shortcut' to your 'My Files' for the file, which creates a URL shortcut to the file which can be followed when using a Linux Window Manager (Gnome, KDE etc) and the link will open up in a browser. Microsoft Windows only supports this option. 2. Use `--sync-shared-files` option to sync all files shared with you to your local disk. If you use this method, you can utilise any 'client side filtering' rules that you have created to filter out files you do not want locally. This option will create a new folder locally, with sub-folders named after the person who shared the data with you. ### Syncing OneDrive Business Shared Files using Option 1 1. As per the above method for adding folders, select the shared file, then select to 'Add shortcut' to the file ![add_shared_file_shortcut](./images/add_shared_file_shortcut.png) 2. The final result online will look like this: ![add_shared_file_shortcut_added](./images/online_shared_file_link.png) When using Microsoft Windows, this shared file will appear as the following: ![windows_view_shared_file_link](./images/windows_view_shared_file_link.png) 3. Sync your data using `onedrive --sync --verbose`. If you have just enabled the `sync_business_shared_items = "true"` configuration option, you will be required to perform a resync. ``` ... All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2 Finished processing /delta JSON response from the OneDrive API Processing 1 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Number of items to download from OneDrive: 1 Downloading file: ./file to share.docx.url ... done Syncing this OneDrive Business Shared Folder: my_shared_folder Fetching /delta response from the OneDrive API for Drive ID: b!BhWyqa7K_kqXqHtSIlsqjR5iJogxpWxDradnpVGTU2VxBOJh82Y6S4he4rdnGPBT Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0 Finished processing /delta JSON response from the OneDrive API No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive Quota information is restricted or not available for this drive. Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!BhWyqa7K_kqXqHtSIlsqjR5iJogxpWxDradnpVGTU2VxBOJh82Y6S4he4rdnGPBT Quota information is restricted or not available for this drive. ... ``` When this is viewed locally, on Linux, this shared folder is seen as the following: ![linux_view_shared_file_link](./images/linux_view_shared_file_link.png) Any shared file link you add can utilise any 'client side filtering' rules that you have created. ### Syncing OneDrive Business Shared Files using Option 2 > [!IMPORTANT] > When using option 2, all files that have been shared with you will be downloaded by default. To reduce this, first use `--list-shared-items` to list all shared items with your account, then use 'client side filtering' rules such as 'sync_list' configuration to selectively sync all the files to your local system. 1. Review all items that have been shared with you by using `onedrive --list-shared-items`. This should display output similar to the following: ``` ... Listing available OneDrive Business Shared Items: ----------------------------------------------------------------------------------- Shared File: large_document_shared.docx Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: no_download_access.docx Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: online_access_only.txt Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: read_only.txt Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: qewrqwerwqer.txt Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: dummy_file_to_share.docx Shared By: testuser2 testuser2 (testuser2@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared Folder: Sub Folder 2 Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared File: file to share.docx Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared Folder: Top Folder Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared Folder: my_shared_folder Shared By: testuser2 testuser2 (testuser2@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- Shared Folder: Jenkins Shared By: test user (testuser@mynasau3.onmicrosoft.com) ----------------------------------------------------------------------------------- ... ``` 2. If applicable, add entries to a 'sync_list' file, to only sync the shared files that are of importance to you. 3. Run the command `onedrive --sync --verbose --sync-shared-files` to sync the shared files to your local file system. This will create a new local folder called 'Files Shared With Me', and will contain sub-directories named after the entity account that has shared the file with you. In that folder will reside the shared file: ``` ... Finished processing /delta JSON response from the OneDrive API No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive Syncing this OneDrive Business Shared Folder: my_shared_folder Fetching /delta response from the OneDrive API for Drive ID: b!BhWyqa7K_kqXqHtSIlsqjR5iJogxpWxDradnpVGTU2VxBOJh82Y6S4he4rdnGPBT Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0 Finished processing /delta JSON response from the OneDrive API No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive Quota information is restricted or not available for this drive. Creating the OneDrive Business Shared Files Local Directory: /home/alex/OneDrive/Files Shared With Me Checking for any applicable OneDrive Business Shared Files which need to be synced locally Creating the OneDrive Business Shared File Users Local Directory: /home/alex/OneDrive/Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com) Creating the OneDrive Business Shared File Users Local Directory: /home/alex/OneDrive/Files Shared With Me/testuser2 testuser2 (testuser2@mynasau3.onmicrosoft.com) Number of items to download from OneDrive: 7 Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/file to share.docx ... done OneDrive returned a 'HTTP 403 - Forbidden' - gracefully handling error Unable to download this file as this was shared as read-only without download permission: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/no_download_access.docx ERROR: File failed to download. Increase logging verbosity to determine why. Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/no_download_access.docx ... failed! Downloading file: Files Shared With Me/testuser2 testuser2 (testuser2@mynasau3.onmicrosoft.com)/dummy_file_to_share.docx ... done Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 0% | ETA --:--:-- Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/online_access_only.txt ... done Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/read_only.txt ... done Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/qewrqwerwqer.txt ... done Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 5% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 10% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 15% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 20% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 25% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 30% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 35% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 40% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 45% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 50% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 55% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 60% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 65% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 70% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 75% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 80% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 85% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 90% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 95% | ETA 00:00:00 Downloading: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... 100% | DONE in 00:00:00 Quota information is restricted or not available for this drive. Downloading file: Files Shared With Me/test user (testuser@mynasau3.onmicrosoft.com)/large_document_shared.docx ... done Quota information is restricted or not available for this drive. Quota information is restricted or not available for this drive. Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!BhWyqa7K_kqXqHtSIlsqjR5iJogxpWxDradnpVGTU2VxBOJh82Y6S4he4rdnGPBT Quota information is restricted or not available for this drive. ... ``` When this is viewed locally, on Linux, this 'Files Shared With Me' and content is seen as the following: ![files_shared_with_me_folder](./images/files_shared_with_me_folder.png) Unfortunately there is no Microsoft Windows equivalent for this capability. ## Known Issues Shared folders, shared with you from people outside of your 'organisation' are unable to be synced. This is due to the Microsoft Graph API not presenting these folders. Shared folders that match this scenario, when you view 'Shared' via OneDrive online, will have a 'world' symbol as per below: ![shared_with_me](./images/shared_with_me.JPG) This issue is being tracked by: [#966](https://github.com/abraunegg/onedrive/issues/966) ================================================ FILE: docs/client-architecture.md ================================================ # OneDrive Client for Linux Application Architecture ## How does the client work at a high level? The client utilises the 'libcurl' library to communicate with Microsoft OneDrive via the Microsoft Graph API. The diagram below shows this high level interaction with the Microsoft and GitHub API services online: ![client_use_of_libcurl](./puml/client_use_of_libcurl.png) Depending on your operational environment, it is possible to 'tweak' the following options which will modify how libcurl operates with it's interaction with Microsoft OneDrive services: * Downgrade all HTTPS operations to use HTTP1.1 (Config Option: `force_http_11`) * Control how long a specific transfer should take before it is considered too slow and aborted (Config Option: `operation_timeout`) * Control libcurl handling of DNS Cache Timeout (Config Option: `dns_timeout`) * Control the maximum time allowed for the connection to be established (Config Option: `connect_timeout`) * Control the timeout for activity on an established HTTPS connection (Config Option: `data_timeout`) * Control what IP protocol version should be used when communicating with OneDrive (Config Option: `ip_protocol_version`) * Control what User Agent is presented to Microsoft services (Config Option: `user_agent`) > [!IMPORTANT] > The default 'user_agent' value conforms to specific Microsoft requirements to identify as an ISV that complies with OneDrive traffic decoration requirements. Changing this value potentially will impact how Microsoft see's your client, thus your traffic may get throttled. For further information please read: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online Diving a little deeper into how the client operates, the diagram below outlines at a high level the operational workflow of the OneDrive Client for Linux, demonstrating how it interacts with the OneDrive API to maintain synchronisation, manage local and cloud data integrity, and ensure that user data is accurately mirrored between the local filesystem and OneDrive cloud storage. ![High Level Application Sequence](./puml/high_level_operational_process.png) The application operational processes have several high level key stages: 1. **Access Token Validation:** Initially, the client validates its access and the existing access token, refreshing it if necessary. This step ensures that the client has the required permissions to interact with the OneDrive API. 2. **Query Microsoft OneDrive API:** The client queries the /delta API endpoint of Microsoft OneDrive, which returns JSON responses. The /delta endpoint is particularly used for syncing changes, helping the client to identify any updates in the OneDrive storage. 3. **Process JSON Responses:** The client processes each JSON response to determine if it represents a 'root' or 'deleted' item. Items not marked as 'root' or 'deleted' are temporarily stored for further processing. For 'root' or 'deleted' items, the client processes them immediately, otherwise, the client evaluates the items against client-side filtering rules to decide whether to discard them or to process and save them in the local database cache for actions like creating directories or downloading files. 4. **Local Cache Database Processing for Data Integrity:** The client processes its local cache database to check for data integrity and differences compared to the OneDrive storage. If differences are found, such as a file or folder change including deletions, the client uploads these changes to OneDrive. Responses from the API, including item metadata, are saved to the local cache database. 5. **Local Filesystem Scanning:** The client scans the local filesystem for new files or folders. Each new item is checked against client-side filtering rules. If an item passes the filtering, it is uploaded to OneDrive. Otherwise, it is discarded if it doesn't meet the filtering criteria. 6. **Final Data True-Up:** Lastly, the client queries the /delta link for a final true-up, processing any further online JSON changes if required. This ensures that the local and OneDrive storages are fully synchronised. ## What are the operational modes of the client? There are 2 main operational modes that the client can utilise: 1. Standalone sync mode that performs a single sync action against Microsoft OneDrive. This method is used when you utilise `--sync`. 2. Ongoing sync mode that continuously syncs your data with Microsoft OneDrive and utilises 'inotify' to watch for local system changes. This method is used when you utilise `--monitor`. By default, both sync modes (`--sync` and `--monitor`)treat the data stored online in Microsoft OneDrive as the 'source-of-truth'. This means the client will first examine your OneDrive account for any changes (additions, modifications, deletions) and apply those changes to your local file system. After this, any local changes are uploaded, and finally, a second check ensures your local state matches the online state. This mirrors the behaviour of the Microsoft OneDrive Client for Windows. ![Default Sync Flow Process](./puml/default_sync_flow.png) When using the client with the `--local-first` option, the sync flow is reversed. The client treats your local files as the 'source-of-truth'. Local changes are processed first and pushed to Microsoft OneDrive online. Only after local changes have been uploaded will the client check for any remote changes (this includes online additions, modifications and deletions) and apply those to your local system as needed, ensuring the final local state is consistent with that what is now online. ![Local First Sync Flow Process](./puml/local_first_sync_process.png) > [!IMPORTANT] > When using `--sync --local-first`, a locally deleted file will only be deleted online if it was already in sync with its online counterpart. > * If the file was never synced, the client cannot know that the corresponding online file should be removed. In this case, the online file may be downloaded again > * Using `--resync` makes this behaviour more likely because it wipes all local knowledge of what was previously synced, so local deletions will not be recognised > > When using `--monitor --local-first`, file system watches (via inotify) will detect local deletions. This event will automatically trigger removal of the online file, and if exists and matches the local data, the file online will be removed. > [!IMPORTANT] > Please be aware that if you designate a network mount point (such as NFS, Windows Network Share, or Samba Network Share) as your `sync_dir`, this setup inherently lacks 'inotify' support. Support for 'inotify' is essential for real-time tracking of file changes, which means that the client's 'Monitor Mode' cannot immediately detect changes in files located on these network shares. Instead, synchronisation between your local filesystem and Microsoft OneDrive will occur at intervals specified by the `monitor_interval` setting. This limitation regarding 'inotify' support on network mount points like NFS or Samba is beyond the control of this client. ## OneDrive Client for Linux High Level Activity Flows The diagrams below show the high level process flow and decision making when running the application ### Main functional activity flows ![Main Activity](./puml/main_activity_flows.png) ### Processing a potentially new local item ![applyPotentiallyNewLocalItem](./puml/applyPotentiallyNewLocalItem.png) ### Processing a potentially changed local item ![applyPotentiallyChangedItem](./puml/applyPotentiallyChangedItem.png) ### Download a file from Microsoft OneDrive ![downloadFile](./puml/downloadFile.png) ### Upload a modified file to Microsoft OneDrive ![uploadModifiedFile](./puml/uploadModifiedFile.png) ### Upload a new local file to Microsoft OneDrive ![uploadFile](./puml/uploadFile.png) ### Determining if an 'item' is synchronised between Microsoft OneDrive and the local file system ![Item Sync Determination](./puml/is_item_in_sync.png) ### Determining if an 'item' is excluded due to 'Client Side Filtering' rules By default, the OneDrive Client for Linux will sync all files and folders between Microsoft OneDrive and the local filesystem. Client Side Filtering in the context of this client refers to user-configured rules that determine what files and directories the client should upload or download from Microsoft OneDrive. These rules are crucial for optimising synchronisation, especially when dealing with large numbers of files or specific file types. The OneDrive Client for Linux offers several configuration options to facilitate this: * **skip_dir:** This option allows the user to specify directories that should not be synchronised with OneDrive. It's particularly useful for omitting large or irrelevant directories from the sync process. * **skip_dotfiles:** Dotfiles, usually configuration files or scripts, can be excluded from the sync. This is useful for users who prefer to keep these files local. * **skip_file:** Specific files can be excluded from synchronisation using this option. It provides flexibility in selecting which files are essential for cloud storage. * **skip_symlinks:** Symlinks often point to files outside the OneDrive directory or to locations that are not relevant for cloud storage. This option prevents them from being included in the sync. This exclusion process can be illustrated by the following activity diagram. A 'true' return value means that the path being evaluated needs to be excluded: ![Client Side Filtering Determination](./puml/client_side_filtering_rules.png) ## Understanding how the client processes online state When you see `Fetching items from the OneDrive API for Drive ID:` or `Generating a /delta response from the OneDrive API for this Drive ID:` the client isn’t stuck—it’s working through paged change sets from Microsoft Graph using your current delta token, reconciling them with the local database, and safely scheduling work. Microsoft Graph returns paged results and signals either `@odata.nextLink` (more pages to fetch) or `@odata.deltaLink` (caught up; keep this token for next time) - the client follows those links until it reaches a stable point. Page sizing and paging behaviour are controlled by the Microsoft Graph API service. ### What a typical cycle looks like 1. **Fetching online state** * **Application Output:** `Fetching items from the OneDrive API for Drive ID: …` or `Generating a /delta response from the OneDrive API for this Drive ID:` * The client requests the next page of changes using your current delta token. 2. **Processing received items** * **Application Output:** `Processing N applicable changes and items received from Microsoft OneDrive` * Each item received is classified (add/update/delete/excluded), matched against local state, and queued for action. 3. **Execute required actions** * Download new or modified files, Delete local data that has been deleted online, Create new local directories 4. **Database Integrity** * **Application Output:** `Performing a database consistency and integrity check on locally stored data` * Integrity pass to prevent state corruption 5. **Local scan for new local data** * **Application Output:** `Scanning the local file system '…' for new data to upload` * Traverse local filesystem, honouring client side filtering rules 6. **True-Up** * **Application Output:** `Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process` * Final scan of online to ensure that everything is in the state it is meant to be ### Why first runs or --resync take longer A first run (or a deliberate `--resync`) must enumerate the entire tree to establish a known-good baseline; subsequent incremental runs are much faster because the delta token limits work to just the changes since last time. ### What affects performance the most * **Item count & Online structure:** Many folders and files dominate metadata work leading to more metadata churn * **Network** Latency and throughput directly affect how quickly we can iterate Microsoft Graph API responses and transfer content. * **Local Disk & filesystem:** SSDs perform metadata and DB work far faster than spinning disks or remote mounts. Your filesystem type (e.g., ext4, XFS, ZFS) matters and should be tuned appropriately. * **File Indexing:** Disable File Indexing (Tracker, Baloo, Searchmonkey, Pinot and others) as these are adding latency and disk I/O to your operations slowing down your performance. * **CPU & memory:** Classification and hashing are CPU-bound; insufficient RAM or swap can slow DB and traversal work. ## Delta Response vs Generated Delta Response By default, the client uses Microsoft Graph’s `/delta` to retrieve changes efficiently. In a few situations, however, using `/delta` would be wrong or unsafe for your intent. In those cases the client generates a delta by walking the relevant online subtree and synthesising the current state before reconciling it locally. This is intentionally slower but correct. ### When the client deliberately generates a delta * Some national cloud deployments where a needed delta endpoint/feature isn’t available. Capabilities differ by resource and cloud; when a required delta isn’t available, we walk the tree and synthesise the change set. * The use of `--single-directory` scope. A naïve drive-level /delta can include changes outside your intended scope. Generating a delta ensures only the in-scope subtree is considered. * The use of `--download-only --cleanup-local-files`. Raw /delta may replay online delete/replace churn that would remove valid local files you intend to keep. Generated delta captures the current online state and intentionally ignores those intermediate events to protect local data. * The use of 'Shared Folders'. Calling `/delta` on a shared path can be rooted at the owner’s drive, so your filters may not match what you see as “the shared folder”. Generated delta walks the shared subtree and normalises paths so the queue reflects what’s truly shared with you. ## File conflict handling - default operational modes When using the default operational modes (`--sync` or `--monitor`) the client application is conforming to how the Microsoft Windows OneDrive client operates in terms of resolving conflicts for files. When using `--resync` this conflict resolution can differ slightly, as, when using `--resync` you are *deleting* the known application state, thus, the application has zero reference as to what was previously in sync with the local file system. Due to this factor, when using `--resync` the online source is always going to be considered accurate and the source-of-truth, regardless of the local file state, local file timestamp or local file hash. When a difference in local file hash is detected, the file will be renamed to prevent local data loss. > [!IMPORTANT] > In v2.5.3 and above, when a local file is renamed due to conflict handling, this will be in the following format pattern to allow easier identification: > > **filename-hostname-safeBackup-number.file_extension** > > For example: > ``` > -rw-------. 1 alex alex 53402 Sep 21 08:25 file5.data > -rw-------. 1 alex alex 53423 Nov 13 18:18 file5-onedrive-client-dev-safeBackup-0001.data > -rw-------. 1 alex alex 53422 Nov 13 18:19 file5-onedrive-client-dev-safeBackup-0002.data > ``` > > In client versions v2.5.2 and below, the renamed file have the following naming convention: > > **filename-hostname-number.file_extension** > > resulting in backup filenames of the following format: > ``` > -rw-------. 1 alex alex 53402 Sep 21 08:25 file5.data > -rw-------. 1 alex alex 53432 Nov 14 05:22 file5-onedrive-client-dev-2.data > -rw-------. 1 alex alex 53435 Nov 14 05:24 file5-onedrive-client-dev-3.data > -rw-------. 1 alex alex 53419 Nov 14 05:22 file5-onedrive-client-dev.data > ``` > > [!CAUTION] > The creation of backup files when there is a conflict to avoid local data loss can be disabled. > > To do this, utilise the configuration option **'bypass_data_preservation'** > ``` > bypass_data_preservation = "true" > ``` > > If enable this option, you may experience data loss on your local data as the existing local file will be over-written with data from OneDrive online. Use with extreme care and caution. > [!TIP] > If you wish to avoid having these backup files from being uploaded to your online OneDrive account, you can utilise the configuration option **'skip_file'** to skip these files from being uploaded. > > For example: > ``` > skip_file = "~*|.~*|*.tmp|*.swp|*.partial|*-safeBackup-*" > ``` > This example retails the application defaults for 'skip_file' and adds an entry to skip any 'safeBackup' generated file. ### Default Operational Modes - Conflict Handling #### Scenario 1. Create a local file 2. Perform a sync with Microsoft OneDrive using `onedrive --sync` 3. Modify file online 4. Modify file locally with different data|contents 5. Perform a sync with Microsoft OneDrive using `onedrive --sync` ![conflict_handling_default](./puml/conflict_handling_default.png) #### Evidence of Conflict Handling ``` ... Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2 Finished processing /delta JSON response from the OneDrive API Processing 1 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Number of items to download from OneDrive: 1 The local file to replace (./1.txt) has been modified locally since the last download. Renaming it to avoid potential local data loss. The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: ./1.txt -> ./1-onedrive-client-dev.txt Downloading file ./1.txt ... done Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing ~/OneDrive The directory has not changed Processing α ... The file has not changed Processing เอกสาร The directory has not changed Processing 1.txt The file has not changed Scanning the local file system '~/OneDrive' for new data to upload ... New items to upload to OneDrive: 1 Total New Data to Upload: 52 Bytes Uploading new file ./1-onedrive-client-dev.txt ... done. Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2 Finished processing /delta JSON response from the OneDrive API Processing 1 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Sync with Microsoft OneDrive is complete Waiting for all internal threads to complete before exiting application ``` ### Default Operational Modes - Conflict Handling with --resync #### Scenario 1. Create a local file 2. Perform a sync with Microsoft OneDrive using `onedrive --sync` 3. Modify file online 4. Modify file locally with different data|contents 5. Perform a sync with Microsoft OneDrive using `onedrive --sync --resync` ![conflict_handling_default_resync](./puml/conflict_handling_default_resync.png) #### Evidence of Conflict Handling ``` ... Deleting the saved application sync status ... Using IPv4 and IPv6 (if configured) for all network operations Checking Application Version ... ... Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 14 Finished processing /delta JSON response from the OneDrive API Processing 13 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Local file time discrepancy detected: ./1.txt This local file has a different modified time 2024-Feb-19 19:32:55Z (UTC) when compared to remote modified time 2024-Feb-19 19:32:36Z (UTC) The local file has a different hash when compared to remote file hash Local item does not exist in local database - replacing with file from OneDrive - failed download? The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: ./1.txt -> ./1-onedrive-client-dev.txt Number of items to download from OneDrive: 1 Downloading file ./1.txt ... done Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing ~/OneDrive The directory has not changed Processing α ... Processing เอกสาร The directory has not changed Processing 1.txt The file has not changed Scanning the local file system '~/OneDrive' for new data to upload ... New items to upload to OneDrive: 1 Total New Data to Upload: 52 Bytes Uploading new file ./1-onedrive-client-dev.txt ... done. Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 2 Finished processing /delta JSON response from the OneDrive API Processing 1 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Sync with Microsoft OneDrive is complete Waiting for all internal threads to complete before exiting application ``` ## File conflict handling - local-first operational mode When using `--local-first` as your operational parameter the client application is now using your local filesystem data as the 'source-of-truth' as to what should be stored online. However - Microsoft OneDrive itself, has *zero* acknowledgement of this concept, thus, conflict handling needs to be aligned to how Microsoft OneDrive on other platforms operate, that is, rename the local offending file. Additionally, when using `--resync` you are *deleting* the known application state, thus, the application has zero reference as to what was previously in sync with the local file system. Due to this factor, when using `--resync` the online source is always going to be considered accurate and the source-of-truth, regardless of the local file state, file timestamp or file hash or use of `--local-first`. ### Local First Operational Modes - Conflict Handling #### Scenario 1. Create a local file 2. Perform a sync with Microsoft OneDrive using `onedrive --sync --local-first` 3. Modify file locally with different data|contents 4. Modify file online with different data|contents 5. Perform a sync with Microsoft OneDrive using `onedrive --sync --local-first` ![conflict_handling_local-first_default](./puml/conflict_handling_local-first_default.png) #### Evidence of Conflict Handling ``` Reading configuration file: /home/alex/.config/onedrive/config ... Using IPv4 and IPv6 (if configured) for all network operations Checking Application Version ... ... Sync Engine Initialised with new Onedrive API instance All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing ~/OneDrive The directory has not changed Processing α The directory has not changed ... The file has not changed Processing เอกสาร The directory has not changed Processing 1.txt Local file time discrepancy detected: 1.txt The file content has changed locally and has a newer timestamp, thus needs to be uploaded to OneDrive Changed local items to upload to OneDrive: 1 The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: 1.txt -> 1-onedrive-client-dev.txt Uploading new file 1-onedrive-client-dev.txt ... done. Scanning the local file system '~/OneDrive' for new data to upload ... Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3 Finished processing /delta JSON response from the OneDrive API Processing 2 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Number of items to download from OneDrive: 1 Downloading file ./1.txt ... done Sync with Microsoft OneDrive is complete Waiting for all internal threads to complete before exiting application ``` ### Local First Operational Modes - Conflict Handling with --resync #### Scenario 1. Create a local file 2. Perform a sync with Microsoft OneDrive using `onedrive --sync --local-first` 3. Modify file locally with different data|contents 4. Modify file online with different data|contents 5. Perform a sync with Microsoft OneDrive using `onedrive --sync --local-first --resync` ![conflict_handling_local-first_resync](./puml/conflict_handling_local-first_resync.png) #### Evidence of Conflict Handling ``` ... Are you sure you wish to proceed with --resync? [Y/N] y Deleting the saved application sync status ... Using IPv4 and IPv6 (if configured) for all network operations ... Sync Engine Initialised with new Onedrive API instance All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive Performing a database consistency and integrity check on locally stored data Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing ~/OneDrive The directory has not changed Scanning the local file system '~/OneDrive' for new data to upload Skipping item - excluded by sync_list config: ./random_25k_files OneDrive Client requested to create this directory online: ./α The requested directory to create was found on OneDrive - skipping creating the directory: ./α ... New items to upload to OneDrive: 9 Total New Data to Upload: 49 KB ... The file we are attempting to upload as a new file already exists on Microsoft OneDrive: ./1.txt Skipping uploading this item as a new file, will upload as a modified file (online file already exists): ./1.txt The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: ./1.txt -> ./1-onedrive-client-dev.txt Uploading new file ./1-onedrive-client-dev.txt ... done. Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 15 Finished processing /delta JSON response from the OneDrive API Processing 14 applicable changes and items received from Microsoft OneDrive Processing OneDrive JSON item batch [1/1] to ensure consistent local state Number of items to download from OneDrive: 1 Downloading file ./1.txt ... done Sync with Microsoft OneDrive is complete Waiting for all internal threads to complete before exiting application ``` ## Client Functional Component Architecture Relationships The diagram below shows the main functional relationship of application code components, and how these relate to each relevant code module within this application: ![Functional Code Components](./puml/code_functional_component_relationships.png) ## Database Schema The diagram below shows the database schema that is used within the application ![Database Schema](./puml/database_schema.png) ================================================ FILE: docs/contributing.md ================================================ # OneDrive Client for Linux: Coding Style Guidelines ## Introduction This document outlines the coding style guidelines for code contributions for the OneDrive Client for Linux. These guidelines are intended to ensure the codebase remains clean, well-organised, and accessible to all contributors, new and experienced alike. ## Code Layout > [!NOTE] > When developing any code contribution, please utilise either Microsoft Visual Studio Code or Notepad++. ### Indentation Most of the codebase utilises tabs for space indentation, with 4 spaces to a tab. Please keep to this convention. ### Line Length Try and keep line lengths to a reasonable length. Do not constrain yourself to short line lengths such as 80 characters. This means when the code is being displayed in the code editor, lines are correctly displayed when using screen resolutions of 1920x1080 and above. If you wish to use shorter line lengths (80 characters for example), please do not follow this sort of example: ```code ... void functionName( string somevar, bool someOtherVar, cost(char) anotherVar=null ){ .... ``` ### Coding Style | Braces Please use 1TBS (One True Brace Style) which is a variation of the K&R (Kernighan & Ritchie) style. This approach is intended to improve readability and maintain consistency throughout the code. When using this coding style, even when the code of the `if`, `else`, `for`, or function definition contains only one statement, braces are used to enclose it. ```code // What this if statement is doing if (condition) { // The condition was true ..... } else { // The condition was false ..... } // Loop 10 times to do something for (int i = 0; i < 10; i++) { // Loop body } // This function is to do this void functionExample() { // Function body } ``` ## Naming Conventions ### Variables and Functions Please use `camelCase` for variable and function names. ### Classes and Interfaces Please use `PascalCase` for classes, interfaces, and structs. ### Constants Use uppercase with underscores between words. ## Documentation ### Language and Spelling To maintain consistency across the project's documentation, comments, and code, all written text must adhere to British English spelling conventions, not American English. This requirement applies to all aspects of the codebase, including variable names, comments, and documentation. For example, use "specialise" instead of "specialize", "colour" instead of "color", and "organise" instead of "organize". This standard ensures that the project maintains a cohesive and consistent linguistic style. ### Code Comments Please comment code at all levels. Use `//` for all line comments. Detail why a statement is needed, or what is expected to happen so future readers or contributors can read through the intent of the code with clarity. If fixing a 'bug', please add a link to the GitHub issue being addressed as a comment, for example: ```code ... // Before discarding change - does this ID still exist on OneDrive - as in IS this // potentially a --single-directory sync and the user 'moved' the file out of the 'sync-dir' to another OneDrive folder // This is a corner edge case - https://github.com/skilion/onedrive/issues/341 // What is the original local path for this ID in the database? Does it match 'syncFolderChildPath' if (itemdb.idInLocalDatabase(driveId, item["id"].str)){ // item is in the database string originalLocalPath = computeItemPath(driveId, item["id"].str); ... ``` All code should be clearly commented. ### Application Logging Output If making changes to any application logging output, please first discuss this either via direct communication or email. For reference, below are the available application logging output functions and examples: ```code // most used addLogEntry("Basic 'info' message", ["info"]); .... or just use addLogEntry("Basic 'info' message"); addLogEntry("Basic 'verbose' message", ["verbose"]); addLogEntry("Basic 'debug' message", ["debug"]); // GUI notify only addLogEntry("Basic 'notify' ONLY message and displayed in GUI if notifications are enabled", ["notify"]); // info and notify addLogEntry("Basic 'info and notify' message and displayed in GUI if notifications are enabled", ["info", "notify"]); // log file only addLogEntry("Information sent to the log file only, and only if logging to a file is enabled", ["logFileOnly"]); // Console only (session based upload|download) addLogEntry("Basic 'Console only with new line' message", ["consoleOnly"]); // Console only with no new line addLogEntry("Basic 'Console only with no new line' message", ["consoleOnlyNoNewLine"]); ``` ### Documentation Updates If the code changes any of the functionality that is documented, it is expected that any PR submission will also include updating the respective section of user documentation and/or man page as part of the code submission. ## Development Testing Whilst there are more modern D compilers available, ensuring client build compatibility with older platforms is a key requirement. The issue stems from Debian and Ubuntu LTS versions - such as Ubuntu 20.04. It's [ldc package](https://packages.ubuntu.com/focal/ldc) is only v1.20.1 , thus, this is the minimum version that all compilation needs to be tested against. The reason LDC v1.20.1 must be used, is that this is the version that is used to compile the packages presented at [OpenSuSE Build Service ](https://build.opensuse.org/package/show/home:npreining:debian-ubuntu-onedrive/onedrive) - which is where most Debian and Ubuntu users will install the client from. It is assumed here that you know how to download and install the correct LDC compiler for your platform. ## Submitting a PR When submitting a PR, please provide your testing evidence in the PR submission of what has been fixed, in the format of: ### Without PR ``` Application output that is doing whatever | or illustration of issue | illustration of bug ``` ### With PR ``` Application output that is doing whatever | or illustration of issue being fixed | illustration of bug being fixed ``` Please also include validation of compilation using the minimum LDC package version. To assist with your testing validation against the minimum LDC compiler version, a script as per below could assist you with this validation: ```bash #!/bin/bash PR= rm -rf ./onedrive-pr${PR} git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR} cd onedrive-pr${PR} git fetch origin pull/${PR}/head:pr${PR} git checkout pr${PR} # MIN LDC Version to compile # MIN Version for ARM / Compiling with LDC source ~/dlang/ldc-1.20.1/activate # Compile code with specific LDC version ./configure --enable-debug --enable-notifications; make clean; make; deactivate ./onedrive --version ``` ## References * D Language Official Style Guide: https://dlang.org/dstyle.html * British English spelling conventions: https://www.collinsdictionary.com/ ================================================ FILE: docs/docker.md ================================================ # Run the OneDrive Client for Linux under Docker This client can be run as a Docker container, with 3 available container base options for you to choose from: | Container Base | Docker Tag | Description | i686 | x86_64 | ARMHF | AARCH64 | |----------------|-------------|----------------------------------------------------------------|:------:|:------:|:-----:|:-------:| | Alpine Linux | edge-alpine | Docker container based on Alpine 3.23 using 'master' |❌|✔|❌|✔| | Alpine Linux | alpine | Docker container based on Alpine 3.23 using latest release |❌|✔|❌|✔| | Debian | debian | Docker container based on Debian 13 using latest release |✔|✔|✔|✔| | Debian | edge | Docker container based on Debian 13 using 'master' |✔|✔|✔|✔| | Debian | edge-debian | Docker container based on Debian 13 using 'master' |✔|✔|✔|✔| | Debian | latest | Docker container based on Debian 13 using latest release |✔|✔|✔|✔| | Fedora | edge-fedora | Docker container based on Fedora 43 using 'master' |❌|✔|❌|✔| | Fedora | fedora | Docker container based on Fedora 43 using latest release |❌|✔|❌|✔| These containers offer a simple monitoring-mode service for the OneDrive Client for Linux. The instructions below have been validated on: * Fedora 40 The instructions below will utilise the 'edge' tag, however this can be substituted for any of the other docker tags such as 'latest' from the table above if desired. The 'edge' Docker Container will align closer to all documentation and features, where as 'latest' is the release version from a static point in time. The 'latest' tag however may contain bugs and/or issues that will have been fixed, and those fixes are contained in 'edge'. Additionally there are specific version release tags for each release. Refer to https://hub.docker.com/r/driveone/onedrive/tags for any other Docker tags you may be interested in. > [!NOTE] > The below instructions for docker has been tested and validated when logging into the system as an unprivileged user (non 'root' user). ## High Level Configuration Steps 1. Install 'docker' as per your distribution platform's instructions if not already installed. 2. Configure 'docker' to allow non-privileged users to run Docker commands 3. Disable 'SELinux' as per your distribution platform's instructions 4. Test 'docker' by running a test container without using `sudo` 5. Prepare the required docker volumes to store the configuration and data 6. Run the 'onedrive' container and perform authorisation 7. Running the 'onedrive' container under 'docker' ## Configuration Steps ### 1. Install 'docker' on your platform Install Docker for your system using the official instructions found at https://docs.docker.com/engine/install/. > [!CAUTION] > If you are using Ubuntu or any distribution based on Ubuntu, do not install Docker from your distribution's repositories, as they may contain obsolete versions. Instead, you must install Docker using the packages provided directly by Docker. ### 2. Configure 'docker' to allow non-privileged users to run Docker commands Read https://docs.docker.com/engine/install/linux-postinstall/ to configure the 'docker' user group with your user account to allow your non 'root' user to run 'docker' commands. ### 3. Disable SELinux on your platform In order to run the Docker container, SELinux must be disabled. Without doing this, when the application is authenticated in the steps below, the following error will be presented: ```text ERROR: The local file system returned an error with the following message: Error Message: /onedrive/conf/refresh_token: Permission denied The database cannot be opened. Please check the permissions of ~/.config/onedrive/items.sqlite3 ``` The only known work-around for the above problem at present is to disable SELinux. Please refer to your distribution platform's instructions on how to perform this step. * Fedora: https://docs.fedoraproject.org/en-US/quick-docs/selinux-changing-states-and-modes/#_disabling_selinux * Red Hat Enterprise Linux: https://access.redhat.com/solutions/3176 Post disabling SELinux and reboot your system, confirm that `getenforce` returns `Disabled`: ```text $ getenforce Disabled ``` If you are still experiencing permission issues despite disabling SELinux, please read https://www.redhat.com/sysadmin/container-permission-denied-errors ### 4. Test 'docker' on your platform Ensure that 'docker' is running as a system service, and is enabled to be activated on system reboot: ```bash sudo systemctl enable --now docker ``` Test that 'docker' is operational for your 'non-root' user, as per below: ```bash [alex@fedora-40-docker-host ~]$ docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 719385e32844: Pull complete Digest: sha256:88ec0acaa3ec199d3b7eaf73588f4518c25f9d34f58ce9a0df68429c5af48e8d Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ [alex@fedora-40-docker-host ~]$ ``` ### 5. Configure the required docker volumes The 'onedrive' Docker container requires 2 docker volumes to operate: * Config Volume * Data Volume The first volume is the configuration volume that stores all the applicable application configuration + current runtime state. In a non-containerised environment, this normally resides in `~/.config/onedrive` - in a containerised environment this is stored in the volume tagged as `/onedrive/conf` The second volume is the data volume, where all your data from Microsoft OneDrive is stored locally. This volume is mapped to an actual directory point on your local filesystem and this is stored in the volume tagged as `/onedrive/data` #### 5.1 Prepare the 'config' volume Create the 'config' volume with the following command: ```bash docker volume create onedrive_conf ``` This will create a docker volume labeled `onedrive_conf`, where all configuration of your onedrive account will be stored. You can add a custom config file in this location at a later point in time if required. #### 5.2 Prepare the 'data' volume Create the 'data' volume with the following command: ```bash docker volume create onedrive_data ``` This will create a docker volume labeled `onedrive_data` and will map to a path on your local filesystem. This is where your data from Microsoft OneDrive will be stored. Keep in mind that: * The owner of this specified folder must not be root * The owner of this specified folder must have permissions for its parent directory * Docker will attempt to change the permissions of the volume to the user the container is configured to run as > [!IMPORTANT] > Issues occur when this target folder is a mounted folder of an external system (NAS, SMB mount, USB Drive etc) as the 'mount' itself is owed by 'root'. If this is your use case, you *must* ensure your normal user can mount your desired target without having the target mounted by 'root'. If you do not fix this, your Docker container will fail to start with the following error message: > ```bash > ROOT level privileges prohibited! > ``` ### 6. First run of Docker container under docker and performing authorisation The 'onedrive' client within the container first needs to be authorised with your Microsoft account. This is achieved by initially running docker in interactive mode. Run the docker image with the commands below and make sure to change the value of `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `export ONEDRIVE_DATA_DIR="/home/abraunegg/OneDrive"`). > [!IMPORTANT] > The 'target' folder of `ONEDRIVE_DATA_DIR` must exist before running the docker container. The script below will create 'ONEDRIVE_DATA_DIR' so that it exists locally for the docker volume mapping to occur. It is also a requirement that the container be run using a non-root uid and gid, you must insert a non-root UID and GID (e.g.` export ONEDRIVE_UID=1000` and export `ONEDRIVE_GID=1000`). The script below will use `id` to evaluate your system environment to use the correct values. ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR} docker run -it --name onedrive -v onedrive_conf:/onedrive/conf \ -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" \ -e "ONEDRIVE_UID=${ONEDRIVE_UID}" \ -e "ONEDRIVE_GID=${ONEDRIVE_GID}" \ driveone/onedrive:edge ``` When the Docker container successfully starts: * You will be asked to open a specific link using your web browser * Login to your Microsoft Account and give the application the permission * After giving the permission, you will be redirected to a blank page * Copy the URI of the blank page into the application prompt to authorise the application Once the 'onedrive' application is authorised, the client will automatically start monitoring your `ONEDRIVE_DATA_DIR` for data changes to be uploaded to OneDrive. Files stored on OneDrive will be downloaded to this location. If the client is working as expected, you can detach from the container with CTRL+P, CTRL+Q. #### 6.1. Read-Only / Upload-Only Sync Scenarios If you are running the Docker container in upload-only mode and want to ensure that the OneDrive client cannot modify the original source files, the data directory must be mounted as read-only. This is controlled at the container mount level, not by ownership (chown) or permissions (chmod) inside the container. If this is your desired configuration, you must mount your 'ONEDRIVE_DATA_DIR' with read-only permissions to ensure your data source is immutable and cannot be changed. Augment the above script in the following manner: ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR} docker run -it --name onedrive -v onedrive_conf:/onedrive/conf \ -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:ro" \ -e "ONEDRIVE_UID=${ONEDRIVE_UID}" \ -e "ONEDRIVE_GID=${ONEDRIVE_GID}" \ -e ONEDRIVE_UPLOADONLY=1 \ driveone/onedrive:edge ``` > [!NOTE] > Essentially, any Docker command where you are mounting your 'ONEDRIVE_DATA_DIR', you need to append `:ro` to the `/onedrive/data` specification to ensure your data directory is mounted in Docker as read-only volume. ### 7. Running the 'onedrive' container under 'docker' #### 7.1 Check if the monitor service is running ```bash docker ps -f name=onedrive ``` #### 7.2 Show 'onedrive' runtime logs ```bash docker logs onedrive ``` #### 7.3 Stop running 'onedrive' container ```bash docker stop onedrive ``` #### 7.4 Start 'onedrive' container ```bash docker start onedrive ``` #### 7.5 Remove 'onedrive' container ```bash docker rm -f onedrive ``` ### Customising OneDrive Runtime Behaviour in Docker When running the OneDrive client inside Docker, the container **always starts** via `entrypoint.sh`, which ensures that the following arguments are added automatically: ``` --confdir /onedrive/conf --syncdir /onedrive/data ``` This design guarantees that: * Your configuration files persist in the `/onedrive/conf` volume. * Your synchronised data persists in the `/onedrive/data` volume. * The container behaves consistently across hosts, upgrades, and architectures. Because these arguments are always supplied, any `sync_dir` or `confdir` values defined in the configuration file are **overridden at runtime by design**. This avoids confusion and ensures predictable behaviour. These specific paths are the bind-mounts between container and host and should **not be changed manually**. #### Default Docker volume behaviour By default, Docker bind mounts and volumes are mounted read-write inside the container. This means that, unless explicitly restricted, the container process may create, modify, rename, or delete files within the mounted directory, subject to normal filesystem permissions. #### Using read-only mounts If you want to prevent the container from modifying the mounted data (for example, in upload-only or backup-style scenarios), the bind mount must be explicitly marked as read-only using the `:ro` mount option. A read-only mount enforces immutability at the container boundary and cannot be overridden from inside the container, regardless of ownership or permissions. ### Supported ways to customise runtime behaviour There are **two supported mechanisms** for adjusting how the client runs inside Docker: 1. **Docker environment variables** Many client options are exposed as environment variables in a reproducible way. For example: ```shell -e ONEDRIVE_DOWNLOADONLY=1 -e ONEDRIVE_SYNC_ONCE=1 -e ONEDRIVE_VERBOSE=1 ``` See the full list here: 👉 [Supported Docker environment variables](https://github.com/abraunegg/onedrive/blob/master/docs/docker.md#supported-docker-environment-variables) 2. **Configuration file inside `/onedrive/conf`** For permanent or advanced options not covered by environment variables, you can create or edit the client configuration file in the mounted config directory. Documentation: 👉 [Editing the running configuration and using a config file](https://github.com/abraunegg/onedrive/blob/master/docs/docker.md#editing-the-running-configuration-and-using-a-config-file) > [!IMPORTANT] > **Do not manually add `--syncdir` or `--confdir`** when overriding the container command. > > If you do: > > * You bypass the `entrypoint.sh` logic that manages UID/GID mapping, privilege dropping, and environment translation. > * You risk syncing data to the wrong location (`~/OneDrive` inside the container) or creating incorrect file ownership on the host. > > Instead: > > * Use existing **Docker environment variables** for controling specific application functionality. > * Use a **config file** and or 'sync_list' file inside `/onedrive/conf` for advanced configuration. ### How to use Docker-compose You can utilise `docker-compose` if available on your platform if you are able to use docker compose schemas > 3. In the following example it is assumed you have a `ONEDRIVE_DATA_DIR` environment variable and have already created the `onedrive_conf` volume. You can also use docker bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`. ``` version: "3" services: onedrive: image: driveone/onedrive:edge restart: unless-stopped environment: - ONEDRIVE_UID=${PUID} - ONEDRIVE_GID=${PGID} volumes: - onedrive_conf:/onedrive/conf - ${ONEDRIVE_DATA_DIR}:/onedrive/data ``` > [!IMPORTANT] > Before you run the container using your compose file you must first authenticate the client following [step 6](https://github.com/abraunegg/onedrive/blob/master/docs/docker.md#6-first-run-of-docker-container-under-docker-and-performing-authorisation) above. > Failure to perform this step before running your container using your compose file will see your container detail that an invalid response uri was entered. ### Editing the running configuration and using a 'config' file The 'onedrive' client should run in default configuration, however you can change this default configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config) Then put it into your onedrive_conf volume path, which can be found with: ```bash docker volume inspect onedrive_conf ``` Or you can map your own config folder to the config volume. Make sure to copy all files from the docker volume into your mapped folder first. The detailed document for the config can be found here: [Application Configuration Options for the OneDrive Client for Linux](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md) ### Syncing multiple accounts There are many ways to do this, the easiest is probably to do the following: 1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work` 2. And start a second docker monitor container (again replace `Work` with your desired name): ``` export ONEDRIVE_DATA_DIR_WORK="/home/abraunegg/OneDriveWork" mkdir -p ${ONEDRIVE_DATA_DIR_WORK} docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Work:/onedrive/conf -v "${ONEDRIVE_DATA_DIR_WORK}:/onedrive/data" driveone/onedrive:edge ``` ### Run or update the Docker container with one script If you are experienced with docker and onedrive, you can use the following script: ```bash # Update ONEDRIVE_DATA_DIR with correct OneDrive directory path ONEDRIVE_DATA_DIR="${HOME}/OneDrive" # Create directory if non-existent mkdir -p ${ONEDRIVE_DATA_DIR} firstRun='-d' docker pull driveone/onedrive:edge docker inspect onedrive_conf > /dev/null 2>&1 || { docker volume create onedrive_conf; firstRun='-it'; } docker inspect onedrive > /dev/null 2>&1 && docker rm -f onedrive docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` ## Supported Docker Environment Variables | Variable | Purpose | Sample Value | | ---------------- | --------------------------------------------------- |:--------------------------------------------------------------------------------------------------------------------------------:| | ONEDRIVE_UID | UserID (UID) to run as | 1000 | | ONEDRIVE_GID | GroupID (GID) to run as | 1000 | | ONEDRIVE_VERBOSE | Controls "--verbose" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DEBUG | Controls "--verbose --verbose" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DEBUG_HTTPS | Controls "--debug-https" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_RESYNC | Controls "--resync" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DOWNLOADONLY | Controls "--download-only" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_CLEANUPLOCAL | Controls "--cleanup-local-files" to cleanup local files and folders if they are removed online. Default is 0 | 1 | | ONEDRIVE_UPLOADONLY | Controls "--upload-only" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_NOREMOTEDELETE | Controls "--no-remote-delete" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_LOGOUT | Controls "--logout" switch. Default is 0 | 1 | | ONEDRIVE_REAUTH | Controls "--reauth" switch. Default is 0 | 1 | | ONEDRIVE_AUTHFILES | Controls "--auth-files" option. Default is "" | Please read [CLI Option: --auth-files](./application-config-options.md#cli-option---auth-files) | | ONEDRIVE_AUTHRESPONSE | Controls "--auth-response" option. Default is "" | Please read [CLI Option: --auth-response](./application-config-options.md#cli-option---auth-response) | | ONEDRIVE_DISPLAY_CONFIG | Controls "--display-running-config" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_SINGLE_DIRECTORY | Controls "--single-directory" option. Default = "" | "mydir" | | ONEDRIVE_DRYRUN | Controls "--dry-run" option. Default is 0 | 1 | | ONEDRIVE_DISABLE_DOWNLOAD_VALIDATION | Controls "--disable-download-validation" option. Default is 0 | 1 | | ONEDRIVE_DISABLE_UPLOAD_VALIDATION | Controls "--disable-upload-validation" option. Default is 0 | 1 | | ONEDRIVE_SYNC_SHARED_FILES | Controls "--sync-shared-files" option. Default is 0 | 1 | | ONEDRIVE_RUNAS_ROOT | Controls if the Docker container should be run as the 'root' user instead of 'onedrive' user. Default is 0 | 1 | | ONEDRIVE_SYNC_ONCE | Controls if the Docker container should be run in Standalone Mode. It will use Monitor Mode otherwise. Default is 0 | 1 | | ONEDRIVE_FILE_FRAGMENT_SIZE | Controls the fragment size when uploading large files to Microsoft OneDrive. The value specified is in MB. Default is 10, Limit is 60 | 25 | | ONEDRIVE_THREADS | Controls the value for the number of worker threads used for parallel upload and download operations. Default is 8, Limit is 16 | 4 | ### Environment Variables Usage Examples **Verbose Output:** ```bash docker container run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Debug Output:** ```bash docker container run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a --resync:** ```bash docker container run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a --resync and --verbose:** ```bash docker container run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a --logout:** ```bash docker container run -it -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a --logout and re-authenticate:** ```bash docker container run -it -e ONEDRIVE_REAUTH=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a sync using ONEDRIVE_SINGLE_DIRECTORY:** ```bash docker container run -e ONEDRIVE_SINGLE_DIRECTORY="path/which/needs/to/be/synced" -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` **Perform a sync specifying UID and GID:** ```bash docker container run -e ONEDRIVE_UID=9999 -e ONEDRIVE_GID=9999 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:edge ``` > [!IMPORTANT] > Is using a Docker Environment Variable that requires you to specify a 'path' (ONEDRIVE_AUTHFILES, ONEDRIVE_AUTHRESPONSE, ONEDRIVE_SINGLE_DIRECTORY), the placement of quotes around the path is critically important. > > Please ensure you are formatting the option correctly: >``` > -e OPTION="path/which/needs/to/be/synced" >``` > Please also ensure that the path specified complies with the actual application usage argument. Please read the relevant config option advice in the [CLI Option Documentation](./application-config-options.md) ## Building a custom Docker image ### Build Environment Requirements * Build environment must have at least 1GB of memory & 2GB swap space You can validate your build environment memory status with the following command: ```text cat /proc/meminfo | grep -E 'MemFree|Swap' ``` This should result in the following similar output: ```text MemFree: 3704644 kB SwapCached: 0 kB SwapTotal: 8117244 kB SwapFree: 8117244 kB ``` If you do not have enough swap space, you can use the following script to dynamically allocate a swapfile for building the Docker container: ```bash cd /var sudo fallocate -l 1.5G swapfile sudo chmod 600 swapfile sudo mkswap swapfile sudo swapon swapfile # make swap permanent sudo nano /etc/fstab # add "/swapfile swap swap defaults 0 0" at the end of file # check it has been assigned swapon -s free -h ``` If you are running a Raspberry Pi, you will need to edit your system configuration to increase your swapfile: * Modify the file `/etc/dphys-swapfile` and edit the `CONF_SWAPSIZE`, for example: `CONF_SWAPSIZE=2048`. > [!IMPORTANT] > A reboot of your Raspberry Pi is required to make this change effective. ### Building and running a custom Docker image You can also build your own image instead of pulling the one from [hub.docker.com](https://hub.docker.com/r/driveone/onedrive): ```bash git clone https://github.com/abraunegg/onedrive cd onedrive docker build . -t local-onedrive -f contrib/docker/Dockerfile docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive:latest ``` There are alternate, smaller images available by using `Dockerfile-debian` or `Dockerfile-alpine`. These [multi-stage builder pattern](https://docs.docker.com/develop/develop-images/multistage-build/) Dockerfiles require Docker version at least 17.05. ### How to build and run a custom Docker image based on Debian ``` bash docker build . -t local-onedrive-debian -f contrib/docker/Dockerfile-debian docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-debian:latest ``` ### How to build and run a custom Docker image based on Alpine Linux ``` bash docker build . -t local-onedrive-alpine -f contrib/docker/Dockerfile-alpine docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-alpine:latest ``` ### How to build and run a custom Docker image for ARMHF (Raspberry Pi) Compatible with: * Raspberry Pi * Raspberry Pi 2 * Raspberry Pi Zero * Raspberry Pi 3 * Raspberry Pi 4 ``` bash docker build . -t local-onedrive-armhf -f contrib/docker/Dockerfile-debian docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-armhf:latest ``` ### How to build and run a custom Docker image for AARCH64 Platforms ``` bash docker build . -t local-onedrive-aarch64 -f contrib/docker/Dockerfile-debian docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-aarch64:latest ``` ### How to support double-byte languages In some geographic regions, you may need to change and/or update the locale specification of the Docker container to better support the local language used for your local filesystem. To do this, follow the example below: ``` FROM driveone/onedrive ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y locales RUN echo "ja_JP.UTF-8 UTF-8" > /etc/locale.gen && \ locale-gen ja_JP.UTF-8 && \ dpkg-reconfigure locales && \ /usr/sbin/update-locale LANG=ja_JP.UTF-8 ENV LC_ALL ja_JP.UTF-8 ``` The above example changes the Docker container to support Japanese. To support your local language, change `ja_JP.UTF-8` to the required entry. ================================================ FILE: docs/install.md ================================================ # Installing or Upgrading the OneDrive Client for Linux ## Table of Contents - [Recommended Installation Method (Using Pre-Built Packages)](#recommended-installation-method-using-pre-built-packages) - [Important Notice for all Debian \| Ubuntu \| Linux Mint \| Pop!_OS \| Raspbian \| Zorin Users](#important-notice-for-all-debian--ubuntu--linux-mint--pop_os--raspbian--zorin-users) - [Which Installation Method Should I Use?](#which-installation-method-should-i-use) - [When Should You Build From Source?](#when-should-you-build-from-source) - [Building from Source](#building-from-source) - [Minimum Build Requirements](#minimum-build-requirements) - [Install Build Dependencies (By Distribution)](#install-build-dependencies-by-distribution) - [Clone, Configure, Build, Install](#clone-configure-build-install) - [High Level Steps to building the OneDrive Client for Linux](#high-level-steps-to-building-the-onedrive-client-for-linux) - [Building the Application Using Default configure Settings](#building-the-application-using-default-configure-settings) - [Build Options for Customising the Application](#build-options-for-customising-the-application) - [Upgrading the Client](#upgrading-the-client) - [If installed from a distribution package](#if-installed-from-a-distribution-package) - [If installed from source](#if-installed-from-source) - [Uninstalling the client](#uninstalling-the-client) - [If installed from a distribution package](#if-installed-from-a-distribution-package-1) - [If installed from source](#if-installed-from-source-1) ## Overview This document explains how to install or upgrade the OneDrive Client for Linux. The preferred installation method is to use pre-built distribution packages wherever they are available and current. On some distributions, particularly Debian, Ubuntu, Linux Mint, and Raspberry Pi OS, the versions provided in the default distribution repositories are outdated and unsupported. These must not be used. If your distribution provides a current maintained package, you should install the client from your package manager. If your distribution does not provide a supported package, or you need to build the client for a custom or minimal environment, building from source is supported and documented below. Before continuing, identify your Linux distribution and follow the installation path appropriate to your system. ## Recommended Installation Method (Using Pre-Built Packages) ### Important Notice for all Debian | Ubuntu | Linux Mint | Pop!_OS | Raspbian | Zorin Users > [!IMPORTANT] > **Do NOT install the OneDrive client from your distribution’s default repositories.** These packaged versions are **outdated, unsupported, and contain known defects.** > > Instead, install the **fully supported and actively maintained version** from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) ### Which Installation Method Should I Use? | Distribution & Version | Distribution Package Name & Link | Distribution Package Version | Correct Installation Method | |----------------------------------------|----------------------------------------------------------------------------------------------------------|:----------------------------------------------------:|-----------------------------| | Alpine Linux | [onedrive](https://pkgs.alpinelinux.org/packages?name=onedrive&branch=edge) |Alpine Linux Edge package | Alpine **Stable** may ship older versions. If your version is outdated, you need to build from source | | Arch Linux

Manjaro Linux | [onedrive-abraunegg](https://aur.archlinux.org/packages/onedrive-abraunegg/) |AUR package| Install via: `pamac build onedrive-abraunegg` from the Arch Linux User Repository (AUR)

**Note:** You must first install 'base-devel' as this is a pre-requisite for using the AUR

**Note:** If asked regarding a provider for 'd-runtime' and 'd-compiler', select 'liblphobos' and 'ldc'

**Note:** System must have at least 1GB of memory & 1GB swap space

AUR package `onedrive-abraunegg` follows the release versions
AUR package `onedrive-abraunegg-git` follows the 'master' branch | | CentOS Stream 8 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 8 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | CentOS Stream 9 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 9 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | CentOS Stream 10 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 10 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | Debian 11 | [onedrive](https://packages.debian.org/bullseye/source/onedrive) |Debian 11 package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Debian 12 | [onedrive](https://packages.debian.org/bookworm/source/onedrive) |Debian 12 package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Debian 13 | [onedrive](https://packages.debian.org/trixie/source/onedrive) |Debian 13 package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Debian Sid | [onedrive](https://packages.debian.org/sid/onedrive) |Debian Sid package| Install via: `sudo apt install --no-install-recommends --no-install-suggests onedrive` | | Fedora | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |Fedora Rawhide package| Install via: `sudo dnf install onedrive` | | FreeBSD | [onedrive](https://www.freshports.org/net/onedrive) |FreeBSD package| Install via: `pkg install onedrive` | | Gentoo | [onedrive](https://packages.gentoo.org/packages/net-misc/onedrive) |Gentoo package| Install via: `sudo emerge net-misc/onedrive` | | Homebrew | [onedrive-cli](https://formulae.brew.sh/formula/onedrive-cli) |Homebrew package | Install via: `brew install onedrive-cli` | | Linux Mint 21.x | [onedrive](https://community.linuxmint.com/software/view/onedrive) |Ubuntu 22.04 package | Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Linux Mint 22.x | [onedrive](https://community.linuxmint.com/software/view/onedrive) |Ubuntu 24.04 package | Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Linux Mint Debian Edition 6 | [onedrive](https://community.linuxmint.com/software/view/onedrive) |Debian 12 package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Linux Mint Debian Edition 7 | [onedrive](https://community.linuxmint.com/software/view/onedrive) |Debian 13 package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | NixOS | [onedrive](https://search.nixos.org/packages?channel=25.05&query=onedrive) |nixpkgs unstable package| Install via: `nix-env -iA nixpkgs.onedrive` **or** `services.onedrive.enable = true` in `configuration.nix` | | MX Linux 25 | [onedrive](https://mxrepo.com/mx/repo/pool/main/o/onedrive/) |MX Linux package| Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | OpenSUSE | [onedrive](https://software.opensuse.org/package/onedrive) |openSUSE Tumbleweed package| Install via: `sudo zypper install onedrive` | | OpenSUSE Build Service | [onedrive](https://build.opensuse.org/package/show/home:npreining:debian-ubuntu-onedrive/onedrive) | No API available for version information | | | Raspbian | [onedrive](https://archive.raspbian.org/raspbian/pool/main/o/onedrive/) |Raspbian Stable package | Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | RedHat Enterprise Linux 8 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 8 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | RedHat Enterprise Linux 9 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 9 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | RedHat Enterprise Linux 10 | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |EPEL 10 package| **Note:** You must install and enable the EPEL Repository first.

Install via: `sudo dnf install onedrive` | | Slackware | [onedrive](https://slackbuilds.org/result/?search=onedrive&sv=) |SlackBuilds package| Install via SlackBuilds: https://slackbuilds.org/result/?search=onedrive | | Solus | [onedrive](https://packages.getsol.us/shannon/o/onedrive/?sort=time&order=desc) |Solus package| Install via: `sudo eopkg install onedrive` | | Ubuntu 22.04 LTS | [onedrive](https://packages.ubuntu.com/jammy/onedrive) |Ubuntu 22.04 package | Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | | Ubuntu 24.04 LTS | [onedrive](https://packages.ubuntu.com/noble/onedrive) |Ubuntu 24.04 package | Install from the openSUSE Build Service (OBS) repository by following the [Ubuntu / Debian Package Installation Guide](ubuntu-package-install.md) | > [!IMPORTANT] > Distribution versions that are considered **End-of-Life (EOL)** are **no longer supported** or tested with current client releases. > [!IMPORTANT] > Distribution package maintainers are volunteers who generously contribute their time to make software available for your system. New releases of the client may take some time to appear in your distribution’s repositories. > > If you believe a new release is significantly delayed, please contact your distribution’s package maintainer directly to request an update. > > **Do not open a bug report or discussion about this here**, as we have no control over the packaging process for your distribution. ### When Should You Build From Source? You should only build from source in the following circumstances: 1. You are packaging for a custom or minimal distribution. 2. Your distribution does not have a package for your to install. Refer to [repology](https://repology.org/project/onedrive/versions) as a source of all 'onedrive' client versions available across tracked distributions. 3. You require code newer than the latest release or are building a Pull Request to validate a bugfix. Outside of these 3 reasons, you should not be building the client yourself. You should endeavour where possible to use a pre-built package. > [!IMPORTANT] > If your distribution does not currently offer a packaged version of the client, you should **request that your distribution maintainers package and support it** as part of their official repositories. ## Building from Source If you need to build the client from source, follow this high-level process: 1. Ensure your system meets the [minimum build requirements](#minimum-build-requirements). 2. Install the necessary build dependencies and a supported D compiler. 3. Clone the repository, configure the build options, compile, and install the client. ### Minimum Build Requirements * For successful compilation of this application, it's crucial that the build environment is equipped with a minimum of 1GB of memory and an additional 1GB of swap space. * Install the required distribution package dependencies covering the required development tools and development libraries for curl, sqlite and dbus where required. * Install the [Digital Mars D Compiler (DMD)](https://dlang.org/download.html), [LDC – the LLVM-based D Compiler](https://github.com/ldc-developers/ldc), or, at least version 15 of the [GNU D Compiler (GDC)](https://www.gdcproject.org/) > [!IMPORTANT] > To compile this application successfully, the minimum supported versions of each compiler are: DMD **2.091.1**, LDC **1.20.1**, and, GDC **15**. Ensuring compatibility and optimal performance necessitates the use of these specific versions or their more recent updates. > > You only need 1 compiler installed. You do not need to install DMD, LDC and GDC. Please *pick* the most applicable compiler for your distribution. #### Installing DMD Compiler To install the DMD Compiler, this can be achieved in the following manner: ```text curl -fsS https://dlang.org/install.sh | bash -s dmd ``` > [!NOTE] > Note the `source ~/dlang/dmd-X.XXX.X/activate` string as this will be needed later when building the client. #### Installing LDC Compiler To install the LDC Compiler, this can be achieved in the following manner: ```text curl -fsS https://dlang.org/install.sh | bash -s ldc ``` > [!NOTE] > Note the `source ~/dlang/ldc-X.XX.X/activate` string as this will be needed later when building the client. #### Installing GDC Compiler You will need at least GDC version 15. If your distribution's repositories include a suitable version, you can install it from there. Common names for the GDC package are listed on the [GDC website](https://www.gdcproject.org/downloads#linux-distribution-packages). If the package is unavailable or its version is too old, you can try building it from source following [these instructions](https://wiki.dlang.org/GDC/Installation). ### Install Build Dependencies (By Distribution) #### Arch Linux | Manjaro Linux ```text sudo pacman -S git make pkg-config curl sqlite dbus ldc ``` For GUI notifications the following is also necessary: ```text sudo pacman -S libnotify ``` #### CentOS 6.x | RHEL 6.x CentOS 6.x and RHEL 6.x reached End of Life status on November 30th 2020 and is no longer supported or tested against. #### CentOS 7.x | RHEL 7.x CentOS 7.x and RHEL 7.x reached End of Life status on June 30th 2024 and is no longer supported or tested against. #### CentOS Stream 8 | CentOS Stream 9 | CentOS Stream 10 ```text sudo dnf groupinstall 'Development Tools' sudo dnf install libcurl-devel sqlite-devel dbus-devel curl -fsS https://dlang.org/install.sh | bash -s dmd ``` For GUI notifications the following is also necessary: ```text sudo dnf install libnotify-devel ``` #### Debian 9 Debian 9 reached the end of its five-year LTS window on July 18th 2020 and is no longer supported or tested against. #### Debian 10 Debian 10 reached the end of its five-year LTS window on September 10th 2022 and is no longer supported or tested against. #### Debian 11 | Debian 12 | Debian 13 | Linux Mint Debian Edition 6 | Linux Mint Debian Edition 7 - x86_64 ```text sudo apt install build-essential sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl systemd-dev libdbus-1-dev curl -fsS https://dlang.org/install.sh | bash -s dmd ``` For GUI notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` #### Debian 11 | Debian 12 | Debian 13 - ARMHF and ARM64 > [!NOTE] > For Debian ARM platforms it is advisable to use the distribution provided 'ldc' package to ensure compiler consistency. ```text sudo apt install build-essential sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl ldc systemd-dev libdbus-1-dev ``` For GUI notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` #### Fedora > [!NOTE] > Fedora 41 and above uses **dnf5** which removes some deprecated aliases, specifically 'groupinstall' in this instance. ```text sudo dnf group install development-tools sudo dnf install libcurl-devel sqlite-devel dbus-devel ``` Before running the dmd install you need to check for the option 'use-keyboxd' in your gnupg common.conf file and comment it out while running the install. ```text curl -fsS https://dlang.org/install.sh | bash -s dmd ``` Or you may get the following error: ```text myuser@fedora:~$ curl -fsS https://dlang.org/install.sh | bash -s dmd Downloading https://dlang.org/d-keyring.gpg ######################################################################## 100.0% gpg: Note: Specified keyrings are ignored due to option "use-keyboxd" gpg: Signature made Thu 06 Mar 2025 10:45:29 GMT gpg: using RSA key F3F896F3274BBD9BBBA59058710592E7FB7AF6CA gpg: Can't check signature: No public key Invalid signature https://dlang.org/d-keyring.gpg.sig ``` For GUI notifications the following is also necessary: ```text sudo dnf install libnotify-devel ``` #### FreeBSD > [!NOTE] > Install the required FreeBSD packages as 'root' unless you have installed 'sudo' > > For FreeBSD it is advisable to use the distribution provided 'ldc' package to ensure compiler consistency. ```text pkg install bash bash-completion gmake pkgconf autoconf automake logrotate libinotify git sqlite3 ldc ``` For GUI notifications the following is also necessary: ```text pkg install libnotify ``` #### Gentoo ```text sudo emerge --onlydeps net-misc/onedrive ``` #### MX Linux 25 ```text sudo apt install build-essential sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl systemd-dev libdbus-1-dev curl -fsS https://dlang.org/install.sh | bash -s dmd ``` For GUI notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` #### OpenSUSE Leap | OpenSUSE Tumbleweed ```text sudo zypper refresh sudo zypper install gcc git libcurl-devel sqlite3-devel dmd phobos-devel phobos-devel-static dbus-1-devel ``` For GUI notifications the following is also necessary: ```text sudo zypper install libnotify-devel ``` #### Raspbian - ARMHF and ARM64 > [!CAUTION] > The minimum LDC compiler version required to compile this application is 1.20.1, which is not available for Debian Buster or distributions based on Debian Buster. You are advised to first upgrade your platform distribution to one that is based on Debian Bullseye (Debian 11) or later. > [!NOTE] > These dependencies were validated using: > * `Linux raspberrypi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64` (2022-01-28-raspios-bullseye-armhf-lite) using Raspberry Pi 3B (revision 1.2) > * `Linux raspberrypi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64` (2022-01-28-raspios-bullseye-arm64-lite) using Raspberry Pi 3B (revision 1.2) > * `Linux ubuntu 5.15.0-1005-raspi #5-Ubuntu SMP PREEMPT Mon Apr 4 12:21:48 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux` (ubuntu-22.04-preinstalled-server-arm64+raspi) using Raspberry Pi 3B (revision 1.2) ```text sudo apt install build-essential sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl ldc systemd-dev libdbus-1-dev ``` For GUI notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` #### RedHat Enterprise Linux (RHEL) 8 | RedHat Enterprise Linux (RHEL) 9 | RedHat Enterprise Linux (RHEL) 10 ```text sudo dnf groupinstall 'Development Tools' sudo dnf install libcurl-devel sqlite-devel dbus-devel curl -fsS https://dlang.org/install.sh | bash -s dmd ``` For GUI notifications the following is also necessary: ```text sudo dnf install libnotify-devel ``` > [!NOTE] > **Make sure repos are enabled/subscribed**. Minimal images/containers sometimes don’t have group metadata; on those, the group may appear “not available” until you enable the right repos (or use a full image). #### Ubuntu 16.x Ubuntu 16.x LTS reached the end of its five-year LTS window on April 30th 2021 and is no longer supported or tested against. #### Ubuntu 18.x Ubuntu 18.x LTS reached the end of its five-year LTS window on May 31th 2023 and is no longer supported or tested against. #### Ubuntu 20.x Ubuntu 20.x LTS reached the end of its five-year LTS window on May 31th 2025 and is no longer supported or tested against. #### Ubuntu 22.x | Ubuntu 24.x > [!NOTE] > These dependency requirements also apply to any distribution derived from Ubuntu, including but not limited to: > * Lubuntu > * Linux Mint > * Pop!_OS > * Peppermint OS > * Zorin OS ```text sudo apt install build-essential sudo apt install libcurl4-openssl-dev libsqlite3-dev pkg-config git curl systemd-dev libdbus-1-dev curl -fsS https://dlang.org/install.sh | bash -s dmd ``` For GUI notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` ## Clone, Configure, Build, Install ### High Level Steps to building the OneDrive Client for Linux The overall process is as follows: 1. Install the required platform dependencies (see above) 2. If necessary, enable your DMD or LDC compiler environment 3. Clone the GitHub repository 4. Run the configure script adding any applicable build options (see below), then build the application 5. Either run the built binary directly from the build directory, or install it system-wide 6. If applicable, deactivate the DMD or LDC compiler environment when finished ### Building the Application Using Default configure Settings #### Building on Linux using DMD, LDC or GDC You must first **activate** the compiler environment before building. For example: ```text source ~/dlang/dmd-2.091.1/activate # or source ~/dlang/ldc-1.20.1/activate ``` This command updates your environment (`PATH`, `LIBRARY_PATH`, `LD_LIBRARY_PATH`, etc.) so that the correct compiler is available. If you skip this step, the build will fail because the compiler will not be found. > [!NOTE] > Replace the `source` string with the compiler environment activation string displayed when you installed the relevant compiler. Once the compiler is activated, clone, build and install the client: ```text git clone https://github.com/abraunegg/onedrive.git cd onedrive ./configure make clean; make; sudo make install deactivate ``` > [!NOTE] > If using GDC ≥ 15, specify it explicitly when configuring the application: > ```text > ./configure DC=gdc > ``` #### Building on FreeBSD using gmake ```text git clone https://github.com/abraunegg/onedrive.git cd onedrive ./configure gmake clean; gmake; gmake install ``` > [!NOTE] > Build and install the application as 'root' unless you have installed 'sudo' #### Building on ARM | Raspberry Pi > [!CAUTION] > The minimum LDC compiler version required to compile this application is 1.20.1, which is not available for Debian Buster or distributions based on Debian Buster. You are advised to first upgrade your platform distribution to one that is based on Debian Bullseye (Debian 11) or later. > [!IMPORTANT] > For successful compilation of this application, it's crucial that the build environment is equipped with a minimum of 1GB of memory and an additional 1GB of swap space. To verify your system's swap space availability, you can use the `swapon` command. Ensuring these requirements are met is vital for the application's compilation process. > [!NOTE] > The `configure` step will detect the correct version of LDC to be used when compiling the client under ARMHF and ARM64 CPU architectures. ```text git clone https://github.com/abraunegg/onedrive.git cd onedrive ./configure; make clean; make; sudo make install ``` ### Build Options for Customising the Application The `configure` script provides several options that allow you to tailor the build to your needs. These options can be used to enable or adjust specific features in the client, including: * Enabling GUI desktop notifications * Enabling shell completion support * Enabling internal debugging to assist with troubleshooting and performance analysis * Specifying a custom systemd service installation directory #### Build Option: Enable GUI Desktop Notifications To enable GUI notification support, include the `--enable-notifications` option when running `configure`, for example: ```text ./configure --enable-notifications ``` Enabling this option allows the client to send GUI notifications through the Display Manager via the DBus interface. > [!TIP] > Package maintainers are encouraged to enable this option. > > When this option is enabled, the client automatically checks at runtime whether GUI notifications can be delivered via the Display Manager through the DBus interface. If this option is **not** enabled, GUI notifications are **disabled**. #### Build Option: Enable Shell Completion Support To enable command-line shell completions, include the `--enable-completions` option when running `configure`, for example: ```text ./configure --enable-completions ``` When enabled, completion scripts will be installed for **bash**, **zsh**, and **fish** shells. By default, the installation directories are detected automatically. If needed, you can manually specify the installation paths using the following options: ```text --with-bash-completion-dir= --with-zsh-completion-dir= --with-fish-completion-dir= ``` > [!TIP] > Package maintainers are encouraged to enable this option. #### Build Option: Enabling internal debugging To enable internal debugging support, include the `--enable-debug` option when running `configure`, for example: ```text ./configure --enable-debug ``` Enabling this option builds the client with additional debug symbols outside of creating a separate debug package build. This is particularly useful when investigating performance issues (e.g. with `perf`) or diagnosing application crashes. **What difference does this make?** Without this option, if the application encounters a crash, the stack trace may contain unresolved symbols, often shown as `??:??`, which makes identifying the cause very difficult. With `--enable-debug` enabled, the resulting crash stack trace includes full source file and line information. This allows the issue to be located and isolated quickly and accurately. > [!TIP] > Package maintainers are encouraged to enable this option. #### Build Option: Customising the Systemd Service Installation Directory By default, systemd service files are installed into the directories detected via `pkg-config --variable=systemdsystemunitdir systemd` and related settings. If you need to override these locations, specify one or both of the following options when running `configure`: ```text --with-systemdsystemunitdir= # System-wide service unit directory --with-systemduserunitdir= # User-level service unit directory ``` To **disable** installation of a service file entirely, pass `no` as the directory value. For example: ```text ./configure --with-systemduserunitdir=no ``` This prevents the corresponding service unit from being installed. ## Upgrading the Client > [!CAUTION] > Before starting any upgrade, **stop any running systemd service for the client**. This ensures the service is restarted using the updated binary. How you upgrade depends on how the client was originally installed: ### If installed from a distribution package When the package maintainer publishes an updated version, the client will be upgraded automatically as part of your normal system package updates (e.g., `apt upgrade`, `dnf upgrade`, `zypper up`, etc.). ### If installed from source To upgrade a source-built installation, the recommended approach is: 1. Uninstall the existing client (see instructions below). 2. Re-clone the repository. 3. Re-compile and re-install the new version. > [!NOTE] > The uninstall process removes all components, including systemd service files. > If you created custom systemd unit files (e.g., for SharePoint library access), you will need to recreate or restore them after re-installation. You **may** choose to skip the uninstall step and simply re-compile and re-install over the top. However, this risks leaving **multiple** `onedrive` **binaries** on your system. Depending on your system `PATH`, the wrong binary may be executed. After installation, verify the version in use: ```text onedrive --version ``` This confirms that the upgrade was successful. ## Uninstalling the client How to uninstall depends on how the client was installed. ### If installed from a distribution package Uninstall the client using your distribution’s package management tools. Refer to your distribution’s documentation for the correct removal command (e.g. `apt remove`, `dnf remove`, `zypper remove`, etc.). ### If installed from source If you built and installed the client from a GitHub clone, run the following command from within the cloned repository directory: ```text sudo make uninstall ``` This removes the installed `onedrive` binary and associated system files. #### Optional: Remove client configuration and state If you do not plan to upgrade or reinstall and wish to remove all client data, run: ```text rm -rf ~/.config/onedrive ``` > [!IMPORTANT] > If you used the `--confdir` option, replace `~/.config/onedrive` with the custom configuration directory you specified. #### Optional: Remove only the application key If you want to retain your items database but remove the stored authentication token, run: ```text rm -f ~/.config/onedrive/refresh_token ``` This preserves sync state while requiring re-authentication on next run. ================================================ FILE: docs/known-issues.md ================================================ # List of Identified Known Issues The following points detail known issues associated with this client: ## Renaming or Moving Files in Standalone Mode causes online deletion and re-upload to occur **Issue Tracker:** [#876](https://github.com/abraunegg/onedrive/issues/876), [#2579](https://github.com/abraunegg/onedrive/issues/2579) **Summary:** Renaming or moving files and/or folders while using the standalone sync option `--sync` this results in unnecessary data deletion online and subsequent re-upload. **Detailed Description:** In standalone mode (`--sync`), the renaming or moving folders locally that have already been synchronized leads to the data being deleted online and then re-uploaded in the next synchronization process. **Technical Explanation:** This behavior is expected from the client under these specific conditions. Renaming or moving files is interpreted as deleting them from their original location and creating them in a new location. In standalone sync mode, the client lacks the capability to track file system changes (including renames and moves) that occur when it is not running. This limitation is the root cause of the observed 'deletion and re-upload' cycle. **Recommended Workaround:** For effective tracking of file and folder renames or moves to new local directories, it is recommended to run the client in service mode (`--monitor`) rather than in standalone mode. This approach allows the client to immediately process these changes, enabling the data to be updated (renamed or moved) in the new location on OneDrive without undergoing deletion and re-upload. ## Application 'stops' running without any visible reason **Issue Tracker:** [#494](https://github.com/abraunegg/onedrive/issues/494), [#753](https://github.com/abraunegg/onedrive/issues/753), [#792](https://github.com/abraunegg/onedrive/issues/792), [#884](https://github.com/abraunegg/onedrive/issues/884), [#1162](https://github.com/abraunegg/onedrive/issues/1162), [#1408](https://github.com/abraunegg/onedrive/issues/1408), [#1520](https://github.com/abraunegg/onedrive/issues/1520), [#1526](https://github.com/abraunegg/onedrive/issues/1526) **Summary:** Users experience sudden shutdowns in a client application during file transfers with Microsoft's Europe Data Centers, likely due to unstable internet or HTTPS inspection issues. This problem, often signaled by an error code of 141, is related to the application's reliance on Curl and OpenSSL. Resolution steps include system updates, seeking support from OS vendors, ISPs, OpenSSL/Curl teams, and providing detailed debug logs to Microsoft for analysis. **Detailed Description:** The application unexpectedly stops functioning during upload or download operations when using the client. This issue occurs without any apparent reason. Running `echo $?` after the unexpected exit may return an error code of 141. This problem predominantly arises when the client interacts with Microsoft's Europe Data Centers. **Technical Explanation:** The client heavily relies on Curl and OpenSSL for operations with the Microsoft OneDrive service. A common observation during this error is an entry in the HTTPS Debug Log stating: ``` OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104 ``` To confirm this as the root cause, a detailed HTTPS debug log can be generated with these commands: ``` --verbose --verbose --debug-https ``` This error typically suggests one of the following issues: * An unstable internet connection between the user and the OneDrive service. * An issue with HTTPS transparent inspection services that monitor the traffic en route to the OneDrive service. **Recommended Resolution Steps:** Recommended steps to address this issue include: * Updating your operating system to the latest version. * Configure the application to only use HTTP/1.1 * Configure the application to use IPv4 only. * Upgrade your 'curl' application to the latest available from the curl developers. * Seeking assistance from your OS vendor. * Contacting your Internet Service Provider (ISP) or your IT Help Desk. * Reporting the issue to the OpenSSL and/or Curl teams for improved handling of such connection failures. * Creating a HTTPS Debug Log during the issue and submitting a support request to Microsoft with the log for their analysis. For more in-depth SSL troubleshooting, please read: https://maulwuff.de/research/ssl-debugging.html ## AADSTS70000 returned during initial authorisation or re-authentication **Summary:** During initial authentication or when running `onedrive --reauth`, the client fails with: ``` AADSTS70000: The provided value for the 'code' parameter is not valid ``` This issue is **not a client bug** and is caused by the authorisation code being invalid at the time it is redeemed. **Detailed Description:** When authenticating, the user is redirected to a Microsoft login page in their web browser. After successful consent, the browser is redirected to a URL of the form: ``` https://login.microsoftonline.com/common/oauth2/nativeclient?code= ``` The user must copy this URL and paste it back into the CLI when prompted. Microsoft authorisation codes are single-use and short-lived. If the code is altered, reused, expired, or otherwise invalidated before the client redeems it, Microsoft Entra ID returns AADSTS70000. **Technical Explanation:** The most common cause is **browser-side interference** with the redirect URL before the user copies it. Privacy and security tooling (such as ad-blockers, URL sanitisation, or “remove tracking parameters” features) can modify or invalidate the `code` query parameter. Other contributing factors include: * Copying the wrong URL (for example, not copying directly from the browser address bar immediately after consent) * Refreshing the page or attempting to reuse the same redirect URI * Waiting too long before pasting the redirect URI back into the CLI Once an authorisation code is invalid, it **cannot** be reused or recovered. **Recommended Resolution Steps:** 1. Re-run authentication using: ``` onedrive --reauth ``` 2. Use a private/incognito browser session or a clean browser profile 3. Temporarily disable browser extensions or privacy features that modify URLs for the Microsoft login pages (for example: uBlock Origin, ClearURLs, Brave Shields) 4. Complete the browser consent flow and immediately copy the redirect URI from the address bar and paste it into the CLI **Additional Notes:** For security reasons, users should **never post full redirect URIs** (they contain sensitive authorisation codes). Any such URLs must be redacted when shared in logs, issues, or support requests. ================================================ FILE: docs/national-cloud-deployments.md ================================================ # How to configure access to specific Microsoft Azure deployments > [!CAUTION] > Before reading this document, please ensure you are running application version [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) or greater. Use `onedrive --version` to determine what application version you are using and upgrade your client if required. ## Process Overview In some cases it is a requirement to utilise specific Microsoft Azure cloud deployments to conform with data and security requirements that requires data to reside within the geographic borders of that country. Current national clouds that are supported are: * Microsoft Cloud for US Government * Microsoft Cloud Germany * Azure and Office365 operated by VNET in China In order to successfully use these specific Microsoft Azure deployments, the following steps are required: 1. Register an application with the Microsoft identity platform using the Azure portal 2. Configure the new application with the appropriate authentication scopes 3. Validate that the authentication / redirect URI is correct for your application registration 4. Configure the onedrive client to use the new application id as provided during application registration 5. Configure the onedrive client to use the right Microsoft Azure deployment region that your application was registered with 6. Authenticate the client ## Step 1: Register a new application with Microsoft Azure 1. Log into your applicable Microsoft Azure Portal with your applicable Office365 identity: | National Cloud Environment | Microsoft Azure Portal | |---|---| | Microsoft Cloud for US Government | https://portal.azure.com/ | | Microsoft Cloud Germany | https://portal.azure.com/ | | Azure and Office365 operated by VNET | https://portal.azure.cn/ | 2. Select 'Azure Active Directory' as the service you wish to configure 3. Under 'Manage', select 'App registrations' to register a new application 4. Click 'New registration' 5. Type in the appropriate details required as per below: ![application_registration](./images/application_registration.jpg) 6. To save the application registration, click 'Register' and something similar to the following will be displayed: ![application_registration_done](./images/application_registration_done.jpg) > [!NOTE] > The Application (client) ID UUID as displayed after client registration, is what is required as the 'application_id' for Step 4 below. ## Step 2: Configure application authentication scopes Configure the API permissions as per the following: | API / Permissions name | Type | Description | Admin consent required | |---|---|---|---| | Files.ReadWrite | Delegated | Have full access to user files | No | | Files.ReadWrite.All | Delegated | Have full access to all files user can access | No | | Sites.ReadWrite.All | Delegated | Have full access to all items in all site collections | No | | offline_access | Delegated | Maintain access to data you have given it access to | No | ![authentication_scopes](./images/authentication_scopes.jpg) ## Step 3: Validate that the authentication / redirect URI is correct Add the appropriate redirect URI for your Azure deployment: ![authentication_response_uri](./images/authentication_response_uri.jpg) A valid entry for the response URI should be one of: * https://login.microsoftonline.us/common/oauth2/nativeclient (Microsoft Cloud for US Government) * https://login.microsoftonline.de/common/oauth2/nativeclient (Microsoft Cloud Germany) * https://login.chinacloudapi.cn/common/oauth2/nativeclient (Azure and Office365 operated by VNET in China) For a single-tenant application, it may be necessary to use your specific tenant id instead of "common": * https://login.microsoftonline.us/example.onmicrosoft.us/oauth2/nativeclient (Microsoft Cloud for US Government) * https://login.microsoftonline.de/example.onmicrosoft.de/oauth2/nativeclient (Microsoft Cloud Germany) * https://login.chinacloudapi.cn/example.onmicrosoft.cn/oauth2/nativeclient (Azure and Office365 operated by VNET in China) ## Step 4: Configure the onedrive client to use new application registration Update to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following: ```text application_id = "insert valid entry here" ``` This will reconfigure the client to use the new application registration you have created. **Example:** ```text application_id = "22c49a0d-d21c-4792-aed1-8f163c982546" ``` ## Step 5: Configure the onedrive client to use the specific Microsoft Azure deployment Update to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following: ```text azure_ad_endpoint = "insert valid entry here" ``` Valid entries are: * USL4 (Microsoft Cloud for US Government) * USL5 (Microsoft Cloud for US Government - DOD) * DE (Microsoft Cloud Germany) * CN (Azure and Office365 operated by VNET in China) This will configure your client to use the correct Azure AD and Graph endpoints as per [https://docs.microsoft.com/en-us/graph/deployments](https://docs.microsoft.com/en-us/graph/deployments) **Example:** ```text azure_ad_endpoint = "USL4" ``` If the Microsoft Azure deployment does not support multi-tenant applications, update to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following: ```text azure_tenant_id = "insert valid entry here" ``` This will configure your client to use the specified tenant id in its Azure AD and Graph endpoint URIs, instead of "common". The tenant id may be the GUID Directory ID (formatted "00000000-0000-0000-0000-000000000000"), or the fully qualified tenant name (e.g. "example.onmicrosoft.us"). The GUID Directory ID may be located in the Azure administration page as per [https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id](https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id). Note that you may need to go to your national-deployment-specific administration page, rather than following the links within that document. The tenant name may be obtained by following the PowerShell instructions on [https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id](https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id); it is shown as the "TenantDomain" upon completion of the "Connect-AzureAD" command. **Example:** ```text azure_tenant_id = "example.onmicrosoft.us" # or azure_tenant_id = "0c4be462-a1ab-499b-99e0-da08ce52a2cc" ``` ## Step 6: Authenticate the client Run the application without any additional command switches. You will be asked to open a specific URL by using your web browser where you will have to login into your Microsoft Account and give the application the permission to access your files. After giving permission to the application, you will be redirected to a blank page. Copy the URI of the blank page into the application. ```text [user@hostname ~]$ onedrive Authorize this app visiting: https://..... Enter the response uri: ``` **Example:** ``` [user@hostname ~]$ onedrive Authorize this app visiting: https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=22c49a0d-d21c-4792-aed1-8f163c982546&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.ReadWrite.All%20offline_access&response_type=code&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient Enter the response uri: https://login.microsoftonline.com/common/oauth2/nativeclient?code= Application has been successfully authorised, however no additional command switches were provided. Please use --help for further assistance in regards to running this application. ``` ================================================ FILE: docs/podman.md ================================================ # Run the OneDrive Client for Linux under Podman This client can be run as a Podman container, with 3 available container base options for you to choose from: | Container Base | Docker Tag | Description | i686 | x86_64 | ARMHF | AARCH64 | |----------------|-------------|----------------------------------------------------------------|:------:|:------:|:-----:|:-------:| | Alpine Linux | edge-alpine | Podman container based on Alpine 3.23 using 'master' |❌|✔|❌|✔| | Alpine Linux | alpine | Podman container based on Alpine 3.23 using latest release |❌|✔|❌|✔| | Debian | debian | Podman container based on Debian 13 using latest release |✔|✔|✔|✔| | Debian | edge | Podman container based on Debian 13 using 'master' |✔|✔|✔|✔| | Debian | edge-debian | Podman container based on Debian 13 using 'master' |✔|✔|✔|✔| | Debian | latest | Podman container based on Debian 13 using latest release |✔|✔|✔|✔| | Fedora | edge-fedora | Podman container based on Fedora 43 using 'master' |❌|✔|❌|✔| | Fedora | fedora | Podman container based on Fedora 43 using latest release |❌|✔|❌|✔| These containers offer a simple monitoring-mode service for the OneDrive Client for Linux. The instructions below have been validated on: * Fedora 40 The instructions below will utilise the 'edge' tag, however this can be substituted for any of the other docker tags such as 'latest' from the table above if desired. The 'edge' Docker Container will align closer to all documentation and features, where as 'latest' is the release version from a static point in time. The 'latest' tag however may contain bugs and/or issues that will have been fixed, and those fixes are contained in 'edge'. Additionally there are specific version release tags for each release. Refer to https://hub.docker.com/r/driveone/onedrive/tags for any other Docker tags you may be interested in. > [!NOTE] > The below instructions for podman has been tested and validated when logging into the system as an unprivileged user (non 'root' user). ## High Level Configuration Steps 1. Install 'podman' as per your distribution platform's instructions if not already installed. 2. Disable 'SELinux' as per your distribution platform's instructions 3. Test 'podman' by running a test container 4. Prepare the required podman volumes to store the configuration and data 5. Run the 'onedrive' container and perform authorisation 6. Running the 'onedrive' container under 'podman' ## Configuration Steps ### 1. Install 'podman' on your platform Install 'podman' as per your distribution platform's instructions if not already installed. ### 2. Disable SELinux on your platform In order to run the Docker container under 'podman', SELinux must be disabled. Without doing this, when the application is authenticated in the steps below, the following error will be presented: ```text ERROR: The local file system returned an error with the following message: Error Message: /onedrive/conf/refresh_token: Permission denied The database cannot be opened. Please check the permissions of ~/.config/onedrive/items.sqlite3 ``` The only known work-around for the above problem at present is to disable SELinux. Please refer to your distribution platform's instructions on how to perform this step. * Fedora: https://docs.fedoraproject.org/en-US/quick-docs/selinux-changing-states-and-modes/#_disabling_selinux * Red Hat Enterprise Linux: https://access.redhat.com/solutions/3176 Post disabling SELinux and reboot your system, confirm that `getenforce` returns `Disabled`: ```text $ getenforce Disabled ``` If you are still experiencing permission issues despite disabling SELinux, please read https://www.redhat.com/sysadmin/container-permission-denied-errors ### 3. Test 'podman' on your platform Test that 'podman' is operational for your 'non-root' user, as per below: ```bash [alex@fedora40-podman ~]$ podman pull fedora Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull registry.fedoraproject.org/fedora:latest... Getting image source signatures Copying blob b30887322388 done | Copying config a1cd3cbf8a done | Writing manifest to image destination a1cd3cbf8adaa422629f2fcdc629fd9297138910a467b11c66e5ddb2c2753dff [alex@fedora40-podman ~]$ podman run fedora /bin/echo "Welcome to the Podman World" Welcome to the Podman World [alex@fedora40-podman ~]$ ``` ### 4. Configure the required podman volumes The 'onedrive' Docker container requires 2 podman volumes to operate: * Config Volume * Data Volume The first volume is the configuration volume that stores all the applicable application configuration + current runtime state. In a non-containerised environment, this normally resides in `~/.config/onedrive` - in a containerised environment this is stored in the volume tagged as `/onedrive/conf` The second volume is the data volume, where all your data from Microsoft OneDrive is stored locally. This volume is mapped to an actual directory point on your local filesystem and this is stored in the volume tagged as `/onedrive/data` #### 4.1 Prepare the 'config' volume Create the 'config' volume with the following command: ```bash podman volume create onedrive_conf ``` This will create a podman volume labeled `onedrive_conf`, where all configuration of your onedrive account will be stored. You can add a custom config file in this location at a later point in time if required. #### 4.2 Prepare the 'data' volume Create the 'data' volume with the following command: ```bash podman volume create onedrive_data ``` This will create a podman volume labeled `onedrive_data` and will map to a path on your local filesystem. This is where your data from Microsoft OneDrive will be stored. Keep in mind that: * The owner of this specified folder must not be root * Podman will attempt to change the permissions of the volume to the user the container is configured to run as > [!IMPORTANT] > Issues occur when this target folder is a mounted folder of an external system (NAS, SMB mount, USB Drive etc) as the 'mount' itself is owed by 'root'. If this is your use case, you *must* ensure your normal user can mount your desired target without having the target mounted by 'root'. If you do not fix this, your Podman container will fail to start with the following error message: > ```bash > ROOT level privileges prohibited! > ``` ### 5. First run of Docker container under podman and performing authorisation The 'onedrive' client within the container first needs to be authorised with your Microsoft account. This is achieved by initially running podman in interactive mode. Run the podman image with the commands below and make sure to change the value of `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `export ONEDRIVE_DATA_DIR="/home/abraunegg/OneDrive"`). > [!IMPORTANT] > The 'target' folder of `ONEDRIVE_DATA_DIR` must exist before running the podman container. The script below will create 'ONEDRIVE_DATA_DIR' so that it exists locally for the podman volume mapping to occur. It is also a requirement that the container be run using a non-root uid and gid, you must insert a non-root UID and GID (e.g.` export ONEDRIVE_UID=1000` and export `ONEDRIVE_GID=1000`). The script below will use `id` to evaluate your system environment to use the correct values. ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR} podman run -it --name onedrive --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" \ -v onedrive_conf:/onedrive/conf:U,Z \ -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" \ driveone/onedrive:edge ``` > [!IMPORTANT] > In some scenarios, 'podman' sets the configuration and data directories to a different UID & GID as specified. To resolve this situation, you must run 'podman' with the `--userns=keep-id` flag to ensure 'podman' uses the UID and GID as specified. The updated script example when using `--userns=keep-id` is below: ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR} podman run -it --name onedrive --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" \ --userns=keep-id \ -v onedrive_conf:/onedrive/conf:U,Z \ -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" \ driveone/onedrive:edge ``` > [!IMPORTANT] > If you plan to use the 'podman' built in auto-updating of container images described in 'Systemd Service & Auto Updating' below, you must pass an additional argument to set a label during the first run. The updated script example to support auto-updating of container images is below: ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR} podman run -it --name onedrive --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" \ --userns=keep-id \ -v onedrive_conf:/onedrive/conf:U,Z \ -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" \ -e PODMAN=1 \ --label "io.containers.autoupdate=image" \ driveone/onedrive:edge ``` When the Podman container successfully starts: * You will be asked to open a specific link using your web browser * Login to your Microsoft Account and give the application the permission * After giving the permission, you will be redirected to a blank page * Copy the URI of the blank page into the application prompt to authorise the application Once the 'onedrive' application is authorised, the client will automatically start monitoring your `ONEDRIVE_DATA_DIR` for data changes to be uploaded to OneDrive. Files stored on OneDrive will be downloaded to this location. If the client is working as expected, you can detach from the container with Ctrl+p, Ctrl+q. ### 6. Running the 'onedrive' container under 'podman' #### 6.1 Check if the monitor service is running ```bash podman ps -f name=onedrive ``` #### 6.2 Show 'onedrive' runtime logs ```bash podman logs onedrive ``` #### 6.3 Stop running 'onedrive' container ```bash podman stop onedrive ``` #### 6.4 Start 'onedrive' container ```bash podman start onedrive ``` #### 6.5 Remove 'onedrive' container ```bash podman rm -f onedrive ``` ## Advanced Usage ### Systemd Service & Auto Updating Podman supports running containers as a systemd service and also auto updating of the container images. Using the existing running container you can generate a systemd unit file to be installed by the **root** user. To have your container image auto-update with podman, it must first be created with the label `"io.containers.autoupdate=image"` mentioned in step 5 above. ``` cd /tmp podman generate systemd --new --restart-policy on-failure --name -f onedrive /tmp/container-onedrive.service # copy the generated systemd unit file to the systemd path and reload the daemon cp -Z ~/container-onedrive.service /usr/lib/systemd/system systemctl daemon-reload #optionally enable it to startup on boot systemctl enable container-onedrive.service #check status systemctl status container-onedrive #start/stop/restart container as a systemd service systemctl stop container-onedrive systemctl start container-onedrive ``` To update the image using podman (Ad-hoc) ``` podman auto-update ``` To update the image using systemd (Automatic/Scheduled) ``` # Enable the podman-auto-update.timer service at system start: systemctl enable podman-auto-update.timer # Start the service systemctl start podman-auto-update.timer # Containers with the autoupdate label will be updated on the next scheduled timer systemctl list-timers --all ``` ### Editing the running configuration and using a 'config' file The 'onedrive' client should run in default configuration, however you can change this default configuration by placing a custom config file in the `onedrive_conf` podman volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config) Then put it into your onedrive_conf volume path, which can be found with: ```bash podman volume inspect onedrive_conf ``` Or you can map your own config folder to the config volume. Make sure to copy all files from the volume into your mapped folder first. The detailed document for the config can be found here: [Application Configuration Options for the OneDrive Client for Linux](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md) ### Syncing multiple accounts There are many ways to do this, the easiest is probably to do the following: 1. Create a second podman config volume (replace `work` with your desired name): `podman volume create onedrive_conf_work` 2. And start a second podman monitor container (again replace `work` with your desired name): ```bash export ONEDRIVE_DATA_DIR_WORK="/home/abraunegg/OneDriveWork" export ONEDRIVE_UID=`id -u` export ONEDRIVE_GID=`id -g` mkdir -p ${ONEDRIVE_DATA_DIR_WORK} podman run -it --name onedrive_work --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" \ --userns=keep-id \ -v onedrive_conf_work:/onedrive/conf:U,Z \ -v "${ONEDRIVE_DATA_DIR_WORK}:/onedrive/data:U,Z" \ -e PODMAN=1 \ --label "io.containers.autoupdate=image" \ driveone/onedrive:edge ``` ## Supported Podman Environment Variables | Variable | Purpose | Sample Value | | ---------------- | --------------------------------------------------- |:-------------:| | ONEDRIVE_UID | UserID (UID) to run as | 1000 | | ONEDRIVE_GID | GroupID (GID) to run as | 1000 | | ONEDRIVE_VERBOSE | Controls "--verbose" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DEBUG | Controls "--verbose --verbose" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DEBUG_HTTPS | Controls "--debug-https" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_RESYNC | Controls "--resync" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_DOWNLOADONLY | Controls "--download-only" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_CLEANUPLOCAL | Controls "--cleanup-local-files" to cleanup local files and folders if they are removed online. Default is 0 | 1 | | ONEDRIVE_UPLOADONLY | Controls "--upload-only" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_NOREMOTEDELETE | Controls "--no-remote-delete" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_LOGOUT | Controls "--logout" switch. Default is 0 | 1 | | ONEDRIVE_REAUTH | Controls "--reauth" switch. Default is 0 | 1 | | ONEDRIVE_AUTHFILES | Controls "--auth-files" option. Default is "" | Please read [CLI Option: --auth-files](./application-config-options.md#cli-option---auth-files) | | ONEDRIVE_AUTHRESPONSE | Controls "--auth-response" option. Default is "" | Please read [CLI Option: --auth-response](./application-config-options.md#cli-option---auth-response) | | ONEDRIVE_DISPLAY_CONFIG | Controls "--display-running-config" switch on onedrive sync. Default is 0 | 1 | | ONEDRIVE_SINGLE_DIRECTORY | Controls "--single-directory" option. Default = "" | "mydir" | | ONEDRIVE_DRYRUN | Controls "--dry-run" option. Default is 0 | 1 | | ONEDRIVE_DISABLE_DOWNLOAD_VALIDATION | Controls "--disable-download-validation" option. Default is 0 | 1 | | ONEDRIVE_DISABLE_UPLOAD_VALIDATION | Controls "--disable-upload-validation" option. Default is 0 | 1 | | ONEDRIVE_SYNC_SHARED_FILES | Controls "--sync-shared-files" option. Default is 0 | 1 | | ONEDRIVE_RUNAS_ROOT | Controls if the Docker container should be run as the 'root' user instead of 'onedrive' user. Default is 0 | 1 | | ONEDRIVE_SYNC_ONCE | Controls if the Docker container should be run in Standalone Mode. It will use Monitor Mode otherwise. Default is 0 | 1 | | ONEDRIVE_FILE_FRAGMENT_SIZE | Controls the fragment size when uploading large files to Microsoft OneDrive. The value specified is in MB. Default is 10, Limit is 60 | 25 | | ONEDRIVE_THREADS | Controls the value for the number of worker threads used for parallel upload and download operations. Default is 8, Limit is 16 | 4 | ### Environment Variables Usage Examples **Verbose Output:** ```bash podman run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Debug Output:** ```bash podman run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Perform a --resync:** ```bash podman run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Perform a --resync and --verbose:** ```bash podman run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Perform a --logout:** ```bash podman run -it -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Perform a --logout and re-authenticate:** ```bash podman run -it -e ONEDRIVE_REAUTH=1 -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` **Perform a sync using ONEDRIVE_SINGLE_DIRECTORY:** ```bash podman run -e ONEDRIVE_SINGLE_DIRECTORY="path/which/needs/to/be/synced" -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" driveone/onedrive:edge ``` > [!IMPORTANT] > Is using a Podman Environment Variable that requires you to specify a 'path' (ONEDRIVE_AUTHFILES, ONEDRIVE_AUTHRESPONSE, ONEDRIVE_SINGLE_DIRECTORY), the placement of quotes around the path is critically important. > > Please ensure you are formatting the option correctly: >``` > -e OPTION="path/which/needs/to/be/synced" >``` > Please also ensure that the path specified complies with the actual application usage argument. Please read the relevant config option advice in the [CLI Option Documentation](./application-config-options.md) ## Building a custom Podman image You can also build your own image instead of pulling the one from [hub.docker.com](https://hub.docker.com/r/driveone/onedrive): ```bash git clone https://github.com/abraunegg/onedrive cd onedrive podman build . -t local-onedrive -f contrib/docker/Dockerfile ``` There are alternate, smaller images available by building Dockerfile-debian or Dockerfile-alpine. These [multi-stage builder pattern](https://docs.docker.com/develop/develop-images/multistage-build/) Dockerfiles require Docker version at least 17.05. ### How to build and run a custom Podman image based on Debian ``` bash podman build . -t local-onedrive-debian -f contrib/docker/Dockerfile-debian podman run -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" --userns=keep-id local-onedrive-debian:latest ``` ### How to build and run a custom Podman image based on Alpine Linux ``` bash podman build . -t local-onedrive-alpine -f contrib/docker/Dockerfile-alpine podman run -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" --userns=keep-id local-onedrive-alpine:latest ``` ### How to build and run a custom Podman image for ARMHF (Raspberry Pi) Compatible with: * Raspberry Pi * Raspberry Pi 2 * Raspberry Pi Zero * Raspberry Pi 3 * Raspberry Pi 4 ``` bash podman build . -t local-onedrive-armhf -f contrib/docker/Dockerfile-debian podman run -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" --userns=keep-id local-onedrive-armhf:latest ``` ### How to build and run a custom Podman image for AARCH64 Platforms ``` bash podman build . -t local-onedrive-aarch64 -f contrib/docker/Dockerfile-debian podman run -v onedrive_conf:/onedrive/conf:U,Z -v "${ONEDRIVE_DATA_DIR}:/onedrive/data:U,Z" --user "${ONEDRIVE_UID}:${ONEDRIVE_GID}" --userns=keep-id local-onedrive-aarch64:latest ``` ================================================ FILE: docs/privacy-policy.md ================================================ # Privacy Policy Effective Date: May 16 2018 ## Introduction This Privacy Policy outlines how OneDrive Client for Linux ("we," "our," or "us") collects, uses, and protects information when you use our software ("OneDrive Client for Linux"). We respect your privacy and are committed to ensuring the confidentiality and security of any information you provide while using the Software. ## Information We Do Not Collect We want to be transparent about the fact that we do not collect any personal data, usage data, or tracking data through the Software. This means: 1. **No Personal Data**: We do not collect any information that can be used to personally identify you, such as your name, email address, phone number, or physical address. 2. **No Usage Data**: We do not collect data about how you use the Software, such as the features you use, the duration of your sessions, or any interactions within the Software. 3. **No Tracking Data**: We do not use cookies or similar tracking technologies to monitor your online behavior or track your activities across websites or apps. ## How We Use Your Information Since we do not collect any personal, usage, or tracking data, there is no information for us to use for any purpose. ## Third-Party Services The Software may include links to third-party websites or services, but we do not have control over the privacy practices or content of these third-party services. We encourage you to review the privacy policies of any third-party services you access through the Software. ## Children's Privacy Since we do not collect any personal, usage, or tracking data, there is no restriction on the use of this application by anyone under the age of 18. ## Information You Choose to Share While we do not collect personal data, usage data, or tracking data through the Software, there may be instances where you voluntarily choose to share information with us, particularly when submitting bug reports. These bug reports may contain sensitive information such as account details, file names, and directory names. It's important to note that these details are included in the logs and debug logs solely for the purpose of diagnosing and resolving technical issues with the Software. We want to emphasize that, even in these cases, we do not have access to your actual data. The logs and debug logs provided in bug reports are used exclusively for technical troubleshooting and debugging purposes. We take measures to treat this information with the utmost care, and it is only accessible to our technical support and development teams. We do not use this information for any other purpose, and we have strict security measures in place to protect it. ## Protecting Your Sensitive Data We are committed to safeguarding your sensitive data and maintaining its confidentiality. To ensure its protection: 1. **Limited Access**: Only authorized personnel within our technical support and development teams have access to the logs and debug logs containing sensitive data, and they are trained in handling this information securely. 2. **Data Encryption**: We use industry-standard encryption protocols to protect the transmission and storage of sensitive data. 3. **Data Retention**: We retain bug report data for a limited time necessary for resolving the reported issue. Once the issue is resolved, we promptly delete or anonymize the data. 4. **Security Measures**: We employ robust security measures to prevent unauthorized access, disclosure, or alteration of sensitive data. By submitting a bug report, you acknowledge and consent to the inclusion of sensitive information in logs and debug logs for the sole purpose of addressing technical issues with the Software. ## Your Responsibilities While we take measures to protect your sensitive data, it is essential for you to exercise caution when submitting bug reports. Please refrain from including any sensitive or personally identifiable information that is not directly related to the technical issue you are reporting. You have the option to redact or obfuscate sensitive details in bug reports to further protect your data. ## Changes to this Privacy Policy We may update this Privacy Policy from time to time to reflect changes in our practices or for other operational, legal, or regulatory reasons. We will notify you of any material changes by posting the updated Privacy Policy on our website or through the Software. We encourage you to review this Privacy Policy periodically. ## Contact Us If you have any questions or concerns about this Privacy Policy or our privacy practices, please contact us at support@mynas.com.au or via GitHub (https://github.com/abraunegg/onedrive) ## Conclusion By using the Software, you agree to the terms outlined in this Privacy Policy. If you do not agree with any part of this policy, please discontinue the use of the Software. ================================================ FILE: docs/puml/applyPotentiallyChangedItem.puml ================================================ @startuml start partition "applyPotentiallyChangedItem" { :Check if existing item path differs from changed item path; if (itemWasMoved) then (yes) :Log moving item; if (destination exists) then (yes) if (item in database) then (yes) :Check if item is synced; if (item is synced) then (yes) :Log destination is in sync; else (no) :Log destination occupied with a different item; :Backup conflicting file; note right: Local data loss prevention endif else (no) :Log destination occupied by an un-synced file; :Backup conflicting file; note right: Local data loss prevention endif endif :Try to rename path; if (dry run) then (yes) :Track as faked id item; :Track path not renamed; else (no) :Rename item; :Flag item as moved; if (item is a file) then (yes) :Set local timestamp to match online; endif endif else (no) endif :Check if eTag changed; if (eTag changed) then (yes) if (item is a file and not moved) then (yes) :Decide if to download based on hash; else (no) :Update database; endif else (no) :Update database if timestamp differs or in specific operational mode; endif } stop @enduml ================================================ FILE: docs/puml/applyPotentiallyNewLocalItem.puml ================================================ @startuml start partition "applyPotentiallyNewLocalItem" { :Check if path exists; if (Path exists?) then (yes) :Log "Path on local disk already exists"; if (Is symbolic link?) then (yes) :Log "Path is a symbolic link"; if (Can read symbolic link?) then (no) :Log "Reading symbolic link failed"; :Log "Skipping item - invalid symbolic link"; stop endif endif :Determine if item is in-sync; note right: Execute 'isItemSynced()' function if (Is item in-sync?) then (yes) :Log "Item in-sync"; :Update/Insert item in DB; stop else (no) :Log "Item not in-sync"; :Compare local & remote modification times; if (Local time > Remote time?) then (yes) if (ID in database?) then (yes) :Log "Local file is newer & ID in DB"; :Fetch latest DB record; if (Times equal?) then (yes) :Log "Times match, keeping local file"; else (no) :Log "Local time newer, keeping file"; note right: Online item has an 'older' modified timestamp wise than the local file\nIt is assumed that the local file is the file to keep endif stop else (no) :Log "Local item not in DB"; if (Bypass data protection?) then (yes) :Log "WARNING: Data protection disabled"; else (no) :Safe backup local file; note right: Local data loss prevention endif stop endif else (no) if (Remote time > Local time?) then (yes) :Log "Remote item is newer"; if (Bypass data protection?) then (yes) :Log "WARNING: Data protection disabled"; else (no) :Safe backup local file; note right: Local data loss prevention endif endif if (Times equal?) then (yes) note left: Specific handling if timestamp was\nadjusted by isItemSynced() :Log "Times equal, no action required"; :Update/Insert item in DB; stop endif endif endif else (no) :Handle as potentially new item; switch (Item type) case (File) :Add to download queue; case (Directory) :Log "Creating local directory"; if (Dry run?) then (no) :Create directory & set attributes; :Save item to DB; else :Log "Dry run, faking directory creation"; :Save item to dry-run DB; endif case (Unknown) :Log "Unknown type, no action"; endswitch endif } stop @enduml ================================================ FILE: docs/puml/client_side_filtering_processing_order.puml ================================================ @startuml |Decision Tree| :Start Client Side Filtering Evaluation; if (check_nosync?) then (true) :Skip item (no sync); else (false) if (skip_dotfiles?) then (true) :Skip file (dotfile); else (false) if (skip_symlinks?) then (true) :Skip item (symlink); else (false) if (skip_dir?) then (true) :Skip directory; else (false) if (skip_file?) then (true) :Skip file; else (false) if (in sync_list?) then (false) :Skip item (not in sync list); else (true) if (skip_size?) then (true) :Skip file (size too large); else (false) :File or Directory flagged\nto be synced; endif endif endif endif endif endif endif :End Client Side Filtering Evaluation; @enduml ================================================ FILE: docs/puml/client_side_filtering_rules.puml ================================================ @startuml start :Start; partition "checkPathAgainstClientSideFiltering" { :Get localFilePath; if (Does path exist?) then (no) :Return false; stop endif if (Check .nosync?) then (yes) :Check for .nosync file; if (.nosync found) then (yes) :Log and return true; stop endif endif if (Skip dotfiles?) then (yes) :Check if dotfile; if (Is dotfile) then (yes) :Log and return true; stop endif endif if (Skip symlinks?) then (yes) :Check if symlink; if (Is symlink) then (yes) if (Config says skip?) then (yes) :Log and return true; stop elseif (Unexisting symlink?) then (yes) :Check if relative link works; if (Relative link ok) then (no) :Log and return true; stop endif endif endif endif if (Skip dir or file?) then (yes) :Check dir or file exclusion; if (Excluded by config?) then (yes) :Log and return true; stop endif endif if (Use sync_list?) then (yes) :Check sync_list exclusions; if (Excluded by sync_list?) then (yes) :Log and return true; stop endif endif if (Check file size?) then (yes) :Check for file size limit; if (File size exceeds limit?) then (yes) :Log and return true; stop endif endif :Return false; } stop @enduml ================================================ FILE: docs/puml/client_use_of_libcurl.puml ================================================ @startuml participant "OneDrive Client\nfor Linux" as od participant "libcurl" as lc participant "Client Web Browser" as browser participant "Microsoft Authentication Service\n(OAuth 2.0 Endpoint)" as oauth participant "GitHub API" as github participant "Microsoft Graph API" as graph activate od activate lc od->od: Generate Authentication\nService URL activate browser od->browser: Navigate to Authentication\nService URL via Client Web Browser browser->oauth: Request access token activate oauth oauth-->browser: Access token browser-->od: Access token deactivate oauth deactivate browser od->lc: Check application version\nvia api.github.com activate github lc->github: Query release status activate github github-->lc: Release information deactivate github lc-->od: Process release information deactivate lc loop API Communication od->lc: Construct HTTPS request (with token) activate lc lc->graph: API Request activate graph graph-->lc: API Response deactivate graph lc-->od: Process response deactivate lc end @enduml ================================================ FILE: docs/puml/code_functional_component_relationships.puml ================================================ @startuml !define DATABASE_ENTITY(x) entity x component main { } component config { } component log { } component curlEngine { } component util { } component onedrive { } component syncEngine { } component itemdb { } component clientSideFiltering { } component monitor { } component sqlite { } component qxor { } DATABASE_ENTITY("Database") main --> config main --> log main --> curlEngine main --> util main --> onedrive main --> syncEngine main --> itemdb main --> clientSideFiltering main --> monitor config --> log config --> util clientSideFiltering --> config clientSideFiltering --> util clientSideFiltering --> log syncEngine --> config syncEngine --> log syncEngine --> util syncEngine --> onedrive syncEngine --> itemdb syncEngine --> clientSideFiltering util --> log util --> config util --> qxor util --> curlEngine sqlite --> log sqlite -> "Database" : uses onedrive --> config onedrive --> log onedrive --> util onedrive --> curlEngine monitor --> config monitor --> util monitor --> log monitor --> clientSideFiltering monitor .> syncEngine : inotify event itemdb --> sqlite itemdb --> util itemdb --> log curlEngine --> log @enduml ================================================ FILE: docs/puml/conflict_handling_default.puml ================================================ @startuml start note left: Operational Mode 'onedrive --sync' :Query OneDrive /delta API for online changes; note left: This data is considered the 'source-of-truth'\nLocal data should be a 'replica' of this data :Process received JSON data; if (JSON item is a file) then (yes) if (Does the file exist locally) then (yes) :Compute relevant file hashes; :Check DB for file record; if (DB record found) then (yes) :Compare file hash with DB hash; if (Is the hash different) then (yes) :Log that the local file was modified locally since last sync; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file else (no) endif else (no) endif else (no) endif :Download file (as per online JSON item) as required; else (no) :Other handling for directories | root objects | deleted items; endif :Performing a database consistency and\nintegrity check on locally stored data; :Scan file system for any new data to upload; note left: The file that was renamed will be uploaded here stop @enduml ================================================ FILE: docs/puml/conflict_handling_default_resync.puml ================================================ @startuml start note left: Operational Mode 'onedrive -sync --resync' :Query OneDrive /delta API for online changes; note left: This data is considered the 'source-of-truth'\nLocal data should be a 'replica' of this data :Process received JSON data; if (JSON item is a file) then (yes) if (Does the file exist locally) then (yes) note left: In a --resync scenario there are no DB\nrecords that can be used or referenced\nuntil the JSON item is processed and\nadded to the local database cache if (Can the file be read) then (yes) :Compute UTC timestamp data from local file and JSON data; if (timestamps are equal) then (yes) else (no) :Log that a local file time discrepancy was detected; if (Do file hashes match) then (yes) :Correct the offending timestamp as hashes match; else (no) :Local file is technically different; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file endif endif else (no) endif else (no) endif :Download file (as per online JSON item) as required; else (no) :Other handling for directories | root objects | deleted items; endif :Performing a database consistency and\nintegrity check on locally stored data; :Scan file system for any new data to upload; note left: The file that was renamed will be uploaded here stop @enduml ================================================ FILE: docs/puml/conflict_handling_local-first_default.puml ================================================ @startuml start note left: Operational Mode 'onedrive -sync -local-first' :Performing a database consistency and\nintegrity check on locally stored data; note left: This data is considered the 'source-of-truth'\nOnline data should be a 'replica' of this data repeat :Process each DB record; if (Is the DB record is in sync with local file) then (yes) else (no) :Log reason for discrepancy; :Flag item to be processed as a modified local file; endif repeat while :Process modified items to upload; if (Does local file DB record match current latest online JSON data) then (yes) else (no) :Log that the local file was modified locally since last sync; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file :Upload renamed local file as new file; endif :Upload modified file; :Scan file system for any new data to upload; :Query OneDrive /delta API for online changes; :Process received JSON data; if (JSON item is a file) then (yes) if (Does the file exist locally) then (yes) :Compute relevant file hashes; :Check DB for file record; if (DB record found) then (yes) :Compare file hash with DB hash; if (Is the hash different) then (yes) :Log that the local file was modified locally since last sync; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file else (no) endif else (no) endif else (no) endif :Download file (as per online JSON item) as required; else (no) :Other handling for directories | root objects | deleted items; endif stop @enduml ================================================ FILE: docs/puml/conflict_handling_local-first_resync.puml ================================================ @startuml start note left: Operational Mode 'onedrive -sync -local-first -resync' :Query OneDrive API and create new database with default root account objects; :Performing a database consistency and\nintegrity check on locally stored data; note left: This data is considered the 'source-of-truth'\nOnline data should be a 'replica' of this data\nHowever the database has only 1 record currently :Scan file system for any new data to upload; note left: This is where in this specific mode all local\n content is assessed for applicability for\nupload to Microsoft OneDrive repeat :For each new local item; if (Is the item a directory) then (yes) if (Is Directory found online) then (yes) :Save directory details from online in local database; else (no) :Create directory online; :Save details in local database; endif else (no) :Flag file as a potentially new item to upload; endif repeat while :Process potential new items to upload; repeat :For each potential file to upload; if (Is File found online) then (yes) if (Does the online JSON data match local file) then (yes) :Save details in local database; else (no) :Log that the local file was modified locally since last sync; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file :Upload renamed local file as new file; endif else (no) :Upload new file; endif repeat while :Query OneDrive /delta API for online changes; :Process received JSON data; if (JSON item is a file) then (yes) if (Does the file exist locally) then (yes) :Compute relevant file hashes; :Check DB for file record; if (DB record found) then (yes) :Compare file hash with DB hash; if (Is the hash different) then (yes) :Log that the local file was modified locally since last sync; :Renaming local file to avoid potential local data loss; note left: Local data loss prevention\nRenamed file will be uploaded as new file else (no) endif else (no) endif else (no) endif :Download file (as per online JSON item) as required; else (no) :Other handling for directories | root objects | deleted items; endif stop @enduml ================================================ FILE: docs/puml/database_schema.puml ================================================ @startuml class item { driveId: TEXT id: TEXT name: TEXT remoteName: TEXT type: TEXT eTag: TEXT cTag: TEXT mtime: TEXT parentId: TEXT quickXorHash: TEXT sha256Hash: TEXT remoteDriveId: TEXT remoteParentId: TEXT remoteId: TEXT remoteType: TEXT deltaLink: TEXT syncStatus: TEXT size: TEXT relocDriveId: TEXT relocParentId: TEXT } note right of item::driveId PRIMARY KEY (driveId, id) FOREIGN KEY (driveId, parentId) REFERENCES item (driveId, id) ON DELETE CASCADE ON UPDATE RESTRICT end note item --|> item : parentId note "Indexes" as N1 note left of N1 name_idx ON item (name) remote_idx ON item (remoteDriveId, remoteId) item_children_idx ON item (driveId, parentId) selectByPath_idx ON item (name, driveId, parentId) end note @enduml ================================================ FILE: docs/puml/default_sync_flow.puml ================================================ @startuml title Default Sync Flow (Online is Source of Truth) start :Step 1 - Scan OneDrive (online); :Detect online changes: - New files or folders - Modified files - Deleted files; :Apply online changes to local: - Download new files or folders - Update modified files - Delete local files or folders; :Step 2 - Scan local files; :Detect local-only changes: - New files or folders - Modified files; :Upload local changes to OneDrive: - Upload new files or folders - Upload modified files; :Step 3 - Final reconciliation; :Rescan OneDrive to ensure: - Any last-minute online changes are applied locally; stop @enduml ================================================ FILE: docs/puml/downloadFile.puml ================================================ @startuml start partition "Download File" { :Get item specifics from JSON; :Calculate item's path; if (Is item malware?) then (yes) :Log malware detected; stop else (no) :Check for file size in JSON; if (File size missing) then (yes) :Log error; stop endif :Configure hashes for comparison; if (Hashes missing) then (yes) :Log error; stop endif if (Does file exist locally?) then (yes) :Check DB for item; if (DB hash match?) then (no) :Log modification; Perform safe backup; note left: Local data loss prevention endif endif :Check local disk space; if (Insufficient space?) then (yes) :Log insufficient space; stop else (no) if (Dry run?) then (yes) :Fake download process; else (no) :Attempt to download file; if (Download exception occurs?) then (yes) :Handle exceptions; Retry download or log error; endif if (File downloaded successfully?) then (yes) :Validate download; if (Validation passes?) then (yes) :Log success; Update DB; else (no) :Log validation failure; Remove file; endif else (no) :Log download failed; endif endif endif endif } stop @enduml ================================================ FILE: docs/puml/high_level_operational_process.puml ================================================ @startuml participant "OneDrive Client\nfor Linux" as Client participant "Microsoft OneDrive\nAPI" as API == Access Token Validation == Client -> Client: Validate access and\nexisting access token\nRefresh if needed == Query Microsoft OneDrive /delta API == Client -> API: Query /delta API API -> Client: JSON responses == Process JSON Responses == loop for each JSON response Client -> Client: Determine if JSON is 'root'\nor 'deleted' item\nElse, push into temporary array for further processing alt if 'root' or 'deleted' Client -> Client: Process 'root' or 'deleted' items else Client -> Client: Evaluate against 'Client Side Filtering' rules alt if unwanted Client -> Client: Discard JSON else Client -> Client: Process JSON (create dir/download file) Client -> Client: Save in local database cache end end end == Local Cache Database Processing for Data Integrity == Client -> Client: Process local cache database\nto check local data integrity and for differences alt if difference found Client -> API: Upload file/folder change including deletion API -> Client: Response with item metadata Client -> Client: Save response to local cache database end == Local Filesystem Scanning == Client -> Client: Scan local filesystem\nfor new files/folders loop for each new item Client -> Client: Check item against 'Client Side Filtering' rules alt if item passes filtering Client -> API: Upload new file/folder change including deletion API -> Client: Response with item metadata Client -> Client: Save response in local\ncache database else Client -> Client: Discard item\n(Does not meet filtering criteria) end end == Final Data True-Up == Client -> API: Query /delta link for true-up API -> Client: Process further online JSON changes if required @enduml ================================================ FILE: docs/puml/is_item_in_sync.puml ================================================ @startuml start partition "Is item in sync" { :Check if path exists; if (path does not exist) then (no) :Return false; stop else (yes) endif :Identify item type; switch (item type) case (file) :Check if path is a file; if (path is not a file) then (no) :Log "item is a directory but should be a file"; :Return false; stop else (yes) endif :Attempt to read local file; if (file is unreadable) then (no) :Log "file cannot be read"; :Return false; stop else (yes) endif :Get local and input item modified time; note right: The 'input item' could be a database reference object, or the online JSON object\nas provided by the Microsoft OneDrive API :Reduce time resolution to seconds; if (localModifiedTime == itemModifiedTime) then (yes) :Return true; stop else (no) :Log time discrepancy; endif :Check if file hash is the same; if (hash is the same) then (yes) :Log "hash match, correcting timestamp"; if (local time > item time) then (yes) if (download only mode) then (no) :Correct timestamp online if not dryRun; else (yes) :Correct local timestamp if not dryRun; endif else (no) :Correct local timestamp if not dryRun; endif :Return false; note right: Specifically return false here as we performed a time correction\nApplication logic will then perform additional handling based on this very specific response. stop else (no) :Log "different hash"; :Return false; stop endif case (dir or remote) :Check if path is a directory; if (path is a directory) then (yes) :Return true; stop else (no) :Log "item is a file but should be a directory"; :Return false; stop endif case (unknown) :Return true but do not sync; stop endswitch } @enduml ================================================ FILE: docs/puml/local_first_sync_process.puml ================================================ @startuml title Local-First Sync Flow (--local-first) start :Step 1 - Scan local files; :Detect local changes: - New files or folders - Modified files - Deleted files; :Apply local changes to OneDrive: - Upload new files or folders - Update modified files - Delete files or folders on OneDrive; :Step 2 - Scan OneDrive (online); :Detect online-only changes: - New files or folders - Modified files - Deleted files; :Apply online changes to local: - Download missing files or folders - Update outdated local files - Delete local files or folders that were deleted online; stop @enduml ================================================ FILE: docs/puml/main_activity_flows.puml ================================================ @startuml start :Validate access and existing access token\nRefresh if needed; :Query /delta API; note right: Query Microsoft OneDrive /delta API :Receive JSON responses; :Process JSON Responses; partition "Process /delta JSON Responses" { while (for each JSON response) is (yes) :Determine if JSON is 'root'\nor 'deleted' item; if ('root' or 'deleted') then (yes) :Process 'root' or 'deleted' items; if ('root' object) then (yes) :Process 'root' JSON; else (no) if (Is 'deleted' object in sync) then (yes) :Process deletion of local item; else (no) :Rename local file as it is not in sync; note right: Deletion event conflict handling\nLocal data loss prevention endif endif else (no) :Evaluate against 'Client Side Filtering' rules; if (unwanted) then (yes) :Discard JSON; else (no) :Process JSON (create dir/download file); if (Is the 'JSON' item in the local cache) then (yes) :Process JSON as a potentially changed local item; note left: Run 'applyPotentiallyChangedItem' function else (no) :Process JSON as potentially new local item; note right: Run 'applyPotentiallyNewLocalItem' function endif :Process objects in download queue; :Download File; note left: Download file from Microsoft OneDrive (Multi Threaded Download) :Save in local database cache; endif endif endwhile } partition "Perform data integrity check based on local cache database" { :Process local cache database\nto check local data integrity and for differences; if (difference found) then (yes) :Upload file/folder change including deletion; note right: Upload local change to Microsoft OneDrive :Receive response with item metadata; :Save response to local cache database; else (no) endif } partition "Local Filesystem Scanning" { :Scan local filesystem\nfor new files/folders; while (for each new item) is (yes) :Check item against 'Client Side Filtering' rules; if (item passes filtering) then (yes) :Upload new file/folder change including deletion; note right: Upload to Microsoft OneDrive :Receive response with item metadata; :Save response in local\ncache database; else (no) :Discard item\n(Does not meet filtering criteria); endif endwhile } partition "Final True-Up" { :Query /delta link for true-up; note right: Final Data True-Up :Process further online JSON changes if required; } stop @enduml ================================================ FILE: docs/puml/onedrive_linux_authentication.puml ================================================ @startuml participant "OneDrive Client for Linux" participant "Microsoft OneDrive\nAuthentication Service\n(login.microsoftonline.com)" as AuthServer participant "User's Device (for MFA)" as UserDevice participant "Microsoft Graph API\n(graph.microsoft.com)" as GraphAPI participant "Microsoft OneDrive" "OneDrive Client for Linux" -> AuthServer: Request Authorization\n(Client Credentials, Scopes) AuthServer -> "OneDrive Client for Linux": Provide Authorization Code "OneDrive Client for Linux" -> AuthServer: Request Access Token\n(Authorization Code, Client Credentials) alt MFA Enabled AuthServer -> UserDevice: Trigger MFA Challenge UserDevice -> AuthServer: Provide MFA Verification AuthServer -> "OneDrive Client for Linux": Return Access Token\n(and Refresh Token) "OneDrive Client for Linux" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "OneDrive Client for Linux" -> AuthServer: Is Access Token Expired? alt Token Expired "OneDrive Client for Linux" -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> "OneDrive Client for Linux": Return New Access Token else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "OneDrive Client for Linux": Provide Data end end else MFA Not Required AuthServer -> "OneDrive Client for Linux": Return Access Token\n(and Refresh Token) "OneDrive Client for Linux" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "OneDrive Client for Linux" -> AuthServer: Is Access Token Expired? alt Token Expired "OneDrive Client for Linux" -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> "OneDrive Client for Linux": Return New Access Token else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "OneDrive Client for Linux": Provide Data end end else MFA Failed or Other Auth Error AuthServer -> "OneDrive Client for Linux": Error Message (e.g., Invalid Credentials, MFA Failure) end @enduml ================================================ FILE: docs/puml/onedrive_windows_ad_authentication.puml ================================================ @startuml participant "Microsoft Windows OneDrive Client" participant "Azure Active Directory\n(Active Directory)\n(login.microsoftonline.com)" as AzureAD participant "Microsoft OneDrive\nAuthentication Service\n(login.microsoftonline.com)" as AuthServer participant "User's Device (for MFA)" as UserDevice participant "Microsoft Graph API\n(graph.microsoft.com)" as GraphAPI participant "Microsoft OneDrive" "Microsoft Windows OneDrive Client" -> AzureAD: Request Authorization\n(Client Credentials, Scopes) AzureAD -> AuthServer: Validate Credentials\n(Forward Request) AuthServer -> AzureAD: Provide Authorization Code AzureAD -> "Microsoft Windows OneDrive Client": Provide Authorization Code (via AzureAD) "Microsoft Windows OneDrive Client" -> AzureAD: Request Access Token\n(Authorization Code, Client Credentials) AzureAD -> AuthServer: Request Access Token\n(Authorization Code, Forwarded Credentials) AuthServer -> AzureAD: Return Access Token\n(and Refresh Token) AzureAD -> "Microsoft Windows OneDrive Client": Return Access Token\n(and Refresh Token) (via AzureAD) alt MFA Enabled AzureAD -> UserDevice: Trigger MFA Challenge UserDevice -> AzureAD: Provide MFA Verification AzureAD -> "Microsoft Windows OneDrive Client": Return Access Token\n(and Refresh Token) (Post MFA) "Microsoft Windows OneDrive Client" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "Microsoft Windows OneDrive Client" -> AzureAD: Is Access Token Expired? AzureAD -> AuthServer: Validate Token Expiry alt Token Expired "Microsoft Windows OneDrive Client" -> AzureAD: Request New Access Token\n(Refresh Token) AzureAD -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> AzureAD: Return New Access Token AzureAD -> "Microsoft Windows OneDrive Client": Return New Access Token (via AzureAD) else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "Microsoft Windows OneDrive Client": Provide Data end end else MFA Not Required AzureAD -> "Microsoft Windows OneDrive Client": Return Access Token\n(and Refresh Token) (Direct) "Microsoft Windows OneDrive Client" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "Microsoft Windows OneDrive Client" -> AzureAD: Is Access Token Expired? AzureAD -> AuthServer: Validate Token Expiry alt Token Expired "Microsoft Windows OneDrive Client" -> AzureAD: Request New Access Token\n(Refresh Token) AzureAD -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> AzureAD: Return New Access Token AzureAD -> "Microsoft Windows OneDrive Client": Return New Access Token (via AzureAD) else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "Microsoft Windows OneDrive Client": Provide Data end end else MFA Failed or Other Auth Error AzureAD -> "Microsoft Windows OneDrive Client": Error Message (e.g., Invalid Credentials, MFA Failure) end @enduml ================================================ FILE: docs/puml/onedrive_windows_authentication.puml ================================================ @startuml participant "Microsoft Windows OneDrive Client" participant "Microsoft OneDrive\nAuthentication Service\n(login.microsoftonline.com)" as AuthServer participant "User's Device (for MFA)" as UserDevice participant "Microsoft Graph API\n(graph.microsoft.com)" as GraphAPI participant "Microsoft OneDrive" "Microsoft Windows OneDrive Client" -> AuthServer: Request Authorization\n(Client Credentials, Scopes) AuthServer -> "Microsoft Windows OneDrive Client": Provide Authorization Code "Microsoft Windows OneDrive Client" -> AuthServer: Request Access Token\n(Authorization Code, Client Credentials) alt MFA Enabled AuthServer -> UserDevice: Trigger MFA Challenge UserDevice -> AuthServer: Provide MFA Verification AuthServer -> "Microsoft Windows OneDrive Client": Return Access Token\n(and Refresh Token) "Microsoft Windows OneDrive Client" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "Microsoft Windows OneDrive Client" -> AuthServer: Is Access Token Expired? alt Token Expired "Microsoft Windows OneDrive Client" -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> "Microsoft Windows OneDrive Client": Return New Access Token else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "Microsoft Windows OneDrive Client": Provide Data end end else MFA Not Required AuthServer -> "Microsoft Windows OneDrive Client": Return Access Token\n(and Refresh Token) "Microsoft Windows OneDrive Client" -> GraphAPI: Request Microsoft OneDrive Data\n(Access Token) loop Token Expiry Check "Microsoft Windows OneDrive Client" -> AuthServer: Is Access Token Expired? alt Token Expired "Microsoft Windows OneDrive Client" -> AuthServer: Request New Access Token\n(Refresh Token) AuthServer -> "Microsoft Windows OneDrive Client": Return New Access Token else Token Valid GraphAPI -> "Microsoft OneDrive": Retrieve Data "Microsoft OneDrive" -> GraphAPI: Return Data GraphAPI -> "Microsoft Windows OneDrive Client": Provide Data end end else MFA Failed or Other Auth Error AuthServer -> "Microsoft Windows OneDrive Client": Error Message (e.g., Invalid Credentials, MFA Failure) end @enduml ================================================ FILE: docs/puml/uploadFile.puml ================================================ @startuml start partition "Upload File" { :Log "fileToUpload"; :Check database for parent path; if (parent path found?) then (yes) if (drive ID not empty?) then (yes) :Proceed; else (no) :Use defaultDriveId; endif else (no) stop endif :Check if file exists locally; if (file exists?) then (yes) :Read local file; if (can read file?) then (yes) if (parent path in DB?) then (yes) :Get file size; if (file size <= max?) then (yes) :Check available space on OneDrive; if (space available?) then (yes) :Check if file exists on OneDrive; if (file exists online?) then (yes) :Save online metadata only; if (if local file newer) then (yes) :Local file is newer; :Upload file as changed local file; else (no) :Remote file is newer; :Perform safe backup; note right: Local data loss prevention :Upload renamed file as new file; endif else (no) :Attempt upload; endif else (no) :Log "Insufficient space"; endif else (no) :Log "File too large"; endif else (no) :Log "Parent path issue"; endif else (no) :Log "Cannot read file"; endif else (no) :Log "File disappeared locally"; endif :Upload success or failure; if (upload failed?) then (yes) :Log failure; else (no) :Update cache; endif } stop @enduml ================================================ FILE: docs/puml/uploadModifiedFile.puml ================================================ @startuml start partition "Upload Modified File" { :Initialize API Instance; :Check for Dry Run; if (Is Dry Run?) then (yes) :Create Fake Response; else (no) :Get Current Online Data; if (Error Fetching Data) then (yes) :Handle Errors; if (Retryable Error?) then (yes) :Retry Fetching Data; detach else (no) :Log and Display Error; endif endif if (filesize > 0 and valid latest online data) then (yes) if (is online file newer) then (yes) :Log that online is newer; :Perform safe backup; note left: Local data loss prevention :Upload renamed local file as new file; endif endif :Determine Upload Method; if (Use Simple Upload?) then (yes) :Perform Simple Upload; if (Upload Error) then (yes) :Handle Upload Errors and Retries; if (Retryable Upload Error?) then (yes) :Retry Upload; detach else (no) :Log and Display Upload Error; endif endif else (no) :Create Upload Session; :Perform Upload via Session; if (Session Upload Error) then (yes) :Handle Session Upload Errors and Retries; if (Retryable Session Error?) then (yes) :Retry Session Upload; detach else (no) :Log and Display Session Error; endif endif endif endif :Finalize; } stop @enduml ================================================ FILE: docs/puml/webhooks.puml ================================================ @startuml skinparam SequenceBoxBackgroundColor<> AliceBlue box "Linux System"<> participant ClientApp as "OneDrive Client for Linux\n(webhook listener 127.0.0.1:8888)" participant Nginx end box participant Firewall as "Firewall | Router" participant GraphAPI as "Microsoft Graph API" ClientApp -> GraphAPI: HTTPS POST /v1.0/subscriptions GraphAPI -> ClientApp: Subscription details response (HTTPS) == Subscription Notification == GraphAPI -> Firewall: HTTPS Notification (port 443) Firewall -> Nginx: Port forwarding to Nginx (port 443) alt Request for /webhooks/onedrive Nginx -> ClientApp: Proxy notification to http://127.0.0.1:8888 ClientApp -> Nginx: Response Nginx -> GraphAPI: Return proxied response (HTTPS) end @enduml ================================================ FILE: docs/server-side-filtering-limitations.md ================================================ # Why 'Server Side Filtering' is not possible with Microsoft OneDrive A common misconception is that `sync_list` or other client-side filtering rules should be able to instruct Microsoft OneDrive or Microsoft Graph to only return a subset of data from the server. This is not how Microsoft OneDrive or Microsoft Graph works. The Microsoft Graph API exposes OneDrive content as `driveItem` resources. Folders are represented as items with a `children` relationship, and changes are tracked through the `delta` API. In other words, the API is built around addressing items, listing children, and tracking changes to those items over time. It is **not** built around applying a user-defined selective sync policy on the server before results are returned. ## The practical reality Server-side selective sync, equivalent to `sync_list`, is not possible with Microsoft Graph today. There is no supported API capability to provide Microsoft Graph with rules such as: * include these folders * exclude these folders * exclude this subtree recursively * apply wildcard or glob rules * return only the logical drive view that matches a client configuration The OneDrive Client for Linux therefore has no ability to tell Microsoft Graph: > only return `/Documents/Work/**`, but exclude `/Documents/Work/Archive/**` That type of policy-driven filesystem view is simply not part of the API surface exposed by Microsoft Graph. ## Why this is a Microsoft Graph platform limitation This is not an implementation gap in the OneDrive Client for Linux. It is a direct result of how Microsoft Graph is designed. The `children` API for drive items supports paging and response-shaping options such as `$expand`, `$select`, `$skipToken`, `$top`, and `$orderby`, but it does **not** support a hierarchical `$filter` capability that could be used to express selective sync rules. Microsoft’s own query parameter guidance also states that support for query parameters varies by API operation, and the supported parameters for each operation are explicitly documented. For `children`, the supported query parameters do not include the type of recursive or path-based filtering that `sync_list` would require. ### Why `$filter` does not solve this Even where Microsoft Graph supports `$filter` on other APIs, that does not make server-side selective sync possible for OneDrive content. Selective sync requires the server to understand and evaluate: * full path ancestry * descendant relationships * recursive subtree inclusion and exclusion * ordered rule processing * wildcard or glob matching * conflict handling between include and exclude rules The OneDrive `children` API does not expose that model. It returns the items in a folder. The client must then decide what those returned items mean in the context of the configured client-side sync rules. ### Why `search` does not solve this It may be tempting to think that the Graph search API could be used instead. It cannot. The Graph search endpoint is a search function over drive content using query text. It is designed to find matching items by search criteria such as filename, metadata, or file content. It is **not** a policy engine, it is not a substitute for authoritative filesystem enumeration, and it cannot be used to enforce deterministic include/exclude boundaries for sync. Search can help find items. It cannot define a complete and correct sync scope. ### Why `delta` does not solve this The Graph `delta` API is also often misunderstood. `delta` is designed to track changes in a `driveItem` and its children over time. Microsoft documents that the app begins by calling `delta` with no parameters, and that the service starts **enumerating the drive's hierarchy**, returning pages of items until the client has received the complete change set. After that, the client applies those changes to its local state. This is important: * `delta` reduces how much metadata needs to be transferred after the initial state is known * `delta` helps the client track change efficiently * `delta` does **not** move selective sync rule evaluation to the server * `delta` still assumes the client is responsible for deciding what to keep, ignore, download, or discard locally ## What the client must do instead Because Microsoft Graph does not provide server-side selective sync, the OneDrive Client for Linux must do the following: 1. Enumerate remote metadata from Microsoft OneDrive 2. Build or refresh its understanding of the remote hierarchy 3. Evaluate configured rules such as `sync_list`, `skip_file`, `skip_dir`, `single_directory`, and other sync controls 4. Decide locally which items should be downloaded, ignored, retained, or removed This is why `sync_list` and other sync controls are correctly described as client side filtering. The rules are applied by the client after Microsoft Graph has returned the relevant metadata required for the client to understand the remote state. ## Why excluded data may still appear to be “seen” Users sometimes ask: > If I’ve excluded most folders using `sync_list`, why does the client still appear to scan the entire remote structure before skipping them? The answer is simple: To decide whether something should be excluded, the client must first know that the item exists in the remote hierarchy. Microsoft Graph returns metadata about drive items and folder children; the client then applies its local filtering rules to determine whether that item should be processed further. So: * the client may enumerate metadata for excluded paths * the client may log that those paths were evaluated * the client may discard them immediately based on local rules * the client is **not** “pulling everything down” in the sense of downloading all file content. What is unavoidable is remote metadata discovery. What is controlled by client-side filtering is what happens after that discovery process. ## Why “only query allowed folders” is not a complete solution Another suggestion is often: > Why not just query only the folders I want? That approach is incomplete and unreliable. A sync client must correctly handle: * new folders created remotely * renames and moves * deleted items * items relocated into or out of an allowed path * invalidated delta tokens * reconciliation of local and remote state across the full hierarchy Without authoritative knowledge of the hierarchy and changes returned by Microsoft Graph, the client cannot safely and correctly maintain sync state. The Graph API is designed around item enumeration and delta tracking, not around returning a server-enforced filtered filesystem view. ## What this means for all Microsoft OneDrive clients This limitation is not unique to the OneDrive Client for Linux. Any OneDrive client built on Microsoft Graph must work within the same platform constraints: * Microsoft Graph returns OneDrive content as addressable resources and collections of `driveItem` objects * folder traversal happens through `children` * change tracking happens through `delta` * filtering decisions (if implemented) beyond what the API explicitly supports must be made by the client ## Summary Server-side selective sync is not available because Microsoft Graph does not provide: * recursive path-based filtering * wildcard rule evaluation * hierarchical include/exclude policy support * a server-defined partial-drive view for sync clients As a result, the client must always enumerate the remote OneDrive metadata to understand the full filesystem structure before any filtering rules can be applied locally. This enumeration phase can take a noticeable amount of time on large datasets (for example, SharePoint libraries with tens of thousands of folders). This is especially evident when using `--resync`, which clears all locally stored sync state and forces a full re-discovery of the remote hierarchy, or when changes to configuration (such as `sync_list`) require the client to re-evaluate the complete remote structure. It is important to understand that this process is **metadata enumeration only** — the client is not downloading all file contents, but it must still query and process all relevant filesystem objects returned by Microsoft Graph. Additionally, this process cannot be arbitrarily parallelised or short-circuited. Microsoft Graph returns data in a paginated and ordered manner, and the client must process these results sequentially to correctly maintain state, handle hierarchy relationships, and ensure consistency (for example, detecting moves, renames, and deletions). Attempting to process this out of order or in parallel would lead to an inconsistent or incorrect sync state. This means that: * initial syncs and `--resync` operations will take longer on large datasets * applying or modifying filtering rules may require full re-evaluation * large numbers of folders or items will increase enumeration time This behaviour is therefore **expected**, **correct**, and **driven by Microsoft Graph platform limitations**, not by a defect in the OneDrive Client for Linux. ================================================ FILE: docs/sharepoint-libraries.md ================================================ # How to configure OneDrive SharePoint Shared Library sync > [!CAUTION] > Before reading this document, please ensure you are running application version [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) or greater. Use `onedrive --version` to determine what application version you are using and upgrade your client if required. > [!CAUTION] > Several users have reported files being overwritten causing data loss as a result of using this client with SharePoint Libraries when running as a systemd service. > > When this has been investigated, the following has been noted as potential root causes: > * File indexing application such as Baloo File Indexer or Tracker3 constantly indexing your OneDrive data > * The use of WPS Office and how it 'saves' files by deleting the existing item and replaces it with the saved data. Do not use WPS Office. > > Additionally there could be a yet unknown bug with the client, however all debugging and data provided previously shows that an 'external' process to the 'onedrive' application modifies the files triggering the undesirable upload to occur. > > **Possible Preventative Actions:** > * Disable all File Indexing for your SharePoint Library data. It is out of scope to detail on how you should do this. > * Disable using a systemd service for syncing your SharePoint Library data. > * Do not use WPS Office to edit your documents. Use OpenOffice or LibreOffice as these do not exhibit the same 'delete to save' action that WPS Office has. > > Additionally has been 100% re-written from v2.5.0 onwards, thus the mechanism for saving data to SharePoint has been critically overhauled to simplify actions to negate the impacts where SharePoint will *modify* your file post upload, breaking file integrity as the file you have locally, is not the file that is stored online. Please read https://github.com/OneDrive/onedrive-api-docs/issues/935 for relevant details. ## Process Overview Syncing a OneDrive SharePoint library requires additional configuration for your 'onedrive' client: 1. Login to OneDrive and under 'Shared Libraries' obtain the shared library name 2. Query that shared library name using the client to obtain the required configuration details 3. Create a unique local folder which will be the SharePoint Library 'root' 4. Configure the client's config file with the required 'drive_id' 5. Test the configuration using '--dry-run' 6. Sync the SharePoint Library as required > [!IMPORTANT] > The `--get-sharepoint-drive-id` process below requires a fully configured 'onedrive' configuration so that the applicable Drive ID for the given SharePoint Shared Library can be determined. It is highly recommended that you do not use the application 'default' configuration directory for any SharePoint Site, and configure separate items for each site you wish to use. ## 1. Listing available OneDrive SharePoint Libraries Login to the OneDrive web interface and determine which shared library you wish to configure the client for: ![shared_libraries](./images/SharedLibraries.jpg) ## 2. Query OneDrive API to obtain required configuration details Run the following command using the 'onedrive' client to query the OneDrive API to obtain the required 'drive_id' of the SharePoint Library that you wish to sync: ```text onedrive --get-sharepoint-drive-id '' ``` This will return something similar to the following: ```text Configuration file successfully loaded Configuring Global Azure AD Endpoints Initializing the Synchronization Engine ... Office 365 Library Name Query: ----------------------------------------------- Site Name: Library Name: drive_id: b!6H_y8B...xU5 Library URL: ----------------------------------------------- ``` If there are no matches to the site you are attempting to search, the following will be displayed: ```text Configuration file successfully loaded Configuring Global Azure AD Endpoints Initializing the Synchronization Engine ... Office 365 Library Name Query: blah ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site. The following SharePoint site names were returned: * * ... * ``` This list of site names can be used as a basis to search for the correct site for which you are searching ## 3. Create a new configuration directory and sync location for this SharePoint Library Create a new configuration directory for this SharePoint Library in the following manner: ```text mkdir ~/.config/SharePoint_My_Library_Name ``` Create a new local folder to store the SharePoint Library data in: ```text mkdir ~/SharePoint_My_Library_Name ``` > [!TIP] > Do not use spaces in the directory name, use '_' as a replacement ## 4. Configure SharePoint Library config file with the required 'drive_id' & 'sync_dir' options Download a copy of the default configuration file by downloading this file from GitHub and saving this file in the directory created above: ```text wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/SharePoint_My_Library_Name/config ``` Update your 'onedrive' configuration file (`~/.config/SharePoint_My_Library_Name/config`) with the local folder where you will store your data: ```text sync_dir = "~/SharePoint_My_Library_Name" ``` Update your 'onedrive' configuration file(`~/.config/SharePoint_My_Library_Name/config`) with the 'drive_id' value obtained in the steps above: ```text drive_id = "insert the drive_id value from above here" ``` The OneDrive client will now be configured to sync this SharePoint shared library to your local system and the location you have configured. > [!IMPORTANT] > After changing `drive_id`, you must perform a full re-synchronization by adding `--resync` to your existing command line. ## 5. Validate and Test the configuration Validate your new configuration using the `--display-config` option to validate you have configured the application correctly: ```text onedrive --confdir="~/.config/SharePoint_My_Library_Name" --display-config ``` Test your new configuration using the `--dry-run` option to validate the application configuration: ```text onedrive --confdir="~/.config/SharePoint_My_Library_Name" --synchronize --verbose --dry-run ``` > [!IMPORTANT] > As this is a *new* configuration, the application will be required to be re-authorised the first time this command is run with the new configuration. ## 6. Sync the SharePoint Library as required Sync the SharePoint Library to your system with either `--synchronize` or `--monitor` operations: ```text onedrive --confdir="~/.config/SharePoint_My_Library_Name" --synchronize --verbose ``` ```text onedrive --confdir="~/.config/SharePoint_My_Library_Name" --monitor --verbose ``` > [!IMPORTANT] > As this is a *new* configuration, the application will be required to be re-authorised the first time this command is run with the new configuration. ## 7. Enable custom systemd service for SharePoint Library Systemd can be used to automatically run this configuration in the background, however, a unique systemd service will need to be setup for this SharePoint Library instance In order to automatically start syncing each SharePoint Library, you will need to create a service file for each SharePoint Library. From the applicable 'systemd folder' where the applicable systemd service file exists: * RHEL / CentOS: `/usr/lib/systemd/system` * Others: `/usr/lib/systemd/user` and `/lib/systemd/system` ### Step1: Create a new systemd service file #### Red Hat Enterprise Linux, CentOS Linux Copy the required service file to a new name: ```text sudo cp /usr/lib/systemd/system/onedrive.service /usr/lib/systemd/system/onedrive-SharePoint_My_Library_Name.service ``` or ```text sudo cp /usr/lib/systemd/system/onedrive@.service /usr/lib/systemd/system/onedrive-SharePoint_My_Library_Name@.service ``` #### Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora Copy the required service file to a new name: ```text sudo cp /usr/lib/systemd/user/onedrive.service /usr/lib/systemd/user/onedrive-SharePoint_My_Library_Name.service ``` or ```text sudo cp /lib/systemd/system/onedrive@.service /lib/systemd/system/onedrive-SharePoint_My_Library_Name@.service ``` ### Step 2: Edit new systemd service file Edit the new systemd file, updating the line beginning with `ExecStart` so that the confdir mirrors the one you used above: ```text ExecStart=/usr/local/bin/onedrive --monitor --confdir="/full/path/to/config/dir" ``` Example: ```text ExecStart=/usr/local/bin/onedrive --monitor --confdir="/home/myusername/.config/SharePoint_My_Library_Name" ``` > [!IMPORTANT] > When running the client manually, `--confdir="~/.config/......` is acceptable. In a systemd configuration file, the full path must be used. The `~` must be manually expanded when editing your systemd file. ### Step 3: Enable the new systemd service Once the file is correctly edited, you can enable the new systemd service using the following commands. #### Red Hat Enterprise Linux, CentOS Linux ```text systemctl enable onedrive-SharePoint_My_Library_Name systemctl start onedrive-SharePoint_My_Library_Name ``` #### Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text systemctl --user enable onedrive-SharePoint_My_Library_Name systemctl --user start onedrive-SharePoint_My_Library_Name ``` or ```text systemctl --user enable onedrive-SharePoint_My_Library_Name@myusername.service systemctl --user start onedrive-SharePoint_My_Library_Name@myusername.service ``` ### Step 4: Viewing systemd status and logs for the custom service #### Viewing systemd service status - Red Hat Enterprise Linux, CentOS Linux ```text systemctl status onedrive-SharePoint_My_Library_Name ``` #### Viewing systemd service status - Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text systemctl --user status onedrive-SharePoint_My_Library_Name ``` #### Viewing journalctl systemd logs - Red Hat Enterprise Linux, CentOS Linux ```text journalctl --unit=onedrive-SharePoint_My_Library_Name -f ``` #### Viewing journalctl systemd logs - Others such as Arch, Ubuntu, Debian, OpenSuSE, Fedora ```text journalctl --user --unit=onedrive-SharePoint_My_Library_Name -f ``` ### Step 5: (Optional) Run custom systemd service at boot without user login In some cases it may be desirable for the systemd service to start without having to login as your 'user' All the systemd steps above that utilise the `--user` option, will run the systemd service as your particular user. As such, the systemd service will not start unless you actually login to your system. To avoid this issue, you need to reconfigure your 'user' account so that the systemd services you have created will startup without you having to login to your system: ```text loginctl enable-linger ``` Example: ```text alex@ubuntu-headless:~$ loginctl enable-linger alex ``` ## 8. Configuration for a SharePoint Library is complete The 'onedrive' client configuration for this particular SharePoint Library is now complete. # How to configure multiple OneDrive SharePoint Shared Library sync Create a new configuration as per the process above. Repeat these steps for each SharePoint Library that you wish to use. ================================================ FILE: docs/terms-of-service.md ================================================ # OneDrive Client for Linux - Software Service Terms of Service ## 1. Introduction These Terms of Service ("Terms") govern your use of the OneDrive Client for Linux ("Application") software and related Microsoft OneDrive services ("Service") provided by Microsoft. By accessing or using the Service, you agree to comply with and be bound by these Terms. If you do not agree to these Terms, please do not use the Service. ## 2. License Compliance The OneDrive Client for Linux software is licensed under the GNU General Public License, version 3.0 (the "GPLv3"). Your use of the software must comply with the terms and conditions of the GPLv3. A copy of the GPLv3 can be found here: https://www.gnu.org/licenses/gpl-3.0.en.html ## 3. Use of the Service ### 3.1. Access and Accounts You may need to create an account or provide personal information to access certain features of the Service. You are responsible for maintaining the confidentiality of your account information and are solely responsible for all activities that occur under your account. ### 3.2. Prohibited Activities You agree not to: - Use the Service in any way that violates applicable laws or regulations. - Use the Service to engage in any unlawful, harmful, or fraudulent activity. - Use the Service in any manner that disrupts, damages, or impairs the Service. ## 4. Intellectual Property The OneDrive Client for Linux software is subject to the GPLv3, and you must respect all copyrights, trademarks, and other intellectual property rights associated with the software. Any contributions you make to the software must also comply with the GPLv3. ## 5. Disclaimer of Warranties The OneDrive Client for Linux software is provided "as is" without any warranties, either expressed or implied. We do not guarantee that the use of the Application will be error-free or uninterrupted. Microsoft is not responsible for OneDrive Client for Linux. Any issues or problems with OneDrive Client for Linux should be raised on GitHub at https://github.com/abraunegg/onedrive or email support@mynas.com.au OneDrive Client for Linux is not responsible for the Microsoft OneDrive Service or the Microsoft Graph API Service that this Application utilises. Any issue with either Microsoft OneDrive or Microsoft Graph API should be raised with Microsoft via their support channel in your country. ## 6. Limitation of Liability To the fullest extent permitted by law, we shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages, or any loss of profits or revenues, whether incurred directly or indirectly, or any loss of data, use, goodwill, or other intangible losses, resulting from (a) your use or inability to use the Service, or (b) any other matter relating to the Service. This limitation of liability explicitly relates to the use of the OneDrive Client for Linux software and does not affect your rights under the GPLv3. ## 7. Changes to Terms We reserve the right to update or modify these Terms at any time without prior notice. Any changes will be effective immediately upon posting on GitHub. Your continued use of the Service after the posting of changes constitutes your acceptance of such changes. Changes can be reviewed on GitHub. ## 8. Governing Law These Terms shall be governed by and construed in accordance with the laws of Australia, without regard to its conflict of law principles. ## 9. Contact Us If you have any questions or concerns about these Terms, please contact us at https://github.com/abraunegg/onedrive or email support@mynas.com.au ================================================ FILE: docs/ubuntu-package-install.md ================================================ # Installation of 'onedrive' package on Debian and Ubuntu This document outlines the steps for installing the 'onedrive' client on Debian, Ubuntu, and their derivatives using the OpenSuSE Build Service Packages. > [!CAUTION] > This information is specifically for the following platforms and distributions: > * Debian > * Deepin > * Elementary OS > * Kali Linux > * Lubuntu > * Linux Mint > * MX Linux > * Pop!_OS > * Peppermint OS > * Raspbian | Raspberry Pi OS > * Ubuntu | Kubuntu | Xubuntu | Ubuntu Mate > * Zorin OS > > Although packages for the 'onedrive' client are available through distribution repositories, it is strongly advised against installing them. These distribution-provided packages are outdated, unsupported, and contain bugs and issues that have already been resolved in newer versions. They should not be used. > [!IMPORTANT] > The distribution versions listed below are **End-of-Life (EOL)** and are **no longer supported** or tested with current client releases. You must upgrade to a supported distribution before proceeding. > * Debian 9 > * Debian 10 > * Ubuntu 16.x > * Ubuntu 18.x > * Ubuntu 20.x ## Determine which instructions to use Ubuntu and its clones are based on various different releases, thus, you must use the correct instructions below, otherwise you may run into package dependency issues and will be unable to install the client. ### Step 1: Remove any configured PPA and associated 'onedrive' package and systemd service files #### Step 1a: Remove PPA if configured Many Internet 'help' pages provide inconsistent details on how to install the OneDrive Client for Linux. A number of these websites continue to point users to install the client via the yann1ck PPA repository however this PPA no longer exists and should not be used. If you have previously configured, or attempted to add this PPA, this needs to be removed. To remove the yann1ck PPA repository, perform the following actions: ```text sudo add-apt-repository --remove ppa:yann1ck/onedrive ``` #### Step 1b: Remove 'onedrive' package installed from Debian / Ubuntu repositories Many Internet 'help' pages provide inconsistent details on how to install the OneDrive Client for Linux. A number of these websites continue to advise users to install the client via `sudo apt install onedrive` without first configuring the OpenSuSE Build Service (OBS) Repository. When installing without OBS, you install an obsolete client version with known bugs that have been fixed, but this package also contains an errant systemd service (see below) that impacts background running of this client. To remove the Ubuntu Universe client, perform the following actions: ```text sudo apt remove onedrive ``` #### Step 1c: Remove errant systemd service file installed by Debian / Ubuntu distribution packages The Debian and Ubuntu distribution packages automatically create and enable a default user-level systemd service when installing the onedrive package so that the client runs automatically after authentication. During installation you may see: ``` Created symlink /etc/systemd/user/default.target.wants/onedrive.service → /usr/lib/systemd/user/onedrive.service. ``` This systemd entry is not part of this project’s installation model and is introduced by Debian/Ubuntu packaging defaults. It should be removed. If left in place, it can cause the following error: ``` Opening the item database ... ERROR: onedrive application is already running - check system process list for active application instances - Use 'sudo ps aufxw | grep onedrive' to potentially determine active running process Waiting for all internal threads to complete before exiting application ``` As the client is built with GUI notifications enabled, each automatic restart of this service may also spam your desktop with notifications. To remove this symbolic link created by the distribution package, run: ``` sudo rm /etc/systemd/user/default.target.wants/onedrive.service ``` If this service is not removed, uninstalling the `onedrive` package may result in repeated systemd restart attempts and log entries similar to: ``` Feb 10 10:32:00 host systemd[USER_A]: Started onedrive.service - OneDrive Client for Linux. Feb 10 10:32:00 host (onedrive)[PID_A]: onedrive.service: Unable to locate executable '/usr/bin/onedrive': No such file or directory Feb 10 10:32:00 host (onedrive)[PID_A]: onedrive.service: Failed at step EXEC spawning /usr/bin/onedrive: No such file or directory Feb 10 10:32:00 host systemd[USER_A]: onedrive.service: Main process exited, code=exited, status=203/EXEC Feb 10 10:32:00 host systemd[USER_A]: onedrive.service: Failed with result 'exit-code'. Feb 10 10:32:02 host systemd[USER_B]: Started onedrive.service - OneDrive Client for Linux. Feb 10 10:32:02 host (onedrive)[PID_B]: onedrive.service: Unable to locate executable '/usr/bin/onedrive': No such file or directory Feb 10 10:32:02 host (onedrive)[PID_B]: onedrive.service: Failed at step EXEC spawning /usr/bin/onedrive: No such file or directory Feb 10 10:32:02 host systemd[USER_B]: onedrive.service: Main process exited, code=exited, status=203/EXEC Feb 10 10:32:02 host systemd[USER_B]: onedrive.service: Failed with result 'exit-code'. Feb 10 10:32:03 host systemd[USER_A]: onedrive.service: Scheduled restart job, restart counter is at 201. Feb 10 10:32:03 host systemd[USER_A]: Starting onedrive.service - OneDrive Client for Linux... Feb 10 10:32:05 host systemd[USER_B]: onedrive.service: Scheduled restart job, restart counter is at 105. Feb 10 10:32:05 host systemd[USER_B]: Starting onedrive.service - OneDrive Client for Linux... ``` This behaviour originates from Debian/Ubuntu packaging defaults and does not occur with the OpenSuSE Build Service packages. ### Step 2: Ensure your system is up-to-date Use a script, similar to the following to ensure your system is updated correctly: ```text #!/bin/bash rm -rf /var/lib/dpkg/lock-frontend rm -rf /var/lib/dpkg/lock apt-get update apt-get upgrade -y apt-get dist-upgrade -y apt-get autoremove -y apt-get autoclean -y ``` Run this script as 'root' by using `su -` to elevate to 'root'. Example below: ```text Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-36-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro Expanded Security Maintenance for Applications is not enabled. 0 updates can be applied immediately. Enable ESM Apps to receive additional future security updates. See https://ubuntu.com/esm or run: sudo pro status The list of available updates is more than a week old. To check for new updates run: sudo apt update Last login: Mon Nov 10 06:42:58 2025 from xxx.xxx.xxx.xxx alex@ubuntu-24-04:~$ su - Password: root@ubuntu-24-04:~# ls -la total 36 drwx------ 5 root root 4096 Nov 10 06:43 . drwxr-xr-x 23 root root 4096 Jun 30 2024 .. -rw------- 1 root root 168 Nov 10 06:43 .bash_history -rw-r--r-- 1 root root 3106 Apr 22 2024 .bashrc drwx------ 2 root root 4096 Apr 24 2024 .cache -rw-r--r-- 1 root root 161 Apr 22 2024 .profile drwx------ 6 root root 4096 Jun 30 2024 snap drwx------ 2 root root 4096 Jun 30 2024 .ssh -rwxr-xr-x 1 root root 174 Nov 10 06:43 update_os.sh root@ubuntu-24-04:~# cat update_os.sh #!/bin/bash rm -rf /var/lib/dpkg/lock-frontend rm -rf /var/lib/dpkg/lock apt-get update apt-get upgrade -y apt-get dist-upgrade -y apt-get autoremove -y apt-get autoclean -y root@ubuntu-24-04:~# ./update_os.sh Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB] Hit:2 http://au.archive.ubuntu.com/ubuntu noble InRelease Get:3 http://au.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB] Get:4 http://au.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB] Get:5 http://au.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,585 kB] .... Unpacking libglx-mesa0:amd64 (25.0.7-0ubuntu0.24.04.2) over (24.0.5-1ubuntu1) ... Preparing to unpack .../6-libgl1-amber-dri_21.3.9-0ubuntu3~24.04.1_amd64.deb ... Unpacking libgl1-amber-dri:amd64 (21.3.9-0ubuntu3~24.04.1) over (21.3.9-0ubuntu2) ... (Reading database ... 152058 files and directories currently installed.) Removing libglapi-mesa:amd64 (24.0.5-1ubuntu1) ... Selecting previously unselected package libglapi-amber:amd64. (Reading database ... 152049 files and directories currently installed.) Preparing to unpack .../00-libglapi-amber_21.3.9-0ubuntu3~24.04.1_amd64.deb ... Unpacking libglapi-amber:amd64 (21.3.9-0ubuntu3~24.04.1) ... Selecting previously unselected package libmalcontent-0-0:amd64. Preparing to unpack .../01-libmalcontent-0-0_0.11.1-1ubuntu1.2_amd64.deb ... Unpacking libmalcontent-0-0:amd64 (0.11.1-1ubuntu1.2) ... Preparing to unpack .../02-gnome-control-center_1%3a46.7-0ubuntu0.24.04.2_amd64.deb ... Unpacking gnome-control-center (1:46.7-0ubuntu0.24.04.2) over (1:46.0.1-1ubuntu7) ... Preparing to unpack .../03-libxatracker2_25.0.7-0ubuntu0.24.04.2_amd64.deb ... Unpacking libxatracker2:amd64 (25.0.7-0ubuntu0.24.04.2) over (24.0.5-1ubuntu1) ... Selecting previously unselected package linux-modules-6.14.0-35-generic. Preparing to unpack .../04-linux-modules-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb ... Unpacking linux-modules-6.14.0-35-generic (6.14.0-35.35~24.04.1) ... Selecting previously unselected package linux-image-6.14.0-35-generic. Preparing to unpack .../05-linux-image-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb ... Unpacking linux-image-6.14.0-35-generic (6.14.0-35.35~24.04.1) ... Selecting previously unselected package linux-modules-extra-6.14.0-35-generic. Preparing to unpack .../06-linux-modules-extra-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb ... .... Del libpam-modules-bin 1.5.3-5ubuntu5.1 [51.9 kB] Del systemd-sysv 255.4-1ubuntu8.1 [11.9 kB] root@ubuntu-24-04:~# ``` Reboot your system after running this process before continuing with Step 3. This ensures that your system is correctly up-to-date and any prior running 'onedrive' process and systemd service is now correctly removed and not running. ```text reboot ``` ### Step 3: Determine what your OS is based on Determine what your OS is based on. To do this, run the following command: ```text lsb_release -a ``` **Example:** ```text alex@ubuntu-24-04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 24.04 LTS Release: 24.04 Codename: noble alex@ubuntu-24-04:~$ ``` ### Step 4: Pick the correct instructions to use If required, review the table below based on your 'lsb_release' information to pick the appropriate instructions to use: | Release & Codename | Instructions to use | |--------------------|---------------------| | Linux Mint 19.x | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Linux Mint 22.x | | Linux Mint 20.x | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Linux Mint 22.x | | Linux Mint 21.x | Use [Ubuntu 22.04](#distribution-ubuntu-2204) instructions below | | Linux Mint 22.x | Use [Ubuntu 24.04](#distribution-ubuntu-2404) instructions below | | Linux Mint Debian Edition (LMDE) 5 / Elsie | Use [Debian 11](#distribution-debian-11) instructions below | | Linux Mint Debian Edition (LMDE) 6 / Faye | Use [Debian 12](#distribution-debian-12) instructions below | | Linux Mint Debian Edition (LMDE) 7 / Gigi | Use [Debian 13](#distribution-debian-13) instructions below | | Debian 9 / stretch | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Debian 13 | | Debian 10 / buster | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Debian 13 | | Debian 11 / bullseye | Use [Debian 11](#distribution-debian-11) instructions below | | Debian 12 / bookworm | Use [Debian 12](#distribution-debian-12) instructions below | | Debian 13 / trixie | Use [Debian 13](#distribution-debian-13) instructions below | | Debian Sid | Refer to https://packages.debian.org/sid/onedrive for assistance | | Raspbian GNU/Linux 10 | You must build from source or upgrade your Operating System to Raspbian GNU/Linux 12 | | Raspbian GNU/Linux 11 | Use [Debian 11](#distribution-debian-11) instructions below | | Raspbian GNU/Linux 12 | Use [Debian 12](#distribution-debian-12) instructions below | | Ubuntu 16.04 / Xenial | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 18.04 / Bionic | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 20.04 / Focal | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 21.04 / Hirsute | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 21.10 / Impish | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 22.04 / Jammy | Use [Ubuntu 22.04](#distribution-ubuntu-2204) instructions below | | Ubuntu 22.10 / Kinetic | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 23.04 / Lunar | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 23.10 / Mantic | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 24.04 | | Ubuntu 24.04 / Noble | Use [Ubuntu 24.04](#distribution-ubuntu-2404) instructions below | | Ubuntu 24.10 / Oracular | This platform is **End-of-Life (EOL)** and no longer supported. You must upgrade to at least Ubuntu 25.04 | | Ubuntu 25.04 / Plucky | Use [Ubuntu 25.04](#distribution-ubuntu-2504) instructions below | | Ubuntu 25.10 / Questing | Use [Ubuntu 25.10](#distribution-ubuntu-2510) instructions below | > [!IMPORTANT] > If your Linux distribution or release is **not listed in the table above**, you have two options: > > 1. Compile the client from source. Refer to [Installing or Upgrading the OneDrive Client for Linux](install.md). > 2. Request packaging support from your distribution’s maintainers so that an official, supported package can be provided. ## Distribution Package Install Instructions ### Distribution: Debian 11 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |✔|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_11/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_11/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Debian 12 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |✔|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_12/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_12/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Debian 13 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |✔|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_13/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/Debian_13/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Ubuntu 22.04 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |❌|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_22.04/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_22.04/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Ubuntu 24.04 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |❌|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_24.04/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_24.04/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Ubuntu 25.04 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |❌|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_25.04/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_25.04/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ### Distribution: Ubuntu 25.10 The packages support the following platform architectures: |  i686  | x86_64 | ARMHF | AARCH64 | |:----:|:------:|:-----:|:-------:| |❌|✔|✔|✔| #### Step 1: Add the OpenSuSE Build Service repository release key Add the OpenSuSE Build Service repository release key using the following command: ```text wget -qO - https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_25.10/Release.key | gpg --dearmor | sudo tee /usr/share/keyrings/obs-onedrive.gpg > /dev/null ``` #### Step 2: Add the OpenSuSE Build Service repository Add the OpenSuSE Build Service repository using the following command: ```text echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/obs-onedrive.gpg] https://download.opensuse.org/repositories/home:/npreining:/debian-ubuntu-onedrive/xUbuntu_25.10/ ./" | sudo tee /etc/apt/sources.list.d/onedrive.list ``` #### Step 3: Update your apt package cache Run: `sudo apt-get update` #### Step 4: Install 'onedrive' Run: `sudo apt install --no-install-recommends --no-install-suggests onedrive` #### Step 5: Read 'Known Issues' with these packages Read and understand the [known issues](#known-issues-with-installing-from-the-above-packages) with these packages below, taking any action that is needed. ## Known Issues with Installing from the above packages There are currently no known issues when installing 'onedrive' from the OpenSuSE Build Service repository. ================================================ FILE: docs/usage.md ================================================ # Using the OneDrive Client for Linux ## Application Version Before reading this document, please ensure you are running application version [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) or greater. Use `onedrive --version` to determine what application version you are using and upgrade your client if required. ## Table of Contents - [Important Notes](#important-notes) - [Memory Usage](#memory-usage) - [Guidelines for Local File and Folder Naming in the Synchronisation Directory](#guidelines-for-local-file-and-folder-naming-in-the-synchronisation-directory) - [Support for Microsoft Azure Information Protected Files](#support-for-microsoft-azure-information-protected-files) - [Compatibility with Editors and Applications Using Atomic Save Operations](#compatibility-with-editors-and-applications-using-atomic-save-operations) - [Compatibility with Obsidian](#compatibility-with-obsidian) - [Compatibility with curl](#compatibility-with-curl) - [First Steps](#first-steps) - [Authorise the Application with Your Microsoft OneDrive Account](#authorise-the-application-with-your-microsoft-onedrive-account) - [Display Your Applicable Runtime Configuration](#display-your-applicable-runtime-configuration) - [Understanding OneDrive Client for Linux Operational Modes](#understanding-onedrive-client-for-linux-operational-modes) - [Standalone Synchronisation Operational Mode (Standalone Mode)](#standalone-synchronisation-operational-mode-standalone-mode) - [Ongoing Synchronisation Operational Mode (Monitor Mode)](#ongoing-synchronisation-operational-mode-monitor-mode) - [Using the OneDrive Client for Linux to synchronise your data](#using-the-onedrive-client-for-linux-to-synchronise-your-data) - [Client Documentation](#client-documentation) - [Increasing application logging level](#increasing-application-logging-level) - [Using 'Client Side Filtering' rules to determine what should be synced with Microsoft OneDrive](#using-client-side-filtering-rules-to-determine-what-should-be-synced-with-microsoft-onedrive) - [Why 'Server Side Filtering' is not possible with Microsoft OneDrive](#why-server-side-filtering-is-not-possible-with-microsoft-onedrive) - [Testing your configuration](#testing-your-configuration) - [Performing a sync with Microsoft OneDrive](#performing-a-sync-with-microsoft-onedrive) - [Performing a single directory synchronisation with Microsoft OneDrive](#performing-a-single-directory-synchronisation-with-microsoft-onedrive) - [Performing a 'one-way' download synchronisation with Microsoft OneDrive](#performing-a-one-way-download-synchronisation-with-microsoft-onedrive) - [Performing a 'one-way' upload synchronisation with Microsoft OneDrive](#performing-a-one-way-upload-synchronisation-with-microsoft-onedrive) - [Performing a selective synchronisation via 'sync_list' file](#performing-a-selective-synchronisation-via-sync_list-file) - [Performing a --resync](#performing-a---resync) - [Performing a --force-sync without a --resync or changing your configuration](#performing-a---force-sync-without-a---resync-or-changing-your-configuration) - [Enabling the Client Activity Log](#enabling-the-client-activity-log) - [Client Activity Log Example:](#client-activity-log-example) - [Client Activity Log Differences](#client-activity-log-differences) - [Display Manager Integration](#display-manager-integration) - [GUI Notifications](#gui-notifications) - [Using a local Recycle Bin](#using-a-local-recycle-bin) - [Handling a Microsoft OneDrive Account Password Change](#handling-a-microsoft-onedrive-account-password-change) - [Determining the synchronisation result](#determining-the-synchronisation-result) - [Resumable Transfers](#resumable-transfers) - [Frequently Asked Configuration Questions](#frequently-asked-configuration-questions) - [How to change the default configuration of the client?](#how-to-change-the-default-configuration-of-the-client) - [How to change where my data from Microsoft OneDrive is stored?](#how-to-change-where-my-data-from-microsoft-onedrive-is-stored) - [Why does the client create 'safeBackup' files?](#why-does-the-client-create-safebackup-files) - [How to change what file and directory permissions are assigned to data that is downloaded from Microsoft OneDrive?](#how-to-change-what-file-and-directory-permissions-are-assigned-to-data-that-is-downloaded-from-microsoft-onedrive) - [How are uploads and downloads managed?](#how-are-uploads-and-downloads-managed) - [How to only sync a specific directory?](#how-to-only-sync-a-specific-directory) - [How to 'skip' files from syncing?](#how-to-skip-files-from-syncing) - [How to 'skip' directories from syncing?](#how-to-skip-directories-from-syncing) - [How to 'skip' .files and .folders from syncing?](#how-to-skip-files-and-folders-from-syncing) - [How to 'skip' files larger than a certain size from syncing?](#how-to-skip-files-larger-than-a-certain-size-from-syncing) - [How to 'rate limit' the application to control bandwidth consumed for upload & download operations?](#how-to-rate-limit-the-application-to-control-bandwidth-consumed-for-upload--download-operations) - [How can I prevent my local disk from filling up?](#how-can-i-prevent-my-local-disk-from-filling-up) - [How does the client handle symbolic links?](#how-does-the-client-handle-symbolic-links) - [How to synchronise OneDrive Personal Shared Folders?](#how-to-synchronise-onedrive-personal-shared-folders) - [How to synchronise OneDrive Business Shared Items (Files and Folders)?](#how-to-synchronise-onedrive-business-shared-items-files-and-folders) - [How to synchronise SharePoint / Office 365 Shared Libraries?](#how-to-synchronise-sharepoint--office-365-shared-libraries) - [How to Create a Shareable Link?](#how-to-create-a-shareable-link) - [How to Synchronise Both Personal and Business Accounts at once?](#how-to-synchronise-both-personal-and-business-accounts-at-once) - [How to Synchronise Multiple SharePoint Libraries simultaneously?](#how-to-synchronise-multiple-sharepoint-libraries-simultaneously) - [How to Receive Real-time Changes from Microsoft OneDrive Service, instead of waiting for the next sync period?](#how-to-receive-real-time-changes-from-microsoft-onedrive-service-instead-of-waiting-for-the-next-sync-period) - [How to initiate the client as a background service?](#how-to-initiate-the-client-as-a-background-service) - [OneDrive service running as root user via init.d](#onedrive-service-running-as-root-user-via-initd) - [OneDrive service running as root user via systemd (Arch, Ubuntu, Debian, OpenSuSE, Fedora)](#onedrive-service-running-as-root-user-via-systemd-arch-ubuntu-debian-opensuse-fedora) - [OneDrive service running as root user via systemd (Red Hat Enterprise Linux, CentOS Linux)](#onedrive-service-running-as-root-user-via-systemd-red-hat-enterprise-linux-centos-linux) - [OneDrive service running as a non-root user via systemd (All Linux Distributions)](#onedrive-service-running-as-a-non-root-user-via-systemd-all-linux-distributions) - [OneDrive service running as a non-root user via systemd (with notifications enabled) (Arch, Ubuntu, Debian, OpenSuSE, Fedora)](#onedrive-service-running-as-a-non-root-user-via-systemd-with-notifications-enabled-arch-ubuntu-debian-opensuse-fedora) - [OneDrive service running as a non-root user via runit (antiX, Devuan, Artix, Void)](#onedrive-service-running-as-a-non-root-user-via-runit-antix-devuan-artix-void) - [How to start a user systemd service at boot without user login?](#how-to-start-a-user-systemd-service-at-boot-without-user-login) - [How to access Microsoft OneDrive service through a proxy](#how-to-access-microsoft-onedrive-service-through-a-proxy) - [How to set up SELinux for a sync folder outside of the home folder](#how-to-set-up-selinux-for-a-sync-folder-outside-of-the-home-folder) - [Advanced Configuration of the OneDrive Client for Linux](#advanced-configuration-of-the-onedrive-client-for-linux) - [Overview of all OneDrive Client for Linux CLI Options](#overview-of-all-onedrive-client-for-linux-cli-options) ## Important Notes ### Memory Usage Starting with version 2.5.x, the application has been completely rewritten. It is crucial to understand the memory requirements to ensure the application runs smoothly on your system. During a `--resync` or full online scan, the OneDrive Client may use approximately 1GB of memory for every 100,000 objects stored online. This is because the client retrieves data for all objects via the OneDrive API before processing them locally. Once this process completes, the memory is freed. To avoid performance issues, ensure your system has sufficient available memory. If the system starts using swap space due to insufficient free memory, this can significantly slow down the application and impact overall performance. To avoid potential system instability or the client being terminated by your Out-Of-Memory (OOM) process monitors, please ensure your system has sufficient memory allocated or configure adequate swap space. ### Guidelines for Local File and Folder Naming in the Synchronisation Directory To ensure seamless synchronisation with Microsoft OneDrive, it's critical to adhere strictly to the prescribed naming conventions for your files and folders within the sync directory. The guidelines detailed below are designed to preempt potential sync failures by aligning with Microsoft Windows Naming Conventions, coupled with specific OneDrive restrictions. > [!WARNING] > Failure to comply will result in synchronisation being bypassed for the offending files or folders, necessitating a rename of the local item to establish sync compatibility. #### Key Restrictions and Limitations * Invalid Characters: * Avoid using the following characters in names of files and folders: `" * : < > ? / \ |` * Names should not start or end with spaces * Names should not end with a fullstop / period character `.` * Prohibited Names: * Certain names are reserved and cannot be used for files or folders: `.lock`, `CON`, `PRN`, `AUX`, `NUL`, `COM0 - COM9`, `LPT0 - LPT9`, `desktop.ini`, any filename starting with `~$` * The text sequence `_vti_` cannot appear anywhere in a file or directory name * A file and folder called `forms` is unsupported at the root level of a synchronisation directory * Path Length * All files and folders stored in your 'sync_dir' (typically `~/OneDrive`) must not have a path length greater than: * 400 characters for OneDrive Business & SharePoint * 430 characters for OneDrive Personal Should a file or folder infringe upon these naming conventions or restrictions, synchronisation will skip the item, indicating an invalid name according to Microsoft Naming Convention. The only remedy is to rename the offending item. This constraint is by design and remains firm. > [!TIP] > The UTF-16 character set provides a capability to use alternative characters to work around the restrictions and limitations imposed by Microsoft OneDrive. An example of some replacement characters are below: > | Standard Invalid Character | Potential UTF-16 Replacement Character | > |--------------------|------------------------------| > | . | ․ (One Dot Leader, `\u2024`) | > | : | ː (Modifier Letter Triangular Colon, `\u02D0`) | > | \| | │ (Box Drawings Light Vertical, `\u2502`) | > [!CAUTION] > The last critically important point is that Microsoft OneDrive does not adhere to POSIX standards, which fundamentally impacts naming conventions. In Unix environments (which are POSIX compliant), files and folders can exist simultaneously with identical names if their capitalisation differs. **This is not possible on Microsoft OneDrive.** If such a scenario occurs, the OneDrive Client for Linux will encounter a conflict, preventing the synchronisation of the conflicting file or folder. This constraint is a conscious design choice and is immutable. To avoid synchronisation issues, preemptive renaming of any conflicting local files or folders is advised. #### Further reading: The above guidelines are essential for maintaining synchronisation integrity with Microsoft OneDrive. Adhering to them ensures your files and folders sync without issue. For additional details, consult the following resources: * [Microsoft Windows Naming Conventions](https://docs.microsoft.com/windows/win32/fileio/naming-a-file) * [Restrictions and limitations in OneDrive and SharePoint](https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa) **Adherence to these guidelines is not optional but mandatory to avoid sync disruptions.** ### Support for Microsoft Azure Information Protected Files > [!CAUTION] > If you are using OneDrive Business Accounts and your organisation implements Azure Information Protection, these AIP files will report as one size & hash online, but when downloaded, will report a totally different size and hash. This is due to how the Microsoft Graph API handles AIP files and how Microsoft SharePoint (the technology behind Microsoft OneDrive for Business) serves these files via the API. > > By default these files will fail integrity checking and be deleted locally, meaning that AIP files will not reside on your platform. These AIP files will be flagged as a failed download during application operation. > > If you chose to enable `--disable-download-validation` , the AIP files will download to your platform, however, if there are any other genuine download failures where the size and hash are different, these too will be retained locally meaning you may experience data integrity loss. This is due to the Microsoft Graph API lacking any capability to identify up-front that a file utilises AIP, thus zero capability to differentiate between AIP and non-AIP files for failure detection. > > Please use the `--disable-download-validation` option with extreme caution and understand the risk if you enable it. ### Compatibility with Editors and Applications Using Atomic Save Operations Many modern editors and applications—including `vi`, `vim`, `nvim`, `emacs`, `LibreOffice`, `Obsidian` and others—use *atomic save* strategies to preserve data integrity when writing files. This section outlines how such operations interact with the `onedrive` client, what users can expect, and why certain side effects (such as editor warnings or perceived timestamp discrepancies) may occur. #### How Atomic Save Operations Work When these applications save a file, they typically follow this sequence: 1. **Create a Temporary File** A new file is written with the updated content, often in the same directory as the original. 2. **Flush to Disk** The temporary file is flushed to disk using `fsync()` or an equivalent method to ensure data safety. 3. **Atomic Rename** The temporary file is renamed to the original filename using the `rename()` syscall. This is an atomic operation on Linux, meaning the original file is *replaced*, not modified. 4. **Remove Lock or Swap Files** Auxiliary files used during editing (e.g., `.swp`, `.#filename`) are deleted. As a result, the saved file is **technically a new file** with a new inode and a new timestamp, even if the filename remains unchanged. #### How This Affects the OneDrive Client When the `onedrive` client observes such an atomic save operation via `inotify`, it detects: - The original file as *deleted*. - A new file (with the same name) as *created*. The client responds accordingly: - The "new" file is uploaded to Microsoft OneDrive. - After upload, Microsoft assigns its own *modification timestamp* to the file. - To ensure consistency between local and remote states, the client updates the local file’s timestamp to match the **exact time** stored in OneDrive. > [!IMPORTANT] > Microsoft OneDrive does **not support fractional-second precision** in file timestamps—only whole seconds. As a result, small discrepancies may occur if the local file system supports higher-resolution timestamps. This behaviour ensures accurate syncing and content integrity, but may lead to subtle side effects in timestamp-sensitive applications. #### Expected Side Effects - **Timestamp Alignment for Atomic Saves** Editors that rely on local file timestamps (rather than content checksums) can issue warnings that a file had changed unexpectedly—typically because the `onedrive` client potentially updated the modification time after upload. This client attempts to preserve the original modification timestamp only if fractional seconds differ, preventing unnecessary local timestamp changes. As a result, editors such as `vi`, `vim`, `nvim`, `emacs`, `LibreOffice` and `Obsidian` should not trigger warnings when saving files using atomic operations. - **False Conflict Prompts (Collaborative Editing)** In collaborative editing scenarios—such as with LibreOffice or shared OneDrive folders—conflict prompts may still occur if another user or device modifies a file, resulting in a meaningful timestamp or content change. However, for local edits using atomic save methods, the client now avoids unnecessary timestamp updates, effectively eliminating false conflicts in those cases. #### Recommendation If you are using editors that rely on strict timestamp semantics and wish to minimise interference from the `onedrive` client: - Save your work, then pause or temporarily stop sync (`onedrive --monitor`). - Resume syncing when finished. - Configure the client to ignore temporary files your editor uses via the `skip_file` setting if they do not need to be synced. - Configure the client to use 'session uploads' for all files via the `force_session_upload` setting. This option, when enabled, forces the client to use a 'session' upload, which, when the 'file' is uploaded by the session, this includes the actual local timestamp (without fractional seconds) of the file that Microsoft OneDrive should store. #### Summary The `onedrive` client is fully compatible with applications that use atomic save operations. Users should be aware that: - Atomic saves result in the file being treated as a new item. - Timestamps may be adjusted post-upload to match OneDrive's stored value. - In rare cases, timestamp-sensitive applications may display warnings or prompts. This behaviour is by design and ensures consistency and data integrity between your local filesystem and the OneDrive cloud. ### Compatibility with Obsidian Obsidian on Linux does not provide a built-in way to disable atomic saves or switch to a backup-copy method via configuration. The application is built on Electron and relies on the default save behaviour of its underlying libraries and editor components (such as CodeMirror), which typically perform *atomic writes* using the following process: 1. A temporary file is created containing the updated content. 2. That temporary file is flushed to disk. 3. The temporary file is atomically renamed to replace the original file. This behaviour is intended to improve data integrity and crash resilience, but it results in high disk I/O — particularly in Obsidian, where auto-save is triggered nearly every keystroke. > [!IMPORTANT] > Obsidian provides no mechanism to change how this save behaviour operates. This is a serious design limitation and should be treated as a bug in the application. The excessive and unnecessary write operations can significantly reduce the lifespan of SSDs over time due to increased wear, leading to broader consequences for system reliability. #### How This Affects the OneDrive Client Because Obsidian is constantly writing files, running the OneDrive Client for Linux in `--monitor` mode causes the client to continually receive inotify events from the local file system. This leads to constant re-uploading of files, regardless of whether meaningful content has changed. The consequences of this are: 1. Continuous upload attempts to Microsoft OneDrive. 2. Potential for repeated overwrites of online data. 3. Excessive API usage, which may result in Microsoft throttling your access — subsequently affecting the client’s ability to synchronise files reliably. #### Recommendation If you use Obsidian, it is *strongly* recommended that you enable the following two configuration options in your OneDrive Client for Linux `config` file: ``` force_session_upload = "true" delay_inotify_processing = "true" ``` These settings introduce a delay in processing local file change events, allowing the OneDrive Client for Linux to batch or debounce Obsidian's frequent writes. By default, this delay is 5 seconds. To adjust this delay, you can add the following configuration option: ``` inotify_delay = "10" ``` This example sets the delay to 10 seconds. > [!CAUTION] > Increasing `inotify_delay` too aggressively may have unintended side effects. All file system events are queued and processed in order, so setting a very high delay could result in large backlogs or undesirable data synchronisation outcomes — particularly in cases of rapid file changes or deletions. > > Adjust this setting with extreme caution and test thoroughly to ensure it does not impact your workflow or data integrity. > [!TIP] > An Obsidian Plugin also exists to 'control' the auto save behaviour of Obsidian. > > Instead of saving every two seconds from start of typing (Obsidian default), this plugin makes Obsidian wait for the user to finish with editing, and after the input stops, it waits for a defined time (by default 10 seconds) and then it only saves once. > > For more information please read: https://github.com/mihasm/obsidian-autosave-control ### Compatibility with curl If your system uses curl < 7.47.0, curl will default to HTTP/1.1 for HTTPS operations, and the client will follow suit, using HTTP/1.1. For systems running curl >= 7.47.0 and < 7.62.0, curl will prefer HTTP/2 for HTTPS, but it will still use HTTP/1.1 as the default for these operations. The client will employ HTTP/1.1 for HTTPS operations as well. However, if your system employs curl >= 7.62.0, curl will, by default, prioritise HTTP/2 over HTTP/1.1. In this case, the client will utilise HTTP/2 for most HTTPS operations and stick with HTTP/1.1 for others. Please note that this distinction is governed by the OneDrive platform, not our client. If you explicitly want to use HTTP/1.1, you can do so by using the `--force-http-11` flag or setting the configuration option `force_http_11 = "true"`. This will compel the application to exclusively use HTTP/1.1. Otherwise, all client operations will align with the curl default settings for your distribution. #### Known curl bugs that impact the use of this client | id | curl bug | fixed in curl version | |----|----------|-----------------------| | 1 | HTTP/2 support: Introduced HTTP/2 support, enabling multiplexed transfers over a single connection | 7.47.0 | | 2 | HTTP/2 issue: Resolved an issue where HTTP/2 connections were not properly reused, leading to unnecessary new connections. | 7.68.0 | | 3 | HTTP/2 issue: Addressed a race condition in HTTP/2 multiplexing that could lead to unexpected behaviour. | 7.74.0 | | 4 | HTTP/2 issue: Improved handling of HTTP/2 priority frames to ensure proper stream prioritisation. | 7.81.0 | | 5 | HTTP/2 issue: Fixed a bug where HTTP/2 connections were prematurely closed, resulting in incomplete data transfers. | 7.88.1 | | 6 | HTTP/2 issue: Resolved a problem with HTTP/2 frame handling that could cause data corruption during transfers. | 8.2.1 | | 7 | HTTP/2 issue: Corrected an issue where HTTP/2 streams were not properly closed, leading to potential memory leaks. | 8.5.0 | | 8 | HTTP/2 issue: Addressed a bug where HTTP/2 connections could hang under specific conditions, improving reliability. | 8.8.0 | | 9 | HTTP/2 issue: Improved handling of HTTP/2 connections to prevent unexpected stream resets and enhance stability. | 8.9.0 | | 10 | SIGPIPE issue: Resolved a problem where SIGPIPE signals were not properly handled, leading to unexpected behaviour. | 8.9.1 | | 11 | SIGPIPE issue: Addressed a SIGPIPE leak that occurred in certain cases starting with version 8.9.1 | 8.10.0 | | 12 | HTTP/2 issue: Stopped offering ALPN `http/1.1` for `http2-prior-knowledge` to ensure proper protocol negotiation. | 8.10.0 | | 13 | HTTP/2 issue: Improved handling of end-of-stream (EOS) and blocked states to prevent unexpected behaviour.| 8.11.0 | | 14 | OneDrive operation encountered an issue with libcurl reading the local SSL CA Certificates issue | 8.14.1 | #### Known curl versions with compatibility issues for this client | curl Version | distribution | curl bugs | |--------------|--------------|-----------| | 7.68.0 | Ubuntu 20.04 LTS (Focal Fossa) | 2,3,4,5,6,7,8,9,10,11,12,13 | | 7.74.0 | Debian 11 (Bullseye) | 4,5,6,7,8,9,10,11,12,13 | | 7.81.0 | Ubuntu 22.04 LTS (Jammy Jellyfish) | 5,6,7,8,9,10,11,12,13 | | 7.88.1 | Debian 12 (Bookworm) | 6,7,8,9,10,11,12,13 | | 8.2.1 | Alpine Linux 3.14 | 7,8,9,10,11,12,13 | | 8.5.0 | Alpine Linux 3.15, Ubuntu 24.04 LTS (Noble Numbat) | 8,9,10,11,12,13 | | 8.9.1 | Ubuntu 24.10 (Oracular Oriole) | 11,12,13 | | 8.10.0 | Alpine Linux 3.17 | 13 | | 8.13.0 | Various + Self Compiled | 14 | | 8.13.1 | Various + Self Compiled | 14 | | 8.14.0 | Various + Self Compiled | 14 | > [!IMPORTANT] > If your distribution provides one of these curl versions you must upgrade your curl version to the latest available, or get your distribution to provide a more modern version of curl. Refer to [curl releases](https://curl.se/docs/releases.html) for curl version information. > > If you are using one of the above curl versions, the following application message will be generated: > ```text > WARNING: Your curl/libcurl version (curl.version.number) has known HTTP/2 bugs that impact the use of this application. > Please report this to your distribution and request that they provide a newer curl version for your platform or upgrade this yourself. > Downgrading all application operations to use HTTP/1.1 to ensure maximum operational stability. > Please read https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl for more information. > ``` > > The WARNING line will be sent to the GUI for notification purposes if notifications have been enabled. To avoid this message and/or the GUI notification your only have 2 options: > 1. Upgrade your curl version on your platform > 2. Configure the client to always downgrade client operations to HTTP/1.1 and use IPv4 only > > If you are unable to upgrade your version of curl, to always downgrade client operations to HTTP/1.1 you must add the following to your config file: > ```text > force_http_11 = "true" > ip_protocol_version = "1" > ``` > When these two options are applied to your application configuration, the following application message will be generated: > ```text > WARNING: Your curl/libcurl version (curl.version.number) has known operational bugs that impact the use of this application. > Please report this to your distribution and request that they provide a newer curl version for your platform or upgrade this yourself. > ``` > > The WARNING line will be now only be written to application logging output, no longer sending a GUI notification message. > [!IMPORTANT] > Outside of the above known broken curl versions, there are significant HTTP/2 bugs in all curl versions < 8.6.x that can lead to HTTP/2 errors such as `Error in the HTTP2 framing layer on handle` or `Stream error in the HTTP/2 framing layer on handle` > > The only options to resolve this issue are the following: > 1. Upgrade your curl version to the latest available, or get your distribution to provide a more modern version of curl. Refer to [curl releases](https://curl.se/docs/releases.html) for curl version information. > 2. Configure the client to only use HTTP/1.1 via the config option `--force-http-11` flag or set the configuration file option `force_http_11 = "true"` > [!IMPORTANT] > Outside of the above known broken curl versions, it has also been evidenced that curl has an internal DNS resolution bug that at random times will skip using IPv4 for DNS resolution and only uses IPv6 DNS resolution when the host system is configured to use IPv4 and IPv6 addressing. > > As a result of this internal curl resolution bug, if your system does not have an IPv6 DNS resolver, and/or does not have a valid IPv6 network path to Microsoft OneDrive, you may encounter these errors: > > * `A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response` > * `Could not connect to server on handle ABC12DEF3456` > > The only options to resolve this issue are the following: > 1. Implement and/or ensure that IPv6 DNS resolution is possible on your system; allow IPv6 network connectivity between your system and Microsoft OneDrive > 2. Configure the client to only use IPv4 DNS resolution via setting the configuration option `ip_protocol_version = "1"` > [!IMPORTANT] > If you are using Debian 12 or Linux Mint Debian Edition (LMDE) 6, you can install the latest curl version from the respective backports repositories to address the bugs present in the default Debian 12 curl version. > [!CAUTION] > If you continue to use a curl/libcurl version with known HTTP/2 bugs the application will automatically downgrade HTTP operations to HTTP/1.1, however you will continue to experience application runtime issues such as randomly exiting for zero reason or incomplete download/upload of your data. ## First Steps ### Authorise the Application with Your Microsoft OneDrive Account Once you've installed the application, you'll need to authorise it using your Microsoft OneDrive Account. This can be done by simply running the application without any additional command switches. Please be aware that some organisations may require you to explicitly add this app to the [Microsoft MyApps portal](https://myapps.microsoft.com/). To add an approved app to your apps, click on the ellipsis in the top-right corner and select "Request new apps." On the next page, you can add this app. If it's not listed, you should make a request through your IT department. This client supports the following methods to authenticate the application with Microsoft OneDrive: * Supports interactive browser-based authentication using OAuth2 and a redirect URI * Supports seamless Single Sign-On (SSO) via Intune using the Microsoft Identity Device Broker D-Bus interface * Supports OAuth2 Device Authorisation Flow for Microsoft Entra ID accounts #### Interactive Authentication using OAuth2 and a redirect URI When you run the application for the first time, you'll be prompted to open a specific URL in your web browser. This URL takes you to the Microsoft login page, where you’ll sign in with your Microsoft Account and grant the application permission to access your files. After granting permission, your browser will redirect you to a blank page, or a page that displays this message: ![microsoft-auth-display-message](./images/microsoft-auth-display-message.png) This is expected behaviour. At this point, copy the full redirect URI shown in your browser's address bar and paste it into the terminal where prompted. **Example Terminal Session:** ```text user@hostname:~$ onedrive D-Bus message bus daemon is available; GUI notifications are now enabled Using IPv4 and IPv6 (if configured) for all network operations Attempting to contact Microsoft OneDrive Login Service Successfully reached Microsoft OneDrive Login Service Configuring Global Azure AD Endpoints Please authorise this application by visiting the following URL: https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=d50ca740-c83f-4d1b-b616-12c519384f0c&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient After completing the authorisation in your browser, copy the full redirect URI (from the address bar) and paste it below. Paste redirect URI here: https://login.microsoftonline.com/common/oauth2/nativeclient?code= The application has been successfully authorised, but no extra command options have been specified. Please use 'onedrive --help' for further assistance in regards to running this application. user@hostname:~$ ``` **Interactive OAuth2 Authentication Process Illustrated:** ![initial_auth_url_access_redacted](./images/initial_auth_url_access_redacted.png) ![copy_redirect_uri_to_application](./images/authorise_client_before_copy_with_arrow.png) ![copy_redirect_uri_to_application_done](./images/authorise_client_after_paste_hashed_out.png) ![client_authorised](./images/authorise_client_now_authorised_hashed_out.png) > [!IMPORTANT] > Without additional input or configuration, the OneDrive Client for Linux will automatically adhere to default application settings during synchronisation processes with Microsoft OneDrive. > [!IMPORTANT] > **Handling a AADSTS70000 response** > > If you paste the redirect URI back into the CLI and receive: > `AADSTS70000: The provided value for the 'code' parameter is not valid.` > this is **not a client bug**. > > Microsoft authorisation codes are single-use and short-lived, so the code you pasted is no longer redeemable. > > **Common causes:** > * Browser extensions / privacy tools modifying the redirect URL (for example, ad-blockers or 'remove tracking parameters' features within browsers) > * Copying the wrong URL (ensure you copy from the browser address bar immediately after consent) > * Refreshing the page or reusing the same redirect URI (codes can only be redeemed once) > * Waiting too long before pasting the URL back > > **Remediation steps for AADSTS70000:** > 1. Re-run: `onedrive --reauth` > 2. Use a private/incognito browser session or a clean browser profile > 3. Temporarily disable URL-filtering/privacy extensions for the Microsoft login pages (uBlock Origin / ClearURLs / Brave Shields / similar), then retry #### Single Sign-On (SSO) via Intune using the Microsoft Identity Device Broker To use this method of authentication, you must add the following configuration to your 'config' file: ``` use_intune_sso = "true" ``` The application will check to ensure that Intune is operational and that the required dbus elements are available. Should these be available, the following will be displayed: ``` ... Client has been configured to use Intune SSO via Microsoft Identity Broker dbus session - checking usage criteria Intune SSO via Microsoft Identity Broker dbus session usage criteria met - will attempt to authenticate via Intune ... ``` > [!NOTE] > The installation and configuration of Intune for your platform is beyond the scope of this documentation. #### OAuth2 Device Authorisation Flow for Microsoft Entra ID accounts To use this method of authentication, you must add the following configuration to your 'config' file: ``` use_device_auth = "true" ``` You will be required to open a URL using a web browser, and enter the code that this application presents: ``` Configuring Global Azure AD Endpoints Authorise this application by visiting: https://microsoft.com/devicelogin Enter the following code when prompted: ABCDEFGHI This code expires at: 2025-Jun-02 15:27:30 ``` You will have ~15 minutes before the code expires. > [!IMPORTANT] > #### Limitation: OAuth2 Device Authorization Flow and Personal Microsoft Accounts > > While the OneDrive Client for Linux fully supports OAuth2 Device Authorisation Flow (`device_code` grant) for **Microsoft Entra ID (Work/School)** accounts, **Microsoft currently does not allow this flow to be used with personal Microsoft accounts (MSA)** unless the application is explicitly authorised by Microsoft. > > **Application Configuration Summary:** > > - `signInAudience`: `AzureADandPersonalMicrosoftAccount` > - `allowPublicClient`: `true` > - Uses Microsoft Identity Platform v2.0 endpoints (`/devicecode`, `/token`, etc.) > - Microsoft Graph scopes properly defined > > Despite this correct configuration, users signing in with a Personal Microsoft OneDrive account will see the following error: > > > **"The code you entered has expired. Get a new code from the device you're trying to sign in to and try again."** > > This occurs even if the code is entered immediately. Microsoft redirects the user to: > > ``` > https://login.live.com/ppsecure/post.srf?username=...... > ``` > > This behaviour confirms that Microsoft **blocks the `device_code` grant flow for MSA accounts** on unapproved (by Microsoft) applications. > > **Recommendation:** > If using a Personal Microsoft OneDrive account (e.g., @outlook.com or @hotmail.com), please complete authentication using the interactive authentication method detailed above. > > **Further Reading:** > 📚 [Microsoft Documentation — OAuth 2.0 device authorisation grant](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-device-code) ### Display Your Applicable Runtime Configuration To verify the configuration that the application will use, use the following command: ```text onedrive --display-config ``` This command will display all the relevant runtime interpretations of the options and configurations you are using. An example output is as follows: ```text Reading configuration file: /home/user/.config/onedrive/config Configuration file successfully loaded onedrive version = vX.Y.Z-A-bcdefghi Config path = /home/user/.config/onedrive Config file found in config path = true Config option 'drive_id' = Config option 'sync_dir' = ~/OneDrive ... Config option 'webhook_enabled' = false ``` > [!IMPORTANT] > When using multiple OneDrive accounts, it's essential to always use the `--confdir` command followed by the appropriate configuration directory. This ensures that the specific configuration you intend to view is correctly displayed. ### Understanding OneDrive Client for Linux Operational Modes There are two modes of operation when using the client: 1. Standalone sync mode that performs a single sync action against Microsoft OneDrive. 2. Ongoing sync mode that continuously syncs your data with Microsoft OneDrive. > [!TIP] > To understand further the client operational modes and how the client operates, please review the [client architecture](client-architecture.md) documentation. > [!IMPORTANT] > The default setting for the OneDrive Client on Linux will sync all data from your Microsoft OneDrive account to your local device. To avoid this and select specific items for synchronisation, you should explore setting up 'Client Side Filtering' rules. This will help you manage and specify what exactly gets synced with your Microsoft OneDrive account. #### Standalone Synchronisation Operational Mode (Standalone Mode) This method of use can be employed by issuing the following option to the client: ```text onedrive --sync ``` For simplicity, this can be shortened to the following: ```text onedrive -s ``` #### Ongoing Synchronisation Operational Mode (Monitor Mode) This method of use can be utilised by issuing the following option to the client: ```text onedrive --monitor ``` For simplicity, this can be shortened to the following: ```text onedrive -m ``` > [!NOTE] > This method of use is used when enabling a systemd service to run the application in the background. Two common errors can occur when using monitor mode: * Initialisation failure * Unable to add a new inotify watch Both of these errors are local environment issues, where the following system variables need to be increased as the current system values are potentially too low: * Open Files Soft limit (current session) * Open Files Hard limit (current session) * `fs.inotify.max_user_watches` To determine what the existing values are on your system, use the following commands: **open files** ```text ulimit -Sn ulimit -Hn ``` **inotify watches** ```text sysctl fs.inotify.max_user_watches ``` Alternatively, when running the client with increased verbosity (see below), the client will display what the current configured system maximum values are: ```text ... All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive OneDrive synchronisation interval (seconds): 300 Maximum allowed open files (soft): 1024 Maximum allowed open files (hard): 262144 Maximum allowed inotify user watches: 29463 Initialising filesystem inotify monitoring ... ... ``` To determine what value to change to, you need to count all the files and folders in your configured 'sync_dir' location: ```text cd /path/to/your/sync/dir ls -laR | wc -l ``` To make a change to these variables using your file and folder count, use the following process: **open files** You can increase the limits for your current shell session temporarily using: ``` ulimit -n ``` Refer to your distribution documentation to make the change persistent across reboots and sessions. > [!NOTE] > systemd overrides these values for user sessions and services. If you are making a system wide change that is persistent across reboots and sessions you will also have to modify your systemd service files in the following manner: > ``` > [Service] > LimitNOFILE= > ``` > Post the modification of systemd service files you will need to reload and restart the services. **inotify watches** ```text sudo sysctl fs.inotify.max_user_watches= ``` Once these values are changed, you will need to restart your client so that the new values are detected and used. To make these changes permanent on your system, refer to your OS reference documentation. ## Using the OneDrive Client for Linux to synchronise your data ### Client Documentation The following documents provide detailed guidance on installing, configuring, and using the OneDrive Client for Linux: * **[advanced-usage.md](https://github.com/abraunegg/onedrive/blob/master/docs/advanced-usage.md)** Instructions for advanced configurations, including multiple account setups, Docker usage, dual-boot scenarios, and syncing to mounted directories. * **[application-config-options.md](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md)** Comprehensive list and explanation of all configuration file and command-line options available in the client. * **[application-security.md](https://github.com/abraunegg/onedrive/blob/master/docs/application-security.md)** Details on security considerations and practices related to the OneDrive client. * **[business-shared-items.md](https://github.com/abraunegg/onedrive/blob/master/docs/business-shared-items.md)** Instructions on syncing shared items in OneDrive for Business accounts. * **[client-architecture.md](https://github.com/abraunegg/onedrive/blob/master/docs/client-architecture.md)** Overview of the client's architecture and design principles. * **[docker.md](https://github.com/abraunegg/onedrive/blob/master/docs/docker.md)** Instructions for running the OneDrive client within Docker containers. * **[known-issues.md](https://github.com/abraunegg/onedrive/blob/master/docs/known-issues.md)** List of known issues and limitations of the OneDrive client. * **[national-cloud-deployments.md](https://github.com/abraunegg/onedrive/blob/master/docs/national-cloud-deployments.md)** Information on deploying the client in national cloud environments. * **[podman.md](https://github.com/abraunegg/onedrive/blob/master/docs/podman.md)** Guide for running the OneDrive client using Podman containers. * **[sharepoint-libraries.md](https://github.com/abraunegg/onedrive/blob/master/docs/sharepoint-libraries.md)** Instructions for syncing SharePoint document libraries. * **[ubuntu-package-install.md](https://github.com/abraunegg/onedrive/blob/master/docs/ubuntu-package-install.md)** Specific instructions for installing the client on Ubuntu systems. * **[webhooks.md](https://github.com/abraunegg/onedrive/blob/master/docs/webhooks.md)** Information on configuring and using webhooks with the OneDrive client. Further documentation not listed above can be found here: https://github.com/abraunegg/onedrive/blob/master/docs/ Please read these additional references to assist you with installing, configuring, and using the OneDrive Client for Linux. ### Increasing application logging level When running a sync (`--sync`) or using monitor mode (`--monitor`), it may be desirable to see additional information regarding the progress and operation of the client. The client supports four levels of logging output: #### 1. Normal (default) Only essential information is shown — suitable for standard usage without additional output. #### 2. Verbose Enables general status and progress information. Use: ```text onedrive --sync --verbose ``` or its short form: ```text onedrive -s -v ``` #### 3. Debug Logging Enables detailed internal logging useful for diagnosing issues. This is activated by specifying the `--verbose` flag twice: ```text onedrive --sync --verbose --verbose ``` #### 4. HTTPS Debug Logging Enables full debug logging including HTTPS request/response information. This is typically only needed for advanced debugging of API or network issues. Activate with: ```text onedrive --sync --verbose --verbose --debug-https ``` > [!IMPORTANT] > When raising a bug report or attempting to understand unexpected behaviour, it is recommended to enable debug logging using `--verbose --verbose`. > > Only use `--debug-https` if explicitly requested or required, as it may expose sensitive information in logs. ### Using 'Client Side Filtering' rules to determine what should be synced with Microsoft OneDrive Client Side Filtering in the context of the OneDrive Client for Linux refers to user-configured rules that determine what files and directories the client should upload or download from Microsoft OneDrive. These rules are crucial for optimising synchronisation, especially when dealing with large numbers of files or specific file types. The OneDrive Client for Linux offers several configuration options to facilitate this: * **check_nosync:** This option allows you to create a `.nosync` file in local directories, to skip that directory from being included in sync operations. * **skip_dir:** This option allows the user to specify directories that should not be synchronised with OneDrive. It's particularly useful for omitting large or irrelevant directories from the sync process. * **skip_dotfiles:** Dotfiles, usually configuration files or scripts, can be excluded from the sync. This is useful for users who prefer to keep these files local. * **skip_file:** Specific files can be excluded from synchronisation using this option. It provides flexibility in selecting which files are essential for cloud storage. * **skip_size:** Skip files greater than this specific size (in MB) * **skip_symlinks:** Symlinks often point to files outside the OneDrive directory or to locations that are not relevant for cloud storage. This option prevents them from being included in the sync. Additionally, the OneDrive Client for Linux allows the implementation of Client Side Filtering rules through a 'sync_list' file. This file explicitly states which directories or files should be included in the synchronisation. By default, any item not listed in the 'sync_list' file is excluded. This method offers a more granular approach to synchronisation, ensuring that only the necessary data is transferred to and from Microsoft OneDrive. These configurable options and the 'sync_list' file provide users with the flexibility to tailor the synchronisation process to their specific needs, conserving bandwidth and storage space while ensuring that important files are always backed up and accessible. > [!IMPORTANT] > Client Side Filtering rules are generally processed in the following order: > 1. 'check_nosync' > 2. 'skip_dotfiles' > 3. 'skip_symlinks' > 4. 'skip_dir' > 5. 'skip_file' > 6. 'sync_list' > 7. 'skip_size' > > This can be best illustrated below: > > ![Client Side Filtering Processing Order](./puml/client_side_filtering_processing_order.png) > > For further details please review the [client architecture](client-architecture.md) documentation. > [!IMPORTANT] > After changing any Client Side Filtering rule, you must perform a full re-synchronisation by using `--resync`. ### Why 'Server Side Filtering' is not possible with Microsoft OneDrive It is important to understand that all filtering performed by this client (including `sync_list`) is client-side filtering. Microsoft OneDrive and the Microsoft Graph API do not support server-side selective sync or the ability to apply include/exclude rules when retrieving data. The client must first enumerate the remote filesystem to understand its structure and state, and only then apply filtering rules locally to determine what should be synchronised. This behaviour is expected and is a direct result of platform limitations, not a defect in the client. For further details please read the [server-side filtering limitations](server-side-filtering-limitations.md) documentation. ### Testing your configuration You can test your configuration by utilising the `--dry-run` CLI option. No files will be downloaded, uploaded, or removed; however, the application will display what 'would' have occurred. For example: ```text onedrive --sync --verbose --dry-run Reading configuration file: /home/user/.config/onedrive/config Configuration file successfully loaded Using 'user' Config Dir: /home/user/.config/onedrive DRY-RUN Configured. Output below shows what 'would' have occurred. DRY-RUN: Copying items.sqlite3 to items-dryrun.sqlite3 to use for dry run operations DRY RUN: Not creating backup config file as --dry-run has been used DRY RUN: Not updating hash files as --dry-run has been used Checking Application Version ... Attempting to initialise the OneDrive API ... Configuring Global Azure AD Endpoints The OneDrive API was initialised successfully Opening the item database ... Sync Engine Initialised with new Onedrive API instance Application version: vX.Y.Z-A-bcdefghi Account Type: Default Drive ID: Default Root ID: Remaining Free Space: 1058488129 KB All application operations will be performed in: /home/user/OneDrive Fetching items from the OneDrive API for Drive ID: .. ... Performing a database consistency and integrity check on locally stored data ... Processing DB entries for this Drive ID: Processing ~/OneDrive The directory has not changed ... Scanning local filesystem '~/OneDrive' for new data to upload ... ... Performing a final true-up scan of online data from Microsoft OneDrive Fetching items from the OneDrive API for Drive ID: .. Sync with Microsoft OneDrive is complete ``` ### Performing a sync with Microsoft OneDrive By default, all files are downloaded in `~/OneDrive`. This download location is controlled by the 'sync_dir' config option. After authorising the application, a sync of your data can be performed by running: ```text onedrive --sync ``` This will synchronise files from your Microsoft OneDrive account to your `~/OneDrive` local directory or to your specified 'sync_dir' location. > [!TIP] > #### Specifying the 'source of truth' for your synchronisation with Microsoft OneDrive > By default, the OneDrive Client for Linux treats your online OneDrive data as the source of truth. This means that when determining which version of a file should be trusted as authoritative, the client prioritises the state of files stored online over local copies. > > In some workflows, you may prefer to treat your local files as the primary reference instead — for example, when you regularly make changes locally and want those to take precedence during conflict resolution. > > To change this behaviour, enable the local-first mode by setting the configuration option in your `config` file: > ```text > local_first = "true" > ``` > or by using the command-line argument at runtime: > ```text > onedrive --sync --local-first > ``` > > When this option is enabled, the client will prioritise local data as the source of truth when comparing file differences and resolving synchronisation conflicts. > ### Performing a single directory synchronisation with Microsoft OneDrive In some cases, it may be desirable to synchronise a single directory under ~/OneDrive without having to change your client configuration. To do this, use the following command: ```text onedrive --sync --single-directory '' ``` > [!TIP] > If the full path is `~/OneDrive/mydir`, the command would be `onedrive --sync --single-directory 'mydir'` ### Performing a 'one-way' download synchronisation with Microsoft OneDrive In some cases, it may be desirable to 'download only' from Microsoft OneDrive. To do this, use the following command: ```text onedrive --sync --download-only ``` This will download all the content from Microsoft OneDrive to your `~/OneDrive` location. Any files that are deleted online will remain locally and will not be removed. > [!IMPORTANT] > There is an application functionality change between v2.4.x and v.2.5x when using this option. > > In prior v2.4.x releases, online deletes were automatically processed, thus automatically deleting local files that were deleted online, however there was zero way to perform a `--download-only` operation to archive the online state. > > In v2.5.x and above, when using `--download-only` the default is that all files will remain locally as an archive of your online data rather than being deleted locally if deleted online. > [!TIP] > If you have the requirement to clean up local files that have been removed online, use the following command: > ```text > onedrive --sync --download-only --cleanup-local-files > ``` ### Performing a 'one-way' upload synchronisation with Microsoft OneDrive In certain scenarios, you might need to perform an 'upload only' operation to Microsoft OneDrive. This means that you'll be uploading data to OneDrive, but not synchronising any changes or additions made elsewhere. Use this command to initiate an upload-only synchronisation: ```text onedrive --sync --upload-only ``` > [!IMPORTANT] > - The 'upload only' mode operates independently of OneDrive's online content. It doesn't check or sync with what's already stored on OneDrive. It only uploads data from the local client. > - If a local file or folder that was previously synchronised with Microsoft OneDrive is now missing locally, it will be deleted from OneDrive during this operation. > [!TIP] > If you have the requirement to ensure that all data on Microsoft OneDrive remains intact (e.g., preventing deletion of items on OneDrive if they're deleted locally), use this command instead: > ```text > onedrive --sync --upload-only --no-remote-delete > ``` > [!IMPORTANT] > - `--upload-only`: This command will only upload local changes to OneDrive. These changes can include additions, modifications, moves, and deletions of files and folders. > - `--no-remote-delete`: Adding this command prevents the deletion of any items on OneDrive, even if they're deleted locally. This creates a one-way archive on OneDrive where files are only added and never removed. ### Performing a selective synchronisation via 'sync_list' file Selective synchronisation allows you to sync only specific files and directories. To enable selective synchronisation, create a file named `sync_list` in your application configuration directory (default is `~/.config/onedrive`). > [!IMPORTANT] > Important points to understand before using 'sync_list'. > * 'sync_list' excludes _everything_ by default on OneDrive. > * 'sync_list' follows an _"exclude overrides include"_ rule, and requires **explicit inclusion**. > * Order specific exclusions before inclusions, so that anything _specifically included_ is included. > * How and where you place your `/` matters for excludes and includes in subdirectories. Each line of the 'sync_list' file represents a relative path from your `sync_dir`. All files and directories not matching any line of the file will be skipped during all operations. > [!CAUTION] > Rules without slashes (`Codes`, `Work`, `Backup`, `notes.txt`, etc.) are the most expensive form of `sync_list` rule as this instructs the client to scan every folder online & local to find a match. As a result, these types of rules can cause: > * High CPU usage > * High disk or network activity > * Increased fan usage (especially on laptops) > > If you want best performance, always prefer fully-qualified or path-scoped 'sync_list' rules. Avoid generic includes unless absolutely necessary. #### Example 'sync_list' rules ```text # ====================================================================== # Example sync_list # ====================================================================== # IMPORTANT: # - 'sync_list' EXCLUDES EVERYTHING by default. # - Exclusions come first. # - Inclusions follow. # # Matching behaviour: # - Rules WITHOUT a slash (e.g., "Backup", "notes.txt") match ANYWHERE. # ⚠️ These rules force exhaustive scanning of ALL online and local folders. # ⚠️ They are computationally expensive. # # - Rules with a leading "/" apply ONLY to the OneDrive ROOT. # # - Rules with a trailing "/" match DIRECTORIES only. # # Wildcards and globbing: # - "*" matches any characters within a single path segment. # - "**" matches directories RECURSIVELY across ANY depth. # ====================================================================== # ---------------------------------------------------------------------- # EXCLUSIONS (ALWAYS PUT THESE FIRST) # ---------------------------------------------------------------------- # Exclude temporary folders inside ANY Documents folder (any level) !Documents/temp* # Exclude Secret_data ONLY in OneDrive root !/Secret_data/* # ---------------------------------------------------------------------- # Modern development / programming exclusions # (Common cache/build folders used by many languages & tools) # ---------------------------------------------------------------------- # Python virtual environments !venv/* !.venv/* !__pycache__/* # Node.js / JavaScript build directories !node_modules/* !.next/* # Java & Kotlin build caches !build/kotlin/* !.kotlin/* # Gradle build system cache !.gradle/* # JetBrains IDE caches !.idea/libraries/* !.idea/caches/* # Generic runtime caches !.cache/* # ---------------------------------------------------------------------- # INCLUSIONS (WHAT YOU *DO* WANT TO SYNC) # ---------------------------------------------------------------------- # Include the Backup folder OR any file/folder named "Backup" ANYWHERE. # ⚠️ High-cost rule — causes full tree scanning. Backup # Include Documents directory ANYWHERE # ⚠️ High-cost rule — causes full tree scanning. Documents/ # Include all PDF files inside any Documents folder # ⚠️ High-cost rule — causes full tree scanning. Documents/*.pdf # Include one specific file, if present inside ANY Documents folder # ⚠️ High-cost rule — causes full tree scanning. Documents/latest_report.docx # Include the /Backup/ folder ONLY in the OneDrive root /Backup/ # Include Blender ONLY if in root /Blender # ---------------------------------------------------------------------- # PROJECT / DEVELOPMENT STRUCTURES WITH WILDCARDS & GLOBBING # ---------------------------------------------------------------------- # Include any folder or file beginning with "Project" inside ANY Work/ folder # ⚠️ High-cost rule — causes full tree scanning. Work/Project* # Include the 'Blog' directory — and ONLY that specific folder # . # ├── Parent # │   ├── Blog # │   │   ├── random_files # │   │   │   ├── CZ9aZRM7U1j7pM21fH0MfP2gywlX7bqW # │   │   │   └── k4GptfTBE2z2meRFqjf54tnvSXcXe30Y # │   │   └── random_images # │   │   ├── cAuQMfX7qsMIOmzyQYdELikZwsXeCYsL # │   │   └── GqjZuo7UBB0qjYM2WUcZXOvToAhCQ29M # │   └── other_stuffs # /Parent/Blog/* # Include Android build directories located ANYWHERE inside ANY project !/Programming/Projects/Android/**/build/* # Include Android NDK /.cxx build trees ANYWHERE inside ANY project !/Programming/Projects/Android/**/.cxx/* # Include Web build output directories across ANY nested depth !/Programming/Projects/Web/**/build/* # Include the entire /Programming directory from OneDrive root /Programming # ---------------------------------------------------------------------- # FILE-BY-NAME MATCHING ANYWHERE # ---------------------------------------------------------------------- # Match all files named exactly "notes.txt" ANYWHERE # ⚠️ High-cost rule — causes full tree scanning. notes.txt # ---------------------------------------------------------------------- # DIRECTORIES WITH SPACES # ---------------------------------------------------------------------- # - There is zero requirement to escape space sequences within the 'sync_list' file # Include directories under ANY Pictures folder # ⚠️ High-cost rule — causes full tree scanning. Pictures/Camera Roll Pictures/Saved Pictures # Include 'Camera Roll' and all files / folders /Pictures/Camera Roll/* # Include 'Saved Pictures' and all files / folders /Pictures/Saved Pictures/* # ---------------------------------------------------------------------- # GENERIC NAME MATCHES (⚠️ VERY EXPENSIVE) # These match ANY file or folder with that name ANYWHERE in OneDrive. # They cause full, exhaustive scanning of ALL online and local folders. # ---------------------------------------------------------------------- Cinema Soc Codes Textbooks Year 2 Documents Pictures Music ``` The following are supported for pattern matching and exclusion rules: * Use the `*` to wildcard select any characters to match for the item to be included * Use either `!` or `-` characters at the start of the line to exclude an otherwise included item > [!IMPORTANT] > After changing the sync_list, you must perform a full re-synchronisation by adding `--resync` to your existing command line - for example: `onedrive --sync --resync` > [!TIP] > When enabling the use of 'sync_list,' utilise the `--display-config` option to validate that your configuration will be used by the application, and test your configuration by adding `--dry-run` to ensure the client will operate as per your requirement. > [!TIP] > In some circumstances, it may be required to sync all the individual files within the 'sync_dir' root, but due to frequent name change / addition / deletion of these files, it is not desirable to constantly change the 'sync_list' file to include / exclude these files and force a resync. To assist with this, enable the following in your configuration file: > ```text > sync_root_files = "true" > ``` > This will tell the application to sync any file that it finds in your 'sync_dir' root by default, negating the need to constantly update your 'sync_list' file. ### Performing a --resync A `--resync` operation instructs the client to delete its local state database and fully rebuild it from the current online OneDrive contents. This is a powerful recovery and re-alignment action that should be used **sparingly** and **with care**. > [!IMPORTANT] > **Do not use --resync as part of normal or routine operation.** > > A `--resync` is **not** a “refresh” or “force sync” button. It is a destructive recovery action that discards the client’s local sync history and forces a rebuild based solely on the current online OneDrive state. > > Habitually using `--resync` has several negative impacts: > * It removes the historical sync context the client uses to safely resolve conflicts. > * It can cause unnecessary uploads, downloads, and renames. > * It increases the chance of triggering rate-limiting (HTTP 429 responses) from the Microsoft Graph API. > * It can mask underlying configuration or permission issues that should be properly diagnosed instead. > > If you are unsure whether the client is in sync, do not run `--resync`. Instead, use: >``` > onedrive --display-sync-status >``` > Only use `--resync` when the client explicitly requests it or when a documented configuration change requires it. #### When a --resync is required You **must** perform a `--resync` after modifying any of the following configuration items: * `check_nosync` * `drive_id` * `sync_dir` * `skip_file` * `skip_dir` * `skip_dotfiles` * `skip_size` * `skip_symlinks` * `sync_business_shared_items` * Creating, modifying, or deleting the `sync_list` file You may also use `--resync` if you believe the local state has become inconsistent with online OneDrive state. However, if you only want to check the current sync status, run: ```text onedrive --display-sync-status ``` This shows whether you are up-to-date without requiring a resynchronisation. #### What happens when you use `--resync` When invoking `--resync`, the client displays one of the following prompts depending on the client version. #### v2.5.9 and below ```text The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist. This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss. If in doubt, backup your local data before using --resync Are you sure you wish to proceed with --resync? [Y/N] ``` #### v2.5.10 and above ```text WARNING: You have asked the client to perform a --resync operation. This operation will delete the client’s local state database and rebuild it entirely from the current online OneDrive state. Because the previous sync state will no longer be available, the following may occur: * Local files that also exist in OneDrive may have local changes overwritten by the cloud version if a conflict cannot be safely resolved. * Local files may be renamed or duplicated locally as part of conflict resolution and data-preservation handling. * The initial synchronisation pass may involve a large number of file uploads and downloads. * The increased activity against the Microsoft Graph API may trigger HTTP 429 (throttling) responses during the synchronisation process. For safest operation: * Ensure you have a current backup of your sync_dir. * Run this command first with --dry-run to confirm all planned actions. * Enable 'use_recycle_bin' so that online deletion events from OneDrive are moved to your system Trash rather than deleted from your local disk. If in doubt, stop now and back up your local data before continuing. Are you sure you wish to proceed with --resync? [Y/N] ``` You must press `Y` or `y` to continue with `--resync` action. Any other entry will exit the application. #### Understanding the --resync risks and behaviour A `--resync` **does not delete local-only files**. When a file exists locally but not in OneDrive, and is not excluded via a `sync_list` rule, it is treated as **new local content** and will be uploaded during the resynchronisation process. Local deletion of such files when using `--resync` only occurs when using the explicit local data destructive modes such as: ```text --download-only --cleanup-local-files ``` The risks associated with `--resync` stem entirely from the loss of the local historic state: * The client no longer knows which side previously held the authoritative version of your data. * Conflict handling still protects data using safe-backup mechanisms, but may result in renamed or duplicated files. * Upload and download volumes may spike significantly. * Increased calls to the Microsoft Graph API may result in temporary throttling (HTTP 429 responses). This makes it essential that users **verify actions with `--dry-run`** and **maintain proper backups**. #### Best-practice guidance when using --resync 1. Always back up your data. This client is **not** a backup system. Ensure your `sync_dir` is protected with real backup tooling such as: - rsnapshot - borg - restic - Timeshift - ZFS or Btrfs snapshots 2. Use `--dry-run` before a real `--resync` Allows you to preview all intended changes without modifying your filesystem. 3. Enable the Recycle Bin feature Set `use_recycle_bin = "true"` in your application configuration. When enabled: - Online deletions received from OneDrive via the Graph API are moved to the FreeDesktop.org-compliant system Trash rather than being permanently deleted from your disk. 4. Avoid using `--resync` unnecessarily Only use it: - When the client explicitly requests it, or - When you’ve confirmed, via logs or sync status, that the local state has become invalid > [!CAUTION] > Avoid configuring `--resync` as a default startup option. #### Automated environments If you **fully understand the implications** and are operating in a scripted or automated environment, you may bypass the confirmation prompt by adding: ```bash --resync-auth ``` This should **only** be used when automation requires non-interactive operation and robust backups are in place. ### Performing a --force-sync without a --resync or changing your configuration In some cases and situations, you may have configured the application to skip certain files and folders using 'skip_file' and 'skip_dir' configuration. You then may have a requirement to actually sync one of these items, but do not wish to modify your configuration, nor perform an entire `--resync` twice. The `--force-sync` option allows you to sync a specific directory, ignoring your 'skip_file' and 'skip_dir' configuration and negating the requirement to perform a `--resync`. To use this option, you must run the application manually in the following manner: ```text onedrive --sync --single-directory '' --force-sync ``` When using `--force-sync`, you'll encounter the following warning and advice: ```text WARNING: Overriding application configuration to use application defaults for skip_dir and skip_file due to --sync --single-directory --force-sync being used Using --force-sync will reconfigure the application to use defaults. This may have unknown future impacts. By proceeding with this option, you accept any impacts, including potential data loss resulting from using --force-sync. Are you sure you want to proceed with --force-sync [Y/N] ``` To proceed with `--force-sync`, you must type 'y' or 'Y' to allow the application to continue. ### Enabling the Client Activity Log When running onedrive, all actions can be logged to a separate log file. This can be enabled by using the `--enable-logging` flag or by adding `enable_logging = "true"` to your 'config' file. By default, log files will be written to `/var/log/onedrive/` and will be in the format of `%username%.onedrive.log`, where `%username%` represents the user who ran the client to allow easy sorting of user to client activity log. > [!NOTE] > You will need to ensure the existence of this directory and that your user has the applicable permissions to write to this directory; otherwise, the following error message will be printed: > ```text > ERROR: Unable to access /var/log/onedrive > ERROR: Please manually create '/var/log/onedrive' and set appropriate permissions to allow write access > ERROR: The requested client activity log will instead be located in your user's home directory > ``` On many systems, ensuring that the log directory exists can be achieved by performing the following: ```text sudo mkdir /var/log/onedrive sudo chown root:users /var/log/onedrive sudo chmod 0775 /var/log/onedrive ``` Additionally, you need to ensure that your user account is part of the 'users' group: ``` cat /etc/group | grep users ``` If your user is not part of this group, then you need to add your user to this group: ``` sudo usermod -a -G users ``` If you need to make a group modification, you will need to 'logout' of all sessions / SSH sessions to log in again to have the new group access applied. If the client is unable to write the client activity log, the following error message will be printed: ```text ERROR: Unable to write the activity log to /var/log/onedrive/%username%.onedrive.log ERROR: Please set appropriate permissions to allow write access to the logging directory for your user account ERROR: The requested client activity log will instead be located in your user's home directory ``` If you receive this error message, you will need to diagnose why your system cannot write to the specified file location. #### Client Activity Log Example: An example of a client activity log for the command `onedrive --sync --enable-logging` is below: ```text 2023-Sep-27 08:16:00.1128806 Configuring Global Azure AD Endpoints 2023-Sep-27 08:16:00.1160620 Sync Engine Initialised with new Onedrive API instance 2023-Sep-27 08:16:00.5227122 All application operations will be performed in: /home/user/OneDrive 2023-Sep-27 08:16:00.5227977 Fetching items from the OneDrive API for Drive ID: 2023-Sep-27 08:16:00.7780979 Processing changes and items received from Microsoft OneDrive ... 2023-Sep-27 08:16:00.7781548 Performing a database consistency and integrity check on locally stored data ... 2023-Sep-27 08:16:00.7785889 Scanning the local file system '~/OneDrive' for new data to upload ... 2023-Sep-27 08:16:00.7813710 Performing a final true-up scan of online data from Microsoft OneDrive 2023-Sep-27 08:16:00.7814668 Fetching items from the OneDrive API for Drive ID: 2023-Sep-27 08:16:01.0141776 Processing changes and items received from Microsoft OneDrive ... 2023-Sep-27 08:16:01.0142454 Sync with Microsoft OneDrive is complete ``` An example of a client activity log for the command `onedrive --sync --verbose --enable-logging` is below: ```text 2023-Sep-27 08:20:05.4600464 Checking Application Version ... 2023-Sep-27 08:20:05.5235017 Attempting to initialise the OneDrive API ... 2023-Sep-27 08:20:05.5237207 Configuring Global Azure AD Endpoints 2023-Sep-27 08:20:05.5238087 The OneDrive API was initialised successfully 2023-Sep-27 08:20:05.5238536 Opening the item database ... 2023-Sep-27 08:20:05.5270612 Sync Engine Initialised with new Onedrive API instance 2023-Sep-27 08:20:05.9226535 Application version: vX.Y.Z-A-bcdefghi 2023-Sep-27 08:20:05.9227079 Account Type: 2023-Sep-27 08:20:05.9227360 Default Drive ID: 2023-Sep-27 08:20:05.9227550 Default Root ID: 2023-Sep-27 08:20:05.9227862 Remaining Free Space: 2023-Sep-27 08:20:05.9228296 All application operations will be performed in: /home/user/OneDrive 2023-Sep-27 08:20:05.9228989 Fetching items from the OneDrive API for Drive ID: 2023-Sep-27 08:20:06.2076569 Performing a database consistency and integrity check on locally stored data ... 2023-Sep-27 08:20:06.2077121 Processing DB entries for this Drive ID: 2023-Sep-27 08:20:06.2078408 Processing ~/OneDrive 2023-Sep-27 08:20:06.2078739 The directory has not changed 2023-Sep-27 08:20:06.2079783 Processing Attachments 2023-Sep-27 08:20:06.2080071 The directory has not changed 2023-Sep-27 08:20:06.2081585 Processing Attachments/file.docx 2023-Sep-27 08:20:06.2082079 The file has not changed 2023-Sep-27 08:20:06.2082760 Processing Documents 2023-Sep-27 08:20:06.2083225 The directory has not changed 2023-Sep-27 08:20:06.2084284 Processing Documents/file.log 2023-Sep-27 08:20:06.2084886 The file has not changed 2023-Sep-27 08:20:06.2085150 Scanning the local file system '~/OneDrive' for new data to upload ... 2023-Sep-27 08:20:06.2087133 Skipping item - excluded by sync_list config: ./random_25k_files 2023-Sep-27 08:20:06.2116235 Performing a final true-up scan of online data from Microsoft OneDrive 2023-Sep-27 08:20:06.2117190 Fetching items from the OneDrive API for Drive ID: 2023-Sep-27 08:20:06.5049743 Sync with Microsoft OneDrive is complete ``` #### Client Activity Log Differences Despite application logging being enabled as early as possible, the following log entries will be missing from the client activity log when compared to console output: **No user configuration file:** ```text No user or system config file found, using application defaults Using 'user' configuration path for application state data: /home/user/.config/onedrive Using the following path to store the runtime application log: /var/log/onedrive ``` **User configuration file:** ```text Reading configuration file: /home/user/.config/onedrive/config Configuration file successfully loaded Using 'user' configuration path for application state data: /home/user/.config/onedrive Using the following path to store the runtime application log: /var/log/onedrive ``` ### Display Manager Integration Modern desktop environments such as GNOME and KDE Plasma provide graphical file managers — Nautilus (GNOME Files) and Dolphin, respectively — to help users navigate their local and remote storage. #### What “Display Manager Integration” means Display Manager Integration refers to an ability to integrate your configured Microsoft OneDrive synchronisation directory (`sync_dir`) with the desktop’s file manager environment. Depending on the platform and desktop environment, this may include: 1. **Sidebar registration** — Adding the OneDrive folder as a “special place” within the sidebar of Nautilus (GNOME) or Dolphin (KDE), providing easy access without manual navigation. 2. **Custom folder icon** — Applying a dedicated OneDrive icon to visually distinguish the synchronised directory within the file manager. 3. **Context-menu extensions** — Adding right-click actions such as “Upload to OneDrive” or “Share via OneDrive” directly inside Nautilus or Dolphin. 4. **File overlay badges** — Displaying icons (check-marks, sync arrows, or cloud symbols) to represent file synchronisation state. 5. **System tray or application indicator** — Presenting sync status, pause/resume controls, or notifications via a tray icon. #### What display manager integration is available in the OneDrive Client for Linux The OneDrive Client for Linux currently supports the following integration features: 1. **Sidebar registration** — The client automatically registers the OneDrive folder as a “special place” within the sidebar of Nautilus (GNOME) or Dolphin (KDE). 2. **Custom folder icon** — The client applies a OneDrive-specific icon to the synchronisation directory where supported by the installed icon theme. Sidebar registration and custom folder icon behaviour is controlled by the configuration option: ```text display_manager_integration = "true" ``` When enabled, the client detects the active desktop session and applies the corresponding integration automatically when the client is running in `--monitor` mode only. > [!NOTE] > Display Manager Integration remains active only while the OneDrive client or its systemd service is running. If the client stops or the service is stopped, the desktop integration is automatically cleared. It is re-applied the next time the client starts. #### Fedora (GNOME) Display Manager Integration Example ![fedora_integration](./images/fedora_integration.png) #### Fedora (KDE) Display Manager Integration Example ![fedora_kde_integration](./images/fedora_kde_integration.png) #### Ubuntu Display Manager Integration Example ![ubuntu_integration](./images/ubuntu_integration.png) #### Kubuntu Display Manager Integration Example ![kubuntu_integration](./images/kubuntu_integration.png) Additionally, the following display manager integrations are independent from the above configuration specification: 1. **GUI Notifications** — The client (when compiled with `--enable-notifications`) will send notifications to the GUI when important events occur. 2. **Recycle Bin** — When `use_recycle_bin = "true"` is enabled, the client uses the FreeDesktop.org Trash Specification–compliant recycle bin for any online deletions that are processed locally. This capability can be utilised even when no GUI is available. #### What about context menu integration? Context-menu integration is a desktop-specific capability, not part of the core OneDrive Client. It can be achieved through desktop-provided extension mechanisms: 1. **Shell-script bridge** — A simple shell script can be registered as a KDE ServiceMenu or a GNOME Nautilus Script to trigger local actions (for example, creating a symbolic link in `~/OneDrive` to upload a file). 2. **Python + Nautilus API (GNOME)** — Implemented via nautilus-python bindings by registering a subclass of `Nautilus.MenuProvider`. 3. **Qt/KIO Plugins (KDE)** — Implemented using C++ or declarative .desktop ServiceMenu definitions under `/usr/share/kservices5/ServiceMenus/`. These methods are optional and operate independently of the core OneDrive Client. They can be used by advanced users or system integrators to provide additional right-click functionality. #### What about file overlay badges? File overlay badges are typically associated with Microsoft’s Files-On-Demand feature, which allows selective file downloads and visual state indicators (online-only, available offline, etc.). Because Files-On-Demand is currently a feature request for this client, overlay badges are not implemented and remain out of scope for now. #### What about a system tray or application indicator? While the core OneDrive Client for Linux does not include its own tray icon or GUI dashboard, the community provides complementary tools that plug into it — exposing sync status, pause/resume controls, tray menus, and GUI configuration front-ends. Below are two popular options: **1. OneDriveGUI** - https://github.com/bpozdena/OneDriveGUI * A full-featured graphical user interface built for the OneDrive Linux client. * Key features include: multi-account support, asynchronous real-time monitoring of multiple OneDrive profiles, a setup wizard for profile creation/import, automatic sync on GUI startup, and GUI-based login. * Includes tray icon support when the desktop environment allows it. * Intended to simplify one-click configuration of the CLI client, help users visualise current operations (uploads/downloads), and manage advanced features such as SharePoint libraries and multiple profiles. **2. onedrive_tray** - https://github.com/DanielBorgesOliveira/onedrive_tray * A lightweight system tray utility written in Qt (using libqt5 or later) that monitors the running OneDrive Linux client and displays status via a tray icon. * Left-click the tray icon to view sync progress; right-click to access a menu of available actions; middle-click shows the PID of the running client. * Ideal for users who just want visual status cues (e.g., “sync in progress”, “idle”, “error”) without a full GUI configuration tool. ### GUI Notifications To enable GUI notifications, you must compile the application with GUI Notification Support. Refer to [GUI Notification Support](install.md#gui-notification-support) for details. Once compiled, GUI notifications will work by default in the display manager session under the following conditions: * A D-Bus message bus daemon must be running. * The environment variables XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS must be set. Without these conditions met, GUI notifications will not function even if the support is compiled in. Once these conditions have been met, the following application events will trigger a GUI notification within the display manager session by default: * Aborting a sync if .nosync file is found * Skipping a particular item due to an invalid name * Skipping a particular item due to an invalid symbolic link * Skipping a particular item due to an invalid UTF sequence * Skipping a particular item due to an invalid character encoding sequence * Cannot create remote directory * Cannot upload file changes (free space issue, breaches maximum allowed size, breaches maximum OneDrive Account path length) * Cannot delete remote file / folder * Cannot move remote file / folder * When a re-authentication is required * When a new client version is available * Files that fail to upload * Files that fail to download Additionally, GUI notifications can also be sent for the following activities: * Successful file download * Successful file upload * Successful deletion locally (files and folders) * Successful deletion online (files and folders) To enable these specific notifications, add the following to your 'config' file: ``` notify_file_actions = "true" ``` To disable *all* GUI notifications, add the following to your 'config' file: ``` disable_notifications = "true" ``` ### Using a local Recycle Bin By default, this application will process online deletions and directly delete the corresponding file or folder directly from your configured 'sync_dir'. In some cases, it may actually be desirable to move these files to your Linux user default 'Recycle Bin', so that you can manually delete the files at your own discretion. To enable this application functionality, add the following to your 'config' file: ``` use_recycle_bin = "true" ``` This capability is designed to be compatible with the [FreeDesktop.org Trash Specification](https://specifications.freedesktop.org/trash/latest/), ensuring interoperability with GUI-based desktop environments such as GNOME (GIO) and KDE (KIO). It follows the required structure by: * Moving deleted files and directories to `~/.local/share/Trash/files/` * Creating matching metadata files in `~/.local/share/Trash/info/` with the correct `.trashinfo` format, including the original absolute path and ISO 8601-formatted deletion timestamp * Resolving filename collisions using a `name.N.ext` pattern (e.g., `Document.2.docx`), consistent with GNOME and KDE behaviour. To specify an explicit 'Recycle Bin' directory, add the following to your 'config' file: ``` recycle_bin_path = "/path/to/desired/location/" ``` The same FreeDesktop.org Trash Specification will be used with this explicit 'Recycle Bin' directory as illustrated below: ![using_recycle_bin](./images/using_recycle_bin.png) ### Handling a Microsoft OneDrive Account Password Change If you change your Microsoft OneDrive Account Password, the client will no longer be authorised to sync, and will generate the following error upon next application run: ```text AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '' and the TokensValidFrom date (before which tokens are not valid) for this user is ''. ERROR: You will need to issue a --reauth and re-authorise this client to obtain a fresh auth token. ``` To re-authorise the client, follow the steps below: 1. If running the client as a system service (init.d or systemd), stop the applicable system service 2. Run the command `onedrive --reauth`. This will clean up the previous authorisation, and will prompt you to re-authorise the client as per initial configuration. Please note, if you are using `--confdir` as part of your application runtime configuration, you must include this when telling the client to re-authenticate. 3. Restart the client if running as a system service or perform the standalone sync operation again The application will now sync with OneDrive with the new credentials. ### Determining the synchronisation result When the client has finished syncing without errors, the following will be displayed: ``` Sync with Microsoft OneDrive is complete ``` If any items failed to sync, the following will be displayed: ``` Sync with Microsoft OneDrive has completed, however there are items that failed to sync. ``` A file list of failed upload or download items will also be listed to allow you to determine your next steps. In order to fix the upload or download failures, you may need to: * Review the application output to determine what happened * Re-try your command utilising a resync to ensure your system is correctly synced with your Microsoft OneDrive Account ### Resumable Transfers The OneDrive Client for Linux supports resumable transfers for both uploads and downloads. This capability enhances the reliability and robustness of file transfers by allowing interrupted operations to continue from the last successful point, instead of restarting from the beginning. This is especially important in environments with unstable network connections or during large file transfers. #### What Are Resumable Transfers? A resumable transfer is a process that: * Detects when a file upload or download was interrupted due to a network error, system shutdown, or other external factors. * Saves the current state of the transfer, including offsets, temporary filenames, and online session metadata. * Upon application restart, automatically detects these incomplete operations and resumes them from where they left off. #### When Does It Occur? Resumable transfers are automatically engaged when: * The application is not started with `--resync`. * Interrupted downloads exist with associated metadata saved to disk. * Interrupted uploads using session-based transfers are pending resumption. > [!IMPORTANT] > If a `--resync` operation is being performed, all resumable transfer metadata is purged to ensure a clean and consistent resynchronisation state. #### How It Works Internally * **Downloads:** Partial download state is stored as a JSON metadata file, including the online hash, download URL, and byte offset. The file itself is saved with a `.partial` suffix. When detected, this metadata is parsed and the download resumes using HTTP range headers. * **Uploads:** Session uploads use OneDrive Upload Sessions. If interrupted, the session URL and transfer state are persisted. On restart, the client attempts to resume the upload using the remaining byte ranges. #### Benefits of Resumable Transfers * Saves bandwidth by avoiding full re-transfer of large files. * Improves reliability in poor network conditions. * Increases performance and reduces recovery time after unexpected shutdowns. #### Considerations Resumable state is only preserved if the client exits gracefully or the system preserves temporary files across sessions. If `--resync` is used, all resumable data is discarded intentionally. #### Recommendations * Avoid using `--resync` unless explicitly required. * Enable logging (`--enable-logging`) to help diagnose resumable transfer behaviour. * For environments where network interruptions are common, ensure that the system does not clean temporary or cache files between reboots. > [!NOTE] > Resumable transfer support is built-in and requires no special configuration. It is automatically applied during both standalone and monitor operational modes when applicable. ## Frequently Asked Configuration Questions ### How to change the default configuration of the client? The OneDrive Client for Linux determines its configuration from three layers, applied in the following order of priority: 1. Application default values – internal defaults built into the client 2. Configuration file values – user-defined settings from a config file (if present) 3. Command-line arguments – values passed at runtime override both of the above The built-in application defaults are sufficient for most users and provide a reliable operational baseline. Adding a configuration file or command-line options is optional, and only required when you want to customise application runtime behaviour. >[!NOTE] > The OneDrive Client does not create a configuration file automatically. > If no configuration file is found, the client runs entirely using its internally defined default values. > You only need to create a config file if you wish to override those defaults. If you want to adjust the default settings, download a copy of the configuration template into your local configuration directory. Valid configuration file locations are: * `~/.config/onedrive` – for per-user configuration * `/etc/onedrive` – for system-wide configuration > [!TIP] > To download a copy of the default configuration template, run: > ```text > mkdir -p ~/.config/onedrive > wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/onedrive/config > ``` For a full list of configuration options and command-line switches, see [application-config-options.md](application-config-options.md) ### How to change where my data from Microsoft OneDrive is stored? By default, the location where your Microsoft OneDrive data is stored, is within your Home Directory under a directory called 'OneDrive'. This replicates as close as possible where the Microsoft Windows OneDrive client stores data. To change this location, the application configuration option 'sync_dir' is used to specify a new local directory where your Microsoft OneDrive data should be stored. > [!IMPORTANT] > Please be aware that if you designate a network mount point (such as NFS, Windows Network Share, or Samba Network Share) as your `sync_dir`, this setup inherently lacks 'inotify' support. Support for 'inotify' is essential for real-time tracking of local file changes, which means that the client's 'Monitor Mode' cannot immediately detect changes in files located on these network shares. Instead, synchronisation between your local filesystem and Microsoft OneDrive will occur at intervals specified by the `monitor_interval` setting. This limitation regarding 'inotify' support on network mount points like NFS or Samba is beyond the control of this client. ### Why does the client create 'safeBackup' files? 'safeBackup' files are created to prevent local data loss whenever the client is about to replace or remove a local file and there’s any chance the current on-disk content might be different to what OneDrive expects. Under the hood, the client makes specific decisions right before a local file would otherwise be overwritten, renamed, or deleted. Instead of risking silent data loss, the client renames your current local file to a clearly marked backup name and then proceeds with the sync action. From v2.5.3+, the backup name is: ``` filename-hostname-safeBackup-0001.ext ``` The client will increment the number if additional backups are needed. #### The most common reasons you’ll see 'safeBackup' files **1. You ran the client with `--resync`** `--resync` intentionally discards the client’s local state, so the client no longer “knows” what used to be in sync. During the first pass after a resync, the online state is treated as source-of-truth. If the client finds a local file whose content differs from the online version (hash mismatch), it will back up your local copy first and then bring the local file in line with OneDrive. If you wish to treat your local files as the source-of-truth, you can set the following configuration option: ``` local_first = "true" ``` **2. Dual-booting and pointing sync_dir at your Windows OneDrive folder.** If you dual boot and set the Linux client’s sync_dir to the same path used by the Windows client, there will be times when files already exist on disk without matching local DB entries or with content that changed while Linux wasn’t running. When the Linux client encounters such a file (e.g. “exists locally but isn’t represented the way the DB expects” or “exists but content/hash differs”), the client will protect the on-disk content by creating a 'safeBackup' before it reconciles the file. **3. The online file was modified (server-side) and now differs from your local copy** If Microsoft OneDrive (or another app) changes a file online, the hash reported by the Graph API won’t match your local content. When the client is about to update the local item to match what’s online, a 'safeBackup' is created so your current local data isn’t lost if the client determines that this action should be taken. #### Can I turn this functionality off? Yes, but be careful. To disable local data protection entirely, set the following configuration option: ``` bypass_data_preservation = "true" ``` If you enable this, the client will not create 'safeBackup' files and may overwrite or remove local content during conflict resolution. **Use with extreme caution.** If you simply don’t want 'safeBackup' files uploaded to OneDrive, it is advisable to keep protection enabled and add a 'skip_file' rule: ``` skip_file = "~*|.~*|*.tmp|*.swp|*.partial|*-safeBackup-*" ``` This allows you to handle the safeBackup files locally, without having to remediate anything online. ### How to change what file and directory permissions are assigned to data that is downloaded from Microsoft OneDrive? The following are the application default permissions for any new directory or file that is created locally when downloaded from Microsoft OneDrive: * Directories: 700 - This provides the following permissions: `drwx------` * Files: 600 - This provides the following permissions: `-rw-------` These default permissions align to the security principal of 'least privilege' so that only you should have access to your data that you download from Microsoft OneDrive. To alter these default permissions, you can adjust the values of two configuration options as follows. You can also use the [Unix Permissions Calculator](https://chmod-calculator.com/) to help you determine the necessary new permissions. ```text sync_dir_permissions = "700" sync_file_permissions = "600" ``` > [!IMPORTANT] > Please note that special permission bits such as setuid, setgid, and the sticky bit are not supported. Valid permission values range from `000` to `777` only. > [!NOTE] > To prevent the application from modifying file or directory permissions and instead rely on the existing file system permission inheritance, add `disable_permission_set = "true"` to your configuration file. ### How are uploads and downloads managed? The system manages downloads and uploads using a multi-threaded approach. Specifically, the application utilises by default 8 threads (a maximum of 16 can be configured) for these processes. Refer to [configuration documentation](application-config-options.md#threads) for further details. ### How to only sync a specific directory? There are two methods to achieve this: * Employ the '--single-directory' option to only sync this specific path * Employ 'sync_list' as part of your 'config' file to configure what files and directories to sync, and what should be excluded ### How to 'skip' files from syncing? There are two methods to achieve this: * Employ 'skip_file' as part of your 'config' file to configure what files to skip * Employ 'sync_list' to configure what files and directories to sync, and what should be excluded For further details please read the ['skip_file' config option documentation](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md#skip_file) ### How to 'skip' directories from syncing? There are three methods available to 'skip' a directory from the sync process: * Employ 'skip_dir' as part of your 'config' file to configure what directories to skip * Employ 'sync_list' to configure what files and directories to sync, and what should be excluded * Employ 'check_nosync' as part of your 'config' file and a '.nosync' empty file within the directory to exclude to skip that directory > [!IMPORTANT] > Entries for 'skip_dir' are *relative* to your 'sync_dir' path. For further details please read the ['skip_dir' config option documentation](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md#skip_dir) ### How to 'skip' .files and .folders from syncing? There are three methods to achieve this: * Employ 'skip_file' or 'skip_dir' to configure what files or folders to skip * Employ 'sync_list' to configure what files and directories to sync, and what should be excluded * Employ 'skip_dotfiles' as part of your 'config' file to skip any dot file (for example: `.Trash-1000` or `.xdg-volume-info`) from syncing to OneDrive ### How to 'skip' files larger than a certain size from syncing? Use `skip_size = "value"` as part of your 'config' file where files larger than this size (in MB) will be skipped. ### How to 'rate limit' the application to control bandwidth consumed for upload & download operations? To optimise Internet bandwidth usage during upload and download processes, include the 'rate_limit' setting in your configuration file. This setting controls the bandwidth allocated to each thread. By default, 'rate_limit' is set to '0', indicating that the application will utilise the maximum available bandwidth across all threads. To check the current 'rate_limit' value, use the `--display-config` command. > [!NOTE] > Since downloads and uploads are processed through multiple threads, the 'rate_limit' value applies to each thread separately. For instance, setting 'rate_limit' to 1048576 (1MB) means that during data transfers, the total bandwidth consumption might reach around 16MB, not just the 1MB configured due to the number of threads being used. ### How can I prevent my local disk from filling up? By default, the application will reserve 50MB of disk space to prevent your filesystem from running out of disk space. This default value can be modified by adding the 'space_reservation' configuration option and the applicable value as part of your 'config' file. You can review the value being used when using `--display-config`. ### How does the client handle symbolic links? Microsoft OneDrive has no concept or understanding of symbolic links, and attempting to upload a symbolic link to Microsoft OneDrive generates a platform API error. All data (files and folders) that are uploaded to OneDrive must be whole files or actual directories. As such, there are only two methods to support symbolic links with this client: 1. Follow the Linux symbolic link and upload whatever the local symbolic link is pointing to to Microsoft OneDrive. This is the default behaviour. 2. Skip symbolic links by configuring the application to do so. When skipping, no data, no link, no reference is uploaded to OneDrive. Use 'skip_symlinks' as part of your 'config' file to configure the skipping of all symbolic links while syncing. ### How to synchronise OneDrive Personal Shared Folders? Folders shared with you can be synchronised by adding them to your OneDrive online. To do that, open your OneDrive account online, go to the Shared files list, right-click on the folder you want to synchronise, and then click on "Add to my OneDrive". ### How to synchronise OneDrive Business Shared Items (Files and Folders)? Folders shared with you can be synchronised by adding them to your OneDrive online. To do that, open your OneDrive account online, go to the Shared files list, right-click on the folder you want to synchronise, and then click on "Add to my OneDrive". Files shared with you can be synchronised using two methods: 1. Add a shortcut link to the file to your OneDrive folder online 2. Sync the actual file locally using the configuration option to sync OneDrive Business Shared Files. Refer to [business-shared-items.md](business-shared-items.md) for further details. ### How to synchronise SharePoint / Office 365 Shared Libraries? There are two methods to achieve this: * SharePoint library can be directly added to your OneDrive online. To do that, open your OneDrive account online, go to the Shared files list, right-click on the SharePoint Library you want to synchronise, and then click on "Add to my OneDrive". * Configure a separate application instance to only synchronise that specific SharePoint Library. Refer to [sharepoint-libraries.md](sharepoint-libraries.md) for configuration assistance. ### How to Create a Shareable Link? In certain situations, you might want to generate a shareable file link and provide this link to other users for accessing a specific file. To accomplish this, employ the following command: ```text onedrive --create-share-link ``` > [!IMPORTANT] > By default, this access permissions for the file link will be read-only. To make the shareable link a read-write link, execute the following command: ```text onedrive --create-share-link --with-editing-perms ``` > [!IMPORTANT] > The order of the file path and option flag is crucial. ### How to Synchronise Both Personal and Business Accounts at once? You need to set up separate instances of the application configuration for each account. Refer to [advanced-usage.md](advanced-usage.md) for guidance on configuration. ### How to Synchronise Multiple SharePoint Libraries simultaneously? For each SharePoint Library, configure a separate instance of the application configuration. Refer to [advanced-usage.md](advanced-usage.md) for configuration instructions. ### How to Receive Real-time Changes from Microsoft OneDrive Service, instead of waiting for the next sync period? Refer to [webhooks.md](webhooks.md) for configuration instructions. ### How to initiate the client as a background service? There are a few ways to employ onedrive as a service: * via init.d * via systemd * via runit #### OneDrive service running as root user via init.d ```text chkconfig onedrive on service onedrive start ``` To view the logs, execute: ```text tail -f /var/log/onedrive/.onedrive.log ``` To alter the 'user' under which the client operates (typically root by default), manually modify the init.d service file and adjust `daemon --user root onedrive_service.sh` to match the correct user. #### OneDrive service running as root user via systemd (Arch, Ubuntu, Debian, OpenSuSE, Fedora) Initially, switch to the root user with `su - root`, then activate the systemd service: ```text systemctl --user enable onedrive systemctl --user start onedrive ``` > [!IMPORTANT] > This will execute the 'onedrive' process with a UID/GID of '0', which means any files or folders created will be owned by 'root'. > [!IMPORTANT] > The `systemctl --user` command is not applicable to Red Hat Enterprise Linux (RHEL) or CentOS Linux platforms - see below. To monitor the service's status, use the following: ```text systemctl --user status onedrive.service ``` To observe the systemd application logs, use: ```text journalctl --user-unit=onedrive -f ``` > [!TIP] > For systemd to function correctly, it requires the presence of XDG environment variables. If you encounter the following error while enabling the systemd service: > ```text > Failed to connect to bus: No such file or directory > ``` > The most likely cause is missing XDG environment variables. To resolve this, add the following lines to `.bashrc` or another file executed upon user login: > ```text > export XDG_RUNTIME_DIR="/run/user/$UID" > export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus" > ``` > > To apply this change, you must log out of all user accounts where it has been made. > [!IMPORTANT] > On certain systems (e.g., Raspbian / Ubuntu / Debian on Raspberry Pi), the XDG fix above may not persist after system reboots. An alternative to starting the client via systemd as root is as follows: > 1. Create a symbolic link from `/home/root/.config/onedrive` to `/root/.config/onedrive/`. > 2. Establish a systemd service using the '@' service file: `systemctl enable onedrive@root.service`. > 3. Start the root@service: `systemctl start onedrive@root.service`. > > This ensures that the service correctly restarts upon system reboot. To examine the systemd application logs, run: ```text journalctl --unit=onedrive@ -f ``` #### OneDrive service running as root user via systemd (Red Hat Enterprise Linux, CentOS Linux) ```text systemctl enable onedrive systemctl start onedrive ``` > [!IMPORTANT] > This will execute the 'onedrive' process with a UID/GID of '0', meaning any files or folders created will be owned by 'root'. To view the systemd application logs, execute: ```text journalctl --unit=onedrive -f ``` #### OneDrive service running as a non-root user via systemd (All Linux Distributions) In some instances, it is preferable to run the OneDrive client as a service without the 'root' user. Follow the instructions below to configure the service for your regular user login. 1. As the user who will run the service, launch the application in standalone mode, authorise it for use, and verify that synchronisation is functioning as expected: ```text onedrive --sync --verbose ``` 2. After validating the application for your user, switch to the 'root' user, where is your username from step 1 above. ```text systemctl enable onedrive@.service systemctl start onedrive@.service ``` 3. To check the service's status for the user, use the following: ```text systemctl status onedrive@.service ``` To observe the systemd application logs, use: ```text journalctl --unit=onedrive@ -f ``` #### OneDrive service running as a non-root user via systemd (with notifications enabled) (Arch, Ubuntu, Debian, OpenSuSE, Fedora) In some scenarios, you may want to receive GUI notifications when using the client as a non-root user. In this case, follow these steps: 1. Log in via the graphical UI as the user you want to enable the service for. 2. Disable any `onedrive@` service files for your username, e.g.: ```text sudo systemctl stop onedrive@alex.service sudo systemctl disable onedrive@alex.service ``` 3. Enable the service as follows: ```text systemctl --user enable onedrive systemctl --user start onedrive ``` To check the service's status for the user, use the following: ```text systemctl --user status onedrive.service ``` To view the systemd application logs, execute: ```text journalctl --user-unit=onedrive -f ``` > [!IMPORTANT] > The `systemctl --user` command is not applicable to Red Hat Enterprise Linux (RHEL) or CentOS Linux platforms. #### OneDrive service running as a non-root user via runit (antiX, Devuan, Artix, Void) 1. Create the following folder if it doesn't already exist: `/etc/sv/runsvdir-` - where `` is the `USER` targeted for the service - e.g., `# mkdir /etc/sv/runsvdir-nolan` 2. Create a file called `run` under the previously created folder with executable permissions - `# touch /etc/sv/runsvdir-/run` - `# chmod 0755 /etc/sv/runsvdir-/run` 3. Edit the `run` file with the following contents (permissions needed): ```sh #!/bin/sh export USER="" export HOME="/home/" groups="$(id -Gn "${USER}" | tr ' ' ':')" svdir="${HOME}/service" exec chpst -u "${USER}:${groups}" runsvdir "${svdir}" ``` - Ensure you replace `` with the `USER` set in step #1. 4. Enable the previously created folder as a service - `# ln -fs /etc/sv/runsvdir- /var/service/` 5. Create a subfolder in the `USER`'s `HOME` directory to store the services (or symlinks) - `$ mkdir ~/service` 6. Create a subfolder specifically for OneDrive - `$ mkdir ~/service/onedrive/` 7. Create a file called `run` under the previously created folder with executable permissions - `$ touch ~/service/onedrive/run` - `$ chmod 0755 ~/service/onedrive/run` 8. Append the following contents to the `run` file ```sh #!/usr/bin/env sh exec /usr/bin/onedrive --monitor ``` - In some scenarios, the path to the `onedrive` binary may vary. You can obtain it by running `$ command -v onedrive`. 9. Reboot to apply the changes 10. Check the status of user-defined services - `$ sv status ~/service/*` > [!NOTE] > For additional details, you can refer to Void's documentation on [Per-User Services](https://docs.voidlinux.org/config/services/user-services.html) ### How to start a user systemd service at boot without user login? In some situations, it may be necessary for the systemd service to start without requiring your 'user' to log in. To address this issue, you need to reconfigure your 'user' account so that the systemd services you've created launch without the need for you to log in to your system: ```text loginctl enable-linger ``` ### How to access Microsoft OneDrive service through a proxy If you have a requirement to run the client through a proxy, there are a couple of ways to achieve this: #### Option 1: Use '.bashrc' to specify the proxy server details Set proxy configuration in `~/.bashrc` to allow the 'onedrive' application to use a specific proxy server: ```text # Set the HTTP proxy export http_proxy="http://your.proxy.server:port" # Set the HTTPS proxy export https_proxy="http://your.proxy.server:port" ``` Once you've edited your `~/.bashrc` file, run the following command to apply the changes: ``` source ~/.bashrc ``` #### Option 2: Update the 'systemd' service file to include the proxy server details If running as a systemd service, edit the applicable systemd service file to include the proxy configuration information: ```text [Unit] Description=OneDrive Client for Linux Documentation=https://github.com/abraunegg/onedrive After=network-online.target Wants=network-online.target [Service] ........ Environment="HTTP_PROXY=http://your.proxy.server:port" Environment="HTTPS_PROXY=http://your.proxy.server:port" ExecStart=/usr/local/bin/onedrive --monitor ........ ``` > [!NOTE] > After modifying the service files, you will need to run `sudo systemctl daemon-reload` to ensure the service file changes are picked up. A restart of the OneDrive service will also be required to pick up the change to send the traffic via the proxy server ### How to set up SELinux for a sync folder outside of the home folder If SELinux is enforced and the sync folder is outside of the home folder, as long as there is no policy for cloud file service providers, label the file system folder to `user_home_t`. ```text sudo semanage fcontext -a -t user_home_t /path/to/onedriveSyncFolder sudo restorecon -R -v /path/to/onedriveSyncFolder ``` To remove this change from SELinux and restore the default behaviour: ```text sudo semanage fcontext -d /path/to/onedriveSyncFolder sudo restorecon -R -v /path/to/onedriveSyncFolder ``` ## Advanced Configuration of the OneDrive Client for Linux Refer to [advanced-usage.md](advanced-usage.md) for further details on the following topics: * Configuring the client to use multiple OneDrive accounts / configurations * Configuring the client to use multiple OneDrive accounts / configurations using Docker * Configuring the client for use in dual-boot (Windows / Linux) situations * Configuring the client for use when 'sync_dir' is a mounted directory * Upload data from the local ~/OneDrive folder to a specific location on OneDrive ## Overview of all OneDrive Client for Linux CLI Options Below is a comprehensive list of all available configuration options for the OneDrive Client for Linux, as shown by the output of `onedrive --help`. These commands provide a range of options for synchronising, monitoring, and managing files between your local system and Microsoft's OneDrive cloud service. The following configuration options are available: ```text onedrive - A client for the Microsoft OneDrive Cloud Service Usage: onedrive [options] --sync Do a one-time synchronisation with Microsoft OneDrive onedrive [options] --monitor Monitor filesystem and synchronise regularly with Microsoft OneDrive onedrive [options] --display-config Display the currently used configuration onedrive [options] --display-sync-status Query OneDrive service and report on pending changes onedrive -h | --help Show this help screen onedrive --version Show version Options: --auth-files '' Perform authentication via files rather than an interactive dialogue. The application reads/writes the required values from/to the specified files --auth-response '' Perform authentication via a supplied response URL rather than an interactive dialogue --check-for-nomount Check for the presence of .nosync in the syncdir root. If found, do not perform sync --check-for-nosync Check for the presence of .nosync in each directory. If found, skip directory from sync --classify-as-big-delete '' Number of children in a path that is locally removed which will be classified as a 'big data delete' --cleanup-local-files Clean up additional local files when using --download-only. This will remove local data --confdir '' Set the directory used to store the configuration files --create-directory '' Create a directory on OneDrive. No synchronisation will be performed --create-share-link '' Create a shareable link for an existing file on OneDrive --debug-https Debug OneDrive HTTPS communication. --destination-directory '' Destination directory for renamed or moved items on OneDrive. No synchronisation will be performed --disable-download-validation Disable download validation when downloading from OneDrive --disable-notifications Do not use desktop notifications in monitor mode --disable-upload-validation Disable upload validation when uploading to OneDrive --display-config Display what options the client will use as currently configured. No synchronisation will be performed --display-quota Display the quota status of the client. No synchronisation will be performed --display-running-config Display what options the client has been configured to use on application startup --display-sync-status Display the sync status of the client. No synchronisation will be performed --download-file '' Download a single file from Microsoft OneDrive --download-only Replicate the OneDrive online state locally, by only downloading changes from OneDrive. Do not upload local changes to OneDrive --dry-run Perform a trial sync with no changes made --enable-logging Enable client activity to a separate log file --file-fragment-size Specify the file fragment size for large file uploads (in MB) --force Force the deletion of data when a 'big delete' is detected --force-http-11 Force the use of HTTP 1.1 for all operations --force-sync Force a synchronisation of a specific folder, only when using --sync --single-directory and ignore all non-default skip_dir and skip_file rules --get-O365-drive-id '' Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library (DEPRECATED) --get-file-link '' Display the file link of a synced file --get-sharepoint-drive-id '' Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library --help -h This help information. --list-shared-items List OneDrive Business Shared Items --local-first Synchronise from the local directory source first, before downloading changes from OneDrive --log-dir '' Directory where logging output is saved to, needs to end with a slash --logout Log out the current user --modified-by '' Display the last modified by details of a given path --monitor -m Keep monitoring for local and remote changes --monitor-fullscan-frequency '' Number of sync runs before performing a full local scan of the synced directory --monitor-interval '' Number of seconds by which each sync operation is undertaken when idle under monitor mode --monitor-log-frequency '' Frequency of logging in monitor mode --no-remote-delete Do not delete local file 'deletes' from OneDrive when using --upload-only --print-access-token Print the access token, useful for debugging --reauth Reauthenticate the client with OneDrive --remove-directory '' Remove a directory on OneDrive. No synchronisation will be performed --remove-source-files Remove source file after successful transfer to OneDrive when using --upload-only --remove-source-folders Remove the local directory structure post successful file transfer to Microsoft OneDrive when using --upload-only --remove-source-files --resync Forget the last saved state, perform a full sync --resync-auth Approve the use of performing a --resync action --share-password '' Require a password to access the shared link when used with --create-share-link --single-directory '' Specify a single local directory within the OneDrive root to sync --skip-dir '' Skip any directories that match this pattern from syncing --skip-dir-strict-match When matching skip_dir directories, only match explicit matches --skip-dot-files Skip dot files and folders from syncing --skip-file '' Skip any files that match this pattern from syncing --skip-size '' Skip new files larger than this size (in MB) --skip-symlinks Skip syncing of symlinks --source-directory '' Source directory to rename or move on OneDrive. No synchronisation will be performed --space-reservation '' The amount of disk space to reserve (in MB) to avoid 100% disk space utilisation --sync -s Perform a synchronisation with Microsoft OneDrive --sync-root-files Sync all files in sync_dir root when using sync_list --sync-shared-files Sync OneDrive Business Shared Files to the local filesystem --syncdir '' Specify the local directory used for synchronisation to OneDrive --synchronize Perform a synchronisation with Microsoft OneDrive (DEPRECATED) --threads Specify a value for the number of worker threads used for parallel upload and download operations --upload-only Replicate the locally configured sync_dir state to OneDrive, by only uploading local changes to OneDrive. Do not download changes from OneDrive --verbose -v+ Print more details, useful for debugging (repeat for extra debugging) --version Print the version and exit --with-editing-perms Create a read-write shareable link for an existing file on OneDrive when used with --create-share-link ``` Refer to [application-config-options.md](application-config-options.md) for in-depth details on all application options. ================================================ FILE: docs/webhooks.md ================================================ # How to configure receiving real-time changes from Microsoft OneDrive using webhooks When operating in 'Monitor Mode,' receiving real-time updates to online data can significantly enhance synchronisation efficiency. This is achieved by enabling 'webhooks,' which allows the client to subscribe to remote updates and receive real-time notifications when certain events occur on Microsoft OneDrive. With this setup, any remote changes are promptly synchronised to your local file system, eliminating the need to wait for the next scheduled synchronisation cycle. > [!IMPORTANT] > In March 2023, Microsoft updated the webhook notification capability in Microsoft Graph to only allow valid HTTPS URLs as the destination for subscription updates. > > This change was part of Microsoft's ongoing efforts to enhance security and ensure that all webhooks used with Microsoft Graph comply with modern security standards. The enforcement of this requirement prevents the registration of subscriptions with non-secure (HTTP) endpoints, thereby improving the security of data transmission. > > Therefore, as a prerequisite, you must have a valid fully qualified domain name (FQDN) for your system that is externally resolvable, or configure Dynamic DNS (DDNS) using a provider such as: > * No-IP > * DynDNS > * DuckDNS > * Afraid.org > * Cloudflare > * Google Domains > * Dynu > * ChangeIP > > This FQDN will allow you to create a valid HTTPS certificate for your system, which can be used by Microsoft Graph for webhook functionality. > > Please note that it is beyond the scope of this document to provide guidance on setting up this requirement. Depending on your environment, a number of steps are required to configure this application functionality. At a very high level these configuration steps are: 1. Application configuration to enable 'webhooks' functionality 2. Install and configure 'nginx' as a reverse proxy for HTTPS traffic 3. Install and configure Let's Encrypt 'certbot' to provide a valid HTTPS certificate for your system using your FQDN 4. Configure your Firewall or Router to forward traffic to your system > [!NOTE] > The configuration steps below were validated on [Fedora 40 Workstation](https://fedoraproject.org/) > > The installation of required components (nginx, certbot) for your platform is beyond the scope of this document and it is assumed you know how to install these components. If you are unsure, please seek support from your Linux distribution support channels. ### Step 1: Application configuration #### Enable the 'webhook' application feature * In your 'config' file, set `webhook_enabled = "true"` to activate the webhook feature. #### Configure the public notification URL * In your 'config' file, set `webhook_public_url = "https:///webhooks/onedrive"` as the public URL that will receive subscription updates from the Microsoft Graph API platform. > [!NOTE] > This URL will utilise your FQDN and must be resolvable from the Internet. This FQDN will also be used within your 'nginx' configuration. #### Testing At this point, if you attempt to test 'webhooks', when they are attempted to be initialised, the following error *should* be generated: ``` ERROR: Microsoft OneDrive API returned an error with the following message: Error Message: HTTP request returned status code 400 (Bad Request) Error Reason: Subscription validation request timed out. Error Code: ValidationError Error Timestamp: YYYY-MM-DDThh:mm:ss API Request ID: eb196382-51d7-4411-984a-45a3fda90463 Will retry creating or renewing subscription in 1 minute ``` This error is 100% normal at this point. ### Step 2: Install and configure 'nginx' > [!NOTE] > Nginx is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. #### Install and enable 'nginx' * Install 'nginx' and any other requirements to install 'nginx' on your platform. It is beyond the scope of this document to advise on how to install this. Enable and start the 'nginx' service. > [!TIP] > You may need to enable firewall rules to allow inbound http and https connections on your system: > ``` > sudo firewall-cmd --permanent --add-service=http > sudo firewall-cmd --permanent --add-service=https > sudo firewall-cmd --reload > ``` #### Verify your 'nginx' installation * From your local machine, attempt to access the local server now running, by using a web browser and pointing at http://127.0.0.1/ ![nginx_verify_install](./images/nginx_verify_install.png) #### Configure 'nginx' to receive the subscription update * Create a basic 'nginx' configuration file to support proxying traffic from Nginx to the local 'onedrive' process, which will, by default, have an HTTP listener running on TCP port 8888 ``` server { listen 80; server_name ; location /webhooks/onedrive { # Proxy Options proxy_http_version 1.1; proxy_pass http://127.0.0.1:8888; } } ``` The configuration above will: * Create an endpoint listener at `https:///webhooks/onedrive` * Proxy the received traffic at this listener to the local listener TCP port > [!TIP] > Save this file in the nginx configuration directory similar to the following path: `/etc/nginx/conf.d/onedrive_webhook.conf`. This will help keep all your configurations organised. * Test your 'nginx' configuration using `sudo nginx -t` to validate that there are no errors. If any are identified, please correct them. * Once tested, reload your 'nginx' configuration to activate the webhook reverse proxy configuration. ### Step 4: Initial Firewall/Router Configuration * Configure your firewall or router to forward all incoming HTTP and HTTPS traffic to the internal address of your system where 'nginx' is running. This is required for to allow the Let's Encrypt `certbot` tool to create a valid HTTPS certificate for your system. ![initial_firewall_config](./images/initial_firewall_config.png) * A valid configuration will be similar to the above illustration. ### Step 5: Use Let's Encrypt 'certbot' to create a SSL Certificate and deploy to your 'nginx' webhook configuration * Install the Let's Encrypt 'certbot' tool along with the associated python module 'python-certbot-nginx' for your platform * Run the 'certbot' tool on your platform to generate a valid HTTPS certificate for your `` by running `certbot --nginx`. This should *detect* your active `server_name` from your 'nginx' configuration and install the certificate in the correct manner. * The resulting 'nginx' configuration will look something like this: ``` server { server_name ; location /webhooks/onedrive { # Proxy Options proxy_http_version 1.1; proxy_pass http://127.0.0.1:8888; } listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live//fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live//privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = ) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name ; return 404; # managed by Certbot } ``` * Test your 'nginx' configuration using `sudo nginx -t` to validate that there are no errors. If any are identified, please correct them. * Once tested, reload your 'nginx' configuration to activate the webhook reverse proxy configuration. > [!IMPORTANT] > It is strongly advised that post doing this step, you implement a method to automatically keep your SSL certificate in a healthy state, as if the SSL certificate expires, webhook functionality will stop working. It is also beyond the scope of this document on how to do this. ### Step 6: Update 'nginx' to only use TLS 1.2 and TLS 1.3 To ensure that you are configuring your 'nginx' configuration to use secure communication, it is advisable for you to add the following to your `onedrive_webhook.conf` within the `server {}` configuration section: ``` # Ensure only TLS 1.2 and TLS 1.3 are used ssl_protocols TLSv1.2 TLSv1.3; ``` The resulting 'nginx' configuration will look something like this: ``` server { server_name ; location /webhooks/onedrive { # Proxy Options proxy_http_version 1.1; proxy_pass http://127.0.0.1:8888; } listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live//fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live//privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot # Ensure only TLS 1.2 and TLS 1.3 are used ssl_protocols TLSv1.2 TLSv1.3; } server { if ($host = ) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name ; return 404; # managed by Certbot } ``` * Test your 'nginx' configuration using `sudo nginx -t` to validate that there are no errors. If any are identified, please correct them. * Once tested, reload your 'nginx' configuration to activate the webhook reverse proxy configuration. To validate that the TLS configuration is working, perform the following tests from a different system that is able to resolve your FQDN externally: ``` curl -I -v --tlsv1.2 --tls-max 1.2 https:// curl -I -v --tlsv1.3 --tls-max 1.3 https:// ``` This should return valid TLS information similar to the following: ``` * Rebuilt URL to: https://your.fully.qualified.domain.name/ * Trying 123.123.123.123... * TCP_NODELAY set * Connected to your.fully.qualified.domain.name (123.123.123.123) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=your.fully.qualified.domain.name * start date: Aug 28 07:18:04 2024 GMT * expire date: Nov 26 07:18:03 2024 GMT * subjectAltName: host "your.fully.qualified.domain.name" matched cert's "your.fully.qualified.domain.name" * issuer: C=US; O=Let's Encrypt; CN=E6 * SSL certificate verify ok. > HEAD / HTTP/1.1 > Host: your.fully.qualified.domain.name > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK HTTP/1.1 200 OK < Server: nginx/1.26.2 Server: nginx/1.26.2 < Date: Sat, 31 Aug 2024 22:36:01 GMT Date: Sat, 31 Aug 2024 22:36:01 GMT < Content-Type: text/html Content-Type: text/html < Content-Length: 8474 Content-Length: 8474 < Last-Modified: Mon, 20 Feb 2023 17:42:39 GMT Last-Modified: Mon, 20 Feb 2023 17:42:39 GMT < Connection: keep-alive Connection: keep-alive < ETag: "63f3b10f-211a" ETag: "63f3b10f-211a" < Accept-Ranges: bytes Accept-Ranges: bytes ``` Lastly, to validate that TLS 1.1 and below is being blocked, perform the following tests from a different system that is able to resolve your FQDN externally: ``` curl -I -v --tlsv1.1 --tls-max 1.1 https:// ``` The response should be similar to the following: ``` * Rebuilt URL to: https://your.fully.qualified.domain.name/ * Trying 123.123.123.123... * TCP_NODELAY set * Connected to your.fully.qualified.domain.name (123.123.123.123) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS alert, internal error (592): * error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available curl: (35) error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available ``` > [!IMPORTANT] > TLS 1.2 and TLS 1.3 support is provided by OpenSSL. > > To correctly support only using these TLS versions, you must be using 'nginx' version 1.15.0 or later combined with OpenSSL 1.1.1 or later. > > If your distribution does not provide these, then please raise this with your distribution or upgrade your distribution to one that does. > [!NOTE] > If you use a version of 'nginx' that supports TLS 1.3 but are using an older version of OpenSSL (e.g., OpenSSL 1.0.x), TLS 1.3 will not be supported even if your 'nginx' configuration requests it. > [!NOTE] > If using 'LetsEncrypt', TLS 1.2 and TLS 1.3 support will be automatically configured in the `/etc/letsencrypt/options-ssl-nginx.conf` include file when the SSL Certificate is added to your 'nginx' configuration. ### Step 7: Secure your 'nginx' configuration to only allow Microsoft 365 to connect Enhance your 'nginx' configuration to only allow the Microsoft 365 platform which includes the Microsoft Graph API to communicate with your configured webhooks endpoint. Review https://www.microsoft.com/en-us/download/details.aspx?id=56519 to assist you. Please note, it is beyond the scope of this document to tell you how to secure your system against unauthorised access of your endpoint listener. > [!IMPORTANT] > The IP address ranges that are part of the Microsoft 365 Common and Office Online services, which also cover Microsoft Graph API can be sourced from the above Microsoft URL. You should regularly update your configuration as Microsoft updates these ranges frequently. > It is recommended to automate these updates accordingly and is also beyond the scope of this document on how to do this. ### Step 8: Test your 'onedrive' application using this configuration * Run the 'onedrive' application using `--monitor --verbose` and the client should now create a new subscription and register itself: ``` ..... Performing initial synchronisation to ensure consistent local state ... Started webhook server Initializing subscription for updates ... Webhook: handled validation request Created new subscription a09ba1cf-3420-4d78-9117-b41373de33ff with expiration: 2024-08-28T08:42:00.637Z Attempting to contact Microsoft OneDrive Login Service Successfully reached Microsoft OneDrive Login Service Starting a sync with Microsoft OneDrive ..... ``` * Review the 'nginx' logs to validate that applicable communication is occurring: ``` 70.37.95.11 - - [28/Aug/2024:18:26:07 +1000] "POST /webhooks/onedrive?validationToken=Validation%3a+Testing+client+application+reachability+for+subscription+Request-Id%3a+25460109-0e8b-4521-8090-dd691b407ed8 HTTP/1.1" 200 128 "-" "-" "-" 137.135.11.116 - - [28/Aug/2024:18:32:02 +1000] "POST /webhooks/onedrive?validationToken=Validation%3a+Testing+client+application+reachability+for+subscription+Request-Id%3a+65e43e3c-cbab-4e74-87ec-0e8fafdef6d3 HTTP/1.1" 200 128 "-" "-" "-" ``` ## Troubleshooting In some circumstances, `SELinux` can provent 'nginx' from communicating with local system processes. When this occurs, the application will generate an error similar to the following: ``` ERROR: Microsoft OneDrive API returned an error with the following message: Error Message: HTTP request returned status code 400 (Bad Request) Error Reason: Subscription validation request failed. Notification endpoint must respond with 200 OK to validation request. Error Code: ValidationError Error Timestamp: 2024-08-28T08:22:34 API Request ID: 36684746-1458-4150-aeab-9871355a106c Calling Function: logSubscriptionError() ``` To correct this issue, use the `setsebool` tool to allow HTTPD processes (which includes 'nginx') to make network connections: ``` sudo setsebool -P httpd_can_network_connect 1 ``` After setting the boolean, restart 'nginx' to apply the SELinux configuration change. ## Resulting configuration When these steps are followed, your environment configuration will be similar to the following diagram: ![webhooks](./puml/webhooks.png) ## Additional Configuration Assistance Refer to [application-config-options.md](application-config-options.md) for further guidance on 'webhook' configuration options. ================================================ FILE: install-sh ================================================ #!/bin/sh # install - install a program, script, or datafile scriptversion=2018-03-11.20; # UTC # This originates from X11R5 (mit/util/scripts/install.sh), which was # later released in X11R6 (xc/config/util/install.sh) with the # following copyright and license. # # Copyright (C) 1994 X Consortium # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to # deal in the Software without restriction, including without limitation the # rights to use, copy, modify, merge, publish, distribute, sublicense, and/or # sell copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN # AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC- # TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # # Except as contained in this notice, the name of the X Consortium shall not # be used in advertising or otherwise to promote the sale, use or other deal- # ings in this Software without prior written authorization from the X Consor- # tium. # # # FSF changes to this file are in the public domain. # # Calling this script install-sh is preferred over install.sh, to prevent # 'make' implicit rules from creating a file called install from it # when there is no Makefile. # # This script is compatible with the BSD install script, but was written # from scratch. tab=' ' nl=' ' IFS=" $tab$nl" # Set DOITPROG to "echo" to test this script. doit=${DOITPROG-} doit_exec=${doit:-exec} # Put in absolute file names if you don't have them in your path; # or use environment vars. chgrpprog=${CHGRPPROG-chgrp} chmodprog=${CHMODPROG-chmod} chownprog=${CHOWNPROG-chown} cmpprog=${CMPPROG-cmp} cpprog=${CPPROG-cp} mkdirprog=${MKDIRPROG-mkdir} mvprog=${MVPROG-mv} rmprog=${RMPROG-rm} stripprog=${STRIPPROG-strip} posix_mkdir= # Desired mode of installed file. mode=0755 chgrpcmd= chmodcmd=$chmodprog chowncmd= mvcmd=$mvprog rmcmd="$rmprog -f" stripcmd= src= dst= dir_arg= dst_arg= copy_on_change=false is_target_a_directory=possibly usage="\ Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE or: $0 [OPTION]... SRCFILES... DIRECTORY or: $0 [OPTION]... -t DIRECTORY SRCFILES... or: $0 [OPTION]... -d DIRECTORIES... In the 1st form, copy SRCFILE to DSTFILE. In the 2nd and 3rd, copy all SRCFILES to DIRECTORY. In the 4th, create DIRECTORIES. Options: --help display this help and exit. --version display version info and exit. -c (ignored) -C install only if different (preserve the last data modification time) -d create directories instead of installing files. -g GROUP $chgrpprog installed files to GROUP. -m MODE $chmodprog installed files to MODE. -o USER $chownprog installed files to USER. -s $stripprog installed files. -t DIRECTORY install into DIRECTORY. -T report an error if DSTFILE is a directory. Environment variables override the default commands: CHGRPPROG CHMODPROG CHOWNPROG CMPPROG CPPROG MKDIRPROG MVPROG RMPROG STRIPPROG " while test $# -ne 0; do case $1 in -c) ;; -C) copy_on_change=true;; -d) dir_arg=true;; -g) chgrpcmd="$chgrpprog $2" shift;; --help) echo "$usage"; exit $?;; -m) mode=$2 case $mode in *' '* | *"$tab"* | *"$nl"* | *'*'* | *'?'* | *'['*) echo "$0: invalid mode: $mode" >&2 exit 1;; esac shift;; -o) chowncmd="$chownprog $2" shift;; -s) stripcmd=$stripprog;; -t) is_target_a_directory=always dst_arg=$2 # Protect names problematic for 'test' and other utilities. case $dst_arg in -* | [=\(\)!]) dst_arg=./$dst_arg;; esac shift;; -T) is_target_a_directory=never;; --version) echo "$0 $scriptversion"; exit $?;; --) shift break;; -*) echo "$0: invalid option: $1" >&2 exit 1;; *) break;; esac shift done # We allow the use of options -d and -T together, by making -d # take the precedence; this is for compatibility with GNU install. if test -n "$dir_arg"; then if test -n "$dst_arg"; then echo "$0: target directory not allowed when installing a directory." >&2 exit 1 fi fi if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then # When -d is used, all remaining arguments are directories to create. # When -t is used, the destination is already specified. # Otherwise, the last argument is the destination. Remove it from $@. for arg do if test -n "$dst_arg"; then # $@ is not empty: it contains at least $arg. set fnord "$@" "$dst_arg" shift # fnord fi shift # arg dst_arg=$arg # Protect names problematic for 'test' and other utilities. case $dst_arg in -* | [=\(\)!]) dst_arg=./$dst_arg;; esac done fi if test $# -eq 0; then if test -z "$dir_arg"; then echo "$0: no input file specified." >&2 exit 1 fi # It's OK to call 'install-sh -d' without argument. # This can happen when creating conditional directories. exit 0 fi if test -z "$dir_arg"; then if test $# -gt 1 || test "$is_target_a_directory" = always; then if test ! -d "$dst_arg"; then echo "$0: $dst_arg: Is not a directory." >&2 exit 1 fi fi fi if test -z "$dir_arg"; then do_exit='(exit $ret); exit $ret' trap "ret=129; $do_exit" 1 trap "ret=130; $do_exit" 2 trap "ret=141; $do_exit" 13 trap "ret=143; $do_exit" 15 # Set umask so as not to create temps with too-generous modes. # However, 'strip' requires both read and write access to temps. case $mode in # Optimize common cases. *644) cp_umask=133;; *755) cp_umask=22;; *[0-7]) if test -z "$stripcmd"; then u_plus_rw= else u_plus_rw='% 200' fi cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;; *) if test -z "$stripcmd"; then u_plus_rw= else u_plus_rw=,u+rw fi cp_umask=$mode$u_plus_rw;; esac fi for src do # Protect names problematic for 'test' and other utilities. case $src in -* | [=\(\)!]) src=./$src;; esac if test -n "$dir_arg"; then dst=$src dstdir=$dst test -d "$dstdir" dstdir_status=$? else # Waiting for this to be detected by the "$cpprog $src $dsttmp" command # might cause directories to be created, which would be especially bad # if $src (and thus $dsttmp) contains '*'. if test ! -f "$src" && test ! -d "$src"; then echo "$0: $src does not exist." >&2 exit 1 fi if test -z "$dst_arg"; then echo "$0: no destination specified." >&2 exit 1 fi dst=$dst_arg # If destination is a directory, append the input filename. if test -d "$dst"; then if test "$is_target_a_directory" = never; then echo "$0: $dst_arg: Is a directory" >&2 exit 1 fi dstdir=$dst dstbase=`basename "$src"` case $dst in */) dst=$dst$dstbase;; *) dst=$dst/$dstbase;; esac dstdir_status=0 else dstdir=`dirname "$dst"` test -d "$dstdir" dstdir_status=$? fi fi case $dstdir in */) dstdirslash=$dstdir;; *) dstdirslash=$dstdir/;; esac obsolete_mkdir_used=false if test $dstdir_status != 0; then case $posix_mkdir in '') # Create intermediate dirs using mode 755 as modified by the umask. # This is like FreeBSD 'install' as of 1997-10-28. umask=`umask` case $stripcmd.$umask in # Optimize common cases. *[2367][2367]) mkdir_umask=$umask;; .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;; *[0-7]) mkdir_umask=`expr $umask + 22 \ - $umask % 100 % 40 + $umask % 20 \ - $umask % 10 % 4 + $umask % 2 `;; *) mkdir_umask=$umask,go-w;; esac # With -d, create the new directory with the user-specified mode. # Otherwise, rely on $mkdir_umask. if test -n "$dir_arg"; then mkdir_mode=-m$mode else mkdir_mode= fi posix_mkdir=false case $umask in *[123567][0-7][0-7]) # POSIX mkdir -p sets u+wx bits regardless of umask, which # is incompatible with FreeBSD 'install' when (umask & 300) != 0. ;; *) # Note that $RANDOM variable is not portable (e.g. dash); Use it # here however when possible just to lower collision chance. tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$ trap 'ret=$?; rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" 2>/dev/null; exit $ret' 0 # Because "mkdir -p" follows existing symlinks and we likely work # directly in world-writeable /tmp, make sure that the '$tmpdir' # directory is successfully created first before we actually test # 'mkdir -p' feature. if (umask $mkdir_umask && $mkdirprog $mkdir_mode "$tmpdir" && exec $mkdirprog $mkdir_mode -p -- "$tmpdir/a/b") >/dev/null 2>&1 then if test -z "$dir_arg" || { # Check for POSIX incompatibilities with -m. # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or # other-writable bit of parent directory when it shouldn't. # FreeBSD 6.1 mkdir -m -p sets mode of existing directory. test_tmpdir="$tmpdir/a" ls_ld_tmpdir=`ls -ld "$test_tmpdir"` case $ls_ld_tmpdir in d????-?r-*) different_mode=700;; d????-?--*) different_mode=755;; *) false;; esac && $mkdirprog -m$different_mode -p -- "$test_tmpdir" && { ls_ld_tmpdir_1=`ls -ld "$test_tmpdir"` test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1" } } then posix_mkdir=: fi rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" else # Remove any dirs left behind by ancient mkdir implementations. rmdir ./$mkdir_mode ./-p ./-- "$tmpdir" 2>/dev/null fi trap '' 0;; esac;; esac if $posix_mkdir && ( umask $mkdir_umask && $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir" ) then : else # The umask is ridiculous, or mkdir does not conform to POSIX, # or it failed possibly due to a race condition. Create the # directory the slow way, step by step, checking for races as we go. case $dstdir in /*) prefix='/';; [-=\(\)!]*) prefix='./';; *) prefix='';; esac oIFS=$IFS IFS=/ set -f set fnord $dstdir shift set +f IFS=$oIFS prefixes= for d do test X"$d" = X && continue prefix=$prefix$d if test -d "$prefix"; then prefixes= else if $posix_mkdir; then (umask=$mkdir_umask && $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break # Don't fail if two instances are running concurrently. test -d "$prefix" || exit 1 else case $prefix in *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;; *) qprefix=$prefix;; esac prefixes="$prefixes '$qprefix'" fi fi prefix=$prefix/ done if test -n "$prefixes"; then # Don't fail if two instances are running concurrently. (umask $mkdir_umask && eval "\$doit_exec \$mkdirprog $prefixes") || test -d "$dstdir" || exit 1 obsolete_mkdir_used=true fi fi fi if test -n "$dir_arg"; then { test -z "$chowncmd" || $doit $chowncmd "$dst"; } && { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } && { test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false || test -z "$chmodcmd" || $doit $chmodcmd $mode "$dst"; } || exit 1 else # Make a couple of temp file names in the proper directory. dsttmp=${dstdirslash}_inst.$$_ rmtmp=${dstdirslash}_rm.$$_ # Trap to clean up those temp files at exit. trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0 # Copy the file name to the temp name. (umask $cp_umask && $doit_exec $cpprog "$src" "$dsttmp") && # and set any options; do chmod last to preserve setuid bits. # # If any of these fail, we abort the whole thing. If we want to # ignore errors from any of these, just make sure not to ignore # errors from the above "$doit $cpprog $src $dsttmp" command. # { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } && { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } && { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } && { test -z "$chmodcmd" || $doit $chmodcmd $mode "$dsttmp"; } && # If -C, don't bother to copy if it wouldn't change the file. if $copy_on_change && old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` && new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` && set -f && set X $old && old=:$2:$4:$5:$6 && set X $new && new=:$2:$4:$5:$6 && set +f && test "$old" = "$new" && $cmpprog "$dst" "$dsttmp" >/dev/null 2>&1 then rm -f "$dsttmp" else # Rename the file to the real destination. $doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null || # The rename failed, perhaps because mv can't rename something else # to itself, or perhaps because mv is so ancient that it does not # support -f. { # Now remove or move aside any old file at destination location. # We try this two ways since rm can't unlink itself on some # systems and the destination file might be busy for other # reasons. In this case, the final cleanup might fail but the new # file should still install successfully. { test ! -f "$dst" || $doit $rmcmd -f "$dst" 2>/dev/null || { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null && { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; } } || { echo "$0: cannot unlink or rename $dst" >&2 (exit 1); exit 1 } } && # Now rename the file to the real destination. $doit $mvcmd "$dsttmp" "$dst" } fi || exit 1 trap '' 0 fi done # Local variables: # eval: (add-hook 'before-save-hook 'time-stamp) # time-stamp-start: "scriptversion=" # time-stamp-format: "%:y-%02m-%02d.%02H" # time-stamp-time-zone: "UTC0" # time-stamp-end: "; # UTC" # End: ================================================ FILE: onedrive.1.in ================================================ .TH ONEDRIVE "1" "@PACKAGE_DATE@" "@PACKAGE_VERSION@" "User Commands" .SH NAME onedrive \- A client for the Microsoft OneDrive Cloud Service .SH SYNOPSIS .B onedrive [\fI\,OPTION\/\fR] --sync .br .B onedrive [\fI\,OPTION\/\fR] --monitor .br .B onedrive [\fI\,OPTION\/\fR] --display-config .br .B onedrive [\fI\,OPTION\/\fR] --display-sync-status .br .B onedrive [\fI\,OPTION\/\fR] -h | --help .br .B onedrive --version .SH DESCRIPTION A fully featured, free, and actively maintained Microsoft OneDrive client that seamlessly supports OneDrive Personal, OneDrive for Business, Microsoft 365 (formerly Office 365), and SharePoint document libraries. .PP Designed for maximum flexibility and reliability, this powerful and highly configurable client works across all major Linux distributions and FreeBSD. It can also be deployed in containerised environments using Docker or Podman. Supporting both one-way and two-way synchronisation modes, the client provides secure and efficient file syncing with Microsoft OneDrive services — tailored to suit both desktop and server environments. .SH FEATURES .br * Compatible with OneDrive Personal, OneDrive for Business, and Microsoft SharePoint Libraries .br * Provides full support for shared folders and files across both Personal and Business accounts .br * Supports single-tenant and multi-tenant Microsoft Entra ID environments .br * Supports national cloud deployments including Microsoft Cloud for US Government, Microsoft Cloud Germany, and Azure/Office 365 operated by VNET in China .br * Supports bi-directional synchronisation (default) to keep local and remote data fully aligned .br * Supports upload-only mode to upload local changes without downloading remote changes .br * Supports download-only mode to download remote changes without uploading local changes .br * Supports a dry-run mode for safely testing configuration changes without modifying data .br * Implements safe conflict handling to minimise data loss by creating local backups when this is determined to be the safest resolution strategy .br * Provides comprehensive rules-based client-side filtering with inclusions, exclusions, wildcard matching (*), and recursive globbing (**) .br * Allows selective synchronisation of specific files, directories, or patterns .br * Caches synchronisation state for efficient processing and improved performance on large or complex sync sets .br * Supports near real-time processing of cloud-side changes using native WebSocket support .br * Supports webhook-based online change notifications where WebSockets are unsuitable (manual configuration required) .br * Monitors local file system changes in real-time using inotify .br * Implements the FreeDesktop.org Trash specification, enabling recovery of files deleted locally due to remote deletions .br * Protects against accidental data loss following configuration changes .br * Supports interruption-tolerant uploads and downloads with automatic transfer resumption .br * Validates file transfers to ensure data integrity .br * Enhances synchronisation performance through multi-threaded file transfers .br * Manages network usage through configurable bandwidth rate limiting .br * Supports desktop notifications for synchronisation events, warnings, and errors using libnotify .br * Provides desktop file-manager integration by registering the OneDrive folder as a sidebar location with a distinctive icon .br * Operates fully in both graphical and headless/server environments, with a graphical environment required only for Intune SSO, desktop notifications, and sidebar integration .SH CONFIGURATION By default, the OneDrive Client for Linux uses a sensible set of built-in defaults to interact with the Microsoft OneDrive service. .PP The client determines its configuration from three layers, applied in the following order of priority: .PP 1. Application default values – internal defaults compiled into the client. .br 2. Configuration file values – user-defined settings loaded from a configuration file (if present). .br 3. Command-line arguments – values specified at runtime override both the configuration file and application defaults. .br .PP The built-in application defaults are sufficient for most users and provide a reliable operational baseline. Creating a configuration file or using command-line options is optional, and only required when you wish to customise runtime behaviour. .TP .B NOTE: The OneDrive Client does not create a configuration file automatically. If no configuration file is found, the client runs entirely using its internally defined default values. You only need to create a configuration file if you wish to override those defaults. .PP If you want to adjust the default settings, download a copy of the default configuration template into your local configuration directory. Valid configuration file locations are: .br .PP \fB~/.config/onedrive\fP – for per-user configuration. .br \fB/etc/onedrive\fP – for system-wide configuration. .TP .B Example: To download a copy of the default configuration template, run: .PP .nf \fB mkdir -p ~/.config/onedrive wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/onedrive/config \fP .fi .PP For a full list of configuration options and command-line switches, refer to the online documentation: .br \fIhttps://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md\fP .SH CLIENT SIDE FILTERING Client Side Filtering in the context of the OneDrive Client for Linux refers to user-configured rules that determine what files and directories the client should upload or download from Microsoft OneDrive. These rules are crucial for optimising synchronisation, especially when dealing with large numbers of files or specific file types. The OneDrive Client for Linux offers several configuration options to facilitate this: .TP .B --skip-dir or 'skip_dir' config file option Specifies directories that should not be synchronised with OneDrive. Useful for omitting large or irrelevant directories from the sync process. .TP .B --skip-dot-files or 'skip_dotfiles' config file option Excludes dotfiles, usually configuration files or scripts, from the sync. Ideal for users who prefer to keep these files local. .TP .B --skip-file or 'skip_file' config file option Allows specifying specific files to exclude from synchronisation. Offers flexibility in selecting essential files for cloud storage. .TP .B --skip-symlinks or 'skip_symlinks' config file option Prevents symlinks, which often point to files outside the OneDrive directory or to irrelevant locations, from being included in the sync. .PP Additionally, the OneDrive Client for Linux allows the implementation of Client Side Filtering rules through a 'sync_list' file. This file explicitly states which directories or files should be included in the synchronisation. By default, any item not listed in the 'sync_list' file is excluded. This approach offers granular control over synchronisation, ensuring that only necessary data is transferred to and from Microsoft OneDrive. .PP These configurable options and the 'sync_list' file provide users with the flexibility to tailor the synchronisation process to their specific needs, conserving bandwidth and storage space while ensuring that important files are always backed up and accessible. .TP .B NOTE: After changing any Client Side Filtering rule, a full re-synchronisation must be performed using --resync .SH FIRST RUN Once you've installed the application, you'll need to authorise it using your Microsoft OneDrive Account. This can be done by simply running the application without any additional command switches. .TP Please be aware that some companies may require you to explicitly add this app to the Microsoft MyApps portal. To add an approved app to your apps, click on the ellipsis in the top-right corner and select "Request new apps." On the next page, you can add this app. If it's not listed, you should make a request through your IT department. .TP When you run the application for the first time, you'll be prompted to open a specific URL using your web browser, where you'll need to log in to your Microsoft Account and grant the application permission to access your files. After granting permission to the application, you'll be redirected to a blank page. Simply copy the URI from the blank page and paste it into the application. .TP This process authenticates your application with your account information, and it is now ready to use to sync your data between your local system and Microsoft OneDrive. .SH GUI NOTIFICATIONS If the client has been compiled with support for notifications, the client will send notifications about client activity via libnotify to the GUI via DBus when the client is being run in --monitor mode. .SH APPLICATION LOGGING When running onedrive all actions can be logged to a separate log file. This can be enabled by using the \fB--enable-logging\fP flag. By default, log files will be written to \fB/var/log/onedrive\fP. All logfiles will be in the format of \fB%username%.onedrive.log\fP, where \fB%username%\fP represents the user who ran the client. .SH ALL CLI OPTIONS The options below allow you to control the behaviour of the onedrive client from the CLI. Without any specific option, if the client is already authenticated, the client will exit without any further action. .TP \fB\-\-sync\fR, -s Do a one-time synchronisation with Microsoft OneDrive. .TP \fB\-\-monitor\fR, -m Monitor filesystem and synchronise regularly with Microsoft OneDrive. .TP \fB\-\-display-config\fR Display the currently used configuration for the onedrive client. .TP \fB\-\-display-sync-status\fR Query OneDrive service and report on pending changes. .TP \fB\-\-auth-files\fR \fIARG\fR Perform authentication not via interactive dialogue but via files that are read/written when using this option. The two files are passed in as \fBARG\fP in the format \fBauthUrl:responseUrl\fP. The authorisation URL is written to the \fBauthUrl\fP file, then \fBonedrive\fP waits for the file \fBresponseUrl\fP to be present, and reads the response from that file. .br Always specify the full path when using this option, otherwise the application will default to using the default configuration path for these files (~/.config/onedrive/) .TP \fB\-\-auth-response\fR \fIARG\fR Perform authentication not via interactive dialogue but via providing the response URL directly. .TP \fB\-\-check-for-nomount\fR Check for the presence of .nosync in the syncdir root. If found, do not perform sync. .TP \fB\-\-check-for-nosync\fR Check for the presence of .nosync in each directory. If found, skip directory from sync. .TP \fB\-\-classify-as-big-delete\fR \fIARG\fR Number of children in a path that is locally removed which will be classified as a 'big data delete'. .TP \fB\-\-cleanup-local-files\fR Clean up additional local files when using --download-only. This will remove local data. .TP \fB\-\-confdir\fR \fIARG\fR Set the directory used to store the configuration files. .TP \fB\-\-create-directory\fR \fIARG\fR Create a directory on OneDrive. No synchronisation will be performed. .TP \fB\-\-create-share-link\fR \fIARG\fR Create a shareable link for an existing file on OneDrive. .br Use --with-editing-perms to create a read-write share link instead of read-only. .br Use --share-password to protect the shared link with a password. .TP \fB\-\-debug-https\fR Debug OneDrive HTTPS communication. .TP \fB\-\-destination-directory\fR \fIARG\fR Destination directory for renamed or moved items on OneDrive. No synchronisation will be performed. .TP \fB\-\-disable-download-validation\fR Disable download validation when downloading from OneDrive. .TP \fB\-\-disable-notifications\fR Do not use desktop notifications in monitor mode. .TP \fB\-\-disable-upload-validation\fR Disable upload validation when uploading to OneDrive. .TP \fB\-\-display-quota\fR Display the quota status of the client. No synchronisation will be performed. .TP \fB\-\-download-file\fR \fIARG\fR Download a single file from Microsoft OneDrive. .br Specify the full online path to the file. No synchronisation will be performed. .TP \fB\-\-display-running-config\fR Display what options the client has been configured to use on application startup. .TP \fB\-\-download-only\fR Replicate the OneDrive online state locally, by only downloading changes from OneDrive. Do not upload local changes to OneDrive. .TP \fB\-\-dry-run\fR Perform a trial sync with no changes made. .TP \fB\-\-enable-logging\fR Enable client activity to a separate log file. .TP \fB\-\-file-fragment-size\fR \fIARG\fR Specify the file fragment size for large file uploads (in MB). .TP \fB\-\-force\fR Force the deletion of data when a 'big delete' is detected. .TP \fB\-\-force-http-11\fR Force the use of HTTP 1.1 for all operations. .TP \fB\-\-force-sync\fR Force a synchronisation of a specific folder, only when using --sync --single-directory and ignore all non-default skip_dir and skip_file rules. .TP \fB\-\-get-O365-drive-id\fR \fIARG\fR Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library (DEPRECATED). .TP \fB\-\-get-file-link\fR \fIARG\fR Display the file link of a synced file. .TP \fB\-\-get-sharepoint-drive-id\fR \fIARG\fR Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library. .TP \fB\-\-help\fR, \fB\-h\fR Display application help. .TP \fB\-\-list-shared-items\fR List OneDrive Business Shared Items. .TP \fB\-\-local-first\fR Synchronise from the local directory source first, before downloading changes from OneDrive. .TP \fB\-\-log-dir\fR \fIARG\fR Directory where logging output is saved to, needs to end with a slash. .TP \fB\-\-logout\fR Log out the current user. .TP \fB\-\-modified-by\fR \fIARG\fR Display the last modified by details of a given path. .TP \fB\-\-monitor-fullscan-frequency\fR \fIARG\fR Number of sync runs before performing a full local scan of the synced directory. .TP \fB\-\-monitor-interval\fR \fIARG\fR Number of seconds by which each sync operation is undertaken when idle under monitor mode. .TP \fB\-\-monitor-log-frequency\fR \fIARG\fR Frequency of logging in monitor mode. .TP \fB\-\-no-remote-delete\fR Do not delete local file 'deletes' from OneDrive when using --upload-only. .TP \fB\-\-print-access-token\fR Print the access token, useful for debugging. .TP \fB\-\-reauth\fR Reauthenticate the client with OneDrive. .TP \fB\-\-remove-directory\fR \fIARG\fR Remove a directory on OneDrive. No synchronisation will be performed. .TP \fB\-\-remove-source-files\fR Remove source file after successful transfer to OneDrive when using --upload-only. .TP \fB\-\-remove-source-folders\fR Remove the local directory structure post successful file transfer to Microsoft OneDrive when using --upload-only --remove-source-files. .TP \fB\-\-resync\fR Forget the last saved state, perform a full sync. .TP \fB\-\-resync-auth\fR Approve the use of performing a --resync action. .TP \fB\-\-share-password\fR \fIARG\fR Require a password to access the shared link when used with --create-share-link . Only supported for OneDrive Business and SharePoint environments that permit password-protected sharing. .TP \fB\-\-single-directory\fR \fIARG\fR Specify a single local directory within the OneDrive root to sync. .TP \fB\-\-skip-dir\fR \fIARG\fR Skip any directories that match this pattern from syncing. .TP \fB\-\-skip-dir-strict-match\fR When matching skip_dir directories, only match explicit matches. .TP \fB\-\-skip-dot-files\fR Skip dot files and folders from syncing. .TP \fB\-\-skip-file\fR \fIARG\fR Skip any files that match this pattern from syncing. .TP \fB\-\-skip-size\fR \fIARG\fR Skip new files larger than this size (in MB). .TP \fB\-\-skip-symlinks\fR Skip syncing of symlinks. .TP \fB\-\-source-directory\fR \fIARG\fR Source directory to rename or move on OneDrive. No synchronisation will be performed. .TP \fB\-\-space-reservation\fR \fIARG\fR The amount of disk space to reserve (in MB) to avoid 100% disk space utilisation. .TP \fB\-\-sync-root-files\fR Sync all files in sync_dir root when using sync_list. .TP \fB\-\-sync-shared-files\fR Sync OneDrive Business Shared Files to the local filesystem. .TP \fB\-\-syncdir\fR \fIARG\fR Specify the local directory used for synchronisation to OneDrive. .TP \fB\-\-synchronize\fR Perform a synchronisation with Microsoft OneDrive (DEPRECATED). .TP \fB\-\-threads\fR \fIARG\fR Specify a value for the number of worker threads used for parallel upload and download operations. .TP \fB\-\-upload-only\fR Replicate the locally configured sync_dir state to OneDrive, by only uploading local changes to OneDrive. Do not download changes from OneDrive. .TP \fB\-\-verbose\fR, \fB\-v+\fR Print more details, useful for debugging (repeat for extra debugging). .TP \fB\-\-version\fR Print the version and exit. .TP \fB\-\-with-editing-perms\fR Create a read-write shareable link for an existing file on OneDrive when used with --create-share-link . .SH DOCUMENTATION All documentation is available on GitHub: https://github.com/abraunegg/onedrive/tree/master/docs/ .SH SEE ALSO .BR curl(1), ================================================ FILE: readme.md ================================================ # OneDrive Client for Linux [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) [![Release Date](https://img.shields.io/github/release-date/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) [![Test Build](https://github.com/abraunegg/onedrive/actions/workflows/testbuild.yaml/badge.svg)](https://github.com/abraunegg/onedrive/actions/workflows/testbuild.yaml) [![Build Docker Images](https://github.com/abraunegg/onedrive/actions/workflows/docker.yaml/badge.svg)](https://github.com/abraunegg/onedrive/actions/workflows/docker.yaml) [![Docker Pulls](https://img.shields.io/docker/pulls/driveone/onedrive)](https://hub.docker.com/r/driveone/onedrive) A fully featured, free, and actively maintained Microsoft OneDrive client that seamlessly supports OneDrive Personal, OneDrive for Business, Microsoft 365 (formerly Office 365), and SharePoint document libraries. Designed for maximum flexibility and reliability, this powerful and highly configurable client works across all major Linux distributions and FreeBSD. It can also be deployed in containerised environments using Docker or Podman. Supporting both one-way and two-way synchronisation modes, the client provides secure and efficient file syncing with Microsoft OneDrive services — tailored to suit both desktop and server environments. ## Project Background This project originated as a fork of the skilion client in early 2018, after a number of proposed improvements and bug fixes — including [Pull Requests #82 and #314](https://github.com/skilion/onedrive/pulls?q=author%3Aabraunegg) — were not merged and development activity of the skilion client had largely stalled. While it’s unclear whether the original developer was unavailable or had stepped away from the project - bug reports and feature requests remained unanswered for extended periods. In 2020, the original developer (skilion) confirmed they had no intention of maintaining or supporting their work ([reference](https://github.com/skilion/onedrive/issues/518#issuecomment-717604726)). The original [skilion repository](https://github.com/skilion/onedrive) was formally archived and made read-only on GitHub in December 2024. While still publicly accessible as a historical reference, an archived repository is no longer maintained, cannot accept contributions, and reflects a frozen snapshot of the codebase. The last code change to the skilion client was merged in November 2021; however, active development had slowed significantly well before then. As such, the skilion client should no longer be considered current or supported — particularly given the major API changes and evolving Microsoft OneDrive platform requirements since that time. Under the terms of the GNU General Public License (GPL), forking and continuing development of open source software is fully permitted — provided that derivative works retain the same license. This client complies with the original GPLv3 licensing, ensuring the same freedoms granted by the original project remain intact. Since forking in early 2018, this client has evolved into a clean re-imagining of the original codebase, resolving long-standing bugs and adding extensive new functionality to better support both personal and enterprise use cases to interact with Microsoft OneDrive from Linux and FreeBSD platforms. ## Features ### Broad Microsoft OneDrive Compatibility * Works with OneDrive Personal, OneDrive for Business, and Microsoft SharePoint Libraries. * Full support for shared folders and files across both Personal and Business accounts. * Supports single-tenant and multi-tenant Microsoft Entra ID environments. * Compatible with national cloud deployments: * Microsoft Cloud for US Government * Microsoft Cloud Germany * Azure/Office 365 operated by VNET in China ### Flexible Synchronisation Modes * Bi-directional sync (default) - keeps local and remote data fully aligned. * Upload-only mode - only uploads local changes; does not download remote changes. * Download-only mode - only downloads remote changes; does not upload local changes. * Dry-run mode - test configuration changes safely without modifying files. * Safe conflict handling minimises data loss by creating local backups whenever this is determined to be the safest conflict-resolution strategy. ### Client-Side Filtering & Granular Sync Control * Comprehensive rules-based client-side filtering (inclusions, exclusions, wildcard `*`, globbing `**`). * Filter specific files, folders, or patterns to tailor precisely what is synced with Microsoft OneDrive. * Efficient cached sync state for fast decision-making during large or complex sync sets. ### Real-Time Monitoring & Online Change Detection * Near real-time processing of cloud-side changes using native WebSocket support. * Webhook support for environments where WebSockets are unsuitable (manual setup). * Real-time local change monitoring via inotify. ### Data Safety, Recovery & Integrity Protection * Implements the FreeDesktop.org Trash specification, enabling recovery of items deleted locally due to online deletion. * Strong safeguards to prevent accidental remote deletion or overwrite after configuration changes. * Interruption-tolerant uploads and downloads, automatically resuming transfers. * Integrity validation for every file transferred. ### Modern Authentication Support * Standard OAuth2 Native Client Authorisation Flow (default), supporting browser-based login, multi-factor authentication (MFA), and modern Microsoft account security requirements. * OAuth2 Device Authorisation Flow for Microsoft Entra ID accounts, ideal for headless systems, servers, and terminal-only environments. * Intune Single Sign-On (SSO) using the Microsoft Identity Device Broker (IDB) via D-Bus, enabling seamless enterprise authentication without manual credential entry. ### Performance, Efficiency & Resource Management * Multi-threaded file transfers for significantly improved sync speeds. * Bandwidth rate limiting to control network consumption. * Highly efficient processing with state caching, reducing API traffic and improving performance. ### Desktop Integration & User Experience * libnotify desktop notifications for sync events, warnings, and errors. * Registers the OneDrive folder as a sidebar location in supported file managers, complete with a distinctive icon. * Works seamlessly in GUI and headless/server environments. A GUI is only required for Intune SSO, notifications, and sidebar integration; all other features function without graphical support. ## What's missing * Ability to encrypt/decrypt files on-the-fly when uploading/downloading files from OneDrive * Support for Windows 'On-Demand' functionality so file is only downloaded when accessed locally ## External Enhancements * A GUI for configuration management: [OneDrive Client for Linux GUI](https://github.com/bpozdena/OneDriveGUI) * Colorful log output terminal modification: [OneDrive Client for Linux Colorful log Output](https://github.com/zzzdeb/dotfiles/blob/master/scripts/tools/onedrive_log) * System Tray Icon: [OneDrive Client for Linux System Tray Icon](https://github.com/DanielBorgesOliveira/onedrive_tray) ## Frequently Asked Questions Refer to [Frequently Asked Questions](https://github.com/abraunegg/onedrive/wiki/Frequently-Asked-Questions) ## Have a question If you have a question or need something clarified, please raise a new discussion post [here](https://github.com/abraunegg/onedrive/discussions) ## Supported Application Version Support is only provided for the current application release version or newer 'master' branch versions. The current release version is: [![Version](https://img.shields.io/github/v/release/abraunegg/onedrive)](https://github.com/abraunegg/onedrive/releases) To check your version, run: `onedrive --version`. Ensure you are using the current release or compile the latest version from the master branch if needed. If you are using an older version, you must upgrade to the current release or newer to receive support. ## Documentation and Configuration Assistance OneDrive Client for Linux includes a rich set of documentation covering installation, configuration options, advanced usage, and integrations. These resources are designed to help new users get started quickly and to give experienced users full control over advanced behaviour. If you are changing configuration, running in production, or using Business/SharePoint features, you should be reading these documents. All documentation is maintained in the [`docs/`](https://github.com/abraunegg/onedrive/tree/master/docs) directory of this repository. ### Getting Started #### Installation Learn how to install the client on various systems — from distribution packages to building from source. Please read the [Install Guide](https://github.com/abraunegg/onedrive/blob/master/docs/install.md) #### Basic Usage & Configuration Covers initial authentication, default settings, basic operational instructions, frequently asked 'how to' questions, and how to tailor the application configuration. Please read the [Usage Guide](https://github.com/abraunegg/onedrive/blob/master/docs/usage.md) ### Advanced Configuration #### Application Configuration Options Full reference for every config option (with descriptions, defaults, and examples) to customise sync behaviour precisely. Please read the [Application Configuration Options Guide](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md) #### Advanced Usage Tips for creating multiple config profiles, custom sync rules, daemon setups, selective sync, dual-booting with Microsoft Windows and more. Please read the [Advanced Usage Guide](https://github.com/abraunegg/onedrive/blob/master/docs/advanced-usage.md) ### Special Use Cases #### Business Shared Items Configuring sync for OneDrive Business shared items (files and folders). Please read the [Business Shared Items Guide](https://github.com/abraunegg/onedrive/blob/master/docs/business-shared-items.md) #### SharePoint & Office 365 Libraries Instructions for syncing SharePoint document libraries (Business or Education tenants). Please read the [SharePoint Library Guide](https://github.com/abraunegg/onedrive/blob/master/docs/sharepoint-libraries.md) #### National Cloud support Instructions for environments like Microsoft Cloud Germany or US Government cloud endpoints. Please read the [National Cloud Deployment Guide](https://github.com/abraunegg/onedrive/blob/master/docs/national-cloud-deployments.md) ### Container Support #### Docker How to run the OneDrive client in a Docker container. Please read the [Docker Guide](https://github.com/abraunegg/onedrive/blob/master/docs/docker.md) #### Podman How to run the OneDrive client with Podman. Please read the [Podman Guide](https://github.com/abraunegg/onedrive/blob/master/docs/podman.md) ## Basic Troubleshooting Steps If you encounter any issues running the application, please follow these steps **before** raising a bug report: 1. **Check the application version** Run `onedrive --version` to confirm which version you are using. - Ensure you are running the latest [release](https://github.com/abraunegg/onedrive/releases). - If you are already on the latest release but still experiencing issues, manually build the client from the `master` branch to test against the very latest code. This includes fixes for bugs discovered since the last tagged release. - If you are using Docker or Podman, ensure you are using the 'edge' Docker Tag. Do not use the 'latest' Docker Tag. 2. **Run in verbose mode** Use the `--verbose` option to provide greater clarity and detailed logging about the issue you are facing. If you are using Docker or Podman, use the ONEDRIVE_VERBOSE environment variable to increase logging verbosity. 3. **Test with IPv4 only** Configure the application to use **IPv4 network connectivity only**, then retest. See the `'ip_protocol_version'` option [documentation](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md#ip_protocol_version) for assistance. 4. **Test with HTTP/1.1 and IPv4** Configure the application to use **HTTP/1.1 over IPv4 only**, then retest. See the `'force_http_11'` option [documentation](https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md#force_http_11) for assistance. 5. **Verify cURL and libcurl versions** If the above steps do not resolve your issue, upgrade both `curl` and `libcurl` to the latest versions provided by the curl developers. - See [Compatibility with curl](https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl) for details on curl bugs that impact this client. - Refer to the official [cURL Releases](https://curl.se/docs/releases.html) page for version information. 6. **Open a new issue** If the problem persists after completing the steps above, proceed to **Reporting an Issue or Bug** below and open a new issue with the requested details and logs. ## Reporting an Issue or Bug > [!IMPORTANT] > Please ensure the problem is a software bug. For installation issues, distribution package/version questions, or dependency problems, start a [Discussion](https://github.com/abraunegg/onedrive/discussions) instead of filing a bug report. If you encounter a bug, you can report it on GitHub. Before opening a new issue report: 1. **Complete the Basic Troubleshooting Steps** Confirm you’ve run through all steps in the section above. 2. **Search existing issues** Check both [Open](https://github.com/abraunegg/onedrive/issues) and [Closed](https://github.com/abraunegg/onedrive/issues?q=is%3Aissue%20state%3Aclosed) issues for a similar problem to avoid duplicates. 3. **Use the issue template** Open a new bug report using the [issue template](https://github.com/abraunegg/onedrive/issues/new?template=bug_report.md) and fill in **all fields**. Complete detail helps us reproduce your environment and replicate the issue. 4. **Generate a debug log** Follow this [process](https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support) to create a debug log. - If you are concerned about personal or business sensitive data in the debug log, you may: - Create a new OneDrive account, configure the client to use it, use **dummy** data to simulate your environment, and reproduce the issue; or - Provide an NDA or confidentiality agreement for signature prior to sharing sensitive logs. 5. **Share the debug log securely** - **Do not post debug logs publicly.** Debug logs can include sensitive details (file paths, filenames, API endpoints, environment info, etc.). - **Send the log via email** to **support@mynas.com.au** using a trusted email account. - **Archive and password-protect** the log before sending (e.g. `.zip` with AES or `.7z`): - Example (zip with password): `zip -e onedrive-debug.zip onedrive-debug.log` - Example (7z with password): `7z a -p onedrive-debug.7z onedrive-debug.log` - **Send the password out-of-band (OOB)** — not in the same email as the archive. Email **support@mynas.com.au** to arrange an OOB method (e.g. separate email thread, phone/SMS, or agreed channel). - **If you require an NDA**, attach your NDA or confidentiality agreement to your email. It will be reviewed and signed prior to exchanging sensitive data. ### What to include in your bug report When raising a new bug report, please include **all details requested in the issue template**, such as: - A clear description of the problem and how to reproduce it - Your operating system and installation method - OneDrive account type and client version - Application configuration and cURL version - Sync directory location, system mount points, and partition types - A full debug log, shared securely as described above Providing complete information makes it much easier to understand, reproduce, and resolve your issue quickly. > [!NOTE] > Submitting a bug report starts a collaboration. To help us help you, please: > - Stay available to answer questions or provide clarifications if needed > - Test and confirm fixes in your own environment when a pull request (PR) is created for your issue > [!TIP] > Reports with missing details are much harder to investigate. Sharing as much as you can up front gives the best chance of a fast and accurate fix. ## Known issues Lists common limitations, known problems, diagnostics, and workarounds. Please read the [Known Issues Advice](https://github.com/abraunegg/onedrive/blob/master/docs/known-issues.md) ================================================ FILE: src/arsd/README.md ================================================ The files in this directory have been obtained form the following places: cgi.d https://github.com/adamdruppe/arsd/blob/a870179988b8881b04126856105f0fad2cc0018d/cgi.d License: Boost Software License - Version 1.0 Copyright 2008-2021, Adam D. Ruppe see https://github.com/adamdruppe/arsd/blob/a870179988b8881b04126856105f0fad2cc0018d/LICENSE ================================================ FILE: src/arsd/cgi.d ================================================ // FIXME: if an exception is thrown, we shouldn't necessarily cache... // FIXME: there's some annoying duplication of code in the various versioned mains // add the Range header in there too. should return 206 // FIXME: cgi per-request arena allocator // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable // but the later one can edit and simplify the api. You'd have to use the subclass tho! /* void foo(int f, @("test") string s) {} void main() { static if(is(typeof(foo) Params == __parameters)) //pragma(msg, __traits(getAttributes, Params[0])); pragma(msg, __traits(getAttributes, Params[1..2])); else pragma(msg, "fail"); } */ // Note: spawn-fcgi can help with fastcgi on nginx // FIXME: to do: add openssl optionally // make sure embedded_httpd doesn't send two answers if one writes() then dies // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections /* Session manager process: it spawns a new process, passing a command line argument, to just be a little key/value store of some serializable struct. On Windows, it CreateProcess. On Linux, it can just fork or maybe fork/exec. The session key is in a cookie. Server-side event process: spawns an async manager. You can push stuff out to channel ids and the clients listen to it. websocket process: spawns an async handler. They can talk to each other or get info from a cgi request. Tempting to put web.d 2.0 in here. It would: * map urls and form generation to functions * have data presentation magic * do the skeleton stuff like 1.0 * auto-cache generated stuff in files (at least if pure?) * introspect functions in json for consumers https://linux.die.net/man/3/posix_spawn */ /++ Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. --- import arsd.cgi; // Instead of writing your own main(), you should write a function // that takes a Cgi param, and use mixin GenericMain // for maximum compatibility with different web servers. void hello(Cgi cgi) { cgi.setResponseContentType("text/plain"); if("name" in cgi.get) cgi.write("Hello, " ~ cgi.get["name"]); else cgi.write("Hello, world!"); } mixin GenericMain!hello; --- Or: --- import arsd.cgi; class MyApi : WebObject { @UrlName("") string hello(string name = null) { if(name is null) return "Hello, world!"; else return "Hello, " ~ name; } } mixin DispatcherMain!( "/".serveApi!MyApi ); --- $(NOTE Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` and `dub add arsd-official:dom` yourself. ) Test on console (works in any interface mode): $(CONSOLE $ ./cgi_hello GET / name=whatever ) If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): $(CONSOLE $ ./cgi_hello --port 8080 # now you can go to http://localhost:8080/?name=whatever ) Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. Build_Configurations: cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. If you are using `dub`, use: ```sdlang subConfiguration "arsd-official:cgi" "VALUE_HERE" ``` or to dub.json: ```json "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} ``` to change versions. The possible options for `VALUE_HERE` are: $(LIST * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. ) With dmd, use: $(TABLE_ROWS * + Interfaces + (mutually exclusive) * - `-version=plain_cgi` - The default building the module alone without dub - a traditional, plain CGI executable will be generated. * - `-version=embedded_httpd` - A HTTP server will be embedded in the generated executable. This is default when building with dub. * - `-version=fastcgi` - A FastCGI executable will be generated. * - `-version=scgi` - A SCGI (SimpleCGI) executable will be generated. * - `-version=embedded_httpd_hybrid` - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. * - `-version=embedded_httpd_threads` - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) * - `-version=embedded_httpd_processes` - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) * - `-version=embedded_httpd_processes_accept_after_fork` - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. * - `-version=stdio_http` - The embedded HTTP server will be spoken over stdin and stdout. * + Tweaks + (can be used together with others) * - `-version=cgi_with_websocket` - The CGI class has websocket server support. (This is on by default now.) * - `-version=with_openssl` - not currently used * - `-version=cgi_embedded_sessions` - The session server will be embedded in the cgi.d server process * - `-version=cgi_session_server_process` - The session will be provided in a separate process, provided by cgi.d. ) For example, For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. Simulating_requests: If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command line shell. Call the program like this: $(CONSOLE ./yourprogram GET / name=adr ) And it will print the result to stdout instead of running a server, regardless of build more.. CGI_Setup_tips: On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. Overview_Of_Basic_Concepts: cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] Redirections: [Cgi.setResponseLocation] Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] A basic program using the lower-level api might look like: --- import arsd.cgi; // you write a request handler which always takes a Cgi object void handler(Cgi cgi) { /+ when the user goes to your site, suppose you are being hosted at http://example.com/yourapp If the user goes to http://example.com/yourapp/test?name=value then the url will be parsed out into the following pieces: cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) cgi.scriptName == "yourapp". With an embedded http server, this will be blank. cgi.host == "example.com" cgi.https == false cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) The query string is further parsed into the `get` and `getArray` members, so: cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` And cgi.getArray == ["name": ["value"]]. Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data if you need it. But since so often you only care about one value, the `get` member provides more convenient access. We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. +/ switch(cgi.pathInfo) { // the home page will be a small html form that can set a cookie. case "/": cgi.write(`
`, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. break; // POSTing to this will set a cookie with our submitted name case "/set-cookie": // HTTP has a number of request methods (also called "verbs") to tell // what you should do with the given resource. // The most common are GET and POST, the ones used in html forms. // You can check which one was used with the `cgi.requestMethod` property. if(cgi.requestMethod == Cgi.RequestMethod.POST) { // headers like redirections need to be set before we call `write` cgi.setResponseLocation("read-cookie"); // just like how url params go into cgi.get/getArray, form data submitted in a POST // body go to cgi.post/postArray. Please note that a POST request can also have get // params in addition to post params. // // There's also a convenience function `cgi.request("name")` which checks post first, // then get if it isn't found there, and then returns a default value if it is in neither. if("name" in cgi.post) { // we can set cookies with a method too // again, cookies need to be set before calling `cgi.write`, since they // are a kind of header. cgi.setCookie("name" , cgi.post["name"]); } // the user will probably never see this, since the response location // is an automatic redirect, but it is still best to say something anyway cgi.write("Redirecting you to see the cookie...", true); } else { // you can write out response codes and headers // as well as response bodies // // But always check the cgi docs before using the generic // `header` method - if there is a specific method for your // header, use it before resorting to the generic one to avoid // a header value from being sent twice. cgi.setResponseLocation("405 Method Not Allowed"); // there is no special accept member, so you can use the generic header function cgi.header("Accept: POST"); // but content type does have a method, so prefer to use it: cgi.setResponseContentType("text/plain"); // all the headers are buffered, and will be sent upon the first body // write. you can actually modify some of them before sending if need be. cgi.write("You must use the POST http verb on this resource.", true); } break; // and GETting this will read the cookie back out case "/read-cookie": // I did NOT pass `,true` here because this is writing a partial response. // It is possible to stream data to the user in chunks by writing partial // responses the calling `cgi.flush();` to send the partial response immediately. // normally, you'd only send partial chunks if you have to - it is better to build // a response as a whole and send it as a whole whenever possible - but here I want // to demo that you can. cgi.write("Hello, "); if("name" in cgi.cookies) { import arsd.dom; // dom.d provides a lot of helpers for html // since the cookie is set, we need to write it out properly to // avoid cross-site scripting attacks. // // Getting this stuff right automatically is a benefit of using the higher // level apis, but this demo is to show the fundamental building blocks, so // we're responsible to take care of it. cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); } else { cgi.write("friend"); } // note that I never called cgi.setResponseContentType, since the default is text/html. // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write // calls. break; default: // no path matched cgi.setResponseStatus("404 Not Found"); cgi.write("Resource not found.", true); } } // and this adds the boilerplate to set up a server according to the // compile version configuration and call your handler as requests come in mixin GenericMain!handler; // the `handler` here is the name of your function --- Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) A basic program using the higher-level apis might look like: --- /+ import arsd.cgi; struct LoginData { string currentUser; } class AppClass : WebObject { string foo() {} } mixin DispatcherMain!( "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory "/".serveApi!AppClass, "/thing/".serveRestObject, ); +/ --- Guide_for_PHP_users: (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) If you are coming from old-style PHP, here's a quick guide to help you get started: $(SIDE_BY_SIDE $(COLUMN ```php ``` ) $(COLUMN --- import arsd.cgi; void app(Cgi cgi) { string foo = cgi.post["foo"]; string bar = cgi.get["bar"]; string baz = cgi.cookies["baz"]; string user_ip = cgi.remoteAddress; string host = cgi.host; string path = cgi.pathInfo; cgi.setCookie("baz", "some value"); cgi.write("hello!"); } mixin GenericMain!app --- ) ) $(H3 Array elements) In PHP, you can give a form element a name like `"something[]"`, and then `$_POST["something"]` gives an array. In D, you can use whatever name you want, and access an array of values with the `cgi.getArray["name"]` and `cgi.postArray["name"]` members. $(H3 Databases) PHP has a lot of stuff in its standard library. cgi.d doesn't include most of these, but the rest of my arsd repository has much of it. For example, to access a MySQL database, download `database.d` and `mysql.d` from my github repo, and try this code (assuming, of course, your database is set up): --- import arsd.cgi; import arsd.mysql; void app(Cgi cgi) { auto database = new MySql("localhost", "username", "password", "database_name"); foreach(row; mysql.query("SELECT count(id) FROM people")) cgi.write(row[0] ~ " people in database"); } mixin GenericMain!app; --- Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, implementing the same basic interface. See_Also: You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making web applications. dom and webtemplate are used by the higher-level api here in cgi.d. For working with json, try [arsd.jsvar]. [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in accessing databases. If you are looking to access a web application via HTTP, try [arsd.http2]. Copyright: cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. Yes, this file is old, and yes, it is still actively maintained and used. +/ module arsd.cgi; // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form /++ This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. +/ unittest { import arsd.cgi; mixin DispatcherMain!( "/".serveStaticFileDirectory(null, true) ); } /++ Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. +/ unittest { import arsd.cgi; void requestHandler(Cgi cgi) { cgi.dispatcher!( "/".serveStaticFileDirectory(null, true) ); } // mixin GenericMain!requestHandler would add this function: void main(string[] args) { // this is all the content of [cgiMainImpl] which you can also call // cgi.d embeds a few add on functions like real time event forwarders // and session servers it can run in other processes. this spawns them, if needed. if(tryAddonServers(args)) return; // cgi.d allows you to easily simulate http requests from the command line, // without actually starting a server. this function will do that. if(trySimulatedRequest!(requestHandler, Cgi)(args)) return; RequestServer server; // you can change the default port here if you like // server.listeningPort = 9000; // then call this to let the command line args override your default server.configureFromCommandLine(args); // here is where you could print out the listeningPort to the user if you wanted // and serve the request(s) according to the compile configuration server.serve!(requestHandler)(); // or you could explicitly choose a serve mode like this: // server.serveEmbeddedHttp!requestHandler(); } } /++ cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that otherwise run through the rest of the internal mechanisms to call your functions without actually spinning up a server. +/ unittest { import arsd.cgi; void requestHandler(Cgi cgi) { } // D doesn't let me embed a unittest inside an example unittest // so this is a function, but you can do it however in your real program /* unittest */ void runTests() { auto tester = new CgiTester(&requestHandler); auto response = tester.GET("/"); assert(response.code == 200); } } static import std.file; // for a single thread, linear request thing, use: // -version=embedded_httpd_threads -version=cgi_no_threads version(Posix) { version(CRuntime_Musl) { } else version(minimal) { } else { version(GNU) { // GDC doesn't support static foreach so I had to cheat on it :( } else version(FreeBSD) { // I never implemented the fancy stuff there either } else version(OpenBSD) { // Fix issue #2977 - adopt same approach as FreeBSD above } else { version=with_breaking_cgi_features; version=with_sendfd; version=with_addon_servers; } } } version(Windows) { version(minimal) { } else { // not too concerned about gdc here since the mingw version is fairly new as well version=with_breaking_cgi_features; } } void cloexec(int fd) { version(Posix) { import core.sys.posix.fcntl; fcntl(fd, F_SETFD, FD_CLOEXEC); } } void cloexec(Socket s) { version(Posix) { import core.sys.posix.fcntl; fcntl(s.handle, F_SETFD, FD_CLOEXEC); } } version(embedded_httpd_hybrid) { version=embedded_httpd_threads; version(cgi_no_fork) {} else version(Posix) version=cgi_use_fork; version=cgi_use_fiber; } version(cgi_use_fork) enum cgi_use_fork_default = true; else enum cgi_use_fork_default = false; // the servers must know about the connections to talk to them; the interfaces are vital version(with_addon_servers) version=with_addon_servers_connections; version(embedded_httpd) { version(linux) version=embedded_httpd_processes; else { version=embedded_httpd_threads; } /* version(with_openssl) { pragma(lib, "crypto"); pragma(lib, "ssl"); } */ } version(embedded_httpd_processes) version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later version(embedded_httpd_threads) { // unless the user overrides the default.. version(cgi_session_server_process) {} else version=cgi_embedded_sessions; } version(scgi) { // unless the user overrides the default.. version(cgi_session_server_process) {} else version=cgi_embedded_sessions; } // fall back if the other is not defined so we can cleanly version it below version(cgi_embedded_sessions) {} else version=cgi_session_server_process; version=cgi_with_websocket; enum long defaultMaxContentLength = 5_000_000; /* To do a file download offer in the browser: cgi.setResponseContentType("text/csv"); cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); */ // FIXME: the location header is supposed to be an absolute url I guess. // FIXME: would be cool to flush part of a dom document before complete // somehow in here and dom.d. // these are public so you can mixin GenericMain. // FIXME: use a function level import instead! public import std.string; public import std.stdio; public import std.conv; import std.concurrency; import std.uri; import std.uni; import std.algorithm.comparison; import std.algorithm.searching; import std.exception; import std.base64; static import std.algorithm; import std.datetime; import std.range; import std.process; import std.zlib; T[] consume(T)(T[] range, int count) { if(count > range.length) count = range.length; return range[count..$]; } int locationOf(T)(T[] data, string item) { const(ubyte[]) d = cast(const(ubyte[])) data; const(ubyte[]) i = cast(const(ubyte[])) item; // this is a vague sanity check to ensure we aren't getting insanely // sized input that will infinite loop below. it should never happen; // even huge file uploads ought to come in smaller individual pieces. if(d.length > (int.max/2)) throw new Exception("excessive block of input"); for(int a = 0; a < d.length; a++) { if(a + i.length > d.length) return -1; if(d[a..a+i.length] == i) return a; } return -1; } /// If you are doing a custom cgi class, mixing this in can take care of /// the required constructors for you mixin template ForwardCgiConstructors() { this(long maxContentLength = defaultMaxContentLength, string[string] env = null, const(ubyte)[] delegate() readdata = null, void delegate(const(ubyte)[]) _rawDataOutput = null, void delegate() _flush = null ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } this(string[] args) { super(args); } this( BufferedInputRange inputData, string address, ushort _port, int pathInfoStarts = 0, bool _https = false, void delegate(const(ubyte)[]) _rawDataOutput = null, void delegate() _flush = null, // this pointer tells if the connection is supposed to be closed after we handle this bool* closeConnection = null) { super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); } this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } } /// thrown when a connection is closed remotely while we waiting on data from it class ConnectionClosedException : Exception { this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { super(message, file, line, next); } } version(Windows) { // FIXME: ugly hack to solve stdin exception problems on Windows: // reading stdin results in StdioException (Bad file descriptor) // this is probably due to https://issues.dlang.org/show_bug.cgi?id=3425 private struct stdin { struct ByChunk { // Replicates std.stdio.ByChunk private: ubyte[] chunk_; public: this(size_t size) in { assert(size, "size must be larger than 0"); } do { chunk_ = new ubyte[](size); popFront(); } @property bool empty() const { return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job } @property nothrow ubyte[] front() { return chunk_; } void popFront() { enforce(!empty, "Cannot call popFront on empty range"); chunk_ = stdin.rawRead(chunk_); } } import core.sys.windows.windows; static: T[] rawRead(T)(T[] buf) { uint bytesRead; auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); if (!result) { auto err = GetLastError(); if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input return buf[0..0]; // Some other error, throw it char* buffer; scope(exit) LocalFree(buffer); // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); throw new Exception(to!string(buffer)); } enforce(!(bytesRead % T.sizeof), "I/O error"); return buf[0..bytesRead / T.sizeof]; } auto byChunk(size_t sz) { return ByChunk(sz); } void close() { std.stdio.stdin.close; } } } /// The main interface with the web request class Cgi { public: /// the methods a request can be enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work // these are defined in the standard, but idk if they are useful for anything OPTIONS, TRACE, CONNECT, // These seem new, I have only recently seen them PATCH, MERGE, // this is an extension for when the method is not specified and you want to assume CommandLine } /+ /++ Cgi provides a per-request memory pool +/ void[] allocateMemory(size_t nBytes) { } /// ditto void[] reallocateMemory(void[] old, size_t nBytes) { } /// ditto void freeMemory(void[] memory) { } +/ /* import core.runtime; auto args = Runtime.args(); we can call the app a few ways: 1) set up the environment variables and call the app (manually simulating CGI) 2) simulate a call automatically: ./app method 'uri' for example: ./app get /path?arg arg2=something Anything on the uri is treated as query string etc on get method, further args are appended to the query string (encoded automatically) on post method, further args are done as post @name means import from file "name". if name == -, it uses stdin (so info=@- means set info to the value of stdin) Other arguments include: --cookie name=value (these are all concated together) --header 'X-Something: cool' --referrer 'something' --port 80 --remote-address some.ip.address.here --https yes --user-agent 'something' --userpass 'user:pass' --authorization 'Basic base64encoded_user:pass' --accept 'content' // FIXME: better example --last-event-id 'something' --host 'something.com' Non-simulation arguments: --port xxx listening port for non-cgi things (valid for the cgi interfaces) --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. */ /** Initializes it with command line arguments (for easy testing) */ this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { rawDataOutput = _rawDataOutput; // these are all set locally so the loop works // without triggering errors in dmd 2.064 // we go ahead and set them at the end of it to the this version int port; string referrer; string remoteAddress; string userAgent; string authorization; string origin; string accept; string lastEventId; bool https; string host; RequestMethod requestMethod; string requestUri; string pathInfo; string queryString; bool lookingForMethod; bool lookingForUri; string nextArgIs; string _cookie; string _queryString; string[][string] _post; string[string] _headers; string[] breakUp(string s) { string k, v; auto idx = s.indexOf("="); if(idx == -1) { k = s; } else { k = s[0 .. idx]; v = s[idx + 1 .. $]; } return [k, v]; } lookingForMethod = true; scriptName = args[0]; scriptFileName = args[0]; environmentVariables = cast(const) environment.toAA; foreach(arg; args[1 .. $]) { if(arg.startsWith("--")) { nextArgIs = arg[2 .. $]; } else if(nextArgIs.length) { if (nextArgIs == "cookie") { auto info = breakUp(arg); if(_cookie.length) _cookie ~= "; "; _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); } else if (nextArgIs == "port") { port = to!int(arg); } else if (nextArgIs == "referrer") { referrer = arg; } else if (nextArgIs == "remote-address") { remoteAddress = arg; } else if (nextArgIs == "user-agent") { userAgent = arg; } else if (nextArgIs == "authorization") { authorization = arg; } else if (nextArgIs == "userpass") { authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; } else if (nextArgIs == "origin") { origin = arg; } else if (nextArgIs == "accept") { accept = arg; } else if (nextArgIs == "last-event-id") { lastEventId = arg; } else if (nextArgIs == "https") { if(arg == "yes") https = true; } else if (nextArgIs == "header") { string thing, other; auto idx = arg.indexOf(":"); if(idx == -1) throw new Exception("need a colon in a http header"); thing = arg[0 .. idx]; other = arg[idx + 1.. $]; _headers[thing.strip.toLower()] = other.strip; } else if (nextArgIs == "host") { host = arg; } // else // skip, we don't know it but that's ok, it might be used elsewhere so no error nextArgIs = null; } else if(lookingForMethod) { lookingForMethod = false; lookingForUri = true; if(arg.asLowerCase().equal("commandline")) requestMethod = RequestMethod.CommandLine; else requestMethod = to!RequestMethod(arg.toUpper()); } else if(lookingForUri) { lookingForUri = false; requestUri = arg; auto idx = arg.indexOf("?"); if(idx == -1) pathInfo = arg; else { pathInfo = arg[0 .. idx]; _queryString = arg[idx + 1 .. $]; } } else { // it is an argument of some sort if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { auto parts = breakUp(arg); _post[parts[0]] ~= parts[1]; allPostNamesInOrder ~= parts[0]; allPostValuesInOrder ~= parts[1]; } else { if(_queryString.length) _queryString ~= "&"; auto parts = breakUp(arg); _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); } } } acceptsGzip = false; keepAliveRequested = false; requestHeaders = cast(immutable) _headers; cookie = _cookie; cookiesArray = getCookieArray(); cookies = keepLastOf(cookiesArray); queryString = _queryString; getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); get = keepLastOf(getArray); postArray = cast(immutable) _post; post = keepLastOf(_post); // FIXME filesArray = null; files = null; isCalledWithCommandLineArguments = true; this.port = port; this.referrer = referrer; this.remoteAddress = remoteAddress; this.userAgent = userAgent; this.authorization = authorization; this.origin = origin; this.accept = accept; this.lastEventId = lastEventId; this.https = https; this.host = host; this.requestMethod = requestMethod; this.requestUri = requestUri; this.pathInfo = pathInfo; this.queryString = queryString; this.postBody = null; } private { string[] allPostNamesInOrder; string[] allPostValuesInOrder; string[] allGetNamesInOrder; string[] allGetValuesInOrder; } CgiConnectionHandle getOutputFileHandle() { return _outputFileHandle; } CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; /** Initializes it using a CGI or CGI-like interface */ this(long maxContentLength = defaultMaxContentLength, // use this to override the environment variable listing in string[string] env = null, // and this should return a chunk of data. return empty when done const(ubyte)[] delegate() readdata = null, // finally, use this to do custom output if needed void delegate(const(ubyte)[]) _rawDataOutput = null, // to flush the custom output void delegate() _flush = null ) { // these are all set locally so the loop works // without triggering errors in dmd 2.064 // we go ahead and set them at the end of it to the this version int port; string referrer; string remoteAddress; string userAgent; string authorization; string origin; string accept; string lastEventId; bool https; string host; RequestMethod requestMethod; string requestUri; string pathInfo; string queryString; isCalledWithCommandLineArguments = false; rawDataOutput = _rawDataOutput; flushDelegate = _flush; auto getenv = delegate string(string var) { if(env is null) return std.process.environment.get(var); auto e = var in env; if(e is null) return null; return *e; }; environmentVariables = env is null ? cast(const) environment.toAA : env; // fetching all the request headers string[string] requestHeadersHere; foreach(k, v; env is null ? cast(const) environment.toAA() : env) { if(k.startsWith("HTTP_")) { requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; } } this.requestHeaders = assumeUnique(requestHeadersHere); requestUri = getenv("REQUEST_URI"); cookie = getenv("HTTP_COOKIE"); cookiesArray = getCookieArray(); cookies = keepLastOf(cookiesArray); referrer = getenv("HTTP_REFERER"); userAgent = getenv("HTTP_USER_AGENT"); remoteAddress = getenv("REMOTE_ADDR"); host = getenv("HTTP_HOST"); pathInfo = getenv("PATH_INFO"); queryString = getenv("QUERY_STRING"); scriptName = getenv("SCRIPT_NAME"); { import core.runtime; auto sfn = getenv("SCRIPT_FILENAME"); scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); } bool iis = false; // Because IIS doesn't pass requestUri, we simulate it here if it's empty. if(requestUri.length == 0) { // IIS sometimes includes the script name as part of the path info - we don't want that if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) pathInfo = pathInfo[scriptName.length .. $]; requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 // FIXME: this works for apache and iis... but what about others? } auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); getArray = assumeUnique(ugh); get = keepLastOf(getArray); // NOTE: on apache, you need to specifically forward this authorization = getenv("HTTP_AUTHORIZATION"); // this is a hack because Apache is a shitload of fuck and // refuses to send the real header to us. Compatible // programs should send both the standard and X- versions // NOTE: if you have access to .htaccess or httpd.conf, you can make this // unnecessary with mod_rewrite, so it is commented //if(authorization.length == 0) // if the std is there, use it // authorization = getenv("HTTP_X_AUTHORIZATION"); // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") port = to!int(getenv("SERVER_PORT")); else port = 0; // this was probably called from the command line auto ae = getenv("HTTP_ACCEPT_ENCODING"); if(ae.length && ae.indexOf("gzip") != -1) acceptsGzip = true; accept = getenv("HTTP_ACCEPT"); lastEventId = getenv("HTTP_LAST_EVENT_ID"); auto ka = getenv("HTTP_CONNECTION"); if(ka.length && ka.asLowerCase().canFind("keep-alive")) keepAliveRequested = true; auto or = getenv("HTTP_ORIGIN"); origin = or; auto rm = getenv("REQUEST_METHOD"); if(rm.length) requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); else requestMethod = RequestMethod.CommandLine; // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); // FIXME: DOCUMENT_ROOT? // FIXME: what about PUT? if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { version(preserveData) // a hack to make forwarding simpler immutable(ubyte)[] data; size_t amountReceived = 0; auto contentType = getenv("CONTENT_TYPE"); // FIXME: is this ever not going to be set? I guess it depends // on if the server de-chunks and buffers... seems like it has potential // to be slow if they did that. The spec says it is always there though. // And it has worked reliably for me all year in the live environment, // but some servers might be different. auto cls = getenv("CONTENT_LENGTH"); auto contentLength = to!size_t(cls.length ? cls : "0"); immutable originalContentLength = contentLength; if(contentLength) { if(maxContentLength > 0 && contentLength > maxContentLength) { setResponseStatus("413 Request entity too large"); write("You tried to upload a file that is too large."); close(); throw new Exception("POST too large"); } prepareForIncomingDataChunks(contentType, contentLength); int processChunk(in ubyte[] chunk) { if(chunk.length > contentLength) { handleIncomingDataChunk(chunk[0..contentLength]); amountReceived += contentLength; contentLength = 0; return 1; } else { handleIncomingDataChunk(chunk); contentLength -= chunk.length; amountReceived += chunk.length; } if(contentLength == 0) return 1; onRequestBodyDataReceived(amountReceived, originalContentLength); return 0; } if(readdata is null) { foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) if(processChunk(chunk)) break; } else { // we have a custom data source.. auto chunk = readdata(); while(chunk.length) { if(processChunk(chunk)) break; chunk = readdata(); } } onRequestBodyDataReceived(amountReceived, originalContentLength); postArray = assumeUnique(pps._post); filesArray = assumeUnique(pps._files); files = keepLastOf(filesArray); post = keepLastOf(postArray); this.postBody = pps.postBody; cleanUpPostDataState(); } version(preserveData) originalPostData = data; } // fixme: remote_user script name this.port = port; this.referrer = referrer; this.remoteAddress = remoteAddress; this.userAgent = userAgent; this.authorization = authorization; this.origin = origin; this.accept = accept; this.lastEventId = lastEventId; this.https = https; this.host = host; this.requestMethod = requestMethod; this.requestUri = requestUri; this.pathInfo = pathInfo; this.queryString = queryString; } /// Cleans up any temporary files. Do not use the object /// after calling this. /// /// NOTE: it is called automatically by GenericMain // FIXME: this should be called if the constructor fails too, if it has created some garbage... void dispose() { foreach(file; files) { if(!file.contentInMemory) if(std.file.exists(file.contentFilename)) std.file.remove(file.contentFilename); } } private { struct PostParserState { string contentType; string boundary; string localBoundary; // the ones used at the end or something lol bool isMultipart; bool needsSavedBody; ulong expectedLength; ulong contentConsumed; immutable(ubyte)[] buffer; // multipart parsing state int whatDoWeWant; bool weHaveAPart; string[] thisOnesHeaders; immutable(ubyte)[] thisOnesData; string postBody; UploadedFile piece; bool isFile = false; size_t memoryCommitted; // do NOT keep mutable references to these anywhere! // I assume they are unique in the constructor once we're all done getting data. string[][string] _post; UploadedFile[][string] _files; } PostParserState pps; } /// This represents a file the user uploaded via a POST request. static struct UploadedFile { /// If you want to create one of these structs for yourself from some data, /// use this function. static UploadedFile fromData(immutable(void)[] data, string name = null) { Cgi.UploadedFile f; f.filename = name; f.content = cast(immutable(ubyte)[]) data; f.contentInMemory = true; return f; } string name; /// The name of the form element. string filename; /// The filename the user set. string contentType; /// The MIME type the user's browser reported. (Not reliable.) /** For small files, cgi.d will buffer the uploaded file in memory, and make it directly accessible to you through the content member. I find this very convenient and somewhat efficient, since it can avoid hitting the disk entirely. (I often want to inspect and modify the file anyway!) I find the file is very large, it is undesirable to eat that much memory just for a file buffer. In those cases, if you pass a large enough value for maxContentLength to the constructor so they are accepted, cgi.d will write the content to a temporary file that you can re-read later. You can override this behavior by subclassing Cgi and overriding the protected handlePostChunk method. Note that the object is not initialized when you write that method - the http headers are available, but the cgi.post method is not. You may parse the file as it streams in using this method. Anyway, if the file is small enough to be in memory, contentInMemory will be set to true, and the content is available in the content member. If not, contentInMemory will be set to false, and the content saved in a file, whose name will be available in the contentFilename member. Tip: if you know you are always dealing with small files, and want the convenience of ignoring this member, construct Cgi with a small maxContentLength. Then, if a large file comes in, it simply throws an exception (and HTTP error response) instead of trying to handle it. The default value of maxContentLength in the constructor is for small files. */ bool contentInMemory = true; // the default ought to always be true immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. /// ulong fileSize() { if(contentInMemory) return content.length; import std.file; return std.file.getSize(contentFilename); } /// void writeToFile(string filenameToSaveTo) const { import std.file; if(contentInMemory) std.file.write(filenameToSaveTo, content); else std.file.rename(contentFilename, filenameToSaveTo); } } // given a content type and length, decide what we're going to do with the data.. protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { pps.expectedLength = contentLength; auto terminator = contentType.indexOf(";"); if(terminator == -1) terminator = contentType.length; pps.contentType = contentType[0 .. terminator]; auto b = contentType[terminator .. $]; if(b.length) { auto idx = b.indexOf("boundary="); if(idx != -1) { pps.boundary = b[idx + "boundary=".length .. $]; pps.localBoundary = "\r\n--" ~ pps.boundary; } } // while a content type SHOULD be sent according to the RFC, it is // not required. We're told we SHOULD guess by looking at the content // but it seems to me that this only happens when it is urlencoded. if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { pps.isMultipart = false; pps.needsSavedBody = false; } else if(pps.contentType == "multipart/form-data") { pps.isMultipart = true; enforce(pps.boundary.length, "no boundary"); } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params // save the body so the application can handle it pps.isMultipart = false; pps.needsSavedBody = true; } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too // save the body so the application can handle it pps.needsSavedBody = true; pps.isMultipart = false; } else { // the rest is 100% handled by the application. just save the body and send it to them pps.needsSavedBody = true; pps.isMultipart = false; } } // handles streaming POST data. If you handle some other content type, you should // override this. If the data isn't the content type you want, you ought to call // super.handleIncomingDataChunk so regular forms and files still work. // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network // input anyway, so I'm not going to get too worked up about it right now. protected void handleIncomingDataChunk(const(ubyte)[] chunk) { if(chunk.length == 0) return; assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so // if we're passed big chunks, it might throw unnecessarily. // just pass it smaller chunks at a time. if(pps.isMultipart) { // multipart/form-data // FIXME: this might want to be factored out and factorized // need to make sure the stream hooks actually work. void pieceHasNewContent() { // we just grew the piece's buffer. Do we have to switch to file backing? if(pps.piece.contentInMemory) { if(pps.piece.content.length <= 10 * 1024 * 1024) // meh, I'm ok with it. return; else { // this is too big. if(!pps.isFile) throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. else { // a file this large is probably acceptable though... let's use a backing file. pps.piece.contentInMemory = false; // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. int count = 0; pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); // odds are this loop will never be entered, but we want it just in case. while(std.file.exists(pps.piece.contentFilename)) { count++; pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); } // I hope this creates the file pretty quickly, or the loop might be useless... // FIXME: maybe I should write some kind of custom transaction here. std.file.write(pps.piece.contentFilename, pps.piece.content); pps.piece.content = null; } } } else { // it's already in a file, so just append it to what we have if(pps.piece.content.length) { // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... std.file.append(pps.piece.contentFilename, pps.piece.content); pps.piece.content = null; } } } void commitPart() { if(!pps.weHaveAPart) return; pieceHasNewContent(); // be sure the new content is handled every time if(pps.isFile) { // I'm not sure if other environments put files in post or not... // I used to not do it, but I think I should, since it is there... pps._post[pps.piece.name] ~= pps.piece.filename; pps._files[pps.piece.name] ~= pps.piece; allPostNamesInOrder ~= pps.piece.name; allPostValuesInOrder ~= pps.piece.filename; } else { pps._post[pps.piece.name] ~= cast(string) pps.piece.content; allPostNamesInOrder ~= pps.piece.name; allPostValuesInOrder ~= cast(string) pps.piece.content; } /* stderr.writeln("RECEIVED: ", pps.piece.name, "=", pps.piece.content.length < 1000 ? to!string(pps.piece.content) : "too long"); */ // FIXME: the limit here pps.memoryCommitted += pps.piece.content.length; pps.weHaveAPart = false; pps.whatDoWeWant = 1; pps.thisOnesHeaders = null; pps.thisOnesData = null; pps.piece = UploadedFile.init; pps.isFile = false; } void acceptChunk() { pps.buffer ~= chunk; chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion } immutable(ubyte)[] consume(size_t howMuch) { pps.contentConsumed += howMuch; auto ret = pps.buffer[0 .. howMuch]; pps.buffer = pps.buffer[howMuch .. $]; return ret; } dataConsumptionLoop: do { switch(pps.whatDoWeWant) { default: assert(0); case 0: acceptChunk(); // the format begins with two extra leading dashes, then we should be at the boundary if(pps.buffer.length < 2) return; assert(pps.buffer[0] == '-', "no leading dash"); consume(1); assert(pps.buffer[0] == '-', "no second leading dash"); consume(1); pps.whatDoWeWant = 1; goto case 1; /* fallthrough */ case 1: // looking for headers // here, we should be lined up right at the boundary, which is followed by a \r\n // want to keep the buffer under control in case we're under attack //stderr.writeln("here once"); //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... // throw new Exception("wtf is up with the huge mime part headers"); acceptChunk(); if(pps.buffer.length < pps.boundary.length) return; // not enough data, since there should always be a boundary here at least if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n // we *should* be at the end here! assert(pps.buffer[0] == '-'); consume(1); assert(pps.buffer[0] == '-'); consume(1); // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, "not lined up on boundary " ~ pps.boundary); consume(pps.boundary.length); assert(pps.buffer[0] == '-'); consume(1); assert(pps.buffer[0] == '-'); consume(1); assert(pps.buffer[0] == '\r'); consume(1); assert(pps.buffer[0] == '\n'); consume(1); assert(pps.buffer.length == 0); assert(pps.contentConsumed == pps.expectedLength); break dataConsumptionLoop; // we're done! } else { // we're not done yet. We should be lined up on a boundary. // But, we want to ensure the headers are here before we consume anything! auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); if(headerEndLocation == -1) return; // they *should* all be here, so we can handle them all at once. assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, "not lined up on boundary " ~ pps.boundary); consume(pps.boundary.length); // the boundary is always followed by a \r\n assert(pps.buffer[0] == '\r'); consume(1); assert(pps.buffer[0] == '\n'); consume(1); } // re-running since by consuming the boundary, we invalidate the old index. auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); assert(headerEndLocation >= 0, "no header"); auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); // now we'll parse the headers foreach(h; pps.thisOnesHeaders) { auto p = h.indexOf(":"); assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); string hn = h[0..p]; string hv = h[p+2..$]; switch(hn.toLower) { default: assert(0); case "content-disposition": auto info = hv.split("; "); foreach(i; info[1..$]) { // skipping the form-data auto o = i.split("="); // FIXME string pn = o[0]; string pv = o[1][1..$-1]; if(pn == "name") { pps.piece.name = pv; } else if (pn == "filename") { pps.piece.filename = pv; pps.isFile = true; } } break; case "content-type": pps.piece.contentType = hv; break; } } pps.whatDoWeWant++; // move to the next step - the data break; case 2: // when we get here, pps.buffer should contain our first chunk of data if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much throw new Exception("wtf is up with the huge mime part buffer"); acceptChunk(); // so the trick is, we want to process all the data up to the boundary, // but what if the chunk's end cuts the boundary off? If we're unsure, we // want to wait for the next chunk. We start by looking for the whole boundary // in the buffer somewhere. auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); if(boundaryLocation != -1) { // this is easy - we can see it in it's entirety! pps.piece.content ~= consume(boundaryLocation); assert(pps.buffer[0] == '\r'); consume(1); assert(pps.buffer[0] == '\n'); consume(1); assert(pps.buffer[0] == '-'); consume(1); assert(pps.buffer[0] == '-'); consume(1); // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. pps.weHaveAPart = true; pps.whatDoWeWant = 1; // back to getting headers for the next part commitPart(); // we're done here } else { // we can't see the whole thing, but what if there's a partial boundary? enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... assert(pps.localBoundary.length > 1); // should already be sane but just in case bool potentialBoundaryFound = false; boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { // we grow the boundary a bit each time. If we think it looks the // same, better pull another chunk to be sure it's not the end. // Starting small because exiting the loop early is desirable, since // we're not keeping any ambiguity and 1 / 256 chance of exiting is // the best we can do. if(a > pps.buffer.length) break; // FIXME: is this right? assert(a <= pps.buffer.length); assert(a > 0); if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { // ok, there *might* be a boundary here, so let's // not treat the end as data yet. The rest is good to // use though, since if there was a boundary there, we'd // have handled it up above after locationOf. pps.piece.content ~= pps.buffer[0 .. $ - a]; consume(pps.buffer.length - a); pieceHasNewContent(); potentialBoundaryFound = true; break boundaryCheck; } } if(!potentialBoundaryFound) { // we can consume the whole thing pps.piece.content ~= pps.buffer; pieceHasNewContent(); consume(pps.buffer.length); } else { // we found a possible boundary, but there was // insufficient data to be sure. assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); return; // wait for the next chunk. } } } } while(pps.buffer.length); // btw all boundaries except the first should have a \r\n before them } else { // application/x-www-form-urlencoded and application/json // not using maxContentLength because that might be cranked up to allow // large file uploads. We can handle them, but a huge post[] isn't any good. if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough throw new Exception("wtf is up with such a gigantic form submission????"); pps.buffer ~= chunk; // simple handling, but it works... until someone bombs us with gigabytes of crap at least... if(pps.buffer.length == pps.expectedLength) { if(pps.needsSavedBody) pps.postBody = cast(string) pps.buffer; else pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); version(preserveData) originalPostData = pps.buffer; } else { // just for debugging } } } protected void cleanUpPostDataState() { pps = PostParserState.init; } /// you can override this function to somehow react /// to an upload in progress. /// /// Take note that parts of the CGI object is not yet /// initialized! Stuff from HTTP headers, including get[], is usable. /// But, none of post[] is usable, and you cannot write here. That's /// why this method is const - mutating the object won't do much anyway. /// /// My idea here was so you can output a progress bar or /// something to a cooperative client (see arsd.rtud for a potential helper) /// /// The default is to do nothing. Subclass cgi and use the /// CustomCgiMain mixin to do something here. void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { // This space intentionally left blank. } /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. /// *closeConnection will be set to true if you should close the connection after handling this request this(BufferedInputRange ir, bool* closeConnection) { isCalledWithCommandLineArguments = false; import al = std.algorithm; immutable(ubyte)[] data; void rdo(const(ubyte)[] d) { //import std.stdio; writeln(d); sendAll(ir.source, d); } auto ira = ir.source.remoteAddress(); auto irLocalAddress = ir.source.localAddress(); ushort port = 80; if(auto ia = cast(InternetAddress) irLocalAddress) { port = ia.port; } else if(auto ia = cast(Internet6Address) irLocalAddress) { port = ia.port; } // that check for UnixAddress is to work around a Phobos bug // see: https://github.com/dlang/phobos/pull/7383 // but this might be more useful anyway tbh for this case version(Posix) this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); else this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); } /** Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host Params: inputData = the incoming data, including headers and other raw http data. When the constructor exits, it will leave this range exactly at the start of the next request on the connection (if there is one). address = the IP address of the remote user _port = the port number of the connection pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. _https = if this connection is encrypted (note that the input data must not actually be encrypted) _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire closeConnection = if the request asks to close the connection, *closeConnection == true. */ this( BufferedInputRange inputData, // string[] headers, immutable(ubyte)[] data, string address, ushort _port, int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment bool _https = false, void delegate(const(ubyte)[]) _rawDataOutput = null, void delegate() _flush = null, // this pointer tells if the connection is supposed to be closed after we handle this bool* closeConnection = null) { // these are all set locally so the loop works // without triggering errors in dmd 2.064 // we go ahead and set them at the end of it to the this version int port; string referrer; string remoteAddress; string userAgent; string authorization; string origin; string accept; string lastEventId; bool https; string host; RequestMethod requestMethod; string requestUri; string pathInfo; string queryString; string scriptName; string[string] get; string[][string] getArray; bool keepAliveRequested; bool acceptsGzip; string cookie; environmentVariables = cast(const) environment.toAA; idlol = inputData; isCalledWithCommandLineArguments = false; https = _https; port = _port; rawDataOutput = _rawDataOutput; flushDelegate = _flush; nph = true; remoteAddress = address; // streaming parser import al = std.algorithm; // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); while(idx == -1) { inputData.popFront(0); idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); } assert(idx != -1); string contentType = ""; string[string] requestHeadersHere; size_t contentLength; bool isChunked; { import core.runtime; scriptFileName = Runtime.args.length ? Runtime.args[0] : null; } int headerNumber = 0; foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) if(line.length) { headerNumber++; auto header = cast(string) line.idup; if(headerNumber == 1) { // request line auto parts = al.splitter(header, " "); requestMethod = to!RequestMethod(parts.front); parts.popFront(); requestUri = parts.front; // FIXME: the requestUri could be an absolute path!!! should I rename it or something? scriptName = requestUri[0 .. pathInfoStarts]; auto question = requestUri.indexOf("?"); if(question == -1) { queryString = ""; // FIXME: double check, this might be wrong since it could be url encoded pathInfo = requestUri[pathInfoStarts..$]; } else { queryString = requestUri[question+1..$]; pathInfo = requestUri[pathInfoStarts..question]; } auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); getArray = cast(string[][string]) assumeUnique(ugh); if(header.indexOf("HTTP/1.0") != -1) { http10 = true; autoBuffer = true; if(closeConnection) { // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) *closeConnection = true; } } } else { // other header auto colon = header.indexOf(":"); if(colon == -1) throw new Exception("HTTP headers should have a colon!"); string name = header[0..colon].toLower; string value = header[colon+2..$]; // skip the colon and the space requestHeadersHere[name] = value; if (name == "accept") { accept = value; } else if (name == "origin") { origin = value; } else if (name == "connection") { if(value == "close" && closeConnection) *closeConnection = true; if(value.asLowerCase().canFind("keep-alive")) { keepAliveRequested = true; // on http 1.0, the connection is closed by default, // but not if they request keep-alive. then we don't close // anymore - undoing the set above if(http10 && closeConnection) { *closeConnection = false; } } } else if (name == "transfer-encoding") { if(value == "chunked") isChunked = true; } else if (name == "last-event-id") { lastEventId = value; } else if (name == "authorization") { authorization = value; } else if (name == "content-type") { contentType = value; } else if (name == "content-length") { contentLength = to!size_t(value); } else if (name == "x-forwarded-for") { remoteAddress = value; } else if (name == "x-forwarded-host" || name == "host") { if(name != "host" || host is null) host = value; } // FIXME: https://tools.ietf.org/html/rfc7239 else if (name == "accept-encoding") { if(value.indexOf("gzip") != -1) acceptsGzip = true; } else if (name == "user-agent") { userAgent = value; } else if (name == "referer") { referrer = value; } else if (name == "cookie") { cookie ~= value; } else if(name == "expect") { if(value == "100-continue") { // FIXME we should probably give user code a chance // to process and reject but that needs to be virtual, // perhaps part of the CGI redesign. // FIXME: if size is > max content length it should // also fail at this point. _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); // FIXME: let the user write out 103 early hints too } } // else // ignore it } } inputData.consume(idx + 4); // done requestHeaders = assumeUnique(requestHeadersHere); ByChunkRange dataByChunk; // reading Content-Length type data // We need to read up the data we have, and write it out as a chunk. if(!isChunked) { dataByChunk = byChunk(inputData, contentLength); } else { // chunked requests happen, but not every day. Since we need to know // the content length (for now, maybe that should change), we'll buffer // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) auto data = dechunk(inputData); // set the range here dataByChunk = byChunk(data); contentLength = data.length; } assert(dataByChunk !is null); if(contentLength) { prepareForIncomingDataChunks(contentType, contentLength); foreach(dataChunk; dataByChunk) { handleIncomingDataChunk(dataChunk); } postArray = assumeUnique(pps._post); filesArray = assumeUnique(pps._files); files = keepLastOf(filesArray); post = keepLastOf(postArray); postBody = pps.postBody; cleanUpPostDataState(); } this.port = port; this.referrer = referrer; this.remoteAddress = remoteAddress; this.userAgent = userAgent; this.authorization = authorization; this.origin = origin; this.accept = accept; this.lastEventId = lastEventId; this.https = https; this.host = host; this.requestMethod = requestMethod; this.requestUri = requestUri; this.pathInfo = pathInfo; this.queryString = queryString; this.scriptName = scriptName; this.get = keepLastOf(getArray); this.getArray = cast(immutable) getArray; this.keepAliveRequested = keepAliveRequested; this.acceptsGzip = acceptsGzip; this.cookie = cookie; cookiesArray = getCookieArray(); cookies = keepLastOf(cookiesArray); } BufferedInputRange idlol; private immutable(string[string]) keepLastOf(in string[][string] arr) { string[string] ca; foreach(k, v; arr) ca[k] = v[$-1]; return assumeUnique(ca); } // FIXME duplication private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { UploadedFile[string] ca; foreach(k, v; arr) ca[k] = v[$-1]; return assumeUnique(ca); } private immutable(string[][string]) getCookieArray() { auto forTheLoveOfGod = decodeVariables(cookie, "; "); return assumeUnique(forTheLoveOfGod); } /// Very simple method to require a basic auth username and password. /// If the http request doesn't include the required credentials, it throws a /// HTTP 401 error, and an exception. /// /// Note: basic auth does not provide great security, especially over unencrypted HTTP; /// the user's credentials are sent in plain text on every request. /// /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the /// application. Either use Apache's built in methods for basic authentication, or add /// something along these lines to your server configuration: /// /// RewriteEngine On /// RewriteCond %{HTTP:Authorization} ^(.*) /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] /// /// To ensure the necessary data is available to cgi.d. void requireBasicAuth(string user, string pass, string message = null) { if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { setResponseStatus("401 Authorization Required"); header ("WWW-Authenticate: Basic realm=\""~message~"\""); close(); throw new Exception("Not authorized; got " ~ authorization); } } /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. /// setCache(true) means it will always be cached for as long as possible. Best for static content. /// Use setResponseExpires and updateResponseExpires for more control void setCache(bool allowCaching) { noCache = !allowCaching; } /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers /// who can accept it bool gzipResponse; immutable bool acceptsGzip; immutable bool keepAliveRequested; /// Set to true if and only if this was initialized with command line arguments immutable bool isCalledWithCommandLineArguments; /// This gets a full url for the current request, including port, protocol, host, path, and query string getCurrentCompleteUri() const { ushort defaultPort = https ? 443 : 80; string uri = "http"; if(https) uri ~= "s"; uri ~= "://"; uri ~= host; /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right thing now version(none) if(!(!port || port == defaultPort)) { uri ~= ":"; uri ~= to!string(port); } +/ uri ~= requestUri; return uri; } /// You can override this if your site base url isn't the same as the script name string logicalScriptName() const { return scriptName; } /++ Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). Note setResponseStatus() must be called *before* you write() any data to the output. History: The `int` overload was added on January 11, 2021. +/ void setResponseStatus(string status) { assert(!outputtedResponseData); responseStatus = status; } /// ditto void setResponseStatus(int statusCode) { setResponseStatus(getHttpCodeText(statusCode)); } private string responseStatus = null; /// Returns true if it is still possible to output headers bool canOutputHeaders() { return !isClosed && !outputtedResponseData; } /// Sets the location header, which the browser will redirect the user to automatically. /// Note setResponseLocation() must be called *before* you write() any data to the output. /// The optional important argument is used if it's a default suggestion rather than something to insist upon. void setResponseLocation(string uri, bool important = true, string status = null) { if(!important && isCurrentResponseLocationImportant) return; // important redirects always override unimportant ones if(uri is null) { responseStatus = "200 OK"; responseLocation = null; isCurrentResponseLocationImportant = important; return; // this just cancels the redirect } assert(!outputtedResponseData); if(status is null) responseStatus = "302 Found"; else responseStatus = status; responseLocation = uri.strip; isCurrentResponseLocationImportant = important; } protected string responseLocation = null; private bool isCurrentResponseLocationImportant = false; /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. /// Note: the when parameter is different than setCookie's expire parameter. void setResponseExpires(long when, bool isPublic = false) { responseExpires = when; setCache(true); // need to enable caching so the date has meaning responseIsPublic = isPublic; responseExpiresRelative = false; } /// Sets a cache-control max-age header for whenFromNow, in seconds. void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { responseExpires = whenFromNow; setCache(true); // need to enable caching so the date has meaning responseIsPublic = isPublic; responseExpiresRelative = true; } private long responseExpires = long.min; private bool responseIsPublic = false; private bool responseExpiresRelative = false; /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program /// output as a whole is as cacheable as the least cacheable part in the chain. /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. void updateResponseExpires(long when, bool isPublic) { if(responseExpires == long.min) setResponseExpires(when, isPublic); else if(when < responseExpires) setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is } /* /// Set to true if you want the result to be cached publicly - that is, is the content shared? /// Should generally be false if the user is logged in. It assumes private cache only. /// setCache(true) also turns on public caching, and setCache(false) sets to private. void setPublicCaching(bool allowPublicCaches) { publicCaching = allowPublicCaches; } private bool publicCaching = false; */ /++ History: Added January 11, 2021 +/ enum SameSitePolicy { Lax, Strict, None } /++ Sets an HTTP cookie, automatically encoding the data to the correct string. expiresIn is how many milliseconds in the future the cookie will expire. TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. Note setCookie() must be called *before* you write() any data to the output. History: Parameter `sameSitePolicy` was added on January 11, 2021. +/ void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { assert(!outputtedResponseData); string cookie = std.uri.encodeComponent(name) ~ "="; cookie ~= std.uri.encodeComponent(data); if(path !is null) cookie ~= "; path=" ~ path; // FIXME: should I just be using max-age here? (also in cache below) if(expiresIn != 0) cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); if(domain !is null) cookie ~= "; domain=" ~ domain; if(secure == true) cookie ~= "; Secure"; if(httpOnly == true ) cookie ~= "; HttpOnly"; final switch(sameSitePolicy) { case SameSitePolicy.Lax: cookie ~= "; SameSite=Lax"; break; case SameSitePolicy.Strict: cookie ~= "; SameSite=Strict"; break; case SameSitePolicy.None: cookie ~= "; SameSite=None"; assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite break; } if(auto idx = name in cookieIndexes) { responseCookies[*idx] = cookie; } else { cookieIndexes[name] = responseCookies.length; responseCookies ~= cookie; } } private string[] responseCookies; private size_t[string] cookieIndexes; /// Clears a previously set cookie with the given name, path, and domain. void clearCookie(string name, string path = null, string domain = null) { assert(!outputtedResponseData); setCookie(name, "", 1, path, domain); } /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image void setResponseContentType(string ct) { assert(!outputtedResponseData); responseContentType = ct; } private string responseContentType = null; /// Adds a custom header. It should be the name: value, but without any line terminator. /// For example: header("X-My-Header: Some value"); /// Note you should use the specialized functions in this object if possible to avoid /// duplicates in the output. void header(string h) { customHeaders ~= h; } /++ I named the original function `header` after PHP, but this pattern more fits the rest of the Cgi object. Either name are allowed. History: Alias added June 17, 2022. +/ alias setResponseHeader = header; private string[] customHeaders; private bool websocketMode; void flushHeaders(const(void)[] t, bool isAll = false) { StackBuffer buffer = StackBuffer(0); prepHeaders(t, isAll, &buffer); if(rawDataOutput !is null) rawDataOutput(cast(const(ubyte)[]) buffer.get()); else { stdout.rawWrite(buffer.get()); } } private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { string terminator = "\n"; if(rawDataOutput !is null) terminator = "\r\n"; if(responseStatus !is null) { if(nph) { if(http10) buffer.add("HTTP/1.0 ", responseStatus, terminator); else buffer.add("HTTP/1.1 ", responseStatus, terminator); } else buffer.add("Status: ", responseStatus, terminator); } else if (nph) { if(http10) buffer.add("HTTP/1.0 200 OK", terminator); else buffer.add("HTTP/1.1 200 OK", terminator); } if(websocketMode) goto websocket; if(nph) { // we're responsible for setting the date too according to http 1.1 char[29] db = void; printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); buffer.add("Date: ", db[], terminator); } // FIXME: what if the user wants to set his own content-length? // The custom header function can do it, so maybe that's best. // Or we could reuse the isAll param. if(responseLocation !is null) { buffer.add("Location: ", responseLocation, terminator); } if(!noCache && responseExpires != long.min) { // an explicit expiration date is set if(responseExpiresRelative) { buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); buffer.add(responseExpires); buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); } else { auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); char[29] db = void; printDateToBuffer(cast(DateTime) expires, db[]); buffer.add("Expires: ", db[], terminator); // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); buffer.add(terminator); } } if(responseCookies !is null && responseCookies.length > 0) { foreach(c; responseCookies) buffer.add("Set-Cookie: ", c, terminator); } if(noCache) { // we specifically do not want caching (this is actually the default) buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); buffer.add("Expires: 0", terminator); buffer.add("Pragma: no-cache", terminator); } else { if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever buffer.add("Cache-Control: public", terminator); buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future } } if(responseContentType !is null) { buffer.add("Content-Type: ", responseContentType, terminator); } else buffer.add("Content-Type: text/html; charset=utf-8", terminator); if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary buffer.add("Content-Encoding: gzip", terminator); } if(!isAll) { if(nph && !http10) { buffer.add("Transfer-Encoding: chunked", terminator); responseChunked = true; } } else { buffer.add("Content-Length: "); buffer.add(t.length); buffer.add(terminator); if(nph && keepAliveRequested) { buffer.add("Connection: Keep-Alive", terminator); } } websocket: foreach(hd; customHeaders) buffer.add(hd, terminator); // FIXME: what about duplicated headers? // end of header indicator buffer.add(terminator); outputtedResponseData = true; } /// Writes the data to the output, flushing headers if they have not yet been sent. void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { assert(!closed, "Output has already been closed"); StackBuffer buffer = StackBuffer(0); if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary // actually gzip the data here auto c = new Compress(HeaderFormat.gzip); // want gzip auto data = c.compress(t); data ~= c.flush(); // std.file.write("/tmp/last-item", data); t = data; } if(!outputtedResponseData && (!autoBuffer || isAll)) { prepHeaders(t, isAll, &buffer); } if(requestMethod != RequestMethod.HEAD && t.length > 0) { if (autoBuffer && !isAll) { outputBuffer ~= cast(ubyte[]) t; } if(!autoBuffer || isAll) { if(rawDataOutput !is null) if(nph && responseChunked) { //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); // we're making the chunk here instead of in a function // to avoid unneeded gc pressure buffer.add(toHex(t.length)); buffer.add("\r\n"); buffer.add(cast(char[]) t, "\r\n"); } else { buffer.add(cast(char[]) t); } else buffer.add(cast(char[]) t); } } if(rawDataOutput !is null) rawDataOutput(cast(const(ubyte)[]) buffer.get()); else stdout.rawWrite(buffer.get()); if(maybeAutoClose && isAll) close(); // if you say it is all, that means we're definitely done // maybeAutoClose can be false though to avoid this (important if you call from inside close()! } /++ Convenience method to set content type to json and write the string as the complete response. History: Added January 16, 2020 +/ void writeJson(string json) { this.setResponseContentType("application/json"); this.write(json, true); } /// Flushes the pending buffer, leaving the connection open so you can send more. void flush() { if(rawDataOutput is null) stdout.flush(); else if(flushDelegate !is null) flushDelegate(); } version(autoBuffer) bool autoBuffer = true; else bool autoBuffer = false; ubyte[] outputBuffer; /// Flushes the buffers to the network, signifying that you are done. /// You should always call this explicitly when you are done outputting data. void close() { if(closed) return; // don't double close if(!outputtedResponseData) write("", true, false); // writing auto buffered data if(requestMethod != RequestMethod.HEAD && autoBuffer) { if(!nph) stdout.rawWrite(outputBuffer); else write(outputBuffer, true, false); // tell it this is everything } // closing the last chunk... if(nph && rawDataOutput !is null && responseChunked) rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); if(flushDelegate) flushDelegate(); closed = true; } // Closes without doing anything, shouldn't be used often void rawClose() { closed = true; } /++ Gets a request variable as a specific type, or the default value of it isn't there or isn't convertible to the request type. Checks both GET and POST variables, preferring the POST variable, if available. A nice trick is using the default value to choose the type: --- /* The return value will match the type of the default. Here, I gave 10 as a default, so the return value will be an int. If the user-supplied value cannot be converted to the requested type, you will get the default value back. */ int a = cgi.request("number", 10); if(cgi.get["number"] == "11") assert(a == 11); // conversion succeeds if("number" !in cgi.get) assert(a == 10); // no value means you can't convert - give the default if(cgi.get["number"] == "twelve") assert(a == 10); // conversion from string to int would fail, so we get the default --- You can use an enum as an easy whitelist, too: --- enum Operations { add, remove, query } auto op = cgi.request("op", Operations.query); if(cgi.get["op"] == "add") assert(op == Operations.add); if(cgi.get["op"] == "remove") assert(op == Operations.remove); if(cgi.get["op"] == "query") assert(op == Operations.query); if(cgi.get["op"] == "random string") assert(op == Operations.query); // the value can't be converted to the enum, so we get the default --- +/ T request(T = string)(in string name, in T def = T.init) const nothrow { try { return (name in post) ? to!T(post[name]) : (name in get) ? to!T(get[name]) : def; } catch(Exception e) { return def; } } /// Is the output already closed? bool isClosed() const { return closed; } /++ Gets a session object associated with the `cgi` request. You can use different type throughout your application. +/ Session!Data getSessionObject(Data)() { if(testInProcess !is null) { // test mode auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); if(obj !is null) return cast(typeof(return)) obj; else { auto o = new MockSession!Data(); testInProcess.setSessionOverride(typeid(typeof(return)), o); return o; } } else { // normal operation return new BasicDataServerSession!Data(this); } } // if it is in test mode; triggers mock sessions. Used by CgiTester version(with_breaking_cgi_features) private CgiTester testInProcess; /* Hooks for redirecting input and output */ private void delegate(const(ubyte)[]) rawDataOutput = null; private void delegate() flushDelegate = null; /* This info is used when handling a more raw HTTP protocol */ private bool nph; private bool http10; private bool closed; private bool responseChunked = false; version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. immutable(ubyte)[] originalPostData; /++ This holds the posted body data if it has not been parsed into [post] and [postArray]. It is intended to be used for JSON and XML request content types, but also may be used for other content types your application can handle. But it will NOT be populated for content types application/x-www-form-urlencoded or multipart/form-data, since those are parsed into the post and postArray members. Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., will be discarded to the client with an error. This helps keep this array from being exploded in size and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent client in certain build modes.) History: Added January 5, 2021 Documented February 21, 2023 (dub v11.0) +/ public immutable string postBody; alias postJson = postBody; // old name /* Internal state flags */ private bool outputtedResponseData; private bool noCache = true; const(string[string]) environmentVariables; /** What follows is data gotten from the HTTP request. It is all fully immutable, partially because it logically is (your code doesn't change what the user requested...) and partially because I hate how bad programs in PHP change those superglobals to do all kinds of hard to follow ugliness. I don't want that to ever happen in D. For some of these, you'll want to refer to the http or cgi specs for more details. */ immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". immutable(char[]) scriptFileName; /// The physical filename of your script immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. */ immutable(char[]) referrer; immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) immutable bool https; /// Was the request encrypted via https? immutable int port; /// On what TCP port number did the server receive the request? /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) /** Represents user uploaded files. When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html
tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. */ immutable(UploadedFile[][string]) filesArray; immutable(UploadedFile[string]) files; /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. /// the order of the arrays is the order the data arrives immutable(string[][string]) getArray; /// like get, but an array of values per name immutable(string[][string]) postArray; /// ditto for post immutable(string[][string]) cookiesArray; /// ditto for cookies // convenience function for appending to a uri without extra ? // matches the name and effect of javascript's location.search property string search() const { if(queryString.length) return "?" ~ queryString; return ""; } // FIXME: what about multiple files with the same name? private: //RequestMethod _requestMethod; } /// use this for testing or other isolated things when you want it to be no-ops Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { // we want to ignore, not use stdout if(outputSink is null) outputSink = delegate void(const(ubyte)[]) { }; string[string] env; env["REQUEST_METHOD"] = to!string(method); env["CONTENT_LENGTH"] = to!string(data.length); auto cgi = new Cgi( 0, env, { return data; }, outputSink, null); return cgi; } /++ A helper test class for request handler unittests. +/ version(with_breaking_cgi_features) class CgiTester { private { SessionObject[TypeInfo] mockSessions; SessionObject getSessionOverride(TypeInfo ti) { if(auto o = ti in mockSessions) return *o; else return null; } void setSessionOverride(TypeInfo ti, SessionObject so) { mockSessions[ti] = so; } } /++ Gets (and creates if necessary) a mock session object for this test. Note it will be the same one used for any test operations through this CgiTester instance. +/ Session!Data getSessionObject(Data)() { auto obj = getSessionOverride(typeid(typeof(return))); if(obj !is null) return cast(typeof(return)) obj; else { auto o = new MockSession!Data(); setSessionOverride(typeid(typeof(return)), o); return o; } } /++ Pass a reference to your request handler when creating the tester. +/ this(void function(Cgi) requestHandler) { this.requestHandler = requestHandler; } /++ You can check response information with these methods after you call the request handler. +/ struct Response { int code; string[string] headers; string responseText; ubyte[] responseBody; } /++ Executes a test request on your request handler, and returns the response. Params: url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. args = additional arguments. Same format as cgi's command line handler. +/ Response GET(string url, string[] args = null) { return executeTest("GET", url, args); } /// ditto Response POST(string url, string[] args = null) { return executeTest("POST", url, args); } /// ditto Response executeTest(string method, string url, string[] args) { ubyte[] outputtedRawData; void outputSink(const(ubyte)[] data) { outputtedRawData ~= data; } auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); cgi.testInProcess = this; scope(exit) cgi.dispose(); requestHandler(cgi); cgi.close(); Response response; if(outputtedRawData.length) { enum LINE = "\r\n"; auto idx = outputtedRawData.locationOf(LINE ~ LINE); assert(idx != -1, to!string(outputtedRawData)); auto headers = cast(string) outputtedRawData[0 .. idx]; response.code = 200; while(headers.length) { auto i = headers.locationOf(LINE); if(i == -1) i = cast(int) headers.length; auto header = headers[0 .. i]; auto c = header.locationOf(":"); if(c != -1) { auto name = header[0 .. c]; auto value = header[c + 2 ..$]; if(name == "Status") response.code = value[0 .. value.locationOf(" ")].to!int; response.headers[name] = value; } else { assert(0); } if(i != headers.length) i += 2; headers = headers[i .. $]; } response.responseBody = outputtedRawData[idx + 4 .. $]; response.responseText = cast(string) response.responseBody; } return response; } private void function(Cgi) requestHandler; } // should this be a separate module? Probably, but that's a hassle. /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). string makeDataUrl(string mimeType, in void[] data) { auto data64 = Base64.encode(cast(const(ubyte[])) data); return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); } // FIXME: I don't think this class correctly decodes/encodes the individual parts /// Represents a url that can be broken down or built up through properties struct Uri { alias toString this; // blargh idk a url really is a string, but should it be implicit? // scheme//userinfo@host:port/path?query#fragment string scheme; /// e.g. "http" in "http://example.com/" string userinfo; /// the username (and possibly a password) in the uri string host; /// the domain name int port; /// port number, if given. Will be zero if a port was not explicitly given string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" string query; /// the stuff after the ? in a uri string fragment; /// the stuff after the # in a uri. // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility // the decode ones need to keep different names anyway because we can't overload on return values... static string encode(string s) { return std.uri.encodeComponent(s); } static string encode(string[string] s) { return encodeVariables(s); } static string encode(string[][string] s) { return encodeVariables(s); } /// Breaks down a uri string to its components this(string uri) { reparse(uri); } private void reparse(string uri) { // from RFC 3986 // the ctRegex triples the compile time and makes ugly errors for no real benefit // it was a nice experiment but just not worth it. // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; /* Captures: 0 = whole url 1 = scheme, with : 2 = scheme, no : 3 = authority, with // 4 = authority, no // 5 = path 6 = query string, with ? 7 = query string, no ? 8 = anchor, with # 9 = anchor, no # */ // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! // instead, I will DIY and cut that down to 0.6s on the same computer. /* Note that authority is user:password@domain:port where the user:password@ part is optional, and the :port is optional. Regex translation: Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. Path cannot have any ? or # in it. It is optional. Query must start with ? and must not have # in it. It is optional. Anchor must start with # and can have anything else in it to end of string. It is optional. */ this = Uri.init; // reset all state // empty uri = nothing special if(uri.length == 0) { return; } size_t idx; scheme_loop: foreach(char c; uri[idx .. $]) { switch(c) { case ':': case '/': case '?': case '#': break scheme_loop; default: } idx++; } if(idx == 0 && uri[idx] == ':') { // this is actually a path! we skip way ahead goto path_loop; } if(idx == uri.length) { // the whole thing is a path, apparently path = uri; return; } if(idx > 0 && uri[idx] == ':') { scheme = uri[0 .. idx]; idx++; } else { // we need to rewind; it found a / but no :, so the whole thing is prolly a path... idx = 0; } if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { // we have an authority.... idx += 2; auto authority_start = idx; authority_loop: foreach(char c; uri[idx .. $]) { switch(c) { case '/': case '?': case '#': break authority_loop; default: } idx++; } auto authority = uri[authority_start .. idx]; auto idx2 = authority.indexOf("@"); if(idx2 != -1) { userinfo = authority[0 .. idx2]; authority = authority[idx2 + 1 .. $]; } if(authority.length && authority[0] == '[') { // ipv6 address special casing idx2 = authority.indexOf(']'); if(idx2 != -1) { auto end = authority[idx2 + 1 .. $]; if(end.length && end[0] == ':') idx2 = idx2 + 1; else idx2 = -1; } } else { idx2 = authority.indexOf(":"); } if(idx2 == -1) { port = 0; // 0 means not specified; we should use the default for the scheme host = authority; } else { host = authority[0 .. idx2]; if(idx2 + 1 < authority.length) port = to!int(authority[idx2 + 1 .. $]); else port = 0; } } path_loop: auto path_start = idx; foreach(char c; uri[idx .. $]) { if(c == '?' || c == '#') break; idx++; } path = uri[path_start .. idx]; if(idx == uri.length) return; // nothing more to examine... if(uri[idx] == '?') { idx++; auto query_start = idx; foreach(char c; uri[idx .. $]) { if(c == '#') break; idx++; } query = uri[query_start .. idx]; } if(idx < uri.length && uri[idx] == '#') { idx++; fragment = uri[idx .. $]; } // uriInvalidated = false; } private string rebuildUri() const { string ret; if(scheme.length) ret ~= scheme ~ ":"; if(userinfo.length || host.length) ret ~= "//"; if(userinfo.length) ret ~= userinfo ~ "@"; if(host.length) ret ~= host; if(port) ret ~= ":" ~ to!string(port); ret ~= path; if(query.length) ret ~= "?" ~ query; if(fragment.length) ret ~= "#" ~ fragment; // uri = ret; // uriInvalidated = false; return ret; } /// Converts the broken down parts back into a complete string string toString() const { // if(uriInvalidated) return rebuildUri(); } /// Returns a new absolute Uri given a base. It treats this one as /// relative where possible, but absolute if not. (If protocol, domain, or /// other info is not set, the new one inherits it from the base.) /// /// Browsers use a function like this to figure out links in html. Uri basedOn(in Uri baseUrl) const { Uri n = this; // copies if(n.scheme == "data") return n; // n.uriInvalidated = true; // make sure we regenerate... // userinfo is not inherited... is this wrong? // if anything is given in the existing url, we don't use the base anymore. if(n.scheme.empty) { n.scheme = baseUrl.scheme; if(n.host.empty) { n.host = baseUrl.host; if(n.port == 0) { n.port = baseUrl.port; if(n.path.length > 0 && n.path[0] != '/') { auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; if(b.length == 0) b = "/"; n.path = b ~ n.path; } else if(n.path.length == 0) { n.path = baseUrl.path; } } } } n.removeDots(); return n; } void removeDots() { auto parts = this.path.split("/"); string[] toKeep; foreach(part; parts) { if(part == ".") { continue; } else if(part == "..") { //if(toKeep.length > 1) toKeep = toKeep[0 .. $-1]; //else //toKeep = [""]; continue; } else { //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) //continue; // skip a `//` situation toKeep ~= part; } } auto path = toKeep.join("/"); if(path.length && path[0] != '/') path = "/" ~ path; this.path = path; } unittest { auto uri = Uri("test.html"); assert(uri.path == "test.html"); uri = Uri("path/1/lol"); assert(uri.path == "path/1/lol"); uri = Uri("http://me@example.com"); assert(uri.scheme == "http"); assert(uri.userinfo == "me"); assert(uri.host == "example.com"); uri = Uri("http://example.com/#a"); assert(uri.scheme == "http"); assert(uri.host == "example.com"); assert(uri.fragment == "a"); uri = Uri("#foo"); assert(uri.fragment == "foo"); uri = Uri("?lol"); assert(uri.query == "lol"); uri = Uri("#foo?lol"); assert(uri.fragment == "foo?lol"); uri = Uri("?lol#foo"); assert(uri.fragment == "foo"); assert(uri.query == "lol"); uri = Uri("http://127.0.0.1/"); assert(uri.host == "127.0.0.1"); assert(uri.port == 0); uri = Uri("http://127.0.0.1:123/"); assert(uri.host == "127.0.0.1"); assert(uri.port == 123); uri = Uri("http://[ff:ff::0]/"); assert(uri.host == "[ff:ff::0]"); uri = Uri("http://[ff:ff::0]:123/"); assert(uri.host == "[ff:ff::0]"); assert(uri.port == 123); } // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover // the possibilities. unittest { auto url = Uri("cool.html"); // checking relative links assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); url = Uri("/something/cool.html"); // same server, different path assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); url = Uri("/test/bar"); assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); assert(Uri("../").basedOn(url) == "/"); url = Uri("http://example.com/"); assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); //auto uriBefore = url; url = Uri("#anchor"); // everything should remain the same except the anchor //uriBefore.anchor = "anchor"); //assert(url == uriBefore); url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. url = Uri("//example.com/example.html"); // same protocol, but different server and path url = Uri("http://example.com/test.html"); // completely absolute link should never be modified url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path // FIXME: add something for port too } // these are like javascript's location.search and location.hash string search() const { return query.length ? ("?" ~ query) : ""; } string hash() const { return fragment.length ? ("#" ~ fragment) : ""; } } /* for session, see web.d */ /// breaks down a url encoded string string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { auto vars = data.split(separator); string[][string] _get; foreach(var; vars) { auto equal = var.indexOf("="); string name; string value; if(equal == -1) { name = decodeComponent(var); value = ""; } else { //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); // stupid + -> space conversion. name = decodeComponent(var[0..equal].replace("+", " ")); value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); } _get[name] ~= value; if(namesInOrder) (*namesInOrder) ~= name; if(valuesInOrder) (*valuesInOrder) ~= value; } return _get; } /// breaks down a url encoded string, but only returns the last value of any array string[string] decodeVariablesSingle(string data) { string[string] va; auto varArray = decodeVariables(data); foreach(k, v; varArray) va[k] = v[$-1]; return va; } /// url encodes the whole string string encodeVariables(in string[string] data) { string ret; bool outputted = false; foreach(k, v; data) { if(outputted) ret ~= "&"; else outputted = true; ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); } return ret; } /// url encodes a whole string string encodeVariables(in string[][string] data) { string ret; bool outputted = false; foreach(k, arr; data) { foreach(v; arr) { if(outputted) ret ~= "&"; else outputted = true; ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); } } return ret; } /// Encodes all but the explicitly unreserved characters per rfc 3986 /// Alphanumeric and -_.~ are the only ones left unencoded /// name is borrowed from php string rawurlencode(in char[] data) { string ret; ret.reserve(data.length * 2); foreach(char c; data) { if( (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.' || c == '~') { ret ~= c; } else { ret ~= '%'; // since we iterate on char, this should give us the octets of the full utf8 string ret ~= toHexUpper(c); } } return ret; } // http helper functions // for chunked responses (which embedded http does whenever possible) version(none) // this is moved up above to avoid making a copy of the data const(ubyte)[] makeChunk(const(ubyte)[] data) { const(ubyte)[] ret; ret = cast(const(ubyte)[]) toHex(data.length); ret ~= cast(const(ubyte)[]) "\r\n"; ret ~= data; ret ~= cast(const(ubyte)[]) "\r\n"; return ret; } string toHex(long num) { string ret; while(num) { int v = num % 16; num /= 16; char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); ret ~= d; } return to!string(array(ret.retro)); } string toHexUpper(long num) { string ret; while(num) { int v = num % 16; num /= 16; char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); ret ~= d; } if(ret.length == 1) ret ~= "0"; // url encoding requires two digits and that's what this function is used for... return to!string(array(ret.retro)); } // the generic mixins /++ Use this instead of writing your own main It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. +/ mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { mixin CustomCgiMain!(Cgi, fun, maxContentLength); } /++ Boilerplate mixin for a main function that uses the [dispatcher] function. You can send `typeof(null)` as the `Presenter` argument to use a generic one. History: Added July 9, 2021 +/ mixin template DispatcherMain(Presenter, DispatcherArgs...) { /++ Handler to the generated presenter you can use from your objects, etc. +/ Presenter activePresenter; /++ Request handler that creates the presenter then forwards to the [dispatcher] function. Renders 404 if the dispatcher did not handle the request. Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" +/ void handler(Cgi cgi) { auto presenter = new Presenter; activePresenter = presenter; scope(exit) activePresenter = null; if(cgi.dispatcher!DispatcherArgs(presenter)) return; switch(cgi.pathInfo) { case "/style.css": cgi.setCache(true); cgi.setResponseContentType("text/css"); cgi.write(presenter.style(), true); break; case "/script.js": cgi.setCache(true); cgi.setResponseContentType("application/javascript"); cgi.write(presenter.script(), true); break; default: presenter.renderBasicError(cgi, 404); } } mixin GenericMain!handler; } mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { class GenericPresenter : WebPresenter!GenericPresenter {} mixin DispatcherMain!(GenericPresenter, DispatcherArgs); } private string simpleHtmlEncode(string s) { return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "
\n"); } string messageFromException(Throwable t) { string message; if(t !is null) { debug message = t.toString(); else message = "An unexpected error has occurred."; } else { message = "Unknown error"; } return message; } string plainHttpError(bool isCgi, string type, Throwable t) { auto message = messageFromException(t); message = simpleHtmlEncode(message); return format("%s %s\r\nContent-Length: %s\r\n\r\n%s", isCgi ? "Status:" : "HTTP/1.0", type, message.length, message); } // returns true if we were able to recover reasonably bool handleException(Cgi cgi, Throwable t) { if(cgi.isClosed) { // if the channel has been explicitly closed, we can't handle it here return true; } if(cgi.outputtedResponseData) { // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. } else { // no headers are sent, we can send a full blown error and recover cgi.setCache(false); cgi.setResponseContentType("text/html"); cgi.setResponseLocation(null); // cancel the redirect cgi.setResponseStatus("500 Internal Server Error"); cgi.write(simpleHtmlEncode(messageFromException(t))); cgi.close(); return true; } } bool isCgiRequestMethod(string s) { s = s.toUpper(); if(s == "COMMANDLINE") return true; foreach(member; __traits(allMembers, Cgi.RequestMethod)) if(s == member) return true; return false; } /// If you want to use a subclass of Cgi with generic main, use this mixin. mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere void main(string[] args) { cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); } } version(embedded_httpd_processes) __gshared int processPoolSize = 8; // Returns true if run. You should exit the program after that. bool tryAddonServers(string[] args) { if(args.length > 1) { // run the special separate processes if needed switch(args[1]) { case "--websocket-server": version(with_addon_servers) websocketServers[args[2]](args[3 .. $]); else printf("Add-on servers not compiled in.\n"); return true; case "--websocket-servers": import core.demangle; version(with_addon_servers_connections) foreach(k, v; websocketServers) writeln(k, "\t", demangle(k)); return true; case "--session-server": version(with_addon_servers) runSessionServer(); else printf("Add-on servers not compiled in.\n"); return true; case "--event-server": version(with_addon_servers) runEventServer(); else printf("Add-on servers not compiled in.\n"); return true; case "--timer-server": version(with_addon_servers) runTimerServer(); else printf("Add-on servers not compiled in.\n"); return true; case "--timed-jobs": import core.demangle; version(with_addon_servers_connections) foreach(k, v; scheduledJobHandlers) writeln(k, "\t", demangle(k)); return true; case "--timed-job": scheduledJobHandlers[args[2]](args[3 .. $]); return true; default: // intentionally blank - do nothing and carry on to run normally } } return false; } /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { // we support command line thing for easy testing everywhere // it needs to be called ./app method uri [other args...] if(args.length >= 3 && isCgiRequestMethod(args[1])) { Cgi cgi = new CustomCgi(args); scope(exit) cgi.dispose(); fun(cgi); cgi.close(); return true; } return false; } /++ A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] to an example of how you can use it. History: Added Sept 26, 2020 (release version 8.5). +/ struct RequestServer { /// string listeningHost = defaultListeningHost(); /// ushort listeningPort = defaultListeningPort(); /++ Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and [stop] may not work as well. History: Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for compatibility. +/ bool useFork = cgi_use_fork_default; /++ Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a default based on the number of cpus modified by the server mode. History: Added August 12, 2022 (dub v10.9) +/ int numberOfThreads = 0; /// this(string defaultHost, ushort defaultPort) { this.listeningHost = defaultHost; this.listeningPort = defaultPort; } /// this(ushort defaultPort) { listeningPort = defaultPort; } /++ Reads the command line arguments into the values here. Possible arguments are `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. +/ void configureFromCommandLine(string[] args) { bool foundPort = false; bool foundHost = false; bool foundUid = false; bool foundGid = false; foreach(arg; args) { if(foundPort) { listeningPort = to!ushort(arg); foundPort = false; } if(foundHost) { listeningHost = arg; foundHost = false; } if(foundUid) { privilegesDropToUid = to!uid_t(arg); foundUid = false; } if(foundGid) { privilegesDropToGid = to!gid_t(arg); foundGid = false; } if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") foundHost = true; else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") foundPort = true; else if(arg == "--uid") foundUid = true; else if(arg == "--gid") foundGid = true; } } version(Windows) { private alias uid_t = int; private alias gid_t = int; } /// user (uid) to drop privileges to /// 0 … do nothing uid_t privilegesDropToUid = 0; /// group (gid) to drop privileges to /// 0 … do nothing gid_t privilegesDropToGid = 0; private void dropPrivileges() { version(Posix) { import core.sys.posix.unistd; if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) throw new Exception("Dropping privileges via setgid() failed."); if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) throw new Exception("Dropping privileges via setuid() failed."); } else { // FIXME: Windows? //pragma(msg, "Dropping privileges is not implemented for this platform"); } // done, set zero privilegesDropToGid = 0; privilegesDropToUid = 0; } /++ Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders History: Added Oct 10, 2020. Example: --- import arsd.cgi; void main() { RequestServer server = RequestServer("127.0.0.1", 6789); string oauthCode; string oauthScope; server.serveHttpOnce!((cgi) { oauthCode = cgi.request("code"); oauthScope = cgi.request("scope"); cgi.write("Thank you, please return to the application."); }); // use the code and scope given } --- +/ void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { import std.socket; bool tcp; void delegate() cleanup; auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); auto connection = socket.accept(); doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); if(cleanup) cleanup(); } /++ Starts serving requests according to the current configuration. +/ void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { version(netman_httpd) { // Obsolete! import arsd.httpd; // what about forwarding the other constructor args? // this probably needs a whole redoing... serveHttp!CustomCgi(&fun, listeningPort);//5005); return; } else version(embedded_httpd_processes) { serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); } else version(embedded_httpd_threads) { serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); } else version(scgi) { serveScgi!(fun, CustomCgi, maxContentLength)(); } else version(fastcgi) { serveFastCgi!(fun, CustomCgi, maxContentLength)(this); } else version(stdio_http) { serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); } else { //version=plain_cgi; handleCgiRequest!(fun, CustomCgi, maxContentLength)(); } } /++ Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. +/ shared void serveEmbeddedHttp(alias fun, T, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(shared T _this) { globalStopFlag = false; static if(__traits(isStaticFunction, fun)) void funToUse(CustomCgi cgi) { fun(_this, cgi); } else void funToUse(CustomCgi cgi) { static if(__VERSION__ > 2097) __traits(child, _inst_this, fun)(_inst_this, cgi); else static assert(0, "Not implemented in your compiler version!"); } auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); manager.listen(); } /++ Runs the embedded SCGI server specifically, regardless of which build configuration you have. +/ void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { globalStopFlag = false; auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); manager.listen(); } /++ Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] History: Added May 29, 2021 +/ void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); } /++ The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't respond to this flag, the library will force the issue. This determines when and how the issue will be forced. +/ enum ForceStop { /++ Stops accepting new requests, but lets ones already in the queue start and complete before exiting. +/ afterQueuedRequestsComplete, /++ Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers should cooperate and exit gracefully, but if they don't, it will continue waiting for them. +/ afterCurrentRequestsComplete, /++ Partial response writes will throw an exception, cancelling any streaming response, but complete writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. +/ cancelStreamingRequestsEarly, /++ All writes will throw. +/ cancelAllRequestsEarly, /++ Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). +/ forciblyTerminate, } version(embedded_httpd_processes) {} else /++ Stops serving after the current requests are completed. Bugs: Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid on Windows however). Only partially implemented on non-Linux posix systems. You might also try SIGINT perhaps. The stopPriority is not yet fully implemented. +/ static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { globalStopFlag = true; version(Posix) { if(cancelfd > 0) { ulong a = 1; core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); } } version(Windows) { if(iocp) { foreach(i; 0 .. 16) // FIXME PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); } } } } private alias AliasSeq(T...) = T; version(with_breaking_cgi_features) mixin(q{ template ThisFor(alias t) { static if(__traits(isStaticFunction, t)) { alias ThisFor = AliasSeq!(); } else { alias ThisFor = __traits(parent, t); } } }); else alias ThisFor(alias t) = AliasSeq!(); private __gshared bool globalStopFlag = false; version(embedded_httpd_processes) void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { import core.sys.posix.unistd; import core.sys.posix.sys.socket; import core.sys.posix.netinet.in_; //import std.c.linux.socket; int sock = socket(AF_INET, SOCK_STREAM, 0); if(sock == -1) throw new Exception("socket"); cloexec(sock); { sockaddr_in addr; addr.sin_family = AF_INET; addr.sin_port = htons(params.listeningPort); auto lh = params.listeningHost; if(lh.length) { if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); } else addr.sin_addr.s_addr = INADDR_ANY; // HACKISH int on = 1; setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); // end hack if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { close(sock); throw new Exception("bind"); } // FIXME: if this queue is full, it will just ignore it // and wait for the client to retransmit it. This is an // obnoxious timeout condition there. if(sock.listen(128) == -1) { close(sock); throw new Exception("listen"); } params.dropPrivileges(); } version(embedded_httpd_processes_accept_after_fork) {} else { int pipeReadFd; int pipeWriteFd; { int[2] pipeFd; if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { import core.stdc.errno; throw new Exception("pipe failed " ~ to!string(errno)); } pipeReadFd = pipeFd[0]; pipeWriteFd = pipeFd[1]; } } int processCount; pid_t newPid; reopen: while(processCount < processPoolSize) { newPid = fork(); if(newPid == 0) { // start serving on the socket //ubyte[4096] backingBuffer; for(;;) { bool closeConnection; uint i; sockaddr addr; i = addr.sizeof; version(embedded_httpd_processes_accept_after_fork) { int s = accept(sock, &addr, &i); int opt = 1; import core.sys.posix.netinet.tcp; // the Cgi class does internal buffering, so disabling this // helps with latency in many cases... setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); cloexec(s); } else { int s; auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); if(readret != s.sizeof) { import core.stdc.errno; throw new Exception("pipe read failed " ~ to!string(errno)); } //writeln("process ", getpid(), " got socket ", s); } try { if(s == -1) throw new Exception("accept"); scope(failure) close(s); //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; auto ir = new BufferedInputRange(s); //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); while(!ir.empty) { //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; Cgi cgi; try { cgi = new CustomCgi(ir, &closeConnection); cgi._outputFileHandle = cast(CgiConnectionHandle) s; // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. if(processPoolSize <= 1) closeConnection = true; //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); } catch(Throwable t) { // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P // anyway let's kill the connection version(CRuntime_Musl) { // LockingTextWriter fails here // so working around it auto estr = t.toString(); stderr.rawWrite(estr); stderr.rawWrite("\n"); } else stderr.writeln(t.toString()); sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); closeConnection = true; break; } assert(cgi !is null); scope(exit) cgi.dispose(); try { fun(cgi); cgi.close(); if(cgi.websocketMode) closeConnection = true; } catch(ConnectionException ce) { closeConnection = true; } catch(Throwable t) { // a processing error can be recovered from version(CRuntime_Musl) { // LockingTextWriter fails here // so working around it auto estr = t.toString(); stderr.rawWrite(estr); } else { stderr.writeln(t.toString); } if(!handleException(cgi, t)) closeConnection = true; } if(closeConnection) { ir.source.close(); break; } else { if(!ir.empty) ir.popFront(); // get the next else if(ir.sourceClosed) { ir.source.close(); } } } ir.source.close(); } catch(Throwable t) { version(CRuntime_Musl) {} else debug writeln(t); // most likely cause is a timeout } } } else if(newPid < 0) { throw new Exception("fork failed"); } else { processCount++; } } // the parent should wait for its children... if(newPid) { import core.sys.posix.sys.wait; version(embedded_httpd_processes_accept_after_fork) {} else { import core.sys.posix.sys.select; int[] fdQueue; while(true) { // writeln("select call"); int nfds = pipeWriteFd; if(sock > pipeWriteFd) nfds = sock; nfds += 1; fd_set read_fds; fd_set write_fds; FD_ZERO(&read_fds); FD_ZERO(&write_fds); FD_SET(sock, &read_fds); if(fdQueue.length) FD_SET(pipeWriteFd, &write_fds); auto ret = select(nfds, &read_fds, &write_fds, null, null); if(ret == -1) { import core.stdc.errno; if(errno == EINTR) goto try_wait; else throw new Exception("wtf select"); } int s = -1; if(FD_ISSET(sock, &read_fds)) { uint i; sockaddr addr; i = addr.sizeof; s = accept(sock, &addr, &i); cloexec(s); import core.sys.posix.netinet.tcp; int opt = 1; setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); } if(FD_ISSET(pipeWriteFd, &write_fds)) { if(s == -1 && fdQueue.length) { s = fdQueue[0]; fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer } write_fd(pipeWriteFd, &s, s.sizeof, s); close(s); // we are done with it, let the other process take ownership } else fdQueue ~= s; } } try_wait: int status; while(-1 != wait(&status)) { version(CRuntime_Musl) {} else { import std.stdio; writeln("Process died ", status); } processCount--; goto reopen; } close(sock); } } version(fastcgi) void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { // SetHandler fcgid-script FCGX_Stream* input, output, error; FCGX_ParamArray env; const(ubyte)[] getFcgiChunk() { const(ubyte)[] ret; while(FCGX_HasSeenEOF(input) != -1) ret ~= cast(ubyte) FCGX_GetChar(input); return ret; } void writeFcgi(const(ubyte)[] data) { FCGX_PutStr(data.ptr, data.length, output); } void doARequest() { string[string] fcgienv; for(auto e = env; e !is null && *e !is null; e++) { string cur = to!string(*e); auto idx = cur.indexOf("="); string name, value; if(idx == -1) name = cur; else { name = cur[0 .. idx]; value = cur[idx + 1 .. $]; } fcgienv[name] = value; } void flushFcgi() { FCGX_FFlush(output); } Cgi cgi; try { cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); } catch(Throwable t) { FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); return; //continue; } assert(cgi !is null); scope(exit) cgi.dispose(); try { fun(cgi); cgi.close(); } catch(Throwable t) { // log it to the error stream FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); // handle it for the user, if we can if(!handleException(cgi, t)) return; // continue; } } auto lp = params.listeningPort; auto host = params.listeningHost; FCGX_Request request; if(lp || !host.empty) { // if a listening port was specified on the command line, we want to spawn ourself // (needed for nginx without spawn-fcgi, e.g. on Windows) FCGX_Init(); int sock; if(host.startsWith("unix:")) { sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); } else if(host.startsWith("abstract:")) { sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); } else { sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); } if(sock < 0) throw new Exception("Couldn't listen on the port"); FCGX_InitRequest(&request, sock, 0); while(FCGX_Accept_r(&request) >= 0) { input = request.inStream; output = request.outStream; error = request.errStream; env = request.envp; doARequest(); } } else { // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) // using the version with a global variable since we are separate processes anyway while(FCGX_Accept(&input, &output, &error, &env) >= 0) { doARequest(); } } } /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. ushort defaultListeningPort() { version(netman_httpd) return 8080; else version(embedded_httpd_processes) return 8085; else version(embedded_httpd_threads) return 8085; else version(scgi) return 4000; else return 0; } /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. string defaultListeningHost() { version(netman_httpd) return null; else version(embedded_httpd_processes) return null; else version(embedded_httpd_threads) return null; else version(scgi) return "127.0.0.1"; else return null; } /++ This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). Params: fun = Your request handler CustomCgi = a subclass of Cgi, if you wise to customize it further maxContentLength = max POST size you want to allow args = command-line arguments History: Documented Sept 26, 2020. +/ void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { if(tryAddonServers(args)) return; if(trySimulatedRequest!(fun, CustomCgi)(args)) return; RequestServer server; // you can change the port here if you like // server.listeningPort = 9000; // then call this to let the command line args override your default server.configureFromCommandLine(args); // and serve the request(s). server.serve!(fun, CustomCgi, maxContentLength)(); } //version(plain_cgi) void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { // standard CGI is the default version // Set stdin to binary mode if necessary to avoid mangled newlines // the fact that stdin is global means this could be trouble but standard cgi request // handling is one per process anyway so it shouldn't actually be threaded here or anything. version(Windows) { version(Win64) _setmode(std.stdio.stdin.fileno(), 0x8000); else setmode(std.stdio.stdin.fileno(), 0x8000); } Cgi cgi; try { cgi = new CustomCgi(maxContentLength); version(Posix) cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout else version(Windows) cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); else static assert(0); } catch(Throwable t) { version(CRuntime_Musl) { // LockingTextWriter fails here // so working around it auto s = t.toString(); stderr.rawWrite(s); stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); } else { stderr.writeln(t.msg); // the real http server will probably handle this; // most likely, this is a bug in Cgi. But, oh well. stdout.write(plainHttpError(true, "400 Bad Request", t)); } return; } assert(cgi !is null); scope(exit) cgi.dispose(); try { fun(cgi); cgi.close(); } catch (Throwable t) { version(CRuntime_Musl) { // LockingTextWriter fails here // so working around it auto s = t.msg; stderr.rawWrite(s); } else { stderr.writeln(t.msg); } if(!handleException(cgi, t)) return; } } private __gshared int cancelfd = -1; /+ The event loop for embedded_httpd_threads will prolly fiber dispatch cgi constructors too, so slow posts will not monopolize a worker thread. May want to provide the worker task system just need to ensure all the fibers has a big enough stack for real work... would also ideally like to reuse them. So prolly bir would switch it to nonblocking. If it would block, it epoll registers one shot with this existing fiber to take it over. new connection comes in. it picks a fiber off the free list, or if there is none, it creates a new one. this fiber handles this connection the whole time. epoll triggers the fiber when something comes in. it is called by a random worker thread, it might change at any time. at least during the constructor. maybe into the main body it will stay tied to a thread just so TLS stuff doesn't randomly change in the middle. but I could specify if you yield all bets are off. when the request is finished, if there's more data buffered, it just keeps going. if there is no more data buffered, it epoll ctls to get triggered when more data comes in. all one shot. when a connection is closed, the fiber returns and is then reset and added to the free list. if the free list is full, the fiber is just freed, this means it will balloon to a certain size but not generally grow beyond that unless the activity keeps going. 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. So the fiber has its own magic methods to read and write. if they would block, it registers for epoll and yields. when it returns, it read/writes and then returns back normal control. basically you issue the command and it tells you when it is done it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued +/ /++ The stack size when a fiber is created. You can set this from your main or from a shared static constructor to optimize your memory use if you know you don't need this much space. Be careful though, some functions use more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! History: Added July 10, 2021. Previously, it used the druntime default of 16 KB. +/ version(cgi_use_fiber) __gshared size_t fiberStackSize = 4096 * 100; version(cgi_use_fiber) class CgiFiber : Fiber { private void function(Socket) f_handler; private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function f_handler(s); } this(void function(Socket) handler) { this.f_handler = handler; this(&f_handler_dg); } this(void delegate(Socket) handler) { this.handler = handler; super(&run, fiberStackSize); } Socket connection; void delegate(Socket) handler; void run() { handler(connection); } void delegate() postYield; private void setPostYield(scope void delegate() py) @nogc { postYield = cast(void delegate()) py; } void proceed() { try { call(); auto py = postYield; postYield = null; if(py !is null) py(); } catch(Exception e) { if(connection) connection.close(); goto terminate; } if(state == State.TERM) { terminate: import core.memory; GC.removeRoot(cast(void*) this); } } } version(cgi_use_fiber) version(Windows) { extern(Windows) private { import core.sys.windows.mswsock; alias GROUP=uint; alias LPWSAPROTOCOL_INFOW = void*; SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); struct WSABUF { ULONG len; CHAR *buf; } alias LPWSABUF = WSABUF*; alias WSAOVERLAPPED = OVERLAPPED; alias LPWSAOVERLAPPED = LPOVERLAPPED; /+ alias LPFN_ACCEPTEX = BOOL function( SOCKET sListenSocket, SOCKET sAcceptSocket, //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, void* lpOutputBuffer, WORD dwReceiveDataLength, WORD dwLocalAddressLength, WORD dwRemoteAddressLength, LPDWORD lpdwBytesReceived, LPOVERLAPPED lpOverlapped ); enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); +/ enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); } private class PseudoblockingOverlappedSocket : Socket { SOCKET handle; CgiFiber fiber; this(AddressFamily af, SocketType st) { auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); if(!handle) throw new Exception("WSASocketW"); this.handle = handle; iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); if(iocp is null) { writeln(GetLastError()); throw new Exception("CreateIoCompletionPort"); } super(cast(socket_t) handle, af); } this() pure nothrow @trusted { assert(0); } override void blocking(bool) {} // meaningless to us, just ignore it. protected override Socket accepting() pure nothrow { assert(0); } bool addressesParsed; Address la; Address ra; private void populateAddresses() { if(addressesParsed) return; addressesParsed = true; int lalen, ralen; sockaddr_in* la; sockaddr_in* ra; lpfnGetAcceptExSockaddrs( scratchBuffer.ptr, 0, // same as in the AcceptEx call! sockaddr_in.sizeof + 16, sockaddr_in.sizeof + 16, cast(sockaddr**) &la, &lalen, cast(sockaddr**) &ra, &ralen ); if(la) this.la = new InternetAddress(*la); if(ra) this.ra = new InternetAddress(*ra); } override @property @trusted Address localAddress() { populateAddresses(); return la; } override @property @trusted Address remoteAddress() { populateAddresses(); return ra; } PseudoblockingOverlappedSocket accepted; __gshared static LPFN_ACCEPTEX lpfnAcceptEx; __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; override Socket accept() @trusted { __gshared static LPFN_ACCEPTEX lpfnAcceptEx; if(lpfnAcceptEx is null) { DWORD dwBytes; GUID GuidAcceptEx = WSAID_ACCEPTEX; auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, &GuidAcceptEx, GuidAcceptEx.sizeof, &lpfnAcceptEx, lpfnAcceptEx.sizeof, &dwBytes, null, null); GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, &GuidAcceptEx, GuidAcceptEx.sizeof, &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, &dwBytes, null, null); } auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); accepted = pfa; SOCKET pendingForAccept = pfa.handle; DWORD ignored; auto ret = lpfnAcceptEx(handle, pendingForAccept, // buffer to receive up front pfa.scratchBuffer.ptr, 0, // size of local and remote addresses. normally + 16. sockaddr_in.sizeof + 16, sockaddr_in.sizeof + 16, &ignored, // bytes would be given through the iocp instead but im not even requesting the thing &overlapped ); return pfa; } override void connect(Address to) { assert(0); } DWORD lastAnswer; ubyte[1024] scratchBuffer; static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); WSABUF[1] buffer; OVERLAPPED overlapped; override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { overlapped = overlapped.init; buffer[0].len = cast(DWORD) buf.length; buffer[0].buf = cast(CHAR*) buf.ptr; fiber.setPostYield( () { if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { if(GetLastError() != 997) { //throw new Exception("WSASend fail"); } } }); Fiber.yield(); return lastAnswer; } override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { overlapped = overlapped.init; buffer[0].len = cast(DWORD) buf.length; buffer[0].buf = cast(CHAR*) buf.ptr; DWORD flags2 = 0; fiber.setPostYield(() { if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { if(GetLastError() != 997) { //writeln("WSARecv ", WSAGetLastError()); //throw new Exception("WSARecv fail"); } } }); Fiber.yield(); return lastAnswer; } // I might go back and implement these for udp things. override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { assert(0); } override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { assert(0); } override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { assert(0); } override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { assert(0); } // lol overload sets alias send = typeof(super).send; alias receive = typeof(super).receive; alias sendTo = typeof(super).sendTo; alias receiveFrom = typeof(super).receiveFrom; } } void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { assert(connection !is null); version(cgi_use_fiber) { auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); version(Windows) { (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; } import core.memory; GC.addRoot(cast(void*) fiber); fiber.connection = connection; fiber.proceed(); } else { doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); } } void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { scope(failure) { // catch all for other errors try { sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); connection.close(); } catch(Exception e) {} // swallow it, we're aborting anyway. } bool closeConnection = alwaysCloseConnection; /+ ubyte[4096] inputBuffer = void; ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; ir.__ctor(connection, inputBuffer[], true); +/ auto ir = new BufferedInputRange(connection); while(!ir.empty) { if(ir.view.length == 0) { ir.popFront(); if(ir.sourceClosed) { connection.close(); closeConnection = true; break; } } Cgi cgi; try { cgi = new CustomCgi(ir, &closeConnection); // There's a bunch of these casts around because the type matches up with // the -version=.... specifiers, just you can also create a RequestServer // and instantiate the things where the types don't match up. It isn't exactly // correct but I also don't care rn. Might FIXME and either remove it later or something. cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; } catch(ConnectionClosedException ce) { closeConnection = true; break; } catch(ConnectionException ce) { // broken pipe or something, just abort the connection closeConnection = true; break; } catch(Throwable t) { // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P // anyway let's kill the connection version(CRuntime_Musl) { stderr.rawWrite(t.toString()); stderr.rawWrite("\n"); } else { stderr.writeln(t.toString()); } sendAll(connection, plainHttpError(false, "400 Bad Request", t)); closeConnection = true; break; } assert(cgi !is null); scope(exit) cgi.dispose(); try { fun(cgi); cgi.close(); if(cgi.websocketMode) closeConnection = true; } catch(ConnectionException ce) { // broken pipe or something, just abort the connection closeConnection = true; } catch(ConnectionClosedException ce) { // broken pipe or something, just abort the connection closeConnection = true; } catch(Throwable t) { // a processing error can be recovered from version(CRuntime_Musl) {} else stderr.writeln(t.toString); if(!handleException(cgi, t)) closeConnection = true; } if(globalStopFlag) closeConnection = true; if(closeConnection || alwaysCloseConnection) { connection.shutdown(SocketShutdown.BOTH); connection.close(); ir.dispose(); closeConnection = false; // don't reclose after loop break; } else { if(ir.front.length) { ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along } else if(ir.sourceClosed) { ir.source.shutdown(SocketShutdown.BOTH); ir.source.close(); ir.dispose(); closeConnection = false; } else { continue; // break; // this was for a keepalive experiment } } } if(closeConnection) { connection.shutdown(SocketShutdown.BOTH); connection.close(); ir.dispose(); } // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! } void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { // and now we can buffer scope(failure) connection.close(); import al = std.algorithm; size_t size; string[string] headers; auto range = new BufferedInputRange(connection); more_data: auto chunk = range.front(); // waiting for colon for header length auto idx = indexOf(cast(string) chunk, ':'); if(idx == -1) { try { range.popFront(); } catch(Exception e) { // it is just closed, no big deal connection.close(); return; } goto more_data; } size = to!size_t(cast(string) chunk[0 .. idx]); chunk = range.consume(idx + 1); // reading headers if(chunk.length < size) range.popFront(0, size + 1); // we are now guaranteed to have enough chunk = range.front(); assert(chunk.length > size); idx = 0; string key; string value; foreach(part; al.splitter(chunk, '\0')) { if(idx & 1) { // odd is value value = cast(string)(part.idup); headers[key] = value; // commit } else key = cast(string)(part.idup); idx++; } enforce(chunk[size] == ','); // the terminator range.consume(size + 1); // reading data // this will be done by Cgi const(ubyte)[] getScgiChunk() { // we are already primed auto data = range.front(); if(data.length == 0 && !range.sourceClosed) { range.popFront(0); data = range.front(); } else if (range.sourceClosed) range.source.close(); return data; } void writeScgi(const(ubyte)[] data) { sendAll(connection, data); } void flushScgi() { // I don't *think* I have to do anything.... } Cgi cgi; try { cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; } catch(Throwable t) { sendAll(connection, plainHttpError(true, "400 Bad Request", t)); connection.close(); return; // this connection is dead } assert(cgi !is null); scope(exit) cgi.dispose(); try { fun(cgi); cgi.close(); connection.close(); } catch(Throwable t) { // no std err if(!handleException(cgi, t)) { connection.close(); return; } else { connection.close(); return; } } } string printDate(DateTime date) { char[29] buffer = void; printDateToBuffer(date, buffer[]); return buffer.idup; } int printDateToBuffer(DateTime date, char[] buffer) @nogc { assert(buffer.length >= 29); // 29 static length ? static immutable daysOfWeek = [ "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" ]; static immutable months = [ null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" ]; buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; buffer[3 .. 5] = ", "; buffer[5] = date.day / 10 + '0'; buffer[6] = date.day % 10 + '0'; buffer[7] = ' '; buffer[8 .. 11] = months[date.month]; buffer[11] = ' '; auto y = date.year; buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; buffer[13] = cast(char) (y / 100 + '0'); y %= 100; buffer[14] = cast(char) (y / 10 + '0'); y %= 10; buffer[15] = cast(char) (y + '0'); buffer[16] = ' '; buffer[17] = date.hour / 10 + '0'; buffer[18] = date.hour % 10 + '0'; buffer[19] = ':'; buffer[20] = date.minute / 10 + '0'; buffer[21] = date.minute % 10 + '0'; buffer[22] = ':'; buffer[23] = date.second / 10 + '0'; buffer[24] = date.second % 10 + '0'; buffer[25 .. $] = " GMT"; return 29; } // Referencing this gigantic typeid seems to remind the compiler // to actually put the symbol in the object file. I guess the immutable // assoc array array isn't actually included in druntime void hackAroundLinkerError() { stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); // this is getting kinda ridiculous btw. Moving assoc arrays // to the library is the pain that keeps on coming. // eh this broke the build on the work server // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); } version(fastcgi) { pragma(lib, "fcgi"); static if(size_t.sizeof == 8) // 64 bit alias long c_int; else alias int c_int; extern(C) { struct FCGX_Stream { ubyte* rdNext; ubyte* wrNext; ubyte* stop; ubyte* stopUnget; c_int isReader; c_int isClosed; c_int wasFCloseCalled; c_int FCGI_errno; void* function(FCGX_Stream* stream) fillBuffProc; void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; void* data; } // note: this is meant to be opaque, so don't access it directly struct FCGX_Request { int requestId; int role; FCGX_Stream* inStream; FCGX_Stream* outStream; FCGX_Stream* errStream; char** envp; void* paramsPtr; int ipcFd; int isBeginProcessed; int keepConnection; int appStatus; int nWriters; int flags; int listen_sock; } int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); void FCGX_Init(); int FCGX_Accept_r(FCGX_Request *request); alias char** FCGX_ParamArray; c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); c_int FCGX_GetChar(FCGX_Stream* stream); c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); int FCGX_HasSeenEOF(FCGX_Stream* stream); c_int FCGX_FFlush(FCGX_Stream *stream); int FCGX_OpenSocket(in char*, int); } } /* This might go int a separate module eventually. It is a network input helper class. */ import std.socket; version(cgi_use_fiber) { import core.thread; version(linux) { import core.sys.linux.epoll; int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. } else version(Windows) { // declaring the iocp thing below... } else static assert(0, "The hybrid fiber server is not implemented on your OS."); } version(Windows) __gshared HANDLE iocp; version(cgi_use_fiber) { version(linux) private enum WakeupEvent { Read = EPOLLIN, Write = EPOLLOUT } else version(Windows) private enum WakeupEvent { Read, Write } else static assert(0); } version(cgi_use_fiber) private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { // static cast since I know what i have in here and don't want to pay for dynamic cast auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); version(linux) { f.setPostYield = () { if(*registered) { // rearm epoll_event evt; evt.events = e | EPOLLONESHOT; evt.data.ptr = cast(void*) f; if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) throw new Exception("epoll_ctl"); } else { // initial registration *registered = true ; int fd = source.handle; epoll_event evt; evt.events = e | EPOLLONESHOT; evt.data.ptr = cast(void*) f; if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) throw new Exception("epoll_ctl"); } }; Fiber.yield(); f.setPostYield(null); } else version(Windows) { Fiber.yield(); } else static assert(0); } version(cgi_use_fiber) void unregisterSource(Socket s) { version(linux) { epoll_event evt; epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); } else version(Windows) { // intentionally blank } else static assert(0); } // it is a class primarily for reference semantics // I might change this interface /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. class BufferedInputRange { version(Posix) this(int source, ubyte[] buffer = null) { this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); } this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { // if they connect but never send stuff to us, we don't want it wasting the process // so setting a time out version(cgi_use_fiber) source.blocking = false; else source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); this.source = source; if(buffer is null) { underlyingBuffer = new ubyte[4096]; this.allowGrowth = true; } else { underlyingBuffer = buffer; this.allowGrowth = allowGrowth; } assert(underlyingBuffer.length); // we assume view.ptr is always inside underlyingBuffer view = underlyingBuffer[0 .. 0]; popFront(); // prime } version(cgi_use_fiber) { bool registered; } void dispose() { version(cgi_use_fiber) { if(registered) unregisterSource(source); } } /** A slight difference from regular ranges is you can give it the maximum number of bytes to consume. IMPORTANT NOTE: the default is to consume nothing, so if you don't call consume() yourself and use a regular foreach, it will infinitely loop! The default is to do what a normal range does, and consume the whole buffer and wait for additional input. You can also specify 0, to append to the buffer, or any other number to remove the front n bytes and wait for more. */ void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { if(sourceClosed) throw new ConnectionClosedException("can't get any more data from a closed source"); if(!skipConsume) consume(maxBytesToConsume); // we might have to grow the buffer if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { if(allowGrowth) { //import std.stdio; writeln("growth"); auto viewStart = view.ptr - underlyingBuffer.ptr; size_t growth = 4096; // make sure we have enough for what we're being asked for if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) growth = minBytesToSettleFor - underlyingBuffer.length; //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); underlyingBuffer.length += growth; view = underlyingBuffer[viewStart .. view.length]; } else throw new Exception("No room left in the buffer"); } do { auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; try_again: auto ret = source.receive(freeSpace); if(ret == Socket.ERROR) { if(wouldHaveBlocked()) { version(cgi_use_fiber) { registerEventWakeup(®istered, source, WakeupEvent.Read); goto try_again; } else { // gonna treat a timeout here as a close sourceClosed = true; return; } } version(Posix) { import core.stdc.errno; if(errno == EINTR || errno == EAGAIN) { goto try_again; } if(errno == ECONNRESET) { sourceClosed = true; return; } } throw new Exception(lastSocketError); // FIXME } if(ret == 0) { sourceClosed = true; return; } //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; //import std.stdio; writeln(cast(string) view); } while(view.length < minBytesToSettleFor); } /// Removes n bytes from the front of the buffer, and returns the new buffer slice. /// You might want to idup the data you are consuming if you store it, since it may /// be overwritten on the new popFront. /// /// You do not need to call this if you always want to wait for more data when you /// consume some. ubyte[] consume(size_t bytes) { view = view[bytes > $ ? $ : bytes .. $]; if(view.length == 0) { view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning /* writeln("HERE"); popFront(0, 0, true); // try to load more if we can, checks if the source is closed writeln(cast(string)front); writeln("DONE"); */ } return front; } bool empty() { return sourceClosed && view.length == 0; } ubyte[] front() { return view; } invariant() { assert(view.ptr >= underlyingBuffer.ptr); // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); } ubyte[] underlyingBuffer; bool allowGrowth; ubyte[] view; Socket source; bool sourceClosed; } private class FakeSocketForStdin : Socket { import std.stdio; this() { } private bool closed; override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { if(closed) throw new Exception("Closed"); return stdin.rawRead(buffer).length; } override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { if(closed) throw new Exception("Closed"); stdout.rawWrite(buffer); return buffer.length; } override void close() @trusted scope { (cast(void delegate() @nogc nothrow) &realClose)(); } override void shutdown(SocketShutdown s) { // FIXME } override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} override void setOption(SocketOptionLevel, SocketOption, Duration) {} override @property @trusted Address remoteAddress() { return null; } override @property @trusted Address localAddress() { return null; } void realClose() { closed = true; try { stdin.close(); stdout.close(); } catch(Exception e) { } } } import core.sync.semaphore; import core.atomic; /** To use this thing: --- void handler(Socket s) { do something... } auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); manager.listen(); --- The 4th parameter is optional. I suggest you use BufferedInputRange(connection) to handle the input. As a packet comes in, you will get control. You can just continue; though to fetch more. FIXME: should I offer an event based async thing like netman did too? Yeah, probably. */ class ListeningConnectionManager { Semaphore semaphore; Socket[256] queue; shared(ubyte) nextIndexFront; ubyte nextIndexBack; shared(int) queueLength; Socket acceptCancelable() { version(Posix) { import core.sys.posix.sys.select; fd_set read_fds; FD_ZERO(&read_fds); FD_SET(listener.handle, &read_fds); if(cancelfd != -1) FD_SET(cancelfd, &read_fds); auto max = listener.handle > cancelfd ? listener.handle : cancelfd; auto ret = select(max + 1, &read_fds, null, null, null); if(ret == -1) { import core.stdc.errno; if(errno == EINTR) return null; else throw new Exception("wtf select"); } if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { return null; } if(FD_ISSET(listener.handle, &read_fds)) return listener.accept(); return null; } else { Socket socket = listener; auto check = new SocketSet(); keep_looping: check.reset(); check.add(socket); // just to check the stop flag on a kinda busy loop. i hate this FIXME auto got = Socket.select(check, null, null, 3.seconds); if(got > 0) return listener.accept(); if(globalStopFlag) return null; else goto keep_looping; } } int defaultNumberOfThreads() { import std.parallelism; version(cgi_use_fiber) { return totalCPUs * 1 + 1; } else { // I times 4 here because there's a good chance some will be blocked on i/o. return totalCPUs * 4; } } void listen() { shared(int) loopBroken; version(Posix) { import core.sys.posix.signal; signal(SIGPIPE, SIG_IGN); } version(linux) { if(cancelfd == -1) cancelfd = eventfd(0, 0); } version(cgi_no_threads) { // NEVER USE THIS // it exists only for debugging and other special occasions // the thread mode is faster and less likely to stall the whole // thing when a request is slow while(!loopBroken && !globalStopFlag) { auto sn = acceptCancelable(); if(sn is null) continue; cloexec(sn); try { handler(sn); } catch(Exception e) { // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) sn.close(); } } } else { if(useFork) { version(linux) { //asm { int 3; } fork(); } } version(cgi_use_fiber) { version(Windows) { listener.accept(); } WorkerThread[] threads = new WorkerThread[](numberOfThreads); foreach(i, ref thread; threads) { thread = new WorkerThread(this, handler, cast(int) i); thread.start(); } bool fiber_crash_check() { bool hasAnyRunning; foreach(thread; threads) { if(!thread.isRunning) { thread.join(); } else hasAnyRunning = true; } return (!hasAnyRunning); } while(!globalStopFlag) { Thread.sleep(1.seconds); if(fiber_crash_check()) break; } } else { semaphore = new Semaphore(); ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); foreach(i, ref thread; threads) { thread = new ConnectionThread(this, handler, cast(int) i); thread.start(); } while(!loopBroken && !globalStopFlag) { Socket sn; bool crash_check() { bool hasAnyRunning; foreach(thread; threads) { if(!thread.isRunning) { thread.join(); } else hasAnyRunning = true; } return (!hasAnyRunning); } void accept_new_connection() { sn = acceptCancelable(); if(sn is null) return; cloexec(sn); if(tcp) { // disable Nagle's algorithm to avoid a 40ms delay when we send/recv // on the socket because we do some buffering internally. I think this helps, // certainly does for small requests, and I think it does for larger ones too sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); } } void existing_connection_new_data() { // wait until a slot opens up //int waited = 0; while(queueLength >= queue.length) { Thread.sleep(1.msecs); //waited ++; } //if(waited) {import std.stdio; writeln(waited);} synchronized(this) { queue[nextIndexBack] = sn; nextIndexBack++; atomicOp!"+="(queueLength, 1); } semaphore.notify(); } accept_new_connection(); if(sn !is null) existing_connection_new_data(); else if(sn is null && globalStopFlag) { foreach(thread; threads) { semaphore.notify(); } Thread.sleep(50.msecs); } if(crash_check()) break; } } // FIXME: i typically stop this with ctrl+c which never // actually gets here. i need to do a sigint handler. if(cleanup) cleanup(); } } //version(linux) //int epoll_fd; bool tcp; void delegate() cleanup; private void function(Socket) fhandler; private void dg_handler(Socket s) { fhandler(s); } this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { fhandler = handler; this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); } this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { this.handler = handler; this.useFork = useFork; this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); listener = startListening(host, port, tcp, cleanup, 128, dropPrivs); version(cgi_use_fiber) if(useFork) listener.blocking = false; // this is the UI control thread and thus gets more priority Thread.getThis.priority = Thread.PRIORITY_MAX; } Socket listener; void delegate(Socket) handler; immutable bool useFork; int numberOfThreads; } Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { Socket listener; if(host.startsWith("unix:")) { version(Posix) { listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); cloexec(listener); string filename = host["unix:".length .. $].idup; listener.bind(new UnixAddress(filename)); cleanup = delegate() { listener.close(); import std.file; remove(filename); }; tcp = false; } else { throw new Exception("unix sockets not supported on this system"); } } else if(host.startsWith("abstract:")) { version(linux) { listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); cloexec(listener); string filename = "\0" ~ host["abstract:".length .. $]; import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); listener.bind(new UnixAddress(filename)); tcp = false; } else { throw new Exception("abstract unix sockets not supported on this system"); } } else { version(cgi_use_fiber) { version(Windows) listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); else listener = new TcpSocket(); } else { listener = new TcpSocket(); } cloexec(listener); listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); listener.bind(host.length ? parseAddress(host, port) : new InternetAddress(port)); cleanup = delegate() { listener.close(); }; tcp = true; } listener.listen(backQueue); if (dropPrivs !is null) // can be null, backwards compatibility dropPrivs(); return listener; } // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { if(data.length == 0) return; ptrdiff_t amount; //import std.stdio; writeln("***",cast(string) data,"///"); do { amount = s.send(data); if(amount == Socket.ERROR) { version(cgi_use_fiber) { if(wouldHaveBlocked()) { bool registered = true; registerEventWakeup(®istered, s, WakeupEvent.Write); continue; } } throw new ConnectionException(s, lastSocketError, file, line); } assert(amount > 0); data = data[amount .. $]; } while(data.length); } class ConnectionException : Exception { Socket socket; this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { this.socket = s; super(msg, file, line); } } alias void delegate(Socket) CMT; import core.thread; /+ cgi.d now uses a hybrid of event i/o and threads at the top level. Top level thread is responsible for accepting sockets and selecting on them. It then indicates to a child that a request is pending, and any random worker thread that is free handles it. It goes into blocking mode and handles that http request to completion. At that point, it goes back into the waiting queue. This concept is only implemented on Linux. On all other systems, it still uses the worker threads and semaphores (which is perfectly fine for a lot of things! Just having a great number of keep-alive connections will break that.) So the algorithm is: select(accept, event, pending) if accept -> send socket to free thread, if any. if not, add socket to queue if event -> send the signaling thread a socket from the queue, if not, mark it free - event might block until it can be *written* to. it is a fifo sending socket fds! A worker only does one http request at a time, then signals its availability back to the boss. The socket the worker was just doing should be added to the one-off epoll read. If it is closed, great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the actual FD will not be kept out here. So: queue = sockets we know are ready to read now, but no worker thread is available idle list = worker threads not doing anything else. they signal back and forth the workers all read off the event fd. This is the semaphore wait the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, it puts it in the queue and writes to the event fd. The child could put the socket back in the epoll thing itself. The child needs to be able to gracefully handle being given a socket that just closed with no work. +/ class ConnectionThread : Thread { this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { this.lcm = lcm; this.dg = dg; this.myThreadNumber = myThreadNumber; super(&run); } void run() { while(true) { // so if there's a bunch of idle keep-alive connections, it can // consume all the worker threads... just sitting there. lcm.semaphore.wait(); if(globalStopFlag) return; Socket socket; synchronized(lcm) { auto idx = lcm.nextIndexFront; socket = lcm.queue[idx]; lcm.queue[idx] = null; atomicOp!"+="(lcm.nextIndexFront, 1); atomicOp!"-="(lcm.queueLength, 1); } try { //import std.stdio; writeln(myThreadNumber, " taking it"); dg(socket); /+ if(socket.isAlive) { // process it more later version(linux) { import core.sys.linux.epoll; epoll_event ev; ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; ev.data.fd = socket.handle; import std.stdio; writeln("adding"); if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { if(errno == EEXIST) { ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; ev.data.fd = socket.handle; if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) throw new Exception("epoll_ctl " ~ to!string(errno)); } else throw new Exception("epoll_ctl " ~ to!string(errno)); } //import std.stdio; writeln("keep alive"); // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later __traits(getMember, socket, "sock") = cast(socket_t) -1; } else { continue; // hope it times out in a reasonable amount of time... } } +/ } catch(ConnectionClosedException e) { // can just ignore this, it is fairly normal socket.close(); } catch(Throwable e) { import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); socket.close(); } } } ListeningConnectionManager lcm; CMT dg; int myThreadNumber; } version(cgi_use_fiber) class WorkerThread : Thread { this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { this.lcm = lcm; this.dg = dg; this.myThreadNumber = myThreadNumber; super(&run); } version(Windows) void run() { auto timeout = INFINITE; PseudoblockingOverlappedSocket key; OVERLAPPED* overlapped; DWORD bytes; while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { if(key is null) continue; key.lastAnswer = bytes; if(key.fiber) { key.fiber.proceed(); } else { // we have a new connection, issue the first receive on it and issue the next accept auto sn = key.accepted; key.accept(); cloexec(sn); if(lcm.tcp) { // disable Nagle's algorithm to avoid a 40ms delay when we send/recv // on the socket because we do some buffering internally. I think this helps, // certainly does for small requests, and I think it does for larger ones too sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); } dg(sn); } } //SleepEx(INFINITE, TRUE); } version(linux) void run() { import core.sys.linux.epoll; epfd = epoll_create1(EPOLL_CLOEXEC); if(epfd == -1) throw new Exception("epoll_create1 " ~ to!string(errno)); scope(exit) { import core.sys.posix.unistd; close(epfd); } { epoll_event ev; ev.events = EPOLLIN; ev.data.fd = cancelfd; epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); } epoll_event ev; ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. ev.data.fd = lcm.listener.handle; if(epoll_ctl(epfd, EPOLL_CTL_ADD, lcm.listener.handle, &ev) == -1) throw new Exception("epoll_ctl " ~ to!string(errno)); while(!globalStopFlag) { Socket sn; epoll_event[64] events; auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); if(nfds == -1) { if(errno == EINTR) continue; throw new Exception("epoll_wait " ~ to!string(errno)); } foreach(idx; 0 .. nfds) { auto flags = events[idx].events; if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { globalStopFlag = true; //import std.stdio; writeln("exit heard"); break; } else if(cast(size_t) events[idx].data.ptr == cast(size_t) lcm.listener.handle) { //import std.stdio; writeln(myThreadNumber, " woken up ", flags); // this try/catch is because it is set to non-blocking mode // and Phobos' stupid api throws an exception instead of returning // if it would block. Why would it block? because a forked process // might have beat us to it, but the wakeup event thundered our herds. try sn = lcm.listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better catch(SocketAcceptException e) { continue; } cloexec(sn); if(lcm.tcp) { // disable Nagle's algorithm to avoid a 40ms delay when we send/recv // on the socket because we do some buffering internally. I think this helps, // certainly does for small requests, and I think it does for larger ones too sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); } dg(sn); } else { if(cast(size_t) events[idx].data.ptr < 1024) { throw new Exception("this doesn't look like a fiber pointer..."); } auto fiber = cast(CgiFiber) events[idx].data.ptr; fiber.proceed(); } } } } ListeningConnectionManager lcm; CMT dg; int myThreadNumber; } /* Done with network helper */ /* Helpers for doing temporary files. Used both here and in web.d */ version(Windows) { import core.sys.windows.windows; extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); alias GetTempPathW GetTempPath; } version(Posix) { static import linux = core.sys.posix.unistd; } string getTempDirectory() { string path; version(Windows) { wchar[1024] buffer; auto len = GetTempPath(1024, buffer.ptr); if(len == 0) throw new Exception("couldn't find a temporary path"); auto b = buffer[0 .. len]; path = to!string(b); } else path = "/tmp/"; return path; } // I like std.date. These functions help keep my old code and data working with phobos changing. long sysTimeToDTime(in SysTime sysTime) { return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); } long dateTimeToDTime(in DateTime dt) { return sysTimeToDTime(cast(SysTime) dt); } long getUtcTime() { // renamed primarily to avoid conflict with std.date itself return sysTimeToDTime(Clock.currTime(UTC())); } // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; return SysTime(hnsecs, tz); } // this is a helper to read HTTP transfer-encoding: chunked responses immutable(ubyte[]) dechunk(BufferedInputRange ir) { immutable(ubyte)[] ret; another_chunk: // If here, we are at the beginning of a chunk. auto a = ir.front(); int chunkSize; int loc = locationOf(a, "\r\n"); while(loc == -1) { ir.popFront(); a = ir.front(); loc = locationOf(a, "\r\n"); } string hex; hex = ""; for(int i = 0; i < loc; i++) { char c = a[i]; if(c >= 'A' && c <= 'Z') c += 0x20; if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { hex ~= c; } else { break; } } assert(hex.length); int power = 1; int size = 0; foreach(cc1; retro(hex)) { dchar cc = cc1; if(cc >= 'a' && cc <= 'z') cc -= 0x20; int val = 0; if(cc >= '0' && cc <= '9') val = cc - '0'; else val = cc - 'A' + 10; size += power * val; power *= 16; } chunkSize = size; assert(size >= 0); if(loc + 2 > a.length) { ir.popFront(0, a.length + loc + 2); a = ir.front(); } a = ir.consume(loc + 2); if(chunkSize == 0) { // we're done with the response // if we got here, will change must be true.... more_footers: loc = locationOf(a, "\r\n"); if(loc == -1) { ir.popFront(); a = ir.front; goto more_footers; } else { assert(loc == 0); ir.consume(loc + 2); goto finish; } } else { // if we got here, will change must be true.... if(a.length < chunkSize + 2) { ir.popFront(0, chunkSize + 2); a = ir.front(); } ret ~= (a[0..chunkSize]); if(!(a.length > chunkSize + 2)) { ir.popFront(0, chunkSize + 2); a = ir.front(); } assert(a[chunkSize] == 13); assert(a[chunkSize+1] == 10); a = ir.consume(chunkSize + 2); chunkSize = 0; goto another_chunk; } finish: return ret; } // I want to be able to get data from multiple sources the same way... interface ByChunkRange { bool empty(); void popFront(); const(ubyte)[] front(); } ByChunkRange byChunk(const(ubyte)[] data) { return new class ByChunkRange { override bool empty() { return !data.length; } override void popFront() { if(data.length > 4096) data = data[4096 .. $]; else data = null; } override const(ubyte)[] front() { return data[0 .. $ > 4096 ? 4096 : $]; } }; } ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { const(ubyte)[] f; f = ir.front; if(f.length > atMost) f = f[0 .. atMost]; return new class ByChunkRange { override bool empty() { return atMost == 0; } override const(ubyte)[] front() { return f; } override void popFront() { ir.consume(f.length); atMost -= f.length; auto a = ir.front(); if(a.length <= atMost) { f = a; atMost -= a.length; a = ir.consume(a.length); if(atMost != 0) ir.popFront(); if(f.length == 0) { f = ir.front(); } } else { // we actually have *more* here than we need.... f = a[0..atMost]; atMost = 0; ir.consume(atMost); } } }; } version(cgi_with_websocket) { // https://tools.ietf.org/html/rfc6455 /** WEBSOCKET SUPPORT: Full example: --- import arsd.cgi; void websocketEcho(Cgi cgi) { if(cgi.websocketRequested()) { if(cgi.origin != "http://arsdnet.net") throw new Exception("bad origin"); auto websocket = cgi.acceptWebsocket(); websocket.send("hello"); websocket.send(" world!"); auto msg = websocket.recv(); while(msg.opcode != WebSocketOpcode.close) { if(msg.opcode == WebSocketOpcode.text) { websocket.send(msg.textData); } else if(msg.opcode == WebSocketOpcode.binary) { websocket.send(msg.data); } msg = websocket.recv(); } websocket.close(); } else assert(0, "i want a web socket!"); } mixin GenericMain!websocketEcho; --- */ class WebSocket { Cgi cgi; private this(Cgi cgi) { this.cgi = cgi; Socket socket = cgi.idlol.source; socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); } // returns true if data available, false if it timed out bool recvAvailable(Duration timeout = dur!"msecs"(0)) { if(!waitForNextMessageWouldBlock()) return true; if(isDataPending(timeout)) return true; // this is kinda a lie. return false; } public bool lowLevelReceive() { auto bfr = cgi.idlol; top: auto got = bfr.front; if(got.length) { if(receiveBuffer.length < receiveBufferUsedLength + got.length) receiveBuffer.length += receiveBufferUsedLength + got.length; receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; receiveBufferUsedLength += got.length; bfr.consume(got.length); return true; } if(bfr.sourceClosed) return false; bfr.popFront(0); if(bfr.sourceClosed) return false; goto top; } bool isDataPending(Duration timeout = 0.seconds) { Socket socket = cgi.idlol.source; auto check = new SocketSet(); check.add(socket); auto got = Socket.select(check, null, null, timeout); if(got > 0) return true; return false; } // note: this blocks WebSocketFrame recv() { return waitForNextMessage(); } private void llclose() { cgi.close(); } private void llsend(ubyte[] data) { cgi.write(data); cgi.flush(); } void unregisterActiveSocket(WebSocket) {} /* copy/paste section { */ private int readyState_; private ubyte[] receiveBuffer; private size_t receiveBufferUsedLength; private Config config; enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. enum OPEN = 1; /// The connection is open and ready to communicate. enum CLOSING = 2; /// The connection is in the process of closing. enum CLOSED = 3; /// The connection is closed or couldn't be opened. /++ +/ /// Group: foundational static struct Config { /++ These control the size of the receive buffer. It starts at the initial size, will temporarily balloon up to the maximum size, and will reuse a buffer up to the likely size. Anything larger than the maximum size will cause the connection to be aborted and an exception thrown. This is to protect you against a peer trying to exhaust your memory, while keeping the user-level processing simple. +/ size_t initialReceiveBufferSize = 4096; size_t likelyReceiveBufferSize = 4096; /// ditto size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto /++ Maximum combined size of a message. +/ size_t maximumMessageSize = 10 * 1024 * 1024; string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; string origin; /// Origin URL to send with the handshake, if desired. string protocol; /// the protocol header, if desired. int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping } /++ Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. +/ int readyState() { return readyState_; } /++ Closes the connection, sending a graceful teardown message to the other side. +/ /// Group: foundational void close(int code = 0, string reason = null) //in (reason.length < 123) in { assert(reason.length < 123); } do { if(readyState_ != OPEN) return; // it cool, we done WebSocketFrame wss; wss.fin = true; wss.opcode = WebSocketOpcode.close; wss.data = cast(ubyte[]) reason.dup; wss.send(&llsend); readyState_ = CLOSING; llclose(); } /++ Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. +/ /// Group: foundational void ping() { WebSocketFrame wss; wss.fin = true; wss.opcode = WebSocketOpcode.ping; wss.send(&llsend); } // automatically handled.... void pong() { WebSocketFrame wss; wss.fin = true; wss.opcode = WebSocketOpcode.pong; wss.send(&llsend); } /++ Sends a text message through the websocket. +/ /// Group: foundational void send(in char[] textData) { WebSocketFrame wss; wss.fin = true; wss.opcode = WebSocketOpcode.text; wss.data = cast(ubyte[]) textData.dup; wss.send(&llsend); } /++ Sends a binary message through the websocket. +/ /// Group: foundational void send(in ubyte[] binaryData) { WebSocketFrame wss; wss.fin = true; wss.opcode = WebSocketOpcode.binary; wss.data = cast(ubyte[]) binaryData.dup; wss.send(&llsend); } /++ Waits for and returns the next complete message on the socket. Note that the onmessage function is still called, right before this returns. +/ /// Group: blocking_api public WebSocketFrame waitForNextMessage() { do { auto m = processOnce(); if(m.populated) return m; } while(lowLevelReceive()); throw new ConnectionClosedException("Websocket receive timed out"); //return WebSocketFrame.init; // FIXME? maybe. } /++ Tells if [waitForNextMessage] would block. +/ /// Group: blocking_api public bool waitForNextMessageWouldBlock() { checkAgain: if(isMessageBuffered()) return false; if(!isDataPending()) return true; while(isDataPending()) lowLevelReceive(); goto checkAgain; } /++ Is there a message in the buffer already? If `true`, [waitForNextMessage] is guaranteed to return immediately. If `false`, check [isDataPending] as the next step. +/ /// Group: blocking_api public bool isMessageBuffered() { ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; auto s = d; if(d.length) { auto orig = d; auto m = WebSocketFrame.read(d); // that's how it indicates that it needs more data if(d !is orig) return true; } return false; } private ubyte continuingType; private ubyte[] continuingData; //private size_t continuingDataLength; private WebSocketFrame processOnce() { ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; auto s = d; // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. WebSocketFrame m; if(d.length) { auto orig = d; m = WebSocketFrame.read(d); // that's how it indicates that it needs more data if(d is orig) return WebSocketFrame.init; m.unmaskInPlace(); switch(m.opcode) { case WebSocketOpcode.continuation: if(continuingData.length + m.data.length > config.maximumMessageSize) throw new Exception("message size exceeded"); continuingData ~= m.data; if(m.fin) { if(ontextmessage) ontextmessage(cast(char[]) continuingData); if(onbinarymessage) onbinarymessage(continuingData); continuingData = null; } break; case WebSocketOpcode.text: if(m.fin) { if(ontextmessage) ontextmessage(m.textData); } else { continuingType = m.opcode; //continuingDataLength = 0; continuingData = null; continuingData ~= m.data; } break; case WebSocketOpcode.binary: if(m.fin) { if(onbinarymessage) onbinarymessage(m.data); } else { continuingType = m.opcode; //continuingDataLength = 0; continuingData = null; continuingData ~= m.data; } break; case WebSocketOpcode.close: readyState_ = CLOSED; if(onclose) onclose(); unregisterActiveSocket(this); break; case WebSocketOpcode.ping: pong(); break; case WebSocketOpcode.pong: // just really references it is still alive, nbd. break; default: // ignore though i could and perhaps should throw too } } // the recv thing can be invalidated so gotta copy it over ugh if(d.length) { m.data = m.data.dup(); } import core.stdc.string; memmove(receiveBuffer.ptr, d.ptr, d.length); receiveBufferUsedLength = d.length; return m; } private void autoprocess() { // FIXME do { processOnce(); } while(lowLevelReceive()); } void delegate() onclose; /// void delegate() onerror; /// void delegate(in char[]) ontextmessage; /// void delegate(in ubyte[]) onbinarymessage; /// void delegate() onopen; /// /++ +/ /// Group: browser_api void onmessage(void delegate(in char[]) dg) { ontextmessage = dg; } /// ditto void onmessage(void delegate(in ubyte[]) dg) { onbinarymessage = dg; } /* } end copy/paste */ } bool websocketRequested(Cgi cgi) { return "sec-websocket-key" in cgi.requestHeaders && "connection" in cgi.requestHeaders && cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") && "upgrade" in cgi.requestHeaders && cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") ; } WebSocket acceptWebsocket(Cgi cgi) { assert(!cgi.closed); assert(!cgi.outputtedResponseData); cgi.setResponseStatus("101 Switching Protocols"); cgi.header("Upgrade: WebSocket"); cgi.header("Connection: upgrade"); string key = cgi.requestHeaders["sec-websocket-key"]; key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec import std.digest.sha; auto hash = sha1Of(key); auto accept = Base64.encode(hash); cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); cgi.websocketMode = true; cgi.write(""); cgi.flush(); return new WebSocket(cgi); } // FIXME get websocket to work on other modes, not just embedded_httpd /* copy/paste in http2.d { */ enum WebSocketOpcode : ubyte { continuation = 0, text = 1, binary = 2, // 3, 4, 5, 6, 7 RESERVED close = 8, ping = 9, pong = 10, // 11,12,13,14,15 RESERVED } public struct WebSocketFrame { private bool populated; bool fin; bool rsv1; bool rsv2; bool rsv3; WebSocketOpcode opcode; // 4 bits bool masked; ubyte lengthIndicator; // don't set this when building one to send ulong realLength; // don't use when sending ubyte[4] maskingKey; // don't set this when sending ubyte[] data; static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { WebSocketFrame msg; msg.fin = true; msg.opcode = opcode; msg.data = cast(ubyte[]) data.dup; return msg; } private void send(scope void delegate(ubyte[]) llsend) { ubyte[64] headerScratch; int headerScratchPos = 0; realLength = data.length; { ubyte b1; b1 |= cast(ubyte) opcode; b1 |= rsv3 ? (1 << 4) : 0; b1 |= rsv2 ? (1 << 5) : 0; b1 |= rsv1 ? (1 << 6) : 0; b1 |= fin ? (1 << 7) : 0; headerScratch[0] = b1; headerScratchPos++; } { headerScratchPos++; // we'll set header[1] at the end of this auto rlc = realLength; ubyte b2; b2 |= masked ? (1 << 7) : 0; assert(headerScratchPos == 2); if(realLength > 65535) { // use 64 bit length b2 |= 0x7f; // FIXME: double check endinaness foreach(i; 0 .. 8) { headerScratch[2 + 7 - i] = rlc & 0x0ff; rlc >>>= 8; } headerScratchPos += 8; } else if(realLength > 125) { // use 16 bit length b2 |= 0x7e; // FIXME: double check endinaness foreach(i; 0 .. 2) { headerScratch[2 + 1 - i] = rlc & 0x0ff; rlc >>>= 8; } headerScratchPos += 2; } else { // use 7 bit length b2 |= realLength & 0b_0111_1111; } headerScratch[1] = b2; } //assert(!masked, "masking key not properly implemented"); if(masked) { // FIXME: randomize this headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; headerScratchPos += 4; // we'll just mask it in place... int keyIdx = 0; foreach(i; 0 .. data.length) { data[i] = data[i] ^ maskingKey[keyIdx]; if(keyIdx == 3) keyIdx = 0; else keyIdx++; } } //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); llsend(headerScratch[0 .. headerScratchPos]); llsend(data); } static WebSocketFrame read(ref ubyte[] d) { WebSocketFrame msg; auto orig = d; WebSocketFrame needsMoreData() { d = orig; return WebSocketFrame.init; } if(d.length < 2) return needsMoreData(); ubyte b = d[0]; msg.populated = true; msg.opcode = cast(WebSocketOpcode) (b & 0x0f); b >>= 4; msg.rsv3 = b & 0x01; b >>= 1; msg.rsv2 = b & 0x01; b >>= 1; msg.rsv1 = b & 0x01; b >>= 1; msg.fin = b & 0x01; b = d[1]; msg.masked = (b & 0b1000_0000) ? true : false; msg.lengthIndicator = b & 0b0111_1111; d = d[2 .. $]; if(msg.lengthIndicator == 0x7e) { // 16 bit length msg.realLength = 0; if(d.length < 2) return needsMoreData(); foreach(i; 0 .. 2) { msg.realLength |= d[0] << ((1-i) * 8); d = d[1 .. $]; } } else if(msg.lengthIndicator == 0x7f) { // 64 bit length msg.realLength = 0; if(d.length < 8) return needsMoreData(); foreach(i; 0 .. 8) { msg.realLength |= ulong(d[0]) << ((7-i) * 8); d = d[1 .. $]; } } else { // 7 bit length msg.realLength = msg.lengthIndicator; } if(msg.masked) { if(d.length < 4) return needsMoreData(); msg.maskingKey = d[0 .. 4]; d = d[4 .. $]; } if(msg.realLength > d.length) { return needsMoreData(); } msg.data = d[0 .. cast(size_t) msg.realLength]; d = d[cast(size_t) msg.realLength .. $]; return msg; } void unmaskInPlace() { if(this.masked) { int keyIdx = 0; foreach(i; 0 .. this.data.length) { this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; if(keyIdx == 3) keyIdx = 0; else keyIdx++; } } } char[] textData() { return cast(char[]) data; } } /* } */ } version(Windows) { version(CRuntime_DigitalMars) { extern(C) int setmode(int, int) nothrow @nogc; } else version(CRuntime_Microsoft) { extern(C) int _setmode(int, int) nothrow @nogc; alias setmode = _setmode; } else static assert(0); } version(Posix) { import core.sys.posix.unistd; version(CRuntime_Musl) {} else { private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); } } // FIXME: these aren't quite public yet. //private: // template for laziness void startAddonServer()(string arg) { version(OSX) { assert(0, "Not implemented"); } else version(linux) { import core.sys.posix.unistd; pid_t pid; const(char)*[16] args; args[0] = "ARSD_CGI_ADDON_SERVER"; args[1] = arg.ptr; posix_spawn(&pid, "/proc/self/exe", null, null, args.ptr, null // env ); } else version(Windows) { wchar[2048] filename; auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); if(len == 0 || len == filename.length) throw new Exception("could not get process name to start helper server"); STARTUPINFOW startupInfo; startupInfo.cb = cast(DWORD) startupInfo.sizeof; PROCESS_INFORMATION processInfo; import std.utf; // I *MIGHT* need to run it as a new job or a service... auto ret = CreateProcessW( filename.ptr, toUTF16z(arg), null, // process attributes null, // thread attributes false, // inherit handles 0, // creation flags null, // environment null, // working directory &startupInfo, &processInfo ); if(!ret) throw new Exception("create process failed"); // when done with those, if we set them /* CloseHandle(hStdInput); CloseHandle(hStdOutput); CloseHandle(hStdError); */ } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); } // template for laziness /* The websocket server is a single-process, single-thread, event I/O thing. It is passed websockets from other CGI processes and is then responsible for handling their messages and responses. Note that the CGI process is responsible for websocket setup, including authentication, etc. It also gets data sent to it by other processes and is responsible for distributing that, as necessary. */ void runWebsocketServer()() { assert(0, "not implemented"); } void sendToWebsocketServer(WebSocket ws, string group) { assert(0, "not implemented"); } void sendToWebsocketServer(string content, string group) { assert(0, "not implemented"); } void runEventServer()() { runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); } void runTimerServer()() { runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); } version(Posix) { alias LocalServerConnectionHandle = int; alias CgiConnectionHandle = int; alias SocketConnectionHandle = int; enum INVALID_CGI_CONNECTION_HANDLE = -1; } else version(Windows) { alias LocalServerConnectionHandle = HANDLE; version(embedded_httpd_threads) { alias CgiConnectionHandle = SOCKET; enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; } else version(fastcgi) { alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. enum INVALID_CGI_CONNECTION_HANDLE = null; } else version(scgi) { alias CgiConnectionHandle = SOCKET; enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; } else { /* version(plain_cgi) */ alias CgiConnectionHandle = HANDLE; enum INVALID_CGI_CONNECTION_HANDLE = null; } alias SocketConnectionHandle = SOCKET; } version(with_addon_servers_connections) LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { version(Posix) { import core.sys.posix.unistd; import core.sys.posix.sys.un; int sock = socket(AF_UNIX, SOCK_STREAM, 0); if(sock == -1) throw new Exception("socket " ~ to!string(errno)); scope(failure) close(sock); cloexec(sock); // add-on server processes are assumed to be local, and thus will // use unix domain sockets. Besides, I want to pass sockets to them, // so it basically must be local (except for the session server, but meh). sockaddr_un addr; addr.sun_family = AF_UNIX; version(linux) { // on linux, we will use the abstract namespace addr.sun_path[0] = 0; addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; } else { // but otherwise, just use a file cuz we must. addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; } bool alreadyTried; try_again: if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { if(!alreadyTried && errno == ECONNREFUSED) { // try auto-spawning the server, then attempt connection again startAddonServer(arg); import core.thread; Thread.sleep(50.msecs); alreadyTried = true; goto try_again; } else throw new Exception("connect " ~ to!string(errno)); } return sock; } else version(Windows) { return null; // FIXME } } version(with_addon_servers_connections) void closeLocalServerConnection(LocalServerConnectionHandle handle) { version(Posix) { import core.sys.posix.unistd; close(handle); } else version(Windows) CloseHandle(handle); } void runSessionServer()() { runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); } version(Posix) private void makeNonBlocking(int fd) { import core.sys.posix.fcntl; auto flags = fcntl(fd, F_GETFL, 0); if(flags == -1) throw new Exception("fcntl get"); flags |= O_NONBLOCK; auto s = fcntl(fd, F_SETFL, flags); if(s == -1) throw new Exception("fcntl set"); } import core.stdc.errno; struct IoOp { @disable this(); @disable this(this); /* So we want to be able to eventually handle generic sockets too. */ enum Read = 1; enum Write = 2; enum Accept = 3; enum ReadSocketHandle = 4; // Your handler may be called in a different thread than the one that initiated the IO request! // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed private void delegate(IoOp*) closeHandler; private void delegate(IoOp*) completeHandler; private int internalFd; private int operation; private int bufferLengthAllocated; private int bufferLengthUsed; private ubyte[1] internalBuffer; // it can be overallocated! ubyte[] allocatedBuffer() return { return internalBuffer.ptr[0 .. bufferLengthAllocated]; } ubyte[] usedBuffer() return { return allocatedBuffer[0 .. bufferLengthUsed]; } void reset() { bufferLengthUsed = 0; } int fd() { return internalFd; } } IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { import core.stdc.stdlib; auto ptr = calloc(IoOp.sizeof + bufferSize, 1); if(ptr is null) assert(0); // out of memory! auto op = cast(IoOp*) ptr; op.handler = handler; op.internalFd = fd; op.operation = operation; op.bufferLengthAllocated = bufferSize; op.bufferLengthUsed = 0; import core.memory; GC.addRoot(ptr); return op; } void freeIoOp(ref IoOp* ptr) { import core.memory; GC.removeRoot(ptr); import core.stdc.stdlib; free(ptr); ptr = null; } version(Posix) version(with_addon_servers_connections) void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { //import std.stdio : writeln; writeln(cast(string) data); import core.sys.posix.unistd; auto ret = write(connection, data.ptr, data.length); if(ret != data.length) { if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { // the file is closed, remove it eis.fileClosed(connection); } else throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME } } version(Windows) version(with_addon_servers_connections) void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { // FIXME } bool isInvalidHandle(CgiConnectionHandle h) { return h == INVALID_CGI_CONNECTION_HANDLE; } /+ https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult +/ /++ You can customize your server by subclassing the appropriate server. Then, register your subclass at compile time with the [registerEventIoServer] template, or implement your own main function and call it yourself. $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) +/ version(with_addon_servers_connections) interface EventIoServer { bool handleLocalConnectionData(IoOp* op, int receivedFd); void handleLocalConnectionClose(IoOp* op); void handleLocalConnectionComplete(IoOp* op); void wait_timeout(); void fileClosed(int fd); void epoll_fd(int fd); } // the sink should buffer it private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { static if(is(T == struct)) { foreach(member; __traits(allMembers, T)) serialize(sink, __traits(getMember, t, member)); } else static if(is(T : int)) { // no need to think of endianness just because this is only used // for local, same-machine stuff anyway. thanks private lol sink((cast(ubyte*) &t)[0 .. t.sizeof]); } else static if(is(T == string) || is(T : const(ubyte)[])) { // these are common enough to optimize int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. sink((cast(ubyte*) &len)[0 .. int.sizeof]); sink(cast(ubyte[]) t[]); } else static if(is(T : A[], A)) { // generic array is less optimal but still prolly ok int len = cast(int) t.length; sink((cast(ubyte*) &len)[0 .. int.sizeof]); foreach(item; t) serialize(sink, item); } else static assert(0, T.stringof); } // all may be stack buffers, so use caution private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { static if(is(T == struct)) { T t; foreach(member; __traits(allMembers, T)) deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); dg(t); } else static if(is(T : int)) { // no need to think of endianness just because this is only used // for local, same-machine stuff anyway. thanks private lol T t; auto data = get(t.sizeof); t = (cast(T[]) data)[0]; dg(t); } else static if(is(T == string) || is(T : const(ubyte)[])) { // these are common enough to optimize int len; auto data = get(len.sizeof); len = (cast(int[]) data)[0]; /* typeof(T[0])[2000] stackBuffer; T buffer; if(len < stackBuffer.length) buffer = stackBuffer[0 .. len]; else buffer = new T(len); data = get(len * typeof(T[0]).sizeof); */ T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); dg(t); } else static if(is(T == E[], E)) { T t; int len; auto data = get(len.sizeof); len = (cast(int[]) data)[0]; t.length = len; foreach(ref e; t) { deserialize!E(get, (ele) { e = ele; }); } dg(t); } else static assert(0, T.stringof); } unittest { serialize((ubyte[] b) { deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); }, 1); serialize((ubyte[] b) { deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); }, 56674); ubyte[1000] buffer; int bufferPoint; void add(ubyte[] b) { buffer[bufferPoint .. bufferPoint + b.length] = b[]; bufferPoint += b.length; } ubyte[] get(int sz) { auto b = buffer[bufferPoint .. bufferPoint + sz]; bufferPoint += sz; return b; } serialize(&add, "test here"); bufferPoint = 0; deserialize!string(&get, (t) { assert(t == "test here"); }); bufferPoint = 0; struct Foo { int a; ubyte c; string d; } serialize(&add, Foo(403, 37, "amazing")); bufferPoint = 0; deserialize!Foo(&get, (t) { assert(t.a == 403); assert(t.c == 37); assert(t.d == "amazing"); }); bufferPoint = 0; } /* Here's the way the RPC interface works: You define the interface that lists the functions you can call on the remote process. The interface may also have static methods for convenience. These forward to a singleton instance of an auto-generated class, which actually sends the args over the pipe. An impl class actually implements it. A receiving server deserializes down the pipe and calls methods on the class. I went with the interface to get some nice compiler checking and documentation stuff. I could have skipped the interface and just implemented it all from the server class definition itself, but then the usage may call the method instead of rpcing it; I just like having the user interface and the implementation separate so you aren't tempted to `new impl` to call the methods. I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. Realistically though the bodies would just be connection.call(this.mangleof, args...) sooooo. FIXME: overloads aren't supported */ /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. interface SessionObject {} private immutable void delegate(string[])[string] scheduledJobHandlers; private immutable void delegate(string[])[string] websocketServers; version(with_breaking_cgi_features) mixin(q{ mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { static import std.traits; // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. static foreach(idx, member; __traits(derivedMembers, T)) { static if(__traits(isVirtualMethod, __traits(getMember, T, member))) mixin( q{ std.traits.ReturnType!(__traits(getMember, T, member)) } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) { SerializationBuffer buffer; auto i = cast(ushort) idx; serialize(&buffer.sink, i); serialize(&buffer.sink, __traits(getMember, T, member).mangleof); foreach(param; params) serialize(&buffer.sink, param); auto sendable = buffer.sendable; version(Posix) {{ auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); if(ret == -1) { throw new Exception("send returned -1, errno: " ~ to!string(errno)); } else if(ret == 0) { throw new Exception("Connection to addon server lost"); } if(ret < sendable.length) throw new Exception("Send failed to send all"); assert(ret == sendable.length); }} // FIXME Windows impl static if(!is(typeof(return) == void)) { // there is a return value; we need to wait for it too version(Posix) { ubyte[3000] revBuffer; auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); auto got = revBuffer[0 .. ret]; int dataLocation; ubyte[] grab(int sz) { auto dataLocation1 = dataLocation; dataLocation += sz; return got[dataLocation1 .. dataLocation]; } typeof(return) retu; deserialize!(typeof(return))(&grab, (a) { retu = a; }); return retu; } else { // FIXME Windows impl return typeof(return).init; } } }}); } private static typeof(this) singletonInstance; private LocalServerConnectionHandle connectionHandle; static typeof(this) connection() { if(singletonInstance is null) { singletonInstance = new typeof(this)(); singletonInstance.connect(); } return singletonInstance; } void connect() { connectionHandle = openLocalServerConnection(serverPath, cmdArg); } void disconnect() { closeLocalServerConnection(connectionHandle); } } void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { ushort calledIdx; string calledFunction; int dataLocation; ubyte[] grab(int sz) { if(sz == 0) assert(0); auto d = data[dataLocation .. dataLocation + sz]; dataLocation += sz; return d; } again: deserialize!ushort(&grab, (a) { calledIdx = a; }); deserialize!string(&grab, (a) { calledFunction = a; }); import std.traits; sw: switch(calledIdx) { foreach(idx, memberName; __traits(derivedMembers, Interface)) static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { case idx: assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); Parameters!(__traits(getMember, Interface, memberName)) params; foreach(ref param; params) deserialize!(typeof(param))(&grab, (a) { param = a; }); static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { __traits(getMember, this_, memberName)(params); } else { auto ret = __traits(getMember, this_, memberName)(params); SerializationBuffer buffer; serialize(&buffer.sink, ret); auto sendable = buffer.sendable; version(Posix) { auto r = send(fd, sendable.ptr, sendable.length, 0); if(r == -1) { throw new Exception("send returned -1, errno: " ~ to!string(errno)); } else if(r == 0) { throw new Exception("Connection to addon client lost"); } if(r < sendable.length) throw new Exception("Send failed to send all"); } // FIXME Windows impl } break sw; } default: assert(0); } if(dataLocation != data.length) goto again; } private struct SerializationBuffer { ubyte[2048] bufferBacking; int bufferLocation; void sink(scope ubyte[] data) { bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; bufferLocation += data.length; } ubyte[] sendable() return { return bufferBacking[0 .. bufferLocation]; } } /* FIXME: add a version command line arg version data in the library management gui as external program at server with event_fd for each run use .mangleof in the at function name i think the at server will have to: pipe args to the child collect child output for logging get child return value for logging on windows timers work differently. idk how to best combine with the io stuff. will have to have dump and restore too, so i can restart without losing stuff. */ /++ A convenience object for talking to the [BasicDataServer] from a higher level. See: [Cgi.getSessionObject]. You pass it a `Data` struct describing the data you want saved in the session. Then, this class will generate getter and setter properties that allow access to that data. Note that each load and store will be done as-accessed; it doesn't front-load mutable data nor does it batch updates out of fear of read-modify-write race conditions. (In fact, right now it does this for everything, but in the future, I might batch load `immutable` members of the Data struct.) At some point in the future, I might also let it do different backends, like a client-side cookie store too, but idk. Note that the plain-old-data members of your `Data` struct are wrapped by this interface via a static foreach to make property functions. See_Also: [MockSession] +/ interface Session(Data) : SessionObject { @property string sessionId() const; /++ Starts a new session. Note that a session is also implicitly started as soon as you write data to it, so if you need to alter these parameters from their defaults, be sure to explicitly call this BEFORE doing any writes to session data. Params: idleLifetime = How long, in seconds, the session should remain in memory when not being read from or written to. The default is one day. NOT IMPLEMENTED useExtendedLifetimeCookie = The session ID is always stored in a HTTP cookie, and by default, that cookie is discarded when the user closes their browser. But if you set this to true, it will use a non-perishable cookie for the given idleLifetime. NOT IMPLEMENTED +/ void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); /++ Regenerates the session ID and updates the associated cookie. This is also your chance to change immutable data (not yet implemented). +/ void regenerateId(); /++ Terminates this session, deleting all saved data. +/ void terminate(); /++ Plain-old-data members of your `Data` struct are wrapped here via the property getters and setters. If the member is a non-string array, it returns a magical array proxy object which allows for atomic appends and replaces via overloaded operators. You can slice this to get a range representing a $(B const) view of the array. This is to protect you against read-modify-write race conditions. +/ static foreach(memberName; __traits(allMembers, Data)) static if(is(typeof(__traits(getMember, Data, memberName)))) mixin(q{ @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); }); } /++ An implementation of [Session] that works on real cgi connections utilizing the [BasicDataServer]. As opposed to a [MockSession] which is made for testing purposes. You will not construct one of these directly. See [Cgi.getSessionObject] instead. +/ class BasicDataServerSession(Data) : Session!Data { private Cgi cgi; private string sessionId_; public @property string sessionId() const { return sessionId_; } protected @property string sessionId(string s) { return this.sessionId_ = s; } private this(Cgi cgi) { this.cgi = cgi; if(auto ptr = "sessionId" in cgi.cookies) sessionId = (*ptr).length ? *ptr : null; } void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { assert(sessionId is null); // FIXME: what if there is a session ID cookie, but no corresponding session on the server? import std.random, std.conv; sessionId = to!string(uniform(1, long.max)); BasicDataServer.connection.createSession(sessionId, idleLifetime); setCookie(); } protected void setCookie() { cgi.setCookie( "sessionId", sessionId, 0 /* expiration */, "/" /* path */, null /* domain */, true /* http only */, cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); } void regenerateId() { if(sessionId is null) { start(); return; } import std.random, std.conv; auto oldSessionId = sessionId; sessionId = to!string(uniform(1, long.max)); BasicDataServer.connection.renameSession(oldSessionId, sessionId); setCookie(); } void terminate() { BasicDataServer.connection.destroySession(sessionId); sessionId = null; setCookie(); } static foreach(memberName; __traits(allMembers, Data)) static if(is(typeof(__traits(getMember, Data, memberName)))) mixin(q{ @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { if(sessionId is null) return typeof(return).init; import std.traits; auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); if(v.length == 0) return typeof(return).init; import std.conv; // why this cast? to doesn't like being given an inout argument. so need to do it without that, then // we need to return it and that needed the cast. It should be fine since we basically respect constness.. // basically. Assuming the session is POD this should be fine. return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); } @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { if(sessionId is null) start(); import std.conv; import std.traits; BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); return value; } }); } /++ A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. Simply stores the data in its instance members. +/ class MockSession(Data) : Session!Data { pure { @property string sessionId() const { return "mock"; } void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} void regenerateId() {} void terminate() {} private Data store_; static foreach(memberName; __traits(allMembers, Data)) static if(is(typeof(__traits(getMember, Data, memberName)))) mixin(q{ @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { return __traits(getMember, store_, memberName); } @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { return __traits(getMember, store_, memberName) = value; } }); } } /++ Direct interface to the basic data add-on server. You can typically use [Cgi.getSessionObject] as a more convenient interface. +/ version(with_addon_servers_connections) interface BasicDataServer { /// void createSession(string sessionId, int lifetime); /// void renewSession(string sessionId, int lifetime); /// void destroySession(string sessionId); /// void renameSession(string oldSessionId, string newSessionId); /// void setSessionData(string sessionId, string dataKey, string dataValue); /// string getSessionData(string sessionId, string dataKey); /// static BasicDataServerConnection connection() { return BasicDataServerConnection.connection(); } } version(with_addon_servers_connections) class BasicDataServerConnection : BasicDataServer { mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); } version(with_addon_servers) final class BasicDataServerImplementation : BasicDataServer, EventIoServer { void createSession(string sessionId, int lifetime) { sessions[sessionId.idup] = Session(lifetime); } void destroySession(string sessionId) { sessions.remove(sessionId); } void renewSession(string sessionId, int lifetime) { sessions[sessionId].lifetime = lifetime; } void renameSession(string oldSessionId, string newSessionId) { sessions[newSessionId.idup] = sessions[oldSessionId]; sessions.remove(oldSessionId); } void setSessionData(string sessionId, string dataKey, string dataValue) { if(sessionId !in sessions) createSession(sessionId, 3600); // FIXME? sessions[sessionId].values[dataKey.idup] = dataValue.idup; } string getSessionData(string sessionId, string dataKey) { if(auto session = sessionId in sessions) { if(auto data = dataKey in (*session).values) return *data; else return null; // no such data } else { return null; // no session } } protected: struct Session { int lifetime; string[string] values; } Session[string] sessions; bool handleLocalConnectionData(IoOp* op, int receivedFd) { auto data = op.usedBuffer; dispatchRpcServer!BasicDataServer(this, data, op.fd); return false; } void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant void wait_timeout() {} void fileClosed(int fd) {} // stateless so irrelevant void epoll_fd(int fd) {} } /++ See [schedule] to make one of these. You then call one of the methods here to set it up: --- schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it --- +/ version(with_addon_servers_connections) struct ScheduledJobHelper { private string func; private string[] args; private bool consumed; private this(string func, string[] args) { this.func = func; this.args = args; } ~this() { assert(consumed); } /++ Schedules the job to be run at the given time. +/ void at(DateTime when, immutable TimeZone timezone = UTC()) { consumed = true; auto conn = ScheduledJobServerConnection.connection; import std.file; auto st = SysTime(when, timezone); auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); } /++ Schedules the job to run at least after the specified delay. +/ void delay(Duration delay) { consumed = true; auto conn = ScheduledJobServerConnection.connection; import std.file; auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); } /++ Runs the job in the background ASAP. $(NOTE It may run in a background thread. Don't segfault!) +/ void asap() { consumed = true; auto conn = ScheduledJobServerConnection.connection; import std.file; auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); } /+ /++ Schedules the job to recur on the given pattern. +/ void recur(string spec) { } +/ } /++ First step to schedule a job on the scheduled job server. The scheduled job needs to be a top-level function that doesn't read any variables from outside its arguments because it may be run in a new process, without any context existing later. You MUST set details on the returned object to actually do anything! +/ template schedule(alias fn, T...) if(is(typeof(fn) == function)) { /// ScheduledJobHelper schedule(T args) { // this isn't meant to ever be called, but instead just to // get the compiler to type check the arguments passed for us auto sample = delegate() { fn(args); }; string[] sargs; foreach(arg; args) sargs ~= to!string(arg); return ScheduledJobHelper(fn.mangleof, sargs); } shared static this() { scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { import std.traits; Parameters!fn args; foreach(idx, ref arg; args) arg = to!(typeof(arg))(sargs[idx]); fn(args); }; } } /// interface ScheduledJobServer { /// Use the [schedule] function for a higher-level interface. int scheduleJob(int whenIs, int when, string executable, string func, string[] args); /// void cancelJob(int jobId); } version(with_addon_servers_connections) class ScheduledJobServerConnection : ScheduledJobServer { mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); } version(with_addon_servers) final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { // FIXME: we need to handle SIGCHLD in this somehow // whenIs is 0 for relative, 1 for absolute protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { auto nj = nextJobId; nextJobId++; version(linux) { import core.sys.linux.timerfd; import core.sys.linux.epoll; import core.sys.posix.unistd; auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); if(fd == -1) throw new Exception("fd timer create failed"); foreach(ref arg; args) arg = arg.idup; auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); itimerspec value; value.it_value.tv_sec = when; value.it_value.tv_nsec = 0; value.it_interval.tv_sec = 0; value.it_interval.tv_nsec = 0; if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) throw new Exception("couldn't set fd timer"); auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { jobs.remove(nj); epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); close(fd); spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); return true; }); scope(failure) freeIoOp(op); epoll_event ev; ev.events = EPOLLIN | EPOLLET; ev.data.ptr = op; if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) throw new Exception("epoll_ctl " ~ to!string(errno)); jobs[nj] = job; return nj; } else assert(0); } protected void cancelJob(int jobId) { version(linux) { auto job = jobId in jobs; if(job is null) return; jobs.remove(jobId); version(linux) { import core.sys.linux.timerfd; import core.sys.linux.epoll; import core.sys.posix.unistd; epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); close(job.timerfd); } } jobs.remove(jobId); } int nextJobId = 1; static struct Job { string executable; string func; string[] args; int timerfd; int id; } Job[int] jobs; // event io server methods below bool handleLocalConnectionData(IoOp* op, int receivedFd) { auto data = op.usedBuffer; dispatchRpcServer!ScheduledJobServer(this, data, op.fd); return false; } void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant void wait_timeout() {} void fileClosed(int fd) {} // stateless so irrelevant int epoll_fd_; void epoll_fd(int fd) {this.epoll_fd_ = fd; } int epoll_fd() { return epoll_fd_; } } /// version(with_addon_servers_connections) interface EventSourceServer { /++ sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. $(WARNING This API is extremely unstable. I might change it or remove it without notice.) See_Also: [sendEvent] +/ public static void adoptConnection(Cgi cgi, in char[] eventUrl) { /* If lastEventId is missing or empty, you just get new events as they come. If it is set from something else, it sends all since then (that are still alive) down the pipe immediately. The reason it can come from the header is that's what the standard defines for browser reconnects. The reason it can come from a query string is just convenience in catching up in a user-defined manner. The reason the header overrides the query string is if the browser tries to reconnect, it will send the header AND the query (it reconnects to the same url), so we just want to do the restart thing. Note that if you ask for "0" as the lastEventId, it will get ALL still living events. */ string lastEventId = cgi.lastEventId; if(lastEventId.length == 0 && "lastEventId" in cgi.get) lastEventId = cgi.get["lastEventId"]; cgi.setResponseContentType("text/event-stream"); cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later cgi.flush(); cgi.closed = true; auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); scope(exit) closeLocalServerConnection(s); version(fastcgi) throw new Exception("sending fcgi connections not supported"); else { auto fd = cgi.getOutputFileHandle(); if(isInvalidHandle(fd)) throw new Exception("bad fd from cgi!"); EventSourceServerImplementation.SendableEventConnection sec; sec.populate(cgi.responseChunked, eventUrl, lastEventId); version(Posix) { auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); assert(res == sec.sizeof); } else version(Windows) { // FIXME } } } /++ Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. $(WARNING This API is extremely unstable. I might change it or remove it without notice.) Params: url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. event = the event type string, which is used in the Javascript addEventListener API on EventSource data = the event data. Available in JS as `event.data`. lifetime = the amount of time to keep this event for replaying on the event server. See_Also: [sendEventToEventServer] +/ public static void sendEvent(string url, string event, string data, int lifetime) { auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); scope(exit) closeLocalServerConnection(s); EventSourceServerImplementation.SendableEvent sev; sev.populate(url, event, data, lifetime); version(Posix) { auto ret = send(s, &sev, sev.sizeof, 0); assert(ret == sev.sizeof); } else version(Windows) { // FIXME } } /++ Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. See_Also: [disconnect] +/ void connect(string url, string forwardUrl); /++ Disconnects `forwardUrl` from `url` See_Also: [connect] +/ void disconnect(string url, string forwardUrl); } /// version(with_addon_servers) final class EventSourceServerImplementation : EventSourceServer, EventIoServer { protected: void connect(string url, string forwardUrl) { pipes[url] ~= forwardUrl; } void disconnect(string url, string forwardUrl) { auto t = url in pipes; if(t is null) return; foreach(idx, n; (*t)) if(n == forwardUrl) { (*t)[idx] = (*t)[$-1]; (*t) = (*t)[0 .. $-1]; break; } } bool handleLocalConnectionData(IoOp* op, int receivedFd) { if(receivedFd != -1) { //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; auto url = got.url.idup; eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); // FIXME: catch up on past messages here } else { auto data = op.usedBuffer; auto event = cast(SendableEvent*) data.ptr; if(event.magic == 0xdeadbeef) { handleInputEvent(event); if(event.url in pipes) foreach(pipe; pipes[event.url]) { event.url = pipe; handleInputEvent(event); } } else { dispatchRpcServer!EventSourceServer(this, data, op.fd); } } return false; } void handleLocalConnectionClose(IoOp* op) { fileClosed(op.fd); } void handleLocalConnectionComplete(IoOp* op) {} void wait_timeout() { // just keeping alive foreach(url, connections; eventConnectionsByUrl) foreach(connection; connections) if(connection.needsChunking) nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); else nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); } void fileClosed(int fd) { outer: foreach(url, ref connections; eventConnectionsByUrl) { foreach(idx, conn; connections) { if(fd == conn.fd) { connections[idx] = connections[$-1]; connections = connections[0 .. $ - 1]; continue outer; } } } } void epoll_fd(int fd) {} private: struct SendableEventConnection { ubyte responseChunked; int urlLength; char[256] urlBuffer = 0; int lastEventIdLength; char[32] lastEventIdBuffer = 0; char[] url() return { return urlBuffer[0 .. urlLength]; } void url(in char[] u) { urlBuffer[0 .. u.length] = u[]; urlLength = cast(int) u.length; } char[] lastEventId() return { return lastEventIdBuffer[0 .. lastEventIdLength]; } void populate(bool responseChunked, in char[] url, in char[] lastEventId) in { assert(url.length < this.urlBuffer.length); assert(lastEventId.length < this.lastEventIdBuffer.length); } do { this.responseChunked = responseChunked ? 1 : 0; this.urlLength = cast(int) url.length; this.lastEventIdLength = cast(int) lastEventId.length; this.urlBuffer[0 .. url.length] = url[]; this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; } } struct SendableEvent { int magic = 0xdeadbeef; int urlLength; char[256] urlBuffer = 0; int typeLength; char[32] typeBuffer = 0; int messageLength; char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. int _lifetime; char[] message() return { return messageBuffer[0 .. messageLength]; } char[] type() return { return typeBuffer[0 .. typeLength]; } char[] url() return { return urlBuffer[0 .. urlLength]; } void url(in char[] u) { urlBuffer[0 .. u.length] = u[]; urlLength = cast(int) u.length; } int lifetime() { return _lifetime; } /// void populate(string url, string type, string message, int lifetime) in { assert(url.length < this.urlBuffer.length); assert(type.length < this.typeBuffer.length); assert(message.length < this.messageBuffer.length); } do { this.urlLength = cast(int) url.length; this.typeLength = cast(int) type.length; this.messageLength = cast(int) message.length; this._lifetime = lifetime; this.urlBuffer[0 .. url.length] = url[]; this.typeBuffer[0 .. type.length] = type[]; this.messageBuffer[0 .. message.length] = message[]; } } struct EventConnection { int fd; bool needsChunking; } private EventConnection[][string] eventConnectionsByUrl; private string[][string] pipes; private void handleInputEvent(scope SendableEvent* event) { static int eventId; static struct StoredEvent { int id; string type; string message; int lifetimeRemaining; } StoredEvent[][string] byUrl; int thisId = ++eventId; if(event.lifetime) byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); auto connectionsPtr = event.url in eventConnectionsByUrl; EventConnection[] connections; if(connectionsPtr is null) return; else connections = *connectionsPtr; char[4096] buffer; char[] formattedMessage; void append(const char[] a) { // the 6's here are to leave room for a HTTP chunk header, if it proves necessary buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; } import std.algorithm.iteration; if(connections.length) { append("id: "); append(to!string(thisId)); append("\n"); append("event: "); append(event.type); append("\n"); foreach(line; event.message.splitter("\n")) { append("data: "); append(line); append("\n"); } append("\n"); } // chunk it for HTTP! auto len = toHex(formattedMessage.length); buffer[4 .. 6] = "\r\n"[]; buffer[4 - len.length .. 4] = len[]; buffer[6 + formattedMessage.length] = '\r'; buffer[6 + formattedMessage.length + 1] = '\n'; auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; // done // FIXME: send back requests when needed // FIXME: send a single ":\n" every 15 seconds to keep alive foreach(connection; connections) { if(connection.needsChunking) { nonBlockingWrite(this, connection.fd, chunkedMessage); } else { nonBlockingWrite(this, connection.fd, formattedMessage); } } } } void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { version(Posix) { import core.sys.posix.unistd; import core.sys.posix.fcntl; import core.sys.posix.sys.un; import core.sys.posix.signal; signal(SIGPIPE, SIG_IGN); static extern(C) void sigchldhandler(int) { int status; import w = core.sys.posix.sys.wait; w.wait(&status); } signal(SIGCHLD, &sigchldhandler); int sock = socket(AF_UNIX, SOCK_STREAM, 0); if(sock == -1) throw new Exception("socket " ~ to!string(errno)); scope(failure) close(sock); cloexec(sock); // add-on server processes are assumed to be local, and thus will // use unix domain sockets. Besides, I want to pass sockets to them, // so it basically must be local (except for the session server, but meh). sockaddr_un addr; addr.sun_family = AF_UNIX; version(linux) { // on linux, we will use the abstract namespace addr.sun_path[0] = 0; addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; } else { // but otherwise, just use a file cuz we must. addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; } if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) throw new Exception("bind " ~ to!string(errno)); if(listen(sock, 128) == -1) throw new Exception("listen " ~ to!string(errno)); makeNonBlocking(sock); version(linux) { import core.sys.linux.epoll; auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); if(epoll_fd == -1) throw new Exception("epoll_create1 " ~ to!string(errno)); scope(failure) close(epoll_fd); } else { import core.sys.posix.poll; } version(linux) eis.epoll_fd = epoll_fd; auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); scope(exit) freeIoOp(acceptOp); version(linux) { epoll_event ev; ev.events = EPOLLIN | EPOLLET; ev.data.ptr = acceptOp; if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) throw new Exception("epoll_ctl " ~ to!string(errno)); epoll_event[64] events; } else { pollfd[] pollfds; IoOp*[int] ioops; pollfds ~= pollfd(sock, POLLIN); ioops[sock] = acceptOp; } import core.time : MonoTime, seconds; MonoTime timeout = MonoTime.currTime + 15.seconds; while(true) { // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently int timeout_milliseconds = 0; // -1; // infinite timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; if(timeout_milliseconds < 0) timeout_milliseconds = 0; //writeln("waiting for ", name); version(linux) { auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); if(nfds == -1) { if(errno == EINTR) continue; throw new Exception("epoll_wait " ~ to!string(errno)); } } else { int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); size_t lastIdx = 0; } if(nfds == 0) { eis.wait_timeout(); timeout += 15.seconds; } foreach(idx; 0 .. nfds) { version(linux) { auto flags = events[idx].events; auto ioop = cast(IoOp*) events[idx].data.ptr; } else { IoOp* ioop; foreach(tidx, thing; pollfds[lastIdx .. $]) { if(thing.revents) { ioop = ioops[thing.fd]; lastIdx += tidx + 1; break; } } } //writeln(flags, " ", ioop.fd); void newConnection() { // on edge triggering, it is important that we get it all while(true) { version(Android) { auto size = cast(int) addr.sizeof; } else { auto size = cast(uint) addr.sizeof; } auto ns = accept(sock, cast(sockaddr*) &addr, &size); if(ns == -1) { if(errno == EAGAIN || errno == EWOULDBLOCK) { // all done, got it all break; } throw new Exception("accept " ~ to!string(errno)); } cloexec(ns); makeNonBlocking(ns); auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); niop.closeHandler = &eis.handleLocalConnectionClose; niop.completeHandler = &eis.handleLocalConnectionComplete; scope(failure) freeIoOp(niop); version(linux) { epoll_event nev; nev.events = EPOLLIN | EPOLLET; nev.data.ptr = niop; if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) throw new Exception("epoll_ctl " ~ to!string(errno)); } else { bool found = false; foreach(ref pfd; pollfds) { if(pfd.fd < 0) { pfd.fd = ns; found = true; } } if(!found) pollfds ~= pollfd(ns, POLLIN); ioops[ns] = niop; } } } bool newConnectionCondition() { version(linux) return ioop.fd == sock && (flags & EPOLLIN); else return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); } if(newConnectionCondition()) { newConnection(); } else if(ioop.operation == IoOp.ReadSocketHandle) { while(true) { int in_fd; auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); if(got == -1) { if(errno == EAGAIN || errno == EWOULDBLOCK) { // all done, got it all if(ioop.completeHandler) ioop.completeHandler(ioop); break; } throw new Exception("recv " ~ to!string(errno)); } if(got == 0) { if(ioop.closeHandler) { ioop.closeHandler(ioop); version(linux) {} // nothing needed else { foreach(ref pfd; pollfds) { if(pfd.fd == ioop.fd) pfd.fd = -1; } } } close(ioop.fd); freeIoOp(ioop); break; } ioop.bufferLengthUsed = cast(int) got; ioop.handler(ioop, in_fd); } } else if(ioop.operation == IoOp.Read) { while(true) { auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); if(got == -1) { if(errno == EAGAIN || errno == EWOULDBLOCK) { // all done, got it all if(ioop.completeHandler) ioop.completeHandler(ioop); break; } throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); } if(got == 0) { if(ioop.closeHandler) ioop.closeHandler(ioop); close(ioop.fd); freeIoOp(ioop); break; } ioop.bufferLengthUsed = cast(int) got; if(ioop.handler(ioop, ioop.fd)) { close(ioop.fd); freeIoOp(ioop); break; } } } // EPOLLHUP? } } } else version(Windows) { // set up a named pipe // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid } else static assert(0); } version(with_sendfd) // copied from the web and ported from C // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { msghdr msg; iovec[1] iov; version(OSX) { // removed } else version(Android) { } else { union ControlUnion { cmsghdr cm; char[CMSG_SPACE(int.sizeof)] control; } ControlUnion control_un; cmsghdr* cmptr; msg.msg_control = control_un.control.ptr; msg.msg_controllen = control_un.control.length; cmptr = CMSG_FIRSTHDR(&msg); cmptr.cmsg_len = CMSG_LEN(int.sizeof); cmptr.cmsg_level = SOL_SOCKET; cmptr.cmsg_type = SCM_RIGHTS; *(cast(int *) CMSG_DATA(cmptr)) = sendfd; } msg.msg_name = null; msg.msg_namelen = 0; iov[0].iov_base = ptr; iov[0].iov_len = nbytes; msg.msg_iov = iov.ptr; msg.msg_iovlen = 1; return sendmsg(fd, &msg, 0); } version(with_sendfd) // copied from the web and ported from C ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { msghdr msg; iovec[1] iov; ssize_t n; int newfd; version(OSX) { //msg.msg_accrights = cast(cattr_t) recvfd; //msg.msg_accrightslen = int.sizeof; } else version(Android) { } else { union ControlUnion { cmsghdr cm; char[CMSG_SPACE(int.sizeof)] control; } ControlUnion control_un; cmsghdr* cmptr; msg.msg_control = control_un.control.ptr; msg.msg_controllen = control_un.control.length; } msg.msg_name = null; msg.msg_namelen = 0; iov[0].iov_base = ptr; iov[0].iov_len = nbytes; msg.msg_iov = iov.ptr; msg.msg_iovlen = 1; if ( (n = recvmsg(fd, &msg, 0)) <= 0) return n; version(OSX) { //if(msg.msg_accrightslen != int.sizeof) //*recvfd = -1; } else version(Android) { } else { if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { if (cmptr.cmsg_level != SOL_SOCKET) throw new Exception("control level != SOL_SOCKET"); if (cmptr.cmsg_type != SCM_RIGHTS) throw new Exception("control type != SCM_RIGHTS"); *recvfd = *(cast(int *) CMSG_DATA(cmptr)); } else *recvfd = -1; /* descriptor was not passed */ } return n; } /* end read_fd */ /* Event source stuff The api is: sendEvent(string url, string type, string data, int timeout = 60*10); attachEventListener(string url, int fd, lastId) It just sends to all attached listeners, and stores it until the timeout for replaying via lastEventId. */ /* Session process stuff it stores it all. the cgi object has a session object that can grab it session may be done in the same process if possible, there is a version switch to choose if you want to override. */ struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; alias handler = dispatchHandler; string urlPrefix; bool rejectFurther; immutable(DispatcherDetails) details; } private string urlify(string name) pure { return beautify(name, '-', true); } private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { if(name == "id") return allLowerCase ? name : "ID"; char[160] buffer; int bufferIndex = 0; bool shouldCap = true; bool shouldSpace; bool lastWasCap; foreach(idx, char ch; name) { if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important if((ch >= 'A' && ch <= 'Z') || ch == '_') { if(lastWasCap) { // two caps in a row, don't change. Prolly acronym. } else { if(idx) shouldSpace = true; // new word, add space } lastWasCap = true; } else { lastWasCap = false; } if(shouldSpace) { buffer[bufferIndex++] = space; if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important shouldSpace = false; } if(shouldCap) { if(ch >= 'a' && ch <= 'z') ch -= 32; shouldCap = false; } if(allLowerCase && ch >= 'A' && ch <= 'Z') ch += 32; buffer[bufferIndex++] = ch; } return buffer[0 .. bufferIndex].idup; } /* string urlFor(alias func)() { return __traits(identifier, func); } */ /++ UDA: The name displayed to the user in auto-generated HTML. Default is `beautify(identifier)`. +/ struct DisplayName { string name; } /++ UDA: The name used in the URL or web parameter. Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. +/ struct UrlName { string name; } /++ UDA: default format to respond for this method +/ struct DefaultFormat { string value; } class MissingArgumentException : Exception { string functionName; string argumentName; string argumentType; this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { this.functionName = functionName; this.argumentName = argumentName; this.argumentType = argumentType; super("Missing Argument: " ~ this.argumentName, file, line, next); } } /++ You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. History: Added December 15, 2021 (dub v10.5) +/ class ResourceNotFoundException : Exception { string resourceType; string resourceId; this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { this.resourceType = resourceType; this.resourceId = resourceId; super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); } } /++ This can be attached to any constructor or function called from the cgi system. If it is present, the function argument can NOT be set from web params, but instead is set to the return value of the given `func`. If `func` can take a parameter of type [Cgi], it will be passed the one representing the current request. Otherwise, it must take zero arguments. Any params in your function of type `Cgi` are automatically assumed to take the cgi object for the connection. Any of type [Session] (with an argument) is also assumed to come from the cgi object. const arguments are also supported. +/ struct ifCalledFromWeb(alias func) {} // it only looks at query params for GET requests, the rest must be in the body for a function argument. auto callFromCgi(alias method, T)(T dg, Cgi cgi) { // FIXME: any array of structs should also be settable or gettable from csv as well. // FIXME: think more about checkboxes and bools. import std.traits; Parameters!method params; alias idents = ParameterIdentifierTuple!method; alias defaults = ParameterDefaults!method; const(string)[] names; const(string)[] values; // first, check for missing arguments and initialize to defaults if necessary static if(is(typeof(method) P == __parameters)) foreach(idx, param; P) {{ // see: mustNotBeSetFromWebParams static if(is(param : Cgi)) { static assert(!is(param == immutable)); cast() params[idx] = cgi; } else static if(is(param == Session!D, D)) { static assert(!is(param == immutable)); cast() params[idx] = cgi.getSessionObject!D(); } else { bool populated; foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { static if(is(uda == ifCalledFromWeb!func, alias func)) { static if(is(typeof(func(cgi)))) params[idx] = func(cgi); else params[idx] = func(); populated = true; } } if(!populated) { static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { params[idx] = param.getAutomaticallyForCgi(cgi); populated = true; } } if(!populated) { auto ident = idents[idx]; if(cgi.requestMethod == Cgi.RequestMethod.GET) { if(ident !in cgi.get) { static if(is(defaults[idx] == void)) { static if(is(param == bool)) params[idx] = false; else throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); } else params[idx] = defaults[idx]; } } else { if(ident !in cgi.post) { static if(is(defaults[idx] == void)) { static if(is(param == bool)) params[idx] = false; else throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); } else params[idx] = defaults[idx]; } } } } }} // second, parse the arguments in order to build up arrays, etc. static bool setVariable(T)(string name, string paramName, T* what, string value) { static if(is(T == struct)) { if(name == paramName) { *what = T.init; return true; } else { // could be a child. gonna allow either obj.field OR obj[field] string afterName; if(name[paramName.length] == '[') { int count = 1; auto idx = paramName.length + 1; while(idx < name.length && count > 0) { if(name[idx] == '[') count++; else if(name[idx] == ']') { count--; if(count == 0) break; } idx++; } if(idx == name.length) return false; // malformed auto insideBrackets = name[paramName.length + 1 .. idx]; afterName = name[idx + 1 .. $]; name = name[0 .. paramName.length]; paramName = insideBrackets; } else if(name[paramName.length] == '.') { paramName = name[paramName.length + 1 .. $]; name = paramName; int p = 0; foreach(ch; paramName) { if(ch == '.' || ch == '[') break; p++; } afterName = paramName[p .. $]; paramName = paramName[0 .. p]; } else { return false; } if(paramName.length) // set the child member switch(paramName) { foreach(idx, memberName; __traits(allMembers, T)) static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { // data member! case memberName: return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); } default: // ok, not a member } } return false; } else static if(is(T == enum)) { *what = to!T(value); return true; } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { *what = to!T(value); return true; } else static if(is(T == bool)) { *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; return true; } else static if(is(T == K[], K)) { K tmp; if(name == paramName) { // direct - set and append if(setVariable(name, paramName, &tmp, value)) { (*what) ~= tmp; return true; } else { return false; } } else { // child, append to last element // FIXME: what about range violations??? auto ptr = &(*what)[(*what).length - 1]; return setVariable(name, paramName, ptr, value); } } else static if(is(T == V[K], K, V)) { // assoc array, name[key] is valid if(name == paramName) { // no action necessary return true; } else if(name[paramName.length] == '[') { int count = 1; auto idx = paramName.length + 1; while(idx < name.length && count > 0) { if(name[idx] == '[') count++; else if(name[idx] == ']') { count--; if(count == 0) break; } idx++; } if(idx == name.length) return false; // malformed auto insideBrackets = name[paramName.length + 1 .. idx]; auto afterName = name[idx + 1 .. $]; auto k = to!K(insideBrackets); V v; if(auto ptr = k in *what) v = *ptr; name = name[0 .. paramName.length]; //writeln(name, afterName, " ", paramName); auto ret = setVariable(name ~ afterName, paramName, &v, value); if(ret) { (*what)[k] = v; return true; } } return false; } else { static assert(0, "unsupported type for cgi call " ~ T.stringof); } //return false; } void setArgument(string name, string value) { int p; foreach(ch; name) { if(ch == '.' || ch == '[') break; p++; } auto paramName = name[0 .. p]; sw: switch(paramName) { static if(is(typeof(method) P == __parameters)) foreach(idx, param; P) { static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { // cannot be set from the outside } else { case idents[idx]: static if(is(param == Cgi.UploadedFile)) { params[idx] = cgi.files[name]; } else { setVariable(name, paramName, ¶ms[idx], value); } break sw; } } default: // ignore; not relevant argument } } if(cgi.requestMethod == Cgi.RequestMethod.GET) { names = cgi.allGetNamesInOrder; values = cgi.allGetValuesInOrder; } else { names = cgi.allPostNamesInOrder; values = cgi.allPostValuesInOrder; } foreach(idx, name; names) { setArgument(name, values[idx]); } static if(is(ReturnType!method == void)) { typeof(null) ret; dg(params); } else { auto ret = dg(params); } // FIXME: format return values // options are: json, html, csv. // also may need to wrap in envelope format: none, html, or json. return ret; } private bool mustNotBeSetFromWebParams(T, attrs...)() { static if(is(T : const(Cgi))) { return true; } else static if(is(T : const(Session!D), D)) { return true; } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { return true; } else { foreach(uda; attrs) static if(is(uda == ifCalledFromWeb!func, alias func)) return true; return false; } } private bool hasIfCalledFromWeb(attrs...)() { foreach(uda; attrs) static if(is(uda == ifCalledFromWeb!func, alias func)) return true; return false; } /++ Implies POST path for the thing itself, then GET will get the automatic form. The given customizer, if present, will be called as a filter on the Form object. History: Added December 27, 2020 +/ template AutomaticForm(alias customizer) { } /++ This is meant to be returned by a function that takes a form POST submission. You want to set the url of the new resource it created, which is set as the http Location header for a "201 Created" result, and you can also set a separate destination for browser users, which it sets via a "Refresh" header. The `resourceRepresentation` should generally be the thing you just created, and it will be the body of the http response when formatted through the presenter. The exact thing is up to you - it could just return an id, or the whole object, or perhaps a partial object. Examples: --- class Test : WebObject { @(Cgi.RequestMethod.POST) CreatedResource!int makeThing(string value) { return CreatedResource!int(value.to!int, "/resources/id"); } } --- History: Added December 18, 2021 +/ struct CreatedResource(T) { static if(!is(T == void)) T resourceRepresentation; string resourceUrl; string refreshUrl; } /+ /++ This can be attached as a UDA to a handler to add a http Refresh header on a successful run. (It will not be attached if the function throws an exception.) This will refresh the browser the given number of seconds after the page loads, to the url returned by `urlFunc`, which can be either a static function or a member method of the current handler object. You might use this for a POST handler that is normally used from ajax, but you want it to degrade gracefully to a temporarily flashed message before reloading the main page. History: Added December 18, 2021 +/ struct Refresh(alias urlFunc) { int waitInSeconds; string url() { static if(__traits(isStaticFunction, urlFunc)) return urlFunc(); else static if(is(urlFunc : string)) return urlFunc; } } +/ /+ /++ Sets a filter to be run before A before function can do validations of params and log and stop the function from running. +/ template Before(alias b) {} template After(alias b) {} +/ /+ Argument conversions: for the most part, it is to!Thing(string). But arrays and structs are a bit different. Arrays come from the cgi array. Thus they are passed arr=foo&arr=bar <-- notice the same name. Structs are first declared with an empty thing, then have their members set individually, with dot notation. The members are not required, just the initial declaration. struct Foo { int a; string b; } void test(Foo foo){} foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members Arrays of structs use this declaration. void test(Foo[] foo) {} foo&foo.a=5&foo.b=bar&foo&foo.a=9 You can use a hidden input field in HTML forms to achieve this. The value of the naked name declaration is ignored. Mind that order matters! The declaration MUST come first in the string. Arrays of struct members follow this rule recursively. struct Foo { int[] a; } foo&foo.a=1&foo.a=2&foo&foo.a=1 Associative arrays are formatted with brackets, after a declaration, like structs: foo&foo[key]=value&foo[other_key]=value Note: for maximum compatibility with outside code, keep your types simple. Some libraries do not support the strict ordering requirements to work with these struct protocols. FIXME: also perhaps accept application/json to better work with outside trash. Return values are also auto-formatted according to user-requested type: for json, it loops over and converts. for html, basic types are strings. Arrays are
    . Structs are
    . Arrays of structs are tables! +/ /++ A web presenter is responsible for rendering things to HTML to be usable in a web browser. They are passed as template arguments to the base classes of [WebObject] Responsible for displaying stuff as HTML. You can put this into your own aggregate and override it. Use forwarding and specialization to customize it. When you inherit from it, pass your own class as the CRTP argument. This lets the base class templates and your overridden templates work with each other. --- class MyPresenter : WebPresenter!(MyPresenter) { @Override void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { // present the CustomType } @Override void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { // handle everything else via the super class, which will call // back to your class when appropriate super.presentSuccessfulReturnAsHtml(cgi, ret); } } --- The meta argument in there can be overridden by your own facility. +/ class WebPresenter(CRTP) { /// A UDA version of the built-in `override`, to be used for static template polymorphism /// If you override a plain method, use `override`. If a template, use `@Override`. enum Override; string script() { return ` `; } string style() { return ` :root { --mild-border: #ccc; --middle-border: #999; --accent-color: #f2f2f2; --sidebar-color: #fefefe; } ` ~ genericFormStyling() ~ genericSiteStyling(); } string genericFormStyling() { return q"css table.automatic-data-display { border-collapse: collapse; border: solid 1px var(--mild-border); } table.automatic-data-display td { vertical-align: top; border: solid 1px var(--mild-border); padding: 2px 4px; } table.automatic-data-display th { border: solid 1px var(--mild-border); border-bottom: solid 1px var(--middle-border); padding: 2px 4px; } ol.automatic-data-display { margin: 0px; list-style-position: inside; padding: 0px; } dl.automatic-data-display { } .automatic-form { max-width: 600px; } .form-field { margin: 0.5em; padding-left: 0.5em; } .label-text { display: block; font-weight: bold; margin-left: -0.5em; } .submit-button-holder { padding-left: 2em; } .add-array-button { } css"; } string genericSiteStyling() { return q"css * { box-sizing: border-box; } html, body { margin: 0px; } body { font-family: sans-serif; } header { background: var(--accent-color); height: 64px; } footer { background: var(--accent-color); height: 64px; } #site-container { display: flex; } main { flex: 1 1 auto; order: 2; min-height: calc(100vh - 64px - 64px); padding: 4px; padding-left: 1em; } #sidebar { flex: 0 0 16em; order: 1; background: var(--sidebar-color); } css"; } import arsd.dom; Element htmlContainer() { auto document = new Document(q"html D Application
    html", true, true); return document.requireSelector("main"); } /// Renders a response as an HTTP error void renderBasicError(Cgi cgi, int httpErrorCode) { cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); auto c = htmlContainer(); c.innerText = getHttpCodeText(httpErrorCode); cgi.setResponseContentType("text/html; charset=utf-8"); cgi.write(c.parentDocument.toString(), true); } template methodMeta(alias method) { enum methodMeta = null; } void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { switch(format) { case "html": (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); break; case "json": import arsd.jsvar; static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { var json; foreach(index, type; Types) { if(ret.contains == index) json = ret.payload[index]; } } else { var json = ret; } var envelope = json; // var.emptyObject; /* envelope.success = true; envelope.result = json; envelope.error = null; */ cgi.setResponseContentType("application/json"); cgi.write(envelope.toJson(), true); break; default: cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. } } /// typeof(null) (which is also used to represent functions returning `void`) do nothing /// in the default presenter - allowing the function to have full low-level control over the /// response. void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { // nothing intentionally! } /// Redirections are forwarded to [Cgi.setResponseLocation] void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); } /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { cgi.setResponseStatus(getHttpCodeText(201)); if(ret.resourceUrl.length) cgi.header("Location: " ~ ret.resourceUrl); if(ret.refreshUrl.length) cgi.header("Refresh: 0;" ~ ret.refreshUrl); static if(!is(R == void)) presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); } /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { bool outputted = false; foreach(index, type; Types) { if(ret.contains == index) { assert(!outputted); outputted = true; (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); } } if(!outputted) assert(0); } /++ An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. +/ void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { cgi.setCache(true); // not necessarily true but meh if(auto fn = ret.filename()) { cgi.header("Content-Disposition: attachment; filename="~fn~";"); } cgi.setResponseContentType(ret.contentType); cgi.write(ret.getData(), true); } /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { auto container = this.htmlContainer(); container.appendChild(formatReturnValueAsHtml(ret)); cgi.write(container.parentDocument.toString(), true); } /++ History: Added January 23, 2023 (dub v11.0) +/ void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { switch(format) { case "html": presentExceptionAsHtml(cgi, t, meta); break; default: } } /++ If you override this, you will need to cast the exception type `t` dynamically, but can then use the template arguments here to refer back to the function. `func` is an alias to the method itself, and `dg` is a callable delegate to the same method on the live object. You could, in theory, change arguments and retry, but I provide that information mostly with the expectation that you will use them to make useful forms or richer error messages for the user. History: BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. I removed this in favor of a `Meta` param. Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` If you used the func for something, move that something into your `methodMeta` template. What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with enabling an easier implementation of [presentExceptionalReturn]. +/ void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { Form af; /+ foreach(attr; __traits(getAttributes, func)) { static if(__traits(isSame, attr, AutomaticForm)) { af = createAutomaticFormForFunction!(func)(dg); } } +/ presentExceptionAsHtmlImpl(cgi, t, af); } void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { if(auto e = cast(ResourceNotFoundException) t) { auto container = this.htmlContainer(); container.addChild("p", e.msg); if(!cgi.outputtedResponseData) cgi.setResponseStatus("404 Not Found"); cgi.write(container.parentDocument.toString(), true); } else if(auto mae = cast(MissingArgumentException) t) { if(automaticForm is null) goto generic; auto container = this.htmlContainer(); if(cgi.requestMethod == Cgi.RequestMethod.POST) container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); container.appendChild(automaticForm); cgi.write(container.parentDocument.toString(), true); } else { generic: auto container = this.htmlContainer(); // import std.stdio; writeln(t.toString()); container.appendChild(exceptionToElement(t)); container.addChild("h4", "GET"); foreach(k, v; cgi.get) { auto deets = container.addChild("details"); deets.addChild("summary", k); deets.addChild("div", v); } container.addChild("h4", "POST"); foreach(k, v; cgi.post) { auto deets = container.addChild("details"); deets.addChild("summary", k); deets.addChild("div", v); } if(!cgi.outputtedResponseData) cgi.setResponseStatus("500 Internal Server Error"); cgi.write(container.parentDocument.toString(), true); } } Element exceptionToElement(Throwable t) { auto div = Element.make("div"); div.addClass("exception-display"); div.addChild("p", t.msg); div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); auto pre = div.addChild("pre"); string s; s = t.toString(); Element currentBox; bool on = false; foreach(line; s.splitLines) { if(!on && line.startsWith("-----")) on = true; if(!on) continue; if(line.indexOf("arsd/") != -1) { if(currentBox is null) { currentBox = pre.addChild("details"); currentBox.addChild("summary", "Framework code"); } currentBox.addChild("span", line ~ "\n"); } else { pre.addChild("span", line ~ "\n"); currentBox = null; } } return div; } /++ Returns an element for a particular type +/ Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { import std.traits; auto div = Element.make("div"); div.addClass("form-field"); static if(is(T == Cgi.UploadedFile)) { Element lbl; if(displayName !is null) { lbl = div.addChild("label"); lbl.addChild("span", displayName, "label-text"); lbl.appendText(" "); } else { lbl = div; } auto i = lbl.addChild("input", name); i.attrs.name = name; i.attrs.type = "file"; } else static if(is(T == enum)) { Element lbl; if(displayName !is null) { lbl = div.addChild("label"); lbl.addChild("span", displayName, "label-text"); lbl.appendText(" "); } else { lbl = div; } auto i = lbl.addChild("select", name); i.attrs.name = name; foreach(memberName; __traits(allMembers, T)) i.addChild("option", memberName); } else static if(is(T == struct)) { if(displayName !is null) div.addChild("span", displayName, "label-text"); auto fieldset = div.addChild("fieldset"); fieldset.addChild("legend", beautify(T.stringof)); // FIXME fieldset.addChild("input", name); foreach(idx, memberName; __traits(allMembers, T)) static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); } } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { Element lbl; if(displayName !is null) { lbl = div.addChild("label"); lbl.addChild("span", displayName, "label-text"); lbl.appendText(" "); } else { lbl = div; } Element i; if(udaSuggestion) { i = udaSuggestion(); lbl.appendChild(i); } else { i = lbl.addChild("input", name); } i.attrs.name = name; static if(isSomeString!T) i.attrs.type = "text"; else i.attrs.type = "number"; if(i.tagName == "textarea") i.textContent = to!string(T.init); else i.attrs.value = to!string(T.init); } else static if(is(T == bool)) { Element lbl; if(displayName !is null) { lbl = div.addChild("label"); lbl.addChild("span", displayName, "label-text"); lbl.appendText(" "); } else { lbl = div; } auto i = lbl.addChild("input", name); i.attrs.type = "checkbox"; i.attrs.value = "true"; i.attrs.name = name; } else static if(is(T == K[], K)) { auto templ = div.addChild("template"); templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); if(displayName !is null) div.addChild("span", displayName, "label-text"); auto btn = div.addChild("button"); btn.addClass("add-array-button"); btn.attrs.type = "button"; btn.innerText = "Add"; btn.attrs.onclick = q{ var a = document.importNode(this.parentNode.firstChild.content, true); this.parentNode.insertBefore(a, this); }; } else static if(is(T == V[K], K, V)) { div.innerText = "assoc array not implemented for automatic form at this time"; } else { static assert(0, "unsupported type for cgi call " ~ T.stringof); } return div; } /// creates a form for gathering the function's arguments Form createAutomaticFormForFunction(alias method, T)(T dg) { auto form = cast(Form) Element.make("form"); form.method = "POST"; // FIXME form.addClass("automatic-form"); string formDisplayName = beautify(__traits(identifier, method)); foreach(attr; __traits(getAttributes, method)) static if(is(typeof(attr) == DisplayName)) formDisplayName = attr.name; form.addChild("h3", formDisplayName); import std.traits; //Parameters!method params; //alias idents = ParameterIdentifierTuple!method; //alias defaults = ParameterDefaults!method; static if(is(typeof(method) P == __parameters)) foreach(idx, _; P) {{ alias param = P[idx .. idx + 1]; static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { string displayName = beautify(__traits(identifier, param)); Element function() element; foreach(attr; __traits(getAttributes, param)) { static if(is(typeof(attr) == DisplayName)) displayName = attr.name; else static if(is(typeof(attr) : typeof(element))) { element = attr; } } auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); if(i.querySelector("input[type=file]") !is null) form.setAttribute("enctype", "multipart/form-data"); } }} form.addChild("div", Html(``), "submit-button-holder"); return form; } /// creates a form for gathering object members (for the REST object thing right now) Form createAutomaticFormForObject(T)(T obj) { auto form = cast(Form) Element.make("form"); form.addClass("automatic-form"); form.addChild("h3", beautify(__traits(identifier, T))); import std.traits; //Parameters!method params; //alias idents = ParameterIdentifierTuple!method; //alias defaults = ParameterDefaults!method; foreach(idx, memberName; __traits(derivedMembers, T)) {{ static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { string displayName = beautify(memberName); Element function() element; foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) static if(is(typeof(attr) == DisplayName)) displayName = attr.name; else static if(is(typeof(attr) : typeof(element))) element = attr; form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); }}} form.addChild("div", Html(``), "submit-button-holder"); return form; } /// Element formatReturnValueAsHtml(T)(T t) { import std.traits; static if(is(T == typeof(null))) { return Element.make("span"); } else static if(is(T : Element)) { return t; } else static if(is(T == MultipleResponses!Types, Types...)) { foreach(index, type; Types) { if(t.contains == index) return formatReturnValueAsHtml(t.payload[index]); } assert(0); } else static if(is(T == Paginated!E, E)) { auto e = Element.make("div").addClass("paginated-result"); e.appendChild(formatReturnValueAsHtml(t.items)); if(t.nextPageUrl.length) e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); return e; } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { return Element.make("span", to!string(t), "automatic-data-display"); } else static if(is(T == V[K], K, V)) { auto dl = Element.make("dl"); dl.addClass("automatic-data-display associative-array"); foreach(k, v; t) { dl.addChild("dt", to!string(k)); dl.addChild("dd", formatReturnValueAsHtml(v)); } return dl; } else static if(is(T == struct)) { auto dl = Element.make("dl"); dl.addClass("automatic-data-display struct"); foreach(idx, memberName; __traits(allMembers, T)) static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { dl.addChild("dt", beautify(memberName)); dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); } return dl; } else static if(is(T == bool)) { return Element.make("span", t ? "true" : "false", "automatic-data-display"); } else static if(is(T == E[], E)) { static if(is(E : RestObject!Proxy, Proxy)) { // treat RestObject similar to struct auto table = cast(Table) Element.make("table"); table.addClass("automatic-data-display"); string[] names; foreach(idx, memberName; __traits(derivedMembers, E)) static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { names ~= beautify(memberName); } table.appendHeaderRow(names); foreach(l; t) { auto tr = table.appendRow(); foreach(idx, memberName; __traits(derivedMembers, E)) static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { static if(memberName == "id") { string val = to!string(__traits(getMember, l, memberName)); tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME } else { tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); } } } return table; } else static if(is(E == struct)) { // an array of structs is kinda special in that I like // having those formatted as tables. auto table = cast(Table) Element.make("table"); table.addClass("automatic-data-display"); string[] names; foreach(idx, memberName; __traits(allMembers, E)) static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { names ~= beautify(memberName); } table.appendHeaderRow(names); foreach(l; t) { auto tr = table.appendRow(); foreach(idx, memberName; __traits(allMembers, E)) static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); } } return table; } else { // otherwise, I will just make a list. auto ol = Element.make("ol"); ol.addClass("automatic-data-display"); foreach(e; t) ol.addChild("li", formatReturnValueAsHtml(e)); return ol; } } else static if(is(T : Object)) { static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface return Element.make("div", t.toHtml()); else return Element.make("div", t.toString()); } else static assert(0, "bad return value for cgi call " ~ T.stringof); assert(0); } } /++ The base class for the [dispatcher] function and object support. +/ class WebObject { //protected Cgi cgi; protected void initialize(Cgi cgi) { //this.cgi = cgi; } } /++ Can return one of the given types, decided at runtime. The syntax is to declare all the possible types in the return value, then you can `return typeof(return)(...value...)` to construct it. It has an auto-generated constructor for each value it can hold. --- MultipleResponses!(Redirection, string) getData(int how) { if(how & 1) return typeof(return)(Redirection("http://dpldocs.info/")); else return typeof(return)("hi there!"); } --- If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. +/ struct MultipleResponses(T...) { private size_t contains; private union { private T payload; } static foreach(index, type; T) public this(type t) { contains = index; payload[index] = t; } /++ This is primarily for testing. It is your way of getting to the response. Let's say you wanted to test that one holding a Redirection and a string actually holds a string, by name of "test": --- auto valueToTest = your_test_function(); valueToTest.visit( (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test (string s) { assert(s == "test"); } // right value, go ahead and test it. ); --- History: Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. It tried to use alias lambdas before, but runtime delegates work much better so I changed it. +/ void visit(Handlers...)(Handlers handlers) { template findHandler(type, int count, HandlersToCheck...) { static if(HandlersToCheck.length == 0) enum findHandler = -1; else { static if(is(typeof(HandlersToCheck[0].init(type.init)))) enum findHandler = count; else enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); } } foreach(index, type; T) { enum handlerIndex = findHandler!(type, 0, Handlers); static if(handlerIndex == -1) static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); else { if(index == this.contains) handlers[handlerIndex](this.payload[index]); } } } /+ auto toArsdJsvar()() { import arsd.jsvar; return var(null); } +/ } // FIXME: implement this somewhere maybe struct RawResponse { int code; string[] headers; const(ubyte)[] responseBody; } /++ You can return this from [WebObject] subclasses for redirections. (though note the static types means that class must ALWAYS redirect if you return this directly. You might want to return [MultipleResponses] if it can be conditional) +/ struct Redirection { string to; /// The URL to redirect to. int code = 303; /// The HTTP code to return. } /++ Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overridden the presenter in the dispatcher. FIXME: explain this better You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, the runtime result of that is undefined. A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. (this might change, like maybe i will use pure as an indicator GET is ok. idk.) $(WARNING --- // legal in D, undefined runtime behavior with cgi.d, it may call either method // even if you put different URL udas on it, the current code ignores them. void foo(int a) {} void foo(string a) {} --- ) See_Also: [serveRestObject], [serveStaticFile] +/ auto serveApi(T)(string urlPrefix) { assert(urlPrefix[$ - 1] == '/'); return serveApiInternal!T(urlPrefix); } private string nextPieceFromSlash(ref string remainingUrl) { if(remainingUrl.length == 0) return remainingUrl; int slash = 0; while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') slash++; // I am specifically passing `null` to differentiate it vs empty string // so in your ctor, `items` means new T(null) and `items/` means new T("") auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; // so if it is the last item, the dot can be used to load an alternative view // otherwise tho the dot is considered part of the identifier // FIXME // again notice "" vs null here! if(slash == remainingUrl.length) remainingUrl = null; else remainingUrl = remainingUrl[slash + 1 .. $]; return ident; } /++ UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. +/ enum AddTrailingSlash; /// ditto enum RemoveTrailingSlash; private auto serveApiInternal(T)(string urlPrefix) { import arsd.dom; import arsd.jsvar; static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; try { // see duplicated code below by searching subresource_ctor // also see mustNotBeSetFromWebParams static if(is(typeof(T.__ctor) P == __parameters)) { P params; foreach(pidx, param; P) { static if(is(param : Cgi)) { static assert(!is(param == immutable)); cast() params[pidx] = cgi; } else static if(is(param == Session!D, D)) { static assert(!is(param == immutable)); cast() params[pidx] = cgi.getSessionObject!D(); } else { static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { static if(is(uda == ifCalledFromWeb!func, alias func)) { static if(is(typeof(func(cgi)))) params[pidx] = func(cgi); else params[pidx] = func(); } } } else { static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { params[pidx] = param.getAutomaticallyForCgi(cgi); } else static if(is(param == string)) { auto ident = nextPieceFromSlash(remainingUrl); params[pidx] = ident; } else static assert(0, "illegal type for subresource " ~ param.stringof); } } } auto obj = new T(params); } else { auto obj = new T(); } return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); } catch(Throwable t) { switch(cgi.request("format", "html")) { case "html": static void dummy() {} presenter.presentExceptionAsHtml(cgi, t, null); return true; case "json": var envelope = var.emptyObject; envelope.success = false; envelope.result = null; envelope.error = t.toString(); cgi.setResponseContentType("application/json"); cgi.write(envelope.toJson(), true); return true; default: throw t; // return true; } // return true; } assert(0); } static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { obj.initialize(cgi); /+ Overload rules: Any unique combination of HTTP verb and url path can be dispatched to function overloads statically. Moreover, some args vs no args can be overloaded dynamically. +/ auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); /+ auto orig = remainingUrl; assert(0, (orig is null ? "__null" : orig) ~ " .. " ~ (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); +/ if(methodNameFromUrl is null) methodNameFromUrl = "__null"; string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; if(remainingUrl.length) hack ~= "/"; switch(hack) { foreach(methodName; __traits(derivedMembers, T)) static if(methodName != "__ctor") foreach(idx, overload; __traits(getOverloads, T, methodName)) { static if(is(typeof(overload) P == __parameters)) static if(is(typeof(overload) R == return)) static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") { static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) case urlNameForMethod: static if(is(R : WebObject)) { // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string // subresource_ctor // also see mustNotBeSetFromWebParams P params; string ident; foreach(pidx, param; P) { static if(is(param : Cgi)) { static assert(!is(param == immutable)); cast() params[pidx] = cgi; } else static if(is(param == typeof(presenter))) { cast() param[pidx] = presenter; } else static if(is(param == Session!D, D)) { static assert(!is(param == immutable)); cast() params[pidx] = cgi.getSessionObject!D(); } else { static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { static if(is(uda == ifCalledFromWeb!func, alias func)) { static if(is(typeof(func(cgi)))) params[pidx] = func(cgi); else params[pidx] = func(); } } } else { static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { params[pidx] = param.getAutomaticallyForCgi(cgi); } else static if(is(param == string)) { ident = nextPieceFromSlash(remainingUrl); if(ident is null) { // trailing slash mandated on subresources cgi.setResponseLocation(cgi.pathInfo ~ "/"); return true; } else { params[pidx] = ident; } } else static assert(0, "illegal type for subresource " ~ param.stringof); } } } auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); } else { // 404 it if any url left - not a subresource means we don't get to play with that! if(remainingUrl.length) return false; bool automaticForm; foreach(attr; __traits(getAttributes, overload)) static if(is(attr == AddTrailingSlash)) { if(remainingUrl is null) { cgi.setResponseLocation(cgi.pathInfo ~ "/"); return true; } } else static if(is(attr == RemoveTrailingSlash)) { if(remainingUrl !is null) { cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); return true; } } else static if(__traits(isSame, AutomaticForm, attr)) { automaticForm = true; } /+ int zeroArgOverload = -1; int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; bool calledWithZeroArgs = true; foreach(k, v; cgi.get) if(k != "format") { calledWithZeroArgs = false; break; } foreach(k, v; cgi.post) if(k != "format") { calledWithZeroArgs = false; break; } // first, we need to go through and see if there is an empty one, since that // changes inside. But otherwise, all the stuff I care about can be done via // simple looping (other improper overloads might be flagged for runtime semantic check) // // an argument of type Cgi is ignored for these purposes static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ static if(is(typeof(overload) P == __parameters)) static if(P.length == 0) zeroArgOverload = cast(int) idx; else static if(P.length == 1 && is(P[0] : Cgi)) zeroArgOverload = cast(int) idx; }} // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. bool overloadHasBeenCalled = false; static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ bool callFunction = true; // there is a zero arg overload and this is NOT it, and we have zero args - don't call this if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) callFunction = false; // if this is the zero-arg overload, obviously it cannot be called if we got any args. if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) callFunction = false; // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. bool hadAnyMethodRestrictions = false; bool foundAcceptableMethod = false; foreach(attr; __traits(getAttributes, overload)) { static if(is(typeof(attr) == Cgi.RequestMethod)) { hadAnyMethodRestrictions = true; if(attr == cgi.requestMethod) foundAcceptableMethod = true; } } if(hadAnyMethodRestrictions && !foundAcceptableMethod) callFunction = false; /+ The overloads we really want to allow are the sane ones from the web perspective. Which is likely on HTTP verbs, for the most part, but might also be potentially based on some args vs zero args, or on argument names. Can't really do argument types very reliable through the web though; those should probably be different URLs. Even names I feel is better done inside the function, so I'm not going to support that here. But the HTTP verbs and zero vs some args makes sense - it lets you define custom forms pretty easily. Moreover, I'm of the opinion that empty overload really only makes sense on GET for this case. On a POST, it is just a missing argument exception and that should be handled by the presenter. But meh, I'll let the user define that, D only allows one empty arg thing anyway so the method UDAs are irrelevant. +/ if(callFunction) +/ auto format = cgi.request("format", defaultFormat!overload()); auto wantsFormFormat = format.startsWith("form-"); if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { // Should I still show the form on a json thing? idk... auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); return true; } try { // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); } catch(Throwable t) { // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); } return true; //}} //cgi.header("Accept: POST"); // FIXME list the real thing //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. //return true; } } } case "GET script.js": cgi.setResponseContentType("text/javascript"); cgi.gzipResponse = true; cgi.write(presenter.script(), true); return true; case "GET style.css": cgi.setResponseContentType("text/css"); cgi.gzipResponse = true; cgi.write(presenter.style(), true); return true; default: return false; } assert(0); } return DispatcherDefinition!internalHandler(urlPrefix, false); } string defaultFormat(alias method)() { bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; foreach(attr; __traits(getAttributes, method)) { static if(is(typeof(attr) == DefaultFormat)) { if(nonConstConditionForWorkingAroundASpuriousDmdWarning) return attr.value; } } return "html"; } struct Paginated(T) { T[] items; string nextPageUrl; } template urlNamesForMethod(alias method, string default_) { string[] helper() { auto verb = Cgi.RequestMethod.GET; bool foundVerb = false; bool foundNoun = false; string def = default_; bool hasAutomaticForm = false; foreach(attr; __traits(getAttributes, method)) { static if(is(typeof(attr) == Cgi.RequestMethod)) { verb = attr; if(foundVerb) assert(0, "Multiple http verbs on one function is not currently supported"); foundVerb = true; } static if(is(typeof(attr) == UrlName)) { if(foundNoun) assert(0, "Multiple url names on one function is not currently supported"); foundNoun = true; def = attr.name; } static if(__traits(isSame, attr, AutomaticForm)) { hasAutomaticForm = true; } } if(def is null) def = "__null"; string[] ret; static if(is(typeof(method) R == return)) { static if(is(R : WebObject)) { def ~= "/"; foreach(v; __traits(allMembers, Cgi.RequestMethod)) ret ~= v ~ " " ~ def; } else { if(hasAutomaticForm) { ret ~= "GET " ~ def; ret ~= "POST " ~ def; } else { ret ~= to!string(verb) ~ " " ~ def; } } } else static assert(0); return ret; } enum urlNamesForMethod = helper(); } enum AccessCheck { allowed, denied, nonExistent, } enum Operation { show, create, replace, remove, update } enum UpdateResult { accessDenied, noSuchResource, success, failure, unnecessary } enum ValidationResult { valid, invalid } /++ The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. WARNING: this is not stable. +/ class RestObject(CRTP) : WebObject { import arsd.dom; import arsd.jsvar; /// Prepare the object to be shown. void show() {} /// ditto void show(string urlId) { load(urlId); show(); } /// Override this to provide access control to this object. AccessCheck accessCheck(string urlId, Operation operation) { return AccessCheck.allowed; } ValidationResult validate() { // FIXME return ValidationResult.valid; } string getUrlSlug() { import std.conv; static if(is(typeof(CRTP.id))) return to!string((cast(CRTP) this).id); else return null; } // The functions with more arguments are the low-level ones, // they forward to the ones with fewer arguments by default. // POST on a parent collection - this is called from a collection class after the members are updated /++ Given a populated object, this creates a new entry. Returns the url identifier of the new object. +/ string create(scope void delegate() applyChanges) { applyChanges(); save(); return getUrlSlug(); } void replace() { save(); } void replace(string urlId, scope void delegate() applyChanges) { load(urlId); applyChanges(); replace(); } void update(string[] fieldList) { save(); } void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { load(urlId); applyChanges(); update(fieldList); } void remove() {} void remove(string urlId) { load(urlId); remove(); } abstract void load(string urlId); abstract void save(); Element toHtml(Presenter)(Presenter presenter) { import arsd.dom; import std.conv; auto obj = cast(CRTP) this; auto div = Element.make("div"); div.addClass("Dclass_" ~ CRTP.stringof); div.dataset.url = getUrlSlug(); bool first = true; foreach(idx, memberName; __traits(derivedMembers, CRTP)) static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { if(!first) div.addChild("br"); else first = false; div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); } return div; } var toJson() { import arsd.jsvar; var v = var.emptyObject(); auto obj = cast(CRTP) this; foreach(idx, memberName; __traits(derivedMembers, CRTP)) static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { v[memberName] = __traits(getMember, obj, memberName); } return v; } /+ auto structOf(this This) { } +/ } // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value // https://use-the-index-luke.com/sql/partial-results/fetch-next-page /++ Base class for REST collections. +/ class CollectionOf(Obj) : RestObject!(CollectionOf) { /// You might subclass this and use the cgi object's query params /// to implement a search filter, for example. /// /// FIXME: design a way to auto-generate that form /// (other than using the WebObject thing above lol // it'll prolly just be some searchParams UDA or maybe an enum. // // pagination too perhaps. // // and sorting too IndexResult index() { return IndexResult.init; } string[] sortableFields() { return null; } string[] searchableFields() { return null; } struct IndexResult { Obj[] results; string[] sortableFields; string previousPageIdentifier; string nextPageIdentifier; string firstPageIdentifier; string lastPageIdentifier; int numberOfPages; } override string create(scope void delegate() applyChanges) { assert(0); } override void load(string urlId) { assert(0); } override void save() { assert(0); } override void show() { index(); } override void show(string urlId) { show(); } /// Proxy POST requests (create calls) to the child collection alias PostProxy = Obj; } /++ Serves a REST object, similar to a Ruby on Rails resource. You put data members in your class. cgi.d will automatically make something out of those. It will call your constructor with the ID from the URL. This may be null. It will then populate the data members from the request. It will then call a method, if present, telling what happened. You don't need to write these! It finally returns a reply. Your methods are passed a list of fields it actually set. The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better with relative linking. But meh.) GET /items -> index. all values not set. GET /items/id -> get. only ID will be set, other params ignored. POST /items -> create. values set as given PUT /items/id -> replace. values set as given or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. PATCH /items/id -> update. values set as given, list of changed fields passed or POST /items/id with cgi.post["_method"] == "PATCH" DELETE /items/id -> destroy. only ID guaranteed to be set or POST /items/id with cgi.post["_method"] == "DELETE" Following the stupid convention, there will never be a trailing slash here, and if it is there, it will redirect you away from it. API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. I will also let you change the default, if you must. // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. You can define sub-resources on your object inside the object. These sub-resources are also REST objects that follow the same thing. They may be individual resources or collections themselves. Your class is expected to have at least the following methods: FIXME: i kinda wanna add a routes object to the initialize call create Create returns the new address on success, some code on failure. show index update remove You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or something like that. Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. NOT IMPLEMENTED Really, a collection is a resource with a bunch of subresources. GET /items index because it is GET on the top resource GET /items/foo item but different than items? class Items { } ... but meh, a collection can be automated. not worth making it a separate thing, let's look at a real example. Users has many items and a virtual one, /users/current. the individual users have properties and two sub-resources: session, which is just one, and comments, a collection. class User : RestObject!() { // no parent int id; string name; // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. // but you can override them to do it differently. // any member which is of type RestObject can be linked automatically via href btw. void show() {} void show(string urlId) {} // automated! GET of this specific thing void create() {} // POST on a parent collection - this is called from a collection class after the members are updated void replace(string urlId) {} // this is the PUT; really, it just updates all fields. void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. void remove(string urlId) {} // DELETE void load(string urlId) {} // the default implementation of show() populates the id, then this() {} mixin Subresource!Session; mixin Subresource!Comment; } class Session : RestObject!() { // the parent object may not be fully constructed/loaded this(User parent) {} } class Comment : CollectionOf!Comment { this(User parent) {} } class Users : CollectionOf!User { // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child } +/ auto serveRestObject(T)(string urlPrefix) { assert(urlPrefix[0] == '/'); assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { string url = cgi.pathInfo[urlPrefix.length .. $]; if(url.length && url[$ - 1] == '/') { // remove the final slash... cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); return true; } return restObjectServeHandler!T(cgi, presenter, url); } return DispatcherDefinition!internalHandler(urlPrefix, false); } /+ /// Convenience method for serving a collection. It will be named the same /// as type T, just with an s at the end. If you need any further, just /// write the class yourself. auto serveRestCollectionOf(T)(string urlPrefix) { assert(urlPrefix[0] == '/'); mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); } +/ bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { string urlId = null; if(url.length && url[0] == '/') { // asking for a subobject urlId = url[1 .. $]; foreach(idx, ch; urlId) { if(ch == '/') { urlId = urlId[0 .. idx]; break; } } } // FIXME handle other subresources static if(is(T : CollectionOf!(C), C)) { if(urlId !is null) { return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); } } // FIXME: support precondition failed, if-modified-since, expectation failed, etc. auto obj = new T(); obj.initialize(cgi); // FIXME: populate reflection info delegates // FIXME: I am not happy with this. switch(urlId) { case "script.js": cgi.setResponseContentType("text/javascript"); cgi.gzipResponse = true; cgi.write(presenter.script(), true); return true; case "style.css": cgi.setResponseContentType("text/css"); cgi.gzipResponse = true; cgi.write(presenter.style(), true); return true; default: // intentionally blank } static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { foreach(idx, memberName; __traits(derivedMembers, Obj)) static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); } } void applyChanges() { applyChangesTemplate(cgi, obj); } string[] modifiedList; void writeObject(bool addFormLinks) { if(cgi.request("format") == "json") { cgi.setResponseContentType("application/json"); cgi.write(obj.toJson().toString, true); } else { auto container = presenter.htmlContainer(); if(addFormLinks) { static if(is(T : CollectionOf!(C), C)) container.appendHtml(` `); else container.appendHtml(` Back
    `); } container.appendChild(obj.toHtml(presenter)); cgi.write(container.parentDocument.toString, true); } } // FIXME: I think I need a set type in here.... // it will be nice to pass sets of members. try switch(cgi.requestMethod) { case Cgi.RequestMethod.GET: // I could prolly use template this parameters in the implementation above for some reflection stuff. // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... // automatic forms here for usable basic auto site from browser. // even if the format is json, it could actually send out the links and formats switch(cgi.request("_method", "GET")) { case "GET": static if(is(T : CollectionOf!(C), C)) { auto results = obj.index(); if(cgi.request("format", "html") == "html") { auto container = presenter.htmlContainer(); auto html = presenter.formatReturnValueAsHtml(results.results); container.appendHtml(`
    `); container.appendChild(html); cgi.write(container.parentDocument.toString, true); } else { cgi.setResponseContentType("application/json"); import arsd.jsvar; var json = var.emptyArray; foreach(r; results.results) { var o = var.emptyObject; foreach(idx, memberName; __traits(derivedMembers, typeof(r))) static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { o[memberName] = __traits(getMember, r, memberName); } json ~= o; } cgi.write(json.toJson(), true); } } else { obj.show(urlId); writeObject(true); } break; case "PATCH": obj.load(urlId); goto case; case "PUT": case "POST": // an editing form for the object auto container = presenter.htmlContainer(); static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); } else { auto form = presenter.createAutomaticFormForObject(obj); } form.attrs.method = "POST"; form.setValue("_method", cgi.request("_method", "GET")); container.appendChild(form); cgi.write(container.parentDocument.toString(), true); break; case "DELETE": // FIXME: a delete form for the object (can be phrased "are you sure?") auto container = presenter.htmlContainer(); container.appendHtml(`
    Are you sure you want to delete this item?
    `); cgi.write(container.parentDocument.toString(), true); break; default: cgi.write("bad method\n", true); } break; case Cgi.RequestMethod.POST: // this is to allow compatibility with HTML forms switch(cgi.request("_method", "POST")) { case "PUT": goto PUT; case "PATCH": goto PATCH; case "DELETE": goto DELETE; case "POST": static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { auto p = new obj.PostProxy(); void specialApplyChanges() { applyChangesTemplate(cgi, p); } string n = p.create(&specialApplyChanges); } else { string n = obj.create(&applyChanges); } auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; cgi.setResponseLocation(newUrl); cgi.setResponseStatus("201 Created"); cgi.write(`The object has been created.`); break; default: cgi.write("bad method\n", true); } // FIXME this should be valid on the collection, but not the child.... // 303 See Other break; case Cgi.RequestMethod.PUT: PUT: obj.replace(urlId, &applyChanges); writeObject(false); break; case Cgi.RequestMethod.PATCH: PATCH: obj.update(urlId, &applyChanges, modifiedList); writeObject(false); break; case Cgi.RequestMethod.DELETE: DELETE: obj.remove(urlId); cgi.setResponseStatus("204 No Content"); break; default: // FIXME: OPTIONS, HEAD } catch(Throwable t) { presenter.presentExceptionAsHtml(cgi, t); } return true; } /+ struct SetOfFields(T) { private void[0][string] storage; void set(string what) { //storage[what] = } void unset(string what) {} void setAll() {} void unsetAll() {} bool isPresent(string what) { return false; } } +/ /+ enum readonly; enum hideonindex; +/ /++ Returns true if I recommend gzipping content of this type. You might want to call it from your Presenter classes before calling cgi.write. --- cgi.setResponseContentType(yourContentType); cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); cgi.write(yourData, true); --- This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. History: Added January 28, 2023 (dub v11.0) +/ bool gzipRecommendedForContentType(string contentType) { if(contentType.startsWith("text/")) return true; if(contentType.startsWith("application/javascript")) return true; return false; } /++ Serves a static file. To be used with [dispatcher]. See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] +/ auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { // https://baus.net/on-tcp_cork/ // man 2 sendfile assert(urlPrefix[0] == '/'); if(filename is null) filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? if(contentType is null) { contentType = contentTypeFromFileExtension(filename); } static struct DispatcherDetails { string filename; string contentType; } static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) cgi.setCache(true); cgi.setResponseContentType(details.contentType); cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); cgi.write(std.file.read(details.filename), true); return true; } return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); } /++ Serves static data. To be used with [dispatcher]. History: Added October 31, 2021 +/ auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { assert(urlPrefix[0] == '/'); if(contentType is null) { contentType = contentTypeFromFileExtension(urlPrefix); } static struct DispatcherDetails { immutable(void)[] data; string contentType; } static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { cgi.setCache(true); cgi.setResponseContentType(details.contentType); cgi.write(details.data, true); return true; } return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); } string contentTypeFromFileExtension(string filename) { if(filename.endsWith(".png")) return "image/png"; if(filename.endsWith(".apng")) return "image/apng"; if(filename.endsWith(".svg")) return "image/svg+xml"; if(filename.endsWith(".jpg")) return "image/jpeg"; if(filename.endsWith(".html")) return "text/html"; if(filename.endsWith(".css")) return "text/css"; if(filename.endsWith(".js")) return "application/javascript"; if(filename.endsWith(".wasm")) return "application/wasm"; if(filename.endsWith(".mp3")) return "audio/mpeg"; if(filename.endsWith(".pdf")) return "application/pdf"; return null; } /// This serves a directory full of static files, figuring out the content-types from file extensions. /// It does not let you to descend into subdirectories (or ascend out of it, of course) auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { assert(urlPrefix[0] == '/'); assert(urlPrefix[$-1] == '/'); static struct DispatcherDetails { string directory; bool recursive; } if(directory is null) directory = urlPrefix[1 .. $]; if(directory.length == 0) directory = "./"; assert(directory[$-1] == '/'); static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct if(details.recursive) { // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier if(file.indexOf("\\") != -1) return false; import std.path; file = std.path.buildNormalizedPath(file); enum upOneDir = ".." ~ std.path.dirSeparator; // also no point doing any kind of up directory things since that makes it more likely to break out of the parent if(file == ".." || file.startsWith(upOneDir)) return false; if(std.path.isAbsolute(file)) return false; // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? // once it passes these filters it is probably ok. } else { if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) return false; } auto contentType = contentTypeFromFileExtension(file); auto fn = details.directory ~ file; if(std.file.exists(fn)) { //if(contentType.indexOf("image/") == 0) //cgi.setCache(true); //else if(contentType.indexOf("audio/") == 0) cgi.setCache(true); cgi.setResponseContentType(contentType); cgi.gzipResponse = gzipRecommendedForContentType(contentType); cgi.write(std.file.read(fn), true); return true; } else { return false; } } return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); } /++ Redirects one url to another See_Also: [dispatcher], [serveStaticFile] +/ auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { assert(urlPrefix[0] == '/'); static struct DispatcherDetails { string redirectTo; string code; } static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { cgi.setResponseLocation(details.redirectTo, true, details.code); return true; } return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); } /// Used exclusively with `dispatchTo` struct DispatcherData(Presenter) { Cgi cgi; /// You can use this cgi object. Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. } /++ Dispatches the URL to a specific function. +/ auto handleWith(alias handler)(string urlPrefix) { // cuz I'm too lazy to do it better right now static class Hack : WebObject { static import std.traits; @UrlName("") auto handle(std.traits.Parameters!handler args) { return handler(args); } } return urlPrefix.serveApiInternal!Hack; } /++ Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: --- bool other(DD)(DD dd) { return dd.dispatcher!( "/whatever".serveRedirect("/success"), "/api/".serveApi!MyClass ); } --- The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. Or, of course, you could just use the exact type in your own code. You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a good job. +/ auto dispatchTo(alias handler)(string urlPrefix) { assert(urlPrefix[0] == '/'); assert(urlPrefix[$-1] != '/'); static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); } return DispatcherDefinition!(internalHandler)(urlPrefix, false); } /++ See [serveStaticFile] if you want to serve a file off disk. History: Added January 28, 2023 (dub v11.0) +/ auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { assert(urlPrefix[0] == '/'); static struct DispatcherDetails { immutable(ubyte)[] data; string contentType; string filenameToSuggestAsDownload; } static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { cgi.setCache(true); cgi.setResponseContentType(details.contentType); if(details.filenameToSuggestAsDownload.length) cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); cgi.write(details.data, true); return true; } return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); } /++ Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. History: Added January 28, 2023 (dub v11.0) +/ alias KeepExistingPresenter = typeof(null); /++ For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, this issues the given errorCode and stops processing. --- bool hasAdminPermissions(Cgi cgi) { return true; } mixin DispatcherMain!( "/admin".dispatchSubsection!( passFilterOrIssueError!(hasAdminPermissions, 403), KeepExistingPresenter, "/".serveApi!AdminFunctions ) ); --- History: Added January 28, 2023 (dub v11.0) +/ template passFilterOrIssueError(alias filter, int errorCode) { bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { if(filter(dd.cgi)) return true; dd.presenter.renderBasicError(dd.cgi, errorCode); return false; } } /++ Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, and then be dispatched to their own handlers. --- /+ // a long-form filter function bool permissionCheck(DispatcherData)(DispatcherData dd) { // you are permitted to call mutable methods on the Cgi object // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data // though much of the request is immutable so there's only so much you're allowed to do to modify it. if(checkPermissionOnRequest(dd.cgi)) { return true; // OK, allow processing to continue } else { dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester return false; // and stop further processing into this subsection } } +/ // but you can also do short-form filters: bool permissionCheck(Cgi cgi) { return ("ok" in cgi.get) !is null; } // handler for the subsection class AdminClass : WebObject { int foo() { return 5; } } // handler for the main site class TheMainSite : WebObject {} mixin DispatcherMain!( "/admin".dispatchSubsection!( // converts our short-form filter into a long-form filter passFilterOrIssueError!(permissionCheck, 403), // can use a new presenter if wanted for the subsection KeepExistingPresenter, // and then provide child route dispatchers "/".serveApi!AdminClass ), // and back to the top level "/".serveApi!TheMainSite ); --- Note you can encapsulate sections in files like this: --- auto adminDispatcher(string urlPrefix) { return urlPrefix.dispatchSubsection!( .... ); } mixin DispatcherMain!( "/admin".adminDispatcher, // and so on ) --- If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. History: Added January 28, 2023 (dub v11.0) +/ auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { assert(urlPrefix[0] == '/'); assert(urlPrefix[$-1] != '/'); static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { static if(!is(PreRequestFilter == typeof(null))) { if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) return true; // we handled it by rejecting it } static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); } else { auto newPresenter = new NewPresenter(); return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); } } return DispatcherDefinition!(internalHandler)(urlPrefix, false); } /++ A URL dispatcher. --- if(cgi.dispatcher!( "/api/".serveApi!MyApiClass, "/objects/lol".serveRestObject!MyRestObject, "/file.js".serveStaticFile, "/admin/".dispatchTo!adminHandler )) return; --- You define a series of url prefixes followed by handlers. You may want to do different pre- and post- processing there, for example, an authorization check and different page layout. You can use different presenters and different function chains. See [dispatchSubsection] for details. [dispatchTo] will send the request to another function for handling. +/ template dispatcher(definitions...) { bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { static if(is(Presenter == typeof(null))) { static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} auto presenter = new GenericWebPresenter(); } else alias presenter = presenterArg; return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); } bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { // I can prolly make this more efficient later but meh. foreach(definition; definitions) { if(definition.rejectFurther) { if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { auto ret = definition.handler( dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], dispatcherData.cgi, dispatcherData.presenter, definition.details); if(ret) return true; } } else if( dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && // cgi.d dispatcher urls must be complete or have a /; // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') ) { auto ret = definition.handler( dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], dispatcherData.cgi, dispatcherData.presenter, definition.details); if(ret) return true; } } return false; } } }); private struct StackBuffer { char[1024] initial = void; char[] buffer; size_t position; this(int a) { buffer = initial[]; position = 0; } void add(in char[] what) { if(position + what.length > buffer.length) buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases buffer[position .. position + what.length] = what[]; position += what.length; } void add(in char[] w1, in char[] w2, in char[] w3 = null) { add(w1); add(w2); add(w3); } void add(long v) { char[16] buffer = void; auto pos = buffer.length; bool negative; if(v < 0) { negative = true; v = -v; } do { buffer[--pos] = cast(char) (v % 10 + '0'); v /= 10; } while(v); if(negative) buffer[--pos] = '-'; auto res = buffer[pos .. $]; add(res[]); } char[] get() @nogc { return buffer[0 .. position]; } } // duplicated in http2.d private static string getHttpCodeText(int code) pure nothrow @nogc { switch(code) { case 200: return "200 OK"; case 201: return "201 Created"; case 202: return "202 Accepted"; case 203: return "203 Non-Authoritative Information"; case 204: return "204 No Content"; case 205: return "205 Reset Content"; case 206: return "206 Partial Content"; // case 300: return "300 Multiple Choices"; case 301: return "301 Moved Permanently"; case 302: return "302 Found"; case 303: return "303 See Other"; case 304: return "304 Not Modified"; case 305: return "305 Use Proxy"; case 307: return "307 Temporary Redirect"; case 308: return "308 Permanent Redirect"; // case 400: return "400 Bad Request"; case 401: return "401 Unauthorized"; case 402: return "402 Payment Required"; case 403: return "403 Forbidden"; case 404: return "404 Not Found"; case 405: return "405 Method Not Allowed"; case 406: return "406 Not Acceptable"; case 407: return "407 Proxy Authentication Required"; case 408: return "408 Request Timeout"; case 409: return "409 Conflict"; case 410: return "410 Gone"; case 411: return "411 Length Required"; case 412: return "412 Precondition Failed"; case 413: return "413 Payload Too Large"; case 414: return "414 URI Too Long"; case 415: return "415 Unsupported Media Type"; case 416: return "416 Range Not Satisfiable"; case 417: return "417 Expectation Failed"; case 418: return "418 I'm a teapot"; case 421: return "421 Misdirected Request"; case 422: return "422 Unprocessable Entity (WebDAV)"; case 423: return "423 Locked (WebDAV)"; case 424: return "424 Failed Dependency (WebDAV)"; case 425: return "425 Too Early"; case 426: return "426 Upgrade Required"; case 428: return "428 Precondition Required"; case 431: return "431 Request Header Fields Too Large"; case 451: return "451 Unavailable For Legal Reasons"; case 500: return "500 Internal Server Error"; case 501: return "501 Not Implemented"; case 502: return "502 Bad Gateway"; case 503: return "503 Service Unavailable"; case 504: return "504 Gateway Timeout"; case 505: return "505 HTTP Version Not Supported"; case 506: return "506 Variant Also Negotiates"; case 507: return "507 Insufficient Storage (WebDAV)"; case 508: return "508 Loop Detected (WebDAV)"; case 510: return "510 Not Extended"; case 511: return "511 Network Authentication Required"; // default: assert(0, "Unsupported http code"); } } /+ /++ This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. It relies on jsvar.d and dom.d. You can get javascript out of it to call. The generated functions need to look like function name(a,b,c,d,e) { return _call("name", {"realName":a,"sds":b}); } And _call returns an object you can call or set up or whatever. +/ bool apiDispatcher()(Cgi cgi) { import arsd.jsvar; import arsd.dom; } +/ version(linux) private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; /* Copyright: Adam D. Ruppe, 2008 - 2023 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. Authors: Adam D. Ruppe Copyright Adam D. Ruppe 2008 - 2023. Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) */ ================================================ FILE: src/clientSideFiltering.d ================================================ // What is this module called? module clientSideFiltering; // What does this module require to function? import std.algorithm; import std.array; import std.file; import std.path; import std.regex; import std.stdio; import std.string; import std.conv; // What other modules that we have created do we need to import? import config; import util; import log; class ClientSideFiltering { // Class variables ApplicationConfig appConfig; string[] syncListRules; string[] syncListIncludePathsOnly; // These are 'include' rules that start with a '/' string[] syncListAnywherePathOnly; // These are 'include' rules that do not start with a '/', thus are to be searched anywhere for inclusion Regex!char fileMask; Regex!char directoryMask; bool skipDirStrictMatch = false; bool skipDotfiles = false; this(ApplicationConfig appConfig) { // Configure the class variable to consume the application configuration this.appConfig = appConfig; } // Initialise the required items bool initialise() { // Log what is being done if (debugLogging) {addLogEntry("Configuring Client Side Filtering (Selective Sync)", ["debug"]);} // Load the sync_list file if it exists if (exists(appConfig.syncListFilePath)){ loadSyncList(appConfig.syncListFilePath); } // Handle skip_dir configuration in config file if (debugLogging) {addLogEntry("Configuring skip_dir ...", ["debug"]);} // Validate skip_dir entries to ensure that this does not contain an invalid configuration // Do not use a skip_dir entry of .* as this will prevent correct searching of local changes to process. foreach(entry; appConfig.getValueString("skip_dir").split("|")){ if (entry == ".*") { // invalid entry element detected addLogEntry(); addLogEntry("ERROR: Invalid skip_dir entry '.*' detected."); addLogEntry(" To exclude hidden directories (those starting with '.'), enable the 'skip_dotfiles' configuration option instead of using wildcard patterns."); addLogEntry(); return false; } } // All skip_dir entries are valid if (debugLogging) {addLogEntry("skip_dir: " ~ appConfig.getValueString("skip_dir"), ["debug"]);} setDirMask(appConfig.getValueString("skip_dir")); // Was --skip-dir-strict-match configured? if (debugLogging) { addLogEntry("Configuring skip_dir_strict_match ...", ["debug"]); addLogEntry("skip_dir_strict_match: " ~ to!string(appConfig.getValueBool("skip_dir_strict_match")), ["debug"]); } if (appConfig.getValueBool("skip_dir_strict_match")) { setSkipDirStrictMatch(); } // Handle skip_file configuration in config file if (debugLogging) {addLogEntry("Configuring skip_file ...", ["debug"]);} // Validate skip_file entries to ensure that this does not contain an invalid configuration // Do not use a skip_file entry of .* as this will prevent correct searching of local changes to process. foreach(entry; appConfig.getValueString("skip_file").split("|")){ if (entry == ".*") { // invalid entry element detected addLogEntry(); addLogEntry("ERROR: Invalid skip_file entry '.*' detected."); addLogEntry(" To exclude hidden files (those starting with '.'), enable the 'skip_dotfiles' configuration option instead of using wildcard patterns."); addLogEntry(); return false; } } // All skip_file entries are valid if (debugLogging) {addLogEntry("skip_file: " ~ appConfig.getValueString("skip_file"), ["debug"]);} setFileMask(appConfig.getValueString("skip_file")); // Was --skip-dot-files configured? if (debugLogging) { addLogEntry("Configuring skip_dotfiles ...", ["debug"]); addLogEntry("skip_dotfiles: " ~ to!string(appConfig.getValueBool("skip_dotfiles")), ["debug"]); } if (appConfig.getValueBool("skip_dotfiles")) { setSkipDotfiles(); } // Validate 'sync_list' include rules are not shadowed by 'skip_file' entries if (!validateSyncListNotShadowedBySkipFile()) { return false; } // Validate 'sync_list' include rules are not shadowed by 'skip_dir' entries if (!validateSyncListNotShadowedBySkipDir()) { // The application configuration is invalid .. 'skip_dir' is shadowing paths included by 'sync_list' return false; } // Client Side Filtering has been configured correctly return true; } // Shutdown components void shutdown() { syncListRules = null; syncListIncludePathsOnly = null; syncListAnywherePathOnly = null; fileMask = regex(""); directoryMask = regex(""); } // Load sync_list file if it exists void loadSyncList(string filepath) { // open file as read only auto file = File(filepath, "r"); auto range = file.byLine(); scope(exit) { file.close(); object.destroy(file); object.destroy(range); } scope(failure) { file.close(); object.destroy(file); object.destroy(range); } foreach (line; range) { auto cleanLine = strip(line); // Skip any line that is empty or just contains whitespace if (cleanLine.length == 0) continue; // Skip comments in file if (cleanLine[0] == ';' || cleanLine[0] == '#') continue; // Invalid exclusion rule patterns if (cleanLine == "!/*" || cleanLine == "!/" || cleanLine == "-/*" || cleanLine == "-/") { string errorMessage = "ERROR: Invalid sync_list rule '" ~ to!string(cleanLine) ~ "' detected. Please read the 'sync_list' documentation."; addLogEntry(); addLogEntry(errorMessage, ["info", "notify"]); addLogEntry(); // do not add this rule continue; } // Legacy include root rule if (cleanLine == "/*" || cleanLine == "/") { string errorMessage = "ERROR: Invalid sync_list rule '" ~ to!string(cleanLine) ~ "' detected. Please use 'sync_root_files = \"true\"' or --sync-root-files option to sync files in the root path."; addLogEntry(); addLogEntry(errorMessage, ["info", "notify"]); addLogEntry(); // do not add this rule continue; } // './' rule warning if ((cleanLine.length > 1) && (cleanLine[0] == '.') && (cleanLine[1] == '/')) { string errorMessage = "ERROR: Invalid sync_list rule '" ~ to!string(cleanLine) ~ "' detected. Rule should not start with './' - please fix your 'sync_list' rule."; addLogEntry(); addLogEntry(errorMessage, ["info", "notify"]); addLogEntry(); // do not add this rule continue; } // Normalise the 'sync_list' rule and store auto normalisedRulePath = buildNormalizedPath(cleanLine); syncListRules ~= normalisedRulePath; // Only add the normalised rule to the specific include list if not an exclude rule if (cleanLine[0] != '!' && cleanLine[0] != '-') { // All include rules get added here syncListIncludePathsOnly ~= normalisedRulePath; // Special case for searching local disk for new data added 'somewhere' if (cleanLine[0] != '/') { // Rule is an 'anywhere' rule within the 'sync_list' syncListAnywherePathOnly ~= normalisedRulePath; } } } // Close the file post reading it file.close(); } // return true or false based on if we have loaded any valid sync_list rules bool validSyncListRules() { // If empty, will return true return syncListRules.empty; } // Configure the regex that will be used for 'skip_file' void setFileMask(const(char)[] mask) { fileMask = wild2regex(mask); if (debugLogging) {addLogEntry("Selective Sync File Mask: " ~ to!string(fileMask), ["debug"]);} } // Configure the regex that will be used for 'skip_dir' void setDirMask(const(char)[] dirmask) { directoryMask = wild2regex(dirmask); if (debugLogging) {addLogEntry("Selective Sync Directory Mask: " ~ to!string(directoryMask), ["debug"]);} } // Configure skipDirStrictMatch if function is called // By default, skipDirStrictMatch = false; void setSkipDirStrictMatch() { skipDirStrictMatch = true; } // Configure skipDotfiles if function is called // By default, skipDotfiles = false; void setSkipDotfiles() { skipDotfiles = true; } // return value of skipDotfiles bool getSkipDotfiles() { return skipDotfiles; } // Match against 'sync_list' only bool isPathExcludedViaSyncList(string path) { // Are there 'sync_list' rules to process? if (count(syncListRules) > 0) { // Perform 'sync_list' rule testing on the given path return isPathExcluded(path); } else { // There are no valid 'sync_list' rules that were loaded return false; // not excluded by 'sync_list' } } // config 'skip_dir' parameter checking bool isDirNameExcluded(string inputPath) { // Returns true if the inputPath matches a skip_dir config entry (directoryMask) // Returns false if no match if (debugLogging) { addLogEntry("skip_dir evaluation for: " ~ inputPath, ["debug"]); } // Build candidate path variants to cover common inputs: // - "./Documents/Uni" (most common from sync engine) // - "Documents/Uni" (relative) // - "/Documents/Uni" (user occasionally prefixes with '/') string name = inputPath; // Normalise leading "./" to relative if (startsWith(name, "./")) { name = name[2 .. $]; if (debugLogging) addLogEntry("skip_dir evaluation (normalised inputPath, removed leading './'): " ~ name, ["debug"]); } // Create a small set of candidates (avoid duplicates) string[] candidates; void addCandidate(string c) { if (c.empty) return; foreach (e; candidates) { if (e == c) return; } candidates ~= c; } addCandidate(name); // If name is rooted, also test relative form if (!name.empty && name[0] == '/') { addCandidate(name[1 .. $]); } else { // If name is relative, also test rooted form (covers skip_dir rules that were authored with a leading '/') addCandidate("/" ~ name); } // Also test trailing-slash equivalence for directory roots // (treat "Documents" and "Documents/" the same, but do not create "//") string[] expanded; foreach (c; candidates) { expanded ~= c; if (c.length > 1 && c[$ - 1] != '/') { expanded ~= (c ~ "/"); } } candidates = expanded; // ------------------------------------------------------------ // 1) Full-path match first (strict semantics) // ------------------------------------------------------------ foreach (c; candidates) { if (!c.matchFirst(directoryMask).empty) { if (debugLogging) addLogEntry("skip_dir full-path match: " ~ c, ["debug"]); return true; } } // ------------------------------------------------------------ // 2) Non-strict mode: test path segments for a match // ------------------------------------------------------------ if (!skipDirStrictMatch) { if (debugLogging) addLogEntry("No Strict Matching Enforced - testing individual path segments", ["debug"]); foreach (c; candidates) { // buildNormalizedPath may introduce a leading '/', so we keep it as-is // and let pathSplitter do its job. We are matching segments, not full paths here. string path = buildNormalizedPath(c); if (debugLogging) addLogEntry("skip_dir segment test path: " ~ path, ["debug"]); foreach_reverse(seg; pathSplitter(path)) { if (seg == "/") continue; // seg is a single component (e.g. "Documents") if (!seg.matchFirst(directoryMask).empty) { if (debugLogging) { addLogEntry("skip_dir segment match: " ~ seg, ["debug"]); } return true; } } } } else { if (debugLogging) addLogEntry("Strict Matching Enforced - no segment testing", ["debug"]); } // No match return false; } // config file skip_file parameter bool isFileNameExcluded(string name) { // Does the file name match skip_file config entry? // Returns true if the name matches a skip_file config entry // Returns false if no match if (debugLogging) {addLogEntry("skip_file evaluation for: " ~ name, ["debug"]);} // Try full path match first if (!name.matchFirst(fileMask).empty) { return true; } else { // check just the file name string filename = baseName(name); if(!filename.matchFirst(fileMask).empty) { return true; } } // no match return false; } // test if the given path is not included in the allowed syncListRules // if there are no allowed syncListRules always return false private bool isPathExcluded(string path) { // function variables bool exclude = false; bool excludeExactMatch = false; // will get updated to true, if there is a pattern match to sync_list entry bool excludeParentMatched = false; // will get updated to true, if there is a pattern match to sync_list entry bool finalResult = true; // will get updated to false, if pattern match to sync_list entry bool anywhereRuleMatched = false; // will get updated if the 'anywhere' rule matches bool excludeAnywhereMatched = false; // will get updated if the 'anywhere' rule matches bool wildcardRuleMatched = false; // will get updated if the 'wildcard' rule matches bool excludeWildcardMatched = false; // will get updated if the 'wildcard' rule matches int offset; string wildcard = "*"; string globbing = "**"; // always allow the root if (path == ".") return false; // if there are no allowed syncListRules always return false, meaning path is not excluded if (syncListRules.empty) return false; // To ensure we are checking the 'right' path, build the path path = buildPath("/", buildNormalizedPath(path)); // Evaluation start point, in order of what is checked as well if (debugLogging) { addLogEntry("******************* SYNC LIST RULES EVALUATION START *******************", ["debug"]); addLogEntry("Evaluation against 'sync_list' rules for this input path: " ~ path, ["debug"]); addLogEntry("[S]excludeExactMatch = " ~ to!string(excludeExactMatch), ["debug"]); addLogEntry("[S]excludeParentMatched = " ~ to!string(excludeParentMatched), ["debug"]); addLogEntry("[S]excludeAnywhereMatched = " ~ to!string(excludeAnywhereMatched), ["debug"]); addLogEntry("[S]excludeWildcardMatched = " ~ to!string(excludeWildcardMatched), ["debug"]); } // Split input path by '/' to create an applicable path segment array // - This is reused below in a number of places string[] pathSegments = path.strip.split("/").filter!(s => !s.empty).array; // Unless path is an exact match, entire sync_list entries need to be processed to ensure negative matches are also correctly detected foreach (syncListRuleEntry; syncListRules) { // There are several matches we need to think of here // Exclusions: // !foldername/* = As there is no preceding '/' (after the !) .. this is a rule that should exclude 'foldername' and all its children ANYWHERE // !*.extension = As there is no preceding '/' (after the !) .. this is a rule that should exclude any item that has the specified extension ANYWHERE // !/path/to/foldername/* = As there IS a preceding '/' (after the !) .. this is a rule that should exclude this specific path and all its children // !/path/to/foldername/*.extension = As there IS a preceding '/' (after the !) .. this is a rule that should exclude any item that has the specified extension in this path ONLY // !/path/to/foldername/*/specific_target/* = As there IS a preceding '/' (after the !) .. this excludes 'specific_target' in any subfolder of '/path/to/foldername/' // // Inclusions: // foldername/* = As there is no preceding '/' .. this is a rule that should INCLUDE 'foldername' and all its children ANYWHERE // *.extension = As there is no preceding '/' .. this is a rule that should INCLUDE any item that has the specified extension ANYWHERE // /path/to/foldername/* = As there IS a preceding '/' .. this is a rule that should INCLUDE this specific path and all its children // /path/to/foldername/*.extension = As there IS a preceding '/' .. this is a rule that should INCLUDE any item that has the specified extension in this path ONLY // /path/to/foldername/*/specific_target/* = As there IS a preceding '/' .. this INCLUDES 'specific_target' in any subfolder of '/path/to/foldername/' if (debugLogging) {addLogEntry("------------------------------ NEW RULE --------------------------------", ["debug"]);} // Is this rule an 'exclude' or 'include' rule? bool thisIsAnExcludeRule = false; // Switch based on first character of rule to determine rule type switch (syncListRuleEntry[0]) { case '-': // sync_list path starts with '-', this user wants to exclude this path exclude = true; // default exclude thisIsAnExcludeRule = true; // exclude rule offset = 1; // To negate the '-' in the rule entry break; case '!': // sync_list path starts with '!', this user wants to exclude this path exclude = true; // default exclude thisIsAnExcludeRule = true; // exclude rule offset = 1; // To negate the '!' in the rule entry break; case '/': // sync_list path starts with '/', this user wants to include this path // but a '/' at the start causes matching issues, so use the offset for comparison exclude = false; // DO NOT EXCLUDE thisIsAnExcludeRule = false; // INCLUDE rule offset = 0; break; default: // no negative pattern, default is to not exclude exclude = false; // DO NOT EXCLUDE thisIsAnExcludeRule = false; // INCLUDE rule offset = 0; } // Update syncListRuleEntry to remove the offset syncListRuleEntry = syncListRuleEntry[offset..$]; // What 'sync_list' rule are we comparing against? if (thisIsAnExcludeRule) { if (debugLogging) {addLogEntry("Evaluation against EXCLUSION 'sync_list' rule: !" ~ syncListRuleEntry, ["debug"]);} } else { if (debugLogging) {addLogEntry("Evaluation against INCLUSION 'sync_list' rule: " ~ syncListRuleEntry, ["debug"]);} } // Split rule path by '/' to create an applicable path segment array // - This is reused below in a number of places string[] ruleSegments = syncListRuleEntry.strip.split("/").filter!(s => !s.empty).array; // Configure logging rule type string ruleKind = thisIsAnExcludeRule ? "exclusion rule" : "inclusion rule"; // Is path is an exact match of the 'sync_list' rule, or do the input path segments (directories) match the 'sync_list' rule? // wildcard (*) rules are below if we get there, if this rule does not contain a wildcard if ((to!string(syncListRuleEntry[0]) == "/") && (!canFind(syncListRuleEntry, wildcard))) { // what sort of rule is this - 'exact match' include or exclude rule? if (debugLogging) {addLogEntry("Testing input path against an exact match 'sync_list' " ~ ruleKind, ["debug"]);} // Print rule and input segments for validation during debug if (debugLogging) { addLogEntry(" - Calculated Rule Segments: " ~ to!string(ruleSegments), ["debug"]); addLogEntry(" - Calculated Path Segments: " ~ to!string(pathSegments), ["debug"]); } // Test for exact segment matching of input path to rule if (exactMatchRuleSegmentsToPathSegments(ruleSegments, pathSegments)) { // EXACT PATH MATCH if (debugLogging) {addLogEntry("Exact path match with 'sync_list' rule entry", ["debug"]);} if (!thisIsAnExcludeRule) { // Include Rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: direct match", ["debug"]);} // final result finalResult = false; // direct match, break and search rules no more given include rule match break; } else { // Exclude rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: exclusion direct match - path to be excluded", ["debug"]);} // flag excludeExactMatch so that a 'wildcard match' will not override this exclude excludeExactMatch = true; exclude = true; // final result finalResult = true; // dont break here, finish checking other rules } } else { // NOT an EXACT MATCH, so check the very first path segment if (debugLogging) {addLogEntry("No exact path match with 'sync_list' rule entry - checking path segments to verify", ["debug"]);} // - This is so that paths in 'sync_list' as specified as /some path/another path/ actually get included|excluded correctly if (matchFirstSegmentToPathFirstSegment(ruleSegments, pathSegments)) { // PARENT ROOT MATCH if (debugLogging) {addLogEntry("Parent root path match with 'sync_list' rule entry", ["debug"]);} // Does the 'rest' of the input path match? // We only need to do this step if the input path has more and 1 segment (the parent folder) if (count(pathSegments) > 1) { // More segments to check, so do a parental path match if (matchRuleSegmentsToPathSegments(ruleSegments, pathSegments)) { // PARENTAL PATH MATCH if (debugLogging) {addLogEntry("Parental path match with 'sync_list' rule entry", ["debug"]);} // What sort of rule was this? if (!thisIsAnExcludeRule) { // Include Rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: parental path match", ["debug"]);} // final result finalResult = false; // parental path match, break and search rules no more given include rule match break; } else { // Exclude rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: exclusion parental path match - path to be excluded", ["debug"]);} excludeParentMatched = true; exclude = true; // final result finalResult = true; // dont break here, finish checking other rules } } } else { // No more segments to check if (!thisIsAnExcludeRule) { // Include Rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: parent root path match to rule", ["debug"]);} // final result finalResult = false; // parental path match, break and search rules no more given include rule match break; } else { // Exclude rule {addLogEntry("Evaluation against 'sync_list' rule result: exclusion parent root path match to rule - path to be excluded", ["debug"]);} excludeParentMatched = true; exclude = true; // final result finalResult = true; // dont break here, finish checking other rules } } } else { // No parental path segment match if (debugLogging) {addLogEntry("No parental path match with 'sync_list' rule entry - exact path matching not possible", ["debug"]);} } } // What 'rule' type are we currently testing? if (!thisIsAnExcludeRule) { // Is the path a parental path match to an include 'sync_list' rule? if (isSyncListPrefixMatch(path)) { // PARENTAL PATH MATCH if (debugLogging) { addLogEntry("Parental path match with 'sync_list' rule entry (syncListIncludePathsOnly)", ["debug"]); addLogEntry("Evaluation against 'sync_list' rule result: parental path match (syncListIncludePathsOnly)", ["debug"]); } // final result finalResult = false; // parental path match, break and search rules no more given include rule match break; } } } // Is the 'sync_list' rule an 'anywhere' rule? // EXCLUSION // !foldername/* // !*.extension // !foldername // INCLUSION // foldername/* // *.extension // foldername if (to!string(syncListRuleEntry[0]) != "/") { // reset anywhereRuleMatched anywhereRuleMatched = false; // what sort of rule is this - 'anywhere' include or exclude rule? if (debugLogging) {addLogEntry("Testing input path against an anywhere 'sync_list' " ~ ruleKind, ["debug"]);} // this is an 'anywhere' rule string anywhereRuleStripped; // If this 'sync_list' rule end in '/*' - if yes, remove it to allow for easier comparison if (syncListRuleEntry.endsWith("/*")) { // strip '/*' from the end of the rule anywhereRuleStripped = syncListRuleEntry.stripRight("/*"); } else { // keep rule 'as-is' anywhereRuleStripped = syncListRuleEntry; } // If the input path is exactly the parent root (single segment) and that segment // matches the rule's first segment, treat it as a match. if (!ruleSegments.empty && count(pathSegments) == 1 && matchFirstSegmentToPathFirstSegment(ruleSegments, pathSegments)) { if (debugLogging) { addLogEntry(" - anywhere rule 'parent root' MATCH with '" ~ ruleSegments[0] ~ "'", ["debug"]); } anywhereRuleMatched = true; } if (!anywhereRuleMatched) { if (canFind(path, anywhereRuleStripped)) { // we matched the path to the rule if (debugLogging) {addLogEntry(" - anywhere rule 'canFind' MATCH", ["debug"]);} anywhereRuleMatched = true; } else { // no 'canFind' match, try via regex if (debugLogging) {addLogEntry(" - anywhere rule 'canFind' NO_MATCH .. trying a regex match", ["debug"]);} // create regex from 'syncListRuleEntry' auto allowedMask = regex(createRegexCompatiblePath(syncListRuleEntry)); // perform regex match attempt if (matchAll(path, allowedMask)) { // we regex matched the path to the rule if (debugLogging) {addLogEntry(" - anywhere rule 'matchAll via regex' MATCH", ["debug"]);} anywhereRuleMatched = true; } else { // no match if (debugLogging) {addLogEntry(" - anywhere rule 'matchAll via regex' NO_MATCH", ["debug"]);} } } } // is this rule matched? if (anywhereRuleMatched) { // Is this an exclude rule? if (thisIsAnExcludeRule) { if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: anywhere rule matched and must be excluded", ["debug"]);} excludeAnywhereMatched = true; exclude = true; finalResult = true; // anywhere match, break and search rules no more break; } else { if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: anywhere rule matched and must be included", ["debug"]);} finalResult = false; excludeAnywhereMatched = false; // anywhere match, break and search rules no more break; } } } // Does the 'sync_list' rule contain a wildcard (*) or globbing (**) reference anywhere in the rule? // EXCLUSION // !/Programming/Projects/Android/**/build/* // !/build/kotlin/* // INCLUSION // /Programming/Projects/Android/**/build/* // /build/kotlin/* if (canFind(syncListRuleEntry, wildcard)) { // A '*' wildcard is in the rule, but we do not know what type of wildcard yet .. // reset the applicable flag wildcardRuleMatched = false; // What sort of rule is this - globbing (**) or wildcard (*) bool globbingRule = false; globbingRule = canFind(syncListRuleEntry, globbing); // The sync_list rule contains some sort of wildcard sequence - lets log this correctly as to the rule type we are testing string ruleType = globbingRule ? "globbing (**)" : "wildcard (*)"; if (debugLogging) {addLogEntry("Testing input path against a " ~ ruleType ~ " 'sync_list' " ~ ruleKind, ["debug"]);} // Does the parents of the input path and rule path match .. meaning we can actually evaluate this wildcard rule against the input path if (matchFirstSegmentToPathFirstSegment(ruleSegments, pathSegments)) { // Is this a globbing rule (**) or just a single wildcard (*) entries if (globbingRule) { // globbing (**) rule processing // globbing rules can only realistically apply if there are enough path segments for the globbing rule to actually apply // otherwise we get a bad match - see: // - https://github.com/abraunegg/onedrive/issues/3122 // - https://github.com/abraunegg/onedrive/issues/3122#issuecomment-2661556789 auto wildcardDepth = firstWildcardDepth(syncListRuleEntry); auto pathCount = count(pathSegments); // Are there enough path segments for this globbing rule to apply? if (pathCount < wildcardDepth) { // there are not enough path segments up to the first wildcard character (*) for this rule to even be applicable if (debugLogging) {addLogEntry(" - This sync list globbing rule cannot not be evaluated as the globbing appears beyond the current input path", ["debug"]);} } else { // There are enough segments in the path and rule to test against this globbing rule if (matchPathAgainstRule(path, syncListRuleEntry)) { // set the applicable flag wildcardRuleMatched = true; if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: globbing pattern match using segment matching", ["debug"]);} } } } else { // wildcard (*) rule processing // create regex from 'syncListRuleEntry' auto allowedMask = regex(createRegexCompatiblePath(syncListRuleEntry)); if (matchAll(path, allowedMask)) { // set the applicable flag wildcardRuleMatched = true; if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: wildcard pattern match", ["debug"]);} } else { // matchAll no match ... try another way just to be sure if (matchPathAgainstRule(path, syncListRuleEntry)) { // set the applicable flag wildcardRuleMatched = true; if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: wildcard pattern match using segment matching", ["debug"]);} } } } // Was the rule matched? if (wildcardRuleMatched) { // Is this an exclude rule? if (thisIsAnExcludeRule) { // Yes exclude rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: wildcard|globbing rule matched and must be excluded", ["debug"]);} excludeWildcardMatched = true; exclude = true; finalResult = true; } else { // include rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: wildcard|globbing pattern matched and must be included", ["debug"]);} finalResult = false; excludeWildcardMatched = false; } } else { if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: No match to 'sync_list' wildcard|globbing rule", ["debug"]);} } } else { // log that parental path in input path does not match the parental path in the rule if (debugLogging) {addLogEntry("Evaluation against 'sync_list' rule result: No evaluation possible - parental input path does not match 'sync_list' rule", ["debug"]);} } } } // debug logging post 'sync_list' rule evaluations if (debugLogging) { // Rule evaluation complete addLogEntry("------------------------------------------------------------------------", ["debug"]); // Interim results after checking each 'sync_list' rule against the input path addLogEntry("[F]excludeExactMatch = " ~ to!string(excludeExactMatch), ["debug"]); addLogEntry("[F]excludeParentMatched = " ~ to!string(excludeParentMatched), ["debug"]); addLogEntry("[F]excludeAnywhereMatched = " ~ to!string(excludeAnywhereMatched), ["debug"]); addLogEntry("[F]excludeWildcardMatched = " ~ to!string(excludeWildcardMatched), ["debug"]); } // Only force exclusion if an exclusion rule actually matched this path if (excludeExactMatch || excludeParentMatched || excludeAnywhereMatched || excludeWildcardMatched) { finalResult = true; } // Final Result if (finalResult) { if (debugLogging) {addLogEntry("Evaluation against 'sync_list' final result: EXCLUDED as no rule included path", ["debug"]);} } else { if (debugLogging) {addLogEntry("Evaluation against 'sync_list' final result: included for sync", ["debug"]);} } if (debugLogging) {addLogEntry("******************* SYNC LIST RULES EVALUATION END *********************", ["debug"]);} return finalResult; } // Calculate wildcard character depth in path int firstWildcardDepth(string syncListRuleEntry) { int depth = 0; foreach (segment; pathSplitter(syncListRuleEntry)) { if (segment.canFind("*")) // Check for wildcard characters return depth; depth++; } return depth; // No wildcard found should be '0' } // Create a wildcard regex compatible string based on the sync list rule string createRegexCompatiblePath(string regexCompatiblePath) { // Escape all special regex characters that could break regex parsing regexCompatiblePath = escaper(regexCompatiblePath).text; // Restore wildcard (*) support with '.*' to be compatible with function and to match any characters regexCompatiblePath = regexCompatiblePath.replace("\\*", ".*"); // Ensure space matches only literal space, not \s (tabs, etc.) regexCompatiblePath = regexCompatiblePath.replace(" ", "\\ "); // Return the regex compatible path return regexCompatiblePath; } // Create a regex compatible string to match a relevant segment bool matchSegment(string ruleSegment, string pathSegment) { // Create the required pattern auto pattern = regex("^" ~ createRegexCompatiblePath(ruleSegment) ~ "$"); // Check if there's a match and return result return !match(pathSegment, pattern).empty; } // Function to handle path matching when using globbing (**) bool matchPathAgainstRule(string path, string rule) { // Split both the path and rule into segments auto pathSegments = pathSplitter(path).filter!(s => !s.empty).array; auto ruleSegments = pathSplitter(rule).filter!(s => !s.empty).array; bool lastSegmentMatchesRule = false; size_t i = 0, j = 0; while (i < pathSegments.length && j < ruleSegments.length) { if (ruleSegments[j] == "**") { if (j == ruleSegments.length - 1) { return true; // '**' at the end matches everything } // Find next matching part after '**' while (i < pathSegments.length && !matchSegment(ruleSegments[j + 1], pathSegments[i])) { i++; } j++; // Move past the '**' in the rule } else { if (!matchSegment(ruleSegments[j], pathSegments[i])) { return false; } else { // increment to next set of values i++; j++; } } } // Ensure that we handle the last segments gracefully if (i >= pathSegments.length && j < ruleSegments.length) { if (j == ruleSegments.length - 1 && ruleSegments[j] == "*") { return true; } if (ruleSegments[j - 1] == pathSegments[i - 1]) { lastSegmentMatchesRule = true; } } return j == ruleSegments.length || (j == ruleSegments.length - 1 && ruleSegments[j] == "**") || lastSegmentMatchesRule; } // Function to perform an exact match of path segments to rule segments bool exactMatchRuleSegmentsToPathSegments(string[] ruleSegments, string[] inputSegments) { // If rule has more segments than input, or input has more segments than rule, no match is possible if ((ruleSegments.length > inputSegments.length) || ( inputSegments.length > ruleSegments.length)) { return false; } // Iterate over each segment and compare for (size_t i = 0; i < ruleSegments.length; ++i) { if (ruleSegments[i] != inputSegments[i]) { if (debugLogging) {addLogEntry("Mismatch at segment " ~ to!string(i) ~ ": Rule Segment = " ~ ruleSegments[i] ~ ", Input Segment = " ~ inputSegments[i], ["debug"]);} return false; // Return false if any segment doesn't match } } // If all segments match, return true if (debugLogging) {addLogEntry("All segments matched: Rule Segments = " ~ to!string(ruleSegments) ~ ", Input Segments = " ~ to!string(inputSegments), ["debug"]);} return true; } // Function to perform a match of path segments to rule segments bool matchRuleSegmentsToPathSegments(string[] ruleSegments, string[] inputSegments) { if (debugLogging) {addLogEntry("Running matchRuleSegmentsToPathSegments()", ["debug"]);} // If rule has more segments than input, no match is possible if (ruleSegments.length > inputSegments.length) { return false; } // Compare segments up to the length of the rule path return equal(ruleSegments, inputSegments[0 .. ruleSegments.length]); } // Function to match the first segment only of the path and rule bool matchFirstSegmentToPathFirstSegment(string[] ruleSegments, string[] inputSegments) { // Check that both segments are not empty if (ruleSegments.length == 0 || inputSegments.length == 0) { return false; // Return false if either segment array is empty } // Compare the first segments only return equal(ruleSegments[0], inputSegments[0]); } // Test the path for prefix matching an include sync_list rule bool isSyncListPrefixMatch(string inputPath) { // Ensure inputPath ends with a '/' if not root, to avoid false positives string inputPrefix = inputPath.endsWith("/") ? inputPath : inputPath ~ "/"; foreach (entry; syncListIncludePathsOnly) { string normalisedEntry = entry; // If rule ends in '/*', treat it as if the '/*' is not there if (normalisedEntry.endsWith("/*")) { normalisedEntry = normalisedEntry[0 .. $ - 2]; // remove '/*' for this rule comparison } // Ensure trailing '/' for safe prefix match string entryWithSlash = normalisedEntry.endsWith("/") ? normalisedEntry : normalisedEntry ~ "/"; // Match input as being equal to or under the rule path, or rule path being under the input path if (entryWithSlash.startsWith(inputPrefix) || inputPrefix.startsWith(entryWithSlash)) { // Debug the exact 'sync_list' inclusion rule this matched if (debugLogging) { addLogEntry("Parental path matched 'sync_list' Inclusion Rule: " ~ to!string(entry), ["debug"]); } return true; } } return false; } // Do any 'anywhere' sync_list' rules exist for inclusion? bool syncListAnywhereInclusionRulesExist() { // Count the entries in syncListAnywherePathOnly auto anywhereRuleCount = count(syncListAnywherePathOnly); if (anywhereRuleCount > 0) { return true; } else { return false; } } // Validate that 'sync_list' *include* rules are not rendered non-viable by 'skip_dir' entries. // If an include rule would be excluded by 'skip_dir' evaluation, it is "shadowed" by that entry. bool validateSyncListNotShadowedBySkipDir() { // No sync_list include rules loaded => nothing to validate if (syncListIncludePathsOnly is null || syncListIncludePathsOnly.empty) return true; // No skip_dir configured => nothing to validate if (appConfig.getValueString("skip_dir").empty) return true; string[] shadowedRules; foreach (rule; syncListIncludePathsOnly) { // syncListIncludePathsOnly should only contain include rules, but be defensive. if (rule.empty) continue; if (rule[0] == '!' || rule[0] == '-') continue; // Normalise the rule to match how skip_dir rules are evaluated at runtime. // skip_dir entries are relative to sync_dir. sync_list entries may be rooted (start with '/'). string candidate = rule; // Normalise leading "./" (defensive) if (candidate.length >= 2 && candidate[0 .. 2] == "./") { candidate = candidate[2 .. $]; } // Normalise sync_list rooted includes: "/Documents" -> "Documents" if (candidate.length >= 1 && candidate[0] == '/') { // Remove only the first '/', sync_list rules are single-rooted relative to sync_dir candidate = candidate[1 .. $]; } if (candidate.empty) continue; // Use the *actual* runtime skip_dir evaluation logic (strict/non-strict) // so the check matches real behaviour. bool shadowed = false; // Test as-is if (isDirNameExcluded(candidate)) { shadowed = true; } else { // Also test with a trailing slash where appropriate, so: // skip_dir = "Documents/" correctly shadows sync_list = "/Documents" // (Users often represent a directory root either way.) if (candidate[$ - 1] != '/') { if (isDirNameExcluded(candidate ~ "/")) { shadowed = true; } } } if (shadowed) { shadowedRules ~= rule; } } if (!shadowedRules.empty) { addLogEntry(); addLogEntry("ERROR: Invalid Client Side Filtering configuration detected.", ["info", "notify"]); addLogEntry(" One or more 'sync_list' inclusion rules are shadowed by 'skip_dir' and will never be viable.", ["info", "notify"]); foreach (r; shadowedRules) { addLogEntry(" Shadowed 'sync_list' rule: " ~ r, ["info", "notify"]); } addLogEntry(" Fix: remove or narrow the conflicting 'skip_dir' entry/entries, or adjust your 'sync_list' rules.", ["info", "notify"]); addLogEntry(" See the 'skip_dir' documentation for correct usage and examples.", ["info", "notify"]); addLogEntry(); return false; } return true; } // Validate that 'sync_list' *include* rules are not rendered non-viable by 'skip_file' entries. // If an include rule would be excluded by 'skip_file' evaluation, it is "shadowed" by that entry. bool validateSyncListNotShadowedBySkipFile() { // No sync_list include rules loaded => nothing to validate if (syncListIncludePathsOnly is null || syncListIncludePathsOnly.empty) return true; // No skip_file configured => nothing to validate if (appConfig.getValueString("skip_file").empty) return true; string[] shadowedRules; foreach (rule; syncListIncludePathsOnly) { // Defensive: ignore empty or explicitly negative rules if (rule.empty) continue; if (rule[0] == '!' || rule[0] == '-') continue; // Only validate file-intent rules: // - If it ends with '/', treat as a directory include and do not apply skip_file shadow validation. // (Users commonly include folders; skip_file patterns like '*.tmp' should not invalidate that.) if (rule.length > 1 && rule[$ - 1] == '/') continue; // Normalise the rule to match how skip_file rules are evaluated at runtime. // skip_file entries are relative to sync_dir. sync_list entries may be rooted (start with '/'). string candidate = rule; // Normalise leading "./" (defensive) if (candidate.length >= 2 && candidate[0 .. 2] == "./") { candidate = candidate[2 .. $]; } // Normalise sync_list rooted includes: "/Documents/file.txt" -> "Documents/file.txt" if (candidate.length >= 1 && candidate[0] == '/') { candidate = candidate[1 .. $]; } if (candidate.empty) continue; // Use the *actual* runtime skip_file evaluation logic so this check matches real behaviour. if (isFileNameExcluded(candidate)) { shadowedRules ~= rule; } } if (!shadowedRules.empty) { addLogEntry(); addLogEntry("ERROR: Invalid Client Side Filtering configuration detected.", ["info", "notify"]); addLogEntry(" One or more 'sync_list' inclusion rules are shadowed by 'skip_file' and will never be viable.", ["info", "notify"]); foreach (r; shadowedRules) { addLogEntry(" Shadowed 'sync_list' rule: " ~ r, ["info", "notify"]); } addLogEntry(" Fix: remove or narrow the conflicting 'skip_file' entry/entries, or adjust your 'sync_list' rules.", ["info", "notify"]); addLogEntry(" See the 'skip_file' documentation for correct usage and examples.", ["info", "notify"]); addLogEntry(); return false; } return true; } } ================================================ FILE: src/config.d ================================================ // What is this module called? module config; // What does this module require to function? import core.stdc.stdlib: EXIT_SUCCESS, EXIT_FAILURE, exit; import std.array; import std.stdio; import std.process; import std.regex; import std.string; import std.algorithm; import std.algorithm.searching; import std.algorithm.sorting; import std.file; import std.conv; import std.path; import std.getopt; import std.format; import std.ascii; import std.datetime; import std.exception; import core.sys.posix.unistd : geteuid, getuid; import std.process : spawnProcess, wait; // What other modules that we have created do we need to import? import log; import util; class ApplicationConfig { // Application default values - these do not change // - Compile time regex immutable auto configRegex = ctRegex!(`^(\w+)\s*=\s*"(.*)"\s*$`); // - Default directory to store data immutable string defaultSyncDir = "~/OneDrive"; // - Default Directory Permissions immutable long defaultDirectoryPermissionMode = 700; // - Default File Permissions immutable long defaultFilePermissionMode = 600; // - Default types of files to skip // v2.0.x - 2.4.x: ~*|.~*|*.tmp // v2.5.x : ~*|.~*|*.tmp|*.swp|*.partial immutable string defaultSkipFile = "~*|.~*|*.tmp|*.swp|*.partial"; // - Default directories to skip (default is skip none) immutable string defaultSkipDir = ""; // - Default application logging directory immutable string defaultLogFileDir = "/var/log/onedrive"; // - Default configuration directory immutable string defaultConfigDirName = "~/.config/onedrive"; // - Default 'OneDrive Business Shared Files' Folder Name immutable string defaultBusinessSharedFilesDirectoryName = "Files Shared With Me"; // - Default file fragment size for uploads immutable long defaultFileFragmentSize = 10; // in MiB immutable long defaultMaxFileFragmentSize = 60; // in MiB immutable long defaultMonitorInterval = 300; // 5 minutes // Microsoft Requirements // - Default Application ID (abraunegg) immutable string defaultApplicationId = "d50ca740-c83f-4d1b-b616-12c519384f0c"; // - Microsoft User Agent ISV Tag immutable string isvTag = "ISV"; // - Microsoft User Agent Company name immutable string companyName = "abraunegg"; // - Microsoft Application name as per Microsoft Azure application registration immutable string appTitle = "OneDrive Client for Linux"; // Comply with OneDrive traffic decoration requirements // https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online // - Identify as ISV and include Company Name, App Name separated by a pipe character and then adding Version number separated with a slash character immutable string defaultUserAgent = isvTag ~ "|" ~ companyName ~ "|" ~ appTitle ~ "/" ~ strip(import("version")); // HTTP Struct items, used for configuring HTTP() // Curl Timeout Handling // libcurl dns_cache_timeout timeout immutable int defaultDnsTimeout = 60; // in seconds // Connect timeout for HTTP|HTTPS connections // Controls CURLOPT_CONNECTTIMEOUT immutable int defaultConnectTimeout = 10; // in seconds // Default data timeout for HTTP operations // curl.d has a default of: _defaultDataTimeout = dur!"minutes"(2); immutable int defaultDataTimeout = 60; // in seconds // Maximum total time (in seconds) that any transfer operation is allowed to take. // This maps directly to libcurl's CURLOPT_TIMEOUT. // // IMPORTANT: // • CURLOPT_TIMEOUT applies to the *entire* operation — DNS lookup, TCP connect, // TLS negotiation, and the full data transfer. // • If this timeout is reached, libcurl will abort the request even if data is // flowing normally. // • For large file downloads, especially on slower links, setting a non-zero // timeout can cause the transfer to be killed prematurely. // // Behaviour: // • A value of 0 disables the limit entirely (libcurl’s default behaviour). // • It is strongly recommended to keep this at 0 unless a hard global cap is // explicitly required by the user or their environment. immutable int defaultOperationTimeout = 0; // 0 = no timeout (safe for extremely large file downloads) // Specify what IP protocol version should be used when communicating with OneDrive immutable int defaultIpProtocol = 0; // 0 = IPv4 + IPv6, 1 = IPv4 Only, 2 = IPv6 Only // Specify how many redirects should be allowed immutable int defaultMaxRedirects = 5; // Azure Active Directory & Graph Explorer Endpoints // - Global & Default immutable string globalAuthEndpoint = "https://login.microsoftonline.com"; immutable string globalGraphEndpoint = "https://graph.microsoft.com"; // - US Government L4 immutable string usl4AuthEndpoint = "https://login.microsoftonline.us"; immutable string usl4GraphEndpoint = "https://graph.microsoft.us"; // - US Government L5 immutable string usl5AuthEndpoint = "https://login.microsoftonline.us"; immutable string usl5GraphEndpoint = "https://dod-graph.microsoft.us"; // - Germany immutable string deAuthEndpoint = "https://login.microsoftonline.de"; immutable string deGraphEndpoint = "https://graph.microsoft.de"; // - China immutable string cnAuthEndpoint = "https://login.chinacloudapi.cn"; immutable string cnGraphEndpoint = "https://microsoftgraph.chinacloudapi.cn"; // Application Version immutable string applicationVersion = "onedrive " ~ strip(import("version")); // Application items that depend on application run-time environment, thus cannot be immutable // Public variables // Logging verbosity count long verbosityCount = 0; // Was the application just authorised - paste of response URI bool applicationAuthoriseResponseURIReceived = false; // Store the refreshToken for use within the application const(char)[] refreshToken; // Store the current accessToken for use within the application const(char)[] accessToken; // Store the 'refresh_token' file path string refreshTokenFilePath = ""; // Store the accessTokenExpiration for use within the application SysTime accessTokenExpiration; // Store the 'session_upload.UNIQUE_STRING' file path string uploadSessionFilePath = ""; // Store the 'resume_download.UNIQUE_STRING' file path string resumeDownloadFilePath = ""; // Store the Intune account information string intuneAccountDetails; // Store the Intune account information on disk for reuse string intuneAccountDetailsFilePath = ""; // API initialisation flags bool apiWasInitialised = false; bool syncEngineWasInitialised = false; // Important Account Details string accountType; string defaultDriveId; string defaultRootId; // Sync Operations bool fullScanTrueUpRequired = false; bool suppressLoggingOutput = false; // WebSocket Operations bool curlSupportsWebSockets = false; bool websocketSupportCheckDone = false; bool websocketNotificationUrlAvailable = false; string websocketEndpointResponse; string websocketNotificationUrl; string websocketUrlExpiry; // Default number of concurrent threads when downloading and uploading data ulong defaultConcurrentThreads = 8; // Default number of seconds inotify actions will be delayed by ulong defaultInotifyDelay = 5; // All application run-time paths are formulated from this as a set of defaults // - What is the home path of the actual 'user' that is running the application string defaultHomePath = ""; // - What is the config path for the application. By default, this is ~/.config/onedrive but can be overridden by using --confdir string configDirName = defaultConfigDirName; // - In case we have to use a system config directory such as '/etc/onedrive' or similar, store that path in this variable private string systemConfigDirName = ""; // - Store the configured converted octal value for directory permissions private int configuredDirectoryPermissionMode; // - Store the configured converted octal value for file permissions private int configuredFilePermissionMode; // - Store the 'delta_link' file path private string deltaLinkFilePath = ""; // - Store the 'items.sqlite3' file path string databaseFilePath = ""; // - Store the 'items-dryrun.sqlite3' file path string databaseFilePathDryRun = ""; // - Store the user 'config' file path private string userConfigFilePath = ""; // - Store the system 'config' file path private string systemConfigFilePath = ""; // - What is the 'config' file path that will be used? private string applicableConfigFilePath = ""; // - Store the 'sync_list' file path string syncListFilePath = ""; // OneDrive Business Shared File handling - what directory will be used? string configuredBusinessSharedFilesDirectoryName = ""; // Hash files so that we can detect when the configuration has changed, in items that will require a --resync private string configHashFile = ""; private string configBackupFile = ""; private string syncListHashFile = ""; // Store the actual 'runtime' hash private string currentConfigHash = ""; private string currentSyncListHash = ""; // Store the previous config files hash values (file contents) private string previousConfigHash = ""; private string previousSyncListHash = ""; // Store items that come in from the 'config' file, otherwise these need to be set the defaults private string configFileSyncDir = defaultSyncDir; private string configFileSkipFile = ""; // Default for now, if post reading in any user configuration, if still empty, default will be used private bool configFileSkipFileReadIn = false; // If we actually read in something from 'config' file, this gets set to true private string configFileSkipDir = ""; // Default here is no directories are skipped private string configFileDriveId = ""; // Default here is that no drive id is specified private bool configFileCheckNoSync = false; private bool configFileSkipDotfiles = false; private bool configFileSkipSymbolicLinks = false; private bool configFileSkipSize = false; private bool configFileSyncBusinessSharedItems = false; // File permission values (set via initialise function) private int convertedPermissionValue; // Array of values that are the actual application runtime configuration // The values stored in these array's are the actual application configuration which can then be accessed by getValue & setValue string[string] stringValues; long[string] longValues; bool[string] boolValues; bool shellEnvironmentSet = false; // GUI Notification Environment variables bool xdg_exists = false; bool dbus_exists = false; // Recycle Bin Configuration // These paths are used by the application, if 'use_recycle_bin' is enabled string recycleBinParentPath; string recycleBinFilePath; string recycleBinInfoPath; // Runtime 'sync_dir' as initialised string runtimeSyncDirectory; // Initialise the application configuration bool initialise(string confdirOption, bool helpRequested) { // Default runtime configuration - entries in config file ~/.config/onedrive/config or derived from variables above // An entry here means it can be set via the config file if there is a corresponding entry, read from config and set via update_from_args() // The below becomes the 'default' application configuration before config file and/or cli options are overlaid on top // - Set the required default values stringValues["application_id"] = defaultApplicationId; stringValues["log_dir"] = defaultLogFileDir; stringValues["skip_dir"] = defaultSkipDir; stringValues["skip_file"] = defaultSkipFile; stringValues["sync_dir"] = defaultSyncDir; stringValues["user_agent"] = defaultUserAgent; // - The 'drive_id' is used when we specify a specific OneDrive ID when attempting to sync Shared Folders and SharePoint items stringValues["drive_id"] = ""; // Support National Azure AD endpoints as per https://docs.microsoft.com/en-us/graph/deployments // By default, if empty, use standard Azure AD URL's // Will support the following options: // - USL4 // AD Endpoint: https://login.microsoftonline.us // Graph Endpoint: https://graph.microsoft.us // - USL5 // AD Endpoint: https://login.microsoftonline.us // Graph Endpoint: https://dod-graph.microsoft.us // - DE // AD Endpoint: https://portal.microsoftazure.de // Graph Endpoint: https://graph.microsoft.de // - CN // AD Endpoint: https://login.chinacloudapi.cn // Graph Endpoint: https://microsoftgraph.chinacloudapi.cn stringValues["azure_ad_endpoint"] = ""; // Support single-tenant applications that are not able to use the "common" multiplexer stringValues["azure_tenant_id"] = ""; // Support synchronising files based on user desire // - default = whatever order these came in as, processed essentially FIFO // - size_asc = file size ascending // - size_dsc = file size descending // - name_asc = file name ascending // - name_dsc = file name descending stringValues["transfer_order"] = "default"; // Recycle Bin Configuration // Enable|Disable feature boolValues["use_recycle_bin"] = false; // Recycle Bin Folder - empty string as a default stringValues["recycle_bin_path"] = ""; // - Store how many times was --verbose added longValues["verbose"] = verbosityCount; // - The amount of time (seconds) between monitor sync loops longValues["monitor_interval"] = defaultMonitorInterval; // - What size of file should be skipped? longValues["skip_size"] = 0; // - How many 'loops' when using --monitor, before we print out high frequency recurring items? longValues["monitor_log_frequency"] = 12; // - Number of N sync runs before performing a full local scan of sync_dir // By default 12 which means every ~60 minutes a full disk scan of sync_dir will occur // 'monitor_interval' * 'monitor_fullscan_frequency' = 3600 = 1 hour longValues["monitor_fullscan_frequency"] = 12; // - Number of children in a path that is locally removed which will be classified as a 'big data delete' longValues["classify_as_big_delete"] = 1000; // - Configure the default folder permission attributes for newly created folders longValues["sync_dir_permissions"] = defaultDirectoryPermissionMode; // - Configure the default file permission attributes for newly created file longValues["sync_file_permissions"] = defaultFilePermissionMode; // - Configure download / upload rate limits longValues["rate_limit"] = 0; // - To ensure we do not fill up the load disk, how much disk space should be reserved by default longValues["space_reservation"] = 50 * 2^^20; // 50 MB as Bytes // - How large should our file fragments be when uploading as an 'upload session' ? longValues["file_fragment_size"] = defaultFileFragmentSize; // whole number, treated as MB, will be converted to bytes within performSessionFileUpload(). Default is 10. // HTTPS & CURL Operation Settings // - Maximum time an operation is allowed to take // This includes dns resolution, connecting, data transfer, etc - controls CURLOPT_TIMEOUT // CURLOPT_TIMEOUT: This option sets the maximum time in seconds that you allow the libcurl transfer operation to take. // This is useful for controlling how long a specific transfer should take before it is considered too slow and aborted. However, it does not directly control the keep-alive time of a socket. longValues["operation_timeout"] = defaultOperationTimeout; // libcurl dns_cache_timeout timeout longValues["dns_timeout"] = defaultDnsTimeout; // Timeout for HTTPS connections - controls CURLOPT_CONNECTTIMEOUT // CURLOPT_CONNECTTIMEOUT: This option sets the timeout, in seconds, for the connection phase. It is the maximum time allowed for the connection to be established. longValues["connect_timeout"] = defaultConnectTimeout; // Timeout for activity on a HTTPS connection longValues["data_timeout"] = defaultDataTimeout; // What IP protocol version should be used when communicating with OneDrive longValues["ip_protocol_version"] = defaultIpProtocol; // 0 = IPv4 + IPv6, 1 = IPv4 Only, 2 = IPv6 Only // What is the default age that a curl engine should be left idle for, before being destroyed longValues["max_curl_idle"] = 120; // Number of concurrent threads longValues["threads"] = defaultConcurrentThreads; // Default is 8, user can increase to max of 16 or decrease // Do we wish to upload only? boolValues["upload_only"] = false; // Do we need to check for the .nomount file on the mount point? boolValues["check_nomount"] = false; // Do we need to check for the .nosync file anywhere? boolValues["check_nosync"] = false; // Do we wish to download only? boolValues["download_only"] = false; // Do we disable notifications? boolValues["disable_notifications"] = false; // Do we bypass all the download validation? // - This is critically important not to disable, but because of SharePoint 'feature' can be highly desirable to enable boolValues["disable_download_validation"] = false; // Do we bypass all the upload validation? // - This is critically important not to disable, but because of SharePoint 'feature' can be highly desirable to enable boolValues["disable_upload_validation"] = false; // Do we enable logging? boolValues["enable_logging"] = false; // Do we force HTTP 1.1 for connections to the OneDrive API // - By default we use the curl library default, which should be HTTP2 for most operations governed by the OneDrive API boolValues["force_http_11"] = false; // Do we treat the local file system as the source of truth for our data? boolValues["local_first"] = false; // Do we ignore local file deletes, so that all files are retained online? boolValues["no_remote_delete"] = false; // Do we skip symbolic links? boolValues["skip_symlinks"] = false; // Do we enable debugging for all HTTPS flows. Critically important for debugging API issues. boolValues["debug_https"] = false; // Do we skip .files and .folders? boolValues["skip_dotfiles"] = false; // Do we perform a 'dry-run' with no local or remote changes actually being performed? boolValues["dry_run"] = false; // Do we sync all the files in the 'sync_dir' root? boolValues["sync_root_files"] = false; // Do we delete source file after successful transfer? boolValues["remove_source_files"] = false; // Do we delete source folders after successful transfer? boolValues["remove_source_folders"] = false; // Do we perform strict matching for skip_dir? boolValues["skip_dir_strict_match"] = false; // Do we perform a --resync? boolValues["resync"] = false; // 'resync' now needs to be acknowledged based on the 'risk' of using it boolValues["resync_auth"] = false; // Ignore data safety checks and overwrite local data rather than preserve & rename // - This is a config file option ONLY boolValues["bypass_data_preservation"] = false; // Allow enable / disable of the syncing of OneDrive Business Shared items (files & folders) via configuration file boolValues["sync_business_shared_items"] = false; // Log to application output running configuration values boolValues["display_running_config"] = false; // Configure read-only authentication scope boolValues["read_only_auth_scope"] = false; // Flag to cleanup local files when using --download-only boolValues["cleanup_local_files"] = false; // Perform a permanentDelete on deletion activities boolValues["permanent_delete"] = false; // Controls how the application handles the Microsoft SharePoint 'feature' of modifying all PDF, MS Office & HTML files with added XML content post upload // - There are 2 ways to solve this: // 1. Download the modified file immediately after upload as per v2.4.x (default) // 2. Create a new online version of the file, which then contributes to the users 'quota' boolValues["create_new_file_version"] = false; // Some Linux editors (vi|vim|nvim|emacs|LibreOffice) use use a safe file-save strategy designed to avoid data corruption. As such, as part of this Process // they 'track' the last modified timestamp of the 'new' file that they create on file save (regardless of new file, modified file) // If *any* other application in the background then 'updates' this timestamp, these Linux editors complain saying that the file has changed: // // WARNING: The file has been changed since reading it!!! // Do you really want to write to it (y/n)? // // This is simply because they are looking at the timestamp and *not* if the content has actually changed .... a poor design on those editors // // This option, when enabled, forces the client to use a 'session' upload, which, when the 'file' is uploaded by the session, this includes the local timestamp of the file // and Microsoft OneDrive should be respecting this timestamp as the timestamp to use|set when storing that file online boolValues["force_session_upload"] = false; // Obsidian Editor has been written in such a way that it is constantly writing each and every keystroke to a file. // Not only is this really bad application behaviour, for this client, this means the application is constantly writing to disk, thus attempting to upload file changes. // Unfortunately Obsidian on Linux does not provide a built-in way to disable atomic saves or switch to a backup-copy method via configuration. // This flag tells the 'onedrive' inotify monitor to 'sleep' for this period of time, so that constant system writes are not creating instant data uploads boolValues["delay_inotify_processing"] = false; longValues["inotify_delay"] = defaultInotifyDelay; // default of 5 seconds // Webhook Feature Options boolValues["webhook_enabled"] = false; stringValues["webhook_public_url"] = ""; stringValues["webhook_listening_host"] = ""; longValues["webhook_listening_port"] = 8888; longValues["webhook_expiration_interval"] = 600; longValues["webhook_renewal_interval"] = 300; longValues["webhook_retry_interval"] = 60; // WebSocket Feature Options boolValues["disable_websocket_support"] = false; // GUI File Transfer and Deletion Notifications boolValues["notify_file_actions"] = false; // Display file transfer metrics // - Enable the calculation of transfer metrics (duration,speed) for the transfer of a file boolValues["display_transfer_metrics"] = false; // Enable writing extended attributes about a file to xattr values // - file creator // - file last modifier boolValues["write_xattr_data"] = false; // Diable setting the permissions for directories and files, using the inherited permissions boolValues["disable_permission_set"] = false; // Use authentication via Intune SSO via Microsoft Identity Broker (microsoft-identity-broker) dbus session boolValues["use_intune_sso"] = false; // Use authentication via OAuth2 Device Authorisation Flow boolValues["use_device_auth"] = false; // GUI | Display Manager Integration boolValues["display_manager_integration"] = false; // Disable GitHub Version check boolValues["disable_version_check"] = false; // EXPAND USERS HOME DIRECTORY // Determine the users home directory. // Need to avoid using ~ here as expandTilde() below does not interpret correctly when running under init.d or systemd scripts // Check for HOME environment variable if (environment.get("HOME") != ""){ // Use HOME environment variable if (debugLogging) {addLogEntry("runtime_environment: HOME environment variable detected, expansion of '~' should be possible", ["debug"]);} defaultHomePath = environment.get("HOME"); shellEnvironmentSet = true; } else { if ((environment.get("SHELL") == "") && (environment.get("USER") == "")){ // No shell is set or username - observed case when running as systemd service under CentOS 7.x if (debugLogging) {addLogEntry("runtime_environment: No HOME, SHELL or USER environment variable configuration detected. Expansion of '~' not possible", ["debug"]);} defaultHomePath = "/root"; shellEnvironmentSet = false; } else { // A shell & valid user is set, but no HOME is set, use ~ which can be expanded if (debugLogging) {addLogEntry("runtime_environment: SHELL and USER environment variable detected, expansion of '~' should be possible", ["debug"]);} defaultHomePath = "~"; shellEnvironmentSet = true; } } // Outcome of setting 'defaultHomePath' if (debugLogging) {addLogEntry("runtime_environment: Calculated defaultHomePath: " ~ defaultHomePath, ["debug"]);} // Configure the default path for the Recycle Bin // Both GNOME and KDE use '~/.local/share/Trash/' as the default path // ~/.local/share/Trash/ // ├── files/ # The actual trashed files // └── info/ # .trashinfo metadata about each file (original path, deletion date) setValueString("recycle_bin_path", defaultHomePath ~ "/.local/share/Trash/"); recycleBinParentPath = getValueString("recycle_bin_path"); // DEVELOPER OPTIONS // display_memory = true | false // - It may be desirable to display the memory usage of the application to assist with diagnosing memory issues with the application // - This is especially beneficial when debugging or performing memory tests with Valgrind boolValues["display_memory"] = false; // monitor_max_loop = long value // - It may be desirable to, when running in monitor mode, force monitor mode to 'quit' after X number of loops // - This is especially beneficial when debugging or performing memory tests with Valgrind longValues["monitor_max_loop"] = 0; // display_sync_options = true | false // - It may be desirable to see what options are being passed into performSync() without enabling the full verbose debug logging boolValues["display_sync_options"] = false; // force_children_scan = true | false // - Force client to use /children rather than /delta to query changes on OneDrive // - This option flags nationalCloudDeployment as true, forcing the client to act like it is using a National Cloud Deployment model boolValues["force_children_scan"] = false; // display_processing_time = true | false // - Enabling this option will add function processing times to the console output // - This then enables tracking of where the application is spending most amount of time when processing data when users have questions re performance boolValues["display_processing_time"] = false; // Function variables string configDirBase; string systemConfigDirBase = "/etc"; bool configurationInitialised = false; // Initialise the application configuration, using the provided --confdir option was passed in if (!confdirOption.empty) { // A CLI 'confdir' was passed in // Clean up any stray " .. these should not be there for correct process handling of the configuration option confdirOption = strip(confdirOption,"\""); if (debugLogging) {addLogEntry("configDirName: CLI override to set configDirName to: " ~ confdirOption, ["debug"]);} // For the passed in --confdir option .. if (canFind(confdirOption,"~")) { // A ~ was found if (debugLogging) {addLogEntry("configDirName: A '~' was found in configDirName, using the calculated 'defaultHomePath' to replace '~'", ["debug"]);} configDirName = defaultHomePath ~ strip(confdirOption,"~","~"); } else { configDirName = confdirOption; } } else { // Determine the base directory relative to which user specific configuration files should be stored if (environment.get("XDG_CONFIG_HOME") != ""){ if (debugLogging) {addLogEntry("configDirBase: XDG_CONFIG_HOME environment variable set", ["debug"]);} configDirBase = environment.get("XDG_CONFIG_HOME"); } else { // XDG_CONFIG_HOME does not exist on systems where X11 is not present - ie - headless systems / servers if (debugLogging) {addLogEntry("configDirBase: WARNING - no XDG_CONFIG_HOME environment variable set", ["debug"]);} configDirBase = buildNormalizedPath(buildPath(defaultHomePath, ".config")); } // Output configDirBase calculation if (debugLogging) { addLogEntry("configDirBase: " ~ configDirBase, ["debug"]); // Set the calculated application configuration directory addLogEntry("configDirName: Configuring application to use calculated config path", ["debug"]); } // configDirBase contains the correct path so we do not need to check for presence of '~' configDirName = buildNormalizedPath(buildPath(configDirBase, "onedrive")); } // systemConfigDirBase contains the correct path, build the correct path for the system config file systemConfigDirName = buildNormalizedPath(buildPath(systemConfigDirBase, "onedrive")); // Configuration directory should now have been correctly identified if (!exists(configDirName)) { // Attempt path creation try { // create the configuration directory mkdirRecurse(configDirName); // Configure the applicable permissions for the folder configDirName.setAttributes(returnRequiredDirectoryPermissions()); } catch (std.file.FileException e) { // Creating the configuration directory failed addLogEntry("ERROR: Unable to create the required application configuration directory: " ~ e.msg, ["info", "notify"]); // Use exit scopes to shutdown API return EXIT_FAILURE; } } else { // The config path exists // The path that exists must be a directory, not a file if (!isDir(configDirName)) { if (!confdirOption.empty) { // the configuration path was passed in by the user .. user error addLogEntry("ERROR: --confdir entered value is an existing file instead of an existing directory"); } else { // other error addLogEntry("ERROR: " ~ confdirOption ~ " is a file rather than a directory"); } // Must exit exit(EXIT_FAILURE); } } // Update application set variables based on configDirName // - What is the full path for the 'refresh_token' refreshTokenFilePath = buildNormalizedPath(buildPath(configDirName, "refresh_token")); // - What is the full path for the 'intune_account' intuneAccountDetailsFilePath = buildNormalizedPath(buildPath(configDirName, "intune_account")); // - What is the full path for the 'delta_link' deltaLinkFilePath = buildNormalizedPath(buildPath(configDirName, "delta_link")); // - What is the full path for the 'items.sqlite3' - the database cache file databaseFilePath = buildNormalizedPath(buildPath(configDirName, "items.sqlite3")); // - What is the full path for the 'items-dryrun.sqlite3' - the dry-run database cache file databaseFilePathDryRun = buildNormalizedPath(buildPath(configDirName, "items-dryrun.sqlite3")); // - What is the full path for the 'resume_upload' uploadSessionFilePath = buildNormalizedPath(buildPath(configDirName, "session_upload")); // - What is the full path for the resume 'resume_download' file resumeDownloadFilePath = buildNormalizedPath(buildPath(configDirName, "resume_download")); // - What is the full path for the 'sync_list' file syncListFilePath = buildNormalizedPath(buildPath(configDirName, "sync_list")); // - What is the full path for the 'config' - the user file to configure the application userConfigFilePath = buildNormalizedPath(buildPath(configDirName, "config")); // - What is the full path for the system 'config' file if it is required systemConfigFilePath = buildNormalizedPath(buildPath(systemConfigDirName, "config")); // To determine if any configuration items has changed, where a --resync would be required, we need to have a hash file for the following items // - 'config.backup' file // - applicable 'config' file // - 'sync_list' file // - 'business_shared_items' file configBackupFile = buildNormalizedPath(buildPath(configDirName, ".config.backup")); configHashFile = buildNormalizedPath(buildPath(configDirName, ".config.hash")); syncListHashFile = buildNormalizedPath(buildPath(configDirName, ".sync_list.hash")); // Debug Output for application set variables based on configDirName if (debugLogging) { addLogEntry("refreshTokenFilePath = " ~ refreshTokenFilePath, ["debug"]); addLogEntry("intuneAccountDetailsFilePath = " ~ intuneAccountDetailsFilePath, ["debug"]); addLogEntry("deltaLinkFilePath = " ~ deltaLinkFilePath, ["debug"]); addLogEntry("databaseFilePath = " ~ databaseFilePath, ["debug"]); addLogEntry("databaseFilePathDryRun = " ~ databaseFilePathDryRun, ["debug"]); addLogEntry("uploadSessionFilePath = " ~ uploadSessionFilePath, ["debug"]); addLogEntry("userConfigFilePath = " ~ userConfigFilePath, ["debug"]); addLogEntry("syncListFilePath = " ~ syncListFilePath, ["debug"]); addLogEntry("systemConfigFilePath = " ~ systemConfigFilePath, ["debug"]); addLogEntry("configBackupFile = " ~ configBackupFile, ["debug"]); addLogEntry("configHashFile = " ~ configHashFile, ["debug"]); addLogEntry("syncListHashFile = " ~ syncListHashFile, ["debug"]); } // Configure the Hash and Backup File Permission Value string valueToConvert = to!string(defaultFilePermissionMode); auto convertedValue = parse!long(valueToConvert, 8); convertedPermissionValue = to!int(convertedValue); // Do not try and load any user configuration file if --help was used if (helpRequested) { return true; } else { // Initialise the application using the configuration file if it exists if (!exists(userConfigFilePath)) { // 'user' configuration file does not exist .. but did the user specify a custom configuration directory via --confdir ? if (confdirOption.empty) { // No --confdir entry // Is there a system configuration file? if (!exists(systemConfigFilePath)) { // 'system' configuration file does not exist if (verboseLogging) {addLogEntry("No user or system config file found, using application defaults", ["verbose"]);} applicableConfigFilePath = userConfigFilePath; configurationInitialised = true; } else { // 'system' configuration file exists // can we load the configuration file without error? if (loadConfigFile(systemConfigFilePath)) { // configuration file loaded without error addLogEntry("System configuration file successfully loaded"); // Set 'applicableConfigFilePath' to equal the 'config' we loaded applicableConfigFilePath = systemConfigFilePath; // Update the configHashFile path value to ensure we are using the system 'config' file for the hash configHashFile = buildNormalizedPath(buildPath(systemConfigDirName, ".config.hash")); configurationInitialised = true; } else { // there was a problem loading the configuration file addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("System configuration file has errors - please check your configuration"); } } } else { // Set 'applicableConfigFilePath' to equal the 'config' path specified via --confdir applicableConfigFilePath = userConfigFilePath; configurationInitialised = true; } } else { // 'user' configuration file exists in the specified path // can we load the configuration file without error? if (loadConfigFile(userConfigFilePath)) { // configuration file loaded without error addLogEntry("Configuration file successfully loaded"); // Set 'applicableConfigFilePath' to equal the 'config' we loaded applicableConfigFilePath = userConfigFilePath; configurationInitialised = true; } else { // there was a problem loading the configuration file addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("Configuration file has errors - please check your configuration"); } } // Advise the user path that we will use for the application state data if (canFind(applicableConfigFilePath, configDirName)) { if (verboseLogging) {addLogEntry("Using 'user' configuration path for application config and state data: " ~ configDirName, ["verbose"]);} } else { if (canFind(applicableConfigFilePath, systemConfigDirName)) { if (verboseLogging) { addLogEntry("Using 'system' configuration path for application config data: " ~ systemConfigDirName, ["verbose"]); addLogEntry("Using 'user' configuration path for application state data: " ~ configDirName, ["verbose"]); } } } } // return if the configuration was initialised return configurationInitialised; } // Create a backup of the 'config' file if it does not exist void createBackupConfigFile() { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); if (!getValueBool("dry_run")) { // Is there a backup of the config file if the config file exists? if (exists(applicableConfigFilePath)) { if (debugLogging) {addLogEntry("Creating a backup of the applicable config file", ["debug"]);} // create backup copy of current config file try { std.file.copy(applicableConfigFilePath, configBackupFile); // File Copy should only be readable by the user who created it - 0600 permissions needed configBackupFile.setAttributes(convertedPermissionValue); } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, configBackupFile, FsErrorSeverity.warning); } } } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not creating backup config file as --dry-run has been used"); } } // Return a given string value based on the provided key string getValueString(string key) { auto p = key in stringValues; if (p) { return *p; } else { throw new Exception("Missing config value: " ~ key); } } // Return a given long value based on the provided key long getValueLong(string key) { auto p = key in longValues; if (p) { return *p; } else { throw new Exception("Missing config value: " ~ key); } } // Return a given bool value based on the provided key bool getValueBool(string key) { auto p = key in boolValues; if (p) { return *p; } else { throw new Exception("Missing config value: " ~ key); } } // Set a given string value based on the provided key void setValueString(string key, string value) { stringValues[key] = value; } // Set a given long value based on the provided key void setValueLong(string key, long value) { longValues[key] = value; } // Set a given long value based on the provided key void setValueBool(string key, bool value) { boolValues[key] = value; } // Configure the directory octal permission value void configureRequiredDirectoryPermissions() { // return the directory permission mode required // - return octal!defaultDirectoryPermissionMode; ... cant be used .. which is odd // Error: variable defaultDirectoryPermissionMode cannot be read at compile time if (getValueLong("sync_dir_permissions") != defaultDirectoryPermissionMode) { // return user configured permissions as octal integer string valueToConvert = to!string(getValueLong("sync_dir_permissions")); auto convertedValue = parse!long(valueToConvert, 8); configuredDirectoryPermissionMode = to!int(convertedValue); } else { // return default as octal integer string valueToConvert = to!string(defaultDirectoryPermissionMode); auto convertedValue = parse!long(valueToConvert, 8); configuredDirectoryPermissionMode = to!int(convertedValue); } } // Configure the file octal permission value void configureRequiredFilePermissions() { // return the file permission mode required // - return octal!defaultFilePermissionMode; ... cant be used .. which is odd // Error: variable defaultFilePermissionMode cannot be read at compile time if (getValueLong("sync_file_permissions") != defaultFilePermissionMode) { // return user configured permissions as octal integer string valueToConvert = to!string(getValueLong("sync_file_permissions")); auto convertedValue = parse!long(valueToConvert, 8); configuredFilePermissionMode = to!int(convertedValue); } else { // return default as octal integer string valueToConvert = to!string(defaultFilePermissionMode); auto convertedValue = parse!long(valueToConvert, 8); configuredFilePermissionMode = to!int(convertedValue); } } // Read the configuredDirectoryPermissionMode and return int returnRequiredDirectoryPermissions() { if (configuredDirectoryPermissionMode == 0) { // the configured value is zero, this means that directories would get // values of d--------- configureRequiredDirectoryPermissions(); } return configuredDirectoryPermissionMode; } // Read the configuredFilePermissionMode and return int returnRequiredFilePermissions() { if (configuredFilePermissionMode == 0) { // the configured value is zero configureRequiredFilePermissions(); } return configuredFilePermissionMode; } // Set file permissions for 'refresh_token' and 'intune_account' to 0600 int returnSecureFilePermission() { string valueToConvert = to!string(defaultFilePermissionMode); auto convertedValue = parse!long(valueToConvert, 8); return to!int(convertedValue); } // Load a configuration file from the provided filename private bool loadConfigFile(string filename) { try { addLogEntry("Reading configuration file: " ~ filename); readText(filename); } catch (std.file.FileException e) { addLogEntry("ERROR: Unable to access " ~ e.msg); return false; } auto file = File(filename, "r"); string lineBuffer; scope(exit) { file.close(); object.destroy(file); object.destroy(lineBuffer); } scope(failure) { file.close(); object.destroy(file); object.destroy(lineBuffer); } foreach (line; file.byLine()) { lineBuffer = stripLeft(line).to!string; if (lineBuffer.empty || lineBuffer[0] == ';' || lineBuffer[0] == '#') continue; auto c = lineBuffer.matchFirst(configRegex); if (c.empty) { addLogEntry("Malformed config line: " ~ lineBuffer); addLogEntry(); addLogEntry("Please review the documentation on how to correctly configure this application."); forceExit(); } c.popFront(); // skip the whole match string key = c.front.dup; c.popFront(); // Handle deprecated keys switch (key) { case "min_notify_changes": case "force_http_2": addLogEntry("The option '" ~ key ~ "' has been deprecated and will be ignored. Please read the updated documentation and update your client configuration to remove this option."); continue; case "sync_business_shared_folders": addLogEntry(); addLogEntry("The option 'sync_business_shared_folders' has been deprecated and the process for synchronising Microsoft OneDrive Business Shared Folders has changed."); addLogEntry("Please review the revised documentation on how to correctly configure this application feature."); addLogEntry("You must update your client configuration and make changes to your local filesystem and online data to use this capability."); return false; default: break; } // Process other keys if (key in boolValues) { // Strip quotes and whitespace string rawValue = to!string(c.front.dup); // Evaluate rawValue if (rawValue == "true") { setValueBool(key, true); // Additional config-specific flags for specific keys if (key == "check_nosync") configFileCheckNoSync = true; if (key == "skip_dotfiles") configFileSkipDotfiles = true; if (key == "skip_symlinks") configFileSkipSymbolicLinks = true; if (key == "sync_business_shared_items") configFileSyncBusinessSharedItems = true; } else if (rawValue == "false") { setValueBool(key, false); } else { addLogEntry("Invalid boolean value for key in config file: " ~ key ~ " = " ~ to!string(c.front.dup)); addLogEntry("ERROR: Only 'true' or 'false' are accepted for this setting."); forceExit(); } } else if (key in stringValues) { string value = c.front.dup; setValueString(key, value); if (key == "sync_dir") { if (!strip(value).empty) { configFileSyncDir = value; } else { addLogEntry(); addLogEntry("Invalid value for key in config file: " ~ key); addLogEntry("ERROR: sync_dir in config file cannot be empty - this is a fatal error and must be corrected"); addLogEntry(); forceExit(); } } else if (key == "skip_file") { // Flag this as true configFileSkipFileReadIn = true; // Merge safely, removing empty entries and de-duplicating configFileSkipFile = mergePipeDelimitedRulesDedup(configFileSkipFile, to!string(c.front.dup)); // Update stored config value setValueString("skip_file", configFileSkipFile); } else if (key == "skip_dir") { // Merge safely, removing empty entries and de-duplicating configFileSkipDir = mergePipeDelimitedRulesDedup(configFileSkipDir, to!string(c.front.dup)); // Update stored config value setValueString("skip_dir", configFileSkipDir); } else if (key == "single_directory") { // --single-directory Strip quotation marks from path // This is an issue when using ONEDRIVE_SINGLE_DIRECTORY with Docker string configFileSingleDirectory = strip(to!string(c.front.dup), "\""); setValueString("single_directory", configFileSingleDirectory); } else if (key == "azure_ad_endpoint") { switch (value) { case "": addLogEntry("Using default config option for Global Azure AD Endpoints"); break; case "USL4": addLogEntry("Using config option for Azure AD for US Government Endpoints"); break; case "USL5": addLogEntry("Using config option for Azure AD for US Government Endpoints (DOD)"); break; case "DE": addLogEntry("Using config option for Azure AD Germany"); break; case "CN": addLogEntry("Using config option for Azure AD China operated by VNET"); break; default: addLogEntry("Unknown Azure AD Endpoint - using Global Azure AD Endpoints"); } } else if (key == "transfer_order") { switch (value) { case "size_asc": addLogEntry("Files will be transferred sorted by ascending size (smallest first)"); break; case "size_dsc": addLogEntry("Files will be transferred sorted by descending size (largest first)"); break; case "name_asc": addLogEntry("Files will be transferred sorted by ascending name (A -> Z)"); break; case "name_dsc": addLogEntry("Files will be transferred sorted by descending name (Z -> A)"); break; default: addLogEntry("Files will be transferred in original order that they were received (FIFO)"); } } else if (key == "application_id") { string tempApplicationId = strip(value); if (tempApplicationId.empty) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); if (debugLogging) {addLogEntry("application_id in config file cannot be empty - using default application_id", ["debug"]);} setValueString("application_id", defaultApplicationId); } } else if (key == "drive_id") { string tempDriveId = strip(value); if (tempDriveId.empty) { addLogEntry(); addLogEntry("Invalid value for key in config file: " ~ key); if (debugLogging) {addLogEntry("drive_id in config file cannot be empty - this is a fatal error and must be corrected by removing this entry from your config file.", ["debug"]);} addLogEntry(); forceExit(); } else { configFileDriveId = tempDriveId; } } else if (key == "log_dir") { string tempLogDir = strip(value); if (tempLogDir.empty) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); if (debugLogging) {addLogEntry("log_dir in config file cannot be empty - using default log_dir", ["debug"]);} setValueString("log_dir", defaultLogFileDir); } } } else if (key in longValues) { ulong thisConfigValue; try { thisConfigValue = to!ulong(c.front.dup); } catch (std.conv.ConvException) { addLogEntry("Invalid value for key in config file: " ~ key); return false; } setValueLong(key, thisConfigValue); if (key == "monitor_interval") { // if key is 'monitor_interval' the value must be 300 or greater ulong tempValue = thisConfigValue; // the temp value needs to be 300 or greater if (tempValue < defaultMonitorInterval) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = defaultMonitorInterval; } setValueLong("monitor_interval", tempValue); } else if (key == "monitor_fullscan_frequency") { // if key is 'monitor_fullscan_frequency' the value must be 12 or greater ulong tempValue = thisConfigValue; // the temp value needs to be 12 or greater if (tempValue < 12) { // If this is not set to zero (0) then we are not disabling 'monitor_fullscan_frequency' if (tempValue != 0) { // invalid value addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = 12; } } setValueLong("monitor_fullscan_frequency", tempValue); } else if (key == "space_reservation") { // if key is 'space_reservation' we have to calculate MB -> bytes ulong tempValue = thisConfigValue; // a value of 0 needs to be made at least 1MB .. if (tempValue == 0) { addLogEntry("Invalid value for key in config file - using 1MB: " ~ key); tempValue = 1; } setValueLong("space_reservation", tempValue * 2^^20); } else if (key == "ip_protocol_version") { ulong tempValue = thisConfigValue; if (tempValue > 2) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = defaultIpProtocol; } setValueLong("ip_protocol_version", tempValue); } else if (key == "threads") { ulong tempValue = thisConfigValue; if (tempValue > 16) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = defaultConcurrentThreads; } setValueLong("threads", tempValue); } else if (key == "inotify_delay") { ulong tempValue = thisConfigValue; if ((tempValue < 5)||(tempValue > 15)) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = defaultInotifyDelay; } setValueLong("inotify_delay", tempValue); } else if (key == "skip_size") { // Flag this for triggering --resync requirement configFileSkipSize = true; ulong tempValue = thisConfigValue; // If set, this must be greater than 0 if (tempValue <= 0) { addLogEntry("Invalid value for key in config file - using default value: " ~ key); tempValue = 0; } setValueLong("skip_size", tempValue); } else if (key == "file_fragment_size") { ulong tempValue = thisConfigValue; // If set, this must be greater than the default, but also aligning to Microsoft upper limit of 60 MiB // Enforce lower bound (must be greater than default) if (tempValue < defaultFileFragmentSize) { addLogEntry("Invalid value for key in config file (too low) - using default value: " ~ key); tempValue = defaultFileFragmentSize; } // Enforce upper bound (safe maximum) else if (tempValue > defaultMaxFileFragmentSize) { addLogEntry("Invalid value for key in config file (too high) - using maximum safe value: " ~ key); tempValue = defaultMaxFileFragmentSize; } setValueLong("file_fragment_size", tempValue); } } else { addLogEntry("Unknown key in config file: " ~ key); return false; } } // If we read in 'skip_file' from the 'config' file, this will be 'true' if (configFileSkipFileReadIn) { // The user added entries, are the application defaults included or were these discarded / discounted? // Check for temporary and/or transient files to skip (application defaults) checkForSkipFileDefaults(); } // Return that we were able to read in the config file and parse the options without issue return true; } // Perform a check on 'skip_file' configuration post reading from 'config' file void checkForSkipFileDefaults() { // Split both the default and user values auto defaultEntries = defaultSkipFile.split('|').map!(a => a.strip).array; auto userEntries = configFileSkipFile.split('|').map!(a => a.strip).array; string[] missingDefaults; // Check if all defaults exist in user config foreach (defaultEntry; defaultEntries) { if (!userEntries.canFind(defaultEntry)) { missingDefaults ~= defaultEntry; } } // Display warning message about missing default entries for temporary and/or transient files that should be skipped if (!missingDefaults.empty) { addLogEntry(); addLogEntry("WARNING: Your 'skip_file' configuration is missing important default entries. Temporary and/or transient files that would normally be skipped may now be included in syncing.", ["info", "notify"]); addLogEntry(); if (verboseLogging) { addLogEntry("By default, the following types of temporary and/or transient files are skipped:", ["verbose"]); addLogEntry(" Files that start with '~' (Temporary or backup files that are not intended to be saved permanently)", ["verbose"]); addLogEntry(" Files that start with '.~' (e.g., LibreOffice lock files)", ["verbose"]); addLogEntry(" Files that end with '.tmp' (Generic temporary files created by applications like browsers, editors, installers)", ["verbose"]); addLogEntry(" Files that end with '.swp' (Transient files created by editors such as vim and vi)", ["verbose"]); addLogEntry(" Files that end with '.partial' (Partially downloaded files, incomplete by nature, should not be synced)", ["verbose"]); addLogEntry(); addLogEntry(" Missing the following important 'skip_file' entries: " ~ missingDefaults.join(", "), ["verbose"]); addLogEntry(); addLogEntry("Reference: https://github.com/abraunegg/onedrive/blob/master/docs/application-config-options.md#skip_file", ["verbose"]); addLogEntry(); } } } // Update the application configuration based on CLI passed in parameters void updateFromArgs(string[] cliArgs) { // Add additional CLI options that are NOT configurable via config file stringValues["create_directory"] = ""; stringValues["create_share_link"] = ""; stringValues["destination_directory"] = ""; stringValues["get_file_link"] = ""; stringValues["modified_by"] = ""; stringValues["sharepoint_library_name"] = ""; stringValues["remove_directory"] = ""; stringValues["single_directory"] = ""; stringValues["source_directory"] = ""; stringValues["auth_files"] = ""; stringValues["auth_response"] = ""; stringValues["share_password"] = ""; stringValues["download_single_file"] = ""; boolValues["display_config"] = false; boolValues["display_sync_status"] = false; boolValues["display_quota"] = false; boolValues["print_token"] = false; boolValues["logout"] = false; boolValues["reauth"] = false; boolValues["monitor"] = false; boolValues["synchronize"] = false; boolValues["force"] = false; boolValues["list_business_shared_items"] = false; boolValues["sync_business_shared_files"] = false; boolValues["force_sync"] = false; boolValues["with_editing_perms"] = false; // Specific options for CLI input handling stringValues["sync_dir_cli"] = ""; // Application Startup option validation try { string tmpStr; bool tmpBol; long tmpVerb; // duplicated from main.d to get full help output! auto opt = getopt( cliArgs, std.getopt.config.bundling, std.getopt.config.caseSensitive, "auth-files", "Perform authentication via files rather than an interactive dialogue. The application reads/writes the required values from/to the specified files", &stringValues["auth_files"], "auth-response", "Perform authentication via a supplied response URL rather than an interactive dialogue", &stringValues["auth_response"], "check-for-nomount", "Check for the presence of .nosync in the syncdir root. If found, do not perform sync", &boolValues["check_nomount"], "check-for-nosync", "Check for the presence of .nosync in each directory. If found, skip directory from sync", &boolValues["check_nosync"], "classify-as-big-delete", "Number of children in a path that is locally removed which will be classified as a 'big data delete'", &longValues["classify_as_big_delete"], "cleanup-local-files", "Clean up additional local files when using --download-only. This will remove local data", &boolValues["cleanup_local_files"], "create-directory", "Create a directory on OneDrive. No synchronisation will be performed", &stringValues["create_directory"], "create-share-link", "Create a shareable link for an existing file on OneDrive", &stringValues["create_share_link"], "debug-https", "Debug OneDrive HTTPS communication.", &boolValues["debug_https"], "destination-directory", "Destination directory for renamed or moved items on OneDrive. No synchronisation will be performed", &stringValues["destination_directory"], "disable-notifications", "Do not use desktop notifications in monitor mode", &boolValues["disable_notifications"], "disable-download-validation", "Disable download validation when downloading from OneDrive", &boolValues["disable_download_validation"], "disable-upload-validation", "Disable upload validation when uploading to OneDrive", &boolValues["disable_upload_validation"], "display-config", "Display what options the client will use as currently configured. No synchronisation will be performed", &boolValues["display_config"], "display-running-config", "Display what options the client has been configured to use on application startup", &boolValues["display_running_config"], "display-sync-status", "Display the sync status of the client. No synchronisation will be performed", &boolValues["display_sync_status"], "display-quota", "Display the quota status of the client. No synchronisation will be performed", &boolValues["display_quota"], "download-only", "Replicate the OneDrive online state locally, by only downloading changes from OneDrive. Do not upload local changes to OneDrive", &boolValues["download_only"], "download-file", "Download a single file from Microsoft OneDrive", &stringValues["download_single_file"], "dry-run", "Perform a trial sync with no changes made", &boolValues["dry_run"], "enable-logging", "Enable client activity to a separate log file", &boolValues["enable_logging"], "file-fragment-size", "Specify the file fragment size for large file uploads (in MB)", &longValues["file_fragment_size"], "force-http-11", "Force the use of HTTP 1.1 for all operations", &boolValues["force_http_11"], "force", "Force the deletion of data when a 'big delete' is detected", &boolValues["force"], "force-sync", "Force a synchronisation of a specific folder, only when using --sync --single-directory and ignore all non-default skip_dir and skip_file rules", &boolValues["force_sync"], "get-file-link", "Display the file link of a synced file", &stringValues["get_file_link"], "get-sharepoint-drive-id", "Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library", &stringValues["sharepoint_library_name"], "get-O365-drive-id", "Query and return the Office 365 Drive ID for a given Office 365 SharePoint Shared Library (DEPRECATED)", &stringValues["sharepoint_library_name"], "list-shared-items", "List OneDrive Business Shared Items", &boolValues["list_business_shared_items"], "sync-shared-files", "Sync OneDrive Business Shared Files to the local filesystem", &boolValues["sync_business_shared_files"], "local-first", "Synchronise from the local directory source first, before downloading changes from OneDrive", &boolValues["local_first"], "log-dir", "Directory where logging output is saved to, needs to end with a slash", &stringValues["log_dir"], "logout", "Log out the current user", &boolValues["logout"], "modified-by", "Display the last modified by details of a given path", &stringValues["modified_by"], "monitor|m", "Keep monitoring for local and remote changes", &boolValues["monitor"], "monitor-interval", "Number of seconds by which each sync operation is undertaken when idle under monitor mode", &longValues["monitor_interval"], "monitor-fullscan-frequency", "Number of sync runs before performing a full local scan of the synced directory", &longValues["monitor_fullscan_frequency"], "monitor-log-frequency", "Frequency of logging in monitor mode", &longValues["monitor_log_frequency"], "no-remote-delete", "Do not delete local file 'deletes' from OneDrive when using --upload-only", &boolValues["no_remote_delete"], "print-access-token", "Print the access token, useful for debugging", &boolValues["print_token"], "reauth", "Reauthenticate the client with OneDrive", &boolValues["reauth"], "resync", "Forget the last saved state, perform a full sync", &boolValues["resync"], "resync-auth", "Approve the use of performing a --resync action", &boolValues["resync_auth"], "remove-directory", "Remove a directory on OneDrive. No synchronisation will be performed", &stringValues["remove_directory"], "remove-source-files", "Remove source file after successful transfer to OneDrive when using --upload-only", &boolValues["remove_source_files"], "remove-source-folders", "Remove the local directory structure post successful file transfer to Microsoft OneDrive when using --upload-only --remove-source-files", &boolValues["remove_source_folders"], "single-directory", "Specify a single local directory within the OneDrive root to sync", &stringValues["single_directory"], "skip-dot-files", "Skip dot files and folders from syncing", &boolValues["skip_dotfiles"], "skip-file", "Skip any files that match this pattern from syncing", &stringValues["skip_file"], "skip-dir", "Skip any directories that match this pattern from syncing", &stringValues["skip_dir"], "skip-size", "Skip new files larger than this size (in MB)", &longValues["skip_size"], "skip-dir-strict-match", "When matching skip_dir directories, only match explicit matches", &boolValues["skip_dir_strict_match"], "skip-symlinks", "Skip syncing of symlinks", &boolValues["skip_symlinks"], "source-directory", "Source directory to rename or move on OneDrive. No synchronisation will be performed", &stringValues["source_directory"], "space-reservation", "The amount of disk space to reserve (in MB) to avoid 100% disk space utilisation", &longValues["space_reservation"], "syncdir", "Specify the local directory used for synchronisation to OneDrive", &stringValues["sync_dir_cli"], "share-password", "Require a password to access the shared link when used with --create-share-link ", &stringValues["share_password"], "sync|s", "Perform a synchronisation with Microsoft OneDrive", &boolValues["synchronize"], "synchronize", "Perform a synchronisation with Microsoft OneDrive (DEPRECATED)", &boolValues["synchronize"], "sync-root-files", "Sync all files in sync_dir root when using sync_list", &boolValues["sync_root_files"], "threads", "Specify a value for the number of worker threads used for parallel upload and download operations", &longValues["threads"], "upload-only", "Replicate the locally configured sync_dir state to OneDrive, by only uploading local changes to OneDrive. Do not download changes from OneDrive", &boolValues["upload_only"], "confdir", "Set the directory used to store the configuration files", &tmpStr, "verbose|v+", "Print more details, useful for debugging (repeat for extra debugging)", &tmpVerb, "version", "Print the version and exit", &tmpBol, "with-editing-perms", "Create a read-write shareable link for an existing file on OneDrive when used with --create-share-link ", &boolValues["with_editing_perms"] ); // Was --syncdir specified? if (!getValueString("sync_dir_cli").empty) { // Build the line we need to update and/or write out string newConfigOptionSyncDirLine = "sync_dir = \"" ~ getValueString("sync_dir_cli") ~ "\""; // Does a 'config' file exist? if (!exists(applicableConfigFilePath)) { // No existing 'config' file exists, create it, and write the 'sync_dir' configuration to it if (!getValueBool("dry_run")) { std.file.write(applicableConfigFilePath, newConfigOptionSyncDirLine); // Config file should only be readable by the user who created it - 0600 permissions needed applicableConfigFilePath.setAttributes(convertedPermissionValue); } } else { // an existing config file exists .. so this now becomes tricky // string replace 'sync_dir' if it exists, in the existing 'config' file, but only if 'sync_dir' (already read in) is different from 'sync_dir_cli' if ( (getValueString("sync_dir")) != (getValueString("sync_dir_cli")) ) { // values are different File applicableConfigFilePathFileHandle = File(applicableConfigFilePath, "r"); string lineBuffer; string[] newConfigFileEntries; // read applicableConfigFilePath line by line auto range = applicableConfigFilePathFileHandle.byLine(); // for each 'config' file line foreach (line; range) { lineBuffer = stripLeft(line).to!string; if (lineBuffer.length == 0 || lineBuffer[0] == ';' || lineBuffer[0] == '#') { newConfigFileEntries ~= [lineBuffer]; } else { auto c = lineBuffer.matchFirst(configRegex); if (!c.empty) { c.popFront(); // skip the whole match string key = c.front.dup; if (key == "sync_dir") { // lineBuffer is the line we want to keep newConfigFileEntries ~= [newConfigOptionSyncDirLine]; } else { newConfigFileEntries ~= [lineBuffer]; } } } } // close original 'config' file if still open if (applicableConfigFilePathFileHandle.isOpen()) { // close open file applicableConfigFilePathFileHandle.close(); } // free memory from file open object.destroy(applicableConfigFilePathFileHandle); // Update the existing item in the file line array if (!getValueBool("dry_run")) { // Open the file with write access using 'w' mode to overwrite existing content File applicableConfigFilePathFileHandleWrite = File(applicableConfigFilePath, "w"); // Write each line from the 'newConfigFileEntries' array to the file foreach (line; newConfigFileEntries) { applicableConfigFilePathFileHandleWrite.writeln(line); } // Is this a running as a container if (entrypointExists) { // write this to the config file so that when config options are checked again, this matches on next run applicableConfigFilePathFileHandleWrite.writeln(newConfigOptionSyncDirLine); } // Flush and close the file handle to ensure all data is written if (applicableConfigFilePathFileHandleWrite.isOpen()) { applicableConfigFilePathFileHandleWrite.flush(); applicableConfigFilePathFileHandleWrite.close(); } // free memory from file open object.destroy(applicableConfigFilePathFileHandleWrite); } } } // Final - configure sync_dir with the value of sync_dir_cli so that it can be used as part of the application configuration and detect change setValueString("sync_dir", getValueString("sync_dir_cli")); } // Was --monitor-interval specified and now set to a value below minimum requirement? if (getValueLong("monitor_interval") < defaultMonitorInterval ) { addLogEntry("Invalid value for --monitor-interval - using default value: " ~ to!string(defaultMonitorInterval)); setValueLong("monitor_interval", defaultMonitorInterval); } // Was --file-fragment-size specified and now set to a value below or above maximum? // Enforce lower bound (must be greater than default) for 'file_fragment_size' if (getValueLong("file_fragment_size") < defaultFileFragmentSize) { addLogEntry("Invalid value for --file-fragment-size (too low) - using default value: " ~ to!string(defaultFileFragmentSize)); setValueLong("file_fragment_size", defaultFileFragmentSize); } // Enforce upper bound (safe maximum) for 'file_fragment_size' if (getValueLong("file_fragment_size") > defaultMaxFileFragmentSize) { addLogEntry("Invalid value for --file-fragment-size (too high) - using maximum safe value: " ~ to!string(defaultMaxFileFragmentSize)); setValueLong("file_fragment_size", defaultMaxFileFragmentSize); } // Was --auth-files used? if (!getValueString("auth_files").empty) { // --auth-files used, need to validate that '~' was not used as a path identifier, and if yes, perform the correct expansion string[] tempAuthFiles = getValueString("auth_files").split(":"); string tempAuthUrl = tempAuthFiles[0]; string tempResponseUrl = tempAuthFiles[1]; string newAuthFilesString; // shell expansion if required if (!shellEnvironmentSet){ // No shell environment is set, no automatic expansion of '~' if present is possible // Does the 'currently configured' tempAuthUrl include a ~ if (canFind(tempAuthUrl, "~")) { // A ~ was found in auth_files(authURL) if (debugLogging) {addLogEntry("auth_files: A '~' was found in 'auth_files(authURL)', using the calculated 'homePath' to replace '~' as no SHELL or USER environment variable set", ["debug"]);} tempAuthUrl = buildNormalizedPath(buildPath(defaultHomePath, strip(tempAuthUrl, "~"))); } // Does the 'currently configured' tempAuthUrl include a ~ if (canFind(tempResponseUrl, "~")) { // A ~ was found in auth_files(authURL) if (debugLogging) {addLogEntry("auth_files: A '~' was found in 'auth_files(tempResponseUrl)', using the calculated 'homePath' to replace '~' as no SHELL or USER environment variable set", ["debug"]);} tempResponseUrl = buildNormalizedPath(buildPath(defaultHomePath, strip(tempResponseUrl, "~"))); } } else { // Shell environment is set, automatic expansion of '~' if present is possible // Does the 'currently configured' tempAuthUrl include a ~ if (canFind(tempAuthUrl, "~")) { // A ~ was found in auth_files(authURL) if (debugLogging) {addLogEntry("auth_files: A '~' was found in the configured 'auth_files(authURL)', automatically expanding as SHELL and USER environment variable is set", ["debug"]);} tempAuthUrl = expandTilde(tempAuthUrl); } // Does the 'currently configured' tempAuthUrl include a ~ if (canFind(tempResponseUrl, "~")) { // A ~ was found in auth_files(authURL) if (debugLogging) {addLogEntry("auth_files: A '~' was found in the configured 'auth_files(tempResponseUrl)', automatically expanding as SHELL and USER environment variable is set", ["debug"]);} tempResponseUrl = expandTilde(tempResponseUrl); } } // Build new string newAuthFilesString = tempAuthUrl ~ ":" ~ tempResponseUrl; if (debugLogging) {addLogEntry("auth_files - updated value: " ~ newAuthFilesString, ["debug"]);} setValueString("auth_files", newAuthFilesString); } if (opt.helpWanted) { outputLongHelp(opt.options); // Shutdown logging, which also flushes all logging buffers shutdownLogging(); // Exit as successful exit(EXIT_SUCCESS); } } catch (GetOptException e) { // getOpt error - must use writeln() here writeln(e.msg); writeln("Try 'onedrive -h' for more information"); // Shutdown logging, which also flushes all logging buffers shutdownLogging(); // Exit as failure exit(EXIT_FAILURE); } catch (Exception e) { // general error - must use writeln() here writeln(e.msg); writeln("Try 'onedrive -h' for more information"); // Shutdown logging, which also flushes all logging buffers shutdownLogging(); // Exit as failure exit(EXIT_FAILURE); } } // Check the arguments passed in for any that will be deprecated void checkDeprecatedOptions(string[] cliArgs) { bool deprecatedCommandsFound = false; foreach (cliArg; cliArgs) { // Check each CLI arg for items that have been deprecated // --synchronize deprecated in v2.5.0, will be removed in future version if (cliArg == "--synchronize") { addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("DEPRECIATION WARNING: --synchronize has been deprecated in favour of --sync or -s"); deprecatedCommandsFound = true; } // --get-O365-drive-id deprecated in v2.5.0, will be removed in future version if (cliArg == "--get-O365-drive-id") { addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("DEPRECIATION WARNING: --get-O365-drive-id has been deprecated in favour of --get-sharepoint-drive-id"); deprecatedCommandsFound = true; } } if (deprecatedCommandsFound) { addLogEntry("DEPRECIATION WARNING: Deprecated commands will be removed in a future release."); addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering } } // Display the applicable application configuration void displayApplicationConfiguration() { if (getValueBool("display_running_config")) { addLogEntry("--------------- Application Runtime Configuration ---------------"); } // Display application version addLogEntry("Application version = " ~ applicationVersion); addLogEntry("Compiled with = " ~ compilerDetails()); addLogEntry("Curl version = " ~ getCurlVersionString()); // Display all of the pertinent configuration options addLogEntry("User Application Config path = " ~ configDirName); addLogEntry("System Application Config path = " ~ systemConfigDirName); // Does a config file exist or are we using application defaults addLogEntry("Applicable Application 'config' location = " ~ applicableConfigFilePath); string configFileStatusMessage; if (exists(applicableConfigFilePath)) { configFileStatusMessage = "true - using 'config' file values to override application defaults"; } else { configFileStatusMessage = "false - using application defaults"; } addLogEntry("Configuration file found in config location = " ~ configFileStatusMessage); // Display where various files should live // - items.sqlite3 // - sync_list // If using the 'system' directory, (/etc/onedrive) for the config file, these should always live in the 'users' home directory addLogEntry("Applicable 'sync_list' location = " ~ syncListFilePath); addLogEntry("Applicable 'items.sqlite3' location = " ~ databaseFilePath); // Is config option drive_id configured? addLogEntry("Config option 'drive_id' = " ~ getValueString("drive_id")); // Config Options as per 'config' file addLogEntry("Config option 'sync_dir' = " ~ getValueString("sync_dir")); // authentication addLogEntry("Config option 'use_intune_sso' = " ~ to!string(getValueBool("use_intune_sso"))); addLogEntry("Config option 'use_device_auth' = " ~ to!string(getValueBool("use_device_auth"))); // logging and notifications addLogEntry("Config option 'enable_logging' = " ~ to!string(getValueBool("enable_logging"))); addLogEntry("Config option 'log_dir' = " ~ getValueString("log_dir")); addLogEntry("Config option 'disable_notifications' = " ~ to!string(getValueBool("disable_notifications"))); // skip files and directory and 'matching' policy addLogEntry("Config option 'skip_dir' = " ~ getValueString("skip_dir")); addLogEntry("Config option 'skip_dir_strict_match' = " ~ to!string(getValueBool("skip_dir_strict_match"))); addLogEntry("Config option 'skip_file' = " ~ getValueString("skip_file")); addLogEntry("Config option 'skip_dotfiles' = " ~ to!string(getValueBool("skip_dotfiles"))); addLogEntry("Config option 'skip_symlinks' = " ~ to!string(getValueBool("skip_symlinks"))); addLogEntry("Config option 'skip_size' = " ~ to!string(getValueLong("skip_size"))); // --monitor sync process options addLogEntry("Config option 'monitor_interval' = " ~ to!string(getValueLong("monitor_interval"))); addLogEntry("Config option 'monitor_log_frequency' = " ~ to!string(getValueLong("monitor_log_frequency"))); addLogEntry("Config option 'monitor_fullscan_frequency' = " ~ to!string(getValueLong("monitor_fullscan_frequency"))); addLogEntry("Config option 'disable_websocket_support' = " ~ to!string(getValueBool("disable_websocket_support"))); // sync process and method addLogEntry("Config option 'read_only_auth_scope' = " ~ to!string(getValueBool("read_only_auth_scope"))); addLogEntry("Config option 'dry_run' = " ~ to!string(getValueBool("dry_run"))); addLogEntry("Config option 'upload_only' = " ~ to!string(getValueBool("upload_only"))); addLogEntry("Config option 'download_only' = " ~ to!string(getValueBool("download_only"))); addLogEntry("Config option 'local_first' = " ~ to!string(getValueBool("local_first"))); addLogEntry("Config option 'check_nosync' = " ~ to!string(getValueBool("check_nosync"))); addLogEntry("Config option 'check_nomount' = " ~ to!string(getValueBool("check_nomount"))); addLogEntry("Config option 'resync' = " ~ to!string(getValueBool("resync"))); addLogEntry("Config option 'resync_auth' = " ~ to!string(getValueBool("resync_auth"))); addLogEntry("Config option 'cleanup_local_files' = " ~ to!string(getValueBool("cleanup_local_files"))); addLogEntry("Config option 'disable_permission_set' = " ~ to!string(getValueBool("disable_permission_set"))); addLogEntry("Config option 'transfer_order' = " ~ getValueString("transfer_order")); addLogEntry("Config option 'delay_inotify_processing' = " ~ to!string(getValueBool("delay_inotify_processing"))); addLogEntry("Config option 'inotify_delay' = " ~ to!string(getValueLong("inotify_delay"))); addLogEntry("Config option 'display_transfer_metrics' = " ~ to!string(getValueBool("display_transfer_metrics"))); addLogEntry("Config option 'force_session_upload' = " ~ to!string(getValueBool("force_session_upload"))); addLogEntry("Config option 'file_fragment_size' = " ~ to!string(getValueLong("file_fragment_size"))); // data integrity addLogEntry("Config option 'classify_as_big_delete' = " ~ to!string(getValueLong("classify_as_big_delete"))); addLogEntry("Config option 'disable_upload_validation' = " ~ to!string(getValueBool("disable_upload_validation"))); addLogEntry("Config option 'disable_download_validation' = " ~ to!string(getValueBool("disable_download_validation"))); addLogEntry("Config option 'bypass_data_preservation' = " ~ to!string(getValueBool("bypass_data_preservation"))); addLogEntry("Config option 'no_remote_delete' = " ~ to!string(getValueBool("no_remote_delete"))); addLogEntry("Config option 'remove_source_files' = " ~ to!string(getValueBool("remove_source_files"))); addLogEntry("Config option 'sync_dir_permissions' = " ~ to!string(getValueLong("sync_dir_permissions"))); addLogEntry("Config option 'sync_file_permissions' = " ~ to!string(getValueLong("sync_file_permissions"))); addLogEntry("Config option 'space_reservation' = " ~ to!string(getValueLong("space_reservation"))); addLogEntry("Config option 'permanent_delete' = " ~ to!string(getValueBool("permanent_delete"))); addLogEntry("Config option 'write_xattr_data' = " ~ to!string(getValueBool("write_xattr_data"))); addLogEntry("Config option 'create_new_file_version' = " ~ to!string(getValueBool("create_new_file_version"))); // curl operations addLogEntry("Config option 'application_id' = " ~ getValueString("application_id")); addLogEntry("Config option 'azure_ad_endpoint' = " ~ getValueString("azure_ad_endpoint")); addLogEntry("Config option 'azure_tenant_id' = " ~ getValueString("azure_tenant_id")); addLogEntry("Config option 'user_agent' = " ~ getValueString("user_agent")); addLogEntry("Config option 'force_http_11' = " ~ to!string(getValueBool("force_http_11"))); addLogEntry("Config option 'debug_https' = " ~ to!string(getValueBool("debug_https"))); addLogEntry("Config option 'rate_limit' = " ~ to!string(getValueLong("rate_limit"))); addLogEntry("Config option 'operation_timeout' = " ~ to!string(getValueLong("operation_timeout"))); addLogEntry("Config option 'dns_timeout' = " ~ to!string(getValueLong("dns_timeout"))); addLogEntry("Config option 'connect_timeout' = " ~ to!string(getValueLong("connect_timeout"))); addLogEntry("Config option 'data_timeout' = " ~ to!string(getValueLong("data_timeout"))); addLogEntry("Config option 'ip_protocol_version' = " ~ to!string(getValueLong("ip_protocol_version"))); addLogEntry("Config option 'threads' = " ~ to!string(getValueLong("threads"))); addLogEntry("Config option 'max_curl_idle' = " ~ to!string(getValueLong("max_curl_idle"))); // GUI notifications version(Notifications) { addLogEntry("Environment var 'XDG_RUNTIME_DIR' = " ~ to!string(xdg_exists)); addLogEntry("Environment var 'DBUS_SESSION_BUS_ADDRESS' = " ~ to!string(dbus_exists)); addLogEntry("Config option 'notify_file_actions' = " ~ to!string(getValueBool("notify_file_actions"))); } else { addLogEntry("Compile time option --enable-notifications = false"); } // Recycle Bin addLogEntry("Config option 'use_recycle_bin' = " ~ to!string(getValueBool("use_recycle_bin"))); addLogEntry("Config option 'recycle_bin_path' = " ~ getValueString("recycle_bin_path")); // Is sync_list configured and contains entries? if (exists(syncListFilePath) && getSize(syncListFilePath) > 0) { addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("Selective sync 'sync_list' configured = true"); addLogEntry("sync_list config option 'sync_root_files' = " ~ to!string(getValueBool("sync_root_files"))); addLogEntry("sync_list contents:"); // Output the sync_list contents auto syncListFile = File(syncListFilePath, "r"); auto range = syncListFile.byLine(); addLogEntry("------------------------------'sync_list'------------------------------"); foreach (line; range) { addLogEntry(to!string(line)); } addLogEntry("-----------------------------------------------------------------------"); // Close reading the 'sync_list' file syncListFile.close(); } else { // file does not exist or file size is not greater than 0 addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering if (exists(syncListFilePath) && getSize(syncListFilePath) == 0) { // 'sync_list' file exists, no entries addLogEntry("Selective sync 'sync_list' configured = file exists but contains zero data"); } else { // no 'sync_list' file addLogEntry("Selective sync 'sync_list' configured = false"); } } // Is sync_business_shared_items enabled and configured ? addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("Config option 'sync_business_shared_items' = " ~ to!string(getValueBool("sync_business_shared_items"))); if (getValueBool("sync_business_shared_items")) { // display what the shared files directory will be addLogEntry("Config option 'Shared Files Directory' = " ~ configuredBusinessSharedFilesDirectoryName); } // Are webhooks enabled? addLogEntry(); // used instead of an empty 'writeln();' to ensure the line break is correct in the buffered console output ordering addLogEntry("Config option 'webhook_enabled' = " ~ to!string(getValueBool("webhook_enabled"))); if (getValueBool("webhook_enabled")) { addLogEntry("Config option 'webhook_public_url' = " ~ getValueString("webhook_public_url")); addLogEntry("Config option 'webhook_listening_host' = " ~ getValueString("webhook_listening_host")); addLogEntry("Config option 'webhook_listening_port' = " ~ to!string(getValueLong("webhook_listening_port"))); addLogEntry("Config option 'webhook_expiration_interval' = " ~ to!string(getValueLong("webhook_expiration_interval"))); addLogEntry("Config option 'webhook_renewal_interval' = " ~ to!string(getValueLong("webhook_renewal_interval"))); addLogEntry("Config option 'webhook_retry_interval' = " ~ to!string(getValueLong("webhook_retry_interval"))); } if (getValueBool("display_running_config")) { addLogEntry(); addLogEntry("--------------------DEVELOPER_OPTIONS----------------------------"); addLogEntry("Config option 'force_children_scan' = " ~ to!string(getValueBool("force_children_scan"))); addLogEntry("Config option 'monitor_max_loop' = " ~ to!string(getValueLong("monitor_max_loop"))); addLogEntry("Config option 'display_memory' = " ~ to!string(getValueBool("display_memory"))); addLogEntry("Config option 'display_sync_options' = " ~ to!string(getValueBool("display_sync_options"))); addLogEntry("Config option 'display_processing_time' = " ~ to!string(getValueBool("display_processing_time"))); } // Close out config output if (getValueBool("display_running_config")) { addLogEntry("-----------------------------------------------------------------"); addLogEntry(); } } // Prompt the user to accept the risk of using --resync bool displayResyncRiskForAcceptance() { // what is the user risk acceptance? bool userRiskAcceptance = false; // Did the user use --resync-auth or 'resync_auth' in the config file to negate presenting this message? if (!getValueBool("resync_auth")) { // need to prompt user char response; // --resync warning message addLogEntry("", ["consoleOnly"]); // new line, console only addLogEntry("WARNING: You have asked the client to perform a --resync operation.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); addLogEntry(" This operation will delete the client’s local state database and rebuild it entirely from the current online OneDrive state.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); addLogEntry(" Because the previous sync state will no longer be available, the following may occur:", ["consoleOnly"]); addLogEntry(" * Local files that also exist in OneDrive may have local changes overwritten by the cloud version if a conflict cannot be safely resolved.", ["consoleOnly"]); addLogEntry(" * Local files may be renamed or duplicated locally as part of conflict resolution and data-preservation handling.", ["consoleOnly"]); addLogEntry(" * The initial synchronisation pass may involve a large number of file uploads and downloads.", ["consoleOnly"]); addLogEntry(" * The increased activity against the Microsoft Graph API may trigger HTTP 429 (throttling) responses during the synchronisation process.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); addLogEntry(" For safest operation:", ["consoleOnly"]); addLogEntry(" * Ensure you have a current backup of your sync_dir.", ["consoleOnly"]); addLogEntry(" * Run this command first with --dry-run to confirm all planned actions.", ["consoleOnly"]); addLogEntry(" * Enable 'use_recycle_bin' so that online deletion events from OneDrive are moved to your system Trash rather than deleted from your local disk.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); addLogEntry("If in doubt, stop now and back up your local data before continuing.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); addLogEntry("Are you sure you wish to proceed with --resync? [Y/N] ", ["consoleOnlyNoNewLine"]); try { // Attempt to read user response string input = readln().strip; if (input.length > 0) { response = std.ascii.toUpper(input[0]); } } catch (std.format.FormatException e) { userRiskAcceptance = false; // Caught an error return EXIT_FAILURE; } // What did the user enter? if (debugLogging) {addLogEntry("--resync warning User Response Entered: " ~ to!string(response), ["debug"]);} // Evaluate user response if ((to!string(response) == "y") || (to!string(response) == "Y")) { // User has accepted --resync risk to proceed userRiskAcceptance = true; // Are you sure you wish .. does not use writeln(); write("\n"); } } else { // resync_auth is true userRiskAcceptance = true; } // Return the --resync acceptance or not return userRiskAcceptance; } // Prompt the user to accept the risk of using --force-sync bool displayForceSyncRiskForAcceptance() { // what is the user risk acceptance? bool userRiskAcceptance = false; // need to prompt user char response; // --force-sync warning message addLogEntry("", ["consoleOnly"]); // new line, console only addLogEntry("The use of --force-sync will reconfigure the application to use defaults. This may have untold and unknown future impacts.", ["consoleOnly"]); addLogEntry("By proceeding in using this option you accept any impacts including any data loss that may occur as a result of using --force-sync.", ["consoleOnly"]); addLogEntry("", ["consoleOnly"]); // new line, console only addLogEntry("Are you sure you wish to proceed with --force-sync [Y/N] ", ["consoleOnlyNoNewLine"]); try { // Attempt to read user response string input = readln().strip; if (input.length > 0) { response = std.ascii.toUpper(input[0]); } } catch (std.format.FormatException e) { userRiskAcceptance = false; // Caught an error return EXIT_FAILURE; } // What did the user enter? if (debugLogging) {addLogEntry("--force-sync warning User Response Entered: " ~ to!string(response), ["debug"]);} // Evaluate user response if ((to!string(response) == "y") || (to!string(response) == "Y")) { // User has accepted --force-sync risk to proceed userRiskAcceptance = true; // Are you sure you wish .. does not use writeln(); write("\n"); } // Return the --resync acceptance or not return userRiskAcceptance; } // Check the application configuration for any changes that need to trigger a --resync // This function is only called if --resync is not present bool applicationChangeWhereResyncRequired() { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Default is that no resync is required bool resyncRequired = false; // Consolidate the flags for different configuration changes bool[11] configOptionsDifferent; // Handle multiple entries of skip_file string backupConfigFileSkipFile; // Handle multiple entries of skip_dir string backupConfigFileSkipDir; // Create and read the required initial hash files createRequiredInitialConfigurationHashFiles(); // Read in the existing hash file values readExistingConfigurationHashFiles(); // can we read the backup config file bool failedToReadBackupConfig = false; // Helper lambda for logging and setting the difference flag auto logAndSetDifference = (string message, size_t index) { if (debugLogging) {addLogEntry(message, ["debug"]);} configOptionsDifferent[index] = true; }; // Check for changes in the sync_list and business_shared_items files if (currentSyncListHash != previousSyncListHash) logAndSetDifference("sync_list file has been updated, --resync needed", 0); // Check for updates in the config file if (currentConfigHash != previousConfigHash) { addLogEntry("Application configuration file has been updated, checking if --resync needed"); if (debugLogging) {addLogEntry("Using this configBackupFile: " ~ configBackupFile, ["debug"]);} if (exists(configBackupFile)) { string[string] backupConfigStringValues; backupConfigStringValues["check_nosync"] = ""; backupConfigStringValues["drive_id"] = ""; backupConfigStringValues["sync_dir"] = ""; backupConfigStringValues["skip_file"] = ""; backupConfigStringValues["skip_dir"] = ""; backupConfigStringValues["skip_dotfiles"] = ""; backupConfigStringValues["skip_size"] = ""; backupConfigStringValues["skip_symlinks"] = ""; backupConfigStringValues["sync_business_shared_items"] = ""; bool check_nosync_present = false; bool drive_id_present = false; bool sync_dir_present = false; bool skip_file_present = false; bool skip_dir_present = false; bool skip_dotfiles_present = false; bool skip_size_present = false; bool skip_symlinks_present = false; bool sync_business_shared_items_present = false; string configOptionModifiedMessage = " was modified since the last time the application was successfully run, --resync required"; File configBackupFileHandle; try { configBackupFileHandle = File(configBackupFile, "r"); } catch (FileException e) { // filesystem error failedToReadBackupConfig = true; displayFileSystemErrorMessage(e.msg, thisFunctionName, configBackupFile, FsErrorSeverity.warning); } catch (std.exception.ErrnoException e) { // filesystem error failedToReadBackupConfig = true; displayFileSystemErrorMessage(e.msg, thisFunctionName, configBackupFile, FsErrorSeverity.warning); } scope(exit) { if (configBackupFileHandle.isOpen()) { configBackupFileHandle.close(); } } if (!failedToReadBackupConfig) { // backup config file was able to be read string lineBuffer; auto range = configBackupFileHandle.byLine(); foreach (line; range) { lineBuffer = stripLeft(line).to!string; if (lineBuffer.length == 0 || lineBuffer[0] == ';' || lineBuffer[0] == '#') continue; auto c = lineBuffer.matchFirst(configRegex); if (!c.empty) { c.popFront(); // skip the whole match string key = c.front.dup; if (debugLogging) {addLogEntry("Backup Config Key: " ~ key, ["debug"]);} auto p = key in backupConfigStringValues; if (p) { c.popFront(); string value = c.front.dup; // Compare each key value with current config if (key == "drive_id") { drive_id_present = true; if (value != getValueString("drive_id")) { logAndSetDifference(key ~ configOptionModifiedMessage, 2); } } if (key == "sync_dir") { sync_dir_present = true; if (value != getValueString("sync_dir")) { logAndSetDifference(key ~ configOptionModifiedMessage, 3); } } // skip_file handling if (key == "skip_file") { skip_file_present = true; // Merge safely, removing empty entries and de-duplicating backupConfigFileSkipFile = mergePipeDelimitedRulesDedup(backupConfigFileSkipFile, to!string(c.front.dup)); } // skip_dir handling if (key == "skip_dir") { skip_dir_present = true; // Merge safely, removing empty entries and de-duplicating backupConfigFileSkipDir = mergePipeDelimitedRulesDedup(backupConfigFileSkipDir, to!string(c.front.dup)); } if (key == "skip_dotfiles") { skip_dotfiles_present = true; if (value != to!string(getValueBool("skip_dotfiles"))) { logAndSetDifference(key ~ configOptionModifiedMessage, 6); } } if (key == "skip_symlinks") { skip_symlinks_present = true; if (value != to!string(getValueBool("skip_symlinks"))) { logAndSetDifference(key ~ configOptionModifiedMessage, 7); } } if (key == "sync_business_shared_items") { sync_business_shared_items_present = true; if (value != to!string(getValueBool("sync_business_shared_items"))) { logAndSetDifference(key ~ configOptionModifiedMessage, 8); } } } } } // Debug logging if (debugLogging) { addLogEntry("skip_file in actual config = " ~ to!string(configFileSkipFileReadIn), ["debug"]); addLogEntry("skip_file in backup config = " ~ to!string(skip_file_present), ["debug"]); addLogEntry("defaultSkipFile value = " ~ to!string(defaultSkipFile), ["debug"]); addLogEntry("configFileSkipFile value = " ~ to!string(configFileSkipFile), ["debug"]); addLogEntry("backupConfigFileSkipFile value = " ~ to!string(backupConfigFileSkipFile), ["debug"]); } // skip_file can be specified multiple times if (skip_file_present && backupConfigFileSkipFile != configFileSkipFile) logAndSetDifference("skip_file" ~ configOptionModifiedMessage, 4); // skip_file can also be an empty string, thus when removed, as an empty string, we are going back to application defaults if (skip_file_present && backupConfigFileSkipFile != defaultSkipFile) logAndSetDifference("skip_file" ~ configOptionModifiedMessage, 4); // skip_dir can be specified multiple times if (skip_dir_present && backupConfigFileSkipDir != configFileSkipDir) logAndSetDifference("skip_dir" ~ configOptionModifiedMessage, 5); // Check for newly added configuration options to the 'config' file vs being present in the 'backup' config file if (!drive_id_present && configFileDriveId != "") logAndSetDifference("drive_id newly added ... --resync needed", 2); if (!sync_dir_present && configFileSyncDir != defaultSyncDir) logAndSetDifference("sync_dir newly added ... --resync needed", 3); if (configFileSkipFileReadIn) { // We actually read a 'skip_file' configuration line from the 'config' file if (!skip_file_present && configFileSkipFile != defaultSkipFile) logAndSetDifference("skip_file newly added ... --resync needed", 4); } // Other options if (!skip_dir_present && configFileSkipDir != "") logAndSetDifference("skip_dir newly added ... --resync needed", 5); if (!skip_dotfiles_present && configFileSkipDotfiles) logAndSetDifference("skip_dotfiles newly added ... --resync needed", 6); if (!skip_symlinks_present && configFileSkipSymbolicLinks) logAndSetDifference("skip_symlinks newly added ... --resync needed", 7); if (!sync_business_shared_items_present && configFileSyncBusinessSharedItems) logAndSetDifference("sync_business_shared_items newly added ... --resync needed", 8); if (!check_nosync_present && configFileCheckNoSync) logAndSetDifference("check_nosync newly added ... --resync needed", 9); if (!skip_size_present && configFileSkipSize) logAndSetDifference("skip_size newly added ... --resync needed", 10); } else { // failed to read backup config file addLogEntry("WARNING: unable to read backup config, unable to validate if any changes made"); } } else { addLogEntry("WARNING: no backup config file was found, unable to validate if any changes made"); } } // config file set options can be changed via CLI input, specifically these will impact sync and a --resync will be needed: // --syncdir ARG // --skip-file ARG // --skip-dir ARG // --skip-dot-files // --skip-symlinks // --check-for-nosync // --skip-size ARG // Check CLI options if (exists(applicableConfigFilePath)) { if (configFileSyncDir != "" && configFileSyncDir != getValueString("sync_dir")) { // config file was set and CLI input changed this // Is this potentially running as a Docker container? if (entrypointExists) { // entrypoint.sh exists if (debugLogging) {addLogEntry("sync_dir: CLI override of config file option, however entrypoint.sh exists, thus most likely running as a container", ["debug"]);} } else { // Not a Docker container, raise that --resync needed due to configuration change logAndSetDifference("sync_dir: CLI override of config file option, --resync needed", 3); } } if (configFileSkipFile != "" && configFileSkipFile != getValueString("skip_file")) logAndSetDifference("skip_file: CLI override of config file option, --resync needed", 4); if (configFileSkipDir != "" && configFileSkipDir != getValueString("skip_dir")) logAndSetDifference("skip_dir: CLI override of config file option, --resync needed", 5); if (!configFileSkipDotfiles && getValueBool("skip_dotfiles")) logAndSetDifference("skip_dotfiles: CLI override of config file option, --resync needed", 6); if (!configFileSkipSymbolicLinks && getValueBool("skip_symlinks")) logAndSetDifference("skip_symlinks: CLI override of config file option, --resync needed", 7); if (!configFileCheckNoSync && getValueBool("check_nosync")) logAndSetDifference("check_nosync: CLI override of config file option, --resync needed", 9); if (!configFileSkipSize && (getValueLong("skip_size") > 0)) logAndSetDifference("skip_size: CLI override of config file option, --resync needed", 10); } // Aggregate the result to determine if a resync is required if (!failedToReadBackupConfig) { foreach (optionDifferent; configOptionsDifferent) { if (optionDifferent) { resyncRequired = true; break; } } } // Final override // In certain situations, regardless of config 'resync' needed status, ignore this so that the application can display 'non-syncable' information // Options that should now be looked at are: // --list-shared-items if (getValueBool("list_business_shared_items")) resyncRequired = false; // Return the calculated boolean return resyncRequired; } // Cleanup hash files that require to be cleaned up when a --resync is issued void cleanupHashFilesDueToResync() { if (!getValueBool("dry_run")) { // cleanup hash files if (debugLogging) {addLogEntry("Cleaning up configuration hash files", ["debug"]);} safeRemove(configHashFile); safeRemove(syncListHashFile); } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not removing hash files as --dry-run has been used"); } } // For each of the config files, update the hash data in the hash files void updateHashContentsForConfigFiles() { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Are we in a --dry-run scenario? if (!getValueBool("dry_run")) { // Not a dry-run scenario, update the applicable files // Update applicable 'config' files if (exists(applicableConfigFilePath)) { // Update the hash of the applicable config file if (debugLogging) {addLogEntry("Updating applicable config file hash", ["debug"]);} try { std.file.write(configHashFile, computeQuickXorHash(applicableConfigFilePath)); // Hash file should only be readable by the user who created it - 0600 permissions needed configHashFile.setAttributes(convertedPermissionValue); } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, configHashFile, FsErrorSeverity.warning); } } // Update 'sync_list' files if (exists(syncListFilePath)) { // update sync_list hash if (debugLogging) {addLogEntry("Updating sync_list hash", ["debug"]);} try { std.file.write(syncListHashFile, computeQuickXorHash(syncListFilePath)); // Hash file should only be readable by the user who created it - 0600 permissions needed syncListHashFile.setAttributes(convertedPermissionValue); } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, syncListHashFile, FsErrorSeverity.warning); } } } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not updating hash files as --dry-run has been used"); } } // Create any required hash files for files that help us determine if the configuration has changed since last run void createRequiredInitialConfigurationHashFiles() { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Does a 'config' file exist with a valid hash file if (exists(applicableConfigFilePath)) { if (!exists(configHashFile)) { // no existing hash file exists try { std.file.write(configHashFile, "initial-hash"); // Hash file should only be readable by the user who created it - 0600 permissions needed configHashFile.setAttributes(convertedPermissionValue); } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, configHashFile, FsErrorSeverity.warning); } } // Generate the runtime hash for the 'config' file currentConfigHash = computeQuickXorHash(applicableConfigFilePath); } // Does a 'sync_list' file exist with a valid hash file if (exists(syncListFilePath)) { if (!exists(syncListHashFile)) { // no existing hash file exists try { std.file.write(syncListHashFile, "initial-hash"); // Hash file should only be readable by the user who created it - 0600 permissions needed syncListHashFile.setAttributes(convertedPermissionValue); } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, syncListHashFile, FsErrorSeverity.warning); } } // Generate the runtime hash for the 'sync_list' file currentSyncListHash = computeQuickXorHash(syncListFilePath); } } // Read in the text values of the previous configurations int readExistingConfigurationHashFiles() { if (exists(configHashFile)) { try { previousConfigHash = readText(configHashFile); } catch (std.file.FileException e) { // Unable to access required hash file addLogEntry("ERROR: Unable to access " ~ e.msg); // Use exit scopes to shutdown API return EXIT_FAILURE; } } if (exists(syncListHashFile)) { try { previousSyncListHash = readText(syncListHashFile); } catch (std.file.FileException e) { // Unable to access required hash file addLogEntry("ERROR: Unable to access " ~ e.msg); // Use exit scopes to shutdown API return EXIT_FAILURE; } } return 0; } // Check for basic option conflicts - flags that should not be used together and/or flag combinations that conflict with each other bool checkForBasicOptionConflicts() { bool operationalConflictDetected = false; // What are the permission that have been set for the application? // These are relevant for: // - The ~/OneDrive parent folder or 'sync_dir' configured item // - Any new folder created under ~/OneDrive or 'sync_dir' // - Any new file created under ~/OneDrive or 'sync_dir' // valid permissions are 000 -> 777 - anything else is invalid long syncDirPermissions = getValueLong("sync_dir_permissions"); long syncFilePermissions = getValueLong("sync_file_permissions"); bool invalidPermissions = false; // Check 'sync_dir_permissions' if (syncDirPermissions < 0 || syncDirPermissions > 777) { addLogEntry("ERROR: Invalid 'User|Group|Other' permissions set for 'sync_dir_permissions' within your config file. Please check your configuration"); invalidPermissions = true; } // Check 'sync_file_permissions' if (syncFilePermissions < 0 || syncFilePermissions > 777) { addLogEntry("ERROR: Invalid 'User|Group|Other' permissions set for 'sync_file_permissions' within your config file. Please check your configuration"); invalidPermissions = true; } // Invalid permissions detected? if (invalidPermissions) { operationalConflictDetected = true; } else { // Debug log output what permissions are being set to if (debugLogging) {addLogEntry("Configuring default new folder permissions as: " ~ to!string(getValueLong("sync_dir_permissions")), ["debug"]);} configureRequiredDirectoryPermissions(); if (debugLogging) {addLogEntry("Configuring default new file permissions as: " ~ to!string(getValueLong("sync_file_permissions")), ["debug"]);} configureRequiredFilePermissions(); } // --upload-only and --download-only cannot be used together if ((getValueBool("upload_only")) && (getValueBool("download_only"))) { addLogEntry("ERROR: --upload-only and --download-only cannot be used together. Use one, not both at the same time"); operationalConflictDetected = true; } // --sync and --monitor cannot be used together if ((getValueBool("synchronize")) && (getValueBool("monitor"))) { addLogEntry("ERROR: --sync and --monitor cannot be used together. Only use one of these options, not both at the same time"); operationalConflictDetected = true; } // --no-remote-delete can ONLY be enabled when --upload-only is used if ((getValueBool("no_remote_delete")) && (!getValueBool("upload_only"))) { addLogEntry("ERROR: --no-remote-delete can only be used with --upload-only"); operationalConflictDetected = true; } // --remove-source-files can ONLY be enabled when --upload-only is used if ((getValueBool("remove_source_files")) && (!getValueBool("upload_only"))) { addLogEntry("ERROR: --remove-source-files can only be used with --upload-only"); operationalConflictDetected = true; } // --cleanup-local-files can ONLY be enabled when --download-only is used if ((getValueBool("cleanup_local_files")) && (!getValueBool("download_only"))) { addLogEntry("ERROR: --cleanup-local-files can only be used with --download-only"); operationalConflictDetected = true; } // --list-shared-folders cannot be used with --resync and/or --resync-auth if ((getValueBool("list_business_shared_items")) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --list-shared-items cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --list-shared-folders cannot be used with --sync or --monitor if ((getValueBool("list_business_shared_items")) && ((getValueBool("synchronize")) || (getValueBool("monitor")))) { addLogEntry("ERROR: --list-shared-items cannot be used with --sync or --monitor"); operationalConflictDetected = true; } // --sync-shared-files can ONLY be used with sync_business_shared_items if ((getValueBool("sync_business_shared_files")) && (!getValueBool("sync_business_shared_items"))) { addLogEntry("ERROR: The --sync-shared-files option can only be utilised if the 'sync_business_shared_items' configuration setting is enabled."); operationalConflictDetected = true; } // --display-sync-status cannot be used with --resync and/or --resync-auth if ((getValueBool("display_sync_status")) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --display-sync-status cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --modified-by cannot be used with --resync and/or --resync-auth if ((!getValueString("modified_by").empty) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --modified-by cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --get-file-link cannot be used with --resync and/or --resync-auth if ((!getValueString("get_file_link").empty) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --get-file-link cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --create-share-link cannot be used with --resync and/or --resync-auth if ((!getValueString("create_share_link").empty) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --create-share-link cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --get-sharepoint-drive-id cannot be used with --resync and/or --resync-auth if ((!getValueString("sharepoint_library_name").empty) && ((getValueBool("resync")) || (getValueBool("resync_auth")))) { addLogEntry("ERROR: --get-sharepoint-drive-id cannot be used with --resync or --resync-auth"); operationalConflictDetected = true; } // --monitor and --display-sync-status cannot be used together if ((getValueBool("monitor")) && (getValueBool("display_sync_status"))) { addLogEntry("ERROR: --monitor and --display-sync-status cannot be used together"); operationalConflictDetected = true; } // --sync and --display-sync-status cannot be used together if ((getValueBool("synchronize")) && (getValueBool("display_sync_status"))) { addLogEntry("ERROR: --sync and --display-sync-status cannot be used together"); operationalConflictDetected = true; } // --monitor and --display-quota cannot be used together if ((getValueBool("monitor")) && (getValueBool("display_quota"))) { addLogEntry("ERROR: --monitor and --display-quota cannot be used together"); operationalConflictDetected = true; } // --sync and --display-quota cannot be used together if ((getValueBool("synchronize")) && (getValueBool("display_quota"))) { addLogEntry("ERROR: --sync and --display-quota cannot be used together"); operationalConflictDetected = true; } // --force-sync can only be used when using --sync and --single-directory if (getValueBool("force_sync")) { bool conflict = false; // Should not be used with --monitor if (getValueBool("monitor")) conflict = true; // single_directory must not be empty if (getValueString("single_directory").empty) conflict = true; if (conflict) { addLogEntry("ERROR: --force-sync can only be used with --sync --single-directory"); operationalConflictDetected = true; } } // When using 'azure_ad_endpoint', 'azure_tenant_id' cannot be empty if ((!getValueString("azure_ad_endpoint").empty) && (getValueString("azure_tenant_id").empty)) { addLogEntry("ERROR: config option 'azure_tenant_id' cannot be empty when 'azure_ad_endpoint' is configured"); operationalConflictDetected = true; } // When using --enable-logging the 'log_dir' cannot be empty if ((getValueBool("enable_logging")) && (getValueString("log_dir").empty)) { addLogEntry("ERROR: config option 'log_dir' cannot be empty when 'enable_logging' is configured"); operationalConflictDetected = true; } // When using --syncdir, the value cannot be empty. if (strip(getValueString("sync_dir")).empty) { addLogEntry("ERROR: --syncdir value cannot be empty"); operationalConflictDetected = true; } // --monitor and --create-directory cannot be used together if ((getValueBool("monitor")) && (!getValueString("create_directory").empty)) { addLogEntry("ERROR: --monitor and --create-directory cannot be used together"); operationalConflictDetected = true; } // --sync and --create-directory cannot be used together if ((getValueBool("synchronize")) && (!getValueString("create_directory").empty)) { addLogEntry("ERROR: --sync and --create-directory cannot be used together"); operationalConflictDetected = true; } // --monitor and --remove-directory cannot be used together if ((getValueBool("monitor")) && (!getValueString("remove_directory").empty)) { addLogEntry("ERROR: --monitor and --remove-directory cannot be used together"); operationalConflictDetected = true; } // --sync and --remove-directory cannot be used together if ((getValueBool("synchronize")) && (!getValueString("remove_directory").empty)) { addLogEntry("ERROR: --sync and --remove-directory cannot be used together"); operationalConflictDetected = true; } // --monitor and --source-directory cannot be used together if ((getValueBool("monitor")) && (!getValueString("source_directory").empty)) { addLogEntry("ERROR: --monitor and --source-directory cannot be used together"); operationalConflictDetected = true; } // --sync and --source-directory cannot be used together if ((getValueBool("synchronize")) && (!getValueString("source_directory").empty)) { addLogEntry("ERROR: --sync and --source-directory cannot be used together"); operationalConflictDetected = true; } // --monitor and --destination-directory cannot be used together if ((getValueBool("monitor")) && (!getValueString("destination_directory").empty)) { addLogEntry("ERROR: --monitor and --destination-directory cannot be used together"); operationalConflictDetected = true; } // --sync and --destination-directory cannot be used together if ((getValueBool("synchronize")) && (!getValueString("destination_directory").empty)) { addLogEntry("ERROR: --sync and --destination-directory cannot be used together"); operationalConflictDetected = true; } // --download-only and --local-first cannot be used together if ((getValueBool("download_only")) && (getValueBool("local_first"))) { addLogEntry("ERROR: --download-only cannot be used with --local-first"); operationalConflictDetected = true; } // Test that '--modified-by ' has a valid argument and not another directive if (getValueString("modified_by") != "") { // Does the string start with '--' ? if (getValueString("modified_by").startsWith("--")) { addLogEntry("ERROR: --modified-by missing a valid entry"); operationalConflictDetected = true; } } // Test that '--get-file-link ' has a valid argument and not another directive if (getValueString("get_file_link") != "") { // Does the string start with '--' ? if (getValueString("get_file_link").startsWith("--")) { addLogEntry("ERROR: --get-file-link missing a valid entry"); operationalConflictDetected = true; } } // Test that '--create-share-link ' has a valid argument and not another directive if (getValueString("create_share_link") != "") { // Does the string start with '--' ? if (getValueString("create_share_link").startsWith("--")) { addLogEntry("ERROR: --create-share-link missing a valid entry"); operationalConflictDetected = true; } } // Test that '--create-directory ' has a valid argument and not another directive if (getValueString("create_directory") != "") { // Does the string start with '--' ? if (getValueString("create_directory").startsWith("--")) { addLogEntry("ERROR: --create-directory missing a valid entry"); operationalConflictDetected = true; } } // Test that '--remove-directory ' has a valid argument and not another directive if (getValueString("remove_directory") != "") { // Does the string start with '--' ? if (getValueString("remove_directory").startsWith("--")) { addLogEntry("ERROR: --remove-directory missing a valid entry"); operationalConflictDetected = true; } } // Test that '--source-directory ' has a valid argument and not another directive if (getValueString("source_directory") != "") { // Does the string start with '--' ? if (getValueString("source_directory").startsWith("--")) { addLogEntry("ERROR: --source-directory missing a valid entry"); operationalConflictDetected = true; } } // Test that '--destination-directory ' has a valid argument and not another directive if (getValueString("destination_directory") != "") { // Does the string start with '--' ? if (getValueString("destination_directory").startsWith("--")) { addLogEntry("ERROR: --destination-directory missing a valid entry"); operationalConflictDetected = true; } } // 'use_intune_sso' and 'use_device_auth' cannot be used together if ((getValueBool("use_intune_sso")) && (getValueBool("use_device_auth"))) { addLogEntry("ERROR: 'use_intune_sso' and 'use_device_auth' cannot be used together"); operationalConflictDetected = true; } // --force and --resync cannot be used together as --resync blows away the database, thus there is no way to calculate large local deletes if ((getValueBool("force")) && (getValueBool("resync"))) { addLogEntry("ERROR: --force and --resync cannot be used together as there is zero way to determine that a big delete has occurred"); operationalConflictDetected = true; } // Return bool value indicating if we have an operational conflict return operationalConflictDetected; } // Reset skip_file and skip_dir to application defaults when --force-sync is used void resetSkipToDefaults() { // skip_file if (debugLogging) { addLogEntry("original skip_file: " ~ getValueString("skip_file"), ["debug"]); addLogEntry("resetting skip_file to application defaults", ["debug"]); } setValueString("skip_file", defaultSkipFile); if (debugLogging) {addLogEntry("reset skip_file: " ~ getValueString("skip_file"), ["debug"]);} // skip_dir if (debugLogging) { addLogEntry("original skip_dir: " ~ getValueString("skip_dir"), ["debug"]); addLogEntry("resetting skip_dir to application defaults", ["debug"]); } setValueString("skip_dir", defaultSkipDir); if (debugLogging) {addLogEntry("reset skip_dir: " ~ getValueString("skip_dir"), ["debug"]);} } // Initialise the correct 'sync_dir' expanding any '~' if present string initialiseRuntimeSyncDirectory() { // Log what we are doing if (debugLogging) {addLogEntry("sync_dir: Setting runtimeSyncDirectory from config value 'sync_dir'", ["debug"]);} if (!shellEnvironmentSet){ if (debugLogging) {addLogEntry("sync_dir: No SHELL or USER environment variable configuration detected", ["debug"]);} // No shell or user set, so expandTilde() will fail - usually headless system running under init.d / systemd or potentially Docker // Does the 'currently configured' sync_dir include a ~ if (canFind(getValueString("sync_dir"), "~")) { // A ~ was found in sync_dir if (debugLogging) {addLogEntry("sync_dir: A '~' was found in 'sync_dir', using the calculated 'homePath' to replace '~' as no SHELL or USER environment variable set", ["debug"]);} runtimeSyncDirectory = buildNormalizedPath(buildPath(defaultHomePath, strip(getValueString("sync_dir"), "~"))); } else { // No ~ found in sync_dir, use as is if (debugLogging) {addLogEntry("sync_dir: Using configured 'sync_dir' path as-is as no SHELL or USER environment variable configuration detected", ["debug"]);} runtimeSyncDirectory = getValueString("sync_dir"); } } else { // A shell and user environment variable is set, expand any ~ as this will be expanded correctly if present if (canFind(getValueString("sync_dir"), "~")) { if (debugLogging) {addLogEntry("sync_dir: A '~' was found in the configured 'sync_dir', automatically expanding as SHELL and USER environment variable is set", ["debug"]);} runtimeSyncDirectory = expandTilde(getValueString("sync_dir")); } else { // No ~ found in sync_dir, does the path begin with a '/' ? if (debugLogging) {addLogEntry("sync_dir: Using configured 'sync_dir' path as-is as however SHELL or USER environment variable configuration detected - should be placed in USER home directory", ["debug"]);} if (!startsWith(getValueString("sync_dir"), "/")) { if (debugLogging) {addLogEntry("Configured 'sync_dir' does not start with a '/' or '~/' - adjusting configured 'sync_dir' to use User Home Directory as base for 'sync_dir' path", ["debug"]);} string updatedPathWithHome = "~/" ~ getValueString("sync_dir"); runtimeSyncDirectory = expandTilde(updatedPathWithHome); } else { if (debugLogging) {addLogEntry("use 'sync_dir' as is - no touch", ["debug"]);} runtimeSyncDirectory = getValueString("sync_dir"); } } } // What will runtimeSyncDirectory be actually set to? if (debugLogging) {addLogEntry("sync_dir: runtimeSyncDirectory set to: " ~ runtimeSyncDirectory, ["debug"]);} // Configure configuredBusinessSharedFilesDirectoryName configuredBusinessSharedFilesDirectoryName = buildNormalizedPath(buildPath(runtimeSyncDirectory, defaultBusinessSharedFilesDirectoryName)); return runtimeSyncDirectory; } // Initialise the correct 'log_dir' when application logging to a separate file is enabled with 'enable_logging' and expanding any '~' if present string calculateLogDirectory() { string configuredLogDirPath; if (debugLogging) {addLogEntry("log_dir: Setting runtime application log from config value 'log_dir'", ["debug"]);} if (getValueString("log_dir") != defaultLogFileDir) { // User modified 'log_dir' to be used with 'enable_logging' // if 'log_dir' contains a '~' this needs to be expanded correctly if (canFind(getValueString("log_dir"), "~")) { // ~ needs to be expanded correctly if (!shellEnvironmentSet) { // No shell or user environment variable set, so expandTilde() will fail - usually headless system running under init.d / systemd or potentially Docker if (debugLogging) {addLogEntry("log_dir: A '~' was found in 'log_dir' however '~' as no SHELL or USER environment variable set", ["debug"]);} configuredLogDirPath = buildNormalizedPath(buildPath(defaultHomePath, strip(getValueString("log_dir"), "~"))); } else { // We have a SHELL or USER environment variable set, so expandTilde() should work correctly if (debugLogging) {addLogEntry("log_dir: A '~' was found in the configured 'log_dir', automatically expanding as SHELL and USER environment variable is set", ["debug"]);} configuredLogDirPath = buildNormalizedPath(expandTilde(getValueString("log_dir"))); } } else { // No '~' present - use as-is, but normalise configuredLogDirPath = buildNormalizedPath(getValueString("log_dir")); } } else { // Default 'log_dir' to be used with 'enable_logging' configuredLogDirPath = defaultLogFileDir; } // Attempt to create 'configuredLogDirPath' if this does not exist, otherwise we need to fall back to the users home directory if (!exists(configuredLogDirPath)) { // 'configuredLogDirPath' path does not exist - try and create it try { mkdirRecurse(configuredLogDirPath); } catch (std.file.FileException e) { // We got an error when attempting to create the directory .. addLogEntry(); addLogEntry("ERROR: Unable to create " ~ configuredLogDirPath); addLogEntry("ERROR: Please manually create '" ~ configuredLogDirPath ~ "' and ensure that the permissions allow write access for your user to this location."); addLogEntry("ERROR: The requested client activity log will instead be located in your users home directory"); addLogEntry(); // Reconfigure 'configuredLogDirPath' to use environment.get("HOME") value, which we have already calculated configuredLogDirPath = defaultHomePath; } } // Verify that we can actually write in configuredLogDirPath // If we cannot, fall back to the user's home directory instead of later crashing try { auto testFile = buildNormalizedPath(buildPath(configuredLogDirPath, ".onedrive_log_write_test")); // Try to append a zero-length string – this will create the file if possible std.file.append(testFile, ""); // Clean up the test file std.file.remove(testFile); } catch (std.file.FileException e) { addLogEntry(); addLogEntry("ERROR: Unable to write to " ~ configuredLogDirPath); addLogEntry("ERROR: Please manually adjust permissions or choose a different 'log_dir' in the configuration file."); addLogEntry("ERROR: The requested client activity log will instead be located in your users home directory"); addLogEntry(); // Reconfigure 'configuredLogDirPath' to use environment.get("HOME") value, which we have already calculated configuredLogDirPath = defaultHomePath; } // Return the initialised application log path return configuredLogDirPath; } // What IP protocol is going to be used to access Microsoft OneDrive void displayIPProtocol() { if (getValueLong("ip_protocol_version") == 0) addLogEntry("Using IPv4 and IPv6 (if configured) for all network operations"); if (getValueLong("ip_protocol_version") == 1) addLogEntry("Forcing client to use IPv4 connections only"); if (getValueLong("ip_protocol_version") == 2) addLogEntry("Forcing client to use IPv6 connections only"); } // Has a 'no-sync' task been requested? bool hasNoSyncOperationBeenRequested() { // Are we performing some sort of 'no-sync' task? // - Are we performing a logout? // - Are we performing a reauth? // - Are we obtaining the Office 365 Drive ID for a given Office 365 SharePoint Shared Library? // - Are we displaying the sync status? // - Are we getting the URL for a file online? // - Are we listing who modified a file last online? // - Are we listing OneDrive Business Shared Items? // - Are we creating a shareable link for an existing file on OneDrive? // - Are we just creating a directory online, without any sync being performed? // - Are we just deleting a directory online, without any sync being performed? // - Are we renaming or moving a directory? // - Are we displaying the quota information? // - Are we downloading a single file? bool noSyncOperation = false; // Return a true|false if any of these have been set, so that we use the 'dry-run' DB copy, to execute these tasks, in case the client is currently operational // --logout if (getValueBool("logout")) { // flag that a no sync operation has been requested noSyncOperation = true; } // --reauth if (getValueBool("reauth")) { // flag that a no sync operation has been requested noSyncOperation = true; } // --get-sharepoint-drive-id - Get the SharePoint Library drive_id if (getValueString("sharepoint_library_name") != "") { // flag that a no sync operation has been requested noSyncOperation = true; } // --display-sync-status - Query the sync status if (getValueBool("display_sync_status")) { // flag that a no sync operation has been requested noSyncOperation = true; } // --get-file-link - Get the URL path for a synced file? if (getValueString("get_file_link") != "") { // flag that a no sync operation has been requested noSyncOperation = true; } // --modified-by - Are we listing the modified-by details of a provided path? if (getValueString("modified_by") != "") { // flag that a no sync operation has been requested noSyncOperation = true; } // --list-shared-items - Are we listing OneDrive Business Shared Items if (getValueBool("list_business_shared_items")) { // flag that a no sync operation has been requested noSyncOperation = true; } // --create-share-link - Are we creating a shareable link for an existing file on OneDrive? if (getValueString("create_share_link") != "") { // flag that a no sync operation has been requested noSyncOperation = true; } // --create-directory - Are we just creating a directory online, without any sync being performed? if ((getValueString("create_directory") != "")) { // flag that a no sync operation has been requested noSyncOperation = true; } // --remove-directory - Are we just deleting a directory online, without any sync being performed? if ((getValueString("remove_directory") != "")) { // flag that a no sync operation has been requested noSyncOperation = true; } // Are we renaming or moving a directory online? // onedrive --source-directory 'path/as/source/' --destination-directory 'path/as/destination' if ((getValueString("source_directory") != "") && (getValueString("destination_directory") != "")) { // flag that a no sync operation has been requested noSyncOperation = true; } // Are we displaying the quota information? if (getValueBool("display_quota")) { // flag that a no sync operation has been requested noSyncOperation = true; } // Are we downloading a single file? if ((getValueString("download_single_file") != "")) { // flag that a no sync operation has been requested noSyncOperation = true; } // Return result return noSyncOperation; } // Are the required GUI logging environment variables for this user available? // Specifically these must be available: // - XDG_RUNTIME_DIR // - DBUS_SESSION_BUS_ADDRESS bool validateGUINotificationEnvironmentVariables() { bool variablesAvailable = false; string xdg_value; string dbus_value; version(Notifications) { // Check XDG_RUNTIME_DIR environment variable try { xdg_value = environment["XDG_RUNTIME_DIR"]; xdg_exists = true; } catch (Exception e) { xdg_exists = false; } // Check DBUS_SESSION_BUS_ADDRESS environment variable try { dbus_value = environment["DBUS_SESSION_BUS_ADDRESS"]; dbus_exists = true; } catch (Exception e) { dbus_exists = false; } // Output the result if (xdg_exists) { if (debugLogging) {addLogEntry("runtime_environment: XDG_RUNTIME_DIR exists with value: " ~ xdg_value , ["debug"]);} } else { if (debugLogging) {addLogEntry("runtime_environment: XDG_RUNTIME_DIR missing from runtime user environment", ["debug"]);} } if (dbus_exists) { if (debugLogging) {addLogEntry("runtime_environment: DBUS_SESSION_BUS_ADDRESS exists with value: " ~ dbus_value, ["debug"]);} } else { if (debugLogging) {addLogEntry("runtime_environment: DBUS_SESSION_BUS_ADDRESS missing from runtime user environment", ["debug"]);} } // Determine result if (xdg_exists && dbus_exists) { variablesAvailable = true; } else { addLogEntry("WARNING: Required environment variables required to enable GUI Notifications are not present"); variablesAvailable = false; } } // Return result return variablesAvailable; } // Set the Recycle Bin Paths void setRecycleBinPaths() { string configured = getValueString("recycle_bin_path"); string basePath; string dirSeparatorString = "/"; // Handle the "no shell / no user" case similarly to sync_dir if (!shellEnvironmentSet) { // No SHELL or USER means expandTilde() will fail if '~' is present if (canFind(configured, "~")) { // Replace '~' with defaultHomePath explicitly basePath = buildNormalizedPath( buildPath(defaultHomePath, strip(configured, "~")) ); } else { basePath = configured; } } else { // Normal case: shell + user are set; we can rely on expandTilde() if (canFind(configured, "~")) { basePath = expandTilde(configured); } else { basePath = configured; } } // Make sure it's normalised and has a trailing '/' basePath = buildNormalizedPath(basePath); if (!basePath.endsWith(dirSeparatorString)) { basePath ~= dirSeparatorString; } // Update Recycle Bin paths recycleBinParentPath = basePath; recycleBinFilePath = basePath ~ "files" ~ dirSeparatorString; recycleBinInfoPath = basePath ~ "info" ~ dirSeparatorString; } // Is 'recycleBinParentPath' a child path of the configured 'runtimeSyncDirectory'? bool checkRecycleBinPathAsChildOfSyncDir() { // Configure the variables to check string syncRoot = runtimeSyncDirectory; string recycleBin = recycleBinParentPath; string sep = "/"; // Make prefix check robust – ensure syncRoot ends with separator if (!syncRoot.endsWith(sep)) { syncRoot ~= sep; } // Make prefix check robust – ensure recycleBin ends with separator if (!recycleBin.endsWith(sep)) { recycleBin ~= sep; } // Perform the check and return the evaluation return startsWith(recycleBin, syncRoot); } // Is the client running under a GUI session? // - GNOME // - KDE bool isGuiSessionDetected() { bool hasDisplay = false; bool hasRuntime = false; bool uidMatches = false; bool homeOK = false; string xdgType; try { xdgType = environment.get("XDG_SESSION_TYPE", ""); } catch (Exception e) { xdgType = ""; } try { hasDisplay = environment.get("WAYLAND_DISPLAY", "").length > 0 || environment.get("DISPLAY", "").length > 0; } catch (Exception e) { hasDisplay = false; } try { hasRuntime = environment.get("XDG_RUNTIME_DIR", "").length > 0; } catch (Exception e) { hasRuntime = false; } try { uidMatches = (geteuid() == getuid()); } catch (Exception e) { uidMatches = false; } try { homeOK = environment.get("HOME", "").length > 0; } catch (Exception e) { homeOK = false; } bool hasGuiElements = hasDisplay || (xdgType == "wayland" || xdgType == "x11"); return hasGuiElements && hasRuntime && uidMatches && homeOK; } // Attempt to detect the running display manager DesktopHints detectDesktop() { string all = ( environment.get("XDG_CURRENT_DESKTOP","") ~ ":" ~ environment.get("XDG_SESSION_DESKTOP","") ~ ":" ~ environment.get("DESKTOP_SESSION","") ~ ":" ~ environment.get("GDMSESSION","") ~ ":" ~ environment.get("KDE_FULL_SESSION","")).toLower(); DesktopHints hints; hints.gnome = all.canFind("gnome"); hints.kde = all.canFind("kde") || all.canFind("plasma"); return hints; } // Generate the correct file:// URI for display manager integration string fileUriFor(string absPath) { // Basic, safe URI for local file return "file://" ~ expandTilde(absPath); } // Add GNOME Bookmark void addGnomeBookmark() { // Configure required variables string uri = fileUriFor(getValueString("sync_dir")); string bookmarksPath = buildPath(expandTilde(environment.get("HOME", "")), ".config", "gtk-3.0", "bookmarks"); // Ensure the bookmarks path exists try { // Attempt bookmarks path creation mkdirRecurse(dirName(bookmarksPath)); } catch (std.file.FileException e) { // Creating the bookmarks path failed addLogEntry("ERROR: Unable to create the GNOME bookmark directory: " ~ e.msg, ["info", "notify"]); return; } // Does the bookmark already exist? string content = exists(bookmarksPath) ? readText(bookmarksPath) : ""; bool present = false; foreach (line; content.splitLines()) { if (line.strip == uri) { present = true; break; } } if (present) return; // Append newline if needed, then our URI string newline = content.length && !content.endsWith("\n") ? "\n" : ""; string updated = content ~ newline ~ uri ~ "\n"; // Atomic write string tmp = bookmarksPath ~ ".tmp"; std.file.write(tmp, updated); rename(tmp, bookmarksPath); // Log outcome addLogEntry("GNOME Desktop Integration: Bookmark added successfully", ["info"]); } // Set the correct folder icon for the 'sync_dir' path void setOneDriveFolderIcon() { // Get the sync directory string syncDir = expandTilde(getValueString("sync_dir")); // Build gio command string[] gioCmd = [ "gio", "set", syncDir, "metadata::custom-icon-name", "onedrive" ]; // Try and set folder icon try { auto p = spawnProcess(gioCmd); int status = p.wait(); if (status == 0) { addLogEntry("GNOME Desktop Integration: Set folder icon to 'onedrive' for " ~ syncDir, ["info"]); } else { addLogEntry("GNOME Desktop Integration: Failed to set folder icon for " ~ syncDir ~ " (gio exit " ~ status.to!string ~ ")", ["info"]); } } catch (Exception e) { addLogEntry("GNOME Desktop Integration: Exception setting folder icon: " ~ e.msg, ["info"]); } } // Remove GNOME Bookmark void removeGnomeBookmark() { // Configure required variables string uri = fileUriFor(getValueString("sync_dir")); string bookmarksPath = buildPath(expandTilde(environment.get("HOME", "")), ".config", "gtk-3.0", "bookmarks"); // Does the bookmark path exist? if (!exists(bookmarksPath)) { return; } // Read existing bookmarks string content = readText(bookmarksPath); auto lines = content.splitLines(); bool changed = false; string[] kept; kept.reserve(lines.length); foreach (line; lines) { // Remove every line that exactly matches the URI (after stripping whitespace) if (line.strip == uri) { changed = true; continue; } kept ~= line; } if (!changed) { return; } // Rebuild file (ensure trailing newline if non-empty) string updated = kept.length ? kept.join("\n") ~ "\n" : ""; // Atomic write const string tmp = bookmarksPath ~ ".tmp"; std.file.write(tmp, updated); rename(tmp, bookmarksPath); // Log outcome addLogEntry("GNOME Desktop Integration: Bookmark removed successfully", ["info"]); } // Remove folder icon void removeOneDriveFolderIcon() { // Get the sync directory string syncDir = expandTilde(getValueString("sync_dir")); // Build gio command string[] gioCmd = [ "gio", "set", syncDir, "metadata::custom-icon-name", "folder" ]; // Try and set folder icon try { auto p = spawnProcess(gioCmd); int status = p.wait(); if (status == 0) { addLogEntry("GNOME Desktop Integration: Reset folder icon to 'default' for " ~ syncDir, ["info"]); } else { addLogEntry("GNOME Desktop Integration: Failed to reset folder icon for " ~ syncDir ~ " (gio exit " ~ status.to!string ~ ")", ["info"]); } } catch (Exception e) { addLogEntry("GNOME Desktop Integration: Exception setting folder icon: " ~ e.msg, ["info"]); } } // Add KDE Places entry void addKDEPlacesEntry() { // Configure required variables string uri = fileUriFor(getValueString("sync_dir")); string xbelPath = buildPath(expandTilde(environment.get("HOME", "")), ".local", "share", "user-places.xbel"); string content; // Ensure the xbelPath path exists try { // Attempt xbelPath creation mkdirRecurse(dirName(xbelPath)); } catch (std.file.FileException e) { // Creating the xbelPath path failed addLogEntry("ERROR: Unable to create the KDE Places directory: " ~ e.msg, ["info", "notify"]); return; } // Does the xbel file exist? if (exists(xbelPath)) { // Path exists - read the file content = readText(xbelPath); // Does the 'sync_dir' path exist in the xbel file? if (content.canFind(`href="` ~ uri ~ `"`)) { return; // already present } } else { // xbel path does not exist, create minimal XBEL skeleton content = " "; } // Insert xbel bookmark before closing tag string bookmark = ` OneDrive `; // Update xbel file with Microsoft OneDrive Bookmark string updated; auto idx = content.lastIndexOf(""); if (idx >= 0) { updated = content[0 .. idx] ~ bookmark ~ "\n" ~ content[idx .. $]; } else { // Fallback: append (still valid for many parsers) updated = content ~ "\n" ~ bookmark ~ "\n\n"; } string tmp = xbelPath ~ ".tmp"; std.file.write(tmp, updated); rename(tmp, xbelPath); // Log outcome addLogEntry("KDE Desktop Integration: KDE/Plasma place added successfully", ["info"]); } // Remove KDE Places entry void removeKDEPlacesEntry() { // Compute paths/values const string uri = fileUriFor(getValueString("sync_dir")); const string xbelPath = buildPath(expandTilde(environment.get("HOME", "")), ".local", "share", "user-places.xbel"); if (!exists(xbelPath)) { return; } string content = readText(xbelPath); auto before = content; // Build a regex that matches: // ... // - tolerate attribute order/whitespace // - accept single or double quotes around URI // - non-greedy body match const esc = regexEscape(uri); auto re = regex(`(?s)]*\bhref\s*=\s*["']` ~ esc ~ `["'][^>]*>.*?`); // Remove all matches content = replaceAll(content, re, ""); // Optional: tidy up multiple blank lines left behind auto cleanup = regex(`\n{3,}`); content = replaceAll(content, cleanup, "\n\n"); // If nothing changed, exit quietly if (content == before) { return; } // Atomic write string tmp = xbelPath ~ ".tmp"; std.file.write(tmp, content); rename(tmp, xbelPath); addLogEntry("KDE Desktop Integration: KDE/Plasma place removed successfully", ["info"]); } // Safely merge multiple '|'-delimited rule strings by normalising, removing empty entries, trimming whitespace, // and de-duplicating rules so malformed or repeated config entries do not corrupt 'skip_dir' / 'skip_file' processing private string mergePipeDelimitedRulesDedup(const string existing, const string incoming) { auto resultString = appender!(string[])(); void addTokens(const string raw) { foreach (part; raw.splitter('|')) { auto t = part.strip(); if (t.empty) continue; // Keep first occurrence if (!resultString.data.canFind(t)) resultString ~= t.idup; } } addTokens(existing); addTokens(incoming); return resultString.data.joiner("|").to!string; } } // Output the full application help when --help is passed in void outputLongHelp(Option[] opt) { auto argsNeedingOptions = [ "--auth-files", "--auth-response", "--confdir", "--create-directory", "--classify-as-big-delete", "--create-share-link", "--destination-directory", "--download-file", "--get-file-link", "--get-O365-drive-id", "--get-sharepoint-drive-id", "--log-dir", "--min-notify-changes", "--modified-by", "--monitor-interval", "--monitor-log-frequency", "--monitor-fullscan-frequency", "--remove-directory", "--single-directory", "--skip-dir", "--skip-file", "--skip-size", "--source-directory", "--space-reservation", "--syncdir", "--share-password", "--user-agent" ]; writeln(`onedrive - A client for the Microsoft OneDrive Cloud Service Usage: onedrive [options] --sync Do a one-time synchronisation with Microsoft OneDrive onedrive [options] --monitor Monitor filesystem and synchronise regularly with Microsoft OneDrive onedrive [options] --display-config Display the currently used configuration onedrive [options] --display-sync-status Query OneDrive service and report on pending changes onedrive -h | --help Show this help screen onedrive --version Show version Options: `); foreach (it; opt.sort!("a.optLong < b.optLong")) { writefln(" %s%s%s%s\n %s", it.optLong, it.optShort == "" ? "" : " " ~ it.optShort, argsNeedingOptions.canFind(it.optLong) ? " ''" : "", it.required ? " (required)" : "", it.help); } // end with a blank line writeln(); } ================================================ FILE: src/curlEngine.d ================================================ // What is this module called? module curlEngine; // What does this module require to function? import std.net.curl; import etc.c.curl; import std.datetime; import std.conv; import std.file; import std.format; import std.json; import std.stdio; import std.range; import core.memory; import core.sys.posix.signal; // Required for WebSocket Support import core.stdc.stdlib : getenv; import core.stdc.string : strcmp; import core.sys.posix.dlfcn : dlopen, dlsym, dlclose, RTLD_NOW; // Posix elements import std.exception : enforce; // for enforce(...) // What other modules that we have created do we need to import? import log; import util; // WebSocket check elements enum CURL_WS_MIN_NUM = 0x075600; // 7.86.0 (version which WebSocket support was added to cURL) extern (C) void sigpipeHandler(int signum) { // Custom handler to ignore SIGPIPE signals addLogEntry("ERROR: Handling a cURL SIGPIPE signal despite CURLOPT_NOSIGNAL being set (cURL Operational Bug) ..."); } // Function pointer types matching libcurl WebSocket (WS) API extern(C) struct curl_ws_frame { uint age; uint flags; size_t len; size_t offset; size_t bytesleft; } // WebSocket alias alias PFN_curl_ws_recv = extern(C) CURLcode function(CURL*, void*, size_t, size_t*, const curl_ws_frame**); alias PFN_curl_ws_send = extern(C) CURLcode function(CURL*, const void*, size_t, size_t*, long /*curl_off_t*/, uint); extern(C) struct curl_slist { char* data; curl_slist* next; } extern(C) curl_slist* curl_slist_append(curl_slist* list, const char* string); extern(C) void curl_slist_free_all(curl_slist* list); // Shared pool of CurlEngine instances accessible across all threads __gshared CurlEngine[] curlEnginePool; // __gshared is used to declare a variable that is shared across all threads private __gshared { void* _curlLib; PFN_curl_ws_recv p_curl_ws_recv; PFN_curl_ws_send p_curl_ws_send; bool _wsSymbolsReady; uint _wsProbeOnce; // 0=not run, 1=success, 2=fail } private void* loadCurlLib() { // Respect LD_LIBRARY_PATH etc. auto h = dlopen("libcurl.so.4", RTLD_NOW); if (h is null) h = dlopen("libcurl.so", RTLD_NOW); return h; } private void* findSymbol(const(char)* name) { return dlsym(_curlLib, name); } private bool probeCurlWsSymbols() { if (_wsProbeOnce == 1) return _wsSymbolsReady; if (_wsProbeOnce == 2) return false; // 1) libcurl version check auto vi = curl_version_info(CURLVERSION_NOW); if (vi is null || vi.version_num < CURL_WS_MIN_NUM) { _wsProbeOnce = 2; _wsSymbolsReady = false; return false; } // 2) load libcurl and resolve symbols _curlLib = loadCurlLib(); if (_curlLib is null) { _wsProbeOnce = 2; _wsSymbolsReady = false; return false; } p_curl_ws_recv = cast(PFN_curl_ws_recv) findSymbol("curl_ws_recv"); p_curl_ws_send = cast(PFN_curl_ws_send) findSymbol("curl_ws_send"); _wsSymbolsReady = (p_curl_ws_recv !is null) && (p_curl_ws_send !is null); _wsProbeOnce = _wsSymbolsReady ? 1 : 2; return _wsSymbolsReady; } bool curlSupportsWebSockets() { return probeCurlWsSymbols(); } class CurlResponse { HTTP.Method method; const(char)[] url; const(char)[][const(char)[]] requestHeaders; const(char)[] postBody; bool hasResponse; string[string] responseHeaders; HTTP.StatusLine statusLine; char[] content; this() { reset(); } ~this() { reset(); } void reset() { method = HTTP.Method.undefined; url = ""; requestHeaders = null; postBody = []; hasResponse = false; responseHeaders = null; statusLine.reset(); content = []; } void addRequestHeader(const(char)[] name, const(char)[] value) { requestHeaders[to!string(name)] = to!string(value); } void connect(HTTP.Method method, const(char)[] url) { this.method = method; this.url = url; } const JSONValue json() { JSONValue json; try { json = content.parseJSON(); } catch (JSONException e) { // Log that a JSON Exception was caught, dont output the HTML response from OneDrive if (debugLogging) {addLogEntry("JSON Exception caught when performing HTTP operations - use --debug-https to diagnose further", ["debug"]);} } return json; }; void update(HTTP *http) { hasResponse = true; this.responseHeaders = http.responseHeaders(); this.statusLine = http.statusLine; // has 'microsoftDataCentre' been set yet? if (microsoftDataCentre.empty) { // Extract the 'x-ms-ags-diagnostic' header if it exists if ("x-ms-ags-diagnostic" in this.responseHeaders) { // try and extract the data centre details try { // attempt to extract the data centre location from the header auto diagHeaderData = parseJSON(this.responseHeaders["x-ms-ags-diagnostic"]); string dataCentre = diagHeaderData["ServerInfo"]["DataCenter"].str; // set the Microsoft Data Centre value microsoftDataCentre = dataCentre; } catch (Exception e) { // do nothing } } } // Output the response headers only if using debug mode + debugging https itself if ((debugLogging) && (debugHTTPSResponse)) { addLogEntry("HTTP Response Headers: " ~ to!string(this.responseHeaders), ["debug"]); addLogEntry("HTTP Status Line: " ~ to!string(this.statusLine), ["debug"]); } } @safe pure HTTP.StatusLine getStatus() { return this.statusLine; } // Return the current value of retryAfterValue int getRetryAfterValue() { int delayBeforeRetry; // Is 'retry-after' in the response headers if ("retry-after" in responseHeaders) { // Set the retry-after value if (debugLogging) { addLogEntry("curlEngine.http.perform() => Received a 'Retry-After' Header Response with the following value: " ~ to!string(responseHeaders["retry-after"]), ["debug"]); addLogEntry("curlEngine.http.perform() => Setting retryAfterValue to: " ~ responseHeaders["retry-after"], ["debug"]); } delayBeforeRetry = to!int(responseHeaders["retry-after"]); } else { // Use a 120 second delay as a default given header value was zero // This value is based on log files and data when determining correct process for 429 response handling delayBeforeRetry = 120; // Update that we are over-riding the provided value with a default if (debugLogging) {addLogEntry("HTTP Response Header retry-after value was missing - Using a preconfigured default of: " ~ to!string(delayBeforeRetry), ["debug"]);} } return delayBeforeRetry; } const string parseRequestHeaders(const(const(char)[][const(char)[]]) headers) { string requestHeadersStr = ""; // Ensure response headers is not null and iterate over keys safely. if (headers !is null) { foreach (string header; headers.byKey()) { if (header == "Authorization") { continue; } // Use the 'in' operator to safely check if the key exists in the associative array. if (auto val = header in headers) { requestHeadersStr ~= "< " ~ header ~ ": " ~ *val ~ "\n"; } } } return requestHeadersStr; } const string parseResponseHeaders(const(string[string]) headers) { string responseHeadersStr = ""; // Ensure response headers is not null and iterate over keys safely. if (headers !is null) { foreach (string header; headers.byKey()) { // Check if the key actually exists before accessing it to avoid RangeError. if (auto val = header in headers) { // 'in' checks for the key and returns a pointer to the value if found. responseHeadersStr ~= "> " ~ header ~ ": " ~ *val ~ "\n"; // Dereference pointer to get the value. } } } return responseHeadersStr; } const string dumpDebug() { import std.range; import std.format : format; string str = ""; str ~= format("< %s %s\n", method, url); if (!requestHeaders.empty) { str ~= parseRequestHeaders(requestHeaders); } if (!postBody.empty) { str ~= format("\n----\n%s\n----\n", postBody); } str ~= format("< %s\n", statusLine); if (!responseHeaders.empty) { str ~= parseResponseHeaders(responseHeaders); } return str; } const string dumpResponse() { import std.range; import std.format : format; string str = ""; if (!content.empty) { str ~= format("\n----\n%s\n----\n", content); } return str; } override string toString() const { string str = "Curl debugging: \n"; str ~= dumpDebug(); if (hasResponse) { str ~= "Curl response: \n"; str ~= dumpResponse(); } return str; } } class CurlEngine { HTTP http; File uploadFile; CurlResponse response; bool keepAlive; ulong dnsTimeout; string internalThreadId; SysTime releaseTimestamp; ulong maxIdleTime; private long resumeFromOffset = -1; this() { http = HTTP(); // Directly initializes HTTP using its default constructor response = null; // Initialize as null internalThreadId = generateAlphanumericString(); // Give this CurlEngine instance a unique ID if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("Created new CurlEngine instance id: " ~ to!string(internalThreadId), ["debug"]);} } // The destructor should only clean up resources owned directly by this CurlEngine instance ~this() { // Is the file still open? if (uploadFile.isOpen()) { uploadFile.close(); } // Is 'response' cleared? object.destroy(response); // Destroy, then set to null response = null; // Is the actual http instance is stopped? if (!http.isStopped) { http.shutdown(); } // Make sure this HTTP instance is destroyed object.destroy(http); // ThreadId needs to be set to null internalThreadId = null; } // We are releasing a curl instance back to the pool void releaseEngine() { // Set timestamp of release releaseTimestamp = Clock.currTime(UTC()); // Log that we are releasing this engine back to the pool if ((debugLogging) && (debugHTTPSResponse)) { addLogEntry("CurlEngine releaseEngine() called on instance id: " ~ to!string(internalThreadId), ["debug"]); addLogEntry("CurlEngine curlEnginePool size before release: " ~ to!string(curlEnginePool.length), ["debug"]); string engineReleaseMessage = format("Release Timestamp for CurlEngine %s: %s", to!string(internalThreadId), to!string(releaseTimestamp)); addLogEntry(engineReleaseMessage, ["debug"]); } // cleanup this curl instance before putting it back in the pool cleanup(true); // Cleanup instance by resetting values and flushing cookie cache synchronized (CurlEngine.classinfo) { curlEnginePool ~= this; if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine curlEnginePool size after release: " ~ to!string(curlEnginePool.length), ["debug"]);} } // Perform Garbage Collection GC.collect(); // Return free memory to the OS GC.minimize(); } // Setup a specific SIGPIPE Signal handler due to curl bugs that ignore CurlOption.nosignal void setupSIGPIPESignalHandler() { // Setup the signal handler sigaction_t curlAction; curlAction.sa_handler = &sigpipeHandler; // Direct function pointer assignment sigaction(SIGPIPE, &curlAction, null); // Broken Pipe signal from curl } // Initialise this curl instance void initialise(ulong dnsTimeout, ulong connectTimeout, ulong dataTimeout, ulong operationTimeout, int maxRedirects, bool httpsDebug, string userAgent, bool httpProtocol, ulong userRateLimit, ulong protocolVersion, ulong maxIdleTime, bool keepAlive=true) { // There are many broken curl versions being used, mainly provided by Ubuntu // Ignore SIGPIPE to prevent the application from exiting without reason with an exit code of 141 when bad curl version generate this signal despite being told not to (CurlOption.nosignal) below setupSIGPIPESignalHandler(); // Setting 'keepAlive' to false ensures that when we close the curl instance, any open sockets are closed - which we need to do when running // multiple threads and API instances at the same time otherwise we run out of local files | sockets pretty quickly this.keepAlive = keepAlive; // Curl DNS Timeout Handling this.dnsTimeout = dnsTimeout; // Curl Timeout Handling this.maxIdleTime = maxIdleTime; // libcurl dns_cache_timeout timeout // https://curl.se/libcurl/c/CURLOPT_DNS_CACHE_TIMEOUT.html // https://dlang.org/library/std/net/curl/http.dns_timeout.html http.dnsTimeout = (dur!"seconds"(dnsTimeout)); // Timeout for HTTPS connections // https://curl.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html // https://dlang.org/library/std/net/curl/http.connect_timeout.html http.connectTimeout = (dur!"seconds"(connectTimeout)); // Timeout for activity on connection // This is a DMD | DLANG specific item, not a libcurl item // https://dlang.org/library/std/net/curl/http.data_timeout.html // https://raw.githubusercontent.com/dlang/phobos/master/std/net/curl.d - private enum _defaultDataTimeout = dur!"minutes"(2); http.dataTimeout = (dur!"seconds"(dataTimeout)); // Maximum time any operation is allowed to take // This includes dns resolution, connecting, data transfer, etc. // https://curl.se/libcurl/c/CURLOPT_TIMEOUT_MS.html // https://dlang.org/library/std/net/curl/http.operation_timeout.html http.operationTimeout = (dur!"seconds"(operationTimeout)); // Specify how many redirects should be allowed http.maxRedirects(maxRedirects); // Debug HTTPS http.verbose = httpsDebug; // Use the configured 'user_agent' value http.setUserAgent = userAgent; // What IP protocol version should be used when using Curl - IPv4 & IPv6, IPv4 or IPv6 http.handle.set(CurlOption.ipresolve,protocolVersion); // 0 = IPv4 + IPv6, 1 = IPv4 Only, 2 = IPv6 Only // What version of HTTP protocol do we use? // Curl >= 7.62.0 defaults to http2 for a significant number of operations if (httpProtocol) { // Downgrade to HTTP 1.1 - yes version = 2 is HTTP 1.1 http.handle.set(CurlOption.http_version,2); } // Configure upload / download rate limits if configured // 131072 = 128 KB/s - minimum for basic application operations to prevent timeouts // A 0 value means rate is unlimited, and is the curl default if (userRateLimit > 0) { // set rate limit http.handle.set(CurlOption.max_send_speed_large,userRateLimit); http.handle.set(CurlOption.max_recv_speed_large,userRateLimit); } // Explicitly set libcurl options to avoid using signal handlers in a multi-threaded environment // See: https://curl.se/libcurl/c/CURLOPT_NOSIGNAL.html // The CURLOPT_NOSIGNAL option is intended for use in multi-threaded programs to ensure that libcurl does not use any signal handling. // Set CURLOPT_NOSIGNAL to 1 to prevent libcurl from using signal handlers, thus avoiding interference with the application's signal handling which could lead to issues such as unstable behavior or application crashes. http.handle.set(CurlOption.nosignal,1); // https://curl.se/libcurl/c/CURLOPT_TCP_NODELAY.html // Ensure that TCP_NODELAY is set to 0 to ensure that TCP NAGLE is enabled http.handle.set(CurlOption.tcp_nodelay,0); // https://curl.se/libcurl/c/CURLOPT_FORBID_REUSE.html // CURLOPT_FORBID_REUSE - make connection get closed at once after use // Setting this to 0 ensures that we ARE reusing connections (we did this in v2.4.xx) to ensure connections remained open and usable // Setting this to 1 ensures that when we close the curl instance, any open sockets are forced closed when the API curl instance is destroyed // The libcurl default is 0 as per the documentation (to REUSE connections) - ensure we are configuring to reuse sockets http.handle.set(CurlOption.forbid_reuse,0); if (httpsDebug) { // Output what options we are using so that in the debug log this can be tracked if ((debugLogging) && (debugHTTPSResponse)) { addLogEntry("http.dnsTimeout = " ~ to!string(dnsTimeout), ["debug"]); addLogEntry("http.connectTimeout = " ~ to!string(connectTimeout), ["debug"]); addLogEntry("http.dataTimeout = " ~ to!string(dataTimeout), ["debug"]); addLogEntry("http.operationTimeout = " ~ to!string(operationTimeout), ["debug"]); addLogEntry("http.maxRedirects = " ~ to!string(maxRedirects), ["debug"]); addLogEntry("http.CurlOption.ipresolve = " ~ to!string(protocolVersion), ["debug"]); addLogEntry("http.header.Connection.keepAlive = " ~ to!string(keepAlive), ["debug"]); } } } void setResponseHolder(CurlResponse response) { if (response is null) { // Create a response instance if it doesn't already exist if (this.response is null) this.response = new CurlResponse(); } else { this.response = response; } } void addRequestHeader(const(char)[] name, const(char)[] value) { setResponseHolder(null); http.addRequestHeader(name, value); response.addRequestHeader(name, value); } void connect(HTTP.Method method, const(char)[] url) { setResponseHolder(null); if (!keepAlive) addRequestHeader("Connection", "close"); http.method = method; http.url = url; response.connect(method, url); } void setContent(const(char)[] contentType, const(char)[] sendData) { setResponseHolder(null); addRequestHeader("Content-Type", contentType); if (sendData) { http.contentLength = sendData.length; http.onSend = (void[] buf) { import std.algorithm: min; size_t minLen = min(buf.length, sendData.length); if (minLen == 0) return 0; buf[0 .. minLen] = cast(void[]) sendData[0 .. minLen]; sendData = sendData[minLen .. $]; return minLen; }; response.postBody = sendData; } } void setFile(string filepath, string contentRange, ulong offset, ulong offsetSize) { setResponseHolder(null); // open file as read-only in binary mode uploadFile = File(filepath, "rb"); if (contentRange.empty) { offsetSize = uploadFile.size(); } else { addRequestHeader("Content-Range", contentRange); uploadFile.seek(offset); } // Setup progress bar to display http.onProgress = delegate int(size_t dltotal, size_t dlnow, size_t ultotal, size_t ulnow) { return 0; }; addRequestHeader("Content-Type", "application/octet-stream"); http.onSend = data => uploadFile.rawRead(data).length; http.contentLength = offsetSize; } void setZeroContentLength() { // Explicit HTTP semantics http.contentLength = 0; addRequestHeader("Content-Length", to!string(0)); // Force libcurl POST-with-empty-body semantics // This prevents libcurl from attempting to read from stdin when performing a POST with no payload. http.handle.set(CurlOption.postfields, ""); http.handle.set(CurlOption.postfieldsize, 0L); // Defensive: ensure we are NOT in upload/read-callback mode http.handle.set(CurlOption.upload, 0); } CurlResponse execute() { scope(exit) { cleanup(); } setResponseHolder(null); http.onReceive = (ubyte[] data) { response.content ~= data; // HTTP Server Response Code Debugging if --https-debug is being used return data.length; }; http.perform(); response.update(&http); return response; } CurlResponse download(string originalFilename, string downloadFilename) { setResponseHolder(null); // Open the file in append mode if resuming, else write mode auto file = (resumeFromOffset > 0) ? File(downloadFilename, "ab") // append binary : File(downloadFilename, "wb"); // write binary // Function exit scope scope(exit) { cleanup(); if (file.isOpen()){ // close open file file.close(); } } // Apply Range header if resuming if (resumeFromOffset > 0) { string rangeHeader = format("bytes=%d-", resumeFromOffset); addRequestHeader("Range", rangeHeader); } // Receive data http.onReceive = (ubyte[] data) { file.rawWrite(data); return data.length; }; // Perform HTTP Operation http.perform(); // close open file - avoids problems with renaming on GCS Buckets and other semi-POSIX systems if (file.isOpen()){ file.close(); } // Rename downloaded file rename(downloadFilename, originalFilename); // Update response and return response response.update(&http); return response; } // Cleanup this instance internal variables that may have been set void cleanup(bool flushCookies = false) { // Reset any values to defaults, freeing any set objects if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine cleanup() called on instance id: " ~ to!string(internalThreadId), ["debug"]);} // Is the instance is stopped? if (!http.isStopped) { // A stopped instance is not usable, these cannot be reset http.clearRequestHeaders(); http.onSend = null; http.onReceive = null; http.onReceiveHeader = null; http.onReceiveStatusLine = null; http.onProgress = delegate int(size_t dltotal, size_t dlnow, size_t ultotal, size_t ulnow) { return 0; }; http.contentLength = 0; // We only do this if we are pushing the curl engine back to the curl pool if (flushCookies) { // Flush the cookie cache as well http.flushCookieJar(); http.clearSessionCookies(); http.clearAllCookies(); } } // set the response to null response = null; // close file if open if (uploadFile.isOpen()){ // close open file uploadFile.close(); } } // Shut down the curl instance & close any open sockets void shutdownCurlHTTPInstance() { // Log that we are attempting to shutdown this curl instance if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine shutdownCurlHTTPInstance() called on instance id: " ~ to!string(internalThreadId), ["debug"]);} // Is this curl instance is stopped? if (!http.isStopped) { if ((debugLogging) && (debugHTTPSResponse)) { addLogEntry("HTTP instance still active: " ~ to!string(internalThreadId), ["debug"]); addLogEntry("HTTP instance isStopped state before http.shutdown(): " ~ to!string(http.isStopped), ["debug"]); } http.shutdown(); if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("HTTP instance isStopped state post http.shutdown(): " ~ to!string(http.isStopped), ["debug"]);} object.destroy(http); // Destroy, however we cant set to null if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("HTTP instance shutdown and destroyed: " ~ to!string(internalThreadId), ["debug"]);} } else { // Already stopped .. destroy it object.destroy(http); // Destroy, however we cant set to null if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("Stopped HTTP instance shutdown and destroyed: " ~ to!string(internalThreadId), ["debug"]);} } // Perform Garbage Collection GC.collect(); // Return free memory to the OS GC.minimize(); } // Disable SSL certificate peer verification for libcurl operations. // // This function disables the verification of the SSL peer's certificate // by setting CURLOPT_SSL_VERIFYPEER to 0. This means that libcurl will // accept any certificate presented by the server, regardless of whether // it is signed by a trusted certificate authority. // // ------------------------------------------------------------------------------------- // WARNING: Disabling SSL peer verification introduces significant security risks: // ------------------------------------------------------------------------------------- // - Man-in-the-Middle (MITM) attacks become trivially possible. // - Malicious servers can impersonate trusted endpoints. // - Confidential data (authentication tokens, file contents) can be intercepted. // - Violates industry security standards and regulatory compliance requirements. // - Should never be used in production environments or on untrusted networks. // // This option should only be enabled for internal testing, debugging self-signed // certificates, or explicitly controlled environments with known risks. // // See also: // https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html void setDisableSSLVerifyPeer() { // Emit a runtime warning if debug logging is enabled if (debugLogging) { addLogEntry("WARNING: SSL peer verification has been DISABLED!", ["debug"]); addLogEntry(" This allows invalid or self-signed certificates to be accepted.", ["debug"]); addLogEntry(" Use ONLY for testing. This severely weakens HTTPS security.", ["debug"]); } // Disable SSL certificate verification (DANGEROUS) http.handle.set(CurlOption.ssl_verifypeer, 0); } // Enable SSL Certificate Verification void setEnableSSLVerifyPeer() { // Enable SSL certificate verification addLogEntry("Enabling SSL peer verification"); http.handle.set(CurlOption.ssl_verifypeer, 1); } // Set an applicable resumable offset point when downloading a file void setDownloadResumeOffset(long offset) { resumeFromOffset = offset; } // reset resumable offset point to negative value void resetDownloadResumeOffset() { resumeFromOffset = -1; } } // Methods to control obtaining and releasing a CurlEngine instance from the curlEnginePool // Get a curl instance for the OneDrive API to use CurlEngine getCurlInstance() { if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine getCurlInstance() called", ["debug"]);} synchronized (CurlEngine.classinfo) { // What is the current pool size if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine curlEnginePool current size: " ~ to!string(curlEnginePool.length), ["debug"]);} if (curlEnginePool.empty) { if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine curlEnginePool is empty - constructing a new CurlEngine instance", ["debug"]);} return new CurlEngine; // Constructs a new CurlEngine with a fresh HTTP instance } else { CurlEngine curlEngine = curlEnginePool[$ - 1]; curlEnginePool.popBack(); // assumes a LIFO (last-in, first-out) usage pattern // Is this engine stopped? if (curlEngine.http.isStopped) { // return a new curl engine as a stopped one cannot be used if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine was in a stopped state (not usable) - constructing a new CurlEngine instance", ["debug"]);} return new CurlEngine; // Constructs a new CurlEngine with a fresh HTTP instance } else { // When was this engine last used? auto elapsedTime = Clock.currTime(UTC()) - curlEngine.releaseTimestamp; if ((debugLogging) && (debugHTTPSResponse)) { string engineIdleMessage = format("CurlEngine %s time since last use: %s", to!string(curlEngine.internalThreadId), to!string(elapsedTime)); addLogEntry(engineIdleMessage, ["debug"]); } // If greater than 120 seconds (default), the treat this as a stale engine, preventing: // * Too old connection (xxx seconds idle), disconnect it // * Connection 0 seems to be dead! // * Closing connection 0 if (elapsedTime > dur!"seconds"(curlEngine.maxIdleTime)) { // Too long idle engine, clean it up and create a new one if ((debugLogging) && (debugHTTPSResponse)) { string curlTooOldMessage = format("CurlEngine idle for > %d seconds .... destroying and returning a new curl engine instance", curlEngine.maxIdleTime); addLogEntry(curlTooOldMessage, ["debug"]); } curlEngine.cleanup(true); // Cleanup instance by resetting values and flushing cookie cache curlEngine.shutdownCurlHTTPInstance(); // Assume proper cleanup of any resources used by HTTP if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("Returning NEW curlEngine instance", ["debug"]);} return new CurlEngine; // Constructs a new CurlEngine with a fresh HTTP instance } else { // return an existing curl engine if ((debugLogging) && (debugHTTPSResponse)) { addLogEntry("CurlEngine was in a valid state - returning existing CurlEngine instance", ["debug"]); addLogEntry("Using CurlEngine instance ID: " ~ curlEngine.internalThreadId, ["debug"]); } // return the existing engine return curlEngine; } } } } } // Release all CurlEngine instances void releaseAllCurlInstances() { if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine releaseAllCurlInstances() called", ["debug"]);} synchronized (CurlEngine.classinfo) { // What is the current pool size if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine curlEnginePool size to release: " ~ to!string(curlEnginePool.length), ["debug"]);} if (curlEnginePool.length > 0) { // Safely iterate and clean up each CurlEngine instance foreach (curlEngineInstance; curlEnginePool) { try { curlEngineInstance.cleanup(true); // Cleanup instance by resetting values and flushing cookie cache curlEngineInstance.shutdownCurlHTTPInstance(); // Assume proper cleanup of any resources used by HTTP } catch (Exception e) { // Log the error or handle it appropriately // e.g., writeln("Error during cleanup/shutdown: ", e.toString()); } // It's safe to destroy the object here assuming no other references exist object.destroy(curlEngineInstance); // Destroy, then set to null curlEngineInstance = null; // Perform Garbage Collection on this destroyed curl engine GC.collect(); // Log release if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine destroyed", ["debug"]);} } // Clear the array after all instances have been handled curlEnginePool.length = 0; // More explicit than curlEnginePool = []; } } // Perform Garbage Collection on the destroyed curl engines GC.collect(); // Return free memory to the OS GC.minimize(); // Log that all curl engines have been released if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("CurlEngine releaseAllCurlInstances() completed", ["debug"]);} } // Return how many curl engines there are ulong curlEnginePoolLength() { return curlEnginePool.length; } ================================================ FILE: src/curlWebsockets.d ================================================ // What is this module called? module curlWebsockets; /****************************************************************************** * Minimal RFC6455 WebSocket client over libcurl (CONNECT_ONLY). ******************************************************************************/ // What does this module require to function? import etc.c.curl : CURL, CURLcode, curl_easy_cleanup, curl_easy_getinfo, curl_easy_init, curl_easy_perform, curl_easy_recv, curl_easy_reset, curl_easy_send, curl_easy_setopt; import core.stdc.string : memcpy, memmove; import core.time : MonoTime, dur; import std.array : Appender, appender; import std.base64 : Base64; import std.meta : AliasSeq; import std.random : Random, unpredictableSeed, uniform; import std.range : empty; import std.string : indexOf, startsWith, toLower, toStringz; import std.exception : collectException; import std.conv; // What other modules that we have created do we need to import? import log; // ========== Logging Shim ========== private void logCurlWebsocketOutput(string s) { if (debugLogging) { addLogEntry("WEBSOCKET: " ~ s, ["debug"]); } } private struct WsFrame { ubyte fin; ubyte opcode; bool masked; ulong payloadLen; ubyte[4] maskKey; ubyte[] payload; } final class CurlWebSocket { private: // libcurl constants defined locally enum int CURLOPT_URL = 10002; enum int CURLOPT_FOLLOWLOCATION = 52; enum int CURLOPT_NOSIGNAL = 99; enum int CURLOPT_USERAGENT = 10018; enum int CURLOPT_SSL_VERIFYPEER = 64; enum int CURLOPT_SSL_VERIFYHOST = 81; enum int CURLOPT_CONNECT_ONLY = 141; enum int CURLOPT_TIMEOUT_MS = 155; enum int CURLOPT_CONNECTTIMEOUT_MS = 156; enum int CURLOPT_VERBOSE = 41; // Additional constants needed for WebSocket handling enum int CURLOPT_HTTP_VERSION = 84; // CURLOPT_HTTP_VERSION enum int CURLOPT_SSL_ENABLE_ALPN = 226; // CURLOPT_SSL_ENABLE_ALPN enum int CURLOPT_SSL_ENABLE_NPN = 225; // CURLOPT_SSL_ENABLE_NPN // HTTP version flags (for CURLOPT_HTTP_VERSION) enum long CURL_HTTP_VERSION_NONE = 0; enum long CURL_HTTP_VERSION_1_0 = 1; enum long CURL_HTTP_VERSION_1_1 = 2; enum long CURL_HTTP_VERSION_2_0 = 3; enum long CURL_HTTP_VERSION_2TLS = 4; enum long CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE = 5; enum long CURL_HTTP_VERSION_3 = 30; // (added in curl 7.66.0+) CURL* curl = null; bool websocketConnected = false; int connectTimeoutMs = 10000; int ioTimeoutMs = 15000; string userAgent = ""; bool httpsDebug = false; string scheme; string host; int port; string hostPort; string pathQuery; ubyte[] recvBuf; Random rng; public: this() { websocketConnected = false; curl = curl_easy_init(); rng = Random(unpredictableSeed); logCurlWebsocketOutput("Created a new instance of a CurlWebSocket object accessing libcurl for HTTP operations"); } ~this() { // No logging output in ~this() if (curl !is null) { curl_easy_cleanup(curl); curl = null; } websocketConnected = false; } bool isConnected() { return websocketConnected; } void setTimeouts(int connectMs, int rwMs) { connectTimeoutMs = connectMs; ioTimeoutMs = rwMs; } void setUserAgent(string ua) { if (!ua.empty) userAgent = ua; } void setHTTPSDebug(bool httpsDebugInput) { httpsDebug = httpsDebugInput; } int connect(string wsUrl) { if (curl is null) { logCurlWebsocketOutput("libcurl handle not initialised"); return -1; } ParsedUrl p = parseWsUrl(wsUrl); if (!p.ok) { logCurlWebsocketOutput("Invalid WebSocket URL: " ~ wsUrl); return -2; } scheme = p.scheme; host = p.host; port = p.port; hostPort = p.hostPort; pathQuery = p.pathQuery; string connectUrl = (scheme == "wss" ? "https://" : "http://") ~ hostPort ~ pathQuery; // Reset 'curl' using curl_easy_reset curl_easy_reset(curl); // Configure required curl options curl_easy_setopt(curl, cast(int)CURLOPT_NOSIGNAL, 1L); curl_easy_setopt(curl, cast(int)CURLOPT_FOLLOWLOCATION, 1L); curl_easy_setopt(curl, cast(int)CURLOPT_USERAGENT, userAgent.toStringz); // NUL-terminated curl_easy_setopt(curl, cast(int)CURLOPT_CONNECTTIMEOUT_MS, cast(long)connectTimeoutMs); curl_easy_setopt(curl, cast(int)CURLOPT_TIMEOUT_MS, cast(long)ioTimeoutMs); curl_easy_setopt(curl, cast(int)CURLOPT_SSL_VERIFYPEER, 1L); curl_easy_setopt(curl, cast(int)CURLOPT_SSL_VERIFYHOST, 2L); curl_easy_setopt(curl, cast(int)CURLOPT_CONNECT_ONLY, 1L); curl_easy_setopt(curl, cast(int)CURLOPT_URL, connectUrl.toStringz); // NUL-terminated // Force HTTP/1.1 and disable ALPN/NPN curl_easy_setopt(curl, cast(int)CURLOPT_SSL_ENABLE_ALPN, 0L); curl_easy_setopt(curl, cast(int)CURLOPT_SSL_ENABLE_NPN, 0L); curl_easy_setopt(curl, cast(int)CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1); // Do we enable HTTPS Debugging? if (httpsDebug) { // Enable curl verbosity curl_easy_setopt(curl, cast(int)CURLOPT_VERBOSE, 1L); } else { // Disable curl verbosity curl_easy_setopt(curl, cast(int)CURLOPT_VERBOSE, 0L); } auto rc = curl_easy_perform(curl); if (rc != 0) { logCurlWebsocketOutput("libcurl connect failed"); return -3; } auto req = buildUpgradeRequest(); if (sendAll(req) != 0) { logCurlWebsocketOutput("Failed sending HTTP upgrade request"); return -4; } // Read headers until CRLFCRLF, with deadline (don’t treat 0-bytes as EOF). string hdrs; enum maxHdr = 16 * 1024; auto deadline = MonoTime.currTime + dur!"seconds"(10); { ubyte[4096] tmp; size_t total; for (;;) { int got = recvSome(tmp[]); if (got < 0) { logCurlWebsocketOutput("Failed receiving HTTP upgrade response"); return -5; } if (got == 0) { if (MonoTime.currTime >= deadline) { logCurlWebsocketOutput("Timeout waiting for HTTP upgrade response"); return -6; } continue; } hdrs ~= cast(const(char)[]) tmp[0 .. cast(size_t)got]; total += cast(size_t)got; auto pos = hdrs.indexOf("\r\n\r\n"); if (pos >= 0) { auto remain = hdrs[(cast(size_t)pos + 4) .. hdrs.length]; if (remain.length > 0) { auto ru = cast(const(ubyte)[]) remain; size_t old = recvBuf.length; recvBuf.length = old + ru.length; memcpy(recvBuf.ptr + old, ru.ptr, ru.length); } hdrs = hdrs[0 .. cast(size_t)pos]; break; } if (total > maxHdr) { logCurlWebsocketOutput("HTTP upgrade headers too large"); return -7; } } } { auto firstLineEnd = hdrs.indexOf("\r\n"); string statusLine = firstLineEnd > 0 ? hdrs[0 .. cast(size_t)firstLineEnd] : hdrs; if (statusLine.indexOf("101") < 0) { logCurlWebsocketOutput("HTTP upgrade failed; status line: " ~ statusLine); return -8; } auto low = hdrs.toLower(); if (low.indexOf("upgrade: websocket") < 0 || low.indexOf("connection: upgrade") < 0) { logCurlWebsocketOutput("HTTP upgrade missing expected headers"); return -9; } } // Log that protocol switch confirmed, upgraded to RFC6455 logCurlWebsocketOutput("Received HTTP 101 Switching Protocols confirmed; Upgraded to RFC6455"); websocketConnected = true; return 0; } int close(ushort code = 1000, string reason = "") { logCurlWebsocketOutput("Running curlWebsocket close()"); if (!websocketConnected) { logCurlWebsocketOutput("Websocket already closed - websocketConnected = false"); return 0; } else { logCurlWebsocketOutput("Running curlWebsocket close() - websocketConnected = true"); } // Build close payload: 2 bytes status code (network order) + optional reason ubyte[] pay; pay.length = 2 + reason.length; pay[0] = cast(ubyte)((code >> 8) & 0xFF); pay[1] = cast(ubyte)(code & 0xFF); foreach (i; 0 .. reason.length) pay[2 + i] = cast(ubyte)reason[i]; auto frame = encodeFrame(0x8, pay); // opcode 0x8 = Close auto rc = sendAll(frame); // Even if sending fails, cleanup below so we don’t leak. logCurlWebsocketOutput("Sending RFC6455 Close (code=" ~ to!string(code) ~ ")"); // Flag we are no longer connected with the websocket websocketConnected = false; return rc; } // Cleanup curl handler void cleanupCurlHandle() { // No logging output for this function if (curl !is null) { curl_easy_cleanup(curl); curl = null; } websocketConnected = false; } int sendText(string payload) { if (!websocketConnected) return -1; auto frame = encodeFrame(0x1, cast(const(ubyte)[])payload); return sendAll(frame); } string recvText() { if (!websocketConnected) return ""; for (;;) { auto f = tryParseFrame(); if (f.opcode == 0xFF) { ubyte[4096] tmp; int got = recvSome(tmp[]); if (got <= 0) return ""; size_t old = recvBuf.length; recvBuf.length = old + cast(size_t)got; memcpy(recvBuf.ptr + old, tmp.ptr, cast(size_t)got); continue; } if (f.opcode == 0x1) { return cast(string) f.payload; } else if (f.opcode == 0x9) { auto pong = encodeFrame(0xA, f.payload); auto _ = sendAll(pong); continue; } else if (f.opcode == 0xA) { continue; } else if (f.opcode == 0x8) { websocketConnected = false; return ""; } else { continue; } } } private: struct ParsedUrl { bool ok; string scheme; string host; int port; string hostPort; string pathQuery; } static int parsePortDec(string s) { if (s.length == 0) return 0; int v = 0; foreach (ch; s) { if (ch < '0' || ch > '9') return 0; v = v * 10 + (cast(int)ch - cast(int)'0'); if (v > 65535) return 0; } return v; } ParsedUrl parseWsUrl(string u) { ParsedUrl p; p.ok = false; auto sidx = u.indexOf("://"); if (sidx <= 0) return p; string sc = u[0 .. cast(size_t)sidx]; string rest = u[(cast(size_t)sidx + 3) .. u.length]; auto scl = sc.toLower(); if (scl == "ws") p.scheme = "ws"; else if (scl == "wss") p.scheme = "wss"; else return p; auto slash = rest.indexOf("/"); string hostport; if (slash < 0) { hostport = rest; p.pathQuery = "/"; } else { hostport = rest[0 .. cast(size_t)slash]; p.pathQuery = rest[cast(size_t)slash .. rest.length]; } auto col = hostport.indexOf(":"); if (col >= 0) { p.host = hostport[0 .. cast(size_t)col]; string ps = hostport[(cast(size_t)col + 1) .. hostport.length]; int prt = parsePortDec(ps); if (prt == 0) return p; p.port = prt; p.hostPort = p.host ~ ":" ~ to!string(p.port); } else { p.host = hostport; p.port = (p.scheme == "wss") ? 443 : 80; p.hostPort = p.host; } if (p.pathQuery.length == 0 || p.pathQuery[0] != '/') p.pathQuery = "/" ~ p.pathQuery; p.ok = true; return p; } string buildUpgradeRequest() { // Sec-WebSocket-Key: random 16 bytes, base64 ubyte[16] keyBytes; foreach (i; 0 .. 16) keyBytes[i] = cast(ubyte) uniform(0, 256, rng); auto keyB64 = Base64.encode(keyBytes[]); // Origin header (some proxies expect it) string origin = (scheme == "wss" ? "https://" : "http://") ~ host; string req = "GET " ~ pathQuery ~ " HTTP/1.1\r\n"; req ~= "Host: " ~ hostPort ~ "\r\n"; req ~= "User-Agent: " ~ userAgent ~ "\r\n"; req ~= "Upgrade: websocket\r\n"; req ~= "Connection: Upgrade\r\n"; req ~= "Sec-WebSocket-Version: 13\r\n"; req ~= "Sec-WebSocket-Key: " ~ keyB64 ~ "\r\n"; req ~= "Origin: " ~ origin ~ "\r\n"; req ~= "\r\n"; return req; } int sendAll(const(char)[] data) { size_t sent = 0; while (sent < data.length) { size_t now = 0; auto rc = curl_easy_send(curl, cast(void*)(data.ptr + sent), data.length - sent, &now); if (rc != 0 && now == 0) return -1; sent += now; } return 0; } int sendAll(const(ubyte)[] data) { size_t sent = 0; while (sent < data.length) { size_t now = 0; auto rc = curl_easy_send(curl, cast(void*)(data.ptr + sent), data.length - sent, &now); if (rc != 0 && now == 0) return -1; sent += now; } return 0; } int recvSome(ubyte[] buf) { size_t got = 0; auto rc = curl_easy_recv(curl, cast(void*)buf.ptr, buf.length, &got); if (rc != 0) return 0; // treat EAGAIN etc. as "no bytes now" return cast(int)got; } ubyte[] encodeFrame(ubyte opcode, const(ubyte)[] payload) { Appender!(ubyte[]) outp = appender!(ubyte[])(); outp.reserve(2 + 4 + payload.length + 8); ubyte b0 = cast(ubyte)(0x80 | (opcode & 0x0F)); // FIN=1 outp.put(b0); ubyte maskBit = 0x80; ulong len = cast(ulong)payload.length; if (len <= 125) { outp.put(cast(ubyte)(maskBit | cast(ubyte)len)); } else if (len <= 0xFFFF) { outp.put(cast(ubyte)(maskBit | 126)); outp.put(cast(ubyte)((len >> 8) & 0xFF)); outp.put(cast(ubyte)(len & 0xFF)); } else { outp.put(cast(ubyte)(maskBit | 127)); foreach (shift; AliasSeq!(56, 48, 40, 32, 24, 16, 8, 0)) { outp.put(cast(ubyte)((len >> shift) & 0xFF)); } } ubyte[4] key; foreach (i; 0 .. 4) key[i] = cast(ubyte) uniform(0, 256, rng); outp.put(key[]); auto masked = new ubyte[payload.length]; foreach (i; 0 .. payload.length) masked[i] = payload[i] ^ key[i % 4]; outp.put(masked[]); return outp.data; } WsFrame tryParseFrame() { WsFrame f; f.opcode = 0xFF; if (recvBuf.length < 2) return f; size_t i = 0; ubyte b0 = recvBuf[i]; i += 1; ubyte b1 = recvBuf[i]; i += 1; bool fin = (b0 & 0x80) != 0; ubyte opcode = cast(ubyte)(b0 & 0x0F); bool masked = (b1 & 0x80) != 0; ulong len = cast(ulong)(b1 & 0x7F); if (len == 126) { if (recvBuf.length < i + 2) return f; len = (cast(ulong)recvBuf[i] << 8) | cast(ulong)recvBuf[i + 1]; i += 2; } else if (len == 127) { if (recvBuf.length < i + 8) return f; len = 0; foreach (shift; AliasSeq!(56, 48, 40, 32, 24, 16, 8, 0)) { len |= (cast(ulong)recvBuf[i] << shift); i += 1; } } ubyte[4] key; if (masked) { if (recvBuf.length < i + 4) return f; foreach (k; 0 .. 4) key[k] = recvBuf[i + k]; i += 4; } if (recvBuf.length < i + cast(size_t)len) return f; auto start = i; auto end = i + cast(size_t)len; auto raw = recvBuf[start .. end]; ubyte[] data; if (masked) { data = new ubyte[raw.length]; foreach (idx; 0 .. raw.length) data[idx] = raw[idx] ^ key[idx % 4]; } else { data = raw.dup; } auto consumed = end; auto remainLen = recvBuf.length - consumed; if (remainLen > 0) { memmove(recvBuf.ptr, recvBuf.ptr + consumed, remainLen); } recvBuf.length = remainLen; f.fin = fin ? 1 : 0; f.opcode = opcode; f.masked = masked; f.payloadLen = len; f.maskKey = key; f.payload = data; return f; } } ================================================ FILE: src/intune.d ================================================ // What is this module called? module intune; // What does this module require to function? import core.stdc.string : strcmp; import core.stdc.stdlib : malloc, free; import core.thread : Thread; import core.time : dur; import std.string : fromStringz, toStringz; import std.conv : to; import std.json : JSONValue, parseJSON, JSONType; import std.uuid : randomUUID; import std.range : empty; import std.format : format; // What 'onedrive' modules do we import? import log; extern(C): alias dbus_bool_t = int; struct DBusError { char* name; char* message; uint[8] dummy; void* padding; } struct DBusConnection; struct DBusMessage; struct DBusMessageIter; enum DBusBusType { DBUS_BUS_SESSION = 0, } void dbus_error_init(DBusError* error); void dbus_error_free(DBusError* error); int dbus_error_is_set(DBusError* error); DBusConnection* dbus_bus_get(DBusBusType type, DBusError* error); dbus_bool_t dbus_bus_name_has_owner(DBusConnection* conn, const char* name, DBusError* error); DBusMessage* dbus_message_new_method_call(const char* dest, const char* path, const char* iface, const char* method); dbus_bool_t dbus_connection_send(DBusConnection* conn, DBusMessage* msg, void* client_serial); void dbus_connection_flush(DBusConnection* conn); DBusMessage* dbus_connection_send_with_reply_and_block(DBusConnection* conn, DBusMessage* msg, int timeout_ms, DBusError* error); void dbus_message_unref(DBusMessage* msg); dbus_bool_t dbus_message_iter_init_append(DBusMessage* msg, DBusMessageIter* iter); dbus_bool_t dbus_message_iter_append_basic(DBusMessageIter* iter, int type, const void* value); dbus_bool_t dbus_message_iter_init(DBusMessage* msg, DBusMessageIter* iter); dbus_bool_t dbus_message_iter_get_arg_type(DBusMessageIter* iter); void dbus_message_iter_get_basic(DBusMessageIter* iter, void* value); enum DBUS_TYPE_STRING = 115; enum DBUS_MESSAGE_ITER_SIZE = 128; bool check_intune_broker_available() { version (linux) { DBusError err; dbus_error_init(&err); DBusConnection* conn = dbus_bus_get(DBusBusType.DBUS_BUS_SESSION, &err); if (dbus_error_is_set(&err)) { dbus_error_free(&err); return false; } if (conn is null) return false; dbus_bool_t hasOwner = dbus_bus_name_has_owner(conn, "com.microsoft.identity.broker1", &err); if (dbus_error_is_set(&err)) { dbus_error_free(&err); return false; } return hasOwner != 0; } else { return false; } } bool wait_for_broker(int timeoutSeconds = 10) { int waited = 0; while (waited < timeoutSeconds) { if (check_intune_broker_available()) return true; Thread.sleep(dur!"seconds"(1)); waited++; } return false; } string build_auth_request(string accountJson = "", string clientId = "") { string header = format(`{ "authParameters": { "clientId": "%s", "redirectUri": "https://login.microsoftonline.com/common/oauth2/nativeclient", "authority": "https://login.microsoftonline.com/common", "requestedScopes": [ "Files.ReadWrite", "Files.ReadWrite.All", "Sites.ReadWrite.All", "offline_access" ]`, clientId); string footer = ` } }`; if (!accountJson.empty) return header ~ `,"account": ` ~ accountJson ~ footer; else return header ~ footer; } struct AuthResult { JSONValue brokerTokenResponse; } // Initiate interactive authentication via D-Bus using the Microsoft Identity Broker AuthResult acquire_token_interactive(string clientId) { AuthResult result; version (linux) { if (!wait_for_broker(10)) { addLogEntry("Timed out waiting for Identity Broker to appear on D-Bus"); return result; } // Step 1: Call acquireTokenInteractively and capture account from result DBusError err; dbus_error_init(&err); DBusConnection* conn = dbus_bus_get(DBusBusType.DBUS_BUS_SESSION, &err); if (dbus_error_is_set(&err) || conn is null) return result; DBusMessage* msg = dbus_message_new_method_call( "com.microsoft.identity.broker1", "/com/microsoft/identity/broker1", "com.microsoft.identity.Broker1", "acquireTokenInteractively" ); if (msg is null) return result; string correlationId = randomUUID().toString(); string accountJson = ""; string requestJson = build_auth_request(accountJson, clientId); DBusMessageIter* args = cast(DBusMessageIter*) malloc(DBUS_MESSAGE_ITER_SIZE); if (!dbus_message_iter_init_append(msg, args)) { dbus_message_unref(msg); free(args); return result; } const(char)* protocol = toStringz("0.0"); const(char)* corrId = toStringz(correlationId); const(char)* reqJson = toStringz(requestJson); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &protocol); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &corrId); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &reqJson); free(args); DBusMessage* reply = dbus_connection_send_with_reply_and_block(conn, msg, 60000, &err); dbus_message_unref(msg); if (dbus_error_is_set(&err) || reply is null) { addLogEntry("Interactive call failed"); return result; } DBusMessageIter* iter = cast(DBusMessageIter*) malloc(DBUS_MESSAGE_ITER_SIZE); if (!dbus_message_iter_init(reply, iter)) { dbus_message_unref(reply); free(iter); return result; } char* responseStr; dbus_message_iter_get_basic(iter, &responseStr); dbus_message_unref(reply); free(iter); string jsonResponse = fromStringz(responseStr).idup; if (debugLogging) {addLogEntry("Interactive raw response: " ~ to!string(jsonResponse), ["debug"]);} JSONValue parsed = parseJSON(jsonResponse); if (parsed.type != JSONType.object) return result; auto obj = parsed.object; if ("brokerTokenResponse" in obj) { result.brokerTokenResponse = obj["brokerTokenResponse"]; } } return result; } // Perform silent authentication via D-Bus using the Microsoft Identity Broker AuthResult acquire_token_silently(string accountJson, string clientId) { AuthResult result; version (linux) { DBusError err; dbus_error_init(&err); DBusConnection* conn = dbus_bus_get(DBusBusType.DBUS_BUS_SESSION, &err); if (dbus_error_is_set(&err) || conn is null) return result; DBusMessage* msg = dbus_message_new_method_call( "com.microsoft.identity.broker1", "/com/microsoft/identity/broker1", "com.microsoft.identity.Broker1", "acquireTokenSilently" ); if (msg is null) return result; string correlationId = randomUUID().toString(); string requestJson = build_auth_request(accountJson, clientId); DBusMessageIter* args = cast(DBusMessageIter*) malloc(DBUS_MESSAGE_ITER_SIZE); if (!dbus_message_iter_init_append(msg, args)) { dbus_message_unref(msg); free(args); return result; } const(char)* protocol = toStringz("0.0"); const(char)* corrId = toStringz(correlationId); const(char)* reqJson = toStringz(requestJson); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &protocol); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &corrId); dbus_message_iter_append_basic(args, DBUS_TYPE_STRING, &reqJson); free(args); DBusMessage* reply = dbus_connection_send_with_reply_and_block(conn, msg, 10000, &err); dbus_message_unref(msg); if (dbus_error_is_set(&err) || reply is null) return result; DBusMessageIter* iter = cast(DBusMessageIter*) malloc(DBUS_MESSAGE_ITER_SIZE); if (!dbus_message_iter_init(reply, iter)) { dbus_message_unref(reply); free(iter); return result; } char* responseStr; dbus_message_iter_get_basic(iter, &responseStr); dbus_message_unref(reply); free(iter); string jsonResponse = fromStringz(responseStr).idup; if (debugLogging) {addLogEntry("Silent raw response: " ~ to!string(jsonResponse), ["debug"]);} JSONValue parsed = parseJSON(jsonResponse); if (parsed.type != JSONType.object) return result; auto obj = parsed.object; if (!("brokerTokenResponse" in obj)) return result; result.brokerTokenResponse = obj["brokerTokenResponse"]; } return result; } ================================================ FILE: src/itemdb.d ================================================ // What is this module called? module itemdb; // What does this module require to function? import std.datetime; import std.exception; import std.path; import std.string; import std.stdio; import std.algorithm.searching; import core.stdc.stdlib; import std.json; import std.conv; import core.sync.mutex; // What other modules that we have created do we need to import? import sqlite; import util; import log; enum ItemType { none, file, dir, remote, root, unknown } struct Item { string driveId; string id; string name; string remoteName; ItemType type; string eTag; string cTag; SysTime mtime; string parentId; string quickXorHash; string sha256Hash; string remoteDriveId; string remoteParentId; string remoteId; ItemType remoteType; string syncStatus; string size; string relocDriveId; string relocParentId; } // Construct an Item DB struct from a JSON driveItem Item makeDatabaseItem(JSONValue driveItem) { Item item = { id: driveItem["id"].str, name: "name" in driveItem ? driveItem["name"].str : null, // name may be missing for deleted files in OneDrive Business eTag: "eTag" in driveItem ? driveItem["eTag"].str : null, // eTag is not returned for the root in OneDrive Business cTag: "cTag" in driveItem ? driveItem["cTag"].str : null, // cTag is missing in old files (and all folders in OneDrive Business) remoteName: "actualOnlineName" in driveItem ? driveItem["actualOnlineName"].str : null, // actualOnlineName is only used with OneDrive Business Shared Folders }; // OneDrive API Change: https://github.com/OneDrive/onedrive-api-docs/issues/834 // OneDrive no longer returns lastModifiedDateTime if the item is deleted by OneDrive if(isItemDeleted(driveItem)) { // Set mtime to SysTime(0) item.mtime = SysTime(0); } else { // Item is not in a deleted state string lastModifiedTimestamp; // Resolve 'Key not found: fileSystemInfo' when then item is a remote item // https://github.com/abraunegg/onedrive/issues/11 if (isItemRemote(driveItem)) { // remoteItem is a OneDrive object that exists on a 'different' OneDrive drive id, when compared to account default // Normally, the 'remoteItem' field will contain 'fileSystemInfo' however, if the user uses the 'Add Shortcut ..' option in OneDrive WebUI // to create a 'link', this object, whilst remote, does not have 'fileSystemInfo' in the expected place, thus leading to a application crash // See: https://github.com/abraunegg/onedrive/issues/1533 if ("fileSystemInfo" in driveItem["remoteItem"]) { // 'fileSystemInfo' is in 'remoteItem' which will be the majority of cases lastModifiedTimestamp = strip(driveItem["remoteItem"]["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp item.mtime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to Clock.currTime(UTC()) to ensure we have a valid UTC value item.mtime = Clock.currTime(UTC()); } } else { // is a remote item, but 'fileSystemInfo' is missing from 'remoteItem' lastModifiedTimestamp = strip(driveItem["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp item.mtime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to Clock.currTime(UTC()) to ensure we have a valid UTC value item.mtime = Clock.currTime(UTC()); } } } else { // Does fileSystemInfo exist at all ? if ("fileSystemInfo" in driveItem) { // fileSystemInfo exists lastModifiedTimestamp = strip(driveItem["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp item.mtime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to Clock.currTime(UTC()) to ensure we have a valid UTC value item.mtime = Clock.currTime(UTC()); } } else { // no timestamp from JSON file addLogEntry("WARNING: No timestamp provided by the Microsoft OneDrive API - using current system time for item!"); // Set mtime to Clock.currTime(UTC()) to ensure we have a valid UTC value item.mtime = Clock.currTime(UTC()); } } } // Set this item object type bool typeSet = false; if (isItemFile(driveItem)) { // 'file' object exists in the JSON if (debugLogging) {addLogEntry("Flagging database item.type as a file", ["debug"]);} typeSet = true; item.type = ItemType.file; } if (isItemFolder(driveItem)) { // 'folder' object exists in the JSON if (debugLogging) {addLogEntry("Flagging database item.type as a directory", ["debug"]);} typeSet = true; item.type = ItemType.dir; } if (isItemRemote(driveItem)) { // 'remote' object exists in the JSON if (debugLogging) {addLogEntry("Flagging database item.type as a remote", ["debug"]);} typeSet = true; item.type = ItemType.remote; } // root and remote items do not have parentReference if (!isItemRoot(driveItem) && ("parentReference" in driveItem) != null) { item.driveId = driveItem["parentReference"]["driveId"].str; if (hasParentReferenceId(driveItem)) { item.parentId = driveItem["parentReference"]["id"].str; } } // extract the file hash and file size if (isItemFile(driveItem) && ("hashes" in driveItem["file"])) { // Get file size if (hasFileSize(driveItem)) { item.size = to!string(driveItem["size"].integer); // Get quickXorHash as default if ("quickXorHash" in driveItem["file"]["hashes"]) { item.quickXorHash = driveItem["file"]["hashes"]["quickXorHash"].str; } else { if (debugLogging) {addLogEntry("quickXorHash is missing from " ~ driveItem["id"].str, ["debug"]);} } // If quickXorHash is empty .. if (item.quickXorHash.empty) { // Is there a sha256Hash? if ("sha256Hash" in driveItem["file"]["hashes"]) { item.sha256Hash = driveItem["file"]["hashes"]["sha256Hash"].str; } else { if (debugLogging) {addLogEntry("sha256Hash is missing from " ~ driveItem["id"].str, ["debug"]);} } } } else { // So that we have at least a zero value here as the API provided no 'size' data for this file item item.size = "0"; } } // Is the object a remote drive item - living on another driveId ? if (isItemRemote(driveItem)) { // Check and assign remoteDriveId if ("parentReference" in driveItem["remoteItem"] && "driveId" in driveItem["remoteItem"]["parentReference"]) { item.remoteDriveId = driveItem["remoteItem"]["parentReference"]["driveId"].str; } // Check and assign remoteParentId if ("parentReference" in driveItem["remoteItem"] && "id" in driveItem["remoteItem"]["parentReference"]) { item.remoteParentId = driveItem["remoteItem"]["parentReference"]["id"].str; } // Check and assign remoteId if ("id" in driveItem["remoteItem"]) { item.remoteId = driveItem["remoteItem"]["id"].str; } // Check and assign remoteType if ("file" in driveItem["remoteItem"].object) { item.remoteType = ItemType.file; } else { item.remoteType = ItemType.dir; } } // We have 4 different operational modes where 'item.syncStatus' is used to flag if an item is synced or not: // - National Cloud Deployments do not support /delta as a query // - When using --single-directory // - When using --download-only --cleanup-local-files // - Are we scanning a Shared Folder // // Thus we need to track in the database that this item is in sync // As we are making an item, set the syncStatus to Y // ONLY when either of the three modes above are being used, all the existing DB entries will get set to N // so when processing /children, it can be identified what the 'deleted' difference is item.syncStatus = "Y"; // Return the created item return item; } final class ItemDatabase { // increment this for every change in the db schema immutable int itemDatabaseVersion = 18; Database db; string insertItemStmt; string updateItemStmt; string deleteOrphanItemStmt; string selectItemByIdStmt; string selectItemByRemoteIdStmt; string selectItemByRemoteDriveIdStmt; string selectItemByParentIdStmt; string selectRemoteTypeByNameStmt; string selectRemoteTypeByRemoteDriveIdStmt; string deleteItemByIdStmt; bool databaseInitialised = false; private Mutex databaseLock; this(string filename) { // Initialise the mutex databaseLock = new Mutex(); db = Database(filename); int dbVersion; try { dbVersion = db.getVersion(); } catch (SqliteException exception) { // An error was generated - what was the error? // - database is locked if (exception.msg == "database is locked" || exception.errorCode == 5) { addLogEntry(); addLogEntry("ERROR: The 'onedrive' application is already running - please check system process list for active application instances" , ["info", "notify"]); addLogEntry(" - Use 'sudo ps aufxw | grep onedrive' to potentially determine active running process"); addLogEntry(); } else { // A different error .. detail the message, detail the actual SQLite Error Code to assist with troubleshooting addLogEntry(); addLogEntry("ERROR: An internal database error occurred: " ~ exception.msg ~ " (SQLite Error Code: " ~ to!string(exception.errorCode) ~ ")"); addLogEntry(); // Give the user some additional information and pointers on this error // The below list is based on user issue / discussion reports since 2018 switch (exception.errorCode) { case 7: // SQLITE_NOMEM addLogEntry("The operation could not be completed due to insufficient memory. Please close unnecessary applications to free up memory and try again.", ["info", "notify"]); break; case 10: // SQLITE_IOERR addLogEntry("A disk I/O error occurred. This could be due to issues with the storage medium (e.g., disk full, hardware failure, filesystem corruption).\nPlease check your disk's health using a disk utility tool, ensure there is enough free space, and check the filesystem for errors.", ["info", "notify"]); break; case 11: // SQLITE_CORRUPT addLogEntry("The database file appears to be corrupt. This could be due to incomplete or failed writes, hardware issues, or unexpected interruptions during database operations.\nPlease perform a --resync operation.", ["info", "notify"]); break; case 14: // SQLITE_CANTOPEN addLogEntry("The database file could not be opened. Please check that the database file exists, has the correct permissions, and is not being blocked by another process or security software.", ["info", "notify"]); break; case 26: // SQLITE_NOTADB addLogEntry("The database file that attempted to be opened does not appear to be a valid SQLite database, or it may have been corrupted to a point where it's no longer recognisable.\nPlease check your application configuration directory and/or perform a --resync operation.", ["info", "notify"]); break; default: addLogEntry("An unexpected error occurred. Please consult the application documentation or request support to resolve this issue.", ["info", "notify"]); break; } // Blank line before exit addLogEntry(); } return; } if (dbVersion == 0) { createTable(); } else if (db.getVersion() != itemDatabaseVersion) { addLogEntry("The item database is incompatible, re-creating database table structures"); db.dropTableIfExists("item"); // Check and drop table if it exists createTable(); } // What is the threadsafe value auto threadsafeValue = db.getThreadsafeValue(); if (debugLogging) {addLogEntry("SQLite Threadsafe database value: " ~ to!string(threadsafeValue), ["debug"]);} try { // Set the enforcement of foreign key constraints. // https://www.sqlite.org/pragma.html#pragma_foreign_keys // PRAGMA foreign_keys = boolean; db.exec("PRAGMA foreign_keys = TRUE;"); // Set the recursive trigger capability // https://www.sqlite.org/pragma.html#pragma_recursive_triggers // PRAGMA recursive_triggers = boolean; db.exec("PRAGMA recursive_triggers = TRUE;"); // Set the journal mode for databases associated with the current connection // https://www.sqlite.org/pragma.html#pragma_journal_mode db.exec("PRAGMA journal_mode = WAL;"); // Only checkpoint if WAL grows past a certain size db.exec("PRAGMA wal_autocheckpoint = 1000;"); // Automatic indexing is enabled by default as of version 3.7.17 // https://www.sqlite.org/pragma.html#pragma_automatic_index // PRAGMA automatic_index = boolean; db.exec("PRAGMA automatic_index = FALSE;"); // Tell SQLite to store temporary tables in memory. This will speed up many read operations that rely on temporary tables, indices, and views. // https://www.sqlite.org/pragma.html#pragma_temp_store db.exec("PRAGMA temp_store = MEMORY;"); // Tell SQlite to cleanup database table size // https://www.sqlite.org/pragma.html#pragma_auto_vacuum // PRAGMA schema.auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL; db.exec("PRAGMA auto_vacuum = FULL;"); // This pragma sets or queries the database connection locking-mode. The locking-mode is either NORMAL or EXCLUSIVE. // https://www.sqlite.org/pragma.html#pragma_locking_mode // PRAGMA schema.locking_mode = NORMAL | EXCLUSIVE db.exec("PRAGMA locking_mode = EXCLUSIVE;"); // The synchronous setting determines how carefully SQLite writes data to disk, balancing between performance and data safety. // https://sqlite.org/pragma.html#pragma_synchronous // PRAGMA synchronous = 0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA; db.exec("PRAGMA synchronous=FULL;"); // Leave this as FULL, this is the sqlite default, ensure this is set to FULL } catch (SqliteException exception) { detailSQLErrorMessage(exception); } insertItemStmt = " INSERT OR REPLACE INTO item (driveId, id, name, remoteName, type, eTag, cTag, mtime, parentId, quickXorHash, sha256Hash, remoteDriveId, remoteParentId, remoteId, remoteType, syncStatus, size, relocDriveId, relocParentId) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19) "; updateItemStmt = " UPDATE item SET name = ?3, remoteName = ?4, type = ?5, eTag = ?6, cTag = ?7, mtime = ?8, parentId = ?9, quickXorHash = ?10, sha256Hash = ?11, remoteDriveId = ?12, remoteParentId = ?13, remoteId = ?14, remoteType = ?15, syncStatus = ?16, size = ?17, relocDriveId = ?18, relocParentId = ?19 WHERE driveId = ?1 AND id = ?2 "; deleteOrphanItemStmt = " DELETE FROM item WHERE driveId = ?1 AND parentId = ?9 AND name = ?3 "; selectItemByIdStmt = " SELECT * FROM item WHERE driveId = ?1 AND id = ?2 "; selectItemByRemoteIdStmt = " SELECT * FROM item WHERE remoteDriveId = ?1 AND remoteId = ?2 "; selectItemByRemoteDriveIdStmt = " SELECT * FROM item WHERE remoteDriveId = ?1 "; selectRemoteTypeByNameStmt = " SELECT * FROM item WHERE type = 'remote' AND name = ?1 "; selectRemoteTypeByRemoteDriveIdStmt = " SELECT * FROM item WHERE type = 'remote' AND remoteDriveId = ?1 AND remoteId = ?2 "; selectItemByParentIdStmt = "SELECT * FROM item WHERE driveId = ? AND parentId = ?"; deleteItemByIdStmt = "DELETE FROM item WHERE driveId = ? AND id = ?"; // flag that the database is accessible and we have control databaseInitialised = true; } ~this() { closeDatabaseFile(); } bool isDatabaseInitialised() { return databaseInitialised; } void closeDatabaseFile() { if (databaseInitialised) { db.close(); } databaseInitialised = false; } void createTable() { db.exec("CREATE TABLE item ( driveId TEXT NOT NULL, id TEXT NOT NULL, name TEXT NOT NULL, remoteName TEXT, type TEXT NOT NULL, eTag TEXT, cTag TEXT, mtime TEXT NOT NULL, parentId TEXT, quickXorHash TEXT, sha256Hash TEXT, remoteDriveId TEXT, remoteParentId TEXT, remoteId TEXT, remoteType TEXT, deltaLink TEXT, syncStatus TEXT, size TEXT, relocDriveId TEXT, relocParentId TEXT, PRIMARY KEY (driveId, id), FOREIGN KEY (driveId, parentId) REFERENCES item (driveId, id) ON DELETE CASCADE ON UPDATE RESTRICT )"); db.exec("CREATE INDEX name_idx ON item (name)"); db.exec("CREATE INDEX remote_idx ON item (remoteDriveId, remoteId)"); db.exec("CREATE INDEX item_children_idx ON item (driveId, parentId)"); db.exec("CREATE INDEX selectByPath_idx ON item (name, driveId, parentId)"); db.setVersion(itemDatabaseVersion); } void detailSQLErrorMessage(SqliteException exception) { addLogEntry(); addLogEntry("A database statement execution error occurred: " ~ exception.msg); addLogEntry(); switch (exception.errorCode) { case 7: // SQLITE_FULL case 8: // SQLITE_READONLY case 10: // SQLITE_SCHEMA case 11: // SQLITE_CORRUPT case 17: // SQLITE_IOERR case 21: // SQLITE_NOMEM case 22: // SQLITE_MISUSE case 26: // SQLITE_NOTADB case 27: // SQLITE_CANTOPEN addLogEntry("Fatal SQLite error encountered. Error code: " ~ to!string(exception.errorCode), ["info", "notify"]); addLogEntry(); // Must exit here forceExit(); // This line is needed, even though the application technically never gets here .. // - Error: switch case fallthrough - use 'goto default;' if intended goto default; default: addLogEntry("Please restart the application with --resync to potentially fix any local database issues."); // Handle non-fatal errors or continue execution break; } } void insert(const ref Item item) { synchronized(databaseLock) { auto p = db.prepare(insertItemStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { bindItem(item, p); p.exec(); } catch (SqliteException exception) { detailSQLErrorMessage(exception); } } } void update(const ref Item item) { synchronized(databaseLock) { auto p = db.prepare(updateItemStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { bindItem(item, p); p.exec(); } catch (SqliteException exception) { detailSQLErrorMessage(exception); } } } void dump_open_statements() { synchronized(databaseLock) { db.dump_open_statements(); } } int db_checkpoint() { synchronized(databaseLock) { return db.db_checkpoint(); } } void upsert(const ref Item item) { synchronized(databaseLock) { Statement selectStmt = db.prepare("SELECT COUNT(*) FROM item WHERE driveId = ? AND id = ?"); Statement selectParentalStmt = db.prepare("SELECT COUNT(*) FROM item WHERE driveId = ? AND parentId = ? AND name = ?"); Statement executionStmt = Statement.init; // Initialise executionStmt to avoid uninitialised variable usage scope(exit) { selectStmt.finalise(); selectParentalStmt.finalise(); executionStmt.finalise(); } try { if (debugLogging) { addLogEntry("Attempting upsert for item: driveId='" ~ item.driveId ~ "', id='" ~ item.id ~ "', parentId='" ~ item.parentId ~ "', name='" ~ item.name ~ "'", ["debug"]); } selectStmt.bind(1, item.driveId); selectStmt.bind(2, item.id); auto result = selectStmt.exec(); size_t count = result.front[0].to!size_t; // If the existing 'driveId' and 'id' are in the DB, then this is a record to update if (count == 0) { // Item with id not found, check for orphaned entry by parentId and name // - If the user has deleted and recreated the folder online with the same name, whilst we may have an existing entry, this will have the old 'id' selectParentalStmt.bind(1, item.driveId); selectParentalStmt.bind(2, item.parentId); selectParentalStmt.bind(3, item.name); auto orphanResult = selectParentalStmt.exec(); size_t orphanCount = orphanResult.front[0].to!size_t; // Were orphans found? if (orphanCount == 0) { // No match on name+parentId either — new insert if (debugLogging) { addLogEntry("Inserting new item: driveId='" ~ item.driveId ~ "', id='" ~ item.id ~ "', parentId='" ~ item.parentId ~ "', name='" ~ item.name ~ "'", ["debug"]); } executionStmt = db.prepare(insertItemStmt); } else { // Orphans found if (debugLogging) { addLogEntry("Orphan lookup: count=" ~ to!string(orphanCount) ~ " for driveId='" ~ item.driveId ~ "', parentId='" ~ item.parentId ~ "', name='" ~ item.name ~ "'", ["debug"]); addLogEntry("Orphaned DB Entry - deleting old entry for name='" ~ item.name ~ "' and parentId='" ~ item.parentId ~ "'", ["debug"]); } // Orphan exists, delete it first auto deleteOrphan = db.prepare(deleteOrphanItemStmt); deleteOrphan.bind(1, item.driveId); deleteOrphan.bind(2, item.parentId); deleteOrphan.bind(3, item.name); deleteOrphan.exec(); deleteOrphan.finalise(); if (debugLogging) { addLogEntry("Deleted orphaned entry — now inserting new item: id='" ~ item.id ~ "', parentId='" ~ item.parentId ~ "', name='" ~ item.name ~ "'", ["debug"]); } executionStmt = db.prepare(insertItemStmt); } } else { // Found by ID — perform update if (debugLogging) { addLogEntry("Updating existing DB record: id='" ~ item.id ~ "', parentId='" ~ item.parentId ~ "', name='" ~ item.name ~ "'", ["debug"]); } executionStmt = db.prepare(updateItemStmt); } bindItem(item, executionStmt); executionStmt.exec(); } catch (SqliteException exception) { // Handle errors appropriately detailSQLErrorMessage(exception); } } } Item[] selectChildren(const(char)[] driveId, const(char)[] id) { synchronized(databaseLock) { Item[] items; auto p = db.prepare(selectItemByParentIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, driveId); p.bind(2, id); auto res = p.exec(); while (!res.empty) { items ~= buildItem(res); res.step(); } return items; } catch (SqliteException exception) { // Handle errors appropriately detailSQLErrorMessage(exception); items = []; return items; // Return an empty array on error } } } bool selectById(const(char)[] driveId, const(char)[] id, out Item item) { synchronized(databaseLock) { auto p = db.prepare(selectItemByIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, driveId); p.bind(2, id); auto r = p.exec(); if (!r.empty) { item = buildItem(r); return true; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return false; } } bool selectByRemoteId(const(char)[] remoteDriveId, const(char)[] remoteId, out Item item) { synchronized(databaseLock) { auto p = db.prepare(selectItemByRemoteIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, remoteDriveId); p.bind(2, remoteId); auto r = p.exec(); if (!r.empty) { item = buildItem(r); return true; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return false; } } // This should return the 'remote' DB entry for a given remote drive id bool selectByRemoteDriveId(const(char)[] remoteDriveId, out Item item) { synchronized(databaseLock) { auto p = db.prepare(selectItemByRemoteDriveIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, remoteDriveId); auto r = p.exec(); if (!r.empty) { item = buildItem(r); return true; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return false; } } // This should return the 'remote' DB entry for the given 'name' bool selectByRemoteEntryByName(const(char)[] entryName, out Item item) { synchronized(databaseLock) { auto p = db.prepare(selectRemoteTypeByNameStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, entryName); auto r = p.exec(); if (!r.empty) { item = buildItem(r); return true; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return false; } } // This should return the 'remote' DB entry for the given 'remoteDriveId' and 'remoteId' bool selectRemoteTypeByRemoteDriveId(const(char)[] remoteDriveId, const(char)[] remoteId, out Item item) { synchronized(databaseLock) { auto p = db.prepare(selectRemoteTypeByRemoteDriveIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, remoteDriveId); p.bind(2, remoteId); auto r = p.exec(); if (!r.empty) { item = buildItem(r); return true; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return false; } } // returns true if an item id is in the database bool idInLocalDatabase(const(string) driveId, const(string) id) { synchronized(databaseLock) { auto p = db.prepare(selectItemByIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, driveId); p.bind(2, id); auto r = p.exec(); return !r.empty; } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); return false; } } } // returns the item with the given path // the path is relative to the sync directory ex: "./Music/file_name.mp3" bool selectByPath(const(char)[] path, string rootDriveId, out Item item) { synchronized(databaseLock) { Item currItem = { driveId: rootDriveId }; // Issue https://github.com/abraunegg/onedrive/issues/578 path = "root/" ~ (startsWith(path, "./") || path == "." ? path.chompPrefix(".") : path); auto s = db.prepare("SELECT * FROM item WHERE name = ?1 AND driveId IS ?2 AND parentId IS ?3"); scope(exit) s.finalise(); // Ensure that the prepared statement is finalised after execution. try { foreach (name; pathSplitter(path)) { s.bind(1, name); s.bind(2, currItem.driveId); s.bind(3, currItem.id); auto r = s.exec(); if (r.empty) return false; currItem = buildItem(r); // If the item is of type remote, substitute it with the child if (currItem.type == ItemType.remote) { if (debugLogging) {addLogEntry("Record is a Remote Object: " ~ to!string(currItem), ["debug"]);} Item child; if (selectById(currItem.remoteDriveId, currItem.remoteId, child)) { assert(child.type != ItemType.remote, "The type of the child cannot be remote"); currItem = child; if (debugLogging) {addLogEntry("Selecting Record that is NOT Remote Object: " ~ to!string(currItem), ["debug"]);} } } } item = currItem; return true; } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); return false; } } } // same as selectByPath() but it does not traverse remote folders, returns the remote element if that is what is required bool selectByPathIncludingRemoteItems(const(char)[] path, string rootDriveId, out Item item) { synchronized(databaseLock) { Item currItem = { driveId: rootDriveId }; // Issue https://github.com/abraunegg/onedrive/issues/578 path = "root/" ~ (startsWith(path, "./") || path == "." ? path.chompPrefix(".") : path); auto s = db.prepare("SELECT * FROM item WHERE name IS ?1 AND driveId IS ?2 AND parentId IS ?3"); scope(exit) s.finalise(); // Ensure that the prepared statement is finalised after execution. try { foreach (name; pathSplitter(path)) { s.bind(1, name); s.bind(2, currItem.driveId); s.bind(3, currItem.id); auto r = s.exec(); if (r.empty) return false; currItem = buildItem(r); } if (currItem.type == ItemType.remote) { if (debugLogging) {addLogEntry("Record selected is a Remote Object: " ~ to!string(currItem), ["debug"]);} } item = currItem; return true; } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); return false; } } } void deleteById(const(char)[] driveId, const(char)[] id) { synchronized(databaseLock) { auto p = db.prepare(deleteItemByIdStmt); scope(exit) p.finalise(); // Ensure that the prepared statement is finalised after execution. try { p.bind(1, driveId); p.bind(2, id); p.exec(); } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } } } private void bindItem(const ref Item item, ref Statement stmt) { with (stmt) with (item) { bind(1, driveId); bind(2, id); bind(3, name); bind(4, remoteName); // type handling string typeStr = null; final switch (type) with (ItemType) { case file: typeStr = "file"; break; case dir: typeStr = "dir"; break; case remote: typeStr = "remote"; break; case root: typeStr = "root"; break; case unknown: typeStr = "unknown"; break; case none: typeStr = null; break; } bind(5, typeStr); bind(6, eTag); bind(7, cTag); bind(8, mtime.toISOExtString()); bind(9, parentId); bind(10, quickXorHash); bind(11, sha256Hash); bind(12, remoteDriveId); bind(13, remoteParentId); bind(14, remoteId); // remoteType handling string remoteTypeStr = null; final switch (remoteType) with (ItemType) { case file: remoteTypeStr = "file"; break; case dir: remoteTypeStr = "dir"; break; case remote: remoteTypeStr = "remote"; break; case root: remoteTypeStr = "root"; break; case unknown: remoteTypeStr = "unknown"; break; case none: remoteTypeStr = null; break; } bind(15, remoteTypeStr); bind(16, syncStatus); bind(17, size); bind(18, relocDriveId); bind(19, relocParentId); } } private Item buildItem(Statement.Result result) { assert(!result.empty, "The DB result must not be empty"); assert(result.front.length == 20, "The DB result must have 20 columns"); // Check the DB record timestamp entry. Rather than assert(), use forceExit() and exit in a more graceful manner // - empty values // - 2024-11-23T01:16:14\x80Z // - ��Ϣc (#3014) // - ����� (#2876) // - non timestamp formatted strings such as 'CurlEngine curlEngin' (#2813) if (!isValidUTCDateTime(result.front[7].dup)) { addLogEntry(); addLogEntry("FATAL: The DB record mtime entry is not a valid ISO timestamp entry. Please attempt a --resync to fix the local database."); addLogEntry(); // Must force exit here, allow logging to be done forceExit(); } Item item = { // column 0: driveId // column 1: id // column 2: name // column 3: remoteName - only used when there is a difference in the local name & remote shared folder name // column 4: type // column 5: eTag // column 6: cTag // column 7: mtime // column 8: parentId // column 9: quickXorHash // column 10: sha256Hash // column 11: remoteDriveId // column 12: remoteParentId // column 13: remoteId // column 14: remoteType // column 15: deltaLink // column 16: syncStatus // column 17: size // column 18: relocDriveId // column 19: relocParentId driveId: result.front[0].dup, id: result.front[1].dup, name: result.front[2].dup, remoteName: result.front[3].dup, // Column 4 is type - not set here eTag: result.front[5].dup, cTag: result.front[6].dup, mtime: SysTime.fromISOExtString(result.front[7].dup), parentId: result.front[8].dup, quickXorHash: result.front[9].dup, sha256Hash: result.front[10].dup, remoteDriveId: result.front[11].dup, remoteParentId: result.front[12].dup, remoteId: result.front[13].dup, // Column 14 is remoteType - not set here // Column 15 is deltaLink - not set here syncStatus: result.front[16].dup, size: result.front[17].dup, relocDriveId: result.front[18].dup, relocParentId: result.front[19].dup }; // Configure item.type switch (result.front[4]) { case "file": item.type = ItemType.file; break; case "dir": item.type = ItemType.dir; break; case "remote": item.type = ItemType.remote; break; case "root": item.type = ItemType.root; break; default: assert(0, "Invalid item type"); } // Configure item.remoteType switch (result.front[14]) { // We only care about 'dir' and 'file' for 'remote' items case "file": item.remoteType = ItemType.file; break; case "dir": item.remoteType = ItemType.dir; break; default: item.remoteType = ItemType.none; break; // Default to ItemType.none } // Return item return item; } // Computes the relative path of the given item ID as stored in the OneDrive item database. // // The path is reconstructed by traversing the item's parent hierarchy via parentId, // optionally resolving relocation fields (relocDriveId and relocParentId) if present. // The returned path is relative to the configured sync directory, e.g. "Music/Turbo Killer.mp3". // // Behaviour includes: // - Handling normal items and directory structures // - Supporting relocated shared folder roots via relocDriveId and relocParentId // - Skipping inclusion of any item with ItemType.root to avoid adding "root" as a path segment // - Ensuring folders named "root" (with ItemType.dir) are still correctly included // // Note: The returned path does not end with a trailing slash, even for directories. string computePath(const(char)[] driveIdInput, const(char)[] itemIdInput) { synchronized(databaseLock) { assert(driveIdInput && itemIdInput); string path; string driveId = driveIdInput.idup; string id = itemIdInput.idup; Item item; // Remember the highest non-root node we saw in this drive string anchorCandidateDriveId; string anchorCandidateItemId; // DB Statements auto s = db.prepare("SELECT * FROM item WHERE driveId = ?1 AND id = ?2"); auto s2 = db.prepare("SELECT driveId, id FROM item WHERE remoteDriveId = ?1 AND remoteId = ?2"); scope(exit) { s.finalise(); // Ensure that the prepared statement is finalised after execution. s2.finalise(); // Ensure that the prepared statement is finalised after execution. } // Attempt to compute the path based on the elements provided try { while (true) { s.bind(1, driveId); s.bind(2, id); auto r = s.exec(); if (!r.empty) { item = buildItem(r); // Track the highest non-root row we encounter if (item.type != ItemType.root) { anchorCandidateDriveId = driveId; anchorCandidateItemId = item.id; } // Build path: Skip only if name == "root" AND item.type == ItemType.root const bool skipAppend = (item.name == "root") && (item.type == ItemType.root); if (!skipAppend) { if (item.type == ItemType.remote) { // replace first segment with remote name auto idx = indexOf(path, '/'); path = (idx >= 0) ? item.name ~ path[idx .. $] : item.name; } else { path = path.length ? item.name ~ "/" ~ path : item.name; } } // Move up one level (within the same drive) id = item.parentId; // Check for relocation and handle the relocation if (item.type == ItemType.root && item.relocDriveId !is null && item.relocParentId !is null) { driveId = item.relocDriveId; id = item.relocParentId; } } else { // We fell off the top (id == null). Try to jump to the anchor (mount point). if (id == null) { // Use the top-most NON-ROOT we saw, not the root we just processed if (anchorCandidateItemId.length) { s2.bind(1, anchorCandidateDriveId); // remoteDriveId s2.bind(2, anchorCandidateItemId); // remoteId (top-most folder) auto r2 = s2.exec(); if (r2.empty) { break; // no anchor -> done } else { // Jump into the drive that contains the remote mount point driveId = r2.front[0].dup; id = r2.front[1].dup; // loop continues; next iteration will fetch the 'remote' row } } else { // no candidate (single item or broken tree) break; } } else { // broken database tree addLogEntry("The following generated a broken database tree query:"); addLogEntry("Drive ID: " ~ to!string(driveId)); addLogEntry("Item ID: " ~ to!string(id)); assert(0); } } } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } if (path.length == 0) { path = "."; } // Return the computed path return path; } } Item[] selectRemoteItems() { synchronized(databaseLock) { Item[] items; auto stmt = db.prepare("SELECT * FROM item WHERE remoteDriveId IS NOT NULL"); scope (exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { auto res = stmt.exec(); while (!res.empty) { items ~= buildItem(res); res.step(); } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return items; } } string getDeltaLink(const(char)[] driveId, const(char)[] id) { synchronized(databaseLock) { // Log what we received if (debugLogging) { addLogEntry("DeltaLink Query (driveId): " ~ to!string(driveId), ["debug"]); addLogEntry("DeltaLink Query (id): " ~ to!string(id), ["debug"]); } // assert if these are null assert(driveId && id); auto stmt = db.prepare("SELECT deltaLink FROM item WHERE driveId = ?1 AND id = ?2"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { stmt.bind(1, driveId); stmt.bind(2, id); auto res = stmt.exec(); if (res.empty) return null; return res.front[0].dup; } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); return null; } } } void setDeltaLink(const(char)[] driveId, const(char)[] id, const(char)[] deltaLink) { synchronized(databaseLock) { assert(driveId && id); assert(deltaLink); auto stmt = db.prepare("UPDATE item SET deltaLink = ?3 WHERE driveId = ?1 AND id = ?2"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { stmt.bind(1, driveId); stmt.bind(2, id); stmt.bind(3, deltaLink); stmt.exec(); } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } } } // We have 4 different operational modes where 'item.syncStatus' is used to flag if an item is synced or not: // - National Cloud Deployments do not support /delta as a query // - When using --single-directory // - When using --download-only --cleanup-local-files // - Are we scanning a Shared Folder // // As we query /children to get all children from OneDrive, update anything in the database // to be flagged as not-in-sync, thus, we can use that flag to determine what was previously // in-sync, but now deleted on OneDrive void downgradeSyncStatusFlag(const(char)[] driveId, const(char)[] id) { synchronized(databaseLock) { assert(driveId); auto stmt = db.prepare("UPDATE item SET syncStatus = 'N' WHERE driveId = ?1 AND id = ?2"); scope(exit) { stmt.finalise(); // Ensure that the prepared statement is finalised after execution. } try { stmt.bind(1, driveId); stmt.bind(2, id); stmt.exec(); } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } } } // We have 4 different operational modes where 'item.syncStatus' is used to flag if an item is synced or not: // - National Cloud Deployments do not support /delta as a query // - When using --single-directory // - When using --download-only --cleanup-local-files // - Are we scanning a Shared Folder // // Select items that have a out-of-sync flag set Item[] selectOutOfSyncItems(const(char)[] driveId) { synchronized(databaseLock) { assert(driveId); Item[] items; auto stmt = db.prepare("SELECT * FROM item WHERE syncStatus = 'N' AND driveId = ?1"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { stmt.bind(1, driveId); auto res = stmt.exec(); while (!res.empty) { items ~= buildItem(res); res.step(); } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return items; } } // OneDrive Business Folders are stored in the database potentially without a root | parentRoot link // Select items associated with the provided driveId Item[] selectByDriveId(const(char)[] driveId) { synchronized(databaseLock) { assert(driveId); Item[] items; auto stmt = db.prepare("SELECT * FROM item WHERE driveId = ?1 AND parentId IS NULL"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { stmt.bind(1, driveId); auto res = stmt.exec(); while (!res.empty) { items ~= buildItem(res); res.step(); } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return items; } } // Perform a vacuum on the database, commit WAL / SHM to file void performVacuum() { synchronized(databaseLock) { // Log what we are attempting to do addLogEntry("Attempting to perform a database vacuum to optimise database"); try { // Check the current DB Status - we have to be in a clean state here db.checkStatus(); // Are there any open statements that need to be closed? if (db.count_open_statements() > 0) { // Dump open statements db.dump_open_statements(); // dump open statements so we know what the are // SIGINT (CTRL-C), SIGTERM (kill) handling if (exitHandlerTriggered) { // The SQLITE_INTERRUPT result code indicates that an operation was interrupted - which if we have open statements, most likely a SIGINT scenario throw new SqliteException(9, "Open SQL Statements due to interrupted operations"); } else { // Try and close open statements db.close_open_statements(); } } // Ensure there are no pending operations by performing a PASSIVE checkpoint db.exec("PRAGMA wal_checkpoint(PASSIVE);"); // Prepare and execute VACUUM statement Statement stmt = db.prepare("VACUUM;"); scope(exit) stmt.finalise(); // Ensure the statement is finalised when we exit stmt.exec(); addLogEntry("Database vacuum is complete"); } catch (SqliteException exception) { addLogEntry(); addLogEntry("ERROR: Unable to perform a database vacuum: " ~ exception.msg); addLogEntry(); } } } // Perform a checkpoint (either TRUNCATE or PASSIVE) by writing the data into to the database from the WAL file void performCheckpoint(string checkpointType) { synchronized(databaseLock) { // Log what we are attempting to do if (debugLogging) {addLogEntry("Attempting to perform a database checkpoint to merge temporary data", ["debug"]);} try { // Check the current DB Status - we have to be in a clean state here db.checkStatus(); // Are there any open statements that need to be closed? if (db.count_open_statements() > 0) { // Dump open statements db.dump_open_statements(); // dump open statements so we know what the are // SIGINT (CTRL-C), SIGTERM (kill) handling if (exitHandlerTriggered) { // The SQLITE_INTERRUPT result code indicates that an operation was interrupted - which if we have open statements, most likely a SIGINT scenario throw new SqliteException(9, "Open SQL Statements due to interrupted operations"); } else { // Try and close open statements db.close_open_statements(); } } // Ensure there are no pending operations by performing a checkpoint string databaseCommand = format("PRAGMA wal_checkpoint(%s);" , checkpointType); db.exec(databaseCommand); if (debugLogging) {addLogEntry("Database checkpoint is complete", ["debug"]);} } catch (SqliteException exception) { addLogEntry(); addLogEntry("ERROR: Unable to perform a database checkpoint: " ~ exception.msg); addLogEntry(); } } } // Select distinct driveId items from database string[] selectDistinctDriveIds() { synchronized(databaseLock) { string[] driveIdArray; auto stmt = db.prepare("SELECT DISTINCT driveId FROM item;"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { auto res = stmt.exec(); if (res.empty) return driveIdArray; while (!res.empty) { driveIdArray ~= res.front[0].dup; res.step(); } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return driveIdArray; } } // Function to get the total number of rows in a table int getTotalRowCount() { synchronized(databaseLock) { int rowCount = 0; auto stmt = db.prepare("SELECT COUNT(*) FROM item;"); scope(exit) stmt.finalise(); // Ensure that the prepared statement is finalised after execution. try { auto res = stmt.exec(); if (!res.empty) { rowCount = res.front[0].to!int; } } catch (SqliteException exception) { // Handle the error appropriately detailSQLErrorMessage(exception); } return rowCount; } } } ================================================ FILE: src/log.d ================================================ // What is this module called? module log; // What does this module require to function? import std.stdio; import std.file; import std.datetime; import std.concurrency; import std.typecons; import core.sync.mutex; import core.sync.condition; import core.thread; import std.format; import std.string; import std.conv; // What other modules that we have created do we need to import? import util; version(Notifications) { import dnotify; } // Shared Application Logging Level Variables __gshared bool verboseLogging = false; __gshared bool debugLogging = false; __gshared bool debugHTTPSResponse = false; __gshared string microsoftDataCentre; // Private Shared Module Objects private __gshared LogBuffer logBuffer; // Timer for logging private __gshared MonoTime lastInsertedTime; // Is logging active private __gshared bool isRunning; class LogBuffer { private string[3][] buffer; private Mutex bufferLock; private Condition condReady; private string logFilePath; private bool writeToFile; private bool verboseLogging; private bool debugLogging; private Thread flushThread; private bool environmentVariablesAvailable; private bool sendGUINotification; this(bool verboseLogging, bool debugLogging) { // Initialise the mutex bufferLock = new Mutex(); condReady = new Condition(bufferLock); // Initialise shared items isRunning = true; // Initialise other items this.logFilePath = ""; this.writeToFile = false; this.verboseLogging = verboseLogging; this.debugLogging = debugLogging; this.environmentVariablesAvailable = false; this.sendGUINotification = false; this.flushThread = new Thread(&flushBuffer); this.flushThread.isDaemon(true); this.flushThread.start(); } ~this() { if (!isRunning) { if (exitHandlerTriggered) { bufferLock.unlock(); } } } // Terminate Logging void terminateLogging() { synchronized { // join all threads thread_joinAll(); if (!isRunning) { return; // Prevent multiple shutdowns } // flag that we are no longer running due to shutting down isRunning = false; condReady.notifyAll(); // Wake up all waiting threads } // Wait for the flush thread to finish outside of the synchronized block to avoid deadlocks if (flushThread.isRunning()) { flushThread.join(true); } // Flush any remaining logs flushBuffer(); // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed // Exit scopes scope(exit) { if (bufferLock !is null) { bufferLock.lock(); } scope(exit) { if (bufferLock !is null) { bufferLock.unlock(); object.destroy(bufferLock); bufferLock = null; } } } scope(failure) { if (bufferLock !is null) { bufferLock.lock(); } scope(exit) { if (bufferLock !is null) { bufferLock.unlock(); object.destroy(bufferLock); bufferLock = null; } } } } // Flush the logging buffer private void flushBuffer() { while (isRunning) { flush(); } stdout.flush(); } // Add the message received to the buffer for logging void logThisMessage(string message, string[] levels = ["info"]) { // Generate the timestamp for this log entry auto timeStamp = leftJustify(Clock.currTime().toString(), 28, '0'); synchronized(bufferLock) { foreach (level; levels) { // Normal application output if (!debugLogging) { if ((level == "info") || ((verboseLogging) && (level == "verbose")) || (level == "logFileOnly") || (level == "consoleOnly") || (level == "consoleOnlyNoNewLine")) { // Add this message to the buffer, with this format buffer ~= [timeStamp, level, format("%s", message)]; } } else { // Debug Logging (--verbose --verbose | -v -v | -vv) output // Add this message, regardless of 'level' to the buffer, with this format buffer ~= [timeStamp, level, format("DEBUG: %s", message)]; // If there are multiple 'levels' configured, ignore this and break as we are doing debug logging break; } // Submit the message to the dbus / notification daemon for display within the GUI being used // Will not send GUI notifications when running in debug mode if ((!debugLogging) && (level == "notify")) { if (sendGUINotification) { notify(message); } } } // Notify thread to wake up condReady.notify(); } } // Send GUI notification if --enable-notifications as been used at compile time void notify(string message) { // Use dnotify's functionality for GUI notifications, if GUI notification support has been compiled in version(Notifications) { try { auto n = new Notification("OneDrive Client for Linux", message, "dialog-information"); n.show(); } catch (NotificationError e) { addLogEntry("Unable to send notification to the D-Bus message bus daemon, disabling GUI notifications: " ~ e.message); sendGUINotification = false; } } } // Flush the logging buffer private void flush() { string[3][] messages; synchronized(bufferLock) { if (isRunning) { while (buffer.empty && isRunning) { // buffer is empty and logging is still active condReady.wait(); } messages = buffer; buffer.length = 0; } } // Are there messages to process? if (messages.length > 0) { // There are messages to process foreach (msg; messages) { // timestamp, logLevel, message // Always write the log line to the console, if level != logFileOnly if (msg[1] != "logFileOnly") { // Console output .. what sort of output if (msg[1] == "consoleOnlyNoNewLine") { // This is used write out a message to the console only, without a new line // This is used in non-verbose mode to indicate something is happening when downloading JSON data from OneDrive or when we need user input from --resync write(msg[2]); } else { // write this to the console with a new line writeln(msg[2]); } } // Was this just console only output? if ((msg[1] != "consoleOnlyNoNewLine") && (msg[1] != "consoleOnly")) { // Write to the logfile only if configured to do so - console only items should not be written out if (writeToFile) { string logFileLine = format("[%s] %s", msg[0], msg[2]); std.file.append(logFilePath, logFileLine ~ "\n"); } } } // Clear Messages messages.length = 0; } } } // Function to initialise the logging system void initialiseLogging(bool verboseLogging = false, bool debugLogging = false) { logBuffer = new LogBuffer(verboseLogging, debugLogging); lastInsertedTime = MonoTime.currTime(); } // Shutdown Logging void shutdownLogging() { if (logBuffer !is null) { // Terminate logging in a safe manner logBuffer.terminateLogging(); logBuffer = null; } } // Function to add a log entry with multiple levels void addLogEntry(string message = "", string[] levels = ["info"]) { // we can only add a log line if we are running ... if (isRunning) { logBuffer.logThisMessage(message, levels); } } // Is logging still active bool loggingActive() { return isRunning; } // Is logging still initialised bool loggingStillInitialised() { return logBuffer !is null; } void addProcessingLogHeaderEntry(string message, long verbosityCount) { if (verbosityCount == 0) { addLogEntry(message, ["logFileOnly"]); // Use the dots to show the application is 'doing something' if verbosityCount == 0 addLogEntry(message ~ " .", ["consoleOnlyNoNewLine"]); } else { // Fallback to normal logging if in verbose or above level addLogEntry(message); } } // Add a processing '.' to indicate activity void addProcessingDotEntry() { if (MonoTime.currTime() - lastInsertedTime < dur!"seconds"(1)) { // Don't flood the log buffer return; } lastInsertedTime = MonoTime.currTime(); addLogEntry(".", ["consoleOnlyNoNewLine"]); } // Finish processing '.' line output void completeProcessingDots() { addLogEntry(" ", ["consoleOnly"]); } // Function to set logFilePath and enable logging to a file void enableLogFileOutput(string configuredLogFilePath) { logBuffer.logFilePath = configuredLogFilePath; logBuffer.writeToFile = true; } // Flag that the environment variables exists so if logging is compiled in, it can be enabled void flagEnvironmentVariablesAvailable(bool variablesAvailable) { logBuffer.environmentVariablesAvailable = variablesAvailable; } // Disable GUI Notifications void disableGUINotifications(bool userConfigDisableNotifications) { logBuffer.sendGUINotification = userConfigDisableNotifications; } // Validate that if GUI Notification support has been compiled in using --enable-notifications, the DBUS Server is actually usable void validateDBUSServerAvailability() { version(Notifications) { if (logBuffer.environmentVariablesAvailable) { auto serverAvailable = dnotify.check_availability(); if (!serverAvailable) { addLogEntry("WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled"); logBuffer.sendGUINotification = false; } else { addLogEntry("D-Bus message bus daemon is available; GUI notifications are now enabled"); if (debugLogging) {addLogEntry("D-Bus message bus daemon server details: " ~ to!string(dnotify.get_server_info()), ["debug"]);} logBuffer.sendGUINotification = true; } } else { addLogEntry("WARNING: The required environment variables to enable GUI Notifications are not available; GUI notifications are disabled"); logBuffer.sendGUINotification = false; } } } ================================================ FILE: src/main.d ================================================ // What is this module called? module main; // What does this module require to function? import core.memory; import core.stdc.stdlib: EXIT_SUCCESS, EXIT_FAILURE, exit; import core.sys.posix.signal; import core.thread; import core.time; import std.algorithm; import std.concurrency; import std.conv; import std.datetime; import std.file; import std.getopt; import std.net.curl: CurlException; import std.parallelism; import std.path; import std.process; import std.socket: SocketException; import std.stdio; import std.string; import std.traits; // What other modules that we have created do we need to import? import config; import log; import curlEngine; import util; import onedrive; import syncEngine; import itemdb; import clientSideFiltering; import monitor; import webhook; import intune; import socketio; // What other constant variables do we require? const int EXIT_RESYNC_REQUIRED = 126; // Class objects ApplicationConfig appConfig; OneDriveWebhook oneDriveWebhook; SyncEngine syncEngineInstance; ItemDatabase itemDB; ClientSideFiltering selectiveSync; Monitor filesystemMonitor; OneDriveSocketIo oneDriveSocketIo; // Class variables // Flag for performing a synchronised shutdown bool shutdownInProgress = false; // Flag if a --dry-run is being performed, as, on shutdown, once config is destroyed, we have no reference here bool dryRun = false; // Configure the runtime database file path so that it is available to us on shutdown so objects can be destroyed and removed if required // - Typically this will be the default, but in a --dry-run scenario, we use a separate database file string runtimeDatabaseFile = ""; // Flag for if we are performing filesystem monitoring bool performFileSystemMonitoring = false; // Flag for if we perform a database vacuum. This gets set to false if we have not performed a 'no-sync' task bool performDatabaseVacuum = true; // Flag if SIGTERM is used bool sigtermHandlerTriggered = false; int main(string[] cliArgs) { // Application Start Time - used during monitor loop to detail how long it has been running for auto applicationStartTime = Clock.currTime(); // Disable buffering on stdout - this is needed so that when we are using plain write() it will go to the terminal without flushing stdout.setvbuf(0, _IONBF); // Required main function variables string genericHelpMessage = "Please use 'onedrive --help' for further assistance in regards to running this application."; // If the user passes in --confdir we need to store this as a variable string confdirOption = ""; // running as what user? string runtimeUserName = ""; // Are we online? bool online = false; // Does the operating environment have shell environment variables set bool shellEnvSet = false; // What is the runtime synchronisation directory that will be used // Typically this will be '~/OneDrive' .. however tilde expansion is unreliable string runtimeSyncDirectory = ""; // Verbosity Logging Count - this defines if verbose or debug logging is being used long verbosityCount = 0; // Monitor loop failures bool monitorFailures = false; // Help requested bool helpRequested = false; // Did the user specify --sync or --monitor bool syncOrMonitorMissing = false; // Was a no-sync type operation requested bool noSyncTaskOperationRequested = false; // DEVELOPER OPTIONS OUTPUT VARIABLES bool displayMemoryUsage = false; bool displaySyncOptions = false; // Application Version immutable string applicationVersion = "onedrive " ~ strip(import("version")); // Define 'exit' and 'failure' scopes scope(exit) { // Detail what scope was called if (debugLogging) {addLogEntry("Exit scope was called", ["debug"]);} // Perform synchronised exit performSynchronisedExitProcess("exitScope"); // Setup signal handling for the exit scope setupExitScopeSignalHandler(); } scope(failure) { // Detail what scope was called if (debugLogging) {addLogEntry("Failure scope was called", ["debug"]);} // Perform synchronised exit performSynchronisedExitProcess("failureScope"); // Setup signal handling for the exit scope setupExitScopeSignalHandler(); } // Read in application options as passed in try { bool printVersion = false; auto cliOptions = getopt( cliArgs, std.getopt.config.passThrough, std.getopt.config.bundling, std.getopt.config.caseSensitive, "confdir", "Set the directory used to store the configuration files", &confdirOption, "verbose|v+", "Print more details, useful for debugging (repeat for extra debugging)", &verbosityCount, "version", "Print the version and exit", &printVersion ); // Print help and exit if (cliOptions.helpWanted) { cliArgs ~= "--help"; helpRequested = true; } // Print the version and exit if (printVersion) { writeln(applicationVersion); exit(EXIT_SUCCESS); } } catch (GetOptException e) { // Option errors writeln(e.msg); writeln(genericHelpMessage); return EXIT_FAILURE; } catch (Exception e) { // Generic error writeln(e.msg); writeln(genericHelpMessage); return EXIT_FAILURE; } // Determine the application logging verbosity // - As these flags are used to reduce application processing when not required, specifically in a 'debug' scenario, both verboseLogging and debugLogging need to be enabled if (verbosityCount == 1) { verboseLogging = true;} // set __gshared bool verboseLogging in log.d if (verbosityCount >= 2) { verboseLogging = true; debugLogging = true;} // set __gshared bool verboseLogging & debugLogging in log.d // Initialize the application logging class, as we know the application verbosity level // If we need to enable logging to a file, we can only do this once we know the application configuration which is done slightly later on initialiseLogging(verboseLogging, debugLogging); // Log application start time, log line has start time if (debugLogging) {addLogEntry("Application started", ["debug"]);} // Who are we running as? This will print the ProcessID, UID, GID and username the application is running as runtimeUserName = getUserName(); // Print the application version and how this was compiled as soon as possible if (debugLogging) { addLogEntry("Application Version: " ~ applicationVersion, ["debug"]); addLogEntry("Application Compiled With: " ~ compilerDetails(), ["debug"]); // How was this application started - what options were passed in addLogEntry("Passed in 'cliArgs': " ~ to!string(cliArgs), ["debug"]); addLogEntry("Note: --confdir and --verbose are not listed in 'cliArgs' array", ["debug"]); addLogEntry("Passed in --confdir if present: " ~ confdirOption, ["debug"]); addLogEntry("Passed in --verbose count if present: " ~ to!string(verbosityCount), ["debug"]); } // Create a new AppConfig object with default values, appConfig = new ApplicationConfig(); // Update the default application configuration with the verbosity count so this can be used throughout the application as needed appConfig.verbosityCount = verbosityCount; // Initialise the application configuration, utilising --confdir if it was passed in // Otherwise application defaults will be used to configure the application if (!appConfig.initialise(confdirOption, helpRequested)) { // There was an error loading the user specified application configuration // Error message already printed return EXIT_FAILURE; } // Update the current runtime application configuration (default or 'config' file read in options) from any passed in command line arguments appConfig.updateFromArgs(cliArgs); // Set the default thread pool value based on configuration or maximum logical CPUs setDefaultApplicationThreads(); // If --debug-https has been used, set the applicable flag debugHTTPSResponse = appConfig.getValueBool("debug_https"); // set __gshared bool debugHTTPSResponse in log.d now that we have read-in any CLI arguments // Read in the configured 'sync_dir' from appConfig with '~' if present correctly expanded based on the user environment runtimeSyncDirectory = appConfig.initialiseRuntimeSyncDirectory(); // Are we doing a --sync or a --monitor operation? Both of these will be false if they are not set if ((!appConfig.getValueBool("synchronize")) && (!appConfig.getValueBool("monitor"))) { syncOrMonitorMissing = true; // --sync or --monitor is missing } // Has the client been configured to use Intune SSO via Microsoft Identity Broker (microsoft-identity-broker) dbus session // This is ONLY possible on Linux, not FreeBSD or other platforms version (linux) { if (appConfig.getValueBool("use_intune_sso")) { // The client is configured to use Intune SSO via Microsoft Identity Broker dbus session addLogEntry("Client has been configured to use Intune SSO via Microsoft Identity Broker dbus session - checking usage criteria"); // We need to check that the available dbus is actually available if(wait_for_broker()) { // Usage criteria met, will attempt to use Intune SSO via dbus addLogEntry("Intune SSO via Microsoft Identity Broker dbus session usage criteria met - will attempt to authenticate via Intune"); } else { // Microsoft Identity Broker dbus is not available addLogEntry(); addLogEntry("Required Microsoft Identity Broker dbus capability not found - disabling authentication via Intune SSO"); addLogEntry(); appConfig.setValueBool("use_intune_sso" , false); } } } else { // Ensure 'use_intune_sso' is disabled appConfig.setValueBool("use_intune_sso" , false); } // Has the user configured to use the 'Recycle Bin' locally, for any files that are deleted online? if (appConfig.getValueBool("use_recycle_bin")) { // Configure the internal application paths which will be used to move rather than delete any online deletes to appConfig.setRecycleBinPaths(); // If we are not using --display-config, test if the Recycle Bin Paths exist on the file system if (!appConfig.getValueBool("display_config")) { // We need to test that the configured 'Recycle Bin' path is not within the configured 'sync_dir' if (appConfig.checkRecycleBinPathAsChildOfSyncDir) { // ERROR: 'Recycle Bin' path is a child of the configured 'sync_dir' addLogEntry(); addLogEntry("ERROR: The configured 'recycle_bin_path' (" ~ appConfig.recycleBinParentPath ~ ") is located within the configured 'sync_dir' (" ~ appConfig.runtimeSyncDirectory ~ ").", ["info", "notify"]); addLogEntry(" This would cause locally recycled items to be re-uploaded to Microsoft OneDrive."); addLogEntry(" Please set 'recycle_bin_path' to a location outside of 'sync_dir' and restart the client."); addLogEntry(); return EXIT_FAILURE; } else { // 'Recycle Bin' path is not within the configured 'sync_dir' // We need to ensure that the Recycle Bin Paths exist on the file system, and if they do not exist, create them // Test for appConfig.recycleBinFilePath if (!exists(appConfig.recycleBinFilePath)) { try { // Attempt to create the 'Recycle Bin' file path we have been configured with mkdirRecurse(appConfig.recycleBinFilePath); // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ appConfig.recycleBinFilePath, ["debug"]);} appConfig.recycleBinFilePath.setAttributes(octal!700); // Set to 0700 as Trash may contain sensitive and is the expected default permissions by GIO or KIO } catch (std.file.FileException e) { // Creating the 'Recycle Bin' file path failed addLogEntry("ERROR: Unable to create the configured local 'Recycle Bin' file directory: " ~ e.msg, ["info", "notify"]); // Use exit scopes to shutdown API return EXIT_FAILURE; } } // Test for appConfig.recycleBinInfoPath if (!exists(appConfig.recycleBinInfoPath)) { try { // Attempt to create the 'Recycle Bin' info path we have been configured with mkdirRecurse(appConfig.recycleBinInfoPath); // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ appConfig.recycleBinInfoPath, ["debug"]);} appConfig.recycleBinInfoPath.setAttributes(octal!700); // Set to 0700 as Trash may contain sensitive and is the expected default permissions by GIO or KIO } catch (std.file.FileException e) { // Creating the 'Recycle Bin' info path failed addLogEntry("ERROR: Unable to create the configured local 'Recycle Bin' info directory: " ~ e.msg, ["info", "notify"]); // Use exit scopes to shutdown API return EXIT_FAILURE; } } } } } // Are we performing some sort of 'no-sync' operation task? noSyncTaskOperationRequested = appConfig.hasNoSyncOperationBeenRequested(); // returns true if we are // If 'syncOrMonitorMissing' is true and 'noSyncTaskOperationRequested' is false (meaning we are not doing some 'no-sync' operation like '--display-sync-status', '--get-sharepoint-drive-id' or '--display-config' // - fail fast here to avoid setting up all the other components, database, initialising the API as this is all pointless if we just fail out later // If we are not using --display-config, perform this check if (!appConfig.getValueBool("display_config")) { if (syncOrMonitorMissing && !noSyncTaskOperationRequested) { // Before failing fast, has the client been authenticated and does the 'refresh_token' contain data if (exists(appConfig.refreshTokenFilePath) && getSize(appConfig.refreshTokenFilePath) > 0) { // fail fast - print error message that --sync or --monitor are missing printMissingOperationalSwitchesError(); // Use exit scopes to shutdown API return EXIT_FAILURE; } } } // If --disable-notifications has not been used, check if everything exists to enable notifications if (!appConfig.getValueBool("disable_notifications")) { // If notifications was compiled in, we need to ensure that these variables are actually available before we enable GUI Notifications flagEnvironmentVariablesAvailable(appConfig.validateGUINotificationEnvironmentVariables()); // If we are not using --display-config attempt to enable GUI notifications if (!appConfig.getValueBool("display_config")) { // Attempt to enable GUI Notifications validateDBUSServerAvailability(); } } // cURL Version Compatibility Test // - Common warning for cURL version issue string distributionWarning = " Please report this to your distribution, requesting an update to a newer cURL version, or consider upgrading it yourself for optimal stability."; // If 'force_http_11' = false, we need to check the curl version being used if (!appConfig.getValueBool("force_http_11")) { // get the curl version string curlVersion = getCurlVersionNumeric(); // Is the version of curl or libcurl being used by the platform a known bad curl version for HTTP/2 support if (isBadCurlVersion(curlVersion)) { // add warning message string curlWarningMessage = format("WARNING: Your cURL/libcurl version (%s) has known HTTP/2 bugs that impact the use of this client.", curlVersion); addLogEntry(); addLogEntry(curlWarningMessage, ["info", "notify"]); addLogEntry(distributionWarning); addLogEntry(" Downgrading all client operations to use HTTP/1.1 to ensure maximum operational stability."); addLogEntry(" Please read https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl for more information."); addLogEntry(); appConfig.setValueBool("force_http_11" , true); } } else { // get the curl version - a bad curl version may still be in use string curlVersion = getCurlVersionNumeric(); // Is the version of curl or libcurl being used by the platform a known bad curl version if (isBadCurlVersion(curlVersion)) { // add warning message string curlWarningMessage = format("WARNING: Your cURL/libcurl version (%s) has known operational bugs that impact the use of this client.", curlVersion); addLogEntry(); addLogEntry(curlWarningMessage); // curl HTTP/1.1 downgrade in place meaning user took steps to remediate, perform standard logging with no GUI notification addLogEntry(distributionWarning); addLogEntry(); } } // In a debug scenario, to assist with understanding the run-time configuration, ensure this flag is set if (debugLogging) { appConfig.setValueBool("display_running_config", true); } // Configure dryRun so that this can be used here & during shutdown dryRun = appConfig.getValueBool("dry_run"); // As early as possible, now re-configure the logging class, given that we have read in any applicable 'config' file and updated the application running config from CLI input: // - Enable logging to a file if this is required // - Disable GUI notifications if this has been configured // Configure application logging to a log file only if this has been enabled // This is the earliest point that this can be done, as the client configuration has been read in, and any CLI arguments have been processed. // Either of those ('config' file, CLI arguments) could be enabling logging, thus this is the earliest point at which this can be validated and enabled. // The buffered logging also ensures that all 'output' to this point is also captured and written out to the log file if (appConfig.getValueBool("enable_logging")) { // Calculate the application logging directory string calculatedLogDirPath = appConfig.calculateLogDirectory(); string calculatedLogFilePath; // Initialise using the configured logging directory if (verboseLogging) {addLogEntry("Using the following path to store the runtime application log: " ~ calculatedLogDirPath, ["verbose"]);} // Calculate the logfile name if (calculatedLogDirPath != appConfig.defaultHomePath) { // Log file is not going to the home directory string logfileName = runtimeUserName ~ ".onedrive.log"; calculatedLogFilePath = buildNormalizedPath(buildPath(calculatedLogDirPath, logfileName)); } else { // Log file is going to the users home directory calculatedLogFilePath = buildNormalizedPath(buildPath(calculatedLogDirPath, "onedrive.log")); } // Update the logging class to use 'calculatedLogFilePath' for the application log file now that this has been determined enableLogFileOutput(calculatedLogFilePath); } // Disable GUI Notifications if configured to do so // - This option is reverse action. If 'disable_notifications' is 'true', we need to send 'false' if (appConfig.getValueBool("disable_notifications")) { // disable_notifications is true, ensure GUI notifications is initialised with false so that NO GUI notification is sent disableGUINotifications(false); addLogEntry("Disabling GUI notifications as per user configuration"); } // Perform a deprecated options check now that the config file (if present) and CLI options have all been parsed to advise the user that their option usage might change appConfig.checkDeprecatedOptions(cliArgs); // Configure Client Side Filtering (selective sync) by parsing and getting a usable regex for skip_file, skip_dir and sync_list config components selectiveSync = new ClientSideFiltering(appConfig); if (!selectiveSync.initialise()) { // exit here as something triggered a selective sync configuration failure return EXIT_FAILURE; } // Set runtimeDatabaseFile, this will get updated if we are using --dry-run runtimeDatabaseFile = appConfig.databaseFilePath; // DEVELOPER OPTIONS OUTPUT // Set to display memory details as early as possible displayMemoryUsage = appConfig.getValueBool("display_memory"); // set to display sync options displaySyncOptions = appConfig.getValueBool("display_sync_options"); // Display the current application configuration (based on all defaults, 'config' file parsing and/or options passed in via the CLI) and exit if --display-config has been used if ((appConfig.getValueBool("display_config")) || (appConfig.getValueBool("display_running_config"))) { // Display the application configuration appConfig.displayApplicationConfiguration(); // Do we exit? We exit only if '--display-config' has been used if (appConfig.getValueBool("display_config")) { return EXIT_SUCCESS; } } // Check for basic application option conflicts - flags that should not be used together and/or flag combinations that conflict with each other, values that should be present and are not if (appConfig.checkForBasicOptionConflicts) { // Any error will have been printed by the function itself, but we need a small delay here to allow the buffered logging to output any error return EXIT_FAILURE; } // Check for --dry-run operation or a 'no-sync' operation where the 'dry-run' DB copy should be used // If this has been requested, we need to ensure that all actions are performed against the dry-run database copy, and, // no actual action takes place - such as deleting files if deleted online, moving files if moved online or local, downloading new & changed files, uploading new & changed files if (dryRun || (noSyncTaskOperationRequested)) { // Cleanup any existing dry-run elements ... these should never be left hanging around and should be cleaned up first cleanupDatabaseFiles(appConfig.databaseFilePathDryRun); // If --dry-run if (dryRun) { // This is a --dry-run operation addLogEntry("DRY-RUN Configured. Output below shows what 'would' have occurred."); // Make a copy of the original items.sqlite3 for use as the dry run copy if it exists if (exists(appConfig.databaseFilePath)) { // In a --dry-run --resync scenario, we should not copy the existing database file if (!appConfig.getValueBool("resync")) { // Copy the existing DB file to the dry-run copy addLogEntry("DRY-RUN: Copying items.sqlite3 to items-dryrun.sqlite3 to use for dry run operations"); copy(appConfig.databaseFilePath,appConfig.databaseFilePathDryRun); } else { // No database copy due to --resync - an empty DB file will be used for the resync operation addLogEntry("DRY-RUN: No database copy created for --dry-run due to --resync also being used"); } } // update runtimeDatabaseFile now that we are using the dry run path runtimeDatabaseFile = appConfig.databaseFilePathDryRun; } } else { // Cleanup any existing dry-run elements ... these should never be left hanging around cleanupDatabaseFiles(appConfig.databaseFilePathDryRun); } // Handle --logout as separate item, do not 'resync' on a --logout if (appConfig.getValueBool("logout")) { if (debugLogging) {addLogEntry("--logout requested", ["debug"]);} addLogEntry("Deleting the saved authentication status ..."); if (!dryRun) { // Remove the 'refresh_token' file if present safeRemove(appConfig.refreshTokenFilePath); // Remove the 'intune_account' file if present safeRemove(appConfig.intuneAccountDetailsFilePath); } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not removing the saved authentication status"); } // Exit return EXIT_SUCCESS; } // Handle --reauth to re-authenticate the client if (appConfig.getValueBool("reauth")) { if (debugLogging) {addLogEntry("--reauth requested", ["debug"]);} addLogEntry("Deleting the saved authentication status ... re-authentication requested"); if (!dryRun) { // Remove the 'refresh_token' file if present safeRemove(appConfig.refreshTokenFilePath); // Remove the 'intune_account' file if present safeRemove(appConfig.intuneAccountDetailsFilePath); } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not removing the saved authentication status"); } } // --resync should be considered a 'last resort item' or if the application configuration has changed, where a resync is needed .. the user needs to 'accept' this warning to proceed // If --resync has not been used (bool value is false), check the application configuration for 'changes' that require a --resync to ensure that the data locally reflects the users requested configuration if (appConfig.getValueBool("resync")) { // what is the risk acceptance for --resync? bool resyncRiskAcceptance = appConfig.displayResyncRiskForAcceptance(); if (debugLogging) {addLogEntry("Returned --resync risk acceptance: " ~ to!string(resyncRiskAcceptance), ["debug"]);} // Action based on user response if (!resyncRiskAcceptance){ // --resync risk not accepted return EXIT_FAILURE; } else { if (debugLogging) {addLogEntry("--resync issued and risk accepted", ["debug"]);} // --resync risk accepted, perform a cleanup of items that require a cleanup appConfig.cleanupHashFilesDueToResync(); // Make a backup of the applicable configuration file appConfig.createBackupConfigFile(); // Update hash files and generate a new config backup appConfig.updateHashContentsForConfigFiles(); // Remove the items database processResyncDatabaseRemoval(runtimeDatabaseFile); } } else { // Is the application currently authenticated? If not, it is pointless checking if a --resync is required until the application is authenticated if (exists(appConfig.refreshTokenFilePath)) { // Has any of our application configuration that would require a --resync been changed? if (appConfig.applicationChangeWhereResyncRequired()) { // Application configuration has changed however --resync not issued, fail fast addLogEntry(); addLogEntry("An application configuration change has been detected where a --resync is required", ["info", "notify"]); addLogEntry(); return EXIT_RESYNC_REQUIRED; } else { // No configuration change that requires a --resync to be issued // Special cases need to be checked - if these options were enabled, it creates a false 'Resync Required' flag, so do not create a backup if ((!appConfig.getValueBool("list_business_shared_items"))) { // Make a backup of the applicable configuration file appConfig.createBackupConfigFile(); // Update hash files and generate a new config backup appConfig.updateHashContentsForConfigFiles(); } } } } // Implement https://github.com/abraunegg/onedrive/issues/1129 // Force a synchronisation of a specific folder, only when using --synchronize --single-directory and ignoring all non-default skip_dir and skip_file rules if (appConfig.getValueBool("force_sync")) { // appConfig.checkForBasicOptionConflicts() has already checked for the basic requirements for --force-sync addLogEntry(); addLogEntry("WARNING: Overriding application configuration to use application defaults for skip_dir and skip_file due to --sync --single-directory --force-sync being used"); addLogEntry(); bool forceSyncRiskAcceptance = appConfig.displayForceSyncRiskForAcceptance(); if (debugLogging) {addLogEntry("Returned --force-sync risk acceptance: " ~ forceSyncRiskAcceptance, ["debug"]);} // Action based on user response if (!forceSyncRiskAcceptance){ // --force-sync risk not accepted return EXIT_FAILURE; } else { // --force-sync risk accepted // reset set config using function to use application defaults appConfig.resetSkipToDefaults(); // update sync engine regex with reset defaults selectiveSync.setDirMask(appConfig.getValueString("skip_dir")); selectiveSync.setFileMask(appConfig.getValueString("skip_file")); } } // What IP Protocol are we going to use to access the network with appConfig.displayIPProtocol(); // Test if OneDrive service can be reached, exit if it cant be reached if (debugLogging) {addLogEntry("Testing network to ensure network connectivity to Microsoft OneDrive Service", ["debug"]);} online = testInternetReachability(appConfig); // If we are not 'online' - how do we handle this situation? if (!online) { // We are unable to initialise the OneDrive API as we are not online if (!appConfig.getValueBool("monitor")) { // Running as --synchronize addLogEntry(); addLogEntry("ERROR: Unable to reach the Microsoft OneDrive API service, unable to initialise application"); addLogEntry(); return EXIT_FAILURE; } else { // Running as --monitor addLogEntry(); addLogEntry("Unable to reach the Microsoft OneDrive API service at this point in time, re-trying network tests based on applicable intervals"); addLogEntry(); // Run the re-try of Internet connectivity test online = retryInternetConnectivityTest(appConfig); } } // This needs to be a separate 'if' statement, as, if this was an 'if-else' from above, if we were originally offline and using --monitor, we would never get to this point if (online) { // Check Application Version if (!appConfig.getValueBool("disable_version_check")) { if (verboseLogging) {addLogEntry("Checking Application Version ...", ["verbose"]);} checkApplicationVersion(); } // Initialise the OneDrive API if (verboseLogging) {addLogEntry("Attempting to initialise the OneDrive API ...", ["verbose"]);} OneDriveApi oneDriveApiInstance = new OneDriveApi(appConfig); appConfig.apiWasInitialised = oneDriveApiInstance.initialise(); // Did the API initialise successfully? if (appConfig.apiWasInitialised) { if (verboseLogging) {addLogEntry("The OneDrive API was initialised successfully", ["verbose"]);} // Flag that we were able to initialise the API in the application config oneDriveApiInstance.debugOutputConfiguredAPIItems(); oneDriveApiInstance.releaseCurlEngine(); object.destroy(oneDriveApiInstance); oneDriveApiInstance = null; // Need to configure the itemDB and syncEngineInstance for 'sync' and 'non-sync' operations if (verboseLogging) {addLogEntry("Opening the item database ...", ["verbose"]);} // Configure the Item Database itemDB = new ItemDatabase(runtimeDatabaseFile); // Was the database successfully initialised? if (!itemDB.isDatabaseInitialised()) { // no .. destroy class itemDB = null; // exit application return EXIT_FAILURE; } // Initialise the syncEngine syncEngineInstance = new SyncEngine(appConfig, itemDB, selectiveSync); appConfig.syncEngineWasInitialised = syncEngineInstance.initialise(); // Are we not doing a --sync or a --monitor operation? if (syncOrMonitorMissing) { // this is 'true' if --sync or a --monitor were not used // Do not perform a vacuum on exit, pointless performDatabaseVacuum = false; // Are we performing some sort of 'no-sync' task? // - Are we obtaining the Office 365 Drive ID for a given Office 365 SharePoint Shared Library? // - Are we displaying the sync status? // - Are we getting the URL for a file online? // - Are we listing who modified a file last online? // - Are we listing OneDrive Business Shared Items? // - Are we creating a shareable link for an existing file on OneDrive? // - Are we just creating a directory online, without any sync being performed? // - Are we just deleting a directory online, without any sync being performed? // - Are we renaming or moving a directory? // - Are we displaying the quota information? // - Did we just authorise the client? // --get-sharepoint-drive-id - Get the SharePoint Library drive_id if (appConfig.getValueString("sharepoint_library_name") != "") { // Get the SharePoint Library drive_id syncEngineInstance.querySiteCollectionForDriveID(appConfig.getValueString("sharepoint_library_name")); // Exit application // Use exit scopes to shutdown API and cleanup data return EXIT_SUCCESS; } // --display-sync-status - Query the sync status if (appConfig.getValueBool("display_sync_status")) { // path to query variable string pathToQueryStatusOn; // What path do we query? if (!appConfig.getValueString("single_directory").empty) { pathToQueryStatusOn = "/" ~ appConfig.getValueString("single_directory"); } else { pathToQueryStatusOn = "/"; } // Query the sync status syncEngineInstance.queryOneDriveForSyncStatus(pathToQueryStatusOn); // Exit application // Use exit scopes to shutdown API and cleanup data return EXIT_SUCCESS; } // --get-file-link - Get the URL path for a synced file if (appConfig.getValueString("get_file_link") != "") { // Query the OneDrive API for the file link syncEngineInstance.queryOneDriveForFileDetails(appConfig.getValueString("get_file_link"), runtimeSyncDirectory, "URL"); // Exit application // Use exit scopes to shutdown API and cleanup data return EXIT_SUCCESS; } // --modified-by - Get listing the modified-by details of a provided path if (appConfig.getValueString("modified_by") != "") { // Query the OneDrive API for the last modified by details syncEngineInstance.queryOneDriveForFileDetails(appConfig.getValueString("modified_by"), runtimeSyncDirectory, "ModifiedBy"); // Exit application // Use exit scopes to shutdown API and cleanup data return EXIT_SUCCESS; } // --list-shared-items - Get listing OneDrive Business Shared Items if (appConfig.getValueBool("list_business_shared_items")) { // Is this a business account type? if (appConfig.accountType == "business") { // List OneDrive Business Shared Items syncEngineInstance.listBusinessSharedObjects(); } else { addLogEntry("ERROR: Unsupported account type for listing OneDrive Business Shared Items"); } // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // --create-share-link - Create a shareable link for an existing file, based on the local path if (appConfig.getValueString("create_share_link") != "") { // Query OneDrive for the file, and if valid, create a shareable link for the file // By default, the shareable link will be read-only. // If the user adds: // --with-editing-perms // this will create a writeable link syncEngineInstance.queryOneDriveForFileDetails(appConfig.getValueString("create_share_link"), runtimeSyncDirectory, "ShareableLink"); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // --create-directory - Are we just creating a directory online, without any sync being performed? if ((appConfig.getValueString("create_directory") != "")) { // Handle the remote path creation and updating of the local database without performing a sync syncEngineInstance.createDirectoryOnline(appConfig.getValueString("create_directory")); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // --remove-directory - Are we just deleting a directory online, without any sync being performed? if ((appConfig.getValueString("remove_directory") != "")) { // Handle the remote path deletion without performing a sync syncEngineInstance.deleteByPathNoSync(appConfig.getValueString("remove_directory")); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // Are we renaming or moving a directory online? // onedrive --source-directory 'path/as/source/' --destination-directory 'path/as/destination' if ((appConfig.getValueString("source_directory") != "") && (appConfig.getValueString("destination_directory") != "")) { // We are renaming or moving a directory syncEngineInstance.moveOrRenameDirectoryOnline(appConfig.getValueString("source_directory"), appConfig.getValueString("destination_directory")); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // --display-quota - Are we displaying the quota information? if (appConfig.getValueBool("display_quota")) { // Query and respond with the quota details syncEngineInstance.queryOneDriveForQuotaDetails(); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // --download-file - Are we downloading a single file from Microsoft OneDrive if ((appConfig.getValueString("download_single_file") != "")) { // Handle downloading the single file syncEngineInstance.downloadSingleFile(appConfig.getValueString("download_single_file")); // Exit application // Use exit scopes to shutdown API return EXIT_SUCCESS; } // If we get to this point, we have not performed a 'no-sync' task .. // Did we just authorise the client? if (appConfig.applicationAuthoriseResponseURIReceived) { // Authorisation activity if (exists(appConfig.refreshTokenFilePath)) { // OneDrive refresh token exists addLogEntry(); addLogEntry("The application has been successfully authorised, but no extra command options have been specified."); addLogEntry(); addLogEntry(genericHelpMessage); addLogEntry(); // Use exit scopes to shutdown API return EXIT_SUCCESS; } else { // We just authorised, but refresh_token does not exist .. probably an auth error? addLogEntry(); addLogEntry("Your application's authorisation was unsuccessful. Please review your URI response entry, then attempt authorisation again with a new URI response."); addLogEntry(); // Use exit scopes to shutdown API return EXIT_FAILURE; } } else { // No authorisation activity - print error message printMissingOperationalSwitchesError(); // Use exit scopes to shutdown API return EXIT_FAILURE; } } } else { // API could not be initialised addLogEntry("The OneDrive API could not be initialised"); return EXIT_FAILURE; } } // Configure the sync directory based on the runtimeSyncDirectory configured directory if (verboseLogging) {addLogEntry("All application operations will be performed in the configured local 'sync_dir' directory: " ~ runtimeSyncDirectory, ["verbose"]);} // Try and set the 'sync_dir', attempt to create if it does not exist try { if (!exists(runtimeSyncDirectory)) { if (debugLogging) {addLogEntry("runtimeSyncDirectory: Configured 'sync_dir' is missing locally. Creating: " ~ runtimeSyncDirectory, ["debug"]);} // At this point 'sync_dir' is missing and we have requested to create it // However ... 'itemDB' is pointing to a valid database file // If this database has any entries, an empty 'sync_dir' will cause the application to think that all content in 'sync_dir' has been deleted // In this scenario, the application, depending on the options being used, may attempt to delete all files online - which is not desirable // Do a sanity check here to ensure that there are no database entries if (itemDB.getTotalRowCount() == 1) { // Technically an 'empty database' // An empty database will just have 1 row in it, that row being the account 'root' data added when the API is initially initialised above try { // Attempt to create the sync dir we have been configured with mkdirRecurse(runtimeSyncDirectory); // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ runtimeSyncDirectory, ["debug"]);} runtimeSyncDirectory.setAttributes(appConfig.returnRequiredDirectoryPermissions()); } catch (std.file.FileException e) { // Creating the sync directory failed addLogEntry("ERROR: Unable to create the configured local 'sync_dir' directory: " ~ e.msg, ["info", "notify"]); // Use exit scopes to shutdown API return EXIT_FAILURE; } } else { // Not an empty database addLogEntry(); addLogEntry("An application cache state issue has been detected where a --resync is required", ["info", "notify"]); addLogEntry(); return EXIT_RESYNC_REQUIRED; } } } catch (std.file.FileException e) { // Creating the sync directory failed addLogEntry("ERROR: Unable to test for the existence of the configured local 'sync_dir' directory: " ~ e.msg); // Use exit scopes to shutdown API return EXIT_FAILURE; } // Try and change to the working directory to the 'sync_dir' as configured try { chdir(runtimeSyncDirectory); // A FileSystem exception was thrown when attempting to change to the configured 'sync_dir' } catch (FileException e) { // Log error message addLogEntry("FATAL: Unable to change to the configured local 'sync_dir' directory: " ~ runtimeSyncDirectory); // A file system exception was generated displayFileSystemErrorMessage(e.msg, strip(getFunctionName!({})), runtimeSyncDirectory, FsErrorSeverity.fatal); // Use exit scopes to shutdown API as if we are unable to change to the 'sync_dir' we need to exit return EXIT_FAILURE; } // Do we need to validate the runtimeSyncDirectory to check for the presence of a '.nosync' file checkForNoMountScenario(); // Is the sync engine initialised correctly? if (appConfig.syncEngineWasInitialised) { // Configure some initial variables string singleDirectoryPath; string localPath = "."; string remotePath = "/"; // If not performing a --resync, check if there are interrupted downloads and/or uploads that need to be completed if (!appConfig.getValueBool("resync")) { // Check if there are any downloads that need to be resumed if (syncEngineInstance.checkForResumableDownloads) { // Need to re-process the the 'resumable data' to resume the download addLogEntry("There are interrupted downloads that need to be resumed ..."); // Process the resumable download files syncEngineInstance.processResumableDownloadFiles(); } // Check if there are interrupted upload session(s) if (syncEngineInstance.checkForInterruptedSessionUploads) { // Need to re-process the session upload files to resume the failed session uploads addLogEntry("There are interrupted session uploads that need to be resumed ..."); // Process the session upload files syncEngineInstance.processInterruptedSessionUploads(); } } else { // Clean up any downloads that were due to be resumed, but will not be resumed due to --resync being used syncEngineInstance.clearInterruptedDownloads(); // Clean up any uploads that were due to be resumed, but will not be resumed due to --resync being used syncEngineInstance.clearInterruptedSessionUploads(); } // Are we doing a single directory operation (--single-directory) ? if (!appConfig.getValueString("single_directory").empty) { // Ensure that the value stored for appConfig.getValueString("single_directory") does not contain any extra quotation marks string originalSingleDirectoryValue = appConfig.getValueString("single_directory"); // Strip quotation marks from provided path to ensure no issues within a Docker environment when using passed in values string updatedSingleDirectoryValue = strip(originalSingleDirectoryValue, "\""); // Set singleDirectoryPath singleDirectoryPath = updatedSingleDirectoryValue; // Ensure that this is a normalised relative path to runtimeSyncDirectory string normalisedRelativePath = replace(buildNormalizedPath(absolutePath(singleDirectoryPath)), buildNormalizedPath(absolutePath(runtimeSyncDirectory)), "." ); // The user provided a directory to sync within the configured 'sync_dir' path // This also validates if the path being used exists online and/or does not have a 'case-insensitive match' syncEngineInstance.setSingleDirectoryScope(normalisedRelativePath); // Does the directory we want to sync actually exist locally? if (!exists(singleDirectoryPath)) { // The requested path to use with --single-directory does not exist locally within the configured 'sync_dir' addLogEntry("WARNING: The requested path for --single-directory does not exist locally. Creating requested path within " ~ runtimeSyncDirectory, ["info", "notify"]); // Attempt path creation try { // Attempt to create the required --single-directory path locally mkdirRecurse(singleDirectoryPath); // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ singleDirectoryPath, ["debug"]);} singleDirectoryPath.setAttributes(appConfig.returnRequiredDirectoryPermissions()); } catch (std.file.FileException e) { // Creating the sync directory failed addLogEntry("ERROR: Unable to create the required --single-directory path: " ~ e.msg, ["info", "notify"]); // Use exit scopes to shutdown API return EXIT_FAILURE; } } // Update the paths that we use to perform the sync actions localPath = singleDirectoryPath; remotePath = singleDirectoryPath; // Display that we are syncing from a specific path due to --single-directory if (verboseLogging) {addLogEntry("Syncing changes from this selected path: " ~ singleDirectoryPath, ["verbose"]);} } // Handle SIGINT, SIGTERM and SIGSEGV signals setupSignalHandler(); // Are we doing a --sync operation? This includes doing any --single-directory operations if (appConfig.getValueBool("synchronize")) { // We are not using this, so destroy it early object.destroy(filesystemMonitor); filesystemMonitor = null; // Did the user specify --upload-only? if (appConfig.getValueBool("upload_only")) { // Perform the --upload-only sync process performUploadOnlySyncProcess(localPath); } // Did the user specify --download-only? if (appConfig.getValueBool("download_only")) { // Only download data from OneDrive syncEngineInstance.syncOneDriveAccountToLocalDisk(); // Perform the DB consistency check // This will also delete any out-of-sync flagged items if configured to do so syncEngineInstance.performDatabaseConsistencyAndIntegrityCheck(); // Do we cleanup local files? // - Deletes of data from online will already have been performed, but what we are now doing is searching the local filesystem // for any new data locally, that usually would be uploaded to OneDrive, but instead, because of the options being // used, will need to be deleted from the local filesystem if (appConfig.getValueBool("cleanup_local_files")) { // Perform the filesystem walk syncEngineInstance.scanLocalFilesystemPathForNewData(localPath); } } // If no use of --upload-only or --download-only if ((!appConfig.getValueBool("upload_only")) && (!appConfig.getValueBool("download_only"))) { // Perform the standard sync process performStandardSyncProcess(localPath); } // Detail the outcome of the sync process displaySyncOutcome(); } // Are we doing a --monitor operation? if (appConfig.getValueBool("monitor")) { // Update the flag given we are running with --monitor performFileSystemMonitoring = true; // Set initial variable for when we last uploaded something or made an online change from a local inotify event lastLocalWrite = MonoTime.currTime() - dur!"hours"(24); // Is Display Manager Integration enabled? if (appConfig.getValueBool("display_manager_integration")) { // Attempt to configure the desktop integration whilst the client is running in --monitor mode attemptFileManagerIntegration(); } // If 'webhooks' are enabled, this is going to conflict with 'websockets' if the OS cURL library supports websockets if (appConfig.getValueBool("webhook_enabled") && appConfig.curlSupportsWebSockets) { // We have to disable 'websocket' support addLogEntry(); addLogEntry("WARNING: WebSocket support has been disabled because Webhooks are already configured to monitor Microsoft Graph API changes."); addLogEntry(" Only one API notification method can be active at a time."); addLogEntry(); // Set the flag that this will not be used appConfig.curlSupportsWebSockets = false; } else { // Double check scenario, this time 'false' checking 'webhook_enabled' if ((!appConfig.getValueBool("webhook_enabled")) && (appConfig.curlSupportsWebSockets)) { // If we are doing --upload-only however .. we need to 'ignore' online change if (!appConfig.getValueBool("upload_only")) { // Did the user configure to disable 'websocket' support? if (!appConfig.getValueBool("disable_websocket_support")) { // Log that we are attempting to enable WebSocket Support addLogEntry("Attempting to enable WebSocket support to monitor Microsoft Graph API changes in near real-time."); // Obtain the WebSocket Notification URL from the API endpoint syncEngineInstance.obtainWebSocketNotificationURL(); // Were we able to correctly obtain the endpoint response and build the socket.io WS endpoint if (appConfig.websocketNotificationUrlAvailable) { // Notification URL is available if (oneDriveSocketIo is null) { oneDriveSocketIo = new OneDriveSocketIo(thisTid, appConfig); oneDriveSocketIo.start(); } addLogEntry("Enabled WebSocket support to monitor Microsoft Graph API changes in near real-time."); } else { addLogEntry("ERROR: Unable to configure WebSocket support to monitor Microsoft Graph API changes in near real-time."); if (debugLogging) {addLogEntry("Setting 'disable_websocket_support' to 'true' to force WebSockets to be disabled.", ["debug"]);} appConfig.setValueBool("disable_websocket_support" , true); } } else { // WebSocket Support has been disabled addLogEntry("WebSocket support has been disabled by user configuration."); } } else { // --upload only being used addLogEntry("Online changes will not be monitored by WebSocket support due to --upload-only"); // Set the flag that this will not be used appConfig.curlSupportsWebSockets = false; } } } // What are the current values for the platform we are running on string maxOpenFilesSoft = strip(to!string(getSoftOpenFilesLimit())); string maxOpenFilesHard = strip(to!string(getHardOpenFilesLimit())); // What is the currently configured maximum inotify watches that can be used string maxInotifyWatches = strip(getMaxInotifyWatches()); // Start the monitor process addLogEntry("OneDrive synchronisation interval (seconds): " ~ to!string(appConfig.getValueLong("monitor_interval"))); // If we are in a --download-only method of operation, the output of these is not required if (!appConfig.getValueBool("download_only")) { if (verboseLogging) { addLogEntry("Maximum allowed open files (soft): " ~ maxOpenFilesSoft, ["verbose"]); addLogEntry("Maximum allowed open files (hard): " ~ maxOpenFilesHard, ["verbose"]); addLogEntry("Maximum allowed inotify user watches: " ~ maxInotifyWatches, ["verbose"]); } } // Configure the monitor class filesystemMonitor = new Monitor(appConfig, selectiveSync); // Delegated function for when inotify detects a new local directory has been created filesystemMonitor.onDirCreated = delegate(string path) { // Handle .folder creation if skip_dotfiles is enabled if ((appConfig.getValueBool("skip_dotfiles")) && (isDotFile(path))) { if (verboseLogging) {addLogEntry("[M] Skipping watching local path - .folder found & --skip-dot-files enabled: " ~ path, ["verbose"]);} } else { if (verboseLogging) {addLogEntry("[M] Local directory created: " ~ path, ["verbose"]);} try { syncEngineInstance.scanLocalFilesystemPathForNewData(path); markLocalWrite(); } catch (CurlException e) { if (verboseLogging) {addLogEntry("Offline, cannot create remote dir: " ~ path, ["verbose"]);} } catch (Exception e) { addLogEntry("Cannot create remote directory: " ~ e.msg, ["info", "notify"]); } } }; // Delegated function for when inotify detects a local file has been changed filesystemMonitor.onFileChanged = delegate(string[] changedLocalFilesToUploadToOneDrive) { // Handle a potentially locally changed file // Logging for this event moved to handleLocalFileTrigger() due to threading and false triggers from scanLocalFilesystemPathForNewData() above syncEngineInstance.handleLocalFileTrigger(changedLocalFilesToUploadToOneDrive); markLocalWrite(); if (verboseLogging) {addLogEntry("[M] Total number of local file(s) added or changed: " ~ to!string(changedLocalFilesToUploadToOneDrive.length), ["verbose"]);} }; // Delegated function for when inotify detects a delete event filesystemMonitor.onDelete = delegate(string path) { if (verboseLogging) {addLogEntry("[M] Local item deleted: " ~ path, ["verbose"]);} try { // The path has been deleted .. we cannot use isDir or isFile to advise what was deleted. This is the best we can Do addLogEntry("The operating system sent a deletion notification. Trying to delete this item as requested: " ~ path); // perform the delete action syncEngineInstance.deleteByPath(path); markLocalWrite(); } catch (CurlException e) { if (verboseLogging) {addLogEntry("Offline, cannot delete item: " ~ path, ["verbose"]);} } catch (SyncException e) { if (e.msg == "The item to delete is not in the local database") { if (verboseLogging) {addLogEntry("Item cannot be deleted from Microsoft OneDrive because it was not found in the local database", ["verbose"]);} } else { addLogEntry("Cannot delete remote item: " ~ e.msg, ["info", "notify"]); } } catch (FileException e) { // Path is gone locally, log and continue. addLogEntry("ERROR: The local file system returned an error with the following message: " ~ e.msg, ["verbose"]); } catch (Exception e) { addLogEntry("Cannot delete remote item: " ~ e.msg, ["info", "notify"]); } }; // Delegated function for when inotify detects a move event filesystemMonitor.onMove = delegate(string from, string to) { if (verboseLogging) {addLogEntry("[M] Local item moved: " ~ from ~ " -> " ~ to, ["verbose"]);} try { // Handle .folder -> folder if skip_dotfiles is enabled if ((appConfig.getValueBool("skip_dotfiles")) && (isDotFile(from))) { // .folder -> folder handling - has to be handled as a new folder syncEngineInstance.scanLocalFilesystemPathForNewData(to); } else { syncEngineInstance.uploadMoveItem(from, to); } markLocalWrite(); } catch (CurlException e) { if (verboseLogging) {addLogEntry("Offline, cannot move item !", ["verbose"]);} } catch (Exception e) { addLogEntry("Cannot move item: " ~ e.msg, ["info", "notify"]); } }; // Initialise the local filesystem monitor class using inotify to monitor for local filesystem changes // If we are in a --download-only method of operation, we do not enable local filesystem monitoring if (!appConfig.getValueBool("download_only")) { // Not using --download-only try { addLogEntry("Initialising filesystem inotify monitoring ...", ["info", "notify"]); filesystemMonitor.initialise(); addLogEntry("Performing initial synchronisation to ensure consistent local state ..."); } catch (MonitorException e) { // monitor class initialisation failed addLogEntry("ERROR: " ~ e.msg); return EXIT_FAILURE; } } // Filesystem monitor loop variables // Immutables immutable auto checkOnlineInterval = dur!"seconds"(appConfig.getValueLong("monitor_interval")); immutable auto githubCheckInterval = dur!"seconds"(86400); immutable auto localEchoDebounce = dur!"seconds"(10); immutable ulong fullScanFrequency = appConfig.getValueLong("monitor_fullscan_frequency"); immutable ulong logOutputSuppressionInterval = appConfig.getValueLong("monitor_log_frequency"); immutable bool webhookEnabled = appConfig.getValueBool("webhook_enabled"); immutable string loopStartOutputMessage = "################################################## NEW LOOP ##################################################"; immutable string loopStopOutputMessage = "################################################ LOOP COMPLETE ###############################################"; // Changeable variables ulong monitorLoopFullCount = 0; ulong fullScanFrequencyLoopCount = 0; ulong monitorLogOutputLoopCount = 0; MonoTime lastCheckTime = MonoTime.currTime(); MonoTime lastGitHubCheckTime = MonoTime.currTime(); while (performFileSystemMonitoring) { // Do we need to validate the runtimeSyncDirectory to check for the presence of a '.nosync' file - the disk may have been ejected .. checkForNoMountScenario(); // If we are in a --download-only method of operation, there is no filesystem monitoring, so no inotify events to check if (!appConfig.getValueBool("download_only")) { // Process any inotify events processInotifyEvents(true); } // WebSocket and Webhook Notification Handling bool notificationReceived = false; // If we are doing --upload-only however .. we need to 'ignore' online change if (!appConfig.getValueBool("upload_only")) { // Check for notifications pushed from Microsoft to the webhook if (webhookEnabled) { // Create a subscription on the first run, or renew the subscription // on subsequent runs when it is about to expire. if (oneDriveWebhook is null) { oneDriveWebhook = new OneDriveWebhook(thisTid, appConfig); oneDriveWebhook.serve(); } else { oneDriveWebhook.createOrRenewSubscription(); } } else { // WebSocket support is enabled by default, but only if the version of libcurl supports it if (appConfig.curlSupportsWebSockets) { // Did the user configure to disable 'websocket' support? if (!appConfig.getValueBool("disable_websocket_support")) { // Do we need to renew the notification URL? auto renewEarly = dur!"seconds"(120); if (appConfig.websocketNotificationUrlAvailable && appConfig.websocketUrlExpiry.length) { auto expiry = SysTime.fromISOExtString(appConfig.websocketUrlExpiry); auto now = Clock.currTime(UTC()); if (expiry - now <= renewEarly) { try { // Obtain the WebSocket Notification URL from the API endpoint syncEngineInstance.obtainWebSocketNotificationURL(); if (debugLogging) addLogEntry("Refreshed WebSocket notification URL prior to expiry", ["debug"]); } catch (Exception e) { if (debugLogging) addLogEntry("Failed to refresh WebSocket notification URL: " ~ e.msg, ["debug"]); } } } } } } } // Get the current time this loop is starting auto currentTime = MonoTime.currTime(); // Do we perform a sync with OneDrive? if ((currentTime - lastCheckTime >= checkOnlineInterval) || (monitorLoopFullCount == 0)) { // Increment relevant counters monitorLoopFullCount++; fullScanFrequencyLoopCount++; monitorLogOutputLoopCount++; // If full scan at a specific frequency enabled? if (fullScanFrequency > 0) { // Full Scan set for some 'frequency' - do we flag to perform a full scan of the online data? if (fullScanFrequencyLoopCount > fullScanFrequency) { // set full scan trigger for true up if (debugLogging) {addLogEntry("Enabling Full Scan True Up (fullScanFrequencyLoopCount > fullScanFrequency), resetting fullScanFrequencyLoopCount = 1", ["debug"]);} fullScanFrequencyLoopCount = 1; appConfig.fullScanTrueUpRequired = true; } else { // unset full scan trigger for true up if (debugLogging) {addLogEntry("Disabling Full Scan True Up", ["debug"]);} appConfig.fullScanTrueUpRequired = false; } } else { // No it is disabled - ensure this is false appConfig.fullScanTrueUpRequired = false; } // Loop Start if (debugLogging) { addLogEntry(loopStartOutputMessage, ["debug"]); addLogEntry("Total Run-Time Loop Number: " ~ to!string(monitorLoopFullCount), ["debug"]); addLogEntry("Full Scan Frequency Loop Number: " ~ to!string(fullScanFrequencyLoopCount), ["debug"]); } SysTime startFunctionProcessingTime = Clock.currTime(); if (debugLogging) {addLogEntry("Start Monitor Loop Time: " ~ to!string(startFunctionProcessingTime), ["debug"]);} // Do we perform any monitor console logging output suppression? // 'monitor_log_frequency' controls how often, in a non-verbose application output mode, how often // the full output of what is occurring is done. This is done to lessen the 'verbosity' of non-verbose // logging, but only when running in --monitor if (monitorLogOutputLoopCount > logOutputSuppressionInterval) { // re-enable the logging output as required monitorLogOutputLoopCount = 1; if (debugLogging) {addLogEntry("Allowing initial sync log output", ["debug"]);} appConfig.suppressLoggingOutput = false; } else { // do we suppress the logging output to absolute minimal if (monitorLoopFullCount == 1) { // application startup with --monitor if (debugLogging) {addLogEntry("Allowing initial sync log output", ["debug"]);} appConfig.suppressLoggingOutput = false; } else { // only suppress if we are not doing --verbose or higher if (appConfig.verbosityCount == 0) { if (debugLogging) {addLogEntry("Suppressing --monitor log output", ["debug"]);} appConfig.suppressLoggingOutput = true; } else { if (debugLogging) {addLogEntry("Allowing log output", ["debug"]);} appConfig.suppressLoggingOutput = false; } } } // How long has the application been running for? auto elapsedTime = Clock.currTime() - applicationStartTime; if (debugLogging) {addLogEntry("Application run-time thus far: " ~ to!string(elapsedTime), ["debug"]);} // Need to re-validate that the client is still online for this loop if (testInternetReachability(appConfig)) { // Starting a sync - we are online addLogEntry("Starting a sync with Microsoft OneDrive"); // Attempt to reset syncFailures from any prior loop syncEngineInstance.resetSyncFailures(); // Update cached quota details from online as this may have changed online in the background outside of this application syncEngineInstance.freshenCachedDriveQuotaDetails(); // Did the user specify --upload-only? if (appConfig.getValueBool("upload_only")) { // Perform the --upload-only sync process performUploadOnlySyncProcess(localPath, filesystemMonitor); } else { // Perform the standard sync process performStandardSyncProcess(localPath, filesystemMonitor); } // Handle any new inotify events processInotifyEvents(true); // Detail the outcome of the sync process displaySyncOutcome(); // Cleanup sync process arrays syncEngineInstance.cleanupArrays(); // Write WAL and SHM data to file for this loop and release memory used by in-memory processing if (debugLogging) {addLogEntry("Merge contents of WAL and SHM files into main database file", ["debug"]);} itemDB.performCheckpoint("PASSIVE"); } else { // Not online addLogEntry("Microsoft OneDrive service is not reachable at this time. Will re-try on next sync attempt."); } // Output end of loop processing times SysTime endFunctionProcessingTime = Clock.currTime(); if (debugLogging) { addLogEntry("End Monitor Loop Time: " ~ to!string(endFunctionProcessingTime), ["debug"]); addLogEntry("Elapsed Monitor Loop Processing Time: " ~ to!string((endFunctionProcessingTime - startFunctionProcessingTime)), ["debug"]); } // Release all the curl instances used during this loop // New curl instances will be established on next loop if (debugLogging) {addLogEntry("CurlEngine Pool Size PRE Cleanup: " ~ to!string(curlEnginePoolLength()), ["debug"]);} releaseAllCurlInstances(); // Release all CurlEngine instances if (debugLogging) {addLogEntry("CurlEngine Pool Size POST Cleanup: " ~ to!string(curlEnginePoolLength()) , ["debug"]);} // Display memory details before garbage collection if (displayMemoryUsage) { addLogEntry("Monitor Loop Count: " ~ to!string(monitorLoopFullCount)); // Get the current time in the local timezone auto timeStamp = leftJustify(Clock.currTime().toString(), 28, '0'); addLogEntry("Timestamp: " ~ to!string(timeStamp)); addLogEntry("Application Run Time: " ~ to!string(elapsedTime)); // Display memory stats before GC cleanup displayMemoryUsagePreGC(); } // Perform Garbage Collection GC.collect(); // Return free memory to the OS GC.minimize(); // Display memory details after garbage collection if (displayMemoryUsage) displayMemoryUsagePostGC(); // Log that this loop is complete if (debugLogging) {addLogEntry(loopStopOutputMessage, ["debug"]);} // performSync complete, set lastCheckTime to current time lastCheckTime = MonoTime.currTime(); // Developer break via config option if (appConfig.getValueLong("monitor_max_loop") > 0) { // developer set option to limit --monitor loops if (monitorLoopFullCount == (appConfig.getValueLong("monitor_max_loop"))) { performFileSystemMonitoring = false; addLogEntry("Exiting after " ~ to!string(monitorLoopFullCount) ~ " loops due to developer set option"); } } } if (performFileSystemMonitoring) { auto nextCheckTime = lastCheckTime + checkOnlineInterval; currentTime = MonoTime.currTime(); auto sleepTime = nextCheckTime - currentTime; if (debugLogging) {addLogEntry("Sleep for " ~ to!string(sleepTime), ["debug"]);} if (filesystemMonitor.initialised || webhookEnabled || oneDriveSocketIo !is null) { if (filesystemMonitor.initialised) { // If local monitor is on and is waiting (previous event was not from webhook) // Obsidian Editor has been written in such a way that it is constantly writing each and every keystroke to a file. // Not only is this really bad application behaviour, for this client, this means the application is constantly writing to disk, thus attempting to upload file changes. // Unfortunately Obsidian on Linux does not provide a built-in way to disable atomic saves or switch to a backup-copy method via configuration. if (appConfig.getValueBool("delay_inotify_processing")) { Thread.sleep(dur!("seconds")(to!int(appConfig.getValueLong("inotify_delay")))); } // Start the filesystem monitor (inotify) worker and wait for inotify event if (!notificationReceived) { filesystemMonitor.send(true); } } // Adjust sleepTime based on webhook/websocket only when NOT upload_only if (!appConfig.getValueBool("upload_only")) { if (webhookEnabled) { Duration nextWebhookCheckDuration = oneDriveWebhook.getNextExpirationCheckDuration(); if (nextWebhookCheckDuration < sleepTime) sleepTime = nextWebhookCheckDuration; notificationReceived = false; } else if (oneDriveSocketIo !is null && !appConfig.getValueBool("disable_websocket_support") && appConfig.curlSupportsWebSockets) { Duration nextWebsocketCheckDuration = oneDriveSocketIo.getNextExpirationCheckDuration(); if (nextWebsocketCheckDuration < sleepTime) sleepTime = nextWebsocketCheckDuration; } } // ALWAYS wait for FS worker, but only track webhook/websocket if NOT '--upload-only' int res = 1; bool onlineSignal = false; if (appConfig.getValueBool("upload_only")) { receiveTimeout(sleepTime, (int msg) { res = msg; }); } else { receiveTimeout(sleepTime, (int msg) { res = msg; }, (ulong _) { onlineSignal = true; }); } // Debug logging of worker status if (debugLogging) { addLogEntry("worker status = " ~ to!string(res), ["debug"]); if (!appConfig.getValueBool("upload_only")) { addLogEntry("notificationReceived = " ~ to!string(onlineSignal), ["debug"]); } } // Empirical evidence shows that Microsoft often sends multiple // notifications for one single change, so we need a loop to exhaust // all signals that were queued up by the webhook. The notifications // do not contain any actual changes, and we will always rely do the // delta endpoint to sync to latest. Therefore, only one sync run is // good enough to catch up for multiple notifications. // Only process online notifications if NOT '--upload-only' if (!appConfig.getValueBool("upload_only") && onlineSignal) { int signalCount = 1; while (true) { auto more = receiveTimeout(dur!"seconds"(-1), (ulong _) {}); if (more) { signalCount++; } else { auto now = MonoTime.currTime(); auto sinceLocal = now - lastLocalWrite; if (sinceLocal < localEchoDebounce) { if (debugLogging) { addLogEntry( "Debounced online refresh signal (" ~ to!string(sinceLocal.total!"msecs"()) ~ " ms since local write; threshold " ~ to!string(localEchoDebounce.total!"msecs"()) ~ " ms)", ["debug"] ); } // Ignore this reflection; skip the immediate online scan. // Next push or the regular monitor cadence will pick up genuine remote changes. break; } // Get the signal timestamp - this is as close as possible to when this was received SysTime signalTimeStamp = Clock.currTime(); signalTimeStamp.fracSecs = Duration.zero; // Log what signal we received if (webhookEnabled) { string webhookLogEntry = format("Received %s signal(s) from Webhook handler (%s)", to!string(signalCount), to!string(signalTimeStamp)); addLogEntry(webhookLogEntry); } else { string websocketLogEntry = format("Received %s signal(s) from WebSocket handler (%s)", to!string(signalCount), to!string(signalTimeStamp)); addLogEntry(websocketLogEntry); } // Perform online callback action oneDriveOnlineCallback(); break; } } } // Worker failure remains outside '--upload-only' filter if (res == -1) { addLogEntry("ERROR: Monitor worker failed."); monitorFailures = true; performFileSystemMonitoring = false; } } else { // no hooks available, nothing to check Thread.sleep(sleepTime); } } } } } else { // Exit application as the sync engine could not be initialised addLogEntry("Application Sync Engine could not be initialised correctly"); // Use exit scope return EXIT_FAILURE; } // Exit application using exit scope if (!syncEngineInstance.syncFailures && !monitorFailures) { return EXIT_SUCCESS; } else { return EXIT_FAILURE; } } // Set default application threads void setDefaultApplicationThreads() { // Read in system values int configuredThreads = to!int(appConfig.getValueLong("threads")); int systemCPUs = totalCPUs; // Warning if configuredThreads is too high if (configuredThreads > systemCPUs) { addLogEntry(); addLogEntry("WARNING: Configured 'threads = " ~ to!string(configuredThreads) ~ "' exceeds available CPU cores (" ~ to!string(systemCPUs) ~ ")."); addLogEntry(" This may lead to reduced performance, CPU contention, and instability. For best results, set 'threads' no higher than the number of physical CPU cores."); addLogEntry(); } // Set the default threads based on configured option defaultPoolThreads(configuredThreads); } // Retrieves the maximum inotify watches allowed by the system string getMaxInotifyWatches() { // Predefined Versions // https://dlang.org/spec/version.html#predefined-versions version (linux) { try { // Read max inotify watches from procfs on Linux return strip(readText("/proc/sys/fs/inotify/max_user_watches")); } catch (Exception e) { return "Unknown (Error reading /proc/sys/fs/inotify/max_user_watches)"; } } else version (FreeBSD) { // FreeBSD uses kqueue instead of inotify, no direct equivalent return "N/A (uses kqueue)"; } else version (OpenBSD) { // OpenBSD uses kqueue instead of inotify, no direct equivalent return "N/A (uses kqueue)"; } else { return "Unsupported platform"; } } // Print error message when --sync or --monitor has not been used and no valid 'no-sync' operation was requested void printMissingOperationalSwitchesError() { // notify the user that --sync or --monitor were missing addLogEntry(); addLogEntry("Your command line input is missing either the '--sync' or '--monitor' switches. Please include one (but not both) of these switches in your command line, or refer to 'onedrive --help' for additional guidance."); addLogEntry(); addLogEntry("It is important to note that you must include one of these two arguments in your command line for the application to perform a synchronisation with Microsoft OneDrive"); addLogEntry(); } // Function used for WebSocket or Webhook callbacks to perform specific activities void oneDriveOnlineCallback() { // If we are in a --download-only method of operation, there is no filesystem monitoring, so no inotify events to check if (!appConfig.getValueBool("download_only")) { // Handle inotify events processInotifyEvents(true); } // Sync any online change down to the local disk // If we are doing --upload-only however .. we need to 'ignore' online change if (!appConfig.getValueBool("upload_only")) { // We are not doing an --upload-only scenario .. sync online change --> local syncEngineInstance.syncOneDriveAccountToLocalDisk(); } if (appConfig.getValueBool("monitor")) { // Handle inotify events processInotifyEvents(true); } } // Perform only an upload of data when using --upload-only void performUploadOnlySyncProcess(string localPath, Monitor filesystemMonitor = null) { // Perform the local database consistency check, picking up locally modified data and uploading this to OneDrive syncEngineInstance.performDatabaseConsistencyAndIntegrityCheck(); if (appConfig.getValueBool("monitor")) { // Handle any inotify events whilst the DB was being scanned processInotifyEvents(true); } // Scan the configured 'sync_dir' for new data to upload syncEngineInstance.scanLocalFilesystemPathForNewData(localPath); if (appConfig.getValueBool("monitor")) { // Handle any new inotify events whilst the local filesystem was being scanned processInotifyEvents(true); } } // Perform the normal application sync process void performStandardSyncProcess(string localPath, Monitor filesystemMonitor = null) { // If we are performing log suppression, output this message so the user knows what is happening if (appConfig.suppressLoggingOutput) { addLogEntry("Syncing changes from Microsoft OneDrive ..."); } // Zero out these arrays syncEngineInstance.fileDownloadFailures = []; syncEngineInstance.fileUploadFailures = []; // Which way do we sync first? // OneDrive first then local changes (normal operational process that uses OneDrive as the source of truth) // Local First then OneDrive changes (alternate operation process to use local files as source of truth) if (appConfig.getValueBool("local_first")) { // Local data first // Perform the local database consistency check, picking up locally modified data and uploading this to OneDrive syncEngineInstance.performDatabaseConsistencyAndIntegrityCheck(); if (appConfig.getValueBool("monitor")) { // Handle any inotify events whilst the DB was being scanned processInotifyEvents(true); } // Scan the configured 'sync_dir' for new data to upload to OneDrive syncEngineInstance.scanLocalFilesystemPathForNewData(localPath); if (appConfig.getValueBool("monitor")) { // Handle any new inotify events whilst the local filesystem was being scanned processInotifyEvents(true); } // Download data from OneDrive last syncEngineInstance.syncOneDriveAccountToLocalDisk(); if (appConfig.getValueBool("monitor")) { // Cancel out any inotify events from downloading data processInotifyEvents(false); } // At this point, we have done a sync from: // local -> online // online -> local // // Everything now should be 'in sync' and the database correctly populated with data // If --resync was used, we need to unset this as sync.d performs certain queries depending on if 'resync' is set or not if (appConfig.getValueBool("resync")) { // unset 'resync' now that everything has been performed appConfig.setValueBool("resync" , false); } } else { // Normal sync process // Download data from OneDrive first syncEngineInstance.syncOneDriveAccountToLocalDisk(); if (appConfig.getValueBool("monitor")) { // Cancel out any inotify events from downloading data processInotifyEvents(false); } // Perform the local database consistency check, picking up locally modified data and uploading this to OneDrive syncEngineInstance.performDatabaseConsistencyAndIntegrityCheck(); if (appConfig.getValueBool("monitor")) { // Handle any inotify events whilst the DB was being scanned processInotifyEvents(true); } // Is --download-only NOT configured? if (!appConfig.getValueBool("download_only")) { // Scan the configured 'sync_dir' for new data to upload to OneDrive syncEngineInstance.scanLocalFilesystemPathForNewData(localPath); if (appConfig.getValueBool("monitor")) { // Handle any new inotify events whilst the local filesystem was being scanned processInotifyEvents(true); } // If we are not doing a 'force_children_scan' perform a true-up // 'force_children_scan' is used when using /children rather than /delta and it is not efficient to re-run this exact same process twice if (!appConfig.getValueBool("force_children_scan")) { // Perform the final true up scan to ensure we have correctly replicated the current online state locally if (!appConfig.suppressLoggingOutput) { addLogEntry("Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process"); } // We pass in the 'appConfig.fullScanTrueUpRequired' value which then flags do we use the configured 'deltaLink' // If 'appConfig.fullScanTrueUpRequired' is true, we do not use the 'deltaLink' if we are in --monitor mode, thus forcing a full scan true up syncEngineInstance.syncOneDriveAccountToLocalDisk(); if (appConfig.getValueBool("monitor")) { // Cancel out any inotify events from downloading data processInotifyEvents(false); } } } // At this point, we have done a sync from: // online -> local // local -> online (if not doing --download-only) // online -> local (if not doing --download-only) // // Everything now should be 'in sync' and the database correctly populated with data // If --resync was used, we need to unset this as sync.d performs certain queries depending on if 'resync' is set or not if (appConfig.getValueBool("resync")) { // unset 'resync' now that everything has been performed appConfig.setValueBool("resync" , false); } } } // Process any inotify events void processInotifyEvents(bool updateFlag) { // Attempt to process or cancel inotify events // filesystemMonitor.update will throw this, thus needs to be caught // monitor.MonitorException@src/monitor.d(549): inotify queue overflow: some events may be lost (Interrupted system call) try { // Process any inotify events or cancel events based on flag value // True = process // False = cancel filesystemMonitor.update(updateFlag); } catch (MonitorException e) { // Catch any exceptions thrown by inotify / monitor engine addLogEntry("ERROR: The following inotify error was generated: " ~ e.msg); } } // Display the sync outcome void displaySyncOutcome() { // Detail any download or upload transfer failures syncEngineInstance.displaySyncFailures(); // Sync is either complete or partially complete if (!syncEngineInstance.syncFailures) { // No download or upload issues if (!appConfig.getValueBool("monitor")) addLogEntry(); // Add an additional line break so that this is clear when using --sync addLogEntry("Sync with Microsoft OneDrive is complete"); } else { addLogEntry(); addLogEntry("Sync with Microsoft OneDrive has completed, however there are items that failed to sync."); // Due to how the OneDrive API works 'changes' such as add new files online, rename files online, delete files online are only sent once when using the /delta API call. // That we failed to download it, we need to track that, and then issue a --resync to download any of these failed files .. unfortunate, but there is no easy way here if (!syncEngineInstance.fileDownloadFailures.empty) { addLogEntry("To fix any download failures you may need to perform a --resync to ensure this system is correctly synced with your Microsoft OneDrive Account"); } if (!syncEngineInstance.fileUploadFailures.empty) { addLogEntry("To fix any upload failures you may need to perform a --resync to ensure this system is correctly synced with your Microsoft OneDrive Account"); } // So that from a logging perspective these messages are clear, add a line break in addLogEntry(); } } // Perform database file removal void processResyncDatabaseRemoval(string databaseFilePathToRemove) { // Log what we are doing if (debugLogging) {addLogEntry("Testing if we have exclusive access to local database file", ["debug"]);} // Are we the only running instance? Test that we can open the database file path itemDB = new ItemDatabase(databaseFilePathToRemove); // did we successfully initialise the database class? if (!itemDB.isDatabaseInitialised()) { // no .. destroy class itemDB = null; // exit application - void function, force exit this way exit(EXIT_FAILURE); } // If we have exclusive access we will not have exited // destroy access test itemDB = null; // delete application sync state addLogEntry("Deleting the saved application sync status ..."); if (!dryRun) { safeRemove(databaseFilePathToRemove); } else { // --dry-run scenario ... technically we should not be making any local file changes ....... addLogEntry("DRY-RUN: Not removing the saved application sync status"); } } // Clean up the local database files void cleanupDatabaseFiles(string activeDatabaseFileName) { // Temp variables string databaseShmFile = activeDatabaseFileName ~ "-shm"; string databaseWalFile = activeDatabaseFileName ~ "-wal"; // Are we performing a --dry-run? if (dryRun) { // If the dry run database exists, clean this up if (exists(activeDatabaseFileName)) { // remove the dry run database file if (debugLogging) {addLogEntry("DRY-RUN: Removing items-dryrun.sqlite3 as it still exists for some reason", ["debug"]);} safeRemove(activeDatabaseFileName); } } else { // we may have not been using --dry-run, however we may have been running some operations that use a dry-run database, and this needs to be explicitly cleaned up if (exists(appConfig.databaseFilePathDryRun)) { if (debugLogging) {addLogEntry("Removing items-dryrun.sqlite3 as it still exists for some reason post being used for non-dryrun operations", ["debug"]);} safeRemove(appConfig.databaseFilePathDryRun); } } // Silent cleanup of -shm file if it exists if (exists(databaseShmFile)) { // Configure the log message string logMessage = "Removing " ~ baseName(databaseShmFile) ~ " as it still exists for some reason"; // Is this a --dry-run scenario if (dryRun) { logMessage = "DRY-RUN: " ~ logMessage; } // Remove -shm file if (debugLogging) {addLogEntry(logMessage, ["debug"]);} safeRemove(databaseShmFile); } // Silent cleanup of wal files if it exists if (exists(databaseWalFile)) { // Configure the log message string logMessage = "Removing " ~ baseName(databaseWalFile) ~ " as it still exists for some reason"; // Is this a --dry-run scenario if (dryRun) { logMessage = "DRY-RUN: " ~ logMessage; } // Remove -wal file if (debugLogging) {addLogEntry(logMessage, ["debug"]);} safeRemove(databaseWalFile); } } // Perform a check to see if this is a mount point, and if the 'mount' has gone void checkForNoMountScenario() { // If this is a 'mounted' folder, the 'mount point' should have this file to help the application stop any action to preserve data because the drive to mount is not currently mounted if (appConfig.getValueBool("check_nomount")) { // we were asked to check the mount point for the presence of a '.nosync' file if (exists(".nosync")) { addLogEntry("ERROR: .nosync file found in directory mount point. Aborting application startup process to safeguard data.", ["info", "notify"]); // Perform the shutdown process performSynchronisedExitProcess("check_nomount"); // Exit exit(EXIT_FAILURE); } } } // Setup a signal handler for catching SIGINT, SIGTERM and SIGSEGV (CTRL-C and others) during application execution void setupSignalHandler() { sigaction_t action; action.sa_handler = &exitViaSignalHandler; // Direct function pointer assignment sigemptyset(&action.sa_mask); // Initialize the signal set to empty action.sa_flags = 0; sigaction(SIGINT, &action, null); // Interrupt from keyboard sigaction(SIGTERM, &action, null); // Termination signal sigaction(SIGSEGV, &action, null); // Invalid Memory Access signal } // Catch SIGINT (CTRL-C), SIGTERM (kill) and SIGSEGV (invalid memory access), handle rapid repeat CTRL-C presses extern(C) nothrow @nogc @system void exitViaSignalHandler(int signo) { // Update global exitHandlerTriggered flag so that objects that depend on this know we are shutting down exitHandlerTriggered = true; // Catch the generation of SIGSEV post SIGINT or SIGTERM event if (signo == SIGSEGV) { // Was SIGTERM used? if (!sigtermHandlerTriggered) { // No .. so most likely SIGINT (CTRL-C) - lets check if (signo == SIGINT) { // Yes - SIGINT was used printf("Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.\n"); exit(130); } else { // Confirmed as SIGSEGV, but not SIGINT and SIGTERM not used printf("FATAL: Segmentation fault (SIGSEGV). The application encountered an internal error and will now exit in a unclean manner.\n"); exit(139); } } else { // High probability of being shutdown by systemd, for example: systemctl --user stop onedrive // Exit in a manner that does not trigger an exit failure in systemd exit(0); } } if (signo == SIGTERM) { // systemd will use SIGTERM to terminate a running process sigtermHandlerTriggered = true; } if (shutdownInProgress) { return; // Ignore subsequent presses } else { // Disable logging suppression appConfig.suppressLoggingOutput = false; // Flag we are shutting down shutdownInProgress = true; try { assumeNoGC ( () { // Log that a termination signal was caught addLogEntry("\nReceived termination signal, attempting to cleanly shutdown application"); // Try and shutdown in a safe and synchronised manner performSynchronisedExitProcess("SIGINT-SIGTERM-HANDLER"); })(); } catch (Exception e) { // Any output here will cause a GC allocation // - Error: `@nogc` function `main.exitHandler` cannot call non-@nogc function `std.stdio.writeln!string.writeln` // - Error: cannot use operator `~` in `@nogc` function `main.exitHandler` // writeln("Exception during shutdown: " ~ e.msg); } // Exit the process with the provided exit code exit(signo); } } // Handle application exit void performSynchronisedExitProcess(string scopeCaller = null) { synchronized { // Perform cleanup and shutdown of various services and resources try { // Log who called this function if (debugLogging) {addLogEntry("performSynchronisedExitProcess called by: " ~ scopeCaller, ["debug"]);} // Remove Desktop integration if(performFileSystemMonitoring) { // Was desktop integration enabled? if (appConfig.getValueBool("display_manager_integration")) { // Attempt removal attemptFileManagerIntegrationRemoval(); } } // Shutdown the OneDrive Webhook instance shutdownOneDriveWebhook(); // Shutdown the OneDrive WebSocket instance shutdownOneDriveSocketIo(); // Shutdown any local filesystem monitoring shutdownFilesystemMonitor(); // Shutdown the sync engine if (scopeCaller == "SIGINT-SIGTERM-HANDLER") { // Wait for all parallel jobs that depend on the database being available to complete addLogEntry("Waiting for any existing upload|download process to complete"); } shutdownSyncEngine(); // Release all CurlEngine instances releaseAllCurlInstances(); // Shutdown the client side filtering objects shutdownSelectiveSync(); // Shutdown the database shutdownDatabase(); // Shutdown the application configuration objects - nothing should be active now shutdownAppConfig(); // Shutdown application logging shutdownApplicationLogging(); } catch (Exception e) { addLogEntry("Error during performStandardExitProcess: " ~ e.toString(), ["error"]); } } } void shutdownOneDriveWebhook() { if (oneDriveWebhook !is null) { if (debugLogging) {addLogEntry("Shutting down OneDrive Webhook instance", ["debug"]);} oneDriveWebhook.stop(); object.destroy(oneDriveWebhook); oneDriveWebhook = null; if (debugLogging) {addLogEntry("Shutdown of OneDrive Webhook instance is complete", ["debug"]);} } } void shutdownOneDriveSocketIo() { if (oneDriveSocketIo !is null) { if (debugLogging) addLogEntry("Shutting down OneDrive WebSocket instance", ["debug"]); oneDriveSocketIo.stop(); object.destroy(oneDriveSocketIo); oneDriveSocketIo = null; if (debugLogging) addLogEntry("Shutdown of OneDrive WebSocket instance complete", ["debug"]); } } void shutdownFilesystemMonitor() { if (filesystemMonitor !is null) { if (debugLogging) {addLogEntry("Shutting down Filesystem Monitoring instance", ["debug"]);} filesystemMonitor.shutdown(); object.destroy(filesystemMonitor); filesystemMonitor = null; if (debugLogging) {addLogEntry("Shutdown of Filesystem Monitoring instance is complete", ["debug"]);} } } void shutdownSelectiveSync() { if (selectiveSync !is null) { if (debugLogging) {addLogEntry("Shutting down Client Side Filtering instance", ["debug"]);} selectiveSync.shutdown(); object.destroy(selectiveSync); selectiveSync = null; if (debugLogging) {addLogEntry("Shutdown of Client Side Filtering instance is complete", ["debug"]);} } } void shutdownSyncEngine() { if (syncEngineInstance !is null) { if (debugLogging) {addLogEntry("Shutting down Sync Engine instance", ["debug"]);} syncEngineInstance.shutdown(); // Make sure any running thread completes first object.destroy(syncEngineInstance); syncEngineInstance = null; if (debugLogging) {addLogEntry("Shutdown Sync Engine instance is complete", ["debug"]);} } } void shutdownDatabase() { if (itemDB !is null && itemDB.isDatabaseInitialised()) { if (debugLogging) {addLogEntry("Shutting down Database instance", ["debug"]);} // Write WAL and SHM data to file if (debugLogging) {addLogEntry("Merge contents of WAL and SHM files into main database file before shutting down database", ["debug"]);} itemDB.performCheckpoint("TRUNCATE"); // Do we perform a database vacuum? if (performDatabaseVacuum) { // Logging to attempt this is denoted from performVacuum() - so no need to confirm here itemDB.performVacuum(); // If this completes, it is denoted from performVacuum() - so no need to confirm here } // Close the DB File Handle itemDB.closeDatabaseFile(); object.destroy(itemDB); cleanupDatabaseFiles(runtimeDatabaseFile); itemDB = null; if (debugLogging) {addLogEntry("Shutdown of Database instance is complete", ["debug"]);} } } void shutdownAppConfig() { if (appConfig !is null) { if (debugLogging) {addLogEntry("Shutting down Application Configuration instance", ["debug"]);} object.destroy(appConfig); appConfig = null; if (debugLogging) {addLogEntry("Shutdown of Application Configuration instance is complete", ["debug"]);} } } void shutdownApplicationLogging() { // Log that we are exiting if (loggingStillInitialised()) { if (loggingActive()) { // join all threads thread_joinAll(); if (debugLogging) {addLogEntry("Application is exiting", ["debug"]);} addLogEntry("#######################################################################################################################################", ["logFileOnly"]); // Destroy the shared logging buffer which flushes any remaining logs if (debugLogging) {addLogEntry("Shutting down Application Logging instance", ["debug"]);} // Allow any logging complete before we exit Thread.sleep(dur!("msecs")(500)); // Shutdown Logging which also sets logBuffer to null shutdownLogging(); } } } string compilerDetails() { version(DigitalMars) enum compiler = "DMD"; else version(LDC) enum compiler = "LDC"; else version(GNU) enum compiler = "GDC"; else enum compiler = "Unknown compiler"; string compilerString = compiler ~ " " ~ to!string(__VERSION__); return compilerString; } void attemptFileManagerIntegration() { // Are we running under a Desktop Manager (GNOME or KDE)? if (appConfig.isGuiSessionDetected()) { // Generate desktop hints auto hints = appConfig.detectDesktop(); // GNOME Desktop File Manager integration if (hints.gnome) { // Attempt integration appConfig.addGnomeBookmark(); appConfig.setOneDriveFolderIcon(); return; } // KDE Desktop File Manager integration if (hints.kde) { // Attempt integration appConfig.addKDEPlacesEntry(); return; } } } void attemptFileManagerIntegrationRemoval() { // Are we running under a Desktop Manager (GNOME or KDE)? if (appConfig.isGuiSessionDetected()) { // Generate desktop hints auto hints = appConfig.detectDesktop(); // GNOME Desktop File Manager integration removal if (hints.gnome) { // Attempt integration removal appConfig.removeGnomeBookmark(); appConfig.removeOneDriveFolderIcon(); return; } // KDE Desktop File Manager integration removal if (hints.kde) { // Attempt integration removal appConfig.removeKDEPlacesEntry(); return; } } } ================================================ FILE: src/monitor.d ================================================ // What is this module called? module monitor; // What does this module require to function? import core.stdc.errno; import core.stdc.stdlib; import core.sys.linux.sys.inotify; import core.sys.posix.poll; import core.sys.posix.unistd; import core.sys.posix.sys.select; import core.thread; import core.time; import std.algorithm; import std.concurrency; import std.exception; import std.file; import std.path; import std.process; import std.regex; import std.stdio; import std.string; import std.conv; import core.sync.mutex; // What other modules that we have created do we need to import? import config; import util; import log; import clientSideFiltering; // Relevant inotify events version(FreeBSD) { private immutable uint32_t mask = IN_CLOSE_WRITE | IN_CREATE | IN_DELETE | IN_MOVE; } else { private immutable uint32_t mask = IN_CLOSE_WRITE | IN_CREATE | IN_DELETE | IN_MOVE | IN_IGNORED | IN_Q_OVERFLOW; } class MonitorException: ErrnoException { @safe this(string msg, string file = __FILE__, size_t line = __LINE__) { super(msg, file, line); } } class MonitorBackgroundWorker { // inotify file descriptor int fd; Pipe p; bool isAlive; this() { isAlive = true; p = pipe(); } shared void initialise() { fd = inotify_init(); if (fd < 0) throw new MonitorException("inotify_init failed"); } // Add this path to be monitored shared int addInotifyWatch(string pathname) { int wd = inotify_add_watch(fd, toStringz(pathname), mask); if (wd < 0) { if (errno() == ENOSPC) { // Predefined Versions // https://dlang.org/spec/version.html#predefined-versions version (linux) { // Read max inotify watches from procfs on Linux ulong maxInotifyWatches = to!int(strip(readText("/proc/sys/fs/inotify/max_user_watches"))); addLogEntry("The user limit on the total number of inotify watches has been reached."); addLogEntry("Your current limit of inotify watches is: " ~ to!string(maxInotifyWatches)); addLogEntry("It is recommended that you change the max number of inotify watches to at least double your existing value."); addLogEntry("To change the current max number of watches to " ~ to!string((maxInotifyWatches * 2)) ~ " run:"); addLogEntry("EXAMPLE: sudo sysctl fs.inotify.max_user_watches=" ~ to!string((maxInotifyWatches * 2))); } else { // some other platform addLogEntry("The user limit on the total number of inotify watches has been reached."); addLogEntry("Please seek support from your distribution on how to increase the max number of inotify watches to at least double your existing value."); } } if (errno() == 13) { if (verboseLogging) {addLogEntry("WARNING: inotify_add_watch failed - permission denied: " ~ pathname, ["verbose"]);} } // Flag any other errors addLogEntry("ERROR: inotify_add_watch failed: " ~ pathname); return wd; } // Add path to inotify watch - required regardless if a '.folder' or 'folder' if (debugLogging) {addLogEntry("inotify_add_watch successfully added for: " ~ pathname, ["debug"]);} // Do we log that we are monitoring this directory? if (isDir(pathname)) { // Log that this is directory is being monitored if (verboseLogging) {addLogEntry("Monitoring directory: " ~ pathname, ["verbose"]);} } return wd; } shared int removeInotifyWatch(int wd) { assert(fd > 0, "File descriptor 'fd' is invalid."); assert(wd > 0, "Watch descriptor 'wd' is invalid."); // Debug logging of the inotify watch being removed if (debugLogging) {addLogEntry("Attempting to remove inotify watch: fd=" ~ fd.to!string ~ ", wd=" ~ wd.to!string, ["debug"]);} // return the value of performing the action return inotify_rm_watch(fd, wd); } shared void watch(Tid callerTid) { // On failure, send -1 to caller int res; // wait for the caller to be ready receiveOnly!int(); while (isAlive) { fd_set fds; FD_ZERO (&fds); FD_SET(fd, &fds); // Listen for messages from the caller FD_SET((cast()p).readEnd.fileno, &fds); res = select(FD_SETSIZE, &fds, null, null, null); if(res == -1) { if(errno() == EINTR) { // Received an interrupt signal but no events are available // directly watch again } else { // Error occurred, tell caller to terminate. callerTid.send(-1); break; } } else { // Wake up caller callerTid.send(1); // wait for the caller to be ready if (isAlive) isAlive = receiveOnly!bool(); } } } shared void interrupt() { isAlive = false; (cast()p).writeEnd.writeln("done"); (cast()p).writeEnd.flush(); } shared void shutdown() { isAlive = false; if (fd > 0) { close(fd); fd = 0; (cast()p).close(); } } } void startMonitorJob(shared(MonitorBackgroundWorker) worker, Tid callerTid) { try { worker.watch(callerTid); } catch (OwnerTerminated error) { // caller is terminated worker.shutdown(); } } enum ActionType { moved, deleted, changed, createDir } struct Action { ActionType type; bool skipped; string src; string dst; } struct ActionHolder { Action[] actions; size_t[string] srcMap; void append(ActionType type, string src, string dst=null) { size_t[] pendingTargets; switch (type) { case ActionType.changed: if (src in srcMap && actions[srcMap[src]].type == ActionType.changed) { // skip duplicate operations return; } break; case ActionType.createDir: break; case ActionType.deleted: if (src in srcMap) { size_t pendingTarget = srcMap[src]; // Skip operations require reading local file that is gone switch (actions[pendingTarget].type) { case ActionType.changed: case ActionType.createDir: actions[srcMap[src]].skipped = true; srcMap.remove(src); break; default: break; } } break; case ActionType.moved: for(int i = 0; i < actions.length; i++) { // Only match for latest operation if (actions[i].src in srcMap) { switch (actions[i].type) { case ActionType.changed: case ActionType.createDir: // check if the source is the prefix of the target string prefix = src ~ "/"; string target = actions[i].src; if (prefix[0] != '.') prefix = "./" ~ prefix; if (target[0] != '.') target = "./" ~ target; string comm = commonPrefix(prefix, target); if (src == actions[i].src || comm.length == prefix.length) { // Hold operations require reading local file that is moved after the target is moved online pendingTargets ~= i; actions[i].skipped = true; srcMap.remove(actions[i].src); if (comm.length == target.length) actions[i].src = dst; else actions[i].src = dst ~ target[comm.length - 1 .. target.length]; } break; default: break; } } } break; default: break; } actions ~= Action(type, false, src, dst); srcMap[src] = actions.length - 1; foreach (pendingTarget; pendingTargets) { actions ~= actions[pendingTarget]; actions[$-1].skipped = false; srcMap[actions[$-1].src] = actions.length - 1; } } } final class Monitor { // Class variables ApplicationConfig appConfig; ClientSideFiltering selectiveSync; // Are we verbose in logging output bool verbose = false; // skip symbolic links bool skip_symlinks = false; // check for .nosync if enabled bool check_nosync = false; // check if initialised bool initialised = false; // Worker Tid Tid workerTid; // Configure Private Class Variables shared(MonitorBackgroundWorker) worker; // map every inotify watch descriptor to its directory private string[int] wdToDirName; // map the inotify cookies of move_from events to their path private string[int] cookieToPath; // buffer to receive the inotify events private void[] buffer; // Mutex to support thread safe access of inotify watch descriptors private Mutex inotifyMutex; // Configure function delegates void delegate(string path) onDirCreated; void delegate(string[] path) onFileChanged; void delegate(string path) onDelete; void delegate(string from, string to) onMove; // List of paths that were moved, not deleted bool[string] movedNotDeleted; // An array of actions ActionHolder actionHolder; // Configure the class variable to consume the application configuration including selective sync this(ApplicationConfig appConfig, ClientSideFiltering selectiveSync) { this.appConfig = appConfig; this.selectiveSync = selectiveSync; inotifyMutex = new Mutex(); // Define a Mutex for thread-safe access } // The destructor should only clean up resources owned directly by this instance ~this() { object.destroy(worker); } // Initialise the monitor class void initialise() { // Configure the variables skip_symlinks = appConfig.getValueBool("skip_symlinks"); check_nosync = appConfig.getValueBool("check_nosync"); if (appConfig.getValueLong("verbose") > 0) { verbose = true; } assert(onDirCreated && onFileChanged && onDelete && onMove); if (!buffer) buffer = new void[4096]; worker = cast(shared) new MonitorBackgroundWorker; worker.initialise(); // from which point do we start watching for changes? string monitorPath; if (appConfig.getValueString("single_directory") != ""){ // single directory in use, monitor only this path monitorPath = "./" ~ appConfig.getValueString("single_directory"); } else { // default monitorPath = "."; } addRecursive(monitorPath); // Start monitoring workerTid = spawn(&startMonitorJob, worker, thisTid); initialised = true; } // Communication with worker void send(bool isAlive) { workerTid.send(isAlive); } // Shutdown the monitor class void shutdown() { if(!initialised) return; initialised = false; // Release all resources synchronized(inotifyMutex) { // Interrupt the worker to allow removal of inotify watch descriptors worker.interrupt(); // Remove all the inotify watch descriptors removeAll(); // Notify the worker that the monitor has been shutdown worker.interrupt(); send(false); wdToDirName = null; } } // Recursively add this path to be monitored private void addRecursive(string dirname) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // skip non existing/disappeared items if (!exists(dirname)) { if (verboseLogging) {addLogEntry("Not adding non-existing/disappeared directory: " ~ dirname, ["verbose"]);} return; } // Issue #3404: If the file is a very short lived file, and exists when the above test is done, but then is removed shortly thereafter, we need to catch this as a filesystem exception try { // Skip the monitoring of any user filtered items if (dirname != ".") { // Is the directory name a match to a skip_dir entry? // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched if (isDir(dirname)) { if (selectiveSync.isDirNameExcluded(dirname.strip('.'))) { // dont add a watch for this item if (debugLogging) {addLogEntry("Skipping monitoring due to skip_dir match: " ~ dirname, ["debug"]);} return; } } if (isFile(dirname)) { // Is the filename a match to a skip_file entry? // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched if (selectiveSync.isFileNameExcluded(dirname.strip('.'))) { // dont add a watch for this item if (debugLogging) {addLogEntry("Skipping monitoring due to skip_file match: " ~ dirname, ["debug"]);} return; } } // Is the path excluded by sync_list? if (selectiveSync.isPathExcludedViaSyncList(buildNormalizedPath(dirname))) { // dont add a watch for this item if (debugLogging) {addLogEntry("Skipping monitoring parent path due to sync_list exclusion: " ~ dirname, ["debug"]);} // However before we return, we need to test this path tree as a branch on this tree may be included by an anywhere exclusion rule. Do 'anywhere' inclusion rules exist? if (isDir(dirname)) { // Do any 'sync_list' anywhere inclusion rules exist? if (selectiveSync.syncListAnywhereInclusionRulesExist()) { // Yes .. if (debugLogging) {addLogEntry("Bypassing 'sync_list' exclusion to test if children should be monitored due to 'sync_list' anywhere rule existence", ["debug"]);} // Traverse this directory traverseDirectory(dirname); } } // For the original path, we return, no inotify watch was added return; } } // skip symlinks if configured if (isSymlink(dirname)) { // if config says so we skip all symlinked items if (skip_symlinks) { // dont add a watch for this directory return; } } // Do we need to check for .nosync? Only if check_nosync is true if (check_nosync) { if (exists(buildNormalizedPath(dirname) ~ "/.nosync")) { if (verboseLogging) {addLogEntry("Skipping watching path - .nosync found & --check-for-nosync enabled: " ~ buildNormalizedPath(dirname), ["verbose"]);} return; } } if (isDir(dirname)) { // This is a directory // is the path excluded if skip_dotfiles configured and path is a .folder? if ((selectiveSync.getSkipDotfiles()) && (isDotFile(dirname))) { // dont add a watch for this directory return; } } // passed all potential exclusions // add inotify watch for this path / directory / file if (debugLogging) {addLogEntry("Calling worker.addInotifyWatch() for this dirname: " ~ dirname, ["debug"]);} int wd = worker.addInotifyWatch(dirname); if (wd > 0) { wdToDirName[wd] = buildNormalizedPath(dirname) ~ "/"; } // if this is a directory, recursively add this path if (isDir(dirname)) { traverseDirectory(dirname); } // Catch any FileException error which is generated } catch (std.file.FileException e) { // Standard filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, dirname); return; } } // Traverse directory to test if this should have an inotify watch added private void traverseDirectory(string dirname) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Current path for error logging string currentPath; // Try and get all the directory entities for this path try { auto pathList = dirEntries(dirname, SpanMode.shallow, false); foreach(DirEntry entry; pathList) { currentPath = entry.name; if (entry.isDir) { if (debugLogging) {addLogEntry("Calling addRecursive() for this directory: " ~ entry.name, ["debug"]);} addRecursive(entry.name); } } // Catch any FileException error which is generated } catch (std.file.FileException e) { // Standard filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, currentPath); return; } catch (Exception e) { // Issue #1154 handling // Need to check for: Failed to stat file in error message if (canFind(e.msg, "Failed to stat file")) { // File system access issue addLogEntry("ERROR: The local file system returned an error with the following message:"); addLogEntry(" Error Message: " ~ e.msg); addLogEntry("ACCESS ERROR: Please check your UID and GID access to this file, as the permissions on this file is preventing this application to read it"); addLogEntry("\nFATAL: Forcing exiting application to avoid deleting data due to local file system access issues\n"); // Must force exit here, allow logging to be done forceExit(); } else { // some other error displayFileSystemErrorMessage(e.msg, thisFunctionName, currentPath); return; } } } // Remove a watch descriptor private void removeAll() { string[int] copy; synchronized(inotifyMutex) { copy = wdToDirName.dup; // Make a thread-safe copy } // Loop through the watch descriptors and remove foreach (wd, path; copy) { remove(wd); } } private void remove(int wd) { assert(wd in wdToDirName); synchronized(inotifyMutex) { int ret = worker.removeInotifyWatch(wd); if (ret < 0) throw new MonitorException("inotify_rm_watch failed"); if (verboseLogging) {addLogEntry("Stopped monitoring directory (inotify watch removed): " ~ to!string(wdToDirName[wd]), ["verbose"]);} wdToDirName.remove(wd); } } // Remove the watch descriptors associated to the given path private void remove(const(char)[] path) { path ~= "/"; foreach (wd, dirname; wdToDirName) { if (dirname.startsWith(path)) { int ret = worker.removeInotifyWatch(wd); if (ret < 0) throw new MonitorException("inotify_rm_watch failed"); wdToDirName.remove(wd); if (verboseLogging) {addLogEntry("Stopped monitoring directory (inotify watch removed): " ~ dirname, ["verbose"]);} } } } // Return the file path from an inotify event private string getPath(const(inotify_event)* event) { string path = wdToDirName[event.wd]; if (event.len > 0) path ~= fromStringz(event.name.ptr); if (debugLogging) {addLogEntry("inotify path event for: " ~ path, ["debug"]);} return path; } // Update void update(bool useCallbacks = true) { if(!initialised) return; pollfd fds = { fd: worker.fd, events: POLLIN }; while (true) { bool hasNotification = false; int sleep_counter = 0; // Batch events up to 5 seconds while (sleep_counter < 5) { int ret = poll(&fds, 1, 0); if (ret == -1) throw new MonitorException("poll failed"); else if (ret == 0) break; // no events available hasNotification = true; size_t length = read(worker.fd, buffer.ptr, buffer.length); if (length == -1) throw new MonitorException("read failed"); int i = 0; while (i < length) { inotify_event *event = cast(inotify_event*) &buffer[i]; string path; string evalPath; // inotify event debug if (debugLogging) { addLogEntry("inotify event wd: " ~ to!string(event.wd), ["debug"]); addLogEntry("inotify event mask: " ~ to!string(event.mask), ["debug"]); addLogEntry("inotify event cookie: " ~ to!string(event.cookie), ["debug"]); addLogEntry("inotify event len: " ~ to!string(event.len), ["debug"]); addLogEntry("inotify event name: " ~ to!string(event.name), ["debug"]); } // inotify event handling if (debugLogging) { if (event.mask & IN_ACCESS) addLogEntry("inotify event flag: IN_ACCESS", ["debug"]); if (event.mask & IN_MODIFY) addLogEntry("inotify event flag: IN_MODIFY", ["debug"]); if (event.mask & IN_ATTRIB) addLogEntry("inotify event flag: IN_ATTRIB", ["debug"]); if (event.mask & IN_CLOSE_WRITE) addLogEntry("inotify event flag: IN_CLOSE_WRITE", ["debug"]); if (event.mask & IN_CLOSE_NOWRITE) addLogEntry("inotify event flag: IN_CLOSE_NOWRITE", ["debug"]); if (event.mask & IN_MOVED_FROM) addLogEntry("inotify event flag: IN_MOVED_FROM", ["debug"]); if (event.mask & IN_MOVED_TO) addLogEntry("inotify event flag: IN_MOVED_TO", ["debug"]); if (event.mask & IN_CREATE) addLogEntry("inotify event flag: IN_CREATE", ["debug"]); if (event.mask & IN_DELETE) addLogEntry("inotify event flag: IN_DELETE", ["debug"]); if (event.mask & IN_DELETE_SELF) addLogEntry("inotify event flag: IN_DELETE_SELF", ["debug"]); if (event.mask & IN_MOVE_SELF) addLogEntry("inotify event flag: IN_MOVE_SELF", ["debug"]); if (event.mask & IN_UNMOUNT) addLogEntry("inotify event flag: IN_UNMOUNT", ["debug"]); if (event.mask & IN_Q_OVERFLOW) addLogEntry("inotify event flag: IN_Q_OVERFLOW", ["debug"]); if (event.mask & IN_IGNORED) addLogEntry("inotify event flag: IN_IGNORED", ["debug"]); if (event.mask & IN_CLOSE) addLogEntry("inotify event flag: IN_CLOSE", ["debug"]); if (event.mask & IN_MOVE) addLogEntry("inotify event flag: IN_MOVE", ["debug"]); if (event.mask & IN_ONLYDIR) addLogEntry("inotify event flag: IN_ONLYDIR", ["debug"]); if (event.mask & IN_DONT_FOLLOW) addLogEntry("inotify event flag: IN_DONT_FOLLOW", ["debug"]); if (event.mask & IN_EXCL_UNLINK) addLogEntry("inotify event flag: IN_EXCL_UNLINK", ["debug"]); if (event.mask & IN_MASK_ADD) addLogEntry("inotify event flag: IN_MASK_ADD", ["debug"]); if (event.mask & IN_ISDIR) addLogEntry("inotify event flag: IN_ISDIR", ["debug"]); if (event.mask & IN_ONESHOT) addLogEntry("inotify event flag: IN_ONESHOT", ["debug"]); if (event.mask & IN_ALL_EVENTS) addLogEntry("inotify event flag: IN_ALL_EVENTS", ["debug"]); } // skip events that need to be ignored if (event.mask & IN_IGNORED) { // forget the directory associated to the watch descriptor wdToDirName.remove(event.wd); goto skip; } else if (event.mask & IN_Q_OVERFLOW) { throw new MonitorException("inotify queue overflow: some events may be lost"); } // if the event is not to be ignored, obtain path path = getPath(event); // configure the skip_dir & skip skip_file comparison item evalPath = path.strip('.'); // Skip events that should be excluded based on application configuration // We cant use isDir or isFile as this information is missing from the inotify event itself // Thus this causes a segfault when attempting to query this - https://github.com/abraunegg/onedrive/issues/995 // Based on the 'type' of event & object type (directory or file) check that path against the 'right' user exclusions // Directory events should only be compared against skip_dir and file events should only be compared against skip_file if (event.mask & IN_ISDIR) { // The event in question contains IN_ISDIR event mask, thus highly likely this is an event on a directory // This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched if (selectiveSync.isDirNameExcluded(evalPath)) { // The path to evaluate matches a path that the user has configured to skip goto skip; } } else { // The event in question missing the IN_ISDIR event mask, thus highly likely this is an event on a file // This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched if (selectiveSync.isFileNameExcluded(evalPath)) { // The path to evaluate matches a file that the user has configured to skip goto skip; } } // is the path, excluded via sync_list if (selectiveSync.isPathExcludedViaSyncList(path)) { // The path to evaluate matches a directory or file that the user has configured not to include in the sync goto skip; } // handle the inotify events if (event.mask & IN_MOVED_FROM) { if (debugLogging) {addLogEntry("event IN_MOVED_FROM: " ~ path, ["debug"]);} cookieToPath[event.cookie] = path; movedNotDeleted[path] = true; // Mark as moved, not deleted } else if (event.mask & IN_MOVED_TO) { if (debugLogging) {addLogEntry("event IN_MOVED_TO: " ~ path, ["debug"]);} if (event.mask & IN_ISDIR) addRecursive(path); auto from = event.cookie in cookieToPath; if (from) { cookieToPath.remove(event.cookie); if (useCallbacks) actionHolder.append(ActionType.moved, *from, path); movedNotDeleted.remove(*from); // Clear moved status } else { // Handle file moved in from outside if (event.mask & IN_ISDIR) { if (useCallbacks) actionHolder.append(ActionType.createDir, path); } else { if (useCallbacks) actionHolder.append(ActionType.changed, path); } } } else if (event.mask & IN_CREATE) { if (debugLogging) {addLogEntry("event IN_CREATE: " ~ path, ["debug"]);} if (event.mask & IN_ISDIR) { // fix from #2586 auto cookieToPath1 = cookieToPath.dup(); foreach (cookie, path1; cookieToPath1) { if (path1 == path) { cookieToPath.remove(cookie); } } addRecursive(path); if (useCallbacks) actionHolder.append(ActionType.createDir, path); } } else if (event.mask & IN_DELETE) { if (path in movedNotDeleted) { movedNotDeleted.remove(path); // Ignore delete for moved files } else { if (debugLogging) {addLogEntry("event IN_DELETE: " ~ path, ["debug"]);} if (useCallbacks) actionHolder.append(ActionType.deleted, path); } } else if ((event.mask & IN_CLOSE_WRITE) && !(event.mask & IN_ISDIR)) { if (debugLogging) {addLogEntry("event IN_CLOSE_WRITE and not IN_ISDIR: " ~ path, ["debug"]);} // fix from #2586 auto cookieToPath1 = cookieToPath.dup(); foreach (cookie, path1; cookieToPath1) { if (path1 == path) { cookieToPath.remove(cookie); } } if (useCallbacks) actionHolder.append(ActionType.changed, path); } else { addLogEntry("inotify event unhandled: " ~ path); assert(0); } skip: i += inotify_event.sizeof + event.len; } // Sleep for one second to prevent missing fast-changing events. if (poll(&fds, 1, 0) == 0) { sleep_counter += 1; Thread.sleep(dur!"seconds"(1)); } } if (!hasNotification) break; processChanges(); // Assume that the items moved outside the watched directory have been deleted foreach (cookie, path; cookieToPath) { if (debugLogging) {addLogEntry("Deleting cookie|watch (post loop): " ~ path, ["debug"]);} if (useCallbacks) onDelete(path); remove(path); cookieToPath.remove(cookie); } // Debug Log that all inotify events are flushed if (debugLogging) {addLogEntry("inotify events flushed", ["debug"]);} } } private void processChanges() { string[] changes; foreach(action; actionHolder.actions) { if (action.skipped) continue; switch (action.type) { case ActionType.changed: changes ~= action.src; break; case ActionType.deleted: onDelete(action.src); break; case ActionType.createDir: onDirCreated(action.src); break; case ActionType.moved: onMove(action.src, action.dst); break; default: break; } } if (!changes.empty) { onFileChanged(changes); } object.destroy(actionHolder); } } ================================================ FILE: src/notifications/README ================================================ The files in this directory have been obtained form the following places: dnotify.d https://github.com/Dav1dde/dnotify/blob/master/dnotify.d License: Creative Commons Zro 1.0 Universal see https://github.com/Dav1dde/dnotify/blob/master/LICENSE notify.d https://github.com/D-Programming-Deimos/libnotify/blob/master/deimos/notify/notify.d License: GNU Lesser General Public License (LGPL) 2.1 or upwards, see file ================================================ FILE: src/notifications/dnotify.d ================================================ module dnotify; import std.path; import std.file; private { import std.string : toStringz; import std.conv : to; import std.traits : isPointer, isArray; import std.variant : Variant; import std.array : appender; import deimos.notify.notify; } public import deimos.notify.notify : NOTIFY_EXPIRES_DEFAULT, NOTIFY_EXPIRES_NEVER, NotifyUrgency; version(NoPragma) { } else { pragma(lib, "notify"); pragma(lib, "gmodule"); pragma(lib, "glib-2.0"); } extern (C) { private void g_free(void* mem); private void g_list_free(GList* glist); } version(NoGdk) { } else { version(NoPragma) { } else { pragma(lib, "gdk_pixbuf"); } private: extern (C) { GdkPixbuf* gdk_pixbuf_new_from_file(const(char)* filename, GError **error); } } class NotificationError : Exception { string message; GError* gerror; this(GError* gerror) { this.message = to!(string)(gerror.message); this.gerror = gerror; super(this.message); } this(string message) { this.message = message; super(message); } } bool check_availability() { // notify_init might return without dbus server actually started // try to check for running dbus server char **ret_name; char **ret_vendor; char **ret_version; char **ret_spec_version; bool ret; try { return notify_get_server_info(ret_name, ret_vendor, ret_version, ret_spec_version); } catch (NotificationError e) { throw new NotificationError("Cannot find dbus server!"); } } void init(in char[] name) { notify_init(name.toStringz()); } alias notify_is_initted is_initted; alias notify_uninit uninit; shared static this() { init(baseName(thisExePath())); } shared static ~this() { uninit(); } string get_app_name() { return to!(string)(notify_get_app_name()); } void set_app_name(in char[] app_name) { notify_set_app_name(app_name.toStringz()); } string[] get_server_caps() { auto result = appender!(string[])(); GList* list = notify_get_server_caps(); if(list !is null) { for(GList* c = list; c !is null; c = c.next) { result.put(to!(string)(cast(char*)c.data)); g_free(c.data); } g_list_free(list); } return result.data; } struct ServerInfo { string name; string vendor; string version_; string spec_version; } ServerInfo get_server_info() { char* name; char* vendor; char* version_; char* spec_version; notify_get_server_info(&name, &vendor, &version_, &spec_version); scope(exit) { g_free(name); g_free(vendor); g_free(version_); g_free(spec_version); } return ServerInfo(to!string(name), to!string(vendor), to!string(version_), to!string(spec_version)); } struct Action { const(char[]) id; const(char[]) label; NotifyActionCallback callback; void* user_ptr; } class Notification { NotifyNotification* notify_notification; const(char)[] summary; const(char)[] body_; const(char)[] icon; bool closed = true; private int _timeout = NOTIFY_EXPIRES_DEFAULT; const(char)[] _category; NotifyUrgency _urgency; GdkPixbuf* _image; Variant[const(char)[]] _hints; const(char)[] _app_name; Action[] _actions; this(in char[] summary, in char[] body_, in char[] icon="") in { assert(is_initted(), "call dnotify.init() before using Notification"); } do { this.summary = summary; this.body_ = body_; this.icon = icon; notify_notification = notify_notification_new(summary.toStringz(), body_.toStringz(), icon.toStringz()); } bool update(in char[] summary, in char[] body_, in char[] icon="") { this.summary = summary; this.body_ = body_; this.icon = icon; return notify_notification_update(notify_notification, summary.toStringz(), body_.toStringz(), icon.toStringz()); } void show() { GError* ge; if(!notify_notification_show(notify_notification, &ge)) { throw new NotificationError(ge); } } @property int timeout() { return _timeout; } @property void timeout(int timeout) { this._timeout = timeout; notify_notification_set_timeout(notify_notification, timeout); } @property const(char[]) category() { return _category; } @property void category(in char[] category) { this._category = category; notify_notification_set_category(notify_notification, category.toStringz()); } @property NotifyUrgency urgency() { return _urgency; } @property void urgency(NotifyUrgency urgency) { this._urgency = urgency; notify_notification_set_urgency(notify_notification, urgency); } void set_image(GdkPixbuf* pixbuf) { notify_notification_set_image_from_pixbuf(notify_notification, pixbuf); //_image = pixbuf; } version(NoGdk) { } else { void set_image(in char[] filename) { GError* ge; // TODO: free pixbuf GdkPixbuf* pixbuf = gdk_pixbuf_new_from_file(filename.toStringz(), &ge); if(pixbuf is null) { if(ge is null) { throw new NotificationError("Unable to load file: " ~ filename.idup); } else { throw new NotificationError(ge); } } assert(notify_notification !is null); notify_notification_set_image_from_pixbuf(notify_notification, pixbuf); // TODO: fix segfault //_image = pixbuf; } } @property GdkPixbuf* image() { return _image; } // using deprecated set_hint_* functions (GVariant is an opaque structure, which needs the glib) void set_hint(T)(in char[] key, T value) { static if(is(T == int)) { notify_notification_set_hint_int32(notify_notification, key, value); } else static if(is(T == uint)) { notify_notification_set_hint_uint32(notify_notification, key, value); } else static if(is(T == double)) { notify_notification_set_hint_double(notify_notification, key, value); } else static if(is(T : const(char)[])) { notify_notification_set_hint_string(notify_notification, key, value.toStringz()); } else static if(is(T == ubyte)) { notify_notification_set_hint_byte(notify_notification, key, value); } else static if(is(T == ubyte[])) { notify_notification_set_hint_byte_array(notify_notification, key, value.ptr, value.length); } else { static assert(false, "unsupported value for Notification.set_hint"); } _hints[key] = Variant(value); } // unset hint? Variant get_hint(in char[] key) { return _hints[key]; } @property const(char)[] app_name() { return _app_name; } @property void app_name(in char[] name) { this._app_name = app_name; notify_notification_set_app_name(notify_notification, app_name.toStringz()); } void add_action(T)(in char[] action, in char[] label, NotifyActionCallback callback, T user_data) { static if(isPointer!T) { void* user_ptr = cast(void*)user_data; } else static if(isArray!T) { void* user_ptr = cast(void*)user_data.ptr; } else { void* user_ptr = cast(void*)&user_data; } notify_notification_add_action(notify_notification, action.toStringz(), label.toStringz(), callback, user_ptr, null); _actions ~= Action(action, label, callback, user_ptr); } void add_action()(Action action) { notify_notification_add_action(notify_notification, action.id.toStringz(), action.label.toStringz(), action.callback, action.user_ptr, null); _actions ~= action; } @property Action[] actions() { return _actions; } void clear_actions() { notify_notification_clear_actions(notify_notification); } void close() { GError* ge; if(!notify_notification_close(notify_notification, &ge)) { throw new NotificationError(ge); } } @property int closed_reason() { return notify_notification_get_closed_reason(notify_notification); } } version(TestMain) { import std.stdio; void main() { writeln(get_app_name()); set_app_name("blargh"); writeln(get_app_name()); writeln(get_server_caps()); writeln(get_server_info()); auto n = new Notification("foo", "bar", "notification-message-im"); n.timeout = 3; n.show(); } } ================================================ FILE: src/notifications/notify.d ================================================ /** * Copyright (C) 2004-2006 Christian Hammond * Copyright (C) 2010 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this library; if not, write to the * Free Software Foundation, Inc., 59 Temple Place - Suite 330, * Boston, MA 02111-1307, USA. */ module deimos.notify.notify; enum NOTIFY_VERSION_MAJOR = 0; enum NOTIFY_VERSION_MINOR = 7; enum NOTIFY_VERSION_MICRO = 5; template NOTIFY_CHECK_VERSION(int major, int minor, int micro) { enum NOTIFY_CHECK_VERSION = ((NOTIFY_VERSION_MAJOR > major) || (NOTIFY_VERSION_MAJOR == major && NOTIFY_VERSION_MINOR > minor) || (NOTIFY_VERSION_MAJOR == major && NOTIFY_VERSION_MINOR == minor && NOTIFY_VERSION_MICRO >= micro)); } alias ulong GType; alias void function(void*) GFreeFunc; struct GError { uint domain; int code; char* message; } struct GList { void* data; GList* next; GList* prev; } // dummies struct GdkPixbuf {} struct GObject {} struct GObjectClass {} struct GVariant {} GType notify_urgency_get_type(); /** * NOTIFY_EXPIRES_DEFAULT: * * The default expiration time on a notification. */ enum NOTIFY_EXPIRES_DEFAULT = -1; /** * NOTIFY_EXPIRES_NEVER: * * The notification never expires. It stays open until closed by the calling API * or the user. */ enum NOTIFY_EXPIRES_NEVER = 0; // #define NOTIFY_TYPE_NOTIFICATION (notify_notification_get_type ()) // #define NOTIFY_NOTIFICATION(o) (G_TYPE_CHECK_INSTANCE_CAST ((o), NOTIFY_TYPE_NOTIFICATION, NotifyNotification)) // #define NOTIFY_NOTIFICATION_CLASS(k) (G_TYPE_CHECK_CLASS_CAST((k), NOTIFY_TYPE_NOTIFICATION, NotifyNotificationClass)) // #define NOTIFY_IS_NOTIFICATION(o) (G_TYPE_CHECK_INSTANCE_TYPE ((o), NOTIFY_TYPE_NOTIFICATION)) // #define NOTIFY_IS_NOTIFICATION_CLASS(k) (G_TYPE_CHECK_CLASS_TYPE ((k), NOTIFY_TYPE_NOTIFICATION)) // #define NOTIFY_NOTIFICATION_GET_CLASS(o) (G_TYPE_INSTANCE_GET_CLASS ((o), NOTIFY_TYPE_NOTIFICATION, NotifyNotificationClass)) extern (C) { struct NotifyNotificationPrivate; struct NotifyNotification { /*< private >*/ GObject parent_object; NotifyNotificationPrivate *priv; } struct NotifyNotificationClass { GObjectClass parent_class; /* Signals */ void function(NotifyNotification *notification) closed; } /** * NotifyUrgency: * @NOTIFY_URGENCY_LOW: Low urgency. Used for unimportant notifications. * @NOTIFY_URGENCY_NORMAL: Normal urgency. Used for most standard notifications. * @NOTIFY_URGENCY_CRITICAL: Critical urgency. Used for very important notifications. * * The urgency level of the notification. */ enum NotifyUrgency { NOTIFY_URGENCY_LOW, NOTIFY_URGENCY_NORMAL, NOTIFY_URGENCY_CRITICAL, } /** * NotifyActionCallback: * @notification: * @action: * @user_data: * * An action callback function. */ alias void function(NotifyNotification* notification, char* action, void* user_data) NotifyActionCallback; GType notify_notification_get_type(); NotifyNotification* notify_notification_new(const(char)* summary, const(char)* body_, const(char)* icon); bool notify_notification_update(NotifyNotification* notification, const(char)* summary, const(char)* body_, const(char)* icon); bool notify_notification_show(NotifyNotification* notification, GError** error); void notify_notification_set_timeout(NotifyNotification* notification, int timeout); void notify_notification_set_category(NotifyNotification* notification, const(char)* category); void notify_notification_set_urgency(NotifyNotification* notification, NotifyUrgency urgency); void notify_notification_set_image_from_pixbuf(NotifyNotification* notification, GdkPixbuf* pixbuf); void notify_notification_set_icon_from_pixbuf(NotifyNotification* notification, GdkPixbuf* icon); void notify_notification_set_hint_int32(NotifyNotification* notification, const(char)* key, int value); void notify_notification_set_hint_uint32(NotifyNotification* notification, const(char)* key, uint value); void notify_notification_set_hint_double(NotifyNotification* notification, const(char)* key, double value); void notify_notification_set_hint_string(NotifyNotification* notification, const(char)* key, const(char)* value); void notify_notification_set_hint_byte(NotifyNotification* notification, const(char)* key, ubyte value); void notify_notification_set_hint_byte_array(NotifyNotification* notification, const(char)* key, const(ubyte)* value, ulong len); void notify_notification_set_hint(NotifyNotification* notification, const(char)* key, GVariant* value); void notify_notification_set_app_name(NotifyNotification* notification, const(char)* app_name); void notify_notification_clear_hints(NotifyNotification* notification); void notify_notification_add_action(NotifyNotification* notification, const(char)* action, const(char)* label, NotifyActionCallback callback, void* user_data, GFreeFunc free_func); void notify_notification_clear_actions(NotifyNotification* notification); bool notify_notification_close(NotifyNotification* notification, GError** error); int notify_notification_get_closed_reason(const NotifyNotification* notification); bool notify_init(const(char)* app_name); void notify_uninit(); bool notify_is_initted(); const(char)* notify_get_app_name(); void notify_set_app_name(const(char)* app_name); GList *notify_get_server_caps(); bool notify_get_server_info(char** ret_name, char** ret_vendor, char** ret_version, char** ret_spec_version); } version(MainTest) { import std.string; void main() { notify_init("test".toStringz()); auto n = notify_notification_new("summary".toStringz(), "body".toStringz(), "none".toStringz()); GError* ge; notify_notification_show(n, &ge); scope(success) notify_uninit(); } } ================================================ FILE: src/onedrive.d ================================================ // What is this module called? module onedrive; // What does this module require to function? import core.stdc.stdlib: EXIT_SUCCESS, EXIT_FAILURE, exit; import core.memory; import core.thread; import std.stdio; import std.string; import std.utf; import std.file; import std.exception; import std.regex; import std.json; import std.algorithm; import std.net.curl; import std.datetime; import std.path; import std.conv; import std.math; import std.uri; import std.array; // Required for webhooks import std.uuid; // What other modules that we have created do we need to import? import config; import log; import util; import curlEngine; import intune; // Define the 'OneDriveException' class class OneDriveException : Exception { // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/concepts/errors int httpStatusCode; const CurlResponse response; private JSONValue _error; // Public property to access the JSON error @property JSONValue error() const { return _error; } this(int httpStatusCode, string reason, const CurlResponse response, string file = __FILE__, size_t line = __LINE__) { this.httpStatusCode = httpStatusCode; this.response = response; this._error = response.json(); string msg = format("HTTP request returned status code %d (%s)\n%s", httpStatusCode, reason, toJSON(_error, true)); super(msg, file, line); } this(int httpStatusCode, string reason, string file = __FILE__, size_t line = __LINE__) { this.httpStatusCode = httpStatusCode; this.response = null; super(msg, file, line); } } // Define the 'OneDriveError' class class OneDriveError: Error { this(string msg) { super(msg); } } // Define the 'OneDriveApi' class class OneDriveApi { // Class variables that use other classes ApplicationConfig appConfig; CurlEngine curlEngine; CurlResponse response; // API Endpoint Constants immutable string defaultDriveUrlAPIEndpoint = "/v1.0/me/drive"; immutable string defaultDriveByIdUrlAPIEndpoint = "/v1.0/drives/"; immutable string defaultSharedWithMeUrlAPIEndpoint = "/v1.0/me/drive/sharedWithMe"; immutable string defaultItemByIdUrlAPIEndpoint = "/v1.0/me/drive/items/"; immutable string defaultItemByPathUrlAPIEndpoint = "/v1.0/me/drive/root:/"; immutable string defaultSiteSearchUrlAPIEndpoint = "/v1.0/sites?search"; immutable string defaultSiteDriveUrlAPIEndpoint = "/v1.0/sites/"; immutable string defaultSubscriptionUrlAPIEndpoint = "/v1.0/subscriptions"; immutable string defaultWebsocketEndpointAPIEndpoint = "/v1.0/me/drive/root/subscriptions/socketIo"; // Class variables string clientId = ""; string companyName = ""; string authUrl = ""; string deviceAuthUrl = ""; string redirectUrl = ""; string tokenUrl = ""; string driveUrl = ""; string driveByIdUrl = ""; string sharedWithMeUrl = ""; string itemByIdUrl = ""; string itemByPathUrl = ""; string siteSearchUrl = ""; string siteDriveUrl = ""; string subscriptionUrl = ""; string tenantId = ""; string authScope = ""; string websocketEndpoint = ""; string websocketEndpointAPIEndpoint = defaultWebsocketEndpointAPIEndpoint; const(char)[] refreshToken = ""; bool dryRun = false; bool keepAlive = false; this(ApplicationConfig appConfig) { // Configure the class variable to consume the application configuration this.appConfig = appConfig; this.curlEngine = null; this.response = null; // Configure the major API Query URL's, based on using application configuration // These however can be updated by config option 'azure_ad_endpoint', thus handled differently // Drive Queries driveUrl = appConfig.globalGraphEndpoint ~ defaultDriveUrlAPIEndpoint; driveByIdUrl = appConfig.globalGraphEndpoint ~ defaultDriveByIdUrlAPIEndpoint; // What is 'shared with me' Query sharedWithMeUrl = appConfig.globalGraphEndpoint ~ defaultSharedWithMeUrlAPIEndpoint; // Item Queries itemByIdUrl = appConfig.globalGraphEndpoint ~ defaultItemByIdUrlAPIEndpoint; itemByPathUrl = appConfig.globalGraphEndpoint ~ defaultItemByPathUrlAPIEndpoint; // Office 365 / SharePoint Queries siteSearchUrl = appConfig.globalGraphEndpoint ~ defaultSiteSearchUrlAPIEndpoint; siteDriveUrl = appConfig.globalGraphEndpoint ~ defaultSiteDriveUrlAPIEndpoint; // Subscriptions subscriptionUrl = appConfig.globalGraphEndpoint ~ defaultSubscriptionUrlAPIEndpoint; // WebSocket Endpoint - sets the default: /v1.0/me/drive/root/subscriptions/socketIo websocketEndpoint = appConfig.globalGraphEndpoint ~ websocketEndpointAPIEndpoint; } // The destructor should only clean up resources owned directly by this instance ~this() { if (response !is null) { object.destroy(response); // calls class CurlResponse destructor response = null; } if (curlEngine !is null) { object.destroy(curlEngine); // calls class CurlEngine destructor curlEngine = null; } if (appConfig !is null) { appConfig = null; } } // Initialise the OneDrive API class bool initialise(bool keepAlive=true) { // Initialise the curl engine this.keepAlive = keepAlive; if (curlEngine is null) { curlEngine = getCurlInstance(); curlEngine.initialise(appConfig.getValueLong("dns_timeout"), appConfig.getValueLong("connect_timeout"), appConfig.getValueLong("data_timeout"), appConfig.getValueLong("operation_timeout"), appConfig.defaultMaxRedirects, appConfig.getValueBool("debug_https"), appConfig.getValueString("user_agent"), appConfig.getValueBool("force_http_11"), appConfig.getValueLong("rate_limit"), appConfig.getValueLong("ip_protocol_version"), appConfig.getValueLong("max_curl_idle"), keepAlive); // WebSocket capability available in OS cURL version if (!appConfig.websocketSupportCheckDone) { // Check the underlying cURL capability to support websockets if (debugLogging) {addLogEntry("Checking cURL Websocket support ...", ["debug"]);} bool websocketSupport = curlSupportsWebSockets(); if (debugLogging) {addLogEntry("Checked cURL Websocket support = " ~ to!string(websocketSupport), ["debug"]);} // Update appConfig flags appConfig.curlSupportsWebSockets = websocketSupport; appConfig.websocketSupportCheckDone = true; // Notify user if cURL version is too old to support websockets, but only if we are in --monitor mode, as this is where this is used // Are we doing a --monitor operation? if (appConfig.getValueBool("monitor")) { if (!websocketSupport) { // cURL/libcurl version is too old addLogEntry(); addLogEntry("WARNING: Your libcurl version is too old for WebSocket support. Please upgrade to libcurl 7.86.0 or newer."); addLogEntry(" The near real-time processing of online changes cannot be enabled on your system."); addLogEntry(); } } } } // Authorised value to return bool authorised = false; // Did the user specify --dry-run dryRun = appConfig.getValueBool("dry_run"); // Set clientId to use the configured 'application_id' clientId = appConfig.getValueString("application_id"); if (clientId != appConfig.defaultApplicationId) { // a custom 'application_id' was set companyName = "custom_application"; } // Do we have a custom Azure Tenant ID? if (!appConfig.getValueString("azure_tenant_id").empty) { // Use the value entered by the user tenantId = appConfig.getValueString("azure_tenant_id"); } else { // set to common tenantId = "common"; } // Did the user specify a 'drive_id' ? if (!appConfig.getValueString("drive_id").empty) { // Update base URL's driveUrl = driveByIdUrl ~ appConfig.getValueString("drive_id"); itemByIdUrl = driveUrl ~ "/items"; itemByPathUrl = driveUrl ~ "/root:/"; // Need to update 'websocketEndpointAPIEndpoint' to /v1.0/drives/{driveId}/root/subscriptions/socketIo websocketEndpointAPIEndpoint = "/v1.0/drives/" ~ appConfig.getValueString("drive_id") ~ "/root/subscriptions/socketIo"; } // Configure the authentication scope if (appConfig.getValueBool("read_only_auth_scope")) { // read-only authentication scopes has been requested if (appConfig.getValueBool("use_device_auth")) { authScope = "&scope=Files.Read%20Files.Read.All%20Sites.Read.All%20offline_access"; } else { authScope = "&scope=Files.Read%20Files.Read.All%20Sites.Read.All%20offline_access&response_type=code&prompt=login&redirect_uri="; } } else { // read-write authentication scopes will be used (default) if (appConfig.getValueBool("use_device_auth")) { authScope = "&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access"; } else { authScope = "&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri="; } } // Configure Azure AD endpoints if 'azure_ad_endpoint' is configured string azureConfigValue = appConfig.getValueString("azure_ad_endpoint"); switch(azureConfigValue) { case "": if (tenantId == "common") { if (!appConfig.apiWasInitialised) addLogEntry("Configuring Global Azure AD Endpoints"); } else { if (!appConfig.apiWasInitialised) addLogEntry("Configuring Global Azure AD Endpoints - Single Tenant Application"); } // Authentication authUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/authorize"; deviceAuthUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/devicecode"; redirectUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; tokenUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/token"; // WebSocket Endpoint websocketEndpoint = appConfig.globalGraphEndpoint ~ websocketEndpointAPIEndpoint; break; case "USL4": if (!appConfig.apiWasInitialised) addLogEntry("Configuring Azure AD for US Government Endpoints"); // Authentication authUrl = appConfig.usl4AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/authorize"; deviceAuthUrl = appConfig.usl4AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/devicecode"; tokenUrl = appConfig.usl4AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/token"; if (clientId == appConfig.defaultApplicationId) { // application_id == default if (debugLogging) {addLogEntry("USL4 AD Endpoint but default application_id, redirectUrl needs to be aligned to globalAuthEndpoint", ["debug"]);} redirectUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } else { // custom application_id redirectUrl = appConfig.usl4AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } // Drive Queries driveUrl = appConfig.usl4GraphEndpoint ~ defaultDriveUrlAPIEndpoint; driveByIdUrl = appConfig.usl4GraphEndpoint ~ defaultDriveByIdUrlAPIEndpoint; // Item Queries itemByIdUrl = appConfig.usl4GraphEndpoint ~ defaultItemByIdUrlAPIEndpoint; itemByPathUrl = appConfig.usl4GraphEndpoint ~ defaultItemByPathUrlAPIEndpoint; // Office 365 / SharePoint Queries siteSearchUrl = appConfig.usl4GraphEndpoint ~ defaultSiteSearchUrlAPIEndpoint; siteDriveUrl = appConfig.usl4GraphEndpoint ~ defaultSiteDriveUrlAPIEndpoint; // Shared With Me sharedWithMeUrl = appConfig.usl4GraphEndpoint ~ defaultSharedWithMeUrlAPIEndpoint; // Subscriptions subscriptionUrl = appConfig.usl4GraphEndpoint ~ defaultSubscriptionUrlAPIEndpoint; // WebSocket Endpoint websocketEndpoint = appConfig.usl4GraphEndpoint ~ websocketEndpointAPIEndpoint; break; case "USL5": if (!appConfig.apiWasInitialised) addLogEntry("Configuring Azure AD for US Government Endpoints (DOD)"); // Authentication authUrl = appConfig.usl5AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/authorize"; deviceAuthUrl = appConfig.usl5AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/devicecode"; tokenUrl = appConfig.usl5AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/token"; if (clientId == appConfig.defaultApplicationId) { // application_id == default if (debugLogging) {addLogEntry("USL5 AD Endpoint but default application_id, redirectUrl needs to be aligned to globalAuthEndpoint", ["debug"]);} redirectUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } else { // custom application_id redirectUrl = appConfig.usl5AuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } // Drive Queries driveUrl = appConfig.usl5GraphEndpoint ~ defaultDriveUrlAPIEndpoint; driveByIdUrl = appConfig.usl5GraphEndpoint ~ defaultDriveByIdUrlAPIEndpoint; // Item Queries itemByIdUrl = appConfig.usl5GraphEndpoint ~ defaultItemByIdUrlAPIEndpoint; itemByPathUrl = appConfig.usl5GraphEndpoint ~ defaultItemByPathUrlAPIEndpoint; // Office 365 / SharePoint Queries siteSearchUrl = appConfig.usl5GraphEndpoint ~ defaultSiteSearchUrlAPIEndpoint; siteDriveUrl = appConfig.usl5GraphEndpoint ~ defaultSiteDriveUrlAPIEndpoint; // Shared With Me sharedWithMeUrl = appConfig.usl5GraphEndpoint ~ defaultSharedWithMeUrlAPIEndpoint; // Subscriptions subscriptionUrl = appConfig.usl5GraphEndpoint ~ defaultSubscriptionUrlAPIEndpoint; // WebSocket Endpoint websocketEndpoint = appConfig.usl5GraphEndpoint ~ websocketEndpointAPIEndpoint; break; case "DE": if (!appConfig.apiWasInitialised) addLogEntry("Configuring Azure AD Germany"); // Authentication authUrl = appConfig.deAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/authorize"; deviceAuthUrl = appConfig.deAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/devicecode"; tokenUrl = appConfig.deAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/token"; if (clientId == appConfig.defaultApplicationId) { // application_id == default if (debugLogging) {addLogEntry("DE AD Endpoint but default application_id, redirectUrl needs to be aligned to globalAuthEndpoint", ["debug"]);} redirectUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } else { // custom application_id redirectUrl = appConfig.deAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } // Drive Queries driveUrl = appConfig.deGraphEndpoint ~ defaultDriveUrlAPIEndpoint; driveByIdUrl = appConfig.deGraphEndpoint ~ defaultDriveByIdUrlAPIEndpoint; // Item Queries itemByIdUrl = appConfig.deGraphEndpoint ~ defaultItemByIdUrlAPIEndpoint; itemByPathUrl = appConfig.deGraphEndpoint ~ defaultItemByPathUrlAPIEndpoint; // Office 365 / SharePoint Queries siteSearchUrl = appConfig.deGraphEndpoint ~ defaultSiteSearchUrlAPIEndpoint; siteDriveUrl = appConfig.deGraphEndpoint ~ defaultSiteDriveUrlAPIEndpoint; // Shared With Me sharedWithMeUrl = appConfig.deGraphEndpoint ~ defaultSharedWithMeUrlAPIEndpoint; // Subscriptions subscriptionUrl = appConfig.deGraphEndpoint ~ defaultSubscriptionUrlAPIEndpoint; // WebSocket Endpoint websocketEndpoint = appConfig.deGraphEndpoint ~ websocketEndpointAPIEndpoint; break; case "CN": if (!appConfig.apiWasInitialised) addLogEntry("Configuring AD China operated by VNET"); // Authentication authUrl = appConfig.cnAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/authorize"; deviceAuthUrl = appConfig.cnAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/devicecode"; tokenUrl = appConfig.cnAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/v2.0/token"; if (clientId == appConfig.defaultApplicationId) { // application_id == default if (debugLogging) {addLogEntry("CN AD Endpoint but default application_id, redirectUrl needs to be aligned to globalAuthEndpoint", ["debug"]);} redirectUrl = appConfig.globalAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } else { // custom application_id redirectUrl = appConfig.cnAuthEndpoint ~ "/" ~ tenantId ~ "/oauth2/nativeclient"; } // Drive Queries driveUrl = appConfig.cnGraphEndpoint ~ defaultDriveUrlAPIEndpoint; driveByIdUrl = appConfig.cnGraphEndpoint ~ defaultDriveByIdUrlAPIEndpoint; // Item Queries itemByIdUrl = appConfig.cnGraphEndpoint ~ defaultItemByIdUrlAPIEndpoint; itemByPathUrl = appConfig.cnGraphEndpoint ~ defaultItemByPathUrlAPIEndpoint; // Office 365 / SharePoint Queries siteSearchUrl = appConfig.cnGraphEndpoint ~ defaultSiteSearchUrlAPIEndpoint; siteDriveUrl = appConfig.cnGraphEndpoint ~ defaultSiteDriveUrlAPIEndpoint; // Shared With Me sharedWithMeUrl = appConfig.cnGraphEndpoint ~ defaultSharedWithMeUrlAPIEndpoint; // Subscriptions subscriptionUrl = appConfig.cnGraphEndpoint ~ defaultSubscriptionUrlAPIEndpoint; // WebSocket Endpoint websocketEndpoint = appConfig.cnGraphEndpoint ~ websocketEndpointAPIEndpoint; break; // Default - all other entries default: if (!appConfig.apiWasInitialised) addLogEntry("Unknown Azure AD Endpoint request - using Global Azure AD Endpoints"); } // Has the application been authenticated? // How do we authenticate - standard method or via Intune? if (appConfig.getValueBool("use_intune_sso")) { // Authenticate via Intune if (appConfig.accessToken.empty) { // No authentication via intune yet authorised = authorise(); } else { // We are already authenticated authorised = true; } } else { // Authenticate via standard method if (!exists(appConfig.refreshTokenFilePath)) { if (debugLogging) {addLogEntry("Application has no 'refresh_token' thus needs to be authenticated", ["debug"]);} authorised = authorise(); } else { // Try and read the value from the appConfig if it is set, rather than trying to read the value from disk if (!appConfig.refreshToken.empty) { if (debugLogging) {addLogEntry("Read token from appConfig", ["debug"]);} refreshToken = strip(appConfig.refreshToken); authorised = true; } else { // Try and read the file from disk try { refreshToken = strip(readText(appConfig.refreshTokenFilePath)); // is the refresh_token empty? if (refreshToken.empty) { addLogEntry("RefreshToken exists but is empty: " ~ appConfig.refreshTokenFilePath); authorised = authorise(); } else { // Existing token not empty authorised = true; // update appConfig.refreshToken appConfig.refreshToken = refreshToken; } } catch (FileException exception) { authorised = authorise(); } catch (std.utf.UTFException exception) { // path contains characters which generate a UTF exception addLogEntry("Cannot read refreshToken from: " ~ appConfig.refreshTokenFilePath); addLogEntry(" Error Reason:" ~ exception.msg); authorised = false; } } if (refreshToken.empty) { // PROBLEM ... CODING TO DO ?????????? if (debugLogging) {addLogEntry("DEBUG: refreshToken is empty !!!!!!!!!!", ["debug"]);} } } } // Return if we are authorised if (debugLogging) {addLogEntry("Authorised State: " ~ to!string(authorised), ["debug"]);} return authorised; } // If the API has been configured correctly, print the items that been configured void debugOutputConfiguredAPIItems() { // Debug output of configured URL's // Application Identification if (debugLogging) { addLogEntry("Configured clientId " ~ clientId, ["debug"]); addLogEntry("Configured userAgent " ~ appConfig.getValueString("user_agent"), ["debug"]); // Authentication addLogEntry("Configured authScope: " ~ authScope, ["debug"]); addLogEntry("Configured authUrl: " ~ authUrl, ["debug"]); addLogEntry("Configured redirectUrl: " ~ redirectUrl, ["debug"]); addLogEntry("Configured tokenUrl: " ~ tokenUrl, ["debug"]); // Drive Queries addLogEntry("Configured driveUrl: " ~ driveUrl, ["debug"]); addLogEntry("Configured driveByIdUrl: " ~ driveByIdUrl, ["debug"]); // Shared With Me addLogEntry("Configured sharedWithMeUrl: " ~ sharedWithMeUrl, ["debug"]); // Item Queries addLogEntry("Configured itemByIdUrl: " ~ itemByIdUrl, ["debug"]); addLogEntry("Configured itemByPathUrl: " ~ itemByPathUrl, ["debug"]); // SharePoint Queries addLogEntry("Configured siteSearchUrl: " ~ siteSearchUrl, ["debug"]); addLogEntry("Configured siteDriveUrl: " ~ siteDriveUrl, ["debug"]); // Websocket addLogEntry("Configured websocketEndpoint: " ~ websocketEndpoint, ["debug"]); } } // Release CurlEngine bask to the Curl Engine Pool void releaseCurlEngine() { // Log that this was called if ((debugLogging) && (debugHTTPSResponse)) {addLogEntry("OneDrive API releaseCurlEngine() Called", ["debug"]);} // Release curl instance back to the pool if (curlEngine !is null) { curlEngine.releaseEngine(); curlEngine = null; } // Release the response response = null; // Perform Garbage Collection GC.collect(); } // Authenticate this client against Microsoft OneDrive API using one of the 3 authentication methods this client supports bool authorise() { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Has the client been configured to use Intune SSO via Microsoft Identity Broker (microsoft-identity-broker) dbus session if (appConfig.getValueBool("use_intune_sso")) { // The client is configured to use Intune SSO via Microsoft Identity Broker dbus session // Do we have a saved account file? if (!exists(appConfig.intuneAccountDetailsFilePath)) { // No file exists locally auto intuneAuthResult = acquire_token_interactive(appConfig.getValueString("application_id")); JSONValue intuneBrokerJSONData = intuneAuthResult.brokerTokenResponse; // Is the response JSON data valid? if ((intuneBrokerJSONData.type() == JSONType.object)) { // Does the JSON data have the required authentication elements: // - accessToken // - account // - expiresOn if ((hasAccessTokenData(intuneBrokerJSONData)) && (hasAccountData(intuneBrokerJSONData)) && (hasExpiresOn(intuneBrokerJSONData))) { // Details exist processIntuneResponse(intuneBrokerJSONData); // Return that we are authenticated return true; } else { // no ... expected values not available addLogEntry("Required JSON elements are not present in the Intune JSON response"); return false; } } else { // Not a valid JSON response addLogEntry("Invalid JSON Intune JSON response when attempting access authentication"); return false; } } else { // The account information is available in a saved file. Read this file in and attempt a silent authentication try { appConfig.intuneAccountDetails = strip(readText(appConfig.intuneAccountDetailsFilePath)); // Is the 'intune_account' empty? if (appConfig.intuneAccountDetails.empty) { addLogEntry("The 'intune_account' file exists but is empty: " ~ appConfig.intuneAccountDetailsFilePath); // No .. remove the file and perform the interactive authentication safeRemove(appConfig.intuneAccountDetailsFilePath); // Attempt interactive authentication authorise(); return true; } else { // We have loaded some Intune Account details, try and use them auto intuneAuthResult = acquire_token_silently(appConfig.intuneAccountDetails, appConfig.getValueString("application_id")); JSONValue intuneBrokerJSONData = intuneAuthResult.brokerTokenResponse; // Is the JSON data valid? if ((intuneBrokerJSONData.type() == JSONType.object)) { // Does the JSON data have the required authentication elements: // - accessToken // - account // - expiresOn if ((hasAccessTokenData(intuneBrokerJSONData)) && (hasAccountData(intuneBrokerJSONData)) && (hasExpiresOn(intuneBrokerJSONData))) { // Details exist processIntuneResponse(intuneBrokerJSONData); // Return that we are authenticated return true; } else { // no ... expected values not available addLogEntry("Required JSON elements are not present in the Intune JSON response"); return false; } } else { // No .. remove the file and perform the interactive authentication safeRemove(appConfig.intuneAccountDetailsFilePath); // Attempt interactive authentication authorise(); return true; } } } catch (FileException exception) { return false; } catch (std.utf.UTFException exception) { // path contains characters which generate a UTF exception addLogEntry("Cannot read 'intune_account' file on disk from: " ~ appConfig.intuneAccountDetailsFilePath); addLogEntry(" Error Reason:" ~ exception.msg); return false; } } } else { // There are 2 options here for normal authentication flow // 1. Use OAuth2 Device Authorisation Flow // 2. Use OAuth2 Interactive Authorisation Flow (application default) string authoriseApplicationRequest = "Please authorise this application by visiting the following URL:\n"; if (appConfig.getValueBool("use_device_auth")) { // Use OAuth2 Device Authorisation Flow // * deviceAuthUrl: Should already be configured based on client configuration // * tokenUrl: Should already be configured based on client configuration // * authScope: Should already be configured with the correct auth scopes string deviceAuthPostData = "client_id=" ~ clientId ~ authScope; // Initiating Device Code Request JSONValue deviceAuthResponse = initiateDeviceAuthorisation(deviceAuthPostData); // Was a valid JSON response provided? if (deviceAuthResponse.type() == JSONType.object) { // A valid JSON was returned // Extract required values string deviceCode = deviceAuthResponse["device_code"].str; string deviceAuthUrl = deviceAuthResponse["verification_uri"].str; string userCode = deviceAuthResponse["user_code"].str; long expiresIn = deviceAuthResponse["expires_in"].integer; long pollInterval = deviceAuthResponse["interval"].integer; SysTime expiresAt = Clock.currTime + dur!"seconds"(expiresIn); expiresAt.fracSecs = Duration.zero; // Display the required items for the user to action addLogEntry(); addLogEntry(authoriseApplicationRequest, ["consoleOnly"]); addLogEntry(deviceAuthUrl ~ "\n", ["consoleOnly"]); addLogEntry("Enter the following code when prompted: " ~ userCode, ["consoleOnly"]); addLogEntry(); addLogEntry("This code expires at: " ~ to!string(expiresAt), ["consoleOnly"]); addLogEntry(); // JSON value to store the poll response data JSONValue deviceAuthPollResponse; // Construct the polling post submission data string pollPostData = format( "client_id=%s&grant_type=urn%%3Aietf%%3Aparams%%3Aoauth%%3Agrant-type%%3Adevice_code&device_code=%s", clientId, deviceCode ); // Poll Microsoft API for authentication to be performed, until the expiry of this device authentication request while (Clock.currTime < expiresAt) { // Try the post to poll if the authentication has been done try { deviceAuthPollResponse = post(tokenUrl, pollPostData, null, true, "application/x-www-form-urlencoded"); // No error ... break out of the loop so the returned data can be processed break; } catch (OneDriveException e) { // Get the polling error JSON response JSONValue errorResponse = e.error; string errorType; if ("error" in errorResponse) { errorType = errorResponse["error"].str; if (errorType == "authorization_pending") { // Calculate remaining time Duration timeRemaining = expiresAt - Clock.currTime; long minutes = timeRemaining.total!"minutes"(); long seconds = timeRemaining.total!"seconds"() % 60; // Log countdown and status addLogEntry(format("[%02dm %02ds remaining] Still pending authorisation ...", minutes, seconds)); } else if (errorType == "authorization_declined") { addLogEntry("Authorisation was declined by the user."); // return false if we get to this point // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } else if (errorType == "expired_token") { addLogEntry("Device code expired before authorisation was completed."); // return false if we get to this point // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } else { addLogEntry("Unhandled error during polling: " ~ errorType); // return false if we get to this point // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } } else { addLogEntry("Unexpected error response from token polling."); // return false if we get to this point // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } } // Sleep until next polling interval Thread.sleep(dur!"seconds"(pollInterval)); } // Broken out of the polling loop // Was a valid JSON response provided? if (deviceAuthPollResponse.type() == JSONType.object) { // is the required 'access_token' available? if ("access_token" in deviceAuthPollResponse) { // We got the applicable access token addLogEntry("Access token acquired!"); // Process this JSON data processAuthenticationJSON(deviceAuthPollResponse); // Return that we are authorised return true; } } // return false if we get to this point // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } else { // No valid JSON response was returned // set 'use_device_auth' to false to fall back to interactive authentication flow appConfig.setValueBool("use_device_auth" , false); return false; } } else { // Use OAuth2 Interactive Authorisation Flow (application default) char[] response; // What URL should be presented to the user to access string url = authUrl ~ "?client_id=" ~ clientId ~ authScope ~ redirectUrl; // Configure automated authentication if --auth-files authUrlFilePath:responseUrlFilePath is being used string authFilesString = appConfig.getValueString("auth_files"); string authResponseString = appConfig.getValueString("auth_response"); if (!authResponseString.empty) { // read the response from authResponseString response = cast(char[]) authResponseString; } else if (authFilesString != "") { string[] authFiles = authFilesString.split(":"); string authUrlFilePath = authFiles[0]; string responseUrlFilePath = authFiles[1]; try { auto authUrlFile = File(authUrlFilePath, "w"); authUrlFile.write(url); authUrlFile.close(); } catch (FileException exception) { // There was a file system error // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, authUrlFilePath); // Must force exit here, allow logging to be done forceExit(); } catch (ErrnoException exception) { // There was a file system error // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, authUrlFilePath); // Must force exit here, allow logging to be done forceExit(); } // Log we are now waiting addLogEntry("Client requires authentication before proceeding. Waiting for --auth-files elements to be available."); while (!exists(responseUrlFilePath)) { Thread.sleep(dur!("msecs")(100)); } // read response from provided from OneDrive try { response = cast(char[]) read(responseUrlFilePath); } catch (OneDriveException exception) { // exception generated displayOneDriveErrorMessage(exception.msg, thisFunctionName); return false; } // try to remove auth files one at a time try { std.file.remove(authUrlFilePath); } catch (FileException exception) { addLogEntry("Cannot remove --auth-files elements - details below"); // There was a file system error - display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, authUrlFilePath); return false; } try { std.file.remove(responseUrlFilePath); } catch (FileException exception) { addLogEntry("Cannot remove --auth-files elements - details below"); // There was a file system error - display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, responseUrlFilePath); return false; } } else { // If we are not running in --dry-run mode, prompt the user to authorise the application if (!appConfig.getValueBool("dry_run")) { // Notify the user of the next step: visit the URL to authorise the client addLogEntry(); addLogEntry(authoriseApplicationRequest, ["consoleOnly"]); addLogEntry(url ~ "\n", ["consoleOnly"]); // Prompt the user to paste the full redirect URI (copied from the browser after login) addLogEntry("After completing the authorisation in your browser, copy the full redirect URI (from the address bar) and paste it below.\n", ["consoleOnly"]); addLogEntry("Paste redirect URI here: ", ["consoleOnlyNoNewLine"]); // Read the user's pasted response URI readln(response); // Flag that a response URI has been received - at this point could be valid or invalid appConfig.applicationAuthoriseResponseURIReceived = true; } else { // The application cannot be authorised when using --dry-run as we have to write out the authentication data, which negates the whole 'dry-run' process addLogEntry(); addLogEntry("The application requires authorisation, which involves saving authentication data on your system. Application authorisation cannot be completed when using the '--dry-run' option."); addLogEntry(); addLogEntry("To authorise the application please use your original command without '--dry-run'."); addLogEntry(); addLogEntry("To exclusively authorise the application without performing any additional actions, do not add '--sync' or '--monitor' to your command line."); addLogEntry(); forceExit(); } } // match the authorisation code auto c = matchFirst(strip(response), r"(?:[?&]code=)([^&]+)"); if (c.empty) { addLogEntry("An empty or invalid response uri was entered"); return false; } c.popFront(); // skip the whole match string authCode = decodeComponent(c.front); redeemToken(authCode); return true; } } } // Process Intune JSON response data void processIntuneResponse(JSONValue intuneBrokerJSONData) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Use the provided JSON data and configure elements, save JSON data to disk for reuse long expiresOnMs = intuneBrokerJSONData["expiresOn"].integer(); // Convert to SysTime SysTime expiryTime = SysTime.fromUnixTime(expiresOnMs / 1000); // Store in appConfig (to match standard flow) appConfig.accessTokenExpiration = expiryTime; addLogEntry("Intune access token expires at: " ~ to!string(appConfig.accessTokenExpiration)); // Configure the 'accessToken' based on Intune response appConfig.accessToken = "bearer " ~ strip(intuneBrokerJSONData["accessToken"].str); // Do we print the current access token debugOutputAccessToken(); // In order to support silent renewal of the access token, when the access token expires, we must store the Intune account data appConfig.intuneAccountDetails = to!string(intuneBrokerJSONData["account"]); // try and update the 'intune_account' file on disk for reuse later try { if (debugLogging) {addLogEntry("Updating 'intune_account' on disk", ["debug"]);} std.file.write(appConfig.intuneAccountDetailsFilePath, appConfig.intuneAccountDetails); if (debugLogging) {addLogEntry("Setting file permissions for: " ~ appConfig.intuneAccountDetailsFilePath, ["debug"]);} appConfig.intuneAccountDetailsFilePath.setAttributes(appConfig.returnSecureFilePermission()); } catch (FileException exception) { // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, appConfig.intuneAccountDetailsFilePath); } } // Initiate OAuth2 Device Authorisation JSONValue initiateDeviceAuthorisation(string deviceAuthPostData) { // Device OAuth2 Device Authorisation requires a HTTP POST return post(deviceAuthUrl, deviceAuthPostData, null, true, "application/x-www-form-urlencoded"); } // Do we print the current access token void debugOutputAccessToken() { if (appConfig.verbosityCount > 1) { if (appConfig.getValueBool("debug_https")) { if (appConfig.getValueBool("print_token")) { // This needs to be highly restricted in output .... if (debugLogging) {addLogEntry("CAUTION - KEEP THIS SAFE: Current access token: " ~ to!string(appConfig.accessToken), ["debug"]);} } } } } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/drive_get JSONValue getDefaultDriveDetails() { string url; url = driveUrl; return get(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get JSONValue getDefaultRootDetails() { string url; url = driveUrl ~ "/root"; return get(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get JSONValue getDriveIdRoot(string driveId) { string url; url = driveByIdUrl ~ driveId ~ "/root"; return get(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/drive_get JSONValue getDriveQuota(string driveId) { string url; url = driveByIdUrl ~ driveId ~ "/"; url ~= "?select=quota"; return get(url); } // Return the details of the specified path, by giving the path we wish to query // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get JSONValue getPathDetails(string path) { string url; if ((path == ".")||(path == "/")) { url = driveUrl ~ "/root/"; } else { url = itemByPathUrl ~ encodeComponent(path) ~ ":/"; } // Add select clause url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,createdBy,lastModifiedBy,package"; return get(url); } // Return the details of the specified item based on its driveID and itemID // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get JSONValue getPathDetailsById(string driveId, string id) { string url; url = driveByIdUrl ~ driveId ~ "/items/" ~ id; url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,createdBy,lastModifiedBy,webUrl,lastModifiedDateTime,package"; return get(url); } // Return all the items that are shared with the user // https://docs.microsoft.com/en-us/graph/api/drive-sharedwithme JSONValue getSharedWithMe() { return get(sharedWithMeUrl); } // Create a shareable link for an existing file on OneDrive based on the accessScope JSON permissions // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createlink JSONValue createShareableLink(string driveId, string id, JSONValue accessScope) { string url; url = driveByIdUrl ~ driveId ~ "/items/" ~ id ~ "/createLink"; return post(url, accessScope.toString()); } // Return the requested details of the specified path on the specified drive id and path // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get JSONValue getPathDetailsByDriveId(string driveId, string path) { string url; // https://learn.microsoft.com/en-us/onedrive/developer/rest-api/concepts/addressing-driveitems?view=odsp-graph-online // Required format: /drives/{drive-id}/root:/{item-path}: url = driveByIdUrl ~ driveId ~ "/root:/" ~ encodeComponent(path) ~ ":"; url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,createdBy,lastModifiedBy,package"; return get(url); } // Track changes for a given driveId // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delta // Your app begins by calling delta without any parameters. The service starts enumerating the drive's hierarchy, returning pages of items and either an @odata.nextLink or an @odata.deltaLink, as described below. // Your app should continue calling with the @odata.nextLink until you no longer see an @odata.nextLink returned, or you see a response with an empty set of changes. // After you have finished receiving all the changes, you may apply them to your local state. To check for changes in the future, call delta again with the @odata.deltaLink from the previous successful response. JSONValue getChangesByItemId(string driveId, string id, string deltaLink) { string[string] requestHeaders; // From March 1st 2025, this needs to be added to ensure that Shared Folders are sent in the Delta Query Response if (appConfig.accountType == "personal") { // OneDrive Personal Account addIncludeFeatureRequestHeader(&requestHeaders); } else { // Business or SharePoint Library // Only add if configured to do so if (appConfig.getValueBool("sync_business_shared_items")) { // Feature enabled, add headers addIncludeFeatureRequestHeader(&requestHeaders); } } string url; // configure deltaLink to query if (deltaLink.empty) { url = driveByIdUrl ~ driveId ~ "/items/" ~ id ~ "/delta"; // Reduce what we ask for in the response - which reduces the data transferred back to us, and reduces what is held in memory during initial JSON processing url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,createdBy,lastModifiedBy,package"; } else { url = deltaLink; } // get the response return get(url, false, requestHeaders); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_list_children JSONValue listChildren(string driveId, string id, string nextLink) { string[string] requestHeaders; // From March 1st 2025, this needs to be added to ensure that Shared Folders are sent in the Delta Query Response if (appConfig.accountType == "personal") { // OneDrive Personal Account addIncludeFeatureRequestHeader(&requestHeaders); } else { // Business or SharePoint Library // Only add if configured to do so if (appConfig.getValueBool("sync_business_shared_items")) { // Feature enabled, add headers addIncludeFeatureRequestHeader(&requestHeaders); } } string url; // configure URL to query if (nextLink.empty) { url = driveByIdUrl ~ driveId ~ "/items/" ~ id ~ "/children"; url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,createdBy,lastModifiedBy,package"; } else { url = nextLink; } return get(url, false, requestHeaders); } // https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_search JSONValue searchDriveForPath(string driveId, string path) { // OData string literal escaping: a single quote inside a '...' literal becomes doubled. // Then URL-encode for safe transport auto odataSafe = path.replace("'", "''"); auto encoded = encodeComponent(odataSafe); string url; url = "https://graph.microsoft.com/v1.0/drives/" ~ driveId ~ "/root/search(q='" ~ encoded ~ "')"; return get(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_update JSONValue updateById(const(char)[] driveId, const(char)[] id, JSONValue data, const(char)[] eTag = null) { string[string] requestHeaders; const(char)[] url = driveByIdUrl ~ driveId ~ "/items/" ~ id; if (eTag) requestHeaders["If-Match"] = to!string(eTag); return patch(url, data.toString(), false, requestHeaders); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delete void deleteById(const(char)[] driveId, const(char)[] id, const(char)[] eTag = null) { // string[string] requestHeaders; const(char)[] url = driveByIdUrl ~ driveId ~ "/items/" ~ id; //TODO: investigate why this always fail with 412 (Precondition Failed) // if (eTag) requestHeaders["If-Match"] = eTag; performDelete(url); } // https://learn.microsoft.com/en-us/graph/api/driveitem-permanentdelete?view=graph-rest-1.0 void permanentDeleteById(const(char)[] driveId, const(char)[] id, const(char)[] eTag = null) { // string[string] requestHeaders; const(char)[] url = driveByIdUrl ~ driveId ~ "/items/" ~ id ~ "/permanentDelete"; //TODO: investigate why this always fail with 412 (Precondition Failed) // if (eTag) requestHeaders["If-Match"] = eTag; // as per documentation, a permanentDelete needs to be a HTTP POST performPermanentDelete(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_post_children JSONValue createById(string parentDriveId, string parentId, JSONValue item) { string url = driveByIdUrl ~ parentDriveId ~ "/items/" ~ parentId ~ "/children"; return post(url, item.toString()); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content JSONValue simpleUpload(string localPath, string parentDriveId, string parentId, string filename) { string url = driveByIdUrl ~ parentDriveId ~ "/items/" ~ parentId ~ ":/" ~ encodeComponent(filename) ~ ":/content"; return put(url, localPath); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content JSONValue simpleUploadReplace(string localPath, string driveId, string id) { string url = driveByIdUrl ~ driveId ~ "/items/" ~ id ~ "/content"; return put(url, localPath); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession //JSONValue createUploadSession(string parentDriveId, string parentId, string filename, string eTag = null, JSONValue item = null) { JSONValue createUploadSession(string parentDriveId, string parentId, string filename, const(char)[] eTag = null, JSONValue item = null) { string[string] requestHeaders; string url = driveByIdUrl ~ parentDriveId ~ "/items/" ~ parentId ~ ":/" ~ encodeComponent(filename) ~ ":/createUploadSession"; // eTag If-Match header addition commented out for the moment // At some point, post the creation of this upload session the eTag is being 'updated' by OneDrive, thus when uploadFragment() is used // this generates a 412 Precondition Failed and then a 416 Requested Range Not Satisfiable // This needs to be investigated further as to why this occurs if (eTag) requestHeaders["If-Match"] = to!string(eTag); return post(url, item.toString(), requestHeaders); } // https://learn.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#upload-bytes-to-the-upload-session JSONValue uploadFragment(string uploadUrl, string filepath, long offset, long offsetSize, long fileSize) { // If we upload a modified file, with the current known online eTag, this gets changed when the session is started - thus, the tail end of uploading // a fragment fails with a 412 Precondition Failed and then a 416 Requested Range Not Satisfiable // For the moment, comment out adding the If-Match header in createUploadSession, which then avoids this issue string contentRange = "bytes " ~ to!string(offset) ~ "-" ~ to!string(offset + offsetSize - 1) ~ "/" ~ to!string(fileSize); if (debugLogging) { addLogEntry("fragment contentRange: " ~ contentRange, ["debug"]); } // Before we submit this 'HTTP PUT' request, pre-emptively check token expiry to avoid future 401s during long uploads checkAccessTokenExpired(); // Perform the HTTP PUT action to upload the file fragment return put(uploadUrl, filepath, true, contentRange, offset, offsetSize); } // https://learn.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#resuming-an-in-progress-upload JSONValue requestUploadStatus(string uploadUrl) { return get(uploadUrl, true); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/site_search?view=odsp-graph-online JSONValue o365SiteSearch(string nextLink) { string url; // configure URL to query if (nextLink.empty) { url = siteSearchUrl ~ "=*"; } else { url = nextLink; } return get(url); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/drive_list?view=odsp-graph-online JSONValue o365SiteDrives(string site_id, string nextLink){ string url; // configure URL to query if (nextLink.empty) { url = siteDriveUrl ~ site_id ~ "/drives"; } else { url = nextLink; } return get(url); } // Create Webhook Subscription JSONValue createSubscription(string notificationUrl, SysTime expirationDateTime) { string driveId; string url = subscriptionUrl; // What do we set for driveId if (appConfig.getValueString("drive_id").length) { // Use the 'config' file option driveId = appConfig.getValueString("drive_id"); } else { // use appConfig.defaultDriveId driveId = appConfig.defaultDriveId; } // Create a resource item based on if we have a driveId now configured string resourceItem; if (driveId.length) { resourceItem = "/drives/" ~ driveId ~ "/root"; } else { resourceItem = "/me/drive/root"; } // create JSON request to create webhook subscription const JSONValue request = [ "changeType": "updated", "notificationUrl": notificationUrl, "resource": resourceItem, "expirationDateTime": expirationDateTime.toISOExtString(), "clientState": randomUUID().toString() ]; return post(url, request.toString()); } // Renew Webhook Subscription JSONValue renewSubscription(string subscriptionId, SysTime expirationDateTime) { string url; url = subscriptionUrl ~ "/" ~ subscriptionId; const JSONValue request = [ "expirationDateTime": expirationDateTime.toISOExtString() ]; return patch(url, request.toString(), true); } // Delete Webhook subscription void deleteSubscription(string subscriptionId) { string url; url = subscriptionUrl ~ "/" ~ subscriptionId; performDelete(url); } // Obtain the Websocket Notification URL JSONValue obtainWebSocketNotificationURL() { if (debugLogging) {addLogEntry("Request a Socket.IO Subscription Endpoint: " ~ websocketEndpoint, ["debug"]);} return get(websocketEndpoint); } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get_content void downloadById(const(char)[] driveId, const(char)[] itemId, string saveToPath, long fileSize, JSONValue onlineHash, long resumeOffset = 0) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // We pass through to 'downloadFile()' // - resumeOffset // - onlineHash // - driveId // - itemId scope(failure) { if (exists(saveToPath)) { // try and remove the file, catch error try { remove(saveToPath); } catch (FileException exception) { // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, saveToPath); } } } // Create the required local parental path structure if this does not exist string parentalPath = dirName(saveToPath); // Does the parental path exist locally? if (!exists(parentalPath)) { try { if (debugLogging) {addLogEntry("Requested local parental path does not exist, creating directory structure: " ~ parentalPath, ["debug"]);} mkdirRecurse(parentalPath); // Has the user disabled the setting of filesystem permissions? if (!appConfig.getValueBool("disable_permission_set")) { // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ parentalPath, ["debug"]);} parentalPath.setAttributes(appConfig.returnRequiredDirectoryPermissions()); } else { // Use inherited permissions if (debugLogging) {addLogEntry("Using inherited filesystem permissions for: " ~ parentalPath, ["debug"]);} } } catch (FileException exception) { // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, parentalPath); } } // Create the URL to download the file const(char)[] url = driveByIdUrl ~ driveId ~ "/items/" ~ itemId ~ "/content?AVOverride=1"; // Download file using the URL created above downloadFile(driveId, itemId, url, saveToPath, fileSize, onlineHash, resumeOffset); // Does downloaded file now exist locally? if (exists(saveToPath)) { // Has the user disabled the setting of filesystem permissions? if (!appConfig.getValueBool("disable_permission_set")) { // File was downloaded successfully - configure the applicable permissions for the file if (debugLogging) {addLogEntry("Setting file permissions for: " ~ saveToPath, ["debug"]);} saveToPath.setAttributes(appConfig.returnRequiredFilePermissions()); } else { // Use inherited permissions if (debugLogging) {addLogEntry("Using inherited filesystem permissions for: " ~ saveToPath, ["debug"]);} } } } // Return the actual siteSearchUrl being used and/or requested when performing 'siteQuery = onedrive.o365SiteSearch(nextLink);' call string getSiteSearchUrl() { return siteSearchUrl; } // Private OneDrive API Functions private void addIncludeFeatureRequestHeader(string[string]* headers) { if (appConfig.accountType == "personal") { // Add logging message for OneDrive Personal Accounts if (debugLogging) {addLogEntry("Adding 'Include-Feature=AddToOneDrive' API request header for OneDrive Personal Account Type", ["debug"]);} } else { // Add logging message for OneDrive Business Accounts if (debugLogging) {addLogEntry("Adding 'Include-Feature=AddToOneDrive' API request header as 'sync_business_shared_items' config option is enabled", ["debug"]);} } // Add feature to request headers (*headers)["Prefer"] = "Include-Feature=AddToOneDrive"; } private void redeemToken(string authCode) { string postData = "client_id=" ~ clientId ~ "&redirect_uri=" ~ encodeComponent(redirectUrl) ~ "&code=" ~ encodeComponent(authCode) ~ "&grant_type=authorization_code"; acquireToken(postData.dup); } private void acquireToken(char[] postData) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Configure the response JSON JSONValue response; // Log what we are doing if (debugLogging) { addLogEntry("acquireToken: requesting new access token using refresh token (value redacted)", ["debug"]); } // Try and process the 'postData' content try { response = post(tokenUrl, postData, null, true, "application/x-www-form-urlencoded"); } catch (OneDriveException exception) { // an error was generated if ((exception.httpStatusCode == 400) || (exception.httpStatusCode == 401)) { // Release curl engine releaseCurlEngine(); // Handle an unauthorised client handleClientUnauthorised(exception.httpStatusCode, exception.error); // Must force exit here, allow logging to be done forceExit(); } else { if (exception.httpStatusCode >= 500) { // There was a HTTP 5xx Server Side Error - retry acquireToken(postData); } else { displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } } if (response.type() == JSONType.object) { // Debug the provided response if (debugLogging) { string scopes = ("scope" in response) ? response["scope"].str() : ""; string tokenType = ("token_type" in response) ? response["token_type"].str() : ""; long expiresIn = ("expires_in" in response) ? response["expires_in"].integer() : -1; addLogEntry("acquireToken post response: token_type=" ~ tokenType ~ ", expires_in=" ~ to!string(expiresIn) ~ ", scope=" ~ scopes, ["debug"]); } // Has the client been configured to use read_only_auth_scope if (appConfig.getValueBool("read_only_auth_scope")) { // read_only_auth_scope has been configured if ("scope" in response){ string effectiveScopes = response["scope"].str(); // Display the effective authentication scopes addLogEntry(); if (verboseLogging) {addLogEntry("Effective API Authentication Scopes: " ~ effectiveScopes, ["verbose"]);} // if we have any write scopes, we need to tell the user to update an remove online prior authentication and exit application if (canFind(effectiveScopes, "Write")) { // effective scopes contain write scopes .. so not a read-only configuration addLogEntry(); addLogEntry("ERROR: You have authentication scopes that allow write operations. You need to remove your existing application access consent"); addLogEntry(); addLogEntry("Please login to https://account.live.com/consent/Manage and remove your existing application access consent"); addLogEntry(); // force exit releaseCurlEngine(); // Must force exit here, allow logging to be done forceExit(); } } } if ("access_token" in response) { // Process the response JSON processAuthenticationJSON(response); } else { // Release curl engine releaseCurlEngine(); // Log error message addLogEntry("\nInvalid authentication response from OneDrive. Please check the response uri\n"); // re-authorize authorise(); } } else { // Release curl engine releaseCurlEngine(); addLogEntry("Invalid response from the Microsoft Graph API. Unable to initialise OneDrive API instance."); // Must force exit here, allow logging to be done forceExit(); } } // Process the authentication JSON private void processAuthenticationJSON(JSONValue response) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Use 'access_token' and set in the application configuration appConfig.accessToken = "bearer " ~ strip(response["access_token"].str); // Do we print the current access token debugOutputAccessToken(); // Obtain the 'refresh_token' and its expiry refreshToken = strip(response["refresh_token"].str); appConfig.accessTokenExpiration = Clock.currTime() + dur!"seconds"(response["expires_in"].integer()); // Debug this response if (debugLogging) {addLogEntry("appConfig.accessTokenExpiration = " ~ to!string(appConfig.accessTokenExpiration), ["debug"]);} if (!dryRun) { // Update the refreshToken in appConfig so that we can reuse it if (appConfig.refreshToken.empty) { // The access token is empty if (debugLogging) {addLogEntry("Updating appConfig.refreshToken with new refreshToken as appConfig.refreshToken is empty", ["debug"]);} appConfig.refreshToken = refreshToken; } else { // Is the access token different? if (appConfig.refreshToken != refreshToken) { // Update the memory version if (debugLogging) {addLogEntry("Updating appConfig.refreshToken with updated refreshToken", ["debug"]);} appConfig.refreshToken = refreshToken; } } // try and update the 'refresh_token' file on disk try { if (debugLogging) {addLogEntry("Updating 'refresh_token' on disk", ["debug"]);} std.file.write(appConfig.refreshTokenFilePath, refreshToken); if (debugLogging) {addLogEntry("Setting file permissions for: " ~ appConfig.refreshTokenFilePath, ["debug"]);} appConfig.refreshTokenFilePath.setAttributes(appConfig.returnSecureFilePermission()); } catch (FileException exception) { // display the error message displayFileSystemErrorMessage(exception.msg, thisFunctionName, appConfig.refreshTokenFilePath); } } } private void generateNewAccessToken() { if (debugLogging) {addLogEntry("Need to generate a new access token for Microsoft OneDrive", ["debug"]);} // Has the client been configured to use Intune SSO via Microsoft Identity Broker (microsoft-identity-broker) dbus session if (appConfig.getValueBool("use_intune_sso")) { // The client is configured to use Intune SSO via Microsoft Identity Broker dbus session auto intuneAuthResult = acquire_token_silently(appConfig.intuneAccountDetails, appConfig.getValueString("application_id")); JSONValue intuneBrokerJSONData = intuneAuthResult.brokerTokenResponse; // Is the JSON data valid? if ((intuneBrokerJSONData.type() == JSONType.object)) { // Does the JSON data have the required renewal elements: // - accessToken // - account // - expiresOn if ((hasAccessTokenData(intuneBrokerJSONData)) && (hasAccountData(intuneBrokerJSONData)) && (hasExpiresOn(intuneBrokerJSONData))) { // Details exist processIntuneResponse(intuneBrokerJSONData); } else { // no ... expected values not available addLogEntry("Required Intune JSON elements are not present in the Intune JSON response"); } } else { // Not a valid JSON response addLogEntry("Invalid Intune JSON response when attempting access token renewal"); } } else { // Normal authentication method auto postData = appender!(string)(); postData ~= "client_id=" ~ clientId; postData ~= "&redirect_uri=" ~ redirectUrl; postData ~= "&refresh_token=" ~ to!string(refreshToken); postData ~= "&grant_type=refresh_token"; acquireToken(postData.data.dup); } } // Check if the existing access token has expired, if it has, generate a new one private void checkAccessTokenExpired() { if (Clock.currTime() >= appConfig.accessTokenExpiration) { if (debugLogging) {addLogEntry("Microsoft OneDrive OAuth2 Access Token has expired. Must generate a new Microsoft OneDrive OAuth2 Access Token", ["debug"]);} generateNewAccessToken(); } else { if (debugLogging) {addLogEntry("Microsoft OneDrive OAuth2 Access Token Valid Until (Local): " ~ to!string(appConfig.accessTokenExpiration), ["debug"]);} } } private string getAccessToken() { checkAccessTokenExpired(); return to!string(appConfig.accessToken); } private void addAccessTokenHeader(string[string]* requestHeaders) { (*requestHeaders)["Authorization"] = getAccessToken(); } private void connect(HTTP.Method method, const(char)[] url, bool skipToken, CurlResponse response, string[string] requestHeaders=null) { // If we are debug logging, output the URL being accessed and the HTTP method being used to access that URL if (debugLogging) {addLogEntry("HTTP " ~ to!string(method) ~ " request to URL: " ~ to!string(url), ["debug"]);} // Check access token first in case the request is overridden if (!skipToken) addAccessTokenHeader(&requestHeaders); curlEngine.setResponseHolder(response); foreach(k, v; requestHeaders) { curlEngine.addRequestHeader(k, v); } curlEngine.connect(method, url); } private void performDelete(const(char)[] url, string[string] requestHeaders=null, string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = false; oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.del, url, false, response, requestHeaders); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } private void performPermanentDelete(const(char)[] url, string[string] requestHeaders=null, string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = false; oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.post, url, false, response, requestHeaders); curlEngine.setZeroContentLength(); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } // Download a file based on the URL request private void downloadFile(const(char)[] driveId, const(char)[] itemId, const(char)[] url, string filename, long fileSize, JSONValue onlineHash, long resumeOffset = 0, string callingFunction=__FUNCTION__, int lineno=__LINE__) { // Threshold for displaying download bar long thresholdFileSize = 4 * 2^^20; // 4 MiB // To support marking of partially-downloaded files string originalFilename = filename; string downloadFilename = filename ~ ".partial"; // To support resumable downloads, configure the 'resumable data' file path string threadResumeDownloadFilePath = appConfig.resumeDownloadFilePath ~ "." ~ generateAlphanumericString(); // Create a JSONValue with download state so this can be used when resuming, to evaluate if the online file has changed, and if we are able to resume in a safe manner JSONValue resumeDownloadData = JSONValue([ "driveId": JSONValue(to!string(driveId)), "itemId": JSONValue(to!string(itemId)), "onlineHash": onlineHash, "originalFilename": JSONValue(originalFilename), "downloadFilename": JSONValue(downloadFilename), "resumeOffset": JSONValue(to!string(resumeOffset)) ]); // ---------------------------------------------------------------------- // Progress state – must live for the whole downloadFile() call so that // retries triggered by oneDriveErrorHandlerWrapper() do NOT reset the // visible progress bar back to 0%. // ---------------------------------------------------------------------- size_t expected_total_segments = 20; SysTime startTime = Clock.currTime(); long start_unix_time = startTime.toUnixTime(); int h, m, s; string etaString; bool barInit = false; real previousProgressPercent = 0.0; // last *displayed* percent real percentCheck = 5.0; size_t segmentCount = 0; // Validate the JSON response bool validateJSONResponse = false; oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.get, url, false, response); if (fileSize >= thresholdFileSize) { // ------------------------------------------------------------------ // Determine an effective resume offset for this attempt. // // - Start from the passed-in resumeOffset (from resume_download.*) // - If a .partial file exists and is larger, prefer its size. // This ensures we never re-download bytes we already have on disk. // ------------------------------------------------------------------ long effectiveResumeOffset = resumeOffset; if (exists(downloadFilename)) { try { auto partialSize = cast(long) getSize(downloadFilename); if (partialSize > effectiveResumeOffset) { if (debugLogging) { addLogEntry( "Resumable download: detected existing partial file '" ~ downloadFilename ~ "' of size " ~ to!string(partialSize) ~ " bytes", ["debug"] ); addLogEntry( "Adjusting resumable offset for '" ~ originalFilename ~ "' from " ~ to!string(effectiveResumeOffset) ~ " to " ~ to!string(partialSize), ["debug"] ); } effectiveResumeOffset = partialSize; } } catch (FileException ex) { if (debugLogging) { addLogEntry( "Failed to obtain size of partial download file '" ~ downloadFilename ~ "': " ~ ex.msg, ["debug"] ); } } } // If we have a resumable offset to use, set this as the offset to use if (effectiveResumeOffset > 0) { curlEngine.setDownloadResumeOffset(effectiveResumeOffset); // Keep the JSON state in sync with the absolute offset resumeDownloadData["resumeOffset"] = JSONValue(to!string(effectiveResumeOffset)); } // Setup progress bar to display curlEngine.http.onProgress = delegate int(size_t dltotal, size_t dlnow, size_t ultotal, size_t ulnow) { string downloadLogEntry = "Downloading: " ~ filename ~ " ... "; // ------------------------------------------------------------------ // Compute absolute progress as bytes_on_disk + bytes_this_transfer. // This ensures that after a retry, the percentage continues from // (for example) 25% instead of restarting at 0%. // ------------------------------------------------------------------ long absoluteNow = effectiveResumeOffset + cast(long)dlnow; long absoluteTotal; if (fileSize > 0) { absoluteTotal = fileSize; } else if (dltotal > 0) { absoluteTotal = effectiveResumeOffset + cast(long)dltotal; } else { absoluteTotal = absoluteNow; // best effort; avoids div-by-zero } if (absoluteTotal <= 0) { absoluteTotal = 1; // safety guard } // Floor to nearest whole number real currentDLPercent = floor( (cast(real) absoluteNow / cast(real) absoluteTotal) * 100.0 ); // Clamp just in case if (currentDLPercent < 0.0) { currentDLPercent = 0.0; } else if (currentDLPercent > 100.0) { currentDLPercent = 100.0; } // Debug logging (optional, but handy while we’re testing) if (debugLogging) { addLogEntry("", ["debug"]); addLogEntry("absoluteNow = " ~ to!string(absoluteNow), ["debug"]); addLogEntry("absoluteTotal = " ~ to!string(absoluteTotal), ["debug"]); addLogEntry("Percent Complete = " ~ to!string(currentDLPercent), ["debug"]); } // Have we started downloading (in absolute terms)? if (currentDLPercent > 0) { // Has the user set a data rate limit? // when using rate_limit, we will get odd download rates, for example: // Percent Complete = 24 // Data Received = 13080163 // Expected Total = 52428800 // Percent Complete = 24 // Data Received = 13685777 // Expected Total = 52428800 // Percent Complete = 26 <---- jumps to 26% missing 25%, thus fmod misses incrementing progress bar // Data Received = 13685777 // Expected Total = 52428800 // Percent Complete = 26 if (appConfig.getValueLong("rate_limit") > 0) { // Under rate limiting, libcurl can "jump" the visible percentage, // e.g. 24% -> 26%, which can skip a clean 5% boundary. // To keep a stable 5% display (5, 10, 15, ...), we use a // catch-up loop that prints every missing 5% step up to // currentDLPercent, based on the *absolute* percentage. real nextPercent = previousProgressPercent + percentCheck; // Emit all missing 5% steps below 100% while (nextPercent < 100.0 && currentDLPercent >= nextPercent) { if (debugLogging) { addLogEntry("Incrementing Progress Bar (rate_limit) to " ~ to!string(nextPercent) ~ "%", ["debug"]); } segmentCount++; etaString = formatETA(calc_eta(segmentCount, expected_total_segments, start_unix_time)); string percentage = leftJustify(to!string(cast(int) nextPercent) ~ "%", 5, ' '); addLogEntry(downloadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); previousProgressPercent = nextPercent; nextPercent += percentCheck; } // Handle 100% exactly once if ((currentDLPercent >= 100.0) && (previousProgressPercent < 100.0)) { SysTime endTime = Clock.currTime(); long end_unix_time = endTime.toUnixTime(); int download_duration = cast(int)(end_unix_time - start_unix_time); dur!"seconds"(download_duration).split!("hours", "minutes", "seconds")(h, m, s); etaString = format!"| DONE in %02d:%02d:%02d"(h, m, s); string percentage = leftJustify("100%", 5, ' '); addLogEntry(downloadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); previousProgressPercent = 100.0; } } else { // Non-rate-limited case: fmod-based behaviour but applied to the absolute percentage if ((isIdentical(fmod(currentDLPercent, percentCheck), 0.0)) && (previousProgressPercent != currentDLPercent)) { // currentDLPercent matches a new increment if (debugLogging) { addLogEntry("Incrementing Progress Bar using fmod match", ["debug"]); } if (currentDLPercent != 100) { // Not 100% yet segmentCount++; etaString = formatETA(calc_eta(segmentCount, expected_total_segments, start_unix_time)); string percentage = leftJustify(to!string(cast(int) currentDLPercent) ~ "%", 5, ' '); addLogEntry(downloadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); } else { // 100% done SysTime endTime = Clock.currTime(); long end_unix_time = endTime.toUnixTime(); int download_duration = cast(int)(end_unix_time - start_unix_time); dur!"seconds"(download_duration).split!("hours", "minutes", "seconds")(h, m, s); etaString = format!"| DONE in %02d:%02d:%02d"(h, m, s); string percentage = leftJustify("100%", 5, ' '); addLogEntry(downloadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); } previousProgressPercent = currentDLPercent; } } // Has our absolute offset advanced? if (absoluteNow > to!long(resumeDownloadData["resumeOffset"].str)) { // Update resumeOffset for this progress event with the latest absolute offset resumeDownloadData["resumeOffset"] = JSONValue(to!string(absoluteNow)); // Save resumable download data - this needs to be saved on every onProgress event that is processed saveResumeDownloadFile(threadResumeDownloadFilePath, resumeDownloadData); } } else { // We may get frequent progress callbacks at 0%, make sure we initialise the bar once per overall download if ((currentDLPercent == 0) && (!barInit)) { etaString = "| ETA --:--:--"; string percentage = leftJustify("0%", 5, ' '); addLogEntry(downloadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); barInit = true; } } return 0; }; } else { // No progress bar, no resumable download } // Capture the result of the download action auto result = curlEngine.download(originalFilename, downloadFilename); // Safe remove 'threadResumeDownloadFilePath' as if we get to this point, the file has been downloaded successfully safeRemove(threadResumeDownloadFilePath); // Reset this curlEngine offset value now that the file has been downloaded successfully curlEngine.resetDownloadResumeOffset(); // Return the applicable result return result; }, validateJSONResponse, callingFunction, lineno); } // Save the resume download data private void saveResumeDownloadFile(string threadResumeDownloadFilePath, JSONValue resumeDownloadData) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); try { std.file.write(threadResumeDownloadFilePath, resumeDownloadData.toString()); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, threadResumeDownloadFilePath); } } private JSONValue get(string url, bool skipToken = false, string[string] requestHeaders=null, string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = true; return oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.get, url, skipToken, response, requestHeaders); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } private JSONValue patch(const(char)[] url, const(char)[] patchData, bool validateJSONResponseInput, string[string] requestHeaders=null, const(char)[] contentType = "application/json", string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = validateJSONResponseInput; return oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.patch, url, false, response, requestHeaders); curlEngine.setContent(contentType, patchData); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } private JSONValue post(const(char)[] url, const(char)[] postData, string[string] requestHeaders=null, bool skipToken = false, const(char)[] contentType = "application/json", string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = true; return oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.post, url, skipToken, response, requestHeaders); curlEngine.setContent(contentType, postData); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } private JSONValue put(const(char)[] url, string filepath, bool skipToken=false, string contentRange=null, ulong offset=0, ulong offsetSize=0, string callingFunction=__FUNCTION__, int lineno=__LINE__) { bool validateJSONResponse = true; return oneDriveErrorHandlerWrapper((CurlResponse response) { connect(HTTP.Method.put, url, skipToken, response); curlEngine.setFile(filepath, contentRange, offset, offsetSize); return curlEngine.execute(); }, validateJSONResponse, callingFunction, lineno); } // Wrapper function for all requests to OneDrive API // - This should throw a OneDriveException so that this exception can be handled appropriately elsewhere in the application private JSONValue oneDriveErrorHandlerWrapper(CurlResponse delegate(CurlResponse response) executer, bool validateJSONResponse, string callingFunction, int lineno) { // Create a new 'curl' response response = new CurlResponse(); // Other wrapper variables int retryAttempts = 0; int baseBackoffInterval = 1; // Base backoff interval in seconds int maxRetryCount = 175200; // Approx 365 days based on maxBackoffInterval + appConfig.defaultDataTimeout //int maxRetryCount = 5; // Temp int maxBackoffInterval = 120; // Maximum backoff interval in seconds int thisBackOffInterval = 0; int timestampAlign = 0; JSONValue result; SysTime currentTime; SysTime retryTime; bool retrySuccess = false; bool transientError = false; bool sslVerifyPeerDisabled = false; while (!retrySuccess) { // Reset thisBackOffInterval thisBackOffInterval = 0; transientError = false; if (retryAttempts >= 1) { // re-try log entry & clock time retryTime = Clock.currTime(); retryTime.fracSecs = Duration.zero; addLogEntry("Retrying the respective Microsoft Graph API call for Internal Thread ID: " ~ to!string(curlEngine.internalThreadId) ~ " (Timestamp: " ~ to!string(retryTime) ~ ") ..."); } try { response.reset(); response = executer(response); // Check for a valid response if (response.hasResponse) { // Process the response result = response.json(); // Print response if 'debugHTTPSResponse' is flagged if (debugHTTPSResponse){ if (debugLogging) {addLogEntry("Microsoft Graph API Response: " ~ response.dumpResponse(), ["debug"]);} } // Check http response code, raise a OneDriveException if the operation was not successfully performed if (checkHttpResponseCode(response.statusLine.code)) { // 'curl' on platforms like Ubuntu does not reliably provide the 'http.statusLine.reason' when using HTTP/2 // This is a curl bug, but because Ubuntu uses old packages and never updates them, we are stuck with working around this bug if (response.statusLine.reason.length == 0) { // No 'reason', fetch what it should have been response.statusLine.reason = getMicrosoftGraphStatusMessage(response.statusLine.code); } // Why are throwing a OneDriveException - do not do this for a 404 error as this is not required as we use a 404 if things are not online, to create them if (response.statusLine.code != 404) { if (debugLogging) { addLogEntry("response.statusLine.code: " ~ to!string(response.statusLine.code), ["debug"]); addLogEntry("response.statusLine.reason: " ~ to!string(response.statusLine.reason), ["debug"]); addLogEntry("actual curl response: " ~ to!string(response), ["debug"]); } } // For every HTTP error status code, including those from 3xx (other Redirection codes excluding 302), 4xx (Client Error), and 5xx (Server Error) series, will trigger the following line of code. throw new OneDriveException(response.statusLine.code, response.statusLine.reason, response); } // Do we need to validate the JSON response? if (validateJSONResponse) { const code = response.statusLine.code; // 204 = No Content is a valid success response for some Graph operations (e.g. PATCH/DELETE). // In that case, there is no JSON payload to validate. if (code != 204) { // If caller expects JSON, an empty body is not acceptable if (response.content.length == 0) { throw new OneDriveException( 0, "Caller requested a JSON object response, but the response body was empty", response); } // Body is present: it must be a JSON object if (result.type() != JSONType.object) { throw new OneDriveException(0, "Caller requested a JSON object response, but the response was not a JSON object", response); } } } // If we get to this point, there is no error from http.perform() on re-try // If retryAttempts is greater than 1, it means we were re-trying the request if (retryAttempts > 1) { // unset the fresh connect option as this then creates performance issues if left enabled unsetFreshConnectOption(); } // On successful http.perform() processing, break out of the loop break; } else { // Throw a custom 506 error // Whilst this error code is a bit more esoteric and typically involves content negotiation issues that lead to a configuration error on the server, but it could be loosely // interpreted to signal that the response received didn't meet the expected criteria or format. throw new OneDriveException(506, "Received an unexpected response from Microsoft OneDrive", response); } // A 'curl' exception was thrown } catch (CurlException exception) { // Handle 'curl' exception errors // Detail the curl exception, debug output only if (debugLogging) { addLogEntry("Handling a curl exception:", ["debug"]); addLogEntry(to!string(response), ["debug"]); } // Parse and display error message received from OneDrive if (debugLogging) {addLogEntry(callingFunction ~ "() - Generated a OneDrive CurlException", ["debug"]);} auto errorArray = splitLines(exception.msg); string errorMessage = errorArray[0]; // Configure libcurl to perform a fresh connection setFreshConnectOption(); // What is contained in the curl error message? // Handle the following: // - Couldn't connect to server on handle // - Could not connect to server on handle (changed noticed in curl 8.14.1, possibly done earlier ...) // - Couldn't resolve host name on handle // - Could not resolve host name on handle (changed noticed in curl 8.14.1, possibly done earlier ...) // - Timeout was reached on handle if (canFind(errorMessage, "connect to server on handle") || canFind(errorMessage, "resolve host name on handle") || canFind(errorMessage, "resolve hostname on handle") || canFind(errorMessage, "Timeout was reached on handle")) { // Connectivity to Microsoft OneDrive was lost addLogEntry("Internet connectivity to Microsoft OneDrive service has been interrupted .. re-trying in the background"); // What caused the initial curl exception? // - DNS resolution issue if (canFind(errorMessage, "resolve host name on handle")) { if (debugLogging) {addLogEntry("Unable to resolve server - DNS access blocked?", ["debug"]);} } // - connection issue if (canFind(errorMessage, "connect to server on handle")) { if (debugLogging) {addLogEntry("Unable to connect to server - HTTPS access blocked?", ["debug"]);} } // - timeout issue if (canFind(errorMessage, "Timeout was reached on handle")) { // Common cause is libcurl trying IPv6 DNS resolution when there are only IPv4 DNS servers available if (verboseLogging) { addLogEntry("A libcurl timeout has been triggered - data transfer too slow, no DNS resolution response, no server response or operational timeout", ["verbose"]); // There are 3 common causes for this issue: // 1. Usually poor DNS resolution where libcurl flip/flops to use IPv6 and is unable to resolve // 2. A device between the user and Microsoft OneDrive is unable to correctly handle HTTP/2 communication // 3. No Internet access from this system at this point in time addLogEntry(" - IPv6 DNS resolution issues may be causing timeouts. Consider setting 'ip_protocol_version' to IPv4 to potentially avoid this", ["verbose"]); addLogEntry(" - HTTP/2 compatibility issues might also be interfering with your system. Use 'force_http_11' to switch to HTTP/1.1 to potentially avoid this", ["verbose"]); addLogEntry(" - Ensure 'operation_timeout' is configured for the conditions of your network, covering DNS lookups, connection setup, TLS negotiation, and how long data transfers normally take", ["verbose"]); addLogEntry(" - If these options do not resolve this timeout issue, please use --debug-https to diagnose this issue further.", ["verbose"]); } } } else { // Some other 'libcurl' error was returned if (canFind(errorMessage, "Problem with the SSL CA cert (path? access rights?) on handle")) { // error setting certificate verify locations: // CAfile: /etc/pki/tls/certs/ca-bundle.crt // CApath: none // // Tell the Curl Engine to bypass SSL check - essentially SSL is passing back a bad value due to 'stdio' compile time option // Further reading: // https://github.com/curl/curl/issues/6090 // https://github.com/openssl/openssl/issues/7536 // https://stackoverflow.com/questions/45829588/brew-install-fails-curl77-error-setting-certificate-verify // https://forum.dlang.org/post/vwvkbubufexgeuaxhqfl@forum.dlang.org string sslCertReadErrorMessage = "System SSL CA certificates are missing or unreadable by libcurl – please ensure the correct CA bundle is installed and is accessible."; addLogEntry("ERROR: " ~ sslCertReadErrorMessage); throw new OneDriveError(sslCertReadErrorMessage); } else { // Was this a curl initialization error? if (canFind(errorMessage, "Failed initialization on handle")) { // initialization error ... prevent a run-away process if we have zero disk space ulong localActualFreeSpace = getAvailableDiskSpace("."); if (localActualFreeSpace == 0) { throw new OneDriveError("Zero disk space detected"); } } else { // Unknown curl error displayGeneralErrorMessage(exception, callingFunction, lineno); // Fallback: Ensure retry interval is enforced in case of unknown CurlException if (thisBackOffInterval == 0) { thisBackOffInterval = calculateBackoff(retryAttempts, baseBackoffInterval, maxBackoffInterval); if (thisBackOffInterval <= 0) { thisBackOffInterval = 1; addLogEntry("WARNING: Enforcing minimum backoff interval of 1 second – unclassified CurlException"); } } } } } // A OneDrive API exception was thrown } catch (OneDriveException exception) { // https://developer.overdrive.com/docs/reference-guide // https://learn.microsoft.com/en-us/onedrive/developer/rest-api/concepts/errors?view=odsp-graph-online // https://learn.microsoft.com/en-us/graph/errors /** HTTP/1.1 Response handling Errors in the OneDrive API are returned using standard HTTP status codes, as well as a JSON error response object. The following HTTP status codes should be expected. Status code Status message Description 100 Continue Continue 200 OK Request was handled OK 201 Created This means you've made a successful POST to checkout, lock in a format, or place a hold 204 No Content This means you've made a successful DELETE to remove a hold or return a title 400 Bad Request Cannot process the request because it is malformed or incorrect. 401 Unauthorized Required authentication information is either missing or not valid for the resource. 403 Forbidden Access is denied to the requested resource. The user might not have enough permission. 404 Not Found The requested resource doesn’t exist. 405 Method Not Allowed The HTTP method in the request is not allowed on the resource. 406 Not Acceptable This service doesn’t support the format requested in the Accept header. 408 Request Time out CUSTOM ERROR - Not expected from OneDrive, but can be used to handle Internet connection failures the same (fallback and try again) 409 Conflict The current state conflicts with what the request expects. For example, the specified parent folder might not exist. 410 Gone The requested resource is no longer available at the server. 411 Length Required A Content-Length header is required on the request. 412 Precondition Failed A precondition provided in the request (such as an if-match header) does not match the resource's current state. 413 Request Entity Too Large The request size exceeds the maximum limit. 415 Unsupported Media Type The content type of the request is a format that is not supported by the service. 416 Requested Range Not Satisfiable The specified byte range is invalid or unavailable. 422 Unprocessable Entity Cannot process the request because it is semantically incorrect. 423 Locked The file is currently checked out or locked for editing by another user 429 Too Many Requests Client application has been throttled and should not attempt to repeat the request until an amount of time has elapsed. 500 Internal Server Error There was an internal server error while processing the request. 501 Not Implemented The requested feature isn’t implemented. 502 Bad Gateway The service was unreachable 503 Service Unavailable The service is temporarily unavailable. You may repeat the request after a delay. There may be a Retry-After header. 504 Gateway Timeout The server, which is acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request 506 Variant Also Negotiates CUSTOM ERROR - Received an unexpected response from Microsoft OneDrive 507 Insufficient Storage The maximum storage quota has been reached. 509 Bandwidth Limit Exceeded Your app has been throttled for exceeding the maximum bandwidth cap. Your app can retry the request again after more time has elapsed. HTTP/2 Response handling 0 OK **/ // Detail the OneDriveAPI exception, debug output only if (debugLogging) { addLogEntry("Handling a OneDrive API exception:", ["debug"]); addLogEntry(to!string(response), ["debug"]); // Parse and display error message received from OneDrive addLogEntry(callingFunction ~ "() - Generated a OneDriveException", ["debug"]); } // Perform action based on the HTTP Status Code switch(exception.httpStatusCode) { // 0 - OK ... HTTP/2 version of 200 OK case 0: break; // 100 - Continue case 100: break; // 408 - Request Time Out // 429 - Too Many Requests, backoff case 408,429: // If OneDrive sends a status code 429 then this function will be used to process the Retry-After response header which contains the value by which we need to wait if (exception.httpStatusCode == 408) { addLogEntry("Handling a Microsoft Graph API HTTP 408 Response Code (Request Time Out) - Internal Thread ID: " ~ to!string(curlEngine.internalThreadId)); } else { addLogEntry("Handling a Microsoft Graph API HTTP 429 Response Code (Too Many Requests) - Internal Thread ID: " ~ to!string(curlEngine.internalThreadId)); } // Read in the Retry-After HTTP header as set and delay as per this value before retrying the request thisBackOffInterval = response.getRetryAfterValue(); if (debugLogging) {addLogEntry("Using Retry-After Value = " ~ to!string(thisBackOffInterval), ["debug"]);} transientError = true; break; // Transient errors // 503 - Service Unavailable // 504 - Gateway Timeout case 503,504: // The server, while acting as a proxy, did not receive a timely response from the upstream server it needed to access in attempting to complete the request auto errorArray = splitLines(exception.msg); addLogEntry(to!string(errorArray[0]) ~ " when attempting to query the Microsoft Graph API Service - retrying applicable request in 30 seconds - Internal Thread ID: " ~ to!string(curlEngine.internalThreadId)); if (debugLogging) {addLogEntry("Thread sleeping for 30 seconds as the server did not receive a timely response from the upstream server it needed to access in attempting to complete the request", ["debug"]);} // Transient error - try again in 30 seconds thisBackOffInterval = 30; transientError = true; break; // Default default: // This exception should be then passed back to the original calling function for handling a OneDriveException throw new OneDriveException(response.statusLine.code, response.statusLine.reason, response); } // A FileSystem exception was thrown from somewhere } catch (FileException exception) { // There was a file system error - display the error message displayFileSystemErrorMessage(exception.msg, callingFunction, ""); // as we have no file path reference here, use a blank input throw new OneDriveException(0, "There was a file system error during OneDrive request: " ~ exception.msg, response); // A OneDriveError was thrown } catch (OneDriveError exception) { // Disk space error or SSL error caused a OneDriveError to be thrown /** DO NOT UNCOMMENT THIS CODE UNLESS TESTING FOR THIS ISSUE: System SSL CA certificates are missing or unreadable by libcurl // Disk space error or SSL error if (getAvailableDiskSpace(".") == 0) { // Must exit forceExit(); } else { // Catch the SSL error addLogEntry("WARNING: Disabling SSL peer verification due to libcurl failing to access the system CA certificate bundle (CAfile missing, unreadable, or misconfigured)."); sslVerifyPeerDisabled = true; curlEngine.setDisableSSLVerifyPeer(); } **/ // Must exit forceExit(); } // Increment re-try counter retryAttempts++; // Configure libcurl to perform a fresh connection on API retry setFreshConnectOption(); // Has maxRetryCount been reached? if (retryAttempts > maxRetryCount) { addLogEntry("ERROR: Unable to reconnect to the Microsoft OneDrive service after " ~ to!string(retryAttempts) ~ " attempts lasting approximately 365 days"); throw new OneDriveException(408, "Request Timeout - HTTP 408 or Internet down?", response); } else { // Was 'thisBackOffInterval' set by a 429 event ? if (thisBackOffInterval == 0) { // Calculate and apply exponential backoff upto a maximum of 120 seconds before the API call is re-tried thisBackOffInterval = calculateBackoff(retryAttempts, baseBackoffInterval, maxBackoffInterval); // If this 'somehow' calculates a negative number, this is not correct .. and this has been seen in testing - unknown cause // // Retry attempt: 31 - Internal Thread ID: ICO4ELBlGXFwyTzh // This attempt timestamp: 2024-Aug-10 10:32:07 // Next retry in approx: -2147483648 seconds // Next retry approx: 1956-Jul-23 07:17:59 // Illegal instruction (core dumped) // // Set to 'maxBackoffInterval' if calculated value is negative if (thisBackOffInterval < 0) { thisBackOffInterval = maxBackoffInterval; } } // set the current time for this thread currentTime = Clock.currTime(); currentTime.fracSecs = Duration.zero; // If verbose logging, detail when we are re-trying the call if (verboseLogging) { auto timeString = currentTime.toString(); addLogEntry("Retry attempt: " ~ to!string(retryAttempts) ~ " - Internal Thread ID: " ~ to!string(curlEngine.internalThreadId), ["verbose"]); addLogEntry(" This attempt timestamp: " ~ timeString, ["verbose"]); // Detail when the next attempt will be tried // Factor in the delay for curl to generate the exception - otherwise the next timestamp appears to be 'out' even though technically correct auto nextRetry = currentTime + dur!"seconds"(thisBackOffInterval) + dur!"seconds"(timestampAlign); addLogEntry(" Next retry in approx: " ~ to!string((thisBackOffInterval + timestampAlign)) ~ " seconds"); addLogEntry(" Next retry approx: " ~ to!string(nextRetry), ["verbose"]); } // Thread sleep Thread.sleep(dur!"seconds"(thisBackOffInterval)); } } // Reset SSL Peer Validation if it was disabled if (sslVerifyPeerDisabled) { curlEngine.setEnableSSLVerifyPeer(); } // Return the result return result; } // Check the HTTP Response code and determine if a OneDriveException should be thrown private bool checkHttpResponseCode(int httpResponseCode) { bool shouldThrow = false; // Redirect Codes immutable acceptedRedirectCodes = [301, 302, 304, 307, 308]; // // This condition checks if the HTTP response code falls within the acceptable range for both HTTP 1.1 and HTTP 2.0. // // For HTTP 1.1: // - Any 1xx response (Informational responses, ranging from 100 to 199) // - Any 2xx response (Successful responses, ranging from 200 to 299) // - A 302 response (Temporary Redirect) // // For HTTP 2.0: // - Any 1xx response (Informational responses, ranging from 100 to 199) // - Any 2xx response (Successful responses, ranging from 200 to 299) // - A 302 response (Temporary Redirect) // - A 0 response (Interpreted as 200 OK based on empirical evidence) // // If the HTTP response code meets any of these conditions, it is considered acceptable, and no exception will be thrown. // if ((httpResponseCode >= 100 && httpResponseCode < 300) || canFind(acceptedRedirectCodes, httpResponseCode) || httpResponseCode == 0) { shouldThrow = false; } else { shouldThrow = true; } // return evaluation return shouldThrow; } // Calculates the delay for exponential backoff private int calculateBackoff(int retryAttempts, int baseInterval, int maxInterval) { int cappedAttempts = min(retryAttempts, 10); // Prevent exponent overflow int backoff = baseInterval * (1 << cappedAttempts); return min(backoff, maxInterval); } // Configure libcurl to perform a fresh connection private void setFreshConnectOption() { if (debugLogging) {addLogEntry("Configuring libcurl to use a fresh connection for re-try", ["debug"]);} curlEngine.http.handle.set(CurlOption.fresh_connect,1); // Set libcurl dns_cache_timeout timeout // https://curl.se/libcurl/c/CURLOPT_DNS_CACHE_TIMEOUT.html // https://dlang.org/library/std/net/curl/http.dns_timeout.html curlEngine.http.dnsTimeout = (dur!"seconds"(0)); } // Unset the libcurl fresh connection options and reset libcurl DNS Cache Timeout private void unsetFreshConnectOption() { if (debugLogging) {addLogEntry("Unsetting libcurl to use a fresh connection as this causes a performance impact if left enabled", ["debug"]);} curlEngine.http.handle.set(CurlOption.fresh_connect,0); // Reset libcurl dns_cache_timeout timeout // https://curl.se/libcurl/c/CURLOPT_DNS_CACHE_TIMEOUT.html // https://dlang.org/library/std/net/curl/http.dns_timeout.html curlEngine.http.dnsTimeout = (dur!"seconds"(appConfig.getValueLong("dns_timeout"))); } // Generate a HTTP 'reason' based on the HTTP 'code' private string getMicrosoftGraphStatusMessage(ushort code) { string message; switch (code) { case 200: message = "OK"; break; case 201: message = "Created"; break; case 202: message = "Accepted"; break; case 204: message = "No Content"; break; case 301: message = "Moved Permanently"; break; case 302: message = "Found"; break; case 304: message = "Not Modified"; break; case 307: message = "Temporary Redirect"; break; case 308: message = "Permanent Redirect"; break; case 400: message = "Bad Request"; break; case 401: message = "Unauthorized"; break; case 402: message = "Payment Required"; break; case 403: message = "Forbidden"; break; case 404: message = "Not Found"; break; case 405: message = "Method Not Allowed"; break; case 406: message = "Not Acceptable"; break; case 409: message = "Conflict"; break; case 410: message = "Gone"; break; case 411: message = "Length Required"; break; case 412: message = "Precondition Failed"; break; case 413: message = "Request Entity Too Large"; break; case 415: message = "Unsupported Media Type"; break; case 416: message = "Requested Range Not Satisfiable"; break; case 422: message = "Unprocessable Entity"; break; case 423: message = "Locked"; break; case 429: message = "Too Many Requests"; break; case 500: message = "Internal Server Error"; break; case 501: message = "Not Implemented"; break; case 503: message = "Service Unavailable"; break; case 504: message = "Gateway Timeout"; break; case 507: message = "Insufficient Storage"; break; case 509: message = "Bandwidth Limit Exceeded"; break; default: message = "Unknown Status Code"; break; } return message; } } ================================================ FILE: src/qxor.d ================================================ // What is this module called? module qxor; // What does this module require to function? import std.algorithm; import std.digest; // Implementation of the QuickXorHash algorithm in D // https://github.com/OneDrive/onedrive-api-docs/blob/live/docs/code-snippets/quickxorhash.md struct QuickXor { private enum int widthInBits = 160; private enum size_t lengthInBytes = (widthInBits - 1) / 8 + 1; private enum size_t lengthInQWords = (widthInBits - 1) / 64 + 1; private enum int bitsInLastCell = widthInBits % 64; // 32 private enum int shift = 11; private ulong[lengthInQWords] _data; private ulong _lengthSoFar; private int _shiftSoFar; nothrow @safe void put(scope const(ubyte)[] array...) { int vectorArrayIndex = _shiftSoFar / 64; int vectorOffset = _shiftSoFar % 64; immutable size_t iterations = min(array.length, widthInBits); for (size_t i = 0; i < iterations; i++) { immutable bool isLastCell = vectorArrayIndex == _data.length - 1; immutable int bitsInVectorCell = isLastCell ? bitsInLastCell : 64; if (vectorOffset <= bitsInVectorCell - 8) { for (size_t j = i; j < array.length; j += widthInBits) { _data[vectorArrayIndex] ^= cast(ulong) array[j] << vectorOffset; } } else { int index1 = vectorArrayIndex; int index2 = isLastCell ? 0 : (vectorArrayIndex + 1); ubyte low = cast(ubyte) (bitsInVectorCell - vectorOffset); ubyte xoredByte = 0; for (size_t j = i; j < array.length; j += widthInBits) { xoredByte ^= array[j]; } _data[index1] ^= cast(ulong) xoredByte << vectorOffset; _data[index2] ^= cast(ulong) xoredByte >> low; } vectorOffset += shift; if (vectorOffset >= bitsInVectorCell) { vectorArrayIndex = isLastCell ? 0 : vectorArrayIndex + 1; vectorOffset -= bitsInVectorCell; } } _shiftSoFar = cast(int) (_shiftSoFar + shift * (array.length % widthInBits)) % widthInBits; _lengthSoFar += array.length; } nothrow @safe void start() { _data = _data.init; _shiftSoFar = 0; _lengthSoFar = 0; } nothrow @trusted ubyte[lengthInBytes] finish() { ubyte[lengthInBytes] tmp; tmp[0 .. lengthInBytes] = (cast(ubyte*) _data)[0 .. lengthInBytes]; for (size_t i = 0; i < 8; i++) { tmp[lengthInBytes - 8 + i] ^= (cast(ubyte*) &_lengthSoFar)[i]; } return tmp; } } ================================================ FILE: src/socketio.d ================================================ // What is this module called? module socketio; // What does this module require to function? import core.atomic : atomicLoad, atomicStore; import core.thread : Thread; import core.time : Duration, dur; import std.concurrency : spawn, Tid, thisTid, send, receiveTimeout; import std.conv : to; import std.datetime : SysTime, Clock, UTC; import std.exception : collectException; import std.json : JSONValue, JSONType, parseJSON; import std.net.curl : CurlException; import std.socket : SocketException; import std.string : indexOf; // What other modules that we have created do we need to import? import log; import util; import config; import curlWebsockets; // ========== Logging Shim ========== private void logSocketIOOutput(string s) { if (debugLogging) { addLogEntry("SOCKETIO: " ~ s, ["debug"]); } } final class OneDriveSocketIo { private Tid parentTid; private ApplicationConfig appConfig; private bool started = false; private Duration renewEarly = dur!"seconds"(120); private string engineSid; private bool expiryWarned = false; private bool renewRequested = false; private string currentNotifUrl; // Worker / state private Tid controllerTid; // main/control thread to notify when the worker exits private Tid workerTid; private shared bool pleaseStop = false; private long pingIntervalMs = 25000; private long pingTimeoutMs = 60000; private bool namespaceOpened = false; private CurlWebSocket ws; private shared bool workerExited = false; // set true by run() on clean exit public: this(Tid parentTid, ApplicationConfig appConfig) { this.parentTid = parentTid; this.appConfig = appConfig; } ~this() { logSocketIOOutput("Signalling to stop a OneDriveSocketIo instance"); stop(); // sets pleaseStop + waits for workerExited if (atomicLoad(workerExited)) { if (ws !is null) { logSocketIOOutput("Attempting to destroy libcurl RFC6455 WebSocket client cleanly"); // Worker has exited; safe to close/cleanup/destroy collectException(ws.close(1000, "client stop")); collectException(ws.cleanupCurlHandle()); logSocketIOOutput("Cleaned up an instance of a CurlWebSocket object via cleanupCurlHandle()"); object.destroy(ws); ws = null; logSocketIOOutput("Destroyed libcurl RFC6455 WebSocket client cleanly"); } } else { // Worker still running; DO NOT touch ws/curl from this thread. logSocketIOOutput("Worker still running; skipping ws destruction to avoid race."); } } void start() { if (started) return; // Get current WebSocket Notification URL currentNotifUrl = appConfig.websocketNotificationUrl; // Reset cooperative flags pleaseStop = false; atomicStore(workerExited, false); // Set Flag started = true; // Spawn worker thread workerTid = spawn(&run, cast(shared) this); } void stop() { if (!started) return; // Ask the worker to stop cooperatively pleaseStop = true; logSocketIOOutput("Flagged to stop WebSocket monitoring of Microsoft Graph API changes."); // Wait up to ~6 seconds for the worker to finish cleanup. // No mailbox usage here to avoid nested receiveTimeout on FreeBSD. enum int totalWaitMs = 6000; enum int stepMs = 100; int waited = 0; while (!atomicLoad(workerExited) && waited < totalWaitMs) { Thread.sleep(dur!"msecs"(stepMs)); waited += stepMs; } // Mark not started only after we know we've requested stop started = false; if (!atomicLoad(workerExited)) { // We asked nicely but didn’t get an ack within the window; continue shutdown anyway. // Keeps behaviour safe; avoids hanging the main shutdown path logSocketIOOutput("Worker stop acknowledgement not received within timeout; continuing shutdown."); } } Duration getNextExpirationCheckDuration() { if (appConfig.websocketUrlExpiry.length == 0) return dur!"seconds"(5); SysTime expiry; auto err = collectException(expiry = SysTime.fromISOExtString(appConfig.websocketUrlExpiry)); if (err !is null) return dur!"seconds"(5); auto now = Clock.currTime(UTC()); if (expiry <= now) return dur!"seconds"(5); auto delta = expiry - now; if (delta > renewEarly) delta -= renewEarly; return (delta > Duration.zero) ? delta : dur!"seconds"(5); } private: // Main function that listens and sends events static void run(shared OneDriveSocketIo _this) { logSocketIOOutput("run() entered"); auto self = cast(OneDriveSocketIo) _this; // Capped exponential backoff: 1s, 2s, 4s, ... up to 60s int backoffSeconds = 1; const int maxBackoffSeconds = 60; bool online; scope(exit) { // Signal that the worker is fully done (visible across threads) atomicStore(self.workerExited, true); // Log that we are exiting the run() function logSocketIOOutput("run() exiting"); } while (!self.pleaseStop) { // Catch network exceptions at the socketio-loop level and treat them as recoverable try { // If we're offline (or OneDrive service not reachable), don't bother trying yet logSocketIOOutput("Testing network to ensure network connectivity to Microsoft OneDrive Service"); online = testInternetReachability(self.appConfig, false); // Will display failures, but nothing if successful .. a quiet check of sorts. if (!online) { logSocketIOOutput("Network or OneDrive service not reachable; delaying reconnect"); logSocketIOOutput("Backoff " ~ to!string(backoffSeconds) ~ "s before retry"); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } else { // We are 'online' // Build Socket.IO WS URL from notificationUrl string notif = self.appConfig.websocketNotificationUrl; if (notif.length == 0) { logSocketIOOutput("No notificationUrl available; will retry"); logSocketIOOutput("Backoff " ~ to!string(backoffSeconds) ~ "s before retry"); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } self.currentNotifUrl = notif; string wsUrl = toSocketIoWsUrl(notif); // Fresh WS instance per attempt self.ws = new CurlWebSocket(); // Use application configuration values self.ws.setUserAgent(self.appConfig.getValueString("user_agent")); self.ws.setHTTPSDebug(self.appConfig.getValueBool("debug_https")); self.ws.setTimeouts(10000, 15000); // Connect to Microsoft Graph API using WebSockets and Socket.IO v4 logSocketIOOutput("Connecting to " ~ wsUrl); auto rc = self.ws.connect(wsUrl); if (rc != 0) { logSocketIOOutput("self.ws.connect failed; will retry"); collectException(self.ws.close(1002, "connect-failed")); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } // Socket.IO handshake: wait for '0{json}' if (!awaitEngineOpen(self.ws, self)) { logSocketIOOutput("Socket.IO open handshake failed; will retry"); collectException(self.ws.close(1002, "handshake-failed")); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } // Open default namespace: send "40" logSocketIOOutput("Sending Socket.IO connect (40) to default namespace"); if (self.ws.sendText("40") != 0) { logSocketIOOutput("Failed to send 40 (open namespace); will retry"); collectException(self.ws.close(1002, "ns40-failed")); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } else { logSocketIOOutput("Sent Socket.IO connect '40' for namespace '/'"); } // Open 'notifications' namespace: send "40/notifications" logSocketIOOutput("Sending Socket.IO connect (40) to '/notifications' namespace"); if (self.ws.sendText("40/notifications") != 0) { logSocketIOOutput("Failed to send 40 for '/notifications' namespace; will retry"); collectException(self.ws.close(1002, "ns40-failed")); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; continue; } else { logSocketIOOutput("Sent Socket.IO connect '40' for namespace '/notifications'"); } // Connected successfully → reset backoff backoffSeconds = 1; // Reset per-connection flags so renew logic and ns-open tracking work after reconnection self.expiryWarned = false; self.renewRequested = false; self.namespaceOpened = false; // Track last server ping received to detect a dead connection SysTime lastPingAt = Clock.currTime(UTC()); // Listen for Socket.IO Events for (;;) { // Stop request if (self.pleaseStop) { logSocketIOOutput("Stop requested; shutting down run() loop"); collectException(self.ws.close(1000, "stop-requested")); collectException(self.ws.cleanupCurlHandle()); logSocketIOOutput("Cleaned up an instance of a CurlWebSocket object via cleanupCurlHandle()"); return; } // Subscription nearing expiry? (informational; renewal happens elsewhere) if (!self.expiryWarned && self.appConfig.websocketUrlExpiry.length > 0) { SysTime expiry; auto e = collectException(expiry = SysTime.fromISOExtString(self.appConfig.websocketUrlExpiry)); if (e is null) { auto remain = expiry - Clock.currTime(UTC()); if (remain <= dur!"minutes"(5)) { self.expiryWarned = true; // emit only once logSocketIOOutput("subscription nearing expiry; renewal required soon"); } } } // Renewal window check (emit once; 2 minutes before) if (!self.renewRequested && self.appConfig.websocketUrlExpiry.length > 0) { SysTime expiry; auto e = collectException(expiry = SysTime.fromISOExtString(self.appConfig.websocketUrlExpiry)); if (e is null) { auto remain = expiry - Clock.currTime(UTC()); if (remain <= dur!"minutes"(2)) { self.renewRequested = true; logSocketIOOutput("Subscription nearing expiry; requesting renewal from main() monitor loop"); send(self.parentTid, "SOCKETIO_RENEWAL_REQUEST"); send(self.parentTid, "SOCKETIO_RENEWAL_CONTEXT:" ~ "id=" ~ self.appConfig.websocketEndpointResponse ~ " url=" ~ self.appConfig.websocketNotificationUrl); } } } // If we haven't seen a server ping within pingInterval + pingTimeout → treat as dead link auto now = Clock.currTime(UTC()); auto maxSilence = dur!"msecs"(self.pingIntervalMs + self.pingTimeoutMs); if (now - lastPingAt > maxSilence) { logSocketIOOutput("No server ping within expected window; restarting WebSocket"); break; // fall out to backoff/retry } // Reconnect to a new endpoint if main updated websocketNotificationUrl if (self.appConfig.websocketNotificationUrl.length > 0 && self.appConfig.websocketNotificationUrl != self.currentNotifUrl) { logSocketIOOutput("Detected new notificationUrl; reconnecting"); collectException(self.ws.close(1000, "reconnect")); collectException(self.ws.cleanupCurlHandle()); logSocketIOOutput("Cleaned up an instance of a CurlWebSocket object via cleanupCurlHandle()"); // Establish a fresh connection and handshakes self.currentNotifUrl = self.appConfig.websocketNotificationUrl; string newWsUrl = toSocketIoWsUrl(self.currentNotifUrl); self.ws = new CurlWebSocket(); self.ws.setUserAgent(self.appConfig.getValueString("user_agent")); self.ws.setTimeouts(10000, 15000); self.ws.setHTTPSDebug(self.appConfig.getValueBool("debug_https")); auto rc2 = self.ws.connect(newWsUrl); if (rc2 != 0) { logSocketIOOutput("reconnect failed"); break; // fall out to backoff/retry } if (!awaitEngineOpen(self.ws, self)) { logSocketIOOutput("Socket.IO open after reconnect failed"); break; // fall out to backoff/retry } // Open default namespace again logSocketIOOutput("Sending Socket.IO connect (40) to default namespace"); if (self.ws.sendText("40") != 0) { logSocketIOOutput("Failed to send 40 (open namespace)"); break; // fall out to backoff/retry } else { logSocketIOOutput("Sent Socket.IO connect '40' for namespace '/'"); } // Open '/notifications' again (best-effort) logSocketIOOutput("Sending Socket.IO connect (40) to '/notifications' namespace"); if (self.ws.sendText("40/notifications") != 0) { logSocketIOOutput("Failed to send 40 for '/notifications' namespace"); break; // fall out to backoff/retry } else { logSocketIOOutput("Sent Socket.IO connect '40' for namespace '/notifications'"); } // Reset ping reference after a clean reconnect lastPingAt = Clock.currTime(UTC()); } // Receive message auto msg = self.ws.recvText(); if (msg.length == 0) { Thread.sleep(dur!"msecs"(20)); continue; } // Socket.IO parsing if (msg.length > 0 && msg[0] == '2') { // Server ping -> immediate pong, and mark last ping time if (self.ws.sendText("3") != 0) { logSocketIOOutput("Failed sending Socket.IO pong '3'"); break; // fall out to backoff/retry } else { lastPingAt = Clock.currTime(UTC()); logSocketIOOutput("Socket.IO ping received, → pong sent"); } continue; } if (msg.length > 0 && msg[0] == '3') { continue; } else if (msg.length > 1 && msg[0] == '4' && msg[1] == '2') { logSocketIOOutput("Received 42 msg = " ~ to!string(msg)); handleSocketIoEvent(msg, self); continue; } else if (msg.length > 1 && msg[0] == '4' && msg[1] == '0') { logSocketIOOutput("Received 40 msg = " ~ to!string(msg)); // 40{"sid":...} or 40/notifications,{...} size_t i = 3; while (i < msg.length && msg[i] != ',') i++; auto ns = msg[3 .. i]; if (ns == "notifications") { logSocketIOOutput("Namespace '/notifications' opened; listening for Socket.IO events via WebSocket Transport"); } else { logSocketIOOutput("Namespace '/' opened; listening for Socket.IO events via WebSocket Transport"); } self.namespaceOpened = true; continue; } else if (msg.length > 1 && msg[0] == '4' && msg[1] == '1') { logSocketIOOutput("got 41 (disconnect)"); break; // fall out to backoff/retry } else if (msg.length > 0 && msg[0] == '0') { parseEngineOpenFromPacket(msg, self); continue; } else { logSocketIOOutput("Received Unhandled Message: " ~ msg); } } // Fell out of the inner loop → close and backoff, then retry logSocketIOOutput("Retrying WebSocket Connection"); collectException(self.ws.close(1001, "reconnect")); logSocketIOOutput("Backoff " ~ to!string(backoffSeconds) ~ "s before retry"); Thread.sleep(dur!"seconds"(backoffSeconds)); if (backoffSeconds < maxBackoffSeconds) backoffSeconds *= 2; } } catch (CurlException e) { // Caught a CurlException addLogEntry("Network error during socketio loop: " ~ e.msg ~ " (will retry)"); Thread.sleep(dur!"seconds"(5)); } catch (SocketException e) { // Caught a SocketException addLogEntry("Socket error during socketio loop: " ~ e.msg ~ " (will retry)"); Thread.sleep(dur!"seconds"(5)); } catch (Exception e) { // Caught some other error addLogEntry("Unexpected error during socketio loop: " ~ e.toString()); Thread.sleep(dur!"seconds"(5)); } } } // Convert the notificationURL into a usable WebSocket URL static string toSocketIoWsUrl(string notificationUrl) { // input: https://host/notifications?token=...&applicationId=... // output: wss://host/socket.io/?EIO=4&transport=websocket&token=...&applicationId=... logSocketIOOutput("toSocketIoWsUrl input: " ~ notificationUrl); size_t schemePos = notificationUrl.length; { auto pos = cast(ptrdiff_t) -1; // manual indexOf("://") without std.string for (size_t i = 0; i + 2 < notificationUrl.length; ++i) { if (notificationUrl[i] == ':' && notificationUrl[i+1] == '/' && notificationUrl[i+2] == '/') { pos = cast(ptrdiff_t)i; break; } } if (pos >= 0) schemePos = cast(size_t)pos; } string hostAndAfter; if (schemePos < notificationUrl.length) { hostAndAfter = notificationUrl[(schemePos + 3) .. notificationUrl.length]; } else { hostAndAfter = notificationUrl; } size_t slash = hostAndAfter.length; foreach (i; 0 .. hostAndAfter.length) { if (hostAndAfter[i] == '/') { slash = i; break; } } string host = (slash < hostAndAfter.length) ? hostAndAfter[0 .. slash] : hostAndAfter; string query = ""; if (slash < hostAndAfter.length) { auto rest = hostAndAfter[slash .. hostAndAfter.length]; size_t qpos = rest.length; foreach (i; 0 .. rest.length) { if (rest[i] == '?') { qpos = i; break; } } if (qpos < rest.length) query = rest[(qpos + 1) .. rest.length]; } string outUrl = "wss://" ~ host ~ "/socket.io/?EIO=4&transport=websocket"; if (query.length > 0) outUrl ~= "&" ~ query; logSocketIOOutput("toSocketIoWsUrl output: " ~ outUrl); return outUrl; } // Wait for Socket.IO to open static bool awaitEngineOpen(curlWebsockets.CurlWebSocket ws, OneDriveSocketIo self) { SysTime deadline = Clock.currTime(UTC()) + dur!"seconds"(10); for (;;) { if (Clock.currTime(UTC()) >= deadline) return false; auto msg = ws.recvText(); if (msg.length == 0) { Thread.sleep(dur!"msecs"(25)); continue; } if (msg.length > 0 && msg[0] == '0') { return parseEngineOpenFromPacket(msg, self); } if (msg.length > 1 && msg[0] == '4' && msg[1] == '0') { self.namespaceOpened = true; return true; } logSocketIOOutput("Pre-open RX: " ~ msg); } } // Parse Socket.IO response static bool parseEngineOpenFromPacket(string packet, OneDriveSocketIo self) { // packet = "0{...json...}" if (packet.length < 2) return false; auto jsonPart = packet[1 .. packet.length]; JSONValue j; auto err = collectException(j = parseJSON(jsonPart)); if (err !is null) { logSocketIOOutput("Failed to parse Socket.IO open JSON"); return false; } if (j.type == JSONType.object) { // sid if ("sid" in j.object) { auto vsid = j["sid"]; if (vsid.type == JSONType.string) { self.engineSid = vsid.str; } } // pingInterval if ("pingInterval" in j.object) { auto v = j["pingInterval"]; if (v.type == JSONType.integer) { self.pingIntervalMs = v.integer; } } // pingTimeout if ("pingTimeout" in j.object) { auto v2 = j["pingTimeout"]; if (v2.type == JSONType.integer) { self.pingTimeoutMs = v2.integer; } } } // Log that we have opened a connection and have a valid SID logSocketIOOutput("Engine open; sid=" ~ self.engineSid ~ " pingInterval=" ~ self.pingIntervalMs.to!string ~ "ms" ~ " pingTimeout=" ~ self.pingTimeoutMs.to!string ~ "ms"); return true; } // Handle Socket.IO Events static void handleSocketIoEvent(string msg, OneDriveSocketIo self) { // Accept both: 42[...] // and: 42/,[...] size_t i = 2; string ns = "/"; // Optional namespace: 42/notifications,[...] if (i < msg.length && msg[i] == '/') { size_t j = i + 1; while (j < msg.length && msg[j] != ',') j++; if (j >= msg.length) { logSocketIOOutput("42 frame (malformed namespace): " ~ msg); return; } ns = msg[(i + 1) .. j]; i = j + 1; // payload starts after comma } if (i >= msg.length || msg[i] != '[') { logSocketIOOutput("42 frame (unexpected payload start): ns='/" ~ ns ~ "' raw=" ~ msg); return; } JSONValue arr; auto ex = collectException(arr = parseJSON(msg[i .. $])); if (ex !is null || arr.type != JSONType.array || arr.array.length == 0) { logSocketIOOutput("42 frame (unparsed): ns='/" ~ ns ~ "' raw=" ~ msg); return; } auto evNameVal = arr.array[0]; if (evNameVal.type != JSONType.string) { logSocketIOOutput("42 frame (no string event name): ns='/" ~ ns ~ "' raw=" ~ msg); return; } string evName = evNameVal.str; // 2nd element may be a JSON string containing the real JSON string dataText = "null"; if (arr.array.length > 1) { auto d = arr.array[1]; if (d.type == JSONType.string) { JSONValue inner; auto ex2 = collectException(inner = parseJSON(d.str)); if (ex2 is null) { dataText = inner.toString(); // normalized JSON } else { dataText = d.str; // raw string if not JSON } } else { dataText = d.toString(); } } if (evName == "notification") { logSocketIOOutput("Notification Event (ns='/" ~ ns ~ "') -> " ~ dataText); // Signal main() monitor loop exactly like webhook does collectException(send(self.parentTid, cast(ulong)1)); } else { // Visibility in case the service uses other event names logSocketIOOutput("Event '" ~ evName ~ "' (ns='/" ~ ns ~ "') -> " ~ dataText); } } } ================================================ FILE: src/sqlite.d ================================================ // What is this module called? module sqlite; // What does this module require to function? import std.stdio; import etc.c.sqlite3; import std.string: fromStringz, toStringz; import core.stdc.stdlib; import std.conv; import std.format; import std.file; // What other modules that we have created do we need to import? import log; import util; extern (C) immutable(char)* sqlite3_errstr(int); // missing from the std library // Callback function to check if table exists extern (C) int tableExistsCallback(void* data, int argc, char** argv, char** colNames) { // Set `tableExists` to 1 if at least one row is returned int* tableExists = cast(int*) data; *tableExists = 1; return 0; // Continue processing } static this() { if (sqlite3_libversion_number() < 3006019) { throw new SqliteException(-1, "SQLite 3.6.19 or newer is required"); } } private string ifromStringz(const(char)* cstr) { return fromStringz(cstr).idup; } class SqliteException: Exception { int errorCode; // Add an errorCode member to store the SQLite error code @safe pure nothrow this(int errorCode, string msg, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { super(msg, file, line, next); this.errorCode = errorCode; // Set the errorCode } @safe pure nothrow this(int errorCode, string msg, Throwable next, string file = __FILE__, size_t line = __LINE__) { super(msg, file, line, next); this.errorCode = errorCode; // Set the errorCode } } struct Database { private sqlite3* pDb; this(const(char)[] filename) { open(filename); } ~this() { close(); } int db_checkpoint() { return sqlite3_wal_checkpoint(pDb, null); } // Dump open statements void dump_open_statements() { if (debugLogging) {addLogEntry("Dumping open SQL statements:", ["debug"]);} auto p = sqlite3_next_stmt(pDb, null); while (p != null) { if (debugLogging) {addLogEntry(" Still Open: " ~ to!string(ifromStringz(sqlite3_sql(p))), ["debug"]);} p = sqlite3_next_stmt(pDb, p); } } // Close open statements void close_open_statements() { if (debugLogging) {addLogEntry("Closing open SQL statements:", ["debug"]);} auto p = sqlite3_next_stmt(pDb, null); while (p != null) { // The sqlite3_finalize() function is called to delete a prepared statement sqlite3_finalize(p); addLogEntry(" Finalised: " ~ to!string(ifromStringz(sqlite3_sql(p)))); p = sqlite3_next_stmt(pDb, p); } } // Count open statements int count_open_statements() { if (debugLogging) {addLogEntry("Counting open SQL statements", ["debug"]);} int openStatementCount = 0; auto p = sqlite3_next_stmt(pDb, null); while (p != null) { openStatementCount++; p = sqlite3_next_stmt(pDb, p); } return openStatementCount; } // Check DB Status void checkStatus() { int rc = sqlite3_errcode(pDb); if (rc != SQLITE_OK) { throw new SqliteException(rc, getErrorMessage()); } } // Open the database file void open(const(char)[] filename) { // https://www.sqlite.org/c3ref/open.html // Safest multithreaded way to open the database // Does the file we need to open actually exist? if (exists(filename)) { if (debugLogging) {addLogEntry("Database file EXISTS on disk", ["debug"]);} } else { if (debugLogging) {addLogEntry("Database file DOES NOT EXIST on disk", ["debug"]);} } int rc = sqlite3_open_v2( toStringz(filename), /* Database filename (UTF-8) */ &pDb, /* OUT: SQLite db handle */ SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, /* Flags */ null /* Optional: Name of the VFS module to use */ ); if (rc != SQLITE_OK) { string errorMsg; if (rc == SQLITE_CANTOPEN) { // Database cannot be opened errorMsg = "The database cannot be opened. Please check the permissions of " ~ to!string(filename); } else { // Some other error errorMsg = "A database access error occurred: " ~ getErrorMessage(); } // Log why we could not open the database file addLogEntry(); addLogEntry(errorMsg); addLogEntry(); close(); throw new SqliteException(rc, getErrorMessage()); } // Opened database file OK // Flag to always use extended result codes for errors sqlite3_extended_result_codes(pDb, 1); } void exec(const(char)[] sql) { // https://www.sqlite.org/c3ref/exec.html if (pDb !is null) { int rc = sqlite3_exec(pDb, toStringz(sql), null, null, null); if (rc != SQLITE_OK) { // Get error message and print it, then exit string errorMessage = getErrorMessage(); close(); // Throw sqlite error throw new SqliteException(rc, errorMessage); } } } // Check if the table exists before dropping it void dropTableIfExists(const(char)[] tableName) { string checkTableQuery = "SELECT name FROM sqlite_master WHERE type='table' AND name='" ~ to!string(tableName) ~ "';"; int tableExists = 0; // Execute query with callback to check if table exists int rc = sqlite3_exec(pDb, toStringz(checkTableQuery), &tableExistsCallback, &tableExists, null); // Only proceed if the query executed successfully if (rc == SQLITE_OK) { // If the table exists, drop it if (tableExists == 1) { exec("DROP TABLE " ~ tableName); } else { // Optionally log that the table does not exist addLogEntry(format("WARNING: Table '%s' does not exist, skipping table drop.", to!string(tableName))); } } else { // Log or handle the error if `sqlite3_exec` fails addLogEntry(format("ERROR: Failed to execute table existence check for '%s'.", to!string(tableName))); } } // Get DB Version int getVersion() { int userVersion; extern (C) int callback(void* user_version, int count, char** column_text, char** column_name) { import core.stdc.stdlib: atoi; *(cast(int*) user_version) = atoi(*column_text); return 0; } int rc = sqlite3_exec(pDb, "PRAGMA user_version", &callback, &userVersion, null); if (rc != SQLITE_OK) { throw new SqliteException(rc, getErrorMessage()); } return userVersion; } // Get the threadsafe value int getThreadsafeValue() { return sqlite3_threadsafe(); } // Get sqlite error message string getErrorMessage() { return ifromStringz(sqlite3_errmsg(pDb)); } void setVersion(int userVersion) { exec("PRAGMA user_version=" ~ to!string(userVersion)); } Statement prepare(const(char)[] zSql) { Statement s; // https://www.sqlite.org/c3ref/prepare.html if (pDb !is null) { int rc = sqlite3_prepare_v2(pDb, zSql.ptr, cast(int) zSql.length, &s.pStmt, null); if (rc != SQLITE_OK) { throw new SqliteException(rc, getErrorMessage()); } } return s; } void close() { // https://www.sqlite.org/c3ref/close.html if (pDb !is null) { sqlite3_close_v2(pDb); pDb = null; } } } struct Statement { struct Result { private sqlite3_stmt* pStmt; private const(char)[][] row; private this(sqlite3_stmt* pStmt) { this.pStmt = pStmt; step(); // initialize the range } @property bool empty() { return row.length == 0; } @property auto front() { return row; } alias step popFront; void step() { // https://www.sqlite.org/c3ref/step.html int rc = sqlite3_step(pStmt); if (rc == SQLITE_BUSY) { // Database is locked by another onedrive process addLogEntry("The database is currently locked by another process - cannot sync"); return; } if (rc == SQLITE_DONE) { row.length = 0; } else if (rc == SQLITE_ROW) { // https://www.sqlite.org/c3ref/data_count.html int count = 0; count = sqlite3_data_count(pStmt); row = new const(char)[][count]; foreach (size_t i, ref column; row) { // https://www.sqlite.org/c3ref/column_blob.html column = fromStringz(sqlite3_column_text(pStmt, to!int(i))); } } else { string errorMessage = getErrorMessage(); // Must force exit here, allow logging to be done throw new SqliteException(rc, errorMessage); } } string getErrorMessage() { return ifromStringz(sqlite3_errmsg(sqlite3_db_handle(pStmt))); } } private sqlite3_stmt* pStmt; ~this() { // Finalise any prepared statement finalise(); } // https://www.sqlite.org/c3ref/finalize.html void finalise() { if (pStmt !is null) { // The sqlite3_finalize() function is called to delete a prepared statement sqlite3_finalize(pStmt); pStmt = null; } } void bind(int index, const(char)[] value) { reset(); // https://www.sqlite.org/c3ref/bind_blob.html int rc = sqlite3_bind_text(pStmt, index, value.ptr, cast(int) value.length, SQLITE_STATIC); if (rc != SQLITE_OK) { throw new SqliteException(rc, getErrorMessage()); } } Result exec() { reset(); return Result(pStmt); } private void reset() { // https://www.sqlite.org/c3ref/reset.html int rc = sqlite3_reset(pStmt); if (rc != SQLITE_OK) { throw new SqliteException(rc, getErrorMessage()); } } string getErrorMessage() { return ifromStringz(sqlite3_errmsg(sqlite3_db_handle(pStmt))); } } ================================================ FILE: src/sync.d ================================================ // What is this module called? module syncEngine; // What does this module require to function? import core.memory; import core.stdc.stdlib: EXIT_SUCCESS, EXIT_FAILURE, exit; import core.thread; import core.time; import std.algorithm; import std.array; import std.concurrency; import std.container.rbtree; import std.conv; import std.datetime; import std.encoding; import std.exception; import std.file; import std.json; import std.parallelism; import std.path; import std.range; import std.regex; import std.stdio; import std.string; import std.uni; import std.uri; import std.utf; import std.math; import std.typecons; // What other modules that we have created do we need to import? import config; import log; import util; import onedrive; import itemdb; import clientSideFiltering; import xattr; class JsonResponseException: Exception { @safe pure this(string inputMessage) { string msg = format(inputMessage); super(msg); } } class PosixException: Exception { @safe pure this(string localTargetName, string remoteTargetName) { string msg = format("POSIX 'case-insensitive match' between '%s' (local) and '%s' (online) which violates the Microsoft OneDrive API namespace convention", localTargetName, remoteTargetName); super(msg); } } class AccountDetailsException: Exception { @safe pure this() { string msg = format("Unable to query OneDrive API to obtain required account details"); super(msg); } } class SyncException: Exception { @nogc @safe pure nothrow this(string msg, string file = __FILE__, size_t line = __LINE__) { super(msg, file, line); } } struct DriveDetailsCache { // - driveId is the drive for the operations were items need to be stored // - quotaRestricted details a bool value as to if that drive is restricting our ability to understand if there is space available. Some 'Business' and 'SharePoint' restrict, and most (if not all) shared folders it cant be determined if there is free space // - quotaAvailable is a long value that stores the value of what the current free space is available online string driveId; bool quotaRestricted; bool quotaAvailable; long quotaRemaining; } struct DeltaLinkDetails { string driveId; string itemId; string latestDeltaLink; } struct DatabaseItemsToDeleteOnline { Item dbItem; string localFilePath; } class SyncEngine { // Class Variables ApplicationConfig appConfig; ItemDatabase itemDB; ClientSideFiltering selectiveSync; // Array of directory databaseItem.id to skip while applying the changes. // These are the 'parent path' id's that are being excluded, so if the parent id is in here, the child needs to be skipped as well RedBlackTree!string skippedItems = redBlackTree!string(); // Array consisting of 'item.driveId', 'item.id' and 'item.parentId' values to delete after all the online changes have been downloaded string[3][] idsToDelete; // Array of JSON items which are files or directories that are not 'root', skipped or to be deleted, that need to be processed JSONValue[] jsonItemsToProcess; // Array of JSON items which are files that are not 'root', skipped or to be deleted, that need to be downloaded JSONValue[] fileJSONItemsToDownload; // Array of paths that failed to download string[] fileDownloadFailures; // Associative array mapping of all OneDrive driveId's that have been seen, mapped with DriveDetailsCache data for reference DriveDetailsCache[string] onlineDriveDetails; // List of items we fake created when using --dry-run string[2][] idsFaked; // List of paths we fake deleted when using --dry-run string[] pathFakeDeletedArray; // Array of database Parent Item ID, Item ID & Local Path where the content has changed and needs to be uploaded string[3][] databaseItemsWhereContentHasChanged; // Array of local file paths that need to be uploaded as new items to OneDrive string[] newLocalFilesToUploadToOneDrive; // Array of local file paths that failed to be uploaded to OneDrive string[] fileUploadFailures; // List of path names changed online, but not changed locally when using --dry-run string[] pathsRenamed; // List of path names retained when using --download-only --cleanup-local-files + using a 'sync_list' string[] pathsRetained; // List of paths that were a POSIX case-insensitive match, thus could not be created online string[] posixViolationPaths; // List of local paths, that, when using the OneDrive Business Shared Folders feature, then disabling it, folder still exists locally and online // This list of local paths need to be skipped string[] businessSharedFoldersOnlineToSkip; // List of interrupted uploads session files that need to be resumed string[] interruptedUploadsSessionFiles; // List of interrupted downloads that need to be resumed string[] interruptedDownloadFiles; // List of validated interrupted uploads session JSON items to resume JSONValue[] jsonItemsToResumeUpload; // List of validated interrupted download JSON items to resume JSONValue[] jsonItemsToResumeDownload; // This list of local paths that need to be created online string[] pathsToCreateOnline; // Array of items from the database that have been deleted locally, that needs to be deleted online DatabaseItemsToDeleteOnline[] databaseItemsToDeleteOnline; // Array of parentId's that have been skipped via 'sync_list' string[] syncListSkippedParentIds; // Array of Microsoft OneNote Notebook Package ID's string[] onenotePackageIdentifiers; // Flag that there were upload or download failures listed bool syncFailures = false; // Is sync_list configured bool syncListConfigured = false; // Was --dry-run used? bool dryRun = false; // Was --upload-only used? bool uploadOnly = false; // Was --remove-source-files used? // Flag to set whether the local file should be deleted once it is successfully uploaded to OneDrive bool localDeleteAfterUpload = false; // Do we configure to disable the download validation routine due to --disable-download-validation // We will always validate our downloads // However, when downloading files from SharePoint, the OneDrive API will not advise the correct file size // which means that the application thinks the file download has failed as the size is different / hash is different // See: https://github.com/abraunegg/onedrive/discussions/1667 bool disableDownloadValidation = false; // Do we configure to disable the upload validation routine due to --disable-upload-validation // We will always validate our uploads // However, when uploading a file that can contain metadata SharePoint will associate some // metadata from the library the file is uploaded to directly in the file which breaks this validation. // See: https://github.com/abraunegg/onedrive/issues/205 // See: https://github.com/OneDrive/onedrive-api-docs/issues/935 bool disableUploadValidation = false; // Do we perform a local cleanup of files that are 'extra' on the local file system, when using --download-only bool cleanupLocalFiles = false; // Are we performing a --single-directory sync ? bool singleDirectoryScope = false; string singleDirectoryScopeDriveId; string singleDirectoryScopeItemId; // Is National Cloud Deployments configured ? bool nationalCloudDeployment = false; // Do we configure not to perform a remote file delete if --upload-only & --no-remote-delete configured bool noRemoteDelete = false; // Is bypass_data_preservation set via config file // Local data loss MAY occur in this scenario bool bypassDataPreservation = false; // Has the user configured to permanently delete files online rather than send to online recycle bin bool permanentDelete = false; // Maximum file size upload // https://support.microsoft.com/en-us/office/invalid-file-names-and-file-types-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-us&rs=en-us&ad=us // July 2020, maximum file size for all accounts is 100GB // January 2021, maximum file size for all accounts is 250GB long maxUploadFileSize = 268435456000; // 250GB // Threshold after which files will be uploaded using an upload session long sessionThresholdFileSize = 4 * 2^^20; // 4 MiB // File size limit for file operations that the user has configured long fileSizeLimit; // Total data to upload long totalDataToUpload; // How many items have been processed for the active operation long processedCount; // Are we creating a simulated /delta response? This is critically important in terms of how we 'update' the database bool generateSimulatedDeltaResponse = false; // Store the latest DeltaLink string latestDeltaLink; // Struct of containing the deltaLink details DeltaLinkDetails deltaLinkCache; // Array of driveId and deltaLink for use when performing the last examination of the most recent online data alias DeltaLinkInfo = string[string]; DeltaLinkInfo deltaLinkInfo; // Flag to denote data cleanup pass when using --download-only --cleanup-local-files bool cleanupDataPass = false; // Create the specific task pool to process items in parallel TaskPool processPool; // Shared Folder Flags for 'sync_list' processing bool sharedFolderDeltaGeneration = false; string currentSharedFolderName = ""; // Directory excluded by 'sync_list flag so that when scanning that directory, if it is excluded, // can be scanned for new data which may be included by other include rule, but parent is excluded bool syncListDirExcluded = false; // Debug Logging Break Lines string debugLogBreakType1 = "-----------------------------------------------------------------------------------------------------------"; string debugLogBreakType2 = "==========================================================================================================="; // Configure this class instance this(ApplicationConfig appConfig, ItemDatabase itemDB, ClientSideFiltering selectiveSync) { // Create the specific task pool to process items in parallel processPool = new TaskPool(to!int(appConfig.getValueLong("threads"))); if (debugLogging) {addLogEntry("Initialised TaskPool worker with threads: " ~ to!string(processPool.size), ["debug"]);} // Configure the class variable to consume the application configuration this.appConfig = appConfig; // Configure the class variable to consume the database configuration this.itemDB = itemDB; // Configure the class variable to consume the selective sync (skip_dir, skip_file and sync_list) configuration this.selectiveSync = selectiveSync; // Configure the dryRun flag to capture if --dry-run was used // Application startup already flagged we are also in a --dry-run state, so no need to output anything else here this.dryRun = appConfig.getValueBool("dry_run"); // Configure file size limit if (appConfig.getValueLong("skip_size") != 0) { fileSizeLimit = appConfig.getValueLong("skip_size") * 2^^20; fileSizeLimit = (fileSizeLimit == 0) ? long.max : fileSizeLimit; } // Is there a sync_list file present? if (exists(appConfig.syncListFilePath)) { // yes there is a file present, but did we load any entries? if (!selectiveSync.validSyncListRules) { // function returned 'false' (array contains valid entries) // flag there are rules to process when we are performing Client Side Filtering if (debugLogging) {addLogEntry("Configuring syncListConfigured flag to TRUE as valid entries were loaded from 'sync_list' file", ["debug"]);} this.syncListConfigured = true; } else { // function returned 'true' meaning there are are zero sync_list rules loaded despite the 'sync_list' file being present // ensure this flag is false so we do not do any extra processing if (debugLogging) {addLogEntry("Configuring syncListConfigured flag to FALSE as no valid entries were loaded from 'sync_list' file", ["debug"]);} this.syncListConfigured = false; } } // Configure the uploadOnly flag to capture if --upload-only was used if (appConfig.getValueBool("upload_only")) { if (debugLogging) {addLogEntry("Configuring uploadOnly flag to TRUE as --upload-only passed in or configured", ["debug"]);} this.uploadOnly = true; } // Configure the localDeleteAfterUpload flag if (appConfig.getValueBool("remove_source_files")) { if (debugLogging) {addLogEntry("Configuring localDeleteAfterUpload flag to TRUE as --remove-source-files passed in or configured", ["debug"]);} this.localDeleteAfterUpload = true; } // Configure the disableDownloadValidation flag if (appConfig.getValueBool("disable_download_validation")) { if (debugLogging) {addLogEntry("Configuring disableDownloadValidation flag to TRUE as --disable-download-validation passed in or configured", ["debug"]);} this.disableDownloadValidation = true; } // Configure the disableUploadValidation flag if (appConfig.getValueBool("disable_upload_validation")) { if (debugLogging) {addLogEntry("Configuring disableUploadValidation flag to TRUE as --disable-upload-validation passed in or configured", ["debug"]);} this.disableUploadValidation = true; } // Do we configure to clean up local files if using --download-only ? if ((appConfig.getValueBool("download_only")) && (appConfig.getValueBool("cleanup_local_files"))) { // --download-only and --cleanup-local-files were passed in addLogEntry(); addLogEntry("WARNING: Application has been configured to cleanup local files that are not present online."); addLogEntry("WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally."); addLogEntry(); // Set the flag this.cleanupLocalFiles = true; } // Do we configure to NOT perform a remote delete if --upload-only & --no-remote-delete configured ? if ((appConfig.getValueBool("upload_only")) && (appConfig.getValueBool("no_remote_delete"))) { // --upload-only and --no-remote-delete were passed in addLogEntry("WARNING: Application has been configured NOT to cleanup remote files that are deleted locally."); // Set the flag this.noRemoteDelete = true; } // Are we configured to use a National Cloud Deployment? if (appConfig.getValueString("azure_ad_endpoint") != "") { // value is configured, is it a valid value? if ((appConfig.getValueString("azure_ad_endpoint") == "USL4") || (appConfig.getValueString("azure_ad_endpoint") == "USL5") || (appConfig.getValueString("azure_ad_endpoint") == "DE") || (appConfig.getValueString("azure_ad_endpoint") == "CN")) { // valid entries to flag we are using a National Cloud Deployment // National Cloud Deployments do not support /delta as a query // https://docs.microsoft.com/en-us/graph/deployments#supported-features // Flag that we have a valid National Cloud Deployment that cannot use /delta queries this.nationalCloudDeployment = true; // Reverse set 'force_children_scan' for completeness appConfig.setValueBool("force_children_scan", true); } } // Are we forcing to use /children scan instead of /delta to simulate National Cloud Deployment use of /children? if (appConfig.getValueBool("force_children_scan")) { addLogEntry("Forcing client to use /children API call rather than /delta API to retrieve objects from the OneDrive API"); this.nationalCloudDeployment = true; } // Are we forcing the client to bypass any data preservation techniques to NOT rename any local files if there is a conflict? // The enabling of this function could lead to data loss if (appConfig.getValueBool("bypass_data_preservation")) { addLogEntry(); addLogEntry("WARNING: Application has been configured to bypass local data preservation in the event of file conflict."); addLogEntry("WARNING: Local data loss MAY occur in this scenario."); addLogEntry(); this.bypassDataPreservation = true; } // Did the user configure a specific rate limit for the application? if (appConfig.getValueLong("rate_limit") > 0) { // User configured rate limit addLogEntry("User Configured Rate Limit: " ~ to!string(appConfig.getValueLong("rate_limit"))); // If user provided rate limit is < 131072, flag that this is too low, setting to the recommended minimum of 131072 if (appConfig.getValueLong("rate_limit") < 131072) { // user provided limit too low addLogEntry("WARNING: User configured rate limit too low for normal application processing and preventing application timeouts. Overriding to recommended minimum of 131072 (128KB/s)"); appConfig.setValueLong("rate_limit", 131072); } } // Did the user downgrade all HTTP operations to force HTTP 1.1 if (appConfig.getValueBool("force_http_11")) { // User is forcing downgrade to curl to use HTTP 1.1 for all operations if (verboseLogging) {addLogEntry("Downgrading all HTTP operations to HTTP/1.1 due to user configuration", ["verbose"]);} } else { // Use curl defaults if (debugLogging) {addLogEntry("Using Curl defaults for HTTP operational protocol version (potentially HTTP/2)", ["debug"]);} } } // Initialise the Sync Engine class bool initialise() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Control whether the worker threads are daemon threads. A daemon thread is automatically terminated when all non-daemon threads have terminated. processPool.isDaemon(true); // daemon thread // Flag for 'no-sync' task bool noSyncTask = false; // Create a new instance of the OneDrive API OneDriveApi oneDriveApiInstance; oneDriveApiInstance = new OneDriveApi(appConfig); // Exit scope - release curl engine back to pool scope(exit) { oneDriveApiInstance.releaseCurlEngine(); // Free object and memory oneDriveApiInstance = null; } // Issue #2941 // If the account being used _only_ has access to specific resources, getDefaultDriveDetails() will generate problems and cause // the application to exit, which, is technically the right thing to do (no access to account details) ... but if: // - are we doing a no-sync task ? // - do we have the 'drive_id' via config file ? // Are we not doing a --sync or a --monitor operation? Both of these will be false if they are not set if ((!appConfig.getValueBool("synchronize")) && (!appConfig.getValueBool("monitor"))) { // set flag noSyncTask = true; } // Can the API be initialised successfully? if (oneDriveApiInstance.initialise()) { // Get the relevant default drive details try { getDefaultDriveDetails(); } catch (AccountDetailsException exception) { // was this a no-sync task? if (!noSyncTask) { // details could not be queried addLogEntry(exception.msg); // Must force exit here, allow logging to be done forceExit(); } } // Get the relevant default root details try { getDefaultRootDetails(); } catch (AccountDetailsException exception) { // details could not be queried addLogEntry(exception.msg); // Must force exit here, allow logging to be done forceExit(); } // Display relevant account details try { // we only do this if we are doing --verbose logging if (verboseLogging) { displaySyncEngineDetails(); } } catch (AccountDetailsException exception) { // details could not be queried addLogEntry(exception.msg); // Must force exit here, allow logging to be done forceExit(); } } else { // API could not be initialised addLogEntry("OneDrive API could not be initialised with previously used details"); // Must force exit here, allow logging to be done forceExit(); } // Has the client been configured to permanently delete files online rather than send these to the online recycle bin? if (appConfig.getValueBool("permanent_delete")) { // This can only be set if not using: // - US Government L4 // - US Government L5 (DOD) // - Azure and Office365 operated by VNET in China // // Additionally, this is not supported by OneDrive Personal accounts: // // This is a doc bug. In fact, OneDrive personal accounts do not support the permanentDelete API, it only applies to OneDrive for Business and SharePoint document libraries. // // Reference: https://learn.microsoft.com/en-us/answers/questions/1501170/onedrive-permanently-delete-a-file string azureConfigValue = appConfig.getValueString("azure_ad_endpoint"); // Now that we know the 'accountType' we can configure this correctly if ((appConfig.accountType != "personal") && (azureConfigValue.empty || azureConfigValue == "DE")) { // Only supported for Global Service and DE based on https://learn.microsoft.com/en-us/graph/api/driveitem-permanentdelete?view=graph-rest-1.0 addLogEntry(); addLogEntry("WARNING: Application has been configured to permanently remove files online rather than send to the recycle bin. Permanently deleted items can't be restored."); addLogEntry("WARNING: Online data loss MAY occur in this scenario."); addLogEntry(); this.permanentDelete = true; } else { // what error message do we present if (appConfig.accountType == "personal") { // personal account type - API not supported addLogEntry(); addLogEntry("WARNING: The application is configured to permanently delete files online; however, this action is not supported by Microsoft OneDrive Personal Accounts."); addLogEntry(); } else { // Not a personal account addLogEntry(); addLogEntry("WARNING: The application is configured to permanently delete files online; however, this action is not supported by the National Cloud Deployment in use."); addLogEntry(); } // ensure this is false regardless this.permanentDelete = false; } } // API was initialised if (verboseLogging) {addLogEntry("Sync Engine Initialised with new Onedrive API instance", ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return required value return true; } // Shutdown the sync engine, wait for anything in processPool to complete void shutdown() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } if (debugLogging) {addLogEntry("SyncEngine: Waiting for all internal threads to complete", ["debug"]);} shutdownProcessPool(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Shut down all running tasks that are potentially running in parallel void shutdownProcessPool() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // TaskPool needs specific shutdown based on compiler version otherwise this causes a segfault if (processPool.size > 0) { // TaskPool is still configured for 'thread' size // Normal TaskPool shutdown process if (debugLogging) {addLogEntry("Shutting down processPool in a thread blocking manner", ["debug"]);} // All worker threads are daemon threads which are automatically terminated when all non-daemon threads have terminated. processPool.finish(true); // If blocking argument is true, wait for all worker threads to terminate before returning. } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Get Default Drive Details for this Account void getDefaultDriveDetails() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables JSONValue defaultOneDriveDriveDetails; bool noSyncTask = false; // Create a new instance of the OneDrive API OneDriveApi getDefaultDriveApiInstance; getDefaultDriveApiInstance = new OneDriveApi(appConfig); getDefaultDriveApiInstance.initialise(); // Are we not doing a --sync or a --monitor operation? Both of these will be false if they are not set if ((!appConfig.getValueBool("synchronize")) && (!appConfig.getValueBool("monitor"))) { // set flag noSyncTask = true; } // Get Default Drive Details for this Account try { if (debugLogging) {addLogEntry("Getting Account Default Drive Details", ["debug"]);} defaultOneDriveDriveDetails = getDefaultDriveApiInstance.getDefaultDriveDetails(); } catch (OneDriveException exception) { if (debugLogging) {addLogEntry("defaultOneDriveDriveDetails = getDefaultDriveApiInstance.getDefaultDriveDetails() generated a OneDriveException", ["debug"]);} if ((exception.httpStatusCode == 400) || (exception.httpStatusCode == 401)) { // Handle the 400 | 401 error handleClientUnauthorised(exception.httpStatusCode, exception.error); } else { // Default operation if not 400,401 errors // - 408,429,503,504 errors are handled as a retry within getDefaultDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } // If the JSON response is a correct JSON object, and has an 'id' we can set these details if ((defaultOneDriveDriveDetails.type() == JSONType.object) && (hasId(defaultOneDriveDriveDetails))) { if (debugLogging) {addLogEntry("OneDrive Account Default Drive Details: " ~ sanitiseJSONItem(defaultOneDriveDriveDetails), ["debug"]);} appConfig.accountType = defaultOneDriveDriveDetails["driveType"].str; // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Test driveId length and validation // Once checked and validated, we only need to check 'driveId' if it does not match exactly 'appConfig.defaultDriveId' appConfig.defaultDriveId = transformToLowerCase(testProvidedDriveIdForLengthIssue(defaultOneDriveDriveDetails["id"].str)); } else { // Use 'defaultOneDriveDriveDetails' as is for all other account types appConfig.defaultDriveId = defaultOneDriveDriveDetails["id"].str; } // Make sure that appConfig.defaultDriveId is in our driveIDs array to use when checking if item is in database // Keep the DriveDetailsCache array with unique entries only DriveDetailsCache cachedOnlineDriveData; if (!canFindDriveId(appConfig.defaultDriveId, cachedOnlineDriveData)) { // Add this driveId to the drive cache, which then also sets for the defaultDriveId: // - quotaRestricted; // - quotaAvailable; // - quotaRemaining; // // In some cases OneDrive Business configurations 'restrict' quota details thus is empty / blank / negative value / zero value // When addOrUpdateOneDriveOnlineDetails() is called, messaging is provided if these are zero, negative or missing (thus quota is being restricted) addOrUpdateOneDriveOnlineDetails(appConfig.defaultDriveId); } // Fetch the details from cachedOnlineDriveData for appConfig.defaultDriveId cachedOnlineDriveData = getDriveDetails(appConfig.defaultDriveId); // - cachedOnlineDriveData.quotaRestricted; // - cachedOnlineDriveData.quotaAvailable; // - cachedOnlineDriveData.quotaRemaining; // What did we set based on the data from the JSON and cached drive data if (debugLogging) { addLogEntry("appConfig.accountType = " ~ appConfig.accountType, ["debug"]); addLogEntry("appConfig.defaultDriveId = " ~ appConfig.defaultDriveId, ["debug"]); addLogEntry("cachedOnlineDriveData.quotaRemaining = " ~ to!string(cachedOnlineDriveData.quotaRemaining), ["debug"]); addLogEntry("cachedOnlineDriveData.quotaAvailable = " ~ to!string(cachedOnlineDriveData.quotaAvailable), ["debug"]); addLogEntry("cachedOnlineDriveData.quotaRestricted = " ~ to!string(cachedOnlineDriveData.quotaRestricted), ["debug"]); } // Regardless of this being all set - based on the JSON response, check for 'quota' being present, to check // for the following valid states: normal | nearing | critical | exceeded // // Based on this, then generate an applicable application message to advise the user of their quota status if ((hasQuota(defaultOneDriveDriveDetails)) && (hasQuotaState(defaultOneDriveDriveDetails))) { // get the current state string quotaState = defaultOneDriveDriveDetails["quota"]["state"].str; // quotaState = normal - no message string nearingMessage = "WARNING: Your Microsoft OneDrive storage is nearing capacity, with less than 10% of your available space remaining."; string criticalMessage = "WARNING: Your Microsoft OneDrive storage is critically low, with less than 1% of your available space remaining."; string exceededMessage = "CRITICAL: Your Microsoft OneDrive storage limit has been exceeded. You can no longer upload new content to Microsoft OneDrive."; string actionRequired = " Delete unneeded files or upgrade your storage plan now, as further uploads will not be possible once storage is exceeded"; // switch to display the right message switch(quotaState) { case "nearing": addLogEntry(); addLogEntry(nearingMessage, ["info", "notify"]); addLogEntry(actionRequired); addLogEntry(); break; case "critical": addLogEntry(); addLogEntry(criticalMessage, ["info", "notify"]); addLogEntry(actionRequired); addLogEntry(); break; case "exceeded": addLogEntry(); addLogEntry("******************************************************************************************************************************"); addLogEntry(exceededMessage, ["info", "notify"]); addLogEntry("******************************************************************************************************************************"); addLogEntry(); break; default: // nothing } } } else { // Did the configuration file contain a 'drive_id' entry // If this exists, this will be a 'documentLibrary' if (appConfig.getValueString("drive_id").length) { // Force set these as for whatever reason we could to query these via the getDefaultDriveDetails API call appConfig.accountType = "documentLibrary"; appConfig.defaultDriveId = appConfig.getValueString("drive_id"); } else { // was this a no-sync task? if (!noSyncTask) { // Handle the invalid JSON response by throwing an exception error throw new AccountDetailsException(); } } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getDefaultDriveApiInstance.releaseCurlEngine(); getDefaultDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Get Default Root Details for this Account void getDefaultRootDetails() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables JSONValue defaultOneDriveRootDetails; bool noSyncTask = false; // Create a new instance of the OneDrive API OneDriveApi getDefaultRootApiInstance; getDefaultRootApiInstance = new OneDriveApi(appConfig); getDefaultRootApiInstance.initialise(); // Are we not doing a --sync or a --monitor operation? Both of these will be false if they are not set if ((!appConfig.getValueBool("synchronize")) && (!appConfig.getValueBool("monitor"))) { // set flag noSyncTask = true; } // Get Default Root Details for this Account try { if (debugLogging) {addLogEntry("Getting Account Default Root Details", ["debug"]);} defaultOneDriveRootDetails = getDefaultRootApiInstance.getDefaultRootDetails(); } catch (OneDriveException exception) { if (debugLogging) {addLogEntry("defaultOneDriveRootDetails = getDefaultRootApiInstance.getDefaultRootDetails() generated a OneDriveException", ["debug"]);} if ((exception.httpStatusCode == 400) || (exception.httpStatusCode == 401)) { // Handle the 400 | 401 error handleClientUnauthorised(exception.httpStatusCode, exception.error); } else { // Default operation if not 400,401 errors // - 408,429,503,504 errors are handled as a retry within getDefaultRootApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } // If the JSON response is a correct JSON object, and has an 'id' we can set these details if ((defaultOneDriveRootDetails.type() == JSONType.object) && (hasId(defaultOneDriveRootDetails))) { // Read the returned JSON data for the root drive details if (debugLogging) {addLogEntry("OneDrive Account Default Root Details: " ~ sanitiseJSONItem(defaultOneDriveRootDetails), ["debug"]);} appConfig.defaultRootId = defaultOneDriveRootDetails["id"].str; if (debugLogging) {addLogEntry("appConfig.defaultRootId = " ~ appConfig.defaultRootId, ["debug"]);} // Save the item to the database, so the account root drive is is always going to be present in the DB saveItem(defaultOneDriveRootDetails); } else { // was this a no-sync task? if (!noSyncTask) { // Handle the invalid JSON response by throwing an exception error throw new AccountDetailsException(); } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getDefaultRootApiInstance.releaseCurlEngine(); getDefaultRootApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Reset syncFailures to false based on file activity void resetSyncFailures() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Log initial status and any non-empty arrays string logMessage = "Evaluating reset of syncFailures: "; if (fileDownloadFailures.length > 0) { logMessage ~= "fileDownloadFailures is not empty; "; } if (fileUploadFailures.length > 0) { logMessage ~= "fileUploadFailures is not empty; "; } // Check if both arrays are empty to reset syncFailures if (fileDownloadFailures.length == 0 && fileUploadFailures.length == 0) { if (syncFailures) { syncFailures = false; logMessage ~= "Resetting syncFailures to false."; } else { logMessage ~= "syncFailures already false."; } } else { // Indicate no reset of syncFailures due to non-empty conditions logMessage ~= "Not resetting syncFailures due to non-empty arrays."; } // Log the final decision and conditions if (debugLogging) {addLogEntry(logMessage, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform a sync of the OneDrive Account // - Query /delta // - If singleDirectoryScope or nationalCloudDeployment is used we need to generate a /delta like response // - Process changes (add, changes, moves, deletes) // - Process any items to add (download data to local) // - Detail any files that we failed to download // - Process any deletes (remove local data) void syncOneDriveAccountToLocalDisk() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // performFullScanTrueUp value if (debugLogging) {addLogEntry("Perform a Full Scan True-Up: " ~ to!string(appConfig.fullScanTrueUpRequired), ["debug"]);} // Fetch the API response of /delta to track changes that were performed online fetchOneDriveDeltaAPIResponse(); // Process any download activities or cleanup actions processDownloadActivities(); // If singleDirectoryScope is false, we are not targeting a single directory // but if true, the target 'could' be a shared folder - so dont try and scan it again if (!singleDirectoryScope) { // OneDrive Shared Folder Handling if (appConfig.accountType == "personal") { // Personal Account Type // https://github.com/OneDrive/onedrive-api-docs/issues/764 // Get the Remote Items from the Database Item[] remoteItems = itemDB.selectRemoteItems(); foreach (remoteItem; remoteItems) { // Check if this path is specifically excluded by 'skip_dir', but only if 'skip_dir' is not empty if (appConfig.getValueString("skip_dir") != "") { // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched if (selectiveSync.isDirNameExcluded(remoteItem.name)) { // This directory name is excluded if (verboseLogging) {addLogEntry("Skipping path - excluded by skip_dir config: " ~ remoteItem.name, ["verbose"]);} continue; } } // Directory name is not excluded or skip_dir is not populated if (!appConfig.suppressLoggingOutput) { // So that we represent correctly where this shared folder is, calculate the path string sharedFolderLogicalPath = computeItemPath(remoteItem.driveId, remoteItem.id); addLogEntry("Syncing this OneDrive Personal Shared Folder: " ~ ensureStartsWithDotSlash(sharedFolderLogicalPath)); } // Check this OneDrive Personal Shared Folder for changes fetchOneDriveDeltaAPIResponse(remoteItem.remoteDriveId, remoteItem.remoteId, remoteItem.name); // Process any download activities or cleanup actions for this OneDrive Personal Shared Folder processDownloadActivities(); } // Clear the array remoteItems = []; } else { // Is this a Business Account with Sync Business Shared Items enabled? if ((appConfig.accountType == "business") && (appConfig.getValueBool("sync_business_shared_items"))) { // Business Account Shared Items Handling // - OneDrive Business Shared Folder // - OneDrive Business Shared Files // - SharePoint Links // Get the Remote Items from the Database Item[] remoteItems = itemDB.selectRemoteItems(); foreach (remoteItem; remoteItems) { // As all remote items are returned, including files, we only want to process directories here if (remoteItem.remoteType == ItemType.dir) { // Check if this path is specifically excluded by 'skip_dir', but only if 'skip_dir' is not empty if (appConfig.getValueString("skip_dir") != "") { // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched if (selectiveSync.isDirNameExcluded(remoteItem.name)) { // This directory name is excluded if (verboseLogging) {addLogEntry("Skipping path - excluded by skip_dir config: " ~ remoteItem.name, ["verbose"]);} continue; } } // Directory name is not excluded or skip_dir is not populated if (!appConfig.suppressLoggingOutput) { // So that we represent correctly where this shared folder is, calculate the path string sharedFolderLogicalPath = computeItemPath(remoteItem.driveId, remoteItem.id); addLogEntry("Syncing this OneDrive Business Shared Folder: " ~ sharedFolderLogicalPath); } // Debug log output if (debugLogging) { addLogEntry("Fetching /delta API response for:", ["debug"]); addLogEntry(" remoteItem.remoteDriveId: " ~ remoteItem.remoteDriveId, ["debug"]); addLogEntry(" remoteItem.remoteId: " ~ remoteItem.remoteId, ["debug"]); } // Check this OneDrive Business Shared Folder for changes fetchOneDriveDeltaAPIResponse(remoteItem.remoteDriveId, remoteItem.remoteId, remoteItem.name); // Process any download activities or cleanup actions for this OneDrive Business Shared Folder processDownloadActivities(); } } // Clear the array remoteItems = []; // OneDrive Business Shared File Handling - but only if this option is enabled if (appConfig.getValueBool("sync_business_shared_files")) { // We need to create a 'new' local folder in the 'sync_dir' where these shared files & associated folder structure will reside // Whilst these files are synced locally, the entire folder structure will need to be excluded from syncing back to OneDrive // But file changes , *if any* , will need to be synced back to the original shared file location // . // ├── Files Shared With Me -> Directory should not be created online | Not Synced // │ └── Display Name (email address) (of Account who shared file) -> Directory should not be created online | Not Synced // │ │ └── shared file.ext -> File synced with original shared file location on remote drive // │ │ └── shared file.ext -> File synced with original shared file location on remote drive // │ │ └── ...... -> File synced with original shared file location on remote drive // │ └── Display Name (email address) ... // │ └── shared file.ext .... -> File synced with original shared file location on remote drive // Does the Local Folder to store the OneDrive Business Shared Files exist? if (!exists(appConfig.configuredBusinessSharedFilesDirectoryName)) { // Folder does not exist locally and needs to be created addLogEntry("Creating the OneDrive Business Shared Files Local Directory: " ~ appConfig.configuredBusinessSharedFilesDirectoryName); if (!dryRun) { // Local folder does not exist, thus needs to be created try { // Attempt path creation mkdirRecurse(appConfig.configuredBusinessSharedFilesDirectoryName); } catch (std.file.FileException e) { // Creating the path failed addLogEntry("ERROR: Unable to create the OneDrive Business Shared Files Local Directory: " ~ e.msg, ["info", "notify"]); } } // As this will not be created online, generate a response so it can be saved to the database Item sharedFilesPath = makeItem(createFakeResponse(baseName(appConfig.configuredBusinessSharedFilesDirectoryName))); // Add DB record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a DB record for storing OneDrive Business Shared Files: " ~ to!string(sharedFilesPath), ["debug"]);} itemDB.upsert(sharedFilesPath); } else { // Folder exists locally, is the folder in the database? // Query DB for this path Item dbRecord; if (!itemDB.selectByPath(baseName(appConfig.configuredBusinessSharedFilesDirectoryName), appConfig.defaultDriveId, dbRecord)) { // As this will not be created online, generate a response so it can be saved to the database Item sharedFilesPath = makeItem(createFakeResponse(baseName(appConfig.configuredBusinessSharedFilesDirectoryName))); // Add DB record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a DB record for storing OneDrive Business Shared Files: " ~ to!string(sharedFilesPath), ["debug"]);} itemDB.upsert(sharedFilesPath); } } // Query for OneDrive Business Shared Files if (verboseLogging) {addLogEntry("Checking for any applicable OneDrive Business Shared Files which need to be synced locally", ["verbose"]);} queryBusinessSharedObjects(); // Download any OneDrive Business Shared Files processDownloadActivities(); } } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Cleanup arrays when used in --monitor loops void cleanupArrays() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Debug what we are doing if (debugLogging) {addLogEntry("Cleaning up all internal arrays used when processing data", ["debug"]);} // Multi Dimensional Arrays idsToDelete.length = 0; idsFaked.length = 0; databaseItemsWhereContentHasChanged.length = 0; // JSON Items Arrays jsonItemsToProcess = []; fileJSONItemsToDownload = []; jsonItemsToResumeUpload = []; jsonItemsToResumeDownload = []; // String Arrays fileDownloadFailures = []; pathFakeDeletedArray = []; pathsRenamed = []; newLocalFilesToUploadToOneDrive = []; fileUploadFailures = []; posixViolationPaths = []; businessSharedFoldersOnlineToSkip = []; interruptedUploadsSessionFiles = []; interruptedDownloadFiles = []; pathsToCreateOnline = []; databaseItemsToDeleteOnline = []; pathsRetained = []; // Perform Garbage Collection on this destroyed curl engine GC.collect(); if (debugLogging) {addLogEntry("Cleaning of internal arrays complete", ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Configure singleDirectoryScope = true if this function is called // By default, singleDirectoryScope = false void setSingleDirectoryScope(string normalisedSingleDirectoryPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables Item searchItem; JSONValue onlinePathData; // Set the main flag singleDirectoryScope = true; // What are we doing? addLogEntry("The OneDrive Client was asked to search for this directory online and create it if it's not located: " ~ normalisedSingleDirectoryPath); // Query the OneDrive API for the specified path online // In a --single-directory scenario, we need to traverse the entire path that we are wanting to sync // and then check the path element does it exist online, if it does, is it a POSIX match, or if it does not, create the path // Once we have searched online, we have the right drive id and item id so that we can downgrade the sync status, then build up // any object items from that location // This is because, in a --single-directory scenario, any folder in the entire path tree could be a 'case-insensitive match' try { onlinePathData = queryOneDriveForSpecificPathAndCreateIfMissing(normalisedSingleDirectoryPath, true); } catch (PosixException e) { displayPosixErrorMessage(e.msg); addLogEntry("ERROR: Requested directory to search for and potentially create has a 'case-insensitive match' to an existing directory on Microsoft OneDrive online."); } // Was a valid JSON response provided? if (onlinePathData.type() == JSONType.object) { // Valid JSON item was returned searchItem = makeItem(onlinePathData); if (debugLogging) {addLogEntry("searchItem: " ~ to!string(searchItem), ["debug"]);} // Is this item a potential Shared Folder? // Is this JSON a remote object if (isItemRemote(onlinePathData)) { // Is this a Personal Account Type or has 'sync_business_shared_items' been enabled? if ((appConfig.accountType == "personal") || (appConfig.getValueBool("sync_business_shared_items"))) { // The path we are seeking is remote to our account drive id searchItem.driveId = onlinePathData["remoteItem"]["parentReference"]["driveId"].str; searchItem.id = onlinePathData["remoteItem"]["id"].str; // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test searchItem.driveId = transformToLowerCase(searchItem.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (searchItem.driveId != appConfig.defaultDriveId) { searchItem.driveId = testProvidedDriveIdForLengthIssue(searchItem.driveId); } } // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(onlinePathData); } else { // This is a shared folder location, but we are not a 'personal' account, and 'sync_business_shared_items' has not been enabled addLogEntry(); addLogEntry("ERROR: The requested --single-directory path to sync is a Shared Folder online and 'sync_business_shared_items' is not enabled"); addLogEntry(); forceExit(); } } // Set these items so that these can be used as required singleDirectoryScopeDriveId = searchItem.driveId; singleDirectoryScopeItemId = searchItem.id; } else { addLogEntry(); addLogEntry("ERROR: The requested --single-directory path to sync has generated an error. Please correct this error and try again."); addLogEntry(); forceExit(); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query OneDrive API for /delta changes and iterate through items online void fetchOneDriveDeltaAPIResponse(string driveIdToQuery = null, string itemIdToQuery = null, string sharedFolderName = null) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } string deltaLink = null; string currentDeltaLink = null; string databaseDeltaLink; JSONValue deltaChanges; long responseBundleCount; long jsonItemsReceived = 0; // Reset jsonItemsToProcess & processedCount jsonItemsToProcess = []; processedCount = 0; // Reset generateSimulatedDeltaResponse generateSimulatedDeltaResponse = false; // Reset Shared Folder Flags for 'sync_list' processing sharedFolderDeltaGeneration = false; currentSharedFolderName = ""; // Was a driveId provided as an input if (strip(driveIdToQuery).empty) { // No provided driveId to query, use the account default driveIdToQuery = appConfig.defaultDriveId; if (debugLogging) { addLogEntry("driveIdToQuery was empty, setting to appConfig.defaultDriveId", ["debug"]); addLogEntry("driveIdToQuery: " ~ driveIdToQuery, ["debug"]); } } // Was an itemId provided as an input if (strip(itemIdToQuery).empty) { // No provided itemId to query, use the account default itemIdToQuery = appConfig.defaultRootId; if (debugLogging) { addLogEntry("itemIdToQuery was empty, setting to appConfig.defaultRootId", ["debug"]); addLogEntry("itemIdToQuery: " ~ itemIdToQuery, ["debug"]); } } // What OneDrive API query do we use? // - Are we running against a National Cloud Deployments that does not support /delta ? // National Cloud Deployments do not support /delta as a query // https://docs.microsoft.com/en-us/graph/deployments#supported-features // // - Are we performing a --single-directory sync, which will exclude many items online, focusing in on a specific online directory // // - Are we performing a --download-only --cleanup-local-files action? // - If we are, and we use a normal /delta query, we get all the local 'deleted' objects as well. // - If the user deletes a folder online, then replaces it online, we download the deletion events and process the new 'upload' via the web interface .. // the net effect of this, is that the valid local files we want to keep, are actually deleted ...... not desirable if ((singleDirectoryScope) || (nationalCloudDeployment) || (cleanupLocalFiles)) { // Generate a simulated /delta response so that we correctly capture the current online state, less any 'online' delete and replace activity generateSimulatedDeltaResponse = true; } // Shared Folders, by nature of where that path has been shared with us, we cannot use /delta against that path, as this queries the entire 'other persons' drive: // Syncing this OneDrive Business Shared Folder: Sub Folder 2 // Fetching /delta response from the OneDrive API for Drive ID: b!fZgJhK-pU0eTQpylvmoYCkE4YgH_KRNDlxjRx9OWNqmV9Q_E_uWdRJKIB5L_ruPN // Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 18 // Skipping path - excluded by sync_list config: Sub Folder Share/Sub Folder 1/Sub Folder 2 // // When using 'sync_list' potentially nothing is going to match, as, we are getting the 'whole' path from their 'root' , not just the folder shared with us if (!sharedFolderName.empty) { // When using 'sync_list' we need to do this sharedFolderDeltaGeneration = true; currentSharedFolderName = sharedFolderName; generateSimulatedDeltaResponse = true; } // Reset latestDeltaLink & deltaLinkCache latestDeltaLink = null; deltaLinkCache.driveId = null; deltaLinkCache.itemId = null; deltaLinkCache.latestDeltaLink = null; // Perform Garbage Collection GC.collect(); // What /delta query do we use? if (!generateSimulatedDeltaResponse) { // This should be the majority default pathway application use // Do we need to perform a Full Scan True Up? Is 'appConfig.fullScanTrueUpRequired' set to 'true'? if (appConfig.fullScanTrueUpRequired) { addLogEntry("Performing a full scan of online data to ensure consistent local state"); if (debugLogging) {addLogEntry("Setting currentDeltaLink = null", ["debug"]);} currentDeltaLink = null; } else { // Try and get the current Delta Link from the internal cache, this saves a DB I/O call currentDeltaLink = getDeltaLinkFromCache(deltaLinkInfo, driveIdToQuery); // Is currentDeltaLink empty (no cached entry found) ? if (currentDeltaLink.empty) { // Try and get the current delta link from the database for this DriveID and RootID databaseDeltaLink = itemDB.getDeltaLink(driveIdToQuery, itemIdToQuery); if (!databaseDeltaLink.empty) { if (debugLogging) {addLogEntry("Using database stored deltaLink", ["debug"]);} currentDeltaLink = databaseDeltaLink; } else { if (debugLogging) {addLogEntry("Zero deltaLink available for use, we will be performing a full online scan", ["debug"]);} currentDeltaLink = null; } } else { // Log that we are using the deltaLink for cache if (debugLogging) {addLogEntry("Using cached deltaLink", ["debug"]);} } } // Dynamic output for non-verbose and verbose run so that the user knows something is being retrieved from the OneDrive API if (appConfig.verbosityCount == 0) { if (!appConfig.suppressLoggingOutput) { addProcessingLogHeaderEntry("Fetching items from the OneDrive API for Drive ID: " ~ driveIdToQuery, appConfig.verbosityCount); } } else { if (verboseLogging) {addLogEntry("Fetching /delta response from the OneDrive API for Drive ID: " ~ driveIdToQuery, ["verbose"]);} } // Create a new API Instance for querying the actual /delta and initialise it OneDriveApi getDeltaDataOneDriveApiInstance; getDeltaDataOneDriveApiInstance = new OneDriveApi(appConfig); getDeltaDataOneDriveApiInstance.initialise(); // Get the /delta changes via the OneDrive API while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Increment responseBundleCount responseBundleCount++; // Ensure deltaChanges is empty before we query /delta deltaChanges = null; // Perform Garbage Collection GC.collect(); // getDeltaChangesByItemId has the re-try logic for transient errors deltaChanges = getDeltaChangesByItemId(driveIdToQuery, itemIdToQuery, currentDeltaLink, getDeltaDataOneDriveApiInstance); // If the initial deltaChanges response is an invalid JSON object, keep trying until we get a valid response .. if (deltaChanges.type() != JSONType.object) { // While the response is not a JSON Object or the Exit Handler has not been triggered while (deltaChanges.type() != JSONType.object) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Handle the invalid JSON response and retry if (debugLogging) {addLogEntry("ERROR: Query of the OneDrive API via deltaChanges = getDeltaChangesByItemId() returned an invalid JSON response", ["debug"]);} deltaChanges = getDeltaChangesByItemId(driveIdToQuery, itemIdToQuery, currentDeltaLink, getDeltaDataOneDriveApiInstance); } } long nrChanges = count(deltaChanges["value"].array); int changeCount = 0; if (appConfig.verbosityCount == 0) { // Dynamic output for a non-verbose run so that the user knows something is happening if (!appConfig.suppressLoggingOutput) { addProcessingDotEntry(); } } else { if (verboseLogging) {addLogEntry("Processing API Response Bundle: " ~ to!string(responseBundleCount) ~ " - Quantity of 'changes|items' in this bundle to process: " ~ to!string(nrChanges), ["verbose"]);} } // Update the count of items received jsonItemsReceived = jsonItemsReceived + nrChanges; // The 'deltaChanges' response may contain either @odata.nextLink or @odata.deltaLink // Check for @odata.nextLink if ("@odata.nextLink" in deltaChanges) { // @odata.nextLink is the pointer within the API to the next '200+' JSON bundle - this is the checkpoint link for this bundle // This URL changes between JSON bundle sets // Log the action of setting currentDeltaLink to @odata.nextLink if (debugLogging) {addLogEntry("Setting currentDeltaLink to @odata.nextLink: " ~ deltaChanges["@odata.nextLink"].str, ["debug"]);} // Update currentDeltaLink to @odata.nextLink for the next '200+' JSON bundle - this is the checkpoint link for this bundle currentDeltaLink = deltaChanges["@odata.nextLink"].str; } // Check for @odata.deltaLink - usually only in the LAST JSON changeset bundle if ("@odata.deltaLink" in deltaChanges) { // @odata.deltaLink is the pointer that finalises all the online 'changes' for this particular checkpoint // When the API is queried again, this is fetched from the DB as this is the starting point // The API issue here is - the LAST JSON bundle will ONLY ever contain this item, meaning if this is then committed to the database // if there has been any file download failures from within this LAST JSON bundle, the only way to EVER re-try the failed items is for the user to perform a --resync // This is an API capability gap: // // .. // @odata.nextLink: https://graph.microsoft.com/v1.0/drives//items//delta?token= // Processing API Response Bundle: 115 - Quantity of 'changes|items' in this bundle to process: 204 // .. // @odata.nextLink: https://graph.microsoft.com/v1.0/drives//items//delta?token= // Processing API Response Bundle: 127 - Quantity of 'changes|items' in this bundle to process: 204 // @odata.nextLink: https://graph.microsoft.com/v1.0/drives//items//delta?token= // Processing API Response Bundle: 128 - Quantity of 'changes|items' in this bundle to process: 176 // @odata.deltaLink: https://graph.microsoft.com/v1.0/drives//items//delta?token= // Finished processing /delta JSON response from the OneDrive API // Log the action of setting currentDeltaLink to @odata.deltaLink if (debugLogging) {addLogEntry("Setting currentDeltaLink to (@odata.deltaLink): " ~ deltaChanges["@odata.deltaLink"].str, ["debug"]);} // Update currentDeltaLink to @odata.deltaLink as the final checkpoint URL for this entire JSON response set currentDeltaLink = deltaChanges["@odata.deltaLink"].str; // Store this currentDeltaLink as latestDeltaLink latestDeltaLink = deltaChanges["@odata.deltaLink"].str; // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test driveIdToQuery = transformToLowerCase(driveIdToQuery); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (driveIdToQuery != appConfig.defaultDriveId) { driveIdToQuery = testProvidedDriveIdForLengthIssue(driveIdToQuery); } } // Update deltaLinkCache deltaLinkCache.driveId = driveIdToQuery; deltaLinkCache.itemId = itemIdToQuery; deltaLinkCache.latestDeltaLink = currentDeltaLink; } // We have a valid deltaChanges JSON array. This means we have at least 200+ JSON items to process. // The API response however cannot be run in parallel as the OneDrive API sends the JSON items in the order in which they must be processed auto jsonArrayToProcess = deltaChanges["value"].array; // To allow for better debugging, what are all the JSON elements in the array the API responded with in this set? if (count(jsonArrayToProcess) > 0) { if (debugLogging) { string debugLogHeader = format("=============================== jsonArrayToProcess - response bundle %s ===================================", to!string(responseBundleCount)); addLogEntry(debugLogHeader, ["debug"]); addLogEntry(to!string(jsonArrayToProcess), ["debug"]); addLogEntry(debugLogBreakType2, ["debug"]); } } // Process the change set foreach (onedriveJSONItem; jsonArrayToProcess) { // increment change count for this item changeCount++; // Process the received OneDrive object item JSON for this JSON bundle // This will determine its initial applicability and perform some initial processing on the JSON if required processDeltaJSONItem(onedriveJSONItem, nrChanges, changeCount, responseBundleCount, singleDirectoryScope); } // Clear up this data jsonArrayToProcess = null; // Perform Garbage Collection GC.collect(); // Is latestDeltaLink matching deltaChanges["@odata.deltaLink"].str ? if ("@odata.deltaLink" in deltaChanges) { if (latestDeltaLink == deltaChanges["@odata.deltaLink"].str) { // break out of the 'while (true)' loop break; } } // Cleanup deltaChanges as this is no longer needed deltaChanges = null; // Perform Garbage Collection GC.collect(); // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } // Terminate getDeltaDataOneDriveApiInstance here getDeltaDataOneDriveApiInstance.releaseCurlEngine(); getDeltaDataOneDriveApiInstance = null; // Perform Garbage Collection on this destroyed curl engine GC.collect(); // To finish off the JSON processing items, this is needed to reflect this in the log if (debugLogging) {addLogEntry(debugLogBreakType1, ["debug"]);} // Log that we have finished querying the /delta API if (appConfig.verbosityCount == 0) { if (!appConfig.suppressLoggingOutput) { // Close out the '....' being printed to the console completeProcessingDots(); } } else { if (verboseLogging) {addLogEntry("Finished processing /delta JSON response from the OneDrive API", ["verbose"]);} } // If this was set, now unset it, as this will have been completed, so that for a true up, we dont do a double full scan if (appConfig.fullScanTrueUpRequired) { if (debugLogging) {addLogEntry("Unsetting fullScanTrueUpRequired as this has been performed", ["debug"]);} appConfig.fullScanTrueUpRequired = false; } // Cleanup deltaChanges as this is no longer needed deltaChanges = null; // Perform Garbage Collection GC.collect(); } else { // Why are we generating a /delta response if (debugLogging) { addLogEntry("Why are we generating a /delta response:", ["debug"]); addLogEntry(" singleDirectoryScope: " ~ to!string(singleDirectoryScope), ["debug"]); addLogEntry(" nationalCloudDeployment: " ~ to!string(nationalCloudDeployment), ["debug"]); addLogEntry(" cleanupLocalFiles: " ~ to!string(cleanupLocalFiles), ["debug"]); addLogEntry(" sharedFolderName: " ~ sharedFolderName, ["debug"]); } // What 'path' are we going to start generating the response for string pathToQuery; // If --single-directory has been called, use the value that has been set if (singleDirectoryScope) { pathToQuery = appConfig.getValueString("single_directory"); } // We could also be syncing a Shared Folder of some description - is this empty? if (!sharedFolderName.empty) { // We need to build 'pathToQuery' to support Shared Folders being anywhere in the directory structure (#2824) // Is the itemIdToQuery in the database? If this is not there, we cannot build the path if (itemDB.idInLocalDatabase(driveIdToQuery, itemIdToQuery)) { // The entries are in our DB, but we need to use our Drive details to compute the actual local path the the point of the 'remote' record and DB Tie Record Item remoteEntryItem; itemDB.selectByRemoteEntryByName(sharedFolderName, remoteEntryItem); // Use the 'remote' item type DB entry to calculate the local path of this item, which then will match the path online for this Shared Folder string computedLocalPathToQuery = computeItemPath(remoteEntryItem.driveId, remoteEntryItem.id); // If we have a computed path, use it, else use 'sharedFolderName' if (!computedLocalPathToQuery.empty) { // computedLocalPathToQuery is not empty pathToQuery = computedLocalPathToQuery; } else { // computedLocalPathToQuery is empty pathToQuery = sharedFolderName; } } else { // shared folder details are not even in the database ... fall back to this pathToQuery = sharedFolderName; } // At this point we have either calculated the shared folder path, or not and can attempt to generate a /delta response from that path entry online } // Generate the simulated /delta response // // The generated /delta response however contains zero deleted JSON items, so the only way that we can track this, is if the object was in sync // we have the object in the database, thus, what we need to do is for every DB object in the tree of items, flag 'syncStatus' as 'N', then when we process // the returned JSON items from the API, we flag the item as back in sync, then we can cleanup any out-of-sync items // // The flagging of the local database items to 'N' is handled within the generateDeltaResponse() function // // When these JSON items are then processed, if the item exists online, and is in the DB, and that the values match, the DB item is flipped back to 'Y' // This then allows the application to look for any remaining 'N' values, and delete these as no longer needed locally deltaChanges = generateDeltaResponse(pathToQuery); // deltaChanges must be a valid JSON object / array of data if (deltaChanges.type() == JSONType.object) { // How many changes were returned? long nrChanges = count(deltaChanges["value"].array); int changeCount = 0; if (debugLogging) {addLogEntry("API Response Bundle: " ~ to!string(responseBundleCount) ~ " - Quantity of 'changes|items' in this bundle to process: " ~ to!string(nrChanges), ["debug"]);} // Update the count of items received jsonItemsReceived = jsonItemsReceived + nrChanges; // The API response however cannot be run in parallel as the OneDrive API sends the JSON items in the order in which they must be processed auto jsonArrayToProcess = deltaChanges["value"].array; foreach (onedriveJSONItem; deltaChanges["value"].array) { // increment change count for this item changeCount++; // Process the received OneDrive object item JSON for this JSON bundle // When we generate a /delta response .. there is no currentDeltaLink value processDeltaJSONItem(onedriveJSONItem, nrChanges, changeCount, responseBundleCount, singleDirectoryScope); } // Clear up this data jsonArrayToProcess = null; // To finish off the JSON processing items, this is needed to reflect this in the log if (debugLogging) {addLogEntry(debugLogBreakType1, ["debug"]);} // Log that we have finished generating our self generated /delta response if (!appConfig.suppressLoggingOutput) { addLogEntry("Finished processing self generated /delta JSON response from the OneDrive API"); } } // Cleanup deltaChanges as this is no longer needed deltaChanges = null; // Perform Garbage Collection GC.collect(); } // Cleanup deltaChanges as this is no longer needed deltaChanges = null; // Perform Garbage Collection GC.collect(); // We have JSON items received from the OneDrive API if (debugLogging) { addLogEntry("Number of JSON Objects received from OneDrive API: " ~ to!string(jsonItemsReceived), ["debug"]); addLogEntry("Number of JSON Objects already processed (root and deleted items): " ~ to!string((jsonItemsReceived - jsonItemsToProcess.length)), ["debug"]); // We should have now at least processed all the JSON items as returned by the /delta call // Additionally, we should have a new array, that now contains all the JSON items we need to process that are non 'root' or deleted items addLogEntry("Number of JSON items submitted for further processing is: " ~ to!string(jsonItemsToProcess.length), ["debug"]); } // Are there items to process? if (jsonItemsToProcess.length > 0) { // Lets deal with the JSON items in a batch process size_t batchSize = 500; long batchCount = (jsonItemsToProcess.length + batchSize - 1) / batchSize; long batchesProcessed = 0; // Dynamic output for a non-verbose run so that the user knows something is happening if (!appConfig.suppressLoggingOutput) { addProcessingLogHeaderEntry("Processing " ~ to!string(jsonItemsToProcess.length) ~ " applicable JSON items received from Microsoft OneDrive", appConfig.verbosityCount); } // For each batch, process the JSON items that need to be now processed. // 'root' and deleted objects have already been handled foreach (batchOfJSONItems; jsonItemsToProcess.chunks(batchSize)) { // Chunk the total items to process into 500 lot items batchesProcessed++; if (appConfig.verbosityCount == 0) { // Dynamic output for a non-verbose run so that the user knows something is happening if (!appConfig.suppressLoggingOutput) { addProcessingDotEntry(); } } else { if (verboseLogging) {addLogEntry("Processing OneDrive JSON item batch [" ~ to!string(batchesProcessed) ~ "/" ~ to!string(batchCount) ~ "] to ensure consistent local state", ["verbose"]);} } // Process the batch processJSONItemsInBatch(batchOfJSONItems, batchesProcessed, batchCount); // To finish off the JSON processing items, this is needed to reflect this in the log if (debugLogging) {addLogEntry(debugLogBreakType1, ["debug"]);} // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); } if (appConfig.verbosityCount == 0) { // close off '.' output if (!appConfig.suppressLoggingOutput) { // Close out the '....' being printed to the console completeProcessingDots(); } } // Debug output - what was processed if (debugLogging) { addLogEntry("Number of JSON items to process is: " ~ to!string(jsonItemsToProcess.length), ["debug"]); addLogEntry("Number of JSON items processed was: " ~ to!string(processedCount), ["debug"]); addLogEntry("", ["debug"]); string jsonProcessingCompleteLineEntry = format("Processing of JSON items from driveId %s and itemId %s is complete", driveIdToQuery, itemIdToQuery); addLogEntry(jsonProcessingCompleteLineEntry, ["debug"]); addLogEntry("", ["debug"]); } // Notification to user regarding number of objects received from OneDrive API if (jsonItemsReceived >= 300000) { // 'driveIdToQuery' should be the drive where the JSON responses came from string objectsExceedLimitWarning = format("WARNING: The number of objects stored online in '%s' exceeds Microsoft OneDrive's recommended limit. This may cause unreliable application behaviour due to inconsistent or incomplete API responses. Immediate action is strongly advised to avoid data integrity issues.", driveIdToQuery); addLogEntry(objectsExceedLimitWarning, ["info", "notify"]); } // Free up memory and items processed as it is pointless now having this data around jsonItemsToProcess = []; // Perform Garbage Collection on this destroyed curl engine GC.collect(); } else { if (!appConfig.suppressLoggingOutput) { addLogEntry("No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive"); } } // Keep the DriveDetailsCache array with unique entries only DriveDetailsCache cachedOnlineDriveData; if (!canFindDriveId(driveIdToQuery, cachedOnlineDriveData)) { // Add this driveId to the drive cache addOrUpdateOneDriveOnlineDetails(driveIdToQuery); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process the /delta API JSON response items void processDeltaJSONItem(JSONValue onedriveJSONItem, long nrChanges, int changeCount, long responseBundleCount, bool singleDirectoryScope) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Variables for this JSON item string thisItemId; bool itemIsRoot = false; bool handleItemAsRootObject = false; bool itemIsDeletedOnline = false; bool itemHasParentReferenceId = false; bool itemHasParentReferencePath = false; bool itemIdMatchesDefaultRootId = false; bool itemNameExplicitMatchRoot = false; bool itemIsRemoteItem = false; string objectParentDriveId; string objectParentId; MonoTime jsonProcessingStartTime; // Debugging the processing start of the JSON item if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); jsonProcessingStartTime = MonoTime.currTime(); addLogEntry("Processing OneDrive Item " ~ to!string(changeCount) ~ " of " ~ to!string(nrChanges) ~ " from API Response Bundle " ~ to!string(responseBundleCount), ["debug"]); } // Issue #3336 - Convert driveId to lowercase if (appConfig.accountType == "personal") { // We must massage this raw JSON record to force the onedriveJSONItem["parentReference"]["driveId"] to lowercase if (hasParentReferenceDriveId(onedriveJSONItem)) { // This JSON record has a driveId we now must manipulate to lowercase string originalDriveIdValue = onedriveJSONItem["parentReference"]["driveId"].str; onedriveJSONItem["parentReference"]["driveId"] = transformToLowerCase(originalDriveIdValue); } } // Debug output of the raw JSON item we are processing if (debugLogging) { addLogEntry("Raw JSON OneDrive Item: " ~ sanitiseJSONItem(onedriveJSONItem), ["debug"]); } // What is this item's id thisItemId = onedriveJSONItem["id"].str; // Is this a deleted item - only calculate this once itemIsDeletedOnline = isItemDeleted(onedriveJSONItem); if (!itemIsDeletedOnline) { // This is not a deleted item if (debugLogging) {addLogEntry("This item is not a OneDrive online deletion change", ["debug"]);} // Only calculate these elements once itemIsRoot = isItemRoot(onedriveJSONItem); itemHasParentReferenceId = hasParentReferenceId(onedriveJSONItem); itemIdMatchesDefaultRootId = (thisItemId == appConfig.defaultRootId); itemNameExplicitMatchRoot = (onedriveJSONItem["name"].str == "root"); objectParentDriveId = onedriveJSONItem["parentReference"]["driveId"].str; if (itemHasParentReferenceId) { objectParentId = onedriveJSONItem["parentReference"]["id"].str; } itemIsRemoteItem = isItemRemote(onedriveJSONItem); // Test is this is the OneDrive Users Root? // Debug output of change evaluation items if (debugLogging) { addLogEntry("defaultRootId = " ~ appConfig.defaultRootId, ["debug"]); addLogEntry("thisItemName = " ~ onedriveJSONItem["name"].str, ["debug"]); addLogEntry("thisItemId = " ~ thisItemId, ["debug"]); addLogEntry("thisItemId == defaultRootId = " ~ to!string(itemIdMatchesDefaultRootId), ["debug"]); addLogEntry("isItemRoot(onedriveJSONItem) = " ~ to!string(itemIsRoot), ["debug"]); addLogEntry("onedriveJSONItem['name'].str == 'root' = " ~ to!string(itemNameExplicitMatchRoot), ["debug"]); addLogEntry("itemHasParentReferenceId = " ~ to!string(itemHasParentReferenceId), ["debug"]); addLogEntry("itemIsRemoteItem = " ~ to!string(itemIsRemoteItem), ["debug"]); } if ( (itemIdMatchesDefaultRootId || singleDirectoryScope) && itemIsRoot && itemNameExplicitMatchRoot) { // This IS a OneDrive Root item or should be classified as such in the case of 'singleDirectoryScope' if (debugLogging) {addLogEntry("JSON item will flagged as a 'root' item", ["debug"]);} handleItemAsRootObject = true; } } // How do we handle this JSON item from the OneDrive API? // Is this a confirmed 'root' item, has no Parent ID, or is a Deleted Item if (handleItemAsRootObject || !itemHasParentReferenceId || itemIsDeletedOnline){ // Is a root item, has no id in parentReference or is a OneDrive deleted item if (debugLogging) { addLogEntry("objectParentDriveId = " ~ objectParentDriveId, ["debug"]); addLogEntry("handleItemAsRootObject = " ~ to!string(handleItemAsRootObject), ["debug"]); addLogEntry("itemHasParentReferenceId = " ~ to!string(itemHasParentReferenceId), ["debug"]); addLogEntry("itemIsDeletedOnline = " ~ to!string(itemIsDeletedOnline), ["debug"]); addLogEntry("Handling change immediately as 'root item', or has no parent reference id or is a deleted item", ["debug"]); } // OK ... do something with this JSON post here .... processRootAndDeletedJSONItems(onedriveJSONItem, objectParentDriveId, handleItemAsRootObject, itemIsDeletedOnline, itemHasParentReferenceId); } else { // Do we need to update this RAW JSON from OneDrive? bool sharedFolderRenameCheck = false; // What account type is this? if (appConfig.accountType == "personal") { // flag this by default as we always sync personal shared folders by default sharedFolderRenameCheck = true; } else { // business | DocumentLibrary if (appConfig.getValueBool("sync_business_shared_items")) { // flag this sharedFolderRenameCheck = true; } } // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { objectParentDriveId = transformToLowerCase(objectParentDriveId); } // Do we check if this JSON needs updating? if ((objectParentDriveId != appConfig.defaultDriveId) && (sharedFolderRenameCheck)) { // Potentially need to update this JSON data if (debugLogging) {addLogEntry("Potentially need to update this source JSON .... need to check the database", ["debug"]);} // Check the DB for 'remote' objects, searching 'remoteDriveId' and 'remoteId' items for this remoteItem.driveId and remoteItem.id Item remoteDBItem; itemDB.selectByRemoteId(objectParentDriveId, thisItemId, remoteDBItem); // Is the data that was returned from the database what we are looking for? if ((remoteDBItem.remoteDriveId == objectParentDriveId) && (remoteDBItem.remoteId == thisItemId)) { // Yes, this is the record we are looking for if (debugLogging) {addLogEntry("DB Item response for remoteDBItem: " ~ to!string(remoteDBItem), ["debug"]);} // Must compare remoteDBItem.name with remoteItem.name if (remoteDBItem.name != onedriveJSONItem["name"].str) { // Update JSON Item string actualOnlineName = onedriveJSONItem["name"].str; if (debugLogging) { addLogEntry("Updating source JSON 'name' to that which is the actual local directory", ["debug"]); addLogEntry("onedriveJSONItem['name'] was: " ~ onedriveJSONItem["name"].str, ["debug"]); addLogEntry("Updating onedriveJSONItem['name'] to: " ~ remoteDBItem.name, ["debug"]); } onedriveJSONItem["name"] = remoteDBItem.name; if (debugLogging) {addLogEntry("onedriveJSONItem['name'] now: " ~ onedriveJSONItem["name"].str, ["debug"]);} // Add the original name to the JSON onedriveJSONItem["actualOnlineName"] = actualOnlineName; } } } // Do we discard this JSON item? bool discardDeltaJSONItem = false; // Microsoft OneNote container objects present neither folder or file but contain a 'package' element // "package": { // "type": "oneNote" // }, // Confirmed with Microsoft OneDrive Personal // Confirmed with Microsoft OneDrive Business if (isOneNotePackageFolder(onedriveJSONItem)) { // This JSON has this element if (verboseLogging) {addLogEntry("Skipping path - The Microsoft OneNote Notebook Package '" ~ generatePathFromJSONData(onedriveJSONItem) ~ "' is not supported by this client", ["verbose"]);} discardDeltaJSONItem = true; // Add this 'id' to onenotePackageIdentifiers as a future 'catch all' for any objects inside this container if (!onenotePackageIdentifiers.canFind(thisItemId)) { if (debugLogging) {addLogEntry("Adding 'thisItemId' to onenotePackageIdentifiers: " ~ to!string(thisItemId), ["debug"]);} onenotePackageIdentifiers ~= thisItemId; } } // Microsoft OneDrive OneNote file objects will report as files but have 'application/msonenote' or 'application/octet-stream' as their mime type and will not have any hash entry // Is there a 'file' JSON element and it has a 'mimeType' element? if (isItemFile(onedriveJSONItem) && hasMimeType(onedriveJSONItem)) { // Is the mimeType 'application/msonenote' or 'application/octet-stream' // However there is API inconsistency here between Personal and Business Accounts // Personal OneNote .onetoc2 and .one items all report mimeType as 'application/msonenote' // Business OneNote .onetoc2 and .one items however are different: // .one = 'application/msonenote' mimeType // .onetoc2 = 'application/octet-stream' mimeType if (isMicrosoftOneNoteMimeType1(onedriveJSONItem) || isMicrosoftOneNoteMimeType2(onedriveJSONItem)) { // We have a 'mimeType' match // What is the file extension? // .one (Type1) // .onetoc2 (Type2) if (isMicrosoftOneNoteFileExtensionType1(onedriveJSONItem) || isMicrosoftOneNoteFileExtensionType2(onedriveJSONItem)) { // Extreme confidence this JSON is a Microsoft OneNote file reference which cannot be supported // Log that this will be skipped as this this is a Microsoft OneNote item and unsupported if (verboseLogging) {addLogEntry("Skipping path - The Microsoft OneNote Notebook File '" ~ generatePathFromJSONData(onedriveJSONItem) ~ "' is not supported by this client", ["verbose"]);} discardDeltaJSONItem = true; // Add the Parent ID to onenotePackageIdentifiers if (itemHasParentReferenceId) { // Add this 'id' to onenotePackageIdentifiers as a future 'catch all' for any objects inside this container if (!onenotePackageIdentifiers.canFind(objectParentId)) { if (debugLogging) {addLogEntry("Adding 'objectParentId' to onenotePackageIdentifiers: " ~ to!string(objectParentId), ["debug"]);} onenotePackageIdentifiers ~= objectParentId; } } } } } // Microsoft OneDrive OneNote 'internal recycle bin' items are a 'folder' , with a 'size' but have a specific name 'OneNote_RecycleBin', for example: // { // .... // "fileSystemInfo": { // "createdDateTime": "2025-03-10T17:11:15Z", // "lastModifiedDateTime": "2025-03-10T17:11:15Z" // }, // "folder": { // "childCount": 2 // }, // "id": "XXXXX", // "lastModifiedBy": { // XXXXX // }, // "name": "OneNote_RecycleBin", // "parentReference": { // "driveId": "abcde", // "driveType": "business", // "id": "abcde", // "name": "PARENT NAME - ONENOTE PACKAGE NAME", // "path": "/drives/path/to/parent", // "siteId": "XXXXX" // }, // "size": 17468 // } // // The only way we can block this download is looking at the 'name' component if (onedriveJSONItem["name"].str == "OneNote_RecycleBin") { // Log that this will be skipped as this this is a Microsoft OneNote item and unsupported if (verboseLogging) {addLogEntry("Skipping path - The Microsoft OneNote Notebook Recycle Bin '" ~ generatePathFromJSONData(onedriveJSONItem) ~ "' is not supported by this client", ["verbose"]);} discardDeltaJSONItem = true; // Add the Parent ID to onenotePackageIdentifiers if (itemHasParentReferenceId) { // Add this 'id' to onenotePackageIdentifiers as a future 'catch all' for any objects inside this container if (!onenotePackageIdentifiers.canFind(objectParentId)) { if (debugLogging) {addLogEntry("Adding 'objectParentId' to onenotePackageIdentifiers: " ~ to!string(objectParentId), ["debug"]);} onenotePackageIdentifiers ~= objectParentId; } } } // If we are not self-generating a /delta response, check this initial /delta JSON bundle item against the basic checks // of applicability against 'skip_file', 'skip_dir' and 'sync_list' // We only do this if we did not generate a /delta response, as generateDeltaResponse() performs the checkJSONAgainstClientSideFiltering() // against elements as it is building the /delta compatible response // If we blindly just 'check again' all JSON responses then there is potentially double JSON processing going on if we used generateDeltaResponse() if (!generateSimulatedDeltaResponse) { // Did we already exclude? if (!discardDeltaJSONItem) { // Check applicability against 'skip_file', 'skip_dir' and 'sync_list' discardDeltaJSONItem = checkJSONAgainstClientSideFiltering(onedriveJSONItem); } } // Add this JSON item for further processing if this is not being discarded if (!discardDeltaJSONItem) { // If 'personal' account type, we must validate ["parentReference"]["driveId"] value in this raw JSON // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { string existingDriveIdEntry = onedriveJSONItem["parentReference"]["driveId"].str; string newDriveIdEntry; // Perform the required length test if (existingDriveIdEntry.length < 16) { // existingDriveIdEntry value is not 16 characters in length // Is this 'driveId' in this JSON a 15 character representation of our actual 'driveId' which we have already corrected? if (appConfig.defaultDriveId.canFind(existingDriveIdEntry)) { // The JSON provided value is our 'driveId' // Debug logging for correction if (debugLogging) {addLogEntry("ONEDRIVE PERSONAL API BUG (Issue #3072): The provided raw JSON ['parentReference']['driveId'] value is not 16 Characters in length - correcting with validated 'appConfig.defaultDriveId' value", ["debug"]);} newDriveIdEntry = appConfig.defaultDriveId; } else { // No match, potentially a Shared Folder ... // Debug logging for correction if (debugLogging) {addLogEntry("ONEDRIVE PERSONAL API BUG (Issue #3072): The provided raw JSON ['parentReference']['driveId'] value is not 16 Characters in length - padding with leading zero's", ["debug"]);} // Generate the change newDriveIdEntry = to!string(existingDriveIdEntry.padLeft('0', 16)); // Explicitly use padLeft for leading zero padding, leave case as-is } // Make the change to the JSON data before submit for further processing onedriveJSONItem["parentReference"]["driveId"] = newDriveIdEntry; } } // Add onedriveJSONItem to jsonItemsToProcess if (debugLogging) { addLogEntry("Adding this raw JSON OneDrive Item to jsonItemsToProcess array for further processing", ["debug"]); if (itemIsRemoteItem) { addLogEntry("- This JSON record represents a online remote folder, thus needs special handling when being processed further", ["debug"]); } } jsonItemsToProcess ~= onedriveJSONItem; } else { // detail we are discarding the json if (debugLogging) {addLogEntry("Discarding this raw JSON OneDrive Item as this has been determined to be unwanted", ["debug"]);} } } // How long to initially process this JSON item if (debugLogging) { Duration jsonProcessingElapsedTime = MonoTime.currTime() - jsonProcessingStartTime; addLogEntry("Initial JSON item processing time: " ~ to!string(jsonProcessingElapsedTime), ["debug"]); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process 'root' and 'deleted' OneDrive JSON items void processRootAndDeletedJSONItems(JSONValue onedriveJSONItem, string driveId, bool handleItemAsRootObject, bool itemIsDeletedOnline, bool itemHasParentReferenceId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Use the JSON elements rather can computing a DB struct via makeItem() string thisItemId = onedriveJSONItem["id"].str; string thisItemDriveId = onedriveJSONItem["parentReference"]["driveId"].str; // Check if the item has been seen before Item existingDatabaseItem; bool existingDBEntry = itemDB.selectById(thisItemDriveId, thisItemId, existingDatabaseItem); // Is the item deleted online? if(!itemIsDeletedOnline) { // Is the item a confirmed root object? // The JSON item should be considered a 'root' item if: // 1. Contains a ["root"] element // 2. Has no ["parentReference"]["id"] ... #323 & #324 highlighted that this is false as some 'root' shared objects now can have an 'id' element .. OneDrive API change // 2. Has no ["parentReference"]["path"] // 3. Was detected by an input flag as to be handled as a root item regardless of actual status if ((handleItemAsRootObject) || (!itemHasParentReferenceId)) { if (debugLogging) {addLogEntry("Handing JSON object as OneDrive 'root' object", ["debug"]);} if (!existingDBEntry) { // we have not seen this item before saveItem(onedriveJSONItem); } } } else { // Change is to delete an item if (debugLogging) {addLogEntry("Handing a OneDrive Online Deleted Item", ["debug"]);} // Is the deleted item in our database? if (existingDBEntry) { // Is the item to delete locally actually in sync with OneDrive currently? // What is the source of this item data? string itemSource = "online"; // Compute this deleted items path based on the database entries string localPathToDelete = computeItemPath(existingDatabaseItem.driveId, existingDatabaseItem.parentId) ~ "/" ~ existingDatabaseItem.name; if (isItemSynced(existingDatabaseItem, localPathToDelete, itemSource)) { // Flag to delete if (debugLogging) {addLogEntry("Flagging to delete item locally due to online deletion event: " ~ to!string(onedriveJSONItem), ["debug"]);} // Use the DB entries returned - add the driveId, itemId and parentId values to the array idsToDelete ~= [existingDatabaseItem.driveId, existingDatabaseItem.id, existingDatabaseItem.parentId]; } else { // Local item is not in sync with the online item, but the online item has been deleted, and we are flagging to delete the local item // We need to determine the trigger for isItemSynced() returning false before we determine if we should make utilise safeBackup() // Is this the exact same file? // Test the file hash against the hash of the file online // Empirical evidence shows that Microsoft do not provide a 'valid' hash in JSON data for online deleted items, for example: // file":{"hashes":{"quickXorHash":"AAAAAAAAAAAAAAAAAAAAAAAAAAA="}}, // Thus this makes using the provided data via the API useless for a hash comparison test // Test the existing database item hash against the hash on the local disk - as this is what we know was in-sync with online prior to online deletion event if (!testFileHash(localPathToDelete, existingDatabaseItem)) { // Current file on disk is different by hash / content // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(localPathToDelete, dryRun, bypassDataPreservation, renamedPath); // Purge the old record from the database as this still exists. The safeBackup() generated file now will be 'new' on the local filesystem itemDB.deleteById(existingDatabaseItem.driveId, existingDatabaseItem.id); } else { // Hash is the same, we can assume the isItemSynced() returning false was due to some sort of timestamp issue // Flag to delete rather than create a backup of the local file if (debugLogging) {addLogEntry("Flagging to delete item locally due to online deletion event: " ~ to!string(onedriveJSONItem), ["debug"]);} // Use the DB entries returned - add the driveId, itemId and parentId values to the array idsToDelete ~= [existingDatabaseItem.driveId, existingDatabaseItem.id, existingDatabaseItem.parentId]; } } } else { // Flag to ignore if (debugLogging) {addLogEntry("Flagging item to skip: " ~ to!string(onedriveJSONItem), ["debug"]);} skippedItems.insert(thisItemId); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process each of the elements contained in jsonItemsToProcess[] void processJSONItemsInBatch(JSONValue[] array, long batchGroup, long batchCount) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } long batchElementCount = array.length; MonoTime jsonProcessingStartTime; foreach (i, onedriveJSONItem; array.enumerate) { // Use the JSON elements rather can computing a DB struct via makeItem() long elementCount = i +1; jsonProcessingStartTime = MonoTime.currTime(); // To show this is the processing for this particular item, start off with this breaker line if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); addLogEntry("Processing OneDrive JSON item " ~ to!string(elementCount) ~ " of " ~ to!string(batchElementCount) ~ " as part of JSON Item Batch " ~ to!string(batchGroup) ~ " of " ~ to!string(batchCount), ["debug"]); addLogEntry("Raw JSON OneDrive Item (Batched Item): " ~ to!string(onedriveJSONItem), ["debug"]); } // Configure required items from the JSON elements string thisItemId = onedriveJSONItem["id"].str; string thisItemDriveId = onedriveJSONItem["parentReference"]["driveId"].str; string thisItemParentId = onedriveJSONItem["parentReference"]["id"].str; string thisItemName = onedriveJSONItem["name"].str; // Create an empty item struct for an existing DB item Item existingDatabaseItem; // Do we NOT want this item? bool unwanted = false; // meaning by default we will WANT this item // Is this parent is in the database bool parentInDatabase = false; // Is this the 'root' folder of a Shared Folder bool rootSharedFolder = false; // What is the full path of the new item string computedItemPath; string newItemPath; // Configure the remoteItem - so if it is used, it can be utilised later Item remoteItem; // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { thisItemDriveId = transformToLowerCase(thisItemDriveId); } // Check the database for an existing entry for this JSON item bool existingDBEntry = itemDB.selectById(thisItemDriveId, thisItemId, existingDatabaseItem); // Calculate if the Parent Item is in the database so that it can be re-used parentInDatabase = itemDB.idInLocalDatabase(thisItemDriveId, thisItemParentId); // Calculate the local path of this JSON item, but we can only do this if the parent is in the database if (parentInDatabase) { // Compute the full local path for an item based on its position within the OneDrive hierarchy // This also accounts for Shared Folders in our account root, plus Shared Folders in a folder (relocated shared folders) computedItemPath = computeItemPath(thisItemDriveId, thisItemParentId); // Is 'thisItemParentId' in the DB as a 'root' object? Item databaseItem; // Is this a remote drive? if (thisItemDriveId != appConfig.defaultDriveId) { // query the database for the actual thisItemParentId record itemDB.selectById(thisItemDriveId, thisItemParentId, databaseItem); } // Calculate newItemPath to // This needs to factor in: // - Shared Folders = ItemType.root with a name of 'root' // - SharePoint Document Root = ItemType.root with a name of the actual shared folder // - Relocatable Shared Folders where a user moves a Shared Folder Link to a sub folder elsewhere within their directory structure online if (databaseItem.type == ItemType.root) { // 'root' database object if (databaseItem.name == "root") { // OneDrive Business Shared Folder 'root' shortcut link // If the record type is now a root record, we dont want to add the name to itself newItemPath = computedItemPath; } else { // OneDrive Business SharePoint Document 'root' shortcut link if (databaseItem.name == thisItemName) { // If the record type is now a root record, we dont want to add the name to itself newItemPath = computedItemPath; } else { // add the item name to the computed path newItemPath = computedItemPath ~ "/" ~ thisItemName; } } // Set this for later use rootSharedFolder = true; } else { // Add the item name to the computed path newItemPath = computedItemPath ~ "/" ~ thisItemName; } // debug logging of what was calculated if (debugLogging) {addLogEntry("JSON Item calculated full path is: " ~ newItemPath, ["debug"]);} } else { // Parent not in the database // Is the parent a 'folder' from another user? ie - is this a 'shared folder' that has been shared with us? // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { thisItemDriveId = transformToLowerCase(thisItemDriveId); } // Lets determine why? if (thisItemDriveId == appConfig.defaultDriveId) { // Parent path does not exist - flagging as unwanted if (debugLogging) {addLogEntry("Flagging as unwanted: thisItemDriveId (" ~ thisItemDriveId ~ "), thisItemParentId (" ~ thisItemParentId ~ ") not in local database", ["debug"]);} // Was this a skipped item? if (thisItemParentId in skippedItems) { // Parent is a skipped item if (debugLogging) {addLogEntry("Reason: thisItemParentId listed within skippedItems", ["debug"]);} } else { // Parent is not in the database, as we are not creating it if (debugLogging) {addLogEntry("Reason: Parent ID is not in the DB .. ", ["debug"]);} } // Flag as unwanted unwanted = true; } else { // Format the OneDrive change into a consumable object for the database remoteItem = makeItem(onedriveJSONItem); // Edge case as the parent (from another users OneDrive account) will never be in the database - potentially a shared object? if (debugLogging) { addLogEntry("The reported parentId is not in the database. This potentially is a shared folder as 'remoteItem.driveId' != 'appConfig.defaultDriveId'. Relevant Details: remoteItem.driveId (" ~ remoteItem.driveId ~ "), remoteItem.parentId (" ~ remoteItem.parentId ~ ")", ["debug"]); addLogEntry("Potential Shared Object JSON: " ~ sanitiseJSONItem(onedriveJSONItem), ["debug"]); } // What account type is this? if (appConfig.accountType == "personal") { // Personal Account Handling if (debugLogging) {addLogEntry("Handling a Personal Shared Item JSON object", ["debug"]);} // Does the JSON have a shared element structure if (hasSharedElement(onedriveJSONItem)) { // Has the Shared JSON structure if (debugLogging) {addLogEntry("Personal Shared Item JSON object has the 'shared' JSON structure", ["debug"]);} // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(onedriveJSONItem); } else { // The Shared JSON structure is missing ..... if (debugLogging) {addLogEntry("Personal Shared Item JSON object is MISSING the 'shared' JSON structure ... API BUG ?", ["debug"]);} } // Ensure that this item has no parent if (debugLogging) {addLogEntry("Setting remoteItem.parentId of Personal Shared Item JSON object to be null", ["debug"]);} remoteItem.parentId = null; // Add this record to the local database if (debugLogging) {addLogEntry("Update/Insert local database with Personal Shared Item JSON object with remoteItem.parentId as null: " ~ to!string(remoteItem), ["debug"]);} itemDB.upsert(remoteItem); // Due to OneDrive API inconsistency with Personal Accounts, again with European Data Centres, as we have handled this JSON - flag as unwanted as processing is complete for this JSON item unwanted = true; } else { // Business or SharePoint Account Handling if (debugLogging) {addLogEntry("Handling a Business or SharePoint Shared Item JSON object", ["debug"]);} if (appConfig.accountType == "business") { // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(onedriveJSONItem); // Ensure that this item has no parent if (debugLogging) {addLogEntry("Setting remoteItem.parentId to be null", ["debug"]);} remoteItem.parentId = null; // Check the DB for 'remote' objects, searching 'remoteDriveId' and 'remoteId' items for this remoteItem.driveId and remoteItem.id Item remoteDBItem; itemDB.selectByRemoteId(remoteItem.driveId, remoteItem.id, remoteDBItem); // Must compare remoteDBItem.name with remoteItem.name if ((!remoteDBItem.name.empty) && (remoteDBItem.name != remoteItem.name)) { // Update DB Item if (debugLogging) { addLogEntry("The shared item stored in OneDrive, has a different name to the actual name on the remote drive", ["debug"]); addLogEntry("Updating remoteItem.name JSON data with the actual name being used on account drive and local folder", ["debug"]); addLogEntry("remoteItem.name was: " ~ remoteItem.name, ["debug"]); addLogEntry("Updating remoteItem.name to: " ~ remoteDBItem.name, ["debug"]); } remoteItem.name = remoteDBItem.name; if (debugLogging) {addLogEntry("Setting remoteItem.remoteName to: " ~ onedriveJSONItem["name"].str, ["debug"]);} // Update JSON Item remoteItem.remoteName = onedriveJSONItem["name"].str; if (debugLogging) { addLogEntry("Updating source JSON 'name' to that which is the actual local directory", ["debug"]); addLogEntry("onedriveJSONItem['name'] was: " ~ onedriveJSONItem["name"].str, ["debug"]); addLogEntry("Updating onedriveJSONItem['name'] to: " ~ remoteDBItem.name, ["debug"]); } onedriveJSONItem["name"] = remoteDBItem.name; if (debugLogging) {addLogEntry("onedriveJSONItem['name'] now: " ~ onedriveJSONItem["name"].str, ["debug"]);} // Update newItemPath value newItemPath = computeItemPath(thisItemDriveId, thisItemParentId) ~ "/" ~ remoteDBItem.name; if (debugLogging) {addLogEntry("New Item updated calculated full path is: " ~ newItemPath, ["debug"]);} } // Add this record to the local database if (debugLogging) {addLogEntry("Update/Insert local database with remoteItem details: " ~ to!string(remoteItem), ["debug"]);} itemDB.upsert(remoteItem); } else { // Sharepoint account type addLogEntry("Handling a SharePoint Shared Item JSON object - NOT IMPLEMENTED YET ........ RAISE A BUG PLEASE", ["info"]); } } } } // Check the skippedItems array for the parent id of this JSONItem if this is something we need to skip if (!unwanted) { if (thisItemParentId in skippedItems) { // Flag this JSON item as unwanted if (debugLogging) {addLogEntry("Flagging as unwanted: find(thisItemParentId).length != 0", ["debug"]);} unwanted = true; // Is this item id in the database? if (existingDBEntry) { // item exists in database, most likely moved out of scope for current client configuration if (debugLogging) {addLogEntry("This item was previously synced / seen by the client", ["debug"]);} if (("name" in onedriveJSONItem["parentReference"]) != null) { // How is this item now out of scope? // is sync_list configured if (syncListConfigured) { // sync_list configured and in use if (selectiveSync.isPathExcludedViaSyncList(onedriveJSONItem["parentReference"]["name"].str)) { // Previously synced item is now out of scope as it has been moved out of what is included in sync_list if (debugLogging) {addLogEntry("This previously synced item is now excluded from being synced due to sync_list exclusion", ["debug"]);} } } // flag to delete local file as it now is no longer in sync with OneDrive if (verboseLogging) {addLogEntry("Flagging to delete item locally as this is now an unwanted item (parental exclusion) and the item currently exists in the local database: ", ["verbose"]);} // Use the configured values - add the driveId, itemId and parentId values to the array idsToDelete ~= [thisItemDriveId, thisItemId, thisItemParentId]; } } } } // Check the item type - if it not an item type that we support, we cant process the JSON item if (!unwanted) { if (isItemFile(onedriveJSONItem)) { if (debugLogging) {addLogEntry("The JSON item we are processing is a file", ["debug"]);} } else if (isItemFolder(onedriveJSONItem)) { if (debugLogging) {addLogEntry("The JSON item we are processing is a folder", ["debug"]);} } else if (isItemRemote(onedriveJSONItem)) { if (debugLogging) {addLogEntry("The JSON item we are processing is a remote item", ["debug"]);} } else { // Why was this unwanted? if (newItemPath.empty) { if (debugLogging) {addLogEntry("OOPS: newItemPath is empty ....... need to calculate it", ["debug"]);} // Compute this item path & need the full path for this file newItemPath = computeItemPath(thisItemDriveId, thisItemParentId) ~ "/" ~ thisItemName; if (debugLogging) {addLogEntry("New Item calculated full path is: " ~ newItemPath, ["debug"]);} } // Microsoft OneNote container objects present as neither folder or file but has file size if ((!isItemFile(onedriveJSONItem)) && (!isItemFolder(onedriveJSONItem)) && (hasFileSize(onedriveJSONItem))) { // Log that this was skipped as this was a Microsoft OneNote item and unsupported if (verboseLogging) {addLogEntry("The Microsoft OneNote Notebook '" ~ newItemPath ~ "' is not supported by this client", ["verbose"]);} } else { // Log that this item was skipped as unsupported if (verboseLogging) {addLogEntry("The OneDrive item '" ~ newItemPath ~ "' is not supported by this client", ["verbose"]);} } unwanted = true; if (debugLogging) {addLogEntry("Flagging as unwanted: item type is not supported", ["debug"]);} } } // Check if this is excluded by config option: skip_dir if (!unwanted) { // Only check path if config is != "" if (!appConfig.getValueString("skip_dir").empty) { // Is the item a folder or a remote item? (which itself is a directory, but is missing the 'folder' JSON element we use to determine JSON being a directory or not) if ((isItemFolder(onedriveJSONItem)) || (isRemoteFolderItem(onedriveJSONItem))) { // work out the 'snippet' path where this folder would be created string simplePathToCheck = ""; string complexPathToCheck = ""; string matchDisplay = ""; if (hasParentReference(onedriveJSONItem)) { // we need to workout the FULL path for this item // simple path calculation if (("name" in onedriveJSONItem["parentReference"]) != null) { // how do we build the simplePathToCheck path up ? // did we flag this as the root shared folder object earlier? if (rootSharedFolder) { // just use item name simplePathToCheck = onedriveJSONItem["name"].str; } else { // add parent name to item name simplePathToCheck = onedriveJSONItem["parentReference"]["name"].str ~ "/" ~ onedriveJSONItem["name"].str; } } else { // just use item name simplePathToCheck = onedriveJSONItem["name"].str; } if (debugLogging) {addLogEntry("skip_dir path to check (simple): " ~ simplePathToCheck, ["debug"]);} // complex path calculation if (parentInDatabase) { // build up complexPathToCheck complexPathToCheck = buildNormalizedPath(newItemPath); } else { if (debugLogging) {addLogEntry("Parent details not in database - unable to compute complex path to check", ["debug"]);} } if (!complexPathToCheck.empty) { if (debugLogging) {addLogEntry("skip_dir path to check (complex): " ~ complexPathToCheck, ["debug"]);} } } else { simplePathToCheck = onedriveJSONItem["name"].str; } // If 'simplePathToCheck' or 'complexPathToCheck' is of the following format: root:/folder // then isDirNameExcluded matching will not work if (simplePathToCheck.canFind(":")) { if (debugLogging) {addLogEntry("Updating simplePathToCheck to remove 'root:'", ["debug"]);} simplePathToCheck = processPathToRemoveRootReference(simplePathToCheck); } if (complexPathToCheck.canFind(":")) { if (debugLogging) {addLogEntry("Updating complexPathToCheck to remove 'root:'", ["debug"]);} complexPathToCheck = processPathToRemoveRootReference(complexPathToCheck); } // OK .. what checks are we doing? if ((!simplePathToCheck.empty) && (complexPathToCheck.empty)) { // just a simple check if (debugLogging) {addLogEntry("Performing a simple check only", ["debug"]);} unwanted = selectiveSync.isDirNameExcluded(simplePathToCheck); } else { // simple and complex if (debugLogging) {addLogEntry("Performing a simple then complex path match if required", ["debug"]);} // simple first if (debugLogging) {addLogEntry("Performing a simple check first", ["debug"]);} unwanted = selectiveSync.isDirNameExcluded(simplePathToCheck); matchDisplay = simplePathToCheck; if (!unwanted) { // simple didnt match, perform a complex check if (debugLogging) {addLogEntry("Simple match was false, attempting complex match", ["debug"]);} unwanted = selectiveSync.isDirNameExcluded(complexPathToCheck); matchDisplay = complexPathToCheck; } } // result if (debugLogging) {addLogEntry("skip_dir exclude result (directory based): " ~ to!string(unwanted), ["debug"]);} if (unwanted) { // This path should be skipped if (verboseLogging) {addLogEntry("Skipping path - excluded by skip_dir config: " ~ matchDisplay, ["verbose"]);} } } // Is the item a file? // We need to check to see if this files path is excluded as well if (isItemFile(onedriveJSONItem)) { string pathToCheck; // does the newItemPath start with '/'? if (!startsWith(newItemPath, "/")){ // path does not start with '/', but we need to check skip_dir entries with and without '/' // so always make sure we are checking a path with '/' pathToCheck = '/' ~ dirName(newItemPath); } else { pathToCheck = dirName(newItemPath); } // perform the check unwanted = selectiveSync.isDirNameExcluded(pathToCheck); // result if (debugLogging) {addLogEntry("skip_dir exclude result (file based): " ~ to!string(unwanted), ["debug"]);} if (unwanted) { // this files path should be skipped if (verboseLogging) {addLogEntry("Skipping file - file path is excluded by skip_dir config: " ~ newItemPath, ["verbose"]);} } } } } // Check if this is excluded by config option: skip_file if (!unwanted) { // Is the JSON item a file? if (isItemFile(onedriveJSONItem)) { // skip_file can contain 4 types of entries: // - wildcard - *.txt // - text + wildcard - name*.txt // - full path + combination of any above two - /path/name*.txt // - full path to file - /path/to/file.txt // is the parent id in the database? if (parentInDatabase) { // Compute this item path & need the full path for this file if (newItemPath.empty) { if (debugLogging) {addLogEntry("OOPS: newItemPath is empty ....... need to calculate it", ["debug"]);} newItemPath = computeItemPath(thisItemDriveId, thisItemParentId) ~ "/" ~ thisItemName; if (debugLogging) {addLogEntry("New Item calculated full path is: " ~ newItemPath, ["debug"]);} } // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched // However, as 'path' used throughout, use a temp variable with this modification so that we use the temp variable for exclusion checks string exclusionTestPath = ""; if (!startsWith(newItemPath, "/")){ // Add '/' to the path exclusionTestPath = '/' ~ newItemPath; } if (debugLogging) {addLogEntry("skip_file item to check: " ~ exclusionTestPath, ["debug"]);} unwanted = selectiveSync.isFileNameExcluded(exclusionTestPath); if (debugLogging) {addLogEntry("Result: " ~ to!string(unwanted), ["debug"]);} if (unwanted) { if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_file config: " ~ thisItemName, ["verbose"]);} } } else { // parent id is not in the database unwanted = true; if (verboseLogging) {addLogEntry("Skipping file - parent path not present in local database", ["verbose"]);} } } } // Check if this is included or excluded by use of sync_list if (!unwanted) { // No need to try and process something against a sync_list if it has been configured if (syncListConfigured) { // Compute the item path if empty - as to check sync_list we need an actual path to check if (newItemPath.empty) { // Calculate this items path if (debugLogging) {addLogEntry("OOPS: newItemPath is empty ....... need to calculate it", ["debug"]);} newItemPath = computeItemPath(thisItemDriveId, thisItemParentId) ~ "/" ~ thisItemName; if (debugLogging) {addLogEntry("New Item calculated full path is: " ~ newItemPath, ["debug"]);} } // What path are we checking? if (debugLogging) {addLogEntry("Path to check against 'sync_list' entries: " ~ newItemPath, ["debug"]);} // Unfortunately there is no avoiding this call to check if the path is excluded|included via sync_list if (selectiveSync.isPathExcludedViaSyncList(newItemPath)) { // selective sync advised to skip, however is this a file and are we configured to upload / download files in the root? if ((isItemFile(onedriveJSONItem)) && (appConfig.getValueBool("sync_root_files")) && (rootName(newItemPath) == "") ) { // This is a file // We are configured to sync all files in the root // This is a file in the logical configured root unwanted = false; // Log that we are retaining this file and why if (verboseLogging) { addLogEntry("Path retained due to 'sync_root_files' override for logical root file: " ~ newItemPath, ["verbose"]); } } else { // path is unwanted - excluded by 'sync_list' unwanted = true; if (verboseLogging) {addLogEntry("Skipping path - excluded by sync_list config: " ~ newItemPath, ["verbose"]);} // flagging to skip this item now, but does this exist in the DB thus needs to be removed / deleted? if (existingDBEntry) { // flag to delete if (verboseLogging) {addLogEntry("Flagging to delete item locally as this is now an unwanted item (sync_list exclusion) and the item currently exists in the local database: ", ["verbose"]);} // Use the configured values - add the driveId, itemId and parentId values to the array idsToDelete ~= [thisItemDriveId, thisItemId, thisItemParentId]; } } } } } // Check if the user has configured to skip downloading .files or .folders: skip_dotfiles if (!unwanted) { if (appConfig.getValueBool("skip_dotfiles")) { if (isDotFile(newItemPath)) { if (verboseLogging) {addLogEntry("Skipping item - .file or .folder: " ~ newItemPath, ["verbose"]);} unwanted = true; } } } // Check if this should be skipped due to a --check-for-nosync directive (.nosync)? if (!unwanted) { if (appConfig.getValueBool("check_nosync")) { // need the parent path for this object string parentPath = dirName(newItemPath); // Check for the presence of a .nosync in the parent path if (exists(parentPath ~ "/.nosync")) { if (verboseLogging) {addLogEntry("Skipping downloading item - .nosync found in parent folder & --check-for-nosync is enabled: " ~ newItemPath, ["verbose"]);} unwanted = true; } } } // Check if this is excluded by a user set maximum filesize to download if (!unwanted) { if (isItemFile(onedriveJSONItem)) { if (fileSizeLimit != 0) { if (onedriveJSONItem["size"].integer >= fileSizeLimit) { if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_size config: " ~ thisItemName ~ " (" ~ to!string(onedriveJSONItem["size"].integer/2^^20) ~ " MB)", ["verbose"]);} unwanted = true; } } } } // At this point all the applicable checks on this JSON object from OneDrive are complete: // - skip_file // - skip_dir // - sync_list // - skip_dotfiles // - check_nosync // - skip_size // - We know if this item exists in the DB or not in the DB // We know if this JSON item is unwanted or not if (unwanted) { // This JSON item is NOT wanted - it is excluded if (debugLogging) {addLogEntry("Skipping OneDrive JSON item as this is determined to be unwanted either through Client Side Filtering Rules or prior processing to this point", ["debug"]);} // Add to the skippedItems array, but only if it is a directory ... pointless adding 'files' here, as it is the 'id' we check as the parent path which can only be a directory if (!isItemFile(onedriveJSONItem)) { skippedItems.insert(thisItemId); } } else { // This JSON item is wanted - we need to process this JSON item further if (debugLogging) { addLogEntry("OneDrive JSON item passed all applicable Client Side Filtering Rules and has been determined this is a wanted item", ["debug"]); addLogEntry("Creating newDatabaseItem object using the provided JSON data", ["debug"]); } // Take the JSON item and create a consumable object for eventual database insertion Item newDatabaseItem = makeItem(onedriveJSONItem); if (existingDBEntry) { // The details of this JSON item are already in the DB // Is the item in the DB the same as the JSON data provided - or is the JSON data advising this is an updated file? if (debugLogging) {addLogEntry("OneDrive JSON item is an update to an existing local item", ["debug"]);} // Compute the existing item path // NOTE: // string existingItemPath = computeItemPath(existingDatabaseItem.driveId, existingDatabaseItem.id); // // This will calculate the path as follows: // // existingItemPath: Document.txt // // Whereas above we use the following // // newItemPath = computeItemPath(newDatabaseItem.driveId, newDatabaseItem.parentId) ~ "/" ~ newDatabaseItem.name; // // Which generates the following path: // // changedItemPath: ./Document.txt // // Need to be consistent here with how 'newItemPath' was calculated string queryDriveID; string queryParentID; // Must query with a valid driveid entry if (existingDatabaseItem.driveId.empty) { queryDriveID = thisItemDriveId; } else { queryDriveID = existingDatabaseItem.driveId; } // Must query with a valid parentid entry if (existingDatabaseItem.parentId.empty) { queryParentID = thisItemParentId; } else { queryParentID = existingDatabaseItem.parentId; } // Calculate the existing path string existingItemPath = computeItemPath(queryDriveID, queryParentID) ~ "/" ~ existingDatabaseItem.name; if (debugLogging) {addLogEntry("existingItemPath calculated full path is: " ~ existingItemPath, ["debug"]);} // Ensure that this path exists if this is an 'existing' database item if (existingDatabaseItem.type == ItemType.dir) { if (!exists(existingItemPath)) { handleLocalDirectoryCreation(existingDatabaseItem, existingItemPath, onedriveJSONItem); } } // Attempt to apply this changed item applyPotentiallyChangedItem(existingDatabaseItem, existingItemPath, newDatabaseItem, newItemPath, onedriveJSONItem); // Is this JSON object a 'remote' item? if(isItemRemote(onedriveJSONItem)) { // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(onedriveJSONItem); } } else { // Action this JSON item as a new item as we have no DB record of it // The actual item may actually exist locally already, meaning that just the database is out-of-date or missing the data due to --resync // But we also cannot compute the newItemPath as the parental objects may not exist as well if (debugLogging) {addLogEntry("OneDrive JSON item is potentially a new local item", ["debug"]);} // Attempt to apply this potentially new item applyPotentiallyNewLocalItem(newDatabaseItem, onedriveJSONItem, newItemPath); } } // How long to process this JSON item in batch if (debugLogging) { Duration jsonProcessingElapsedTime = MonoTime.currTime() - jsonProcessingStartTime; addLogEntry("Batched JSON item processing time: " ~ to!string(jsonProcessingElapsedTime), ["debug"]); } // Tracking as to if this item was processed processedCount++; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the download of any required objects in parallel void processDownloadActivities() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Are there any items to delete locally? Cleanup space locally first if (!idsToDelete.empty) { // There are elements that potentially need to be deleted locally if (verboseLogging) {addLogEntry("Items to potentially delete locally: " ~ to!string(idsToDelete.length), ["verbose"]);} if (appConfig.getValueBool("download_only")) { // Download only has been configured if (cleanupLocalFiles) { // Process online deleted items if (verboseLogging) {addLogEntry("Processing local deletion activity as --download-only & --cleanup-local-files configured", ["verbose"]);} processDeleteItems(); } else { // Not cleaning up local files if (verboseLogging) {addLogEntry("Skipping local deletion activity as --download-only has been used", ["verbose"]);} // List files and directories we are not deleting locally listDeletedItems(); } } else { // Not using --download-only process normally processDeleteItems(); } // Cleanup array memory idsToDelete = []; } // Are there any items to download post fetching and processing the /delta data? if (!fileJSONItemsToDownload.empty) { // There are elements to download addLogEntry("Number of items to download from Microsoft OneDrive: " ~ to!string(fileJSONItemsToDownload.length)); downloadOneDriveItems(); // Cleanup array memory fileJSONItemsToDownload = []; } // Are there any skipped items still? if (!skippedItems.empty) { // Cleanup array memory skippedItems.clear(); } // If deltaLinkCache.latestDeltaLink is not empty, update the deltaLink in the database for this driveId so that we can reuse this now that jsonItemsToProcess has been fully processed if (!deltaLinkCache.latestDeltaLink.empty) { if (debugLogging) {addLogEntry("Updating completed deltaLink for driveID " ~ deltaLinkCache.driveId ~ " in DB to: " ~ deltaLinkCache.latestDeltaLink, ["debug"]);} itemDB.setDeltaLink(deltaLinkCache.driveId, deltaLinkCache.itemId, deltaLinkCache.latestDeltaLink); // Now that the DB is updated, when we perform the last examination of the most recent online data, cache this so this can be obtained this from memory cacheLatestDeltaLink(deltaLinkInfo, deltaLinkCache.driveId, deltaLinkCache.latestDeltaLink); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Function to add or update a key pair in the deltaLinkInfo array void cacheLatestDeltaLink(ref DeltaLinkInfo deltaLinkInfo, string driveId, string latestDeltaLink) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } if (driveId !in deltaLinkInfo) { if (debugLogging) {addLogEntry("Added new latestDeltaLink entry: " ~ driveId ~ " -> " ~ latestDeltaLink, ["debug"]);} } else { if (debugLogging) {addLogEntry("Updated latestDeltaLink entry for " ~ driveId ~ " from " ~ deltaLinkInfo[driveId] ~ " to " ~ latestDeltaLink, ["debug"]);} } deltaLinkInfo[driveId] = latestDeltaLink; // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Function to get the latestDeltaLink based on driveId string getDeltaLinkFromCache(ref DeltaLinkInfo deltaLinkInfo, string driveId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } string cachedDeltaLink; if (driveId in deltaLinkInfo) { cachedDeltaLink = deltaLinkInfo[driveId]; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return value return cachedDeltaLink; } // If the JSON item is not in the database, it is potentially a new item that we need to action void applyPotentiallyNewLocalItem(Item newDatabaseItem, JSONValue onedriveJSONItem, string newItemPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the 'return' code as-is, so that this function operates as efficiently as possible. // Whilst this means some extra code / duplication in this function, it cannot be helped // The JSON and Database items being passed in here have passed the following checks: // - skip_file // - skip_dir // - sync_list // - skip_dotfiles // - check_nosync // - skip_size // - Is not currently cached in the local database // As such, we should not be doing any other checks here to determine if the JSON item is wanted .. it is if (exists(newItemPath)) { if (debugLogging) {addLogEntry("Path on local disk already exists", ["debug"]);} // Issue #2209 fix - test if path is a bad symbolic link if (isSymlink(newItemPath)) { if (debugLogging) {addLogEntry("Path on local disk is a symbolic link ........", ["debug"]);} if (!exists(readLink(newItemPath))) { // reading the symbolic link failed if (debugLogging) {addLogEntry("Reading the symbolic link target failed ........ ", ["debug"]);} addLogEntry("Skipping item - invalid symbolic link: " ~ newItemPath, ["info", "notify"]); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return - invalid symbolic link return; } } // Path exists locally, is not a bad symbolic link // Test if this item is actually in-sync // What is the source of this item data? string itemSource = "remote"; if (isItemSynced(newDatabaseItem, newItemPath, itemSource)) { // Issue #3115 - Personal Account Shared Folder // What account type is this? if (appConfig.accountType == "personal") { // Is this a 'remote' DB record if (newDatabaseItem.type == ItemType.remote) { // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the 'actual' OneDrive Personal driveId value and is 15 character checked string actualOnlineDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(newDatabaseItem.remoteDriveId)); newDatabaseItem.remoteDriveId = actualOnlineDriveId; } } // Item details from OneDrive and local item details in database are in-sync if (debugLogging) { addLogEntry("The item to sync is already present on the local filesystem and is in-sync with what is reported online", ["debug"]); addLogEntry("Update/Insert local database with item details: " ~ to!string(newDatabaseItem), ["debug"]); } // Add item to database itemDB.upsert(newDatabaseItem); // With the 'newDatabaseItem' saved to the database, regardless of --dry-run situation - was that new database item a 'remote' item? // If this is this a 'Shared Folder' item - ensure we have created / updated any relevant Database Tie Records // This should be applicable for all account types if (newDatabaseItem.type == ItemType.remote) { // yes this is a remote item type if (debugLogging) {addLogEntry("The 'newDatabaseItem' (applyPotentiallyNewLocalItem) is a remote item type - we need to create all of the associated database tie records for this database entry" , ["debug"]);} string relocatedFolderDriveId; string relocatedFolderParentId; // Is this a relocated Shared Folder? OneDrive Personal and Business supports the relocation of Shared Folder links to other folders // Is this parentId equal to our defaultRootId .. if not it is highly likely that this Shared Folder is in a sub folder in our online folder structure if (newDatabaseItem.parentId != appConfig.defaultRootId) { // The parentId is not our defaultRootId .. most likely a relocated shared folder if (debugLogging) { addLogEntry("The folder path for this Shared Folder is not our account root, thus is a relocated Shared Folder item. We must pass in the correct parent details for this Shared Folder 'root' object" , ["debug"]); // What are we setting addLogEntry("Setting relocatedFolderDriveId to: " ~ newDatabaseItem.driveId); addLogEntry("Setting relocatedFolderParentId to: " ~ newDatabaseItem.parentId); } // Configure the relocated folders data relocatedFolderDriveId = newDatabaseItem.driveId; relocatedFolderParentId = newDatabaseItem.parentId; } // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner // We pass in the JSON element so we can create the right records + if this is a relocated shared folder, give the local parental record identifier createRequiredSharedFolderDatabaseRecords(onedriveJSONItem, relocatedFolderDriveId, relocatedFolderParentId); } // Did the user configure to save xattr data about this file? if (appConfig.getValueBool("write_xattr_data")) { writeXattrData(newItemPath, onedriveJSONItem); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // all done processing this potential new local item return; } else { // Item details from OneDrive and local item details in database are NOT in-sync if (debugLogging) {addLogEntry("The item to sync exists locally but is potentially not in the local database - otherwise this would be handled as changed item", ["debug"]);} // Which object is newer? The local file or the remote file? SysTime localModifiedTime = timeLastModified(newItemPath).toUTC(); SysTime itemModifiedTime = newDatabaseItem.mtime; // Reduce time resolution to seconds before comparing localModifiedTime.fracSecs = Duration.zero; itemModifiedTime.fracSecs = Duration.zero; // Is the local modified time greater than that from OneDrive? if (localModifiedTime > itemModifiedTime) { // Local file is newer than item on OneDrive based on file modified time // Is this item id in the database? if (itemDB.idInLocalDatabase(newDatabaseItem.driveId, newDatabaseItem.id)) { // item id is in the database // no local rename // no download needed // Fetch the latest DB record - as this could have been updated by the isItemSynced if the date online was being corrected, then the DB updated as a result Item latestDatabaseItem; itemDB.selectById(newDatabaseItem.driveId, newDatabaseItem.id, latestDatabaseItem); if (debugLogging) {addLogEntry("latestDatabaseItem: " ~ to!string(latestDatabaseItem), ["debug"]);} SysTime latestItemModifiedTime = latestDatabaseItem.mtime; // Reduce time resolution to seconds before comparing latestItemModifiedTime.fracSecs = Duration.zero; if (localModifiedTime == latestItemModifiedTime) { // Log action if (verboseLogging) {addLogEntry("Local file modified time matches existing database record - keeping local file", ["verbose"]);} if (debugLogging) {addLogEntry("Skipping OneDrive change as this is determined to be unwanted due to local file modified time matching database data", ["debug"]);} } else { // Log action if (verboseLogging) {addLogEntry("Local file modified time is newer based on UTC time conversion - keeping local file as this exists in the local database", ["verbose"]);} if (debugLogging) {addLogEntry("Skipping OneDrive change as this is determined to be unwanted due to local file modified time being newer than OneDrive file and present in the sqlite database", ["debug"]);} } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return as no further action needed return; } else { // item id is not in the database .. maybe a --resync ? // file exists locally but is not in the sqlite database - maybe a failed download? if (verboseLogging) {addLogEntry("Local item does not exist in local database - replacing with file from OneDrive - failed download?", ["verbose"]);} // In a --resync scenario or if items.sqlite3 was deleted before startup we have zero way of knowing IF the local file is meant to be the right file // To this pint we have passed the following checks: // 1. Any client side filtering checks - this determined this is a file that is wanted // 2. A file with the exact name exists locally // 3. The local modified time > remote modified time // 4. The id of the item from OneDrive is not in the database // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(newItemPath, dryRun, bypassDataPreservation, renamedPath); } } else { // Is the remote newer? if (localModifiedTime < itemModifiedTime) { // Remote file is newer than the existing local item if (verboseLogging) {addLogEntry("Remote item modified time is newer based on UTC time conversion", ["verbose"]);} // correct message, remote item is newer if (debugLogging) { addLogEntry("localModifiedTime (local file): " ~ to!string(localModifiedTime), ["debug"]); addLogEntry("itemModifiedTime (OneDrive item): " ~ to!string(itemModifiedTime), ["debug"]); } // Is this the exact same file? // Test the file hash if (!testFileHash(newItemPath, newDatabaseItem)) { // File on disk is different by hash / content // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(newItemPath, dryRun, bypassDataPreservation, renamedPath); } else { // File on disk is the same by hash / content, but is a different timestamp // The file contents have not changed, but the modified timestamp has if (verboseLogging) {addLogEntry("The last modified timestamp online has changed however the local file content has not changed", ["verbose"]);} // Update the local timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, newItemPath, newDatabaseItem.mtime); } } // Are the timestamps equal? if (localModifiedTime == itemModifiedTime) { // yes they are equal if (debugLogging) { addLogEntry("File timestamps are equal, no further action required", ["debug"]); // correct message as timestamps are equal addLogEntry("Update/Insert local database with item details: " ~ to!string(newDatabaseItem), ["debug"]); } // Add item to database itemDB.upsert(newDatabaseItem); // Did the user configure to save xattr data about this file? if (appConfig.getValueBool("write_xattr_data")) { writeXattrData(newItemPath, onedriveJSONItem); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // everything all OK, DB updated return; } } } } // Path does not exist locally (should not exist locally if renamed file) - this will be a new file download or new folder creation // How to handle this Potentially New Local Item JSON ? final switch (newDatabaseItem.type) { case ItemType.file: // Add to the file to the download array for processing later fileJSONItemsToDownload ~= onedriveJSONItem; goto functionCompletion; case ItemType.dir: // Create the directory immediately as we depend on its entry existing handleLocalDirectoryCreation(newDatabaseItem, newItemPath, onedriveJSONItem); goto functionCompletion; case ItemType.remote: // Add to the directory and relevant details for processing later if (newDatabaseItem.remoteType == ItemType.dir) { handleLocalDirectoryCreation(newDatabaseItem, newItemPath, onedriveJSONItem); } else { // Add to the file to the download array for processing later fileJSONItemsToDownload ~= onedriveJSONItem; } goto functionCompletion; case ItemType.root: case ItemType.unknown: case ItemType.none: // Unknown type - we dont action or sync these items goto functionCompletion; } // To correctly handle a switch|case statement we use goto post the switch|case statement as if 'break' is used, we never get to this point functionCompletion: // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Handle the creation of a new local directory void handleLocalDirectoryCreation(Item newDatabaseItem, string newItemPath, JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // To create a path, 'newItemPath' must not be empty if (!newItemPath.empty) { // Update the logging output to be consistent if (verboseLogging) {addLogEntry("Creating local directory: " ~ "./" ~ buildNormalizedPath(newItemPath), ["verbose"]);} if (!dryRun) { try { // Create the new directory if (debugLogging) {addLogEntry("Requested local path does not exist, creating directory structure: " ~ newItemPath, ["debug"]);} mkdirRecurse(newItemPath); // Has the user disabled the setting of filesystem permissions? if (!appConfig.getValueBool("disable_permission_set")) { // Configure the applicable permissions for the folder if (debugLogging) {addLogEntry("Setting directory permissions for: " ~ newItemPath, ["debug"]);} newItemPath.setAttributes(appConfig.returnRequiredDirectoryPermissions()); } else { // Use inherited permissions if (debugLogging) {addLogEntry("Using inherited filesystem permissions for: " ~ newItemPath, ["debug"]);} } // Update the time of the folder to match the last modified time as is provided by OneDrive // If there are any files then downloaded into this folder, the last modified time will get // updated by the local Operating System with the latest timestamp - as this is normal operation // as the directory has been modified // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, newItemPath, newDatabaseItem.mtime); // Save the newDatabaseItem to the database saveDatabaseItem(newDatabaseItem); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, newItemPath); } } else { // we dont create the directory, but we need to track that we 'faked it' idsFaked ~= [newDatabaseItem.driveId, newDatabaseItem.id]; // Save the newDatabaseItem to the database saveDatabaseItem(newDatabaseItem); } // With the 'newDatabaseItem' saved to the database, regardless of --dry-run situation - was that new database item a 'remote' item? // Is this folder that has been created locally a 'Shared Folder' online? // This should be applicable for all account types if (newDatabaseItem.type == ItemType.remote) { // yes this is a remote item type if (debugLogging) {addLogEntry("The 'newDatabaseItem' (handleLocalDirectoryCreation) is a remote item type - we need to create all of the associated database tie records for this database entry" , ["debug"]);} string relocatedFolderDriveId; string relocatedFolderParentId; // Is this a relocated Shared Folder? OneDrive Personal and Business supports the relocation of Shared Folder links to other folders // Is this parentId equal to our defaultRootId .. if not it is highly likely that this Shared Folder is in a sub folder in our online folder structure if (newDatabaseItem.parentId != appConfig.defaultRootId) { // The parentId is not our defaultRootId .. most likely a relocated shared folder if (debugLogging) { addLogEntry("The folder path for this Shared Folder is not our account root, thus is a relocated Shared Folder item. We must pass in the correct parent details for this Shared Folder 'root' object" , ["debug"]); // What are we setting addLogEntry("Setting relocatedFolderDriveId to: " ~ newDatabaseItem.driveId); addLogEntry("Setting relocatedFolderParentId to: " ~ newDatabaseItem.parentId); } // Configure the relocated folders data relocatedFolderDriveId = newDatabaseItem.driveId; relocatedFolderParentId = newDatabaseItem.parentId; } // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner // We pass in the JSON element so we can create the right records + if this is a relocated shared folder, give the local parental record identifier createRequiredSharedFolderDatabaseRecords(onedriveJSONItem, relocatedFolderDriveId, relocatedFolderParentId); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Create 'root' DB Tie Record and 'Shared Folder' DB Record in a consistent manner void createRequiredSharedFolderDatabaseRecords(JSONValue onedriveJSONItem, string relocatedFolderDriveId = null, string relocatedFolderParentId = null) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return code, so that this function operates as efficiently as possible. // Whilst this means some extra code / duplication in this function, it cannot be helped // Detail what we are doing if (debugLogging) {addLogEntry("We have been requested to create 'root' and 'Shared Folder' DB Tie Records in a consistent manner" , ["debug"]);} JSONValue onlineParentData; string parentDriveId; string parentObjectId; OneDriveApi onlineParentOneDriveApiInstance; onlineParentOneDriveApiInstance = new OneDriveApi(appConfig); onlineParentOneDriveApiInstance.initialise(); // Using the onlineParentData JSON data make a DB record for this parent item so that it exists in the database Item sharedFolderDatabaseTie; // A Shared Folder should have ["remoteItem"]["parentReference"] elements bool remoteItemElementsExist = false; // Test that the required elements exist for Shared Folder DB entry creations to occur if (isItemRemote(onedriveJSONItem)) { // Required ["remoteItem"] element exists in the JSON data if ((hasRemoteParentDriveId(onedriveJSONItem)) && (hasRemoteItemId(onedriveJSONItem))) { // Required elements exist remoteItemElementsExist = true; // What account type is this? This needs to be configured correctly so this can be queried correctly // - The setting of this is the 'same' for account types, but previously this was shown to need different data. Future code optimisation potentially here. if (appConfig.accountType == "personal") { // OneDrive Personal JSON has this structure that we need to use parentDriveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str; parentObjectId = onedriveJSONItem["remoteItem"]["id"].str; } else { // OneDrive Business|Sharepoint JSON has this structure that we need to use parentDriveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str; parentObjectId = onedriveJSONItem["remoteItem"]["id"].str; } } } // If the required elements do not exist, the Shared Folder DB elements cannot be created if (!remoteItemElementsExist) { // We cannot create the required entries in the database if (debugLogging) {addLogEntry("Unable to create 'root' and 'Shared Folder' DB Tie Records in a consistent manner - required elements missing from provided JSON record" , ["debug"]);} return; } // Issue #3115 - Validate 'parentDriveId' length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test parentDriveId = transformToLowerCase(parentDriveId); // Test if the 'parentDriveId' is not equal to appConfig.defaultDriveId if (parentDriveId != appConfig.defaultDriveId) { // Test 'parentDriveId' for length and validation - 15 character API bug parentDriveId = testProvidedDriveIdForLengthIssue(parentDriveId); } } // Try and fetch this shared folder parent's details try { if (debugLogging) {addLogEntry(format("Fetching Shared Folder online data for parentDriveId '%s' and parentObjectId '%s'", parentDriveId, parentObjectId), ["debug"]);} onlineParentData = onlineParentOneDriveApiInstance.getPathDetailsById(parentDriveId, parentObjectId); } catch (OneDriveException exception) { // If we get a 404 .. the shared item does not exist online ... perhaps a broken 'Add shortcut to My files' link in the account holders directory? if ((exception.httpStatusCode == 403) || (exception.httpStatusCode == 404)) { // The API call returned a 404 error response if (debugLogging) {addLogEntry("onlineParentData = onlineParentOneDriveApiInstance.getPathDetailsById(parentDriveId, parentObjectId); generated a 404 - shared folder path does not exist online", ["debug"]);} string errorMessage = format("WARNING: The OneDrive Shared Folder link target '%s' cannot be found online using the provided online data.", onedriveJSONItem["name"].str); // detail what this 404 error response means addLogEntry(); addLogEntry(errorMessage); addLogEntry("WARNING: This is potentially a broken online OneDrive Shared Folder link or you no longer have access to it. Please correct this error online."); addLogEntry(); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory onlineParentOneDriveApiInstance.releaseCurlEngine(); onlineParentOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // we have to return at this point return; } else { // Catch all other errors // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory onlineParentOneDriveApiInstance.releaseCurlEngine(); onlineParentOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // If we get an error, we cannot do much else return; } } // Create a 'root' DB Tie Record for a Shared Folder from the parent folder JSON data // - This maps the Shared Folder 'driveId' with the parent folder where the shared folder exists, so we can call the parent folder to query for changes to this Shared Folder createDatabaseRootTieRecordForOnlineSharedFolder(onlineParentData, relocatedFolderDriveId, relocatedFolderParentId); // Log that we are created the Shared Folder Tie record now if (debugLogging) {addLogEntry("Creating the Shared Folder DB Tie Record that binds the 'root' record to the 'shared folder'" , ["debug"]);} // Make an item from the online JSON data sharedFolderDatabaseTie = makeItem(onlineParentData); // Ensure we use our online name, as we may have renamed the folder in our location sharedFolderDatabaseTie.name = onedriveJSONItem["name"].str; // use this as the name .. this is the name of the folder online in our OneDrive account, not the online parent name // Is sharedFolderDatabaseTie.driveId empty? if (sharedFolderDatabaseTie.driveId.empty) { // This cannot be empty - set to the correct reference for the Shared Folder DB Tie record if (debugLogging) {addLogEntry("The Shared Folder DB Tie record entry for 'driveId' is empty ... correcting it" , ["debug"]);} sharedFolderDatabaseTie.driveId = onlineParentData["parentReference"]["driveId"].str; } // Ensure 'parentId' is not empty, except for Personal Accounts if (appConfig.accountType != "personal") { // Is sharedFolderDatabaseTie.parentId.empty? if (sharedFolderDatabaseTie.parentId.empty) { // This cannot be empty - set to the correct reference for the Shared Folder DB Tie record if (debugLogging) {addLogEntry("The Shared Folder DB Tie record entry for 'parentId' is empty ... correcting it" , ["debug"]);} sharedFolderDatabaseTie.parentId = onlineParentData["id"].str; } } else { // The database Tie Record for Personal Accounts must be empty .. no change, leave 'parentId' empty } // If a user has added the 'whole' SharePoint Document Library, then the DB Shared Folder Tie Record and 'root' record are the 'same' if ((isItemRoot(onlineParentData)) && (onlineParentData["parentReference"]["driveType"].str == "documentLibrary")) { // Yes this is a DocumentLibrary 'root' object if (debugLogging) { addLogEntry("Updating Shared Folder DB Tie record entry with correct values as this is a 'root' object as it is a SharePoint Library Root Object" , ["debug"]); addLogEntry(" sharedFolderDatabaseTie.parentId = null", ["debug"]); addLogEntry(" sharedFolderDatabaseTie.type = ItemType.root", ["debug"]); } sharedFolderDatabaseTie.parentId = null; sharedFolderDatabaseTie.type = ItemType.root; } // Personal Account Shared Folder Handling if (appConfig.accountType == "personal") { // Yes this is a personal account if (debugLogging) { addLogEntry("Updating Shared Folder DB Tie record entry with correct type value as this as it is a Personal Shared Folder Object" , ["debug"]); addLogEntry(" sharedFolderDatabaseTie.type = ItemType.dir", ["debug"]); } sharedFolderDatabaseTie.type = ItemType.dir; } // Issue #3115 - Validate sharedFolderDatabaseTie.driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test sharedFolderDatabaseTie.driveId = transformToLowerCase(sharedFolderDatabaseTie.driveId); // Test sharedFolderDatabaseTie.driveId length and validation if the sharedFolderDatabaseTie.driveId we are testing is not equal to appConfig.defaultDriveId if (sharedFolderDatabaseTie.driveId != appConfig.defaultDriveId) { sharedFolderDatabaseTie.driveId = testProvidedDriveIdForLengthIssue(sharedFolderDatabaseTie.driveId); } } // Log action addLogEntry("Creating|Updating a DB Tie Record for this Shared Folder from the online parental data: " ~ sharedFolderDatabaseTie.name, ["debug"]); addLogEntry("Shared Folder DB Tie Record data: " ~ to!string(sharedFolderDatabaseTie), ["debug"]); // Is this a dry-run excercise? if (dryRun) { // We need to ensure we add this to our faked entries idsFaked ~= [sharedFolderDatabaseTie.driveId, sharedFolderDatabaseTie.id]; } // Save item itemDB.upsert(sharedFolderDatabaseTie); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory onlineParentOneDriveApiInstance.releaseCurlEngine(); onlineParentOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // If the JSON item IS in the database, this will be an update to an existing in-sync item void applyPotentiallyChangedItem(Item existingDatabaseItem, string existingItemPath, Item changedOneDriveItem, string changedItemPath, JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // If we are moving the item, we do not need to download it again bool itemWasMoved = false; // Do we need to actually update the database with the details that were provided by the OneDrive API? // Calculate these time items from the provided items SysTime existingItemModifiedTime = existingDatabaseItem.mtime; existingItemModifiedTime.fracSecs = Duration.zero; SysTime changedOneDriveItemModifiedTime = changedOneDriveItem.mtime; changedOneDriveItemModifiedTime.fracSecs = Duration.zero; // Did the eTag change? if (existingDatabaseItem.eTag != changedOneDriveItem.eTag) { // The eTag has changed to what we previously cached if (existingItemPath != changedItemPath) { // Log that we are changing / moving an item to a new name addLogEntry("Moving " ~ existingItemPath ~ " to " ~ changedItemPath); // Is the destination path empty .. or does something exist at that location? if (exists(changedItemPath)) { // Destination we are moving to exists ... Item changedLocalItem; // Query DB for this changed item in specified path that exists and see if it is in-sync if (itemDB.selectByPath(changedItemPath, changedOneDriveItem.driveId, changedLocalItem)) { // The 'changedItemPath' is in the database string itemSource = "database"; if (isItemSynced(changedLocalItem, changedItemPath, itemSource)) { // The destination item is in-sync if (verboseLogging) {addLogEntry("Destination is in sync and will be overwritten", ["verbose"]);} } else { // The destination item is different if (verboseLogging) {addLogEntry("The destination is occupied with a different item, renaming the conflicting file...", ["verbose"]);} // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(changedItemPath, dryRun, bypassDataPreservation, renamedPath); } } else { // The to be overwritten item is not already in the itemdb, so it should saved to avoid data loss if (verboseLogging) {addLogEntry("The destination is occupied by an existing un-synced file, renaming the conflicting file...", ["verbose"]);} // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(changedItemPath, dryRun, bypassDataPreservation, renamedPath); } } // We should no longer need a try block for safeRename() as retry / error handling occurs within safeRename() and setLocalPathTimestamp() .. but keeping this for the moment try { // If we are in a --dry-run situation? if(!dryRun) { // We are not in a --dry-run situation // Attempt rename (returns true only if rename succeeded) bool renamedOk = safeRename(existingItemPath, changedItemPath, dryRun); // Was the rename successful? if (renamedOk) { // Flag that the item was moved | renamed itemWasMoved = true; // If the item is a file, make sure that the local timestamp now is the same as the timestamp online // Otherwise when we do the DB check, the move on the file system, the file technically has a newer timestamp // which is 'correct' .. but we need to report locally the online timestamp here as the move was made online if (changedOneDriveItem.type == ItemType.file) { // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, changedItemPath, changedOneDriveItem.mtime); } } else { // Rename failed - do NOT track as moved, do NOT touch timestamps on the target path addLogEntry("ERROR: Local rename failed; item will not be treated as moved: " ~ to!string(existingItemPath) ~ " -> " ~ to!string(changedItemPath), ["error", "notify"]); // We need to return here and stop processing this JSON item ... return; } } else { // --dry-run situation - the actual rename did not occur - but we need to track like it did // Track this as a faked id item idsFaked ~= [changedOneDriveItem.driveId, changedOneDriveItem.id]; // We also need to track that we did not rename this path // When we are checking entries in this array, paths need to have './' added pathsRenamed ~= [ensureStartsWithDotSlash(buildNormalizedPath(existingItemPath))]; } } catch (FileException e) { // Display the error message from the filesystem displayFileSystemErrorMessage(e.msg, thisFunctionName, existingItemPath); } } // What sort of changed item is this? // Is it a file or remote file, and we did not move it .. if (((changedOneDriveItem.type == ItemType.file) && (!itemWasMoved)) || (((changedOneDriveItem.type == ItemType.remote) && (changedOneDriveItem.remoteType == ItemType.file)) && (!itemWasMoved))) { // The eTag is notorious for being 'changed' online by some backend Microsoft process if (existingDatabaseItem.quickXorHash != changedOneDriveItem.quickXorHash) { // Add to the items to download array for processing - the file hash we previously recorded is not the same as online fileJSONItemsToDownload ~= onedriveJSONItem; } else { // If the timestamp is different, or we are running a client operational mode that does not support /delta queries - we have to update the DB with the details from OneDrive // Unfortunately because of the consequence of National Cloud Deployments not supporting /delta queries, the application uses the local database to flag what is out-of-date / track changes // This means that the constant disk writing to the database fix implemented with https://github.com/abraunegg/onedrive/pull/2004 cannot be utilised when using these operational modes // as all records are touched / updated when performing the OneDrive sync operations. The impacted operational modes are: // - National Cloud Deployments do not support /delta as a query // - When using --single-directory // - When using --download-only --cleanup-local-files // Is the last modified timestamp in the DB the same as the API data or are we running an operational mode where we simulated the /delta response? if ((existingItemModifiedTime != changedOneDriveItemModifiedTime) || (generateSimulatedDeltaResponse)) { // Save this item in the database // Issue #3115 - Personal Account Shared Folder // What account type is this? if (appConfig.accountType == "personal") { // Is this a 'remote' DB record if (changedOneDriveItem.type == ItemType.remote) { // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the actual OneDrive Personal driveId value and is 15 character checked string actualOnlineDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(changedOneDriveItem.remoteDriveId)); changedOneDriveItem.remoteDriveId = actualOnlineDriveId; } } // Add to the local database if (debugLogging) {addLogEntry("Adding changed OneDrive Item to database: " ~ to!string(changedOneDriveItem), ["debug"]);} itemDB.upsert(changedOneDriveItem); } } } else { // Save this item in the database saveItem(onedriveJSONItem); // If the 'Add shortcut to My files' link was the item that was actually renamed .. we have to update our DB records if (changedOneDriveItem.type == ItemType.remote) { // Select remote item data from the database Item existingRemoteDbItem; itemDB.selectById(changedOneDriveItem.remoteDriveId, changedOneDriveItem.remoteId, existingRemoteDbItem); // Update the 'name' in existingRemoteDbItem and save it back to the database // This is the local name stored on disk that was just 'moved' existingRemoteDbItem.name = changedOneDriveItem.name; itemDB.upsert(existingRemoteDbItem); } } } else { // The existingDatabaseItem.eTag == changedOneDriveItem.eTag .. nothing has changed eTag wise // If the timestamp is different, or we are running a client operational mode that does not support /delta queries - we have to update the DB with the details from OneDrive // Unfortunately because of the consequence of National Cloud Deployments not supporting /delta queries, the application uses the local database to flag what is out-of-date / track changes // This means that the constant disk writing to the database fix implemented with https://github.com/abraunegg/onedrive/pull/2004 cannot be utilised when using these operational modes // as all records are touched / updated when performing the OneDrive sync operations. The impacted operational modes are: // - National Cloud Deployments do not support /delta as a query // - When using --single-directory // - When using --download-only --cleanup-local-files // Is the last modified timestamp in the DB the same as the API data or are we running an operational mode where we simulated the /delta response? if ((existingItemModifiedTime != changedOneDriveItemModifiedTime) || (generateSimulatedDeltaResponse)) { // Database update needed for this item because our local record is out-of-date // Issue #3115 - Personal Account Shared Folder // What account type is this? if (appConfig.accountType == "personal") { // Is this a 'remote' DB record if (changedOneDriveItem.type == ItemType.remote) { // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the actual OneDrive Personal driveId value and is 15 character checked string actualOnlineDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(changedOneDriveItem.remoteDriveId)); changedOneDriveItem.remoteDriveId = actualOnlineDriveId; } } // Add to the local database if (debugLogging) {addLogEntry("Adding changed OneDrive Item to database: " ~ to!string(changedOneDriveItem), ["debug"]);} itemDB.upsert(changedOneDriveItem); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Download new/changed file items as identified void downloadOneDriveItems() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Lets deal with all the JSON items that need to be downloaded in a batch process size_t batchSize = to!int(appConfig.getValueLong("threads")); long batchCount = (fileJSONItemsToDownload.length + batchSize - 1) / batchSize; long batchesProcessed = 0; // Transfer order string transferOrder = appConfig.getValueString("transfer_order"); // Has the user configured to specify the transfer order of files? if (transferOrder != "default") { // If we have more than 1 item to download, sort the items if (count(fileJSONItemsToDownload) > 1) { // Perform sorting based on transferOrder if (transferOrder == "size_asc") { fileJSONItemsToDownload.sort!((a, b) => a["size"].integer < b["size"].integer); // sort the array by ascending size } else if (transferOrder == "size_dsc") { fileJSONItemsToDownload.sort!((a, b) => a["size"].integer > b["size"].integer); // sort the array by descending size } else if (transferOrder == "name_asc") { fileJSONItemsToDownload.sort!((a, b) => a["name"].str < b["name"].str); // sort the array by ascending name } else if (transferOrder == "name_dsc") { fileJSONItemsToDownload.sort!((a, b) => a["name"].str > b["name"].str); // sort the array by descending name } } } // Process fileJSONItemsToDownload foreach (chunk; fileJSONItemsToDownload.chunks(batchSize)) { // send an array containing 'appConfig.getValueLong("threads")' JSON items to download downloadOneDriveItemsInParallel(chunk); } // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Download items in parallel void downloadOneDriveItemsInParallel(JSONValue[] array) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function received an array of JSON items to download, the number of elements based on appConfig.getValueLong("threads") foreach (i, onedriveJSONItem; processPool.parallel(array)) { // Take each JSON item and download it downloadFileItem(onedriveJSONItem); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the actual download of an object from OneDrive void downloadFileItem(JSONValue onedriveJSONItem, bool ignoreDataPreservationCheck = false, long resumeOffset = 0) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables bool downloadFailed = false; string OneDriveFileXORHash; string OneDriveFileSHA256Hash; long jsonFileSize = 0; Item databaseItem; bool fileFoundInDB = false; // Create a JSONValue to store the online hash for resumable file checking JSONValue onlineHash; // Capture what time this download started SysTime downloadStartTime = Clock.currTime(); // Download item specifics string downloadItemId = onedriveJSONItem["id"].str; string downloadItemName = onedriveJSONItem["name"].str; string downloadDriveId = onedriveJSONItem["parentReference"]["driveId"].str; string downloadParentId = onedriveJSONItem["parentReference"]["id"].str; // Calculate this items path string newItemPath = computeItemPath(downloadDriveId, downloadParentId) ~ "/" ~ downloadItemName; if (debugLogging) {addLogEntry("JSON Item calculated full path for download is: " ~ newItemPath, ["debug"]);} // Is the item reported as Malware ? if (isMalware(onedriveJSONItem)){ // OneDrive reports that this file is malware addLogEntry("ERROR: MALWARE DETECTED IN FILE - DOWNLOAD SKIPPED: " ~ newItemPath, ["info", "notify"]); downloadFailed = true; } else { // Grab this file's filesize if (hasFileSize(onedriveJSONItem)) { // Use the configured filesize as reported by OneDrive jsonFileSize = onedriveJSONItem["size"].integer; } else { // filesize missing if (debugLogging) {addLogEntry("ERROR: onedriveJSONItem['size'] is missing", ["debug"]);} } // Configure the hashes for comparison post download if (hasHashes(onedriveJSONItem)) { // File details returned hash details // QuickXorHash if (hasQuickXorHash(onedriveJSONItem)) { // Use the provided quickXorHash as reported by OneDrive if (onedriveJSONItem["file"]["hashes"]["quickXorHash"].str != "") { OneDriveFileXORHash = onedriveJSONItem["file"]["hashes"]["quickXorHash"].str; } // Assign to JSONValue as object for resumable file checking onlineHash = JSONValue([ "quickXorHash": JSONValue(OneDriveFileXORHash) ]); } else { // Fallback: Check for SHA256Hash if (hasSHA256Hash(onedriveJSONItem)) { // Use the provided sha256Hash as reported by OneDrive if (onedriveJSONItem["file"]["hashes"]["sha256Hash"].str != "") { OneDriveFileSHA256Hash = onedriveJSONItem["file"]["hashes"]["sha256Hash"].str; } // Assign to JSONValue as object for resumable file checking onlineHash = JSONValue([ "sha256Hash": JSONValue(OneDriveFileSHA256Hash) ]); } } } else { // file hash data missing if (debugLogging) {addLogEntry("ERROR: onedriveJSONItem['file']['hashes'] is missing - unable to compare file hash after download to verify integrity of the downloaded file", ["debug"]);} // Assign to JSONValue as object for resumable file checking onlineHash = JSONValue([ "hashMissing": JSONValue("none") ]); } // Does the file already exist in the path locally? if (exists(newItemPath)) { // To accommodate forcing the download of a file, post upload to Microsoft OneDrive, we need to ignore the checking of hashes and making a safe backup if (!ignoreDataPreservationCheck) { // file exists locally already foreach (driveId; onlineDriveDetails.keys) { if (itemDB.selectByPath(newItemPath, driveId, databaseItem)) { fileFoundInDB = true; break; } } // Log the DB details if (debugLogging) {addLogEntry("File to download exists locally and this is the DB record: " ~ to!string(databaseItem), ["debug"]);} // Does the DB (what we think is in sync) hash match the existing local file hash? if (!testFileHash(newItemPath, databaseItem)) { // local file is different to what we know to be true addLogEntry("The local file to replace (" ~ newItemPath ~ ") has been modified locally since the last download. Renaming it to avoid potential local data loss."); // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not // In case the renamed path is needed string renamedPath; safeBackup(newItemPath, dryRun, bypassDataPreservation, renamedPath); } } } // Is there enough free space locally to download the file // - We can use '.' here as we change the current working directory to the configured 'sync_dir' long localActualFreeSpace = to!long(getAvailableDiskSpace(".")); // So that we are not responsible in making the disk 100% full if we can download the file, compare the current available space against the reservation set and file size // The reservation value is user configurable in the config file, 50MB by default long freeSpaceReservation = appConfig.getValueLong("space_reservation"); // debug output if (debugLogging) { addLogEntry("Local Disk Space Actual: " ~ to!string(localActualFreeSpace), ["debug"]); addLogEntry("Free Space Reservation: " ~ to!string(freeSpaceReservation), ["debug"]); addLogEntry("File Size to Download: " ~ to!string(jsonFileSize), ["debug"]); } // Calculate if we can actually download file - is there enough free space? if ((localActualFreeSpace < freeSpaceReservation) || (jsonFileSize > localActualFreeSpace)) { // localActualFreeSpace is less than freeSpaceReservation .. insufficient free space // jsonFileSize is greater than localActualFreeSpace .. insufficient free space addLogEntry("Downloading file: " ~ newItemPath ~ " ... failed!", ["info", "notify"]); addLogEntry("Insufficient local disk space to download file"); downloadFailed = true; } else { // If we are in a --dry-run situation - if not, actually perform the download if (!dryRun) { // Attempt to download the file as there is enough free space locally OneDriveApi downloadFileOneDriveApiInstance; try { // Initialise API instance downloadFileOneDriveApiInstance = new OneDriveApi(appConfig); downloadFileOneDriveApiInstance.initialise(); // OneDrive Business Shared Files - update the driveId where to get the file from if (isItemRemote(onedriveJSONItem)) { downloadDriveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str; } // Perform the download with any applicable set offset downloadFileOneDriveApiInstance.downloadById(downloadDriveId, downloadItemId, newItemPath, jsonFileSize, onlineHash, resumeOffset); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory downloadFileOneDriveApiInstance.releaseCurlEngine(); downloadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException exception) { if (debugLogging) {addLogEntry("downloadFileOneDriveApiInstance.downloadById(downloadDriveId, downloadItemId, newItemPath, jsonFileSize, onlineHash, resumeOffset); generated a OneDriveException", ["debug"]);} // HTTP request returned status code 403 if ((exception.httpStatusCode == 403) && (appConfig.getValueBool("sync_business_shared_files"))) { // We attempted to download a file, that was shared with us, but this was shared with us as read-only and no download permission addLogEntry("Unable to download this file as this was shared as read-only without download permission: " ~ newItemPath); downloadFailed = true; } else { // Default operation if not a 403 error // - 408,429,503,504 errors are handled as a retry within downloadFileOneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } catch (FileException e) { // There was a file system error - display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, newItemPath, FsErrorSeverity.error); if (verboseLogging) {addLogEntry("Download failed (local file system error): " ~ newItemPath, ["verbose"]);} downloadFailed = true; } catch (ErrnoException e) { // There was a file system error - display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, newItemPath, FsErrorSeverity.error); if (verboseLogging) {addLogEntry("Download failed (local file system error): " ~ newItemPath, ["verbose"]);} downloadFailed = true; } // If we get to this point, something was downloaded .. does it match what we expected? // Does it still exist? if (exists(newItemPath)) { // When downloading some files from SharePoint, the OneDrive API reports one file size, // but the SharePoint HTTP Server sends a totally different byte count for the same file // we have implemented --disable-download-validation to disable these checks // Regardless of --disable-download-validation we still need to set the file timestamp correctly // Get the mtime from the JSON data SysTime itemModifiedTime; string lastModifiedTimestamp; if (isItemRemote(onedriveJSONItem)) { // remote file item lastModifiedTimestamp = strip(onedriveJSONItem["remoteItem"]["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp itemModifiedTime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to Clock.currTime(UTC()) given that the time in the JSON should be a UTC timestamp itemModifiedTime = Clock.currTime(UTC()); } } else { // not a remote item lastModifiedTimestamp = strip(onedriveJSONItem["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp itemModifiedTime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to Clock.currTime(UTC()) given that the time in the JSON should be a UTC timestamp itemModifiedTime = Clock.currTime(UTC()); } } // Did the user configure --disable-download-validation ? if (!disableDownloadValidation) { // A 'file' was downloaded - does what we downloaded = reported jsonFileSize or if there is some sort of funky local disk compression going on // Does the file hash OneDrive reports match what we have locally? string onlineFileHash; string downloadedFileHash; long downloadFileSize = getSize(newItemPath); if (!OneDriveFileXORHash.empty) { onlineFileHash = OneDriveFileXORHash; // Calculate the QuickXOHash for this file downloadedFileHash = computeQuickXorHash(newItemPath); } else { onlineFileHash = OneDriveFileSHA256Hash; // Fallback: Calculate the SHA256 Hash for this file downloadedFileHash = computeSHA256Hash(newItemPath); } if ((downloadFileSize == jsonFileSize) && (downloadedFileHash == onlineFileHash)) { // Downloaded file matches size and hash if (debugLogging) {addLogEntry("Downloaded file matches reported size and reported file hash", ["debug"]);} // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, newItemPath, itemModifiedTime); } else { // QuickXorHash in this client incorporates the file length into the final digest, so a size mismatch would be expected to also produce a hash mismatch. // However, QuickXorHash is not collision-resistant, so we treat the hash mismatch as the definitive integrity failure condition and log size mismatches // as advisory // Downloaded file does not match size or hash .. which is it? bool downloadValueMismatch = false; // Size difference between what was written to disk and what the API reported as the file size if (downloadFileSize != jsonFileSize) { // downloaded file size does not match downloadValueMismatch = true; if (debugLogging) { addLogEntry("Actual file size on disk: " ~ to!string(downloadFileSize), ["debug"]); addLogEntry("OneDrive API reported size: " ~ to!string(jsonFileSize), ["debug"]); } if ((verboseLogging)||(debugLogging)) { // verbose or debug message addLogEntry("WARNING: Download validation failed (size mismatch): " ~ newItemPath ~ " | expected=" ~ to!string(jsonFileSize) ~ " | actual=" ~ to!string(downloadFileSize), ["verbose"]); } else { // non-verbose message addLogEntry("WARNING: File download size mismatch. Re-run with --verbose for additional diagnostic information to assist with troubleshooting."); } } // Hash difference between what was written to disk and then QuickXOR calculated and what the API reported as the file hash online if (downloadedFileHash != onlineFileHash) { // downloaded file hash does not match downloadValueMismatch = true; if (debugLogging) { addLogEntry("Actual local file hash: " ~ downloadedFileHash, ["debug"]); addLogEntry("OneDrive API reported hash: " ~ onlineFileHash, ["debug"]); } if ((verboseLogging)||(debugLogging)) { // verbose or debug message addLogEntry("ERROR: Download validation failed (hash mismatch): " ~ newItemPath ~ " | expected=" ~ onlineFileHash ~ " | actual=" ~ downloadedFileHash, ["verbose"]); } else { // non-verbose message addLogEntry("ERROR: File download hash mismatch. Re-run with --verbose for additional diagnostic information to assist with troubleshooting."); } } // .heic data loss check // - https://github.com/abraunegg/onedrive/issues/2471 // - https://github.com/OneDrive/onedrive-api-docs/issues/1532 // - https://github.com/OneDrive/onedrive-api-docs/issues/1723 if (downloadValueMismatch && (toLower(extension(newItemPath)) == ".heic")) { // Need to display a message to the user that they have experienced data loss addLogEntry("DATA-LOSS: File downloaded has experienced data loss due to a Microsoft OneDrive API bug. DO NOT DELETE THIS FILE ONLINE: " ~ newItemPath, ["info", "notify"]); if (verboseLogging) {addLogEntry(" Please read https://github.com/OneDrive/onedrive-api-docs/issues/1723 for more details.", ["verbose"]);} } // Add some workaround messaging for SharePoint if (appConfig.accountType == "documentLibrary"){ // It has been seen where SharePoint / OneDrive API reports one size via the JSON // but the content length and file size written to disk is totally different - example: // From JSON: "size": 17133 // From HTTPS Server: < Content-Length: 19340 // with no logical reason for the difference, except for a 302 redirect before file download addLogEntry("INFO: It is most likely that a SharePoint OneDrive API issue is the root cause. Add --disable-download-validation to work around this issue but downloaded data integrity cannot be guaranteed."); } else { // other account types addLogEntry("INFO: Potentially add --disable-download-validation to work around this issue but downloaded data integrity cannot be guaranteed."); } // If the computed hash does not equal provided online hash, consider this a failed download if (downloadedFileHash != onlineFileHash) { // We do not want this local file to remain on the local file system as it failed the integrity checks addLogEntry("Removing local file " ~ newItemPath ~ " due to failed integrity checks"); if (!dryRun) { safeRemove(newItemPath); } // Was this item previously in-sync with the local system? // We previously searched for the file in the DB, we need to use that record if (fileFoundInDB) { // Purge DB record so that the deleted local file does not cause an online deletion // In a --dry-run scenario, this is being done against a DB copy addLogEntry("Removing DB record due to failed integrity checks"); itemDB.deleteById(databaseItem.driveId, databaseItem.id); } // Flag that the download failed downloadFailed = true; } } } else { // Download validation checks were disabled if (debugLogging) {addLogEntry("Downloaded file validation disabled due to --disable-download-validation", ["debug"]);} if (verboseLogging) {addLogEntry("WARNING: Skipping download integrity check for: " ~ newItemPath, ["verbose"]);} // Whilst the download integrity checks were disabled, we still have to set the correct timestamp on the file // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, newItemPath, itemModifiedTime); // Azure Information Protection (AIP) protected files potentially have missing data and/or inconsistent data if (appConfig.accountType != "personal") { // AIP Protected Files cause issues here, as the online size & hash are not what has been downloaded // There is ZERO way to determine if this is an AIP protected file either from the JSON data // Calculate the local file hash and get the local file size string localFileHash = computeQuickXorHash(newItemPath); long downloadFileSize = getSize(newItemPath); if ((OneDriveFileXORHash != localFileHash) && (jsonFileSize != downloadFileSize)) { // High potential to be an AIP protected file given the following scenario // Business | SharePoint Account Type (not a personal account) // --disable-download-validation is being used .. meaning the user has specifically configured this due the Microsoft SharePoint Enrichment Feature (bug) // The file downloaded but the XOR hash and file size locally is not as per the provided JSON - both are different // // Update the 'onedriveJSONItem' JSON data with the local values ..... if (debugLogging) { string aipLogMessage = format("POTENTIAL AIP FILE (Issue 3070) - Changing the source JSON data provided by Graph API to use actual on-disk values (quickXorHash,size): %s", newItemPath); addLogEntry(aipLogMessage, ["debug"]); addLogEntry(" - Online XOR : " ~ to!string(OneDriveFileXORHash), ["debug"]); addLogEntry(" - Online Size : " ~ to!string(jsonFileSize), ["debug"]); addLogEntry(" - Local XOR : " ~ to!string(computeQuickXorHash(newItemPath)), ["debug"]); addLogEntry(" - Local Size : " ~ to!string(getSize(newItemPath)), ["debug"]); } // Make the change in the JSON using local values onedriveJSONItem["file"]["hashes"]["quickXorHash"] = localFileHash; onedriveJSONItem["size"] = downloadFileSize; } } } // end of (!disableDownloadValidation) } else { // File does not exist locally ... so the download failed if ((verboseLogging)||(debugLogging)) { // If we are doing verbose logging, addLogEntry("ERROR: Download failed (file not present after download): " ~ newItemPath ~ " | expectedSize=" ~ to!string(jsonFileSize) ~ " | resumeOffset=" ~ to!string(resumeOffset), ["verbose"]); } else { addLogEntry("ERROR: File failed to download. Re-run with --verbose for additional diagnostic information to assist with troubleshooting."); } // Was this item previously in-sync with the local system? // We previously searched for the file in the DB, we need to use that record if (fileFoundInDB) { // Purge DB record so that the deleted local file does not cause an online deletion // In a --dry-run scenario, this is being done against a DB copy addLogEntry("Removing existing DB record due to failed file download."); itemDB.deleteById(databaseItem.driveId, databaseItem.id); } // Flag that the download failed downloadFailed = true; } } } // File should have been downloaded if (!downloadFailed) { // Download did not fail addLogEntry("Downloading file: " ~ newItemPath ~ " ... done", fileTransferNotifications()); // As no download failure, calculate transfer metrics in a consistent manner displayTransferMetrics(newItemPath, jsonFileSize, downloadStartTime, Clock.currTime()); // Save this item into the database saveItem(onedriveJSONItem); // If we are in a --dry-run situation - if we are, we need to track that we faked the download if (dryRun) { // track that we 'faked it' idsFaked ~= [downloadDriveId, downloadItemId]; } // If, the initial download failed, but, during the 'Performing a last examination of the most recent online data within Microsoft OneDrive' Process // the file downloads without issue, check if the path is in 'fileDownloadFailures' and if this is in this array, remove this entry as it is technically no longer valid to be in there if (canFind(fileDownloadFailures, newItemPath)) { // Remove 'newItemPath' from 'fileDownloadFailures' as this is no longer a failed download fileDownloadFailures = fileDownloadFailures.filter!(item => item != newItemPath).array; } // Did the user configure to save xattr data about this file? if (appConfig.getValueBool("write_xattr_data")) { writeXattrData(newItemPath, onedriveJSONItem); } } else { // Output to the user that the file download failed addLogEntry("Downloading file: " ~ newItemPath ~ " ... failed!", ["info", "notify"]); // Add the path to a list of items that failed to download if (!canFind(fileDownloadFailures, newItemPath)) { fileDownloadFailures ~= newItemPath; // Add newItemPath if it's not already present } // Since the file download failed: // - The file should not exist locally // - The download identifiers should not exist in the local database if (!exists(newItemPath)) { // The local path does not exist if (itemDB.idInLocalDatabase(downloadDriveId, downloadItemId)) { // Since the path does not exist, but the driveId and itemId exists in the database, when we do the DB consistency check, we will think this file has been 'deleted' // The driveId and itemId online exists in our database - it needs to be removed so this does not occur addLogEntry("Removing existing DB record due to failed file download."); itemDB.deleteById(downloadDriveId, downloadItemId); } } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Write xattr data if configured to do so void writeXattrData(string filePath, JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // We can only set xattr values when not performing a --dry-run operation if (!dryRun) { // This function will write the following xattr attributes based on the JSON data received from Microsoft onedrive // - createdBy using the 'displayName' value // - lastModifiedBy using the 'displayName' value string createdBy; string lastModifiedBy; // Configure 'createdBy' from the JSON data if (hasCreatedByUserDisplayName(onedriveJSONItem)) { createdBy = onedriveJSONItem["createdBy"]["user"]["displayName"].str; } else { // required data not in JSON data createdBy = "Unknown"; } // Configure 'lastModifiedBy' from the JSON data if (hasLastModifiedByUserDisplayName(onedriveJSONItem)) { lastModifiedBy = onedriveJSONItem["lastModifiedBy"]["user"]["displayName"].str; } else { // required data not in JSON data lastModifiedBy = "Unknown"; } // Set the xattr values, file must exist to set these values if (exists(filePath)) { setXAttr(filePath, "user.onedrive.createdBy", createdBy); setXAttr(filePath, "user.onedrive.lastModifiedBy", lastModifiedBy); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Test if the given item is in-sync. Returns true if the given item corresponds to the local one bool isItemSynced(Item item, string path, string itemSource) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return ; code, so that this function operates as efficiently as possible. // It is pointless having the entire code run through and performing additional needless checks where it is not required // Whilst this means some extra code / duplication in this function, it cannot be helped if (!exists(path)) { if (debugLogging) {addLogEntry("Unable to determine the sync state of this file as it does not exist: " ~ path, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return false; } // Combine common logic for readability and file check into a single block if (item.type == ItemType.file || ((item.type == ItemType.remote) && (item.remoteType == ItemType.file))) { // Can we actually read the local file? if (!readLocalFile(path)) { // Unable to read local file addLogEntry("Unable to determine the sync state of this file as it cannot be read (file permissions or file corruption): " ~ path); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return false; } // Get time values SysTime localModifiedTime = timeLastModified(path).toUTC(); SysTime itemModifiedTime = item.mtime; // Reduce time resolution to seconds before comparing localModifiedTime.fracSecs = Duration.zero; itemModifiedTime.fracSecs = Duration.zero; if (localModifiedTime == itemModifiedTime) { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return true; } else { // The file has a different timestamp ... is the hash the same meaning no file modification? if (verboseLogging) { addLogEntry("Local file time discrepancy detected: " ~ path, ["verbose"]); addLogEntry("This local file has a different modified time " ~ to!string(localModifiedTime) ~ " (UTC) when compared to " ~ itemSource ~ " modified time " ~ to!string(itemModifiedTime) ~ " (UTC)", ["verbose"]); } // The file has a different timestamp ... is the hash the same meaning no file modification? // Test the file hash as the date / time stamp is different // Generating a hash is computationally expensive - we only generate the hash if timestamp was different if (testFileHash(path, item)) { // The hash is the same .. so we need to fix-up the timestamp depending on where it is wrong if (verboseLogging) {addLogEntry("Local item has the same hash value as the item online - correcting the applicable file timestamp", ["verbose"]);} // Correction logic based on the configuration and the comparison of timestamps if (localModifiedTime > itemModifiedTime) { // Local file is newer timestamp wise, but has the same hash .. are we in a --download-only situation? if (!appConfig.getValueBool("download_only") && !dryRun) { // Not --download-only .. but are we in a --resync scenario? if (appConfig.getValueBool("resync")) { // --resync was used // The source of the out-of-date timestamp was the local item and needs to be corrected ... but why is it newer - indexing application potentially changing the timestamp ? if (verboseLogging) {addLogEntry("The source of the incorrect timestamp was the local file - correcting timestamp locally due to --resync", ["verbose"]);} // Fix the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, path, item.mtime); } else { // The source of the out-of-date timestamp was OneDrive and this needs to be corrected to avoid always generating a hash test if timestamp is different if (verboseLogging) {addLogEntry("The source of the incorrect timestamp was OneDrive online - correcting timestamp online", ["verbose"]);} // Attempt to update the online date time stamp // We need to use the correct driveId and itemId, especially if we are updating a OneDrive Business Shared File timestamp if (item.type == ItemType.file) { // Not a remote file uploadLastModifiedTime(item, item.driveId, item.id, localModifiedTime, item.eTag); } else { // Remote file, remote values need to be used uploadLastModifiedTime(item, item.remoteDriveId, item.remoteId, localModifiedTime, item.eTag); } } } else if (!dryRun) { // --download-only is being used ... local file needs to be corrected ... but why is it newer - indexing application potentially changing the timestamp ? if (verboseLogging) {addLogEntry("The source of the incorrect timestamp was the local file - correcting timestamp locally due to --download-only", ["verbose"]);} // Fix the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, path, item.mtime); } } else if (!dryRun) { // The source of the out-of-date timestamp was the local file and this needs to be corrected to avoid always generating a hash test if timestamp is different if (verboseLogging) {addLogEntry("The source of the incorrect timestamp was the local file - correcting timestamp locally", ["verbose"]);} // Fix the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, path, item.mtime); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return false; } else { // The hash is different so the content of the file has to be different as to what is stored online if (verboseLogging) {addLogEntry("The local file has a different hash when compared to " ~ itemSource ~ " file hash", ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return false; } } } else if (item.type == ItemType.dir || ((item.type == ItemType.remote) && (item.remoteType == ItemType.dir))) { // item is a directory // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return true; } else { // ItemType.unknown or ItemType.none // Logically, we might not want to sync these items, but a more nuanced approach may be needed based on application context // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return true; } } // Get the /delta data using the provided details JSONValue getDeltaChangesByItemId(string selectedDriveId, string selectedItemId, string providedDeltaLink, OneDriveApi getDeltaQueryOneDriveApiInstance) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables JSONValue deltaChangesBundle; // Get the /delta data for this account | driveId | deltaLink combination if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); addLogEntry("selectedDriveId: " ~ selectedDriveId, ["debug"]); addLogEntry("selectedItemId: " ~ selectedItemId, ["debug"]); addLogEntry("providedDeltaLink: " ~ providedDeltaLink, ["debug"]); addLogEntry(debugLogBreakType1, ["debug"]); } try { deltaChangesBundle = getDeltaQueryOneDriveApiInstance.getChangesByItemId(selectedDriveId, selectedItemId, providedDeltaLink); } catch (OneDriveException exception) { // caught an exception if (debugLogging) {addLogEntry("getDeltaQueryOneDriveApiInstance.getChangesByItemId(selectedDriveId, selectedItemId, providedDeltaLink) generated a OneDriveException", ["debug"]);} // get the error message auto errorArray = splitLines(exception.msg); // Error handling operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within getDeltaQueryOneDriveApiInstance if (exception.httpStatusCode == 410) { addLogEntry(); addLogEntry("WARNING: The OneDrive API responded with an error that indicates the locally stored deltaLink value is invalid"); // Essentially the 'providedDeltaLink' that we have stored is no longer available ... re-try without the stored deltaLink addLogEntry("WARNING: Retrying OneDrive API call without using the locally stored deltaLink value"); // Configure an empty deltaLink if (debugLogging) {addLogEntry("Delta link expired for 'getDeltaQueryOneDriveApiInstance.getChangesByItemId(selectedDriveId, selectedItemId, providedDeltaLink)', setting 'deltaLink = null'", ["debug"]);} string emptyDeltaLink = ""; // retry with empty deltaLink deltaChangesBundle = getDeltaQueryOneDriveApiInstance.getChangesByItemId(selectedDriveId, selectedItemId, emptyDeltaLink); } else { // Display what the error is addLogEntry("CODING TO DO: Hitting this failure error output after getting a httpStatusCode != 410 when the API responded the deltaLink was invalid"); displayOneDriveErrorMessage(exception.msg, thisFunctionName); deltaChangesBundle = null; // Perform Garbage Collection GC.collect(); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return data return deltaChangesBundle; } // If the JSON response is not correct JSON object, exit void invalidJSONResponseFromOneDriveAPI() { addLogEntry("ERROR: Query of the OneDrive API returned an invalid JSON response"); // Must force exit here, allow logging to be done forceExit(); } // Handle an unhandled API error void defaultUnhandledHTTPErrorCode(OneDriveException exception) { // compute function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // display error displayOneDriveErrorMessage(exception.msg, thisFunctionName); // Must force exit here, allow logging to be done forceExit(); } // Display the pertinent details of the sync engine void displaySyncEngineDetails() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Display accountType, defaultDriveId, defaultRootId & remainingFreeSpace for verbose logging purposes addLogEntry("Application Version: " ~ appConfig.applicationVersion, ["verbose"]); addLogEntry("Account Type: " ~ appConfig.accountType, ["verbose"]); addLogEntry("Default Drive ID: " ~ appConfig.defaultDriveId, ["verbose"]); addLogEntry("Default Root ID: " ~ appConfig.defaultRootId, ["verbose"]); addLogEntry("Microsoft Data Centre: " ~ microsoftDataCentre, ["verbose"]); // Fetch the details from cachedOnlineDriveData DriveDetailsCache cachedOnlineDriveData; cachedOnlineDriveData = getDriveDetails(appConfig.defaultDriveId); // What do we display here for space remaining if (cachedOnlineDriveData.quotaRemaining > 0) { // Display the actual value addLogEntry("Remaining Free Space: " ~ to!string(byteToGibiByte(cachedOnlineDriveData.quotaRemaining)) ~ " GB (" ~ to!string(cachedOnlineDriveData.quotaRemaining) ~ " bytes)", ["verbose"]); } else { // zero or non-zero value or restricted if (!cachedOnlineDriveData.quotaRestricted){ addLogEntry("Remaining Free Space: 0 KB", ["verbose"]); } else { addLogEntry("Remaining Free Space: Not Available", ["verbose"]); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query itemdb.computePath() and catch potential assert when DB consistency issue occurs // This function returns what that local physical path should be on the local disk string computeItemPath(string thisDriveId, string thisItemId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // static declare this for this function static import core.exception; string calculatedPath; // Issue #3336 - Convert thisDriveId to lowercase before any test if (appConfig.accountType == "personal") { thisDriveId = transformToLowerCase(thisDriveId); } // What driveID and itemID we trying to calculate the path for if (debugLogging) { string initialComputeLogMessage = format("Attempting to calculate local filesystem path for '%s' and '%s'", thisDriveId, thisItemId); addLogEntry(initialComputeLogMessage, ["debug"]); } // Perform the original calculation of the path using the values provided try { // The 'itemDB.computePath' will calculate the full path for the combination of provided driveId and itemId values. // This function traverses the parent chain of a given item (e.g., folder or file) using stored parent-child relationships // in the database, reconstructing the correct path from the item's root to itself. calculatedPath = itemDB.computePath(thisDriveId, thisItemId); if (debugLogging) {addLogEntry("Calculated local path via itemDB.computePath() = " ~ to!string(calculatedPath), ["debug"]);} } catch (core.exception.AssertError) { // broken tree in the database, we cant compute the path for this item id, exit addLogEntry("ERROR: A database consistency issue has been caught. A --resync is needed to rebuild the database."); // Must force exit here, allow logging to be done forceExit(); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return calculated path as string return calculatedPath; } // Try and compute the file hash for the given item bool testFileHash(string path, Item item) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return ; code, so that this function operates as efficiently as possible. // It is pointless having the entire code run through and performing additional needless checks where it is not required // Whilst this means some extra code / duplication in this function, it cannot be helped // Generate QuickXORHash first before attempting to generate any other type of hash if (item.quickXorHash) { if (item.quickXorHash == computeQuickXorHash(path)) { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return true; } } else if (item.sha256Hash) { if (item.sha256Hash == computeSHA256Hash(path)) { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return true; } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return false; } // Process items that need to be removed from the local filesystem as they were removed online void processDeleteItems() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Has the user configured to use the 'Recycle Bin' locally, for any files that are deleted online? if (!appConfig.getValueBool("use_recycle_bin")) { if (debugLogging) {addLogEntry("Performing filesystem deletion, using reverse order of items to delete", ["debug"]);} foreach_reverse (i; idsToDelete) { Item item; string path; if (!itemDB.selectById(i[0], i[1], item)) continue; // check if the item is in the db // Compute this item path path = computeItemPath(i[0], i[1]); // Log the action if the path exists .. it may of already been removed and this is a legacy array item if (exists(path)) { if (item.type == ItemType.file) { addLogEntry("Trying to delete local file: " ~ path); } else { addLogEntry("Trying to delete local directory: " ~ path); } } // Process the database entry removal. In a --dry-run scenario, this is being done against a DB copy itemDB.deleteById(item.driveId, item.id); if (item.remoteDriveId != null) { // delete the linked remote folder itemDB.deleteById(item.remoteDriveId, item.remoteId); } // Add to pathFakeDeletedArray // We dont want to try and upload this item again, so we need to track this objects removal if (dryRun) { // We need to add './' here so that it can be correctly searched to ensure it is not uploaded string pathToAdd = "./" ~ path; pathFakeDeletedArray ~= pathToAdd; } bool needsRemoval = false; if (exists(path)) { // path exists on the local system // make sure that the path refers to the correct item Item pathItem; if (itemDB.selectByPath(path, item.driveId, pathItem)) { if (pathItem.id == item.id) { needsRemoval = true; } else { addLogEntry("Skipping local path removal due to 'id' difference!"); } } else { // item has disappeared completely needsRemoval = true; } } if (needsRemoval) { // Log the action if (item.type == ItemType.file) { addLogEntry("Deleting local file: " ~ path, fileTransferNotifications()); } else { addLogEntry("Deleting local directory: " ~ path, fileTransferNotifications()); } // Perform the action if (!dryRun) { if (isFile(path)) { safeRemove(path); } else { try { // Remove any children of this path if they still exist // Resolve 'Directory not empty' error when deleting local files foreach (DirEntry child; dirEntries(path, SpanMode.depth, false)) { attrIsDir(child.linkAttributes) ? rmdir(child.name) : safeRemove(child.name); } // Remove the path now that it is empty of children rmdirRecurse(path); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, path); } } } } } } else { if (debugLogging) {addLogEntry("Moving online deleted files to configured local Recycle Bin", ["debug"]);} // Process in normal order, so that the parent, if a folder, gets moved 'first' mirroring how files / folders are deleted in GNOME and KDE foreach (i; idsToDelete) { Item item; string path; if (!itemDB.selectById(i[0], i[1], item)) continue; // check if the item is in the db // Compute this item path path = computeItemPath(i[0], i[1]); // Log the action if the path exists .. it may of already been removed and this is a legacy array item if (exists(path)) { if (item.type == ItemType.file) { addLogEntry("Trying to move this local file to the configured 'Recycle Bin': " ~ path); } else { addLogEntry("Trying to move this local directory to the configured 'Recycle Bin': " ~ path); } } // Process the database entry removal. In a --dry-run scenario, this is being done against a DB copy itemDB.deleteById(item.driveId, item.id); if (item.remoteDriveId != null) { // delete the linked remote folder itemDB.deleteById(item.remoteDriveId, item.remoteId); } // Add to pathFakeDeletedArray // We dont want to try and upload this item again, so we need to track this objects removal if (dryRun) { // We need to add './' here so that it can be correctly searched to ensure it is not uploaded string pathToAdd = "./" ~ path; pathFakeDeletedArray ~= pathToAdd; } // Local path removal bool needsRemoval = false; if (exists(path)) { // path exists on the local system // make sure that the path refers to the correct item Item pathItem; if (itemDB.selectByPath(path, item.driveId, pathItem)) { if (pathItem.id == item.id) { needsRemoval = true; } else { addLogEntry("Skipping local path removal due to 'id' difference!"); } } else { // item has disappeared completely needsRemoval = true; } } if (needsRemoval) { // Log the action if (item.type == ItemType.file) { addLogEntry("Moving this local file to the configured 'Recycle Bin': " ~ path, fileTransferNotifications()); } else { addLogEntry("Moving this local directory to the configured 'Recycle Bin': " ~ path, fileTransferNotifications()); } // Perform the action if (!dryRun) { // Move the 'path' to the configured recycle bin movePathToRecycleBin(path); } } } } if (!dryRun) { // Cleanup array memory idsToDelete = []; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Move to the 'Recycle Bin' rather than a hard delete locally of the online deleted item void movePathToRecycleBin(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This is a 2 step process // 1. Move the file // - If the destination 'name' already exists, the file being moved to the 'Recycle Bin' needs to have a number added to it. // 2. Create the metadata about where the file came from // - This is in a specific format: // [Trash Info] // Path=/original/absolute/path/to/the/file/or/folder // DeletionDate=YYYY-MM-DDTHH:MM:SS // Calculate all the initial paths required string computedFullLocalPath = absolutePath(path); string fileNameOnly = baseName(path); string computedRecycleBinFilePath = appConfig.recycleBinFilePath ~ fileNameOnly; string computedRecycleBinInfoPath = appConfig.recycleBinInfoPath ~ fileNameOnly ~ ".trashinfo"; bool isPathFile = isFile(computedFullLocalPath); // The 'destination' needs to be unique, but if there is a 'collision' the RecycleBin paths need to be updated to be: // - file1.data (1) // - file1.data (1).trashinfo if (exists(computedRecycleBinFilePath)) { // There is an existing file with the same name already in the 'Recycle Bin' // - Testing has show that this counter MUST start at 2 to be compatible with FreeDesktop.org Trash Specification .... int n = 2; // We need to split this out string nameOnly = stripExtension(fileNameOnly); // "file1" string extension = extension(fileNameOnly); // ".data" // We need to test for this: nameOnly.n.extension while (exists(format(appConfig.recycleBinFilePath ~ nameOnly ~ ".%d." ~ extension, n))) { n++; } // Generate newFileNameOnly string newFileNameOnly = format(nameOnly ~ ".%d." ~ extension, n); // UPDATE: // - computedRecycleBinFilePath // - computedRecycleBinInfoPath computedRecycleBinFilePath = appConfig.recycleBinFilePath ~ newFileNameOnly; computedRecycleBinInfoPath = appConfig.recycleBinInfoPath ~ newFileNameOnly ~ ".trashinfo"; } // Move the file to the 'Recycle Bin' path computedRecycleBinFilePath // - DMD has no 'move' specifically, it uses 'rename' to achieve this // https://forum.dlang.org/thread/kwnwrlqtjehldckyfmau@forum.dlang.org // Use rename() as Linux is POSIX compliant, we have an atomic operation where at no point in time the 'to' is missing. try { rename(computedFullLocalPath, computedRecycleBinFilePath); } catch (Exception e) { // Handle exceptions, e.g., log error if (isPathFile) { addLogEntry("Move of local file failed for " ~ to!string(path) ~ ": " ~ e.msg, ["error"]); } else { addLogEntry("Move of local directory failed for " ~ to!string(path) ~ ": " ~ e.msg, ["error"]); } } // Generate the 'Recycle Bin' metadata file using computedRecycleBinInfoPath auto now = Clock.currTime().toLocalTime(); string deletionDate = format("%04d-%02d-%02dT%02d:%02d:%02d",now.year, now.month, now.day, now.hour, now.minute, now.second); // Format the content of the .trashinfo file string content = format("[Trash Info]\nPath=%s\nDeletionDate=%s\n", computedFullLocalPath, deletionDate); // Write the metadata file try { std.file.write(computedRecycleBinInfoPath, content); } catch (Exception e) { // Handle exceptions, e.g., log error addLogEntry("Writing of .trashinfo metadata file failed for " ~ computedRecycleBinInfoPath ~ ": " ~ e.msg, ["error"]); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // List items that were deleted online, but, due to --download-only being used, will not be deleted locally void listDeletedItems() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // For each id in the idsToDelete array foreach_reverse (i; idsToDelete) { Item item; string path; if (!itemDB.selectById(i[0], i[1], item)) continue; // check if the item is in the db // Compute this item path path = computeItemPath(i[0], i[1]); // Log the action if the path exists .. it may of already been removed and this is a legacy array item if (exists(path)) { if (item.type == ItemType.file) { if (verboseLogging) {addLogEntry("Skipping local deletion for file " ~ path, ["verbose"]);} } else { if (verboseLogging) {addLogEntry("Skipping local deletion for directory " ~ path, ["verbose"]);} } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Update the timestamp of an object online void uploadLastModifiedTime(Item originItem, string driveId, string id, SysTime mtime, string eTag) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } string itemModifiedTime; itemModifiedTime = mtime.toISOExtString(); JSONValue data = [ "fileSystemInfo": JSONValue([ "lastModifiedDateTime": itemModifiedTime ]) ]; // What eTag value do we use? string eTagValue; if (appConfig.accountType == "personal") { // Nullify the eTag to avoid 412 errors as much as possible eTagValue = null; } else { eTagValue = eTag; } JSONValue response; OneDriveApi uploadLastModifiedTimeApiInstance; // Try and update the online last modified time try { // Create a new OneDrive API instance uploadLastModifiedTimeApiInstance = new OneDriveApi(appConfig); uploadLastModifiedTimeApiInstance.initialise(); // Use this instance response = uploadLastModifiedTimeApiInstance.updateById(driveId, id, data, eTagValue); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadLastModifiedTimeApiInstance.releaseCurlEngine(); uploadLastModifiedTimeApiInstance = null; // Perform Garbage Collection GC.collect(); // Do we actually save the response? // Special case here .. if the DB record item (originItem) is a remote object, thus, if we save the 'response' we will have a DB FOREIGN KEY constraint failed problem // Update 'originItem.mtime' with the correct timestamp // Update 'originItem.size' with the correct size from the response // Update 'originItem.eTag' with the correct eTag from the response // Update 'originItem.cTag' with the correct cTag from the response // Update 'originItem.quickXorHash' with the correct quickXorHash from the response // Everything else should remain the same .. and then save this DB record to the DB .. // However, we did this, for the local modified file right before calling this function to update the online timestamp ... so .. do we need to do this again, effectively performing a double DB write for the same data? if ((originItem.type != ItemType.remote) && (originItem.remoteType != ItemType.file)) { if (response.type() == JSONType.object) { // Save the response JSON saveItem(response); } else { // Log why we are not saving if (debugLogging) {addLogEntry("uploadLastModifiedTime: updateById returned no JSON payload (likely HTTP 204); skipping saveItem()", ["debug"]);} } } } catch (OneDriveException exception) { // Handle a 409 - ETag does not match current item's value // Handle a 412 - A precondition provided in the request (such as an if-match header) does not match the resource's current state. if ((exception.httpStatusCode == 409) || (exception.httpStatusCode == 412)) { // Handle the 409 if (exception.httpStatusCode == 409) { // OneDrive threw a 412 error if (verboseLogging) {addLogEntry("OneDrive returned a 'HTTP 409 - ETag does not match current item's value' when attempting file time stamp update - gracefully handling error", ["verbose"]);} if (debugLogging) { addLogEntry("File Metadata Update Failed - OneDrive eTag / cTag match issue", ["debug"]); addLogEntry("Retrying Function: " ~ thisFunctionName, ["debug"]); } } // Handle the 412 if (exception.httpStatusCode == 412) { // OneDrive threw a 412 error if (verboseLogging) {addLogEntry("OneDrive returned a 'HTTP 412 - Precondition Failed' when attempting file time stamp update - gracefully handling error", ["verbose"]);} if (debugLogging) { addLogEntry("File Metadata Update Failed - OneDrive eTag / cTag match issue", ["debug"]); addLogEntry("Retrying Function: " ~ thisFunctionName, ["debug"]); } } // Retry without eTag uploadLastModifiedTime(originItem, driveId, id, mtime, null); } else { // Any other error that should be handled // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadLastModifiedTimeApiInstance.releaseCurlEngine(); uploadLastModifiedTimeApiInstance = null; // Perform Garbage Collection GC.collect(); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform a database integrity check - checking all the items that are in-sync at the moment, validating what we know should be on disk, to what is actually on disk void performDatabaseConsistencyAndIntegrityCheck() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Log what we are doing if (!appConfig.suppressLoggingOutput) { addProcessingLogHeaderEntry("Performing a database consistency and integrity check on locally stored data", appConfig.verbosityCount); } // What driveIDsArray do we use? If we are doing a --single-directory we need to use just the drive id associated with that operation string[] consistencyCheckDriveIdsArray; if (singleDirectoryScope) { consistencyCheckDriveIdsArray ~= singleDirectoryScopeDriveId; } else { // Query the DB for all unique DriveID's consistencyCheckDriveIdsArray = itemDB.selectDistinctDriveIds(); } // Create a new DB blank item Item item; // Use the array we populate, rather than selecting all distinct driveId's from the database foreach (driveId; consistencyCheckDriveIdsArray) { // Make the logging more accurate - we cant update driveId as this then breaks the below queries if (verboseLogging) {addLogEntry("Processing DB entries for this Drive ID: " ~ driveId, ["verbose"]);} // Initialise the array Item[] driveItems = []; // Freshen the cached quota details for this driveID addOrUpdateOneDriveOnlineDetails(driveId); // What OneDrive API query do we use? // - Are we running against a National Cloud Deployments that does not support /delta ? // National Cloud Deployments do not support /delta as a query // https://docs.microsoft.com/en-us/graph/deployments#supported-features // // - Are we performing a --single-directory sync, which will exclude many items online, focusing in on a specific online directory // - Are we performing a --download-only --cleanup-local-files action? // - Are we scanning a Shared Folder // // If we did, we self generated a /delta response, thus need to now process elements that are still flagged as out-of-sync if ((singleDirectoryScope) || (nationalCloudDeployment) || (cleanupLocalFiles) || sharedFolderDeltaGeneration) { // Any entry in the DB than is flagged as out-of-sync needs to be cleaned up locally first before we scan the entire DB // Normally, this is done at the end of processing all /delta queries, however when using --single-directory or a National Cloud Deployments is configured // We cant use /delta to query the OneDrive API as National Cloud Deployments dont support /delta // https://docs.microsoft.com/en-us/graph/deployments#supported-features // We dont use /delta for --single-directory as, in order to sync a single path with /delta, we need to query the entire OneDrive API JSON data to then filter out // objects that we dont want, thus, it is easier to use the same method as National Cloud Deployments, but query just the objects we are after // For each unique OneDrive driveID we know about Item[] outOfSyncItems = itemDB.selectOutOfSyncItems(driveId); foreach (outOfSyncItem; outOfSyncItems) { if (!dryRun) { // clean up idsToDelete idsToDelete.length = 0; assumeSafeAppend(idsToDelete); // flag to delete local file as it now is no longer in sync with OneDrive if (debugLogging) { addLogEntry("Flagging to delete local item as it now is no longer in sync with OneDrive", ["debug"]); addLogEntry("outOfSyncItem: " ~ to!string(outOfSyncItem), ["debug"]); } // Use the configured values - add the driveId, itemId and parentId values to the array idsToDelete ~= [outOfSyncItem.driveId, outOfSyncItem.id, outOfSyncItem.parentId]; // delete items in idsToDelete if (idsToDelete.length > 0) processDeleteItems(); } } // Clear array outOfSyncItems = []; // Fetch database items associated with this path if (singleDirectoryScope) { // Use the --single-directory items we previously configured // - query database for children objects using those items driveItems = getChildren(singleDirectoryScopeDriveId, singleDirectoryScopeItemId); } else { // Check everything associated with each driveId we know about if (debugLogging) {addLogEntry("Selecting DB items via itemDB.selectByDriveId(driveId)", ["debug"]);} // Query database driveItems = itemDB.selectByDriveId(driveId); } // Log DB items to process if (debugLogging) {addLogEntry("Database items to process for this driveId: " ~ to!string(driveItems.count), ["debug"]);} // Process each database item associated with the driveId foreach(dbItem; driveItems) { // Does it still exist on disk in the location the DB thinks it is checkDatabaseItemForConsistency(dbItem); } } else { // Check everything associated with each driveId we know about if (debugLogging) {addLogEntry("Selecting DB items via itemDB.selectByDriveId(driveId)", ["debug"]);} // Query database driveItems = itemDB.selectByDriveId(driveId); if (debugLogging) {addLogEntry("Database items to process for this driveId: " ~ to!string(driveItems.count), ["debug"]);} // Process each database item associated with the driveId foreach(dbItem; driveItems) { // Does it still exist on disk in the location the DB thinks it is checkDatabaseItemForConsistency(dbItem); } } // Clear the array driveItems = []; } // Close out the '....' being printed to the console if (!appConfig.suppressLoggingOutput) { if (appConfig.verbosityCount == 0) { completeProcessingDots(); } } // Are we doing a --download-only sync? if (!appConfig.getValueBool("download_only")) { // Do we have any known items, where they have been deleted locally, that now need to be deleted online? if (databaseItemsToDeleteOnline.length > 0) { // There are items to delete online addLogEntry("Deleted local items to delete on Microsoft OneDrive: " ~ to!string(databaseItemsToDeleteOnline.length)); foreach(localItemToDeleteOnline; databaseItemsToDeleteOnline) { // Upload to OneDrive the instruction to delete this item. This will handle the 'noRemoteDelete' flag if set uploadDeletedItem(localItemToDeleteOnline.dbItem, localItemToDeleteOnline.localFilePath); } // Cleanup array memory databaseItemsToDeleteOnline = []; } // Do we have any known items, where the content has changed locally, that needs to be uploaded? if (databaseItemsWhereContentHasChanged.length > 0) { // There are changed local files that were in the DB to upload addLogEntry("Changed local items to upload to Microsoft OneDrive: " ~ to!string(databaseItemsWhereContentHasChanged.length)); processChangedLocalItemsToUpload(); // Cleanup array memory databaseItemsWhereContentHasChanged = []; } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Check this Database Item for its consistency on disk void checkDatabaseItemForConsistency(Item dbItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return ; code, so that this function operates as efficiently as possible. // It is pointless having the entire code run through and performing additional needless checks where it is not required // Whilst this means some extra code / duplication in this function, it cannot be helped // What is the local path item string localFilePath; // Do we want to onward process this item? bool unwanted = false; // Remote directory items we can 'skip' if ((dbItem.type == ItemType.remote) && (dbItem.remoteType == ItemType.dir)) { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return .. nothing to check here, no logging needed return; } // Compute this dbItem path early as we we use this path often localFilePath = buildNormalizedPath(computeItemPath(dbItem.driveId, dbItem.id)); // To improve logging output for this function, what is the 'logical path'? string logOutputPath; if (localFilePath == ".") { // get the configured sync_dir logOutputPath = buildNormalizedPath(appConfig.getValueString("sync_dir")); } else { // Use the path that was computed logOutputPath = localFilePath; } // Log what we are doing if (verboseLogging) {addLogEntry("Processing: " ~ logOutputPath, ["verbose"]);} // Add a processing '.' if (!appConfig.suppressLoggingOutput) { if (appConfig.verbosityCount == 0) { addProcessingDotEntry(); } } // Debug logging of paths being checked if (debugLogging) { addLogEntry("Database item being checked: " ~ to!string(dbItem), ["debug"]); addLogEntry("Local Path being checked: " ~ localFilePath, ["debug"]); } // Determine which action to take final switch (dbItem.type) { case ItemType.file: // Logging output result is handled by checkFileDatabaseItemForConsistency checkFileDatabaseItemForConsistency(dbItem, localFilePath); goto functionCompletion; case ItemType.dir, ItemType.root: // Logging output result is handled by checkDirectoryDatabaseItemForConsistency checkDirectoryDatabaseItemForConsistency(dbItem, localFilePath); goto functionCompletion; case ItemType.remote: // DB items that match: dbItem.remoteType == ItemType.dir - these should have been skipped above // This means that anything that hits here should be: dbItem.remoteType == ItemType.file checkFileDatabaseItemForConsistency(dbItem, localFilePath); goto functionCompletion; case ItemType.unknown: case ItemType.none: // Unknown type - we dont action these items goto functionCompletion; } // To correctly handle a switch|case statement we use goto post the switch|case statement as if 'break' is used, we never get to this point functionCompletion: // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the database consistency check on this file item void checkFileDatabaseItemForConsistency(Item dbItem, string localFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // What is the source of this item data? string itemSource = "database"; // Does this item|file still exist on disk? if (exists(localFilePath)) { // Path exists locally, is this path a file? if (isFile(localFilePath)) { // Can we actually read the local file? if (readLocalFile(localFilePath)){ // File is readable SysTime localModifiedTime = timeLastModified(localFilePath).toUTC(); SysTime itemModifiedTime = dbItem.mtime; // Reduce time resolution to seconds before comparing itemModifiedTime.fracSecs = Duration.zero; localModifiedTime.fracSecs = Duration.zero; if (localModifiedTime != itemModifiedTime) { // The modified dates are different if (verboseLogging) { addLogEntry("Local file time discrepancy detected: " ~ localFilePath, ["verbose"]); addLogEntry("This local file has a different modified time " ~ to!string(localModifiedTime) ~ " (UTC) when compared to " ~ itemSource ~ " modified time " ~ to!string(itemModifiedTime) ~ " (UTC)", ["verbose"]); } // Test the file hash if (!testFileHash(localFilePath, dbItem)) { // Is the local file 'newer' or 'older' (ie was an old file 'restored locally' by a different backup / replacement process?) if (localModifiedTime >= itemModifiedTime) { // Local file is newer if (!appConfig.getValueBool("download_only")) { if (verboseLogging) {addLogEntry("The file content has changed locally and has a newer timestamp, thus needs to be uploaded to OneDrive", ["verbose"]);} // Add to an array of files we need to upload as this file has changed locally in-between doing the /delta check and performing this check databaseItemsWhereContentHasChanged ~= [dbItem.driveId, dbItem.id, localFilePath]; } else { if (verboseLogging) {addLogEntry("The file content has changed locally and has a newer timestamp. The file will remain different to online file due to --download-only being used", ["verbose"]);} } } else { // Local file is older - data recovery process? something else? if (!appConfig.getValueBool("download_only")) { if (verboseLogging) {addLogEntry("The file content has changed locally and file now has a older timestamp. Uploading this file to OneDrive may potentially cause data-loss online", ["verbose"]);} // Add to an array of files we need to upload as this file has changed locally in-between doing the /delta check and performing this check databaseItemsWhereContentHasChanged ~= [dbItem.driveId, dbItem.id, localFilePath]; } else { if (verboseLogging) {addLogEntry("The file content has changed locally and file now has a older timestamp. The file will remain different to online file due to --download-only being used", ["verbose"]);} } } } else { // The file contents have not changed, but the modified timestamp has if (verboseLogging) {addLogEntry("The last modified timestamp has changed however the file content has not changed", ["verbose"]);} // Local file is newer .. are we in a --download-only situation? if (!appConfig.getValueBool("download_only")) { // Not a --download-only scenario if (!dryRun) { // Attempt to update the timestamp in the correct location // We need to use the correct driveId and itemId, especially if we are updating a OneDrive Business Shared File timestamp if (dbItem.type == ItemType.file) { // Not a remote file // Where should the timestamp update be performed ? if (localModifiedTime >= itemModifiedTime) { // Log what is being done if (verboseLogging) {addLogEntry("The local item has the same hash value as the item online but with a newer local timestamp - correcting online timestamp", ["verbose"]);} // Correct timestamp uploadLastModifiedTime(dbItem, dbItem.driveId, dbItem.id, localModifiedTime.toUTC(), dbItem.eTag); } else { // Log what is being done if (verboseLogging) {addLogEntry("The local item has the same hash value as the item online but with an older local timestamp - correcting local timestamp", ["verbose"]);} // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, localFilePath, dbItem.mtime); } } else { // Remote file, remote values need to be used, we may not even have permission to change timestamp, update local file if (verboseLogging) {addLogEntry("The local item has the same hash value as the item online, however file is a OneDrive Business Shared File - correcting local timestamp", ["verbose"]);} // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, localFilePath, dbItem.mtime); } } } else { // --download-only being used if (verboseLogging) {addLogEntry("The local item has the same hash value as the item online - correcting local timestamp due to --download-only being used to ensure local file matches timestamp online", ["verbose"]);} // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, localFilePath, dbItem.mtime); } } } else { // The file has not changed if (verboseLogging) {addLogEntry("The file has not changed", ["verbose"]);} } } else { //The file is not readable - skipped addLogEntry("Skipping processing this file as it cannot be read (file permissions or file corruption): " ~ localFilePath); } } else { // The item was a file but now is a directory if (verboseLogging) {addLogEntry("The item was a file but now is a directory", ["verbose"]);} } } else { // File does not exist locally, but is in our database as a dbItem containing all the data was passed into this function // If we are in a --dry-run situation - this file may never have existed as we never downloaded it if (!dryRun) { // Not --dry-run situation if (verboseLogging) {addLogEntry("The file has been deleted locally", ["verbose"]);} // Add this to the array to handle post checking all database items databaseItemsToDeleteOnline ~= [DatabaseItemsToDeleteOnline(dbItem, localFilePath)]; } else { // We are in a --dry-run situation, file appears to have been deleted locally - this file may never have existed locally as we never downloaded it due to --dry-run // Did we 'fake create it' as part of --dry-run ? bool idsFakedMatch = false; // Check the file id - was this faked foreach (i; idsFaked) { if (i[1] == dbItem.id) { if (debugLogging) {addLogEntry("Matched faked file which is 'supposed' to exist locally but not created|renamed due to --dry-run use", ["debug"]);} if (verboseLogging) {addLogEntry("The file has not changed", ["verbose"]);} idsFakedMatch = true; } } // Check if the parent folder was faked being changed in any way .. so we need to check the parent id foreach (i; idsFaked) { if (i[1] == dbItem.parentId) { if (debugLogging) {addLogEntry("Matched faked parental directory which is 'supposed' to exist locally but not created|renamed due to --dry-run use", ["debug"]);} if (verboseLogging) {addLogEntry("The file has not changed", ["verbose"]);} idsFakedMatch = true; } } // file id or parent id of the file did not match anything we faked changing due to --dry-run if (!idsFakedMatch) { // dbItem.id did not match a 'faked' download new file creation - so this in-sync object was actually deleted locally, but we are in a --dry-run situation if (verboseLogging) {addLogEntry("The file has been deleted locally", ["verbose"]);} // Add this to the array to handle post checking all database items databaseItemsToDeleteOnline ~= [DatabaseItemsToDeleteOnline(dbItem, localFilePath)]; } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the database consistency check on this directory item void checkDirectoryDatabaseItemForConsistency(Item dbItem, string localFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // What is the source of this item data? string itemSource = "database"; // Does this item|directory still exist on disk? if (exists(localFilePath)) { // Fix https://github.com/abraunegg/onedrive/issues/1915 try { if (!isDir(localFilePath)) { if (verboseLogging) {addLogEntry("The item was a directory but now it is a file", ["verbose"]);} uploadDeletedItem(dbItem, localFilePath); uploadNewFile(localFilePath); } else { // Directory still exists locally if (verboseLogging) {addLogEntry("The directory has not changed", ["verbose"]);} // When we are using --single-directory, we use the getChildren() call to get all children of a path, meaning all children are already traversed // Thus, if we traverse the path of this directory .. we end up with double processing & log output .. which is not ideal if (!singleDirectoryScope) { // loop through the children Item[] childrenFromDatabase = itemDB.selectChildren(dbItem.driveId, dbItem.id); foreach (Item child; childrenFromDatabase) { checkDatabaseItemForConsistency(child); } // Clear DB response array childrenFromDatabase = []; } } } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, localFilePath); } } else { // Directory does not exist locally, but it is in our database as a dbItem containing all the data was passed into this function // If we are in a --dry-run situation - this directory may never have existed as we never created it if (!dryRun) { // Not --dry-run situation if (!appConfig.getValueBool("monitor")) { // Not in --monitor mode if (verboseLogging) {addLogEntry("The directory has been deleted locally", ["verbose"]);} } else { // Appropriate message as we are in --monitor mode if (verboseLogging) {addLogEntry("The directory appears to have been deleted locally .. but we are running in --monitor mode. This may have been 'moved' on the local filesystem rather than being 'deleted'", ["verbose"]);} if (debugLogging) {addLogEntry("Most likely cause - 'inotify' event was missing for whatever action was taken locally or action taken when application was stopped", ["debug"]);} } // A moved directory will be uploaded as 'new', delete the old directory and database reference // Add this to the array to handle post checking all database items databaseItemsToDeleteOnline ~= [DatabaseItemsToDeleteOnline(dbItem, localFilePath)]; } else { // We are in a --dry-run situation, directory appears to have been deleted locally - this directory may never have existed locally as we never created it due to --dry-run // Did we 'fake create it' as part of --dry-run ? bool idsFakedMatch = false; foreach (i; idsFaked) { if (i[1] == dbItem.id) { if (debugLogging) {addLogEntry("Matched faked directory which is 'supposed' to exist locally but not created|renamed due to --dry-run use", ["debug"]);} if (verboseLogging) {addLogEntry("The directory has not changed", ["verbose"]);} idsFakedMatch = true; } } if (!idsFakedMatch) { // dbItem.id did not match a 'faked' download new directory creation - so this in-sync object was actually deleted locally, but we are in a --dry-run situation if (verboseLogging) {addLogEntry("The directory has been deleted locally", ["verbose"]);} // Add this to the array to handle post checking all database items databaseItemsToDeleteOnline ~= [DatabaseItemsToDeleteOnline(dbItem, localFilePath)]; } else { // When we are using --single-directory, we use a the getChildren() call to get all children of a path, meaning all children are already traversed // Thus, if we traverse the path of this directory .. we end up with double processing & log output .. which is not ideal if (!singleDirectoryScope) { // loop through the children Item[] childrenFromDatabase = itemDB.selectChildren(dbItem.driveId, dbItem.id); foreach (Item child; childrenFromDatabase) { checkDatabaseItemForConsistency(child); } // Clear DB response array childrenFromDatabase = []; } } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Does this local path (directory or file) conform with the Microsoft Naming Restrictions? It needs to conform otherwise we cannot create the directory or upload the file. bool checkPathAgainstMicrosoftNamingRestrictions(string localFilePath, string logModifier = "item") { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Check if the given path violates certain Microsoft restrictions and limitations // Return a true|false response bool invalidPath = false; // Check path against Microsoft OneDrive restriction and limitations about Windows naming for files and folders if (!invalidPath) { if (!isValidName(localFilePath)) { // This will return false if this is not a valid name according to the OneDrive API specifications addLogEntry("Skipping " ~ logModifier ~" - invalid name (Microsoft Naming Convention): " ~ localFilePath, ["info", "notify"]); invalidPath = true; } } // Check path for bad whitespace items if (!invalidPath) { if (containsBadWhiteSpace(localFilePath)) { // This will return true if this contains a bad whitespace character addLogEntry("Skipping " ~ logModifier ~" - invalid name (Contains an invalid whitespace character): " ~ localFilePath, ["info", "notify"]); invalidPath = true; } } // Check path for HTML ASCII Codes if (!invalidPath) { if (containsASCIIHTMLCodes(localFilePath)) { // This will return true if this contains HTML ASCII Codes addLogEntry("Skipping " ~ logModifier ~" - invalid name (Contains HTML ASCII Code): " ~ localFilePath, ["info", "notify"]); invalidPath = true; } } // Validate that the path is a valid UTF-16 encoded path if (!invalidPath) { if (!isValidUTF16(localFilePath)) { // This will return true if this is a valid UTF-16 encoded path, so we are checking for 'false' as response addLogEntry("Skipping " ~ logModifier ~" - invalid name (Invalid UTF-16 encoded path): " ~ localFilePath, ["info", "notify"]); invalidPath = true; } } // Check path for ASCII Control Codes if (!invalidPath) { if (containsASCIIControlCodes(localFilePath)) { // This will return true if this contains ASCII Control Codes addLogEntry("Skipping " ~ logModifier ~" - invalid name (Contains ASCII Control Codes): " ~ localFilePath, ["info", "notify"]); invalidPath = true; } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return if this is a valid path return invalidPath; } // Does this local path (directory or file) get excluded from any operation based on any client side filtering rules? bool checkPathAgainstClientSideFiltering(string localFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Check the path against client side filtering rules // - check_nosync // - skip_dotfiles // - skip_symlinks // - skip_file // - skip_dir // - sync_list // - skip_size // Return a true|false response bool clientSideRuleExcludesPath = false; // Reset global syncListDirExcluded syncListDirExcluded = false; // does the path exist? if (!exists(localFilePath)) { // path does not exist - we cant review any client side rules on something that does not exist locally return clientSideRuleExcludesPath; } // - check_nosync if (!clientSideRuleExcludesPath) { // Do we need to check for .nosync? Only if --check-for-nosync was passed in if (appConfig.getValueBool("check_nosync")) { if (exists(localFilePath ~ "/.nosync")) { if (verboseLogging) {addLogEntry("Skipping item - .nosync found & --check-for-nosync enabled: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } } } // - skip_dotfiles if (!clientSideRuleExcludesPath) { // Do we need to check skip dot files if configured if (appConfig.getValueBool("skip_dotfiles")) { if (isDotFile(localFilePath)) { if (!syncListConfigured) { // 'sync_list' is not in use if (verboseLogging) {addLogEntry("Skipping item - .file or .folder: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } else { // 'sync_list' is in use - potentially skipping .file or .folder but it may be included via 'sync_list' if (verboseLogging) {addLogEntry("Potentially skipping item - .file or .folder (sync_list inclusion check to be done): " ~ localFilePath, ["verbose"]);} } } } } // - skip_symlinks if (!clientSideRuleExcludesPath) { // Is the path a symbolic link if (isSymlink(localFilePath)) { // if config says so we skip all symlinked items if (appConfig.getValueBool("skip_symlinks")) { if (verboseLogging) {addLogEntry("Skipping item - skip symbolic links configured: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } // skip unexisting symbolic links else if (!exists(readLink(localFilePath))) { // reading the symbolic link failed - is the link a relative symbolic link // drwxrwxr-x. 2 alex alex 46 May 30 09:16 . // drwxrwxr-x. 3 alex alex 35 May 30 09:14 .. // lrwxrwxrwx. 1 alex alex 61 May 30 09:16 absolute.txt -> /home/alex/OneDrivePersonal/link_tests/intercambio/prueba.txt // lrwxrwxrwx. 1 alex alex 13 May 30 09:16 relative.txt -> ../prueba.txt // // absolute links will be able to be read, but 'relative' links will fail, because they cannot be read based on the current working directory 'sync_dir' string currentSyncDir = getcwd(); string fullLinkPath = buildNormalizedPath(absolutePath(localFilePath)); string fileName = baseName(fullLinkPath); string parentLinkPath = dirName(fullLinkPath); // test if this is a 'relative' symbolic link chdir(parentLinkPath); auto relativeLink = readLink(fileName); auto relativeLinkTest = exists(readLink(fileName)); // reset back to our 'sync_dir' chdir(currentSyncDir); // results if (relativeLinkTest) { if (debugLogging) {addLogEntry("Not skipping item - symbolic link is a 'relative link' to target ('" ~ relativeLink ~ "') which can be supported: " ~ localFilePath, ["debug"]);} } else { addLogEntry("Skipping item - invalid symbolic link: "~ localFilePath, ["info", "notify"]); clientSideRuleExcludesPath = true; } } } } // Is this item excluded by user configuration of skip_dir or skip_file? if (!clientSideRuleExcludesPath) { if (localFilePath != ".") { // skip_dir handling if (isDir(localFilePath)) { if (debugLogging) {addLogEntry("Checking local path: " ~ localFilePath, ["debug"]);} // Only check path if config is != "" if (appConfig.getValueString("skip_dir") != "") { // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched if (selectiveSync.isDirNameExcluded(localFilePath.strip('.'))) { if (verboseLogging) {addLogEntry("Skipping path - excluded by skip_dir config: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } } } // skip_file handling if (isFile(localFilePath)) { if (debugLogging) {addLogEntry("Checking file: " ~ localFilePath, ["debug"]);} // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched if (selectiveSync.isFileNameExcluded(localFilePath.strip('.'))) { if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_file config: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } } } } // Is this item excluded by user configuration of sync_list? if (!clientSideRuleExcludesPath) { if (localFilePath != ".") { if (syncListConfigured) { // sync_list configured and in use if (selectiveSync.isPathExcludedViaSyncList(localFilePath)) { if ((isFile(localFilePath)) && (appConfig.getValueBool("sync_root_files")) && (rootName(localFilePath.strip('.').strip('/')) == "")) { if (debugLogging) {addLogEntry("Not skipping path due to sync_root_files inclusion: " ~ localFilePath, ["debug"]);} } else { if (exists(appConfig.syncListFilePath)){ // skipped most likely due to inclusion in sync_list // is this path a file or directory? if (isFile(localFilePath)) { // file if (verboseLogging) {addLogEntry("Skipping file - excluded by sync_list config: " ~ localFilePath, ["verbose"]);} } else { // directory if (verboseLogging) {addLogEntry("Skipping path - excluded by sync_list config: " ~ localFilePath, ["verbose"]);} // update syncListDirExcluded syncListDirExcluded = true; } // flag as excluded clientSideRuleExcludesPath = true; } else { // skipped for some other reason if (verboseLogging) {addLogEntry("Skipping path - excluded by user config: " ~ localFilePath, ["verbose"]);} clientSideRuleExcludesPath = true; } } } } } } // Check if this is excluded by a user set maximum filesize to upload if (!clientSideRuleExcludesPath) { if (isFile(localFilePath)) { if (fileSizeLimit != 0) { // Get the file size long thisFileSize = getSize(localFilePath); if (thisFileSize >= fileSizeLimit) { if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_size config: " ~ localFilePath ~ " (" ~ to!string(thisFileSize/2^^20) ~ " MB)", ["verbose"]);} clientSideRuleExcludesPath = true; } } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return if path is excluded return clientSideRuleExcludesPath; } // Does this JSON item (as received from OneDrive API) get excluded from any operation based on any client side filtering rules? // This function is used when we are fetching objects from the OneDrive API using a /children query to help speed up what object we query or when checking OneDrive Business Shared Files bool checkJSONAgainstClientSideFiltering(JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Debug what JSON we are evaluating against Client Side Filtering Rules if (debugLogging) {addLogEntry("Checking this JSON against Client Side Filtering Rules: " ~ sanitiseJSONItem(onedriveJSONItem), ["debug"]);} // Function flag bool clientSideRuleExcludesPath = false; // Check the path against client side filtering rules // - check_nosync (MISSING) // - skip_dotfiles (MISSING) // - skip_symlinks (MISSING) // - skip_dir // - skip_file // - sync_list // - skip_size // Return a true|false response // Use the JSON elements rather than computing a DB struct via makeItem() string thisItemId = onedriveJSONItem["id"].str; string thisItemDriveId = onedriveJSONItem["parentReference"]["driveId"].str; string thisItemParentId = onedriveJSONItem["parentReference"]["id"].str; string thisItemName = onedriveJSONItem["name"].str; // Issue #3336 - Convert thisItemDriveId to lowercase before any test if (appConfig.accountType == "personal") { thisItemDriveId = transformToLowerCase(thisItemDriveId); } // Is this parent is in the database bool parentInDatabase = false; string calculatedParentalPath; // Calculate if the Parent Item is in the database so that this flag can be reused parentInDatabase = itemDB.idInLocalDatabase(thisItemDriveId, thisItemParentId); if (parentInDatabase) { // Calculate this items path based on database entries if (debugLogging) {addLogEntry("Parent path details are in DB - computing 'calculatedParentalPath' using computeItemPath()", ["debug"]);} calculatedParentalPath = computeItemPath(thisItemDriveId, thisItemParentId); if (debugLogging) {addLogEntry("Resulting 'calculatedParentalPath' using computeItemPath() = " ~ calculatedParentalPath, ["debug"]);} } // Check if this is excluded by config option: skip_dir if (!clientSideRuleExcludesPath) { // Is the item a folder? if (isItemFolder(onedriveJSONItem)) { // Only check path if config is != "" if (!appConfig.getValueString("skip_dir").empty) { // work out the 'snippet' path where this folder would be created string simplePathToCheck = ""; string complexPathToCheck = ""; string matchDisplay = ""; if (hasParentReference(onedriveJSONItem)) { // we need to workout the FULL path for this item // simple path if (("name" in onedriveJSONItem["parentReference"]) != null) { simplePathToCheck = onedriveJSONItem["parentReference"]["name"].str ~ "/" ~ onedriveJSONItem["name"].str; } else { simplePathToCheck = onedriveJSONItem["name"].str; } if (debugLogging) {addLogEntry("skip_dir path to check (simple): " ~ simplePathToCheck, ["debug"]);} // complex path calculation if (parentInDatabase) { // build up complexPathToCheck based on database data complexPathToCheck = calculatedParentalPath ~ "/" ~ thisItemName; if (debugLogging) {addLogEntry("Updated 'complexPathToCheck' to '"~ complexPathToCheck ~"' for 'skip_dir' validation to determine if this directory should be excluded.", ["debug"]);} } else { if (debugLogging) {addLogEntry("Parent details not in database - unable to compute complex path to check using database data", ["debug"]);} // use onedriveJSONItem["parentReference"]["path"].str string selfBuiltPath = onedriveJSONItem["parentReference"]["path"].str ~ "/" ~ onedriveJSONItem["name"].str; // Check for ':' and split if present auto splitIndex = selfBuiltPath.indexOf(":"); if (splitIndex != -1) { // Keep only the part after ':' selfBuiltPath = selfBuiltPath[splitIndex + 1 .. $]; } // set complexPathToCheck to selfBuiltPath and be compatible with computeItemPath() output complexPathToCheck = "." ~ selfBuiltPath; } // were we able to compute a complexPathToCheck ? if (!complexPathToCheck.empty) { // complexPathToCheck must at least start with './' to ensure logging output consistency but also for pattern matching consistency if (!startsWith(complexPathToCheck, "./")) { complexPathToCheck = "./" ~ complexPathToCheck; } // log the complex path to check if (debugLogging) {addLogEntry("skip_dir path to check (complex): " ~ complexPathToCheck, ["debug"]);} } } else { simplePathToCheck = onedriveJSONItem["name"].str; } // If 'simplePathToCheck' or 'complexPathToCheck' is of the following format: root:/folder // then isDirNameExcluded matching will not work if (simplePathToCheck.canFind(":")) { if (debugLogging) {addLogEntry("Updating simplePathToCheck to remove 'root:'", ["debug"]);} simplePathToCheck = processPathToRemoveRootReference(simplePathToCheck); } if (complexPathToCheck.canFind(":")) { if (debugLogging) {addLogEntry("Updating complexPathToCheck to remove 'root:'", ["debug"]);} complexPathToCheck = processPathToRemoveRootReference(complexPathToCheck); } // OK .. what checks are we doing? if ((!simplePathToCheck.empty) && (complexPathToCheck.empty)) { // just a simple check if (debugLogging) {addLogEntry("Performing a simple check only", ["debug"]);} clientSideRuleExcludesPath = selectiveSync.isDirNameExcluded(simplePathToCheck); } else { // simple and complex if (debugLogging) {addLogEntry("Performing a simple then complex path match if required", ["debug"]);} // simple first if (debugLogging) {addLogEntry("Performing a simple check first", ["debug"]);} clientSideRuleExcludesPath = selectiveSync.isDirNameExcluded(simplePathToCheck); if (!clientSideRuleExcludesPath) { if (debugLogging) {addLogEntry("Simple match was false, attempting complex match", ["debug"]);} // simple didnt match, perform a complex check clientSideRuleExcludesPath = selectiveSync.isDirNameExcluded(complexPathToCheck); } } // End Result if (debugLogging) {addLogEntry("skip_dir exclude result (directory based): " ~ to!string(clientSideRuleExcludesPath), ["debug"]);} if (clientSideRuleExcludesPath) { // what path should be displayed if we are excluding if (!complexPathToCheck.empty) { // try and always use the complex path as it is more complete for application output matchDisplay = complexPathToCheck; } else { matchDisplay = simplePathToCheck; } // This path should be skipped if (verboseLogging) {addLogEntry("Skipping path - excluded by skip_dir config: " ~ matchDisplay, ["verbose"]);} } } } // Is the item a file? // We need to check to see if this files path is excluded as well if (isItemFile(onedriveJSONItem)) { // Only check path if config is != "" if (!appConfig.getValueString("skip_dir").empty) { // variable to check the file path against skip_dir string pathToCheck; if (parentInDatabase) { // Parent is in the database - use those details to compute this files parental path pathToCheck = calculatedParentalPath; if (debugLogging) {addLogEntry("Updated 'pathToCheck' to '"~ pathToCheck ~"' for 'skip_dir' validation to determine if this file should be excluded.", ["debug"]);} } else { // Parent is not in the database .. compute manually if (hasParentReference(onedriveJSONItem)) { // use onedriveJSONItem["parentReference"]["path"].str string selfBuiltPath = onedriveJSONItem["parentReference"]["path"].str; if (debugLogging) {addLogEntry("Initial file based selfBuiltPath = " ~ selfBuiltPath, ["debug"]);} // Check for ':' and split if present within 'selfBuiltPath' auto splitIndex = selfBuiltPath.indexOf(":"); if (splitIndex != -1) { // Keep only the part after ':' string pathAfterSplit = selfBuiltPath[splitIndex + 1 .. $]; if (debugLogging) {addLogEntry("pathAfterSplit = " ~ pathAfterSplit, ["debug"]);} if (pathAfterSplit.empty) { // Empty path, thus this is most likely a file in the account root selfBuiltPath = "/"; } else { // There is a path after the split, this is the path we are interested in // However ... in a Shared Folder scenario, this path now is the absolute path on the remote driveID .. could be problematic selfBuiltPath = pathAfterSplit; } // Result after split if (debugLogging) {addLogEntry("selfBuiltPath after splitting at : = " ~ selfBuiltPath, ["debug"]);} } // Update file path to check against 'skip_dir' using the self built details pathToCheck = selfBuiltPath; if (debugLogging) {addLogEntry("Updated (manual computation) 'pathToCheck' to '"~ pathToCheck ~"' for 'skip_dir' validation to determine if this file should be excluded.", ["debug"]);} } } // Build the consistent path for logging output string logItemPath = ensureStartsWithDotSlash(buildNormalizedPath(pathToCheck ~ "/" ~ onedriveJSONItem["name"].str)); // Perform the skip_dir check for file path if (debugLogging) {addLogEntry("skip_dir path to check (file based): " ~ to!string(pathToCheck), ["debug"]);} clientSideRuleExcludesPath = selectiveSync.isDirNameExcluded(pathToCheck); // 'skip_dir' result if (debugLogging) {addLogEntry("skip_dir exclude result (file based): " ~ to!string(clientSideRuleExcludesPath), ["debug"]);} if (clientSideRuleExcludesPath) { // this files path should be skipped if (verboseLogging) {addLogEntry("Skipping file - file path is excluded by skip_dir config: " ~ logItemPath, ["verbose"]);} } } } } // Check if this is excluded by config option: skip_file if (!clientSideRuleExcludesPath) { // is the item a file ? if (isFileItem(onedriveJSONItem)) { // JSON item is a file // skip_file can contain 4 types of entries: // - wildcard - *.txt // - text + wildcard - name*.txt // - full path + combination of any above two - /path/name*.txt // - full path to file - /path/to/file.txt string exclusionTestPath = ""; // is the parent id in the database? if (parentInDatabase) { // parent id is in the database, so we can try and calculate the full file path string newItemPath = ""; // Compute this item path & need the full path for this file newItemPath = calculatedParentalPath ~ "/" ~ thisItemName; // The path that needs to be checked needs to include the '/' // This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched // However, as 'path' used throughout, use a temp variable with this modification so that we use the temp variable for exclusion checks if (!startsWith(newItemPath, "/")){ // Add '/' to the path exclusionTestPath = '/' ~ newItemPath; } // Normalise the path to ensure any initial sequence of '/./././' or similar is normalised exclusionTestPath = buildNormalizedPath(exclusionTestPath); // what are we checking if (debugLogging) {addLogEntry("Updated 'newItemPath' to '"~ newItemPath ~"' for 'skip_file' validation to determine if this file should be excluded.", ["debug"]);} } else { // parent not in database, we can only check using this JSON item's name if (!startsWith(thisItemName, "/")){ // Add '/' to the path exclusionTestPath = '/' ~ thisItemName; } // what are we checking if (debugLogging) {addLogEntry("skip_file item to check (file name only - parent path not in database): " ~ exclusionTestPath, ["debug"]);} } // Perform the 'skip_file' evaluation clientSideRuleExcludesPath = selectiveSync.isFileNameExcluded(exclusionTestPath); if (debugLogging) {addLogEntry("skip_file evaluation result: " ~ to!string(clientSideRuleExcludesPath), ["debug"]);} if (clientSideRuleExcludesPath) { // This path should be skipped if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_file config: " ~ exclusionTestPath, ["verbose"]);} } } } // Check if this is included or excluded by use of sync_list if (!clientSideRuleExcludesPath) { // No need to try and process something against a sync_list if it has been configured if (syncListConfigured) { // Compute the item path if empty - as to check sync_list we need an actual path to check // What is the path of the new item string newItemPath; // Is the parent in the database? If not, we cannot compute the full path based on the database entries // In a --resync scenario - the database is empty if (parentInDatabase) { // Calculate this items path based on database entries newItemPath = calculatedParentalPath ~ "/" ~ thisItemName; if (debugLogging) {addLogEntry("Updated 'newItemPath' to '"~ newItemPath ~"' for 'sync_list' validation to determine if this directory should be included.", ["debug"]);} } else { // Parent is not in the database .. we need to compute it .. why ???? if (appConfig.getValueBool("resync")) { if (debugLogging) {addLogEntry("Parent NOT in DB .. we need to manually compute this path due to --resync being used", ["debug"]);} } else { if (debugLogging) {addLogEntry("Parent NOT in DB .. we need to manually compute this path .......", ["debug"]);} } // gather the applicable path details if (("path" in onedriveJSONItem["parentReference"]) != null) { // If there is a parent reference path, try and use it string selfBuiltPath = onedriveJSONItem["parentReference"]["path"].str ~ "/" ~ onedriveJSONItem["name"].str; // Check for ':' and split if present string[] splitPaths; auto splitIndex = selfBuiltPath.indexOf(":"); if (splitIndex != -1) { // Keep only the part after ':' splitPaths = selfBuiltPath.split(":"); selfBuiltPath = splitPaths[1]; } // Debug output what the self-built path currently is if (debugLogging) {addLogEntry(" - selfBuiltPath currently calculated as: " ~ selfBuiltPath, ["debug"]);} // Issue #2731 // Get the remoteDriveId from JSON record string remoteDriveId = onedriveJSONItem["parentReference"]["driveId"].str; // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { remoteDriveId = transformToLowerCase(remoteDriveId); } // Is this potentially a shared folder? This is the only reliable way to determine this ... if (remoteDriveId != appConfig.defaultDriveId) { // Yes this JSON is from a Shared Folder // Query the database for the 'remote' folder details from the database if (debugLogging) {addLogEntry("Query database for this 'remoteDriveId' record: " ~ to!string(remoteDriveId), ["debug"]);} Item remoteItem; itemDB.selectByRemoteDriveId(remoteDriveId, remoteItem); if (debugLogging) {addLogEntry("Query returned result (itemDB.selectByRemoteDriveId): " ~ to!string(remoteItem), ["debug"]);} // Shared Folders present a unique challenge to determine what path needs to be used, especially in a --resync scenario where there are near zero records available to use computeItemPath() // Update the path that will be used to check 'sync_list' with the 'name' of the remoteDriveId database record // Issue #3331 // Avoid duplicating the shared folder root name if already present if (!selfBuiltPath.startsWith("/" ~ remoteItem.name ~ "/")) { selfBuiltPath = remoteItem.name ~ selfBuiltPath; if (debugLogging) {addLogEntry("selfBuiltPath after 'Shared Folder' DB details update = " ~ to!string(selfBuiltPath), ["debug"]);} } else { if (debugLogging) {addLogEntry("Shared Folder name already present in path; no update needed to selfBuiltPath", ["debug"]);} } } // Issue #2740 // If selfBuiltPath is containing any sort of URL encoding, due to special characters (spaces, umlaut, or any other character that is HTML encoded, this specific path now needs to be HTML decoded // Does the path contain HTML encoding? if (containsURLEncodedItems(selfBuiltPath)) { // decode it if (debugLogging) {addLogEntry("selfBuiltPath for sync_list check needs decoding: " ~ selfBuiltPath, ["debug"]);} try { // try and decode selfBuiltPath newItemPath = decodeComponent(selfBuiltPath); } catch (URIException exception) { // why? if (verboseLogging) { addLogEntry("ERROR: Unable to URL Decode path: " ~ exception.msg, ["verbose"]); addLogEntry("ERROR: To resolve, rename this item online: " ~ selfBuiltPath, ["verbose"]); } // have to use as-is due to decode error newItemPath = selfBuiltPath; } } else { // use as-is newItemPath = selfBuiltPath; } // The final format of newItemPath when self building needs to be the same as newItemPath when computed using computeItemPath .. this is handled later below if (debugLogging) {addLogEntry("newItemPath as manually computed by selfBuiltPath process = " ~ to!string(selfBuiltPath), ["debug"]);} } else { // no parent reference path available in provided JSON newItemPath = thisItemName; } } // The 'newItemPath' needs to be updated to ensure it is in the right format // Regardless of built from DB or computed it needs to be in this format: // ./path/path/ etc // This then makes the path output with 'sync_list' consistent, and, more importantly consistent for 'sync_list' evaluations newItemPath = ensureStartsWithDotSlash(newItemPath); // Check for HTML entities (e.g., '%20' for space) in newItemPath if (containsURLEncodedItems(newItemPath)) { if (verboseLogging) { addLogEntry("CAUTION: The JSON element transmitted by the Microsoft OneDrive API includes HTML URL encoded items, which may complicate pattern matching and potentially lead to synchronisation problems for this item.", ["verbose"]); addLogEntry("WORKAROUND: An alternative solution could be to change the name of this item through the online platform: " ~ newItemPath, ["verbose"]); addLogEntry("See: https://github.com/OneDrive/onedrive-api-docs/issues/1765 for further details", ["verbose"]); } } // What path are we checking against sync_list? if (debugLogging) {addLogEntry("Path to check against 'sync_list' entries: " ~ newItemPath, ["debug"]);} // Unfortunately there is no avoiding this call to check if the path is excluded|included via sync_list if (selectiveSync.isPathExcludedViaSyncList(newItemPath)) { // selective sync advised to skip, however is this a file and are we configured to upload / download files in the root? if ((isItemFile(onedriveJSONItem)) && (appConfig.getValueBool("sync_root_files")) && (rootName(newItemPath) == "") ) { // This is a file // We are configured to sync all files in the root // This is a file in the logical root clientSideRuleExcludesPath = false; } else { // Path is unwanted, flag to exclude clientSideRuleExcludesPath = true; // Has this itemId already been flagged as being skipped? if (!syncListSkippedParentIds.canFind(thisItemId)) { if (isItemFolder(onedriveJSONItem)) { // Detail we are skipping this JSON data from online if (verboseLogging) {addLogEntry("Skipping path - excluded by sync_list config: " ~ newItemPath, ["verbose"]);} // Add this folder id to the elements we have already detailed we are skipping, so we do no output this again syncListSkippedParentIds ~= thisItemId; } } // Is this is a 'add shortcut to onedrive' link? if (isItemRemote(onedriveJSONItem)) { // Detail we are skipping this JSON data from online if (verboseLogging) {addLogEntry("Skipping Shared Folder Link - excluded by sync_list config: " ~ newItemPath, ["verbose"]);} // Add this folder id to the elements we have already detailed we are skipping, so we do no output this again syncListSkippedParentIds ~= thisItemId; } } } else { // Is this a file or directory? if (isItemFile(onedriveJSONItem)) { // File included due to 'sync_list' match if (verboseLogging) {addLogEntry("Including file - included by sync_list config: " ~ newItemPath, ["verbose"]);} // Is the parent item in the database? if (!parentInDatabase) { // Parental database structure needs to be created string newParentalPath = dirName(newItemPath); // Log that this parental structure needs to be created if (verboseLogging) {addLogEntry("Parental Path structure needs to be created to support included file: " ~ newParentalPath, ["verbose"]);} // Recursively, stepping backward from 'thisItemParentId', query online, save entry to DB and create the local path structure createLocalPathStructure(onedriveJSONItem, newParentalPath); // If this is --dry-run if (dryRun) { // we dont create the directory, but we need to track that we 'faked it' idsFaked ~= [onedriveJSONItem["parentReference"]["driveId"].str, onedriveJSONItem["parentReference"]["id"].str]; } } } else { // Directory included due to 'sync_list' match if (verboseLogging) {addLogEntry("Including path - included by sync_list config: " ~ newItemPath, ["verbose"]);} // So that this path is in the DB, we need to add onedriveJSONItem to the DB so that this record can be used to build paths if required if (parentInDatabase) { // Parent is in DB .. is this a 'new' object or an 'existing' object? // Issue #3501 - If an online name name is done, the item needs to be 'renamed' via applyPotentiallyChangedItem() later // Only save to the database at this point, if this JSON 'id' is not already in the database to allow applyPotentiallyChangedItem() to operate as expected Item tempDBItem; itemDB.selectById(onedriveJSONItem["parentReference"]["driveId"].str, onedriveJSONItem["id"].str, tempDBItem); // Was a valid DB response returned if (tempDBItem.driveId.empty) { // No .. so this is a new item // Save this JSON now saveItem(onedriveJSONItem); } } } } } } // Check if this is excluded by a user set maximum filesize to download if (!clientSideRuleExcludesPath) { if (isItemFile(onedriveJSONItem)) { if (fileSizeLimit != 0) { if (onedriveJSONItem["size"].integer >= fileSizeLimit) { if (verboseLogging) {addLogEntry("Skipping file - excluded by skip_size config: " ~ thisItemName ~ " (" ~ to!string(onedriveJSONItem["size"].integer/2^^20) ~ " MB)", ["verbose"]);} clientSideRuleExcludesPath = true; } } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return if path is excluded return clientSideRuleExcludesPath; } // Ensure the path passed in, is in the correct format to use when evaluating 'sync_list' rules string ensureStartsWithDotSlash(string inputPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Check if the path starts with './' if (inputPath.startsWith("./")) { return inputPath; // No modification needed } // Check if the path starts with '/' or does not start with '.' at all if (inputPath.startsWith("/")) { return "." ~ inputPath; // Prepend '.' to ensure it starts with './' } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // If the path starts with any other character or is missing './', add './' return "./" ~ inputPath; } // When using 'sync_list' if a file is to be included, ensure that the path that the file resides in, is available locally and in the database, and the path exists locally void createLocalPathStructure(JSONValue onedriveJSONItem, string newLocalParentalPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables bool parentInDatabase; JSONValue onlinePathData; OneDriveApi onlinePathOneDriveApiInstance; onlinePathOneDriveApiInstance = new OneDriveApi(appConfig); onlinePathOneDriveApiInstance.initialise(); string thisItemDriveId; string thisItemParentId; // Log what we received to analyse if (debugLogging) { addLogEntry("createLocalPathStructure input onedriveJSONItem: " ~ to!string(onedriveJSONItem), ["debug"]); addLogEntry("createLocalPathStructure input newLocalParentalPath: " ~ newLocalParentalPath, ["debug"]); } // Configure these variables based on the JSON input thisItemDriveId = onedriveJSONItem["parentReference"]["driveId"].str; // OneDrive Personal JSON responses are in-consistent with not having 'id' available if (hasParentReferenceId(onedriveJSONItem)) { // Use the parent reference id thisItemParentId = onedriveJSONItem["parentReference"]["id"].str; } // To continue, thisItemDriveId and thisItemParentId must not be empty if ((thisItemDriveId != "") && (thisItemParentId != "")) { // Calculate if the Parent Item is in the database so that it can be re-used parentInDatabase = itemDB.idInLocalDatabase(thisItemDriveId, thisItemParentId); // Is the parent in the database? if (!parentInDatabase) { // Get data from online for this driveId and JSON item parent .. so we have the parent details if (debugLogging) {addLogEntry("createLocalPathStructure parent is not in database, fetching parental details from online", ["debug"]);} try { onlinePathData = onlinePathOneDriveApiInstance.getPathDetailsById(thisItemDriveId, thisItemParentId); } catch (OneDriveException exception) { // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // There needs to be a valid JSON to process if (onlinePathData.type() == JSONType.object) { // Does this JSON match the root name of a shared folder we may be trying to match? if (sharedFolderDeltaGeneration) { if (currentSharedFolderName == onlinePathData["name"].str) { if (debugLogging) {addLogEntry("createLocalPathStructure parent matches the current shared folder name, creating applicable shared folder database records", ["debug"]);} // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(onlinePathData); } } // Configure the grandparent items string grandparentItemDriveId; string grandparentItemParentId; grandparentItemDriveId = onlinePathData["parentReference"]["driveId"].str; // OneDrive Personal JSON responses are in-consistent with not having 'id' available if (hasParentReferenceId(onlinePathData)) { // Use the parent reference id grandparentItemParentId = onlinePathData["parentReference"]["id"].str; } else { // Testing evidence shows that for Personal accounts, use the 'id' itself grandparentItemParentId = onlinePathData["id"].str; } // Is this item's grandparent data in the database? if (!itemDB.idInLocalDatabase(grandparentItemDriveId, grandparentItemParentId)) { // grandparent needs to be added createLocalPathStructure(onlinePathData, dirName(newLocalParentalPath)); } // If this is --dry-run if (dryRun) { // we dont create the directory, but we need to track that we 'faked it' idsFaked ~= [grandparentItemDriveId, grandparentItemParentId]; } // Does the parental path exist locally? if (!exists(newLocalParentalPath)) { // the required path does not exist locally - logging is done in handleLocalDirectoryCreation // create a db item record for the online data Item newDatabaseItem = makeItem(onlinePathData); // create the path locally, save the data to the database post path creation handleLocalDirectoryCreation(newDatabaseItem, newLocalParentalPath, onlinePathData); } else { // parent path exists locally, save the data to the database saveItem(onlinePathData); } } else { // No valid JSON was responded with - unable to create local path structure addLogEntry("Unable to create the local path structure as the Microsoft OneDrive API returned an invalid response"); } } else { if (debugLogging) {addLogEntry("createLocalPathStructure parent is in the database", ["debug"]);} } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory onlinePathOneDriveApiInstance.releaseCurlEngine(); onlinePathOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process the list of local changes to upload to OneDrive void processChangedLocalItemsToUpload() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Each element in this array 'databaseItemsWhereContentHasChanged' is an Database Item ID that has been modified locally size_t batchSize = to!int(appConfig.getValueLong("threads")); long batchCount = (databaseItemsWhereContentHasChanged.length + batchSize - 1) / batchSize; long batchesProcessed = 0; // For each batch of files to upload, upload the changed data to OneDrive foreach (chunk; databaseItemsWhereContentHasChanged.chunks(batchSize)) { processChangedLocalItemsToUploadInParallel(chunk); } // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process all the changed local items in parallel void processChangedLocalItemsToUploadInParallel(string[3][] array) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function received an array of string items to upload, the number of elements based on appConfig.getValueLong("threads") foreach (i, localItemDetails; processPool.parallel(array)) { if (debugLogging) {addLogEntry("Upload Thread " ~ to!string(i) ~ " Starting: " ~ to!string(Clock.currTime()), ["debug"]);} uploadChangedLocalFileToOneDrive(localItemDetails); if (debugLogging) {addLogEntry("Upload Thread " ~ to!string(i) ~ " Finished: " ~ to!string(Clock.currTime()), ["debug"]);} } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Upload changed local files to OneDrive in parallel void uploadChangedLocalFileToOneDrive(string[3] localItemDetails) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // These are the details of the item we need to upload string changedItemDriveId = localItemDetails[0]; string changedItemId = localItemDetails[1]; string localFilePath = localItemDetails[2]; // Log the path that was modified if (debugLogging) {addLogEntry("uploadChangedLocalFileToOneDrive: " ~ localFilePath, ["debug"]);} // How much space is remaining on OneDrive long remainingFreeSpace; // Did the upload fail? bool uploadFailed = false; // Did we skip due to exceeding maximum allowed size? bool skippedMaxSize = false; // Did we skip to an exception error? bool skippedExceptionError = false; // Flag for if space is available online bool spaceAvailableOnline = false; // Capture what time this upload started SysTime uploadStartTime = Clock.currTime(); // When we are uploading OneDrive Business Shared Files, we need to be targeting the right driveId and itemId string targetDriveId; string targetItemId; // Unfortunately, we cant store an array of Item's ... so we have to re-query the DB again - unavoidable extra processing here // This is because the Item[] has no other functions to allow is to parallel process those elements, so we have to use a string array as input to this function Item dbItem; itemDB.selectById(changedItemDriveId, changedItemId, dbItem); // Was a valid DB response returned if (!dbItem.driveId.empty) { // Is this a remote driveId target based on the database response? if ((dbItem.type == ItemType.remote) && (dbItem.remoteType == ItemType.file)) { // This is a remote file targetDriveId = dbItem.remoteDriveId; targetItemId = dbItem.remoteId; // we are going to make the assumption here that as this is a OneDrive Business Shared File, that there is space available spaceAvailableOnline = true; } else { // This is not a remote file targetDriveId = dbItem.driveId; targetItemId = dbItem.id; } } else { // No valid DB response was provided if (debugLogging) { string logMessage = format("No valid DB response was provided when searching for '%s' and '%s'", changedItemDriveId, changedItemId); addLogEntry(logMessage, ["debug"]); // Fetch the online data again for this file addLogEntry("Fetching latest online details for this item due to zero DB data available", ["debug"]); } OneDriveApi checkFileOneDriveApiInstance; JSONValue fileDetailsFromOneDrive; // Create a new API Instance for this thread and initialise it checkFileOneDriveApiInstance = new OneDriveApi(appConfig); checkFileOneDriveApiInstance.initialise(); // Try and get the absolute latest object details from online to potentially build a DB record we can use try { fileDetailsFromOneDrive = checkFileOneDriveApiInstance.getPathDetailsById(changedItemDriveId, changedItemId); } catch (OneDriveException exception) { // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory checkFileOneDriveApiInstance.releaseCurlEngine(); checkFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Turn 'fileDetailsFromOneDrive' into a DB item if (fileDetailsFromOneDrive.type() == JSONType.object) { // Yes if (debugLogging) {addLogEntry("Creating DB item from online API response: " ~ to!string(fileDetailsFromOneDrive), ["debug"]);} dbItem = makeItem(fileDetailsFromOneDrive); } else { // No addLogEntry("Unable to upload this modified file at this point in time: " ~ localFilePath); return; } } // Are we in an --upload-only & --remove-source-files scenario? // - In this scenario, and even more so in a --resync scenario when using these options, there is potentially 100% zero database entry for the modified file we are uploading // This will be in the logs when we are in this scenario: // Skipping adding to database as --upload-only & --remove-source-files configured if ((uploadOnly) && (localDeleteAfterUpload)) { // We are in the potential scenario where 'targetDriveId' and 'targetItemId' are still an empty value(s) // Check targetDriveId if (targetDriveId.empty) { if (debugLogging) { string logMessage = format("Updating 'targetDriveId' to '%s' due to --upload-only and --remove-source-files being used", changedItemDriveId); addLogEntry(logMessage, ["debug"]); } // set the value targetDriveId = changedItemDriveId; } // Check targetItemId if (targetItemId.empty) { if (debugLogging) { string logMessage = format("Updating 'targetItemId' to '%s' due to --upload-only and --remove-source-files being used", changedItemId); addLogEntry(logMessage, ["debug"]); } // set the value targetItemId = changedItemId; } } // Fetch the details from cachedOnlineDriveData if this is available // - cachedOnlineDriveData.quotaRestricted; // - cachedOnlineDriveData.quotaAvailable; // - cachedOnlineDriveData.quotaRemaining; DriveDetailsCache cachedOnlineDriveData; // Make sure that parentItem.driveId is in our driveIDs array to use when checking if item is in database // Keep the DriveDetailsCache array with unique entries only if (!canFindDriveId(targetDriveId, cachedOnlineDriveData)) { // Add this driveId to the drive cache, which then also sets for the defaultDriveId: // - quotaRestricted; // - quotaAvailable; // - quotaRemaining; addOrUpdateOneDriveOnlineDetails(targetDriveId); } // Query the details using the correct 'targetDriveId' for this modified file to be uploaded cachedOnlineDriveData = getDriveDetails(targetDriveId); // Configure 'remainingFreeSpace' based on the 'targetDriveId' remainingFreeSpace = cachedOnlineDriveData.quotaRemaining; // Get the file size from the actual file long thisFileSizeLocal = getSize(localFilePath); // Get the file size from the DB data, if DB data was returned, otherwise we have zero size value from the DB long thisFileSizeFromDB; if (!dbItem.size.empty) { thisFileSizeFromDB = to!long(dbItem.size); } else { thisFileSizeFromDB = 0; } // 'remainingFreeSpace' online includes the current file online // We need to remove the online file (add back the existing file size) then take away the new local file size to get a new approximate value long calculatedSpaceOnlinePostUpload = (remainingFreeSpace + thisFileSizeFromDB) - thisFileSizeLocal; // Based on what we know, for this thread - can we safely upload this modified local file? if (debugLogging) { string estimatedMessage = format("This Thread (Upload Changed File) Estimated Free Space Online (%s): ", targetDriveId); addLogEntry(estimatedMessage ~ to!string(remainingFreeSpace), ["debug"]); addLogEntry("This Thread (Upload Changed File) Calculated Free Space Online Post Upload: " ~ to!string(calculatedSpaceOnlinePostUpload), ["debug"]); } // Is there quota available for the given drive where we are uploading to? // If 'personal' accounts, if driveId == defaultDriveId, then we will have quota data - cachedOnlineDriveData.quotaRemaining will be updated so it can be reused // If 'personal' accounts, if driveId != defaultDriveId, then we will not have quota data - cachedOnlineDriveData.quotaRestricted will be set as true // If 'business' accounts, if driveId == defaultDriveId, then we will potentially have quota data - cachedOnlineDriveData.quotaRemaining will be updated so it can be reused // If 'business' accounts, if driveId != defaultDriveId, then we will potentially have quota data, but it most likely will be a 0 value - cachedOnlineDriveData.quotaRestricted will be set as true if (cachedOnlineDriveData.quotaAvailable) { // Our query told us we have free space online .. if we upload this file, will we exceed space online - thus upload will fail during upload? if (calculatedSpaceOnlinePostUpload > 0) { // Based on this thread action, we believe that there is space available online to upload - proceed spaceAvailableOnline = true; } } // Is quota being restricted? if (cachedOnlineDriveData.quotaRestricted) { // Space available online is being restricted - so we have no way to really know if there is space available online spaceAvailableOnline = true; } // Do we have space available or is space available being restricted (so we make the blind assumption that there is space available) JSONValue uploadResponse; if (spaceAvailableOnline) { // Does this file exceed the maximum file size to upload to OneDrive? if (thisFileSizeLocal <= maxUploadFileSize) { // Attempt to upload the modified file // Error handling is in performModifiedFileUpload(), and the JSON that is responded with - will either be null or a valid JSON object containing the upload result uploadResponse = performModifiedFileUpload(dbItem, localFilePath, thisFileSizeLocal); // Evaluate the returned JSON uploadResponse // If there was an error uploading the file, uploadResponse should be empty and invalid if (uploadResponse.type() != JSONType.object) { uploadFailed = true; skippedExceptionError = true; } } else { // Skip file - too large uploadFailed = true; skippedMaxSize = true; } } else { // Cant upload this file - no space available uploadFailed = true; } // Did the upload fail? if (uploadFailed) { // Upload failed .. why? // No space available online if (!spaceAvailableOnline) { addLogEntry("Skipping uploading modified file: " ~ localFilePath ~ " due to insufficient free space available on Microsoft OneDrive", ["info", "notify"]); } // File exceeds max allowed size if (skippedMaxSize) { addLogEntry("Skipping uploading this modified file as it exceeds the maximum size allowed by Microsoft OneDrive: " ~ localFilePath, ["info", "notify"]); } // Generic message if (skippedExceptionError) { // normal failure message if API or exception error generated // If Issue #2626 | Case 2-1 is triggered, the file we tried to upload was renamed, then uploaded as a new name if (exists(localFilePath)) { // Issue #2626 | Case 2-1 was not triggered, file still exists on local filesystem addLogEntry("Uploading modified file: " ~ localFilePath ~ " ... failed!", ["info", "notify"]); } } } else { // Upload was successful addLogEntry("Uploading modified file: " ~ localFilePath ~ " ... done", fileTransferNotifications()); // As no upload failure, calculate transfer metrics in a consistent manner displayTransferMetrics(localFilePath, thisFileSizeLocal, uploadStartTime, Clock.currTime()); // What do we save to the DB? Is this a OneDrive Business Shared File? if ((dbItem.type == ItemType.remote) && (dbItem.remoteType == ItemType.file)) { // We need to 'massage' the old DB record, with data from online, as the DB record was specifically crafted for OneDrive Business Shared Files Item tempItem = makeItem(uploadResponse); dbItem.eTag = tempItem.eTag; dbItem.cTag = tempItem.cTag; dbItem.mtime = tempItem.mtime; dbItem.quickXorHash = tempItem.quickXorHash; dbItem.sha256Hash = tempItem.sha256Hash; dbItem.size = tempItem.size; itemDB.upsert(dbItem); } else { // Save the response JSON item in database as is saveItem(uploadResponse); } // Update the 'cachedOnlineDriveData' record for this 'targetDriveId' so that this is tracked as accurately as possible for other threads updateDriveDetailsCache(targetDriveId, cachedOnlineDriveData.quotaRestricted, cachedOnlineDriveData.quotaAvailable, thisFileSizeLocal); // Check the integrity of the uploaded modified file if not in a --dry-run scenario if (!dryRun) { bool uploadIntegrityPassed; // Check the integrity of the uploaded modified file, if the local file still exists uploadIntegrityPassed = performUploadIntegrityValidationChecks(uploadResponse, localFilePath, thisFileSizeLocal); // Update the date / time of the file online to match the local item // Get the local file last modified time SysTime localModifiedTime = timeLastModified(localFilePath).toUTC(); // Drop fractional seconds for upload timestamp modification as Microsoft OneDrive does not support fractional seconds localModifiedTime.fracSecs = Duration.zero; // Get the latest eTag, and use that string etagFromUploadResponse = uploadResponse["eTag"].str; // Attempt to update the online lastModifiedDateTime value based on our local timestamp data if (appConfig.accountType == "personal") { // Personal Account Handling for Modified File Upload // // Did the upload integrity check pass or fail? if (!uploadIntegrityPassed) { // upload integrity check failed for the modified file if (!appConfig.getValueBool("create_new_file_version")) { // warn that file differences will exist online // as this is a 'personal' account .. we have no idea / reason potentially, so do not download the 'online' file addLogEntry("WARNING: The file uploaded to Microsoft OneDrive does not match your local version. Data loss may occur."); } else { // Create a new online version of the file by updating the online metadata uploadLastModifiedTime(dbItem, targetDriveId, targetItemId, localModifiedTime, etagFromUploadResponse); } } else { // Upload of the modified file passed integrity checks // We need to make sure that the local file on disk has this timestamp from this JSON, otherwise on the next application run: // The last modified timestamp has changed however the file content has not changed // The local item has the same hash value as the item online - correcting timestamp online // This then creates another version online which we do not want to do .. unless configured to do so if (!appConfig.getValueBool("create_new_file_version")) { // Are we in an --upload-only scenario? // In in an --upload-only scenario, it is pointless updating the local timestamp with that what is now online if(!uploadOnly){ // Create an applicable DB item from the upload JSON response Item onlineItem; onlineItem = makeItem(uploadResponse); // Correct the local file timestamp to avoid creating a new version online // Set the local timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, localFilePath, onlineItem.mtime); } } else { // Create a new online version of the file by updating the metadata, which negates the need to download the file uploadLastModifiedTime(dbItem, targetDriveId, targetItemId, localModifiedTime, etagFromUploadResponse); } } } else { // Business | SharePoint Account Handling for Modified File Upload // // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. // This means that the file which was uploaded, is potentially no longer the file we have locally // There are 2 ways to solve this: // 1. Download the modified file immediately after upload as per v2.4.x (default) // 2. Create a new online version of the file, which then contributes to the users 'quota' // Did the upload integrity check pass or fail? if (!uploadIntegrityPassed) { // upload integrity check failed for the modified file if (!appConfig.getValueBool("create_new_file_version")) { // Are we in an --upload-only scenario? if(!uploadOnly){ // Download the now online modified file addLogEntry("WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded."); addLogEntry("WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details."); // Download the file directly using the prior upload JSON response downloadFileItem(uploadResponse, true); } else { // --upload-only being used // we are not downloading a file, warn that file differences will exist addLogEntry("WARNING: The file uploaded to Microsoft OneDrive has been modified through its SharePoint 'enrichment' process and no longer matches your local version."); addLogEntry("WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details."); } } else { // Create a new online version of the file by updating the metadata, which negates the need to download the file uploadLastModifiedTime(dbItem, targetDriveId, targetItemId, localModifiedTime, etagFromUploadResponse); } } else { // Upload of the modified file passed integrity checks // We need to make sure that the local file on disk has this timestamp from this JSON, otherwise on the next application run: // The last modified timestamp has changed however the file content has not changed // The local item has the same hash value as the item online - correcting timestamp online // This then creates another version online which we do not want to do .. unless configured to do so if (!appConfig.getValueBool("create_new_file_version")) { // Are we in an --upload-only scenario? // In in an --upload-only scenario, it is pointless updating the local timestamp with that what is now online if(!uploadOnly){ // Create an applicable DB item from the upload JSON response Item onlineItem; onlineItem = makeItem(uploadResponse); // Correct the local file timestamp to avoid creating a new version online // Set the timestamp, logging and error handling done within function setLocalPathTimestamp(dryRun, localFilePath, onlineItem.mtime); } } else { // Create a new online version of the file by updating the metadata, which negates the need to download the file uploadLastModifiedTime(dbItem, targetDriveId, targetItemId, localModifiedTime, etagFromUploadResponse); } } } // Are we in an --upload-only & --remove-source-files scenario? if ((uploadOnly) && (localDeleteAfterUpload)) { // Perform the local file deletion removeLocalFilePostUpload(localFilePath); } } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Remove the local file if using --upload-only & --remove-source-files scenario in a consistent manner void removeLocalFilePostUpload(string localPathToRemove) { // File has to exist before removal if (exists(localPathToRemove)) { // Log that we are deleting a local item addLogEntry("Attempting removal of local file as --upload-only & --remove-source-files configured"); // Are we in a --dry-run scenario? if (!dryRun) { // Not in a --dry-run scenario if (debugLogging) {addLogEntry("Removing local file: " ~ localPathToRemove, ["debug"]);} safeRemove(localPathToRemove); addLogEntry("Removed local file: " ~ localPathToRemove); // Do we try and attempt to remove the local source tree? if (appConfig.getValueBool("remove_source_folders")) { // Remove the source directory structure but only if it is empty addLogEntry("Attempting removal of local directory structure as --upload-only & --remove-source-files & --remove-source-folders configured"); string parentPath = dirName(localPathToRemove); removeEmptyParents(localPathToRemove); addLogEntry("Removed parental path: " ~ parentPath); } } else { // --dry-run scenario addLogEntry("Not removing local file as --dry-run configured"); } } else { // Log that the path to remove does not exist locally addLogEntry("Removing local file not possible as local file does not exist"); } } // Remove empty parent directories of `filePath` upwards until: // - we hit a non-empty directory, or // - we reach the visible root (i.e. dirName(current) == "."). // Never tries to remove ".". void removeEmptyParents(string filePath) { // Work with a normalised *relative* path inside the chrooted configured 'sync_dir' // If someone passed an absolute path, normalise it anyway; your codebase // likely already ensures paths are relative within the sync root. string current = dirName(buildNormalizedPath(filePath)); while (current.length && current != ".") { // Safety: don’t descend into symlinks if (isSymlink(current)) { if (debugLogging) addLogEntry("Skipping removal; parent is a symlink: " ~ current, ["debug"]); break; } // Stop at first non-empty directory if (!isDirEmpty(current)) { if (debugLogging) addLogEntry("Stopping prune; directory not empty: " ~ current, ["debug"]); break; } if (!dryRun) { if (debugLogging) addLogEntry("Removing empty directory: " ~ current, ["debug"]); // rmdir only succeeds for empty directories; errors are collected not thrown collectException(rmdir(current)); } else { addLogEntry("Not removing empty directory as --dry-run configured: " ~ current); } // Move up one level string next = dirName(current); if (next == current) { // Just in case (shouldn’t happen with relative paths) break; } current = next; } } // Perform the upload of a locally modified file to OneDrive JSONValue performModifiedFileUpload(Item dbItem, string localFilePath, long thisFileSizeLocal) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables JSONValue uploadResponse; OneDriveApi uploadFileOneDriveApiInstance; uploadFileOneDriveApiInstance = new OneDriveApi(appConfig); uploadFileOneDriveApiInstance.initialise(); // Configure JSONValue variables we use for a session upload JSONValue currentOnlineJSONData; Item currentOnlineItemData; JSONValue uploadSessionData; string currentETag; // When we are uploading OneDrive Business Shared Files, we need to be targeting the right driveId and itemId string targetDriveId; string targetParentId; string targetItemId; // Is this a remote target? if ((dbItem.type == ItemType.remote) && (dbItem.remoteType == ItemType.file)) { // This is a remote file targetDriveId = dbItem.remoteDriveId; targetParentId = dbItem.remoteParentId; targetItemId = dbItem.remoteId; } else { // This is not a remote file targetDriveId = dbItem.driveId; targetParentId = dbItem.parentId; targetItemId = dbItem.id; } // Is this a dry-run scenario? if (!dryRun) { // Do we use simpleUpload or create an upload session? bool useSimpleUpload = false; // Try and get the absolute latest object details from online, so we get the latest eTag to try and avoid a 412 eTag error try { currentOnlineJSONData = uploadFileOneDriveApiInstance.getPathDetailsById(targetDriveId, targetItemId); } catch (OneDriveException exception) { // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // Was a valid JSON response provided? if (currentOnlineJSONData.type() == JSONType.object) { // Does the response contain an eTag? if (hasETag(currentOnlineJSONData)) { // Use the value returned from online as this will attempt to avoid a 412 response if we are creating a session upload currentETag = currentOnlineJSONData["eTag"].str; } else { // Use the database value - greater potential for a 412 error to occur if we are creating a session upload if (debugLogging) {addLogEntry("Online data for file returned zero eTag - using database eTag value", ["debug"]);} currentETag = dbItem.eTag; } // Make a reusable item from this online JSON data currentOnlineItemData = makeItem(currentOnlineJSONData); } else { // no valid JSON response - greater potential for a 412 error to occur if we are creating a session upload if (debugLogging) {addLogEntry("Online data returned was invalid - using database eTag value", ["debug"]);} currentETag = dbItem.eTag; } // What upload method should be used? if (thisFileSizeLocal <= sessionThresholdFileSize) { // file size is below session threshold useSimpleUpload = true; } // Use Session Upload regardless if (appConfig.getValueBool("force_session_upload")) { // Forcing session upload if (debugLogging) {addLogEntry("Forcing to perform upload using a session (modified)", ["debug"]);} useSimpleUpload = false; } // If the filesize is greater than zero , and we have valid 'latest' online data is the online file matching what we think is in the database? if ((thisFileSizeLocal > 0) && (currentOnlineJSONData.type() == JSONType.object)) { // Issue #2626 | Case 2-1 // If the 'online' file is newer, this will be overwritten with the file from the local filesystem - potentially constituting online data loss Item onlineFile = makeItem(currentOnlineJSONData); // Which file is technically newer? The local file or the remote file? SysTime localModifiedTime = timeLastModified(localFilePath).toUTC(); SysTime onlineModifiedTime = onlineFile.mtime; // Reduce time resolution to seconds before comparing localModifiedTime.fracSecs = Duration.zero; onlineModifiedTime.fracSecs = Duration.zero; // Which file is newer? If local is newer, it will be uploaded as a modified file in the correct manner if (localModifiedTime < onlineModifiedTime) { // Online File is actually newer than the locally modified file if (debugLogging) { addLogEntry("currentOnlineJSONData: " ~ to!string(currentOnlineJSONData), ["debug"]); addLogEntry("onlineFile: " ~ to!string(onlineFile), ["debug"]); addLogEntry("database item: " ~ to!string(dbItem), ["debug"]); } addLogEntry("Skipping uploading this item as a locally modified file, will upload as a new file (online file already exists and is newer): " ~ localFilePath); // Online is newer, rename local, then upload the renamed file // We need to know the renamed path so we can upload it string renamedPath; // Rename the local path - we WANT this to occur regardless of bypassDataPreservation setting safeBackup(localFilePath, dryRun, false, renamedPath); // Upload renamed local file as a new file uploadNewFile(renamedPath); // Process the database entry removal for the original file. In a --dry-run scenario, this is being done against a DB copy. // This is done so we can download the newer online file itemDB.deleteById(targetDriveId, targetItemId); // This file is now uploaded, return from here, but this will trigger a response that the upload failed (technically for the original filename it did, but we renamed it, then uploaded it return uploadResponse; } } // We can only upload zero size files via simpleFileUpload regardless of account type // Reference: https://github.com/OneDrive/onedrive-api-docs/issues/53 // Additionally, all files where file size is < 4MB should be uploaded by simpleUploadReplace - everything else should use a session to upload the modified file if ((thisFileSizeLocal == 0) || (useSimpleUpload)) { // Must use Simple Upload to replace the file online try { uploadResponse = uploadFileOneDriveApiInstance.simpleUploadReplace(localFilePath, targetDriveId, targetItemId); } catch (OneDriveException exception) { // HTTP request returned status code 403 if ((exception.httpStatusCode == 403) && (appConfig.getValueBool("sync_business_shared_files"))) { // We attempted to upload a file, that was shared with us, but this was shared with us as read-only addLogEntry("Unable to upload this modified file as this was shared as read-only: " ~ localFilePath); } // HTTP request returned status code 423 // Resolve https://github.com/abraunegg/onedrive/issues/36 if (exception.httpStatusCode == 423) { // The file is currently checked out or locked for editing by another user // We cant upload this file at this time addLogEntry("Unable to upload this modified file as this is currently checked out or locked for editing by another user: " ~ localFilePath); } else { // Handle all other HTTP status codes // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } catch (FileException e) { // filesystem error displayFileSystemErrorMessage(e.msg, thisFunctionName, localFilePath); } } else { // As this is a unique thread, the sessionFilePath for where we save the data needs to be unique // The best way to do this is generate a 10 digit alphanumeric string, and use this as the file extension string threadUploadSessionFilePath = appConfig.uploadSessionFilePath ~ "." ~ generateAlphanumericString(); // Create the upload session using the latest online data 'currentOnlineData' etag try { // create the session uploadSessionData = createSessionForFileUpload(uploadFileOneDriveApiInstance, localFilePath, targetDriveId, targetParentId, baseName(localFilePath), currentOnlineItemData.eTag, threadUploadSessionFilePath); } catch (OneDriveException exception) { // HTTP request returned status code 403 if ((exception.httpStatusCode == 403) && (appConfig.getValueBool("sync_business_shared_files"))) { // We attempted to upload a file, that was shared with us, but this was shared with us as read-only addLogEntry("Unable to upload this modified file as this was shared as read-only: " ~ localFilePath); return uploadResponse; } // HTTP request returned status code 423 // Resolve https://github.com/abraunegg/onedrive/issues/36 if (exception.httpStatusCode == 423) { // The file is currently checked out or locked for editing by another user // We cant upload this file at this time addLogEntry("Unable to upload this modified file as this is currently checked out or locked for editing by another user: " ~ localFilePath); return uploadResponse; } else { // Handle all other HTTP status codes // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } catch (FileException e) { // Display filesystem exception error message displayFileSystemErrorMessage(e.msg, thisFunctionName, threadUploadSessionFilePath); } // Do we have a valid session URL that we can use ? if (uploadSessionData.type() == JSONType.object) { // This is a valid JSON object // Perform the upload using the session that has been created try { // so that we have this data available if we need to re-create the session // - targetDriveId, targetParentId, baseName(localFilePath), currentOnlineItemData.eTag, threadUploadSessionFilePath uploadSessionData["targetDriveId"] = targetDriveId; uploadSessionData["targetParentId"] = targetParentId; uploadSessionData["currentETag"] = currentOnlineItemData.eTag; // attempt the session upload using the session data provided uploadResponse = performSessionFileUpload(uploadFileOneDriveApiInstance, thisFileSizeLocal, uploadSessionData, threadUploadSessionFilePath); } catch (OneDriveException exception) { // Handle all other HTTP status codes // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } catch (FileException e) { // Display filesystem exception error message displayFileSystemErrorMessage(e.msg, thisFunctionName, threadUploadSessionFilePath); } } else { // Create session Upload URL failed if (debugLogging) {addLogEntry("Unable to upload modified file as the creation of the upload session URL failed", ["debug"]);} } } } else { // We are in a --dry-run scenario uploadResponse = createFakeResponse(localFilePath); } // Debug Log the modified upload response if (debugLogging) {addLogEntry("Modified File Upload Response: " ~ to!string(uploadResponse), ["debug"]);} // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return JSON return uploadResponse; } // Query the OneDrive API using the provided driveId to get the latest quota details string[3][] getRemainingFreeSpaceOnline(string sourceDriveId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Get the quota details for this sourceDriveId // Quota details are ONLY available for the main default sourceDriveId, as the OneDrive API does not provide quota details for shared folders JSONValue currentDriveQuota; bool quotaRestricted = false; // Assume quota is not restricted unless "remaining" is missing bool quotaAvailable = false; long quotaRemainingOnline = 0; string[3][] result; OneDriveApi getCurrentDriveQuotaApiInstance; string driveId; // Issue #3115 - Validate sourceDriveId length // What account type is this? if (appConfig.accountType == "personal") { // Test sourceDriveId length and validation if (!sourceDriveId.empty) { // We were provided a sourceDriveId - that is what we check driveId = transformToLowerCase(testProvidedDriveIdForLengthIssue(sourceDriveId)); } else { // No sourceDriveId provided - use appConfig.defaultDriveId and validate that driveId = transformToLowerCase(testProvidedDriveIdForLengthIssue(appConfig.defaultDriveId)); } } else { // This is not a personal account type // Ensure that we have a valid driveId to query if (sourceDriveId.empty) { // No 'driveId' was provided, use the application default driveId = appConfig.defaultDriveId; } else { // A 'driveId' was provided, use the provided 'sourceDriveId' driveId = sourceDriveId; } } // Try and query the quota for the provided driveId try { // Create a new OneDrive API instance getCurrentDriveQuotaApiInstance = new OneDriveApi(appConfig); getCurrentDriveQuotaApiInstance.initialise(); if (debugLogging) {addLogEntry("Seeking available quota for this drive id: " ~ driveId, ["debug"]);} currentDriveQuota = getCurrentDriveQuotaApiInstance.getDriveQuota(driveId); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getCurrentDriveQuotaApiInstance.releaseCurlEngine(); getCurrentDriveQuotaApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException e) { if (debugLogging) {addLogEntry("currentDriveQuota = onedrive.getDriveQuota(driveId) generated a OneDriveException", ["debug"]);} // If an exception occurs, it's unclear if quota is restricted, but quota details are not available quotaRestricted = true; // Considering restricted due to failure to access // Return result result ~= [to!string(quotaRestricted), to!string(quotaAvailable), to!string(quotaRemainingOnline)]; // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getCurrentDriveQuotaApiInstance.releaseCurlEngine(); getCurrentDriveQuotaApiInstance = null; // Perform Garbage Collection GC.collect(); return result; } // Validate that currentDriveQuota is a JSON value if (currentDriveQuota.type() == JSONType.object && "quota" in currentDriveQuota) { // Response from API contains valid data // If 'personal' accounts, if driveId == defaultDriveId, then we will have data // If 'personal' accounts, if driveId != defaultDriveId, then we will not have quota data // If 'business' accounts, if driveId == defaultDriveId, then we will have data // If 'business' accounts, if driveId != defaultDriveId, then we will have data, but it will be a 0 value JSONValue quota = currentDriveQuota["quota"]; // debug output the entire 'quota' JSON response if (debugLogging) {addLogEntry("Quota Details: " ~ to!string(quota), ["debug"]);} // Does the 'quota' JSON struct contain 'remaining' ? if ("remaining" in quota) { // Issue #2806 // If this is a negative value, quota["remaining"].integer can potentially convert to a huge positive number. Convert a different way. string tempQuotaRemainingOnlineString; // is quota["remaining"] an integer type? if (quota["remaining"].type() == JSONType.integer) { // debug logging of the 'remaining' JSON struct if (debugLogging) { addLogEntry("quota remaining is an integer value - using this value: " ~ to!string(quota["remaining"].integer), ["debug"]); } // extract as integer and convert to string tempQuotaRemainingOnlineString = to!string(quota["remaining"].integer); } // Is 'tempQuotaRemainingOnlineString' still empty post integer check? if (tempQuotaRemainingOnlineString.empty) { // debug log that 'tempQuotaRemainingOnlineString' is still empty post integer check if (debugLogging) { addLogEntry("tempQuotaRemainingOnlineString is still empty post integer JSON value analysis ..", ["debug"]); } // is quota["remaining"] an string type? if (quota["remaining"].type() == JSONType.string) { // debug logging of the 'remaining' JSON struct if (debugLogging) { addLogEntry("quota remaining is an string value - using this value: " ~ to!string(quota["remaining"].str), ["debug"]); } // extract JSON value as string tempQuotaRemainingOnlineString = quota["remaining"].str; } } // Fallback if tempQuotaRemainingOnlineString is still empty if (tempQuotaRemainingOnlineString.empty) { // debug log that 'tempQuotaRemainingOnlineString' is still empty if (debugLogging) { addLogEntry("tempQuotaRemainingOnlineString is still empty post integer and string JSON value analysis .. this means quota 'remaining' element was not a string or integer value", ["debug"]); } // Fetch the details from cachedOnlineDriveData DriveDetailsCache cachedOnlineDriveData; cachedOnlineDriveData = getDriveDetails(appConfig.defaultDriveId); // Use cachedOnlineDriveData.quotaRemaining as this is the last value we potentially had if ((cachedOnlineDriveData.quotaRemaining) > 0) { // the last known quota remaining was above zero if (debugLogging) { addLogEntry("cachedOnlineDriveData.quotaRemaining is a positive value, using this last known value for tempQuotaRemainingOnlineString", ["debug"]); } // set tempQuotaRemainingOnlineString to cachedOnlineDriveData.quotaRemaining tempQuotaRemainingOnlineString = to!string(cachedOnlineDriveData.quotaRemaining); } else { if (debugLogging) { addLogEntry("cachedOnlineDriveData.quotaRemaining is zero or negative value, setting tempQuotaRemainingOnlineString to zero", ["debug"]); } // no option but to set to zero tempQuotaRemainingOnlineString = "0"; } } // What did we set 'tempQuotaRemainingOnlineString' to? if (debugLogging) { addLogEntry("tempQuotaRemainingOnlineString = " ~ tempQuotaRemainingOnlineString, ["debug"]); } // Update quotaRemainingOnline to use the converted string value quotaRemainingOnline = to!long(tempQuotaRemainingOnlineString); // What did we set 'quotaRemainingOnline' to? if (debugLogging) { addLogEntry("quotaRemainingOnline = " ~ to!string(quotaRemainingOnline), ["debug"]); } // Set the applicable 'quotaAvailable' value quotaAvailable = quotaRemainingOnline > 0; // If "remaining" is present but its value is <= 0, it's not restricted but exhausted if (quotaRemainingOnline <= 0) { if (appConfig.accountType == "personal") { addLogEntry("ERROR: OneDrive account currently has zero space available. Please free up some space online or purchase additional capacity."); } else { // Assuming 'business' or 'sharedLibrary' if (verboseLogging) {addLogEntry("WARNING: OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator." , ["verbose"]);} } } } else { // "remaining" not present, indicating restricted quota information quotaRestricted = true; // what sort of account type is this? if (appConfig.accountType == "personal") { if (verboseLogging) {addLogEntry("ERROR: OneDrive quota information is missing. Your OneDrive account potentially has zero space available. Please free up some space online.", ["verbose"]);} } else { // quota details not available if (verboseLogging) {addLogEntry("WARNING: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.", ["verbose"]);} } } } else { // When valid quota details are not fetched if (verboseLogging) {addLogEntry("Failed to fetch or query quota details for OneDrive Drive ID: " ~ driveId, ["verbose"]);} quotaRestricted = true; // Considering restricted due to failure to interpret } // What was the determined available quota? if (debugLogging) {addLogEntry("Reported Available Online Quota for driveID '" ~ driveId ~ "': " ~ to!string(quotaRemainingOnline), ["debug"]);} // Return result result ~= [to!string(quotaRestricted), to!string(quotaAvailable), to!string(quotaRemainingOnline)]; // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return new drive array data return result; } // Perform a filesystem walk to uncover new data to upload to OneDrive void scanLocalFilesystemPathForNewData(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Cleanup array memory before we start adding files pathsToCreateOnline = []; newLocalFilesToUploadToOneDrive = []; // Perform a filesystem walk to uncover new data scanLocalFilesystemPathForNewDataToUpload(path); // Create new directories online that has been identified processNewDirectoriesToCreateOnline(); // Upload new data that has been identified processNewLocalItemsToUpload(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Scan the local filesystem for new data to upload void scanLocalFilesystemPathForNewDataToUpload(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // To improve logging output for this function, what is the 'logical path' we are scanning for file & folder differences? string logPath; if (path == ".") { // get the configured sync_dir logPath = buildNormalizedPath(appConfig.getValueString("sync_dir")); } else { // use what was passed in if (!appConfig.getValueBool("monitor")) { logPath = buildNormalizedPath(appConfig.getValueString("sync_dir")) ~ "/" ~ path; } else { logPath = path; } } // Log the action that we are performing, however only if this is a directory if (exists(path)) { if (isDir(path)) { if (!appConfig.suppressLoggingOutput) { if (!cleanupLocalFiles) { addProcessingLogHeaderEntry("Scanning the local file system '" ~ logPath ~ "' for new data to upload", appConfig.verbosityCount); } else { addProcessingLogHeaderEntry("Scanning the local file system '" ~ logPath ~ "' for data to cleanup", appConfig.verbosityCount); // Set the cleanup flag cleanupDataPass = true; } } } } SysTime startTime; if (debugLogging) { startTime = Clock.currTime(); addLogEntry("Starting Filesystem Walk (Local Time): " ~ to!string(startTime), ["debug"]); } // Add a processing '.' if this is a directory we are scanning if (exists(path)) { if (isDir(path)) { if (!appConfig.suppressLoggingOutput) { if (appConfig.verbosityCount == 0) { addProcessingDotEntry(); } } } } // Perform the filesystem walk of this path, building an array of new items to upload scanPathForNewData(path); // Reset flag cleanupDataPass = false; // Close processing '.' if this is a directory we are scanning if (exists(path)) { if (isDir(path)) { if (appConfig.verbosityCount == 0) { if (!appConfig.suppressLoggingOutput) { // Close out the '....' being printed to the console completeProcessingDots(); } } } } // To finish off the processing items, this is needed to reflect this in the log if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); // finish filesystem walk time SysTime finishTime = Clock.currTime(); addLogEntry("Finished Filesystem Walk (Local Time): " ~ to!string(finishTime), ["debug"]); // duration Duration elapsedTime = finishTime - startTime; addLogEntry("Elapsed Time Filesystem Walk: " ~ to!string(elapsedTime), ["debug"]); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Ensure we have a full list of unique paths to create online void addPathToCreateOnline(string pathToAdd) { // Is this a valid path to add? // The requested directory to create was not found on OneDrive - creating remote directory: ./. // OneDrive generated an error when creating this path: ./. // ERROR: Microsoft OneDrive API returned an error with the following message: // Error Message: HTTP request returned status code 400 (Bad Request) // Error Reason: Invalid request // Error Code: invalidRequest // Error Timestamp: 2025-05-02T20:31:46 // API Request ID: 23c2e2cd-6968-4a99-ac80-f9da786a18fd // Calling Function: syncEngine.createDirectoryOnline() // Is this a valid path to add? if ((pathToAdd == ".")||(pathToAdd == "./.")) { // matches paths we should not attempt to create online if (debugLogging) {addLogEntry("attempted to add as path to create online - rejecting: " ~ pathToAdd, ["debug"]);} // We can never add or create online the OneDrive 'root' return; } // Only add unique paths if (!pathsToCreateOnline.canFind(pathToAdd)) { // Add this unique path to the created online // are we in a --dry-run scenario? if (!dryRun) { // Add this to the list to create online pathsToCreateOnline ~= pathToAdd; } else { // We are in a --dry-run scenario .. this might have been a directory we 'faked' doing something with. // pathsRenamed contains all the paths that were 'renamed' if (pathsRenamed.canFind(ensureStartsWithDotSlash(buildNormalizedPath(pathToAdd)))) { // Path was renamed .. but faked due to --dry-run if (debugLogging) {addLogEntry("DRY-RUN: Skipping creating this directory online as this was a faked local change", ["debug"]);} } else { // Add this to the list to create online pathsToCreateOnline ~= pathToAdd; } } } } // Create new directories online void processNewDirectoriesToCreateOnline() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This list of local paths that need to be created online string[] uniquePathsToCreateOnline; // Are there any new local directories to create online? if (!pathsToCreateOnline.empty) { // There are new directories to create online addLogEntry("New directories to create on Microsoft OneDrive: " ~ to!string(pathsToCreateOnline.length)); if (debugLogging) {addLogEntry("pathsToCreateOnline = " ~ to!string(pathsToCreateOnline), ["debug"]);} // Process 'pathsToCreateOnline' into each array element, then create each path based on path segments foreach (fullPath; pathsToCreateOnline) { // Normalise path and strip leading "./" if present string normalised = fullPath; if (normalised.startsWith("./")) normalised = normalised[2 .. $]; if (normalised.endsWith("/")) normalised = normalised[0 .. $ - 1]; auto segments = normalised.split("/").filter!(s => !s.empty).array; string pathToCreate = "."; foreach (i; 0 .. segments.length) { pathToCreate = buildPath(pathToCreate, segments[i]); // Only add unique paths to avoid duplication of the same path creation request if (!uniquePathsToCreateOnline.canFind(pathToCreate)) { // Add this unique path to the created online uniquePathsToCreateOnline ~= pathToCreate; } } } } // Now that all the paths have been rationalised and potential duplicate creation requests filtered out, create the paths online if (debugLogging) {addLogEntry("uniquePathsToCreateOnline = " ~ to!string(uniquePathsToCreateOnline), ["debug"]);} // For each path in the array, attempt to create this online foreach (onlinePathToCreate; uniquePathsToCreateOnline) { try { // Try and create the required path online createDirectoryOnline(onlinePathToCreate); } catch (Exception e) { addLogEntry("ERROR: Failed to create directory online: " ~ onlinePathToCreate ~ " => " ~ e.msg); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Upload new data that has been identified to Microsoft OneDrive void processNewLocalItemsToUpload() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Are there any new local items to upload? if (!newLocalFilesToUploadToOneDrive.empty) { // There are elements to upload addLogEntry("New items to upload to Microsoft OneDrive: " ~ to!string(newLocalFilesToUploadToOneDrive.length) ); // Reset totalDataToUpload totalDataToUpload = 0; // How much data do we need to upload? This is important, as, we need to know how much data to determine if all the files can be uploaded foreach (uploadFilePath; newLocalFilesToUploadToOneDrive) { // validate that the path actually exists so that it can be counted if (exists(uploadFilePath)) { totalDataToUpload = totalDataToUpload + getSize(uploadFilePath); } } // How much data is there to upload if (verboseLogging) { if (totalDataToUpload < 1024) { // Display as Bytes to upload addLogEntry("Total New Data to Upload: " ~ to!string(totalDataToUpload) ~ " Bytes", ["verbose"]); } else { if ((totalDataToUpload > 1024) && (totalDataToUpload < 1048576)) { // Display as KB to upload addLogEntry("Total New Data to Upload: " ~ to!string((totalDataToUpload / 1024)) ~ " KB", ["verbose"]); } else { // Display as MB to upload addLogEntry("Total New Data to Upload: " ~ to!string((totalDataToUpload / 1024 / 1024)) ~ " MB", ["verbose"]); } } } // How much space is available // The file, could be uploaded to a shared folder, which, we are not tracking how much free space is available there ... // Iterate through all the drives we have cached thus far, that we know about if (debugLogging) { foreach (driveId, driveDetails; onlineDriveDetails) { // Log how much space is available for each driveId addLogEntry("Current Available Space Online (" ~ driveId ~ "): " ~ to!string((driveDetails.quotaRemaining / 1024 / 1024)) ~ " MB", ["debug"]); } } // Perform the upload uploadNewLocalFileItems(); // Cleanup array memory after uploading all files newLocalFilesToUploadToOneDrive = []; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Scan this path for new data void scanPathForNewData(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Skip symlinks as early as possible, including dangling symlinks if (isSymlink(path)) { // Should this path be skipped? if (appConfig.getValueBool("skip_symlinks")) { if (verboseLogging) {addLogEntry("Skipping item - skip symbolic links configured: " ~ path, ["verbose"]);} return; } } // Add a processing '.' if path exists if (exists(path)) { if (isDir(path)) { if (!appConfig.suppressLoggingOutput) { if (appConfig.verbosityCount == 0) { addProcessingDotEntry(); } } } } long maxPathLength; long pathWalkLength; // Add this logging break to assist with what was checked for each path if (path != ".") { if (debugLogging) {addLogEntry(debugLogBreakType1, ["debug"]);} } // https://support.microsoft.com/en-us/help/3125202/restrictions-and-limitations-when-you-sync-files-and-folders // If the path is greater than allowed characters, then one drive will return a '400 - Bad Request' // Need to ensure that the URI is encoded before the check is made: // - 400 Character Limit for OneDrive Business / Office 365 // - 430 Character Limit for OneDrive Personal // Configure maxPathLength based on account type if (appConfig.accountType == "personal") { // Personal Account maxPathLength = 430; } else { // Business Account / Office365 / SharePoint maxPathLength = 400; } // OneDrive Business Shared Files Handling - if we make a 'backup' locally of a file shared with us (because we modified it, and then maybe did a --resync), it will be treated as a new file to upload ... // The issue here is - the 'source' was a shared file - we may not even have permission to upload a 'renamed' file to the shared file's parent folder // In this case, we need to skip adding this new local file - we do not upload it (we cant , and we should not) if (appConfig.accountType == "business") { // Check appConfig.configuredBusinessSharedFilesDirectoryName against 'path' if (canFind(path, baseName(appConfig.configuredBusinessSharedFilesDirectoryName))) { // Log why this path is being skipped addLogEntry("Skipping scanning path for new files as this is reserved for OneDrive Business Shared Files: " ~ path, ["info"]); return; } } // A short lived item that has already disappeared will cause an error - is the path still valid? if (!exists(path)) { addLogEntry("Skipping path - path has disappeared: " ~ path); return; } // Calculate the path length by walking the path and catch any UTF-8 sequence errors at the same time // https://github.com/skilion/onedrive/issues/57 // https://github.com/abraunegg/onedrive/issues/487 // https://github.com/abraunegg/onedrive/issues/1192 try { pathWalkLength = path.byGrapheme.walkLength; } catch (std.utf.UTFException e) { // Path contains characters which generate a UTF exception addLogEntry("Skipping item - invalid UTF sequence: " ~ path, ["info", "notify"]); if (debugLogging) {addLogEntry(" Error Reason:" ~ e.msg, ["debug"]);} return; } // Is the path length is less than maxPathLength if (pathWalkLength < maxPathLength) { // Is this path unwanted bool unwanted = false; // First check of this item - if we are in a --dry-run scenario, we may have 'fake deleted' this path // thus, the entries are not in the dry-run DB copy, thus, at this point the client thinks that this is an item to upload // Check this 'path' for an entry in pathFakeDeletedArray - if it is there, this is unwanted if (dryRun) { // Is this path in the array of fake deleted items? If yes, return early, nothing else to do, save processing if (canFind(pathFakeDeletedArray, path)) return; } // Check if item if found in database bool itemFoundInDB = pathFoundInDatabase(path); // If the item is already found in the database, it is redundant to perform these checks if (!itemFoundInDB) { // This not a Client Side Filtering check, nor a Microsoft Check, but is a sanity check that the path provided is UTF encoded correctly // Check the std.encoding of the path against: Unicode 5.0, ASCII, ISO-8859-1, ISO-8859-2, WINDOWS-1250, WINDOWS-1251, WINDOWS-1252 if (!unwanted) { if(!isValid(path)) { // Path is not valid according to https://dlang.org/phobos/std_encoding.html addLogEntry("Skipping item - invalid character encoding sequence: " ~ path, ["info", "notify"]); unwanted = true; } } // Check this path against the Client Side Filtering Rules // - check_nosync // - skip_dotfiles // - skip_symlinks // - skip_file // - skip_dir // - sync_list // - skip_size if (!unwanted) { // If this is not the cleanup data pass when using --download-only --cleanup-local-files we dont want to exclude files we need to delete locally when using 'sync_list' if (!cleanupDataPass) { unwanted = checkPathAgainstClientSideFiltering(path); } } // Check this path against the Microsoft Naming Conventions & Restrictions // - Check path against Microsoft OneDrive restriction and limitations about Windows naming for files and folders // - Check path for bad whitespace items // - Check path for HTML ASCII Codes // - Check path for ASCII Control Codes if (!unwanted) { unwanted = checkPathAgainstMicrosoftNamingRestrictions(path); } } // Before we traverse this 'path', we need to make a last check to see if this was just excluded bool skipFolderTraverse = skipBusinessSharedFolder(path); // Current path for error logging string currentPath; if (!unwanted) { // At this point, this path, we want to scan for new data as it is not excluded if (isDir(path)) { // Was the path found in the database? if (!itemFoundInDB) { // Path not found in database when searching all drive id's if (!cleanupLocalFiles) { // --download-only --cleanup-local-files not used // Create this directory on OneDrive so that we can upload files to it // Add this path to an array so that the directory online can be created before we upload files if (debugLogging) {addLogEntry("Adding path to create online (directory inclusion): " ~ path, ["debug"]);} addPathToCreateOnline(path); } else { // we need to clean up this directory if (verboseLogging) {addLogEntry("Attempting removal of local directory as --download-only & --cleanup-local-files configured", ["verbose"]);} // Remove any children of this path if they still exist // Resolve 'Directory not empty' error when deleting local files try { // the cleanup code should only operate on the immediate children of the current directory auto directoryEntries = dirEntries(path, SpanMode.shallow); foreach (DirEntry child; directoryEntries) { // Normalise the child path once and use it consistently everywhere string normalisedChildPath = ensureStartsWithDotSlash(buildNormalizedPath(child.name)); // Default action is to remove unless a retention condition is met bool pathShouldBeRemoved = true; // 1. If this path was already retained earlier, never delete it if (canFind(pathsRetained, normalisedChildPath)) { pathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Path already marked for retention - retaining path: " ~ normalisedChildPath, ["verbose"]);} } // 2. If not already retained, evaluate via sync_list if (pathShouldBeRemoved && syncListConfigured) { // selectiveSync.isPathExcludedViaSyncList() returns: // true = excluded by sync_list // false = included / must be retained if (!selectiveSync.isPathExcludedViaSyncList(child.name)) { pathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Path retained due to 'sync_list' inclusion: " ~ normalisedChildPath, ["verbose"]);} } } // What action should be taken? if (pathShouldBeRemoved) { // Path should be removed if (isDir(child.name)) { if (verboseLogging) {addLogEntry("Attempting removal of local directory: " ~ normalisedChildPath, ["verbose"]);} } else { if (verboseLogging) {addLogEntry("Attempting removal of local file: " ~ normalisedChildPath, ["verbose"]);} } // Are we in a --dry-run scenario? if (!dryRun) { // No --dry-run ... process local delete if (exists(child.name)) { try { if (attrIsDir(child.linkAttributes)) { rmdir(child.name); // Log removal if (verboseLogging) {addLogEntry("Removed local directory: " ~ normalisedChildPath, ["verbose"]);} } else { safeRemove(child.name); // Log removal if (verboseLogging) {addLogEntry("Removed local file: " ~ normalisedChildPath, ["verbose"]);} } } catch (FileException e) { displayFileSystemErrorMessage(e.msg, thisFunctionName, normalisedChildPath); } } } } else { // Path should be retained if (isDir(child.name)) { if (verboseLogging) {addLogEntry("Local directory should be retained due to 'sync_list' inclusion: " ~ child.name, ["verbose"]);} } else { if (verboseLogging) {addLogEntry("Local file should be retained due to 'sync_list' inclusion: " ~ child.name, ["verbose"]);} } // Add this path to the retention list if not already present if (!canFind(pathsRetained, normalisedChildPath)) { pathsRetained ~= normalisedChildPath; } // Child retained, do not perform any further delete logic for this child continue; } } // Clear directoryEntries object.destroy(directoryEntries); // Determine whether the parent path itself should be removed bool parentalPathShouldBeRemoved = true; string normalisedParentPath = ensureStartsWithDotSlash(buildNormalizedPath(path)); string parentPrefix = normalisedParentPath ~ "/"; // 1. sync_list evaluation for the parent path itself if (syncListConfigured) { // selectiveSync.isPathExcludedViaSyncList() returns: // true = excluded by sync_list // false = included / must be retained if (!selectiveSync.isPathExcludedViaSyncList(path)) { parentalPathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Parent path retained due to 'sync_list' inclusion: " ~ path, ["verbose"]);} } } // 2. If parent path exists in the database, it must be retained if (parentalPathShouldBeRemoved && pathFoundInDatabase(normalisedParentPath)) { parentalPathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Parent path found in database - retain path: " ~ normalisedParentPath, ["verbose"]);} } // 3. If any retained path is this parent or is beneath this parent, retain the parent if (parentalPathShouldBeRemoved) { foreach (retainedPath; pathsRetained) { if ((retainedPath == normalisedParentPath) || retainedPath.startsWith(parentPrefix)) { parentalPathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Parent path retained because child path is retained: " ~ retainedPath, ["verbose"]);} break; } } } // What action should be taken? if (parentalPathShouldBeRemoved) { // Remove the parental path now that it is empty of children if (verboseLogging) {addLogEntry("Attempting removal of local directory: " ~ path, ["verbose"]);} // are we in a --dry-run scenario? if (!dryRun) { // No --dry-run ... process local delete if (exists(path)) { try { rmdirRecurse(path); if (verboseLogging) {addLogEntry("Removed local directory: " ~ path, ["verbose"]);} } catch (FileException e) { displayFileSystemErrorMessage(e.msg, thisFunctionName, path); } } } } else { // Path needs to be retained if (verboseLogging) {addLogEntry("Local parent directory should be retained due to 'sync_list' inclusion: " ~ path, ["verbose"]);} // Add the parent path to the retention list if not already present if (!canFind(pathsRetained, normalisedParentPath)) { pathsRetained ~= normalisedParentPath; } } } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, currentPath); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return as there was an error return; } } } // Do we actually traverse this path? if (!skipFolderTraverse) { // Try and access this directory and any path below if (exists(path)) { try { auto directoryEntries = dirEntries(path, SpanMode.shallow, false); foreach (DirEntry entry; directoryEntries) { currentPath = entry.name; scanPathForNewData(entry.name); } // Clear directoryEntries object.destroy(directoryEntries); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, currentPath); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return as there was an error return; } } } } else { // https://github.com/abraunegg/onedrive/issues/984 // path is not a directory, is it a valid file? // pipes - whilst technically valid files, are not valid for this client // prw-rw-r--. 1 user user 0 Jul 7 05:55 my_pipe if (isFile(path)) { // Is the file a '.nosync' file? if (canFind(path, ".nosync")) { if (debugLogging) {addLogEntry("Skipping .nosync file", ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return as there was an error return; } // Was the file found in the database? if (!itemFoundInDB) { // File not found in database when searching all drive id's // Do we upload the file or clean up the file? if (!cleanupLocalFiles) { // --download-only --cleanup-local-files not used // Ensure this directory on OneDrive so that we can upload files to it // Add this path to an array so that the directory online can be created before we upload files string parentPath = dirName(path); if (debugLogging) {addLogEntry("Adding parental path to create online (file inclusion): " ~ parentPath, ["debug"]);} addPathToCreateOnline(parentPath); // Add this path as a file we need to upload if (debugLogging) {addLogEntry("OneDrive Client flagging to upload this file to Microsoft OneDrive: " ~ path, ["debug"]);} if (!dryRun) { // Add to the array newLocalFilesToUploadToOneDrive ~= path; } else { // In a --dry-run scenario, we may have locally fake changed a directory name, thus, this path we are checking needs to checked against 'pathsRenamed' if (pathsRenamed.canFind(ensureStartsWithDotSlash(buildNormalizedPath(parentPath)))) { // Parental path was renamed if (debugLogging) {addLogEntry("DRY-RUN: parentPath found in 'pathsRenamed' ... skipping uploading this file", ["debug"]);} } else { // Add to the array newLocalFilesToUploadToOneDrive ~= path; } } } else { // Normalise the file path once and use it consistently everywhere string normalisedFilePath = ensureStartsWithDotSlash(buildNormalizedPath(path)); // Default action is to remove unless a retention condition is met bool pathShouldBeRemoved = true; // 1. If this path was already retained earlier, never delete it if (canFind(pathsRetained, normalisedFilePath)) { pathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Path already marked for retention - retaining path: " ~ normalisedFilePath, ["verbose"]);} } // 2. If not already retained, evaluate via sync_list if (pathShouldBeRemoved && syncListConfigured) { // selectiveSync.isPathExcludedViaSyncList() returns: // true = excluded by sync_list // false = included / must be retained if (!selectiveSync.isPathExcludedViaSyncList(path)) { pathShouldBeRemoved = false; if (verboseLogging) {addLogEntry("Path retained due to 'sync_list' inclusion: " ~ normalisedFilePath, ["verbose"]);} } } // What action should be taken? if (pathShouldBeRemoved) { // we need to clean up this file if (verboseLogging) {addLogEntry("Attempting removal of local file as --download-only & --cleanup-local-files configured", ["verbose"]);} // are we in a --dry-run scenario? if (verboseLogging) {addLogEntry("Attempting removal of local file: " ~ normalisedFilePath, ["verbose"]);} if (!dryRun) { // No --dry-run ... process local file delete safeRemove(path); // Log removal if (verboseLogging) {addLogEntry("Removed local file: " ~ normalisedFilePath, ["verbose"]);} } } else { // Path should be retained if (verboseLogging) {addLogEntry("Local file should be retained due to 'sync_list' inclusion: " ~ normalisedFilePath, ["verbose"]);} // Add this path to the retention list if not already present if (!canFind(pathsRetained, normalisedFilePath)) { pathsRetained ~= normalisedFilePath; } } } } } else { // path is not a valid file addLogEntry("Skipping item - item is not a valid file: " ~ path, ["info", "notify"]); } } } else { // Issue #3126 - https://github.com/abraunegg/onedrive/discussions/3126 // At this point, this path that we want to scan for new data has been excluded .. we may have an include 'sync_list' rule for a subfolder of this excluded parent ... // If the data is created online, this is not usually a problem, but essentially if we create new data locally, in a folder we are expecting to included by an existing configuration, // unless we actually scan the entire tree, including those directories that are excluded, we are not going to detect the new locally added data in a parent that has been excluded, // but the child content has to be included if (isDir(path)) { // Do we actually traverse this path? if (!skipFolderTraverse) { // Not a Business Shared Folder that must not be traversed if 'sync_business_shared_folders' is not enabled // Was this path excluded by the 'sync_list' exclusion process if (syncListDirExcluded) { // yes .. this parent path was excluded by the 'sync_list' ... we need to scan this path for potential new data that may be included bool parentalInclusionSyncListRule = selectiveSync.isSyncListPrefixMatch(path); bool syncListAnywhereInclusionRulesExist = selectiveSync.syncListAnywhereInclusionRulesExist(); bool mustTraversePath = false; if ((parentalInclusionSyncListRule) || (syncListAnywhereInclusionRulesExist)) { mustTraversePath = true; } // Log what we are testing if (debugLogging) { addLogEntry("Local path was excluded by 'sync_list' but is this in anyway included in a specific 'inclusion' rule?", ["debug"]); // Is this path in the 'sync_list' inclusion path array? addLogEntry("Testing path against the specific 'sync_list' inclusion rules: " ~ path, ["debug"]); addLogEntry("Should we traverse this local path to scan for new data: " ~ to!string(mustTraversePath), ["debug"]); addLogEntry(" - parentalInclusionSyncListRule: " ~ to!string(parentalInclusionSyncListRule), ["debug"]); addLogEntry(" - syncListAnywhereInclusionRulesExist: " ~ to!string(syncListAnywhereInclusionRulesExist), ["debug"]); } // Was traversal of this excluded path triggered? if (mustTraversePath) { // We must traverse this path .. if (verboseLogging) { // Why ... if (syncListAnywhereInclusionRulesExist) { addLogEntry("Bypassing 'sync_list' exclusion to scan directory for potential new data that may be included due to 'sync_list' anywhere rule existence", ["verbose"]); } else { addLogEntry("Bypassing 'sync_list' exclusion to scan directory for potential new data that may be included", ["verbose"]); } } // Try and go through the excluded directory path try { auto directoryEntries = dirEntries(path, SpanMode.shallow, false); foreach (DirEntry entry; directoryEntries) { currentPath = entry.name; scanPathForNewData(entry.name); } // Clear directoryEntries object.destroy(directoryEntries); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, currentPath); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return as there was an error return; } } } } } } } else { // This path was skipped - why? addLogEntry("Skipping item '" ~ path ~ "' due to the full path exceeding " ~ to!string(maxPathLength) ~ " characters (Microsoft OneDrive limitation)", ["info", "notify"]); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Do we skip this path as it might be an Online Business Shared Folder bool skipBusinessSharedFolder(string path) { // Is this a business account? if (appConfig.accountType == "business") { // search businessSharedFoldersOnlineToSkip for this path if (canFind(businessSharedFoldersOnlineToSkip, path)) { // This path was skipped - why? addLogEntry("Skipping item '" ~ path ~ "' due to this path matching an existing online Business Shared Folder name", ["info", "notify"]); addLogEntry("To sync this Business Shared Folder, consider enabling 'sync_business_shared_folders' within your application configuration.", ["info"]); return true; } } // return value return false; } // Handle a single file inotify trigger when using --monitor void handleLocalFileTrigger(string[] changedLocalFilesToUploadToOneDrive) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Is this path a new file or an existing one? // Normally we would use pathFoundInDatabase() to calculate, but we need 'databaseItem' as well if the item is in the database foreach (localFilePath; changedLocalFilesToUploadToOneDrive) { try { Item databaseItem; bool fileFoundInDB = false; foreach (driveId; onlineDriveDetails.keys) { if (itemDB.selectByPath(localFilePath, driveId, databaseItem)) { fileFoundInDB = true; // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // file found, search no more break; } } // Was the file found in the database? if (!fileFoundInDB) { // This is a new file as it is not in the database // Log that the file has been added locally if (verboseLogging) {addLogEntry("[M] New local file added: " ~ localFilePath, ["verbose"]);} scanLocalFilesystemPathForNewDataToUpload(localFilePath); } else { // This is a potentially modified file, needs to be handled as such. Is the item truly modified? if (!testFileHash(localFilePath, databaseItem)) { // The local file failed the hash comparison test - there is a data difference // Log that the file has changed locally if (verboseLogging) {addLogEntry("[M] Local file changed: " ~ localFilePath, ["verbose"]);} // Add the modified item to the array to upload uploadChangedLocalFileToOneDrive([databaseItem.driveId, databaseItem.id, localFilePath]); } } } catch(Exception e) { addLogEntry("Cannot upload file changes/creation: " ~ e.msg, ["info", "notify"]); } } processNewLocalItemsToUpload(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query the database to determine if this path is within the existing database bool pathFoundInDatabase(string searchPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Normalise input IF required if (!startsWith(searchPath, "./")) { if (searchPath != ".") { // Log that the path needs normalising if (debugLogging) {addLogEntry("searchPath does not start with './' ... searchPath needs normalising", ["debug"]);} searchPath = ensureStartsWithDotSlash(buildNormalizedPath(searchPath)); } } // Check if this path in the database Item databaseItem; if (debugLogging) {addLogEntry("Search DB for this path: " ~ searchPath, ["debug"]);} foreach (driveId; onlineDriveDetails.keys) { if (itemDB.selectByPath(searchPath, driveId, databaseItem)) { if (debugLogging) {addLogEntry("DB Record for search path: " ~ to!string(databaseItem), ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } if (debugLogging) {addLogEntry("Path found in database - early exit", ["debug"]);} return true; // Early exit on finding the path in the DB } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } if (debugLogging) {addLogEntry("Path not found in database after exhausting all driveId entries: " ~ searchPath, ["debug"]);} return false; // Return false if path is not found in any drive } // Create a new directory online on OneDrive // - Test if we can get the parent path details from the database, otherwise we need to search online // for the path flow and create the folder that way void createDirectoryOnline(string thisNewPathToCreate) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Is this a valid path to create? // We need to avoid this sort of error: // // OneDrive generated an error when creating this path: . // ERROR: Microsoft OneDrive API returned an error with the following message: // Error Message: HTTP request returned status code 400 (Bad Request) // Error Reason: Invalid request // Error Code: invalidRequest // Error Timestamp: 2025-08-01T21:08:26 // API Request ID: dca77bd6-1e9a-432a-bc6c-1c6b5380745d if (isRootEquivalent(thisNewPathToCreate)) return; // Log what path we are attempting to create online if (verboseLogging) {addLogEntry("OneDrive Client requested to create this directory online: " ~ thisNewPathToCreate, ["verbose"]);} // Function variables Item parentItem; JSONValue onlinePathData; // Special Folder Handling: Do NOT create the folder online if it is being used for OneDrive Business Shared Files // These are local copy files, in a self created directory structure which is not to be replicated online // Check appConfig.configuredBusinessSharedFilesDirectoryName against 'thisNewPathToCreate' if (canFind(thisNewPathToCreate, baseName(appConfig.configuredBusinessSharedFilesDirectoryName))) { // Log why this is being skipped addLogEntry("Skipping creating '" ~ thisNewPathToCreate ~ "' as this path is used for handling OneDrive Business Shared Files", ["info", "notify"]); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return early as skipping return; } // Create a new API Instance for this thread and initialise it OneDriveApi createDirectoryOnlineOneDriveApiInstance; createDirectoryOnlineOneDriveApiInstance = new OneDriveApi(appConfig); createDirectoryOnlineOneDriveApiInstance.initialise(); // What parent path to use? string parentPath = dirName(thisNewPathToCreate); // will be either . or something else // Configure the parentItem by if this is the account 'root' use the root details, or search the database for the parent details if (parentPath == ".") { // Parent path is '.' which is the account root // Use client defaults parentItem.driveId = appConfig.defaultDriveId; parentItem.id = appConfig.defaultRootId; } else { // Query the parent path online if (debugLogging) {addLogEntry("Attempting to query Local Database for this parent path: " ~ parentPath, ["debug"]);} // Attempt a 2 step process to work out where to create the directory // Step 1: Query the DB first for the parent path, to try and avoid an API call // Step 2: Query online as last resort // Step 1: Check if this parent path in the database Item databaseItem; bool parentPathFoundInDB = false; foreach (driveId; onlineDriveDetails.keys) { // driveId comes from the DB .. trust it is has been validated if (debugLogging) {addLogEntry("Query DB with this driveID for the Parent Path: " ~ driveId, ["debug"]);} // Query the database for this parent path using each driveId that we know about if (itemDB.selectByPath(parentPath, driveId, databaseItem)) { parentPathFoundInDB = true; if (debugLogging) { addLogEntry("Parent databaseItem: " ~ to!string(databaseItem), ["debug"]); addLogEntry("parentPathFoundInDB: " ~ to!string(parentPathFoundInDB), ["debug"]); } // Set parentItem to the item returned from the database parentItem = databaseItem; } } // After querying all DB entries for each driveID for the parent path, what are the details in parentItem? if (debugLogging) {addLogEntry("Parent parentItem after DB Query exhausted: " ~ to!string(parentItem), ["debug"]);} // Step 2: Query for the path online if NOT found in the local database if (!parentPathFoundInDB) { // parent path not found in database try { if (debugLogging) {addLogEntry("Attempting to query OneDrive Online for this parent path as path not found in local database: " ~ parentPath, ["debug"]);} onlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetails(parentPath); if (debugLogging) {addLogEntry("Online Parent Path Query Response: " ~ to!string(onlinePathData), ["debug"]);} // Make the parentItem from the online data parentItem = makeItem(onlinePathData); // Before we 'save' this item to the database, is the parent of this parent in the database? // We need to go and check the grandparent item for this parent item Item grandparentDatabaseItem; bool grandparentInDatabase = itemDB.selectById(onlinePathData["parentReference"]["driveId"].str, onlinePathData["parentReference"]["id"].str, grandparentDatabaseItem); // Is the 'grandparent' in the database? if (!grandparentInDatabase) { // No .. string grandParentPath = dirName(parentPath); // create/add grandparent path online, add to database createDirectoryOnline(grandParentPath); } // Save parent item to the database saveItem(onlinePathData); } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // Parent does not exist ... need to create parent if (debugLogging) {addLogEntry("Parent path does not exist online: " ~ parentPath, ["debug"]);} createDirectoryOnline(parentPath); // no return here as we need to continue, but need to re-query the OneDrive API to get the right parental details now that they exist onlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetails(parentPath); parentItem = makeItem(onlinePathData); } else { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } } } // Make sure the full path does not exist online, this should generate a 404 response, to which then the folder will be created online try { // Try and query the OneDrive API for the path we need to create if (debugLogging) { addLogEntry("Attempting to query OneDrive API for this path: " ~ thisNewPathToCreate, ["debug"]); addLogEntry("parentItem details: " ~ to!string(parentItem), ["debug"]); } // Depending on the data within parentItem, will depend on what method we are using to search // A Shared Folder will be 'remote' so we need to check the remote parent id, rather than parentItem details Item queryItem; // If we are doing a normal sync, 'parentItem.type == ItemType.remote' comparison works // If we are doing a --local-first 'parentItem.type == ItemType.remote' fails as the returned object is not a remote item, but is remote based on the 'driveId' if (parentItem.type == ItemType.remote) { // This folder is a potential shared object if (debugLogging) {addLogEntry("ParentItem is a remote item object", ["debug"]);} // Is this a Personal Account Type or has 'sync_business_shared_items' been enabled? if ((appConfig.accountType == "personal") || (appConfig.getValueBool("sync_business_shared_items"))) { // Update the queryItem values queryItem.driveId = parentItem.remoteDriveId; queryItem.id = parentItem.remoteId; } else { // This is a shared folder location, but we are not a 'personal' account, and 'sync_business_shared_items' has not been enabled addLogEntry("ERROR: Unable to create directory online as 'sync_business_shared_items' is not enabled"); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return as we cannot continue here return; } } else { // Use parent item for the query item if (debugLogging) {addLogEntry("Standard Query, use parentItem", ["debug"]);} queryItem = parentItem; } // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test queryItem.driveId = transformToLowerCase(queryItem.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (queryItem.driveId != appConfig.defaultDriveId) { queryItem.driveId = testProvidedDriveIdForLengthIssue(queryItem.driveId); } } if (queryItem.driveId == appConfig.defaultDriveId) { // Use getPathDetailsByDriveId if (debugLogging) {addLogEntry("Selecting getPathDetailsByDriveId to query OneDrive API for path data", ["debug"]);} onlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetailsByDriveId(queryItem.driveId, thisNewPathToCreate); } else { // Use searchDriveForPath to query OneDrive if (debugLogging) {addLogEntry("Selecting searchDriveForPath to query OneDrive API for path data", ["debug"]);} // If the queryItem.driveId is not our driveId - the path we are looking for will not be at the logical location that getPathDetailsByDriveId // can use - as it will always return a 404 .. even if the path actually exists (which is the whole point of this test) // Search the queryItem.driveId for any folder name match that we are going to create, then compare response JSON items with queryItem.id // If no match, the folder we want to create does not exist at the location we are seeking to create it at, thus generate a 404 onlinePathData = createDirectoryOnlineOneDriveApiInstance.searchDriveForPath(queryItem.driveId, baseName(thisNewPathToCreate)); if (debugLogging) {addLogEntry("onlinePathData: " ~to!string(onlinePathData), ["debug"]);} // Process the response from searching the drive long responseCount = count(onlinePathData["value"].array); if (responseCount > 0) { // Search 'name' matches were found .. need to match these against queryItem.id bool foundDirectoryOnline = false; JSONValue foundDirectoryJSONItem; // Items were returned .. but is one of these what we are looking for? foreach (childJSON; onlinePathData["value"].array) { // Is this item not a file? if (!isFileItem(childJSON)) { Item thisChildItem = makeItem(childJSON); // Direct Match Check if ((queryItem.id == thisChildItem.parentId) && (baseName(thisNewPathToCreate) == thisChildItem.name)) { // High confidence that this child folder is a direct match we are trying to create and it already exists online if (debugLogging) { addLogEntry("Path we are searching for exists online (Direct Match): " ~ baseName(thisNewPathToCreate), ["debug"]); addLogEntry("childJSON: " ~ sanitiseJSONItem(childJSON), ["debug"]); } foundDirectoryOnline = true; foundDirectoryJSONItem = childJSON; break; } // Full Lower Case POSIX Match Check string childAsLower = toLower(childJSON["name"].str); string thisFolderNameAsLower = toLower(baseName(thisNewPathToCreate)); // Child name check if (childAsLower == thisFolderNameAsLower) { // This is a POSIX 'case in-sensitive match' ..... in folder name only // - Local item name has a 'case-insensitive match' to an existing item on OneDrive // The 'parentId' of this JSON object must match the parentId of where the folder was created // - why .. we might have the same folder name, but somewhere totally different if (queryItem.id == thisChildItem.parentId) { // Found the directory in the location, using case in-sensitive matching if (debugLogging) { addLogEntry("Path we are searching for exists online (POSIX 'case in-sensitive match'): " ~ baseName(thisNewPathToCreate), ["debug"]); addLogEntry("childJSON: " ~ sanitiseJSONItem(childJSON), ["debug"]); } foundDirectoryOnline = true; foundDirectoryJSONItem = childJSON; break; } } } } if (foundDirectoryOnline) { // Directory we are seeking was found online ... if (debugLogging) {addLogEntry("The directory we are seeking was found online by using searchDriveForPath ...", ["debug"]);} onlinePathData = foundDirectoryJSONItem; } else { // No 'search item matches found' - raise a 404 so that the exception handling will take over to create the folder throw new OneDriveException(404, "Name not found via search"); } } else { // No 'search item matches found' - raise a 404 so that the exception handling will take over to create the folder throw new OneDriveException(404, "Name not found via search"); } } } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // This is a good error - it means that the directory to create 100% does not exist online // The directory was not found on the drive id we queried if (verboseLogging) {addLogEntry("The requested directory to create was not found on OneDrive - creating remote directory: " ~ thisNewPathToCreate, ["verbose"]);} // Build up the online create directory request string requiredDriveId; string requiredParentItemId; JSONValue createDirectoryOnlineAPIResponse; JSONValue newDriveItem = [ "name": JSONValue(baseName(thisNewPathToCreate)), "folder": parseJSON("{}") ]; // Submit the creation request // Fix for https://github.com/skilion/onedrive/issues/356 if (!dryRun) { try { // Attempt to create a new folder on the required driveId and parent item id // Is the item a Remote Object (Shared Folder) ? if (parentItem.type == ItemType.remote) { // Yes .. Shared Folder if (debugLogging) {addLogEntry("parentItem data: " ~ to!string(parentItem), ["debug"]);} requiredDriveId = parentItem.remoteDriveId; requiredParentItemId = parentItem.remoteId; } else { // Not a Shared Folder requiredDriveId = parentItem.driveId; requiredParentItemId = parentItem.id; } // Where are we creating this new folder? if (debugLogging) { addLogEntry("requiredDriveId: " ~ requiredDriveId, ["debug"]); addLogEntry("requiredParentItemId: " ~ requiredParentItemId, ["debug"]); addLogEntry("newDriveItem JSON: " ~ sanitiseJSONItem(newDriveItem), ["debug"]); } // Create the new folder createDirectoryOnlineAPIResponse = createDirectoryOnlineOneDriveApiInstance.createById(requiredDriveId, requiredParentItemId, newDriveItem); // Log that the directory was created addLogEntry("Successfully created the remote directory " ~ thisNewPathToCreate ~ " on Microsoft OneDrive"); // Is the response a valid JSON object - validation checking done in saveItem, printing of the JSON object is done in saveItem() saveItem(createDirectoryOnlineAPIResponse); } catch (OneDriveException exception) { if (exception.httpStatusCode == 409) { // OneDrive API returned a 404 (far above) to say the directory did not exist // but when we attempted to create it, OneDrive responded that it now already exists with a 409 if (verboseLogging) {addLogEntry("OneDrive reported that " ~ thisNewPathToCreate ~ " already exists .. OneDrive API race condition", ["verbose"]);} // Try to recover race condition by querying the parent's children for the folder we are trying to create createDirectoryOnlineAPIResponse = resolveOnlineCreationRaceCondition(requiredDriveId, requiredParentItemId, thisNewPathToCreate); // Log that the directory details were obtained addLogEntry("Successfully obtained the remote directory details " ~ thisNewPathToCreate ~ " from Microsoft OneDrive"); // Is the response a valid JSON object - validation checking done in saveItem, printing of the JSON object is done in saveItem() saveItem(createDirectoryOnlineAPIResponse); // Shutdown this API instance, as we will create API instances as required, when required createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); // Free object and memory createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } else { // some other error from OneDrive was returned - display what it is addLogEntry("OneDrive generated an error when creating this path: " ~ thisNewPathToCreate); displayOneDriveErrorMessage(exception.msg, thisFunctionName); // Shutdown this API instance, as we will create API instances as required, when required createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); // Free object and memory createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return due to OneDriveException return; } } else { // Simulate a successful 'directory create' & save it to the dryRun database copy addLogEntry("Successfully created the remote directory " ~ thisNewPathToCreate ~ " on Microsoft OneDrive"); // The simulated response has to pass 'makeItem' as part of saveItem auto fakeResponse = createFakeResponse(thisNewPathToCreate); // Save item to the database saveItem(fakeResponse); } // Shutdown this API instance, as we will create API instances as required, when required createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); // Free object and memory createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // shutdown & return return; } else { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within createDirectoryOnlineOneDriveApiInstance // If we get a 400 error, there is an issue creating this folder on Microsoft OneDrive for some reason // If the error is not 400, re-try, else fail if (exception.httpStatusCode != 400) { // Attempt a re-try createDirectoryOnline(thisNewPathToCreate); } else { // We cant create this directory online if (debugLogging) {addLogEntry("This folder cannot be created online: " ~ buildNormalizedPath(absolutePath(thisNewPathToCreate)), ["debug"]);} } } } // If we get to this point - onlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetailsByDriveId(parentItem.driveId, thisNewPathToCreate) generated a 'valid' response .... // This means that the folder potentially exists online .. which is odd .. as it should not have existed if (onlinePathData.type() == JSONType.object) { // A valid object was responded with if (onlinePathData["name"].str == baseName(thisNewPathToCreate)) { // OneDrive 'name' matches local path name if (debugLogging) { addLogEntry("The path to query/search for online was found online", ["debug"]); addLogEntry(" onlinePathData via query/search: " ~ to!string(onlinePathData), ["debug"]); } // Now we know the location of this folder via query/search - go get the actual path details using the 'onlinePathData' Item onlineItem = makeItem(onlinePathData); // Fetch the real data in a consistent manner to ensure the JSON response contains the elements we are expecting JSONValue realOnlinePathData; // Get drive details for the provided driveId try { realOnlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetailsById(onlineItem.driveId, onlineItem.id); if (debugLogging) { addLogEntry(" realOnlinePathData via getPathDetailsById call: " ~ to!string(realOnlinePathData), ["debug"]); } } catch (OneDriveException exception) { // An error was generated if (debugLogging) {addLogEntry("realOnlinePathData = createDirectoryOnlineOneDriveApiInstance.getPathDetailsById(onlineItem.driveId, onlineItem.id) generated a OneDriveException", ["debug"]);} // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); // abort .. return; } // OneDrive Personal Shared Folder Check - Use the REAL online data here if (appConfig.accountType == "personal") { // We are a personal account, this existing online folder, it could be a Shared Online Folder could be a 'Add shortcut to My files' item // Is this a remote folder if (isItemRemote(realOnlinePathData)) { // The folder is a remote item ... if (debugLogging) {addLogEntry("The existing Online Folder and 'realOnlinePathData' indicate this is most likely a OneDrive Personal Shared Folder Link added by 'Add shortcut to My files'", ["debug"]);} // It is a 'remote' JSON item denoting a potential shared folder // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(realOnlinePathData); } } // OneDrive Business Shared Folder Check if (appConfig.accountType == "business") { // We are a business account, this existing online folder, it could be a Shared Online Folder could be a 'Add shortcut to My files' item // Is this a remote folder if (isItemRemote(realOnlinePathData)) { // The folder is a remote item ... if (debugLogging) {addLogEntry("The existing Online Folder and 'realOnlinePathData' indicate this is most likely a OneDrive Shared Business Folder Link added by 'Add shortcut to My files'", ["debug"]);} // Is Shared Business Folder Syncing actually enabled? if (!appConfig.getValueBool("sync_business_shared_items")) { // Shared Business Folder Syncing is NOT enabled if (debugLogging) {addLogEntry("We need to skip this path: " ~ thisNewPathToCreate, ["debug"]);} // Add this path to businessSharedFoldersOnlineToSkip businessSharedFoldersOnlineToSkip ~= [thisNewPathToCreate]; // no save to database, no online create // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return due to skipped path return; } else { // Shared Business Folder Syncing IS enabled // It is a 'remote' JSON item denoting a potential shared folder // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(realOnlinePathData); } } } // Path found online if (verboseLogging) {addLogEntry("The requested directory to create was found on OneDrive - skipping creating the directory online: " ~ thisNewPathToCreate, ["verbose"]);} // Is the response a valid JSON object - validation checking done in saveItem saveItem(realOnlinePathData); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return due to path found online return; } else { // Normally this would throw an error, however we cant use throw new PosixException() string msg = format("POSIX 'case-insensitive match' between '%s' (local) and '%s' (online) which violates the Microsoft OneDrive API namespace convention", baseName(thisNewPathToCreate), onlinePathData["name"].str); displayPosixErrorMessage(msg); addLogEntry("ERROR: Requested directory to create has a 'case-insensitive match' to an existing directory on Microsoft OneDrive online."); addLogEntry("ERROR: To resolve, rename this local directory: " ~ buildNormalizedPath(absolutePath(thisNewPathToCreate))); addLogEntry("Skipping creating this directory online due to 'case-insensitive match': " ~ thisNewPathToCreate); // Add this path to posixViolationPaths posixViolationPaths ~= [thisNewPathToCreate]; // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // manual POSIX exception return; } } else { // response is not valid JSON, an error was returned from OneDrive addLogEntry("ERROR: There was an error performing this operation on Microsoft OneDrive"); addLogEntry("ERROR: Increase logging verbosity to assist determining why."); addLogEntry("Skipping: " ~ buildNormalizedPath(absolutePath(thisNewPathToCreate))); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory createDirectoryOnlineOneDriveApiInstance.releaseCurlEngine(); createDirectoryOnlineOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // generic error return; } } // In the event that the online creation triggered a 404 then a 409 on creation attempt, this function explicitly is used to query that parent for the child being sought // This should return a usable JSON response of the folder being sought JSONValue resolveOnlineCreationRaceCondition(string requiredDriveId, string requiredParentItemId, string thisNewPathToCreate) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Create a new API Instance for this thread and initialise it OneDriveApi raceConditionResolutionOneDriveApiInstance; raceConditionResolutionOneDriveApiInstance = new OneDriveApi(appConfig); raceConditionResolutionOneDriveApiInstance.initialise(); // What is the folder we are seeking string searchFolder = baseName(thisNewPathToCreate); // Where should we store the details of the online folder we are seeking? JSONValue targetOnlineFolderDetails; // Required variables for listChildren to operate JSONValue topLevelChildren; string nextLink; bool directoryFoundOnline = false; // To handle ^c events, we need this Code while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Query this remote object for its children topLevelChildren = raceConditionResolutionOneDriveApiInstance.listChildren(requiredDriveId, requiredParentItemId, nextLink); // Process each child that has been returned foreach (child; topLevelChildren["value"].array) { // We are specifically seeking a 'folder' object if (isItemFolder(child)) { // Is this the child folder we are looking for, and is a POSIX match? // We know that Microsoft OneDrive is not POSIX aware, thus there cannot be 2 folders of the same name with different case sensitivity if (child["name"].str == searchFolder) { // EXACT MATCH including case sensitivity: Flag that we found the folder online directoryFoundOnline = true; // Use these details for raceCondition response targetOnlineFolderDetails = child; break; } else { string childAsLower = toLower(child["name"].str); string thisFolderNameAsLower = toLower(searchFolder); try { if (childAsLower == thisFolderNameAsLower) { // This is a POSIX 'case in-sensitive match' ..... // Local item name has a 'case-insensitive match' to an existing item on OneDrive throw new PosixException(searchFolder, child["name"].str); } } catch (PosixException e) { // Display POSIX error message displayPosixErrorMessage(e.msg); addLogEntry("ERROR: Requested directory to search for and potentially create has a 'case-insensitive match' to an existing directory on Microsoft OneDrive online."); addLogEntry("ERROR: To resolve, rename this local directory: " ~ thisNewPathToCreate); } } } } // That set of returned objects - did we find the folder? if (directoryFoundOnline) { // We found the folder, no need to continue searching nextLink data break; } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in topLevelChildren) { // Update nextLink to next changeSet bundle if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} nextLink = topLevelChildren["@odata.nextLink"].str; } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } // Shutdown this API instance, as we will create API instances as required, when required raceConditionResolutionOneDriveApiInstance.releaseCurlEngine(); // Free object and memory raceConditionResolutionOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the JSON with the folder details return targetOnlineFolderDetails; } // Test that the online name actually matches the requested local name bool performPosixTest(string localNameToCheck, string onlineName) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file // Do not assume case sensitivity. For example, consider the names OSCAR, Oscar, and oscar to be the same, // even though some file systems (such as a POSIX-compliant file system) may consider them as different. // Note that NTFS supports POSIX semantics for case sensitivity but this is not the default behavior. bool posixIssue = false; // Check for a POSIX casing mismatch if (localNameToCheck != onlineName) { // The input items are different .. how are they different? if (toLower(localNameToCheck) == toLower(onlineName)) { // Names differ only by case -> POSIX issue if (debugLogging) {addLogEntry("performPosixTest: Names differ only by case -> POSIX issue", ["debug"]);} // Local item name has a 'case-insensitive match' to an existing item on OneDrive posixIssue = true; } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return the posix evaluation return posixIssue; } // Upload new file items as identified void uploadNewLocalFileItems() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Lets deal with the new local items in a batch process size_t batchSize = to!int(appConfig.getValueLong("threads")); long batchCount = (newLocalFilesToUploadToOneDrive.length + batchSize - 1) / batchSize; long batchesProcessed = 0; // Transfer order string transferOrder = appConfig.getValueString("transfer_order"); // Has the user configured to specify the transfer order of files? if (transferOrder != "default") { // If we have more than 1 item to upload, sort the items if (count(newLocalFilesToUploadToOneDrive) > 1) { // Create an array of tuples (file path, file size) auto fileInfo = newLocalFilesToUploadToOneDrive .map!(file => tuple(file, getSize(file))) // Get file size for each file that needs to be uploaded .array; // Perform sorting based on transferOrder if (transferOrder == "size_asc") { fileInfo.sort!((a, b) => a[1] < b[1]); // sort the array by ascending size } else if (transferOrder == "size_dsc") { fileInfo.sort!((a, b) => a[1] > b[1]); // sort the array by descending size } else if (transferOrder == "name_asc") { fileInfo.sort!((a, b) => a[0] < b[0]); // sort the array by ascending name } else if (transferOrder == "name_dsc") { fileInfo.sort!((a, b) => a[0] > b[0]); // sort the array by descending name } // Extract sorted file paths newLocalFilesToUploadToOneDrive = fileInfo.map!(t => t[0]).array; } } // Process newLocalFilesToUploadToOneDrive foreach (chunk; newLocalFilesToUploadToOneDrive.chunks(batchSize)) { // send an array containing 'appConfig.getValueLong("threads")' local files to upload uploadNewLocalFileItemsInParallel(chunk); } // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Upload the file batches in parallel void uploadNewLocalFileItemsInParallel(string[] array) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function received an array of string items to upload, the number of elements based on appConfig.getValueLong("threads") foreach (i, fileToUpload; processPool.parallel(array)) { if (debugLogging) {addLogEntry("Upload Thread " ~ to!string(i) ~ " Starting: " ~ to!string(Clock.currTime()), ["debug"]);} uploadNewFile(fileToUpload); if (debugLogging) {addLogEntry("Upload Thread " ~ to!string(i) ~ " Finished: " ~ to!string(Clock.currTime()), ["debug"]);} } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Upload a new file to OneDrive void uploadNewFile(string fileToUpload) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Debug for the moment if (debugLogging) {addLogEntry("fileToUpload: " ~ fileToUpload, ["debug"]);} // These are the details of the item we need to upload // How much space is remaining on OneDrive long remainingFreeSpaceOnline; // Did the upload fail? bool uploadFailed = false; // Did we skip due to exceeding maximum allowed size? bool skippedMaxSize = false; // Did we skip to an exception error? bool skippedExceptionError = false; // Is the parent path in the item database? bool parentPathFoundInDB = false; // Get this file size long thisFileSize; // Is there space available online bool spaceAvailableOnline = false; // Flag to track if there is zero data traversal bool zeroDataTraversal = false; DriveDetailsCache cachedOnlineDriveData; long calculatedSpaceOnlinePostUpload; OneDriveApi checkFileOneDriveApiInstance; // Check the database for the parent path of fileToUpload Item parentItem; // What parent path to use? string parentPath = dirName(fileToUpload); // will be either . or something else if (parentPath == "."){ // Assume this is a new file in the users configured sync_dir root // Use client defaults parentItem.id = appConfig.defaultRootId; // Should give something like 12345ABCDE1234A1!101 parentItem.driveId = appConfig.defaultDriveId; // Should give something like 12345abcde1234a1 parentPathFoundInDB = true; } else { // Query the database using each of the driveId's we are using foreach (driveId; onlineDriveDetails.keys) { // Query the database for this parent path using each driveId Item dbResponse; if(itemDB.selectByPath(parentPath, driveId, dbResponse)){ // parent path was found in the database parentItem = dbResponse; parentPathFoundInDB = true; } } } // If the parent path was found in the DB, to ensure we are uploading the right location 'parentItem.driveId' must not be empty if ((parentPathFoundInDB) && (parentItem.driveId.empty)) { // switch to using defaultDriveId if (debugLogging) {addLogEntry("parentItem.driveId is empty - using defaultDriveId for upload API calls", ["debug"]);} parentItem.driveId = appConfig.defaultDriveId; } // Check if the path still exists locally before we try to upload if (exists(fileToUpload)) { // Can we read the file - as a permissions issue or actual file corruption will cause a failure // Resolves: https://github.com/abraunegg/onedrive/issues/113 if (readLocalFile(fileToUpload)) { // The local file can be read - so we can read it to attempt to upload it in this thread // Is the path parent in the DB? if (parentPathFoundInDB) { // Parent path is in the database // Get the new file size // Even if the permissions on the file are: -rw-------. 1 root root 8 Jan 11 09:42 // we can still obtain the file size, however readLocalFile() also tests if the file can be read (permission check) thisFileSize = getSize(fileToUpload); // Does this file exceed the maximum filesize for OneDrive // Resolves: https://github.com/skilion/onedrive/issues/121 , https://github.com/skilion/onedrive/issues/294 , https://github.com/skilion/onedrive/issues/329 if (thisFileSize <= maxUploadFileSize) { // Is there enough free space on OneDrive as compared to when we started this thread, to safely upload the file to OneDrive? // Make sure that parentItem.driveId is in our driveIDs array to use when checking if item is in database // Keep the DriveDetailsCache array with unique entries only if (!canFindDriveId(parentItem.driveId, cachedOnlineDriveData)) { // Add this driveId to the drive cache, which then also sets for the defaultDriveId: // - quotaRestricted; // - quotaAvailable; // - quotaRemaining; addOrUpdateOneDriveOnlineDetails(parentItem.driveId); // Fetch the details from cachedOnlineDriveData cachedOnlineDriveData = getDriveDetails(parentItem.driveId); } // Fetch the details from cachedOnlineDriveData // - cachedOnlineDriveData.quotaRestricted; // - cachedOnlineDriveData.quotaAvailable; // - cachedOnlineDriveData.quotaRemaining; remainingFreeSpaceOnline = cachedOnlineDriveData.quotaRemaining; // When we compare the space online to the total we are trying to upload - is there space online? calculatedSpaceOnlinePostUpload = remainingFreeSpaceOnline - thisFileSize; // Based on what we know, for this thread - can we safely upload this modified local file? if (debugLogging) { string estimatedMessage = format("This Thread (Upload New File) Estimated Free Space Online (%s): ", parentItem.driveId); addLogEntry(estimatedMessage ~ to!string(remainingFreeSpaceOnline), ["debug"]); addLogEntry("This Thread (Upload New File) Calculated Free Space Online Post Upload: " ~ to!string(calculatedSpaceOnlinePostUpload), ["debug"]); } // If 'personal' accounts, if driveId == defaultDriveId, then we will have data - appConfig.quotaAvailable will be updated // If 'personal' accounts, if driveId != defaultDriveId, then we will not have quota data - appConfig.quotaRestricted will be set as true // If 'business' accounts, if driveId == defaultDriveId, then we will have data // If 'business' accounts, if driveId != defaultDriveId, then we will have data, but it will be a 0 value - appConfig.quotaRestricted will be set as true if (remainingFreeSpaceOnline > totalDataToUpload) { // Space available spaceAvailableOnline = true; } else { // we need to look more granular // What was the latest getRemainingFreeSpace() value? if (cachedOnlineDriveData.quotaAvailable) { // Our query told us we have free space online .. if we upload this file, will we exceed space online - thus upload will fail during upload? if (calculatedSpaceOnlinePostUpload > 0) { // Based on this thread action, we believe that there is space available online to upload - proceed spaceAvailableOnline = true; } } } // Is quota being restricted? if (cachedOnlineDriveData.quotaRestricted) { // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { parentItem.driveId = transformToLowerCase(parentItem.driveId); } // If the upload target drive is not our drive id, then it is a shared folder .. we need to print a space warning message if (parentItem.driveId != appConfig.defaultDriveId) { // Different message depending on account type if (appConfig.accountType == "personal") { if (verboseLogging) {addLogEntry("WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.", ["verbose"]);} } else { if (verboseLogging) {addLogEntry("WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.", ["verbose"]);} } } else { if (appConfig.accountType == "personal") { if (verboseLogging) {addLogEntry("WARNING: OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.", ["verbose"]);} } else { if (verboseLogging) {addLogEntry("WARNING: OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.", ["verbose"]);} } } // Space available online is being restricted - so we have no way to really know if there is space available online spaceAvailableOnline = true; } // Do we have space available or is space available being restricted (so we make the blind assumption that there is space available) if (spaceAvailableOnline) { // We need to check that this new local file does not exist on OneDrive JSONValue fileDetailsFromOneDrive; // https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file // Do not assume case sensitivity. For example, consider the names OSCAR, Oscar, and oscar to be the same, // even though some file systems (such as a POSIX-compliant file systems that Linux use) may consider them as different. // Note that NTFS supports POSIX semantics for case sensitivity but this is not the default behavior, OneDrive does not use this. // In order to upload this file - this query HAS to respond with a '404 - Not Found' so that the upload is triggered // Does this 'file' already exist on OneDrive? try { // Create a new API Instance for this thread and initialise it checkFileOneDriveApiInstance = new OneDriveApi(appConfig); checkFileOneDriveApiInstance.initialise(); // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { parentItem.driveId = transformToLowerCase(parentItem.driveId); } if (parentItem.driveId == appConfig.defaultDriveId) { // getPathDetailsByDriveId is only reliable when the driveId is our driveId fileDetailsFromOneDrive = checkFileOneDriveApiInstance.getPathDetailsByDriveId(parentItem.driveId, fileToUpload); } else { // We need to curate a response by listing the children of this parentItem.driveId and parentItem.id , without traversing directories // So that IF the file is on a Shared Folder, it can be found, and, if it exists, checked correctly fileDetailsFromOneDrive = searchDriveItemForFile(parentItem.driveId, parentItem.id, fileToUpload); // Was the file found? if (fileDetailsFromOneDrive.type() != JSONType.object) { // No .... throw new OneDriveException(404, "Name not found via searchDriveItemForFile"); } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory checkFileOneDriveApiInstance.releaseCurlEngine(); checkFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // No 404 which means a file was found with the path we are trying to upload to if (debugLogging) {addLogEntry("fileDetailsFromOneDrive JSON data after exist online check: " ~ to!string(fileDetailsFromOneDrive), ["debug"]);} // Portable Operating System Interface (POSIX) testing of JSON response from OneDrive API if (hasName(fileDetailsFromOneDrive)) { // Perform the POSIX evaluation test against the names if (performPosixTest(baseName(fileToUpload), fileDetailsFromOneDrive["name"].str)) { throw new PosixException(baseName(fileToUpload), fileDetailsFromOneDrive["name"].str); } } else { throw new JsonResponseException("Unable to perform POSIX test as the OneDrive API request generated an invalid JSON response"); } // If we get to this point, the OneDrive API returned a 200 OK with valid JSON data that indicates a 'file' exists at this location already // and that it matches the POSIX filename of the local item we are trying to upload as a new file if (verboseLogging) {addLogEntry("The file we are attempting to upload as a new file already exists on Microsoft OneDrive: " ~ fileToUpload, ["verbose"]);} // Does the data from online match our local file that we are attempting to upload as a new file? if (!disableUploadValidation && performUploadIntegrityValidationChecks(fileDetailsFromOneDrive, fileToUpload, thisFileSize)) { // Need a check here around the 'upload_only' and 'remove_source_files' // Are we in an --upload-only & --remove-source-files scenario? if ((uploadOnly) && (localDeleteAfterUpload)) { // Perform the local file deletion as the file exists online, hash matches, no upload removeLocalFilePostUpload(fileToUpload); // As file is now removed, we have nothing to add to the local database if (debugLogging) {addLogEntry("Skipping adding to database as --upload-only & --remove-source-files configured", ["debug"]);} } else { // No data movement, file exists online, local file matches what is online zeroDataTraversal = true; // Save online item details to the database saveItem(fileDetailsFromOneDrive); } } else { // The local file we are attempting to upload as a new file is different to the existing file online if (debugLogging) {addLogEntry("Triggering newfile upload target already exists edge case, where the online item does not match what we are trying to upload", ["debug"]);} // Issue #2626 | Case 2-2 (resync) // If the 'online' file is newer, this will be overwritten with the file from the local filesystem - potentially constituting online data loss // The file 'version history' online will have to be used to 'recover' the prior online file string changedItemParentDriveId = fileDetailsFromOneDrive["parentReference"]["driveId"].str; string changedItemId = fileDetailsFromOneDrive["id"].str; addLogEntry("Skipping uploading this item as a new file, will upload as a modified file (online file already exists): " ~ fileToUpload); // In order for the processing of the local item as a 'changed' item, unfortunately we need to save the online data of the existing online file to the local DB saveItem(fileDetailsFromOneDrive); // Which file is technically newer? The local file or the remote file? Item onlineFile = makeItem(fileDetailsFromOneDrive); SysTime localModifiedTime = timeLastModified(fileToUpload).toUTC(); SysTime onlineModifiedTime = onlineFile.mtime; // Reduce time resolution to seconds before comparing localModifiedTime.fracSecs = Duration.zero; onlineModifiedTime.fracSecs = Duration.zero; // Which file is newer? if (localModifiedTime >= onlineModifiedTime) { // Upload the locally modified file as-is, as it is newer uploadChangedLocalFileToOneDrive([changedItemParentDriveId, changedItemId, fileToUpload]); } else { // Online is newer, rename local, then upload the renamed file // We need to know the renamed path so we can upload it string renamedPath; // Rename the local path - we WANT this to occur regardless of bypassDataPreservation setting safeBackup(fileToUpload, dryRun, false, renamedPath); // Upload renamed local file as a new file uploadNewFile(renamedPath); // Process the database entry removal for the original file. In a --dry-run scenario, this is being done against a DB copy. // This is done so we can download the newer online file itemDB.deleteById(changedItemParentDriveId, changedItemId); } } } catch (OneDriveException exception) { // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory checkFileOneDriveApiInstance.releaseCurlEngine(); checkFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // If we get a 404 .. the file is not online .. this is what we want .. file does not exist online if (exception.httpStatusCode == 404) { // The file has been checked, client side filtering checked, does not exist online - we need to upload it if (debugLogging) {addLogEntry("fileDetailsFromOneDrive = checkFileOneDriveApiInstance.getPathDetailsByDriveId(parentItem.driveId, fileToUpload); generated a 404 - file does not exist online - must upload it", ["debug"]);} uploadFailed = performNewFileUpload(parentItem, fileToUpload, thisFileSize); } else { // some other error // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } catch (PosixException e) { // Display POSIX error message displayPosixErrorMessage(e.msg); addLogEntry("ERROR: Requested file to upload has a 'case-insensitive match' to an existing item on Microsoft OneDrive online."); addLogEntry("ERROR: To resolve, rename this local file: " ~ fileToUpload); addLogEntry("Skipping uploading this new file due to 'case-insensitive match': " ~ fileToUpload); uploadFailed = true; } catch (JsonResponseException e) { // Display JSON error message if (debugLogging) {addLogEntry(e.msg, ["debug"]);} uploadFailed = true; } } else { // skip file upload - insufficient space to upload addLogEntry("Skipping uploading this new file as it exceeds the available free space on Microsoft OneDrive: " ~ fileToUpload); uploadFailed = true; } } else { // Skip file upload - too large addLogEntry("Skipping uploading this new file as it exceeds the maximum size allowed by Microsoft OneDrive: " ~ fileToUpload); uploadFailed = true; } } else { // why was the parent path not in the database? if (canFind(posixViolationPaths, parentPath)) { addLogEntry("ERROR: POSIX 'case-insensitive match' for the parent path which violates the Microsoft OneDrive API namespace convention."); } else { addLogEntry("ERROR: Parent path is not in the database or online: " ~ parentPath); } addLogEntry("ERROR: Unable to upload this file: " ~ fileToUpload); uploadFailed = true; } } else { // Unable to read local file addLogEntry("Skipping uploading this file as it cannot be read (file permissions or file corruption): " ~ fileToUpload); uploadFailed = true; } } else { // File disappeared before upload addLogEntry("File disappeared locally before upload: " ~ fileToUpload); // dont set uploadFailed = true; as the file disappeared before upload, thus nothing here failed } // Upload success or failure? if (!uploadFailed) { // Did we actually upload a file - that is, potentially change the online quota available state? if (!zeroDataTraversal) { // Update the 'cachedOnlineDriveData' record for this 'dbItem.driveId' so that this is tracked as accurately as possible for other threads updateDriveDetailsCache(parentItem.driveId, cachedOnlineDriveData.quotaRestricted, cachedOnlineDriveData.quotaAvailable, thisFileSize); } else { // There was zero data traversal if (debugLogging) {addLogEntry("No file upload, no data movement - cachedOnlineDriveData.quotaRemaining = " ~ to!string(cachedOnlineDriveData.quotaRemaining), ["debug"]);} } } else { // Need to add this to fileUploadFailures to capture at the end fileUploadFailures ~= fileToUpload; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the actual upload to OneDrive bool performNewFileUpload(Item parentItem, string fileToUpload, long thisFileSize) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Assume that by default the upload fails bool uploadFailed = true; // OneDrive API Upload Response JSONValue uploadResponse; // Create the OneDriveAPI Upload Instance OneDriveApi uploadFileOneDriveApiInstance; // Capture what time this upload started SysTime uploadStartTime = Clock.currTime(); // Is this a dry-run scenario? if (!dryRun) { // Not a dry-run situation // Do we use simpleUpload or create an upload session? bool useSimpleUpload = false; // What upload method should be used? if (thisFileSize <= sessionThresholdFileSize) { useSimpleUpload = true; } // Use Session Upload regardless if (appConfig.getValueBool("force_session_upload")) { // Forcing session upload if (debugLogging) {addLogEntry("Forcing to perform upload using a session (newfile)", ["debug"]);} useSimpleUpload = false; } // We can only upload zero size files via simpleFileUpload regardless of account type // Reference: https://github.com/OneDrive/onedrive-api-docs/issues/53 // Additionally, only where file size is < 4MB should be uploaded by simpleUpload - everything else should use a session to upload if ((thisFileSize == 0) || (useSimpleUpload)) { try { // Initialise API for simple upload uploadFileOneDriveApiInstance = new OneDriveApi(appConfig); uploadFileOneDriveApiInstance.initialise(); // Attempt to upload the zero byte file using simpleUpload for all account types uploadResponse = uploadFileOneDriveApiInstance.simpleUpload(fileToUpload, parentItem.driveId, parentItem.id, baseName(fileToUpload)); uploadFailed = false; addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... done", fileTransferNotifications()); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException exception) { // An error was responded with - what was it // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); displayOneDriveErrorMessage(exception.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (FileException e) { // display the error message addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); displayFileSystemErrorMessage(e.msg, thisFunctionName, fileToUpload); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } } else { // Initialise API for session upload uploadFileOneDriveApiInstance = new OneDriveApi(appConfig); uploadFileOneDriveApiInstance.initialise(); // Session Upload for this criteria: // - Personal Account and file size > 4MB // - All Business | Office365 | SharePoint files > 0 bytes JSONValue uploadSessionData; // As this is a unique thread, the sessionFilePath for where we save the data needs to be unique // The best way to do this is generate a 10 digit alphanumeric string, and use this as the file extension string threadUploadSessionFilePath = appConfig.uploadSessionFilePath ~ "." ~ generateAlphanumericString(); // Attempt to upload the > 4MB file using an upload session for all account types try { // Create the Upload Session uploadSessionData = createSessionForFileUpload(uploadFileOneDriveApiInstance, fileToUpload, parentItem.driveId, parentItem.id, baseName(fileToUpload), null, threadUploadSessionFilePath); } catch (OneDriveException exception) { // An error was responded with - what was it // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); displayOneDriveErrorMessage(exception.msg, thisFunctionName); } catch (FileException e) { // display the error message addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); displayFileSystemErrorMessage(e.msg, thisFunctionName, fileToUpload); } // Do we have a valid session URL that we can use ? if (uploadSessionData.type() == JSONType.object) { // This is a valid JSON object bool sessionDataValid = true; // Validate that we have the following items which we need if (!hasUploadURL(uploadSessionData)) { sessionDataValid = false; if (debugLogging) {addLogEntry("Session data missing 'uploadUrl'", ["debug"]);} } if (!hasNextExpectedRanges(uploadSessionData)) { sessionDataValid = false; if (debugLogging) {addLogEntry("Session data missing 'nextExpectedRanges'", ["debug"]);} } if (!hasLocalPath(uploadSessionData)) { sessionDataValid = false; if (debugLogging) {addLogEntry("Session data missing 'localPath'", ["debug"]);} } if (sessionDataValid) { // We have a valid Upload Session Data we can use try { // Try and perform the upload session uploadResponse = performSessionFileUpload(uploadFileOneDriveApiInstance, thisFileSize, uploadSessionData, threadUploadSessionFilePath); if (uploadResponse.type() == JSONType.object) { uploadFailed = false; addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... done", fileTransferNotifications()); } else { addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); uploadFailed = true; } } catch (OneDriveException exception) { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } else { // No Upload URL or nextExpectedRanges or localPath .. not a valid JSON we can use if (verboseLogging) {addLogEntry("Session data is missing required elements to perform a session upload.", ["verbose"]);} addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); } } else { // Create session Upload URL failed addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... failed!", ["info", "notify"]); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } } else { // We are in a --dry-run scenario uploadResponse = createFakeResponse(fileToUpload); uploadFailed = false; addLogEntry("Uploading new file: " ~ fileToUpload ~ " ... done", fileTransferNotifications()); } // If no upload failure, calculate transfer metrics, perform integrity validation if (!uploadFailed) { // Upload did not fail ... // As no upload failure, calculate transfer metrics in a consistent manner displayTransferMetrics(fileToUpload, thisFileSize, uploadStartTime, Clock.currTime()); // OK as the upload did not fail, we need to save the response from OneDrive, but it has to be a valid JSON response if (uploadResponse.type() == JSONType.object) { // check if the path still exists locally before we try to set the file times online - as short lived files, whilst we uploaded it - it may not exist locally already if (exists(fileToUpload)) { // Are we in a --dry-run scenario if (!dryRun) { bool uploadIntegrityPassed; // Check the integrity of the uploaded file, if the local file still exists uploadIntegrityPassed = performUploadIntegrityValidationChecks(uploadResponse, fileToUpload, thisFileSize); // Update the file modified time on OneDrive and save item details to database // Update the item's metadata on OneDrive SysTime mtime = timeLastModified(fileToUpload).toUTC(); mtime.fracSecs = Duration.zero; string newFileId = uploadResponse["id"].str; string newFileETag = uploadResponse["eTag"].str; // Attempt to update the online date time stamp based on our local data if (appConfig.accountType == "personal") { // Business | SharePoint we used a session to upload the data, thus, local timestamps are given when the session is created uploadLastModifiedTime(parentItem, parentItem.driveId, newFileId, mtime, newFileETag); } else { // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. // This means that the file which was uploaded, is potentially no longer the file we have locally // There are 2 ways to solve this: // 1. Download the modified file immediately after upload as per v2.4.x (default) // 2. Create a new online version of the file, which then contributes to the users 'quota' if (!uploadIntegrityPassed) { // upload integrity check failed // We do not want to create a new online file version .. unless configured to do so if (!appConfig.getValueBool("create_new_file_version")) { // are we in an --upload-only scenario if(!uploadOnly){ // Download the now online modified file addLogEntry("WARNING: Microsoft OneDrive modified your uploaded file via its SharePoint 'enrichment' feature. To keep your local and online versions consistent, the altered file will now be downloaded."); addLogEntry("WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details."); // Download the file directly using the prior upload JSON response downloadFileItem(uploadResponse, true); } else { // --upload-only being used // we are not downloading a file, warn that file differences will exist addLogEntry("WARNING: The file uploaded to Microsoft OneDrive has been modified through its SharePoint 'enrichment' process and no longer matches your local version."); addLogEntry("WARNING: The online metadata will now be modified to match your local file which will create a new file version."); addLogEntry("WARNING: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details."); // Create a new online version of the file by updating the metadata - this ensures that the file we uploaded is the file online uploadLastModifiedTime(parentItem, parentItem.driveId, newFileId, mtime, newFileETag); } } else { // Create a new online version of the file by updating the metadata, which negates the need to download the file uploadLastModifiedTime(parentItem, parentItem.driveId, newFileId, mtime, newFileETag); } } else { // integrity checks passed // save the uploadResponse to the database saveItem(uploadResponse); } } } // Are we in an --upload-only & --remove-source-files scenario? // Use actual config values as we are doing an upload session recovery if ((uploadOnly) && (localDeleteAfterUpload)) { // Perform the local file deletion removeLocalFilePostUpload(fileToUpload); } } else { // will be removed in different event! addLogEntry("File disappeared locally after upload: " ~ fileToUpload); } } else { // Log that an invalid JSON object was returned if (debugLogging) {addLogEntry("uploadFileOneDriveApiInstance.simpleUpload or session.upload call returned an invalid JSON Object from the OneDrive API", ["debug"]);} } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return upload status return uploadFailed; } // Create the OneDrive Upload Session JSONValue createSessionForFileUpload(OneDriveApi activeOneDriveApiInstance, string fileToUpload, string parentDriveId, string parentId, string filename, string eTag, string threadUploadSessionFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Upload file via a OneDrive API session JSONValue uploadSession; // Calculate modification time SysTime localFileLastModifiedTime = timeLastModified(fileToUpload).toUTC(); localFileLastModifiedTime.fracSecs = Duration.zero; // Construct the fileSystemInfo JSON component needed to create the Upload Session JSONValue fileSystemInfo = [ "item": JSONValue([ "@microsoft.graph.conflictBehavior": JSONValue("replace"), "fileSystemInfo": JSONValue([ "lastModifiedDateTime": localFileLastModifiedTime.toISOExtString() ]) ]) ]; // Try to create the upload session for this file uploadSession = activeOneDriveApiInstance.createUploadSession(parentDriveId, parentId, filename, eTag, fileSystemInfo); if (uploadSession.type() == JSONType.object) { // a valid session object was created if ("uploadUrl" in uploadSession) { // Add the file path we are uploading to this JSON Session Data uploadSession["localPath"] = fileToUpload; // Save this session saveSessionFile(threadUploadSessionFilePath, uploadSession); } // When does this upload URL expire? displayUploadSessionExpiry(uploadSession); } else { // no valid session was created if (verboseLogging) {addLogEntry("Creation of OneDrive API Upload Session failed.", ["verbose"]);} // return upload() will return a JSONValue response, create an empty JSONValue response to return uploadSession = null; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the JSON return uploadSession; } // Display upload session expiry time void displayUploadSessionExpiry(JSONValue uploadSessionData) { try { // Step 1: Extract the ISO 8601 UTC string from the JSON string utcExpiry = uploadSessionData["expirationDateTime"].str; // Step 2: Convert ISO 8601 string to SysTime (assumes Zulu / UTC timezone) SysTime expiryUTC = SysTime.fromISOExtString(utcExpiry); // Step 3: Convert to local time auto expiryLocal = expiryUTC.toLocalTime(); // Step 4: Print both UTC and Local times if (debugLogging) { addLogEntry("Upload session URL expires at (UTC): " ~ to!string(expiryUTC), ["debug"]); addLogEntry("Upload session URL expires at (Local): " ~ to!string(expiryLocal), ["debug"]); } } catch (Exception e) { // nothing } } // Save the session upload data void saveSessionFile(string threadUploadSessionFilePath, JSONValue uploadSessionData) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } try { std.file.write(threadUploadSessionFilePath, uploadSessionData.toString()); } catch (FileException e) { // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, threadUploadSessionFilePath); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform the upload of file via the Upload Session that was created JSONValue performSessionFileUpload(OneDriveApi activeOneDriveApiInstance, long thisFileSize, JSONValue uploadSessionData, string threadUploadSessionFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Response for upload JSONValue uploadResponse; // https://learn.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#upload-bytes-to-the-upload-session // You can upload the entire file, or split the file into multiple byte ranges, as long as the maximum bytes in any given request is less than 60 MiB. // Calculate File Fragment Size (must be valid multiple of 320 KiB) long baseSize; long fragmentSize; enum CHUNK_SIZE = 327_680L; // 320 KiB enum MAX_FRAGMENT_BYTES = 60L * 1_048_576L; // 60 MiB = 62,914,560 bytes // Time sensitive and ETA string items SysTime currentTime = Clock.currTime(); long start_unix_time = currentTime.toUnixTime(); int h, m, s; string etaString; // Upload string template string uploadLogEntry = "Uploading: " ~ uploadSessionData["localPath"].str ~ " ... "; // Calculate base size using configured fragment size baseSize = appConfig.getValueLong("file_fragment_size") * 2^^20; // Ensure 'fragmentSize' is a multiple of 327680 bytes and < 60 MiB if (baseSize >= MAX_FRAGMENT_BYTES) { // Use the maximum valid size below 60 MiB, rounded down to nearest 320 KiB multiple fragmentSize = ((MAX_FRAGMENT_BYTES - 1) / CHUNK_SIZE) * CHUNK_SIZE; } else { fragmentSize = (baseSize / CHUNK_SIZE) * CHUNK_SIZE; } // Set the fragment count and fragSize size_t fragmentCount = 0; long fragSize = 0; // Extract current upload offset from session data long offset = uploadSessionData["nextExpectedRanges"][0].str.splitter('-').front.to!long; // Estimate total number of expected fragments size_t expected_total_fragments = cast(size_t) ceil(double(thisFileSize) / double(fragmentSize)); // If we get a 404, create a new upload session and store it here JSONValue newUploadSession; // Start the session upload using the active API instance for this thread while (true) { // fragment upload fragmentCount++; if (debugLogging) {addLogEntry("Fragment: " ~ to!string(fragmentCount) ~ " of " ~ to!string(expected_total_fragments), ["debug"]);} // Generate ETA time output etaString = formatETA(calc_eta((fragmentCount -1), expected_total_fragments, start_unix_time)); // Calculate this progress output auto ratio = cast(double)(fragmentCount - 1) / expected_total_fragments; // Convert the ratio to a percentage and format it to two decimal places string percentage = leftJustify(format("%d%%", cast(int)(ratio * 100)), 5, ' '); addLogEntry(uploadLogEntry ~ percentage ~ etaString, ["consoleOnly"]); // What fragment size will be used? if (debugLogging) {addLogEntry("fragmentSize: " ~ to!string(fragmentSize) ~ " offset: " ~ to!string(offset) ~ " thisFileSize: " ~ to!string(thisFileSize), ["debug"]);} fragSize = fragmentSize < thisFileSize - offset ? fragmentSize : thisFileSize - offset; if (debugLogging) {addLogEntry("Using fragSize: " ~ to!string(fragSize), ["debug"]);} // fragSize must not be a negative value if (fragSize < 0) { // Session upload will fail // not a JSON object - fragment upload failed if (verboseLogging) {addLogEntry("File upload session failed - invalid calculation of fragment size", ["verbose"]);} if (exists(threadUploadSessionFilePath)) { safeRemove(threadUploadSessionFilePath); } // set uploadResponse to null as error uploadResponse = null; return uploadResponse; } // If the resume upload fails, we need to check for a return code here try { uploadResponse = activeOneDriveApiInstance.uploadFragment( uploadSessionData["uploadUrl"].str, uploadSessionData["localPath"].str, offset, fragSize, thisFileSize ); } catch (OneDriveException exception) { // if a 100 uploadResponse is generated, continue if (exception.httpStatusCode == 100) { continue; } // Issue #3355: https://github.com/abraunegg/onedrive/issues/3355 if (exception.httpStatusCode == 403 && (exception.msg.canFind("accessDenied") || exception.msg.canFind("You do not have authorization to access the file"))) { addLogEntry("ERROR: Upload session has expired (403 - Access Denied)"); addLogEntry("Probable Cause: The 'tempauth' token embedded in the upload URL has most likely expired."); addLogEntry(" Microsoft issues this token when the upload session is first created. It cannot be refreshed, extended, or queried for its expiry time."); addLogEntry(" The only way to infer its validity is by measuring the time from session creation to this 403 failure."); addLogEntry(" The upload session URL itself may still appear active (based on expirationDateTime), but the upload URL is no longer usable once this 'tempauth' token expires."); addLogEntry(" A new upload session will now be created. Upload will restart from the beginning using the new session URL and new 'tempauth' token."); // Attempt creation of new upload session newUploadSession = createSessionForFileUpload( activeOneDriveApiInstance, uploadSessionData["localPath"].str, uploadSessionData["targetDriveId"].str, uploadSessionData["targetParentId"].str, baseName(uploadSessionData["localPath"].str), null, threadUploadSessionFilePath ); // Attempt retry (which will start upload again from scratch) with new session upload URL continue; } // There was an error uploadResponse from OneDrive when uploading the file fragment if (exception.httpStatusCode == 404) { // The upload session was not found .. ?? we just created it .. maybe the backend is still creating it or failed to create it if (debugLogging) {addLogEntry("The upload session was not found .... re-create session");} newUploadSession = createSessionForFileUpload( activeOneDriveApiInstance, uploadSessionData["localPath"].str, uploadSessionData["targetDriveId"].str, uploadSessionData["targetParentId"].str, baseName(uploadSessionData["localPath"].str), null, threadUploadSessionFilePath ); } // Issue https://github.com/abraunegg/onedrive/issues/2747 // if a 416 uploadResponse is generated, continue if (exception.httpStatusCode == 416) { continue; } // Handle transient errors: // 408 - Request Time Out // 429 - Too Many Requests // 503 - Service Unavailable // 504 - Gateway Timeout // Insert a new line as well, so that the below error is inserted on the console in the right location if (verboseLogging) {addLogEntry("Fragment upload failed - received an exception response from OneDrive API", ["verbose"]);} // display what the error is if we have not already continued if (exception.httpStatusCode != 404) { displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // retry fragment upload in case error is transient if (verboseLogging) {addLogEntry("Retrying fragment upload", ["verbose"]);} // Retry fragment upload logic try { string effectiveRetryUploadURL; string effectiveLocalPath; // If we re-created the session, use the new data on re-try if (newUploadSession.type() == JSONType.object) { if ("uploadUrl" in newUploadSession) { // get this from 'newUploadSession' effectiveRetryUploadURL = newUploadSession["uploadUrl"].str; effectiveLocalPath = newUploadSession["localPath"].str; } else { // get this from the original input effectiveRetryUploadURL = uploadSessionData["uploadUrl"].str; effectiveLocalPath = uploadSessionData["localPath"].str; } // retry the fragment upload uploadResponse = activeOneDriveApiInstance.uploadFragment( effectiveRetryUploadURL, effectiveLocalPath, offset, fragSize, thisFileSize ); } else { // newUploadSession not a JSON uploadResponse = null; return uploadResponse; } } catch (OneDriveException e) { // OneDrive threw another error on retry if (verboseLogging) {addLogEntry("Retry to upload fragment failed", ["verbose"]);} // display what the error is displayOneDriveErrorMessage(e.msg, thisFunctionName); // set uploadResponse to null as the fragment upload was in error twice uploadResponse = null; } catch (std.exception.ErrnoException e) { // There was a file system error - display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, newUploadSession["localPath"].str); return uploadResponse; } } catch (ErrnoException e) { // There was a file system error // display the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, uploadSessionData["localPath"].str); uploadResponse = null; return uploadResponse; } // was the fragment uploaded without issue? if (uploadResponse.type() == JSONType.object) { // Fragment uploaded if (debugLogging) {addLogEntry("Fragment upload complete", ["debug"]);} // Use updated offset from response, not fixed increment if ("nextExpectedRanges" in uploadResponse && uploadResponse["nextExpectedRanges"].type() == JSONType.array && !uploadResponse["nextExpectedRanges"].array.empty) { offset = uploadResponse["nextExpectedRanges"].array[0].str.splitter('-').front.to!long; } else { // No nextExpectedRanges? Assume upload complete break; } // update the uploadSessionData details uploadSessionData["expirationDateTime"] = uploadResponse["expirationDateTime"]; uploadSessionData["nextExpectedRanges"] = uploadResponse["nextExpectedRanges"]; // Log URL 'updated' expirationDateTime as 'UTC' and 'localTime' if (debugLogging) { // Convert expiration time to localTime string utcExpiry = uploadResponse["expirationDateTime"].str; SysTime expiryUTC = SysTime.fromISOExtString(utcExpiry); SysTime expiryLocal = expiryUTC.toLocalTime(); // Display updated URL expiry as UTC and localTime addLogEntry("Upload Session URL expiration extended to (UTC): " ~ to!string(expiryUTC), ["debug"]); addLogEntry("Upload Session URL expiration extended to (Local): " ~ to!string(expiryLocal), ["debug"]); addLogEntry("", ["debug"]); // Add new line as this fragment is complete } // Save for reuse saveSessionFile(threadUploadSessionFilePath, uploadSessionData); } else { // not a JSON object - fragment upload failed if (verboseLogging) {addLogEntry("File upload session failed - invalid response from OneDrive API", ["verbose"]);} // cleanup session data if (exists(threadUploadSessionFilePath)) { safeRemove(threadUploadSessionFilePath); } // set uploadResponse to null as error uploadResponse = null; return uploadResponse; } } // Upload complete long end_unix_time = Clock.currTime.toUnixTime(); auto upload_duration = cast(int)(end_unix_time - start_unix_time); dur!"seconds"(upload_duration).split!("hours", "minutes", "seconds")(h, m, s); etaString = format!"| DONE in %02d:%02d:%02d"(h, m, s); addLogEntry(uploadLogEntry ~ "100% " ~ etaString, ["consoleOnly"]); // Remove session file if it exists if (exists(threadUploadSessionFilePath)) { safeRemove(threadUploadSessionFilePath); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the session upload response return uploadResponse; } // Delete an item on OneDrive void uploadDeletedItem(Item itemToDelete, string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } OneDriveApi uploadDeletedItemOneDriveApiInstance; // Are we in a situation where we HAVE to keep the data online - do not delete the remote object if (noRemoteDelete) { if ((itemToDelete.type == ItemType.dir)) { // Do not process remote directory delete if (verboseLogging) {addLogEntry("Skipping remote directory delete as --upload-only & --no-remote-delete configured", ["verbose"]);} } else { // Do not process remote file delete if (verboseLogging) {addLogEntry("Skipping remote file delete as --upload-only & --no-remote-delete configured", ["verbose"]);} } } else { // Is this a --download-only operation? if (!appConfig.getValueBool("download_only")) { // Process the delete - delete the object online addLogEntry("Deleting item from Microsoft OneDrive: " ~ path, fileTransferNotifications()); bool flagAsBigDelete = false; Item[] children; long itemsToDelete; if ((itemToDelete.type == ItemType.dir)) { // Query the database - how many objects will this remove? children = getChildren(itemToDelete.driveId, itemToDelete.id); // Count the returned items + the original item (1) itemsToDelete = count(children) + 1; if (debugLogging) {addLogEntry("Number of items online to delete: " ~ to!string(itemsToDelete), ["debug"]);} } else { itemsToDelete = 1; } // Clear array children = []; // A local delete of a file|folder when using --monitor will issue a inotify event, which will trigger the local & remote data immediately be deleted // The user may also be --sync process, so we are checking if something was deleted between application use if (itemsToDelete >= appConfig.getValueLong("classify_as_big_delete")) { // A big delete has been detected flagAsBigDelete = true; if (!appConfig.getValueBool("force")) { // Send this message to the GUI addLogEntry("ERROR: An attempt to remove a large volume of data from OneDrive has been detected. Exiting client to preserve your data on Microsoft OneDrive", ["info", "notify"]); // Additional application logging addLogEntry("ERROR: The total number of items being deleted is: " ~ to!string(itemsToDelete)); addLogEntry("ERROR: To delete a large volume of data use --force or increase the config value 'classify_as_big_delete' to a larger value"); // Must exit here to preserve data on online , allow logging to be done forceExit(); } } // Are we in a --dry-run scenario? if (!dryRun) { // We are not in a dry run scenario if (debugLogging) { addLogEntry("itemToDelete: " ~ to!string(itemToDelete), ["debug"]); // what item are we trying to delete? addLogEntry("Attempting to delete this single item id: " ~ itemToDelete.id ~ " from drive: " ~ itemToDelete.driveId, ["debug"]); } // Configure these item variables to handle OneDrive Business Shared Folder Deletion Item actualItemToDelete; Item remoteShortcutLinkItem; // OneDrive Shared Folder Link Handling // - If the item to delete is on a remote drive ... technically we do not own this and should not be deleting this online // We should however be deleting the 'link' in our account online, and, remove the DB link entries (root / folder DB Tie records) bool businessSharingEnabled = false; // OneDrive Business Shared Folder Deletion Handling // Is this a Business Account with Sync Business Shared Items enabled? if ((appConfig.accountType == "business") && (appConfig.getValueBool("sync_business_shared_items"))) { // Syncing Business Shared Items is enabled businessSharingEnabled = true; } // Is this a 'personal' account type or is this a Business Account with Sync Business Shared Items enabled? if ((appConfig.accountType == "personal") || businessSharingEnabled) { // Personal account type or syncing Business Shared Items is enabled // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { itemToDelete.driveId = transformToLowerCase(itemToDelete.driveId); } // Is the 'drive' where this is to be deleted on 'our' drive or is this a remote 'drive' ? if (itemToDelete.driveId != appConfig.defaultDriveId) { // The item to delete is on a remote drive ... this must be handled in a specific way if (itemToDelete.type == ItemType.dir) { // Select the 'remote' database object type using these details // Get the DB entry for this 'remote' item itemDB.selectRemoteTypeByRemoteDriveId(itemToDelete.driveId, itemToDelete.id, remoteShortcutLinkItem); } } // We potentially now have the correct details to delete in our account if (remoteShortcutLinkItem.type == ItemType.remote) { // A valid 'remote' DB entry was returned if (debugLogging) {addLogEntry("remoteShortcutLinkItem: " ~ to!string(remoteShortcutLinkItem), ["debug"]);} // Set actualItemToDelete to this data actualItemToDelete = remoteShortcutLinkItem; // Delete the shortcut reference in the local database if (appConfig.accountType == "personal") { // Personal Shared Folder deletion message if (debugLogging) {addLogEntry("Deleted OneDrive Personal Shared Folder 'Shortcut Link'", ["debug"]);} } else { // Business Shared Folder deletion message if (debugLogging) {addLogEntry("Deleted OneDrive Business Shared Folder 'Shortcut Link'", ["debug"]);} } // Perform action deletion from database itemDB.deleteById(remoteShortcutLinkItem.driveId, remoteShortcutLinkItem.id); } else { // No data was returned, use the original data actualItemToDelete = itemToDelete; } } else { // Set actualItemToDelete to original data actualItemToDelete = itemToDelete; } // Try the online deletion using the 'actualItemToDelete' values try { // Create new OneDrive API Instance uploadDeletedItemOneDriveApiInstance = new OneDriveApi(appConfig); uploadDeletedItemOneDriveApiInstance.initialise(); if (!permanentDelete) { // Perform the delete via the default OneDrive API instance uploadDeletedItemOneDriveApiInstance.deleteById(actualItemToDelete.driveId, actualItemToDelete.id); } else { // Perform the permanent delete via the default OneDrive API instance uploadDeletedItemOneDriveApiInstance.permanentDeleteById(actualItemToDelete.driveId, actualItemToDelete.id); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadDeletedItemOneDriveApiInstance.releaseCurlEngine(); uploadDeletedItemOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException e) { if (e.httpStatusCode == 404) { // item.id, item.eTag could not be found on the specified driveId if (verboseLogging) {addLogEntry("OneDrive reported: The resource could not be found to be deleted.", ["verbose"]);} } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadDeletedItemOneDriveApiInstance.releaseCurlEngine(); uploadDeletedItemOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } // Delete the reference in the local database - use the original input itemDB.deleteById(itemToDelete.driveId, itemToDelete.id); // Was the original item a 'Shared Folder' ? if (remoteShortcutLinkItem.type == ItemType.remote) { // Are there any other 'children' for itemToDelete parent ... this parent may have other Shared Folders added to our account that we have not removed .. Item[] remainingChildren; remainingChildren ~= itemDB.selectChildren(itemToDelete.driveId, itemToDelete.parentId); // Only if there are zero children for this parent item, remove the 'root' record if (count(remainingChildren) == 0) { // No more children for this parental object itemDB.deleteById(itemToDelete.driveId, itemToDelete.parentId); } } } else { // log that this is a dry-run activity addLogEntry("DRY-RUN: No delete activity"); } } else { // --download-only operation, we are not uploading any delete event to OneDrive if (debugLogging) {addLogEntry("Not pushing local delete to Microsoft OneDrive due to --download-only being used", ["debug"]);} } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Get the children of an item id from the database Item[] getChildren(string driveId, string id) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } Item[] children; children ~= itemDB.selectChildren(driveId, id); foreach (Item child; children) { if (child.type != ItemType.file) { // recursively get the children of this child children ~= getChildren(child.driveId, child.id); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return the database records return children; } // Perform a 'reverse' delete of all child objects on OneDrive void performReverseDeletionOfOneDriveItems(Item[] children, Item itemToDelete) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Log what is happening if (debugLogging) {addLogEntry("Attempting a reverse delete of all child objects from OneDrive", ["debug"]);} // Create a new API Instance for this thread and initialise it OneDriveApi performReverseDeletionOneDriveApiInstance; performReverseDeletionOneDriveApiInstance = new OneDriveApi(appConfig); performReverseDeletionOneDriveApiInstance.initialise(); foreach_reverse (Item child; children) { // Log the action if (debugLogging) {addLogEntry("Attempting to delete this child item id: " ~ child.id ~ " from drive: " ~ child.driveId, ["debug"]);} if (!permanentDelete) { // Perform the delete via the default OneDrive API instance performReverseDeletionOneDriveApiInstance.deleteById(child.driveId, child.id, child.eTag); } else { // Perform the permanent delete via the default OneDrive API instance performReverseDeletionOneDriveApiInstance.permanentDeleteById(child.driveId, child.id, child.eTag); } // delete the child reference in the local database itemDB.deleteById(child.driveId, child.id); } // Log the action if (debugLogging) {addLogEntry("Attempting to delete this parent item id: " ~ itemToDelete.id ~ " from drive: " ~ itemToDelete.driveId, ["debug"]);} if (!permanentDelete) { // Perform the delete via the default OneDrive API instance performReverseDeletionOneDriveApiInstance.deleteById(itemToDelete.driveId, itemToDelete.id, itemToDelete.eTag); } else { // Perform the permanent delete via the default OneDrive API instance performReverseDeletionOneDriveApiInstance.permanentDeleteById(itemToDelete.driveId, itemToDelete.id, itemToDelete.eTag); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory performReverseDeletionOneDriveApiInstance.releaseCurlEngine(); performReverseDeletionOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Create a fake OneDrive response suitable for use with saveItem JSONValue createFakeResponse(string path) { import std.digest.sha; // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Generate a simulated JSON response which can be used // At a minimum we need: // 1. eTag // 2. cTag // 3. fileSystemInfo // 4. file or folder. if file, hash of file // 5. id // 6. name // 7. parent reference string fakeDriveId = appConfig.defaultDriveId; string fakeRootId = appConfig.defaultRootId; SysTime mtime = exists(path) ? timeLastModified(path).toUTC() : Clock.currTime(UTC()); auto sha1 = new SHA1Digest(); ubyte[] fakedOneDriveItemValues = sha1.digest(path); JSONValue fakeResponse; string parentPath = dirName(path); if (parentPath != "." && exists(path)) { foreach (searchDriveId; onlineDriveDetails.keys) { Item databaseItem; if (itemDB.selectByPath(parentPath, searchDriveId, databaseItem)) { fakeDriveId = databaseItem.driveId; fakeRootId = databaseItem.id; break; // Exit loop after finding the first match } } } fakeResponse = [ "id": JSONValue(toHexString(fakedOneDriveItemValues)), "cTag": JSONValue(toHexString(fakedOneDriveItemValues)), "eTag": JSONValue(toHexString(fakedOneDriveItemValues)), "fileSystemInfo": JSONValue([ "createdDateTime": mtime.toISOExtString(), "lastModifiedDateTime": mtime.toISOExtString() ]), "name": JSONValue(baseName(path)), "parentReference": JSONValue([ "driveId": JSONValue(fakeDriveId), "driveType": JSONValue(appConfig.accountType), "id": JSONValue(fakeRootId) ]) ]; if (exists(path)) { if (isDir(path)) { fakeResponse["folder"] = JSONValue(""); } else { string quickXorHash = computeQuickXorHash(path); fakeResponse["file"] = JSONValue([ "hashes": JSONValue(["quickXorHash": JSONValue(quickXorHash)]) ]); } } else { // Assume directory if path does not exist fakeResponse["folder"] = JSONValue(""); } if (debugLogging) {addLogEntry("Generated Fake OneDrive Response: " ~ to!string(fakeResponse), ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return the generated fake API response return fakeResponse; } // Save JSON item details into the item database void saveItem(JSONValue jsonItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // jsonItem has to be a valid object if (jsonItem.type() == JSONType.object) { // Issue #3336 - Convert driveId to lowercase if (appConfig.accountType == "personal") { // We must massage this raw JSON record to force the jsonItem["parentReference"]["driveId"] to lowercase if (hasParentReferenceDriveId(jsonItem)) { // This JSON record has a driveId we now must manipulate to lowercase string originalDriveIdValue = jsonItem["parentReference"]["driveId"].str; jsonItem["parentReference"]["driveId"] = transformToLowerCase(originalDriveIdValue); } } // Check if the response JSON has an 'id', otherwise makeItem() fails with 'Key not found: id' if (hasId(jsonItem)) { // Are we in a --upload-only & --remove-source-files scenario? // We do not want to add the item to the database in this situation as there is no local reference to the file post file deletion // If the item is a directory, we need to add this to the DB, if this is a file, we dont add this, the parent path is not in DB, thus any new files in this directory are not added if ((uploadOnly) && (localDeleteAfterUpload) && (isItemFile(jsonItem))) { // Log that we skipping adding item to the local DB and the reason why if (debugLogging) {addLogEntry("Skipping adding to database as --upload-only & --remove-source-files configured", ["debug"]);} } else { // Takes a JSON input and formats to an item which can be used by the database Item item = makeItem(jsonItem); // Is this JSON item a 'root' item? if ((isItemRoot(jsonItem)) && (item.name == "root")) { if (debugLogging) { addLogEntry("Creating 'root' DB item from this JSON: " ~ sanitiseJSONItem(jsonItem), ["debug"]); addLogEntry("Updating DB Item object with correct values as this is a 'root' object", ["debug"]); addLogEntry(" item.parentId = null", ["debug"]); addLogEntry(" item.type = ItemType.root", ["debug"]); } item.parentId = null; // ensures that this database entry has no parent item.type = ItemType.root; // Check for parentReference if (hasParentReference(jsonItem)) { // Set the correct item.driveId if (debugLogging) { addLogEntry("The 'root' JSON Item HAS a parentReference .... setting item.driveId = jsonItem['parentReference']['driveId'].str from the provided JSON record", ["debug"]); string logMessage = format(" item.driveId = '%s'", jsonItem["parentReference"]["driveId"].str); addLogEntry(logMessage, ["debug"]); } item.driveId = jsonItem["parentReference"]["driveId"].str; } // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test item.driveId = transformToLowerCase(item.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (item.driveId != appConfig.defaultDriveId) { item.driveId = testProvidedDriveIdForLengthIssue(item.driveId); } } // We only should be adding our account 'root' to the database, not shared folder 'root' items if (item.driveId != appConfig.defaultDriveId) { // Shared Folder drive 'root' object .. we dont want this item if (debugLogging) {addLogEntry("NOT adding 'remote root' object to database: " ~ to!string(item), ["debug"]);} return; } } // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test item.driveId = transformToLowerCase(item.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (item.driveId != appConfig.defaultDriveId) { item.driveId = testProvidedDriveIdForLengthIssue(item.driveId); } } // Add to the local database if (debugLogging) {addLogEntry("Saving this DB item record: " ~ to!string(item), ["debug"]);} itemDB.upsert(item); // If we have a remote drive ID, add this to our list of known drive id's if (!item.remoteDriveId.empty) { // Keep the DriveDetailsCache array with unique entries only DriveDetailsCache cachedOnlineDriveData; if (!canFindDriveId(item.remoteDriveId, cachedOnlineDriveData)) { // Add this driveId to the drive cache if (debugLogging) {addLogEntry("Database item is a remote drive object, need to fetch online details for this drive: " ~ to!string(item.remoteDriveId), ["debug"]);} addOrUpdateOneDriveOnlineDetails(item.remoteDriveId); } } } } else { // log error addLogEntry("ERROR: OneDrive response missing required 'id' element"); addLogEntry("ERROR: " ~ sanitiseJSONItem(jsonItem)); } } else { // Log that the provided JSON could not be processed addLogEntry("ERROR: Invalid JSON object - the provided data cannot be processed or stored in the database."); // What level of next message is provided? if (appConfig.verbosityCount == 0) { // Standard error message addLogEntry("ERROR: Please rerun the application with --verbose enabled to obtain additional diagnostic information."); } else { // verbose or debug addLogEntry("ERROR: The following JSON data failed validation and could not be saved: " ~ to!string(jsonItem)); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Save an already created database object into the database void saveDatabaseItem(Item newDatabaseItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Issue #3115 - Personal Account Shared Folder // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase for the DB record string actualOnlineDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(newDatabaseItem.driveId)); newDatabaseItem.driveId = actualOnlineDriveId; // Is this a 'remote' DB record if (newDatabaseItem.type == ItemType.remote) { // Issue #3336 - Convert remoteDriveId to lowercase before any test newDatabaseItem.remoteDriveId = transformToLowerCase(newDatabaseItem.remoteDriveId); // Test remoteDriveId length and validation if the remoteDriveId we are testing is not equal to appConfig.defaultDriveId if (newDatabaseItem.remoteDriveId != appConfig.defaultDriveId) { // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the actual OneDrive Personal remoteDriveId value and is 15 character checked string actualOnlineRemoteDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(newDatabaseItem.remoteDriveId)); newDatabaseItem.remoteDriveId = actualOnlineRemoteDriveId; } } } // Add the database record if (debugLogging) {addLogEntry("Creating a new database record for a new local path that has been created: " ~ to!string(newDatabaseItem), ["debug"]);} itemDB.upsert(newDatabaseItem); // If we have a remote drive ID, add this to our list of known drive id's if (!newDatabaseItem.remoteDriveId.empty) { // Keep the DriveDetailsCache array with unique entries only DriveDetailsCache cachedOnlineDriveData; if (!canFindDriveId(newDatabaseItem.remoteDriveId, cachedOnlineDriveData)) { // Add this driveId to the drive cache if (debugLogging) {addLogEntry("New database record is a remote drive object, need to fetch online details for this drive: " ~ to!string(newDatabaseItem.remoteDriveId), ["debug"]);} addOrUpdateOneDriveOnlineDetails(newDatabaseItem.remoteDriveId); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Wrapper function for makeDatabaseItem so we can check to ensure that the item has the required hashes Item makeItem(JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Make the DB Item from the JSON data provided Item newDatabaseItem = makeDatabaseItem(onedriveJSONItem); // Is this a 'file' item that has not been deleted? Deleted items have no hash if ((newDatabaseItem.type == ItemType.file) && (!isItemDeleted(onedriveJSONItem))) { // Does this item have a file size attribute? if (hasFileSize(onedriveJSONItem)) { // Is the file size greater than 0? if (onedriveJSONItem["size"].integer > 0) { // Does the DB item have any hashes as per the API provided JSON data? if ((newDatabaseItem.quickXorHash.empty) && (newDatabaseItem.sha256Hash.empty)) { // Odd .. there is no hash for this item .. why is that? // Is there a 'file' JSON element? if ("file" in onedriveJSONItem) { // Microsoft OneDrive OneNote objects will report as files but have 'application/msonenote' and 'application/octet-stream' as mime types if ((isMicrosoftOneNoteMimeType1(onedriveJSONItem)) || (isMicrosoftOneNoteMimeType2(onedriveJSONItem))) { // Debug log output that this is a potential OneNote object if (debugLogging) {addLogEntry("This item is potentially an associated Microsoft OneNote Object Item", ["debug"]);} } else { // Not a Microsoft OneNote Mime Type Object .. string apiWarningMessage = "WARNING: OneDrive API inconsistency - this file does not have any hash: "; // This is computationally expensive .. but we are only doing this if there are no hashes provided bool parentInDatabase = itemDB.idInLocalDatabase(newDatabaseItem.driveId, newDatabaseItem.parentId); // Is the parent id in the database? if (parentInDatabase) { // This is again computationally expensive .. calculate this item path to advise the user the actual path of this item that has no hash string newItemPath = computeItemPath(newDatabaseItem.driveId, newDatabaseItem.parentId) ~ "/" ~ newDatabaseItem.name; addLogEntry(apiWarningMessage ~ newItemPath); } else { // Parent is not in the database .. why? // Check if the parent item had been skipped .. if (newDatabaseItem.parentId in skippedItems) { if (debugLogging) {addLogEntry(apiWarningMessage ~ "newDatabaseItem.parentId listed within skippedItems", ["debug"]);} } else { // Use the item ID .. there is no other reference available, parent is not being skipped, so we should have been able to calculate this - but we could not addLogEntry(apiWarningMessage ~ newDatabaseItem.id); } } } } } } else { // zero file size if (debugLogging) {addLogEntry("This item file is zero size - potentially no hash provided by the OneDrive API", ["debug"]);} } } } // OneDrive Personal Account driveId and remoteDriveId length check // Issue #3072 (https://github.com/abraunegg/onedrive/issues/3072) illustrated that the OneDrive API is inconsistent in response when the Drive ID starts with a zero ('0') // - driveId // - remoteDriveId // // Example: // 024470056F5C3E43 (driveId) // 24470056f5c3e43 (remoteDriveId) // If this is a OneDrive Personal Account, ensure this value is 16 characters, padded by leading zero's if eventually required // What account type is this? if (appConfig.accountType == "personal") { // Check the newDatabaseItem.remoteDriveId if (!newDatabaseItem.remoteDriveId.empty) { // Issue #3136, #3139 #3143 // Test searchItem.driveId length and validation // - This check the length, fetch online value and return a 16 character driveId newDatabaseItem.remoteDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(newDatabaseItem.remoteDriveId)); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the new database item return newDatabaseItem; } // For OneDrive Personal Accounts, the case sensitivity depending on the API call means the 'driveId' can be uppercase or lowercase // For this application use, this causes issues as, in POSIX environments - 024470056F5C3E43 != 024470056f5c3e43 despite on Windows this being treated as the same // This function does NOT do a 15 character driveId validation string fetchRealOnlineDriveIdentifier(string inputDriveId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // What are we doing if (debugLogging) { string fetchRealValueLogMessage = format("Fetching actual online 'driveId' value for '%s'", inputDriveId); addLogEntry(fetchRealValueLogMessage, ["debug"]); } // variables for this function JSONValue remoteDriveDetails; OneDriveApi fetchDriveDetailsOneDriveApiInstance; string outputDriveId; // Create new OneDrive API Instance fetchDriveDetailsOneDriveApiInstance = new OneDriveApi(appConfig); fetchDriveDetailsOneDriveApiInstance.initialise(); // Get root details for the provided driveId try { remoteDriveDetails = fetchDriveDetailsOneDriveApiInstance.getDriveIdRoot(inputDriveId); } catch (OneDriveException exception) { if (debugLogging) {addLogEntry("remoteDriveDetails = fetchDriveDetailsOneDriveApiInstance.getDriveIdRoot(inputDriveId) generated a OneDriveException", ["debug"]);} // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory fetchDriveDetailsOneDriveApiInstance.releaseCurlEngine(); fetchDriveDetailsOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Do we have details we can use? if (hasParentReferenceDriveId(remoteDriveDetails)) { // We have a [parentReference][driveId] reference driveId to use outputDriveId = remoteDriveDetails["parentReference"]["driveId"].str; } else { // We dont have a value from online we can use // Test existing driveId length and validation outputDriveId = inputDriveId; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the outputDriveId return outputDriveId; } // Print the fileDownloadFailures and fileUploadFailures arrays if they are not empty void displaySyncFailures() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } bool logFailures(string[] failures, string operation) { if (failures.empty) return false; addLogEntry(); addLogEntry("Failed items to " ~ operation ~ " to/from Microsoft OneDrive: " ~ to!string(failures.length)); foreach (failedFile; failures) { addLogEntry("Failed to " ~ operation ~ ": " ~ failedFile, ["info", "notify"]); foreach (searchDriveId; onlineDriveDetails.keys) { Item dbItem; if (itemDB.selectByPath(failedFile, searchDriveId, dbItem)) { addLogEntry("ERROR: Failed " ~ operation ~ " path found in database, must delete this item from the database .. it should not be in there if the file failed to " ~ operation); itemDB.deleteById(dbItem.driveId, dbItem.id); if (dbItem.remoteDriveId != null) { itemDB.deleteById(dbItem.remoteDriveId, dbItem.remoteId); } } } } return true; } bool downloadFailuresLogged = logFailures(fileDownloadFailures, "download"); bool uploadFailuresLogged = logFailures(fileUploadFailures, "upload"); syncFailures = downloadFailuresLogged || uploadFailuresLogged; // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Generate a /delta compatible response - for use when we cant actually use /delta // This is required when the application is configured to use National Azure AD deployments as these do not support /delta queries // The same technique can also be used when we are using --single-directory. The parent objects up to the single directory target can be added, // then once the target of the --single-directory request is hit, all of the children of that path can be queried, giving a much more focused // JSON response which can then be processed, negating the need to continuously traverse the tree and 'exclude' items JSONValue generateDeltaResponse(string pathToQuery = null) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // JSON value which will be responded with JSONValue selfGeneratedDeltaResponse; // Function variables bool remotePathObject = false; Item searchItem; JSONValue rootData; JSONValue driveData; JSONValue pathData; JSONValue topLevelChildren; JSONValue[] childrenData; string nextLink; OneDriveApi generateDeltaResponseOneDriveApiInstance; // Was a path to query passed in? if (pathToQuery.empty) { // Will query for the 'root' pathToQuery = "."; } // Create new OneDrive API Instance generateDeltaResponseOneDriveApiInstance = new OneDriveApi(appConfig); generateDeltaResponseOneDriveApiInstance.initialise(); // Is this a --single-directory invocation? if (!singleDirectoryScope) { // In a --resync scenario, there is no DB data to query, so we have to query the OneDrive API here to get relevant details try { // Query the OneDrive API, using the path, which will query 'our' OneDrive Account pathData = generateDeltaResponseOneDriveApiInstance.getPathDetails(pathToQuery); // Is the path on OneDrive local or remote to our account drive id? if (!isItemRemote(pathData)) { // The path we are seeking is local to our account drive id searchItem.driveId = pathData["parentReference"]["driveId"].str; searchItem.id = pathData["id"].str; } else { // The path we are seeking is remote to our account drive id searchItem.driveId = pathData["remoteItem"]["parentReference"]["driveId"].str; searchItem.id = pathData["remoteItem"]["id"].str; remotePathObject = true; // Issue #3115 - Personal Account Shared Folder // What account type is this? if (appConfig.accountType == "personal") { // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the actual OneDrive Personal driveId value. The check of 'searchItem.driveId' to comply with 16 characters is done below string actualOnlineDriveId = fetchRealOnlineDriveIdentifier(searchItem.driveId); searchItem.driveId = actualOnlineDriveId; } } } catch (OneDriveException exception) { // Display error message displayOneDriveErrorMessage(exception.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory generateDeltaResponseOneDriveApiInstance.releaseCurlEngine(); generateDeltaResponseOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Must force exit here, allow logging to be done forceExit(); } } else { // When setSingleDirectoryScope() was called, the following were set to the correct items, even if the path was remote: // - singleDirectoryScopeDriveId // - singleDirectoryScopeItemId // Reuse these prior set values searchItem.driveId = singleDirectoryScopeDriveId; searchItem.id = singleDirectoryScopeItemId; } // Issue #3072 - Validate searchItem.driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test searchItem.driveId = transformToLowerCase(searchItem.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (searchItem.driveId != appConfig.defaultDriveId) { searchItem.driveId = testProvidedDriveIdForLengthIssue(searchItem.driveId); } } // Before we get any data from the OneDrive API, flag any child object in the database as out-of-sync for this driveId & and object id // Downgrade ONLY files associated with this driveId and idToQuery if (debugLogging) {addLogEntry("Downgrading all children for this searchItem.driveId (" ~ searchItem.driveId ~ ") and searchItem.id (" ~ searchItem.id ~ ") to an out-of-sync state", ["debug"]);} Item[] drivePathChildren = getChildren(searchItem.driveId, searchItem.id); if (count(drivePathChildren) > 0) { // Children to process and flag as out-of-sync foreach (drivePathChild; drivePathChildren) { // Flag any object in the database as out-of-sync for this driveId & and object id if (debugLogging) {addLogEntry("Downgrading item as out-of-sync: " ~ drivePathChild.id, ["debug"]);} itemDB.downgradeSyncStatusFlag(drivePathChild.driveId, drivePathChild.id); } } // Clear DB response array drivePathChildren = []; // Get drive details for the provided driveId try { driveData = generateDeltaResponseOneDriveApiInstance.getPathDetailsById(searchItem.driveId, searchItem.id); } catch (OneDriveException exception) { // An error was generated if (debugLogging) {addLogEntry("driveData = generateDeltaResponseOneDriveApiInstance.getPathDetailsById(searchItem.driveId, searchItem.id) generated a OneDriveException", ["debug"]);} // Was this a 403 or 404 ? if ((exception.httpStatusCode == 403) || (exception.httpStatusCode == 404)) { // The API call returned a 404 error response if (debugLogging) {addLogEntry("onlineParentData = onlineParentOneDriveApiInstance.getPathDetailsById(parentDriveId, parentObjectId); generated a 404 - shared folder path does not exist online", ["debug"]);} string errorMessage = format("WARNING: The OneDrive Shared Folder link target '%s' cannot be found online using the provided online data.", pathToQuery); // detail what this 404 error response means addLogEntry(); addLogEntry(errorMessage); addLogEntry("WARNING: This is potentially a broken online OneDrive Shared Folder link or you no longer have access to it. Please correct this error online."); addLogEntry(); // Release curl engine generateDeltaResponseOneDriveApiInstance.releaseCurlEngine(); // Free object and memory generateDeltaResponseOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Return the generated JSON response return selfGeneratedDeltaResponse; } else { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } // Was a valid JSON response for 'driveData' provided? if (driveData.type() == JSONType.object) { // Dynamic output for a non-verbose run so that the user knows something is happening string generatingDeltaResponseMessage = format("Generating a /delta response from the OneDrive API for this Drive ID: %s and Item ID: %s", searchItem.driveId, searchItem.id); if (appConfig.verbosityCount == 0) { if (!appConfig.suppressLoggingOutput) { addProcessingLogHeaderEntry(generatingDeltaResponseMessage, appConfig.verbosityCount); } } else { if (verboseLogging) {addLogEntry(generatingDeltaResponseMessage, ["verbose"]);} } // Process this initial JSON response if (!isItemRoot(driveData)) { // Are we generating a /delta response for a Shared Folder, if not, then we need to add the drive root details first if (!sharedFolderDeltaGeneration) { // Get root details for the provided driveId try { rootData = generateDeltaResponseOneDriveApiInstance.getDriveIdRoot(searchItem.driveId); } catch (OneDriveException exception) { if (debugLogging) {addLogEntry("rootData = onedrive.getDriveIdRoot(searchItem.driveId) generated a OneDriveException", ["debug"]);} // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // Add driveData JSON data to array if (verboseLogging) {addLogEntry("Adding OneDrive root details for processing", ["verbose"]);} childrenData ~= rootData; } } // Add driveData JSON data to array if (verboseLogging) {addLogEntry("Adding OneDrive parent folder details for processing", ["verbose"]);} // What 'driveData' are we adding? if (debugLogging) { addLogEntry("Adding this 'driveData' to childrenData = " ~ to!string(driveData), ["debug"]); } // add the responded 'driveData' to the childrenData to process later childrenData ~= driveData; } else { // driveData is an invalid JSON object addLogEntry("CODING TO DO: The query of OneDrive API to getPathDetailsById generated an invalid JSON response - thus we cant build our own /delta simulated response ... how to handle?"); // Release curl engine generateDeltaResponseOneDriveApiInstance.releaseCurlEngine(); // Free object and memory generateDeltaResponseOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Must force exit here, allow logging to be done forceExit(); } // For each child object, query the OneDrive API while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // query top level children try { topLevelChildren = generateDeltaResponseOneDriveApiInstance.listChildren(searchItem.driveId, searchItem.id, nextLink); } catch (OneDriveException exception) { // OneDrive threw an error if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); addLogEntry("Query Error: topLevelChildren = generateDeltaResponseOneDriveApiInstance.listChildren(searchItem.driveId, searchItem.id, nextLink)", ["debug"]); addLogEntry("driveId: " ~ searchItem.driveId, ["debug"]); addLogEntry("idToQuery: " ~ searchItem.id, ["debug"]); addLogEntry("nextLink: " ~ nextLink, ["debug"]); } // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // Process top level children if (!remotePathObject) { // Main account root folder if (verboseLogging) {addLogEntry("Adding " ~ to!string(count(topLevelChildren["value"].array)) ~ " OneDrive items for processing from the OneDrive 'root' Folder", ["verbose"]);} } else { // Shared Folder if (verboseLogging) {addLogEntry("Adding " ~ to!string(count(topLevelChildren["value"].array)) ~ " OneDrive items for processing from the OneDrive Shared Folder", ["verbose"]);} } foreach (child; topLevelChildren["value"].array) { // Check for any Client Side Filtering here ... we should skip querying the OneDrive API for 'folders' that we are going to just process and skip anyway. // This avoids needless calls to the OneDrive API, and potentially speeds up this process. if (!checkJSONAgainstClientSideFiltering(child)) { // add this child to the array of objects childrenData ~= child; // is this child a folder? if (isItemFolder(child)) { // We have to query this folders children if childCount > 0 if (child["folder"]["childCount"].integer > 0){ // This child folder has children string childIdToQuery = child["id"].str; string childDriveToQuery = child["parentReference"]["driveId"].str; auto childParentPath = child["parentReference"]["path"].str.split(":"); string folderPathToScan = childParentPath[1] ~ "/" ~ child["name"].str; string pathForLogging; // Are we in a --single-directory situation? If we are, the path we are using for logging needs to use the input path as a base if (singleDirectoryScope) { pathForLogging = appConfig.getValueString("single_directory") ~ "/" ~ child["name"].str; } else { pathForLogging = child["name"].str; } // Query the children of this item JSONValue[] grandChildrenData = queryForChildren(childDriveToQuery, childIdToQuery, folderPathToScan, pathForLogging); foreach (grandChild; grandChildrenData.array) { // add the grandchild to the array childrenData ~= grandChild; } } } // As we are generating a /delta response we need to check if this 'child' JSON is a 'remoteItem' and then handle appropriately // Is this a remote folder JSON ? if (isItemRemote(child)) { // Check account type if (appConfig.accountType == "personal") { // The folder is a remote item ... OneDrive Personal Shared Folder if (debugLogging) {addLogEntry("The JSON data indicates this is most likely a OneDrive Personal Shared Folder Link added by 'Add shortcut to My files'", ["debug"]);} // It is a 'remote' JSON item denoting a potential shared folder // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(child); } if (appConfig.accountType == "business") { // The folder is a remote item ... OneDrive Business Shared Folder if (debugLogging) {addLogEntry("The JSON data indicates this is most likely a OneDrive Shared Business Folder Link added by 'Add shortcut to My files'", ["debug"]);} // Is Shared Business Folder Syncing actually enabled? if (appConfig.getValueBool("sync_business_shared_items")) { // Shared Business Folder Syncing IS enabled // It is a 'remote' JSON item denoting a potential shared folder // Create a 'root' and 'Shared Folder' DB Tie Records for this JSON object in a consistent manner createRequiredSharedFolderDatabaseRecords(child); } } } } } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in topLevelChildren) { // Update nextLink to next changeSet bundle if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} nextLink = topLevelChildren["@odata.nextLink"].str; } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } if (appConfig.verbosityCount == 0) { // Dynamic output for a non-verbose run so that the user knows something is happening if (!appConfig.suppressLoggingOutput) { // Close out the '....' being printed to the console completeProcessingDots(); } } // Craft response from all returned JSON elements selfGeneratedDeltaResponse = [ "@odata.context": JSONValue("https://graph.microsoft.com/v1.0/$metadata#Collection(driveItem)"), "value": JSONValue(childrenData.array) ]; // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory generateDeltaResponseOneDriveApiInstance.releaseCurlEngine(); generateDeltaResponseOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return the generated JSON response return selfGeneratedDeltaResponse; } // Query the OneDrive API for the specified child id for any children objects JSONValue[] queryForChildren(string driveId, string idToQuery, string childParentPath, string pathForLogging) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // function variables JSONValue thisLevelChildren; JSONValue[] thisLevelChildrenData; string nextLink; // Create new OneDrive API Instance OneDriveApi queryChildrenOneDriveApiInstance; queryChildrenOneDriveApiInstance = new OneDriveApi(appConfig); queryChildrenOneDriveApiInstance.initialise(); // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test driveId = transformToLowerCase(driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (driveId != appConfig.defaultDriveId) { driveId = testProvidedDriveIdForLengthIssue(driveId); } } while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Query this level children try { thisLevelChildren = queryThisLevelChildren(driveId, idToQuery, nextLink, queryChildrenOneDriveApiInstance); } catch (OneDriveException exception) { // MAY NEED FUTURE WORK HERE .. YET TO TRIGGER THIS addLogEntry("CODING TO DO: EXCEPTION HANDLING NEEDED: thisLevelChildren = queryThisLevelChildren(driveId, idToQuery, nextLink, queryChildrenOneDriveApiInstance)"); } if (appConfig.verbosityCount == 0) { // Dynamic output for a non-verbose run so that the user knows something is happening if (!appConfig.suppressLoggingOutput) { addProcessingDotEntry(); } } // Was a paging token error detected? if ((thisLevelChildren.type() == JSONType.string) && (thisLevelChildren.str == "INVALID_PAGING_TOKEN")) { // Invalid paging token: failed to parse integer value from token if (debugLogging) addLogEntry("Upstream detected invalid paging token – clearing nextLink and retrying", ["debug"]); nextLink = null; thisLevelChildren = queryThisLevelChildren(driveId, idToQuery, nextLink, queryChildrenOneDriveApiInstance); } // Was a valid JSON response for 'thisLevelChildren' provided? if (thisLevelChildren.type() == JSONType.object) { // process this level children if (!childParentPath.empty) { // We dont use childParentPath to log, as this poses an information leak risk. // The full parent path of the child, as per the JSON might be: // /Level 1/Level 2/Level 3/Child Shared Folder/some folder/another folder // But 'Child Shared Folder' is what is shared, thus '/Level 1/Level 2/Level 3/' is a potential information leak if logged. // Plus, the application output now shows accurately what is being shared - so that is a good thing. if (verboseLogging) {addLogEntry("Adding " ~ to!string(count(thisLevelChildren["value"].array)) ~ " OneDrive JSON items for further processing from " ~ pathForLogging, ["verbose"]);} } foreach (child; thisLevelChildren["value"].array) { // Check for any Client Side Filtering here ... we should skip querying the OneDrive API for 'folders' that we are going to just process and skip anyway. // This avoids needless calls to the OneDrive API, and potentially speeds up this process. if (!checkJSONAgainstClientSideFiltering(child)) { // add this child to the array of objects thisLevelChildrenData ~= child; // is this child a folder? if (isItemFolder(child)){ // We have to query this folders children if childCount > 0 if (child["folder"]["childCount"].integer > 0){ // This child folder has children string childIdToQuery = child["id"].str; string childDriveToQuery = child["parentReference"]["driveId"].str; auto grandchildParentPath = child["parentReference"]["path"].str.split(":"); string folderPathToScan = grandchildParentPath[1] ~ "/" ~ child["name"].str; string newLoggingPath = pathForLogging ~ "/" ~ child["name"].str; JSONValue[] grandChildrenData = queryForChildren(childDriveToQuery, childIdToQuery, folderPathToScan, newLoggingPath); foreach (grandChild; grandChildrenData.array) { // add the grandchild to the array thisLevelChildrenData ~= grandChild; } } } } } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in thisLevelChildren) { // Update nextLink to next changeSet bundle nextLink = thisLevelChildren["@odata.nextLink"].str; if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} } else break; } else { // Invalid JSON response when querying this level children if (debugLogging) {addLogEntry("INVALID JSON response when attempting a retry of parent function - queryForChildren(driveId, idToQuery, childParentPath, pathForLogging)", ["debug"]);} // retry thisLevelChildren = queryThisLevelChildren if (debugLogging) {addLogEntry("Thread sleeping for an additional 30 seconds", ["debug"]);} Thread.sleep(dur!"seconds"(30)); if (debugLogging) {addLogEntry("Retry this call thisLevelChildren = queryThisLevelChildren(driveId, idToQuery, nextLink, queryChildrenOneDriveApiInstance)", ["debug"]);} thisLevelChildren = queryThisLevelChildren(driveId, idToQuery, nextLink, queryChildrenOneDriveApiInstance); } // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryChildrenOneDriveApiInstance.releaseCurlEngine(); queryChildrenOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return response return thisLevelChildrenData; } // Query the OneDrive API for the child objects for this element JSONValue queryThisLevelChildren(string driveId, string idToQuery, string nextLink, OneDriveApi queryChildrenOneDriveApiInstance) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Issue #3115 - Validate driveId length // - The function 'queryForChildren' checks the 'driveId' value and that value is the input to this function. // It is redundant to then check 'driveid' again as this is not changed when this function is called // function variables JSONValue thisLevelChildren; // query children try { // attempt API call if (debugLogging) {addLogEntry("Attempting Query: thisLevelChildren = queryChildrenOneDriveApiInstance.listChildren(driveId, idToQuery, nextLink)", ["debug"]);} thisLevelChildren = queryChildrenOneDriveApiInstance.listChildren(driveId, idToQuery, nextLink); if (debugLogging) {addLogEntry("Query 'thisLevelChildren = queryChildrenOneDriveApiInstance.listChildren(driveId, idToQuery, nextLink)' performed successfully", ["debug"]);} } catch (OneDriveException exception) { // OneDrive threw an error if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); addLogEntry("Query Error: thisLevelChildren = queryChildrenOneDriveApiInstance.listChildren(driveId, idToQuery, nextLink)", ["debug"]); addLogEntry("driveId: " ~ driveId, ["debug"]); addLogEntry("idToQuery: " ~ idToQuery, ["debug"]); addLogEntry("nextLink: " ~ nextLink, ["debug"]); } // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); // With the error displayed, testing of PR #3381 for #3375 generated this error: // Error Message: HTTP request returned status code 400 (Bad Request) // Error Reason: Invalid paging token: failed to parse integer value from token. if ((exception.httpStatusCode == 400) && (exception.msg.canFind("Invalid paging token"))) { // Log and return a known marker that bypasses JSONType.object check if (debugLogging) addLogEntry("Detected invalid paging token – signaling upstream", ["debug"]); return JSONValue("INVALID_PAGING_TOKEN"); } // Generic failure return thisLevelChildren; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return response return thisLevelChildren; } // Traverses the provided path online, via the OneDrive API, following correct parent driveId and itemId elements across the account // to find if this full path exists. If this path exists online, the last item in the object path will be returned as a full JSON item. // // If the createPathIfMissing = false + no path exists online, a null invalid JSON item will be returned. // If the createPathIfMissing = true + no path exists online, the requested path will be created in the correct location online. The resulting // response to the directory creation will then be returned. // // This function also ensures that each path in the requested path actually matches the requested element to ensure that the OneDrive API response // is not falsely matching a 'case insensitive' match to the actual request which is a POSIX compliance issue. JSONValue queryOneDriveForSpecificPathAndCreateIfMissing(string thisNewPathToSearch, bool createPathIfMissing) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // function variables JSONValue getPathDetailsAPIResponse; string currentPathTree; Item parentDetails; JSONValue topLevelChildren; string nextLink; bool directoryFoundOnline = false; bool posixIssue = false; // Create a new API Instance for this thread and initialise it OneDriveApi queryOneDriveForSpecificPath; queryOneDriveForSpecificPath = new OneDriveApi(appConfig); queryOneDriveForSpecificPath.initialise(); foreach (thisFolderName; pathSplitter(thisNewPathToSearch)) { if (debugLogging) {addLogEntry("Testing for the existence online of this folder path: " ~ thisFolderName, ["debug"]);} directoryFoundOnline = false; // If this is '.' this is the account root if (thisFolderName == ".") { currentPathTree = thisFolderName; } else { currentPathTree = currentPathTree ~ "/" ~ thisFolderName; } // What path are we querying if (debugLogging) {addLogEntry("Attempting to query OneDrive for this path: " ~ currentPathTree, ["debug"]);} // What query do we use? if (thisFolderName == ".") { // Query the root, set the right details try { getPathDetailsAPIResponse = queryOneDriveForSpecificPath.getPathDetails(currentPathTree); parentDetails = makeItem(getPathDetailsAPIResponse); // Save item to the database saveItem(getPathDetailsAPIResponse); directoryFoundOnline = true; } catch (OneDriveException exception) { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } else { // Ensure we have a valid driveId to search here if (parentDetails.driveId.empty) { parentDetails.driveId = appConfig.defaultDriveId; } // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { parentDetails.driveId = transformToLowerCase(parentDetails.driveId); } // If the prior JSON 'getPathDetailsAPIResponse' is on this account driveId .. then continue to use getPathDetails if (parentDetails.driveId == appConfig.defaultDriveId) { try { // Query OneDrive API for this path getPathDetailsAPIResponse = queryOneDriveForSpecificPath.getPathDetails(currentPathTree); // Portable Operating System Interface (POSIX) testing of JSON response from OneDrive API if (hasName(getPathDetailsAPIResponse)) { // Perform the POSIX evaluation test against the names if (performPosixTest(thisFolderName, getPathDetailsAPIResponse["name"].str)) { throw new PosixException(thisFolderName, getPathDetailsAPIResponse["name"].str); } } else { throw new JsonResponseException("Unable to perform POSIX test as the OneDrive API request generated an invalid JSON response"); } // No POSIX issue with requested path element parentDetails = makeItem(getPathDetailsAPIResponse); // Save item to the database saveItem(getPathDetailsAPIResponse); directoryFoundOnline = true; // Is this JSON a remote object if (debugLogging) {addLogEntry("Testing if this is a remote Shared Folder", ["debug"]);} if (isItemRemote(getPathDetailsAPIResponse)) { // Remote Directory .. need a DB Tie Record createDatabaseTieRecordForOnlineSharedFolder(parentDetails); // Temp DB Item to bind the 'remote' path to our parent path Item tempDBItem; // Set the name tempDBItem.name = parentDetails.name; // Set the correct item type tempDBItem.type = ItemType.dir; // Set the right elements using the 'remote' of the parent as the 'actual' for this DB Tie tempDBItem.driveId = parentDetails.remoteDriveId; tempDBItem.id = parentDetails.remoteId; // Set the correct mtime tempDBItem.mtime = parentDetails.mtime; // Update parentDetails to use this temp record parentDetails = tempDBItem; } } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { directoryFoundOnline = false; } else { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } catch (PosixException e) { // Display POSIX error message displayPosixErrorMessage(e.msg); addLogEntry("ERROR: Requested directory to search for and potentially create has a 'case-insensitive match' to an existing directory on Microsoft OneDrive online."); addLogEntry("ERROR: To resolve, rename this local directory: " ~ currentPathTree); } catch (JsonResponseException e) { if (debugLogging) {addLogEntry(e.msg, ["debug"]);} } } else { // parentDetails.driveId is not the account drive id - thus will be a remote shared item if (debugLogging) {addLogEntry("This parent directory is a remote object this next path will be on a remote drive", ["debug"]);} // For this parentDetails.driveId, parentDetails.id object, query the OneDrive API for it's children while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Query this remote object for its children topLevelChildren = queryOneDriveForSpecificPath.listChildren(parentDetails.driveId, parentDetails.id, nextLink); // Process each child foreach (child; topLevelChildren["value"].array) { // Is this child a folder? if (isItemFolder(child)) { // Is this the child folder we are looking for, and is a POSIX match? if (child["name"].str == thisFolderName) { // EXACT MATCH including case sensitivity: Flag that we found the folder online directoryFoundOnline = true; // Use these details for the next entry path getPathDetailsAPIResponse = child; parentDetails = makeItem(getPathDetailsAPIResponse); // Save item to the database saveItem(getPathDetailsAPIResponse); // No need to continue searching break; } else { string childAsLower = toLower(child["name"].str); string thisFolderNameAsLower = toLower(thisFolderName); try { if (childAsLower == thisFolderNameAsLower) { // This is a POSIX 'case in-sensitive match' ..... // Local item name has a 'case-insensitive match' to an existing item on OneDrive posixIssue = true; throw new PosixException(thisFolderName, child["name"].str); } } catch (PosixException e) { // Display POSIX error message displayPosixErrorMessage(e.msg); addLogEntry("ERROR: Requested directory to search for and potentially create has a 'case-insensitive match' to an existing directory on Microsoft OneDrive online."); addLogEntry("ERROR: To resolve, rename this local directory: " ~ currentPathTree); } } } } if (directoryFoundOnline) { // We found the folder, no need to continue searching nextLink data break; } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in topLevelChildren) { // Update nextLink to next changeSet bundle if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} nextLink = topLevelChildren["@odata.nextLink"].str; } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } } } // If we did not find the folder, we need to create this folder if (!directoryFoundOnline) { // Folder not found online // Set any response to be an invalid JSON item getPathDetailsAPIResponse = null; // Was there a POSIX issue? if (!posixIssue) { // No POSIX issue if (createPathIfMissing) { // Create this path as it is missing on OneDrive online and there is no POSIX issue with a 'case-insensitive match' if (debugLogging) { addLogEntry("FOLDER NOT FOUND ONLINE AND WE ARE REQUESTED TO CREATE IT", ["debug"]); addLogEntry("Create folder on this drive: " ~ parentDetails.driveId, ["debug"]); addLogEntry("Create folder as a child on this object: " ~ parentDetails.id, ["debug"]); addLogEntry("Create this folder name: " ~ thisFolderName, ["debug"]); } // Generate the JSON needed to create the folder online JSONValue newDriveItem = [ "name": JSONValue(thisFolderName), "folder": parseJSON("{}") ]; JSONValue createByIdAPIResponse; // Submit the creation request // Fix for https://github.com/skilion/onedrive/issues/356 if (!dryRun) { try { // Attempt to create a new folder on the configured parent driveId & parent id createByIdAPIResponse = queryOneDriveForSpecificPath.createById(parentDetails.driveId, parentDetails.id, newDriveItem); // Is the response a valid JSON object - validation checking done in saveItem saveItem(createByIdAPIResponse); // Set getPathDetailsAPIResponse to createByIdAPIResponse getPathDetailsAPIResponse = createByIdAPIResponse; } catch (OneDriveException e) { // 409 - API Race Condition if (e.httpStatusCode == 409) { // When we attempted to create it, OneDrive responded that it now already exists if (verboseLogging) {addLogEntry("OneDrive reported that " ~ thisFolderName ~ " already exists .. OneDrive API race condition", ["verbose"]);} } else { // some other error from OneDrive was returned - display what it is addLogEntry("OneDrive generated an error when creating this path: " ~ thisFolderName); displayOneDriveErrorMessage(e.msg, thisFunctionName); } } } else { // Simulate a successful 'directory create' & save it to the dryRun database copy // The simulated response has to pass 'makeItem' as part of saveItem auto fakeResponse = createFakeResponse(thisNewPathToSearch); // Save item to the database saveItem(fakeResponse); } } } } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryOneDriveForSpecificPath.releaseCurlEngine(); queryOneDriveForSpecificPath = null; // Perform Garbage Collection GC.collect(); // Output our search results if (debugLogging) {addLogEntry("queryOneDriveForSpecificPathAndCreateIfMissing.getPathDetailsAPIResponse = " ~ to!string(getPathDetailsAPIResponse), ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return JSON result return getPathDetailsAPIResponse; } // Delete an item by it's path // This function is only used in --monitor mode to remove a directory online void deleteByPath(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // function variables Item dbItem; // Need to check all driveid's we know about, not just the defaultDriveId bool itemInDB = false; foreach (searchDriveId; onlineDriveDetails.keys) { if (itemDB.selectByPath(path, searchDriveId, dbItem)) { // item was found in the DB itemInDB = true; break; } } // Was the item found in the database? if (!itemInDB) { // path to delete is not in the local database .. // was this a --remove-directory attempt? if (!appConfig.getValueBool("monitor")) { // --remove-directory deletion attempt addLogEntry("The item to delete is not in the local database - unable to delete online"); return; } else { // normal use .. --monitor being used throw new SyncException("The item to delete is not in the local database"); } } // This needs to be enforced as we have to know the parent id of the object being deleted if (dbItem.parentId == null) { // the item is a remote folder, need to do the operation on the parent enforce(itemDB.selectByPathIncludingRemoteItems(path, appConfig.defaultDriveId, dbItem)); } try { if (noRemoteDelete) { // do not process remote delete if (verboseLogging) {addLogEntry("Skipping remote delete as --upload-only & --no-remote-delete configured", ["verbose"]);} } else { uploadDeletedItem(dbItem, path); } } catch (FileException e) { // filesystem generated an error message - display error message displayFileSystemErrorMessage(e.msg, thisFunctionName, path); } catch (OneDriveException e) { if (e.httpStatusCode == 404) { addLogEntry(e.msg); } else { // display what the error is displayOneDriveErrorMessage(e.msg, thisFunctionName); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Delete an item by it's path // Delete a directory on OneDrive without syncing. This function is only used with --remove-directory void deleteByPathNoSync(string path) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Attempt to delete the requested path within OneDrive without performing a sync addLogEntry("Attempting to delete the requested path within Microsoft OneDrive"); // function variables JSONValue getPathDetailsAPIResponse; OneDriveApi deleteByPathNoSyncAPIInstance; // test if the path we are going to exists on OneDrive try { // Create a new API Instance for this thread and initialise it deleteByPathNoSyncAPIInstance = new OneDriveApi(appConfig); deleteByPathNoSyncAPIInstance.initialise(); getPathDetailsAPIResponse = deleteByPathNoSyncAPIInstance.getPathDetails(path); // If we get here, no error, the path to delete exists online } catch (OneDriveException exception) { // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory deleteByPathNoSyncAPIInstance.releaseCurlEngine(); deleteByPathNoSyncAPIInstance = null; // Perform Garbage Collection GC.collect(); // Log that an error was generated if (debugLogging) {addLogEntry("deleteByPathNoSyncAPIInstance.getPathDetails(path) generated a OneDriveException", ["debug"]);} if (exception.httpStatusCode == 404) { // The directory was not found on OneDrive - no need to delete it addLogEntry("The requested directory to delete was not found on OneDrive - skipping removing the remote directory online as it does not exist"); return; } // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); return; } // Make a DB item from the JSON data that was returned via the API call Item deletionItem = makeItem(getPathDetailsAPIResponse); // Is the item to remove the correct type if (deletionItem.type == ItemType.dir) { // Item is a directory to remove // Log that the path | item was found, is a directory addLogEntry("The requested directory to delete was found on OneDrive - attempting deletion"); // Try the online deletion try { if (!permanentDelete) { // Perform the delete via the default OneDrive API instance deleteByPathNoSyncAPIInstance.deleteById(deletionItem.driveId, deletionItem.id); } else { // Perform the permanent delete via the default OneDrive API instance deleteByPathNoSyncAPIInstance.permanentDeleteById(deletionItem.driveId, deletionItem.id); } // If we get here without error, directory was deleted addLogEntry("The requested directory to delete online has been deleted"); } catch (OneDriveException exception) { // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } else { // --remove-directory is for removing directories // Log that the path | item was found, is a directory addLogEntry("The requested path to delete is not a directory - aborting deletion attempt"); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory deleteByPathNoSyncAPIInstance.releaseCurlEngine(); deleteByPathNoSyncAPIInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_move // This function is only called in monitor mode when an move event is coming from // inotify and we try to move the item. void uploadMoveItem(string oldPath, string newPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Log that we are doing a move addLogEntry("Moving " ~ oldPath ~ " to " ~ newPath); // Is this move unwanted? bool unwanted = false; // Item variables Item oldItem, newItem, parentItem; // This not a Client Side Filtering check, nor a Microsoft Check, but is a sanity check that the path provided is UTF encoded correctly // Check the std.encoding of the path against: Unicode 5.0, ASCII, ISO-8859-1, ISO-8859-2, WINDOWS-1250, WINDOWS-1251, WINDOWS-1252 if (!unwanted) { if(!isValid(newPath)) { // Path is not valid according to https://dlang.org/phobos/std_encoding.html addLogEntry("Skipping item - invalid character encoding sequence: " ~ newPath, ["info", "notify"]); unwanted = true; } } // Check this path against the Client Side Filtering Rules // - check_nosync // - skip_dotfiles // - skip_symlinks // - skip_file // - skip_dir // - sync_list // - skip_size if (!unwanted) { unwanted = checkPathAgainstClientSideFiltering(newPath); } // Check this path against the Microsoft Naming Conventions & Restrictions // - Check path against Microsoft OneDrive restriction and limitations about Windows naming for files and folders // - Check path for bad whitespace items // - Check path for HTML ASCII Codes // - Check path for ASCII Control Codes if (!unwanted) { unwanted = checkPathAgainstMicrosoftNamingRestrictions(newPath); } // 'newPath' has passed client side filtering validation if (!unwanted) { if (!itemDB.selectByPath(oldPath, appConfig.defaultDriveId, oldItem)) { // The old path|item is not synced with the database, upload as a new file addLogEntry("Moved local item was not in-sync with local database - uploading as new item"); scanLocalFilesystemPathForNewData(newPath); return; } if (oldItem.parentId == null) { // the item is a remote folder, need to do the operation on the parent enforce(itemDB.selectByPathIncludingRemoteItems(oldPath, appConfig.defaultDriveId, oldItem)); } if (itemDB.selectByPath(newPath, appConfig.defaultDriveId, newItem)) { // the destination has been overwritten addLogEntry("Moved local item overwrote an existing item - deleting old online item"); uploadDeletedItem(newItem, newPath); } if (!itemDB.selectByPath(dirName(newPath), appConfig.defaultDriveId, parentItem)) { // the parent item is not in the database throw new SyncException("Can't move an item to an unsynchronised directory"); } if (oldItem.driveId != parentItem.driveId) { // items cannot be moved between drives uploadDeletedItem(oldItem, oldPath); // what sort of move is this? if (isFile(newPath)) { // newPath is a file uploadNewFile(newPath); } else { // newPath is a directory scanLocalFilesystemPathForNewData(newPath); } } else { if (!exists(newPath)) { // is this --monitor use? if (appConfig.getValueBool("monitor")) { if (verboseLogging) {addLogEntry("uploadMoveItem target has disappeared: " ~ newPath, ["verbose"]);} return; } } // Configure the modification JSON item SysTime mtime; if (appConfig.getValueBool("monitor")) { // Use the newPath modified timestamp mtime = timeLastModified(newPath).toUTC(); } else { // Use the current system time mtime = Clock.currTime().toUTC(); } JSONValue data = [ "name": JSONValue(baseName(newPath)), "parentReference": JSONValue([ "id": parentItem.id ]), "fileSystemInfo": JSONValue([ "lastModifiedDateTime": mtime.toISOExtString() ]) ]; // Perform the move operation on OneDrive bool isMoveSuccess = false; JSONValue response; string eTag = oldItem.eTag; // Create a new API Instance for this thread and initialise it OneDriveApi movePathOnlineApiInstance; movePathOnlineApiInstance = new OneDriveApi(appConfig); movePathOnlineApiInstance.initialise(); // Try the online move for (int i = 0; i < 3; i++) { try { response = movePathOnlineApiInstance.updateById(oldItem.driveId, oldItem.id, data, eTag); isMoveSuccess = true; break; } catch (OneDriveException e) { // Handle a 412 - A precondition provided in the request (such as an if-match header) does not match the resource's current state. if (e.httpStatusCode == 412) { // OneDrive threw a 412 error, most likely: ETag does not match current item's value // Retry without eTag if (debugLogging) {addLogEntry("File Move Failed - OneDrive eTag / cTag match issue", ["debug"]);} if (verboseLogging) {addLogEntry("OneDrive returned a 'HTTP 412 - Precondition Failed' when attempting to move the file - gracefully handling error", ["verbose"]);} eTag = null; // Retry to move the file but without the eTag, via the for() loop } else if (e.httpStatusCode == 409) { // Destination item already exists and is a conflict, delete existing item first addLogEntry("Moved local item will overwrite an existing online item - deleting old online item first"); uploadDeletedItem(newItem, newPath); } else break; } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory movePathOnlineApiInstance.releaseCurlEngine(); movePathOnlineApiInstance = null; // Perform Garbage Collection GC.collect(); // Save the move response from OneDrive in the database if (isMoveSuccess && response.type() == JSONType.object) { saveItem(response); } else { // Log why we are not saving if (debugLogging) {addLogEntry("uploadMoveItem: skipping saveItem() (no JSON payload returned or move not successful)", ["debug"]);} } } } else { // Moved item is unwanted addLogEntry("Item has been moved to a location that is excluded from sync operations. Removing item from OneDrive"); uploadDeletedItem(oldItem, oldPath); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Perform integrity validation of the file that was uploaded bool performUploadIntegrityValidationChecks(JSONValue uploadResponse, string localFilePath, long localFileSize) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } bool integrityValid = false; if (!disableUploadValidation) { // Integrity validation has not been disabled (this is the default so we are always integrity checking our uploads) if (uploadResponse.type() == JSONType.object) { // Provided JSON is a valid JSON long uploadFileSize; string uploadFileHash; string localFileHash; // Regardless if valid JSON is responded with, 'size' and 'quickXorHash' must be present if (hasFileSize(uploadResponse) && hasQuickXorHash(uploadResponse)) { uploadFileSize = uploadResponse["size"].integer; uploadFileHash = uploadResponse["file"]["hashes"]["quickXorHash"].str; localFileHash = computeQuickXorHash(localFilePath); } else { if (verboseLogging) { addLogEntry("Online file validation unable to be performed: input JSON whilst valid did not contain data which could be validated", ["verbose"]); addLogEntry("WARNING: Skipping upload integrity check for: " ~ localFilePath, ["verbose"]); } return integrityValid; } // compare values if ((localFileSize == uploadFileSize) && (localFileHash == uploadFileHash)) { // Uploaded file integrity intact if (debugLogging) {addLogEntry("Uploaded local file matches reported online size and hash values", ["debug"]);} // set to true and return integrityValid = true; return integrityValid; } else { // Upload integrity failure .. what failed? // There are 2 scenarios where this happens: // 1. Failed Transfer // 2. Upload file is going to a SharePoint Site, where Microsoft enriches the file with additional metadata with no way to disable addLogEntry("WARNING: Online file integrity failure for: " ~ localFilePath, ["info", "notify"]); // What integrity failed - size? if (localFileSize != uploadFileSize) { if (verboseLogging) {addLogEntry("WARNING: Online file integrity failure - Size Mismatch", ["verbose"]);} } // What integrity failed - hash? if (localFileHash != uploadFileHash) { if (verboseLogging) {addLogEntry("WARNING: Online file integrity failure - Hash Mismatch", ["verbose"]);} } // What account type is this? if (appConfig.accountType != "personal") { // Not a personal account, thus the integrity failure is most likely due to SharePoint if (verboseLogging) { addLogEntry("CAUTION: When you upload files to Microsoft OneDrive that uses SharePoint as its backend, Microsoft OneDrive will alter your files post upload.", ["verbose"]); addLogEntry("CAUTION: This will lead to technical differences between the version stored online and your local original file, potentially causing issues with the accuracy or consistency of your data.", ["verbose"]); addLogEntry("CAUTION: Please refer to https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details.", ["verbose"]); } } // How can this be disabled? addLogEntry("To disable the integrity checking of uploaded files use --disable-upload-validation"); } } else { if (verboseLogging) { addLogEntry("Online file validation unable to be performed: input JSON whilst valid did not contain data which could be validated", ["verbose"]); addLogEntry("WARNING: Skipping upload integrity check for: " ~ localFilePath, ["verbose"]); } } } else { // Skipping upload integrity check, do not notify the user via the GUI ... they have explicitly disabled upload validation if (verboseLogging) {addLogEntry("WARNING: Skipping upload integrity check for: " ~ localFilePath, ["verbose"]);} // We are bypassing integrity checks due to --disable-upload-validation if (debugLogging) { addLogEntry("Online file validation disabled due to --disable-upload-validation", ["debug"]); addLogEntry("- Assuming file integrity is OK and valid", ["debug"]); } // Ensure we return 'true', but this is in a false sense, as we are skipping the integrity check, so we assume the file is good integrityValid = true; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Is the file integrity online valid? return integrityValid; } // Query Office 365 SharePoint Shared Library site name to obtain it's Drive ID void querySiteCollectionForDriveID(string sharepointLibraryNameToQuery) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Steps to get the ID: // 1. Query https://graph.microsoft.com/v1.0/sites?search= with the name entered // 2. Evaluate the response. A valid response will contain the description and the id. If the response comes back with nothing, the site name cannot be found or no access // 3. If valid, use the returned ID and query the site drives // https://graph.microsoft.com/v1.0/sites//drives // 4. Display Shared Library Name & Drive ID string site_id; string drive_id; bool found = false; JSONValue siteQuery; string nextLink; string[] siteSearchResults; // Create a new API Instance for this thread and initialise it OneDriveApi querySharePointLibraryNameApiInstance; querySharePointLibraryNameApiInstance = new OneDriveApi(appConfig); querySharePointLibraryNameApiInstance.initialise(); // The account type must not be a personal account type if (appConfig.accountType == "personal") { addLogEntry("ERROR: A OneDrive Personal Account cannot be used with --get-sharepoint-drive-id. Please re-authenticate your client using a OneDrive Business Account."); return; } // What query are we performing? addLogEntry(); addLogEntry("Office 365 Library Name Query: " ~ sharepointLibraryNameToQuery); while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } try { siteQuery = querySharePointLibraryNameApiInstance.o365SiteSearch(nextLink); } catch (OneDriveException e) { addLogEntry("ERROR: Query of OneDrive for Office 365 Library Name failed"); // Forbidden - most likely authentication scope needs to be updated if (e.httpStatusCode == 403) { addLogEntry("ERROR: Authentication scope needs to be updated. Use --reauth and re-authenticate client."); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // Requested resource cannot be found if (e.httpStatusCode == 404) { string siteSearchUrl; if (nextLink.empty) { siteSearchUrl = querySharePointLibraryNameApiInstance.getSiteSearchUrl(); } else { siteSearchUrl = nextLink; } // log the error addLogEntry("ERROR: Your OneDrive Account and Authentication Scope cannot access this OneDrive API: " ~ siteSearchUrl); addLogEntry("ERROR: To resolve, please discuss this issue with whomever supports your OneDrive and SharePoint environment."); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(e.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // is siteQuery a valid JSON object & contain data we can use? if ((siteQuery.type() == JSONType.object) && ("value" in siteQuery)) { // valid JSON object if (debugLogging) {addLogEntry("O365 Query Response: " ~ to!string(siteQuery), ["debug"]);} foreach (searchResult; siteQuery["value"].array) { // Need an 'exclusive' match here with sharepointLibraryNameToQuery as entered if (debugLogging) {addLogEntry("Found O365 Site: " ~ to!string(searchResult), ["debug"]);} // 'displayName' and 'id' have to be present in the search result record in order to query the site if (("displayName" in searchResult) && ("id" in searchResult)) { if (sharepointLibraryNameToQuery == searchResult["displayName"].str){ // 'displayName' matches search request site_id = searchResult["id"].str; JSONValue siteDriveQuery; string nextLinkDrive; while (true) { try { siteDriveQuery = querySharePointLibraryNameApiInstance.o365SiteDrives(site_id, nextLinkDrive); } catch (OneDriveException e) { addLogEntry("ERROR: Query of OneDrive for Office Site ID failed"); // display what the error is displayOneDriveErrorMessage(e.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // is siteDriveQuery a valid JSON object & contain data we can use? if ((siteDriveQuery.type() == JSONType.object) && ("value" in siteDriveQuery)) { // valid JSON object foreach (driveResult; siteDriveQuery["value"].array) { // Display results found = true; addLogEntry("-----------------------------------------------"); if (debugLogging) {addLogEntry("Site Details: " ~ to!string(driveResult), ["debug"]);} addLogEntry("Site Name: " ~ searchResult["displayName"].str); addLogEntry("Library Name: " ~ driveResult["name"].str); addLogEntry("drive_id: " ~ driveResult["id"].str); addLogEntry("Library URL: " ~ driveResult["webUrl"].str); } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in siteDriveQuery) { // Update nextLink to next set of SharePoint library names nextLinkDrive = siteDriveQuery["@odata.nextLink"].str; if (debugLogging) {addLogEntry("Setting nextLinkDrive to (@odata.nextLink): " ~ nextLinkDrive, ["debug"]);} // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } else { // closeout addLogEntry("-----------------------------------------------"); break; } } else { // not a valid JSON object addLogEntry("ERROR: There was an error performing this operation on Microsoft OneDrive"); addLogEntry("ERROR: Increase logging verbosity to assist determining why."); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } } } } else { // 'displayName', 'id' or ''webUrl' not present in JSON results for a specific site string siteNameAvailable = "Site 'name' was restricted by OneDrive API permissions"; bool displayNameAvailable = false; bool idAvailable = false; if ("name" in searchResult) siteNameAvailable = searchResult["name"].str; if ("displayName" in searchResult) displayNameAvailable = true; if ("id" in searchResult) idAvailable = true; // Display error details for this site data addLogEntry(); addLogEntry("ERROR: SharePoint Site details not provided for: " ~ siteNameAvailable); addLogEntry("ERROR: The SharePoint Site results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator."); addLogEntry("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName' or 'id'"); if (verboseLogging) { addLogEntry(" - Is 'displayName' available = " ~ to!string(displayNameAvailable), ["verbose"]); addLogEntry(" - Is 'id' available = " ~ to!string(idAvailable), ["verbose"]); } addLogEntry("ERROR: To debug this further, please increase application output verbosity to provide further insight as to what details are actually being returned."); } } if(!found) { // The SharePoint site we are searching for was not found in this bundle set // Add to siteSearchResults so we can display what we did find string siteSearchResultsEntry; foreach (searchResult; siteQuery["value"].array) { // We can only add the displayName if it is available if ("displayName" in searchResult) { // Use the displayName siteSearchResultsEntry = " * " ~ searchResult["displayName"].str; siteSearchResults ~= siteSearchResultsEntry; } else { // Add, but indicate displayName unavailable, use id if ("id" in searchResult) { siteSearchResultsEntry = " * " ~ "Unknown displayName (Data not provided by API), Site ID: " ~ searchResult["id"].str; siteSearchResults ~= siteSearchResultsEntry; } else { // displayName and id unavailable, display in debug log the entry if (debugLogging) {addLogEntry("Bad SharePoint Data for site: " ~ to!string(searchResult), ["debug"]);} } } } } } else { // not a valid JSON object addLogEntry("ERROR: There was an error performing this operation on Microsoft OneDrive"); addLogEntry("ERROR: Increase logging verbosity to assist determining why."); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in siteQuery) { // Update nextLink to next set of SharePoint library names nextLink = siteQuery["@odata.nextLink"].str; if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } // Was the intended target found? if(!found) { // Was the search a wildcard? if (sharepointLibraryNameToQuery != "*") { // Only print this out if the search was not a wildcard addLogEntry(); addLogEntry("ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site."); } // List all sites returned to assist user addLogEntry(); addLogEntry("The following SharePoint site names were returned:"); foreach (searchResultEntry; siteSearchResults) { // list the display name that we use to match against the user query addLogEntry(searchResultEntry); } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory querySharePointLibraryNameApiInstance.releaseCurlEngine(); querySharePointLibraryNameApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query the sync status of the client and the local system void queryOneDriveForSyncStatus(string pathToQueryStatusOn) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Query the account driveId and rootId to get the /delta JSON information // Process that JSON data for relevancy // Function variables long downloadSize = 0; string deltaLink = null; string driveIdToQuery = appConfig.defaultDriveId; string itemIdToQuery = appConfig.defaultRootId; JSONValue deltaChanges; // Array of JSON items JSONValue[] jsonItemsArray; // Query Database for a potential deltaLink starting point deltaLink = itemDB.getDeltaLink(driveIdToQuery, itemIdToQuery); // Log what we are doing addProcessingLogHeaderEntry("Querying the change status of Drive ID: " ~ driveIdToQuery, appConfig.verbosityCount); // Create a new API Instance for querying the actual /delta and initialise it OneDriveApi getDeltaDataOneDriveApiInstance; getDeltaDataOneDriveApiInstance = new OneDriveApi(appConfig); getDeltaDataOneDriveApiInstance.initialise(); while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Add a processing '.' if (appConfig.verbosityCount == 0) { addProcessingDotEntry(); } // Get the /delta changes via the OneDrive API // getDeltaChangesByItemId has the re-try logic for transient errors deltaChanges = getDeltaChangesByItemId(driveIdToQuery, itemIdToQuery, deltaLink, getDeltaDataOneDriveApiInstance); // If the initial deltaChanges response is an invalid JSON object, keep trying until we get a valid response .. if (deltaChanges.type() != JSONType.object) { // While the response is not a JSON Object or the Exit Handler has not been triggered while (deltaChanges.type() != JSONType.object) { // Handle the invalid JSON response and retry if (debugLogging) {addLogEntry("ERROR: Query of the OneDrive API via deltaChanges = getDeltaChangesByItemId() returned an invalid JSON response", ["debug"]);} deltaChanges = getDeltaChangesByItemId(driveIdToQuery, itemIdToQuery, deltaLink, getDeltaDataOneDriveApiInstance); } } // We have a valid deltaChanges JSON array. This means we have at least 200+ JSON items to process. // The API response however cannot be run in parallel as the OneDrive API sends the JSON items in the order in which they must be processed foreach (onedriveJSONItem; deltaChanges["value"].array) { // is the JSON a root object - we dont want to count this if (!isItemRoot(onedriveJSONItem)) { // Files are the only item that we want to calculate if (isItemFile(onedriveJSONItem)) { // JSON item is a file // Is the item filtered out due to client side filtering rules? if (!checkJSONAgainstClientSideFiltering(onedriveJSONItem)) { // Is the path of this JSON item 'in-scope' or 'out-of-scope' ? if (pathToQueryStatusOn != "/") { // We need to check the path of this item against pathToQueryStatusOn string thisItemPath = ""; if (("path" in onedriveJSONItem["parentReference"]) != null) { // If there is a parent reference path, try and use it string selfBuiltPath = onedriveJSONItem["parentReference"]["path"].str ~ "/" ~ onedriveJSONItem["name"].str; // Check for ':' and split if present auto splitIndex = selfBuiltPath.indexOf(":"); if (splitIndex != -1) { // Keep only the part after ':' selfBuiltPath = selfBuiltPath[splitIndex + 1 .. $]; } // Set thisItemPath to the self built path thisItemPath = selfBuiltPath; } else { // no parent reference path available thisItemPath = onedriveJSONItem["name"].str; } // can we find 'pathToQueryStatusOn' in 'thisItemPath' ? if (canFind(thisItemPath, pathToQueryStatusOn)) { // Add this to the array for processing jsonItemsArray ~= onedriveJSONItem; } } else { // We are not doing a --single-directory check // Add this to the array for processing jsonItemsArray ~= onedriveJSONItem; } } } } } // The response may contain either @odata.deltaLink or @odata.nextLink if ("@odata.deltaLink" in deltaChanges) { deltaLink = deltaChanges["@odata.deltaLink"].str; if (debugLogging) {addLogEntry("Setting next deltaLink to (@odata.deltaLink): " ~ deltaLink, ["debug"]);} } // Update deltaLink to next changeSet bundle if ("@odata.nextLink" in deltaChanges) { deltaLink = deltaChanges["@odata.nextLink"].str; if (debugLogging) {addLogEntry("Setting next deltaLink to (@odata.nextLink): " ~ deltaLink, ["debug"]);} } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } // Terminate getDeltaDataOneDriveApiInstance here getDeltaDataOneDriveApiInstance.releaseCurlEngine(); getDeltaDataOneDriveApiInstance = null; // Perform Garbage Collection on this destroyed curl engine GC.collect(); // Needed after printing out '....' when fetching changes from OneDrive API if (appConfig.verbosityCount == 0) { completeProcessingDots(); } // Are there any JSON items to process? if (count(jsonItemsArray) != 0) { // There are items to process foreach (onedriveJSONItem; jsonItemsArray.array) { // variables we need string thisItemParentDriveId; string thisItemId; string thisItemHash; bool existingDBEntry = false; // Is this file a remote item (on a shared folder) ? if (isItemRemote(onedriveJSONItem)) { // remote drive item thisItemParentDriveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str; thisItemId = onedriveJSONItem["id"].str; } else { // standard drive item thisItemParentDriveId = onedriveJSONItem["parentReference"]["driveId"].str; thisItemId = onedriveJSONItem["id"].str; } // Get the file hash if (hasHashes(onedriveJSONItem)) { // At a minimum we require 'quickXorHash' to exist if (hasQuickXorHash(onedriveJSONItem)) { // JSON item has a hash we can use thisItemHash = onedriveJSONItem["file"]["hashes"]["quickXorHash"].str; } // Check if the item has been seen before Item existingDatabaseItem; existingDBEntry = itemDB.selectById(thisItemParentDriveId, thisItemId, existingDatabaseItem); if (existingDBEntry) { // item exists in database .. do the database details match the JSON record? if (existingDatabaseItem.quickXorHash != thisItemHash) { // file hash is different, this will trigger a download event if (hasFileSize(onedriveJSONItem)) { downloadSize = downloadSize + onedriveJSONItem["size"].integer; } } } else { // item does not exist in the database // this item has already passed client side filtering rules (skip_dir, skip_file, sync_list) // this will trigger a download event if (hasFileSize(onedriveJSONItem)) { downloadSize = downloadSize + onedriveJSONItem["size"].integer; } } } } } // Was anything detected that would constitute a download? if (downloadSize > 0) { // we have something to download if (pathToQueryStatusOn != "/") { addLogEntry("The selected local directory via --single-directory is out of sync with Microsoft OneDrive"); } else { addLogEntry("The configured local 'sync_dir' directory is out of sync with Microsoft OneDrive"); } addLogEntry("Approximate data to download from Microsoft OneDrive: " ~ to!string(downloadSize/1024) ~ " KB"); } else { // No changes were returned addLogEntry("There are no pending changes from Microsoft OneDrive; your local directory matches the data online."); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query OneDrive for file details of a given path, returning either the 'webURL' or 'lastModifiedBy' JSON facet void queryOneDriveForFileDetails(string inputFilePath, string runtimePath, string outputType) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } OneDriveApi queryOneDriveForFileDetailsApiInstance; // Calculate the full local file path string fullLocalFilePath = buildNormalizedPath(buildPath(runtimePath, inputFilePath)); // Query if file is valid locally if (exists(fullLocalFilePath)) { // search drive_id list string[] distinctDriveIds = itemDB.selectDistinctDriveIds(); bool pathInDB = false; Item dbItem; foreach (searchDriveId; distinctDriveIds) { // Does this path exist in the database, use the 'inputFilePath' if (itemDB.selectByPath(inputFilePath, searchDriveId, dbItem)) { // item is in the database pathInDB = true; JSONValue fileDetailsFromOneDrive; // Create a new API Instance for this thread and initialise it queryOneDriveForFileDetailsApiInstance = new OneDriveApi(appConfig); queryOneDriveForFileDetailsApiInstance.initialise(); try { fileDetailsFromOneDrive = queryOneDriveForFileDetailsApiInstance.getPathDetailsById(dbItem.driveId, dbItem.id); // Dont cleanup here as if we are creating a shareable file link (below) it is still needed } catch (OneDriveException exception) { // display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryOneDriveForFileDetailsApiInstance.releaseCurlEngine(); queryOneDriveForFileDetailsApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // Is the API response a valid JSON file? if (fileDetailsFromOneDrive.type() == JSONType.object) { // debug output of response if (debugLogging) {addLogEntry("API Response: " ~ to!string(fileDetailsFromOneDrive), ["debug"]);} // What sort of response to we generate // --get-file-link response if (outputType == "URL") { if ((fileDetailsFromOneDrive.type() == JSONType.object) && ("webUrl" in fileDetailsFromOneDrive)) { // Valid JSON object addLogEntry(); writeln("WebURL: ", fileDetailsFromOneDrive["webUrl"].str); } } // --modified-by response if (outputType == "ModifiedBy") { if ((fileDetailsFromOneDrive.type() == JSONType.object) && ("lastModifiedBy" in fileDetailsFromOneDrive)) { // Valid JSON object writeln(); writeln("Last modified: ", fileDetailsFromOneDrive["lastModifiedDateTime"].str); writeln("Last modified by: ", fileDetailsFromOneDrive["lastModifiedBy"]["user"]["displayName"].str); // if 'email' provided, add this to the output if ("email" in fileDetailsFromOneDrive["lastModifiedBy"]["user"]) { writeln("Email Address: ", fileDetailsFromOneDrive["lastModifiedBy"]["user"]["email"].str); } } } // --create-share-link response if (outputType == "ShareableLink") { JSONValue accessScope; JSONValue createShareableLinkResponse; string thisDriveId = fileDetailsFromOneDrive["parentReference"]["driveId"].str; string thisItemId = fileDetailsFromOneDrive["id"].str; string fileShareLink; bool writeablePermissions = appConfig.getValueBool("with_editing_perms"); // What sort of shareable link is required? if (writeablePermissions) { // configure the read-write access scope accessScope = [ "type": "edit", "scope": "anonymous" ]; } else { // configure the read-only access scope (default) accessScope = [ "type": "view", "scope": "anonymous" ]; } // If a share-password was passed use it when creating the link if (strip(appConfig.getValueString("share_password")) != "") { accessScope["password"] = appConfig.getValueString("share_password"); } // Try and create the shareable file link try { createShareableLinkResponse = queryOneDriveForFileDetailsApiInstance.createShareableLink(thisDriveId, thisItemId, accessScope); } catch (OneDriveException exception) { // display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); return; } // Is the API response a valid JSON file? if ((createShareableLinkResponse.type() == JSONType.object) && ("link" in createShareableLinkResponse)) { // Extract the file share link from the JSON response fileShareLink = createShareableLinkResponse["link"]["webUrl"].str; writeln("File Shareable Link: ", fileShareLink); if (writeablePermissions) { writeln("Shareable Link has read-write permissions - use and provide with caution"); } } } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryOneDriveForFileDetailsApiInstance.releaseCurlEngine(); queryOneDriveForFileDetailsApiInstance = null; // Perform Garbage Collection GC.collect(); } } // was path found? if (!pathInDB) { // File has not been synced with OneDrive addLogEntry("Selected path has not been synced with Microsoft OneDrive: " ~ inputFilePath); } } else { // File does not exist locally addLogEntry("Selected path not found on local system: " ~ inputFilePath); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query OneDrive for the quota details void queryOneDriveForQuotaDetails() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function is similar to getRemainingFreeSpace() but is different in data being analysed and output method JSONValue currentDriveQuota; string driveId; OneDriveApi getCurrentDriveQuotaApiInstance; if (appConfig.getValueString("drive_id").length) { driveId = appConfig.getValueString("drive_id"); } else { driveId = appConfig.defaultDriveId; } try { // Create a new OneDrive API instance getCurrentDriveQuotaApiInstance = new OneDriveApi(appConfig); getCurrentDriveQuotaApiInstance.initialise(); if (debugLogging) {addLogEntry("Seeking available quota for this drive id: " ~ driveId, ["debug"]);} currentDriveQuota = getCurrentDriveQuotaApiInstance.getDriveQuota(driveId); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getCurrentDriveQuotaApiInstance.releaseCurlEngine(); getCurrentDriveQuotaApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException e) { if (debugLogging) {addLogEntry("currentDriveQuota = onedrive.getDriveQuota(driveId) generated a OneDriveException", ["debug"]);} // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getCurrentDriveQuotaApiInstance.releaseCurlEngine(); getCurrentDriveQuotaApiInstance = null; // Perform Garbage Collection GC.collect(); } // validate that currentDriveQuota is a JSON value if (currentDriveQuota.type() == JSONType.object) { // was 'quota' in response? if ("quota" in currentDriveQuota) { // debug output of response if (debugLogging) {addLogEntry("currentDriveQuota: " ~ to!string(currentDriveQuota), ["debug"]);} // human readable output of response string deletedValue = "Not Provided"; string remainingValue = "Not Provided"; string stateValue = "Not Provided"; string totalValue = "Not Provided"; string usedValue = "Not Provided"; // Update values if ("deleted" in currentDriveQuota["quota"]) { deletedValue = byteToGibiByte(currentDriveQuota["quota"]["deleted"].integer); } if ("remaining" in currentDriveQuota["quota"]) { remainingValue = byteToGibiByte(currentDriveQuota["quota"]["remaining"].integer); } if ("state" in currentDriveQuota["quota"]) { stateValue = currentDriveQuota["quota"]["state"].str; } if ("total" in currentDriveQuota["quota"]) { totalValue = byteToGibiByte(currentDriveQuota["quota"]["total"].integer); } if ("used" in currentDriveQuota["quota"]) { usedValue = byteToGibiByte(currentDriveQuota["quota"]["used"].integer); } writeln("Microsoft OneDrive quota information as reported for this Drive ID: ", driveId); writeln(); writeln("Deleted: ", deletedValue, " GB (", currentDriveQuota["quota"]["deleted"].integer, " bytes)"); writeln("Remaining: ", remainingValue, " GB (", currentDriveQuota["quota"]["remaining"].integer, " bytes)"); writeln("State: ", stateValue); writeln("Total: ", totalValue, " GB (", currentDriveQuota["quota"]["total"].integer, " bytes)"); writeln("Used: ", usedValue, " GB (", currentDriveQuota["quota"]["used"].integer, " bytes)"); writeln(); } else { writeln("Microsoft OneDrive quota information is being restricted for this Drive ID: ", driveId); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query the system for session_upload.* files bool checkForInterruptedSessionUploads() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } bool interruptedUploads = false; long interruptedUploadsCount; // Scan the filesystem for the files we are interested in, build up interruptedUploadsSessionFiles array foreach (sessionFile; dirEntries(appConfig.configDirName, "session_upload.*", SpanMode.shallow)) { // calculate the full path string tempPath = buildNormalizedPath(buildPath(appConfig.configDirName, sessionFile)); // add to array interruptedUploadsSessionFiles ~= [tempPath]; } // Count all 'session_upload' files in appConfig.configDirName interruptedUploadsCount = count(interruptedUploadsSessionFiles); if (interruptedUploadsCount != 0) { interruptedUploads = true; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return if there are interrupted uploads to process return interruptedUploads; } // Query the system for resume_download.* files bool checkForResumableDownloads() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } bool resumableDownloads = false; long resumableDownloadsCount; // Scan the filesystem for the files we are interested in, build up interruptedDownloadFiles array foreach (resumeDownloadFile; dirEntries(appConfig.configDirName, "resume_download.*", SpanMode.shallow)) { // calculate the full path string tempPath = buildNormalizedPath(buildPath(appConfig.configDirName, resumeDownloadFile)); // add to array interruptedDownloadFiles ~= [tempPath]; } // Count all 'resume_download' files in appConfig.configDirName resumableDownloadsCount = count(interruptedDownloadFiles); if (resumableDownloadsCount != 0) { resumableDownloads = true; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return if there are interrupted uploads to process return resumableDownloads; } // Clear any session_upload.* files void clearInterruptedSessionUploads() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Scan the filesystem for the files we are interested in, build up interruptedUploadsSessionFiles array foreach (sessionFile; dirEntries(appConfig.configDirName, "session_upload.*", SpanMode.shallow)) { // calculate the full path string tempPath = buildNormalizedPath(buildPath(appConfig.configDirName, sessionFile)); JSONValue sessionFileData = readText(tempPath).parseJSON(); addLogEntry("Removing interrupted session upload file due to --resync for: " ~ sessionFileData["localPath"].str, ["info"]); // Process removal if (!dryRun) { safeRemove(tempPath); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Clear any resume_download.* files void clearInterruptedDownloads() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Scan the filesystem for the files we are interested in, build up interruptedDownloadFiles array foreach (resumeDownloadFile; dirEntries(appConfig.configDirName, "resume_download.*", SpanMode.shallow)) { // calculate the full path string tempPath = buildNormalizedPath(buildPath(appConfig.configDirName, resumeDownloadFile)); JSONValue resumeFileData = readText(tempPath).parseJSON(); addLogEntry("Removing interrupted download file due to --resync for: " ~ resumeFileData["originalFilename"].str, ["info"]); string resumeFilename = resumeFileData["downloadFilename"].str; // Process removal if (!dryRun) { // remove the .partial file safeRemove(resumeFilename); // remove the resume_download. file safeRemove(tempPath); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process interrupted 'session_upload' files void processInterruptedSessionUploads() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // For each upload_session file that has been found, process the data to ensure it is still valid foreach (sessionFilePath; interruptedUploadsSessionFiles) { // What session data are we trying to restore if (verboseLogging) {addLogEntry("Attempting to restore file upload session using this session data file: " ~ sessionFilePath, ["verbose"]);} // Does this pass validation? if (!validateUploadSessionFileData(sessionFilePath)) { // Remove upload_session file as it is invalid // upload_session file contains an error - cant resume this session if (verboseLogging) {addLogEntry("Restore file upload session failed - cleaning up resumable session data file: " ~ sessionFilePath, ["verbose"]);} // cleanup session path if (exists(sessionFilePath)) { if (!dryRun) { safeRemove(sessionFilePath); } } } } // At this point we should have an array of JSON items to resume uploading if (count(jsonItemsToResumeUpload) > 0) { // there are valid items to resume upload // Lets deal with all the JSON items that need to be resumed for upload in a batch process size_t batchSize = to!int(appConfig.getValueLong("threads")); long batchCount = (jsonItemsToResumeUpload.length + batchSize - 1) / batchSize; long batchesProcessed = 0; foreach (chunk; jsonItemsToResumeUpload.chunks(batchSize)) { // send an array containing 'appConfig.getValueLong("threads")' JSON items to resume upload resumeSessionUploadsInParallel(chunk); } // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Process 'resumable download' files that were found void processResumableDownloadFiles() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // For each 'resume_download' file that has been found, process the data to ensure it is still valid foreach (resumeDownloadFile; interruptedDownloadFiles) { // What 'resumable data' are we trying to resume if (verboseLogging) {addLogEntry("Attempting to resume file download using this 'resumable data' file: " ~ resumeDownloadFile, ["verbose"]);} // Does this pass validation? if (!validateResumableDownloadFileData(resumeDownloadFile)) { // Remove 'resume_download' file as it is invalid if (verboseLogging) {addLogEntry("Resume file download verification failed - cleaning up resumable download data file: " ~ resumeDownloadFile, ["verbose"]);} // Cleanup 'resume_download' file if (exists(resumeDownloadFile)) { if (!dryRun) { safeRemove(resumeDownloadFile); } } } } // At this point we should have an array of JSON items to resume downloading if (count(jsonItemsToResumeDownload) > 0) { // There are valid items to resume download // Lets deal with all the JSON items that need to be resumed for download in a batch process size_t batchSize = to!int(appConfig.getValueLong("threads")); long batchCount = (jsonItemsToResumeDownload.length + batchSize - 1) / batchSize; long batchesProcessed = 0; foreach (chunk; jsonItemsToResumeDownload.chunks(batchSize)) { // send an array containing 'appConfig.getValueLong("threads")' JSON items to resume download resumeDownloadsInParallel(chunk); } // For this set of items, perform a DB PASSIVE checkpoint itemDB.performCheckpoint("PASSIVE"); } // Cleanup all 'resume_download' files foreach (resumeDownloadFile; interruptedDownloadFiles) { safeRemove(resumeDownloadFile); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // A resume session upload file needs to be valid to be used // This function validates this data bool validateUploadSessionFileData(string sessionFilePath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return ; code, so that this function operates as efficiently as possible. // It is pointless having the entire code run through and performing additional needless checks where it is not required // Whilst this means some extra code / duplication in this function, it cannot be helped JSONValue sessionFileData; OneDriveApi validateUploadSessionFileDataApiInstance; // Try and read the text from the session file as a JSON array try { if (getSize(sessionFilePath) > 0) { // There is data to read in sessionFileData = readText(sessionFilePath).parseJSON(); } else { // No data to read in - invalid file if (debugLogging) {addLogEntry("SESSION-RESUME: Invalid JSON file: " ~ sessionFilePath, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } catch (JSONException e) { if (debugLogging) {addLogEntry("SESSION-RESUME: Invalid JSON data in: " ~ sessionFilePath, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Does the file we wish to resume uploading exist locally still? if ("localPath" in sessionFileData) { string sessionLocalFilePath = sessionFileData["localPath"].str; if (debugLogging) {addLogEntry("SESSION-RESUME: sessionLocalFilePath: " ~ sessionLocalFilePath, ["debug"]);} // Does the file exist? if (!exists(sessionLocalFilePath)) { if (verboseLogging) {addLogEntry("The local file to upload does not exist locally anymore", ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Can we read the file? if (!readLocalFile(sessionLocalFilePath)) { // filesystem error already returned if unable to read // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } else { if (debugLogging) {addLogEntry("SESSION-RESUME: No localPath data in: " ~ sessionFilePath, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Check the session data for expirationDateTime if ("expirationDateTime" in sessionFileData) { SysTime expiration; string expirationTimestamp; expirationTimestamp = strip(sessionFileData["expirationDateTime"].str); // is expirationTimestamp valid? if (isValidUTCDateTime(expirationTimestamp)) { // string is a valid timestamp expiration = SysTime.fromISOExtString(expirationTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ expirationTimestamp); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // valid timestamp if (expiration < Clock.currTime()) { if (verboseLogging) {addLogEntry("The upload session has expired for: " ~ sessionFilePath, ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } else { if (debugLogging) {addLogEntry("SESSION-RESUME: No expirationDateTime data in: " ~ sessionFilePath, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Check the online upload status, using the uloadURL in sessionFileData if ("uploadUrl" in sessionFileData) { JSONValue response; try { // Create a new OneDrive API instance validateUploadSessionFileDataApiInstance = new OneDriveApi(appConfig); validateUploadSessionFileDataApiInstance.initialise(); // Request upload status response = validateUploadSessionFileDataApiInstance.requestUploadStatus(sessionFileData["uploadUrl"].str); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory validateUploadSessionFileDataApiInstance.releaseCurlEngine(); validateUploadSessionFileDataApiInstance = null; // Perform Garbage Collection GC.collect(); // no error .. potentially all still valid } catch (OneDriveException e) { // handle any onedrive error response as invalid if (debugLogging) {addLogEntry("SESSION-RESUME: Invalid response when using uploadUrl in: " ~ sessionFilePath, ["debug"]);} // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory validateUploadSessionFileDataApiInstance.releaseCurlEngine(); validateUploadSessionFileDataApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Do we have a valid response from OneDrive? if (response.type() == JSONType.object) { // Valid JSON object was returned if (("expirationDateTime" in response) && ("nextExpectedRanges" in response)) { // The 'uploadUrl' is valid, and the response contains elements we need sessionFileData["expirationDateTime"] = response["expirationDateTime"]; sessionFileData["nextExpectedRanges"] = response["nextExpectedRanges"]; if (sessionFileData["nextExpectedRanges"].array.length == 0) { if (verboseLogging) {addLogEntry("The upload session was already completed", ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } else { if (debugLogging) {addLogEntry("SESSION-RESUME: No expirationDateTime & nextExpectedRanges data in Microsoft OneDrive API response: " ~ to!string(response), ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } else { // not a JSON object if (verboseLogging) {addLogEntry("Restore file upload session failed - invalid response from Microsoft OneDrive", ["verbose"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } } else { if (debugLogging) {addLogEntry("SESSION-RESUME: No uploadUrl data in: " ~ sessionFilePath, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is invalid return false; } // Add 'sessionFilePath' to 'sessionFileData' so that it can be used when we reuse the JSON data to resume the upload sessionFileData["sessionFilePath"] = sessionFilePath; // Add sessionFileData to jsonItemsToResumeUpload as it is now valid jsonItemsToResumeUpload ~= sessionFileData; // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return session file is valid return true; } // A 'resumable download' file needs to be valid to be used bool validateResumableDownloadFileData(string resumeDownloadFile) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables JSONValue resumeDownloadFileData; JSONValue latestOnlineFileDetails; OneDriveApi validateResumableDownloadFileDataApiInstance; string driveId; string itemId; string existingHash; string downloadFilename; long resumeOffset; string OneDriveFileXORHash; string OneDriveFileSHA256Hash; // Try and read the text from the 'resumable download' file as a JSON array try { if (getSize(resumeDownloadFile) > 0) { // There is data to read in resumeDownloadFileData = readText(resumeDownloadFile).parseJSON(); } else { // No data to read in - invalid file if (debugLogging) {addLogEntry("SESSION-RESUME: Invalid JSON file: " ~ resumeDownloadFile, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return 'resumable download' file is invalid return false; } } catch (JSONException e) { if (debugLogging) {addLogEntry("SESSION-RESUME: Invalid JSON data in: " ~ resumeDownloadFile, ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return 'resumable download' file is invalid return false; } // What needs to be checked? // - JSON has 'downloadFilename' - critical to check the online state // - JSON has 'driveId' - critical to check the online state // - JSON has 'itemId' - critical to check the online state // - JSON has 'resumeOffset' - critical to check the online state // - JSON has 'onlineHash' with an applicable hash value - critical to check the online state if (!hasDownloadFilename(resumeDownloadFileData)) { // no downloadFilename present - file invalid if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'downloadFilename'", ["verbose"]);} // Return 'resumable download' file is invalid return false; } else { // Configure search variables downloadFilename = resumeDownloadFileData["downloadFilename"].str; // Does the file specified by 'downloadFilename' exist on disk? if (!exists(downloadFilename)) { // File that is supposed to contain our resumable if (verboseLogging) {addLogEntry("The 'resumable download' file no longer exists on your local disk: " ~ downloadFilename, ["verbose"]);} // Return 'resumable download' file is invalid return false; } } // If we get to this point 'downloadFilename' has a file name and the file exists on disk. // If any of the other validations fail, we can remove the file if (!hasDriveId(resumeDownloadFileData)) { // no driveId present - file invalid if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'driveId'", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } else { // Configure search variables driveId = resumeDownloadFileData["driveId"].str; } if (!hasItemId(resumeDownloadFileData)) { // no itemId present - file invalid if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'itemId'", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } else { // Configure search variables itemId = resumeDownloadFileData["itemId"].str; } if (!hasResumeOffset(resumeDownloadFileData)) { // no resumeOffset present - file invalid if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'resumeOffset'", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } else { // we have a resumeOffset value resumeOffset = to!long(resumeDownloadFileData["resumeOffset"].str); // We need to check 'resumeOffset' against the 'downloadFilename' on-disk size long onDiskSize = getSize(downloadFilename); if (resumeOffset != onDiskSize) { // The size of the offset location does not equal the size on disk .. if we resume that file, the file will be corrupt string logMessage = format("The 'resumable download' file on disk is a different size to the resumable offset: %s vs %s", to!string(resumeOffset), to!string(onDiskSize)); if (verboseLogging) {addLogEntry(logMessage, ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } } if (!hasOnlineHash(resumeDownloadFileData)) { // no onlineHash present - file invalid if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'onlineHash'", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } else { // Configure hash variable from the resume data // QuickXorHash Check if (hasQuickXorHashResume(resumeDownloadFileData)) { // We have a quickXorHash value existingHash = resumeDownloadFileData["onlineHash"]["quickXorHash"].str; } else { // Fallback: Check for SHA256Hash if (hasSHA256HashResume(resumeDownloadFileData)) { // We have a sha256Hash value existingHash = resumeDownloadFileData["onlineHash"]["sha256Hash"].str; } } // At this point if we do not have a existingHash value, its a fail if (existingHash.empty) { if (verboseLogging) {addLogEntry("The 'resumable download' file contains invalid data: Missing 'onlineHash' value", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } } // At this point we have elements in the 'resumable download' JSON data that will allow is to check if the online file has been modified - if it has, resuming the download is pointless try { // Create a new OneDrive API instance validateResumableDownloadFileDataApiInstance = new OneDriveApi(appConfig); validateResumableDownloadFileDataApiInstance.initialise(); // Request latest file details latestOnlineFileDetails = validateResumableDownloadFileDataApiInstance.getPathDetailsById(driveId, itemId); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory validateResumableDownloadFileDataApiInstance.releaseCurlEngine(); validateResumableDownloadFileDataApiInstance = null; // Perform Garbage Collection GC.collect(); // no error .. potentially all still valid } catch (OneDriveException e) { // handle any onedrive error response as invalid // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory validateResumableDownloadFileDataApiInstance.releaseCurlEngine(); validateResumableDownloadFileDataApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return 'resumable download' file is invalid return false; } // Configure the hashes from the online data for comparison if (hasHashes(latestOnlineFileDetails)) { // File details returned hash details // QuickXorHash if (hasQuickXorHash(latestOnlineFileDetails)) { // Use the provided quickXorHash as reported by OneDrive if (latestOnlineFileDetails["file"]["hashes"]["quickXorHash"].str != "") { OneDriveFileXORHash = latestOnlineFileDetails["file"]["hashes"]["quickXorHash"].str; } } else { // Fallback: Check for SHA256Hash if (hasSHA256Hash(latestOnlineFileDetails)) { // Use the provided sha256Hash as reported by OneDrive if (latestOnlineFileDetails["file"]["hashes"]["sha256Hash"].str != "") { OneDriveFileSHA256Hash = latestOnlineFileDetails["file"]["hashes"]["sha256Hash"].str; } } } } // Last check - has the online file changed since we attempted to do the download that we are trying to resume? // Test 'existingHash' against the potential 2 online hashes for a match // As we dont know what type of hash 'existingHash' is, we have to test it against the 2 known online types bool hashesMatch = (existingHash == OneDriveFileXORHash) || (existingHash == OneDriveFileSHA256Hash); // Do the hashes match? if (!hashesMatch) { // Hashes do not match if (verboseLogging) {addLogEntry("The 'online file' has changed in content since the download was last attempted. Aborting this resumable download attempt.", ["verbose"]);} // Remove local file safeRemove(downloadFilename); // Return 'resumable download' file is invalid return false; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Augment 'latestOnlineFileDetails' with our resume point latestOnlineFileDetails["resumeOffset"] = JSONValue(to!string(resumeOffset)); // Add latestOnlineFileDetails to jsonItemsToResumeDownload as it is now valid jsonItemsToResumeDownload ~= latestOnlineFileDetails; // Return 'resumable download' file is valid return true; } // Resume all resumable session uploads in parallel void resumeSessionUploadsInParallel(JSONValue[] array) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function received an array of JSON items to resume upload, the number of elements based on appConfig.getValueLong("threads") foreach (i, jsonItemToResume; processPool.parallel(array)) { // Take each JSON item and resume upload using the JSON data JSONValue uploadResponse; OneDriveApi uploadFileOneDriveApiInstance; // Create a new API instance uploadFileOneDriveApiInstance = new OneDriveApi(appConfig); uploadFileOneDriveApiInstance.initialise(); // Pull out data from this JSON element string threadUploadSessionFilePath = jsonItemToResume["sessionFilePath"].str; long thisFileSizeLocal = getSize(jsonItemToResume["localPath"].str); // Try to resume the session upload using the provided data try { uploadResponse = performSessionFileUpload(uploadFileOneDriveApiInstance, thisFileSizeLocal, jsonItemToResume, threadUploadSessionFilePath); } catch (OneDriveException exception) { writeln("CODING TO DO: Handle an exception when performing a resume session upload"); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory uploadFileOneDriveApiInstance.releaseCurlEngine(); uploadFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Was the response from the OneDrive API a valid JSON item? if (uploadResponse.type() == JSONType.object) { // A valid JSON object was returned - session resumption upload successful // Are we in an --upload-only & --remove-source-files scenario? // Use actual config values as we are doing an upload session recovery if ((uploadOnly) && (localDeleteAfterUpload)) { // Perform the local file deletion removeLocalFilePostUpload(jsonItemToResume["localPath"].str); // as file is removed, we have nothing to add to the local database if (debugLogging) {addLogEntry("Skipping adding to database as --upload-only & --remove-source-files configured", ["debug"]);} } else { // Save JSON item in database saveItem(uploadResponse); } } else { // No valid response was returned addLogEntry("CODING TO DO: what to do when session upload resumption JSON data is not valid ... nothing ? error message ?"); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Resume all resumable downloads in parallel void resumeDownloadsInParallel(JSONValue[] array) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // This function received an array of JSON items to resume download, the number of elements based on appConfig.getValueLong("threads") foreach (i, jsonItemToResume; processPool.parallel(array)) { // Take each JSON item and resume download using the JSON data // Extract the 'offset' from the JSON data long resumeOffset; resumeOffset = to!long(jsonItemToResume["resumeOffset"].str); // Take each JSON item and download it using the offset downloadFileItem(jsonItemToResume, false, resumeOffset); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Function to process the path by removing prefix up to ':' - remove '/drive/root:' from a path string string processPathToRemoveRootReference(ref string pathToCheck) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } size_t colonIndex = pathToCheck.indexOf(":"); if (colonIndex != -1) { if (debugLogging) {addLogEntry("Updating " ~ pathToCheck ~ " to remove prefix up to ':'", ["debug"]);} pathToCheck = pathToCheck[colonIndex + 1 .. $]; if (debugLogging) {addLogEntry("Updated path: " ~ pathToCheck, ["debug"]);} } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return updated path return pathToCheck; } // Generate path from JSON data string generatePathFromJSONData(JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function variables string parentPath; string combinedPath; string computedItemPath; bool parentInDatabase = false; // Set itemName string itemName = onedriveJSONItem["name"].str; // If this item is on our 'driveId' then use the following, otherwise we need to calculate parental path to display the 'correct' path string thisItemDriveId = onedriveJSONItem["parentReference"]["driveId"].str; string thisItemParentId = onedriveJSONItem["parentReference"]["id"].str; // Issue #3336 - Convert driveId to lowercase before any test if (appConfig.accountType == "personal") { thisItemDriveId = transformToLowerCase(thisItemDriveId); } if (thisItemDriveId == appConfig.defaultDriveId) { // As this is on our driveId, use the path details as is parentPath = onedriveJSONItem["parentReference"]["path"].str; combinedPath = buildNormalizedPath(buildPath(parentPath, itemName)); } else { // As this is not our driveId, the 'path' reference above is the 'full' remote path, which is not reflective of our location' // Are the 'parent' details in the database? parentInDatabase = itemDB.idInLocalDatabase(thisItemDriveId, thisItemParentId); if (parentInDatabase) { // Parent in DB .. we can calculate path computedItemPath = computeItemPath(thisItemDriveId, thisItemParentId); combinedPath = buildNormalizedPath(buildPath(computedItemPath, itemName)); } else { // We cant calculate this path parentPath = onedriveJSONItem["parentReference"]["name"].str; combinedPath = buildNormalizedPath(buildPath(parentPath, itemName)); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return processPathToRemoveRootReference(combinedPath); } // Function to find a given DriveId in the onlineDriveDetails associative array that maps driveId to DriveDetailsCache // If 'true' will return 'driveDetails' containing the struct data 'DriveDetailsCache' bool canFindDriveId(string driveId, out DriveDetailsCache driveDetails) { // Not adding performance metrics to this function auto ptr = driveId in onlineDriveDetails; if (ptr !is null) { driveDetails = *ptr; // Dereference the pointer to get the value return true; } else { return false; } } // Add this driveId plus relevant details for future reference and use void addOrUpdateOneDriveOnlineDetails(string driveId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } bool quotaRestricted; bool quotaAvailable; long quotaRemaining; // Get the data from online auto onlineDriveData = getRemainingFreeSpaceOnline(driveId); quotaRestricted = to!bool(onlineDriveData[0][0]); quotaAvailable = to!bool(onlineDriveData[0][1]); quotaRemaining = to!long(onlineDriveData[0][2]); onlineDriveDetails[driveId] = DriveDetailsCache(driveId, quotaRestricted, quotaAvailable, quotaRemaining); // Debug log what the cached array now contains if (debugLogging) {addLogEntry("onlineDriveDetails: " ~ to!string(onlineDriveDetails), ["debug"]);} // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Return a specific 'driveId' details from 'onlineDriveDetails' DriveDetailsCache getDriveDetails(string driveId) { // Not adding performance metrics to this function auto ptr = driveId in onlineDriveDetails; if (ptr !is null) { return *ptr; // Dereference the pointer to get the value } else { // Return a default DriveDetailsCache or handle the case where the driveId is not found return DriveDetailsCache.init; // Return default-initialised struct } } // Search a given Drive ID, Item ID and filename to see if this exists in the location specified JSONValue searchDriveItemForFile(string parentItemDriveId, string parentItemId, string fileToUpload) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } JSONValue onedriveJSONItem; string searchName = baseName(fileToUpload); JSONValue thisLevelChildren; string nextLink; // Create a new API Instance for this thread and initialise it OneDriveApi checkFileOneDriveApiInstance; checkFileOneDriveApiInstance = new OneDriveApi(appConfig); checkFileOneDriveApiInstance.initialise(); while (true) { // Check if exitHandlerTriggered is true if (exitHandlerTriggered) { // break out of the 'while (true)' loop break; } // Try and query top level children try { thisLevelChildren = checkFileOneDriveApiInstance.listChildren(parentItemDriveId, parentItemId, nextLink); } catch (OneDriveException exception) { // OneDrive threw an error if (debugLogging) { addLogEntry(debugLogBreakType1, ["debug"]); addLogEntry("Query Error: thisLevelChildren = checkFileOneDriveApiInstance.listChildren(parentItemDriveId, parentItemId, nextLink)", ["debug"]); addLogEntry("driveId: " ~ parentItemDriveId, ["debug"]); addLogEntry("idToQuery: " ~ parentItemId, ["debug"]); addLogEntry("nextLink: " ~ nextLink, ["debug"]); } // Handle the 404 error code - the parent item id was not found on the drive id specified if (exception.httpStatusCode == 404) { // Return an empty JSON item, as parent item could not be found, thus any child object will never be found return onedriveJSONItem; } else { // Default operation if not 408,429,503,504 errors // - 408,429,503,504 errors are handled as a retry within oneDriveApiInstance // Display what the error is displayOneDriveErrorMessage(exception.msg, thisFunctionName); } } // 'thisLevelChildren' must be a valid JSON response to progress any further if (thisLevelChildren.type() == JSONType.object) { // Process thisLevelChildren response foreach (child; thisLevelChildren["value"].array) { // Only looking at files if ((child["name"].str == searchName) && (("file" in child) != null)) { // Found the matching file, return its JSON representation // Operations in this thread are done / complete // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory checkFileOneDriveApiInstance.releaseCurlEngine(); checkFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return child as found item return child; } } // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in thisLevelChildren) { // Update nextLink to next changeSet bundle if (debugLogging) {addLogEntry("Setting nextLink to (@odata.nextLink): " ~ nextLink, ["debug"]);} nextLink = thisLevelChildren["@odata.nextLink"].str; } else break; // Sleep for a while to avoid busy-waiting Thread.sleep(dur!"msecs"(100)); // Adjust the sleep duration as needed } else { // API response was not a valid response // Break out of the 'while (true)' loop break; } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory checkFileOneDriveApiInstance.releaseCurlEngine(); checkFileOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // return an empty JSON item, as search item was not found return onedriveJSONItem; } // Update 'onlineDriveDetails' with the latest data about this drive void updateDriveDetailsCache(string driveId, bool quotaRestricted, bool quotaAvailable, long localFileSize) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // As each thread is running differently, what is the current 'quotaRemaining' for 'driveId' ? long quotaRemaining; DriveDetailsCache cachedOnlineDriveData; cachedOnlineDriveData = getDriveDetails(driveId); quotaRemaining = cachedOnlineDriveData.quotaRemaining; // Update 'quotaRemaining' quotaRemaining = quotaRemaining - localFileSize; // Do the flags get updated? if (quotaRemaining <= 0) { if (appConfig.accountType == "personal"){ // Issue #3336 - Convert driveId to lowercase before any test driveId = transformToLowerCase(driveId); if (driveId == appConfig.defaultDriveId) { // zero space available on our drive addLogEntry("ERROR: OneDrive account currently has zero space available. Please free up some space online or purchase additional capacity."); quotaRemaining = 0; quotaAvailable = false; } } else { // zero space available is being reported, maybe being restricted? if (verboseLogging) {addLogEntry("WARNING: OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.", ["verbose"]);} quotaRemaining = 0; quotaRestricted = true; } } // Updated the details onlineDriveDetails[driveId] = DriveDetailsCache(driveId, quotaRestricted, quotaAvailable, quotaRemaining); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Update all of the known cached driveId quota details void freshenCachedDriveQuotaDetails() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } foreach (driveId; onlineDriveDetails.keys) { // Update this driveid quota details if (debugLogging) {addLogEntry("Freshen Quota Details for this driveId: " ~ driveId, ["debug"]);} addOrUpdateOneDriveOnlineDetails(driveId); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Create a 'root' DB Tie Record for a Shared Folder from the JSON data void createDatabaseRootTieRecordForOnlineSharedFolder(JSONValue onedriveJSONItem, string relocatedFolderDriveId = null, string relocatedFolderParentId = null) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Creating|Updating a DB Tie if (debugLogging) { addLogEntry("Creating|Updating a 'root' DB Tie Record for this Shared Folder (Actual 'Shared With Me' Folder Name): " ~ onedriveJSONItem["name"].str, ["debug"]); addLogEntry("Raw JSON for 'root' DB Tie Record: " ~ to!string(onedriveJSONItem), ["debug"]); } // New DB Tie Item to detail the 'root' of the Shared Folder Item tieDBItem; string lastModifiedTimestamp; tieDBItem.name = "root"; // Get the right parentReference details if (isItemRemote(onedriveJSONItem)) { tieDBItem.driveId = onedriveJSONItem["remoteItem"]["parentReference"]["driveId"].str; tieDBItem.id = onedriveJSONItem["remoteItem"]["id"].str; } else { if (onedriveJSONItem["name"].str != "root") { tieDBItem.driveId = onedriveJSONItem["parentReference"]["driveId"].str; // OneDrive Personal JSON responses are in-consistent with not having 'id' available if (hasParentReferenceId(onedriveJSONItem)) { // Use the parent reference id tieDBItem.id = onedriveJSONItem["parentReference"]["id"].str; } else { // Testing evidence shows that for Personal accounts, use the 'id' itself tieDBItem.id = onedriveJSONItem["id"].str; } } else { tieDBItem.driveId = onedriveJSONItem["parentReference"]["driveId"].str; tieDBItem.id = onedriveJSONItem["id"].str; } } // set the item type tieDBItem.type = ItemType.root; // get the lastModifiedDateTime lastModifiedTimestamp = strip(onedriveJSONItem["fileSystemInfo"]["lastModifiedDateTime"].str); // is lastModifiedTimestamp valid? if (isValidUTCDateTime(lastModifiedTimestamp)) { // string is a valid timestamp tieDBItem.mtime = SysTime.fromISOExtString(lastModifiedTimestamp); } else { // invalid timestamp from JSON file addLogEntry("WARNING: Invalid timestamp provided by the Microsoft OneDrive API: " ~ lastModifiedTimestamp); // Set mtime to SysTime(0) tieDBItem.mtime = SysTime(0); } // Ensure there is no parentId for this DB record tieDBItem.parentId = null; // OneDrive Personal and Business supports relocating Shared Folders to other folders. // This means, in our DB, we need this DB record to have the correct parentId of the parental folder, if this is relocated shared folder // This is stored in the 'relocParentId' DB entry // This 'relocatedFolderParentId' variable is only ever set if using OneDrive Business account types and the shared folder is located online in another folder if ((!relocatedFolderDriveId.empty) && (!relocatedFolderParentId.empty)) { // Ensure that we set the relocParentId to the provided relocatedFolderParentId record if (debugLogging) {addLogEntry("Relocated Shared Folder references were provided - adding these to the 'root' DB Tie Record", ["debug"]);} tieDBItem.relocDriveId = relocatedFolderDriveId; tieDBItem.relocParentId = relocatedFolderParentId; } // Issue #3115 - Validate driveId length // What account type is this? if (appConfig.accountType == "personal") { // Issue #3336 - Convert driveId to lowercase before any test tieDBItem.driveId = transformToLowerCase(tieDBItem.driveId); // Test driveId length and validation if the driveId we are testing is not equal to appConfig.defaultDriveId if (tieDBItem.driveId != appConfig.defaultDriveId) { tieDBItem.driveId = testProvidedDriveIdForLengthIssue(tieDBItem.driveId); } } // Add this DB Tie parent record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a 'root' DB Tie record for a OneDrive Shared Folder online: " ~ to!string(tieDBItem), ["debug"]);} itemDB.upsert(tieDBItem); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Create a DB Tie Record for a Shared Folder void createDatabaseTieRecordForOnlineSharedFolder(Item parentItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Creating|Updating a DB Tie if (debugLogging) { //addLogEntry("Creating|Updating a DB Tie Record for this Shared Folder: " ~ parentItem.name, ["debug"]); addLogEntry("Creating|Updating a DB Tie Record for this Shared Folder from the provided parental data: " ~ parentItem.name, ["debug"]); addLogEntry("Parent Item Record: " ~ to!string(parentItem), ["debug"]); } // New DB Tie Item to bind the 'remote' path to our parent path in the database Item tieDBItem; tieDBItem.name = parentItem.name; tieDBItem.id = parentItem.remoteId; tieDBItem.type = ItemType.dir; tieDBItem.mtime = parentItem.mtime; // Initially set this tieDBItem.driveId = parentItem.remoteDriveId; // What account type is this as this determines what 'tieDBItem.parentId' should be set to // There is a difference in the JSON responses between 'personal' and 'business' account types for Shared Folders // Essentially an API inconsistency if (appConfig.accountType == "personal") { // Set tieDBItem.parentId to null tieDBItem.parentId = null; tieDBItem.type = ItemType.root; // Issue #3136, #3139 #3143 // Fetch the actual online record for this item // This returns the actual OneDrive Personal driveId value and is 15 character checked string actualOnlineDriveId = testProvidedDriveIdForLengthIssue(fetchRealOnlineDriveIdentifier(tieDBItem.driveId)); tieDBItem.driveId = actualOnlineDriveId; } else { // The tieDBItem.parentId needs to be the correct driveId id reference // Query the DB Item[] rootDriveItems; Item dbRecord; rootDriveItems = itemDB.selectByDriveId(parentItem.remoteDriveId); // Fix Issue #2883 if (rootDriveItems.length > 0) { // Use the first record returned dbRecord = rootDriveItems[0]; tieDBItem.parentId = dbRecord.id; } else { // Business Account ... but itemDB.selectByDriveId returned no entries ... need to query for this item online to get the correct details given they are not in the database if (debugLogging) {addLogEntry("itemDB.selectByDriveId(parentItem.remoteDriveId) returned zero database entries for this remoteDriveId: " ~ to!string(parentItem.remoteDriveId), ["debug"]);} // Create a new API Instance for this query and initialise it OneDriveApi getPathDetailsApiInstance; JSONValue latestOnlineDetails; getPathDetailsApiInstance = new OneDriveApi(appConfig); getPathDetailsApiInstance.initialise(); try { // Get the latest online details latestOnlineDetails = getPathDetailsApiInstance.getPathDetailsById(parentItem.remoteDriveId, parentItem.remoteId); if (debugLogging) {addLogEntry("Parent JSON details from Online Query: " ~ to!string(latestOnlineDetails), ["debug"]);} // Convert JSON to a database compatible item Item tempOnlineRecord = makeItem(latestOnlineDetails); // Configure tieDBItem.parentId to use tempOnlineRecord.id tieDBItem.parentId = tempOnlineRecord.id; // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getPathDetailsApiInstance.releaseCurlEngine(); getPathDetailsApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException e) { // Display error message displayOneDriveErrorMessage(e.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory getPathDetailsApiInstance.releaseCurlEngine(); getPathDetailsApiInstance = null; // Perform Garbage Collection GC.collect(); return; } } // Free the array memory rootDriveItems = []; } // Add tie DB record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a DB Tie record: " ~ to!string(tieDBItem), ["debug"]);} itemDB.upsert(tieDBItem); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // List all the OneDrive Business Shared Items for the user to see void listBusinessSharedObjects() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } JSONValue sharedWithMeItems; // Create a new API Instance for this thread and initialise it OneDriveApi sharedWithMeOneDriveApiInstance; sharedWithMeOneDriveApiInstance = new OneDriveApi(appConfig); sharedWithMeOneDriveApiInstance.initialise(); try { sharedWithMeItems = sharedWithMeOneDriveApiInstance.getSharedWithMe(); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory sharedWithMeOneDriveApiInstance.releaseCurlEngine(); sharedWithMeOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); } catch (OneDriveException e) { // Display error message displayOneDriveErrorMessage(e.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory sharedWithMeOneDriveApiInstance.releaseCurlEngine(); sharedWithMeOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); return; } if (sharedWithMeItems.type() == JSONType.object) { if (count(sharedWithMeItems["value"].array) > 0) { // No shared items addLogEntry(); addLogEntry("Listing available OneDrive Business Shared Items:"); addLogEntry(); // Iterate through the array foreach (searchResult; sharedWithMeItems["value"].array) { // loop variables for each item string sharedByName; string sharedByEmail; // Debug response output if (debugLogging) {addLogEntry("shared folder entry: " ~ to!string(searchResult), ["debug"]);} // Configure 'who' this was shared by if ("sharedBy" in searchResult["remoteItem"]["shared"]) { // we have shared by details we can use if ("displayName" in searchResult["remoteItem"]["shared"]["sharedBy"]["user"]) { sharedByName = searchResult["remoteItem"]["shared"]["sharedBy"]["user"]["displayName"].str; } if ("email" in searchResult["remoteItem"]["shared"]["sharedBy"]["user"]) { sharedByEmail = searchResult["remoteItem"]["shared"]["sharedBy"]["user"]["email"].str; } } // Output query result addLogEntry(debugLogBreakType1); if (isItemFile(searchResult)) { addLogEntry("Shared File: " ~ to!string(searchResult["name"].str)); } else { addLogEntry("Shared Folder: " ~ to!string(searchResult["name"].str)); } // Detail 'who' shared this if ((sharedByName != "") && (sharedByEmail != "")) { addLogEntry("Shared By: " ~ sharedByName ~ " (" ~ sharedByEmail ~ ")"); } else { if (sharedByName != "") { addLogEntry("Shared By: " ~ sharedByName); } } // More detail if --verbose is being used if (verboseLogging) { addLogEntry("Item Id: " ~ searchResult["remoteItem"]["id"].str, ["verbose"]); addLogEntry("Parent Drive Id: " ~ searchResult["remoteItem"]["parentReference"]["driveId"].str, ["verbose"]); if ("id" in searchResult["remoteItem"]["parentReference"]) { addLogEntry("Parent Item Id: " ~ searchResult["remoteItem"]["parentReference"]["id"].str, ["verbose"]); } } } // Close out the loop addLogEntry(debugLogBreakType1); addLogEntry(); } else { // No shared items addLogEntry(); addLogEntry("No OneDrive Business Shared Folders were returned"); addLogEntry(); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Query all the OneDrive Business Shared Objects to sync only Shared Files void queryBusinessSharedObjects() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } JSONValue sharedWithMeItems; Item sharedFilesRootDirectoryDatabaseRecord; // Create a new API Instance for this thread and initialise it OneDriveApi sharedWithMeOneDriveApiInstance; sharedWithMeOneDriveApiInstance = new OneDriveApi(appConfig); sharedWithMeOneDriveApiInstance.initialise(); try { sharedWithMeItems = sharedWithMeOneDriveApiInstance.getSharedWithMe(); // We cant shutdown the API instance here, as we reuse it below } catch (OneDriveException e) { // Display error message displayOneDriveErrorMessage(e.msg, thisFunctionName); // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory sharedWithMeOneDriveApiInstance.releaseCurlEngine(); sharedWithMeOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); return; } // Valid JSON response if (sharedWithMeItems.type() == JSONType.object) { // Get the configuredBusinessSharedFilesDirectoryName DB item // We need this as we need to 'fake' create all the folders for the shared files // Then fake create the file entries for the database with the correct parent folder that is the local folder itemDB.selectByPath(baseName(appConfig.configuredBusinessSharedFilesDirectoryName), appConfig.defaultDriveId, sharedFilesRootDirectoryDatabaseRecord); // For each item returned, if a file, process it foreach (searchResult; sharedWithMeItems["value"].array) { // Shared Business Folders are added to the account using 'Add shortcut to My files' // We only care here about any remaining 'files' that are shared with the user if (isItemFile(searchResult)) { // Debug response output if (debugLogging) {addLogEntry("getSharedWithMe Response Shared File JSON: " ~ sanitiseJSONItem(searchResult), ["debug"]);} // Make a DB item from this JSON Item sharedFileOriginalData = makeItem(searchResult); // Variables for each item string sharedByName; string sharedByEmail; string sharedByFolderName; string newLocalSharedFilePath; string newItemPath; Item sharedFilesPath; JSONValue fileToDownload; JSONValue detailsToUpdate; JSONValue latestOnlineDetails; // Configure 'who' this was shared by if ("sharedBy" in searchResult["remoteItem"]["shared"]) { // we have shared by details we can use if ("displayName" in searchResult["remoteItem"]["shared"]["sharedBy"]["user"]) { sharedByName = searchResult["remoteItem"]["shared"]["sharedBy"]["user"]["displayName"].str; } if ("email" in searchResult["remoteItem"]["shared"]["sharedBy"]["user"]) { sharedByEmail = searchResult["remoteItem"]["shared"]["sharedBy"]["user"]["email"].str; } } // Configure 'who' shared this, so that we can create the directory for that users shared files with us if ((sharedByName != "") && (sharedByEmail != "")) { sharedByFolderName = sharedByName ~ " (" ~ sharedByEmail ~ ")"; } else { if (debugLogging) {addLogEntry("Either name or email is not defined -> check specifically. Currently: " ~ to!string(sharedByName) ~ " / " ~ to!string(sharedByEmail), ["debug"]);} if (sharedByName != "") { sharedByFolderName = sharedByName; } else { sharedByFolderName = sharedByEmail; } } if (debugLogging) {addLogEntry("Combined folder set to " ~ to!string(sharedByFolderName), ["debug"]);} // Create the local path to store this users shared files with us newLocalSharedFilePath = buildNormalizedPath(buildPath(appConfig.configuredBusinessSharedFilesDirectoryName, sharedByFolderName)); if (debugLogging) {addLogEntry("newLocalSharedFilePath is located at " ~ to!string(newLocalSharedFilePath), ["debug"]);} // Does the Shared File Users Local Directory to store the shared file(s) exist? if (!exists(newLocalSharedFilePath)) { // Folder does not exist locally and needs to be created addLogEntry("Creating the OneDrive Business Shared File Users Local Directory: " ~ newLocalSharedFilePath); if (!dryRun) { // Local folder does not exist, thus needs to be created try { // Attempt path creation mkdirRecurse(newLocalSharedFilePath); } catch (std.file.FileException e) { // Creating the path failed addLogEntry("ERROR: Unable to create the OneDrive Business Shared File Users Local Directory: " ~ e.msg, ["info", "notify"]); } } // As this will not be created online, generate a response so it can be saved to the database sharedFilesPath = makeItem(createFakeResponse(baseName(newLocalSharedFilePath))); // Update sharedFilesPath parent items to that of sharedFilesRootDirectoryDatabaseRecord sharedFilesPath.parentId = sharedFilesRootDirectoryDatabaseRecord.id; // Add DB record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a DB record for storing OneDrive Business Shared Files: " ~ to!string(sharedFilesPath), ["debug"]);} itemDB.upsert(sharedFilesPath); } else { // Folder exists locally, is the folder in the database? // Query DB for this path Item dbRecord; if (!itemDB.selectByPath(baseName(newLocalSharedFilePath), appConfig.defaultDriveId, dbRecord)) { // As this will not be created online, generate a response so it can be saved to the database sharedFilesPath = makeItem(createFakeResponse(baseName(newLocalSharedFilePath))); // Update sharedFilesPath parent items to that of sharedFilesRootDirectoryDatabaseRecord sharedFilesPath.parentId = sharedFilesRootDirectoryDatabaseRecord.id; // Add DB record to the local database if (debugLogging) {addLogEntry("Creating|Updating into local database a DB record for storing OneDrive Business Shared Files: " ~ to!string(sharedFilesPath), ["debug"]);} itemDB.upsert(sharedFilesPath); } else { // If the folder exists in the db, assign the variable to have the parentID available sharedFilesPath = dbRecord; if (debugLogging) {addLogEntry("Recreating local database record for storing OneDrive Business Shared Files: " ~ to!string(sharedFilesPath), ["debug"]);} } } // The file to download JSON details fileToDownload = searchResult; // Get the latest online details latestOnlineDetails = sharedWithMeOneDriveApiInstance.getPathDetailsById(sharedFileOriginalData.remoteDriveId, sharedFileOriginalData.remoteId); Item tempOnlineRecord = makeItem(latestOnlineDetails); // With the local folders created, now update 'fileToDownload' to download the file to our location: // "parentReference": { // "driveId": "", // "driveType": "business", // "id": "", // }, // The getSharedWithMe() JSON response also contains an API bug where the 'hash' of the file is not provided // Use the 'latestOnlineDetails' response to obtain the hash // "file": { // "hashes": { // "quickXorHash": "" // } // }, // // The getSharedWithMe() JSON response also contains an API bug where the 'size' of the file is not the actual size of the file // The getSharedWithMe() JSON response also contains an API bug where the 'eTag' of the file is not present // The getSharedWithMe() JSON response also contains an API bug where the 'lastModifiedDateTime' of the file is date when the file was shared, not the actual date last modified detailsToUpdate = [ "parentReference": JSONValue([ "driveId": JSONValue(appConfig.defaultDriveId), "driveType": JSONValue("business"), "id": JSONValue(sharedFilesPath.id) ]), "file": JSONValue([ "hashes":JSONValue([ "quickXorHash": JSONValue(tempOnlineRecord.quickXorHash) ]) ]), "eTag": JSONValue(tempOnlineRecord.eTag) ]; foreach (string key, JSONValue value; detailsToUpdate.object) { fileToDownload[key] = value; } // Update specific items // Update 'size' fileToDownload["size"] = to!int(tempOnlineRecord.size); fileToDownload["remoteItem"]["size"] = to!int(tempOnlineRecord.size); // Update 'lastModifiedDateTime' fileToDownload["lastModifiedDateTime"] = latestOnlineDetails["fileSystemInfo"]["lastModifiedDateTime"].str; fileToDownload["fileSystemInfo"]["lastModifiedDateTime"] = latestOnlineDetails["fileSystemInfo"]["lastModifiedDateTime"].str; fileToDownload["remoteItem"]["lastModifiedDateTime"] = latestOnlineDetails["fileSystemInfo"]["lastModifiedDateTime"].str; fileToDownload["remoteItem"]["fileSystemInfo"]["lastModifiedDateTime"] = latestOnlineDetails["fileSystemInfo"]["lastModifiedDateTime"].str; // Final JSON that will be used to download the file if (debugLogging) {addLogEntry("Final fileToDownload: " ~ to!string(fileToDownload), ["debug"]);} // Make the new DB item from the consolidated JSON item Item downloadSharedFileDbItem = makeItem(fileToDownload); // Calculate the full local path for this shared file newItemPath = computeItemPath(downloadSharedFileDbItem.driveId, downloadSharedFileDbItem.parentId) ~ "/" ~ downloadSharedFileDbItem.name; // Does this potential file exists on disk? if (!exists(newItemPath)) { // The shared file does not exists locally // Is this something we actually want? Check the JSON against Client Side Filtering Rules bool unwanted = checkJSONAgainstClientSideFiltering(fileToDownload); if (!unwanted) { // File has not been excluded via Client Side Filtering // Submit this shared file to be processed further for downloading applyPotentiallyNewLocalItem(downloadSharedFileDbItem, fileToDownload, newItemPath); } } else { // A file, in the desired local location already exists with the same name // Is this local file in sync? string itemSource = "remote"; if (!isItemSynced(downloadSharedFileDbItem, newItemPath, itemSource)) { // Not in sync .... Item existingDatabaseItem; bool existingDBEntry = itemDB.selectById(downloadSharedFileDbItem.driveId, downloadSharedFileDbItem.id, existingDatabaseItem); // Is there a DB entry? if (existingDBEntry) { // Existing DB entry // Need to be consistent here with how 'newItemPath' was calculated string existingItemPath = computeItemPath(existingDatabaseItem.driveId, existingDatabaseItem.parentId) ~ "/" ~ existingDatabaseItem.name; // Attempt to apply this changed item applyPotentiallyChangedItem(existingDatabaseItem, existingItemPath, downloadSharedFileDbItem, newItemPath, fileToDownload); } else { // File exists locally, it is not in sync, there is no record in the DB of this file // In case the renamed path is needed string renamedPath; // If local data protection is configured (bypassDataPreservation = false), safeBackup the local file, passing in if we are performing a --dry-run or not safeBackup(newItemPath, dryRun, bypassDataPreservation, renamedPath); // Submit this shared file to be processed further for downloading applyPotentiallyNewLocalItem(downloadSharedFileDbItem, fileToDownload, newItemPath); } } else { // Item is in sync, ensure the DB record is the same itemDB.upsert(downloadSharedFileDbItem); } } } } } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory sharedWithMeOneDriveApiInstance.releaseCurlEngine(); sharedWithMeOneDriveApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Renaming or moving a directory online manually using --source-directory 'path/as/source/' --destination-directory 'path/as/destination' void moveOrRenameDirectoryOnline(string sourcePath, string destinationPath) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Function Variables bool sourcePathExists = false; bool destinationPathExists = false; bool invalidDestination = false; JSONValue sourcePathData; JSONValue destinationPathData; JSONValue parentPathData; Item sourceItem; Item parentItem; // Log that we are doing a move addLogEntry("Moving " ~ sourcePath ~ " to " ~ destinationPath); // Create a new API Instance for this thread and initialise it OneDriveApi onlineMoveApiInstance; onlineMoveApiInstance = new OneDriveApi(appConfig); onlineMoveApiInstance.initialise(); // In order to move, the 'source' needs to exist online, so this is the first check try { sourcePathData = onlineMoveApiInstance.getPathDetails(sourcePath); sourceItem = makeItem(sourcePathData); sourcePathExists = true; } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // The item to search was not found. If it does not exist, how can we move it? addLogEntry("The source path to move does not exist online - unable to move|rename a path that does not already exist online"); forceExit(); } else { // An error, regardless of what it is ... not good // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); forceExit(); } } // The second check needs to be that the destination does not already exist try { destinationPathData = onlineMoveApiInstance.getPathDetails(destinationPath); destinationPathExists = true; addLogEntry("The destination path to move to exists online - unable to move|rename to a path that already exists online"); forceExit(); } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // The item to search was not found. This is good as the destination path is empty } else { // An error, regardless of what it is ... not good // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); forceExit(); } } // Can we move? if ((sourcePathExists) && (!destinationPathExists)) { // Make an item we can use Item onlineItem = makeItem(sourcePathData); // The directory to move MUST be a directory if (onlineItem.type == ItemType.dir) { // Validate that the 'destination' is valid // This not a Client Side Filtering check, nor a Microsoft Check, but is a sanity check that the path provided is UTF encoded correctly // Check the std.encoding of the path against: Unicode 5.0, ASCII, ISO-8859-1, ISO-8859-2, WINDOWS-1250, WINDOWS-1251, WINDOWS-1252 if (!invalidDestination) { if(!isValid(destinationPath)) { // Path is not valid according to https://dlang.org/phobos/std_encoding.html addLogEntry("Skipping move - invalid character encoding sequence: " ~ destinationPath, ["info", "notify"]); invalidDestination = true; } } // We do not check this path against the Client Side Filtering Rules as this is 100% an online move only // Check this path against the Microsoft Naming Conventions & Restrictions // - Check path against Microsoft OneDrive restriction and limitations about Windows naming for files and folders // - Check path for bad whitespace items // - Check path for HTML ASCII Codes // - Check path for ASCII Control Codes if (!invalidDestination) { invalidDestination = checkPathAgainstMicrosoftNamingRestrictions(destinationPath, "move"); } // Is the destination location invalid? if (!invalidDestination) { // We can perform the online move // We need to query for the parent information of the destination path string parentPath = dirName(destinationPath); // Configure the parentItem by if this is the account 'root' use the root details, or query online for the parent details if (parentPath == ".") { // Parent path is '.' which is the account root - use client defaults parentItem.driveId = appConfig.defaultDriveId; // Should give something like 12345abcde1234a1 parentItem.id = appConfig.defaultRootId; // Should give something like 12345ABCDE1234A1!101 } else { // Need to query to obtain the details try { if (debugLogging) {addLogEntry("Attempting to query OneDrive Online for this parent path: " ~ parentPath, ["debug"]);} parentPathData = onlineMoveApiInstance.getPathDetails(parentPath); if (debugLogging) {addLogEntry("Online Parent Path Query Response: " ~ to!string(parentPathData), ["debug"]);} parentItem = makeItem(parentPathData); } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // The item to search was not found. If it does not exist, how can we move it? addLogEntry("The parent path to move to does not exist online - unable to move|rename a path to a parent that does exist online"); forceExit(); } else { // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); forceExit(); } } } // Configure the modification JSON item SysTime mtime; // Use the current system time mtime = Clock.currTime().toUTC(); JSONValue data = [ "name": JSONValue(baseName(destinationPath)), "parentReference": JSONValue([ "id": parentItem.id ]), "fileSystemInfo": JSONValue([ "lastModifiedDateTime": mtime.toISOExtString() ]) ]; // Try the online move try { onlineMoveApiInstance.updateById(sourceItem.driveId, sourceItem.id, data, sourceItem.eTag); // Log that it was successful addLogEntry("Successfully moved " ~ sourcePath ~ " to " ~ destinationPath); } catch (OneDriveException exception) { // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); forceExit(); } } } else { // The source item is not a directory addLogEntry("ERROR: The source path to move is not a directory"); forceExit(); } } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Return an array of the notification parameters when this is called. This implements FR #2760 string[] fileTransferNotifications() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Based on the configuration option, send the file transfer actions to the GUI notifications if configured // GUI notifications are already sent for files that meet this criteria: // - Skipping a particular item due to an invalid name // - Skipping a particular item due to an invalid symbolic link // - Skipping a particular item due to an invalid UTF sequence // - Skipping a particular item due to an invalid character encoding sequence // - Files that fail to upload // - Files that fail to download // // This is about notifying on: // - Successful file download // - Successful file upload // - Successful deletion locally // - Successful deletion online string[] loggingOptions; if (appConfig.getValueBool("notify_file_actions")) { // Add the 'notify' to enable GUI notifications loggingOptions = ["info", "notify"]; } else { // Logging to console and/or logfile only loggingOptions = ["info"]; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } return loggingOptions; } // OneDrive Personal driveId or parentReference driveId must be 16 characters in length string testProvidedDriveIdForLengthIssue(string objectParentDriveId) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Due to this function, we need to keep the return ; code, so that this function operates as efficiently as possible. // Whilst this means some extra code / duplication in this function, it cannot be helped // OneDrive Personal Account driveId and remoteDriveId length check // Issue #3072 (https://github.com/abraunegg/onedrive/issues/3072) illustrated that the OneDrive API is inconsistent in response when the Drive ID starts with a zero ('0') // - driveId // - remoteDriveId // // Example: // 024470056F5C3E43 (driveId) // 24470056f5c3e43 (remoteDriveId) // // If this is a OneDrive Personal Account, ensure this value is 16 characters, padded by leading zero's if eventually required string oldEntry; string newEntry; // Check the provided objectParentDriveId if (!objectParentDriveId.empty) { // Ensure objectParentDriveId is 16 characters long by padding with leading zeros if required if (debugLogging) { string validationMessage = format("Validating that the provided OneDrive Personal 'driveId' value '%s' is 16 characters", objectParentDriveId); addLogEntry(validationMessage, ["debug"]); } // Is this less than 16 characters if (objectParentDriveId.length < 16) { // Debug logging if (debugLogging) {addLogEntry("ONEDRIVE PERSONAL API BUG (Issue #3072): The provided 'driveId' is not 16 characters in length - fetching the correct value from Microsoft Graph API via getDriveIdRoot call", ["debug"]);} // Generate the change oldEntry = objectParentDriveId; string onlineDriveValue; // Fetch the actual online record for this item // This returns the actual OneDrive Personal driveId value based on the input value. // The function 'fetchRealOnlineDriveIdentifier' does not check for length issue, this is done below onlineDriveValue = fetchRealOnlineDriveIdentifier(oldEntry); // Check the onlineDriveValue value for 15 character issue if (!onlineDriveValue.empty) { // Ensure remoteDriveId is 16 characters long by padding with leading zeros if required if (onlineDriveValue.length < 16) { // online value is not 16 characters in length // Debug logging if (debugLogging) {addLogEntry("ONEDRIVE PERSONAL API BUG (Issue #3072): The provided online ['parentReference']['driveId'] value is not 16 Characters in length - padding with leading zero's", ["debug"]);} // Generate the change newEntry = to!string(onlineDriveValue.padLeft('0', 16)); // Explicitly use padLeft for leading zero padding, leave case as-is } else { // Online value is 16 characters in length, use as-is newEntry = onlineDriveValue; } } // Debug Logging of result if (debugLogging) { addLogEntry(" - old 'driveId' value = " ~ oldEntry, ["debug"]); addLogEntry(" - new 'driveId' value = " ~ newEntry, ["debug"]); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Issue #3336 - Convert driveId to lowercase // Return the new calculated value as lowercase return transformToLowerCase(newEntry); } else { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Issue #3336 - Convert driveId to lowercase // Return input value as-is as lowercase return transformToLowerCase(objectParentDriveId); } } else { // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Issue #3336 - Convert driveId to lowercase // Return input value as-is as lowercase return transformToLowerCase(objectParentDriveId); } } // Transform OneDrive Personal driveId or parentReference driveId to lowercase string transformToLowerCase(string objectParentDriveId) { // Since 14 June 2025 (possibly earlier), the Microsoft Graph API has started returning inconsistent casing for driveId values across multiple OneDrive Personal API endpoints. // https://github.com/OneDrive/onedrive-api-docs/issues/1902 // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } string transformedDriveIdValue; transformedDriveIdValue = toLower(objectParentDriveId); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return transformed value return transformedDriveIdValue; } // Calculate the transfer metrics for the file to aid in performance discussions when they are raised void displayTransferMetrics(string fileTransferred, long transferredBytes, SysTime transferStartTime, SysTime transferEndTime) { // We only calculate this if 'display_transfer_metrics' is enabled or we are doing debug logging if (appConfig.getValueBool("display_transfer_metrics") || debugLogging) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Calculations must be done on files > 0 transferredBytes if (transferredBytes > 0) { // Calculate transfer metrics auto transferDuration = transferEndTime - transferStartTime; double transferDurationAsSeconds = (transferDuration.total!"msecs"/1e3); // msec --> seconds double transferSpeedAsMbps = ((transferredBytes / transferDurationAsSeconds) / 1024 / 1024); // bytes --> Mbps // Output the transfer metrics string transferMetrics = format("File: %s | Size: %d Bytes | Duration: %.2f Seconds | Speed: %.2f Mbps (approx)", fileTransferred, transferredBytes, transferDurationAsSeconds, transferSpeedAsMbps); addLogEntry("Transfer Metrics - " ~ transferMetrics); } else { // Zero bytes - not applicable addLogEntry("Transfer Metrics - N/A (Zero Byte File)"); } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } } // Recursively validate JSONValue for UTF-8 compliance bool validateUTF8JSON(in JSONValue json) { switch (json.type) { case JSONType.string: return isValidUTF8(json.str); case JSONType.array: foreach (ref item; json.array) { if (!validateUTF8JSON(item)) return false; } break; case JSONType.object: foreach (key, ref value; json.object) { if (!isValidUTF8(key) || !validateUTF8JSON(value)) return false; } break; default: break; // Other types (null, bool, int, float) don't need UTF-8 validation } return true; } // Sanitise the provided onedriveJSONItem into a string that can actually be printed without error or issue string sanitiseJSONItem(JSONValue onedriveJSONItem) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } // Validate UTF-8 before serialisation if (!validateUTF8JSON(onedriveJSONItem)) { return "JSON Validation Failed: JSON data from OneDrive API contains invalid UTF-8 characters"; } // Redact PII in JSON before serialisation redactPII(onedriveJSONItem); // Eventual output variable string sanitisedJSONString; // Try and serialise the JSON into a string try { auto app = appender!string(); toJSON(app, onedriveJSONItem); sanitisedJSONString = app.data; } catch (Exception e) { sanitisedJSONString = "JSON Serialisation Failed: " ~ e.msg; } // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } // Return sanitised JSON string for logging output return sanitisedJSONString; } // Recursively redact PII and sensitive elements from JSONValue void redactPII(ref JSONValue j) { if (j.type == JSONType.object) { foreach (key, ref value; j.object) { // Match Graph's actual keys directly if (key == "email") { value = JSONValue(""); continue; } if (key == "displayName") { value = JSONValue(""); continue; } // Recurse redactPII(value); } } else if (j.type == JSONType.array) { foreach (ref value; j.array) { redactPII(value); } } } // Obtain the Websocket Notification URL void obtainWebSocketNotificationURL() { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } string websocketURL; // Create a new API Instance for this thread and initialise it OneDriveApi queryWebsocketURLApiInstance; queryWebsocketURLApiInstance = new OneDriveApi(appConfig); queryWebsocketURLApiInstance.initialise(); // Try and query Websocket Notification URL try { JSONValue endpointResponse = queryWebsocketURLApiInstance.obtainWebSocketNotificationURL(); // Was a valid JSON response provided? if (endpointResponse.type() == JSONType.object) { // Log response if (debugLogging) {addLogEntry("Response for a Socket.IO Subscription Endpoint: " ~ to!string(endpointResponse), ["debug"]);} // Store the JSON in the configuration for reuse appConfig.websocketEndpointResponse = to!string(endpointResponse); // Extract and store the Notification URL from the response we received (no transformation) websocketURL = endpointResponse["notificationUrl"].str; // Extract and store the expiry appConfig.websocketUrlExpiry = endpointResponse["expirationDateTime"].str; SysTime expiryUTC = SysTime.fromISOExtString(appConfig.websocketUrlExpiry); SysTime expiryLocal = expiryUTC.toLocalTime(); // Do we have a valid Notification URL ? if (!websocketURL.empty) { // Store the websocket notification URL appConfig.websocketNotificationUrl = websocketURL; // Set flag appConfig.websocketNotificationUrlAvailable = true; // Log WebSocket specifics if (debugLogging) { addLogEntry("WebSocket Notification URL: " ~ websocketURL, ["debug"]); addLogEntry("WebSocket Expiry (UTC): " ~ to!string(expiryUTC), ["debug"]); addLogEntry("WebSocket Expiry (Local): " ~ to!string(expiryLocal), ["debug"]); } } } } catch (OneDriveException exception) { // An error, regardless of what it is ... not good // Display what the error is // - 408,429,503,504 errors are handled as a retry within uploadFileOneDriveApiInstance displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryWebsocketURLApiInstance.releaseCurlEngine(); queryWebsocketURLApiInstance = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } // Download a single file via --download-file void downloadSingleFile(string pathToQuery) { // Function Start Time SysTime functionStartTime; string logKey; string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // Only set this if we are generating performance processing times if (appConfig.getValueBool("display_processing_time") && debugLogging) { functionStartTime = Clock.currTime(); logKey = generateAlphanumericString(); displayFunctionProcessingStart(thisFunctionName, logKey); } OneDriveApi queryPathDetailsOnline; JSONValue onlinePathData; // Was a path to query passed in? if (pathToQuery.empty) { // Nothing to query addLogEntry("No path to query"); return; } // Create new OneDrive API Instance queryPathDetailsOnline = new OneDriveApi(appConfig); queryPathDetailsOnline.initialise(); try { // Query the OneDrive API, using the path, which will query 'our' OneDrive Account onlinePathData = queryPathDetailsOnline.getPathDetails(pathToQuery); } catch (OneDriveException exception) { if (exception.httpStatusCode == 404) { // Path does not exist online ... addLogEntry("ERROR: The requested path does not exist online. Please check for your file online."); } else { // Display error message displayOneDriveErrorMessage(exception.msg, thisFunctionName); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryPathDetailsOnline.releaseCurlEngine(); queryPathDetailsOnline = null; // Perform Garbage Collection GC.collect(); // Return .. nothing to do return; } // Was a valid JSON response provided? if (onlinePathData.type() == JSONType.object) { // Valid JSON item was returned // Is the item a file ? if (isFileItem(onlinePathData)) { // JSON item is a file // Download the file based on the data returned downloadFileItem(onlinePathData); } else { // The provided path is not a file addLogEntry(); addLogEntry("ERROR: The requested path to download is not a file. Please correct this error and try again."); addLogEntry(); } } else { addLogEntry(); addLogEntry("ERROR: The requested file to download has generated an error. Please correct this error and try again."); addLogEntry(); } // OneDrive API Instance Cleanup - Shutdown API, free curl object and memory queryPathDetailsOnline.releaseCurlEngine(); queryPathDetailsOnline = null; // Perform Garbage Collection GC.collect(); // Display function processing time if configured to do so if (appConfig.getValueBool("display_processing_time") && debugLogging) { // Combine module name & running Function displayFunctionProcessingTime(thisFunctionName, functionStartTime, Clock.currTime(), logKey); } } } ================================================ FILE: src/util.d ================================================ // What is this module called? module util; // What does this module require to function? import core.memory; import core.stdc.errno : ENOENT, EINTR, EBUSY, EXDEV, EAGAIN, EPERM, EACCES, EROFS; import core.stdc.stdlib; import core.stdc.string; import core.sys.posix.pwd; import core.sys.posix.signal; import core.sys.posix.sys.resource; import core.sys.posix.sys.stat; import core.sys.posix.unistd; import core.thread; import etc.c.curl; import std.algorithm; import std.array; import std.ascii; import std.base64; import std.conv; import std.datetime; import std.digest.crc; import std.digest.sha; import std.exception; import std.file; import std.format; import std.json; import std.math; import std.net.curl; import std.path; import std.process; import std.random; import std.range; import std.regex; import std.socket; import std.stdio; import std.string; import std.traits; import std.uri; import std.utf; // What other modules that we have created do we need to import? import log; import config; import qxor; import curlEngine; // Global variable for the device name __gshared string deviceName; // Global flag for SIGINT (CTRL-C) and SIGTERM (kill) state __gshared bool exitHandlerTriggered = false; // Global variable for when we last uploaded something or made an online change from a local inotify event __gshared MonoTime lastLocalWrite; // util module variable ulong previousRSS; struct DesktopHints { bool gnome; bool kde; } shared static this() { deviceName = Socket.hostName; } // To assist with filesystem severity issues, configure an enum that can be used enum FsErrorSeverity { warning, error, fatal, permission } // Creates a safe backup of the given item, and only performs the function if not in a --dry-run scenario. // If the path already ends with "--safeBackup-####", the counter is incremented // instead of appending another "--safeBackup-". void safeBackup(const(char)[] path, bool dryRun, bool bypassDataPreservation, out string renamedPath) { // Ensure this is currently null renamedPath = null; bool isDirectory = false; // If the path doesn’t exist, there is nothing to back up if (!exists(path)) { if (debugLogging) { addLogEntry("safeBackup: Skipping backup as local path does not exist: " ~ to!string(path), ["debug"]); } return; } // Is the path a directory? try { isDirectory = isDir(path); } catch (FileException e) { // Path disappeared or became inaccessible between exists() and isDir() if (verboseLogging) { addLogEntry("Path to backup no longer exists or is inaccessible: " ~ to!string(path) ~ " : " ~ e.msg, ["verbose"]); } // Nothing left to back up — exit safely return; } // Is the input path a folder|directory? These should never be renamed if (isDirectory) { if (verboseLogging) { addLogEntry("Renaming request of local directory is being ignored: " ~ to!string(path), ["verbose"]); } return; } // Has the user configured to IGNORE local data protection rules? if (bypassDataPreservation) { addLogEntry("WARNING: Local Data Protection has been disabled - not renaming local file. You may experience data loss on this file: " ~ to!string(path), ["info", "notify"]); return; } // Convert once for convenience const string spath = to!string(path); const string ext = extension(spath); // Compute stem without extension (handles no-extension case too) const size_t stemLen = spath.length >= ext.length ? spath.length - ext.length : spath.length; string stem = spath[0 .. stemLen]; // Tag used for our safe backups string tag = "-" ~ deviceName ~ "-safeBackup-"; // Detect if already a tagged safeBackup on THIS device; if so, bump the 4-digit counter int startN = 1; string baseStem = stem; if (stem.length >= tag.length + 4) { // Slice out last 4 chars and the tag position auto last4 = stem[$ - 4 .. $]; auto tagSpan = stem[$ - (tag.length + 4) .. $ - 4]; bool fourDigits = true; foreach (c; last4) { if (!c.isDigit) { fourDigits = false; break; } } if (fourDigits && tagSpan == tag) { // Already a backup from this device — bump the counter startN = to!int(last4) + 1; baseStem = stem[0 .. $ - (tag.length + 4)]; } } // Find the first available name, capped at 1000 attempts int n = startN; string candidate; while (n <= 1000) { candidate = baseStem ~ tag ~ format("%04d", n) ~ ext; if (!exists(candidate)) break; ++n; } // If we exhausted our attempts, fail out if (n > 1000) { addLogEntry("Failed to backup " ~ spath ~ ": Unique file name could not be found after 1000 attempts", ["error"]); return; } // Log intent if (verboseLogging) { addLogEntry("The local item is out-of-sync with OneDrive, renaming to preserve existing file and prevent local data loss: " ~ spath ~ " -> " ~ candidate, ["verbose"]); } // Perform (or simulate) the rename if (!dryRun) { // Not a --dry-run scenario - attempt the file rename to create a safe backup // Use safeRename() if (safeRename(spath, candidate, dryRun)) { renamedPath = candidate; } else { // Failed to rename using safeRename() addLogEntry("Renaming of local file failed for " ~ spath ~ " -> " ~ candidate, ["error"]); } } else { if (debugLogging) { addLogEntry("DRY-RUN: Skipping renaming local file to preserve existing file and prevent data loss: " ~ spath ~ " -> " ~ candidate, ["debug"]); } } } // Rename the given item, and only performs the function if not in a --dry-run scenario bool safeRename(const(char)[] oldPath, const(char)[] newPath, bool dryRun) { string thisFunctionName = format("%s.%s", strip(__MODULE__), strip(getFunctionName!({}))); if (dryRun) { if (debugLogging) { addLogEntry("DRY-RUN: Skipping local file rename", ["debug"]); } return true; } int maxAttempts = 5; foreach (attempt; 0 .. maxAttempts) { try { if (debugLogging) { addLogEntry("Calling rename(oldPath, newPath)", ["debug"]); } // There are 2 options to rename a file // rename() - https://dlang.org/library/std/file/rename.html // std.file.copy() - https://dlang.org/library/std/file/copy.html // // rename: // It is not possible to rename a file across different mount points or drives. On POSIX, the operation is atomic. That means, if to already exists there will be no time period during the operation where to is missing. // // std.file.copy // Copy file from to file to. File timestamps are preserved. File attributes are preserved, if preserve equals Yes.preserveAttributes // // Use rename() as Linux is POSIX compliant, we have an atomic operation where at no point in time the 'to' is missing. rename(oldPath, newPath); return true; } catch (FileException e) { // Retry on EINTR if (e.errno == EINTR) { // Interrupted by signal → retry // 10ms backoff to avoid spinning if signals are frequent Thread.sleep(dur!"msecs"(10 * (attempt + 1))); continue; } // Retry on EBUSY if (e.errno == EBUSY) { // Filesystem was busy → retry // 25ms backoff to avoid spinning if signals are frequent Thread.sleep(dur!"msecs"(25 * (attempt + 1))); continue; } // Cross-device rename: not retryable if (e.errno == EXDEV) { displayFileSystemErrorMessage("Rename failed (cross-filesystem): " ~ e.msg, thisFunctionName, "oldPath=" ~ to!string(oldPath) ~ " newPath=" ~ to!string(newPath)); return false; } // Everything else: log once and return displayFileSystemErrorMessage(e.msg, thisFunctionName, "oldPath=" ~ to!string(oldPath) ~ " newPath=" ~ to!string(newPath)); return false; } } // If we get here, we exhausted retries // Log the last failure displayFileSystemErrorMessage("Failed to rename after retries: ", thisFunctionName, "oldPath=" ~ to!string(oldPath) ~ " newPath=" ~ to!string(newPath)); return false; } // Deletes the specified file without throwing an exception if there is an issue void safeRemove(const(char)[] path) { string thisFunctionName = format("%s.%s", strip(__MODULE__), strip(getFunctionName!({}))); int maxAttempts = 5; foreach (attempt; 0 .. maxAttempts) { try { // Attempt to remove; no pre-check to avoid TOCTTOU remove(path); return; } catch (FileException e) { if (e.errno == ENOENT) return; // already gone → fine // Retry on EINTR if (e.errno == EINTR) { // Interrupted by signal → retry // 10ms backoff to avoid spinning if signals are frequent Thread.sleep(dur!"msecs"(10 * (attempt + 1))); continue; } // Retry on EBUSY if (e.errno == EBUSY) { // Filesystem was busy → retry // 25ms backoff to avoid spinning if signals are frequent Thread.sleep(dur!"msecs"(25 * (attempt + 1))); continue; } // Anything else is noteworthy (EISDIR, EACCES, etc.) displayFileSystemErrorMessage(e.msg, thisFunctionName, to!string(path)); return; } } // If we get here, we exhausted retries // Log the last failure displayFileSystemErrorMessage("Failed to remove file after retries: " ~ to!string(path), thisFunctionName, to!string(path)); } // Returns the quickXorHash base64 string of a file, or an empty string on failure string computeQuickXorHash(string path) { QuickXor qxor; File file; bool fileOpened = false; scope(exit) { if (fileOpened) { file.close(); } } try { // Open file for reading file = File(path, "rb"); fileOpened = true; // Single stat call for BOTH size and preferred block size ulong fs = 0; size_t blockSize = 4096; // sensible default try { auto de = DirEntry(path); auto st = de.statBuf; // POSIX stat struct inferred if (st.st_size > 0) fs = cast(ulong) st.st_size; if (st.st_blksize > 0) blockSize = cast(size_t) st.st_blksize; } catch (Exception e) { // Best-effort only; keep defaults if stat fails addLogEntry("Unexpected error while stat'ing file for hash sizing: " ~ path ~ " - " ~ e.msg); } // Choose factor based on file size size_t factor; if (fs == 0) { factor = 256; // unknown size -> moderate buffer } else if (fs < 1_048_576UL) { // < 1 MiB factor = 16; // small buffer } else if (fs < 1_073_741_824UL) { // < 1 GiB factor = 256; // medium buffer } else { // >= 1 GiB factor = 512; // larger buffer } // Compute bufSize and clamp to [64 KiB, 8 MiB] size_t bufSize = blockSize * factor; if (bufSize < 64 * 1024) bufSize = 64 * 1024; if (bufSize > 8 * 1024 * 1024) bufSize = 8 * 1024 * 1024; // Allocate outside GC to avoid scanning big buffers auto raw = cast(ubyte*) malloc(bufSize); if (raw is null) { addLogEntry("Failed to compute QuickXor Hash for file: " ~ path ~ " - out of memory allocating buffer"); return ""; } scope(exit) free(raw); ubyte[] buf = raw[0 .. bufSize]; // Large sequential reads, minimal syscall overhead for (;;) { auto chunk = file.rawRead(buf); // returns slice of bytes read if (chunk.length == 0) break; // EOF qxor.put(chunk); } } catch (ErrnoException e) { addLogEntry("Failed to compute QuickXor Hash for file: " ~ path ~ " - " ~ e.msg); return ""; } catch (Exception e) { addLogEntry("Unexpected error while computing QuickXor Hash for file: " ~ path ~ " - " ~ e.msg); return ""; } auto hashResult = qxor.finish(); return Base64.encode(hashResult).idup; } // Returns the SHA256 hash hex string of a file, or an empty string on failure string computeSHA256Hash(string path) { SHA256 sha256; File file; bool fileOpened = false; scope(exit) { if (fileOpened) { file.close(); } } try { // Open file for reading file = File(path, "rb"); fileOpened = true; // Single stat call for BOTH size and preferred block size ulong fs = 0; size_t blockSize = 4096; // sensible default try { auto de = DirEntry(path); auto st = de.statBuf; // POSIX stat struct inferred if (st.st_size > 0) fs = cast(ulong) st.st_size; if (st.st_blksize > 0) blockSize = cast(size_t) st.st_blksize; } catch (Exception e) { // Best-effort only; keep defaults if stat fails addLogEntry("Unexpected error while stat'ing file for hash sizing: " ~ path ~ " - " ~ e.msg); } // Choose factor based on file size size_t factor; if (fs == 0) { factor = 256; // unknown size -> moderate buffer } else if (fs < 1_048_576UL) { // < 1 MiB factor = 16; // small buffer } else if (fs < 1_073_741_824UL) { // < 1 GiB factor = 256; // medium buffer } else { // >= 1 GiB factor = 512; // larger buffer } // Compute bufSize and clamp to [64 KiB, 8 MiB] size_t bufSize = blockSize * factor; if (bufSize < 64 * 1024) bufSize = 64 * 1024; if (bufSize > 8 * 1024 * 1024) bufSize = 8 * 1024 * 1024; // Allocate outside GC to avoid scanning big buffers auto raw = cast(ubyte*) malloc(bufSize); if (raw is null) { addLogEntry("Failed to compute SHA256 Hash for file: " ~ path ~ " - out of memory allocating buffer"); return ""; } scope(exit) free(raw); ubyte[] buf = raw[0 .. bufSize]; // Large sequential reads, minimal syscall overhead for (;;) { auto chunk = file.rawRead(buf); // returns slice of bytes read if (chunk.length == 0) break; // EOF sha256.put(chunk); } } catch (ErrnoException e) { addLogEntry("Failed to compute SHA256 Hash for file: " ~ path ~ " - " ~ e.msg); return ""; } catch (Exception e) { addLogEntry("Unexpected error while computing SHA256 Hash for file: " ~ path ~ " - " ~ e.msg); return ""; } auto hashResult = sha256.finish(); return toHexString(hashResult).idup; } // Converts wildcards (*, ?) to regex // The changes here need to be 100% regression tested before full release Regex!char wild2regex(const(char)[] pattern) { string str; str.reserve(pattern.length + 2); str ~= "^"; foreach (c; pattern) { switch (c) { case '*': str ~= ".*"; // Changed to match any character. Was: str ~= "[^/]*"; break; case '.': str ~= "\\."; break; case '?': str ~= "."; // Changed to match any single character. Was: str ~= "[^/]"; break; case '|': str ~= "$|^"; break; case '+': str ~= "\\+"; break; case ' ': str ~= "\\s"; // Changed to match exactly one whitespace. Was: str ~= "\\s+"; break; case '/': str ~= "\\/"; break; case '(': str ~= "\\("; break; case ')': str ~= "\\)"; break; default: str ~= c; break; } } str ~= "$"; return regex(str, "i"); } // Test Internet access to Microsoft OneDrive using a simple HTTP HEAD request bool testInternetReachability(ApplicationConfig appConfig, bool displayLogging = true) { HTTP http = HTTP(); http.url = "https://login.microsoftonline.com"; // Configure timeouts based on application configuration http.dnsTimeout = dur!"seconds"(appConfig.getValueLong("dns_timeout")); http.connectTimeout = dur!"seconds"(appConfig.getValueLong("connect_timeout")); http.dataTimeout = dur!"seconds"(appConfig.getValueLong("data_timeout")); http.operationTimeout = dur!"seconds"(appConfig.getValueLong("operation_timeout")); // Set IP protocol version http.handle.set(CurlOption.ipresolve, appConfig.getValueLong("ip_protocol_version")); // Explicitly set libcurl options to avoid using signal handlers in a multi-threaded environment // https://curl.se/libcurl/c/CURLOPT_NOSIGNAL.html http.handle.set(CurlOption.nosignal,1); // Explicitly set the use of TCP NAGLE // https://curl.se/libcurl/c/CURLOPT_TCP_NODELAY.html // Ensure that TCP_NODELAY is set to 0 to ensure that TCP NAGLE is enabled http.handle.set(CurlOption.tcp_nodelay,0); // Explicitly set to ensure libcurl keep the connection open for possible later reuse // https://curl.se/libcurl/c/CURLOPT_FORBID_REUSE.html http.handle.set(CurlOption.forbid_reuse,0); // Set HTTP method to HEAD for minimal data transfer http.method = HTTP.Method.head; bool reachedService = false; // Exit scope to ensure cleanup http object scope(exit) { // Shut http down http object http.shutdown(); } // Execute the request and handle exceptions try { if (displayLogging) { addLogEntry("Attempting to contact the Microsoft OneDrive Service"); } http.perform(); // Check response for HTTP status code - consider 2xx and 3xx as "reachable" if (http.statusLine.code >= 200 && http.statusLine.code < 400) { if (displayLogging) { addLogEntry("Successfully reached the Microsoft OneDrive Service"); } reachedService = true; } else { addLogEntry("Failed to reach the Microsoft OneDrive Service. HTTP status code: " ~ to!string(http.statusLine.code)); reachedService = false; } } catch (SocketException e) { addLogEntry("Cannot connect to the Microsoft OneDrive Service - Socket Issue: " ~ e.msg); displayOneDriveErrorMessage(e.msg, getFunctionName!({})); reachedService = false; } catch (CurlException e) { addLogEntry("Cannot connect to the Microsoft OneDrive Service - Network Connection Issue: " ~ e.msg); displayOneDriveErrorMessage(e.msg, getFunctionName!({})); reachedService = false; } catch (Exception e) { addLogEntry("An unexpected error occurred: " ~ e.toString()); displayOneDriveErrorMessage(e.toString(), getFunctionName!({})); reachedService = false; } // Return state return reachedService; } // Retry Internet access test to Microsoft OneDrive bool retryInternetConnectivityTest(ApplicationConfig appConfig) { int retryAttempts = 0; int backoffInterval = 1; // initial backoff interval in seconds int maxBackoffInterval = 3600; // maximum backoff interval in seconds int maxRetryCount = 100; // max retry attempts, reduced for practicality bool isOnline = false; while (retryAttempts < maxRetryCount && !isOnline) { if (backoffInterval < maxBackoffInterval) { backoffInterval = min(backoffInterval * 2, maxBackoffInterval); // exponential increase } if (debugLogging) { addLogEntry(" Retry Attempt: " ~ to!string(retryAttempts + 1), ["debug"]); addLogEntry(" Retry In (seconds): " ~ to!string(backoffInterval), ["debug"]); } Thread.sleep(dur!"seconds"(backoffInterval)); isOnline = testInternetReachability(appConfig); // assuming this function is defined elsewhere if (isOnline) { addLogEntry("Internet connectivity to Microsoft OneDrive service has been restored"); } retryAttempts++; } if (!isOnline) { addLogEntry("ERROR: Was unable to reconnect to the Microsoft OneDrive service after " ~ to!string(maxRetryCount) ~ " attempts!"); } // Return state return isOnline; } // Can we read the local file - as a permissions issue or file corruption will cause a failure // https://github.com/abraunegg/onedrive/issues/113 // returns true if file can be accessed bool readLocalFile(string path) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__) , strip(getFunctionName!({}))); // What is the file size if (getSize(path) != 0) { try { // Attempt to read up to the first 1 byte of the file auto data = read(path, 1); // Check if the read operation was successful if (data.length != 1) { // Read operation not successful addLogEntry("Failed to read the required amount from the file: " ~ path); return false; } } catch (std.file.FileException e) { // Unable to read the file, log the error message displayFileSystemErrorMessage(e.msg, thisFunctionName, path); return false; } return true; } else { // zero byte files cannot be read, return true return true; } } // Calls globMatch for each string in pattern separated by '|' bool multiGlobMatch(const(char)[] path, const(char)[] pattern) { if (path.length == 0 || pattern.length == 0) { return false; } if (!pattern.canFind('|')) { return globMatch!(std.path.CaseSensitive.yes)(path, pattern); } foreach (glob; pattern.split('|')) { if (globMatch!(std.path.CaseSensitive.yes)(path, glob)) { return true; } } return false; } // Check if the provided item name is a reserved Microsoft / Windows device name // This must catch both: // - exact reserved names, e.g. "CON" // - reserved names followed by an extension, e.g. "CON.txt", "NUL.tar.gz" // Microsoft documents that reserved names remain invalid even when followed by an extension. bool isReservedMicrosoftName(string itemName, const(bool[string]) disallowedSet) { // Ensure case-insensitive comparisons string candidate = itemName.toLower(); // Exact match if (disallowedSet.get(candidate, false)) { return true; } // Reserved device names followed by an extension, e.g. "CON.txt" auto firstDot = countUntil(candidate, "."); if (firstDot > 0) { string deviceRoot = candidate[0 .. firstDot]; if (disallowedSet.get(deviceRoot, false)) { return true; } } return false; } // Does the path pass the Microsoft restriction and limitations about naming files and folders bool isValidName(string path) { // Restriction and limitations about windows naming files and folders // https://msdn.microsoft.com/en-us/library/aa365247 // https://support.microsoft.com/en-us/help/3125202/restrictions-and-limitations-when-you-sync-files-and-folders if (path == ".") { return true; } string itemName = baseName(path).toLower(); // Ensure case-insensitivity // Check for explicitly disallowed names // https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-us&rs=en-us&ad=us#invalidfilefoldernames string[] disallowedNames = [ ".lock", "desktop.ini", "CON", "PRN", "AUX", "NUL", "COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9" ]; // Creating an associative array for faster lookup bool[string] disallowedSet; foreach (name; disallowedNames) { disallowedSet[name.toLower()] = true; // Normalise to lowercase } if (isReservedMicrosoftName(itemName, disallowedSet) || itemName.startsWith("~$") || canFind(itemName, "_vti_")) { return false; } // Regular expression for invalid patterns // https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-us&rs=en-us&ad=us#invalidcharacters // Leading whitespace and trailing whitespace // Invalid characters // Trailing dot '.' (not documented above) , however see issue https://github.com/abraunegg/onedrive/issues/2678 //auto invalidNameReg = ctRegex!(`^\s.*|^.*[\s\.]$|.*[<>:"\|\?*/\\].*`); - original to remove at some point auto invalidNameReg = ctRegex!(`^\s+|\s$|\.$|[<>:"\|\?*/\\]`); // revised 25/3/2024 // - ^\s+ matches one or more whitespace characters at the start of the string. The + ensures we match one or more whitespaces, making it more efficient than .* for detecting leading whitespaces. // - \s$ matches a whitespace character at the end of the string. This is more precise than [\s\.]$ because we'll handle the dot separately. // - \.$ specifically matches a dot character at the end of the string, addressing the requirement to catch trailing dots as invalid. // - [<>:"\|\?*/\\] matches any single instance of the specified invalid characters: ", *, :, <, >, ?, /, \, | auto matchResult = match(itemName, invalidNameReg); if (!matchResult.empty) { return false; } // Determine if the path is at the root level, if yes, check that 'forms' is not the first folder auto segments = pathSplitter(path).array; if (segments.length <= 2 && segments.back.toLower() == "forms") { // Check only the last segment, convert to lower as OneDrive is not POSIX compliant, easier to compare return false; } return true; } // Does the path contain any bad whitespace characters bool containsBadWhiteSpace(string path) { // Check for null or empty string if (path.length == 0) { return false; } // Check for root item if (path == ".") { return false; } // https://github.com/abraunegg/onedrive/issues/35 // Issue #35 presented an interesting issue where the filename contained a newline item // 'State-of-the-art, challenges, and open issues in the integration of Internet of'$'\n''Things and Cloud Computing.pdf' // When the check to see if this file was present the GET request queries as follows: // /v1.0/me/drive/root:/.%2FState-of-the-art%2C%20challenges%2C%20and%20open%20issues%20in%20the%20integration%20of%20Internet%20of%0AThings%20and%20Cloud%20Computing.pdf // The '$'\n'' is translated to %0A which causes the OneDrive query to fail // Check for the presence of '%0A' via regex string itemName = encodeComponent(baseName(path)); // Check for encoded newline character return itemName.indexOf("%0A") != -1; } // Does the path contain any ASCII HTML Codes bool containsASCIIHTMLCodes(string path) { // Check for null or empty string if (path.length == 0) { return false; } // Check for root item if (path == ".") { return false; } // https://github.com/abraunegg/onedrive/issues/151 // If a filename contains ASCII HTML codes, it generates an error when attempting to upload this to Microsoft OneDrive // Check if the filename contains an ASCII HTML code sequence // Check for the pattern &# followed by 1 to 4 digits and a semicolon auto invalidASCIICode = ctRegex!(`&#[0-9]{1,4};`); // Use match to search for ASCII HTML codes in the path auto matchResult = match(path, invalidASCIICode); // Return true if ASCII HTML codes are found return !matchResult.empty; } // Does the path contain any ASCII Control Codes bool containsASCIIControlCodes(string path) { // Check for null or empty string if (path.length == 0) { return false; } // Check for root item if (path == ".") { return false; } // https://github.com/abraunegg/onedrive/discussions/2553#discussioncomment-7995254 // Define a ctRegex pattern for ASCII control codes and specific non-ASCII control characters // This pattern includes the ASCII control range and common non-ASCII control characters // Adjust the pattern as needed to include specific characters of concern auto controlCodePattern = ctRegex!(`[\x00-\x1F\x7F]|\p{Cc}`); // Blocks ƒ†¯~‰ (#2553) , allows α (#2598) // Use match to search for ASCII control codes in the path auto matchResult = match(path, controlCodePattern); // Return true if matchResult is not empty (indicating a control code was found) return !matchResult.empty; } // Is the string a valid UTF-8 timestamp string? bool isValidUTF8Timestamp(string input) { try { // Validate the entire string for UTF-8 correctness validate(input); // Throws UTFException if invalid UTF-8 is found // Validate the input against UTF-8 test cases if (!isValidUTF8(input)) { // error message already printed return false; } // Additional edge-case handling because the input format is known and controlled: // Ensure input length is within the expected range for a UTC datetime if (input.length < 20 || input.length > 30) { // not the correct length addLogEntry("UTF-8 validation failed: Input '" ~ input ~ "' is not within the expected length range for UTC datetime strings (20-30 characters)."); return false; } return true; } catch (UTFException) { addLogEntry("UTF-8 validation failed: Input '" ~ input ~ "' contains invalid UTF-8 characters."); return false; } } // Is the string a valid UTF-8 string? bool isValidUTF8(string input) { try { // Validate the entire string for UTF-8 correctness validate(input); // Throws UTFException if invalid UTF-8 is found // Iterate through each character using byUTF to ensure proper UTF-8 decoding auto it = input.byUTF!(char); foreach (_; it) { // Iterating over the range ensures every UTF-8 sequence in the string is decoded into valid `dchar`s. // Throws a UTFException if an invalid UTF-8 sequence is encountered during decoding. } // Check for replacement characters if (input.count!((dchar c) => c == '\uFFFD') > 0) { // contains replacement character addLogEntry("UTF-8 validation failed: Input contains replacement characters (�)."); return false; } // return true return true; } catch (UTFException) { addLogEntry("UTF-8 validation failed: Input '" ~ input ~ "' contains invalid UTF-8 characters."); return false; } } // Is the path a valid UTF-16 encoded path? bool isValidUTF16(string path) { // Check for null or empty string if (path.length == 0) { return true; } // Check for root item if (path == ".") { return true; } auto wpath = toUTF16(path); // Convert to UTF-16 encoding auto it = wpath.byCodeUnit; while (!it.empty) { ushort current = it.front; // Check for valid single unit if (current <= 0xD7FF || (current >= 0xE000 && current <= 0xFFFF)) { it.popFront(); } // Check for valid surrogate pair else if (current >= 0xD800 && current <= 0xDBFF) { it.popFront(); if (it.empty || it.front < 0xDC00 || it.front > 0xDFFF) { return false; // Invalid surrogate pair } it.popFront(); } else { return false; // Invalid code unit } } return true; } // Validate that the provided string is a valid date time stamp in UTC format bool isValidUTCDateTime(string dateTimeString) { // Regular expression for validating the string against UTC datetime format // Allows for an optional fractional second part (e.g., .123 or .123456789) auto pattern = regex(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z$"); // Validate for UTF-8 first if (!isValidUTF8Timestamp(dateTimeString)) { if (dateTimeString.empty) { // empty string addLogEntry("BAD TIMESTAMP (UTF-8 FAIL): empty string"); } else { // log string that caused UTF-8 failure addLogEntry("BAD TIMESTAMP (UTF-8 FAIL): " ~ dateTimeString); } return false; } // First, check if the string matches the pattern if (!match(dateTimeString, pattern)) { addLogEntry("BAD TIMESTAMP (REGEX FAIL): " ~ dateTimeString); return false; } // Attempt to parse the string into a DateTime object try { auto dt = SysTime.fromISOExtString(dateTimeString); return true; } catch (TimeException) { addLogEntry("BAD TIMESTAMP (CONVERSION FAIL): " ~ dateTimeString); return false; } } // Does the path contain any HTML URL encoded items (e.g., '%20' for space) bool containsURLEncodedItems(string path) { // Check for null or empty string if (path.length == 0) { return false; } // Pattern for percent encoding: % followed by two hexadecimal digits auto urlEncodedPattern = ctRegex!(`%[0-9a-fA-F]{2}`); // Search for URL encoded items in the string auto matchResult = match(path, urlEncodedPattern); // Return true if URL encoded items are found return !matchResult.empty; } // Parse and display error message received from OneDrive void displayOneDriveErrorMessage(string message, string callingFunction) { addLogEntry(); addLogEntry("ERROR: Microsoft OneDrive API returned an error with the following message:"); auto errorArray = splitLines(message); addLogEntry(" Error Message: " ~ to!string(errorArray[0])); // Extract 'message' as the reason JSONValue errorMessage = parseJSON(replace(message, errorArray[0], "")); // What is the reason for the error if (errorMessage.type() == JSONType.object) { // configure the error reason string errorReason; string errorCode; string requestDate; string requestId; string localizedMessage; // set the reason for the error try { // Use error_description as reason errorReason = errorMessage["error_description"].str; } catch (JSONException e) { // we dont want to do anything here } // set the reason for the error try { // Use ["error"]["message"] as reason errorReason = errorMessage["error"]["message"].str; } catch (JSONException e) { // we dont want to do anything here } // Microsoft has started adding 'localizedMessage' to error JSON responses. If this is available, use this try { // Use ["error"]["localizedMessage"] as localised reason localizedMessage = errorMessage["error"]["localizedMessage"].str; } catch (JSONException e) { // we dont want to do anything here if not available } // Display the error reason if (errorReason.startsWith(" 0) { // First line: usually the most useful errorMessage = to!string(errorLines[0]); addLogEntry(" Error Message: " ~ errorMessage); // Remaining lines (if any) often contain errno / path / syscall details if (errorLines.length > 1) { addLogEntry(" Error Details:"); foreach (i, line; errorLines[1 .. $]) { // Avoid logging empty lines, but keep order if (!line.empty) { addLogEntry(" - " ~ to!string(line)); } } } } else { addLogEntry(" Error Message: No error message available"); } // Disk space diagnostics (best-effort) - if this is not a permission issue if (severity != FsErrorSeverity.permission) { // We intentionally probe both the current directory and the target path directory when possible. try { // Always check the current working directory as a baseline ulong freeCwd = to!ulong(getAvailableDiskSpace(".")); addLogEntry(" Disk Space (CWD): " ~ to!string(freeCwd) ~ " bytes available"); // If we have a context path, also check its parent directory when possible. // We keep this conservative: if anything throws, just log the exception. if (!contextPath.empty) { string targetProbePath = contextPath; // If it's a file path, probe the parent directory (where writes/renames happen). // Avoid throwing if parentDir isn't available or contextPath is weird. try { // std.path.dirName handles both file/dir paths; if it returns ".", keep as-is. import std.path : dirName; auto parent = dirName(contextPath); if (!parent.empty) targetProbePath = parent; } catch (Exception e) { addLogEntry(" NOTE: Failed to derive parent directory from path: " ~ e.msg); } ulong freeTarget = to!ulong(getAvailableDiskSpace(targetProbePath)); addLogEntry(" Disk Space (Path): " ~ to!string(freeTarget) ~ " bytes available (parent path: " ~ targetProbePath ~ ")"); // Preserve existing behaviour: if disk space check returns 0, force exit. // (Assumes getAvailableDiskSpace returns 0 on a hard failure in your implementation.) if (freeTarget == 0 || freeCwd == 0) { // Must force exit here, allow logging to be done forceExit(); } } else { // Preserve existing behaviour: if disk space check returns 0, force exit. if (freeCwd == 0) { forceExit(); } } } catch (Exception e) { // Handle exceptions from disk space check or type conversion addLogEntry(" NOTE: Exception during disk space check: " ~ e.msg); } } // Add note for WARNING messages if (severity == FsErrorSeverity.warning) { addLogEntry(); addLogEntry("NOTE: This warning is non-fatal; the client will continue to operate, but this may affect future operations if not resolved"); addLogEntry(); } // Add note for filesystem permission messages if (severity == FsErrorSeverity.permission) { addLogEntry(); addLogEntry("NOTE: Sync will continue. This file’s timestamps could not be updated because the effective user does not own the file."); addLogEntry(" Potential Fix:"); addLogEntry(" Run the client as the file owner, or change ownership of the sync tree so it is owned by the user running the client."); addLogEntry(" Learn more about File Ownership:"); addLogEntry(" https://www.redhat.com/en/blog/linux-file-permissions-explained"); addLogEntry(" https://unix.stackexchange.com/questions/191940/difference-between-owner-root-and-ruid-euid"); addLogEntry(); } // Add note for ERROR messages if (severity == FsErrorSeverity.error) { addLogEntry(); addLogEntry("NOTE: This error requires attention; the client may continue running, but functionality is impaired and the issue should be resolved."); addLogEntry(); } // Add note for FATAL messages if (severity == FsErrorSeverity.fatal) { addLogEntry(); addLogEntry("NOTE: This error is fatal; the client cannot continue and this issue must be corrected before retrying. The client will now attempt to exit in a safe and orderly manner."); addLogEntry(); } } // Display the POSIX Error Message void displayPosixErrorMessage(string message) { addLogEntry(); // used rather than writeln addLogEntry("ERROR: Microsoft OneDrive API returned data that highlights a POSIX compliance issue:"); addLogEntry(" Error Message: " ~ message); } // Display the Error Message void displayGeneralErrorMessage(Exception e, string callingFunction=__FUNCTION__, int lineno=__LINE__) { addLogEntry(); // used rather than writeln addLogEntry("ERROR: Encountered a " ~ e.classinfo.name ~ ":"); addLogEntry(" Error Message: " ~ e.msg); addLogEntry(" Calling Function: " ~ callingFunction); addLogEntry(" Line number: " ~ to!string(lineno)); } // Get the function name that is being called to assist with identifying where an error is being generated string getFunctionName(alias func)() { return __traits(identifier, __traits(parent, func)) ~ "()\n"; } JSONValue fetchOnlineURLContent(string url) { // Function variables char[] content; JSONValue onlineContent; // Setup HTTP request HTTP http = HTTP(); // Exit scope to ensure cleanup scope(exit) { // Shut http down and destroy http.shutdown(); object.destroy(http); // Perform Garbage Collection GC.collect(); // Return free memory to the OS GC.minimize(); } // Configure the URL to access http.url = url; // HTTP the connection method http.method = HTTP.Method.get; // Data receive handler http.onReceive = (ubyte[] data) { content ~= data; // Append data as it's received return data.length; }; // Perform HTTP request http.perform(); // Parse Content onlineContent = parseJSON(to!string(content)); // Return onlineResponse return onlineContent; } // Get the latest release version from GitHub JSONValue getLatestReleaseDetails() { JSONValue githubLatest; JSONValue versionDetails; string latestTag; string publishedDate; // Query GitHub for the 'latest' release details try { githubLatest = fetchOnlineURLContent("https://api.github.com/repos/abraunegg/onedrive/releases/latest"); } catch (CurlException e) { if (debugLogging) {addLogEntry("CurlException: Unable to query GitHub for latest release - " ~ e.msg, ["debug"]);} } catch (JSONException e) { if (debugLogging) {addLogEntry("JSONException: Unable to parse GitHub JSON response - " ~ e.msg, ["debug"]);} } // githubLatest has to be a valid JSON object if (githubLatest.type() == JSONType.object){ // use the returned tag_name if ("tag_name" in githubLatest) { // use the provided tag // "tag_name": "vA.B.CC" and strip 'v' latestTag = strip(githubLatest["tag_name"].str, "v"); } else { // set to latestTag zeros if (debugLogging) {addLogEntry("'tag_name' unavailable in JSON response. Setting GitHub 'tag_name' release version to 0.0.0", ["debug"]);} latestTag = "0.0.0"; } // use the returned published_at date if ("published_at" in githubLatest) { // use the provided value publishedDate = githubLatest["published_at"].str; } else { // set to v2.0.0 release date if (debugLogging) {addLogEntry("'published_at' unavailable in JSON response. Setting GitHub 'published_at' date to 2018-07-18T18:00:00Z", ["debug"]);} publishedDate = "2018-07-18T18:00:00Z"; } } else { // JSONValue is not an object if (debugLogging) {addLogEntry("Invalid JSON Object response from GitHub. Setting GitHub 'tag_name' release version to 0.0.0", ["debug"]);} latestTag = "0.0.0"; if (debugLogging) {addLogEntry("Invalid JSON Object. Setting GitHub 'published_at' date to 2018-07-18T18:00:00Z", ["debug"]);} publishedDate = "2018-07-18T18:00:00Z"; } // return the latest github version and published date as our own JSON versionDetails = [ "latestTag": JSONValue(latestTag), "publishedDate": JSONValue(publishedDate) ]; // return JSON return versionDetails; } // Get the release details from the 'current' running version JSONValue getCurrentVersionDetails(string thisVersion) { JSONValue githubDetails; JSONValue versionDetails; string versionTag = "v" ~ thisVersion; string publishedDate; // Query GitHub for the release details to match the running version try { githubDetails = fetchOnlineURLContent("https://api.github.com/repos/abraunegg/onedrive/releases"); } catch (CurlException e) { if (debugLogging) {addLogEntry("CurlException: Unable to query GitHub for release details - " ~ e.msg, ["debug"]);} return parseJSON(`{"Error": "CurlException", "message": "` ~ e.msg ~ `"}`); } catch (JSONException e) { if (debugLogging) {addLogEntry("JSONException: Unable to parse GitHub JSON response - " ~ e.msg, ["debug"]);} return parseJSON(`{"Error": "JSONException", "message": "` ~ e.msg ~ `"}`); } // githubDetails has to be a valid JSON array if (githubDetails.type() == JSONType.array){ foreach (searchResult; githubDetails.array) { // searchResult["tag_name"].str; if (searchResult["tag_name"].str == versionTag) { if (debugLogging) { addLogEntry("MATCHED version", ["debug"]); addLogEntry("tag_name: " ~ searchResult["tag_name"].str, ["debug"]); addLogEntry("published_at: " ~ searchResult["published_at"].str, ["debug"]); } publishedDate = searchResult["published_at"].str; } } if (publishedDate.empty) { // empty .. no version match ? // set to v2.0.0 release date if (debugLogging) {addLogEntry("'published_at' unavailable in JSON response. Setting GitHub 'published_at' date to 2018-07-18T18:00:00Z", ["debug"]);} publishedDate = "2018-07-18T18:00:00Z"; } } else { // JSONValue is not an Array if (debugLogging) {addLogEntry("Invalid JSON Array. Setting GitHub 'published_at' date to 2018-07-18T18:00:00Z", ["debug"]);} publishedDate = "2018-07-18T18:00:00Z"; } // return the latest github version and published date as our own JSON versionDetails = [ "versionTag": JSONValue(thisVersion), "publishedDate": JSONValue(publishedDate) ]; // return JSON return versionDetails; } // Check the application version versus GitHub latestTag void checkApplicationVersion() { // Get the latest details from GitHub JSONValue latestVersionDetails = getLatestReleaseDetails(); string latestVersion = latestVersionDetails["latestTag"].str; SysTime publishedDate = SysTime.fromISOExtString(latestVersionDetails["publishedDate"].str).toUTC(); SysTime releaseGracePeriod = publishedDate; SysTime currentTime = Clock.currTime().toUTC(); // drop fraction seconds publishedDate.fracSecs = Duration.zero; currentTime.fracSecs = Duration.zero; releaseGracePeriod.fracSecs = Duration.zero; // roll the grace period forward to allow distributions to catch up based on their release cycles releaseGracePeriod = releaseGracePeriod.add!"months"(1); // what is this clients version? auto currentVersionArray = strip(strip(import("version"), "v")).split("-"); string applicationVersion = currentVersionArray[0]; // debug output if (debugLogging) { addLogEntry("applicationVersion: " ~ applicationVersion, ["debug"]); addLogEntry("latestVersion: " ~ latestVersion, ["debug"]); addLogEntry("publishedDate: " ~ to!string(publishedDate), ["debug"]); addLogEntry("currentTime: " ~ to!string(currentTime), ["debug"]); addLogEntry("releaseGracePeriod: " ~ to!string(releaseGracePeriod), ["debug"]); } // display details if not current // is application version is older than available on GitHub if (applicationVersion != latestVersion) { // application version is different bool displayObsolete = false; // what warning do we present? if (applicationVersion < latestVersion) { // go get this running version details JSONValue thisVersionDetails = getCurrentVersionDetails(applicationVersion); SysTime thisVersionPublishedDate = SysTime.fromISOExtString(thisVersionDetails["publishedDate"].str).toUTC(); thisVersionPublishedDate.fracSecs = Duration.zero; if (debugLogging) {addLogEntry("thisVersionPublishedDate: " ~ to!string(thisVersionPublishedDate), ["debug"]);} // the running version grace period is its release date + 1 month SysTime thisVersionReleaseGracePeriod = thisVersionPublishedDate; thisVersionReleaseGracePeriod = thisVersionReleaseGracePeriod.add!"months"(1); if (debugLogging) {addLogEntry("thisVersionReleaseGracePeriod: " ~ to!string(thisVersionReleaseGracePeriod), ["debug"]);} // Is this running version obsolete ? if (!displayObsolete) { // if releaseGracePeriod > currentTime // display an information warning that there is a new release available if (releaseGracePeriod.toUnixTime() > currentTime.toUnixTime()) { // inside release grace period ... set flag to false displayObsolete = false; } else { // outside grace period displayObsolete = true; } } // display version response addLogEntry(); if (!displayObsolete) { // display the new version is available message addLogEntry("INFO: A new onedrive client version is available. Please upgrade your client version when possible.", ["info", "notify"]); } else { // display the obsolete message addLogEntry("WARNING: Your onedrive client version is now obsolete and unsupported. Please upgrade your client version.", ["info", "notify"]); } addLogEntry("Current Application Version: " ~ applicationVersion); addLogEntry("Version Available: " ~ latestVersion); addLogEntry(); } } } bool hasId(JSONValue item) { return ("id" in item) != null; } bool hasMimeType(const ref JSONValue item) { return ("mimeType" in item["file"]) != null; } bool hasQuota(JSONValue item) { return ("quota" in item) != null; } bool hasQuotaState(JSONValue item) { return ("state" in item["quota"]) != null; } bool isItemDeleted(JSONValue item) { return ("deleted" in item) != null; } bool isItemRoot(JSONValue item) { return ("root" in item) != null; } bool hasParentReference(const ref JSONValue item) { return ("parentReference" in item) != null; } bool hasParentReferenceDriveId(JSONValue item) { return ("driveId" in item["parentReference"]) != null; } bool hasParentReferenceId(JSONValue item) { return ("id" in item["parentReference"]) != null; } bool hasParentReferencePath(JSONValue item) { return ("path" in item["parentReference"]) != null; } bool isFolderItem(const ref JSONValue item) { return ("folder" in item) != null; } bool isRemoteFolderItem(const ref JSONValue item) { if (isItemRemote(item)) { return ("folder" in item["remoteItem"]) != null; } else { return false; } } bool isFileItem(const ref JSONValue item) { return ("file" in item) != null; } bool isItemRemote(const ref JSONValue item) { return ("remoteItem" in item) != null; } // Check if ["remoteItem"]["parentReference"]["driveId"] exists bool hasRemoteParentDriveId(const ref JSONValue item) { return ("remoteItem" in item) && ("parentReference" in item["remoteItem"]) && ("driveId" in item["remoteItem"]["parentReference"]); } // Check if ["remoteItem"]["id"] exists bool hasRemoteItemId(const ref JSONValue item) { return ("remoteItem" in item) && ("id" in item["remoteItem"]); } bool isItemFile(const ref JSONValue item) { return ("file" in item) != null; } bool isItemFolder(const ref JSONValue item) { return ("folder" in item) != null; } bool hasFileSize(const ref JSONValue item) { return ("size" in item) != null; } // Function to determine if the final component of the provided path is a .file or .folder bool isDotFile(const(string) path) { // Check for null or empty path if (path is null || path.length == 0) { return false; } // Special case for root if (path == ".") { return false; } // Extract the last component of the path auto paths = pathSplitter(buildNormalizedPath(path)); // Optimised way to fetch the last component string lastComponent = paths.empty ? "" : paths.back; // Check if the last component starts with a dot return startsWith(lastComponent, "."); } bool isMalware(const ref JSONValue item) { return ("malware" in item) != null; } bool isOneNotePackageFolder(const ref JSONValue item) { if ("package" in item) { auto pkg = item["package"]; if ("type" in pkg && pkg["type"].type == JSONType.string) { return pkg["type"].str == "oneNote"; } } return false; } bool hasHashes(const ref JSONValue item) { return ("hashes" in item["file"]) != null; } bool hasZeroHashes(const ref JSONValue item) { // Check if "hashes" exists under "file" and is empty if ("hashes" in item["file"]) { auto hashes = item["file"]["hashes"]; if (hashes.type == JSONType.object && hashes.object.keys.length == 0) { return true; } } return false; } bool hasQuickXorHash(const ref JSONValue item) { return ("quickXorHash" in item["file"]["hashes"]) != null; } bool hasSHA256Hash(const ref JSONValue item) { return ("sha256Hash" in item["file"]["hashes"]) != null; } bool isMicrosoftOneNoteMimeType1(const ref JSONValue item) { return (item["file"]["mimeType"].str) == "application/msonenote"; } bool isMicrosoftOneNoteMimeType2(const ref JSONValue item) { return (item["file"]["mimeType"].str) == "application/octet-stream"; } bool isMicrosoftOneNoteFileExtensionType1(const ref JSONValue item) { return item["name"].str.endsWith(".one"); } bool isMicrosoftOneNoteFileExtensionType2(const ref JSONValue item) { return item["name"].str.endsWith(".onetoc2"); } bool hasUploadURL(const ref JSONValue item) { return ("uploadUrl" in item) != null; } bool hasNextExpectedRanges(const ref JSONValue item) { return ("nextExpectedRanges" in item) != null; } bool hasLocalPath(const ref JSONValue item) { return ("localPath" in item) != null; } bool hasETag(const ref JSONValue item) { return ("eTag" in item) != null; } bool hasSharedElement(const ref JSONValue item) { return ("shared" in item) != null; } bool hasName(const ref JSONValue item) { return ("name" in item) != null; } bool hasCreatedBy(const ref JSONValue item) { return ("createdBy" in item) != null; } bool hasCreatedByUser(const ref JSONValue item) { return ("user" in item["createdBy"]) != null; } bool hasCreatedByUserDisplayName(const ref JSONValue item) { if (hasCreatedBy(item)) { if (hasCreatedByUser(item)) { return ("displayName" in item["createdBy"]["user"]) != null; } else { return false; } } else { return false; } } bool hasLastModifiedBy(const ref JSONValue item) { return ("lastModifiedBy" in item) != null; } bool hasLastModifiedByUser(const ref JSONValue item) { return ("user" in item["lastModifiedBy"]) != null; } bool hasLastModifiedByUserDisplayName(const ref JSONValue item) { if (hasLastModifiedBy(item)) { if (hasLastModifiedByUser(item)) { return ("displayName" in item["lastModifiedBy"]["user"]) != null; } else { return false; } } else { return false; } } // Check Intune JSON response for 'accessToken' bool hasAccessTokenData(const ref JSONValue item) { return ("accessToken" in item) != null; } // Check Intune JSON response for 'account' bool hasAccountData(const ref JSONValue item) { return ("account" in item) != null; } // Check Intune JSON response for 'expiresOn' bool hasExpiresOn(const ref JSONValue item) { return ("expiresOn" in item) != null; } // Resumable Download checks bool hasDriveId(const ref JSONValue item) { return ("driveId" in item) != null; } bool hasItemId(const ref JSONValue item) { return ("itemId" in item) != null; } bool hasDownloadFilename(const ref JSONValue item) { return ("downloadFilename" in item) != null; } bool hasResumeOffset(const ref JSONValue item) { return ("resumeOffset" in item) != null; } bool hasOnlineHash(const ref JSONValue item) { return ("onlineHash" in item) != null; } bool hasQuickXorHashResume(const ref JSONValue item) { return ("quickXorHash" in item["onlineHash"]) != null; } bool hasSHA256HashResume(const ref JSONValue item) { return ("sha256Hash" in item["onlineHash"]) != null; } // Test if a path is the equivalent of root '.' bool isRootEquivalent(string inputPath) { auto normalisedPath = buildNormalizedPath(inputPath); return normalisedPath == "." || normalisedPath == ""; } // Convert bytes to GB string byteToGibiByte(ulong bytes) { if (bytes == 0) { return "0.00"; // or handle the zero case as needed } double gib = bytes / 1073741824.0; // 1024^3 for direct conversion return format("%.2f", gib); // Format to ensure two decimal places } // Test if entrypoint.sh exists on the root filesystem bool entrypointExists(string basePath = "/") { try { // Build the path to the entrypoint.sh file string entrypointPath = buildNormalizedPath(buildPath(basePath, "entrypoint.sh")); // Check if the path exists and return the result return exists(entrypointPath); } catch (Exception e) { // Handle any exceptions (e.g., permission issues, invalid path) addLogEntry("An error occurred: " ~ e.msg); return false; } } // Generate a random alphanumeric string with specified length string generateAlphanumericString(size_t length = 16) { // Ensure length is not zero if (length == 0) { throw new Exception("Length must be greater than 0"); } auto asciiLetters = to!(dchar[])(letters); auto asciiDigits = to!(dchar[])(digits); dchar[] randomString; randomString.length = length; // Create a random number generator auto rndGen = Random(unpredictableSeed); // Fill the string with random alphanumeric characters fill(randomString[], randomCover(chain(asciiLetters, asciiDigits), rndGen)); return to!string(randomString); } // Display internal memory stats pre garbage collection void displayMemoryUsagePreGC() { // Display memory usage addLogEntry(); addLogEntry("Memory Usage PRE Garbage Collection (KB)"); addLogEntry("-----------------------------------------------------"); writeMemoryStats(); addLogEntry(); } // Display internal memory stats post garbage collection + RSS (actual memory being used) void displayMemoryUsagePostGC() { // Display memory usage title addLogEntry("Memory Usage POST Garbage Collection (KB)"); addLogEntry("-----------------------------------------------------"); writeMemoryStats(); // Assuming this function logs memory stats correctly // Query the actual Resident Set Size (RSS) for the PID pid_t pid = getCurrentPID(); ulong rss = getRSS(pid); // Check and log the previous RSS value if (previousRSS != 0) { addLogEntry("previous Resident Set Size (RSS) = " ~ to!string(previousRSS) ~ " KB"); // Calculate and log the difference in RSS long difference = rss - previousRSS; // 'difference' can be negative, use 'long' to handle it string sign = difference > 0 ? "+" : (difference < 0 ? "" : ""); // Determine the sign for display, no sign for zero addLogEntry("difference in Resident Set Size (RSS) = " ~ sign ~ to!string(difference) ~ " KB"); } // Update previous RSS with the new value previousRSS = rss; // Closeout addLogEntry(); } // Write internal memory stats void writeMemoryStats() { addLogEntry("current memory usedSize = " ~ to!string((GC.stats.usedSize/1024))); // number of used bytes on the GC heap (might only get updated after a collection) addLogEntry("current memory freeSize = " ~ to!string((GC.stats.freeSize/1024))); // number of free bytes on the GC heap (might only get updated after a collection) addLogEntry("current memory allocatedInCurrentThread = " ~ to!string((GC.stats.allocatedInCurrentThread/1024))); // number of bytes allocated for current thread since program start // Query the actual Resident Set Size (RSS) for the PID pid_t pid = getCurrentPID(); ulong rss = getRSS(pid); // The RSS includes all memory that is currently marked as occupied by the process. // Over time, the heap can become fragmented. Even after garbage collection, fragmented memory blocks may not be contiguous enough to be returned to the OS, leading to an increase in the reported memory usage despite having free space. // This includes memory that might not be actively used but has not been returned to the system. // The GC.minimize() function can sometimes cause an increase in RSS due to how memory pages are managed and freed. addLogEntry("current Resident Set Size (RSS) = " ~ to!string(rss) ~ " KB"); // actual memory in RAM used by the process at this point in time } // Return the username of the UID running the 'onedrive' process string getUserName() { // Retrieve the UID of the current user auto uid = getuid(); // Retrieve password file entry for the user auto pw = getpwuid(uid); // If user info is not found (e.g. no /etc/passwd entry), fallback to environment if (pw is null) { if (debugLogging) { addLogEntry("Unable to retrieve user info for UID: " ~ to!string(uid), ["debug"]); addLogEntry("Falling back to environment variable USER or returning 'unknown'", ["debug"]); } // Try environment variable string userEnv = environment.get("USER", "unknown"); return userEnv.length > 0 ? userEnv : "unknown"; } // If pw is valid, we can safely access pw.pw_name string userName = to!string(fromStringz(pw.pw_name)); // Log User identifiers from process if (debugLogging) { addLogEntry("Process ID: " ~ to!string(pw), ["debug"]); addLogEntry("User UID: " ~ to!string(pw.pw_uid), ["debug"]); addLogEntry("User GID: " ~ to!string(pw.pw_gid), ["debug"]); } // Check if username is valid if (!userName.empty) { if (debugLogging) {addLogEntry("User Name: " ~ userName, ["debug"]);} return userName; } else { // Log and return unknown user if (debugLogging) {addLogEntry("User Name: unknown", ["debug"]);} return "unknown"; } } // Get resource limit in POSIX portable manner (soft limit max open files) ulong getSoftOpenFilesLimit() { rlimit lim; if (getrlimit(RLIMIT_NOFILE, &lim) == 0) return cast(ulong) lim.rlim_cur; // soft limit return 0; } // Get resource limit in POSIX portable manner (hard limit max open files) ulong getHardOpenFilesLimit() { rlimit lim; if (getrlimit(RLIMIT_NOFILE, &lim) == 0) return cast(ulong) lim.rlim_max; // hard limit return 0; // or throw / handle error } // Calculate the ETA for when a 'large file' will be completed (upload & download operations) int calc_eta(size_t counter, size_t iterations, long start_time) { if (counter == 0) { return 0; // Avoid division by zero } // Get the current time as a Unix timestamp (seconds since the epoch, January 1, 1970, 00:00:00 UTC) SysTime currentTime = Clock.currTime(); long current_time = currentTime.toUnixTime(); // 'start_time' must be less than 'current_time' otherwise ETA will have negative values if (start_time > current_time) { if (debugLogging) { addLogEntry("Warning: start_time is in the future. Cannot calculate ETA.", ["debug"]); } return 0; } // Calculate duration long duration = (current_time - start_time); // Calculate the ratio we are at double ratio = cast(double) counter / iterations; // Calculate segments left to download auto segments_remaining = (iterations > counter) ? (iterations - counter) : 0; // Calculate the average time per iteration so far double avg_time_per_iteration = cast(double) duration / counter; // Debug output for the ETA calculation if (debugLogging) { addLogEntry("counter: " ~ to!string(counter), ["debug"]); addLogEntry("iterations: " ~ to!string(iterations), ["debug"]); addLogEntry("segments_remaining: " ~ to!string(segments_remaining), ["debug"]); addLogEntry("ratio: " ~ format("%.2f", ratio), ["debug"]); addLogEntry("start_time: " ~ to!string(start_time), ["debug"]); addLogEntry("current_time: " ~ to!string(current_time), ["debug"]); addLogEntry("duration: " ~ to!string(duration), ["debug"]); addLogEntry("avg_time_per_iteration: " ~ format("%.2f", avg_time_per_iteration), ["debug"]); } // Return the ETA or duration if (counter != iterations) { auto eta_sec = avg_time_per_iteration * segments_remaining; // ETA Debug if (debugLogging) { addLogEntry("eta_sec: " ~ to!string(eta_sec), ["debug"]); addLogEntry("estimated_total_time: " ~ to!string(avg_time_per_iteration * iterations), ["debug"]); } // Return ETA return eta_sec > 0 ? cast(int) ceil(eta_sec) : 0; } else { // Return the average time per iteration for the last iteration return cast(int) ceil(avg_time_per_iteration); } } // Use the ETA value and return a formatted string in a consistent manner string formatETA(int eta) { // How do we format the ETA string. Guard against zero and negative values if (eta <= 0) { return "| ETA --:--:--"; } int h, m, s; dur!"seconds"(eta).split!("hours", "minutes", "seconds")(h, m, s); return format!"| ETA %02d:%02d:%02d"(h, m, s); } // Force Exit due to failure void forceExit() { // Allow any logging complete before we force exit Thread.sleep(dur!("msecs")(500)); // Shutdown logging, which also flushes all logging buffers shutdownLogging(); // Setup signal handling for the exit scope setupExitScopeSignalHandler(); // Force Exit exit(EXIT_FAILURE); } // Get the current PID of the application pid_t getCurrentPID() { // The '/proc/self' is a symlink to the current process's proc directory string path = "/proc/self/stat"; // Read the content of the stat file string content; try { content = readText(path); } catch (Exception e) { writeln("Failed to read stat file: ", e.msg); return 0; } // The first value in the stat file is the PID auto parts = split(content); return to!pid_t(parts[0]); // Convert the first part to pid_t } // Access the Resident Set Size (RSS) based on the PID of the running application ulong getRSS(pid_t pid) { // Construct the path to the statm file for the given PID string path = format("/proc/%s/statm", to!string(pid)); // Read the content of the file string content; try { content = readText(path); } catch (Exception e) { writeln("Failed to read statm file: ", e.msg); return 0; } // Split the content and get the RSS (second value) auto stats = split(content); if (stats.length < 2) { writeln("Unexpected format in statm file."); return 0; } // RSS is in pages, convert it to kilobytes ulong rssPages = to!ulong(stats[1]); ulong rssKilobytes = rssPages * sysconf(_SC_PAGESIZE) / 1024; return rssKilobytes; } // Getting around the @nogc problem // https://p0nce.github.io/d-idioms/#Bypassing-@nogc auto assumeNoGC(T) (T t) if (isFunctionPointer!T || isDelegate!T) { enum attrs = functionAttributes!T | FunctionAttribute.nogc; return cast(SetFunctionAttributes!(T, functionLinkage!T, attrs)) t; } // When using exit scopes, set up this to catch any undesirable signal void setupExitScopeSignalHandler() { sigaction_t action; action.sa_handler = &exitScopeSignalHandler; // Direct function pointer assignment sigemptyset(&action.sa_mask); // Initialize the signal set to empty action.sa_flags = 0; sigaction(SIGSEGV, &action, null); // Invalid Memory Access signal } // Catch any SIGSEV generated by the exit scopes extern(C) nothrow @nogc @system void exitScopeSignalHandler(int signo) { if (signo == SIGSEGV) { assumeNoGC ( () { // Caught a SIGSEGV but everything was shutdown cleanly ..... //printf("Caught a SIGSEGV but everything was shutdown cleanly .....\n"); exit(0); })(); } } // Return the compiler details string compilerDetails() { version(DigitalMars) enum compiler = "DMD"; else version(LDC) enum compiler = "LDC"; else version(GNU) enum compiler = "GDC"; else enum compiler = "Unknown compiler"; string compilerString = compiler ~ " " ~ to!string(__VERSION__); return compilerString; } // Return the curl version details string getCurlVersionString() { // Get curl version auto versionInfo = curl_version(); return to!string(versionInfo); } // Function to return the decoded curl version as a string string getCurlVersionNumeric() { // Get curl version info using curl_version_info auto curlVersionDetails = curl_version_info(CURLVERSION_NOW); // Extract the major, minor, and patch numbers from version_num uint versionNum = curlVersionDetails.version_num; // The version number is in the format 0xXXYYZZ uint major = (versionNum >> 16) & 0xFF; // Extract XX (major version) uint minor = (versionNum >> 8) & 0xFF; // Extract YY (minor version) uint patch = versionNum & 0xFF; // Extract ZZ (patch version) // Return the version in the format "major.minor.patch" return major.to!string ~ "." ~ minor.to!string ~ "." ~ patch.to!string; } // Test the curl version against known curl versions with HTTP/2 issues bool isBadCurlVersion(string curlVersion) { // List of known curl versions with HTTP/2 issues string[] supportedVersions = [ "7.68.0", // Ubuntu 20.x "7.74.0", // Debian 11 "7.81.0", // Ubuntu 22.x "7.88.1", // Debian 12 "8.2.1", // Ubuntu 23.10 "8.5.0", // Ubuntu 24.04 "8.9.1", // Ubuntu 24.10 "8.10.0", // Various - HTTP/2 bug which was fixed in 8.10.1 "8.13.0", // Has a SSL Certificate read issue fixed by 8.14.1 "8.13.1", // Has a SSL Certificate read issue fixed by 8.14.1 "8.14.0", // Has a SSL Certificate read issue fixed by 8.14.1 ]; // Check if the current version matches one of the supported versions return canFind(supportedVersions, curlVersion); } // Is the operation a transient error? private bool isTransientErrno(int err) @safe nothrow { // EINTR: interrupted system call // EBUSY: resource busy (can be transient on some FS / mount scenarios) // EAGAIN: try again (transient) return err == EINTR || err == EBUSY || err == EAGAIN; } // Retry wrapper for getTimes() private bool safeGetTimes(string path, out SysTime accessTime, out SysTime modTime, string thisFunctionName) { int maxAttempts = 5; foreach (attempt; 0 .. maxAttempts) { try { getTimes(path, accessTime, modTime); return true; } catch (FileException e) { // If path vanished between checks / operations, treat as non-fatal for this workflow if (e.errno == ENOENT) { return false; } if (isTransientErrno(e.errno)) { // bounded backoff to avoid spinning Thread.sleep(dur!"msecs"(10 * (attempt + 1))); continue; } displayFileSystemErrorMessage(e.msg, thisFunctionName, path); return false; } } displayFileSystemErrorMessage("Failed to read file timestamps after retries", thisFunctionName, path); return false; } // Some errnos are 'expected' in the wild (permissions, RO mounts, immutable files) // What is this errno private bool isExpectedPermissionStyleErrno(int err) { // Return true of this is an expected error due to permission issues return err == EPERM || err == EACCES || err == EROFS; } // Helper function to determine path mismatch against UID|GID and process effective UID private bool getPathOwnerMismatch(string path, out uint fileUid, out uint effectiveUid) { version (Posix) { stat_t st; // Default outputs fileUid = 0; effectiveUid = cast(uint) geteuid(); try { // absolutePath can throw; keep this helper non-throwing auto fullPath = absolutePath(path); // Ensure we pass a NUL-terminated string to the C API auto cpath = toStringz(fullPath); if (lstat(cpath, &st) != 0) { if (debugLogging) { addLogEntry("getPathOwnerMismatch(): lstat() failed for '" ~ path ~ "'", ["debug"]); } return false; } fileUid = cast(uint) st.st_uid; // effectiveUid already set above return fileUid != effectiveUid; } catch (Exception e) { if (debugLogging) { addLogEntry("getPathOwnerMismatch(): exception for '" ~ path ~ "': " ~ e.msg, ["debug"]); } return false; } } else { fileUid = 0; effectiveUid = 0; return false; } } // Retry wrapper for setTimes() private bool safeSetTimes(string path, SysTime accessTime, SysTime modTime, string thisFunctionName) { enum int maxAttempts = 5; foreach (attempt; 0 .. maxAttempts) { try { setTimes(path, accessTime, modTime); return true; } catch (FileException e) { // If the path disappeared before we could set, there's nothing useful to do if (e.errno == ENOENT) { return false; } // Transient filesystem error: retry with backoff if (isTransientErrno(e.errno)) { if (debugLogging) { // Log that we hit a transient error when doing debugging, otherwise nothing addLogEntry("safeSetTimes() transient filesystem error response: " ~ e.msg ~ "\n - Attempting retry for setTimes()", ["debug"]); } // Backoff and retry Thread.sleep(dur!"msecs"(15 * (attempt + 1))); continue; } // Non-transient: special-case common permission errors // The user running the client needs to be the owner of the files if the client needs to set explicit timestamps // See https://github.com/abraunegg/onedrive/issues/3651 for details if (isExpectedPermissionStyleErrno(e.errno)) { // Configure application message to display string permissionErrorMessage = "Unable to set local file timestamps (mtime/atime): Operation not permitted"; if (e.errno == EPERM) { permissionErrorMessage = permissionErrorMessage ~ " (EPERM)"; } if (e.errno == EACCES) { permissionErrorMessage = permissionErrorMessage ~ " (EACCES)"; } if (e.errno == EROFS) { permissionErrorMessage = permissionErrorMessage ~ " (EROFS)"; } // Get extra details if required string extraHint; uint fileUid; uint effectiveUid; if (e.errno == EPERM && getPathOwnerMismatch(path, fileUid, effectiveUid)) { extraHint = "\nThe onedrive client user does not own this file. onedrive user effective UID=" ~ to!string(effectiveUid) ~ ", file owner UID=" ~ to!string(fileUid) ~ "." ~ "\nOn Unix-like systems, setting explicit file timestamps typically requires the process to be the file owner or run with sufficient privileges."; // Update permissionErrorMessage to add extraHint permissionErrorMessage = permissionErrorMessage ~ extraHint; } // If we are doing --verbose or --debug display this file system error if (verboseLogging) { // Display applicable message for the user regarding permission error on path displayFileSystemErrorMessage( permissionErrorMessage, thisFunctionName, path, FsErrorSeverity.permission ); } // It is pointless attempting a re-try in this scenario as those conditions will not change by retrying 15ms later. return false; } // Everything else: preserve existing behaviour displayFileSystemErrorMessage(e.msg, thisFunctionName, path); return false; } } // Only reached if transient errors never resolved displayFileSystemErrorMessage("Failed to set path timestamps after retries", thisFunctionName, path); return false; } // Set the timestamp of the provided path to ensure this is done in a consistent manner void setLocalPathTimestamp(bool dryRun, string inputPath, SysTime newTimeStamp) { // Set this function name string thisFunctionName = format("%s.%s", strip(__MODULE__), strip(getFunctionName!({}))); if (dryRun) { // Keep behaviour consistent: do nothing in dry-run return; } if (debugLogging) { string logMessage = format("Setting 'lastAccessTime' and 'lastModificationTime' properties for: %s to %s if required", inputPath, to!string(newTimeStamp)); addLogEntry(logMessage, ["debug"]); } // Read existing times (with retry protection) SysTime existingAccessTime; SysTime existingModificationTime; if (!safeGetTimes(inputPath, existingAccessTime, existingModificationTime, thisFunctionName)) { // safeGetTimes already logged non-transient errors; ENOENT etc just returns false quietly return; } if (debugLogging) { addLogEntry("Existing timestamp values:", ["debug"]); addLogEntry(" Access Time: " ~ to!string(existingAccessTime), ["debug"]); addLogEntry(" Modification Time: " ~ to!string(existingModificationTime), ["debug"]); } // Compare timestamps using UTC and truncated fractional seconds (OneDrive has no fractional seconds) SysTime newTimeStampZeroFracSec = newTimeStamp.toUTC(); SysTime existingTimeStampZeroFracSec = existingModificationTime.toUTC(); newTimeStampZeroFracSec.fracSecs = Duration.zero; existingTimeStampZeroFracSec.fracSecs = Duration.zero; if (debugLogging) { addLogEntry("Comparison timestamp values:", ["debug"]); addLogEntry(" newTimeStampZeroFracSec = " ~ to!string(newTimeStampZeroFracSec), ["debug"]); addLogEntry(" existingTimeStampZeroFracSec = " ~ to!string(existingTimeStampZeroFracSec), ["debug"]); } // Only update if the whole-second timestamp differs bool makeTimestampChange = (newTimeStampZeroFracSec != existingTimeStampZeroFracSec); SysTime updatedModificationTime; if (!makeTimestampChange) { if (debugLogging) { addLogEntry("Fractional seconds only difference in modification time; preserving existing modification time", ["debug"]); addLogEntry("No local timestamp change required", ["debug"]); } return; } if (debugLogging) { addLogEntry("New timestamp is different to existing timestamp; using new modification time", ["debug"]); addLogEntry("Calling setTimes() for the given path", ["debug"]); } updatedModificationTime = newTimeStamp; // Apply new timestamp if (!safeSetTimes(inputPath, existingAccessTime, updatedModificationTime, thisFunctionName)) { // safeSetTimes logs non-transient errors; ENOENT just returns false quietly return; } if (debugLogging) { addLogEntry("Timestamp updated for this path: " ~ inputPath, ["debug"]); } // Post-check to ensure timestamp is set SysTime newAccessTime; SysTime newModificationTime; if (safeGetTimes(inputPath, newAccessTime, newModificationTime, thisFunctionName) && debugLogging) { addLogEntry("Current timestamp values post any change (if required):", ["debug"]); addLogEntry(" Access Time: " ~ to!string(newAccessTime), ["debug"]); addLogEntry(" Modification Time: " ~ to!string(newModificationTime), ["debug"]); } } // Generate the initial function processing time log entry void displayFunctionProcessingStart(string functionName, string logKey) { // Output the function processing header addLogEntry(format("[%s] Application Function '%s' Started", strip(logKey), strip(functionName))); } // Calculate the time taken to perform the application Function void displayFunctionProcessingTime(string functionName, SysTime functionStartTime, SysTime functionEndTime, string logKey) { // Calculate processing time auto functionDuration = functionEndTime - functionStartTime; double functionDurationAsSeconds = (functionDuration.total!"msecs"/1e3); // msec --> seconds // Output the function processing time string processingTime = format("[%s] Application Function '%s' Processing Time = %.4f Seconds", strip(logKey), strip(functionName), functionDurationAsSeconds); addLogEntry(processingTime); } // Return true if `dir` exists and has no entries. // Symlinks are treated as non-removable. bool isDirEmpty(string dir) { if (!exists(dir) || !isDir(dir) || isSymlink(dir)) return false; foreach (_; dirEntries(dir, SpanMode.shallow)) { // Found at least one entry return false; } return true; } // Escape a string for literal use inside a regex string regexEscape(string s) { auto b = appender!string(); foreach (c; s) { // characters with special meaning in regex immutable specials = "\\.^$|?*+()[]{}"; if (specials.canFind(c)) b.put('\\'); b.put(c); } return b.data; } // Update lastLocalWrite to denote we just performed a local-originated write void markLocalWrite() { lastLocalWrite = MonoTime.currTime(); } ================================================ FILE: src/webhook.d ================================================ // What is this module called? module webhook; // What does this module require to function? import core.atomic : atomicOp; import std.datetime; import std.concurrency; import std.json; // What other modules that we have created do we need to import? import arsd.cgi; import config; import onedrive; import log; import util; class OneDriveWebhook { private RequestServer server; private string host; private ushort port; private Tid parentTid; private bool started; private ApplicationConfig appConfig; private OneDriveApi oneDriveApiInstance; string subscriptionId = ""; SysTime subscriptionExpiration, subscriptionLastErrorAt; Duration subscriptionExpirationInterval, subscriptionRenewalInterval, subscriptionRetryInterval; string notificationUrl = ""; private uint count; this(Tid parentTid, ApplicationConfig appConfig) { this.host = appConfig.getValueString("webhook_listening_host"); this.port = to!ushort(appConfig.getValueLong("webhook_listening_port")); this.parentTid = parentTid; this.appConfig = appConfig; subscriptionExpiration = Clock.currTime(UTC()); subscriptionLastErrorAt = SysTime.fromUnixTime(0); subscriptionExpirationInterval = dur!"seconds"(appConfig.getValueLong("webhook_expiration_interval")); subscriptionRenewalInterval = dur!"seconds"(appConfig.getValueLong("webhook_renewal_interval")); subscriptionRetryInterval = dur!"seconds"(appConfig.getValueLong("webhook_retry_interval")); notificationUrl = appConfig.getValueString("webhook_public_url"); } // The static serve() is necessary because spawn() does not like instance methods void serve() { if (this.started) { return; } this.started = true; this.count = 0; server.listeningHost = this.host; server.listeningPort = this.port; spawn(&serveImpl, cast(shared) this); addLogEntry("Started OneDrive API Webhook server"); // Subscriptions oneDriveApiInstance = new OneDriveApi(this.appConfig); oneDriveApiInstance.initialise(); createOrRenewSubscription(); } void stop() { if (!this.started) return; server.stop(); this.started = false; addLogEntry("Stopped OneDrive API Webhook server"); object.destroy(server); // Delete subscription if there exists any try { deleteSubscription(); } catch (OneDriveException e) { logSubscriptionError(e); } // Release API instance back to the pool oneDriveApiInstance.releaseCurlEngine(); object.destroy(oneDriveApiInstance); oneDriveApiInstance = null; } private static void handle(shared OneDriveWebhook _this, Cgi cgi) { if (debugHTTPSResponse) { addLogEntry("Webhook request: " ~ to!string(cgi.requestMethod) ~ " " ~ to!string(cgi.requestUri)); if (!cgi.postBody.empty) { addLogEntry("Webhook post body: " ~ to!string(cgi.postBody)); } } cgi.setResponseContentType("text/plain"); if ("validationToken" in cgi.get) { // For validation requests, respond with the validation token passed in the query string // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/concepts/webhook-receiver-validation-request cgi.write(cgi.get["validationToken"]); addLogEntry("OneDrive API Webhook: handled validation request"); } else { // Notifications don't include any information about the changes that triggered them. // Put a refresh signal in the queue and let the main monitor loop process it. // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/concepts/using-webhooks _this.count.atomicOp!"+="(1); send(cast()_this.parentTid, to!ulong(_this.count)); cgi.write("OK"); addLogEntry("OneDrive API Webhook: sent refresh signal #" ~ to!string(_this.count)); } } private static void serveImpl(shared OneDriveWebhook _this) { _this.server.serveEmbeddedHttp!(handle, OneDriveWebhook)(_this); } // Create a new subscription or renew the existing subscription void createOrRenewSubscription() { auto elapsed = Clock.currTime(UTC()) - subscriptionLastErrorAt; if (elapsed < subscriptionRetryInterval) { return; } try { if (!hasValidSubscription()) { createSubscription(); } else if (isSubscriptionUpForRenewal()) { renewSubscription(); } } catch (OneDriveException e) { logSubscriptionError(e); subscriptionLastErrorAt = Clock.currTime(UTC()); addLogEntry("Will retry creating or renewing subscription in " ~ to!string(subscriptionRetryInterval)); } catch (JSONException e) { addLogEntry("ERROR: Unexpected JSON error when attempting to validate subscription: " ~ e.msg); subscriptionLastErrorAt = Clock.currTime(UTC()); addLogEntry("Will retry creating or renewing subscription in " ~ to!string(subscriptionRetryInterval)); } } // Return the duration to next subscriptionExpiration check Duration getNextExpirationCheckDuration() { SysTime now = Clock.currTime(UTC()); if (hasValidSubscription()) { Duration elapsed = Clock.currTime(UTC()) - subscriptionLastErrorAt; // Check if we are waiting for the next retry if (elapsed < subscriptionRetryInterval) return subscriptionRetryInterval - elapsed; else return subscriptionExpiration - now - subscriptionRenewalInterval; } else return subscriptionRetryInterval; } private bool hasValidSubscription() { return !subscriptionId.empty && subscriptionExpiration > Clock.currTime(UTC()); } private bool isSubscriptionUpForRenewal() { return subscriptionExpiration < Clock.currTime(UTC()) + subscriptionRenewalInterval; } private void createSubscription() { addLogEntry("Initialising webhook subscription for updates ..."); auto expirationDateTime = Clock.currTime(UTC()) + subscriptionExpirationInterval; try { JSONValue response = oneDriveApiInstance.createSubscription(notificationUrl, expirationDateTime); // Save important subscription metadata including id and expiration subscriptionId = response["id"].str; subscriptionExpiration = SysTime.fromISOExtString(response["expirationDateTime"].str); addLogEntry("Created new subscription " ~ subscriptionId ~ " with expiration: " ~ to!string(subscriptionExpiration.toISOExtString())); } catch (OneDriveException e) { if (e.httpStatusCode == 409) { // Take over an existing subscription on HTTP 409. // // Sample 409 error: // { // "error": { // "code": "ObjectIdentifierInUse", // "innerError": { // "client-request-id": "615af209-467a-4ab7-8eff-27c1d1efbc2d", // "date": "2023-09-26T09:27:45", // "request-id": "615af209-467a-4ab7-8eff-27c1d1efbc2d" // }, // "message": "Subscription Id c0bba80e-57a3-43a7-bac2-e6f525a76e7c already exists for the requested combination" // } // } // Make sure the error code is "ObjectIdentifierInUse" try { if (e.error["error"]["code"].str != "ObjectIdentifierInUse") { throw e; } } catch (JSONException jsonEx) { throw e; } // Extract the existing subscription id from the error message import std.regex; auto idReg = ctRegex!(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "i"); auto m = matchFirst(e.error["error"]["message"].str, idReg); if (!m) { throw e; } // Save the subscription id and renew it immediately since we don't know the expiration timestamp subscriptionId = m[0]; addLogEntry("Found existing webhook subscription " ~ subscriptionId); renewSubscription(); } else { throw e; } } } private void renewSubscription() { addLogEntry("Renewing webhook subscription for updates ..."); auto expirationDateTime = Clock.currTime(UTC()) + subscriptionExpirationInterval; try { JSONValue response = oneDriveApiInstance.renewSubscription(subscriptionId, expirationDateTime); // Update subscription expiration from the response subscriptionExpiration = SysTime.fromISOExtString(response["expirationDateTime"].str); addLogEntry("Renewed webhook subscription " ~ subscriptionId ~ " with expiration: " ~ to!string(subscriptionExpiration.toISOExtString())); } catch (OneDriveException e) { if (e.httpStatusCode == 404) { addLogEntry("The subscription is not found on the server. Recreating subscription ..."); subscriptionId = null; subscriptionExpiration = Clock.currTime(UTC()); createSubscription(); } else { throw e; } } } private void deleteSubscription() { if (!hasValidSubscription()) { return; } oneDriveApiInstance.deleteSubscription(subscriptionId); addLogEntry("Deleted subscription"); } private void logSubscriptionError(OneDriveException e) { string errorMsg; try { // Attempt to extract the specific error message from the JSON if possible if (e.error.type == JSONType.object && "error" in e.error && e.error["error"].type == JSONType.object && "message" in e.error["error"]) { errorMsg = e.error["error"]["message"].str; } else { throw new Exception("Invalid error structure"); } } catch (Exception ex) { // Fallback to the message stored in the exception if the JSON is malformed or not structured as expected errorMsg = e.msg; } // Log a message to the GUI only addLogEntry("ERROR: An issue has occurred with webhook subscriptions: " ~ errorMsg, ["notify"]); // Use the standard OneDrive API logging method displayOneDriveErrorMessage(errorMsg, getFunctionName!({})); } } ================================================ FILE: src/xattr.d ================================================ module xattr; import core.sys.posix.sys.types; import core.stdc.errno; import core.stdc.stdlib; import core.stdc.string; import core.stdc.stdio; import std.string; import std.conv; version (linux) { extern (C) { int setxattr(const(char)* path, const(char)* name, const(void)* value, size_t size, int flags); ssize_t getxattr(const(char)* path, const(char)* name, void* value, size_t size); } } version (FreeBSD) { extern (C) { int extattr_set_file(const(char)* path, int attrnamespace, const(char)* name, const(void)* value, size_t size); ssize_t extattr_get_file(const(char)* path, int attrnamespace, const(char)* name, void* value, size_t size); } enum EXTATTR_NAMESPACE_USER = 1; } class XAttrException : Exception { this(string message) { super(message); } } // Sets an extended attribute for a given file. // Throws `XAttrException` on failure. void setXAttr(string filePath, string attrName, string attrValue) { version (linux) { int result = setxattr(filePath.toStringz(), attrName.toStringz(), cast(const(void)*)attrValue.ptr, attrValue.length, 0); if (result != 0) { throw new XAttrException("Failed to set xattr '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } } else version (FreeBSD) { int result = extattr_set_file(filePath.toStringz(), EXTATTR_NAMESPACE_USER, attrName.toStringz(), cast(const(void)*)attrValue.ptr, attrValue.length); if (result < 0) { throw new XAttrException("Failed to set xattr '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } } else { throw new XAttrException("xattr not supported on this platform"); } } // Retrieves an extended attribute value from a file. // Returns the attribute value as a string or throws `XAttrException` on failure. string getXAttr(string filePath, string attrName) { version (linux) { // First, determine the size of the attribute value ssize_t size = getxattr(filePath.toStringz(), attrName.toStringz(), null, 0); if (size < 0) { throw new XAttrException("Failed to get xattr size for '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } void* buffer = malloc(size); scope(exit) free(buffer); ssize_t ret = getxattr(filePath.toStringz(), attrName.toStringz(), buffer, cast(size_t)size); if (ret < 0) { throw new XAttrException("Failed to get xattr '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } return cast(string)(buffer[0 .. size]); } else version (FreeBSD) { // First, determine the size ssize_t size = extattr_get_file(filePath.toStringz(), EXTATTR_NAMESPACE_USER, attrName.toStringz(), null, 0); if (size < 0) { throw new XAttrException("Failed to get xattr size for '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } void* buffer = malloc(size); scope(exit) free(buffer); ssize_t ret = extattr_get_file(filePath.toStringz(), EXTATTR_NAMESPACE_USER, attrName.toStringz(), buffer, cast(size_t)size); if (ret < 0) { throw new XAttrException("Failed to get xattr '" ~ attrName ~ "' on '" ~ filePath ~ "': " ~ to!string(strerror(errno))); } return cast(string)(buffer[0 .. size]); } else { throw new XAttrException("xattr not supported on this platform"); } } ================================================ FILE: tests/makefiles.sh ================================================ #!/bin/bash ONEDRIVEALT=~/OneDriveALT if [ ! -d ${ONEDRIVEALT} ]; then mkdir -p ${ONEDRIVEALT} else rm -rf ${ONEDRIVEALT}/* fi BADFILES=${ONEDRIVEALT}/bad_files TESTFILES=${ONEDRIVEALT}/test_files mkdir -p ${BADFILES} mkdir -p ${TESTFILES} dd if=/dev/urandom of=${TESTFILES}/large_file1.txt count=15 bs=1572864 dd if=/dev/urandom of=${TESTFILES}/large_file2.txt count=20 bs=1572864 # Create bad files that should be skipped touch "${BADFILES}/ leading_white_space" touch "${BADFILES}/trailing_white_space " touch "${BADFILES}/trailing_dot." touch "${BADFILES}/includes < in the filename" touch "${BADFILES}/includes > in the filename" touch "${BADFILES}/includes : in the filename" touch "${BADFILES}/includes \" in the filename" touch "${BADFILES}/includes | in the filename" touch "${BADFILES}/includes ? in the filename" touch "${BADFILES}/includes * in the filename" touch "${BADFILES}/includes \\ in the filename" touch "${BADFILES}/includes \\\\ in the filename" touch "${BADFILES}/CON" touch "${BADFILES}/CON.text" touch "${BADFILES}/PRN" touch "${BADFILES}/AUX" touch "${BADFILES}/NUL" touch "${BADFILES}/COM0" touch "${BADFILES}/COM1" touch "${BADFILES}/COM2" touch "${BADFILES}/COM3" touch "${BADFILES}/COM4" touch "${BADFILES}/COM5" touch "${BADFILES}/COM6" touch "${BADFILES}/COM7" touch "${BADFILES}/COM8" touch "${BADFILES}/COM9" touch "${BADFILES}/LPT0" touch "${BADFILES}/LPT1" touch "${BADFILES}/LPT2" touch "${BADFILES}/LPT3" touch "${BADFILES}/LPT4" touch "${BADFILES}/LPT5" touch "${BADFILES}/LPT6" touch "${BADFILES}/LPT7" touch "${BADFILES}/LPT8" touch "${BADFILES}/LPT9" # Test files from cases # File contains invalid whitespace characters tar xf ./bad-file-name.tar.xz -C ${BADFILES}/ # HelloCOM2.rar should be allowed dd if=/dev/urandom of=${TESTFILES}/HelloCOM2.rar count=5 bs=1572864