Repository: digitalec/deemon Branch: main Commit: 6ab04eddbe04 Files: 68 Total size: 324.3 KB Directory structure: gitextract_dh1lrv7v/ ├── .github/ │ └── workflows/ │ ├── deploy-docker.yml │ ├── discord-release-msg.yml │ ├── purge-old-betas.yml │ └── python-publish.yml ├── .gitignore ├── Dockerfile ├── LICENSE ├── MANIFEST.in ├── README.md ├── deemon/ │ ├── __init__.py │ ├── __main__.py │ ├── assets/ │ │ └── index.html │ ├── cli.py │ ├── cmd/ │ │ ├── __init__.py │ │ ├── artistconfig.py │ │ ├── backup.py │ │ ├── download.py │ │ ├── extra.py │ │ ├── generate.py │ │ ├── monitor.py │ │ ├── profile.py │ │ ├── refresh.py │ │ ├── rollback.py │ │ ├── search.py │ │ ├── show.py │ │ └── upgradelib.py │ ├── core/ │ │ ├── __init__.py │ │ ├── api.py │ │ ├── common.py │ │ ├── config.py │ │ ├── db.py │ │ ├── dmi.py │ │ ├── exceptions.py │ │ ├── logger.py │ │ └── notifier.py │ └── utils/ │ ├── __init__.py │ ├── dataprocessor.py │ ├── dates.py │ ├── performance.py │ ├── startup.py │ ├── ui.py │ └── validate.py ├── docs/ │ ├── _config.yml │ ├── _sass/ │ │ └── custom/ │ │ └── custom.scss │ ├── docs/ │ │ ├── automations/ │ │ │ ├── automations.md │ │ │ ├── cron.md │ │ │ └── task-scheduler.md │ │ ├── commands/ │ │ │ ├── backup.md │ │ │ ├── commands.md │ │ │ ├── config.md │ │ │ ├── download.md │ │ │ ├── library.md │ │ │ ├── monitor.md │ │ │ ├── profile.md │ │ │ ├── refresh.md │ │ │ ├── reset.md │ │ │ ├── rollback.md │ │ │ ├── search.md │ │ │ ├── show.md │ │ │ └── test.md │ │ ├── configuration.md │ │ ├── installation.md │ │ └── troubleshooting/ │ │ ├── logs.md │ │ ├── queue.md │ │ └── troubleshooting.md │ └── index.md ├── requirements.txt └── setup.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/workflows/deploy-docker.yml ================================================ name: Deploy Docker on: release: types: [released] workflow_dispatch: jobs: docker: if: "!github.event.release.prerelease" runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Login to GHCR if: github.event_name != 'pull_request' uses: docker/login-action@v1 with: registry: ghcr.io username: ${{ github.repository_owner }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build and push uses: docker/build-push-action@v2 with: context: . platforms: linux/amd64, linux/arm64, linux/arm/v7 push: true tags: ghcr.io/digitalec/deemon:latest labels: ${{ steps.meta.outputs.labels }} ================================================ FILE: .github/workflows/discord-release-msg.yml ================================================ name: Release messages to discord announcement channel on: release: types: - created workflow_dispatch: jobs: run_main: runs-on: ubuntu-22.04 name: Sends custom message steps: - name: Sending message uses: digitalec/discord-styled-releases@main with: webhook_id: ${{ secrets.DISCORD_WEBHOOK_ID }} webhook_token: ${{ secrets.DISCORD_WEBHOOK_TOKEN }} ================================================ FILE: .github/workflows/purge-old-betas.yml ================================================ name: Delete old beta releases on: workflow_dispatch: jobs: deploy: runs-on: ubuntu-latest steps: - uses: dev-drprasad/delete-older-releases@v0.2.0 with: repo: digitalec/deemon # defaults to current repo keep_latest: 0 delete_tag_pattern: b # defaults to "" env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ================================================ FILE: .github/workflows/python-publish.yml ================================================ # This workflow will upload a Python Package using Twine when a release is created # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries name: Upload Python Package on: release: types: [created] workflow_dispatch: jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up Python uses: actions/setup-python@v2 with: python-version: '3.x' - name: Install dependencies run: | python -m pip install --upgrade pip pip install setuptools wheel twine packaging - name: Build and publish env: TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} run: | python setup.py sdist bdist_wheel twine upload dist/* ================================================ FILE: .gitignore ================================================ __pycache__/ .idea/ build deemon.egg-info dist test.txt venv .cache tests.py **/.DS_Store artists.txt /deemon/tests/ .vscode/settings.json .vscode/launch.json createenv.sh .python-version ================================================ FILE: Dockerfile ================================================ FROM ubuntu:22.04 RUN apt-get update -y && \ apt-get install -y python3-pip COPY ./requirements.txt /requirements.txt WORKDIR / RUN pip3 install -r requirements.txt && \ mkdir /config && mkdir /deemix && mkdir /downloads && mkdir /import && \ mkdir /root/.config && \ ln -s /config /root/.config/deemon && \ ln -s /deemix /root/.config/deemix COPY deemon /app/deemon ENV PYTHONPATH="$PYTHONPATH:/app" VOLUME /config /downloads /import /deemix ================================================ FILE: LICENSE ================================================ GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . ================================================ FILE: MANIFEST.in ================================================ recursive-include deemon/assets * include README.md include LICENSE include requirements.txt ================================================ FILE: README.md ================================================ deemon [About](#about) | [Installation](#installation) | [Docker](#docker) | [Documentation](https://digitalec.github.io/deemon) | [Support](#support) ![PyPI](https://img.shields.io/pypi/v/deemon?style=for-the-badge) ![Downloads](https://img.shields.io/pepy/dt/deemon?style=for-the-badge) ![GitHub last release](https://img.shields.io/github/release-date/digitalec/deemon?style=for-the-badge) ![GitHub last commit](https://img.shields.io/github/last-commit/digitalec/deemon?style=for-the-badge) ![Docker](https://img.shields.io/github/actions/workflow/status/digitalec/deemon/deploy-docker.yml?branch=main&style=for-the-badge&logo=docker) ![Discord](https://img.shields.io/discord/831356172464160838?style=for-the-badge&logo=discord) ### About deemon is a command line tool written in Python that monitors artists for new releases, provides email notifications and can also integrate with the deemix library to automatically download new releases. ### Support [Open an Issue](https://github.com/digitalec/deemon/issues/new) | [Discord](https://discord.gg/KzNCG2tkvn) ### Installation #### Using pip ```bash $ pip install deemon ``` #### From source ```bash $ pip install -r requirements.txt $ python3 -m deemon ``` #### Docker Docker support has been added for `amd64`, `arm64` and `armv7` architectures. It is recommended to save your `docker run` command as a script to execute via cron/Task Scheduler. **Note:** Inside deemon's `config.json`, download_location **must** be set to `/downloads` until I can integrate this myself. **Example: Refreshing an existing database** ``` docker run --name deemon \ --rm \ -v /path/to/deemon/config:/config \ -v /path/to/music:/downloads \ -v /path/to/deemix/config:/deemix \ ghcr.io/digitalec/deemon:latest \ python3 -m deemon refresh ``` #### Unraid Install Python/PIP using either Nerd-tools Plugin (Unraid 6), Python 3 for UNRAID Plugin (Unraid 6 or 7), or manually via command line. See the installation instructions [here](https://digitalec.github.io/deemon/docs/installation.html) or install as root (**NOT** recommended!): ```bash pip install deemon ``` Then: ```bash deemon --init ``` If deemon is not found in your path, you can also call it as a python module: ```bash python3 -m deemon --init ``` If installed using the **root** account, the config.json will be located at: **/root/.config/deemon/config.json**. Edit your configuration using the documentation located [here](https://digitalec.github.io/deemon/docs/configuration.html). Use `deemon monitor -h` for help on adding artists, playlists, or albums to monitor for new releases. #### Installation in a Python Virtual Environment (venv) If you wish to install deemon and it's dependencies in a sandbox-style environment, I would recommend using venv. Create a venv and install deemon (you may need to use `python3` and `pip3` depending on your system): ```commandline $ python -m venv venv $ source ./venv/bin/activate $ pip install deemon ``` When you are finished, close the terminal or exit our venv: ```commandline $ deactivate ``` Next time you want to run deemon, activate the venv first: ```commandline $ source ./venv/bin/activate $ deemon refresh ``` If you are moving to venv from the Docker container, be sure to update your cron/Task Scheduler scripts. ### Getting Started You have to manually add artists, playlists, albums, etc.. deemon does not automatically pull artists unless they're being monitored. Refer to the documentation [here](https://digitalec.github.io/deemon/docs/commands/monitor.html). ================================================ FILE: deemon/__init__.py ================================================ #!/usr/bin/env python3 from deemon.utils import startup __version__ = '2.22' __dbversion__ = '3.7' appdata = startup.get_appdata_dir() startup.init_appdata_dir(appdata) ================================================ FILE: deemon/__main__.py ================================================ from deemon import cli def main(): cli.run() if __name__ == "__main__": main() ================================================ FILE: deemon/assets/index.html ================================================ deemon
deemon
{UPDATE_MESSAGE}
{NEW_RELEASE_COUNT} new release(s) were found!
{NEW_RELEASE_LIST}
Open an issue on GitHub or join us on Discord
================================================ FILE: deemon/cli.py ================================================ import logging import platform import sys import time from pathlib import Path import click from packaging.version import parse as parse_version from deemon import __version__ from deemon.cmd import download, rollback, backup, extra, tests, upgradelib from deemon.cmd.artistconfig import artist_lookup from deemon.cmd.monitor import Monitor from deemon.cmd.profile import ProfileConfig from deemon.cmd.refresh import Refresh from deemon.cmd.search import Search from deemon.cmd.show import Show from deemon.core import notifier from deemon.core.config import Config, LoadProfile from deemon.core.db import Database from deemon.core.logger import setup_logger from deemon.utils import startup, dataprocessor, validate logger = None config = None db = None CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help']) @click.group(context_settings=CONTEXT_SETTINGS, invoke_without_command=True, no_args_is_help=True) @click.option('--whats-new', is_flag=True, help="Show release notes from this version") @click.option('--init', is_flag=True, help="""Initialize deemon application data directory. Warning: if directory exists, this will delete existing config and database.""") @click.option('--arl', help="Update ARL") @click.option('-P', '--profile', help="Specify profile to run deemon as") @click.version_option(__version__, '-V', '--version', message='deemon %(version)s') @click.option('-v', '--verbose', is_flag=True, help="Show debug output") def run(whats_new, init, arl, verbose, profile): """Monitoring and alerting tool for new music releases using the Deezer API. deemon is a free and open source tool. To report issues or to contribute, please visit https://github.com/digitalec/deemon """ global logger global config global db setup_logger(log_level='DEBUG' if verbose else 'INFO', log_file=startup.get_log_file()) logger = logging.getLogger(__name__) logger.debug(f"deemon {__version__}") logger.debug(f"command: \"{' '.join([x for x in sys.argv[1:]])}\"") logger.debug("Python " + platform.python_version()) logger.debug(platform.platform()) logger.debug(f"deemon appdata is located at {startup.get_appdata_dir()}") if whats_new: return startup.get_changelog(__version__) if init: app_data_path = startup.get_appdata_dir() startup.reinit_appdata_dir(app_data_path) config = Config() db = Database() db.do_upgrade() tid = db.get_next_transaction_id() config.set('tid', tid, validate=False) if arl: if config.set("arl", arl): config._Config__write_modified_config() reload_config = Config() return print(f"ARL has been successfully updated to: {reload_config.arl()}") else: return print("Error when updating ARL.") if profile: profile_config = db.get_profile(profile) if profile_config: LoadProfile(profile_config) else: logger.error(f"Profile {profile} does not exist.") sys.exit(1) else: profile_config = db.get_profile_by_id(1) if profile_config: LoadProfile(profile_config) if not any(x in sys.argv[1:] for x in ['-h', '--help']): last_checked: int = int(db.last_update_check()) next_check: int = last_checked + (config.check_update() * 86400) if config.release_channel() != db.get_release_channel()['value']: # If release_channel has changed, check for latest release logger.debug(f"Release channel changed to '{config.release_channel()}'") db.set_release_channel() last_checked = 0 if time.time() >= next_check or last_checked == 0: logger.info(f"Checking for updates ({config.release_channel()})...") config.set('update_available', 0, False) latest_ver = str(startup.get_latest_version(config.release_channel())) if latest_ver: db.set_latest_version(latest_ver) db.set_last_update_check() new_version = db.get_latest_ver() if parse_version(new_version) > parse_version(__version__): if parse_version(new_version).major > parse_version(__version__).major: config.set('update_available', new_version, False) print("*" * 80) logger.info(f"deemon {parse_version(new_version).major} is available. " f"Please see the release notes before upgrading.") logger.info("Release notes available at: https://github.com/digitalec/deemon/releases") print("*" * 80) else: config.set('update_available', new_version, False) print("*" * 50) logger.info(f"* New version is available: v{__version__} -> v{new_version}") if config.release_channel() == "beta": logger.info("* To upgrade, run `pip install --upgrade --pre deemon`") else: logger.info("* To upgrade, run `pip install --upgrade deemon`") print("*" * 50) print("") config.set("start_time", int(time.time()), False) @run.command(name='test') @click.option('-e', '--email', is_flag=True, help="Send test notification to configured email") @click.option('-E', '--exclusions', metavar="URL", type=str, help="Test exclude regex pattern against URL") def test(email, exclusions): """Run tests on email configuration, exclusion filters, etc.""" if email: notification = notifier.Notify() notification.test() elif exclusions: if config.exclusion_patterns() or config.exclusion_keywords: tests.exclusion_test(exclusions) else: logger.info("You don't have any exclusions configured or they're disabled") @run.command(name='download', no_args_is_help=True) @click.argument('artist', nargs=-1, required=False) @click.option('-m', '--monitored', is_flag=True, help='Download all currently monitored artists') @click.option('-i', '--artist-id', multiple=True, metavar='ID', type=int, help='Download by artist ID') @click.option('-A', '--album-id', multiple=True, metavar='ID', type=int, help='Download by album ID') @click.option('-T', '--track-id', multiple=True, metavar='ID', type=int, help='Download by track ID') @click.option('-u', '--url', metavar='URL', multiple=True, help='Download by URL of artist/album/track/playlist') @click.option('-f', '--file', metavar='FILE', help='Download batch of artists or artist IDs from file', hidden=True) @click.option('--artist-file', metavar='FILE', help='Download batch of artists or artist IDs from file') @click.option('--album-file', metavar='FILE', help='Download batch of album IDs from file') @click.option('--track-file', metavar='FILE', help='Download batch of track IDs from file') @click.option('-a', '--after', 'from_date', metavar="YYYY-MM-DD", type=str, help='Grab releases released after this date') @click.option('-B', '--before', 'to_date', metavar="YYYY-MM-DD", type=str, help='Grab releases released before this date') @click.option('-b', '--bitrate', metavar="BITRATE", help='Set custom bitrate for this operation') @click.option('-o', '--download-path', metavar="PATH", type=str, help='Specify custom download directory') @click.option('-t', '--record-type', metavar="TYPE", type=str, help='Specify record types to download') def download_command(artist, artist_id, album_id, url, file, bitrate, record_type, download_path, from_date, to_date, monitored, track_id, track_file, artist_file, album_file): """ Download specific artist, album ID or by URL \b Examples: download Mozart download -i 100 -t album -b 9 """ if bitrate: config.set('bitrate', bitrate) if download_path: config.set('download_path', download_path) if record_type: config.set('record_type', record_type) artists = dataprocessor.csv_to_list(artist) if artist else None artist_ids = [x for x in artist_id] if artist_id else None album_ids = [x for x in album_id] if album_id else None track_ids = [x for x in track_id] if track_id else None urls = [x for x in url] if url else None if file: logger.info("WARNING: -f/--file has been replaced with --artist-file and will be removed in future versions.") artist_file = file if download_path and download_path != "": if Path(download_path).exists: config.set('download_path', download_path) logger.debug(f"Download path has changed: {config.download_path()}") else: return logger.error(f"Invalid download path: {download_path}") dl = download.Download() dl.set_dates(from_date, to_date) dl.download(artists, artist_ids, album_ids, urls, artist_file, track_file, album_file, track_ids, monitored=monitored) @run.command(name='monitor', context_settings={"ignore_unknown_options": False}, no_args_is_help=True) @click.argument('artist', nargs=-1) @click.option('-a', '--alerts', is_flag=True, help="Enable or disable alerts") @click.option('-b', '--bitrate', metavar="BITRATE", help="Specify bitrate") @click.option('-D', '--download', 'dl', is_flag=True, help='Download all releases matching record type') @click.option('-d', '--download-path', type=str, metavar="PATH", help='Specify custom download directory') @click.option('-I', '--import', 'im', metavar="PATH", help="Monitor artists/IDs from file or directory") @click.option('-i', '--artist-id', is_flag=True, help="Monitor artist by ID") @click.option('-p', '--playlist', is_flag=True, help='Monitor Deezer playlist by URL') @click.option('--include-artists', is_flag=True, help='Also monitor artists from playlist') @click.option('-u', '--url', is_flag=True, help='Monitor artist by URL') @click.option('-R', '--remove', is_flag=True, help='Stop monitoring an artist') @click.option('-s', '--search', 'search_flag', is_flag=True, help='Show similar artist results to choose from') @click.option('-T', '--time-machine', type=str, metavar="YYYY-MM-DD", help="Refresh newly added artists on this date") @click.option('-t', '--record-type', metavar="TYPE", type=str, help='Specify record types to download') def monitor_command(artist, im, playlist, include_artists, bitrate, record_type, alerts, artist_id, dl, remove, url, download_path, search_flag, time_machine): """ Monitor artist for new releases by ID, URL or name. \b Examples: monitor Mozart monitor --artist-id 100 monitor --url https://www.deezer.com/us/artist/000 """ monitor = Monitor() if download_path: if not Path(download_path).exists(): return logger.error("Invalid download path provided") if time_machine: validated = validate.validate_date(time_machine) if validated: monitor.time_machine = validated else: return logger.error("Date for time machine is invalid") if not alerts: alerts = None monitor.set_options(remove, dl, search_flag) monitor.set_config(bitrate, alerts, record_type, download_path) if url: artist_id = True urls = [x.replace(",", "") for x in artist] artist = [] for u in urls: id_from_url = u.split('/artist/') try: aid = int(id_from_url[1]) except (IndexError, ValueError): logger.error(f"Invalid artist URL -- {u}") return artist.append(aid) if playlist: urls = [x.replace(",", "") for x in artist] playlist_id = [] for u in urls: id_from_url = u.split('/playlist/') try: aid = int(id_from_url[1]) except (IndexError, ValueError): logger.error(f"Invalid playlist URL -- {u}") return playlist_id.append(aid) if im: monitor.importer(im) elif playlist: monitor.playlists(playlist_id, include_artists) elif artist_id: monitor.artist_ids(dataprocessor.csv_to_list(artist)) elif artist: monitor.artists(dataprocessor.csv_to_list(artist)) @run.command(name='refresh') @click.argument('NAME', nargs=-1, type=str, required=False) @click.option('-p', '--playlist', is_flag=True, help="Refresh a specific playlist by name") @click.option('-s', '--skip-download', is_flag=True, help="Skips downloading of new releases") @click.option('-T', '--time-machine', metavar='DATE', type=str, help='Refresh as if it were this date (YYYY-MM-DD)') def refresh_command(name, playlist, skip_download, time_machine): """Check artists for new releases""" if time_machine: time_machine = validate.validate_date(time_machine) if not time_machine: return logger.error("Date for time machine is invalid") logger.info(":: Starting database refresh") refresh = Refresh(time_machine, skip_download) if playlist: if not len(name): return logger.warning("You must provide the name of a playlist") refresh.run(playlists=dataprocessor.csv_to_list(name)) elif name: refresh.run(artists=dataprocessor.csv_to_list(name)) else: refresh.run() @click.group(name="show") def show_command(): """ Show monitored artists and latest releases """ @show_command.command(name="artists") @click.argument('artist', nargs=-1, required=False) @click.option('-c', '--csv', is_flag=True, help='Output artists as CSV') @click.option('-e', '--export', type=Path, help='Export CSV data to file; same as -ce') @click.option('-f', '--filter', type=str, help='Specify filter for CSV output') @click.option('-H', '--hide-header', is_flag=True, help='Hide header on CSV output') @click.option('-b', '--backup', type=Path, help='Backup artist IDs to CSV, same as -cHf id -e ...') def show_artists(artist, csv, export, filter, hide_header, backup): """Show artist info monitored by profile""" if artist: artist = ' '.join([x for x in artist]) show = Show() show.monitoring(artist=True, query=artist, export_csv=csv, save_path=export, filter=filter, hide_header=hide_header, backup=backup) @show_command.command(name="playlists") @click.argument('title', nargs=-1, required=False) @click.option('-c', '--csv', is_flag=True, help='Output artists as CSV') @click.option('-f', '--filter', type=str, help='Specify filter for CSV output') @click.option('-H', '--hide-header', is_flag=True, help='Hide header on CSV output') @click.option('-i', '--playlist-id', is_flag=True, help='Show playlist info by playlist ID') def show_artists(title, playlist_id, csv, filter, hide_header): """Show playlist info monitored by profile""" if title: title = ' '.join([x for x in title]) show = Show() show.monitoring(artist=False, query=title, export_csv=csv, filter=filter, hide_header=hide_header, is_id=playlist_id) @show_command.command(name="releases") @click.argument('N', default=7) @click.option('-f', '--future', is_flag=True, help='Display future releases') def show_releases(n, future): """ Show list of new or future releases """ show = Show() show.releases(n, future) run.add_command(show_command) @run.command(name="backup") @click.option('-i', '--include-logs', is_flag=True, help='include log files in backup') @click.option('-r', '--restore', is_flag=True, help='Restore from existing backup') def backup_command(restore, include_logs): """Backup configuration and database to a tar file""" if restore: backup.restore() else: backup.run(include_logs) # TODO @click.option does not support nargs=-1; unable to use spaces without quotations @run.command(name="api", help="View raw API data for artist, artist ID or playlist ID", hidden=True) @click.option('-A', '--album-id', type=int, help='Get album ID result via API') @click.option('-a', '--artist', type=str, help='Get artist result via API') @click.option('-i', '--artist-id', type=int, help='Get artist ID result via API') @click.option('-l', '--limit', type=int, help='Set max number of artist results; default=1', default=1) @click.option('-p', '--playlist-id', type=int, help='Get playlist ID result via API') @click.option('-r', '--raw', is_flag=True, help='Dump as raw data returned from API') def api_test(artist, artist_id, album_id, playlist_id, limit, raw): """View API result - for testing purposes""" import deezer dz = deezer.Deezer() if artist or artist_id: if artist: result = dz.api.search_artist(artist, limit=limit)['data'] else: result = dz.api.get_artist(artist_id) if raw: if isinstance(result, list): for row in result: for key, value in row.items(): print(f"{key}: {value}") print("\n") else: for key, value in result.items(): print(f"{key}: {value}") else: if isinstance(result, list): for row in result: print(f"Artist ID: {row['id']}\nArtist Name: {row['name']}\n") else: print(f"Artist ID: {result['id']}\nArtist Name: {result['name']}") if album_id: result = dz.api.get_album(album_id) if raw: for key, value in result.items(): print(f"{key}: {value}") else: print(f"Album ID: {result['id']}\nAlbum Title: {result['title']}") if playlist_id: result = dz.api.get_playlist(playlist_id) if raw: for key, value in result.items(): print(f"{key}: {value}") else: print(f"Playlist ID: {result['id']}\nPlaylist Title: {result['title']}") @run.command(name="reset") def reset_db(): """Reset monitoring database""" logger.warning("** WARNING: All artists and playlists will be removed regardless of profile! **") confirm = input(":: Type 'reset' to confirm: ") if confirm.lower() == "reset": print("") db.reset_database() else: logger.info("Reset aborted. Database has NOT been modified.") return @run.command(name='profile') @click.argument('profile', required=False) @click.option('-a', '--add', is_flag=True, help="Add new profile") @click.option('-c', '--clear', is_flag=True, help="Clear config for existing profile") @click.option('-d', '--delete', is_flag=True, help="Delete an existing profile") @click.option('-e', '--edit', is_flag=True, help="Edit an existing profile") def profile_command(profile, add, clear, delete, edit): """Add, modify and delete configuration profiles""" pc = ProfileConfig(profile) if profile: if add: pc.add() elif clear: pc.clear() elif delete: pc.delete() elif edit: pc.edit() else: pc.show() else: pc.show() @run.command(name="extra") def extra_command(): """Fetch extra release info""" extra.main() @run.command(name="search") @click.argument('query', nargs=-1, required=False) def search(query): """Interactively search and download/monitor artists""" if query: query = ' '.join(query) client = Search() client.search_menu(query) @run.command(name="config") @click.argument('artist', nargs=-1, required=True) def config_command(artist): """Configure per-artist settings by name or ID""" artist = ' '.join([x for x in artist]) artist_lookup(artist) @run.command(name="rollback", no_args_is_help=True) @click.argument('num', type=int, required=False) @click.option('-v', '--view', is_flag=True, help="View recent refresh transactions") def rollback_command(num, view): """Rollback a previous monitor or refresh transaction""" if view: rollback.view_transactions() elif num: rollback.rollback_last(num) @click.group(name="library") def library_command(): """ Library options such as upgrading from MP3 to FLAC """ @library_command.command(name="upgrade") @click.argument('library', metavar='PATH') @click.option('-A', '--album-only', is_flag=True, help="Get album IDs instead of track IDs (Fastest)") @click.option('-E', '--allow-exclusions', is_flag=True, help="Allow exclusions to be applied") @click.option('-O', '--output', metavar='PATH', help="Output file to save IDs (default: current directory)") def library_upgrade_command(library, output, album_only, allow_exclusions): """ (BETA) Scans MP3 files in PATH and generates a text file containing album/track IDs """ if not output: output = Path.cwd() upgradelib.upgrade(library, output, album_only, allow_exclusions) run.add_command(library_command) ================================================ FILE: deemon/cmd/__init__.py ================================================ ================================================ FILE: deemon/cmd/artistconfig.py ================================================ import logging from deemon.core.config import Config as config from deemon.core.db import Database logger = logging.getLogger(__name__) db = Database() def print_header(message: str = None): from os import system, name if name == 'nt': _ = system('cls') else: _ = system('clear') print("deemon Artist Configurator") if message: print(":: " + message + "\n") else: print("") def get_artist(query: str): artist_as_id = False artist_fromdb = None try: artist_as_id = int(query) except ValueError: pass by_name = db.get_monitored_artist_by_name(query) if by_name: logger.debug("Artist found by name") artist_fromdb = by_name if artist_as_id: by_id = db.get_monitored_artist_by_id(artist_as_id) if by_id: logger.debug(f"Artist found by ID") if not artist_fromdb: artist_fromdb = by_id else: logger.debug("Artist found by both ID and name, prompting user") while True: prompt = input(":: Multiple artists found. Was that a name or ID? ") if prompt.lower() == "name": logger.debug("Artist confirmed by user to be a name") return artist_fromdb elif prompt.lower() == "id": logger.debug("Artist confirmed by user to be an ID") return by_id else: return logger.error(f"Artist/Artist ID not found for '{query}'") if artist_fromdb: return artist_fromdb else: return logger.error(f"Artist/Artist ID not found for '{query}'") def artist_lookup(query): result = get_artist(query) if not result: return print_header(f"Configuring '{result['artist_name']}' (Artist ID: {result['artist_id']})") modified = 0 for property in result: if property not in ['alerts', 'bitrate', 'record_type', 'download_path']: continue allowed_opts = config.allowed_values(property) if isinstance(allowed_opts, dict): allowed_opts = [str(x.lower()) for x in allowed_opts.values()] while True: friendly_text = property.replace("_", " ").title() user_input = input(f"{friendly_text} [{result[property]}]: ") if property != "download_path": user_input = user_input.lower() if user_input == "": break elif user_input == "false" or user_input == "0": user_input = False elif user_input == "true" or user_input == "1": user_input = True if user_input == "none": user_input = None elif allowed_opts: if user_input not in allowed_opts: print(f"Allowed options: " + ', '.join(str(x) for x in allowed_opts)) continue logger.debug(f"User set {property} to {user_input}") result[property] = user_input modified += 1 break if modified > 0: i = input("\n:: Save these settings? [y|N] ") if i.lower() != "y": logger.info("No changes made, exiting...") else: db.update_artist(result) print(f"\nArtist '{result['artist_name']}' has been updated!") else: print("No changes made, exiting...") ================================================ FILE: deemon/cmd/backup.py ================================================ import logging import os import tarfile from datetime import datetime from pathlib import Path from packaging.version import parse as parse_version from tqdm import tqdm from deemon import __version__ from deemon.utils import startup, dates logger = logging.getLogger(__name__) def run(include_logs: bool = False): def filter_func(item): includes = ['deemon', 'deemon/config.json', 'deemon/deemon.db'] if include_logs: if 'deemon/logs' in item.name: includes.append(item.name) if item.name in includes: return item backup_tar = dates.generate_date_filename("backup-" + __version__ + "-") + ".tar" backup_path = startup.get_backup_dir() with tarfile.open(backup_path / backup_tar, "w") as tar: tar.add(startup.get_appdata_dir(), arcname='deemon', filter=filter_func) logger.info(f"Backed up to {backup_path / backup_tar}") def restore(): restore_file_list = ['deemon/config.json', 'deemon/deemon.db'] def inspect_tar(fn: Path) -> dict: fn_name = fn.name fn_name = fn_name.replace('.tar', '').split('-') if fn_name[0] == "backup" and len(fn_name) > 3: if check_tar_contents(fn): backup_appversion = '-'.join(fn_name[1:-2]) if is_newer_backup(backup_appversion): logger.debug(f"Backup found for newer version {backup_appversion} is not compatible!") return backup_time = datetime.strptime(fn_name[-1], "%H%M%S") backup_date = datetime.strptime(fn_name[-2], "%Y%m%d") try: friendly_time = datetime.strftime(backup_time, "%-I:%M:%S %p") except ValueError: # Gotta keep Windows happy... friendly_time = datetime.strftime(backup_time, "%#I:%M:%S %p") try: friendly_date = datetime.strftime(backup_date, "%b %-d, %Y") except ValueError: # Gotta keep Windows happy... friendly_date = datetime.strftime(backup_date, "%b %#d, %Y") backup_info = { 'version': backup_appversion, 'date': friendly_date, 'time': friendly_time, 'age': fn_name[-2] + fn_name[-1], 'filename': fn } return backup_info else: return def check_tar_contents(archive: Path): tar = tarfile.open(archive) file_list = tar.getmembers() files = [x.name for x in file_list] if all(item in files for item in restore_file_list): return True logger.debug("Archive is invalid or corrupt: " + str(archive)) def restore_tarfile(archive: dict): logger.debug("Restoring backup from `" + str(archive['filename'].name + "`")) extract_dir = startup.get_appdata_dir() tar = tarfile.open(archive['filename']) progress = tqdm(tar.getmembers(), ascii=" #", bar_format='{desc} [{bar}] {percentage:3.0f}%') for member in progress: if member.isreg(): if member.name in restore_file_list: member.name = os.path.basename(member.name) logger.info(f"Restoring {member.name}...") progress.set_description_str(f"Restoring {member.name}") tar.extract(member, extract_dir) logger.debug(f"Restored {member.name} to {extract_dir}") if member == tar.getmembers()[-1]: progress.set_description_str("Restore complete") def is_newer_backup(version: str): if parse_version(version) > parse_version(__version__): return True def display_backup_list(available_backups: list): print("deemon Backup Manager\n") for index, backup in enumerate(available_backups, start=1): print(f"{index}. {backup['date']} @ {backup['time']} (ver {backup['version']})") selected_backup = int while selected_backup not in range(len(available_backups)): selected_backup = input("\nSelect a backup to restore (or press Enter to exit): ") if selected_backup == "": return try: selected_backup = int(selected_backup) selected_backup -= 1 except ValueError: logger.warning("Invalid entry. Enter a number corresponding to the backup you wish to restore.") print("") restore_tarfile(available_backups[selected_backup]) backups = [] backup_path = startup.get_backup_dir() file_list = [x for x in sorted(Path(backup_path).glob('*.tar'))] for backup in file_list: tar_files = inspect_tar(backup) if tar_files: backups.append(tar_files) if backups: backups = sorted(backups, key=lambda x: x['age'], reverse=True) display_backup_list(backups) else: logger.info("No backups available to restore") ================================================ FILE: deemon/cmd/download.py ================================================ import logging import os import sys import requests from concurrent.futures import ThreadPoolExecutor from pathlib import Path from tqdm import tqdm import deemix.errors import deezer from deezer import errors import plexapi.exceptions from plexapi.server import PlexServer from deemon import utils from deemon.core import dmi, db, api, common from deemon.core.config import Config as config from deemon.utils import ui, dataprocessor, startup, dates logger = logging.getLogger(__name__) class QueueItem: # TODO - Accept new playlist tracks for output/alerts def __init__(self, artist=None, album=None, track=None, playlist=None, bitrate: str = None, download_path: str = None, release_full: dict = None): self.artist_name = None self.album_id = None self.album_title = None self.track_id = None self.track_title = None self.url = None self.playlist_title = None self.bitrate = bitrate or config.bitrate() self.download_path = download_path or config.download_path() self.release_type = None if release_full: self.artist_name = release_full['artist_name'] self.album_id = release_full['id'] self.album_title = release_full['title'] self.url = f"https://www.deezer.com/album/{self.album_id}" self.release_type = release_full['record_type'] self.bitrate = release_full['bitrate'] self.download_path = release_full['download_path'] if artist: try: self.artist_name = artist["artist_name"] except KeyError: self.artist_name = artist["name"] if not album and not track: self.url = artist["link"] if album: if not artist: self.artist_name = album["artist"]["name"] self.album_id = album["id"] self.album_title = album["title"] try: self.url = album["link"] except KeyError: self.url = f"https://www.deezer.com/album/{album['id']}" if track: self.artist_name = track["artist"]["name"] self.track_id = track["id"] self.track_title = track["title"] self.url = f"https://deezer.com/track/{self.track_id}" if playlist: try: self.url = playlist["link"] except KeyError: logger.debug("DEPRECATED dict key: playlist['url'] should not be used in favor of playlist['link']") self.url = playlist.get("url", None) self.playlist_title = playlist["title"] def get_deemix_bitrate(bitrate: str): for bitrate_id, bitrate_name in config.allowed_values('bitrate').items(): if bitrate_name.lower() == bitrate.lower(): logger.debug(f"Setting deemix bitrate to {str(bitrate_id)}") return bitrate_id def get_plex_server(): if (config.plex_baseurl() != "") and (config.plex_token() != ""): session = None if not config.plex_ssl_verify(): requests.packages.urllib3.disable_warnings() session = requests.Session() session.verify = False try: print("Plex settings found, trying to connect (10s)... ", end="") plex_server = PlexServer(config.plex_baseurl(), config.plex_token(), timeout=10, session=session) print(" OK") return plex_server except Exception as e: print(" FAILED") logger.error("Error: Unable to reach Plex server, please refresh manually.") logger.debug(e) return False def refresh_plex(plexobj): try: plexobj.library.section(config.plex_library()).update() logger.debug("Plex library refreshed successfully") except plexapi.exceptions.BadRequest as e: logger.error("Error occurred while refreshing your library. See logs for additional info.") logger.debug(f"Error during Plex refresh: {e}") except plexapi.exceptions.NotFound as e: logger.error("Error: Plex library not found. See logs for additional info.") logger.debug(f"Error during Plex refresh: {e}") class Download: def __init__(self, active_api=None): super().__init__() self.api = active_api or api.PlatformAPI() self.dz = deezer.Deezer() self.di = dmi.DeemixInterface() self.queue_list = [] self.db = db.Database() self.bitrate = None self.release_from = None self.release_to = None self.verbose = os.environ.get("VERBOSE") self.duplicate_id_count = 0 def set_dates(self, from_date: str = None, to_date: str = None) -> None: """Set to/from dates to get while downloading""" if from_date: try: self.release_from = dates.str_to_datetime_obj(from_date) except ValueError as e: raise ValueError(f"Invalid date provided - {from_date}: {e}") if to_date: try: self.release_to = dates.str_to_datetime_obj(to_date) except ValueError as e: raise ValueError(f"Invalid date provided - {to_date}: {e}") # @performance.timeit def download_queue(self, queue_list: list = None): if queue_list: self.queue_list = queue_list if not self.di.login(): logger.error("Failed to login, aborting download...") return False if self.queue_list: plex = get_plex_server() print("") logger.info(":: Sending " + str(len(self.queue_list)) + " release(s) to deemix for download:") with open(startup.get_appdata_dir() / "queue.csv", "w", encoding="utf-8") as f: f.writelines(','.join([str(x) for x in vars(self.queue_list[0]).keys()]) + "\n") logger.debug(f"Writing queue to CSV file - {len(self.queue_list)} items in queue") for q in self.queue_list: raw_values = [str(x) for x in vars(q).values()] # TODO move this to shared function for i, v in enumerate(raw_values): if '"' in v: raw_values[i] = v.replace('"', "'") if ',' in v: raw_values[i] = f'"{v}"' f.writelines(','.join(raw_values) + "\n") logger.debug(f"Queue exported to {startup.get_appdata_dir()}/queue.csv") failed_count = [] download_progress = tqdm( self.queue_list, total=len(self.queue_list), desc="Downloading releases...", ascii=" #", bar_format=ui.TQDM_FORMAT ) for index, item in enumerate(download_progress): i = str(index + 1) t = str(len(download_progress)) download_progress.set_description_str(f"Downloading release {i} of {t}...") dx_bitrate = get_deemix_bitrate(item.bitrate) if self.verbose == "true": logger.debug(f"Processing queue item {vars(item)}") try: if item.download_path: download_path = item.download_path else: download_path = None if item.artist_name: if item.album_title: logger.info(f" > {item.artist_name} - {item.album_title}... ") self.di.download_url([item.url], dx_bitrate, download_path) else: logger.info(f" > {item.artist_name} - {item.track_title}... ") self.di.download_url([item.url], dx_bitrate, download_path) else: logger.info(f" > {item.playlist_title} (playlist)...") self.di.download_url([item.url], dx_bitrate, download_path, override_deemix=True) except (deemix.errors.GenerationError, errors.WrongGeolocation) as e: logger.debug(e) failed_count.append([(item, "No tracks listed or unavailable in your country")]) except Exception as e: if item.artist_name and item.album_title: logger.info(f"The following error occured while downloading {item.artist_name} - {item.album_title}: {e}") elif item.artist_name and item.track_title: logger.info(f"The following error occured while downloading {item.artist_name} - {item.track_title}: {e}") else: logger.info(f"The following error occured while downloading {item.playlist_title}: {e}") pass failed_count = [x for x in failed_count if x] print("") if len(failed_count): logger.info(f" [!] Downloads completed with {len(failed_count)} error(s):") with open(startup.get_appdata_dir() / "failed.csv", "w", encoding="utf-8") as f: f.writelines(','.join([str(x) for x in vars(self.queue_list[0]).keys()]) + "\n") for failed in failed_count: try: raw_values = [str(x) for x in vars(failed[0]).values()] except TypeError as e: print(f"Error reading from failed.csv. Entry that failed was either invalid or empty: {failed}") logger.error(e) else: # TODO move this to shared function for i, v in enumerate(raw_values): if '"' in v: raw_values[i] = v.replace('"', "'") if ',' in v: raw_values[i] = f'"{v}"' f.writelines(','.join(raw_values) + "\n") print(f"+ {failed[0].artist_name} - {failed[0].album_title} --- Reason: {failed[1]}") print("") logger.info(f":: Failed downloads exported to: {startup.get_appdata_dir()}/failed.csv") else: logger.info(" Downloads complete!") if plex and (config.plex_library() != ""): refresh_plex(plex) return True def download(self, artist, artist_id, album_id, url, artist_file, track_file, album_file, track_id, auto=True, monitored=False): def filter_artist_by_record_type(artist): album_api = self.api.get_artist_albums(query={'artist_name': '', 'artist_id': artist['id']}) filtered_albums = [] for album in album_api['releases']: if (album['record_type'] == config.record_type()) or config.record_type() == "all": album_date = dates.str_to_datetime_obj(album['release_date']) if self.release_from and self.release_to: if album_date > self.release_from and album_date < self.release_to: filtered_albums.append(album) elif self.release_from: if album_date > self.release_from: filtered_albums.append(album) elif self.release_to: if album_date < self.release_to: filtered_albums.append(album) else: filtered_albums.append(album) return filtered_albums def get_api_result(artist=None, artist_id=None, album_id=None, track_id=None): if artist: try: return self.api.search_artist(artist, limit=1)['results'][0] except (deezer.api.DataException, IndexError): logger.error(f"Artist {artist} not found.") if artist_id: try: return self.api.get_artist_by_id(artist_id) except (deezer.api.DataException, IndexError): logger.error(f"Artist ID {artist_id} not found.") if album_id: try: return self.api.get_album(album_id) except (deezer.api.DataException, IndexError): logger.error(f"Album ID {album_id} not found.") if track_id: try: return self.api.get_track(track_id) except (deezer.api.DataException, IndexError): logger.error(f"Track ID {track_id} not found.") def queue_filtered_releases(api_object): filtered = filter_artist_by_record_type(api_object) filtered = common.exclude_filtered_versions(filtered) for album in filtered: if not queue_item_exists(album['id']): self.queue_list.append(QueueItem(artist=api_object, album=album)) def queue_item_exists(i): for q in self.queue_list: if q.album_id == i: logger.debug(f"Album ID {i} is already in queue") self.duplicate_id_count += 1 return True return False def process_artist_by_name(name): artist_result = get_api_result(artist=name) if not artist_result: return logger.debug(f"Requested Artist: '{name}', Found: '{artist_result['name']}'") if artist_result: queue_filtered_releases(artist_result) def process_artist_by_id(i): artist_id_result = get_api_result(artist_id=i) if not artist_id_result: return logger.debug(f"Requested Artist ID: {i}, Found: {artist_id_result['name']}") if artist_id_result: queue_filtered_releases(artist_id_result) def process_album_by_id(i): logger.debug("Processing album by ID") album_id_result = get_api_result(album_id=i) if not album_id_result: logger.debug(f"Album ID {i} was not found") return logger.debug(f"Requested album: {i}, " f"Found: {album_id_result['artist']['name']} - {album_id_result['title']}") if album_id_result and not queue_item_exists(album_id_result['id']): self.queue_list.append(QueueItem(album=album_id_result)) def process_track_by_id(id): logger.debug("Processing track by ID") track_id_result = get_api_result(track_id=id) if not track_id_result: return logger.debug(f"Requested track: {id}, " f"Found: {track_id_result['artist']['name']} - {track_id_result['title']}") if track_id_result and not queue_item_exists(id): self.queue_list.append(QueueItem(track=track_id_result)) def process_track_file(id): if not queue_item_exists(id): track_data = { "artist": { "name": "TRACK ID" }, "id": id, "title": id } self.queue_list.append(QueueItem(track=track_data)) def process_playlist_by_id(id): playlist_api = self.api.get_playlist(id) self.queue_list.append(QueueItem(playlist=playlist_api)) def extract_id_from_url(url): id_group = ['artist', 'album', 'track', 'playlist'] for group in id_group: id_type = group try: # Strip ID from URL id_from_url = url.split(f'/{group}/')[1] # Support for share links: http://deezer.com/us/track/12345?utm_campaign... id_from_url_extra = id_from_url.split('?')[0] id = int(id_from_url_extra) logger.debug(f"Extracted group={id_type}, id={id}") return id_type, id except (IndexError, ValueError) as e: continue return False, False logger.info("[!] Queueing releases, this might take awhile...") if self.release_from or self.release_to: if self.release_from and self.release_to: logger.info(":: Getting releases that were released between " f"{dates.ui_date(self.release_from)} and " f"{dates.ui_date(self.release_to)}") elif self.release_from: logger.info(":: Getting releases that were released after " f"{dates.ui_date(self.release_from)}") elif self.release_to: logger.info(":: Getting releases that were released before " f"{dates.ui_date(self.release_to)}") if monitored: artist_id = self.db.get_all_monitored_artist_ids() if artist: [process_artist_by_name(a) for a in artist] if artist_id: [process_artist_by_id(i) for i in artist_id] if album_id: [process_album_by_id(i) for i in album_id] if track_id: [process_track_by_id(i) for i in track_id] if album_file: logger.info(f":: Reading from file {album_file}") if Path(album_file).exists(): album_list = utils.dataprocessor.read_file_as_csv(album_file, split_new_line=False) album_list = utils.dataprocessor.process_input_file(album_list) if album_list: if isinstance(album_list[0], int): with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: _api_results = list(tqdm(ex.map(process_album_by_id, album_list), total=len(album_list), desc=f"Fetching album data for {len(album_list)} " f"album(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) else: logger.debug(f"Invalid album ID: \"{album_list[0]}\"") logger.error(f"Invalid album ID file detected.") else: logger.error(f"The file {album_file} could not be found") sys.exit() if artist_file: # TODO artist_file is in different format than album_file and track_file # TODO is one continuous CSV line better than separate lines? logger.info(f":: Reading from file {artist_file}") if Path(artist_file).exists(): artist_list = utils.dataprocessor.read_file_as_csv(artist_file) if artist_list: if isinstance(artist_list[0], int): logger.debug(f"{artist_file} contains artist IDs") with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: _api_results = list(tqdm(ex.map(process_artist_by_id, artist_list), total=len(artist_list), desc=f"Fetching artist release data for {len(artist_list)} " f"artist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) elif isinstance(artist_list[0], str): logger.debug(f"{artist_file} contains artist names") with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: _api_results = list(tqdm(ex.map(process_artist_by_name, artist_list), total=len(artist_list), desc=f"Fetching artist release data for {len(artist_list)} " f"artist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) else: logger.error(f"The file {artist_file} could not be found") sys.exit() if track_file: logger.info(f":: Reading from file {track_file}") if Path(track_file).exists(): track_list = utils.dataprocessor.read_file_as_csv(track_file, split_new_line=False) try: track_list = [int(x) for x in track_list] except TypeError: logger.info("Track file must only contain track IDs") return if track_list: with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: _api_results = list(tqdm(ex.map(process_track_file, track_list), total=len(track_list), desc=f"Fetching track release data for {len(track_list)} " f"track(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) else: logger.error(f"The file {track_file} could not be found") sys.exit() if url: logger.debug("Processing URLs") for u in url: egroup, eid = extract_id_from_url(u) if not egroup or not eid: logger.error(f"Invalid URL -- {u}") continue if egroup == "artist": process_artist_by_id(eid) elif egroup == "album": process_album_by_id(eid) elif egroup == "playlist": process_playlist_by_id(eid) elif egroup == "track": process_track_by_id(eid) if self.duplicate_id_count > 0: logger.info(f"Cleaned up {self.duplicate_id_count} duplicate release(s). See log for additional info.") if auto: if len(self.queue_list): self.download_queue() else: print("") logger.info("No releases found matching applied filters.") ================================================ FILE: deemon/cmd/extra.py ================================================ from concurrent.futures import ThreadPoolExecutor import logging from tqdm import tqdm from deemon.core import db as dbase from deemon.core.api import PlatformAPI from deemon.core.config import Config as config from deemon.utils import ui, performance logger = logging.getLogger(__name__) def debugger(message: str, payload = None): if config.debug_mode(): if not payload: payload = "" logger.debug(f"DEBUG_MODE: {message} {str(payload)}") def main(): db = dbase.Database() api = PlatformAPI() releases = db.get_artist_releases() if not len(releases): return logger.warning("No releases found in local database") logger.debug("Fetching extra release data...") debugger("SpawningThreads", api.max_threads) with ThreadPoolExecutor(max_workers=api.max_threads) as ex: api_result = list( tqdm(ex.map(api.get_extra_release_info, releases), total=len(releases), desc=f"Fetching extra release data for {len(releases):,} " "releases, please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT) ) if len(api_result): logger.info(":: Saving changes to database, this can take several minutes...") db.add_extra_release_info(api_result) db.commit() print("") performance.operation_time(config.get('start_time')) logger.info("Extra release info has been updated") ================================================ FILE: deemon/cmd/generate.py ================================================ from pathlib import Path import tqdm as tqdm from deezer import Deezer def read_album_ids_from_file(filename): if not Path(filename).exists(): raise Exception("File does not exist") ids = [] with open(filename, 'r') as f: f.readline() for l in f: ids.append(l.encode("ascii", "ignore")) print("Total lines read from text file: " + str(len(ids))) return ids def clean_absolute_paths(album_list): stripped = [] for line in album_list: stripped.append(line.split("\\")) return stripped def clean_year_from_album(album_list, level: int = 5): artist_album = [] for line in album_list: if len(line) == level: strip_year = line[3][-6:].strip("()") try: int(strip_year) except Exception: artist_album.append([line[2], line[3]]) continue artist_album.append([line[2], line[3][:-6]]) return artist_album def clean_artist_album_text(album_list: list): stripped = [] for line in album_list: line = line.decode() line = line.replace('\n', '') line = line.replace('�', '') line = line.replace('¡', '') line = line.replace('É', 'E') line = line.replace('.', '') line = line.replace('+', ' ') line = line.replace('/', ' ') stripped.append(line.split(" - ")) return stripped def get_artist_album(filename: str, absolute_path: bool = False): list_from_file = read_album_ids_from_file(filename) if absolute_path: stripped_paths = clean_absolute_paths(list_from_file) artist_album = clean_year_from_album(stripped_paths, level=5) else: artist_album = clean_artist_album_text(list_from_file) print("Total albums to lookup: " + str(len(artist_album))) return sorted(x for x in artist_album) def get_api_results(album_list, artist_name: str = None): dz = Deezer() for x in album_list: album_list.set_description_str(f"Pass: {str(len(id_list))} | Fail: {str(len(fail_list))}") # For testing, only process this artist if artist_name and artist_name != x[0]: continue # Remove things like "(Bonus Tracks)" artist_from_file = x[0] album_from_file = x[1] stripped_album_from_file = x[1].split(" (")[0] api_artist = dz.api.search_artist(x[0], limit=10)['data'] found_artist = True for artist_result in api_artist: # TODO name decode - replace unknown with ? - use as wildcard encoded_name = artist_result['name'].encode("ascii", "replace") decoded_name = encoded_name.decode() if artist_from_file == decoded_name: api_artist = artist_result found_artist = True break else: found_artist = False # TODO make this add albums to id and break out if found_artist is False: for artist in api_artist: get_albums = dz.api.get_artist_albums(artist['id'])['data'] if album_from_file in [x['title'] for x in get_albums]: api_artist = artist break # # print("Searched all albums, nothing matches!") # exit() try: all_albums = dz.api.get_artist_albums(api_artist['id'])['data'] except: fail_list.append(f"{x[0]} - {x[1]}") continue api_albums = [[x['title'].split(" (")[0], x['id']] for x in all_albums] for [title, id] in api_albums: clean_title = title clean_title = clean_title.replace('¡', '') clean_title = clean_title.replace('É', 'E') clean_title = clean_title.replace('.', '') clean_title = clean_title.replace(' + ', ' ') clean_title = clean_title.replace(' / ', ' ') if album_from_file.lower() == clean_title.lower() or stripped_album_from_file.lower() == clean_title.lower(): if id not in id_list: id_list.append(id) break if album_from_file.lower() in clean_title.lower() or stripped_album_from_file.lower() in clean_title.lower(): if id not in id_list: id_list.append(id) break else: fail_list.append(f"{x[0]} - {x[1]}") id_list = [] fail_list = [] input_file_or_directory: str = None output_file_passing: str = None output_file_failing: str = None album_list = get_artist_album(input_file_or_directory) progress = tqdm.tqdm(album_list, ascii=" #", bar_format='{desc}... {n_fmt}/{total_fmt} [{bar:40}] {percentage:3.0f}%') progress.set_description_str("Getting IDs") get_api_results(progress) print("PASS: " + str(len(id_list))) with open(output_file_passing, "w") as f: for id in id_list: f.write(str(id) + "\n") print("FAIL: " + str(len(fail_list))) with open(output_file_failing, "w") as f: for id in fail_list: f.write(str(id) + "\n") ================================================ FILE: deemon/cmd/monitor.py ================================================ import logging from concurrent.futures import ThreadPoolExecutor from pathlib import Path from tqdm import tqdm from deemon.cmd import search from deemon.cmd.refresh import Refresh from deemon.core.api import PlatformAPI from deemon.core.config import Config as config from deemon.core.db import Database from deemon.utils import dataprocessor, ui logger = logging.getLogger(__name__) class Monitor: def __init__(self, active_api=None): self.bitrate = None self.alerts = False self.record_type = None self.download_path = None self.remove = False self.refresh = True self.is_search = False self.duplicates = 0 self.time_machine = None self.dl = None self.db = Database() self.api = active_api or PlatformAPI() def set_config(self, bitrate: str, alerts: bool, record_type: str, download_path: Path): self.bitrate = bitrate self.alerts = alerts self.record_type = record_type self.download_path = download_path self.debugger("SetConfig", {'bitrate': bitrate, 'alerts': alerts, 'type': record_type, 'path': download_path}) def set_options(self, remove, dl_all, search): self.remove = True if remove else False self.dl = True if dl_all else False self.is_search = True if search else False self.debugger("SetOptions", {'remove': remove, 'dl': dl_all, 'search': search}) def debugger(self, message: str, payload=None): if config.debug_mode(): if not payload: payload = "" logger.debug(f"DEBUG_MODE: {message} {str(payload)}") def get_best_result(self, api_result) -> list: name = api_result['query'] if self.is_search: logger.debug("Waiting for user input...") prompt = self.prompt_search(name, api_result['results']) if prompt: logger.debug(f"User selected {prompt}") return [prompt] matches = [r for r in api_result['results'] if r['name'].lower() == name.lower()] self.debugger("Matches", matches) if len(matches) == 1: return [matches[0]] elif len(matches) > 1: logger.debug(f"Multiple matches were found for artist \"{api_result['query']}\"") if config.prompt_duplicates(): logger.debug("Waiting for user input...") prompt = self.prompt_search(name, matches) if prompt: logger.debug(f"User selected {prompt}") return [prompt] else: logger.info(f"No selection made, skipping {name}...") return [] else: self.duplicates += 1 return [matches[0]] elif not len(matches): logger.debug(f" [!] No matches were found for artist \"{api_result['query']}\"") if config.prompt_no_matches() and len(api_result['results']): logger.debug("Waiting for user input...") prompt = self.prompt_search(name, api_result['results']) if prompt: logger.debug(f"User selected {prompt}") return [prompt] else: logger.info(f"No selection made, skipping {name}...") return [] else: logger.info(f" [!] Artist {name} not found") return [] def prompt_search(self, value, api_result): menu = search.Search(active_api=self.api) ask_user = menu.artist_menu(value, api_result, True) if ask_user: return {'id': ask_user['id'], 'name': ask_user['name']} return logger.debug("No artist selected, skipping...") # @performance.timeit def build_artist_query(self, api_result: list): existing = self.db.get_all_monitored_artist_ids() artists_to_add = [] pbar = tqdm(api_result, total=len(api_result), desc="Setting up artists for monitoring...", ascii=" #", bar_format=ui.TQDM_FORMAT) for artist in pbar: if artist is None: continue if artist['id'] in existing: logger.info(f" - Already monitoring {artist['name']}, skipping...") else: artist.update({'bitrate': self.bitrate, 'alerts': self.alerts, 'record_type': self.record_type, 'download_path': self.download_path, 'profile_id': config.profile_id(), 'trans_id': config.transaction_id()}) artists_to_add.append(artist) if len(artists_to_add): logger.debug("New artists have been monitored. Saving changes to the database...") self.db.new_transaction() self.db.fast_monitor(artists_to_add) self.db.commit() return True def build_playlist_query(self, api_result: list, include_artists: bool): if include_artists: include_artists = '1' existing = self.db.get_all_monitored_playlist_ids() or [] playlists_to_add = [] pbar = tqdm(api_result, total=len(api_result), desc="Setting up playlists for monitoring...", ascii=" #", bar_format=ui.TQDM_FORMAT) for i, playlist in enumerate(pbar): if not playlist: continue if playlist['id'] in existing: logger.info(f" Already monitoring {playlist['title']}, skipping...") else: playlist.update( { 'bitrate': self.bitrate, 'alerts': self.alerts, 'download_path': self.download_path, 'profile_id': config.profile_id(), 'trans_id': config.transaction_id(), 'monitor_artists': include_artists } ) playlists_to_add.append(playlist) if len(playlists_to_add): logger.debug("New playlists have been monitored. Saving changes to the database...") self.db.new_transaction() self.db.fast_monitor_playlist(playlists_to_add) self.db.commit() return True def call_refresh(self): refresh = Refresh(self.time_machine, ignore_filters=self.dl, active_api=self.api) refresh.run() # @performance.timeit def artists(self, names: list) -> None: """ Return list of dictionaries containing each artist """ if self.remove: return self.purge_artists(names=names) self.debugger("SpawningThreads", self.api.max_threads) with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: api_result = list( tqdm(ex.map(self.api.search_artist, names), total=len(names), desc=f"Fetching artist data for {len(names):,} artist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) select_artist = tqdm(api_result, total=len(api_result), desc="Examining results for best match...", ascii=" #", bar_format=ui.TQDM_FORMAT) to_monitor = [] for artist in select_artist: best_result = self.get_best_result(artist) if best_result: to_monitor.append(best_result) to_process = [item for elem in to_monitor for item in elem if len(item)] if self.build_artist_query(to_process): self.call_refresh() else: print("") logger.info("No new artists have been added, skipping refresh.") # @performance.timeit def artist_ids(self, ids: list): ids = [int(x) for x in ids] if self.remove: return self.purge_artists(ids=ids) self.debugger("SpawningThreads", self.api.max_threads) with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: api_result = list( tqdm(ex.map(self.api.get_artist_by_id, ids), total=len(ids), desc=f"Fetching artist data for {len(ids):,} artist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) if self.build_artist_query(api_result): self.call_refresh() else: print("") logger.info("No new artists have been added, skipping refresh.") # @performance.timeit def importer(self, import_path: str): if Path(import_path).is_file(): imported_file = dataprocessor.read_file_as_csv(import_path) artist_list = dataprocessor.process_input_file(imported_file) if isinstance(artist_list[0], int): self.artist_ids(artist_list) else: self.artists(artist_list) elif Path(import_path).is_dir(): import_list = [x.relative_to(import_path).name for x in sorted(Path(import_path).iterdir()) if x.is_dir()] if import_list: self.artists(import_list) else: logger.error(f"File or directory not found: {import_path}") return # @performance.timeit def playlists(self, playlists: list, include_artists: bool): if self.remove: return self.purge_playlists(ids=playlists) ids = [int(x) for x in playlists] self.debugger("SpawningThreads", self.api.max_threads) with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: api_result = list( tqdm(ex.map(self.api.get_playlist, ids), total=len(ids), desc=f"Fetching playlist data for {len(ids):,} playlist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT)) if self.build_playlist_query(api_result, include_artists): self.call_refresh() else: print("") logger.info("No new playlists have been added, skipping refresh.") # @performance.timeit def purge_artists(self, names: list = None, ids: list = None): if names: for n in names: monitored = self.db.get_monitored_artist_by_name(n) if monitored: self.db.remove_monitored_artist(monitored['artist_id']) logger.info(f"No longer monitoring {monitored['artist_name']}") else: logger.info(f"{n} is not being monitored yet") if ids: for i in ids: monitored = self.db.get_monitored_artist_by_id(i) if monitored: self.db.remove_monitored_artist(monitored['artist_id']) logger.info(f"\nNo longer monitoring {monitored['artist_name']}") else: logger.info(f"{i} is not being monitored yet") def purge_playlists(self, titles: list = None, ids: list = None): if ids: for i in ids: monitored = self.db.get_monitored_playlist_by_id(i) if monitored: self.db.remove_monitored_playlists(monitored['id']) logger.info(f"\nNo longer monitoring {monitored['title']}") else: logger.info(f"{i} is not being monitored yet") ================================================ FILE: deemon/cmd/profile.py ================================================ import logging from deemon.core.config import Config as config from deemon.core.db import Database logger = logging.getLogger(__name__) class ProfileConfig: def __init__(self, profile_name): self.db = Database() self.profile_name = profile_name self.profile = None # TODO move this to utils @staticmethod def print_header(message: str = None): print("deemon Profile Editor") if message: print(":: " + message + "\n") else: print("") def edit(self): profile = self.db.get_profile(self.profile_name) self.print_header(f"Configuring '{profile['name']}' (Profile ID: {profile['id']})") modified = 0 for property in profile: if property == "id": continue allowed_opts = config.allowed_values(property) if isinstance(allowed_opts, dict): allowed_opts = [str(x.lower()) for x in allowed_opts.values()] while True: friendly_text = property.replace("_", " ").title() user_input = input(f"{friendly_text} [{profile[property]}]: ").lower() if user_input == "": break # TODO move to function to share with Config.set()? elif user_input == "false" or user_input == "0": user_input = False elif user_input == "true" or user_input == "1": user_input = True elif property == "name" and self.profile_name != user_input: if self.db.get_profile(user_input): print("Name already in use") continue if user_input == "none" and property != "name": user_input = None elif allowed_opts: if user_input not in allowed_opts: print(f"Allowed options: " + ', '.join(str(x) for x in allowed_opts)) continue logger.debug(f"User set {property} to {user_input}") profile[property] = user_input modified += 1 break if modified > 0: user_input = input("\n:: Save these settings? [y|N] ") if user_input.lower() != "y": logger.info("No changes made, exiting...") else: self.db.update_profile(profile) print(f"\nProfile '{profile['name']}' has been updated!") else: print("No changes made, exiting...") def add(self): new_profile = {} profile_config = self.db.get_profile(self.profile_name) if profile_config: return logger.error(f"Profile {self.profile_name} already exists") else: logger.info("Adding new profile: " + self.profile_name) print("** Any option left blank will fallback to global config **\n") new_profile['name'] = self.profile_name menu = [ {'setting': 'email', 'type': str, 'text': 'Email address', 'allowed': []}, {'setting': 'alerts', 'type': bool, 'text': 'Alerts', 'allowed': config.allowed_values('alerts')}, {'setting': 'bitrate', 'type': str, 'text': 'Bitrate', 'allowed': config.allowed_values('bitrate').values()}, {'setting': 'record_type', 'type': str, 'text': 'Record Type', 'allowed': config.allowed_values('record_type')}, {'setting': 'plex_baseurl', 'type': str, 'text': 'Plex Base URL', 'allowed': []}, {'setting': 'plex_token', 'type': str, 'text': 'Plex Token', 'allowed': []}, {'setting': 'plex_library', 'type': str, 'text': 'Plex Library', 'allowed': []}, {'setting': 'download_path', 'type': str, 'text': 'Download Path', 'allowed': []}, ] for m in menu: repeat = True while repeat: i = input(m['text'] + ": ") if i == "": new_profile[m['setting']] = None break if not isinstance(i, m['type']): try: i = int(i) except ValueError: print(" - Allowed options: " + ', '.join(str(x) for x in m['allowed'])) continue if len(m['allowed']) > 0: if i not in m['allowed']: print(" - Allowed options: " + ', '.join(str(x) for x in m['allowed'])) continue new_profile[m['setting']] = i break print("\n") i = input(":: Save these settings? [y|N] ") if i.lower() != "y": return logger.info("Operation cancelled. No changes saved.") else: self.db.create_profile(new_profile) logger.debug(f"New profile created with the following configuration: {new_profile}") def delete(self): profile_config = self.db.get_profile(self.profile_name) if not profile_config: return logger.error(f"Profile {self.profile_name} not found") if profile_config['id'] == 1: return logger.info("You cannot delete the default profile.") i = input(f":: Remove the profile '{self.profile_name}'? [y|N] ") if i.lower() == "y": self.db.delete_profile(self.profile_name) return logger.info("Profile " + self.profile_name + " deleted.") else: return logger.info("Operation cancelled") def show(self): if not self.profile_name: profile = self.db.get_all_profiles() self.print_header(f"Showing all profiles") else: profile = [self.db.get_profile(self.profile_name)] self.print_header(f"Showing profile '{profile[0]['name']}' (Profile ID: {profile[0]['id']})") if len(profile) == 0: return logger.error(f"Profile {self.profile_name} not found") print("{:<10} {:<40} {:<8} {:<8} {:<8} {:<25} " "{:<20} {:<20} {:<20}".format('Name', 'Email', 'Alerts', 'Bitrate', 'Type', 'Plex Base URL', 'Plex Token', 'Plex Library', 'Download Path')) for u in profile: id, name, email, alerts, bitrate, rtype, url, token, \ lib, dl_path = [x if x is not None else '' for x in u.values()] print("{:<10} {:<40} {:<8} {:<8} {:<8} {:<25} " "{:<20} {:<20} {:<20}".format(name, email, alerts, bitrate, rtype, url, token, lib, dl_path)) print("") def clear(self): profile = self.db.get_profile(self.profile_name) self.print_header(f"Configuring '{profile['name']}' (Profile ID: {profile['id']})") if not profile: return logger.error(f"Profile {self.profile_name} not found") for value in profile: if value in ["id", "name"]: continue profile[value] = None self.db.update_profile(profile) logger.info("All values have been cleared.") ================================================ FILE: deemon/cmd/refresh.py ================================================ import logging import re import time from concurrent.futures import ThreadPoolExecutor from datetime import datetime, timedelta from tqdm import tqdm from deemon.cmd.download import QueueItem, Download from deemon.core import db, api, notifier, common from deemon.core.config import Config as config from deemon.utils import dates, ui, performance logger = logging.getLogger(__name__) class Refresh: def __init__(self, time_machine: datetime = None, skip_download: bool = False, ignore_filters: bool = False, active_api=None): self.db = db.Database() self.refresh_date = datetime.now() self.max_refresh_date = None self.api = active_api or api.PlatformAPI() self.new_releases = [] self.new_releases_alert = [] self.new_playlist_releases = [] self.time_machine = time_machine self.total_new_releases = 0 self.queue_list = [] self.skip_download = skip_download self.download_all = ignore_filters self.seen = None if self.time_machine: logger.info(f":: Time Machine active: {datetime.strftime(self.time_machine, '%b %d, %Y')}!") config._CONFIG['new_releases']['release_max_age'] = 0 if not self.waiting_for_refresh(): self.db.remove_specific_releases({'tm_date': str(self.time_machine)}) self.db.commit() @staticmethod def debugger(message: str, payload = None): if config.debug_mode(): if not payload: payload = "" logger.debug(f"DEBUG_MODE: {message} {str(payload)}") def remove_existing_releases(self, payload: dict, seen: dict) -> list: """ Return list of releases that have not been stored in the database """ new_releases = [] if payload.get('artist_id'): seen_releases = seen if seen_releases: seen_releases = [v for x in seen_releases for k, v in x.items() if not x.get('future_release', 0)] new_releases = [x for x in payload['releases'] if type(x) == dict for k, v in x.items() if k == "id" and v not in seen_releases] return new_releases return [x for x in payload['releases']] if payload.get('tracks'): playlist_id = payload['id'] seen_releases = self.db.get_playlist_tracks(playlist_id) if seen_releases: seen_releases = [v for x in seen_releases for k, v in x.items()] new_releases = [x for x in payload['tracks'] if type(x) == dict for k, v in x.items() if k == "id" and v not in seen_releases] return new_releases return [x for x in payload['tracks']] return new_releases def filter_artist_releases(self, payload: dict): """ Inspect artist releases and decide what to do with each release """ self.debugger(f"{payload['artist_name']} has {len(payload['releases'])} new releases") for release in payload['releases']: release['artist_id'] = payload['artist_id'] release['artist_name'] = payload['artist_name'] release['bitrate'] = payload['bitrate'] or config.bitrate() release['download_path'] = payload['download_path'] or config.download_path() release['future'] = self.is_future_release(release['release_date']) release['alerts'] = payload['alerts'] if release['explicit_lyrics'] != 1: release['explicit_lyrics'] = 0 self.append_database_release(release) if release['future']: continue if not common.exclude_filtered_versions([{'title': release['title']}]): # exclude_filtered_versions returns empty list if excluded continue explicit_album_id = self.explicit_id(release['title'], payload['releases']) if explicit_album_id: if explicit_album_id == release['id']: logger.debug(f"An explicit release was found for {release['title']}") else: continue if self.download_all: self.queue_release(release) continue if not self.allowed_record_type(payload['record_type'], release['record_type']): logger.debug(f"Record type \"{release['record_type']}\" has been filtered out, skipping release " f"{release['id']}") continue if self.release_too_old(release['release_date']): logger.debug(f"Release {release['id']} is too old, skipping it.") continue if not payload['refreshed'] and not self.time_machine: continue self.queue_release(release) def append_database_release(self, new_release: dict): self.new_releases.append(new_release) @staticmethod def explicit_id(release_title: str, payload: list): for release in payload: if release['title'] == release_title: if release['explicit_lyrics'] == 1: return release['id'] def release_too_old(self, release_date: str): release_date_dt = dates.str_to_datetime_obj(release_date) if self.time_machine: if release_date_dt <= self.time_machine: self.debugger(f"Release date \"{release_date}\" is older than TIME_MACHINE ({str(dates.ui_date(self.time_machine))})") return True if config.release_max_age(): if release_date_dt < (self.refresh_date - timedelta(config.release_max_age())): self.debugger(f"Release date \"{release_date}\" is older than RELEASE_MAX_AGE ({config.release_max_age()} day(s))") return True @staticmethod def is_future_release(release_date: str): """ Return 1 if release date is in future, otherwise return 0 """ release_date_dt = dates.str_to_datetime_obj(release_date) if release_date_dt > datetime.now(): return 1 else: return 0 @staticmethod def allowed_record_type(artist_rec_type, release_rec_type: str): """ Compare actual record_type against allowable """ if artist_rec_type: if artist_rec_type == release_rec_type or artist_rec_type == "all": return True else: return elif config.record_type() == release_rec_type: return True elif config.record_type() == "all": return True def queue_release(self, release: dict): """ Add release to download queue and create alert notification """ # Create notification of release if per-artist is set to True if release['alerts'] is not False and config.alerts(): self.create_notification(release) self.queue_list.append(QueueItem(release_full=release)) def filter_playlist_releases(self, payload: dict): self.debugger(f"Filtering {len(payload['tracks'])} tracks for playlist {payload['title']}") if len(payload['tracks']): for track in payload['tracks']: new_track = track.copy() new_track['playlist_id'] = payload['id'] self.new_playlist_releases.append(new_track) if payload['refreshed'] == 0: return queue_obj = QueueItem(playlist=payload, bitrate=payload['bitrate'], download_path=payload['download_path']) self.debugger("QueuePlaylistItem", queue_obj) self.queue_list.append(queue_obj) def waiting_for_refresh(self): playlists = self.db.get_unrefreshed_playlists() artists = self.db.get_unrefreshed_artists() if len(playlists) or len(artists): return {'artists': artists, 'playlists': playlists} def prep_payload(self, p): if len(p): p['releases'] = self.remove_existing_releases(p, self.seen) self.filter_artist_releases(p) else: logger.debug("No payload provided") def run(self, artists: list = None, playlists: list = None): if config.check_account_status(): if self.api.account_type == "free" and config.bitrate() != "128": notification = notifier.Notify() notification.expired_arl() return logger.error(" [X] ARL expired? Deezer account only allows low" " quality. If you wish to download " "anyway, set `check_account_status` " "to False in the config.") if artists: self.debugger("ManualRefresh", artists) monitored_artists = [x for x in (self.db.get_monitored_artist_by_name(a) for a in artists) if x] if not len(monitored_artists): return logger.warning("Specified artist(s) were not found") api_result = self.get_release_data({'artists': monitored_artists}) elif playlists: self.debugger("ManualRefresh", playlists) monitored_playlists = [x for x in (self.db.get_monitored_playlist_by_name(p) for p in playlists) if x] if not len(monitored_playlists): return logger.warning("Specified playlist(s) were not found") api_result = self.get_release_data({'playlists': monitored_playlists}) else: waiting = self.waiting_for_refresh() if waiting: logger.debug(f"There are {len(waiting['playlists'])} playlist(s) and " f"{len(waiting['artists'])} artist(s) waiting to be refreshed.") api_result = self.get_release_data(waiting) else: self.debugger("FullRefresh") monitored_playlists = self.db.get_all_monitored_playlists() monitored_artists = self.db.get_all_monitored_artists() if not len(monitored_playlists) and not len(monitored_artists): return logger.warning("No artists found to refresh") api_result = self.get_release_data({'artists': monitored_artists, 'playlists': monitored_playlists}) if len(api_result): self.seen = self.db.get_artist_releases() payload_container = tqdm(api_result['artists'], total=len(api_result['artists']), desc=f"Scanning release data for new releases...", ascii=" #", bar_format=ui.TQDM_FORMAT) for payload in payload_container: self.prep_payload(payload) playlist_monitor_artists = [] for payload in api_result['playlists']: if payload and len(payload): self.seen = self.db.get_playlist_tracks(payload['id']) payload['tracks'] = self.remove_existing_releases(payload, self.seen) self.filter_playlist_releases(payload) if payload['monitor_artists']: logger.debug(f"Artists from this playlist ({payload['id']}) are to be monitored!") for track in payload['tracks']: playlist_monitor_artists.append(track['artist_id']) playlist_monitor_artists = list(set(playlist_monitor_artists)) if self.skip_download: logger.info(f" [!] You have opted to skip downloads, clearing {len(self.queue_list):,} item(s) from queue...") self.queue_list.clear() self.new_releases_alert.clear() if len(self.queue_list): dl = Download(active_api=self.api) dl.download_queue(self.queue_list) if len(self.new_playlist_releases) or len(self.new_releases): if len(self.new_playlist_releases): logger.debug("Updating playlist releases in database...") self.db.add_new_playlist_releases(self.new_playlist_releases) if len(self.new_releases): logger.debug("Updating artist releases in database...") self.db.add_new_releases(self.new_releases) self.db.commit() self.db_stats() performance.operation_time(config.get('start_time')) logger.info("Database is up-to-date.") else: self.db_stats() performance.operation_time(config.get('start_time')) logger.info("Database is up-to-date. No new releases were found.") if len(self.new_releases_alert) > 0: notification = notifier.Notify(self.new_releases_alert) notification.send() if playlist_monitor_artists: print("") logger.info(":: New artists to monitor, stand by...") time.sleep(2) from deemon.cmd.monitor import Monitor monitor = Monitor(active_api=self.api) monitor.artist_ids(playlist_monitor_artists) def db_stats(self): artists = len(self.db.get_all_monitored_artist_ids()) playlists = len(self.db.get_all_monitored_playlist_ids()) releases = len(self.db.get_artist_releases()) future = len(self.db.get_future_releases()) print("") print(f"+ Artists monitored: {artists:,}") print(f"+ Playlists monitored: {playlists:,}") print(f"+ Releases seen: {releases:,}") print(f"+ Pending future releases: {future:,}") print("") def get_release_data(self, to_refresh: dict) -> dict: """ Generate a list of dictionaries containing artist (DB) and release (API) information. """ api_result = {'artists': [], 'playlists': []} logger.debug(f"Standby, starting refresh...") if to_refresh.get('playlists') and len(to_refresh.get('playlists')): logger.debug("Fetching playlist track data...") self.debugger("SpawningThreads", self.api.max_threads) with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: api_result['playlists'] = list( tqdm(ex.map(self.api.get_playlist_tracks, to_refresh['playlists']), total=len(to_refresh['playlists']), desc=f"Fetching playlist track data for " f"{len(to_refresh['playlists'])} playlist(s), " "please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT) ) if to_refresh.get('artists') and len(to_refresh['artists']): logger.debug("Fetching artist release data...") self.debugger("SpawningThreads", self.api.max_threads) with ThreadPoolExecutor(max_workers=self.api.max_threads) as ex: api_result['artists'] = list( tqdm(ex.map(self.api.get_artist_albums, to_refresh['artists']), total=len(to_refresh['artists']), desc=f"Fetching artist release data for {len(to_refresh['artists']):,} artist(s), please wait...", ascii=" #", bar_format=ui.TQDM_FORMAT) ) return api_result def create_notification(self, release: dict): for days in self.new_releases_alert: for key in days: if key == "release_date": if release['release_date'] in days[key]: days["releases"].append( { 'artist': release['artist_name'], 'album': release['title'], 'cover': release['cover_big'], 'url': release['link'], 'track_num': release.get('nb_tracks', None), 'record_type': release['record_type'], } ) return self.new_releases_alert.append( { 'release_date': release['release_date'], 'releases': [ { 'artist': release['artist_name'], 'album': release['title'], 'cover': release['cover_big'], 'url': release['link'], 'track_num': release.get('nb_tracks', None), 'record_type': release['record_type'], } ] } ) ================================================ FILE: deemon/cmd/rollback.py ================================================ import logging from deemon.core.db import Database from deemon.utils import dates logger = logging.getLogger(__name__) db = Database() def view_transactions(): transactions = db.get_transactions() if not transactions: return logger.info("No transactions are available to be rolled back.") for i, transaction in enumerate(transactions, start=1): release_id = [] artist_names = [] playlist_titles = [] for k, v in transaction.items(): if (k == "releases" or k == "playlist_tracks") and transaction[k]: for item in transaction[k]: for key, val in item.items(): if key == "album_id" or key == "track_id": if item[key] not in release_id: release_id.append(item[key]) if k == "monitor" and transaction[k]: if transaction[k] not in artist_names: artist_names = [x['artist_name'] for x in transaction[k]] if k == "playlists" and transaction[k]: if transaction[k] not in playlist_titles: playlist_titles.append(transaction[k][0]['title']) release_id_len = len(release_id) if release_id_len == 1: release_text = f" and {release_id_len} release" elif release_id_len > 1: release_text = f" and {release_id_len} releases" else: release_text = "" if len(artist_names) > 1: if len(playlist_titles) > 1: playlist_text = f", {len(playlist_titles)} playlists" elif len(playlist_titles) == 1: playlist_text = f", {len(playlist_titles)} playlist" else: playlist_text = "" output_text = f"Added {artist_names[0]} + {len(artist_names) - 1} artist(s){playlist_text}{release_text}" elif len(artist_names) == 1: if len(playlist_titles) > 1: playlist_text = f", {len(playlist_titles)} playlists" elif len(playlist_titles) == 1: playlist_text = f", {len(playlist_titles)} playlist" else: playlist_text = "" output_text = f"Added {artist_names[0]}{playlist_text}{release_text}" else: if len(playlist_titles) > 1: output_text = f"Added {playlist_titles[0]} + {len(playlist_titles) - 1}{release_text}" elif len(playlist_titles) == 1: output_text = f"Added {playlist_titles[0]}{release_text}" else: output_text = f"Added {release_text[4:]}" print(f"{i}. {dates.get_friendly_date(transaction['timestamp'])} - {output_text}") rollback = None while rollback not in range(len(transactions)): rollback = input("\nSelect specific refresh to rollback (or press Enter to exit): ") if rollback == "": return try: rollback = int(rollback) - 1 except ValueError: logger.error("Invalid input") rollback = transactions[rollback]['id'] logger.debug(f"Rolling back transaction {rollback}") db.rollback_refresh(rollback) def rollback_last(i: int): db.rollback_last_refresh(i) logger.info(f"Rolled back the last {i} transaction(s).") ================================================ FILE: deemon/cmd/search.py ================================================ import logging import sys from deezer import Deezer from deemon.cmd import download from deemon.cmd import monitor as mon from deemon.core import db, api from deemon.core.config import Config as config from deemon.utils import dates logger = logging.getLogger(__name__) class Search: def __init__(self, active_api=None): self.api = active_api or api.PlatformAPI() self.artist_id: int = None self.artist: str = None self.choices: list = [] self.status_message: str = None self.queue_list = [] self.select_mode = False self.explicit_only = False self.search_results: str = None self.sort: str = "release_date" self.filter: str = None self.desc: bool = True self.gte_year = None self.lte_year = None self.eq_year = None self.db = db.Database() self.dz = Deezer() @staticmethod def truncate_artist(name: str): if len(name) > 45: return name[0:40] + "..." return name def get_latest_release(self, artist_id: int): try: all_releases = self.dz.api.get_artist_albums(artist_id)['data'] sorted_releases = sorted(all_releases, key=lambda x: x['release_date'], reverse=True) latest_release = sorted_releases[0] except IndexError: return " - No releases found" return f" - Latest release: {latest_release['title']} ({dates.get_year(latest_release['release_date'])})" def display_monitored_status(self, artist_id: int): if self.db.get_monitored_artist_by_id(artist_id): return "[M] " return " " @staticmethod def has_duplicate_artists(name: str, artist_dicts: dict): names = [x['name'] for x in artist_dicts if x['name'] == name] if len(names) > 1: return True def show_mini_queue(self): num_queued = len(self.queue_list) if num_queued > 0: return f" ({str(num_queued)} Queued)" return "" def search_menu(self, query: str = None): exit_search: bool = False quick_search: bool = False while exit_search is False: self.clear() print("deemon Interactive Search Client\n") if len(self.queue_list) > 0: self.display_options(options="(d) Download Queue (Q) Show Queue") if query: search_query = query query = None else: search_query = input(f":: Enter an artist to search for{self.show_mini_queue()}: ") if search_query == "exit": if self.exit_search(): sys.exit() continue elif search_query == "d": if len(self.queue_list) > 0: self.start_queue() continue elif search_query == "Q": if len(self.queue_list) > 0: self.queue_menu() else: self.status_message = "Queue is empty" continue elif search_query == "": continue self.clear() print("deemon Interactive Search Client\n") self.search_results = self.api.search_artist(search_query, config.query_limit()) if not self.search_results['results']: self.status_message = "No results found for: " + search_query continue smart_search = None if config.smart_search(): for result in self.search_results['results']: if result['name'].lower() == search_query.lower(): if not smart_search: smart_search = result else: smart_search = None break if smart_search: self.artist = smart_search['name'] album_selected = self.album_menu(smart_search) if album_selected: return [album_selected] artist_selected = self.artist_menu(self.search_results['query'], self.search_results['results'], quick_search) if artist_selected: return [artist_selected] def queue_menu_options(self): ui_options = ("(d) Download Queue (c) Clear Queue (b) Back") self.display_options(options=ui_options) def artist_menu(self, query: str, results: dict, artist_only: bool = False): exit_artist: bool = False while exit_artist is False: self.clear() print("Search results for artist: " + query) for idx, option in enumerate(results, start=1): print(f"{self.display_monitored_status(option['id'])}{idx}. {self.truncate_artist(option['name'])}") if self.has_duplicate_artists(option['name'], results): print(self.get_latest_release(option['id'])) print(" - Artist ID: " + str(option['id'])) if not option.get('nb_album'): option['nb_album'] = self.dz.api.get_artist(option['id'])['nb_album'] print(" - Total releases: " + str(option['nb_album'])) self.status_message = "Duplicate artists found" # TODO make options smarter/modular if len(self.queue_list) > 0: self.display_options(options="(b) Back (d) Download Queue (Q) Show Queue") else: self.display_options(options="(b) Back") response = input(f":: Please choose an option or type 'exit'{self.show_mini_queue()}: ") if response == "d": if len(self.queue_list) > 0: self.start_queue() continue elif response == "Q": if len(self.queue_list) > 0: self.queue_menu() else: self.status_message = "Queue is empty" continue elif response == "b": break elif response == "exit": if self.exit_search() and not artist_only: sys.exit() else: return elif response == "": continue try: response = int(response) except ValueError: self.status_message = f"Invalid selection: {response}" else: response = response - 1 if response in range(len(results)): self.artist = results[response]['name'] if artist_only: self.clear() return results[response] self.album_menu(results[response]) else: self.status_message = f"Invalid selection: {response}" continue def get_filtered_year(self): if self.gte_year and self.lte_year: return f"{self.gte_year} - {self.lte_year}" elif self.gte_year: return f">={self.gte_year}" elif self.lte_year: return f"<={self.lte_year}" elif self.eq_year: return f"{self.eq_year}" else: return "All" def album_menu_header(self, artist: str): filter_text = "All" if not self.filter else self.filter.title() filter_year = self.get_filtered_year() if self.explicit_only: filter_text = filter_text + " (Explicit Only)" desc_text = "desc" if self.desc else "asc" sort_text = self.sort.replace("_", " ").title() + " (" + desc_text + ")" print("Discography for artist: " + artist) print("Filter: " + filter_text + " | Sort: " + sort_text + " | Year: " + filter_year + "\n") def album_menu_options(self, monitored): print("") if not monitored: monitor_opt = "(m) Monitor" else: monitor_opt = "(m) Stop Monitoring" ui_filter = "Filters: (*) All (a) Albums (e) EP (s) Singles - (E) Explicit (r) Reset" ui_sort = " Sort: (y) Release Date (desc) (Y) Release Date (asc) (t) Title (desc) (T) Title (asc)" ui_mode = " Mode: (S) Toggle Select" ui_options = ("(b) Back (d) Download Queue (Q) Show Queue (f) Queue Filtered " f"{monitor_opt}") self.display_options(ui_filter, ui_sort, ui_mode, ui_options) @staticmethod def explicit_lyrics(is_explicit): if is_explicit > 0: return f"[E]" else: return f" " def item_selected(self, id): if self.select_mode: if [x for x in self.queue_list if x.album_id == id or x.track_id == id]: return "[*] " else: return "[ ] " else: return " " def show_mode(self): if self.select_mode: return "[SELECT] " return "" def album_menu(self, artist: dict): exit_album_menu: bool = False # Rewrite DICT to follow old format used by get_artist_albums artist_tmp = {'artist_id': artist['id'], 'artist_name': artist['name']} artist_albums = self.api.get_artist_albums(artist_tmp)['releases'] while exit_album_menu is False: self.clear() self.album_menu_header(artist['name']) filtered_choices = self.filter_choices(artist_albums) for idx, album in enumerate(filtered_choices, start=1): print(f"{self.explicit_lyrics(album['explicit_lyrics'])} {self.item_selected(album['id'])}{idx}. ({dates.get_year(album['release_date'])}) " f"{album['title']}") monitored = self.db.get_monitored_artist_by_id(artist['id']) self.album_menu_options(monitored) prompt = input(f":: {self.show_mode()}Please choose an option or type 'exit'{self.show_mini_queue()}: ") if prompt == "a": self.filter = "album" elif prompt == "e": self.filter = "ep" elif prompt == "s": self.filter = "single" elif prompt == "*": self.filter = None elif prompt == "E": self.explicit_only ^= True elif prompt == "r": self.filter = None self.explicit_only = False self.sort = "release_date" self.desc = True self.gte_year = None self.lte_year = None self.eq_year = None elif prompt.startswith(">="): self.eq_year = None self.gte_year = int(prompt[2:]) elif prompt.startswith("<="): self.eq_year = None self.lte_year = int(prompt[2:]) elif prompt.startswith("="): self.lte_year = None self.gte_year = None self.eq_year = int(prompt[1:]) elif prompt == "y": self.sort = "release_date" self.desc = True elif prompt == "Y": self.sort = "release_date" self.desc = False elif prompt == "t": self.sort = "title" self.desc = True elif prompt == "T": self.sort = "title" self.desc = False elif prompt == "S": self.select_mode ^= True elif prompt == "m": if monitored: stop = True else: stop = False record_type = self.filter or config.record_type() self.clear() monitor = mon.Monitor() monitor.set_config(None, None, record_type, None) monitor.set_options(stop, False, False) monitor.artist_ids([artist['id']]) elif prompt == "f": if len(filtered_choices) > 0: for item in filtered_choices: self.send_to_queue(item) else: self.status_message = "No items to add" elif prompt == "d": if len(self.queue_list) > 0: self.start_queue() elif prompt == "Q": if len(self.queue_list) > 0: self.queue_menu() else: self.status_message = "Queue is empty" elif prompt == "b": break elif prompt == "": self.status_message = "Hint: to exit, type 'exit'!" continue elif prompt == "exit": if self.exit_search(): sys.exit() else: try: selected_index = (int(prompt) - 1) except ValueError: self.status_message = "Invalid filter, sort or option provided" continue except IndexError: self.status_message = "Invalid selection, please choose from above" continue if selected_index in range(len(filtered_choices)): if self.select_mode: selected_item = filtered_choices[selected_index] self.send_to_queue(selected_item) continue else: self.track_menu(filtered_choices[selected_index]) else: self.status_message = "Invalid selection, please choose from above" continue def track_menu_options(self): ui_options = ("(b) Back (d) Download Queue (Q) Show Queue") self.display_options(options=ui_options) def track_menu_header(self, album): print("deemon Interactive Search Client") print(f"Artist: {self.artist} | Album: {album['title']}\n") def track_menu(self, album): exit_track_menu: bool = False track_list = self.dz.api.get_album_tracks(album['id'])['data'] self.select_mode = True while not exit_track_menu: self.clear() self.track_menu_header(album) for idx, track in enumerate(track_list, start=1): print(f"{self.item_selected(track['id'])}{idx}. {track['title']}") self.track_menu_options() prompt = input(f":: {self.show_mode()}Please choose an option or type 'exit'{self.show_mini_queue()}: ") if prompt == "d": if len(self.queue_list) > 0: self.start_queue() else: self.status_message = "Queue is empty" elif prompt == "Q": if len(self.queue_list) > 0: self.queue_menu() else: self.status_message = "Queue is empty" elif prompt == "b": self.select_mode = False break elif prompt == "": self.status_message = "Hint: to exit, type 'exit'!" continue elif prompt == "exit": if self.exit_search(): sys.exit() else: try: selected_index = (int(prompt) - 1) except ValueError: self.status_message = "Invalid filter, sort or option provided" continue except IndexError: self.status_message = "Invalid selection, please choose from above" continue if selected_index in range(len(track_list)): selected_item = track_list[selected_index] selected_item['record_type'] = 'track' self.send_to_queue(selected_item) continue else: self.status_message = "Invalid selection, please choose from above" continue def search_header(self): pass def queue_menu(self): exit_queue_list = False while exit_queue_list is False: self.clear() for idx, q in enumerate(self.queue_list, start=1): if q.album_title: print(f"{idx}. {q.artist_name} - {q.album_title}") else: print(f"{idx}. {q.artist_name} - {q.track_title}") print("") self.queue_menu_options() response = input(f":: Please choose an option or type exit {self.show_mini_queue()}: ") if response == "d": if len(self.queue_list) > 0: self.start_queue() else: self.status_message = "Queue is empty" if response == "c": self.queue_list = [] break if response == "b": break if response == "exit": if self.exit_search(): sys.exit() try: response = int(response) - 1 except ValueError: continue if response in range(len(self.queue_list)): self.queue_list.pop(response) if len(self.queue_list) == 0: break def exit_search(self): if len(self.queue_list) > 0: exit_all = input(":: Quit before downloading queue? [y|N] ") if exit_all.lower() != 'y': return False return True def display_options(self, filter=None, sort=None, mode=None, options=None): if filter: print(filter) if sort: print(sort) if mode: print(mode) if options: print("") print(options) if self.status_message: print("** " + self.status_message + " **") self.status_message = None @staticmethod def clear(): from os import system, name if name == 'nt': _ = system('cls') else: _ = system('clear') def filter_choices(self, choices): apply_filter = [x for x in choices if x['record_type'] == self.filter or self.filter is None] if self.explicit_only: apply_filter = [x for x in apply_filter if x['explicit_lyrics'] > 0] if any([self.gte_year, self.lte_year, self.eq_year]): if self.eq_year: apply_filter = [x for x in apply_filter if dates.get_year(x['release_date']) == self.eq_year] elif self.gte_year and self.lte_year: apply_filter = [x for x in apply_filter if dates.get_year(x['release_date']) >= self.gte_year and dates.get_year(x['release_date']) <= self.lte_year] elif self.gte_year: apply_filter = [x for x in apply_filter if dates.get_year(x['release_date']) >= self.gte_year] elif self.lte_year: apply_filter = [x for x in apply_filter if dates.get_year(x['release_date']) <= self.lte_year] return sorted(apply_filter, key=lambda x: x[self.sort], reverse=self.desc) def start_queue(self): self.clear() dl = download.Download(active_api=self.api) dl.queue_list = self.queue_list download_result = dl.download_queue() self.queue_list.clear() if download_result: self.status_message = "Downloads complete" else: self.status_message = "Downloads failed, please check logs" def send_to_queue(self, item): if item['record_type'] in ['album', 'ep', 'single']: album = { 'id': item['id'], 'title': item['title'], 'link': item['link'], 'artist': { 'name': self.artist } } for i, q in enumerate(self.queue_list): if q.album_id == album['id']: del self.queue_list[i] return self.queue_list.append(download.QueueItem(album=album)) elif item['record_type'] == 'track': track = { 'id': item['id'], 'title': item['title'], 'link': item['link'], 'artist': { 'name': self.artist } } for i, q in enumerate(self.queue_list): if q.track_id == track['id']: del self.queue_list[i] return self.queue_list.append(download.QueueItem(track=track)) else: logger.error("Unknown record type. Please report this to add support:") logger.error(item) ================================================ FILE: deemon/cmd/show.py ================================================ import csv import logging import os import time from pathlib import Path from typing import Union from deemon.core.db import Database from deemon.utils.dates import generate_date_filename logger = logging.getLogger(__name__) class Show: def __init__(self): self.db = Database() def monitoring(self, artist: bool = True, query: str = None, export_csv: bool = False, save_path: Union[str, Path] = None, filter: str = None, hide_header: bool = False, is_id: bool = False, backup: Union[str, Path] = None): def csv_output(line: str): if save_path: output_to_file.append(line) else: print(line) output_to_file = [] if backup: export_csv = True filter = "id" hide_header = True save_path = backup if artist: if query: db_result = self.db.get_monitored_artist_by_name(query) else: db_result = self.db.get_all_monitored_artists() if not db_result: if query: return logger.error("Artist not found: " + str(query)) else: return logger.error("No artists are being monitored") else: if query: if is_id: try: query = int(query) except ValueError: return logger.error(f"Invalid Playlist ID - {query}") db_result = self.db.get_monitored_playlist_by_id(query) else: db_result = self.db.get_monitored_playlist_by_name(query) else: db_result = self.db.get_all_monitored_playlists() if not db_result: if query: return logger.error("Playlist/ID not found: " + str(query)) else: return logger.error("No playlists are being monitored") if artist and query: for key, val in db_result.items(): if val == None: db_result[key] = "-" print("{:<10} {:<35} {:<10} {:<10} {:<10} {:<25}".format('ID', 'Artist', 'Alerts', 'Bitrate', 'Type', 'Download Path')) print("{!s:<10} {!s:<35} {!s:<10} {!s:<10} {!s:<10} {!s:<25}".format(db_result['artist_id'], db_result['artist_name'], db_result['alerts'], db_result['bitrate'], db_result['record_type'], db_result['download_path'])) print("") elif not artist and query: for key, val in db_result.items(): if val == None: db_result[key] = "-" print("{:<15} {:<30} {:<50} {:<10} {:<10} {:<25}".format('ID', 'Title', 'URL', 'Alerts', 'Bitrate', 'Download Path')) print("{!s:<15} {!s:<30} {!s:<50} {!s:<10} {!s:<10} {!s:<25}".format(db_result['id'], db_result['title'], db_result['url'], db_result['alerts'], db_result['bitrate'], db_result['download_path'])) print("") else: if export_csv or save_path: if artist: if not filter: filter = "name,id,bitrate,alerts,type,path" filter = filter.split(',') logger.debug(f"Generating CSV data using filters: {', '.join(filter)}") column_names = ['artist_id' if x == 'id' else x for x in filter] column_names = ['artist_name' if x == 'name' else x for x in column_names] column_names = ['record_type' if x == 'type' else x for x in column_names] column_names = ['download_path' if x == 'path' else x for x in column_names] else: if not filter: filter = "id,title,url,bitrate,alerts,path" filter = filter.split(',') logger.debug(f"Generating CSV data using filters: {', '.join(filter)}") column_names = ['download_path' if x == 'path' else x for x in filter] for column in column_names: if not len([x for x in db_result if column in x.keys()]): logger.warning(f"Unknown filter specified: {column}") column_names.remove(column) if not hide_header: csv_output(','.join(filter)) for artist in db_result: filtered_artists = [] for key, value in artist.items(): if value is None: artist[key] = "" for column in column_names: filtered_artists.append(str(artist[column])) if len(filtered_artists): for i, a in enumerate(filtered_artists): if '"' in a: a = a.replace('"', "'") if ',' in a: filtered_artists[i] = f'"{a}"' csv_output(",".join(filtered_artists)) if output_to_file: if Path(save_path).is_dir(): output_filename = Path(save_path / f"{generate_date_filename('deemon_')}.csv") else: output_filename = Path(save_path) with open(output_filename, 'w', encoding="utf-8") as f: for line in output_to_file: if line == output_to_file[-1]: f.writelines(line) break f.writelines(line + "\n") return logger.info("CSV data has been saved to: " + str(output_filename)) return if artist: db_result = [x['artist_name'] for x in db_result] else: db_result = [x['title'] for x in db_result] if len(db_result) < 10: for artist in db_result: print(artist) else: db_result = self.truncate_long_artists(db_result) try: size = os.get_terminal_size() max_cols = (int(size.columns / 30)) except: max_cols = 5 if max_cols > 5: max_cols = 5 while len(db_result) % max_cols != 0: db_result.append(" ") if max_cols >= 5: for a, b, c, d, e in zip(db_result[0::5], db_result[1::5], db_result[2::5], db_result[3::5], db_result[4::5]): print('{:<30}{:<30}{:<30}{:<30}{:<30}'.format(a, b, c, d, e)) elif max_cols >= 4: for a, b, c, d in zip(db_result[0::4], db_result[1::4], db_result[2::4], db_result[3::4]): print('{:<30}{:<30}{:<30}{:<30}'.format(a, b, c, d)) elif max_cols >= 3: for a, b, c in zip(db_result[0::3], db_result[1::3], db_result[2::3]): print('{:<30}{:<30}{:<30}'.format(a, b, c)) elif max_cols >= 2: for a, b in zip(db_result[0::2], db_result[1::2]): print('{:<30}{:<30}'.format(a, b)) else: for a in db_result: print(a) def playlists(self, csv=False): monitored_playlists = self.db.get_all_monitored_playlists() for p in monitored_playlists: print(f"{p[1]} ({p[2]})") @staticmethod def truncate_long_artists(all_artists): for idx, artist in enumerate(all_artists): if len(artist) > 25: all_artists[idx] = artist[:22] + "..." all_artists[idx] = artist return all_artists def releases(self, days, future): if future: future_releases = self.db.get_future_releases() future_release_list = [x for x in future_releases] if len(future_release_list) > 0: logger.info(f"Future releases:") print("") future_release_list.sort(key=lambda x: x['album_release'], reverse=True) for release in future_release_list: print('+ [%-10s] %s - %s' % (release['album_release'], release['artist_name'], release['album_name'])) else: logger.info("No future releases have been detected") else: seconds_per_day = 86400 days_in_seconds = (days * seconds_per_day) now = int(time.time()) back_date = (now - days_in_seconds) releases = self.db.show_new_releases(back_date, now) release_list = [x for x in releases] if len(release_list) > 0: logger.info(f"New releases found within last {days} day(s):") print("") release_list.sort(key=lambda x: x['album_release'], reverse=True) for release in release_list: print('+ [%-10s] %s - %s' % (release['album_release'], release['artist_name'], release['album_name'])) else: logger.info(f"No releases found in the last {days} day(s)") ================================================ FILE: deemon/cmd/upgradelib.py ================================================ import sys import time import logging from datetime import timedelta from pathlib import Path from mutagen.easyid3 import EasyID3 from itertools import groupby from operator import itemgetter from deezer import Deezer from concurrent.futures import ThreadPoolExecutor from unidecode import unidecode from tqdm import tqdm from deemon.core.common import exclude_filtered_versions from deemon.core.config import Config as config logger = logging.getLogger(__name__) dz = Deezer() LIBRARY_ROOT = None ALBUM_ONLY = None ALLOW_EXCLUSIONS = None # TODO - Add an 'exclusions' key to albums/tracks for count # TODO - to improve album title matching, extract all a-zA-Z0-9 and compare (remove special chars) library_metadata = [] performance = { 'startID3': 0, 'endID3': 0, 'completeID3': 0, 'startAPI': 0, 'endAPI': 0, 'completeAPI': 0 } class Performance: def __init__(self): self.startID3 = 0 self.endID3 = 0 self.completeID3 = 0 self.startAPI = 0 self.endAPI = 0 self.completeAPI = 0 def start(self, module: str): if module == 'ID3': self.startID3 = time.time() elif module == 'API': self.startAPI = time.time() def end(self, module: str): if module == 'ID3': self.endID3 = time.time() self.completeID3 = self.endID3 - self.startID3 elif module == 'API': self.endAPI = time.time() self.completeAPI = self.endAPI - self.startAPI def read_metadata(file): metadata = { 'abs_path': file, 'rel_path': str(file).replace(LIBRARY_ROOT, ".."), 'error': None } try: _audio = EasyID3(file) # Remove featured artists from artist tag metadata['artist'] = _audio['artist'][0].split("/")[0].strip() # Remove special character replacement for search query metadata['album'] = _audio['album'][0].replace("_", " ").strip() metadata['title'] = _audio['title'][0].strip() except Exception as e: metadata['error'] = e return metadata def get_time_from_secs(secs): td_str = str(timedelta(seconds=secs)) x = td_str.split(":") x[2] = x[2].split(".")[0] if x[0] != "0": friendly_time = f"{x[0]} Hours {x[1]} Minutes {x[2]} Seconds" elif x[1] != "00": friendly_time = f"{x[1]} Minutes {x[2]} Seconds" elif x[2] == "00": friendly_time = f"Less than 1 second" else: friendly_time = f"{x[2]} Seconds" return friendly_time def invalid_metadata(track: dict) -> bool: if not all([track['artist'], track['album'], track['title']]): return True else: return False def get_artist_api(name: str) -> list: """ Get list of artists with exact name matches from API """ artist_api = dz.gw.search(name)['ARTIST']['data'] artist_matches = [] for artist in artist_api: if artist['ART_NAME'].lower() == name.lower(): artist_matches.append(artist) return artist_matches def get_artist_discography_api(artist_name, artist_id) -> list: """ Get list of albums with exact name matches from API """ album_search = dz.gw.search(artist_name)['ALBUM']['data'] album_gw = dz.gw.get_artist_discography(artist_id)['data'] album_api = dz.api.get_artist_albums(artist_id)['data'] albums = [] for album in album_api: if album['record_type'] == 'single': album['record_type'] = '0' elif album['record_type'] == 'album': album['record_type'] = '1' elif album['record_type'] == 'compilation': album['record_type'] = '2' elif album['record_type'] == 'ep': album['record_type'] = '3' if album['explicit_lyrics']: album['explicit_lyrics'] = '1' else: album['explicit_lyrics'] = '0' alb = { 'ALB_ID': str(album['id']), 'ALB_TITLE': album['title'], 'EXPLICIT_LYRICS': album['explicit_lyrics'], 'TYPE': album['record_type'] } albums.append(alb) for album in album_gw: if album['ALB_ID'] not in [x['ALB_ID'] for x in albums]: albums.append(album) for album in album_search: if album['ART_ID'] == artist_id: if album['ALB_ID'] not in [x['ALB_ID'] for x in albums]: # Album returned via Search is missing EXPLICIT_LYRICS key if not album.get('EXPLICIT_LYRICS'): album['EXPLICIT_LYRICS'] = '0' albums.append(album) return albums def get_album_tracklist_api(album_id: str) -> list: """ Get tracklist for album based on album_id """ tracklist_api = dz.gw.get_album_tracks(album_id) return tracklist_api def retrieve_track_ids_per_artist(discography: tuple): artist = discography[0] albums = discography[1] # TODO - Need to implement this duplicate_artists = False found_artist = False track_ids = [] album_ids = [] api_artists = get_artist_api(artist) tqdm.write(f"Getting track IDs for tracks by \"{artist}\"") if len(api_artists): if len(api_artists) > 1: tqdm.write(f"Duplicate artists detected for \"{artist}\"") duplicate_artists = True for api_artist in api_artists: if found_artist: tqdm.write("Found correct artist, skipping duplicates") break if duplicate_artists: tqdm.write(f"Searching with: {api_artist['ART_NAME']} ({api_artist['ART_ID']})") discog = get_artist_discography_api(api_artist['ART_NAME'], api_artist['ART_ID']) for album, tracks in groupby(albums, key=itemgetter('album')): # Convert itertools.groupby to list so we can use it more than once tracks = [track for track in tracks] api_album_matches = [alb for alb in discog if alb['ALB_TITLE'].lower() == album.lower()] if ALLOW_EXCLUSIONS: filtered_album_matches = exclude_filtered_versions(api_album_matches) api_album = get_preferred_album(filtered_album_matches, len(tracks)) else: api_album = get_preferred_album(api_album_matches, len(tracks)) if ALBUM_ONLY: if api_album: album_ids.append(api_album) continue else: album_ids.append( { 'artist': artist, 'title': album, 'info': "Album not found" } ) continue if api_album: tracklist = get_album_tracklist_api(api_album['ALB_ID']) for track in tracks: track_variations = [track['title'].lower(), unidecode(track['title']).lower()] for i, track_api in enumerate(tracklist, start=1): if track_api['SNG_TITLE'].lower() in track_variations: found_artist = True track['id'] = track_api['SNG_ID'] track_ids.append(track) break elif f"{track_api['SNG_TITLE']} {track_api.get('VERSION', '')}".lower() in track_variations: found_artist = True track['id'] = track_api['SNG_ID'] track_ids.append(track) break if i == len(tracklist): track['info'] = "Track not found" tqdm.write(f"{track['info']}: {track['title']}") track_ids.append(track) break else: if duplicate_artists: info = f"Album not found under artist ID {api_artist['ART_ID']}" else: info = f"Album not found" tqdm.write(f"{info}: {album}") for track in tracks: track['info'] = info track_ids.append(track) else: tqdm.write(f"Artist not found: {artist}") for album, tracks in groupby(albums, key=itemgetter('album')): for track in tracks: track['info'] = "Artist not found" track_ids.append(track) if ALBUM_ONLY: return album_ids else: return track_ids def get_preferred_album(api_albums: list, num_tracks: int): """ Return preferred album order based on config.prefer_explicit() """ preferred_album = None if num_tracks < 4: preferred_album = [album for album in api_albums if album['EXPLICIT_LYRICS'] == '1' and album['TYPE'] == '0'] if not preferred_album: preferred_album = [album for album in api_albums if album['TYPE'] == '0'] if num_tracks >= 4 or not preferred_album: preferred_album = [album for album in api_albums if album['EXPLICIT_LYRICS'] == '1' and album['TYPE'] in ['1', '2', '3']] if not preferred_album: preferred_album = [album for album in api_albums if album['TYPE'] in ['1', '2', '3']] if preferred_album: return preferred_album[0] def get_preferred_track_id(title: str, tracklist: list): """ Return preferred track ID by comparing against title of local track """ track_id = None for track in tracklist: if track.get('VERSION'): api_title = f"{track['SNG_TITLE']} {track['VERSION']}".lower() if title.lower() == api_title: return track['SNG_ID'] else: if track['SNG_TITLE'].lower() == title.lower(): track_id = track['SNG_ID'] return track_id def upgrade(library, output, albums=False, exclusions=False): global ALBUM_ONLY global ALLOW_EXCLUSIONS global LIBRARY_ROOT ALBUM_ONLY = albums ALLOW_EXCLUSIONS = exclusions LIBRARY_ROOT = library output_ids = Path(output) / "library_upgrade_ids.txt" output_log = Path(output) / "library_upgrade.log" perf = Performance() logger.info("Scanning library, standby...") logger.debug(f"Library path: {LIBRARY_ROOT}") library_files = Path(LIBRARY_ROOT).glob("**/*.mp3") files = [file for file in library_files if not file.name.startswith(".")] files.sort() if files: print(f"Found {len(files)} MP3 files") else: print("No MP3 files found") sys.exit() perf.start('ID3') with ThreadPoolExecutor(10) as executor: library_metadata = list( tqdm( executor.map(read_metadata, files), total=(len(files)), desc="Reading metadata", ) ) perf.end('ID3') library_metadata_errors = [file for file in library_metadata if file.get('error')] library_metadata = [file for file in library_metadata if not file.get('error')] artists = sorted(library_metadata, key=itemgetter('artist')) artist_list = [(artist, list(albums)) for artist, albums in groupby(artists, key=itemgetter('artist'))] perf.start('API') with ThreadPoolExecutor(20) as executor: result = list( tqdm( executor.map(retrieve_track_ids_per_artist, artist_list), total=len(artist_list), desc="Processing tracks by artist" ) ) if ALBUM_ONLY: album_result = result track_result = [] else: track_result = result album_result = [] perf.end('API') albums = [album for artist in album_result for album in artist] album_ids = [album['ALB_ID'] for album in albums if album.get('ALB_ID')] album_not_found = [album for album in albums if not album.get('ALB_ID')] tracks = [track for artist in track_result for track in artist] track_ids = [track['id'] for track in tracks if track.get('id') and not track.get('error')] track_not_found = [track for track in tracks if not track.get('id') and not track.get('error')] # TODO move this to function for reuse in f.write() below print(f"Found: {len(track_ids)} | Not Found: {len(track_not_found)} | " f"Errors: {len(library_metadata_errors)} | Total: {len(files)}\n\n") print(f"Time to read metadata: {get_time_from_secs(perf.completeID3)}") print(f"Time to retrieve API data: {get_time_from_secs(perf.completeAPI)}\n\n") if (ALBUM_ONLY and album_ids) or track_ids: with open(output_ids, "w") as f: if ALBUM_ONLY and album_ids: f.write(', '.join(album_ids)) elif track_ids: f.write(', '.join(track_ids)) with open(output_log, "w") as f: if ALBUM_ONLY: f.write(f"Albums Found: {len(album_ids)} | Albums Not Found: {len(album_not_found)} | " f"ID3 Errors: {len(library_metadata_errors)} | Total Files: {len(files)}\n\n") f.write(f"Time to read metadata: {get_time_from_secs(perf.completeID3)}\n") f.write(f"Time to retrieve API data: {get_time_from_secs(perf.completeAPI)}\n\n") if library_metadata_errors: f.write("The following files had missing/invalid ID3 tag data:\n\n") for track in library_metadata_errors: if track.get('error'): f.write(f"\tFile: {track['rel_path']}\n") f.write(f"\t\tError: {track['error']}\n\n") f.write("\n") if album_not_found: artists_not_found = [track['artist'] for track in tracks if track['info'] == 'Artist not found'] if artists_not_found: f.write("The following artists were not found:\n") for artist in artists_not_found: f.write("\n\t{track['artist']\n") f.write("The following albums were not found:\n") for album in album_not_found: f.write(f"\n\tArtist: {album['artist']}\n") f.write(f"\tAlbum: {album['title']}\n") if album.get('info'): f.write(f"\tInfo: {album['info']}\n") else: f.write(f"Tracks Found: {len(track_ids)} | Tracks Not Found: {len(track_not_found)} | " f"ID3 Errors: {len(library_metadata_errors)} | Total Files: {len(files)}\n\n") f.write(f"Time to read metadata: {get_time_from_secs(perf.completeID3)}\n") f.write(f"Time to retrieve API data: {get_time_from_secs(perf.completeAPI)}\n\n") if library_metadata_errors: f.write("The following files had missing/invalid ID3 tag data:\n\n") for track in library_metadata_errors: if track.get('error'): f.write(f"\tFile: {track['rel_path']}\n") f.write(f"\t\tError: {track['error']}\n\n") f.write("\n") if track_not_found: artists_not_found = [track['artist'] for track in tracks if track.get('info', '') == 'Artist not found'] if artists_not_found: f.write("The following artists were not found:\n") for artist in artists_not_found: f.write("\n\t{track['artist']\n") f.write("The following tracks were not found:\n") for track in track_not_found: f.write(f"\n\tArtist: {track['artist']}\n") f.write(f"\tAlbum: {track['album']}\n") f.write(f"\tTrack: {track['title']}\n") f.write(f"\tFile: {track['rel_path']}\n") if track.get('info'): f.write(f"\tInfo: {track['info']}\n") ================================================ FILE: deemon/core/__init__.py ================================================ ================================================ FILE: deemon/core/api.py ================================================ import json import logging from datetime import datetime import deezer.errors from deezer import Deezer from deemon.core.config import Config as config logger = logging.getLogger(__name__) class PlatformAPI: def __init__(self): self.max_threads = 2 self.dz = Deezer() self.platform = self.get_platform() self.account_type = None self.api = self.set_platform() if config.check_account_status(): self.account_type = self.get_account_type() def debugger(self, message: str, payload = None): if config.debug_mode(): if not payload: payload = "" logger.debug(f"DEBUG_MODE: {message} {str(payload)}") def get_platform(self): if config.fast_api(): return "deezer-gw" return "deezer-api" def set_platform(self): if self.platform == "deezer-gw": self.max_threads = config.fast_api_threads() if self.max_threads > 50: self.max_threads = 50 if self.max_threads < 1: self.max_threads = 1 logger.debug("Using GW API, max_threads set " f"to {self.max_threads}") return self.dz.gw else: return self.dz.api def get_account_type(self): logger.info("Verifying ARL, please wait...") temp_dz = Deezer() temp_dz.login_via_arl(config.arl()) if temp_dz.get_session()['current_user'].get('can_stream_lossless'): logger.debug(f"Deezer account type is \"Hi-Fi\"") return "hifi" elif temp_dz.get_session()['current_user'].get('can_stream_hq'): logger.debug(f"Deezer account type is \"Premium\"") return "premium" else: logger.debug(f"Deezer account type is \"Free\"") return "free" #TODO GW API appears to ignore limit; must implement afterwards def search_artist(self, query: str, limit: int = 5): """ Return a list of dictionaries from API containing {'id': int, 'name': str} """ if self.platform == "deezer-gw": api_result = [] try: logger.info(f"Searching for {query}, please wait...") result = self.api.search(query=query)['ARTIST']['data'][:limit] except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while searching for artist {query}, retrying...") try: result = self.api.search(query=query)['ARTIST']['data'][:limit] except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response while searching for artist {query}") return [] for r in result: api_result.append({'id': int(r['ART_ID']), 'name': r['ART_NAME']}) else: api_result = self.api.search_artist(query=query, limit=limit)['data'] return {'query': query, 'results': api_result} def get_artist_by_id(self, query: int, limit: int = 1): """ Return a dictionary from API containing {'id': int, 'name': str} """ if self.platform == "deezer-gw": try: result = self.api.get_artist(query) except deezer.errors.GWAPIError as e: logger.debug(f"API error on artist ID {query}: {e}") return except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for artist ID {query}, retrying...") try: result = self.api.get_artist(query) except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response for artist ID {query}") return return {'id': int(result['ART_ID']), 'name': result['ART_NAME']} else: try: result = self.api.get_artist(query) except deezer.errors.DataException as e: logger.debug(f"API error: {e}") return return {'id': result['id'], 'name': result['name']} def get_album(self, query: int) -> dict: """Return a dictionary from API containing album info""" if self.platform == "deezer-gw": try: result = self.api.get_album(query) except deezer.errors.GWAPIError as e: logger.debug(f"API error: {e}") return {} except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for album ID {query}, retrying...") try: result = self.api.get_album(query) except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response for album ID {query}") return {} return {'id': int(result['ALB_ID']), 'title': result['ALB_TITLE'], 'artist': {'name': result['ART_NAME']}} else: try: result = self.api.get_album(query) except deezer.errors.DataException as e: logger.debug(f"API error: {e}") return else: return result def get_track(self, query: int) -> dict: """Return a dictionary from API containing album info""" if self.platform == "deezer-gw": try: result = self.api.get_track(query) except deezer.errors.GWAPIError as e: logger.debug(f"API error: {e}") return {} except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for album ID {query}, retrying...") try: result = self.api.get_album(query) except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response for album ID {query}") return {} return {'id': int(result['SNG_ID']), 'title': result['SNG_TITLE'], 'artist': {'name': result['ART_NAME']}} else: try: result = self.api.get_track(query) except deezer.errors.DataException as e: logger.debug(f"API error: {e}") else: return result def get_extra_release_info(self, query: dict): album = {'id': query['album_id'], 'label': None} if self.platform == "deezer-gw": album_details = self.api.get_album(query['album_id']) if album_details.get('LABEL_NAME'): album['label'] = album_details['LABEL_NAME'] else: album_details = self.api.get_album(query['album_id']) if album_details.get('label'): album['label'] = album_details['label'] return album def get_artist_albums(self, query: dict, limit: int = -1): """ Return a list of dictionaries from API containing """ self.debugger(f"Refreshing artist releases for {query['artist_name']} ({query['artist_id']})") if self.platform == "deezer-gw": try: result = self.api.get_artist_discography(art_id=query['artist_id'], limit=limit)['data'] except deezer.errors.GWAPIError as e: if "UNKNOWN" in str(e): logger.debug(e) logger.warning(f" [!] Artist discography is not available for " f"{query['artist_name']} ({query['artist_id']})") else: logger.debug(e) logger.error(f"An error occured while attempting to get the discography for " f"{query['artist_name']} ({query['artist_id']})") query['releases'] = [] return query except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for discography for {query['artist_name']}, retrying...") try: result = self.api.get_artist_discography(art_id=query['artist_id'], limit=limit)['data'] except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response for discography for {query['artist_name']}") query['releases'] = [] return query api_result = [] for r in result: # Remove ID check to get compilations if (r['ART_ID'] == str(query['artist_id']) and r['ARTISTS_ALBUMS_IS_OFFICIAL']) or (r['ART_ID'] == str(query['artist_id']) and config.allow_unofficial()) or config.allow_compilations(): # TYPE 0 - single, TYPE 1 - album, TYPE 2 - compilation, TYPE 3 - ep if r['TYPE'] == '0': r['TYPE'] = "single" elif r['TYPE'] == '1' and r['ART_ID'] != str(query['artist_id']): if not config.allow_featured_in(): logger.debug(f"Featured In for {query['artist_name']} detected but are disabled in config") continue else: logger.debug(f"Featured In detected for artist {query['artist_name']}: {r['ALB_TITLE']}") r['TYPE'] = "album" # TODO set unique r['TYPE'] for FEATURED IN elif r['TYPE'] == '2': if not config.allow_compilations(): logger.debug(f"Compilation for {query['artist_name']} detected but are disabled in config") continue else: logger.debug(f"Compilation detected for artist {query['artist_name']}: {r['ALB_TITLE']}") r['TYPE'] = "album" # TODO set unique r['TYPE'] for COMPILATIONS elif r['TYPE'] == '3': r['TYPE'] = "ep" else: r['TYPE'] = "album" if r['ORIGINAL_RELEASE_DATE'] != "0000-00-00": release_date = r['ORIGINAL_RELEASE_DATE'] elif r['PHYSICAL_RELEASE_DATE'] != "0000-00-00": release_date = r['PHYSICAL_RELEASE_DATE'] elif r['DIGITAL_RELEASE_DATE'] != "0000-00-00": release_date = r['DIGITAL_RELEASE_DATE'] else: # In the event of an unknown release date, set it to today's date # See album ID: 417403 logger.warning(f" [!] Found release without release date, assuming today: " f"{query['artist_name']} - {r['ALB_TITLE']}") release_date = datetime.strftime(datetime.today(), "%Y-%m-%d") cover_art = f"https://e-cdns-images.dzcdn.net/images/cover/{r['ALB_PICTURE']}/500x500-00000-80-0-0.jpg" album_url = f"https://www.deezer.com/album/{r['ALB_ID']}" api_result.append( { 'id': int(r['ALB_ID']), 'title': r['ALB_TITLE'], 'release_date': release_date, 'explicit_lyrics': r['EXPLICIT_ALBUM_CONTENT']['EXPLICIT_LYRICS_STATUS'], 'record_type': r['TYPE'], 'cover_big': cover_art, 'link': album_url, 'nb_tracks': r['NUMBER_TRACK'], } ) else: api_result = self.api.get_artist_albums(artist_id=query['artist_id'], limit=limit)['data'] query['releases'] = api_result return query @staticmethod def get_playlist(query: int): try: api_result = Deezer().api.get_playlist(query) except deezer.errors.PermissionException: logger.warning(f" [!] Permission Denied: Playlist {query} is private") return except deezer.errors.DataException: logger.warning(f" [!] Playlist ID {query} was not found") return except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for playlist ID {query}, retrying...") try: api_result = Deezer().api.get_playlist(query) except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response while getting data for playlist ID {query}") return return {'id': query, 'title': api_result['title'], 'link': f"https://deezer.com/playlist/{str(api_result['id'])}"} @staticmethod def get_playlist_tracks(query: dict): track_list = [] try: api_result = Deezer().api.get_playlist_tracks(query['id']) except deezer.errors.PermissionException: logger.warning(f" [!] Permission Denied: Playlist {query['title']} ({query['id']}) is private") return except deezer.errors.DataException: logger.warning(f" [!] Playlist ID {query} was not found") return except json.decoder.JSONDecodeError: logger.error(f" [!] Empty response from API while getting data for playlist ID {query['id']}") try: api_result = Deezer().api.get_playlist_tracks(query['id']) except json.decoder.JSONDecodeError: logger.error(f" [!] API still sending empty response while getting data for playlist ID {query['id']}") return for track in api_result['data']: track_list.append({'id': track['id'], 'title': track['title'], 'artist_id': track['artist']['id'], 'artist_name': track['artist']['name']}) query['tracks'] = track_list return query ================================================ FILE: deemon/core/common.py ================================================ import re import logging from deemon.core.config import Config as config logger = logging.getLogger(__name__) def exclude_filtered_versions(albums: list) -> list: """ Remove album versions containing specified text """ exclusion_patterns = config.exclusion_patterns() exclusion_keywords = config.exclusion_keywords() allowed = [] if exclusion_keywords or exclusion_patterns: for album in albums: album_title = album['title'] exclusion_pattern_match = [p for p in exclusion_patterns if re.search(p, album_title)] keyword_search = re.search(r'\(([^)]+)\)|\[([^)]+)]', album_title.lower()) exclusion_keyword_match = [e for e in exclusion_keywords if keyword_search and e in keyword_search.group()] result = exclusion_pattern_match + exclusion_keyword_match result = '", "'.join(result) if exclusion_keyword_match or exclusion_pattern_match: logger.info(f" Album \"{album_title}\" excluded by filter: \"{result}\"") continue else: allowed.append(album) return allowed else: return albums ================================================ FILE: deemon/core/config.py ================================================ import json import logging from copy import deepcopy from pathlib import Path from typing import Optional from deemon.core.exceptions import ValueNotAllowed, UnknownValue, PropertyTypeMismatch from deemon.utils import startup logger = logging.getLogger(__name__) ALLOWED_VALUES = { 'bitrate': {1: "128", 3: "320", 9: "FLAC"}, 'alerts': [True, False], 'record_type': ['all', 'album', 'ep', 'single'], 'release_channel': ['stable', 'beta'] } DEFAULT_CONFIG = { "check_update": 1, "debug_mode": False, "release_channel": "stable", "query_limit": 5, "smart_search": True, "rollback_view_limit": 10, "prompt_duplicates": False, "prompt_no_matches": True, "fast_api": True, "fast_api_threads": 25, "exclusions": { "enable_exclusions": True, "patterns": [], "keywords": [], }, "new_releases": { "release_max_age": 90, "include_unofficial": False, "include_compilations": False, "include_featured_in": False, }, "global": { "bitrate": "320", "alerts": False, "record_type": "all", "download_path": "", "email": "" }, "deemix": { "path": "", "arl": "", "check_account_status": True, "halt_download_on_error": False, }, "smtp_settings": { "server": "", "port": 465, "starttls": False, "username": "", "password": "", "from_addr": "" }, "plex": { "base_url": "", "ssl_verify": True, "token": "", "library": "" } } class Config(object): _CONFIG_FILE: Optional[Path] = startup.get_config() _CONFIG: Optional[dict] = None def __init__(self): if not Config._CONFIG_FILE.exists(): self.__create_default_config() if Config._CONFIG is None: with open(Config._CONFIG_FILE, 'r', encoding='utf8') as f: try: Config._CONFIG = json.load(f) except json.decoder.JSONDecodeError as e: logger.exception(f"An error occured while reading from config: {e}") raise if self.validate() > 0: self.__write_modified_config() if not self.arl(): logger.debug("Attempting to locate deemix's .arl file") arl_file = startup.get_appdata_root() / 'deemix' / '.arl' if Path(arl_file).is_file(): with open(arl_file) as f: arl_from_file = f.readline().replace("\n", "") self.set('arl', arl_from_file) logger.debug("Successfully loaded ARL") if len(self.arl()) > 0 and len(self.arl()) != 192: logger.warning(f" [!] Possible invalid ARL detected (length: {len(self.arl())}). ARL should " "be 192 characters") # Set default for runtime settings self.set('profile_id', 1, validate=False) @staticmethod def __create_default_config(): with open(Config._CONFIG_FILE, 'w', encoding='utf8') as f: json.dump(DEFAULT_CONFIG, f, indent=4) @staticmethod def __write_modified_config(): with open(Config._CONFIG_FILE, 'w', encoding='utf8') as f: json.dump(Config._CONFIG, f, indent=4) @staticmethod def validate(): modified = 0 def process_config(dict1, dict2): """ Process user configuration, applying values to a default config """ for key, value in dict1.items(): if key in dict2.keys(): if isinstance(dict1[key], dict): process_config(dict1[key], dict2[key]) else: dict2[key] = dict1[key] return dict2 def find_position(d, key): for k, v in d.items(): if isinstance(v, dict): next = find_position(v, key) if next: return [k] + next elif k == key: return [k] def update_config_layout(user_config, reference_config): """ Used to move existing values to new property names/locations """ nonlocal modified if user_config.get('exclude'): user_config['exclusions'] = { 'enable_exclusions': True, 'patterns': user_config['exclude'], 'keywords': [] } modified += 1 if user_config.get('experimental'): if user_config['experimental'].get('allow_unofficial_releases'): user_config['new_releases']['include_unofficial'] = True modified += 1 if user_config['experimental'].get('allow_compilations'): user_config['new_releases']['include_compilations'] = True modified += 1 if user_config['experimental'].get('allow_featured_in'): user_config['new_releases']['include_featured_in'] = True modified += 1 if user_config.get('new_releases'): if not user_config['new_releases'].get('by_release_date', True): user_config['new_releases']['release_max_age'] = 0 modified += 1 migration_map = [ {'check_update': 'check_update'}, {'plex_baseurl': 'base_url'}, {'plex_token': 'token'}, {'plex_library': 'library'}, {'deemix_path': 'path'}, {'arl': 'arl'}, {'smtp_recipient': 'email'}, {'smtp_server': 'server'}, {'smtp_user': 'username'}, {'smtp_pass': 'password'}, {'smtp_port': 'port'}, {'smtp_sender': 'from_addr'}, {'bitrate': 'bitrate'}, {'alerts': 'alerts'}, {'record_type': 'record_type'}, {'download_path': 'download_path'}, {'release_max_days': 'release_max_age'}, {'ranked_duplicates': 'prompt_duplicates'} ] for mlist in migration_map: for old, new in mlist.items(): user_config_tmp = deepcopy(user_config) user_config_copy = user_config if not user_config.get(old): continue old_position = find_position(user_config, old) or [old] new_position = find_position(reference_config, new) or [new] for i in old_position[:-1]: user_config_tmp = user_config_tmp.setdefault(i, {}) for i in new_position[:-1]: user_config_copy = user_config_copy.setdefault(i, {}) if user_config_tmp != user_config_copy: logger.debug("Migrating " + ':'.join([str(x) for x in old_position]) + " -> " + ':'.join( [str(x) for x in new_position])) user_config_copy[new_position[-1]] = user_config_tmp[old_position[-1]] modified += 1 return user_config def test_values(dict1, dict2): nonlocal modified for key, value in dict1.items(): if key in dict2.keys(): if isinstance(dict1[key], dict): test_values(dict1[key], dict2[key]) else: if key == "starttls" and value is True: if dict1['port'] == 465: logger.debug("STARTTLS enabled in config but port set to 465 (SSL); disabling STARTTLS") dict1['starttls'] = False modified += 1 if key in ALLOWED_VALUES: if isinstance(ALLOWED_VALUES[key], dict): if key == "bitrate" and value in ["1", "3", "9"]: if value == "1": dict1['bitrate'] = "128" if value == "3": dict1['bitrate'] = "320" if value == "9": dict1['bitrate'] = "FLAC" modified += 1 elif value in ALLOWED_VALUES[key].keys(): dict1_tmp = dict1 pos = find_position(dict1_tmp, key) for i in pos[:-1]: dict1_tmp = dict1.setdefault(i, {}) dict1_tmp[key] = ALLOWED_VALUES[key][value] modified += 1 elif ( value in ALLOWED_VALUES[key].values() or value.upper() in ALLOWED_VALUES[key].values() ): continue else: raise UnknownValue( f"Unknown value in config - '{key}': {value} (type: {type(value).__name__})") elif not isinstance(dict1[key], type(dict2[key])): if isinstance(dict2[key], bool): if dict1[key] == 1: dict1[key] = True modified += 1 if dict1[key] == 0: dict1[key] = False modified += 1 else: raise UnknownValue( f"Unknown value in config - '{key}': {value} (type: {type(value).__name__})") else: if value in ALLOWED_VALUES[key]: continue else: raise UnknownValue( f"Unknown value in config - '{key}': {value} (type: {type(value).__name__})") elif not isinstance(dict1[key], type(dict2[key])): if isinstance(dict2[key], bool): if dict1[key] == 1: dict1[key] = True modified += 1 if dict1[key] == 0: dict1[key] = False modified += 1 else: raise PropertyTypeMismatch( f"Invalid type in config - '{str(key)}' incorrectly set as {type(value).__name__}") else: pass def add_new_options(dict1, dict2): nonlocal modified for key, value in dict1.items(): if key not in dict2.keys(): logger.debug(f"New option added to config: {key}") dict2[key] = value modified += 1 elif isinstance(value, dict): for k, v in value.items(): if k not in dict2[key].keys(): logger.debug("New option added to config: " f"{key}/{k}") dict2[key][k] = v modified += 1 logger.debug("Loading configuration, please wait...") add_new_options(DEFAULT_CONFIG, Config._CONFIG) migrated_config = update_config_layout(Config._CONFIG, DEFAULT_CONFIG) Config._CONFIG = process_config(migrated_config, DEFAULT_CONFIG) test_values(Config._CONFIG, DEFAULT_CONFIG) return modified @staticmethod def get_config_file() -> Path: return Config._CONFIG_FILE @staticmethod def get_config() -> dict: return Config._CONFIG @staticmethod def plex_baseurl() -> str: return Config._CONFIG.get('plex').get('base_url') @staticmethod def plex_token() -> str: return Config._CONFIG.get('plex').get('token') @staticmethod def plex_library() -> str: return Config._CONFIG.get('plex').get('library') @staticmethod def download_path() -> str: return Config._CONFIG.get('global').get('download_path') @staticmethod def deemix_path() -> str: return Config._CONFIG.get('deemix').get('path') @staticmethod def arl() -> str: return Config._CONFIG.get('deemix').get('arl') @staticmethod def release_max_age() -> int: return Config._CONFIG.get('new_releases').get('release_max_age') @staticmethod def bitrate() -> str: return Config._CONFIG.get('global').get('bitrate') @staticmethod def alerts() -> bool: return Config._CONFIG.get('global').get('alerts') @staticmethod def record_type() -> str: return Config._CONFIG.get('global').get('record_type') @staticmethod def smtp_server() -> str: return Config._CONFIG.get('smtp_settings').get('server') @staticmethod def smtp_port() -> int: return Config._CONFIG.get('smtp_settings').get('port') @staticmethod def smtp_user() -> str: return Config._CONFIG.get('smtp_settings').get('username') @staticmethod def smtp_pass() -> str: return Config._CONFIG.get('smtp_settings').get('password') @staticmethod def smtp_sender() -> str: return Config._CONFIG.get('smtp_settings').get('from_addr') @staticmethod def smtp_recipient() -> list: return Config._CONFIG.get('global').get('email') @staticmethod def smtp_starttls() -> bool: return Config._CONFIG.get('smtp_settings').get('starttls') @staticmethod def check_update() -> int: return Config._CONFIG.get('check_update') @staticmethod def debug_mode() -> bool: return Config._CONFIG.get('debug_mode') @staticmethod def profile_id() -> int: return Config._CONFIG.get('profile_id') @staticmethod def update_available() -> int: return Config._CONFIG.get('update_available') @staticmethod def query_limit() -> int: return Config._CONFIG.get('query_limit') @staticmethod def prompt_duplicates() -> int: return Config._CONFIG.get('prompt_duplicates') @staticmethod def prompt_no_matches() -> bool: return Config._CONFIG.get('prompt_no_matches') @staticmethod def allowed_values(prop): return ALLOWED_VALUES.get(prop) @staticmethod def release_channel() -> str: return Config._CONFIG.get('release_channel') @staticmethod def rollback_view_limit() -> int: return Config._CONFIG.get('rollback_view_limit') @staticmethod def transaction_id() -> int: return Config._CONFIG.get('tid') @staticmethod def check_account_status() -> bool: return Config._CONFIG.get('deemix').get('check_account_status') @staticmethod def fast_api() -> bool: return Config._CONFIG['fast_api'] @staticmethod def fast_api_threads() -> int: return Config._CONFIG['fast_api_threads'] @staticmethod def allow_compilations() -> bool: return Config._CONFIG['new_releases']['include_compilations'] @staticmethod def allow_featured_in() -> bool: return Config._CONFIG['new_releases']['include_featured_in'] @staticmethod def allow_unofficial() -> bool: return Config._CONFIG['new_releases']['include_unofficial'] @staticmethod def enable_exclusions() -> bool: return Config._CONFIG['exclusions']['enable_exclusions'] @staticmethod def exclusion_keywords() -> list: if Config._CONFIG['exclusions']['enable_exclusions']: return [x.lower() for x in Config._CONFIG['exclusions']['keywords']] else: return [] @staticmethod def exclusion_patterns() -> list: if Config._CONFIG['exclusions']['enable_exclusions']: return Config._CONFIG['exclusions']['patterns'] else: return [] @staticmethod def plex_ssl_verify() -> bool: return Config._CONFIG.get('plex').get('ssl_verify') @staticmethod def halt_download_on_error() -> bool: return Config._CONFIG.get('deemix').get('halt_download_on_error') @staticmethod def smart_search() -> bool: return Config._CONFIG.get('smart_search') @staticmethod def find_position(d, property): for k, v in d.items(): if isinstance(v, dict): next = Config.find_position(v, property) if next: return [k] + next elif k == property: return [k] @staticmethod def get(property): return Config._CONFIG.get(property) @staticmethod def set(property, value, validate=True): if not validate: Config._CONFIG[property] = value if Config._CONFIG.get(property): if property in ALLOWED_VALUES: if value.lower() == "true" or value == "1": value = True elif value.lower() == "false" or value == "0": value = "" if value in ALLOWED_VALUES[property]: Config._CONFIG[property] = value return raise ValueNotAllowed(f"Property {property} requires one of " f"{', '.join(ALLOWED_VALUES[property])}, not {value}.") if isinstance(value, type(Config._CONFIG[property])): Config._CONFIG[property] = value return else: raise PropertyTypeMismatch(f"Type mismatch while setting {property} " f"to {value} (type: {type(value).__name__})") else: property_path = Config.find_position(Config._CONFIG, property) tmpConfig = Config._CONFIG for k in property_path[:-1]: tmpConfig = tmpConfig.setdefault(k, {}) if property in ALLOWED_VALUES: if isinstance(value, str): if value.lower() == "true" or value == "1": value = True elif value.lower() == "false" or value == "0": value = False if isinstance(ALLOWED_VALUES[property], dict): if value in [str(x.lower()) for x in ALLOWED_VALUES[property].values()]: tmpConfig[property_path[-1]] = value return True if value in ALLOWED_VALUES[property]: tmpConfig[property_path[-1]] = value return True raise ValueNotAllowed(f"Value for {property} is invalid: {value} (type: {type(value).__name__})") if isinstance(value, type(tmpConfig[property])): tmpConfig[property] = value return True else: raise PropertyTypeMismatch(f"Type mismatch while setting {property} " f"to {value} (type: {type(value).__name__})") class LoadProfile(object): def __init__(self, profile: dict): logger.debug(f"Loaded config for profile {str(profile['id'])} ({str(profile['name'])})") # Rename keys to match config profile["profile_id"] = profile.pop("id") profile["base_url"] = profile.pop("plex_baseurl") profile["token"] = profile.pop("plex_token") profile["library"] = profile.pop("plex_library") # Append to config for debug output; Remove profile name from dict Config.set("profile_name", profile.pop("name"), validate=False) for key, value in profile.items(): if value is None: continue Config.set(key, value) for key, value in Config.get_config().items(): if isinstance(value, dict): for k, v in value.items(): if key in ['smtp_settings'] or k in ['arl', 'email', 'token']: if v: v = "********" logger.debug(f"> {key}/{k}: {v}") else: logger.debug(f"> {key}: {value}") ================================================ FILE: deemon/core/db.py ================================================ import logging import sqlite3 import time from datetime import datetime from pathlib import Path from packaging.version import parse as parse_version from deemon import __dbversion__, __version__ from deemon.core.config import Config as config from deemon.utils import startup, performance, dates logger = logging.getLogger(__name__) class Database(object): def __init__(self): self.conn = None self.cursor = None self.db = startup.get_database() if not Path(self.db).exists(): self.connect() self.create_new_database() else: self.connect() def __enter__(self): return self def __exit__(self): self.close() @staticmethod def dict_factory(cursor, row): d = {} for idx, col in enumerate(cursor.description): d[col[0]] = row[idx] return d def connect(self): try: self.conn = sqlite3.connect(self.db) self.conn.row_factory = self.dict_factory self.cursor = self.conn.cursor() except sqlite3.OperationalError as e: logger.error(f"Error opening database: {e}") def close(self): if self.conn: self.conn.commit() self.conn.close() def commit(self): self.conn.commit() def commit_and_close(self): self.commit() self.close() def create_new_database(self): logger.debug("DATABASE CREATION IN PROGRESS!") self.query("CREATE TABLE monitor (" "'artist_id' INTEGER," "'artist_name' TEXT," "'bitrate' TEXT," "'record_type' TEXT," "'alerts' INTEGER," "'profile_id' INTEGER DEFAULT 1," "'download_path' TEXT," "'refreshed' INTEGER DEFAULT 0," "'trans_id' INTEGER)") self.query("CREATE TABLE playlists (" "'id' INTEGER UNIQUE," "'title' TEXT," "'url' TEXT," "'bitrate' TEXT," "'alerts' INTEGER," "'profile_id' INTEGER DEFAULT 1," "'download_path' TEXT," "'refreshed' INTEGER DEFAULT 0," "'trans_id' INTEGER," "'monitor_artists' INTEGER DEFAULT 0)") self.query("CREATE TABLE playlist_tracks (" "'track_id' INTEGER," "'playlist_id' INTEGER," "'artist_id' INTEGER," "'artist_name' TEXT," "'track_name' TEXT," "'profile_id' INTEGER DEFAULT 1," "'track_added' TEXT," "'trans_id' INTEGER)") self.query("CREATE TABLE releases (" "'artist_id' INTEGER," "'artist_name' TEXT," "'album_id' INTEGER," "'album_name' TEXT," "'album_release' TEXT," "'album_added' INTEGER," "'explicit' INTEGER," "'label' TEXT," "'record_type' INTEGER," "'profile_id' INTEGER DEFAULT 1," "'future_release' INTEGER DEFAULT 0," "'trans_id' INTEGER," "unique(album_id, profile_id))") self.query("CREATE TABLE 'deemon' (" "'property' TEXT," "'value' TEXT)") self.query("CREATE TABLE 'profiles' (" "'id' INTEGER," "'name' TEXT," "'email' TEXT," "'alerts' INTEGER," "'bitrate' TEXT," "'record_type' TEXT," "'plex_baseurl' TEXT," "'plex_token' TEXT," "'plex_library' TEXT," "'download_path' TEXT," "PRIMARY KEY('id' AUTOINCREMENT))") self.query("CREATE TABLE transactions (" "'id' INTEGER," "'timestamp' INTEGER," "'profile_id' INTEGER DEFAULT 1," "PRIMARY KEY('id' AUTOINCREMENT))") self.query("CREATE UNIQUE INDEX 'idx_property' ON 'deemon' ('property')") self.query("CREATE INDEX 'artist' ON 'releases' ('artist_id', 'profile_id')") self.query(f"INSERT INTO 'deemon' ('property', 'value') VALUES ('version', '{__dbversion__}')") self.query("INSERT OR REPLACE INTO 'deemon' ('property', 'value') VALUES ('latest_ver', '')") self.query("INSERT INTO 'deemon' ('property', 'value') VALUES ('last_update_check', 0)") self.query("INSERT INTO 'deemon' ('property', 'value') VALUES ('release_channel', 'stable')") self.query("INSERT INTO 'profiles' ('name') VALUES ('default')") self.commit() def get_latest_ver(self): return self.query("SELECT value FROM deemon WHERE property = 'latest_ver'").fetchone()['value'] def get_db_version(self): try: version = self.query(f"SELECT value FROM deemon WHERE property = 'version'").fetchone()['value'] except sqlite3.OperationalError: version = '0.0.0' logger.debug(f"Database version {version}") return version def do_upgrade(self): current_ver = parse_version(self.get_db_version()) app_db_version = parse_version(__dbversion__) if current_ver == app_db_version: return logger.debug("DATABASE UPGRADE IN PROGRESS!") if current_ver < parse_version("3.5"): logger.error("Due to database changes, you must be on at least " f"v2.5 before upgrading to v{__version__}.") exit() if current_ver < parse_version("3.5.2"): self.query("CREATE TABLE releases_tmp (" "'artist_id' INTEGER," "'artist_name' TEXT," "'album_id' INTEGER," "'album_name' TEXT," "'album_release' TEXT," "'album_added' INTEGER," "'explicit' INTEGER," "'label' TEXT," "'record_type' INTEGER," "'profile_id' INTEGER DEFAULT 1," "'future_release' INTEGER DEFAULT 0," "'trans_id' INTEGER," "unique(album_id, profile_id))") self.query("INSERT OR REPLACE INTO releases_tmp(artist_id, artist_name, " "album_id, album_name, album_release, album_added, " "explicit, label, record_type, profile_id, " "future_release, trans_id) SELECT artist_id, " "artist_name, album_id, album_name, album_release, " "album_added, explicit, label, record_type, " "profile_id, future_release, trans_id FROM releases") self.query("DROP TABLE releases") self.query("ALTER TABLE releases_tmp RENAME TO releases") self.query("INSERT OR REPLACE INTO 'deemon' ('property', 'value') VALUES ('version', '3.5.2')") self.commit() if current_ver < parse_version("3.6"): # album_release_ts REMOVED pass if current_ver < parse_version("3.7"): self.query("CREATE TABLE playlists_tmp (" "'id' INTEGER UNIQUE," "'title' TEXT," "'url' TEXT," "'bitrate' TEXT," "'alerts' INTEGER," "'profile_id' INTEGER DEFAULT 1," "'download_path' TEXT," "'refreshed' INTEGER DEFAULT 0," "'trans_id' INTEGER," "'monitor_artists' INTEGER DEFAULT 0)") self.query("INSERT OR REPLACE INTO playlists_tmp (id, title, url, bitrate, " "alerts, profile_id, download_path, refreshed, trans_id) SELECT " "id, title, url, bitrate, alerts, profile_id, download_path, " "refreshed, trans_id FROM playlists") self.query("DROP TABLE playlists") self.query("ALTER TABLE playlists_tmp RENAME TO playlists") self.query("INSERT OR REPLACE INTO 'deemon' ('property', 'value') VALUES ('version', '3.7')") self.commit() logger.debug(f"Database upgraded to version 3.7") def query(self, query, values=None): if values is None: values = {} return self.cursor.execute(query, values) def reset_future(self, album_id): logger.debug("Clearing future_release flag from " + str(album_id)) values = {'album_id': album_id, 'profile_id': config.profile_id()} sql = "UPDATE 'releases' SET future_release = 0 WHERE album_id = :album_id AND profile_id = :profile_id" self.query(sql, values) def get_all_monitored_artists(self): vals = {'profile_id': config.profile_id()} return self.query(f"SELECT * FROM monitor WHERE profile_id = :profile_id " f"ORDER BY artist_name COLLATE NOCASE ASC", vals).fetchall() def get_monitored_artist_by_id(self, artist_id: int): values = {'id': artist_id, 'profile_id': config.profile_id()} return self.query(f"SELECT * FROM monitor WHERE artist_id = :id AND profile_id = :profile_id", values).fetchone() def get_monitored_artist_by_name(self, name: str): values = {'name': name, 'profile_id': config.profile_id()} return self.query(f"SELECT * FROM monitor WHERE artist_name = :name COLLATE NOCASE " f"AND profile_id = :profile_id", values).fetchone() def get_all_monitored_playlist_ids(self): vals = {'profile_id': config.profile_id()} query = self.query("SELECT id FROM playlists WHERE profile_id = :profile_id", vals).fetchall() return [v for x in query for v in x.values()] def get_all_monitored_playlists(self): vals = {'profile_id': config.profile_id()} return self.query("SELECT * FROM playlists WHERE profile_id = :profile_id " "ORDER BY title COLLATE NOCASE ASC", vals).fetchall() def get_monitored_playlist_by_id(self, playlist_id): values = {'id': playlist_id, 'profile_id': config.profile_id()} return self.query("SELECT * FROM playlists WHERE id = :id AND profile_id = :profile_id", values).fetchone() def get_monitored_playlist_by_name(self, title): values = {'title': title, 'profile_id': config.profile_id()} return self.query("SELECT * FROM playlists WHERE title = :title COLLATE NOCASE " "AND profile_id = :profile_id", values).fetchone() def monitor_artist(self, artist: dict, artist_config: dict): self.new_transaction() vals = { 'artist_id': artist['id'], 'artist_name': artist['name'], 'bitrate': artist_config['bitrate'], 'record_type': artist_config['record_type'], 'alerts': artist_config['alerts'], 'download_path': artist_config['download_path'], 'profile_id': config.profile_id(), 'trans_id': config.transaction_id() } query = ("INSERT INTO monitor " "(artist_id, artist_name, bitrate, record_type, alerts, download_path, profile_id, trans_id) " "VALUES " "(:artist_id, :artist_name, :bitrate, :record_type, :alerts, :download_path, :profile_id, :trans_id)") self.query(query, vals) self.commit() def get_artist_releases(self, artist_id=None): sql_values = {'artist_id': artist_id, 'profile_id': config.profile_id()} if artist_id: query = "SELECT album_id, future_release FROM 'releases' WHERE artist_id = :artist_id AND profile_id = :profile_id" else: query = "SELECT album_id, future_release FROM 'releases' WHERE profile_id = :profile_id" return self.query(query, sql_values).fetchall() def get_future_releases(self): vals = {'profile_id': config.profile_id()} return self.query("SELECT * FROM releases " "WHERE future_release = 1 AND profile_id = :profile_id", vals).fetchall() def get_playlist_tracks(self, playlist_id): sql_values = {'playlist_id': playlist_id, 'profile_id': config.profile_id()} query = "SELECT * FROM 'playlist_tracks' WHERE playlist_id = :playlist_id AND profile_id = :profile_id" return self.query(query, sql_values).fetchall() def get_track_from_playlist(self, playlist_id, track_id): values = {'pid': playlist_id, 'tid': track_id, 'profile_id': config.profile_id()} query = "SELECT * FROM 'playlist_tracks' WHERE track_id = :tid AND playlist_id = :pid AND profile_id = :profile_id" result = self.query(query, values).fetchone() if result: return True def monitor_playlist(self, api_result): self.new_transaction() values = {'id': api_result['id'], 'title': api_result['title'], 'url': api_result['link'], 'bitrate': api_result['bitrate'], 'alerts': api_result['alerts'], 'download_path': api_result['download_path'], 'profile_id': config.profile_id(), 'trans_id': config.transaction_id()} query = ("INSERT INTO playlists ('id', 'title', 'url', 'bitrate', 'alerts', 'download_path'," "'profile_id', 'trans_id') " "VALUES (:id, :title, :url, :bitrate, :alerts, :download_path, :profile_id, :trans_id)") self.query(query, values) self.commit() def remove_monitored_artist(self, id: int = None): values = {'id': id, 'profile_id': config.profile_id()} self.query("DELETE FROM monitor WHERE artist_id = :id AND profile_id = :profile_id", values) self.query("DELETE FROM releases WHERE artist_id = :id AND profile_id = :profile_id", values) self.commit() def remove_monitored_playlists(self, id: int = None): values = {'id': id, 'profile_id': config.profile_id()} self.query("DELETE FROM playlists WHERE id = :id AND profile_id = :profile_id", values) self.query("DELETE FROM playlist_tracks WHERE playlist_id = :id AND profile_id = :profile_id", values) self.commit() def get_specified_artist(self, artist): values = {'artist': artist, 'profile_id': config.profile_id()} if type(artist) is int: return self.query("SELECT * FROM monitor WHERE artist_id = :artist " "AND profile_id = :profile_id", values).fetchone() else: return self.query("SELECT * FROM monitor WHERE artist_name = ':artist' " "AND profile_id = :profile_id COLLATE NOCASE", values).fetchone() def set_all_artists_refreshed(self): self.query("UPDATE monitor SET refreshed = 1 WHERE refreshed = 0") def set_all_playlists_refreshed(self): self.query("UPDATE playlists SET refreshed = 1 WHERE refreshed = 0") def add_new_releases(self, values): self.new_transaction() sql = (f"INSERT OR REPLACE INTO releases ('artist_id', 'artist_name', 'album_id', 'album_name', 'album_release', " f"'album_added', 'future_release', 'explicit', 'record_type', 'profile_id', 'trans_id') " f"VALUES (:artist_id, :artist_name, :id, :title, :release_date, {int(time.time())}, :future, " f":explicit_lyrics, :record_type, {config.profile_id()}, {config.transaction_id()})") self.cursor.executemany(sql, values) self.set_all_artists_refreshed() def add_new_playlist_releases(self, values): self.new_transaction() sql = (f"INSERT INTO playlist_tracks ('artist_id', 'artist_name', 'track_id', 'track_name', 'playlist_id', " f"'track_added', 'profile_id', 'trans_id') VALUES (:artist_id, :artist_name, :id, :title, :playlist_id, " f"{int(time.time())}, {config.profile_id()}, {config.transaction_id()})") self.cursor.executemany(sql, values) self.set_all_playlists_refreshed() def show_new_releases(self, from_date_ts, now_ts): today_date = datetime.utcfromtimestamp(now_ts).strftime('%Y-%m-%d') from_date = datetime.utcfromtimestamp(from_date_ts).strftime('%Y-%m-%d') values = {'from': from_date, 'today': today_date, 'profile_id': config.profile_id()} sql = "SELECT * FROM 'releases' WHERE album_release >= :from AND album_release <= :today AND profile_id = :profile_id" return self.query(sql, values).fetchall() def get_album_by_id(self, album_id): values = {'id': album_id, 'profile_id': config.profile_id()} sql = "SELECT * FROM 'releases' WHERE album_id = :id AND profile_id = :profile_id" return self.query(sql, values).fetchone() def reset_database(self): self.query("DELETE FROM monitor") self.query("DELETE FROM releases") self.query("DELETE FROM playlists") self.query("DELETE FROM playlist_tracks") self.query("DELETE FROM transactions") self.commit() logger.info("Database has been reset") def update_artist(self, settings: dict): self.query("UPDATE monitor SET bitrate = :bitrate, alerts = :alerts, record_type = :record_type," "download_path = :download_path WHERE artist_id = :artist_id AND profile_id = :profile_id", settings) self.commit() def add_playlist_track(self, playlist, track): self.new_transaction() values = {'pid': playlist['id'], 'tid': track['id'], 'tname': track['title'], 'aid': track['artist']['id'], 'aname': track['artist']['name'], 'time': int(time.time()), 'profile_id': config.profile_id(), 'trans_id': config.transaction_id()} query = ("INSERT INTO 'playlist_tracks' " "('track_id', 'playlist_id', 'artist_id', 'artist_name', 'track_name', 'track_added', 'profile_id'," "'trans_id') VALUES (:tid, :pid, :aid, :aname, :tname, :time, :profile_id, :trans_id)") return self.query(query, values) def create_profile(self, settings: dict): self.query("INSERT INTO profiles (name, email, alerts, bitrate, record_type, plex_baseurl, plex_token," "plex_library, download_path) VALUES (:name, :email, :alerts, :bitrate, :record_type," ":plex_baseurl, :plex_token, :plex_library, :download_path)", settings) self.commit() def delete_profile(self, profile_name: str): profile = self.get_profile(profile_name) vals = {'profile_id': profile['id']} self.query("DELETE FROM monitor WHERE profile_id = :profile_id", vals) self.query("DELETE FROM releases WHERE profile_id = :profile_id", vals) self.query("DELETE FROM playlists WHERE profile_id = :profile_id", vals) self.query("DELETE FROM playlist_tracks WHERE profile_id = :profile_id", vals) self.query("DELETE FROM profiles WHERE id = :profile_id", vals) self.commit() def get_all_profiles(self): return self.query("SELECT * FROM profiles").fetchall() def get_profile(self, profile_name: str): vals = {'profile': profile_name} return self.query("SELECT * FROM profiles WHERE name = :profile COLLATE NOCASE", vals).fetchone() def get_profile_by_id(self, profile_id: int): vals = {'profile_id': profile_id} return self.query("SELECT * FROM profiles WHERE id = :profile_id", vals).fetchone() def update_profile(self, settings: dict): self.query("UPDATE profiles SET name = :name, email = :email, alerts = :alerts, bitrate = :bitrate," "record_type = :record_type, plex_baseurl = :plex_baseurl, plex_token = :plex_token," "plex_library = :plex_library, download_path = :download_path " "WHERE id = :id", settings) self.commit() def last_update_check(self): return self.query("SELECT value FROM 'deemon' WHERE property = 'last_update_check'").fetchone()['value'] def set_last_update_check(self): now = int(time.time()) self.query(f"UPDATE deemon SET value = {now} WHERE property = 'last_update_check'") self.commit() def get_next_transaction_id(self): tid = self.query(f"SELECT seq FROM sqlite_sequence WHERE name = 'transactions'").fetchone() if not tid: return 0 return tid['seq'] + 1 def new_transaction(self): check_exists = self.query(f"SELECT * FROM transactions WHERE id = {config.transaction_id()}").fetchone() if not check_exists: current_time = int(time.time()) vals = {'timestamp': current_time, 'profile_id': config.profile_id()} self.query(f"INSERT INTO transactions ('timestamp', 'profile_id') " f"VALUES (:timestamp, :profile_id)", vals) self.commit() def rollback_last_refresh(self, rollback: int): vals = {'rollback': rollback, 'profile_id': config.profile_id()} transactions = self.query("SELECT id FROM transactions WHERE profile_id = :profile_id " f"ORDER BY id DESC LIMIT {rollback}", vals).fetchall() for t in transactions: vals = {'id': t['id'], 'profile_id': config.profile_id()} self.query(f"DELETE FROM monitor WHERE trans_id = :id AND profile_id = :profile_id", vals) self.query(f"DELETE FROM releases WHERE trans_id = :id AND profile_id = :profile_id", vals) self.query(f"DELETE FROM playlist_tracks WHERE trans_id = :id AND profile_id = :profile_id", vals) self.query(f"DELETE FROM transactions WHERE id = :id AND profile_id = :profile_id", vals) self.commit() def rollback_refresh(self, rollback: int): vals = {'rollback': rollback, 'profile_id': config.profile_id()} self.query(f"DELETE FROM monitor WHERE trans_id = {rollback} AND profile_id = :profile_id", vals) self.query(f"DELETE FROM releases WHERE trans_id = {rollback} AND profile_id = :profile_id", vals) self.query(f"DELETE FROM playlist_tracks WHERE trans_id = {rollback} AND profile_id = :profile_id", vals) self.query(f"DELETE FROM transactions WHERE id = {rollback} AND profile_id = :profile_id", vals) self.commit() def set_artist_refreshed(self, id): vals = {'id': id, 'profile_id': config.profile_id()} return self.query("UPDATE monitor SET refreshed = 1 WHERE artist_id = :id AND profile_id = :profile_id", vals) def set_playlist_refreshed(self, id): vals = {'id': id, 'profile_id': config.profile_id()} return self.query("UPDATE playlists SET refreshed = 1 WHERE id = :id AND profile_id = :profile_id", vals) def set_latest_version(self, version): vals = {'version': version} self.query("INSERT OR REPLACE INTO deemon (property, value) VALUES ('latest_ver', :version)", vals) return self.commit() def get_release_channel(self): return self.query("SELECT value FROM deemon WHERE property = 'release_channel'").fetchone() def set_release_channel(self): self.query(f"INSERT OR REPLACE INTO deemon (property, value) " f"VALUES ('release_channel', '{config.release_channel()}')") return self.commit() def get_transactions(self): vals = {'profile_id': config.profile_id(), 'trans_limit': config.rollback_view_limit()} transaction_list = self.query("SELECT id, timestamp FROM transactions WHERE profile_id = :profile_id " "ORDER BY id DESC LIMIT :trans_limit", vals).fetchall() results = [] for tid in transaction_list: vals = {'tid': tid['id'], 'profile_id': config.profile_id()} transaction = {} transaction['id'] = tid['id'] transaction['timestamp'] = tid['timestamp'] transaction['releases'] = self.query("SELECT album_id " "FROM releases " "WHERE trans_id = :tid " "AND profile_id = :profile_id", vals).fetchall() transaction['playlist_tracks'] = self.query("SELECT track_id " "FROM playlist_tracks " "WHERE trans_id = :tid " "AND profile_id = :profile_id", vals).fetchall() transaction['playlists'] = self.query("SELECT title " "FROM playlists " "WHERE trans_id = :tid " "AND profile_id = :profile_id", vals).fetchall() transaction['monitor'] = self.query("SELECT artist_name " "FROM monitor " "WHERE trans_id = :tid " "AND profile_id = :profile_id", vals).fetchall() results.append(transaction) return results def get_all_monitored_artist_ids(self): values = {"profile_id": config.profile_id()} query = self.query("SELECT artist_id FROM monitor WHERE profile_id = :profile_id", values).fetchall() return [v for x in query for v in x.values()] @performance.timeit def get_monitored(self): values = {"profile_id": config.profile_id()} query = self.query("SELECT artist_id, artist_name FROM monitor WHERE profile_id = :profile_id", values).fetchall() return query def get_unrefreshed_artists(self): values = {"profile_id": config.profile_id()} return self.query("SELECT * FROM monitor WHERE profile_id = :profile_id AND refreshed = 0", values).fetchall() def get_unrefreshed_playlists(self): values = {"profile_id": config.profile_id()} return self.query("SELECT * FROM playlists WHERE profile_id = :profile_id AND refreshed = 0", values).fetchall() def fast_monitor(self, values): self.cursor.executemany( "INSERT OR REPLACE INTO monitor (artist_id, artist_name, bitrate, record_type, alerts, profile_id, download_path, trans_id) VALUES (:id, :name, :bitrate, :record_type, :alerts, :profile_id, :download_path, :trans_id)", values) def fast_monitor_playlist(self, values): self.cursor.executemany( "INSERT OR REPLACE INTO playlists (id, title, url, bitrate, alerts, profile_id, download_path, trans_id, monitor_artists) VALUES (:id, :title, :link, :bitrate, :alerts, :profile_id, :download_path, :trans_id, :monitor_artists)", values) def insert_multiple(self, table, values): self.cursor.executemany( f"INSERT INTO {table} (artist_id, artist_name, album_id, album_name, album_release, album_added, profile_id, future_release, trans_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", values) def remove_by_name(self, values): self.cursor.executemany(f"DELETE FROM monitor WHERE profile_id = {config.profile_id()} AND artist_name = ?", values) self.cursor.executemany(f"DELETE FROM releases WHERE profile_id = {config.profile_id()} AND artist_name = ?", values) self.commit() def remove_by_id(self, values): self.cursor.executemany(f"DELETE FROM monitor WHERE profile_id = {config.profile_id()} AND artist_id = ?", values) self.cursor.executemany(f"DELETE FROM releases WHERE profile_id = {config.profile_id()} AND artist_id = ?", values) self.commit() # @performance.timeit def remove_specific_releases(self, values): self.query(f"DELETE FROM releases WHERE album_release > :tm_date AND profile_id = {config.profile_id()}", values) def add_extra_release_info(self, values): self.new_transaction() sql = ("UPDATE releases SET label = :label WHERE album_id = :id AND " f"profile_id = {config.profile_id()}") self.cursor.executemany(sql, values) ================================================ FILE: deemon/core/dmi.py ================================================ import logging import sys from pathlib import Path import deemix import deemix.utils.localpaths as localpaths from deemix import generateDownloadObject from deemix.downloader import Downloader from deemix.settings import load as LoadSettings from deemix.types.DownloadObjects import Collection from deemix.utils import formatListener, pathtemplates from deezer import Deezer from deezer.api import APIError from deezer.gw import GWAPIError from deezer.utils import map_user_playlist, LyricsStatus, map_track from deemon.core import notifier from deemon.core.config import Config as config from deemon.core.db import Database logger = logging.getLogger(__name__) class DeemixLogListener: @classmethod def send(cls, key, value=None): if isinstance(value, dict): if value.get('failed') and value['failed'] == True: if value.get('stack') and "WrongGeolocation" in value['stack']: logger.error(f" [!] Not available in your country: {value['data']['title']} by {value['data']['artist']}") else: logger.error(f" [!] Error while downloading {value['data']['title']} by {value['data']['artist']}: {value['error']}") if config.halt_download_on_error(): logger.info("[X] Exiting due to halt_download_on_error being set to True in config.") sys.exit() log_string = formatListener(key, value) if config.debug_mode(): if log_string: logger.debug(f"[DEEMIX] {log_string}") class DeemixInterface: def __init__(self): logger.debug("Initializing deemix library") self.db = Database() self.dz = Deezer() if config.deemix_path() == "": self.config_dir = localpaths.getConfigFolder() else: self.config_dir = Path(config.deemix_path()) self.dx_settings = LoadSettings(self.config_dir) logger.debug("deemix " + deemix.__version__) logger.debug(f"deemix config path: {self.config_dir}") def download_url(self, url, bitrate, download_path, override_deemix=True): listener = DeemixLogListener() if override_deemix: deemix.generatePlaylistItem = self.generatePlaylistItem if download_path: self.dx_settings['downloadLocation'] = download_path logger.debug(f"deemix download path set to: {self.dx_settings['downloadLocation']}") links = [] for link in url: if ';' in link: for l in link.split(";"): links.append(l) else: links.append(link) for link in links: download_object = generateDownloadObject(self.dz, link, bitrate, listener=listener) if isinstance(download_object, list): for obj in download_object: Downloader(self.dz, obj, self.dx_settings, listener=listener).start() else: Downloader(self.dz, download_object, self.dx_settings, listener=listener).start() def deezer_acct_type(self): user_session = self.dz.get_session()['current_user'] if user_session.get('can_stream_lossless'): logger.debug("Deezer account connected and supports lossless") config.set('deezer_quality', 'lossless', validate=False) elif user_session.get('can_stream_hq'): logger.debug("Deezer account connected and supports high quality") config.set('deezer_quality', 'hq', validate=False) else: logger.warning("Deezer account connected but only supports 128k") config.set('deezer_quality', 'lq', validate=False) def verify_arl(self, arl): if not self.dz.login_via_arl(arl): print("FAILED") logger.debug(f"ARL Failed: {arl}") return False self.deezer_acct_type() print("OK") logger.debug("ARL is valid") return True def login(self): failed_logins = 0 logger.debug("Looking for ARL...") if config.arl(): logger.debug("ARL found in deemon config") print(":: Found ARL in deemon config, checking... ", end="") if self.verify_arl(config.arl()): return True else: logger.error("Unable to login using ARL found in deemon config") failed_logins += 1 else: logger.debug("ARL was not found in deemon config, checking if deemix has it...") if self.config_dir.is_dir(): if Path(self.config_dir / '.arl').is_file(): with open(self.config_dir / '.arl', 'r') as f: arl_from_file = f.readline().rstrip("\n") logger.debug("ARL found in deemix config") print(":: Found ARL in deemix .arl file, checking... ", end="") if self.verify_arl(arl_from_file): return True else: logger.error("Unable to login using ARL found in deemix config directory") failed_logins += 1 else: logger.debug(f"ARL not found in {self.config_dir}") else: logger.error(f"ARL directory {self.config_dir} was not found") if failed_logins > 1: notification = notifier.Notify() notification.expired_arl() else: logger.error("No ARL was found, aborting...") return False def generatePlaylistItem(self, dz, link_id, bitrate, playlistAPI=None, playlistTracksAPI=None): if not playlistAPI: if not str(link_id).isdecimal(): raise InvalidID(f"https://deezer.com/playlist/{link_id}") # Get essential playlist info try: playlistAPI = dz.api.get_playlist(link_id) except APIError: playlistAPI = None # Fallback to gw api if the playlist is private if not playlistAPI: try: userPlaylist = dz.gw.get_playlist_page(link_id) playlistAPI = map_user_playlist(userPlaylist['DATA']) except GWAPIError as e: raise GenerationError(f"https://deezer.com/playlist/{link_id}", str(e)) from e # Check if private playlist and owner if not playlistAPI.get('public', False) and playlistAPI['creator']['id'] != str(self.dz.current_user['id']): logger.warning("You can't download others private playlists.") raise NotYourPrivatePlaylist(f"https://deezer.com/playlist/{link_id}") if not playlistTracksAPI: playlistTracksAPI = dz.gw.get_playlist_tracks(link_id) playlistAPI['various_artist'] = dz.api.get_artist(5080) # Useful for save as compilation totalSize = len(playlistTracksAPI) playlistAPI['nb_tracks'] = totalSize collection = [] for pos, trackAPI in enumerate(playlistTracksAPI, start=1): # # BEGIN DEEMON PATCH # vals = {'track_id': trackAPI['SNG_ID'], 'playlist_id': playlistAPI['id']} sql = "SELECT * FROM 'playlist_tracks' WHERE track_id = :track_id AND playlist_id = :playlist_id" result = self.db.query(sql, vals).fetchone() if result: continue # # END DEEMON PATCH # trackAPI = map_track(trackAPI) if trackAPI['explicit_lyrics']: playlistAPI['explicit'] = True if 'track_token' in trackAPI: del trackAPI['track_token'] trackAPI['position'] = pos collection.append(trackAPI) if 'explicit' not in playlistAPI: playlistAPI['explicit'] = False return Collection({ 'type': 'playlist', 'id': link_id, 'bitrate': bitrate, 'title': playlistAPI['title'], 'artist': playlistAPI['creator']['name'], 'cover': playlistAPI['picture_small'][:-24] + '/75x75-000000-80-0-0.jpg', 'explicit': playlistAPI['explicit'], 'size': totalSize, 'collection': { 'tracks': collection, 'playlistAPI': playlistAPI } }) class GenerationError(Exception): def __init__(self, link, message, errid=None): super().__init__() self.link = link self.message = message self.errid = errid def toDict(self): return { 'link': self.link, 'error': self.message, 'errid': self.errid } class InvalidID(GenerationError): def __init__(self, link): super().__init__(link, "Link ID is invalid!", "invalidID") class NotYourPrivatePlaylist(GenerationError): def __init__(self, link): super().__init__(link, "You can't download others private playlists.", "notYourPrivatePlaylist") ================================================ FILE: deemon/core/exceptions.py ================================================ class ValueNotAllowed(Exception): pass class PropertyTypeMismatch(Exception): pass class UnknownValue(Exception): pass ================================================ FILE: deemon/core/logger.py ================================================ import logging from logging.handlers import RotatingFileHandler import tqdm LOG_FORMATS = { 'DEBUG': '%(asctime)s [%(levelname)s] %(name)s: %(message)s', 'INFO': '%(message)s', } LOG_DATE = '%Y-%m-%d %H:%M:%S' class tqdmStream(object): @classmethod def write(cls, msg): tqdm.tqdm.write(msg, end='') def setup_logger(log_level='DEBUG', log_file=None): """ Configure logging for the deemon application """ logger = logging.getLogger() logger.setLevel(logging.DEBUG) deemon_logger = logging.getLogger("deemon") deemon_logger.setLevel(logging.DEBUG) urllib3_logger = logging.getLogger("urllib3") urllib3_logger.setLevel(logging.ERROR) spotipy_logger = logging.getLogger("spotipy") spotipy_logger.setLevel(logging.INFO) del logger.handlers[:] if log_file is not None: rotate = RotatingFileHandler(log_file, maxBytes=10485760, backupCount=0, encoding="utf-8") rotate.setLevel(logging.DEBUG) rotate.setFormatter(logging.Formatter(LOG_FORMATS['DEBUG'], datefmt=LOG_DATE)) logger.addHandler(rotate) stream = logging.StreamHandler(stream=tqdmStream) stream.setLevel(log_level) stream.setFormatter(logging.Formatter(LOG_FORMATS[log_level], datefmt=LOG_DATE)) deemon_logger.addHandler(stream) ================================================ FILE: deemon/core/notifier.py ================================================ import logging import platform import smtplib import ssl from datetime import datetime from email.message import EmailMessage from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.utils import formataddr, formatdate import pkgutil from deemon import __version__ from deemon.core.config import Config as config logger = logging.getLogger(__name__) class Notify: def __init__(self, new_releases: list = None): logger.debug(f"Sending notification for {new_releases} release(s)") self.subject = "deemon Notification" self.releases = new_releases def send(self, body=None, test=False): """ Send email notification message """ if not all([config.smtp_server(), config.smtp_port(), config.smtp_user(), config.smtp_pass(), config.smtp_sender(), config.smtp_recipient()]): if test: logger.info(" [!] Unable to send test notification. Please configure " "email server settings and provide recipient address.") logger.debug("Email not configured, no notifications will be sent") return False if not body: body = self.html_message() context = ssl.create_default_context() logger.debug("Sending notification email") logger.debug(f"Using server: {config.smtp_server()}:{config.smtp_port()}") if config.smtp_starttls(): with smtplib.SMTP(config.smtp_server(), config.smtp_port()) as server: server.starttls() server.login(config.smtp_user(), config.smtp_pass()) server.sendmail(config.smtp_sender(), config.smtp_recipient(), body.as_string()) logger.debug("Email notification has been sent") else: with smtplib.SMTP_SSL(config.smtp_server(), config.smtp_port(), context=context) as server: server.login(config.smtp_user(), config.smtp_pass()) server.sendmail(config.smtp_sender(), config.smtp_recipient(), body.as_string()) logger.debug("Email notification has been sent") def construct_header(self, is_plain_text=True, subject=None): subject = subject or self.subject if is_plain_text: message = EmailMessage() else: message = MIMEMultipart('mixed') message['To'] = config.smtp_recipient() message['From'] = formataddr(('deemon', config.smtp_sender())) message['Subject'] = subject message['Date'] = formatdate(localtime=True) return message def html_message(self): """ Builds HTML messages """ html_body = MIMEText(self.html_new_releases(), 'html') msg = self.construct_header(is_plain_text=False) msg.attach(html_body) return msg def test(self): """ Verify SMTP settings by sending test email """ msg = self.construct_header(subject="deemon Test Notification") msg.set_content("Congrats! You'll now receive new release notifications.") self.send(msg, test=True) def expired_arl(self): """ Notify user of expired ARL """ msg = self.construct_header(subject="deemon - ARL expired") msg.set_content("Your ARL has expired. Please update your ARL to receive new releases.") self.send(msg) def expired_sub(self): """ Notify user of expired subscription """ msg = self.construct_header(subject="deemon - Subscription expired") msg.set_content("Your Deezer subscription appears to have expired.") self.send(msg) def plaintext_message(self) -> str: """ Plaintext version of email to send """ message = "The following new releases were detected:\n\n" for release in self.releases: release_date_ts = datetime.strptime(release["release_date"], "%Y-%m-%d") release_date_str = datetime.strftime(release_date_ts, "%A, %B %-d") message += f"\n{release_date_str}\n" for album in release["releases"]: message += f"+ {album['artist']} - {album['album']}\n" return message def html_new_releases(self): app_version = f"deemon {__version__}" py_version = f"python {platform.python_version()}" sys_version = f"{platform.system()} {platform.release()}" new_release_list_spacer = f"""

 

""" all_new_releases = "" self.releases = sorted(self.releases, key=lambda x: x['release_date'], reverse=True) new_release_count = 0 for release in self.releases: if all_new_releases != "": all_new_releases += new_release_list_spacer release_date_ts = datetime.strptime(release["release_date"], "%Y-%m-%d") release_date_str = datetime.strftime(release_date_ts, "%A, %B %d").replace(" 0", " ") new_release_list_header = f"""
{release_date_str}
""" new_release_list_item = "" for album in release["releases"]: new_release_count += 1 if album['record_type'].lower() == "ep": record_type = "EP" else: record_type = album['record_type'].title() if not album['track_num']: album_info = record_type else: album_info = f"{record_type} | {album['track_num']} track(s)" new_release_list_item += f"""
{album['artist']}
{album_info}
""" all_new_releases += new_release_list_header + new_release_list_item if new_release_count > 1: self.subject = f"{str(new_release_count)} new releases found!" else: self.subject = f"1 new release found!" html_output = pkgutil.get_data('deemon', 'assets/index.html').decode('ascii') if config.update_available(): html_output = html_output.replace("{UPDATE_MESSAGE}", f"Update to v{config.update_available()} is now available!") else: html_output = html_output.replace("{UPDATE_MESSAGE}", "") html_output = html_output.replace("{VIEW_UPDATE_MESSAGE}", "display:none;") html_output = html_output.replace("{NEW_RELEASE_COUNT}", str(new_release_count)) html_output = html_output.replace("{NEW_RELEASE_LIST}", all_new_releases) html_output = html_output.replace("{DEEMON_VER}", app_version) html_output = html_output.replace("{PY_VER}", py_version) html_output = html_output.replace("{SYS_VER}", sys_version) return html_output ================================================ FILE: deemon/utils/__init__.py ================================================ ================================================ FILE: deemon/utils/dataprocessor.py ================================================ import logging from csv import reader logger = logging.getLogger(__name__) def read_file_as_csv(file, split_new_line=True): with open(file, 'r', encoding="utf-8-sig", errors="replace") as f: make_csv = f.read() if split_new_line: csv_to_list = make_csv.split('\n') else: csv_to_list = make_csv.split(', ') sorted_list = sorted(list(filter(None, csv_to_list))) sorted_list = list(set(sorted_list)) try: sorted_list = [int(s) for s in sorted_list] except ValueError: pass return sorted_list def process_input_file(artist_list): logger.debug("Processing file contents") try: artists = [int(x) for x in artist_list] total_artist_count = len(artists) logger.debug(f"File detected as containing {total_artist_count} artist IDs") logger.debug("Checking for duplicates") artists_removed_duplicates = list(set(artists)) new_artists_count = len(artists_removed_duplicates) duplicates = total_artist_count - new_artists_count if duplicates: logger.debug(f"Removed {duplicates} duplicate(s)") else: logger.debug("No duplicates found, continuing...") except ValueError: artists = [x for x in artist_list] total_artist_count = len(artists) logger.debug(f"File detected as containing {len(artists)} artist names") artists_removed_duplicates = list(set(artists)) new_artists_count = len(artists_removed_duplicates) duplicates = total_artist_count - new_artists_count if duplicates: logger.debug(f"Removed {duplicates} duplicate(s)") logger.info(f"Detected {new_artists_count} unique artists") return artists_removed_duplicates def csv_to_list(all_artists) -> list: """ Separate artists and replace delimiter to find artists containing commas in their name """ all_artists = [str(x) for x in all_artists] processed_artists = [] for artist in all_artists: if artist[-1] == ',': processed_artists.append(artist[:-1] + "|") else: processed_artists.append(artist) processed_artists = ' '.join(processed_artists) processed_artists = processed_artists.split('|') result = [] csv_artists = reader(processed_artists, delimiter="|") for line in csv_artists: combined_line = ([x.lstrip() for x in line]) result.append(','.join(combined_line)) return (result) ================================================ FILE: deemon/utils/dates.py ================================================ import logging import time from datetime import datetime, date logger = logging.getLogger(__name__) def get_todays_date(): now_ts = int(time.time()) today_date = datetime.utcfromtimestamp(now_ts).strftime('%Y-%m-%d') return today_date def generate_date_filename(prefix: str): return prefix + datetime.today().strftime('%Y%m%d-%H%M%S') def get_max_release_date(days): day_in_secs = 86400 input_days_in_secs = days * day_in_secs max_date_ts = int(time.time()) - input_days_in_secs max_date = datetime.utcfromtimestamp(max_date_ts).strftime('%Y-%m-%d') return max_date def get_year(release_date: str): return datetime.strptime(release_date, '%Y-%m-%d').year def format_date_string(d: str): date_string = datetime.strptime(d, "%Y-%m-%d") return datetime.strftime(date_string, "%Y-%m-%d") def ui_date(d: datetime): return datetime.strftime(d, '%b %d, %Y') def str_to_datetime_obj(d: str) -> datetime: if d == "0000-00-00": d = "1980-01-01" return datetime.strptime(d, "%Y-%m-%d") def get_friendly_date(d: int): input_date = datetime.fromtimestamp(d).date() input_time = datetime.fromtimestamp(d).time() today = date.today() delta = today - input_date if delta.days == 0: try: return f"{input_time.strftime('%-I:%M %p')}" except ValueError: # Gotta keep Windows happy... return f"{input_time.strftime('%#I:%M %p')}" elif delta.days == 1: try: return f"Yesterday, {input_time.strftime('%-I:%M %p')}" except ValueError: # Gotta keep Windows happy... return f"Yesterday, {input_time.strftime('%#I:%M %p')}" elif 1 < delta.days < 7: try: return input_date.strftime("%A, ") + input_time.strftime("%-I:%M %p") except ValueError: # Gotta keep Windows happy... return input_date.strftime("%A, ") + input_time.strftime("%#I:%M %p") else: try: return input_date.strftime("%Y-%m-%d - ") + input_time.strftime("%-I:%M %p") except ValueError: # Gotta keep Windows happy... return input_date.strftime("%Y-%m-%d - ") + input_time.strftime("%#I:%M %p") ================================================ FILE: deemon/utils/performance.py ================================================ import logging import time logger = logging.getLogger(__name__) def timeit(method): def timed(*args, **kwargs): ts = time.time() result = method(*args, **kwargs) te = time.time() logger.debug(f"{method.__name__} finished in ({str((te - ts))})") return result return timed def operation_time(start_time): end_time = int(time.time()) duration = end_time - start_time output = time.strftime("%H:%M:%S", time.gmtime(duration)) logger.info(f"Operation completed in {output}") ================================================ FILE: deemon/utils/startup.py ================================================ import logging import os import sys from pathlib import Path import requests from packaging.version import parse as parse_version logger = logging.getLogger(__name__) def get_appdata_root(): home_dir = Path.home() if os.getenv("XDG_CONFIG_HOME"): appdata_dir = Path(os.getenv("XDG_CONFIG_HOME")) elif os.getenv("APPDATA"): appdata_dir = Path(os.getenv("APPDATA")) elif sys.platform.startswith('darwin'): appdata_dir = home_dir / 'Library' / 'Application Support' else: appdata_dir = home_dir / '.config' return appdata_dir def get_appdata_dir(): """ Get appdata directory where configuration and data is stored """ return get_appdata_root() / 'deemon' def get_backup_dir(): return Path(get_appdata_dir() / "backups") def init_appdata_dir(appdata): Path(appdata / 'logs').mkdir(parents=True, exist_ok=True) Path(appdata / 'backups').mkdir(exist_ok=True) def delete_appdata(appdata): import shutil try: shutil.rmtree(appdata) except OSError as e: logger.info(f"Error while deleting path: {e}") def reinit_appdata_dir(appdata): if appdata.exists(): logger.info("Deleting existing application data directory (config, database, etc.)") delete_appdata(appdata) logger.info("Initializing new application data directory...") init_appdata_dir(appdata) def get_config(): return get_appdata_dir() / 'config.json' def get_database(): return get_appdata_dir() / 'deemon.db' def get_log_file(): """ Get path to log file """ return Path(get_appdata_dir() / 'logs' / 'deemon.log') def get_latest_version(release_type): latest_ver = "https://pypi.org/pypi/deemon/json" try: response = requests.get(latest_ver) except requests.exceptions.ConnectionError: return latest_stable = parse_version(response.json()['info']['version']) if release_type == "beta": all_releases = [parse_version(x) for x in response.json()['releases']] sorted_releases = sorted(all_releases, reverse=True) for release in sorted_releases: if "b" in str(release) or "rc" in str(release): if release > latest_stable: return release else: return latest_stable else: return latest_stable def get_changelog(ver: str): try: response = requests.get("https://api.github.com/repos/digitalec/" "deemon/releases") except requests.exceptions.ConnectionError: return print("Unable to reach GitHub API") for release in response.json(): if release['name'] == ver: return print(release['body']) return print(f"Changelog for v{ver} was not found.") ================================================ FILE: deemon/utils/ui.py ================================================ import os TQDM_FORMAT = ":: {desc} {percentage:3.0f}%" def get_progress_bar_size() -> int: try: screen_size = int(os.get_terminal_size().columns) except OSError: screen_size = 80 dynamic_size = int(screen_size / 4) if dynamic_size > 30: return 30 elif dynamic_size < 16: return 16 else: return dynamic_size def set_progress_bar_text(msg: str, max_length: int) -> str: max_cols = get_progress_bar_size() max_length += 11 if max_length < max_cols: max_cols = max_length while len(msg) < max_cols: msg += " " while len(msg) > max_cols: msg = msg[:-1] msg += "..." return msg ================================================ FILE: deemon/utils/validate.py ================================================ import logging from datetime import datetime logger = logging.getLogger(__name__) def validate_date(d): try: return datetime.strptime(d, '%Y-%m-%d') except ValueError: return False ================================================ FILE: docs/_config.yml ================================================ remote_theme: pmarsceill/just-the-docs ================================================ FILE: docs/_sass/custom/custom.scss ================================================ @import url('https://fonts.googleapis.com/css2?family=Baloo+Tammudu+2:wght@500&display=swap'); .site-title { font-family: 'Baloo Tammudu 2', cursive; color: #6200EE; } ================================================ FILE: docs/docs/automations/automations.md ================================================ --- layout: default title: Automations nav_order: 5 has_children: true permalink: /docs/automations --- ================================================ FILE: docs/docs/automations/cron.md ================================================ --- layout: default title: cron (Linux/macOS) parent: Automations nav_order: 1 --- # cron {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- A cron job is the ideal way to run deemon in regular intervals to check for new releases: ## Check for New Releases This example checks for new releases every day at 06:00: ```bash $ crontab -l 0 6 * * * /home/user/.local/bin/deemon refresh ``` ================================================ FILE: docs/docs/automations/task-scheduler.md ================================================ --- layout: default title: Task Scheduler (Windows) parent: Automations --- # Task Scheduler {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- ## Create a New Task ![task-scheduler-1](../assets/win-setup-1.png) Inside Task Scheduler, select the **Action** menu button and click _Create Task..._ ## Name the Task ![task-scheduler-2](../assets/win-setup-2.png) Give the task a name so you'll know what it's for. In this case, this task will be setup to run `deemon refresh` so that's what I've named it. ## Setup the Schedule ![task-scheduler-3](../assets/win-setup-3.png) This is where Task Scheduler becomes a little cumbersome. If you want to run a task every few minutes/hours, you have to select _Daily_ (1) and under _Advanced settings_ check the box to repeat the task (2). The drop down box for _Repeat task every:_ doesn't give many options, but thankfully you can type in your preference. I went ahead and put in _12 hours_ with a duration of _indefinitely_. With these settings, our task will run every 12 hours until we disable it. ## Set the Application and Arguments ![task-scheduler-4](../assets/win-setup-4.png) Under _Program/script_ (1) we need to set the **absolute path** to deemon: `_%localappdata%\Programs\Python\Python39\Scripts\deemon.exe` Under _Add arguments (optional)_ (2) is where you put the deemon command you wish to run. Since this task is being setup for doing a refresh, we only need to put _refresh_ in this box. When Task Scheduler runs this task, it would be the equivalent of typing `deemon refresh` at a command prompt. ## Configure Task Settings ![task-scheduler-5](../assets/win-setup-5.png) Here you can configure your various preferences with how to handle various situations/conditions. The most important ones are: (1) _Allow task to be run on demand_ (2) _Run task as soon as possible after a scheduled start is missed_ This will allow us to manually run the task to test it and will also run the task if your computer was off or sleeping the last scheduled run. ================================================ FILE: docs/docs/commands/backup.md ================================================ --- layout: default title: backup parent: Commands --- # backup {: .no_toc } --- Introduced in version 1.0, you can now make backups of your deemon installation including your _config.json_, _deemon.db_ and (optionally) the _logs_ directory. ## Creating a Backup A backup can be created by using the `backup` command: ```bash user@localhost:~$ deemon backup --include-logs Backed up to /home/user/.config/deemon/backups/20210603-233151.tar ``` **Warning: ** Do **NOT** send your backup to anyone unless you have personally removed all sensitive data from your configuration such as email addresses, SMTP server settings and your ARL. ## Restore a Backup To restore from an existing backup, use `backup --restore` to be presented with valid backups to restore from: ```bash user@localhost:~$ deemon backup --restore deemon Backup Manager 1. Sep 23, 2021 @ 1:32:05 AM (ver 2.0) Select a backup to restore: ``` **Note: ** You cannot restore from a backup that is newer than the version of deemon you are presently running. For example, if you create a backup while using deemon 2.9 you cannot restore it on version 2.8. However, you can restore manually by extracting the tar file and replacing the files as necessary. This is to prevent users from using an incompatible configuration or database versions. ================================================ FILE: docs/docs/commands/commands.md ================================================ --- layout: default title: Commands nav_order: 4 has_children: true permalink: /docs/commands --- # arguments {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- The following arguments may be run directly after `deemon`: `--whats-new` - prints current version release notes from GitHub `--init` - Initializes deemon's application data directory. (*If directory exists, it will be deleted!*) `--arl` - Update your ARL quickly without having to edit your config.json `-P ID`, `--profile ID` - Uses specified profile ID `-V` - Prints current version and exits `-v`, `--verbose` - Show all verbose log messages ================================================ FILE: docs/docs/commands/config.md ================================================ --- layout: default title: config parent: Commands --- # config {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- The `config` command allows you to specify a per-artist configuration that overrides _global_ and _profile_ configurations for one specific artist. If you wish to clear a particular setting for an artist, type 'none'. Providing no input leaves the setting unchanged. ```bash user@localhost:~$ deemon config ARTIST deemon Artist Configurator :: Configuring 'ARTIST' (Artist ID: ...) Bitrate [None]: 320 Record Type [None]: album Alerts [None]: true Download Path [None]: :: Save these settings? [y|N] y Artist 'ARTIST' has been updated! ``` ================================================ FILE: docs/docs/commands/download.md ================================================ --- layout: default title: download parent: Commands --- # download {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- deemon includes a command line interface to the deemix library allowing you to download directly by artist name, artist ID, album ID or URL. The `download` command inherits all global settings configured in `config.json` such as bitrate and record type. These settings can be overriden using options available with the `download` command. The `download` command is fairly straightforward and usage information including options can be found by running `deemon download --help`. Below are a few common usages of the `download` command. ### Download by artist name To download by artist name, simply run the `download` command followed by the artist's name: ```bash user@localhost:~$ deemon download John Doe ``` ### Download by artist ID To download by the artist's ID: ```bash user@localhost:~$ deemon download --artist-id 100 ``` ### Download by URL In the below example, you can download a specific URL (artist, album, track or playlist): ```bash user@localhost:~$ deemon download --url https://... ``` ### Download all monitored artists If you'd like to download all releases by all artists currently being monitored, you can use the `--monitored` option to do so: ```bash user@localhost:~$ deemon download --monitored ``` ### Download a date range Introduced in version 2.5, you can now specify a date range when downloading releases. To download releases by all monitored artists between January 1, 2022 and January 31, 2022: ```bash user@localhost:~$ deemon download --monitored --after 2021-12-13 --before 2022-02-01 ``` ================================================ FILE: docs/docs/commands/library.md ================================================ --- layout: default title: library parent: Commands --- # library {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- **Warning: This feature is a working prototype and is provided as-is. It should work but it requires accurate local metadata when querying for track/album IDs.** Starting in v2.18, deemon includes a library upgrade script to upgrade your existing collection from MP3 to FLAC by generating a file containing track/album IDs to be used with the `download` command. ## Generate Track IDs To generate a file containing track IDs: ```bash user@localhost:~$ deemon library upgrade /path/to/music/library ``` This will generate a file in the current working directory called `library_upgrade_ids.txt`. ## Generate Album IDs To generate a file containing album IDs: ```bash user@localhost:~$ deemon library upgrade -A /path/to/music/library ``` ## Specify output path of ID file ```bash user@localhost:~$ deemon library upgrade -O /path/to/save /path/to/music/library ``` ## Obey exclusions set in config.json If you'd like to obey the exclusions defined in your config.json file, add `-E` or `--allow-exclusions`. ```bash user@localhost:~$ deemon library upgrade -E /path/to/music/library ``` ## Using library_upgrade_ids.txt To process this file for downloading of the tracks/albums, use one of the following commands depending on which type of file you have generated: Track IDs: ```bash user@localhost:~$ deemon download --track-file library_upgrade_ids.txt ``` Album IDs: ```bash user@localhost:~$ deemon download --album-file library_upgrade_ids.txt ``` ================================================ FILE: docs/docs/commands/monitor.md ================================================ --- layout: default title: monitor parent: Commands --- # monitor {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- Monitoring artists is the core feature of deemon. Using the `monitor` command, you can monitor artists by name, their Deezer ID or Deezer URL. Starting with version 1.1, you can now provide multiple values in CSV format. ## Monitor by Artist Name This is the easiest way to monitor an artist but has some limitations. When using an artist name, deemon searches Deezer via an API call which returns the most likely result. In some situations you may find yourself monitoring the wrong artist. In this case, it would be best to [monitor the artist by ID](#monitor-by-artist-id). You can monitor multiple artists at once by comma separating the artist names. ```bash user@localhost:~$ deemon monitor ArtistA, ArtistB, ArtistC ``` ## Monitor by Artist ID The Artist ID is the number located in the URL after `/artist/` and can be used to monitor an artist directly. This is the most accurate way to monitor an artist because this number is unique. If monitoring by artist name doesn't give you the correct artist or an artist has more than one artist profile, this method is guaranteed to give you this exact artist. ```bash $ deemon monitor --artist-id 1234 ``` ## Monitor by URL Monitoring by URL was implemented with the intention of using it for integration with automation tools like Siri Shortcuts. ```bash $ deemon monitor --url https://www.deezer.com/us/artist/1234 ``` ## Monitor by Playlist Deemon will monitor the playlist URL, and will download any new additions to the playlist. ```bash $ deemon monitor --playlist https://www.deezer.com/en/playlist/1234 ``` ## Monitor Playlist including Playlist Artists If you'd also like to setup monitoring for artists included in your playlist, you can add `--include-artists`: ```bash $ deemon monitor --playlist https://www.deezer.com/en/playlist/1234 --include-artists ``` ## Import artists from file or directory This method imports artist names or IDs from a file (CSV or Text) or a directory and stores them in the database. >**Note**: As of version 1.3, this does not actively monitor a file or directory for changes. This strictly imports the artists. **File Method:** ```bash $ deemon monitor --import file.csv ``` **Directory Method:** ```bash $ deemon monitor --import /home/user/Music ``` ## Specify custom bitrate, record type and alerts By default, deemon uses the settings configured in the `config.json` configuration file for all operations. This can be overridden at any time by using the available options such as `--bitrate`, `--record-type` and `--alerts`. ```bash $ deemon monitor ArtistA --bitrate FLAC ``` ## Monitor and download existing releases When setting up an artist (or artists) for monitoring, you can use the `-D` or `--download` flag to also download all releases matching the configured `record_type`. ## Stop Monitoring an Artist If you no longer wish to monitor an artist, include the `--remove` flag with one of the above methods and they will be removed from the database. ```bash $ deemon monitor --remove ArtistA $ deemon monitor --remove --artist-id 1234 ``` ## Stop Monitoring a Playlist If you no longer wish to monitor an playlist, include the `--remove --playlist` flags along with the playlist URL. ```bash $ deemon monitor --remove --playlist https://www.deezer.com/en/playlist/1234 ``` ## Reset database To reset the database and remove all artists/playlists from monitoring: ```bash $ deemon monitor --reset ** ALL ARTISTS AND PLAYLISTS WILL BE REMOVED! ** Type 'reset' to confirm: reset Database has been reset ``` ## Skip Refresh If you'd like to monitor an artist and wish to bypass refreshing the database afterwards, simply use `--no-refresh`. ================================================ FILE: docs/docs/commands/profile.md ================================================ --- layout: default title: profile parent: Commands --- # profile {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- Profiles are a new feature of deemon and give you the ability to apply a set of configuration options to a completely separate set of monitored artists. Profiles were developed with the intention of keeping up-to-date on multiple, separate music libraries. The default profile cannot be deleted but can be renamed and modified. To unset a setting, type "none" when prompted. ## Viewing an Existing Profile You can view an existing profile and it's configuration by running the below command: ```bash user@localhost:~$ deemon profile default deemon Profile Editor :: Showing profile 'default' (Profile ID: 1) Name Email Alerts Bitrate Type Plex Base URL Plex Token Plex Library Download Path default ``` You can also view all existing profiles: ```bash user@localhost:~$ deemon profile  ✔ deemon Profile Editor :: Showing all profiles Name Email Alerts Bitrate Type Plex Base URL Plex Token Plex Library Download Path default Soundtracks 0 320 album /music/soundtracks Favorites 1 FLAC all /music/favorites ``` ## Managing Profiles You can add, edit and delete profiles using the following syntax: ```bash deemon profile --add NAME deemon profile --edit NAME deemon profile --delete NAME ``` ================================================ FILE: docs/docs/commands/refresh.md ================================================ --- layout: default title: refresh parent: Commands --- # refresh {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- The `refresh` command is used to check for new releases, update the database and queue new releases with deemix for download. By default, running `refresh` will refresh the both artists and playlists for the active profile. ## Refresh By executing the `refresh` command by itself, deemon will refresh the releases for all artists and playlists contained in your database. > **Note:** For large databases, this can take several minutes to complete. ```bash user@localhost:~$ deemon refresh ``` ## Refreshing a single artist The `refresh` command has the ability to refresh a single artist or your entire database. To refresh an artist, simply specify that artists name after the `refresh` command: ```bash user@localhost:~$ deemon refresh Artist Name ``` ## Refreshing a single playlist The `refresh` command also has the ability to refresh a single playlist. To refresh a playlist, specify that playlists name after the `refresh` command: ```bash user@localhost:~$ deemon refresh My Awesome Playlist ``` ## Refreshing with downloads disabled If you wish to run a refresh without downloading any releases automatically, you can specify `--skip-download`. ## Refresh to a specific date The `refresh` command has a feature developed for resetting the database to a certain point in time called _time machine_. This makes rebuilding a music collection simple and can also ensure you have all releases released after a certain date. Let's say for example you want to download all releases released on or after January 1, 2022 for your entire database. All you have to do is run _time machine_ with the date of the day prior: ```bash user@localhost:~$ deemon refresh --time-machine 2021-12-31 ``` This tells deemon to first clear any release from the database that is newer than _December 31, 2021_ and then will do a full refresh. Any releases found between _January 1, 2022_ and today's date will be queued for download. In the event a release is found with a release date in the future, deemon will save this to the database and flag it is a _future release_. Once the release date of the _future release_ has come, that release will then be queued for download. ================================================ FILE: docs/docs/commands/reset.md ================================================ --- layout: default title: reset parent: Commands --- # reset {: .no_toc } --- The `reset` command allows you to remove **ALL** monitored artists and releases regardless of profile. The `reset` command does not remove any profiles. To remove a profile, see the [profile]({{ site.baseurl }}{% link docs/commands/profile.md %}) docs. ```bash user@localhost:~$ deemon reset ** ALL ARTISTS AND PLAYLISTS WILL BE REMOVED! ** :: Type 'reset' to confirm: reset Database has been reset ``` ================================================ FILE: docs/docs/commands/rollback.md ================================================ --- layout: default title: rollback parent: Commands --- # reset {: .no_toc } --- The `rollback` command allows you to rollback the last N transactions or a specific transaction. A _transaction_ is created anytime an artist is monitored or a refresh finds new releases. This does not delete any downloaded files but can be useful in the event a download failed and you want to quickly re-download those releases. ## Rollback by last _N_ transactions ```bash user@localhost:~$ deemon rollback 2  ✔  2m 22s  Rolled back the last 2 transaction(s). ``` ## Rollback a specific transaction By default, deemon shows only the last 10 transactions. To change this, edit _rollback_view_limit_ in your config.json file to increase or lower this amount. ```bash user@localhost:~$ deemon rollback --view 1. 10:00 AM - Added Ludwig van Beethoven and 389 releases 2. Yesterday, 8:22 PM - Added Mozart and 31 releases Select specific refresh to rollback (or Enter to exit): ``` ================================================ FILE: docs/docs/commands/search.md ================================================ --- layout: default title: search parent: Commands --- # deemon Interactive Search Client (dISC) {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- The `search` command is an interactive client to search for artists and download albums or setup monitoring. To open the search client: ```bash $ deemon search ``` You will then be prompted for an artist to search for: ```bash deemon Interactive Search Client :: Enter an artist to search for: ``` You can also save yourself a step and include your query as an argument to the search command: ```bash $ deemon search ArtistA ``` ## Search Results When searching for an artist, you'll be presented with a list of results. If you have `smart_search` enabled, you will be taken to the releases menu provided that there was only one exact match. In the event more than one artist is found with the exact same name, you'll be presented with some data regarding that artist to help you determine the one you're looking for: ```bash Search results for artist: John Doe 1. John Doe - Latest release: One Hit Wonder (1996) - Artist ID: 1234 - Total releases: 1 2. John Doe - Latest release: Broken (2020) - Artist ID: 3210329 - Total releases: 14 (b) Back ** Duplicate artists found ** :: Please choose an option or type 'exit': ```
Search results showing a duplicate match

Judging by the results, it's possible that both artists are the same artist but for some reason have two separate artist profiles. It's also possible that they are unrelated and both artists happen to have the same name. You can then (hopefully) tell them apart based on their _Latest release_ and also by their _Total releases_. The _Artist ID_ is provided for reference so you can visit that specific artist in a browser if you need to. ## Navigation ```bash Discography for artist: John Doe Filter by: All | Sort by: Year (desc) 1. (2020) Broken 2. (2018) Greatest Hits Filters: (*) All (a) Albums (e) EP (s) Singles - (E) Explicit (r) Reset Sort: (y) Year Desc (Y) Year Asc (t) Title Desc (T) Title Asc Mode: (S) Toggle Select (b) Back (d) Download Queue (Q) Show Queue (f) Queue Filtered (m) Stop Monitoring :: Please choose an option or type 'exit': ```
The interface when viewing an artist's discography.

At the top of the screen is information related to the menu you are currently on. In the example above you can see that you are looking at the _Discography_ for the artist _John Doe_. At the bottom of the screen you see four rows: Filters, Sort, Mode and Actions. ### Filters Here you can filter albums by type (albums, EP and singles), show only explicit releases and reset filters back to their default view. As of v2.22, you can now filter by date using `>=`, `<=` or `=` followed by the four digit year. For example, to find all releases between (and including) 2000 and 2003: Released in year 2000 or later: ```bash :: [SELECT] Please choose an option or type 'exit': >=2000 ``` Released in year 2003 or prior: ```bash :: [SELECT] Please choose an option or type 'exit': <=2003 ``` You'll notice the header has updated to reflect the filter: ```bash Filter: All | Sort: Release Date (desc) | Year: 2000 - 2003 ``` ### Sorting You can sort the results by release year and title, both ascending and descending. ### Modes There are two different _Modes_ available when it comes to selecting releases. _Regular_ mode is the default mode which displays a number to the left of each menu item. _Select_ mode allows you to select an album or track to add to the queue and is identified by two brackets: `[ ]` for unselected and `[*]` for selected. When selecting items, the prompt will change to let you know you are in _Select_ mode and how many items are in the queue: ```bash :: [SELECT] Please choose an option or type 'exit' (1 Queued): ``` ### Actions The _Actions_ bar is a group of actions you can perform while in the current view. Below is a list of all available actions throughout the dISC interface. You can only use an option if it is available in the _Actions_ bar. |Key|Name|Description| |-|--|---| |b|Back|Go back to the previous screen| |d|Download Queue|Download items currently in queue| |Q|Show Queue|Show all items presently in the queue| |c|Clear Queue|Clear all items from the queue| |f|Queue Filtered|Queue all items currently visible on the screen| |m|Start/Stop Monitoring|Toggle monitoring status of an artist| ================================================ FILE: docs/docs/commands/show.md ================================================ --- layout: default title: show parent: Commands --- # show {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- Using the `show` command, you can currently view information pertaining to artists, playlists and new releases. You can also view artist and playlist data in CSV format and export that data to a CSV file. ## Viewing artists or playlists To show a list of monitored artists: ```bash $ deemon show artists ArtistA ArtistB ``` To show a list of monitored playlists: ```bash $ deemon show playlists Summer Hits Playlist 2022 Indie Folk ``` ## CSV, filters and exporting data The `show` command allows you to view and export more data pertaining to each artist other than just their names. ### Viewing the data in CSV format First, we'll use the `-c` or `--csv` option to see all the data available: ```bash $ deemon show artists --csv name,id,bitrate,alerts,type,path John Doe,1234,,,, ``` You can also toggle the header of the CSV output by passing `--hide-header` or `-H`: ```bash $ deemon show artists -cH John Doe,1234,,,, ``` ### Using filters to customize the CSV output Next, we can apply filters to view only certain pieces of this data. If you would like to generate a CSV file containing only the ID and artists' name, you can do so by using the `-f` or `--filter` option: ```bash $ deemon show artists --csv --filter id,name id,name 1234,John Doe ``` Notice how the data is presented in the order in which you specify the filters. Filters can also be used more than once if you find the need to do so: ```bash $ deemon show artists -cf id,name,id id,name,id 1234,John Doe,1234 ``` ### Exporting monitored artists Another option for the `show artists` command is `-e` or `--export`. This allows you to export your artists to a CSV file. You can also combine this with `--hide-headers` and `--filter` to customize the data output to meet your needs. ```bash $ deemon show artists --export artists.csv CSV data has been saved to: artists.csv $ cat artists.csv name,id,bitrate,alerts,type,path John Doe,1234,,,, ``` A common use case is backing up a list of all monitored artist IDs to a CSV file which you can then import into deemon if you ever want to rebuild your database: ```bash $ deemon show artists -Hf id -e artists.csv CSV data has been saved to: artists.csv $ cat artists.csv 1234 ``` Introduced in version 2.9 is a new alias option `-b` or `--backup`. This option does the exact same thing as the previous example but does so in a much more user-friendly and simple way: ```bash $ deemon show artists --backup artists.csv CSV data has been saved to: artists.csv $ cat artists.csv 1234 ``` ## New releases You can view a list of releases that have been detected in chronological order (newest to oldest). By default, `show releases` will display the last 7 days worth of releases. You can override this by specify a number like so: ```bash $ deemon show releases 30 ``` ## Future releases You may have seen `Pending future releases` displayed in the output of the `refresh` command. When deemon detects a release with a release date in the future, it is flagged and is stored in the database until the release date approaches. If you'd like to view these future releases, you can use the `show` command: ```bash $ deemon show releases --future ``` ================================================ FILE: docs/docs/commands/test.md ================================================ --- layout: default title: test parent: Commands --- # test {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- ## SMTP settings test To verify that your SMTP settings are correctly configured, you may use `test --email` to send a test email to yourself. If you don't receive the email, confirm your SMTP settings with your mail provider and check the logs for additional information. --- ## Exclusions test If you have opted to use exclusion patterns or keywords to filter out releases, you may test those exclusions against any release URL to identify if that URL will be appropriately filtered out: ``` Artist: Various Artists Album: Broken Bow (Remix) Checking for the following patterns: 1. (?i)\bremix\b >> ** MATCH ** Checking for the following keywords: 1. remix >> ** MATCH ** 2. deluxe >> NO MATCH 3. bonus >> NO MATCH 4. special >> NO MATCH 5. live >> NO MATCH Result: This release would be excluded ``` ================================================ FILE: docs/docs/configuration.md ================================================ --- layout: default title: Configuration nav_order: 3 --- # Configuration {: .no_toc } deemon has some specific configuration parameters that can be defined in your config.json file. {: .fs-6 .fw-300 } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- ## Configuration Location Depending on your operating system, your config.json file will be located in one of the locations below. When deemon is run with a command, it will automatically generate a default config if an existing configuration file is not present. For example: to generate this configuration file, run `deemon refresh`. - **Linux**: /home/user/.config/deemon - **macOS**: /User/user/Library/Application Support/deemon - **Windows**: %appdata%\deemon ### Default Configuration (Version 2.19.2) ```json { "check_update": 1, "debug_mode": false, "release_channel": "stable", "query_limit": 5, "smart_search": true, "rollback_view_limit": 10, "prompt_duplicates": false, "prompt_no_matches": true, "fast_api": true, "fast_api_threads": 25, "exclusions": { "enable_exclusions": true, "patterns": [], "keywords": [], }, "new_releases": { "release_max_age": 90, "include_unofficial": false, "include_compilations": false, "include_featured_in": false, }, "global": { "bitrate": "320", "alerts": false, "record_type": "all", "download_path": "", "email": "" }, "deemix": { "path": "", "arl": "", "check_account_status": true, "halt_download_on_error": false, }, "smtp_settings": { "server": "", "port": 465, "starttls": false, "username": "", "password": "", "from_addr": "" }, "plex": { "base_url": "", "ssl_verify": true, "token": "", "library": "" } } ``` ## Configuration Types There are technically three different "levels" of configuration in deemon. The first level (and default) is the "global" configuration, the second level is the "profile" configuration and the third level is the "per-artist" configuration. ### Global Configuration The global configuration is defined inside the `config.json` configuration file. This configuration is used by default when running deemon and the values defined in this configuration may be temporarily superseded by a [Profile](#profile-configuration-optional) or an [Artist](#per-artist-configuration-optional) configuration. ### Profile Configuration *(optional)* Within deemon, you can configure multiple profiles (using the `profile` command) which monitors their own sets of artists and can use settings that override the _Global Configuration_ while inheriting the settings that are not defined. You can also think of these profiles as separate users. For example, if you create a new profile and only specify a download path, that profile will inherit the other settings from the _Global Configuration_. ### Per-Artist Configuration *(optional)* Within deemon, you can configure different settings for each individual artist (using the `config` command). This gives you the flexibility to disable alerts for certain artists or specify a certain release type, bitrate or download path.
## Global Configuration This section outlines each setting available in the configuration file and their respective options. --- ### Global settings These settings affect the general operation of deemon. |Setting|Description| |-|---| |**check_update**


|This option allows you to specify how frequently (in days) to check for new updates to deemon. To disable checking for updates, change this to `0`.

| |**debug_mode**
options: _true, false_

|This option will allow you to print extra debug messages in the logs or on screen if used with `--verbose`.

| |**release_channel**
options: _stable, beta_


|When checking for updates (if enabled), this option allows you to choose what updates you are notified about. Most users will want to use _stable_. If you are interested in testing pre-release versions of deemon, you can set this to _beta_.

| |**query_limit**
options: _number_


|This option allows you to specify how many results are displayed when using the `search` command or when prompted using the `monitor` command (see _prompt_duplicates_ and _prompt_no_matches_ below).

| |**smart_search**
options: _true, false_


|This option allows you to skip the list of artist search results and proceed directly to the list of artist albums, provided there is only one exact match of the artists name (case insensitive).


| |**rollback_view_limit**


|This option allows you to specify the maximum number of transactions to display using the `rollback` command

| |**prompt_duplicates**
options: _true, false_



|When adding a new artist using the `monitor` command, deemon will choose the highest ranked artist in situations where two artists have identical names. Instead, you can set this option to `true` which will prompt you with choices including the latest release from each artist to help you better decide which is the artist you're looking for.

| |**prompt_no_matches**
options: _true, false_


|When adding a new artist using the `monitor` command, if deemon does not find an **exact** match for the artist you're searching for, it will prompt you with a list of results returned from the Deezer API.

| |**fast_api**
options: _true, false_

|In previous versions of deemon, this was referred to as the _experimental_api_ and has been the default API since version 2.1.

| |**fast_api_threads**
options: _number_

|This sets the number of threads to spawn when accessing the API. The higher the number, the faster artist data is retrieved. However, setting this number too high may result in a temporary ban of your IP address. **It is recommended to keep this number below 50.**

| --- ### Exclusion settings Exclusions can be setup to ignore releases matching a specific regular expression ("pattern") or by matching against specific keywords. You can test your exclusion settings by using the `test` command. |Setting|Description| |-|---| |**enable_exclusions**
options: _true, false_

|This setting enables exclusion patterns and keywords (below) to filter out releases.


| |**patterns**
options: _regex_

|Provide a list of regular expressions to filter out releases.


| |**keywords**
options: _true, false_

|Provide a list of keywords to filter out releases (_remix, deluxe, bonus_).


| --- ### New Release settings These settings affect what releases are filtered out by deemon. As of version 2.9, these settings are global only and cannot be configured in a profile nor per-artist configuration. |Setting|Description| |-|---| |**release_max_age**







|This lets you control how far back to consider a new release 'new' from it's actual release date. This setting is helpful if you only run a `refresh` monthly or if there is a delay in Deezer adding a release to their catalogue. If you wish to get all releases regardless of when they were released, set this to _0_.

**Note:** Version 2.8.x and earlier relied on a separate toggle _by_release_date_ which has been deprecated in favor of setting _release_max_age_ to _0_.

| |**include_unofficial**
options: _true, false_





|Each release result returned from the API includes a boolean value designating the release as either an official artist release or not. In many cases, this flag is incorrectly set resulting in some official releases not being detected.

**Warning:** If you are enabling this option on an existing database, you could potentially have hundreds or thousands of new releases detected.

| |**include_compilations**
options: _true, false_




|If you want to include all compilation albums that your artists are featured on, you can enable this.

**Warning:** If you are enabling this option on an existing database, you could potentially have hundreds or thousands of new releases detected.

| |**include_featured_in**
options: _true, false_





|Enabling this option will include every single release/track an artist is associated with. It is highly recommend to avoid using this but is included for testing purposes.

**Warning:** If you are enabling this option on an existing database, you could potentially have **_tens of thousands_** of new releases detected. Most users will NOT want this option enabled.

| --- ### Global settings These settings can be overriden within deemon using _profiles_ or by specifying a _per-artist configuration_. |Setting|Description| |-|---| |**bitrate**
options: _128, 320, FLAC_

|This option allows you to specify the bitrate used for downloads.


| |**alerts**
options: _true, false_

|Enable or disable email notification alerts when new releases are present. You must also have your email settings configured (see _[SMTP Settings](#smtp-settings)_).

| |**record_type**
options: all, album, ep, single


|This option allows you to specify what release types you wish to download upon release. Keep in mind, most EPs are labelled as albums in the Deezer API.

**Limitation:** Currently only one option at a time is allowed.

| |**download_path**





|This option allows you to specify where downloads are saved. If no path is provided, downloads will be saved in the default directory specified by deemix.

**Windows users:** When providing a path, you **must** use double backslashes: `C:\\Music` or forward slashes: `C:/Music`.

| |**email**


|This option allows you to specify the default email address to use when alerts are enabled and SMTP settings are defined.

| --- ### deemix settings These settings are needed for deemon to interface with deemix which is a third party library that does the actual downloading. |Setting|Description| |-|---| |**path**


|Specify the path to where your deemix configuration is stored. For most users, leave this blank. If you have moved the deemix configuration, you must specify that here.

| |**arl**



|This is your authorization token required by `deemix` to authenticate your Deezer account. This is stored in a cookie named `arl` in your browser after logging in to Deezer.

| |**check_account_status**
options: _true, false_



|This option allows you to force account verification before doing a refresh. If you have _bitrate_ set to FLAC and your account type is not HiFi, deemon will exit until you correct the issue (expired ARL or subscription). This option is useful for preventing low quality downloads due to an expired subscription.

| |**halt_download_on_error**
options: _true, false_

|If enabled, deemon will exit if deemix reports any errors when downloading. This prevents releases from being logged in the database so that you can try again later.

| --- ### SMTP settings These settings are needed for deemon to interface with deemix which is a third party library that does the actual downloading. |Setting|Description| |-|---| |**server**

|Server address for your mail server

| |**port**

|Port used to connect to your mail server (typically 587 or 465)

| |**starttls**
options: _true, false_
|If your mail server requires STARTTLS, you can enable that here.
| |**username**

|Username required to login to your mail server

| |**password**

|Password used to authenticate your account with your mail server

| |**from_addr**


|This is the email address your email is to be sent _from_ and typically must be a real address associated with your account on your mail server.

| --- ### Plex integration settings deemon features support for refreshing your Plex library after a download operation is complete. Plex has the ability to automatically detect changes and rescan but if you're having issues with that, this should help. |Setting|Description| |-|---| |**base_url**






|This is the URL to reach your Plex server including the port and protocol.

_Example_: http://192.168.0.2:32400

**Note:** You may need to use _https_ if your Plex setup is configured for secure connections.

| |**ssl_verify**
options: _true, false_
|If you have Plex configured to require secure connections, but have not provided a custom certificate, keep this set to _false_ to avoid SSL certificate errors.

| |**token**



|Authentication token required to connect to your Plex server

For instructions on how to find your token, [click here](https://support.plex.tv/articles/204059436-finding-an-authentication-token-x-plex-token/).

| |**library**

|The name of your Plex library to be refreshed.

| ================================================ FILE: docs/docs/installation.md ================================================ --- layout: default title: Installation nav_order: 2 --- # Installation {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- ## Step 1 - Required Dependencies In order to install and run deemon, you'll need to have at least Python 3.6 or higher installed along with the `pip` package manager. Please refer to [python.org](https://www.python.org/downloads/) for more information. ### Step 2 - Installing deemon Once you have at least Python 3.6 installed, go ahead and install deemon using `pip`. On some distributions, the `pip` command is for Python2. In this case, substitute `pip` for `pip3` in the commands below. **Windows users**: These commands should be typed in a Command Prompt, Windows Terminal or Powershell window. ```bash # Latest stable release $ pip install deemon # Latest release (including pre-release/beta) $ pip install --pre deemon ``` At this point, pip will download deemon and any other modules required to allow deemon to function. Once it's complete, use the following command to make sure deemon is installed: ```bash $ deemon -V deemon 2.19.2 ``` ## Configuration & First Use Congrats! If you've made it this far, you have successfully installed deemon. There are a few things you should configure before using deemon. Head on over to the [configuration](configuration.md) page to learn more. ================================================ FILE: docs/docs/troubleshooting/logs.md ================================================ --- layout: default title: Logs parent: Troubleshooting --- # Logs {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- If you're having issues or would like to see how things are running, you can view the log files which are located inside the [deemon config directory](/docs/configuration#configuration-file). As of version 1.0, the logs from deemix are now included (as well as other third-party modules used) in the deemon log file. ================================================ FILE: docs/docs/troubleshooting/queue.md ================================================ --- layout: default title: Queue parent: Troubleshooting --- # Queue {: .no_toc } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- Each time deemon finds new releases, they are dumped to `queue.csv` inside the deemon application directory. This provides a way to recover a failed job. To redownload a queue, you can manually extract the `album_id` or `track_id` column and save it into a new file to pass to deemon: Album IDs: `deemon download --album-file file.csv` Track IDs: `deemon download --track-file file.csv` ================================================ FILE: docs/docs/troubleshooting/troubleshooting.md ================================================ --- layout: default title: Troubleshooting nav_order: 6 has_children: true permalink: /docs/troubleshooting --- ================================================ FILE: docs/index.md ================================================ --- layout: default title: Home nav_order: 1 description: "deemon is a monitoring utility for new artist releases that can provide email alerts and automate downloading via the deemix library" permalink: / --- # deemon 2.22 Documentation {: .fs-9 } deemon is a monitoring utility for new artist releases that can provide email alerts and automate downloading via the deemix library {: .fs-6 .fw-300 } [Get started now](#getting-started){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-2 } [View it on GitHub](https://github.com/digitalec/deemon){: .btn .fs-5 .mb-4 .mb-md-0 } Version 2.22 --- ## Disclaimer deemon does not download anything by itself. It requires a third party library called *deemix* in order to do this and will be installed automatically when installed via pip. --- ## Getting started ### Dependencies deemon depends on various python modules and libraries to perform all of its functions. Please refer to the `requirements.txt` file to see what those dependencies are. ### Installation & Configuration When you're ready to install deemon, head on over to the [installation]({{ site.baseurl }}{% link docs/installation.md %}) page. Once you've installed deemon, it is important to [configure]({{ site.baseurl }}{% link docs/configuration.md %}) it properly to get the best experience. --- ## About the project deemon (**dee**zer **mon**itor) is an open source project that came from the need to stay on top of new releases by some of my favorite artists. ### License deemon is distributed by a [GPL-3.0 license](https://github.com/digitalec/deemon/blob/main/LICENSE). ================================================ FILE: requirements.txt ================================================ deemix~=3.6 packaging~=23.0 requests~=2.28.0 click~=8.1.0 setuptools~=66.1.0 wheel~=0.43.0 PlexAPI~=4.5.2 tqdm~=4.61.0 mutagen~=1.46 Unidecode~=1.3.6 ================================================ FILE: setup.py ================================================ from pathlib import Path from setuptools import setup, find_packages from deemon import __version__ with open('requirements.txt') as f: required = f.read().splitlines() HERE = Path(__file__).parent README = (HERE / "README.md").read_text() DESCRIPTION = "Monitor new releases by a specified list of artists and auto download using the deemix library" setup( name="deemon", version=__version__, author="digitalec", description=DESCRIPTION, long_description=README, long_description_content_type="text/markdown", license="GPL3", classifiers=[ "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Programming Language :: Python :: 3.8", "Operating System :: OS Independent", ], packages=find_packages(), include_package_data=True, python_requires=">=3.8", install_requires=required, url="https://github.com/digitalec/deemon", entry_points = { "console_scripts": ["deemon=deemon.__main__:main"], } )