[
{
"path": "LICENSE",
"content": " GNU AFFERO GENERAL PUBLIC LICENSE\n Version 3, 19 November 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU Affero General Public License is a free, copyleft license for\nsoftware and other kinds of works, specifically designed to ensure\ncooperation with the community in the case of network server software.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nour General Public Licenses are intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n Developers that use our General Public Licenses protect your rights\nwith two steps: (1) assert copyright on the software, and (2) offer\nyou this License which gives you legal permission to copy, distribute\nand/or modify the software.\n\n A secondary benefit of defending all users' freedom is that\nimprovements made in alternate versions of the program, if they\nreceive widespread use, become available for other developers to\nincorporate. Many developers of free software are heartened and\nencouraged by the resulting cooperation. However, in the case of\nsoftware used on network servers, this result may fail to come about.\nThe GNU General Public License permits making a modified version and\nletting the public access it on a server without ever releasing its\nsource code to the public.\n\n The GNU Affero General Public License is designed specifically to\nensure that, in such cases, the modified source code becomes available\nto the community. It requires the operator of a network server to\nprovide the source code of the modified version running there to the\nusers of that server. Therefore, public use of a modified version, on\na publicly accessible server, gives the public access to the source\ncode of the modified version.\n\n An older license, called the Affero General Public License and\npublished by Affero, was designed to accomplish similar goals. This is\na different license, not a version of the Affero GPL, but Affero has\nreleased a new version of the Affero GPL which permits relicensing under\nthis license.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU Affero General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Remote Network Interaction; Use with the GNU General Public License.\n\n Notwithstanding any other provision of this License, if you modify the\nProgram, your modified version must prominently offer all users\ninteracting with it remotely through a computer network (if your version\nsupports such interaction) an opportunity to receive the Corresponding\nSource of your version by providing access to the Corresponding Source\nfrom a network server at no charge, through some standard or customary\nmeans of facilitating copying of software. This Corresponding Source\nshall include the Corresponding Source for any work covered by version 3\nof the GNU General Public License that is incorporated pursuant to the\nfollowing paragraph.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the work with which it is combined will remain governed by version\n3 of the GNU General Public License.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU Affero General Public License from time to time. Such new versions\nwill be similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU Affero General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU Affero General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU Affero General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU Affero General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU Affero General Public License for more details.\n\n You should have received a copy of the GNU Affero General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If your software can interact with users remotely through a computer\nnetwork, you should also make sure that it provides a way for users to\nget its source. For example, if your program is a web application, its\ninterface could display a \"Source\" link that leads users to an archive\nof the code. There are many ways you could offer source, and different\nsolutions will be better for different programs; see section 13 for the\nspecific requirements.\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU AGPL, see\n.\n"
},
{
"path": "README.md",
"content": "# yolov8-object-tracking \n\nThis is compatible only with `ultralytics==8.0.0`. However, I highly recommend using the latest version of the Ultralytics package and referring to the official Ultralytics codebase here: [GitHub Repository](https://github.com/ultralytics/ultralytics/).\n\n[](https://muhammadrizwanmunawar.medium.com/train-yolov8-on-custom-data-6d28cd348262)\n\n### Steps to run Code\n\n- Clone the repository\n```bash\nhttps://github.com/RizwanMunawar/yolov8-object-tracking.git\n```\n\n- Move to the cloned folder\n\n```bash\ncd yolov8-object-tracking\n```\n\n- Install the ultralytics package\n```bash\npip install ultralytics==8.0.0\n```\n\n- Do tracking with the mentioned command below\n```bash\n#video file\npython yolo\\v8\\detect\\detect_and_trk.py model=yolov8s.pt source=\"test.mp4\" show=True\n\n#imagefile\npython yolo\\v8\\detect\\detect_and_trk.py model=yolov8m.pt source=\"path to image\"\n\n#Webcam\npython yolo\\v8\\detect\\detect_and_trk.py model=yolov8m.pt source=0 show=True\n\n#External Camera\npython yolo\\v8\\detect\\detect_and_trk.py model=yolov8m.pt source=1 show=True\n```\n\n- Output file will be created in the `runs/detect/train` with the original filename\n\n\n### Results π\n\n \n | YOLOv8s Object Tracking | \n YOLOv8m Object Tracking | \n
\n \n ![](\"https://user-images.githubusercontent.com/62513924/211671576-7d39829a-f8f5-4e25-b30a-530548c11a24.png\") | \n ![](\"https://user-images.githubusercontent.com/62513924/211672010-7415ef8b-7941-4545-8434-377d94675299.png\") | \n
\n
\n\n### Star History\n\n[](https://www.star-history.com/#RizwanMunawar/yolov8-object-tracking&type=date&legend=top-left)\n\n\n### References π\n- π https://github.com/ultralytics/ultralytics\n- π https://github.com/abewley/sort\n- π https://docs.ultralytics.com/\n\n**Some of my articles/research papers | Computer vision awesome resources for learning | How do I appear to the world? π**\n\n| Article Title & Link | Published Date |\n|-----------------------|----------------|\n| [Ultralytics YOLO11: Object Detection and Instance Segmentationπ€―](https://muhammadrizwanmunawar.medium.com/ultralytics-yolo11-object-detection-and-instance-segmentation-88ef0239a811) |  |\n| [Parking Management using Ultralytics YOLO11](https://muhammadrizwanmunawar.medium.com/parking-management-using-ultralytics-yolo11-fba4c6bc62bc) |  |\n| [My ποΈComputer Vision Hobby Projects that Yielded Earnings](https://muhammadrizwanmunawar.medium.com/my-οΈcomputer-vision-hobby-projects-that-yielded-earnings-7923c9b9eead) |  |\n| [Best Resources to Learn Computer Vision](https://muhammadrizwanmunawar.medium.com/best-resources-to-learn-computer-vision-311352ed0833) |  |\n| [Roadmap for Computer Vision Engineer](https://medium.com/augmented-startups/roadmap-for-computer-vision-engineer-45167b94518c) |  |\n| [How did I spend 2022 in the Computer Vision Field](https://www.linkedin.com/pulse/how-did-i-spend-2022-computer-vision-field-muhammad-rizwan-munawar) |  |\n| [Domain Feature Mapping with YOLOv7 for Automated Edge-Based Pallet Racking Inspections](https://www.mdpi.com/1424-8220/22/18/6927) |  |\n| [Exudate Regeneration for Automated Exudate Detection in Retinal Fundus Images](https://ieeexplore.ieee.org/document/9885192) |  |\n| [Feature Mapping for Rice Leaf Defect Detection Based on a Custom Convolutional Architecture](https://www.mdpi.com/2304-8158/11/23/3914) |  |\n| [Yolov5, Yolo-x, Yolo-r, Yolov7 Performance Comparison: A Survey](https://aircconline.com/csit/papers/vol12/csit121602.pdf) |  |\n| [Explainable AI in Drug Sensitivity Prediction on Cancer Cell Lines](https://ieeexplore.ieee.org/document/9922931) |  |\n| [Train YOLOv8 on Custom Data](https://medium.com/augmented-startups/train-yolov8-on-custom-data-6d28cd348262) |  |\n\n\n**More Information**\n\nFor more details, you can reach out to me on [Medium](https://muhammadrizwanmunawar.medium.com/) or connect with me on [LinkedIn](https://www.linkedin.com/in/muhammadrizwanmunawar/)\n"
},
{
"path": "__init__.py",
"content": "from hub import checks\r\nfrom engine.model import YOLO\r\nfrom utils import ops\r\nfrom . import v8"
},
{
"path": "models/v8/yolov8l.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 1.00 # scales module repeats\r\nwidth_multiple: 1.00 # scales convolution channels\r\n\r\n# YOLOv8.0l backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [512, True]]\r\n - [-1, 1, SPPF, [512, 5]] # 9\r\n\r\n# YOLOv8.0l head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2f, [512]] # 13\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2f, [256]] # 17 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 12], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2f, [512]] # 20 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 9], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2f, [512]] # 23 (P5/32-large)\r\n\r\n - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)\r\n"
},
{
"path": "models/v8/yolov8m.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 0.67 # scales module repeats\r\nwidth_multiple: 0.75 # scales convolution channels\r\n\r\n# YOLOv8.0m backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [768, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [768, True]]\r\n - [-1, 1, SPPF, [768, 5]] # 9\r\n\r\n# YOLOv8.0m head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2f, [512]] # 13\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2f, [256]] # 17 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 12], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2f, [512]] # 20 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 9], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2f, [768]] # 23 (P5/32-large)\r\n\r\n - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)\r\n"
},
{
"path": "models/v8/yolov8n.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 0.33 # scales module repeats\r\nwidth_multiple: 0.25 # scales convolution channels\r\n\r\n# YOLOv8.0n backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [1024, True]]\r\n - [-1, 1, SPPF, [1024, 5]] # 9\r\n\r\n# YOLOv8.0n head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2f, [512]] # 13\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2f, [256]] # 17 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 12], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2f, [512]] # 20 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 9], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2f, [1024]] # 23 (P5/32-large)\r\n\r\n - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)\r\n"
},
{
"path": "models/v8/yolov8s.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 0.33 # scales module repeats\r\nwidth_multiple: 0.50 # scales convolution channels\r\n\r\n# YOLOv8.0s backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [1024, True]]\r\n - [-1, 1, SPPF, [1024, 5]] # 9\r\n\r\n# YOLOv8.0s head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2f, [512]] # 13\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2f, [256]] # 17 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 12], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2f, [512]] # 20 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 9], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2f, [1024]] # 23 (P5/32-large)\r\n\r\n - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)\r\n"
},
{
"path": "models/v8/yolov8x.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 1.00 # scales module repeats\r\nwidth_multiple: 1.25 # scales convolution channels\r\n\r\n# YOLOv8.0x backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [512, True]]\r\n - [-1, 1, SPPF, [512, 5]] # 9\r\n\r\n# YOLOv8.0x head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2f, [512]] # 13\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2f, [256]] # 17 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 12], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2f, [512]] # 20 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 9], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2f, [512]] # 23 (P5/32-large)\r\n\r\n - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)\r\n"
},
{
"path": "models/v8/yolov8x6.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\n# Parameters\r\nnc: 80 # number of classes\r\ndepth_multiple: 1.00 # scales module repeats\r\nwidth_multiple: 1.25 # scales convolution channels\r\n\r\n# YOLOv8.0x6 backbone\r\nbackbone:\r\n # [from, repeats, module, args]\r\n - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\r\n - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\r\n - [-1, 3, C2f, [128, True]]\r\n - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\r\n - [-1, 6, C2f, [256, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\r\n - [-1, 6, C2f, [512, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 7-P5/32\r\n - [-1, 3, C2f, [512, True]]\r\n - [-1, 1, Conv, [512, 3, 2]] # 9-P6/64\r\n - [-1, 3, C2f, [512, True]]\r\n - [-1, 1, SPPF, [512, 5]] # 11\r\n\r\n# YOLOv8.0x6 head\r\nhead:\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 8], 1, Concat, [1]] # cat backbone P5\r\n - [-1, 3, C2, [512, False]] # 14\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 6], 1, Concat, [1]] # cat backbone P4\r\n - [-1, 3, C2, [512, False]] # 17\r\n\r\n - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\r\n - [[-1, 4], 1, Concat, [1]] # cat backbone P3\r\n - [-1, 3, C2, [256, False]] # 20 (P3/8-small)\r\n\r\n - [-1, 1, Conv, [256, 3, 2]]\r\n - [[-1, 17], 1, Concat, [1]] # cat head P4\r\n - [-1, 3, C2, [512, False]] # 23 (P4/16-medium)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 14], 1, Concat, [1]] # cat head P5\r\n - [-1, 3, C2, [512, False]] # 26 (P5/32-large)\r\n\r\n - [-1, 1, Conv, [512, 3, 2]]\r\n - [[-1, 11], 1, Concat, [1]] # cat head P6\r\n - [-1, 3, C2, [512, False]] # 29 (P6/64-xlarge)\r\n\r\n - [[20, 23, 26, 29], 1, Detect, [nc]] # Detect(P3, P4, P5, P6)\r\n"
},
{
"path": "nn/__init__.py",
"content": ""
},
{
"path": "nn/autobackend.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport json\r\nimport platform\r\nfrom collections import OrderedDict, namedtuple\r\nfrom pathlib import Path\r\nfrom urllib.parse import urlparse\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport torch\r\nimport torch.nn as nn\r\nfrom PIL import Image\r\n\r\nfrom yolo.utils import LOGGER, ROOT, yaml_load\r\nfrom yolo.utils.checks import check_requirements, check_suffix, check_version\r\nfrom yolo.utils.downloads import attempt_download, is_url\r\nfrom yolo.utils.ops import xywh2xyxy\r\n\r\n\r\nclass AutoBackend(nn.Module):\r\n\r\n def __init__(self, weights='yolov8n.pt', device=torch.device('cpu'), dnn=False, data=None, fp16=False, fuse=True):\r\n \"\"\"\r\n Ultralytics YOLO MultiBackend class for python inference on various backends\r\n\r\n Args:\r\n weights: the path to the weights file. Defaults to yolov8n.pt\r\n device: The device to run the model on.\r\n dnn: If you want to use OpenCV's DNN module to run the inference, set this to True. Defaults to\r\n False\r\n data: a dictionary containing the following keys:\r\n fp16: If true, will use half precision. Defaults to False\r\n fuse: whether to fuse the model or not. Defaults to True\r\n\r\n Supported format and their usage:\r\n | Platform | weights |\r\n |-----------------------|------------------|\r\n | PyTorch | *.pt |\r\n | TorchScript | *.torchscript |\r\n | ONNX Runtime | *.onnx |\r\n | ONNX OpenCV DNN | *.onnx --dnn |\r\n | OpenVINO | *.xml |\r\n | CoreML | *.mlmodel |\r\n | TensorRT | *.engine |\r\n | TensorFlow SavedModel | *_saved_model |\r\n | TensorFlow GraphDef | *.pb |\r\n | TensorFlow Lite | *.tflite |\r\n | TensorFlow Edge TPU | *_edgetpu.tflite |\r\n | PaddlePaddle | *_paddle_model |\r\n \"\"\"\r\n super().__init__()\r\n w = str(weights[0] if isinstance(weights, list) else weights)\r\n nn_module = isinstance(weights, torch.nn.Module)\r\n pt, jit, onnx, xml, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle, triton = self._model_type(w)\r\n fp16 &= pt or jit or onnx or engine or nn_module # FP16\r\n nhwc = coreml or saved_model or pb or tflite or edgetpu # BHWC formats (vs torch BCWH)\r\n stride = 32 # default stride\r\n cuda = torch.cuda.is_available() and device.type != 'cpu' # use CUDA\r\n if not (pt or triton or nn_module):\r\n w = attempt_download(w) # download if not local\r\n\r\n # NOTE: special case: in-memory pytorch model\r\n if nn_module:\r\n model = weights.to(device)\r\n model = model.fuse() if fuse else model\r\n names = model.module.names if hasattr(model, 'module') else model.names # get class names\r\n model.half() if fp16 else model.float()\r\n self.model = model # explicitly assign for to(), cpu(), cuda(), half()\r\n pt = True\r\n elif pt: # PyTorch\r\n from nn.tasks import attempt_load_weights\r\n model = attempt_load_weights(weights if isinstance(weights, list) else w,\r\n device=device,\r\n inplace=True,\r\n fuse=fuse)\r\n stride = max(int(model.stride.max()), 32) # model stride\r\n names = model.module.names if hasattr(model, 'module') else model.names # get class names\r\n model.half() if fp16 else model.float()\r\n self.model = model # explicitly assign for to(), cpu(), cuda(), half()\r\n elif jit: # TorchScript\r\n LOGGER.info(f'Loading {w} for TorchScript inference...')\r\n extra_files = {'config.txt': ''} # model metadata\r\n model = torch.jit.load(w, _extra_files=extra_files, map_location=device)\r\n model.half() if fp16 else model.float()\r\n if extra_files['config.txt']: # load metadata dict\r\n d = json.loads(extra_files['config.txt'],\r\n object_hook=lambda d: {int(k) if k.isdigit() else k: v\r\n for k, v in d.items()})\r\n stride, names = int(d['stride']), d['names']\r\n elif dnn: # ONNX OpenCV DNN\r\n LOGGER.info(f'Loading {w} for ONNX OpenCV DNN inference...')\r\n check_requirements('opencv-python>=4.5.4')\r\n net = cv2.dnn.readNetFromONNX(w)\r\n elif onnx: # ONNX Runtime\r\n LOGGER.info(f'Loading {w} for ONNX Runtime inference...')\r\n check_requirements(('onnx', 'onnxruntime-gpu' if cuda else 'onnxruntime'))\r\n import onnxruntime\r\n providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] if cuda else ['CPUExecutionProvider']\r\n session = onnxruntime.InferenceSession(w, providers=providers)\r\n output_names = [x.name for x in session.get_outputs()]\r\n meta = session.get_modelmeta().custom_metadata_map # metadata\r\n if 'stride' in meta:\r\n stride, names = int(meta['stride']), eval(meta['names'])\r\n elif xml: # OpenVINO\r\n LOGGER.info(f'Loading {w} for OpenVINO inference...')\r\n check_requirements('openvino') # requires openvino-dev: https://pypi.org/project/openvino-dev/\r\n from openvino.runtime import Core, Layout, get_batch # noqa\r\n ie = Core()\r\n if not Path(w).is_file(): # if not *.xml\r\n w = next(Path(w).glob('*.xml')) # get *.xml file from *_openvino_model dir\r\n network = ie.read_model(model=w, weights=Path(w).with_suffix('.bin'))\r\n if network.get_parameters()[0].get_layout().empty:\r\n network.get_parameters()[0].set_layout(Layout(\"NCHW\"))\r\n batch_dim = get_batch(network)\r\n if batch_dim.is_static:\r\n batch_size = batch_dim.get_length()\r\n executable_network = ie.compile_model(network, device_name=\"CPU\") # device_name=\"MYRIAD\" for Intel NCS2\r\n stride, names = self._load_metadata(Path(w).with_suffix('.yaml')) # load metadata\r\n elif engine: # TensorRT\r\n LOGGER.info(f'Loading {w} for TensorRT inference...')\r\n import tensorrt as trt # https://developer.nvidia.com/nvidia-tensorrt-download\r\n check_version(trt.__version__, '7.0.0', hard=True) # require tensorrt>=7.0.0\r\n if device.type == 'cpu':\r\n device = torch.device('cuda:0')\r\n Binding = namedtuple('Binding', ('name', 'dtype', 'shape', 'data', 'ptr'))\r\n logger = trt.Logger(trt.Logger.INFO)\r\n with open(w, 'rb') as f, trt.Runtime(logger) as runtime:\r\n model = runtime.deserialize_cuda_engine(f.read())\r\n context = model.create_execution_context()\r\n bindings = OrderedDict()\r\n output_names = []\r\n fp16 = False # default updated below\r\n dynamic = False\r\n for i in range(model.num_bindings):\r\n name = model.get_binding_name(i)\r\n dtype = trt.nptype(model.get_binding_dtype(i))\r\n if model.binding_is_input(i):\r\n if -1 in tuple(model.get_binding_shape(i)): # dynamic\r\n dynamic = True\r\n context.set_binding_shape(i, tuple(model.get_profile_shape(0, i)[2]))\r\n if dtype == np.float16:\r\n fp16 = True\r\n else: # output\r\n output_names.append(name)\r\n shape = tuple(context.get_binding_shape(i))\r\n im = torch.from_numpy(np.empty(shape, dtype=dtype)).to(device)\r\n bindings[name] = Binding(name, dtype, shape, im, int(im.data_ptr()))\r\n binding_addrs = OrderedDict((n, d.ptr) for n, d in bindings.items())\r\n batch_size = bindings['images'].shape[0] # if dynamic, this is instead max batch size\r\n elif coreml: # CoreML\r\n LOGGER.info(f'Loading {w} for CoreML inference...')\r\n import coremltools as ct\r\n model = ct.models.MLModel(w)\r\n elif saved_model: # TF SavedModel\r\n LOGGER.info(f'Loading {w} for TensorFlow SavedModel inference...')\r\n import tensorflow as tf\r\n keras = False # assume TF1 saved_model\r\n model = tf.keras.models.load_model(w) if keras else tf.saved_model.load(w)\r\n elif pb: # GraphDef https://www.tensorflow.org/guide/migrate#a_graphpb_or_graphpbtxt\r\n LOGGER.info(f'Loading {w} for TensorFlow GraphDef inference...')\r\n import tensorflow as tf\r\n\r\n def wrap_frozen_graph(gd, inputs, outputs):\r\n x = tf.compat.v1.wrap_function(lambda: tf.compat.v1.import_graph_def(gd, name=\"\"), []) # wrapped\r\n ge = x.graph.as_graph_element\r\n return x.prune(tf.nest.map_structure(ge, inputs), tf.nest.map_structure(ge, outputs))\r\n\r\n def gd_outputs(gd):\r\n name_list, input_list = [], []\r\n for node in gd.node: # tensorflow.core.framework.node_def_pb2.NodeDef\r\n name_list.append(node.name)\r\n input_list.extend(node.input)\r\n return sorted(f'{x}:0' for x in list(set(name_list) - set(input_list)) if not x.startswith('NoOp'))\r\n\r\n gd = tf.Graph().as_graph_def() # TF GraphDef\r\n with open(w, 'rb') as f:\r\n gd.ParseFromString(f.read())\r\n frozen_func = wrap_frozen_graph(gd, inputs=\"x:0\", outputs=gd_outputs(gd))\r\n elif tflite or edgetpu: # https://www.tensorflow.org/lite/guide/python#install_tensorflow_lite_for_python\r\n try: # https://coral.ai/docs/edgetpu/tflite-python/#update-existing-tf-lite-code-for-the-edge-tpu\r\n from tflite_runtime.interpreter import Interpreter, load_delegate\r\n except ImportError:\r\n import tensorflow as tf\r\n Interpreter, load_delegate = tf.lite.Interpreter, tf.lite.experimental.load_delegate,\r\n if edgetpu: # TF Edge TPU https://coral.ai/software/#edgetpu-runtime\r\n LOGGER.info(f'Loading {w} for TensorFlow Lite Edge TPU inference...')\r\n delegate = {\r\n 'Linux': 'libedgetpu.so.1',\r\n 'Darwin': 'libedgetpu.1.dylib',\r\n 'Windows': 'edgetpu.dll'}[platform.system()]\r\n interpreter = Interpreter(model_path=w, experimental_delegates=[load_delegate(delegate)])\r\n else: # TFLite\r\n LOGGER.info(f'Loading {w} for TensorFlow Lite inference...')\r\n interpreter = Interpreter(model_path=w) # load TFLite model\r\n interpreter.allocate_tensors() # allocate\r\n input_details = interpreter.get_input_details() # inputs\r\n output_details = interpreter.get_output_details() # outputs\r\n elif tfjs: # TF.js\r\n raise NotImplementedError('ERROR: YOLOv5 TF.js inference is not supported')\r\n elif paddle: # PaddlePaddle\r\n LOGGER.info(f'Loading {w} for PaddlePaddle inference...')\r\n check_requirements('paddlepaddle-gpu' if cuda else 'paddlepaddle')\r\n import paddle.inference as pdi\r\n if not Path(w).is_file(): # if not *.pdmodel\r\n w = next(Path(w).rglob('*.pdmodel')) # get *.xml file from *_openvino_model dir\r\n weights = Path(w).with_suffix('.pdiparams')\r\n config = pdi.Config(str(w), str(weights))\r\n if cuda:\r\n config.enable_use_gpu(memory_pool_init_size_mb=2048, device_id=0)\r\n predictor = pdi.create_predictor(config)\r\n input_handle = predictor.get_input_handle(predictor.get_input_names()[0])\r\n output_names = predictor.get_output_names()\r\n elif triton: # NVIDIA Triton Inference Server\r\n LOGGER.info('Triton Inference Server not supported...')\r\n '''\r\n TODO:\r\n check_requirements('tritonclient[all]')\r\n from utils.triton import TritonRemoteModel\r\n model = TritonRemoteModel(url=w)\r\n nhwc = model.runtime.startswith(\"tensorflow\")\r\n '''\r\n else:\r\n raise NotImplementedError(f'ERROR: {w} is not a supported format')\r\n\r\n # class names\r\n if 'names' not in locals():\r\n names = yaml_load(data)['names'] if data else {i: f'class{i}' for i in range(999)}\r\n if names[0] == 'n01440764' and len(names) == 1000: # ImageNet\r\n names = yaml_load(ROOT / 'yolo/data/datasets/ImageNet.yaml')['names'] # human-readable names\r\n\r\n self.__dict__.update(locals()) # assign all variables to self\r\n\r\n def forward(self, im, augment=False, visualize=False):\r\n \"\"\"\r\n Runs inference on the given model\r\n\r\n Args:\r\n im: the image tensor\r\n augment: whether to augment the image. Defaults to False\r\n visualize: if True, then the network will output the feature maps of the last convolutional layer.\r\n Defaults to False\r\n \"\"\"\r\n # YOLOv5 MultiBackend inference\r\n b, ch, h, w = im.shape # batch, channel, height, width\r\n if self.fp16 and im.dtype != torch.float16:\r\n im = im.half() # to FP16\r\n if self.nhwc:\r\n im = im.permute(0, 2, 3, 1) # torch BCHW to numpy BHWC shape(1,320,192,3)\r\n\r\n if self.pt or self.nn_module: # PyTorch\r\n y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)\r\n elif self.jit: # TorchScript\r\n y = self.model(im)\r\n elif self.dnn: # ONNX OpenCV DNN\r\n im = im.cpu().numpy() # torch to numpy\r\n self.net.setInput(im)\r\n y = self.net.forward()\r\n elif self.onnx: # ONNX Runtime\r\n im = im.cpu().numpy() # torch to numpy\r\n y = self.session.run(self.output_names, {self.session.get_inputs()[0].name: im})\r\n elif self.xml: # OpenVINO\r\n im = im.cpu().numpy() # FP32\r\n y = list(self.executable_network([im]).values())\r\n elif self.engine: # TensorRT\r\n if self.dynamic and im.shape != self.bindings['images'].shape:\r\n i = self.model.get_binding_index('images')\r\n self.context.set_binding_shape(i, im.shape) # reshape if dynamic\r\n self.bindings['images'] = self.bindings['images']._replace(shape=im.shape)\r\n for name in self.output_names:\r\n i = self.model.get_binding_index(name)\r\n self.bindings[name].data.resize_(tuple(self.context.get_binding_shape(i)))\r\n s = self.bindings['images'].shape\r\n assert im.shape == s, f\"input size {im.shape} {'>' if self.dynamic else 'not equal to'} max model size {s}\"\r\n self.binding_addrs['images'] = int(im.data_ptr())\r\n self.context.execute_v2(list(self.binding_addrs.values()))\r\n y = [self.bindings[x].data for x in sorted(self.output_names)]\r\n elif self.coreml: # CoreML\r\n im = im.cpu().numpy()\r\n im = Image.fromarray((im[0] * 255).astype('uint8'))\r\n # im = im.resize((192, 320), Image.ANTIALIAS)\r\n y = self.model.predict({'image': im}) # coordinates are xywh normalized\r\n if 'confidence' in y:\r\n box = xywh2xyxy(y['coordinates'] * [[w, h, w, h]]) # xyxy pixels\r\n conf, cls = y['confidence'].max(1), y['confidence'].argmax(1).astype(np.float)\r\n y = np.concatenate((box, conf.reshape(-1, 1), cls.reshape(-1, 1)), 1)\r\n else:\r\n y = list(reversed(y.values())) # reversed for segmentation models (pred, proto)\r\n elif self.paddle: # PaddlePaddle\r\n im = im.cpu().numpy().astype(np.float32)\r\n self.input_handle.copy_from_cpu(im)\r\n self.predictor.run()\r\n y = [self.predictor.get_output_handle(x).copy_to_cpu() for x in self.output_names]\r\n elif self.triton: # NVIDIA Triton Inference Server\r\n y = self.model(im)\r\n else: # TensorFlow (SavedModel, GraphDef, Lite, Edge TPU)\r\n im = im.cpu().numpy()\r\n if self.saved_model: # SavedModel\r\n y = self.model(im, training=False) if self.keras else self.model(im)\r\n elif self.pb: # GraphDef\r\n y = self.frozen_func(x=self.tf.constant(im))\r\n else: # Lite or Edge TPU\r\n input = self.input_details[0]\r\n int8 = input['dtype'] == np.uint8 # is TFLite quantized uint8 model\r\n if int8:\r\n scale, zero_point = input['quantization']\r\n im = (im / scale + zero_point).astype(np.uint8) # de-scale\r\n self.interpreter.set_tensor(input['index'], im)\r\n self.interpreter.invoke()\r\n y = []\r\n for output in self.output_details:\r\n x = self.interpreter.get_tensor(output['index'])\r\n if int8:\r\n scale, zero_point = output['quantization']\r\n x = (x.astype(np.float32) - zero_point) * scale # re-scale\r\n y.append(x)\r\n y = [x if isinstance(x, np.ndarray) else x.numpy() for x in y]\r\n y[0][..., :4] *= [w, h, w, h] # xywh normalized to pixels\r\n\r\n if isinstance(y, (list, tuple)):\r\n return self.from_numpy(y[0]) if len(y) == 1 else [self.from_numpy(x) for x in y]\r\n else:\r\n return self.from_numpy(y)\r\n\r\n def from_numpy(self, x):\r\n \"\"\"\r\n `from_numpy` converts a numpy array to a tensor\r\n\r\n Args:\r\n x: the numpy array to convert\r\n \"\"\"\r\n return torch.from_numpy(x).to(self.device) if isinstance(x, np.ndarray) else x\r\n\r\n def warmup(self, imgsz=(1, 3, 640, 640)):\r\n \"\"\"\r\n Warmup model by running inference once\r\n\r\n Args:\r\n imgsz: the size of the image you want to run inference on.\r\n \"\"\"\r\n warmup_types = self.pt, self.jit, self.onnx, self.engine, self.saved_model, self.pb, self.triton, self.nn_module\r\n if any(warmup_types) and (self.device.type != 'cpu' or self.triton):\r\n im = torch.empty(*imgsz, dtype=torch.half if self.fp16 else torch.float, device=self.device) # input\r\n for _ in range(2 if self.jit else 1): #\r\n self.forward(im) # warmup\r\n\r\n @staticmethod\r\n def _model_type(p='path/to/model.pt'):\r\n \"\"\"\r\n This function takes a path to a model file and returns the model type\r\n\r\n Args:\r\n p: path to the model file. Defaults to path/to/model.pt\r\n \"\"\"\r\n # Return model type from model path, i.e. path='path/to/model.onnx' -> type=onnx\r\n # types = [pt, jit, onnx, xml, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle]\r\n from yolo.engine.exporter import export_formats\r\n sf = list(export_formats().Suffix) # export suffixes\r\n if not is_url(p, check=False) and not isinstance(p, str):\r\n check_suffix(p, sf) # checks\r\n url = urlparse(p) # if url may be Triton inference server\r\n types = [s in Path(p).name for s in sf]\r\n types[8] &= not types[9] # tflite &= not edgetpu\r\n triton = not any(types) and all([any(s in url.scheme for s in [\"http\", \"grpc\"]), url.netloc])\r\n return types + [triton]\r\n\r\n @staticmethod\r\n def _load_metadata(f=Path('path/to/meta.yaml')):\r\n \"\"\"\r\n > Loads the metadata from a yaml file\r\n\r\n Args:\r\n f: The path to the metadata file.\r\n \"\"\"\r\n from yolo.utils.files import yaml_load\r\n\r\n # Load metadata from meta.yaml if it exists\r\n if f.exists():\r\n d = yaml_load(f)\r\n return d['stride'], d['names'] # assign stride, names\r\n return None, None\r\n"
},
{
"path": "nn/modules.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nCommon modules\r\n\"\"\"\r\n\r\nimport math\r\nimport warnings\r\nfrom copy import copy\r\nfrom pathlib import Path\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport pandas as pd\r\nimport requests\r\nimport torch\r\nimport torch.nn as nn\r\nfrom PIL import Image, ImageOps\r\nfrom torch.cuda import amp\r\n\r\nfrom nn.autobackend import AutoBackend\r\nfrom yolo.data.augment import LetterBox\r\nfrom yolo.utils import LOGGER, colorstr\r\nfrom yolo.utils.files import increment_path\r\nfrom yolo.utils.ops import Profile, make_divisible, non_max_suppression, scale_boxes, xyxy2xywh\r\nfrom yolo.utils.plotting import Annotator, colors, save_one_box\r\nfrom yolo.utils.tal import dist2bbox, make_anchors\r\nfrom yolo.utils.torch_utils import copy_attr, smart_inference_mode\r\n\r\n# from utils.plots import feature_visualization TODO\r\n\r\n\r\ndef autopad(k, p=None, d=1): # kernel, padding, dilation\r\n # Pad to 'same' shape outputs\r\n if d > 1:\r\n k = d * (k - 1) + 1 if isinstance(k, int) else [d * (x - 1) + 1 for x in k] # actual kernel-size\r\n if p is None:\r\n p = k // 2 if isinstance(k, int) else [x // 2 for x in k] # auto-pad\r\n return p\r\n\r\n\r\nclass Conv(nn.Module):\r\n # Standard convolution with args(ch_in, ch_out, kernel, stride, padding, groups, dilation, activation)\r\n default_act = nn.SiLU() # default activation\r\n\r\n def __init__(self, c1, c2, k=1, s=1, p=None, g=1, d=1, act=True):\r\n super().__init__()\r\n self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)\r\n self.bn = nn.BatchNorm2d(c2)\r\n self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()\r\n\r\n def forward(self, x):\r\n return self.act(self.bn(self.conv(x)))\r\n\r\n def forward_fuse(self, x):\r\n return self.act(self.conv(x))\r\n\r\n\r\nclass DWConv(Conv):\r\n # Depth-wise convolution\r\n def __init__(self, c1, c2, k=1, s=1, d=1, act=True): # ch_in, ch_out, kernel, stride, dilation, activation\r\n super().__init__(c1, c2, k, s, g=math.gcd(c1, c2), d=d, act=act)\r\n\r\n\r\nclass DWConvTranspose2d(nn.ConvTranspose2d):\r\n # Depth-wise transpose convolution\r\n def __init__(self, c1, c2, k=1, s=1, p1=0, p2=0): # ch_in, ch_out, kernel, stride, padding, padding_out\r\n super().__init__(c1, c2, k, s, p1, p2, groups=math.gcd(c1, c2))\r\n\r\n\r\nclass ConvTranspose(nn.Module):\r\n # Convolution transpose 2d layer\r\n default_act = nn.SiLU() # default activation\r\n\r\n def __init__(self, c1, c2, k=2, s=2, p=0, bn=True, act=True):\r\n super().__init__()\r\n self.conv_transpose = nn.ConvTranspose2d(c1, c2, k, s, p, bias=not bn)\r\n self.bn = nn.BatchNorm2d(c2) if bn else nn.Identity()\r\n self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()\r\n\r\n def forward(self, x):\r\n return self.act(self.bn(self.conv_transpose(x)))\r\n\r\n\r\nclass DFL(nn.Module):\r\n # DFL module\r\n def __init__(self, c1=16):\r\n super().__init__()\r\n self.conv = nn.Conv2d(c1, 1, 1, bias=False).requires_grad_(False)\r\n x = torch.arange(c1, dtype=torch.float)\r\n self.conv.weight.data[:] = nn.Parameter(x.view(1, c1, 1, 1))\r\n self.c1 = c1\r\n\r\n def forward(self, x):\r\n b, c, a = x.shape # batch, channels, anchors\r\n return self.conv(x.view(b, 4, self.c1, a).transpose(2, 1).softmax(1)).view(b, 4, a)\r\n # return self.conv(x.view(b, self.c1, 4, a).softmax(1)).view(b, 4, a)\r\n\r\n\r\nclass TransformerLayer(nn.Module):\r\n # Transformer layer https://arxiv.org/abs/2010.11929 (LayerNorm layers removed for better performance)\r\n def __init__(self, c, num_heads):\r\n super().__init__()\r\n self.q = nn.Linear(c, c, bias=False)\r\n self.k = nn.Linear(c, c, bias=False)\r\n self.v = nn.Linear(c, c, bias=False)\r\n self.ma = nn.MultiheadAttention(embed_dim=c, num_heads=num_heads)\r\n self.fc1 = nn.Linear(c, c, bias=False)\r\n self.fc2 = nn.Linear(c, c, bias=False)\r\n\r\n def forward(self, x):\r\n x = self.ma(self.q(x), self.k(x), self.v(x))[0] + x\r\n x = self.fc2(self.fc1(x)) + x\r\n return x\r\n\r\n\r\nclass TransformerBlock(nn.Module):\r\n # Vision Transformer https://arxiv.org/abs/2010.11929\r\n def __init__(self, c1, c2, num_heads, num_layers):\r\n super().__init__()\r\n self.conv = None\r\n if c1 != c2:\r\n self.conv = Conv(c1, c2)\r\n self.linear = nn.Linear(c2, c2) # learnable position embedding\r\n self.tr = nn.Sequential(*(TransformerLayer(c2, num_heads) for _ in range(num_layers)))\r\n self.c2 = c2\r\n\r\n def forward(self, x):\r\n if self.conv is not None:\r\n x = self.conv(x)\r\n b, _, w, h = x.shape\r\n p = x.flatten(2).permute(2, 0, 1)\r\n return self.tr(p + self.linear(p)).permute(1, 2, 0).reshape(b, self.c2, w, h)\r\n\r\n\r\nclass Bottleneck(nn.Module):\r\n # Standard bottleneck\r\n def __init__(self, c1, c2, shortcut=True, g=1, k=(3, 3), e=0.5): # ch_in, ch_out, shortcut, kernels, groups, expand\r\n super().__init__()\r\n c_ = int(c2 * e) # hidden channels\r\n self.cv1 = Conv(c1, c_, k[0], 1)\r\n self.cv2 = Conv(c_, c2, k[1], 1, g=g)\r\n self.add = shortcut and c1 == c2\r\n\r\n def forward(self, x):\r\n return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x))\r\n\r\n\r\nclass BottleneckCSP(nn.Module):\r\n # CSP Bottleneck https://github.com/WongKinYiu/CrossStagePartialNetworks\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n c_ = int(c2 * e) # hidden channels\r\n self.cv1 = Conv(c1, c_, 1, 1)\r\n self.cv2 = nn.Conv2d(c1, c_, 1, 1, bias=False)\r\n self.cv3 = nn.Conv2d(c_, c_, 1, 1, bias=False)\r\n self.cv4 = Conv(2 * c_, c2, 1, 1)\r\n self.bn = nn.BatchNorm2d(2 * c_) # applied to cat(cv2, cv3)\r\n self.act = nn.SiLU()\r\n self.m = nn.Sequential(*(Bottleneck(c_, c_, shortcut, g, e=1.0) for _ in range(n)))\r\n\r\n def forward(self, x):\r\n y1 = self.cv3(self.m(self.cv1(x)))\r\n y2 = self.cv2(x)\r\n return self.cv4(self.act(self.bn(torch.cat((y1, y2), 1))))\r\n\r\n\r\nclass C3(nn.Module):\r\n # CSP Bottleneck with 3 convolutions\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n c_ = int(c2 * e) # hidden channels\r\n self.cv1 = Conv(c1, c_, 1, 1)\r\n self.cv2 = Conv(c1, c_, 1, 1)\r\n self.cv3 = Conv(2 * c_, c2, 1) # optional act=FReLU(c2)\r\n self.m = nn.Sequential(*(Bottleneck(c_, c_, shortcut, g, e=1.0) for _ in range(n)))\r\n\r\n def forward(self, x):\r\n return self.cv3(torch.cat((self.m(self.cv1(x)), self.cv2(x)), 1))\r\n\r\n\r\nclass C2(nn.Module):\r\n # CSP Bottleneck with 2 convolutions\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n self.c = int(c2 * e) # hidden channels\r\n self.cv1 = Conv(c1, 2 * self.c, 1, 1)\r\n self.cv2 = Conv(2 * self.c, c2, 1) # optional act=FReLU(c2)\r\n # self.attention = ChannelAttention(2 * self.c) # or SpatialAttention()\r\n self.m = nn.Sequential(*(Bottleneck(self.c, self.c, shortcut, g, k=((3, 3), (3, 3)), e=1.0) for _ in range(n)))\r\n\r\n def forward(self, x):\r\n a, b = self.cv1(x).split((self.c, self.c), 1)\r\n return self.cv2(torch.cat((self.m(a), b), 1))\r\n\r\n\r\nclass C2f(nn.Module):\r\n # CSP Bottleneck with 2 convolutions\r\n def __init__(self, c1, c2, n=1, shortcut=False, g=1, e=0.5): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n self.c = int(c2 * e) # hidden channels\r\n self.cv1 = Conv(c1, 2 * self.c, 1, 1)\r\n self.cv2 = Conv((2 + n) * self.c, c2, 1) # optional act=FReLU(c2)\r\n self.m = nn.ModuleList(Bottleneck(self.c, self.c, shortcut, g, k=((3, 3), (3, 3)), e=1.0) for _ in range(n))\r\n\r\n def forward(self, x):\r\n y = list(self.cv1(x).split((self.c, self.c), 1))\r\n y.extend(m(y[-1]) for m in self.m)\r\n return self.cv2(torch.cat(y, 1))\r\n\r\n\r\nclass ChannelAttention(nn.Module):\r\n # Channel-attention module https://github.com/open-mmlab/mmdetection/tree/v3.0.0rc1/configs/rtmdet\r\n def __init__(self, channels: int) -> None:\r\n super().__init__()\r\n self.pool = nn.AdaptiveAvgPool2d(1)\r\n self.fc = nn.Conv2d(channels, channels, 1, 1, 0, bias=True)\r\n self.act = nn.Sigmoid()\r\n\r\n def forward(self, x: torch.Tensor) -> torch.Tensor:\r\n return x * self.act(self.fc(self.pool(x)))\r\n\r\n\r\nclass SpatialAttention(nn.Module):\r\n # Spatial-attention module\r\n def __init__(self, kernel_size=7):\r\n super().__init__()\r\n assert kernel_size in (3, 7), 'kernel size must be 3 or 7'\r\n padding = 3 if kernel_size == 7 else 1\r\n self.cv1 = nn.Conv2d(2, 1, kernel_size, padding=padding, bias=False)\r\n self.act = nn.Sigmoid()\r\n\r\n def forward(self, x):\r\n return x * self.act(self.cv1(torch.cat([torch.mean(x, 1, keepdim=True), torch.max(x, 1, keepdim=True)[0]], 1)))\r\n\r\n\r\nclass CBAM(nn.Module):\r\n # CSP Bottleneck with 3 convolutions\r\n def __init__(self, c1, ratio=16, kernel_size=7): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n self.channel_attention = ChannelAttention(c1)\r\n self.spatial_attention = SpatialAttention(kernel_size)\r\n\r\n def forward(self, x):\r\n return self.spatial_attention(self.channel_attention(x))\r\n\r\n\r\nclass C1(nn.Module):\r\n # CSP Bottleneck with 3 convolutions\r\n def __init__(self, c1, c2, n=1): # ch_in, ch_out, number, shortcut, groups, expansion\r\n super().__init__()\r\n self.cv1 = Conv(c1, c2, 1, 1)\r\n self.m = nn.Sequential(*(Conv(c2, c2, 3) for _ in range(n)))\r\n\r\n def forward(self, x):\r\n y = self.cv1(x)\r\n return self.m(y) + y\r\n\r\n\r\nclass C3x(C3):\r\n # C3 module with cross-convolutions\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5):\r\n super().__init__(c1, c2, n, shortcut, g, e)\r\n self.c_ = int(c2 * e)\r\n self.m = nn.Sequential(*(Bottleneck(self.c_, self.c_, shortcut, g, k=((1, 3), (3, 1)), e=1) for _ in range(n)))\r\n\r\n\r\nclass C3TR(C3):\r\n # C3 module with TransformerBlock()\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5):\r\n super().__init__(c1, c2, n, shortcut, g, e)\r\n c_ = int(c2 * e)\r\n self.m = TransformerBlock(c_, c_, 4, n)\r\n\r\n\r\nclass C3Ghost(C3):\r\n # C3 module with GhostBottleneck()\r\n def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5):\r\n super().__init__(c1, c2, n, shortcut, g, e)\r\n c_ = int(c2 * e) # hidden channels\r\n self.m = nn.Sequential(*(GhostBottleneck(c_, c_) for _ in range(n)))\r\n\r\n\r\nclass SPP(nn.Module):\r\n # Spatial Pyramid Pooling (SPP) layer https://arxiv.org/abs/1406.4729\r\n def __init__(self, c1, c2, k=(5, 9, 13)):\r\n super().__init__()\r\n c_ = c1 // 2 # hidden channels\r\n self.cv1 = Conv(c1, c_, 1, 1)\r\n self.cv2 = Conv(c_ * (len(k) + 1), c2, 1, 1)\r\n self.m = nn.ModuleList([nn.MaxPool2d(kernel_size=x, stride=1, padding=x // 2) for x in k])\r\n\r\n def forward(self, x):\r\n x = self.cv1(x)\r\n with warnings.catch_warnings():\r\n warnings.simplefilter('ignore') # suppress torch 1.9.0 max_pool2d() warning\r\n return self.cv2(torch.cat([x] + [m(x) for m in self.m], 1))\r\n\r\n\r\nclass SPPF(nn.Module):\r\n # Spatial Pyramid Pooling - Fast (SPPF) layer for YOLOv5 by Glenn Jocher\r\n def __init__(self, c1, c2, k=5): # equivalent to SPP(k=(5, 9, 13))\r\n super().__init__()\r\n c_ = c1 // 2 # hidden channels\r\n self.cv1 = Conv(c1, c_, 1, 1)\r\n self.cv2 = Conv(c_ * 4, c2, 1, 1)\r\n self.m = nn.MaxPool2d(kernel_size=k, stride=1, padding=k // 2)\r\n\r\n def forward(self, x):\r\n x = self.cv1(x)\r\n with warnings.catch_warnings():\r\n warnings.simplefilter('ignore') # suppress torch 1.9.0 max_pool2d() warning\r\n y1 = self.m(x)\r\n y2 = self.m(y1)\r\n return self.cv2(torch.cat((x, y1, y2, self.m(y2)), 1))\r\n\r\n\r\nclass Focus(nn.Module):\r\n # Focus wh information into c-space\r\n def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): # ch_in, ch_out, kernel, stride, padding, groups\r\n super().__init__()\r\n self.conv = Conv(c1 * 4, c2, k, s, p, g, act=act)\r\n # self.contract = Contract(gain=2)\r\n\r\n def forward(self, x): # x(b,c,w,h) -> y(b,4c,w/2,h/2)\r\n return self.conv(torch.cat((x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]), 1))\r\n # return self.conv(self.contract(x))\r\n\r\n\r\nclass GhostConv(nn.Module):\r\n # Ghost Convolution https://github.com/huawei-noah/ghostnet\r\n def __init__(self, c1, c2, k=1, s=1, g=1, act=True): # ch_in, ch_out, kernel, stride, groups\r\n super().__init__()\r\n c_ = c2 // 2 # hidden channels\r\n self.cv1 = Conv(c1, c_, k, s, None, g, act=act)\r\n self.cv2 = Conv(c_, c_, 5, 1, None, c_, act=act)\r\n\r\n def forward(self, x):\r\n y = self.cv1(x)\r\n return torch.cat((y, self.cv2(y)), 1)\r\n\r\n\r\nclass GhostBottleneck(nn.Module):\r\n # Ghost Bottleneck https://github.com/huawei-noah/ghostnet\r\n def __init__(self, c1, c2, k=3, s=1): # ch_in, ch_out, kernel, stride\r\n super().__init__()\r\n c_ = c2 // 2\r\n self.conv = nn.Sequential(\r\n GhostConv(c1, c_, 1, 1), # pw\r\n DWConv(c_, c_, k, s, act=False) if s == 2 else nn.Identity(), # dw\r\n GhostConv(c_, c2, 1, 1, act=False)) # pw-linear\r\n self.shortcut = nn.Sequential(DWConv(c1, c1, k, s, act=False), Conv(c1, c2, 1, 1,\r\n act=False)) if s == 2 else nn.Identity()\r\n\r\n def forward(self, x):\r\n return self.conv(x) + self.shortcut(x)\r\n\r\n\r\nclass Concat(nn.Module):\r\n # Concatenate a list of tensors along dimension\r\n def __init__(self, dimension=1):\r\n super().__init__()\r\n self.d = dimension\r\n\r\n def forward(self, x):\r\n return torch.cat(x, self.d)\r\n\r\n\r\nclass AutoShape(nn.Module):\r\n # YOLOv5 input-robust model wrapper for passing cv2/np/PIL/torch inputs. Includes preprocessing, inference and NMS\r\n conf = 0.25 # NMS confidence threshold\r\n iou = 0.45 # NMS IoU threshold\r\n agnostic = False # NMS class-agnostic\r\n multi_label = False # NMS multiple labels per box\r\n classes = None # (optional list) filter by class, i.e. = [0, 15, 16] for COCO persons, cats and dogs\r\n max_det = 1000 # maximum number of detections per image\r\n amp = False # Automatic Mixed Precision (AMP) inference\r\n\r\n def __init__(self, model, verbose=True):\r\n super().__init__()\r\n if verbose:\r\n LOGGER.info('Adding AutoShape... ')\r\n copy_attr(self, model, include=('yaml', 'nc', 'hyp', 'names', 'stride', 'abc'), exclude=()) # copy attributes\r\n self.dmb = isinstance(model, AutoBackend) # DetectMultiBackend() instance\r\n self.pt = not self.dmb or model.pt # PyTorch model\r\n self.model = model.eval()\r\n if self.pt:\r\n m = self.model.model.model[-1] if self.dmb else self.model.model[-1] # Detect()\r\n m.inplace = False # Detect.inplace=False for safe multithread inference\r\n m.export = True # do not output loss values\r\n\r\n def _apply(self, fn):\r\n # Apply to(), cpu(), cuda(), half() to model tensors that are not parameters or registered buffers\r\n self = super()._apply(fn)\r\n if self.pt:\r\n m = self.model.model.model[-1] if self.dmb else self.model.model[-1] # Detect()\r\n m.stride = fn(m.stride)\r\n m.grid = list(map(fn, m.grid))\r\n if isinstance(m.anchor_grid, list):\r\n m.anchor_grid = list(map(fn, m.anchor_grid))\r\n return self\r\n\r\n @smart_inference_mode()\r\n def forward(self, ims, size=640, augment=False, profile=False):\r\n # Inference from various sources. For size(height=640, width=1280), RGB images example inputs are:\r\n # file: ims = 'data/images/zidane.jpg' # str or PosixPath\r\n # URI: = 'https://com/images/zidane.jpg'\r\n # OpenCV: = cv2.imread('image.jpg')[:,:,::-1] # HWC BGR to RGB x(640,1280,3)\r\n # PIL: = Image.open('image.jpg') or ImageGrab.grab() # HWC x(640,1280,3)\r\n # numpy: = np.zeros((640,1280,3)) # HWC\r\n # torch: = torch.zeros(16,3,320,640) # BCHW (scaled to size=640, 0-1 values)\r\n # multiple: = [Image.open('image1.jpg'), Image.open('image2.jpg'), ...] # list of images\r\n\r\n dt = (Profile(), Profile(), Profile())\r\n with dt[0]:\r\n if isinstance(size, int): # expand\r\n size = (size, size)\r\n p = next(self.model.parameters()) if self.pt else torch.empty(1, device=self.model.device) # param\r\n autocast = self.amp and (p.device.type != 'cpu') # Automatic Mixed Precision (AMP) inference\r\n if isinstance(ims, torch.Tensor): # torch\r\n with amp.autocast(autocast):\r\n return self.model(ims.to(p.device).type_as(p), augment=augment) # inference\r\n\r\n # Pre-process\r\n n, ims = (len(ims), list(ims)) if isinstance(ims, (list, tuple)) else (1, [ims]) # number, list of images\r\n shape0, shape1, files = [], [], [] # image and inference shapes, filenames\r\n for i, im in enumerate(ims):\r\n f = f'image{i}' # filename\r\n if isinstance(im, (str, Path)): # filename or uri\r\n im, f = Image.open(requests.get(im, stream=True).raw if str(im).startswith('http') else im), im\r\n im = np.asarray(ImageOps.exif_transpose(im))\r\n elif isinstance(im, Image.Image): # PIL Image\r\n im, f = np.asarray(ImageOps.exif_transpose(im)), getattr(im, 'filename', f) or f\r\n files.append(Path(f).with_suffix('.jpg').name)\r\n if im.shape[0] < 5: # image in CHW\r\n im = im.transpose((1, 2, 0)) # reverse dataloader .transpose(2, 0, 1)\r\n im = im[..., :3] if im.ndim == 3 else cv2.cvtColor(im, cv2.COLOR_GRAY2BGR) # enforce 3ch input\r\n s = im.shape[:2] # HWC\r\n shape0.append(s) # image shape\r\n g = max(size) / max(s) # gain\r\n shape1.append([y * g for y in s])\r\n ims[i] = im if im.data.contiguous else np.ascontiguousarray(im) # update\r\n shape1 = [make_divisible(x, self.stride) for x in np.array(shape1).max(0)] if self.pt else size # inf shape\r\n x = [LetterBox(shape1, auto=False)(image=im)[\"img\"] for im in ims] # pad\r\n x = np.ascontiguousarray(np.array(x).transpose((0, 3, 1, 2))) # stack and BHWC to BCHW\r\n x = torch.from_numpy(x).to(p.device).type_as(p) / 255 # uint8 to fp16/32\r\n\r\n with amp.autocast(autocast):\r\n # Inference\r\n with dt[1]:\r\n y = self.model(x, augment=augment) # forward\r\n\r\n # Post-process\r\n with dt[2]:\r\n y = non_max_suppression(y if self.dmb else y[0],\r\n self.conf,\r\n self.iou,\r\n self.classes,\r\n self.agnostic,\r\n self.multi_label,\r\n max_det=self.max_det) # NMS\r\n for i in range(n):\r\n scale_boxes(shape1, y[i][:, :4], shape0[i])\r\n\r\n return Detections(ims, y, files, dt, self.names, x.shape)\r\n\r\n\r\nclass Detections:\r\n # YOLOv5 detections class for inference results\r\n def __init__(self, ims, pred, files, times=(0, 0, 0), names=None, shape=None):\r\n super().__init__()\r\n d = pred[0].device # device\r\n gn = [torch.tensor([*(im.shape[i] for i in [1, 0, 1, 0]), 1, 1], device=d) for im in ims] # normalizations\r\n self.ims = ims # list of images as numpy arrays\r\n self.pred = pred # list of tensors pred[0] = (xyxy, conf, cls)\r\n self.names = names # class names\r\n self.files = files # image filenames\r\n self.times = times # profiling times\r\n self.xyxy = pred # xyxy pixels\r\n self.xywh = [xyxy2xywh(x) for x in pred] # xywh pixels\r\n self.xyxyn = [x / g for x, g in zip(self.xyxy, gn)] # xyxy normalized\r\n self.xywhn = [x / g for x, g in zip(self.xywh, gn)] # xywh normalized\r\n self.n = len(self.pred) # number of images (batch size)\r\n self.t = tuple(x.t / self.n * 1E3 for x in times) # timestamps (ms)\r\n self.s = tuple(shape) # inference BCHW shape\r\n\r\n def _run(self, pprint=False, show=False, save=False, crop=False, render=False, labels=True, save_dir=Path('')):\r\n s, crops = '', []\r\n for i, (im, pred) in enumerate(zip(self.ims, self.pred)):\r\n s += f'\\nimage {i + 1}/{len(self.pred)}: {im.shape[0]}x{im.shape[1]} ' # string\r\n if pred.shape[0]:\r\n for c in pred[:, -1].unique():\r\n n = (pred[:, -1] == c).sum() # detections per class\r\n s += f\"{n} {self.names[int(c)]}{'s' * (n > 1)}, \" # add to string\r\n s = s.rstrip(', ')\r\n if show or save or render or crop:\r\n annotator = Annotator(im, example=str(self.names))\r\n for *box, conf, cls in reversed(pred): # xyxy, confidence, class\r\n label = f'{self.names[int(cls)]} {conf:.2f}'\r\n if crop:\r\n file = save_dir / 'crops' / self.names[int(cls)] / self.files[i] if save else None\r\n crops.append({\r\n 'box': box,\r\n 'conf': conf,\r\n 'cls': cls,\r\n 'label': label,\r\n 'im': save_one_box(box, im, file=file, save=save)})\r\n else: # all others\r\n annotator.box_label(box, label if labels else '', color=colors(cls))\r\n im = annotator.im\r\n else:\r\n s += '(no detections)'\r\n\r\n im = Image.fromarray(im.astype(np.uint8)) if isinstance(im, np.ndarray) else im # from np\r\n if show:\r\n im.show(self.files[i]) # show\r\n if save:\r\n f = self.files[i]\r\n im.save(save_dir / f) # save\r\n if i == self.n - 1:\r\n LOGGER.info(f\"Saved {self.n} image{'s' * (self.n > 1)} to {colorstr('bold', save_dir)}\")\r\n if render:\r\n self.ims[i] = np.asarray(im)\r\n if pprint:\r\n s = s.lstrip('\\n')\r\n return f'{s}\\nSpeed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {self.s}' % self.t\r\n if crop:\r\n if save:\r\n LOGGER.info(f'Saved results to {save_dir}\\n')\r\n return crops\r\n\r\n def show(self, labels=True):\r\n self._run(show=True, labels=labels) # show results\r\n\r\n def save(self, labels=True, save_dir='runs/detect/exp', exist_ok=False):\r\n save_dir = increment_path(save_dir, exist_ok, mkdir=True) # increment save_dir\r\n self._run(save=True, labels=labels, save_dir=save_dir) # save results\r\n\r\n def crop(self, save=True, save_dir='runs/detect/exp', exist_ok=False):\r\n save_dir = increment_path(save_dir, exist_ok, mkdir=True) if save else None\r\n return self._run(crop=True, save=save, save_dir=save_dir) # crop results\r\n\r\n def render(self, labels=True):\r\n self._run(render=True, labels=labels) # render results\r\n return self.ims\r\n\r\n def pandas(self):\r\n # return detections as pandas DataFrames, i.e. print(results.pandas().xyxy[0])\r\n new = copy(self) # return copy\r\n ca = 'xmin', 'ymin', 'xmax', 'ymax', 'confidence', 'class', 'name' # xyxy columns\r\n cb = 'xcenter', 'ycenter', 'width', 'height', 'confidence', 'class', 'name' # xywh columns\r\n for k, c in zip(['xyxy', 'xyxyn', 'xywh', 'xywhn'], [ca, ca, cb, cb]):\r\n a = [[x[:5] + [int(x[5]), self.names[int(x[5])]] for x in x.tolist()] for x in getattr(self, k)] # update\r\n setattr(new, k, [pd.DataFrame(x, columns=c) for x in a])\r\n return new\r\n\r\n def tolist(self):\r\n # return a list of Detections objects, i.e. 'for result in results.tolist():'\r\n r = range(self.n) # iterable\r\n x = [Detections([self.ims[i]], [self.pred[i]], [self.files[i]], self.times, self.names, self.s) for i in r]\r\n # for d in x:\r\n # for k in ['ims', 'pred', 'xyxy', 'xyxyn', 'xywh', 'xywhn']:\r\n # setattr(d, k, getattr(d, k)[0]) # pop out of list\r\n return x\r\n\r\n def print(self):\r\n LOGGER.info(self.__str__())\r\n\r\n def __len__(self): # override len(results)\r\n return self.n\r\n\r\n def __str__(self): # override print(results)\r\n return self._run(pprint=True) # print results\r\n\r\n def __repr__(self):\r\n return f'YOLOv5 {self.__class__} instance\\n' + self.__str__()\r\n\r\n\r\nclass Proto(nn.Module):\r\n # YOLOv8 mask Proto module for segmentation models\r\n def __init__(self, c1, c_=256, c2=32): # ch_in, number of protos, number of masks\r\n super().__init__()\r\n self.cv1 = Conv(c1, c_, k=3)\r\n self.upsample = nn.ConvTranspose2d(c_, c_, 2, 2, 0, bias=True) # nn.Upsample(scale_factor=2, mode='nearest')\r\n self.cv2 = Conv(c_, c_, k=3)\r\n self.cv3 = Conv(c_, c2)\r\n\r\n def forward(self, x):\r\n return self.cv3(self.cv2(self.upsample(self.cv1(x))))\r\n\r\n\r\nclass Ensemble(nn.ModuleList):\r\n # Ensemble of models\r\n def __init__(self):\r\n super().__init__()\r\n\r\n def forward(self, x, augment=False, profile=False, visualize=False):\r\n y = [module(x, augment, profile, visualize)[0] for module in self]\r\n # y = torch.stack(y).max(0)[0] # max ensemble\r\n # y = torch.stack(y).mean(0) # mean ensemble\r\n y = torch.cat(y, 1) # nms ensemble\r\n return y, None # inference, train output\r\n\r\n\r\n# heads\r\nclass Detect(nn.Module):\r\n # YOLOv5 Detect head for detection models\r\n dynamic = False # force grid reconstruction\r\n export = False # export mode\r\n shape = None\r\n anchors = torch.empty(0) # init\r\n strides = torch.empty(0) # init\r\n\r\n def __init__(self, nc=80, ch=()): # detection layer\r\n super().__init__()\r\n self.nc = nc # number of classes\r\n self.nl = len(ch) # number of detection layers\r\n self.reg_max = 16 # DFL channels (ch[0] // 16 to scale 4/8/12/16/20 for n/s/m/l/x)\r\n self.no = nc + self.reg_max * 4 # number of outputs per anchor\r\n self.stride = torch.zeros(self.nl) # strides computed during build\r\n\r\n c2, c3 = max((16, ch[0] // 4, self.reg_max * 4)), max(ch[0], self.nc) # channels\r\n self.cv2 = nn.ModuleList(\r\n nn.Sequential(Conv(x, c2, 3), Conv(c2, c2, 3), nn.Conv2d(c2, 4 * self.reg_max, 1)) for x in ch)\r\n self.cv3 = nn.ModuleList(nn.Sequential(Conv(x, c3, 3), Conv(c3, c3, 3), nn.Conv2d(c3, self.nc, 1)) for x in ch)\r\n self.dfl = DFL(self.reg_max) if self.reg_max > 1 else nn.Identity()\r\n\r\n def forward(self, x):\r\n shape = x[0].shape # BCHW\r\n for i in range(self.nl):\r\n x[i] = torch.cat((self.cv2[i](x[i]), self.cv3[i](x[i])), 1)\r\n if self.training:\r\n return x\r\n elif self.dynamic or self.shape != shape:\r\n self.anchors, self.strides = (x.transpose(0, 1) for x in make_anchors(x, self.stride, 0.5))\r\n self.shape = shape\r\n\r\n box, cls = torch.cat([xi.view(shape[0], self.no, -1) for xi in x], 2).split((self.reg_max * 4, self.nc), 1)\r\n dbox = dist2bbox(self.dfl(box), self.anchors.unsqueeze(0), xywh=True, dim=1) * self.strides\r\n y = torch.cat((dbox, cls.sigmoid()), 1)\r\n return y if self.export else (y, x)\r\n\r\n def bias_init(self):\r\n # Initialize Detect() biases, WARNING: requires stride availability\r\n m = self # self.model[-1] # Detect() module\r\n # cf = torch.bincount(torch.tensor(np.concatenate(dataset.labels, 0)[:, 0]).long(), minlength=nc) + 1\r\n # ncf = math.log(0.6 / (m.nc - 0.999999)) if cf is None else torch.log(cf / cf.sum()) # nominal class frequency\r\n for a, b, s in zip(m.cv2, m.cv3, m.stride): # from\r\n a[-1].bias.data[:] = 1.0 # box\r\n b[-1].bias.data[:m.nc] = math.log(5 / m.nc / (640 / s) ** 2) # cls (.01 objects, 80 classes, 640 img)\r\n\r\n\r\nclass Segment(Detect):\r\n # YOLOv5 Segment head for segmentation models\r\n def __init__(self, nc=80, nm=32, npr=256, ch=()):\r\n super().__init__(nc, ch)\r\n self.nm = nm # number of masks\r\n self.npr = npr # number of protos\r\n self.proto = Proto(ch[0], self.npr, self.nm) # protos\r\n self.detect = Detect.forward\r\n\r\n c4 = max(ch[0] // 4, self.nm)\r\n self.cv4 = nn.ModuleList(nn.Sequential(Conv(x, c4, 3), Conv(c4, c4, 3), nn.Conv2d(c4, self.nm, 1)) for x in ch)\r\n\r\n def forward(self, x):\r\n p = self.proto(x[0]) # mask protos\r\n bs = p.shape[0] # batch size\r\n\r\n mc = torch.cat([self.cv4[i](x[i]).view(bs, self.nm, -1) for i in range(self.nl)], 2) # mask coefficients\r\n x = self.detect(self, x)\r\n if self.training:\r\n return x, mc, p\r\n return (torch.cat([x, mc], 1), p) if self.export else (torch.cat([x[0], mc], 1), (x[1], mc, p))\r\n\r\n\r\nclass Classify(nn.Module):\r\n # YOLOv5 classification head, i.e. x(b,c1,20,20) to x(b,c2)\r\n def __init__(self, c1, c2, k=1, s=1, p=None, g=1): # ch_in, ch_out, kernel, stride, padding, groups\r\n super().__init__()\r\n c_ = 1280 # efficientnet_b0 size\r\n self.conv = Conv(c1, c_, k, s, autopad(k, p), g)\r\n self.pool = nn.AdaptiveAvgPool2d(1) # to x(b,c_,1,1)\r\n self.drop = nn.Dropout(p=0.0, inplace=True)\r\n self.linear = nn.Linear(c_, c2) # to x(b,c2)\r\n\r\n def forward(self, x):\r\n if isinstance(x, list):\r\n x = torch.cat(x, 1)\r\n return self.linear(self.drop(self.pool(self.conv(x)).flatten(1)))\r\n"
},
{
"path": "nn/tasks.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport contextlib\r\nfrom copy import deepcopy\r\n\r\nimport thop\r\nimport torch\r\nimport torch.nn as nn\r\n\r\nfrom nn.modules import (C1, C2, C3, C3TR, SPP, SPPF, Bottleneck, BottleneckCSP, C2f, C3Ghost, C3x, Classify,\r\n Concat, Conv, ConvTranspose, Detect, DWConv, DWConvTranspose2d, Ensemble, Focus,\r\n GhostBottleneck, GhostConv, Segment)\r\nfrom yolo.utils import DEFAULT_CONFIG_DICT, DEFAULT_CONFIG_KEYS, LOGGER, colorstr, yaml_load\r\nfrom yolo.utils.checks import check_yaml\r\nfrom yolo.utils.torch_utils import (fuse_conv_and_bn, initialize_weights, intersect_dicts, make_divisible,\r\n model_info, scale_img, time_sync)\r\n\r\n\r\nclass BaseModel(nn.Module):\r\n '''\r\n The BaseModel class is a base class for all the models in the Ultralytics YOLO family.\r\n '''\r\n\r\n def forward(self, x, profile=False, visualize=False):\r\n \"\"\"\r\n > `forward` is a wrapper for `_forward_once` that runs the model on a single scale\r\n\r\n Args:\r\n x: the input image\r\n profile: whether to profile the model. Defaults to False\r\n visualize: if True, will return the intermediate feature maps. Defaults to False\r\n\r\n Returns:\r\n The output of the network.\r\n \"\"\"\r\n return self._forward_once(x, profile, visualize)\r\n\r\n def _forward_once(self, x, profile=False, visualize=False):\r\n \"\"\"\r\n > Forward pass of the network\r\n\r\n Args:\r\n x: input to the model\r\n profile: if True, the time taken for each layer will be printed. Defaults to False\r\n visualize: If True, it will save the feature maps of the model. Defaults to False\r\n\r\n Returns:\r\n The last layer of the model.\r\n \"\"\"\r\n y, dt = [], [] # outputs\r\n for m in self.model:\r\n if m.f != -1: # if not from previous layer\r\n x = y[m.f] if isinstance(m.f, int) else [x if j == -1 else y[j] for j in m.f] # from earlier layers\r\n if profile:\r\n self._profile_one_layer(m, x, dt)\r\n x = m(x) # run\r\n y.append(x if m.i in self.save else None) # save output\r\n if visualize:\r\n pass\r\n # TODO: feature_visualization(x, m.type, m.i, save_dir=visualize)\r\n return x\r\n\r\n def _profile_one_layer(self, m, x, dt):\r\n \"\"\"\r\n It takes a model, an input, and a list of times, and it profiles the model on the input, appending\r\n the time to the list\r\n\r\n Args:\r\n m: the model\r\n x: the input image\r\n dt: list of time taken for each layer\r\n \"\"\"\r\n c = m == self.model[-1] # is final layer, copy input as inplace fix\r\n o = thop.profile(m, inputs=(x.copy() if c else x,), verbose=False)[0] / 1E9 * 2 if thop else 0 # FLOPs\r\n t = time_sync()\r\n for _ in range(10):\r\n m(x.copy() if c else x)\r\n dt.append((time_sync() - t) * 100)\r\n if m == self.model[0]:\r\n LOGGER.info(f\"{'time (ms)':>10s} {'GFLOPs':>10s} {'params':>10s} module\")\r\n LOGGER.info(f'{dt[-1]:10.2f} {o:10.2f} {m.np:10.0f} {m.type}')\r\n if c:\r\n LOGGER.info(f\"{sum(dt):10.2f} {'-':>10s} {'-':>10s} Total\")\r\n\r\n def fuse(self):\r\n \"\"\"\r\n > It takes a model and fuses the Conv2d() and BatchNorm2d() layers into a single layer\r\n\r\n Returns:\r\n The model is being returned.\r\n \"\"\"\r\n LOGGER.info('Fusing layers... ')\r\n for m in self.model.modules():\r\n if isinstance(m, (Conv, DWConv)) and hasattr(m, 'bn'):\r\n m.conv = fuse_conv_and_bn(m.conv, m.bn) # update conv\r\n delattr(m, 'bn') # remove batchnorm\r\n m.forward = m.forward_fuse # update forward\r\n self.info()\r\n return self\r\n\r\n def info(self, verbose=False, imgsz=640):\r\n \"\"\"\r\n Prints model information\r\n\r\n Args:\r\n verbose: if True, prints out the model information. Defaults to False\r\n imgsz: the size of the image that the model will be trained on. Defaults to 640\r\n \"\"\"\r\n model_info(self, verbose, imgsz)\r\n\r\n def _apply(self, fn):\r\n \"\"\"\r\n `_apply()` is a function that applies a function to all the tensors in the model that are not\r\n parameters or registered buffers\r\n\r\n Args:\r\n fn: the function to apply to the model\r\n\r\n Returns:\r\n A model that is a Detect() object.\r\n \"\"\"\r\n self = super()._apply(fn)\r\n m = self.model[-1] # Detect()\r\n if isinstance(m, (Detect, Segment)):\r\n m.stride = fn(m.stride)\r\n m.anchors = fn(m.anchors)\r\n m.strides = fn(m.strides)\r\n return self\r\n\r\n def load(self, weights):\r\n \"\"\"\r\n > This function loads the weights of the model from a file\r\n\r\n Args:\r\n weights: The weights to load into the model.\r\n \"\"\"\r\n # Force all tasks to implement this function\r\n raise NotImplementedError(\"This function needs to be implemented by derived classes!\")\r\n\r\n\r\nclass DetectionModel(BaseModel):\r\n # YOLOv5 detection model\r\n def __init__(self, cfg='yolov8n.yaml', ch=3, nc=None, verbose=True): # model, input channels, number of classes\r\n super().__init__()\r\n self.yaml = cfg if isinstance(cfg, dict) else yaml_load(check_yaml(cfg), append_filename=True) # cfg dict\r\n\r\n # Define model\r\n ch = self.yaml['ch'] = self.yaml.get('ch', ch) # input channels\r\n if nc and nc != self.yaml['nc']:\r\n LOGGER.info(f\"Overriding model.yaml nc={self.yaml['nc']} with nc={nc}\")\r\n self.yaml['nc'] = nc # override yaml value\r\n self.model, self.save = parse_model(deepcopy(self.yaml), ch=[ch], verbose=verbose) # model, savelist\r\n self.names = {i: f'{i}' for i in range(self.yaml['nc'])} # default names dict\r\n self.inplace = self.yaml.get('inplace', True)\r\n\r\n # Build strides\r\n m = self.model[-1] # Detect()\r\n if isinstance(m, (Detect, Segment)):\r\n s = 256 # 2x min stride\r\n m.inplace = self.inplace\r\n forward = lambda x: self.forward(x)[0] if isinstance(m, Segment) else self.forward(x)\r\n m.stride = torch.tensor([s / x.shape[-2] for x in forward(torch.zeros(1, ch, s, s))]) # forward\r\n self.stride = m.stride\r\n m.bias_init() # only run once\r\n\r\n # Init weights, biases\r\n initialize_weights(self)\r\n if verbose:\r\n self.info()\r\n LOGGER.info('')\r\n\r\n def forward(self, x, augment=False, profile=False, visualize=False):\r\n if augment:\r\n return self._forward_augment(x) # augmented inference, None\r\n return self._forward_once(x, profile, visualize) # single-scale inference, train\r\n\r\n def _forward_augment(self, x):\r\n img_size = x.shape[-2:] # height, width\r\n s = [1, 0.83, 0.67] # scales\r\n f = [None, 3, None] # flips (2-ud, 3-lr)\r\n y = [] # outputs\r\n for si, fi in zip(s, f):\r\n xi = scale_img(x.flip(fi) if fi else x, si, gs=int(self.stride.max()))\r\n yi = self._forward_once(xi)[0] # forward\r\n # cv2.imwrite(f'img_{si}.jpg', 255 * xi[0].cpu().numpy().transpose((1, 2, 0))[:, :, ::-1]) # save\r\n yi = self._descale_pred(yi, fi, si, img_size)\r\n y.append(yi)\r\n y = self._clip_augmented(y) # clip augmented tails\r\n return torch.cat(y, -1), None # augmented inference, train\r\n\r\n @staticmethod\r\n def _descale_pred(p, flips, scale, img_size, dim=1):\r\n # de-scale predictions following augmented inference (inverse operation)\r\n p[:, :4] /= scale # de-scale\r\n x, y, wh, cls = p.split((1, 1, 2, p.shape[dim] - 4), dim)\r\n if flips == 2:\r\n y = img_size[0] - y # de-flip ud\r\n elif flips == 3:\r\n x = img_size[1] - x # de-flip lr\r\n return torch.cat((x, y, wh, cls), dim)\r\n\r\n def _clip_augmented(self, y):\r\n # Clip YOLOv5 augmented inference tails\r\n nl = self.model[-1].nl # number of detection layers (P3-P5)\r\n g = sum(4 ** x for x in range(nl)) # grid points\r\n e = 1 # exclude layer count\r\n i = (y[0].shape[-1] // g) * sum(4 ** x for x in range(e)) # indices\r\n y[0] = y[0][..., :-i] # large\r\n i = (y[-1].shape[-1] // g) * sum(4 ** (nl - 1 - x) for x in range(e)) # indices\r\n y[-1] = y[-1][..., i:] # small\r\n return y\r\n\r\n def load(self, weights, verbose=True):\r\n csd = weights.float().state_dict() # checkpoint state_dict as FP32\r\n csd = intersect_dicts(csd, self.state_dict()) # intersect\r\n self.load_state_dict(csd, strict=False) # load\r\n if verbose:\r\n LOGGER.info(f'Transferred {len(csd)}/{len(self.model.state_dict())} items from pretrained weights')\r\n\r\n\r\nclass SegmentationModel(DetectionModel):\r\n # YOLOv5 segmentation model\r\n def __init__(self, cfg='yolov8n-seg.yaml', ch=3, nc=None, verbose=True):\r\n super().__init__(cfg, ch, nc, verbose)\r\n\r\n\r\nclass ClassificationModel(BaseModel):\r\n # YOLOv5 classification model\r\n def __init__(self,\r\n cfg=None,\r\n model=None,\r\n ch=3,\r\n nc=1000,\r\n cutoff=10,\r\n verbose=True): # yaml, model, number of classes, cutoff index\r\n super().__init__()\r\n self._from_detection_model(model, nc, cutoff) if model is not None else self._from_yaml(cfg, ch, nc, verbose)\r\n\r\n def _from_detection_model(self, model, nc=1000, cutoff=10):\r\n # Create a YOLOv5 classification model from a YOLOv5 detection model\r\n from nn.autobackend import AutoBackend\r\n if isinstance(model, AutoBackend):\r\n model = model.model # unwrap DetectMultiBackend\r\n model.model = model.model[:cutoff] # backbone\r\n m = model.model[-1] # last layer\r\n ch = m.conv.in_channels if hasattr(m, 'conv') else m.cv1.conv.in_channels # ch into module\r\n c = Classify(ch, nc) # Classify()\r\n c.i, c.f, c.type = m.i, m.f, 'models.common.Classify' # index, from, type\r\n model.model[-1] = c # replace\r\n self.model = model.model\r\n self.stride = model.stride\r\n self.save = []\r\n self.nc = nc\r\n\r\n def _from_yaml(self, cfg, ch, nc, verbose):\r\n self.yaml = cfg if isinstance(cfg, dict) else yaml_load(check_yaml(cfg), append_filename=True) # cfg dict\r\n # Define model\r\n ch = self.yaml['ch'] = self.yaml.get('ch', ch) # input channels\r\n if nc and nc != self.yaml['nc']:\r\n LOGGER.info(f\"Overriding model.yaml nc={self.yaml['nc']} with nc={nc}\")\r\n self.yaml['nc'] = nc # override yaml value\r\n self.model, self.save = parse_model(deepcopy(self.yaml), ch=[ch], verbose=verbose) # model, savelist\r\n self.names = {i: f'{i}' for i in range(self.yaml['nc'])} # default names dict\r\n self.info()\r\n\r\n def load(self, weights):\r\n model = weights[\"model\"] if isinstance(weights, dict) else weights # torchvision models are not dicts\r\n csd = model.float().state_dict()\r\n csd = intersect_dicts(csd, self.state_dict()) # intersect\r\n self.load_state_dict(csd, strict=False) # load\r\n\r\n @staticmethod\r\n def reshape_outputs(model, nc):\r\n # Update a TorchVision classification model to class count 'n' if required\r\n name, m = list((model.model if hasattr(model, 'model') else model).named_children())[-1] # last module\r\n if isinstance(m, Classify): # YOLO Classify() head\r\n if m.linear.out_features != nc:\r\n m.linear = nn.Linear(m.linear.in_features, nc)\r\n elif isinstance(m, nn.Linear): # ResNet, EfficientNet\r\n if m.out_features != nc:\r\n setattr(model, name, nn.Linear(m.in_features, nc))\r\n elif isinstance(m, nn.Sequential):\r\n types = [type(x) for x in m]\r\n if nn.Linear in types:\r\n i = types.index(nn.Linear) # nn.Linear index\r\n if m[i].out_features != nc:\r\n m[i] = nn.Linear(m[i].in_features, nc)\r\n elif nn.Conv2d in types:\r\n i = types.index(nn.Conv2d) # nn.Conv2d index\r\n if m[i].out_channels != nc:\r\n m[i] = nn.Conv2d(m[i].in_channels, nc, m[i].kernel_size, m[i].stride, bias=m[i].bias is not None)\r\n\r\n\r\n# Functions ------------------------------------------------------------------------------------------------------------\r\n\r\n\r\ndef attempt_load_weights(weights, device=None, inplace=True, fuse=False):\r\n # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a\r\n from yolo.utils.downloads import attempt_download\r\n\r\n model = Ensemble()\r\n for w in weights if isinstance(weights, list) else [weights]:\r\n ckpt = torch.load(attempt_download(w), map_location='cpu') # load\r\n args = {**DEFAULT_CONFIG_DICT, **ckpt['train_args']} # combine model and default args, preferring model args\r\n ckpt = (ckpt.get('ema') or ckpt['model']).to(device).float() # FP32 model\r\n\r\n # Model compatibility updates\r\n ckpt.args = {k: v for k, v in args.items() if k in DEFAULT_CONFIG_KEYS} # attach args to model\r\n ckpt.pt_path = weights # attach *.pt file path to model\r\n if not hasattr(ckpt, 'stride'):\r\n ckpt.stride = torch.tensor([32.])\r\n\r\n # Append\r\n model.append(ckpt.fuse().eval() if fuse and hasattr(ckpt, 'fuse') else ckpt.eval()) # model in eval mode\r\n\r\n # Module compatibility updates\r\n for m in model.modules():\r\n t = type(m)\r\n if t in (nn.Hardswish, nn.LeakyReLU, nn.ReLU, nn.ReLU6, nn.SiLU, Detect, Segment):\r\n m.inplace = inplace # torch 1.7.0 compatibility\r\n elif t is nn.Upsample and not hasattr(m, 'recompute_scale_factor'):\r\n m.recompute_scale_factor = None # torch 1.11.0 compatibility\r\n\r\n # Return model\r\n if len(model) == 1:\r\n return model[-1]\r\n\r\n # Return ensemble\r\n print(f'Ensemble created with {weights}\\n')\r\n for k in 'names', 'nc', 'yaml':\r\n setattr(model, k, getattr(model[0], k))\r\n model.stride = model[torch.argmax(torch.tensor([m.stride.max() for m in model])).int()].stride # max stride\r\n assert all(model[0].nc == m.nc for m in model), f'Models have different class counts: {[m.nc for m in model]}'\r\n return model\r\n\r\n\r\ndef attempt_load_one_weight(weight, device=None, inplace=True, fuse=False):\r\n # Loads a single model weights\r\n from yolo.utils.downloads import attempt_download\r\n\r\n ckpt = torch.load(attempt_download(weight), map_location='cpu') # load\r\n args = {**DEFAULT_CONFIG_DICT, **ckpt['train_args']} # combine model and default args, preferring model args\r\n model = (ckpt.get('ema') or ckpt['model']).to(device).float() # FP32 model\r\n\r\n # Model compatibility updates\r\n model.args = {k: v for k, v in args.items() if k in DEFAULT_CONFIG_KEYS} # attach args to model\r\n model.pt_path = weight # attach *.pt file path to model\r\n if not hasattr(model, 'stride'):\r\n model.stride = torch.tensor([32.])\r\n\r\n model = model.fuse().eval() if fuse and hasattr(model, 'fuse') else model.eval() # model in eval mode\r\n\r\n # Module compatibility updates\r\n for m in model.modules():\r\n t = type(m)\r\n if t in (nn.Hardswish, nn.LeakyReLU, nn.ReLU, nn.ReLU6, nn.SiLU, Detect, Segment):\r\n m.inplace = inplace # torch 1.7.0 compatibility\r\n elif t is nn.Upsample and not hasattr(m, 'recompute_scale_factor'):\r\n m.recompute_scale_factor = None # torch 1.11.0 compatibility\r\n\r\n # Return model and ckpt\r\n return model, ckpt\r\n\r\n\r\ndef parse_model(d, ch, verbose=True): # model_dict, input_channels(3)\r\n # Parse a YOLO model.yaml dictionary\r\n if verbose:\r\n LOGGER.info(f\"\\n{'':>3}{'from':>20}{'n':>3}{'params':>10} {'module':<45}{'arguments':<30}\")\r\n nc, gd, gw, act = d['nc'], d['depth_multiple'], d['width_multiple'], d.get('activation')\r\n if act:\r\n Conv.default_act = eval(act) # redefine default activation, i.e. Conv.default_act = nn.SiLU()\r\n if verbose:\r\n LOGGER.info(f\"{colorstr('activation:')} {act}\") # print\r\n\r\n layers, save, c2 = [], [], ch[-1] # layers, savelist, ch out\r\n for i, (f, n, m, args) in enumerate(d['backbone'] + d['head']): # from, number, module, args\r\n m = eval(m) if isinstance(m, str) else m # eval strings\r\n for j, a in enumerate(args):\r\n with contextlib.suppress(NameError):\r\n args[j] = eval(a) if isinstance(a, str) else a # eval strings\r\n\r\n n = n_ = max(round(n * gd), 1) if n > 1 else n # depth gain\r\n if m in {\r\n Classify, Conv, ConvTranspose, GhostConv, Bottleneck, GhostBottleneck, SPP, SPPF, DWConv, Focus,\r\n BottleneckCSP, C1, C2, C2f, C3, C3TR, C3Ghost, nn.ConvTranspose2d, DWConvTranspose2d, C3x}:\r\n c1, c2 = ch[f], args[0]\r\n if c2 != nc: # if c2 not equal to number of classes (i.e. for Classify() output)\r\n c2 = make_divisible(c2 * gw, 8)\r\n\r\n args = [c1, c2, *args[1:]]\r\n if m in {BottleneckCSP, C1, C2, C2f, C3, C3TR, C3Ghost, C3x}:\r\n args.insert(2, n) # number of repeats\r\n n = 1\r\n elif m is nn.BatchNorm2d:\r\n args = [ch[f]]\r\n elif m is Concat:\r\n c2 = sum(ch[x] for x in f)\r\n elif m in {Detect, Segment}:\r\n args.append([ch[x] for x in f])\r\n if m is Segment:\r\n args[2] = make_divisible(args[2] * gw, 8)\r\n else:\r\n c2 = ch[f]\r\n\r\n m_ = nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args) # module\r\n t = str(m)[8:-2].replace('__main__.', '') # module type\r\n m.np = sum(x.numel() for x in m_.parameters()) # number params\r\n m_.i, m_.f, m_.type = i, f, t # attach index, 'from' index, type\r\n if verbose:\r\n LOGGER.info(f'{i:>3}{str(f):>20}{n_:>3}{m.np:10.0f} {t:<45}{str(args):<30}') # print\r\n save.extend(x % i for x in ([f] if isinstance(f, int) else f) if x != -1) # append to savelist\r\n layers.append(m_)\r\n if i == 0:\r\n ch = []\r\n ch.append(c2)\r\n return nn.Sequential(*layers), sorted(save)\r\n"
},
{
"path": "requirements.txt",
"content": "# Ultralytics requirements\r\n# Usage: pip install -r requirements.txt\r\n\r\n# Base ----------------------------------------\r\nhydra-core>=1.2.0\r\nmatplotlib>=3.2.2\r\nnumpy>=1.18.5\r\nopencv-python>=4.1.1\r\nPillow>=7.1.2\r\nPyYAML>=5.3.1\r\nrequests>=2.23.0\r\nscipy>=1.4.1\r\ntorch>=1.7.0\r\ntorchvision>=0.8.1\r\ntqdm>=4.64.0\r\nultralytics==8.0.0\r\n\r\n# Logging -------------------------------------\r\ntensorboard>=2.4.1\r\n# clearml\r\n# comet\r\n\r\n#tracking\r\nfilterpy\r\nscikit-image\r\n\r\n# Plotting ------------------------------------\r\npandas>=1.1.4\r\nseaborn>=0.11.0\r\n\r\n# Export --------------------------------------\r\n# coremltools>=6.0 # CoreML export\r\n# onnx>=1.12.0 # ONNX export\r\n# onnx-simplifier>=0.4.1 # ONNX simplifier\r\n# nvidia-pyindex # TensorRT export\r\n# nvidia-tensorrt # TensorRT export\r\n# scikit-learn==0.19.2 # CoreML quantization\r\n# tensorflow>=2.4.1 # TF exports (-cpu, -aarch64, -macos)\r\n# tensorflowjs>=3.9.0 # TF.js export\r\n# openvino-dev # OpenVINO export\r\n\r\n# Extras --------------------------------------\r\nipython # interactive notebook\r\npsutil # system utilization\r\nthop>=0.1.1 # FLOPs computation\r\n# albumentations>=1.0.3\r\n# pycocotools>=2.0.6 # COCO mAP\r\n# roboflow\r\n\r\n# HUB -----------------------------------------\r\nGitPython>=3.1.24\r\n"
},
{
"path": "yolo/cli.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport shutil\r\nfrom pathlib import Path\r\n\r\nimport hydra\r\n\r\nimport hub, yolo\r\nfrom yolo.utils import DEFAULT_CONFIG, LOGGER, colorstr\r\n\r\nDIR = Path(__file__).parent\r\n\r\n\r\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent.relative_to(DIR)), config_name=DEFAULT_CONFIG.name)\r\ndef cli(cfg):\r\n \"\"\"\r\n Run a specified task and mode with the given configuration.\r\n\r\n Args:\r\n cfg (DictConfig): Configuration for the task and mode.\r\n \"\"\"\r\n # LOGGER.info(f\"{colorstr(f'Ultralytics YOLO v{ultralytics.__version__}')}\")\r\n task, mode = cfg.task.lower(), cfg.mode.lower()\r\n\r\n # Special case for initializing the configuration\r\n if task == \"init\":\r\n shutil.copy2(DEFAULT_CONFIG, Path.cwd())\r\n LOGGER.info(f\"\"\"\r\n {colorstr(\"YOLO:\")} configuration saved to {Path.cwd() / DEFAULT_CONFIG.name}.\r\n To run experiments using custom configuration:\r\n yolo task='task' mode='mode' --config-name config_file.yaml\r\n \"\"\")\r\n return\r\n\r\n # Mapping from task to module\r\n task_module_map = {\"detect\": yolo.v8.detect, \"segment\": yolo.v8.segment, \"classify\": yolo.v8.classify}\r\n module = task_module_map.get(task)\r\n if not module:\r\n raise SyntaxError(f\"task not recognized. Choices are {', '.join(task_module_map.keys())}\")\r\n\r\n # Mapping from mode to function\r\n mode_func_map = {\r\n \"train\": module.train,\r\n \"val\": module.val,\r\n \"predict\": module.predict,\r\n \"export\": yolo.engine.exporter.export,\r\n \"checks\": hub.checks}\r\n func = mode_func_map.get(mode)\r\n if not func:\r\n raise SyntaxError(f\"mode not recognized. Choices are {', '.join(mode_func_map.keys())}\")\r\n\r\n func(cfg)\r\n"
},
{
"path": "yolo/configs/__init__.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom pathlib import Path\r\nfrom typing import Dict, Union\r\n\r\nfrom omegaconf import DictConfig, OmegaConf\r\n\r\nfrom ultralytics.yolo.configs.hydra_patch import check_config_mismatch\r\n\r\n\r\ndef get_config(config: Union[str, DictConfig], overrides: Union[str, Dict] = None):\r\n \"\"\"\r\n Load and merge configuration data from a file or dictionary.\r\n\r\n Args:\r\n config (Union[str, DictConfig]): Configuration data in the form of a file name or a DictConfig object.\r\n overrides (Union[str, Dict], optional): Overrides in the form of a file name or a dictionary. Default is None.\r\n\r\n Returns:\r\n OmegaConf.Namespace: Training arguments namespace.\r\n \"\"\"\r\n if overrides is None:\r\n overrides = {}\r\n if isinstance(config, (str, Path)):\r\n config = OmegaConf.load(config)\r\n elif isinstance(config, Dict):\r\n config = OmegaConf.create(config)\r\n # override\r\n if isinstance(overrides, str):\r\n overrides = OmegaConf.load(overrides)\r\n elif isinstance(overrides, Dict):\r\n overrides = OmegaConf.create(overrides)\r\n\r\n check_config_mismatch(dict(overrides).keys(), dict(config).keys())\r\n\r\n return OmegaConf.merge(config, overrides)\r\n"
},
{
"path": "yolo/configs/default.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# Default training settings and hyperparameters for medium-augmentation COCO training\r\n\r\ntask: \"detect\" # choices=['detect', 'segment', 'classify', 'init'] # init is a special case. Specify task to run.\r\nmode: \"train\" # choices=['train', 'val', 'predict'] # mode to run task in.\r\n\r\n# Train settings -------------------------------------------------------------------------------------------------------\r\nmodel: null # i.e. yolov8n.pt, yolov8n.yaml. Path to model file\r\ndata: null # i.e. coco128.yaml. Path to data file\r\nepochs: 100 # number of epochs to train for\r\npatience: 50 # TODO: epochs to wait for no observable improvement for early stopping of training\r\nbatch: 16 # number of images per batch\r\nimgsz: 640 # size of input images\r\nsave: True # save checkpoints\r\ncache: False # True/ram, disk or False. Use cache for data loading\r\ndevice: null # cuda device, i.e. 0 or 0,1,2,3 or cpu. Device to run on\r\nworkers: 8 # number of worker threads for data loading\r\nproject: null # project name\r\nname: null # experiment name\r\nexist_ok: False # whether to overwrite existing experiment\r\npretrained: False # whether to use a pretrained model\r\noptimizer: 'SGD' # optimizer to use, choices=['SGD', 'Adam', 'AdamW', 'RMSProp']\r\nverbose: False # whether to print verbose output\r\nseed: 0 # random seed for reproducibility\r\ndeterministic: True # whether to enable deterministic mode\r\nsingle_cls: False # train multi-class data as single-class\r\nimage_weights: False # use weighted image selection for training\r\nrect: False # support rectangular training\r\ncos_lr: False # use cosine learning rate scheduler\r\nclose_mosaic: 10 # disable mosaic augmentation for final 10 epochs\r\nresume: False # resume training from last checkpoint\r\n# Segmentation\r\noverlap_mask: True # masks should overlap during training\r\nmask_ratio: 4 # mask downsample ratio\r\n# Classification\r\ndropout: 0.0 # use dropout regularization\r\n\r\n# Val/Test settings ----------------------------------------------------------------------------------------------------\r\nval: True # validate/test during training\r\nsave_json: False # save results to JSON file\r\nsave_hybrid: False # save hybrid version of labels (labels + additional predictions)\r\nconf: null # object confidence threshold for detection (default 0.25 predict, 0.001 val)\r\niou: 0.7 # intersection over union (IoU) threshold for NMS\r\nmax_det: 300 # maximum number of detections per image\r\nhalf: False # use half precision (FP16)\r\ndnn: False # use OpenCV DNN for ONNX inference\r\nplots: True # show plots during training\r\n\r\n# Prediction settings --------------------------------------------------------------------------------------------------\r\nsource: null # source directory for images or videos\r\nshow: False # show results if possible\r\nsave_txt: False # save results as .txt file\r\nsave_conf: False # save results with confidence scores\r\nsave_crop: False # save cropped images with results\r\nhide_labels: False # hide labels\r\nhide_conf: True # hide confidence scores\r\nvid_stride: 1 # video frame-rate stride\r\nline_thickness: 3 # bounding box thickness (pixels)\r\nvisualize: False # visualize results\r\naugment: False # apply data augmentation to images\r\nagnostic_nms: False # class-agnostic NMS\r\nretina_masks: False # use retina masks for object detection\r\n\r\n# Export settings ------------------------------------------------------------------------------------------------------\r\nformat: torchscript # format to export to\r\nkeras: False # use Keras\r\noptimize: False # TorchScript: optimize for mobile\r\nint8: False # CoreML/TF INT8 quantization\r\ndynamic: False # ONNX/TF/TensorRT: dynamic axes\r\nsimplify: False # ONNX: simplify model\r\nopset: 17 # ONNX: opset version\r\nworkspace: 4 # TensorRT: workspace size (GB)\r\nnms: False # CoreML: add NMS\r\n\r\n# Hyperparameters ------------------------------------------------------------------------------------------------------\r\nlr0: 0.01 # initial learning rate (SGD=1E-2, Adam=1E-3)\r\nlrf: 0.01 # final OneCycleLR learning rate (lr0 * lrf)\r\nmomentum: 0.937 # SGD momentum/Adam beta1\r\nweight_decay: 0.0005 # optimizer weight decay 5e-4\r\nwarmup_epochs: 3.0 # warmup epochs (fractions ok)\r\nwarmup_momentum: 0.8 # warmup initial momentum\r\nwarmup_bias_lr: 0.1 # warmup initial bias lr\r\nbox: 7.5 # box loss gain\r\ncls: 0.5 # cls loss gain (scale with pixels)\r\ndfl: 1.5 # dfl loss gain\r\nfl_gamma: 0.0 # focal loss gamma (efficientDet default gamma=1.5)\r\nlabel_smoothing: 0.0\r\nnbs: 64 # nominal batch size\r\nhsv_h: 0.015 # image HSV-Hue augmentation (fraction)\r\nhsv_s: 0.7 # image HSV-Saturation augmentation (fraction)\r\nhsv_v: 0.4 # image HSV-Value augmentation (fraction)\r\ndegrees: 0.0 # image rotation (+/- deg)\r\ntranslate: 0.1 # image translation (+/- fraction)\r\nscale: 0.5 # image scale (+/- gain)\r\nshear: 0.0 # image shear (+/- deg)\r\nperspective: 0.0 # image perspective (+/- fraction), range 0-0.001\r\nflipud: 0.0 # image flip up-down (probability)\r\nfliplr: 0.5 # image flip left-right (probability)\r\nmosaic: 1.0 # image mosaic (probability)\r\nmixup: 0.0 # image mixup (probability)\r\ncopy_paste: 0.0 # segment copy-paste (probability)\r\n\r\n# Hydra configs --------------------------------------------------------------------------------------------------------\r\nhydra:\r\n output_subdir: null # disable hydra directory creation\r\n run:\r\n dir: .\r\n\r\n# Debug, do not modify -------------------------------------------------------------------------------------------------\r\nv5loader: False # use legacy YOLOv5 dataloader\r\n"
},
{
"path": "yolo/configs/hydra_patch.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport sys\r\nfrom difflib import get_close_matches\r\nfrom textwrap import dedent\r\n\r\nimport hydra\r\nfrom hydra.errors import ConfigCompositionException\r\nfrom omegaconf import OmegaConf, open_dict # noqa\r\nfrom omegaconf.errors import ConfigAttributeError, ConfigKeyError, OmegaConfBaseException # noqa\r\n\r\nfrom yolo.utils import LOGGER, colorstr\r\n\r\n\r\ndef override_config(overrides, cfg):\r\n override_keys = [override.key_or_group for override in overrides]\r\n check_config_mismatch(override_keys, cfg.keys())\r\n for override in overrides:\r\n if override.package is not None:\r\n raise ConfigCompositionException(f\"Override {override.input_line} looks like a config group\"\r\n f\" override, but config group '{override.key_or_group}' does not exist.\")\r\n\r\n key = override.key_or_group\r\n value = override.value()\r\n try:\r\n if override.is_delete():\r\n config_val = OmegaConf.select(cfg, key, throw_on_missing=False)\r\n if config_val is None:\r\n raise ConfigCompositionException(f\"Could not delete from config. '{override.key_or_group}'\"\r\n \" does not exist.\")\r\n elif value is not None and value != config_val:\r\n raise ConfigCompositionException(\"Could not delete from config. The value of\"\r\n f\" '{override.key_or_group}' is {config_val} and not\"\r\n f\" {value}.\")\r\n\r\n last_dot = key.rfind(\".\")\r\n with open_dict(cfg):\r\n if last_dot == -1:\r\n del cfg[key]\r\n else:\r\n node = OmegaConf.select(cfg, key[:last_dot])\r\n del node[key[last_dot + 1:]]\r\n\r\n elif override.is_add():\r\n if OmegaConf.select(cfg, key, throw_on_missing=False) is None or isinstance(value, (dict, list)):\r\n OmegaConf.update(cfg, key, value, merge=True, force_add=True)\r\n else:\r\n assert override.input_line is not None\r\n raise ConfigCompositionException(\r\n dedent(f\"\"\"\\\r\n Could not append to config. An item is already at '{override.key_or_group}'.\r\n Either remove + prefix: '{override.input_line[1:]}'\r\n Or add a second + to add or override '{override.key_or_group}': '+{override.input_line}'\r\n \"\"\"))\r\n elif override.is_force_add():\r\n OmegaConf.update(cfg, key, value, merge=True, force_add=True)\r\n else:\r\n try:\r\n OmegaConf.update(cfg, key, value, merge=True)\r\n except (ConfigAttributeError, ConfigKeyError) as ex:\r\n raise ConfigCompositionException(f\"Could not override '{override.key_or_group}'.\"\r\n f\"\\nTo append to your config use +{override.input_line}\") from ex\r\n except OmegaConfBaseException as ex:\r\n raise ConfigCompositionException(f\"Error merging override {override.input_line}\").with_traceback(\r\n sys.exc_info()[2]) from ex\r\n\r\n\r\ndef check_config_mismatch(overrides, cfg):\r\n mismatched = [option for option in overrides if option not in cfg and 'hydra.' not in option]\r\n\r\n for option in mismatched:\r\n LOGGER.info(f\"{colorstr(option)} is not a valid key. Similar keys: {get_close_matches(option, cfg, 3, 0.6)}\")\r\n if mismatched:\r\n exit()\r\n\r\n\r\nhydra._internal.config_loader_impl.ConfigLoaderImpl._apply_overrides_to_config = override_config\r\n"
},
{
"path": "yolo/data/__init__.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom .base import BaseDataset\r\nfrom .build import build_classification_dataloader, build_dataloader\r\nfrom .dataset import ClassificationDataset, SemanticDataset, YOLODataset\r\nfrom .dataset_wrappers import MixAndRectDataset\r\n"
},
{
"path": "yolo/data/augment.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport math\r\nimport random\r\nfrom copy import deepcopy\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport torch\r\nimport torchvision.transforms as T\r\n\r\nfrom ..utils import LOGGER, colorstr\r\nfrom ..utils.checks import check_version\r\nfrom ..utils.instance import Instances\r\nfrom ..utils.metrics import bbox_ioa\r\nfrom ..utils.ops import segment2box\r\nfrom .utils import IMAGENET_MEAN, IMAGENET_STD, polygons2masks, polygons2masks_overlap\r\n\r\n\r\n# TODO: we might need a BaseTransform to make all these augments be compatible with both classification and semantic\r\nclass BaseTransform:\r\n\r\n def __init__(self) -> None:\r\n pass\r\n\r\n def apply_image(self, labels):\r\n pass\r\n\r\n def apply_instances(self, labels):\r\n pass\r\n\r\n def apply_semantic(self, labels):\r\n pass\r\n\r\n def __call__(self, labels):\r\n self.apply_image(labels)\r\n self.apply_instances(labels)\r\n self.apply_semantic(labels)\r\n\r\n\r\nclass Compose:\r\n\r\n def __init__(self, transforms):\r\n self.transforms = transforms\r\n\r\n def __call__(self, data):\r\n for t in self.transforms:\r\n data = t(data)\r\n return data\r\n\r\n def append(self, transform):\r\n self.transforms.append(transform)\r\n\r\n def tolist(self):\r\n return self.transforms\r\n\r\n def __repr__(self):\r\n format_string = f\"{self.__class__.__name__}(\"\r\n for t in self.transforms:\r\n format_string += \"\\n\"\r\n format_string += f\" {t}\"\r\n format_string += \"\\n)\"\r\n return format_string\r\n\r\n\r\nclass BaseMixTransform:\r\n \"\"\"This implementation is from mmyolo\"\"\"\r\n\r\n def __init__(self, dataset, pre_transform=None, p=0.0) -> None:\r\n self.dataset = dataset\r\n self.pre_transform = pre_transform\r\n self.p = p\r\n\r\n def __call__(self, labels):\r\n if random.uniform(0, 1) > self.p:\r\n return labels\r\n\r\n # get index of one or three other images\r\n indexes = self.get_indexes()\r\n if isinstance(indexes, int):\r\n indexes = [indexes]\r\n\r\n # get images information will be used for Mosaic or MixUp\r\n mix_labels = [self.dataset.get_label_info(i) for i in indexes]\r\n\r\n if self.pre_transform is not None:\r\n for i, data in enumerate(mix_labels):\r\n mix_labels[i] = self.pre_transform(data)\r\n labels[\"mix_labels\"] = mix_labels\r\n\r\n # Mosaic or MixUp\r\n labels = self._mix_transform(labels)\r\n labels.pop(\"mix_labels\", None)\r\n return labels\r\n\r\n def _mix_transform(self, labels):\r\n raise NotImplementedError\r\n\r\n def get_indexes(self):\r\n raise NotImplementedError\r\n\r\n\r\nclass Mosaic(BaseMixTransform):\r\n \"\"\"Mosaic augmentation.\r\n Args:\r\n imgsz (Sequence[int]): Image size after mosaic pipeline of single\r\n image. The shape order should be (height, width).\r\n Default to (640, 640).\r\n \"\"\"\r\n\r\n def __init__(self, dataset, imgsz=640, p=1.0, border=(0, 0)):\r\n assert 0 <= p <= 1.0, \"The probability should be in range [0, 1]. \" f\"got {p}.\"\r\n super().__init__(dataset=dataset, p=p)\r\n self.dataset = dataset\r\n self.imgsz = imgsz\r\n self.border = border\r\n\r\n def get_indexes(self):\r\n return [random.randint(0, len(self.dataset) - 1) for _ in range(3)]\r\n\r\n def _mix_transform(self, labels):\r\n mosaic_labels = []\r\n assert labels.get(\"rect_shape\", None) is None, \"rect and mosaic is exclusive.\"\r\n assert len(labels.get(\"mix_labels\", [])) > 0, \"There are no other images for mosaic augment.\"\r\n s = self.imgsz\r\n yc, xc = (int(random.uniform(-x, 2 * s + x)) for x in self.border) # mosaic center x, y\r\n for i in range(4):\r\n labels_patch = (labels if i == 0 else labels[\"mix_labels\"][i - 1]).copy()\r\n # Load image\r\n img = labels_patch[\"img\"]\r\n h, w = labels_patch[\"resized_shape\"]\r\n\r\n # place img in img4\r\n if i == 0: # top left\r\n img4 = np.full((s * 2, s * 2, img.shape[2]), 114, dtype=np.uint8) # base image with 4 tiles\r\n x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc # xmin, ymin, xmax, ymax (large image)\r\n x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h # xmin, ymin, xmax, ymax (small image)\r\n elif i == 1: # top right\r\n x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\r\n x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\r\n elif i == 2: # bottom left\r\n x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\r\n x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, w, min(y2a - y1a, h)\r\n elif i == 3: # bottom right\r\n x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\r\n x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\r\n\r\n img4[y1a:y2a, x1a:x2a] = img[y1b:y2b, x1b:x2b] # img4[ymin:ymax, xmin:xmax]\r\n padw = x1a - x1b\r\n padh = y1a - y1b\r\n\r\n labels_patch = self._update_labels(labels_patch, padw, padh)\r\n mosaic_labels.append(labels_patch)\r\n final_labels = self._cat_labels(mosaic_labels)\r\n final_labels[\"img\"] = img4\r\n return final_labels\r\n\r\n def _update_labels(self, labels, padw, padh):\r\n \"\"\"Update labels\"\"\"\r\n nh, nw = labels[\"img\"].shape[:2]\r\n labels[\"instances\"].convert_bbox(format=\"xyxy\")\r\n labels[\"instances\"].denormalize(nw, nh)\r\n labels[\"instances\"].add_padding(padw, padh)\r\n return labels\r\n\r\n def _cat_labels(self, mosaic_labels):\r\n if len(mosaic_labels) == 0:\r\n return {}\r\n cls = []\r\n instances = []\r\n for labels in mosaic_labels:\r\n cls.append(labels[\"cls\"])\r\n instances.append(labels[\"instances\"])\r\n final_labels = {\r\n \"ori_shape\": mosaic_labels[0][\"ori_shape\"],\r\n \"resized_shape\": (self.imgsz * 2, self.imgsz * 2),\r\n \"im_file\": mosaic_labels[0][\"im_file\"],\r\n \"cls\": np.concatenate(cls, 0),\r\n \"instances\": Instances.concatenate(instances, axis=0)}\r\n final_labels[\"instances\"].clip(self.imgsz * 2, self.imgsz * 2)\r\n return final_labels\r\n\r\n\r\nclass MixUp(BaseMixTransform):\r\n\r\n def __init__(self, dataset, pre_transform=None, p=0.0) -> None:\r\n super().__init__(dataset=dataset, pre_transform=pre_transform, p=p)\r\n\r\n def get_indexes(self):\r\n return random.randint(0, len(self.dataset) - 1)\r\n\r\n def _mix_transform(self, labels):\r\n # Applies MixUp augmentation https://arxiv.org/pdf/1710.09412.pdf\r\n r = np.random.beta(32.0, 32.0) # mixup ratio, alpha=beta=32.0\r\n labels2 = labels[\"mix_labels\"][0]\r\n labels[\"img\"] = (labels[\"img\"] * r + labels2[\"img\"] * (1 - r)).astype(np.uint8)\r\n labels[\"instances\"] = Instances.concatenate([labels[\"instances\"], labels2[\"instances\"]], axis=0)\r\n labels[\"cls\"] = np.concatenate([labels[\"cls\"], labels2[\"cls\"]], 0)\r\n return labels\r\n\r\n\r\nclass RandomPerspective:\r\n\r\n def __init__(self, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, border=(0, 0)):\r\n self.degrees = degrees\r\n self.translate = translate\r\n self.scale = scale\r\n self.shear = shear\r\n self.perspective = perspective\r\n # mosaic border\r\n self.border = border\r\n\r\n def affine_transform(self, img):\r\n # Center\r\n C = np.eye(3)\r\n\r\n C[0, 2] = -img.shape[1] / 2 # x translation (pixels)\r\n C[1, 2] = -img.shape[0] / 2 # y translation (pixels)\r\n\r\n # Perspective\r\n P = np.eye(3)\r\n P[2, 0] = random.uniform(-self.perspective, self.perspective) # x perspective (about y)\r\n P[2, 1] = random.uniform(-self.perspective, self.perspective) # y perspective (about x)\r\n\r\n # Rotation and Scale\r\n R = np.eye(3)\r\n a = random.uniform(-self.degrees, self.degrees)\r\n # a += random.choice([-180, -90, 0, 90]) # add 90deg rotations to small rotations\r\n s = random.uniform(1 - self.scale, 1 + self.scale)\r\n # s = 2 ** random.uniform(-scale, scale)\r\n R[:2] = cv2.getRotationMatrix2D(angle=a, center=(0, 0), scale=s)\r\n\r\n # Shear\r\n S = np.eye(3)\r\n S[0, 1] = math.tan(random.uniform(-self.shear, self.shear) * math.pi / 180) # x shear (deg)\r\n S[1, 0] = math.tan(random.uniform(-self.shear, self.shear) * math.pi / 180) # y shear (deg)\r\n\r\n # Translation\r\n T = np.eye(3)\r\n T[0, 2] = random.uniform(0.5 - self.translate, 0.5 + self.translate) * self.size[0] # x translation (pixels)\r\n T[1, 2] = random.uniform(0.5 - self.translate, 0.5 + self.translate) * self.size[1] # y translation (pixels)\r\n\r\n # Combined rotation matrix\r\n M = T @ S @ R @ P @ C # order of operations (right to left) is IMPORTANT\r\n # affine image\r\n if (self.border[0] != 0) or (self.border[1] != 0) or (M != np.eye(3)).any(): # image changed\r\n if self.perspective:\r\n img = cv2.warpPerspective(img, M, dsize=self.size, borderValue=(114, 114, 114))\r\n else: # affine\r\n img = cv2.warpAffine(img, M[:2], dsize=self.size, borderValue=(114, 114, 114))\r\n return img, M, s\r\n\r\n def apply_bboxes(self, bboxes, M):\r\n \"\"\"apply affine to bboxes only.\r\n\r\n Args:\r\n bboxes(ndarray): list of bboxes, xyxy format, with shape (num_bboxes, 4).\r\n M(ndarray): affine matrix.\r\n Returns:\r\n new_bboxes(ndarray): bboxes after affine, [num_bboxes, 4].\r\n \"\"\"\r\n n = len(bboxes)\r\n if n == 0:\r\n return bboxes\r\n\r\n xy = np.ones((n * 4, 3))\r\n xy[:, :2] = bboxes[:, [0, 1, 2, 3, 0, 3, 2, 1]].reshape(n * 4, 2) # x1y1, x2y2, x1y2, x2y1\r\n xy = xy @ M.T # transform\r\n xy = (xy[:, :2] / xy[:, 2:3] if self.perspective else xy[:, :2]).reshape(n, 8) # perspective rescale or affine\r\n\r\n # create new boxes\r\n x = xy[:, [0, 2, 4, 6]]\r\n y = xy[:, [1, 3, 5, 7]]\r\n return np.concatenate((x.min(1), y.min(1), x.max(1), y.max(1))).reshape(4, n).T\r\n\r\n def apply_segments(self, segments, M):\r\n \"\"\"apply affine to segments and generate new bboxes from segments.\r\n\r\n Args:\r\n segments(ndarray): list of segments, [num_samples, 500, 2].\r\n M(ndarray): affine matrix.\r\n Returns:\r\n new_segments(ndarray): list of segments after affine, [num_samples, 500, 2].\r\n new_bboxes(ndarray): bboxes after affine, [N, 4].\r\n \"\"\"\r\n n, num = segments.shape[:2]\r\n if n == 0:\r\n return [], segments\r\n\r\n xy = np.ones((n * num, 3))\r\n segments = segments.reshape(-1, 2)\r\n xy[:, :2] = segments\r\n xy = xy @ M.T # transform\r\n xy = xy[:, :2] / xy[:, 2:3]\r\n segments = xy.reshape(n, -1, 2)\r\n bboxes = np.stack([segment2box(xy, self.size[0], self.size[1]) for xy in segments], 0)\r\n return bboxes, segments\r\n\r\n def apply_keypoints(self, keypoints, M):\r\n \"\"\"apply affine to keypoints.\r\n\r\n Args:\r\n keypoints(ndarray): keypoints, [N, 17, 2].\r\n M(ndarray): affine matrix.\r\n Return:\r\n new_keypoints(ndarray): keypoints after affine, [N, 17, 2].\r\n \"\"\"\r\n n = len(keypoints)\r\n if n == 0:\r\n return keypoints\r\n new_keypoints = np.ones((n * 17, 3))\r\n new_keypoints[:, :2] = keypoints.reshape(n * 17, 2) # num_kpt is hardcoded to 17\r\n new_keypoints = new_keypoints @ M.T # transform\r\n new_keypoints = (new_keypoints[:, :2] / new_keypoints[:, 2:3]).reshape(n, 34) # perspective rescale or affine\r\n new_keypoints[keypoints.reshape(-1, 34) == 0] = 0\r\n x_kpts = new_keypoints[:, list(range(0, 34, 2))]\r\n y_kpts = new_keypoints[:, list(range(1, 34, 2))]\r\n\r\n x_kpts[np.logical_or.reduce((x_kpts < 0, x_kpts > self.size[0], y_kpts < 0, y_kpts > self.size[1]))] = 0\r\n y_kpts[np.logical_or.reduce((x_kpts < 0, x_kpts > self.size[0], y_kpts < 0, y_kpts > self.size[1]))] = 0\r\n new_keypoints[:, list(range(0, 34, 2))] = x_kpts\r\n new_keypoints[:, list(range(1, 34, 2))] = y_kpts\r\n return new_keypoints.reshape(n, 17, 2)\r\n\r\n def __call__(self, labels):\r\n \"\"\"\r\n Affine images and targets.\r\n\r\n Args:\r\n labels(Dict): a dict of `bboxes`, `segments`, `keypoints`.\r\n \"\"\"\r\n img = labels[\"img\"]\r\n cls = labels[\"cls\"]\r\n instances = labels.pop(\"instances\")\r\n # make sure the coord formats are right\r\n instances.convert_bbox(format=\"xyxy\")\r\n instances.denormalize(*img.shape[:2][::-1])\r\n\r\n self.size = img.shape[1] + self.border[1] * 2, img.shape[0] + self.border[0] * 2 # w, h\r\n # M is affine matrix\r\n # scale for func:`box_candidates`\r\n img, M, scale = self.affine_transform(img)\r\n\r\n bboxes = self.apply_bboxes(instances.bboxes, M)\r\n\r\n segments = instances.segments\r\n keypoints = instances.keypoints\r\n # update bboxes if there are segments.\r\n if len(segments):\r\n bboxes, segments = self.apply_segments(segments, M)\r\n\r\n if keypoints is not None:\r\n keypoints = self.apply_keypoints(keypoints, M)\r\n new_instances = Instances(bboxes, segments, keypoints, bbox_format=\"xyxy\", normalized=False)\r\n # clip\r\n new_instances.clip(*self.size)\r\n\r\n # filter instances\r\n instances.scale(scale_w=scale, scale_h=scale, bbox_only=True)\r\n # make the bboxes have the same scale with new_bboxes\r\n i = self.box_candidates(box1=instances.bboxes.T,\r\n box2=new_instances.bboxes.T,\r\n area_thr=0.01 if len(segments) else 0.10)\r\n labels[\"instances\"] = new_instances[i]\r\n labels[\"cls\"] = cls[i]\r\n labels[\"img\"] = img\r\n labels[\"resized_shape\"] = img.shape[:2]\r\n return labels\r\n\r\n def box_candidates(self, box1, box2, wh_thr=2, ar_thr=100, area_thr=0.1, eps=1e-16): # box1(4,n), box2(4,n)\r\n # Compute box candidates: box1 before augment, box2 after augment, wh_thr (pixels), aspect_ratio_thr, area_ratio\r\n w1, h1 = box1[2] - box1[0], box1[3] - box1[1]\r\n w2, h2 = box2[2] - box2[0], box2[3] - box2[1]\r\n ar = np.maximum(w2 / (h2 + eps), h2 / (w2 + eps)) # aspect ratio\r\n return (w2 > wh_thr) & (h2 > wh_thr) & (w2 * h2 / (w1 * h1 + eps) > area_thr) & (ar < ar_thr) # candidates\r\n\r\n\r\nclass RandomHSV:\r\n\r\n def __init__(self, hgain=0.5, sgain=0.5, vgain=0.5) -> None:\r\n self.hgain = hgain\r\n self.sgain = sgain\r\n self.vgain = vgain\r\n\r\n def __call__(self, labels):\r\n img = labels[\"img\"]\r\n if self.hgain or self.sgain or self.vgain:\r\n r = np.random.uniform(-1, 1, 3) * [self.hgain, self.sgain, self.vgain] + 1 # random gains\r\n hue, sat, val = cv2.split(cv2.cvtColor(img, cv2.COLOR_BGR2HSV))\r\n dtype = img.dtype # uint8\r\n\r\n x = np.arange(0, 256, dtype=r.dtype)\r\n lut_hue = ((x * r[0]) % 180).astype(dtype)\r\n lut_sat = np.clip(x * r[1], 0, 255).astype(dtype)\r\n lut_val = np.clip(x * r[2], 0, 255).astype(dtype)\r\n\r\n im_hsv = cv2.merge((cv2.LUT(hue, lut_hue), cv2.LUT(sat, lut_sat), cv2.LUT(val, lut_val)))\r\n cv2.cvtColor(im_hsv, cv2.COLOR_HSV2BGR, dst=img) # no return needed\r\n return labels\r\n\r\n\r\nclass RandomFlip:\r\n\r\n def __init__(self, p=0.5, direction=\"horizontal\") -> None:\r\n assert direction in [\"horizontal\", \"vertical\"], f\"Support direction `horizontal` or `vertical`, got {direction}\"\r\n assert 0 <= p <= 1.0\r\n\r\n self.p = p\r\n self.direction = direction\r\n\r\n def __call__(self, labels):\r\n img = labels[\"img\"]\r\n instances = labels.pop(\"instances\")\r\n instances.convert_bbox(format=\"xywh\")\r\n h, w = img.shape[:2]\r\n h = 1 if instances.normalized else h\r\n w = 1 if instances.normalized else w\r\n\r\n # Flip up-down\r\n if self.direction == \"vertical\" and random.random() < self.p:\r\n img = np.flipud(img)\r\n instances.flipud(h)\r\n if self.direction == \"horizontal\" and random.random() < self.p:\r\n img = np.fliplr(img)\r\n instances.fliplr(w)\r\n labels[\"img\"] = np.ascontiguousarray(img)\r\n labels[\"instances\"] = instances\r\n return labels\r\n\r\n\r\nclass LetterBox:\r\n \"\"\"Resize image and padding for detection, instance segmentation, pose\"\"\"\r\n\r\n def __init__(self, new_shape=(640, 640), auto=False, scaleFill=False, scaleup=True, stride=32):\r\n self.new_shape = new_shape\r\n self.auto = auto\r\n self.scaleFill = scaleFill\r\n self.scaleup = scaleup\r\n self.stride = stride\r\n\r\n def __call__(self, labels=None, image=None):\r\n if labels is None:\r\n labels = {}\r\n img = labels.get(\"img\") if image is None else image\r\n shape = img.shape[:2] # current shape [height, width]\r\n new_shape = labels.pop(\"rect_shape\", self.new_shape)\r\n if isinstance(new_shape, int):\r\n new_shape = (new_shape, new_shape)\r\n\r\n # Scale ratio (new / old)\r\n r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])\r\n if not self.scaleup: # only scale down, do not scale up (for better val mAP)\r\n r = min(r, 1.0)\r\n\r\n # Compute padding\r\n ratio = r, r # width, height ratios\r\n new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))\r\n dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh padding\r\n if self.auto: # minimum rectangle\r\n dw, dh = np.mod(dw, self.stride), np.mod(dh, self.stride) # wh padding\r\n elif self.scaleFill: # stretch\r\n dw, dh = 0.0, 0.0\r\n new_unpad = (new_shape[1], new_shape[0])\r\n ratio = new_shape[1] / shape[1], new_shape[0] / shape[0] # width, height ratios\r\n\r\n dw /= 2 # divide padding into 2 sides\r\n dh /= 2\r\n if labels.get(\"ratio_pad\"):\r\n labels[\"ratio_pad\"] = (labels[\"ratio_pad\"], (dw, dh)) # for evaluation\r\n\r\n if shape[::-1] != new_unpad: # resize\r\n img = cv2.resize(img, new_unpad, interpolation=cv2.INTER_LINEAR)\r\n top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))\r\n left, right = int(round(dw - 0.1)), int(round(dw + 0.1))\r\n img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT,\r\n value=(114, 114, 114)) # add border\r\n\r\n if len(labels):\r\n labels = self._update_labels(labels, ratio, dw, dh)\r\n labels[\"img\"] = img\r\n labels[\"resized_shape\"] = new_shape\r\n return labels\r\n else:\r\n return img\r\n\r\n def _update_labels(self, labels, ratio, padw, padh):\r\n \"\"\"Update labels\"\"\"\r\n labels[\"instances\"].convert_bbox(format=\"xyxy\")\r\n labels[\"instances\"].denormalize(*labels[\"img\"].shape[:2][::-1])\r\n labels[\"instances\"].scale(*ratio)\r\n labels[\"instances\"].add_padding(padw, padh)\r\n return labels\r\n\r\n\r\nclass CopyPaste:\r\n\r\n def __init__(self, p=0.5) -> None:\r\n self.p = p\r\n\r\n def __call__(self, labels):\r\n # Implement Copy-Paste augmentation https://arxiv.org/abs/2012.07177, labels as nx5 np.array(cls, xyxy)\r\n im = labels[\"img\"]\r\n cls = labels[\"cls\"]\r\n instances = labels.pop(\"instances\")\r\n instances.convert_bbox(format=\"xyxy\")\r\n if self.p and len(instances.segments):\r\n n = len(instances)\r\n _, w, _ = im.shape # height, width, channels\r\n im_new = np.zeros(im.shape, np.uint8)\r\n\r\n # calculate ioa first then select indexes randomly\r\n ins_flip = deepcopy(instances)\r\n ins_flip.fliplr(w)\r\n\r\n ioa = bbox_ioa(ins_flip.bboxes, instances.bboxes) # intersection over area, (N, M)\r\n indexes = np.nonzero((ioa < 0.30).all(1))[0] # (N, )\r\n n = len(indexes)\r\n for j in random.sample(list(indexes), k=round(self.p * n)):\r\n cls = np.concatenate((cls, cls[[j]]), axis=0)\r\n instances = Instances.concatenate((instances, ins_flip[[j]]), axis=0)\r\n cv2.drawContours(im_new, instances.segments[[j]].astype(np.int32), -1, (1, 1, 1), cv2.FILLED)\r\n\r\n result = cv2.flip(im, 1) # augment segments (flip left-right)\r\n i = cv2.flip(im_new, 1).astype(bool)\r\n im[i] = result[i] # cv2.imwrite('debug.jpg', im) # debug\r\n\r\n labels[\"img\"] = im\r\n labels[\"cls\"] = cls\r\n labels[\"instances\"] = instances\r\n return labels\r\n\r\n\r\nclass Albumentations:\r\n # YOLOv5 Albumentations class (optional, only used if package is installed)\r\n def __init__(self, p=1.0):\r\n self.p = p\r\n self.transform = None\r\n prefix = colorstr(\"albumentations: \")\r\n try:\r\n import albumentations as A\r\n\r\n check_version(A.__version__, \"1.0.3\", hard=True) # version requirement\r\n\r\n T = [\r\n A.Blur(p=0.01),\r\n A.MedianBlur(p=0.01),\r\n A.ToGray(p=0.01),\r\n A.CLAHE(p=0.01),\r\n A.RandomBrightnessContrast(p=0.0),\r\n A.RandomGamma(p=0.0),\r\n A.ImageCompression(quality_lower=75, p=0.0),] # transforms\r\n self.transform = A.Compose(T, bbox_params=A.BboxParams(format=\"yolo\", label_fields=[\"class_labels\"]))\r\n\r\n LOGGER.info(prefix + \", \".join(f\"{x}\".replace(\"always_apply=False, \", \"\") for x in T if x.p))\r\n except ImportError: # package not installed, skip\r\n pass\r\n except Exception as e:\r\n LOGGER.info(f\"{prefix}{e}\")\r\n\r\n def __call__(self, labels):\r\n im = labels[\"img\"]\r\n cls = labels[\"cls\"]\r\n if len(cls):\r\n labels[\"instances\"].convert_bbox(\"xywh\")\r\n labels[\"instances\"].normalize(*im.shape[:2][::-1])\r\n bboxes = labels[\"instances\"].bboxes\r\n # TODO: add supports of segments and keypoints\r\n if self.transform and random.random() < self.p:\r\n new = self.transform(image=im, bboxes=bboxes, class_labels=cls) # transformed\r\n labels[\"img\"] = new[\"image\"]\r\n labels[\"cls\"] = np.array(new[\"class_labels\"])\r\n labels[\"instances\"].update(bboxes=bboxes)\r\n return labels\r\n\r\n\r\n# TODO: technically this is not an augmentation, maybe we should put this to another files\r\nclass Format:\r\n\r\n def __init__(self,\r\n bbox_format=\"xywh\",\r\n normalize=True,\r\n return_mask=False,\r\n return_keypoint=False,\r\n mask_ratio=4,\r\n mask_overlap=True,\r\n batch_idx=True):\r\n self.bbox_format = bbox_format\r\n self.normalize = normalize\r\n self.return_mask = return_mask # set False when training detection only\r\n self.return_keypoint = return_keypoint\r\n self.mask_ratio = mask_ratio\r\n self.mask_overlap = mask_overlap\r\n self.batch_idx = batch_idx # keep the batch indexes\r\n\r\n def __call__(self, labels):\r\n img = labels[\"img\"]\r\n h, w = img.shape[:2]\r\n cls = labels.pop(\"cls\")\r\n instances = labels.pop(\"instances\")\r\n instances.convert_bbox(format=self.bbox_format)\r\n instances.denormalize(w, h)\r\n nl = len(instances)\r\n\r\n if self.return_mask:\r\n if nl:\r\n masks, instances, cls = self._format_segments(instances, cls, w, h)\r\n masks = torch.from_numpy(masks)\r\n else:\r\n masks = torch.zeros(1 if self.mask_overlap else nl, img.shape[0] // self.mask_ratio,\r\n img.shape[1] // self.mask_ratio)\r\n labels[\"masks\"] = masks\r\n if self.normalize:\r\n instances.normalize(w, h)\r\n labels[\"img\"] = self._format_img(img)\r\n labels[\"cls\"] = torch.from_numpy(cls) if nl else torch.zeros(nl)\r\n labels[\"bboxes\"] = torch.from_numpy(instances.bboxes) if nl else torch.zeros((nl, 4))\r\n if self.return_keypoint:\r\n labels[\"keypoints\"] = torch.from_numpy(instances.keypoints) if nl else torch.zeros((nl, 17, 2))\r\n # then we can use collate_fn\r\n if self.batch_idx:\r\n labels[\"batch_idx\"] = torch.zeros(nl)\r\n return labels\r\n\r\n def _format_img(self, img):\r\n if len(img.shape) < 3:\r\n img = np.expand_dims(img, -1)\r\n img = np.ascontiguousarray(img.transpose(2, 0, 1)[::-1])\r\n img = torch.from_numpy(img)\r\n return img\r\n\r\n def _format_segments(self, instances, cls, w, h):\r\n \"\"\"convert polygon points to bitmap\"\"\"\r\n segments = instances.segments\r\n if self.mask_overlap:\r\n masks, sorted_idx = polygons2masks_overlap((h, w), segments, downsample_ratio=self.mask_ratio)\r\n masks = masks[None] # (640, 640) -> (1, 640, 640)\r\n instances = instances[sorted_idx]\r\n cls = cls[sorted_idx]\r\n else:\r\n masks = polygons2masks((h, w), segments, color=1, downsample_ratio=self.mask_ratio)\r\n\r\n return masks, instances, cls\r\n\r\n\r\ndef mosaic_transforms(dataset, imgsz, hyp):\r\n pre_transform = Compose([\r\n Mosaic(dataset, imgsz=imgsz, p=hyp.mosaic, border=[-imgsz // 2, -imgsz // 2]),\r\n CopyPaste(p=hyp.copy_paste),\r\n RandomPerspective(\r\n degrees=hyp.degrees,\r\n translate=hyp.translate,\r\n scale=hyp.scale,\r\n shear=hyp.shear,\r\n perspective=hyp.perspective,\r\n border=[-imgsz // 2, -imgsz // 2],\r\n ),])\r\n return Compose([\r\n pre_transform,\r\n MixUp(dataset, pre_transform=pre_transform, p=hyp.mixup),\r\n Albumentations(p=1.0),\r\n RandomHSV(hgain=hyp.hsv_h, sgain=hyp.hsv_s, vgain=hyp.hsv_v),\r\n RandomFlip(direction=\"vertical\", p=hyp.flipud),\r\n RandomFlip(direction=\"horizontal\", p=hyp.fliplr),]) # transforms\r\n\r\n\r\ndef affine_transforms(imgsz, hyp):\r\n return Compose([\r\n LetterBox(new_shape=(imgsz, imgsz)),\r\n RandomPerspective(\r\n degrees=hyp.degrees,\r\n translate=hyp.translate,\r\n scale=hyp.scale,\r\n shear=hyp.shear,\r\n perspective=hyp.perspective,\r\n border=[0, 0],\r\n ),\r\n Albumentations(p=1.0),\r\n RandomHSV(hgain=hyp.hsv_h, sgain=hyp.hsv_s, vgain=hyp.hsv_v),\r\n RandomFlip(direction=\"vertical\", p=hyp.flipud),\r\n RandomFlip(direction=\"horizontal\", p=hyp.fliplr),]) # transforms\r\n\r\n\r\n# Classification augmentations -----------------------------------------------------------------------------------------\r\ndef classify_transforms(size=224):\r\n # Transforms to apply if albumentations not installed\r\n assert isinstance(size, int), f\"ERROR: classify_transforms size {size} must be integer, not (list, tuple)\"\r\n # T.Compose([T.ToTensor(), T.Resize(size), T.CenterCrop(size), T.Normalize(IMAGENET_MEAN, IMAGENET_STD)])\r\n return T.Compose([CenterCrop(size), ToTensor(), T.Normalize(IMAGENET_MEAN, IMAGENET_STD)])\r\n\r\n\r\ndef classify_albumentations(\r\n augment=True,\r\n size=224,\r\n scale=(0.08, 1.0),\r\n hflip=0.5,\r\n vflip=0.0,\r\n jitter=0.4,\r\n mean=IMAGENET_MEAN,\r\n std=IMAGENET_STD,\r\n auto_aug=False,\r\n):\r\n # YOLOv5 classification Albumentations (optional, only used if package is installed)\r\n prefix = colorstr(\"albumentations: \")\r\n try:\r\n import albumentations as A\r\n from albumentations.pytorch import ToTensorV2\r\n\r\n check_version(A.__version__, \"1.0.3\", hard=True) # version requirement\r\n if augment: # Resize and crop\r\n T = [A.RandomResizedCrop(height=size, width=size, scale=scale)]\r\n if auto_aug:\r\n # TODO: implement AugMix, AutoAug & RandAug in albumentation\r\n LOGGER.info(f\"{prefix}auto augmentations are currently not supported\")\r\n else:\r\n if hflip > 0:\r\n T += [A.HorizontalFlip(p=hflip)]\r\n if vflip > 0:\r\n T += [A.VerticalFlip(p=vflip)]\r\n if jitter > 0:\r\n color_jitter = (float(jitter),) * 3 # repeat value for brightness, contrast, saturation, 0 hue\r\n T += [A.ColorJitter(*color_jitter, 0)]\r\n else: # Use fixed crop for eval set (reproducibility)\r\n T = [A.SmallestMaxSize(max_size=size), A.CenterCrop(height=size, width=size)]\r\n T += [A.Normalize(mean=mean, std=std), ToTensorV2()] # Normalize and convert to Tensor\r\n LOGGER.info(prefix + \", \".join(f\"{x}\".replace(\"always_apply=False, \", \"\") for x in T if x.p))\r\n return A.Compose(T)\r\n\r\n except ImportError: # package not installed, skip\r\n pass\r\n except Exception as e:\r\n LOGGER.info(f\"{prefix}{e}\")\r\n\r\n\r\nclass ClassifyLetterBox:\r\n # YOLOv5 LetterBox class for image preprocessing, i.e. T.Compose([LetterBox(size), ToTensor()])\r\n def __init__(self, size=(640, 640), auto=False, stride=32):\r\n super().__init__()\r\n self.h, self.w = (size, size) if isinstance(size, int) else size\r\n self.auto = auto # pass max size integer, automatically solve for short side using stride\r\n self.stride = stride # used with auto\r\n\r\n def __call__(self, im): # im = np.array HWC\r\n imh, imw = im.shape[:2]\r\n r = min(self.h / imh, self.w / imw) # ratio of new/old\r\n h, w = round(imh * r), round(imw * r) # resized image\r\n hs, ws = (math.ceil(x / self.stride) * self.stride for x in (h, w)) if self.auto else self.h, self.w\r\n top, left = round((hs - h) / 2 - 0.1), round((ws - w) / 2 - 0.1)\r\n im_out = np.full((self.h, self.w, 3), 114, dtype=im.dtype)\r\n im_out[top:top + h, left:left + w] = cv2.resize(im, (w, h), interpolation=cv2.INTER_LINEAR)\r\n return im_out\r\n\r\n\r\nclass CenterCrop:\r\n # YOLOv5 CenterCrop class for image preprocessing, i.e. T.Compose([CenterCrop(size), ToTensor()])\r\n def __init__(self, size=640):\r\n super().__init__()\r\n self.h, self.w = (size, size) if isinstance(size, int) else size\r\n\r\n def __call__(self, im): # im = np.array HWC\r\n imh, imw = im.shape[:2]\r\n m = min(imh, imw) # min dimension\r\n top, left = (imh - m) // 2, (imw - m) // 2\r\n return cv2.resize(im[top:top + m, left:left + m], (self.w, self.h), interpolation=cv2.INTER_LINEAR)\r\n\r\n\r\nclass ToTensor:\r\n # YOLOv5 ToTensor class for image preprocessing, i.e. T.Compose([LetterBox(size), ToTensor()])\r\n def __init__(self, half=False):\r\n super().__init__()\r\n self.half = half\r\n\r\n def __call__(self, im): # im = np.array HWC in BGR order\r\n im = np.ascontiguousarray(im.transpose((2, 0, 1))[::-1]) # HWC to CHW -> BGR to RGB -> contiguous\r\n im = torch.from_numpy(im) # to torch\r\n im = im.half() if self.half else im.float() # uint8 to fp16/32\r\n im /= 255.0 # 0-255 to 0.0-1.0\r\n return im\r\n"
},
{
"path": "yolo/data/base.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport glob\r\nimport math\r\nimport os\r\nfrom multiprocessing.pool import ThreadPool\r\nfrom pathlib import Path\r\nfrom typing import Optional\r\n\r\nimport cv2\r\nimport numpy as np\r\nfrom torch.utils.data import Dataset\r\nfrom tqdm import tqdm\r\n\r\nfrom ..utils import NUM_THREADS, TQDM_BAR_FORMAT\r\nfrom .utils import HELP_URL, IMG_FORMATS, LOCAL_RANK\r\n\r\n\r\nclass BaseDataset(Dataset):\r\n \"\"\"Base Dataset.\r\n Args:\r\n img_path (str): image path.\r\n pipeline (dict): a dict of image transforms.\r\n label_path (str): label path, this can also be an ann_file or other custom label path.\r\n \"\"\"\r\n\r\n def __init__(\r\n self,\r\n img_path,\r\n imgsz=640,\r\n label_path=None,\r\n cache=False,\r\n augment=True,\r\n hyp=None,\r\n prefix=\"\",\r\n rect=False,\r\n batch_size=None,\r\n stride=32,\r\n pad=0.5,\r\n single_cls=False,\r\n ):\r\n super().__init__()\r\n self.img_path = img_path\r\n self.imgsz = imgsz\r\n self.label_path = label_path\r\n self.augment = augment\r\n self.single_cls = single_cls\r\n self.prefix = prefix\r\n\r\n self.im_files = self.get_img_files(self.img_path)\r\n self.labels = self.get_labels()\r\n if self.single_cls:\r\n self.update_labels(include_class=[])\r\n\r\n self.ni = len(self.labels)\r\n\r\n # rect stuff\r\n self.rect = rect\r\n self.batch_size = batch_size\r\n self.stride = stride\r\n self.pad = pad\r\n if self.rect:\r\n assert self.batch_size is not None\r\n self.set_rectangle()\r\n\r\n # cache stuff\r\n self.ims = [None] * self.ni\r\n self.npy_files = [Path(f).with_suffix(\".npy\") for f in self.im_files]\r\n if cache:\r\n self.cache_images(cache)\r\n\r\n # transforms\r\n self.transforms = self.build_transforms(hyp=hyp)\r\n\r\n def get_img_files(self, img_path):\r\n \"\"\"Read image files.\"\"\"\r\n try:\r\n f = [] # image files\r\n for p in img_path if isinstance(img_path, list) else [img_path]:\r\n p = Path(p) # os-agnostic\r\n if p.is_dir(): # dir\r\n f += glob.glob(str(p / \"**\" / \"*.*\"), recursive=True)\r\n # f = list(p.rglob('*.*')) # pathlib\r\n elif p.is_file(): # file\r\n with open(p) as t:\r\n t = t.read().strip().splitlines()\r\n parent = str(p.parent) + os.sep\r\n f += [x.replace(\"./\", parent) if x.startswith(\"./\") else x for x in t] # local to global path\r\n # f += [p.parent / x.lstrip(os.sep) for x in t] # local to global path (pathlib)\r\n else:\r\n raise FileNotFoundError(f\"{self.prefix}{p} does not exist\")\r\n im_files = sorted(x.replace(\"/\", os.sep) for x in f if x.split(\".\")[-1].lower() in IMG_FORMATS)\r\n # self.img_files = sorted([x for x in f if x.suffix[1:].lower() in IMG_FORMATS]) # pathlib\r\n assert im_files, f\"{self.prefix}No images found\"\r\n except Exception as e:\r\n raise FileNotFoundError(f\"{self.prefix}Error loading data from {img_path}: {e}\\n{HELP_URL}\") from e\r\n return im_files\r\n\r\n def update_labels(self, include_class: Optional[list]):\r\n \"\"\"include_class, filter labels to include only these classes (optional)\"\"\"\r\n include_class_array = np.array(include_class).reshape(1, -1)\r\n for i in range(len(self.labels)):\r\n if include_class:\r\n cls = self.labels[i][\"cls\"]\r\n bboxes = self.labels[i][\"bboxes\"]\r\n segments = self.labels[i][\"segments\"]\r\n j = (cls == include_class_array).any(1)\r\n self.labels[i][\"cls\"] = cls[j]\r\n self.labels[i][\"bboxes\"] = bboxes[j]\r\n if segments:\r\n self.labels[i][\"segments\"] = segments[j]\r\n if self.single_cls:\r\n self.labels[i][\"cls\"] = 0\r\n\r\n def load_image(self, i):\r\n # Loads 1 image from dataset index 'i', returns (im, resized hw)\r\n im, f, fn = self.ims[i], self.im_files[i], self.npy_files[i]\r\n if im is None: # not cached in RAM\r\n if fn.exists(): # load npy\r\n im = np.load(fn)\r\n else: # read image\r\n im = cv2.imread(f) # BGR\r\n assert im is not None, f\"Image Not Found {f}\"\r\n h0, w0 = im.shape[:2] # orig hw\r\n r = self.imgsz / max(h0, w0) # ratio\r\n if r != 1: # if sizes are not equal\r\n interp = cv2.INTER_LINEAR if (self.augment or r > 1) else cv2.INTER_AREA\r\n im = cv2.resize(im, (math.ceil(w0 * r), math.ceil(h0 * r)), interpolation=interp)\r\n return im, (h0, w0), im.shape[:2] # im, hw_original, hw_resized\r\n return self.ims[i], self.im_hw0[i], self.im_hw[i] # im, hw_original, hw_resized\r\n\r\n def cache_images(self, cache):\r\n # cache images to memory or disk\r\n gb = 0 # Gigabytes of cached images\r\n self.im_hw0, self.im_hw = [None] * self.ni, [None] * self.ni\r\n fcn = self.cache_images_to_disk if cache == \"disk\" else self.load_image\r\n results = ThreadPool(NUM_THREADS).imap(fcn, range(self.ni))\r\n pbar = tqdm(enumerate(results), total=self.ni, bar_format=TQDM_BAR_FORMAT, disable=LOCAL_RANK > 0)\r\n for i, x in pbar:\r\n if cache == \"disk\":\r\n gb += self.npy_files[i].stat().st_size\r\n else: # 'ram'\r\n self.ims[i], self.im_hw0[i], self.im_hw[i] = x # im, hw_orig, hw_resized = load_image(self, i)\r\n gb += self.ims[i].nbytes\r\n pbar.desc = f\"{self.prefix}Caching images ({gb / 1E9:.1f}GB {cache})\"\r\n pbar.close()\r\n\r\n def cache_images_to_disk(self, i):\r\n # Saves an image as an *.npy file for faster loading\r\n f = self.npy_files[i]\r\n if not f.exists():\r\n np.save(f.as_posix(), cv2.imread(self.im_files[i]))\r\n\r\n def set_rectangle(self):\r\n bi = np.floor(np.arange(self.ni) / self.batch_size).astype(int) # batch index\r\n nb = bi[-1] + 1 # number of batches\r\n\r\n s = np.array([x.pop(\"shape\") for x in self.labels]) # hw\r\n ar = s[:, 0] / s[:, 1] # aspect ratio\r\n irect = ar.argsort()\r\n self.im_files = [self.im_files[i] for i in irect]\r\n self.labels = [self.labels[i] for i in irect]\r\n ar = ar[irect]\r\n\r\n # Set training image shapes\r\n shapes = [[1, 1]] * nb\r\n for i in range(nb):\r\n ari = ar[bi == i]\r\n mini, maxi = ari.min(), ari.max()\r\n if maxi < 1:\r\n shapes[i] = [maxi, 1]\r\n elif mini > 1:\r\n shapes[i] = [1, 1 / mini]\r\n\r\n self.batch_shapes = np.ceil(np.array(shapes) * self.imgsz / self.stride + self.pad).astype(int) * self.stride\r\n self.batch = bi # batch index of image\r\n\r\n def __getitem__(self, index):\r\n return self.transforms(self.get_label_info(index))\r\n\r\n def get_label_info(self, index):\r\n label = self.labels[index].copy()\r\n label[\"img\"], label[\"ori_shape\"], label[\"resized_shape\"] = self.load_image(index)\r\n label[\"ratio_pad\"] = (\r\n label[\"resized_shape\"][0] / label[\"ori_shape\"][0],\r\n label[\"resized_shape\"][1] / label[\"ori_shape\"][1],\r\n ) # for evaluation\r\n if self.rect:\r\n label[\"rect_shape\"] = self.batch_shapes[self.batch[index]]\r\n label = self.update_labels_info(label)\r\n return label\r\n\r\n def __len__(self):\r\n return len(self.im_files)\r\n\r\n def update_labels_info(self, label):\r\n \"\"\"custom your label format here\"\"\"\r\n return label\r\n\r\n def build_transforms(self, hyp=None):\r\n \"\"\"Users can custom augmentations here\r\n like:\r\n if self.augment:\r\n # training transforms\r\n return Compose([])\r\n else:\r\n # val transforms\r\n return Compose([])\r\n \"\"\"\r\n raise NotImplementedError\r\n\r\n def get_labels(self):\r\n \"\"\"Users can custom their own format here.\r\n Make sure your output is a list with each element like below:\r\n dict(\r\n im_file=im_file,\r\n shape=shape, # format: (height, width)\r\n cls=cls,\r\n bboxes=bboxes, # xywh\r\n segments=segments, # xy\r\n keypoints=keypoints, # xy\r\n normalized=True, # or False\r\n bbox_format=\"xyxy\", # or xywh, ltwh\r\n )\r\n \"\"\"\r\n raise NotImplementedError\r\n"
},
{
"path": "yolo/data/build.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport os\r\nimport random\r\n\r\nimport numpy as np\r\nimport torch\r\nfrom torch.utils.data import DataLoader, dataloader, distributed\r\n\r\nfrom ..utils import LOGGER, colorstr\r\nfrom ..utils.torch_utils import torch_distributed_zero_first\r\nfrom .dataset import ClassificationDataset, YOLODataset\r\nfrom .utils import PIN_MEMORY, RANK\r\n\r\n\r\nclass InfiniteDataLoader(dataloader.DataLoader):\r\n \"\"\"Dataloader that reuses workers\r\n\r\n Uses same syntax as vanilla DataLoader\r\n \"\"\"\r\n\r\n def __init__(self, *args, **kwargs):\r\n super().__init__(*args, **kwargs)\r\n object.__setattr__(self, \"batch_sampler\", _RepeatSampler(self.batch_sampler))\r\n self.iterator = super().__iter__()\r\n\r\n def __len__(self):\r\n return len(self.batch_sampler.sampler)\r\n\r\n def __iter__(self):\r\n for _ in range(len(self)):\r\n yield next(self.iterator)\r\n\r\n\r\nclass _RepeatSampler:\r\n \"\"\"Sampler that repeats forever\r\n\r\n Args:\r\n sampler (Sampler)\r\n \"\"\"\r\n\r\n def __init__(self, sampler):\r\n self.sampler = sampler\r\n\r\n def __iter__(self):\r\n while True:\r\n yield from iter(self.sampler)\r\n\r\n\r\ndef seed_worker(worker_id):\r\n # Set dataloader worker seed https://pytorch.org/docs/stable/notes/randomness.html#dataloader\r\n worker_seed = torch.initial_seed() % 2 ** 32\r\n np.random.seed(worker_seed)\r\n random.seed(worker_seed)\r\n\r\n\r\ndef build_dataloader(cfg, batch_size, img_path, stride=32, label_path=None, rank=-1, mode=\"train\"):\r\n assert mode in [\"train\", \"val\"]\r\n shuffle = mode == \"train\"\r\n if cfg.rect and shuffle:\r\n LOGGER.warning(\"WARNING β οΈ --rect is incompatible with DataLoader shuffle, setting shuffle=False\")\r\n shuffle = False\r\n with torch_distributed_zero_first(rank): # init dataset *.cache only once if DDP\r\n dataset = YOLODataset(\r\n img_path=img_path,\r\n label_path=label_path,\r\n imgsz=cfg.imgsz,\r\n batch_size=batch_size,\r\n augment=mode == \"train\", # augmentation\r\n hyp=cfg, # TODO: probably add a get_hyps_from_cfg function\r\n rect=cfg.rect if mode == \"train\" else True, # rectangular batches\r\n cache=cfg.get(\"cache\", None),\r\n single_cls=cfg.get(\"single_cls\", False),\r\n stride=int(stride),\r\n pad=0.0 if mode == \"train\" else 0.5,\r\n prefix=colorstr(f\"{mode}: \"),\r\n use_segments=cfg.task == \"segment\",\r\n use_keypoints=cfg.task == \"keypoint\")\r\n\r\n batch_size = min(batch_size, len(dataset))\r\n nd = torch.cuda.device_count() # number of CUDA devices\r\n workers = cfg.workers if mode == \"train\" else cfg.workers * 2\r\n nw = min([os.cpu_count() // max(nd, 1), batch_size if batch_size > 1 else 0, workers]) # number of workers\r\n sampler = None if rank == -1 else distributed.DistributedSampler(dataset, shuffle=shuffle)\r\n loader = DataLoader if cfg.image_weights or cfg.close_mosaic else InfiniteDataLoader # allow attribute updates\r\n generator = torch.Generator()\r\n generator.manual_seed(6148914691236517205 + RANK)\r\n return loader(dataset=dataset,\r\n batch_size=batch_size,\r\n shuffle=shuffle and sampler is None,\r\n num_workers=nw,\r\n sampler=sampler,\r\n pin_memory=PIN_MEMORY,\r\n collate_fn=getattr(dataset, \"collate_fn\", None),\r\n worker_init_fn=seed_worker,\r\n generator=generator), dataset\r\n\r\n\r\n# build classification\r\n# TODO: using cfg like `build_dataloader`\r\ndef build_classification_dataloader(path,\r\n imgsz=224,\r\n batch_size=16,\r\n augment=True,\r\n cache=False,\r\n rank=-1,\r\n workers=8,\r\n shuffle=True):\r\n # Returns Dataloader object to be used with YOLOv5 Classifier\r\n with torch_distributed_zero_first(rank): # init dataset *.cache only once if DDP\r\n dataset = ClassificationDataset(root=path, imgsz=imgsz, augment=augment, cache=cache)\r\n batch_size = min(batch_size, len(dataset))\r\n nd = torch.cuda.device_count()\r\n nw = min([os.cpu_count() // max(nd, 1), batch_size if batch_size > 1 else 0, workers])\r\n sampler = None if rank == -1 else distributed.DistributedSampler(dataset, shuffle=shuffle)\r\n generator = torch.Generator()\r\n generator.manual_seed(6148914691236517205 + RANK)\r\n return InfiniteDataLoader(dataset,\r\n batch_size=batch_size,\r\n shuffle=shuffle and sampler is None,\r\n num_workers=nw,\r\n sampler=sampler,\r\n pin_memory=PIN_MEMORY,\r\n worker_init_fn=seed_worker,\r\n generator=generator) # or DataLoader(persistent_workers=True)\r\n"
},
{
"path": "yolo/data/dataloaders/__init__.py",
"content": ""
},
{
"path": "yolo/data/dataloaders/stream_loaders.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport glob\r\nimport math\r\nimport os\r\nimport time\r\nfrom pathlib import Path\r\nfrom threading import Thread\r\nfrom urllib.parse import urlparse\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport torch\r\n\r\nfrom ultralytics.yolo.data.augment import LetterBox\r\nfrom ultralytics.yolo.data.utils import IMG_FORMATS, VID_FORMATS\r\nfrom ultralytics.yolo.utils import LOGGER, is_colab, is_kaggle, ops\r\nfrom ultralytics.yolo.utils.checks import check_requirements\r\n\r\n\r\nclass LoadStreams:\r\n # YOLOv5 streamloader, i.e. `python detect.py --source 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP streams`\r\n def __init__(self, sources='file.streams', imgsz=640, stride=32, auto=True, transforms=None, vid_stride=1):\r\n torch.backends.cudnn.benchmark = True # faster for fixed-size inference\r\n self.mode = 'stream'\r\n self.imgsz = imgsz\r\n self.stride = stride\r\n self.vid_stride = vid_stride # video frame-rate stride\r\n sources = Path(sources).read_text().rsplit() if os.path.isfile(sources) else [sources]\r\n n = len(sources)\r\n self.sources = [ops.clean_str(x) for x in sources] # clean source names for later\r\n self.imgs, self.fps, self.frames, self.threads = [None] * n, [0] * n, [0] * n, [None] * n\r\n for i, s in enumerate(sources): # index, source\r\n # Start thread to read frames from video stream\r\n st = f'{i + 1}/{n}: {s}... '\r\n if urlparse(s).hostname in ('www.youtube.com', 'youtube.com', 'youtu.be'): # if source is YouTube video\r\n # YouTube format i.e. 'https://www.youtube.com/watch?v=Zgi9g1ksQHc' or 'https://youtu.be/Zgi9g1ksQHc'\r\n check_requirements(('pafy', 'youtube_dl==2020.12.2'))\r\n import pafy\r\n s = pafy.new(s).getbest(preftype=\"mp4\").url # YouTube URL\r\n s = eval(s) if s.isnumeric() else s # i.e. s = '0' local webcam\r\n if s == 0:\r\n assert not is_colab(), '--source 0 webcam unsupported on Colab. Rerun command in a local environment.'\r\n assert not is_kaggle(), '--source 0 webcam unsupported on Kaggle. Rerun command in a local environment.'\r\n cap = cv2.VideoCapture(s)\r\n assert cap.isOpened(), f'{st}Failed to open {s}'\r\n w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\r\n h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\r\n fps = cap.get(cv2.CAP_PROP_FPS) # warning: may return 0 or nan\r\n self.frames[i] = max(int(cap.get(cv2.CAP_PROP_FRAME_COUNT)), 0) or float('inf') # infinite stream fallback\r\n self.fps[i] = max((fps if math.isfinite(fps) else 0) % 100, 0) or 30 # 30 FPS fallback\r\n\r\n _, self.imgs[i] = cap.read() # guarantee first frame\r\n self.threads[i] = Thread(target=self.update, args=([i, cap, s]), daemon=True)\r\n LOGGER.info(f\"{st} Success ({self.frames[i]} frames {w}x{h} at {self.fps[i]:.2f} FPS)\")\r\n self.threads[i].start()\r\n LOGGER.info('') # newline\r\n\r\n # check for common shapes\r\n s = np.stack([LetterBox(imgsz, auto, stride=stride)(image=x).shape for x in self.imgs])\r\n self.rect = np.unique(s, axis=0).shape[0] == 1 # rect inference if all shapes equal\r\n self.auto = auto and self.rect\r\n self.transforms = transforms # optional\r\n if not self.rect:\r\n LOGGER.warning('WARNING β οΈ Stream shapes differ. For optimal performance supply similarly-shaped streams.')\r\n\r\n def update(self, i, cap, stream):\r\n # Read stream `i` frames in daemon thread\r\n n, f = 0, self.frames[i] # frame number, frame array\r\n while cap.isOpened() and n < f:\r\n n += 1\r\n cap.grab() # .read() = .grab() followed by .retrieve()\r\n if n % self.vid_stride == 0:\r\n success, im = cap.retrieve()\r\n if success:\r\n self.imgs[i] = im\r\n else:\r\n LOGGER.warning('WARNING β οΈ Video stream unresponsive, please check your IP camera connection.')\r\n self.imgs[i] = np.zeros_like(self.imgs[i])\r\n cap.open(stream) # re-open stream if signal was lost\r\n time.sleep(0.0) # wait time\r\n\r\n def __iter__(self):\r\n self.count = -1\r\n return self\r\n\r\n def __next__(self):\r\n self.count += 1\r\n if not all(x.is_alive() for x in self.threads) or cv2.waitKey(1) == ord('q'): # q to quit\r\n cv2.destroyAllWindows()\r\n raise StopIteration\r\n\r\n im0 = self.imgs.copy()\r\n if self.transforms:\r\n im = np.stack([self.transforms(x) for x in im0]) # transforms\r\n else:\r\n im = np.stack([LetterBox(self.imgsz, self.auto, stride=self.stride)(image=x) for x in im0])\r\n im = im[..., ::-1].transpose((0, 3, 1, 2)) # BGR to RGB, BHWC to BCHW\r\n im = np.ascontiguousarray(im) # contiguous\r\n\r\n return self.sources, im, im0, None, ''\r\n\r\n def __len__(self):\r\n return len(self.sources) # 1E12 frames = 32 streams at 30 FPS for 30 years\r\n\r\n\r\nclass LoadScreenshots:\r\n # YOLOv5 screenshot dataloader, i.e. `python detect.py --source \"screen 0 100 100 512 256\"`\r\n def __init__(self, source, imgsz=640, stride=32, auto=True, transforms=None):\r\n # source = [screen_number left top width height] (pixels)\r\n check_requirements('mss')\r\n import mss\r\n\r\n source, *params = source.split()\r\n self.screen, left, top, width, height = 0, None, None, None, None # default to full screen 0\r\n if len(params) == 1:\r\n self.screen = int(params[0])\r\n elif len(params) == 4:\r\n left, top, width, height = (int(x) for x in params)\r\n elif len(params) == 5:\r\n self.screen, left, top, width, height = (int(x) for x in params)\r\n self.imgsz = imgsz\r\n self.stride = stride\r\n self.transforms = transforms\r\n self.auto = auto\r\n self.mode = 'stream'\r\n self.frame = 0\r\n self.sct = mss.mss()\r\n\r\n # Parse monitor shape\r\n monitor = self.sct.monitors[self.screen]\r\n self.top = monitor[\"top\"] if top is None else (monitor[\"top\"] + top)\r\n self.left = monitor[\"left\"] if left is None else (monitor[\"left\"] + left)\r\n self.width = width or monitor[\"width\"]\r\n self.height = height or monitor[\"height\"]\r\n self.monitor = {\"left\": self.left, \"top\": self.top, \"width\": self.width, \"height\": self.height}\r\n\r\n def __iter__(self):\r\n return self\r\n\r\n def __next__(self):\r\n # mss screen capture: get raw pixels from the screen as np array\r\n im0 = np.array(self.sct.grab(self.monitor))[:, :, :3] # [:, :, :3] BGRA to BGR\r\n s = f\"screen {self.screen} (LTWH): {self.left},{self.top},{self.width},{self.height}: \"\r\n\r\n if self.transforms:\r\n im = self.transforms(im0) # transforms\r\n else:\r\n im = LetterBox(self.imgsz, self.auto, stride=self.stride)(image=im0)\r\n im = im.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB\r\n im = np.ascontiguousarray(im) # contiguous\r\n self.frame += 1\r\n return str(self.screen), im, im0, None, s # screen, img, original img, im0s, s\r\n\r\n\r\nclass LoadImages:\r\n # YOLOv5 image/video dataloader, i.e. `python detect.py --source image.jpg/vid.mp4`\r\n def __init__(self, path, imgsz=640, stride=32, auto=True, transforms=None, vid_stride=1):\r\n if isinstance(path, str) and Path(path).suffix == \".txt\": # *.txt file with img/vid/dir on each line\r\n path = Path(path).read_text().rsplit()\r\n files = []\r\n for p in sorted(path) if isinstance(path, (list, tuple)) else [path]:\r\n p = str(Path(p).resolve())\r\n if '*' in p:\r\n files.extend(sorted(glob.glob(p, recursive=True))) # glob\r\n elif os.path.isdir(p):\r\n files.extend(sorted(glob.glob(os.path.join(p, '*.*')))) # dir\r\n elif os.path.isfile(p):\r\n files.append(p) # files\r\n else:\r\n raise FileNotFoundError(f'{p} does not exist')\r\n\r\n images = [x for x in files if x.split('.')[-1].lower() in IMG_FORMATS]\r\n videos = [x for x in files if x.split('.')[-1].lower() in VID_FORMATS]\r\n ni, nv = len(images), len(videos)\r\n\r\n self.imgsz = imgsz\r\n self.stride = stride\r\n self.files = images + videos\r\n self.nf = ni + nv # number of files\r\n self.video_flag = [False] * ni + [True] * nv\r\n self.mode = 'image'\r\n self.auto = auto\r\n self.transforms = transforms # optional\r\n self.vid_stride = vid_stride # video frame-rate stride\r\n if any(videos):\r\n self._new_video(videos[0]) # new video\r\n else:\r\n self.cap = None\r\n assert self.nf > 0, f'No images or videos found in {p}. ' \\\r\n f'Supported formats are:\\nimages: {IMG_FORMATS}\\nvideos: {VID_FORMATS}'\r\n\r\n def __iter__(self):\r\n self.count = 0\r\n return self\r\n\r\n def __next__(self):\r\n if self.count == self.nf:\r\n raise StopIteration\r\n path = self.files[self.count]\r\n\r\n if self.video_flag[self.count]:\r\n # Read video\r\n self.mode = 'video'\r\n for _ in range(self.vid_stride):\r\n self.cap.grab()\r\n ret_val, im0 = self.cap.retrieve()\r\n while not ret_val:\r\n self.count += 1\r\n self.cap.release()\r\n if self.count == self.nf: # last video\r\n raise StopIteration\r\n path = self.files[self.count]\r\n self._new_video(path)\r\n ret_val, im0 = self.cap.read()\r\n\r\n self.frame += 1\r\n # im0 = self._cv2_rotate(im0) # for use if cv2 autorotation is False\r\n s = f'video {self.count + 1}/{self.nf} ({self.frame}/{self.frames}) {path}: '\r\n\r\n else:\r\n # Read image\r\n self.count += 1\r\n im0 = cv2.imread(path) # BGR\r\n assert im0 is not None, f'Image Not Found {path}'\r\n s = f'image {self.count}/{self.nf} {path}: '\r\n\r\n if self.transforms:\r\n im = self.transforms(im0) # transforms\r\n else:\r\n im = LetterBox(self.imgsz, self.auto, stride=self.stride)(image=im0)\r\n im = im.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB\r\n im = np.ascontiguousarray(im) # contiguous\r\n\r\n return path, im, im0, self.cap, s\r\n\r\n def _new_video(self, path):\r\n # Create a new video capture object\r\n self.frame = 0\r\n self.cap = cv2.VideoCapture(path)\r\n self.frames = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT) / self.vid_stride)\r\n self.orientation = int(self.cap.get(cv2.CAP_PROP_ORIENTATION_META)) # rotation degrees\r\n # self.cap.set(cv2.CAP_PROP_ORIENTATION_AUTO, 0) # disable https://github.com/ultralytics/yolov5/issues/8493\r\n\r\n def _cv2_rotate(self, im):\r\n # Rotate a cv2 video manually\r\n if self.orientation == 0:\r\n return cv2.rotate(im, cv2.ROTATE_90_CLOCKWISE)\r\n elif self.orientation == 180:\r\n return cv2.rotate(im, cv2.ROTATE_90_COUNTERCLOCKWISE)\r\n elif self.orientation == 90:\r\n return cv2.rotate(im, cv2.ROTATE_180)\r\n return im\r\n\r\n def __len__(self):\r\n return self.nf # number of files\r\n"
},
{
"path": "yolo/data/dataloaders/v5augmentations.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nImage augmentation functions\r\n\"\"\"\r\n\r\nimport math\r\nimport random\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport torch\r\nimport torchvision.transforms as T\r\nimport torchvision.transforms.functional as TF\r\n\r\nfrom ultralytics.yolo.utils import LOGGER, colorstr\r\nfrom ultralytics.yolo.utils.checks import check_version\r\nfrom ultralytics.yolo.utils.metrics import bbox_ioa\r\nfrom ultralytics.yolo.utils.ops import resample_segments, segment2box, xywhn2xyxy\r\n\r\nIMAGENET_MEAN = 0.485, 0.456, 0.406 # RGB mean\r\nIMAGENET_STD = 0.229, 0.224, 0.225 # RGB standard deviation\r\n\r\n\r\nclass Albumentations:\r\n # YOLOv5 Albumentations class (optional, only used if package is installed)\r\n def __init__(self, size=640):\r\n self.transform = None\r\n prefix = colorstr('albumentations: ')\r\n try:\r\n import albumentations as A\r\n check_version(A.__version__, '1.0.3', hard=True) # version requirement\r\n\r\n T = [\r\n A.RandomResizedCrop(height=size, width=size, scale=(0.8, 1.0), ratio=(0.9, 1.11), p=0.0),\r\n A.Blur(p=0.01),\r\n A.MedianBlur(p=0.01),\r\n A.ToGray(p=0.01),\r\n A.CLAHE(p=0.01),\r\n A.RandomBrightnessContrast(p=0.0),\r\n A.RandomGamma(p=0.0),\r\n A.ImageCompression(quality_lower=75, p=0.0)] # transforms\r\n self.transform = A.Compose(T, bbox_params=A.BboxParams(format='yolo', label_fields=['class_labels']))\r\n\r\n LOGGER.info(prefix + ', '.join(f'{x}'.replace('always_apply=False, ', '') for x in T if x.p))\r\n except ImportError: # package not installed, skip\r\n pass\r\n except Exception as e:\r\n LOGGER.info(f'{prefix}{e}')\r\n\r\n def __call__(self, im, labels, p=1.0):\r\n if self.transform and random.random() < p:\r\n new = self.transform(image=im, bboxes=labels[:, 1:], class_labels=labels[:, 0]) # transformed\r\n im, labels = new['image'], np.array([[c, *b] for c, b in zip(new['class_labels'], new['bboxes'])])\r\n return im, labels\r\n\r\n\r\ndef normalize(x, mean=IMAGENET_MEAN, std=IMAGENET_STD, inplace=False):\r\n # Denormalize RGB images x per ImageNet stats in BCHW format, i.e. = (x - mean) / std\r\n return TF.normalize(x, mean, std, inplace=inplace)\r\n\r\n\r\ndef denormalize(x, mean=IMAGENET_MEAN, std=IMAGENET_STD):\r\n # Denormalize RGB images x per ImageNet stats in BCHW format, i.e. = x * std + mean\r\n for i in range(3):\r\n x[:, i] = x[:, i] * std[i] + mean[i]\r\n return x\r\n\r\n\r\ndef augment_hsv(im, hgain=0.5, sgain=0.5, vgain=0.5):\r\n # HSV color-space augmentation\r\n if hgain or sgain or vgain:\r\n r = np.random.uniform(-1, 1, 3) * [hgain, sgain, vgain] + 1 # random gains\r\n hue, sat, val = cv2.split(cv2.cvtColor(im, cv2.COLOR_BGR2HSV))\r\n dtype = im.dtype # uint8\r\n\r\n x = np.arange(0, 256, dtype=r.dtype)\r\n lut_hue = ((x * r[0]) % 180).astype(dtype)\r\n lut_sat = np.clip(x * r[1], 0, 255).astype(dtype)\r\n lut_val = np.clip(x * r[2], 0, 255).astype(dtype)\r\n\r\n im_hsv = cv2.merge((cv2.LUT(hue, lut_hue), cv2.LUT(sat, lut_sat), cv2.LUT(val, lut_val)))\r\n cv2.cvtColor(im_hsv, cv2.COLOR_HSV2BGR, dst=im) # no return needed\r\n\r\n\r\ndef hist_equalize(im, clahe=True, bgr=False):\r\n # Equalize histogram on BGR image 'im' with im.shape(n,m,3) and range 0-255\r\n yuv = cv2.cvtColor(im, cv2.COLOR_BGR2YUV if bgr else cv2.COLOR_RGB2YUV)\r\n if clahe:\r\n c = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))\r\n yuv[:, :, 0] = c.apply(yuv[:, :, 0])\r\n else:\r\n yuv[:, :, 0] = cv2.equalizeHist(yuv[:, :, 0]) # equalize Y channel histogram\r\n return cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR if bgr else cv2.COLOR_YUV2RGB) # convert YUV image to RGB\r\n\r\n\r\ndef replicate(im, labels):\r\n # Replicate labels\r\n h, w = im.shape[:2]\r\n boxes = labels[:, 1:].astype(int)\r\n x1, y1, x2, y2 = boxes.T\r\n s = ((x2 - x1) + (y2 - y1)) / 2 # side length (pixels)\r\n for i in s.argsort()[:round(s.size * 0.5)]: # smallest indices\r\n x1b, y1b, x2b, y2b = boxes[i]\r\n bh, bw = y2b - y1b, x2b - x1b\r\n yc, xc = int(random.uniform(0, h - bh)), int(random.uniform(0, w - bw)) # offset x, y\r\n x1a, y1a, x2a, y2a = [xc, yc, xc + bw, yc + bh]\r\n im[y1a:y2a, x1a:x2a] = im[y1b:y2b, x1b:x2b] # im4[ymin:ymax, xmin:xmax]\r\n labels = np.append(labels, [[labels[i, 0], x1a, y1a, x2a, y2a]], axis=0)\r\n\r\n return im, labels\r\n\r\n\r\ndef letterbox(im, new_shape=(640, 640), color=(114, 114, 114), auto=True, scaleFill=False, scaleup=True, stride=32):\r\n # Resize and pad image while meeting stride-multiple constraints\r\n shape = im.shape[:2] # current shape [height, width]\r\n if isinstance(new_shape, int):\r\n new_shape = (new_shape, new_shape)\r\n\r\n # Scale ratio (new / old)\r\n r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])\r\n if not scaleup: # only scale down, do not scale up (for better val mAP)\r\n r = min(r, 1.0)\r\n\r\n # Compute padding\r\n ratio = r, r # width, height ratios\r\n new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))\r\n dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh padding\r\n if auto: # minimum rectangle\r\n dw, dh = np.mod(dw, stride), np.mod(dh, stride) # wh padding\r\n elif scaleFill: # stretch\r\n dw, dh = 0.0, 0.0\r\n new_unpad = (new_shape[1], new_shape[0])\r\n ratio = new_shape[1] / shape[1], new_shape[0] / shape[0] # width, height ratios\r\n\r\n dw /= 2 # divide padding into 2 sides\r\n dh /= 2\r\n\r\n if shape[::-1] != new_unpad: # resize\r\n im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)\r\n top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))\r\n left, right = int(round(dw - 0.1)), int(round(dw + 0.1))\r\n im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add border\r\n return im, ratio, (dw, dh)\r\n\r\n\r\ndef random_perspective(im,\r\n targets=(),\r\n segments=(),\r\n degrees=10,\r\n translate=.1,\r\n scale=.1,\r\n shear=10,\r\n perspective=0.0,\r\n border=(0, 0)):\r\n # torchvision.transforms.RandomAffine(degrees=(-10, 10), translate=(0.1, 0.1), scale=(0.9, 1.1), shear=(-10, 10))\r\n # targets = [cls, xyxy]\r\n\r\n height = im.shape[0] + border[0] * 2 # shape(h,w,c)\r\n width = im.shape[1] + border[1] * 2\r\n\r\n # Center\r\n C = np.eye(3)\r\n C[0, 2] = -im.shape[1] / 2 # x translation (pixels)\r\n C[1, 2] = -im.shape[0] / 2 # y translation (pixels)\r\n\r\n # Perspective\r\n P = np.eye(3)\r\n P[2, 0] = random.uniform(-perspective, perspective) # x perspective (about y)\r\n P[2, 1] = random.uniform(-perspective, perspective) # y perspective (about x)\r\n\r\n # Rotation and Scale\r\n R = np.eye(3)\r\n a = random.uniform(-degrees, degrees)\r\n # a += random.choice([-180, -90, 0, 90]) # add 90deg rotations to small rotations\r\n s = random.uniform(1 - scale, 1 + scale)\r\n # s = 2 ** random.uniform(-scale, scale)\r\n R[:2] = cv2.getRotationMatrix2D(angle=a, center=(0, 0), scale=s)\r\n\r\n # Shear\r\n S = np.eye(3)\r\n S[0, 1] = math.tan(random.uniform(-shear, shear) * math.pi / 180) # x shear (deg)\r\n S[1, 0] = math.tan(random.uniform(-shear, shear) * math.pi / 180) # y shear (deg)\r\n\r\n # Translation\r\n T = np.eye(3)\r\n T[0, 2] = random.uniform(0.5 - translate, 0.5 + translate) * width # x translation (pixels)\r\n T[1, 2] = random.uniform(0.5 - translate, 0.5 + translate) * height # y translation (pixels)\r\n\r\n # Combined rotation matrix\r\n M = T @ S @ R @ P @ C # order of operations (right to left) is IMPORTANT\r\n if (border[0] != 0) or (border[1] != 0) or (M != np.eye(3)).any(): # image changed\r\n if perspective:\r\n im = cv2.warpPerspective(im, M, dsize=(width, height), borderValue=(114, 114, 114))\r\n else: # affine\r\n im = cv2.warpAffine(im, M[:2], dsize=(width, height), borderValue=(114, 114, 114))\r\n\r\n # Visualize\r\n # import matplotlib.pyplot as plt\r\n # ax = plt.subplots(1, 2, figsize=(12, 6))[1].ravel()\r\n # ax[0].imshow(im[:, :, ::-1]) # base\r\n # ax[1].imshow(im2[:, :, ::-1]) # warped\r\n\r\n # Transform label coordinates\r\n n = len(targets)\r\n if n:\r\n use_segments = any(x.any() for x in segments)\r\n new = np.zeros((n, 4))\r\n if use_segments: # warp segments\r\n segments = resample_segments(segments) # upsample\r\n for i, segment in enumerate(segments):\r\n xy = np.ones((len(segment), 3))\r\n xy[:, :2] = segment\r\n xy = xy @ M.T # transform\r\n xy = xy[:, :2] / xy[:, 2:3] if perspective else xy[:, :2] # perspective rescale or affine\r\n\r\n # clip\r\n new[i] = segment2box(xy, width, height)\r\n\r\n else: # warp boxes\r\n xy = np.ones((n * 4, 3))\r\n xy[:, :2] = targets[:, [1, 2, 3, 4, 1, 4, 3, 2]].reshape(n * 4, 2) # x1y1, x2y2, x1y2, x2y1\r\n xy = xy @ M.T # transform\r\n xy = (xy[:, :2] / xy[:, 2:3] if perspective else xy[:, :2]).reshape(n, 8) # perspective rescale or affine\r\n\r\n # create new boxes\r\n x = xy[:, [0, 2, 4, 6]]\r\n y = xy[:, [1, 3, 5, 7]]\r\n new = np.concatenate((x.min(1), y.min(1), x.max(1), y.max(1))).reshape(4, n).T\r\n\r\n # clip\r\n new[:, [0, 2]] = new[:, [0, 2]].clip(0, width)\r\n new[:, [1, 3]] = new[:, [1, 3]].clip(0, height)\r\n\r\n # filter candidates\r\n i = box_candidates(box1=targets[:, 1:5].T * s, box2=new.T, area_thr=0.01 if use_segments else 0.10)\r\n targets = targets[i]\r\n targets[:, 1:5] = new[i]\r\n\r\n return im, targets\r\n\r\n\r\ndef copy_paste(im, labels, segments, p=0.5):\r\n # Implement Copy-Paste augmentation https://arxiv.org/abs/2012.07177, labels as nx5 np.array(cls, xyxy)\r\n n = len(segments)\r\n if p and n:\r\n h, w, c = im.shape # height, width, channels\r\n im_new = np.zeros(im.shape, np.uint8)\r\n\r\n # calculate ioa first then select indexes randomly\r\n boxes = np.stack([w - labels[:, 3], labels[:, 2], w - labels[:, 1], labels[:, 4]], axis=-1) # (n, 4)\r\n ioa = bbox_ioa(boxes, labels[:, 1:5]) # intersection over area\r\n indexes = np.nonzero((ioa < 0.30).all(1))[0] # (N, )\r\n n = len(indexes)\r\n for j in random.sample(list(indexes), k=round(p * n)):\r\n l, box, s = labels[j], boxes[j], segments[j]\r\n labels = np.concatenate((labels, [[l[0], *box]]), 0)\r\n segments.append(np.concatenate((w - s[:, 0:1], s[:, 1:2]), 1))\r\n cv2.drawContours(im_new, [segments[j].astype(np.int32)], -1, (1, 1, 1), cv2.FILLED)\r\n\r\n result = cv2.flip(im, 1) # augment segments (flip left-right)\r\n i = cv2.flip(im_new, 1).astype(bool)\r\n im[i] = result[i] # cv2.imwrite('debug.jpg', im) # debug\r\n\r\n return im, labels, segments\r\n\r\n\r\ndef cutout(im, labels, p=0.5):\r\n # Applies image cutout augmentation https://arxiv.org/abs/1708.04552\r\n if random.random() < p:\r\n h, w = im.shape[:2]\r\n scales = [0.5] * 1 + [0.25] * 2 + [0.125] * 4 + [0.0625] * 8 + [0.03125] * 16 # image size fraction\r\n for s in scales:\r\n mask_h = random.randint(1, int(h * s)) # create random masks\r\n mask_w = random.randint(1, int(w * s))\r\n\r\n # box\r\n xmin = max(0, random.randint(0, w) - mask_w // 2)\r\n ymin = max(0, random.randint(0, h) - mask_h // 2)\r\n xmax = min(w, xmin + mask_w)\r\n ymax = min(h, ymin + mask_h)\r\n\r\n # apply random color mask\r\n im[ymin:ymax, xmin:xmax] = [random.randint(64, 191) for _ in range(3)]\r\n\r\n # return unobscured labels\r\n if len(labels) and s > 0.03:\r\n box = np.array([[xmin, ymin, xmax, ymax]], dtype=np.float32)\r\n ioa = bbox_ioa(box, xywhn2xyxy(labels[:, 1:5], w, h))[0] # intersection over area\r\n labels = labels[ioa < 0.60] # remove >60% obscured labels\r\n\r\n return labels\r\n\r\n\r\ndef mixup(im, labels, im2, labels2):\r\n # Applies MixUp augmentation https://arxiv.org/pdf/1710.09412.pdf\r\n r = np.random.beta(32.0, 32.0) # mixup ratio, alpha=beta=32.0\r\n im = (im * r + im2 * (1 - r)).astype(np.uint8)\r\n labels = np.concatenate((labels, labels2), 0)\r\n return im, labels\r\n\r\n\r\ndef box_candidates(box1, box2, wh_thr=2, ar_thr=100, area_thr=0.1, eps=1e-16): # box1(4,n), box2(4,n)\r\n # Compute candidate boxes: box1 before augment, box2 after augment, wh_thr (pixels), aspect_ratio_thr, area_ratio\r\n w1, h1 = box1[2] - box1[0], box1[3] - box1[1]\r\n w2, h2 = box2[2] - box2[0], box2[3] - box2[1]\r\n ar = np.maximum(w2 / (h2 + eps), h2 / (w2 + eps)) # aspect ratio\r\n return (w2 > wh_thr) & (h2 > wh_thr) & (w2 * h2 / (w1 * h1 + eps) > area_thr) & (ar < ar_thr) # candidates\r\n\r\n\r\ndef classify_albumentations(\r\n augment=True,\r\n size=224,\r\n scale=(0.08, 1.0),\r\n ratio=(0.75, 1.0 / 0.75), # 0.75, 1.33\r\n hflip=0.5,\r\n vflip=0.0,\r\n jitter=0.4,\r\n mean=IMAGENET_MEAN,\r\n std=IMAGENET_STD,\r\n auto_aug=False):\r\n # YOLOv5 classification Albumentations (optional, only used if package is installed)\r\n prefix = colorstr('albumentations: ')\r\n try:\r\n import albumentations as A\r\n from albumentations.pytorch import ToTensorV2\r\n check_version(A.__version__, '1.0.3', hard=True) # version requirement\r\n if augment: # Resize and crop\r\n T = [A.RandomResizedCrop(height=size, width=size, scale=scale, ratio=ratio)]\r\n if auto_aug:\r\n # TODO: implement AugMix, AutoAug & RandAug in albumentation\r\n LOGGER.info(f'{prefix}auto augmentations are currently not supported')\r\n else:\r\n if hflip > 0:\r\n T += [A.HorizontalFlip(p=hflip)]\r\n if vflip > 0:\r\n T += [A.VerticalFlip(p=vflip)]\r\n if jitter > 0:\r\n color_jitter = (float(jitter),) * 3 # repeat value for brightness, contrast, satuaration, 0 hue\r\n T += [A.ColorJitter(*color_jitter, 0)]\r\n else: # Use fixed crop for eval set (reproducibility)\r\n T = [A.SmallestMaxSize(max_size=size), A.CenterCrop(height=size, width=size)]\r\n T += [A.Normalize(mean=mean, std=std), ToTensorV2()] # Normalize and convert to Tensor\r\n LOGGER.info(prefix + ', '.join(f'{x}'.replace('always_apply=False, ', '') for x in T if x.p))\r\n return A.Compose(T)\r\n\r\n except ImportError: # package not installed, skip\r\n LOGGER.warning(f'{prefix}β οΈ not found, install with `pip install albumentations` (recommended)')\r\n except Exception as e:\r\n LOGGER.info(f'{prefix}{e}')\r\n\r\n\r\ndef classify_transforms(size=224):\r\n # Transforms to apply if albumentations not installed\r\n assert isinstance(size, int), f'ERROR: classify_transforms size {size} must be integer, not (list, tuple)'\r\n # T.Compose([T.ToTensor(), T.Resize(size), T.CenterCrop(size), T.Normalize(IMAGENET_MEAN, IMAGENET_STD)])\r\n return T.Compose([CenterCrop(size), ToTensor(), T.Normalize(IMAGENET_MEAN, IMAGENET_STD)])\r\n\r\n\r\nclass LetterBox:\r\n # YOLOv5 LetterBox class for image preprocessing, i.e. T.Compose([LetterBox(size), ToTensor()])\r\n def __init__(self, size=(640, 640), auto=False, stride=32):\r\n super().__init__()\r\n self.h, self.w = (size, size) if isinstance(size, int) else size\r\n self.auto = auto # pass max size integer, automatically solve for short side using stride\r\n self.stride = stride # used with auto\r\n\r\n def __call__(self, im): # im = np.array HWC\r\n imh, imw = im.shape[:2]\r\n r = min(self.h / imh, self.w / imw) # ratio of new/old\r\n h, w = round(imh * r), round(imw * r) # resized image\r\n hs, ws = (math.ceil(x / self.stride) * self.stride for x in (h, w)) if self.auto else self.h, self.w\r\n top, left = round((hs - h) / 2 - 0.1), round((ws - w) / 2 - 0.1)\r\n im_out = np.full((self.h, self.w, 3), 114, dtype=im.dtype)\r\n im_out[top:top + h, left:left + w] = cv2.resize(im, (w, h), interpolation=cv2.INTER_LINEAR)\r\n return im_out\r\n\r\n\r\nclass CenterCrop:\r\n # YOLOv5 CenterCrop class for image preprocessing, i.e. T.Compose([CenterCrop(size), ToTensor()])\r\n def __init__(self, size=640):\r\n super().__init__()\r\n self.h, self.w = (size, size) if isinstance(size, int) else size\r\n\r\n def __call__(self, im): # im = np.array HWC\r\n imh, imw = im.shape[:2]\r\n m = min(imh, imw) # min dimension\r\n top, left = (imh - m) // 2, (imw - m) // 2\r\n return cv2.resize(im[top:top + m, left:left + m], (self.w, self.h), interpolation=cv2.INTER_LINEAR)\r\n\r\n\r\nclass ToTensor:\r\n # YOLOv5 ToTensor class for image preprocessing, i.e. T.Compose([LetterBox(size), ToTensor()])\r\n def __init__(self, half=False):\r\n super().__init__()\r\n self.half = half\r\n\r\n def __call__(self, im): # im = np.array HWC in BGR order\r\n im = np.ascontiguousarray(im.transpose((2, 0, 1))[::-1]) # HWC to CHW -> BGR to RGB -> contiguous\r\n im = torch.from_numpy(im) # to torch\r\n im = im.half() if self.half else im.float() # uint8 to fp16/32\r\n im /= 255.0 # 0-255 to 0.0-1.0\r\n return im\r\n"
},
{
"path": "yolo/data/dataloaders/v5loader.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nDataloaders and dataset utils\r\n\"\"\"\r\n\r\nimport contextlib\r\nimport glob\r\nimport hashlib\r\nimport json\r\nimport math\r\nimport os\r\nimport random\r\nimport shutil\r\nimport time\r\nfrom itertools import repeat\r\nfrom multiprocessing.pool import Pool, ThreadPool\r\nfrom pathlib import Path\r\nfrom threading import Thread\r\nfrom urllib.parse import urlparse\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport psutil\r\nimport torch\r\nimport torchvision\r\nimport yaml\r\nfrom PIL import ExifTags, Image, ImageOps\r\nfrom torch.utils.data import DataLoader, Dataset, dataloader, distributed\r\nfrom tqdm import tqdm\r\n\r\nfrom ultralytics.yolo.data.utils import check_dataset, unzip_file\r\nfrom ultralytics.yolo.utils import DATASETS_DIR, LOGGER, NUM_THREADS, TQDM_BAR_FORMAT, is_colab, is_kaggle\r\nfrom ultralytics.yolo.utils.checks import check_requirements, check_yaml\r\nfrom ultralytics.yolo.utils.ops import clean_str, segments2boxes, xyn2xy, xywh2xyxy, xywhn2xyxy, xyxy2xywhn\r\nfrom ultralytics.yolo.utils.torch_utils import torch_distributed_zero_first\r\n\r\nfrom .v5augmentations import (Albumentations, augment_hsv, classify_albumentations, classify_transforms, copy_paste,\r\n letterbox, mixup, random_perspective)\r\n\r\n# Parameters\r\nHELP_URL = 'See https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data'\r\nIMG_FORMATS = 'bmp', 'dng', 'jpeg', 'jpg', 'mpo', 'png', 'tif', 'tiff', 'webp', 'pfm' # include image suffixes\r\nVID_FORMATS = 'asf', 'avi', 'gif', 'm4v', 'mkv', 'mov', 'mp4', 'mpeg', 'mpg', 'ts', 'wmv' # include video suffixes\r\nLOCAL_RANK = int(os.getenv('LOCAL_RANK', -1)) # https://pytorch.org/docs/stable/elastic/run.html\r\nRANK = int(os.getenv('RANK', -1))\r\nPIN_MEMORY = str(os.getenv('PIN_MEMORY', True)).lower() == 'true' # global pin_memory for dataloaders\r\n\r\n# Get orientation exif tag\r\nfor orientation in ExifTags.TAGS.keys():\r\n if ExifTags.TAGS[orientation] == 'Orientation':\r\n break\r\n\r\n\r\ndef get_hash(paths):\r\n # Returns a single hash value of a list of paths (files or dirs)\r\n size = sum(os.path.getsize(p) for p in paths if os.path.exists(p)) # sizes\r\n h = hashlib.md5(str(size).encode()) # hash sizes\r\n h.update(''.join(paths).encode()) # hash paths\r\n return h.hexdigest() # return hash\r\n\r\n\r\ndef exif_size(img):\r\n # Returns exif-corrected PIL size\r\n s = img.size # (width, height)\r\n with contextlib.suppress(Exception):\r\n rotation = dict(img._getexif().items())[orientation]\r\n if rotation in [6, 8]: # rotation 270 or 90\r\n s = (s[1], s[0])\r\n return s\r\n\r\n\r\ndef exif_transpose(image):\r\n \"\"\"\r\n Transpose a PIL image accordingly if it has an EXIF Orientation tag.\r\n Inplace version of https://github.com/python-pillow/Pillow/blob/master/src/PIL/ImageOps.py exif_transpose()\r\n\r\n :param image: The image to transpose.\r\n :return: An image.\r\n \"\"\"\r\n exif = image.getexif()\r\n orientation = exif.get(0x0112, 1) # default 1\r\n if orientation > 1:\r\n method = {\r\n 2: Image.FLIP_LEFT_RIGHT,\r\n 3: Image.ROTATE_180,\r\n 4: Image.FLIP_TOP_BOTTOM,\r\n 5: Image.TRANSPOSE,\r\n 6: Image.ROTATE_270,\r\n 7: Image.TRANSVERSE,\r\n 8: Image.ROTATE_90}.get(orientation)\r\n if method is not None:\r\n image = image.transpose(method)\r\n del exif[0x0112]\r\n image.info[\"exif\"] = exif.tobytes()\r\n return image\r\n\r\n\r\ndef seed_worker(worker_id):\r\n # Set dataloader worker seed https://pytorch.org/docs/stable/notes/randomness.html#dataloader\r\n worker_seed = torch.initial_seed() % 2 ** 32\r\n np.random.seed(worker_seed)\r\n random.seed(worker_seed)\r\n\r\n\r\ndef create_dataloader(path,\r\n imgsz,\r\n batch_size,\r\n stride,\r\n single_cls=False,\r\n hyp=None,\r\n augment=False,\r\n cache=False,\r\n pad=0.0,\r\n rect=False,\r\n rank=-1,\r\n workers=8,\r\n image_weights=False,\r\n close_mosaic=False,\r\n min_items=0,\r\n prefix='',\r\n shuffle=False,\r\n seed=0):\r\n if rect and shuffle:\r\n LOGGER.warning('WARNING β οΈ --rect is incompatible with DataLoader shuffle, setting shuffle=False')\r\n shuffle = False\r\n with torch_distributed_zero_first(rank): # init dataset *.cache only once if DDP\r\n dataset = LoadImagesAndLabels(\r\n path,\r\n imgsz,\r\n batch_size,\r\n augment=augment, # augmentation\r\n hyp=hyp, # hyperparameters\r\n rect=rect, # rectangular batches\r\n cache_images=cache,\r\n single_cls=single_cls,\r\n stride=int(stride),\r\n pad=pad,\r\n image_weights=image_weights,\r\n min_items=min_items,\r\n prefix=prefix)\r\n\r\n batch_size = min(batch_size, len(dataset))\r\n nd = torch.cuda.device_count() # number of CUDA devices\r\n nw = min([os.cpu_count() // max(nd, 1), batch_size if batch_size > 1 else 0, workers]) # number of workers\r\n sampler = None if rank == -1 else distributed.DistributedSampler(dataset, shuffle=shuffle)\r\n loader = DataLoader if image_weights or close_mosaic else InfiniteDataLoader # DataLoader allows attribute updates\r\n generator = torch.Generator()\r\n generator.manual_seed(6148914691236517205 + seed + RANK)\r\n return loader(dataset,\r\n batch_size=batch_size,\r\n shuffle=shuffle and sampler is None,\r\n num_workers=nw,\r\n sampler=sampler,\r\n pin_memory=PIN_MEMORY,\r\n collate_fn=LoadImagesAndLabels.collate_fn,\r\n worker_init_fn=seed_worker,\r\n generator=generator), dataset\r\n\r\n\r\nclass InfiniteDataLoader(dataloader.DataLoader):\r\n \"\"\" Dataloader that reuses workers\r\n\r\n Uses same syntax as vanilla DataLoader\r\n \"\"\"\r\n\r\n def __init__(self, *args, **kwargs):\r\n super().__init__(*args, **kwargs)\r\n object.__setattr__(self, 'batch_sampler', _RepeatSampler(self.batch_sampler))\r\n self.iterator = super().__iter__()\r\n\r\n def __len__(self):\r\n return len(self.batch_sampler.sampler)\r\n\r\n def __iter__(self):\r\n for _ in range(len(self)):\r\n yield next(self.iterator)\r\n\r\n\r\nclass _RepeatSampler:\r\n \"\"\" Sampler that repeats forever\r\n\r\n Args:\r\n sampler (Sampler)\r\n \"\"\"\r\n\r\n def __init__(self, sampler):\r\n self.sampler = sampler\r\n\r\n def __iter__(self):\r\n while True:\r\n yield from iter(self.sampler)\r\n\r\n\r\nclass LoadScreenshots:\r\n # YOLOv5 screenshot dataloader, i.e. `python detect.py --source \"screen 0 100 100 512 256\"`\r\n def __init__(self, source, img_size=640, stride=32, auto=True, transforms=None):\r\n # source = [screen_number left top width height] (pixels)\r\n check_requirements('mss')\r\n import mss\r\n\r\n source, *params = source.split()\r\n self.screen, left, top, width, height = 0, None, None, None, None # default to full screen 0\r\n if len(params) == 1:\r\n self.screen = int(params[0])\r\n elif len(params) == 4:\r\n left, top, width, height = (int(x) for x in params)\r\n elif len(params) == 5:\r\n self.screen, left, top, width, height = (int(x) for x in params)\r\n self.img_size = img_size\r\n self.stride = stride\r\n self.transforms = transforms\r\n self.auto = auto\r\n self.mode = 'stream'\r\n self.frame = 0\r\n self.sct = mss.mss()\r\n\r\n # Parse monitor shape\r\n monitor = self.sct.monitors[self.screen]\r\n self.top = monitor[\"top\"] if top is None else (monitor[\"top\"] + top)\r\n self.left = monitor[\"left\"] if left is None else (monitor[\"left\"] + left)\r\n self.width = width or monitor[\"width\"]\r\n self.height = height or monitor[\"height\"]\r\n self.monitor = {\"left\": self.left, \"top\": self.top, \"width\": self.width, \"height\": self.height}\r\n\r\n def __iter__(self):\r\n return self\r\n\r\n def __next__(self):\r\n # mss screen capture: get raw pixels from the screen as np array\r\n im0 = np.array(self.sct.grab(self.monitor))[:, :, :3] # [:, :, :3] BGRA to BGR\r\n s = f\"screen {self.screen} (LTWH): {self.left},{self.top},{self.width},{self.height}: \"\r\n\r\n if self.transforms:\r\n im = self.transforms(im0) # transforms\r\n else:\r\n im = letterbox(im0, self.img_size, stride=self.stride, auto=self.auto)[0] # padded resize\r\n im = im.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB\r\n im = np.ascontiguousarray(im) # contiguous\r\n self.frame += 1\r\n return str(self.screen), im, im0, None, s # screen, img, original img, im0s, s\r\n\r\n\r\nclass LoadImages:\r\n # YOLOv5 image/video dataloader, i.e. `python detect.py --source image.jpg/vid.mp4`\r\n def __init__(self, path, img_size=640, stride=32, auto=True, transforms=None, vid_stride=1):\r\n if isinstance(path, str) and Path(path).suffix == \".txt\": # *.txt file with img/vid/dir on each line\r\n path = Path(path).read_text().rsplit()\r\n files = []\r\n for p in sorted(path) if isinstance(path, (list, tuple)) else [path]:\r\n p = str(Path(p).resolve())\r\n if '*' in p:\r\n files.extend(sorted(glob.glob(p, recursive=True))) # glob\r\n elif os.path.isdir(p):\r\n files.extend(sorted(glob.glob(os.path.join(p, '*.*')))) # dir\r\n elif os.path.isfile(p):\r\n files.append(p) # files\r\n else:\r\n raise FileNotFoundError(f'{p} does not exist')\r\n\r\n images = [x for x in files if x.split('.')[-1].lower() in IMG_FORMATS]\r\n videos = [x for x in files if x.split('.')[-1].lower() in VID_FORMATS]\r\n ni, nv = len(images), len(videos)\r\n\r\n self.img_size = img_size\r\n self.stride = stride\r\n self.files = images + videos\r\n self.nf = ni + nv # number of files\r\n self.video_flag = [False] * ni + [True] * nv\r\n self.mode = 'image'\r\n self.auto = auto\r\n self.transforms = transforms # optional\r\n self.vid_stride = vid_stride # video frame-rate stride\r\n if any(videos):\r\n self._new_video(videos[0]) # new video\r\n else:\r\n self.cap = None\r\n assert self.nf > 0, f'No images or videos found in {p}. ' \\\r\n f'Supported formats are:\\nimages: {IMG_FORMATS}\\nvideos: {VID_FORMATS}'\r\n\r\n def __iter__(self):\r\n self.count = 0\r\n return self\r\n\r\n def __next__(self):\r\n if self.count == self.nf:\r\n raise StopIteration\r\n path = self.files[self.count]\r\n\r\n if self.video_flag[self.count]:\r\n # Read video\r\n self.mode = 'video'\r\n for _ in range(self.vid_stride):\r\n self.cap.grab()\r\n ret_val, im0 = self.cap.retrieve()\r\n while not ret_val:\r\n self.count += 1\r\n self.cap.release()\r\n if self.count == self.nf: # last video\r\n raise StopIteration\r\n path = self.files[self.count]\r\n self._new_video(path)\r\n ret_val, im0 = self.cap.read()\r\n\r\n self.frame += 1\r\n # im0 = self._cv2_rotate(im0) # for use if cv2 autorotation is False\r\n s = f'video {self.count + 1}/{self.nf} ({self.frame}/{self.frames}) {path}: '\r\n\r\n else:\r\n # Read image\r\n self.count += 1\r\n im0 = cv2.imread(path) # BGR\r\n assert im0 is not None, f'Image Not Found {path}'\r\n s = f'image {self.count}/{self.nf} {path}: '\r\n\r\n if self.transforms:\r\n im = self.transforms(im0) # transforms\r\n else:\r\n im = letterbox(im0, self.img_size, stride=self.stride, auto=self.auto)[0] # padded resize\r\n im = im.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB\r\n im = np.ascontiguousarray(im) # contiguous\r\n\r\n return path, im, im0, self.cap, s\r\n\r\n def _new_video(self, path):\r\n # Create a new video capture object\r\n self.frame = 0\r\n self.cap = cv2.VideoCapture(path)\r\n self.frames = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT) / self.vid_stride)\r\n self.orientation = int(self.cap.get(cv2.CAP_PROP_ORIENTATION_META)) # rotation degrees\r\n # self.cap.set(cv2.CAP_PROP_ORIENTATION_AUTO, 0) # disable https://github.com/ultralytics/yolov5/issues/8493\r\n\r\n def _cv2_rotate(self, im):\r\n # Rotate a cv2 video manually\r\n if self.orientation == 0:\r\n return cv2.rotate(im, cv2.ROTATE_90_CLOCKWISE)\r\n elif self.orientation == 180:\r\n return cv2.rotate(im, cv2.ROTATE_90_COUNTERCLOCKWISE)\r\n elif self.orientation == 90:\r\n return cv2.rotate(im, cv2.ROTATE_180)\r\n return im\r\n\r\n def __len__(self):\r\n return self.nf # number of files\r\n\r\n\r\nclass LoadStreams:\r\n # YOLOv5 streamloader, i.e. `python detect.py --source 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP streams`\r\n def __init__(self, sources='file.streams', img_size=640, stride=32, auto=True, transforms=None, vid_stride=1):\r\n torch.backends.cudnn.benchmark = True # faster for fixed-size inference\r\n self.mode = 'stream'\r\n self.img_size = img_size\r\n self.stride = stride\r\n self.vid_stride = vid_stride # video frame-rate stride\r\n sources = Path(sources).read_text().rsplit() if os.path.isfile(sources) else [sources]\r\n n = len(sources)\r\n self.sources = [clean_str(x) for x in sources] # clean source names for later\r\n self.imgs, self.fps, self.frames, self.threads = [None] * n, [0] * n, [0] * n, [None] * n\r\n for i, s in enumerate(sources): # index, source\r\n # Start thread to read frames from video stream\r\n st = f'{i + 1}/{n}: {s}... '\r\n if urlparse(s).hostname in ('www.youtube.com', 'youtube.com', 'youtu.be'): # if source is YouTube video\r\n # YouTube format i.e. 'https://www.youtube.com/watch?v=Zgi9g1ksQHc' or 'https://youtu.be/Zgi9g1ksQHc'\r\n check_requirements(('pafy', 'youtube_dl==2020.12.2'))\r\n import pafy\r\n s = pafy.new(s).getbest(preftype=\"mp4\").url # YouTube URL\r\n s = eval(s) if s.isnumeric() else s # i.e. s = '0' local webcam\r\n if s == 0:\r\n assert not is_colab(), '--source 0 webcam unsupported on Colab. Rerun command in a local environment.'\r\n assert not is_kaggle(), '--source 0 webcam unsupported on Kaggle. Rerun command in a local environment.'\r\n cap = cv2.VideoCapture(s)\r\n assert cap.isOpened(), f'{st}Failed to open {s}'\r\n w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\r\n h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\r\n fps = cap.get(cv2.CAP_PROP_FPS) # warning: may return 0 or nan\r\n self.frames[i] = max(int(cap.get(cv2.CAP_PROP_FRAME_COUNT)), 0) or float('inf') # infinite stream fallback\r\n self.fps[i] = max((fps if math.isfinite(fps) else 0) % 100, 0) or 30 # 30 FPS fallback\r\n\r\n _, self.imgs[i] = cap.read() # guarantee first frame\r\n self.threads[i] = Thread(target=self.update, args=([i, cap, s]), daemon=True)\r\n LOGGER.info(f\"{st} Success ({self.frames[i]} frames {w}x{h} at {self.fps[i]:.2f} FPS)\")\r\n self.threads[i].start()\r\n LOGGER.info('') # newline\r\n\r\n # check for common shapes\r\n s = np.stack([letterbox(x, img_size, stride=stride, auto=auto)[0].shape for x in self.imgs])\r\n self.rect = np.unique(s, axis=0).shape[0] == 1 # rect inference if all shapes equal\r\n self.auto = auto and self.rect\r\n self.transforms = transforms # optional\r\n if not self.rect:\r\n LOGGER.warning('WARNING β οΈ Stream shapes differ. For optimal performance supply similarly-shaped streams.')\r\n\r\n def update(self, i, cap, stream):\r\n # Read stream `i` frames in daemon thread\r\n n, f = 0, self.frames[i] # frame number, frame array\r\n while cap.isOpened() and n < f:\r\n n += 1\r\n cap.grab() # .read() = .grab() followed by .retrieve()\r\n if n % self.vid_stride == 0:\r\n success, im = cap.retrieve()\r\n if success:\r\n self.imgs[i] = im\r\n else:\r\n LOGGER.warning('WARNING β οΈ Video stream unresponsive, please check your IP camera connection.')\r\n self.imgs[i] = np.zeros_like(self.imgs[i])\r\n cap.open(stream) # re-open stream if signal was lost\r\n time.sleep(0.0) # wait time\r\n\r\n def __iter__(self):\r\n self.count = -1\r\n return self\r\n\r\n def __next__(self):\r\n self.count += 1\r\n if not all(x.is_alive() for x in self.threads) or cv2.waitKey(1) == ord('q'): # q to quit\r\n cv2.destroyAllWindows()\r\n raise StopIteration\r\n\r\n im0 = self.imgs.copy()\r\n if self.transforms:\r\n im = np.stack([self.transforms(x) for x in im0]) # transforms\r\n else:\r\n im = np.stack([letterbox(x, self.img_size, stride=self.stride, auto=self.auto)[0] for x in im0]) # resize\r\n im = im[..., ::-1].transpose((0, 3, 1, 2)) # BGR to RGB, BHWC to BCHW\r\n im = np.ascontiguousarray(im) # contiguous\r\n\r\n return self.sources, im, im0, None, ''\r\n\r\n def __len__(self):\r\n return len(self.sources) # 1E12 frames = 32 streams at 30 FPS for 30 years\r\n\r\n\r\ndef img2label_paths(img_paths):\r\n # Define label paths as a function of image paths\r\n sa, sb = f'{os.sep}images{os.sep}', f'{os.sep}labels{os.sep}' # /images/, /labels/ substrings\r\n return [sb.join(x.rsplit(sa, 1)).rsplit('.', 1)[0] + '.txt' for x in img_paths]\r\n\r\n\r\nclass LoadImagesAndLabels(Dataset):\r\n # YOLOv5 train_loader/val_loader, loads images and labels for training and validation\r\n cache_version = 0.6 # dataset labels *.cache version\r\n rand_interp_methods = [cv2.INTER_NEAREST, cv2.INTER_LINEAR, cv2.INTER_CUBIC, cv2.INTER_AREA, cv2.INTER_LANCZOS4]\r\n\r\n def __init__(self,\r\n path,\r\n img_size=640,\r\n batch_size=16,\r\n augment=False,\r\n hyp=None,\r\n rect=False,\r\n image_weights=False,\r\n cache_images=False,\r\n single_cls=False,\r\n stride=32,\r\n pad=0.0,\r\n min_items=0,\r\n prefix=''):\r\n self.img_size = img_size\r\n self.augment = augment\r\n self.hyp = hyp\r\n self.image_weights = image_weights\r\n self.rect = False if image_weights else rect\r\n self.mosaic = self.augment and not self.rect # load 4 images at a time into a mosaic (only during training)\r\n self.mosaic_border = [-img_size // 2, -img_size // 2]\r\n self.stride = stride\r\n self.path = path\r\n self.albumentations = Albumentations(size=img_size) if augment else None\r\n\r\n try:\r\n f = [] # image files\r\n for p in path if isinstance(path, list) else [path]:\r\n p = Path(p) # os-agnostic\r\n if p.is_dir(): # dir\r\n f += glob.glob(str(p / '**' / '*.*'), recursive=True)\r\n # f = list(p.rglob('*.*')) # pathlib\r\n elif p.is_file(): # file\r\n with open(p) as t:\r\n t = t.read().strip().splitlines()\r\n parent = str(p.parent) + os.sep\r\n f += [x.replace('./', parent, 1) if x.startswith('./') else x for x in t] # to global path\r\n # f += [p.parent / x.lstrip(os.sep) for x in t] # to global path (pathlib)\r\n else:\r\n raise FileNotFoundError(f'{prefix}{p} does not exist')\r\n self.im_files = sorted(x.replace('/', os.sep) for x in f if x.split('.')[-1].lower() in IMG_FORMATS)\r\n # self.img_files = sorted([x for x in f if x.suffix[1:].lower() in IMG_FORMATS]) # pathlib\r\n assert self.im_files, f'{prefix}No images found'\r\n except Exception as e:\r\n raise FileNotFoundError(f'{prefix}Error loading data from {path}: {e}\\n{HELP_URL}') from e\r\n\r\n # Check cache\r\n self.label_files = img2label_paths(self.im_files) # labels\r\n cache_path = (p if p.is_file() else Path(self.label_files[0]).parent).with_suffix('.cache')\r\n try:\r\n cache, exists = np.load(cache_path, allow_pickle=True).item(), True # load dict\r\n assert cache['version'] == self.cache_version # matches current version\r\n assert cache['hash'] == get_hash(self.label_files + self.im_files) # identical hash\r\n except Exception:\r\n cache, exists = self.cache_labels(cache_path, prefix), False # run cache ops\r\n\r\n # Display cache\r\n nf, nm, ne, nc, n = cache.pop('results') # found, missing, empty, corrupt, total\r\n if exists and LOCAL_RANK in {-1, 0}:\r\n d = f\"Scanning {cache_path}... {nf} images, {nm + ne} backgrounds, {nc} corrupt\"\r\n tqdm(None, desc=prefix + d, total=n, initial=n, bar_format=TQDM_BAR_FORMAT) # display cache results\r\n if cache['msgs']:\r\n LOGGER.info('\\n'.join(cache['msgs'])) # display warnings\r\n assert nf > 0 or not augment, f'{prefix}No labels found in {cache_path}, can not start training. {HELP_URL}'\r\n\r\n # Read cache\r\n [cache.pop(k) for k in ('hash', 'version', 'msgs')] # remove items\r\n labels, shapes, self.segments = zip(*cache.values())\r\n nl = len(np.concatenate(labels, 0)) # number of labels\r\n assert nl > 0 or not augment, f'{prefix}All labels empty in {cache_path}, can not start training. {HELP_URL}'\r\n self.labels = list(labels)\r\n self.shapes = np.array(shapes)\r\n self.im_files = list(cache.keys()) # update\r\n self.label_files = img2label_paths(cache.keys()) # update\r\n\r\n # Filter images\r\n if min_items:\r\n include = np.array([len(x) >= min_items for x in self.labels]).nonzero()[0].astype(int)\r\n LOGGER.info(f'{prefix}{n - len(include)}/{n} images filtered from dataset')\r\n self.im_files = [self.im_files[i] for i in include]\r\n self.label_files = [self.label_files[i] for i in include]\r\n self.labels = [self.labels[i] for i in include]\r\n self.segments = [self.segments[i] for i in include]\r\n self.shapes = self.shapes[include] # wh\r\n\r\n # Create indices\r\n n = len(self.shapes) # number of images\r\n bi = np.floor(np.arange(n) / batch_size).astype(int) # batch index\r\n nb = bi[-1] + 1 # number of batches\r\n self.batch = bi # batch index of image\r\n self.n = n\r\n self.indices = range(n)\r\n\r\n # Update labels\r\n include_class = [] # filter labels to include only these classes (optional)\r\n include_class_array = np.array(include_class).reshape(1, -1)\r\n for i, (label, segment) in enumerate(zip(self.labels, self.segments)):\r\n if include_class:\r\n j = (label[:, 0:1] == include_class_array).any(1)\r\n self.labels[i] = label[j]\r\n if segment:\r\n self.segments[i] = segment[j]\r\n if single_cls: # single-class training, merge all classes into 0\r\n self.labels[i][:, 0] = 0\r\n\r\n # Rectangular Training\r\n if self.rect:\r\n # Sort by aspect ratio\r\n s = self.shapes # wh\r\n ar = s[:, 1] / s[:, 0] # aspect ratio\r\n irect = ar.argsort()\r\n self.im_files = [self.im_files[i] for i in irect]\r\n self.label_files = [self.label_files[i] for i in irect]\r\n self.labels = [self.labels[i] for i in irect]\r\n self.segments = [self.segments[i] for i in irect]\r\n self.shapes = s[irect] # wh\r\n ar = ar[irect]\r\n\r\n # Set training image shapes\r\n shapes = [[1, 1]] * nb\r\n for i in range(nb):\r\n ari = ar[bi == i]\r\n mini, maxi = ari.min(), ari.max()\r\n if maxi < 1:\r\n shapes[i] = [maxi, 1]\r\n elif mini > 1:\r\n shapes[i] = [1, 1 / mini]\r\n\r\n self.batch_shapes = np.ceil(np.array(shapes) * img_size / stride + pad).astype(int) * stride\r\n\r\n # Cache images into RAM/disk for faster training\r\n if cache_images == 'ram' and not self.check_cache_ram(prefix=prefix):\r\n cache_images = False\r\n self.ims = [None] * n\r\n self.npy_files = [Path(f).with_suffix('.npy') for f in self.im_files]\r\n if cache_images:\r\n b, gb = 0, 1 << 30 # bytes of cached images, bytes per gigabytes\r\n self.im_hw0, self.im_hw = [None] * n, [None] * n\r\n fcn = self.cache_images_to_disk if cache_images == 'disk' else self.load_image\r\n results = ThreadPool(NUM_THREADS).imap(fcn, range(n))\r\n pbar = tqdm(enumerate(results), total=n, bar_format=TQDM_BAR_FORMAT, disable=LOCAL_RANK > 0)\r\n for i, x in pbar:\r\n if cache_images == 'disk':\r\n b += self.npy_files[i].stat().st_size\r\n else: # 'ram'\r\n self.ims[i], self.im_hw0[i], self.im_hw[i] = x # im, hw_orig, hw_resized = load_image(self, i)\r\n b += self.ims[i].nbytes\r\n pbar.desc = f'{prefix}Caching images ({b / gb:.1f}GB {cache_images})'\r\n pbar.close()\r\n\r\n def check_cache_ram(self, safety_margin=0.1, prefix=''):\r\n # Check image caching requirements vs available memory\r\n b, gb = 0, 1 << 30 # bytes of cached images, bytes per gigabytes\r\n n = min(self.n, 30) # extrapolate from 30 random images\r\n for _ in range(n):\r\n im = cv2.imread(random.choice(self.im_files)) # sample image\r\n ratio = self.img_size / max(im.shape[0], im.shape[1]) # max(h, w) # ratio\r\n b += im.nbytes * ratio ** 2\r\n mem_required = b * self.n / n # GB required to cache dataset into RAM\r\n mem = psutil.virtual_memory()\r\n cache = mem_required * (1 + safety_margin) < mem.available # to cache or not to cache, that is the question\r\n if not cache:\r\n LOGGER.info(f\"{prefix}{mem_required / gb:.1f}GB RAM required, \"\r\n f\"{mem.available / gb:.1f}/{mem.total / gb:.1f}GB available, \"\r\n f\"{'caching images β
' if cache else 'not caching images β οΈ'}\")\r\n return cache\r\n\r\n def cache_labels(self, path=Path('./labels.cache'), prefix=''):\r\n # Cache dataset labels, check images and read shapes\r\n x = {} # dict\r\n nm, nf, ne, nc, msgs = 0, 0, 0, 0, [] # number missing, found, empty, corrupt, messages\r\n desc = f\"{prefix}Scanning {path.parent / path.stem}...\"\r\n with Pool(NUM_THREADS) as pool:\r\n pbar = tqdm(pool.imap(verify_image_label, zip(self.im_files, self.label_files, repeat(prefix))),\r\n desc=desc,\r\n total=len(self.im_files),\r\n bar_format=TQDM_BAR_FORMAT)\r\n for im_file, lb, shape, segments, nm_f, nf_f, ne_f, nc_f, msg in pbar:\r\n nm += nm_f\r\n nf += nf_f\r\n ne += ne_f\r\n nc += nc_f\r\n if im_file:\r\n x[im_file] = [lb, shape, segments]\r\n if msg:\r\n msgs.append(msg)\r\n pbar.desc = f\"{desc} {nf} images, {nm + ne} backgrounds, {nc} corrupt\"\r\n\r\n pbar.close()\r\n if msgs:\r\n LOGGER.info('\\n'.join(msgs))\r\n if nf == 0:\r\n LOGGER.warning(f'{prefix}WARNING β οΈ No labels found in {path}. {HELP_URL}')\r\n x['hash'] = get_hash(self.label_files + self.im_files)\r\n x['results'] = nf, nm, ne, nc, len(self.im_files)\r\n x['msgs'] = msgs # warnings\r\n x['version'] = self.cache_version # cache version\r\n try:\r\n np.save(path, x) # save cache for next time\r\n path.with_suffix('.cache.npy').rename(path) # remove .npy suffix\r\n LOGGER.info(f'{prefix}New cache created: {path}')\r\n except Exception as e:\r\n LOGGER.warning(f'{prefix}WARNING β οΈ Cache directory {path.parent} is not writeable: {e}') # not writeable\r\n return x\r\n\r\n def __len__(self):\r\n return len(self.im_files)\r\n\r\n # def __iter__(self):\r\n # self.count = -1\r\n # print('ran dataset iter')\r\n # #self.shuffled_vector = np.random.permutation(self.nF) if self.augment else np.arange(self.nF)\r\n # return self\r\n\r\n def __getitem__(self, index):\r\n index = self.indices[index] # linear, shuffled, or image_weights\r\n\r\n hyp = self.hyp\r\n mosaic = self.mosaic and random.random() < hyp['mosaic']\r\n if mosaic:\r\n # Load mosaic\r\n img, labels = self.load_mosaic(index)\r\n shapes = None\r\n\r\n # MixUp augmentation\r\n if random.random() < hyp['mixup']:\r\n img, labels = mixup(img, labels, *self.load_mosaic(random.randint(0, self.n - 1)))\r\n\r\n else:\r\n # Load image\r\n img, (h0, w0), (h, w) = self.load_image(index)\r\n\r\n # Letterbox\r\n shape = self.batch_shapes[self.batch[index]] if self.rect else self.img_size # final letterboxed shape\r\n img, ratio, pad = letterbox(img, shape, auto=False, scaleup=self.augment)\r\n shapes = (h0, w0), ((h / h0, w / w0), pad) # for COCO mAP rescaling\r\n\r\n labels = self.labels[index].copy()\r\n if labels.size: # normalized xywh to pixel xyxy format\r\n labels[:, 1:] = xywhn2xyxy(labels[:, 1:], ratio[0] * w, ratio[1] * h, padw=pad[0], padh=pad[1])\r\n\r\n if self.augment:\r\n img, labels = random_perspective(img,\r\n labels,\r\n degrees=hyp['degrees'],\r\n translate=hyp['translate'],\r\n scale=hyp['scale'],\r\n shear=hyp['shear'],\r\n perspective=hyp['perspective'])\r\n\r\n nl = len(labels) # number of labels\r\n if nl:\r\n labels[:, 1:5] = xyxy2xywhn(labels[:, 1:5], w=img.shape[1], h=img.shape[0], clip=True, eps=1E-3)\r\n\r\n if self.augment:\r\n # Albumentations\r\n img, labels = self.albumentations(img, labels)\r\n nl = len(labels) # update after albumentations\r\n\r\n # HSV color-space\r\n augment_hsv(img, hgain=hyp['hsv_h'], sgain=hyp['hsv_s'], vgain=hyp['hsv_v'])\r\n\r\n # Flip up-down\r\n if random.random() < hyp['flipud']:\r\n img = np.flipud(img)\r\n if nl:\r\n labels[:, 2] = 1 - labels[:, 2]\r\n\r\n # Flip left-right\r\n if random.random() < hyp['fliplr']:\r\n img = np.fliplr(img)\r\n if nl:\r\n labels[:, 1] = 1 - labels[:, 1]\r\n\r\n # Cutouts\r\n # labels = cutout(img, labels, p=0.5)\r\n # nl = len(labels) # update after cutout\r\n\r\n labels_out = torch.zeros((nl, 6))\r\n if nl:\r\n labels_out[:, 1:] = torch.from_numpy(labels)\r\n\r\n # Convert\r\n img = img.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB\r\n img = np.ascontiguousarray(img)\r\n\r\n return torch.from_numpy(img), labels_out, self.im_files[index], shapes\r\n\r\n def load_image(self, i):\r\n # Loads 1 image from dataset index 'i', returns (im, original hw, resized hw)\r\n im, f, fn = self.ims[i], self.im_files[i], self.npy_files[i],\r\n if im is None: # not cached in RAM\r\n if fn.exists(): # load npy\r\n im = np.load(fn)\r\n else: # read image\r\n im = cv2.imread(f) # BGR\r\n assert im is not None, f'Image Not Found {f}'\r\n h0, w0 = im.shape[:2] # orig hw\r\n r = self.img_size / max(h0, w0) # ratio\r\n if r != 1: # if sizes are not equal\r\n interp = cv2.INTER_LINEAR if (self.augment or r > 1) else cv2.INTER_AREA\r\n im = cv2.resize(im, (math.ceil(w0 * r), math.ceil(h0 * r)), interpolation=interp)\r\n return im, (h0, w0), im.shape[:2] # im, hw_original, hw_resized\r\n return self.ims[i], self.im_hw0[i], self.im_hw[i] # im, hw_original, hw_resized\r\n\r\n def cache_images_to_disk(self, i):\r\n # Saves an image as an *.npy file for faster loading\r\n f = self.npy_files[i]\r\n if not f.exists():\r\n np.save(f.as_posix(), cv2.imread(self.im_files[i]))\r\n\r\n def load_mosaic(self, index):\r\n # YOLOv5 4-mosaic loader. Loads 1 image + 3 random images into a 4-image mosaic\r\n labels4, segments4 = [], []\r\n s = self.img_size\r\n yc, xc = (int(random.uniform(-x, 2 * s + x)) for x in self.mosaic_border) # mosaic center x, y\r\n indices = [index] + random.choices(self.indices, k=3) # 3 additional image indices\r\n random.shuffle(indices)\r\n for i, index in enumerate(indices):\r\n # Load image\r\n img, _, (h, w) = self.load_image(index)\r\n\r\n # place img in img4\r\n if i == 0: # top left\r\n img4 = np.full((s * 2, s * 2, img.shape[2]), 114, dtype=np.uint8) # base image with 4 tiles\r\n x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc # xmin, ymin, xmax, ymax (large image)\r\n x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h # xmin, ymin, xmax, ymax (small image)\r\n elif i == 1: # top right\r\n x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc\r\n x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h\r\n elif i == 2: # bottom left\r\n x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)\r\n x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, w, min(y2a - y1a, h)\r\n elif i == 3: # bottom right\r\n x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)\r\n x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)\r\n\r\n img4[y1a:y2a, x1a:x2a] = img[y1b:y2b, x1b:x2b] # img4[ymin:ymax, xmin:xmax]\r\n padw = x1a - x1b\r\n padh = y1a - y1b\r\n\r\n # Labels\r\n labels, segments = self.labels[index].copy(), self.segments[index].copy()\r\n if labels.size:\r\n labels[:, 1:] = xywhn2xyxy(labels[:, 1:], w, h, padw, padh) # normalized xywh to pixel xyxy format\r\n segments = [xyn2xy(x, w, h, padw, padh) for x in segments]\r\n labels4.append(labels)\r\n segments4.extend(segments)\r\n\r\n # Concat/clip labels\r\n labels4 = np.concatenate(labels4, 0)\r\n for x in (labels4[:, 1:], *segments4):\r\n np.clip(x, 0, 2 * s, out=x) # clip when using random_perspective()\r\n # img4, labels4 = replicate(img4, labels4) # replicate\r\n\r\n # Augment\r\n img4, labels4, segments4 = copy_paste(img4, labels4, segments4, p=self.hyp['copy_paste'])\r\n img4, labels4 = random_perspective(img4,\r\n labels4,\r\n segments4,\r\n degrees=self.hyp['degrees'],\r\n translate=self.hyp['translate'],\r\n scale=self.hyp['scale'],\r\n shear=self.hyp['shear'],\r\n perspective=self.hyp['perspective'],\r\n border=self.mosaic_border) # border to remove\r\n\r\n return img4, labels4\r\n\r\n def load_mosaic9(self, index):\r\n # YOLOv5 9-mosaic loader. Loads 1 image + 8 random images into a 9-image mosaic\r\n labels9, segments9 = [], []\r\n s = self.img_size\r\n indices = [index] + random.choices(self.indices, k=8) # 8 additional image indices\r\n random.shuffle(indices)\r\n hp, wp = -1, -1 # height, width previous\r\n for i, index in enumerate(indices):\r\n # Load image\r\n img, _, (h, w) = self.load_image(index)\r\n\r\n # place img in img9\r\n if i == 0: # center\r\n img9 = np.full((s * 3, s * 3, img.shape[2]), 114, dtype=np.uint8) # base image with 4 tiles\r\n h0, w0 = h, w\r\n c = s, s, s + w, s + h # xmin, ymin, xmax, ymax (base) coordinates\r\n elif i == 1: # top\r\n c = s, s - h, s + w, s\r\n elif i == 2: # top right\r\n c = s + wp, s - h, s + wp + w, s\r\n elif i == 3: # right\r\n c = s + w0, s, s + w0 + w, s + h\r\n elif i == 4: # bottom right\r\n c = s + w0, s + hp, s + w0 + w, s + hp + h\r\n elif i == 5: # bottom\r\n c = s + w0 - w, s + h0, s + w0, s + h0 + h\r\n elif i == 6: # bottom left\r\n c = s + w0 - wp - w, s + h0, s + w0 - wp, s + h0 + h\r\n elif i == 7: # left\r\n c = s - w, s + h0 - h, s, s + h0\r\n elif i == 8: # top left\r\n c = s - w, s + h0 - hp - h, s, s + h0 - hp\r\n\r\n padx, pady = c[:2]\r\n x1, y1, x2, y2 = (max(x, 0) for x in c) # allocate coords\r\n\r\n # Labels\r\n labels, segments = self.labels[index].copy(), self.segments[index].copy()\r\n if labels.size:\r\n labels[:, 1:] = xywhn2xyxy(labels[:, 1:], w, h, padx, pady) # normalized xywh to pixel xyxy format\r\n segments = [xyn2xy(x, w, h, padx, pady) for x in segments]\r\n labels9.append(labels)\r\n segments9.extend(segments)\r\n\r\n # Image\r\n img9[y1:y2, x1:x2] = img[y1 - pady:, x1 - padx:] # img9[ymin:ymax, xmin:xmax]\r\n hp, wp = h, w # height, width previous\r\n\r\n # Offset\r\n yc, xc = (int(random.uniform(0, s)) for _ in self.mosaic_border) # mosaic center x, y\r\n img9 = img9[yc:yc + 2 * s, xc:xc + 2 * s]\r\n\r\n # Concat/clip labels\r\n labels9 = np.concatenate(labels9, 0)\r\n labels9[:, [1, 3]] -= xc\r\n labels9[:, [2, 4]] -= yc\r\n c = np.array([xc, yc]) # centers\r\n segments9 = [x - c for x in segments9]\r\n\r\n for x in (labels9[:, 1:], *segments9):\r\n np.clip(x, 0, 2 * s, out=x) # clip when using random_perspective()\r\n # img9, labels9 = replicate(img9, labels9) # replicate\r\n\r\n # Augment\r\n img9, labels9, segments9 = copy_paste(img9, labels9, segments9, p=self.hyp['copy_paste'])\r\n img9, labels9 = random_perspective(img9,\r\n labels9,\r\n segments9,\r\n degrees=self.hyp['degrees'],\r\n translate=self.hyp['translate'],\r\n scale=self.hyp['scale'],\r\n shear=self.hyp['shear'],\r\n perspective=self.hyp['perspective'],\r\n border=self.mosaic_border) # border to remove\r\n\r\n return img9, labels9\r\n\r\n @staticmethod\r\n def collate_fn(batch):\r\n # YOLOv8 collate function, outputs dict\r\n im, label, path, shapes = zip(*batch) # transposed\r\n for i, lb in enumerate(label):\r\n lb[:, 0] = i # add target image index for build_targets()\r\n batch_idx, cls, bboxes = torch.cat(label, 0).split((1, 1, 4), dim=1)\r\n return {\r\n 'ori_shape': tuple((x[0] if x else None) for x in shapes),\r\n 'ratio_pad': tuple((x[1] if x else None) for x in shapes),\r\n 'im_file': path,\r\n 'img': torch.stack(im, 0),\r\n 'cls': cls,\r\n 'bboxes': bboxes,\r\n 'batch_idx': batch_idx.view(-1)}\r\n\r\n @staticmethod\r\n def collate_fn_old(batch):\r\n # YOLOv5 original collate function\r\n im, label, path, shapes = zip(*batch) # transposed\r\n for i, lb in enumerate(label):\r\n lb[:, 0] = i # add target image index for build_targets()\r\n return torch.stack(im, 0), torch.cat(label, 0), path, shapes\r\n\r\n\r\n# Ancillary functions --------------------------------------------------------------------------------------------------\r\ndef flatten_recursive(path=DATASETS_DIR / 'coco128'):\r\n # Flatten a recursive directory by bringing all files to top level\r\n new_path = Path(f'{str(path)}_flat')\r\n if os.path.exists(new_path):\r\n shutil.rmtree(new_path) # delete output folder\r\n os.makedirs(new_path) # make new output folder\r\n for file in tqdm(glob.glob(f'{str(Path(path))}/**/*.*', recursive=True)):\r\n shutil.copyfile(file, new_path / Path(file).name)\r\n\r\n\r\ndef extract_boxes(path=DATASETS_DIR / 'coco128'): # from utils.dataloaders import *; extract_boxes()\r\n # Convert detection dataset into classification dataset, with one directory per class\r\n path = Path(path) # images dir\r\n shutil.rmtree(path / 'classification') if (path / 'classification').is_dir() else None # remove existing\r\n files = list(path.rglob('*.*'))\r\n n = len(files) # number of files\r\n for im_file in tqdm(files, total=n):\r\n if im_file.suffix[1:] in IMG_FORMATS:\r\n # image\r\n im = cv2.imread(str(im_file))[..., ::-1] # BGR to RGB\r\n h, w = im.shape[:2]\r\n\r\n # labels\r\n lb_file = Path(img2label_paths([str(im_file)])[0])\r\n if Path(lb_file).exists():\r\n with open(lb_file) as f:\r\n lb = np.array([x.split() for x in f.read().strip().splitlines()], dtype=np.float32) # labels\r\n\r\n for j, x in enumerate(lb):\r\n c = int(x[0]) # class\r\n f = (path / 'classifier') / f'{c}' / f'{path.stem}_{im_file.stem}_{j}.jpg' # new filename\r\n if not f.parent.is_dir():\r\n f.parent.mkdir(parents=True)\r\n\r\n b = x[1:] * [w, h, w, h] # box\r\n # b[2:] = b[2:].max() # rectangle to square\r\n b[2:] = b[2:] * 1.2 + 3 # pad\r\n b = xywh2xyxy(b.reshape(-1, 4)).ravel().astype(int)\r\n\r\n b[[0, 2]] = np.clip(b[[0, 2]], 0, w) # clip boxes outside of image\r\n b[[1, 3]] = np.clip(b[[1, 3]], 0, h)\r\n assert cv2.imwrite(str(f), im[b[1]:b[3], b[0]:b[2]]), f'box failure in {f}'\r\n\r\n\r\ndef autosplit(path=DATASETS_DIR / 'coco128/images', weights=(0.9, 0.1, 0.0), annotated_only=False):\r\n \"\"\" Autosplit a dataset into train/val/test splits and save path/autosplit_*.txt files\r\n Usage: from utils.dataloaders import *; autosplit()\r\n Arguments\r\n path: Path to images directory\r\n weights: Train, val, test weights (list, tuple)\r\n annotated_only: Only use images with an annotated txt file\r\n \"\"\"\r\n path = Path(path) # images dir\r\n files = sorted(x for x in path.rglob('*.*') if x.suffix[1:].lower() in IMG_FORMATS) # image files only\r\n n = len(files) # number of files\r\n random.seed(0) # for reproducibility\r\n indices = random.choices([0, 1, 2], weights=weights, k=n) # assign each image to a split\r\n\r\n txt = ['autosplit_train.txt', 'autosplit_val.txt', 'autosplit_test.txt'] # 3 txt files\r\n for x in txt:\r\n if (path.parent / x).exists():\r\n (path.parent / x).unlink() # remove existing\r\n\r\n print(f'Autosplitting images from {path}' + ', using *.txt labeled images only' * annotated_only)\r\n for i, img in tqdm(zip(indices, files), total=n):\r\n if not annotated_only or Path(img2label_paths([str(img)])[0]).exists(): # check label\r\n with open(path.parent / txt[i], 'a') as f:\r\n f.write(f'./{img.relative_to(path.parent).as_posix()}' + '\\n') # add image to txt file\r\n\r\n\r\ndef verify_image_label(args):\r\n # Verify one image-label pair\r\n im_file, lb_file, prefix = args\r\n nm, nf, ne, nc, msg, segments = 0, 0, 0, 0, '', [] # number (missing, found, empty, corrupt), message, segments\r\n try:\r\n # verify images\r\n im = Image.open(im_file)\r\n im.verify() # PIL verify\r\n shape = exif_size(im) # image size\r\n assert (shape[0] > 9) & (shape[1] > 9), f'image size {shape} <10 pixels'\r\n assert im.format.lower() in IMG_FORMATS, f'invalid image format {im.format}'\r\n if im.format.lower() in ('jpg', 'jpeg'):\r\n with open(im_file, 'rb') as f:\r\n f.seek(-2, 2)\r\n if f.read() != b'\\xff\\xd9': # corrupt JPEG\r\n ImageOps.exif_transpose(Image.open(im_file)).save(im_file, 'JPEG', subsampling=0, quality=100)\r\n msg = f'{prefix}WARNING β οΈ {im_file}: corrupt JPEG restored and saved'\r\n\r\n # verify labels\r\n if os.path.isfile(lb_file):\r\n nf = 1 # label found\r\n with open(lb_file) as f:\r\n lb = [x.split() for x in f.read().strip().splitlines() if len(x)]\r\n if any(len(x) > 6 for x in lb): # is segment\r\n classes = np.array([x[0] for x in lb], dtype=np.float32)\r\n segments = [np.array(x[1:], dtype=np.float32).reshape(-1, 2) for x in lb] # (cls, xy1...)\r\n lb = np.concatenate((classes.reshape(-1, 1), segments2boxes(segments)), 1) # (cls, xywh)\r\n lb = np.array(lb, dtype=np.float32)\r\n nl = len(lb)\r\n if nl:\r\n assert lb.shape[1] == 5, f'labels require 5 columns, {lb.shape[1]} columns detected'\r\n assert (lb >= 0).all(), f'negative label values {lb[lb < 0]}'\r\n assert (lb[:, 1:] <= 1).all(), f'non-normalized or out of bounds coordinates {lb[:, 1:][lb[:, 1:] > 1]}'\r\n _, i = np.unique(lb, axis=0, return_index=True)\r\n if len(i) < nl: # duplicate row check\r\n lb = lb[i] # remove duplicates\r\n if segments:\r\n segments = [segments[x] for x in i]\r\n msg = f'{prefix}WARNING β οΈ {im_file}: {nl - len(i)} duplicate labels removed'\r\n else:\r\n ne = 1 # label empty\r\n lb = np.zeros((0, 5), dtype=np.float32)\r\n else:\r\n nm = 1 # label missing\r\n lb = np.zeros((0, 5), dtype=np.float32)\r\n return im_file, lb, shape, segments, nm, nf, ne, nc, msg\r\n except Exception as e:\r\n nc = 1\r\n msg = f'{prefix}WARNING β οΈ {im_file}: ignoring corrupt image/label: {e}'\r\n return [None, None, None, None, nm, nf, ne, nc, msg]\r\n\r\n\r\nclass HUBDatasetStats():\r\n \"\"\" Class for generating HUB dataset JSON and `-hub` dataset directory\r\n\r\n Arguments\r\n path: Path to data.yaml or data.zip (with data.yaml inside data.zip)\r\n autodownload: Attempt to download dataset if not found locally\r\n\r\n Usage\r\n from utils.dataloaders import HUBDatasetStats\r\n stats = HUBDatasetStats('coco128.yaml', autodownload=True) # usage 1\r\n stats = HUBDatasetStats('path/to/coco128.zip') # usage 2\r\n stats.get_json(save=False)\r\n stats.process_images()\r\n \"\"\"\r\n\r\n def __init__(self, path='coco128.yaml', autodownload=False):\r\n # Initialize class\r\n zipped, data_dir, yaml_path = self._unzip(Path(path))\r\n try:\r\n with open(check_yaml(yaml_path), errors='ignore') as f:\r\n data = yaml.safe_load(f) # data dict\r\n if zipped:\r\n data['path'] = data_dir\r\n except Exception as e:\r\n raise Exception(\"error/HUB/dataset_stats/yaml_load\") from e\r\n\r\n check_dataset(data, autodownload) # download dataset if missing\r\n self.hub_dir = Path(data['path'] + '-hub')\r\n self.im_dir = self.hub_dir / 'images'\r\n self.im_dir.mkdir(parents=True, exist_ok=True) # makes /images\r\n self.stats = {'nc': data['nc'], 'names': list(data['names'].values())} # statistics dictionary\r\n self.data = data\r\n\r\n @staticmethod\r\n def _find_yaml(dir):\r\n # Return data.yaml file\r\n files = list(dir.glob('*.yaml')) or list(dir.rglob('*.yaml')) # try root level first and then recursive\r\n assert files, f'No *.yaml file found in {dir}'\r\n if len(files) > 1:\r\n files = [f for f in files if f.stem == dir.stem] # prefer *.yaml files that match dir name\r\n assert files, f'Multiple *.yaml files found in {dir}, only 1 *.yaml file allowed'\r\n assert len(files) == 1, f'Multiple *.yaml files found: {files}, only 1 *.yaml file allowed in {dir}'\r\n return files[0]\r\n\r\n def _unzip(self, path):\r\n # Unzip data.zip\r\n if not str(path).endswith('.zip'): # path is data.yaml\r\n return False, None, path\r\n assert Path(path).is_file(), f'Error unzipping {path}, file not found'\r\n unzip_file(path, path=path.parent)\r\n dir = path.with_suffix('') # dataset directory == zip name\r\n assert dir.is_dir(), f'Error unzipping {path}, {dir} not found. path/to/abc.zip MUST unzip to path/to/abc/'\r\n return True, str(dir), self._find_yaml(dir) # zipped, data_dir, yaml_path\r\n\r\n def _hub_ops(self, f, max_dim=1920):\r\n # HUB ops for 1 image 'f': resize and save at reduced quality in /dataset-hub for web/app viewing\r\n f_new = self.im_dir / Path(f).name # dataset-hub image filename\r\n try: # use PIL\r\n im = Image.open(f)\r\n r = max_dim / max(im.height, im.width) # ratio\r\n if r < 1.0: # image too large\r\n im = im.resize((int(im.width * r), int(im.height * r)))\r\n im.save(f_new, 'JPEG', quality=50, optimize=True) # save\r\n except Exception as e: # use OpenCV\r\n LOGGER.info(f'WARNING β οΈ HUB ops PIL failure {f}: {e}')\r\n im = cv2.imread(f)\r\n im_height, im_width = im.shape[:2]\r\n r = max_dim / max(im_height, im_width) # ratio\r\n if r < 1.0: # image too large\r\n im = cv2.resize(im, (int(im_width * r), int(im_height * r)), interpolation=cv2.INTER_AREA)\r\n cv2.imwrite(str(f_new), im)\r\n\r\n def get_json(self, save=False, verbose=False):\r\n # Return dataset JSON for Ultralytics HUB\r\n def _round(labels):\r\n # Update labels to integer class and 6 decimal place floats\r\n return [[int(c), *(round(x, 4) for x in points)] for c, *points in labels]\r\n\r\n for split in 'train', 'val', 'test':\r\n if self.data.get(split) is None:\r\n self.stats[split] = None # i.e. no test set\r\n continue\r\n dataset = LoadImagesAndLabels(self.data[split]) # load dataset\r\n x = np.array([\r\n np.bincount(label[:, 0].astype(int), minlength=self.data['nc'])\r\n for label in tqdm(dataset.labels, total=dataset.n, desc='Statistics')]) # shape(128x80)\r\n self.stats[split] = {\r\n 'instance_stats': {\r\n 'total': int(x.sum()),\r\n 'per_class': x.sum(0).tolist()},\r\n 'image_stats': {\r\n 'total': dataset.n,\r\n 'unlabelled': int(np.all(x == 0, 1).sum()),\r\n 'per_class': (x > 0).sum(0).tolist()},\r\n 'labels': [{\r\n str(Path(k).name): _round(v.tolist())} for k, v in zip(dataset.im_files, dataset.labels)]}\r\n\r\n # Save, print and return\r\n if save:\r\n stats_path = self.hub_dir / 'stats.json'\r\n print(f'Saving {stats_path.resolve()}...')\r\n with open(stats_path, 'w') as f:\r\n json.dump(self.stats, f) # save stats.json\r\n if verbose:\r\n print(json.dumps(self.stats, indent=2, sort_keys=False))\r\n return self.stats\r\n\r\n def process_images(self):\r\n # Compress images for Ultralytics HUB\r\n for split in 'train', 'val', 'test':\r\n if self.data.get(split) is None:\r\n continue\r\n dataset = LoadImagesAndLabels(self.data[split]) # load dataset\r\n desc = f'{split} images'\r\n for _ in tqdm(ThreadPool(NUM_THREADS).imap(self._hub_ops, dataset.im_files), total=dataset.n, desc=desc):\r\n pass\r\n print(f'Done. All images saved to {self.im_dir}')\r\n return self.im_dir\r\n\r\n\r\n# Classification dataloaders -------------------------------------------------------------------------------------------\r\nclass ClassificationDataset(torchvision.datasets.ImageFolder):\r\n \"\"\"\r\n YOLOv5 Classification Dataset.\r\n Arguments\r\n root: Dataset path\r\n transform: torchvision transforms, used by default\r\n album_transform: Albumentations transforms, used if installed\r\n \"\"\"\r\n\r\n def __init__(self, root, augment, imgsz, cache=False):\r\n super().__init__(root=root)\r\n self.torch_transforms = classify_transforms(imgsz)\r\n self.album_transforms = classify_albumentations(augment, imgsz) if augment else None\r\n self.cache_ram = cache is True or cache == 'ram'\r\n self.cache_disk = cache == 'disk'\r\n self.samples = [list(x) + [Path(x[0]).with_suffix('.npy'), None] for x in self.samples] # file, index, npy, im\r\n\r\n def __getitem__(self, i):\r\n f, j, fn, im = self.samples[i] # filename, index, filename.with_suffix('.npy'), image\r\n if self.cache_ram and im is None:\r\n im = self.samples[i][3] = cv2.imread(f)\r\n elif self.cache_disk:\r\n if not fn.exists(): # load npy\r\n np.save(fn.as_posix(), cv2.imread(f))\r\n im = np.load(fn)\r\n else: # read image\r\n im = cv2.imread(f) # BGR\r\n if self.album_transforms:\r\n sample = self.album_transforms(image=cv2.cvtColor(im, cv2.COLOR_BGR2RGB))[\"image\"]\r\n else:\r\n sample = self.torch_transforms(im)\r\n return sample, j\r\n\r\n\r\ndef create_classification_dataloader(path,\r\n imgsz=224,\r\n batch_size=16,\r\n augment=True,\r\n cache=False,\r\n rank=-1,\r\n workers=8,\r\n shuffle=True):\r\n # Returns Dataloader object to be used with YOLOv5 Classifier\r\n with torch_distributed_zero_first(rank): # init dataset *.cache only once if DDP\r\n dataset = ClassificationDataset(root=path, imgsz=imgsz, augment=augment, cache=cache)\r\n batch_size = min(batch_size, len(dataset))\r\n nd = torch.cuda.device_count()\r\n nw = min([os.cpu_count() // max(nd, 1), batch_size if batch_size > 1 else 0, workers])\r\n sampler = None if rank == -1 else distributed.DistributedSampler(dataset, shuffle=shuffle)\r\n generator = torch.Generator()\r\n generator.manual_seed(6148914691236517205 + RANK)\r\n return InfiniteDataLoader(dataset,\r\n batch_size=batch_size,\r\n shuffle=shuffle and sampler is None,\r\n num_workers=nw,\r\n sampler=sampler,\r\n pin_memory=PIN_MEMORY,\r\n worker_init_fn=seed_worker,\r\n generator=generator) # or DataLoader(persistent_workers=True)\r\n"
},
{
"path": "yolo/data/dataset.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom itertools import repeat\r\nfrom multiprocessing.pool import Pool\r\nfrom pathlib import Path\r\n\r\nimport torchvision\r\nfrom tqdm import tqdm\r\n\r\nfrom ..utils import NUM_THREADS, TQDM_BAR_FORMAT\r\nfrom .augment import *\r\nfrom .base import BaseDataset\r\nfrom .utils import HELP_URL, LOCAL_RANK, get_hash, img2label_paths, verify_image_label\r\n\r\n\r\nclass YOLODataset(BaseDataset):\r\n cache_version = 1.0 # dataset labels *.cache version, >= 1.0 for YOLOv8\r\n rand_interp_methods = [cv2.INTER_NEAREST, cv2.INTER_LINEAR, cv2.INTER_CUBIC, cv2.INTER_AREA, cv2.INTER_LANCZOS4]\r\n \"\"\"YOLO Dataset.\r\n Args:\r\n img_path (str): image path.\r\n prefix (str): prefix.\r\n \"\"\"\r\n\r\n def __init__(\r\n self,\r\n img_path,\r\n imgsz=640,\r\n label_path=None,\r\n cache=False,\r\n augment=True,\r\n hyp=None,\r\n prefix=\"\",\r\n rect=False,\r\n batch_size=None,\r\n stride=32,\r\n pad=0.0,\r\n single_cls=False,\r\n use_segments=False,\r\n use_keypoints=False,\r\n ):\r\n self.use_segments = use_segments\r\n self.use_keypoints = use_keypoints\r\n assert not (self.use_segments and self.use_keypoints), \"Can not use both segments and keypoints.\"\r\n super().__init__(img_path, imgsz, label_path, cache, augment, hyp, prefix, rect, batch_size, stride, pad,\r\n single_cls)\r\n\r\n def cache_labels(self, path=Path(\"./labels.cache\")):\r\n # Cache dataset labels, check images and read shapes\r\n x = {\"labels\": []}\r\n nm, nf, ne, nc, msgs = 0, 0, 0, 0, [] # number missing, found, empty, corrupt, messages\r\n desc = f\"{self.prefix}Scanning {path.parent / path.stem}...\"\r\n with Pool(NUM_THREADS) as pool:\r\n pbar = tqdm(\r\n pool.imap(verify_image_label,\r\n zip(self.im_files, self.label_files, repeat(self.prefix), repeat(self.use_keypoints))),\r\n desc=desc,\r\n total=len(self.im_files),\r\n bar_format=TQDM_BAR_FORMAT,\r\n )\r\n for im_file, lb, shape, segments, keypoint, nm_f, nf_f, ne_f, nc_f, msg in pbar:\r\n nm += nm_f\r\n nf += nf_f\r\n ne += ne_f\r\n nc += nc_f\r\n if im_file:\r\n x[\"labels\"].append(\r\n dict(\r\n im_file=im_file,\r\n shape=shape,\r\n cls=lb[:, 0:1], # n, 1\r\n bboxes=lb[:, 1:], # n, 4\r\n segments=segments,\r\n keypoints=keypoint,\r\n normalized=True,\r\n bbox_format=\"xywh\",\r\n ))\r\n if msg:\r\n msgs.append(msg)\r\n pbar.desc = f\"{desc} {nf} images, {nm + ne} backgrounds, {nc} corrupt\"\r\n\r\n pbar.close()\r\n if msgs:\r\n LOGGER.info(\"\\n\".join(msgs))\r\n if nf == 0:\r\n LOGGER.warning(f\"{self.prefix}WARNING β οΈ No labels found in {path}. {HELP_URL}\")\r\n x[\"hash\"] = get_hash(self.label_files + self.im_files)\r\n x[\"results\"] = nf, nm, ne, nc, len(self.im_files)\r\n x[\"msgs\"] = msgs # warnings\r\n x[\"version\"] = self.cache_version # cache version\r\n try:\r\n np.save(path, x) # save cache for next time\r\n path.with_suffix(\".cache.npy\").rename(path) # remove .npy suffix\r\n LOGGER.info(f\"{self.prefix}New cache created: {path}\")\r\n except Exception as e:\r\n LOGGER.warning(\r\n f\"{self.prefix}WARNING β οΈ Cache directory {path.parent} is not writeable: {e}\") # not writeable\r\n return x\r\n\r\n def get_labels(self):\r\n self.label_files = img2label_paths(self.im_files)\r\n cache_path = Path(self.label_files[0]).parent.with_suffix(\".cache\")\r\n try:\r\n cache, exists = np.load(str(cache_path), allow_pickle=True).item(), True # load dict\r\n assert cache[\"version\"] == self.cache_version # matches current version\r\n assert cache[\"hash\"] == get_hash(self.label_files + self.im_files) # identical hash\r\n except Exception:\r\n cache, exists = self.cache_labels(cache_path), False # run cache ops\r\n\r\n # Display cache\r\n nf, nm, ne, nc, n = cache.pop(\"results\") # found, missing, empty, corrupt, total\r\n if exists and LOCAL_RANK in {-1, 0}:\r\n d = f\"Scanning {cache_path}... {nf} images, {nm + ne} backgrounds, {nc} corrupt\"\r\n tqdm(None, desc=self.prefix + d, total=n, initial=n, bar_format=TQDM_BAR_FORMAT) # display cache results\r\n if cache[\"msgs\"]:\r\n LOGGER.info(\"\\n\".join(cache[\"msgs\"])) # display warnings\r\n assert nf > 0, f\"{self.prefix}No labels found in {cache_path}, can not start training. {HELP_URL}\"\r\n\r\n # Read cache\r\n [cache.pop(k) for k in (\"hash\", \"version\", \"msgs\")] # remove items\r\n labels = cache[\"labels\"]\r\n nl = len(np.concatenate([label[\"cls\"] for label in labels], 0)) # number of labels\r\n assert nl > 0, f\"{self.prefix}All labels empty in {cache_path}, can not start training. {HELP_URL}\"\r\n return labels\r\n\r\n # TODO: use hyp config to set all these augmentations\r\n def build_transforms(self, hyp=None):\r\n if self.augment:\r\n mosaic = self.augment and not self.rect\r\n transforms = mosaic_transforms(self, self.imgsz, hyp) if mosaic else affine_transforms(self.imgsz, hyp)\r\n else:\r\n transforms = Compose([LetterBox(new_shape=(self.imgsz, self.imgsz), scaleup=False)])\r\n transforms.append(\r\n Format(bbox_format=\"xywh\",\r\n normalize=True,\r\n return_mask=self.use_segments,\r\n return_keypoint=self.use_keypoints,\r\n batch_idx=True))\r\n return transforms\r\n\r\n def close_mosaic(self, hyp):\r\n self.transforms = affine_transforms(self.imgsz, hyp)\r\n self.transforms.append(\r\n Format(bbox_format=\"xywh\",\r\n normalize=True,\r\n return_mask=self.use_segments,\r\n return_keypoint=self.use_keypoints,\r\n batch_idx=True))\r\n\r\n def update_labels_info(self, label):\r\n \"\"\"custom your label format here\"\"\"\r\n # NOTE: cls is not with bboxes now, classification and semantic segmentation need an independent cls label\r\n # we can make it also support classification and semantic segmentation by add or remove some dict keys there.\r\n bboxes = label.pop(\"bboxes\")\r\n segments = label.pop(\"segments\")\r\n keypoints = label.pop(\"keypoints\", None)\r\n bbox_format = label.pop(\"bbox_format\")\r\n normalized = label.pop(\"normalized\")\r\n label[\"instances\"] = Instances(bboxes, segments, keypoints, bbox_format=bbox_format, normalized=normalized)\r\n return label\r\n\r\n @staticmethod\r\n def collate_fn(batch):\r\n # TODO: returning a dict can make thing easier and cleaner when using dataset in training\r\n # but I don't know if this will slow down a little bit.\r\n new_batch = {}\r\n keys = batch[0].keys()\r\n values = list(zip(*[list(b.values()) for b in batch]))\r\n for i, k in enumerate(keys):\r\n value = values[i]\r\n if k == \"img\":\r\n value = torch.stack(value, 0)\r\n if k in [\"masks\", \"keypoints\", \"bboxes\", \"cls\"]:\r\n value = torch.cat(value, 0)\r\n new_batch[k] = value\r\n new_batch[\"batch_idx\"] = list(new_batch[\"batch_idx\"])\r\n for i in range(len(new_batch[\"batch_idx\"])):\r\n new_batch[\"batch_idx\"][i] += i # add target image index for build_targets()\r\n new_batch[\"batch_idx\"] = torch.cat(new_batch[\"batch_idx\"], 0)\r\n return new_batch\r\n\r\n\r\n# Classification dataloaders -------------------------------------------------------------------------------------------\r\nclass ClassificationDataset(torchvision.datasets.ImageFolder):\r\n \"\"\"\r\n YOLOv5 Classification Dataset.\r\n Arguments\r\n root: Dataset path\r\n transform: torchvision transforms, used by default\r\n album_transform: Albumentations transforms, used if installed\r\n \"\"\"\r\n\r\n def __init__(self, root, augment, imgsz, cache=False):\r\n super().__init__(root=root)\r\n self.torch_transforms = classify_transforms(imgsz)\r\n self.album_transforms = classify_albumentations(augment, imgsz) if augment else None\r\n self.cache_ram = cache is True or cache == \"ram\"\r\n self.cache_disk = cache == \"disk\"\r\n self.samples = [list(x) + [Path(x[0]).with_suffix(\".npy\"), None] for x in self.samples] # file, index, npy, im\r\n\r\n def __getitem__(self, i):\r\n f, j, fn, im = self.samples[i] # filename, index, filename.with_suffix('.npy'), image\r\n if self.cache_ram and im is None:\r\n im = self.samples[i][3] = cv2.imread(f)\r\n elif self.cache_disk:\r\n if not fn.exists(): # load npy\r\n np.save(fn.as_posix(), cv2.imread(f))\r\n im = np.load(fn)\r\n else: # read image\r\n im = cv2.imread(f) # BGR\r\n if self.album_transforms:\r\n sample = self.album_transforms(image=cv2.cvtColor(im, cv2.COLOR_BGR2RGB))[\"image\"]\r\n else:\r\n sample = self.torch_transforms(im)\r\n return {'img': sample, 'cls': j}\r\n\r\n def __len__(self) -> int:\r\n return len(self.samples)\r\n\r\n\r\n# TODO: support semantic segmentation\r\nclass SemanticDataset(BaseDataset):\r\n\r\n def __init__(self):\r\n pass\r\n"
},
{
"path": "yolo/data/dataset_wrappers.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport collections\r\nfrom copy import deepcopy\r\n\r\nfrom .augment import LetterBox\r\n\r\n\r\nclass MixAndRectDataset:\r\n \"\"\"A wrapper of multiple images mixed dataset.\r\n\r\n Args:\r\n dataset (:obj:`BaseDataset`): The dataset to be mixed.\r\n transforms (Sequence[dict]): config dict to be composed.\r\n \"\"\"\r\n\r\n def __init__(self, dataset):\r\n self.dataset = dataset\r\n self.imgsz = dataset.imgsz\r\n\r\n def __len__(self):\r\n return len(self.dataset)\r\n\r\n def __getitem__(self, index):\r\n labels = deepcopy(self.dataset[index])\r\n for transform in self.dataset.transforms.tolist():\r\n # mosaic and mixup\r\n if hasattr(transform, \"get_indexes\"):\r\n indexes = transform.get_indexes(self.dataset)\r\n if not isinstance(indexes, collections.abc.Sequence):\r\n indexes = [indexes]\r\n mix_labels = [deepcopy(self.dataset[index]) for index in indexes]\r\n labels[\"mix_labels\"] = mix_labels\r\n if self.dataset.rect and isinstance(transform, LetterBox):\r\n transform.new_shape = self.dataset.batch_shapes[self.dataset.batch[index]]\r\n labels = transform(labels)\r\n if \"mix_labels\" in labels:\r\n labels.pop(\"mix_labels\")\r\n return labels\r\n"
},
{
"path": "yolo/data/datasets/Argoverse.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# Argoverse-HD dataset (ring-front-center camera) http://www.cs.cmu.edu/~mengtial/proj/streaming/ by Argo AI\r\n# Example usage: python train.py --data Argoverse.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ Argoverse β downloads here (31.3 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/Argoverse # dataset root dir\r\ntrain: Argoverse-1.1/images/train/ # train images (relative to 'path') 39384 images\r\nval: Argoverse-1.1/images/val/ # val images (relative to 'path') 15062 images\r\ntest: Argoverse-1.1/images/test/ # test images (optional) https://eval.ai/web/challenges/challenge-page/800/overview\r\n\r\n# Classes\r\nnames:\r\n 0: person\r\n 1: bicycle\r\n 2: car\r\n 3: motorcycle\r\n 4: bus\r\n 5: truck\r\n 6: traffic_light\r\n 7: stop_sign\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n import json\r\n\r\n from tqdm import tqdm\r\n from utils.general import download, Path\r\n\r\n\r\n def argoverse2yolo(set):\r\n labels = {}\r\n a = json.load(open(set, \"rb\"))\r\n for annot in tqdm(a['annotations'], desc=f\"Converting {set} to YOLOv5 format...\"):\r\n img_id = annot['image_id']\r\n img_name = a['images'][img_id]['name']\r\n img_label_name = f'{img_name[:-3]}txt'\r\n\r\n cls = annot['category_id'] # instance class id\r\n x_center, y_center, width, height = annot['bbox']\r\n x_center = (x_center + width / 2) / 1920.0 # offset and scale\r\n y_center = (y_center + height / 2) / 1200.0 # offset and scale\r\n width /= 1920.0 # scale\r\n height /= 1200.0 # scale\r\n\r\n img_dir = set.parents[2] / 'Argoverse-1.1' / 'labels' / a['seq_dirs'][a['images'][annot['image_id']]['sid']]\r\n if not img_dir.exists():\r\n img_dir.mkdir(parents=True, exist_ok=True)\r\n\r\n k = str(img_dir / img_label_name)\r\n if k not in labels:\r\n labels[k] = []\r\n labels[k].append(f\"{cls} {x_center} {y_center} {width} {height}\\n\")\r\n\r\n for k in labels:\r\n with open(k, \"w\") as f:\r\n f.writelines(labels[k])\r\n\r\n\r\n # Download\r\n dir = Path(yaml['path']) # dataset root dir\r\n urls = ['https://argoverse-hd.s3.us-east-2.amazonaws.com/Argoverse-HD-Full.zip']\r\n download(urls, dir=dir, delete=False)\r\n\r\n # Convert\r\n annotations_dir = 'Argoverse-HD/annotations/'\r\n (dir / 'Argoverse-1.1' / 'tracking').rename(dir / 'Argoverse-1.1' / 'images') # rename 'tracking' to 'images'\r\n for d in \"train.json\", \"val.json\":\r\n argoverse2yolo(dir / annotations_dir / d) # convert VisDrone annotations to YOLO labels\r\n"
},
{
"path": "yolo/data/datasets/GlobalWheat2020.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# Global Wheat 2020 dataset http://www.global-wheat.com/ by University of Saskatchewan\r\n# Example usage: python train.py --data GlobalWheat2020.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ GlobalWheat2020 β downloads here (7.0 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/GlobalWheat2020 # dataset root dir\r\ntrain: # train images (relative to 'path') 3422 images\r\n - images/arvalis_1\r\n - images/arvalis_2\r\n - images/arvalis_3\r\n - images/ethz_1\r\n - images/rres_1\r\n - images/inrae_1\r\n - images/usask_1\r\nval: # val images (relative to 'path') 748 images (WARNING: train set contains ethz_1)\r\n - images/ethz_1\r\ntest: # test images (optional) 1276 images\r\n - images/utokyo_1\r\n - images/utokyo_2\r\n - images/nau_1\r\n - images/uq_1\r\n\r\n# Classes\r\nnames:\r\n 0: wheat_head\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n from utils.general import download, Path\r\n\r\n\r\n # Download\r\n dir = Path(yaml['path']) # dataset root dir\r\n urls = ['https://zenodo.org/record/4298502/files/global-wheat-codalab-official.zip',\r\n 'https://github.com/ultralytics/yolov5/releases/download/v1.0/GlobalWheat2020_labels.zip']\r\n download(urls, dir=dir)\r\n\r\n # Make Directories\r\n for p in 'annotations', 'images', 'labels':\r\n (dir / p).mkdir(parents=True, exist_ok=True)\r\n\r\n # Move\r\n for p in 'arvalis_1', 'arvalis_2', 'arvalis_3', 'ethz_1', 'rres_1', 'inrae_1', 'usask_1', \\\r\n 'utokyo_1', 'utokyo_2', 'nau_1', 'uq_1':\r\n (dir / p).rename(dir / 'images' / p) # move to /images\r\n f = (dir / p).with_suffix('.json') # json file\r\n if f.exists():\r\n f.rename((dir / 'annotations' / p).with_suffix('.json')) # move to /annotations\r\n"
},
{
"path": "yolo/data/datasets/ImageNet.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# ImageNet-1k dataset https://www.image-net.org/index.php by Stanford University\r\n# Simplified class names from https://github.com/anishathalye/imagenet-simple-labels\r\n# Example usage: python classify/train.py --data imagenet\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ imagenet β downloads here (144 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/imagenet # dataset root dir\r\ntrain: train # train images (relative to 'path') 1281167 images\r\nval: val # val images (relative to 'path') 50000 images\r\ntest: # test images (optional)\r\n\r\n# Classes\r\nnames:\r\n 0: tench\r\n 1: goldfish\r\n 2: great white shark\r\n 3: tiger shark\r\n 4: hammerhead shark\r\n 5: electric ray\r\n 6: stingray\r\n 7: cock\r\n 8: hen\r\n 9: ostrich\r\n 10: brambling\r\n 11: goldfinch\r\n 12: house finch\r\n 13: junco\r\n 14: indigo bunting\r\n 15: American robin\r\n 16: bulbul\r\n 17: jay\r\n 18: magpie\r\n 19: chickadee\r\n 20: American dipper\r\n 21: kite\r\n 22: bald eagle\r\n 23: vulture\r\n 24: great grey owl\r\n 25: fire salamander\r\n 26: smooth newt\r\n 27: newt\r\n 28: spotted salamander\r\n 29: axolotl\r\n 30: American bullfrog\r\n 31: tree frog\r\n 32: tailed frog\r\n 33: loggerhead sea turtle\r\n 34: leatherback sea turtle\r\n 35: mud turtle\r\n 36: terrapin\r\n 37: box turtle\r\n 38: banded gecko\r\n 39: green iguana\r\n 40: Carolina anole\r\n 41: desert grassland whiptail lizard\r\n 42: agama\r\n 43: frilled-necked lizard\r\n 44: alligator lizard\r\n 45: Gila monster\r\n 46: European green lizard\r\n 47: chameleon\r\n 48: Komodo dragon\r\n 49: Nile crocodile\r\n 50: American alligator\r\n 51: triceratops\r\n 52: worm snake\r\n 53: ring-necked snake\r\n 54: eastern hog-nosed snake\r\n 55: smooth green snake\r\n 56: kingsnake\r\n 57: garter snake\r\n 58: water snake\r\n 59: vine snake\r\n 60: night snake\r\n 61: boa constrictor\r\n 62: African rock python\r\n 63: Indian cobra\r\n 64: green mamba\r\n 65: sea snake\r\n 66: Saharan horned viper\r\n 67: eastern diamondback rattlesnake\r\n 68: sidewinder\r\n 69: trilobite\r\n 70: harvestman\r\n 71: scorpion\r\n 72: yellow garden spider\r\n 73: barn spider\r\n 74: European garden spider\r\n 75: southern black widow\r\n 76: tarantula\r\n 77: wolf spider\r\n 78: tick\r\n 79: centipede\r\n 80: black grouse\r\n 81: ptarmigan\r\n 82: ruffed grouse\r\n 83: prairie grouse\r\n 84: peacock\r\n 85: quail\r\n 86: partridge\r\n 87: grey parrot\r\n 88: macaw\r\n 89: sulphur-crested cockatoo\r\n 90: lorikeet\r\n 91: coucal\r\n 92: bee eater\r\n 93: hornbill\r\n 94: hummingbird\r\n 95: jacamar\r\n 96: toucan\r\n 97: duck\r\n 98: red-breasted merganser\r\n 99: goose\r\n 100: black swan\r\n 101: tusker\r\n 102: echidna\r\n 103: platypus\r\n 104: wallaby\r\n 105: koala\r\n 106: wombat\r\n 107: jellyfish\r\n 108: sea anemone\r\n 109: brain coral\r\n 110: flatworm\r\n 111: nematode\r\n 112: conch\r\n 113: snail\r\n 114: slug\r\n 115: sea slug\r\n 116: chiton\r\n 117: chambered nautilus\r\n 118: Dungeness crab\r\n 119: rock crab\r\n 120: fiddler crab\r\n 121: red king crab\r\n 122: American lobster\r\n 123: spiny lobster\r\n 124: crayfish\r\n 125: hermit crab\r\n 126: isopod\r\n 127: white stork\r\n 128: black stork\r\n 129: spoonbill\r\n 130: flamingo\r\n 131: little blue heron\r\n 132: great egret\r\n 133: bittern\r\n 134: crane (bird)\r\n 135: limpkin\r\n 136: common gallinule\r\n 137: American coot\r\n 138: bustard\r\n 139: ruddy turnstone\r\n 140: dunlin\r\n 141: common redshank\r\n 142: dowitcher\r\n 143: oystercatcher\r\n 144: pelican\r\n 145: king penguin\r\n 146: albatross\r\n 147: grey whale\r\n 148: killer whale\r\n 149: dugong\r\n 150: sea lion\r\n 151: Chihuahua\r\n 152: Japanese Chin\r\n 153: Maltese\r\n 154: Pekingese\r\n 155: Shih Tzu\r\n 156: King Charles Spaniel\r\n 157: Papillon\r\n 158: toy terrier\r\n 159: Rhodesian Ridgeback\r\n 160: Afghan Hound\r\n 161: Basset Hound\r\n 162: Beagle\r\n 163: Bloodhound\r\n 164: Bluetick Coonhound\r\n 165: Black and Tan Coonhound\r\n 166: Treeing Walker Coonhound\r\n 167: English foxhound\r\n 168: Redbone Coonhound\r\n 169: borzoi\r\n 170: Irish Wolfhound\r\n 171: Italian Greyhound\r\n 172: Whippet\r\n 173: Ibizan Hound\r\n 174: Norwegian Elkhound\r\n 175: Otterhound\r\n 176: Saluki\r\n 177: Scottish Deerhound\r\n 178: Weimaraner\r\n 179: Staffordshire Bull Terrier\r\n 180: American Staffordshire Terrier\r\n 181: Bedlington Terrier\r\n 182: Border Terrier\r\n 183: Kerry Blue Terrier\r\n 184: Irish Terrier\r\n 185: Norfolk Terrier\r\n 186: Norwich Terrier\r\n 187: Yorkshire Terrier\r\n 188: Wire Fox Terrier\r\n 189: Lakeland Terrier\r\n 190: Sealyham Terrier\r\n 191: Airedale Terrier\r\n 192: Cairn Terrier\r\n 193: Australian Terrier\r\n 194: Dandie Dinmont Terrier\r\n 195: Boston Terrier\r\n 196: Miniature Schnauzer\r\n 197: Giant Schnauzer\r\n 198: Standard Schnauzer\r\n 199: Scottish Terrier\r\n 200: Tibetan Terrier\r\n 201: Australian Silky Terrier\r\n 202: Soft-coated Wheaten Terrier\r\n 203: West Highland White Terrier\r\n 204: Lhasa Apso\r\n 205: Flat-Coated Retriever\r\n 206: Curly-coated Retriever\r\n 207: Golden Retriever\r\n 208: Labrador Retriever\r\n 209: Chesapeake Bay Retriever\r\n 210: German Shorthaired Pointer\r\n 211: Vizsla\r\n 212: English Setter\r\n 213: Irish Setter\r\n 214: Gordon Setter\r\n 215: Brittany\r\n 216: Clumber Spaniel\r\n 217: English Springer Spaniel\r\n 218: Welsh Springer Spaniel\r\n 219: Cocker Spaniels\r\n 220: Sussex Spaniel\r\n 221: Irish Water Spaniel\r\n 222: Kuvasz\r\n 223: Schipperke\r\n 224: Groenendael\r\n 225: Malinois\r\n 226: Briard\r\n 227: Australian Kelpie\r\n 228: Komondor\r\n 229: Old English Sheepdog\r\n 230: Shetland Sheepdog\r\n 231: collie\r\n 232: Border Collie\r\n 233: Bouvier des Flandres\r\n 234: Rottweiler\r\n 235: German Shepherd Dog\r\n 236: Dobermann\r\n 237: Miniature Pinscher\r\n 238: Greater Swiss Mountain Dog\r\n 239: Bernese Mountain Dog\r\n 240: Appenzeller Sennenhund\r\n 241: Entlebucher Sennenhund\r\n 242: Boxer\r\n 243: Bullmastiff\r\n 244: Tibetan Mastiff\r\n 245: French Bulldog\r\n 246: Great Dane\r\n 247: St. Bernard\r\n 248: husky\r\n 249: Alaskan Malamute\r\n 250: Siberian Husky\r\n 251: Dalmatian\r\n 252: Affenpinscher\r\n 253: Basenji\r\n 254: pug\r\n 255: Leonberger\r\n 256: Newfoundland\r\n 257: Pyrenean Mountain Dog\r\n 258: Samoyed\r\n 259: Pomeranian\r\n 260: Chow Chow\r\n 261: Keeshond\r\n 262: Griffon Bruxellois\r\n 263: Pembroke Welsh Corgi\r\n 264: Cardigan Welsh Corgi\r\n 265: Toy Poodle\r\n 266: Miniature Poodle\r\n 267: Standard Poodle\r\n 268: Mexican hairless dog\r\n 269: grey wolf\r\n 270: Alaskan tundra wolf\r\n 271: red wolf\r\n 272: coyote\r\n 273: dingo\r\n 274: dhole\r\n 275: African wild dog\r\n 276: hyena\r\n 277: red fox\r\n 278: kit fox\r\n 279: Arctic fox\r\n 280: grey fox\r\n 281: tabby cat\r\n 282: tiger cat\r\n 283: Persian cat\r\n 284: Siamese cat\r\n 285: Egyptian Mau\r\n 286: cougar\r\n 287: lynx\r\n 288: leopard\r\n 289: snow leopard\r\n 290: jaguar\r\n 291: lion\r\n 292: tiger\r\n 293: cheetah\r\n 294: brown bear\r\n 295: American black bear\r\n 296: polar bear\r\n 297: sloth bear\r\n 298: mongoose\r\n 299: meerkat\r\n 300: tiger beetle\r\n 301: ladybug\r\n 302: ground beetle\r\n 303: longhorn beetle\r\n 304: leaf beetle\r\n 305: dung beetle\r\n 306: rhinoceros beetle\r\n 307: weevil\r\n 308: fly\r\n 309: bee\r\n 310: ant\r\n 311: grasshopper\r\n 312: cricket\r\n 313: stick insect\r\n 314: cockroach\r\n 315: mantis\r\n 316: cicada\r\n 317: leafhopper\r\n 318: lacewing\r\n 319: dragonfly\r\n 320: damselfly\r\n 321: red admiral\r\n 322: ringlet\r\n 323: monarch butterfly\r\n 324: small white\r\n 325: sulphur butterfly\r\n 326: gossamer-winged butterfly\r\n 327: starfish\r\n 328: sea urchin\r\n 329: sea cucumber\r\n 330: cottontail rabbit\r\n 331: hare\r\n 332: Angora rabbit\r\n 333: hamster\r\n 334: porcupine\r\n 335: fox squirrel\r\n 336: marmot\r\n 337: beaver\r\n 338: guinea pig\r\n 339: common sorrel\r\n 340: zebra\r\n 341: pig\r\n 342: wild boar\r\n 343: warthog\r\n 344: hippopotamus\r\n 345: ox\r\n 346: water buffalo\r\n 347: bison\r\n 348: ram\r\n 349: bighorn sheep\r\n 350: Alpine ibex\r\n 351: hartebeest\r\n 352: impala\r\n 353: gazelle\r\n 354: dromedary\r\n 355: llama\r\n 356: weasel\r\n 357: mink\r\n 358: European polecat\r\n 359: black-footed ferret\r\n 360: otter\r\n 361: skunk\r\n 362: badger\r\n 363: armadillo\r\n 364: three-toed sloth\r\n 365: orangutan\r\n 366: gorilla\r\n 367: chimpanzee\r\n 368: gibbon\r\n 369: siamang\r\n 370: guenon\r\n 371: patas monkey\r\n 372: baboon\r\n 373: macaque\r\n 374: langur\r\n 375: black-and-white colobus\r\n 376: proboscis monkey\r\n 377: marmoset\r\n 378: white-headed capuchin\r\n 379: howler monkey\r\n 380: titi\r\n 381: Geoffroy's spider monkey\r\n 382: common squirrel monkey\r\n 383: ring-tailed lemur\r\n 384: indri\r\n 385: Asian elephant\r\n 386: African bush elephant\r\n 387: red panda\r\n 388: giant panda\r\n 389: snoek\r\n 390: eel\r\n 391: coho salmon\r\n 392: rock beauty\r\n 393: clownfish\r\n 394: sturgeon\r\n 395: garfish\r\n 396: lionfish\r\n 397: pufferfish\r\n 398: abacus\r\n 399: abaya\r\n 400: academic gown\r\n 401: accordion\r\n 402: acoustic guitar\r\n 403: aircraft carrier\r\n 404: airliner\r\n 405: airship\r\n 406: altar\r\n 407: ambulance\r\n 408: amphibious vehicle\r\n 409: analog clock\r\n 410: apiary\r\n 411: apron\r\n 412: waste container\r\n 413: assault rifle\r\n 414: backpack\r\n 415: bakery\r\n 416: balance beam\r\n 417: balloon\r\n 418: ballpoint pen\r\n 419: Band-Aid\r\n 420: banjo\r\n 421: baluster\r\n 422: barbell\r\n 423: barber chair\r\n 424: barbershop\r\n 425: barn\r\n 426: barometer\r\n 427: barrel\r\n 428: wheelbarrow\r\n 429: baseball\r\n 430: basketball\r\n 431: bassinet\r\n 432: bassoon\r\n 433: swimming cap\r\n 434: bath towel\r\n 435: bathtub\r\n 436: station wagon\r\n 437: lighthouse\r\n 438: beaker\r\n 439: military cap\r\n 440: beer bottle\r\n 441: beer glass\r\n 442: bell-cot\r\n 443: bib\r\n 444: tandem bicycle\r\n 445: bikini\r\n 446: ring binder\r\n 447: binoculars\r\n 448: birdhouse\r\n 449: boathouse\r\n 450: bobsleigh\r\n 451: bolo tie\r\n 452: poke bonnet\r\n 453: bookcase\r\n 454: bookstore\r\n 455: bottle cap\r\n 456: bow\r\n 457: bow tie\r\n 458: brass\r\n 459: bra\r\n 460: breakwater\r\n 461: breastplate\r\n 462: broom\r\n 463: bucket\r\n 464: buckle\r\n 465: bulletproof vest\r\n 466: high-speed train\r\n 467: butcher shop\r\n 468: taxicab\r\n 469: cauldron\r\n 470: candle\r\n 471: cannon\r\n 472: canoe\r\n 473: can opener\r\n 474: cardigan\r\n 475: car mirror\r\n 476: carousel\r\n 477: tool kit\r\n 478: carton\r\n 479: car wheel\r\n 480: automated teller machine\r\n 481: cassette\r\n 482: cassette player\r\n 483: castle\r\n 484: catamaran\r\n 485: CD player\r\n 486: cello\r\n 487: mobile phone\r\n 488: chain\r\n 489: chain-link fence\r\n 490: chain mail\r\n 491: chainsaw\r\n 492: chest\r\n 493: chiffonier\r\n 494: chime\r\n 495: china cabinet\r\n 496: Christmas stocking\r\n 497: church\r\n 498: movie theater\r\n 499: cleaver\r\n 500: cliff dwelling\r\n 501: cloak\r\n 502: clogs\r\n 503: cocktail shaker\r\n 504: coffee mug\r\n 505: coffeemaker\r\n 506: coil\r\n 507: combination lock\r\n 508: computer keyboard\r\n 509: confectionery store\r\n 510: container ship\r\n 511: convertible\r\n 512: corkscrew\r\n 513: cornet\r\n 514: cowboy boot\r\n 515: cowboy hat\r\n 516: cradle\r\n 517: crane (machine)\r\n 518: crash helmet\r\n 519: crate\r\n 520: infant bed\r\n 521: Crock Pot\r\n 522: croquet ball\r\n 523: crutch\r\n 524: cuirass\r\n 525: dam\r\n 526: desk\r\n 527: desktop computer\r\n 528: rotary dial telephone\r\n 529: diaper\r\n 530: digital clock\r\n 531: digital watch\r\n 532: dining table\r\n 533: dishcloth\r\n 534: dishwasher\r\n 535: disc brake\r\n 536: dock\r\n 537: dog sled\r\n 538: dome\r\n 539: doormat\r\n 540: drilling rig\r\n 541: drum\r\n 542: drumstick\r\n 543: dumbbell\r\n 544: Dutch oven\r\n 545: electric fan\r\n 546: electric guitar\r\n 547: electric locomotive\r\n 548: entertainment center\r\n 549: envelope\r\n 550: espresso machine\r\n 551: face powder\r\n 552: feather boa\r\n 553: filing cabinet\r\n 554: fireboat\r\n 555: fire engine\r\n 556: fire screen sheet\r\n 557: flagpole\r\n 558: flute\r\n 559: folding chair\r\n 560: football helmet\r\n 561: forklift\r\n 562: fountain\r\n 563: fountain pen\r\n 564: four-poster bed\r\n 565: freight car\r\n 566: French horn\r\n 567: frying pan\r\n 568: fur coat\r\n 569: garbage truck\r\n 570: gas mask\r\n 571: gas pump\r\n 572: goblet\r\n 573: go-kart\r\n 574: golf ball\r\n 575: golf cart\r\n 576: gondola\r\n 577: gong\r\n 578: gown\r\n 579: grand piano\r\n 580: greenhouse\r\n 581: grille\r\n 582: grocery store\r\n 583: guillotine\r\n 584: barrette\r\n 585: hair spray\r\n 586: half-track\r\n 587: hammer\r\n 588: hamper\r\n 589: hair dryer\r\n 590: hand-held computer\r\n 591: handkerchief\r\n 592: hard disk drive\r\n 593: harmonica\r\n 594: harp\r\n 595: harvester\r\n 596: hatchet\r\n 597: holster\r\n 598: home theater\r\n 599: honeycomb\r\n 600: hook\r\n 601: hoop skirt\r\n 602: horizontal bar\r\n 603: horse-drawn vehicle\r\n 604: hourglass\r\n 605: iPod\r\n 606: clothes iron\r\n 607: jack-o'-lantern\r\n 608: jeans\r\n 609: jeep\r\n 610: T-shirt\r\n 611: jigsaw puzzle\r\n 612: pulled rickshaw\r\n 613: joystick\r\n 614: kimono\r\n 615: knee pad\r\n 616: knot\r\n 617: lab coat\r\n 618: ladle\r\n 619: lampshade\r\n 620: laptop computer\r\n 621: lawn mower\r\n 622: lens cap\r\n 623: paper knife\r\n 624: library\r\n 625: lifeboat\r\n 626: lighter\r\n 627: limousine\r\n 628: ocean liner\r\n 629: lipstick\r\n 630: slip-on shoe\r\n 631: lotion\r\n 632: speaker\r\n 633: loupe\r\n 634: sawmill\r\n 635: magnetic compass\r\n 636: mail bag\r\n 637: mailbox\r\n 638: tights\r\n 639: tank suit\r\n 640: manhole cover\r\n 641: maraca\r\n 642: marimba\r\n 643: mask\r\n 644: match\r\n 645: maypole\r\n 646: maze\r\n 647: measuring cup\r\n 648: medicine chest\r\n 649: megalith\r\n 650: microphone\r\n 651: microwave oven\r\n 652: military uniform\r\n 653: milk can\r\n 654: minibus\r\n 655: miniskirt\r\n 656: minivan\r\n 657: missile\r\n 658: mitten\r\n 659: mixing bowl\r\n 660: mobile home\r\n 661: Model T\r\n 662: modem\r\n 663: monastery\r\n 664: monitor\r\n 665: moped\r\n 666: mortar\r\n 667: square academic cap\r\n 668: mosque\r\n 669: mosquito net\r\n 670: scooter\r\n 671: mountain bike\r\n 672: tent\r\n 673: computer mouse\r\n 674: mousetrap\r\n 675: moving van\r\n 676: muzzle\r\n 677: nail\r\n 678: neck brace\r\n 679: necklace\r\n 680: nipple\r\n 681: notebook computer\r\n 682: obelisk\r\n 683: oboe\r\n 684: ocarina\r\n 685: odometer\r\n 686: oil filter\r\n 687: organ\r\n 688: oscilloscope\r\n 689: overskirt\r\n 690: bullock cart\r\n 691: oxygen mask\r\n 692: packet\r\n 693: paddle\r\n 694: paddle wheel\r\n 695: padlock\r\n 696: paintbrush\r\n 697: pajamas\r\n 698: palace\r\n 699: pan flute\r\n 700: paper towel\r\n 701: parachute\r\n 702: parallel bars\r\n 703: park bench\r\n 704: parking meter\r\n 705: passenger car\r\n 706: patio\r\n 707: payphone\r\n 708: pedestal\r\n 709: pencil case\r\n 710: pencil sharpener\r\n 711: perfume\r\n 712: Petri dish\r\n 713: photocopier\r\n 714: plectrum\r\n 715: Pickelhaube\r\n 716: picket fence\r\n 717: pickup truck\r\n 718: pier\r\n 719: piggy bank\r\n 720: pill bottle\r\n 721: pillow\r\n 722: ping-pong ball\r\n 723: pinwheel\r\n 724: pirate ship\r\n 725: pitcher\r\n 726: hand plane\r\n 727: planetarium\r\n 728: plastic bag\r\n 729: plate rack\r\n 730: plow\r\n 731: plunger\r\n 732: Polaroid camera\r\n 733: pole\r\n 734: police van\r\n 735: poncho\r\n 736: billiard table\r\n 737: soda bottle\r\n 738: pot\r\n 739: potter's wheel\r\n 740: power drill\r\n 741: prayer rug\r\n 742: printer\r\n 743: prison\r\n 744: projectile\r\n 745: projector\r\n 746: hockey puck\r\n 747: punching bag\r\n 748: purse\r\n 749: quill\r\n 750: quilt\r\n 751: race car\r\n 752: racket\r\n 753: radiator\r\n 754: radio\r\n 755: radio telescope\r\n 756: rain barrel\r\n 757: recreational vehicle\r\n 758: reel\r\n 759: reflex camera\r\n 760: refrigerator\r\n 761: remote control\r\n 762: restaurant\r\n 763: revolver\r\n 764: rifle\r\n 765: rocking chair\r\n 766: rotisserie\r\n 767: eraser\r\n 768: rugby ball\r\n 769: ruler\r\n 770: running shoe\r\n 771: safe\r\n 772: safety pin\r\n 773: salt shaker\r\n 774: sandal\r\n 775: sarong\r\n 776: saxophone\r\n 777: scabbard\r\n 778: weighing scale\r\n 779: school bus\r\n 780: schooner\r\n 781: scoreboard\r\n 782: CRT screen\r\n 783: screw\r\n 784: screwdriver\r\n 785: seat belt\r\n 786: sewing machine\r\n 787: shield\r\n 788: shoe store\r\n 789: shoji\r\n 790: shopping basket\r\n 791: shopping cart\r\n 792: shovel\r\n 793: shower cap\r\n 794: shower curtain\r\n 795: ski\r\n 796: ski mask\r\n 797: sleeping bag\r\n 798: slide rule\r\n 799: sliding door\r\n 800: slot machine\r\n 801: snorkel\r\n 802: snowmobile\r\n 803: snowplow\r\n 804: soap dispenser\r\n 805: soccer ball\r\n 806: sock\r\n 807: solar thermal collector\r\n 808: sombrero\r\n 809: soup bowl\r\n 810: space bar\r\n 811: space heater\r\n 812: space shuttle\r\n 813: spatula\r\n 814: motorboat\r\n 815: spider web\r\n 816: spindle\r\n 817: sports car\r\n 818: spotlight\r\n 819: stage\r\n 820: steam locomotive\r\n 821: through arch bridge\r\n 822: steel drum\r\n 823: stethoscope\r\n 824: scarf\r\n 825: stone wall\r\n 826: stopwatch\r\n 827: stove\r\n 828: strainer\r\n 829: tram\r\n 830: stretcher\r\n 831: couch\r\n 832: stupa\r\n 833: submarine\r\n 834: suit\r\n 835: sundial\r\n 836: sunglass\r\n 837: sunglasses\r\n 838: sunscreen\r\n 839: suspension bridge\r\n 840: mop\r\n 841: sweatshirt\r\n 842: swimsuit\r\n 843: swing\r\n 844: switch\r\n 845: syringe\r\n 846: table lamp\r\n 847: tank\r\n 848: tape player\r\n 849: teapot\r\n 850: teddy bear\r\n 851: television\r\n 852: tennis ball\r\n 853: thatched roof\r\n 854: front curtain\r\n 855: thimble\r\n 856: threshing machine\r\n 857: throne\r\n 858: tile roof\r\n 859: toaster\r\n 860: tobacco shop\r\n 861: toilet seat\r\n 862: torch\r\n 863: totem pole\r\n 864: tow truck\r\n 865: toy store\r\n 866: tractor\r\n 867: semi-trailer truck\r\n 868: tray\r\n 869: trench coat\r\n 870: tricycle\r\n 871: trimaran\r\n 872: tripod\r\n 873: triumphal arch\r\n 874: trolleybus\r\n 875: trombone\r\n 876: tub\r\n 877: turnstile\r\n 878: typewriter keyboard\r\n 879: umbrella\r\n 880: unicycle\r\n 881: upright piano\r\n 882: vacuum cleaner\r\n 883: vase\r\n 884: vault\r\n 885: velvet\r\n 886: vending machine\r\n 887: vestment\r\n 888: viaduct\r\n 889: violin\r\n 890: volleyball\r\n 891: waffle iron\r\n 892: wall clock\r\n 893: wallet\r\n 894: wardrobe\r\n 895: military aircraft\r\n 896: sink\r\n 897: washing machine\r\n 898: water bottle\r\n 899: water jug\r\n 900: water tower\r\n 901: whiskey jug\r\n 902: whistle\r\n 903: wig\r\n 904: window screen\r\n 905: window shade\r\n 906: Windsor tie\r\n 907: wine bottle\r\n 908: wing\r\n 909: wok\r\n 910: wooden spoon\r\n 911: wool\r\n 912: split-rail fence\r\n 913: shipwreck\r\n 914: yawl\r\n 915: yurt\r\n 916: website\r\n 917: comic book\r\n 918: crossword\r\n 919: traffic sign\r\n 920: traffic light\r\n 921: dust jacket\r\n 922: menu\r\n 923: plate\r\n 924: guacamole\r\n 925: consomme\r\n 926: hot pot\r\n 927: trifle\r\n 928: ice cream\r\n 929: ice pop\r\n 930: baguette\r\n 931: bagel\r\n 932: pretzel\r\n 933: cheeseburger\r\n 934: hot dog\r\n 935: mashed potato\r\n 936: cabbage\r\n 937: broccoli\r\n 938: cauliflower\r\n 939: zucchini\r\n 940: spaghetti squash\r\n 941: acorn squash\r\n 942: butternut squash\r\n 943: cucumber\r\n 944: artichoke\r\n 945: bell pepper\r\n 946: cardoon\r\n 947: mushroom\r\n 948: Granny Smith\r\n 949: strawberry\r\n 950: orange\r\n 951: lemon\r\n 952: fig\r\n 953: pineapple\r\n 954: banana\r\n 955: jackfruit\r\n 956: custard apple\r\n 957: pomegranate\r\n 958: hay\r\n 959: carbonara\r\n 960: chocolate syrup\r\n 961: dough\r\n 962: meatloaf\r\n 963: pizza\r\n 964: pot pie\r\n 965: burrito\r\n 966: red wine\r\n 967: espresso\r\n 968: cup\r\n 969: eggnog\r\n 970: alp\r\n 971: bubble\r\n 972: cliff\r\n 973: coral reef\r\n 974: geyser\r\n 975: lakeshore\r\n 976: promontory\r\n 977: shoal\r\n 978: seashore\r\n 979: valley\r\n 980: volcano\r\n 981: baseball player\r\n 982: bridegroom\r\n 983: scuba diver\r\n 984: rapeseed\r\n 985: daisy\r\n 986: yellow lady's slipper\r\n 987: corn\r\n 988: acorn\r\n 989: rose hip\r\n 990: horse chestnut seed\r\n 991: coral fungus\r\n 992: agaric\r\n 993: gyromitra\r\n 994: stinkhorn mushroom\r\n 995: earth star\r\n 996: hen-of-the-woods\r\n 997: bolete\r\n 998: ear\r\n 999: toilet paper\r\n\r\n\r\n# Download script/URL (optional)\r\ndownload: data/scripts/get_imagenet.sh\r\n"
},
{
"path": "yolo/data/datasets/Objects365.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# Objects365 dataset https://www.objects365.org/ by Megvii\r\n# Example usage: python train.py --data Objects365.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ Objects365 β downloads here (712 GB = 367G data + 345G zips)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/Objects365 # dataset root dir\r\ntrain: images/train # train images (relative to 'path') 1742289 images\r\nval: images/val # val images (relative to 'path') 80000 images\r\ntest: # test images (optional)\r\n\r\n# Classes\r\nnames:\r\n 0: Person\r\n 1: Sneakers\r\n 2: Chair\r\n 3: Other Shoes\r\n 4: Hat\r\n 5: Car\r\n 6: Lamp\r\n 7: Glasses\r\n 8: Bottle\r\n 9: Desk\r\n 10: Cup\r\n 11: Street Lights\r\n 12: Cabinet/shelf\r\n 13: Handbag/Satchel\r\n 14: Bracelet\r\n 15: Plate\r\n 16: Picture/Frame\r\n 17: Helmet\r\n 18: Book\r\n 19: Gloves\r\n 20: Storage box\r\n 21: Boat\r\n 22: Leather Shoes\r\n 23: Flower\r\n 24: Bench\r\n 25: Potted Plant\r\n 26: Bowl/Basin\r\n 27: Flag\r\n 28: Pillow\r\n 29: Boots\r\n 30: Vase\r\n 31: Microphone\r\n 32: Necklace\r\n 33: Ring\r\n 34: SUV\r\n 35: Wine Glass\r\n 36: Belt\r\n 37: Monitor/TV\r\n 38: Backpack\r\n 39: Umbrella\r\n 40: Traffic Light\r\n 41: Speaker\r\n 42: Watch\r\n 43: Tie\r\n 44: Trash bin Can\r\n 45: Slippers\r\n 46: Bicycle\r\n 47: Stool\r\n 48: Barrel/bucket\r\n 49: Van\r\n 50: Couch\r\n 51: Sandals\r\n 52: Basket\r\n 53: Drum\r\n 54: Pen/Pencil\r\n 55: Bus\r\n 56: Wild Bird\r\n 57: High Heels\r\n 58: Motorcycle\r\n 59: Guitar\r\n 60: Carpet\r\n 61: Cell Phone\r\n 62: Bread\r\n 63: Camera\r\n 64: Canned\r\n 65: Truck\r\n 66: Traffic cone\r\n 67: Cymbal\r\n 68: Lifesaver\r\n 69: Towel\r\n 70: Stuffed Toy\r\n 71: Candle\r\n 72: Sailboat\r\n 73: Laptop\r\n 74: Awning\r\n 75: Bed\r\n 76: Faucet\r\n 77: Tent\r\n 78: Horse\r\n 79: Mirror\r\n 80: Power outlet\r\n 81: Sink\r\n 82: Apple\r\n 83: Air Conditioner\r\n 84: Knife\r\n 85: Hockey Stick\r\n 86: Paddle\r\n 87: Pickup Truck\r\n 88: Fork\r\n 89: Traffic Sign\r\n 90: Balloon\r\n 91: Tripod\r\n 92: Dog\r\n 93: Spoon\r\n 94: Clock\r\n 95: Pot\r\n 96: Cow\r\n 97: Cake\r\n 98: Dinning Table\r\n 99: Sheep\r\n 100: Hanger\r\n 101: Blackboard/Whiteboard\r\n 102: Napkin\r\n 103: Other Fish\r\n 104: Orange/Tangerine\r\n 105: Toiletry\r\n 106: Keyboard\r\n 107: Tomato\r\n 108: Lantern\r\n 109: Machinery Vehicle\r\n 110: Fan\r\n 111: Green Vegetables\r\n 112: Banana\r\n 113: Baseball Glove\r\n 114: Airplane\r\n 115: Mouse\r\n 116: Train\r\n 117: Pumpkin\r\n 118: Soccer\r\n 119: Skiboard\r\n 120: Luggage\r\n 121: Nightstand\r\n 122: Tea pot\r\n 123: Telephone\r\n 124: Trolley\r\n 125: Head Phone\r\n 126: Sports Car\r\n 127: Stop Sign\r\n 128: Dessert\r\n 129: Scooter\r\n 130: Stroller\r\n 131: Crane\r\n 132: Remote\r\n 133: Refrigerator\r\n 134: Oven\r\n 135: Lemon\r\n 136: Duck\r\n 137: Baseball Bat\r\n 138: Surveillance Camera\r\n 139: Cat\r\n 140: Jug\r\n 141: Broccoli\r\n 142: Piano\r\n 143: Pizza\r\n 144: Elephant\r\n 145: Skateboard\r\n 146: Surfboard\r\n 147: Gun\r\n 148: Skating and Skiing shoes\r\n 149: Gas stove\r\n 150: Donut\r\n 151: Bow Tie\r\n 152: Carrot\r\n 153: Toilet\r\n 154: Kite\r\n 155: Strawberry\r\n 156: Other Balls\r\n 157: Shovel\r\n 158: Pepper\r\n 159: Computer Box\r\n 160: Toilet Paper\r\n 161: Cleaning Products\r\n 162: Chopsticks\r\n 163: Microwave\r\n 164: Pigeon\r\n 165: Baseball\r\n 166: Cutting/chopping Board\r\n 167: Coffee Table\r\n 168: Side Table\r\n 169: Scissors\r\n 170: Marker\r\n 171: Pie\r\n 172: Ladder\r\n 173: Snowboard\r\n 174: Cookies\r\n 175: Radiator\r\n 176: Fire Hydrant\r\n 177: Basketball\r\n 178: Zebra\r\n 179: Grape\r\n 180: Giraffe\r\n 181: Potato\r\n 182: Sausage\r\n 183: Tricycle\r\n 184: Violin\r\n 185: Egg\r\n 186: Fire Extinguisher\r\n 187: Candy\r\n 188: Fire Truck\r\n 189: Billiards\r\n 190: Converter\r\n 191: Bathtub\r\n 192: Wheelchair\r\n 193: Golf Club\r\n 194: Briefcase\r\n 195: Cucumber\r\n 196: Cigar/Cigarette\r\n 197: Paint Brush\r\n 198: Pear\r\n 199: Heavy Truck\r\n 200: Hamburger\r\n 201: Extractor\r\n 202: Extension Cord\r\n 203: Tong\r\n 204: Tennis Racket\r\n 205: Folder\r\n 206: American Football\r\n 207: earphone\r\n 208: Mask\r\n 209: Kettle\r\n 210: Tennis\r\n 211: Ship\r\n 212: Swing\r\n 213: Coffee Machine\r\n 214: Slide\r\n 215: Carriage\r\n 216: Onion\r\n 217: Green beans\r\n 218: Projector\r\n 219: Frisbee\r\n 220: Washing Machine/Drying Machine\r\n 221: Chicken\r\n 222: Printer\r\n 223: Watermelon\r\n 224: Saxophone\r\n 225: Tissue\r\n 226: Toothbrush\r\n 227: Ice cream\r\n 228: Hot-air balloon\r\n 229: Cello\r\n 230: French Fries\r\n 231: Scale\r\n 232: Trophy\r\n 233: Cabbage\r\n 234: Hot dog\r\n 235: Blender\r\n 236: Peach\r\n 237: Rice\r\n 238: Wallet/Purse\r\n 239: Volleyball\r\n 240: Deer\r\n 241: Goose\r\n 242: Tape\r\n 243: Tablet\r\n 244: Cosmetics\r\n 245: Trumpet\r\n 246: Pineapple\r\n 247: Golf Ball\r\n 248: Ambulance\r\n 249: Parking meter\r\n 250: Mango\r\n 251: Key\r\n 252: Hurdle\r\n 253: Fishing Rod\r\n 254: Medal\r\n 255: Flute\r\n 256: Brush\r\n 257: Penguin\r\n 258: Megaphone\r\n 259: Corn\r\n 260: Lettuce\r\n 261: Garlic\r\n 262: Swan\r\n 263: Helicopter\r\n 264: Green Onion\r\n 265: Sandwich\r\n 266: Nuts\r\n 267: Speed Limit Sign\r\n 268: Induction Cooker\r\n 269: Broom\r\n 270: Trombone\r\n 271: Plum\r\n 272: Rickshaw\r\n 273: Goldfish\r\n 274: Kiwi fruit\r\n 275: Router/modem\r\n 276: Poker Card\r\n 277: Toaster\r\n 278: Shrimp\r\n 279: Sushi\r\n 280: Cheese\r\n 281: Notepaper\r\n 282: Cherry\r\n 283: Pliers\r\n 284: CD\r\n 285: Pasta\r\n 286: Hammer\r\n 287: Cue\r\n 288: Avocado\r\n 289: Hamimelon\r\n 290: Flask\r\n 291: Mushroom\r\n 292: Screwdriver\r\n 293: Soap\r\n 294: Recorder\r\n 295: Bear\r\n 296: Eggplant\r\n 297: Board Eraser\r\n 298: Coconut\r\n 299: Tape Measure/Ruler\r\n 300: Pig\r\n 301: Showerhead\r\n 302: Globe\r\n 303: Chips\r\n 304: Steak\r\n 305: Crosswalk Sign\r\n 306: Stapler\r\n 307: Camel\r\n 308: Formula 1\r\n 309: Pomegranate\r\n 310: Dishwasher\r\n 311: Crab\r\n 312: Hoverboard\r\n 313: Meat ball\r\n 314: Rice Cooker\r\n 315: Tuba\r\n 316: Calculator\r\n 317: Papaya\r\n 318: Antelope\r\n 319: Parrot\r\n 320: Seal\r\n 321: Butterfly\r\n 322: Dumbbell\r\n 323: Donkey\r\n 324: Lion\r\n 325: Urinal\r\n 326: Dolphin\r\n 327: Electric Drill\r\n 328: Hair Dryer\r\n 329: Egg tart\r\n 330: Jellyfish\r\n 331: Treadmill\r\n 332: Lighter\r\n 333: Grapefruit\r\n 334: Game board\r\n 335: Mop\r\n 336: Radish\r\n 337: Baozi\r\n 338: Target\r\n 339: French\r\n 340: Spring Rolls\r\n 341: Monkey\r\n 342: Rabbit\r\n 343: Pencil Case\r\n 344: Yak\r\n 345: Red Cabbage\r\n 346: Binoculars\r\n 347: Asparagus\r\n 348: Barbell\r\n 349: Scallop\r\n 350: Noddles\r\n 351: Comb\r\n 352: Dumpling\r\n 353: Oyster\r\n 354: Table Tennis paddle\r\n 355: Cosmetics Brush/Eyeliner Pencil\r\n 356: Chainsaw\r\n 357: Eraser\r\n 358: Lobster\r\n 359: Durian\r\n 360: Okra\r\n 361: Lipstick\r\n 362: Cosmetics Mirror\r\n 363: Curling\r\n 364: Table Tennis\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n from tqdm import tqdm\r\n\r\n from utils.general import Path, check_requirements, download, np, xyxy2xywhn\r\n\r\n check_requirements(('pycocotools>=2.0',))\r\n from pycocotools.coco import COCO\r\n\r\n # Make Directories\r\n dir = Path(yaml['path']) # dataset root dir\r\n for p in 'images', 'labels':\r\n (dir / p).mkdir(parents=True, exist_ok=True)\r\n for q in 'train', 'val':\r\n (dir / p / q).mkdir(parents=True, exist_ok=True)\r\n\r\n # Train, Val Splits\r\n for split, patches in [('train', 50 + 1), ('val', 43 + 1)]:\r\n print(f\"Processing {split} in {patches} patches ...\")\r\n images, labels = dir / 'images' / split, dir / 'labels' / split\r\n\r\n # Download\r\n url = f\"https://dorc.ks3-cn-beijing.ksyun.com/data-set/2020Objects365%E6%95%B0%E6%8D%AE%E9%9B%86/{split}/\"\r\n if split == 'train':\r\n download([f'{url}zhiyuan_objv2_{split}.tar.gz'], dir=dir, delete=False) # annotations json\r\n download([f'{url}patch{i}.tar.gz' for i in range(patches)], dir=images, curl=True, delete=False, threads=8)\r\n elif split == 'val':\r\n download([f'{url}zhiyuan_objv2_{split}.json'], dir=dir, delete=False) # annotations json\r\n download([f'{url}images/v1/patch{i}.tar.gz' for i in range(15 + 1)], dir=images, curl=True, delete=False, threads=8)\r\n download([f'{url}images/v2/patch{i}.tar.gz' for i in range(16, patches)], dir=images, curl=True, delete=False, threads=8)\r\n\r\n # Move\r\n for f in tqdm(images.rglob('*.jpg'), desc=f'Moving {split} images'):\r\n f.rename(images / f.name) # move to /images/{split}\r\n\r\n # Labels\r\n coco = COCO(dir / f'zhiyuan_objv2_{split}.json')\r\n names = [x[\"name\"] for x in coco.loadCats(coco.getCatIds())]\r\n for cid, cat in enumerate(names):\r\n catIds = coco.getCatIds(catNms=[cat])\r\n imgIds = coco.getImgIds(catIds=catIds)\r\n for im in tqdm(coco.loadImgs(imgIds), desc=f'Class {cid + 1}/{len(names)} {cat}'):\r\n width, height = im[\"width\"], im[\"height\"]\r\n path = Path(im[\"file_name\"]) # image filename\r\n try:\r\n with open(labels / path.with_suffix('.txt').name, 'a') as file:\r\n annIds = coco.getAnnIds(imgIds=im[\"id\"], catIds=catIds, iscrowd=None)\r\n for a in coco.loadAnns(annIds):\r\n x, y, w, h = a['bbox'] # bounding box in xywh (xy top-left corner)\r\n xyxy = np.array([x, y, x + w, y + h])[None] # pixels(1,4)\r\n x, y, w, h = xyxy2xywhn(xyxy, w=width, h=height, clip=True)[0] # normalized and clipped\r\n file.write(f\"{cid} {x:.5f} {y:.5f} {w:.5f} {h:.5f}\\n\")\r\n except Exception as e:\r\n print(e)\r\n"
},
{
"path": "yolo/data/datasets/SKU-110K.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# SKU-110K retail items dataset https://github.com/eg4000/SKU110K_CVPR19 by Trax Retail\r\n# Example usage: python train.py --data SKU-110K.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ SKU-110K β downloads here (13.6 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/SKU-110K # dataset root dir\r\ntrain: train.txt # train images (relative to 'path') 8219 images\r\nval: val.txt # val images (relative to 'path') 588 images\r\ntest: test.txt # test images (optional) 2936 images\r\n\r\n# Classes\r\nnames:\r\n 0: object\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n import shutil\r\n from tqdm import tqdm\r\n from utils.general import np, pd, Path, download, xyxy2xywh\r\n\r\n\r\n # Download\r\n dir = Path(yaml['path']) # dataset root dir\r\n parent = Path(dir.parent) # download dir\r\n urls = ['http://trax-geometry.s3.amazonaws.com/cvpr_challenge/SKU110K_fixed.tar.gz']\r\n download(urls, dir=parent, delete=False)\r\n\r\n # Rename directories\r\n if dir.exists():\r\n shutil.rmtree(dir)\r\n (parent / 'SKU110K_fixed').rename(dir) # rename dir\r\n (dir / 'labels').mkdir(parents=True, exist_ok=True) # create labels dir\r\n\r\n # Convert labels\r\n names = 'image', 'x1', 'y1', 'x2', 'y2', 'class', 'image_width', 'image_height' # column names\r\n for d in 'annotations_train.csv', 'annotations_val.csv', 'annotations_test.csv':\r\n x = pd.read_csv(dir / 'annotations' / d, names=names).values # annotations\r\n images, unique_images = x[:, 0], np.unique(x[:, 0])\r\n with open((dir / d).with_suffix('.txt').__str__().replace('annotations_', ''), 'w') as f:\r\n f.writelines(f'./images/{s}\\n' for s in unique_images)\r\n for im in tqdm(unique_images, desc=f'Converting {dir / d}'):\r\n cls = 0 # single-class dataset\r\n with open((dir / 'labels' / im).with_suffix('.txt'), 'a') as f:\r\n for r in x[images == im]:\r\n w, h = r[6], r[7] # image width, height\r\n xywh = xyxy2xywh(np.array([[r[1] / w, r[2] / h, r[3] / w, r[4] / h]]))[0] # instance\r\n f.write(f\"{cls} {xywh[0]:.5f} {xywh[1]:.5f} {xywh[2]:.5f} {xywh[3]:.5f}\\n\") # write label\r\n"
},
{
"path": "yolo/data/datasets/VOC.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# PASCAL VOC dataset http://host.robots.ox.ac.uk/pascal/VOC by University of Oxford\r\n# Example usage: python train.py --data VOC.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ VOC β downloads here (2.8 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/VOC\r\ntrain: # train images (relative to 'path') 16551 images\r\n - images/train2012\r\n - images/train2007\r\n - images/val2012\r\n - images/val2007\r\nval: # val images (relative to 'path') 4952 images\r\n - images/test2007\r\ntest: # test images (optional)\r\n - images/test2007\r\n\r\n# Classes\r\nnames:\r\n 0: aeroplane\r\n 1: bicycle\r\n 2: bird\r\n 3: boat\r\n 4: bottle\r\n 5: bus\r\n 6: car\r\n 7: cat\r\n 8: chair\r\n 9: cow\r\n 10: diningtable\r\n 11: dog\r\n 12: horse\r\n 13: motorbike\r\n 14: person\r\n 15: pottedplant\r\n 16: sheep\r\n 17: sofa\r\n 18: train\r\n 19: tvmonitor\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n import xml.etree.ElementTree as ET\r\n\r\n from tqdm import tqdm\r\n from utils.general import download, Path\r\n\r\n\r\n def convert_label(path, lb_path, year, image_id):\r\n def convert_box(size, box):\r\n dw, dh = 1. / size[0], 1. / size[1]\r\n x, y, w, h = (box[0] + box[1]) / 2.0 - 1, (box[2] + box[3]) / 2.0 - 1, box[1] - box[0], box[3] - box[2]\r\n return x * dw, y * dh, w * dw, h * dh\r\n\r\n in_file = open(path / f'VOC{year}/Annotations/{image_id}.xml')\r\n out_file = open(lb_path, 'w')\r\n tree = ET.parse(in_file)\r\n root = tree.getroot()\r\n size = root.find('size')\r\n w = int(size.find('width').text)\r\n h = int(size.find('height').text)\r\n\r\n names = list(yaml['names'].values()) # names list\r\n for obj in root.iter('object'):\r\n cls = obj.find('name').text\r\n if cls in names and int(obj.find('difficult').text) != 1:\r\n xmlbox = obj.find('bndbox')\r\n bb = convert_box((w, h), [float(xmlbox.find(x).text) for x in ('xmin', 'xmax', 'ymin', 'ymax')])\r\n cls_id = names.index(cls) # class id\r\n out_file.write(\" \".join([str(a) for a in (cls_id, *bb)]) + '\\n')\r\n\r\n\r\n # Download\r\n dir = Path(yaml['path']) # dataset root dir\r\n url = 'https://github.com/ultralytics/yolov5/releases/download/v1.0/'\r\n urls = [f'{url}VOCtrainval_06-Nov-2007.zip', # 446MB, 5012 images\r\n f'{url}VOCtest_06-Nov-2007.zip', # 438MB, 4953 images\r\n f'{url}VOCtrainval_11-May-2012.zip'] # 1.95GB, 17126 images\r\n download(urls, dir=dir / 'images', delete=False, curl=True, threads=3)\r\n\r\n # Convert\r\n path = dir / 'images/VOCdevkit'\r\n for year, image_set in ('2012', 'train'), ('2012', 'val'), ('2007', 'train'), ('2007', 'val'), ('2007', 'test'):\r\n imgs_path = dir / 'images' / f'{image_set}{year}'\r\n lbs_path = dir / 'labels' / f'{image_set}{year}'\r\n imgs_path.mkdir(exist_ok=True, parents=True)\r\n lbs_path.mkdir(exist_ok=True, parents=True)\r\n\r\n with open(path / f'VOC{year}/ImageSets/Main/{image_set}.txt') as f:\r\n image_ids = f.read().strip().split()\r\n for id in tqdm(image_ids, desc=f'{image_set}{year}'):\r\n f = path / f'VOC{year}/JPEGImages/{id}.jpg' # old img path\r\n lb_path = (lbs_path / f.name).with_suffix('.txt') # new label path\r\n f.rename(imgs_path / f.name) # move image\r\n convert_label(path, lb_path, year, id) # convert labels to YOLO format\r\n"
},
{
"path": "yolo/data/datasets/VisDrone.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# VisDrone2019-DET dataset https://github.com/VisDrone/VisDrone-Dataset by Tianjin University\r\n# Example usage: python train.py --data VisDrone.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ VisDrone β downloads here (2.3 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/VisDrone # dataset root dir\r\ntrain: VisDrone2019-DET-train/images # train images (relative to 'path') 6471 images\r\nval: VisDrone2019-DET-val/images # val images (relative to 'path') 548 images\r\ntest: VisDrone2019-DET-test-dev/images # test images (optional) 1610 images\r\n\r\n# Classes\r\nnames:\r\n 0: pedestrian\r\n 1: people\r\n 2: bicycle\r\n 3: car\r\n 4: van\r\n 5: truck\r\n 6: tricycle\r\n 7: awning-tricycle\r\n 8: bus\r\n 9: motor\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n from utils.general import download, os, Path\r\n\r\n def visdrone2yolo(dir):\r\n from PIL import Image\r\n from tqdm import tqdm\r\n\r\n def convert_box(size, box):\r\n # Convert VisDrone box to YOLO xywh box\r\n dw = 1. / size[0]\r\n dh = 1. / size[1]\r\n return (box[0] + box[2] / 2) * dw, (box[1] + box[3] / 2) * dh, box[2] * dw, box[3] * dh\r\n\r\n (dir / 'labels').mkdir(parents=True, exist_ok=True) # make labels directory\r\n pbar = tqdm((dir / 'annotations').glob('*.txt'), desc=f'Converting {dir}')\r\n for f in pbar:\r\n img_size = Image.open((dir / 'images' / f.name).with_suffix('.jpg')).size\r\n lines = []\r\n with open(f, 'r') as file: # read annotation.txt\r\n for row in [x.split(',') for x in file.read().strip().splitlines()]:\r\n if row[4] == '0': # VisDrone 'ignored regions' class 0\r\n continue\r\n cls = int(row[5]) - 1\r\n box = convert_box(img_size, tuple(map(int, row[:4])))\r\n lines.append(f\"{cls} {' '.join(f'{x:.6f}' for x in box)}\\n\")\r\n with open(str(f).replace(os.sep + 'annotations' + os.sep, os.sep + 'labels' + os.sep), 'w') as fl:\r\n fl.writelines(lines) # write label.txt\r\n\r\n\r\n # Download\r\n dir = Path(yaml['path']) # dataset root dir\r\n urls = ['https://github.com/ultralytics/yolov5/releases/download/v1.0/VisDrone2019-DET-train.zip',\r\n 'https://github.com/ultralytics/yolov5/releases/download/v1.0/VisDrone2019-DET-val.zip',\r\n 'https://github.com/ultralytics/yolov5/releases/download/v1.0/VisDrone2019-DET-test-dev.zip',\r\n 'https://github.com/ultralytics/yolov5/releases/download/v1.0/VisDrone2019-DET-test-challenge.zip']\r\n download(urls, dir=dir, curl=True, threads=4)\r\n\r\n # Convert\r\n for d in 'VisDrone2019-DET-train', 'VisDrone2019-DET-val', 'VisDrone2019-DET-test-dev':\r\n visdrone2yolo(dir / d) # convert VisDrone annotations to YOLO labels\r\n"
},
{
"path": "yolo/data/datasets/coco.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# COCO 2017 dataset http://cocodataset.org by Microsoft\r\n# Example usage: python train.py --data coco.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ coco β downloads here (20.1 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/coco # dataset root dir\r\ntrain: train2017.txt # train images (relative to 'path') 118287 images\r\nval: val2017.txt # val images (relative to 'path') 5000 images\r\ntest: test-dev2017.txt # 20288 of 40670 images, submit to https://competitions.codalab.org/competitions/20794\r\n\r\n# Classes\r\nnames:\r\n 0: person\r\n 1: bicycle\r\n 2: car\r\n 3: motorcycle\r\n 4: airplane\r\n 5: bus\r\n 6: train\r\n 7: truck\r\n 8: boat\r\n 9: traffic light\r\n 10: fire hydrant\r\n 11: stop sign\r\n 12: parking meter\r\n 13: bench\r\n 14: bird\r\n 15: cat\r\n 16: dog\r\n 17: horse\r\n 18: sheep\r\n 19: cow\r\n 20: elephant\r\n 21: bear\r\n 22: zebra\r\n 23: giraffe\r\n 24: backpack\r\n 25: umbrella\r\n 26: handbag\r\n 27: tie\r\n 28: suitcase\r\n 29: frisbee\r\n 30: skis\r\n 31: snowboard\r\n 32: sports ball\r\n 33: kite\r\n 34: baseball bat\r\n 35: baseball glove\r\n 36: skateboard\r\n 37: surfboard\r\n 38: tennis racket\r\n 39: bottle\r\n 40: wine glass\r\n 41: cup\r\n 42: fork\r\n 43: knife\r\n 44: spoon\r\n 45: bowl\r\n 46: banana\r\n 47: apple\r\n 48: sandwich\r\n 49: orange\r\n 50: broccoli\r\n 51: carrot\r\n 52: hot dog\r\n 53: pizza\r\n 54: donut\r\n 55: cake\r\n 56: chair\r\n 57: couch\r\n 58: potted plant\r\n 59: bed\r\n 60: dining table\r\n 61: toilet\r\n 62: tv\r\n 63: laptop\r\n 64: mouse\r\n 65: remote\r\n 66: keyboard\r\n 67: cell phone\r\n 68: microwave\r\n 69: oven\r\n 70: toaster\r\n 71: sink\r\n 72: refrigerator\r\n 73: book\r\n 74: clock\r\n 75: vase\r\n 76: scissors\r\n 77: teddy bear\r\n 78: hair drier\r\n 79: toothbrush\r\n\r\n\r\n# Download script/URL (optional)\r\ndownload: |\r\n from utils.general import download, Path\r\n # Download labels\r\n segments = True # segment or box labels\r\n dir = Path(yaml['path']) # dataset root dir\r\n url = 'https://github.com/ultralytics/yolov5/releases/download/v1.0/'\r\n urls = [url + ('coco2017labels-segments.zip' if segments else 'coco2017labels.zip')] # labels\r\n download(urls, dir=dir.parent)\r\n # Download data\r\n urls = ['http://images.cocodataset.org/zips/train2017.zip', # 19G, 118k images\r\n 'http://images.cocodataset.org/zips/val2017.zip', # 1G, 5k images\r\n 'http://images.cocodataset.org/zips/test2017.zip'] # 7G, 41k images (optional)\r\n download(urls, dir=dir / 'images', threads=3)"
},
{
"path": "yolo/data/datasets/coco128-seg.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# COCO128-seg dataset https://www.kaggle.com/ultralytics/coco128 (first 128 images from COCO train2017) by Ultralytics\r\n# Example usage: python train.py --data coco128.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ coco128-seg β downloads here (7 MB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/coco128-seg # dataset root dir\r\ntrain: images/train2017 # train images (relative to 'path') 128 images\r\nval: images/train2017 # val images (relative to 'path') 128 images\r\ntest: # test images (optional)\r\n\r\n# Classes\r\nnames:\r\n 0: person\r\n 1: bicycle\r\n 2: car\r\n 3: motorcycle\r\n 4: airplane\r\n 5: bus\r\n 6: train\r\n 7: truck\r\n 8: boat\r\n 9: traffic light\r\n 10: fire hydrant\r\n 11: stop sign\r\n 12: parking meter\r\n 13: bench\r\n 14: bird\r\n 15: cat\r\n 16: dog\r\n 17: horse\r\n 18: sheep\r\n 19: cow\r\n 20: elephant\r\n 21: bear\r\n 22: zebra\r\n 23: giraffe\r\n 24: backpack\r\n 25: umbrella\r\n 26: handbag\r\n 27: tie\r\n 28: suitcase\r\n 29: frisbee\r\n 30: skis\r\n 31: snowboard\r\n 32: sports ball\r\n 33: kite\r\n 34: baseball bat\r\n 35: baseball glove\r\n 36: skateboard\r\n 37: surfboard\r\n 38: tennis racket\r\n 39: bottle\r\n 40: wine glass\r\n 41: cup\r\n 42: fork\r\n 43: knife\r\n 44: spoon\r\n 45: bowl\r\n 46: banana\r\n 47: apple\r\n 48: sandwich\r\n 49: orange\r\n 50: broccoli\r\n 51: carrot\r\n 52: hot dog\r\n 53: pizza\r\n 54: donut\r\n 55: cake\r\n 56: chair\r\n 57: couch\r\n 58: potted plant\r\n 59: bed\r\n 60: dining table\r\n 61: toilet\r\n 62: tv\r\n 63: laptop\r\n 64: mouse\r\n 65: remote\r\n 66: keyboard\r\n 67: cell phone\r\n 68: microwave\r\n 69: oven\r\n 70: toaster\r\n 71: sink\r\n 72: refrigerator\r\n 73: book\r\n 74: clock\r\n 75: vase\r\n 76: scissors\r\n 77: teddy bear\r\n 78: hair drier\r\n 79: toothbrush\r\n\r\n\r\n# Download script/URL (optional)\r\ndownload: https://ultralytics.com/assets/coco128-seg.zip"
},
{
"path": "yolo/data/datasets/coco128.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# COCO128 dataset https://www.kaggle.com/ultralytics/coco128 (first 128 images from COCO train2017) by Ultralytics\r\n# Example usage: python train.py --data coco128.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ coco128 β downloads here (7 MB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/coco128 # dataset root dir\r\ntrain: images/train2017 # train images (relative to 'path') 128 images\r\nval: images/train2017 # val images (relative to 'path') 128 images\r\ntest: # test images (optional)\r\n\r\n# Classes\r\nnames:\r\n 0: person\r\n 1: bicycle\r\n 2: car\r\n 3: motorcycle\r\n 4: airplane\r\n 5: bus\r\n 6: train\r\n 7: truck\r\n 8: boat\r\n 9: traffic light\r\n 10: fire hydrant\r\n 11: stop sign\r\n 12: parking meter\r\n 13: bench\r\n 14: bird\r\n 15: cat\r\n 16: dog\r\n 17: horse\r\n 18: sheep\r\n 19: cow\r\n 20: elephant\r\n 21: bear\r\n 22: zebra\r\n 23: giraffe\r\n 24: backpack\r\n 25: umbrella\r\n 26: handbag\r\n 27: tie\r\n 28: suitcase\r\n 29: frisbee\r\n 30: skis\r\n 31: snowboard\r\n 32: sports ball\r\n 33: kite\r\n 34: baseball bat\r\n 35: baseball glove\r\n 36: skateboard\r\n 37: surfboard\r\n 38: tennis racket\r\n 39: bottle\r\n 40: wine glass\r\n 41: cup\r\n 42: fork\r\n 43: knife\r\n 44: spoon\r\n 45: bowl\r\n 46: banana\r\n 47: apple\r\n 48: sandwich\r\n 49: orange\r\n 50: broccoli\r\n 51: carrot\r\n 52: hot dog\r\n 53: pizza\r\n 54: donut\r\n 55: cake\r\n 56: chair\r\n 57: couch\r\n 58: potted plant\r\n 59: bed\r\n 60: dining table\r\n 61: toilet\r\n 62: tv\r\n 63: laptop\r\n 64: mouse\r\n 65: remote\r\n 66: keyboard\r\n 67: cell phone\r\n 68: microwave\r\n 69: oven\r\n 70: toaster\r\n 71: sink\r\n 72: refrigerator\r\n 73: book\r\n 74: clock\r\n 75: vase\r\n 76: scissors\r\n 77: teddy bear\r\n 78: hair drier\r\n 79: toothbrush\r\n\r\n\r\n# Download script/URL (optional)\r\ndownload: https://ultralytics.com/assets/coco128.zip"
},
{
"path": "yolo/data/datasets/xView.yaml",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n# DIUx xView 2018 Challenge https://challenge.xviewdataset.org by U.S. National Geospatial-Intelligence Agency (NGA)\r\n# -------- DOWNLOAD DATA MANUALLY and jar xf val_images.zip to 'datasets/xView' before running train command! --------\r\n# Example usage: python train.py --data xView.yaml\r\n# parent\r\n# βββ yolov5\r\n# βββ datasets\r\n# βββ xView β downloads here (20.7 GB)\r\n\r\n\r\n# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\r\npath: ../datasets/xView # dataset root dir\r\ntrain: images/autosplit_train.txt # train images (relative to 'path') 90% of 847 train images\r\nval: images/autosplit_val.txt # train images (relative to 'path') 10% of 847 train images\r\n\r\n# Classes\r\nnames:\r\n 0: Fixed-wing Aircraft\r\n 1: Small Aircraft\r\n 2: Cargo Plane\r\n 3: Helicopter\r\n 4: Passenger Vehicle\r\n 5: Small Car\r\n 6: Bus\r\n 7: Pickup Truck\r\n 8: Utility Truck\r\n 9: Truck\r\n 10: Cargo Truck\r\n 11: Truck w/Box\r\n 12: Truck Tractor\r\n 13: Trailer\r\n 14: Truck w/Flatbed\r\n 15: Truck w/Liquid\r\n 16: Crane Truck\r\n 17: Railway Vehicle\r\n 18: Passenger Car\r\n 19: Cargo Car\r\n 20: Flat Car\r\n 21: Tank car\r\n 22: Locomotive\r\n 23: Maritime Vessel\r\n 24: Motorboat\r\n 25: Sailboat\r\n 26: Tugboat\r\n 27: Barge\r\n 28: Fishing Vessel\r\n 29: Ferry\r\n 30: Yacht\r\n 31: Container Ship\r\n 32: Oil Tanker\r\n 33: Engineering Vehicle\r\n 34: Tower crane\r\n 35: Container Crane\r\n 36: Reach Stacker\r\n 37: Straddle Carrier\r\n 38: Mobile Crane\r\n 39: Dump Truck\r\n 40: Haul Truck\r\n 41: Scraper/Tractor\r\n 42: Front loader/Bulldozer\r\n 43: Excavator\r\n 44: Cement Mixer\r\n 45: Ground Grader\r\n 46: Hut/Tent\r\n 47: Shed\r\n 48: Building\r\n 49: Aircraft Hangar\r\n 50: Damaged Building\r\n 51: Facility\r\n 52: Construction Site\r\n 53: Vehicle Lot\r\n 54: Helipad\r\n 55: Storage Tank\r\n 56: Shipping container lot\r\n 57: Shipping Container\r\n 58: Pylon\r\n 59: Tower\r\n\r\n\r\n# Download script/URL (optional) ---------------------------------------------------------------------------------------\r\ndownload: |\r\n import json\r\n import os\r\n from pathlib import Path\r\n\r\n import numpy as np\r\n from PIL import Image\r\n from tqdm import tqdm\r\n\r\n from utils.dataloaders import autosplit\r\n from utils.general import download, xyxy2xywhn\r\n\r\n\r\n def convert_labels(fname=Path('xView/xView_train.geojson')):\r\n # Convert xView geoJSON labels to YOLO format\r\n path = fname.parent\r\n with open(fname) as f:\r\n print(f'Loading {fname}...')\r\n data = json.load(f)\r\n\r\n # Make dirs\r\n labels = Path(path / 'labels' / 'train')\r\n os.system(f'rm -rf {labels}')\r\n labels.mkdir(parents=True, exist_ok=True)\r\n\r\n # xView classes 11-94 to 0-59\r\n xview_class2index = [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, -1, 3, -1, 4, 5, 6, 7, 8, -1, 9, 10, 11,\r\n 12, 13, 14, 15, -1, -1, 16, 17, 18, 19, 20, 21, 22, -1, 23, 24, 25, -1, 26, 27, -1, 28, -1,\r\n 29, 30, 31, 32, 33, 34, 35, 36, 37, -1, 38, 39, 40, 41, 42, 43, 44, 45, -1, -1, -1, -1, 46,\r\n 47, 48, 49, -1, 50, 51, -1, 52, -1, -1, -1, 53, 54, -1, 55, -1, -1, 56, -1, 57, -1, 58, 59]\r\n\r\n shapes = {}\r\n for feature in tqdm(data['features'], desc=f'Converting {fname}'):\r\n p = feature['properties']\r\n if p['bounds_imcoords']:\r\n id = p['image_id']\r\n file = path / 'train_images' / id\r\n if file.exists(): # 1395.tif missing\r\n try:\r\n box = np.array([int(num) for num in p['bounds_imcoords'].split(\",\")])\r\n assert box.shape[0] == 4, f'incorrect box shape {box.shape[0]}'\r\n cls = p['type_id']\r\n cls = xview_class2index[int(cls)] # xView class to 0-60\r\n assert 59 >= cls >= 0, f'incorrect class index {cls}'\r\n\r\n # Write YOLO label\r\n if id not in shapes:\r\n shapes[id] = Image.open(file).size\r\n box = xyxy2xywhn(box[None].astype(np.float), w=shapes[id][0], h=shapes[id][1], clip=True)\r\n with open((labels / id).with_suffix('.txt'), 'a') as f:\r\n f.write(f\"{cls} {' '.join(f'{x:.6f}' for x in box[0])}\\n\") # write label.txt\r\n except Exception as e:\r\n print(f'WARNING: skipping one label for {file}: {e}')\r\n\r\n\r\n # Download manually from https://challenge.xviewdataset.org\r\n dir = Path(yaml['path']) # dataset root dir\r\n # urls = ['https://d307kc0mrhucc3.cloudfront.net/train_labels.zip', # train labels\r\n # 'https://d307kc0mrhucc3.cloudfront.net/train_images.zip', # 15G, 847 train images\r\n # 'https://d307kc0mrhucc3.cloudfront.net/val_images.zip'] # 5G, 282 val images (no labels)\r\n # download(urls, dir=dir, delete=False)\r\n\r\n # Convert labels\r\n convert_labels(dir / 'xView_train.geojson')\r\n\r\n # Move images\r\n images = Path(dir / 'images')\r\n images.mkdir(parents=True, exist_ok=True)\r\n Path(dir / 'train_images').rename(dir / 'images' / 'train')\r\n Path(dir / 'val_images').rename(dir / 'images' / 'val')\r\n\r\n # Split\r\n autosplit(dir / 'images' / 'train')\r\n"
},
{
"path": "yolo/data/scripts/download_weights.sh",
"content": "#!/bin/bash\r\n# Ultralytics YOLO π, GPL-3.0 license\r\n# Download latest models from https://github.com/ultralytics/yolov5/releases\r\n# Example usage: bash data/scripts/download_weights.sh\r\n# parent\r\n# βββ yolov5\r\n# βββ yolov5s.pt β downloads here\r\n# βββ yolov5m.pt\r\n# βββ ...\r\n\r\npython - < 9) & (shape[1] > 9), f\"image size {shape} <10 pixels\"\r\n assert im.format.lower() in IMG_FORMATS, f\"invalid image format {im.format}\"\r\n if im.format.lower() in (\"jpg\", \"jpeg\"):\r\n with open(im_file, \"rb\") as f:\r\n f.seek(-2, 2)\r\n if f.read() != b\"\\xff\\xd9\": # corrupt JPEG\r\n ImageOps.exif_transpose(Image.open(im_file)).save(im_file, \"JPEG\", subsampling=0, quality=100)\r\n msg = f\"{prefix}WARNING β οΈ {im_file}: corrupt JPEG restored and saved\"\r\n\r\n # verify labels\r\n if os.path.isfile(lb_file):\r\n nf = 1 # label found\r\n with open(lb_file) as f:\r\n lb = [x.split() for x in f.read().strip().splitlines() if len(x)]\r\n if any(len(x) > 6 for x in lb) and (not keypoint): # is segment\r\n classes = np.array([x[0] for x in lb], dtype=np.float32)\r\n segments = [np.array(x[1:], dtype=np.float32).reshape(-1, 2) for x in lb] # (cls, xy1...)\r\n lb = np.concatenate((classes.reshape(-1, 1), segments2boxes(segments)), 1) # (cls, xywh)\r\n lb = np.array(lb, dtype=np.float32)\r\n nl = len(lb)\r\n if nl:\r\n if keypoint:\r\n assert lb.shape[1] == 56, \"labels require 56 columns each\"\r\n assert (lb[:, 5::3] <= 1).all(), \"non-normalized or out of bounds coordinate labels\"\r\n assert (lb[:, 6::3] <= 1).all(), \"non-normalized or out of bounds coordinate labels\"\r\n kpts = np.zeros((lb.shape[0], 39))\r\n for i in range(len(lb)):\r\n kpt = np.delete(lb[i, 5:], np.arange(2, lb.shape[1] - 5,\r\n 3)) # remove the occlusion parameter from the GT\r\n kpts[i] = np.hstack((lb[i, :5], kpt))\r\n lb = kpts\r\n assert lb.shape[1] == 39, \"labels require 39 columns each after removing occlusion parameter\"\r\n else:\r\n assert lb.shape[1] == 5, f\"labels require 5 columns, {lb.shape[1]} columns detected\"\r\n assert (lb >= 0).all(), f\"negative label values {lb[lb < 0]}\"\r\n assert (lb[:, 1:] <=\r\n 1).all(), f\"non-normalized or out of bounds coordinates {lb[:, 1:][lb[:, 1:] > 1]}\"\r\n _, i = np.unique(lb, axis=0, return_index=True)\r\n if len(i) < nl: # duplicate row check\r\n lb = lb[i] # remove duplicates\r\n if segments:\r\n segments = [segments[x] for x in i]\r\n msg = f\"{prefix}WARNING β οΈ {im_file}: {nl - len(i)} duplicate labels removed\"\r\n else:\r\n ne = 1 # label empty\r\n lb = np.zeros((0, 39), dtype=np.float32) if keypoint else np.zeros((0, 5), dtype=np.float32)\r\n else:\r\n nm = 1 # label missing\r\n lb = np.zeros((0, 39), dtype=np.float32) if keypoint else np.zeros((0, 5), dtype=np.float32)\r\n if keypoint:\r\n keypoints = lb[:, 5:].reshape(-1, 17, 2)\r\n lb = lb[:, :5]\r\n return im_file, lb, shape, segments, keypoints, nm, nf, ne, nc, msg\r\n except Exception as e:\r\n nc = 1\r\n msg = f\"{prefix}WARNING β οΈ {im_file}: ignoring corrupt image/label: {e}\"\r\n return [None, None, None, None, None, nm, nf, ne, nc, msg]\r\n\r\n\r\ndef polygon2mask(imgsz, polygons, color=1, downsample_ratio=1):\r\n \"\"\"\r\n Args:\r\n imgsz (tuple): The image size.\r\n polygons (np.ndarray): [N, M], N is the number of polygons, M is the number of points(Be divided by 2).\r\n color (int): color\r\n downsample_ratio (int): downsample ratio\r\n \"\"\"\r\n mask = np.zeros(imgsz, dtype=np.uint8)\r\n polygons = np.asarray(polygons)\r\n polygons = polygons.astype(np.int32)\r\n shape = polygons.shape\r\n polygons = polygons.reshape(shape[0], -1, 2)\r\n cv2.fillPoly(mask, polygons, color=color)\r\n nh, nw = (imgsz[0] // downsample_ratio, imgsz[1] // downsample_ratio)\r\n # NOTE: fillPoly firstly then resize is trying the keep the same way\r\n # of loss calculation when mask-ratio=1.\r\n mask = cv2.resize(mask, (nw, nh))\r\n return mask\r\n\r\n\r\ndef polygons2masks(imgsz, polygons, color, downsample_ratio=1):\r\n \"\"\"\r\n Args:\r\n imgsz (tuple): The image size.\r\n polygons (list[np.ndarray]): each polygon is [N, M], N is number of polygons, M is number of points (M % 2 = 0)\r\n color (int): color\r\n downsample_ratio (int): downsample ratio\r\n \"\"\"\r\n masks = []\r\n for si in range(len(polygons)):\r\n mask = polygon2mask(imgsz, [polygons[si].reshape(-1)], color, downsample_ratio)\r\n masks.append(mask)\r\n return np.array(masks)\r\n\r\n\r\ndef polygons2masks_overlap(imgsz, segments, downsample_ratio=1):\r\n \"\"\"Return a (640, 640) overlap mask.\"\"\"\r\n masks = np.zeros((imgsz[0] // downsample_ratio, imgsz[1] // downsample_ratio),\r\n dtype=np.int32 if len(segments) > 255 else np.uint8)\r\n areas = []\r\n ms = []\r\n for si in range(len(segments)):\r\n mask = polygon2mask(\r\n imgsz,\r\n [segments[si].reshape(-1)],\r\n downsample_ratio=downsample_ratio,\r\n color=1,\r\n )\r\n ms.append(mask)\r\n areas.append(mask.sum())\r\n areas = np.asarray(areas)\r\n index = np.argsort(-areas)\r\n ms = np.array(ms)[index]\r\n for i in range(len(segments)):\r\n mask = ms[i] * (i + 1)\r\n masks = masks + mask\r\n masks = np.clip(masks, a_min=0, a_max=i + 1)\r\n return masks, index\r\n\r\n\r\ndef check_dataset_yaml(data, autodownload=True):\r\n # Download, check and/or unzip dataset if not found locally\r\n data = check_file(data)\r\n DATASETS_DIR = (Path.cwd() / \"../datasets\").resolve() # TODO: handle global dataset dir\r\n # Download (optional)\r\n extract_dir = ''\r\n if isinstance(data, (str, Path)) and (is_zipfile(data) or is_tarfile(data)):\r\n download(data, dir=f'{DATASETS_DIR}/{Path(data).stem}', unzip=True, delete=False, curl=False, threads=1)\r\n data = next((DATASETS_DIR / Path(data).stem).rglob('*.yaml'))\r\n extract_dir, autodownload = data.parent, False\r\n # Read yaml (optional)\r\n if isinstance(data, (str, Path)):\r\n data = yaml_load(data, append_filename=True) # dictionary\r\n\r\n # Checks\r\n for k in 'train', 'val', 'names':\r\n assert k in data, f\"data.yaml '{k}:' field missing β\"\r\n if isinstance(data['names'], (list, tuple)): # old array format\r\n data['names'] = dict(enumerate(data['names'])) # convert to dict\r\n data['nc'] = len(data['names'])\r\n\r\n # Resolve paths\r\n path = Path(extract_dir or data.get('path') or '') # optional 'path' default to '.'\r\n if not path.is_absolute():\r\n path = (Path.cwd() / path).resolve()\r\n data['path'] = path # download scripts\r\n for k in 'train', 'val', 'test':\r\n if data.get(k): # prepend path\r\n if isinstance(data[k], str):\r\n x = (path / data[k]).resolve()\r\n if not x.exists() and data[k].startswith('../'):\r\n x = (path / data[k][3:]).resolve()\r\n data[k] = str(x)\r\n else:\r\n data[k] = [str((path / x).resolve()) for x in data[k]]\r\n\r\n # Parse yaml\r\n train, val, test, s = (data.get(x) for x in ('train', 'val', 'test', 'download'))\r\n if val:\r\n val = [Path(x).resolve() for x in (val if isinstance(val, list) else [val])] # val path\r\n if not all(x.exists() for x in val):\r\n LOGGER.info('\\nDataset not found β οΈ, missing paths %s' % [str(x) for x in val if not x.exists()])\r\n if not s or not autodownload:\r\n raise FileNotFoundError('Dataset not found β')\r\n t = time.time()\r\n if s.startswith('http') and s.endswith('.zip'): # URL\r\n f = Path(s).name # filename\r\n LOGGER.info(f'Downloading {s} to {f}...')\r\n torch.hub.download_url_to_file(s, f)\r\n Path(DATASETS_DIR).mkdir(parents=True, exist_ok=True) # create root\r\n unzip_file(f, path=DATASETS_DIR) # unzip\r\n Path(f).unlink() # remove zip\r\n r = None # success\r\n elif s.startswith('bash '): # bash script\r\n LOGGER.info(f'Running {s} ...')\r\n r = os.system(s)\r\n else: # python script\r\n r = exec(s, {'yaml': data}) # return None\r\n dt = f'({round(time.time() - t, 1)}s)'\r\n s = f\"success β
{dt}, saved to {colorstr('bold', DATASETS_DIR)}\" if r in (0, None) else f\"failure {dt} β\"\r\n LOGGER.info(f\"Dataset download {s}\")\r\n check_font('Arial.ttf' if is_ascii(data['names']) else 'Arial.Unicode.ttf', progress=True) # download fonts\r\n return data # dictionary\r\n\r\n\r\ndef check_dataset(dataset: str):\r\n \"\"\"\r\n Check a classification dataset such as Imagenet.\r\n\r\n Copy code\r\n This function takes a `dataset` name as input and returns a dictionary containing information about the dataset.\r\n If the dataset is not found, it attempts to download the dataset from the internet and save it to the local file system.\r\n\r\n Args:\r\n dataset (str): Name of the dataset.\r\n\r\n Returns:\r\n data (dict): A dictionary containing the following keys and values:\r\n 'train': Path object for the directory containing the training set of the dataset\r\n 'val': Path object for the directory containing the validation set of the dataset\r\n 'nc': Number of classes in the dataset\r\n 'names': List of class names in the dataset\r\n \"\"\"\r\n data_dir = (Path.cwd() / \"datasets\" / dataset).resolve()\r\n if not data_dir.is_dir():\r\n LOGGER.info(f'\\nDataset not found β οΈ, missing path {data_dir}, attempting download...')\r\n t = time.time()\r\n if dataset == 'imagenet':\r\n subprocess.run(f\"bash {ROOT / 'data/scripts/get_imagenet.sh'}\", shell=True, check=True)\r\n else:\r\n url = f'https://github.com/ultralytics/yolov5/releases/download/v1.0/{dataset}.zip'\r\n download(url, dir=data_dir.parent)\r\n s = f\"Dataset download success β
({time.time() - t:.1f}s), saved to {colorstr('bold', data_dir)}\\n\"\r\n LOGGER.info(s)\r\n train_set = data_dir / \"train\"\r\n test_set = data_dir / 'test' if (data_dir / 'test').exists() else data_dir / 'val' # data/test or data/val\r\n nc = len([x for x in (data_dir / 'train').glob('*') if x.is_dir()]) # number of classes\r\n names = [x.name for x in (data_dir / 'train').iterdir() if x.is_dir()] # class names list\r\n names = dict(enumerate(sorted(names)))\r\n return {\"train\": train_set, \"val\": test_set, \"nc\": nc, \"names\": names}\r\n"
},
{
"path": "yolo/engine/__init__.py",
"content": ""
},
{
"path": "yolo/engine/exporter.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nExport a YOLOv5 PyTorch model to other formats. TensorFlow exports authored by https://github.com/zldrobit\r\n\r\nFormat | `format=argument` | Model\r\n--- | --- | ---\r\nPyTorch | - | yolov8n.pt\r\nTorchScript | `torchscript` | yolov8n.torchscript\r\nONNX | `onnx` | yolov8n.onnx\r\nOpenVINO | `openvino` | yolov8n_openvino_model/\r\nTensorRT | `engine` | yolov8n.engine\r\nCoreML | `coreml` | yolov8n.mlmodel\r\nTensorFlow SavedModel | `saved_model` | yolov8n_saved_model/\r\nTensorFlow GraphDef | `pb` | yolov8n.pb\r\nTensorFlow Lite | `tflite` | yolov8n.tflite\r\nTensorFlow Edge TPU | `edgetpu` | yolov8n_edgetpu.tflite\r\nTensorFlow.js | `tfjs` | yolov8n_web_model/\r\nPaddlePaddle | `paddle` | yolov8n_paddle_model/\r\n\r\nRequirements:\r\n $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu # CPU\r\n $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime-gpu openvino-dev tensorflow # GPU\r\n\r\nPython:\r\n from ultralytics import YOLO\r\n model = YOLO('yolov8n.yaml')\r\n results = model.export(format='onnx')\r\n\r\nCLI:\r\n $ yolo mode=export model=yolov8n.pt format=onnx\r\n\r\nInference:\r\n $ python detect.py --weights yolov8n.pt # PyTorch\r\n yolov8n.torchscript # TorchScript\r\n yolov8n.onnx # ONNX Runtime or OpenCV DNN with --dnn\r\n yolov8n_openvino_model # OpenVINO\r\n yolov8n.engine # TensorRT\r\n yolov8n.mlmodel # CoreML (macOS-only)\r\n yolov8n_saved_model # TensorFlow SavedModel\r\n yolov8n.pb # TensorFlow GraphDef\r\n yolov8n.tflite # TensorFlow Lite\r\n yolov8n_edgetpu.tflite # TensorFlow Edge TPU\r\n yolov8n_paddle_model # PaddlePaddle\r\n\r\nTensorFlow.js:\r\n $ cd .. && git clone https://github.com/zldrobit/tfjs-yolov5-example.git && cd tfjs-yolov5-example\r\n $ npm install\r\n $ ln -s ../../yolov5/yolov8n_web_model public/yolov8n_web_model\r\n $ npm start\r\n\"\"\"\r\nimport contextlib\r\nimport json\r\nimport os\r\nimport platform\r\nimport re\r\nimport subprocess\r\nimport time\r\nimport warnings\r\nfrom collections import defaultdict\r\nfrom copy import deepcopy\r\nfrom pathlib import Path\r\n\r\nimport hydra\r\nimport numpy as np\r\nimport pandas as pd\r\nimport torch\r\n\r\nimport ultralytics\r\nfrom nn.modules import Detect, Segment\r\nfrom nn.tasks import ClassificationModel, DetectionModel, SegmentationModel\r\nfrom yolo.configs import get_config\r\nfrom yolo.data.dataloaders.stream_loaders import LoadImages\r\nfrom yolo.data.utils import check_dataset\r\nfrom yolo.utils import DEFAULT_CONFIG, LOGGER, callbacks, colorstr, get_default_args, yaml_save\r\nfrom yolo.utils.checks import check_imgsz, check_requirements, check_version, check_yaml\r\nfrom yolo.utils.files import file_size\r\nfrom yolo.utils.ops import Profile\r\nfrom yolo.utils.torch_utils import guess_task_from_head, select_device, smart_inference_mode\r\n\r\nMACOS = platform.system() == 'Darwin' # macOS environment\r\n\r\n\r\ndef export_formats():\r\n # YOLOv5 export formats\r\n x = [\r\n ['PyTorch', '-', '.pt', True, True],\r\n ['TorchScript', 'torchscript', '.torchscript', True, True],\r\n ['ONNX', 'onnx', '.onnx', True, True],\r\n ['OpenVINO', 'openvino', '_openvino_model', True, False],\r\n ['TensorRT', 'engine', '.engine', False, True],\r\n ['CoreML', 'coreml', '.mlmodel', True, False],\r\n ['TensorFlow SavedModel', 'saved_model', '_saved_model', True, True],\r\n ['TensorFlow GraphDef', 'pb', '.pb', True, True],\r\n ['TensorFlow Lite', 'tflite', '.tflite', True, False],\r\n ['TensorFlow Edge TPU', 'edgetpu', '_edgetpu.tflite', False, False],\r\n ['TensorFlow.js', 'tfjs', '_web_model', False, False],\r\n ['PaddlePaddle', 'paddle', '_paddle_model', True, True],]\r\n return pd.DataFrame(x, columns=['Format', 'Argument', 'Suffix', 'CPU', 'GPU'])\r\n\r\n\r\ndef try_export(inner_func):\r\n # YOLOv5 export decorator, i..e @try_export\r\n inner_args = get_default_args(inner_func)\r\n\r\n def outer_func(*args, **kwargs):\r\n prefix = inner_args['prefix']\r\n try:\r\n with Profile() as dt:\r\n f, model = inner_func(*args, **kwargs)\r\n LOGGER.info(f'{prefix} export success β
{dt.t:.1f}s, saved as {f} ({file_size(f):.1f} MB)')\r\n return f, model\r\n except Exception as e:\r\n LOGGER.info(f'{prefix} export failure β {dt.t:.1f}s: {e}')\r\n return None, None\r\n\r\n return outer_func\r\n\r\n\r\nclass Exporter:\r\n \"\"\"\r\n Exporter\r\n\r\n A class for exporting a model.\r\n\r\n Attributes:\r\n args (OmegaConf): Configuration for the exporter.\r\n save_dir (Path): Directory to save results.\r\n \"\"\"\r\n\r\n def __init__(self, config=DEFAULT_CONFIG, overrides=None):\r\n \"\"\"\r\n Initializes the Exporter class.\r\n\r\n Args:\r\n config (str, optional): Path to a configuration file. Defaults to DEFAULT_CONFIG.\r\n overrides (dict, optional): Configuration overrides. Defaults to None.\r\n \"\"\"\r\n if overrides is None:\r\n overrides = {}\r\n self.args = get_config(config, overrides)\r\n self.callbacks = defaultdict(list, {k: [v] for k, v in callbacks.default_callbacks.items()}) # add callbacks\r\n callbacks.add_integration_callbacks(self)\r\n\r\n @smart_inference_mode()\r\n def __call__(self, model=None):\r\n self.run_callbacks(\"on_export_start\")\r\n t = time.time()\r\n format = self.args.format.lower() # to lowercase\r\n fmts = tuple(export_formats()['Argument'][1:]) # available export formats\r\n flags = [x == format for x in fmts]\r\n assert sum(flags), f'ERROR: Invalid format={format}, valid formats are {fmts}'\r\n jit, onnx, xml, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle = flags # export booleans\r\n\r\n # Load PyTorch model\r\n self.device = select_device('cpu' if self.args.device is None else self.args.device)\r\n if self.args.half:\r\n if self.device.type == 'cpu' and not coreml:\r\n LOGGER.info('half=True only compatible with GPU or CoreML export, i.e. use device=0 or format=coreml')\r\n self.args.half = False\r\n assert not self.args.dynamic, '--half not compatible with --dynamic, i.e. use either --half or --dynamic'\r\n\r\n # Checks\r\n # if self.args.batch == model.args['batch_size']: # user has not modified training batch_size\r\n self.args.batch = 1\r\n self.imgsz = check_imgsz(self.args.imgsz, stride=model.stride, min_dim=2) # check image size\r\n if self.args.optimize:\r\n assert self.device.type == 'cpu', '--optimize not compatible with cuda devices, i.e. use --device cpu'\r\n\r\n # Input\r\n im = torch.zeros(self.args.batch, 3, *self.imgsz).to(self.device)\r\n file = Path(getattr(model, 'pt_path', None) or getattr(model, 'yaml_file', None) or model.yaml['yaml_file'])\r\n if file.suffix == '.yaml':\r\n file = Path(file.name)\r\n\r\n # Update model\r\n model = deepcopy(model).to(self.device)\r\n for p in model.parameters():\r\n p.requires_grad = False\r\n model.eval()\r\n model = model.fuse()\r\n for k, m in model.named_modules():\r\n if isinstance(m, (Detect, Segment)):\r\n m.dynamic = self.args.dynamic\r\n m.export = True\r\n\r\n y = None\r\n for _ in range(2):\r\n y = model(im) # dry runs\r\n if self.args.half and not coreml:\r\n im, model = im.half(), model.half() # to FP16\r\n shape = tuple((y[0] if isinstance(y, tuple) else y).shape) # model output shape\r\n LOGGER.info(\r\n f\"\\n{colorstr('PyTorch:')} starting from {file} with output shape {shape} ({file_size(file):.1f} MB)\")\r\n\r\n # Warnings\r\n warnings.filterwarnings('ignore', category=torch.jit.TracerWarning) # suppress TracerWarning\r\n warnings.filterwarnings('ignore', category=UserWarning) # suppress shape prim::Constant missing ONNX warning\r\n warnings.filterwarnings('ignore', category=DeprecationWarning) # suppress CoreML np.bool deprecation warning\r\n\r\n # Assign\r\n self.im = im\r\n self.model = model\r\n self.file = file\r\n self.output_shape = tuple(y.shape) if isinstance(y, torch.Tensor) else (x.shape for x in y)\r\n self.metadata = {'stride': int(max(model.stride)), 'names': model.names} # model metadata\r\n self.pretty_name = self.file.stem.replace('yolo', 'YOLO')\r\n\r\n # Exports\r\n f = [''] * len(fmts) # exported filenames\r\n if jit: # TorchScript\r\n f[0], _ = self._export_torchscript()\r\n if engine: # TensorRT required before ONNX\r\n f[1], _ = self._export_engine()\r\n if onnx or xml: # OpenVINO requires ONNX\r\n f[2], _ = self._export_onnx()\r\n if xml: # OpenVINO\r\n f[3], _ = self._export_openvino()\r\n if coreml: # CoreML\r\n f[4], _ = self._export_coreml()\r\n if any((saved_model, pb, tflite, edgetpu, tfjs)): # TensorFlow formats\r\n raise NotImplementedError('YOLOv8 TensorFlow export support is still under development. '\r\n 'Please consider contributing to the effort if you have TF expertise. Thank you!')\r\n assert not isinstance(model, ClassificationModel), 'ClassificationModel TF exports not yet supported.'\r\n nms = False\r\n f[5], s_model = self._export_saved_model(nms=nms or self.args.agnostic_nms or tfjs,\r\n agnostic_nms=self.args.agnostic_nms or tfjs)\r\n if pb or tfjs: # pb prerequisite to tfjs\r\n f[6], _ = self._export_pb(s_model)\r\n if tflite or edgetpu:\r\n f[7], _ = self._export_tflite(s_model,\r\n int8=self.args.int8 or edgetpu,\r\n data=self.args.data,\r\n nms=nms,\r\n agnostic_nms=self.args.agnostic_nms)\r\n if edgetpu:\r\n f[8], _ = self._export_edgetpu()\r\n self._add_tflite_metadata(f[8] or f[7], num_outputs=len(s_model.outputs))\r\n if tfjs:\r\n f[9], _ = self._export_tfjs()\r\n if paddle: # PaddlePaddle\r\n f[10], _ = self._export_paddle()\r\n\r\n # Finish\r\n f = [str(x) for x in f if x] # filter out '' and None\r\n if any(f):\r\n task = guess_task_from_head(model.yaml[\"head\"][-1][-2])\r\n s = \"-WARNING β οΈ not yet supported for YOLOv8 exported models\"\r\n LOGGER.info(f'\\nExport complete ({time.time() - t:.1f}s)'\r\n f\"\\nResults saved to {colorstr('bold', file.parent.resolve())}\"\r\n f\"\\nPredict: yolo task={task} mode=predict model={f[-1]} {s}\"\r\n f\"\\nValidate: yolo task={task} mode=val model={f[-1]} {s}\"\r\n f\"\\nVisualize: https://netron.app\")\r\n\r\n self.run_callbacks(\"on_export_end\")\r\n return f # return list of exported files/dirs\r\n\r\n @try_export\r\n def _export_torchscript(self, prefix=colorstr('TorchScript:')):\r\n # YOLOv8 TorchScript model export\r\n LOGGER.info(f'\\n{prefix} starting export with torch {torch.__version__}...')\r\n f = self.file.with_suffix('.torchscript')\r\n\r\n ts = torch.jit.trace(self.model, self.im, strict=False)\r\n d = {\"shape\": self.im.shape, \"stride\": int(max(self.model.stride)), \"names\": self.model.names}\r\n extra_files = {'config.txt': json.dumps(d)} # torch._C.ExtraFilesMap()\r\n if self.args.optimize: # https://pytorch.org/tutorials/recipes/mobile_interpreter.html\r\n LOGGER.info(f'{prefix} optimizing for mobile...')\r\n from torch.utils.mobile_optimizer import optimize_for_mobile\r\n optimize_for_mobile(ts)._save_for_lite_interpreter(str(f), _extra_files=extra_files)\r\n else:\r\n ts.save(str(f), _extra_files=extra_files)\r\n return f, None\r\n\r\n @try_export\r\n def _export_onnx(self, prefix=colorstr('ONNX:')):\r\n # YOLOv8 ONNX export\r\n check_requirements('onnx>=1.12.0')\r\n import onnx # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with onnx {onnx.__version__}...')\r\n f = str(self.file.with_suffix('.onnx'))\r\n\r\n output_names = ['output0', 'output1'] if isinstance(self.model, SegmentationModel) else ['output0']\r\n dynamic = self.args.dynamic\r\n if dynamic:\r\n dynamic = {'images': {0: 'batch', 2: 'height', 3: 'width'}} # shape(1,3,640,640)\r\n if isinstance(self.model, SegmentationModel):\r\n dynamic['output0'] = {0: 'batch', 1: 'anchors'} # shape(1,25200,85)\r\n dynamic['output1'] = {0: 'batch', 2: 'mask_height', 3: 'mask_width'} # shape(1,32,160,160)\r\n elif isinstance(self.model, DetectionModel):\r\n dynamic['output0'] = {0: 'batch', 1: 'anchors'} # shape(1,25200,85)\r\n\r\n torch.onnx.export(\r\n self.model.cpu() if dynamic else self.model, # --dynamic only compatible with cpu\r\n self.im.cpu() if dynamic else self.im,\r\n f,\r\n verbose=False,\r\n opset_version=self.args.opset,\r\n do_constant_folding=True, # WARNING: DNN inference with torch>=1.12 may require do_constant_folding=False\r\n input_names=['images'],\r\n output_names=output_names,\r\n dynamic_axes=dynamic or None)\r\n\r\n # Checks\r\n model_onnx = onnx.load(f) # load onnx model\r\n onnx.checker.check_model(model_onnx) # check onnx model\r\n\r\n # Metadata\r\n d = {'stride': int(max(self.model.stride)), 'names': self.model.names}\r\n for k, v in d.items():\r\n meta = model_onnx.metadata_props.add()\r\n meta.key, meta.value = k, str(v)\r\n onnx.save(model_onnx, f)\r\n\r\n # Simplify\r\n if self.args.simplify:\r\n try:\r\n check_requirements('onnxsim')\r\n import onnxsim\r\n\r\n LOGGER.info(f'{prefix} simplifying with onnx-simplifier {onnxsim.__version__}...')\r\n subprocess.run(f'onnxsim {f} {f}', shell=True)\r\n except Exception as e:\r\n LOGGER.info(f'{prefix} simplifier failure: {e}')\r\n return f, model_onnx\r\n\r\n @try_export\r\n def _export_openvino(self, prefix=colorstr('OpenVINO:')):\r\n # YOLOv8 OpenVINO export\r\n check_requirements('openvino-dev') # requires openvino-dev: https://pypi.org/project/openvino-dev/\r\n import openvino.inference_engine as ie # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with openvino {ie.__version__}...')\r\n f = str(self.file).replace(self.file.suffix, f'_openvino_model{os.sep}')\r\n f_onnx = self.file.with_suffix('.onnx')\r\n\r\n cmd = f\"mo --input_model {f_onnx} --output_dir {f} --data_type {'FP16' if self.args.half else 'FP32'}\"\r\n subprocess.run(cmd.split(), check=True, env=os.environ) # export\r\n yaml_save(Path(f) / self.file.with_suffix('.yaml').name, self.metadata) # add metadata.yaml\r\n return f, None\r\n\r\n @try_export\r\n def _export_paddle(self, prefix=colorstr('PaddlePaddle:')):\r\n # YOLOv8 Paddle export\r\n check_requirements(('paddlepaddle', 'x2paddle'))\r\n import x2paddle # noqa\r\n from x2paddle.convert import pytorch2paddle # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with X2Paddle {x2paddle.__version__}...')\r\n f = str(self.file).replace(self.file.suffix, f'_paddle_model{os.sep}')\r\n\r\n pytorch2paddle(module=self.model, save_dir=f, jit_type='trace', input_examples=[self.im]) # export\r\n yaml_save(Path(f) / self.file.with_suffix('.yaml').name, self.metadata) # add metadata.yaml\r\n return f, None\r\n\r\n @try_export\r\n def _export_coreml(self, prefix=colorstr('CoreML:')):\r\n # YOLOv8 CoreML export\r\n check_requirements('coremltools>=6.0')\r\n import coremltools as ct # noqa\r\n\r\n class iOSModel(torch.nn.Module):\r\n # Wrap an Ultralytics YOLO model for iOS export\r\n def __init__(self, model, im):\r\n super().__init__()\r\n b, c, h, w = im.shape # batch, channel, height, width\r\n self.model = model\r\n self.nc = len(model.names) # number of classes\r\n if w == h:\r\n self.normalize = 1.0 / w # scalar\r\n else:\r\n self.normalize = torch.tensor([1.0 / w, 1.0 / h, 1.0 / w, 1.0 / h]) # broadcast (slower, smaller)\r\n\r\n def forward(self, x):\r\n xywh, cls = self.model(x)[0].transpose(0, 1).split((4, self.nc), 1)\r\n return cls, xywh * self.normalize # confidence (3780, 80), coordinates (3780, 4)\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with coremltools {ct.__version__}...')\r\n f = self.file.with_suffix('.mlmodel')\r\n\r\n model = iOSModel(self.model, self.im) if self.args.nms else self.model\r\n ts = torch.jit.trace(model, self.im, strict=False) # TorchScript model\r\n ct_model = ct.convert(ts, inputs=[ct.ImageType('image', shape=self.im.shape, scale=1 / 255, bias=[0, 0, 0])])\r\n bits, mode = (8, 'kmeans_lut') if self.args.int8 else (16, 'linear') if self.args.half else (32, None)\r\n if bits < 32:\r\n if MACOS: # quantization only supported on macOS\r\n ct_model = ct.models.neural_network.quantization_utils.quantize_weights(ct_model, bits, mode)\r\n else:\r\n LOGGER.info(f'{prefix} quantization only supported on macOS, skipping...')\r\n if self.args.nms:\r\n ct_model = self._pipeline_coreml(ct_model)\r\n\r\n ct_model.save(str(f))\r\n return f, ct_model\r\n\r\n @try_export\r\n def _export_engine(self, workspace=4, verbose=False, prefix=colorstr('TensorRT:')):\r\n # YOLOv8 TensorRT export https://developer.nvidia.com/tensorrt\r\n assert self.im.device.type != 'cpu', 'export running on CPU but must be on GPU, i.e. `device==0`'\r\n try:\r\n import tensorrt as trt # noqa\r\n except ImportError:\r\n if platform.system() == 'Linux':\r\n check_requirements('nvidia-tensorrt', cmds='-U --index-url https://pypi.ngc.nvidia.com')\r\n import tensorrt as trt # noqa\r\n\r\n check_version(trt.__version__, '7.0.0', hard=True) # require tensorrt>=8.0.0\r\n self._export_onnx()\r\n onnx = self.file.with_suffix('.onnx')\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with TensorRT {trt.__version__}...')\r\n assert onnx.exists(), f'failed to export ONNX file: {onnx}'\r\n f = self.file.with_suffix('.engine') # TensorRT engine file\r\n logger = trt.Logger(trt.Logger.INFO)\r\n if verbose:\r\n logger.min_severity = trt.Logger.Severity.VERBOSE\r\n\r\n builder = trt.Builder(logger)\r\n config = builder.create_builder_config()\r\n config.max_workspace_size = workspace * 1 << 30\r\n # config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, workspace << 30) # fix TRT 8.4 deprecation notice\r\n\r\n flag = (1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))\r\n network = builder.create_network(flag)\r\n parser = trt.OnnxParser(network, logger)\r\n if not parser.parse_from_file(str(onnx)):\r\n raise RuntimeError(f'failed to load ONNX file: {onnx}')\r\n\r\n inputs = [network.get_input(i) for i in range(network.num_inputs)]\r\n outputs = [network.get_output(i) for i in range(network.num_outputs)]\r\n for inp in inputs:\r\n LOGGER.info(f'{prefix} input \"{inp.name}\" with shape{inp.shape} {inp.dtype}')\r\n for out in outputs:\r\n LOGGER.info(f'{prefix} output \"{out.name}\" with shape{out.shape} {out.dtype}')\r\n\r\n if self.args.dynamic:\r\n shape = self.im.shape\r\n if shape[0] <= 1:\r\n LOGGER.warning(f\"{prefix} WARNING β οΈ --dynamic model requires maximum --batch-size argument\")\r\n profile = builder.create_optimization_profile()\r\n for inp in inputs:\r\n profile.set_shape(inp.name, (1, *shape[1:]), (max(1, shape[0] // 2), *shape[1:]), shape)\r\n config.add_optimization_profile(profile)\r\n\r\n LOGGER.info(\r\n f'{prefix} building FP{16 if builder.platform_has_fast_fp16 and self.args.half else 32} engine as {f}')\r\n if builder.platform_has_fast_fp16 and self.args.half:\r\n config.set_flag(trt.BuilderFlag.FP16)\r\n with builder.build_engine(network, config) as engine, open(f, 'wb') as t:\r\n t.write(engine.serialize())\r\n return f, None\r\n\r\n @try_export\r\n def _export_saved_model(self,\r\n nms=False,\r\n agnostic_nms=False,\r\n topk_per_class=100,\r\n topk_all=100,\r\n iou_thres=0.45,\r\n conf_thres=0.25,\r\n prefix=colorstr('TensorFlow SavedModel:')):\r\n\r\n # YOLOv8 TensorFlow SavedModel export\r\n try:\r\n import tensorflow as tf # noqa\r\n except ImportError:\r\n check_requirements(f\"tensorflow{'' if torch.cuda.is_available() else '-macos' if MACOS else '-cpu'}\")\r\n import tensorflow as tf # noqa\r\n check_requirements((\"onnx\", \"onnx2tf\", \"sng4onnx\", \"onnxsim\", \"onnx_graphsurgeon\"),\r\n cmds=\"--extra-index-url https://pypi.ngc.nvidia.com \")\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with tensorflow {tf.__version__}...')\r\n f = str(self.file).replace(self.file.suffix, '_saved_model')\r\n\r\n # Export to ONNX\r\n self._export_onnx()\r\n onnx = self.file.with_suffix('.onnx')\r\n\r\n # Export to TF SavedModel\r\n subprocess.run(f'onnx2tf -i {onnx} --output_signaturedefs -o {f}', shell=True)\r\n\r\n # Load saved_model\r\n keras_model = tf.saved_model.load(f, tags=None, options=None)\r\n\r\n return f, keras_model\r\n\r\n @try_export\r\n def _export_saved_model_OLD(self,\r\n nms=False,\r\n agnostic_nms=False,\r\n topk_per_class=100,\r\n topk_all=100,\r\n iou_thres=0.45,\r\n conf_thres=0.25,\r\n prefix=colorstr('TensorFlow SavedModel:')):\r\n # YOLOv8 TensorFlow SavedModel export\r\n try:\r\n import tensorflow as tf # noqa\r\n except ImportError:\r\n check_requirements(f\"tensorflow{'' if torch.cuda.is_available() else '-macos' if MACOS else '-cpu'}\")\r\n import tensorflow as tf # noqa\r\n # from models.tf import TFModel\r\n from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2 # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with tensorflow {tf.__version__}...')\r\n f = str(self.file).replace(self.file.suffix, '_saved_model')\r\n batch_size, ch, *imgsz = list(self.im.shape) # BCHW\r\n\r\n tf_models = None # TODO: no TF modules available\r\n tf_model = tf_models.TFModel(cfg=self.model.yaml, model=self.model.cpu(), nc=self.model.nc, imgsz=imgsz)\r\n im = tf.zeros((batch_size, *imgsz, ch)) # BHWC order for TensorFlow\r\n _ = tf_model.predict(im, nms, agnostic_nms, topk_per_class, topk_all, iou_thres, conf_thres)\r\n inputs = tf.keras.Input(shape=(*imgsz, ch), batch_size=None if self.args.dynamic else batch_size)\r\n outputs = tf_model.predict(inputs, nms, agnostic_nms, topk_per_class, topk_all, iou_thres, conf_thres)\r\n keras_model = tf.keras.Model(inputs=inputs, outputs=outputs)\r\n keras_model.trainable = False\r\n keras_model.summary()\r\n if self.args.keras:\r\n keras_model.save(f, save_format='tf')\r\n else:\r\n spec = tf.TensorSpec(keras_model.inputs[0].shape, keras_model.inputs[0].dtype)\r\n m = tf.function(lambda x: keras_model(x)) # full model\r\n m = m.get_concrete_function(spec)\r\n frozen_func = convert_variables_to_constants_v2(m)\r\n tfm = tf.Module()\r\n tfm.__call__ = tf.function(lambda x: frozen_func(x)[:4] if nms else frozen_func(x), [spec])\r\n tfm.__call__(im)\r\n tf.saved_model.save(tfm,\r\n f,\r\n options=tf.saved_model.SaveOptions(experimental_custom_gradients=False)\r\n if check_version(tf.__version__, '2.6') else tf.saved_model.SaveOptions())\r\n return f, keras_model\r\n\r\n @try_export\r\n def _export_pb(self, keras_model, file, prefix=colorstr('TensorFlow GraphDef:')):\r\n # YOLOv8 TensorFlow GraphDef *.pb export https://github.com/leimao/Frozen_Graph_TensorFlow\r\n import tensorflow as tf # noqa\r\n from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2 # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with tensorflow {tf.__version__}...')\r\n f = file.with_suffix('.pb')\r\n\r\n m = tf.function(lambda x: keras_model(x)) # full model\r\n m = m.get_concrete_function(tf.TensorSpec(keras_model.inputs[0].shape, keras_model.inputs[0].dtype))\r\n frozen_func = convert_variables_to_constants_v2(m)\r\n frozen_func.graph.as_graph_def()\r\n tf.io.write_graph(graph_or_graph_def=frozen_func.graph, logdir=str(f.parent), name=f.name, as_text=False)\r\n return f, None\r\n\r\n @try_export\r\n def _export_tflite(self, keras_model, int8, data, nms, agnostic_nms, prefix=colorstr('TensorFlow Lite:')):\r\n # YOLOv8 TensorFlow Lite export\r\n import tensorflow as tf # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with tensorflow {tf.__version__}...')\r\n batch_size, ch, *imgsz = list(self.im.shape) # BCHW\r\n f = str(self.file).replace(self.file.suffix, '-fp16.tflite')\r\n\r\n converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)\r\n converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]\r\n converter.target_spec.supported_types = [tf.float16]\r\n converter.optimizations = [tf.lite.Optimize.DEFAULT]\r\n if int8:\r\n\r\n def representative_dataset_gen(dataset, n_images=100):\r\n # Dataset generator for use with converter.representative_dataset, returns a generator of np arrays\r\n for n, (path, img, im0s, vid_cap, string) in enumerate(dataset):\r\n im = np.transpose(img, [1, 2, 0])\r\n im = np.expand_dims(im, axis=0).astype(np.float32)\r\n im /= 255\r\n yield [im]\r\n if n >= n_images:\r\n break\r\n\r\n dataset = LoadImages(check_dataset(check_yaml(data))['train'], imgsz=imgsz, auto=False)\r\n converter.representative_dataset = lambda: representative_dataset_gen(dataset, n_images=100)\r\n converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\r\n converter.target_spec.supported_types = []\r\n converter.inference_input_type = tf.uint8 # or tf.int8\r\n converter.inference_output_type = tf.uint8 # or tf.int8\r\n converter.experimental_new_quantizer = True\r\n f = str(self.file).replace(self.file.suffix, '-int8.tflite')\r\n if nms or agnostic_nms:\r\n converter.target_spec.supported_ops.append(tf.lite.OpsSet.SELECT_TF_OPS)\r\n\r\n tflite_model = converter.convert()\r\n open(f, \"wb\").write(tflite_model)\r\n return f, None\r\n\r\n @try_export\r\n def _export_edgetpu(self, prefix=colorstr('Edge TPU:')):\r\n # YOLOv8 Edge TPU export https://coral.ai/docs/edgetpu/models-intro/\r\n cmd = 'edgetpu_compiler --version'\r\n help_url = 'https://coral.ai/docs/edgetpu/compiler/'\r\n assert platform.system() == 'Linux', f'export only supported on Linux. See {help_url}'\r\n if subprocess.run(f'{cmd} >/dev/null', shell=True).returncode != 0:\r\n LOGGER.info(f'\\n{prefix} export requires Edge TPU compiler. Attempting install from {help_url}')\r\n sudo = subprocess.run('sudo --version >/dev/null', shell=True).returncode == 0 # sudo installed on system\r\n for c in (\r\n 'curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -',\r\n 'echo \"deb https://packages.cloud.google.com/apt coral-edgetpu-stable main\" | ' # no comma\r\n 'sudo tee /etc/apt/sources.list.d/coral-edgetpu.list',\r\n 'sudo apt-get update',\r\n 'sudo apt-get install edgetpu-compiler'):\r\n subprocess.run(c if sudo else c.replace('sudo ', ''), shell=True, check=True)\r\n ver = subprocess.run(cmd, shell=True, capture_output=True, check=True).stdout.decode().split()[-1]\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with Edge TPU compiler {ver}...')\r\n f = str(self.file).replace(self.file.suffix, '-int8_edgetpu.tflite') # Edge TPU model\r\n f_tfl = str(self.file).replace(self.file.suffix, '-int8.tflite') # TFLite model\r\n\r\n cmd = f\"edgetpu_compiler -s -d -k 10 --out_dir {self.file.parent} {f_tfl}\"\r\n subprocess.run(cmd.split(), check=True)\r\n return f, None\r\n\r\n @try_export\r\n def _export_tfjs(self, prefix=colorstr('TensorFlow.js:')):\r\n # YOLOv8 TensorFlow.js export\r\n check_requirements('tensorflowjs')\r\n import tensorflowjs as tfjs # noqa\r\n\r\n LOGGER.info(f'\\n{prefix} starting export with tensorflowjs {tfjs.__version__}...')\r\n f = str(self.file).replace(self.file.suffix, '_web_model') # js dir\r\n f_pb = self.file.with_suffix('.pb') # *.pb path\r\n f_json = Path(f) / 'model.json' # *.json path\r\n\r\n cmd = f'tensorflowjs_converter --input_format=tf_frozen_model ' \\\r\n f'--output_node_names=Identity,Identity_1,Identity_2,Identity_3 {f_pb} {f}'\r\n subprocess.run(cmd.split())\r\n\r\n with open(f_json, 'w') as j: # sort JSON Identity_* in ascending order\r\n subst = re.sub(\r\n r'{\"outputs\": {\"Identity.?.?\": {\"name\": \"Identity.?.?\"}, '\r\n r'\"Identity.?.?\": {\"name\": \"Identity.?.?\"}, '\r\n r'\"Identity.?.?\": {\"name\": \"Identity.?.?\"}, '\r\n r'\"Identity.?.?\": {\"name\": \"Identity.?.?\"}}}', r'{\"outputs\": {\"Identity\": {\"name\": \"Identity\"}, '\r\n r'\"Identity_1\": {\"name\": \"Identity_1\"}, '\r\n r'\"Identity_2\": {\"name\": \"Identity_2\"}, '\r\n r'\"Identity_3\": {\"name\": \"Identity_3\"}}}', f_json.read_text())\r\n j.write(subst)\r\n return f, None\r\n\r\n def _add_tflite_metadata(self, file, num_outputs):\r\n # Add metadata to *.tflite models per https://www.tensorflow.org/lite/models/convert/metadata\r\n with contextlib.suppress(ImportError):\r\n # check_requirements('tflite_support')\r\n from tflite_support import flatbuffers # noqa\r\n from tflite_support import metadata as _metadata # noqa\r\n from tflite_support import metadata_schema_py_generated as _metadata_fb # noqa\r\n\r\n tmp_file = Path('/tmp/meta.txt')\r\n with open(tmp_file, 'w') as meta_f:\r\n meta_f.write(str(self.metadata))\r\n\r\n model_meta = _metadata_fb.ModelMetadataT()\r\n label_file = _metadata_fb.AssociatedFileT()\r\n label_file.name = tmp_file.name\r\n model_meta.associatedFiles = [label_file]\r\n\r\n subgraph = _metadata_fb.SubGraphMetadataT()\r\n subgraph.inputTensorMetadata = [_metadata_fb.TensorMetadataT()]\r\n subgraph.outputTensorMetadata = [_metadata_fb.TensorMetadataT()] * num_outputs\r\n model_meta.subgraphMetadata = [subgraph]\r\n\r\n b = flatbuffers.Builder(0)\r\n b.Finish(model_meta.Pack(b), _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)\r\n metadata_buf = b.Output()\r\n\r\n populator = _metadata.MetadataPopulator.with_model_file(file)\r\n populator.load_metadata_buffer(metadata_buf)\r\n populator.load_associated_files([str(tmp_file)])\r\n populator.populate()\r\n tmp_file.unlink()\r\n\r\n def _pipeline_coreml(self, model, prefix=colorstr('CoreML Pipeline:')):\r\n # YOLOv8 CoreML pipeline\r\n import coremltools as ct # noqa\r\n\r\n LOGGER.info(f'{prefix} starting pipeline with coremltools {ct.__version__}...')\r\n batch_size, ch, h, w = list(self.im.shape) # BCHW\r\n\r\n # Output shapes\r\n spec = model.get_spec()\r\n out0, out1 = iter(spec.description.output)\r\n if MACOS:\r\n from PIL import Image\r\n img = Image.new('RGB', (w, h)) # img(192 width, 320 height)\r\n # img = torch.zeros((*opt.img_size, 3)).numpy() # img size(320,192,3) iDetection\r\n out = model.predict({'image': img})\r\n out0_shape = out[out0.name].shape\r\n out1_shape = out[out1.name].shape\r\n else: # linux and windows can not run model.predict(), get sizes from pytorch output y\r\n out0_shape = self.output_shape[1], self.output_shape[2] - 5 # (3780, 80)\r\n out1_shape = self.output_shape[1], 4 # (3780, 4)\r\n\r\n # Checks\r\n names = self.metadata['names']\r\n nx, ny = spec.description.input[0].type.imageType.width, spec.description.input[0].type.imageType.height\r\n na, nc = out0_shape\r\n # na, nc = out0.type.multiArrayType.shape # number anchors, classes\r\n assert len(names) == nc, f'{len(names)} names found for nc={nc}' # check\r\n\r\n # Define output shapes (missing)\r\n out0.type.multiArrayType.shape[:] = out0_shape # (3780, 80)\r\n out1.type.multiArrayType.shape[:] = out1_shape # (3780, 4)\r\n # spec.neuralNetwork.preprocessing[0].featureName = '0'\r\n\r\n # Flexible input shapes\r\n # from coremltools.models.neural_network import flexible_shape_utils\r\n # s = [] # shapes\r\n # s.append(flexible_shape_utils.NeuralNetworkImageSize(320, 192))\r\n # s.append(flexible_shape_utils.NeuralNetworkImageSize(640, 384)) # (height, width)\r\n # flexible_shape_utils.add_enumerated_image_sizes(spec, feature_name='image', sizes=s)\r\n # r = flexible_shape_utils.NeuralNetworkImageSizeRange() # shape ranges\r\n # r.add_height_range((192, 640))\r\n # r.add_width_range((192, 640))\r\n # flexible_shape_utils.update_image_size_range(spec, feature_name='image', size_range=r)\r\n\r\n # Print\r\n print(spec.description)\r\n\r\n # Model from spec\r\n model = ct.models.MLModel(spec)\r\n\r\n # 3. Create NMS protobuf\r\n nms_spec = ct.proto.Model_pb2.Model()\r\n nms_spec.specificationVersion = 5\r\n for i in range(2):\r\n decoder_output = model._spec.description.output[i].SerializeToString()\r\n nms_spec.description.input.add()\r\n nms_spec.description.input[i].ParseFromString(decoder_output)\r\n nms_spec.description.output.add()\r\n nms_spec.description.output[i].ParseFromString(decoder_output)\r\n\r\n nms_spec.description.output[0].name = 'confidence'\r\n nms_spec.description.output[1].name = 'coordinates'\r\n\r\n output_sizes = [nc, 4]\r\n for i in range(2):\r\n ma_type = nms_spec.description.output[i].type.multiArrayType\r\n ma_type.shapeRange.sizeRanges.add()\r\n ma_type.shapeRange.sizeRanges[0].lowerBound = 0\r\n ma_type.shapeRange.sizeRanges[0].upperBound = -1\r\n ma_type.shapeRange.sizeRanges.add()\r\n ma_type.shapeRange.sizeRanges[1].lowerBound = output_sizes[i]\r\n ma_type.shapeRange.sizeRanges[1].upperBound = output_sizes[i]\r\n del ma_type.shape[:]\r\n\r\n nms = nms_spec.nonMaximumSuppression\r\n nms.confidenceInputFeatureName = out0.name # 1x507x80\r\n nms.coordinatesInputFeatureName = out1.name # 1x507x4\r\n nms.confidenceOutputFeatureName = 'confidence'\r\n nms.coordinatesOutputFeatureName = 'coordinates'\r\n nms.iouThresholdInputFeatureName = 'iouThreshold'\r\n nms.confidenceThresholdInputFeatureName = 'confidenceThreshold'\r\n nms.iouThreshold = 0.45\r\n nms.confidenceThreshold = 0.25\r\n nms.pickTop.perClass = True\r\n nms.stringClassLabels.vector.extend(names.values())\r\n nms_model = ct.models.MLModel(nms_spec)\r\n\r\n # 4. Pipeline models together\r\n pipeline = ct.models.pipeline.Pipeline(input_features=[('image', ct.models.datatypes.Array(3, ny, nx)),\r\n ('iouThreshold', ct.models.datatypes.Double()),\r\n ('confidenceThreshold', ct.models.datatypes.Double())],\r\n output_features=['confidence', 'coordinates'])\r\n pipeline.add_model(model)\r\n pipeline.add_model(nms_model)\r\n\r\n # Correct datatypes\r\n pipeline.spec.description.input[0].ParseFromString(model._spec.description.input[0].SerializeToString())\r\n pipeline.spec.description.output[0].ParseFromString(nms_model._spec.description.output[0].SerializeToString())\r\n pipeline.spec.description.output[1].ParseFromString(nms_model._spec.description.output[1].SerializeToString())\r\n\r\n # Update metadata\r\n pipeline.spec.specificationVersion = 5\r\n pipeline.spec.description.metadata.versionString = f'Ultralytics YOLOv{__version__}'\r\n pipeline.spec.description.metadata.shortDescription = f'Ultralytics {self.pretty_name} CoreML model'\r\n pipeline.spec.description.metadata.author = 'Ultralytics (https://com)'\r\n pipeline.spec.description.metadata.license = 'GPL-3.0 license (https://com/license)'\r\n pipeline.spec.description.metadata.userDefined.update({\r\n 'IoU threshold': str(nms.iouThreshold),\r\n 'Confidence threshold': str(nms.confidenceThreshold)})\r\n\r\n # Save the model\r\n model = ct.models.MLModel(pipeline.spec)\r\n model.input_description['image'] = 'Input image'\r\n model.input_description['iouThreshold'] = f'(optional) IOU threshold override (default: {nms.iouThreshold})'\r\n model.input_description['confidenceThreshold'] = \\\r\n f'(optional) Confidence threshold override (default: {nms.confidenceThreshold})'\r\n model.output_description['confidence'] = 'Boxes Γ Class confidence (see user-defined metadata \"classes\")'\r\n model.output_description['coordinates'] = 'Boxes Γ [x, y, width, height] (relative to image size)'\r\n LOGGER.info(f'{prefix} pipeline success')\r\n return model\r\n\r\n def run_callbacks(self, event: str):\r\n for callback in self.callbacks.get(event, []):\r\n callback(self)\r\n\r\n\r\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent), config_name=DEFAULT_CONFIG.name)\r\ndef export(cfg):\r\n cfg.model = cfg.model or \"yolov8n.yaml\"\r\n cfg.format = cfg.format or \"torchscript\"\r\n\r\n # exporter = Exporter(cfg)\r\n #\r\n # model = None\r\n # if isinstance(cfg.model, (str, Path)):\r\n # if Path(cfg.model).suffix == '.yaml':\r\n # model = DetectionModel(cfg.model)\r\n # elif Path(cfg.model).suffix == '.pt':\r\n # model = attempt_load_weights(cfg.model, fuse=True)\r\n # else:\r\n # TypeError(f'Unsupported model type {cfg.model}')\r\n # exporter(model=model)\r\n\r\n from ultralytics import YOLO\r\n model = YOLO(cfg.model)\r\n model.export(**cfg)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n \"\"\"\r\n CLI:\r\n yolo mode=export model=yolov8n.yaml format=onnx\r\n \"\"\"\r\n export()\r\n"
},
{
"path": "yolo/engine/model.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom pathlib import Path\r\n\r\nfrom ultralytics import yolo # noqa\r\nfrom nn.tasks import ClassificationModel, DetectionModel, SegmentationModel, attempt_load_one_weight\r\nfrom yolo.configs import get_config\r\nfrom yolo.engine.exporter import Exporter\r\nfrom yolo.utils import DEFAULT_CONFIG, LOGGER, yaml_load\r\nfrom yolo.utils.checks import check_imgsz, check_yaml\r\nfrom yolo.utils.torch_utils import guess_task_from_head, smart_inference_mode\r\n\r\n# Map head to model, trainer, validator, and predictor classes\r\nMODEL_MAP = {\r\n \"classify\": [\r\n ClassificationModel, 'yolo.TYPE.classify.ClassificationTrainer', 'yolo.TYPE.classify.ClassificationValidator',\r\n 'yolo.TYPE.classify.ClassificationPredictor'],\r\n \"detect\": [\r\n DetectionModel, 'yolo.TYPE.detect.DetectionTrainer', 'yolo.TYPE.detect.DetectionValidator',\r\n 'yolo.TYPE.detect.DetectionPredictor'],\r\n \"segment\": [\r\n SegmentationModel, 'yolo.TYPE.segment.SegmentationTrainer', 'yolo.TYPE.segment.SegmentationValidator',\r\n 'yolo.TYPE.segment.SegmentationPredictor']}\r\n\r\n\r\nclass YOLO:\r\n \"\"\"\r\n YOLO\r\n\r\n A python interface which emulates a model-like behaviour by wrapping trainers.\r\n \"\"\"\r\n\r\n def __init__(self, model='yolov8n.yaml', type=\"v8\") -> None:\r\n \"\"\"\r\n > Initializes the YOLO object.\r\n\r\n Args:\r\n model (str, Path): model to load or create\r\n type (str): Type/version of models to use. Defaults to \"v8\".\r\n \"\"\"\r\n self.type = type\r\n self.ModelClass = None # model class\r\n self.TrainerClass = None # trainer class\r\n self.ValidatorClass = None # validator class\r\n self.PredictorClass = None # predictor class\r\n self.model = None # model object\r\n self.trainer = None # trainer object\r\n self.task = None # task type\r\n self.ckpt = None # if loaded from *.pt\r\n self.cfg = None # if loaded from *.yaml\r\n self.ckpt_path = None\r\n self.overrides = {} # overrides for trainer object\r\n\r\n # Load or create new YOLO model\r\n {'.pt': self._load, '.yaml': self._new}[Path(model).suffix](model)\r\n\r\n def __call__(self, source, **kwargs):\r\n return self.predict(source, **kwargs)\r\n\r\n def _new(self, cfg: str, verbose=True):\r\n \"\"\"\r\n > Initializes a new model and infers the task type from the model definitions.\r\n\r\n Args:\r\n cfg (str): model configuration file\r\n verbose (bool): display model info on load\r\n \"\"\"\r\n cfg = check_yaml(cfg) # check YAML\r\n cfg_dict = yaml_load(cfg, append_filename=True) # model dict\r\n self.task = guess_task_from_head(cfg_dict[\"head\"][-1][-2])\r\n self.ModelClass, self.TrainerClass, self.ValidatorClass, self.PredictorClass = \\\r\n self._guess_ops_from_task(self.task)\r\n self.model = self.ModelClass(cfg_dict, verbose=verbose) # initialize\r\n self.cfg = cfg\r\n\r\n def _load(self, weights: str):\r\n \"\"\"\r\n > Initializes a new model and infers the task type from the model head.\r\n\r\n Args:\r\n weights (str): model checkpoint to be loaded\r\n \"\"\"\r\n self.model, self.ckpt = attempt_load_one_weight(weights)\r\n self.ckpt_path = weights\r\n self.task = self.model.args[\"task\"]\r\n self.overrides = self.model.args\r\n self._reset_ckpt_args(self.overrides)\r\n self.ModelClass, self.TrainerClass, self.ValidatorClass, self.PredictorClass = \\\r\n self._guess_ops_from_task(self.task)\r\n\r\n def reset(self):\r\n \"\"\"\r\n > Resets the model modules.\r\n \"\"\"\r\n for m in self.model.modules():\r\n if hasattr(m, 'reset_parameters'):\r\n m.reset_parameters()\r\n for p in self.model.parameters():\r\n p.requires_grad = True\r\n\r\n def info(self, verbose=False):\r\n \"\"\"\r\n > Logs model info.\r\n\r\n Args:\r\n verbose (bool): Controls verbosity.\r\n \"\"\"\r\n self.model.info(verbose=verbose)\r\n\r\n def fuse(self):\r\n self.model.fuse()\r\n\r\n @smart_inference_mode()\r\n def predict(self, source, **kwargs):\r\n \"\"\"\r\n Visualize prediction.\r\n\r\n Args:\r\n source (str): Accepts all source types accepted by yolo\r\n **kwargs : Any other args accepted by the predictors. To see all args check 'configuration' section in docs\r\n \"\"\"\r\n overrides = self.overrides.copy()\r\n overrides[\"conf\"] = 0.25\r\n overrides.update(kwargs)\r\n overrides[\"mode\"] = \"predict\"\r\n overrides[\"save\"] = kwargs.get(\"save\", False) # not save files by default\r\n predictor = self.PredictorClass(overrides=overrides)\r\n\r\n predictor.args.imgsz = check_imgsz(predictor.args.imgsz, min_dim=2) # check image size\r\n predictor.setup(model=self.model, source=source)\r\n return predictor()\r\n\r\n @smart_inference_mode()\r\n def val(self, data=None, **kwargs):\r\n \"\"\"\r\n > Validate a model on a given dataset .\r\n\r\n Args:\r\n data (str): The dataset to validate on. Accepts all formats accepted by yolo\r\n **kwargs : Any other args accepted by the validators. To see all args check 'configuration' section in docs\r\n \"\"\"\r\n overrides = self.overrides.copy()\r\n overrides.update(kwargs)\r\n overrides[\"mode\"] = \"val\"\r\n args = get_config(config=DEFAULT_CONFIG, overrides=overrides)\r\n args.data = data or args.data\r\n args.task = self.task\r\n\r\n validator = self.ValidatorClass(args=args)\r\n validator(model=self.model)\r\n\r\n @smart_inference_mode()\r\n def export(self, **kwargs):\r\n \"\"\"\r\n > Export model.\r\n\r\n Args:\r\n **kwargs : Any other args accepted by the predictors. To see all args check 'configuration' section in docs\r\n \"\"\"\r\n\r\n overrides = self.overrides.copy()\r\n overrides.update(kwargs)\r\n args = get_config(config=DEFAULT_CONFIG, overrides=overrides)\r\n args.task = self.task\r\n\r\n exporter = Exporter(overrides=args)\r\n exporter(model=self.model)\r\n\r\n def train(self, **kwargs):\r\n \"\"\"\r\n > Trains the model on a given dataset.\r\n\r\n Args:\r\n **kwargs (Any): Any number of arguments representing the training configuration. List of all args can be found in 'config' section.\r\n You can pass all arguments as a yaml file in `cfg`. Other args are ignored if `cfg` file is passed\r\n \"\"\"\r\n overrides = self.overrides.copy()\r\n overrides.update(kwargs)\r\n if kwargs.get(\"cfg\"):\r\n LOGGER.info(f\"cfg file passed. Overriding default params with {kwargs['cfg']}.\")\r\n overrides = yaml_load(check_yaml(kwargs[\"cfg\"]), append_filename=True)\r\n overrides[\"task\"] = self.task\r\n overrides[\"mode\"] = \"train\"\r\n if not overrides.get(\"data\"):\r\n raise AttributeError(\"dataset not provided! Please define `data` in config.yaml or pass as an argument.\")\r\n if overrides.get(\"resume\"):\r\n overrides[\"resume\"] = self.ckpt_path\r\n\r\n self.trainer = self.TrainerClass(overrides=overrides)\r\n if not overrides.get(\"resume\"): # manually set model only if not resuming\r\n self.trainer.model = self.trainer.get_model(weights=self.model if self.ckpt else None, cfg=self.model.yaml)\r\n self.model = self.trainer.model\r\n self.trainer.train()\r\n\r\n def to(self, device):\r\n \"\"\"\r\n > Sends the model to the given device.\r\n\r\n Args:\r\n device (str): device\r\n \"\"\"\r\n self.model.to(device)\r\n\r\n def _guess_ops_from_task(self, task):\r\n model_class, train_lit, val_lit, pred_lit = MODEL_MAP[task]\r\n # warning: eval is unsafe. Use with caution\r\n trainer_class = eval(train_lit.replace(\"TYPE\", f\"{self.type}\"))\r\n validator_class = eval(val_lit.replace(\"TYPE\", f\"{self.type}\"))\r\n predictor_class = eval(pred_lit.replace(\"TYPE\", f\"{self.type}\"))\r\n\r\n return model_class, trainer_class, validator_class, predictor_class\r\n\r\n @staticmethod\r\n def _reset_ckpt_args(args):\r\n args.pop(\"device\", None)\r\n args.pop(\"project\", None)\r\n args.pop(\"name\", None)\r\n args.pop(\"batch\", None)\r\n args.pop(\"epochs\", None)\r\n args.pop(\"cache\", None)\r\n args.pop(\"save_json\", None)\r\n"
},
{
"path": "yolo/engine/predictor.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nRun prediction on images, videos, directories, globs, YouTube, webcam, streams, etc.\r\nUsage - sources:\r\n $ yolo task=... mode=predict model=s.pt --source 0 # webcam\r\n img.jpg # image\r\n vid.mp4 # video\r\n screen # screenshot\r\n path/ # directory\r\n list.txt # list of images\r\n list.streams # list of streams\r\n 'path/*.jpg' # glob\r\n 'https://youtu.be/Zgi9g1ksQHc' # YouTube\r\n 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream\r\nUsage - formats:\r\n $ yolo task=... mode=predict --weights yolov8n.pt # PyTorch\r\n yolov8n.torchscript # TorchScript\r\n yolov8n.onnx # ONNX Runtime or OpenCV DNN with --dnn\r\n yolov8n_openvino_model # OpenVINO\r\n yolov8n.engine # TensorRT\r\n yolov8n.mlmodel # CoreML (macOS-only)\r\n yolov8n_saved_model # TensorFlow SavedModel\r\n yolov8n.pb # TensorFlow GraphDef\r\n yolov8n.tflite # TensorFlow Lite\r\n yolov8n_edgetpu.tflite # TensorFlow Edge TPU\r\n yolov8n_paddle_model # PaddlePaddle\r\n \"\"\"\r\nimport platform\r\nfrom collections import defaultdict\r\nfrom pathlib import Path\r\nimport cv2\r\nfrom sort import *\r\nfrom nn.autobackend import AutoBackend\r\nfrom yolo.configs import get_config\r\nfrom yolo.data.dataloaders.stream_loaders import LoadImages, LoadScreenshots, LoadStreams\r\nfrom yolo.data.utils import IMG_FORMATS, VID_FORMATS\r\nfrom yolo.utils import DEFAULT_CONFIG, LOGGER, SETTINGS, callbacks, colorstr, ops\r\nfrom yolo.utils.checks import check_file, check_imgsz, check_imshow\r\nfrom yolo.utils.files import increment_path\r\nfrom yolo.utils.torch_utils import select_device, smart_inference_mode\r\n\r\n\r\n\r\nclass BasePredictor:\r\n \"\"\"\r\n BasePredictor\r\n\r\n A base class for cre ating predictors.\r\n\r\n Attributes:\r\n args (OmegaConf): Configuration for the predictor.\r\n save_dir (Path): Directory to save results.\r\n done_setup (bool): Whether the predictor has finished setup.\r\n model (nn.Module): Model used for prediction.\r\n data (dict): Data configuration.\r\n device (torch.device): Device used for prediction.\r\n dataset (Dataset): Dataset used for prediction.\r\n vid_path (str): Path to video file.\r\n vid_writer (cv2.VideoWriter): Video writer for saving video output.\r\n annotator (Annotator): Annotator used for prediction.\r\n data_path (str): Path to data.\r\n \"\"\"\r\n\r\n def __init__(self, config=DEFAULT_CONFIG, overrides=None):\r\n \"\"\"\r\n Initializes the BasePredictor class.\r\n\r\n Args:\r\n config (str, optional): Path to a configuration file. Defaults to DEFAULT_CONFIG.\r\n overrides (dict, optional): Configuration overrides. Defaults to None.\r\n \"\"\"\r\n print(\"This is a tracker\",tracker)\r\n if overrides is None:\r\n overrides = {}\r\n self.args = get_config(config, overrides)\r\n project = self.args.project or Path(SETTINGS['runs_dir']) / self.args.task\r\n name = self.args.name or f\"{self.args.mode}\"\r\n self.save_dir = increment_path(Path(project) / name, exist_ok=self.args.exist_ok)\r\n if self.args.save:\r\n (self.save_dir / 'labels' if self.args.save_txt else self.save_dir).mkdir(parents=True, exist_ok=True)\r\n if self.args.conf is None:\r\n self.args.conf = 0.25 # default conf=0.25\r\n self.done_setup = False\r\n \r\n \r\n # Usable if setup is done\r\n self.model = None\r\n self.data = self.args.data # data_dict\r\n self.device = None\r\n self.dataset = None\r\n self.vid_path, self.vid_writer = None, None\r\n self.annotator = None\r\n self.data_path = None\r\n self.callbacks = defaultdict(list, {k: [v] for k, v in callbacks.default_callbacks.items()}) # add callbacks\r\n callbacks.add_integration_callbacks(self)\r\n\r\n \r\n def preprocess(self, img):\r\n pass\r\n\r\n def get_annotator(self, img):\r\n raise NotImplementedError(\"get_annotator function needs to be implemented\")\r\n\r\n def get_tracker(self,img):\r\n \r\n def write_results(self, pred, batch, print_string):\r\n raise NotImplementedError(\"print_results function needs to be implemented\")\r\n\r\n def postprocess(self, preds, img, orig_img):\r\n return preds\r\n\r\n def setup(self, source=None, model=None):\r\n \r\n # source\r\n source = str(source if source is not None else self.args.source)\r\n is_file = Path(source).suffix[1:] in (IMG_FORMATS + VID_FORMATS)\r\n is_url = source.lower().startswith(('rtsp://', 'rtmp://', 'http://', 'https://'))\r\n webcam = source.isnumeric() or source.endswith('.streams') or (is_url and not is_file)\r\n screenshot = source.lower().startswith('screen')\r\n if is_url and is_file:\r\n source = check_file(source) # download\r\n \r\n \r\n # model\r\n device = select_device(self.args.device)\r\n model = model or self.args.model\r\n self.args.half &= device.type != 'cpu' # half precision only supported on CUDA\r\n model = AutoBackend(model, device=device, dnn=self.args.dnn, fp16=self.args.half)\r\n stride, pt = model.stride, model.pt\r\n imgsz = check_imgsz(self.args.imgsz, stride=stride) # check image size\r\n\r\n # Dataloader\r\n bs = 1 # batch_size\r\n if self.args.show:\r\n self.args.show = check_imshow(warn=True)\r\n if webcam:\r\n self.dataset = LoadStreams(source,\r\n imgsz=imgsz,\r\n stride=stride,\r\n auto=pt,\r\n transforms=getattr(model.model, 'transforms', None),\r\n vid_stride=self.args.vid_stride)\r\n bs = len(self.dataset)\r\n elif screenshot:\r\n self.dataset = LoadScreenshots(source,\r\n imgsz=imgsz,\r\n stride=stride,\r\n auto=pt,\r\n transforms=getattr(model.model, 'transforms', None))\r\n else:\r\n self.dataset = LoadImages(source,\r\n imgsz=imgsz,\r\n stride=stride,\r\n auto=pt,\r\n transforms=getattr(model.model, 'transforms', None),\r\n vid_stride=self.args.vid_stride)\r\n self.vid_path, self.vid_writer = [None] * bs, [None] * bs\r\n model.warmup(imgsz=(1 if pt or model.triton else bs, 3, *imgsz)) # warmup\r\n\r\n self.model = model\r\n self.webcam = webcam\r\n self.screenshot = screenshot\r\n self.imgsz = imgsz\r\n self.done_setup = True\r\n self.device = device\r\n\r\n return model\r\n\r\n @smart_inference_mode()\r\n def __call__(self, source=None, model=None):\r\n \r\n self.run_callbacks(\"on_predict_start\")\r\n model= self.model if self.done_setup else self.setup(source, model)\r\n model.eval()\r\n print(tracker)\r\n self.seen, self.windows, self.dt = 0, [], (ops.Profile(), ops.Profile(), ops.Profile())\r\n self.all_outputs = []\r\n for batch in self.dataset:\r\n self.run_callbacks(\"on_predict_batch_start\")\r\n path, im, im0s, vid_cap, s = batch\r\n visualize = increment_path(self.save_dir / Path(path).stem, mkdir=True) if self.args.visualize else False\r\n with self.dt[0]:\r\n im = self.preprocess(im)\r\n if len(im.shape) == 3:\r\n im = im[None] \r\n # Inference\r\n with self.dt[1]:\r\n preds = model(im, augment=self.args.augment, visualize=visualize)\r\n\r\n # postprocess\r\n with self.dt[2]:\r\n preds = self.postprocess(preds, im, im0s)\r\n\r\n for i in range(len(im)):\r\n if self.webcam:\r\n path, im0s = path[i], im0s[i]\r\n p = Path(path)\r\n s += self.write_results(i, preds, (p, im, im0s))\r\n\r\n if self.args.show:\r\n self.show(p)\r\n\r\n if self.args.save:\r\n self.save_preds(vid_cap, i, str(self.save_dir / p.name))\r\n\r\n # Print time (inference-only)\r\n LOGGER.info(f\"{s}{'' if len(preds) else '(no detections), '}{self.dt[1].dt * 1E3:.1f}ms\")\r\n\r\n self.run_callbacks(\"on_predict_batch_end\")\r\n\r\n # Print results\r\n t = tuple(x.t / self.seen * 1E3 for x in self.dt) # speeds per image\r\n LOGGER.info(\r\n f'Speed: %.1fms pre-process, %.1fms inference, %.1fms postprocess per image at shape {(1, 3, *self.imgsz)}'\r\n % t)\r\n \r\n if self.args.save_txt or self.args.save:\r\n s = f\"\\n{len(list(self.save_dir.glob('labels/*.txt')))} labels saved to {self.save_dir / 'labels'}\" if self.args.save_txt else ''\r\n LOGGER.info(f\"Results saved to {colorstr('bold', self.save_dir)}{s}\")\r\n\r\n self.run_callbacks(\"on_predict_end\")\r\n return self.all_outputs\r\n\r\n def show(self, p):\r\n im0 = self.annotator.result()\r\n if platform.system() == 'Linux' and p not in self.windows:\r\n self.windows.append(p)\r\n cv2.namedWindow(str(p), cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO) # allow window resize (Linux)\r\n cv2.resizeWindow(str(p), im0.shape[1], im0.shape[0])\r\n cv2.imshow(str(p), im0)\r\n cv2.waitKey(1) # 1 millisecond\r\n\r\n def save_preds(self, vid_cap, idx, save_path):\r\n im0 = self.annotator.result()\r\n # save imgs\r\n if self.dataset.mode == 'image':\r\n cv2.imwrite(save_path, im0)\r\n else: # 'video' or 'stream'\r\n if self.vid_path[idx] != save_path: # new video\r\n self.vid_path[idx] = save_path\r\n if isinstance(self.vid_writer[idx], cv2.VideoWriter):\r\n self.vid_writer[idx].release() # release previous video writer\r\n if vid_cap: # video\r\n fps = vid_cap.get(cv2.CAP_PROP_FPS)\r\n w = int(vid_cap.get(cv2.CAP_PROP_FRAME_WIDTH))\r\n h = int(vid_cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\r\n else: # stream\r\n fps, w, h = 30, im0.shape[1], im0.shape[0]\r\n save_path = str(Path(save_path).with_suffix('.mp4')) # force *.mp4 suffix on results videos\r\n self.vid_writer[idx] = cv2.VideoWriter(save_path, cv2.VideoWriter_fourcc(*'mp4v'), fps, (w, h))\r\n self.vid_writer[idx].write(im0)\r\n\r\n def run_callbacks(self, event: str):\r\n for callback in self.callbacks.get(event, []):\r\n callback(self)\r\n"
},
{
"path": "yolo/engine/sort.py",
"content": "from __future__ import print_function\n\nimport os\nimport numpy as np\nimport matplotlib\nmatplotlib.use('Agg')\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nfrom skimage import io\n\nimport glob\nimport time\nimport argparse\nfrom filterpy.kalman import KalmanFilter\n\nnp.random.seed(0)\n\ndef linear_assignment(cost_matrix):\n try:\n import lap #linear assignment problem solver\n _, x, y = lap.lapjv(cost_matrix, extend_cost = True)\n return np.array([[y[i],i] for i in x if i>=0])\n except ImportError:\n from scipy.optimize import linear_sum_assignment\n x,y = linear_sum_assignment(cost_matrix)\n return np.array(list(zip(x,y)))\n\n\n\"\"\"From SORT: Computes IOU between two boxes in the form [x1,y1,x2,y2]\"\"\"\ndef iou_batch(bb_test, bb_gt):\n \n bb_gt = np.expand_dims(bb_gt, 0)\n bb_test = np.expand_dims(bb_test, 1)\n \n xx1 = np.maximum(bb_test[...,0], bb_gt[..., 0])\n yy1 = np.maximum(bb_test[..., 1], bb_gt[..., 1])\n xx2 = np.minimum(bb_test[..., 2], bb_gt[..., 2])\n yy2 = np.minimum(bb_test[..., 3], bb_gt[..., 3])\n w = np.maximum(0., xx2 - xx1)\n h = np.maximum(0., yy2 - yy1)\n wh = w * h\n o = wh / ((bb_test[..., 2] - bb_test[..., 0]) * (bb_test[..., 3] - bb_test[..., 1]) \n + (bb_gt[..., 2] - bb_gt[..., 0]) * (bb_gt[..., 3] - bb_gt[..., 1]) - wh)\n return(o)\n\n\n\"\"\"Takes a bounding box in the form [x1,y1,x2,y2] and returns z in the form [x,y,s,r] where x,y is the center of the box and s is the scale/area and r is the aspect ratio\"\"\"\ndef convert_bbox_to_z(bbox):\n w = bbox[2] - bbox[0]\n h = bbox[3] - bbox[1]\n x = bbox[0] + w/2.\n y = bbox[1] + h/2.\n s = w * h \n #scale is just area\n r = w / float(h)\n return np.array([x, y, s, r]).reshape((4, 1))\n\n\n\"\"\"Takes a bounding box in the centre form [x,y,s,r] and returns it in the form\n [x1,y1,x2,y2] where x1,y1 is the top left and x2,y2 is the bottom right\"\"\"\ndef convert_x_to_bbox(x, score=None):\n w = np.sqrt(x[2] * x[3])\n h = x[2] / w\n if(score==None):\n return np.array([x[0]-w/2.,x[1]-h/2.,x[0]+w/2.,x[1]+h/2.]).reshape((1,4))\n else:\n return np.array([x[0]-w/2.,x[1]-h/2.,x[0]+w/2.,x[1]+h/2.,score]).reshape((1,5))\n\n\"\"\"This class represents the internal state of individual tracked objects observed as bbox.\"\"\"\nclass KalmanBoxTracker(object):\n \n count = 0\n def __init__(self, bbox):\n \"\"\"\n Initialize a tracker using initial bounding box\n \n Parameter 'bbox' must have 'detected class' int number at the -1 position.\n \"\"\"\n self.kf = KalmanFilter(dim_x=7, dim_z=4)\n self.kf.F = np.array([[1,0,0,0,1,0,0],[0,1,0,0,0,1,0],[0,0,1,0,0,0,1],[0,0,0,1,0,0,0],[0,0,0,0,1,0,0],[0,0,0,0,0,1,0],[0,0,0,0,0,0,1]])\n self.kf.H = np.array([[1,0,0,0,0,0,0],[0,1,0,0,0,0,0],[0,0,1,0,0,0,0],[0,0,0,1,0,0,0]])\n\n self.kf.R[2:,2:] *= 10. # R: Covariance matrix of measurement noise (set to high for noisy inputs -> more 'inertia' of boxes')\n self.kf.P[4:,4:] *= 1000. #give high uncertainty to the unobservable initial velocities\n self.kf.P *= 10.\n self.kf.Q[-1,-1] *= 0.5 # Q: Covariance matrix of process noise (set to high for erratically moving things)\n self.kf.Q[4:,4:] *= 0.5\n\n self.kf.x[:4] = convert_bbox_to_z(bbox) # STATE VECTOR\n self.time_since_update = 0\n self.id = KalmanBoxTracker.count\n KalmanBoxTracker.count += 1\n self.history = []\n self.hits = 0\n self.hit_streak = 0\n self.age = 0\n self.centroidarr = []\n CX = (bbox[0]+bbox[2])//2\n CY = (bbox[1]+bbox[3])//2\n self.centroidarr.append((CX,CY))\n \n #keep yolov5 detected class information\n self.detclass = bbox[5]\n\n # If we want to store bbox\n self.bbox_history = [bbox]\n \n def update(self, bbox):\n \"\"\"\n Updates the state vector with observed bbox\n \"\"\"\n self.time_since_update = 0\n self.history = []\n self.hits += 1\n self.hit_streak += 1\n self.kf.update(convert_bbox_to_z(bbox))\n self.detclass = bbox[5]\n CX = (bbox[0]+bbox[2])//2\n CY = (bbox[1]+bbox[3])//2\n self.centroidarr.append((CX,CY))\n self.bbox_history.append(bbox)\n \n def predict(self):\n \"\"\"\n Advances the state vector and returns the predicted bounding box estimate\n \"\"\"\n if((self.kf.x[6]+self.kf.x[2])<=0):\n self.kf.x[6] *= 0.0\n self.kf.predict()\n self.age += 1\n if(self.time_since_update>0):\n self.hit_streak = 0\n self.time_since_update += 1\n self.history.append(convert_x_to_bbox(self.kf.x))\n # bbox=self.history[-1]\n # CX = (bbox[0]+bbox[2])/2\n # CY = (bbox[1]+bbox[3])/2\n # self.centroidarr.append((CX,CY))\n \n return self.history[-1]\n \n \n def get_state(self):\n \"\"\"\n Returns the current bounding box estimate\n # test\n arr1 = np.array([[1,2,3,4]])\n arr2 = np.array([0])\n arr3 = np.expand_dims(arr2, 0)\n np.concatenate((arr1,arr3), axis=1)\n \"\"\"\n arr_detclass = np.expand_dims(np.array([self.detclass]), 0)\n \n arr_u_dot = np.expand_dims(self.kf.x[4],0)\n arr_v_dot = np.expand_dims(self.kf.x[5],0)\n arr_s_dot = np.expand_dims(self.kf.x[6],0)\n \n return np.concatenate((convert_x_to_bbox(self.kf.x), arr_detclass, arr_u_dot, arr_v_dot, arr_s_dot), axis=1)\n \ndef associate_detections_to_trackers(detections, trackers, iou_threshold = 0.3):\n \"\"\"\n Assigns detections to tracked object (both represented as bounding boxes)\n Returns 3 lists of \n 1. matches,\n 2. unmatched_detections\n 3. unmatched_trackers\n \"\"\"\n if(len(trackers)==0):\n return np.empty((0,2),dtype=int), np.arange(len(detections)), np.empty((0,5),dtype=int)\n \n iou_matrix = iou_batch(detections, trackers)\n \n if min(iou_matrix.shape) > 0:\n a = (iou_matrix > iou_threshold).astype(np.int32)\n if a.sum(1).max() == 1 and a.sum(0).max() ==1:\n matched_indices = np.stack(np.where(a), axis=1)\n else:\n matched_indices = linear_assignment(-iou_matrix)\n else:\n matched_indices = np.empty(shape=(0,2))\n \n unmatched_detections = []\n for d, det in enumerate(detections):\n if(d not in matched_indices[:,0]):\n unmatched_detections.append(d)\n \n unmatched_trackers = []\n for t, trk in enumerate(trackers):\n if(t not in matched_indices[:,1]):\n unmatched_trackers.append(t)\n \n #filter out matched with low IOU\n matches = []\n for m in matched_indices:\n if(iou_matrix[m[0], m[1]]= self.min_hits or self.frame_count <= self.min_hits):\n ret.append(np.concatenate((d, [trk.id+1])).reshape(1,-1)) #+1'd because MOT benchmark requires positive value\n i -= 1\n #remove dead tracklet\n if(trk.time_since_update >self.max_age):\n self.trackers.pop(i)\n if(len(ret) > 0):\n return np.concatenate(ret)\n return np.empty((0,6))\n\ndef parse_args():\n \"\"\"Parse input arguments.\"\"\"\n parser = argparse.ArgumentParser(description='SORT demo')\n parser.add_argument('--display', dest='display', help='Display online tracker output (slow) [False]',action='store_true')\n parser.add_argument(\"--seq_path\", help=\"Path to detections.\", type=str, default='data')\n parser.add_argument(\"--phase\", help=\"Subdirectory in seq_path.\", type=str, default='train')\n parser.add_argument(\"--max_age\", \n help=\"Maximum number of frames to keep alive a track without associated detections.\", \n type=int, default=1)\n parser.add_argument(\"--min_hits\", \n help=\"Minimum number of associated detections before track is initialised.\", \n type=int, default=3)\n parser.add_argument(\"--iou_threshold\", help=\"Minimum IOU for match.\", type=float, default=0.3)\n args = parser.parse_args()\n return args\n\nif __name__ == '__main__':\n # all train\n args = parse_args()\n display = args.display\n phase = args.phase\n total_time = 0.0\n total_frames = 0\n colours = np.random.rand(32, 3) #used only for display\n if(display):\n if not os.path.exists('mot_benchmark'):\n print('\\n\\tERROR: mot_benchmark link not found!\\n\\n Create a symbolic link to the MOT benchmark\\n (https://motchallenge.net/data/2D_MOT_2015/#download). E.g.:\\n\\n $ ln -s /path/to/MOT2015_challenge/2DMOT2015 mot_benchmark\\n\\n')\n exit()\n plt.ion()\n fig = plt.figure()\n ax1 = fig.add_subplot(111, aspect='equal')\n\n if not os.path.exists('output'):\n os.makedirs('output')\n pattern = os.path.join(args.seq_path, phase, '*', 'det', 'det.txt')\n for seq_dets_fn in glob.glob(pattern):\n mot_tracker = Sort(max_age=args.max_age, \n min_hits=args.min_hits,\n iou_threshold=args.iou_threshold) #create instance of the SORT tracker\n seq_dets = np.loadtxt(seq_dets_fn, delimiter=',')\n seq = seq_dets_fn[pattern.find('*'):].split(os.path.sep)[0]\n \n with open(os.path.join('output', '%s.txt'%(seq)),'w') as out_file:\n print(\"Processing %s.\"%(seq))\n for frame in range(int(seq_dets[:,0].max())):\n frame += 1 #detection and frame numbers begin at 1\n dets = seq_dets[seq_dets[:, 0]==frame, 2:7]\n dets[:, 2:4] += dets[:, 0:2] #convert to [x1,y1,w,h] to [x1,y1,x2,y2]\n total_frames += 1\n\n if(display):\n fn = os.path.join('mot_benchmark', phase, seq, 'img1', '%06d.jpg'%(frame))\n im =io.imread(fn)\n ax1.imshow(im)\n plt.title(seq + ' Tracked Targets')\n\n start_time = time.time()\n trackers = mot_tracker.update(dets)\n cycle_time = time.time() - start_time\n total_time += cycle_time\n\n for d in trackers:\n print('%d,%d,%.2f,%.2f,%.2f,%.2f,1,-1,-1,-1'%(frame,d[4],d[0],d[1],d[2]-d[0],d[3]-d[1]),file=out_file)\n if(display):\n d = d.astype(np.int32)\n ax1.add_patch(patches.Rectangle((d[0],d[1]),d[2]-d[0],d[3]-d[1],fill=False,lw=3,ec=colours[d[4]%32,:]))\n\n if(display):\n fig.canvas.flush_events()\n plt.draw()\n ax1.cla()\n\n print(\"Total Tracking took: %.3f seconds for %d frames or %.1f FPS\" % (total_time, total_frames, total_frames / total_time))\n\n if(display):\n print(\"Note: to get real runtime results run without the option: --display\")\n"
},
{
"path": "yolo/engine/trainer.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nSimple training loop; Boilerplate that could apply to any arbitrary neural network,\r\n\"\"\"\r\n\r\nimport os\r\nimport subprocess\r\nimport time\r\nfrom collections import defaultdict\r\nfrom copy import deepcopy\r\nfrom datetime import datetime\r\nfrom pathlib import Path\r\n\r\nimport numpy as np\r\nimport torch\r\nimport torch.distributed as dist\r\nimport torch.nn as nn\r\nfrom omegaconf import OmegaConf # noqa\r\nfrom omegaconf import open_dict\r\nfrom torch.cuda import amp\r\nfrom torch.nn.parallel import DistributedDataParallel as DDP\r\nfrom torch.optim import lr_scheduler\r\nfrom tqdm import tqdm\r\n\r\nimport yolo.utils as utils\r\nfrom ultralytics import __version__\r\nfrom nn.tasks import attempt_load_one_weight\r\nfrom yolo.configs import get_config\r\nfrom yolo.data.utils import check_dataset, check_dataset_yaml\r\nfrom yolo.utils import (DEFAULT_CONFIG, LOGGER, RANK, SETTINGS, TQDM_BAR_FORMAT, callbacks, colorstr,\r\n yaml_save)\r\nfrom yolo.utils.autobatch import check_train_batch_size\r\nfrom yolo.utils.checks import check_file, print_args\r\nfrom yolo.utils.dist import ddp_cleanup, generate_ddp_command\r\nfrom yolo.utils.files import get_latest_run, increment_path\r\nfrom yolo.utils.torch_utils import ModelEMA, de_parallel, init_seeds, one_cycle, strip_optimizer\r\n\r\n\r\nclass BaseTrainer:\r\n \"\"\"\r\n BaseTrainer\r\n\r\n > A base class for creating trainers.\r\n\r\n Attributes:\r\n args (OmegaConf): Configuration for the trainer.\r\n check_resume (method): Method to check if training should be resumed from a saved checkpoint.\r\n console (logging.Logger): Logger instance.\r\n validator (BaseValidator): Validator instance.\r\n model (nn.Module): Model instance.\r\n callbacks (defaultdict): Dictionary of callbacks.\r\n save_dir (Path): Directory to save results.\r\n wdir (Path): Directory to save weights.\r\n last (Path): Path to last checkpoint.\r\n best (Path): Path to best checkpoint.\r\n batch_size (int): Batch size for training.\r\n epochs (int): Number of epochs to train for.\r\n start_epoch (int): Starting epoch for training.\r\n device (torch.device): Device to use for training.\r\n amp (bool): Flag to enable AMP (Automatic Mixed Precision).\r\n scaler (amp.GradScaler): Gradient scaler for AMP.\r\n data (str): Path to data.\r\n trainset (torch.utils.data.Dataset): Training dataset.\r\n testset (torch.utils.data.Dataset): Testing dataset.\r\n ema (nn.Module): EMA (Exponential Moving Average) of the model.\r\n lf (nn.Module): Loss function.\r\n scheduler (torch.optim.lr_scheduler._LRScheduler): Learning rate scheduler.\r\n best_fitness (float): The best fitness value achieved.\r\n fitness (float): Current fitness value.\r\n loss (float): Current loss value.\r\n tloss (float): Total loss value.\r\n loss_names (list): List of loss names.\r\n csv (Path): Path to results CSV file.\r\n \"\"\"\r\n\r\n def __init__(self, config=DEFAULT_CONFIG, overrides=None):\r\n \"\"\"\r\n > Initializes the BaseTrainer class.\r\n\r\n Args:\r\n config (str, optional): Path to a configuration file. Defaults to DEFAULT_CONFIG.\r\n overrides (dict, optional): Configuration overrides. Defaults to None.\r\n \"\"\"\r\n if overrides is None:\r\n overrides = {}\r\n self.args = get_config(config, overrides)\r\n self.check_resume()\r\n self.console = LOGGER\r\n self.validator = None\r\n self.model = None\r\n self.callbacks = defaultdict(list)\r\n init_seeds(self.args.seed + 1 + RANK, deterministic=self.args.deterministic)\r\n\r\n # Dirs\r\n project = self.args.project or Path(SETTINGS['runs_dir']) / self.args.task\r\n name = self.args.name or f\"{self.args.mode}\"\r\n self.save_dir = Path(\r\n self.args.get(\r\n \"save_dir\",\r\n increment_path(Path(project) / name, exist_ok=self.args.exist_ok if RANK in {-1, 0} else True)))\r\n self.wdir = self.save_dir / 'weights' # weights dir\r\n if RANK in {-1, 0}:\r\n self.wdir.mkdir(parents=True, exist_ok=True) # make dir\r\n with open_dict(self.args):\r\n self.args.save_dir = str(self.save_dir)\r\n yaml_save(self.save_dir / 'args.yaml', OmegaConf.to_container(self.args, resolve=True)) # save run args\r\n self.last, self.best = self.wdir / 'last.pt', self.wdir / 'best.pt' # checkpoint paths\r\n\r\n self.batch_size = self.args.batch\r\n self.epochs = self.args.epochs\r\n self.start_epoch = 0\r\n if RANK == -1:\r\n print_args(dict(self.args))\r\n\r\n # Device\r\n self.device = utils.torch_utils.select_device(self.args.device, self.batch_size)\r\n self.amp = self.device.type != 'cpu'\r\n self.scaler = amp.GradScaler(enabled=self.amp)\r\n if self.device.type == 'cpu':\r\n self.args.workers = 0 # faster CPU training as time dominated by inference, not dataloading\r\n\r\n # Model and Dataloaders.\r\n self.model = self.args.model\r\n self.data = self.args.data\r\n if self.data.endswith(\".yaml\"):\r\n self.data = check_dataset_yaml(self.data)\r\n else:\r\n self.data = check_dataset(self.data)\r\n self.trainset, self.testset = self.get_dataset(self.data)\r\n self.ema = None\r\n\r\n # Optimization utils init\r\n self.lf = None\r\n self.scheduler = None\r\n\r\n # Epoch level metrics\r\n self.best_fitness = None\r\n self.fitness = None\r\n self.loss = None\r\n self.tloss = None\r\n self.loss_names = ['Loss']\r\n self.csv = self.save_dir / 'results.csv'\r\n self.plot_idx = [0, 1, 2]\r\n\r\n # Callbacks\r\n self.callbacks = defaultdict(list, {k: [v] for k, v in callbacks.default_callbacks.items()}) # add callbacks\r\n if RANK in {0, -1}:\r\n callbacks.add_integration_callbacks(self)\r\n\r\n def add_callback(self, event: str, callback):\r\n \"\"\"\r\n > Appends the given callback.\r\n \"\"\"\r\n self.callbacks[event].append(callback)\r\n\r\n def set_callback(self, event: str, callback):\r\n \"\"\"\r\n > Overrides the existing callbacks with the given callback.\r\n \"\"\"\r\n self.callbacks[event] = [callback]\r\n\r\n def run_callbacks(self, event: str):\r\n for callback in self.callbacks.get(event, []):\r\n callback(self)\r\n\r\n def train(self):\r\n world_size = torch.cuda.device_count()\r\n if world_size > 1 and \"LOCAL_RANK\" not in os.environ:\r\n command = generate_ddp_command(world_size, self)\r\n try:\r\n subprocess.run(command)\r\n except Exception as e:\r\n self.console(e)\r\n finally:\r\n ddp_cleanup(command, self)\r\n else:\r\n self._do_train(int(os.getenv(\"RANK\", -1)), world_size)\r\n\r\n def _setup_ddp(self, rank, world_size):\r\n # os.environ['MASTER_ADDR'] = 'localhost'\r\n # os.environ['MASTER_PORT'] = '9020'\r\n torch.cuda.set_device(rank)\r\n self.device = torch.device('cuda', rank)\r\n self.console.info(f\"DDP settings: RANK {rank}, WORLD_SIZE {world_size}, DEVICE {self.device}\")\r\n dist.init_process_group(\"nccl\" if dist.is_nccl_available() else \"gloo\", rank=rank, world_size=world_size)\r\n\r\n def _setup_train(self, rank, world_size):\r\n \"\"\"\r\n > Builds dataloaders and optimizer on correct rank process.\r\n \"\"\"\r\n # model\r\n self.run_callbacks(\"on_pretrain_routine_start\")\r\n ckpt = self.setup_model()\r\n self.model = self.model.to(self.device)\r\n self.set_model_attributes()\r\n if world_size > 1:\r\n self.model = DDP(self.model, device_ids=[rank])\r\n\r\n # Batch size\r\n if self.batch_size == -1:\r\n if RANK == -1: # single-GPU only, estimate best batch size\r\n self.batch_size = check_train_batch_size(self.model, self.args.imgsz, self.amp)\r\n else:\r\n SyntaxError('batch=-1 to use AutoBatch is only available in Single-GPU training. '\r\n 'Please pass a valid batch size value for Multi-GPU DDP training, i.e. batch=16')\r\n\r\n # Optimizer\r\n self.accumulate = max(round(self.args.nbs / self.batch_size), 1) # accumulate loss before optimizing\r\n self.args.weight_decay *= self.batch_size * self.accumulate / self.args.nbs # scale weight_decay\r\n self.optimizer = self.build_optimizer(model=self.model,\r\n name=self.args.optimizer,\r\n lr=self.args.lr0,\r\n momentum=self.args.momentum,\r\n decay=self.args.weight_decay)\r\n # Scheduler\r\n if self.args.cos_lr:\r\n self.lf = one_cycle(1, self.args.lrf, self.epochs) # cosine 1->hyp['lrf']\r\n else:\r\n self.lf = lambda x: (1 - x / self.epochs) * (1.0 - self.args.lrf) + self.args.lrf # linear\r\n self.scheduler = lr_scheduler.LambdaLR(self.optimizer, lr_lambda=self.lf)\r\n self.scheduler.last_epoch = self.start_epoch - 1 # do not move\r\n\r\n # dataloaders\r\n batch_size = self.batch_size // world_size if world_size > 1 else self.batch_size\r\n self.train_loader = self.get_dataloader(self.trainset, batch_size=batch_size, rank=rank, mode=\"train\")\r\n if rank in {0, -1}:\r\n self.test_loader = self.get_dataloader(self.testset, batch_size=batch_size * 2, rank=-1, mode=\"val\")\r\n self.validator = self.get_validator()\r\n metric_keys = self.validator.metrics.keys + self.label_loss_items(prefix=\"val\")\r\n self.metrics = dict(zip(metric_keys, [0] * len(metric_keys))) # TODO: init metrics for plot_results()?\r\n self.ema = ModelEMA(self.model)\r\n self.resume_training(ckpt)\r\n self.run_callbacks(\"on_pretrain_routine_end\")\r\n\r\n def _do_train(self, rank=-1, world_size=1):\r\n if world_size > 1:\r\n self._setup_ddp(rank, world_size)\r\n\r\n self._setup_train(rank, world_size)\r\n\r\n self.epoch_time = None\r\n self.epoch_time_start = time.time()\r\n self.train_time_start = time.time()\r\n nb = len(self.train_loader) # number of batches\r\n nw = max(round(self.args.warmup_epochs * nb), 100) # number of warmup iterations\r\n last_opt_step = -1\r\n self.run_callbacks(\"on_train_start\")\r\n self.log(f\"Image sizes {self.args.imgsz} train, {self.args.imgsz} val\\n\"\r\n f'Using {self.train_loader.num_workers * (world_size or 1)} dataloader workers\\n'\r\n f\"Logging results to {colorstr('bold', self.save_dir)}\\n\"\r\n f\"Starting training for {self.epochs} epochs...\")\r\n if self.args.close_mosaic:\r\n base_idx = (self.epochs - self.args.close_mosaic) * nb\r\n self.plot_idx.extend([base_idx, base_idx + 1, base_idx + 2])\r\n for epoch in range(self.start_epoch, self.epochs):\r\n self.epoch = epoch\r\n self.run_callbacks(\"on_train_epoch_start\")\r\n self.model.train()\r\n if rank != -1:\r\n self.train_loader.sampler.set_epoch(epoch)\r\n pbar = enumerate(self.train_loader)\r\n # Update dataloader attributes (optional)\r\n if epoch == (self.epochs - self.args.close_mosaic):\r\n self.console.info(\"Closing dataloader mosaic\")\r\n if hasattr(self.train_loader.dataset, 'mosaic'):\r\n self.train_loader.dataset.mosaic = False\r\n if hasattr(self.train_loader.dataset, 'close_mosaic'):\r\n self.train_loader.dataset.close_mosaic(hyp=self.args)\r\n\r\n if rank in {-1, 0}:\r\n self.console.info(self.progress_string())\r\n pbar = tqdm(enumerate(self.train_loader), total=nb, bar_format=TQDM_BAR_FORMAT)\r\n self.tloss = None\r\n self.optimizer.zero_grad()\r\n for i, batch in pbar:\r\n self.run_callbacks(\"on_train_batch_start\")\r\n # Warmup\r\n ni = i + nb * epoch\r\n if ni <= nw:\r\n xi = [0, nw] # x interp\r\n self.accumulate = max(1, np.interp(ni, xi, [1, self.args.nbs / self.batch_size]).round())\r\n for j, x in enumerate(self.optimizer.param_groups):\r\n # bias lr falls from 0.1 to lr0, all other lrs rise from 0.0 to lr0\r\n x['lr'] = np.interp(\r\n ni, xi, [self.args.warmup_bias_lr if j == 0 else 0.0, x['initial_lr'] * self.lf(epoch)])\r\n if 'momentum' in x:\r\n x['momentum'] = np.interp(ni, xi, [self.args.warmup_momentum, self.args.momentum])\r\n\r\n # Forward\r\n with torch.cuda.amp.autocast(self.amp):\r\n batch = self.preprocess_batch(batch)\r\n preds = self.model(batch[\"img\"])\r\n self.loss, self.loss_items = self.criterion(preds, batch)\r\n if rank != -1:\r\n self.loss *= world_size\r\n self.tloss = (self.tloss * i + self.loss_items) / (i + 1) if self.tloss is not None \\\r\n else self.loss_items\r\n\r\n # Backward\r\n self.scaler.scale(self.loss).backward()\r\n\r\n # Optimize - https://pytorch.org/docs/master/notes/amp_examples.html\r\n if ni - last_opt_step >= self.accumulate:\r\n self.optimizer_step()\r\n last_opt_step = ni\r\n\r\n # Log\r\n mem = f'{torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0:.3g}G' # (GB)\r\n loss_len = self.tloss.shape[0] if len(self.tloss.size()) else 1\r\n losses = self.tloss if loss_len > 1 else torch.unsqueeze(self.tloss, 0)\r\n if rank in {-1, 0}:\r\n pbar.set_description(\r\n ('%11s' * 2 + '%11.4g' * (2 + loss_len)) %\r\n (f'{epoch + 1}/{self.epochs}', mem, *losses, batch[\"cls\"].shape[0], batch[\"img\"].shape[-1]))\r\n self.run_callbacks('on_batch_end')\r\n if self.args.plots and ni in self.plot_idx:\r\n self.plot_training_samples(batch, ni)\r\n\r\n self.run_callbacks(\"on_train_batch_end\")\r\n\r\n self.lr = {f\"lr/pg{ir}\": x['lr'] for ir, x in enumerate(self.optimizer.param_groups)} # for loggers\r\n\r\n self.scheduler.step()\r\n self.run_callbacks(\"on_train_epoch_end\")\r\n\r\n if rank in {-1, 0}:\r\n\r\n # Validation\r\n self.ema.update_attr(self.model, include=['yaml', 'nc', 'args', 'names', 'stride', 'class_weights'])\r\n final_epoch = (epoch + 1 == self.epochs)\r\n if self.args.val or final_epoch:\r\n self.metrics, self.fitness = self.validate()\r\n self.save_metrics(metrics={**self.label_loss_items(self.tloss), **self.metrics, **self.lr})\r\n\r\n # Save model\r\n if self.args.save or (epoch + 1 == self.epochs):\r\n self.save_model()\r\n self.run_callbacks('on_model_save')\r\n\r\n tnow = time.time()\r\n self.epoch_time = tnow - self.epoch_time_start\r\n self.epoch_time_start = tnow\r\n self.run_callbacks(\"on_fit_epoch_end\")\r\n # TODO: termination condition\r\n\r\n if rank in {-1, 0}:\r\n # Do final val with best.pt\r\n self.log(f'\\n{epoch - self.start_epoch + 1} epochs completed in '\r\n f'{(time.time() - self.train_time_start) / 3600:.3f} hours.')\r\n self.final_eval()\r\n if self.args.plots:\r\n self.plot_metrics()\r\n self.log(f\"Results saved to {colorstr('bold', self.save_dir)}\")\r\n self.run_callbacks('on_train_end')\r\n torch.cuda.empty_cache()\r\n self.run_callbacks('teardown')\r\n\r\n def save_model(self):\r\n ckpt = {\r\n 'epoch': self.epoch,\r\n 'best_fitness': self.best_fitness,\r\n 'model': deepcopy(de_parallel(self.model)).half(),\r\n 'ema': deepcopy(self.ema.ema).half(),\r\n 'updates': self.ema.updates,\r\n 'optimizer': self.optimizer.state_dict(),\r\n 'train_args': self.args,\r\n 'date': datetime.now().isoformat(),\r\n 'version': __version__}\r\n\r\n # Save last, best and delete\r\n torch.save(ckpt, self.last)\r\n if self.best_fitness == self.fitness:\r\n torch.save(ckpt, self.best)\r\n del ckpt\r\n\r\n def get_dataset(self, data):\r\n \"\"\"\r\n > Get train, val path from data dict if it exists. Returns None if data format is not recognized.\r\n \"\"\"\r\n return data[\"train\"], data.get(\"val\") or data.get(\"test\")\r\n\r\n def setup_model(self):\r\n \"\"\"\r\n > load/create/download model for any task.\r\n \"\"\"\r\n if isinstance(self.model, torch.nn.Module): # if model is loaded beforehand. No setup needed\r\n return\r\n\r\n model, weights = self.model, None\r\n ckpt = None\r\n if str(model).endswith(\".pt\"):\r\n weights, ckpt = attempt_load_one_weight(model)\r\n cfg = ckpt[\"model\"].yaml\r\n else:\r\n cfg = model\r\n self.model = self.get_model(cfg=cfg, weights=weights) # calls Model(cfg, weights)\r\n return ckpt\r\n\r\n def optimizer_step(self):\r\n self.scaler.unscale_(self.optimizer) # unscale gradients\r\n torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=10.0) # clip gradients\r\n self.scaler.step(self.optimizer)\r\n self.scaler.update()\r\n self.optimizer.zero_grad()\r\n if self.ema:\r\n self.ema.update(self.model)\r\n\r\n def preprocess_batch(self, batch):\r\n \"\"\"\r\n > Allows custom preprocessing model inputs and ground truths depending on task type.\r\n \"\"\"\r\n return batch\r\n\r\n def validate(self):\r\n \"\"\"\r\n > Runs validation on test set using self.validator. The returned dict is expected to contain \"fitness\" key.\r\n \"\"\"\r\n metrics = self.validator(self)\r\n fitness = metrics.pop(\"fitness\", -self.loss.detach().cpu().numpy()) # use loss as fitness measure if not found\r\n if not self.best_fitness or self.best_fitness < fitness:\r\n self.best_fitness = fitness\r\n return metrics, fitness\r\n\r\n def log(self, text, rank=-1):\r\n \"\"\"\r\n > Logs the given text to given ranks process if provided, otherwise logs to all ranks.\r\n\r\n Args\"\r\n text (str): text to log\r\n rank (List[Int]): process rank\r\n\r\n \"\"\"\r\n if rank in {-1, 0}:\r\n self.console.info(text)\r\n\r\n def get_model(self, cfg=None, weights=None, verbose=True):\r\n raise NotImplementedError(\"This task trainer doesn't support loading cfg files\")\r\n\r\n def get_validator(self):\r\n raise NotImplementedError(\"get_validator function not implemented in trainer\")\r\n\r\n def get_dataloader(self, dataset_path, batch_size=16, rank=0):\r\n \"\"\"\r\n > Returns dataloader derived from torch.data.Dataloader.\r\n \"\"\"\r\n raise NotImplementedError(\"get_dataloader function not implemented in trainer\")\r\n\r\n def criterion(self, preds, batch):\r\n \"\"\"\r\n > Returns loss and individual loss items as Tensor.\r\n \"\"\"\r\n raise NotImplementedError(\"criterion function not implemented in trainer\")\r\n\r\n def label_loss_items(self, loss_items=None, prefix=\"train\"):\r\n \"\"\"\r\n Returns a loss dict with labelled training loss items tensor\r\n \"\"\"\r\n # Not needed for classification but necessary for segmentation & detection\r\n return {\"loss\": loss_items} if loss_items is not None else [\"loss\"]\r\n\r\n def set_model_attributes(self):\r\n \"\"\"\r\n To set or update model parameters before training.\r\n \"\"\"\r\n self.model.names = self.data[\"names\"]\r\n\r\n def build_targets(self, preds, targets):\r\n pass\r\n\r\n def progress_string(self):\r\n return \"\"\r\n\r\n # TODO: may need to put these following functions into callback\r\n def plot_training_samples(self, batch, ni):\r\n pass\r\n\r\n def save_metrics(self, metrics):\r\n keys, vals = list(metrics.keys()), list(metrics.values())\r\n n = len(metrics) + 1 # number of cols\r\n s = '' if self.csv.exists() else (('%23s,' * n % tuple(['epoch'] + keys)).rstrip(',') + '\\n') # header\r\n with open(self.csv, 'a') as f:\r\n f.write(s + ('%23.5g,' * n % tuple([self.epoch] + vals)).rstrip(',') + '\\n')\r\n\r\n def plot_metrics(self):\r\n pass\r\n\r\n def final_eval(self):\r\n for f in self.last, self.best:\r\n if f.exists():\r\n strip_optimizer(f) # strip optimizers\r\n if f is self.best:\r\n self.console.info(f'\\nValidating {f}...')\r\n self.validator.args.save_json = True\r\n self.metrics = self.validator(model=f)\r\n self.metrics.pop('fitness', None)\r\n self.run_callbacks('on_fit_epoch_end')\r\n\r\n def check_resume(self):\r\n resume = self.args.resume\r\n if resume:\r\n last = Path(check_file(resume) if isinstance(resume, str) else get_latest_run())\r\n args_yaml = last.parent.parent / 'args.yaml' # train options yaml\r\n if args_yaml.is_file():\r\n args = get_config(args_yaml) # replace\r\n args.model, resume = str(last), True # reinstate\r\n self.args = args\r\n self.resume = resume\r\n\r\n def resume_training(self, ckpt):\r\n if ckpt is None:\r\n return\r\n best_fitness = 0.0\r\n start_epoch = ckpt['epoch'] + 1\r\n if ckpt['optimizer'] is not None:\r\n self.optimizer.load_state_dict(ckpt['optimizer']) # optimizer\r\n best_fitness = ckpt['best_fitness']\r\n if self.ema and ckpt.get('ema'):\r\n self.ema.ema.load_state_dict(ckpt['ema'].float().state_dict()) # EMA\r\n self.ema.updates = ckpt['updates']\r\n if self.resume:\r\n assert start_epoch > 0, \\\r\n f'{self.args.model} training to {self.epochs} epochs is finished, nothing to resume.\\n' \\\r\n f\"Start a new training without --resume, i.e. 'yolo task=... mode=train model={self.args.model}'\"\r\n LOGGER.info(\r\n f'Resuming training from {self.args.model} from epoch {start_epoch} to {self.epochs} total epochs')\r\n if self.epochs < start_epoch:\r\n LOGGER.info(\r\n f\"{self.model} has been trained for {ckpt['epoch']} epochs. Fine-tuning for {self.epochs} more epochs.\")\r\n self.epochs += ckpt['epoch'] # finetune additional epochs\r\n self.best_fitness = best_fitness\r\n self.start_epoch = start_epoch\r\n\r\n @staticmethod\r\n def build_optimizer(model, name='Adam', lr=0.001, momentum=0.9, decay=1e-5):\r\n \"\"\"\r\n > Builds an optimizer with the specified parameters and parameter groups.\r\n\r\n Args:\r\n model (nn.Module): model to optimize\r\n name (str): name of the optimizer to use\r\n lr (float): learning rate\r\n momentum (float): momentum\r\n decay (float): weight decay\r\n\r\n Returns:\r\n optimizer (torch.optim.Optimizer): the built optimizer\r\n \"\"\"\r\n g = [], [], [] # optimizer parameter groups\r\n bn = tuple(v for k, v in nn.__dict__.items() if 'Norm' in k) # normalization layers, i.e. BatchNorm2d()\r\n for v in model.modules():\r\n if hasattr(v, 'bias') and isinstance(v.bias, nn.Parameter): # bias (no decay)\r\n g[2].append(v.bias)\r\n if isinstance(v, bn): # weight (no decay)\r\n g[1].append(v.weight)\r\n elif hasattr(v, 'weight') and isinstance(v.weight, nn.Parameter): # weight (with decay)\r\n g[0].append(v.weight)\r\n\r\n if name == 'Adam':\r\n optimizer = torch.optim.Adam(g[2], lr=lr, betas=(momentum, 0.999)) # adjust beta1 to momentum\r\n elif name == 'AdamW':\r\n optimizer = torch.optim.AdamW(g[2], lr=lr, betas=(momentum, 0.999), weight_decay=0.0)\r\n elif name == 'RMSProp':\r\n optimizer = torch.optim.RMSprop(g[2], lr=lr, momentum=momentum)\r\n elif name == 'SGD':\r\n optimizer = torch.optim.SGD(g[2], lr=lr, momentum=momentum, nesterov=True)\r\n else:\r\n raise NotImplementedError(f'Optimizer {name} not implemented.')\r\n\r\n optimizer.add_param_group({'params': g[0], 'weight_decay': decay}) # add g0 with weight_decay\r\n optimizer.add_param_group({'params': g[1], 'weight_decay': 0.0}) # add g1 (BatchNorm2d weights)\r\n LOGGER.info(f\"{colorstr('optimizer:')} {type(optimizer).__name__}(lr={lr}) with parameter groups \"\r\n f\"{len(g[1])} weight(decay=0.0), {len(g[0])} weight(decay={decay}), {len(g[2])} bias\")\r\n return optimizer\r\n"
},
{
"path": "yolo/engine/validator.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport json\r\nfrom collections import defaultdict\r\nfrom pathlib import Path\r\n\r\nimport torch\r\nfrom omegaconf import OmegaConf # noqa\r\nfrom tqdm import tqdm\r\n\r\nfrom nn.autobackend import AutoBackend\r\nfrom yolo.data.utils import check_dataset, check_dataset_yaml\r\nfrom yolo.utils import DEFAULT_CONFIG, LOGGER, RANK, SETTINGS, TQDM_BAR_FORMAT, callbacks\r\nfrom yolo.utils.checks import check_imgsz\r\nfrom yolo.utils.files import increment_path\r\nfrom yolo.utils.ops import Profile\r\nfrom yolo.utils.torch_utils import de_parallel, select_device, smart_inference_mode\r\n\r\n\r\nclass BaseValidator:\r\n \"\"\"\r\n BaseValidator\r\n\r\n A base class for creating validators.\r\n\r\n Attributes:\r\n dataloader (DataLoader): Dataloader to use for validation.\r\n pbar (tqdm): Progress bar to update during validation.\r\n logger (logging.Logger): Logger to use for validation.\r\n args (OmegaConf): Configuration for the validator.\r\n model (nn.Module): Model to validate.\r\n data (dict): Data dictionary.\r\n device (torch.device): Device to use for validation.\r\n batch_i (int): Current batch index.\r\n training (bool): Whether the model is in training mode.\r\n speed (float): Batch processing speed in seconds.\r\n jdict (dict): Dictionary to store validation results.\r\n save_dir (Path): Directory to save results.\r\n \"\"\"\r\n\r\n def __init__(self, dataloader=None, save_dir=None, pbar=None, logger=None, args=None):\r\n \"\"\"\r\n Initializes a BaseValidator instance.\r\n\r\n Args:\r\n dataloader (torch.utils.data.DataLoader): Dataloader to be used for validation.\r\n save_dir (Path): Directory to save results.\r\n pbar (tqdm.tqdm): Progress bar for displaying progress.\r\n logger (logging.Logger): Logger to log messages.\r\n args (OmegaConf): Configuration for the validator.\r\n \"\"\"\r\n self.dataloader = dataloader\r\n self.pbar = pbar\r\n self.logger = logger or LOGGER\r\n self.args = args or OmegaConf.load(DEFAULT_CONFIG)\r\n self.model = None\r\n self.data = None\r\n self.device = None\r\n self.batch_i = None\r\n self.training = True\r\n self.speed = None\r\n self.jdict = None\r\n\r\n project = self.args.project or Path(SETTINGS['runs_dir']) / self.args.task\r\n name = self.args.name or f\"{self.args.mode}\"\r\n self.save_dir = save_dir or increment_path(Path(project) / name,\r\n exist_ok=self.args.exist_ok if RANK in {-1, 0} else True)\r\n (self.save_dir / 'labels' if self.args.save_txt else self.save_dir).mkdir(parents=True, exist_ok=True)\r\n\r\n if self.args.conf is None:\r\n self.args.conf = 0.001 # default conf=0.001\r\n\r\n self.callbacks = defaultdict(list, {k: [v] for k, v in callbacks.default_callbacks.items()}) # add callbacks\r\n\r\n @smart_inference_mode()\r\n def __call__(self, trainer=None, model=None):\r\n \"\"\"\r\n Supports validation of a pre-trained model if passed or a model being trained\r\n if trainer is passed (trainer gets priority).\r\n \"\"\"\r\n self.training = trainer is not None\r\n if self.training:\r\n self.device = trainer.device\r\n self.data = trainer.data\r\n model = trainer.ema.ema or trainer.model\r\n self.args.half = self.device.type != 'cpu' # force FP16 val during training\r\n model = model.half() if self.args.half else model.float()\r\n self.model = model\r\n self.loss = torch.zeros_like(trainer.loss_items, device=trainer.device)\r\n self.args.plots = trainer.epoch == trainer.epochs - 1 # always plot final epoch\r\n model.eval()\r\n else:\r\n callbacks.add_integration_callbacks(self)\r\n self.run_callbacks('on_val_start')\r\n assert model is not None, \"Either trainer or model is needed for validation\"\r\n self.device = select_device(self.args.device, self.args.batch)\r\n self.args.half &= self.device.type != 'cpu'\r\n model = AutoBackend(model, device=self.device, dnn=self.args.dnn, fp16=self.args.half)\r\n self.model = model\r\n stride, pt, jit, engine = model.stride, model.pt, model.jit, model.engine\r\n imgsz = check_imgsz(self.args.imgsz, stride=stride)\r\n if engine:\r\n self.args.batch = model.batch_size\r\n else:\r\n self.device = model.device\r\n if not pt and not jit:\r\n self.args.batch = 1 # export.py models default to batch-size 1\r\n self.logger.info(\r\n f'Forcing --batch-size 1 square inference (1,3,{imgsz},{imgsz}) for non-PyTorch models')\r\n\r\n if isinstance(self.args.data, str) and self.args.data.endswith(\".yaml\"):\r\n self.data = check_dataset_yaml(self.args.data)\r\n else:\r\n self.data = check_dataset(self.args.data)\r\n\r\n if self.device.type == 'cpu':\r\n self.args.workers = 0 # faster CPU val as time dominated by inference, not dataloading\r\n self.dataloader = self.dataloader or \\\r\n self.get_dataloader(self.data.get(\"val\") or self.data.set(\"test\"), self.args.batch)\r\n\r\n model.eval()\r\n model.warmup(imgsz=(1 if pt else self.args.batch, 3, imgsz, imgsz)) # warmup\r\n\r\n dt = Profile(), Profile(), Profile(), Profile()\r\n n_batches = len(self.dataloader)\r\n desc = self.get_desc()\r\n # NOTE: keeping `not self.training` in tqdm will eliminate pbar after segmentation evaluation during training,\r\n # which may affect classification task since this arg is in yolov5/classify/val.py.\r\n # bar = tqdm(self.dataloader, desc, n_batches, not self.training, bar_format=TQDM_BAR_FORMAT)\r\n bar = tqdm(self.dataloader, desc, n_batches, bar_format=TQDM_BAR_FORMAT)\r\n self.init_metrics(de_parallel(model))\r\n self.jdict = [] # empty before each val\r\n for batch_i, batch in enumerate(bar):\r\n self.run_callbacks('on_val_batch_start')\r\n self.batch_i = batch_i\r\n # pre-process\r\n with dt[0]:\r\n batch = self.preprocess(batch)\r\n\r\n # inference\r\n with dt[1]:\r\n preds = model(batch[\"img\"])\r\n\r\n # loss\r\n with dt[2]:\r\n if self.training:\r\n self.loss += trainer.criterion(preds, batch)[1]\r\n\r\n # pre-process predictions\r\n with dt[3]:\r\n preds = self.postprocess(preds)\r\n\r\n self.update_metrics(preds, batch)\r\n if self.args.plots and batch_i < 3:\r\n self.plot_val_samples(batch, batch_i)\r\n self.plot_predictions(batch, preds, batch_i)\r\n\r\n self.run_callbacks('on_val_batch_end')\r\n stats = self.get_stats()\r\n self.check_stats(stats)\r\n self.print_results()\r\n self.speed = tuple(x.t / len(self.dataloader.dataset) * 1E3 for x in dt) # speeds per image\r\n self.run_callbacks('on_val_end')\r\n if self.training:\r\n model.float()\r\n results = {**stats, **trainer.label_loss_items(self.loss.cpu() / len(self.dataloader), prefix=\"val\")}\r\n return {k: round(float(v), 5) for k, v in results.items()} # return results as 5 decimal place floats\r\n else:\r\n self.logger.info('Speed: %.1fms pre-process, %.1fms inference, %.1fms loss, %.1fms post-process per image' %\r\n self.speed)\r\n if self.args.save_json and self.jdict:\r\n with open(str(self.save_dir / \"predictions.json\"), 'w') as f:\r\n self.logger.info(f\"Saving {f.name}...\")\r\n json.dump(self.jdict, f) # flatten and save\r\n stats = self.eval_json(stats) # update stats\r\n return stats\r\n\r\n def run_callbacks(self, event: str):\r\n for callback in self.callbacks.get(event, []):\r\n callback(self)\r\n\r\n def get_dataloader(self, dataset_path, batch_size):\r\n raise NotImplementedError(\"get_dataloader function not implemented for this validator\")\r\n\r\n def preprocess(self, batch):\r\n return batch\r\n\r\n def postprocess(self, preds):\r\n return preds\r\n\r\n def init_metrics(self, model):\r\n pass\r\n\r\n def update_metrics(self, preds, batch):\r\n pass\r\n\r\n def get_stats(self):\r\n return {}\r\n\r\n def check_stats(self, stats):\r\n pass\r\n\r\n def print_results(self):\r\n pass\r\n\r\n def get_desc(self):\r\n pass\r\n\r\n @property\r\n def metric_keys(self):\r\n return []\r\n\r\n # TODO: may need to put these following functions into callback\r\n def plot_val_samples(self, batch, ni):\r\n pass\r\n\r\n def plot_predictions(self, batch, preds, ni):\r\n pass\r\n\r\n def pred_to_json(self, preds, batch):\r\n pass\r\n\r\n def eval_json(self, stats):\r\n pass\r\n"
},
{
"path": "yolo/utils/__init__.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport contextlib\r\nimport inspect\r\nimport logging.config\r\nimport os\r\nimport platform\r\nimport subprocess\r\nimport sys\r\nimport tempfile\r\nimport threading\r\nimport uuid\r\nfrom pathlib import Path\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport pandas as pd\r\nimport torch\r\nimport yaml\r\n\r\n# Constants\r\nFILE = Path(__file__).resolve()\r\nROOT = FILE.parents[2] # YOLO\r\nDEFAULT_CONFIG = ROOT / \"yolo/configs/default.yaml\"\r\nRANK = int(os.getenv('RANK', -1))\r\nNUM_THREADS = min(8, max(1, os.cpu_count() - 1)) # number of YOLOv5 multiprocessing threads\r\nAUTOINSTALL = str(os.getenv('YOLOv5_AUTOINSTALL', True)).lower() == 'true' # global auto-install mode\r\nFONT = 'Arial.ttf' # https://com/assets/Arial.ttf\r\nVERBOSE = str(os.getenv('YOLOv5_VERBOSE', True)).lower() == 'true' # global verbose mode\r\nTQDM_BAR_FORMAT = '{l_bar}{bar:10}{r_bar}' # tqdm bar format\r\nLOGGING_NAME = 'yolov5'\r\nHELP_MSG = \\\r\n \"\"\"\r\n Usage examples for running YOLOv8:\r\n\r\n 1. Install the ultralytics package:\r\n\r\n pip install ultralytics\r\n\r\n 2. Use the Python SDK:\r\n\r\n from ultralytics import YOLO\r\n\r\n model = YOLO('yolov8n.yaml') # build a new model from scratch\r\n model = YOLO('yolov8n.pt') # load a pretrained model (recommended for best training results)\r\n results = model.train(data='coco128.yaml') # train the model\r\n results = model.val() # evaluate model performance on the validation set\r\n results = model.predict(source='bus.jpg') # predict on an image\r\n success = model.export(format='onnx') # export the model to ONNX format\r\n\r\n 3. Use the command line interface (CLI):\r\n\r\n yolo task=detect mode=train model=yolov8n.yaml args...\r\n classify predict yolov8n-cls.yaml args...\r\n segment val yolov8n-seg.yaml args...\r\n export yolov8n.pt format=onnx args...\r\n\r\n Docs: https://docs.com\r\n Community: https://community.com\r\n GitHub: https://github.com/ultralytics/ultralytics\r\n \"\"\"\r\n\r\n# Settings\r\ntorch.set_printoptions(linewidth=320, precision=5, profile='long')\r\nnp.set_printoptions(linewidth=320, formatter={'float_kind': '{:11.5g}'.format}) # format short g, %precision=5\r\npd.options.display.max_columns = 10\r\ncv2.setNumThreads(0) # prevent OpenCV from multithreading (incompatible with PyTorch DataLoader)\r\nos.environ['NUMEXPR_MAX_THREADS'] = str(NUM_THREADS) # NumExpr max threads\r\nos.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' # for deterministic training\r\n\r\n# Default config dictionary\r\nwith open(DEFAULT_CONFIG, errors='ignore') as f:\r\n DEFAULT_CONFIG_DICT = yaml.safe_load(f)\r\nDEFAULT_CONFIG_KEYS = DEFAULT_CONFIG_DICT.keys()\r\n\r\n\r\ndef is_colab():\r\n \"\"\"\r\n Check if the current script is running inside a Google Colab notebook.\r\n\r\n Returns:\r\n bool: True if running inside a Colab notebook, False otherwise.\r\n \"\"\"\r\n # Check if the google.colab module is present in sys.modules\r\n return 'google.colab' in sys.modules\r\n\r\n\r\ndef is_kaggle():\r\n \"\"\"\r\n Check if the current script is running inside a Kaggle kernel.\r\n\r\n Returns:\r\n bool: True if running inside a Kaggle kernel, False otherwise.\r\n \"\"\"\r\n return os.environ.get('PWD') == '/kaggle/working' and os.environ.get('KAGGLE_URL_BASE') == 'https://www.kaggle.com'\r\n\r\n\r\ndef is_jupyter_notebook():\r\n \"\"\"\r\n Check if the current script is running inside a Jupyter Notebook.\r\n Verified on Colab, Jupyterlab, Kaggle, Paperspace.\r\n\r\n Returns:\r\n bool: True if running inside a Jupyter Notebook, False otherwise.\r\n \"\"\"\r\n # Check if the get_ipython function exists\r\n # (it does not exist when running as a standalone script)\r\n try:\r\n from IPython import get_ipython\r\n return get_ipython() is not None\r\n except ImportError:\r\n return False\r\n\r\n\r\ndef is_docker() -> bool:\r\n \"\"\"\r\n Determine if the script is running inside a Docker container.\r\n\r\n Returns:\r\n bool: True if the script is running inside a Docker container, False otherwise.\r\n \"\"\"\r\n with open('/proc/self/cgroup') as f:\r\n return 'docker' in f.read()\r\n\r\n\r\ndef is_git_directory() -> bool:\r\n \"\"\"\r\n Check if the current working directory is inside a git repository.\r\n\r\n Returns:\r\n bool: True if the current working directory is inside a git repository, False otherwise.\r\n \"\"\"\r\n from git import Repo\r\n try:\r\n # Check if the current working directory is a git repository\r\n Repo(search_parent_directories=True)\r\n return True\r\n except Exception:\r\n return False\r\n\r\n\r\ndef is_pip_package(filepath: str = __name__) -> bool:\r\n \"\"\"\r\n Determines if the file at the given filepath is part of a pip package.\r\n\r\n Args:\r\n filepath (str): The filepath to check.\r\n\r\n Returns:\r\n bool: True if the file is part of a pip package, False otherwise.\r\n \"\"\"\r\n import importlib.util\r\n\r\n # Get the spec for the module\r\n spec = importlib.util.find_spec(filepath)\r\n\r\n # Return whether the spec is not None and the origin is not None (indicating it is a package)\r\n return spec is not None and spec.origin is not None\r\n\r\n\r\ndef is_dir_writeable(dir_path: str) -> bool:\r\n \"\"\"\r\n Check if a directory is writeable.\r\n\r\n Args:\r\n dir_path (str): The path to the directory.\r\n\r\n Returns:\r\n bool: True if the directory is writeable, False otherwise.\r\n \"\"\"\r\n try:\r\n with tempfile.TemporaryFile(dir=dir_path):\r\n pass\r\n return True\r\n except OSError:\r\n return False\r\n\r\n\r\ndef get_git_root_dir():\r\n \"\"\"\r\n Determines whether the current file is part of a git repository and if so, returns the repository root directory.\r\n If the current file is not part of a git repository, returns None.\r\n \"\"\"\r\n try:\r\n output = subprocess.run([\"git\", \"rev-parse\", \"--git-dir\"], capture_output=True, check=True)\r\n return Path(output.stdout.strip().decode('utf-8')).parent # parent/.git\r\n except subprocess.CalledProcessError:\r\n return None\r\n\r\n\r\ndef get_default_args(func):\r\n # Get func() default arguments\r\n signature = inspect.signature(func)\r\n return {k: v.default for k, v in signature.parameters.items() if v.default is not inspect.Parameter.empty}\r\n\r\n\r\ndef get_user_config_dir(sub_dir='Ultralytics'):\r\n \"\"\"\r\n Get the user config directory.\r\n\r\n Args:\r\n sub_dir (str): The name of the subdirectory to create.\r\n\r\n Returns:\r\n Path: The path to the user config directory.\r\n \"\"\"\r\n # Get the operating system name\r\n os_name = platform.system()\r\n\r\n # Return the appropriate config directory for each operating system\r\n if os_name == 'Windows':\r\n path = Path.home() / 'AppData' / 'Roaming' / sub_dir\r\n elif os_name == 'Darwin': # macOS\r\n path = Path.home() / 'Library' / 'Application Support' / sub_dir\r\n elif os_name == 'Linux':\r\n path = Path.home() / '.config' / sub_dir\r\n else:\r\n raise ValueError(f'Unsupported operating system: {os_name}')\r\n\r\n # GCP and AWS lambda fix, only /tmp is writeable\r\n if not is_dir_writeable(str(path.parent)):\r\n path = Path('/tmp') / sub_dir\r\n\r\n # Create the subdirectory if it does not exist\r\n path.mkdir(parents=True, exist_ok=True)\r\n\r\n return path\r\n\r\n\r\nUSER_CONFIG_DIR = get_user_config_dir() # Ultralytics settings dir\r\n\r\n\r\ndef emojis(string=''):\r\n # Return platform-dependent emoji-safe version of string\r\n return string.encode().decode('ascii', 'ignore') if platform.system() == 'Windows' else string\r\n\r\n\r\ndef colorstr(*input):\r\n # Colors a string https://en.wikipedia.org/wiki/ANSI_escape_code, i.e. colorstr('blue', 'hello world')\r\n *args, string = input if len(input) > 1 else (\"blue\", \"bold\", input[0]) # color arguments, string\r\n colors = {\r\n \"black\": \"\\033[30m\", # basic colors\r\n \"red\": \"\\033[31m\",\r\n \"green\": \"\\033[32m\",\r\n \"yellow\": \"\\033[33m\",\r\n \"blue\": \"\\033[34m\",\r\n \"magenta\": \"\\033[35m\",\r\n \"cyan\": \"\\033[36m\",\r\n \"white\": \"\\033[37m\",\r\n \"bright_black\": \"\\033[90m\", # bright colors\r\n \"bright_red\": \"\\033[91m\",\r\n \"bright_green\": \"\\033[92m\",\r\n \"bright_yellow\": \"\\033[93m\",\r\n \"bright_blue\": \"\\033[94m\",\r\n \"bright_magenta\": \"\\033[95m\",\r\n \"bright_cyan\": \"\\033[96m\",\r\n \"bright_white\": \"\\033[97m\",\r\n \"end\": \"\\033[0m\", # misc\r\n \"bold\": \"\\033[1m\",\r\n \"underline\": \"\\033[4m\",}\r\n return \"\".join(colors[x] for x in args) + f\"{string}\" + colors[\"end\"]\r\n\r\n\r\ndef set_logging(name=LOGGING_NAME, verbose=True):\r\n # sets up logging for the given name\r\n rank = int(os.getenv('RANK', -1)) # rank in world for Multi-GPU trainings\r\n level = logging.INFO if verbose and rank in {-1, 0} else logging.ERROR\r\n logging.config.dictConfig({\r\n \"version\": 1,\r\n \"disable_existing_loggers\": False,\r\n \"formatters\": {\r\n name: {\r\n \"format\": \"%(message)s\"}},\r\n \"handlers\": {\r\n name: {\r\n \"class\": \"logging.StreamHandler\",\r\n \"formatter\": name,\r\n \"level\": level,}},\r\n \"loggers\": {\r\n name: {\r\n \"level\": level,\r\n \"handlers\": [name],\r\n \"propagate\": False,}}})\r\n\r\n\r\nclass TryExcept(contextlib.ContextDecorator):\r\n # YOLOv5 TryExcept class. Usage: @TryExcept() decorator or 'with TryExcept():' context manager\r\n def __init__(self, msg=''):\r\n self.msg = msg\r\n\r\n def __enter__(self):\r\n pass\r\n\r\n def __exit__(self, exc_type, value, traceback):\r\n if value:\r\n print(emojis(f\"{self.msg}{': ' if self.msg else ''}{value}\"))\r\n return True\r\n\r\n\r\ndef threaded(func):\r\n # Multi-threads a target function and returns thread. Usage: @threaded decorator\r\n def wrapper(*args, **kwargs):\r\n thread = threading.Thread(target=func, args=args, kwargs=kwargs, daemon=True)\r\n thread.start()\r\n return thread\r\n\r\n return wrapper\r\n\r\n\r\ndef yaml_save(file='data.yaml', data=None):\r\n \"\"\"\r\n Save YAML data to a file.\r\n\r\n Args:\r\n file (str, optional): File name. Default is 'data.yaml'.\r\n data (dict, optional): Data to save in YAML format. Default is None.\r\n\r\n Returns:\r\n None: Data is saved to the specified file.\r\n \"\"\"\r\n file = Path(file)\r\n if not file.parent.exists():\r\n # Create parent directories if they don't exist\r\n file.parent.mkdir(parents=True, exist_ok=True)\r\n\r\n with open(file, 'w') as f:\r\n # Dump data to file in YAML format, converting Path objects to strings\r\n yaml.safe_dump({k: str(v) if isinstance(v, Path) else v for k, v in data.items()}, f, sort_keys=False)\r\n\r\n\r\ndef yaml_load(file='data.yaml', append_filename=False):\r\n \"\"\"\r\n Load YAML data from a file.\r\n\r\n Args:\r\n file (str, optional): File name. Default is 'data.yaml'.\r\n append_filename (bool): Add the YAML filename to the YAML dictionary. Default is False.\r\n\r\n Returns:\r\n dict: YAML data and file name.\r\n \"\"\"\r\n with open(file, errors='ignore') as f:\r\n # Add YAML filename to dict and return\r\n return {**yaml.safe_load(f), 'yaml_file': str(file)} if append_filename else yaml.safe_load(f)\r\n\r\n\r\ndef get_settings(file=USER_CONFIG_DIR / 'settings.yaml'):\r\n \"\"\"\r\n Loads a global settings YAML file or creates one with default values if it does not exist.\r\n\r\n Args:\r\n file (Path): Path to the settings YAML file. Defaults to 'settings.yaml' in the USER_CONFIG_DIR.\r\n\r\n Returns:\r\n dict: Dictionary of settings key-value pairs.\r\n \"\"\"\r\n from yolo.utils.torch_utils import torch_distributed_zero_first\r\n\r\n root = get_git_root_dir() or Path('') # not is_pip_package()\r\n defaults = {\r\n 'datasets_dir': str(root / 'datasets'), # default datasets directory.\r\n 'weights_dir': str(root / 'weights'), # default weights directory.\r\n 'runs_dir': str(root / 'runs'), # default runs directory.\r\n 'sync': True, # sync analytics to help with YOLO development\r\n 'uuid': uuid.getnode()} # device UUID to align analytics\r\n\r\n with torch_distributed_zero_first(RANK):\r\n if not file.exists():\r\n yaml_save(file, defaults)\r\n\r\n settings = yaml_load(file)\r\n\r\n # Check that settings keys and types match defaults\r\n correct = settings.keys() == defaults.keys() and \\\r\n all(type(a) == type(b) for a, b in zip(settings.values(), defaults.values()))\r\n if not correct:\r\n LOGGER.warning('WARNING β οΈ Different global settings detected, resetting to defaults. '\r\n 'This may be due to an ultralytics package update. '\r\n f'View and update your global settings directly in {file}')\r\n settings = defaults # merge **defaults with **settings (prefer **settings)\r\n yaml_save(file, settings) # save updated defaults\r\n\r\n return settings\r\n\r\n\r\n# Run below code on utils init -----------------------------------------------------------------------------------------\r\n\r\n# Set logger\r\nset_logging(LOGGING_NAME) # run before defining LOGGER\r\nLOGGER = logging.getLogger(LOGGING_NAME) # define globally (used in train.py, val.py, detect.py, etc.)\r\nif platform.system() == 'Windows':\r\n for fn in LOGGER.info, LOGGER.warning:\r\n setattr(LOGGER, fn.__name__, lambda x: fn(emojis(x))) # emoji safe logging\r\n\r\n# Check first-install steps\r\nSETTINGS = get_settings()\r\nDATASETS_DIR = Path(SETTINGS['datasets_dir']) # global datasets directory\r\n\r\n\r\ndef set_settings(kwargs, file=USER_CONFIG_DIR / 'settings.yaml'):\r\n \"\"\"\r\n Function that runs on a first-time ultralytics package installation to set up global settings and create necessary\r\n directories.\r\n \"\"\"\r\n SETTINGS.update(kwargs)\r\n\r\n yaml_save(file, SETTINGS)\r\n"
},
{
"path": "yolo/utils/autobatch.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nAuto-batch utils\r\n\"\"\"\r\n\r\nfrom copy import deepcopy\r\n\r\nimport numpy as np\r\nimport torch\r\n\r\nfrom yolo.utils import LOGGER, colorstr\r\nfrom yolo.utils.torch_utils import profile\r\n\r\n\r\ndef check_train_batch_size(model, imgsz=640, amp=True):\r\n # Check YOLOv5 training batch size\r\n with torch.cuda.amp.autocast(amp):\r\n return autobatch(deepcopy(model).train(), imgsz) # compute optimal batch size\r\n\r\n\r\ndef autobatch(model, imgsz=640, fraction=0.7, batch_size=16):\r\n # Automatically estimate best YOLOv5 batch size to use `fraction` of available CUDA memory\r\n # Usage:\r\n # import torch\r\n # from utils.autobatch import autobatch\r\n # model = torch.hub.load('ultralytics/yolov5', 'yolov5s', autoshape=False)\r\n # print(autobatch(model))\r\n\r\n # Check device\r\n prefix = colorstr('AutoBatch: ')\r\n LOGGER.info(f'{prefix}Computing optimal batch size for --imgsz {imgsz}')\r\n device = next(model.parameters()).device # get model device\r\n if device.type == 'cpu':\r\n LOGGER.info(f'{prefix}CUDA not detected, using default CPU batch-size {batch_size}')\r\n return batch_size\r\n if torch.backends.cudnn.benchmark:\r\n LOGGER.info(f'{prefix} β οΈ Requires torch.backends.cudnn.benchmark=False, using default batch-size {batch_size}')\r\n return batch_size\r\n\r\n # Inspect CUDA memory\r\n gb = 1 << 30 # bytes to GiB (1024 ** 3)\r\n d = str(device).upper() # 'CUDA:0'\r\n properties = torch.cuda.get_device_properties(device) # device properties\r\n t = properties.total_memory / gb # GiB total\r\n r = torch.cuda.memory_reserved(device) / gb # GiB reserved\r\n a = torch.cuda.memory_allocated(device) / gb # GiB allocated\r\n f = t - (r + a) # GiB free\r\n LOGGER.info(f'{prefix}{d} ({properties.name}) {t:.2f}G total, {r:.2f}G reserved, {a:.2f}G allocated, {f:.2f}G free')\r\n\r\n # Profile batch sizes\r\n batch_sizes = [1, 2, 4, 8, 16]\r\n try:\r\n img = [torch.empty(b, 3, imgsz, imgsz) for b in batch_sizes]\r\n results = profile(img, model, n=3, device=device)\r\n except Exception as e:\r\n LOGGER.warning(f'{prefix}{e}')\r\n\r\n # Fit a solution\r\n y = [x[2] for x in results if x] # memory [2]\r\n p = np.polyfit(batch_sizes[:len(y)], y, deg=1) # first degree polynomial fit\r\n b = int((f * fraction - p[1]) / p[0]) # y intercept (optimal batch size)\r\n if None in results: # some sizes failed\r\n i = results.index(None) # first fail index\r\n if b >= batch_sizes[i]: # y intercept above failure point\r\n b = batch_sizes[max(i - 1, 0)] # select prior safe point\r\n if b < 1 or b > 1024: # b outside of safe range\r\n b = batch_size\r\n LOGGER.warning(f'{prefix}WARNING β οΈ CUDA anomaly detected, recommend restart environment and retry command.')\r\n\r\n fraction = (np.polyval(p, b) + r + a) / t # actual fraction predicted\r\n LOGGER.info(f'{prefix}Using batch-size {b} for {d} {t * fraction:.2f}G/{t:.2f}G ({fraction * 100:.0f}%) β
')\r\n return b\r\n"
},
{
"path": "yolo/utils/callbacks/__init__.py",
"content": "from .base import add_integration_callbacks, default_callbacks\r\n"
},
{
"path": "yolo/utils/callbacks/base.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nBase callbacks\r\n\"\"\"\r\n\r\n\r\n# Trainer callbacks ----------------------------------------------------------------------------------------------------\r\ndef on_pretrain_routine_start(trainer):\r\n pass\r\n\r\n\r\ndef on_pretrain_routine_end(trainer):\r\n pass\r\n\r\n\r\ndef on_train_start(trainer):\r\n pass\r\n\r\n\r\ndef on_train_epoch_start(trainer):\r\n pass\r\n\r\n\r\ndef on_train_batch_start(trainer):\r\n pass\r\n\r\n\r\ndef optimizer_step(trainer):\r\n pass\r\n\r\n\r\ndef on_before_zero_grad(trainer):\r\n pass\r\n\r\n\r\ndef on_train_batch_end(trainer):\r\n pass\r\n\r\n\r\ndef on_train_epoch_end(trainer):\r\n pass\r\n\r\n\r\ndef on_fit_epoch_end(trainer):\r\n pass\r\n\r\n\r\ndef on_model_save(trainer):\r\n pass\r\n\r\n\r\ndef on_train_end(trainer):\r\n pass\r\n\r\n\r\ndef on_params_update(trainer):\r\n pass\r\n\r\n\r\ndef teardown(trainer):\r\n pass\r\n\r\n\r\n# Validator callbacks --------------------------------------------------------------------------------------------------\r\ndef on_val_start(validator):\r\n pass\r\n\r\n\r\ndef on_val_batch_start(validator):\r\n pass\r\n\r\n\r\ndef on_val_batch_end(validator):\r\n pass\r\n\r\n\r\ndef on_val_end(validator):\r\n pass\r\n\r\n\r\n# Predictor callbacks --------------------------------------------------------------------------------------------------\r\ndef on_predict_start(predictor):\r\n pass\r\n\r\n\r\ndef on_predict_batch_start(predictor):\r\n pass\r\n\r\n\r\ndef on_predict_batch_end(predictor):\r\n pass\r\n\r\n\r\ndef on_predict_end(predictor):\r\n pass\r\n\r\n\r\n# Exporter callbacks ---------------------------------------------------------------------------------------------------\r\ndef on_export_start(exporter):\r\n pass\r\n\r\n\r\ndef on_export_end(exporter):\r\n pass\r\n\r\n\r\ndefault_callbacks = {\r\n # Run in trainer\r\n 'on_pretrain_routine_start': on_pretrain_routine_start,\r\n 'on_pretrain_routine_end': on_pretrain_routine_end,\r\n 'on_train_start': on_train_start,\r\n 'on_train_epoch_start': on_train_epoch_start,\r\n 'on_train_batch_start': on_train_batch_start,\r\n 'optimizer_step': optimizer_step,\r\n 'on_before_zero_grad': on_before_zero_grad,\r\n 'on_train_batch_end': on_train_batch_end,\r\n 'on_train_epoch_end': on_train_epoch_end,\r\n 'on_fit_epoch_end': on_fit_epoch_end, # fit = train + val\r\n 'on_model_save': on_model_save,\r\n 'on_train_end': on_train_end,\r\n 'on_params_update': on_params_update,\r\n 'teardown': teardown,\r\n\r\n # Run in validator\r\n 'on_val_start': on_val_start,\r\n 'on_val_batch_start': on_val_batch_start,\r\n 'on_val_batch_end': on_val_batch_end,\r\n 'on_val_end': on_val_end,\r\n\r\n # Run in predictor\r\n 'on_predict_start': on_predict_start,\r\n 'on_predict_batch_start': on_predict_batch_start,\r\n 'on_predict_batch_end': on_predict_batch_end,\r\n 'on_predict_end': on_predict_end,\r\n\r\n # Run in exporter\r\n 'on_export_start': on_export_start,\r\n 'on_export_end': on_export_end}\r\n\r\n\r\ndef add_integration_callbacks(instance):\r\n from .clearml import callbacks as clearml_callbacks\r\n from .comet import callbacks as comet_callbacks\r\n from .hub import callbacks as hub_callbacks\r\n from .tensorboard import callbacks as tb_callbacks\r\n\r\n for x in clearml_callbacks, comet_callbacks, hub_callbacks, tb_callbacks:\r\n for k, v in x.items():\r\n instance.callbacks[k].append(v) # callback[name].append(func)\r\n"
},
{
"path": "yolo/utils/callbacks/clearml.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom yolo.utils.torch_utils import get_flops, get_num_params\r\n\r\ntry:\r\n import clearml\r\n from clearml import Task\r\n\r\n assert hasattr(clearml, '__version__')\r\nexcept (ImportError, AssertionError):\r\n clearml = None\r\n\r\n\r\ndef _log_images(imgs_dict, group=\"\", step=0):\r\n task = Task.current_task()\r\n if task:\r\n for k, v in imgs_dict.items():\r\n task.get_logger().report_image(group, k, step, v)\r\n\r\n\r\ndef on_pretrain_routine_start(trainer):\r\n # TODO: reuse existing task\r\n task = Task.init(project_name=trainer.args.project or \"YOLOv8\",\r\n task_name=trainer.args.name,\r\n tags=['YOLOv8'],\r\n output_uri=True,\r\n reuse_last_task_id=False,\r\n auto_connect_frameworks={'pytorch': False})\r\n task.connect(dict(trainer.args), name='General')\r\n\r\n\r\ndef on_train_epoch_end(trainer):\r\n if trainer.epoch == 1:\r\n _log_images({f.stem: str(f) for f in trainer.save_dir.glob('train_batch*.jpg')}, \"Mosaic\", trainer.epoch)\r\n\r\n\r\ndef on_fit_epoch_end(trainer):\r\n if trainer.epoch == 0:\r\n model_info = {\r\n \"Parameters\": get_num_params(trainer.model),\r\n \"GFLOPs\": round(get_flops(trainer.model), 3),\r\n \"Inference speed (ms/img)\": round(trainer.validator.speed[1], 3)}\r\n Task.current_task().connect(model_info, name='Model')\r\n\r\n\r\ndef on_train_end(trainer):\r\n Task.current_task().update_output_model(model_path=str(trainer.best),\r\n model_name=trainer.args.name,\r\n auto_delete_file=False)\r\n\r\n\r\ncallbacks = {\r\n \"on_pretrain_routine_start\": on_pretrain_routine_start,\r\n \"on_train_epoch_end\": on_train_epoch_end,\r\n \"on_fit_epoch_end\": on_fit_epoch_end,\r\n \"on_train_end\": on_train_end} if clearml else {}\r\n"
},
{
"path": "yolo/utils/callbacks/comet.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom yolo.utils.torch_utils import get_flops, get_num_params\r\n\r\ntry:\r\n import comet_ml\r\n\r\nexcept (ModuleNotFoundError, ImportError):\r\n comet_ml = None\r\n\r\n\r\ndef on_pretrain_routine_start(trainer):\r\n experiment = comet_ml.Experiment(project_name=trainer.args.project or \"YOLOv8\",)\r\n experiment.log_parameters(dict(trainer.args))\r\n\r\n\r\ndef on_train_epoch_end(trainer):\r\n experiment = comet_ml.get_global_experiment()\r\n experiment.log_metrics(trainer.label_loss_items(trainer.tloss, prefix=\"train\"), step=trainer.epoch + 1)\r\n if trainer.epoch == 1:\r\n for f in trainer.save_dir.glob('train_batch*.jpg'):\r\n experiment.log_image(f, name=f.stem, step=trainer.epoch + 1)\r\n\r\n\r\ndef on_fit_epoch_end(trainer):\r\n experiment = comet_ml.get_global_experiment()\r\n experiment.log_metrics(trainer.metrics, step=trainer.epoch + 1)\r\n if trainer.epoch == 0:\r\n model_info = {\r\n \"model/parameters\": get_num_params(trainer.model),\r\n \"model/GFLOPs\": round(get_flops(trainer.model), 3),\r\n \"model/speed(ms)\": round(trainer.validator.speed[1], 3)}\r\n experiment.log_metrics(model_info, step=trainer.epoch + 1)\r\n\r\n\r\ndef on_train_end(trainer):\r\n experiment = comet_ml.get_global_experiment()\r\n experiment.log_model(\"YOLOv8\", file_or_folder=trainer.best, file_name=\"best.pt\", overwrite=True)\r\n\r\n\r\ncallbacks = {\r\n \"on_pretrain_routine_start\": on_pretrain_routine_start,\r\n \"on_train_epoch_end\": on_train_epoch_end,\r\n \"on_fit_epoch_end\": on_fit_epoch_end,\r\n \"on_train_end\": on_train_end} if comet_ml else {}\r\n"
},
{
"path": "yolo/utils/callbacks/hub.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport json\r\nfrom time import time\r\n\r\nimport torch\r\n\r\nfrom hub.utils import PREFIX, sync_analytics\r\nfrom yolo.utils import LOGGER\r\n\r\n\r\ndef on_pretrain_routine_end(trainer):\r\n session = getattr(trainer, 'hub_session', None)\r\n if session:\r\n # Start timer for upload rate limit\r\n LOGGER.info(f\"{PREFIX}View model at https://hub.com/models/{session.model_id} π\")\r\n session.t = {'metrics': time(), 'ckpt': time()} # start timer on self.rate_limit\r\n\r\n\r\ndef on_fit_epoch_end(trainer):\r\n session = getattr(trainer, 'hub_session', None)\r\n if session:\r\n session.metrics_queue[trainer.epoch] = json.dumps(trainer.metrics) # json string\r\n if time() - session.t['metrics'] > session.rate_limits['metrics']:\r\n session.upload_metrics()\r\n session.t['metrics'] = time() # reset timer\r\n session.metrics_queue = {} # reset queue\r\n\r\n\r\ndef on_model_save(trainer):\r\n session = getattr(trainer, 'hub_session', None)\r\n if session:\r\n # Upload checkpoints with rate limiting\r\n is_best = trainer.best_fitness == trainer.fitness\r\n if time() - session.t['ckpt'] > session.rate_limits['ckpt']:\r\n LOGGER.info(f\"{PREFIX}Uploading checkpoint {session.model_id}\")\r\n session.upload_model(trainer.epoch, trainer.last, is_best)\r\n session.t['ckpt'] = time() # reset timer\r\n\r\n\r\ndef on_train_end(trainer):\r\n session = getattr(trainer, 'hub_session', None)\r\n if session:\r\n # Upload final model and metrics with exponential standoff\r\n LOGGER.info(f\"{PREFIX}Training completed successfully β
\\n\"\r\n f\"{PREFIX}Uploading final {session.model_id}\")\r\n session.upload_model(trainer.epoch, trainer.best, map=trainer.metrics['metrics/mAP50-95(B)'], final=True)\r\n session.alive = False # stop heartbeats\r\n LOGGER.info(f\"{PREFIX}View model at https://hub.com/models/{session.model_id} π\")\r\n\r\n\r\ndef on_train_start(trainer):\r\n sync_analytics(trainer.args)\r\n\r\n\r\ndef on_val_start(validator):\r\n sync_analytics(validator.args)\r\n\r\n\r\ndef on_predict_start(predictor):\r\n sync_analytics(predictor.args)\r\n\r\n\r\ndef on_export_start(exporter):\r\n sync_analytics(exporter.args)\r\n\r\n\r\ncallbacks = {\r\n \"on_pretrain_routine_end\": on_pretrain_routine_end,\r\n \"on_fit_epoch_end\": on_fit_epoch_end,\r\n \"on_model_save\": on_model_save,\r\n \"on_train_end\": on_train_end,\r\n \"on_train_start\": on_train_start,\r\n \"on_val_start\": on_val_start,\r\n \"on_predict_start\": on_predict_start,\r\n \"on_export_start\": on_export_start}\r\n"
},
{
"path": "yolo/utils/callbacks/tensorboard.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom torch.utils.tensorboard import SummaryWriter\r\n\r\nwriter = None # TensorBoard SummaryWriter instance\r\n\r\n\r\ndef _log_scalars(scalars, step=0):\r\n for k, v in scalars.items():\r\n writer.add_scalar(k, v, step)\r\n\r\n\r\ndef on_pretrain_routine_start(trainer):\r\n global writer\r\n writer = SummaryWriter(str(trainer.save_dir))\r\n\r\n\r\ndef on_batch_end(trainer):\r\n _log_scalars(trainer.label_loss_items(trainer.tloss, prefix=\"train\"), trainer.epoch + 1)\r\n\r\n\r\ndef on_fit_epoch_end(trainer):\r\n _log_scalars(trainer.metrics, trainer.epoch + 1)\r\n\r\n\r\ncallbacks = {\r\n \"on_pretrain_routine_start\": on_pretrain_routine_start,\r\n \"on_fit_epoch_end\": on_fit_epoch_end,\r\n \"on_batch_end\": on_batch_end}\r\n"
},
{
"path": "yolo/utils/checks.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport glob\r\nimport inspect\r\nimport math\r\nimport platform\r\nimport urllib\r\nfrom pathlib import Path\r\nfrom subprocess import check_output\r\nfrom typing import Optional\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport pkg_resources as pkg\r\nimport torch\r\n\r\nfrom yolo.utils import (AUTOINSTALL, FONT, LOGGER, ROOT, USER_CONFIG_DIR, TryExcept, colorstr, emojis,\r\n is_docker, is_jupyter_notebook)\r\n\r\n\r\ndef is_ascii(s) -> bool:\r\n \"\"\"\r\n Check if a string is composed of only ASCII characters.\r\n\r\n Args:\r\n s (str): String to be checked.\r\n\r\n Returns:\r\n bool: True if the string is composed only of ASCII characters, False otherwise.\r\n \"\"\"\r\n # Convert list, tuple, None, etc. to string\r\n s = str(s)\r\n\r\n # Check if the string is composed of only ASCII characters\r\n return all(ord(c) < 128 for c in s)\r\n\r\n\r\ndef check_imgsz(imgsz, stride=32, min_dim=1, floor=0):\r\n \"\"\"\r\n Verify image size is a multiple of the given stride in each dimension. If the image size is not a multiple of the\r\n stride, update it to the nearest multiple of the stride that is greater than or equal to the given floor value.\r\n\r\n Args:\r\n imgsz (int or List[int]): Image size.\r\n stride (int): Stride value.\r\n min_dim (int): Minimum number of dimensions.\r\n floor (int): Minimum allowed value for image size.\r\n\r\n Returns:\r\n List[int]: Updated image size.\r\n \"\"\"\r\n # Convert stride to integer if it is a tensor\r\n stride = int(stride.max() if isinstance(stride, torch.Tensor) else stride)\r\n\r\n # Convert image size to list if it is an integer\r\n if isinstance(imgsz, int):\r\n imgsz = [imgsz]\r\n\r\n # Make image size a multiple of the stride\r\n sz = [max(math.ceil(x / stride) * stride, floor) for x in imgsz]\r\n\r\n # Print warning message if image size was updated\r\n if sz != imgsz:\r\n LOGGER.warning(f'WARNING β οΈ --img-size {imgsz} must be multiple of max stride {stride}, updating to {sz}')\r\n\r\n # Add missing dimensions if necessary\r\n sz = [sz[0], sz[0]] if min_dim == 2 and len(sz) == 1 else sz[0] if min_dim == 1 and len(sz) == 1 else sz\r\n\r\n return sz\r\n\r\n\r\ndef check_version(current: str = \"0.0.0\",\r\n minimum: str = \"0.0.0\",\r\n name: str = \"version \",\r\n pinned: bool = False,\r\n hard: bool = False,\r\n verbose: bool = False) -> bool:\r\n \"\"\"\r\n Check current version against the required minimum version.\r\n\r\n Args:\r\n current (str): Current version.\r\n minimum (str): Required minimum version.\r\n name (str): Name to be used in warning message.\r\n pinned (bool): If True, versions must match exactly. If False, minimum version must be satisfied.\r\n hard (bool): If True, raise an AssertionError if the minimum version is not met.\r\n verbose (bool): If True, print warning message if minimum version is not met.\r\n\r\n Returns:\r\n bool: True if minimum version is met, False otherwise.\r\n \"\"\"\r\n from pkg_resources import parse_version\r\n current, minimum = (parse_version(x) for x in (current, minimum))\r\n result = (current == minimum) if pinned else (current >= minimum) # bool\r\n warning_message = f\"WARNING β οΈ {name}{minimum} is required by YOLOv5, but {name}{current} is currently installed\"\r\n if hard:\r\n assert result, emojis(warning_message) # assert min requirements met\r\n if verbose and not result:\r\n LOGGER.warning(warning_message)\r\n return result\r\n\r\n\r\ndef check_font(font: str = FONT, progress: bool = False) -> None:\r\n \"\"\"\r\n Download font file to the user's configuration directory if it does not already exist.\r\n\r\n Args:\r\n font (str): Path to font file.\r\n progress (bool): If True, display a progress bar during the download.\r\n\r\n Returns:\r\n None\r\n \"\"\"\r\n font = Path(font)\r\n\r\n # Destination path for the font file\r\n file = USER_CONFIG_DIR / font.name\r\n\r\n # Check if font file exists at the source or destination path\r\n if not font.exists() and not file.exists():\r\n # Download font file\r\n url = f'https://com/assets/{font.name}'\r\n LOGGER.info(f'Downloading {url} to {file}...')\r\n torch.hub.download_url_to_file(url, str(file), progress=progress)\r\n\r\n\r\ndef check_online() -> bool:\r\n \"\"\"\r\n Check internet connectivity by attempting to connect to a known online host.\r\n\r\n Returns:\r\n bool: True if connection is successful, False otherwise.\r\n \"\"\"\r\n import socket\r\n try:\r\n # Check host accessibility by attempting to establish a connection\r\n socket.create_connection((\"1.1.1.1\", 443), timeout=5)\r\n return True\r\n except OSError:\r\n return False\r\n\r\n\r\ndef check_python(minimum: str = '3.7.0') -> bool:\r\n \"\"\"\r\n Check current python version against the required minimum version.\r\n\r\n Args:\r\n minimum (str): Required minimum version of python.\r\n\r\n Returns:\r\n None\r\n \"\"\"\r\n check_version(platform.python_version(), minimum, name='Python ', hard=True)\r\n\r\n\r\n@TryExcept()\r\ndef check_requirements(requirements=ROOT.parent / 'requirements.txt', exclude=(), install=True, cmds=''):\r\n # Check installed dependencies meet YOLOv5 requirements (pass *.txt file or list of packages or single package str)\r\n prefix = colorstr('red', 'bold', 'requirements:')\r\n check_python() # check python version\r\n if isinstance(requirements, Path): # requirements.txt file\r\n file = requirements.resolve()\r\n assert file.exists(), f\"{prefix} {file} not found, check failed.\"\r\n with file.open() as f:\r\n requirements = [f'{x.name}{x.specifier}' for x in pkg.parse_requirements(f) if x.name not in exclude]\r\n elif isinstance(requirements, str):\r\n requirements = [requirements]\r\n\r\n s = ''\r\n n = 0\r\n for r in requirements:\r\n try:\r\n pkg.require(r)\r\n except (pkg.VersionConflict, pkg.DistributionNotFound): # exception if requirements not met\r\n s += f'\"{r}\" '\r\n n += 1\r\n\r\n if s and install and AUTOINSTALL: # check environment variable\r\n LOGGER.info(f\"{prefix} YOLOv5 requirement{'s' * (n > 1)} {s}not found, attempting AutoUpdate...\")\r\n try:\r\n assert check_online(), \"AutoUpdate skipped (offline)\"\r\n LOGGER.info(check_output(f'pip install {s} {cmds}', shell=True).decode())\r\n source = file if 'file' in locals() else requirements\r\n s = f\"{prefix} {n} package{'s' * (n > 1)} updated per {source}\\n\" \\\r\n f\"{prefix} β οΈ {colorstr('bold', 'Restart runtime or rerun command for updates to take effect')}\\n\"\r\n LOGGER.info(s)\r\n except Exception as e:\r\n LOGGER.warning(f'{prefix} β {e}')\r\n\r\n\r\ndef check_suffix(file='yolov8n.pt', suffix=('.pt',), msg=''):\r\n # Check file(s) for acceptable suffix\r\n if file and suffix:\r\n if isinstance(suffix, str):\r\n suffix = [suffix]\r\n for f in file if isinstance(file, (list, tuple)) else [file]:\r\n s = Path(f).suffix.lower() # file suffix\r\n if len(s):\r\n assert s in suffix, f\"{msg}{f} acceptable suffix is {suffix}\"\r\n\r\n\r\ndef check_file(file, suffix=''):\r\n # Search/download file (if necessary) and return path\r\n check_suffix(file, suffix) # optional\r\n file = str(file) # convert to str()\r\n if Path(file).is_file() or not file: # exists\r\n return file\r\n elif file.startswith(('http:/', 'https:/')): # download\r\n url = file # warning: Pathlib turns :// -> :/\r\n file = Path(urllib.parse.unquote(file).split('?')[0]).name # '%2F' to '/', split https://url.com/file.txt?auth\r\n if Path(file).is_file():\r\n LOGGER.info(f'Found {url} locally at {file}') # file already exists\r\n else:\r\n LOGGER.info(f'Downloading {url} to {file}...')\r\n torch.hub.download_url_to_file(url, file)\r\n assert Path(file).exists() and Path(file).stat().st_size > 0, f'File download failed: {url}' # check\r\n return file\r\n else: # search\r\n files = []\r\n for d in 'models', 'yolo/data': # search directories\r\n files.extend(glob.glob(str(ROOT / d / '**' / file), recursive=True)) # find file\r\n assert len(files), f'File not found: {file}' # assert file was found\r\n assert len(files) == 1, f\"Multiple files match '{file}', specify exact path: {files}\" # assert unique\r\n return files[0] # return file\r\n\r\n\r\ndef check_yaml(file, suffix=('.yaml', '.yml')):\r\n # Search/download YAML file (if necessary) and return path, checking suffix\r\n return check_file(file, suffix)\r\n\r\n\r\ndef check_imshow(warn=False):\r\n # Check if environment supports image displays\r\n try:\r\n assert not is_jupyter_notebook()\r\n assert not is_docker()\r\n cv2.imshow('test', np.zeros((1, 1, 3)))\r\n cv2.waitKey(1)\r\n cv2.destroyAllWindows()\r\n cv2.waitKey(1)\r\n return True\r\n except Exception as e:\r\n if warn:\r\n LOGGER.warning(f'WARNING β οΈ Environment does not support cv2.imshow() or PIL Image.show()\\n{e}')\r\n return False\r\n\r\n\r\ndef git_describe(path=ROOT): # path must be a directory\r\n # Return human-readable git description, i.e. v5.0-5-g3e25f1e https://git-scm.com/docs/git-describe\r\n try:\r\n assert (Path(path) / '.git').is_dir()\r\n return check_output(f'git -C {path} describe --tags --long --always', shell=True).decode()[:-1]\r\n except Exception:\r\n return ''\r\n\r\n\r\ndef print_args(args: Optional[dict] = None, show_file=True, show_func=False):\r\n # Print function arguments (optional args dict)\r\n x = inspect.currentframe().f_back # previous frame\r\n file, _, func, _, _ = inspect.getframeinfo(x)\r\n if args is None: # get args automatically\r\n args, _, _, frm = inspect.getargvalues(x)\r\n args = {k: v for k, v in frm.items() if k in args}\r\n try:\r\n file = Path(file).resolve().relative_to(ROOT).with_suffix('')\r\n except ValueError:\r\n file = Path(file).stem\r\n s = (f'{file}: ' if show_file else '') + (f'{func}: ' if show_func else '')\r\n LOGGER.info(colorstr(s) + ', '.join(f'{k}={v}' for k, v in args.items()))\r\n"
},
{
"path": "yolo/utils/dist.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport os\r\nimport shutil\r\nimport socket\r\nimport sys\r\nimport tempfile\r\n\r\nfrom . import USER_CONFIG_DIR\r\n\r\n\r\ndef find_free_network_port() -> int:\r\n # https://github.com/Lightning-AI/lightning/blob/master/src/lightning_lite/plugins/environments/lightning.py\r\n \"\"\"Finds a free port on localhost.\r\n\r\n It is useful in single-node training when we don't want to connect to a real main node but have to set the\r\n `MASTER_PORT` environment variable.\r\n \"\"\"\r\n s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\r\n s.bind((\"\", 0))\r\n port = s.getsockname()[1]\r\n s.close()\r\n return port\r\n\r\n\r\ndef generate_ddp_file(trainer):\r\n import_path = '.'.join(str(trainer.__class__).split(\".\")[1:-1])\r\n\r\n if not trainer.resume:\r\n shutil.rmtree(trainer.save_dir) # remove the save_dir\r\n content = f'''config = {dict(trainer.args)} \\nif __name__ == \"__main__\":\r\n from {import_path} import {trainer.__class__.__name__}\r\n\r\n trainer = {trainer.__class__.__name__}(config=config)\r\n trainer.train()'''\r\n (USER_CONFIG_DIR / 'DDP').mkdir(exist_ok=True)\r\n with tempfile.NamedTemporaryFile(prefix=\"_temp_\",\r\n suffix=f\"{id(trainer)}.py\",\r\n mode=\"w+\",\r\n encoding='utf-8',\r\n dir=USER_CONFIG_DIR / 'DDP',\r\n delete=False) as file:\r\n file.write(content)\r\n return file.name\r\n\r\n\r\ndef generate_ddp_command(world_size, trainer):\r\n import __main__ # noqa local import to avoid https://github.com/Lightning-AI/lightning/issues/15218\r\n file_name = os.path.abspath(sys.argv[0])\r\n using_cli = not file_name.endswith(\".py\")\r\n if using_cli:\r\n file_name = generate_ddp_file(trainer)\r\n return [\r\n sys.executable, \"-m\", \"torch.distributed.run\", \"--nproc_per_node\", f\"{world_size}\", \"--master_port\",\r\n f\"{find_free_network_port()}\", file_name] + sys.argv[1:]\r\n\r\n\r\ndef ddp_cleanup(command, trainer):\r\n # delete temp file if created\r\n tempfile_suffix = f\"{id(trainer)}.py\"\r\n if tempfile_suffix in \"\".join(command):\r\n for chunk in command:\r\n if tempfile_suffix in chunk:\r\n os.remove(chunk)\r\n break\r\n"
},
{
"path": "yolo/utils/downloads.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport logging\r\nimport os\r\nimport subprocess\r\nimport urllib\r\nfrom itertools import repeat\r\nfrom multiprocessing.pool import ThreadPool\r\nfrom pathlib import Path\r\nfrom zipfile import ZipFile\r\n\r\nimport requests\r\nimport torch\r\n\r\nfrom yolo.utils import LOGGER\r\n\r\n\r\ndef safe_download(file, url, url2=None, min_bytes=1E0, error_msg=''):\r\n # Attempts to download file from url or url2, checks and removes incomplete downloads < min_bytes\r\n file = Path(file)\r\n assert_msg = f\"Downloaded file '{file}' does not exist or size is < min_bytes={min_bytes}\"\r\n try: # url1\r\n LOGGER.info(f'Downloading {url} to {file}...')\r\n torch.hub.download_url_to_file(url, str(file), progress=LOGGER.level <= logging.INFO)\r\n assert file.exists() and file.stat().st_size > min_bytes, assert_msg # check\r\n except Exception as e: # url2\r\n if file.exists():\r\n file.unlink() # remove partial downloads\r\n LOGGER.info(f'ERROR: {e}\\nRe-attempting {url2 or url} to {file}...')\r\n os.system(f\"curl -# -L '{url2 or url}' -o '{file}' --retry 3 -C -\") # curl download, retry and resume on fail\r\n finally:\r\n if not file.exists() or file.stat().st_size < min_bytes: # check\r\n if file.exists():\r\n file.unlink() # remove partial downloads\r\n LOGGER.info(f\"ERROR: {assert_msg}\\n{error_msg}\")\r\n LOGGER.info('')\r\n\r\n\r\ndef is_url(url, check=True):\r\n # Check if string is URL and check if URL exists\r\n try:\r\n url = str(url)\r\n result = urllib.parse.urlparse(url)\r\n assert all([result.scheme, result.netloc]) # check if is url\r\n return (urllib.request.urlopen(url).getcode() == 200) if check else True # check if exists online\r\n except (AssertionError, urllib.request.HTTPError):\r\n return False\r\n\r\n\r\ndef attempt_download(file, repo='ultralytics/assets', release='v0.0.0'):\r\n # Attempt file download from GitHub release assets if not found locally. release = 'latest', 'v6.2', etc.\r\n\r\n def github_assets(repository, version='latest'):\r\n # Return GitHub repo tag and assets (i.e. ['yolov8n.pt', 'yolov5m.pt', ...])\r\n # Return GitHub repo tag and assets (i.e. ['yolov8n.pt', 'yolov8s.pt', ...])\r\n if version != 'latest':\r\n version = f'tags/{version}' # i.e. tags/v6.2\r\n response = requests.get(f'https://api.github.com/repos/{repository}/releases/{version}').json() # github api\r\n return response['tag_name'], [x['name'] for x in response['assets']] # tag, assets\r\n\r\n file = Path(str(file).strip().replace(\"'\", ''))\r\n if not file.exists():\r\n # URL specified\r\n name = Path(urllib.parse.unquote(str(file))).name # decode '%2F' to '/' etc.\r\n if str(file).startswith(('http:/', 'https:/')): # download\r\n url = str(file).replace(':/', '://') # Pathlib turns :// -> :/\r\n file = name.split('?')[0] # parse authentication https://url.com/file.txt?auth...\r\n if Path(file).is_file():\r\n LOGGER.info(f'Found {url} locally at {file}') # file already exists\r\n else:\r\n safe_download(file=file, url=url, min_bytes=1E5)\r\n return file\r\n\r\n # GitHub assets\r\n assets = [f'yolov5{size}{suffix}.pt' for size in 'nsmlx' for suffix in ('', '6', '-cls', '-seg')] # default\r\n assets = [f'yolov8{size}{suffix}.pt' for size in 'nsmlx' for suffix in ('', '6', '-cls', '-seg')] # default\r\n try:\r\n tag, assets = github_assets(repo, release)\r\n except Exception:\r\n try:\r\n tag, assets = github_assets(repo) # latest release\r\n except Exception:\r\n try:\r\n tag = subprocess.check_output('git tag', shell=True, stderr=subprocess.STDOUT).decode().split()[-1]\r\n except Exception:\r\n tag = release\r\n\r\n file.parent.mkdir(parents=True, exist_ok=True) # make parent dir (if required)\r\n if name in assets:\r\n url3 = 'https://drive.google.com/drive/folders/1EFQTEUeXWSFww0luse2jB9M1QNZQGwNl' # backup gdrive mirror\r\n safe_download(\r\n file,\r\n url=f'https://github.com/{repo}/releases/download/{tag}/{name}',\r\n min_bytes=1E5,\r\n error_msg=f'{file} missing, try downloading from https://github.com/{repo}/releases/{tag} or {url3}')\r\n\r\n return str(file)\r\n\r\n\r\ndef download(url, dir=Path.cwd(), unzip=True, delete=True, curl=False, threads=1, retry=3):\r\n # Multithreaded file download and unzip function, used in data.yaml for autodownload\r\n def download_one(url, dir):\r\n # Download 1 file\r\n success = True\r\n if Path(url).is_file():\r\n f = Path(url) # filename\r\n else: # does not exist\r\n f = dir / Path(url).name\r\n LOGGER.info(f'Downloading {url} to {f}...')\r\n for i in range(retry + 1):\r\n if curl:\r\n s = 'sS' if threads > 1 else '' # silent\r\n r = os.system(\r\n f'curl -# -{s}L \"{url}\" -o \"{f}\" --retry 9 -C -') # curl download with retry, continue\r\n success = r == 0\r\n else:\r\n torch.hub.download_url_to_file(url, f, progress=threads == 1) # torch download\r\n success = f.is_file()\r\n if success:\r\n break\r\n elif i < retry:\r\n LOGGER.warning(f'β οΈ Download failure, retrying {i + 1}/{retry} {url}...')\r\n else:\r\n LOGGER.warning(f'β Failed to download {url}...')\r\n\r\n if unzip and success and f.suffix in ('.zip', '.tar', '.gz'):\r\n LOGGER.info(f'Unzipping {f}...')\r\n if f.suffix == '.zip':\r\n ZipFile(f).extractall(path=dir) # unzip\r\n elif f.suffix == '.tar':\r\n os.system(f'tar xf {f} --directory {f.parent}') # unzip\r\n elif f.suffix == '.gz':\r\n os.system(f'tar xfz {f} --directory {f.parent}') # unzip\r\n if delete:\r\n f.unlink() # remove zip\r\n\r\n dir = Path(dir)\r\n dir.mkdir(parents=True, exist_ok=True) # make directory\r\n if threads > 1:\r\n pool = ThreadPool(threads)\r\n pool.imap(lambda x: download_one(*x), zip(url, repeat(dir))) # multithreaded\r\n pool.close()\r\n pool.join()\r\n else:\r\n for u in [url] if isinstance(url, (str, Path)) else url:\r\n download_one(u, dir)\r\n"
},
{
"path": "yolo/utils/files.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport contextlib\r\nimport glob\r\nimport os\r\nimport urllib\r\nfrom datetime import datetime\r\nfrom pathlib import Path\r\nfrom zipfile import ZipFile\r\n\r\n\r\nclass WorkingDirectory(contextlib.ContextDecorator):\r\n # Usage: @WorkingDirectory(dir) decorator or 'with WorkingDirectory(dir):' context manager\r\n def __init__(self, new_dir):\r\n self.dir = new_dir # new dir\r\n self.cwd = Path.cwd().resolve() # current dir\r\n\r\n def __enter__(self):\r\n os.chdir(self.dir)\r\n\r\n def __exit__(self, exc_type, exc_val, exc_tb):\r\n os.chdir(self.cwd)\r\n\r\n\r\ndef increment_path(path, exist_ok=False, sep='', mkdir=False):\r\n \"\"\"\r\n Increments a file or directory path, i.e. runs/exp --> runs/exp{sep}2, runs/exp{sep}3, ... etc.\r\n\r\n If the path exists and exist_ok is not set to True, the path will be incremented by appending a number and sep to\r\n the end of the path. If the path is a file, the file extension will be preserved. If the path is a directory, the\r\n number will be appended directly to the end of the path. If mkdir is set to True, the path will be created as a\r\n directory if it does not already exist.\r\n\r\n Args:\r\n path (str or pathlib.Path): Path to increment.\r\n exist_ok (bool, optional): If True, the path will not be incremented and will be returned as-is. Defaults to False.\r\n sep (str, optional): Separator to use between the path and the incrementation number. Defaults to an empty string.\r\n mkdir (bool, optional): If True, the path will be created as a directory if it does not exist. Defaults to False.\r\n\r\n Returns:\r\n pathlib.Path: Incremented path.\r\n \"\"\"\r\n path = Path(path) # os-agnostic\r\n if path.exists() and not exist_ok:\r\n path, suffix = (path.with_suffix(''), path.suffix) if path.is_file() else (path, '')\r\n\r\n # Method 1\r\n for n in range(2, 9999):\r\n p = f'{path}{sep}{n}{suffix}' # increment path\r\n if not os.path.exists(p): #\r\n break\r\n path = Path(p)\r\n\r\n if mkdir:\r\n path.mkdir(parents=True, exist_ok=True) # make directory\r\n\r\n return path\r\n\r\n\r\ndef unzip_file(file, path=None, exclude=('.DS_Store', '__MACOSX')):\r\n # Unzip a *.zip file to path/, excluding files containing strings in exclude list\r\n if path is None:\r\n path = Path(file).parent # default path\r\n with ZipFile(file) as zipObj:\r\n for f in zipObj.namelist(): # list all archived filenames in the zip\r\n if all(x not in f for x in exclude):\r\n zipObj.extract(f, path=path)\r\n\r\n\r\ndef file_age(path=__file__):\r\n # Return days since last file update\r\n dt = (datetime.now() - datetime.fromtimestamp(Path(path).stat().st_mtime)) # delta\r\n return dt.days # + dt.seconds / 86400 # fractional days\r\n\r\n\r\ndef file_date(path=__file__):\r\n # Return human-readable file modification date, i.e. '2021-3-26'\r\n t = datetime.fromtimestamp(Path(path).stat().st_mtime)\r\n return f'{t.year}-{t.month}-{t.day}'\r\n\r\n\r\ndef file_size(path):\r\n # Return file/dir size (MB)\r\n mb = 1 << 20 # bytes to MiB (1024 ** 2)\r\n path = Path(path)\r\n if path.is_file():\r\n return path.stat().st_size / mb\r\n elif path.is_dir():\r\n return sum(f.stat().st_size for f in path.glob('**/*') if f.is_file()) / mb\r\n else:\r\n return 0.0\r\n\r\n\r\ndef url2file(url):\r\n # Convert URL to filename, i.e. https://url.com/file.txt?auth -> file.txt\r\n url = str(Path(url)).replace(':/', '://') # Pathlib turns :// -> :/\r\n return Path(urllib.parse.unquote(url)).name.split('?')[0] # '%2F' to '/', split https://url.com/file.txt?auth\r\n\r\n\r\ndef get_latest_run(search_dir='.'):\r\n # Return path to most recent 'last.pt' in /runs (i.e. to --resume from)\r\n last_list = glob.glob(f'{search_dir}/**/last*.pt', recursive=True)\r\n return max(last_list, key=os.path.getctime) if last_list else ''\r\n"
},
{
"path": "yolo/utils/instance.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom collections import abc\r\nfrom itertools import repeat\r\nfrom numbers import Number\r\nfrom typing import List\r\n\r\nimport numpy as np\r\n\r\nfrom .ops import ltwh2xywh, ltwh2xyxy, resample_segments, xywh2ltwh, xywh2xyxy, xyxy2ltwh, xyxy2xywh\r\n\r\n\r\n# From PyTorch internals\r\ndef _ntuple(n):\r\n\r\n def parse(x):\r\n return x if isinstance(x, abc.Iterable) else tuple(repeat(x, n))\r\n\r\n return parse\r\n\r\n\r\nto_4tuple = _ntuple(4)\r\n\r\n# `xyxy` means left top and right bottom\r\n# `xywh` means center x, center y and width, height(yolo format)\r\n# `ltwh` means left top and width, height(coco format)\r\n_formats = [\"xyxy\", \"xywh\", \"ltwh\"]\r\n\r\n__all__ = [\"Bboxes\"]\r\n\r\n\r\nclass Bboxes:\r\n \"\"\"Now only numpy is supported\"\"\"\r\n\r\n def __init__(self, bboxes, format=\"xyxy\") -> None:\r\n assert format in _formats\r\n bboxes = bboxes[None, :] if bboxes.ndim == 1 else bboxes\r\n assert bboxes.ndim == 2\r\n assert bboxes.shape[1] == 4\r\n self.bboxes = bboxes\r\n self.format = format\r\n # self.normalized = normalized\r\n\r\n # def convert(self, format):\r\n # assert format in _formats\r\n # if self.format == format:\r\n # bboxes = self.bboxes\r\n # elif self.format == \"xyxy\":\r\n # if format == \"xywh\":\r\n # bboxes = xyxy2xywh(self.bboxes)\r\n # else:\r\n # bboxes = xyxy2ltwh(self.bboxes)\r\n # elif self.format == \"xywh\":\r\n # if format == \"xyxy\":\r\n # bboxes = xywh2xyxy(self.bboxes)\r\n # else:\r\n # bboxes = xywh2ltwh(self.bboxes)\r\n # else:\r\n # if format == \"xyxy\":\r\n # bboxes = ltwh2xyxy(self.bboxes)\r\n # else:\r\n # bboxes = ltwh2xywh(self.bboxes)\r\n #\r\n # return Bboxes(bboxes, format)\r\n\r\n def convert(self, format):\r\n assert format in _formats\r\n if self.format == format:\r\n return\r\n elif self.format == \"xyxy\":\r\n bboxes = xyxy2xywh(self.bboxes) if format == \"xywh\" else xyxy2ltwh(self.bboxes)\r\n elif self.format == \"xywh\":\r\n bboxes = xywh2xyxy(self.bboxes) if format == \"xyxy\" else xywh2ltwh(self.bboxes)\r\n else:\r\n bboxes = ltwh2xyxy(self.bboxes) if format == \"xyxy\" else ltwh2xywh(self.bboxes)\r\n self.bboxes = bboxes\r\n self.format = format\r\n\r\n def areas(self):\r\n self.convert(\"xyxy\")\r\n return (self.bboxes[:, 2] - self.bboxes[:, 0]) * (self.bboxes[:, 3] - self.bboxes[:, 1])\r\n\r\n # def denormalize(self, w, h):\r\n # if not self.normalized:\r\n # return\r\n # assert (self.bboxes <= 1.0).all()\r\n # self.bboxes[:, 0::2] *= w\r\n # self.bboxes[:, 1::2] *= h\r\n # self.normalized = False\r\n #\r\n # def normalize(self, w, h):\r\n # if self.normalized:\r\n # return\r\n # assert (self.bboxes > 1.0).any()\r\n # self.bboxes[:, 0::2] /= w\r\n # self.bboxes[:, 1::2] /= h\r\n # self.normalized = True\r\n\r\n def mul(self, scale):\r\n \"\"\"\r\n Args:\r\n scale (tuple | List | int): the scale for four coords.\r\n \"\"\"\r\n if isinstance(scale, Number):\r\n scale = to_4tuple(scale)\r\n assert isinstance(scale, (tuple, list))\r\n assert len(scale) == 4\r\n self.bboxes[:, 0] *= scale[0]\r\n self.bboxes[:, 1] *= scale[1]\r\n self.bboxes[:, 2] *= scale[2]\r\n self.bboxes[:, 3] *= scale[3]\r\n\r\n def add(self, offset):\r\n \"\"\"\r\n Args:\r\n offset (tuple | List | int): the offset for four coords.\r\n \"\"\"\r\n if isinstance(offset, Number):\r\n offset = to_4tuple(offset)\r\n assert isinstance(offset, (tuple, list))\r\n assert len(offset) == 4\r\n self.bboxes[:, 0] += offset[0]\r\n self.bboxes[:, 1] += offset[1]\r\n self.bboxes[:, 2] += offset[2]\r\n self.bboxes[:, 3] += offset[3]\r\n\r\n def __len__(self):\r\n return len(self.bboxes)\r\n\r\n @classmethod\r\n def concatenate(cls, boxes_list: List[\"Bboxes\"], axis=0) -> \"Bboxes\":\r\n \"\"\"\r\n Concatenates a list of Boxes into a single Bboxes\r\n\r\n Arguments:\r\n boxes_list (list[Bboxes])\r\n\r\n Returns:\r\n Bboxes: the concatenated Boxes\r\n \"\"\"\r\n assert isinstance(boxes_list, (list, tuple))\r\n if not boxes_list:\r\n return cls(np.empty(0))\r\n assert all(isinstance(box, Bboxes) for box in boxes_list)\r\n\r\n if len(boxes_list) == 1:\r\n return boxes_list[0]\r\n return cls(np.concatenate([b.bboxes for b in boxes_list], axis=axis))\r\n\r\n def __getitem__(self, index) -> \"Bboxes\":\r\n \"\"\"\r\n Args:\r\n index: int, slice, or a BoolArray\r\n\r\n Returns:\r\n Bboxes: Create a new :class:`Bboxes` by indexing.\r\n \"\"\"\r\n if isinstance(index, int):\r\n return Bboxes(self.bboxes[index].view(1, -1))\r\n b = self.bboxes[index]\r\n assert b.ndim == 2, f\"Indexing on Bboxes with {index} failed to return a matrix!\"\r\n return Bboxes(b)\r\n\r\n\r\nclass Instances:\r\n\r\n def __init__(self, bboxes, segments=None, keypoints=None, bbox_format=\"xywh\", normalized=True) -> None:\r\n \"\"\"\r\n Args:\r\n bboxes (ndarray): bboxes with shape [N, 4].\r\n segments (list | ndarray): segments.\r\n keypoints (ndarray): keypoints with shape [N, 17, 2].\r\n \"\"\"\r\n if segments is None:\r\n segments = []\r\n self._bboxes = Bboxes(bboxes=bboxes, format=bbox_format)\r\n self.keypoints = keypoints\r\n self.normalized = normalized\r\n\r\n if len(segments) > 0:\r\n # list[np.array(1000, 2)] * num_samples\r\n segments = resample_segments(segments)\r\n # (N, 1000, 2)\r\n segments = np.stack(segments, axis=0)\r\n else:\r\n segments = np.zeros((0, 1000, 2), dtype=np.float32)\r\n self.segments = segments\r\n\r\n def convert_bbox(self, format):\r\n self._bboxes.convert(format=format)\r\n\r\n def bbox_areas(self):\r\n self._bboxes.areas()\r\n\r\n def scale(self, scale_w, scale_h, bbox_only=False):\r\n \"\"\"this might be similar with denormalize func but without normalized sign\"\"\"\r\n self._bboxes.mul(scale=(scale_w, scale_h, scale_w, scale_h))\r\n if bbox_only:\r\n return\r\n self.segments[..., 0] *= scale_w\r\n self.segments[..., 1] *= scale_h\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] *= scale_w\r\n self.keypoints[..., 1] *= scale_h\r\n\r\n def denormalize(self, w, h):\r\n if not self.normalized:\r\n return\r\n self._bboxes.mul(scale=(w, h, w, h))\r\n self.segments[..., 0] *= w\r\n self.segments[..., 1] *= h\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] *= w\r\n self.keypoints[..., 1] *= h\r\n self.normalized = False\r\n\r\n def normalize(self, w, h):\r\n if self.normalized:\r\n return\r\n self._bboxes.mul(scale=(1 / w, 1 / h, 1 / w, 1 / h))\r\n self.segments[..., 0] /= w\r\n self.segments[..., 1] /= h\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] /= w\r\n self.keypoints[..., 1] /= h\r\n self.normalized = True\r\n\r\n def add_padding(self, padw, padh):\r\n # handle rect and mosaic situation\r\n assert not self.normalized, \"you should add padding with absolute coordinates.\"\r\n self._bboxes.add(offset=(padw, padh, padw, padh))\r\n self.segments[..., 0] += padw\r\n self.segments[..., 1] += padh\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] += padw\r\n self.keypoints[..., 1] += padh\r\n\r\n def __getitem__(self, index) -> \"Instances\":\r\n \"\"\"\r\n Args:\r\n index: int, slice, or a BoolArray\r\n\r\n Returns:\r\n Instances: Create a new :class:`Instances` by indexing.\r\n \"\"\"\r\n segments = self.segments[index] if len(self.segments) else self.segments\r\n keypoints = self.keypoints[index] if self.keypoints is not None else None\r\n bboxes = self.bboxes[index]\r\n bbox_format = self._bboxes.format\r\n return Instances(\r\n bboxes=bboxes,\r\n segments=segments,\r\n keypoints=keypoints,\r\n bbox_format=bbox_format,\r\n normalized=self.normalized,\r\n )\r\n\r\n def flipud(self, h):\r\n if self._bboxes.format == \"xyxy\":\r\n y1 = self.bboxes[:, 1].copy()\r\n y2 = self.bboxes[:, 3].copy()\r\n self.bboxes[:, 1] = h - y2\r\n self.bboxes[:, 3] = h - y1\r\n else:\r\n self.bboxes[:, 1] = h - self.bboxes[:, 1]\r\n self.segments[..., 1] = h - self.segments[..., 1]\r\n if self.keypoints is not None:\r\n self.keypoints[..., 1] = h - self.keypoints[..., 1]\r\n\r\n def fliplr(self, w):\r\n if self._bboxes.format == \"xyxy\":\r\n x1 = self.bboxes[:, 0].copy()\r\n x2 = self.bboxes[:, 2].copy()\r\n self.bboxes[:, 0] = w - x2\r\n self.bboxes[:, 2] = w - x1\r\n else:\r\n self.bboxes[:, 0] = w - self.bboxes[:, 0]\r\n self.segments[..., 0] = w - self.segments[..., 0]\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] = w - self.keypoints[..., 0]\r\n\r\n def clip(self, w, h):\r\n ori_format = self._bboxes.format\r\n self.convert_bbox(format=\"xyxy\")\r\n self.bboxes[:, [0, 2]] = self.bboxes[:, [0, 2]].clip(0, w)\r\n self.bboxes[:, [1, 3]] = self.bboxes[:, [1, 3]].clip(0, h)\r\n if ori_format != \"xyxy\":\r\n self.convert_bbox(format=ori_format)\r\n self.segments[..., 0] = self.segments[..., 0].clip(0, w)\r\n self.segments[..., 1] = self.segments[..., 1].clip(0, h)\r\n if self.keypoints is not None:\r\n self.keypoints[..., 0] = self.keypoints[..., 0].clip(0, w)\r\n self.keypoints[..., 1] = self.keypoints[..., 1].clip(0, h)\r\n\r\n def update(self, bboxes, segments=None, keypoints=None):\r\n new_bboxes = Bboxes(bboxes, format=self._bboxes.format)\r\n self._bboxes = new_bboxes\r\n if segments is not None:\r\n self.segments = segments\r\n if keypoints is not None:\r\n self.keypoints = keypoints\r\n\r\n def __len__(self):\r\n return len(self.bboxes)\r\n\r\n @classmethod\r\n def concatenate(cls, instances_list: List[\"Instances\"], axis=0) -> \"Instances\":\r\n \"\"\"\r\n Concatenates a list of Boxes into a single Bboxes\r\n\r\n Arguments:\r\n instances_list (list[Bboxes])\r\n axis\r\n\r\n Returns:\r\n Boxes: the concatenated Boxes\r\n \"\"\"\r\n assert isinstance(instances_list, (list, tuple))\r\n if not instances_list:\r\n return cls(np.empty(0))\r\n assert all(isinstance(instance, Instances) for instance in instances_list)\r\n\r\n if len(instances_list) == 1:\r\n return instances_list[0]\r\n\r\n use_keypoint = instances_list[0].keypoints is not None\r\n bbox_format = instances_list[0]._bboxes.format\r\n normalized = instances_list[0].normalized\r\n\r\n cat_boxes = np.concatenate([ins.bboxes for ins in instances_list], axis=axis)\r\n cat_segments = np.concatenate([b.segments for b in instances_list], axis=axis)\r\n cat_keypoints = np.concatenate([b.keypoints for b in instances_list], axis=axis) if use_keypoint else None\r\n return cls(cat_boxes, cat_segments, cat_keypoints, bbox_format, normalized)\r\n\r\n @property\r\n def bboxes(self):\r\n return self._bboxes.bboxes\r\n"
},
{
"path": "yolo/utils/loss.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.nn.functional as F\r\n\r\nfrom .metrics import bbox_iou\r\nfrom .tal import bbox2dist\r\n\r\n\r\nclass VarifocalLoss(nn.Module):\r\n # Varifocal loss by Zhang et al. https://arxiv.org/abs/2008.13367\r\n def __init__(self):\r\n super().__init__()\r\n\r\n def forward(self, pred_score, gt_score, label, alpha=0.75, gamma=2.0):\r\n weight = alpha * pred_score.sigmoid().pow(gamma) * (1 - label) + gt_score * label\r\n with torch.cuda.amp.autocast(enabled=False):\r\n loss = (F.binary_cross_entropy_with_logits(pred_score.float(), gt_score.float(), reduction=\"none\") *\r\n weight).sum()\r\n return loss\r\n\r\n\r\nclass BboxLoss(nn.Module):\r\n\r\n def __init__(self, reg_max, use_dfl=False):\r\n super().__init__()\r\n self.reg_max = reg_max\r\n self.use_dfl = use_dfl\r\n\r\n def forward(self, pred_dist, pred_bboxes, anchor_points, target_bboxes, target_scores, target_scores_sum, fg_mask):\r\n # IoU loss\r\n weight = torch.masked_select(target_scores.sum(-1), fg_mask).unsqueeze(-1)\r\n iou = bbox_iou(pred_bboxes[fg_mask], target_bboxes[fg_mask], xywh=False, CIoU=True)\r\n loss_iou = ((1.0 - iou) * weight).sum() / target_scores_sum\r\n\r\n # DFL loss\r\n if self.use_dfl:\r\n target_ltrb = bbox2dist(anchor_points, target_bboxes, self.reg_max)\r\n loss_dfl = self._df_loss(pred_dist[fg_mask].view(-1, self.reg_max + 1), target_ltrb[fg_mask]) * weight\r\n loss_dfl = loss_dfl.sum() / target_scores_sum\r\n else:\r\n loss_dfl = torch.tensor(0.0).to(pred_dist.device)\r\n\r\n return loss_iou, loss_dfl\r\n\r\n @staticmethod\r\n def _df_loss(pred_dist, target):\r\n # Return sum of left and right DFL losses\r\n tl = target.long() # target left\r\n tr = tl + 1 # target right\r\n wl = tr - target # weight left\r\n wr = 1 - wl # weight right\r\n return (F.cross_entropy(pred_dist, tl.view(-1), reduction=\"none\").view(tl.shape) * wl +\r\n F.cross_entropy(pred_dist, tr.view(-1), reduction=\"none\").view(tl.shape) * wr).mean(-1, keepdim=True)\r\n"
},
{
"path": "yolo/utils/metrics.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\"\"\"\r\nModel validation metrics\r\n\"\"\"\r\nimport math\r\nimport warnings\r\nfrom pathlib import Path\r\n\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\nimport torch\r\nimport torch.nn as nn\r\n\r\nfrom yolo.utils import TryExcept\r\n\r\n\r\n# boxes\r\ndef box_area(box):\r\n # box = xyxy(4,n)\r\n return (box[2] - box[0]) * (box[3] - box[1])\r\n\r\n\r\ndef bbox_ioa(box1, box2, eps=1e-7):\r\n \"\"\"Returns the intersection over box2 area given box1, box2. Boxes are x1y1x2y2\r\n box1: np.array of shape(nx4)\r\n box2: np.array of shape(mx4)\r\n returns: np.array of shape(nxm)\r\n \"\"\"\r\n\r\n # Get the coordinates of bounding boxes\r\n b1_x1, b1_y1, b1_x2, b1_y2 = box1.T\r\n b2_x1, b2_y1, b2_x2, b2_y2 = box2.T\r\n\r\n # Intersection area\r\n inter_area = (np.minimum(b1_x2[:, None], b2_x2) - np.maximum(b1_x1[:, None], b2_x1)).clip(0) * \\\r\n (np.minimum(b1_y2[:, None], b2_y2) - np.maximum(b1_y1[:, None], b2_y1)).clip(0)\r\n\r\n # box2 area\r\n box2_area = (b2_x2 - b2_x1) * (b2_y2 - b2_y1) + eps\r\n\r\n # Intersection over box2 area\r\n return inter_area / box2_area\r\n\r\n\r\ndef box_iou(box1, box2, eps=1e-7):\r\n # https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py\r\n \"\"\"\r\n Return intersection-over-union (Jaccard index) of boxes.\r\n Both sets of boxes are expected to be in (x1, y1, x2, y2) format.\r\n Arguments:\r\n box1 (Tensor[N, 4])\r\n box2 (Tensor[M, 4])\r\n Returns:\r\n iou (Tensor[N, M]): the NxM matrix containing the pairwise\r\n IoU values for every element in boxes1 and boxes2\r\n \"\"\"\r\n\r\n # inter(N,M) = (rb(N,M,2) - lt(N,M,2)).clamp(0).prod(2)\r\n (a1, a2), (b1, b2) = box1.unsqueeze(1).chunk(2, 2), box2.unsqueeze(0).chunk(2, 2)\r\n inter = (torch.min(a2, b2) - torch.max(a1, b1)).clamp(0).prod(2)\r\n\r\n # IoU = inter / (area1 + area2 - inter)\r\n return inter / ((a2 - a1).prod(2) + (b2 - b1).prod(2) - inter + eps)\r\n\r\n\r\ndef bbox_iou(box1, box2, xywh=True, GIoU=False, DIoU=False, CIoU=False, eps=1e-7):\r\n # Returns Intersection over Union (IoU) of box1(1,4) to box2(n,4)\r\n\r\n # Get the coordinates of bounding boxes\r\n if xywh: # transform from xywh to xyxy\r\n (x1, y1, w1, h1), (x2, y2, w2, h2) = box1.chunk(4, -1), box2.chunk(4, -1)\r\n w1_, h1_, w2_, h2_ = w1 / 2, h1 / 2, w2 / 2, h2 / 2\r\n b1_x1, b1_x2, b1_y1, b1_y2 = x1 - w1_, x1 + w1_, y1 - h1_, y1 + h1_\r\n b2_x1, b2_x2, b2_y1, b2_y2 = x2 - w2_, x2 + w2_, y2 - h2_, y2 + h2_\r\n else: # x1, y1, x2, y2 = box1\r\n b1_x1, b1_y1, b1_x2, b1_y2 = box1.chunk(4, -1)\r\n b2_x1, b2_y1, b2_x2, b2_y2 = box2.chunk(4, -1)\r\n w1, h1 = b1_x2 - b1_x1, b1_y2 - b1_y1 + eps\r\n w2, h2 = b2_x2 - b2_x1, b2_y2 - b2_y1 + eps\r\n\r\n # Intersection area\r\n inter = (b1_x2.minimum(b2_x2) - b1_x1.maximum(b2_x1)).clamp(0) * \\\r\n (b1_y2.minimum(b2_y2) - b1_y1.maximum(b2_y1)).clamp(0)\r\n\r\n # Union Area\r\n union = w1 * h1 + w2 * h2 - inter + eps\r\n\r\n # IoU\r\n iou = inter / union\r\n if CIoU or DIoU or GIoU:\r\n cw = b1_x2.maximum(b2_x2) - b1_x1.minimum(b2_x1) # convex (smallest enclosing box) width\r\n ch = b1_y2.maximum(b2_y2) - b1_y1.minimum(b2_y1) # convex height\r\n if CIoU or DIoU: # Distance or Complete IoU https://arxiv.org/abs/1911.08287v1\r\n c2 = cw ** 2 + ch ** 2 + eps # convex diagonal squared\r\n rho2 = ((b2_x1 + b2_x2 - b1_x1 - b1_x2) ** 2 + (b2_y1 + b2_y2 - b1_y1 - b1_y2) ** 2) / 4 # center dist ** 2\r\n if CIoU: # https://github.com/Zzh-tju/DIoU-SSD-pytorch/blob/master/utils/box/box_utils.py#L47\r\n v = (4 / math.pi ** 2) * (torch.atan(w2 / h2) - torch.atan(w1 / h1)).pow(2)\r\n with torch.no_grad():\r\n alpha = v / (v - iou + (1 + eps))\r\n return iou - (rho2 / c2 + v * alpha) # CIoU\r\n return iou - rho2 / c2 # DIoU\r\n c_area = cw * ch + eps # convex area\r\n return iou - (c_area - union) / c_area # GIoU https://arxiv.org/pdf/1902.09630.pdf\r\n return iou # IoU\r\n\r\n\r\ndef mask_iou(mask1, mask2, eps=1e-7):\r\n \"\"\"\r\n mask1: [N, n] m1 means number of predicted objects\r\n mask2: [M, n] m2 means number of gt objects\r\n Note: n means image_w x image_h\r\n return: masks iou, [N, M]\r\n \"\"\"\r\n intersection = torch.matmul(mask1, mask2.t()).clamp(0)\r\n union = (mask1.sum(1)[:, None] + mask2.sum(1)[None]) - intersection # (area1 + area2) - intersection\r\n return intersection / (union + eps)\r\n\r\n\r\ndef masks_iou(mask1, mask2, eps=1e-7):\r\n \"\"\"\r\n mask1: [N, n] m1 means number of predicted objects\r\n mask2: [N, n] m2 means number of gt objects\r\n Note: n means image_w x image_h\r\n return: masks iou, (N, )\r\n \"\"\"\r\n intersection = (mask1 * mask2).sum(1).clamp(0) # (N, )\r\n union = (mask1.sum(1) + mask2.sum(1))[None] - intersection # (area1 + area2) - intersection\r\n return intersection / (union + eps)\r\n\r\n\r\ndef smooth_BCE(eps=0.1): # https://github.com/ultralytics/yolov3/issues/238#issuecomment-598028441\r\n # return positive, negative label smoothing BCE targets\r\n return 1.0 - 0.5 * eps, 0.5 * eps\r\n\r\n\r\n# losses\r\nclass FocalLoss(nn.Module):\r\n # Wraps focal loss around existing loss_fcn(), i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5)\r\n def __init__(self, loss_fcn, gamma=1.5, alpha=0.25):\r\n super().__init__()\r\n self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss()\r\n self.gamma = gamma\r\n self.alpha = alpha\r\n self.reduction = loss_fcn.reduction\r\n self.loss_fcn.reduction = 'none' # required to apply FL to each element\r\n\r\n def forward(self, pred, true):\r\n loss = self.loss_fcn(pred, true)\r\n # p_t = torch.exp(-loss)\r\n # loss *= self.alpha * (1.000001 - p_t) ** self.gamma # non-zero power for gradient stability\r\n\r\n # TF implementation https://github.com/tensorflow/addons/blob/v0.7.1/tensorflow_addons/losses/focal_loss.py\r\n pred_prob = torch.sigmoid(pred) # prob from logits\r\n p_t = true * pred_prob + (1 - true) * (1 - pred_prob)\r\n alpha_factor = true * self.alpha + (1 - true) * (1 - self.alpha)\r\n modulating_factor = (1.0 - p_t) ** self.gamma\r\n loss *= alpha_factor * modulating_factor\r\n\r\n if self.reduction == 'mean':\r\n return loss.mean()\r\n elif self.reduction == 'sum':\r\n return loss.sum()\r\n else: # 'none'\r\n return loss\r\n\r\n\r\nclass ConfusionMatrix:\r\n # Updated version of https://github.com/kaanakan/object_detection_confusion_matrix\r\n def __init__(self, nc, conf=0.25, iou_thres=0.45):\r\n self.matrix = np.zeros((nc + 1, nc + 1))\r\n self.nc = nc # number of classes\r\n self.conf = conf\r\n self.iou_thres = iou_thres\r\n\r\n def process_batch(self, detections, labels):\r\n \"\"\"\r\n Return intersection-over-union (Jaccard index) of boxes.\r\n Both sets of boxes are expected to be in (x1, y1, x2, y2) format.\r\n Arguments:\r\n detections (Array[N, 6]), x1, y1, x2, y2, conf, class\r\n labels (Array[M, 5]), class, x1, y1, x2, y2\r\n Returns:\r\n None, updates confusion matrix accordingly\r\n \"\"\"\r\n if detections is None:\r\n gt_classes = labels.int()\r\n for gc in gt_classes:\r\n self.matrix[self.nc, gc] += 1 # background FN\r\n return\r\n\r\n detections = detections[detections[:, 4] > self.conf]\r\n gt_classes = labels[:, 0].int()\r\n detection_classes = detections[:, 5].int()\r\n iou = box_iou(labels[:, 1:], detections[:, :4])\r\n\r\n x = torch.where(iou > self.iou_thres)\r\n if x[0].shape[0]:\r\n matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy()\r\n if x[0].shape[0] > 1:\r\n matches = matches[matches[:, 2].argsort()[::-1]]\r\n matches = matches[np.unique(matches[:, 1], return_index=True)[1]]\r\n matches = matches[matches[:, 2].argsort()[::-1]]\r\n matches = matches[np.unique(matches[:, 0], return_index=True)[1]]\r\n else:\r\n matches = np.zeros((0, 3))\r\n\r\n n = matches.shape[0] > 0\r\n m0, m1, _ = matches.transpose().astype(int)\r\n for i, gc in enumerate(gt_classes):\r\n j = m0 == i\r\n if n and sum(j) == 1:\r\n self.matrix[detection_classes[m1[j]], gc] += 1 # correct\r\n else:\r\n self.matrix[self.nc, gc] += 1 # true background\r\n\r\n if n:\r\n for i, dc in enumerate(detection_classes):\r\n if not any(m1 == i):\r\n self.matrix[dc, self.nc] += 1 # predicted background\r\n\r\n def matrix(self):\r\n return self.matrix\r\n\r\n def tp_fp(self):\r\n tp = self.matrix.diagonal() # true positives\r\n fp = self.matrix.sum(1) - tp # false positives\r\n # fn = self.matrix.sum(0) - tp # false negatives (missed detections)\r\n return tp[:-1], fp[:-1] # remove background class\r\n\r\n @TryExcept('WARNING β οΈ ConfusionMatrix plot failure')\r\n def plot(self, normalize=True, save_dir='', names=()):\r\n import seaborn as sn\r\n\r\n array = self.matrix / ((self.matrix.sum(0).reshape(1, -1) + 1E-9) if normalize else 1) # normalize columns\r\n array[array < 0.005] = np.nan # don't annotate (would appear as 0.00)\r\n\r\n fig, ax = plt.subplots(1, 1, figsize=(12, 9), tight_layout=True)\r\n nc, nn = self.nc, len(names) # number of classes, names\r\n sn.set(font_scale=1.0 if nc < 50 else 0.8) # for label size\r\n labels = (0 < nn < 99) and (nn == nc) # apply names to ticklabels\r\n ticklabels = (names + ['background']) if labels else \"auto\"\r\n with warnings.catch_warnings():\r\n warnings.simplefilter('ignore') # suppress empty matrix RuntimeWarning: All-NaN slice encountered\r\n sn.heatmap(array,\r\n ax=ax,\r\n annot=nc < 30,\r\n annot_kws={\r\n \"size\": 8},\r\n cmap='Blues',\r\n fmt='.2f',\r\n square=True,\r\n vmin=0.0,\r\n xticklabels=ticklabels,\r\n yticklabels=ticklabels).set_facecolor((1, 1, 1))\r\n ax.set_ylabel('True')\r\n ax.set_ylabel('Predicted')\r\n ax.set_title('Confusion Matrix')\r\n fig.savefig(Path(save_dir) / 'confusion_matrix.png', dpi=250)\r\n plt.close(fig)\r\n\r\n def print(self):\r\n for i in range(self.nc + 1):\r\n print(' '.join(map(str, self.matrix[i])))\r\n\r\n\r\ndef smooth(y, f=0.05):\r\n # Box filter of fraction f\r\n nf = round(len(y) * f * 2) // 2 + 1 # number of filter elements (must be odd)\r\n p = np.ones(nf // 2) # ones padding\r\n yp = np.concatenate((p * y[0], y, p * y[-1]), 0) # y padded\r\n return np.convolve(yp, np.ones(nf) / nf, mode='valid') # y-smoothed\r\n\r\n\r\ndef plot_pr_curve(px, py, ap, save_dir=Path('pr_curve.png'), names=()):\r\n # Precision-recall curve\r\n fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)\r\n py = np.stack(py, axis=1)\r\n\r\n if 0 < len(names) < 21: # display per-class legend if < 21 classes\r\n for i, y in enumerate(py.T):\r\n ax.plot(px, y, linewidth=1, label=f'{names[i]} {ap[i, 0]:.3f}') # plot(recall, precision)\r\n else:\r\n ax.plot(px, py, linewidth=1, color='grey') # plot(recall, precision)\r\n\r\n ax.plot(px, py.mean(1), linewidth=3, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())\r\n ax.set_xlabel('Recall')\r\n ax.set_ylabel('Precision')\r\n ax.set_xlim(0, 1)\r\n ax.set_ylim(0, 1)\r\n ax.legend(bbox_to_anchor=(1.04, 1), loc=\"upper left\")\r\n ax.set_title('Precision-Recall Curve')\r\n fig.savefig(save_dir, dpi=250)\r\n plt.close(fig)\r\n\r\n\r\ndef plot_mc_curve(px, py, save_dir=Path('mc_curve.png'), names=(), xlabel='Confidence', ylabel='Metric'):\r\n # Metric-confidence curve\r\n fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)\r\n\r\n if 0 < len(names) < 21: # display per-class legend if < 21 classes\r\n for i, y in enumerate(py):\r\n ax.plot(px, y, linewidth=1, label=f'{names[i]}') # plot(confidence, metric)\r\n else:\r\n ax.plot(px, py.T, linewidth=1, color='grey') # plot(confidence, metric)\r\n\r\n y = smooth(py.mean(0), 0.05)\r\n ax.plot(px, y, linewidth=3, color='blue', label=f'all classes {y.max():.2f} at {px[y.argmax()]:.3f}')\r\n ax.set_xlabel(xlabel)\r\n ax.set_ylabel(ylabel)\r\n ax.set_xlim(0, 1)\r\n ax.set_ylim(0, 1)\r\n ax.legend(bbox_to_anchor=(1.04, 1), loc=\"upper left\")\r\n ax.set_title(f'{ylabel}-Confidence Curve')\r\n fig.savefig(save_dir, dpi=250)\r\n plt.close(fig)\r\n\r\n\r\ndef compute_ap(recall, precision):\r\n \"\"\" Compute the average precision, given the recall and precision curves\r\n # Arguments\r\n recall: The recall curve (list)\r\n precision: The precision curve (list)\r\n # Returns\r\n Average precision, precision curve, recall curve\r\n \"\"\"\r\n\r\n # Append sentinel values to beginning and end\r\n mrec = np.concatenate(([0.0], recall, [1.0]))\r\n mpre = np.concatenate(([1.0], precision, [0.0]))\r\n\r\n # Compute the precision envelope\r\n mpre = np.flip(np.maximum.accumulate(np.flip(mpre)))\r\n\r\n # Integrate area under curve\r\n method = 'interp' # methods: 'continuous', 'interp'\r\n if method == 'interp':\r\n x = np.linspace(0, 1, 101) # 101-point interp (COCO)\r\n ap = np.trapz(np.interp(x, mrec, mpre), x) # integrate\r\n else: # 'continuous'\r\n i = np.where(mrec[1:] != mrec[:-1])[0] # points where x-axis (recall) changes\r\n ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1]) # area under curve\r\n\r\n return ap, mpre, mrec\r\n\r\n\r\ndef ap_per_class(tp, conf, pred_cls, target_cls, plot=False, save_dir=Path(), names=(), eps=1e-16, prefix=\"\"):\r\n \"\"\" Compute the average precision, given the recall and precision curves.\r\n Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.\r\n # Arguments\r\n tp: True positives (nparray, nx1 or nx10).\r\n conf: Objectness value from 0-1 (nparray).\r\n pred_cls: Predicted object classes (nparray).\r\n target_cls: True object classes (nparray).\r\n plot: Plot precision-recall curve at mAP@0.5\r\n save_dir: Plot save directory\r\n # Returns\r\n The average precision as computed in py-faster-rcnn.\r\n \"\"\"\r\n\r\n # Sort by objectness\r\n i = np.argsort(-conf)\r\n tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]\r\n\r\n # Find unique classes\r\n unique_classes, nt = np.unique(target_cls, return_counts=True)\r\n nc = unique_classes.shape[0] # number of classes, number of detections\r\n\r\n # Create Precision-Recall curve and compute AP for each class\r\n px, py = np.linspace(0, 1, 1000), [] # for plotting\r\n ap, p, r = np.zeros((nc, tp.shape[1])), np.zeros((nc, 1000)), np.zeros((nc, 1000))\r\n for ci, c in enumerate(unique_classes):\r\n i = pred_cls == c\r\n n_l = nt[ci] # number of labels\r\n n_p = i.sum() # number of predictions\r\n if n_p == 0 or n_l == 0:\r\n continue\r\n\r\n # Accumulate FPs and TPs\r\n fpc = (1 - tp[i]).cumsum(0)\r\n tpc = tp[i].cumsum(0)\r\n\r\n # Recall\r\n recall = tpc / (n_l + eps) # recall curve\r\n r[ci] = np.interp(-px, -conf[i], recall[:, 0], left=0) # negative x, xp because xp decreases\r\n\r\n # Precision\r\n precision = tpc / (tpc + fpc) # precision curve\r\n p[ci] = np.interp(-px, -conf[i], precision[:, 0], left=1) # p at pr_score\r\n\r\n # AP from recall-precision curve\r\n for j in range(tp.shape[1]):\r\n ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])\r\n if plot and j == 0:\r\n py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5\r\n\r\n # Compute F1 (harmonic mean of precision and recall)\r\n f1 = 2 * p * r / (p + r + eps)\r\n names = [v for k, v in names.items() if k in unique_classes] # list: only classes that have data\r\n names = dict(enumerate(names)) # to dict\r\n if plot:\r\n plot_pr_curve(px, py, ap, save_dir / f'{prefix}PR_curve.png', names)\r\n plot_mc_curve(px, f1, save_dir / f'{prefix}F1_curve.png', names, ylabel='F1')\r\n plot_mc_curve(px, p, save_dir / f'{prefix}P_curve.png', names, ylabel='Precision')\r\n plot_mc_curve(px, r, save_dir / f'{prefix}R_curve.png', names, ylabel='Recall')\r\n\r\n i = smooth(f1.mean(0), 0.1).argmax() # max F1 index\r\n p, r, f1 = p[:, i], r[:, i], f1[:, i]\r\n tp = (r * nt).round() # true positives\r\n fp = (tp / (p + eps) - tp).round() # false positives\r\n return tp, fp, p, r, f1, ap, unique_classes.astype(int)\r\n\r\n\r\nclass Metric:\r\n\r\n def __init__(self) -> None:\r\n self.p = [] # (nc, )\r\n self.r = [] # (nc, )\r\n self.f1 = [] # (nc, )\r\n self.all_ap = [] # (nc, 10)\r\n self.ap_class_index = [] # (nc, )\r\n\r\n @property\r\n def ap50(self):\r\n \"\"\"AP@0.5 of all classes.\r\n Return:\r\n (nc, ) or [].\r\n \"\"\"\r\n return self.all_ap[:, 0] if len(self.all_ap) else []\r\n\r\n @property\r\n def ap(self):\r\n \"\"\"AP@0.5:0.95\r\n Return:\r\n (nc, ) or [].\r\n \"\"\"\r\n return self.all_ap.mean(1) if len(self.all_ap) else []\r\n\r\n @property\r\n def mp(self):\r\n \"\"\"mean precision of all classes.\r\n Return:\r\n float.\r\n \"\"\"\r\n return self.p.mean() if len(self.p) else 0.0\r\n\r\n @property\r\n def mr(self):\r\n \"\"\"mean recall of all classes.\r\n Return:\r\n float.\r\n \"\"\"\r\n return self.r.mean() if len(self.r) else 0.0\r\n\r\n @property\r\n def map50(self):\r\n \"\"\"Mean AP@0.5 of all classes.\r\n Return:\r\n float.\r\n \"\"\"\r\n return self.all_ap[:, 0].mean() if len(self.all_ap) else 0.0\r\n\r\n @property\r\n def map(self):\r\n \"\"\"Mean AP@0.5:0.95 of all classes.\r\n Return:\r\n float.\r\n \"\"\"\r\n return self.all_ap.mean() if len(self.all_ap) else 0.0\r\n\r\n def mean_results(self):\r\n \"\"\"Mean of results, return mp, mr, map50, map\"\"\"\r\n return [self.mp, self.mr, self.map50, self.map]\r\n\r\n def class_result(self, i):\r\n \"\"\"class-aware result, return p[i], r[i], ap50[i], ap[i]\"\"\"\r\n return self.p[i], self.r[i], self.ap50[i], self.ap[i]\r\n\r\n def get_maps(self, nc):\r\n maps = np.zeros(nc) + self.map\r\n for i, c in enumerate(self.ap_class_index):\r\n maps[c] = self.ap[i]\r\n return maps\r\n\r\n def fitness(self):\r\n # Model fitness as a weighted combination of metrics\r\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]\r\n return (np.array(self.mean_results()) * w).sum()\r\n\r\n def update(self, results):\r\n \"\"\"\r\n Args:\r\n results: tuple(p, r, ap, f1, ap_class)\r\n \"\"\"\r\n self.p, self.r, self.f1, self.all_ap, self.ap_class_index = results\r\n\r\n\r\nclass DetMetrics:\r\n\r\n def __init__(self, save_dir=Path(\".\"), plot=False, names=()) -> None:\r\n self.save_dir = save_dir\r\n self.plot = plot\r\n self.names = names\r\n self.metric = Metric()\r\n\r\n def process(self, tp, conf, pred_cls, target_cls):\r\n results = ap_per_class(tp, conf, pred_cls, target_cls, plot=self.plot, save_dir=self.save_dir,\r\n names=self.names)[2:]\r\n self.metric.update(results)\r\n\r\n @property\r\n def keys(self):\r\n return [\"metrics/precision(B)\", \"metrics/recall(B)\", \"metrics/mAP50(B)\", \"metrics/mAP50-95(B)\"]\r\n\r\n def mean_results(self):\r\n return self.metric.mean_results()\r\n\r\n def class_result(self, i):\r\n return self.metric.class_result(i)\r\n\r\n def get_maps(self, nc):\r\n return self.metric.get_maps(nc)\r\n\r\n @property\r\n def fitness(self):\r\n return self.metric.fitness()\r\n\r\n @property\r\n def ap_class_index(self):\r\n return self.metric.ap_class_index\r\n\r\n @property\r\n def results_dict(self):\r\n return dict(zip(self.keys + [\"fitness\"], self.mean_results() + [self.fitness]))\r\n\r\n\r\nclass SegmentMetrics:\r\n\r\n def __init__(self, save_dir=Path(\".\"), plot=False, names=()) -> None:\r\n self.save_dir = save_dir\r\n self.plot = plot\r\n self.names = names\r\n self.metric_box = Metric()\r\n self.metric_mask = Metric()\r\n\r\n def process(self, tp_m, tp_b, conf, pred_cls, target_cls):\r\n results_mask = ap_per_class(tp_m,\r\n conf,\r\n pred_cls,\r\n target_cls,\r\n plot=self.plot,\r\n save_dir=self.save_dir,\r\n names=self.names,\r\n prefix=\"Mask\")[2:]\r\n self.metric_mask.update(results_mask)\r\n results_box = ap_per_class(tp_b,\r\n conf,\r\n pred_cls,\r\n target_cls,\r\n plot=self.plot,\r\n save_dir=self.save_dir,\r\n names=self.names,\r\n prefix=\"Box\")[2:]\r\n self.metric_box.update(results_box)\r\n\r\n @property\r\n def keys(self):\r\n return [\r\n \"metrics/precision(B)\", \"metrics/recall(B)\", \"metrics/mAP50(B)\", \"metrics/mAP50-95(B)\",\r\n \"metrics/precision(M)\", \"metrics/recall(M)\", \"metrics/mAP50(M)\", \"metrics/mAP50-95(M)\"]\r\n\r\n def mean_results(self):\r\n return self.metric_box.mean_results() + self.metric_mask.mean_results()\r\n\r\n def class_result(self, i):\r\n return self.metric_box.class_result(i) + self.metric_mask.class_result(i)\r\n\r\n def get_maps(self, nc):\r\n return self.metric_box.get_maps(nc) + self.metric_mask.get_maps(nc)\r\n\r\n @property\r\n def fitness(self):\r\n return self.metric_mask.fitness() + self.metric_box.fitness()\r\n\r\n @property\r\n def ap_class_index(self):\r\n # boxes and masks have the same ap_class_index\r\n return self.metric_box.ap_class_index\r\n\r\n @property\r\n def results_dict(self):\r\n return dict(zip(self.keys + [\"fitness\"], self.mean_results() + [self.fitness]))\r\n\r\n\r\nclass ClassifyMetrics:\r\n\r\n def __init__(self) -> None:\r\n self.top1 = 0\r\n self.top5 = 0\r\n\r\n def process(self, targets, pred):\r\n # target classes and predicted classes\r\n pred, targets = torch.cat(pred), torch.cat(targets)\r\n correct = (targets[:, None] == pred).float()\r\n acc = torch.stack((correct[:, 0], correct.max(1).values), dim=1) # (top1, top5) accuracy\r\n self.top1, self.top5 = acc.mean(0).tolist()\r\n\r\n @property\r\n def fitness(self):\r\n return self.top5\r\n\r\n @property\r\n def results_dict(self):\r\n return dict(zip(self.keys + [\"fitness\"], [self.top1, self.top5, self.fitness]))\r\n\r\n @property\r\n def keys(self):\r\n return [\"metrics/accuracy_top1\", \"metrics/accuracy_top5\"]\r\n"
},
{
"path": "yolo/utils/ops.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport contextlib\r\nimport math\r\nimport re\r\nimport time\r\n\r\nimport cv2\r\nimport numpy as np\r\nimport torch\r\nimport torch.nn.functional as F\r\nimport torchvision\r\n\r\nfrom yolo.utils import LOGGER\r\n\r\nfrom .metrics import box_iou\r\n\r\n\r\nclass Profile(contextlib.ContextDecorator):\r\n # YOLOv5 Profile class. Usage: @Profile() decorator or 'with Profile():' context manager\r\n def __init__(self, t=0.0):\r\n self.t = t\r\n self.cuda = torch.cuda.is_available()\r\n\r\n def __enter__(self):\r\n self.start = self.time()\r\n return self\r\n\r\n def __exit__(self, type, value, traceback):\r\n self.dt = self.time() - self.start # delta-time\r\n self.t += self.dt # accumulate dt\r\n\r\n def time(self):\r\n if self.cuda:\r\n torch.cuda.synchronize()\r\n return time.time()\r\n\r\n\r\ndef coco80_to_coco91_class(): # converts 80-index (val2014) to 91-index (paper)\r\n # https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/\r\n # a = np.loadtxt('data/coco.names', dtype='str', delimiter='\\n')\r\n # b = np.loadtxt('data/coco_paper.names', dtype='str', delimiter='\\n')\r\n # x1 = [list(a[i] == b).index(True) + 1 for i in range(80)] # darknet to coco\r\n # x2 = [list(b[i] == a).index(True) if any(b[i] == a) else None for i in range(91)] # coco to darknet\r\n return [\r\n 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 31, 32, 33, 34,\r\n 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,\r\n 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90]\r\n\r\n\r\ndef segment2box(segment, width=640, height=640):\r\n \"\"\"\r\n > Convert 1 segment label to 1 box label, applying inside-image constraint, i.e. (xy1, xy2, ...) to\r\n (xyxy)\r\n Args:\r\n segment: the segment label\r\n width: the width of the image. Defaults to 640\r\n height: The height of the image. Defaults to 640\r\n\r\n Returns:\r\n the minimum and maximum x and y values of the segment.\r\n \"\"\"\r\n # Convert 1 segment label to 1 box label, applying inside-image constraint, i.e. (xy1, xy2, ...) to (xyxy)\r\n x, y = segment.T # segment xy\r\n inside = (x >= 0) & (y >= 0) & (x <= width) & (y <= height)\r\n x, y, = x[inside], y[inside]\r\n return np.array([x.min(), y.min(), x.max(), y.max()]) if any(x) else np.zeros(4) # xyxy\r\n\r\n\r\ndef scale_boxes(img1_shape, boxes, img0_shape, ratio_pad=None):\r\n \"\"\"\r\n > Rescale boxes (xyxy) from img1_shape to img0_shape\r\n Args:\r\n img1_shape: The shape of the image that the bounding boxes are for.\r\n boxes: the bounding boxes of the objects in the image\r\n img0_shape: the shape of the original image\r\n ratio_pad: a tuple of (ratio, pad)\r\n\r\n Returns:\r\n The boxes are being returned.\r\n \"\"\"\r\n #\r\n if ratio_pad is None: # calculate from img0_shape\r\n gain = min(img1_shape[0] / img0_shape[0], img1_shape[1] / img0_shape[1]) # gain = old / new\r\n pad = (img1_shape[1] - img0_shape[1] * gain) / 2, (img1_shape[0] - img0_shape[0] * gain) / 2 # wh padding\r\n else:\r\n gain = ratio_pad[0][0]\r\n pad = ratio_pad[1]\r\n\r\n boxes[..., [0, 2]] -= pad[0] # x padding\r\n boxes[..., [1, 3]] -= pad[1] # y padding\r\n boxes[..., :4] /= gain\r\n clip_boxes(boxes, img0_shape)\r\n return boxes\r\n\r\n\r\ndef make_divisible(x, divisor):\r\n # Returns nearest x divisible by divisor\r\n if isinstance(divisor, torch.Tensor):\r\n divisor = int(divisor.max()) # to int\r\n return math.ceil(x / divisor) * divisor\r\n\r\n\r\ndef non_max_suppression(\r\n prediction,\r\n conf_thres=0.25,\r\n iou_thres=0.45,\r\n classes=None,\r\n agnostic=False,\r\n multi_label=False,\r\n labels=(),\r\n max_det=300,\r\n nm=0, # number of masks\r\n):\r\n \"\"\"\r\n > Perform non-maximum suppression (NMS) on a set of boxes, with support for masks and multiple labels per box.\r\n\r\n Arguments:\r\n prediction (torch.Tensor): A tensor of shape (batch_size, num_boxes, num_classes + 4 + num_masks)\r\n containing the predicted boxes, classes, and masks. The tensor should be in the format\r\n output by a model, such as YOLO.\r\n conf_thres (float): The confidence threshold below which boxes will be filtered out.\r\n Valid values are between 0.0 and 1.0.\r\n iou_thres (float): The IoU threshold below which boxes will be filtered out during NMS.\r\n Valid values are between 0.0 and 1.0.\r\n classes (List[int]): A list of class indices to consider. If None, all classes will be considered.\r\n agnostic (bool): If True, the model is agnostic to the number of classes, and all\r\n classes will be considered as one.\r\n multi_label (bool): If True, each box may have multiple labels.\r\n labels (List[List[Union[int, float, torch.Tensor]]]): A list of lists, where each inner\r\n list contains the apriori labels for a given image. The list should be in the format\r\n output by a dataloader, with each label being a tuple of (class_index, x1, y1, x2, y2).\r\n max_det (int): The maximum number of boxes to keep after NMS.\r\n nm (int): The number of masks output by the model.\r\n\r\n Returns:\r\n List[torch.Tensor]: A list of length batch_size, where each element is a tensor of\r\n shape (num_boxes, 6 + num_masks) containing the kept boxes, with columns\r\n (x1, y1, x2, y2, confidence, class, mask1, mask2, ...).\r\n \"\"\"\r\n\r\n # Checks\r\n assert 0 <= conf_thres <= 1, f'Invalid Confidence threshold {conf_thres}, valid values are between 0.0 and 1.0'\r\n assert 0 <= iou_thres <= 1, f'Invalid IoU {iou_thres}, valid values are between 0.0 and 1.0'\r\n if isinstance(prediction, (list, tuple)): # YOLOv5 model in validation model, output = (inference_out, loss_out)\r\n prediction = prediction[0] # select only inference output\r\n\r\n device = prediction.device\r\n mps = 'mps' in device.type # Apple MPS\r\n if mps: # MPS not fully supported yet, convert tensors to CPU before NMS\r\n prediction = prediction.cpu()\r\n bs = prediction.shape[0] # batch size\r\n nc = prediction.shape[1] - nm - 4 # number of classes\r\n mi = 4 + nc # mask start index\r\n xc = prediction[:, 4:mi].amax(1) > conf_thres # candidates\r\n\r\n # Settings\r\n # min_wh = 2 # (pixels) minimum box width and height\r\n max_wh = 7680 # (pixels) maximum box width and height\r\n max_nms = 30000 # maximum number of boxes into torchvision.ops.nms()\r\n time_limit = 0.5 + 0.05 * bs # seconds to quit after\r\n redundant = True # require redundant detections\r\n multi_label &= nc > 1 # multiple labels per box (adds 0.5ms/img)\r\n merge = False # use merge-NMS\r\n\r\n t = time.time()\r\n output = [torch.zeros((0, 6 + nm), device=prediction.device)] * bs\r\n for xi, x in enumerate(prediction): # image index, image inference\r\n # Apply constraints\r\n # x[((x[:, 2:4] < min_wh) | (x[:, 2:4] > max_wh)).any(1), 4] = 0 # width-height\r\n x = x.transpose(0, -1)[xc[xi]] # confidence\r\n\r\n # Cat apriori labels if autolabelling\r\n if labels and len(labels[xi]):\r\n lb = labels[xi]\r\n v = torch.zeros((len(lb), nc + nm + 5), device=x.device)\r\n v[:, :4] = lb[:, 1:5] # box\r\n v[range(len(lb)), lb[:, 0].long() + 4] = 1.0 # cls\r\n x = torch.cat((x, v), 0)\r\n\r\n # If none remain process next image\r\n if not x.shape[0]:\r\n continue\r\n\r\n # Detections matrix nx6 (xyxy, conf, cls)\r\n box, cls, mask = x.split((4, nc, nm), 1)\r\n box = xywh2xyxy(box) # center_x, center_y, width, height) to (x1, y1, x2, y2)\r\n if multi_label:\r\n i, j = (cls > conf_thres).nonzero(as_tuple=False).T\r\n x = torch.cat((box[i], x[i, 4 + j, None], j[:, None].float(), mask[i]), 1)\r\n else: # best class only\r\n conf, j = cls.max(1, keepdim=True)\r\n x = torch.cat((box, conf, j.float(), mask), 1)[conf.view(-1) > conf_thres]\r\n\r\n # Filter by class\r\n if classes is not None:\r\n x = x[(x[:, 5:6] == torch.tensor(classes, device=x.device)).any(1)]\r\n\r\n # Apply finite constraint\r\n # if not torch.isfinite(x).all():\r\n # x = x[torch.isfinite(x).all(1)]\r\n\r\n # Check shape\r\n n = x.shape[0] # number of boxes\r\n if not n: # no boxes\r\n continue\r\n x = x[x[:, 4].argsort(descending=True)[:max_nms]] # sort by confidence and remove excess boxes\r\n\r\n # Batched NMS\r\n c = x[:, 5:6] * (0 if agnostic else max_wh) # classes\r\n boxes, scores = x[:, :4] + c, x[:, 4] # boxes (offset by class), scores\r\n i = torchvision.ops.nms(boxes, scores, iou_thres) # NMS\r\n i = i[:max_det] # limit detections\r\n if merge and (1 < n < 3E3): # Merge NMS (boxes merged using weighted mean)\r\n # update boxes as boxes(i,4) = weights(i,n) * boxes(n,4)\r\n iou = box_iou(boxes[i], boxes) > iou_thres # iou matrix\r\n weights = iou * scores[None] # box weights\r\n x[i, :4] = torch.mm(weights, x[:, :4]).float() / weights.sum(1, keepdim=True) # merged boxes\r\n if redundant:\r\n i = i[iou.sum(1) > 1] # require redundancy\r\n\r\n output[xi] = x[i]\r\n if mps:\r\n output[xi] = output[xi].to(device)\r\n if (time.time() - t) > time_limit:\r\n LOGGER.warning(f'WARNING β οΈ NMS time limit {time_limit:.3f}s exceeded')\r\n break # time limit exceeded\r\n\r\n return output\r\n\r\n\r\ndef clip_boxes(boxes, shape):\r\n \"\"\"\r\n > It takes a list of bounding boxes and a shape (height, width) and clips the bounding boxes to the\r\n shape\r\n\r\n Args:\r\n boxes: the bounding boxes to clip\r\n shape: the shape of the image\r\n \"\"\"\r\n if isinstance(boxes, torch.Tensor): # faster individually\r\n boxes[..., 0].clamp_(0, shape[1]) # x1\r\n boxes[..., 1].clamp_(0, shape[0]) # y1\r\n boxes[..., 2].clamp_(0, shape[1]) # x2\r\n boxes[..., 3].clamp_(0, shape[0]) # y2\r\n else: # np.array (faster grouped)\r\n boxes[..., [0, 2]] = boxes[..., [0, 2]].clip(0, shape[1]) # x1, x2\r\n boxes[..., [1, 3]] = boxes[..., [1, 3]].clip(0, shape[0]) # y1, y2\r\n\r\n\r\ndef clip_coords(boxes, shape):\r\n # Clip bounding xyxy bounding boxes to image shape (height, width)\r\n if isinstance(boxes, torch.Tensor): # faster individually\r\n boxes[:, 0].clamp_(0, shape[1]) # x1\r\n boxes[:, 1].clamp_(0, shape[0]) # y1\r\n boxes[:, 2].clamp_(0, shape[1]) # x2\r\n boxes[:, 3].clamp_(0, shape[0]) # y2\r\n else: # np.array (faster grouped)\r\n boxes[:, [0, 2]] = boxes[:, [0, 2]].clip(0, shape[1]) # x1, x2\r\n boxes[:, [1, 3]] = boxes[:, [1, 3]].clip(0, shape[0]) # y1, y2\r\n\r\n\r\ndef scale_image(im1_shape, masks, im0_shape, ratio_pad=None):\r\n \"\"\"\r\n > It takes a mask, and resizes it to the original image size\r\n\r\n Args:\r\n im1_shape: model input shape, [h, w]\r\n masks: [h, w, num]\r\n im0_shape: the original image shape\r\n ratio_pad: the ratio of the padding to the original image.\r\n\r\n Returns:\r\n The masks are being returned.\r\n \"\"\"\r\n # Rescale coordinates (xyxy) from im1_shape to im0_shape\r\n if ratio_pad is None: # calculate from im0_shape\r\n gain = min(im1_shape[0] / im0_shape[0], im1_shape[1] / im0_shape[1]) # gain = old / new\r\n pad = (im1_shape[1] - im0_shape[1] * gain) / 2, (im1_shape[0] - im0_shape[0] * gain) / 2 # wh padding\r\n else:\r\n pad = ratio_pad[1]\r\n top, left = int(pad[1]), int(pad[0]) # y, x\r\n bottom, right = int(im1_shape[0] - pad[1]), int(im1_shape[1] - pad[0])\r\n\r\n if len(masks.shape) < 2:\r\n raise ValueError(f'\"len of masks shape\" should be 2 or 3, but got {len(masks.shape)}')\r\n masks = masks[top:bottom, left:right]\r\n # masks = masks.permute(2, 0, 1).contiguous()\r\n # masks = F.interpolate(masks[None], im0_shape[:2], mode='bilinear', align_corners=False)[0]\r\n # masks = masks.permute(1, 2, 0).contiguous()\r\n masks = cv2.resize(masks, (im0_shape[1], im0_shape[0]))\r\n\r\n if len(masks.shape) == 2:\r\n masks = masks[:, :, None]\r\n return masks\r\n\r\n\r\ndef xyxy2xywh(x):\r\n \"\"\"\r\n > It takes a list of bounding boxes, and converts them from the format [x1, y1, x2, y2] to [x, y, w,\r\n h] where xy1=top-left, xy2=bottom-right\r\n\r\n Args:\r\n x: the input tensor\r\n\r\n Returns:\r\n the center of the box, the width and the height of the box.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[..., 0] = (x[..., 0] + x[..., 2]) / 2 # x center\r\n y[..., 1] = (x[..., 1] + x[..., 3]) / 2 # y center\r\n y[..., 2] = x[..., 2] - x[..., 0] # width\r\n y[..., 3] = x[..., 3] - x[..., 1] # height\r\n return y\r\n\r\n\r\ndef xywh2xyxy(x):\r\n \"\"\"\r\n > It converts the bounding box from x,y,w,h to x1,y1,x2,y2 where xy1=top-left, xy2=bottom-right\r\n\r\n Args:\r\n x: the input tensor\r\n\r\n Returns:\r\n the top left and bottom right coordinates of the bounding box.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[..., 0] = x[..., 0] - x[..., 2] / 2 # top left x\r\n y[..., 1] = x[..., 1] - x[..., 3] / 2 # top left y\r\n y[..., 2] = x[..., 0] + x[..., 2] / 2 # bottom right x\r\n y[..., 3] = x[..., 1] + x[..., 3] / 2 # bottom right y\r\n return y\r\n\r\n\r\ndef xywhn2xyxy(x, w=640, h=640, padw=0, padh=0):\r\n \"\"\"\r\n > It converts the normalized coordinates to the actual coordinates [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right\r\n\r\n Args:\r\n x: the bounding box coordinates\r\n w: width of the image. Defaults to 640\r\n h: height of the image. Defaults to 640\r\n padw: padding width. Defaults to 0\r\n padh: height of the padding. Defaults to 0\r\n\r\n Returns:\r\n the xyxy coordinates of the bounding box.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[..., 0] = w * (x[..., 0] - x[..., 2] / 2) + padw # top left x\r\n y[..., 1] = h * (x[..., 1] - x[..., 3] / 2) + padh # top left y\r\n y[..., 2] = w * (x[..., 0] + x[..., 2] / 2) + padw # bottom right x\r\n y[..., 3] = h * (x[..., 1] + x[..., 3] / 2) + padh # bottom right y\r\n return y\r\n\r\n\r\ndef xyxy2xywhn(x, w=640, h=640, clip=False, eps=0.0):\r\n \"\"\"\r\n > It takes in a list of bounding boxes, and returns a list of bounding boxes, but with the x and y\r\n coordinates normalized to the width and height of the image\r\n\r\n Args:\r\n x: the bounding box coordinates\r\n w: width of the image. Defaults to 640\r\n h: height of the image. Defaults to 640\r\n clip: If True, the boxes will be clipped to the image boundaries. Defaults to False\r\n eps: the minimum value of the box's width and height.\r\n\r\n Returns:\r\n the xywhn format of the bounding boxes.\r\n \"\"\"\r\n if clip:\r\n clip_boxes(x, (h - eps, w - eps)) # warning: inplace clip\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[..., 0] = ((x[..., 0] + x[..., 2]) / 2) / w # x center\r\n y[..., 1] = ((x[..., 1] + x[..., 3]) / 2) / h # y center\r\n y[..., 2] = (x[..., 2] - x[..., 0]) / w # width\r\n y[..., 3] = (x[..., 3] - x[..., 1]) / h # height\r\n return y\r\n\r\n\r\ndef xyn2xy(x, w=640, h=640, padw=0, padh=0):\r\n \"\"\"\r\n > It converts normalized segments into pixel segments of shape (n,2)\r\n\r\n Args:\r\n x: the normalized coordinates of the bounding box\r\n w: width of the image. Defaults to 640\r\n h: height of the image. Defaults to 640\r\n padw: padding width. Defaults to 0\r\n padh: padding height. Defaults to 0\r\n\r\n Returns:\r\n the x and y coordinates of the top left corner of the bounding box.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[..., 0] = w * x[..., 0] + padw # top left x\r\n y[..., 1] = h * x[..., 1] + padh # top left y\r\n return y\r\n\r\n\r\ndef xywh2ltwh(x):\r\n \"\"\"\r\n > It converts the bounding box from [x, y, w, h] to [x1, y1, w, h] where xy1=top-left\r\n\r\n Args:\r\n x: the x coordinate of the center of the bounding box\r\n\r\n Returns:\r\n the top left x and y coordinates of the bounding box.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left x\r\n y[:, 1] = x[:, 1] - x[:, 3] / 2 # top left y\r\n return y\r\n\r\n\r\ndef xyxy2ltwh(x):\r\n \"\"\"\r\n > Convert nx4 boxes from [x1, y1, x2, y2] to [x1, y1, w, h] where xy1=top-left, xy2=bottom-right\r\n\r\n Args:\r\n x: the input tensor\r\n\r\n Returns:\r\n the xyxy2ltwh function.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[:, 2] = x[:, 2] - x[:, 0] # width\r\n y[:, 3] = x[:, 3] - x[:, 1] # height\r\n return y\r\n\r\n\r\ndef ltwh2xywh(x):\r\n \"\"\"\r\n > Convert nx4 boxes from [x1, y1, w, h] to [x, y, w, h] where xy1=top-left, xy=center\r\n\r\n Args:\r\n x: the input tensor\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[:, 0] = x[:, 0] + x[:, 2] / 2 # center x\r\n y[:, 1] = x[:, 1] + x[:, 3] / 2 # center y\r\n return y\r\n\r\n\r\ndef ltwh2xyxy(x):\r\n \"\"\"\r\n > It converts the bounding box from [x1, y1, w, h] to [x1, y1, x2, y2] where xy1=top-left,\r\n xy2=bottom-right\r\n\r\n Args:\r\n x: the input image\r\n\r\n Returns:\r\n the xyxy coordinates of the bounding boxes.\r\n \"\"\"\r\n y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)\r\n y[:, 2] = x[:, 2] + x[:, 0] # width\r\n y[:, 3] = x[:, 3] + x[:, 1] # height\r\n return y\r\n\r\n\r\ndef segments2boxes(segments):\r\n \"\"\"\r\n > It converts segment labels to box labels, i.e. (cls, xy1, xy2, ...) to (cls, xywh)\r\n\r\n Args:\r\n segments: list of segments, each segment is a list of points, each point is a list of x, y\r\n coordinates\r\n\r\n Returns:\r\n the xywh coordinates of the bounding boxes.\r\n \"\"\"\r\n boxes = []\r\n for s in segments:\r\n x, y = s.T # segment xy\r\n boxes.append([x.min(), y.min(), x.max(), y.max()]) # cls, xyxy\r\n return xyxy2xywh(np.array(boxes)) # cls, xywh\r\n\r\n\r\ndef resample_segments(segments, n=1000):\r\n \"\"\"\r\n > It takes a list of segments (n,2) and returns a list of segments (n,2) where each segment has been\r\n up-sampled to n points\r\n\r\n Args:\r\n segments: a list of (n,2) arrays, where n is the number of points in the segment.\r\n n: number of points to resample the segment to. Defaults to 1000\r\n\r\n Returns:\r\n the resampled segments.\r\n \"\"\"\r\n for i, s in enumerate(segments):\r\n s = np.concatenate((s, s[0:1, :]), axis=0)\r\n x = np.linspace(0, len(s) - 1, n)\r\n xp = np.arange(len(s))\r\n segments[i] = np.concatenate([np.interp(x, xp, s[:, i]) for i in range(2)]).reshape(2, -1).T # segment xy\r\n return segments\r\n\r\n\r\ndef crop_mask(masks, boxes):\r\n \"\"\"\r\n > It takes a mask and a bounding box, and returns a mask that is cropped to the bounding box\r\n\r\n Args:\r\n masks: [h, w, n] tensor of masks\r\n boxes: [n, 4] tensor of bbox coords in relative point form\r\n\r\n Returns:\r\n The masks are being cropped to the bounding box.\r\n \"\"\"\r\n n, h, w = masks.shape\r\n x1, y1, x2, y2 = torch.chunk(boxes[:, :, None], 4, 1) # x1 shape(1,1,n)\r\n r = torch.arange(w, device=masks.device, dtype=x1.dtype)[None, None, :] # rows shape(1,w,1)\r\n c = torch.arange(h, device=masks.device, dtype=x1.dtype)[None, :, None] # cols shape(h,1,1)\r\n\r\n return masks * ((r >= x1) * (r < x2) * (c >= y1) * (c < y2))\r\n\r\n\r\ndef process_mask_upsample(protos, masks_in, bboxes, shape):\r\n \"\"\"\r\n > It takes the output of the mask head, and applies the mask to the bounding boxes. This produces masks of higher\r\n quality but is slower.\r\n\r\n Args:\r\n protos: [mask_dim, mask_h, mask_w]\r\n masks_in: [n, mask_dim], n is number of masks after nms\r\n bboxes: [n, 4], n is number of masks after nms\r\n shape: the size of the input image\r\n\r\n Returns:\r\n mask\r\n \"\"\"\r\n c, mh, mw = protos.shape # CHW\r\n masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)\r\n masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW\r\n masks = crop_mask(masks, bboxes) # CHW\r\n return masks.gt_(0.5)\r\n\r\n\r\ndef process_mask(protos, masks_in, bboxes, shape, upsample=False):\r\n \"\"\"\r\n > It takes the output of the mask head, and applies the mask to the bounding boxes. This is faster but produces\r\n downsampled quality of mask\r\n\r\n Args:\r\n protos: [mask_dim, mask_h, mask_w]\r\n masks_in: [n, mask_dim], n is number of masks after nms\r\n bboxes: [n, 4], n is number of masks after nms\r\n shape: the size of the input image\r\n\r\n Returns:\r\n mask\r\n \"\"\"\r\n\r\n c, mh, mw = protos.shape # CHW\r\n ih, iw = shape\r\n masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw) # CHW\r\n\r\n downsampled_bboxes = bboxes.clone()\r\n downsampled_bboxes[:, 0] *= mw / iw\r\n downsampled_bboxes[:, 2] *= mw / iw\r\n downsampled_bboxes[:, 3] *= mh / ih\r\n downsampled_bboxes[:, 1] *= mh / ih\r\n\r\n masks = crop_mask(masks, downsampled_bboxes) # CHW\r\n if upsample:\r\n masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW\r\n return masks.gt_(0.5)\r\n\r\n\r\ndef process_mask_native(protos, masks_in, bboxes, shape):\r\n \"\"\"\r\n > It takes the output of the mask head, and crops it after upsampling to the bounding boxes.\r\n\r\n Args:\r\n protos: [mask_dim, mask_h, mask_w]\r\n masks_in: [n, mask_dim], n is number of masks after nms\r\n bboxes: [n, 4], n is number of masks after nms\r\n shape: input_image_size, (h, w)\r\n\r\n Returns:\r\n masks: [h, w, n]\r\n \"\"\"\r\n c, mh, mw = protos.shape # CHW\r\n masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)\r\n gain = min(mh / shape[0], mw / shape[1]) # gain = old / new\r\n pad = (mw - shape[1] * gain) / 2, (mh - shape[0] * gain) / 2 # wh padding\r\n top, left = int(pad[1]), int(pad[0]) # y, x\r\n bottom, right = int(mh - pad[1]), int(mw - pad[0])\r\n masks = masks[:, top:bottom, left:right]\r\n\r\n masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW\r\n masks = crop_mask(masks, bboxes) # CHW\r\n return masks.gt_(0.5)\r\n\r\n\r\ndef scale_segments(img1_shape, segments, img0_shape, ratio_pad=None, normalize=False):\r\n \"\"\"\r\n > Rescale segment coords (xyxy) from img1_shape to img0_shape\r\n\r\n Args:\r\n img1_shape: The shape of the image that the segments are from.\r\n segments: the segments to be scaled\r\n img0_shape: the shape of the image that the segmentation is being applied to\r\n ratio_pad: the ratio of the image size to the padded image size.\r\n normalize: If True, the coordinates will be normalized to the range [0, 1]. Defaults to False\r\n\r\n Returns:\r\n the segmented image.\r\n \"\"\"\r\n if ratio_pad is None: # calculate from img0_shape\r\n gain = min(img1_shape[0] / img0_shape[0], img1_shape[1] / img0_shape[1]) # gain = old / new\r\n pad = (img1_shape[1] - img0_shape[1] * gain) / 2, (img1_shape[0] - img0_shape[0] * gain) / 2 # wh padding\r\n else:\r\n gain = ratio_pad[0][0]\r\n pad = ratio_pad[1]\r\n\r\n segments[:, 0] -= pad[0] # x padding\r\n segments[:, 1] -= pad[1] # y padding\r\n segments /= gain\r\n clip_segments(segments, img0_shape)\r\n if normalize:\r\n segments[:, 0] /= img0_shape[1] # width\r\n segments[:, 1] /= img0_shape[0] # height\r\n return segments\r\n\r\n\r\ndef masks2segments(masks, strategy='largest'):\r\n \"\"\"\r\n > It takes a list of masks(n,h,w) and returns a list of segments(n,xy)\r\n\r\n Args:\r\n masks: the output of the model, which is a tensor of shape (batch_size, 160, 160)\r\n strategy: 'concat' or 'largest'. Defaults to largest\r\n\r\n Returns:\r\n segments (List): list of segment masks\r\n \"\"\"\r\n segments = []\r\n for x in masks.int().cpu().numpy().astype('uint8'):\r\n c = cv2.findContours(x, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]\r\n if c:\r\n if strategy == 'concat': # concatenate all segments\r\n c = np.concatenate([x.reshape(-1, 2) for x in c])\r\n elif strategy == 'largest': # select largest segment\r\n c = np.array(c[np.array([len(x) for x in c]).argmax()]).reshape(-1, 2)\r\n else:\r\n c = np.zeros((0, 2)) # no segments found\r\n segments.append(c.astype('float32'))\r\n return segments\r\n\r\n\r\ndef clip_segments(segments, shape):\r\n \"\"\"\r\n > It takes a list of line segments (x1,y1,x2,y2) and clips them to the image shape (height, width)\r\n\r\n Args:\r\n segments: a list of segments, each segment is a list of points, each point is a list of x,y\r\n coordinates\r\n shape: the shape of the image\r\n \"\"\"\r\n if isinstance(segments, torch.Tensor): # faster individually\r\n segments[:, 0].clamp_(0, shape[1]) # x\r\n segments[:, 1].clamp_(0, shape[0]) # y\r\n else: # np.array (faster grouped)\r\n segments[:, 0] = segments[:, 0].clip(0, shape[1]) # x\r\n segments[:, 1] = segments[:, 1].clip(0, shape[0]) # y\r\n\r\n\r\ndef clean_str(s):\r\n # Cleans a string by replacing special characters with underscore _\r\n return re.sub(pattern=\"[|@#!‘·$β¬%&()=?ΒΏ^*;:,¨´><+]\", repl=\"_\", string=s)\r\n"
},
{
"path": "yolo/utils/plotting.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport contextlib\r\nimport math\r\nfrom pathlib import Path\r\nfrom urllib.error import URLError\r\n\r\nimport cv2\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\nimport pandas as pd\r\nimport torch\r\nfrom PIL import Image, ImageDraw, ImageFont\r\n\r\nfrom yolo.utils import FONT, USER_CONFIG_DIR, threaded\r\n\r\nfrom .checks import check_font, check_requirements, is_ascii\r\nfrom .files import increment_path\r\nfrom .ops import clip_coords, scale_image, xywh2xyxy, xyxy2xywh\r\n\r\n\r\nclass Colors:\r\n # Ultralytics color palette https://com/\r\n def __init__(self):\r\n # hex = matplotlib.colors.TABLEAU_COLORS.values()\r\n hexs = ('7fff00', '7fff00', '7fff00', '7fff00', '7fff00', '7fff00', '7fff00', '7fff00', '1A9334', '00D4BB',\r\n '2C99A8', '00C2FF', '344593', '6473FF', '0018EC', '8438FF', '520085', 'CB38FF', 'FF95C8', 'FF37C7')\r\n self.palette = [self.hex2rgb(f'#{c}') for c in hexs]\r\n self.n = len(self.palette)\r\n\r\n def __call__(self, i, bgr=False):\r\n c = self.palette[int(i) % self.n]\r\n return (c[2], c[1], c[0]) if bgr else c\r\n\r\n @staticmethod\r\n def hex2rgb(h): # rgb order (PIL)\r\n return tuple(int(h[1 + i:1 + i + 2], 16) for i in (0, 2, 4))\r\n\r\n\r\ncolors = Colors() # create instance for 'from utils.plots import colors'\r\n\r\n\r\nclass Annotator:\r\n # YOLOv5 Annotator for train/val mosaics and jpgs and detect/hub inference annotations\r\n def __init__(self, im, line_width=None, font_size=None, font='Arial.ttf', pil=False, example='abc'):\r\n assert im.data.contiguous, 'Image not contiguous. Apply np.ascontiguousarray(im) to Annotator() input images.'\r\n non_ascii = not is_ascii(example) # non-latin labels, i.e. asian, arabic, cyrillic\r\n self.pil = pil or non_ascii\r\n if self.pil: # use PIL\r\n self.im = im if isinstance(im, Image.Image) else Image.fromarray(im)\r\n self.draw = ImageDraw.Draw(self.im)\r\n self.font = check_pil_font(font='Arial.Unicode.ttf' if non_ascii else font,\r\n size=font_size or max(round(sum(self.im.size) / 2 * 0.035), 12))\r\n else: # use cv2\r\n self.im = im\r\n self.lw = line_width or max(round(sum(im.shape) / 2 * 0.003), 2) # line width\r\n\r\n def box_label(self, box, label='', color=(128, 128, 128), txt_color=(255, 255, 255)):\r\n # Add one xyxy box to image with label\r\n if self.pil or not is_ascii(label):\r\n self.draw.rectangle(box, width=self.lw, outline=color) # box\r\n if label:\r\n w, h = self.font.getsize(label) # text width, height\r\n outside = box[1] - h >= 0 # label fits outside box\r\n self.draw.rectangle(\r\n (box[0], box[1] - h if outside else box[1], box[0] + w + 1,\r\n box[1] + 1 if outside else box[1] + h + 1),\r\n fill=color,\r\n )\r\n # self.draw.text((box[0], box[1]), label, fill=txt_color, font=self.font, anchor='ls') # for PIL>8.0\r\n self.draw.text((box[0], box[1] - h if outside else box[1]), label, fill=txt_color, font=self.font)\r\n else: # cv2\r\n p1, p2 = (int(box[0]), int(box[1])), (int(box[2]), int(box[3]))\r\n cv2.rectangle(self.im, p1, p2, (0,255,127), thickness=self.lw, lineType=cv2.LINE_AA)\r\n if label:\r\n tf = max(self.lw - 1, 1) # font thickness\r\n w, h = cv2.getTextSize(label, 0, fontScale=self.lw / 3, thickness=tf)[0] # text width, height\r\n outside = p1[1] - h >= 3\r\n p2 = p1[0] + w, p1[1] - h - 3 if outside else p1[1] + h + 3\r\n cv2.rectangle(self.im, p1, p2, color, -1, cv2.LINE_AA) # filled\r\n cv2.putText(self.im,\r\n label, (p1[0], p1[1] - 2 if outside else p1[1] + h + 2),\r\n 0,\r\n self.lw / 3,\r\n txt_color,\r\n thickness=tf,\r\n lineType=cv2.LINE_AA)\r\n\r\n def masks(self, masks, colors, im_gpu, alpha=0.5, retina_masks=False):\r\n \"\"\"Plot masks at once.\r\n Args:\r\n masks (tensor): predicted masks on cuda, shape: [n, h, w]\r\n colors (List[List[Int]]): colors for predicted masks, [[r, g, b] * n]\r\n im_gpu (tensor): img is in cuda, shape: [3, h, w], range: [0, 1]\r\n alpha (float): mask transparency: 0.0 fully transparent, 1.0 opaque\r\n \"\"\"\r\n if self.pil:\r\n # convert to numpy first\r\n self.im = np.asarray(self.im).copy()\r\n if len(masks) == 0:\r\n self.im[:] = im_gpu.permute(1, 2, 0).contiguous().cpu().numpy() * 255\r\n colors = torch.tensor(colors, device=im_gpu.device, dtype=torch.float32) / 255.0\r\n colors = colors[:, None, None] # shape(n,1,1,3)\r\n masks = masks.unsqueeze(3) # shape(n,h,w,1)\r\n masks_color = masks * (colors * alpha) # shape(n,h,w,3)\r\n\r\n inv_alph_masks = (1 - masks * alpha).cumprod(0) # shape(n,h,w,1)\r\n mcs = (masks_color * inv_alph_masks).sum(0) * 2 # mask color summand shape(n,h,w,3)\r\n\r\n im_gpu = im_gpu.flip(dims=[0]) # flip channel\r\n im_gpu = im_gpu.permute(1, 2, 0).contiguous() # shape(h,w,3)\r\n im_gpu = im_gpu * inv_alph_masks[-1] + mcs\r\n im_mask = (im_gpu * 255)\r\n im_mask_np = im_mask.byte().cpu().numpy()\r\n self.im[:] = im_mask_np if retina_masks else scale_image(im_gpu.shape, im_mask_np, self.im.shape)\r\n if self.pil:\r\n # convert im back to PIL and update draw\r\n self.fromarray(self.im)\r\n\r\n def rectangle(self, xy, fill=None, outline=None, width=1):\r\n # Add rectangle to image (PIL-only)\r\n self.draw.rectangle(xy, fill, outline, width)\r\n\r\n def text(self, xy, text, txt_color=(255, 255, 255), anchor='top'):\r\n # Add text to image (PIL-only)\r\n if anchor == 'bottom': # start y from font bottom\r\n w, h = self.font.getsize(text) # text width, height\r\n xy[1] += 1 - h\r\n self.draw.text(xy, text, fill=txt_color, font=self.font)\r\n\r\n def fromarray(self, im):\r\n # Update self.im from a numpy array\r\n self.im = im if isinstance(im, Image.Image) else Image.fromarray(im)\r\n self.draw = ImageDraw.Draw(self.im)\r\n\r\n def result(self):\r\n # Return annotated image as array\r\n return np.asarray(self.im)\r\n\r\n\r\ndef check_pil_font(font=FONT, size=10):\r\n # Return a PIL TrueType Font, downloading to CONFIG_DIR if necessary\r\n font = Path(font)\r\n font = font if font.exists() else (USER_CONFIG_DIR / font.name)\r\n try:\r\n return ImageFont.truetype(str(font) if font.exists() else font.name, size)\r\n except Exception: # download if missing\r\n try:\r\n check_font(font)\r\n return ImageFont.truetype(str(font), size)\r\n except TypeError:\r\n check_requirements('Pillow>=8.4.0') # known issue https://github.com/ultralytics/yolov5/issues/5374\r\n except URLError: # not online\r\n return ImageFont.load_default()\r\n\r\n\r\ndef save_one_box(xyxy, im, file=Path('im.jpg'), gain=1.02, pad=10, square=False, BGR=False, save=True):\r\n # Save image crop as {file} with crop size multiple {gain} and {pad} pixels. Save and/or return crop\r\n xyxy = torch.tensor(xyxy).view(-1, 4)\r\n b = xyxy2xywh(xyxy) # boxes\r\n if square:\r\n b[:, 2:] = b[:, 2:].max(1)[0].unsqueeze(1) # attempt rectangle to square\r\n b[:, 2:] = b[:, 2:] * gain + pad # box wh * gain + pad\r\n xyxy = xywh2xyxy(b).long()\r\n clip_coords(xyxy, im.shape)\r\n crop = im[int(xyxy[0, 1]):int(xyxy[0, 3]), int(xyxy[0, 0]):int(xyxy[0, 2]), ::(1 if BGR else -1)]\r\n if save:\r\n file.parent.mkdir(parents=True, exist_ok=True) # make directory\r\n f = str(increment_path(file).with_suffix('.jpg'))\r\n # cv2.imwrite(f, crop) # save BGR, https://github.com/ultralytics/yolov5/issues/7007 chroma subsampling issue\r\n Image.fromarray(crop[..., ::-1]).save(f, quality=95, subsampling=0) # save RGB\r\n return crop\r\n\r\n\r\n@threaded\r\ndef plot_images(images,\r\n batch_idx,\r\n cls,\r\n bboxes,\r\n masks=np.zeros(0, dtype=np.uint8),\r\n paths=None,\r\n fname='images.jpg',\r\n names=None):\r\n # Plot image grid with labels\r\n if isinstance(images, torch.Tensor):\r\n images = images.cpu().float().numpy()\r\n if isinstance(cls, torch.Tensor):\r\n cls = cls.cpu().numpy()\r\n if isinstance(bboxes, torch.Tensor):\r\n bboxes = bboxes.cpu().numpy()\r\n if isinstance(masks, torch.Tensor):\r\n masks = masks.cpu().numpy().astype(int)\r\n if isinstance(batch_idx, torch.Tensor):\r\n batch_idx = batch_idx.cpu().numpy()\r\n\r\n max_size = 1920 # max image size\r\n max_subplots = 16 # max image subplots, i.e. 4x4\r\n bs, _, h, w = images.shape # batch size, _, height, width\r\n bs = min(bs, max_subplots) # limit plot images\r\n ns = np.ceil(bs ** 0.5) # number of subplots (square)\r\n if np.max(images[0]) <= 1:\r\n images *= 255 # de-normalise (optional)\r\n\r\n # Build Image\r\n mosaic = np.full((int(ns * h), int(ns * w), 3), 255, dtype=np.uint8) # init\r\n for i, im in enumerate(images):\r\n if i == max_subplots: # if last batch has fewer images than we expect\r\n break\r\n x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin\r\n im = im.transpose(1, 2, 0)\r\n mosaic[y:y + h, x:x + w, :] = im\r\n\r\n # Resize (optional)\r\n scale = max_size / ns / max(h, w)\r\n if scale < 1:\r\n h = math.ceil(scale * h)\r\n w = math.ceil(scale * w)\r\n mosaic = cv2.resize(mosaic, tuple(int(x * ns) for x in (w, h)))\r\n\r\n # Annotate\r\n fs = int((h + w) * ns * 0.01) # font size\r\n annotator = Annotator(mosaic, line_width=round(fs / 10), font_size=fs, pil=True, example=names)\r\n for i in range(i + 1):\r\n x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin\r\n annotator.rectangle([x, y, x + w, y + h], None, (255, 255, 255), width=2) # borders\r\n if paths:\r\n annotator.text((x + 5, y + 5 + h), text=Path(paths[i]).name[:40], txt_color=(220, 220, 220)) # filenames\r\n if len(cls) > 0:\r\n idx = batch_idx == i\r\n\r\n boxes = xywh2xyxy(bboxes[idx, :4]).T\r\n classes = cls[idx].astype('int')\r\n labels = bboxes.shape[1] == 4 # labels if no conf column\r\n conf = None if labels else bboxes[idx, 4] # check for confidence presence (label vs pred)\r\n\r\n if boxes.shape[1]:\r\n if boxes.max() <= 1.01: # if normalized with tolerance 0.01\r\n boxes[[0, 2]] *= w # scale to pixels\r\n boxes[[1, 3]] *= h\r\n elif scale < 1: # absolute coords need scale if image scales\r\n boxes *= scale\r\n boxes[[0, 2]] += x\r\n boxes[[1, 3]] += y\r\n for j, box in enumerate(boxes.T.tolist()):\r\n c = classes[j]\r\n color = colors(c)\r\n c = names[c] if names else c\r\n if labels or conf[j] > 0.25: # 0.25 conf thresh\r\n label = f'{c}' if labels else f'{c} {conf[j]:.1f}'\r\n annotator.box_label(box, label, color=color)\r\n\r\n # Plot masks\r\n if len(masks):\r\n if masks.max() > 1.0: # mean that masks are overlap\r\n image_masks = masks[[i]] # (1, 640, 640)\r\n nl = idx.sum()\r\n index = np.arange(nl).reshape(nl, 1, 1) + 1\r\n image_masks = np.repeat(image_masks, nl, axis=0)\r\n image_masks = np.where(image_masks == index, 1.0, 0.0)\r\n else:\r\n image_masks = masks[idx]\r\n\r\n im = np.asarray(annotator.im).copy()\r\n for j, box in enumerate(boxes.T.tolist()):\r\n if labels or conf[j] > 0.25: # 0.25 conf thresh\r\n color = colors(classes[j])\r\n mh, mw = image_masks[j].shape\r\n if mh != h or mw != w:\r\n mask = image_masks[j].astype(np.uint8)\r\n mask = cv2.resize(mask, (w, h))\r\n mask = mask.astype(bool)\r\n else:\r\n mask = image_masks[j].astype(bool)\r\n with contextlib.suppress(Exception):\r\n im[y:y + h, x:x + w, :][mask] = im[y:y + h, x:x + w, :][mask] * 0.4 + np.array(color) * 0.6\r\n annotator.fromarray(im)\r\n annotator.im.save(fname) # save\r\n\r\n\r\ndef plot_results(file='path/to/results.csv', dir='', segment=False):\r\n # Plot training results.csv. Usage: from utils.plots import *; plot_results('path/to/results.csv')\r\n save_dir = Path(file).parent if file else Path(dir)\r\n if segment:\r\n fig, ax = plt.subplots(2, 8, figsize=(18, 6), tight_layout=True)\r\n index = [1, 2, 3, 4, 5, 6, 9, 10, 13, 14, 15, 16, 7, 8, 11, 12]\r\n else:\r\n fig, ax = plt.subplots(2, 5, figsize=(12, 6), tight_layout=True)\r\n index = [1, 2, 3, 4, 5, 8, 9, 10, 6, 7]\r\n ax = ax.ravel()\r\n files = list(save_dir.glob('results*.csv'))\r\n assert len(files), f'No results.csv files found in {save_dir.resolve()}, nothing to plot.'\r\n for f in files:\r\n try:\r\n data = pd.read_csv(f)\r\n s = [x.strip() for x in data.columns]\r\n x = data.values[:, 0]\r\n for i, j in enumerate(index):\r\n y = data.values[:, j].astype('float')\r\n # y[y == 0] = np.nan # don't show zero values\r\n ax[i].plot(x, y, marker='.', label=f.stem, linewidth=2, markersize=8)\r\n ax[i].set_title(s[j], fontsize=12)\r\n # if j in [8, 9, 10]: # share train and val loss y axes\r\n # ax[i].get_shared_y_axes().join(ax[i], ax[i - 5])\r\n except Exception as e:\r\n print(f'Warning: Plotting error for {f}: {e}')\r\n ax[1].legend()\r\n fig.savefig(save_dir / 'results.png', dpi=200)\r\n plt.close()\r\n\r\n\r\ndef output_to_target(output, max_det=300):\r\n # Convert model output to target format [batch_id, class_id, x, y, w, h, conf] for plotting\r\n targets = []\r\n for i, o in enumerate(output):\r\n box, conf, cls = o[:max_det, :6].cpu().split((4, 1, 1), 1)\r\n j = torch.full((conf.shape[0], 1), i)\r\n targets.append(torch.cat((j, cls, xyxy2xywh(box), conf), 1))\r\n targets = torch.cat(targets, 0).numpy()\r\n return targets[:, 0], targets[:, 1], targets[:, 2:]\r\n"
},
{
"path": "yolo/utils/tal.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.nn.functional as F\r\n\r\nfrom .checks import check_version\r\nfrom .metrics import bbox_iou\r\n\r\nTORCH_1_10 = check_version(torch.__version__, '1.10.0')\r\n\r\n\r\ndef select_candidates_in_gts(xy_centers, gt_bboxes, eps=1e-9):\r\n \"\"\"select the positive anchor center in gt\r\n\r\n Args:\r\n xy_centers (Tensor): shape(h*w, 4)\r\n gt_bboxes (Tensor): shape(b, n_boxes, 4)\r\n Return:\r\n (Tensor): shape(b, n_boxes, h*w)\r\n \"\"\"\r\n n_anchors = xy_centers.shape[0]\r\n bs, n_boxes, _ = gt_bboxes.shape\r\n lt, rb = gt_bboxes.view(-1, 1, 4).chunk(2, 2) # left-top, right-bottom\r\n bbox_deltas = torch.cat((xy_centers[None] - lt, rb - xy_centers[None]), dim=2).view(bs, n_boxes, n_anchors, -1)\r\n # return (bbox_deltas.min(3)[0] > eps).to(gt_bboxes.dtype)\r\n return bbox_deltas.amin(3).gt_(eps)\r\n\r\n\r\ndef select_highest_overlaps(mask_pos, overlaps, n_max_boxes):\r\n \"\"\"if an anchor box is assigned to multiple gts,\r\n the one with the highest iou will be selected.\r\n\r\n Args:\r\n mask_pos (Tensor): shape(b, n_max_boxes, h*w)\r\n overlaps (Tensor): shape(b, n_max_boxes, h*w)\r\n Return:\r\n target_gt_idx (Tensor): shape(b, h*w)\r\n fg_mask (Tensor): shape(b, h*w)\r\n mask_pos (Tensor): shape(b, n_max_boxes, h*w)\r\n \"\"\"\r\n # (b, n_max_boxes, h*w) -> (b, h*w)\r\n fg_mask = mask_pos.sum(-2)\r\n if fg_mask.max() > 1: # one anchor is assigned to multiple gt_bboxes\r\n mask_multi_gts = (fg_mask.unsqueeze(1) > 1).repeat([1, n_max_boxes, 1]) # (b, n_max_boxes, h*w)\r\n max_overlaps_idx = overlaps.argmax(1) # (b, h*w)\r\n is_max_overlaps = F.one_hot(max_overlaps_idx, n_max_boxes) # (b, h*w, n_max_boxes)\r\n is_max_overlaps = is_max_overlaps.permute(0, 2, 1).to(overlaps.dtype) # (b, n_max_boxes, h*w)\r\n mask_pos = torch.where(mask_multi_gts, is_max_overlaps, mask_pos) # (b, n_max_boxes, h*w)\r\n fg_mask = mask_pos.sum(-2)\r\n # find each grid serve which gt(index)\r\n target_gt_idx = mask_pos.argmax(-2) # (b, h*w)\r\n return target_gt_idx, fg_mask, mask_pos\r\n\r\n\r\nclass TaskAlignedAssigner(nn.Module):\r\n\r\n def __init__(self, topk=13, num_classes=80, alpha=1.0, beta=6.0, eps=1e-9):\r\n super().__init__()\r\n self.topk = topk\r\n self.num_classes = num_classes\r\n self.bg_idx = num_classes\r\n self.alpha = alpha\r\n self.beta = beta\r\n self.eps = eps\r\n\r\n @torch.no_grad()\r\n def forward(self, pd_scores, pd_bboxes, anc_points, gt_labels, gt_bboxes, mask_gt):\r\n \"\"\"This code referenced to\r\n https://github.com/Nioolek/PPYOLOE_pytorch/blob/master/ppyoloe/assigner/tal_assigner.py\r\n\r\n Args:\r\n pd_scores (Tensor): shape(bs, num_total_anchors, num_classes)\r\n pd_bboxes (Tensor): shape(bs, num_total_anchors, 4)\r\n anc_points (Tensor): shape(num_total_anchors, 2)\r\n gt_labels (Tensor): shape(bs, n_max_boxes, 1)\r\n gt_bboxes (Tensor): shape(bs, n_max_boxes, 4)\r\n mask_gt (Tensor): shape(bs, n_max_boxes, 1)\r\n Returns:\r\n target_labels (Tensor): shape(bs, num_total_anchors)\r\n target_bboxes (Tensor): shape(bs, num_total_anchors, 4)\r\n target_scores (Tensor): shape(bs, num_total_anchors, num_classes)\r\n fg_mask (Tensor): shape(bs, num_total_anchors)\r\n \"\"\"\r\n self.bs = pd_scores.size(0)\r\n self.n_max_boxes = gt_bboxes.size(1)\r\n\r\n if self.n_max_boxes == 0:\r\n device = gt_bboxes.device\r\n return (torch.full_like(pd_scores[..., 0], self.bg_idx).to(device), torch.zeros_like(pd_bboxes).to(device),\r\n torch.zeros_like(pd_scores).to(device), torch.zeros_like(pd_scores[..., 0]).to(device),\r\n torch.zeros_like(pd_scores[..., 0]).to(device))\r\n\r\n mask_pos, align_metric, overlaps = self.get_pos_mask(pd_scores, pd_bboxes, gt_labels, gt_bboxes, anc_points,\r\n mask_gt)\r\n\r\n target_gt_idx, fg_mask, mask_pos = select_highest_overlaps(mask_pos, overlaps, self.n_max_boxes)\r\n\r\n # assigned target\r\n target_labels, target_bboxes, target_scores = self.get_targets(gt_labels, gt_bboxes, target_gt_idx, fg_mask)\r\n\r\n # normalize\r\n align_metric *= mask_pos\r\n pos_align_metrics = align_metric.amax(axis=-1, keepdim=True) # b, max_num_obj\r\n pos_overlaps = (overlaps * mask_pos).amax(axis=-1, keepdim=True) # b, max_num_obj\r\n norm_align_metric = (align_metric * pos_overlaps / (pos_align_metrics + self.eps)).amax(-2).unsqueeze(-1)\r\n target_scores = target_scores * norm_align_metric\r\n\r\n return target_labels, target_bboxes, target_scores, fg_mask.bool(), target_gt_idx\r\n\r\n def get_pos_mask(self, pd_scores, pd_bboxes, gt_labels, gt_bboxes, anc_points, mask_gt):\r\n # get anchor_align metric, (b, max_num_obj, h*w)\r\n align_metric, overlaps = self.get_box_metrics(pd_scores, pd_bboxes, gt_labels, gt_bboxes)\r\n # get in_gts mask, (b, max_num_obj, h*w)\r\n mask_in_gts = select_candidates_in_gts(anc_points, gt_bboxes)\r\n # get topk_metric mask, (b, max_num_obj, h*w)\r\n mask_topk = self.select_topk_candidates(align_metric * mask_in_gts,\r\n topk_mask=mask_gt.repeat([1, 1, self.topk]).bool())\r\n # merge all mask to a final mask, (b, max_num_obj, h*w)\r\n mask_pos = mask_topk * mask_in_gts * mask_gt\r\n\r\n return mask_pos, align_metric, overlaps\r\n\r\n def get_box_metrics(self, pd_scores, pd_bboxes, gt_labels, gt_bboxes):\r\n ind = torch.zeros([2, self.bs, self.n_max_boxes], dtype=torch.long) # 2, b, max_num_obj\r\n ind[0] = torch.arange(end=self.bs).view(-1, 1).repeat(1, self.n_max_boxes) # b, max_num_obj\r\n ind[1] = gt_labels.long().squeeze(-1) # b, max_num_obj\r\n # get the scores of each grid for each gt cls\r\n bbox_scores = pd_scores[ind[0], :, ind[1]] # b, max_num_obj, h*w\r\n\r\n overlaps = bbox_iou(gt_bboxes.unsqueeze(2), pd_bboxes.unsqueeze(1), xywh=False, CIoU=True).squeeze(3).clamp(0)\r\n align_metric = bbox_scores.pow(self.alpha) * overlaps.pow(self.beta)\r\n return align_metric, overlaps\r\n\r\n def select_topk_candidates(self, metrics, largest=True, topk_mask=None):\r\n \"\"\"\r\n Args:\r\n metrics: (b, max_num_obj, h*w).\r\n topk_mask: (b, max_num_obj, topk) or None\r\n \"\"\"\r\n\r\n num_anchors = metrics.shape[-1] # h*w\r\n # (b, max_num_obj, topk)\r\n topk_metrics, topk_idxs = torch.topk(metrics, self.topk, dim=-1, largest=largest)\r\n if topk_mask is None:\r\n topk_mask = (topk_metrics.max(-1, keepdim=True) > self.eps).tile([1, 1, self.topk])\r\n # (b, max_num_obj, topk)\r\n topk_idxs = torch.where(topk_mask, topk_idxs, 0)\r\n # (b, max_num_obj, topk, h*w) -> (b, max_num_obj, h*w)\r\n is_in_topk = F.one_hot(topk_idxs, num_anchors).sum(-2)\r\n # filter invalid bboxes\r\n is_in_topk = torch.where(is_in_topk > 1, 0, is_in_topk)\r\n return is_in_topk.to(metrics.dtype)\r\n\r\n def get_targets(self, gt_labels, gt_bboxes, target_gt_idx, fg_mask):\r\n \"\"\"\r\n Args:\r\n gt_labels: (b, max_num_obj, 1)\r\n gt_bboxes: (b, max_num_obj, 4)\r\n target_gt_idx: (b, h*w)\r\n fg_mask: (b, h*w)\r\n \"\"\"\r\n\r\n # assigned target labels, (b, 1)\r\n batch_ind = torch.arange(end=self.bs, dtype=torch.int64, device=gt_labels.device)[..., None]\r\n target_gt_idx = target_gt_idx + batch_ind * self.n_max_boxes # (b, h*w)\r\n target_labels = gt_labels.long().flatten()[target_gt_idx] # (b, h*w)\r\n\r\n # assigned target boxes, (b, max_num_obj, 4) -> (b, h*w)\r\n target_bboxes = gt_bboxes.view(-1, 4)[target_gt_idx]\r\n\r\n # assigned target scores\r\n target_labels.clamp(0)\r\n target_scores = F.one_hot(target_labels, self.num_classes) # (b, h*w, 80)\r\n fg_scores_mask = fg_mask[:, :, None].repeat(1, 1, self.num_classes) # (b, h*w, 80)\r\n target_scores = torch.where(fg_scores_mask > 0, target_scores, 0)\r\n\r\n return target_labels, target_bboxes, target_scores\r\n\r\n\r\ndef make_anchors(feats, strides, grid_cell_offset=0.5):\r\n \"\"\"Generate anchors from features.\"\"\"\r\n anchor_points, stride_tensor = [], []\r\n assert feats is not None\r\n dtype, device = feats[0].dtype, feats[0].device\r\n for i, stride in enumerate(strides):\r\n _, _, h, w = feats[i].shape\r\n sx = torch.arange(end=w, device=device, dtype=dtype) + grid_cell_offset # shift x\r\n sy = torch.arange(end=h, device=device, dtype=dtype) + grid_cell_offset # shift y\r\n sy, sx = torch.meshgrid(sy, sx, indexing='ij') if TORCH_1_10 else torch.meshgrid(sy, sx)\r\n anchor_points.append(torch.stack((sx, sy), -1).view(-1, 2))\r\n stride_tensor.append(torch.full((h * w, 1), stride, dtype=dtype, device=device))\r\n return torch.cat(anchor_points), torch.cat(stride_tensor)\r\n\r\n\r\ndef dist2bbox(distance, anchor_points, xywh=True, dim=-1):\r\n \"\"\"Transform distance(ltrb) to box(xywh or xyxy).\"\"\"\r\n lt, rb = torch.split(distance, 2, dim)\r\n x1y1 = anchor_points - lt\r\n x2y2 = anchor_points + rb\r\n if xywh:\r\n c_xy = (x1y1 + x2y2) / 2\r\n wh = x2y2 - x1y1\r\n return torch.cat((c_xy, wh), dim) # xywh bbox\r\n return torch.cat((x1y1, x2y2), dim) # xyxy bbox\r\n\r\n\r\ndef bbox2dist(anchor_points, bbox, reg_max):\r\n \"\"\"Transform bbox(xyxy) to dist(ltrb).\"\"\"\r\n x1y1, x2y2 = torch.split(bbox, 2, -1)\r\n return torch.cat((anchor_points - x1y1, x2y2 - anchor_points), -1).clamp(0, reg_max - 0.01) # dist (lt, rb)\r\n"
},
{
"path": "yolo/utils/torch_utils.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport math\r\nimport os\r\nimport platform\r\nimport random\r\nimport time\r\nfrom contextlib import contextmanager\r\nfrom copy import deepcopy\r\nfrom pathlib import Path\r\n\r\nimport numpy as np\r\nimport thop\r\nimport torch\r\nimport torch.distributed as dist\r\nimport torch.nn as nn\r\nimport torch.nn.functional as F\r\nfrom torch.nn.parallel import DistributedDataParallel as DDP\r\n\r\nimport ultralytics\r\nfrom yolo.utils import DEFAULT_CONFIG_DICT, DEFAULT_CONFIG_KEYS, LOGGER\r\nfrom yolo.utils.checks import git_describe\r\n\r\nfrom .checks import check_version\r\n\r\nLOCAL_RANK = int(os.getenv('LOCAL_RANK', -1)) # https://pytorch.org/docs/stable/elastic/run.html\r\nRANK = int(os.getenv('RANK', -1))\r\nWORLD_SIZE = int(os.getenv('WORLD_SIZE', 1))\r\n\r\n\r\n@contextmanager\r\ndef torch_distributed_zero_first(local_rank: int):\r\n # Decorator to make all processes in distributed training wait for each local_master to do something\r\n initialized = torch.distributed.is_initialized() # prevent 'Default process group has not been initialized' errors\r\n if initialized and local_rank not in {-1, 0}:\r\n dist.barrier(device_ids=[local_rank])\r\n yield\r\n if initialized and local_rank == 0:\r\n dist.barrier(device_ids=[0])\r\n\r\n\r\ndef smart_inference_mode(torch_1_9=check_version(torch.__version__, '1.9.0')):\r\n # Applies torch.inference_mode() decorator if torch>=1.9.0 else torch.no_grad() decorator\r\n def decorate(fn):\r\n return (torch.inference_mode if torch_1_9 else torch.no_grad)()(fn)\r\n\r\n return decorate\r\n\r\n\r\ndef DDP_model(model):\r\n # Model DDP creation with checks\r\n assert not check_version(torch.__version__, '1.12.0', pinned=True), \\\r\n 'torch==1.12.0 torchvision==0.13.0 DDP training is not supported due to a known issue. ' \\\r\n 'Please upgrade or downgrade torch to use DDP. See https://github.com/ultralytics/yolov5/issues/8395'\r\n if check_version(torch.__version__, '1.11.0'):\r\n return DDP(model, device_ids=[LOCAL_RANK], output_device=LOCAL_RANK, static_graph=True)\r\n else:\r\n return DDP(model, device_ids=[LOCAL_RANK], output_device=LOCAL_RANK)\r\n\r\n\r\ndef select_device(device='', batch_size=0, newline=False):\r\n # device = None or 'cpu' or 0 or '0' or '0,1,2,3'\r\n ver = git_describe() or __version__ # git commit or pip package version\r\n s = f'Ultralytics YOLOv{ver} π Python-{platform.python_version()} torch-{torch.__version__} '\r\n device = str(device).strip().lower().replace('cuda:', '').replace('none', '') # to string, 'cuda:0' to '0'\r\n cpu = device == 'cpu'\r\n mps = device == 'mps' # Apple Metal Performance Shaders (MPS)\r\n if cpu or mps:\r\n os.environ['CUDA_VISIBLE_DEVICES'] = '-1' # force torch.cuda.is_available() = False\r\n elif device: # non-cpu device requested\r\n os.environ['CUDA_VISIBLE_DEVICES'] = device # set environment variable - must be before assert is_available()\r\n assert torch.cuda.is_available() and torch.cuda.device_count() >= len(device.replace(',', '')), \\\r\n f\"Invalid CUDA '--device {device}' requested, use '--device cpu' or pass valid CUDA device(s)\"\r\n\r\n if not cpu and not mps and torch.cuda.is_available(): # prefer GPU if available\r\n devices = device.split(',') if device else '0' # range(torch.cuda.device_count()) # i.e. 0,1,6,7\r\n n = len(devices) # device count\r\n if n > 1 and batch_size > 0: # check batch_size is divisible by device_count\r\n assert batch_size % n == 0, f'batch-size {batch_size} not multiple of GPU count {n}'\r\n space = ' ' * (len(s) + 1)\r\n for i, d in enumerate(devices):\r\n p = torch.cuda.get_device_properties(i)\r\n s += f\"{'' if i == 0 else space}CUDA:{d} ({p.name}, {p.total_memory / (1 << 20):.0f}MiB)\\n\" # bytes to MB\r\n arg = 'cuda:0'\r\n elif mps and getattr(torch, 'has_mps', False) and torch.backends.mps.is_available(): # prefer MPS if available\r\n s += 'MPS\\n'\r\n arg = 'mps'\r\n else: # revert to CPU\r\n s += 'CPU\\n'\r\n arg = 'cpu'\r\n\r\n if RANK == -1:\r\n LOGGER.info(s if newline else s.rstrip())\r\n return torch.device(arg)\r\n\r\n\r\ndef time_sync():\r\n # PyTorch-accurate time\r\n if torch.cuda.is_available():\r\n torch.cuda.synchronize()\r\n return time.time()\r\n\r\n\r\ndef fuse_conv_and_bn(conv, bn):\r\n # Fuse Conv2d() and BatchNorm2d() layers https://tehnokv.com/posts/fusing-batchnorm-and-conv/\r\n fusedconv = nn.Conv2d(conv.in_channels,\r\n conv.out_channels,\r\n kernel_size=conv.kernel_size,\r\n stride=conv.stride,\r\n padding=conv.padding,\r\n dilation=conv.dilation,\r\n groups=conv.groups,\r\n bias=True).requires_grad_(False).to(conv.weight.device)\r\n\r\n # Prepare filters\r\n w_conv = conv.weight.clone().view(conv.out_channels, -1)\r\n w_bn = torch.diag(bn.weight.div(torch.sqrt(bn.eps + bn.running_var)))\r\n fusedconv.weight.copy_(torch.mm(w_bn, w_conv).view(fusedconv.weight.shape))\r\n\r\n # Prepare spatial bias\r\n b_conv = torch.zeros(conv.weight.size(0), device=conv.weight.device) if conv.bias is None else conv.bias\r\n b_bn = bn.bias - bn.weight.mul(bn.running_mean).div(torch.sqrt(bn.running_var + bn.eps))\r\n fusedconv.bias.copy_(torch.mm(w_bn, b_conv.reshape(-1, 1)).reshape(-1) + b_bn)\r\n\r\n return fusedconv\r\n\r\n\r\ndef model_info(model, verbose=False, imgsz=640):\r\n # Model information. imgsz may be int or list, i.e. imgsz=640 or imgsz=[640, 320]\r\n n_p = get_num_params(model)\r\n n_g = get_num_gradients(model) # number gradients\r\n if verbose:\r\n print(f\"{'layer':>5} {'name':>40} {'gradient':>9} {'parameters':>12} {'shape':>20} {'mu':>10} {'sigma':>10}\")\r\n for i, (name, p) in enumerate(model.named_parameters()):\r\n name = name.replace('module_list.', '')\r\n print('%5g %40s %9s %12g %20s %10.3g %10.3g' %\r\n (i, name, p.requires_grad, p.numel(), list(p.shape), p.mean(), p.std()))\r\n\r\n flops = get_flops(model, imgsz)\r\n fs = f', {flops:.1f} GFLOPs' if flops else ''\r\n m = Path(getattr(model, 'yaml_file', '') or model.yaml.get('yaml_file', '')).stem.replace('yolo', 'YOLO') or 'Model'\r\n LOGGER.info(f\"{m} summary: {len(list(model.modules()))} layers, {n_p} parameters, {n_g} gradients{fs}\")\r\n\r\n\r\ndef get_num_params(model):\r\n return sum(x.numel() for x in model.parameters())\r\n\r\n\r\ndef get_num_gradients(model):\r\n return sum(x.numel() for x in model.parameters() if x.requires_grad)\r\n\r\n\r\ndef get_flops(model, imgsz=640):\r\n try:\r\n model = de_parallel(model)\r\n p = next(model.parameters())\r\n stride = max(int(model.stride.max()), 32) if hasattr(model, 'stride') else 32 # max stride\r\n im = torch.empty((1, p.shape[1], stride, stride), device=p.device) # input image in BCHW format\r\n flops = thop.profile(deepcopy(model), inputs=(im,), verbose=False)[0] / 1E9 * 2 # stride GFLOPs\r\n imgsz = imgsz if isinstance(imgsz, list) else [imgsz, imgsz] # expand if int/float\r\n flops = flops * imgsz[0] / stride * imgsz[1] / stride # 640x640 GFLOPs\r\n return flops\r\n except Exception:\r\n return 0\r\n\r\n\r\ndef initialize_weights(model):\r\n for m in model.modules():\r\n t = type(m)\r\n if t is nn.Conv2d:\r\n pass # nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')\r\n elif t is nn.BatchNorm2d:\r\n m.eps = 1e-3\r\n m.momentum = 0.03\r\n elif t in [nn.Hardswish, nn.LeakyReLU, nn.ReLU, nn.ReLU6, nn.SiLU]:\r\n m.inplace = True\r\n\r\n\r\ndef scale_img(img, ratio=1.0, same_shape=False, gs=32): # img(16,3,256,416)\r\n # Scales img(bs,3,y,x) by ratio constrained to gs-multiple\r\n if ratio == 1.0:\r\n return img\r\n h, w = img.shape[2:]\r\n s = (int(h * ratio), int(w * ratio)) # new size\r\n img = F.interpolate(img, size=s, mode='bilinear', align_corners=False) # resize\r\n if not same_shape: # pad/crop img\r\n h, w = (math.ceil(x * ratio / gs) * gs for x in (h, w))\r\n return F.pad(img, [0, w - s[1], 0, h - s[0]], value=0.447) # value = imagenet mean\r\n\r\n\r\ndef make_divisible(x, divisor):\r\n # Returns nearest x divisible by divisor\r\n if isinstance(divisor, torch.Tensor):\r\n divisor = int(divisor.max()) # to int\r\n return math.ceil(x / divisor) * divisor\r\n\r\n\r\ndef copy_attr(a, b, include=(), exclude=()):\r\n # Copy attributes from b to a, options to only include [...] and to exclude [...]\r\n for k, v in b.__dict__.items():\r\n if (len(include) and k not in include) or k.startswith('_') or k in exclude:\r\n continue\r\n else:\r\n setattr(a, k, v)\r\n\r\n\r\ndef intersect_dicts(da, db, exclude=()):\r\n # Dictionary intersection of matching keys and shapes, omitting 'exclude' keys, using da values\r\n return {k: v for k, v in da.items() if k in db and all(x not in k for x in exclude) and v.shape == db[k].shape}\r\n\r\n\r\ndef is_parallel(model):\r\n # Returns True if model is of type DP or DDP\r\n return type(model) in (nn.parallel.DataParallel, nn.parallel.DistributedDataParallel)\r\n\r\n\r\ndef de_parallel(model):\r\n # De-parallelize a model: returns single-GPU model if model is of type DP or DDP\r\n return model.module if is_parallel(model) else model\r\n\r\n\r\ndef one_cycle(y1=0.0, y2=1.0, steps=100):\r\n # lambda function for sinusoidal ramp from y1 to y2 https://arxiv.org/pdf/1812.01187.pdf\r\n return lambda x: ((1 - math.cos(x * math.pi / steps)) / 2) * (y2 - y1) + y1\r\n\r\n\r\ndef init_seeds(seed=0, deterministic=False):\r\n # Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html\r\n random.seed(seed)\r\n np.random.seed(seed)\r\n torch.manual_seed(seed)\r\n torch.cuda.manual_seed(seed)\r\n torch.cuda.manual_seed_all(seed) # for Multi-GPU, exception safe\r\n # torch.backends.cudnn.benchmark = True # AutoBatch problem https://github.com/ultralytics/yolov5/issues/9287\r\n if deterministic and check_version(torch.__version__, '1.12.0'): # https://github.com/ultralytics/yolov5/pull/8213\r\n torch.use_deterministic_algorithms(True)\r\n torch.backends.cudnn.deterministic = True\r\n os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'\r\n os.environ['PYTHONHASHSEED'] = str(seed)\r\n\r\n\r\nclass ModelEMA:\r\n \"\"\" Updated Exponential Moving Average (EMA) from https://github.com/rwightman/pytorch-image-models\r\n Keeps a moving average of everything in the model state_dict (parameters and buffers)\r\n For EMA details see https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage\r\n \"\"\"\r\n\r\n def __init__(self, model, decay=0.9999, tau=2000, updates=0):\r\n # Create EMA\r\n self.ema = deepcopy(de_parallel(model)).eval() # FP32 EMA\r\n self.updates = updates # number of EMA updates\r\n self.decay = lambda x: decay * (1 - math.exp(-x / tau)) # decay exponential ramp (to help early epochs)\r\n for p in self.ema.parameters():\r\n p.requires_grad_(False)\r\n\r\n def update(self, model):\r\n # Update EMA parameters\r\n self.updates += 1\r\n d = self.decay(self.updates)\r\n\r\n msd = de_parallel(model).state_dict() # model state_dict\r\n for k, v in self.ema.state_dict().items():\r\n if v.dtype.is_floating_point: # true for FP16 and FP32\r\n v *= d\r\n v += (1 - d) * msd[k].detach()\r\n # assert v.dtype == msd[k].dtype == torch.float32, f'{k}: EMA {v.dtype} and model {msd[k].dtype} must be FP32'\r\n\r\n def update_attr(self, model, include=(), exclude=('process_group', 'reducer')):\r\n # Update EMA attributes\r\n copy_attr(self.ema, model, include, exclude)\r\n\r\n\r\ndef strip_optimizer(f='best.pt', s=''):\r\n \"\"\"\r\n Strip optimizer from 'f' to finalize training, optionally save as 's'.\r\n\r\n Usage:\r\n from yolo.utils.torch_utils import strip_optimizer\r\n from pathlib import Path\r\n for f in Path('/Users/glennjocher/Downloads/weights').glob('*.pt'):\r\n strip_optimizer(f)\r\n\r\n Args:\r\n f (str): file path to model state to strip the optimizer from. Default is 'best.pt'.\r\n s (str): file path to save the model with stripped optimizer to. Default is ''. If not provided, the original file will be overwritten.\r\n\r\n Returns:\r\n None\r\n \"\"\"\r\n x = torch.load(f, map_location=torch.device('cpu'))\r\n args = {**DEFAULT_CONFIG_DICT, **x['train_args']} # combine model args with default args, preferring model args\r\n if x.get('ema'):\r\n x['model'] = x['ema'] # replace model with ema\r\n for k in 'optimizer', 'best_fitness', 'ema', 'updates': # keys\r\n x[k] = None\r\n x['epoch'] = -1\r\n x['model'].half() # to FP16\r\n for p in x['model'].parameters():\r\n p.requires_grad = False\r\n x['train_args'] = {k: v for k, v in args.items() if k in DEFAULT_CONFIG_KEYS} # strip non-default keys\r\n torch.save(x, s or f)\r\n mb = os.path.getsize(s or f) / 1E6 # filesize\r\n LOGGER.info(f\"Optimizer stripped from {f},{f' saved as {s},' if s else ''} {mb:.1f}MB\")\r\n\r\n\r\ndef guess_task_from_head(head):\r\n task = None\r\n if head.lower() in [\"classify\", \"classifier\", \"cls\", \"fc\"]:\r\n task = \"classify\"\r\n if head.lower() in [\"detect\"]:\r\n task = \"detect\"\r\n if head.lower() in [\"segment\"]:\r\n task = \"segment\"\r\n\r\n if not task:\r\n raise SyntaxError(\"task or model not recognized! Please refer the docs at : \") # TODO: add docs links\r\n\r\n return task\r\n\r\n\r\ndef profile(input, ops, n=10, device=None):\r\n \"\"\" YOLOv5 speed/memory/FLOPs profiler\r\n Usage:\r\n input = torch.randn(16, 3, 640, 640)\r\n m1 = lambda x: x * torch.sigmoid(x)\r\n m2 = nn.SiLU()\r\n profile(input, [m1, m2], n=100) # profile over 100 iterations\r\n \"\"\"\r\n results = []\r\n if not isinstance(device, torch.device):\r\n device = select_device(device)\r\n print(f\"{'Params':>12s}{'GFLOPs':>12s}{'GPU_mem (GB)':>14s}{'forward (ms)':>14s}{'backward (ms)':>14s}\"\r\n f\"{'input':>24s}{'output':>24s}\")\r\n\r\n for x in input if isinstance(input, list) else [input]:\r\n x = x.to(device)\r\n x.requires_grad = True\r\n for m in ops if isinstance(ops, list) else [ops]:\r\n m = m.to(device) if hasattr(m, 'to') else m # device\r\n m = m.half() if hasattr(m, 'half') and isinstance(x, torch.Tensor) and x.dtype is torch.float16 else m\r\n tf, tb, t = 0, 0, [0, 0, 0] # dt forward, backward\r\n try:\r\n flops = thop.profile(m, inputs=(x,), verbose=False)[0] / 1E9 * 2 # GFLOPs\r\n except Exception:\r\n flops = 0\r\n\r\n try:\r\n for _ in range(n):\r\n t[0] = time_sync()\r\n y = m(x)\r\n t[1] = time_sync()\r\n try:\r\n _ = (sum(yi.sum() for yi in y) if isinstance(y, list) else y).sum().backward()\r\n t[2] = time_sync()\r\n except Exception: # no backward method\r\n # print(e) # for debug\r\n t[2] = float('nan')\r\n tf += (t[1] - t[0]) * 1000 / n # ms per op forward\r\n tb += (t[2] - t[1]) * 1000 / n # ms per op backward\r\n mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0 # (GB)\r\n s_in, s_out = (tuple(x.shape) if isinstance(x, torch.Tensor) else 'list' for x in (x, y)) # shapes\r\n p = sum(x.numel() for x in m.parameters()) if isinstance(m, nn.Module) else 0 # parameters\r\n print(f'{p:12}{flops:12.4g}{mem:>14.3f}{tf:14.4g}{tb:14.4g}{str(s_in):>24s}{str(s_out):>24s}')\r\n results.append([p, flops, mem, tf, tb, s_in, s_out])\r\n except Exception as e:\r\n print(e)\r\n results.append(None)\r\n torch.cuda.empty_cache()\r\n return results\r\n"
},
{
"path": "yolo/v8/__init__.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom pathlib import Path\r\n\r\nfrom yolo.v8 import classify, detect, segment\r\n\r\nROOT = Path(__file__).parents[0] # yolov8 ROOT\r\n\r\n__all__ = [\"classify\", \"segment\", \"detect\"]\r\n\r\nfrom yolo.configs import hydra_patch # noqa (patch hydra cli)\r\n"
},
{
"path": "yolo/v8/detect/__init__.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom .predict import DetectionPredictor, predict\r\nfrom .train import DetectionTrainer, train\r\nfrom .val import DetectionValidator, val\r\n"
},
{
"path": "yolo/v8/detect/detect_and_trk.py",
"content": "import hydra\r\nimport torch\r\nimport cv2\r\nfrom random import randint\r\nfrom sort import *\r\nfrom ultralytics.yolo.engine.predictor import BasePredictor\r\nfrom ultralytics.yolo.utils import DEFAULT_CONFIG, ROOT, ops\r\nfrom ultralytics.yolo.utils.checks import check_imgsz\r\nfrom ultralytics.yolo.utils.plotting import Annotator, colors, save_one_box\r\n\r\ntracker = None\r\n\r\ndef init_tracker():\r\n global tracker\r\n \r\n sort_max_age = 5 \r\n sort_min_hits = 2\r\n sort_iou_thresh = 0.2\r\n tracker =Sort(max_age=sort_max_age,min_hits=sort_min_hits,iou_threshold=sort_iou_thresh)\r\n\r\nrand_color_list = []\r\n \r\ndef draw_boxes(img, bbox, identities=None, categories=None, names=None, offset=(0, 0)):\r\n\r\n for i, box in enumerate(bbox):\r\n x1, y1, x2, y2 = [int(i) for i in box]\r\n x1 += offset[0]\r\n x2 += offset[0]\r\n y1 += offset[1]\r\n y2 += offset[1]\r\n id = int(identities[i]) if identities is not None else 0\r\n box_center = (int((box[0]+box[2])/2),(int((box[1]+box[3])/2)))\r\n label = str(id)\r\n (w, h), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.6, 1)\r\n cv2.rectangle(img, (x1, y1), (x2, y2), (0, 255, 253), 2)\r\n cv2.rectangle(img, (x1, y1 - 20), (x1 + w, y1), (255,144,30), -1)\r\n cv2.putText(img, label, (x1, y1 - 5),cv2.FONT_HERSHEY_SIMPLEX, 0.6, [255, 255, 255], 1)\r\n \r\n \r\n return img\r\n\r\ndef random_color_list():\r\n global rand_color_list\r\n rand_color_list = []\r\n for i in range(0,5005):\r\n r = randint(0, 255)\r\n g = randint(0, 255)\r\n b = randint(0, 255)\r\n rand_color = (r, g, b)\r\n rand_color_list.append(rand_color)\r\n #......................................\r\n \r\n\r\n\r\nclass DetectionPredictor(BasePredictor):\r\n \r\n def get_annotator(self, img):\r\n return Annotator(img, line_width=self.args.line_thickness, example=str(self.model.names))\r\n\r\n def preprocess(self, img):\r\n img = torch.from_numpy(img).to(self.model.device)\r\n img = img.half() if self.model.fp16 else img.float() # uint8 to fp16/32\r\n img /= 255 # 0 - 255 to 0.0 - 1.0\r\n return img\r\n\r\n def postprocess(self, preds, img, orig_img):\r\n preds = ops.non_max_suppression(preds,\r\n self.args.conf,\r\n self.args.iou,\r\n agnostic=self.args.agnostic_nms,\r\n max_det=self.args.max_det)\r\n\r\n for i, pred in enumerate(preds):\r\n shape = orig_img[i].shape if self.webcam else orig_img.shape\r\n pred[:, :4] = ops.scale_boxes(img.shape[2:], pred[:, :4], shape).round()\r\n\r\n return preds\r\n\r\n def write_results(self, idx, preds, batch):\r\n\r\n p, im, im0 = batch\r\n log_string = \"\"\r\n if len(im.shape) == 3:\r\n im = im[None] # expand for batch dim\r\n self.seen += 1\r\n im0 = im0.copy()\r\n if self.webcam: # batch_size >= 1\r\n log_string += f'{idx}: '\r\n frame = self.dataset.count\r\n else:\r\n frame = getattr(self.dataset, 'frame', 0)\r\n # tracker\r\n self.data_path = p\r\n \r\n save_path = str(self.save_dir / p.name) # im.jpg\r\n self.txt_path = str(self.save_dir / 'labels' / p.stem) + ('' if self.dataset.mode == 'image' else f'_{frame}')\r\n log_string += '%gx%g ' % im.shape[2:] # print string\r\n self.annotator = self.get_annotator(im0)\r\n \r\n det = preds[idx]\r\n self.all_outputs.append(det)\r\n if len(det) == 0:\r\n return log_string\r\n for c in det[:, 5].unique():\r\n n = (det[:, 5] == c).sum() # detections per class\r\n log_string += f\"{n} {self.model.names[int(c)]}{'s' * (n > 1)}, \"\r\n \r\n \r\n # #..................USE TRACK FUNCTION....................\r\n dets_to_sort = np.empty((0,6))\r\n \r\n for x1,y1,x2,y2,conf,detclass in det.cpu().detach().numpy():\r\n dets_to_sort = np.vstack((dets_to_sort, \r\n np.array([x1, y1, x2, y2, conf, detclass])))\r\n \r\n tracked_dets = tracker.update(dets_to_sort)\r\n tracks =tracker.getTrackers()\r\n \r\n for track in tracks:\r\n [cv2.line(im0, (int(track.centroidarr[i][0]),\r\n int(track.centroidarr[i][1])), \r\n (int(track.centroidarr[i+1][0]),\r\n int(track.centroidarr[i+1][1])),\r\n rand_color_list[track.id], thickness=3) \r\n for i,_ in enumerate(track.centroidarr) \r\n if i < len(track.centroidarr)-1 ] \r\n \r\n\r\n if len(tracked_dets)>0:\r\n bbox_xyxy = tracked_dets[:,:4]\r\n identities = tracked_dets[:, 8]\r\n categories = tracked_dets[:, 4]\r\n draw_boxes(im0, bbox_xyxy, identities, categories, self.model.names)\r\n \r\n gn = torch.tensor(im0.shape)[[1, 0, 1, 0]] # normalization gain whwh\r\n \r\n return log_string\r\n\r\n\r\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent), config_name=DEFAULT_CONFIG.name)\r\ndef predict(cfg):\r\n init_tracker()\r\n random_color_list()\r\n \r\n cfg.model = cfg.model or \"yolov8n.pt\"\r\n cfg.imgsz = check_imgsz(cfg.imgsz, min_dim=2) # check image size\r\n cfg.source = cfg.source if cfg.source is not None else ROOT / \"assets\"\r\n predictor = DetectionPredictor(cfg)\r\n predictor()\r\n\r\n\r\nif __name__ == \"__main__\":\r\n predict()\r\n"
},
{
"path": "yolo/v8/detect/predict.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\n\nimport hydra\nimport torch\n\nfrom ultralytics.yolo.engine.predictor import BasePredictor\nfrom ultralytics.yolo.utils import DEFAULT_CONFIG, ROOT, ops\nfrom ultralytics.yolo.utils.checks import check_imgsz\nfrom ultralytics.yolo.utils.plotting import Annotator, colors, save_one_box\n\n\nclass DetectionPredictor(BasePredictor):\n\n def get_annotator(self, img):\n return Annotator(img, line_width=self.args.line_thickness, example=str(self.model.names))\n\n def preprocess(self, img):\n img = torch.from_numpy(img).to(self.model.device)\n img = img.half() if self.model.fp16 else img.float() # uint8 to fp16/32\n img /= 255 # 0 - 255 to 0.0 - 1.0\n return img\n\n def postprocess(self, preds, img, orig_img):\n preds = ops.non_max_suppression(preds,\n self.args.conf,\n self.args.iou,\n agnostic=self.args.agnostic_nms,\n max_det=self.args.max_det)\n\n for i, pred in enumerate(preds):\n shape = orig_img[i].shape if self.webcam else orig_img.shape\n pred[:, :4] = ops.scale_boxes(img.shape[2:], pred[:, :4], shape).round()\n\n return preds\n\n def write_results(self, idx, preds, batch):\n p, im, im0 = batch\n log_string = \"\"\n if len(im.shape) == 3:\n im = im[None] # expand for batch dim\n self.seen += 1\n im0 = im0.copy()\n if self.webcam: # batch_size >= 1\n log_string += f'{idx}: '\n frame = self.dataset.count\n else:\n frame = getattr(self.dataset, 'frame', 0)\n\n self.data_path = p\n # save_path = str(self.save_dir / p.name) # im.jpg\n self.txt_path = str(self.save_dir / 'labels' / p.stem) + ('' if self.dataset.mode == 'image' else f'_{frame}')\n log_string += '%gx%g ' % im.shape[2:] # print string\n self.annotator = self.get_annotator(im0)\n\n det = preds[idx]\n self.all_outputs.append(det)\n if len(det) == 0:\n return log_string\n for c in det[:, 5].unique():\n n = (det[:, 5] == c).sum() # detections per class\n log_string += f\"{n} {self.model.names[int(c)]}{'s' * (n > 1)}, \"\n # write\n gn = torch.tensor(im0.shape)[[1, 0, 1, 0]] # normalization gain whwh\n for *xyxy, conf, cls in reversed(det):\n if self.args.save_txt: # Write to file\n xywh = (ops.xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh\n line = (cls, *xywh, conf) if self.args.save_conf else (cls, *xywh) # label format\n with open(f'{self.txt_path}.txt', 'a') as f:\n f.write(('%g ' * len(line)).rstrip() % line + '\\n')\n\n if self.args.save or self.args.save_crop or self.args.show: # Add bbox to image\n c = int(cls) # integer class\n label = None if self.args.hide_labels else (\n self.model.names[c] if self.args.hide_conf else f'{self.model.names[c]} {conf:.2f}')\n self.annotator.box_label(xyxy, label, color=colors(c, True))\n if self.args.save_crop:\n imc = im0.copy()\n save_one_box(xyxy,\n imc,\n file=self.save_dir / 'crops' / self.model.model.names[c] / f'{self.data_path.stem}.jpg',\n BGR=True)\n\n return log_string\n\n\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent), config_name=DEFAULT_CONFIG.name)\ndef predict(cfg):\n cfg.model = cfg.model or \"yolov8n.pt\"\n cfg.imgsz = check_imgsz(cfg.imgsz, min_dim=2) # check image size\n cfg.source = cfg.source or ROOT / \"assets\"\n predictor = DetectionPredictor(cfg)\n predictor()\n\n\nif __name__ == \"__main__\":\n predict()\n"
},
{
"path": "yolo/v8/detect/sort.py",
"content": "from __future__ import print_function\n\nimport os\nimport numpy as np\nimport matplotlib\nmatplotlib.use('Agg')\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nfrom skimage import io\n\nimport glob\nimport time\nimport argparse\nfrom filterpy.kalman import KalmanFilter\n\nnp.random.seed(0)\n\ndef linear_assignment(cost_matrix):\n try:\n import lap #linear assignment problem solver\n _, x, y = lap.lapjv(cost_matrix, extend_cost = True)\n return np.array([[y[i],i] for i in x if i>=0])\n except ImportError:\n from scipy.optimize import linear_sum_assignment\n x,y = linear_sum_assignment(cost_matrix)\n return np.array(list(zip(x,y)))\n\n\n\"\"\"From SORT: Computes IOU between two boxes in the form [x1,y1,x2,y2]\"\"\"\ndef iou_batch(bb_test, bb_gt):\n \n bb_gt = np.expand_dims(bb_gt, 0)\n bb_test = np.expand_dims(bb_test, 1)\n \n xx1 = np.maximum(bb_test[...,0], bb_gt[..., 0])\n yy1 = np.maximum(bb_test[..., 1], bb_gt[..., 1])\n xx2 = np.minimum(bb_test[..., 2], bb_gt[..., 2])\n yy2 = np.minimum(bb_test[..., 3], bb_gt[..., 3])\n w = np.maximum(0., xx2 - xx1)\n h = np.maximum(0., yy2 - yy1)\n wh = w * h\n o = wh / ((bb_test[..., 2] - bb_test[..., 0]) * (bb_test[..., 3] - bb_test[..., 1]) \n + (bb_gt[..., 2] - bb_gt[..., 0]) * (bb_gt[..., 3] - bb_gt[..., 1]) - wh)\n return(o)\n\n\n\"\"\"Takes a bounding box in the form [x1,y1,x2,y2] and returns z in the form [x,y,s,r] where x,y is the center of the box and s is the scale/area and r is the aspect ratio\"\"\"\ndef convert_bbox_to_z(bbox):\n w = bbox[2] - bbox[0]\n h = bbox[3] - bbox[1]\n x = bbox[0] + w/2.\n y = bbox[1] + h/2.\n s = w * h \n #scale is just area\n r = w / float(h)\n return np.array([x, y, s, r]).reshape((4, 1))\n\n\n\"\"\"Takes a bounding box in the centre form [x,y,s,r] and returns it in the form\n [x1,y1,x2,y2] where x1,y1 is the top left and x2,y2 is the bottom right\"\"\"\ndef convert_x_to_bbox(x, score=None):\n w = np.sqrt(x[2] * x[3])\n h = x[2] / w\n if(score==None):\n return np.array([x[0]-w/2.,x[1]-h/2.,x[0]+w/2.,x[1]+h/2.]).reshape((1,4))\n else:\n return np.array([x[0]-w/2.,x[1]-h/2.,x[0]+w/2.,x[1]+h/2.,score]).reshape((1,5))\n\n\"\"\"This class represents the internal state of individual tracked objects observed as bbox.\"\"\"\nclass KalmanBoxTracker(object):\n \n count = 0\n def __init__(self, bbox):\n \"\"\"\n Initialize a tracker using initial bounding box\n \n Parameter 'bbox' must have 'detected class' int number at the -1 position.\n \"\"\"\n self.kf = KalmanFilter(dim_x=7, dim_z=4)\n self.kf.F = np.array([[1,0,0,0,1,0,0],[0,1,0,0,0,1,0],[0,0,1,0,0,0,1],[0,0,0,1,0,0,0],[0,0,0,0,1,0,0],[0,0,0,0,0,1,0],[0,0,0,0,0,0,1]])\n self.kf.H = np.array([[1,0,0,0,0,0,0],[0,1,0,0,0,0,0],[0,0,1,0,0,0,0],[0,0,0,1,0,0,0]])\n\n self.kf.R[2:,2:] *= 10. # R: Covariance matrix of measurement noise (set to high for noisy inputs -> more 'inertia' of boxes')\n self.kf.P[4:,4:] *= 1000. #give high uncertainty to the unobservable initial velocities\n self.kf.P *= 10.\n self.kf.Q[-1,-1] *= 0.5 # Q: Covariance matrix of process noise (set to high for erratically moving things)\n self.kf.Q[4:,4:] *= 0.5\n\n self.kf.x[:4] = convert_bbox_to_z(bbox) # STATE VECTOR\n self.time_since_update = 0\n self.id = KalmanBoxTracker.count\n KalmanBoxTracker.count += 1\n self.history = []\n self.hits = 0\n self.hit_streak = 0\n self.age = 0\n self.centroidarr = []\n CX = (bbox[0]+bbox[2])//2\n CY = (bbox[1]+bbox[3])//2\n self.centroidarr.append((CX,CY))\n \n #keep yolov5 detected class information\n self.detclass = bbox[5]\n\n # If we want to store bbox\n self.bbox_history = [bbox]\n \n def update(self, bbox):\n \"\"\"\n Updates the state vector with observed bbox\n \"\"\"\n self.time_since_update = 0\n self.history = []\n self.hits += 1\n self.hit_streak += 1\n self.kf.update(convert_bbox_to_z(bbox))\n self.detclass = bbox[5]\n CX = (bbox[0]+bbox[2])//2\n CY = (bbox[1]+bbox[3])//2\n self.centroidarr.append((CX,CY))\n self.bbox_history.append(bbox)\n \n def predict(self):\n \"\"\"\n Advances the state vector and returns the predicted bounding box estimate\n \"\"\"\n if((self.kf.x[6]+self.kf.x[2])<=0):\n self.kf.x[6] *= 0.0\n self.kf.predict()\n self.age += 1\n if(self.time_since_update>0):\n self.hit_streak = 0\n self.time_since_update += 1\n self.history.append(convert_x_to_bbox(self.kf.x))\n # bbox=self.history[-1]\n # CX = (bbox[0]+bbox[2])/2\n # CY = (bbox[1]+bbox[3])/2\n # self.centroidarr.append((CX,CY))\n \n return self.history[-1]\n \n \n def get_state(self):\n \"\"\"\n Returns the current bounding box estimate\n # test\n arr1 = np.array([[1,2,3,4]])\n arr2 = np.array([0])\n arr3 = np.expand_dims(arr2, 0)\n np.concatenate((arr1,arr3), axis=1)\n \"\"\"\n arr_detclass = np.expand_dims(np.array([self.detclass]), 0)\n \n arr_u_dot = np.expand_dims(self.kf.x[4],0)\n arr_v_dot = np.expand_dims(self.kf.x[5],0)\n arr_s_dot = np.expand_dims(self.kf.x[6],0)\n \n return np.concatenate((convert_x_to_bbox(self.kf.x), arr_detclass, arr_u_dot, arr_v_dot, arr_s_dot), axis=1)\n \ndef associate_detections_to_trackers(detections, trackers, iou_threshold = 0.3):\n \"\"\"\n Assigns detections to tracked object (both represented as bounding boxes)\n Returns 3 lists of \n 1. matches,\n 2. unmatched_detections\n 3. unmatched_trackers\n \"\"\"\n if(len(trackers)==0):\n return np.empty((0,2),dtype=int), np.arange(len(detections)), np.empty((0,5),dtype=int)\n \n iou_matrix = iou_batch(detections, trackers)\n \n if min(iou_matrix.shape) > 0:\n a = (iou_matrix > iou_threshold).astype(np.int32)\n if a.sum(1).max() == 1 and a.sum(0).max() ==1:\n matched_indices = np.stack(np.where(a), axis=1)\n else:\n matched_indices = linear_assignment(-iou_matrix)\n else:\n matched_indices = np.empty(shape=(0,2))\n \n unmatched_detections = []\n for d, det in enumerate(detections):\n if(d not in matched_indices[:,0]):\n unmatched_detections.append(d)\n \n unmatched_trackers = []\n for t, trk in enumerate(trackers):\n if(t not in matched_indices[:,1]):\n unmatched_trackers.append(t)\n \n #filter out matched with low IOU\n matches = []\n for m in matched_indices:\n if(iou_matrix[m[0], m[1]]= self.min_hits or self.frame_count <= self.min_hits):\n ret.append(np.concatenate((d, [trk.id+1])).reshape(1,-1)) #+1'd because MOT benchmark requires positive value\n i -= 1\n #remove dead tracklet\n if(trk.time_since_update >self.max_age):\n self.trackers.pop(i)\n if(len(ret) > 0):\n return np.concatenate(ret)\n return np.empty((0,6))\n\ndef parse_args():\n \"\"\"Parse input arguments.\"\"\"\n parser = argparse.ArgumentParser(description='SORT demo')\n parser.add_argument('--display', dest='display', help='Display online tracker output (slow) [False]',action='store_true')\n parser.add_argument(\"--seq_path\", help=\"Path to detections.\", type=str, default='data')\n parser.add_argument(\"--phase\", help=\"Subdirectory in seq_path.\", type=str, default='train')\n parser.add_argument(\"--max_age\", \n help=\"Maximum number of frames to keep alive a track without associated detections.\", \n type=int, default=1)\n parser.add_argument(\"--min_hits\", \n help=\"Minimum number of associated detections before track is initialised.\", \n type=int, default=3)\n parser.add_argument(\"--iou_threshold\", help=\"Minimum IOU for match.\", type=float, default=0.3)\n args = parser.parse_args()\n return args\n\nif __name__ == '__main__':\n # all train\n args = parse_args()\n display = args.display\n phase = args.phase\n total_time = 0.0\n total_frames = 0\n colours = np.random.rand(32, 3) #used only for display\n if(display):\n if not os.path.exists('mot_benchmark'):\n print('\\n\\tERROR: mot_benchmark link not found!\\n\\n Create a symbolic link to the MOT benchmark\\n (https://motchallenge.net/data/2D_MOT_2015/#download). E.g.:\\n\\n $ ln -s /path/to/MOT2015_challenge/2DMOT2015 mot_benchmark\\n\\n')\n exit()\n plt.ion()\n fig = plt.figure()\n ax1 = fig.add_subplot(111, aspect='equal')\n\n if not os.path.exists('output'):\n os.makedirs('output')\n pattern = os.path.join(args.seq_path, phase, '*', 'det', 'det.txt')\n for seq_dets_fn in glob.glob(pattern):\n mot_tracker = Sort(max_age=args.max_age, \n min_hits=args.min_hits,\n iou_threshold=args.iou_threshold) #create instance of the SORT tracker\n seq_dets = np.loadtxt(seq_dets_fn, delimiter=',')\n seq = seq_dets_fn[pattern.find('*'):].split(os.path.sep)[0]\n \n with open(os.path.join('output', '%s.txt'%(seq)),'w') as out_file:\n print(\"Processing %s.\"%(seq))\n for frame in range(int(seq_dets[:,0].max())):\n frame += 1 #detection and frame numbers begin at 1\n dets = seq_dets[seq_dets[:, 0]==frame, 2:7]\n dets[:, 2:4] += dets[:, 0:2] #convert to [x1,y1,w,h] to [x1,y1,x2,y2]\n total_frames += 1\n\n if(display):\n fn = os.path.join('mot_benchmark', phase, seq, 'img1', '%06d.jpg'%(frame))\n im =io.imread(fn)\n ax1.imshow(im)\n plt.title(seq + ' Tracked Targets')\n\n start_time = time.time()\n trackers = mot_tracker.update(dets)\n cycle_time = time.time() - start_time\n total_time += cycle_time\n\n for d in trackers:\n print('%d,%d,%.2f,%.2f,%.2f,%.2f,1,-1,-1,-1'%(frame,d[4],d[0],d[1],d[2]-d[0],d[3]-d[1]),file=out_file)\n if(display):\n d = d.astype(np.int32)\n ax1.add_patch(patches.Rectangle((d[0],d[1]),d[2]-d[0],d[3]-d[1],fill=False,lw=3,ec=colours[d[4]%32,:]))\n\n if(display):\n fig.canvas.flush_events()\n plt.draw()\n ax1.cla()\n\n print(\"Total Tracking took: %.3f seconds for %d frames or %.1f FPS\" % (total_time, total_frames, total_frames / total_time))\n\n if(display):\n print(\"Note: to get real runtime results run without the option: --display\")\n"
},
{
"path": "yolo/v8/detect/train.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nfrom copy import copy\r\n\r\nimport hydra\r\nimport torch\r\nimport torch.nn as nn\r\n\r\nfrom ultralytics.nn.tasks import DetectionModel\r\nfrom ultralytics.yolo import v8\r\nfrom ultralytics.yolo.data import build_dataloader\r\nfrom ultralytics.yolo.data.dataloaders.v5loader import create_dataloader\r\nfrom ultralytics.yolo.engine.trainer import BaseTrainer\r\nfrom ultralytics.yolo.utils import DEFAULT_CONFIG, colorstr\r\nfrom ultralytics.yolo.utils.loss import BboxLoss\r\nfrom ultralytics.yolo.utils.ops import xywh2xyxy\r\nfrom ultralytics.yolo.utils.plotting import plot_images, plot_results\r\nfrom ultralytics.yolo.utils.tal import TaskAlignedAssigner, dist2bbox, make_anchors\r\nfrom ultralytics.yolo.utils.torch_utils import de_parallel\r\n\r\n\r\n# BaseTrainer python usage\r\nclass DetectionTrainer(BaseTrainer):\r\n\r\n def get_dataloader(self, dataset_path, batch_size, mode=\"train\", rank=0):\r\n # TODO: manage splits differently\r\n # calculate stride - check if model is initialized\r\n gs = max(int(de_parallel(self.model).stride.max() if self.model else 0), 32)\r\n return create_dataloader(path=dataset_path,\r\n imgsz=self.args.imgsz,\r\n batch_size=batch_size,\r\n stride=gs,\r\n hyp=dict(self.args),\r\n augment=mode == \"train\",\r\n cache=self.args.cache,\r\n pad=0 if mode == \"train\" else 0.5,\r\n rect=self.args.rect,\r\n rank=rank,\r\n workers=self.args.workers,\r\n close_mosaic=self.args.close_mosaic != 0,\r\n prefix=colorstr(f'{mode}: '),\r\n shuffle=mode == \"train\",\r\n seed=self.args.seed)[0] if self.args.v5loader else \\\r\n build_dataloader(self.args, batch_size, img_path=dataset_path, stride=gs, rank=rank, mode=mode)[0]\r\n\r\n def preprocess_batch(self, batch):\r\n batch[\"img\"] = batch[\"img\"].to(self.device, non_blocking=True).float() / 255\r\n return batch\r\n\r\n def set_model_attributes(self):\r\n nl = de_parallel(self.model).model[-1].nl # number of detection layers (to scale hyps)\r\n self.args.box *= 3 / nl # scale to layers\r\n # self.args.cls *= self.data[\"nc\"] / 80 * 3 / nl # scale to classes and layers\r\n self.args.cls *= (self.args.imgsz / 640) ** 2 * 3 / nl # scale to image size and layers\r\n self.model.nc = self.data[\"nc\"] # attach number of classes to model\r\n self.model.args = self.args # attach hyperparameters to model\r\n # TODO: self.model.class_weights = labels_to_class_weights(dataset.labels, nc).to(device) * nc\r\n self.model.names = self.data[\"names\"]\r\n\r\n def get_model(self, cfg=None, weights=None, verbose=True):\r\n model = DetectionModel(cfg, ch=3, nc=self.data[\"nc\"], verbose=verbose)\r\n if weights:\r\n model.load(weights)\r\n\r\n return model\r\n\r\n def get_validator(self):\r\n self.loss_names = 'box_loss', 'cls_loss', 'dfl_loss'\r\n return v8.detect.DetectionValidator(self.test_loader,\r\n save_dir=self.save_dir,\r\n logger=self.console,\r\n args=copy(self.args))\r\n\r\n def criterion(self, preds, batch):\r\n if not hasattr(self, 'compute_loss'):\r\n self.compute_loss = Loss(de_parallel(self.model))\r\n return self.compute_loss(preds, batch)\r\n\r\n def label_loss_items(self, loss_items=None, prefix=\"train\"):\r\n \"\"\"\r\n Returns a loss dict with labelled training loss items tensor\r\n \"\"\"\r\n # Not needed for classification but necessary for segmentation & detection\r\n keys = [f\"{prefix}/{x}\" for x in self.loss_names]\r\n if loss_items is not None:\r\n loss_items = [round(float(x), 5) for x in loss_items] # convert tensors to 5 decimal place floats\r\n return dict(zip(keys, loss_items))\r\n else:\r\n return keys\r\n\r\n def progress_string(self):\r\n return ('\\n' + '%11s' *\r\n (4 + len(self.loss_names))) % ('Epoch', 'GPU_mem', *self.loss_names, 'Instances', 'Size')\r\n\r\n def plot_training_samples(self, batch, ni):\r\n plot_images(images=batch[\"img\"],\r\n batch_idx=batch[\"batch_idx\"],\r\n cls=batch[\"cls\"].squeeze(-1),\r\n bboxes=batch[\"bboxes\"],\r\n paths=batch[\"im_file\"],\r\n fname=self.save_dir / f\"train_batch{ni}.jpg\")\r\n\r\n def plot_metrics(self):\r\n plot_results(file=self.csv) # save results.png\r\n\r\n\r\n# Criterion class for computing training losses\r\nclass Loss:\r\n\r\n def __init__(self, model): # model must be de-paralleled\r\n\r\n device = next(model.parameters()).device # get model device\r\n h = model.args # hyperparameters\r\n\r\n m = model.model[-1] # Detect() module\r\n self.bce = nn.BCEWithLogitsLoss(reduction='none')\r\n self.hyp = h\r\n self.stride = m.stride # model strides\r\n self.nc = m.nc # number of classes\r\n self.no = m.no\r\n self.reg_max = m.reg_max\r\n self.device = device\r\n\r\n self.use_dfl = m.reg_max > 1\r\n self.assigner = TaskAlignedAssigner(topk=10, num_classes=self.nc, alpha=0.5, beta=6.0)\r\n self.bbox_loss = BboxLoss(m.reg_max - 1, use_dfl=self.use_dfl).to(device)\r\n self.proj = torch.arange(m.reg_max, dtype=torch.float, device=device)\r\n\r\n def preprocess(self, targets, batch_size, scale_tensor):\r\n if targets.shape[0] == 0:\r\n out = torch.zeros(batch_size, 0, 5, device=self.device)\r\n else:\r\n i = targets[:, 0] # image index\r\n _, counts = i.unique(return_counts=True)\r\n out = torch.zeros(batch_size, counts.max(), 5, device=self.device)\r\n for j in range(batch_size):\r\n matches = i == j\r\n n = matches.sum()\r\n if n:\r\n out[j, :n] = targets[matches, 1:]\r\n out[..., 1:5] = xywh2xyxy(out[..., 1:5].mul_(scale_tensor))\r\n return out\r\n\r\n def bbox_decode(self, anchor_points, pred_dist):\r\n if self.use_dfl:\r\n b, a, c = pred_dist.shape # batch, anchors, channels\r\n pred_dist = pred_dist.view(b, a, 4, c // 4).softmax(3).matmul(self.proj.type(pred_dist.dtype))\r\n # pred_dist = pred_dist.view(b, a, c // 4, 4).transpose(2,3).softmax(3).matmul(self.proj.type(pred_dist.dtype))\r\n # pred_dist = (pred_dist.view(b, a, c // 4, 4).softmax(2) * self.proj.type(pred_dist.dtype).view(1, 1, -1, 1)).sum(2)\r\n return dist2bbox(pred_dist, anchor_points, xywh=False)\r\n\r\n def __call__(self, preds, batch):\r\n loss = torch.zeros(3, device=self.device) # box, cls, dfl\r\n feats = preds[1] if isinstance(preds, tuple) else preds\r\n pred_distri, pred_scores = torch.cat([xi.view(feats[0].shape[0], self.no, -1) for xi in feats], 2).split(\r\n (self.reg_max * 4, self.nc), 1)\r\n\r\n pred_scores = pred_scores.permute(0, 2, 1).contiguous()\r\n pred_distri = pred_distri.permute(0, 2, 1).contiguous()\r\n\r\n dtype = pred_scores.dtype\r\n batch_size = pred_scores.shape[0]\r\n imgsz = torch.tensor(feats[0].shape[2:], device=self.device, dtype=dtype) * self.stride[0] # image size (h,w)\r\n anchor_points, stride_tensor = make_anchors(feats, self.stride, 0.5)\r\n\r\n # targets\r\n targets = torch.cat((batch[\"batch_idx\"].view(-1, 1), batch[\"cls\"].view(-1, 1), batch[\"bboxes\"]), 1)\r\n targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])\r\n gt_labels, gt_bboxes = targets.split((1, 4), 2) # cls, xyxy\r\n mask_gt = gt_bboxes.sum(2, keepdim=True).gt_(0)\r\n\r\n # pboxes\r\n pred_bboxes = self.bbox_decode(anchor_points, pred_distri) # xyxy, (b, h*w, 4)\r\n\r\n _, target_bboxes, target_scores, fg_mask, _ = self.assigner(\r\n pred_scores.detach().sigmoid(), (pred_bboxes.detach() * stride_tensor).type(gt_bboxes.dtype),\r\n anchor_points * stride_tensor, gt_labels, gt_bboxes, mask_gt)\r\n\r\n target_bboxes /= stride_tensor\r\n target_scores_sum = target_scores.sum()\r\n\r\n # cls loss\r\n # loss[1] = self.varifocal_loss(pred_scores, target_scores, target_labels) / target_scores_sum # VFL way\r\n loss[1] = self.bce(pred_scores, target_scores.to(dtype)).sum() / target_scores_sum # BCE\r\n\r\n # bbox loss\r\n if fg_mask.sum():\r\n loss[0], loss[2] = self.bbox_loss(pred_distri, pred_bboxes, anchor_points, target_bboxes, target_scores,\r\n target_scores_sum, fg_mask)\r\n\r\n loss[0] *= self.hyp.box # box gain\r\n loss[1] *= self.hyp.cls # cls gain\r\n loss[2] *= self.hyp.dfl # dfl gain\r\n\r\n return loss.sum() * batch_size, loss.detach() # loss(box, cls, dfl)\r\n\r\n\r\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent), config_name=DEFAULT_CONFIG.name)\r\ndef train(cfg):\r\n cfg.model = cfg.model or \"yolov8n.yaml\"\r\n cfg.data = cfg.data or \"coco128.yaml\" # or yolo.ClassificationDataset(\"mnist\")\r\n # trainer = DetectionTrainer(cfg)\r\n # trainer.train()\r\n from ultralytics import YOLO\r\n model = YOLO(cfg.model)\r\n model.train(**cfg)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n \"\"\"\r\n CLI usage:\r\n python ultralytics/yolo/v8/detect/train.py model=yolov8n.yaml data=coco128 epochs=100 imgsz=640\r\n\r\n TODO:\r\n yolo task=detect mode=train model=yolov8n.yaml data=coco128.yaml epochs=100\r\n \"\"\"\r\n train()\r\n"
},
{
"path": "yolo/v8/detect/val.py",
"content": "# Ultralytics YOLO π, GPL-3.0 license\r\n\r\nimport os\r\nfrom pathlib import Path\r\n\r\nimport hydra\r\nimport numpy as np\r\nimport torch\r\n\r\nfrom ultralytics.yolo.data import build_dataloader\r\nfrom ultralytics.yolo.data.dataloaders.v5loader import create_dataloader\r\nfrom ultralytics.yolo.engine.validator import BaseValidator\r\nfrom ultralytics.yolo.utils import DEFAULT_CONFIG, colorstr, ops, yaml_load\r\nfrom ultralytics.yolo.utils.checks import check_file, check_requirements\r\nfrom ultralytics.yolo.utils.metrics import ConfusionMatrix, DetMetrics, box_iou\r\nfrom ultralytics.yolo.utils.plotting import output_to_target, plot_images\r\nfrom ultralytics.yolo.utils.torch_utils import de_parallel\r\n\r\n\r\nclass DetectionValidator(BaseValidator):\r\n\r\n def __init__(self, dataloader=None, save_dir=None, pbar=None, logger=None, args=None):\r\n super().__init__(dataloader, save_dir, pbar, logger, args)\r\n self.data_dict = yaml_load(check_file(self.args.data), append_filename=True) if self.args.data else None\r\n self.is_coco = False\r\n self.class_map = None\r\n self.metrics = DetMetrics(save_dir=self.save_dir, plot=self.args.plots)\r\n self.iouv = torch.linspace(0.5, 0.95, 10) # iou vector for mAP@0.5:0.95\r\n self.niou = self.iouv.numel()\r\n\r\n def preprocess(self, batch):\r\n batch[\"img\"] = batch[\"img\"].to(self.device, non_blocking=True)\r\n batch[\"img\"] = (batch[\"img\"].half() if self.args.half else batch[\"img\"].float()) / 255\r\n for k in [\"batch_idx\", \"cls\", \"bboxes\"]:\r\n batch[k] = batch[k].to(self.device)\r\n\r\n nb, _, height, width = batch[\"img\"].shape\r\n batch[\"bboxes\"] *= torch.tensor((width, height, width, height), device=self.device) # to pixels\r\n self.lb = [torch.cat([batch[\"cls\"], batch[\"bboxes\"]], dim=-1)[batch[\"batch_idx\"] == i]\r\n for i in range(nb)] if self.args.save_hybrid else [] # for autolabelling\r\n\r\n return batch\r\n\r\n def init_metrics(self, model):\r\n head = model.model[-1] if self.training else model.model.model[-1]\r\n self.is_coco = self.data.get('val', '').endswith(f'coco{os.sep}val2017.txt') # is COCO dataset\r\n self.class_map = ops.coco80_to_coco91_class() if self.is_coco else list(range(1000))\r\n self.args.save_json |= self.is_coco and not self.training # run on final val if training COCO\r\n self.nc = head.nc\r\n self.names = model.names\r\n self.metrics.names = self.names\r\n self.confusion_matrix = ConfusionMatrix(nc=self.nc)\r\n self.seen = 0\r\n self.jdict = []\r\n self.stats = []\r\n\r\n def get_desc(self):\r\n return ('%22s' + '%11s' * 6) % ('Class', 'Images', 'Instances', 'Box(P', \"R\", \"mAP50\", \"mAP50-95)\")\r\n\r\n def postprocess(self, preds):\r\n preds = ops.non_max_suppression(preds,\r\n self.args.conf,\r\n self.args.iou,\r\n labels=self.lb,\r\n multi_label=True,\r\n agnostic=self.args.single_cls,\r\n max_det=self.args.max_det)\r\n return preds\r\n\r\n def update_metrics(self, preds, batch):\r\n # Metrics\r\n for si, pred in enumerate(preds):\r\n idx = batch[\"batch_idx\"] == si\r\n cls = batch[\"cls\"][idx]\r\n bbox = batch[\"bboxes\"][idx]\r\n nl, npr = cls.shape[0], pred.shape[0] # number of labels, predictions\r\n shape = batch[\"ori_shape\"][si]\r\n correct_bboxes = torch.zeros(npr, self.niou, dtype=torch.bool, device=self.device) # init\r\n self.seen += 1\r\n\r\n if npr == 0:\r\n if nl:\r\n self.stats.append((correct_bboxes, *torch.zeros((2, 0), device=self.device), cls.squeeze(-1)))\r\n if self.args.plots:\r\n self.confusion_matrix.process_batch(detections=None, labels=cls.squeeze(-1))\r\n continue\r\n\r\n # Predictions\r\n if self.args.single_cls:\r\n pred[:, 5] = 0\r\n predn = pred.clone()\r\n ops.scale_boxes(batch[\"img\"][si].shape[1:], predn[:, :4], shape,\r\n ratio_pad=batch[\"ratio_pad\"][si]) # native-space pred\r\n\r\n # Evaluate\r\n if nl:\r\n tbox = ops.xywh2xyxy(bbox) # target boxes\r\n ops.scale_boxes(batch[\"img\"][si].shape[1:], tbox, shape,\r\n ratio_pad=batch[\"ratio_pad\"][si]) # native-space labels\r\n labelsn = torch.cat((cls, tbox), 1) # native-space labels\r\n correct_bboxes = self._process_batch(predn, labelsn)\r\n # TODO: maybe remove these `self.` arguments as they already are member variable\r\n if self.args.plots:\r\n self.confusion_matrix.process_batch(predn, labelsn)\r\n self.stats.append((correct_bboxes, pred[:, 4], pred[:, 5], cls.squeeze(-1))) # (conf, pcls, tcls)\r\n\r\n # Save\r\n if self.args.save_json:\r\n self.pred_to_json(predn, batch[\"im_file\"][si])\r\n # if self.args.save_txt:\r\n # save_one_txt(predn, save_conf, shape, file=save_dir / 'labels' / f'{path.stem}.txt')\r\n\r\n def get_stats(self):\r\n stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*self.stats)] # to numpy\r\n if len(stats) and stats[0].any():\r\n self.metrics.process(*stats)\r\n self.nt_per_class = np.bincount(stats[-1].astype(int), minlength=self.nc) # number of targets per class\r\n return self.metrics.results_dict\r\n\r\n def print_results(self):\r\n pf = '%22s' + '%11i' * 2 + '%11.3g' * len(self.metrics.keys) # print format\r\n self.logger.info(pf % (\"all\", self.seen, self.nt_per_class.sum(), *self.metrics.mean_results()))\r\n if self.nt_per_class.sum() == 0:\r\n self.logger.warning(\r\n f'WARNING β οΈ no labels found in {self.args.task} set, can not compute metrics without labels')\r\n\r\n # Print results per class\r\n if (self.args.verbose or not self.training) and self.nc > 1 and len(self.stats):\r\n for i, c in enumerate(self.metrics.ap_class_index):\r\n self.logger.info(pf % (self.names[c], self.seen, self.nt_per_class[c], *self.metrics.class_result(i)))\r\n\r\n if self.args.plots:\r\n self.confusion_matrix.plot(save_dir=self.save_dir, names=list(self.names.values()))\r\n\r\n def _process_batch(self, detections, labels):\r\n \"\"\"\r\n Return correct prediction matrix\r\n Arguments:\r\n detections (array[N, 6]), x1, y1, x2, y2, conf, class\r\n labels (array[M, 5]), class, x1, y1, x2, y2\r\n Returns:\r\n correct (array[N, 10]), for 10 IoU levels\r\n \"\"\"\r\n iou = box_iou(labels[:, 1:], detections[:, :4])\r\n correct = np.zeros((detections.shape[0], self.iouv.shape[0])).astype(bool)\r\n correct_class = labels[:, 0:1] == detections[:, 5]\r\n for i in range(len(self.iouv)):\r\n x = torch.where((iou >= self.iouv[i]) & correct_class) # IoU > threshold and classes match\r\n if x[0].shape[0]:\r\n matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]),\r\n 1).cpu().numpy() # [label, detect, iou]\r\n if x[0].shape[0] > 1:\r\n matches = matches[matches[:, 2].argsort()[::-1]]\r\n matches = matches[np.unique(matches[:, 1], return_index=True)[1]]\r\n # matches = matches[matches[:, 2].argsort()[::-1]]\r\n matches = matches[np.unique(matches[:, 0], return_index=True)[1]]\r\n correct[matches[:, 1].astype(int), i] = True\r\n return torch.tensor(correct, dtype=torch.bool, device=detections.device)\r\n\r\n def get_dataloader(self, dataset_path, batch_size):\r\n # TODO: manage splits differently\r\n # calculate stride - check if model is initialized\r\n gs = max(int(de_parallel(self.model).stride if self.model else 0), 32)\r\n return create_dataloader(path=dataset_path,\r\n imgsz=self.args.imgsz,\r\n batch_size=batch_size,\r\n stride=gs,\r\n hyp=dict(self.args),\r\n cache=False,\r\n pad=0.5,\r\n rect=True,\r\n workers=self.args.workers,\r\n prefix=colorstr(f'{self.args.mode}: '),\r\n shuffle=False,\r\n seed=self.args.seed)[0] if self.args.v5loader else \\\r\n build_dataloader(self.args, batch_size, img_path=dataset_path, stride=gs, mode=\"val\")[0]\r\n\r\n def plot_val_samples(self, batch, ni):\r\n plot_images(batch[\"img\"],\r\n batch[\"batch_idx\"],\r\n batch[\"cls\"].squeeze(-1),\r\n batch[\"bboxes\"],\r\n paths=batch[\"im_file\"],\r\n fname=self.save_dir / f\"val_batch{ni}_labels.jpg\",\r\n names=self.names)\r\n\r\n def plot_predictions(self, batch, preds, ni):\r\n plot_images(batch[\"img\"],\r\n *output_to_target(preds, max_det=15),\r\n paths=batch[\"im_file\"],\r\n fname=self.save_dir / f'val_batch{ni}_pred.jpg',\r\n names=self.names) # pred\r\n\r\n def pred_to_json(self, predn, filename):\r\n stem = Path(filename).stem\r\n image_id = int(stem) if stem.isnumeric() else stem\r\n box = ops.xyxy2xywh(predn[:, :4]) # xywh\r\n box[:, :2] -= box[:, 2:] / 2 # xy center to top-left corner\r\n for p, b in zip(predn.tolist(), box.tolist()):\r\n self.jdict.append({\r\n 'image_id': image_id,\r\n 'category_id': self.class_map[int(p[5])],\r\n 'bbox': [round(x, 3) for x in b],\r\n 'score': round(p[4], 5)})\r\n\r\n def eval_json(self, stats):\r\n if self.args.save_json and self.is_coco and len(self.jdict):\r\n anno_json = self.data['path'] / \"annotations/instances_val2017.json\" # annotations\r\n pred_json = self.save_dir / \"predictions.json\" # predictions\r\n self.logger.info(f'\\nEvaluating pycocotools mAP using {pred_json} and {anno_json}...')\r\n try: # https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb\r\n check_requirements('pycocotools>=2.0.6')\r\n from pycocotools.coco import COCO # noqa\r\n from pycocotools.cocoeval import COCOeval # noqa\r\n\r\n for x in anno_json, pred_json:\r\n assert x.is_file(), f\"{x} file not found\"\r\n anno = COCO(str(anno_json)) # init annotations api\r\n pred = anno.loadRes(str(pred_json)) # init predictions api (must pass string, not Path)\r\n eval = COCOeval(anno, pred, 'bbox')\r\n if self.is_coco:\r\n eval.params.imgIds = [int(Path(x).stem) for x in self.dataloader.dataset.im_files] # images to eval\r\n eval.evaluate()\r\n eval.accumulate()\r\n eval.summarize()\r\n stats[self.metrics.keys[-1]], stats[self.metrics.keys[-2]] = eval.stats[:2] # update mAP50-95 and mAP50\r\n except Exception as e:\r\n self.logger.warning(f'pycocotools unable to run: {e}')\r\n return stats\r\n\r\n\r\n@hydra.main(version_base=None, config_path=str(DEFAULT_CONFIG.parent), config_name=DEFAULT_CONFIG.name)\r\ndef val(cfg):\r\n cfg.data = cfg.data or \"coco128.yaml\"\r\n validator = DetectionValidator(args=cfg)\r\n validator(model=cfg.model)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n val()\r\n"
}
]