[
{
"path": ".gitignore",
"content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# launch bash\n*.sh\n# nsight system report files\n*.nsys-rep\n*.sqlite\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don't work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n"
},
{
"path": "LICENSE",
"content": " MIT License\n\n Copyright (c) Microsoft Corporation.\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE\n"
},
{
"path": "MODELHUB.md",
"content": "Access code for `baidu` is `swin`.\n\n## ImageNet-1K and ImageNet-22K Pretrained Swin-V1 Models\n\n| name | pretrain | resolution |acc@1 | acc@5 | #params | FLOPs | FPS| 22K model | 1K model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---: |:---: |\n| Swin-T | ImageNet-1K | 224x224 | 81.2 | 95.5 | 28M | 4.5G | 755 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/156nWJy4Q28rDlrX-rRbI3w)/[config](configs/swin/swin_tiny_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745562/log_swin_tiny_patch4_window7_224.txt) |\n| Swin-S | ImageNet-1K | 224x224 | 83.2 | 96.2 | 50M | 8.7G | 437 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/1KFjpj3Efey3LmtE1QqPeQg)/[config](configs/swin/swin_small_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745563/log_swin_small_patch4_window7_224.txt) |\n| Swin-B | ImageNet-1K | 224x224 | 83.5 | 96.5 | 88M | 15.4G | 278 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/16bqCTEc70nC_isSsgBSaqQ)/[config](configs/swin/swin_base_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745564/log_swin_base_patch4_window7_224.txt) |\n| Swin-B | ImageNet-1K | 384x384 | 84.5 | 97.0 | 88M | 47.1G | 85 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pth)/[baidu](https://pan.baidu.com/s/1xT1cu740-ejW7htUdVLnmw)/[config](configs/swin/swin_base_patch4_window12_384_finetune.yaml) |\n| Swin-T | ImageNet-22K | 224x224 | 80.9 | 96.0 | 28M | 4.5G | 755 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1vct0VYwwQQ8PYkBjwSSBZQ?pwd=swin)/[config](configs/swin/swin_tiny_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pth)/[baidu](https://pan.baidu.com/s/1K0OO-nGZDPkR8fm_r83e8Q?pwd=swin)/[config](configs/swin/swin_tiny_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-S | ImageNet-22K | 224x224 | 83.2 | 97.0 | 50M | 8.7G | 437 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/11NC1xdT5BAGBgazdTme5Sg?pwd=swin)/[config](configs/swin/swin_small_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth)/[baidu](https://pan.baidu.com/s/10RFVfjQJhwPfeHrmxQUaLw?pwd=swin)/[config](configs/swin/swin_small_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-B | ImageNet-22K | 224x224 | 85.2 | 97.5 | 88M | 15.4G | 278 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1y1Ec3UlrKSI8IMtEs-oBXA)/[config](configs/swin/swin_base_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1n_wNkcbRxVXit8r_KrfAVg)/[config](configs/swin/swin_base_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-B | ImageNet-22K | 384x384 | 86.4 | 98.0 | 88M | 47.1G | 85 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth)/[baidu](https://pan.baidu.com/s/1vwJxnJcVqcLZAw9HaqiR6g) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1caKTSdoLJYoi4WBcnmWuWg)/[config](configs/swin/swin_base_patch4_window12_384_22kto1k_finetune.yaml) |\n| Swin-L | ImageNet-22K | 224x224 | 86.3 | 97.9 | 197M | 34.5G | 141 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1pws3rOTFuOebBYP3h6Kx8w)/[config](configs/swin/swin_large_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1NkQApMWUhxBGjk1ne6VqBQ)/[config](configs/swin/swin_large_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-L | ImageNet-22K | 384x384 | 87.3 | 98.2 | 197M | 103.9G | 42 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pth)/[baidu](https://pan.baidu.com/s/1sl7o_bJA143OD7UqSLAMoA) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1X0FLHQyPOC6Kmv2CmgxJvA)/[config](configs/swin/swin_large_patch4_window12_384_22kto1k_finetune.yaml) |\n\n## ImageNet-1K and ImageNet-22K Pretrained Swin-V2 Models\n\n| name | pretrain | resolution | window |acc@1 | acc@5 | #params | FLOPs | FPS |22K model | 1K model |\n|:---------------------:| :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---:|:---: |:---: |\n| SwinV2-T | ImageNet-1K | 256x256 | 8x8 | 81.8 | 95.9 | 28M | 5.9G | 572 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1RzLkAH_5OtfRCJe6Vlg6rg?pwd=swin)/[config](configs/swinv2/swinv2_tiny_patch4_window8_256.yaml) |\n| SwinV2-S | ImageNet-1K | 256x256 | 8x8 | 83.7 | 96.6 | 50M | 11.5G | 327 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/195PdA41szEduW3jEtRSa4Q?pwd=swin)/[config](configs/swinv2/swinv2_small_patch4_window8_256.yaml) |\n| SwinV2-B | ImageNet-1K | 256x256 | 8x8 | 84.2 | 96.9 | 88M | 20.3G | 217 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/18AfMSz3dPyzIvP1dKuERvQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window8_256.yaml) |\n| SwinV2-T | ImageNet-1K | 256x256 | 16x16 | 82.8 | 96.2 | 28M | 6.6G | 437 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1dyK3cK9Xipmv6RnTtrPocw?pwd=swin)/[config](configs/swinv2/swinv2_tiny_patch4_window16_256.yaml) |\n| SwinV2-S | ImageNet-1K | 256x256 | 16x16 | 84.1 | 96.8 | 50M | 12.6G | 257 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1ZIPiSfWNKTPp821Ka-Mifw?pwd=swin)/[config](configs/swinv2/swinv2_small_patch4_window16_256.yaml) |\n| SwinV2-B | ImageNet-1K | 256x256 | 16x16 | 84.6 | 97.0 | 88M | 21.8G | 174 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1dlDQGn8BXCmnh7wQSM5Nhw?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window16_256.yaml) |\n| SwinV2-B\\* | ImageNet-22K | 256x256 | 16x16 | 86.2 | 97.9 | 88M | 21.8G | 174 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/1Xc2rsSsRQz_sy5mjgfxrMQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/1sgstld4MgGsZxhUAW7MlmQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.yaml) |\n| SwinV2-B\\* | ImageNet-22K | 384x384 | 24x24 | 87.1 | 98.2 | 88M | 54.7G | 57 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/1Xc2rsSsRQz_sy5mjgfxrMQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/17u3sEQaUYlvfL195rrORzQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.yaml) |\n| SwinV2-L\\* | ImageNet-22K | 256x256 | 16x16 | 86.9 | 98.0 | 197M | 47.5G | 95 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/11PhCV7qAGXtZ8dXNgyiGOw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/1pqp31N80qIWjFPbudzB6Bw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.yaml) |\n| SwinV2-L\\* | ImageNet-22K | 384x384 | 24x24 | 87.6 | 98.3 | 197M | 115.4G | 33 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/11PhCV7qAGXtZ8dXNgyiGOw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/13URdNkygr3Xn0N3e6IwjgA?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.yaml) |\n\nNote:\n\n- SwinV2-B\\* (SwinV2-L\\*) with input resolution of 256x256 and 384x384 both fine-tuned from the\n same pre-training model using a smaller input resolution of 192x192.\n- SwinV2-B\\* (384x384) achieves 78.08 acc@1 on ImageNet-1K-V2 while SwinV2-L\\* (384x384) achieves\n 78.31.\n\n## ImageNet-1K Pretrained Swin MLP Models\n\n| name | pretrain | resolution |acc@1 | acc@5 | #params | FLOPs | FPS | 1K model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| [Mixer-B/16](https://arxiv.org/pdf/2105.01601.pdf) | ImageNet-1K | 224x224 | 76.4 | - | 59M | 12.7G | - | [official repo](https://github.com/google-research/vision_transformer) |\n| [ResMLP-S24](https://arxiv.org/abs/2105.03404) | ImageNet-1K | 224x224 | 79.4 | - | 30M | 6.0G | 715 | [timm](https://github.com/rwightman/pytorch-image-models) |\n| [ResMLP-B24](https://arxiv.org/abs/2105.03404) | ImageNet-1K | 224x224 | 81.0 | - | 116M | 23.0G | 231 | [timm](https://github.com/rwightman/pytorch-image-models) |\n| Swin-T/C24 | ImageNet-1K | 256x256 | 81.6 | 95.7 | 28M | 5.9G | 563 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_tiny_c24_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/17k-7l6Sxt7uZ7IV0f26GNQ)/[config](configs/swin/swin_tiny_c24_patch4_window8_256.yaml) |\n| SwinMLP-T/C24 | ImageNet-1K | 256x256 | 79.4 | 94.6 | 20M | 4.0G | 807 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c24_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1Sa4vP5R0M2RjfIe9HIga-Q)/[config](configs/swin/swin_mlp_tiny_c24_patch4_window8_256.yaml) |\n| SwinMLP-T/C12 | ImageNet-1K | 256x256 | 79.6 | 94.7 | 21M | 4.0G | 792 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c12_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1mM9J2_DEVZHUB5ASIpFl0w)/[config](configs/swin/swin_mlp_tiny_c12_patch4_window8_256.yaml) |\n| SwinMLP-T/C6 | ImageNet-1K | 256x256 | 79.7 | 94.9 | 23M | 4.0G | 766 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c6_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1hUTYVT2W1CsjICw-3W-Vjg)/[config](configs/swin/swin_mlp_tiny_c6_patch4_window8_256.yaml) |\n| SwinMLP-B | ImageNet-1K | 224x224 | 81.3 | 95.3 | 61M | 10.4G | 409 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_base_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/1zww3dnbX3GxNiGfb-GwyUg)/[config](configs/swin/swin_mlp_base_patch4_window7_224.yaml) |\n\nNote: C24 means each head has 24 channels.\n\n## ImageNet-22K Pretrained Swin-MoE Models\n\n| name | #experts | k | router | resolution | window | IN-22K acc@1 | IN-1K/ft acc@1 | IN-1K/5-shot acc@1 | 22K model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Swin-MoE-S | 1 (dense) | - | - | 192x192 | 8x8 | 35.5| 83.5 | 70.3 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_densebaseline_22k.zip)/[baidu](https://pan.baidu.com/s/1O1m9jT2pGoago_RiRX914w?pwd=swin)/[config](configs/swinmoe/swin_moe_small_patch4_window12_192_densebaseline_22k.yaml) |\n| Swin-MoE-S | 8 | 1 | Linear | 192x192 | 8x8 | 36.8 | 84.5 | 75.2 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_8expert_32gpu_22k.zip)/[baidu](https://pan.baidu.com/s/198IlYUrWOxEUp7wNdoJT5Q?pwd=swin)/[config](configs/swinmoe/swin_moe_small_patch4_window12_192_8expert_32gpu_22k.yaml) |\n| Swin-MoE-S | 16 | 1 | Linear |192x192 | 8x8 | 37.6 | 84.9 | 76.5 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_16expert_32gpu_22k.zip)/[baidu](https://pan.baidu.com/s/1vRQweedtT42VwMTqe9-r2A?pwd=swin)/[config](configs/swinmoe/swin_moe_small_patch4_window12_192_16expert_32gpu_22k.yaml) |\n| Swin-MoE-S | 32 | 1 | Linear | 192x192 | 8x8 | 37.4 | 84.7 | 75.9 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.zip)/[baidu](https://pan.baidu.com/s/1i7rImt5pwO8gJC-PRRuZwQ?pwd=swin)/[config](configs/swinmoe/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.yaml) |\n| Swin-MoE-S | 32 | 1 | Cosine | 192x192 | 8x8 | 37.2 | 84.3 | 75.2 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_cosine_router_32expert_32gpu_22k.zip)/[baidu](https://pan.baidu.com/s/1Yghr_12ntSrv01I9yatPDQ?pwd=swin)/[config](configs/swinmoe/swin_moe_small_patch4_window12_192_cosine_router_32expert_32gpu_22k.yaml) |\n| Swin-MoE-S | 64 | 1 | Linear | 192x192 | 8x8 | 37.8 | 84.7 | 75.7 | - |\n| Swin-MoE-S | 128 | 1 | Linear | 192x192 | 8x8 | 37.4 | 84.5 | 75.4 | - |\n| Swin-MoE-B | 1 (dense) | - | - | 192x192 | 8x8 | 37.3 | 85.1 | 75.9 | [config](configs/swinmoe/swin_moe_base_patch4_window12_192_densebaseline_22k.yaml) |\n| Swin-MoE-B | 8 | 1 | Linear | 192x192 | 8x8 | 38.1 | 85.3 | 77.2 | [config](configs/swinmoe/swin_moe_base_patch4_window12_192_8expert_32gpu_22k.yaml) |\n| Swin-MoE-B | 16 | 1 | Linear | 192x192 | 8x8 | 38.7 | 85.5 | 78.2 | [config](configs/swinmoe/swin_moe_base_patch4_window12_192_16expert_32gpu_22k.yaml) |\n| Swin-MoE-B | 32 | 1 | Linear | 192x192 | 8x8 | 38.6 | 85.5 | 77.9 | [config](configs/swinmoe/swin_moe_base_patch4_window12_192_32expert_32gpu_22k.yaml) |\n| Swin-MoE-B | 32 | 1 | Cosine | 192x192 | 8x8 | 38.5 | 85.3 | 77.3 | [config](configs/swinmoe/swin_moe_base_patch4_window12_192_cosine_router_32expert_32gpu_22k.yaml) |\n| Swin-MoE-B | 32 | 2 | Linear | 192x192 | 8x8 | 38.6 | 85.5 | 78.7 | - |\n\n## SimMIM Pretrained Swin-V2 Models\n\n> Please note that all SimMIM pretrained Swin-V2 models will be stored in the Huggingface repository starting July 2024. For more details, refer to the [huggingface repository](https://huggingface.co/zdaxie/SimMIM).\n\n- **Model size** only includes the backbone weights and excludes weights in the decoders/classification heads.\n- **Batch size** for all models is set to 2048.\n- **Validation loss** is calculated on the ImageNet-1K validation set.\n- **Fine-tuned acc@1** refers to the top-1 accuracy on the ImageNet-1K validation set after fine-tuning.\n\n| name | model size | pre-train dataset | pre-train iterations | validation loss | fine-tuned acc@1 | pre-trained model | fine-tuned model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| SwinV2-Small | 49M | ImageNet-1K 10% | 125k | 0.4820 | 82.69 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper10_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper10_125k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 10% | 250k | 0.4961 | 83.11 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper10_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper10_250k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 10% | 500k | 0.5115 | 83.17 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper10_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper10_500k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 20% | 125k | 0.4751 | 83.05 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper20_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper20_125k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 20% | 250k | 0.4722 | 83.56 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper20_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper20_250k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 20% | 500k | 0.4734 | 83.75 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper20_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper20_500k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 50% | 125k | 0.4732 | 83.04 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper50_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper50_125k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 50% | 250k | 0.4681 | 83.67 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper50_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper50_250k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K 50% | 500k | 0.4646 | 83.96 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1kper50_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1kper50_500k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K | 125k | 0.4728 | 82.92 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1k_125k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K | 250k | 0.4674 | 83.66 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1k_250k.pth?download=true) |\n| SwinV2-Small | 49M | ImageNet-1K | 500k | 0.4641 | 84.08 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_small_1k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_small_1k_500k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 10% | 125k | 0.4822 | 83.33 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper10_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper10_125k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 10% | 250k | 0.4997 | 83.60 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper10_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper10_250k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 10% | 500k | 0.5112 | 83.41 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper10_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper10_500k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 20% | 125k | 0.4703 | 83.86 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper20_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper20_125k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 20% | 250k | 0.4679 | 84.37 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper20_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper20_250k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 20% | 500k | 0.4711 | 84.61 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper20_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper20_500k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 50% | 125k | 0.4683 | 84.04 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper50_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper50_125k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 50% | 250k | 0.4633 | 84.57 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper50_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper50_250k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K 50% | 500k | 0.4598 | 84.95 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1kper50_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1kper50_500k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K | 125k | 0.4680 | 84.13 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1k_125k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K | 250k | 0.4626 | 84.65 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1k_250k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-1K | 500k | 0.4588 | 85.04 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_1k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_1k_500k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-22K | 125k | 0.4695 | 84.11 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_22k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_22k_125k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-22K | 250k | 0.4649 | 84.57 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_22k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_22k_250k.pth?download=true) |\n| SwinV2-Base | 87M | ImageNet-22K | 500k | 0.4614 | 85.11 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_base_22k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_base_22k_500k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 10% | 125k | 0.4995 | 83.69 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper10_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper10_125k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 10% | 250k | 0.5140 | 83.66 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper10_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper10_250k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 10% | 500k | 0.5150 | 83.50 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper10_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper10_500k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 20% | 125k | 0.4675 | 84.38 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper20_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper20_125k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 20% | 250k | 0.4746 | 84.71 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper20_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper20_250k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 20% | 500k | 0.4960 | 84.59 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper20_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper20_500k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 50% | 125k | 0.4622 | 84.78 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper50_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper50_125k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 50% | 250k | 0.4566 | 85.38 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper50_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper50_250k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K 50% | 500k | 0.4530 | 85.80 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1kper50_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1kper50_500k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K | 125k | 0.4611 | 84.98 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1k_125k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K | 250k | 0.4552 | 85.45 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1k_250k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-1K | 500k | 0.4507 | 85.91 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_1k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_1k_500k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-22K | 125k | 0.4649 | 84.61 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_22k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_22k_125k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-22K | 250k | 0.4586 | 85.39 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_22k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_22k_250k.pth?download=true) |\n| SwinV2-Large | 195M | ImageNet-22K | 500k | 0.4536 | 85.81 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_large_22k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_large_22k_500k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 20% | 125k | 0.4789 | 84.35 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper20_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper20_125k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 20% | 250k | 0.5038 | 84.16 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper20_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper20_250k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 20% | 500k | 0.5071 | 83.44 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper20_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper20_500k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 50% | 125k | 0.4549 | 85.09 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper50_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper50_125k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 50% | 250k | 0.4511 | 85.64 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper50_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper50_250k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K 50% | 500k | 0.4559 | 85.69 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1kper50_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1kper50_500k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K | 125k | 0.4531 | 85.23 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1k_125k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K | 250k | 0.4464 | 85.90 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1k_250k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-1K | 500k | 0.4416 | 86.34 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_1k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_1k_500k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-22K | 125k | 0.4564 | 85.14 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_22k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_22k_125k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-22K | 250k | 0.4499 | 85.86 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_22k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_22k_250k.pth?download=true) |\n| SwinV2-Huge | 655M | ImageNet-22K | 500k | 0.4444 | 86.27 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_huge_22k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_huge_22k_500k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K 50% | 125k | 0.4534 | 85.44 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1kper50_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1kper50_125k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K 50% | 250k | 0.4515 | 85.76 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1kper50_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1kper50_250k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K 50% | 500k | 0.4719 | 85.51 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1kper50_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1kper50_500k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K | 125k | 0.4513 | 85.57 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1k_125k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K | 250k | 0.4442 | 86.12 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1k_250k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-1K | 500k | 0.4395 | 86.46 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_1k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_1k_500k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-22K | 125k | 0.4544 | 85.39 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_22k_125k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_22k_125k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-22K | 250k | 0.4475 | 85.96 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_22k_250k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_22k_250k.pth?download=true) |\n| SwinV2-giant | 1.06B | ImageNet-22K | 500k | 0.4416 | 86.53 | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_pretrain_models/swinv2_giant_22k_500k.pth?download=true) | [huggingface](https://huggingface.co/zdaxie/SimMIM/resolve/main/simmim_swinv2_finetune_models/finetune_swinv2_giant_22k_500k.pth?download=true) |\n\n## SimMIM Pretrained Swin-V1 Models\n\n**ImageNet-1K Pre-trained and Fine-tuned Models**\n\n| name | pre-train epochs | pre-train resolution | fine-tune resolution | acc@1 | pre-trained model | fine-tuned model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Swin-Base | 100 | 192x192 | 192x192 | 82.8 | [google](https://drive.google.com/file/d/1Wcbr66JL26FF30Kip9fZa_0lXrDAKP-d/view?usp=sharing)/[config](configs/swin_base__100ep/simmim_pretrain__swin_base__img192_window6__100ep.yaml) | [google](https://drive.google.com/file/d/1RsgHfjB4B1ZYblXEQVT-FPX3WSvBrxcs/view?usp=sharing)/[config](configs/swin_base__100ep/simmim_finetune__swin_base__img192_window6__100ep.yaml) |\n| Swin-Base | 100 | 192x192 | 224x224 | 83.5 | [google](https://drive.google.com/file/d/1Wcbr66JL26FF30Kip9fZa_0lXrDAKP-d/view?usp=sharing)/[config](configs/swin_base__100ep/simmim_pretrain__swin_base__img192_window6__100ep.yaml) | [google](https://drive.google.com/file/d/1mb43BkW56F5smwiX-g7QUUD7f1Rftq8u/view?usp=sharing)/[config](configs/swin_base__100ep/simmim_finetune__swin_base__img224_window7__100ep.yaml) |\n| Swin-Base | 800 | 192x192 | 224x224 | 84.0 | [google](https://drive.google.com/file/d/15zENvGjHlM71uKQ3d2FbljWPubtrPtjl/view?usp=sharing)/[config](configs/swin_base__800ep/simmim_pretrain__swin_base__img192_window6__800ep.yaml) | [google](https://drive.google.com/file/d/1xEKyfMTsdh6TfnYhk5vbw0Yz7a-viZ0w/view?usp=sharing)/[config](configs/swin_base__800ep/simmim_finetune__swin_base__img224_window7__800ep.yaml) |\n| Swin-Large | 800 | 192x192 | 224x224 | 85.4 | [google](https://drive.google.com/file/d/1qDxrTl2YUDB0505_4QrU5LU2R1kKmcBP/view?usp=sharing)/[config](configs/swin_large__800ep/simmim_pretrain__swin_large__img192_window12__800ep.yaml) | [google](https://drive.google.com/file/d/1mf0ZpXttEvFsH87Www4oQ-t8Kwr0x485/view?usp=sharing)/[config](configs/swin_large__800ep/simmim_finetune__swin_large__img224_window14__800ep.yaml) |\n| SwinV2-Huge | 800 | 192x192 | 224x224 | 85.7 | / | / |\n| SwinV2-Huge | 800 | 192x192 | 512x512 | 87.1 | / | / |\n"
},
{
"path": "README.md",
"content": "# Swin Transformer\n\n[](https://paperswithcode.com/sota/object-detection-on-coco?p=swin-transformer-v2-scaling-up-capacity-and)\n[](https://paperswithcode.com/sota/instance-segmentation-on-coco?p=swin-transformer-v2-scaling-up-capacity-and)\n[](https://paperswithcode.com/sota/semantic-segmentation-on-ade20k?p=swin-transformer-v2-scaling-up-capacity-and)\n[](https://paperswithcode.com/sota/action-classification-on-kinetics-400?p=swin-transformer-v2-scaling-up-capacity-and)\n\nThis repo is the official implementation of [\"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows\"](https://arxiv.org/pdf/2103.14030.pdf) as well as the follow-ups. It currently includes code and models for the following tasks:\n\n> **Image Classification**: Included in this repo. See [get_started.md](get_started.md) for a quick start.\n\n> **Object Detection and Instance Segmentation**: See [Swin Transformer for Object Detection](https://github.com/SwinTransformer/Swin-Transformer-Object-Detection).\n\n> **Semantic Segmentation**: See [Swin Transformer for Semantic Segmentation](https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation).\n\n> **Video Action Recognition**: See [Video Swin Transformer](https://github.com/SwinTransformer/Video-Swin-Transformer).\n\n> **Semi-Supervised Object Detection**: See [Soft Teacher](https://github.com/microsoft/SoftTeacher).\n\n> **SSL: Contrasitive Learning**: See [Transformer-SSL](https://github.com/SwinTransformer/Transformer-SSL).\n\n> **SSL: Masked Image Modeling**: See [get_started.md#simmim-support](https://github.com/microsoft/Swin-Transformer/blob/main/get_started.md#simmim-support).\n\n> **Mixture-of-Experts**: See [get_started](get_started.md#mixture-of-experts-support) for more instructions.\n\n> **Feature-Distillation**: See [Feature-Distillation](https://github.com/SwinTransformer/Feature-Distillation).\n\n## Updates\n\n***12/29/2022***\n\n1. **Nvidia**'s [FasterTransformer](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/swin_guide.md) now supports Swin Transformer V2 inference, which have significant speed improvements on `T4 and A100 GPUs`.\n\n***11/30/2022***\n\n1. Models and codes of **Feature Distillation** are released. Please refer to [Feature-Distillation](https://github.com/SwinTransformer/Feature-Distillation) for details, and the checkpoints (FD-EsViT-Swin-B, FD-DeiT-ViT-B, FD-DINO-ViT-B, FD-CLIP-ViT-B, FD-CLIP-ViT-L).\n\n***09/24/2022***\n\n1. Merged [SimMIM](https://github.com/microsoft/SimMIM), which is a **Masked Image Modeling** based pre-training approach applicable to Swin and SwinV2 (and also applicable for ViT and ResNet). Please refer to [get started with SimMIM](get_started.md#simmim-support) to play with SimMIM pre-training.\n\n2. Released a series of Swin and SwinV2 models pre-trained using the SimMIM approach (see [MODELHUB for SimMIM](MODELHUB.md#simmim-pretrained-swin-v2-models)), with model size ranging from SwinV2-Small-50M to SwinV2-giant-1B, data size ranging from ImageNet-1K-10% to ImageNet-22K, and iterations from 125k to 500k. You may leverage these models to study the properties of MIM methods. Please look into the [data scaling](https://arxiv.org/abs/2206.04664) paper for more details.\n\n***07/09/2022***\n\n`News`: \n\n1. SwinV2-G achieves `61.4 mIoU` on ADE20K semantic segmentation (+1.5 mIoU over the previous SwinV2-G model), using an additional [feature distillation (FD)](https://github.com/SwinTransformer/Feature-Distillation) approach, **setting a new recrod** on this benchmark. FD is an approach that can generally improve the fine-tuning performance of various pre-trained models, including DeiT, DINO, and CLIP. Particularly, it improves CLIP pre-trained ViT-L by +1.6% to reach `89.0%` on ImageNet-1K image classification, which is **the most accurate ViT-L model**.\n2. Merged a PR from **Nvidia** that links to faster Swin Transformer inference that have significant speed improvements on `T4 and A100 GPUs`.\n3. Merged a PR from **Nvidia** that enables an option to use `pure FP16 (Apex O2)` in training, while almost maintaining the accuracy.\n\n***06/03/2022***\n\n1. Added **Swin-MoE**, the Mixture-of-Experts variant of Swin Transformer implemented using [Tutel](https://github.com/microsoft/tutel) (an optimized Mixture-of-Experts implementation). **Swin-MoE** is introduced in the [TuTel](https://arxiv.org/abs/2206.03382) paper.\n\n***05/12/2022***\n\n1. Pretrained models of [Swin Transformer V2](https://arxiv.org/abs/2111.09883) on ImageNet-1K and ImageNet-22K are released. \n2. ImageNet-22K pretrained models for Swin-V1-Tiny and Swin-V2-Small are released.\n\n***03/02/2022***\n\n1. Swin Transformer V2 and SimMIM got accepted by CVPR 2022. [SimMIM](https://github.com/microsoft/SimMIM) is a self-supervised pre-training approach based on masked image modeling, a key technique that works out the 3-billion-parameter Swin V2 model using `40x less labelled data` than that of previous billion-scale models based on JFT-3B. \n\n***02/09/2022***\n\n1. Integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo [](https://huggingface.co/spaces/akhaliq/Swin-Transformer)\n\n***10/12/2021***\n\n1. Swin Transformer received ICCV 2021 best paper award (Marr Prize).\n\n***08/09/2021***\n1. [Soft Teacher](https://arxiv.org/pdf/2106.09018v2.pdf) will appear at ICCV2021. The code will be released at [GitHub Repo](https://github.com/microsoft/SoftTeacher). `Soft Teacher` is an end-to-end semi-supervisd object detection method, achieving a new record on the COCO test-dev: `61.3 box AP` and `53.0 mask AP`.\n \n***07/03/2021***\n1. Add **Swin MLP**, which is an adaption of `Swin Transformer` by replacing all multi-head self-attention (MHSA) blocks by MLP layers (more precisely it is a group linear layer). The shifted window configuration can also significantly improve the performance of vanilla MLP architectures. \n\n***06/25/2021***\n1. [Video Swin Transformer](https://arxiv.org/abs/2106.13230) is released at [Video-Swin-Transformer](https://github.com/SwinTransformer/Video-Swin-Transformer).\n`Video Swin Transformer` achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including action recognition (`84.9` top-1 accuracy on Kinetics-400 and `86.1` top-1 accuracy on Kinetics-600 with `~20x` less pre-training data and `~3x` smaller model size) and temporal modeling (`69.6` top-1 accuracy on Something-Something v2).\n\n***05/12/2021***\n1. Used as a backbone for `Self-Supervised Learning`: [Transformer-SSL](https://github.com/SwinTransformer/Transformer-SSL)\n\nUsing Swin-Transformer as the backbone for self-supervised learning enables us to evaluate the transferring performance of the learnt representations on down-stream tasks, which is missing in previous works due to the use of ViT/DeiT, which has not been well tamed for down-stream tasks.\n\n***04/12/2021***\n\nInitial commits:\n\n1. Pretrained models on ImageNet-1K ([Swin-T-IN1K](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth), [Swin-S-IN1K](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth), [Swin-B-IN1K](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth)) and ImageNet-22K ([Swin-B-IN22K](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth), [Swin-L-IN22K](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth)) are provided.\n2. The supported code and models for ImageNet-1K image classification, COCO object detection and ADE20K semantic segmentation are provided.\n3. The cuda kernel implementation for the [local relation layer](https://arxiv.org/pdf/1904.11491.pdf) is provided in branch [LR-Net](https://github.com/microsoft/Swin-Transformer/tree/LR-Net).\n\n## Introduction\n\n**Swin Transformer** (the name `Swin` stands for **S**hifted **win**dow) is initially described in [arxiv](https://arxiv.org/abs/2103.14030), which capably serves as a\ngeneral-purpose backbone for computer vision. It is basically a hierarchical Transformer whose representation is\ncomputed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention\ncomputation to non-overlapping local windows while also allowing for cross-window connection.\n\nSwin Transformer achieves strong performance on COCO object detection (`58.7 box AP` and `51.1 mask AP` on test-dev) and\nADE20K semantic segmentation (`53.5 mIoU` on val), surpassing previous models by a large margin.\n\n\n\n## Main Results on ImageNet with Pretrained Models\n\n**ImageNet-1K and ImageNet-22K Pretrained Swin-V1 Models**\n\n| name | pretrain | resolution |acc@1 | acc@5 | #params | FLOPs | FPS| 22K model | 1K model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---: |:---: |\n| Swin-T | ImageNet-1K | 224x224 | 81.2 | 95.5 | 28M | 4.5G | 755 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/156nWJy4Q28rDlrX-rRbI3w)/[config](configs/swin/swin_tiny_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745562/log_swin_tiny_patch4_window7_224.txt) |\n| Swin-S | ImageNet-1K | 224x224 | 83.2 | 96.2 | 50M | 8.7G | 437 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/1KFjpj3Efey3LmtE1QqPeQg)/[config](configs/swin/swin_small_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745563/log_swin_small_patch4_window7_224.txt) |\n| Swin-B | ImageNet-1K | 224x224 | 83.5 | 96.5 | 88M | 15.4G | 278 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/16bqCTEc70nC_isSsgBSaqQ)/[config](configs/swin/swin_base_patch4_window7_224.yaml)/[log](https://github.com/SwinTransformer/storage/files/7745564/log_swin_base_patch4_window7_224.txt) |\n| Swin-B | ImageNet-1K | 384x384 | 84.5 | 97.0 | 88M | 47.1G | 85 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pth)/[baidu](https://pan.baidu.com/s/1xT1cu740-ejW7htUdVLnmw)/[config](configs/swin/swin_base_patch4_window12_384_finetune.yaml) |\n| Swin-T | ImageNet-22K | 224x224 | 80.9 | 96.0 | 28M | 4.5G | 755 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1vct0VYwwQQ8PYkBjwSSBZQ?pwd=swin)/[config](configs/swin/swin_tiny_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pth)/[baidu](https://pan.baidu.com/s/1K0OO-nGZDPkR8fm_r83e8Q?pwd=swin)/[config](configs/swin/swin_tiny_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-S | ImageNet-22K | 224x224 | 83.2 | 97.0 | 50M | 8.7G | 437 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/11NC1xdT5BAGBgazdTme5Sg?pwd=swin)/[config](configs/swin/swin_small_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth)/[baidu](https://pan.baidu.com/s/10RFVfjQJhwPfeHrmxQUaLw?pwd=swin)/[config](configs/swin/swin_small_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-B | ImageNet-22K | 224x224 | 85.2 | 97.5 | 88M | 15.4G | 278 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1y1Ec3UlrKSI8IMtEs-oBXA)/[config](configs/swin/swin_base_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1n_wNkcbRxVXit8r_KrfAVg)/[config](configs/swin/swin_base_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-B | ImageNet-22K | 384x384 | 86.4 | 98.0 | 88M | 47.1G | 85 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth)/[baidu](https://pan.baidu.com/s/1vwJxnJcVqcLZAw9HaqiR6g) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1caKTSdoLJYoi4WBcnmWuWg)/[config](configs/swin/swin_base_patch4_window12_384_22kto1k_finetune.yaml) |\n| Swin-L | ImageNet-22K | 224x224 | 86.3 | 97.9 | 197M | 34.5G | 141 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth)/[baidu](https://pan.baidu.com/s/1pws3rOTFuOebBYP3h6Kx8w)/[config](configs/swin/swin_large_patch4_window7_224_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1NkQApMWUhxBGjk1ne6VqBQ)/[config](configs/swin/swin_large_patch4_window7_224_22kto1k_finetune.yaml) |\n| Swin-L | ImageNet-22K | 384x384 | 87.3 | 98.2 | 197M | 103.9G | 42 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pth)/[baidu](https://pan.baidu.com/s/1sl7o_bJA143OD7UqSLAMoA) | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pth)/[baidu](https://pan.baidu.com/s/1X0FLHQyPOC6Kmv2CmgxJvA)/[config](configs/swin/swin_large_patch4_window12_384_22kto1k_finetune.yaml) |\n\n**ImageNet-1K and ImageNet-22K Pretrained Swin-V2 Models**\n\n| name | pretrain | resolution | window |acc@1 | acc@5 | #params | FLOPs | FPS |22K model | 1K model |\n|:---------------------:| :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---:|:---: |:---: |\n| SwinV2-T | ImageNet-1K | 256x256 | 8x8 | 81.8 | 95.9 | 28M | 5.9G | 572 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1RzLkAH_5OtfRCJe6Vlg6rg?pwd=swin)/[config](configs/swinv2/swinv2_tiny_patch4_window8_256.yaml) |\n| SwinV2-S | ImageNet-1K | 256x256 | 8x8 | 83.7 | 96.6 | 50M | 11.5G | 327 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/195PdA41szEduW3jEtRSa4Q?pwd=swin)/[config](configs/swinv2/swinv2_small_patch4_window8_256.yaml) |\n| SwinV2-B | ImageNet-1K | 256x256 | 8x8 | 84.2 | 96.9 | 88M | 20.3G | 217 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/18AfMSz3dPyzIvP1dKuERvQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window8_256.yaml) |\n| SwinV2-T | ImageNet-1K | 256x256 | 16x16 | 82.8 | 96.2 | 28M | 6.6G | 437 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1dyK3cK9Xipmv6RnTtrPocw?pwd=swin)/[config](configs/swinv2/swinv2_tiny_patch4_window16_256.yaml) |\n| SwinV2-S | ImageNet-1K | 256x256 | 16x16 | 84.1 | 96.8 | 50M | 12.6G | 257 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1ZIPiSfWNKTPp821Ka-Mifw?pwd=swin)/[config](configs/swinv2/swinv2_small_patch4_window16_256.yaml) |\n| SwinV2-B | ImageNet-1K | 256x256 | 16x16 | 84.6 | 97.0 | 88M | 21.8G | 174 | - | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window16_256.pth)/[baidu](https://pan.baidu.com/s/1dlDQGn8BXCmnh7wQSM5Nhw?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window16_256.yaml) |\n| SwinV2-B\\* | ImageNet-22K | 256x256 | 16x16 | 86.2 | 97.9 | 88M | 21.8G | 174 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/1Xc2rsSsRQz_sy5mjgfxrMQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/1sgstld4MgGsZxhUAW7MlmQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.yaml) |\n| SwinV2-B\\* | ImageNet-22K | 384x384 | 24x24 | 87.1 | 98.2 | 88M | 54.7G | 57 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/1Xc2rsSsRQz_sy5mjgfxrMQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/17u3sEQaUYlvfL195rrORzQ?pwd=swin)/[config](configs/swinv2/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.yaml) |\n| SwinV2-L\\* | ImageNet-22K | 256x256 | 16x16 | 86.9 | 98.0 | 197M | 47.5G | 95 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/11PhCV7qAGXtZ8dXNgyiGOw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/1pqp31N80qIWjFPbudzB6Bw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.yaml) |\n| SwinV2-L\\* | ImageNet-22K | 384x384 | 24x24 | 87.6 | 98.3 | 197M | 115.4G | 33 | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth)/[baidu](https://pan.baidu.com/s/11PhCV7qAGXtZ8dXNgyiGOw?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12_192_22k.yaml) | [github](https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.pth)/[baidu](https://pan.baidu.com/s/13URdNkygr3Xn0N3e6IwjgA?pwd=swin)/[config](configs/swinv2/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.yaml) |\n\nNote: \n- SwinV2-B\\* (SwinV2-L\\*) with input resolution of 256x256 and 384x384 both fine-tuned from the same pre-training model using a smaller input resolution of 192x192.\n- SwinV2-B\\* (384x384) achieves 78.08 acc@1 on ImageNet-1K-V2 while SwinV2-L\\* (384x384) achieves 78.31.\n\n**ImageNet-1K Pretrained Swin MLP Models**\n\n| name | pretrain | resolution |acc@1 | acc@5 | #params | FLOPs | FPS | 1K model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| [Mixer-B/16](https://arxiv.org/pdf/2105.01601.pdf) | ImageNet-1K | 224x224 | 76.4 | - | 59M | 12.7G | - | [official repo](https://github.com/google-research/vision_transformer) |\n| [ResMLP-S24](https://arxiv.org/abs/2105.03404) | ImageNet-1K | 224x224 | 79.4 | - | 30M | 6.0G | 715 | [timm](https://github.com/rwightman/pytorch-image-models) |\n| [ResMLP-B24](https://arxiv.org/abs/2105.03404) | ImageNet-1K | 224x224 | 81.0 | - | 116M | 23.0G | 231 | [timm](https://github.com/rwightman/pytorch-image-models) |\n| Swin-T/C24 | ImageNet-1K | 256x256 | 81.6 | 95.7 | 28M | 5.9G | 563 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_tiny_c24_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/17k-7l6Sxt7uZ7IV0f26GNQ)/[config](configs/swin/swin_tiny_c24_patch4_window8_256.yaml) |\n| SwinMLP-T/C24 | ImageNet-1K | 256x256 | 79.4 | 94.6 | 20M | 4.0G | 807 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c24_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1Sa4vP5R0M2RjfIe9HIga-Q)/[config](configs/swin/swin_mlp_tiny_c24_patch4_window8_256.yaml) |\n| SwinMLP-T/C12 | ImageNet-1K | 256x256 | 79.6 | 94.7 | 21M | 4.0G | 792 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c12_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1mM9J2_DEVZHUB5ASIpFl0w)/[config](configs/swin/swin_mlp_tiny_c12_patch4_window8_256.yaml) |\n| SwinMLP-T/C6 | ImageNet-1K | 256x256 | 79.7 | 94.9 | 23M | 4.0G | 766 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_tiny_c6_patch4_window8_256.pth)/[baidu](https://pan.baidu.com/s/1hUTYVT2W1CsjICw-3W-Vjg)/[config](configs/swin/swin_mlp_tiny_c6_patch4_window8_256.yaml) |\n| SwinMLP-B | ImageNet-1K | 224x224 | 81.3 | 95.3 | 61M | 10.4G | 409 | [github](https://github.com/SwinTransformer/storage/releases/download/v1.0.5/swin_mlp_base_patch4_window7_224.pth)/[baidu](https://pan.baidu.com/s/1zww3dnbX3GxNiGfb-GwyUg)/[config](configs/swin/swin_mlp_base_patch4_window7_224.yaml) |\n\nNote: access code for `baidu` is `swin`. C24 means each head has 24 channels.\n\n**ImageNet-22K Pretrained Swin-MoE Models**\n\n- Please refer to [get_started](get_started.md#mixture-of-experts-support) for instructions on running Swin-MoE. \n- Pretrained models for Swin-MoE can be found in [MODEL HUB](MODELHUB.md#imagenet-22k-pretrained-swin-moe-models)\n\n## Main Results on Downstream Tasks\n\n**COCO Object Detection (2017 val)**\n\n| Backbone | Method | pretrain | Lr Schd | box mAP | mask mAP | #params | FLOPs |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Swin-T | Mask R-CNN | ImageNet-1K | 3x | 46.0 | 41.6 | 48M | 267G |\n| Swin-S | Mask R-CNN | ImageNet-1K | 3x | 48.5 | 43.3 | 69M | 359G |\n| Swin-T | Cascade Mask R-CNN | ImageNet-1K | 3x | 50.4 | 43.7 | 86M | 745G |\n| Swin-S | Cascade Mask R-CNN | ImageNet-1K | 3x | 51.9 | 45.0 | 107M | 838G |\n| Swin-B | Cascade Mask R-CNN | ImageNet-1K | 3x | 51.9 | 45.0 | 145M | 982G |\n| Swin-T | RepPoints V2 | ImageNet-1K | 3x | 50.0 | - | 45M | 283G |\n| Swin-T | Mask RepPoints V2 | ImageNet-1K | 3x | 50.3 | 43.6 | 47M | 292G |\n| Swin-B | HTC++ | ImageNet-22K | 6x | 56.4 | 49.1 | 160M | 1043G |\n| Swin-L | HTC++ | ImageNet-22K | 3x | 57.1 | 49.5 | 284M | 1470G |\n| Swin-L | HTC++* | ImageNet-22K | 3x | 58.0 | 50.4 | 284M | - |\n\nNote: * indicates multi-scale testing.\n\n**ADE20K Semantic Segmentation (val)**\n\n| Backbone | Method | pretrain | Crop Size | Lr Schd | mIoU | mIoU (ms+flip) | #params | FLOPs |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Swin-T | UPerNet | ImageNet-1K | 512x512 | 160K | 44.51 | 45.81 | 60M | 945G |\n| Swin-S | UperNet | ImageNet-1K | 512x512 | 160K | 47.64 | 49.47 | 81M | 1038G |\n| Swin-B | UperNet | ImageNet-1K | 512x512 | 160K | 48.13 | 49.72 | 121M | 1188G |\n| Swin-B | UPerNet | ImageNet-22K | 640x640 | 160K | 50.04 | 51.66 | 121M | 1841G |\n| Swin-L | UperNet | ImageNet-22K | 640x640 | 160K | 52.05 | 53.53 | 234M | 3230G |\n\n## Citing Swin Transformer\n\n```\n@inproceedings{liu2021Swin,\n title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},\n author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},\n booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n year={2021}\n}\n```\n## Citing Local Relation Networks (the first full-attention visual backbone)\n```\n@inproceedings{hu2019local,\n title={Local Relation Networks for Image Recognition},\n author={Hu, Han and Zhang, Zheng and Xie, Zhenda and Lin, Stephen},\n booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n pages={3464--3473},\n year={2019}\n}\n```\n## Citing Swin Transformer V2\n```\n@inproceedings{liu2021swinv2,\n title={Swin Transformer V2: Scaling Up Capacity and Resolution}, \n author={Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},\n booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},\n year={2022}\n}\n```\n## Citing SimMIM (a self-supervised approach that enables SwinV2-G)\n```\n@inproceedings{xie2021simmim,\n title={SimMIM: A Simple Framework for Masked Image Modeling},\n author={Xie, Zhenda and Zhang, Zheng and Cao, Yue and Lin, Yutong and Bao, Jianmin and Yao, Zhuliang and Dai, Qi and Hu, Han},\n booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},\n year={2022}\n}\n```\n## Citing SimMIM-data-scaling\n```\n@article{xie2022data,\n title={On Data Scaling in Masked Image Modeling},\n author={Xie, Zhenda and Zhang, Zheng and Cao, Yue and Lin, Yutong and Wei, Yixuan and Dai, Qi and Hu, Han},\n journal={arXiv preprint arXiv:2206.04664},\n year={2022}\n}\n```\n## Citing Swin-MoE\n```\n@misc{hwang2022tutel,\n title={Tutel: Adaptive Mixture-of-Experts at Scale}, \n author={Changho Hwang and Wei Cui and Yifan Xiong and Ziyue Yang and Ze Liu and Han Hu and Zilong Wang and Rafael Salas and Jithin Jose and Prabhat Ram and Joe Chau and Peng Cheng and Fan Yang and Mao Yang and Yongqiang Xiong},\n year={2022},\n eprint={2206.03382},\n archivePrefix={arXiv}\n}\n```\n\n## Getting Started\n\n- For **Image Classification**, please see [get_started.md](get_started.md) for detailed instructions.\n- For **Object Detection and Instance Segmentation**, please see [Swin Transformer for Object Detection](https://github.com/SwinTransformer/Swin-Transformer-Object-Detection).\n- For **Semantic Segmentation**, please see [Swin Transformer for Semantic Segmentation](https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation).\n- For **Self-Supervised Learning**, please see [Transformer-SSL](https://github.com/SwinTransformer/Transformer-SSL).\n- For **Video Recognition**, please see [Video Swin Transformer](https://github.com/SwinTransformer/Video-Swin-Transformer).\n\n## Third-party Usage and Experiments\n\n***In this pargraph, we cross link third-party repositories which use Swin and report results. You can let us know by raising an issue*** \n\n(`Note please report accuracy numbers and provide trained models in your new repository to facilitate others to get sense of correctness and model behavior`)\n\n[12/29/2022] Swin Transformers (V2) inference implemented in FasterTransformer: [FasterTransformer](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/swin_guide.md)\n\n[06/30/2022] Swin Transformers (V1) inference implemented in FasterTransformer: [FasterTransformer](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/swin_guide.md)\n\n[05/12/2022] Swin Transformers (V1) implemented in TensorFlow with the pre-trained parameters ported into them. Find the implementation,\nTensorFlow weights, code example here in [this repository](https://github.com/sayakpaul/swin-transformers-tf/).\n\n[04/06/2022] Swin Transformer for Audio Classification: [Hierarchical Token Semantic Audio Transformer](https://github.com/RetroCirce/HTS-Audio-Transformer).\n\n[12/21/2021] Swin Transformer for StyleGAN: [StyleSwin](https://github.com/microsoft/StyleSwin)\n\n[12/13/2021] Swin Transformer for Face Recognition: [FaceX-Zoo](https://github.com/JDAI-CV/FaceX-Zoo)\n\n[08/29/2021] Swin Transformer for Image Restoration: [SwinIR](https://github.com/JingyunLiang/SwinIR)\n\n[08/12/2021] Swin Transformer for person reID: [https://github.com/layumi/Person_reID_baseline_pytorch](https://github.com/layumi/Person_reID_baseline_pytorch)\n\n[06/29/2021] Swin-Transformer in PaddleClas and inference based on whl package: [https://github.com/PaddlePaddle/PaddleClas](https://github.com/PaddlePaddle/PaddleClas)\n\n[04/14/2021] Swin for RetinaNet in Detectron: https://github.com/xiaohu2015/SwinT_detectron2.\n\n[04/16/2021] Included in a famous model zoo: https://github.com/rwightman/pytorch-image-models.\n\n[04/20/2021] Swin-Transformer classifier inference using TorchServe: https://github.com/kamalkraj/Swin-Transformer-Serve\n\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft \ntrademarks or logos is subject to and must follow \n[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n"
},
{
"path": "SECURITY.md",
"content": "\n\n## Security\n\nMicrosoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).\n\nIf you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.\n\n## Reporting Security Issues\n\n**Please do not report security vulnerabilities through public GitHub issues.**\n\nInstead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).\n\nIf you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).\n\nYou should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). \n\nPlease include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:\n\n * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)\n * Full paths of source file(s) related to the manifestation of the issue\n * The location of the affected source code (tag/branch/commit or direct URL)\n * Any special configuration required to reproduce the issue\n * Step-by-step instructions to reproduce the issue\n * Proof-of-concept or exploit code (if possible)\n * Impact of the issue, including how an attacker might exploit the issue\n\nThis information will help us triage your report more quickly.\n\nIf you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs.\n\n## Preferred Languages\n\nWe prefer all communications to be in English.\n\n## Policy\n\nMicrosoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).\n\n"
},
{
"path": "SUPPORT.md",
"content": "# TODO: The maintainer of this repo has not yet edited this file\r\n\r\n**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?\r\n\r\n- **No CSS support:** Fill out this template with information about how to file issues and get help.\r\n- **Yes CSS support:** Fill out an intake form at [aka.ms/spot](https://aka.ms/spot). CSS will work with/help you to determine next steps. More details also available at [aka.ms/onboardsupport](https://aka.ms/onboardsupport).\r\n- **Not sure?** Fill out a SPOT intake as though the answer were \"Yes\". CSS will help you decide.\r\n\r\n*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*\r\n\r\n# Support\r\n\r\n## How to file issues and get help \r\n\r\nThis project uses GitHub Issues to track bugs and feature requests. Please search the existing \r\nissues before filing new issues to avoid duplicates. For new issues, file your bug or \r\nfeature request as a new Issue.\r\n\r\nFor help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE \r\nFOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER\r\nCHANNEL. WHERE WILL YOU HELP PEOPLE?**.\r\n\r\n## Microsoft Support Policy \r\n\r\nSupport for this **PROJECT or PRODUCT** is limited to the resources listed above.\r\n"
},
{
"path": "config.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------'\n\nimport os\nimport torch\nimport yaml\nfrom yacs.config import CfgNode as CN\n\n# pytorch major version (1.x or 2.x)\nPYTORCH_MAJOR_VERSION = int(torch.__version__.split('.')[0])\n\n_C = CN()\n\n# Base config files\n_C.BASE = ['']\n\n# -----------------------------------------------------------------------------\n# Data settings\n# -----------------------------------------------------------------------------\n_C.DATA = CN()\n# Batch size for a single GPU, could be overwritten by command line argument\n_C.DATA.BATCH_SIZE = 128\n# Path to dataset, could be overwritten by command line argument\n_C.DATA.DATA_PATH = ''\n# Dataset name\n_C.DATA.DATASET = 'imagenet'\n# Input image size\n_C.DATA.IMG_SIZE = 224\n# Interpolation to resize image (random, bilinear, bicubic)\n_C.DATA.INTERPOLATION = 'bicubic'\n# Use zipped dataset instead of folder dataset\n# could be overwritten by command line argument\n_C.DATA.ZIP_MODE = False\n# Cache Data in Memory, could be overwritten by command line argument\n_C.DATA.CACHE_MODE = 'part'\n# Pin CPU memory in DataLoader for more efficient (sometimes) transfer to GPU.\n_C.DATA.PIN_MEMORY = True\n# Number of data loading threads\n_C.DATA.NUM_WORKERS = 8\n\n# [SimMIM] Mask patch size for MaskGenerator\n_C.DATA.MASK_PATCH_SIZE = 32\n# [SimMIM] Mask ratio for MaskGenerator\n_C.DATA.MASK_RATIO = 0.6\n\n# -----------------------------------------------------------------------------\n# Model settings\n# -----------------------------------------------------------------------------\n_C.MODEL = CN()\n# Model type\n_C.MODEL.TYPE = 'swin'\n# Model name\n_C.MODEL.NAME = 'swin_tiny_patch4_window7_224'\n# Pretrained weight from checkpoint, could be imagenet22k pretrained weight\n# could be overwritten by command line argument\n_C.MODEL.PRETRAINED = ''\n# Checkpoint to resume, could be overwritten by command line argument\n_C.MODEL.RESUME = ''\n# Number of classes, overwritten in data preparation\n_C.MODEL.NUM_CLASSES = 1000\n# Dropout rate\n_C.MODEL.DROP_RATE = 0.0\n# Drop path rate\n_C.MODEL.DROP_PATH_RATE = 0.1\n# Label Smoothing\n_C.MODEL.LABEL_SMOOTHING = 0.1\n\n# Swin Transformer parameters\n_C.MODEL.SWIN = CN()\n_C.MODEL.SWIN.PATCH_SIZE = 4\n_C.MODEL.SWIN.IN_CHANS = 3\n_C.MODEL.SWIN.EMBED_DIM = 96\n_C.MODEL.SWIN.DEPTHS = [2, 2, 6, 2]\n_C.MODEL.SWIN.NUM_HEADS = [3, 6, 12, 24]\n_C.MODEL.SWIN.WINDOW_SIZE = 7\n_C.MODEL.SWIN.MLP_RATIO = 4.\n_C.MODEL.SWIN.QKV_BIAS = True\n_C.MODEL.SWIN.QK_SCALE = None\n_C.MODEL.SWIN.APE = False\n_C.MODEL.SWIN.PATCH_NORM = True\n\n# Swin Transformer V2 parameters\n_C.MODEL.SWINV2 = CN()\n_C.MODEL.SWINV2.PATCH_SIZE = 4\n_C.MODEL.SWINV2.IN_CHANS = 3\n_C.MODEL.SWINV2.EMBED_DIM = 96\n_C.MODEL.SWINV2.DEPTHS = [2, 2, 6, 2]\n_C.MODEL.SWINV2.NUM_HEADS = [3, 6, 12, 24]\n_C.MODEL.SWINV2.WINDOW_SIZE = 7\n_C.MODEL.SWINV2.MLP_RATIO = 4.\n_C.MODEL.SWINV2.QKV_BIAS = True\n_C.MODEL.SWINV2.APE = False\n_C.MODEL.SWINV2.PATCH_NORM = True\n_C.MODEL.SWINV2.PRETRAINED_WINDOW_SIZES = [0, 0, 0, 0]\n\n# Swin Transformer MoE parameters\n_C.MODEL.SWIN_MOE = CN()\n_C.MODEL.SWIN_MOE.PATCH_SIZE = 4\n_C.MODEL.SWIN_MOE.IN_CHANS = 3\n_C.MODEL.SWIN_MOE.EMBED_DIM = 96\n_C.MODEL.SWIN_MOE.DEPTHS = [2, 2, 6, 2]\n_C.MODEL.SWIN_MOE.NUM_HEADS = [3, 6, 12, 24]\n_C.MODEL.SWIN_MOE.WINDOW_SIZE = 7\n_C.MODEL.SWIN_MOE.MLP_RATIO = 4.\n_C.MODEL.SWIN_MOE.QKV_BIAS = True\n_C.MODEL.SWIN_MOE.QK_SCALE = None\n_C.MODEL.SWIN_MOE.APE = False\n_C.MODEL.SWIN_MOE.PATCH_NORM = True\n_C.MODEL.SWIN_MOE.MLP_FC2_BIAS = True\n_C.MODEL.SWIN_MOE.INIT_STD = 0.02\n_C.MODEL.SWIN_MOE.PRETRAINED_WINDOW_SIZES = [0, 0, 0, 0]\n_C.MODEL.SWIN_MOE.MOE_BLOCKS = [[-1], [-1], [-1], [-1]]\n_C.MODEL.SWIN_MOE.NUM_LOCAL_EXPERTS = 1\n_C.MODEL.SWIN_MOE.TOP_VALUE = 1\n_C.MODEL.SWIN_MOE.CAPACITY_FACTOR = 1.25\n_C.MODEL.SWIN_MOE.COSINE_ROUTER = False\n_C.MODEL.SWIN_MOE.NORMALIZE_GATE = False\n_C.MODEL.SWIN_MOE.USE_BPR = True\n_C.MODEL.SWIN_MOE.IS_GSHARD_LOSS = False\n_C.MODEL.SWIN_MOE.GATE_NOISE = 1.0\n_C.MODEL.SWIN_MOE.COSINE_ROUTER_DIM = 256\n_C.MODEL.SWIN_MOE.COSINE_ROUTER_INIT_T = 0.5\n_C.MODEL.SWIN_MOE.MOE_DROP = 0.0\n_C.MODEL.SWIN_MOE.AUX_LOSS_WEIGHT = 0.01\n\n# Swin MLP parameters\n_C.MODEL.SWIN_MLP = CN()\n_C.MODEL.SWIN_MLP.PATCH_SIZE = 4\n_C.MODEL.SWIN_MLP.IN_CHANS = 3\n_C.MODEL.SWIN_MLP.EMBED_DIM = 96\n_C.MODEL.SWIN_MLP.DEPTHS = [2, 2, 6, 2]\n_C.MODEL.SWIN_MLP.NUM_HEADS = [3, 6, 12, 24]\n_C.MODEL.SWIN_MLP.WINDOW_SIZE = 7\n_C.MODEL.SWIN_MLP.MLP_RATIO = 4.\n_C.MODEL.SWIN_MLP.APE = False\n_C.MODEL.SWIN_MLP.PATCH_NORM = True\n\n# [SimMIM] Norm target during training\n_C.MODEL.SIMMIM = CN()\n_C.MODEL.SIMMIM.NORM_TARGET = CN()\n_C.MODEL.SIMMIM.NORM_TARGET.ENABLE = False\n_C.MODEL.SIMMIM.NORM_TARGET.PATCH_SIZE = 47\n\n# -----------------------------------------------------------------------------\n# Training settings\n# -----------------------------------------------------------------------------\n_C.TRAIN = CN()\n_C.TRAIN.START_EPOCH = 0\n_C.TRAIN.EPOCHS = 300\n_C.TRAIN.WARMUP_EPOCHS = 20\n_C.TRAIN.WEIGHT_DECAY = 0.05\n_C.TRAIN.BASE_LR = 5e-4\n_C.TRAIN.WARMUP_LR = 5e-7\n_C.TRAIN.MIN_LR = 5e-6\n# Clip gradient norm\n_C.TRAIN.CLIP_GRAD = 5.0\n# Auto resume from latest checkpoint\n_C.TRAIN.AUTO_RESUME = True\n# Gradient accumulation steps\n# could be overwritten by command line argument\n_C.TRAIN.ACCUMULATION_STEPS = 1\n# Whether to use gradient checkpointing to save memory\n# could be overwritten by command line argument\n_C.TRAIN.USE_CHECKPOINT = False\n\n# LR scheduler\n_C.TRAIN.LR_SCHEDULER = CN()\n_C.TRAIN.LR_SCHEDULER.NAME = 'cosine'\n# Epoch interval to decay LR, used in StepLRScheduler\n_C.TRAIN.LR_SCHEDULER.DECAY_EPOCHS = 30\n# LR decay rate, used in StepLRScheduler\n_C.TRAIN.LR_SCHEDULER.DECAY_RATE = 0.1\n# warmup_prefix used in CosineLRScheduler\n_C.TRAIN.LR_SCHEDULER.WARMUP_PREFIX = True\n# [SimMIM] Gamma / Multi steps value, used in MultiStepLRScheduler\n_C.TRAIN.LR_SCHEDULER.GAMMA = 0.1\n_C.TRAIN.LR_SCHEDULER.MULTISTEPS = []\n\n# Optimizer\n_C.TRAIN.OPTIMIZER = CN()\n_C.TRAIN.OPTIMIZER.NAME = 'adamw'\n# Optimizer Epsilon\n_C.TRAIN.OPTIMIZER.EPS = 1e-8\n# Optimizer Betas\n_C.TRAIN.OPTIMIZER.BETAS = (0.9, 0.999)\n# SGD momentum\n_C.TRAIN.OPTIMIZER.MOMENTUM = 0.9\n\n# [SimMIM] Layer decay for fine-tuning\n_C.TRAIN.LAYER_DECAY = 1.0\n\n# MoE\n_C.TRAIN.MOE = CN()\n# Only save model on master device\n_C.TRAIN.MOE.SAVE_MASTER = False\n# -----------------------------------------------------------------------------\n# Augmentation settings\n# -----------------------------------------------------------------------------\n_C.AUG = CN()\n# Color jitter factor\n_C.AUG.COLOR_JITTER = 0.4\n# Use AutoAugment policy. \"v0\" or \"original\"\n_C.AUG.AUTO_AUGMENT = 'rand-m9-mstd0.5-inc1'\n# Random erase prob\n_C.AUG.REPROB = 0.25\n# Random erase mode\n_C.AUG.REMODE = 'pixel'\n# Random erase count\n_C.AUG.RECOUNT = 1\n# Mixup alpha, mixup enabled if > 0\n_C.AUG.MIXUP = 0.8\n# Cutmix alpha, cutmix enabled if > 0\n_C.AUG.CUTMIX = 1.0\n# Cutmix min/max ratio, overrides alpha and enables cutmix if set\n_C.AUG.CUTMIX_MINMAX = None\n# Probability of performing mixup or cutmix when either/both is enabled\n_C.AUG.MIXUP_PROB = 1.0\n# Probability of switching to cutmix when both mixup and cutmix enabled\n_C.AUG.MIXUP_SWITCH_PROB = 0.5\n# How to apply mixup/cutmix params. Per \"batch\", \"pair\", or \"elem\"\n_C.AUG.MIXUP_MODE = 'batch'\n\n# -----------------------------------------------------------------------------\n# Testing settings\n# -----------------------------------------------------------------------------\n_C.TEST = CN()\n# Whether to use center crop when testing\n_C.TEST.CROP = True\n# Whether to use SequentialSampler as validation sampler\n_C.TEST.SEQUENTIAL = False\n_C.TEST.SHUFFLE = False\n\n# -----------------------------------------------------------------------------\n# Misc\n# -----------------------------------------------------------------------------\n# [SimMIM] Whether to enable pytorch amp, overwritten by command line argument\n_C.ENABLE_AMP = False\n\n# Enable Pytorch automatic mixed precision (amp).\n_C.AMP_ENABLE = True\n# [Deprecated] Mixed precision opt level of apex, if O0, no apex amp is used ('O0', 'O1', 'O2')\n_C.AMP_OPT_LEVEL = ''\n# Path to output folder, overwritten by command line argument\n_C.OUTPUT = ''\n# Tag of experiment, overwritten by command line argument\n_C.TAG = 'default'\n# Frequency to save checkpoint\n_C.SAVE_FREQ = 1\n# Frequency to logging info\n_C.PRINT_FREQ = 10\n# Fixed random seed\n_C.SEED = 0\n# Perform evaluation only, overwritten by command line argument\n_C.EVAL_MODE = False\n# Test throughput only, overwritten by command line argument\n_C.THROUGHPUT_MODE = False\n# local rank for DistributedDataParallel, given by command line argument\n_C.LOCAL_RANK = 0\n# for acceleration\n_C.FUSED_WINDOW_PROCESS = False\n_C.FUSED_LAYERNORM = False\n\n\ndef _update_config_from_file(config, cfg_file):\n config.defrost()\n with open(cfg_file, 'r') as f:\n yaml_cfg = yaml.load(f, Loader=yaml.FullLoader)\n\n for cfg in yaml_cfg.setdefault('BASE', ['']):\n if cfg:\n _update_config_from_file(\n config, os.path.join(os.path.dirname(cfg_file), cfg)\n )\n print('=> merge config from {}'.format(cfg_file))\n config.merge_from_file(cfg_file)\n config.freeze()\n\n\ndef update_config(config, args):\n _update_config_from_file(config, args.cfg)\n\n config.defrost()\n if args.opts:\n config.merge_from_list(args.opts)\n\n def _check_args(name):\n if hasattr(args, name) and eval(f'args.{name}'):\n return True\n return False\n\n # merge from specific arguments\n if _check_args('batch_size'):\n config.DATA.BATCH_SIZE = args.batch_size\n if _check_args('data_path'):\n config.DATA.DATA_PATH = args.data_path\n if _check_args('zip'):\n config.DATA.ZIP_MODE = True\n if _check_args('cache_mode'):\n config.DATA.CACHE_MODE = args.cache_mode\n if _check_args('pretrained'):\n config.MODEL.PRETRAINED = args.pretrained\n if _check_args('resume'):\n config.MODEL.RESUME = args.resume\n if _check_args('accumulation_steps'):\n config.TRAIN.ACCUMULATION_STEPS = args.accumulation_steps\n if _check_args('use_checkpoint'):\n config.TRAIN.USE_CHECKPOINT = True\n if _check_args('amp_opt_level'):\n print(\"[warning] Apex amp has been deprecated, please use pytorch amp instead!\")\n if args.amp_opt_level == 'O0':\n config.AMP_ENABLE = False\n if _check_args('disable_amp'):\n config.AMP_ENABLE = False\n if _check_args('output'):\n config.OUTPUT = args.output\n if _check_args('tag'):\n config.TAG = args.tag\n if _check_args('eval'):\n config.EVAL_MODE = True\n if _check_args('throughput'):\n config.THROUGHPUT_MODE = True\n\n # [SimMIM]\n if _check_args('enable_amp'):\n config.ENABLE_AMP = args.enable_amp\n\n # for acceleration\n if _check_args('fused_window_process'):\n config.FUSED_WINDOW_PROCESS = True\n if _check_args('fused_layernorm'):\n config.FUSED_LAYERNORM = True\n ## Overwrite optimizer if not None, currently we use it for [fused_adam, fused_lamb]\n if _check_args('optim'):\n config.TRAIN.OPTIMIZER.NAME = args.optim\n\n # set local rank for distributed training\n if PYTORCH_MAJOR_VERSION == 1:\n config.LOCAL_RANK = args.local_rank\n else:\n config.LOCAL_RANK = int(os.environ['LOCAL_RANK'])\n\n # output folder\n config.OUTPUT = os.path.join(config.OUTPUT, config.MODEL.NAME, config.TAG)\n\n config.freeze()\n\n\ndef get_config(args):\n \"\"\"Get a yacs CfgNode object with default values.\"\"\"\n # Return a clone so that the defaults will not be altered\n # This is for the \"local variable\" use pattern\n config = _C.clone()\n update_config(config, args)\n\n return config\n"
},
{
"path": "configs/simmim/simmim_finetune__swin_base__img224_window7__800ep.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: simmim_finetune\n DROP_PATH_RATE: 0.1\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 7\nDATA:\n IMG_SIZE: 224\nTRAIN:\n EPOCHS: 100\n WARMUP_EPOCHS: 20\n BASE_LR: 1.25e-3\n WARMUP_LR: 2.5e-7\n MIN_LR: 2.5e-7\n WEIGHT_DECAY: 0.05\n LAYER_DECAY: 0.8\nPRINT_FREQ: 100\nSAVE_FREQ: 5\nTAG: simmim_finetune__swin_base__img224_window7__800ep"
},
{
"path": "configs/simmim/simmim_finetune__swinv2_base__img224_window14__800ep.yaml",
"content": "MODEL:\n TYPE: swinv2\n NAME: simmim_finetune\n DROP_PATH_RATE: 0.1\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 14\n PRETRAINED_WINDOW_SIZES: [ 12, 12, 12, 6 ]\nDATA:\n IMG_SIZE: 224\nTRAIN:\n EPOCHS: 100\n WARMUP_EPOCHS: 20\n BASE_LR: 1.25e-3\n WARMUP_LR: 2.5e-7\n MIN_LR: 2.5e-7\n WEIGHT_DECAY: 0.05\n LAYER_DECAY: 0.75\nPRINT_FREQ: 100\nSAVE_FREQ: 5\nTAG: simmim_finetune__swinv2_base__img224_window14__800ep"
},
{
"path": "configs/simmim/simmim_pretrain__swin_base__img192_window6__800ep.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: simmim_pretrain\n DROP_PATH_RATE: 0.0\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 6\nDATA:\n IMG_SIZE: 192\n MASK_PATCH_SIZE: 32\n MASK_RATIO: 0.6\nTRAIN:\n EPOCHS: 800\n WARMUP_EPOCHS: 10\n BASE_LR: 1e-4\n WARMUP_LR: 5e-7\n WEIGHT_DECAY: 0.05\n LR_SCHEDULER:\n NAME: 'multistep'\n GAMMA: 0.1\n MULTISTEPS: [700,]\nPRINT_FREQ: 100\nSAVE_FREQ: 5\nTAG: simmim_pretrain__swin_base__img192_window6__800ep"
},
{
"path": "configs/simmim/simmim_pretrain__swinv2_base__img192_window12__800ep.yaml",
"content": "MODEL:\n TYPE: swinv2\n NAME: simmim_pretrain\n DROP_PATH_RATE: 0.1\n SIMMIM:\n NORM_TARGET:\n ENABLE: True\n PATCH_SIZE: 47\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\nDATA:\n IMG_SIZE: 192\n MASK_PATCH_SIZE: 32\n MASK_RATIO: 0.6\nTRAIN:\n EPOCHS: 800\n WARMUP_EPOCHS: 10\n BASE_LR: 1e-4\n WARMUP_LR: 5e-7\n WEIGHT_DECAY: 0.05\n LR_SCHEDULER:\n NAME: 'multistep'\n GAMMA: 0.1\n MULTISTEPS: [700,]\nPRINT_FREQ: 100\nSAVE_FREQ: 5\nTAG: simmim_pretrain__swinv2_base__img192_window12__800ep"
},
{
"path": "configs/swin/swin_base_patch4_window12_384_22kto1k_finetune.yaml",
"content": "DATA:\n IMG_SIZE: 384\nMODEL:\n TYPE: swin\n NAME: swin_base_patch4_window12_384_22kto1k_finetune\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07\nTEST:\n CROP: False"
},
{
"path": "configs/swin/swin_base_patch4_window12_384_finetune.yaml",
"content": "DATA:\n IMG_SIZE: 384\nMODEL:\n TYPE: swin\n NAME: swin_base_patch4_window12_384_finetune\n DROP_PATH_RATE: 0.5\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07\nTEST:\n CROP: False"
},
{
"path": "configs/swin/swin_base_patch4_window7_224.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_base_patch4_window7_224\n DROP_PATH_RATE: 0.5\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 7"
},
{
"path": "configs/swin/swin_base_patch4_window7_224_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\nMODEL:\n TYPE: swin\n NAME: swin_base_patch4_window7_224_22k\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.05\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swin/swin_base_patch4_window7_224_22kto1k_finetune.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_base_patch4_window7_224_22kto1k_finetune\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swin/swin_large_patch4_window12_384_22kto1k_finetune.yaml",
"content": "DATA:\n IMG_SIZE: 384\nMODEL:\n TYPE: swin\n NAME: swin_large_patch4_window12_384_22kto1k_finetune\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 12\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07\nTEST:\n CROP: False"
},
{
"path": "configs/swin/swin_large_patch4_window7_224_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\nMODEL:\n TYPE: swin\n NAME: swin_large_patch4_window7_224_22k\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.05\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swin/swin_large_patch4_window7_224_22kto1k_finetune.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_large_patch4_window7_224_22kto1k_finetune\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swin/swin_small_patch4_window7_224.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_small_patch4_window7_224\n DROP_PATH_RATE: 0.3\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7"
},
{
"path": "configs/swin/swin_small_patch4_window7_224_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\nMODEL:\n TYPE: swin\n NAME: swin_small_patch4_window7_224_22k\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.05\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swin/swin_small_patch4_window7_224_22kto1k_finetune.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_small_patch4_window7_224_22kto1k_finetune\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swin/swin_tiny_c24_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swin\n NAME: swin_tiny_c24_patch4_window8_256\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swin/swin_tiny_patch4_window7_224.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_tiny_patch4_window7_224\n DROP_PATH_RATE: 0.2\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7"
},
{
"path": "configs/swin/swin_tiny_patch4_window7_224_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\nMODEL:\n TYPE: swin\n NAME: swin_tiny_patch4_window7_224_22k\n DROP_PATH_RATE: 0.1\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.05\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swin/swin_tiny_patch4_window7_224_22kto1k_finetune.yaml",
"content": "MODEL:\n TYPE: swin\n NAME: swin_tiny_patch4_window7_224_22kto1k_finetune\n DROP_PATH_RATE: 0.1\n SWIN:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 7\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swinmlp/swin_mlp_base_patch4_window7_224.yaml",
"content": "MODEL:\n TYPE: swin_mlp\n NAME: swin_mlp_base_patch4_window7_224\n DROP_PATH_RATE: 0.5\n SWIN_MLP:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 7\n"
},
{
"path": "configs/swinmlp/swin_mlp_tiny_c12_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swin_mlp\n NAME: swin_mlp_tiny_c12_patch4_window8_256\n DROP_PATH_RATE: 0.2\n SWIN_MLP:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 8, 16, 32, 64 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swinmlp/swin_mlp_tiny_c24_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swin_mlp\n NAME: swin_mlp_tiny_c24_patch4_window8_256\n DROP_PATH_RATE: 0.2\n SWIN_MLP:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swinmlp/swin_mlp_tiny_c6_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swin_mlp\n NAME: swin_mlp_tiny_c6_patch4_window8_256\n DROP_PATH_RATE: 0.2\n SWIN_MLP:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 16, 32, 64, 128 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swinmoe/swin_moe_base_patch4_window12_192_16expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_base_patch4_window12_192_16expert_32gpu_22k\n DROP_PATH_RATE: 0.3\n SWIN_MOE:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: -2\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_base_patch4_window12_192_32expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_base_patch4_window12_192_32expert_32gpu_22k\n DROP_PATH_RATE: 0.3\n SWIN_MOE:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: 1\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_base_patch4_window12_192_8expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_base_patch4_window12_192_8expert_32gpu_22k\n DROP_PATH_RATE: 0.3\n SWIN_MOE:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: -4\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_base_patch4_window12_192_cosine_router_32expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_base_patch4_window12_192_cosine_router_32expert_32gpu_22k\n DROP_PATH_RATE: 0.3\n SWIN_MOE:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: 1\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n COSINE_ROUTER: True\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_base_patch4_window12_192_densebaseline_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_base_patch4_window12_192_densebaseline_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ -1 ], [ -1 ] ]\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\n MOE:\n SAVE_MASTER: True\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_16expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_16expert_32gpu_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: -2\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_32expert_32gpu_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: 1\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_64expert_64gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_64expert_64gpu_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: 1\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_8expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_8expert_32gpu_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: -4\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_cosine_router_32expert_32gpu_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_cosine_router_32expert_32gpu_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n INIT_STD: 0.005\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ 1, 3, 5, 7, 9, 11, 13, 15, 17 ], [ 1 ] ]\n NUM_LOCAL_EXPERTS: 1\n TOP_VALUE: 1\n CAPACITY_FACTOR: 1.25\n COSINE_ROUTER: True\n IS_GSHARD_LOSS: False\n MOE_DROP: 0.1\n AUX_LOSS_WEIGHT: 0.01\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinmoe/swin_moe_small_patch4_window12_192_densebaseline_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swin_moe\n NAME: swin_moe_small_patch4_window12_192_densebaseline_22k\n DROP_PATH_RATE: 0.2\n SWIN_MOE:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 12\n MLP_FC2_BIAS: False\n MOE_BLOCKS: [ [ -1 ], [ -1 ], [ -1 ], [ -1 ] ]\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 10\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6\n CLIP_GRAD: 3.0\n MOE:\n SAVE_MASTER: True\nTEST:\n SHUFFLE: True"
},
{
"path": "configs/swinv2/swinv2_base_patch4_window12_192_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window12_192_22k\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 12\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swinv2/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window12to16_192to256_22kto1k_ft\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 16\n PRETRAINED_WINDOW_SIZES: [ 12, 12, 12, 6 ]\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swinv2/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.yaml",
"content": "DATA:\n IMG_SIZE: 384\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window12to24_192to384_22kto1k_ft\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 24\n PRETRAINED_WINDOW_SIZES: [ 12, 12, 12, 6 ]\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07\nTEST:\n CROP: False"
},
{
"path": "configs/swinv2/swinv2_base_patch4_window16_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window16_256\n DROP_PATH_RATE: 0.5\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 16"
},
{
"path": "configs/swinv2/swinv2_base_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window8_256\n DROP_PATH_RATE: 0.5\n SWINV2:\n EMBED_DIM: 128\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 4, 8, 16, 32 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swinv2/swinv2_large_patch4_window12_192_22k.yaml",
"content": "DATA:\n DATASET: imagenet22K\n IMG_SIZE: 192\nMODEL:\n TYPE: swinv2\n NAME: swinv2_large_patch4_window12_192_22k\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 12\nTRAIN:\n EPOCHS: 90\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 0.1\n BASE_LR: 1.25e-4 # 4096 batch-size\n WARMUP_LR: 1.25e-7\n MIN_LR: 1.25e-6"
},
{
"path": "configs/swinv2/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_base_patch4_window12to16_192to256_22kto1k_ft\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 16\n PRETRAINED_WINDOW_SIZES: [ 12, 12, 12, 6 ]\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07"
},
{
"path": "configs/swinv2/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.yaml",
"content": "DATA:\n IMG_SIZE: 384\nMODEL:\n TYPE: swinv2\n NAME: swinv2_large_patch4_window12to24_192to384_22kto1k_ft\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 192\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 6, 12, 24, 48 ]\n WINDOW_SIZE: 24\n PRETRAINED_WINDOW_SIZES: [ 12, 12, 12, 6 ]\nTRAIN:\n EPOCHS: 30\n WARMUP_EPOCHS: 5\n WEIGHT_DECAY: 1e-8\n BASE_LR: 2e-05\n WARMUP_LR: 2e-08\n MIN_LR: 2e-07\nTEST:\n CROP: False"
},
{
"path": "configs/swinv2/swinv2_small_patch4_window16_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_small_patch4_window16_256\n DROP_PATH_RATE: 0.3\n SWINV2:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 16"
},
{
"path": "configs/swinv2/swinv2_small_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_small_patch4_window8_256\n DROP_PATH_RATE: 0.3\n SWINV2:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 18, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 8"
},
{
"path": "configs/swinv2/swinv2_tiny_patch4_window16_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_tiny_patch4_window16_256\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 16"
},
{
"path": "configs/swinv2/swinv2_tiny_patch4_window8_256.yaml",
"content": "DATA:\n IMG_SIZE: 256\nMODEL:\n TYPE: swinv2\n NAME: swinv2_tiny_patch4_window8_256\n DROP_PATH_RATE: 0.2\n SWINV2:\n EMBED_DIM: 96\n DEPTHS: [ 2, 2, 6, 2 ]\n NUM_HEADS: [ 3, 6, 12, 24 ]\n WINDOW_SIZE: 8"
},
{
"path": "data/__init__.py",
"content": "from .build import build_loader as _build_loader\nfrom .data_simmim_pt import build_loader_simmim\nfrom .data_simmim_ft import build_loader_finetune\n\n\ndef build_loader(config, simmim=False, is_pretrain=False):\n if not simmim:\n return _build_loader(config)\n if is_pretrain:\n return build_loader_simmim(config)\n else:\n return build_loader_finetune(config)\n"
},
{
"path": "data/build.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport os\nimport torch\nimport numpy as np\nimport torch.distributed as dist\nfrom torchvision import datasets, transforms\nfrom timm.data.constants import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD\nfrom timm.data import Mixup\nfrom timm.data import create_transform\n\nfrom .cached_image_folder import CachedImageFolder\nfrom .imagenet22k_dataset import IN22KDATASET\nfrom .samplers import SubsetRandomSampler\n\ntry:\n from torchvision.transforms import InterpolationMode\n\n\n def _pil_interp(method):\n if method == 'bicubic':\n return InterpolationMode.BICUBIC\n elif method == 'lanczos':\n return InterpolationMode.LANCZOS\n elif method == 'hamming':\n return InterpolationMode.HAMMING\n else:\n # default bilinear, do we want to allow nearest?\n return InterpolationMode.BILINEAR\n\n\n import timm.data.transforms as timm_transforms\n\n timm_transforms._pil_interp = _pil_interp\nexcept:\n from timm.data.transforms import _pil_interp\n\n\ndef build_loader(config):\n config.defrost()\n dataset_train, config.MODEL.NUM_CLASSES = build_dataset(is_train=True, config=config)\n config.freeze()\n print(f\"local rank {config.LOCAL_RANK} / global rank {dist.get_rank()} successfully build train dataset\")\n dataset_val, _ = build_dataset(is_train=False, config=config)\n print(f\"local rank {config.LOCAL_RANK} / global rank {dist.get_rank()} successfully build val dataset\")\n\n num_tasks = dist.get_world_size()\n global_rank = dist.get_rank()\n if config.DATA.ZIP_MODE and config.DATA.CACHE_MODE == 'part':\n indices = np.arange(dist.get_rank(), len(dataset_train), dist.get_world_size())\n sampler_train = SubsetRandomSampler(indices)\n else:\n sampler_train = torch.utils.data.DistributedSampler(\n dataset_train, num_replicas=num_tasks, rank=global_rank, shuffle=True\n )\n\n if config.TEST.SEQUENTIAL:\n sampler_val = torch.utils.data.SequentialSampler(dataset_val)\n else:\n sampler_val = torch.utils.data.distributed.DistributedSampler(\n dataset_val, shuffle=config.TEST.SHUFFLE\n )\n\n data_loader_train = torch.utils.data.DataLoader(\n dataset_train, sampler=sampler_train,\n batch_size=config.DATA.BATCH_SIZE,\n num_workers=config.DATA.NUM_WORKERS,\n pin_memory=config.DATA.PIN_MEMORY,\n drop_last=True,\n )\n\n data_loader_val = torch.utils.data.DataLoader(\n dataset_val, sampler=sampler_val,\n batch_size=config.DATA.BATCH_SIZE,\n shuffle=False,\n num_workers=config.DATA.NUM_WORKERS,\n pin_memory=config.DATA.PIN_MEMORY,\n drop_last=False\n )\n\n # setup mixup / cutmix\n mixup_fn = None\n mixup_active = config.AUG.MIXUP > 0 or config.AUG.CUTMIX > 0. or config.AUG.CUTMIX_MINMAX is not None\n if mixup_active:\n mixup_fn = Mixup(\n mixup_alpha=config.AUG.MIXUP, cutmix_alpha=config.AUG.CUTMIX, cutmix_minmax=config.AUG.CUTMIX_MINMAX,\n prob=config.AUG.MIXUP_PROB, switch_prob=config.AUG.MIXUP_SWITCH_PROB, mode=config.AUG.MIXUP_MODE,\n label_smoothing=config.MODEL.LABEL_SMOOTHING, num_classes=config.MODEL.NUM_CLASSES)\n\n return dataset_train, dataset_val, data_loader_train, data_loader_val, mixup_fn\n\n\ndef build_dataset(is_train, config):\n transform = build_transform(is_train, config)\n if config.DATA.DATASET == 'imagenet':\n prefix = 'train' if is_train else 'val'\n if config.DATA.ZIP_MODE:\n ann_file = prefix + \"_map.txt\"\n prefix = prefix + \".zip@/\"\n dataset = CachedImageFolder(config.DATA.DATA_PATH, ann_file, prefix, transform,\n cache_mode=config.DATA.CACHE_MODE if is_train else 'part')\n else:\n root = os.path.join(config.DATA.DATA_PATH, prefix)\n dataset = datasets.ImageFolder(root, transform=transform)\n nb_classes = 1000\n elif config.DATA.DATASET == 'imagenet22K':\n prefix = 'ILSVRC2011fall_whole'\n if is_train:\n ann_file = prefix + \"_map_train.txt\"\n else:\n ann_file = prefix + \"_map_val.txt\"\n dataset = IN22KDATASET(config.DATA.DATA_PATH, ann_file, transform)\n nb_classes = 21841\n else:\n raise NotImplementedError(\"We only support ImageNet Now.\")\n\n return dataset, nb_classes\n\n\ndef build_transform(is_train, config):\n resize_im = config.DATA.IMG_SIZE > 32\n if is_train:\n # this should always dispatch to transforms_imagenet_train\n transform = create_transform(\n input_size=config.DATA.IMG_SIZE,\n is_training=True,\n color_jitter=config.AUG.COLOR_JITTER if config.AUG.COLOR_JITTER > 0 else None,\n auto_augment=config.AUG.AUTO_AUGMENT if config.AUG.AUTO_AUGMENT != 'none' else None,\n re_prob=config.AUG.REPROB,\n re_mode=config.AUG.REMODE,\n re_count=config.AUG.RECOUNT,\n interpolation=config.DATA.INTERPOLATION,\n )\n if not resize_im:\n # replace RandomResizedCropAndInterpolation with\n # RandomCrop\n transform.transforms[0] = transforms.RandomCrop(config.DATA.IMG_SIZE, padding=4)\n return transform\n\n t = []\n if resize_im:\n if config.TEST.CROP:\n size = int((256 / 224) * config.DATA.IMG_SIZE)\n t.append(\n transforms.Resize(size, interpolation=_pil_interp(config.DATA.INTERPOLATION)),\n # to maintain same ratio w.r.t. 224 images\n )\n t.append(transforms.CenterCrop(config.DATA.IMG_SIZE))\n else:\n t.append(\n transforms.Resize((config.DATA.IMG_SIZE, config.DATA.IMG_SIZE),\n interpolation=_pil_interp(config.DATA.INTERPOLATION))\n )\n\n t.append(transforms.ToTensor())\n t.append(transforms.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD))\n return transforms.Compose(t)\n"
},
{
"path": "data/cached_image_folder.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport io\nimport os\nimport time\nimport torch.distributed as dist\nimport torch.utils.data as data\nfrom PIL import Image\n\nfrom .zipreader import is_zip_path, ZipReader\n\n\ndef has_file_allowed_extension(filename, extensions):\n \"\"\"Checks if a file is an allowed extension.\n Args:\n filename (string): path to a file\n Returns:\n bool: True if the filename ends with a known image extension\n \"\"\"\n filename_lower = filename.lower()\n return any(filename_lower.endswith(ext) for ext in extensions)\n\n\ndef find_classes(dir):\n classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]\n classes.sort()\n class_to_idx = {classes[i]: i for i in range(len(classes))}\n return classes, class_to_idx\n\n\ndef make_dataset(dir, class_to_idx, extensions):\n images = []\n dir = os.path.expanduser(dir)\n for target in sorted(os.listdir(dir)):\n d = os.path.join(dir, target)\n if not os.path.isdir(d):\n continue\n\n for root, _, fnames in sorted(os.walk(d)):\n for fname in sorted(fnames):\n if has_file_allowed_extension(fname, extensions):\n path = os.path.join(root, fname)\n item = (path, class_to_idx[target])\n images.append(item)\n\n return images\n\n\ndef make_dataset_with_ann(ann_file, img_prefix, extensions):\n images = []\n with open(ann_file, \"r\") as f:\n contents = f.readlines()\n for line_str in contents:\n path_contents = [c for c in line_str.split('\\t')]\n im_file_name = path_contents[0]\n class_index = int(path_contents[1])\n\n assert str.lower(os.path.splitext(im_file_name)[-1]) in extensions\n item = (os.path.join(img_prefix, im_file_name), class_index)\n\n images.append(item)\n\n return images\n\n\nclass DatasetFolder(data.Dataset):\n \"\"\"A generic data loader where the samples are arranged in this way: ::\n root/class_x/xxx.ext\n root/class_x/xxy.ext\n root/class_x/xxz.ext\n root/class_y/123.ext\n root/class_y/nsdf3.ext\n root/class_y/asd932_.ext\n Args:\n root (string): Root directory path.\n loader (callable): A function to load a sample given its path.\n extensions (list[string]): A list of allowed extensions.\n transform (callable, optional): A function/transform that takes in\n a sample and returns a transformed version.\n E.g, ``transforms.RandomCrop`` for images.\n target_transform (callable, optional): A function/transform that takes\n in the target and transforms it.\n Attributes:\n samples (list): List of (sample path, class_index) tuples\n \"\"\"\n\n def __init__(self, root, loader, extensions, ann_file='', img_prefix='', transform=None, target_transform=None,\n cache_mode=\"no\"):\n # image folder mode\n if ann_file == '':\n _, class_to_idx = find_classes(root)\n samples = make_dataset(root, class_to_idx, extensions)\n # zip mode\n else:\n samples = make_dataset_with_ann(os.path.join(root, ann_file),\n os.path.join(root, img_prefix),\n extensions)\n\n if len(samples) == 0:\n raise (RuntimeError(\"Found 0 files in subfolders of: \" + root + \"\\n\" +\n \"Supported extensions are: \" + \",\".join(extensions)))\n\n self.root = root\n self.loader = loader\n self.extensions = extensions\n\n self.samples = samples\n self.labels = [y_1k for _, y_1k in samples]\n self.classes = list(set(self.labels))\n\n self.transform = transform\n self.target_transform = target_transform\n\n self.cache_mode = cache_mode\n if self.cache_mode != \"no\":\n self.init_cache()\n\n def init_cache(self):\n assert self.cache_mode in [\"part\", \"full\"]\n n_sample = len(self.samples)\n global_rank = dist.get_rank()\n world_size = dist.get_world_size()\n\n samples_bytes = [None for _ in range(n_sample)]\n start_time = time.time()\n for index in range(n_sample):\n if index % (n_sample // 10) == 0:\n t = time.time() - start_time\n print(f'global_rank {dist.get_rank()} cached {index}/{n_sample} takes {t:.2f}s per block')\n start_time = time.time()\n path, target = self.samples[index]\n if self.cache_mode == \"full\":\n samples_bytes[index] = (ZipReader.read(path), target)\n elif self.cache_mode == \"part\" and index % world_size == global_rank:\n samples_bytes[index] = (ZipReader.read(path), target)\n else:\n samples_bytes[index] = (path, target)\n self.samples = samples_bytes\n\n def __getitem__(self, index):\n \"\"\"\n Args:\n index (int): Index\n Returns:\n tuple: (sample, target) where target is class_index of the target class.\n \"\"\"\n path, target = self.samples[index]\n sample = self.loader(path)\n if self.transform is not None:\n sample = self.transform(sample)\n if self.target_transform is not None:\n target = self.target_transform(target)\n\n return sample, target\n\n def __len__(self):\n return len(self.samples)\n\n def __repr__(self):\n fmt_str = 'Dataset ' + self.__class__.__name__ + '\\n'\n fmt_str += ' Number of datapoints: {}\\n'.format(self.__len__())\n fmt_str += ' Root Location: {}\\n'.format(self.root)\n tmp = ' Transforms (if any): '\n fmt_str += '{0}{1}\\n'.format(tmp, self.transform.__repr__().replace('\\n', '\\n' + ' ' * len(tmp)))\n tmp = ' Target Transforms (if any): '\n fmt_str += '{0}{1}'.format(tmp, self.target_transform.__repr__().replace('\\n', '\\n' + ' ' * len(tmp)))\n return fmt_str\n\n\nIMG_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif']\n\n\ndef pil_loader(path):\n # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)\n if isinstance(path, bytes):\n img = Image.open(io.BytesIO(path))\n elif is_zip_path(path):\n data = ZipReader.read(path)\n img = Image.open(io.BytesIO(data))\n else:\n with open(path, 'rb') as f:\n img = Image.open(f)\n return img.convert('RGB')\n return img.convert('RGB')\n\n\ndef accimage_loader(path):\n import accimage\n try:\n return accimage.Image(path)\n except IOError:\n # Potentially a decoding problem, fall back to PIL.Image\n return pil_loader(path)\n\n\ndef default_img_loader(path):\n from torchvision import get_image_backend\n if get_image_backend() == 'accimage':\n return accimage_loader(path)\n else:\n return pil_loader(path)\n\n\nclass CachedImageFolder(DatasetFolder):\n \"\"\"A generic data loader where the images are arranged in this way: ::\n root/dog/xxx.png\n root/dog/xxy.png\n root/dog/xxz.png\n root/cat/123.png\n root/cat/nsdf3.png\n root/cat/asd932_.png\n Args:\n root (string): Root directory path.\n transform (callable, optional): A function/transform that takes in an PIL image\n and returns a transformed version. E.g, ``transforms.RandomCrop``\n target_transform (callable, optional): A function/transform that takes in the\n target and transforms it.\n loader (callable, optional): A function to load an image given its path.\n Attributes:\n imgs (list): List of (image path, class_index) tuples\n \"\"\"\n\n def __init__(self, root, ann_file='', img_prefix='', transform=None, target_transform=None,\n loader=default_img_loader, cache_mode=\"no\"):\n super(CachedImageFolder, self).__init__(root, loader, IMG_EXTENSIONS,\n ann_file=ann_file, img_prefix=img_prefix,\n transform=transform, target_transform=target_transform,\n cache_mode=cache_mode)\n self.imgs = self.samples\n\n def __getitem__(self, index):\n \"\"\"\n Args:\n index (int): Index\n Returns:\n tuple: (image, target) where target is class_index of the target class.\n \"\"\"\n path, target = self.samples[index]\n image = self.loader(path)\n if self.transform is not None:\n img = self.transform(image)\n else:\n img = image\n if self.target_transform is not None:\n target = self.target_transform(target)\n\n return img, target\n"
},
{
"path": "data/data_simmim_ft.py",
"content": "# --------------------------------------------------------\n# SimMIM\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Zhenda Xie\n# --------------------------------------------------------\n\nimport os\nimport torch.distributed as dist\nfrom torch.utils.data import DataLoader, DistributedSampler\nfrom torchvision import datasets, transforms\nfrom timm.data.constants import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD\nfrom timm.data import Mixup\nfrom timm.data import create_transform\nfrom timm.data.transforms import _pil_interp\n\n\ndef build_loader_finetune(config):\n config.defrost()\n dataset_train, config.MODEL.NUM_CLASSES = build_dataset(is_train=True, config=config)\n config.freeze()\n dataset_val, _ = build_dataset(is_train=False, config=config)\n\n num_tasks = dist.get_world_size()\n global_rank = dist.get_rank()\n sampler_train = DistributedSampler(\n dataset_train, num_replicas=num_tasks, rank=global_rank, shuffle=True\n )\n sampler_val = DistributedSampler(\n dataset_val, num_replicas=num_tasks, rank=global_rank, shuffle=False\n )\n\n data_loader_train = DataLoader(\n dataset_train, sampler=sampler_train,\n batch_size=config.DATA.BATCH_SIZE,\n num_workers=config.DATA.NUM_WORKERS,\n pin_memory=config.DATA.PIN_MEMORY,\n drop_last=True,\n )\n\n data_loader_val = DataLoader(\n dataset_val, sampler=sampler_val,\n batch_size=config.DATA.BATCH_SIZE,\n num_workers=config.DATA.NUM_WORKERS,\n pin_memory=config.DATA.PIN_MEMORY,\n drop_last=False,\n )\n\n # setup mixup / cutmix\n mixup_fn = None\n mixup_active = config.AUG.MIXUP > 0 or config.AUG.CUTMIX > 0. or config.AUG.CUTMIX_MINMAX is not None\n if mixup_active:\n mixup_fn = Mixup(\n mixup_alpha=config.AUG.MIXUP, cutmix_alpha=config.AUG.CUTMIX, cutmix_minmax=config.AUG.CUTMIX_MINMAX,\n prob=config.AUG.MIXUP_PROB, switch_prob=config.AUG.MIXUP_SWITCH_PROB, mode=config.AUG.MIXUP_MODE,\n label_smoothing=config.MODEL.LABEL_SMOOTHING, num_classes=config.MODEL.NUM_CLASSES)\n\n return dataset_train, dataset_val, data_loader_train, data_loader_val, mixup_fn\n\n\ndef build_dataset(is_train, config):\n transform = build_transform(is_train, config)\n \n if config.DATA.DATASET == 'imagenet':\n prefix = 'train' if is_train else 'val'\n root = os.path.join(config.DATA.DATA_PATH, prefix)\n dataset = datasets.ImageFolder(root, transform=transform)\n nb_classes = 1000\n else:\n raise NotImplementedError(\"We only support ImageNet Now.\")\n\n return dataset, nb_classes\n\n\ndef build_transform(is_train, config):\n resize_im = config.DATA.IMG_SIZE > 32\n if is_train:\n # this should always dispatch to transforms_imagenet_train\n transform = create_transform(\n input_size=config.DATA.IMG_SIZE,\n is_training=True,\n color_jitter=config.AUG.COLOR_JITTER if config.AUG.COLOR_JITTER > 0 else None,\n auto_augment=config.AUG.AUTO_AUGMENT if config.AUG.AUTO_AUGMENT != 'none' else None,\n re_prob=config.AUG.REPROB,\n re_mode=config.AUG.REMODE,\n re_count=config.AUG.RECOUNT,\n interpolation=config.DATA.INTERPOLATION,\n )\n if not resize_im:\n # replace RandomResizedCropAndInterpolation with\n # RandomCrop\n transform.transforms[0] = transforms.RandomCrop(config.DATA.IMG_SIZE, padding=4)\n return transform\n\n t = []\n if resize_im:\n if config.TEST.CROP:\n size = int((256 / 224) * config.DATA.IMG_SIZE)\n t.append(\n transforms.Resize(size, interpolation=_pil_interp(config.DATA.INTERPOLATION)),\n # to maintain same ratio w.r.t. 224 images\n )\n t.append(transforms.CenterCrop(config.DATA.IMG_SIZE))\n else:\n t.append(\n transforms.Resize((config.DATA.IMG_SIZE, config.DATA.IMG_SIZE),\n interpolation=_pil_interp(config.DATA.INTERPOLATION))\n )\n\n t.append(transforms.ToTensor())\n t.append(transforms.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD))\n return transforms.Compose(t)\n"
},
{
"path": "data/data_simmim_pt.py",
"content": "# --------------------------------------------------------\n# SimMIM\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Zhenda Xie\n# --------------------------------------------------------\n\nimport math\nimport random\nimport numpy as np\n\nimport torch\nimport torch.distributed as dist\nimport torchvision.transforms as T\nfrom torch.utils.data import DataLoader, DistributedSampler\nfrom torch.utils.data._utils.collate import default_collate\nfrom torchvision.datasets import ImageFolder\nfrom timm.data import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD\n\n\nclass MaskGenerator:\n def __init__(self, input_size=192, mask_patch_size=32, model_patch_size=4, mask_ratio=0.6):\n self.input_size = input_size\n self.mask_patch_size = mask_patch_size\n self.model_patch_size = model_patch_size\n self.mask_ratio = mask_ratio\n \n assert self.input_size % self.mask_patch_size == 0\n assert self.mask_patch_size % self.model_patch_size == 0\n \n self.rand_size = self.input_size // self.mask_patch_size\n self.scale = self.mask_patch_size // self.model_patch_size\n \n self.token_count = self.rand_size ** 2\n self.mask_count = int(np.ceil(self.token_count * self.mask_ratio))\n \n def __call__(self):\n mask_idx = np.random.permutation(self.token_count)[:self.mask_count]\n mask = np.zeros(self.token_count, dtype=int)\n mask[mask_idx] = 1\n \n mask = mask.reshape((self.rand_size, self.rand_size))\n mask = mask.repeat(self.scale, axis=0).repeat(self.scale, axis=1)\n \n return mask\n\n\nclass SimMIMTransform:\n def __init__(self, config):\n self.transform_img = T.Compose([\n T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),\n T.RandomResizedCrop(config.DATA.IMG_SIZE, scale=(0.67, 1.), ratio=(3. / 4., 4. / 3.)),\n T.RandomHorizontalFlip(),\n T.ToTensor(),\n T.Normalize(mean=torch.tensor(IMAGENET_DEFAULT_MEAN),std=torch.tensor(IMAGENET_DEFAULT_STD)),\n ])\n \n if config.MODEL.TYPE in ['swin', 'swinv2']:\n model_patch_size=config.MODEL.SWIN.PATCH_SIZE\n else:\n raise NotImplementedError\n \n self.mask_generator = MaskGenerator(\n input_size=config.DATA.IMG_SIZE,\n mask_patch_size=config.DATA.MASK_PATCH_SIZE,\n model_patch_size=model_patch_size,\n mask_ratio=config.DATA.MASK_RATIO,\n )\n \n def __call__(self, img):\n img = self.transform_img(img)\n mask = self.mask_generator()\n \n return img, mask\n\n\ndef collate_fn(batch):\n if not isinstance(batch[0][0], tuple):\n return default_collate(batch)\n else:\n batch_num = len(batch)\n ret = []\n for item_idx in range(len(batch[0][0])):\n if batch[0][0][item_idx] is None:\n ret.append(None)\n else:\n ret.append(default_collate([batch[i][0][item_idx] for i in range(batch_num)]))\n ret.append(default_collate([batch[i][1] for i in range(batch_num)]))\n return ret\n\n\ndef build_loader_simmim(config):\n transform = SimMIMTransform(config)\n dataset = ImageFolder(config.DATA.DATA_PATH, transform)\n \n sampler = DistributedSampler(dataset, num_replicas=dist.get_world_size(), rank=dist.get_rank(), shuffle=True)\n dataloader = DataLoader(dataset, config.DATA.BATCH_SIZE, sampler=sampler, num_workers=config.DATA.NUM_WORKERS, pin_memory=True, drop_last=True, collate_fn=collate_fn)\n \n return dataloader"
},
{
"path": "data/imagenet22k_dataset.py",
"content": "import os\nimport json\nimport torch.utils.data as data\nimport numpy as np\nfrom PIL import Image\n\nimport warnings\n\nwarnings.filterwarnings(\"ignore\", \"(Possibly )?corrupt EXIF data\", UserWarning)\n\n\nclass IN22KDATASET(data.Dataset):\n def __init__(self, root, ann_file='', transform=None, target_transform=None):\n super(IN22KDATASET, self).__init__()\n\n self.data_path = root\n self.ann_path = os.path.join(self.data_path, ann_file)\n self.transform = transform\n self.target_transform = target_transform\n # id & label: https://github.com/google-research/big_transfer/issues/7\n # total: 21843; only 21841 class have images: map 21841->9205; 21842->15027\n self.database = json.load(open(self.ann_path))\n\n def _load_image(self, path):\n try:\n im = Image.open(path)\n except:\n print(\"ERROR IMG LOADED: \", path)\n random_img = np.random.rand(224, 224, 3) * 255\n im = Image.fromarray(np.uint8(random_img))\n return im\n\n def __getitem__(self, index):\n \"\"\"\n Args:\n index (int): Index\n Returns:\n tuple: (image, target) where target is class_index of the target class.\n \"\"\"\n idb = self.database[index]\n\n # images\n images = self._load_image(self.data_path + '/' + idb[0]).convert('RGB')\n if self.transform is not None:\n images = self.transform(images)\n\n # target\n target = int(idb[1])\n if self.target_transform is not None:\n target = self.target_transform(target)\n\n return images, target\n\n def __len__(self):\n return len(self.database)\n"
},
{
"path": "data/map22kto1k.txt",
"content": "359\n368\n460\n475\n486\n492\n496\n514\n516\n525\n547\n548\n556\n563\n575\n641\n648\n723\n733\n765\n801\n826\n852\n858\n878\n896\n900\n905\n908\n910\n935\n946\n947\n994\n999\n1003\n1005\n1010\n1027\n1029\n1048\n1055\n1064\n1065\n1069\n1075\n1079\n1081\n1085\n1088\n1093\n1106\n1143\n1144\n1145\n1147\n1168\n1171\n1178\n1187\n1190\n1197\n1205\n1216\n1223\n1230\n1236\n1241\n1245\n1257\n1259\n1260\n1267\n1268\n1269\n1271\n1272\n1273\n1277\n1303\n1344\n1349\n1355\n1357\n1384\n1388\n1391\n1427\n1429\n1432\n1437\n1450\n1461\n1462\n1474\n1502\n1503\n1512\n1552\n1555\n1577\n1584\n1587\n1589\n1599\n1615\n1616\n1681\n1692\n1701\n1716\n1729\n1757\n1759\n1764\n1777\n1786\n1822\n1841\n1842\n1848\n1850\n1856\n1860\n1861\n1864\n1876\n1897\n1898\n1910\n1913\n1918\n1922\n1928\n1932\n1935\n1947\n1951\n1953\n1970\n1977\n1979\n2001\n2017\n2067\n2081\n2087\n2112\n2128\n2135\n2147\n2174\n2175\n2176\n2177\n2178\n2181\n2183\n2184\n2187\n2189\n2190\n2191\n2192\n2193\n2197\n2202\n2203\n2206\n2208\n2209\n2211\n2212\n2213\n2214\n2215\n2216\n2217\n2219\n2222\n2223\n2224\n2225\n2226\n2227\n2228\n2229\n2230\n2236\n2238\n2240\n2241\n2242\n2243\n2244\n2245\n2247\n2248\n2249\n2250\n2251\n2252\n2255\n2256\n2257\n2262\n2263\n2264\n2265\n2266\n2268\n2270\n2271\n2272\n2273\n2275\n2276\n2279\n2280\n2281\n2282\n2285\n2289\n2292\n2295\n2296\n2297\n2298\n2299\n2300\n2301\n2302\n2303\n2304\n2305\n2306\n2309\n2310\n2312\n2313\n2314\n2315\n2316\n2318\n2319\n2321\n2322\n2326\n2329\n2330\n2331\n2332\n2334\n2335\n2336\n2337\n2338\n2339\n2341\n2342\n2343\n2344\n2346\n2348\n2349\n2351\n2352\n2353\n2355\n2357\n2358\n2359\n2360\n2364\n2365\n2368\n2369\n2377\n2382\n2383\n2385\n2397\n2398\n2400\n2402\n2405\n2412\n2421\n2428\n2431\n2432\n2433\n2436\n2441\n2445\n2450\n2453\n2454\n2465\n2469\n2532\n2533\n2538\n2544\n2547\n2557\n2565\n2578\n2612\n2658\n2702\n2722\n2731\n2738\n2741\n2747\n2810\n2818\n2833\n2844\n2845\n2867\n2874\n2882\n2884\n2888\n2889\n3008\n3012\n3019\n3029\n3033\n3042\n3091\n3106\n3138\n3159\n3164\n3169\n3280\n3296\n3311\n3318\n3320\n3324\n3330\n3366\n3375\n3381\n3406\n3419\n3432\n3434\n3435\n3493\n3495\n3503\n3509\n3511\n3513\n3517\n3521\n3526\n3546\n3554\n3600\n3601\n3606\n3612\n3613\n3616\n3622\n3623\n3627\n3632\n3634\n3636\n3638\n3644\n3646\n3649\n3650\n3651\n3656\n3663\n3673\n3674\n3689\n3690\n3702\n3733\n3769\n3971\n3974\n4065\n4068\n4073\n4102\n4136\n4140\n4151\n4159\n4165\n4207\n4219\n4226\n4249\n4256\n4263\n4270\n4313\n4321\n4378\n4386\n4478\n4508\n4512\n4536\n4542\n4550\n4560\n4562\n4570\n4571\n4572\n4583\n4588\n4594\n4604\n4608\n4623\n4634\n4636\n4646\n4651\n4652\n4686\n4688\n4691\n4699\n4724\n4727\n4737\n4770\n4774\n4789\n4802\n4807\n4819\n4880\n4886\n4908\n4927\n4931\n4936\n4964\n4976\n4993\n5028\n5033\n5043\n5046\n5096\n5111\n5114\n5131\n5132\n5183\n5199\n5235\n5275\n5291\n5293\n5294\n5343\n5360\n5362\n5364\n5390\n5402\n5418\n5428\n5430\n5437\n5443\n5473\n5484\n5486\n5505\n5507\n5508\n5510\n5567\n5578\n5580\n5584\n5606\n5613\n5629\n5672\n5676\n5692\n5701\n5760\n5769\n5770\n5779\n5814\n5850\n5871\n5893\n5911\n5949\n5954\n6005\n6006\n6012\n6017\n6023\n6024\n6040\n6050\n6054\n6087\n6105\n6157\n6235\n6237\n6256\n6259\n6286\n6291\n6306\n6339\n6341\n6343\n6379\n6383\n6393\n6405\n6479\n6511\n6517\n6541\n6561\n6608\n6611\n6615\n6678\n6682\n6707\n6752\n6798\n6850\n6880\n6885\n6890\n6920\n6981\n7000\n7009\n7038\n7049\n7050\n7052\n7073\n7078\n7098\n7111\n7165\n7198\n7204\n7280\n7283\n7286\n7287\n7293\n7294\n7305\n7318\n7341\n7346\n7354\n7382\n7427\n7428\n7435\n7445\n7450\n7455\n7467\n7469\n7497\n7502\n7506\n7514\n7523\n7651\n7661\n7664\n7672\n7679\n7685\n7696\n7730\n7871\n7873\n7895\n7914\n7915\n7920\n7934\n7935\n7949\n8009\n8036\n8051\n8065\n8074\n8090\n8112\n8140\n8164\n8168\n8178\n8182\n8198\n8212\n8216\n8230\n8242\n8288\n8289\n8295\n8318\n8352\n8368\n8371\n8375\n8376\n8401\n8416\n8419\n8436\n8460\n8477\n8478\n8482\n8498\n8500\n8539\n8543\n8552\n8555\n8580\n8584\n8586\n8594\n8598\n8601\n8606\n8610\n8611\n8622\n8627\n8639\n8649\n8650\n8653\n8654\n8667\n8672\n8673\n8674\n8676\n8684\n8720\n8723\n8750\n8753\n8801\n8815\n8831\n8835\n8842\n8845\n8858\n8897\n8916\n8951\n8954\n8959\n8970\n8976\n8981\n8983\n8989\n8991\n8993\n9019\n9039\n9042\n9043\n9056\n9057\n9070\n9087\n9098\n9106\n9130\n9131\n9155\n9171\n9183\n9198\n9199\n9201\n9204\n9212\n9221\n9225\n9229\n9250\n9260\n9271\n9279\n9295\n9300\n9310\n9322\n9345\n9352\n9376\n9377\n9382\n9392\n9401\n9405\n9441\n9449\n9464\n9475\n9502\n9505\n9514\n9515\n9545\n9567\n9576\n9608\n9609\n9624\n9633\n9639\n9643\n9656\n9674\n9740\n9752\n9760\n9767\n9778\n9802\n9820\n9839\n9879\n9924\n9956\n9961\n9963\n9970\n9997\n10010\n10031\n10040\n10052\n10073\n10075\n10078\n10094\n10097\n10109\n10118\n10121\n10124\n10158\n10226\n10276\n10304\n10307\n10314\n10315\n10332\n10337\n10338\n10413\n10423\n10451\n10463\n10465\n10487\n10519\n10522\n10523\n10532\n10534\n10535\n10551\n10559\n10574\n10583\n10586\n10589\n10612\n10626\n10635\n10638\n10677\n10683\n10726\n10776\n10782\n10783\n10807\n10837\n10840\n10848\n10859\n10871\n10881\n10884\n10908\n10914\n10921\n10936\n10947\n10951\n10952\n10957\n10999\n11003\n11018\n11023\n11025\n11027\n11045\n11055\n11095\n11110\n11137\n5564\n11168\n11186\n11221\n11223\n11242\n11255\n11259\n11279\n11306\n11311\n11331\n11367\n11377\n11389\n11392\n11401\n11407\n11437\n11449\n11466\n11469\n11473\n11478\n11483\n11484\n11507\n11536\n11558\n11566\n11575\n11584\n11594\n11611\n11612\n11619\n11621\n11640\n11643\n11664\n11674\n11689\n11709\n11710\n11716\n11721\n11726\n11729\n11743\n11760\n11771\n11837\n11839\n11856\n11876\n11878\n11884\n11889\n11896\n11917\n11923\n11930\n11944\n11952\n11980\n11984\n12214\n12229\n12239\n12241\n12242\n12247\n12283\n12349\n12369\n12373\n12422\n12560\n12566\n12575\n12688\n12755\n12768\n12778\n12780\n12812\n12832\n12835\n12836\n12843\n12847\n12849\n12850\n12856\n12858\n12873\n12938\n12971\n13017\n13038\n13046\n13059\n13085\n13086\n13088\n13094\n13134\n13182\n13230\n13406\n13444\n13614\n13690\n13698\n13709\n13749\n13804\n13982\n14051\n14059\n14219\n14246\n14256\n14264\n14294\n14324\n14367\n14389\n14394\n14438\n14442\n14965\n15732\n16744\n18037\n18205\n18535\n18792\n19102\n20019\n20462\n21026\n21045\n21163\n21171\n21181\n21196\n21200\n21369\n21817"
},
{
"path": "data/samplers.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport torch\n\n\nclass SubsetRandomSampler(torch.utils.data.Sampler):\n r\"\"\"Samples elements randomly from a given list of indices, without replacement.\n\n Arguments:\n indices (sequence): a sequence of indices\n \"\"\"\n\n def __init__(self, indices):\n self.epoch = 0\n self.indices = indices\n\n def __iter__(self):\n return (self.indices[i] for i in torch.randperm(len(self.indices)))\n\n def __len__(self):\n return len(self.indices)\n\n def set_epoch(self, epoch):\n self.epoch = epoch\n"
},
{
"path": "data/zipreader.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport os\nimport zipfile\nimport io\nimport numpy as np\nfrom PIL import Image\nfrom PIL import ImageFile\n\nImageFile.LOAD_TRUNCATED_IMAGES = True\n\n\ndef is_zip_path(img_or_path):\n \"\"\"judge if this is a zip path\"\"\"\n return '.zip@' in img_or_path\n\n\nclass ZipReader(object):\n \"\"\"A class to read zipped files\"\"\"\n zip_bank = dict()\n\n def __init__(self):\n super(ZipReader, self).__init__()\n\n @staticmethod\n def get_zipfile(path):\n zip_bank = ZipReader.zip_bank\n if path not in zip_bank:\n zfile = zipfile.ZipFile(path, 'r')\n zip_bank[path] = zfile\n return zip_bank[path]\n\n @staticmethod\n def split_zip_style_path(path):\n pos_at = path.index('@')\n assert pos_at != -1, \"character '@' is not found from the given path '%s'\" % path\n\n zip_path = path[0: pos_at]\n folder_path = path[pos_at + 1:]\n folder_path = str.strip(folder_path, '/')\n return zip_path, folder_path\n\n @staticmethod\n def list_folder(path):\n zip_path, folder_path = ZipReader.split_zip_style_path(path)\n\n zfile = ZipReader.get_zipfile(zip_path)\n folder_list = []\n for file_foler_name in zfile.namelist():\n file_foler_name = str.strip(file_foler_name, '/')\n if file_foler_name.startswith(folder_path) and \\\n len(os.path.splitext(file_foler_name)[-1]) == 0 and \\\n file_foler_name != folder_path:\n if len(folder_path) == 0:\n folder_list.append(file_foler_name)\n else:\n folder_list.append(file_foler_name[len(folder_path) + 1:])\n\n return folder_list\n\n @staticmethod\n def list_files(path, extension=None):\n if extension is None:\n extension = ['.*']\n zip_path, folder_path = ZipReader.split_zip_style_path(path)\n\n zfile = ZipReader.get_zipfile(zip_path)\n file_lists = []\n for file_foler_name in zfile.namelist():\n file_foler_name = str.strip(file_foler_name, '/')\n if file_foler_name.startswith(folder_path) and \\\n str.lower(os.path.splitext(file_foler_name)[-1]) in extension:\n if len(folder_path) == 0:\n file_lists.append(file_foler_name)\n else:\n file_lists.append(file_foler_name[len(folder_path) + 1:])\n\n return file_lists\n\n @staticmethod\n def read(path):\n zip_path, path_img = ZipReader.split_zip_style_path(path)\n zfile = ZipReader.get_zipfile(zip_path)\n data = zfile.read(path_img)\n return data\n\n @staticmethod\n def imread(path):\n zip_path, path_img = ZipReader.split_zip_style_path(path)\n zfile = ZipReader.get_zipfile(zip_path)\n data = zfile.read(path_img)\n try:\n im = Image.open(io.BytesIO(data))\n except:\n print(\"ERROR IMG LOADED: \", path_img)\n random_img = np.random.rand(224, 224, 3) * 255\n im = Image.fromarray(np.uint8(random_img))\n return im\n"
},
{
"path": "get_started.md",
"content": "# Swin Transformer for Image Classification\n\nThis folder contains the implementation of the Swin Transformer for image classification.\n\n## Model Zoo\n\nPlease refer to [MODEL HUB](MODELHUB.md#imagenet-22k-pretrained-swin-moe-models) for more pre-trained models.\n\n## Usage\n\n### Install\n\nWe recommend using the pytorch docker `nvcr>=21.05` by\nnvidia: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch.\n\n- Clone this repo:\n\n```bash\ngit clone https://github.com/microsoft/Swin-Transformer.git\ncd Swin-Transformer\n```\n\n- Create a conda virtual environment and activate it:\n\n```bash\nconda create -n swin python=3.7 -y\nconda activate swin\n```\n\n- Install `CUDA>=10.2` with `cudnn>=7` following\n the [official installation instructions](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)\n- Install `PyTorch>=1.8.0` and `torchvision>=0.9.0` with `CUDA>=10.2`:\n\n```bash\nconda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch\n```\n\n- Install `timm==0.4.12`:\n\n```bash\npip install timm==0.4.12\n```\n\n- Install other requirements:\n\n```bash\npip install opencv-python==4.4.0.46 termcolor==1.1.0 yacs==0.1.8 pyyaml scipy\n```\n\n- Install fused window process for acceleration, activated by passing `--fused_window_process` in the running script\n```bash\ncd kernels/window_process\npython setup.py install #--user\n```\n\n### Data preparation\n\nWe use standard ImageNet dataset, you can download it from http://image-net.org/. We provide the following two ways to\nload data:\n\n- For standard folder dataset, move validation images to labeled sub-folders. The file structure should look like:\n ```bash\n $ tree data\n imagenet\n ├── train\n │ ├── class1\n │ │ ├── img1.jpeg\n │ │ ├── img2.jpeg\n │ │ └── ...\n │ ├── class2\n │ │ ├── img3.jpeg\n │ │ └── ...\n │ └── ...\n └── val\n ├── class1\n │ ├── img4.jpeg\n │ ├── img5.jpeg\n │ └── ...\n ├── class2\n │ ├── img6.jpeg\n │ └── ...\n └── ...\n \n ```\n- To boost the slow speed when reading images from massive small files, we also support zipped ImageNet, which includes\n four files:\n - `train.zip`, `val.zip`: which store the zipped folder for train and validate splits.\n - `train_map.txt`, `val_map.txt`: which store the relative path in the corresponding zip file and ground truth\n label. Make sure the data folder looks like this:\n\n ```bash\n $ tree data\n data\n └── ImageNet-Zip\n ├── train_map.txt\n ├── train.zip\n ├── val_map.txt\n └── val.zip\n \n $ head -n 5 data/ImageNet-Zip/val_map.txt\n ILSVRC2012_val_00000001.JPEG\t65\n ILSVRC2012_val_00000002.JPEG\t970\n ILSVRC2012_val_00000003.JPEG\t230\n ILSVRC2012_val_00000004.JPEG\t809\n ILSVRC2012_val_00000005.JPEG\t516\n \n $ head -n 5 data/ImageNet-Zip/train_map.txt\n n01440764/n01440764_10026.JPEG\t0\n n01440764/n01440764_10027.JPEG\t0\n n01440764/n01440764_10029.JPEG\t0\n n01440764/n01440764_10040.JPEG\t0\n n01440764/n01440764_10042.JPEG\t0\n ```\n- For ImageNet-22K dataset, make a folder named `fall11_whole` and move all images to labeled sub-folders in this\n folder. Then download the train-val split\n file ([ILSVRC2011fall_whole_map_train.txt](https://github.com/SwinTransformer/storage/releases/download/v2.0.1/ILSVRC2011fall_whole_map_train.txt)\n & [ILSVRC2011fall_whole_map_val.txt](https://github.com/SwinTransformer/storage/releases/download/v2.0.1/ILSVRC2011fall_whole_map_val.txt))\n , and put them in the parent directory of `fall11_whole`. The file structure should look like:\n\n ```bash\n $ tree imagenet22k/\n imagenet22k/\n ├── ILSVRC2011fall_whole_map_train.txt\n ├── ILSVRC2011fall_whole_map_val.txt\n └── fall11_whole\n ├── n00004475\n ├── n00005787\n ├── n00006024\n ├── n00006484\n └── ...\n ```\n\n### Evaluation\n\nTo evaluate a pre-trained `Swin Transformer` on ImageNet val, run:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node --master_port 12345 main.py --eval \\\n--cfg --resume --data-path \n```\n\nFor example, to evaluate the `Swin-B` with a single GPU:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 1 --master_port 12345 main.py --eval \\\n--cfg configs/swin/swin_base_patch4_window7_224.yaml --resume swin_base_patch4_window7_224.pth --data-path \n```\n\n### Training from scratch on ImageNet-1K\n\nTo train a `Swin Transformer` on ImageNet from scratch, run:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node --master_port 12345 main.py \\ \n--cfg --data-path [--batch-size --output --tag ]\n```\n\n**Notes**:\n\n- To use zipped ImageNet instead of folder dataset, add `--zip` to the parameters.\n - To cache the dataset in the memory instead of reading from files every time, add `--cache-mode part`, which will\n shard the dataset into non-overlapping pieces for different GPUs and only load the corresponding one for each GPU.\n- When GPU memory is not enough, you can try the following suggestions:\n - Use gradient accumulation by adding `--accumulation-steps `, set appropriate `` according to your need.\n - Use gradient checkpointing by adding `--use-checkpoint`, e.g., it saves about 60% memory when training `Swin-B`.\n Please refer to [this page](https://pytorch.org/docs/stable/checkpoint.html) for more details.\n - We recommend using multi-node with more GPUs for training very large models, a tutorial can be found\n in [this page](https://pytorch.org/tutorials/intermediate/dist_tuto.html).\n- To change config options in general, you can use `--opts KEY1 VALUE1 KEY2 VALUE2`, e.g.,\n `--opts TRAIN.EPOCHS 100 TRAIN.WARMUP_EPOCHS 5` will change total epochs to 100 and warm-up epochs to 5.\n- For additional options, see [config](config.py) and run `python main.py --help` to get detailed message.\n\nFor example, to train `Swin Transformer` with 8 GPU on a single node for 300 epochs, run:\n\n`Swin-T`:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_tiny_patch4_window7_224.yaml --data-path --batch-size 128 \n```\n\n`Swin-S`:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_small_patch4_window7_224.yaml --data-path --batch-size 128 \n```\n\n`Swin-B`:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_base_patch4_window7_224.yaml --data-path --batch-size 64 \\\n--accumulation-steps 2 [--use-checkpoint]\n```\n\n### Pre-training on ImageNet-22K\n\nFor example, to pre-train a `Swin-B` model on ImageNet-22K:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_base_patch4_window7_224_22k.yaml --data-path --batch-size 64 \\\n--accumulation-steps 8 [--use-checkpoint]\n```\n\n### Fine-tuning on higher resolution\n\nFor example, to fine-tune a `Swin-B` model pre-trained on 224x224 resolution to 384x384 resolution:\n\n```bashs\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_base_patch4_window12_384_finetune.yaml --pretrained swin_base_patch4_window7_224.pth \\\n--data-path --batch-size 64 --accumulation-steps 2 [--use-checkpoint]\n```\n\n### Fine-tuning from a ImageNet-22K(21K) pre-trained model\n\nFor example, to fine-tune a `Swin-B` model pre-trained on ImageNet-22K(21K):\n\n```bashs\npython -m torch.distributed.launch --nproc_per_node 8 --master_port 12345 main.py \\\n--cfg configs/swin/swin_base_patch4_window7_224_22kto1k_finetune.yaml --pretrained swin_base_patch4_window7_224_22k.pth \\\n--data-path --batch-size 64 --accumulation-steps 2 [--use-checkpoint]\n```\n\n### Throughput\n\nTo measure the throughput, run:\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 1 --master_port 12345 main.py \\\n--cfg --data-path --batch-size 64 --throughput --disable_amp\n```\n\n\n## Mixture-of-Experts Support\n\n### Install [Tutel](https://github.com/microsoft/tutel)\n```bash\npython3 -m pip uninstall tutel -y \npython3 -m pip install --user --upgrade git+https://github.com/microsoft/tutel@main\n```\n\n### Training Swin-MoE \nFor example, to train a `Swin-MoE-S` model with 32 experts on ImageNet-22K with 32 GPUs (4 nodes):\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --nnode=4 \\\n--node_rank= --master_addr= --master_port 12345 main_moe.py \\\n--cfg configs/swinmoe/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.yaml --data-path --batch-size 128\n```\n\n### Evaluating Swin-MoE\n\nTo evaluate a `Swin-MoE-S` with 32 experts on ImageNet-22K with 32 GPUs (4 nodes):\n\n1. Download the zip file [swin_moe_small_patch4_window12_192_32expert_32gpu_22k.zip](https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.zip) which contains the pre-trained models for each rank, and unzip them to the folder \"swin_moe_small_patch4_window12_192_32expert_32gpu_22k\".\n2. Run the following evaluation command, note the checkpoint path should not contain the \".rank\\\" suffix.\n\n```bash\npython -m torch.distributed.launch --nproc_per_node 8 --nnode=4 \\\n--node_rank= --master_addr= --master_port 12345 main_moe.py \\\n--cfg configs/swinmoe/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.yaml --data-path --batch-size 128 \\\n--resume swin_moe_small_patch4_window12_192_32expert_32gpu_22k/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.pth \n```\n\nMore Swin-MoE models can be found in [MODEL HUB](MODELHUB.md#imagenet-22k-pretrained-swin-moe-models)\n\n## SimMIM Support\n\n### Evaluating provided models\n\nTo evaluate a provided model on ImageNet validation set, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node main_simmim_ft.py \\\n--eval --cfg --resume --data-path \n```\n\nFor example, to evaluate the `Swin Base` model on a single GPU, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node 1 main_simmim_ft.py \\\n--eval --cfg configs/simmim/simmim_finetune__swin_base__img224_window7__800ep.yaml --resume simmim_finetune__swin_base__img224_window7__800ep.pth --data-path \n```\n\n### Pre-training with SimMIM\nTo pre-train models with `SimMIM`, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node main_simmim_pt.py \\ \n--cfg --data-path /train [--batch-size --output --tag ]\n```\n\nFor example, to pre-train `Swin Base` for 800 epochs on one DGX-2 server, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node 16 main_simmim_pt.py \\ \n--cfg configs/simmim/simmim_pretrain__swin_base__img192_window6__800ep.yaml --batch-size 128 --data-path /train [--output --tag ]\n```\n\n### Fine-tuning pre-trained models\nTo fine-tune models pre-trained by `SimMIM`, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node main_simmim_ft.py \\ \n--cfg --data-path --pretrained [--batch-size --output --tag ]\n```\n\nFor example, to fine-tune `Swin Base` pre-trained by `SimMIM` on one DGX-2 server, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node 16 main_simmim_ft.py \\ \n--cfg configs/simmim/simmim_finetune__swin_base__img224_window7__800ep.yaml --batch-size 128 --data-path --pretrained [--output --tag ]\n```"
},
{
"path": "kernels/window_process/setup.py",
"content": "from setuptools import setup\nfrom torch.utils.cpp_extension import BuildExtension, CUDAExtension\n\n\nsetup(name='swin_window_process',\n ext_modules=[\n CUDAExtension('swin_window_process', [\n 'swin_window_process.cpp',\n 'swin_window_process_kernel.cu',\n ])\n ],\n cmdclass={'build_ext': BuildExtension})"
},
{
"path": "kernels/window_process/swin_window_process.cpp",
"content": "/*\n * Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\n#include \n#include \n\n\nat::Tensor roll_and_window_partition_forward_cuda(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size);\n\n\nat::Tensor roll_and_window_partition_backward_cuda(\n at::Tensor & grad_in, \n //at::Tensor & grad_out,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size);\n\n\nat::Tensor window_merge_and_roll_forward_cuda(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size);\n\nat::Tensor window_merge_and_roll_backward_cuda(\n at::Tensor & grad_in, \n //at::Tensor & grad_out,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size);\n\n\n#define CHECK_CUDA(x) AT_ASSERTM(x.type().is_cuda(), #x \" must be a CUDA tensor\")\n#define CHECK_CONTIGUOUS(x) AT_ASSERTM(x.is_contiguous(), #x \" must be contiguous\")\n#define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)\n\n\n\nat::Tensor roll_and_window_partition_forward(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n CHECK_INPUT(input);\n return roll_and_window_partition_forward_cuda(input, B, H, W, C, shift_size, window_size);\n}\n\n\nat::Tensor roll_and_window_partition_backward(\n at::Tensor & grad_in, \n //at::Tensor & grad_out,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n CHECK_INPUT(grad_in);\n return roll_and_window_partition_backward_cuda(grad_in, B, H, W, C, shift_size, window_size);\n}\n\n\nat::Tensor window_merge_and_roll_forward(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n CHECK_INPUT(input);\n return window_merge_and_roll_forward_cuda(input, B, H, W, C, shift_size, window_size);\n}\n\n\nat::Tensor window_merge_and_roll_backward(\n at::Tensor & grad_in, \n //at::Tensor & grad_out,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n CHECK_INPUT(grad_in);\n return window_merge_and_roll_backward_cuda(grad_in, B, H, W, C, shift_size, window_size);\n}\n\n\n\nPYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {\n m.def(\"roll_and_window_partition_forward\", &roll_and_window_partition_forward, \"torch.roll and window_partition.\");\n m.def(\"roll_and_window_partition_backward\", &roll_and_window_partition_backward, \"torch.roll and window_partition.\");\n m.def(\"window_merge_and_roll_forward\", &window_merge_and_roll_forward, \"window merge and torch.roll.\");\n m.def(\"window_merge_and_roll_backward\", &window_merge_and_roll_backward, \"window merge and torch.roll.\");\n}"
},
{
"path": "kernels/window_process/swin_window_process_kernel.cu",
"content": "/*\n * Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\n#include \n#include \n#include \n#include \n#include \n#include \n\nint best_block_dim(int feat_dim){\n int best_dim;\n if (feat_dim < 384){\n best_dim = 64;\n }\n else{\n if (feat_dim < 1024){\n best_dim = 128;\n }\n else{\n best_dim = 256;\n }\n }\n return best_dim;\n}\n\n\ntemplate \n__global__ void roll_and_window_partition_forward_cuda_kernel(\n T* input, \n T* output, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size,\n const int nH,\n const int nW){\n // start\n //bool qual = threadIdx.x < C;\n int index = threadIdx.x;\n int offset;\n for (int i = index; i < C; i += blockDim.x) {\n offset = ((blockIdx.z * gridDim.y + blockIdx.y) * gridDim.x + blockIdx.x) * C + i; // C = blocksize\n int input_offset = blockIdx.z / (nH * nW) * H * W * C +\n (blockIdx.z % (nH * nW) / nW * window_size + blockIdx.y - shift_size + H) % H * W * C + \n (blockIdx.z % nW * window_size + blockIdx.x - shift_size + W) % W * C +\n i;\n output[offset] = (T)(__ldg(input + input_offset));\n }\n}\n\n\ntemplate \n__global__ void roll_and_window_partition_backward_cuda_kernel(\n T* grad_in, \n T* grad_out, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size,\n const int nH,\n const int nW){\n // start\n int index = threadIdx.x;\n int offset;\n for (int i = index; i < C; i += blockDim.x) {\n offset = ((blockIdx.z * gridDim.y + blockIdx.y) * gridDim.x + blockIdx.x) * C + i; // C = blocksize\n int input_offset = \n (blockIdx.z * nH * nW + (blockIdx.y + shift_size + H) % H / window_size * nW + (blockIdx.x + shift_size + W) % W / window_size) * window_size * window_size * C +\n (blockIdx.y + shift_size + H ) % H % window_size * window_size * C +\n (blockIdx.x + shift_size + W ) % W % window_size * C +\n i;\n grad_out[offset] = (T)(__ldg(grad_in + input_offset));\n }\n}\n\n\ntemplate \n__global__ void window_merge_and_roll_forward_cuda_kernel(\n T* input, \n T* output, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size,\n const int nH,\n const int nW){\n // start\n int index = threadIdx.x;\n int offset;\n for (int i = index; i < C; i += blockDim.x) {\n offset = ((blockIdx.z * gridDim.y + blockIdx.y) * gridDim.x + blockIdx.x) * C + i; // C = blocksize\n int input_offset = \n (blockIdx.z * nH * nW + (blockIdx.y - shift_size + H) % H / window_size * nH + (blockIdx.x - shift_size + W) % W / window_size) * window_size * window_size * C +\n (blockIdx.y - shift_size + H) % window_size * window_size * C + \n (blockIdx.x - shift_size + W) % window_size * C +\n i;\n output[offset] = (T)(__ldg(input + input_offset));\n }\n}\n\n\n\ntemplate \n__global__ void window_merge_and_roll_backward_cuda_kernel(\n T* grad_in, \n T* grad_out, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size,\n const int nH,\n const int nW){\n // start\n int index = threadIdx.x;\n int offset;\n for (int i = index; i < C; i += blockDim.x) {\n offset = ((blockIdx.z * gridDim.y + blockIdx.y) * gridDim.x + blockIdx.x) * C + i; // C = blocksize\n int input_offset = \n (blockIdx.z / (nH * nW)) * H * W * C +\n (blockIdx.z % (nH * nW) / nW * window_size + blockIdx.y + shift_size + H) % H * W * C +\n (blockIdx.z % nW * window_size + blockIdx.x + shift_size + W) % W * C +\n i;\n grad_out[offset] = (T)(__ldg(grad_in + input_offset));\n }\n}\n\n// input: [B, H, W, C]\n// output: [B*nH*nW, window_size, window_size, C]\nat::Tensor roll_and_window_partition_forward_cuda(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n \n int nH = H / window_size;\n int nW = W / window_size;\n\n dim3 grid(window_size, window_size, B * nH * nW);\n //dim3 block((C + 31) / 32 * 32);\n int blocknum = best_block_dim(C);\n dim3 block(blocknum);\n\n at::Tensor output;\n if (input.scalar_type() == torch::kFloat16){\n output = torch::empty({B*nH*nW, window_size, window_size, C}, torch::dtype(torch::kFloat16).device(torch::kCUDA).requires_grad(true));\n }\n else{\n output = torch::empty({B*nH*nW, window_size, window_size, C}, torch::dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(true));\n }\n\n AT_DISPATCH_FLOATING_TYPES_AND_HALF(input.type(), \"roll_and_window_partition_forward_cuda_kernel\", ([&] {\n roll_and_window_partition_forward_cuda_kernel<<>>(\n input.data(),\n output.data(),\n B,\n H,\n W,\n C,\n shift_size,\n window_size,\n nH,\n nW);\n }));\n return output;\n}\n\n\n// grad_in: [B*nH*nW, window_size, window_size, C]\n// grad_out: [B, H, W, C]\nat::Tensor roll_and_window_partition_backward_cuda(\n at::Tensor & grad_in, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n \n int nH = H / window_size;\n int nW = W / window_size;\n\n dim3 grid(W, H, B);\n //dim3 block((C + 31) / 32 * 32);\n int blocknum = best_block_dim(C);\n dim3 block(blocknum);\n\n at::Tensor grad_out;\n if (grad_in.scalar_type() == torch::kFloat16){\n grad_out = torch::empty({B, H, W, C}, torch::dtype(torch::kFloat16).device(torch::kCUDA).requires_grad(false));\n }\n else{\n grad_out = torch::empty({B, H, W, C}, torch::dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(false));\n }\n\n AT_DISPATCH_FLOATING_TYPES_AND_HALF(grad_in.type(), \"roll_and_window_partition_backward_cuda_kernel\", ([&] {\n roll_and_window_partition_backward_cuda_kernel<<>>(\n grad_in.data(),\n grad_out.data(),\n B,\n H,\n W,\n C,\n shift_size,\n window_size,\n nH,\n nW);\n }));\n return grad_out;\n}\n\n\n// input: [B*nH*nW, window_size, window_size, C]\n// output: [B, H, W, C]\nat::Tensor window_merge_and_roll_forward_cuda(\n at::Tensor & input, \n //at::Tensor & output,\n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n \n int nH = H / window_size;\n int nW = W / window_size;\n\n dim3 grid(W, H, B);\n //dim3 block((C + 31) / 32 * 32);\n int blocknum = best_block_dim(C);\n dim3 block(blocknum);\n\n //generate output tensor inside\n at::Tensor output;\n if (input.scalar_type() == torch::kFloat16){\n output = torch::empty({B, H, W, C}, torch::dtype(torch::kFloat16).device(torch::kCUDA).requires_grad(true));\n }\n else{\n output = torch::empty({B, H, W, C}, torch::dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(true));\n }\n\n AT_DISPATCH_FLOATING_TYPES_AND_HALF(input.type(), \"window_merge_and_roll_forward_cuda_kernel\", ([&] {\n window_merge_and_roll_forward_cuda_kernel<<>>(\n input.data(),\n output.data(),\n B,\n H,\n W,\n C,\n shift_size,\n window_size,\n nH,\n nW);\n }));\n return output;\n}\n\n\nat::Tensor window_merge_and_roll_backward_cuda(\n at::Tensor & grad_in, \n const int B,\n const int H,\n const int W,\n const int C,\n const int shift_size,\n const int window_size){\n \n int nH = H / window_size;\n int nW = W / window_size;\n\n dim3 grid(window_size, window_size, B * nH * nW);\n //dim3 block((C + 31) / 32 * 32);\n int blocknum = best_block_dim(C);\n dim3 block(blocknum);\n\n at::Tensor grad_out;\n if (grad_in.scalar_type() == torch::kFloat16){\n grad_out = torch::empty({B*nH*nW, window_size, window_size, C}, torch::dtype(torch::kFloat16).device(torch::kCUDA).requires_grad(false));\n }\n else{\n grad_out = torch::empty({B*nH*nW, window_size, window_size, C}, torch::dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(false));\n }\n\n AT_DISPATCH_FLOATING_TYPES_AND_HALF(grad_in.type(), \"window_merge_and_roll_backward_cuda_kernel\", ([&] {\n window_merge_and_roll_backward_cuda_kernel<<>>(\n grad_in.data(),\n grad_out.data(),\n B,\n H,\n W,\n C,\n shift_size,\n window_size,\n nH,\n nW);\n }));\n return grad_out;\n}"
},
{
"path": "kernels/window_process/unit_test.py",
"content": "# --------------------------------------------------------\n# Fused kernel for window process for SwinTransformer\n# Copyright (c) 2022 Nvidia\n# Licensed under The MIT License [see LICENSE for details]\n# --------------------------------------------------------\n\nimport torch\nimport swin_window_process\nimport random\nimport time\nimport unittest\n\n\nclass WindowProcess(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, B, H, W, C, shift_size, window_size):\n output = swin_window_process.roll_and_window_partition_forward(input, B, H, W, C, shift_size, window_size)\n\n ctx.B = B\n ctx.H = H\n ctx.W = W \n ctx.C = C \n ctx.shift_size = shift_size\n ctx.window_size = window_size\n return output\n\n @staticmethod\n def backward(ctx, grad_in):\n B = ctx.B\n H = ctx.H\n W = ctx.W \n C = ctx.C \n shift_size = ctx.shift_size\n window_size = ctx.window_size\n\n grad_out = swin_window_process.roll_and_window_partition_backward(grad_in, B, H, W, C, shift_size, window_size)\n return grad_out, None, None, None, None, None, None, None\n\n\nclass WindowProcessReverse(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, B, H, W, C, shift_size, window_size):\n output = swin_window_process.window_merge_and_roll_forward(input, B, H, W, C, shift_size, window_size)\n\n ctx.B = B\n ctx.H = H\n ctx.W = W \n ctx.C = C \n ctx.shift_size = shift_size\n ctx.window_size = window_size\n\n return output\n\n @staticmethod\n def backward(ctx, grad_in):\n B = ctx.B\n H = ctx.H\n W = ctx.W \n C = ctx.C \n shift_size = ctx.shift_size\n window_size = ctx.window_size\n\n grad_out = swin_window_process.window_merge_and_roll_backward(grad_in, B, H, W, C, shift_size, window_size)\n return grad_out, None, None, None, None, None, None, None\n\n\ndef window_partition(x, window_size):\n \"\"\"\n Args:\n x: (B, H, W, C)\n window_size (int): window size\n Returns:\n windows: (num_windows*B, window_size, window_size, C)\n \"\"\"\n B, H, W, C = x.shape\n x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)\n windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)\n return windows\n\ndef window_reverse(windows, window_size, H, W):\n \"\"\"\n Args:\n windows: (num_windows*B, window_size, window_size, C)\n window_size (int): Window size\n H (int): Height of image\n W (int): Width of image\n Returns:\n x: (B, H, W, C)\n \"\"\"\n B = int(windows.shape[0] / (H * W / window_size / window_size))\n x = windows.view(B, H // window_size, W // window_size, window_size, window_size, -1)\n x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, -1)\n return x\n\n\ndef pyt_forward(x, shift_size, window_size):\n # x in shape(B, H, W, C)\n # cyclic shift\n if shift_size > 0:\n shifted_x = torch.roll(x, shifts=(-shift_size, -shift_size), dims=(1, 2))\n else:\n shifted_x = x\n # partition windows\n x_windows = window_partition(shifted_x, window_size)\n return x_windows\n\n\ndef reverse_pyt_forward(attn_windows, shift_size, window_size, H, W):\n # x in shape(B*nH*nW, window_size, window_size, C)\n shifted_x = window_reverse(attn_windows, window_size, H, W)\n if shift_size > 0:\n x = torch.roll(shifted_x, shifts=(shift_size, shift_size), dims=(1, 2))\n else:\n x = shifted_x\n return x\n\n\ndef copy_one_tensor(input, requires_grad=True):\n input1 = input.clone().detach().requires_grad_(requires_grad).cuda()\n return input1\n\nclass Test_WindowProcess(unittest.TestCase):\n def setUp(self):\n self.B = 192\n self.H = 56\n self.W = 56\n self.C = 96\n self.shift_size = 2\n self.window_size = 7\n self.nH = self.H // self.window_size\n self.nW = self.W // self.window_size\n \n def test_roll_and_window_partition_forward(self, dtype=torch.float32):\n input = torch.randn((self.B, self.H, self.W, self.C), dtype=dtype, requires_grad=True).cuda()\n \n input1 = copy_one_tensor(input, True)\n input2 = copy_one_tensor(input, True)\n\n with torch.no_grad():\n # ori\n expected = pyt_forward(input1, self.shift_size, self.window_size)\n # fused kernel\n fused_output = WindowProcess.apply(input2, self.B, self.H, self.W, self.C, -self.shift_size, self.window_size)\n \n self.assertTrue(torch.equal(expected, fused_output))\n #self.assertTrue(torch.allclose(expected, fused_output, rtol=1e-05, atol=1e-08))\n \n def test_roll_and_window_partition_backward(self, dtype=torch.float32):\n input = torch.randn((self.B, self.H, self.W, self.C), dtype=dtype, requires_grad=True).cuda()\n d_loss_tensor = torch.randn((self.B*self.nW*self.nH, self.window_size, self.window_size, self.C), dtype=dtype).cuda()\n \n input1 = copy_one_tensor(input, True)\n input2 = copy_one_tensor(input, True)\n\n # ori\n expected = pyt_forward(input1, self.shift_size, self.window_size)\n expected.backward(d_loss_tensor)\n # fused kernel\n fused_output = WindowProcess.apply(input2, self.B, self.H, self.W, self.C, -self.shift_size, self.window_size)\n fused_output.backward(d_loss_tensor)\n \n self.assertTrue(torch.equal(expected, fused_output))\n #self.assertTrue(torch.allclose(expected, fused_output, rtol=1e-05, atol=1e-08))\n\n def test_window_merge_and_roll_forward(self, dtype=torch.float32):\n input = torch.randn((self.B*self.nH*self.nW, self.window_size, self.window_size, self.C), dtype=dtype, requires_grad=True).cuda()\n \n input1 = copy_one_tensor(input, True)\n input2 = copy_one_tensor(input, True)\n\n with torch.no_grad():\n # ori\n expected = reverse_pyt_forward(input1, self.shift_size, self.window_size, self.H, self.W)\n # fused kernel\n fused_output = WindowProcessReverse.apply(input2, self.B, self.H, self.W, self.C, self.shift_size, self.window_size)\n \n self.assertTrue(torch.equal(expected, fused_output))\n #self.assertTrue(torch.allclose(expected, fused_output, rtol=1e-05, atol=1e-08))\n \n\n def test_window_merge_and_roll_backward(self, dtype=torch.float32):\n input = torch.randn((self.B*self.nH*self.nW, self.window_size, self.window_size, self.C), dtype=dtype, requires_grad=True).cuda()\n d_loss_tensor = torch.randn((self.B, self.H, self.W, self.C), dtype=dtype, requires_grad=True).cuda()\n \n input1 = copy_one_tensor(input, True)\n input2 = copy_one_tensor(input, True)\n\n # ori\n expected = reverse_pyt_forward(input1, self.shift_size, self.window_size, self.H, self.W)\n expected.backward(d_loss_tensor)\n # fused kernel\n fused_output = WindowProcessReverse.apply(input2, self.B, self.H, self.W, self.C, self.shift_size, self.window_size)\n fused_output.backward(d_loss_tensor)\n \n self.assertTrue(torch.equal(expected, fused_output))\n #self.assertTrue(torch.allclose(expected, fused_output, rtol=1e-05, atol=1e-08))\n\n def test_forward_backward_speed(self, dtype=torch.float32, times=1000):\n input = torch.randn((self.B*self.nH*self.nW, self.window_size, self.window_size, self.C), dtype=dtype, requires_grad=True).cuda()\n d_loss_tensor = torch.randn((self.B, self.H, self.W, self.C), dtype=dtype, requires_grad=True).cuda()\n \n input1 = copy_one_tensor(input, True)\n input2 = copy_one_tensor(input, True)\n\n # SwinTransformer official\n def run_pyt(t=1000):\n for _ in range(t):\n expected = reverse_pyt_forward(input1, self.shift_size, self.window_size, self.H, self.W)\n expected.backward(d_loss_tensor)\n\n # my op\n def run_fusedop(t=1000):\n for _ in range(t):\n fused_output = WindowProcessReverse.apply(input2, self.B, self.H, self.W, self.C, self.shift_size, self.window_size)\n fused_output.backward(d_loss_tensor)\n \n torch.cuda.synchronize()\n t1 = time.time()\n run_pyt(t=times)\n torch.cuda.synchronize()\n t2 = time.time()\n run_fusedop(t=times)\n torch.cuda.synchronize()\n t3 = time.time()\n self.assertTrue((t3 - t2) < (t2 - t1))\n\n print('Run {} times'.format(times))\n print('Original time cost: {}'.format(t2 - t1))\n print('Fused op time cost: {}'.format(t3 - t2))\n \n def test_roll_and_window_partition_forward_fp16(self, dtype=torch.float16):\n self.test_roll_and_window_partition_forward(dtype=dtype)\n\n def test_roll_and_window_partition_backward_fp16(self, dtype=torch.float16):\n self.test_roll_and_window_partition_backward(dtype=dtype)\n\n def test_window_merge_and_roll_forward_fp16(self, dtype=torch.float16):\n self.test_window_merge_and_roll_forward(dtype=dtype)\n \n def test_window_merge_and_roll_backward_fp16(self, dtype=torch.float16):\n self.test_window_merge_and_roll_backward(dtype=dtype)\n\n def test_forward_backward_speed_fp16(self, dtype=torch.float16, times=1000):\n self.test_forward_backward_speed(dtype=dtype, times=times)\n\n\nif __name__ == '__main__':\n print('Pass only two tensors are exactly the same (using torch.equal).\\n')\n torch.manual_seed(0)\n unittest.main(verbosity=2)\n"
},
{
"path": "kernels/window_process/window_process.py",
"content": "# --------------------------------------------------------\n# Fused kernel for window process for SwinTransformer\n# Copyright (c) 2022 Nvidia\n# Licensed under The MIT License [see LICENSE for details]\n# --------------------------------------------------------\n\nimport torch\nimport swin_window_process\n\n\nclass WindowProcess(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, B, H, W, C, shift_size, window_size):\n output = swin_window_process.roll_and_window_partition_forward(input, B, H, W, C, shift_size, window_size)\n\n ctx.B = B\n ctx.H = H\n ctx.W = W \n ctx.C = C \n ctx.shift_size = shift_size\n ctx.window_size = window_size\n return output\n\n @staticmethod\n def backward(ctx, grad_in):\n B = ctx.B\n H = ctx.H\n W = ctx.W \n C = ctx.C \n shift_size = ctx.shift_size\n window_size = ctx.window_size\n\n grad_out = swin_window_process.roll_and_window_partition_backward(grad_in, B, H, W, C, shift_size, window_size)\n return grad_out, None, None, None, None, None, None, None\n\n\nclass WindowProcessReverse(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, B, H, W, C, shift_size, window_size):\n output = swin_window_process.window_merge_and_roll_forward(input, B, H, W, C, shift_size, window_size)\n\n ctx.B = B\n ctx.H = H\n ctx.W = W \n ctx.C = C \n ctx.shift_size = shift_size\n ctx.window_size = window_size\n\n return output\n\n @staticmethod\n def backward(ctx, grad_in):\n B = ctx.B\n H = ctx.H\n W = ctx.W \n C = ctx.C \n shift_size = ctx.shift_size\n window_size = ctx.window_size\n\n #grad_out = ctx.saved_tensors[0]\n #grad_out = torch.zeros((B, H, W, C), dtype=dtype).cuda()\n grad_out = swin_window_process.window_merge_and_roll_backward(grad_in, B, H, W, C, shift_size, window_size)\n return grad_out, None, None, None, None, None, None, None\n"
},
{
"path": "logger.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport os\nimport sys\nimport logging\nimport functools\nfrom termcolor import colored\n\n\n@functools.lru_cache()\ndef create_logger(output_dir, dist_rank=0, name=''):\n # create logger\n logger = logging.getLogger(name)\n logger.setLevel(logging.DEBUG)\n logger.propagate = False\n\n # create formatter\n fmt = '[%(asctime)s %(name)s] (%(filename)s %(lineno)d): %(levelname)s %(message)s'\n color_fmt = colored('[%(asctime)s %(name)s]', 'green') + \\\n colored('(%(filename)s %(lineno)d)', 'yellow') + ': %(levelname)s %(message)s'\n\n # create console handlers for master process\n if dist_rank == 0:\n console_handler = logging.StreamHandler(sys.stdout)\n console_handler.setLevel(logging.DEBUG)\n console_handler.setFormatter(\n logging.Formatter(fmt=color_fmt, datefmt='%Y-%m-%d %H:%M:%S'))\n logger.addHandler(console_handler)\n\n # create file handlers\n file_handler = logging.FileHandler(os.path.join(output_dir, f'log_rank{dist_rank}.txt'), mode='a')\n file_handler.setLevel(logging.DEBUG)\n file_handler.setFormatter(logging.Formatter(fmt=fmt, datefmt='%Y-%m-%d %H:%M:%S'))\n logger.addHandler(file_handler)\n\n return logger\n"
},
{
"path": "lr_scheduler.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport bisect\n\nimport torch\nfrom timm.scheduler.cosine_lr import CosineLRScheduler\nfrom timm.scheduler.step_lr import StepLRScheduler\nfrom timm.scheduler.scheduler import Scheduler\n\n\ndef build_scheduler(config, optimizer, n_iter_per_epoch):\n num_steps = int(config.TRAIN.EPOCHS * n_iter_per_epoch)\n warmup_steps = int(config.TRAIN.WARMUP_EPOCHS * n_iter_per_epoch)\n decay_steps = int(config.TRAIN.LR_SCHEDULER.DECAY_EPOCHS * n_iter_per_epoch)\n multi_steps = [i * n_iter_per_epoch for i in config.TRAIN.LR_SCHEDULER.MULTISTEPS]\n\n lr_scheduler = None\n if config.TRAIN.LR_SCHEDULER.NAME == 'cosine':\n lr_scheduler = CosineLRScheduler(\n optimizer,\n t_initial=(num_steps - warmup_steps) if config.TRAIN.LR_SCHEDULER.WARMUP_PREFIX else num_steps,\n t_mul=1.,\n lr_min=config.TRAIN.MIN_LR,\n warmup_lr_init=config.TRAIN.WARMUP_LR,\n warmup_t=warmup_steps,\n cycle_limit=1,\n t_in_epochs=False,\n warmup_prefix=config.TRAIN.LR_SCHEDULER.WARMUP_PREFIX,\n )\n elif config.TRAIN.LR_SCHEDULER.NAME == 'linear':\n lr_scheduler = LinearLRScheduler(\n optimizer,\n t_initial=num_steps,\n lr_min_rate=0.01,\n warmup_lr_init=config.TRAIN.WARMUP_LR,\n warmup_t=warmup_steps,\n t_in_epochs=False,\n )\n elif config.TRAIN.LR_SCHEDULER.NAME == 'step':\n lr_scheduler = StepLRScheduler(\n optimizer,\n decay_t=decay_steps,\n decay_rate=config.TRAIN.LR_SCHEDULER.DECAY_RATE,\n warmup_lr_init=config.TRAIN.WARMUP_LR,\n warmup_t=warmup_steps,\n t_in_epochs=False,\n )\n elif config.TRAIN.LR_SCHEDULER.NAME == 'multistep':\n lr_scheduler = MultiStepLRScheduler(\n optimizer,\n milestones=multi_steps,\n gamma=config.TRAIN.LR_SCHEDULER.GAMMA,\n warmup_lr_init=config.TRAIN.WARMUP_LR,\n warmup_t=warmup_steps,\n t_in_epochs=False,\n )\n\n return lr_scheduler\n\n\nclass LinearLRScheduler(Scheduler):\n def __init__(self,\n optimizer: torch.optim.Optimizer,\n t_initial: int,\n lr_min_rate: float,\n warmup_t=0,\n warmup_lr_init=0.,\n t_in_epochs=True,\n noise_range_t=None,\n noise_pct=0.67,\n noise_std=1.0,\n noise_seed=42,\n initialize=True,\n ) -> None:\n super().__init__(\n optimizer, param_group_field=\"lr\",\n noise_range_t=noise_range_t, noise_pct=noise_pct, noise_std=noise_std, noise_seed=noise_seed,\n initialize=initialize)\n\n self.t_initial = t_initial\n self.lr_min_rate = lr_min_rate\n self.warmup_t = warmup_t\n self.warmup_lr_init = warmup_lr_init\n self.t_in_epochs = t_in_epochs\n if self.warmup_t:\n self.warmup_steps = [(v - warmup_lr_init) / self.warmup_t for v in self.base_values]\n super().update_groups(self.warmup_lr_init)\n else:\n self.warmup_steps = [1 for _ in self.base_values]\n\n def _get_lr(self, t):\n if t < self.warmup_t:\n lrs = [self.warmup_lr_init + t * s for s in self.warmup_steps]\n else:\n t = t - self.warmup_t\n total_t = self.t_initial - self.warmup_t\n lrs = [v - ((v - v * self.lr_min_rate) * (t / total_t)) for v in self.base_values]\n return lrs\n\n def get_epoch_values(self, epoch: int):\n if self.t_in_epochs:\n return self._get_lr(epoch)\n else:\n return None\n\n def get_update_values(self, num_updates: int):\n if not self.t_in_epochs:\n return self._get_lr(num_updates)\n else:\n return None\n\n\nclass MultiStepLRScheduler(Scheduler):\n def __init__(self, optimizer: torch.optim.Optimizer, milestones, gamma=0.1, warmup_t=0, warmup_lr_init=0, t_in_epochs=True) -> None:\n super().__init__(optimizer, param_group_field=\"lr\")\n \n self.milestones = milestones\n self.gamma = gamma\n self.warmup_t = warmup_t\n self.warmup_lr_init = warmup_lr_init\n self.t_in_epochs = t_in_epochs\n if self.warmup_t:\n self.warmup_steps = [(v - warmup_lr_init) / self.warmup_t for v in self.base_values]\n super().update_groups(self.warmup_lr_init)\n else:\n self.warmup_steps = [1 for _ in self.base_values]\n \n assert self.warmup_t <= min(self.milestones)\n \n def _get_lr(self, t):\n if t < self.warmup_t:\n lrs = [self.warmup_lr_init + t * s for s in self.warmup_steps]\n else:\n lrs = [v * (self.gamma ** bisect.bisect_right(self.milestones, t)) for v in self.base_values]\n return lrs\n\n def get_epoch_values(self, epoch: int):\n if self.t_in_epochs:\n return self._get_lr(epoch)\n else:\n return None\n\n def get_update_values(self, num_updates: int):\n if not self.t_in_epochs:\n return self._get_lr(num_updates)\n else:\n return None\n"
},
{
"path": "main.py",
"content": "# --------------------------------------------------------\n# Swin Transformer\n# Copyright (c) 2021 Microsoft\n# Licensed under The MIT License [see LICENSE for details]\n# Written by Ze Liu\n# --------------------------------------------------------\n\nimport os\nimport time\nimport json\nimport random\nimport argparse\nimport datetime\nimport numpy as np\n\nimport torch\nimport torch.backends.cudnn as cudnn\nimport torch.distributed as dist\n\nfrom timm.loss import LabelSmoothingCrossEntropy, SoftTargetCrossEntropy\nfrom timm.utils import accuracy, AverageMeter\n\nfrom config import get_config\nfrom models import build_model\nfrom data import build_loader\nfrom lr_scheduler import build_scheduler\nfrom optimizer import build_optimizer\nfrom logger import create_logger\nfrom utils import load_checkpoint, load_pretrained, save_checkpoint, NativeScalerWithGradNormCount, auto_resume_helper, \\\n reduce_tensor\n\n# pytorch major version (1.x or 2.x)\nPYTORCH_MAJOR_VERSION = int(torch.__version__.split('.')[0])\n\n\ndef parse_option():\n parser = argparse.ArgumentParser('Swin Transformer training and evaluation script', add_help=False)\n parser.add_argument('--cfg', type=str, required=True, metavar=\"FILE\", help='path to config file', )\n parser.add_argument(\n \"--opts\",\n help=\"Modify config options by adding 'KEY VALUE' pairs. \",\n default=None,\n nargs='+',\n )\n\n # easy config modification\n parser.add_argument('--batch-size', type=int, help=\"batch size for single GPU\")\n parser.add_argument('--data-path', type=str, help='path to dataset')\n parser.add_argument('--zip', action='store_true', help='use zipped dataset instead of folder dataset')\n parser.add_argument('--cache-mode', type=str, default='part', choices=['no', 'full', 'part'],\n help='no: no cache, '\n 'full: cache all data, '\n 'part: sharding the dataset into nonoverlapping pieces and only cache one piece')\n parser.add_argument('--pretrained',\n help='pretrained weight from checkpoint, could be imagenet22k pretrained weight')\n parser.add_argument('--resume', help='resume from checkpoint')\n parser.add_argument('--accumulation-steps', type=int, help=\"gradient accumulation steps\")\n parser.add_argument('--use-checkpoint', action='store_true',\n help=\"whether to use gradient checkpointing to save memory\")\n parser.add_argument('--disable_amp', action='store_true', help='Disable pytorch amp')\n parser.add_argument('--amp-opt-level', type=str, choices=['O0', 'O1', 'O2'],\n help='mixed precision opt level, if O0, no amp is used (deprecated!)')\n parser.add_argument('--output', default='output', type=str, metavar='PATH',\n help='root of output folder, the full path is