[ { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "README.md", "content": "## ___***FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling***___\n\n### šŸ”„šŸ”„šŸ”„ FreeNoise for longer high-quality video generation is now released!\n\n
\n

\nāœ… totally no tuning     \nāœ… less than 20% extra time     \nāœ… support 512 frames     \n

\n\n      \n      \n [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/MoonQiu/FreeNoise)      \n [![Replicate](https://replicate.com/cjwbw/longercrafter/badge)](https://replicate.com/cjwbw/longercrafter)\n\n\n_**[Haonan Qiu](http://haonanqiu.com/), [Menghan Xia*](https://menghanxia.github.io), [Yong Zhang](https://yzhang2016.github.io), [Yingqing He](https://github.com/YingqingHe), \n
\n[Xintao Wang](https://xinntao.github.io), [Ying Shan](https://scholar.google.com/citations?hl=zh-CN&user=4oXBp9UAAAAJ), and [Ziwei Liu*](https://liuziwei7.github.io/)**_\n

\n(* corresponding author)\n\nFrom Tencent AI Lab and Nanyang Technological University.\n\n\n

Input: \"A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect\"; \n
\nResolution: 1024 x 576; Frames: 64.

\n\n

Input: \"Campfire at night in a snowy forest with starry sky in the background\"; \n
\nResolution: 1024 x 576; Frames: 64.

\n
\n \n## šŸ”† Introduction\n\n\nšŸ¤—šŸ¤—šŸ¤— LongerCrafter (FreeNoise) is a tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models.\n\n### 1. Longer Single-Prompt Text-to-video Generation\n\n
\n\n

Longer single-prompt results. Resolution: 256 x 256; Frames: 512. (Compressed)

\n
\n\n### 2. Longer Multi-Prompt Text-to-video Generation\n\n
\n\n

Longer multi-prompt results. Resolution: 256 x 256; Frames: 256. (Compressed)

\n
\n\n## šŸ“ Changelog\n- __[2024.01.28]__: šŸ”„šŸ”„ Support FreeNoise on VideoCrafter2!\n- __[2024.01.23]__: šŸ”„šŸ”„ Support FreeNoise on other two video frameworks AnimateDiff and LaVie!\n- __[2023.10.25]__: šŸ”„šŸ”„ Release the 256x256 model and support multi-prompt generation!\n- __[2023.10.24]__: šŸ”„šŸ”„ Release the LongerCrafter (FreeNoise), longer video generation!\n
\n\n\n## šŸ§° Models\n\n|Model|Resolution|Checkpoint|Description\n|:---------|:---------|:--------|:--------|\n|VideoCrafter (Text2Video)|576x1024|[Hugging Face](https://huggingface.co/VideoCrafter/Text2Video-1024-v1.0/blob/main/model.ckpt)|Support 64 frames on NVIDIA A100 (40GB)\n|VideoCrafter (Text2Video)|256x256|[Hugging Face](https://huggingface.co/VideoCrafter)|Support 512 frames on NVIDIA A100 (40GB)\n|VideoCrafter2 (Text2Video)|320x512|[Hugging Face](https://huggingface.co/VideoCrafter/VideoCrafter2/blob/main/model.ckpt)|Support 128 frames on NVIDIA A100 (40GB)\n\n(Reduce the number of frames when you have smaller GPUs, e.g. 256x256 resolutions with 64 frames.)\n\n## āš™ļø Setup\n\n### Install Environment via Anaconda (Recommended)\n```bash\nconda create -n freenoise python=3.8.5\nconda activate freenoise\npip install -r requirements.txt\n```\n\n\n## šŸ’« Inference \n### 1. Longer Text-to-Video\n\n\n\n1) Download pretrained T2V models via [Hugging Face](https://huggingface.co/VideoCrafter/Text2Video-1024-v1.0/blob/main/model.ckpt), and put the `model.ckpt` in `checkpoints/base_1024_v1/model.ckpt`.\n2) Input the following commands in the terminal.\n```bash\nsh scripts/run_text2video_freenoise_1024.sh\n```\n\n### 2. Longer Multi-Prompt Text-to-Video\n\n1) Download pretrained T2V models via [Hugging Face](https://huggingface.co/VideoCrafter), and put the `model.ckpt` in `checkpoints/base_256_v1/model.ckpt`.\n2) Input the following commands in the terminal.\n```bash\nsh scripts/run_text2video_freenoise_mp_256.sh\n```\n\n\n## šŸ§² Support For Other Models\n\nFreeNoise is supposed to work on other similar frameworks. An easy way to test compatibility is by shuffling the noise to see whether a new similar video can be generated (set eta to 0). If you have any questions about applying FreeNoise to other frameworks, feel free to contact [Haonan Qiu](http://haonanqiu.com/).\n\nCurrent official implementation: [FreeNoise-VideoCrafter](https://github.com/AILab-CVC/FreeNoise), [FreeNoise-AnimateDiff](https://github.com/arthur-qiu/FreeNoise-AnimateDiff), [FreeNoise-LaVie](https://github.com/arthur-qiu/FreeNoise-LaVie) \n\n## šŸš€ My Free Series\n[FreeScale](https://github.com/ali-vilab/FreeScale): Tuning-free method for high-resolution image/video generation.\n\n[FreeTraj](https://github.com/arthur-qiu/FreeTraj): Tuning-free method for trajectory control.\n\n## šŸ‘« Crafter Family\n[VideoCrafter](https://github.com/AILab-CVC/VideoCrafter): Framework for high-quality video generation.\n\n[ScaleCrafter](https://github.com/YingqingHe/ScaleCrafter): Tuning-free method for high-resolution image/video generation.\n\n[TaleCrafter](https://github.com/AILab-CVC/TaleCrafter): An interactive story visualization tool that supports multiple characters. \n\n\n## šŸ˜‰ Citation\n```bib\n@misc{qiu2023freenoise,\n title={FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling}, \n author={Haonan Qiu and Menghan Xia and Yong Zhang and Yingqing He and Xintao Wang and Ying Shan and Ziwei Liu},\n year={2023},\n eprint={2310.15169},\n archivePrefix={arXiv},\n primaryClass={cs.CV}\n}\n```\n\n\n## šŸ“¢ Disclaimer\nWe develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes.\n****\n\n" }, { "path": "cog.yaml", "content": "# Configuration for Cog āš™ļø\n# Reference: https://github.com/replicate/cog/blob/main/docs/yaml.md\n\nbuild:\n gpu: true\n system_packages:\n - \"libgl1-mesa-glx\"\n - \"libglib2.0-0\"\n python_version: \"3.11\"\n python_packages:\n - \"decord==0.6.0\"\n - \"einops==0.3.0\"\n - \"imageio==2.9.0\"\n - \"numpy==1.24.2\"\n - \"omegaconf==2.1.1\"\n - \"opencv_python==4.8.1.78\"\n - \"pandas==2.0.0\"\n - \"Pillow==9.5.0\"\n - \"pytorch_lightning==1.8.3\"\n - \"PyYAML==6.0\"\n - \"setuptools==65.6.3\"\n - \"torch==2.0.1\"\n - \"torchvision==0.15.2\"\n - \"tqdm==4.65.0\"\n - \"transformers==4.25.1\"\n - \"moviepy==1.0.3\"\n - \"av==10.0.0\"\n - \"xformers==0.0.22\"\n - \"timm==0.9.8\"\n - \"scikit-learn==1.3.2\"\n - \"open_clip_torch==2.23.0\"\n - \"kornia==0.7.0\"\npredict: \"predict.py:Predictor\"\n" }, { "path": "configs/inference_t2v_1024_v1.0.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 72\n - 128\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: true\n fix_scale_bug: true\n unet_config:\n target: lvdm.modules.networks.openaimodel3d.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: false\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: true\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "configs/inference_t2v_1024_v1.0_freenoise.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 72\n - 128\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: true\n fix_scale_bug: true\n unet_config:\n target: lvdm.modules.networks.openaimodel3d_freenoise.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: false\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: true\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "configs/inference_t2v_tconv256_v1.0.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 32\n - 32\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: false\n fix_scale_bug: true\n unet_config:\n target: lvdm.modules.networks.openaimodel3d_freenoise.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: true\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: false\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "configs/inference_t2v_tconv256_v1.0_freenoise.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 32\n - 32\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: false\n fix_scale_bug: true\n unet_config:\n target: lvdm.modules.networks.openaimodel3d_freenoise.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: true\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: false\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "configs/inference_t2v_tconv512_v2.0.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 40\n - 64\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: true\n scale_b: 0.7\n unet_config:\n target: lvdm.modules.networks.openaimodel3d.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: true\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: false\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "configs/inference_t2v_tconv512_v2.0_freenoise.yaml", "content": "model:\n target: lvdm.models.ddpm3d.LatentDiffusion\n params:\n linear_start: 0.00085\n linear_end: 0.012\n num_timesteps_cond: 1\n timesteps: 1000\n first_stage_key: video\n cond_stage_key: caption\n cond_stage_trainable: false\n conditioning_key: crossattn\n image_size:\n - 40\n - 64\n channels: 4\n scale_by_std: false\n scale_factor: 0.18215\n use_ema: false\n uncond_type: empty_seq\n use_scale: true\n scale_b: 0.7\n unet_config:\n target: lvdm.modules.networks.openaimodel3d_freenoise.UNetModel\n params:\n in_channels: 4\n out_channels: 4\n model_channels: 320\n attention_resolutions:\n - 4\n - 2\n - 1\n num_res_blocks: 2\n channel_mult:\n - 1\n - 2\n - 4\n - 4\n num_head_channels: 64\n transformer_depth: 1\n context_dim: 1024\n use_linear: true\n use_checkpoint: true\n temporal_conv: true\n temporal_attention: true\n temporal_selfatt_only: true\n use_relative_position: false\n use_causal_attention: false\n temporal_length: 16\n addition_attention: true\n fps_cond: true\n first_stage_config:\n target: lvdm.models.autoencoder.AutoencoderKL\n params:\n embed_dim: 4\n monitor: val/rec_loss\n ddconfig:\n double_z: true\n z_channels: 4\n resolution: 512\n in_channels: 3\n out_ch: 3\n ch: 128\n ch_mult:\n - 1\n - 2\n - 4\n - 4\n num_res_blocks: 2\n attn_resolutions: []\n dropout: 0.0\n lossconfig:\n target: torch.nn.Identity\n cond_stage_config:\n target: lvdm.modules.encoders.condition.FrozenOpenCLIPEmbedder\n params:\n freeze: true\n layer: penultimate\n" }, { "path": "lvdm/basics.py", "content": "# adopted from\n# https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/gaussian_diffusion.py\n# and\n# https://github.com/lucidrains/denoising-diffusion-pytorch/blob/7706bdfc6f527f58d33f84b7b522e61e6e3164b3/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py\n# and\n# https://github.com/openai/guided-diffusion/blob/0ba878e517b276c45d1195eb29f6f5f72659a05b/guided_diffusion/nn.py\n#\n# thanks!\n\nimport torch.nn as nn\nfrom utils.utils import instantiate_from_config\n\n\ndef disabled_train(self, mode=True):\n \"\"\"Overwrite model.train with this function to make sure train/eval mode\n does not change anymore.\"\"\"\n return self\n\ndef zero_module(module):\n \"\"\"\n Zero out the parameters of a module and return it.\n \"\"\"\n for p in module.parameters():\n p.detach().zero_()\n return module\n\ndef scale_module(module, scale):\n \"\"\"\n Scale the parameters of a module and return it.\n \"\"\"\n for p in module.parameters():\n p.detach().mul_(scale)\n return module\n\n\ndef conv_nd(dims, *args, **kwargs):\n \"\"\"\n Create a 1D, 2D, or 3D convolution module.\n \"\"\"\n if dims == 1:\n return nn.Conv1d(*args, **kwargs)\n elif dims == 2:\n return nn.Conv2d(*args, **kwargs)\n elif dims == 3:\n return nn.Conv3d(*args, **kwargs)\n raise ValueError(f\"unsupported dimensions: {dims}\")\n\n\ndef linear(*args, **kwargs):\n \"\"\"\n Create a linear module.\n \"\"\"\n return nn.Linear(*args, **kwargs)\n\n\ndef avg_pool_nd(dims, *args, **kwargs):\n \"\"\"\n Create a 1D, 2D, or 3D average pooling module.\n \"\"\"\n if dims == 1:\n return nn.AvgPool1d(*args, **kwargs)\n elif dims == 2:\n return nn.AvgPool2d(*args, **kwargs)\n elif dims == 3:\n return nn.AvgPool3d(*args, **kwargs)\n raise ValueError(f\"unsupported dimensions: {dims}\")\n\n\ndef nonlinearity(type='silu'):\n if type == 'silu':\n return nn.SiLU()\n elif type == 'leaky_relu':\n return nn.LeakyReLU()\n\n\nclass GroupNormSpecific(nn.GroupNorm):\n def forward(self, x):\n return super().forward(x.float()).type(x.dtype)\n\n\ndef normalization(channels, num_groups=32):\n \"\"\"\n Make a standard normalization layer.\n :param channels: number of input channels.\n :return: an nn.Module for normalization.\n \"\"\"\n return GroupNormSpecific(num_groups, channels)\n\n\nclass HybridConditioner(nn.Module):\n\n def __init__(self, c_concat_config, c_crossattn_config):\n super().__init__()\n self.concat_conditioner = instantiate_from_config(c_concat_config)\n self.crossattn_conditioner = instantiate_from_config(c_crossattn_config)\n\n def forward(self, c_concat, c_crossattn):\n c_concat = self.concat_conditioner(c_concat)\n c_crossattn = self.crossattn_conditioner(c_crossattn)\n return {'c_concat': [c_concat], 'c_crossattn': [c_crossattn]}" }, { "path": "lvdm/common.py", "content": "import math\nfrom inspect import isfunction\nimport torch\nfrom torch import nn\nimport torch.distributed as dist\n\n\ndef gather_data(data, return_np=True):\n ''' gather data from multiple processes to one list '''\n data_list = [torch.zeros_like(data) for _ in range(dist.get_world_size())]\n dist.all_gather(data_list, data) # gather not supported with NCCL\n if return_np:\n data_list = [data.cpu().numpy() for data in data_list]\n return data_list\n\ndef autocast(f):\n def do_autocast(*args, **kwargs):\n with torch.cuda.amp.autocast(enabled=True,\n dtype=torch.get_autocast_gpu_dtype(),\n cache_enabled=torch.is_autocast_cache_enabled()):\n return f(*args, **kwargs)\n return do_autocast\n\n\ndef extract_into_tensor(a, t, x_shape):\n b, *_ = t.shape\n out = a.gather(-1, t)\n return out.reshape(b, *((1,) * (len(x_shape) - 1)))\n\n\ndef noise_like(shape, device, repeat=False):\n repeat_noise = lambda: torch.randn((1, *shape[1:]), device=device).repeat(shape[0], *((1,) * (len(shape) - 1)))\n noise = lambda: torch.randn(shape, device=device)\n return repeat_noise() if repeat else noise()\n\n\ndef default(val, d):\n if exists(val):\n return val\n return d() if isfunction(d) else d\n\ndef exists(val):\n return val is not None\n\ndef identity(*args, **kwargs):\n return nn.Identity()\n\ndef uniq(arr):\n return{el: True for el in arr}.keys()\n\ndef mean_flat(tensor):\n \"\"\"\n Take the mean over all non-batch dimensions.\n \"\"\"\n return tensor.mean(dim=list(range(1, len(tensor.shape))))\n\ndef ismap(x):\n if not isinstance(x, torch.Tensor):\n return False\n return (len(x.shape) == 4) and (x.shape[1] > 3)\n\ndef isimage(x):\n if not isinstance(x,torch.Tensor):\n return False\n return (len(x.shape) == 4) and (x.shape[1] == 3 or x.shape[1] == 1)\n\ndef max_neg_value(t):\n return -torch.finfo(t.dtype).max\n\ndef shape_to_str(x):\n shape_str = \"x\".join([str(x) for x in x.shape])\n return shape_str\n\ndef init_(tensor):\n dim = tensor.shape[-1]\n std = 1 / math.sqrt(dim)\n tensor.uniform_(-std, std)\n return tensor\n\nckpt = torch.utils.checkpoint.checkpoint\ndef checkpoint(func, inputs, params, flag):\n \"\"\"\n Evaluate a function without caching intermediate activations, allowing for\n reduced memory at the expense of extra compute in the backward pass.\n :param func: the function to evaluate.\n :param inputs: the argument sequence to pass to `func`.\n :param params: a sequence of parameters `func` depends on but does not\n explicitly take as arguments.\n :param flag: if False, disable gradient checkpointing.\n \"\"\"\n if flag:\n return ckpt(func, *inputs)\n else:\n return func(*inputs)\n\n" }, { "path": "lvdm/distributions.py", "content": "import torch\nimport numpy as np\n\n\nclass AbstractDistribution:\n def sample(self):\n raise NotImplementedError()\n\n def mode(self):\n raise NotImplementedError()\n\n\nclass DiracDistribution(AbstractDistribution):\n def __init__(self, value):\n self.value = value\n\n def sample(self):\n return self.value\n\n def mode(self):\n return self.value\n\n\nclass DiagonalGaussianDistribution(object):\n def __init__(self, parameters, deterministic=False):\n self.parameters = parameters\n self.mean, self.logvar = torch.chunk(parameters, 2, dim=1)\n self.logvar = torch.clamp(self.logvar, -30.0, 20.0)\n self.deterministic = deterministic\n self.std = torch.exp(0.5 * self.logvar)\n self.var = torch.exp(self.logvar)\n if self.deterministic:\n self.var = self.std = torch.zeros_like(self.mean).to(device=self.parameters.device)\n\n def sample(self, noise=None):\n if noise is None:\n noise = torch.randn(self.mean.shape)\n \n x = self.mean + self.std * noise.to(device=self.parameters.device)\n return x\n\n def kl(self, other=None):\n if self.deterministic:\n return torch.Tensor([0.])\n else:\n if other is None:\n return 0.5 * torch.sum(torch.pow(self.mean, 2)\n + self.var - 1.0 - self.logvar,\n dim=[1, 2, 3])\n else:\n return 0.5 * torch.sum(\n torch.pow(self.mean - other.mean, 2) / other.var\n + self.var / other.var - 1.0 - self.logvar + other.logvar,\n dim=[1, 2, 3])\n\n def nll(self, sample, dims=[1,2,3]):\n if self.deterministic:\n return torch.Tensor([0.])\n logtwopi = np.log(2.0 * np.pi)\n return 0.5 * torch.sum(\n logtwopi + self.logvar + torch.pow(sample - self.mean, 2) / self.var,\n dim=dims)\n\n def mode(self):\n return self.mean\n\n\ndef normal_kl(mean1, logvar1, mean2, logvar2):\n \"\"\"\n source: https://github.com/openai/guided-diffusion/blob/27c20a8fab9cb472df5d6bdd6c8d11c8f430b924/guided_diffusion/losses.py#L12\n Compute the KL divergence between two gaussians.\n Shapes are automatically broadcasted, so batches can be compared to\n scalars, among other use cases.\n \"\"\"\n tensor = None\n for obj in (mean1, logvar1, mean2, logvar2):\n if isinstance(obj, torch.Tensor):\n tensor = obj\n break\n assert tensor is not None, \"at least one argument must be a Tensor\"\n\n # Force variances to be Tensors. Broadcasting helps convert scalars to\n # Tensors, but it does not work for torch.exp().\n logvar1, logvar2 = [\n x if isinstance(x, torch.Tensor) else torch.tensor(x).to(tensor)\n for x in (logvar1, logvar2)\n ]\n\n return 0.5 * (\n -1.0\n + logvar2\n - logvar1\n + torch.exp(logvar1 - logvar2)\n + ((mean1 - mean2) ** 2) * torch.exp(-logvar2)\n )\n" }, { "path": "lvdm/ema.py", "content": "import torch\nfrom torch import nn\n\n\nclass LitEma(nn.Module):\n def __init__(self, model, decay=0.9999, use_num_upates=True):\n super().__init__()\n if decay < 0.0 or decay > 1.0:\n raise ValueError('Decay must be between 0 and 1')\n\n self.m_name2s_name = {}\n self.register_buffer('decay', torch.tensor(decay, dtype=torch.float32))\n self.register_buffer('num_updates', torch.tensor(0,dtype=torch.int) if use_num_upates\n else torch.tensor(-1,dtype=torch.int))\n\n for name, p in model.named_parameters():\n if p.requires_grad:\n #remove as '.'-character is not allowed in buffers\n s_name = name.replace('.','')\n self.m_name2s_name.update({name:s_name})\n self.register_buffer(s_name,p.clone().detach().data)\n\n self.collected_params = []\n\n def forward(self,model):\n decay = self.decay\n\n if self.num_updates >= 0:\n self.num_updates += 1\n decay = min(self.decay,(1 + self.num_updates) / (10 + self.num_updates))\n\n one_minus_decay = 1.0 - decay\n\n with torch.no_grad():\n m_param = dict(model.named_parameters())\n shadow_params = dict(self.named_buffers())\n\n for key in m_param:\n if m_param[key].requires_grad:\n sname = self.m_name2s_name[key]\n shadow_params[sname] = shadow_params[sname].type_as(m_param[key])\n shadow_params[sname].sub_(one_minus_decay * (shadow_params[sname] - m_param[key]))\n else:\n assert not key in self.m_name2s_name\n\n def copy_to(self, model):\n m_param = dict(model.named_parameters())\n shadow_params = dict(self.named_buffers())\n for key in m_param:\n if m_param[key].requires_grad:\n m_param[key].data.copy_(shadow_params[self.m_name2s_name[key]].data)\n else:\n assert not key in self.m_name2s_name\n\n def store(self, parameters):\n \"\"\"\n Save the current parameters for restoring later.\n Args:\n parameters: Iterable of `torch.nn.Parameter`; the parameters to be\n temporarily stored.\n \"\"\"\n self.collected_params = [param.clone() for param in parameters]\n\n def restore(self, parameters):\n \"\"\"\n Restore the parameters stored with the `store` method.\n Useful to validate the model with EMA parameters without affecting the\n original optimization process. Store the parameters before the\n `copy_to` method. After validation (or model saving), use this to\n restore the former parameters.\n Args:\n parameters: Iterable of `torch.nn.Parameter`; the parameters to be\n updated with the stored parameters.\n \"\"\"\n for c_param, param in zip(self.collected_params, parameters):\n param.data.copy_(c_param.data)\n" }, { "path": "lvdm/models/autoencoder.py", "content": "import os\nfrom contextlib import contextmanager\nimport torch\nimport numpy as np\nfrom einops import rearrange\nimport torch.nn.functional as F\nimport pytorch_lightning as pl\nfrom lvdm.modules.networks.ae_modules import Encoder, Decoder\nfrom lvdm.distributions import DiagonalGaussianDistribution\nfrom utils.utils import instantiate_from_config\n\n\nclass AutoencoderKL(pl.LightningModule):\n def __init__(self,\n ddconfig,\n lossconfig,\n embed_dim,\n ckpt_path=None,\n ignore_keys=[],\n image_key=\"image\",\n colorize_nlabels=None,\n monitor=None,\n test=False,\n logdir=None,\n input_dim=4,\n test_args=None,\n ):\n super().__init__()\n self.image_key = image_key\n self.encoder = Encoder(**ddconfig)\n self.decoder = Decoder(**ddconfig)\n self.loss = instantiate_from_config(lossconfig)\n assert ddconfig[\"double_z\"]\n self.quant_conv = torch.nn.Conv2d(2*ddconfig[\"z_channels\"], 2*embed_dim, 1)\n self.post_quant_conv = torch.nn.Conv2d(embed_dim, ddconfig[\"z_channels\"], 1)\n self.embed_dim = embed_dim\n self.input_dim = input_dim\n self.test = test\n self.test_args = test_args\n self.logdir = logdir\n if colorize_nlabels is not None:\n assert type(colorize_nlabels)==int\n self.register_buffer(\"colorize\", torch.randn(3, colorize_nlabels, 1, 1))\n if monitor is not None:\n self.monitor = monitor\n if ckpt_path is not None:\n self.init_from_ckpt(ckpt_path, ignore_keys=ignore_keys)\n if self.test:\n self.init_test()\n \n def init_test(self,):\n self.test = True\n save_dir = os.path.join(self.logdir, \"test\")\n if 'ckpt' in self.test_args:\n ckpt_name = os.path.basename(self.test_args.ckpt).split('.ckpt')[0] + f'_epoch{self._cur_epoch}'\n self.root = os.path.join(save_dir, ckpt_name)\n else:\n self.root = save_dir\n if 'test_subdir' in self.test_args:\n self.root = os.path.join(save_dir, self.test_args.test_subdir)\n\n self.root_zs = os.path.join(self.root, \"zs\")\n self.root_dec = os.path.join(self.root, \"reconstructions\")\n self.root_inputs = os.path.join(self.root, \"inputs\")\n os.makedirs(self.root, exist_ok=True)\n\n if self.test_args.save_z:\n os.makedirs(self.root_zs, exist_ok=True)\n if self.test_args.save_reconstruction:\n os.makedirs(self.root_dec, exist_ok=True)\n if self.test_args.save_input:\n os.makedirs(self.root_inputs, exist_ok=True)\n assert(self.test_args is not None)\n self.test_maximum = getattr(self.test_args, 'test_maximum', None) \n self.count = 0\n self.eval_metrics = {}\n self.decodes = []\n self.save_decode_samples = 2048\n\n def init_from_ckpt(self, path, ignore_keys=list()):\n sd = torch.load(path, map_location=\"cpu\")\n try:\n self._cur_epoch = sd['epoch']\n sd = sd[\"state_dict\"]\n except:\n self._cur_epoch = 'null'\n keys = list(sd.keys())\n for k in keys:\n for ik in ignore_keys:\n if k.startswith(ik):\n print(\"Deleting key {} from state_dict.\".format(k))\n del sd[k]\n self.load_state_dict(sd, strict=False)\n # self.load_state_dict(sd, strict=True)\n print(f\"Restored from {path}\")\n\n def encode(self, x, **kwargs):\n \n h = self.encoder(x)\n moments = self.quant_conv(h)\n posterior = DiagonalGaussianDistribution(moments)\n return posterior\n\n def decode(self, z, **kwargs):\n z = self.post_quant_conv(z)\n dec = self.decoder(z)\n return dec\n\n def forward(self, input, sample_posterior=True):\n posterior = self.encode(input)\n if sample_posterior:\n z = posterior.sample()\n else:\n z = posterior.mode()\n dec = self.decode(z)\n return dec, posterior\n\n def get_input(self, batch, k):\n x = batch[k]\n if x.dim() == 5 and self.input_dim == 4:\n b,c,t,h,w = x.shape\n self.b = b\n self.t = t \n x = rearrange(x, 'b c t h w -> (b t) c h w')\n\n return x\n\n def training_step(self, batch, batch_idx, optimizer_idx):\n inputs = self.get_input(batch, self.image_key)\n reconstructions, posterior = self(inputs)\n\n if optimizer_idx == 0:\n # train encoder+decoder+logvar\n aeloss, log_dict_ae = self.loss(inputs, reconstructions, posterior, optimizer_idx, self.global_step,\n last_layer=self.get_last_layer(), split=\"train\")\n self.log(\"aeloss\", aeloss, prog_bar=True, logger=True, on_step=True, on_epoch=True)\n self.log_dict(log_dict_ae, prog_bar=False, logger=True, on_step=True, on_epoch=False)\n return aeloss\n\n if optimizer_idx == 1:\n # train the discriminator\n discloss, log_dict_disc = self.loss(inputs, reconstructions, posterior, optimizer_idx, self.global_step,\n last_layer=self.get_last_layer(), split=\"train\")\n\n self.log(\"discloss\", discloss, prog_bar=True, logger=True, on_step=True, on_epoch=True)\n self.log_dict(log_dict_disc, prog_bar=False, logger=True, on_step=True, on_epoch=False)\n return discloss\n\n def validation_step(self, batch, batch_idx):\n inputs = self.get_input(batch, self.image_key)\n reconstructions, posterior = self(inputs)\n aeloss, log_dict_ae = self.loss(inputs, reconstructions, posterior, 0, self.global_step,\n last_layer=self.get_last_layer(), split=\"val\")\n\n discloss, log_dict_disc = self.loss(inputs, reconstructions, posterior, 1, self.global_step,\n last_layer=self.get_last_layer(), split=\"val\")\n\n self.log(\"val/rec_loss\", log_dict_ae[\"val/rec_loss\"])\n self.log_dict(log_dict_ae)\n self.log_dict(log_dict_disc)\n return self.log_dict\n \n def configure_optimizers(self):\n lr = self.learning_rate\n opt_ae = torch.optim.Adam(list(self.encoder.parameters())+\n list(self.decoder.parameters())+\n list(self.quant_conv.parameters())+\n list(self.post_quant_conv.parameters()),\n lr=lr, betas=(0.5, 0.9))\n opt_disc = torch.optim.Adam(self.loss.discriminator.parameters(),\n lr=lr, betas=(0.5, 0.9))\n return [opt_ae, opt_disc], []\n\n def get_last_layer(self):\n return self.decoder.conv_out.weight\n\n @torch.no_grad()\n def log_images(self, batch, only_inputs=False, **kwargs):\n log = dict()\n x = self.get_input(batch, self.image_key)\n x = x.to(self.device)\n if not only_inputs:\n xrec, posterior = self(x)\n if x.shape[1] > 3:\n # colorize with random projection\n assert xrec.shape[1] > 3\n x = self.to_rgb(x)\n xrec = self.to_rgb(xrec)\n log[\"samples\"] = self.decode(torch.randn_like(posterior.sample()))\n log[\"reconstructions\"] = xrec\n log[\"inputs\"] = x\n return log\n\n def to_rgb(self, x):\n assert self.image_key == \"segmentation\"\n if not hasattr(self, \"colorize\"):\n self.register_buffer(\"colorize\", torch.randn(3, x.shape[1], 1, 1).to(x))\n x = F.conv2d(x, weight=self.colorize)\n x = 2.*(x-x.min())/(x.max()-x.min()) - 1.\n return x\n\nclass IdentityFirstStage(torch.nn.Module):\n def __init__(self, *args, vq_interface=False, **kwargs):\n self.vq_interface = vq_interface # TODO: Should be true by default but check to not break older stuff\n super().__init__()\n\n def encode(self, x, *args, **kwargs):\n return x\n\n def decode(self, x, *args, **kwargs):\n return x\n\n def quantize(self, x, *args, **kwargs):\n if self.vq_interface:\n return x, None, [None, None, None]\n return x\n\n def forward(self, x, *args, **kwargs):\n return x\n" }, { "path": "lvdm/models/ddpm3d.py", "content": "\"\"\"\nwild mixture of\nhttps://github.com/openai/improved-diffusion/blob/e94489283bb876ac1477d5dd7709bbbd2d9902ce/improved_diffusion/gaussian_diffusion.py\nhttps://github.com/lucidrains/denoising-diffusion-pytorch/blob/7706bdfc6f527f58d33f84b7b522e61e6e3164b3/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py\nhttps://github.com/CompVis/taming-transformers\n-- merci\n\"\"\"\n\nfrom functools import partial\nfrom contextlib import contextmanager\nimport numpy as np\nfrom tqdm import tqdm\nfrom einops import rearrange, repeat\nimport logging\nmainlogger = logging.getLogger('mainlogger')\nimport torch\nimport torch.nn as nn\nfrom torchvision.utils import make_grid\nimport pytorch_lightning as pl\nfrom utils.utils import instantiate_from_config\nfrom lvdm.ema import LitEma\nfrom lvdm.distributions import DiagonalGaussianDistribution\nfrom lvdm.models.utils_diffusion import make_beta_schedule\nfrom lvdm.modules.encoders.ip_resampler import ImageProjModel, Resampler\nfrom lvdm.basics import disabled_train\nfrom lvdm.common import (\n extract_into_tensor,\n noise_like,\n exists,\n default\n)\n\n\n__conditioning_keys__ = {'concat': 'c_concat',\n 'crossattn': 'c_crossattn',\n 'adm': 'y'}\n\nclass DDPM(pl.LightningModule):\n # classic DDPM with Gaussian diffusion, in image space\n def __init__(self,\n unet_config,\n timesteps=1000,\n beta_schedule=\"linear\",\n loss_type=\"l2\",\n ckpt_path=None,\n ignore_keys=[],\n load_only_unet=False,\n monitor=None,\n use_ema=True,\n first_stage_key=\"image\",\n image_size=256,\n channels=3,\n log_every_t=100,\n clip_denoised=True,\n linear_start=1e-4,\n linear_end=2e-2,\n cosine_s=8e-3,\n given_betas=None,\n original_elbo_weight=0.,\n v_posterior=0., # weight for choosing posterior variance as sigma = (1-v) * beta_tilde + v * beta\n l_simple_weight=1.,\n conditioning_key=None,\n parameterization=\"eps\", # all assuming fixed variance schedules\n scheduler_config=None,\n use_positional_encodings=False,\n learn_logvar=False,\n logvar_init=0.\n ):\n super().__init__()\n assert parameterization in [\"eps\", \"x0\"], 'currently only supporting \"eps\" and \"x0\"'\n self.parameterization = parameterization\n mainlogger.info(f\"{self.__class__.__name__}: Running in {self.parameterization}-prediction mode\")\n self.cond_stage_model = None\n self.clip_denoised = clip_denoised\n self.log_every_t = log_every_t\n self.first_stage_key = first_stage_key\n self.channels = channels\n self.temporal_length = unet_config.params.temporal_length\n self.image_size = image_size \n if isinstance(self.image_size, int):\n self.image_size = [self.image_size, self.image_size]\n self.use_positional_encodings = use_positional_encodings\n self.model = DiffusionWrapper(unet_config, conditioning_key)\n self.use_ema = use_ema\n if self.use_ema:\n self.model_ema = LitEma(self.model)\n mainlogger.info(f\"Keeping EMAs of {len(list(self.model_ema.buffers()))}.\")\n\n self.use_scheduler = scheduler_config is not None\n if self.use_scheduler:\n self.scheduler_config = scheduler_config\n\n self.v_posterior = v_posterior\n self.original_elbo_weight = original_elbo_weight\n self.l_simple_weight = l_simple_weight\n\n if monitor is not None:\n self.monitor = monitor\n if ckpt_path is not None:\n self.init_from_ckpt(ckpt_path, ignore_keys=ignore_keys, only_model=load_only_unet)\n\n self.register_schedule(given_betas=given_betas, beta_schedule=beta_schedule, timesteps=timesteps,\n linear_start=linear_start, linear_end=linear_end, cosine_s=cosine_s)\n\n self.loss_type = loss_type\n\n self.learn_logvar = learn_logvar\n self.logvar = torch.full(fill_value=logvar_init, size=(self.num_timesteps,))\n if self.learn_logvar:\n self.logvar = nn.Parameter(self.logvar, requires_grad=True)\n\n\n def register_schedule(self, given_betas=None, beta_schedule=\"linear\", timesteps=1000,\n linear_start=1e-4, linear_end=2e-2, cosine_s=8e-3):\n if exists(given_betas):\n betas = given_betas\n else:\n betas = make_beta_schedule(beta_schedule, timesteps, linear_start=linear_start, linear_end=linear_end,\n cosine_s=cosine_s)\n alphas = 1. - betas\n alphas_cumprod = np.cumprod(alphas, axis=0)\n alphas_cumprod_prev = np.append(1., alphas_cumprod[:-1])\n\n timesteps, = betas.shape\n self.num_timesteps = int(timesteps)\n self.linear_start = linear_start\n self.linear_end = linear_end\n assert alphas_cumprod.shape[0] == self.num_timesteps, 'alphas have to be defined for each timestep'\n\n to_torch = partial(torch.tensor, dtype=torch.float32)\n\n self.register_buffer('betas', to_torch(betas))\n self.register_buffer('alphas_cumprod', to_torch(alphas_cumprod))\n self.register_buffer('alphas_cumprod_prev', to_torch(alphas_cumprod_prev))\n\n # calculations for diffusion q(x_t | x_{t-1}) and others\n self.register_buffer('sqrt_alphas_cumprod', to_torch(np.sqrt(alphas_cumprod)))\n self.register_buffer('sqrt_one_minus_alphas_cumprod', to_torch(np.sqrt(1. - alphas_cumprod)))\n self.register_buffer('log_one_minus_alphas_cumprod', to_torch(np.log(1. - alphas_cumprod)))\n self.register_buffer('sqrt_recip_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod)))\n self.register_buffer('sqrt_recipm1_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod - 1)))\n\n # calculations for posterior q(x_{t-1} | x_t, x_0)\n posterior_variance = (1 - self.v_posterior) * betas * (1. - alphas_cumprod_prev) / (\n 1. - alphas_cumprod) + self.v_posterior * betas\n # above: equal to 1. / (1. / (1. - alpha_cumprod_tm1) + alpha_t / beta_t)\n self.register_buffer('posterior_variance', to_torch(posterior_variance))\n # below: log calculation clipped because the posterior variance is 0 at the beginning of the diffusion chain\n self.register_buffer('posterior_log_variance_clipped', to_torch(np.log(np.maximum(posterior_variance, 1e-20))))\n self.register_buffer('posterior_mean_coef1', to_torch(\n betas * np.sqrt(alphas_cumprod_prev) / (1. - alphas_cumprod)))\n self.register_buffer('posterior_mean_coef2', to_torch(\n (1. - alphas_cumprod_prev) * np.sqrt(alphas) / (1. - alphas_cumprod)))\n\n if self.parameterization == \"eps\":\n lvlb_weights = self.betas ** 2 / (\n 2 * self.posterior_variance * to_torch(alphas) * (1 - self.alphas_cumprod))\n elif self.parameterization == \"x0\":\n lvlb_weights = 0.5 * np.sqrt(torch.Tensor(alphas_cumprod)) / (2. * 1 - torch.Tensor(alphas_cumprod))\n else:\n raise NotImplementedError(\"mu not supported\")\n # TODO how to choose this term\n lvlb_weights[0] = lvlb_weights[1]\n self.register_buffer('lvlb_weights', lvlb_weights, persistent=False)\n assert not torch.isnan(self.lvlb_weights).all()\n\n @contextmanager\n def ema_scope(self, context=None):\n if self.use_ema:\n self.model_ema.store(self.model.parameters())\n self.model_ema.copy_to(self.model)\n if context is not None:\n mainlogger.info(f\"{context}: Switched to EMA weights\")\n try:\n yield None\n finally:\n if self.use_ema:\n self.model_ema.restore(self.model.parameters())\n if context is not None:\n mainlogger.info(f\"{context}: Restored training weights\")\n\n def init_from_ckpt(self, path, ignore_keys=list(), only_model=False):\n sd = torch.load(path, map_location=\"cpu\")\n if \"state_dict\" in list(sd.keys()):\n sd = sd[\"state_dict\"]\n keys = list(sd.keys())\n for k in keys:\n for ik in ignore_keys:\n if k.startswith(ik):\n mainlogger.info(\"Deleting key {} from state_dict.\".format(k))\n del sd[k]\n missing, unexpected = self.load_state_dict(sd, strict=False) if not only_model else self.model.load_state_dict(\n sd, strict=False)\n mainlogger.info(f\"Restored from {path} with {len(missing)} missing and {len(unexpected)} unexpected keys\")\n if len(missing) > 0:\n mainlogger.info(f\"Missing Keys: {missing}\")\n if len(unexpected) > 0:\n mainlogger.info(f\"Unexpected Keys: {unexpected}\")\n\n def q_mean_variance(self, x_start, t):\n \"\"\"\n Get the distribution q(x_t | x_0).\n :param x_start: the [N x C x ...] tensor of noiseless inputs.\n :param t: the number of diffusion steps (minus 1). Here, 0 means one step.\n :return: A tuple (mean, variance, log_variance), all of x_start's shape.\n \"\"\"\n mean = (extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start)\n variance = extract_into_tensor(1.0 - self.alphas_cumprod, t, x_start.shape)\n log_variance = extract_into_tensor(self.log_one_minus_alphas_cumprod, t, x_start.shape)\n return mean, variance, log_variance\n\n def predict_start_from_noise(self, x_t, t, noise):\n return (\n extract_into_tensor(self.sqrt_recip_alphas_cumprod, t, x_t.shape) * x_t -\n extract_into_tensor(self.sqrt_recipm1_alphas_cumprod, t, x_t.shape) * noise\n )\n\n def q_posterior(self, x_start, x_t, t):\n posterior_mean = (\n extract_into_tensor(self.posterior_mean_coef1, t, x_t.shape) * x_start +\n extract_into_tensor(self.posterior_mean_coef2, t, x_t.shape) * x_t\n )\n posterior_variance = extract_into_tensor(self.posterior_variance, t, x_t.shape)\n posterior_log_variance_clipped = extract_into_tensor(self.posterior_log_variance_clipped, t, x_t.shape)\n return posterior_mean, posterior_variance, posterior_log_variance_clipped\n\n def p_mean_variance(self, x, t, clip_denoised: bool):\n model_out = self.model(x, t)\n if self.parameterization == \"eps\":\n x_recon = self.predict_start_from_noise(x, t=t, noise=model_out)\n elif self.parameterization == \"x0\":\n x_recon = model_out\n if clip_denoised:\n x_recon.clamp_(-1., 1.)\n\n model_mean, posterior_variance, posterior_log_variance = self.q_posterior(x_start=x_recon, x_t=x, t=t)\n return model_mean, posterior_variance, posterior_log_variance\n\n @torch.no_grad()\n def p_sample(self, x, t, clip_denoised=True, repeat_noise=False):\n b, *_, device = *x.shape, x.device\n model_mean, _, model_log_variance = self.p_mean_variance(x=x, t=t, clip_denoised=clip_denoised)\n noise = noise_like(x.shape, device, repeat_noise)\n # no noise when t == 0\n nonzero_mask = (1 - (t == 0).float()).reshape(b, *((1,) * (len(x.shape) - 1)))\n return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise\n\n @torch.no_grad()\n def p_sample_loop(self, shape, return_intermediates=False):\n device = self.betas.device\n b = shape[0]\n img = torch.randn(shape, device=device)\n intermediates = [img]\n for i in tqdm(reversed(range(0, self.num_timesteps)), desc='Sampling t', total=self.num_timesteps):\n img = self.p_sample(img, torch.full((b,), i, device=device, dtype=torch.long),\n clip_denoised=self.clip_denoised)\n if i % self.log_every_t == 0 or i == self.num_timesteps - 1:\n intermediates.append(img)\n if return_intermediates:\n return img, intermediates\n return img\n\n @torch.no_grad()\n def sample(self, batch_size=16, return_intermediates=False):\n image_size = self.image_size\n channels = self.channels\n return self.p_sample_loop((batch_size, channels, image_size, image_size),\n return_intermediates=return_intermediates)\n\n def q_sample(self, x_start, t, noise=None):\n noise = default(noise, lambda: torch.randn_like(x_start))\n return (extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start *\n extract_into_tensor(self.scale_arr, t, x_start.shape) +\n extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise)\n\n def get_input(self, batch, k):\n x = batch[k]\n x = x.to(memory_format=torch.contiguous_format).float()\n return x\n\n def _get_rows_from_list(self, samples):\n n_imgs_per_row = len(samples)\n denoise_grid = rearrange(samples, 'n b c h w -> b n c h w')\n denoise_grid = rearrange(denoise_grid, 'b n c h w -> (b n) c h w')\n denoise_grid = make_grid(denoise_grid, nrow=n_imgs_per_row)\n return denoise_grid\n\n @torch.no_grad()\n def log_images(self, batch, N=8, n_row=2, sample=True, return_keys=None, **kwargs):\n log = dict()\n x = self.get_input(batch, self.first_stage_key)\n N = min(x.shape[0], N)\n n_row = min(x.shape[0], n_row)\n x = x.to(self.device)[:N]\n log[\"inputs\"] = x\n\n # get diffusion row\n diffusion_row = list()\n x_start = x[:n_row]\n\n for t in range(self.num_timesteps):\n if t % self.log_every_t == 0 or t == self.num_timesteps - 1:\n t = repeat(torch.tensor([t]), '1 -> b', b=n_row)\n t = t.to(self.device).long()\n noise = torch.randn_like(x_start)\n x_noisy = self.q_sample(x_start=x_start, t=t, noise=noise)\n diffusion_row.append(x_noisy)\n\n log[\"diffusion_row\"] = self._get_rows_from_list(diffusion_row)\n\n if sample:\n # get denoise row\n with self.ema_scope(\"Plotting\"):\n samples, denoise_row = self.sample(batch_size=N, return_intermediates=True)\n\n log[\"samples\"] = samples\n log[\"denoise_row\"] = self._get_rows_from_list(denoise_row)\n\n if return_keys:\n if np.intersect1d(list(log.keys()), return_keys).shape[0] == 0:\n return log\n else:\n return {key: log[key] for key in return_keys}\n return log\n\n\nclass LatentDiffusion(DDPM):\n \"\"\"main class\"\"\"\n def __init__(self,\n first_stage_config,\n cond_stage_config,\n num_timesteps_cond=None,\n cond_stage_key=\"caption\",\n cond_stage_trainable=False,\n cond_stage_forward=None,\n conditioning_key=None,\n uncond_prob=0.2,\n uncond_type=\"empty_seq\",\n scale_factor=1.0,\n scale_by_std=False,\n encoder_type=\"2d\",\n only_model=False,\n use_scale=False,\n scale_a=1,\n scale_b=0.3,\n mid_step=400,\n fix_scale_bug=False,\n *args, **kwargs):\n self.num_timesteps_cond = default(num_timesteps_cond, 1)\n self.scale_by_std = scale_by_std\n assert self.num_timesteps_cond <= kwargs['timesteps']\n # for backwards compatibility after implementation of DiffusionWrapper\n ckpt_path = kwargs.pop(\"ckpt_path\", None)\n ignore_keys = kwargs.pop(\"ignore_keys\", [])\n conditioning_key = default(conditioning_key, 'crossattn')\n super().__init__(conditioning_key=conditioning_key, *args, **kwargs)\n\n self.cond_stage_trainable = cond_stage_trainable\n self.cond_stage_key = cond_stage_key\n\n # scale factor\n self.use_scale=use_scale\n if self.use_scale:\n self.scale_a=scale_a\n self.scale_b=scale_b\n if fix_scale_bug:\n scale_step=self.num_timesteps-mid_step\n else: #bug\n scale_step = self.num_timesteps\n\n scale_arr1 = np.linspace(scale_a, scale_b, mid_step)\n scale_arr2 = np.full(scale_step, scale_b)\n scale_arr = np.concatenate((scale_arr1, scale_arr2))\n scale_arr_prev = np.append(scale_a, scale_arr[:-1])\n to_torch = partial(torch.tensor, dtype=torch.float32)\n self.register_buffer('scale_arr', to_torch(scale_arr))\n\n try:\n self.num_downs = len(first_stage_config.params.ddconfig.ch_mult) - 1\n except:\n self.num_downs = 0\n if not scale_by_std:\n self.scale_factor = scale_factor\n else:\n self.register_buffer('scale_factor', torch.tensor(scale_factor))\n self.instantiate_first_stage(first_stage_config)\n self.instantiate_cond_stage(cond_stage_config)\n self.first_stage_config = first_stage_config\n self.cond_stage_config = cond_stage_config \n self.clip_denoised = False\n\n self.cond_stage_forward = cond_stage_forward\n self.encoder_type = encoder_type\n assert(encoder_type in [\"2d\", \"3d\"])\n self.uncond_prob = uncond_prob\n self.classifier_free_guidance = True if uncond_prob > 0 else False\n assert(uncond_type in [\"zero_embed\", \"empty_seq\"])\n self.uncond_type = uncond_type\n\n\n self.restarted_from_ckpt = False\n if ckpt_path is not None:\n self.init_from_ckpt(ckpt_path, ignore_keys, only_model=only_model)\n self.restarted_from_ckpt = True\n \n\n def make_cond_schedule(self, ):\n self.cond_ids = torch.full(size=(self.num_timesteps,), fill_value=self.num_timesteps - 1, dtype=torch.long)\n ids = torch.round(torch.linspace(0, self.num_timesteps - 1, self.num_timesteps_cond)).long()\n self.cond_ids[:self.num_timesteps_cond] = ids\n\n def q_sample(self, x_start, t, noise=None):\n noise = default(noise, lambda: torch.randn_like(x_start))\n if self.use_scale: \n return (extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start *\n extract_into_tensor(self.scale_arr, t, x_start.shape) +\n extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise)\n else:\n return (extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start +\n extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise)\n\n\n def _freeze_model(self):\n for name, para in self.model.diffusion_model.named_parameters():\n para.requires_grad = False\n\n def instantiate_first_stage(self, config):\n model = instantiate_from_config(config)\n self.first_stage_model = model.eval()\n self.first_stage_model.train = disabled_train\n for param in self.first_stage_model.parameters():\n param.requires_grad = False\n\n def instantiate_cond_stage(self, config):\n if not self.cond_stage_trainable:\n model = instantiate_from_config(config)\n self.cond_stage_model = model.eval()\n self.cond_stage_model.train = disabled_train\n for param in self.cond_stage_model.parameters():\n param.requires_grad = False\n else:\n model = instantiate_from_config(config)\n self.cond_stage_model = model\n \n def get_learned_conditioning(self, c):\n if self.cond_stage_forward is None:\n if hasattr(self.cond_stage_model, 'encode') and callable(self.cond_stage_model.encode):\n c = self.cond_stage_model.encode(c)\n if isinstance(c, DiagonalGaussianDistribution):\n c = c.mode()\n else:\n c = self.cond_stage_model(c)\n else:\n assert hasattr(self.cond_stage_model, self.cond_stage_forward)\n c = getattr(self.cond_stage_model, self.cond_stage_forward)(c)\n return c\n\n def get_first_stage_encoding(self, encoder_posterior, noise=None):\n if isinstance(encoder_posterior, DiagonalGaussianDistribution):\n z = encoder_posterior.sample(noise=noise)\n elif isinstance(encoder_posterior, torch.Tensor):\n z = encoder_posterior\n else:\n raise NotImplementedError(f\"encoder_posterior of type '{type(encoder_posterior)}' not yet implemented\")\n return self.scale_factor * z\n \n @torch.no_grad()\n def encode_first_stage(self, x):\n if self.encoder_type == \"2d\" and x.dim() == 5:\n b, _, t, _, _ = x.shape\n x = rearrange(x, 'b c t h w -> (b t) c h w')\n reshape_back = True\n else:\n reshape_back = False\n \n encoder_posterior = self.first_stage_model.encode(x)\n results = self.get_first_stage_encoding(encoder_posterior).detach()\n \n if reshape_back:\n results = rearrange(results, '(b t) c h w -> b c t h w', b=b,t=t)\n \n return results\n \n @torch.no_grad()\n def encode_first_stage_2DAE(self, x):\n\n b, _, t, _, _ = x.shape\n results = torch.cat([self.get_first_stage_encoding(self.first_stage_model.encode(x[:,:,i])).detach().unsqueeze(2) for i in range(t)], dim=2)\n \n return results\n \n def decode_core(self, z, **kwargs):\n if self.encoder_type == \"2d\" and z.dim() == 5:\n b, _, t, _, _ = z.shape\n z = rearrange(z, 'b c t h w -> (b t) c h w')\n reshape_back = True\n else:\n reshape_back = False\n \n z = 1. / self.scale_factor * z\n\n results = self.first_stage_model.decode(z, **kwargs)\n \n if reshape_back:\n results = rearrange(results, '(b t) c h w -> b c t h w', b=b,t=t)\n return results\n\n @torch.no_grad()\n def decode_first_stage(self, z, **kwargs):\n return self.decode_core(z, **kwargs)\n\n def apply_model(self, x_noisy, t, cond, **kwargs):\n if isinstance(cond, dict):\n # hybrid case, cond is exptected to be a dict\n pass\n else:\n if not isinstance(cond, list):\n cond = [cond]\n key = 'c_concat' if self.model.conditioning_key == 'concat' else 'c_crossattn'\n cond = {key: cond}\n\n x_recon = self.model(x_noisy, t, **cond, **kwargs)\n\n if isinstance(x_recon, tuple):\n return x_recon[0]\n else:\n return x_recon\n\n def _get_denoise_row_from_list(self, samples, desc=''):\n denoise_row = []\n for zd in tqdm(samples, desc=desc):\n denoise_row.append(self.decode_first_stage(zd.to(self.device)))\n n_log_timesteps = len(denoise_row)\n\n denoise_row = torch.stack(denoise_row) # n_log_timesteps, b, C, H, W\n \n if denoise_row.dim() == 5:\n # img, num_imgs= n_log_timesteps * bs, grid_size=[bs,n_log_timesteps]\n denoise_grid = rearrange(denoise_row, 'n b c h w -> b n c h w')\n denoise_grid = rearrange(denoise_grid, 'b n c h w -> (b n) c h w')\n denoise_grid = make_grid(denoise_grid, nrow=n_log_timesteps)\n elif denoise_row.dim() == 6:\n # video, grid_size=[n_log_timesteps*bs, t]\n video_length = denoise_row.shape[3]\n denoise_grid = rearrange(denoise_row, 'n b c t h w -> b n c t h w')\n denoise_grid = rearrange(denoise_grid, 'b n c t h w -> (b n) c t h w')\n denoise_grid = rearrange(denoise_grid, 'n c t h w -> (n t) c h w')\n denoise_grid = make_grid(denoise_grid, nrow=video_length)\n else:\n raise ValueError\n\n return denoise_grid\n \n\n @torch.no_grad()\n def decode_first_stage_2DAE(self, z, **kwargs):\n\n b, _, t, _, _ = z.shape\n z = 1. / self.scale_factor * z\n results = torch.cat([self.first_stage_model.decode(z[:,:,i], **kwargs).unsqueeze(2) for i in range(t)], dim=2)\n\n return results\n\n\n def p_mean_variance(self, x, c, t, clip_denoised: bool, return_x0=False, score_corrector=None, corrector_kwargs=None, **kwargs):\n t_in = t\n model_out = self.apply_model(x, t_in, c, **kwargs)\n\n if score_corrector is not None:\n assert self.parameterization == \"eps\"\n model_out = score_corrector.modify_score(self, model_out, x, t, c, **corrector_kwargs)\n\n if self.parameterization == \"eps\":\n x_recon = self.predict_start_from_noise(x, t=t, noise=model_out)\n elif self.parameterization == \"x0\":\n x_recon = model_out\n else:\n raise NotImplementedError()\n\n if clip_denoised:\n x_recon.clamp_(-1., 1.)\n\n model_mean, posterior_variance, posterior_log_variance = self.q_posterior(x_start=x_recon, x_t=x, t=t)\n\n if return_x0:\n return model_mean, posterior_variance, posterior_log_variance, x_recon\n else:\n return model_mean, posterior_variance, posterior_log_variance\n\n @torch.no_grad()\n def p_sample(self, x, c, t, clip_denoised=False, repeat_noise=False, return_x0=False, \\\n temperature=1., noise_dropout=0., score_corrector=None, corrector_kwargs=None, **kwargs):\n b, *_, device = *x.shape, x.device\n outputs = self.p_mean_variance(x=x, c=c, t=t, clip_denoised=clip_denoised, return_x0=return_x0, \\\n score_corrector=score_corrector, corrector_kwargs=corrector_kwargs, **kwargs)\n if return_x0:\n model_mean, _, model_log_variance, x0 = outputs\n else:\n model_mean, _, model_log_variance = outputs\n\n noise = noise_like(x.shape, device, repeat_noise) * temperature\n if noise_dropout > 0.:\n noise = torch.nn.functional.dropout(noise, p=noise_dropout)\n # no noise when t == 0\n nonzero_mask = (1 - (t == 0).float()).reshape(b, *((1,) * (len(x.shape) - 1)))\n\n if return_x0:\n return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise, x0\n else:\n return model_mean + nonzero_mask * (0.5 * model_log_variance).exp() * noise\n\n @torch.no_grad()\n def p_sample_loop(self, cond, shape, return_intermediates=False, x_T=None, verbose=True, callback=None, \\\n timesteps=None, mask=None, x0=None, img_callback=None, start_T=None, log_every_t=None, **kwargs):\n\n if not log_every_t:\n log_every_t = self.log_every_t\n device = self.betas.device\n b = shape[0] \n # sample an initial noise\n if x_T is None:\n img = torch.randn(shape, device=device)\n else:\n img = x_T\n\n intermediates = [img]\n if timesteps is None:\n timesteps = self.num_timesteps\n if start_T is not None:\n timesteps = min(timesteps, start_T)\n\n iterator = tqdm(reversed(range(0, timesteps)), desc='Sampling t', total=timesteps) if verbose else reversed(range(0, timesteps))\n\n if mask is not None:\n assert x0 is not None\n assert x0.shape[2:3] == mask.shape[2:3] # spatial size has to match\n\n for i in iterator:\n ts = torch.full((b,), i, device=device, dtype=torch.long)\n if self.shorten_cond_schedule:\n assert self.model.conditioning_key != 'hybrid'\n tc = self.cond_ids[ts].to(cond.device)\n cond = self.q_sample(x_start=cond, t=tc, noise=torch.randn_like(cond))\n\n img = self.p_sample(img, cond, ts, clip_denoised=self.clip_denoised, **kwargs)\n if mask is not None:\n img_orig = self.q_sample(x0, ts)\n img = img_orig * mask + (1. - mask) * img\n\n if i % log_every_t == 0 or i == timesteps - 1:\n intermediates.append(img)\n if callback: callback(i)\n if img_callback: img_callback(img, i)\n\n if return_intermediates:\n return img, intermediates\n return img\n\n\nclass LatentVisualDiffusion(LatentDiffusion):\n def __init__(self, cond_img_config, finegrained=False, random_cond=False, *args, **kwargs):\n super().__init__(*args, **kwargs)\n self.random_cond = random_cond\n self.instantiate_img_embedder(cond_img_config, freeze=True)\n num_tokens = 16 if finegrained else 4\n self.image_proj_model = self.init_projector(use_finegrained=finegrained, num_tokens=num_tokens, input_dim=1024,\\\n cross_attention_dim=1024, dim=1280) \n\n def instantiate_img_embedder(self, config, freeze=True):\n embedder = instantiate_from_config(config)\n if freeze:\n self.embedder = embedder.eval()\n self.embedder.train = disabled_train\n for param in self.embedder.parameters():\n param.requires_grad = False\n\n def init_projector(self, use_finegrained, num_tokens, input_dim, cross_attention_dim, dim):\n if not use_finegrained:\n image_proj_model = ImageProjModel(clip_extra_context_tokens=num_tokens, cross_attention_dim=cross_attention_dim,\n clip_embeddings_dim=input_dim\n )\n else:\n image_proj_model = Resampler(dim=input_dim, depth=4, dim_head=64, heads=12, num_queries=num_tokens,\n embedding_dim=dim, output_dim=cross_attention_dim, ff_mult=4\n )\n return image_proj_model\n\n ## Never delete this func: it is used in log_images() and inference stage\n def get_image_embeds(self, batch_imgs):\n ## img: b c h w\n img_token = self.embedder(batch_imgs)\n img_emb = self.image_proj_model(img_token)\n return img_emb\n\n\nclass DiffusionWrapper(pl.LightningModule):\n def __init__(self, diff_model_config, conditioning_key):\n super().__init__()\n self.diffusion_model = instantiate_from_config(diff_model_config)\n self.conditioning_key = conditioning_key\n\n def forward(self, x, t, c_concat: list = None, c_crossattn: list = None,\n c_adm=None, s=None, mask=None, **kwargs):\n # temporal_context = fps is foNone\n if self.conditioning_key is None:\n out = self.diffusion_model(x, t)\n elif self.conditioning_key == 'concat':\n xc = torch.cat([x] + c_concat, dim=1)\n out = self.diffusion_model(xc, t, **kwargs)\n elif self.conditioning_key == 'crossattn':\n cc = torch.cat(c_crossattn, 1)\n out = self.diffusion_model(x, t, context=cc, **kwargs)\n elif self.conditioning_key == 'hybrid':\n ## it is just right [b,c,t,h,w]: concatenate in channel dim\n xc = torch.cat([x] + c_concat, dim=1)\n cc = torch.cat(c_crossattn, 1)\n out = self.diffusion_model(xc, t, context=cc)\n elif self.conditioning_key == 'resblockcond':\n cc = c_crossattn[0]\n out = self.diffusion_model(x, t, context=cc)\n elif self.conditioning_key == 'adm':\n cc = c_crossattn[0]\n out = self.diffusion_model(x, t, y=cc)\n elif self.conditioning_key == 'hybrid-adm':\n assert c_adm is not None\n xc = torch.cat([x] + c_concat, dim=1)\n cc = torch.cat(c_crossattn, 1)\n out = self.diffusion_model(xc, t, context=cc, y=c_adm)\n elif self.conditioning_key == 'hybrid-time':\n assert s is not None\n xc = torch.cat([x] + c_concat, dim=1)\n cc = torch.cat(c_crossattn, 1)\n out = self.diffusion_model(xc, t, context=cc, s=s)\n elif self.conditioning_key == 'concat-time-mask':\n # assert s is not None\n # mainlogger.info('x & mask:',x.shape,c_concat[0].shape)\n xc = torch.cat([x] + c_concat, dim=1)\n out = self.diffusion_model(xc, t, context=None, s=s, mask=mask)\n elif self.conditioning_key == 'concat-adm-mask':\n # assert s is not None\n # mainlogger.info('x & mask:',x.shape,c_concat[0].shape)\n if c_concat is not None:\n xc = torch.cat([x] + c_concat, dim=1)\n else:\n xc = x\n out = self.diffusion_model(xc, t, context=None, y=s, mask=mask)\n elif self.conditioning_key == 'hybrid-adm-mask':\n cc = torch.cat(c_crossattn, 1)\n if c_concat is not None:\n xc = torch.cat([x] + c_concat, dim=1)\n else:\n xc = x\n out = self.diffusion_model(xc, t, context=cc, y=s, mask=mask)\n elif self.conditioning_key == 'hybrid-time-adm': # adm means y, e.g., class index\n # assert s is not None\n assert c_adm is not None\n xc = torch.cat([x] + c_concat, dim=1)\n cc = torch.cat(c_crossattn, 1)\n out = self.diffusion_model(xc, t, context=cc, s=s, y=c_adm)\n else:\n raise NotImplementedError()\n\n return out" }, { "path": "lvdm/models/samplers/ddim.py", "content": "import numpy as np\nfrom tqdm import tqdm\nimport torch\nfrom lvdm.models.utils_diffusion import make_ddim_sampling_parameters, make_ddim_timesteps\nfrom lvdm.common import noise_like\n\n\nclass DDIMSampler(object):\n def __init__(self, model, schedule=\"linear\", **kwargs):\n super().__init__()\n self.model = model\n self.ddpm_num_timesteps = model.num_timesteps\n self.schedule = schedule\n self.counter = 0\n\n def register_buffer(self, name, attr):\n if type(attr) == torch.Tensor:\n if attr.device != torch.device(\"cuda\"):\n attr = attr.to(torch.device(\"cuda\"))\n setattr(self, name, attr)\n\n def make_schedule(self, ddim_num_steps, ddim_discretize=\"uniform\", ddim_eta=0., verbose=True):\n self.ddim_timesteps = make_ddim_timesteps(ddim_discr_method=ddim_discretize, num_ddim_timesteps=ddim_num_steps,\n num_ddpm_timesteps=self.ddpm_num_timesteps,verbose=verbose)\n alphas_cumprod = self.model.alphas_cumprod\n assert alphas_cumprod.shape[0] == self.ddpm_num_timesteps, 'alphas have to be defined for each timestep'\n to_torch = lambda x: x.clone().detach().to(torch.float32).to(self.model.device)\n\n self.register_buffer('betas', to_torch(self.model.betas))\n self.register_buffer('alphas_cumprod', to_torch(alphas_cumprod))\n self.register_buffer('alphas_cumprod_prev', to_torch(self.model.alphas_cumprod_prev))\n self.use_scale = self.model.use_scale\n print('DDIM scale', self.use_scale)\n\n if self.use_scale:\n self.register_buffer('scale_arr', to_torch(self.model.scale_arr))\n ddim_scale_arr = self.scale_arr.cpu()[self.ddim_timesteps]\n self.register_buffer('ddim_scale_arr', ddim_scale_arr)\n ddim_scale_arr = np.asarray([self.scale_arr.cpu()[0]] + self.scale_arr.cpu()[self.ddim_timesteps[:-1]].tolist())\n self.register_buffer('ddim_scale_arr_prev', ddim_scale_arr)\n\n # calculations for diffusion q(x_t | x_{t-1}) and others\n self.register_buffer('sqrt_alphas_cumprod', to_torch(np.sqrt(alphas_cumprod.cpu())))\n self.register_buffer('sqrt_one_minus_alphas_cumprod', to_torch(np.sqrt(1. - alphas_cumprod.cpu())))\n self.register_buffer('log_one_minus_alphas_cumprod', to_torch(np.log(1. - alphas_cumprod.cpu())))\n self.register_buffer('sqrt_recip_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod.cpu())))\n self.register_buffer('sqrt_recipm1_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod.cpu() - 1)))\n\n # ddim sampling parameters\n ddim_sigmas, ddim_alphas, ddim_alphas_prev = make_ddim_sampling_parameters(alphacums=alphas_cumprod.cpu(),\n ddim_timesteps=self.ddim_timesteps,\n eta=ddim_eta,verbose=verbose)\n self.register_buffer('ddim_sigmas', ddim_sigmas)\n self.register_buffer('ddim_alphas', ddim_alphas)\n self.register_buffer('ddim_alphas_prev', ddim_alphas_prev)\n self.register_buffer('ddim_sqrt_one_minus_alphas', np.sqrt(1. - ddim_alphas))\n sigmas_for_original_sampling_steps = ddim_eta * torch.sqrt(\n (1 - self.alphas_cumprod_prev) / (1 - self.alphas_cumprod) * (\n 1 - self.alphas_cumprod / self.alphas_cumprod_prev))\n self.register_buffer('ddim_sigmas_for_original_num_steps', sigmas_for_original_sampling_steps)\n\n @torch.no_grad()\n def sample(self,\n S,\n batch_size,\n shape,\n conditioning=None,\n callback=None,\n normals_sequence=None,\n img_callback=None,\n quantize_x0=False,\n eta=0.,\n mask=None,\n x0=None,\n temperature=1.,\n noise_dropout=0.,\n score_corrector=None,\n corrector_kwargs=None,\n verbose=True,\n schedule_verbose=False,\n x_T=None,\n log_every_t=100,\n unconditional_guidance_scale=1.,\n unconditional_conditioning=None,\n # this has to come in the same format as the conditioning, # e.g. as encoded tokens, ...\n **kwargs\n ):\n \n # check condition bs\n if conditioning is not None:\n if isinstance(conditioning, dict):\n try:\n cbs = conditioning[list(conditioning.keys())[0]].shape[0]\n except:\n cbs = conditioning[list(conditioning.keys())[0]][0].shape[0]\n\n if cbs != batch_size:\n print(f\"Warning: Got {cbs} conditionings but batch-size is {batch_size}\")\n else:\n if conditioning.shape[0] != batch_size:\n print(f\"Warning: Got {conditioning.shape[0]} conditionings but batch-size is {batch_size}\")\n\n self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=schedule_verbose)\n \n # make shape\n if len(shape) == 3:\n C, H, W = shape\n size = (batch_size, C, H, W)\n elif len(shape) == 4:\n C, T, H, W = shape\n size = (batch_size, C, T, H, W)\n # print(f'Data shape for DDIM sampling is {size}, eta {eta}')\n \n samples, intermediates = self.ddim_sampling(conditioning, size,\n callback=callback,\n img_callback=img_callback,\n quantize_denoised=quantize_x0,\n mask=mask, x0=x0,\n ddim_use_original_steps=False,\n noise_dropout=noise_dropout,\n temperature=temperature,\n score_corrector=score_corrector,\n corrector_kwargs=corrector_kwargs,\n x_T=x_T,\n log_every_t=log_every_t,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning,\n verbose=verbose,\n **kwargs)\n return samples, intermediates\n\n @torch.no_grad()\n def ddim_sampling(self, cond, shape,\n x_T=None, ddim_use_original_steps=False,\n callback=None, timesteps=None, quantize_denoised=False,\n mask=None, x0=None, img_callback=None, log_every_t=100,\n temperature=1., noise_dropout=0., score_corrector=None, corrector_kwargs=None,\n unconditional_guidance_scale=1., unconditional_conditioning=None, verbose=True,\n cond_tau=1., target_size=None, start_timesteps=None,\n **kwargs):\n device = self.model.betas.device \n print('ddim device', device)\n b = shape[0]\n if x_T is None:\n img = torch.randn(shape, device=device)\n else:\n img = x_T\n \n if timesteps is None:\n timesteps = self.ddpm_num_timesteps if ddim_use_original_steps else self.ddim_timesteps\n elif timesteps is not None and not ddim_use_original_steps:\n subset_end = int(min(timesteps / self.ddim_timesteps.shape[0], 1) * self.ddim_timesteps.shape[0]) - 1\n timesteps = self.ddim_timesteps[:subset_end]\n \n intermediates = {'x_inter': [img], 'pred_x0': [img]}\n time_range = reversed(range(0,timesteps)) if ddim_use_original_steps else np.flip(timesteps)\n total_steps = timesteps if ddim_use_original_steps else timesteps.shape[0]\n if verbose:\n iterator = tqdm(time_range, desc='DDIM Sampler', total=total_steps)\n else:\n iterator = time_range\n\n init_x0 = False\n clean_cond = kwargs.pop(\"clean_cond\", False)\n for i, step in enumerate(iterator):\n index = total_steps - i - 1\n ts = torch.full((b,), step, device=device, dtype=torch.long)\n if start_timesteps is not None:\n assert x0 is not None\n if step > start_timesteps*time_range[0]:\n continue\n elif not init_x0:\n img = self.model.q_sample(x0, ts) \n init_x0 = True\n\n # use mask to blend noised original latent (img_orig) & new sampled latent (img)\n if mask is not None:\n assert x0 is not None\n if clean_cond:\n img_orig = x0\n else:\n img_orig = self.model.q_sample(x0, ts) # TODO: deterministic forward pass? \n img = img_orig * mask + (1. - mask) * img # keep original & modify use img\n \n index_clip = int((1 - cond_tau) * total_steps)\n if index <= index_clip and target_size is not None:\n target_size_ = [target_size[0], target_size[1]//8, target_size[2]//8]\n img = torch.nn.functional.interpolate(\n img,\n size=target_size_,\n mode=\"nearest\",\n )\n outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,\n quantize_denoised=quantize_denoised, temperature=temperature,\n noise_dropout=noise_dropout, score_corrector=score_corrector,\n corrector_kwargs=corrector_kwargs,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning,\n x0=x0,\n **kwargs)\n \n img, pred_x0 = outs\n if callback: callback(i)\n if img_callback: img_callback(pred_x0, i)\n\n if index % log_every_t == 0 or index == total_steps - 1:\n intermediates['x_inter'].append(img)\n intermediates['pred_x0'].append(pred_x0)\n\n return img, intermediates\n\n @torch.no_grad()\n def p_sample_ddim(self, x, c, t, index, repeat_noise=False, use_original_steps=False, quantize_denoised=False,\n temperature=1., noise_dropout=0., score_corrector=None, corrector_kwargs=None,\n unconditional_guidance_scale=1., unconditional_conditioning=None,\n uc_type=None, conditional_guidance_scale_temporal=None, **kwargs):\n b, *_, device = *x.shape, x.device\n if x.dim() == 5:\n is_video = True\n else:\n is_video = False\n if unconditional_conditioning is None or unconditional_guidance_scale == 1.:\n e_t = self.model.apply_model(x, t, c, **kwargs) # unet denoiser\n else:\n # with unconditional condition\n if isinstance(c, torch.Tensor):\n e_t = self.model.apply_model(x, t, c, **kwargs)\n e_t_uncond = self.model.apply_model(x, t, unconditional_conditioning, **kwargs)\n elif isinstance(c, dict):\n e_t = self.model.apply_model(x, t, c, **kwargs)\n e_t_uncond = self.model.apply_model(x, t, unconditional_conditioning, **kwargs)\n else:\n raise NotImplementedError\n # text cfg\n if uc_type is None:\n e_t = e_t_uncond + unconditional_guidance_scale * (e_t - e_t_uncond)\n else:\n if uc_type == 'cfg_original':\n e_t = e_t + unconditional_guidance_scale * (e_t - e_t_uncond)\n elif uc_type == 'cfg_ours':\n e_t = e_t + unconditional_guidance_scale * (e_t_uncond - e_t)\n else:\n raise NotImplementedError\n # temporal guidance\n if conditional_guidance_scale_temporal is not None:\n e_t_temporal = self.model.apply_model(x, t, c, **kwargs)\n e_t_image = self.model.apply_model(x, t, c, no_temporal_attn=True, **kwargs)\n e_t = e_t + conditional_guidance_scale_temporal * (e_t_temporal - e_t_image)\n\n if score_corrector is not None:\n assert self.model.parameterization == \"eps\"\n e_t = score_corrector.modify_score(self.model, e_t, x, t, c, **corrector_kwargs)\n\n alphas = self.model.alphas_cumprod if use_original_steps else self.ddim_alphas\n alphas_prev = self.model.alphas_cumprod_prev if use_original_steps else self.ddim_alphas_prev\n sqrt_one_minus_alphas = self.model.sqrt_one_minus_alphas_cumprod if use_original_steps else self.ddim_sqrt_one_minus_alphas\n sigmas = self.model.ddim_sigmas_for_original_num_steps if use_original_steps else self.ddim_sigmas\n # select parameters corresponding to the currently considered timestep\n \n if is_video:\n size = (b, 1, 1, 1, 1)\n else:\n size = (b, 1, 1, 1)\n a_t = torch.full(size, alphas[index], device=device)\n a_prev = torch.full(size, alphas_prev[index], device=device)\n sigma_t = torch.full(size, sigmas[index], device=device)\n sqrt_one_minus_at = torch.full(size, sqrt_one_minus_alphas[index],device=device)\n\n # current prediction for x_0\n pred_x0 = (x - sqrt_one_minus_at * e_t) / a_t.sqrt()\n if quantize_denoised:\n pred_x0, _, *_ = self.model.first_stage_model.quantize(pred_x0)\n # direction pointing to x_t\n dir_xt = (1. - a_prev - sigma_t**2).sqrt() * e_t\n\n noise = sigma_t * noise_like(x.shape, device, repeat_noise) * temperature\n if noise_dropout > 0.:\n noise = torch.nn.functional.dropout(noise, p=noise_dropout)\n \n alphas = self.model.alphas_cumprod if use_original_steps else self.ddim_alphas\n if self.use_scale:\n scale_arr = self.model.scale_arr if use_original_steps else self.ddim_scale_arr\n scale_t = torch.full(size, scale_arr[index], device=device)\n scale_arr_prev = self.model.scale_arr_prev if use_original_steps else self.ddim_scale_arr_prev\n scale_t_prev = torch.full(size, scale_arr_prev[index], device=device)\n pred_x0 /= scale_t \n x_prev = a_prev.sqrt() * scale_t_prev * pred_x0 + dir_xt + noise\n else:\n x_prev = a_prev.sqrt() * pred_x0 + dir_xt + noise\n\n return x_prev, pred_x0\n\n\n @torch.no_grad()\n def stochastic_encode(self, x0, t, use_original_steps=False, noise=None):\n # fast, but does not allow for exact reconstruction\n # t serves as an index to gather the correct alphas\n if use_original_steps:\n sqrt_alphas_cumprod = self.sqrt_alphas_cumprod\n sqrt_one_minus_alphas_cumprod = self.sqrt_one_minus_alphas_cumprod\n else:\n sqrt_alphas_cumprod = torch.sqrt(self.ddim_alphas)\n sqrt_one_minus_alphas_cumprod = self.ddim_sqrt_one_minus_alphas\n\n if noise is None:\n noise = torch.randn_like(x0)\n\n def extract_into_tensor(a, t, x_shape):\n b, *_ = t.shape\n out = a.gather(-1, t)\n return out.reshape(b, *((1,) * (len(x_shape) - 1)))\n\n return (extract_into_tensor(sqrt_alphas_cumprod, t, x0.shape) * x0 +\n extract_into_tensor(sqrt_one_minus_alphas_cumprod, t, x0.shape) * noise)\n\n @torch.no_grad()\n def decode(self, x_latent, cond, t_start, unconditional_guidance_scale=1.0, unconditional_conditioning=None,\n use_original_steps=False):\n\n timesteps = np.arange(self.ddpm_num_timesteps) if use_original_steps else self.ddim_timesteps\n timesteps = timesteps[:t_start]\n\n time_range = np.flip(timesteps)\n total_steps = timesteps.shape[0]\n print(f\"Running DDIM Sampling with {total_steps} timesteps\")\n\n iterator = tqdm(time_range, desc='Decoding image', total=total_steps)\n x_dec = x_latent\n for i, step in enumerate(iterator):\n index = total_steps - i - 1\n ts = torch.full((x_latent.shape[0],), step, device=x_latent.device, dtype=torch.long)\n x_dec, _ = self.p_sample_ddim(x_dec, cond, ts, index=index, use_original_steps=use_original_steps,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning)\n return x_dec\n\n" }, { "path": "lvdm/models/samplers/ddim_mp.py", "content": "import numpy as np\nfrom tqdm import tqdm\nimport torch\nfrom lvdm.models.utils_diffusion import make_ddim_sampling_parameters, make_ddim_timesteps\nfrom lvdm.common import noise_like\n\n\nclass DDIMSampler(object):\n def __init__(self, model, schedule=\"linear\", **kwargs):\n super().__init__()\n self.model = model\n self.ddpm_num_timesteps = model.num_timesteps\n self.schedule = schedule\n self.counter = 0\n\n def register_buffer(self, name, attr):\n if type(attr) == torch.Tensor:\n if attr.device != torch.device(\"cuda\"):\n attr = attr.to(torch.device(\"cuda\"))\n setattr(self, name, attr)\n\n def make_schedule(self, ddim_num_steps, ddim_discretize=\"uniform\", ddim_eta=0., verbose=True):\n self.ddim_timesteps = make_ddim_timesteps(ddim_discr_method=ddim_discretize, num_ddim_timesteps=ddim_num_steps,\n num_ddpm_timesteps=self.ddpm_num_timesteps,verbose=verbose)\n alphas_cumprod = self.model.alphas_cumprod\n assert alphas_cumprod.shape[0] == self.ddpm_num_timesteps, 'alphas have to be defined for each timestep'\n to_torch = lambda x: x.clone().detach().to(torch.float32).to(self.model.device)\n\n self.register_buffer('betas', to_torch(self.model.betas))\n self.register_buffer('alphas_cumprod', to_torch(alphas_cumprod))\n self.register_buffer('alphas_cumprod_prev', to_torch(self.model.alphas_cumprod_prev))\n self.use_scale = self.model.use_scale\n print('DDIM scale', self.use_scale)\n\n if self.use_scale:\n self.register_buffer('scale_arr', to_torch(self.model.scale_arr))\n ddim_scale_arr = self.scale_arr.cpu()[self.ddim_timesteps]\n self.register_buffer('ddim_scale_arr', ddim_scale_arr)\n ddim_scale_arr = np.asarray([self.scale_arr.cpu()[0]] + self.scale_arr.cpu()[self.ddim_timesteps[:-1]].tolist())\n self.register_buffer('ddim_scale_arr_prev', ddim_scale_arr)\n\n # calculations for diffusion q(x_t | x_{t-1}) and others\n self.register_buffer('sqrt_alphas_cumprod', to_torch(np.sqrt(alphas_cumprod.cpu())))\n self.register_buffer('sqrt_one_minus_alphas_cumprod', to_torch(np.sqrt(1. - alphas_cumprod.cpu())))\n self.register_buffer('log_one_minus_alphas_cumprod', to_torch(np.log(1. - alphas_cumprod.cpu())))\n self.register_buffer('sqrt_recip_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod.cpu())))\n self.register_buffer('sqrt_recipm1_alphas_cumprod', to_torch(np.sqrt(1. / alphas_cumprod.cpu() - 1)))\n\n # ddim sampling parameters\n ddim_sigmas, ddim_alphas, ddim_alphas_prev = make_ddim_sampling_parameters(alphacums=alphas_cumprod.cpu(),\n ddim_timesteps=self.ddim_timesteps,\n eta=ddim_eta,verbose=verbose)\n self.register_buffer('ddim_sigmas', ddim_sigmas)\n self.register_buffer('ddim_alphas', ddim_alphas)\n self.register_buffer('ddim_alphas_prev', ddim_alphas_prev)\n self.register_buffer('ddim_sqrt_one_minus_alphas', np.sqrt(1. - ddim_alphas))\n sigmas_for_original_sampling_steps = ddim_eta * torch.sqrt(\n (1 - self.alphas_cumprod_prev) / (1 - self.alphas_cumprod) * (\n 1 - self.alphas_cumprod / self.alphas_cumprod_prev))\n self.register_buffer('ddim_sigmas_for_original_num_steps', sigmas_for_original_sampling_steps)\n\n @torch.no_grad()\n def sample(self,\n S,\n batch_size,\n shape,\n conditioning=None,\n callback=None,\n normals_sequence=None,\n img_callback=None,\n quantize_x0=False,\n eta=0.,\n mask=None,\n x0=None,\n temperature=1.,\n noise_dropout=0.,\n score_corrector=None,\n corrector_kwargs=None,\n verbose=True,\n schedule_verbose=False,\n x_T=None,\n log_every_t=100,\n unconditional_guidance_scale=1.,\n unconditional_conditioning=None,\n # this has to come in the same format as the conditioning, # e.g. as encoded tokens, ...\n **kwargs\n ):\n \n # check condition bs\n # if conditioning is not None:\n # if isinstance(conditioning, dict):\n # try:\n # cbs = conditioning[list(conditioning.keys())[0]].shape[0]\n # except:\n # cbs = conditioning[list(conditioning.keys())[0]][0].shape[0]\n\n # if cbs != batch_size:\n # print(f\"Warning: Got {cbs} conditionings but batch-size is {batch_size}\")\n # else:\n # if conditioning.shape[0] != batch_size:\n # print(f\"Warning: Got {conditioning.shape[0]} conditionings but batch-size is {batch_size}\")\n\n self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=schedule_verbose)\n \n # make shape\n if len(shape) == 3:\n C, H, W = shape\n size = (batch_size, C, H, W)\n elif len(shape) == 4:\n C, T, H, W = shape\n size = (batch_size, C, T, H, W)\n # print(f'Data shape for DDIM sampling is {size}, eta {eta}')\n \n samples, intermediates = self.ddim_sampling(conditioning, size,\n callback=callback,\n img_callback=img_callback,\n quantize_denoised=quantize_x0,\n mask=mask, x0=x0,\n ddim_use_original_steps=False,\n noise_dropout=noise_dropout,\n temperature=temperature,\n score_corrector=score_corrector,\n corrector_kwargs=corrector_kwargs,\n x_T=x_T,\n log_every_t=log_every_t,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning,\n verbose=verbose,\n **kwargs)\n return samples, intermediates\n\n @torch.no_grad()\n def ddim_sampling(self, cond, shape,\n x_T=None, ddim_use_original_steps=False,\n callback=None, timesteps=None, quantize_denoised=False,\n mask=None, x0=None, img_callback=None, log_every_t=100,\n temperature=1., noise_dropout=0., score_corrector=None, corrector_kwargs=None,\n unconditional_guidance_scale=1., unconditional_conditioning=None, verbose=True,\n cond_tau=1., target_size=None, start_timesteps=None,\n **kwargs):\n device = self.model.betas.device \n print('ddim device', device)\n b = shape[0]\n if x_T is None:\n img = torch.randn(shape, device=device)\n else:\n img = x_T\n \n if timesteps is None:\n timesteps = self.ddpm_num_timesteps if ddim_use_original_steps else self.ddim_timesteps\n elif timesteps is not None and not ddim_use_original_steps:\n subset_end = int(min(timesteps / self.ddim_timesteps.shape[0], 1) * self.ddim_timesteps.shape[0]) - 1\n timesteps = self.ddim_timesteps[:subset_end]\n \n intermediates = {'x_inter': [img], 'pred_x0': [img]}\n time_range = reversed(range(0,timesteps)) if ddim_use_original_steps else np.flip(timesteps)\n total_steps = timesteps if ddim_use_original_steps else timesteps.shape[0]\n if verbose:\n iterator = tqdm(time_range, desc='DDIM Sampler', total=total_steps)\n else:\n iterator = time_range\n\n init_x0 = False\n clean_cond = kwargs.pop(\"clean_cond\", False)\n for i, step in enumerate(iterator):\n index = total_steps - i - 1\n ts = torch.full((b,), step, device=device, dtype=torch.long)\n if start_timesteps is not None:\n assert x0 is not None\n if step > start_timesteps*time_range[0]:\n continue\n elif not init_x0:\n img = self.model.q_sample(x0, ts) \n init_x0 = True\n\n # use mask to blend noised original latent (img_orig) & new sampled latent (img)\n if mask is not None:\n assert x0 is not None\n if clean_cond:\n img_orig = x0\n else:\n img_orig = self.model.q_sample(x0, ts) # TODO: deterministic forward pass? \n img = img_orig * mask + (1. - mask) * img # keep original & modify use img\n \n index_clip = int((1 - cond_tau) * total_steps)\n if index <= index_clip and target_size is not None:\n target_size_ = [target_size[0], target_size[1]//8, target_size[2]//8]\n img = torch.nn.functional.interpolate(\n img,\n size=target_size_,\n mode=\"nearest\",\n )\n outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,\n quantize_denoised=quantize_denoised, temperature=temperature,\n noise_dropout=noise_dropout, score_corrector=score_corrector,\n corrector_kwargs=corrector_kwargs,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning,\n x0=x0,\n step=i,\n **kwargs)\n \n img, pred_x0 = outs\n if callback: callback(i)\n if img_callback: img_callback(pred_x0, i)\n\n if index % log_every_t == 0 or index == total_steps - 1:\n intermediates['x_inter'].append(img)\n intermediates['pred_x0'].append(pred_x0)\n\n return img, intermediates\n\n @torch.no_grad()\n def p_sample_ddim(self, x, c, t, index, repeat_noise=False, use_original_steps=False, quantize_denoised=False,\n temperature=1., noise_dropout=0., score_corrector=None, corrector_kwargs=None,\n unconditional_guidance_scale=1., unconditional_conditioning=None,\n uc_type=None, conditional_guidance_scale_temporal=None, step=0, **kwargs):\n b, *_, device = *x.shape, x.device\n if x.dim() == 5:\n is_video = True\n else:\n is_video = False\n if unconditional_conditioning is None or unconditional_guidance_scale == 1.:\n e_t = self.model.apply_model(x, t, c, **kwargs) # unet denoiser\n else:\n # with unconditional condition\n if step < 5 or step > 15:\n e_t = self.model.apply_model(x, t, c, use_injection=True, **kwargs)\n e_t_uncond = self.model.apply_model(x, t, unconditional_conditioning, **kwargs)\n elif isinstance(c, torch.Tensor):\n e_t = self.model.apply_model(x, t, c, **kwargs)\n e_t_uncond = self.model.apply_model(x, t, unconditional_conditioning, **kwargs)\n elif isinstance(c, dict):\n e_t = self.model.apply_model(x, t, c, **kwargs)\n e_t_uncond = self.model.apply_model(x, t, unconditional_conditioning, **kwargs)\n else:\n raise NotImplementedError\n # text cfg\n if uc_type is None:\n e_t = e_t_uncond + unconditional_guidance_scale * (e_t - e_t_uncond)\n else:\n if uc_type == 'cfg_original':\n e_t = e_t + unconditional_guidance_scale * (e_t - e_t_uncond)\n elif uc_type == 'cfg_ours':\n e_t = e_t + unconditional_guidance_scale * (e_t_uncond - e_t)\n else:\n raise NotImplementedError\n # temporal guidance\n if conditional_guidance_scale_temporal is not None:\n e_t_temporal = self.model.apply_model(x, t, c, **kwargs)\n e_t_image = self.model.apply_model(x, t, c, no_temporal_attn=True, **kwargs)\n e_t = e_t + conditional_guidance_scale_temporal * (e_t_temporal - e_t_image)\n\n if score_corrector is not None:\n assert self.model.parameterization == \"eps\"\n e_t = score_corrector.modify_score(self.model, e_t, x, t, c, **corrector_kwargs)\n\n alphas = self.model.alphas_cumprod if use_original_steps else self.ddim_alphas\n alphas_prev = self.model.alphas_cumprod_prev if use_original_steps else self.ddim_alphas_prev\n sqrt_one_minus_alphas = self.model.sqrt_one_minus_alphas_cumprod if use_original_steps else self.ddim_sqrt_one_minus_alphas\n sigmas = self.model.ddim_sigmas_for_original_num_steps if use_original_steps else self.ddim_sigmas\n # select parameters corresponding to the currently considered timestep\n \n if is_video:\n size = (b, 1, 1, 1, 1)\n else:\n size = (b, 1, 1, 1)\n a_t = torch.full(size, alphas[index], device=device)\n a_prev = torch.full(size, alphas_prev[index], device=device)\n sigma_t = torch.full(size, sigmas[index], device=device)\n sqrt_one_minus_at = torch.full(size, sqrt_one_minus_alphas[index],device=device)\n\n # current prediction for x_0\n pred_x0 = (x - sqrt_one_minus_at * e_t) / a_t.sqrt()\n if quantize_denoised:\n pred_x0, _, *_ = self.model.first_stage_model.quantize(pred_x0)\n # direction pointing to x_t\n dir_xt = (1. - a_prev - sigma_t**2).sqrt() * e_t\n\n noise = sigma_t * noise_like(x.shape, device, repeat_noise) * temperature\n if noise_dropout > 0.:\n noise = torch.nn.functional.dropout(noise, p=noise_dropout)\n \n alphas = self.model.alphas_cumprod if use_original_steps else self.ddim_alphas\n if self.use_scale:\n scale_arr = self.model.scale_arr if use_original_steps else self.ddim_scale_arr\n scale_t = torch.full(size, scale_arr[index], device=device)\n scale_arr_prev = self.model.scale_arr_prev if use_original_steps else self.ddim_scale_arr_prev\n scale_t_prev = torch.full(size, scale_arr_prev[index], device=device)\n pred_x0 /= scale_t \n x_prev = a_prev.sqrt() * scale_t_prev * pred_x0 + dir_xt + noise\n else:\n x_prev = a_prev.sqrt() * pred_x0 + dir_xt + noise\n\n return x_prev, pred_x0\n\n\n @torch.no_grad()\n def stochastic_encode(self, x0, t, use_original_steps=False, noise=None):\n # fast, but does not allow for exact reconstruction\n # t serves as an index to gather the correct alphas\n if use_original_steps:\n sqrt_alphas_cumprod = self.sqrt_alphas_cumprod\n sqrt_one_minus_alphas_cumprod = self.sqrt_one_minus_alphas_cumprod\n else:\n sqrt_alphas_cumprod = torch.sqrt(self.ddim_alphas)\n sqrt_one_minus_alphas_cumprod = self.ddim_sqrt_one_minus_alphas\n\n if noise is None:\n noise = torch.randn_like(x0)\n\n def extract_into_tensor(a, t, x_shape):\n b, *_ = t.shape\n out = a.gather(-1, t)\n return out.reshape(b, *((1,) * (len(x_shape) - 1)))\n\n return (extract_into_tensor(sqrt_alphas_cumprod, t, x0.shape) * x0 +\n extract_into_tensor(sqrt_one_minus_alphas_cumprod, t, x0.shape) * noise)\n\n @torch.no_grad()\n def decode(self, x_latent, cond, t_start, unconditional_guidance_scale=1.0, unconditional_conditioning=None,\n use_original_steps=False):\n\n timesteps = np.arange(self.ddpm_num_timesteps) if use_original_steps else self.ddim_timesteps\n timesteps = timesteps[:t_start]\n\n time_range = np.flip(timesteps)\n total_steps = timesteps.shape[0]\n print(f\"Running DDIM Sampling with {total_steps} timesteps\")\n\n iterator = tqdm(time_range, desc='Decoding image', total=total_steps)\n x_dec = x_latent\n for i, step in enumerate(iterator):\n index = total_steps - i - 1\n ts = torch.full((x_latent.shape[0],), step, device=x_latent.device, dtype=torch.long)\n x_dec, _ = self.p_sample_ddim(x_dec, cond, ts, index=index, use_original_steps=use_original_steps,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_conditioning=unconditional_conditioning)\n return x_dec\n\n" }, { "path": "lvdm/models/utils_diffusion.py", "content": "import math\nimport numpy as np\nfrom einops import repeat\nimport torch\nimport torch.nn.functional as F\n\n\ndef timestep_embedding(timesteps, dim, max_period=10000, repeat_only=False):\n \"\"\"\n Create sinusoidal timestep embeddings.\n :param timesteps: a 1-D Tensor of N indices, one per batch element.\n These may be fractional.\n :param dim: the dimension of the output.\n :param max_period: controls the minimum frequency of the embeddings.\n :return: an [N x dim] Tensor of positional embeddings.\n \"\"\"\n if not repeat_only:\n half = dim // 2\n freqs = torch.exp(\n -math.log(max_period) * torch.arange(start=0, end=half, dtype=torch.float32) / half\n ).to(device=timesteps.device)\n args = timesteps[:, None].float() * freqs[None]\n embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)\n if dim % 2:\n embedding = torch.cat([embedding, torch.zeros_like(embedding[:, :1])], dim=-1)\n else:\n embedding = repeat(timesteps, 'b -> b d', d=dim)\n return embedding\n\n\ndef make_beta_schedule(schedule, n_timestep, linear_start=1e-4, linear_end=2e-2, cosine_s=8e-3):\n if schedule == \"linear\":\n betas = (\n torch.linspace(linear_start ** 0.5, linear_end ** 0.5, n_timestep, dtype=torch.float64) ** 2\n )\n\n elif schedule == \"cosine\":\n timesteps = (\n torch.arange(n_timestep + 1, dtype=torch.float64) / n_timestep + cosine_s\n )\n alphas = timesteps / (1 + cosine_s) * np.pi / 2\n alphas = torch.cos(alphas).pow(2)\n alphas = alphas / alphas[0]\n betas = 1 - alphas[1:] / alphas[:-1]\n betas = np.clip(betas, a_min=0, a_max=0.999)\n\n elif schedule == \"sqrt_linear\":\n betas = torch.linspace(linear_start, linear_end, n_timestep, dtype=torch.float64)\n elif schedule == \"sqrt\":\n betas = torch.linspace(linear_start, linear_end, n_timestep, dtype=torch.float64) ** 0.5\n else:\n raise ValueError(f\"schedule '{schedule}' unknown.\")\n return betas.numpy()\n\n\ndef make_ddim_timesteps(ddim_discr_method, num_ddim_timesteps, num_ddpm_timesteps, verbose=True):\n if ddim_discr_method == 'uniform':\n c = num_ddpm_timesteps // num_ddim_timesteps\n ddim_timesteps = np.asarray(list(range(0, num_ddpm_timesteps, c)))\n elif ddim_discr_method == 'quad':\n ddim_timesteps = ((np.linspace(0, np.sqrt(num_ddpm_timesteps * .8), num_ddim_timesteps)) ** 2).astype(int)\n else:\n raise NotImplementedError(f'There is no ddim discretization method called \"{ddim_discr_method}\"')\n\n # assert ddim_timesteps.shape[0] == num_ddim_timesteps\n # add one to get the final alpha values right (the ones from first scale to data during sampling)\n steps_out = ddim_timesteps + 1\n if verbose:\n print(f'Selected timesteps for ddim sampler: {steps_out}')\n return steps_out\n\n\ndef make_ddim_sampling_parameters(alphacums, ddim_timesteps, eta, verbose=True):\n # select alphas for computing the variance schedule\n # print(f'ddim_timesteps={ddim_timesteps}, len_alphacums={len(alphacums)}')\n alphas = alphacums[ddim_timesteps]\n alphas_prev = np.asarray([alphacums[0]] + alphacums[ddim_timesteps[:-1]].tolist())\n\n # according the the formula provided in https://arxiv.org/abs/2010.02502\n sigmas = eta * np.sqrt((1 - alphas_prev) / (1 - alphas) * (1 - alphas / alphas_prev))\n if verbose:\n print(f'Selected alphas for ddim sampler: a_t: {alphas}; a_(t-1): {alphas_prev}')\n print(f'For the chosen value of eta, which is {eta}, '\n f'this results in the following sigma_t schedule for ddim sampler {sigmas}')\n return sigmas, alphas, alphas_prev\n\n\ndef betas_for_alpha_bar(num_diffusion_timesteps, alpha_bar, max_beta=0.999):\n \"\"\"\n Create a beta schedule that discretizes the given alpha_t_bar function,\n which defines the cumulative product of (1-beta) over time from t = [0,1].\n :param num_diffusion_timesteps: the number of betas to produce.\n :param alpha_bar: a lambda that takes an argument t from 0 to 1 and\n produces the cumulative product of (1-beta) up to that\n part of the diffusion process.\n :param max_beta: the maximum beta to use; use values lower than 1 to\n prevent singularities.\n \"\"\"\n betas = []\n for i in range(num_diffusion_timesteps):\n t1 = i / num_diffusion_timesteps\n t2 = (i + 1) / num_diffusion_timesteps\n betas.append(min(1 - alpha_bar(t2) / alpha_bar(t1), max_beta))\n return np.array(betas)" }, { "path": "lvdm/modules/attention.py", "content": "from functools import partial\nimport torch\nfrom torch import nn, einsum\nimport torch.nn.functional as F\nfrom einops import rearrange, repeat\ntry:\n import xformers\n import xformers.ops\n XFORMERS_IS_AVAILBLE = True\nexcept:\n XFORMERS_IS_AVAILBLE = False\nfrom lvdm.common import (\n checkpoint,\n exists,\n default,\n)\nfrom lvdm.basics import (\n zero_module,\n)\n\nclass RelativePosition(nn.Module):\n \"\"\" https://github.com/evelinehong/Transformer_Relative_Position_PyTorch/blob/master/relative_position.py \"\"\"\n\n def __init__(self, num_units, max_relative_position):\n super().__init__()\n self.num_units = num_units\n self.max_relative_position = max_relative_position\n self.embeddings_table = nn.Parameter(torch.Tensor(max_relative_position * 2 + 1, num_units))\n nn.init.xavier_uniform_(self.embeddings_table)\n\n def forward(self, length_q, length_k):\n device = self.embeddings_table.device\n range_vec_q = torch.arange(length_q, device=device)\n range_vec_k = torch.arange(length_k, device=device)\n distance_mat = range_vec_k[None, :] - range_vec_q[:, None]\n distance_mat_clipped = torch.clamp(distance_mat, -self.max_relative_position, self.max_relative_position)\n final_mat = distance_mat_clipped + self.max_relative_position\n final_mat = final_mat.long()\n embeddings = self.embeddings_table[final_mat]\n return embeddings\n\n\nclass CrossAttention(nn.Module):\n\n def __init__(self, query_dim, context_dim=None, heads=8, dim_head=64, dropout=0., \n relative_position=False, temporal_length=None, img_cross_attention=False):\n super().__init__()\n inner_dim = dim_head * heads\n context_dim = default(context_dim, query_dim)\n\n self.scale = dim_head**-0.5\n self.heads = heads\n self.dim_head = dim_head\n self.to_q = nn.Linear(query_dim, inner_dim, bias=False)\n self.to_k = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_v = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_out = nn.Sequential(nn.Linear(inner_dim, query_dim), nn.Dropout(dropout))\n\n self.image_cross_attention_scale = 1.0\n self.text_context_len = 77\n self.img_cross_attention = img_cross_attention\n if self.img_cross_attention:\n self.to_k_ip = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_v_ip = nn.Linear(context_dim, inner_dim, bias=False)\n \n self.relative_position = relative_position\n if self.relative_position:\n assert(temporal_length is not None)\n self.relative_position_k = RelativePosition(num_units=dim_head, max_relative_position=temporal_length)\n self.relative_position_v = RelativePosition(num_units=dim_head, max_relative_position=temporal_length)\n else:\n ## only used for spatial attention, while NOT for temporal attention\n if XFORMERS_IS_AVAILBLE and temporal_length is None:\n self.forward = self.efficient_forward\n\n def forward(self, x, context=None, mask=None):\n h = self.heads\n\n q = self.to_q(x)\n context = default(context, x)\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n context, context_img = context[:,:self.text_context_len,:], context[:,self.text_context_len:,:]\n k = self.to_k(context)\n v = self.to_v(context)\n k_ip = self.to_k_ip(context_img)\n v_ip = self.to_v_ip(context_img)\n else:\n k = self.to_k(context)\n v = self.to_v(context)\n\n q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (q, k, v))\n sim = torch.einsum('b i d, b j d -> b i j', q, k) * self.scale\n if self.relative_position:\n len_q, len_k, len_v = q.shape[1], k.shape[1], v.shape[1]\n k2 = self.relative_position_k(len_q, len_k)\n sim2 = einsum('b t d, t s d -> b t s', q, k2) * self.scale # TODO check \n sim += sim2\n del k\n\n if exists(mask):\n ## feasible for causal attention mask only\n max_neg_value = -torch.finfo(sim.dtype).max\n mask = repeat(mask, 'b i j -> (b h) i j', h=h)\n sim.masked_fill_(~(mask>0.5), max_neg_value)\n\n # attention, what we cannot get enough of\n sim = sim.softmax(dim=-1)\n out = torch.einsum('b i j, b j d -> b i d', sim, v)\n if self.relative_position:\n v2 = self.relative_position_v(len_q, len_v)\n out2 = einsum('b t s, t s d -> b t d', sim, v2) # TODO check\n out += out2\n out = rearrange(out, '(b h) n d -> b n (h d)', h=h)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n k_ip, v_ip = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (k_ip, v_ip))\n sim_ip = torch.einsum('b i d, b j d -> b i j', q, k_ip) * self.scale\n del k_ip\n sim_ip = sim_ip.softmax(dim=-1)\n out_ip = torch.einsum('b i j, b j d -> b i d', sim_ip, v_ip)\n out_ip = rearrange(out_ip, '(b h) n d -> b n (h d)', h=h)\n out = out + self.image_cross_attention_scale * out_ip\n del q\n\n return self.to_out(out)\n \n def efficient_forward(self, x, context=None, mask=None):\n q = self.to_q(x)\n context = default(context, x)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n context, context_img = context[:,:self.text_context_len,:], context[:,self.text_context_len:,:]\n k = self.to_k(context)\n v = self.to_v(context)\n k_ip = self.to_k_ip(context_img)\n v_ip = self.to_v_ip(context_img)\n else:\n k = self.to_k(context)\n v = self.to_v(context)\n\n b, _, _ = q.shape\n q, k, v = map(\n lambda t: t.unsqueeze(3)\n .reshape(b, t.shape[1], self.heads, self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b * self.heads, t.shape[1], self.dim_head)\n .contiguous(),\n (q, k, v),\n )\n # actually compute the attention, what we cannot get enough of\n out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=None)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n k_ip, v_ip = map(\n lambda t: t.unsqueeze(3)\n .reshape(b, t.shape[1], self.heads, self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b * self.heads, t.shape[1], self.dim_head)\n .contiguous(),\n (k_ip, v_ip),\n )\n out_ip = xformers.ops.memory_efficient_attention(q, k_ip, v_ip, attn_bias=None, op=None)\n out_ip = (\n out_ip.unsqueeze(0)\n .reshape(b, self.heads, out.shape[1], self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b, out.shape[1], self.heads * self.dim_head)\n )\n\n if exists(mask):\n raise NotImplementedError\n out = (\n out.unsqueeze(0)\n .reshape(b, self.heads, out.shape[1], self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b, out.shape[1], self.heads * self.dim_head)\n )\n if context is not None and self.img_cross_attention:\n out = out + self.image_cross_attention_scale * out_ip\n return self.to_out(out)\n\n\nclass BasicTransformerBlock(nn.Module):\n\n def __init__(self, dim, n_heads, d_head, dropout=0., context_dim=None, gated_ff=True, checkpoint=True,\n disable_self_attn=False, attention_cls=None, img_cross_attention=False):\n super().__init__()\n attn_cls = CrossAttention if attention_cls is None else attention_cls\n self.disable_self_attn = disable_self_attn\n self.attn1 = attn_cls(query_dim=dim, heads=n_heads, dim_head=d_head, dropout=dropout,\n context_dim=context_dim if self.disable_self_attn else None)\n self.ff = FeedForward(dim, dropout=dropout, glu=gated_ff)\n self.attn2 = attn_cls(query_dim=dim, context_dim=context_dim, heads=n_heads, dim_head=d_head, dropout=dropout,\n img_cross_attention=img_cross_attention)\n self.norm1 = nn.LayerNorm(dim)\n self.norm2 = nn.LayerNorm(dim)\n self.norm3 = nn.LayerNorm(dim)\n self.checkpoint = checkpoint\n\n def forward(self, x, context=None, mask=None):\n ## implementation tricks: because checkpointing doesn't support non-tensor (e.g. None or scalar) arguments\n input_tuple = (x,) ## should not be (x), otherwise *input_tuple will decouple x into multiple arguments\n if context is not None:\n input_tuple = (x, context)\n if mask is not None:\n forward_mask = partial(self._forward, mask=mask)\n return checkpoint(forward_mask, (x,), self.parameters(), self.checkpoint)\n if context is not None and mask is not None:\n input_tuple = (x, context, mask)\n return checkpoint(self._forward, input_tuple, self.parameters(), self.checkpoint)\n\n def _forward(self, x, context=None, mask=None):\n x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None, mask=mask) + x\n x = self.attn2(self.norm2(x), context=context, mask=mask) + x\n x = self.ff(self.norm3(x)) + x\n return x\n\n\nclass SpatialTransformer(nn.Module):\n \"\"\"\n Transformer block for image-like data in spatial axis.\n First, project the input (aka embedding)\n and reshape to b, t, d.\n Then apply standard transformer action.\n Finally, reshape to image\n NEW: use_linear for more efficiency instead of the 1x1 convs\n \"\"\"\n\n def __init__(self, in_channels, n_heads, d_head, depth=1, dropout=0., context_dim=None,\n use_checkpoint=True, disable_self_attn=False, use_linear=False, img_cross_attention=False):\n super().__init__()\n self.in_channels = in_channels\n inner_dim = n_heads * d_head\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n if not use_linear:\n self.proj_in = nn.Conv2d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n else:\n self.proj_in = nn.Linear(in_channels, inner_dim)\n\n self.transformer_blocks = nn.ModuleList([\n BasicTransformerBlock(\n inner_dim,\n n_heads,\n d_head,\n dropout=dropout,\n context_dim=context_dim,\n img_cross_attention=img_cross_attention,\n disable_self_attn=disable_self_attn,\n checkpoint=use_checkpoint) for d in range(depth)\n ])\n if not use_linear:\n self.proj_out = zero_module(nn.Conv2d(inner_dim, in_channels, kernel_size=1, stride=1, padding=0))\n else:\n self.proj_out = zero_module(nn.Linear(inner_dim, in_channels))\n self.use_linear = use_linear\n\n\n def forward(self, x, context=None):\n b, c, h, w = x.shape\n x_in = x\n x = self.norm(x)\n if not self.use_linear:\n x = self.proj_in(x)\n x = rearrange(x, 'b c h w -> b (h w) c').contiguous()\n if self.use_linear:\n x = self.proj_in(x)\n for i, block in enumerate(self.transformer_blocks):\n x = block(x, context=context)\n if self.use_linear:\n x = self.proj_out(x)\n x = rearrange(x, 'b (h w) c -> b c h w', h=h, w=w).contiguous()\n if not self.use_linear:\n x = self.proj_out(x)\n return x + x_in\n \n \nclass TemporalTransformer(nn.Module):\n \"\"\"\n Transformer block for image-like data in temporal axis.\n First, reshape to b, t, d.\n Then apply standard transformer action.\n Finally, reshape to image\n \"\"\"\n def __init__(self, in_channels, n_heads, d_head, depth=1, dropout=0., context_dim=None,\n use_checkpoint=True, use_linear=False, only_self_att=True, causal_attention=False,\n relative_position=False, temporal_length=None):\n super().__init__()\n self.only_self_att = only_self_att\n self.relative_position = relative_position\n self.causal_attention = causal_attention\n self.in_channels = in_channels\n inner_dim = n_heads * d_head\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n self.proj_in = nn.Conv1d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n if not use_linear:\n self.proj_in = nn.Conv1d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n else:\n self.proj_in = nn.Linear(in_channels, inner_dim)\n\n if relative_position:\n assert(temporal_length is not None)\n attention_cls = partial(CrossAttention, relative_position=True, temporal_length=temporal_length)\n else:\n attention_cls = None\n if self.causal_attention:\n assert(temporal_length is not None)\n self.mask = torch.tril(torch.ones([1, temporal_length, temporal_length]))\n\n if self.only_self_att:\n context_dim = None\n self.transformer_blocks = nn.ModuleList([\n BasicTransformerBlock(\n inner_dim,\n n_heads,\n d_head,\n dropout=dropout,\n context_dim=context_dim,\n attention_cls=attention_cls,\n checkpoint=use_checkpoint) for d in range(depth)\n ])\n if not use_linear:\n self.proj_out = zero_module(nn.Conv1d(inner_dim, in_channels, kernel_size=1, stride=1, padding=0))\n else:\n self.proj_out = zero_module(nn.Linear(inner_dim, in_channels))\n self.use_linear = use_linear\n\n def forward(self, x, context=None):\n b, c, t, h, w = x.shape\n x_in = x\n x = self.norm(x)\n x = rearrange(x, 'b c t h w -> (b h w) c t').contiguous()\n if not self.use_linear:\n x = self.proj_in(x)\n x = rearrange(x, 'bhw c t -> bhw t c').contiguous()\n if self.use_linear:\n x = self.proj_in(x)\n\n if self.causal_attention:\n mask = self.mask.to(x.device)\n mask = repeat(mask, 'l i j -> (l bhw) i j', bhw=b*h*w)\n else:\n mask = None\n\n if self.only_self_att:\n ## note: if no context is given, cross-attention defaults to self-attention\n for i, block in enumerate(self.transformer_blocks):\n x = block(x, mask=mask)\n x = rearrange(x, '(b hw) t c -> b hw t c', b=b).contiguous()\n else:\n x = rearrange(x, '(b hw) t c -> b hw t c', b=b).contiguous()\n context = rearrange(context, '(b t) l con -> b t l con', t=t).contiguous()\n for i, block in enumerate(self.transformer_blocks):\n # calculate each batch one by one (since number in shape could not greater then 65,535 for some package)\n for j in range(b):\n context_j = repeat(\n context[j],\n 't l con -> (t r) l con', r=(h * w) // t, t=t).contiguous()\n ## note: causal mask will not applied in cross-attention case\n x[j] = block(x[j], context=context_j)\n \n if self.use_linear:\n x = self.proj_out(x)\n x = rearrange(x, 'b (h w) t c -> b c t h w', h=h, w=w).contiguous()\n if not self.use_linear:\n x = rearrange(x, 'b hw t c -> (b hw) c t').contiguous()\n x = self.proj_out(x)\n x = rearrange(x, '(b h w) c t -> b c t h w', b=b, h=h, w=w).contiguous()\n\n return x + x_in\n \n\nclass GEGLU(nn.Module):\n def __init__(self, dim_in, dim_out):\n super().__init__()\n self.proj = nn.Linear(dim_in, dim_out * 2)\n\n def forward(self, x):\n x, gate = self.proj(x).chunk(2, dim=-1)\n return x * F.gelu(gate)\n\n\nclass FeedForward(nn.Module):\n def __init__(self, dim, dim_out=None, mult=4, glu=False, dropout=0.):\n super().__init__()\n inner_dim = int(dim * mult)\n dim_out = default(dim_out, dim)\n project_in = nn.Sequential(\n nn.Linear(dim, inner_dim),\n nn.GELU()\n ) if not glu else GEGLU(dim, inner_dim)\n\n self.net = nn.Sequential(\n project_in,\n nn.Dropout(dropout),\n nn.Linear(inner_dim, dim_out)\n )\n\n def forward(self, x):\n return self.net(x)\n\n\nclass LinearAttention(nn.Module):\n def __init__(self, dim, heads=4, dim_head=32):\n super().__init__()\n self.heads = heads\n hidden_dim = dim_head * heads\n self.to_qkv = nn.Conv2d(dim, hidden_dim * 3, 1, bias = False)\n self.to_out = nn.Conv2d(hidden_dim, dim, 1)\n\n def forward(self, x):\n b, c, h, w = x.shape\n qkv = self.to_qkv(x)\n q, k, v = rearrange(qkv, 'b (qkv heads c) h w -> qkv b heads c (h w)', heads = self.heads, qkv=3)\n k = k.softmax(dim=-1) \n context = torch.einsum('bhdn,bhen->bhde', k, v)\n out = torch.einsum('bhde,bhdn->bhen', context, q)\n out = rearrange(out, 'b heads c (h w) -> b (heads c) h w', heads=self.heads, h=h, w=w)\n return self.to_out(out)\n\n\nclass SpatialSelfAttention(nn.Module):\n def __init__(self, in_channels):\n super().__init__()\n self.in_channels = in_channels\n\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n self.q = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.k = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.v = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.proj_out = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n\n def forward(self, x):\n h_ = x\n h_ = self.norm(h_)\n q = self.q(h_)\n k = self.k(h_)\n v = self.v(h_)\n\n # compute attention\n b,c,h,w = q.shape\n q = rearrange(q, 'b c h w -> b (h w) c')\n k = rearrange(k, 'b c h w -> b c (h w)')\n w_ = torch.einsum('bij,bjk->bik', q, k)\n\n w_ = w_ * (int(c)**(-0.5))\n w_ = torch.nn.functional.softmax(w_, dim=2)\n\n # attend to values\n v = rearrange(v, 'b c h w -> b c (h w)')\n w_ = rearrange(w_, 'b i j -> b j i')\n h_ = torch.einsum('bij,bjk->bik', v, w_)\n h_ = rearrange(h_, 'b c (h w) -> b c h w', h=h)\n h_ = self.proj_out(h_)\n\n return x+h_\n" }, { "path": "lvdm/modules/attention_freenoise.py", "content": "from functools import partial\nimport torch\nfrom torch import nn, einsum\nimport torch.nn.functional as F\nfrom einops import rearrange, repeat\ntry:\n import xformers\n import xformers.ops\n XFORMERS_IS_AVAILBLE = True\nexcept:\n XFORMERS_IS_AVAILBLE = False\nfrom lvdm.common import (\n checkpoint,\n exists,\n default,\n)\nfrom lvdm.basics import (\n zero_module,\n)\n\ndef generate_weight_sequence(n):\n if n % 2 == 0:\n max_weight = n // 2\n weight_sequence = list(range(1, max_weight + 1, 1)) + list(range(max_weight, 0, -1))\n else:\n max_weight = (n + 1) // 2\n weight_sequence = list(range(1, max_weight, 1)) + [max_weight] + list(range(max_weight - 1, 0, -1))\n return weight_sequence\n\nclass RelativePosition(nn.Module):\n \"\"\" https://github.com/evelinehong/Transformer_Relative_Position_PyTorch/blob/master/relative_position.py \"\"\"\n\n def __init__(self, num_units, max_relative_position):\n super().__init__()\n self.num_units = num_units\n self.max_relative_position = max_relative_position\n self.embeddings_table = nn.Parameter(torch.Tensor(max_relative_position * 2 + 1, num_units))\n nn.init.xavier_uniform_(self.embeddings_table)\n\n def forward(self, length_q, length_k):\n device = self.embeddings_table.device\n range_vec_q = torch.arange(length_q, device=device)\n range_vec_k = torch.arange(length_k, device=device)\n distance_mat = range_vec_k[None, :] - range_vec_q[:, None]\n distance_mat_clipped = torch.clamp(distance_mat, -self.max_relative_position, self.max_relative_position)\n final_mat = distance_mat_clipped + self.max_relative_position\n final_mat = final_mat.long()\n embeddings = self.embeddings_table[final_mat]\n return embeddings\n\n\nclass CrossAttention(nn.Module):\n\n def __init__(self, query_dim, context_dim=None, heads=8, dim_head=64, dropout=0., \n relative_position=False, temporal_length=None, img_cross_attention=False, injection=False):\n super().__init__()\n inner_dim = dim_head * heads\n context_dim = default(context_dim, query_dim)\n\n self.scale = dim_head**-0.5\n self.heads = heads\n self.dim_head = dim_head\n self.to_q = nn.Linear(query_dim, inner_dim, bias=False)\n self.to_k = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_v = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_out = nn.Sequential(nn.Linear(inner_dim, query_dim), nn.Dropout(dropout))\n\n self.image_cross_attention_scale = 1.0\n self.text_context_len = 77\n self.img_cross_attention = img_cross_attention\n if self.img_cross_attention:\n self.to_k_ip = nn.Linear(context_dim, inner_dim, bias=False)\n self.to_v_ip = nn.Linear(context_dim, inner_dim, bias=False)\n \n self.relative_position = relative_position\n if self.relative_position:\n assert(temporal_length is not None)\n self.relative_position_k = RelativePosition(num_units=dim_head, max_relative_position=temporal_length)\n self.relative_position_v = RelativePosition(num_units=dim_head, max_relative_position=temporal_length)\n else:\n ## only used for spatial attention, while NOT for temporal attention\n if XFORMERS_IS_AVAILBLE and temporal_length is None:\n self.forward = self.efficient_forward\n\n self.injection = injection\n\n def forward(self, x, context=None, mask=None, context_next=None, use_injection=False):\n\n sa_flag = False\n if context is None:\n sa_flag = True\n\n h = self.heads\n\n all_q = self.to_q(x)\n context = default(context, x)\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n context, context_img = context[:,:self.text_context_len,:], context[:,self.text_context_len:,:]\n all_k = self.to_k(context)\n all_v = self.to_v(context)\n all_k_ip = self.to_k_ip(context_img)\n all_v_ip = self.to_v_ip(context_img)\n else:\n all_k = self.to_k(context)\n all_v = self.to_v(context)\n\n count = torch.zeros_like(all_k)\n value = torch.zeros_like(all_k)\n\n if (sa_flag) and (context_next is not None):\n all_q, all_k, all_v = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (all_q, all_k, all_v))\n if context is not None and self.img_cross_attention:\n all_k_ip, all_v_ip = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (all_k_ip, all_v_ip))\n for t_start, t_end in context_next:\n weight_sequence = generate_weight_sequence(t_end - t_start)\n weight_tensor = torch.ones_like(count[:, t_start:t_end])\n weight_tensor = weight_tensor * torch.Tensor(weight_sequence).to(x.device).unsqueeze(0).unsqueeze(-1)\n\n q = all_q[:, t_start:t_end]\n k = all_k[:, t_start:t_end]\n v = all_v[:, t_start:t_end]\n\n sim = torch.einsum('b i d, b j d -> b i j', q, k) * self.scale\n if self.relative_position:\n len_q, len_k, len_v = q.shape[1], k.shape[1], v.shape[1]\n k2 = self.relative_position_k(len_q, len_k)\n sim2 = einsum('b t d, t s d -> b t s', q, k2) * self.scale # TODO check \n sim += sim2\n del k\n\n if exists(mask):\n ## feasible for causal attention mask only\n max_neg_value = -torch.finfo(sim.dtype).max\n mask = repeat(mask, 'b i j -> (b h) i j', h=h)\n sim.masked_fill_(~(mask>0.5), max_neg_value)\n\n # attention, what we cannot get enough of\n sim = sim.softmax(dim=-1)\n out = torch.einsum('b i j, b j d -> b i d', sim, v)\n if self.relative_position:\n v2 = self.relative_position_v(len_q, len_v)\n out2 = einsum('b t s, t s d -> b t d', sim, v2) # TODO check\n out += out2\n out = rearrange(out, '(b h) n d -> b n (h d)', h=h)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n k_ip = all_k_ip[:, t_start:t_end]\n v_ip = all_v_ip[:, t_start:t_end]\n sim_ip = torch.einsum('b i d, b j d -> b i j', q, k_ip) * self.scale\n del k_ip\n sim_ip = sim_ip.softmax(dim=-1)\n out_ip = torch.einsum('b i j, b j d -> b i d', sim_ip, v_ip)\n out_ip = rearrange(out_ip, '(b h) n d -> b n (h d)', h=h)\n out = out + self.image_cross_attention_scale * out_ip\n del q\n\n value[:,t_start:t_end] += out * weight_tensor\n count[:,t_start:t_end] += weight_tensor\n\n final_out = torch.where(count>0, value/count, value)\n\n else:\n q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (all_q, all_k, all_v))\n sim = torch.einsum('b i d, b j d -> b i j', q, k) * self.scale\n if self.relative_position:\n len_q, len_k, len_v = q.shape[1], k.shape[1], v.shape[1]\n k2 = self.relative_position_k(len_q, len_k)\n sim2 = einsum('b t d, t s d -> b t s', q, k2) * self.scale # TODO check \n sim += sim2\n del k\n\n if exists(mask):\n ## feasible for causal attention mask only\n max_neg_value = -torch.finfo(sim.dtype).max\n mask = repeat(mask, 'b i j -> (b h) i j', h=h)\n sim.masked_fill_(~(mask>0.5), max_neg_value)\n\n # attention, what we cannot get enough of\n sim = sim.softmax(dim=-1)\n out = torch.einsum('b i j, b j d -> b i d', sim, v)\n if self.relative_position:\n v2 = self.relative_position_v(len_q, len_v)\n out2 = einsum('b t s, t s d -> b t d', sim, v2) # TODO check\n out += out2\n final_out = rearrange(out, '(b h) n d -> b n (h d)', h=h)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n k_ip, v_ip = map(lambda t: rearrange(t, 'b n (h d) -> (b h) n d', h=h), (all_k_ip, all_v_ip))\n sim_ip = torch.einsum('b i d, b j d -> b i j', q, k_ip) * self.scale\n del k_ip\n sim_ip = sim_ip.softmax(dim=-1)\n out_ip = torch.einsum('b i j, b j d -> b i d', sim_ip, v_ip)\n out_ip = rearrange(out_ip, '(b h) n d -> b n (h d)', h=h)\n final_out = final_out + self.image_cross_attention_scale * out_ip\n del q\n\n return self.to_out(final_out)\n \n def efficient_forward(self, x, context=None, mask=None, context_next=None, use_injection=False):\n\n sa_flag = False\n if context is None:\n sa_flag = True\n\n q = self.to_q(x)\n context = default(context, x)\n\n if not sa_flag: \n sq_size = x.shape[0]\n if self.injection and use_injection:\n context_new = context[-sq_size:]\n else:\n context_new = context[:sq_size]\n else:\n context_new = context.clone()\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n context, context_img = context_new[:,:self.text_context_len,:], context_new[:,self.text_context_len:,:]\n k = self.to_k(context)\n v = self.to_v(context)\n k_ip = self.to_k_ip(context_img)\n v_ip = self.to_v_ip(context_img)\n else:\n k = self.to_k(context_new)\n v = self.to_v(context_new)\n\n b, _, _ = q.shape\n q, k, v = map(\n lambda t: t.unsqueeze(3)\n .reshape(b, t.shape[1], self.heads, self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b * self.heads, t.shape[1], self.dim_head)\n .contiguous(),\n (q, k, v),\n )\n # actually compute the attention, what we cannot get enough of\n out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=None)\n\n ## considering image token additionally\n if context is not None and self.img_cross_attention:\n k_ip, v_ip = map(\n lambda t: t.unsqueeze(3)\n .reshape(b, t.shape[1], self.heads, self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b * self.heads, t.shape[1], self.dim_head)\n .contiguous(),\n (k_ip, v_ip),\n )\n out_ip = xformers.ops.memory_efficient_attention(q, k_ip, v_ip, attn_bias=None, op=None)\n out_ip = (\n out_ip.unsqueeze(0)\n .reshape(b, self.heads, out.shape[1], self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b, out.shape[1], self.heads * self.dim_head)\n )\n\n if exists(mask):\n raise NotImplementedError\n out = (\n out.unsqueeze(0)\n .reshape(b, self.heads, out.shape[1], self.dim_head)\n .permute(0, 2, 1, 3)\n .reshape(b, out.shape[1], self.heads * self.dim_head)\n )\n if context is not None and self.img_cross_attention:\n out = out + self.image_cross_attention_scale * out_ip\n return self.to_out(out)\n\n\nclass BasicTransformerBlock(nn.Module):\n\n def __init__(self, dim, n_heads, d_head, dropout=0., context_dim=None, gated_ff=True, checkpoint=True,\n disable_self_attn=False, attention_cls=None, img_cross_attention=False, injection=False):\n super().__init__()\n attn_cls = CrossAttention if attention_cls is None else attention_cls\n self.disable_self_attn = disable_self_attn\n self.attn1 = attn_cls(query_dim=dim, heads=n_heads, dim_head=d_head, dropout=dropout,\n context_dim=context_dim if self.disable_self_attn else None, injection=injection)\n self.ff = FeedForward(dim, dropout=dropout, glu=gated_ff)\n self.attn2 = attn_cls(query_dim=dim, context_dim=context_dim, heads=n_heads, dim_head=d_head, dropout=dropout,\n img_cross_attention=img_cross_attention, injection=injection)\n self.norm1 = nn.LayerNorm(dim)\n self.norm2 = nn.LayerNorm(dim)\n self.norm3 = nn.LayerNorm(dim)\n self.checkpoint = checkpoint\n\n def forward(self, x, context=None, mask=None, context_next=None, use_injection=False, **kwargs):\n ## implementation tricks: because checkpointing doesn't support non-tensor (e.g. None or scalar) arguments\n input_tuple = (x,) ## should not be (x), otherwise *input_tuple will decouple x into multiple arguments\n if context is not None:\n input_tuple = (x, context)\n if mask is not None:\n forward_mask = partial(self._forward, mask=mask)\n return checkpoint(forward_mask, (x,), self.parameters(), self.checkpoint)\n if context is not None and mask is not None:\n input_tuple = (x, context, mask)\n input_tuple = (x, context, mask, context_next, use_injection)\n return checkpoint(self._forward, input_tuple, self.parameters(), self.checkpoint)\n\n def _forward(self, x, context=None, mask=None, context_next=None, use_injection=False):\n x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None, mask=mask, context_next=context_next, use_injection=False) + x\n x = self.attn2(self.norm2(x), context=context, mask=mask, context_next=context_next, use_injection=use_injection) + x\n x = self.ff(self.norm3(x)) + x\n return x\n\n\nclass SpatialTransformer(nn.Module):\n \"\"\"\n Transformer block for image-like data in spatial axis.\n First, project the input (aka embedding)\n and reshape to b, t, d.\n Then apply standard transformer action.\n Finally, reshape to image\n NEW: use_linear for more efficiency instead of the 1x1 convs\n \"\"\"\n\n def __init__(self, in_channels, n_heads, d_head, depth=1, dropout=0., context_dim=None,\n use_checkpoint=True, disable_self_attn=False, use_linear=False, img_cross_attention=False, injection=False):\n super().__init__()\n self.in_channels = in_channels\n inner_dim = n_heads * d_head\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n if not use_linear:\n self.proj_in = nn.Conv2d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n else:\n self.proj_in = nn.Linear(in_channels, inner_dim)\n\n self.transformer_blocks = nn.ModuleList([\n BasicTransformerBlock(\n inner_dim,\n n_heads,\n d_head,\n dropout=dropout,\n context_dim=context_dim,\n img_cross_attention=img_cross_attention,\n disable_self_attn=disable_self_attn,\n checkpoint=use_checkpoint,\n injection=injection) for d in range(depth)\n ])\n if not use_linear:\n self.proj_out = zero_module(nn.Conv2d(inner_dim, in_channels, kernel_size=1, stride=1, padding=0))\n else:\n self.proj_out = zero_module(nn.Linear(inner_dim, in_channels))\n self.use_linear = use_linear\n\n\n def forward(self, x, context=None, **kwargs):\n b, c, h, w = x.shape\n x_in = x\n x = self.norm(x)\n if not self.use_linear:\n x = self.proj_in(x)\n x = rearrange(x, 'b c h w -> b (h w) c').contiguous()\n if self.use_linear:\n x = self.proj_in(x)\n for i, block in enumerate(self.transformer_blocks):\n x = block(x, context=context, **kwargs)\n if self.use_linear:\n x = self.proj_out(x)\n x = rearrange(x, 'b (h w) c -> b c h w', h=h, w=w).contiguous()\n if not self.use_linear:\n x = self.proj_out(x)\n return x + x_in\n \n \nclass TemporalTransformer(nn.Module):\n \"\"\"\n Transformer block for image-like data in temporal axis.\n First, reshape to b, t, d.\n Then apply standard transformer action.\n Finally, reshape to image\n \"\"\"\n def __init__(self, in_channels, n_heads, d_head, depth=1, dropout=0., context_dim=None,\n use_checkpoint=True, use_linear=False, only_self_att=True, causal_attention=False,\n relative_position=False, temporal_length=None, injection=False):\n super().__init__()\n self.only_self_att = only_self_att\n self.relative_position = relative_position\n self.causal_attention = causal_attention\n self.in_channels = in_channels\n inner_dim = n_heads * d_head\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n self.proj_in = nn.Conv1d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n if not use_linear:\n self.proj_in = nn.Conv1d(in_channels, inner_dim, kernel_size=1, stride=1, padding=0)\n else:\n self.proj_in = nn.Linear(in_channels, inner_dim)\n\n if relative_position:\n assert(temporal_length is not None)\n attention_cls = partial(CrossAttention, relative_position=True, temporal_length=temporal_length)\n else:\n attention_cls = partial(CrossAttention, temporal_length=temporal_length)\n if self.causal_attention:\n assert(temporal_length is not None)\n self.mask = torch.tril(torch.ones([1, temporal_length, temporal_length]))\n\n if self.only_self_att:\n context_dim = None\n self.transformer_blocks = nn.ModuleList([\n BasicTransformerBlock(\n inner_dim,\n n_heads,\n d_head,\n dropout=dropout,\n context_dim=context_dim,\n attention_cls=attention_cls,\n checkpoint=use_checkpoint,\n injection=injection) for d in range(depth)\n ])\n if not use_linear:\n self.proj_out = zero_module(nn.Conv1d(inner_dim, in_channels, kernel_size=1, stride=1, padding=0))\n else:\n self.proj_out = zero_module(nn.Linear(inner_dim, in_channels))\n self.use_linear = use_linear\n\n def forward(self, x, context=None, **kwargs):\n b, c, t, h, w = x.shape\n x_in = x\n x = self.norm(x)\n x = rearrange(x, 'b c t h w -> (b h w) c t').contiguous()\n if not self.use_linear:\n x = self.proj_in(x)\n x = rearrange(x, 'bhw c t -> bhw t c').contiguous()\n if self.use_linear:\n x = self.proj_in(x)\n\n if self.causal_attention:\n mask = self.mask.to(x.device)\n mask = repeat(mask, 'l i j -> (l bhw) i j', bhw=b*h*w)\n else:\n mask = None\n\n if self.only_self_att:\n ## note: if no context is given, cross-attention defaults to self-attention\n for i, block in enumerate(self.transformer_blocks):\n x = block(x, mask=mask, **kwargs)\n x = rearrange(x, '(b hw) t c -> b hw t c', b=b).contiguous()\n else:\n x = rearrange(x, '(b hw) t c -> b hw t c', b=b).contiguous()\n context = rearrange(context, '(b t) l con -> b t l con', t=t).contiguous()\n for i, block in enumerate(self.transformer_blocks):\n # calculate each batch one by one (since number in shape could not greater then 65,535 for some package)\n for j in range(b):\n context_j = repeat(\n context[j],\n 't l con -> (t r) l con', r=(h * w) // t, t=t).contiguous()\n ## note: causal mask will not applied in cross-attention case\n x[j] = block(x[j], context=context_j, **kwargs)\n \n if self.use_linear:\n x = self.proj_out(x)\n x = rearrange(x, 'b (h w) t c -> b c t h w', h=h, w=w).contiguous()\n if not self.use_linear:\n x = rearrange(x, 'b hw t c -> (b hw) c t').contiguous()\n x = self.proj_out(x)\n x = rearrange(x, '(b h w) c t -> b c t h w', b=b, h=h, w=w).contiguous()\n\n return x + x_in\n \n\nclass GEGLU(nn.Module):\n def __init__(self, dim_in, dim_out):\n super().__init__()\n self.proj = nn.Linear(dim_in, dim_out * 2)\n\n def forward(self, x):\n x, gate = self.proj(x).chunk(2, dim=-1)\n return x * F.gelu(gate)\n\n\nclass FeedForward(nn.Module):\n def __init__(self, dim, dim_out=None, mult=4, glu=False, dropout=0.):\n super().__init__()\n inner_dim = int(dim * mult)\n dim_out = default(dim_out, dim)\n project_in = nn.Sequential(\n nn.Linear(dim, inner_dim),\n nn.GELU()\n ) if not glu else GEGLU(dim, inner_dim)\n\n self.net = nn.Sequential(\n project_in,\n nn.Dropout(dropout),\n nn.Linear(inner_dim, dim_out)\n )\n\n def forward(self, x):\n return self.net(x)\n\n\nclass LinearAttention(nn.Module):\n def __init__(self, dim, heads=4, dim_head=32):\n super().__init__()\n self.heads = heads\n hidden_dim = dim_head * heads\n self.to_qkv = nn.Conv2d(dim, hidden_dim * 3, 1, bias = False)\n self.to_out = nn.Conv2d(hidden_dim, dim, 1)\n\n def forward(self, x):\n b, c, h, w = x.shape\n qkv = self.to_qkv(x)\n q, k, v = rearrange(qkv, 'b (qkv heads c) h w -> qkv b heads c (h w)', heads = self.heads, qkv=3)\n k = k.softmax(dim=-1) \n context = torch.einsum('bhdn,bhen->bhde', k, v)\n out = torch.einsum('bhde,bhdn->bhen', context, q)\n out = rearrange(out, 'b heads c (h w) -> b (heads c) h w', heads=self.heads, h=h, w=w)\n return self.to_out(out)\n\n\nclass SpatialSelfAttention(nn.Module):\n def __init__(self, in_channels):\n super().__init__()\n self.in_channels = in_channels\n\n self.norm = torch.nn.GroupNorm(num_groups=32, num_channels=in_channels, eps=1e-6, affine=True)\n self.q = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.k = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.v = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.proj_out = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n\n def forward(self, x, **kwargs):\n h_ = x\n h_ = self.norm(h_)\n q = self.q(h_)\n k = self.k(h_)\n v = self.v(h_)\n\n # compute attention\n b,c,h,w = q.shape\n q = rearrange(q, 'b c h w -> b (h w) c')\n k = rearrange(k, 'b c h w -> b c (h w)')\n w_ = torch.einsum('bij,bjk->bik', q, k)\n\n w_ = w_ * (int(c)**(-0.5))\n w_ = torch.nn.functional.softmax(w_, dim=2)\n\n # attend to values\n v = rearrange(v, 'b c h w -> b c (h w)')\n w_ = rearrange(w_, 'b i j -> b j i')\n h_ = torch.einsum('bij,bjk->bik', v, w_)\n h_ = rearrange(h_, 'b c (h w) -> b c h w', h=h)\n h_ = self.proj_out(h_)\n\n return x+h_\n" }, { "path": "lvdm/modules/encoders/condition.py", "content": "import torch\nimport torch.nn as nn\nfrom torch.utils.checkpoint import checkpoint\nimport kornia\nimport open_clip\nfrom transformers import T5Tokenizer, T5EncoderModel, CLIPTokenizer, CLIPTextModel\nfrom lvdm.common import autocast\nfrom utils.utils import count_params\n\nclass AbstractEncoder(nn.Module):\n def __init__(self):\n super().__init__()\n\n def encode(self, *args, **kwargs):\n raise NotImplementedError\n\n\nclass IdentityEncoder(AbstractEncoder):\n\n def encode(self, x):\n return x\n\n\nclass ClassEmbedder(nn.Module):\n def __init__(self, embed_dim, n_classes=1000, key='class', ucg_rate=0.1):\n super().__init__()\n self.key = key\n self.embedding = nn.Embedding(n_classes, embed_dim)\n self.n_classes = n_classes\n self.ucg_rate = ucg_rate\n\n def forward(self, batch, key=None, disable_dropout=False):\n if key is None:\n key = self.key\n # this is for use in crossattn\n c = batch[key][:, None]\n if self.ucg_rate > 0. and not disable_dropout:\n mask = 1. - torch.bernoulli(torch.ones_like(c) * self.ucg_rate)\n c = mask * c + (1 - mask) * torch.ones_like(c) * (self.n_classes - 1)\n c = c.long()\n c = self.embedding(c)\n return c\n\n def get_unconditional_conditioning(self, bs, device=\"cuda\"):\n uc_class = self.n_classes - 1 # 1000 classes --> 0 ... 999, one extra class for ucg (class 1000)\n uc = torch.ones((bs,), device=device) * uc_class\n uc = {self.key: uc}\n return uc\n\n\ndef disabled_train(self, mode=True):\n \"\"\"Overwrite model.train with this function to make sure train/eval mode\n does not change anymore.\"\"\"\n return self\n\n\nclass FrozenT5Embedder(AbstractEncoder):\n \"\"\"Uses the T5 transformer encoder for text\"\"\"\n\n def __init__(self, version=\"google/t5-v1_1-large\", device=\"cuda\", max_length=77,\n freeze=True): # others are google/t5-v1_1-xl and google/t5-v1_1-xxl\n super().__init__()\n self.tokenizer = T5Tokenizer.from_pretrained(version)\n self.transformer = T5EncoderModel.from_pretrained(version)\n self.device = device\n self.max_length = max_length # TODO: typical value?\n if freeze:\n self.freeze()\n\n def freeze(self):\n self.transformer = self.transformer.eval()\n # self.train = disabled_train\n for param in self.parameters():\n param.requires_grad = False\n\n def forward(self, text):\n batch_encoding = self.tokenizer(text, truncation=True, max_length=self.max_length, return_length=True,\n return_overflowing_tokens=False, padding=\"max_length\", return_tensors=\"pt\")\n tokens = batch_encoding[\"input_ids\"].to(self.device)\n outputs = self.transformer(input_ids=tokens)\n\n z = outputs.last_hidden_state\n return z\n\n def encode(self, text):\n return self(text)\n\n\nclass FrozenCLIPEmbedder(AbstractEncoder):\n \"\"\"Uses the CLIP transformer encoder for text (from huggingface)\"\"\"\n LAYERS = [\n \"last\",\n \"pooled\",\n \"hidden\"\n ]\n\n def __init__(self, version=\"openai/clip-vit-large-patch14\", device=\"cuda\", max_length=77,\n freeze=True, layer=\"last\", layer_idx=None): # clip-vit-base-patch32\n super().__init__()\n assert layer in self.LAYERS\n self.tokenizer = CLIPTokenizer.from_pretrained(version)\n self.transformer = CLIPTextModel.from_pretrained(version)\n self.device = device\n self.max_length = max_length\n if freeze:\n self.freeze()\n self.layer = layer\n self.layer_idx = layer_idx\n if layer == \"hidden\":\n assert layer_idx is not None\n assert 0 <= abs(layer_idx) <= 12\n\n def freeze(self):\n self.transformer = self.transformer.eval()\n # self.train = disabled_train\n for param in self.parameters():\n param.requires_grad = False\n\n def forward(self, text):\n batch_encoding = self.tokenizer(text, truncation=True, max_length=self.max_length, return_length=True,\n return_overflowing_tokens=False, padding=\"max_length\", return_tensors=\"pt\")\n tokens = batch_encoding[\"input_ids\"].to(self.device)\n outputs = self.transformer(input_ids=tokens, output_hidden_states=self.layer == \"hidden\")\n if self.layer == \"last\":\n z = outputs.last_hidden_state\n elif self.layer == \"pooled\":\n z = outputs.pooler_output[:, None, :]\n else:\n z = outputs.hidden_states[self.layer_idx]\n return z\n\n def encode(self, text):\n return self(text)\n\n\nclass ClipImageEmbedder(nn.Module):\n def __init__(\n self,\n model,\n jit=False,\n device='cuda' if torch.cuda.is_available() else 'cpu',\n antialias=True,\n ucg_rate=0.\n ):\n super().__init__()\n from clip import load as load_clip\n self.model, _ = load_clip(name=model, device=device, jit=jit)\n\n self.antialias = antialias\n\n self.register_buffer('mean', torch.Tensor([0.48145466, 0.4578275, 0.40821073]), persistent=False)\n self.register_buffer('std', torch.Tensor([0.26862954, 0.26130258, 0.27577711]), persistent=False)\n self.ucg_rate = ucg_rate\n\n def preprocess(self, x):\n # normalize to [0,1]\n x = kornia.geometry.resize(x, (224, 224),\n interpolation='bicubic', align_corners=True,\n antialias=self.antialias)\n x = (x + 1.) / 2.\n # re-normalize according to clip\n x = kornia.enhance.normalize(x, self.mean, self.std)\n return x\n\n def forward(self, x, no_dropout=False):\n # x is assumed to be in range [-1,1]\n out = self.model.encode_image(self.preprocess(x))\n out = out.to(x.dtype)\n if self.ucg_rate > 0. and not no_dropout:\n out = torch.bernoulli((1. - self.ucg_rate) * torch.ones(out.shape[0], device=out.device))[:, None] * out\n return out\n\n\nclass FrozenOpenCLIPEmbedder(AbstractEncoder):\n \"\"\"\n Uses the OpenCLIP transformer encoder for text\n \"\"\"\n LAYERS = [\n # \"pooled\",\n \"last\",\n \"penultimate\"\n ]\n\n def __init__(self, arch=\"ViT-H-14\", version=\"laion2b_s32b_b79k\", device=\"cuda\", max_length=77,\n freeze=True, layer=\"last\"):\n super().__init__()\n assert layer in self.LAYERS\n model, _, _ = open_clip.create_model_and_transforms(arch, device=torch.device('cpu'))\n del model.visual\n self.model = model\n\n self.device = device\n self.max_length = max_length\n if freeze:\n self.freeze()\n self.layer = layer\n if self.layer == \"last\":\n self.layer_idx = 0\n elif self.layer == \"penultimate\":\n self.layer_idx = 1\n else:\n raise NotImplementedError()\n\n def freeze(self):\n self.model = self.model.eval()\n for param in self.parameters():\n param.requires_grad = False\n\n def forward(self, text):\n self.device = self.model.positional_embedding.device\n tokens = open_clip.tokenize(text)\n z = self.encode_with_transformer(tokens.to(self.device))\n return z\n\n def encode_with_transformer(self, text):\n x = self.model.token_embedding(text) # [batch_size, n_ctx, d_model]\n x = x + self.model.positional_embedding\n x = x.permute(1, 0, 2) # NLD -> LND\n x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask)\n x = x.permute(1, 0, 2) # LND -> NLD\n x = self.model.ln_final(x)\n return x\n\n def text_transformer_forward(self, x: torch.Tensor, attn_mask=None):\n for i, r in enumerate(self.model.transformer.resblocks):\n if i == len(self.model.transformer.resblocks) - self.layer_idx:\n break\n if self.model.transformer.grad_checkpointing and not torch.jit.is_scripting():\n x = checkpoint(r, x, attn_mask)\n else:\n x = r(x, attn_mask=attn_mask)\n return x\n\n def encode(self, text):\n return self(text)\n\n\nclass FrozenOpenCLIPImageEmbedder(AbstractEncoder):\n \"\"\"\n Uses the OpenCLIP vision transformer encoder for images\n \"\"\"\n\n def __init__(self, arch=\"ViT-H-14\", version=\"laion2b_s32b_b79k\", device=\"cuda\", max_length=77,\n freeze=True, layer=\"pooled\", antialias=True, ucg_rate=0.):\n super().__init__()\n model, _, _ = open_clip.create_model_and_transforms(arch, device=torch.device('cpu'),\n pretrained=version, )\n del model.transformer\n self.model = model\n\n self.device = device\n self.max_length = max_length\n if freeze:\n self.freeze()\n self.layer = layer\n if self.layer == \"penultimate\":\n raise NotImplementedError()\n self.layer_idx = 1\n\n self.antialias = antialias\n\n self.register_buffer('mean', torch.Tensor([0.48145466, 0.4578275, 0.40821073]), persistent=False)\n self.register_buffer('std', torch.Tensor([0.26862954, 0.26130258, 0.27577711]), persistent=False)\n self.ucg_rate = ucg_rate\n\n def preprocess(self, x):\n # normalize to [0,1]\n x = kornia.geometry.resize(x, (224, 224),\n interpolation='bicubic', align_corners=True,\n antialias=self.antialias)\n x = (x + 1.) / 2.\n # renormalize according to clip\n x = kornia.enhance.normalize(x, self.mean, self.std)\n return x\n\n def freeze(self):\n self.model = self.model.eval()\n for param in self.parameters():\n param.requires_grad = False\n\n @autocast\n def forward(self, image, no_dropout=False):\n z = self.encode_with_vision_transformer(image)\n if self.ucg_rate > 0. and not no_dropout:\n z = torch.bernoulli((1. - self.ucg_rate) * torch.ones(z.shape[0], device=z.device))[:, None] * z\n return z\n\n def encode_with_vision_transformer(self, img):\n img = self.preprocess(img)\n x = self.model.visual(img)\n return x\n\n def encode(self, text):\n return self(text)\n\n\n\nclass FrozenOpenCLIPImageEmbedderV2(AbstractEncoder):\n \"\"\"\n Uses the OpenCLIP vision transformer encoder for images\n \"\"\"\n\n def __init__(self, arch=\"ViT-H-14\", version=\"laion2b_s32b_b79k\", device=\"cuda\",\n freeze=True, layer=\"pooled\", antialias=True):\n super().__init__()\n model, _, _ = open_clip.create_model_and_transforms(arch, device=torch.device('cpu'),\n pretrained=version, )\n del model.transformer\n self.model = model\n self.device = device\n\n if freeze:\n self.freeze()\n self.layer = layer\n if self.layer == \"penultimate\":\n raise NotImplementedError()\n self.layer_idx = 1\n\n self.antialias = antialias\n self.register_buffer('mean', torch.Tensor([0.48145466, 0.4578275, 0.40821073]), persistent=False)\n self.register_buffer('std', torch.Tensor([0.26862954, 0.26130258, 0.27577711]), persistent=False)\n\n\n def preprocess(self, x):\n # normalize to [0,1]\n x = kornia.geometry.resize(x, (224, 224),\n interpolation='bicubic', align_corners=True,\n antialias=self.antialias)\n x = (x + 1.) / 2.\n # renormalize according to clip\n x = kornia.enhance.normalize(x, self.mean, self.std)\n return x\n\n def freeze(self):\n self.model = self.model.eval()\n for param in self.model.parameters():\n param.requires_grad = False\n\n def forward(self, image, no_dropout=False):\n ## image: b c h w\n z = self.encode_with_vision_transformer(image)\n return z\n\n def encode_with_vision_transformer(self, x):\n x = self.preprocess(x)\n\n # to patches - whether to use dual patchnorm - https://arxiv.org/abs/2302.01327v1\n if self.model.visual.input_patchnorm:\n # einops - rearrange(x, 'b c (h p1) (w p2) -> b (h w) (c p1 p2)')\n x = x.reshape(x.shape[0], x.shape[1], self.model.visual.grid_size[0], self.model.visual.patch_size[0], self.model.visual.grid_size[1], self.model.visual.patch_size[1])\n x = x.permute(0, 2, 4, 1, 3, 5)\n x = x.reshape(x.shape[0], self.model.visual.grid_size[0] * self.model.visual.grid_size[1], -1)\n x = self.model.visual.patchnorm_pre_ln(x)\n x = self.model.visual.conv1(x)\n else:\n x = self.model.visual.conv1(x) # shape = [*, width, grid, grid]\n x = x.reshape(x.shape[0], x.shape[1], -1) # shape = [*, width, grid ** 2]\n x = x.permute(0, 2, 1) # shape = [*, grid ** 2, width]\n\n # class embeddings and positional embeddings\n x = torch.cat(\n [self.model.visual.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device),\n x], dim=1) # shape = [*, grid ** 2 + 1, width]\n x = x + self.model.visual.positional_embedding.to(x.dtype)\n\n # a patch_dropout of 0. would mean it is disabled and this function would do nothing but return what was passed in\n x = self.model.visual.patch_dropout(x)\n x = self.model.visual.ln_pre(x)\n\n x = x.permute(1, 0, 2) # NLD -> LND\n x = self.model.visual.transformer(x)\n x = x.permute(1, 0, 2) # LND -> NLD\n\n return x\n\n\nclass FrozenCLIPT5Encoder(AbstractEncoder):\n def __init__(self, clip_version=\"openai/clip-vit-large-patch14\", t5_version=\"google/t5-v1_1-xl\", device=\"cuda\",\n clip_max_length=77, t5_max_length=77):\n super().__init__()\n self.clip_encoder = FrozenCLIPEmbedder(clip_version, device, max_length=clip_max_length)\n self.t5_encoder = FrozenT5Embedder(t5_version, device, max_length=t5_max_length)\n print(f\"{self.clip_encoder.__class__.__name__} has {count_params(self.clip_encoder) * 1.e-6:.2f} M parameters, \"\n f\"{self.t5_encoder.__class__.__name__} comes with {count_params(self.t5_encoder) * 1.e-6:.2f} M params.\")\n\n def encode(self, text):\n return self(text)\n\n def forward(self, text):\n clip_z = self.clip_encoder.encode(text)\n t5_z = self.t5_encoder.encode(text)\n return [clip_z, t5_z]" }, { "path": "lvdm/modules/encoders/ip_resampler.py", "content": "# modified from https://github.com/mlfoundations/open_flamingo/blob/main/open_flamingo/src/helpers.py\nimport math\nimport torch\nimport torch.nn as nn\n\n\nclass ImageProjModel(nn.Module):\n \"\"\"Projection Model\"\"\"\n def __init__(self, cross_attention_dim=1024, clip_embeddings_dim=1024, clip_extra_context_tokens=4):\n super().__init__() \n self.cross_attention_dim = cross_attention_dim\n self.clip_extra_context_tokens = clip_extra_context_tokens\n self.proj = nn.Linear(clip_embeddings_dim, self.clip_extra_context_tokens * cross_attention_dim)\n self.norm = nn.LayerNorm(cross_attention_dim)\n \n def forward(self, image_embeds):\n #embeds = image_embeds\n embeds = image_embeds.type(list(self.proj.parameters())[0].dtype)\n clip_extra_context_tokens = self.proj(embeds).reshape(-1, self.clip_extra_context_tokens, self.cross_attention_dim)\n clip_extra_context_tokens = self.norm(clip_extra_context_tokens)\n return clip_extra_context_tokens\n\n# FFN\ndef FeedForward(dim, mult=4):\n inner_dim = int(dim * mult)\n return nn.Sequential(\n nn.LayerNorm(dim),\n nn.Linear(dim, inner_dim, bias=False),\n nn.GELU(),\n nn.Linear(inner_dim, dim, bias=False),\n )\n \n \ndef reshape_tensor(x, heads):\n bs, length, width = x.shape\n #(bs, length, width) --> (bs, length, n_heads, dim_per_head)\n x = x.view(bs, length, heads, -1)\n # (bs, length, n_heads, dim_per_head) --> (bs, n_heads, length, dim_per_head)\n x = x.transpose(1, 2)\n # (bs, n_heads, length, dim_per_head) --> (bs*n_heads, length, dim_per_head)\n x = x.reshape(bs, heads, length, -1)\n return x\n\n\nclass PerceiverAttention(nn.Module):\n def __init__(self, *, dim, dim_head=64, heads=8):\n super().__init__()\n self.scale = dim_head**-0.5\n self.dim_head = dim_head\n self.heads = heads\n inner_dim = dim_head * heads\n\n self.norm1 = nn.LayerNorm(dim)\n self.norm2 = nn.LayerNorm(dim)\n\n self.to_q = nn.Linear(dim, inner_dim, bias=False)\n self.to_kv = nn.Linear(dim, inner_dim * 2, bias=False)\n self.to_out = nn.Linear(inner_dim, dim, bias=False)\n\n\n def forward(self, x, latents):\n \"\"\"\n Args:\n x (torch.Tensor): image features\n shape (b, n1, D)\n latent (torch.Tensor): latent features\n shape (b, n2, D)\n \"\"\"\n x = self.norm1(x)\n latents = self.norm2(latents)\n \n b, l, _ = latents.shape\n\n q = self.to_q(latents)\n kv_input = torch.cat((x, latents), dim=-2)\n k, v = self.to_kv(kv_input).chunk(2, dim=-1)\n \n q = reshape_tensor(q, self.heads)\n k = reshape_tensor(k, self.heads)\n v = reshape_tensor(v, self.heads)\n\n # attention\n scale = 1 / math.sqrt(math.sqrt(self.dim_head))\n weight = (q * scale) @ (k * scale).transpose(-2, -1) # More stable with f16 than dividing afterwards\n weight = torch.softmax(weight.float(), dim=-1).type(weight.dtype)\n out = weight @ v\n \n out = out.permute(0, 2, 1, 3).reshape(b, l, -1)\n\n return self.to_out(out)\n\n\nclass Resampler(nn.Module):\n def __init__(\n self,\n dim=1024,\n depth=8,\n dim_head=64,\n heads=16,\n num_queries=8,\n embedding_dim=768,\n output_dim=1024,\n ff_mult=4,\n ):\n super().__init__()\n \n self.latents = nn.Parameter(torch.randn(1, num_queries, dim) / dim**0.5)\n \n self.proj_in = nn.Linear(embedding_dim, dim)\n\n self.proj_out = nn.Linear(dim, output_dim)\n self.norm_out = nn.LayerNorm(output_dim)\n \n self.layers = nn.ModuleList([])\n for _ in range(depth):\n self.layers.append(\n nn.ModuleList(\n [\n PerceiverAttention(dim=dim, dim_head=dim_head, heads=heads),\n FeedForward(dim=dim, mult=ff_mult),\n ]\n )\n )\n\n def forward(self, x):\n \n latents = self.latents.repeat(x.size(0), 1, 1)\n \n x = self.proj_in(x)\n \n for attn, ff in self.layers:\n latents = attn(x, latents) + latents\n latents = ff(latents) + latents\n \n latents = self.proj_out(latents)\n return self.norm_out(latents)" }, { "path": "lvdm/modules/networks/ae_modules.py", "content": "# pytorch_diffusion + derived encoder decoder\nimport math\nimport torch\nimport numpy as np\nimport torch.nn as nn\nfrom einops import rearrange\nfrom utils.utils import instantiate_from_config\nfrom lvdm.modules.attention import LinearAttention\n\ndef nonlinearity(x):\n # swish\n return x*torch.sigmoid(x)\n\n\ndef Normalize(in_channels, num_groups=32):\n return torch.nn.GroupNorm(num_groups=num_groups, num_channels=in_channels, eps=1e-6, affine=True)\n\n\n\nclass LinAttnBlock(LinearAttention):\n \"\"\"to match AttnBlock usage\"\"\"\n def __init__(self, in_channels):\n super().__init__(dim=in_channels, heads=1, dim_head=in_channels)\n\n\nclass AttnBlock(nn.Module):\n def __init__(self, in_channels):\n super().__init__()\n self.in_channels = in_channels\n\n self.norm = Normalize(in_channels)\n self.q = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.k = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.v = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n self.proj_out = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n\n def forward(self, x):\n h_ = x\n h_ = self.norm(h_)\n q = self.q(h_)\n k = self.k(h_)\n v = self.v(h_)\n\n # compute attention\n b,c,h,w = q.shape\n q = q.reshape(b,c,h*w) # bcl\n q = q.permute(0,2,1) # bcl -> blc l=hw\n k = k.reshape(b,c,h*w) # bcl\n \n w_ = torch.bmm(q,k) # b,hw,hw w[b,i,j]=sum_c q[b,i,c]k[b,c,j]\n w_ = w_ * (int(c)**(-0.5))\n w_ = torch.nn.functional.softmax(w_, dim=2)\n\n # attend to values\n v = v.reshape(b,c,h*w)\n w_ = w_.permute(0,2,1) # b,hw,hw (first hw of k, second of q)\n h_ = torch.bmm(v,w_) # b, c,hw (hw of q) h_[b,c,j] = sum_i v[b,c,i] w_[b,i,j]\n h_ = h_.reshape(b,c,h,w)\n\n h_ = self.proj_out(h_)\n\n return x+h_\n\ndef make_attn(in_channels, attn_type=\"vanilla\"):\n assert attn_type in [\"vanilla\", \"linear\", \"none\"], f'attn_type {attn_type} unknown'\n #print(f\"making attention of type '{attn_type}' with {in_channels} in_channels\")\n if attn_type == \"vanilla\":\n return AttnBlock(in_channels)\n elif attn_type == \"none\":\n return nn.Identity(in_channels)\n else:\n return LinAttnBlock(in_channels)\n \nclass Downsample(nn.Module):\n def __init__(self, in_channels, with_conv):\n super().__init__()\n self.with_conv = with_conv\n self.in_channels = in_channels\n if self.with_conv:\n # no asymmetric padding in torch conv, must do it ourselves\n self.conv = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=3,\n stride=2,\n padding=0)\n def forward(self, x):\n if self.with_conv:\n pad = (0,1,0,1)\n x = torch.nn.functional.pad(x, pad, mode=\"constant\", value=0)\n x = self.conv(x)\n else:\n x = torch.nn.functional.avg_pool2d(x, kernel_size=2, stride=2)\n return x\n\nclass Upsample(nn.Module):\n def __init__(self, in_channels, with_conv):\n super().__init__()\n self.with_conv = with_conv\n self.in_channels = in_channels\n if self.with_conv:\n self.conv = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, x):\n x = torch.nn.functional.interpolate(x, scale_factor=2.0, mode=\"nearest\")\n if self.with_conv:\n x = self.conv(x)\n return x\n\ndef get_timestep_embedding(timesteps, embedding_dim):\n \"\"\"\n This matches the implementation in Denoising Diffusion Probabilistic Models:\n From Fairseq.\n Build sinusoidal embeddings.\n This matches the implementation in tensor2tensor, but differs slightly\n from the description in Section 3.5 of \"Attention Is All You Need\".\n \"\"\"\n assert len(timesteps.shape) == 1\n\n half_dim = embedding_dim // 2\n emb = math.log(10000) / (half_dim - 1)\n emb = torch.exp(torch.arange(half_dim, dtype=torch.float32) * -emb)\n emb = emb.to(device=timesteps.device)\n emb = timesteps.float()[:, None] * emb[None, :]\n emb = torch.cat([torch.sin(emb), torch.cos(emb)], dim=1)\n if embedding_dim % 2 == 1: # zero pad\n emb = torch.nn.functional.pad(emb, (0,1,0,0))\n return emb\n\n\n\nclass ResnetBlock(nn.Module):\n def __init__(self, *, in_channels, out_channels=None, conv_shortcut=False,\n dropout, temb_channels=512):\n super().__init__()\n self.in_channels = in_channels\n out_channels = in_channels if out_channels is None else out_channels\n self.out_channels = out_channels\n self.use_conv_shortcut = conv_shortcut\n\n self.norm1 = Normalize(in_channels)\n self.conv1 = torch.nn.Conv2d(in_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n if temb_channels > 0:\n self.temb_proj = torch.nn.Linear(temb_channels,\n out_channels)\n self.norm2 = Normalize(out_channels)\n self.dropout = torch.nn.Dropout(dropout)\n self.conv2 = torch.nn.Conv2d(out_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n if self.in_channels != self.out_channels:\n if self.use_conv_shortcut:\n self.conv_shortcut = torch.nn.Conv2d(in_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n else:\n self.nin_shortcut = torch.nn.Conv2d(in_channels,\n out_channels,\n kernel_size=1,\n stride=1,\n padding=0)\n\n def forward(self, x, temb):\n h = x\n h = self.norm1(h)\n h = nonlinearity(h)\n h = self.conv1(h)\n\n if temb is not None:\n h = h + self.temb_proj(nonlinearity(temb))[:,:,None,None]\n\n h = self.norm2(h)\n h = nonlinearity(h)\n h = self.dropout(h)\n h = self.conv2(h)\n\n if self.in_channels != self.out_channels:\n if self.use_conv_shortcut:\n x = self.conv_shortcut(x)\n else:\n x = self.nin_shortcut(x)\n\n return x+h\n\nclass Model(nn.Module):\n def __init__(self, *, ch, out_ch, ch_mult=(1,2,4,8), num_res_blocks,\n attn_resolutions, dropout=0.0, resamp_with_conv=True, in_channels,\n resolution, use_timestep=True, use_linear_attn=False, attn_type=\"vanilla\"):\n super().__init__()\n if use_linear_attn: attn_type = \"linear\"\n self.ch = ch\n self.temb_ch = self.ch*4\n self.num_resolutions = len(ch_mult)\n self.num_res_blocks = num_res_blocks\n self.resolution = resolution\n self.in_channels = in_channels\n\n self.use_timestep = use_timestep\n if self.use_timestep:\n # timestep embedding\n self.temb = nn.Module()\n self.temb.dense = nn.ModuleList([\n torch.nn.Linear(self.ch,\n self.temb_ch),\n torch.nn.Linear(self.temb_ch,\n self.temb_ch),\n ])\n\n # downsampling\n self.conv_in = torch.nn.Conv2d(in_channels,\n self.ch,\n kernel_size=3,\n stride=1,\n padding=1)\n\n curr_res = resolution\n in_ch_mult = (1,)+tuple(ch_mult)\n self.down = nn.ModuleList()\n for i_level in range(self.num_resolutions):\n block = nn.ModuleList()\n attn = nn.ModuleList()\n block_in = ch*in_ch_mult[i_level]\n block_out = ch*ch_mult[i_level]\n for i_block in range(self.num_res_blocks):\n block.append(ResnetBlock(in_channels=block_in,\n out_channels=block_out,\n temb_channels=self.temb_ch,\n dropout=dropout))\n block_in = block_out\n if curr_res in attn_resolutions:\n attn.append(make_attn(block_in, attn_type=attn_type))\n down = nn.Module()\n down.block = block\n down.attn = attn\n if i_level != self.num_resolutions-1:\n down.downsample = Downsample(block_in, resamp_with_conv)\n curr_res = curr_res // 2\n self.down.append(down)\n\n # middle\n self.mid = nn.Module()\n self.mid.block_1 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n self.mid.attn_1 = make_attn(block_in, attn_type=attn_type)\n self.mid.block_2 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n\n # upsampling\n self.up = nn.ModuleList()\n for i_level in reversed(range(self.num_resolutions)):\n block = nn.ModuleList()\n attn = nn.ModuleList()\n block_out = ch*ch_mult[i_level]\n skip_in = ch*ch_mult[i_level]\n for i_block in range(self.num_res_blocks+1):\n if i_block == self.num_res_blocks:\n skip_in = ch*in_ch_mult[i_level]\n block.append(ResnetBlock(in_channels=block_in+skip_in,\n out_channels=block_out,\n temb_channels=self.temb_ch,\n dropout=dropout))\n block_in = block_out\n if curr_res in attn_resolutions:\n attn.append(make_attn(block_in, attn_type=attn_type))\n up = nn.Module()\n up.block = block\n up.attn = attn\n if i_level != 0:\n up.upsample = Upsample(block_in, resamp_with_conv)\n curr_res = curr_res * 2\n self.up.insert(0, up) # prepend to get consistent order\n\n # end\n self.norm_out = Normalize(block_in)\n self.conv_out = torch.nn.Conv2d(block_in,\n out_ch,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, x, t=None, context=None):\n #assert x.shape[2] == x.shape[3] == self.resolution\n if context is not None:\n # assume aligned context, cat along channel axis\n x = torch.cat((x, context), dim=1)\n if self.use_timestep:\n # timestep embedding\n assert t is not None\n temb = get_timestep_embedding(t, self.ch)\n temb = self.temb.dense[0](temb)\n temb = nonlinearity(temb)\n temb = self.temb.dense[1](temb)\n else:\n temb = None\n\n # downsampling\n hs = [self.conv_in(x)]\n for i_level in range(self.num_resolutions):\n for i_block in range(self.num_res_blocks):\n h = self.down[i_level].block[i_block](hs[-1], temb)\n if len(self.down[i_level].attn) > 0:\n h = self.down[i_level].attn[i_block](h)\n hs.append(h)\n if i_level != self.num_resolutions-1:\n hs.append(self.down[i_level].downsample(hs[-1]))\n\n # middle\n h = hs[-1]\n h = self.mid.block_1(h, temb)\n h = self.mid.attn_1(h)\n h = self.mid.block_2(h, temb)\n\n # upsampling\n for i_level in reversed(range(self.num_resolutions)):\n for i_block in range(self.num_res_blocks+1):\n h = self.up[i_level].block[i_block](\n torch.cat([h, hs.pop()], dim=1), temb)\n if len(self.up[i_level].attn) > 0:\n h = self.up[i_level].attn[i_block](h)\n if i_level != 0:\n h = self.up[i_level].upsample(h)\n\n # end\n h = self.norm_out(h)\n h = nonlinearity(h)\n h = self.conv_out(h)\n return h\n\n def get_last_layer(self):\n return self.conv_out.weight\n\n\nclass Encoder(nn.Module):\n def __init__(self, *, ch, out_ch, ch_mult=(1,2,4,8), num_res_blocks,\n attn_resolutions, dropout=0.0, resamp_with_conv=True, in_channels,\n resolution, z_channels, double_z=True, use_linear_attn=False, attn_type=\"vanilla\",\n **ignore_kwargs):\n super().__init__()\n if use_linear_attn: attn_type = \"linear\"\n self.ch = ch\n self.temb_ch = 0\n self.num_resolutions = len(ch_mult)\n self.num_res_blocks = num_res_blocks\n self.resolution = resolution\n self.in_channels = in_channels\n\n # downsampling\n self.conv_in = torch.nn.Conv2d(in_channels,\n self.ch,\n kernel_size=3,\n stride=1,\n padding=1)\n\n curr_res = resolution\n in_ch_mult = (1,)+tuple(ch_mult)\n self.in_ch_mult = in_ch_mult\n self.down = nn.ModuleList()\n for i_level in range(self.num_resolutions):\n block = nn.ModuleList()\n attn = nn.ModuleList()\n block_in = ch*in_ch_mult[i_level]\n block_out = ch*ch_mult[i_level]\n for i_block in range(self.num_res_blocks):\n block.append(ResnetBlock(in_channels=block_in,\n out_channels=block_out,\n temb_channels=self.temb_ch,\n dropout=dropout))\n block_in = block_out\n if curr_res in attn_resolutions:\n attn.append(make_attn(block_in, attn_type=attn_type))\n down = nn.Module()\n down.block = block\n down.attn = attn\n if i_level != self.num_resolutions-1:\n down.downsample = Downsample(block_in, resamp_with_conv)\n curr_res = curr_res // 2\n self.down.append(down)\n\n # middle\n self.mid = nn.Module()\n self.mid.block_1 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n self.mid.attn_1 = make_attn(block_in, attn_type=attn_type)\n self.mid.block_2 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n\n # end\n self.norm_out = Normalize(block_in)\n self.conv_out = torch.nn.Conv2d(block_in,\n 2*z_channels if double_z else z_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, x):\n # timestep embedding\n temb = None\n\n # print(f'encoder-input={x.shape}')\n # downsampling\n hs = [self.conv_in(x)]\n # print(f'encoder-conv in feat={hs[0].shape}')\n for i_level in range(self.num_resolutions):\n for i_block in range(self.num_res_blocks):\n h = self.down[i_level].block[i_block](hs[-1], temb)\n # print(f'encoder-down feat={h.shape}')\n if len(self.down[i_level].attn) > 0:\n h = self.down[i_level].attn[i_block](h)\n hs.append(h)\n if i_level != self.num_resolutions-1:\n # print(f'encoder-downsample (input)={hs[-1].shape}')\n hs.append(self.down[i_level].downsample(hs[-1]))\n # print(f'encoder-downsample (output)={hs[-1].shape}')\n\n # middle\n h = hs[-1]\n h = self.mid.block_1(h, temb)\n # print(f'encoder-mid1 feat={h.shape}')\n h = self.mid.attn_1(h)\n h = self.mid.block_2(h, temb)\n # print(f'encoder-mid2 feat={h.shape}')\n\n # end\n h = self.norm_out(h)\n h = nonlinearity(h)\n h = self.conv_out(h)\n # print(f'end feat={h.shape}')\n return h\n\n\nclass Decoder(nn.Module):\n def __init__(self, *, ch, out_ch, ch_mult=(1,2,4,8), num_res_blocks,\n attn_resolutions, dropout=0.0, resamp_with_conv=True, in_channels,\n resolution, z_channels, give_pre_end=False, tanh_out=False, use_linear_attn=False,\n attn_type=\"vanilla\", **ignorekwargs):\n super().__init__()\n if use_linear_attn: attn_type = \"linear\"\n self.ch = ch\n self.temb_ch = 0\n self.num_resolutions = len(ch_mult)\n self.num_res_blocks = num_res_blocks\n self.resolution = resolution\n self.in_channels = in_channels\n self.give_pre_end = give_pre_end\n self.tanh_out = tanh_out\n\n # compute in_ch_mult, block_in and curr_res at lowest res\n in_ch_mult = (1,)+tuple(ch_mult)\n block_in = ch*ch_mult[self.num_resolutions-1]\n curr_res = resolution // 2**(self.num_resolutions-1)\n self.z_shape = (1,z_channels,curr_res,curr_res)\n print(\"AE working on z of shape {} = {} dimensions.\".format(\n self.z_shape, np.prod(self.z_shape)))\n\n # z to block_in\n self.conv_in = torch.nn.Conv2d(z_channels,\n block_in,\n kernel_size=3,\n stride=1,\n padding=1)\n\n # middle\n self.mid = nn.Module()\n self.mid.block_1 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n self.mid.attn_1 = make_attn(block_in, attn_type=attn_type)\n self.mid.block_2 = ResnetBlock(in_channels=block_in,\n out_channels=block_in,\n temb_channels=self.temb_ch,\n dropout=dropout)\n\n # upsampling\n self.up = nn.ModuleList()\n for i_level in reversed(range(self.num_resolutions)):\n block = nn.ModuleList()\n attn = nn.ModuleList()\n block_out = ch*ch_mult[i_level]\n for i_block in range(self.num_res_blocks+1):\n block.append(ResnetBlock(in_channels=block_in,\n out_channels=block_out,\n temb_channels=self.temb_ch,\n dropout=dropout))\n block_in = block_out\n if curr_res in attn_resolutions:\n attn.append(make_attn(block_in, attn_type=attn_type))\n up = nn.Module()\n up.block = block\n up.attn = attn\n if i_level != 0:\n up.upsample = Upsample(block_in, resamp_with_conv)\n curr_res = curr_res * 2\n self.up.insert(0, up) # prepend to get consistent order\n\n # end\n self.norm_out = Normalize(block_in)\n self.conv_out = torch.nn.Conv2d(block_in,\n out_ch,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, z):\n #assert z.shape[1:] == self.z_shape[1:]\n self.last_z_shape = z.shape\n\n # print(f'decoder-input={z.shape}')\n # timestep embedding\n temb = None\n\n # z to block_in\n h = self.conv_in(z)\n # print(f'decoder-conv in feat={h.shape}')\n\n # middle\n h = self.mid.block_1(h, temb)\n h = self.mid.attn_1(h)\n h = self.mid.block_2(h, temb)\n # print(f'decoder-mid feat={h.shape}')\n\n # upsampling\n for i_level in reversed(range(self.num_resolutions)):\n for i_block in range(self.num_res_blocks+1):\n h = self.up[i_level].block[i_block](h, temb)\n if len(self.up[i_level].attn) > 0:\n h = self.up[i_level].attn[i_block](h)\n # print(f'decoder-up feat={h.shape}')\n if i_level != 0:\n h = self.up[i_level].upsample(h)\n # print(f'decoder-upsample feat={h.shape}')\n\n # end\n if self.give_pre_end:\n return h\n\n h = self.norm_out(h)\n h = nonlinearity(h)\n h = self.conv_out(h)\n # print(f'decoder-conv_out feat={h.shape}')\n if self.tanh_out:\n h = torch.tanh(h)\n return h\n\n\nclass SimpleDecoder(nn.Module):\n def __init__(self, in_channels, out_channels, *args, **kwargs):\n super().__init__()\n self.model = nn.ModuleList([nn.Conv2d(in_channels, in_channels, 1),\n ResnetBlock(in_channels=in_channels,\n out_channels=2 * in_channels,\n temb_channels=0, dropout=0.0),\n ResnetBlock(in_channels=2 * in_channels,\n out_channels=4 * in_channels,\n temb_channels=0, dropout=0.0),\n ResnetBlock(in_channels=4 * in_channels,\n out_channels=2 * in_channels,\n temb_channels=0, dropout=0.0),\n nn.Conv2d(2*in_channels, in_channels, 1),\n Upsample(in_channels, with_conv=True)])\n # end\n self.norm_out = Normalize(in_channels)\n self.conv_out = torch.nn.Conv2d(in_channels,\n out_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, x):\n for i, layer in enumerate(self.model):\n if i in [1,2,3]:\n x = layer(x, None)\n else:\n x = layer(x)\n\n h = self.norm_out(x)\n h = nonlinearity(h)\n x = self.conv_out(h)\n return x\n\n\nclass UpsampleDecoder(nn.Module):\n def __init__(self, in_channels, out_channels, ch, num_res_blocks, resolution,\n ch_mult=(2,2), dropout=0.0):\n super().__init__()\n # upsampling\n self.temb_ch = 0\n self.num_resolutions = len(ch_mult)\n self.num_res_blocks = num_res_blocks\n block_in = in_channels\n curr_res = resolution // 2 ** (self.num_resolutions - 1)\n self.res_blocks = nn.ModuleList()\n self.upsample_blocks = nn.ModuleList()\n for i_level in range(self.num_resolutions):\n res_block = []\n block_out = ch * ch_mult[i_level]\n for i_block in range(self.num_res_blocks + 1):\n res_block.append(ResnetBlock(in_channels=block_in,\n out_channels=block_out,\n temb_channels=self.temb_ch,\n dropout=dropout))\n block_in = block_out\n self.res_blocks.append(nn.ModuleList(res_block))\n if i_level != self.num_resolutions - 1:\n self.upsample_blocks.append(Upsample(block_in, True))\n curr_res = curr_res * 2\n\n # end\n self.norm_out = Normalize(block_in)\n self.conv_out = torch.nn.Conv2d(block_in,\n out_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n\n def forward(self, x):\n # upsampling\n h = x\n for k, i_level in enumerate(range(self.num_resolutions)):\n for i_block in range(self.num_res_blocks + 1):\n h = self.res_blocks[i_level][i_block](h, None)\n if i_level != self.num_resolutions - 1:\n h = self.upsample_blocks[k](h)\n h = self.norm_out(h)\n h = nonlinearity(h)\n h = self.conv_out(h)\n return h\n\n\nclass LatentRescaler(nn.Module):\n def __init__(self, factor, in_channels, mid_channels, out_channels, depth=2):\n super().__init__()\n # residual block, interpolate, residual block\n self.factor = factor\n self.conv_in = nn.Conv2d(in_channels,\n mid_channels,\n kernel_size=3,\n stride=1,\n padding=1)\n self.res_block1 = nn.ModuleList([ResnetBlock(in_channels=mid_channels,\n out_channels=mid_channels,\n temb_channels=0,\n dropout=0.0) for _ in range(depth)])\n self.attn = AttnBlock(mid_channels)\n self.res_block2 = nn.ModuleList([ResnetBlock(in_channels=mid_channels,\n out_channels=mid_channels,\n temb_channels=0,\n dropout=0.0) for _ in range(depth)])\n\n self.conv_out = nn.Conv2d(mid_channels,\n out_channels,\n kernel_size=1,\n )\n\n def forward(self, x):\n x = self.conv_in(x)\n for block in self.res_block1:\n x = block(x, None)\n x = torch.nn.functional.interpolate(x, size=(int(round(x.shape[2]*self.factor)), int(round(x.shape[3]*self.factor))))\n x = self.attn(x)\n for block in self.res_block2:\n x = block(x, None)\n x = self.conv_out(x)\n return x\n\n\nclass MergedRescaleEncoder(nn.Module):\n def __init__(self, in_channels, ch, resolution, out_ch, num_res_blocks,\n attn_resolutions, dropout=0.0, resamp_with_conv=True,\n ch_mult=(1,2,4,8), rescale_factor=1.0, rescale_module_depth=1):\n super().__init__()\n intermediate_chn = ch * ch_mult[-1]\n self.encoder = Encoder(in_channels=in_channels, num_res_blocks=num_res_blocks, ch=ch, ch_mult=ch_mult,\n z_channels=intermediate_chn, double_z=False, resolution=resolution,\n attn_resolutions=attn_resolutions, dropout=dropout, resamp_with_conv=resamp_with_conv,\n out_ch=None)\n self.rescaler = LatentRescaler(factor=rescale_factor, in_channels=intermediate_chn,\n mid_channels=intermediate_chn, out_channels=out_ch, depth=rescale_module_depth)\n\n def forward(self, x):\n x = self.encoder(x)\n x = self.rescaler(x)\n return x\n\n\nclass MergedRescaleDecoder(nn.Module):\n def __init__(self, z_channels, out_ch, resolution, num_res_blocks, attn_resolutions, ch, ch_mult=(1,2,4,8),\n dropout=0.0, resamp_with_conv=True, rescale_factor=1.0, rescale_module_depth=1):\n super().__init__()\n tmp_chn = z_channels*ch_mult[-1]\n self.decoder = Decoder(out_ch=out_ch, z_channels=tmp_chn, attn_resolutions=attn_resolutions, dropout=dropout,\n resamp_with_conv=resamp_with_conv, in_channels=None, num_res_blocks=num_res_blocks,\n ch_mult=ch_mult, resolution=resolution, ch=ch)\n self.rescaler = LatentRescaler(factor=rescale_factor, in_channels=z_channels, mid_channels=tmp_chn,\n out_channels=tmp_chn, depth=rescale_module_depth)\n\n def forward(self, x):\n x = self.rescaler(x)\n x = self.decoder(x)\n return x\n\n\nclass Upsampler(nn.Module):\n def __init__(self, in_size, out_size, in_channels, out_channels, ch_mult=2):\n super().__init__()\n assert out_size >= in_size\n num_blocks = int(np.log2(out_size//in_size))+1\n factor_up = 1.+ (out_size % in_size)\n print(f\"Building {self.__class__.__name__} with in_size: {in_size} --> out_size {out_size} and factor {factor_up}\")\n self.rescaler = LatentRescaler(factor=factor_up, in_channels=in_channels, mid_channels=2*in_channels,\n out_channels=in_channels)\n self.decoder = Decoder(out_ch=out_channels, resolution=out_size, z_channels=in_channels, num_res_blocks=2,\n attn_resolutions=[], in_channels=None, ch=in_channels,\n ch_mult=[ch_mult for _ in range(num_blocks)])\n\n def forward(self, x):\n x = self.rescaler(x)\n x = self.decoder(x)\n return x\n\n\nclass Resize(nn.Module):\n def __init__(self, in_channels=None, learned=False, mode=\"bilinear\"):\n super().__init__()\n self.with_conv = learned\n self.mode = mode\n if self.with_conv:\n print(f\"Note: {self.__class__.__name} uses learned downsampling and will ignore the fixed {mode} mode\")\n raise NotImplementedError()\n assert in_channels is not None\n # no asymmetric padding in torch conv, must do it ourselves\n self.conv = torch.nn.Conv2d(in_channels,\n in_channels,\n kernel_size=4,\n stride=2,\n padding=1)\n\n def forward(self, x, scale_factor=1.0):\n if scale_factor==1.0:\n return x\n else:\n x = torch.nn.functional.interpolate(x, mode=self.mode, align_corners=False, scale_factor=scale_factor)\n return x\n\nclass FirstStagePostProcessor(nn.Module):\n\n def __init__(self, ch_mult:list, in_channels,\n pretrained_model:nn.Module=None,\n reshape=False,\n n_channels=None,\n dropout=0.,\n pretrained_config=None):\n super().__init__()\n if pretrained_config is None:\n assert pretrained_model is not None, 'Either \"pretrained_model\" or \"pretrained_config\" must not be None'\n self.pretrained_model = pretrained_model\n else:\n assert pretrained_config is not None, 'Either \"pretrained_model\" or \"pretrained_config\" must not be None'\n self.instantiate_pretrained(pretrained_config)\n\n self.do_reshape = reshape\n\n if n_channels is None:\n n_channels = self.pretrained_model.encoder.ch\n\n self.proj_norm = Normalize(in_channels,num_groups=in_channels//2)\n self.proj = nn.Conv2d(in_channels,n_channels,kernel_size=3,\n stride=1,padding=1)\n\n blocks = []\n downs = []\n ch_in = n_channels\n for m in ch_mult:\n blocks.append(ResnetBlock(in_channels=ch_in,out_channels=m*n_channels,dropout=dropout))\n ch_in = m * n_channels\n downs.append(Downsample(ch_in, with_conv=False))\n\n self.model = nn.ModuleList(blocks)\n self.downsampler = nn.ModuleList(downs)\n\n\n def instantiate_pretrained(self, config):\n model = instantiate_from_config(config)\n self.pretrained_model = model.eval()\n # self.pretrained_model.train = False\n for param in self.pretrained_model.parameters():\n param.requires_grad = False\n\n\n @torch.no_grad()\n def encode_with_pretrained(self,x):\n c = self.pretrained_model.encode(x)\n if isinstance(c, DiagonalGaussianDistribution):\n c = c.mode()\n return c\n\n def forward(self,x):\n z_fs = self.encode_with_pretrained(x)\n z = self.proj_norm(z_fs)\n z = self.proj(z)\n z = nonlinearity(z)\n\n for submodel, downmodel in zip(self.model,self.downsampler):\n z = submodel(z,temb=None)\n z = downmodel(z)\n\n if self.do_reshape:\n z = rearrange(z,'b c h w -> b (h w) c')\n return z\n\n" }, { "path": "lvdm/modules/networks/openaimodel3d.py", "content": "from functools import partial\nfrom abc import abstractmethod\nimport torch\nimport torch.nn as nn\nfrom einops import rearrange\nimport torch.nn.functional as F\nfrom lvdm.models.utils_diffusion import timestep_embedding\nfrom lvdm.common import checkpoint\nfrom lvdm.basics import (\n zero_module,\n conv_nd,\n linear,\n avg_pool_nd,\n normalization\n)\nfrom lvdm.modules.attention import SpatialTransformer, TemporalTransformer\n\n\nclass TimestepBlock(nn.Module):\n \"\"\"\n Any module where forward() takes timestep embeddings as a second argument.\n \"\"\"\n @abstractmethod\n def forward(self, x, emb):\n \"\"\"\n Apply the module to `x` given `emb` timestep embeddings.\n \"\"\"\n\n\nclass TimestepEmbedSequential(nn.Sequential, TimestepBlock):\n \"\"\"\n A sequential module that passes timestep embeddings to the children that\n support it as an extra input.\n \"\"\"\n\n def forward(self, x, emb, context=None, batch_size=None):\n for layer in self:\n if isinstance(layer, TimestepBlock):\n x = layer(x, emb, batch_size)\n elif isinstance(layer, SpatialTransformer):\n x = layer(x, context)\n elif isinstance(layer, TemporalTransformer):\n x = rearrange(x, '(b f) c h w -> b c f h w', b=batch_size)\n x = layer(x, context)\n x = rearrange(x, 'b c f h w -> (b f) c h w')\n else:\n x = layer(x,)\n return x\n\n\nclass Downsample(nn.Module):\n \"\"\"\n A downsampling layer with an optional convolution.\n :param channels: channels in the inputs and outputs.\n :param use_conv: a bool determining if a convolution is applied.\n :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then\n downsampling occurs in the inner-two dimensions.\n \"\"\"\n\n def __init__(self, channels, use_conv, dims=2, out_channels=None, padding=1):\n super().__init__()\n self.channels = channels\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.dims = dims\n stride = 2 if dims != 3 else (1, 2, 2)\n if use_conv:\n self.op = conv_nd(\n dims, self.channels, self.out_channels, 3, stride=stride, padding=padding\n )\n else:\n assert self.channels == self.out_channels\n self.op = avg_pool_nd(dims, kernel_size=stride, stride=stride)\n\n def forward(self, x):\n assert x.shape[1] == self.channels\n return self.op(x)\n\n\nclass Upsample(nn.Module):\n \"\"\"\n An upsampling layer with an optional convolution.\n :param channels: channels in the inputs and outputs.\n :param use_conv: a bool determining if a convolution is applied.\n :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then\n upsampling occurs in the inner-two dimensions.\n \"\"\"\n\n def __init__(self, channels, use_conv, dims=2, out_channels=None, padding=1):\n super().__init__()\n self.channels = channels\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.dims = dims\n if use_conv:\n self.conv = conv_nd(dims, self.channels, self.out_channels, 3, padding=padding)\n\n def forward(self, x):\n assert x.shape[1] == self.channels\n if self.dims == 3:\n x = F.interpolate(x, (x.shape[2], x.shape[3] * 2, x.shape[4] * 2), mode='nearest')\n else:\n x = F.interpolate(x, scale_factor=2, mode='nearest')\n if self.use_conv:\n x = self.conv(x)\n return x\n\n\nclass ResBlock(TimestepBlock):\n \"\"\"\n A residual block that can optionally change the number of channels.\n :param channels: the number of input channels.\n :param emb_channels: the number of timestep embedding channels.\n :param dropout: the rate of dropout.\n :param out_channels: if specified, the number of out channels.\n :param use_conv: if True and out_channels is specified, use a spatial\n convolution instead of a smaller 1x1 convolution to change the\n channels in the skip connection.\n :param dims: determines if the signal is 1D, 2D, or 3D.\n :param up: if True, use this block for upsampling.\n :param down: if True, use this block for downsampling.\n \"\"\"\n\n def __init__(\n self,\n channels,\n emb_channels,\n dropout,\n out_channels=None,\n use_scale_shift_norm=False,\n dims=2,\n use_checkpoint=False,\n use_conv=False,\n up=False,\n down=False,\n use_temporal_conv=False,\n tempspatial_aware=False\n ):\n super().__init__()\n self.channels = channels\n self.emb_channels = emb_channels\n self.dropout = dropout\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.use_checkpoint = use_checkpoint\n self.use_scale_shift_norm = use_scale_shift_norm\n self.use_temporal_conv = use_temporal_conv\n\n self.in_layers = nn.Sequential(\n normalization(channels),\n nn.SiLU(),\n conv_nd(dims, channels, self.out_channels, 3, padding=1),\n )\n\n self.updown = up or down\n\n if up:\n self.h_upd = Upsample(channels, False, dims)\n self.x_upd = Upsample(channels, False, dims)\n elif down:\n self.h_upd = Downsample(channels, False, dims)\n self.x_upd = Downsample(channels, False, dims)\n else:\n self.h_upd = self.x_upd = nn.Identity()\n\n self.emb_layers = nn.Sequential(\n nn.SiLU(),\n nn.Linear(\n emb_channels,\n 2 * self.out_channels if use_scale_shift_norm else self.out_channels,\n ),\n )\n self.out_layers = nn.Sequential(\n normalization(self.out_channels),\n nn.SiLU(),\n nn.Dropout(p=dropout),\n zero_module(nn.Conv2d(self.out_channels, self.out_channels, 3, padding=1)),\n )\n\n if self.out_channels == channels:\n self.skip_connection = nn.Identity()\n elif use_conv:\n self.skip_connection = conv_nd(dims, channels, self.out_channels, 3, padding=1)\n else:\n self.skip_connection = conv_nd(dims, channels, self.out_channels, 1)\n\n if self.use_temporal_conv:\n self.temopral_conv = TemporalConvBlock(\n self.out_channels,\n self.out_channels,\n dropout=0.1,\n spatial_aware=tempspatial_aware\n )\n\n def forward(self, x, emb, batch_size=None):\n \"\"\"\n Apply the block to a Tensor, conditioned on a timestep embedding.\n :param x: an [N x C x ...] Tensor of features.\n :param emb: an [N x emb_channels] Tensor of timestep embeddings.\n :return: an [N x C x ...] Tensor of outputs.\n \"\"\"\n input_tuple = (x, emb,)\n if batch_size:\n forward_batchsize = partial(self._forward, batch_size=batch_size)\n return checkpoint(forward_batchsize, input_tuple, self.parameters(), self.use_checkpoint)\n return checkpoint(self._forward, input_tuple, self.parameters(), self.use_checkpoint)\n\n def _forward(self, x, emb, batch_size=None,):\n if self.updown:\n in_rest, in_conv = self.in_layers[:-1], self.in_layers[-1]\n h = in_rest(x)\n h = self.h_upd(h)\n x = self.x_upd(x)\n h = in_conv(h)\n else:\n h = self.in_layers(x)\n emb_out = self.emb_layers(emb).type(h.dtype)\n while len(emb_out.shape) < len(h.shape):\n emb_out = emb_out[..., None]\n if self.use_scale_shift_norm:\n out_norm, out_rest = self.out_layers[0], self.out_layers[1:]\n scale, shift = torch.chunk(emb_out, 2, dim=1)\n h = out_norm(h) * (1 + scale) + shift\n h = out_rest(h)\n else:\n h = h + emb_out\n h = self.out_layers(h)\n h = self.skip_connection(x) + h\n \n if self.use_temporal_conv and batch_size:\n h = rearrange(h, '(b t) c h w -> b c t h w', b=batch_size)\n h = self.temopral_conv(h)\n h = rearrange(h, 'b c t h w -> (b t) c h w')\n return h\n\n\nclass TemporalConvBlock(nn.Module):\n \"\"\"\n Adapted from modelscope: https://github.com/modelscope/modelscope/blob/master/modelscope/models/multi_modal/video_synthesis/unet_sd.py\n \"\"\"\n\n def __init__(self, in_channels, out_channels=None, dropout=0.0, spatial_aware=False):\n super(TemporalConvBlock, self).__init__()\n if out_channels is None:\n out_channels = in_channels\n self.in_channels = in_channels\n self.out_channels = out_channels\n kernel_shape = (3, 1, 1) if not spatial_aware else (3, 3, 3)\n padding_shape = (1, 0, 0) if not spatial_aware else (1, 1, 1)\n\n # conv layers\n self.conv1 = nn.Sequential(\n nn.GroupNorm(32, in_channels), nn.SiLU(),\n nn.Conv3d(in_channels, out_channels, kernel_shape, padding=padding_shape))\n self.conv2 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, kernel_shape, padding=padding_shape))\n self.conv3 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, (3, 1, 1), padding=(1, 0, 0)))\n self.conv4 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, (3, 1, 1), padding=(1, 0, 0)))\n\n # zero out the last layer params,so the conv block is identity\n nn.init.zeros_(self.conv4[-1].weight)\n nn.init.zeros_(self.conv4[-1].bias)\n\n def forward(self, x):\n identity = x\n x = self.conv1(x)\n x = self.conv2(x)\n x = self.conv3(x)\n x = self.conv4(x)\n\n return x + identity\n\n\nclass UNetModel(nn.Module):\n \"\"\"\n The full UNet model with attention and timestep embedding.\n :param in_channels: in_channels in the input Tensor.\n :param model_channels: base channel count for the model.\n :param out_channels: channels in the output Tensor.\n :param num_res_blocks: number of residual blocks per downsample.\n :param attention_resolutions: a collection of downsample rates at which\n attention will take place. May be a set, list, or tuple.\n For example, if this contains 4, then at 4x downsampling, attention\n will be used.\n :param dropout: the dropout probability.\n :param channel_mult: channel multiplier for each level of the UNet.\n :param conv_resample: if True, use learned convolutions for upsampling and\n downsampling.\n :param dims: determines if the signal is 1D, 2D, or 3D.\n :param num_classes: if specified (as an int), then this model will be\n class-conditional with `num_classes` classes.\n :param use_checkpoint: use gradient checkpointing to reduce memory usage.\n :param num_heads: the number of attention heads in each attention layer.\n :param num_heads_channels: if specified, ignore num_heads and instead use\n a fixed channel width per attention head.\n :param num_heads_upsample: works with num_heads to set a different number\n of heads for upsampling. Deprecated.\n :param use_scale_shift_norm: use a FiLM-like conditioning mechanism.\n :param resblock_updown: use residual blocks for up/downsampling.\n \"\"\"\n\n def __init__(self,\n in_channels,\n model_channels,\n out_channels,\n num_res_blocks,\n attention_resolutions,\n dropout=0.0,\n channel_mult=(1, 2, 4, 8),\n conv_resample=True,\n dims=2,\n context_dim=None,\n use_scale_shift_norm=False,\n resblock_updown=False,\n num_heads=-1,\n num_head_channels=-1,\n transformer_depth=1,\n use_linear=False,\n use_checkpoint=False,\n temporal_conv=False,\n tempspatial_aware=False,\n temporal_attention=True,\n temporal_selfatt_only=True,\n use_relative_position=True,\n use_causal_attention=False,\n temporal_length=None,\n use_fp16=False,\n addition_attention=False,\n use_image_attention=False,\n temporal_transformer_depth=1,\n fps_cond=False,\n ):\n super(UNetModel, self).__init__()\n if num_heads == -1:\n assert num_head_channels != -1, 'Either num_heads or num_head_channels has to be set'\n if num_head_channels == -1:\n assert num_heads != -1, 'Either num_heads or num_head_channels has to be set'\n\n self.in_channels = in_channels\n self.model_channels = model_channels\n self.out_channels = out_channels\n self.num_res_blocks = num_res_blocks\n self.attention_resolutions = attention_resolutions\n self.dropout = dropout\n self.channel_mult = channel_mult\n self.conv_resample = conv_resample\n self.temporal_attention = temporal_attention\n time_embed_dim = model_channels * 4\n self.use_checkpoint = use_checkpoint\n self.dtype = torch.float16 if use_fp16 else torch.float32\n self.addition_attention=addition_attention\n self.use_image_attention = use_image_attention\n self.fps_cond=fps_cond\n\n\n\n self.time_embed = nn.Sequential(\n linear(model_channels, time_embed_dim),\n nn.SiLU(),\n linear(time_embed_dim, time_embed_dim),\n )\n if self.fps_cond:\n self.fps_embedding = nn.Sequential(\n linear(model_channels, time_embed_dim),\n nn.SiLU(),\n linear(time_embed_dim, time_embed_dim),\n )\n\n self.input_blocks = nn.ModuleList(\n [\n TimestepEmbedSequential(conv_nd(dims, in_channels, model_channels, 3, padding=1))\n ]\n )\n if self.addition_attention:\n self.init_attn=TimestepEmbedSequential(\n TemporalTransformer(\n model_channels,\n n_heads=8,\n d_head=num_head_channels,\n depth=transformer_depth,\n context_dim=context_dim,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length))\n \n input_block_chans = [model_channels]\n ch = model_channels\n ds = 1\n for level, mult in enumerate(channel_mult):\n for _ in range(num_res_blocks):\n layers = [\n ResBlock(ch, time_embed_dim, dropout,\n out_channels=mult * model_channels, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n ]\n ch = mult * model_channels\n if ds in attention_resolutions:\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers.append(\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention\n )\n )\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n self.input_blocks.append(TimestepEmbedSequential(*layers))\n input_block_chans.append(ch)\n if level != len(channel_mult) - 1:\n out_ch = ch\n self.input_blocks.append(\n TimestepEmbedSequential(\n ResBlock(ch, time_embed_dim, dropout, \n out_channels=out_ch, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm,\n down=True\n )\n if resblock_updown\n else Downsample(ch, conv_resample, dims=dims, out_channels=out_ch)\n )\n )\n ch = out_ch\n input_block_chans.append(ch)\n ds *= 2\n\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers = [\n ResBlock(ch, time_embed_dim, dropout,\n dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n ),\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention\n )\n ]\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n layers.append(\n ResBlock(ch, time_embed_dim, dropout,\n dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n )\n self.middle_block = TimestepEmbedSequential(*layers)\n\n self.output_blocks = nn.ModuleList([])\n for level, mult in list(enumerate(channel_mult))[::-1]:\n for i in range(num_res_blocks + 1):\n ich = input_block_chans.pop()\n layers = [\n ResBlock(ch + ich, time_embed_dim, dropout,\n out_channels=mult * model_channels, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n ]\n ch = model_channels * mult\n if ds in attention_resolutions:\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers.append(\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention\n )\n )\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n if level and i == num_res_blocks:\n out_ch = ch\n layers.append(\n ResBlock(ch, time_embed_dim, dropout,\n out_channels=out_ch, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm,\n up=True\n )\n if resblock_updown\n else Upsample(ch, conv_resample, dims=dims, out_channels=out_ch)\n )\n ds //= 2\n self.output_blocks.append(TimestepEmbedSequential(*layers))\n\n self.out = nn.Sequential(\n normalization(ch),\n nn.SiLU(),\n zero_module(conv_nd(dims, model_channels, out_channels, 3, padding=1)),\n )\n\n def forward(self, x, timesteps, context=None, features_adapter=None, fps=16, **kwargs):\n t_emb = timestep_embedding(timesteps, self.model_channels, repeat_only=False)\n emb = self.time_embed(t_emb)\n\n if self.fps_cond:\n if type(fps) == int:\n fps = torch.full_like(timesteps, fps)\n fps_emb = timestep_embedding(fps,self.model_channels, repeat_only=False)\n emb += self.fps_embedding(fps_emb)\n\n b,_,t,_,_ = x.shape\n ## repeat t times for context [(b t) 77 768] & time embedding\n if len(context.shape) < 4:\n context = context.repeat_interleave(repeats=t, dim=0)\n else:\n context = context.view(-1, context.shape[2], context.shape[3])\n # context = context.repeat_interleave(repeats=t, dim=0)\n emb = emb.repeat_interleave(repeats=t, dim=0)\n\n ## always in shape (b t) c h w, except for temporal layer\n x = rearrange(x, 'b c t h w -> (b t) c h w')\n\n h = x.type(self.dtype)\n adapter_idx = 0\n hs = []\n for id, module in enumerate(self.input_blocks):\n h = module(h, emb, context=context, batch_size=b)\n if id ==0 and self.addition_attention:\n h = self.init_attn(h, emb, context=context, batch_size=b)\n ## plug-in adapter features\n if ((id+1)%3 == 0) and features_adapter is not None:\n h = h + features_adapter[adapter_idx]\n adapter_idx += 1\n hs.append(h)\n if features_adapter is not None:\n assert len(features_adapter)==adapter_idx, 'Wrong features_adapter'\n\n h = self.middle_block(h, emb, context=context, batch_size=b)\n for module in self.output_blocks:\n h = torch.cat([h, hs.pop()], dim=1)\n h = module(h, emb, context=context, batch_size=b)\n h = h.type(x.dtype)\n y = self.out(h)\n \n # reshape back to (b c t h w)\n y = rearrange(y, '(b t) c h w -> b c t h w', b=b)\n return y\n " }, { "path": "lvdm/modules/networks/openaimodel3d_freenoise.py", "content": "from functools import partial\nfrom abc import abstractmethod\nimport torch\nimport torch.nn as nn\nfrom einops import rearrange\nimport torch.nn.functional as F\nfrom lvdm.models.utils_diffusion import timestep_embedding\nfrom lvdm.common import checkpoint\nfrom lvdm.basics import (\n zero_module,\n conv_nd,\n linear,\n avg_pool_nd,\n normalization\n)\nfrom lvdm.modules.attention_freenoise import SpatialTransformer, TemporalTransformer\n\n\nclass TimestepBlock(nn.Module):\n \"\"\"\n Any module where forward() takes timestep embeddings as a second argument.\n \"\"\"\n @abstractmethod\n def forward(self, x, emb):\n \"\"\"\n Apply the module to `x` given `emb` timestep embeddings.\n \"\"\"\n\n\nclass TimestepEmbedSequential(nn.Sequential, TimestepBlock):\n \"\"\"\n A sequential module that passes timestep embeddings to the children that\n support it as an extra input.\n \"\"\"\n\n def forward(self, x, emb, context=None, batch_size=None, use_injection=False, **kwargs):\n for layer in self:\n if isinstance(layer, TimestepBlock):\n x = layer(x, emb, batch_size)\n elif isinstance(layer, SpatialTransformer):\n x = layer(x, context, use_injection=use_injection)\n elif isinstance(layer, TemporalTransformer):\n x = rearrange(x, '(b f) c h w -> b c f h w', b=batch_size)\n x = layer(x, context, **kwargs)\n x = rearrange(x, 'b c f h w -> (b f) c h w')\n else:\n x = layer(x,)\n return x\n\n\nclass Downsample(nn.Module):\n \"\"\"\n A downsampling layer with an optional convolution.\n :param channels: channels in the inputs and outputs.\n :param use_conv: a bool determining if a convolution is applied.\n :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then\n downsampling occurs in the inner-two dimensions.\n \"\"\"\n\n def __init__(self, channels, use_conv, dims=2, out_channels=None, padding=1):\n super().__init__()\n self.channels = channels\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.dims = dims\n stride = 2 if dims != 3 else (1, 2, 2)\n if use_conv:\n self.op = conv_nd(\n dims, self.channels, self.out_channels, 3, stride=stride, padding=padding\n )\n else:\n assert self.channels == self.out_channels\n self.op = avg_pool_nd(dims, kernel_size=stride, stride=stride)\n\n def forward(self, x):\n assert x.shape[1] == self.channels\n return self.op(x)\n\n\nclass Upsample(nn.Module):\n \"\"\"\n An upsampling layer with an optional convolution.\n :param channels: channels in the inputs and outputs.\n :param use_conv: a bool determining if a convolution is applied.\n :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then\n upsampling occurs in the inner-two dimensions.\n \"\"\"\n\n def __init__(self, channels, use_conv, dims=2, out_channels=None, padding=1):\n super().__init__()\n self.channels = channels\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.dims = dims\n if use_conv:\n self.conv = conv_nd(dims, self.channels, self.out_channels, 3, padding=padding)\n\n def forward(self, x):\n assert x.shape[1] == self.channels\n if self.dims == 3:\n x = F.interpolate(x, (x.shape[2], x.shape[3] * 2, x.shape[4] * 2), mode='nearest')\n else:\n x = F.interpolate(x, scale_factor=2, mode='nearest')\n if self.use_conv:\n x = self.conv(x)\n return x\n\n\nclass ResBlock(TimestepBlock):\n \"\"\"\n A residual block that can optionally change the number of channels.\n :param channels: the number of input channels.\n :param emb_channels: the number of timestep embedding channels.\n :param dropout: the rate of dropout.\n :param out_channels: if specified, the number of out channels.\n :param use_conv: if True and out_channels is specified, use a spatial\n convolution instead of a smaller 1x1 convolution to change the\n channels in the skip connection.\n :param dims: determines if the signal is 1D, 2D, or 3D.\n :param up: if True, use this block for upsampling.\n :param down: if True, use this block for downsampling.\n \"\"\"\n\n def __init__(\n self,\n channels,\n emb_channels,\n dropout,\n out_channels=None,\n use_scale_shift_norm=False,\n dims=2,\n use_checkpoint=False,\n use_conv=False,\n up=False,\n down=False,\n use_temporal_conv=False,\n tempspatial_aware=False\n ):\n super().__init__()\n self.channels = channels\n self.emb_channels = emb_channels\n self.dropout = dropout\n self.out_channels = out_channels or channels\n self.use_conv = use_conv\n self.use_checkpoint = use_checkpoint\n self.use_scale_shift_norm = use_scale_shift_norm\n self.use_temporal_conv = use_temporal_conv\n\n self.in_layers = nn.Sequential(\n normalization(channels),\n nn.SiLU(),\n conv_nd(dims, channels, self.out_channels, 3, padding=1),\n )\n\n self.updown = up or down\n\n if up:\n self.h_upd = Upsample(channels, False, dims)\n self.x_upd = Upsample(channels, False, dims)\n elif down:\n self.h_upd = Downsample(channels, False, dims)\n self.x_upd = Downsample(channels, False, dims)\n else:\n self.h_upd = self.x_upd = nn.Identity()\n\n self.emb_layers = nn.Sequential(\n nn.SiLU(),\n nn.Linear(\n emb_channels,\n 2 * self.out_channels if use_scale_shift_norm else self.out_channels,\n ),\n )\n self.out_layers = nn.Sequential(\n normalization(self.out_channels),\n nn.SiLU(),\n nn.Dropout(p=dropout),\n zero_module(nn.Conv2d(self.out_channels, self.out_channels, 3, padding=1)),\n )\n\n if self.out_channels == channels:\n self.skip_connection = nn.Identity()\n elif use_conv:\n self.skip_connection = conv_nd(dims, channels, self.out_channels, 3, padding=1)\n else:\n self.skip_connection = conv_nd(dims, channels, self.out_channels, 1)\n\n if self.use_temporal_conv:\n self.temopral_conv = TemporalConvBlock(\n self.out_channels,\n self.out_channels,\n dropout=0.1,\n spatial_aware=tempspatial_aware\n )\n\n def forward(self, x, emb, batch_size=None):\n \"\"\"\n Apply the block to a Tensor, conditioned on a timestep embedding.\n :param x: an [N x C x ...] Tensor of features.\n :param emb: an [N x emb_channels] Tensor of timestep embeddings.\n :return: an [N x C x ...] Tensor of outputs.\n \"\"\"\n input_tuple = (x, emb,)\n if batch_size:\n forward_batchsize = partial(self._forward, batch_size=batch_size)\n return checkpoint(forward_batchsize, input_tuple, self.parameters(), self.use_checkpoint)\n return checkpoint(self._forward, input_tuple, self.parameters(), self.use_checkpoint)\n\n def _forward(self, x, emb, batch_size=None,):\n if self.updown:\n in_rest, in_conv = self.in_layers[:-1], self.in_layers[-1]\n h = in_rest(x)\n h = self.h_upd(h)\n x = self.x_upd(x)\n h = in_conv(h)\n else:\n h = self.in_layers(x)\n emb_out = self.emb_layers(emb).type(h.dtype)\n while len(emb_out.shape) < len(h.shape):\n emb_out = emb_out[..., None]\n if self.use_scale_shift_norm:\n out_norm, out_rest = self.out_layers[0], self.out_layers[1:]\n scale, shift = torch.chunk(emb_out, 2, dim=1)\n h = out_norm(h) * (1 + scale) + shift\n h = out_rest(h)\n else:\n h = h + emb_out\n h = self.out_layers(h)\n h = self.skip_connection(x) + h\n \n if self.use_temporal_conv and batch_size:\n h = rearrange(h, '(b t) c h w -> b c t h w', b=batch_size)\n h = self.temopral_conv(h)\n h = rearrange(h, 'b c t h w -> (b t) c h w')\n return h\n\n\nclass TemporalConvBlock(nn.Module):\n \"\"\"\n Adapted from modelscope: https://github.com/modelscope/modelscope/blob/master/modelscope/models/multi_modal/video_synthesis/unet_sd.py\n \"\"\"\n\n def __init__(self, in_channels, out_channels=None, dropout=0.0, spatial_aware=False):\n super(TemporalConvBlock, self).__init__()\n if out_channels is None:\n out_channels = in_channels\n self.in_channels = in_channels\n self.out_channels = out_channels\n kernel_shape = (3, 1, 1) if not spatial_aware else (3, 3, 3)\n padding_shape = (1, 0, 0) if not spatial_aware else (1, 1, 1)\n\n # conv layers\n self.conv1 = nn.Sequential(\n nn.GroupNorm(32, in_channels), nn.SiLU(),\n nn.Conv3d(in_channels, out_channels, kernel_shape, padding=padding_shape))\n self.conv2 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, kernel_shape, padding=padding_shape))\n self.conv3 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, (3, 1, 1), padding=(1, 0, 0)))\n self.conv4 = nn.Sequential(\n nn.GroupNorm(32, out_channels), nn.SiLU(), nn.Dropout(dropout),\n nn.Conv3d(out_channels, in_channels, (3, 1, 1), padding=(1, 0, 0)))\n\n # zero out the last layer params,so the conv block is identity\n nn.init.zeros_(self.conv4[-1].weight)\n nn.init.zeros_(self.conv4[-1].bias)\n\n def forward(self, x):\n identity = x\n x = self.conv1(x)\n x = self.conv2(x)\n x = self.conv3(x)\n x = self.conv4(x)\n\n return x + identity\n\n\nclass UNetModel(nn.Module):\n \"\"\"\n The full UNet model with attention and timestep embedding.\n :param in_channels: in_channels in the input Tensor.\n :param model_channels: base channel count for the model.\n :param out_channels: channels in the output Tensor.\n :param num_res_blocks: number of residual blocks per downsample.\n :param attention_resolutions: a collection of downsample rates at which\n attention will take place. May be a set, list, or tuple.\n For example, if this contains 4, then at 4x downsampling, attention\n will be used.\n :param dropout: the dropout probability.\n :param channel_mult: channel multiplier for each level of the UNet.\n :param conv_resample: if True, use learned convolutions for upsampling and\n downsampling.\n :param dims: determines if the signal is 1D, 2D, or 3D.\n :param num_classes: if specified (as an int), then this model will be\n class-conditional with `num_classes` classes.\n :param use_checkpoint: use gradient checkpointing to reduce memory usage.\n :param num_heads: the number of attention heads in each attention layer.\n :param num_heads_channels: if specified, ignore num_heads and instead use\n a fixed channel width per attention head.\n :param num_heads_upsample: works with num_heads to set a different number\n of heads for upsampling. Deprecated.\n :param use_scale_shift_norm: use a FiLM-like conditioning mechanism.\n :param resblock_updown: use residual blocks for up/downsampling.\n \"\"\"\n\n def __init__(self,\n in_channels,\n model_channels,\n out_channels,\n num_res_blocks,\n attention_resolutions,\n dropout=0.0,\n channel_mult=(1, 2, 4, 8),\n conv_resample=True,\n dims=2,\n context_dim=None,\n use_scale_shift_norm=False,\n resblock_updown=False,\n num_heads=-1,\n num_head_channels=-1,\n transformer_depth=1,\n use_linear=False,\n use_checkpoint=False,\n temporal_conv=False,\n tempspatial_aware=False,\n temporal_attention=True,\n temporal_selfatt_only=True,\n use_relative_position=True,\n use_causal_attention=False,\n temporal_length=None,\n use_fp16=False,\n addition_attention=False,\n use_image_attention=False,\n temporal_transformer_depth=1,\n fps_cond=False,\n ):\n super(UNetModel, self).__init__()\n if num_heads == -1:\n assert num_head_channels != -1, 'Either num_heads or num_head_channels has to be set'\n if num_head_channels == -1:\n assert num_heads != -1, 'Either num_heads or num_head_channels has to be set'\n\n self.in_channels = in_channels\n self.model_channels = model_channels\n self.out_channels = out_channels\n self.num_res_blocks = num_res_blocks\n self.attention_resolutions = attention_resolutions\n self.dropout = dropout\n self.channel_mult = channel_mult\n self.conv_resample = conv_resample\n self.temporal_attention = temporal_attention\n time_embed_dim = model_channels * 4\n self.use_checkpoint = use_checkpoint\n self.dtype = torch.float16 if use_fp16 else torch.float32\n self.addition_attention=addition_attention\n self.use_image_attention = use_image_attention\n self.fps_cond=fps_cond\n\n\n\n self.time_embed = nn.Sequential(\n linear(model_channels, time_embed_dim),\n nn.SiLU(),\n linear(time_embed_dim, time_embed_dim),\n )\n if self.fps_cond:\n self.fps_embedding = nn.Sequential(\n linear(model_channels, time_embed_dim),\n nn.SiLU(),\n linear(time_embed_dim, time_embed_dim),\n )\n\n self.input_blocks = nn.ModuleList(\n [\n TimestepEmbedSequential(conv_nd(dims, in_channels, model_channels, 3, padding=1))\n ]\n )\n if self.addition_attention:\n self.init_attn=TimestepEmbedSequential(\n TemporalTransformer(\n model_channels,\n n_heads=8,\n d_head=num_head_channels,\n depth=transformer_depth,\n context_dim=context_dim,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length))\n \n input_block_chans = [model_channels]\n ch = model_channels\n ds = 1\n for level, mult in enumerate(channel_mult):\n for _ in range(num_res_blocks):\n layers = [\n ResBlock(ch, time_embed_dim, dropout,\n out_channels=mult * model_channels, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n ]\n ch = mult * model_channels\n if ds in attention_resolutions:\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers.append(\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention, injection=True\n )\n )\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n self.input_blocks.append(TimestepEmbedSequential(*layers))\n input_block_chans.append(ch)\n if level != len(channel_mult) - 1:\n out_ch = ch\n self.input_blocks.append(\n TimestepEmbedSequential(\n ResBlock(ch, time_embed_dim, dropout, \n out_channels=out_ch, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm,\n down=True\n )\n if resblock_updown\n else Downsample(ch, conv_resample, dims=dims, out_channels=out_ch)\n )\n )\n ch = out_ch\n input_block_chans.append(ch)\n ds *= 2\n\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers = [\n ResBlock(ch, time_embed_dim, dropout,\n dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n ),\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention, injection=True\n )\n ]\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n layers.append(\n ResBlock(ch, time_embed_dim, dropout,\n dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n )\n self.middle_block = TimestepEmbedSequential(*layers)\n\n self.output_blocks = nn.ModuleList([])\n for level, mult in list(enumerate(channel_mult))[::-1]:\n for i in range(num_res_blocks + 1):\n ich = input_block_chans.pop()\n layers = [\n ResBlock(ch + ich, time_embed_dim, dropout,\n out_channels=mult * model_channels, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm, tempspatial_aware=tempspatial_aware,\n use_temporal_conv=temporal_conv\n )\n ]\n ch = model_channels * mult\n if ds in attention_resolutions:\n if num_head_channels == -1:\n dim_head = ch // num_heads\n else:\n num_heads = ch // num_head_channels\n dim_head = num_head_channels\n layers.append(\n SpatialTransformer(ch, num_heads, dim_head, \n depth=transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, disable_self_attn=False,\n img_cross_attention=self.use_image_attention, injection=False\n )\n )\n if self.temporal_attention:\n layers.append(\n TemporalTransformer(ch, num_heads, dim_head,\n depth=temporal_transformer_depth, context_dim=context_dim, use_linear=use_linear,\n use_checkpoint=use_checkpoint, only_self_att=temporal_selfatt_only, \n causal_attention=use_causal_attention, relative_position=use_relative_position, \n temporal_length=temporal_length\n )\n )\n if level and i == num_res_blocks:\n out_ch = ch\n layers.append(\n ResBlock(ch, time_embed_dim, dropout,\n out_channels=out_ch, dims=dims, use_checkpoint=use_checkpoint,\n use_scale_shift_norm=use_scale_shift_norm,\n up=True\n )\n if resblock_updown\n else Upsample(ch, conv_resample, dims=dims, out_channels=out_ch)\n )\n ds //= 2\n self.output_blocks.append(TimestepEmbedSequential(*layers))\n\n self.out = nn.Sequential(\n normalization(ch),\n nn.SiLU(),\n zero_module(conv_nd(dims, model_channels, out_channels, 3, padding=1)),\n )\n\n def forward(self, x, timesteps, context=None, features_adapter=None, fps=16, **kwargs):\n t_emb = timestep_embedding(timesteps, self.model_channels, repeat_only=False)\n emb = self.time_embed(t_emb)\n\n if self.fps_cond:\n if type(fps) == int:\n fps = torch.full_like(timesteps, fps)\n fps_emb = timestep_embedding(fps,self.model_channels, repeat_only=False)\n emb += self.fps_embedding(fps_emb)\n\n b,_,t,_,_ = x.shape\n ## repeat t times for context [(b t) 77 768] & time embedding\n if len(context.shape) < 4:\n context = context.repeat_interleave(repeats=t, dim=0)\n else:\n context = context.view(-1, context.shape[2], context.shape[3])\n # context = context.repeat_interleave(repeats=t, dim=0)\n emb = emb.repeat_interleave(repeats=t, dim=0)\n\n ## always in shape (b t) c h w, except for temporal layer\n x = rearrange(x, 'b c t h w -> (b t) c h w')\n\n h = x.type(self.dtype)\n adapter_idx = 0\n hs = []\n for id, module in enumerate(self.input_blocks):\n h = module(h, emb, context=context, batch_size=b, **kwargs)\n if id ==0 and self.addition_attention:\n h = self.init_attn(h, emb, context=context, batch_size=b, **kwargs)\n ## plug-in adapter features\n if ((id+1)%3 == 0) and features_adapter is not None:\n h = h + features_adapter[adapter_idx]\n adapter_idx += 1\n hs.append(h)\n if features_adapter is not None:\n assert len(features_adapter)==adapter_idx, 'Wrong features_adapter'\n\n h = self.middle_block(h, emb, context=context, batch_size=b, **kwargs)\n for module in self.output_blocks:\n h = torch.cat([h, hs.pop()], dim=1)\n h = module(h, emb, context=context, batch_size=b, **kwargs)\n h = h.type(x.dtype)\n y = self.out(h)\n \n # reshape back to (b c t h w)\n y = rearrange(y, '(b t) c h w -> b c t h w', b=b)\n return y\n " }, { "path": "lvdm/modules/x_transformer.py", "content": "\"\"\"shout-out to https://github.com/lucidrains/x-transformers/tree/main/x_transformers\"\"\"\nfrom functools import partial\nfrom inspect import isfunction\nfrom collections import namedtuple\nfrom einops import rearrange, repeat\nimport torch\nfrom torch import nn, einsum\nimport torch.nn.functional as F\n\n# constants\nDEFAULT_DIM_HEAD = 64\n\nIntermediates = namedtuple('Intermediates', [\n 'pre_softmax_attn',\n 'post_softmax_attn'\n])\n\nLayerIntermediates = namedtuple('Intermediates', [\n 'hiddens',\n 'attn_intermediates'\n])\n\n\nclass AbsolutePositionalEmbedding(nn.Module):\n def __init__(self, dim, max_seq_len):\n super().__init__()\n self.emb = nn.Embedding(max_seq_len, dim)\n self.init_()\n\n def init_(self):\n nn.init.normal_(self.emb.weight, std=0.02)\n\n def forward(self, x):\n n = torch.arange(x.shape[1], device=x.device)\n return self.emb(n)[None, :, :]\n\n\nclass FixedPositionalEmbedding(nn.Module):\n def __init__(self, dim):\n super().__init__()\n inv_freq = 1. / (10000 ** (torch.arange(0, dim, 2).float() / dim))\n self.register_buffer('inv_freq', inv_freq)\n\n def forward(self, x, seq_dim=1, offset=0):\n t = torch.arange(x.shape[seq_dim], device=x.device).type_as(self.inv_freq) + offset\n sinusoid_inp = torch.einsum('i , j -> i j', t, self.inv_freq)\n emb = torch.cat((sinusoid_inp.sin(), sinusoid_inp.cos()), dim=-1)\n return emb[None, :, :]\n\n\n# helpers\n\ndef exists(val):\n return val is not None\n\n\ndef default(val, d):\n if exists(val):\n return val\n return d() if isfunction(d) else d\n\n\ndef always(val):\n def inner(*args, **kwargs):\n return val\n return inner\n\n\ndef not_equals(val):\n def inner(x):\n return x != val\n return inner\n\n\ndef equals(val):\n def inner(x):\n return x == val\n return inner\n\n\ndef max_neg_value(tensor):\n return -torch.finfo(tensor.dtype).max\n\n\n# keyword argument helpers\n\ndef pick_and_pop(keys, d):\n values = list(map(lambda key: d.pop(key), keys))\n return dict(zip(keys, values))\n\n\ndef group_dict_by_key(cond, d):\n return_val = [dict(), dict()]\n for key in d.keys():\n match = bool(cond(key))\n ind = int(not match)\n return_val[ind][key] = d[key]\n return (*return_val,)\n\n\ndef string_begins_with(prefix, str):\n return str.startswith(prefix)\n\n\ndef group_by_key_prefix(prefix, d):\n return group_dict_by_key(partial(string_begins_with, prefix), d)\n\n\ndef groupby_prefix_and_trim(prefix, d):\n kwargs_with_prefix, kwargs = group_dict_by_key(partial(string_begins_with, prefix), d)\n kwargs_without_prefix = dict(map(lambda x: (x[0][len(prefix):], x[1]), tuple(kwargs_with_prefix.items())))\n return kwargs_without_prefix, kwargs\n\n\n# classes\nclass Scale(nn.Module):\n def __init__(self, value, fn):\n super().__init__()\n self.value = value\n self.fn = fn\n\n def forward(self, x, **kwargs):\n x, *rest = self.fn(x, **kwargs)\n return (x * self.value, *rest)\n\n\nclass Rezero(nn.Module):\n def __init__(self, fn):\n super().__init__()\n self.fn = fn\n self.g = nn.Parameter(torch.zeros(1))\n\n def forward(self, x, **kwargs):\n x, *rest = self.fn(x, **kwargs)\n return (x * self.g, *rest)\n\n\nclass ScaleNorm(nn.Module):\n def __init__(self, dim, eps=1e-5):\n super().__init__()\n self.scale = dim ** -0.5\n self.eps = eps\n self.g = nn.Parameter(torch.ones(1))\n\n def forward(self, x):\n norm = torch.norm(x, dim=-1, keepdim=True) * self.scale\n return x / norm.clamp(min=self.eps) * self.g\n\n\nclass RMSNorm(nn.Module):\n def __init__(self, dim, eps=1e-8):\n super().__init__()\n self.scale = dim ** -0.5\n self.eps = eps\n self.g = nn.Parameter(torch.ones(dim))\n\n def forward(self, x):\n norm = torch.norm(x, dim=-1, keepdim=True) * self.scale\n return x / norm.clamp(min=self.eps) * self.g\n\n\nclass Residual(nn.Module):\n def forward(self, x, residual):\n return x + residual\n\n\nclass GRUGating(nn.Module):\n def __init__(self, dim):\n super().__init__()\n self.gru = nn.GRUCell(dim, dim)\n\n def forward(self, x, residual):\n gated_output = self.gru(\n rearrange(x, 'b n d -> (b n) d'),\n rearrange(residual, 'b n d -> (b n) d')\n )\n\n return gated_output.reshape_as(x)\n\n\n# feedforward\n\nclass GEGLU(nn.Module):\n def __init__(self, dim_in, dim_out):\n super().__init__()\n self.proj = nn.Linear(dim_in, dim_out * 2)\n\n def forward(self, x):\n x, gate = self.proj(x).chunk(2, dim=-1)\n return x * F.gelu(gate)\n\n\nclass FeedForward(nn.Module):\n def __init__(self, dim, dim_out=None, mult=4, glu=False, dropout=0.):\n super().__init__()\n inner_dim = int(dim * mult)\n dim_out = default(dim_out, dim)\n project_in = nn.Sequential(\n nn.Linear(dim, inner_dim),\n nn.GELU()\n ) if not glu else GEGLU(dim, inner_dim)\n\n self.net = nn.Sequential(\n project_in,\n nn.Dropout(dropout),\n nn.Linear(inner_dim, dim_out)\n )\n\n def forward(self, x):\n return self.net(x)\n\n\n# attention.\nclass Attention(nn.Module):\n def __init__(\n self,\n dim,\n dim_head=DEFAULT_DIM_HEAD,\n heads=8,\n causal=False,\n mask=None,\n talking_heads=False,\n sparse_topk=None,\n use_entmax15=False,\n num_mem_kv=0,\n dropout=0.,\n on_attn=False\n ):\n super().__init__()\n if use_entmax15:\n raise NotImplementedError(\"Check out entmax activation instead of softmax activation!\")\n self.scale = dim_head ** -0.5\n self.heads = heads\n self.causal = causal\n self.mask = mask\n\n inner_dim = dim_head * heads\n\n self.to_q = nn.Linear(dim, inner_dim, bias=False)\n self.to_k = nn.Linear(dim, inner_dim, bias=False)\n self.to_v = nn.Linear(dim, inner_dim, bias=False)\n self.dropout = nn.Dropout(dropout)\n\n # talking heads\n self.talking_heads = talking_heads\n if talking_heads:\n self.pre_softmax_proj = nn.Parameter(torch.randn(heads, heads))\n self.post_softmax_proj = nn.Parameter(torch.randn(heads, heads))\n\n # explicit topk sparse attention\n self.sparse_topk = sparse_topk\n\n # entmax\n #self.attn_fn = entmax15 if use_entmax15 else F.softmax\n self.attn_fn = F.softmax\n\n # add memory key / values\n self.num_mem_kv = num_mem_kv\n if num_mem_kv > 0:\n self.mem_k = nn.Parameter(torch.randn(heads, num_mem_kv, dim_head))\n self.mem_v = nn.Parameter(torch.randn(heads, num_mem_kv, dim_head))\n\n # attention on attention\n self.attn_on_attn = on_attn\n self.to_out = nn.Sequential(nn.Linear(inner_dim, dim * 2), nn.GLU()) if on_attn else nn.Linear(inner_dim, dim)\n\n def forward(\n self,\n x,\n context=None,\n mask=None,\n context_mask=None,\n rel_pos=None,\n sinusoidal_emb=None,\n prev_attn=None,\n mem=None\n ):\n b, n, _, h, talking_heads, device = *x.shape, self.heads, self.talking_heads, x.device\n kv_input = default(context, x)\n\n q_input = x\n k_input = kv_input\n v_input = kv_input\n\n if exists(mem):\n k_input = torch.cat((mem, k_input), dim=-2)\n v_input = torch.cat((mem, v_input), dim=-2)\n\n if exists(sinusoidal_emb):\n # in shortformer, the query would start at a position offset depending on the past cached memory\n offset = k_input.shape[-2] - q_input.shape[-2]\n q_input = q_input + sinusoidal_emb(q_input, offset=offset)\n k_input = k_input + sinusoidal_emb(k_input)\n\n q = self.to_q(q_input)\n k = self.to_k(k_input)\n v = self.to_v(v_input)\n\n q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h=h), (q, k, v))\n\n input_mask = None\n if any(map(exists, (mask, context_mask))):\n q_mask = default(mask, lambda: torch.ones((b, n), device=device).bool())\n k_mask = q_mask if not exists(context) else context_mask\n k_mask = default(k_mask, lambda: torch.ones((b, k.shape[-2]), device=device).bool())\n q_mask = rearrange(q_mask, 'b i -> b () i ()')\n k_mask = rearrange(k_mask, 'b j -> b () () j')\n input_mask = q_mask * k_mask\n\n if self.num_mem_kv > 0:\n mem_k, mem_v = map(lambda t: repeat(t, 'h n d -> b h n d', b=b), (self.mem_k, self.mem_v))\n k = torch.cat((mem_k, k), dim=-2)\n v = torch.cat((mem_v, v), dim=-2)\n if exists(input_mask):\n input_mask = F.pad(input_mask, (self.num_mem_kv, 0), value=True)\n\n dots = einsum('b h i d, b h j d -> b h i j', q, k) * self.scale\n mask_value = max_neg_value(dots)\n\n if exists(prev_attn):\n dots = dots + prev_attn\n\n pre_softmax_attn = dots\n\n if talking_heads:\n dots = einsum('b h i j, h k -> b k i j', dots, self.pre_softmax_proj).contiguous()\n\n if exists(rel_pos):\n dots = rel_pos(dots)\n\n if exists(input_mask):\n dots.masked_fill_(~input_mask, mask_value)\n del input_mask\n\n if self.causal:\n i, j = dots.shape[-2:]\n r = torch.arange(i, device=device)\n mask = rearrange(r, 'i -> () () i ()') < rearrange(r, 'j -> () () () j')\n mask = F.pad(mask, (j - i, 0), value=False)\n dots.masked_fill_(mask, mask_value)\n del mask\n\n if exists(self.sparse_topk) and self.sparse_topk < dots.shape[-1]:\n top, _ = dots.topk(self.sparse_topk, dim=-1)\n vk = top[..., -1].unsqueeze(-1).expand_as(dots)\n mask = dots < vk\n dots.masked_fill_(mask, mask_value)\n del mask\n\n attn = self.attn_fn(dots, dim=-1)\n post_softmax_attn = attn\n\n attn = self.dropout(attn)\n\n if talking_heads:\n attn = einsum('b h i j, h k -> b k i j', attn, self.post_softmax_proj).contiguous()\n\n out = einsum('b h i j, b h j d -> b h i d', attn, v)\n out = rearrange(out, 'b h n d -> b n (h d)')\n\n intermediates = Intermediates(\n pre_softmax_attn=pre_softmax_attn,\n post_softmax_attn=post_softmax_attn\n )\n\n return self.to_out(out), intermediates\n\n\nclass AttentionLayers(nn.Module):\n def __init__(\n self,\n dim,\n depth,\n heads=8,\n causal=False,\n cross_attend=False,\n only_cross=False,\n use_scalenorm=False,\n use_rmsnorm=False,\n use_rezero=False,\n rel_pos_num_buckets=32,\n rel_pos_max_distance=128,\n position_infused_attn=False,\n custom_layers=None,\n sandwich_coef=None,\n par_ratio=None,\n residual_attn=False,\n cross_residual_attn=False,\n macaron=False,\n pre_norm=True,\n gate_residual=False,\n **kwargs\n ):\n super().__init__()\n ff_kwargs, kwargs = groupby_prefix_and_trim('ff_', kwargs)\n attn_kwargs, _ = groupby_prefix_and_trim('attn_', kwargs)\n\n dim_head = attn_kwargs.get('dim_head', DEFAULT_DIM_HEAD)\n\n self.dim = dim\n self.depth = depth\n self.layers = nn.ModuleList([])\n\n self.has_pos_emb = position_infused_attn\n self.pia_pos_emb = FixedPositionalEmbedding(dim) if position_infused_attn else None\n self.rotary_pos_emb = always(None)\n\n assert rel_pos_num_buckets <= rel_pos_max_distance, 'number of relative position buckets must be less than the relative position max distance'\n self.rel_pos = None\n\n self.pre_norm = pre_norm\n\n self.residual_attn = residual_attn\n self.cross_residual_attn = cross_residual_attn\n\n norm_class = ScaleNorm if use_scalenorm else nn.LayerNorm\n norm_class = RMSNorm if use_rmsnorm else norm_class\n norm_fn = partial(norm_class, dim)\n\n norm_fn = nn.Identity if use_rezero else norm_fn\n branch_fn = Rezero if use_rezero else None\n\n if cross_attend and not only_cross:\n default_block = ('a', 'c', 'f')\n elif cross_attend and only_cross:\n default_block = ('c', 'f')\n else:\n default_block = ('a', 'f')\n\n if macaron:\n default_block = ('f',) + default_block\n\n if exists(custom_layers):\n layer_types = custom_layers\n elif exists(par_ratio):\n par_depth = depth * len(default_block)\n assert 1 < par_ratio <= par_depth, 'par ratio out of range'\n default_block = tuple(filter(not_equals('f'), default_block))\n par_attn = par_depth // par_ratio\n depth_cut = par_depth * 2 // 3 # 2 / 3 attention layer cutoff suggested by PAR paper\n par_width = (depth_cut + depth_cut // par_attn) // par_attn\n assert len(default_block) <= par_width, 'default block is too large for par_ratio'\n par_block = default_block + ('f',) * (par_width - len(default_block))\n par_head = par_block * par_attn\n layer_types = par_head + ('f',) * (par_depth - len(par_head))\n elif exists(sandwich_coef):\n assert sandwich_coef > 0 and sandwich_coef <= depth, 'sandwich coefficient should be less than the depth'\n layer_types = ('a',) * sandwich_coef + default_block * (depth - sandwich_coef) + ('f',) * sandwich_coef\n else:\n layer_types = default_block * depth\n\n self.layer_types = layer_types\n self.num_attn_layers = len(list(filter(equals('a'), layer_types)))\n\n for layer_type in self.layer_types:\n if layer_type == 'a':\n layer = Attention(dim, heads=heads, causal=causal, **attn_kwargs)\n elif layer_type == 'c':\n layer = Attention(dim, heads=heads, **attn_kwargs)\n elif layer_type == 'f':\n layer = FeedForward(dim, **ff_kwargs)\n layer = layer if not macaron else Scale(0.5, layer)\n else:\n raise Exception(f'invalid layer type {layer_type}')\n\n if isinstance(layer, Attention) and exists(branch_fn):\n layer = branch_fn(layer)\n\n if gate_residual:\n residual_fn = GRUGating(dim)\n else:\n residual_fn = Residual()\n\n self.layers.append(nn.ModuleList([\n norm_fn(),\n layer,\n residual_fn\n ]))\n\n def forward(\n self,\n x,\n context=None,\n mask=None,\n context_mask=None,\n mems=None,\n return_hiddens=False\n ):\n hiddens = []\n intermediates = []\n prev_attn = None\n prev_cross_attn = None\n\n mems = mems.copy() if exists(mems) else [None] * self.num_attn_layers\n\n for ind, (layer_type, (norm, block, residual_fn)) in enumerate(zip(self.layer_types, self.layers)):\n is_last = ind == (len(self.layers) - 1)\n\n if layer_type == 'a':\n hiddens.append(x)\n layer_mem = mems.pop(0)\n\n residual = x\n\n if self.pre_norm:\n x = norm(x)\n\n if layer_type == 'a':\n out, inter = block(x, mask=mask, sinusoidal_emb=self.pia_pos_emb, rel_pos=self.rel_pos,\n prev_attn=prev_attn, mem=layer_mem)\n elif layer_type == 'c':\n out, inter = block(x, context=context, mask=mask, context_mask=context_mask, prev_attn=prev_cross_attn)\n elif layer_type == 'f':\n out = block(x)\n\n x = residual_fn(out, residual)\n\n if layer_type in ('a', 'c'):\n intermediates.append(inter)\n\n if layer_type == 'a' and self.residual_attn:\n prev_attn = inter.pre_softmax_attn\n elif layer_type == 'c' and self.cross_residual_attn:\n prev_cross_attn = inter.pre_softmax_attn\n\n if not self.pre_norm and not is_last:\n x = norm(x)\n\n if return_hiddens:\n intermediates = LayerIntermediates(\n hiddens=hiddens,\n attn_intermediates=intermediates\n )\n\n return x, intermediates\n\n return x\n\n\nclass Encoder(AttentionLayers):\n def __init__(self, **kwargs):\n assert 'causal' not in kwargs, 'cannot set causality on encoder'\n super().__init__(causal=False, **kwargs)\n\n\n\nclass TransformerWrapper(nn.Module):\n def __init__(\n self,\n *,\n num_tokens,\n max_seq_len,\n attn_layers,\n emb_dim=None,\n max_mem_len=0.,\n emb_dropout=0.,\n num_memory_tokens=None,\n tie_embedding=False,\n use_pos_emb=True\n ):\n super().__init__()\n assert isinstance(attn_layers, AttentionLayers), 'attention layers must be one of Encoder or Decoder'\n\n dim = attn_layers.dim\n emb_dim = default(emb_dim, dim)\n\n self.max_seq_len = max_seq_len\n self.max_mem_len = max_mem_len\n self.num_tokens = num_tokens\n\n self.token_emb = nn.Embedding(num_tokens, emb_dim)\n self.pos_emb = AbsolutePositionalEmbedding(emb_dim, max_seq_len) if (\n use_pos_emb and not attn_layers.has_pos_emb) else always(0)\n self.emb_dropout = nn.Dropout(emb_dropout)\n\n self.project_emb = nn.Linear(emb_dim, dim) if emb_dim != dim else nn.Identity()\n self.attn_layers = attn_layers\n self.norm = nn.LayerNorm(dim)\n\n self.init_()\n\n self.to_logits = nn.Linear(dim, num_tokens) if not tie_embedding else lambda t: t @ self.token_emb.weight.t()\n\n # memory tokens (like [cls]) from Memory Transformers paper\n num_memory_tokens = default(num_memory_tokens, 0)\n self.num_memory_tokens = num_memory_tokens\n if num_memory_tokens > 0:\n self.memory_tokens = nn.Parameter(torch.randn(num_memory_tokens, dim))\n\n # let funnel encoder know number of memory tokens, if specified\n if hasattr(attn_layers, 'num_memory_tokens'):\n attn_layers.num_memory_tokens = num_memory_tokens\n\n def init_(self):\n nn.init.normal_(self.token_emb.weight, std=0.02)\n\n def forward(\n self,\n x,\n return_embeddings=False,\n mask=None,\n return_mems=False,\n return_attn=False,\n mems=None,\n **kwargs\n ):\n b, n, device, num_mem = *x.shape, x.device, self.num_memory_tokens\n x = self.token_emb(x)\n x += self.pos_emb(x)\n x = self.emb_dropout(x)\n\n x = self.project_emb(x)\n\n if num_mem > 0:\n mem = repeat(self.memory_tokens, 'n d -> b n d', b=b)\n x = torch.cat((mem, x), dim=1)\n\n # auto-handle masking after appending memory tokens\n if exists(mask):\n mask = F.pad(mask, (num_mem, 0), value=True)\n\n x, intermediates = self.attn_layers(x, mask=mask, mems=mems, return_hiddens=True, **kwargs)\n x = self.norm(x)\n\n mem, x = x[:, :num_mem], x[:, num_mem:]\n\n out = self.to_logits(x) if not return_embeddings else x\n\n if return_mems:\n hiddens = intermediates.hiddens\n new_mems = list(map(lambda pair: torch.cat(pair, dim=-2), zip(mems, hiddens))) if exists(mems) else hiddens\n new_mems = list(map(lambda t: t[..., -self.max_mem_len:, :].detach(), new_mems))\n return out, new_mems\n\n if return_attn:\n attn_maps = list(map(lambda t: t.post_softmax_attn, intermediates.attn_intermediates))\n return out, attn_maps\n\n return out\n\n" }, { "path": "predict.py", "content": "# Prediction interface for Cog āš™ļø\n# https://github.com/replicate/cog/blob/main/docs/python.md\n\n\nimport os\nimport sys\nimport argparse\nimport random\nfrom omegaconf import OmegaConf\nfrom einops import rearrange, repeat\nimport torch\nimport torchvision\nfrom pytorch_lightning import seed_everything\nfrom cog import BasePredictor, Input, Path\n\nsys.path.insert(0, \"scripts/evaluation\")\nfrom funcs import (\n batch_ddim_sampling_freenoise,\n load_model_checkpoint,\n load_image_batch,\n get_filelist,\n)\nfrom utils.utils import instantiate_from_config\n\n\nclass Predictor(BasePredictor):\n def setup(self) -> None:\n \"\"\"Load the model into memory to make running multiple predictions efficient\"\"\"\n\n ckpt_path_1024 = \"checkpoints/base_1024_v1/model.ckpt\"\n config_1024 = \"configs/inference_t2v_1024_v1.0_freenoise.yaml\"\n ckpt_path_256 = \"checkpoints/base_256_v1/model.pth\"\n config_256 = \"configs/inference_t2v_tconv256_v1.0_freenoise.yaml\"\n\n config_1024 = OmegaConf.load(config_1024)\n model_config_1024 = config_1024.pop(\"model\", OmegaConf.create())\n self.model_1024 = instantiate_from_config(model_config_1024)\n self.model_1024 = self.model_1024.cuda()\n self.model_1024 = load_model_checkpoint(self.model_1024, ckpt_path_1024)\n self.model_1024.eval()\n\n config_256 = OmegaConf.load(config_256)\n model_config_256 = config_256.pop(\"model\", OmegaConf.create())\n self.model_256 = instantiate_from_config(model_config_256)\n self.model_256 = self.model_256.cuda()\n self.model_256 = load_model_checkpoint(self.model_256, ckpt_path_256)\n self.model_256.eval()\n\n def predict(\n self,\n prompt: str = Input(\n description=\"Prompt for video generation.\",\n default=\"A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect.\",\n ),\n output_size: str = Input(\n description=\"Choose the size of the output video.\",\n choices=[\"576x1024\", \"256x256\"],\n default=\"576x1024\",\n ),\n num_frames: int = Input(\n description=\"Number for frames to generate.\", default=32\n ),\n ddim_steps: int = Input(description=\"Number of denoising steps.\", default=50),\n unconditional_guidance_scale: float = Input(\n description=\"Classifier-free guidance scale.\", default=12.0\n ),\n seed: int = Input(\n description=\"Random seed. Leave blank to randomize the seed\", default=None\n ),\n save_fps: int = Input(\n description=\"Frame per second for the generated video.\", default=10\n ),\n window_size: int = Input(description=\"Window size.\", default=16),\n window_stride: int = Input(description=\"Window stride.\", default=4),\n ) -> Path:\n\n width = 1024 if output_size == \"576x1024\" else 256\n height = 576 if output_size == \"576x1024\" else 256\n fps = 28 if output_size == \"576x1024\" else 8\n model = self.model_1024 if output_size == \"576x1024\" else self.model_256\n\n if seed is None:\n seed = int.from_bytes(os.urandom(2), \"big\")\n print(f\"Using seed: {seed}\")\n seed_everything(seed)\n\n args = argparse.Namespace(\n mode=\"base\",\n savefps=save_fps,\n n_samples=1,\n ddim_steps=ddim_steps,\n ddim_eta=0.0,\n bs=1,\n height=height,\n width=width,\n frames=num_frames,\n fps=fps,\n unconditional_guidance_scale=unconditional_guidance_scale,\n unconditional_guidance_scale_temporal=None,\n cond_input=None,\n window_size=window_size,\n window_stride=window_stride,\n )\n\n ## latent noise shape\n h, w = args.height // 8, args.width // 8\n frames = model.temporal_length if args.frames < 0 else args.frames\n channels = model.channels\n\n x_T_total = torch.randn(\n [args.n_samples, 1, channels, frames, h, w], device=model.device\n ).repeat(1, args.bs, 1, 1, 1, 1)\n for frame_index in range(args.window_size, args.frames, args.window_stride):\n list_index = list(\n range(\n frame_index - args.window_size,\n frame_index + args.window_stride - args.window_size,\n )\n )\n random.shuffle(list_index)\n x_T_total[\n :, :, :, frame_index : frame_index + args.window_stride\n ] = x_T_total[:, :, :, list_index]\n\n batch_size = 1\n noise_shape = [batch_size, channels, frames, h, w]\n fps = torch.tensor([args.fps] * batch_size).to(model.device).long()\n prompts = [prompt]\n text_emb = model.get_learned_conditioning(prompts)\n\n if args.mode == \"base\":\n cond = {\"c_crossattn\": [text_emb], \"fps\": fps}\n elif args.mode == \"i2v\":\n cond_images = load_image_batch(\n cond_inputs_rank[idx_s:idx_e], (args.height, args.width)\n )\n cond_images = cond_images.to(model.device)\n img_emb = model.get_image_embeds(cond_images)\n imtext_cond = torch.cat([text_emb, img_emb], dim=1)\n cond = {\"c_crossattn\": [imtext_cond], \"fps\": fps}\n else:\n raise NotImplementedError\n\n ## inference\n batch_samples = batch_ddim_sampling_freenoise(\n model,\n cond,\n noise_shape,\n args.n_samples,\n args.ddim_steps,\n args.ddim_eta,\n args.unconditional_guidance_scale,\n args=args,\n x_T_total=x_T_total,\n )\n\n out_path = \"/tmp/output.mp4\"\n vid_tensor = batch_samples[0]\n video = vid_tensor.detach().cpu()\n video = torch.clamp(video.float(), -1.0, 1.0)\n video = video.permute(2, 0, 1, 3, 4) # t,n,c,h,w\n\n frame_grids = [\n torchvision.utils.make_grid(framesheet, nrow=int(args.n_samples))\n for framesheet in video\n ] # [3, 1*h, n*w]\n grid = torch.stack(frame_grids, dim=0) # stack in temporal dim [t, 3, n*h, w]\n grid = (grid + 1.0) / 2.0\n grid = (grid * 255).to(torch.uint8).permute(0, 2, 3, 1)\n torchvision.io.write_video(\n out_path,\n grid,\n fps=args.savefps,\n video_codec=\"h264\",\n options={\"crf\": \"10\"},\n )\n return Path(out_path)\n" }, { "path": "prompts/mp_prompts.txt", "content": "A bigfoot giving a thumbs up in the snow, towards the camera;A bigfoot waving hands in the snow, towards the camera\nA woman with red dress waving hands on the beach in sunset;A woman with red dress dancing on the beach in sunset" }, { "path": "prompts/single_prompts.txt", "content": "A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect\nA corgi is swimming" }, { "path": "requirements.txt", "content": "decord==0.6.0\neinops==0.3.0\nimageio==2.9.0\nnumpy==1.24.2\nomegaconf==2.1.1\nopencv_python\npandas==2.0.0\nPillow==9.5.0\npytorch_lightning==1.8.3\nPyYAML==6.0\nsetuptools==65.6.3\ntorch==2.0.0\ntorchvision\ntqdm==4.65.0\ntransformers==4.25.1\nmoviepy\nav\nxformers\ngradio\ntimm\nscikit-learn \nopen_clip_torch\nkornia" }, { "path": "scripts/evaluation/ddp_wrapper.py", "content": "import datetime\r\nimport argparse, importlib\r\nfrom pytorch_lightning import seed_everything\r\n\r\nimport torch\r\nimport torch.distributed as dist\r\n\r\ndef setup_dist(local_rank):\r\n if dist.is_initialized():\r\n return\r\n torch.cuda.set_device(local_rank)\r\n torch.distributed.init_process_group('nccl', init_method='env://')\r\n\r\n\r\ndef get_dist_info():\r\n if dist.is_available():\r\n initialized = dist.is_initialized()\r\n else:\r\n initialized = False\r\n if initialized:\r\n rank = dist.get_rank()\r\n world_size = dist.get_world_size()\r\n else:\r\n rank = 0\r\n world_size = 1\r\n return rank, world_size\r\n\r\n\r\nif __name__ == '__main__':\r\n now = datetime.datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"--module\", type=str, help=\"module name\", default=\"inference\")\r\n parser.add_argument(\"--local_rank\", type=int, nargs=\"?\", help=\"for ddp\", default=0)\r\n args, unknown = parser.parse_known_args()\r\n inference_api = importlib.import_module(args.module, package=None)\r\n\r\n inference_parser = inference_api.get_parser()\r\n inference_args, unknown = inference_parser.parse_known_args()\r\n\r\n seed_everything(inference_args.seed)\r\n setup_dist(args.local_rank)\r\n torch.backends.cudnn.benchmark = True\r\n rank, gpu_num = get_dist_info()\r\n\r\n print(\"@CoLVDM Inference [rank%d]: %s\"%(rank, now))\r\n inference_api.run_inference(inference_args, gpu_num, rank)" }, { "path": "scripts/evaluation/funcs.py", "content": "import os, sys, glob\r\nimport numpy as np\r\nfrom collections import OrderedDict\r\nfrom decord import VideoReader, cpu\r\nimport cv2\r\n\r\nimport torch\r\nimport torchvision\r\nsys.path.insert(1, os.path.join(sys.path[0], '..', '..'))\r\nfrom lvdm.models.samplers.ddim import DDIMSampler\r\nfrom lvdm.models.samplers.ddim_mp import DDIMSampler as DDIMSampler_mp\r\n\r\ndef get_views(video_length, window_size=16, stride=4):\r\n num_blocks_time = (video_length - window_size) // stride + 1\r\n views = []\r\n for i in range(num_blocks_time):\r\n t_start = int(i * stride)\r\n t_end = t_start + window_size\r\n views.append((t_start,t_end))\r\n return views\r\n\r\ndef batch_ddim_sampling(model, cond, noise_shape, n_samples=1, ddim_steps=50, ddim_eta=1.0,\\\r\n cfg_scale=1.0, temporal_cfg_scale=None, **kwargs):\r\n ddim_sampler = DDIMSampler(model)\r\n uncond_type = model.uncond_type\r\n batch_size = noise_shape[0]\r\n\r\n ## construct unconditional guidance\r\n if cfg_scale != 1.0:\r\n if uncond_type == \"empty_seq\":\r\n prompts = batch_size * [\"\"]\r\n #prompts = N * T * [\"\"] ## if is_imgbatch=True\r\n uc_emb = model.get_learned_conditioning(prompts)\r\n elif uncond_type == \"zero_embed\":\r\n c_emb = cond[\"c_crossattn\"][0] if isinstance(cond, dict) else cond\r\n uc_emb = torch.zeros_like(c_emb)\r\n \r\n ## process image embedding token\r\n if hasattr(model, 'embedder'):\r\n uc_img = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n ## img: b c h w >> b l c\r\n uc_img = model.get_image_embeds(uc_img)\r\n uc_emb = torch.cat([uc_emb, uc_img], dim=1)\r\n \r\n if isinstance(cond, dict):\r\n uc = {key:cond[key] for key in cond.keys()}\r\n uc.update({'c_crossattn': [uc_emb]})\r\n else:\r\n uc = uc_emb\r\n else:\r\n uc = None\r\n \r\n x_T = None\r\n batch_variants = []\r\n #batch_variants1, batch_variants2 = [], []\r\n for _ in range(n_samples):\r\n if ddim_sampler is not None:\r\n kwargs.update({\"clean_cond\": True})\r\n samples, _ = ddim_sampler.sample(S=ddim_steps,\r\n conditioning=cond,\r\n batch_size=noise_shape[0],\r\n shape=noise_shape[1:],\r\n verbose=False,\r\n unconditional_guidance_scale=cfg_scale,\r\n unconditional_conditioning=uc,\r\n eta=ddim_eta,\r\n temporal_length=noise_shape[2],\r\n conditional_guidance_scale_temporal=temporal_cfg_scale,\r\n x_T=x_T,\r\n **kwargs\r\n )\r\n ## reconstruct from latent to pixel space\r\n batch_images = model.decode_first_stage_2DAE(samples)\r\n batch_variants.append(batch_images)\r\n ## batch, , c, t, h, w\r\n batch_variants = torch.stack(batch_variants, dim=1)\r\n return batch_variants\r\n\r\ndef batch_ddim_sampling_freenoise(model, cond, noise_shape, n_samples=1, ddim_steps=50, ddim_eta=1.0,\\\r\n cfg_scale=1.0, temporal_cfg_scale=None, args=None, x_T_total=None, **kwargs):\r\n ddim_sampler = DDIMSampler(model)\r\n uncond_type = model.uncond_type\r\n batch_size = noise_shape[0]\r\n\r\n ## construct unconditional guidance\r\n if cfg_scale != 1.0:\r\n if uncond_type == \"empty_seq\":\r\n prompts = batch_size * [\"\"]\r\n #prompts = N * T * [\"\"] ## if is_imgbatch=True\r\n uc_emb = model.get_learned_conditioning(prompts)\r\n elif uncond_type == \"zero_embed\":\r\n c_emb = cond[\"c_crossattn\"][0] if isinstance(cond, dict) else cond\r\n uc_emb = torch.zeros_like(c_emb)\r\n \r\n ## process image embedding token\r\n if hasattr(model, 'embedder'):\r\n uc_img = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n ## img: b c h w >> b l c\r\n uc_img = model.get_image_embeds(uc_img)\r\n uc_emb = torch.cat([uc_emb, uc_img], dim=1)\r\n \r\n if isinstance(cond, dict):\r\n uc = {key:cond[key] for key in cond.keys()}\r\n uc.update({'c_crossattn': [uc_emb]})\r\n else:\r\n uc = uc_emb\r\n else:\r\n uc = None\r\n\r\n views = get_views(args.frames, args.window_size, args.window_stride)\r\n\r\n batch_variants = []\r\n #batch_variants1, batch_variants2 = [], []\r\n for _ in range(n_samples):\r\n x_T = x_T_total[_]\r\n if ddim_sampler is not None:\r\n kwargs.update({\"clean_cond\": True})\r\n samples, _ = ddim_sampler.sample(S=ddim_steps,\r\n conditioning=cond,\r\n batch_size=noise_shape[0],\r\n shape=noise_shape[1:],\r\n verbose=False,\r\n unconditional_guidance_scale=cfg_scale,\r\n unconditional_conditioning=uc,\r\n eta=ddim_eta,\r\n temporal_length=noise_shape[2],\r\n conditional_guidance_scale_temporal=temporal_cfg_scale,\r\n x_T=x_T,\r\n context_next=views,\r\n **kwargs\r\n )\r\n ## reconstruct from latent to pixel space\r\n batch_images = model.decode_first_stage_2DAE(samples)\r\n batch_variants.append(batch_images)\r\n ## batch, , c, t, h, w\r\n batch_variants = torch.stack(batch_variants, dim=1)\r\n return batch_variants\r\n\r\ndef batch_ddim_sampling_freenoise_mp(model, cond, noise_shape, n_samples=1, ddim_steps=50, ddim_eta=1.0,\\\r\n cfg_scale=1.0, temporal_cfg_scale=None, args=None, x_T_total=None, **kwargs):\r\n ddim_sampler = DDIMSampler_mp(model)\r\n uncond_type = model.uncond_type\r\n batch_size = noise_shape[0]\r\n\r\n ## construct unconditional guidance\r\n if cfg_scale != 1.0:\r\n if uncond_type == \"empty_seq\":\r\n prompts = batch_size * [\"\"]\r\n #prompts = N * T * [\"\"] ## if is_imgbatch=True\r\n uc_emb = model.get_learned_conditioning(prompts)\r\n elif uncond_type == \"zero_embed\":\r\n c_emb = cond[\"c_crossattn\"][0] if isinstance(cond, dict) else cond\r\n uc_emb = torch.zeros_like(c_emb)\r\n \r\n ## process image embedding token\r\n if hasattr(model, 'embedder'):\r\n uc_img = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n ## img: b c h w >> b l c\r\n uc_img = model.get_image_embeds(uc_img)\r\n uc_emb = torch.cat([uc_emb, uc_img], dim=1)\r\n \r\n if isinstance(cond, dict):\r\n uc = {key:cond[key] for key in cond.keys()}\r\n uc.update({'c_crossattn': [uc_emb]})\r\n else:\r\n uc = uc_emb\r\n else:\r\n uc = None\r\n\r\n views = get_views(args.frames, args.window_size, args.window_stride)\r\n\r\n conditioning = cond['c_crossattn'][0]\r\n len1 = int(args.frames * 3 // 8)\r\n len2 = args.frames - len1 * 2\r\n cond_diff1 = (conditioning[[1]] - conditioning[[0]]) / (len2 - 1)\r\n cond_list1 = []\r\n for i in range(len2):\r\n cond_list1.append((conditioning[[0]] + cond_diff1 * i).unsqueeze(0))\r\n\r\n cond1 = torch.cat([conditioning[[0]].unsqueeze(0).repeat(1, len1, 1, 1), torch.cat(cond_list1, 1), conditioning[[1]].unsqueeze(0).repeat(1, len1, 1, 1)], 1)\r\n cond2 = torch.cat([conditioning[[1]].unsqueeze(0).repeat(1, args.frames, 1, 1)], 1)\r\n \r\n cond_all = torch.cat([cond1, cond2], 0)\r\n\r\n cond['c_crossattn'] = [cond_all]\r\n\r\n batch_variants = []\r\n #batch_variants1, batch_variants2 = [], []\r\n for _ in range(n_samples):\r\n x_T = x_T_total[_]\r\n if ddim_sampler is not None:\r\n kwargs.update({\"clean_cond\": True})\r\n samples, _ = ddim_sampler.sample(S=ddim_steps,\r\n conditioning=cond,\r\n batch_size=noise_shape[0],\r\n shape=noise_shape[1:],\r\n verbose=False,\r\n unconditional_guidance_scale=cfg_scale,\r\n unconditional_conditioning=uc,\r\n eta=ddim_eta,\r\n temporal_length=noise_shape[2],\r\n conditional_guidance_scale_temporal=temporal_cfg_scale,\r\n x_T=x_T,\r\n context_next=views,\r\n **kwargs\r\n )\r\n ## reconstruct from latent to pixel space\r\n batch_images = model.decode_first_stage_2DAE(samples)\r\n batch_variants.append(batch_images)\r\n ## batch, , c, t, h, w\r\n batch_variants = torch.stack(batch_variants, dim=1)\r\n return batch_variants\r\n\r\ndef get_filelist(data_dir, ext='*'):\r\n file_list = glob.glob(os.path.join(data_dir, '*.%s'%ext))\r\n file_list.sort()\r\n return file_list\r\n\r\ndef get_dirlist(path):\r\n list = []\r\n if (os.path.exists(path)):\r\n files = os.listdir(path)\r\n for file in files:\r\n m = os.path.join(path,file)\r\n if (os.path.isdir(m)):\r\n list.append(m)\r\n list.sort()\r\n return list\r\n\r\n\r\ndef load_model_checkpoint(model, ckpt):\r\n def load_checkpoint(model, ckpt, full_strict):\r\n state_dict = torch.load(ckpt, map_location=\"cpu\")\r\n try:\r\n ## deepspeed\r\n new_pl_sd = OrderedDict()\r\n for key in state_dict['module'].keys():\r\n new_pl_sd[key[16:]]=state_dict['module'][key]\r\n model.load_state_dict(new_pl_sd, strict=full_strict)\r\n except:\r\n if \"state_dict\" in list(state_dict.keys()):\r\n state_dict = state_dict[\"state_dict\"]\r\n model.load_state_dict(state_dict, strict=full_strict)\r\n return model\r\n load_checkpoint(model, ckpt, full_strict=True)\r\n print('>>> model checkpoint loaded.')\r\n return model\r\n\r\n\r\ndef load_prompts(prompt_file):\r\n f = open(prompt_file, 'r')\r\n prompt_list = []\r\n for idx, line in enumerate(f.readlines()):\r\n l = line.strip()\r\n if len(l) != 0:\r\n prompt_list.append(l)\r\n f.close()\r\n return prompt_list\r\n\r\ndef load_prompts_mp(prompt_file):\r\n f = open(prompt_file, 'r')\r\n prompt_list = []\r\n for idx, line in enumerate(f.readlines()):\r\n l = []\r\n line = line.strip()\r\n prompts = line.split(';')\r\n for prompt in prompts:\r\n prompt = prompt.strip()\r\n if len(prompt) != 0:\r\n l.append(prompt)\r\n if len(l) != 0:\r\n prompt_list.append(l)\r\n f.close()\r\n print(prompt_list)\r\n return prompt_list\r\n\r\ndef load_video_batch(filepath_list, frame_stride, video_size=(256,256), video_frames=16):\r\n '''\r\n Notice about some special cases:\r\n 1. video_frames=-1 means to take all the frames (with fs=1)\r\n 2. when the total video frames is less than required, padding strategy will be used (repreated last frame)\r\n '''\r\n fps_list = []\r\n batch_tensor = []\r\n assert frame_stride > 0, \"valid frame stride should be a positive integer!\"\r\n for filepath in filepath_list:\r\n padding_num = 0\r\n vidreader = VideoReader(filepath, ctx=cpu(0), width=video_size[1], height=video_size[0])\r\n fps = vidreader.get_avg_fps()\r\n total_frames = len(vidreader)\r\n max_valid_frames = (total_frames-1) // frame_stride + 1\r\n if video_frames < 0:\r\n ## all frames are collected: fs=1 is a must\r\n required_frames = total_frames\r\n frame_stride = 1\r\n else:\r\n required_frames = video_frames\r\n query_frames = min(required_frames, max_valid_frames)\r\n frame_indices = [frame_stride*i for i in range(query_frames)]\r\n\r\n ## [t,h,w,c] -> [c,t,h,w]\r\n frames = vidreader.get_batch(frame_indices)\r\n frame_tensor = torch.tensor(frames.asnumpy()).permute(3, 0, 1, 2).float()\r\n frame_tensor = (frame_tensor / 255. - 0.5) * 2\r\n if max_valid_frames < required_frames:\r\n padding_num = required_frames - max_valid_frames\r\n frame_tensor = torch.cat([frame_tensor, *([frame_tensor[:,-1:,:,:]]*padding_num)], dim=1)\r\n print(f'{os.path.split(filepath)[1]} is not long enough: {padding_num} frames padded.')\r\n batch_tensor.append(frame_tensor)\r\n sample_fps = int(fps/frame_stride)\r\n fps_list.append(sample_fps)\r\n \r\n return torch.stack(batch_tensor, dim=0)\r\n\r\nfrom PIL import Image\r\ndef load_image_batch(filepath_list, image_size=(256,256)):\r\n batch_tensor = []\r\n for filepath in filepath_list:\r\n _, filename = os.path.split(filepath)\r\n _, ext = os.path.splitext(filename)\r\n if ext == '.mp4':\r\n vidreader = VideoReader(filepath, ctx=cpu(0), width=image_size[1], height=image_size[0])\r\n frame = vidreader.get_batch([0])\r\n img_tensor = torch.tensor(frame.asnumpy()).squeeze(0).permute(2, 0, 1).float()\r\n elif ext == '.png' or ext == '.jpg':\r\n img = Image.open(filepath).convert(\"RGB\")\r\n rgb_img = np.array(img, np.float32)\r\n #bgr_img = cv2.imread(filepath, cv2.IMREAD_COLOR)\r\n #bgr_img = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB)\r\n rgb_img = cv2.resize(rgb_img, (image_size[1],image_size[0]), interpolation=cv2.INTER_LINEAR)\r\n img_tensor = torch.from_numpy(rgb_img).permute(2, 0, 1).float()\r\n else:\r\n print(f'ERROR: <{ext}> image loading only support format: [mp4], [png], [jpg]')\r\n raise NotImplementedError\r\n img_tensor = (img_tensor / 255. - 0.5) * 2\r\n batch_tensor.append(img_tensor)\r\n return torch.stack(batch_tensor, dim=0)\r\n\r\n\r\ndef save_videos(batch_tensors, savedir, filenames, fps=10):\r\n # b,samples,c,t,h,w\r\n n_samples = batch_tensors.shape[1]\r\n for idx, vid_tensor in enumerate(batch_tensors):\r\n video = vid_tensor.detach().cpu()\r\n video = torch.clamp(video.float(), -1., 1.)\r\n video = video.permute(2, 0, 1, 3, 4) # t,n,c,h,w\r\n frame_grids = [torchvision.utils.make_grid(framesheet, nrow=int(n_samples)) for framesheet in video] #[3, 1*h, n*w]\r\n grid = torch.stack(frame_grids, dim=0) # stack in temporal dim [t, 3, n*h, w]\r\n grid = (grid + 1.0) / 2.0\r\n grid = (grid * 255).to(torch.uint8).permute(0, 2, 3, 1)\r\n savepath = os.path.join(savedir, f\"{filenames[idx]}.mp4\")\r\n torchvision.io.write_video(savepath, grid, fps=fps, video_codec='h264', options={'crf': '10'})\r\n\r\n" }, { "path": "scripts/evaluation/inference.py", "content": "import argparse, os, sys, glob, yaml, math, random\r\nimport datetime, time\r\nimport numpy as np\r\nfrom omegaconf import OmegaConf\r\nfrom collections import OrderedDict\r\nfrom tqdm import trange, tqdm\r\nfrom einops import repeat\r\nfrom einops import rearrange, repeat\r\nfrom functools import partial\r\nimport torch\r\nfrom pytorch_lightning import seed_everything\r\n\r\nfrom funcs import load_model_checkpoint, load_prompts, load_image_batch, get_filelist, save_videos\r\nfrom funcs import batch_ddim_sampling\r\nfrom utils.utils import instantiate_from_config\r\n\r\n\r\ndef get_parser():\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"--seed\", type=int, default=20230211, help=\"seed for seed_everything\")\r\n parser.add_argument(\"--mode\", default=\"base\", type=str, help=\"which kind of inference mode: {'base', 'i2v'}\")\r\n parser.add_argument(\"--ckpt_path\", type=str, default=None, help=\"checkpoint path\")\r\n parser.add_argument(\"--config\", type=str, help=\"config (yaml) path\")\r\n parser.add_argument(\"--prompt_file\", type=str, default=None, help=\"a text file containing many prompts\")\r\n parser.add_argument(\"--savedir\", type=str, default=None, help=\"results saving path\")\r\n parser.add_argument(\"--savefps\", type=str, default=10, help=\"video fps to generate\")\r\n parser.add_argument(\"--n_samples\", type=int, default=1, help=\"num of samples per prompt\",)\r\n parser.add_argument(\"--ddim_steps\", type=int, default=50, help=\"steps of ddim if positive, otherwise use DDPM\",)\r\n parser.add_argument(\"--ddim_eta\", type=float, default=1.0, help=\"eta for ddim sampling (0.0 yields deterministic sampling)\",)\r\n parser.add_argument(\"--bs\", type=int, default=1, help=\"batch size for inference\")\r\n parser.add_argument(\"--height\", type=int, default=512, help=\"image height, in pixel space\")\r\n parser.add_argument(\"--width\", type=int, default=512, help=\"image width, in pixel space\")\r\n parser.add_argument(\"--frames\", type=int, default=-1, help=\"frames num to inference\")\r\n parser.add_argument(\"--fps\", type=int, default=24)\r\n parser.add_argument(\"--unconditional_guidance_scale\", type=float, default=1.0, help=\"prompt classifier-free guidance\")\r\n parser.add_argument(\"--unconditional_guidance_scale_temporal\", type=float, default=None, help=\"temporal consistency guidance\")\r\n ## for conditional i2v only\r\n parser.add_argument(\"--cond_input\", type=str, default=None, help=\"data dir of conditional input\")\r\n return parser\r\n\r\n\r\ndef run_inference(args, gpu_num, gpu_no, **kwargs):\r\n ## step 1: model config\r\n ## -----------------------------------------------------------------\r\n config = OmegaConf.load(args.config)\r\n #data_config = config.pop(\"data\", OmegaConf.create())\r\n model_config = config.pop(\"model\", OmegaConf.create())\r\n model = instantiate_from_config(model_config)\r\n model = model.cuda(gpu_no)\r\n assert os.path.exists(args.ckpt_path), f\"Error: checkpoint [{args.ckpt_path}] Not Found!\"\r\n model = load_model_checkpoint(model, args.ckpt_path)\r\n model.eval()\r\n\r\n ## sample shape\r\n assert (args.height % 16 == 0) and (args.width % 16 == 0), \"Error: image size [h,w] should be multiples of 16!\"\r\n ## latent noise shape\r\n h, w = args.height // 8, args.width // 8\r\n frames = model.temporal_length if args.frames < 0 else args.frames\r\n channels = model.channels\r\n \r\n ## saving folders\r\n os.makedirs(args.savedir, exist_ok=True)\r\n\r\n ## step 2: load data\r\n ## -----------------------------------------------------------------\r\n assert os.path.exists(args.prompt_file), \"Error: prompt file NOT Found!\"\r\n prompt_list = load_prompts(args.prompt_file)\r\n num_samples = len(prompt_list)\r\n filename_list = [f\"{id+1:04d}\" for id in range(num_samples)]\r\n\r\n samples_split = num_samples // gpu_num\r\n residual_tail = num_samples % gpu_num\r\n print(f'[rank:{gpu_no}] {samples_split}/{num_samples} samples loaded.')\r\n indices = list(range(samples_split*gpu_no, samples_split*(gpu_no+1)))\r\n if gpu_no == 0 and residual_tail != 0:\r\n indices = indices + list(range(num_samples-residual_tail, num_samples))\r\n prompt_list_rank = [prompt_list[i] for i in indices]\r\n\r\n ## conditional input\r\n if args.mode == \"i2v\":\r\n ## each video or frames dir per prompt\r\n cond_inputs = get_filelist(args.cond_input, ext='[mpj][pn][4gj]') # '[mpj][pn][4gj]'\r\n assert len(cond_inputs) == num_samples, f\"Error: conditional input ({len(cond_inputs)}) NOT match prompt ({num_samples})!\"\r\n filename_list = [f\"{os.path.split(cond_inputs[id])[-1][:-4]}\" for id in range(num_samples)]\r\n cond_inputs_rank = [cond_inputs[i] for i in indices]\r\n\r\n filename_list_rank = [filename_list[i] for i in indices]\r\n\r\n ## step 3: run over samples\r\n ## -----------------------------------------------------------------\r\n start = time.time()\r\n n_rounds = len(prompt_list_rank) // args.bs\r\n n_rounds = n_rounds+1 if len(prompt_list_rank) % args.bs != 0 else n_rounds\r\n for idx in range(0, n_rounds):\r\n print(f'[rank:{gpu_no}] batch-{idx+1} ({args.bs})x{args.n_samples} ...')\r\n idx_s = idx*args.bs\r\n idx_e = min(idx_s+args.bs, len(prompt_list_rank))\r\n batch_size = idx_e - idx_s\r\n filenames = filename_list_rank[idx_s:idx_e]\r\n noise_shape = [batch_size, channels, frames, h, w]\r\n fps = torch.tensor([args.fps]*batch_size).to(model.device).long()\r\n\r\n prompts = prompt_list_rank[idx_s:idx_e]\r\n if isinstance(prompts, str):\r\n prompts = [prompts]\r\n #prompts = batch_size * [\"\"]\r\n text_emb = model.get_learned_conditioning(prompts)\r\n\r\n if args.mode == 'base':\r\n cond = {\"c_crossattn\": [text_emb], \"fps\": fps}\r\n elif args.mode == 'i2v':\r\n #cond_images = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n cond_images = load_image_batch(cond_inputs_rank[idx_s:idx_e], (args.height, args.width))\r\n cond_images = cond_images.to(model.device)\r\n img_emb = model.get_image_embeds(cond_images)\r\n imtext_cond = torch.cat([text_emb, img_emb], dim=1)\r\n cond = {\"c_crossattn\": [imtext_cond], \"fps\": fps}\r\n else:\r\n raise NotImplementedError\r\n\r\n ## inference\r\n batch_samples = batch_ddim_sampling(model, cond, noise_shape, args.n_samples, \\\r\n args.ddim_steps, args.ddim_eta, args.unconditional_guidance_scale, **kwargs)\r\n ## b,samples,c,t,h,w\r\n save_videos(batch_samples, args.savedir, filenames, fps=args.savefps)\r\n\r\n print(f\"Saved in {args.savedir}. Time used: {(time.time() - start):.2f} seconds\")\r\n\r\n\r\nif __name__ == '__main__':\r\n now = datetime.datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\r\n print(\"@CoLVDM Inference: %s\"%now)\r\n parser = get_parser()\r\n args = parser.parse_args()\r\n seed_everything(args.seed)\r\n rank, gpu_num = 0, 1\r\n run_inference(args, gpu_num, rank)" }, { "path": "scripts/evaluation/inference_freenoise.py", "content": "import argparse, os, sys, glob, yaml, math, random\r\nimport datetime, time\r\nimport numpy as np\r\nfrom omegaconf import OmegaConf\r\nfrom collections import OrderedDict\r\nfrom tqdm import trange, tqdm\r\nfrom einops import repeat\r\nfrom einops import rearrange, repeat\r\nfrom functools import partial\r\nimport torch\r\nfrom pytorch_lightning import seed_everything\r\n\r\nfrom funcs import load_model_checkpoint, load_prompts, load_image_batch, get_filelist, save_videos\r\nfrom funcs import batch_ddim_sampling_freenoise\r\nfrom utils.utils import instantiate_from_config\r\n\r\n\r\ndef get_parser():\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"--seed\", type=int, default=20230211, help=\"seed for seed_everything\")\r\n parser.add_argument(\"--mode\", default=\"base\", type=str, help=\"which kind of inference mode: {'base', 'i2v'}\")\r\n parser.add_argument(\"--ckpt_path\", type=str, default=None, help=\"checkpoint path\")\r\n parser.add_argument(\"--config\", type=str, help=\"config (yaml) path\")\r\n parser.add_argument(\"--prompt_file\", type=str, default=None, help=\"a text file containing many prompts\")\r\n parser.add_argument(\"--savedir\", type=str, default=None, help=\"results saving path\")\r\n parser.add_argument(\"--savefps\", type=str, default=10, help=\"video fps to generate\")\r\n parser.add_argument(\"--n_samples\", type=int, default=1, help=\"num of samples per prompt\",)\r\n parser.add_argument(\"--ddim_steps\", type=int, default=50, help=\"steps of ddim if positive, otherwise use DDPM\",)\r\n parser.add_argument(\"--ddim_eta\", type=float, default=1.0, help=\"eta for ddim sampling (0.0 yields deterministic sampling)\",)\r\n parser.add_argument(\"--bs\", type=int, default=1, help=\"batch size for inference\")\r\n parser.add_argument(\"--height\", type=int, default=512, help=\"image height, in pixel space\")\r\n parser.add_argument(\"--width\", type=int, default=512, help=\"image width, in pixel space\")\r\n parser.add_argument(\"--frames\", type=int, default=-1, help=\"frames num to inference\")\r\n parser.add_argument(\"--fps\", type=int, default=24)\r\n parser.add_argument(\"--unconditional_guidance_scale\", type=float, default=1.0, help=\"prompt classifier-free guidance\")\r\n parser.add_argument(\"--unconditional_guidance_scale_temporal\", type=float, default=None, help=\"temporal consistency guidance\")\r\n ## for conditional i2v only\r\n parser.add_argument(\"--cond_input\", type=str, default=None, help=\"data dir of conditional input\")\r\n ## for freenoise only\r\n parser.add_argument(\"--window_size\", type=int, default=16, help=\"window_size\")\r\n parser.add_argument(\"--window_stride\", type=int, default=4, help=\"window_stride\")\r\n return parser\r\n\r\n\r\ndef run_inference(args, gpu_num, gpu_no, **kwargs):\r\n ## step 1: model config\r\n ## -----------------------------------------------------------------\r\n config = OmegaConf.load(args.config)\r\n #data_config = config.pop(\"data\", OmegaConf.create())\r\n model_config = config.pop(\"model\", OmegaConf.create())\r\n model = instantiate_from_config(model_config)\r\n model = model.cuda(gpu_no)\r\n assert os.path.exists(args.ckpt_path), f\"Error: checkpoint [{args.ckpt_path}] Not Found!\"\r\n model = load_model_checkpoint(model, args.ckpt_path)\r\n model.eval()\r\n\r\n ## sample shape\r\n assert (args.height % 16 == 0) and (args.width % 16 == 0), \"Error: image size [h,w] should be multiples of 16!\"\r\n ## latent noise shape\r\n h, w = args.height // 8, args.width // 8\r\n frames = model.temporal_length if args.frames < 0 else args.frames\r\n channels = model.channels\r\n \r\n ## saving folders\r\n os.makedirs(args.savedir, exist_ok=True)\r\n\r\n ## step 2: load data\r\n ## -----------------------------------------------------------------\r\n assert os.path.exists(args.prompt_file), \"Error: prompt file NOT Found!\"\r\n prompt_list = load_prompts(args.prompt_file)\r\n num_samples = len(prompt_list)\r\n filename_list = [f\"{id+1:04d}\" for id in range(num_samples)]\r\n\r\n samples_split = num_samples // gpu_num\r\n residual_tail = num_samples % gpu_num\r\n print(f'[rank:{gpu_no}] {samples_split}/{num_samples} samples loaded.')\r\n indices = list(range(samples_split*gpu_no, samples_split*(gpu_no+1)))\r\n if gpu_no == 0 and residual_tail != 0:\r\n indices = indices + list(range(num_samples-residual_tail, num_samples))\r\n prompt_list_rank = [prompt_list[i] for i in indices]\r\n\r\n ## conditional input\r\n if args.mode == \"i2v\":\r\n ## each video or frames dir per prompt\r\n cond_inputs = get_filelist(args.cond_input, ext='[mpj][pn][4gj]') # '[mpj][pn][4gj]'\r\n assert len(cond_inputs) == num_samples, f\"Error: conditional input ({len(cond_inputs)}) NOT match prompt ({num_samples})!\"\r\n filename_list = [f\"{os.path.split(cond_inputs[id])[-1][:-4]}\" for id in range(num_samples)]\r\n cond_inputs_rank = [cond_inputs[i] for i in indices]\r\n\r\n filename_list_rank = [filename_list[i] for i in indices]\r\n\r\n ## step 3: run over samples\r\n ## -----------------------------------------------------------------\r\n start = time.time()\r\n n_rounds = len(prompt_list_rank) // args.bs\r\n n_rounds = n_rounds+1 if len(prompt_list_rank) % args.bs != 0 else n_rounds\r\n\r\n x_T_total = torch.randn([args.n_samples, 1, channels, frames, h, w], device=model.device).repeat(1, args.bs, 1, 1, 1, 1)\r\n for frame_index in range(args.window_size, args.frames, args.window_stride):\r\n list_index = list(range(frame_index-args.window_size, frame_index+args.window_stride-args.window_size))\r\n random.shuffle(list_index)\r\n x_T_total[:, :, :, frame_index:frame_index+args.window_stride] = x_T_total[:, :, :, list_index]\r\n\r\n for idx in range(0, n_rounds):\r\n print(f'[rank:{gpu_no}] batch-{idx+1} ({args.bs})x{args.n_samples} ...')\r\n idx_s = idx*args.bs\r\n idx_e = min(idx_s+args.bs, len(prompt_list_rank))\r\n batch_size = idx_e - idx_s\r\n filenames = filename_list_rank[idx_s:idx_e]\r\n noise_shape = [batch_size, channels, frames, h, w]\r\n fps = torch.tensor([args.fps]*batch_size).to(model.device).long()\r\n\r\n prompts = prompt_list_rank[idx_s:idx_e]\r\n if isinstance(prompts, str):\r\n prompts = [prompts]\r\n #prompts = batch_size * [\"\"]\r\n text_emb = model.get_learned_conditioning(prompts)\r\n\r\n if args.mode == 'base':\r\n cond = {\"c_crossattn\": [text_emb], \"fps\": fps}\r\n elif args.mode == 'i2v':\r\n #cond_images = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n cond_images = load_image_batch(cond_inputs_rank[idx_s:idx_e], (args.height, args.width))\r\n cond_images = cond_images.to(model.device)\r\n img_emb = model.get_image_embeds(cond_images)\r\n imtext_cond = torch.cat([text_emb, img_emb], dim=1)\r\n cond = {\"c_crossattn\": [imtext_cond], \"fps\": fps}\r\n else:\r\n raise NotImplementedError\r\n\r\n ## inference\r\n batch_samples = batch_ddim_sampling_freenoise(model, cond, noise_shape, args.n_samples, \\\r\n args.ddim_steps, args.ddim_eta, args.unconditional_guidance_scale, args=args, x_T_total=x_T_total, **kwargs)\r\n ## b,samples,c,t,h,w\r\n save_videos(batch_samples, args.savedir, filenames, fps=args.savefps)\r\n\r\n print(f\"Saved in {args.savedir}. Time used: {(time.time() - start):.2f} seconds\")\r\n\r\n\r\nif __name__ == '__main__':\r\n now = datetime.datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\r\n print(\"@CoLVDM Inference: %s\"%now)\r\n parser = get_parser()\r\n args = parser.parse_args()\r\n seed_everything(args.seed)\r\n rank, gpu_num = 0, 1\r\n run_inference(args, gpu_num, rank)" }, { "path": "scripts/evaluation/inference_freenoise_mp.py", "content": "import argparse, os, sys, glob, yaml, math, random\r\nimport datetime, time\r\nimport numpy as np\r\nfrom omegaconf import OmegaConf\r\nfrom collections import OrderedDict\r\nfrom tqdm import trange, tqdm\r\nfrom einops import repeat\r\nfrom einops import rearrange, repeat\r\nfrom functools import partial\r\nimport torch\r\nfrom pytorch_lightning import seed_everything\r\n\r\nfrom funcs import load_model_checkpoint, load_prompts_mp, load_image_batch, get_filelist, save_videos\r\nfrom funcs import batch_ddim_sampling_freenoise_mp\r\nfrom utils.utils import instantiate_from_config\r\n\r\n\r\ndef get_parser():\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"--seed\", type=int, default=20230211, help=\"seed for seed_everything\")\r\n parser.add_argument(\"--mode\", default=\"base\", type=str, help=\"which kind of inference mode: {'base', 'i2v'}\")\r\n parser.add_argument(\"--ckpt_path\", type=str, default=None, help=\"checkpoint path\")\r\n parser.add_argument(\"--config\", type=str, help=\"config (yaml) path\")\r\n parser.add_argument(\"--prompt_file\", type=str, default=None, help=\"a text file containing many prompts\")\r\n parser.add_argument(\"--savedir\", type=str, default=None, help=\"results saving path\")\r\n parser.add_argument(\"--savefps\", type=str, default=10, help=\"video fps to generate\")\r\n parser.add_argument(\"--n_samples\", type=int, default=1, help=\"num of samples per prompt\",)\r\n parser.add_argument(\"--ddim_steps\", type=int, default=50, help=\"steps of ddim if positive, otherwise use DDPM\",)\r\n parser.add_argument(\"--ddim_eta\", type=float, default=1.0, help=\"eta for ddim sampling (0.0 yields deterministic sampling)\",)\r\n parser.add_argument(\"--bs\", type=int, default=1, help=\"batch size for inference\")\r\n parser.add_argument(\"--height\", type=int, default=512, help=\"image height, in pixel space\")\r\n parser.add_argument(\"--width\", type=int, default=512, help=\"image width, in pixel space\")\r\n parser.add_argument(\"--frames\", type=int, default=-1, help=\"frames num to inference\")\r\n parser.add_argument(\"--fps\", type=int, default=24)\r\n parser.add_argument(\"--unconditional_guidance_scale\", type=float, default=1.0, help=\"prompt classifier-free guidance\")\r\n parser.add_argument(\"--unconditional_guidance_scale_temporal\", type=float, default=None, help=\"temporal consistency guidance\")\r\n ## for conditional i2v only\r\n parser.add_argument(\"--cond_input\", type=str, default=None, help=\"data dir of conditional input\")\r\n ## for freenoise only\r\n parser.add_argument(\"--window_size\", type=int, default=16, help=\"window_size\")\r\n parser.add_argument(\"--window_stride\", type=int, default=4, help=\"window_stride\")\r\n return parser\r\n\r\n\r\ndef run_inference(args, gpu_num, gpu_no, **kwargs):\r\n ## step 1: model config\r\n ## -----------------------------------------------------------------\r\n config = OmegaConf.load(args.config)\r\n #data_config = config.pop(\"data\", OmegaConf.create())\r\n model_config = config.pop(\"model\", OmegaConf.create())\r\n model = instantiate_from_config(model_config)\r\n model = model.cuda(gpu_no)\r\n assert os.path.exists(args.ckpt_path), f\"Error: checkpoint [{args.ckpt_path}] Not Found!\"\r\n model = load_model_checkpoint(model, args.ckpt_path)\r\n model.eval()\r\n\r\n ## sample shape\r\n assert (args.height % 16 == 0) and (args.width % 16 == 0), \"Error: image size [h,w] should be multiples of 16!\"\r\n ## latent noise shape\r\n h, w = args.height // 8, args.width // 8\r\n frames = model.temporal_length if args.frames < 0 else args.frames\r\n channels = model.channels\r\n \r\n ## saving folders\r\n os.makedirs(args.savedir, exist_ok=True)\r\n\r\n ## step 2: load data\r\n ## -----------------------------------------------------------------\r\n assert os.path.exists(args.prompt_file), \"Error: prompt file NOT Found!\"\r\n prompt_list = load_prompts_mp(args.prompt_file)\r\n num_samples = len(prompt_list)\r\n filename_list = [f\"{id+1:04d}\" for id in range(num_samples)]\r\n\r\n samples_split = num_samples // gpu_num\r\n residual_tail = num_samples % gpu_num\r\n print(f'[rank:{gpu_no}] {samples_split}/{num_samples} samples loaded.')\r\n indices = list(range(samples_split*gpu_no, samples_split*(gpu_no+1)))\r\n if gpu_no == 0 and residual_tail != 0:\r\n indices = indices + list(range(num_samples-residual_tail, num_samples))\r\n prompt_list_rank = [prompt_list[i] for i in indices]\r\n\r\n ## conditional input\r\n if args.mode == \"i2v\":\r\n ## each video or frames dir per prompt\r\n cond_inputs = get_filelist(args.cond_input, ext='[mpj][pn][4gj]') # '[mpj][pn][4gj]'\r\n assert len(cond_inputs) == num_samples, f\"Error: conditional input ({len(cond_inputs)}) NOT match prompt ({num_samples})!\"\r\n filename_list = [f\"{os.path.split(cond_inputs[id])[-1][:-4]}\" for id in range(num_samples)]\r\n cond_inputs_rank = [cond_inputs[i] for i in indices]\r\n\r\n filename_list_rank = [filename_list[i] for i in indices]\r\n\r\n ## step 3: run over samples\r\n ## -----------------------------------------------------------------\r\n start = time.time()\r\n n_rounds = len(prompt_list_rank) // args.bs\r\n n_rounds = n_rounds+1 if len(prompt_list_rank) % args.bs != 0 else n_rounds\r\n\r\n x_T_total = torch.randn([args.n_samples, 1, channels, frames, h, w], device=model.device).repeat(1, args.bs, 1, 1, 1, 1)\r\n for frame_index in range(args.window_size, args.frames, args.window_stride):\r\n list_index = list(range(frame_index-args.window_size, frame_index+args.window_stride-args.window_size))\r\n random.shuffle(list_index)\r\n x_T_total[:, :, :, frame_index:frame_index+args.window_stride] = x_T_total[:, :, :, list_index]\r\n\r\n for idx in range(0, n_rounds):\r\n print(f'[rank:{gpu_no}] batch-{idx+1} ({args.bs})x{args.n_samples} ...')\r\n idx_s = idx*args.bs\r\n idx_e = min(idx_s+args.bs, len(prompt_list_rank))\r\n batch_size = idx_e - idx_s\r\n filenames = filename_list_rank[idx_s:idx_e]\r\n noise_shape = [batch_size, channels, frames, h, w]\r\n fps = torch.tensor([args.fps]*batch_size).to(model.device).long()\r\n\r\n prompts_list = prompt_list_rank[idx_s:idx_e]\r\n if isinstance(prompts_list, str):\r\n prompts_list = [prompts_list]\r\n #prompts = batch_size * [\"\"]\r\n text_emb_list = []\r\n for prompts in prompts_list:\r\n # text_emb = model.get_learned_conditioning(prompts)\r\n # text_emb_list.append(text_emb)\r\n text_emb = [model.get_learned_conditioning(prompt) for prompt in prompts]\r\n text_emb = torch.cat(text_emb, 0)\r\n text_emb_list.append(text_emb)\r\n\r\n if args.mode == 'base':\r\n cond = {\"c_crossattn\": text_emb_list, \"fps\": fps}\r\n elif args.mode == 'i2v':\r\n #cond_images = torch.zeros(noise_shape[0],3,224,224).to(model.device)\r\n cond_images = load_image_batch(cond_inputs_rank[idx_s:idx_e], (args.height, args.width))\r\n cond_images = cond_images.to(model.device)\r\n img_emb = model.get_image_embeds(cond_images)\r\n imtext_cond = torch.cat([text_emb, img_emb], dim=1)\r\n cond = {\"c_crossattn\": [imtext_cond], \"fps\": fps}\r\n else:\r\n raise NotImplementedError\r\n\r\n ## inference\r\n batch_samples = batch_ddim_sampling_freenoise_mp(model, cond, noise_shape, args.n_samples, \\\r\n args.ddim_steps, args.ddim_eta, args.unconditional_guidance_scale, args=args, x_T_total=x_T_total, **kwargs)\r\n ## b,samples,c,t,h,w\r\n save_videos(batch_samples, args.savedir, filenames, fps=args.savefps)\r\n\r\n print(f\"Saved in {args.savedir}. Time used: {(time.time() - start):.2f} seconds\")\r\n\r\n\r\nif __name__ == '__main__':\r\n now = datetime.datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\r\n print(\"@CoLVDM Inference: %s\"%now)\r\n parser = get_parser()\r\n args = parser.parse_args()\r\n seed_everything(args.seed)\r\n rank, gpu_num = 0, 1\r\n run_inference(args, gpu_num, rank)" }, { "path": "scripts/run_text2video.sh", "content": "name=\"base_512_test\"\n\nckpt='checkpoints/base_512_v1/model.ckpt'\nconfig='configs/inference_t2v_tconv512_v1.0.yaml'\n\nprompt_file=\"prompts/single_prompts.txt\"\nres_dir=\"results_single_512\"\n\npython3 scripts/evaluation/inference.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 320 --width 512 \\\n--unconditional_guidance_scale 12.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 8 \\\n--frames 16 \n" }, { "path": "scripts/run_text2video_freenoise_1024.sh", "content": "name=\"base_1024_test\"\n\nckpt='checkpoints/base_1024_v1/model.ckpt'\nconfig='configs/inference_t2v_1024_v1.0_freenoise.yaml'\n\nprompt_file=\"prompts/single_prompts.txt\"\nres_dir=\"results_freenoise_single_1024\"\n\npython3 scripts/evaluation/inference_freenoise.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 576 --width 1024 \\\n--unconditional_guidance_scale 12.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 28 \\\n--frames 64 \\\n--window_size 16 \\\n--window_stride 4 \n" }, { "path": "scripts/run_text2video_freenoise_256.sh", "content": "name=\"base_256_test\"\n\nckpt='checkpoints/base_256_v1/model.ckpt'\nconfig='configs/inference_t2v_tconv256_v1.0_freenoise.yaml'\n\nprompt_file=\"prompts/single_prompts.txt\"\nres_dir=\"results_freenoise_single_256\"\n\npython3 scripts/evaluation/inference_freenoise.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 256 --width 256 \\\n--unconditional_guidance_scale 15.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 8 \\\n--frames 128 \\\n--window_size 16 \\\n--window_stride 4 \n\n" }, { "path": "scripts/run_text2video_freenoise_512.sh", "content": "name=\"base_512_test\"\n\nckpt='checkpoints/base_512_v2/model.ckpt'\nconfig='configs/inference_t2v_tconv512_v2.0_freenoise.yaml'\n\nprompt_file=\"prompts/single_prompts.txt\"\nres_dir=\"results_freenoise_single_512\"\n\npython3 scripts/evaluation/inference_freenoise.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 320 --width 512 \\\n--unconditional_guidance_scale 12.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 16 \\\n--frames 64 \\\n--window_size 16 \\\n--window_stride 4 \n" }, { "path": "scripts/run_text2video_freenoise_mp_256.sh", "content": "name=\"base_256_test\"\n\nckpt='checkpoints/base_256_v1/model.ckpt'\nconfig='configs/inference_t2v_tconv256_v1.0_freenoise.yaml'\n\nprompt_file=\"prompts/mp_prompts.txt\"\nres_dir=\"results_freenoise_mp_256\"\n\npython3 scripts/evaluation/inference_freenoise_mp.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 256 --width 256 \\\n--unconditional_guidance_scale 15.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 8 \\\n--frames 64 \\\n--window_size 16 \\\n--window_stride 4 \n\n" }, { "path": "scripts/run_text2video_freenoise_mp_512.sh", "content": "name=\"base_512_test\"\n\nckpt='checkpoints/base_512_v2/model.ckpt'\nconfig='configs/inference_t2v_tconv512_v2.0_freenoise.yaml'\n\nprompt_file=\"prompts/mp_prompts.txt\"\nres_dir=\"results_freenoise_mp_512\"\n\npython3 scripts/evaluation/inference_freenoise_mp.py \\\n--seed 123 \\\n--mode 'base' \\\n--ckpt_path $ckpt \\\n--config $config \\\n--savedir $res_dir/$name \\\n--n_samples 3 \\\n--bs 1 --height 320 --width 512 \\\n--unconditional_guidance_scale 12.0 \\\n--ddim_steps 50 \\\n--ddim_eta 0.0 \\\n--prompt_file $prompt_file \\\n--fps 16 \\\n--frames 64 \\\n--window_size 16 \\\n--window_stride 4 \n" }, { "path": "utils/utils.py", "content": "import importlib\nimport numpy as np\nimport cv2\nimport torch\nimport torch.distributed as dist\n\n\ndef count_params(model, verbose=False):\n total_params = sum(p.numel() for p in model.parameters())\n if verbose:\n print(f\"{model.__class__.__name__} has {total_params*1.e-6:.2f} M params.\")\n return total_params\n\n\ndef check_istarget(name, para_list):\n \"\"\" \n name: full name of source para\n para_list: partial name of target para \n \"\"\"\n istarget=False\n for para in para_list:\n if para in name:\n return True\n return istarget\n\n\ndef instantiate_from_config(config):\n if not \"target\" in config:\n if config == '__is_first_stage__':\n return None\n elif config == \"__is_unconditional__\":\n return None\n raise KeyError(\"Expected key `target` to instantiate.\")\n return get_obj_from_str(config[\"target\"])(**config.get(\"params\", dict()))\n\n\ndef get_obj_from_str(string, reload=False):\n module, cls = string.rsplit(\".\", 1)\n if reload:\n module_imp = importlib.import_module(module)\n importlib.reload(module_imp)\n return getattr(importlib.import_module(module, package=None), cls)\n\n\ndef load_npz_from_dir(data_dir):\n data = [np.load(os.path.join(data_dir, data_name))['arr_0'] for data_name in os.listdir(data_dir)]\n data = np.concatenate(data, axis=0)\n return data\n\n\ndef load_npz_from_paths(data_paths):\n data = [np.load(data_path)['arr_0'] for data_path in data_paths]\n data = np.concatenate(data, axis=0)\n return data \n\n\ndef resize_numpy_image(image, max_resolution=512 * 512, resize_short_edge=None):\n h, w = image.shape[:2]\n if resize_short_edge is not None:\n k = resize_short_edge / min(h, w)\n else:\n k = max_resolution / (h * w)\n k = k**0.5\n h = int(np.round(h * k / 64)) * 64\n w = int(np.round(w * k / 64)) * 64\n image = cv2.resize(image, (w, h), interpolation=cv2.INTER_LANCZOS4)\n return image\n\n\ndef setup_dist(args):\n if dist.is_initialized():\n return\n torch.cuda.set_device(args.local_rank)\n torch.distributed.init_process_group(\n 'nccl',\n init_method='env://'\n )" } ]