[
{
"path": ".gitignore",
"content": "**/.ipynb_checkpoints\n**/__pycache__\n*.mp4\n*.npy\n*.npz\n*.dae\n*.json\nblender-3.2.2-linux-x64\ndata/material_lib/mat_images\ndata/material_lib/pbr_maps\nexperiments\nexperiments_szy\nexperiments_tmp\ndata_0\noutput\ntmp\nmain_mod.py\nscripts/gpt_eval\nsom/ckpts/swinl_only_sam_many2many.pth\n.idea\n.vscode"
},
{
"path": "INSTALL.md",
"content": "### πΎ Installation\n\n1. Install basic modules: torch and packages in requirements.txt\n \n ```bash\n git clone https://github.com/Aleafy/Make_it_Real.git\n cd Make_it_Real\n\n conda create -n mkreal python=3.8 \n pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html\n pip install -r requirements.txt\n ```\n\n2. Install rendering utils: Blender and Kaolin\n \n Download [blender-3.2.2-linux-x64.tar.xz_original](https://download.blender.org/release/Blender3.2/) or [blender_3.2.2_with_installed_modules](https://drive.google.com/file/d/1PKbCS7VymPo_xVYavT0CDd42OmXbfurY/view?usp=sharing)\n\n ```bash\n tar -xvf blender-3.2.2-linux-x64.tar.xz\n export PATH=$PATH:path_to_blender/blender-3.2.2-linux-x64\n ```\n\n and Kaolin\n ```bash\n pip install kaolin==0.13.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.10.1_cu111.html\n ```\n \n3. Install segment & mark utils:\n ```\n # install Semantic-SAM\n pip install git+https://github.com/UX-Decoder/Semantic-SAM.git@package\n # install Deformable Convolution for Semantic-SAM\n cd som/ops && sh make.sh && cd ..\n # install detectron2\n python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html\n ```\n Get model weight through `bash som/ckpts/download_ckpt.sh`"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2024 Aleafy\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "# ![](\"assets/icon.png\")
Make-it-Real\n\n**[Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials](https://arxiv.org/abs/2404.16829)**\n\n[Ye Fang](https://github.com/Aleafy)\\*,\n[Zeyi Sun](https://github.com/SunzeY)\\*,\n[Tong Wu](https://wutong16.github.io/),\n[Jiaqi Wang](https://myownskyw7.github.io/),\n[Ziwei Liu](https://liuziwei7.github.io/),\n[Gordon Wetzstein](https://web.stanford.edu/~gordonwz/),\n[Dahua Lin](http://dahua.site/)\n\n*Equal Contribution
\n\n![](\"https://img.shields.io/badge/arXiv-Paper-<color)
\">\n![](\"https://img.shields.io/badge/Project-Website-red\")
\n![](\"https://img.shields.io/static/v1?label=Demo&message=Video&color=orange\")
\n\n![](\"https://visitor-badge.laobi.icu/badge?page_id=Aleafy.Make_it_Real&left_color=gray&right_color=blue\")
\n\n
\n\n\n\n\n\n## π News\nπ [2024/6/8] We release our [inference pipeline of Make-it-Real](#β‘-quick-start), including material matching and generation of albedo-only 3D objects.\n\nπ [2024/6/8] [Material library annotations](#π¦-data-preparation) generated by GPT-4V and [data engine](#β‘-quick-start) are released!\n\nπ [2024/4/26] The [paper](https://arxiv.org/abs/2404.16829) and [project page](https://sunzey.github.io/Make-it-Real) are released!\n\n## π‘ Highlights\n- π₯ We first demonstrate that **GPT-4V** can effectively **recognize and describe materials**, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.\n- π₯ We construct a **Material Library** containing thousands of materials with highly\ndetailed descriptions readily for MLLMs to look up and assign.\n- π₯ **An effective pipeline** for texture segmentation, material identification and matching, enabling the high-quality application of materials to\n3D assets.\n\n## π¨βπ» Todo\n- [ ] Evaluation for Existed and Model-Generated Assets (both code & test assets)\n- [ ] More Interactive Demos (huggingface, jupyter) \n- [x] Make-it-Real Pipeline Inference Code\n- [x] Highly detailed Material Library annotations (generated by GPT-4V) \n- [x] Paper and Web Demos\n\n## πΎ Installation\nPrepare basic modules for deep learning 3d modeling tool(kaolin), rendering engine(blender), and segmentation model. See details in [INSTALL.md](INSTALL.md).\n\n\n## π¦ Data Preparation\n 1. **Annotations**: in `data/material_lib/annotations` [folder](data/material_lib/annotations), include:\n - Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the materialβs visual characteristics and rich semantic information.\n - Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.\n 2. **PBR Maps**: You can download the complete PBR data collection at [Huggingface](https://huggingface.co/datasets/gvecchio/MatSynth/tree/main), or download the data used in our project at [OpenXLab](https://openxlab.org.cn/datasets/YeFang/MatSynth/tree/main) (Recommended). (If you have any questions, please refer to [issue#5](https://github.com/Aleafy/Make_it_Real/issues/5))\n 3. **Material Images(optinal)**: You can download the material images file [here](https://drive.google.com/file/d/1ob7CV6JiaqFyjuCzlmSnBuNRkzt2qMSG/view?usp=sharing), to check and visualize the material appearance.\n\n\nMake_it_Real\nβββ data\n βββ material_lib\n βββ annotations\n βββ mat_images\n βββ pbr_maps\n βββ train\n βββ Ceremic\n βββ Concrete\n βββ ...\n βββ Wood\n
\n\n\n\n## β‘ Quick Start\n#### Inference\n```bash\npython main.py --obj_dir --exp_name --api_key \n```\n- To ensure proper network connectivity for GPT-4V, add proxy environment settings in [main.py](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/main.py#L18) (optional). Also, please verify the reachability of your [API host](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/utils/gpt4_query.py#L68).\n- Result visualization (blender engine) is located in the `output/refine_output` dir. You can compare the result with that in `output/ori_output`. \n\n#### Annotation Engine\n\n```bash\ncd scripts/gpt_anno\npython gpt4_query_mat.py\n```\n`Note`: Besides functinoning as annotation engine, you can also use this code ([gpt4_query_mat.py](https://github.com/Aleafy/Make_it_Real/blob/main/scripts/gpt_anno/gpt4_query_mat.py)) to test the GPT-4V connection simply.\n\n\n