[ { "path": "LICENSE.txt", "content": "BSD 3-Clause License\n\nCopyright (c) 2022 Salesforce, Inc.\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\n3. Neither the name of Salesforce.com nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "MANIFEST.in", "content": "recursive-include lavis/configs *.yaml *.json\nrecursive-include lavis/projects *.yaml *.json\n\nrecursive-exclude lavis/datasets/download_scripts *\nrecursive-exclude lavis/output *\n\ninclude requirements.txt\n" }, { "path": "README.md", "content": "# [NeurIPS 2023] Self-Chained Image-Language Model for Video Localization and Question Answering\n\n* Authors: [Shoubin Yu](https://yui010206.github.io/), [Jaemin Cho](https://j-min.io), [Prateek Yadav](https://prateek-yadav.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/)\n* Paper: [arXiv](https://arxiv.org/abs/2305.06988)\n* Online Demo: Try our Gradio demo on Hugging Face[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/Shoubin/SeViLA)\n\n\"teaser\n\n\"teaser\n\n\"teaser\n\n\n# Code structure\n```bash\n\n# data & data preprocessing\n./sevila_data\n\n# pretrained checkpoints\n./sevila_checkpoints\n\n# SeViLA code\n./lavis/\n\n# running scripts for SeViLA localizer/answerer training/inference\n./run_scripts\n\n```\n\n# Setup\n\n## Install Dependencies\n\n1. (Optional) Creating conda environment\n\n```bash\nconda create -n sevila python=3.8\nconda activate sevila\n```\n\n2. build from source\n\n```bash\npip install -e .\n```\n\n## Download Pretrained Models\nWe pre-train SeViLA localizer on QVHighlights and hold checkpoints via [Hugging Face](https://huggingface.co/Shoubin/SeViLA/resolve/main/sevila_pretrained.pth).\nDownload checkpoints and put it under /sevila_checkpoints.\nThe checkpoints (814.55M) contains pre-trained localizer and zero-shot answerer.\n\nIf you want to pre-train your own localizer, you can download [qformer_loc.pth](https://drive.google.com/file/d/13hE_BQDflkzYrHVmVGddRSt8VMa0ouGB/view?usp=sharing), which is a copy of the original BLIP-2 Q-former to initialize the localizer (with changed model keys).\n\n# Run Gradio Demo Locally\nWe also provide a UI for testing our SeViLA locally that is built with gradio. \nRunning demo locally requires about 12GB of memory.\n\n* Installing Gradio:\n\n```bash\npip install gradio==3.30.0\n```\n\n* Running the following command in a terminal will launch the demo:\n\n```bash\npython app.py\n```\n\n# Dataset Preparation\nWe test our model on:\n+ [NExT-QA](https://doc-doc.github.io/docs/nextqa.html)\n\n+ [STAR](https://star.csail.mit.edu/)\n\n+ [How2QA](https://value-benchmark.github.io/index.html)\n\n+ [TVQA](https://tvqa.cs.unc.edu/)\n\n+ [VLEP](https://value-benchmark.github.io/index.html)\n\n+ [QVHighlights](https://github.com/jayleicn/moment_detr)\n\nPlease download original QA data and preprocess them via our [scripts](sevila_data/).\n\n\n# Training and Inference\nWe provide SeViLA training and inference script examples as follows.\n\nAnd please refer to [dataset page](sevila_data/) to custom your data path.\n\n## 1) Localizer Pre-training\n```bash\nsh run_scripts/sevila/pre-train/pretrain_qvh.sh\n```\n\n## 2) Answerer Fine-tuning\n\n```bash\nsh run_scripts/sevila/finetune/nextqa_ft.sh\n```\n\n## 3) Localizer Self-refinement\n\n```bash\nsh run_scripts/sevila/refinement/nextqa_sr.sh\n```\n\n## 4) Inference\n\n```bash\nsh run_scripts/sevila/inference/nextqa_infer.sh\n```\n\n\n# Acknowledgments\nWe thank the developers of [LAVIS](https://github.com/salesforce/LAVIS), [BLIP-2](https://github.com/salesforce/LAVIS/tree/main/projects/blip2), [CLIP](https://github.com/openai/CLIP), [All-in-One](https://github.com/showlab/all-in-one), for their public code release.\n\n\n# Reference\nPlease cite our paper if you use our models in your works:\n\n\n```bibtex\n@inproceedings{yu2023self,\n title = {Self-Chained Image-Language Model for Video Localization and Question Answering},\n author = {Yu, Shoubin and Cho, Jaemin and Yadav, Prateek and Bansal, Mohit},\n booktitle = {NeurIPS},\n year = {2023}\n}\n" }, { "path": "app/__init__.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nfrom PIL import Image\nimport requests\n\nimport streamlit as st\nimport torch\n\n\n@st.cache()\ndef load_demo_image():\n img_url = (\n \"https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg\"\n )\n raw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\n return raw_image\n\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\ncache_root = \"/export/home/.cache/lavis/\"\n" }, { "path": "app/calculate_coco_features.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nfrom PIL import Image\nimport requests\nimport torch\n\nimport os\n\nfrom lavis.common.registry import registry\nfrom lavis.processors import *\nfrom lavis.models import *\nfrom lavis.common.utils import build_default_model\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n\ndef load_demo_image():\n img_url = (\n \"https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg\"\n )\n raw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\n\n return raw_image\n\n\ndef read_img(filepath):\n raw_image = Image.open(filepath).convert(\"RGB\")\n\n return raw_image\n\n\n# model\nmodel_url = \"https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth\"\nfeature_extractor = BlipFeatureExtractor(pretrained=model_url)\n\nfeature_extractor.eval()\nfeature_extractor = feature_extractor.to(device)\n\n# preprocessors\nvis_processor = BlipImageEvalProcessor(image_size=224)\ntext_processor = BlipCaptionProcessor()\n\n# files to process\n# file_root = \"/export/home/.cache/lavis/coco/images/val2014\"\nfile_root = \"/export/home/.cache/lavis/coco/images/train2014\"\nfilepaths = os.listdir(file_root)\n\nprint(len(filepaths))\n\ncaption = \"dummy\"\n\npath2feat = dict()\nbsz = 256\n\nimages_in_batch = []\nfilepaths_in_batch = []\n\nfor i, filename in enumerate(filepaths):\n if i % bsz == 0 and i > 0:\n images_in_batch = torch.cat(images_in_batch, dim=0).to(device)\n with torch.no_grad():\n image_features = feature_extractor(\n images_in_batch, caption, mode=\"image\", normalized=True\n )[:, 0]\n\n for filepath, image_feat in zip(filepaths_in_batch, image_features):\n path2feat[os.path.basename(filepath)] = image_feat.detach().cpu()\n\n images_in_batch = []\n filepaths_in_batch = []\n\n print(len(path2feat), image_features.shape)\n else:\n filepath = os.path.join(file_root, filename)\n\n image = read_img(filepath)\n image = vis_processor(image).unsqueeze(0)\n\n images_in_batch.append(image)\n filepaths_in_batch.append(filepath)\n\ntorch.save(path2feat, \"path2feat_coco_train2014.pth\")\n" }, { "path": "app/caption.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport streamlit as st\nfrom app import device, load_demo_image\nfrom app.utils import load_model_cache\nfrom lavis.processors import load_processor\nfrom PIL import Image\n\n\ndef app():\n # ===== layout =====\n model_type = st.sidebar.selectbox(\"Model:\", [\"BLIP_base\", \"BLIP_large\"])\n\n sampling_method = st.sidebar.selectbox(\n \"Sampling method:\", [\"Beam search\", \"Nucleus sampling\"]\n )\n\n st.markdown(\n \"

Image Description Generation

\",\n unsafe_allow_html=True,\n )\n\n instructions = \"\"\"Try the provided image or upload your own:\"\"\"\n file = st.file_uploader(instructions)\n\n use_beam = sampling_method == \"Beam search\"\n\n col1, col2 = st.columns(2)\n\n if file:\n raw_img = Image.open(file).convert(\"RGB\")\n else:\n raw_img = load_demo_image()\n\n col1.header(\"Image\")\n\n w, h = raw_img.size\n scaling_factor = 720 / w\n resized_image = raw_img.resize((int(w * scaling_factor), int(h * scaling_factor)))\n\n col1.image(resized_image, use_column_width=True)\n col2.header(\"Description\")\n\n cap_button = st.button(\"Generate\")\n\n # ==== event ====\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n\n if cap_button:\n if model_type.startswith(\"BLIP\"):\n blip_type = model_type.split(\"_\")[1].lower()\n model = load_model_cache(\n \"blip_caption\",\n model_type=f\"{blip_type}_coco\",\n is_eval=True,\n device=device,\n )\n\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n captions = generate_caption(\n model=model, image=img, use_nucleus_sampling=not use_beam\n )\n\n col2.write(\"\\n\\n\".join(captions), use_column_width=True)\n\n\ndef generate_caption(\n model, image, use_nucleus_sampling=False, num_beams=3, max_length=40, min_length=5\n):\n samples = {\"image\": image}\n\n captions = []\n if use_nucleus_sampling:\n for _ in range(5):\n caption = model.generate(\n samples,\n use_nucleus_sampling=True,\n max_length=max_length,\n min_length=min_length,\n top_p=0.9,\n )\n captions.append(caption[0])\n else:\n caption = model.generate(\n samples,\n use_nucleus_sampling=False,\n num_beams=num_beams,\n max_length=max_length,\n min_length=min_length,\n )\n captions.append(caption[0])\n\n return captions\n" }, { "path": "app/classification.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport plotly.graph_objects as go\nimport requests\nimport streamlit as st\nimport torch\nfrom lavis.models import load_model\nfrom lavis.processors import load_processor\nfrom lavis.processors.blip_processors import BlipCaptionProcessor\nfrom PIL import Image\n\nfrom app import device, load_demo_image\nfrom app.utils import load_blip_itm_model\nfrom lavis.processors.clip_processors import ClipImageEvalProcessor\n\n\n@st.cache()\ndef load_demo_image(img_url=None):\n if not img_url:\n img_url = \"https://img.atlasobscura.com/yDJ86L8Ou6aIjBsxnlAy5f164w1rjTgcHZcx2yUs4mo/rt:fit/w:1200/q:81/sm:1/scp:1/ar:1/aHR0cHM6Ly9hdGxh/cy1kZXYuczMuYW1h/em9uYXdzLmNvbS91/cGxvYWRzL3BsYWNl/X2ltYWdlcy85MDll/MDRjOS00NTJjLTQx/NzQtYTY4MS02NmQw/MzI2YWIzNjk1ZGVk/MGZhMTJiMTM5MmZi/NGFfUmVhcl92aWV3/X29mX3RoZV9NZXJs/aW9uX3N0YXR1ZV9h/dF9NZXJsaW9uX1Bh/cmssX1NpbmdhcG9y/ZSxfd2l0aF9NYXJp/bmFfQmF5X1NhbmRz/X2luX3RoZV9kaXN0/YW5jZV8tXzIwMTQw/MzA3LmpwZw.jpg\"\n raw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\n return raw_image\n\n\n@st.cache(\n hash_funcs={\n torch.nn.parameter.Parameter: lambda parameter: parameter.data.detach()\n .cpu()\n .numpy()\n },\n allow_output_mutation=True,\n)\ndef load_model_cache(model_type, device):\n if model_type == \"blip\":\n model = load_model(\n \"blip_feature_extractor\", model_type=\"base\", is_eval=True, device=device\n )\n elif model_type == \"albef\":\n model = load_model(\n \"albef_feature_extractor\", model_type=\"base\", is_eval=True, device=device\n )\n elif model_type == \"CLIP_ViT-B-32\":\n model = load_model(\n \"clip_feature_extractor\", \"ViT-B-32\", is_eval=True, device=device\n )\n elif model_type == \"CLIP_ViT-B-16\":\n model = load_model(\n \"clip_feature_extractor\", \"ViT-B-16\", is_eval=True, device=device\n )\n elif model_type == \"CLIP_ViT-L-14\":\n model = load_model(\n \"clip_feature_extractor\", \"ViT-L-14\", is_eval=True, device=device\n )\n\n return model\n\n\ndef app():\n model_type = st.sidebar.selectbox(\n \"Model:\",\n [\"ALBEF\", \"BLIP_Base\", \"CLIP_ViT-B-32\", \"CLIP_ViT-B-16\", \"CLIP_ViT-L-14\"],\n )\n score_type = st.sidebar.selectbox(\"Score type:\", [\"Cosine\", \"Multimodal\"])\n\n # ===== layout =====\n st.markdown(\n \"

Zero-shot Classification

\",\n unsafe_allow_html=True,\n )\n\n instructions = \"\"\"Try the provided image or upload your own:\"\"\"\n file = st.file_uploader(instructions)\n\n st.header(\"Image\")\n if file:\n raw_img = Image.open(file).convert(\"RGB\")\n else:\n raw_img = load_demo_image()\n\n st.image(raw_img) # , use_column_width=True)\n\n col1, col2 = st.columns(2)\n\n col1.header(\"Categories\")\n\n cls_0 = col1.text_input(\"category 1\", value=\"merlion\")\n cls_1 = col1.text_input(\"category 2\", value=\"sky\")\n cls_2 = col1.text_input(\"category 3\", value=\"giraffe\")\n cls_3 = col1.text_input(\"category 4\", value=\"fountain\")\n cls_4 = col1.text_input(\"category 5\", value=\"marina bay\")\n\n cls_names = [cls_0, cls_1, cls_2, cls_3, cls_4]\n cls_names = [cls_nm for cls_nm in cls_names if len(cls_nm) > 0]\n\n if len(cls_names) != len(set(cls_names)):\n st.error(\"Please provide unique class names\")\n return\n\n button = st.button(\"Submit\")\n\n col2.header(\"Prediction\")\n\n # ===== event =====\n\n if button:\n if model_type.startswith(\"BLIP\"):\n text_processor = BlipCaptionProcessor(prompt=\"A picture of \")\n cls_prompt = [text_processor(cls_nm) for cls_nm in cls_names]\n\n if score_type == \"Cosine\":\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=224)\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n\n feature_extractor = load_model_cache(model_type=\"blip\", device=device)\n\n sample = {\"image\": img, \"text_input\": cls_prompt}\n\n with torch.no_grad():\n image_features = feature_extractor.extract_features(\n sample, mode=\"image\"\n ).image_embeds_proj[:, 0]\n text_features = feature_extractor.extract_features(\n sample, mode=\"text\"\n ).text_embeds_proj[:, 0]\n sims = (image_features @ text_features.t())[\n 0\n ] / feature_extractor.temp\n\n else:\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n\n model = load_blip_itm_model(device)\n\n output = model(img, cls_prompt, match_head=\"itm\")\n sims = output[:, 1]\n\n sims = torch.nn.Softmax(dim=0)(sims)\n inv_sims = [sim * 100 for sim in sims.tolist()[::-1]]\n\n elif model_type.startswith(\"ALBEF\"):\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=224)\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n\n text_processor = BlipCaptionProcessor(prompt=\"A picture of \")\n cls_prompt = [text_processor(cls_nm) for cls_nm in cls_names]\n\n feature_extractor = load_model_cache(model_type=\"albef\", device=device)\n\n sample = {\"image\": img, \"text_input\": cls_prompt}\n\n with torch.no_grad():\n image_features = feature_extractor.extract_features(\n sample, mode=\"image\"\n ).image_embeds_proj[:, 0]\n text_features = feature_extractor.extract_features(\n sample, mode=\"text\"\n ).text_embeds_proj[:, 0]\n\n st.write(image_features.shape)\n st.write(text_features.shape)\n\n sims = (image_features @ text_features.t())[0] / feature_extractor.temp\n\n sims = torch.nn.Softmax(dim=0)(sims)\n inv_sims = [sim * 100 for sim in sims.tolist()[::-1]]\n\n elif model_type.startswith(\"CLIP\"):\n if model_type == \"CLIP_ViT-B-32\":\n model = load_model_cache(model_type=\"CLIP_ViT-B-32\", device=device)\n elif model_type == \"CLIP_ViT-B-16\":\n model = load_model_cache(model_type=\"CLIP_ViT-B-16\", device=device)\n elif model_type == \"CLIP_ViT-L-14\":\n model = load_model_cache(model_type=\"CLIP_ViT-L-14\", device=device)\n else:\n raise ValueError(f\"Unknown model type {model_type}\")\n\n if score_type == \"Cosine\":\n # image_preprocess = ClipImageEvalProcessor(image_size=336)\n image_preprocess = ClipImageEvalProcessor(image_size=224)\n img = image_preprocess(raw_img).unsqueeze(0).to(device)\n\n sample = {\"image\": img, \"text_input\": cls_names}\n\n with torch.no_grad():\n clip_features = model.extract_features(sample)\n\n image_features = clip_features.image_embeds_proj\n text_features = clip_features.text_embeds_proj\n\n sims = (100.0 * image_features @ text_features.T)[0].softmax(dim=-1)\n inv_sims = sims.tolist()[::-1]\n else:\n st.warning(\"CLIP does not support multimodal scoring.\")\n return\n\n fig = go.Figure(\n go.Bar(\n x=inv_sims,\n y=cls_names[::-1],\n text=[\"{:.2f}\".format(s) for s in inv_sims],\n orientation=\"h\",\n )\n )\n fig.update_traces(\n textfont_size=12,\n textangle=0,\n textposition=\"outside\",\n cliponaxis=False,\n )\n col2.plotly_chart(fig, use_container_width=True)\n" }, { "path": "app/dataset_browser.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport random\nfrom collections import OrderedDict\nfrom functools import reduce\nfrom tkinter import N\n\nimport streamlit as st\nfrom lavis.common.registry import registry\nfrom lavis.datasets.builders import dataset_zoo, load_dataset\nfrom lavis.datasets.builders.base_dataset_builder import load_dataset_config\nfrom PIL import Image\n\nIMAGE_LAYOUT = 3, 4\nVIDEO_LAYOUT = 1, 2\n\nPREV_STR = \"Prev\"\nNEXT_STR = \"Next\"\n\n\ndef sample_dataset(dataset, indices):\n samples = [dataset.displ_item(idx) for idx in indices]\n\n return samples\n\n\ndef get_concat_v(im1, im2):\n margin = 5\n\n canvas_size = (im1.width + im2.width + margin, max(im1.height, im2.height))\n canvas = Image.new(\"RGB\", canvas_size, \"White\")\n canvas.paste(im1, (0, 0))\n canvas.paste(im2, (im1.width + margin, 0))\n\n return canvas\n\n\ndef resize_img_w(raw_img, new_w=224):\n if isinstance(raw_img, list):\n resized_imgs = [resize_img_w(img, 196) for img in raw_img]\n # concatenate images\n resized_image = reduce(get_concat_v, resized_imgs)\n else:\n w, h = raw_img.size\n scaling_factor = new_w / w\n resized_image = raw_img.resize(\n (int(w * scaling_factor), int(h * scaling_factor))\n )\n\n return resized_image\n\n\ndef get_visual_key(dataset):\n if \"image\" in dataset[0]:\n return \"image\"\n elif \"image0\" in dataset[0]: # NLVR2 dataset\n return \"image\"\n elif \"video\" in dataset[0]:\n return \"video\"\n else:\n raise ValueError(\"Visual key not found.\")\n\n\ndef gather_items(samples, exclude=[]):\n gathered = []\n\n for s in samples:\n ns = OrderedDict()\n for k in s.keys():\n if k not in exclude:\n ns[k] = s[k]\n\n gathered.append(ns)\n\n return gathered\n\n\n@st.cache(allow_output_mutation=True)\ndef load_dataset_cache(name):\n return load_dataset(name)\n\n\ndef format_text(text):\n md = \"\\n\\n\".join([f\"**{k}**: {v}\" for k, v in text.items()])\n\n return md\n\n\ndef show_samples(dataset, offset=0, is_next=False):\n visual_key = get_visual_key(dataset)\n\n num_rows, num_cols = IMAGE_LAYOUT if visual_key == \"image\" else VIDEO_LAYOUT\n n_samples = num_rows * num_cols\n\n if not shuffle:\n if is_next:\n start = min(int(start_idx) + offset + n_samples, len(dataset) - n_samples)\n else:\n start = max(0, int(start_idx) + offset - n_samples)\n\n st.session_state.last_start = start\n end = min(start + n_samples, len(dataset))\n\n indices = list(range(start, end))\n else:\n indices = random.sample(range(len(dataset)), n_samples)\n samples = sample_dataset(dataset, indices)\n\n visual_info = (\n iter([resize_img_w(s[visual_key]) for s in samples])\n if visual_key == \"image\"\n # else iter([s[visual_key] for s in samples])\n else iter([s[\"file\"] for s in samples])\n )\n text_info = gather_items(samples, exclude=[\"image\", \"video\"])\n text_info = iter([format_text(s) for s in text_info])\n\n st.markdown(\n \"\"\"
\"\"\",\n unsafe_allow_html=True,\n )\n for _ in range(num_rows):\n with st.container():\n for col in st.columns(num_cols):\n # col.text(next(text_info))\n # col.caption(next(text_info))\n try:\n col.markdown(next(text_info))\n if visual_key == \"image\":\n col.image(next(visual_info), use_column_width=True, clamp=True)\n elif visual_key == \"video\":\n col.markdown(\n \"![Alt Text](https://media.giphy.com/media/vFKqnCdLPNOKc/giphy.gif)\"\n )\n except StopIteration:\n break\n\n st.markdown(\n \"\"\"
\"\"\",\n unsafe_allow_html=True,\n )\n\n st.session_state.n_display = n_samples\n\n\nif __name__ == \"__main__\":\n st.set_page_config(\n page_title=\"LAVIS Dataset Explorer\",\n # layout=\"wide\",\n initial_sidebar_state=\"expanded\",\n )\n\n dataset_name = st.sidebar.selectbox(\"Dataset:\", dataset_zoo.get_names())\n\n function = st.sidebar.selectbox(\"Function:\", [\"Browser\"], index=0)\n\n if function == \"Browser\":\n shuffle = st.sidebar.selectbox(\"Shuffled:\", [True, False], index=0)\n\n dataset = load_dataset_cache(dataset_name)\n split = st.sidebar.selectbox(\"Split:\", dataset.keys())\n\n dataset_len = len(dataset[split])\n st.success(\n f\"Loaded {dataset_name}/{split} with **{dataset_len}** records. **Image/video directory**: {dataset[split].vis_root}\"\n )\n\n if \"last_dataset\" not in st.session_state:\n st.session_state.last_dataset = dataset_name\n st.session_state.last_split = split\n\n if \"last_start\" not in st.session_state:\n st.session_state.last_start = 0\n\n if \"start_idx\" not in st.session_state:\n st.session_state.start_idx = 0\n\n if \"shuffle\" not in st.session_state:\n st.session_state.shuffle = shuffle\n\n if \"first_run\" not in st.session_state:\n st.session_state.first_run = True\n elif (\n st.session_state.last_dataset != dataset_name\n or st.session_state.last_split != split\n ):\n st.session_state.first_run = True\n\n st.session_state.last_dataset = dataset_name\n st.session_state.last_split = split\n elif st.session_state.shuffle != shuffle:\n st.session_state.shuffle = shuffle\n st.session_state.first_run = True\n\n if not shuffle:\n n_col, p_col = st.columns([0.05, 1])\n\n prev_button = n_col.button(PREV_STR)\n next_button = p_col.button(NEXT_STR)\n\n else:\n next_button = st.button(NEXT_STR)\n\n if not shuffle:\n start_idx = st.sidebar.text_input(f\"Begin from (total {dataset_len})\", 0)\n\n if not start_idx.isdigit():\n st.error(f\"Input to 'Begin from' must be digits, found {start_idx}.\")\n else:\n if int(start_idx) != st.session_state.start_idx:\n st.session_state.start_idx = int(start_idx)\n st.session_state.last_start = int(start_idx)\n\n if prev_button:\n show_samples(\n dataset[split],\n offset=st.session_state.last_start - st.session_state.start_idx,\n is_next=False,\n )\n\n if next_button:\n show_samples(\n dataset[split],\n offset=st.session_state.last_start - st.session_state.start_idx,\n is_next=True,\n )\n\n if st.session_state.first_run:\n st.session_state.first_run = False\n\n show_samples(\n dataset[split],\n offset=st.session_state.last_start - st.session_state.start_idx,\n is_next=True,\n )\n" }, { "path": "app/image_text_match.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport numpy as np\nimport streamlit as st\nimport torch\nfrom lavis.models.blip_models.blip_image_text_matching import compute_gradcam\nfrom lavis.processors import load_processor\nfrom PIL import Image\n\nfrom app import device, load_demo_image\nfrom app.utils import getAttMap, init_bert_tokenizer, load_blip_itm_model\n\n\ndef app():\n model_type = st.sidebar.selectbox(\"Model:\", [\"BLIP_base\", \"BLIP_large\"])\n\n if model_type.startswith(\"BLIP\"):\n blip_type = model_type.split(\"_\")[1]\n model = load_blip_itm_model(device, model_type=blip_type)\n\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n\n st.markdown(\n \"

Image Text Matching

\",\n unsafe_allow_html=True,\n )\n\n values = list(range(1, 12))\n default_layer_num = values.index(7)\n layer_num = (\n st.sidebar.selectbox(\"Layer number\", values, index=default_layer_num) - 1\n )\n\n instructions = \"\"\"Try the provided image or upload your own:\"\"\"\n file = st.file_uploader(instructions)\n\n col1, col2 = st.columns(2)\n col1.header(\"Image\")\n col2.header(\"GradCam\")\n if file:\n raw_img = Image.open(file).convert(\"RGB\")\n else:\n raw_img = load_demo_image()\n\n w, h = raw_img.size\n scaling_factor = 720 / w\n resized_image = raw_img.resize((int(w * scaling_factor), int(h * scaling_factor)))\n col1.image(resized_image, use_column_width=True)\n\n col3, col4 = st.columns(2)\n col3.header(\"Text\")\n user_question = col3.text_input(\n \"Input your sentence!\", \"a woman sitting on the beach with a dog\"\n )\n submit_button = col3.button(\"Submit\")\n\n col4.header(\"Matching score\")\n\n if submit_button:\n tokenizer = init_bert_tokenizer()\n\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n text_processor = load_processor(\"blip_caption\").build()\n\n qry = text_processor(user_question)\n\n norm_img = np.float32(resized_image) / 255\n\n qry_tok = tokenizer(qry, return_tensors=\"pt\").to(device)\n gradcam, output = compute_gradcam(model, img, qry, qry_tok, block_num=layer_num)\n\n avg_gradcam = getAttMap(norm_img, gradcam[0][1], blur=True)\n\n col2.image(avg_gradcam, use_column_width=True, clamp=True)\n # output = model(img, question)\n itm_score = torch.nn.functional.softmax(output, dim=1)\n new_title = (\n '

\\n{:.3f}%

'.format(\n itm_score[0][1].item() * 100\n )\n )\n col4.markdown(new_title, unsafe_allow_html=True)\n" }, { "path": "app/main.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nfrom app.multipage import MultiPage\nfrom app import vqa, caption\nfrom app import image_text_match as itm\nfrom app import text_localization as tl\nfrom app import multimodal_search as ms\nfrom app import classification as cl\n\n\nif __name__ == \"__main__\":\n app = MultiPage()\n\n app.add_page(\"Image Description Generation\", caption.app)\n app.add_page(\"Multimodal Search\", ms.app)\n app.add_page(\"Visual Question Answering\", vqa.app)\n app.add_page(\"Image Text Matching\", itm.app)\n app.add_page(\"Text Localization\", tl.app)\n app.add_page(\"Classification\", cl.app)\n app.run()\n" }, { "path": "app/multimodal_search.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport os\n\nimport numpy as np\nimport streamlit as st\nimport torch\nimport torch.nn.functional as F\nfrom app import cache_root, device\nfrom app.utils import (\n getAttMap,\n init_bert_tokenizer,\n load_blip_itm_model,\n read_img,\n resize_img,\n)\nfrom lavis.models import load_model\nfrom lavis.processors import load_processor\n\n\n@st.cache(\n hash_funcs={\n torch.nn.parameter.Parameter: lambda parameter: parameter.data.detach()\n .cpu()\n .numpy()\n },\n allow_output_mutation=True,\n)\ndef load_feat():\n from lavis.common.utils import download_url\n\n dirname = os.path.join(os.path.dirname(__file__), \"assets\")\n filename = \"path2feat_coco_train2014.pth\"\n filepath = os.path.join(dirname, filename)\n url = \"https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/path2feat_coco_train2014.pth\"\n\n if not os.path.exists(filepath):\n download_url(url=url, root=dirname, filename=\"path2feat_coco_train2014.pth\")\n\n path2feat = torch.load(filepath)\n paths = sorted(path2feat.keys())\n\n all_img_feats = torch.stack([path2feat[k] for k in paths], dim=0).to(device)\n\n return path2feat, paths, all_img_feats\n\n\n@st.cache(\n hash_funcs={\n torch.nn.parameter.Parameter: lambda parameter: parameter.data.detach()\n .cpu()\n .numpy()\n },\n allow_output_mutation=True,\n)\ndef load_feature_extractor_model(device):\n model_url = \"https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth\"\n\n model = load_model(\n \"blip_feature_extractor\", model_type=\"base\", is_eval=True, device=device\n )\n model.load_from_pretrained(model_url)\n\n return model\n\n\ndef app():\n # === layout ===\n model_type = st.sidebar.selectbox(\"Model:\", [\"BLIP_base\", \"BLIP_large\"])\n file_root = os.path.join(cache_root, \"coco/images/train2014/\")\n\n values = [12, 24, 48]\n default_layer_num = values.index(24)\n num_display = st.sidebar.selectbox(\n \"Number of images:\", values, index=default_layer_num\n )\n show_gradcam = st.sidebar.selectbox(\"Show GradCam:\", [True, False], index=1)\n itm_ranking = st.sidebar.selectbox(\"Multimodal re-ranking:\", [True, False], index=0)\n\n # st.title('Multimodal Search')\n st.markdown(\n \"

Multimodal Search

\", unsafe_allow_html=True\n )\n\n # === event ===\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n text_processor = load_processor(\"blip_caption\")\n\n user_question = st.text_input(\n \"Search query\", \"A dog running on the grass.\", help=\"Type something to search.\"\n )\n user_question = text_processor(user_question)\n feature_extractor = load_feature_extractor_model(device)\n\n # ======= ITC =========\n sample = {\"text_input\": user_question}\n\n with torch.no_grad():\n text_feature = feature_extractor.extract_features(\n sample, mode=\"text\"\n ).text_embeds_proj[0, 0]\n\n path2feat, paths, all_img_feats = load_feat()\n all_img_feats.to(device)\n all_img_feats = F.normalize(all_img_feats, dim=1)\n\n num_cols = 4\n num_rows = int(num_display / num_cols)\n\n similarities = text_feature @ all_img_feats.T\n indices = torch.argsort(similarities, descending=True)[:num_display]\n\n top_paths = [paths[ind.detach().cpu().item()] for ind in indices]\n sorted_similarities = [similarities[idx] for idx in indices]\n filenames = [os.path.join(file_root, p) for p in top_paths]\n\n # ========= ITM and GradCam ==========\n bsz = 4 # max number of images to avoid cuda oom\n if model_type.startswith(\"BLIP\"):\n blip_type = model_type.split(\"_\")[1]\n\n itm_model = load_blip_itm_model(device, model_type=blip_type)\n\n tokenizer = init_bert_tokenizer()\n queries_batch = [user_question] * bsz\n queries_tok_batch = tokenizer(queries_batch, return_tensors=\"pt\").to(device)\n\n num_batches = int(num_display / bsz)\n\n avg_gradcams = []\n all_raw_images = []\n itm_scores = []\n\n for i in range(num_batches):\n filenames_in_batch = filenames[i * bsz : (i + 1) * bsz]\n raw_images, images = read_and_process_images(filenames_in_batch, vis_processor)\n gradcam, itm_output = compute_gradcam_batch(\n itm_model, images, queries_batch, queries_tok_batch\n )\n\n all_raw_images.extend([resize_img(r_img) for r_img in raw_images])\n norm_imgs = [np.float32(r_img) / 255 for r_img in raw_images]\n\n for norm_img, grad_cam in zip(norm_imgs, gradcam):\n avg_gradcam = getAttMap(norm_img, grad_cam[0], blur=True)\n avg_gradcams.append(avg_gradcam)\n\n with torch.no_grad():\n itm_score = torch.nn.functional.softmax(itm_output, dim=1)\n\n itm_scores.append(itm_score)\n\n # ========= ITM re-ranking =========\n itm_scores = torch.cat(itm_scores)[:, 1]\n if itm_ranking:\n itm_scores_sorted, indices = torch.sort(itm_scores, descending=True)\n\n avg_gradcams_sorted = []\n all_raw_images_sorted = []\n for idx in indices:\n avg_gradcams_sorted.append(avg_gradcams[idx])\n all_raw_images_sorted.append(all_raw_images[idx])\n\n avg_gradcams = avg_gradcams_sorted\n all_raw_images = all_raw_images_sorted\n\n if show_gradcam:\n images_to_show = iter(avg_gradcams)\n else:\n images_to_show = iter(all_raw_images)\n\n for _ in range(num_rows):\n with st.container():\n for col in st.columns(num_cols):\n col.image(next(images_to_show), use_column_width=True, clamp=True)\n\n\ndef read_and_process_images(image_paths, vis_processor):\n raw_images = [read_img(path) for path in image_paths]\n images = [vis_processor(r_img) for r_img in raw_images]\n images_tensors = torch.stack(images).to(device)\n\n return raw_images, images_tensors\n\n\ndef compute_gradcam_batch(model, visual_input, text_input, tokenized_text, block_num=6):\n model.text_encoder.base_model.base_model.encoder.layer[\n block_num\n ].crossattention.self.save_attention = True\n\n output = model({\"image\": visual_input, \"text_input\": text_input}, match_head=\"itm\")\n loss = output[:, 1].sum()\n\n model.zero_grad()\n loss.backward()\n with torch.no_grad():\n mask = tokenized_text.attention_mask.view(\n tokenized_text.attention_mask.size(0), 1, -1, 1, 1\n ) # (bsz,1,token_len, 1,1)\n token_length = mask.sum() - 2\n token_length = token_length.cpu()\n # grads and cams [bsz, num_head, seq_len, image_patch]\n grads = model.text_encoder.base_model.base_model.encoder.layer[\n block_num\n ].crossattention.self.get_attn_gradients()\n cams = model.text_encoder.base_model.base_model.encoder.layer[\n block_num\n ].crossattention.self.get_attention_map()\n\n # assume using vit large with 576 num image patch\n cams = cams[:, :, :, 1:].reshape(visual_input.size(0), 12, -1, 24, 24) * mask\n grads = (\n grads[:, :, :, 1:].clamp(0).reshape(visual_input.size(0), 12, -1, 24, 24)\n * mask\n )\n\n gradcam = cams * grads\n # [enc token gradcam, average gradcam across token, gradcam for individual token]\n # gradcam = torch.cat((gradcam[0:1,:], gradcam[1:token_length+1, :].sum(dim=0, keepdim=True)/token_length, gradcam[1:, :]))\n gradcam = gradcam.mean(1).cpu().detach()\n gradcam = (\n gradcam[:, 1 : token_length + 1, :].sum(dim=1, keepdim=True) / token_length\n )\n\n return gradcam, output\n" }, { "path": "app/multipage.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\n\"\"\"\nThis file is the framework for generating multiple Streamlit applications\nthrough an object oriented framework.\n\"\"\"\n\n# Import necessary libraries\nimport streamlit as st\n\n# Define the multipage class to manage the multiple apps in our program\nclass MultiPage:\n \"\"\"Framework for combining multiple streamlit applications.\"\"\"\n\n def __init__(self) -> None:\n \"\"\"Constructor class to generate a list which will store all our applications as an instance variable.\"\"\"\n self.pages = []\n\n def add_page(self, title, func) -> None:\n \"\"\"Class Method to Add pages to the project\n Args:\n title ([str]): The title of page which we are adding to the list of apps\n\n func: Python function to render this page in Streamlit\n \"\"\"\n\n self.pages.append({\"title\": title, \"function\": func})\n\n def run(self):\n # Drodown to select the page to run\n page = st.sidebar.selectbox(\n \"Navigation\", self.pages, format_func=lambda page: page[\"title\"]\n )\n\n # run the app function\n page[\"function\"]()\n" }, { "path": "app/text_localization.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport math\n\nimport numpy as np\nimport streamlit as st\nfrom lavis.models.blip_models.blip_image_text_matching import compute_gradcam\nfrom lavis.processors import load_processor\nfrom PIL import Image\n\nfrom app import device, load_demo_image\nfrom app.utils import getAttMap, init_bert_tokenizer, load_blip_itm_model\n\n\ndef app():\n model_type = st.sidebar.selectbox(\"Model:\", [\"BLIP_base\", \"BLIP_large\"])\n\n values = list(range(1, 12))\n default_layer_num = values.index(7)\n layer_num = (\n st.sidebar.selectbox(\"Layer number\", values, index=default_layer_num) - 1\n )\n\n st.markdown(\n \"

Text Localization

\", unsafe_allow_html=True\n )\n\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n text_processor = load_processor(\"blip_caption\")\n\n tokenizer = init_bert_tokenizer()\n\n instructions = \"Try the provided image and text or use your own ones.\"\n file = st.file_uploader(instructions)\n\n query = st.text_input(\n \"Try a different input.\", \"A girl playing with her dog on the beach.\"\n )\n\n submit_button = st.button(\"Submit\")\n\n col1, col2 = st.columns(2)\n\n if file:\n raw_img = Image.open(file).convert(\"RGB\")\n else:\n raw_img = load_demo_image()\n\n col1.header(\"Image\")\n w, h = raw_img.size\n scaling_factor = 720 / w\n resized_image = raw_img.resize((int(w * scaling_factor), int(h * scaling_factor)))\n col1.image(resized_image, use_column_width=True)\n\n col2.header(\"GradCam\")\n\n if submit_button:\n if model_type.startswith(\"BLIP\"):\n blip_type = model_type.split(\"_\")[1]\n model = load_blip_itm_model(device, model_type=blip_type)\n\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n qry = text_processor(query)\n\n qry_tok = tokenizer(qry, return_tensors=\"pt\").to(device)\n\n norm_img = np.float32(resized_image) / 255\n\n gradcam, _ = compute_gradcam(model, img, qry, qry_tok, block_num=layer_num)\n\n avg_gradcam = getAttMap(norm_img, gradcam[0][1], blur=True)\n col2.image(avg_gradcam, use_column_width=True, clamp=True)\n\n num_cols = 4.0\n num_tokens = len(qry_tok.input_ids[0]) - 2\n\n num_rows = int(math.ceil(num_tokens / num_cols))\n\n gradcam_iter = iter(gradcam[0][2:-1])\n token_id_iter = iter(qry_tok.input_ids[0][1:-1])\n\n for _ in range(num_rows):\n with st.container():\n for col in st.columns(int(num_cols)):\n token_id = next(token_id_iter, None)\n if not token_id:\n break\n gradcam_img = next(gradcam_iter)\n\n word = tokenizer.decode([token_id])\n gradcam_todraw = getAttMap(norm_img, gradcam_img, blur=True)\n\n new_title = (\n '

{}

'.format(\n word\n )\n )\n col.markdown(new_title, unsafe_allow_html=True)\n # st.image(image, channels=\"BGR\")\n col.image(gradcam_todraw, use_column_width=True, clamp=True)\n" }, { "path": "app/utils.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport numpy as np\nimport streamlit as st\nimport torch\nfrom lavis.models import BlipBase, load_model\nfrom matplotlib import pyplot as plt\nfrom PIL import Image\nfrom scipy.ndimage import filters\nfrom skimage import transform as skimage_transform\n\n\ndef resize_img(raw_img):\n w, h = raw_img.size\n scaling_factor = 240 / w\n resized_image = raw_img.resize((int(w * scaling_factor), int(h * scaling_factor)))\n return resized_image\n\n\ndef read_img(filepath):\n raw_image = Image.open(filepath).convert(\"RGB\")\n\n return raw_image\n\n\n@st.cache(\n hash_funcs={\n torch.nn.parameter.Parameter: lambda parameter: parameter.data.detach()\n .cpu()\n .numpy()\n },\n allow_output_mutation=True,\n)\ndef load_model_cache(name, model_type, is_eval, device):\n return load_model(name, model_type, is_eval, device)\n\n\n@st.cache(allow_output_mutation=True)\ndef init_bert_tokenizer():\n tokenizer = BlipBase.init_tokenizer()\n return tokenizer\n\n\ndef getAttMap(img, attMap, blur=True, overlap=True):\n attMap -= attMap.min()\n if attMap.max() > 0:\n attMap /= attMap.max()\n attMap = skimage_transform.resize(attMap, (img.shape[:2]), order=3, mode=\"constant\")\n if blur:\n attMap = filters.gaussian_filter(attMap, 0.02 * max(img.shape[:2]))\n attMap -= attMap.min()\n attMap /= attMap.max()\n cmap = plt.get_cmap(\"jet\")\n attMapV = cmap(attMap)\n attMapV = np.delete(attMapV, 3, 2)\n if overlap:\n attMap = (\n 1 * (1 - attMap**0.7).reshape(attMap.shape + (1,)) * img\n + (attMap**0.7).reshape(attMap.shape + (1,)) * attMapV\n )\n return attMap\n\n\n@st.cache(\n hash_funcs={\n torch.nn.parameter.Parameter: lambda parameter: parameter.data.detach()\n .cpu()\n .numpy()\n },\n allow_output_mutation=True,\n)\ndef load_blip_itm_model(device, model_type=\"base\"):\n model = load_model(\n \"blip_image_text_matching\", model_type, is_eval=True, device=device\n )\n return model\n" }, { "path": "app/vqa.py", "content": "\"\"\"\n # Copyright (c) 2022, salesforce.com, inc.\n # All rights reserved.\n # SPDX-License-Identifier: BSD-3-Clause\n # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\"\"\"\n\nimport streamlit as st\nfrom app import load_demo_image, device\nfrom app.utils import load_model_cache\nfrom lavis.processors import load_processor\nfrom PIL import Image\n\n\ndef app():\n model_type = st.sidebar.selectbox(\"Model:\", [\"BLIP\"])\n\n # ===== layout =====\n st.markdown(\n \"

Visual Question Answering

\",\n unsafe_allow_html=True,\n )\n\n instructions = \"\"\"Try the provided image or upload your own:\"\"\"\n file = st.file_uploader(instructions)\n\n col1, col2 = st.columns(2)\n\n col1.header(\"Image\")\n if file:\n raw_img = Image.open(file).convert(\"RGB\")\n else:\n raw_img = load_demo_image()\n\n w, h = raw_img.size\n scaling_factor = 720 / w\n resized_image = raw_img.resize((int(w * scaling_factor), int(h * scaling_factor)))\n\n col1.image(resized_image, use_column_width=True)\n col2.header(\"Question\")\n\n user_question = col2.text_input(\"Input your question!\", \"What are objects there?\")\n qa_button = st.button(\"Submit\")\n\n col2.header(\"Answer\")\n\n # ===== event =====\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=480)\n text_processor = load_processor(\"blip_question\").build()\n\n if qa_button:\n if model_type.startswith(\"BLIP\"):\n model = load_model_cache(\n \"blip_vqa\", model_type=\"vqav2\", is_eval=True, device=device\n )\n\n img = vis_processor(raw_img).unsqueeze(0).to(device)\n question = text_processor(user_question)\n\n vqa_samples = {\"image\": img, \"text_input\": [question]}\n answers = model.predict_answers(vqa_samples, inference_method=\"generate\")\n\n col2.write(\"\\n\".join(answers), use_column_width=True)\n" }, { "path": "app.py", "content": "import gradio as gr\nimport os\nimport torch\nfrom torchvision import transforms\nfrom lavis.processors import transforms_video\nfrom lavis.datasets.data_utils import load_video_demo\nfrom lavis.processors.blip_processors import ToUint8, ToTHWC\nfrom lavis.models.sevila_models.sevila import SeViLA\nfrom typing import Optional\nimport warnings\n# model config\nimg_size = 224\nnum_query_token = 32\nt5_model = 'google/flan-t5-xl'\ndrop_path_rate = 0\nuse_grad_checkpoint = False\nvit_precision = \"fp16\"\nfreeze_vit = True\nprompt = ''\nmax_txt_len = 77\nanswer_num = 5\napply_lemmatizer = False\ntask = 'freeze_loc_freeze_qa_vid'\n\n# prompt\nLOC_propmpt = 'Does the information within the frame provide the necessary details to accurately answer the given question?'\nQA_prompt = 'Considering the information presented in the frame, select the correct answer from the options.'\n\n# processors config\nmean = (0.48145466, 0.4578275, 0.40821073)\nstd = (0.26862954, 0.26130258, 0.27577711)\nnormalize = transforms.Normalize(mean, std)\nimage_size = img_size\ntransform = transforms.Compose([ToUint8(), ToTHWC(), transforms_video.ToTensorVideo(), normalize])\n\nprint('Model Loading \\nLoading the SeViLA model can take a few minutes (typically 2-3).')\nsevila = SeViLA(\n img_size=img_size,\n drop_path_rate=drop_path_rate,\n use_grad_checkpoint=use_grad_checkpoint,\n vit_precision=vit_precision,\n freeze_vit=freeze_vit,\n num_query_token=num_query_token,\n t5_model=t5_model,\n prompt=prompt,\n max_txt_len=max_txt_len,\n apply_lemmatizer=apply_lemmatizer,\n frame_num=4,\n answer_num=answer_num,\n task=task,\n )\n\nsevila.load_checkpoint(url_or_filename='https://huggingface.co/Shoubin/SeViLA/resolve/main/sevila_pretrained.pth')\nprint('Model Loaded')\n\nANS_MAPPING = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D', 4 : 'E'}\n\n# os.mkdir('video')\n\ndef sevila_demo(video, \n question, \n option1, option2, option3, \n video_frame_num, \n keyframe_num):\n \n if torch.cuda.is_available():\n device = 0\n else:\n device = 'cpu'\n \n global sevila \n if device == \"cpu\":\n sevila = sevila.float()\n else:\n sevila = sevila.to(int(device))\n \n vpath = video \n raw_clip, indice, fps, vlen = load_video_demo(\n video_path=vpath,\n n_frms=int(video_frame_num),\n height=image_size,\n width=image_size,\n sampling=\"uniform\",\n clip_proposal=None\n )\n clip = transform(raw_clip.permute(1,0,2,3))\n clip = clip.float().to(int(device))\n clip = clip.unsqueeze(0)\n # check\n if option1[-1] != '.':\n option1 += '.'\n if option2[-1] != '.':\n option2 += '.' \n if option3[-1] != '.':\n option3 += '.'\n option_dict = {0:option1, 1:option2, 2:option3}\n options = 'Option A:{} Option B:{} Option C:{}'.format(option1, option2, option3)\n text_input_qa = 'Question: ' + question + ' ' + options + ' ' + QA_prompt\n text_input_loc = 'Question: ' + question + ' ' + options + ' ' + LOC_propmpt\n \n out = sevila.generate_demo(clip, text_input_qa, text_input_loc, int(keyframe_num))\n # print(out)\n answer_id = out['output_text'][0]\n answer = option_dict[answer_id]\n select_index = out['frame_idx'][0]\n # images = [] \n keyframes = []\n timestamps =[]\n \n # print('raw_clip', len(raw_clip))\n # for j in range(int(video_frame_num)):\n # image = raw_clip[:, j, :, :].int()\n # image = image.permute(1, 2, 0).numpy() \n # images.append(image)\n \n video_len = vlen/fps # seconds\n \n for i in select_index:\n image = raw_clip[:, i, :, :].int()\n image = image.permute(1, 2, 0).numpy() \n keyframes.append(image)\n select_i = indice[i]\n time = round((select_i / vlen) * video_len, 2)\n timestamps.append(str(time)+'s')\n \n gr.components.Gallery(keyframes)\n #gr.components.Gallery(images)\n timestamps_des = ''\n for i in range(len(select_index)):\n timestamps_des += 'Keyframe {}: {} \\n'.format(str(i+1), timestamps[i])\n \n return keyframes, timestamps_des, answer\n\nwith gr.Blocks(title=\"SeViLA demo\") as demo:\n description = \"\"\"

\n Self-Chained Image-Language Model for Video Localization and Question Answering\n
\n \n Shoubin Yu, \n Jaemin Cho, \n Prateek Yadav, \n Mohit Bansal\n \n
\n \n [GitHub]\n [Paper]\n \n

\n

\n To locate keyframes in a video and answer question, please:\n
\n (1) upolad your video; (2) write your question/options and set # video frame/# keyframe; (3) click Locate and Answer!\n
\n Just a heads up - loading the SeViLA model can take a few minutes (typically 2-3), and running examples requires about 12GB of memory.\n
\n We've got you covered! We've provided some example videos and questions below to help you get started. Feel free to try out SeViLA with these!\n

\n \"\"\"\n gr.HTML(description)\n with gr.Row():\n with gr.Column(scale=1, min_width=600):\n video = gr.Video(label='Video') \n question = gr.Textbox(placeholder=\"Why did the two ladies put their hands above their eyes while staring out?\", label='Question')\n with gr.Row():\n option1 = gr.Textbox(placeholder=\"practicing cheer\", label='Option 1')\n option2 = gr.Textbox(placeholder=\"posing for photo\", label='Option 2')\n option3 = gr.Textbox(placeholder=\"to see better\", label='Option 3')\n with gr.Row():\n video_frame_num = gr.Textbox(placeholder=32, label='# Video Frame')\n keyframe_num = gr.Textbox(placeholder=4, label='# Keyframe') \n # device = gr.Textbox(placeholder=0, label='Device') \n gen_btn = gr.Button(value='Locate and Answer!')\n with gr.Column(scale=1, min_width=600): \n keyframes = gr.Gallery(\n label=\"Keyframes\", show_label=False, elem_id=\"gallery\",\n ).style(columns=[4], rows=[1], object_fit=\"contain\", max_width=100, max_height=100)\n #keyframes = gr.Gallery(label='Keyframes')\n timestamps = gr.outputs.Textbox(label=\"Keyframe Timestamps\")\n answer = gr.outputs.Textbox(label=\"Output Answer\")\n \n gen_btn.click(\n sevila_demo,\n inputs=[video, question, option1, option2, option3, video_frame_num, keyframe_num],\n outputs=[keyframes, timestamps, answer],\n queue=True\n )\n #demo = gr.Interface(sevila_demo,\n # inputs=[gr.Video(), question, option1, option2, option3, video_frame_num, keyframe_num, device],\n # outputs=['gallery', timestamps, answer],\n # examples=[['videos/demo1.mp4', 'Why did the two ladies put their hands above their eyes while staring out?', 'practicing cheer.', 'play ball.', 'to see better.', 32, 4, 0],\n # ['videos/demo2.mp4', 'What did both of them do after completing skiing?', 'jump and pose.' , 'bend down.','raised their hands.', 32, 4, 0],\n # ['videos/demo3.mp4', 'What room was Wilson breaking into when House found him?', 'the kitchen.' , 'the dining room.','the bathroom.', 32, 4, 0]]\n # )\n with gr.Column():\n gr.Examples(\n inputs=[video, question, option1, option2, option3, video_frame_num, keyframe_num],\n outputs=[keyframes, timestamps, answer],\n fn=sevila_demo,\n examples=[['videos/demo1.mp4', 'Why did the two ladies put their hands above their eyes while staring out?', 'practicing cheer', 'to place wreaths', 'to see better', 32, 4],\n ['videos/demo2.mp4', 'What did both of them do after completing skiing?', 'jump and pose' , 'bend down','raised their hands', 32, 4],\n ['videos/demo3.mp4', 'What room was Wilson breaking into when House found him?', 'the bedroom' , 'the bathroom','the kitchen', 32, 4]],\n cache_examples=False,\n )\ndemo.queue(concurrency_count=1, api_open=False) \ndemo.launch(share=False) " }, { "path": "docs/Makefile", "content": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS ?=\nSPHINXBUILD ?= sphinx-build\nSOURCEDIR = source\nBUILDDIR = build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option. $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n" }, { "path": "docs/benchmark.rst", "content": "Benchmark\n############\n\nWe provide scripts for evaluating and training models on task datasets. The following benchmark results are included for reference.\n\n\nALBEF\n*******\n.. list-table::\n :widths: 30 80 20\n\n * - **Pretraining**\n - COCO (`download `__)\n - `script `__\n * -\n - Visual Genome (`download `__)\n -\n * -\n - SBU (`download `__)\n -\n * -\n - CC3M (`download `__)\n -\n * -\n - CC12M (`download `__)\n -\n\n.. list-table::\n :widths: 30 40 20 20 20 30 30\n :header-rows: 1\n\n * -\n - **Retrieval**\n - **R1**\n - **R5**\n - **R10**\n - **Training**\n - **Evaluation**\n * - TR\n - COCO (`download `__)\n - 77.6\n - 94.1\n - 97.2\n - `script `__\n - `script `__\n * - IR\n - COCO (`download `__)\n - 61.0\n - 84.5\n - 90.7\n - `script `__\n - `script `__\n * - TR\n - Flickr30k (`download `__)\n - 77.6\n - 94.1\n - 97.2\n - `script `__\n - `script `__\n * - IR\n - Flickr30k (`download `__)\n - 61.0\n - 84.5\n - 90.7\n - `script `__\n - `script `__\n\n\n.. list-table::\n :widths: 20 20 20 20 20\n :header-rows: 1\n\n * - **VQA**\n - **test-dev**\n - **test-std/test**\n - **Training**\n - **Evaluation**\n * - VQAv2 (`download `__)\n - 76.35\n - 76.54\n - `script `__\n - `script `__\n * - OKVQA (`download `__)\n - NA\n - 54.7 \n - `script `__\n - NA\n * - AOKVQA (`download `__)\n - 54.5\n - NA\n - `script `__\n - NA\n\n \n.. list-table::\n :widths: 20 20 20 20 20\n :header-rows: 1\n\n * - **Multimodal Classification**\n - **val**\n - **test**\n - **Training**\n - **Evaluation**\n * - SNLI-VE (`download `__)\n - 80.60\n - 81.04\n - `script `__\n - `script `__\n * - NLVR2 (`download `__)\n - 82.47 \n - 82.91 \n - `script `__\n - `script `__\n \nBLIP\n*******\n.. list-table::\n :widths: 30 80 20\n\n * - **Pretraining (14M)**\n - COCO (`download `__)\n - `script `__\n * -\n - Visual Genome (`download `__)\n -\n * -\n - SBU (`download `__)\n -\n * -\n - CC3M (`download `__)\n -\n * -\n - CC12M (`download `__)\n -\n\n.. list-table::\n :widths: 30 40 20 20 20 30 30\n :header-rows: 1\n\n * - **Tasks**\n - **Retrieval**\n - **R1**\n - **R5**\n - **R10**\n - **Training**\n - **Evaluation**\n * - TR\n - COCO (`download `__)\n - 82.0\n - 95.8\n - 98.1\n - `script `__\n - `script `__\n * - IR\n - COCO (`download `__)\n - 64.5\n - 86.0\n - 91.7\n - `script `__\n - `script `__\n * - TR\n - Flickr30k (`download `__)\n - 96.9\n - 99.9\n - 100.0\n - `script `__\n - `script `__\n * - IR\n - Flickr30k (`download `__)\n - 87.5\n - 97.6\n - 98.9\n - `script `__\n - `script `__\n\n\n.. list-table::\n :widths: 20 20 20 20 20\n :header-rows: 1\n\n * - **VQA**\n - **test-dev**\n - **test-std/test**\n - **Training**\n - **Evaluation**\n * - VQAv2 (`download `__)\n - 78.23\n - 78.29\n - `script `__\n - `script `__\n * - OKVQA (`download `__)\n - NA\n - 55.4 \n - `script `__\n - `script `__\n * - AOKVQA (`download `__)\n - 56.2\n - 50.1 \n - `script `__\n - `script `__\n\n\n.. list-table::\n :widths: 20 20 20 20 20 20\n :header-rows: 1\n\n * - **Image Captioning**\n - **BLEU@4**\n - **CIDEr**\n - **SPICE**\n - **Training**\n - **Evaluation**\n * - COCO (`download `__)\n - 39.9\n - 133.5\n - 23.7\n - `script `__\n - `script `__\n * - NoCaps (`download `__)\n - 31.9\n - 109.1\n - 14.7\n - NA\n - `script `__\n\n\n.. list-table::\n :widths: 20 20 20 20 20\n :header-rows: 1\n\n * - **Multimodal Classification**\n - **val**\n - **test**\n - **Training**\n - **Evaluation**\n * - NLVR2 (`download `__)\n - 82.48\n - 83.25\n - `script `__\n - `script `__\n\nCLIP\n*******\n.. list-table::\n :widths: 30 40 20 20 20 30\n :header-rows: 1\n\n * - **Tasks**\n - **Retrieval (Zero-shot)**\n - **R1**\n - **R5**\n - **R10**\n - **Evaluation**\n * - TR\n - COCO (`download `__)\n - 57.2\n - 80.5\n - 87.8\n - `script `__\n * - IR\n - COCO (`download `__)\n - 36.5\n - 60.8\n - 71.0\n - `script `__\n * - TR\n - Flickr30k (`download `__)\n - 86.5\n - 98.0\n - 99.1\n - `script `__\n * - IR\n - Flickr30k (`download `__)\n - 67.0\n - 88.9\n - 93.3\n - `script `__\n\n.. list-table::\n :widths: 20 20 20\n :header-rows: 1\n\n * - **Multimodal Classification**\n - **val**\n - **Evaluation**\n * - ImageNet \n - 76.5 \n - `script `__\n\n\nALPRO\n*******\n.. list-table::\n :widths: 30 40 20 20 20 20 30\n :header-rows: 1\n\n * - **Tasks**\n - **Retrieval**\n - **R1**\n - **R5**\n - **R10**\n - **Training**\n - **Evaluation**\n * - TR\n - MSRVTT (`download `__)\n - 33.2\n - 60.5 \n - 71.7 \n - `script `__\n - `script `__\n * - VR\n - MSRVTT (`download `__)\n - 33.8\n - 61.4\n - 72.7\n - `script `__\n - `script `__\n * - TR\n - DiDeMo (`download `__)\n - 38.8 \n - 66.4\n - 76.8\n - `script `__\n - `script `__\n * - VR\n - DiDeMo (`download `__)\n - 36.6\n - 67.5\n - 77.9\n - `script `__\n - `script `__\n\n.. list-table::\n :widths: 20 20 20 20\n :header-rows: 1\n\n * - **Video QA**\n - **test**\n - **Training**\n - **Evaluation**\n * - MSRVTT \n - 42.1 \n - `script `__\n - `script `__\n * - MSVD \n - 46.0 \n - `script `__ \n - `script `__" }, { "path": "docs/build_docs.sh", "content": "#!/bin/bash\nset -euo pipefail\n\n# Change to root directory of repo\nDIRNAME=$(cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd)\ncd \"${DIRNAME}/..\"\n\n# # Set up virtual environment\npip3 install setuptools wheel virtualenv\nif [ ! -d venv ]; then\n rm -f venv\n virtualenv venv\nfi\nsource venv/bin/activate\n\n# # Get current git branch & stash unsaved changes\nGIT_BRANCH=$(git branch --show-current)\nif [ -z \"${GIT_BRANCH}\" ]; then\n GIT_BRANCH=\"main\"\nfi\ngit stash\n\n# Set up exit handler to restore git state & delete temp branches\n# function exit_handler {\n# git reset --hard\n# git checkout \"${GIT_BRANCH}\" --\n# git stash pop || true\n# for version in $(git tag --list 'v[0-9]*'); do\n# branch=\"${version}_local_docs_only\"\n# if git show-ref --verify --quiet \"refs/heads/$branch\"; then\n# git branch -D \"$branch\"\n# fi\n# done\n# }\n# trap exit_handler EXIT\n\n# Clean up build directory and install Sphinx requirements\npip3 install -r \"${DIRNAME}/requirements.txt\"\nsphinx-build -M clean \"${DIRNAME}\" \"${DIRNAME}/_build\"\n\n# Build API docs for current head\nexport current_version=\"latest\"\npip3 install \".\"\nsphinx-build -b html \"${DIRNAME}\" \"${DIRNAME}/_build/html/${current_version}\" -W --keep-going\nrm -rf \"${DIRNAME}/_build/html/${current_version}/.doctrees\"\n#pip3 uninstall -y omnixai\n\n# Install all previous released versions\n# and use them to build the appropriate API docs.\n# Uninstall after we're done with each one.\n# versions=()\n# checkout_files=(\"${DIRNAME}/*.rst\" \"lavis\" \"tutorials\" \"setup.py\")\n# for version in $(git tag --list 'v[0-9]*'); do\n# versions+=(\"$version\")\n# git checkout -b \"${version}_local_docs_only\"\n# for f in $(git diff --name-only --diff-filter=A \"tags/${version}\" \"${DIRNAME}/*.rst\"); do\n# git rm \"$f\"\n# done\n# git checkout \"tags/${version}\" -- \"${checkout_files[@]}\"\n# export current_version=${version}\n# pip3 install \".[all]\"\n# sphinx-build -b html \"${DIRNAME}\" \"${DIRNAME}/_build/html/${current_version}\" -W --keep-going\n# rm -rf \"${DIRNAME}/_build/html/${current_version}/.doctrees\"\n# #pip3 uninstall -y omnixai\n# git reset --hard\n# git checkout \"${GIT_BRANCH}\" --\n# done\n\n# Determine the latest stable version if there is one\n# if (( ${#versions[@]} > 0 )); then\n# stable_hash=$(git rev-list --tags --max-count=1)\n# stable_version=$(git describe --tags \"$stable_hash\")\n# export stable_version\n# else\nexport stable_version=\"latest\"\n# fi\n\n# Create dummy HTML's for the stable version in the base directory\nwhile read -r filename; do\n filename=$(echo \"$filename\" | sed \"s/\\.\\///\")\n n_sub=$(echo \"$filename\" | (grep -o \"/\" || true) | wc -l)\n prefix=\"\"\n for (( i=0; i \"${DIRNAME}/_build/html/$filename\" <\n\n \n LAVIS Documentation\n \n \n \n

Please wait while you're redirected to our documentation.

\n \n\nEOF\ndone < <(cd \"${DIRNAME}/_build/html/$stable_version\" && find . -name \"*.html\")\necho \"Finished writing to _build/html.\"" }, { "path": "docs/conf.py", "content": "# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common options. For a full\n# list see the documentation:\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\n\n# -- Path setup --------------------------------------------------------------\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#\n# import os\n# import sys\n# sys.path.insert(0, os.path.abspath('.'))\n\n\n# -- Project information -----------------------------------------------------\n\nproject = \"LAVIS\"\ncopyright = \"2022, salesforce.com inc.\"\nauthor = (\n \"Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, Steven C.H. Hoi\"\n)\n\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\"nbsphinx\"]\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\"_templates\"]\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = []\n\n\n# -- Options for HTML output -------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\n#\n# html_theme = \"alabaster\"\nhtml_theme = \"sphinx_rtd_theme\"\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = [\"_static\"]\n\n# pygments_style = \"sphinx\"\n" }, { "path": "docs/getting_started.rst", "content": "Dataset Zoo\n##################\nLAVIS inherently supports a wide variety of common language-vision datasets by providing automatic download scripts to help download and organize these datasets; \nand implements PyTorch datasets for these datasets. To view supported datasets, use the following code:\n\n.. code-block:: python\n\n from lavis.datasets.builders import dataset_zoo\n dataset_names = dataset_zoo.get_names()\n print(dataset_names)\n # ['aok_vqa', 'coco_caption', 'coco_retrieval', 'coco_vqa', 'conceptual_caption_12m',\n # 'conceptual_caption_3m', 'didemo_retrieval', 'flickr30k', 'imagenet', 'laion2B_multi',\n # 'msrvtt_caption', 'msrvtt_qa', 'msrvtt_retrieval', 'msvd_caption', 'msvd_qa', 'nlvr',\n # 'nocaps', 'ok_vqa', 'sbu_caption', 'snli_ve', 'vatex_caption', 'vg_caption', 'vg_vqa']\n print(len(dataset_names))\n # 23\n\n\nAuto-Downloading and Loading Datasets\n######################################\nWe now take COCO caption dataset as an example to demonstrate how to download and prepare the dataset.\n\nIn ``lavis/datasets/download_scripts/``, we provide tools to download most common public language-vision datasets supported by LAVIS.\nThe COCO caption dataset uses images from COCO dataset. Therefore, we first download COCO images via:\n\n.. code-block:: bash\n \n cd lavis/datasets/download_scripts/ && python download_coco.py\n\nThis will automatically download and extract COCO images to the default LAVIS cache location.\nThe default cache location is ``~/.cache/lavis``, defined in ``lavis/configs/default.yaml``.\n\nAfter downloading the images, we can use ``load_dataset()`` to obtain the dataset. On the first run, this will automatically download and cache annotation files.\n\n.. code-block:: python\n\n from lavis.datasets.builders import load_dataset\n coco_dataset = load_dataset(\"coco_caption\")\n\n print(coco_dataset.keys())\n # dict_keys(['train', 'val', 'test'])\n\n print(len(coco_dataset[\"train\"]))\n # 566747\n\n print(coco_dataset[\"train\"][0])\n # {'image': ,\n # 'text_input': 'A woman wearing a net on her head cutting a cake. ',\n # 'image_id': 0}\n\nIf you already host a local copy of the dataset, you can pass in the ``vis_path`` argument to change the default location to load images.\n\n.. code-block:: python\n\n coco_dataset = load_dataset(\"coco_caption\", vis_path=YOUR_LOCAL_PATH)\n\n\nModel Zoo\n####################################\nLAVIS supports a growing list of pre-trained models for different tasks,\ndatatsets and of varying sizes. Let's get started by viewing the supported models.\n\n.. code-block:: python\n\n from lavis.models import model_zoo\n print(model_zoo)\n # ==================================================\n # Architectures Types\n # ==================================================\n # albef_classification base, ve\n # albef_nlvr base\n # albef_pretrain base\n # albef_retrieval base, coco, flickr\n # albef_vqa base, vqav2\n # alpro_qa base, msrvtt, msvd\n # alpro_retrieval base, msrvtt, didemo\n # blip_caption base, base_coco, large, large_coco\n # blip_classification base\n # blip_feature_extractor base\n # blip_nlvr base\n # blip_pretrain base\n # blip_retrieval base, coco, flickr\n # blip_vqa base, vqav2\n # clip ViT-B-32, ViT-B-16, ViT-L-14, ViT-L-14-336, RN50\n\n # show total number of support model variants\n len(model_zoo)\n # 33\n\n\nInference with Pre-trained Models\n####################################\n\nNow let's see how to use models in LAVIS to perform inference on example data. We first\nload a sample image from local.\n\n.. code-block:: python\n\n from PIL import Image\n\n # setup device to use\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n # load sample image\n raw_image = Image.open(\"docs/_static/merlion.png\").convert(\"RGB\")\n\nThis example image shows `Merlion park `_ (`image credit `_), a landmark in Singapore.\n\n.. image:: _static/merlion.png\n\nImage Captioning\n*******************************\nWe now use the BLIP model to generate a caption for the image. To make inference even easier, we also associate each\npre-trained model with its preprocessors (transforms), we use ``load_model_and_preprocess()`` with the following arguments:\n\n- ``name``: The name of the model to load. This could be a pre-trained model, task model, or feature extractor. See ``model_zoo`` for a full list of model names.\n- ``model_type``: Each architecture has variants trained on different datasets and at different scale. See Types column in ``model_zoo`` for a full list of model types.\n- ``is_eval``: if `True`, set the model to evaluation mode. This is desired for inference or feature extraction.\n- ``devce``: device to load the model to.\n\n.. code-block:: python\n\n from lavis.models import load_model_and_preprocess\n # loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset.\n # this also loads the associated image processors\n model, vis_processors, _ = load_model_and_preprocess(name=\"blip_caption\", model_type=\"base_coco\", is_eval=True, device=device)\n\n # preprocess the image\n # vis_processors stores image transforms for \"train\" and \"eval\" (validation / testing / inference)\n image = vis_processors[\"eval\"](raw_image).unsqueeze(0).to(device)\n\n # generate caption\n model.generate({\"image\": image})\n # ['a large fountain spewing water into the air']\n\n\nYou may also load models and their preprocessors separately via ``load_model()`` and ``load_processor()``.\nIn BLIP, you can also generate diverse captions by turning nucleus sampling on.\n\n.. code-block:: python\n\n from lavis.processors import load_processor\n from lavis.models import load_model\n\n # load image preprocesser used for BLIP\n vis_processor = load_processor(\"blip_image_eval\").build(image_size=384)\n model = load_model(name=\"blip_caption\", model_type=\"base_coco\", is_eval=True, device=device)\n\n image = vis_processor(image).unsqueeze(0).to(device)\n model.generate({\"image\": raw_image}, use_nucleus_sampling=True)\n # one generated random sample: ['some very pretty buildings and some water jets']\n\n\nVisual question answering (VQA)\n*******************************\nBLIP model is able to answer free-form questions about images in natural language.\nTo access the VQA model, simply replace the ``name`` and ``model_type`` arguments \npassed to ``load_model_and_preprocess()``.\n\n.. code-block:: python\n\n from lavis.models import load_model_and_preprocess\n model, vis_processors, txt_processors = load_model_and_preprocess(name=\"blip_vqa\", model_type=\"vqav2\", is_eval=True, device=device)\n\n # ask a random question.\n question = \"Which city is this photo taken?\"\n \n image = vis_processors[\"eval\"](raw_image).unsqueeze(0).to(device)\n question = txt_processors[\"eval\"](question)\n\n model.predict_answers(samples={\"image\": image, \"text_input\": question}, inference_method=\"generate\")\n # ['singapore']\n\n\nUnified Feature Extraction Interface\n####################################\n\nLAVIS provides a unified interface to extract multimodal features from each architecture.\nTo extract features, we load the feature extractor variants of each model.\nThe multimodal feature can be used for multimodal classification. The low-dimensional unimodal features can be used to compute cross-modal similarity.\n\n.. code-block:: python\n\n from lavis.models import load_model_and_preprocess \n \n model, vis_processors, txt_processors = load_model_and_preprocess(name=\"blip_feature_extractor\", model_type=\"base\", is_eval=True, device=device)\n caption = \"a large fountain spewing water into the air\"\n\n image = vis_processors[\"eval\"](raw_image).unsqueeze(0).to(device)\n text_input = txt_processors[\"eval\"](caption)\n\n sample = {\"image\": image, \"text_input\": [text_input]}\n\n features_multimodal = model.extract_features(sample)\n print(features_multimodal.keys())\n # odict_keys(['image_embeds', 'multimodal_embeds'])\n print(features_multimodal.multimodal_embeds.shape)\n # torch.Size([1, 12, 768]), use features_multimodal[:, 0, :] for multimodal classification tasks\n\n features_image = model.extract_features(sample, mode=\"image\")\n print(features_image.keys())\n # odict_keys(['image_embeds', 'image_embeds_proj'])\n print(features_image.image_embeds.shape)\n # torch.Size([1, 197, 768])\n print(features_image.image_embeds_proj.shape)\n # torch.Size([1, 197, 256])\n\n features_text = model.extract_features(sample, mode=\"text\")\n print(features_text.keys())\n # odict_keys(['text_embeds', 'text_embeds_proj'])\n print(features_text.text_embeds.shape)\n # torch.Size([1, 12, 768])\n print(features_text.text_embeds_proj.shape)\n # torch.Size([1, 12, 256])\n \n similarity = features_image.image_embeds_proj[:, 0, :] @ features_text.text_embeds_proj[:, 0, :].t()\n print(similarity)\n # tensor([[0.2622]])\n\nSince LAVIS supports a unified feature extraction interface, minimal changes are necessary to use a different model as feature extractor. For example,\nto use ALBEF as the feature extractor, one only needs to change the following line:\n\n.. code-block:: python\n\n model, vis_processors, txt_processors = load_model_and_preprocess(name=\"albef_feature_extractor\", model_type=\"base\", is_eval=True, device=device)\n\nSimilarly, to use CLIP as feature extractor: \n\n.. code-block:: python\n\n model, vis_processors, txt_processors = load_model_and_preprocess(name=\"clip_feature_extractor\", model_type=\"base\", is_eval=True, device=device)\n # model, vis_processors, txt_processors = load_model_and_preprocess(name=\"clip_feature_extractor\", model_type=\"RN50\", is_eval=True, device=device)\n # model, vis_processors, txt_processors = load_model_and_preprocess(name=\"clip_feature_extractor\", model_type=\"ViT-L-14\", is_eval=True, device=device)\n" }, { "path": "docs/index.rst", "content": ".. LAVIS documentation master file, created by\n sphinx-quickstart on Sun Jul 31 10:32:27 2022.\n You can adapt this file completely to your liking, but it should at least\n contain the root `toctree` directive.\n\nWelcome to LAVIS's documentation!\n=================================\n\n.. toctree::\n :maxdepth: 1\n :caption: Introduction\n\n intro\n\n\n.. toctree::\n :maxdepth: 1\n :caption: Getting Started\n\n getting_started\n\n\n.. :maxdepth: 1\n.. :caption: Advanced Training\n\n.. advanced_training\n\n\n.. toctree::\n :maxdepth: 2\n :caption: Advanced Usage\n\n benchmark\n tutorial\n\n\n.. Documentations\n.. ===================\n\n\nIndices and tables\n==================\n\n* :ref:`genindex`\n* :ref:`modindex`\n* :ref:`search`\n" }, { "path": "docs/intro.rst", "content": "What is LAVIS?\n####################################\n\nLAVIS is a Python deep learning library for LAnguage-and-VISion research and applications.\nIt features a unified design to access state-of-the-art foundation language-vision models (`ALBEF `_,\n`BLIP `_, `ALPRO `_, `CLIP `_), common tasks \n(retrieval, captioning, visual question answering, multimodal classification etc.) and datasets (COCO, Flickr, Nocaps, Conceptual\nCommons, SBU, etc.).\n\nThis library aims to provide engineers and researchers with a one-stop solution to rapidly develop models for their specific multimodal\nscenarios, and benchmark them across standard and customized datasets. \n\nKey features of LAVIS include:\n\n- **Modular and Extensible Library Design**: facilitating to easily utilize and repurpose existing modules (datasets, models, preprocessors), also to add new modules.\n\n- **Easy Off-the-shelf Inference and Feature Extraction**: readily available pre-trained models let you take advantage of state-of-the-art multimodal understanding and generation capabilities on your own data.\n\n- **Reproducible Model Zoo**: provided training/pre-training recipies to easily replicate and extend state-of-the-art models.\n\n- **Dataset Zoo and Automatic Downloading Tools**: it can be a hassle to prepare the many language-vision datasets. LAVIS provides automatic downloaing scripts to help prepare a large variety of datasets and their annotations.\n\nOther features include:\n\n- **Distributed Training** using multiple GPUs on one machine or across multiple machines.\n\n- **Web Demo**: try supported models on your own pictures, questions etc.\n\n- **Leaderboard**: comparing state-of-the-art models across standard datasets. \n\n- **Dataset Explorer**: help browse and understand language-vision datasets.\n\nSupported Tasks, Models and Datasets\n####################################\n\nThe following table shows the supported models and language-vision tasks by LAVIS. Adapting existing models to more tasks is possible and next to come in future releases.\n\n======================================== =========================== ============================================= ============ \nTasks Supported Models Supported Datasets Modalities \n======================================== =========================== ============================================= ============ \nImage-text Pre-training ALBEF, BLIP COCO, VisualGenome, SBU, ConceptualCaptions image, text \nImage-text Retrieval ALBEF, BLIP, CLIP COCO, Flickr30k image, text \nText-image Retrieval ALBEF, BLIP, CLIP COCO, Flickr30k image, text \nVisual Question Answering ALBEF, BLIP VQAv2, OKVQA, A-OKVQA image, text \nImage Captioning BLIP COCO, NoCaps image, text \nImage Classification CLIP ImageNet image \nNatural Language Visual Reasoning (NLVR) ALBEF, BLIP NLVR2 image, text \nVisual Entailment (VE) ALBEF SNLI-VE image, text \nVisual Dialogue BLIP VisDial image, text \nVideo-text Retrieval BLIP, ALPRO MSRVTT, DiDeMo video, text \nText-video Retrieval BLIP, ALPRO MSRVTT, DiDeMo video, text \nVideo Question Answering (VideoQA) BLIP, ALPRO MSRVTT, MSVD video, text \nVideo Dialogue VGD-GPT AVSD video, text \nMultimodal Feature Extraction ALBEF, CLIP, BLIP, ALPRO customized image, text \n======================================== =========================== ============================================= ============ \n\nLibrary Design\n####################################\n\n.. image:: _static/architecture.png\n :width: 550\n\nLAVIS has six key modules.\n\n- ``lavis.runners`` manages the overall training and evaluation lifecycle. It is also responsible for creating required components lazily as per demand, such as optimizers, learning rate schedulers and dataloaders. Currently ``RunnerBase`` implements epoch-based training and ``RunerIters`` implements iteration-based training.\n- ``lavis.tasks`` implements concrete training and evaluation logic per task. A task could be, for example, retrieval, captioning, pre-training. The rationale to have an abstraction of task is to accomodate task-specific training and evaluation. For example, evaluating a retrieval model is different from a classification model.\n- ``lavis.datasets`` is responsible for creating datasets, where ``lavis.datasets.builders`` loads dataset configurations, downloads annotations and returns a dataset object; ``lavis.datasets.datasets`` defines the supported datasets, each is a ``torch.utils.data.Dataset`` instance. We also provide `automatic dataset downloading tools` in ``datasets/download_scripts`` to help prepare common public datasets.\n- ``lavis.models`` holds definition for the supported models and shared model layers.\n- ``lavis.processors`` handles preprocessing of text and images/videos before feeding the model. For images and videos, a processor can be thought as transfroms in torchvision; for text input, this may include lowering case, truncation etc.\n- ``lavis.common`` module contains shared classes and methods used by multiple other modules. For example,\n\n - ``lavis.common.config`` contains classes to store and manipulate configuration files used by LAVIS. In particular, we use a hierarchical configuration design, to allow highly customizable training and evaluation.\n - ``lavis.common.registry`` serves as a centralized place to manage modules that share the same functionalities. It allows building datasets, models, tasks, and learning rate schedulers during runtime, by specifying their names as string in the configuration file.\n - ``lavis.common.optims`` contains definitions of learning rate schedulers.\n - ``lavis.common.dist_utils`` contains utilities for distributed training and evaluation.\n - ``lavis.common.utils`` contains miscellaneous utilities, mostly IO-related helper functions.\n\n\nInstallation\n############\n1. (Optional) Creating conda environment\n\n.. code-block:: bash\n\n conda create -n lavis python=3.8\n conda activate lavis\n\n2. Cloning and building from source\n\n.. code-block:: bash\n\n git clone https://github.com/salesforce/LAVIS.git\n cd LAVIS\n pip install .\n\nIf you would like to develop on LAVIS, you may find it easier to build with editable mode::\n\n pip install -e .\n\n" }, { "path": "docs/make.bat", "content": "@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sphinx-build\r\n)\r\nset SOURCEDIR=source\r\nset BUILDDIR=build\r\n\r\nif \"%1\" == \"\" goto help\r\n\r\n%SPHINXBUILD% >NUL 2>NUL\r\nif errorlevel 9009 (\r\n\techo.\r\n\techo.The 'sphinx-build' command was not found. Make sure you have Sphinx\r\n\techo.installed, then set the SPHINXBUILD environment variable to point\r\n\techo.to the full path of the 'sphinx-build' executable. Alternatively you\r\n\techo.may add the Sphinx directory to PATH.\r\n\techo.\r\n\techo.If you don't have Sphinx installed, grab it from\r\n\techo.http://sphinx-doc.org/\r\n\texit /b 1\r\n)\r\n\r\n%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\ngoto end\r\n\r\n:help\r\n%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\n\r\n:end\r\npopd\r\n" }, { "path": "docs/requirements.txt", "content": "GitPython\nipykernel\nnbsphinx==0.8.7\npandoc\nsphinx\nsphinx_autodoc_typehints\nsphinx_rtd_theme" }, { "path": "docs/tutorial.configs.rst", "content": ".. _config:\n\nTraining Models on Task Datasets (Commands and Configurations) \n#################################################################\n\nLAVIS provides scripts to pre-train and finetune supported models on standard language-vision tasks, stored at ``lavis/run_scripts/``. \nTo replicate the experiments, just run these bash scripts. For example, to train BLIP model on the image-text retrieval task with MSCOCO dataset, we can run\n\n.. code-block::\n\n bash run_scripts/lavis/blip/train/train_retrieval_coco.sh\n\nInside the scripts, we can see \n\n.. code-block:: bash\n\n python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis/projects/blip/train/retrieval_coco_ft.yaml\n\nwhere we start a pytorch distributed training on 8 GPUs (you may change according to your own hardware setup). The ``--cfg-path`` specifys a `runtime configuration file`, specifying\nthe task, model, dataset and training recipes. \n\nAvailable options and their descriptions are as below.\n\n.. LAVIS executes training and evaluation based on arguments specified in the configuration files. The default model and dataset configurations are defined in ``lavis/configs``. The task-specific configurations are defined in ``lavis/projects``. Task-specific configurations have higher priority over the default configurations.\n\n.. The following tables provide explanations for the arguments in the configuration files.\n\n.. list-table::\n :widths: 30 40\n :header-rows: 1\n\n * - Model Configurations\n - Functionalities\n * - arch\n - | name of the model from the model zoo\n | default: task-dependent\n * - model_type\n - | the type of the model (e.g., base)\n | default: task-dependent\n * - load_pretrained\n - | load pretrained weights\n | default: True (for finetuning task) | False (for pretraining task) \n * - load_finetuned\n - | load task-specific finetuned weights\n | default: False (for finetuning task) | True (for evaluation) \n * - pretrained \n - | URL or local path which stores the pretrained model, defined in the default model configuration file\n | default: task-dependent \n * - finetuned\n - | URL or local path which stores the finetuned model, defined in the default model configuration file\n | default: task-dependent\n\n.. list-table::\n :widths: 30 50\n :header-rows: 1\n\n * - Dataset Configurations\n - Functionalities\n * - vis_processor\n - | pre-processing of visual input\n | default: task-dependent\n * - text_processor\n - | pre-processing of text input\n | default: task-dependent\n * - build_info\n - | dataset information including the storage location, defined in the default dataset configuration file\n | default: task-dependent\n\n.. list-table::\n :widths: 30 50\n :header-rows: 1\n\n * - Runtime Configurations\n - Functionalities\n * - task\n - | name of the task\n | default: task-dependent\n * - lr_sched\n - | learning rate schedular\n | default: linear_warmup_cosine_lr\n * - init_lr\n - | initial learning rate (after warmup)\n | default: task-dependent\n * - min_lr\n - | final learning rate after decay\n | default: task-dependent\n * - warmup_lr\n - | starting learning rate for warmup\n | default: init_lr (no warmup)\n * - lr_decay_rate\n - | learning rate decay per epoch for step_lr_shedule\n | default: 0.9\n * - warmup_steps\n - | number of steps for learning rate warmup\n | default: 0\n * - max_epoch\n - | total number of training epochs\n | default: task-dependent\n * - weight_decay\n - | weight decay coefficient for the optimizer\n | default: 0.05\n * - batch_size_train\n - | batch size during training\n | default: task-dependent\n * - batch_size_eval\n - | batch size during evaluation\n | default: task-dependent\n * - seed\n - | pseudo random number generator seed\n | default: 42\n * - output_dir\n - | directory to store logs, results and checkpoints\n | default: task-dependent\n * - resume_ckpt_path\n - | path of the checkpoint to resume training from\n | default: None\n * - evaluate\n - | only perform evaluation without training\n | default: False\n * - train_splits\n - | dataset splits used for training\n | default: [\"train\"]\n * - valid_splits\n - | dataset splits used for validation\n | default: [\"val\"]\n * - test\n - | dataset splits used for test\n | default: [\"test\"]\n * - device\n - | use cpu or gpu (cuda)\n | default: cuda\n * - world_size\n - | number of processes participating in the job\n | default: 1\n * - dist_url\n - | URL specifying how to initialize the process group\n | default: \"env://\"\n * - distributed\n - | use distributed training\n | default: True\n * - amp\n - | use automatic mixed precision training\n | default: False\n\n.. list-table::\n :widths: 40 50\n :header-rows: 1\n\n * - Text Generation Configurations\n - Functionalities\n * - max_len\n - | maximum number of text tokens to generate\n | default: 20 (for image captioning)\n * - min_len\n - | minimum number of text tokens to generate\n | default: 5 (for image captioning)\n * - num_beams\n - | number of beams to perform beam search\n | default: 3\n\n.. list-table::\n :widths: 40 50\n :header-rows: 1\n\n * - Multimodal Retrieval Configurations\n - Functionalities\n * - negative_all_rank\n - | collect negatives from all processes for the image-text matching loss\n | default: True (for coco)\n * - k_test\n - | number of retrieval candidates ranked from contrastive similarity\n | default: 256 (for coco)\n" }, { "path": "docs/tutorial.datasets.rst", "content": "Adding Datasets\n################################################\n\nThis is a tutorial on adding a new dataset using ``lavis.datasets`` module. \n\nThe LAVIS library includes a standard dataset module, which allows customization to add new datasets. \nThe ``lavis.datasets`` module is designed such that any new dataset class can be easily added and adapted from our code base, including creating dataset configuration, and defining and associating new dataset classes.\n\nIn this tutorial, we will replicate the steps to add a dataset class for the `Audio-Visual Scene-Aware Dialogue (AVSD) `_ benchmark for the video-grounded dialogue task.\n\nDataset Configuration ``lavis.configs.datasets``\n**************************************************************\n\nFirst, we define the basic configurations for this dataset, including a new dataset class ``avsd_dialogue``, dataset card, and data types. \nWe can define any new dataset configuration in ``lavis.configs.datasets``. For instance, under this module, we can set up a configuration file ``avsd/defaults_dial.yaml`` as follows: \n\n.. code-block:: yaml\n\n datasets:\n avsd_dialogue: # name of the dataset builder\n dataset_card: dataset_card/avsd_dialogue.md # path to the dataset card \n data_type: features # [images|videos|features] we use features in this case for extracted video features \n\n build_info:\n # Be careful not to append minus sign (-) before split to avoid itemizing\n annotations:\n train:\n url: /export/home/data/avsd/train_set4DSTC7-AVSD.json\n storage: avsd/annotations/train.json\n val:\n url: /export/home/data/avsd/valid_set4DSTC7-AVSD.json\n storage: avsd/annotations/val.json \n test:\n url: /export/home/data/avsd/test_set4DSTC7-AVSD.json\n storage: avsd/annotations/test.json \n features:\n storage: /export/home/data/avsd/features/ \n\n\nDataset Card\n===============\nOne optional step to set up dataset configuration is defining a dataset card, which contains more details about the dataset such as description, tasks, and metrics. \nFor instance, we can define a dataset card for the AVSD benchmark in ``dataset_card/avsd_dialogue.md``.\nDepending on the dataset, we included in its corresponding dataset card the command for auto-downloading data (with python code defined in ``lavis.datasets.download_scripts``) that will automatically load the data and store it in a specific folder.\nElse, you should describe in the dataset card the external download instructions from the original data source to load the dataset properly. \n\nOne example of a dataset card for the AVSD benchmark is: \n\n.. code-block:: md\n\n ![Samples from the AVSD dataset (Image credit: \"https://arxiv.org/pdf/1901.09107.pdf\").](imgs/avsd_dialogue.png)(Samples from the AVSD dataset. Image credit: \"https://arxiv.org/pdf/1901.09107.pdf\")\n \n # Audio-Visual Scene-Aware Dialogues (AVSD) \n \n ## Description\n [Audio-Visual Scene-Aware Dialogues (AVSD)](https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge) contains more than 10,000 dialogues, each of which is grounded on a unique video. In the test split, for each test sample, 6 reference dialogue responses are provided. \n \n \n ## Task\n \n (https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge)\n \n In a **video-grounded dialogue task**, the system must generate responses to user input in the context of a given dialog.\n This context consists of a dialog history (previous utterances by both user and system) in addition to video and audio information that comprise the scene. The quality of a system’s automatically generated sentences is evaluated using objective measures to determine whether or not the generated responses are natural and informative\n \n ## Metrics\n Models are typically evaluated according to [BLEU](https://aclanthology.org/P02-1040/), [CIDER](https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Vedantam_CIDEr_Consensus-Based_Image_2015_CVPR_paper.pdf), [METEOR](https://aclanthology.org/W05-0909/), and [ROUGE-L](https://aclanthology.org/W04-1013/) metrics. \n \n ## Leaderboard\n \n ....\n \n \n ## Auto-Downloading\n \n Please refer to [benchmark webite](https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge) for instructions to download the dataset. \n \n \n ## References\n \"Audio Visual Scene-Aware Dialog\", Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, Devi Parikh\n\nVisual Data Type\n==============================\nWe currently limit the visual data types to one of three options: ``images``, ``videos``, and ``features``. \n\"Images\" and \"videos\" refer to the raw visual data, which is appropriate for models processing visual data in their original forms (e.g. ViT models). \n\"Features\" are visual representations extracted from pretrained models (e.g. CNN models). \nIn this tutorial, the AVSD benchmark consists of video features extracted from 3D-CNN models. \n\nBuild Info\n==============================\nBuild info refers to the specific locations where data is stored and cached. \n\nFor text annotations (e.g. captioning or dialogues), by default, we include three data splits, namely \"train\", \"val\", and \"test\", typically used in all machine learning projects. \nFor each split, we specify 2 parameters: ``url`` and ``storage``.\n``url`` can be either an online URL where the dataset can be loaded automatically (e.g. from *googleapis*), or a local directory where data is already downloaded beforehand. \n``storage`` is the directory where the data will be cached over time, avoiding downloading data repeatedly.\n\nFor visual data annotations, ensure the field name matches the data types defined earlier (e.g. one of \"images\", \"videos\" or features\"). \nAs visual features are usually large and should be downloaded beforehand, we maintain only a ``storage`` parameter where visual data is cached. \n\nDataset ``lavis.datasets.datasets``\n**************************************************************\n\nBase Dataset ``lavis.datasets.datasets.base_dataset``\n=======================================================\nIn this step, we want to define new dataset classes that inherit our base dataset class ``lavis.datasets.datasets.base_dataset``. This base dataset class already defines standard methods such as ``collater`` which uses the default collator from Pytorch. \n\n.. code-block:: python\n\n import json\n from typing import Iterable\n \n from torch.utils.data import Dataset, ConcatDataset\n from torch.utils.data.dataloader import default_collate\n \n class BaseDataset(Dataset):\n def __init__(\n self, vis_processor=None, text_processor=None, vis_root=None, ann_paths=[]\n ):\n \"\"\"\n vis_root (string): Root directory of images (e.g. coco/images/)\n ann_root (string): directory to store the annotation file\n \"\"\"\n self.vis_root = vis_root\n \n self.annotation = []\n for ann_path in ann_paths:\n self.annotation.extend(json.load(open(ann_path, \"r\")))\n \n self.vis_processor = vis_processor\n self.text_processor = text_processor\n \n self._add_instance_ids()\n \n def __len__(self):\n return len(self.annotation)\n \n def collater(self, samples):\n return default_collate(samples)\n \n def set_processors(self, vis_processor, text_processor):\n self.vis_processor = vis_processor\n self.text_processor = text_processor\n \n def _add_instance_ids(self, key=\"instance_id\"):\n for idx, ann in enumerate(self.annotation):\n ann[key] = str(idx)\n\nAny dataset subclass will inherit these methods and it is optional to define and overwrite these methods accordingly to the specifications of the dataset. \nWe encourage users not to modify the base dataset class as any modification will have cascading impacts on any other dataset classes that inherit this base dataset. \nInstead, the users should independently create new dataset classes to cater to their specific requirements. \n\nDialogue Datasets ``lavis.datasets.datasets.dialogue_datasets``\n======================================================================\n\nFor example, for the AVSD dataset, we want to define a new dataset subclass ``DialogueDataset`` for dialogue tasks. We can define this dataset class in ``lavis.datasets.datasets.dialogue_datasets`` as following: \n\n.. code-block:: python\n\n import os\n from collections import OrderedDict\n \n from lavis.datasets.datasets.base_dataset import BaseDataset\n \n import json \n import copy \n\n class DialogueDataset(BaseDataset):\n def __init__(self, vis_processor, text_processor, vis_root, ann_paths):\n \"\"\"\n vis_processor (string): visual processor \n text_processor (string): textual processor \n vis_root (string): Root directory of images (e.g. coco/images/)\n ann_paths (string): Root directory of images (e.g. coco/images/)\n \"\"\"\n \n self.vis_root = vis_root\n \n self.annotation = []\n for ann_path in ann_paths:\n dialogs = json.load(open(ann_path, \"r\"))['dialogs']\n for dialog in dialogs: \n all_turns = dialog['dialog']\n dialogue_context = [] \n for turn in all_turns: \n dialog_instance = copy.deepcopy(dialog)\n question = turn['question']\n answer = turn['answer'] \n \n dialog_instance['dialog'] = copy.deepcopy(dialogue_context) \n dialog_instance['question'] = question\n dialog_instance['answer'] = answer \n self.annotation.append(dialog_instance)\n dialogue_context.append(turn)\n \n self.vis_processor = vis_processor\n self.text_processor = text_processor\n \n self._add_instance_ids()\n \n self.img_ids = {}\n n = 0\n for ann in self.annotation:\n img_id = ann[\"image_id\"]\n if img_id not in self.img_ids.keys():\n self.img_ids[img_id] = n\n n += 1\n\nClass inheritance allows us to define multiple subclasses. For instance, we want another dialogue dataset class that is defined only for the test split. We can define another dataset class ``DialogueEvalDataset`` as similarly defined above but the annotations are processed differently. \nTypically, in dialogue tasks, during test time, only a single test sample is constructed per dialogue (rather than decomposing all dialogue turns as samples during training time).\nThe dataset class can then be defined as: \n\n.. code-block:: python\n\n class DialogueEvalDataset(BaseDataset):\n def __init__(self, vis_processor, text_processor, vis_root, ann_paths):\n # ...\n # defined similarly as DialogueDataset above \n # except for the loading of dialogue annotation data \n \n self.annotation = []\n for ann_path in ann_paths:\n dialogs = json.load(open(ann_path, \"r\"))['dialogs']\n for dialog in dialogs: \n all_turns = dialog['dialog']\n dialogue_context = all_turns[:-1]\n last_turn = all_turns[-1] \n \n question = last_turn['question']\n answer = last_turn['answer'] \n \n dialog['dialog'] = dialogue_context\n dialog['question'] = question\n dialog['answer'] = answer\n \n self.annotation.append(dialog)\n\n\nUsing class inheritance to define datasets also allows us to develop more fine-grain class implementations, each of which is specifically designated for a benchmark. \nFor instance, under the dialogue-based tasks, we can further define another dataset subclass that is specified for the AVSD dataset. \nWe can define a new class ``AVSDDialDataset`` that further specifies how to load individual samples and collate them accordingly to specific requirements: \n\n.. code-block:: python\n\n import os\n from lavis.datasets.datasets.base_dataset import BaseDataset\n from lavis.datasets.datasets.dialogue_datasets import DialogueDataset, DialogueEvalDataset\n \n import torch \n \n class AVSDDialDataset(DialogueDataset):\n def __init__(self, vis_processor, text_processor, vis_root, ann_paths):\n\n super().__init__(vis_processor, text_processor, vis_root, ann_paths)\n \n def __getitem__(self, index):\n \n ann = self.annotation[index]\n \n vname = ann[\"image_id\"]\n \n video = self.vis_processor(self.vis_root, vname)\n \n dialogue = self.text_processor(ann)\n \n return {\n \"video_fts\": video['video_fts'],\n \"video_token_type_ids\": video['token_type_ids'], \n \"input_ids\": dialogue['input_ids'], \n \"token_type_ids\": dialogue['token_type_ids'],\n \"labels\": dialogue['labels'], \n \"image_id\": ann[\"image_id\"],\n \"instance_id\": ann[\"instance_id\"]\n }\n \n def collater(self, samples):\n \n input_ids, token_type_ids, labels, video_fts, video_token_type_ids = [], [], [], [], []\n \n for i in samples:\n input_ids.append(i['input_ids'])\n token_type_ids.append(i['token_type_ids'])\n labels.append(i['labels'])\n video_fts.append(i['video_fts'])\n video_token_type_ids.append(i['video_token_type_ids'])\n \n input_ids = self.text_processor.padding(input_ids)\n \n labels = self.text_processor.padding(labels, -1)\n video_fts = self.vis_processor.padding(video_fts)\n \n token_type_ids = self.text_processor.padding(token_type_ids)\n video_token_type_ids = self.text_processor.padding(video_token_type_ids)\n token_type_ids = torch.cat([video_token_type_ids, token_type_ids], dim=1)\n \n attn_mask = self.text_processor.get_attention_mask(input_ids)\n video_mask = self.vis_processor.get_attention_mask(video_fts)\n attn_mask = torch.cat([video_mask, attn_mask], dim=1)\n \n video_labels = torch.ones((video_fts.size(0), video_fts.size(1))).long() * -1 # ignore token indice -1 by default \n\n labels = torch.cat([video_labels, labels], dim=1)\n \n samples = {}\n samples['input_ids'] = input_ids\n samples['token_type_ids'] = token_type_ids\n samples['labels'] = labels\n samples['video_fts'] = video_fts\n samples['attn_mask'] = attn_mask\n \n return samples \n\nNote that in a dataset subclass, if methods such as ``__getitem__`` and ``collater`` are not defined, the same functions from the corresponding superclass will be used. \nFor instance, by default, we always use the collater from the ``BaseDataset`` class to collate data samples. \n\nDataset Builder ``lavis.datasets.builders``\n**************************************************************\nDataset Builder is the data processing module that controls the dataset classes (by training or evaluation split) and associates the specific dataset configurations to these dataset classes. \n\nBase Dataset Builder ``lavis.datasets.builders.base_dataset_builder``\n======================================================================\n\nNote that any new builder class definition should inherit the base dataset builder class ``lavis.datasets.builders.base_dataset_builder``:\n\n.. code-block:: python\n\n class BaseDatasetBuilder:\n train_dataset_cls, eval_dataset_cls = None, None\n ...\n\nThis allows us to standardize the operations of dataset builders across all builder classes. We advise the users to carefully review the standard methods defined in the base builder class, including methods such as ``_download_data`` and ``build_dataset`` that will load download the data and create instances of dataset classes: \n\n.. code-block:: python\n\n class BaseDatasetBuilder:\n ...\n\n def build_datasets(self):\n # download, split, etc...\n # only called on 1 GPU/TPU in distributed\n \n if is_main_process():\n self._download_data()\n \n if is_dist_avail_and_initialized():\n dist.barrier()\n \n # at this point, all the annotations and image/videos should be all downloaded to the specified locations.\n logging.info(\"Building datasets...\")\n datasets = self.build() # dataset['train'/'val'/'test']\n \n return datasets\n \n def _download_data(self):\n self._download_ann()\n self._download_vis()\n \nWe encourage users not to modify the implementation of the base dataset builder class as this will affect all existing dataset builder subclasses.\n\nDialogue Dataset Builder ``lavis.datasets.builders.dialogue_builder``\n======================================================================\nWe can define any new builder subclass and associate this builder with the corresponding dataset classes and dataset configurations. \nFor instance, for the AVSD dataset, we can define a builder ``lavis.datasets.builders.dialogue_builder`` for dialogue-based datasets as follows: \n\n.. code-block:: python\n\n from lavis.datasets.builders.base_dataset_builder import BaseDatasetBuilder\n from lavis.datasets.datasets.avsd_dialogue_datasets import (\n AVSDDialDataset, \n AVSDDialEvalDataset \n )\n \n from lavis.common.registry import registry\n \n \n @registry.register_builder(\"avsd_dialogue\")\n class AVSDDialBuilder(BaseDatasetBuilder):\n train_dataset_cls = AVSDDialDataset \n eval_dataset_cls = AVSDDialEvalDataset \n \n DATASET_CONFIG_DICT = {\n \"default\": \"configs/datasets/avsd/defaults_dial.yaml\"\n }\n\nNote that we chose to separately define the parameters ``train_dataset_cls`` and ``eval_dataset_cls`` to consider cases where data is processed differently between training and test time. \nFor instance, in captioning tasks, during test time, each data sample often includes multiple ground-truth captions rather than just a single ground-truth during training time. \nIf the data processing is the same in both training and test time, the two parameters can be linked to the same dataset class. \n\nFinally, define ``DATASET_CONFIG_DICT`` to associate the dataset configurations to the assigned dataset classes. \n\nRegistering Builder ``lavis.datasets.builders.__init__``\n======================================================================\n\nTo add a new builder class, ensure to first include the class within the ``__init__.py``. For instance, to define a new builder for the AVSD dataset: \n\n.. code-block:: python\n\n from lavis.datasets.builders.dialogue_builder import (\n AVSDDialBuilder\n )\n \n __all__ = [\n ...,\n \"AVSDDialBuilder\"\n ]\n\nAssigning Builder \n======================================================================\nNote that during data loading and processing, the builder being assigned must have the correct registry to be able to load it properly. \nFor instance, the following should be specified in a configuration file e.g. ``dialogue_avsd_ft.yaml``: \n\n.. code-block:: yaml\n\n datasets:\n avsd_dialogue: # name of the dataset builder\n ...\n # processor configuration \n ...\n\nSubsequently, any processes (e.g. training) should load this configuration file to assign the correct builder which will then associate the correct dataset classes to construct data samples. \n\n.. code-block:: sh\n\n python train.py --cfg-path dialogue_avsd_ft.yaml\n" }, { "path": "docs/tutorial.evaluation.rst", "content": "Evaluating Pre-trained Models on Task Datasets\n###############################################\nLAVIS provides pre-trained and finetuned model for off-the-shelf evaluation on task dataset. \nLet's now see an example to evaluate BLIP model on the captioning task, using MSCOCO dataset.\n\n.. _prep coco:\n\nPreparing Datasets\n******************\nFirst, let's download the dataset. LAVIS provides `automatic downloading scripts` to help prepare \nmost of the public dataset, to download MSCOCO dataset, simply run\n\n.. code-block:: bash\n\n cd lavis/datasets/download_scripts && bash download_coco.py\n\nThis will put the downloaded dataset at a default cache location ``cache`` used by LAVIS.\n\nIf you want to use a different cache location, you can specify it by updating ``cache_root`` in ``lavis/configs/default.yaml``.\n\nIf you have a local copy of the dataset, it is recommended to create a symlink from the cache location to the local copy, e.g.\n\n.. code-block:: bash\n\n ln -s /path/to/local/coco cache/coco\n\nEvaluating pre-trained models\n******************************\n\nTo evaluate pre-trained model, simply run\n\n.. code-block:: bash\n\n bash run_scripts/lavis/blip/eval/eval_coco_cap.sh\n\nOr to evaluate a large model:\n\n.. code-block:: bash\n\n bash run_scripts/lavis/blip/eval/eval_coco_cap_large.sh" }, { "path": "docs/tutorial.models.rst", "content": "Adding Models\n####################################\n\nThis is a tutorial on adding new models using ``lavis.models`` module.\n\nThe LAVIS library includes a standard model module that builds the foundation for many major language-vision models such as `ALBEF `_,\n`BLIP `_, `ALPRO `_, and `CLIP `_. \nThe ``lavis.models`` module is designed such that any new models can be added and integrated into the LAVIS library, with minimal steps to develop training and testing procedures. \nIn this tutorial, we will replicate the steps to add a GPT-style model specifically for `video-grounded dialogue tasks `_. \n\nBase Model ``lavis.models.base_model``\n**************************************************************\n\nNote that any new model definition should inherit the base model class ``BaseModel``:\n\n.. code-block:: python\n\n from omegaconf import OmegaConf\n \n import numpy as np\n \n import torch\n import torch.nn as nn\n \n from lavis.common.utils import get_abs_path\n \n class BaseModel(nn.Module):\n \"\"\"Base class for models.\"\"\"\n \n def __init__(self):\n super().__init__()\n \n def forward_features(self, *args, **kwargs):\n \"\"\"Similar to *forward* but only return features.\"\"\"\n raise NotImplementedError\n \n def load_from_pretrained(self, url_or_filename):\n raise NotImplementedError\n \n @classmethod\n def _from_config(cls, cfg=None, model_type=\"base\"):\n if not cfg:\n # useful when building model without a provided configuration file\n cfg = OmegaConf.load(cls.default_config_path(model_type)).model\n \n return cls.from_config(cfg)\n \n @classmethod\n def from_pretrained(cls, model_type=\"base\"):\n \"\"\"\n Build a pretrained model from the default configuration file, specified by model_type.\n \"\"\"\n return cls._from_config(cfg=None, model_type=model_type)\n \n @property\n def device(self):\n return list(self.parameters())[0].device\n \n @classmethod\n def default_config_path(cls, model_type=\"base\"):\n assert (\n model_type in cls.PRETRAINED_MODEL_CONFIG_DICT\n ), \"Unknown model type {}\".format(model_type)\n return get_abs_path(cls.PRETRAINED_MODEL_CONFIG_DICT[model_type])\n \n def before_evaluation(self, **kwargs):\n pass\n \n def show_n_params(self, return_str=True):\n tot = 0\n for p in self.parameters():\n w = 1\n for x in p.shape:\n w *= x\n tot += w\n if return_str:\n if tot >= 1e6:\n return \"{:.1f}M\".format(tot / 1e6)\n else:\n return \"{:.1f}K\".format(tot / 1e3)\n else:\n return tot\n\n\nIn this base model, we already declare and standardize many common methods such as ``_from_config`` and ``_from_pretrained``. \nInheriting this base model class allows us to standardize operations of models across all model classes while still allowing customizations. \nWe advise users not to change the implementation of the base model class as this will affect all existing model subclasses.\n\nGPT-style Video-grounded Dialogue Model ``lavis.models.gpt_models.gpt_dialogue``\n********************************************************************************\n\nIn this step, we can define a new model class, e.g. under ``lavis.models.gpt_models.gpt_dialogue``, for GPT-based dialogue models designed specifically for video-grounded dialogues. \nNote that we assume the model class inherits from the standard model super class ``GPT2LMHeadModel`` from the ``transformers`` `library `_.\nWe also enforce model integration to the LAVIS framework through the inheritance of the ``BaseModel`` from the LAVIS library, as the secondary super class.\n\n.. code-block:: python\n\n import torch\n from lavis.common.registry import registry\n from lavis.models.base_model import BaseModel\n \n from transformers import GPT2Model, GPT2LMHeadModel\n from transformers.modeling_outputs import CausalLMOutputWithCrossAttentions\n import math\n import torch\n import torch.nn as nn\n from torch.nn import CrossEntropyLoss, MSELoss\n \n @registry.register_model(\"gpt_dialogue\")\n class GPTDialogue(GPT2LMHeadModel, BaseModel):\n ...\n \nNext, we can modify the architecture of the model during model initialization to fit the tasks of interest, i.e. video-grounded dialogues. \nIn this case, we want to add additional model parameters for a linear network to transform the video feature representations to the model dimension. \n\n.. code-block:: python\n\n class GPTDialogue(GPT2LMHeadModel, BaseModel):\n\n def __init__(self, config, len_video_ft=4224):\n \n super().__init__(config)\n \n self.video_ff = nn.Linear(len_video_ft, config.n_embd)\n \n # Model parallel\n self.model_parallel = False\n self.device_map = None\n \n # Initialize weights and apply final processing\n self.post_init()\n \nNote that for each new model class, we advise redefining the ``from_config`` method which is inherited from the ``BaseModel`` class.\nAs each model usually has its own unique configurations, redefining the method will ensure the model instances are created properly. \nFor instance, ``GPTDialogue`` requires an additional parameter of video feature length (``len_video_ft``) which should be part of the model initialization procedure. \nAnother additional parameter is the number of tokens/words (as we include additional special tokens in the vocabulary for dialogue tasks). \n\n.. code-block:: python\n\n class GPTDialogue(GPT2LMHeadModel, BaseModel):\n ...\n @classmethod\n def from_config(cls, cfg):\n model = cls.from_pretrained('gpt2', len_video_ft=cfg['len_video_ft']) \n model.resize_token_embeddings(cfg['len_tokenizer'])\n return model\n\nOther basic methods should also be defined explicitly in the new model class, including the ``forward`` function. \nFor instance, in GPT models for video-grounded dialogue tasks, we want the forward operation also includes the transformation and integration of video features before passing the representations to the Transformer layers. \n\n.. code-block:: python\n\n class GPTDialogue(GPT2LMHeadModel, BaseModel):\n ...\n\n def forward(self, samples, \n past_key_values=None,\n position_ids=None,\n head_mask=None,\n encoder_hidden_states=None,\n encoder_attention_mask=None,\n use_cache=None,\n output_attentions=None,\n output_hidden_states=None,\n return_dict=None): \n \n input_embs = self.transformer.wte(samples['input_ids'])\n video_embs = self.video_ff(samples['video_fts'])\n input_embs = torch.cat([video_embs, input_embs], dim=1)\n \n transformer_outputs = self.transformer(\n attention_mask=samples['attn_mask'],\n token_type_ids=samples['token_type_ids'],\n inputs_embeds=input_embs,\n position_ids=position_ids,\n head_mask=head_mask,\n encoder_hidden_states=encoder_hidden_states,\n encoder_attention_mask=encoder_attention_mask,\n use_cache=use_cache,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n hidden_states = transformer_outputs[0]\n \n lm_logits = self.lm_head(hidden_states)\n ...\n\nRegistering New Model ``lavis.models.__init__``\n********************************************************************************\n\nAny new model must be officially registered as part of the ``lavis.models`` module. \nFor instance, to add a model class for GPT-based dialogue models, we can modify the ``__init__.py`` as follows:\n\n.. code-block:: python\n\n from lavis.models.gpt_models.gpt_dialogue import GPTDialogue\n \n __all__ = [\n ...\n \"GPTDialogue\"\n ]\n\nAssigning Model\n********************************************************************************\n\nFrom the above example of a model class, note that we define a ``from_config method`` for the new model class. \nThis method will process a configuration file and pass specific parameters to initialize the model classes properly. \nTo do this, we can assign/ associate the correct registry of model classes in a configuration file. \nFor instance, the following should be specified in a configuration file e.g. ``dialogue_avsd_ft.yaml``:\n\n.. code-block:: yaml\n\n model:\n arch: gpt_dialogue # name of the model \n model_type: base\n\n\nSubsequently, any processes (e.g. training) should load this configuration file to assign the correct model.\n\n.. code-block:: sh\n\n python train.py --cfg-path dialogue_avsd_ft.yaml\n\nNote that to simplify the model configuration, we only enable two main parameters here: ``arch`` and ``model_type``. ``arch`` refers to the model class registry, and ``model_type`` is the corresponding model type under this model family.\nFor instance, with ``gpt_dialogue``, we have a model ``base`` which has its own configuration in a separate configuration file e.g. ``gpt_dialogue_base.yaml``:\n\n.. code-block:: yaml\n\n model:\n arch: gpt_dialogue\n len_tokenizer: 50264 # 50257 tokens from gpt2 default tokenizer + additional special tokens \n len_video_ft: 4224 # i3d_rgb: 2048 i3d_flow: 2048 vggish: 128 \n\nWe can pass load this configuration and pass the parameters to the above ``from_config`` method to initialize the model accordingly. \nWe advise the users to maintain a dictionary that contains default paths to model configurations, in the model class definition. \nBy default, the LAVIS framework will search for configurations from each model class defined as ``model.PRETRAINED_MODEL_CONFIG_DICT``.\n\n.. code-block:: python\n\n class GPTDialogue(GPT2LMHeadModel, BaseModel):\n PRETRAINED_MODEL_CONFIG_DICT = {\n \"base\": \"configs/models/gpt_dialogue_base.yaml\"\n }\n ...\n" }, { "path": "docs/tutorial.processors.rst", "content": "Adding Processors\n################################################\n\nThis is a tutorial on adding new processors using ``lavis.processors`` module. \n\nThe LAVIS library includes a standard processor module that preprocesses data e.g. image transformation and sequence concatenation.\nThe ``lavis.processors`` module is designed such that any processors can be added, specifically to the requirements of corresponding models of interest. \nIn this tutorial, we will replicate the steps to add visual and textual processors specifically for `video-grounded dialogue tasks `_. \nIn addition, we also want the processors to have processing features to make the data samples compatible with GPT-style models.\n\nBase Processor ``lavis.processors.base_processors``\n*****************************************************\n\nNote that any new processor definition should inherit the base processor class ``BaseProcessor``:\n\n.. code-block:: python\n\n from omegaconf import OmegaConf\n \n class BaseProcessor:\n def __init__(self):\n self.transform = lambda x: x\n return\n \n def __call__(self, item):\n return self.transform(item)\n \n @classmethod\n def from_config(cls, cfg=None):\n return cls()\n \n def build(self, **kwargs):\n cfg = OmegaConf.create(kwargs)\n \n return self.from_config(cfg)\n\nThis allows us to standardize operations of processors across all processor classes while still allowing customization of processors specifically to data and model types. \nWe encourage users not to modify the implementation of the base processor class as this will have an impact on all existing processor subclasses.\n\nGPT-style Processors ``lavis.processors.gpt_processors``\n**************************************************************\nIn this step, we can define new processor classes, e.g. under ``lavis.processors.gpt_processors``, for GPT models designed specifically for video-grounded dialogues. \nFirst, we want to process video features by defining ``GPTVideoFeatureProcessor`` class.\nIn this tutorial, we assume video features are extracted beforehand and this processor simply loads the features from ``npy`` files.\nOther methods that are specifically defined are ``padding`` (which is used by dataset instances to pad multiple video samples) and ``get_attention_mask`` (which creates an attention mask for Transformer attention in GPT models). \n\n.. code-block:: python \n\n SPECIAL_TOKENS_DICT = {'bos_token': \"\", 'eos_token': \"\", 'additional_special_tokens': [\"\", \"\", \"