Showing preview only (279K chars total). Download the full file or copy to clipboard to get everything.
Repository: dmarx/video-killed-the-radio-star
Branch: main
Commit: 7d1053356aa8
Files: 14
Total size: 270.1 KB
Directory structure:
gitextract_j7_i1yfm/
├── .github/
│ └── workflows/
│ └── python-publish.yml
├── .gitignore
├── LICENSE
├── README.md
├── VERSION
├── Video_Killed_The_Radio_Star_Defusion.ipynb
├── pyproject.toml
└── vktrs/
├── __init__.py
├── api.py
├── asr.py
├── hf.py
├── tsp.py
├── utils.py
└── youtube.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/python-publish.yml
================================================
# This workflow will upload a Python Package using Twine when a release is created
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Upload Python Package
on:
release:
types: [published]
permissions:
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build
- name: Publish package
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
with:
user: __token__
password: ${{ secrets.PYPI_TOKEN }}
================================================
FILE: .gitignore
================================================
_venv
*.srv2
*.vtt
*.webm
*.mp3
*.mp4
*.pyc
*.egg-info/
**/frames
**/archive
*.yaml
*.whl
*.tar.gz
dist
# local huggingface model directories and files
feature_extractor
safety_checker
scheduler
text_encoder
tokenizer
unet
vae
model_index.json
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 David Marx
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Video Killed The Radio Star [](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)
## Requirements
* ffmpeg - https://ffmpeg.org/
* pytorch - https://pytorch.org/get-started/locally/
* vktrs - (this repo) - `pip install vktrs[api]`
* stability_sdk api token - https://beta.dreamstudio.ai/ > circular icon in top right > membership > API Key
* whisper - `pip install git+https://github.com/openai/whisper`
## FAQ
**What is this?**
TLDR: Automated music video maker, given an mp3 or a youtube URL
**How does this animation technique work?**
For each text prompt you provide, the notebook will...
1. Generate an image based on that text prompt (using stable diffusion)
2. Use the generated image as the `init_image` to recombine with the text prompt to generate variations similar to the first image. This produces a sequence of extremely similar images based on the original text prompt
3. Images are then intelligently reordered to find the smoothest animation sequence of those frames
3. This image sequence is then repeated to pad out the animation duration as needed
The technique demonstrated in this notebook was inspired by a [video](https://www.youtube.com/watch?v=WJaxFbdjm8c) created by Ben Gillin.
**How are lyrics transcribed?**
This notebook uses openai's recently released 'whisper' model for performing automatic speech recognition.
OpenAI was kind of to offer several different sizes of this model which each have their own pros and cons.
This notebook uses the largest whisper model for transcribing the actual lyrics. Additionally, we use the
smallest model for performing the lyric segmentation. Neither of these models is perfect, but the results
so far seem pretty decent.
The first draft of this notebook relied on subtitles from youtube videos to determine timing, which was
then aligned with user-provided lyrics. Youtube's automated captions are powerful and I'll update the
notebook shortly to leverage those again, but for the time being we're just using whisper for everything
and not referencing user-provided captions at all.
**Something didn't work quite right in the transcription process. How do fix the timing or the actual lyrics?**
The notebook is divided into several steps. Between each step, a "storyboard" file is updated. If you want to
make modifications, you can edit this file directly and those edits should be reflected when you next load the
file. Depending on what you changed and what step you run next, your changes may be ignored or even overwritten.
Still playing with different solutions here.
**Can I provide my own images to 'bring to life' and associate with certain lyrics/sequences?**
Yes, you can! As described above: you just need to modify the storyboard. Will describe this functionality in
greater detail after the implementation stabilizes a bit more.
**This gave me an idea and I'd like to use just a part of your process here. What's the best way to reuse just some of the machinery you've developed here?**
Most of the functionality in this notebook has been offloaded to library I published to pypi called `vktrs`. I strongly encourage you to import anything you need
from there rather than cutting and pasting function into a notebook. Similarly, if you have ideas for improvements, please don't hesitate to submit a PR!
## Dev notes
```
!pip install --upgrade setuptools build
!git clone https://github.com/dmarx/video-killed-the-radio-star/
!cd video-killed-the-radio-star; python -m build; python -m pip install -e .[api,hf]
!pip install ipykernel ipywidgets panel prefetch_generator
```
================================================
FILE: VERSION
================================================
0.1.8
================================================
FILE: Video_Killed_The_Radio_Star_Defusion.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "mgXxoDhMAiti"
},
"source": [
"# $ \\text{Video Killed The Radio Star}$ $\\color{red}{...Diffusion}$\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Notebook by David Marx ([@DigThatData](https://twitter.com/digthatdata))\n",
"\n",
"Shared under MIT license\n",
"\n",
"Last updated: 2023-06-15\n",
"\n",
"Latest stable notebook revision: [](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)\n",
"\n",
"# Introduction\n",
"\n",
"VKTRS is a tool for planning and generating complex animations using audio with spoken/sung words as conditioning input. The planning work can be used independently from generating an animation: all of your work is saved to a human-readable and modifiable `storyboard.yaml` file. Parameter names and notation are also compatible with other AI animation tools, such as a1111-deforum.\n",
"\n",
"The inspiration for this notebook and the \"variations\" animation modes were [this video](https://www.youtube.com/watch?v=WJaxFbdjm8c) created by Ben Gillin.\n",
"\n",
"# General Workflow\n",
"\n",
"If you are reading this in colab, open the \"Table of contents\" tab on the left sidebar for a more detailed enumeration of the steps described below.\n",
"\n",
"1. Pick an audio source, like a youtube video\n",
"2. The notebook will download the video if necessary\n",
"3. OpenAI's \"whisper\" model is used to transcribe the lyrics\n",
"4. The timing of this transcription is used to segment the timeline into a sequence of \"scenes\"\n",
"5. The musical structure of the audio is analyzed to group thematically similar themes\n",
"6. Scene/theme specific settings (prompts, camera motion) can then be added\n",
"7. The audio can be further processed to isolate signals for driving animation parameters for \"audioreactive\" effects. This includes isolating instruments (demucs) and chaining manipulations (librosa)\n",
"8. Generate or specify starting images for each scene\n",
"9. Generate the remaining frames to animate each scene\n",
"\n",
"**NB: the `img2img` animation mode is currently only supported when using the stability ai animation api (DreamStudio) to generate animations**. The audioreactivity features are currently only relevant in img2img mode. If you don't have a DreamStudio account, you can still use this notebook to parameterize an animation, you'll just need to cut-and-paste settings out of the \"export settings\" cell (deforum compatible) or the `storyboard.yaml` file.\n",
"\n",
"# The \"Storyboard\"\n",
"\n",
"Start at the top of the notebook and work your way down. Each decision you make contributes information to a file called \"the storyboard\", which persists project state. The notebook is designed to facilitate working iteratively: once you've made it past a given decision point, you should generally be able to jump back to that point to tweak whatever the decision was and return to where you were.\n",
"\n",
"The storyboard isn't just a logical abstraction, it's a physical file you can modify directly, and is reasonably human readable.\n",
"\n",
"Every project you start will create a new storyboard. If you set your project name to one you used previously, the notebook will attempt to find that project and load its storyboard.\n",
"\n",
"The original motivation behind VKTRS was to be a toy for automating generative music videos. It has evolved quite a bit since, and I now mostly think about it as a tool for building and manipulating these storyboards. The storyboard parameterizes an animation, but the actual animation doesn't need to be generated by this tool.\n",
"\n",
"\n",
"# $\\text{FAQ}$\n",
"\n",
"**Why the name?**\n",
"\n",
"The notebook's name is an ironic homage to the first music video played on MTV: [The Bugles - Video Killed The Radio Star](https://www.youtube.com/watch?v=W8r-tXRLazs). Quoting [wikipedia](https://en.wikipedia.org/wiki/Video_Killed_the_Radio_Star):\n",
"\n",
"> \"The song relates to concerns about, and mixed attitudes toward 20th-century inventions and machines for the media arts.\"\n",
"\n",
"**Something didn't work quite right in the transcription process. How do fix the timing or the actual lyrics?**\n",
"\n",
"The notebook is divided into several steps. Between each step, a \"storyboard\" file is updated. If you want to\n",
"make modifications, you can edit this file directly and those edits should be reflected when you next load the\n",
"file. Depending on what you changed and what step you run next, your changes may be ignored or even overwritten.\n",
"Still playing with different solutions here.\n",
"\n",
"**Can I provide my own images to 'bring to life' and associate with certain lyrics/sequences?**\n",
"\n",
"Yes, you can! As described above: you just need to modify the storyboard. Will describe this functionality in\n",
"greater detail after the implementation stabilizes a bit more.\n",
"\n",
"**How can I support your work or work like it?**\n",
"\n",
"This notebook was made possible thanks to ongoing support from [stability.ai](https://stability.ai/). The best way to support my work is to share it with your friends, [report bugs](https://github.com/dmarx/video-killed-the-radio-star/issues/new), [suggest features](https://github.com/dmarx/video-killed-the-radio-star/discussions) or to donate to open source non-profits :)\n",
"\n",
"# Examples of content made with VKTRS\n",
"\n",
"[](https://www.youtube.com/watch?v=dx8LmqalrmU)\n",
"\n",
"**The Sudden - Fine (Official Music Video)** `variations` animation mode and theme prompts.\n",
"\n",
"<div><a href=\"https://www.youtube.com/watch?v=ScbXSLYyzq4\"><img src=\"https://i.ytimg.com/vi_webp/ScbXSLYyzq4/oar2.webp\" alt=\"Hi-Standard - Asian Pride\" width=\"300\"/></a></div>\n",
"\n",
" **Hi-Standard - Asian Pride** `img2img` animation mode (via the Stability.AI animation API), with camera movement and simple audioreactivity (vocals stem driving `strength_curve` and `noise_curve`). This was one of my test animations and doesn't do the notebook justice, but you'll get the idea.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sM147HP4kAdY"
},
"source": [
"## $0.$ Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "ZnTe8clZuZuj",
"tags": []
},
"outputs": [],
"source": [
"# @title # 📊 Check GPU Status\n",
"\n",
"import pandas as pd\n",
"import subprocess\n",
"\n",
"def gpu_info():\n",
" outv = subprocess.run([\n",
" 'nvidia-smi',\n",
" # these lines concatenate into a single query string\n",
" '--query-gpu='\n",
" 'timestamp,'\n",
" 'name,'\n",
" 'utilization.gpu,'\n",
" 'utilization.memory,'\n",
" 'memory.used,'\n",
" 'memory.free,'\n",
" ,\n",
" '--format=csv'\n",
" ],\n",
" stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
"\n",
" header, rec = outv.split('\\n')[:-1]\n",
" return pd.DataFrame({' '.join(k.strip().split('.')).capitalize():v for k,v in zip(header.split(','), rec.split(','))}, index=[0]).T\n",
"\n",
"gpu_info()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "FIJ-gPjcVXby",
"tags": []
},
"outputs": [],
"source": [
"#%%capture\n",
"\n",
"# @title # 🛠️ Setup: Install Dependencies\n",
"\n",
"# Install dependencies\n",
"\n",
"try:\n",
" import google.colab\n",
" local=False\n",
"except:\n",
" local=True\n",
"\n",
"# TODO: pin versions\n",
"\n",
"# local only additional dependencies\n",
"if local:\n",
" %pip install pandas torch pillow beautifulsoup4 scipy toolz numpy lxml librosa scikit-learn rich\n",
"\n",
"# dependencies for both colab and local\n",
"%pip install yt-dlp python-tsp stability-sdk[anim_ui] diffusers transformers ftfy accelerate omegaconf\n",
"%pip install openai-whisper panel huggingface_hub ipywidgets safetensors keyframed demucs parse"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OrHUOTwdgCfK",
"tags": [],
"cellView": "form"
},
"outputs": [],
"source": [
"#%%capture\n",
"\n",
"# @title # 🛠️ Setup: Imports and Definitions\n",
"\n",
"# Definitions and imports\n",
"\n",
"from collections import defaultdict\n",
"import copy\n",
"import datetime as dt\n",
"import gc\n",
"import io\n",
"from itertools import chain, cycle\n",
"import json\n",
"import os\n",
"from pathlib import Path\n",
"import random\n",
"import re\n",
"import shutil\n",
"import string\n",
"import subprocess\n",
"from subprocess import Popen, PIPE\n",
"import time\n",
"import warnings\n",
"\n",
"from bokeh.models.widgets.tables import (\n",
" NumberFormatter,\n",
" BooleanFormatter,\n",
" CheckboxEditor,\n",
")\n",
"from diffusers import (\n",
" StableDiffusionImg2ImgPipeline,\n",
" StableDiffusionPipeline,\n",
")\n",
"from IPython.display import display\n",
"import keyframed\n",
"import keyframed as kf # TODO...\n",
"import keyframed.dsl\n",
"from keyframed.serialization import from_dict as load_curve\n",
"import librosa\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from omegaconf import OmegaConf, DictConfig\n",
"import pandas as pd\n",
"import panel as pn\n",
"import parse\n",
"import PIL\n",
"from PIL import Image\n",
"from PIL import Image, ImageDraw, ImageFont\n",
"from python_tsp.exact import solve_tsp_dynamic_programming\n",
"import rich\n",
"from safetensors.numpy import save_file as save_safetensors\n",
"from safetensors.numpy import load_file as load_safetensors\n",
"import scipy\n",
"from scipy.spatial.distance import pdist, squareform\n",
"import sklearn.cluster\n",
"import textwrap\n",
"from tqdm.autonotebook import tqdm\n",
"import torch\n",
"from torch import autocast\n",
"import whisper\n",
"\n",
"from stability_sdk.api import Context\n",
"from stability_sdk.animation import AnimationArgs, Animator\n",
"\n",
"from stability_sdk.animation import (\n",
" AnimationArgs,\n",
" Animator,\n",
" AnimationSettings,\n",
" BasicSettings,\n",
" CoherenceSettings,\n",
" ColorSettings,\n",
" DepthSettings,\n",
" InpaintingSettings,\n",
" Rendering3dSettings,\n",
" CameraSettings,\n",
" VideoInputSettings,\n",
" VideoOutputSettings,\n",
")\n",
"\n",
"try:\n",
" import google.colab\n",
" local=False\n",
"except:\n",
" local=True\n",
"\n",
"\n",
"def sanitize_folder_name(fp):\n",
" outv = ''\n",
" whitelist = string.ascii_letters + string.digits + '-_'\n",
" for token in str(fp):\n",
" if token not in whitelist:\n",
" token = '-'\n",
" outv += token\n",
" return outv\n",
"\n",
"# to do: is there a way to check if this is in the env already?\n",
"#pn.extension('tabulator')\n",
"\n",
"\n",
"def establish_workspace(\n",
" use_stability_api,\n",
" mount_gdrive,\n",
" application_name=\"VideoKilledTheRadioStar\",\n",
" active_project=None,\n",
"):\n",
" \"\"\"\n",
" This function constructs a local file called `config.yaml` that maintains state that will be used elsewhere.\n",
" It mostly sets the names of project folders and a handful of settings. The reason for doing things this way\n",
" is to facilitate \"resume\" functionality and creating new projects without overwriting previously created assets.\n",
"\n",
" By convention, when loaded the config.yaml is referred to as the `workspace` object.\n",
"\n",
" Most project-specific content will be located in a project-specific config -- `storyboard.yaml` -- which should be\n",
" located in the folder path given by `workspace.project_root`. By convention, when loaded this is referred to as the\n",
" `storyboard` object.\n",
"\n",
" If everything is set up correctly, you should be able to load the currently configured workspace and storyboard via:\n",
"\n",
" workspace, storyboard = load_storyboard()\n",
" \"\"\"\n",
" # yeah... so... this shouldn't be necessary....\n",
" import os\n",
"\n",
" # infer if we're on colab or not, since this impacts gdrive mounting\n",
" try:\n",
" import google.colab\n",
" local=False\n",
" except:\n",
" local=True\n",
"\n",
" if local:\n",
" mount_gdrive=False\n",
"\n",
" # Infer directory locations\n",
" os.environ['XDG_CACHE_HOME'] = os.environ.get(\n",
" 'XDG_CACHE_HOME',\n",
" str(Path('~/.cache').expanduser())\n",
" )\n",
" if mount_gdrive:\n",
" from google.colab import drive\n",
" drive.mount('/content/drive')\n",
" Path('/content/drive/MyDrive/AI/models/.cache/').mkdir(parents=True, exist_ok=True)\n",
" os.environ['XDG_CACHE_HOME']='/content/drive/MyDrive/AI/models/.cache'\n",
"\n",
" model_dir_str=str(Path(os.environ['XDG_CACHE_HOME']))\n",
" proj_root_str = '${active_project}'\n",
" application_root = str(Path('.').absolute())\n",
" if mount_gdrive:\n",
" application_root = '/content/drive/MyDrive/AI/VideoKilledTheRadioStar'\n",
"\n",
"\n",
" # Build config file that defines the \"workspace\" abstraction\n",
" workspace = OmegaConf.create({\n",
" 'active_project': active_project if active_project else str(time.time()),\n",
" 'application_root':application_root,\n",
" 'project_root':\"${application_root}/${active_project}\",\n",
" 'shared_assets_root':\"${application_root}/shared_assets\",\n",
" 'gdrive_mounted':mount_gdrive,\n",
" 'use_stability_api':use_stability_api,\n",
" 'model_dir':model_dir_str,\n",
" 'output_dir':'${project_root}/frames'\n",
" })\n",
"\n",
" Path(workspace.project_root).mkdir(parents=True, exist_ok=True)\n",
" Path(workspace.model_dir).mkdir(parents=True, exist_ok=True)\n",
" Path(workspace.output_dir).mkdir(parents=True, exist_ok=True)\n",
"\n",
" ###################\n",
"\n",
" # Assign tracking locations for A/V assets and generally useful outputs\n",
"\n",
" assets_dir = Path(workspace.shared_assets_root)\n",
" assets_dir.mkdir(parents=True, exist_ok=True)\n",
"\n",
" # TODO: yaml -> jsonl ?\n",
" video_assets_meta_fname = assets_dir / 'video_assets_meta.yaml'\n",
" if not video_assets_meta_fname.exists():\n",
" video_assets_meta = OmegaConf.create()\n",
" video_assets_meta.videos = []\n",
" with video_assets_meta_fname.open('w') as fp:\n",
" OmegaConf.save(config=video_assets_meta, f=fp.name)\n",
" else:\n",
" video_assets_meta = OmegaConf.load(video_assets_meta_fname)\n",
"\n",
" audio_assets_meta_fname = assets_dir / 'audio_assets_meta.yaml'\n",
" if not audio_assets_meta_fname.exists():\n",
" audio_assets_meta = OmegaConf.create()\n",
" audio_assets_meta.content = []\n",
" with audio_assets_meta_fname.open('w') as fp:\n",
" OmegaConf.save(config=audio_assets_meta, f=fp.name)\n",
" else:\n",
" audio_assets_meta = OmegaConf.load(audio_assets_meta_fname)\n",
"\n",
" ###################\n",
"\n",
" # Request user provide credentials as needed\n",
"\n",
" # if use_stability_api:\n",
" # import os, getpass\n",
" # if not os.environ.get('STABILITY_KEY'):\n",
" # os.environ['STABILITY_KEY'] = getpass.getpass('Enter your Stability API Key, then press enter to continue')\n",
" # else:\n",
" # # TODO: check for HF token in environment\n",
" # if not local:\n",
" # from google.colab import output\n",
" # output.enable_custom_widget_manager()\n",
"\n",
" # from huggingface_hub import notebook_login\n",
" # notebook_login()\n",
"\n",
" ###################\n",
"\n",
" with open('config.yaml','w') as fp:\n",
" OmegaConf.save(config=workspace, f=fp.name)\n",
"\n",
" return workspace\n",
"\n",
"########################\n",
"\n",
"# wrap some of the loading logic for portability\n",
"\n",
"\n",
"def resolve_scene_ids_and_start_end_times(storyboard):\n",
" \"\"\"\n",
" 1. Force first scene to start at frame 0\n",
" 2. Force last scene to end in accordance with duration\n",
" 3. Force each scene's `end` attr to correspond to the `start` attr of the subsequent scene\n",
" 4. Force scene_id to correspond to scenes index position\n",
" \"\"\"\n",
" # nothing to see here, move along.\n",
" if not storyboard.params.get('video_duration'):\n",
" return storyboard\n",
"\n",
" storyboard.prompt_starts[0]['start']=0\n",
" storyboard.prompt_starts[-1]['end']=max(storyboard.params.video_duration, storyboard.prompt_starts[-1]['end'])\n",
" for idx, rec in enumerate(storyboard.prompt_starts):\n",
" rec['scene_id']=idx\n",
" if idx==0:\n",
" prev_rec = rec\n",
" continue\n",
" prev_rec['end'] = rec['start']\n",
" for rec in storyboard.prompt_starts:\n",
" rec['duration_'] = rec['end'] - rec['start']\n",
" return storyboard\n",
"\n",
"def save_storyboard(storyboard):\n",
" #if storyboard.params.get('video_duration'):\n",
" if storyboard.prompt_starts:\n",
" storyboard = resolve_scene_ids_and_start_end_times(storyboard)\n",
" root = Path(load_workspace().project_root)\n",
" root.mkdir(parents=True, exist_ok=True)\n",
" storyboard_fname = root / 'storyboard.yaml'\n",
" with open(storyboard_fname, 'w') as fp:\n",
" OmegaConf.save(config=storyboard, f=fp.name)\n",
"\n",
"def load_workspace():\n",
" return OmegaConf.load('config.yaml')\n",
"\n",
"def load_storyboard():\n",
" workspace = load_workspace()\n",
" root = Path(workspace.project_root)\n",
" storyboard_fname = root / 'storyboard.yaml'\n",
" storyboard = OmegaConf.load(storyboard_fname)\n",
" return workspace, storyboard\n",
"\n",
"def load_audio_meta(workspace, storyboard):\n",
" assets_dir = Path(workspace.shared_assets_root)\n",
" audio_assets_meta_fname = assets_dir / 'audio_assets_meta.yaml'\n",
" audio_assets_meta = OmegaConf.load(audio_assets_meta_fname)\n",
" audio_meta=dict()\n",
" for idx, rec in enumerate(audio_assets_meta.content):\n",
" if rec.audio_fpath == storyboard.params.audio_fpath:\n",
" audio_meta = rec\n",
" break\n",
" return audio_meta\n",
"\n",
"#######################\n",
"\n",
"# EXTRA SEGMENTATION STUFF\n",
"\n",
"\n",
"def calculate_interword_gaps(segment):\n",
" end_prev = -1\n",
" gaps = []\n",
" for word in segment['words']:\n",
" if end_prev < 0:\n",
" end_prev = word['end']\n",
" continue\n",
" gap = word['start'] - end_prev\n",
" gaps.append(gap)\n",
" end_prev = word['end']\n",
" return gaps\n",
"\n",
"def trivial_subsegmentation(segment, threshold=0, gaps=None):\n",
" \"\"\"\n",
" split on gaps in detected vocal activity.\n",
" Contiguity = gap between adjacent tokens is less than the input threshold.\n",
" \"\"\"\n",
" if gaps is None:\n",
" gaps = calculate_interword_gaps(seg)\n",
" out_segments = []\n",
" this_segment = [seg['words'][0]]\n",
" for word, preceding_pause in zip(seg['words'][1:], gaps):\n",
" if preceding_pause <= threshold:\n",
" this_segment.append(word)\n",
" else:\n",
" out_segments.append(this_segment)\n",
" this_segment = [word]\n",
" out_segments.append(this_segment)\n",
"\n",
" outv = [dict(\n",
" start=seg[0]['start'],\n",
" end=seg[-1]['end'],\n",
" text=''.join([w['word'] for w in seg]).strip(),\n",
" ) for seg in out_segments]\n",
"\n",
" return outv\n",
"\n",
"##############################################################\n",
"\n",
"# audio processing\n",
"\n",
"\n",
"def analyze_audio_structure(\n",
" audio_fpath,\n",
" BINS_PER_OCTAVE = 12 * 3, # should be a multiple of twelve: https://github.com/MTG/essentia/blob/master/src/examples/python/tutorial_spectral_constantq-nsg.ipynb\n",
" N_OCTAVES = 7,\n",
"):\n",
" \"\"\"\n",
" via librosa docs\n",
" https://librosa.org/doc/latest/auto_examples/plot_segmentation.html#sphx-glr-auto-examples-plot-segmentation-py\n",
" cites: McFee and Ellis, 2014 - https://brianmcfee.net/papers/ismir2014_spectral.pdf\n",
" \"\"\"\n",
" y, sr = librosa.load(audio_fpath)\n",
"\n",
" C = librosa.amplitude_to_db(np.abs(librosa.cqt(y=y, sr=sr,\n",
" bins_per_octave=BINS_PER_OCTAVE,\n",
" n_bins=N_OCTAVES * BINS_PER_OCTAVE)),\n",
" ref=np.max)\n",
"\n",
" # reduce dimensionality via beat-synchronization\n",
" tempo, beats = librosa.beat.beat_track(y=y, sr=sr, trim=False)\n",
" Csync = librosa.util.sync(C, beats, aggregate=np.median)\n",
"\n",
" # I have concerns about this frame fixing operation\n",
" beat_times = librosa.frames_to_time(librosa.util.fix_frames(beats, x_min=0), sr=sr)\n",
"\n",
" # width=3 prevents links within the same bar\n",
" # mode=’affinity’ here implements S_rep (after Eq. 8)\n",
" R = librosa.segment.recurrence_matrix(Csync, width=3, mode='affinity', sym=True)\n",
" # Enhance diagonals with a median filter (Equation 2)\n",
" df = librosa.segment.timelag_filter(scipy.ndimage.median_filter)\n",
" Rf = df(R, size=(1, 7))\n",
" # build the sequence matrix (S_loc) using mfcc-similarity\n",
" mfcc = librosa.feature.mfcc(y=y, sr=sr)\n",
" Msync = librosa.util.sync(mfcc, beats)\n",
" path_distance = np.sum(np.diff(Msync, axis=1)**2, axis=0)\n",
" sigma = np.median(path_distance)\n",
" path_sim = np.exp(-path_distance / sigma)\n",
" R_path = np.diag(path_sim, k=1) + np.diag(path_sim, k=-1)\n",
" # compute the balanced combination\n",
" deg_path = np.sum(R_path, axis=1)\n",
" deg_rec = np.sum(Rf, axis=1)\n",
" mu = deg_path.dot(deg_path + deg_rec) / np.sum((deg_path + deg_rec)**2)\n",
" A = mu * Rf + (1 - mu) * R_path\n",
"\n",
" # compute normalized laplacian and its spectrum\n",
" L = scipy.sparse.csgraph.laplacian(A, normed=True)\n",
" evals, evecs = scipy.linalg.eigh(L)\n",
" # clean this up with a median filter. can help smooth over discontinuities\n",
" evecs = scipy.ndimage.median_filter(evecs, size=(9, 1))\n",
" return dict(\n",
" y=y,\n",
" sr=np.array(sr).astype(np.uint32),\n",
" tempo=tempo,\n",
" beats=beats,\n",
" beat_times=beat_times,\n",
" evecs=evecs,\n",
" )\n",
"\n",
"\n",
"def laplacian_segmentation(\n",
" audio_fpath=None,\n",
" evecs=None,\n",
" n_clusters = 5,\n",
" n_spectral_features = None,\n",
"):\n",
" \"\"\"\n",
" segment audio by clustering a self-similarity matrix.\n",
" via librosa docs\n",
" https://librosa.org/doc/latest/auto_examples/plot_segmentation.html#sphx-glr-auto-examples-plot-segmentation-py\n",
" cites: McFee and Ellis, 2014 - https://brianmcfee.net/papers/ismir2014_spectral.pdf\n",
" \"\"\"\n",
" if evecs is None:\n",
" if audio_fpath is None:\n",
" raise Exception(\"One of `audio_fpath` or `evecs` must be provided\")\n",
" features = analyze_audio_structure(audio_fpath)\n",
" evecs = features['evecs']\n",
"\n",
" if n_clusters < 2:\n",
" seg_ids = np.zeros(evecs.shape[0], dtype=int)\n",
" return seg_ids\n",
"\n",
" if n_spectral_features is None:\n",
" n_spectral_features = n_clusters\n",
"\n",
" # cumulative normalization is needed for symmetric normalize laplacian eigenvectors\n",
" Cnorm = np.cumsum(evecs**2, axis=1)**0.5\n",
" k = n_spectral_features\n",
" X = evecs[:, :k] / Cnorm[:, k-1:k]\n",
"\n",
"\n",
" # use these k components to cluster beats into segments\n",
" KM = sklearn.cluster.KMeans(n_clusters=n_clusters, n_init=\"auto\")\n",
" seg_ids = KM.fit_predict(X)\n",
"\n",
" return seg_ids #, beat_times, tempo\n",
"\n",
"\n",
"# for video duration\n",
"def get_audio_duration_seconds(audio_fpath):\n",
" outv = subprocess.run([\n",
" 'ffprobe'\n",
" ,'-i',audio_fpath\n",
" ,'-show_entries', 'format=duration'\n",
" ,'-v','quiet'\n",
" ,'-of','csv=p=0'\n",
" ],\n",
" stdout=subprocess.PIPE\n",
" ).stdout.decode('utf-8')\n",
" return float(outv.strip())\n",
"\n",
"\n",
"##########################################\n",
"\n",
"# animation stuff\n",
"\n",
"# TODO: update this stuff to reflect updates to API/sdk\n",
"def get_image_for_prompt_sai(prompt, max_retries=5, **kargs):\n",
" stability_api = client.StabilityInference(\n",
" key=os.environ['STABILITY_KEY'],\n",
" verbose=False,\n",
" )\n",
"\n",
" # auto-retry if mitigation triggered\n",
" while max_retries:\n",
" try:\n",
" answers = stability_api.generate(prompt=prompt, **kargs)\n",
" response = process_response(answers)\n",
" for img in response:\n",
" yield img\n",
" break\n",
"\n",
" # TODO: better regen handling\n",
" except RuntimeError:\n",
" print(\"runtime error\")\n",
" max_retries -= 1\n",
" warnings.warn(f\"mitigation triggered, retries remaining: {max_retries}\")\n",
"\n",
"def process_response(answers):\n",
" for resp in answers:\n",
" for artifact in resp.artifacts:\n",
" if artifact.finish_reason == generation.FILTER:\n",
" warnings.warn(\n",
" \"Your request activated the API's safety filters and could not be processed.\"\n",
" \"Please modify the prompt and try again.\")\n",
" raise RuntimeError\n",
" if artifact.type == generation.ARTIFACT_IMAGE:\n",
" img = Image.open(io.BytesIO(artifact.binary))\n",
" yield img\n",
"\n",
"\n",
"########################################\n",
"\n",
"# misc utils\n",
"\n",
"def rand_str(n_char=5):\n",
" return ''.join(random.choice(string.ascii_lowercase) for i in range(n_char))\n",
"\n",
"def save_frame(\n",
" img: Image,\n",
" idx:int=0,\n",
" root_path=Path('./frames'),\n",
" name=None,\n",
"):\n",
" root_path.mkdir(parents=True, exist_ok=True)\n",
" if name is None:\n",
" name = rand_str()\n",
" outpath = root_path / f\"{idx}-{name}.png\"\n",
" img.save(outpath)\n",
" return str(outpath)\n",
"\n",
"def get_image_sequence(idx, root, init_first=True):\n",
" root = Path(root)\n",
" images = (root / 'frames' ).glob(f'{idx}-*.png')\n",
" images = [str(fp) for fp in images]\n",
" if init_first:\n",
" init_image = None\n",
" images2 = []\n",
" for i, fp in enumerate(images):\n",
" if 'anchor' in fp:\n",
" init_image = fp\n",
" else:\n",
" images2.append(fp)\n",
" if not init_image:\n",
" try:\n",
" init_image, images2 = images2[0], images2[1:]\n",
" images = [init_image] + images2\n",
" except IndexError:\n",
" images = images2\n",
" return images\n",
"\n",
"def archive_images(idx, root, archive_root = None):\n",
" root = Path(root)\n",
" if archive_root is None:\n",
" archive_root = root / 'archive'\n",
" archive_root = Path(archive_root)\n",
" archive_root.mkdir(parents=True, exist_ok=True)\n",
" old_images = get_image_sequence(idx, root=root)\n",
" if not old_images:\n",
" return\n",
" print(f\"moving {len(old_images)} old images for scene {idx} to {archive_root}\")\n",
" for old_fp in old_images:\n",
" old_fp = Path(old_fp)\n",
" im_name = Path(old_fp.name)\n",
" new_path = archive_root / im_name\n",
" if new_path.exists():\n",
" im_name = f\"{im_name.stem}-{time.time()}{im_name.suffix}\"\n",
" new_path = archive_root / im_name\n",
" old_fp.rename(new_path)\n",
"\n",
"\n",
"############################\n",
"\n",
"# video compilation stuff\n",
"\n",
"# TODO: Sorting algorithm that can tolerate more than 15-ish frames (GPU?)\n",
"def tsp_sort(frames):\n",
" frames_m = np.array([np.array(f).ravel() for f in frames])\n",
" dmat = pdist(frames_m, metric='cosine')\n",
" dmat = squareform(dmat)\n",
" permutation, _ = solve_tsp_dynamic_programming(dmat)\n",
" return permutation\n",
"\n",
"def add_caption2image(\n",
" image,\n",
" caption,\n",
" text_font='LiberationSans-Regular.ttf',\n",
" font_size=20,\n",
" fill_color=(255, 255, 255),\n",
" stroke_color=(0, 0, 0), #stroke_fill\n",
" stroke_width=2,\n",
" align='center',\n",
" ):\n",
" # via https://stackoverflow.com/a/59104505/819544\n",
" wrapper = textwrap.TextWrapper(width=50)\n",
" word_list = wrapper.wrap(text=caption)\n",
" caption_new = ''\n",
" for ii in word_list[:-1]:\n",
" caption_new = caption_new + ii + '\\n'\n",
" caption_new += word_list[-1]\n",
"\n",
" draw = ImageDraw.Draw(image)\n",
"\n",
" # Download the Font and Replace the font with the font file.\n",
" font = ImageFont.truetype(text_font, size=font_size)\n",
" w,h = draw.textsize(caption_new, font=font, stroke_width=stroke_width)\n",
" W,H = image.size\n",
" x,y = 0.5*(W-w),0.90*H-h\n",
" draw.text(\n",
" (x,y),\n",
" caption_new,\n",
" font=font,\n",
" fill=fill_color,\n",
" stroke_fill=stroke_color,\n",
" stroke_width=stroke_width,\n",
" align=align,\n",
" )\n",
"\n",
" return image\n",
"\n",
"##########################################################\n",
"\n",
"# audioreactivity stuff\n",
"\n",
"\n",
"def full_width_plot():\n",
" ax = plt.gca()\n",
" ax.figure.set_figwidth(20)\n",
" plt.show()\n",
"\n",
"def display_signal(y, sr, show_spec=True, title=None, start_time=0, end_time=9999):\n",
"\n",
"# if show_spec:\n",
"# frame_time = librosa.samples_to_time(np.arange(len(normalized_signal)), sr=sr)\n",
"# else:\n",
"# frame_time = librosa.frames_to_time(np.arange(len(normalized_signal)), sr=sr)\n",
"\n",
" if show_spec:\n",
" #librosa.display.waveshow(y, sr=sr)\n",
" times = librosa.samples_to_time(np.arange(len(y)), sr=sr)\n",
" else:\n",
" #times = librosa.times_like(y, sr=sr).ravel()\n",
" times = librosa.frames_to_time(np.arange(len(y)), sr=sr).ravel()\n",
"\n",
" start_idx = np.argmax(start_time <= times)\n",
" #end_idx = len(times) - np.argmax([end_time <= times][::-1])\n",
" end_idx = np.argmax(end_time <= times)\n",
" if start_idx >= end_idx:\n",
" end_idx = -1\n",
"\n",
" times = times[start_idx:end_idx]\n",
" y = y[start_idx:end_idx]\n",
"\n",
" plt.plot(times, y)\n",
" if title:\n",
" plt.title(title)\n",
" full_width_plot()\n",
"\n",
" if show_spec:\n",
" try:\n",
" M = librosa.feature.melspectrogram(y=y, sr=sr)\n",
" librosa.display.specshow(librosa.power_to_db(M, ref=np.max),\n",
" y_axis='mel', x_axis='time')\n",
" full_width_plot()\n",
"\n",
" except:\n",
" pass\n",
"\n",
" # plt.plot(frame_time, y)\n",
" # if title:\n",
" # plt.title(title)\n",
" # full_width_plot()\n",
"\n",
"\n",
"# https://github.com/pytti-tools/pytti-core/blob/9e8568365cfdc123d2d2fbc20d676ca0f8715341/src/pytti/AudioParse.py#L95\n",
"from scipy.signal import butter, sosfilt, sosfreqz\n",
"\n",
"def butter_bandpass(lowcut, highcut, fs, order):\n",
" nyq = 0.5 * fs\n",
" low = lowcut / nyq\n",
" high = highcut / nyq\n",
" sos = butter(order, [low, high], analog=False, btype='bandpass', output='sos')\n",
" return sos\n",
"\n",
"def butter_bandpass_filter(y, sr, lowcut, highcut, order=10):\n",
" sos = butter_bandpass(lowcut, highcut, sr, order=order)\n",
" y = sosfilt(sos, y)\n",
" return y\n",
"\n",
"########################################################################\n",
"\n",
"def is_multi_valued_curve(curve_str):\n",
" try:\n",
" return (\": (\" in curve_str) and (\"), \" in curve_str)\n",
" except:\n",
" return False\n",
"\n",
"def show_storyboard(storyboard=None):\n",
" if storyboard is None:\n",
" workspace, storyboard = load_storyboard()\n",
" storyboard = resolve_scene_ids_and_start_end_times(storyboard)\n",
"\n",
" # TODO: fix this...\n",
" reactive_signal_map = {}\n",
" if storyboard.get('audioreactive'):\n",
" reactive_signal_map = storyboard.audioreactive.get('reactive_signal_map')\n",
"\n",
" # TODO: should just invoke compile for basically all of this.\n",
" try:\n",
" # ... borks on ifps...\n",
" settings = compile_storyboard(ignore_defaults=True)\n",
" except:\n",
" settings = []\n",
"\n",
" for idx, rec in enumerate(storyboard.prompt_starts):\n",
" report = f\"scene: {idx}\\t start: {rec['start']:.2f}\"\n",
" if rec.get('duration_'):\n",
" report += f\"\\t duration: {rec.get('duration_'):.2f}\"\n",
" report += f\"\\nspoken text: {rec.get('text')}\\n\"\n",
"\n",
" # TODO: wrap prompt construction logic in a function (better yet use omegaconf substitution variables)\n",
"\n",
" #'_theme':'theme', 'structural_segmentation_label':\n",
" if rec.get('_theme'):\n",
" report += f\"theme prompt: {rec['_theme']}\\n\"\n",
" #f\"image prompt: {rec['_prompt']}\\n\"\n",
" prompt = rec.get('prompt')\n",
" #if not prompt:\n",
" # prompt = ...\n",
" if prompt:\n",
" report += f\"image prompt: {rec['_prompt']}\\n\"\n",
"\n",
" if rec.get('animation_mode'):\n",
" report += f\"animation mode: {rec['animation_mode']}\"\n",
" print(report)\n",
" im_path = rec.get('frame0_fpath')\n",
" if im_path and Path(im_path).exists():\n",
" display(Image.open(rec['frame0_fpath']))\n",
"\n",
" #if reactive_signal_map:\n",
" n = rec['frames']\n",
" if n <1:\n",
" continue\n",
"\n",
" if not settings:\n",
" continue\n",
" scene_settings = settings[idx]\n",
" #for signal_name in reactive_signal_map.keys():\n",
" for signal_name, signal_val in scene_settings.items():\n",
" if not is_multi_valued_curve(signal_val):\n",
" continue\n",
"\n",
" curve = kf.dsl.curve_from_cn_string(signal_val)\n",
" xs = [i for i in range(n)]\n",
" ys = [curve[i] for i in xs]\n",
" plt.plot(xs, ys, label=signal_name)\n",
" plt.title(f\"scene {idx} - {signal_name}\")\n",
" plt.xlabel(\"frame index within scene\")\n",
" plt.legend()\n",
" plt.show()\n",
"\n",
"#########################################\n",
"\n",
"def get_path_to_stems():\n",
" workspace, storyboard = load_storyboard()\n",
" assets_root = Path(workspace.application_root) / 'shared_assets'\n",
" #stems_path = root / \"stems\"\n",
" stems_path = assets_root / \"stems\"\n",
" stems_outpath = stems_path / 'htdemucs_ft' / Path(storyboard.params.audio_fpath).stem\n",
" return stems_outpath\n",
"\n",
"def ensure_stems_separated():\n",
" stems_outpath = get_path_to_stems()\n",
" stems_path = str(stems_outpath.parent.parent)\n",
" if not stems_outpath.exists():\n",
" !demucs -n htdemucs_ft -o \"{stems_path}\" \"{storyboard.params.audio_fpath}\"\n",
"\n",
"def get_stem(instrument_name):\n",
" ensure_stems_separated()\n",
" stems_outpath = get_path_to_stems()\n",
" stem_fpaths = list(stems_outpath.glob('*.wav'))\n",
"\n",
" for stem_fpath in stem_fpaths:\n",
" if instrument_name in str(stem_fpath):\n",
" y, sr = librosa.load(stem_fpath)\n",
" return y, sr\n",
" raise ValueError(\n",
" f\"Unable to locate stem for instrument: {instrument_name}\\n\"\n",
" f\"in folder: {stems_outpath}\"\n",
" )\n",
"\n",
"\n",
"##########################################################################################################\n",
"\n",
"# deforum compatibility sprint\n",
"\n",
"\n",
"import math\n",
"import numpy as np\n",
"\n",
"def build_eval_scope(storyboard):\n",
" # preload eval scope with math stuff\n",
" math_env = {\n",
" \"abs\": abs,\n",
" \"max\": max,\n",
" \"min\": min,\n",
" \"pow\": pow,\n",
" \"round\": round,\n",
" \"np\": np,\n",
" \"__builtins__\": None,\n",
" }\n",
" math_env.update(\n",
" {key: getattr(math, key) for key in dir(math) if \"_\" not in key}\n",
" )\n",
"\n",
" # add signals to scope\n",
" for signal_name, sig_curve in storyboard.signals.items():\n",
" sig_curve = OmegaConf.to_container(sig_curve) # zomg...\n",
" curve = load_curve(sig_curve)\n",
" math_env[signal_name] = curve\n",
" return math_env\n",
"\n",
"\n",
"#eval(signal_mappings['noise_curve'], math_env, t=0)\n",
"#math_env['t']=0\n",
"#eval(signal_mappings['noise_curve'], math_env)\n",
"\n",
"\n",
"#################\n",
"\n",
"\n",
"true=True\n",
"false=False\n",
"DEFORUM_DEFAULTS = {\n",
" \"W\": 512,\n",
" \"H\": 512,\n",
" \"show_info_on_ui\": true,\n",
" \"tiling\": false,\n",
" \"restore_faces\": false,\n",
" \"seed_resize_from_w\": 0,\n",
" \"seed_resize_from_h\": 0,\n",
" \"seed\": -1,\n",
" \"sampler\": \"Euler a\",\n",
" \"steps\": 25,\n",
" \"batch_name\": \"Deforum_{timestring}\",\n",
" \"seed_behavior\": \"iter\",\n",
" \"seed_iter_N\": 1,\n",
" \"use_init\": false,\n",
" \"strength\": 0.8,\n",
" \"strength_0_no_init\": true,\n",
" \"init_image\": \"https://deforum.github.io/a1/I1.png\",\n",
" \"use_mask\": false,\n",
" \"use_alpha_as_mask\": false,\n",
" \"mask_file\": \"https://deforum.github.io/a1/M1.jpg\",\n",
" \"invert_mask\": false,\n",
" \"mask_contrast_adjust\": 1.0,\n",
" \"mask_brightness_adjust\": 1.0,\n",
" \"overlay_mask\": true,\n",
" \"mask_overlay_blur\": 4,\n",
" \"fill\": 1,\n",
" \"full_res_mask\": true,\n",
" \"full_res_mask_padding\": 4,\n",
" \"reroll_blank_frames\": \"ignore\",\n",
" \"reroll_patience\": 10.0,\n",
" # \"prompts\": {\n",
" # \"0\": \"tiny cute bunny, vibrant diffraction, highly detailed, intricate, ultra hd, sharp photo, crepuscular rays, in focus, by tomasz alen kopera\",\n",
" # \"30\": \"anthropomorphic clean cat, surrounded by fractals, epic angle and pose, symmetrical, 3d, depth of field, ruan jia and fenghua zhong\",\n",
" # \"60\": \"a beautiful coconut --neg photo, realistic\",\n",
" # \"90\": \"a beautiful durian, trending on Artstation\"\n",
" # },\n",
" \"animation_prompts_positive\": \"\",\n",
" \"animation_prompts_negative\": \"nsfw, nude\",\n",
" \"animation_mode\": \"2D\",\n",
" \"max_frames\": 1,\n",
" \"border\": \"replicate\",\n",
" \"angle\": \"0: (0)\",\n",
" \"zoom\": \"0: (1.0025+0.002*sin(1.25*3.14*t/30))\",\n",
" \"translation_x\": \"0: (0)\",\n",
" \"translation_y\": \"0: (0)\",\n",
" \"translation_z\": \"0: (1.75)\",\n",
" \"transform_center_x\": \"0: (0.5)\",\n",
" \"transform_center_y\": \"0: (0.5)\",\n",
" \"rotation_3d_x\": \"0: (0)\",\n",
" \"rotation_3d_y\": \"0: (0)\",\n",
" \"rotation_3d_z\": \"0: (0)\",\n",
" \"enable_perspective_flip\": false,\n",
" \"perspective_flip_theta\": \"0: (0)\",\n",
" \"perspective_flip_phi\": \"0: (0)\",\n",
" \"perspective_flip_gamma\": \"0: (0)\",\n",
" \"perspective_flip_fv\": \"0: (53)\",\n",
" \"noise_schedule\": \"0: (0.065)\",\n",
" \"strength_schedule\": \"0: (0.65)\",\n",
" \"contrast_schedule\": \"0: (1.0)\",\n",
" \"cfg_scale_schedule\": \"0: (7)\",\n",
" \"enable_steps_scheduling\": false,\n",
" \"steps_schedule\": \"0: (25)\",\n",
" \"fov_schedule\": \"0: (70)\",\n",
" \"aspect_ratio_schedule\": \"0: (1)\",\n",
" \"aspect_ratio_use_old_formula\": false,\n",
" \"near_schedule\": \"0: (200)\",\n",
" \"far_schedule\": \"0: (10000)\",\n",
" \"seed_schedule\": \"0:(s), 1:(-1), \\\"max_f-2\\\":(-1), \\\"max_f-1\\\":(s)\",\n",
" \"pix2pix_img_cfg_scale_schedule\": \"0:(1.5)\",\n",
" \"enable_subseed_scheduling\": false,\n",
" \"subseed_schedule\": \"0: (1)\",\n",
" \"subseed_strength_schedule\": \"0: (0)\",\n",
" \"enable_sampler_scheduling\": false,\n",
" \"sampler_schedule\": \"0: (\\\"Euler a\\\")\",\n",
" \"use_noise_mask\": false,\n",
" \"mask_schedule\": \"0: (\\\"{video_mask}\\\")\",\n",
" \"noise_mask_schedule\": \"0: (\\\"{video_mask}\\\")\",\n",
" \"enable_checkpoint_scheduling\": false,\n",
" \"checkpoint_schedule\": \"0: (\\\"model1.ckpt\\\"), 100: (\\\"model2.safetensors\\\")\",\n",
" \"enable_clipskip_scheduling\": false,\n",
" \"clipskip_schedule\": \"0: (2)\",\n",
" \"enable_noise_multiplier_scheduling\": true,\n",
" \"noise_multiplier_schedule\": \"0: (1.05)\",\n",
" \"resume_from_timestring\": false,\n",
" \"resume_timestring\": \"20230621175028\",\n",
" \"enable_ddim_eta_scheduling\": false,\n",
" \"ddim_eta_schedule\": \"0: (0)\",\n",
" \"enable_ancestral_eta_scheduling\": false,\n",
" \"ancestral_eta_schedule\": \"0: (1)\",\n",
" \"amount_schedule\": \"0: (0.1)\",\n",
" \"kernel_schedule\": \"0: (5)\",\n",
" \"sigma_schedule\": \"0: (1)\",\n",
" \"threshold_schedule\": \"0: (0)\",\n",
" \"color_coherence\": \"LAB\",\n",
" \"color_coherence_image_path\": \"\",\n",
" \"color_coherence_video_every_N_frames\": 1,\n",
" \"color_force_grayscale\": false,\n",
" \"legacy_colormatch\": false,\n",
" \"diffusion_cadence\": 2,\n",
" \"optical_flow_cadence\": \"None\",\n",
" \"cadence_flow_factor_schedule\": \"0: (1)\",\n",
" \"optical_flow_redo_generation\": \"None\",\n",
" \"redo_flow_factor_schedule\": \"0: (1)\",\n",
" \"diffusion_redo\": \"0\",\n",
" \"noise_type\": \"perlin\",\n",
" \"perlin_octaves\": 4,\n",
" \"perlin_persistence\": 0.5,\n",
" \"use_depth_warping\": true,\n",
" \"depth_algorithm\": \"Midas-3-Hybrid\",\n",
" \"midas_weight\": 0.2,\n",
" \"padding_mode\": \"border\",\n",
" \"sampling_mode\": \"bicubic\",\n",
" \"save_depth_maps\": false,\n",
" \"video_init_path\": \"https://deforum.github.io/a1/V1.mp4\",\n",
" \"extract_nth_frame\": 1,\n",
" \"extract_from_frame\": 0,\n",
" \"extract_to_frame\": -1,\n",
" \"overwrite_extracted_frames\": false,\n",
" \"use_mask_video\": false,\n",
" \"video_mask_path\": \"https://deforum.github.io/a1/VM1.mp4\",\n",
" \"hybrid_comp_alpha_schedule\": \"0:(0.5)\",\n",
" \"hybrid_comp_mask_blend_alpha_schedule\": \"0:(0.5)\",\n",
" \"hybrid_comp_mask_contrast_schedule\": \"0:(1)\",\n",
" \"hybrid_comp_mask_auto_contrast_cutoff_high_schedule\": \"0:(100)\",\n",
" \"hybrid_comp_mask_auto_contrast_cutoff_low_schedule\": \"0:(0)\",\n",
" \"hybrid_flow_factor_schedule\": \"0:(1)\",\n",
" \"hybrid_generate_inputframes\": false,\n",
" \"hybrid_generate_human_masks\": \"None\",\n",
" \"hybrid_use_first_frame_as_init_image\": true,\n",
" \"hybrid_motion\": \"None\",\n",
" \"hybrid_motion_use_prev_img\": false,\n",
" \"hybrid_flow_consistency\": false,\n",
" \"hybrid_consistency_blur\": 2,\n",
" \"hybrid_flow_method\": \"RAFT\",\n",
" \"hybrid_composite\": \"None\",\n",
" \"hybrid_use_init_image\": false,\n",
" \"hybrid_comp_mask_type\": \"None\",\n",
" \"hybrid_comp_mask_inverse\": false,\n",
" \"hybrid_comp_mask_equalize\": \"None\",\n",
" \"hybrid_comp_mask_auto_contrast\": false,\n",
" \"hybrid_comp_save_extra_frames\": false,\n",
" \"parseq_manifest\": \"\",\n",
" \"parseq_use_deltas\": true,\n",
" \"use_looper\": false,\n",
" \"init_images\": \"{\\n \\\"0\\\": \\\"https://deforum.github.io/a1/Gi1.png\\\",\\n \\\"max_f/4-5\\\": \\\"https://deforum.github.io/a1/Gi2.png\\\",\\n \\\"max_f/2-10\\\": \\\"https://deforum.github.io/a1/Gi3.png\\\",\\n \\\"3*max_f/4-15\\\": \\\"https://deforum.github.io/a1/Gi4.jpg\\\",\\n \\\"max_f-20\\\": \\\"https://deforum.github.io/a1/Gi1.png\\\"\\n}\",\n",
" \"image_strength_schedule\": \"0:(0.75)\",\n",
" \"blendFactorMax\": \"0:(0.35)\",\n",
" \"blendFactorSlope\": \"0:(0.25)\",\n",
" \"tweening_frames_schedule\": \"0:(20)\",\n",
" \"color_correction_factor\": \"0:(0.075)\",\n",
" \"skip_video_creation\": false,\n",
" \"fps\": 15,\n",
" \"make_gif\": false,\n",
" \"delete_imgs\": false,\n",
" \"add_soundtrack\": \"None\",\n",
" \"soundtrack_path\": \"https://deforum.github.io/a1/A1.mp3\",\n",
" \"r_upscale_video\": false,\n",
" \"r_upscale_factor\": \"x2\",\n",
" \"r_upscale_model\": \"realesr-animevideov3\",\n",
" \"r_upscale_keep_imgs\": true,\n",
" \"store_frames_in_ram\": false,\n",
" \"frame_interpolation_engine\": \"None\",\n",
" \"frame_interpolation_x_amount\": 2,\n",
" \"frame_interpolation_slow_mo_enabled\": false,\n",
" \"frame_interpolation_slow_mo_amount\": 2,\n",
" \"frame_interpolation_keep_imgs\": true,\n",
" \"frame_interpolation_use_upscaled\": false,\n",
" \"sd_model_name\": \"v1-5-pruned-emaonly.ckpt\",\n",
" \"sd_model_hash\": \"81761151\",\n",
" \"deforum_git_commit_id\": \"b58056f9\",\n",
"}\n",
"\n",
"\n",
"def resolve_prompt(idx, storyboard):\n",
" prompt_lag = storyboard.params.get(\"prompt_lag\",True)\n",
" rec = storyboard.prompt_starts[idx]\n",
" if not rec.get('_prompt'):\n",
" theme = rec.get('_theme')\n",
" prompt = rec.get('prompt')\n",
" if not prompt:\n",
" prompt = f\"{rec['text']}, {theme}\"\n",
"\n",
" if prompt_lag and (idx > 0):\n",
" rec_prev = storyboard.prompt_starts[idx -1]\n",
" prev_text = rec_prev.get('text','')\n",
" if not prev_text:\n",
" prev_text = rec_prev.get('prompt','').split(',')[0]\n",
" this_text = rec.get('text','')\n",
" if this_text:\n",
" prompt = f\"{prev_text} {this_text}, {theme}\"\n",
" else:\n",
" prompt = rec_prev['_prompt']\n",
" rec['_prompt'] = prompt\n",
"\n",
"\n",
"def resolve_signals(scene_id=0, storyboard=None):\n",
" if storyboard is None:\n",
" _, storyboard = load_storyboard()\n",
" idx=scene_id\n",
" rec = storyboard.prompt_starts[scene_id]\n",
" # resolve signals\n",
" default_mappings = storyboard.get('signal_mappings',{})\n",
" signal_mappings = rec.get('signal_mappings', default_mappings)\n",
"\n",
" math_env = build_eval_scope(storyboard)\n",
" curves = {}\n",
" for param_name, param_meta in signal_mappings.items():\n",
" param_expr, attr_hi, attr_low = param_meta['parameter_value'], param_meta['param_max'], param_meta['param_min']\n",
" curve_chunks = []\n",
" start=rec['start']\n",
" for frame_idx in range(rec['frames']):\n",
" curr_time = start + frame_idx * ifps\n",
" # SUPPORTED VARIABLES\n",
" # TODO: describe this somewhere\n",
" math_env['t'] = curr_time\n",
" math_env['scene_id'] = idx\n",
" math_env['frame_id'] = frame_idx\n",
" math_env['theme_id'] = rec.get('structural_segmentation_label',0)\n",
" signal_value=eval(param_expr, math_env)\n",
" #if inverse_relationship:\n",
" if param_meta['invert_relationship_to_signal']:\n",
" signal_value = 1-signal_value\n",
" attr_value = signal_value*(attr_hi-attr_low)+attr_low\n",
" curve_chunks.append(f\"{frame_idx}: ({attr_value})\")\n",
" curve_str = ', '.join(curve_chunks)\n",
" #scene_args[param_name] = curve_str\n",
" #scene_args = OmegaConf.to_container(scene_args)\n",
" curves[param_name] = curve_str\n",
" return curves\n",
"\n",
"\n",
"def compile_storyboard(storyboard=None, ignore_defaults=False):\n",
" if storyboard is None:\n",
" _, storyboard = load_storyboard()\n",
" scenes = []\n",
" story_defaults = storyboard.get('animation_args',{})\n",
" for idx, rec in enumerate(storyboard.prompt_starts):\n",
"\n",
" # establish defaults\n",
" scene_args=copy.deepcopy(story_defaults)\n",
" if not ignore_defaults:\n",
" scene_args = rec.get('animation_args',{})\n",
" try:\n",
" scene_args = OmegaConf.to_container(scene_args)\n",
" except ValueError:\n",
" pass\n",
"\n",
" # translate settings to deforum names\n",
" resolve_prompt(idx, storyboard)\n",
" #scene_args['prompts'] = {\"0\":rec['_prompt']}\n",
" #scene_args['max_frames'] = rec['frames']\n",
" scene_args['_prompt'] = rec['_prompt']\n",
" scene_args['frames'] = rec['frames']\n",
" scene_args['init_image'] = rec.get('frame0_fpath')\n",
"\n",
" curves = resolve_signals(idx, storyboard)\n",
" scene_args.update(curves)\n",
" scenes.append(scene_args)\n",
" return scenes\n",
"\n",
"# TODO: copy init images into this folder, add to settings.txt's\n",
"\n",
"\n",
"\n",
"def make_compatible_for_deforum(settings, backfill_deforum_defaults = True):\n",
" fields_mapping={\n",
" #'prompts':'prompts',\n",
" 'fps':'fs',\n",
" 'extract_nth_frame':'extract_nth_frame',\n",
" 'angle':'angle',\n",
" 'zoom':'zoom',\n",
" 'translation_x': 'translation_x',\n",
" 'translation_y': 'translation_y',\n",
" 'translation_z': 'translation_z',\n",
" 'rotation_x': 'rotation_3d_x',\n",
" 'rotation_y': 'rotation_3d_y',\n",
" 'rotation_z': 'rotation_3d_z',\n",
" 'frames':'max_frames',\n",
" #'noise_add_curve':'noise_schedule',\n",
" 'noise_curve':'noise_schedule',\n",
" 'noise_scale_curve':'noise_multiplier_curve',\n",
" 'strength_curve':'strength_schedule',\n",
" 'steps_curve':'steps_schedule',\n",
" 'srength_curve': 'strength_schedule',\n",
" 'steps_curve': 'steps_schedule',\n",
" }\n",
" # SAI_scalars_2_deforum_curves = {\n",
" # 'cfg_scale':'cfg_scale_schedule',\n",
" # 'seed':'seed_schedule',\n",
" # }\n",
" #keep_params = set(fields_mapping) + set(SAI_scalars_2_deforum_curves)\n",
"\n",
" outv = {}\n",
" if backfill_deforum_defaults:\n",
" outv.update(DEFORUM_DEFAULTS )\n",
"\n",
" for k,v in settings.items():\n",
" if k in fields_mapping:\n",
" outv[fields_mapping[k]] = v\n",
" elif k == '_prompt':\n",
" outv['prompts'] = {\"0\":v}\n",
" elif k == 'cfg_scale':\n",
" outv['cfg_scale_schedule'] = f\"0: ({v})\"\n",
" elif k == 'seed':\n",
" if v == -1:\n",
" outv['seed_schedule'] = \"0: (-1)\"\n",
" else:\n",
" seed_schedule_chunks = []\n",
" for i in range(settings['max_frames']):\n",
" #if v == -1:\n",
" # v = random.randrange(0, 4294967295)\n",
" seed_schedule_chunks.append(f\"{i}: ({v})\")\n",
" v+=1\n",
" outv['seed_schedule'] = ', '.join(seed_schedule_chunks)\n",
" return outv\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PknSJ48jAmuP"
},
"source": [
"## 1. 📋💬 Build Base Storyboard\n",
"* Initial setup\n",
"* Infer keyframes for scene segmentation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "cM8cux9b7F4v",
"tags": []
},
"outputs": [],
"source": [
"# @title ### 📋 Attach Storyboard (create or resume project)\n",
"\n",
"project_name = '' # @param {type:'string'}\n",
"\n",
"# TODO: make it so I can change this value without restarting the kernel....\n",
"use_stability_api = False # @param {type:'boolean'}\n",
"mount_gdrive = True # @param {type:'boolean'}\n",
"resume=True # @param {type:'boolean'}\n",
"\n",
"# TODO: add support for whisper API\n",
"\n",
"# @markdown Enter a unique `project_name`.\n",
"# @markdown If left blank, the current unix timestamp will be used\n",
"# @markdown (seconds since 1970-01-01 00:00).\n",
"\n",
"# @markdown If you use the name of an existing project, the workspace will switch to that project (even if `resume` is unchecked. Each project needs a unique name).\n",
"\n",
"# @markdown Non-alphanumeric characters (excluding '-' and '_') will be replaced with hyphens.\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown # Detailed Discussion\n",
"# @markdown In VKTRS, a \"project\" is encapsulated by a folder.\n",
"# @markdown With google drive loaded, it will be the `<project_name>` subfolder under the path:\n",
"# @markdown `/content/drive/MyDrive/AI/VideoKilledTheRadioStar/`\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown Depending on your settings and environment, running this cell may prompt you to enter one or more API Keys below.\n",
"# @markdown Don't forget to press \"enter\" after providing a requested key.\n",
"\n",
"\n",
"##########################\n",
"\n",
"try:\n",
" import google.colab\n",
" local=False\n",
"except:\n",
" local=True\n",
"\n",
"if local:\n",
" mount_gdrive=False\n",
"\n",
"\n",
"##################################################################\n",
"\n",
"resuming = False\n",
"if resume:\n",
" try:\n",
" workspace, storyboard = load_storyboard()\n",
" print(\"loading storyboard\")\n",
" resuming=True\n",
" except:\n",
" resuming = False\n",
"\n",
"if not resuming:\n",
" if not project_name:\n",
" project_name = str(time.time())\n",
" project_name = sanitize_folder_name(project_name)\n",
"\n",
" print(\"creating workspace\")\n",
"\n",
" workspace = establish_workspace(\n",
" use_stability_api=use_stability_api,\n",
" mount_gdrive=mount_gdrive,\n",
" application_name=\"VideoKilledTheRadioStar\",\n",
" active_project=project_name,\n",
" )\n",
"\n",
" print(\"creating new storyboard\")\n",
" storyboard = OmegaConf.create()\n",
" storyboard.params = {}\n",
" storyboard.prompt_starts=[]\n",
"\n",
"\n",
"# f.\n",
"workspace.use_stability_api = use_stability_api\n",
"with open('config.yaml','w') as fp:\n",
" OmegaConf.save(config=workspace, f=fp.name)\n",
"\n",
"#if workspace.use_stability_api:\n",
"if use_stability_api:\n",
"\n",
" import os, getpass\n",
" if not os.environ.get('STABILITY_KEY'):\n",
" os.environ['STABILITY_KEY'] = getpass.getpass('Enter your Stability API Key, then press enter to continue')\n",
"\n",
"\n",
" # @markdown To get your API key visit https://dreamstudio.ai/account\n",
" STABILITY_HOST = \"grpc.stability.ai:443\" #@param {type:\"string\"}\n",
"\n",
"\n",
"\n",
" ###################################\n",
"\n",
" STABILITY_KEY = os.environ.get('STABILITY_KEY')\n",
"\n",
" # Connect to Stability API\n",
" context = Context(STABILITY_HOST, STABILITY_KEY)\n",
"\n",
" # Test the connection\n",
" context.get_user_info()\n",
"\n",
"\n",
"\n",
"else:\n",
" # TODO: check for HF token in environment\n",
" if not local:\n",
" from google.colab import output\n",
" output.enable_custom_widget_manager()\n",
"\n",
" from huggingface_hub import notebook_login\n",
" notebook_login()\n",
"\n",
"##################################################################\n",
"\n",
"root = Path(workspace.project_root)\n",
"\n",
"assets_dir = Path(workspace.shared_assets_root)\n",
"\n",
"video_assets_meta_fname = assets_dir / 'video_assets_meta.yaml'\n",
"audio_assets_meta_fname = assets_dir / 'audio_assets_meta.yaml'\n",
"\n",
"video_assets_meta = OmegaConf.load( assets_dir/'video_assets_meta.yaml' )\n",
"audio_assets_meta = OmegaConf.load( assets_dir/'audio_assets_meta.yaml' )\n",
"\n",
"# just to be safe\n",
"save_storyboard(storyboard)\n",
"workspace, storyboard = load_storyboard()\n",
"\n",
"import rich\n",
"rich.print(dict(workspace))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "SyIJRMhEgCfL",
"tags": []
},
"outputs": [],
"source": [
"# @title ⬇ Set Audio Source\n",
"\n",
"d_ = dict(\n",
" # all the underscore does is make it so each of the following lines can be preceded with a comma\n",
" # otw the first parameter would be offset from the other in the colab form\n",
" _=\"\"\n",
"\n",
" , video_url = 'https://www.youtube.com/watch?v=REojIUxX4rw' # @param {type:'string'}\n",
" , audio_fpath = '' # @param {type:'string'}\n",
")\n",
"d_.pop('_')\n",
"\n",
"# @markdown `video_url` - URL of a youtube video to download as a source for audio and potentially for text transcription as well.\n",
"\n",
"# @markdown `audio_fpath` - Optionally provide an audio file instead of relying on a youtube download. Name it something other than 'audio.mp3',\n",
"# @markdown otherwise it might get overwritten accidentally.\n",
"\n",
"\n",
"storyboard.params = d_\n",
"\n",
"storyboard_fname = root / 'storyboard.yaml'\n",
"with open(storyboard_fname,'wb') as fp:\n",
" OmegaConf.save(config=storyboard, f=fp.name)\n",
"\n",
"\n",
"###############################\n",
"# Download audio from youtube #\n",
"###############################\n",
"\n",
"# this should modify the existing record for the URL rather than creating a new one...\n",
"force_redownload=False\n",
"\n",
"video_url = storyboard.params.video_url\n",
"download_video=True\n",
"\n",
"if not force_redownload:\n",
" for rec in video_assets_meta.videos:\n",
" if rec.video_url == video_url:\n",
" if rec.get('video_fpath'):\n",
" print(\"previously downloaded video detected\")\n",
" download_video=False\n",
" # populate storyboard with previous processing results\n",
" if rec.get('audio_fpath'):\n",
" storyboard.params.audio_fpath = rec.get('audio_fpath')\n",
" else:\n",
" download_video=True # should be redundant?\n",
" break\n",
"\n",
"\n",
"\n",
"if download_video:\n",
" # check if user provided an audio filepath (or we already have one from youtube) before attempting to download\n",
" video_assets_meta_record = {}\n",
" video_assets_meta_record['video_url'] = video_url\n",
"\n",
" ytdl_prefix = \"DOWNLOADED__\"\n",
" ytdl_fname = f\"{str(assets_dir / ytdl_prefix)}%(title)s.%(ext)s\"\n",
"\n",
" !yt-dlp -o \"{ytdl_fname}\" {video_url}\n",
"\n",
" matched_files = assets_dir.glob(ytdl_prefix+\"*\")\n",
" most_recent_file = max(matched_files, key=os.path.getctime)\n",
" print(f\"downloaded: {most_recent_file}\")\n",
" ytdl_fname = most_recent_file\n",
"\n",
" video_assets_meta_record['video_fpath'] = str(ytdl_fname.absolute())\n",
"\n",
" # TODO: right here, we should be adding this to the audio meta\n",
" audio_fpath = ytdl_fname.with_suffix('.m4a')\n",
" input_audio = ytdl_fname\n",
" !ffmpeg -y -i \"{input_audio}\" -vn -c:a aac \"{audio_fpath}\"\n",
"\n",
" storyboard.params.audio_fpath = audio_fpath\n",
" video_assets_meta_record['audio_fpath'] = str(audio_fpath.absolute())\n",
"\n",
" video_assets_meta.videos.append(video_assets_meta_record)\n",
"\n",
"\n",
"save_storyboard(storyboard)\n",
"\n",
"with open(video_assets_meta_fname, 'wb') as fp:\n",
" OmegaConf.save(config=video_assets_meta, f=fp.name)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "9zT0u4-q_fMF",
"tags": []
},
"outputs": [],
"source": [
"# @title ### 🔊 Initial Audio Processing\n",
"\n",
"# @markdown * Transcribe and segment speech using whisper\n",
"\n",
"# @markdown * Analyze major musical structure\n",
"\n",
"# @markdown If this audio source has been previoussly processed by this notebook, that should be detected and processing won't be repeated.\n",
"\n",
"##################################################\n",
"# 💬 Transcribe and segment speech using whisper #\n",
"##################################################\n",
"\n",
"audio_fpath = str(storyboard.params.audio_fpath)\n",
"\n",
"separate_stems = False # @param {type:'boolean'}\n",
"\n",
"# these are just confusing i think\n",
"force_retranscription = False # #@#param {type:'boolean'}\n",
"override_storyboard_transcription = False # #@#param {type:'boolean'}\n",
"\n",
"whisper_seg = None\n",
"\n",
"if audio_fpath in audio_assets_meta:\n",
" print(\"previously processed audio detected\")\n",
" audio_meta = audio_assets_meta[audio_fpath]\n",
"else:\n",
" audio_assets_meta[audio_fpath] = {}\n",
" audio_meta = audio_assets_meta[audio_fpath]\n",
"\n",
"if (not force_retranscription) and audio_meta.get('whisper_segmentation') and Path(audio_meta.whisper_segmentation).exists():\n",
" print(\"Using pre-existing whisper transcription\")\n",
" whisper_seg_fpath = Path(audio_meta.whisper_segmentation)\n",
" with whisper_seg_fpath.open() as f:\n",
" timings = json.load(f)\n",
" whisper_seg = timings['segments']\n",
"\n",
"\n",
"if force_retranscription or (whisper_seg is None):\n",
" print(\"Transcribing...\")\n",
" #audio_meta['audio_fpath'] = storyboard.params.audio_fpath redundant\n",
" # outputs text files as audio.* locally\n",
" !whisper --model large --word_timestamps True -o {str(assets_dir)} \"{storyboard.params.audio_fpath}\"\n",
"\n",
" whisper_seg_fpath = Path(storyboard.params.audio_fpath).with_suffix('.json')\n",
" audio_meta['whisper_segmentation'] = str(whisper_seg_fpath)\n",
" audio_meta['duration'] = get_audio_duration_seconds(audio_fpath)\n",
"\n",
" with whisper_seg_fpath.open() as f:\n",
" timings = json.load(f)\n",
" whisper_seg = timings['segments']\n",
"\n",
" audio_assets_meta[audio_fpath] = audio_meta\n",
" with open(audio_assets_meta_fname, 'wb') as fp:\n",
" OmegaConf.save(config=audio_assets_meta, f=fp.name)\n",
"\n",
"if not storyboard.get('prompt_starts') or override_storyboard_transcription:\n",
"\n",
" # Ta da!\n",
" storyboard.prompt_starts = [{k:rec[k] for k in ('start','end','text')} for rec in whisper_seg]\n",
"\n",
" storyboard.params['video_duration'] = audio_meta['duration']\n",
" # unsure if below method is reliable.\n",
" #storyboard.params['video_duration'] = storyboard.prompt_starts[-1]['end']\n",
"\n",
"\n",
"# TODO: enforce scene zero starts at t=0 and last scene ends at duration\n",
"\n",
"#storyboard.prompt_starts = prompt_starts\n",
"save_storyboard(storyboard)\n",
"\n",
"# Music Structure Analysis\n",
"\n",
"# Music Structure Analysis\n",
"# - beat and tempo detection\n",
"# - Self-similarity graph\n",
"\n",
"audio_structure_features = analyze_audio_structure(audio_fpath=storyboard.params.audio_fpath)\n",
"\n",
"audio_features_fpath = Path(storyboard.params.audio_fpath).with_suffix('.audio_features.safetensors')\n",
"save_safetensors(audio_structure_features, audio_features_fpath)\n",
"\n",
"# TODO: fix inconsistent variable naming\n",
"structural_features = audio_structure_features\n",
"\n",
"# TODO: un-\"PosixPath\" `storyboard.params.audio_fpath`\n",
"# TODO: this is a confusing af way to access this.\n",
"audio_assets_meta[str(storyboard.params.audio_fpath)]['structural_features'] = audio_features_fpath\n",
"\n",
"\n",
"with open(audio_assets_meta_fname, 'wb') as fp:\n",
" OmegaConf.save(config=audio_assets_meta, f=fp.name)\n",
"\n",
"if separate_stems:\n",
" ensure_stems_separated()\n",
"\n",
"# TODO: ~PLACEHOLDER~ Give user opportunity to correct the transcription at shared_asset level rather than project\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "IcKwmF1BUyup"
},
"outputs": [],
"source": [
"# @title ### 🎞️ Show Current Storyboard\n",
"workspace, storyboard = load_storyboard() # just to ensure consistency\n",
"show_storyboard()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "B3gw1cYNgCfL",
"tags": []
},
"outputs": [],
"source": [
"# @title ### ✂️ Subdivide Unusually Long Scenes\n",
"\n",
"# @markdown Attempts to identify unusually long scenes and splits them into multiple shorter scenes.\n",
"# @markdown Estimates the expected scene duration to identify outliers.\n",
"\n",
"# TODO: wrap these steps in functions for legibility/portability\n",
"# TODO: make this threshold parameterizable via storyboard (and save analysis to storyboard)\n",
"# TODO: use beat counts to estimate a smart scene duration\n",
"\n",
"subdivide_long_scenes = True # @param {'type':'boolean'}\n",
"\n",
"#TODO: merge short scenes\n",
"\n",
"# TODO: expose this to colab\n",
"threshold_duration = None\n",
"\n",
"######################################################\n",
"# estimate parameters of scene duration distribution #\n",
"######################################################\n",
"\n",
"# TODO: use beat onsets/counts\n",
"scene_durations = []\n",
"scenes_ = []\n",
"for idx, rec in enumerate(storyboard.prompt_starts):\n",
" rec=dict(rec)\n",
" if idx > 0:\n",
" # are we maybe doubling up 'start' time stamps? like there are more unique 'end's than 'start's?\n",
" duration = rec['start'] - prev['start']\n",
" prev['duration_'] = duration\n",
" scene_durations.append(duration)\n",
" prev = rec\n",
" scenes_.append(prev)\n",
"\n",
"# handle last record\n",
"else:\n",
" duration = rec['end'] - rec['start']\n",
" rec['duration_'] = duration\n",
" scenes_.append(rec)\n",
" scene_durations.append(duration)\n",
"\n",
"# handle case where there's only one scene\n",
"if idx == 0:\n",
" duration = rec['end'] - rec['start']\n",
" rec['duration_'] = duration\n",
" scenes_.append(rec)\n",
" scene_durations.append(duration)\n",
"\n",
"mu = sum(scene_durations)/len(scene_durations)\n",
"sigma = np.std(scene_durations)\n",
"\n",
"# 1sd filter to concentrate on mode\n",
"scene_durations2 = [s for s in scene_durations if (mu - sigma) < s < (mu+sigma)]\n",
"try:\n",
" mu2 = sum(scene_durations2)/len(scene_durations2)\n",
" sigma2 = np.std(scene_durations2)\n",
" if mu2==0:\n",
" subdivide_long_scenes=False\n",
"except ZeroDivisionError:\n",
" subdivide_long_scenes = False\n",
" warnings.warn(\n",
" \"Sorry, you've hit an unsupported edge case. \"\n",
" \"We need some sort of segmentation to start with, but right now \"\n",
" \"our initial heuristics have everything in one long scene. \"\n",
" \"If you're trying to animate an instrumental track, VKTRS \"\n",
" \"doesn't intelligent scene segmentation for that use case yet. \"\n",
" \"Scene segmentation needs lyrics at the moment.\"\n",
" )\n",
"\n",
"###########\n",
"\n",
"# break up \"outlier\" segments into smaller chunks\n",
"# this heuristic could be improved with beat synchronization and onset detection.\n",
"# ... also could probably leverage the 'end' time of the scene\n",
"# TODO: hierarchical theme structure analysis\n",
"# TODO: MSA segmentation for fully instrumental (i.e. arbitrary) audio\n",
"\n",
"if subdivide_long_scenes:\n",
"\n",
" threshhold = mu2 + sigma\n",
" if threshold_duration is not None:\n",
" threshhold = threshold_duration\n",
"\n",
" scenes = []\n",
" #for rec in storyboard.prompt_starts:\n",
" for rec in list(scenes_):\n",
" gap_remaining = rec['duration_']\n",
" while gap_remaining > threshhold:\n",
" #step = min(max(mu2-sigma, (np.random.normal() + mu2)*sigma), mu2+sigma)\n",
" step=mu2\n",
" step = float(step)\n",
" # TODO: move duration computation somewhere that it will happen necessarily\n",
" rec['duration_'] = step\n",
" new_rec = copy.deepcopy(rec)\n",
" new_rec['start'] = rec['start'] + step\n",
" # TODO: deal with new value here\n",
" #new_rec['end'] = ??\n",
" new_rec['duration_'] = step\n",
" ### maybe i could add a flag or something to clarify that this was an \"inferred\" subscene\n",
" #new_rec['parent_scene'] = rec.get('uid') # something like this?\n",
" new_rec['inferred_subscene'] = True # or this?\n",
"\n",
" scenes.append(rec)\n",
" rec = new_rec\n",
" gap_remaining -= step\n",
" scenes.append(rec)\n",
"\n",
" storyboard.prompt_starts = scenes\n",
"\n",
"# TODO: get last scene \"end\" from story duration\n",
"\n",
"# TODO: compute duration regardless of extra segmentation\n",
"\n",
"# TODO: add scene indices back to rec's to facilitate editing the text file\n",
"\n",
"save_storyboard(storyboard)\n",
"show_storyboard(storyboard)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "5LrrcvX2gCfM",
"tags": []
},
"outputs": [],
"source": [
"# @title ### 🤓 Math\n",
"\n",
"# @markdown Construct an initial estimate for how many frames will be needed for\n",
"# @markdown each scene, based on the scene start times and animation framerate.\n",
"\n",
"# @markdown * `fps` - The desired framerate of the animation (frames per second).\n",
"# @markdown * `animation_mode=\"img2img\"` - We will generate an image for every frame, so higher fps will take longer but improves audioreactivity.\n",
"# @markdown * `animation_mode=\"variations\"` - We will only need to generate `n_variations` images per scene. The FPS setting only affects how the video will be compiled.\n",
"# @markdown The fps and n_variations settings will interact to define the frequency of a visual pulsing effect, e.g.\n",
"# @markdown * fps=12, n_variations=4 : the animation will appear to \"pulse\" 3 times per second as the sequence of image variations is cycled.\n",
"\n",
"#################################################\n",
"# Math #\n",
"# #\n",
"# This block computes how many frames are #\n",
"# needed for each segment based on the start #\n",
"# times for each prompt #\n",
"#################################################\n",
"\n",
"# TODO: move variations settings up here\n",
"\n",
"# TODO: leverage previous beat detection, onsets, etc. for frame timings\n",
"# - TODO: isolated cells for calculating and suggesting parameters (fps, n_variations)\n",
"\n",
"# TODO: experiment with tying instantaneous framerate to a music attribute (i.e. so motion can change speed mid animation)\n",
"\n",
"fps = 30 # @param {type:'integer'}\n",
"storyboard.params.fps = fps\n",
"\n",
"ifps = 1/fps\n",
"\n",
"# estimate video end\n",
"if not storyboard.params.get('video_duration'):\n",
" storyboard.params['video_duration'] = get_audio_duration_seconds(storyboard.params.audio_fpath)\n",
"video_duration = storyboard.params['video_duration']\n",
"\n",
"# dummy prompt for last scene duration\n",
"prompt_starts = OmegaConf.to_container(storyboard.prompt_starts)\n",
"prompt_starts.append({'start':video_duration})\n",
"\n",
"# TODO: do we still need anim_start? is that used in rendering?\n",
"# TODO: add per-frame timings to incorporate beat information\n",
"# make sure we respect the duration of the previous phrase\n",
"frame_start=0\n",
"prompt_starts[0]['anim_start']=frame_start\n",
"for i, rec in enumerate(prompt_starts[1:], start=1):\n",
" rec_prev = prompt_starts[i-1]\n",
" k=0\n",
" while (rec_prev['anim_start'] + k*ifps) < rec['start']:\n",
" k+=1\n",
" k-=1\n",
" rec_prev['frames'] = k\n",
" rec_prev['anim_duration'] = k*ifps\n",
" frame_start+=k*ifps\n",
" rec['anim_start']=frame_start\n",
"\n",
"# drop the dummy frame\n",
"prompt_starts = prompt_starts[:-1]\n",
"\n",
"# TODO: given a 0 duration prompt, assume its duration is captured in the next prompt\n",
"# and guesstimate a corrected prompt start time and duration\n",
"# - or rather.. why are there ever sero duration prompts?\n",
"\n",
"\n",
"storyboard.prompt_starts = prompt_starts\n",
"\n",
"save_storyboard(storyboard)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TcCOlx_vCo31"
},
"source": [
"## $2.$ 🎥🌈 Define your animation's aesthetics"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Bu1FKypFO8C2"
},
"source": [
"### 2.1 📋 Assign Themes to Scenes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "SB0X2mXGgCfL",
"tags": []
},
"outputs": [],
"source": [
"\n",
"\n",
"# @markdown `theme_prompt` - Text that will be appended to the end of each lyric, useful for e.g. applying a consistent aesthetic style.\n",
"# @markdown To provide multiple themes (only one will be used per scene), separate theme prompts with the `|` (pipe) symbol.\n",
"\n",
"# @markdown `infer_thematic_structure` - if False, themes will be rotated sequentially such that no two adjacent frames\n",
"# @markdown will use the same theme prompt (if multiple theme prompts were provided). If True, song structure analysis will cluster related scenes into as many groups as there are theme prompts to attempt to associate a visual themes with respective musical themes.\n",
"\n",
"# @markdown The analysis runs quick. If you didn't understand that explanation, just try it both ways and you'll probably get the idea.\n",
"\n",
"theme_prompt = 'zombie wedding | zombie dance party | zombies in love' # @param {type:'string'}\n",
"\n",
"infer_thematic_structure = True # @param {type:'boolean'}\n",
"\n",
"storyboard.params.theme_prompt = theme_prompt\n",
"themes = [prompt.strip() for prompt in theme_prompt.split('|') if prompt.strip()]\n",
"\n",
"# communicate theme_id mappings\n",
"df_themes = pd.DataFrame({'theme prompt':themes})\n",
"df_themes.index.name=\"theme_id\"\n",
"\n",
"rich.print(df_themes)\n",
"print()\n",
"\n",
"if (len(themes) > 1):\n",
" if infer_thematic_structure:\n",
"\n",
" # audio_assets_meta[storyboard.params.audio_fpath]['structural_features']\n",
" beat_times = audio_structure_features['beat_times']\n",
" evecs = audio_structure_features['evecs']\n",
" segment_labels = laplacian_segmentation(\n",
" evecs=evecs,\n",
"\n",
" # TODO: publish these parameters to the user\n",
" n_clusters=len(themes), # be sure to explain this is an upper bound, stochastic\n",
" n_spectral_features=len(themes),\n",
" )\n",
"\n",
" # beatsynch scene start times\n",
"\n",
" # TODO: swap out rec['end'] -> rec_prev['start'] here\n",
" for rec in storyboard.prompt_starts:\n",
" beat_indices = np.where((beat_times >= rec['start']) & (beat_times <= rec['end']))[0]\n",
" segments_this_interval = segment_labels[beat_indices]\n",
" if len(segments_this_interval) == 0:\n",
" dominant_label = 0\n",
" else:\n",
" dominant_label = int(np.argmax(np.bincount(segments_this_interval)))\n",
" rec['structural_segmentation_label'] = dominant_label\n",
" rec['_theme'] = themes[dominant_label]\n",
" else:\n",
" for rec in storyboard.prompt_starts:\n",
" rec['_theme'] = themes[idx % len(themes)]\n",
"else:\n",
" for rec in storyboard.prompt_starts:\n",
" rec['_theme'] = theme_prompt\n",
"\n",
"\n",
"save_storyboard(storyboard)\n",
"\n",
"# TODO: show dataframe if no init images?\n",
"show_storyboard(storyboard)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "zQvgGq3SgCfM",
"tags": []
},
"outputs": [],
"source": [
"# @title #### 📝 (Optional) modify theme prompt without impacting structure label assignments\n",
"\n",
"# @markdown Themes will be assigned to the `structural_segmentation` label that maps to their ordering\n",
"# @markdown in the theme prompt. To change which theme goes where, simply modify the order in which\n",
"# @markdown they appear in your prompt.\n",
"\n",
"# @markdown **Why the second cell?** The procedure that assigns labels to themes\n",
"# @markdown has some randomness, so re-running the cell above with the same\n",
"# @markdown settings might produce a different order (i.e. a different theme-label assignment)\n",
"\n",
"theme_prompt = 'rusted industrial machinery | kaiju robot CGI | paperclips! paperclips! | robotics for beginners | rusted industrial machinery' # @param {type:'string'}\n",
"\n",
"#####################################################\n",
"\n",
"storyboard.params.theme_prompt = theme_prompt\n",
"themes = [prompt.strip() for prompt in theme_prompt.split('|') if prompt.strip()]\n",
"\n",
"for rec in storyboard.prompt_starts:\n",
" theme_idx = rec.get('structural_segmentation_label',0)\n",
" rec['_theme'] = themes[theme_idx]\n",
"\n",
"save_storyboard(storyboard)\n",
"show_storyboard(storyboard)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sYkcY1XPAFUF"
},
"source": [
"### 2.2 🎛️🎚️ Animation Settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "edxBKv5TgCfM",
"tags": []
},
"outputs": [],
"source": [
"# @title #### 🎚️ (Advanced) `img2img` Animation settings\n",
"\n",
"# @markdown NB: Skip this cell if you're not using the `img2img` animation mode. Go directly to the next cell, \"📌\".\n",
"\n",
"# TODO: let's move `variations` settings into their own dedicated area,\n",
"# call this \"`img2img` animation settings\" or something like that instead\n",
"\n",
"# @markdown Run This cell to reveal a multi-tab UI for specifying `img2img` animation settings. When you are happy with your settings,\n",
"# @markdown run the cell which follows this one to attach the settings you have selected to one or more scenes/themes.\n",
"\n",
"# @markdown To reset values to default, simply re-run this cell.\n",
"\n",
"# @markdown The `img2img` animation mode is currently only supported through the Stability.AI Animation API (i.e. DreamStudio API).\n",
"# @markdown The `img2img` animation parameters are mostly compatible with deforum: if you don't want to animate via the Stability.AI,\n",
"# @markdown you can still use this notebook to configure your animation and then port the relevant settings from the `storyboard.yaml`\n",
"# @markdown to your tool of choice.\n",
"\n",
"if not workspace.use_stability_api:\n",
" warnings.warn(\"img2img animation currently only supported if you're using the stability API\")\n",
" arg_objs=[]\n",
"else:\n",
"\n",
"\n",
" show_documentation = True # @param {type:'boolean'}\n",
" # TODO: Move/add context to setup cell, and/or generation cells\n",
"\n",
"\n",
" ###################\n",
"\n",
" args_generation = BasicSettings()\n",
" args_animation = AnimationSettings()\n",
" args_camera = CameraSettings()\n",
" args_coherence = CoherenceSettings()\n",
" args_color = ColorSettings()\n",
" args_depth = DepthSettings()\n",
" args_render_3d = Rendering3dSettings()\n",
" args_inpaint = InpaintingSettings()\n",
" args_vid_in = VideoInputSettings()\n",
" args_vid_out = VideoOutputSettings()\n",
" arg_objs = (\n",
" args_generation,\n",
" args_animation,\n",
" args_camera,\n",
" args_coherence,\n",
" args_color,\n",
" args_depth,\n",
" args_render_3d,\n",
" args_inpaint,\n",
" args_vid_in,\n",
" args_vid_out,\n",
" )\n",
"\n",
" def _show_docs(component):\n",
" cols = []\n",
" for k, v in component.param.objects().items():\n",
" if k == 'name':\n",
" continue\n",
" col = pn.Column(v, v.doc)\n",
" cols.append(col)\n",
" return pn.Column(*cols)\n",
"\n",
" def build(component):\n",
" if show_documentation:\n",
" component = _show_docs(component)\n",
" return pn.Row(component, width=1000)\n",
"\n",
"pn.extension()\n",
"\n",
"pn.Tabs(*[(a.name[:-5], build(a)) for a in arg_objs])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BX3TBm3bFoY5"
},
"source": [
"### 2.3 📌 Attach settings to scenes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "efgDV40OVO9S",
"tags": []
},
"outputs": [],
"source": [
"# @markdown If using img2img animation, use this cell to specify which scenes the settings you provided above should apply to.\n",
"\n",
"# @markdown Otherwise, you can ignore the cell above and just do everything here.\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown `override_storyboard` - If settings conflict with values already set on the storyboard, the values on the storyboard will take priority.\n",
"\n",
"# @markdown `scene_ids` - Comma separated list of integers specifying scenes to attach to.\n",
"\n",
"# @markdown `theme_ids` - Comma separated list of integers specifying themes the animation parameters should be associated with.\n",
"\n",
"# @markdown NB: both `scene_ids` and `theme_ids` are zero-indexed.\n",
"\n",
"# @markdown If you're not sure what the appropriate scene/theme ids are that you want to reference, use the next cell to load and view the current storyboard state.\n",
"\n",
"# CURRENT ARGS APPLY TO...\n",
"\n",
"all_scenes = True # @param {'type':'boolean'}\n",
"\n",
"scene_ids = '' # @param {'type':'string'}\n",
"theme_ids = '' # @param {'type':'string'}\n",
"\n",
"override_storyboard = True # @param {'type':'boolean'}\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown ## Animation Modes.\n",
"# @markdown * `static`: turns off animation, static image for duration of scene.\n",
"# @markdown * `variations`: injects a small bit of \"life\" into the image. Cheap and fast.\n",
"# @markdown * `variations tsp`: variations animation with frames reordered for smoother motion. Cheap and fast, but a tad slower.\n",
"# @markdown * `img2img`: Fancy deforum-esque animation, only supported for stability api at present. Neither cheap nor fast.\n",
"# #@markdown * `default`: `img2img` if stability api enabled, `variations tsp` if not.\n",
"\n",
"# oh baby `jittered init`\n",
"# markdown * `jittered init`: The VKTRS special! Sample a variation and use that as an init image for img2img. Basically built for audioreactivity\n",
"# to do --> combine this with a referring expression mask\n",
"# make sure animation is configured so we can curve prompts\n",
"\n",
"animation_mode = 'variations' # @param [\"static\", \"variations\", \"variations tsp\", \"img2img\"]\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown ## Parameters for `variations` Animation Modes\n",
"\n",
"# TODO: these parameters should get set elsewhere. maybe with animation mode?\n",
"\n",
"# @markdown `n_variations` - How many unique variations to generate for a given text prompt. This determines the frequency of the visual \"pulsing\" effect\n",
"\n",
"# @markdown `image_consistency` - controls similarity between images generated by the prompt.\n",
"# @markdown - 0: ignore the init image\n",
"# @markdown - 1: true as possible to the init image\n",
"\n",
"## @markdown `max_video_duration_in_seconds` - Early stopping if you don't want to generate a video the full duration of the provided audio. Default = 5min.\n",
"\n",
"\n",
"n_variations=8 # @param {type:'integer'}\n",
"image_consistency=0.72 # @param {type:\"slider\", min:0, max:1, step:0.01}\n",
"\n",
"variations_settings = {\n",
" 'n_variations':n_variations,\n",
" 'image_consistency':image_consistency,\n",
"}\n",
"\n",
"\n",
"# let's try this:\n",
"# - user specifies animation stuff in cell above\n",
"# - then user comes down here and persists the animation settings either to every scene, or to specific scenes.\n",
"# - scene specification can just be a list of numbers\n",
"# - feels like this should be moved after init image generation. maybe push init image generation up to run right after theme prompts and scene count is finalized?\n",
"\n",
"# TODO: add storyboard -> parsec converter\n",
"\n",
"# TODO: PR to deforum for storyboard.yaml support\n",
"\n",
"# TODO: allow user to constrain attention to specific parameters or sets of parameters to set\n",
"\n",
"def param2json(args_obj):\n",
" args = args_obj.param.serialize_parameters()\n",
" args = json.loads(args)\n",
" args.pop('name')\n",
" return args\n",
"\n",
"def collect_animation_args():\n",
" if 'arg_objs' in globals():\n",
" args_d = {}\n",
" [args_d.update(a.param.values()) for a in arg_objs]\n",
" args=AnimationArgs(**args_d)\n",
" else:\n",
" import warnings\n",
" warnings.warn(\n",
" \"Looks like you're animating in img2img mode \"\n",
" \"without having specified any animation parameters. \"\n",
" \"Using SDK defaults. \"\n",
" )\n",
" args = AnimationArgs()\n",
" return param2json(args)\n",
"\n",
"\n",
"##########################\n",
"\n",
"scene_ids = [int(v.strip()) for v in scene_ids.split(',') if v]\n",
"theme_ids = [int(v.strip()) for v in theme_ids.split(',') if v]\n",
"\n",
"# build list of scenes args will apply to\n",
"\n",
"applicable_scenes = []\n",
"if all_scenes:\n",
" applicable_scenes = [idx for idx, _ in enumerate(storyboard.prompt_starts)]\n",
"else:\n",
" applicable_scenes += scene_ids\n",
" applicable_scenes += theme_ids\n",
"\n",
"###---------------------------------###\n",
"\n",
"animation_mode = animation_mode.lower()\n",
"if animation_mode == 'default':\n",
" animation_mode = 'img2img' if workspace.use_stability_api else 'variations tsp'\n",
"\n",
"###---------------------------------###\n",
"\n",
"# apply args to appropriate scenes\n",
"\n",
"for idx, rec in enumerate(storyboard.prompt_starts):\n",
" if idx not in applicable_scenes:\n",
" continue\n",
"\n",
" board_args = rec.get('animation_args')\n",
" if board_args:\n",
" board_args = OmegaConf.to_container(board_args) # coerce to dict\n",
"\n",
" new_args = collect_animation_args()\n",
" if not override_storyboard:\n",
" new_args.update(board_args)\n",
" #variations_settings\n",
" rec['animation_args'] = new_args\n",
" rec.update(variations_settings)\n",
"\n",
" if override_storyboard or (not rec.get('animation_mode')):\n",
" rec['animation_mode'] = animation_mode\n",
"\n",
" # oooo lookat me being fancy with custom code!\n",
" #if (idx >=23) or idx ==16:\n",
" # rec['animation_mode'] = 'img2img'\n",
"\n",
"save_storyboard(storyboard)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "zHJ5gPUOIhIY",
"tags": []
},
"outputs": [],
"source": [
"# @title ### 🎞️ Show Current Storyboard\n",
"workspace, storyboard = load_storyboard() # just to ensure consistency\n",
"show_storyboard()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fikaqNVJ-mmP"
},
"source": [
"### 2.4 🎧📈 (Advanced) Audioreactivity"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0twvUVRTgCfM",
"tags": [],
"cellView": "form"
},
"outputs": [],
"source": [
"# @title ### Choose a driving signal\n",
"\n",
"audio_assets_meta = load_audio_meta(*load_storyboard())\n",
"#structural_features = audio_assets_meta[str(storyboard.params.audio_fpath)]['structural_features']\n",
"\n",
"driving_signal_name = \"vocals stem\" #@param ['audio input','user specified','vocals stem','bass stem','other stem','drum stem']\n",
"\n",
"# TODO: test this\n",
"custom_signal_fpath = '' # @param {'type':'string'}\n",
"\n",
"def get_user_specified_signal():\n",
" y, sr = librosa.load(custom_signal_fpath)\n",
" return y, sr\n",
"\n",
"y = structural_features['y']\n",
"sr = structural_features['sr']\n",
"\n",
"driving_signals = {\n",
" 'audio input': lambda: (y, sr),\n",
" 'user specified': get_user_specified_signal,\n",
" 'vocals stem':lambda: get_stem('vocals'),\n",
" 'bass stem':lambda: get_stem('bass'),\n",
" 'other stem':lambda: get_stem('other'),\n",
" 'drum stem':lambda: get_stem('drum'),\n",
"}\n",
"\n",
"raw_driving_signal, sr = driving_signals[driving_signal_name]()\n",
"display_signal(raw_driving_signal, sr, title=driving_signal_name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NA2UEjQ6PYM6"
},
"source": [
"#### Manipulate the signal"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "hFOIMefxgCfM",
"tags": []
},
"outputs": [],
"source": [
"import numpy as np\n",
"from scipy import signal\n",
"from inspect import signature\n",
"from functools import partial\n",
"from scipy.signal import find_peaks\n",
"from sklearn.cluster import KMeans\n",
"#sklearn_extra.cluster.KMedoids\n",
"\n",
"# @markdown To apply multiple operations, separate manipulation names with a '|' or just re-run this cell.\n",
"# @markdown When you're satisfied, run the next cell to replace the loaded driving signal with the manipulated signal.\n",
"\n",
"# @markdown **Tips**\n",
"# @markdown * Add manipulations one at a time, re-running this cell each time to see what the effect of each subsequent manipulation is and to figure out if you need to change settings before adding others.\n",
"# @markdown * Start by isolating the thing you're interested in, then finish by cleaning it up for animation.\n",
"# @markdown * The outputs of this process will be saved in `storyboard.yaml` and can be used in other AI animation tools such as deforum.\n",
"\n",
"# @markdown ### Available Signal Manipulations\n",
"# @markdown NB: if a signal's name is followed by an asterisk (`*`), this means it modifies the signal's time domain. Only one time domain transformation per sequence is supported.\n",
"\n",
"# @markdown NB: if a signal's name is followed by two asterisks (`**`), this means it can't be used after a time-domain modifying manipulation has been applied.\n",
"# @markdown * `rms`* - Root mean squared. Converts raw signal to signal power\n",
"# @markdown * `novelty`* - Estimates strength of sound event onsets.\n",
"# @markdown * `predominant_pulse`* - Combined estimate of beat timing and strength\n",
"# @markdown * `bandpass(low, high)`** - Isolate signal to frequencies between `[low, high]`\n",
"# @markdown * `harmonic`** - Isolate the harmonic component.\n",
"# @markdown * `percussive`** - Isolate the percussive component.\n",
"# @markdown * `sustain(k)` - Treats signal as a self-exciting process that decays slowly over a window of `k`. Smooths asymmetrically (to the right)\n",
"# @markdown * `stretch` - Increases gap between high and low amplitude signals. Alias for `pow(2)`\n",
"# @markdown * `smoosh` - Reduces gap between high and low amplitude signals. Alias for `pow(1/2)`\n",
"# @markdown * `pow(k)` - Raises signal to the power of k.\n",
"# @markdown * `normalize` - Transform signal to `[0,1]` range. This will always be the last step, even if you don't specify it.\n",
"# @markdown * `threshold(low)` - Zero the signal where amplitude is less than `low`\n",
"# @markdown * `clamp(high)` - Clamp the signal such that no values have amplitude greater than `high`\n",
"# @markdown * `modulo(k)` - Detects peaks in signal and returns only every Kth-peak\n",
"# @markdown * `quantize(k)` - Discretizes the signal into K unique values after clustering amplitudes.\n",
"\n",
"# we can have some undocumented manipulations I think.\n",
"# too cluttered as it is.\n",
"# #@markdown * `raw` - no op\n",
"# #@markdown * `pow2` - Square the signal. Increases gap between high and low amplitude signals\n",
"# #@markdown * `sqrt` - Square root of signal. Reduces gap between high and low amplitude signals\n",
"# #@markdown * `smooth(k)` - Smoothes the waveworm, using a symmetric smoothing window of `k`. Bigger `k` = flatter signal\n",
"\n",
"\n",
"# signal_manipulations = 'bandpass(300, 500) | pow2 | pow2 | rms | pow2 | threshold(5e-6) ' # @param {'type':'string'}\n",
"signal_manipulations = 'harmonic | bandpass(300, 500) | rms | smoosh ' # @param {'type':'string'}\n",
"\n",
"manipulations = [m.strip() for m in signal_manipulations.split('|') if m.strip()]\n",
"\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown Running this cell will produce a series of plots, one for each signal manipulation.\n",
"# @markdown Use these options to zoom in on a particular time window.\n",
"\n",
"# TODO: add scene division markers to audio plots\n",
"plotting_window_start_time = 0 # @param {'type': 'number'}\n",
"plotting_window_end_time = 9999 # @param {'type': 'number'}\n",
"\n",
"# @markdown ---\n",
"\n",
"# @markdown ### Audio EQ cheat sheet\n",
"\n",
"# @markdown \n",
"\n",
"\n",
"# TODO: specify which transforms are available in spectral space vs. not\n",
"\n",
"\n",
"# trim/trim(q) -> y[y>quantile(y,.1)]\n",
"# that 4part way of parameterizing a wave... hit, sustain, decay,..?\n",
"\n",
"\n",
gitextract_j7_i1yfm/
├── .github/
│ └── workflows/
│ └── python-publish.yml
├── .gitignore
├── LICENSE
├── README.md
├── VERSION
├── Video_Killed_The_Radio_Star_Defusion.ipynb
├── pyproject.toml
└── vktrs/
├── __init__.py
├── api.py
├── asr.py
├── hf.py
├── tsp.py
├── utils.py
└── youtube.py
SYMBOL INDEX (28 symbols across 6 files)
FILE: vktrs/api.py
function get_image_for_prompt (line 10) | def get_image_for_prompt(prompt, max_retries=3, **kargs):
function process_response (line 36) | def process_response(answers):
FILE: vktrs/asr.py
function whisper_transcribe (line 15) | def whisper_transcribe(
function whisper_align (line 40) | def whisper_align(whispers):
function whisper_transmit_meta_across_alignment (line 56) | def whisper_transmit_meta_across_alignment(
function whisper_segment_transcription (line 115) | def whisper_segment_transcription(
function whisper_lyrics (line 178) | def whisper_lyrics(audio_fpath="audio.mp3"):
FILE: vktrs/hf.py
class HfHelper (line 19) | class HfHelper:
method __init__ (line 20) | def __init__(
method load_pipelines (line 41) | def load_pipelines(
method get_image_for_prompt (line 74) | def get_image_for_prompt(
FILE: vktrs/tsp.py
function tsp_permute_frames (line 9) | def tsp_permute_frames(frames, verbose=False):
function batched_tsp_permute_frames (line 27) | def batched_tsp_permute_frames(frames, batch_size):
FILE: vktrs/utils.py
function gpu_info (line 12) | def gpu_info():
function get_audio_duration_seconds (line 32) | def get_audio_duration_seconds(audio_fpath):
function rand_str (line 45) | def rand_str(n_char=5):
function remove_punctuation (line 49) | def remove_punctuation(s):
function sanitize_folder_name (line 54) | def sanitize_folder_name(fp):
function add_caption2image (line 64) | def add_caption2image(
function save_frame (line 102) | def save_frame(
function get_image_sequence (line 116) | def get_image_sequence(idx, root, init_first=True):
function archive_images (line 137) | def archive_images(idx, root, archive_root = None):
FILE: vktrs/youtube.py
class YoutubeHelper (line 15) | class YoutubeHelper:
method __init__ (line 16) | def __init__(
method get_subtitles (line 32) | def get_subtitles(
function parse_timestamp (line 49) | def parse_timestamp(ts):
function vtt_to_token_timestamps (line 58) | def vtt_to_token_timestamps(captions):
function srv2_to_token_timestamps (line 88) | def srv2_to_token_timestamps(srv2_xml):
Condensed preview — 14 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (294K chars).
[
{
"path": ".github/workflows/python-publish.yml",
"chars": 1087,
"preview": "# This workflow will upload a Python Package using Twine when a release is created\n# For more information see: https://h"
},
{
"path": ".gitignore",
"chars": 247,
"preview": "_venv\n\n\n*.srv2\n*.vtt\n*.webm\n*.mp3\n*.mp4\n*.pyc\n*.egg-info/\n**/frames\n**/archive\n*.yaml\n*.whl\n*.tar.gz\ndist\n\n# local huggi"
},
{
"path": "LICENSE",
"chars": 1067,
"preview": "MIT License\n\nCopyright (c) 2022 David Marx\n\nPermission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "README.md",
"chars": 3740,
"preview": "# Video Killed The Radio Star [](https://colab"
},
{
"path": "VERSION",
"chars": 6,
"preview": "0.1.8\n"
},
{
"path": "Video_Killed_The_Radio_Star_Defusion.ipynb",
"chars": 251066,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mgXxoDhMAiti\"\n },\n \"sou"
},
{
"path": "pyproject.toml",
"chars": 886,
"preview": "[build-system]\nrequires = [\"setuptools\", \"setuptools-scm\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"vk"
},
{
"path": "vktrs/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "vktrs/api.py",
"chars": 1539,
"preview": "import io\nimport os\nfrom PIL import Image, ImageDraw, ImageFont\nimport warnings\n\nfrom stability_sdk import client\nimport"
},
{
"path": "vktrs/asr.py",
"chars": 5933,
"preview": "\"\"\"\n# Automatic Speech Recognition utilities\n\nCurrently uses openai/whisper. To install: \n pip install git+https://gith"
},
{
"path": "vktrs/hf.py",
"chars": 2662,
"preview": "from pathlib import Path\nimport torch\nfrom torch import autocast\nfrom diffusers import (\n StableDiffusionImg2ImgPipel"
},
{
"path": "vktrs/tsp.py",
"chars": 1258,
"preview": "import time\n\nimport numpy as np\nfrom scipy.spatial.distance import pdist, squareform\nfrom toolz.itertoolz import partiti"
},
{
"path": "vktrs/utils.py",
"chars": 4222,
"preview": "from pathlib import Path\nimport random\nimport string\nimport subprocess\nimport textwrap\nimport time\n\nimport pandas as pd\n"
},
{
"path": "vktrs/youtube.py",
"chars": 2903,
"preview": "import datetime as dt\nimport re\n\nfrom bs4 import BeautifulSoup\nimport yt_dlp\n\n\n# embedding yt-dlp instead of CLI let's u"
}
]
About this extraction
This page contains the full source code of the dmarx/video-killed-the-radio-star GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 14 files (270.1 KB), approximately 105.2k tokens, and a symbol index with 28 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.