Repository: dmarx/video-killed-the-radio-star
Branch: main
Commit: 7d1053356aa8
Files: 14
Total size: 270.1 KB
Directory structure:
gitextract_j7_i1yfm/
├── .github/
│ └── workflows/
│ └── python-publish.yml
├── .gitignore
├── LICENSE
├── README.md
├── VERSION
├── Video_Killed_The_Radio_Star_Defusion.ipynb
├── pyproject.toml
└── vktrs/
├── __init__.py
├── api.py
├── asr.py
├── hf.py
├── tsp.py
├── utils.py
└── youtube.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/python-publish.yml
================================================
# This workflow will upload a Python Package using Twine when a release is created
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Upload Python Package
on:
release:
types: [published]
permissions:
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build
- name: Publish package
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
with:
user: __token__
password: ${{ secrets.PYPI_TOKEN }}
================================================
FILE: .gitignore
================================================
_venv
*.srv2
*.vtt
*.webm
*.mp3
*.mp4
*.pyc
*.egg-info/
**/frames
**/archive
*.yaml
*.whl
*.tar.gz
dist
# local huggingface model directories and files
feature_extractor
safety_checker
scheduler
text_encoder
tokenizer
unet
vae
model_index.json
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 David Marx
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Video Killed The Radio Star [](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)
## Requirements
* ffmpeg - https://ffmpeg.org/
* pytorch - https://pytorch.org/get-started/locally/
* vktrs - (this repo) - `pip install vktrs[api]`
* stability_sdk api token - https://beta.dreamstudio.ai/ > circular icon in top right > membership > API Key
* whisper - `pip install git+https://github.com/openai/whisper`
## FAQ
**What is this?**
TLDR: Automated music video maker, given an mp3 or a youtube URL
**How does this animation technique work?**
For each text prompt you provide, the notebook will...
1. Generate an image based on that text prompt (using stable diffusion)
2. Use the generated image as the `init_image` to recombine with the text prompt to generate variations similar to the first image. This produces a sequence of extremely similar images based on the original text prompt
3. Images are then intelligently reordered to find the smoothest animation sequence of those frames
3. This image sequence is then repeated to pad out the animation duration as needed
The technique demonstrated in this notebook was inspired by a [video](https://www.youtube.com/watch?v=WJaxFbdjm8c) created by Ben Gillin.
**How are lyrics transcribed?**
This notebook uses openai's recently released 'whisper' model for performing automatic speech recognition.
OpenAI was kind of to offer several different sizes of this model which each have their own pros and cons.
This notebook uses the largest whisper model for transcribing the actual lyrics. Additionally, we use the
smallest model for performing the lyric segmentation. Neither of these models is perfect, but the results
so far seem pretty decent.
The first draft of this notebook relied on subtitles from youtube videos to determine timing, which was
then aligned with user-provided lyrics. Youtube's automated captions are powerful and I'll update the
notebook shortly to leverage those again, but for the time being we're just using whisper for everything
and not referencing user-provided captions at all.
**Something didn't work quite right in the transcription process. How do fix the timing or the actual lyrics?**
The notebook is divided into several steps. Between each step, a "storyboard" file is updated. If you want to
make modifications, you can edit this file directly and those edits should be reflected when you next load the
file. Depending on what you changed and what step you run next, your changes may be ignored or even overwritten.
Still playing with different solutions here.
**Can I provide my own images to 'bring to life' and associate with certain lyrics/sequences?**
Yes, you can! As described above: you just need to modify the storyboard. Will describe this functionality in
greater detail after the implementation stabilizes a bit more.
**This gave me an idea and I'd like to use just a part of your process here. What's the best way to reuse just some of the machinery you've developed here?**
Most of the functionality in this notebook has been offloaded to library I published to pypi called `vktrs`. I strongly encourage you to import anything you need
from there rather than cutting and pasting function into a notebook. Similarly, if you have ideas for improvements, please don't hesitate to submit a PR!
## Dev notes
```
!pip install --upgrade setuptools build
!git clone https://github.com/dmarx/video-killed-the-radio-star/
!cd video-killed-the-radio-star; python -m build; python -m pip install -e .[api,hf]
!pip install ipykernel ipywidgets panel prefetch_generator
```
================================================
FILE: VERSION
================================================
0.1.8
================================================
FILE: Video_Killed_The_Radio_Star_Defusion.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "mgXxoDhMAiti"
},
"source": [
"# $ \\text{Video Killed The Radio Star}$ $\\color{red}{...Diffusion}$\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Notebook by David Marx ([@DigThatData](https://twitter.com/digthatdata))\n",
"\n",
"Shared under MIT license\n",
"\n",
"Last updated: 2023-06-15\n",
"\n",
"Latest stable notebook revision: [](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)\n",
"\n",
"# Introduction\n",
"\n",
"VKTRS is a tool for planning and generating complex animations using audio with spoken/sung words as conditioning input. The planning work can be used independently from generating an animation: all of your work is saved to a human-readable and modifiable `storyboard.yaml` file. Parameter names and notation are also compatible with other AI animation tools, such as a1111-deforum.\n",
"\n",
"The inspiration for this notebook and the \"variations\" animation modes were [this video](https://www.youtube.com/watch?v=WJaxFbdjm8c) created by Ben Gillin.\n",
"\n",
"# General Workflow\n",
"\n",
"If you are reading this in colab, open the \"Table of contents\" tab on the left sidebar for a more detailed enumeration of the steps described below.\n",
"\n",
"1. Pick an audio source, like a youtube video\n",
"2. The notebook will download the video if necessary\n",
"3. OpenAI's \"whisper\" model is used to transcribe the lyrics\n",
"4. The timing of this transcription is used to segment the timeline into a sequence of \"scenes\"\n",
"5. The musical structure of the audio is analyzed to group thematically similar themes\n",
"6. Scene/theme specific settings (prompts, camera motion) can then be added\n",
"7. The audio can be further processed to isolate signals for driving animation parameters for \"audioreactive\" effects. This includes isolating instruments (demucs) and chaining manipulations (librosa)\n",
"8. Generate or specify starting images for each scene\n",
"9. Generate the remaining frames to animate each scene\n",
"\n",
"**NB: the `img2img` animation mode is currently only supported when using the stability ai animation api (DreamStudio) to generate animations**. The audioreactivity features are currently only relevant in img2img mode. If you don't have a DreamStudio account, you can still use this notebook to parameterize an animation, you'll just need to cut-and-paste settings out of the \"export settings\" cell (deforum compatible) or the `storyboard.yaml` file.\n",
"\n",
"# The \"Storyboard\"\n",
"\n",
"Start at the top of the notebook and work your way down. Each decision you make contributes information to a file called \"the storyboard\", which persists project state. The notebook is designed to facilitate working iteratively: once you've made it past a given decision point, you should generally be able to jump back to that point to tweak whatever the decision was and return to where you were.\n",
"\n",
"The storyboard isn't just a logical abstraction, it's a physical file you can modify directly, and is reasonably human readable.\n",
"\n",
"Every project you start will create a new storyboard. If you set your project name to one you used previously, the notebook will attempt to find that project and load its storyboard.\n",
"\n",
"The original motivation behind VKTRS was to be a toy for automating generative music videos. It has evolved quite a bit since, and I now mostly think about it as a tool for building and manipulating these storyboards. The storyboard parameterizes an animation, but the actual animation doesn't need to be generated by this tool.\n",
"\n",
"\n",
"# $\\text{FAQ}$\n",
"\n",
"**Why the name?**\n",
"\n",
"The notebook's name is an ironic homage to the first music video played on MTV: [The Bugles - Video Killed The Radio Star](https://www.youtube.com/watch?v=W8r-tXRLazs). Quoting [wikipedia](https://en.wikipedia.org/wiki/Video_Killed_the_Radio_Star):\n",
"\n",
"> \"The song relates to concerns about, and mixed attitudes toward 20th-century inventions and machines for the media arts.\"\n",
"\n",
"**Something didn't work quite right in the transcription process. How do fix the timing or the actual lyrics?**\n",
"\n",
"The notebook is divided into several steps. Between each step, a \"storyboard\" file is updated. If you want to\n",
"make modifications, you can edit this file directly and those edits should be reflected when you next load the\n",
"file. Depending on what you changed and what step you run next, your changes may be ignored or even overwritten.\n",
"Still playing with different solutions here.\n",
"\n",
"**Can I provide my own images to 'bring to life' and associate with certain lyrics/sequences?**\n",
"\n",
"Yes, you can! As described above: you just need to modify the storyboard. Will describe this functionality in\n",
"greater detail after the implementation stabilizes a bit more.\n",
"\n",
"**How can I support your work or work like it?**\n",
"\n",
"This notebook was made possible thanks to ongoing support from [stability.ai](https://stability.ai/). The best way to support my work is to share it with your friends, [report bugs](https://github.com/dmarx/video-killed-the-radio-star/issues/new), [suggest features](https://github.com/dmarx/video-killed-the-radio-star/discussions) or to donate to open source non-profits :)\n",
"\n",
"# Examples of content made with VKTRS\n",
"\n",
"[](https://www.youtube.com/watch?v=dx8LmqalrmU)\n",
"\n",
"**The Sudden - Fine (Official Music Video)** `variations` animation mode and theme prompts.\n",
"\n",
"![\"Hi-Standard](\"https://i.ytimg.com/vi_webp/ScbXSLYyzq4/oar2.webp\")