Showing preview only (588K chars total). Download the full file or copy to clipboard to get everything.
Repository: XmYx/deforum-comfy-nodes
Branch: main
Commit: 25c5e78b5c59
Files: 50
Total size: 564.5 KB
Directory structure:
gitextract_kwee4pen/
├── .gitattributes
├── .github/
│ └── workflows/
│ └── publish_action.yml
├── .gitignore
├── LICENSE
├── README.md
├── __init__.py
├── deforum_nodes/
│ ├── __init__.py
│ ├── exec_hijack.py
│ ├── mapping.py
│ ├── modules/
│ │ ├── __init__.py
│ │ ├── better_resize/
│ │ │ ├── __init__.py
│ │ │ ├── interp_methods.py
│ │ │ └── resize_right.py
│ │ ├── deforum_comfy_sampler.py
│ │ ├── deforum_comfyui_helpers.py
│ │ ├── deforum_constants.py
│ │ ├── deforum_node_base.py
│ │ ├── deforum_ui_data.py
│ │ ├── interp.py
│ │ └── standalone_cadence.py
│ └── nodes/
│ ├── __init__.py
│ ├── deforum_advnoise_node.py
│ ├── deforum_audiosync_nodes.py
│ ├── deforum_cache_nodes.py
│ ├── deforum_cnet_nodes.py
│ ├── deforum_cond_nodes.py
│ ├── deforum_data_nodes.py
│ ├── deforum_framewarp_node.py
│ ├── deforum_hybrid_nodes.py
│ ├── deforum_image_nodes.py
│ ├── deforum_interpolation_nodes.py
│ ├── deforum_iteration_nodes.py
│ ├── deforum_legacy_nodes.py
│ ├── deforum_logic_nodes.py
│ ├── deforum_noise_nodes.py
│ ├── deforum_prompt_nodes.py
│ ├── deforum_sampler_nodes.py
│ ├── deforum_schedule_visualizer.py
│ ├── deforum_video_nodes.py
│ └── redirect_console_node.py
├── examples/
│ ├── deforum_base.json
│ ├── deforum_cadence.json
│ ├── deforum_integrated.json
│ ├── deforum_ip_adapter.json
│ ├── deforum_simple.json
│ ├── deforum_stablecascade.json
│ └── deforum_stablecascade_legacy.json
├── install.py
├── node_list.json
└── web/
└── js/
└── deforumIterateNode.js
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
# Auto detect text files and perform LF normalization
* text=auto
================================================
FILE: .github/workflows/publish_action.yml
================================================
name: Publish to Comfy registry
on:
workflow_dispatch:
push:
branches:
- main
paths:
- "pyproject.toml"
jobs:
publish-node:
name: Publish Custom Node to registry
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Publish Custom Node
uses: Comfy-Org/publish-node-action@main
with:
## Add your own personal access token to your Github Repository secrets and reference it here.
personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintainted in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2024 Deforum LLC.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Deforum for ComfyUI
Deforum integration for ComfyUI.
## Installation
To get started with Deforum Comfy Nodes, please make sure ComfyUI is installed and you are using Python v3.10 or these nodes will not work. We recommend using a virtual environment.
Follow the steps below depending on your method of preference.
### ComfyUI Manager
Look for `Deforum Nodes` by `XmYx`
### Manual Install
To install Deforum for ComfyUI we will clone this repo into the `custom_nodes` folder
```bash
git clone https://github.com/XmYx/deforum-comfy-nodes.git
```
## Recommended Custom Nodes
Here is a list of extra custom nodes that greatly improves the experience of using Deforum.
```bash
https://github.com/rgthree/rgthree-comfy
https://github.com/a1lazydog/ComfyUI-AudioScheduler
https://github.com/cubiq/ComfyUI_IPAdapter_plus
https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite
https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet
https://github.com/WASasquatch/was-node-suite-comfyui
https://github.com/11cafe/comfyui-workspace-manager
https://github.com/cubiq/ComfyUI_essentials
https://github.com/FizzleDorf/ComfyUI_FizzNodes
https://github.com/ltdrdata/ComfyUI-Impact-Pack
https://github.com/Fannovel16/ComfyUI-Frame-Interpolation
https://github.com/Fannovel16/ComfyUI-Video-Matting
https://github.com/crystian/ComfyUI-Crystools
```
## Usage
1. Launch ComfyUI
2. Load any of the example workflows from the examples folder.
3. Queue prompt, this will generate your first frame, you can enable Auto queueing, or batch as many images as long you'd
like your animation to be.
## Contribution
We welcome contributions from the community! If you're interested in improving Deforum Comfy Nodes or have ideas for new features, please follow these steps:
1. Fork the repository on GitHub.
2. Create a new branch for your feature or fix.
3. Commit your changes with clear, descriptive messages.
4. Push your changes to the branch and open a pull request.
## License
Deforum Comfy Nodes is licensed under the MIT License. For more details, see the LICENSE file in the repository.
## Community and Support
Join our Discord community to discuss Deforum Comfy Nodes, share your creations, and get help from the developers and other users:
[Join Discord](https://discord.gg/deforum)
[Visit our website](https://deforum.art)
================================================
FILE: __init__.py
================================================
from .deforum_nodes.mapping import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
WEB_DIRECTORY = "./web"
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS', "WEB_DIRECTORY"]
================================================
FILE: deforum_nodes/__init__.py
================================================
================================================
FILE: deforum_nodes/exec_hijack.py
================================================
import execution, nodes
orig_exec = execution.map_node_over_list
def map_node_over_list(obj, input_data_all, func, allow_interrupt=False, execution_block_cb=None, pre_execute_cb=None):
try:
# check if node wants the lists
input_is_list = getattr(obj, "INPUT_IS_LIST", False)
if len(input_data_all) == 0:
max_len_input = 0
else:
max_len_input = max([len(x) for x in input_data_all.values() if x is not None])
# get a slice of inputs, repeat last input when list isn't long enough
def slice_dict(d, i):
d_new = dict()
for k, v in d.items():
if v is None: # Skip if any of the values is None
return None
d_new[k] = v[i if len(v) > i else -1]
return d_new
results = []
for k, v in input_data_all.items():
if v == "skip":
print("[deforum] Skipping execution of", obj)
return []
def process_inputs(inputs, index=None):
if allow_interrupt:
nodes.before_node_execution()
execution_block = None
for k, v in inputs.items():
if isinstance(v, ExecutionBlocker):
execution_block = execution_block_cb(v) if execution_block_cb else v
break
if execution_block is None:
if pre_execute_cb is not None and index is not None:
pre_execute_cb(index)
results.append(getattr(obj, func)(**inputs))
else:
results.append(execution_block)
if input_is_list:
process_inputs(input_data_all, 0)
elif max_len_input == 0:
process_inputs({})
else:
for i in range(max_len_input):
sliced_input = slice_dict(input_data_all, i)
if sliced_input is None: # Skip this iteration if there's a None value after slicing
continue
if allow_interrupt:
nodes.before_node_execution()
for k, v in sliced_input.items():
if v == "skip":
print("[deforum] Skipping execution of", obj)
return []
process_inputs(sliced_input, i)
return results
except:
print("[deforum] Executor HiJack Failed and was deactivated, please report the issue on GitHub")
execution.map_node_over_list = orig_exec
return orig_exec(obj, input_data_all, func, allow_interrupt, execution_block_cb, pre_execute_cb)
execution.map_node_over_list = map_node_over_list
print("[deforum] Execution HiJack Active")
================================================
FILE: deforum_nodes/mapping.py
================================================
import importlib
import inspect
import sys
"""
DEFORUM STORAGE IMPORT
"""
from .modules.deforum_constants import DeforumStorage
gs = DeforumStorage()
"""
NODE CLASS IMPORTS
"""
from .nodes.deforum_audiosync_nodes import *
from .nodes.deforum_cache_nodes import *
from .nodes.deforum_cnet_nodes import *
from .nodes.deforum_cond_nodes import *
from .nodes.deforum_data_nodes import *
from .nodes.deforum_framewarp_node import *
from .nodes.deforum_hybrid_nodes import *
from .nodes.deforum_interpolation_nodes import *
from .nodes.deforum_image_nodes import *
from .nodes.deforum_iteration_nodes import *
from .nodes.deforum_legacy_nodes import *
from .nodes.deforum_logic_nodes import *
try:
from .nodes.deforum_noise_nodes import AddCustomNoiseNode
except:
pass
try:
from .nodes.deforum_advnoise_node import AddAdvancedNoiseNode
except:
pass
from .nodes.deforum_prompt_nodes import *
from .nodes.redirect_console_node import DeforumRedirectConsole
from .nodes.deforum_sampler_nodes import *
from .nodes.deforum_schedule_visualizer import *
from .nodes.deforum_video_nodes import *
from . import exec_hijack
# Create an empty dictionary for class mappings
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
# Iterate through all classes defined in your code
# Get a reference to the current module to inspect its imported node classes
deforum_node_module = sys.modules[__name__]
# Iterate through all classes defined in the current module
for name, obj in inspect.getmembers(deforum_node_module):
# Check if the member is a class
if inspect.isclass(obj) and hasattr(obj, "INPUT_TYPES"):
# Extract the class name and display name
class_name = name
display_name = getattr(obj, "display_name", name) # Use class attribute or default to class name
# Add the class to the mappings
NODE_CLASS_MAPPINGS[class_name] = obj
NODE_DISPLAY_NAME_MAPPINGS[name] = "(deforum) " + display_name
================================================
FILE: deforum_nodes/modules/__init__.py
================================================
================================================
FILE: deforum_nodes/modules/better_resize/__init__.py
================================================
================================================
FILE: deforum_nodes/modules/better_resize/interp_methods.py
================================================
from math import pi
try:
import torch
except ImportError:
torch = None
try:
import numpy
except ImportError:
numpy = None
if numpy is None and torch is None:
raise ImportError("Must have either Numpy or PyTorch but both not found")
def set_framework_dependencies(x):
if type(x) is numpy.ndarray:
to_dtype = lambda a: a
fw = numpy
else:
to_dtype = lambda a: a.to(x.dtype)
fw = torch
eps = fw.finfo(fw.float32).eps
return fw, to_dtype, eps
def support_sz(sz):
def wrapper(f):
f.support_sz = sz
return f
return wrapper
@support_sz(4)
def cubic(x):
fw, to_dtype, eps = set_framework_dependencies(x)
absx = fw.abs(x)
absx2 = absx ** 2
absx3 = absx ** 3
return ((1.5 * absx3 - 2.5 * absx2 + 1.) * to_dtype(absx <= 1.) +
(-0.5 * absx3 + 2.5 * absx2 - 4. * absx + 2.) *
to_dtype((1. < absx) & (absx <= 2.)))
@support_sz(4)
def lanczos2(x):
fw, to_dtype, eps = set_framework_dependencies(x)
return (((fw.sin(pi * x) * fw.sin(pi * x / 2) + eps) /
((pi**2 * x**2 / 2) + eps)) * to_dtype(abs(x) < 2))
@support_sz(6)
def lanczos3(x):
fw, to_dtype, eps = set_framework_dependencies(x)
return (((fw.sin(pi * x) * fw.sin(pi * x / 3) + eps) /
((pi**2 * x**2 / 3) + eps)) * to_dtype(abs(x) < 3))
@support_sz(2)
def linear(x):
fw, to_dtype, eps = set_framework_dependencies(x)
return ((x + 1) * to_dtype((-1 <= x) & (x < 0)) + (1 - x) *
to_dtype((0 <= x) & (x <= 1)))
@support_sz(1)
def box(x):
fw, to_dtype, eps = set_framework_dependencies(x)
return to_dtype((-1 <= x) & (x < 0)) + to_dtype((0 <= x) & (x <= 1))
================================================
FILE: deforum_nodes/modules/better_resize/resize_right.py
================================================
from typing import Tuple
import warnings
from math import ceil
from . import interp_methods
from fractions import Fraction
class NoneClass:
pass
try:
import torch
from torch import nn
nnModuleWrapped = nn.Module
except ImportError:
warnings.warn('No PyTorch found, will work only with Numpy')
torch = None
nnModuleWrapped = NoneClass
try:
import numpy
except ImportError:
warnings.warn('No Numpy found, will work only with PyTorch')
numpy = None
if numpy is None and torch is None:
raise ImportError("Must have either Numpy or PyTorch but both not found")
def resize(input, scale_factors=None, out_shape=None,
interp_method=interp_methods.cubic, support_sz=None,
antialiasing=True, by_convs=False, scale_tolerance=None,
max_numerator=10, pad_mode='constant'):
# get properties of the input tensor
in_shape, n_dims = input.shape, input.ndim
# fw stands for framework that can be either numpy or torch,
# determined by the input type
fw = numpy if type(input) is numpy.ndarray else torch
eps = fw.finfo(fw.float32).eps
device = input.device if fw is torch else None
# set missing scale factors or output shapem one according to another,
# scream if both missing. this is also where all the defults policies
# take place. also handling the by_convs attribute carefully.
scale_factors, out_shape, by_convs = set_scale_and_out_sz(in_shape,
out_shape,
scale_factors,
by_convs,
scale_tolerance,
max_numerator,
eps, fw)
# sort indices of dimensions according to scale of each dimension.
# since we are going dim by dim this is efficient
sorted_filtered_dims_and_scales = [(dim, scale_factors[dim], by_convs[dim],
in_shape[dim], out_shape[dim])
for dim in sorted(range(n_dims),
key=lambda ind: scale_factors[ind])
if scale_factors[dim] != 1.]
# unless support size is specified by the user, it is an attribute
# of the interpolation method
if support_sz is None:
support_sz = interp_method.support_sz
# output begins identical to input and changes with each iteration
output = input
# iterate over dims
for (dim, scale_factor, dim_by_convs, in_sz, out_sz
) in sorted_filtered_dims_and_scales:
# STEP 1- PROJECTED GRID: The non-integer locations of the projection
# of output pixel locations to the input tensor
projected_grid = get_projected_grid(in_sz, out_sz,
scale_factor, fw, dim_by_convs,
device)
# STEP 1.5: ANTIALIASING- If antialiasing is taking place, we modify
# the window size and the interpolation method (see inside function)
cur_interp_method, cur_support_sz = apply_antialiasing_if_needed(
interp_method,
support_sz,
scale_factor,
antialiasing)
# STEP 2- FIELDS OF VIEW: for each output pixels, map the input pixels
# that influence it. Also calculate needed padding and update grid
# accoedingly
field_of_view = get_field_of_view(projected_grid, cur_support_sz, fw,
eps, device)
# STEP 2.5- CALCULATE PAD AND UPDATE: according to the field of view,
# the input should be padded to handle the boundaries, coordinates
# should be updated. actual padding only occurs when weights are
# aplied (step 4). if using by_convs for this dim, then we need to
# calc right and left boundaries for each filter instead.
pad_sz, projected_grid, field_of_view = calc_pad_sz(in_sz, out_sz,
field_of_view,
projected_grid,
scale_factor,
dim_by_convs, fw,
device)
# STEP 3- CALCULATE WEIGHTS: Match a set of weights to the pixels in
# the field of view for each output pixel
weights = get_weights(cur_interp_method, projected_grid, field_of_view)
# STEP 4- APPLY WEIGHTS: Each output pixel is calculated by multiplying
# its set of weights with the pixel values in its field of view.
# We now multiply the fields of view with their matching weights.
# We do this by tensor multiplication and broadcasting.
# if by_convs is true for this dim, then we do this action by
# convolutions. this is equivalent but faster.
if not dim_by_convs:
output = apply_weights(output, field_of_view, weights, dim, n_dims,
pad_sz, pad_mode, fw)
else:
output = apply_convs(output, scale_factor, in_sz, out_sz, weights,
dim, pad_sz, pad_mode, fw)
return output
def get_projected_grid(in_sz, out_sz, scale_factor, fw, by_convs, device=None):
# we start by having the ouput coordinates which are just integer locations
# in the special case when usin by_convs, we only need two cycles of grid
# points. the first and last.
grid_sz = out_sz if not by_convs else scale_factor.numerator
out_coordinates = fw_arange(grid_sz, fw, device)
# This is projecting the ouput pixel locations in 1d to the input tensor,
# as non-integer locations.
# the following fomrula is derived in the paper
# "From Discrete to Continuous Convolutions" by Shocher et al.
return (out_coordinates / float(scale_factor) +
(in_sz - 1) / 2 - (out_sz - 1) / (2 * float(scale_factor)))
def get_field_of_view(projected_grid, cur_support_sz, fw, eps, device):
# for each output pixel, map which input pixels influence it, in 1d.
# we start by calculating the leftmost neighbor, using half of the window
# size (eps is for when boundary is exact int)
left_boundaries = fw_ceil(projected_grid - cur_support_sz / 2 - eps, fw)
# then we simply take all the pixel centers in the field by counting
# window size pixels from the left boundary
ordinal_numbers = fw_arange(ceil(cur_support_sz - eps), fw, device)
return left_boundaries[:, None] + ordinal_numbers
def calc_pad_sz(in_sz, out_sz, field_of_view, projected_grid, scale_factor,
dim_by_convs, fw, device):
if not dim_by_convs:
# determine padding according to neighbor coords out of bound.
# this is a generalized notion of padding, when pad<0 it means crop
pad_sz = [-field_of_view[0, 0].item(),
field_of_view[-1, -1].item() - in_sz + 1]
# since input image will be changed by padding, coordinates of both
# field_of_view and projected_grid need to be updated
field_of_view += pad_sz[0]
projected_grid += pad_sz[0]
else:
# only used for by_convs, to calc the boundaries of each filter the
# number of distinct convolutions is the numerator of the scale factor
num_convs, stride = scale_factor.numerator, scale_factor.denominator
# calculate left and right boundaries for each conv. left can also be
# negative right can be bigger than in_sz. such cases imply padding if
# needed. however if# both are in-bounds, it means we need to crop,
# practically apply the conv only on part of the image.
left_pads = -field_of_view[:, 0]
# next calc is tricky, explanation by rows:
# 1) counting output pixels between the first position of each filter
# to the right boundary of the input
# 2) dividing it by number of filters to count how many 'jumps'
# each filter does
# 3) multiplying by the stride gives us the distance over the input
# coords done by all these jumps for each filter
# 4) to this distance we add the right boundary of the filter when
# placed in its leftmost position. so now we get the right boundary
# of that filter in input coord.
# 5) the padding size needed is obtained by subtracting the rightmost
# input coordinate. if the result is positive padding is needed. if
# negative then negative padding means shaving off pixel columns.
right_pads = (((out_sz - fw_arange(num_convs, fw, device) - 1) # (1)
// num_convs) # (2)
* stride # (3)
+ field_of_view[:, -1] # (4)
- in_sz + 1) # (5)
# in the by_convs case pad_sz is a list of left-right pairs. one per
# each filter
pad_sz = list(zip(left_pads, right_pads))
return pad_sz, projected_grid, field_of_view
def get_weights(interp_method, projected_grid, field_of_view):
# the set of weights per each output pixels is the result of the chosen
# interpolation method applied to the distances between projected grid
# locations and the pixel-centers in the field of view (distances are
# directed, can be positive or negative)
weights = interp_method(projected_grid[:, None] - field_of_view)
# we now carefully normalize the weights to sum to 1 per each output pixel
sum_weights = weights.sum(1, keepdims=True)
sum_weights[sum_weights == 0] = 1
return weights / sum_weights
def apply_weights(input, field_of_view, weights, dim, n_dims, pad_sz, pad_mode,
fw):
# for this operation we assume the resized dim is the first one.
# so we transpose and will transpose back after multiplying
tmp_input = fw_swapaxes(input, dim, 0, fw)
# apply padding
tmp_input = fw_pad(tmp_input, fw, pad_sz, pad_mode)
# field_of_view is a tensor of order 2: for each output (1d location
# along cur dim)- a list of 1d neighbors locations.
# note that this whole operations is applied to each dim separately,
# this is why it is all in 1d.
# neighbors = tmp_input[field_of_view] is a tensor of order image_dims+1:
# for each output pixel (this time indicated in all dims), these are the
# values of the neighbors in the 1d field of view. note that we only
# consider neighbors along the current dim, but such set exists for every
# multi-dim location, hence the final tensor order is image_dims+1.
neighbors = tmp_input[field_of_view]
# weights is an order 2 tensor: for each output location along 1d- a list
# of weights matching the field of view. we augment it with ones, for
# broadcasting, so that when multiplies some tensor the weights affect
# only its first dim.
tmp_weights = fw.reshape(weights, (*weights.shape, * [1] * (n_dims - 1)))
# now we simply multiply the weights with the neighbors, and then sum
# along the field of view, to get a single value per out pixel
tmp_output = (neighbors * tmp_weights).sum(1)
# we transpose back the resized dim to its original position
return fw_swapaxes(tmp_output, 0, dim, fw)
def apply_convs(input, scale_factor, in_sz, out_sz, weights, dim, pad_sz,
pad_mode, fw):
# for this operations we assume the resized dim is the last one.
# so we transpose and will transpose back after multiplying
input = fw_swapaxes(input, dim, -1, fw)
# the stride for all convs is the denominator of the scale factor
stride, num_convs = scale_factor.denominator, scale_factor.numerator
# prepare an empty tensor for the output
tmp_out_shape = list(input.shape)
tmp_out_shape[-1] = out_sz
tmp_output = fw_empty(tuple(tmp_out_shape), fw, input.device)
# iterate over the conv operations. we have as many as the numerator
# of the scale-factor. for each we need boundaries and a filter.
for conv_ind, (pad_sz, filt) in enumerate(zip(pad_sz, weights)):
# apply padding (we pad last dim, padding can be negative)
pad_dim = input.ndim - 1
tmp_input = fw_pad(input, fw, pad_sz, pad_mode, dim=pad_dim)
# apply convolution over last dim. store in the output tensor with
# positional strides so that when the loop is comlete conv results are
# interwind
tmp_output[..., conv_ind::num_convs] = fw_conv(tmp_input, filt, stride)
return fw_swapaxes(tmp_output, -1, dim, fw)
def set_scale_and_out_sz(in_shape, out_shape, scale_factors, by_convs,
scale_tolerance, max_numerator, eps, fw):
# eventually we must have both scale-factors and out-sizes for all in/out
# dims. however, we support many possible partial arguments
if scale_factors is None and out_shape is None:
raise ValueError("either scale_factors or out_shape should be "
"provided")
if out_shape is not None:
# if out_shape has less dims than in_shape, we defaultly resize the
# first dims for numpy and last dims for torch
out_shape = (list(out_shape) + list(in_shape[len(out_shape):])
if fw is numpy
else list(in_shape[:-len(out_shape)]) + list(out_shape))
if scale_factors is None:
# if no scale given, we calculate it as the out to in ratio
# (not recomended)
scale_factors = [out_sz / in_sz for out_sz, in_sz
in zip(out_shape, in_shape)]
if scale_factors is not None:
# by default, if a single number is given as scale, we assume resizing
# two dims (most common are images with 2 spatial dims)
scale_factors = (scale_factors
if isinstance(scale_factors, (list, tuple))
else [scale_factors, scale_factors])
# if less scale_factors than in_shape dims, we defaultly resize the
# first dims for numpy and last dims for torch
scale_factors = (list(scale_factors) + [1] *
(len(in_shape) - len(scale_factors)) if fw is numpy
else [1] * (len(in_shape) - len(scale_factors)) +
list(scale_factors))
if out_shape is None:
# when no out_shape given, it is calculated by multiplying the
# scale by the in_shape (not recomended)
out_shape = [ceil(scale_factor * in_sz)
for scale_factor, in_sz in
zip(scale_factors, in_shape)]
# next part intentionally after out_shape determined for stability
# we fix by_convs to be a list of truth values in case it is not
if not isinstance(by_convs, (list, tuple)):
by_convs = [by_convs] * len(out_shape)
# next loop fixes the scale for each dim to be either frac or float.
# this is determined by by_convs and by tolerance for scale accuracy.
for ind, (sf, dim_by_convs) in enumerate(zip(scale_factors, by_convs)):
# first we fractionaize
if dim_by_convs:
frac = Fraction(1/sf).limit_denominator(max_numerator)
frac = Fraction(numerator=frac.denominator, denominator=frac.numerator)
# if accuracy is within tolerance scale will be frac. if not, then
# it will be float and the by_convs attr will be set false for
# this dim
if scale_tolerance is None:
scale_tolerance = eps
if dim_by_convs and abs(frac - sf) < scale_tolerance:
scale_factors[ind] = frac
else:
scale_factors[ind] = float(sf)
by_convs[ind] = False
return scale_factors, out_shape, by_convs
def apply_antialiasing_if_needed(interp_method, support_sz, scale_factor,
antialiasing):
# antialiasing is "stretching" the field of view according to the scale
# factor (only for downscaling). this is low-pass filtering. this
# requires modifying both the interpolation (stretching the 1d
# function and multiplying by the scale-factor) and the window size.
scale_factor = float(scale_factor)
if scale_factor >= 1.0 or not antialiasing:
return interp_method, support_sz
cur_interp_method = (lambda arg: scale_factor *
interp_method(scale_factor * arg))
cur_support_sz = support_sz / scale_factor
return cur_interp_method, cur_support_sz
def fw_ceil(x, fw):
if fw is numpy:
return fw.int_(fw.ceil(x))
else:
return x.ceil().long()
def fw_floor(x, fw):
if fw is numpy:
return fw.int_(fw.floor(x))
else:
return x.floor().long()
def fw_cat(x, fw):
if fw is numpy:
return fw.concatenate(x)
else:
return fw.cat(x)
def fw_swapaxes(x, ax_1, ax_2, fw):
if fw is numpy:
return fw.swapaxes(x, ax_1, ax_2)
else:
return x.transpose(ax_1, ax_2)
def fw_pad(x, fw, pad_sz, pad_mode, dim=0):
if pad_sz == (0, 0):
return x
if fw is numpy:
pad_vec = [(0, 0)] * x.ndim
pad_vec[dim] = pad_sz
return fw.pad(x, pad_width=pad_vec, mode=pad_mode)
else:
if x.ndim < 3:
x = x[None, None, ...]
pad_vec = [0] * ((x.ndim - 2) * 2)
pad_vec[0:2] = pad_sz
return fw.nn.functional.pad(x.transpose(dim, -1), pad=pad_vec,
mode=pad_mode).transpose(dim, -1)
def fw_conv(input, filter, stride):
# we want to apply 1d conv to any nd array. the way to do it is to reshape
# the input to a 4D tensor. first two dims are singeletons, 3rd dim stores
# all the spatial dims that we are not convolving along now. then we can
# apply conv2d with a 1xK filter. This convolves the same way all the other
# dims stored in the 3d dim. like depthwise conv over these.
# TODO: numpy support
reshaped_input = input.reshape(1, 1, -1, input.shape[-1])
reshaped_output = torch.nn.functional.conv2d(reshaped_input,
filter.view(1, 1, 1, -1),
stride=(1, stride))
return reshaped_output.reshape(*input.shape[:-1], -1)
def fw_arange(upper_bound, fw, device):
if fw is numpy:
return fw.arange(upper_bound)
else:
return fw.arange(upper_bound, device=device)
def fw_empty(shape, fw, device):
if fw is numpy:
return fw.empty(shape)
else:
return fw.empty(size=(*shape,), device=device)
================================================
FILE: deforum_nodes/modules/deforum_comfy_sampler.py
================================================
import secrets
import torch
import numpy as np
from PIL import Image
from deforum import ImageRNGNoise
from deforum.utils.deforum_cond_utils import blend_tensors
rng = None
def common_ksampler_with_custom_noise(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent,
denoise=1.0, disable_noise=False, start_step=None, last_step=None,
force_full_denoise=False, noise=None):
latent_image = latent["samples"]
if noise is not None:
noise = noise.next()#.detach().cpu()
# noise = rng_noise.clone()
else:
if disable_noise:
noise = torch.zeros(latent_image.size(), dtype=latent_image.dtype, layout=latent_image.layout, device="cpu")
else:
batch_inds = latent["batch_index"] if "batch_index" in latent else None
from comfy.sample import prepare_noise
noise = prepare_noise(latent_image, seed, batch_inds)
noise_mask = None
if "noise_mask" in latent:
noise_mask = latent["noise_mask"]
# callback = latent_preview.prepare_callback(model, steps)
# disable_pbar = not comfy.utils.PROGRESS_BAR_ENABLED
from comfy.sample import sample as sample_k
samples = sample_k(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
denoise=denoise, disable_noise=disable_noise, start_step=start_step,
last_step=last_step,
force_full_denoise=force_full_denoise, noise_mask=noise_mask, callback=None,
disable_pbar=False, seed=seed)
out = latent.copy()
out["samples"] = samples
return (out,)
def sample_deforum(
model,
clip,
vae,
controlnet=None,
prompt="",
pooled_prompts=None,
next_prompt=None,
prompt_blend=None,
negative_prompt="",
steps=25,
scale=7.5,
sampler_name="dpmpp_2m_sde",
scheduler="karras",
width=None,
height=None,
seed=-1,
strength=1.0,
init_image=None,
subseed=-1,
subseed_strength=0.6,
cnet_image=None,
cond=None,
n_cond=None,
return_latent=None,
latent=None,
last_step=None,
seed_resize_from_h=1024,
seed_resize_from_w=1024,
reset_noise=False,
enable_prompt_blend=True,
use_areas= False,
areas= None,
*args,
**kwargs):
# import comfy.sd
# model, clip, vae, clipvision = comfy.sd.load_checkpoint_guess_config(self.model_path,
# output_vae=True,
# output_clip=True,
# embedding_directory="models/embeddings",
# output_clipvision=False,
# )
if seed == -1:
seed = secrets.randbelow(18446744073709551615)
# strength = 1 - strength
if strength <= 0.0 or strength >= 1.0:
strength = 1.0
reset_noise = True
init_image = None
if subseed == -1:
subseed = secrets.randbelow(18446744073709551615)
if cnet_image is not None:
cnet_image = torch.from_numpy(np.array(cnet_image).astype(np.float32) / 255.0).unsqueeze(0)
if init_image is None or reset_noise:
strength = 1.0
if latent is None:
if width is None:
width = 1024
if height is None:
height = 960
latent = generate_latent(width, height, seed, subseed, subseed_strength, seed_resize_from_h,
seed_resize_from_w, reset_noise)
else:
if isinstance(latent, torch.Tensor):
latent = {"samples":latent}
elif isinstance(latent, list):
latent = {"samples":torch.stack(latent, dim=0)}
else:
latent = latent
else:
latent = torch.from_numpy(np.array(init_image).astype(np.float32) / 255.0).unsqueeze(0)
latent = encode_latent(vae, latent, seed, subseed, subseed_strength, seed_resize_from_h, seed_resize_from_w)
assert isinstance(latent, dict), \
"Our Latents have to be in a dict format with the latent being the 'samples' value"
cond = []
if pooled_prompts is None and prompt is not None:
cond = get_conds(clip, prompt)
elif pooled_prompts is not None:
cond = pooled_prompts
if use_areas and areas is not None:
from nodes import ConditioningSetArea
area_setter = ConditioningSetArea()
for area in areas:
print("[deforum] Area Prompt:", area)
prompt = area.get("prompt", None)
if prompt:
new_cond = get_conds(clip, area["prompt"])
new_cond = area_setter.append(conditioning=new_cond, width=int(area["w"]), height=int(area["h"]), x=int(area["x"]),
y=int(area["y"]), strength=area["s"])[0]
cond += new_cond
n_cond = get_conds(clip, negative_prompt)
if next_prompt is not None and enable_prompt_blend:
if next_prompt != prompt and next_prompt != "":
if 0.0 < prompt_blend < 1.0:
next_cond = get_conds(clip, next_prompt)
cond = blend_tensors(cond[0], next_cond[0], blend_value=prompt_blend)
if cnet_image is not None:
cond = apply_controlnet(cond, controlnet, cnet_image, 1.0)
from nodes import common_ksampler as ksampler
#steps = int((strength) * steps) if (strength != 1.0 or not reset_noise) else steps
last_step = steps# if last_step is None else last_step
sample = common_ksampler_with_custom_noise(model=model,
seed=seed,
steps=steps,
cfg=scale,
sampler_name=sampler_name,
scheduler=scheduler,
positive=cond,
negative=n_cond,
latent=latent,
denoise=strength,
disable_noise=False,
start_step=0,
last_step=last_step,
force_full_denoise=True,
noise=rng)
if sample[0]["samples"].shape[0] == 1:
decoded = decode_sample(vae, sample[0]["samples"])
np_array = np.clip(255. * decoded.cpu().numpy(), 0, 255).astype(np.uint8)[0]
image = Image.fromarray(np_array)
# image = Image.fromarray(np.clip(255. * decoded.cpu().numpy(), 0, 255).astype(np.uint8)[0])
image = image.convert("RGB")
image.save('test.png', "PNG")
if return_latent:
return sample[0]["samples"], image
else:
return image
else:
images = []
x_samples = vae.decode_tiled(sample[0]["samples"])
for sample in x_samples:
np_array = np.clip(255. * sample.cpu().numpy(), 0, 255).astype(np.uint8)
image = Image.fromarray(np_array)
# image = Image.fromarray(np.clip(255. * decoded.cpu().numpy(), 0, 255).astype(np.uint8)[0])
image = image.convert("RGB")
images.append(image)
return images
def encode_latent(vae, latent, seed, subseed, subseed_strength, seed_resize_from_h, seed_resize_from_w, reset_noise=False):
global rng
subseed_strength = 0.6
with torch.inference_mode():
latent = latent.to(torch.float32)
latent = vae.encode_tiled(latent[:, :, :, :3])
latent = latent.to("cuda")
if rng is None or reset_noise:
rng = ImageRNGNoise(shape=latent[0].shape, seeds=[seed], subseeds=[subseed], subseed_strength=subseed_strength,
seed_resize_from_h=seed_resize_from_h, seed_resize_from_w=seed_resize_from_w)
# noise = self.rng.first()
# noise = slerp(subseed_strength, noise, latent)
# else:
# #noise = self.rng.next()
# #noise = slerp(subseed_strength, noise, latent)
# noise = latent
return {"samples": latent}
def generate_latent(width, height, seed, subseed, subseed_strength, seed_resize_from_h=None,
seed_resize_from_w=None, reset_noise=False):
shape = [4, height // 8, width // 8]
global rng
if rng is None or reset_noise:
rng = ImageRNGNoise(shape=shape, seeds=[seed], subseeds=[subseed], subseed_strength=subseed_strength,
seed_resize_from_h=seed_resize_from_h, seed_resize_from_w=seed_resize_from_w)
noise = rng.next()
# noise = torch.zeros([1, 4, width // 8, height // 8])
return {"samples": noise}
def get_conds(clip, prompt):
with torch.inference_mode():
# clip_skip = -1
# if clip_skip != clip_skip or clip.layer_idx != clip_skip:
# clip.layer_idx = clip_skip
# clip.clip_layer(clip_skip)
# self.clip_skip = clip_skip
tokens = clip.tokenize(prompt)
cond, pooled = clip.encode_from_tokens(tokens, return_pooled=True)
return [[cond, {"pooled_output": pooled}]]
def apply_controlnet(conditioning, control_net, image, strength):
with torch.inference_mode():
if strength == 0:
return (conditioning,)
c = []
control_hint = image.movedim(-1, 1)
for t in conditioning:
n = [t[0], t[1].copy()]
c_net = control_net.copy().set_cond_hint(control_hint, strength)
if 'control' in t[1]:
c_net.set_previous_controlnet(t[1]['control'])
n[1]['control'] = c_net
n[1]['control_apply_to_uncond'] = True
c.append(n)
return c
def decode_sample(vae, sample):
# with torch.inference_mode():
# sample = sample.to(torch.float32)
# vae.first_stage_model.cuda()
decoded = vae.decode_tiled(sample).detach()
return decoded
================================================
FILE: deforum_nodes/modules/deforum_comfyui_helpers.py
================================================
import base64
import math
import os
import random
import re
from io import BytesIO
import cv2
import numexpr
import numpy as np
import pandas as pd
import torch
from PIL import Image
from deforum.pipelines.deforum_animation.animation_helpers import DeforumAnimKeys
from deforum.pipelines.deforum_animation.pipeline_deforum_animation import interpolate_areas
blend_methods = ["linear", "sigmoidal", "gaussian", "pyramid", "none"]
import torch
import torch.nn.functional as F
import math
def pad_to_match(tensor1, tensor2):
"""
Pad the smaller tensor with zeros to match the size of the larger tensor.
"""
if tensor1.size() == tensor2.size():
return tensor1, tensor2
else:
# Find the size difference between the two tensors
diff_dims = [abs(s1 - s2) for s1, s2 in zip(tensor1.size(), tensor2.size())]
# Padding format (left, right, top, bottom) for 4D tensors, e.g., (batch, channels, height, width)
padding = []
for diff in reversed(diff_dims): # Reverse to start padding from last dimensions
padding.extend([diff // 2, diff - diff // 2])
# Apply padding to the smaller tensor
if tensor1.numel() < tensor2.numel():
tensor1 = F.pad(tensor1, padding)
else:
tensor2 = F.pad(tensor2, padding)
return tensor1, tensor2
def parse_widget(widget_info: dict) -> tuple:
parsed_widget = None
t = widget_info["type"]
if t == "dropdown":
parsed_widget = (widget_info["choices"],)
elif t == "checkbox":
parsed_widget = ("BOOLEAN", {"default": widget_info['default']})
elif t == "lineedit":
parsed_widget = ("STRING", {"default": widget_info['default']})
elif t == "spinbox":
parsed_widget = ("INT", {"default": widget_info['default']})
elif t == "doublespinbox":
parsed_widget = ("FLOAT", {"default": widget_info['default']})
return parsed_widget
def get_node_params(input_params):
data_info = {"required": {}, }
if input_params:
for name, widget_info in input_params.items():
data_info["required"][name] = parse_widget(widget_info)
data_info["optional"] = {"deforum_data": ("deforum_data",)}
return data_info
def get_current_keys(anim_args, seed, root, parseq_args=None, video_args=None, area_prompts=None):
use_parseq = False if parseq_args == None else True
anim_args.max_frames += 2
keys = DeforumAnimKeys(anim_args, seed) # if not use_parseq else ParseqAnimKeys(parseq_args, video_args)
areas = None
# Always enable pseudo-3d with parseq. No need for an extra toggle:
# Whether it's used or not in practice is defined by the schedules
if use_parseq:
anim_args.flip_2d_perspective = True
# expand prompts out to per-frame
if use_parseq and keys.manages_prompts():
prompt_series = keys.prompts
else:
prompt_series = None
if hasattr(root, 'animation_prompts'):
if root.animation_prompts is not None:
prompt_series = pd.Series([np.nan for a in range(anim_args.max_frames + 1)])
for i, prompt in root.animation_prompts.items():
if str(i).isdigit():
prompt_series[int(i)] = prompt
else:
prompt_series[int(numexpr.evaluate(i))] = prompt
prompt_series = prompt_series.ffill().bfill()
if area_prompts is not None:
areas = interpolate_areas(area_prompts, anim_args.max_frames)
anim_args.max_frames -= 2
return keys, prompt_series, areas
def tensor2pil(image):
if image is not None:
#with torch.inference_mode():
return Image.fromarray(np.clip(255. * image.detach().cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
else:
return None
# PIL to Tensor
def pil2tensor(image):
return torch.from_numpy(np.array(image).astype(np.float32) / 255.0).unsqueeze(0)
def tensor2np(img):
np_img = np.array(tensor2pil(img))
return np_img
def get_latent_with_seed(seed):
return torch.randn(generator=torch.manual_seed(seed))
def pil_image_to_base64(pil_image):
buffer = BytesIO()
pil_image.save(buffer, format="WEBP") # Or JPEG
return base64.b64encode(buffer.getvalue()).decode()
def tensor_to_webp_base64(tensor):
# Ensure tensor is in CPU and detach it from the computation graph
tensor = tensor.clone().detach().cpu()
# Convert tensor to a numpy array with value range [0, 255]
# Transpose the tensor to have the channel in the correct order (H, W, C)
np_image = (tensor.numpy().squeeze() * 255).clip(0, 255).astype(np.uint8)
if np_image.ndim == 2: # if it's a grayscale image, convert to RGB
np_image = cv2.cvtColor(np_image, cv2.COLOR_GRAY2RGB)
elif np_image.shape[0] == 3: # Convert CHW to HWC
np_image = np_image.transpose(1, 2, 0)
np_image = cv2.cvtColor(np_image, cv2.COLOR_BGR2RGB)
# Encode the numpy array to WEBP using OpenCV
compression_quality = 80
_, encoded_image = cv2.imencode('.webp', np_image, [cv2.IMWRITE_WEBP_QUALITY, compression_quality])
# Convert the encoded image to base64
base64_str = base64.b64encode(encoded_image).decode()
return base64_str
def generate_seed_list(max_frames, mode='fixed', start_seed=0, step=1):
"""
Generates a list of seed integers compatible with PyTorch in various manners.
Parameters:
- max_frames (int): The maximum number of frames/length of the seed list.
- mode (str): The mode of seed generation, one of 'fixed', 'random', 'ladder', 'incrementing', or 'decrementing'.
- start_seed (int): The starting seed value for modes other than 'random'.
- step (int): The step size for 'incrementing', 'decrementing', and 'ladder' modes.
Returns:
- list: A list of seed integers.
"""
if mode == 'fixed':
return [start_seed for _ in range(max_frames)]
elif mode == 'random':
return [random.randint(0, 2**32 - 1) for _ in range(max_frames)]
elif mode == 'ladder':
# Generate a ladder sequence where the sequence is repeated after reaching the max_frames
return [(start_seed + i // 2 * step if i % 2 == 0 else start_seed + (i // 2 + 1) * step) % (2**32) for i in range(max_frames)]
elif mode == 'incrementing' or 'iter':
return [(start_seed + i * step) % (2**32) for i in range(max_frames)]
elif mode == 'decrementing':
return [(start_seed - i * step) % (2**32) for i in range(max_frames)]
else:
raise ValueError("Invalid mode specified. Choose among 'fixed', 'random', 'ladder', 'incrementing', 'decrementing'.")
def find_next_index(output_dir, filename_prefix, format):
"""
Finds the next index for an MP4 file given an output directory and a filename prefix.
Parameters:
- output_dir: The directory where the MP4 files are saved.
- filename_prefix: The prefix for the filenames.
Returns:
- An integer representing the next index for a new MP4 file.
"""
# Compile a regular expression pattern to match the filenames
# This assumes the index is at the end of the filename, before the .mp4 extension
pattern = re.compile(rf"^{re.escape(filename_prefix)}_(\d+)\.{format.lower()}$")
max_index = -1
for filename in os.listdir(output_dir):
match = pattern.match(filename)
if match:
# Extract the current index from the filename
current_index = int(match.group(1))
# Update the max index found
if current_index > max_index:
max_index = current_index
# The next index is one more than the highest index found
next_index = max_index + 1
return next_index
import torch.nn.functional as F
def pyramid_blend(tensor1, tensor2, blend_value):
tensor1, tensor2 = pad_to_match(tensor1, tensor2)
# For simplicity, we'll use two levels of blending
downsampled1 = F.avg_pool2d(tensor1, 2)
downsampled2 = F.avg_pool2d(tensor2, 2)
blended_low = (1 - blend_value) * downsampled1 + blend_value * downsampled2
blended_high = tensor1 + tensor2 - F.interpolate(blended_low, scale_factor=2)
return blended_high
def gaussian_blend(tensor2, tensor1, blend_value):
tensor1, tensor2 = pad_to_match(tensor1, tensor2)
sigma = 0.5 # Adjust for desired smoothness
weight = math.exp(-((blend_value - 0.5) ** 2) / (2 * sigma ** 2))
return (1 - weight) * tensor1 + weight * tensor2
def sigmoidal_blend(tensor1, tensor2, blend_value):
tensor1, tensor2 = pad_to_match(tensor1, tensor2)
# Convert blend_value into a tensor with the same shape as tensor1 and tensor2
blend_tensor = torch.full_like(tensor1, blend_value)
weight = 1 / (1 + torch.exp(-10 * (blend_tensor - 0.5))) # Sigmoid function centered at 0.5
return (1 - weight) * tensor1 + weight * tensor2
def blend_tensors(obj1, obj2, blend_value, blend_method="linear"):
"""
Blends tensors in two given objects based on a blend value using various blending strategies.
"""
if blend_method == "linear":
weight = blend_value
obj1[0], obj2[0] = pad_to_match(obj1[0], obj2[0])
obj1[1]['pooled_output'], obj2[1]['pooled_output'] = pad_to_match(obj1[1]['pooled_output'], obj2[1]['pooled_output'])
blended_cond = (1 - weight) * obj1[0] + weight * obj2[0]
blended_pooled = (1 - weight) * obj1[1]['pooled_output'] + weight * obj2[1]['pooled_output']
elif blend_method == "sigmoidal":
blended_cond = sigmoidal_blend(obj1[0], obj2[0], blend_value)
blended_pooled = sigmoidal_blend(obj1[1]['pooled_output'], obj2[1]['pooled_output'], blend_value)
elif blend_method == "gaussian":
blended_cond = gaussian_blend(obj1[0], obj2[0], blend_value)
blended_pooled = gaussian_blend(obj1[1]['pooled_output'], obj2[1]['pooled_output'], blend_value)
elif blend_method == "pyramid":
blended_cond = pyramid_blend(obj1[0], obj2[0], blend_value)
blended_pooled = pyramid_blend(obj1[1]['pooled_output'], obj2[1]['pooled_output'], blend_value)
return [[blended_cond, {"pooled_output": blended_pooled}]]
================================================
FILE: deforum_nodes/modules/deforum_constants.py
================================================
class DeforumStorage:
_instance = None # Class attribute that holds the singleton instance
def __new__(cls, *args, **kwargs):
if cls._instance is None:
cls._instance = super(DeforumStorage, cls).__new__(cls, *args, **kwargs)
# Initialize your object here if needed
cls._instance.deforum_models = {}
cls._instance.deforum_cache = {}
cls._instance.deforum_depth_algo = ""
cls._instance.reset = None
return cls._instance
def __init__(self):
# Initialization code here. If there are attributes that should not be reinitialized,
# you can check if they already exist before setting them.
pass
================================================
FILE: deforum_nodes/modules/deforum_node_base.py
================================================
class DeforumDataBase:
# @classmethod
# def INPUT_TYPES(s):
# return s.params
RETURN_TYPES = (("deforum_data",))
FUNCTION = "get"
OUTPUT_NODE = True
CATEGORY = f"deforum/parameters"
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
def get(self, deforum_data=None, *args, **kwargs):
if deforum_data:
deforum_data.update(**kwargs)
else:
deforum_data = kwargs
return (deforum_data,)
================================================
FILE: deforum_nodes/modules/deforum_ui_data.py
================================================
import comfy
deforum_base_params = {
"width": {
"type": "spinbox",
"default": 512,
"min": 64,
"max": 4096,
"step": 64
},
"height": {
"type": "spinbox",
"default": 512,
"min": 64,
"max": 4096,
"step": 64
},
"seed_schedule": {
"type": "lineedit",
"default": "0: (-1)",
},
"seed_behavior": {
"type": "dropdown",
"choices": ["fixed", "random"]
},
"sampler_name": {
"type": "dropdown",
"choices": comfy.samplers.KSampler.SAMPLERS
},
"scheduler": {
"type": "dropdown",
"choices": comfy.samplers.KSampler.SCHEDULERS
},
# "steps": {
# "type": "spinbox",
# "default": 25,
# "min": 0,
# "max": 10000,
# "step": 1
# },
# "scale": {
# "type": "spinbox",
# "default": 7,
# "min": 0,
# "max": 10000,
# "step": 1
# },
# "n_batch": {
# "type": "spinbox",
# "default": 1,
# "min": 0,
# "max": 10000,
# "step": 1
# },
# "batch_name": {
# "type": "lineedit",
# "default": "Deforum_{timestring}"
# },
#
# "seed_iter_N": {
# "type": "spinbox",
# "default": 1,
# "min": 0,
# "max": 10000,
# "step": 1
# },
# "outdir": {
# "type": "lineedit",
# "default": "output/deforum"
# },
# "strength": {
# "type": "doublespinbox",
# "default": 0.8,
# "min": 0,
# "max": 1,
# "step": 0.01
# },
# "save_settings": {
# "type": "checkbox",
# "default": True
# },
# "save_sample_per_step": {
# "type": "checkbox",
# "default": False
# },
"prompt_weighting": {
"type": "checkbox",
"default": True
},
"normalize_prompt_weights": {
"type": "checkbox",
"default": True
},
"log_weighted_subprompts": {
"type": "checkbox",
"default": False
},
# "prompt": {
# "type": "lineedit",
# "default": ""
# },
# "timestring": {
# "type": "lineedit",
# "default": ""
# },
# "seed_internal": {
# "type": "lineedit",
# "default": "-1",
# },
}
deforum_anim_params = {
"animation_mode": {
"type": "dropdown",
"choices": ["None", "2D", "3D", "Video Input", "Interpolation"]
},
"max_frames": {
"type": "spinbox",
"min": 1,
"max": 100000,
"default": 120,
"step": 1
},
"border": {
"type": "dropdown",
"choices": ["wrap", "replicate", "zeros"]
},
# "resume_from_timestring": {
# "type": "checkbox",
# "default": False
# },
# "resume_timestring": {
# "type": "lineedit",
# "default": "20230129210106"
# },
# "use_looper": {
# "type": "checkbox",
# "default": False
# },
}
deforum_translation_params = {
"angle": {
"type": "lineedit",
"default": "0:(0)"
},
"zoom": {
"type": "lineedit",
"default": "0:(1.0025+0.002*sin(1.25*3.14*t/30))"
},
"translation_x": {
"type": "lineedit",
"default": "0:(0)"
},
"translation_y": {
"type": "lineedit",
"default": "0:(0)"
},
"translation_z": {
"type": "lineedit",
"default": "0:(1.75)"
},
"transform_center_x": {
"type": "lineedit",
"default": "0:(0.5)"
},
"transform_center_y": {
"type": "lineedit",
"default": "0:(0.5)"
},
"rotation_3d_x": {
"type": "lineedit",
"default": "0:(0)"
},
"rotation_3d_y": {
"type": "lineedit",
"default": "0:(0)"
},
"rotation_3d_z": {
"type": "lineedit",
"default": "0:(0)"
},
}
deforum_hybrid_video_params = {
# "hybrid_generate_inputframes": {
# "type": "checkbox",
# "default": False
# },
# "hybrid_generate_human_masks": {
# "type": "dropdown",
# "choices": ["None", "PNGs", "Video", "Both"],
# "default": "None"
# },
"hybrid_use_first_frame_as_init_image": {
"type": "checkbox",
"default": True
},
"hybrid_motion": {
"type": "dropdown",
"choices": ["None", "Optical Flow", "Perspective", "Affine"],
"default": "None"
},
"hybrid_motion_use_prev_img": {
"type": "checkbox",
"default": False
},
"hybrid_flow_consistency": {
"type": "checkbox",
"default": False
},
"hybrid_consistency_blur": {
"type": "spinbox",
"min": 0,
"max": 10,
"default": 2,
"step": 1
},
"hybrid_flow_method": {
"type": "dropdown",
"choices": ["RAFT", "DIS Medium", "DIS Fine", "Farneback"],
"default": "RAFT"
},
"hybrid_composite": {
"type": "dropdown",
"choices": ["None", "Normal", "Before Motion", "After Generation"],
"default": "None"
},
# "hybrid_use_init_image": {
# "type": "checkbox",
# "default": False
# },
# "hybrid_comp_mask_type": {
# "type": "dropdown",
# "choices": ["None", "Depth", "Video Depth", "Blend", "Difference"],
# "default": "None"
# },
# "hybrid_comp_mask_inverse": {
# "type": "checkbox",
# "default": False
# },
# "hybrid_comp_mask_equalize": {
# "type": "dropdown",
# "choices": ["None", "Before", "After", "Both"],
# "default": "None"
# },
# "hybrid_comp_mask_auto_contrast": {
# "type": "checkbox",
# "default": False
# },
# "hybrid_comp_save_extra_frames": {
# "type": "checkbox",
# "default": False
# },
}
deforum_hybrid_video_schedules = {
# "hybrid_comp_alpha_schedule": {
# "type": "lineedit",
# "default": "0:(0.5)"
# },
# "hybrid_comp_mask_blend_alpha_schedule": {
# "type": "lineedit",
# "default": "0:(0.5)"
# },
# "hybrid_comp_mask_contrast_schedule": {
# "type": "lineedit",
# "default": "0:(1)"
# },
# "hybrid_comp_mask_auto_contrast_cutoff_high_schedule": {
# "type": "lineedit",
# "default": "0:(100)"
# },
# "hybrid_comp_mask_auto_contrast_cutoff_low_schedule": {
# "type": "lineedit",
# "default": "0:(0)"
# },
"hybrid_flow_factor_schedule": {
"type": "lineedit",
"default": "0:(1)"
},
}
deforum_color_coherence_params = {
"color_coherence": {
"type": "dropdown",
"choices": ["None", "HSV", "LAB", "RGB", "Video Input", "Image"]
},
"color_coherence_image_path": {
"type": "lineedit",
"default": ""
},
"color_coherence_video_every_N_frames": {
"type": "spinbox",
"min": 1,
"max": 100,
"default": 1,
"step": 1
},
"color_force_grayscale": {
"type": "checkbox",
"default": False
},
"legacy_colormatch": {
"type": "checkbox",
"default": False
},
}
deforum_depth_params = {
"use_depth_warping": {
"type": "checkbox",
"default": True
},
"depth_algorithm": {
"type": "dropdown",
"choices": [
"Midas-3-Hybrid",
"Midas+AdaBins (old)",
"Midas-3.1-BeitLarge",
"AdaBins",
"Leres",
"Zoe",
"Zoe+AdaBins (old)",
],
"default": "Zoe"
},
"midas_weight": {
"type": "doublespinbox",
"min": 0,
"max": 1,
"default": 0.2,
"step": 0.01
},
"padding_mode": {
"type": "dropdown",
"choices": ["border", "reflection", "zeros"],
"default": "border"
},
"sampling_mode": {
"type": "dropdown",
"choices": ["bicubic", "bilinear", "nearest"],
"default": "bicubic"
},
"save_depth_maps": {
"type": "checkbox",
"default": False
},
}
deforum_cadence_params = {
"diffusion_cadence": {
"type": "spinbox",
"min": 0,
"max": 100,
"default": 4,
"step": 1
},
"optical_flow_cadence": {
"type": "dropdown",
"choices": ["None", "RAFT", "DIS Medium", "DIS Fine", "Farneback"]
},
"cadence_flow_factor_schedule": {
"type": "lineedit",
"default": "0: (1)"
},
"optical_flow_redo_generation": {
"type": "dropdown",
"choices": ["None", "RAFT", "DIS Medium", "DIS Fine", "Farneback"]
},
"redo_flow_factor_schedule": {
"type": "lineedit",
"default": "0: (1)"
},
"diffusion_redo": {
"type": "lineedit",
"default": "0"
},
}
deforum_video_init_params = {
"video_init_path": {
"type": "lineedit",
"default": "https://deforum.github.io/a1/V1.mp4"
},
"extract_nth_frame": {
"type": "spinbox",
"min": 1,
"max": 100,
"default": 1,
"step": 1
},
"extract_from_frame": {
"type": "spinbox",
"min": 0,
"max": 1000,
"default": 0,
"step": 1
},
"extract_to_frame": {
"type": "spinbox",
"min": -1,
"max": 1000,
"default": -1,
"step": 1
},
"overwrite_extracted_frames": {
"type": "checkbox",
"default": True
},
"use_mask_video": {
"type": "checkbox",
"default": False
},
"video_mask_path": {
"type": "lineedit",
"default": "https://deforum.github.io/a1/VM1.mp4"
},
}
deforum_masking_params = {
"use_mask": {
"type": "checkbox",
"default": False
},
"use_alpha_as_mask": {
"type": "checkbox",
"default": False
},
"mask_file": {
"type": "lineedit",
"default": "https://deforum.github.io/a1/M1.jpg"
},
"mask_image": {
"type": "lineedit",
"default": None
},
"noise_mask": {
"type": "lineedit",
"default": None
},
"invert_mask": {
"type": "checkbox",
"default": False
},
"mask_contrast_adjust": {
"type": "doublespinbox",
"default": 1.0,
"min": 0,
"max": 10,
"step": 0.1
},
"mask_brightness_adjust": {
"type": "doublespinbox",
"default": 1.0,
"min": 0,
"max": 10,
"step": 0.1
},
"overlay_mask": {
"type": "checkbox",
"default": True
},
"mask_overlay_blur": {
"type": "spinbox",
"default": 4,
"min": 0,
"max": 100,
"step": 1
},
"fill": {
"type": "spinbox",
"default": 1,
"min": 0,
"max": 10000,
"step": 1
},
"full_res_mask": {
"type": "checkbox",
"default": True
},
"full_res_mask_padding": {
"type": "spinbox",
"default": 4,
"min": 0,
"max": 10000,
"step": 1
},
}
deforum_diffusion_schedule_params = {
"noise_schedule": {
"type": "lineedit",
"default": "0: (0.065)"
},
"strength_schedule": {
"type": "lineedit",
"default": "0: (0.65)"
},
"contrast_schedule": {
"type": "lineedit",
"default": "0: (1.0)"
},
"cfg_scale_schedule": {
"type": "lineedit",
"default": "0: (7)"
},
"enable_steps_scheduling": {
"type": "checkbox",
"default": False
},
"steps_schedule": {
"type": "lineedit",
"default": "0: (25)"
},
"enable_ddim_eta_scheduling": {
"type": "checkbox",
"default": False
},
"ddim_eta_schedule": {
"type": "lineedit",
"default": "0:(0)"
},
"enable_ancestral_eta_scheduling": {
"type": "checkbox",
"default": False
},
"ancestral_eta_schedule": {
"type": "lineedit",
"default": "0:(1)"
}
}
deforum_noise_params = {
"enable_noise_multiplier_scheduling": {
"type": "checkbox",
"default": True
},
"noise_multiplier_schedule": {
"type": "lineedit",
"default": "0: (1.05)"
},
"amount_schedule": {
"type": "lineedit",
"default": "0: (0.1)"
},
"kernel_schedule": {
"type": "lineedit",
"default": "0: (5)"
},
"sigma_schedule": {
"type": "lineedit",
"default": "0: (1.0)"
},
"threshold_schedule": {
"type": "lineedit",
"default": "0: (0.0)"
},
"noise_type": {
"type": "dropdown",
"choices": ["uniform", "perlin"]
},
"perlin_w": {
"type": "spinbox",
"min": 1,
"max": 100,
"default": 8,
"step": 1
},
"perlin_h": {
"type": "spinbox",
"min": 1,
"max": 100,
"default": 8,
"step": 1
},
"perlin_octaves": {
"type": "spinbox",
"min": 1,
"max": 10,
"default": 4,
"step": 1
},
"perlin_persistence": {
"type": "doublespinbox",
"min": 0.1,
"max": 1.0,
"default": 0.5,
"step": 0.1
},
}
deforum_args_layout = {
"enable_perspective_flip": {
"type": "checkbox",
"default": False
},
"perspective_flip_theta": {
"type": "lineedit",
"default": "0:(0)"
},
"perspective_flip_phi": {
"type": "lineedit",
"default": "0:(0)"
},
"perspective_flip_gamma": {
"type": "lineedit",
"default": "0:(0)"
},
"perspective_flip_fv": {
"type": "lineedit",
"default": "0:(53)"
},
"fov_schedule": {
"type": "lineedit",
"default": "0: (70)"
},
"aspect_ratio_schedule": {
"type": "lineedit",
"default": "0: (1)"
},
"aspect_ratio_use_old_formula": {
"type": "checkbox",
"default": False
},
"near_schedule": {
"type": "lineedit",
"default": "0: (200)"
},
"far_schedule": {
"type": "lineedit",
"default": "0: (10000)"
},
"seed_schedule": {
"type": "lineedit",
"default": "0:(s), 1:(-1), \"max_f-2\":(-1), \"max_f-1\":(s)"
},
"pix2pix_img_cfg_scale": {
"type": "doublespinbox",
"min": 0.0,
"max": 10.0,
"default": 1.5,
"step": 0.1
},
"pix2pix_img_cfg_scale_schedule": {
"type": "lineedit",
"default": "0:(1.5)"
},
"enable_subseed_scheduling": {
"type": "checkbox",
"default": False
},
"subseed_schedule": {
"type": "lineedit",
"default": "0:(1)"
},
"subseed_strength_schedule": {
"type": "lineedit",
"default": "0:(0)"
},
"enable_sampler_scheduling": {
"type": "checkbox",
"default": False
},
"sampler_schedule": {
"type": "lineedit",
"default": "0: (\"Euler a\")"
},
"use_noise_mask": {
"type": "checkbox",
"default": False
},
"mask_schedule": {
"type": "lineedit",
"default": "0: (\"{video_mask}\")"
},
"noise_mask_schedule": {
"type": "lineedit",
"default": "0: (\"{video_mask}\")"
},
"enable_checkpoint_scheduling": {
"type": "checkbox",
"default": False
},
"checkpoint_schedule": {
"type": "lineedit",
"default": "0: (\"model1.ckpt\"), 100: (\"model2.safetensors\")"
},
"enable_clipskip_scheduling": {
"type": "checkbox",
"default": False
},
"clipskip_schedule": {
"type": "lineedit",
"default": "0: (2)"
},
}
deforum_image_init_params = {
"strength_0_no_init": {
"type": "checkbox",
"default": True
},
"init_image": {
"type": "lineedit",
"default": "https://deforum.github.io/a1/I1.png"
},
"init_sample": {
"type": "lineedit",
"default": None
},
"use_init": {
"type": "checkbox",
"default": False
},
}
================================================
FILE: deforum_nodes/modules/interp.py
================================================
import os
import random
import cv2
import numpy as np
# def optical_flow_cadence(i1, i2, cadence, method="DIS Medium"):
# imgs = []
# if i1 is not None and i2 is not None:
# imgs.append(i1)
# flow = get_flow_from_images(i1, i2, method)
# i2 = image_transform_optical_flow(i2, -flow)
# for i in range(1, cadence):
# weight = i / cadence
# flow_inc = flow * weight
# img = cv2.addWeighted(i1, 1 - weight, i2, weight, 0)
# img = image_transform_optical_flow(img, flow_inc, cv2.BORDER_REPLICATE)
# imgs.append(img)
# imgs.append(i2)
# return imgs
def optical_flow_cadence(i1, i2, cadence, method="DIS Medium"):
imgs = []
if i1 is not None and i2 is not None:
flow = get_flow_from_images(i1, i2, method)
# Warp i2 using the negative of the calculated flow to align it with i1
i2_warped = image_transform_optical_flow(i2, -flow)
for i in range(cadence):
weight = i / (cadence - 1) # Adjust weight calculation to include both endpoints
# Directly interpolate between i1 and warped i2 for each frame
img = cv2.addWeighted(i1, 1 - weight, i2_warped, weight, 0)
flow_inc = flow * weight
img = image_transform_optical_flow(img, flow_inc, cv2.BORDER_REPLICATE)
imgs.append(img)
# Ensure the last frame is exactly i2, to avoid any discrepancies
#imgs[-1] = i2
return imgs
def get_matrix_for_hybrid_motion(frame_idx, dimensions, inputfiles, hybrid_motion):
img1 = cv2.cvtColor(get_resized_image_from_filename(str(inputfiles[frame_idx - 1]), dimensions), cv2.COLOR_BGR2GRAY)
img2 = cv2.cvtColor(get_resized_image_from_filename(str(inputfiles[frame_idx]), dimensions), cv2.COLOR_BGR2GRAY)
matrix = get_transformation_matrix_from_images(img1, img2, hybrid_motion)
print(f"[deforum] Calculating {hybrid_motion} RANSAC matrix for frames {frame_idx} to {frame_idx + 1}")
return matrix
def get_matrix_for_hybrid_motion_prev(frame_idx, dimensions, inputfiles, prev_img, hybrid_motion):
# first handle invalid images from cadence by returning default matrix
height, width = prev_img.shape[:2]
if height == 0 or width == 0 or prev_img != np.uint8:
return get_hybrid_motion_default_matrix(hybrid_motion)
else:
prev_img_gray = cv2.cvtColor(prev_img, cv2.COLOR_BGR2GRAY)
img = cv2.cvtColor(get_resized_image_from_filename(str(inputfiles[frame_idx]), dimensions), cv2.COLOR_BGR2GRAY)
matrix = get_transformation_matrix_from_images(prev_img_gray, img, hybrid_motion)
print(f"[deforum] Calculating {hybrid_motion} RANSAC matrix for frames {frame_idx} to {frame_idx + 1}")
return matrix
def get_flow_for_hybrid_motion(frame_idx, dimensions, inputfiles, hybrid_frame_path, method,
do_flow_visualization=False):
print(f"[deforum] Calculating {method} optical flow for frames {frame_idx} to {frame_idx + 1}")
i1 = get_resized_image_from_filename(str(inputfiles[frame_idx]), dimensions)
i2 = get_resized_image_from_filename(str(inputfiles[frame_idx + 1]), dimensions)
flow = get_flow_from_images(i1, i2, method)
if do_flow_visualization:
save_flow_visualization(frame_idx, dimensions, flow, inputfiles, hybrid_frame_path)
return flow
def get_flow_for_hybrid_motion_prev(frame_idx, dimensions, inputfiles, hybrid_frame_path, prev_img, method,
do_flow_visualization=False):
print(f"[deforum] Calculating {method} optical flow for frames {frame_idx} to {frame_idx + 1}")
# first handle invalid images from cadence by returning default matrix
height, width = prev_img.shape[:2]
if height == 0 or width == 0:
flow = get_hybrid_motion_default_flow(dimensions)
else:
i1 = prev_img.astype(np.uint8)
i2 = get_resized_image_from_filename(str(inputfiles[frame_idx]), dimensions)
flow = get_flow_from_images(i1, i2, method)
if do_flow_visualization:
save_flow_visualization(frame_idx, dimensions, flow, inputfiles, hybrid_frame_path)
return flow
def get_flow_for_hybrid_motion_prev_imgs(frame_idx, dimensions, current_img, prev_img, method,
do_flow_visualization=False):
print(f"[deforum] Calculating {method} optical flow for frames {frame_idx} to {frame_idx + 1}")
# first handle invalid images from cadence by returning default matrix
height, width = prev_img.shape[:2]
if height == 0 or width == 0:
flow = get_hybrid_motion_default_flow(dimensions)
else:
i1 = prev_img.astype(np.uint8)
i2 = current_img.astype(np.uint8)
flow = get_flow_from_images(i1, i2, method)
# if do_flow_visualization:
# save_flow_visualization(frame_idx, dimensions, flow, inputfiles, hybrid_frame_path)
return flow
def image_transform_ransac(image_cv2, xform, hybrid_motion, border_mode=cv2.BORDER_REPLICATE):
if hybrid_motion == "Perspective":
return image_transform_perspective(image_cv2, xform, border_mode=border_mode)
else: # Affine
return image_transform_affine(image_cv2, xform, border_mode=border_mode)
def image_transform_optical_flow(img, flow, border_mode=cv2.BORDER_REPLICATE, flow_reverse=False):
if not flow_reverse:
flow = -flow
h, w = img.shape[:2]
flow[:, :, 0] += np.arange(w)
flow[:, :, 1] += np.arange(h)[:, np.newaxis]
return remap(img, flow, border_mode)
def image_transform_affine(image_cv2, xform, border_mode=cv2.BORDER_REPLICATE):
return cv2.warpAffine(
image_cv2,
xform,
(image_cv2.shape[1], image_cv2.shape[0]),
borderMode=border_mode
)
def image_transform_perspective(image_cv2, xform, border_mode=cv2.BORDER_REPLICATE):
return cv2.warpPerspective(
image_cv2,
xform,
(image_cv2.shape[1], image_cv2.shape[0]),
borderMode=border_mode
)
def get_hybrid_motion_default_matrix(hybrid_motion):
if hybrid_motion == "Perspective":
arr = np.array([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]])
else:
arr = np.array([[1., 0., 0.], [0., 1., 0.]])
return arr
def get_hybrid_motion_default_flow(dimensions):
cols, rows = dimensions
flow = np.zeros((rows, cols, 2), np.float32)
return flow
def get_transformation_matrix_from_images(img1, img2, hybrid_motion, max_corners=200, quality_level=0.01,
min_distance=30, block_size=3):
# Detect feature points in previous frame
prev_pts = cv2.goodFeaturesToTrack(img1,
maxCorners=max_corners,
qualityLevel=quality_level,
minDistance=min_distance,
blockSize=block_size)
if prev_pts is None or len(prev_pts) < 8 or img1 is None or img2 is None:
return get_hybrid_motion_default_matrix(hybrid_motion)
# Get optical flow
curr_pts, status, err = cv2.calcOpticalFlowPyrLK(img1, img2, prev_pts, None)
# Filter only valid points
idx = np.where(status == 1)[0]
prev_pts = prev_pts[idx]
curr_pts = curr_pts[idx]
if len(prev_pts) < 8 or len(curr_pts) < 8:
return get_hybrid_motion_default_matrix(hybrid_motion)
if hybrid_motion == "Perspective": # Perspective - Find the transformation between points
transformation_matrix, mask = cv2.findHomography(prev_pts, curr_pts, cv2.RANSAC, 5.0)
return transformation_matrix
else: # Affine - Compute a rigid transformation (without depth, only scale + rotation + translation)
transformation_rigid_matrix, rigid_mask = cv2.estimateAffinePartial2D(prev_pts, curr_pts)
return transformation_rigid_matrix
def get_flow_from_images(i1, i2, method):
if method == "DIS Medium":
r = get_flow_from_images_DIS(i1, i2, cv2.DISOPTICAL_FLOW_PRESET_MEDIUM)
elif method == "DIS Fast":
r = get_flow_from_images_DIS(i1, i2, cv2.DISOPTICAL_FLOW_PRESET_FAST)
elif method == "DIS UltraFast":
r = get_flow_from_images_DIS(i1, i2, cv2.DISOPTICAL_FLOW_PRESET_ULTRAFAST)
elif method == "DenseRLOF": # requires running opencv-contrib-python (full opencv) INSTEAD of opencv-python
r = get_flow_from_images_Dense_RLOF(i1, i2)
elif method == "SF": # requires running opencv-contrib-python (full opencv) INSTEAD of opencv-python
r = get_flow_from_images_SF(i1, i2)
elif method == "Farneback Fine":
r = get_flow_from_images_Farneback(i1, i2, 'fine')
else: # Farneback Normal:
r = get_flow_from_images_Farneback(i1, i2)
return r
def get_flow_from_images_DIS(i1, i2, preset):
i1 = cv2.cvtColor(i1, cv2.COLOR_BGR2GRAY)
i2 = cv2.cvtColor(i2, cv2.COLOR_BGR2GRAY)
dis = cv2.DISOpticalFlow_create(preset)
return dis.calc(i1, i2, None)
def get_flow_from_images_Dense_RLOF(i1, i2, last_flow=None):
return cv2.optflow.calcOpticalFlowDenseRLOF(i1, i2, flow=last_flow)
def get_flow_from_images_SF(i1, i2, last_flow=None, layers=3, averaging_block_size=2, max_flow=4):
return cv2.optflow.calcOpticalFlowSF(i1, i2, layers, averaging_block_size, max_flow)
def get_flow_from_images_Farneback(i1, i2, preset="normal", last_flow=None, pyr_scale=0.5, levels=3, winsize=15,
iterations=3, poly_n=5, poly_sigma=1.2, flags=0):
flags = cv2.OPTFLOW_FARNEBACK_GAUSSIAN # Specify the operation flags
pyr_scale = 0.5 # The image scale (<1) to build pyramids for each image
if preset == "fine":
levels = 13 # The number of pyramid layers, including the initial image
winsize = 77 # The averaging window size
iterations = 13 # The number of iterations at each pyramid level
poly_n = 15 # The size of the pixel neighborhood used to find polynomial expansion in each pixel
poly_sigma = 0.8 # The standard deviation of the Gaussian used to smooth derivatives used as a basis for the polynomial expansion
else: # "normal"
levels = 5 # The number of pyramid layers, including the initial image
winsize = 21 # The averaging window size
iterations = 5 # The number of iterations at each pyramid level
poly_n = 7 # The size of the pixel neighborhood used to find polynomial expansion in each pixel
poly_sigma = 1.2 # The standard deviation of the Gaussian used to smooth derivatives used as a basis for the polynomial expansion
i1 = cv2.cvtColor(i1, cv2.COLOR_BGR2GRAY)
i2 = cv2.cvtColor(i2, cv2.COLOR_BGR2GRAY)
flags = 0 # flags = cv2.OPTFLOW_USE_INITIAL_FLOW
flow = cv2.calcOpticalFlowFarneback(i1, i2, last_flow, pyr_scale, levels, winsize, iterations, poly_n, poly_sigma,
flags)
return flow
def save_flow_visualization(frame_idx, dimensions, flow, inputfiles, hybrid_frame_path):
flow_img_file = os.path.join(hybrid_frame_path, f"flow{frame_idx:05}.jpg")
flow_img = cv2.imread(str(inputfiles[frame_idx]))
flow_img = cv2.resize(flow_img, (dimensions[0], dimensions[1]), cv2.INTER_AREA)
flow_img = cv2.cvtColor(flow_img, cv2.COLOR_RGB2GRAY)
flow_img = cv2.cvtColor(flow_img, cv2.COLOR_GRAY2BGR)
flow_img = draw_flow_lines_in_grid_in_color(flow_img, flow)
flow_img = cv2.cvtColor(flow_img, cv2.COLOR_BGR2RGB)
cv2.imwrite(flow_img_file, flow_img)
print(f"[deforum] Saved optical flow visualization: {flow_img_file}")
def draw_flow_lines_in_grid_in_color(img, flow, step=8, magnitude_multiplier=1, min_magnitude=1, max_magnitude=10000):
flow = flow * magnitude_multiplier
h, w = img.shape[:2]
y, x = np.mgrid[step / 2:h:step, step / 2:w:step].reshape(2, -1).astype(int)
fx, fy = flow[y, x].T
lines = np.vstack([x, y, x + fx, y + fy]).T.reshape(-1, 2, 2)
lines = np.int32(lines + 0.5)
vis = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
vis = cv2.cvtColor(vis, cv2.COLOR_GRAY2BGR)
mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])
hsv = np.zeros((flow.shape[0], flow.shape[1], 3), dtype=np.uint8)
hsv[..., 0] = ang * 180 / np.pi / 2
hsv[..., 1] = 255
hsv[..., 2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX)
bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
vis = cv2.add(vis, bgr)
# Iterate through the lines
for (x1, y1), (x2, y2) in lines:
# Calculate the magnitude of the line
magnitude = np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2)
# Only draw the line if it falls within the magnitude range
if min_magnitude <= magnitude <= max_magnitude:
b = int(bgr[y1, x1, 0])
g = int(bgr[y1, x1, 1])
r = int(bgr[y1, x1, 2])
color = (b, g, r)
cv2.arrowedLine(vis, (x1, y1), (x2, y2), color, thickness=1, tipLength=0.1)
return vis
def draw_flow_lines_in_color(img, flow, threshold=3, magnitude_multiplier=1, min_magnitude=0, max_magnitude=10000):
# h, w = img.shape[:2]
vis = img.copy() # Create a copy of the input image
# Find the locations in the flow field where the magnitude of the flow is greater than the threshold
mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])
idx = np.where(mag > threshold)
# Create HSV image
hsv = np.zeros((flow.shape[0], flow.shape[1], 3), dtype=np.uint8)
hsv[..., 0] = ang * 180 / np.pi / 2
hsv[..., 1] = 255
hsv[..., 2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX)
# Convert HSV image to BGR
bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
# Add color from bgr
vis = cv2.add(vis, bgr)
# Draw an arrow at each of these locations to indicate the direction of the flow
for i, (y, x) in enumerate(zip(idx[0], idx[1])):
# Calculate the magnitude of the line
x2 = x + magnitude_multiplier * int(flow[y, x, 0])
y2 = y + magnitude_multiplier * int(flow[y, x, 1])
magnitude = np.sqrt((x2 - x) ** 2 + (y2 - y) ** 2)
# Only draw the line if it falls within the magnitude range
if min_magnitude <= magnitude <= max_magnitude:
if i % random.randint(100, 200) == 0:
b = int(bgr[y, x, 0])
g = int(bgr[y, x, 1])
r = int(bgr[y, x, 2])
color = (b, g, r)
cv2.arrowedLine(vis, (x, y), (x2, y2), color, thickness=1, tipLength=0.25)
return vis
def autocontrast_grayscale(image, low_cutoff=0, high_cutoff=100):
# Perform autocontrast on a grayscale np array image.
# Find the minimum and maximum values in the image
min_val = np.percentile(image, low_cutoff)
max_val = np.percentile(image, high_cutoff)
# Scale the image so that the minimum value is 0 and the maximum value is 255
image = 255 * (image - min_val) / (max_val - min_val)
# Clip values that fall outside the range [0, 255]
image = np.clip(image, 0, 255)
return image
def get_resized_image_from_filename(im, dimensions):
img = cv2.imread(im)
return cv2.resize(img, (dimensions[0], dimensions[1]), cv2.INTER_AREA)
def remap(img, flow, border_mode=cv2.BORDER_REFLECT_101):
# copyMakeBorder doesn't support wrap, but supports replicate. Replaces wrap with reflect101.
if border_mode == cv2.BORDER_WRAP:
border_mode = cv2.BORDER_REFLECT_101
h, w = img.shape[:2]
displacement = int(h * 0.25), int(w * 0.25)
larger_img = cv2.copyMakeBorder(img, displacement[0], displacement[0], displacement[1], displacement[1],
border_mode)
lh, lw = larger_img.shape[:2]
larger_flow = extend_flow(flow, lw, lh)
remapped_img = cv2.remap(larger_img, larger_flow, None, cv2.INTER_LINEAR, border_mode)
output_img = center_crop_image(remapped_img, w, h)
return output_img
def center_crop_image(img, w, h):
y, x, _ = img.shape
width_indent = int((x - w) / 2)
height_indent = int((y - h) / 2)
cropped_img = img[height_indent:y - height_indent, width_indent:x - width_indent]
return cropped_img
def extend_flow(flow, w, h):
# Get the shape of the original flow image
flow_h, flow_w = flow.shape[:2]
# Calculate the position of the image in the new image
x_offset = int((w - flow_w) / 2)
y_offset = int((h - flow_h) / 2)
# Generate the X and Y grids
x_grid, y_grid = np.meshgrid(np.arange(w), np.arange(h))
# Create the new flow image and set it to the X and Y grids
new_flow = np.dstack((x_grid, y_grid)).astype(np.float32)
# Shift the values of the original flow by the size of the border
flow[:, :, 0] += x_offset
flow[:, :, 1] += y_offset
# Overwrite the middle of the grid with the original flow
new_flow[y_offset:y_offset + flow_h, x_offset:x_offset + flow_w, :] = flow
# Return the extended image
return new_flow
================================================
FILE: deforum_nodes/modules/standalone_cadence.py
================================================
import os
import cv2
import numpy as np
import torch
from deforum.generators.deforum_flow_generator import rel_flow_to_abs_flow, abs_flow_to_rel_flow, get_flow_from_images, \
get_flow_for_hybrid_motion_prev_imgs
from deforum.utils.deforum_framewarp_utils import anim_frame_warp
from deforum.utils.deforum_hybrid_animation import get_matrix_for_hybrid_motion_prev_imgs
from deforum.utils.image_utils import image_transform_optical_flow, image_transform_ransac
import cv2
import numpy as np
import PIL.Image
class CadenceInterpolator:
def __init__(self):
self.turbo_prev_image, self.turbo_prev_frame_idx = None, 0
self.turbo_next_image, self.turbo_next_frame_idx = None, 0
self.start_frame = 0
self.depth = None
self.prev_img = None
self.prev_flow = None
def new_standalone_cadence(self, args, anim_args, root, keys, frame_idx, depth_model, raft_model, depth_strength=1.0, logger=None, hybrid_provider=None):
# global self.turbo_prev_image, self.turbo_next_image, self.turbo_prev_frame_idx, self.turbo_next_frame_idx, self.start_frame, self.depth
if not hasattr(self, 'prev_flow'):
self.prev_flow = None
self.prev_img = self.turbo_prev_image
goal_img = self.turbo_next_image
turbo_steps = int(anim_args.diffusion_cadence)
cadence_flow_factor = keys.cadence_flow_factor_schedule_series[frame_idx]
hybrid_flow_factor = keys.hybrid_flow_factor_schedule_series[frame_idx]
hybrid_comp_schedules = keys.hybrid_comp_alpha_schedule_series[frame_idx]
imgs = []
x = 0
# emit in-between frames
if turbo_steps > 1:
tween_frame_start_idx = max(self.start_frame, frame_idx - turbo_steps)
cadence_flow = None
for tween_frame_idx in range(tween_frame_start_idx, frame_idx):
# update progress during cadence
# state.job = f"frame {tween_frame_idx + 1}/{anim_args.max_frames}"
# state.job_no = tween_frame_idx + 1
# cadence vars
tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx)
advance_prev = self.turbo_prev_image is not None and tween_frame_idx > self.turbo_prev_frame_idx
advance_next = tween_frame_idx > self.turbo_next_frame_idx
# optical flow cadence setup before animation warping
if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None':
if keys.strength_schedule_series[tween_frame_start_idx] > 0:
if cadence_flow is None and self.turbo_prev_image is not None and self.turbo_next_image is not None:
cadence_flow = get_flow_from_images(self.turbo_prev_image, self.turbo_next_image,
anim_args.optical_flow_cadence, raft_model) / 2
self.turbo_next_image = image_transform_optical_flow(self.turbo_next_image, -cadence_flow, 1)
# if opts.data.get("deforum_save_gen_info_as_srt"):
# params_to_print = opts.data.get("deforum_save_gen_info_as_srt_params", ['Seed'])
# params_string = format_animation_params(keys, prompt_series, tween_frame_idx, params_to_print)
# write_frame_subtitle(srt_filename, tween_frame_idx, srt_frame_duration,
# f"F#: {tween_frame_idx}; Cadence: {tween < 1.0}; Seed: {args.seed}; {params_string}")
# params_string = None
# if logger:
# logger.update_row("Status", ["Cadence", str(tween_frame_idx), f"tween:{tween:0.2f}"])
# print(
# f"[deforum] Creating in-between {'' if cadence_flow is None else anim_args.optical_flow_cadence + ' optical flow '}cadence frame: {tween_frame_idx}; tween:{tween:0.2f};")
if depth_model is not None:
assert (self.turbo_next_image is not None)
self.depth = depth_model.predict(self.turbo_next_image, anim_args.midas_weight, root.half_precision) * depth_strength
if advance_prev:
self.turbo_prev_image, _, _ = anim_frame_warp(self.turbo_prev_image, args, anim_args, keys, tween_frame_idx,
depth_model, depth=self.depth, device=root.device,
half_precision=root.half_precision)
if advance_next:
self.turbo_next_image, _, _ = anim_frame_warp(self.turbo_next_image, args, anim_args, keys, tween_frame_idx,
depth_model, depth=self.depth, device=root.device,
half_precision=root.half_precision)
# hybrid video motion - warps self.turbo_prev_image or self.turbo_next_image to match motion
if tween_frame_idx > 0:
if anim_args.hybrid_motion in ['Affine', 'Perspective']:
if anim_args.hybrid_motion_use_prev_img:
matrix = get_matrix_for_hybrid_motion_prev_imgs(tween_frame_idx - 1, (args.width, args.height),
self.turbo_next_image, self.turbo_prev_image, anim_args.hybrid_motion)
if advance_prev:
self.turbo_prev_image = image_transform_ransac(self.turbo_prev_image, matrix, anim_args.hybrid_motion)
if advance_next:
self.turbo_next_image = image_transform_ransac(self.turbo_next_image, matrix, anim_args.hybrid_motion)
else:
if hybrid_provider is not None:
# matrix = get_matrix_for_hybrid_motion(tween_frame_idx - 1, (args.W, args.H), inputfiles,
# anim_args.hybrid_motion)
matrix = get_matrix_for_hybrid_motion_prev_imgs(tween_frame_idx - 1,
(args.width, args.height),
hybrid_provider[x + 1], hybrid_provider[x + 1],
anim_args.hybrid_motion)
if advance_prev:
self.turbo_prev_image = image_transform_ransac(self.turbo_prev_image, matrix, anim_args.hybrid_motion)
if advance_next:
self.turbo_next_image = image_transform_ransac(self.turbo_next_image, matrix, anim_args.hybrid_motion)
if anim_args.hybrid_motion in ['Optical Flow']:
if anim_args.hybrid_motion_use_prev_img:
flow = get_flow_for_hybrid_motion_prev_imgs(self.turbo_next_image,
self.prev_flow,
self.turbo_prev_image,
anim_args.hybrid_flow_method,
raft_model)
if advance_prev:
self.turbo_prev_image = image_transform_optical_flow(self.turbo_prev_image, flow,
#hybrid_comp_schedules['flow_factor'])
hybrid_flow_factor)
if advance_next:
self.turbo_next_image = image_transform_optical_flow(self.turbo_next_image, flow,
#hybrid_comp_schedules['flow_factor'])
hybrid_flow_factor)
self.prev_flow = flow
else:
if hybrid_provider is not None:
flow = get_flow_for_hybrid_motion_prev_imgs(hybrid_provider[x + 1],
self.prev_flow,
hybrid_provider[x],
anim_args.hybrid_flow_method,
raft_model)
if advance_prev:
self.turbo_prev_image = image_transform_optical_flow(self.turbo_prev_image, flow,
hybrid_flow_factor)
if advance_next:
self.turbo_next_image = image_transform_optical_flow(self.turbo_next_image, flow,
hybrid_flow_factor)
self.prev_flow = flow
# do optical flow cadence after animation warping
if cadence_flow is not None:
cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.width, args.height)
cadence_flow, _, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model,
depth=self.depth, device=root.device, half_precision=root.half_precision)
cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.width, args.height) * tween
if advance_prev:
self.turbo_prev_image = image_transform_optical_flow(self.turbo_prev_image, cadence_flow_inc,
cadence_flow_factor)
if advance_next:
self.turbo_next_image = image_transform_optical_flow(self.turbo_next_image, cadence_flow_inc,
cadence_flow_factor)
self.turbo_prev_frame_idx = self.turbo_next_frame_idx = tween_frame_idx
if self.turbo_prev_image is not None and tween < 1.0:
img = self.turbo_prev_image * (1.0 - tween) + self.turbo_next_image * tween
else:
img = self.turbo_next_image
# intercept and override to grayscale
if anim_args.color_force_grayscale:
img = cv2.cvtColor(img.astype(np.uint8), cv2.COLOR_BGR2GRAY)
img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
# overlay mask
# if args.overlay_mask and (anim_args.use_mask_video or args.use_mask):
# img = do_overlay_mask(args, anim_args, img, tween_frame_idx, True)
if logger:
# preview = preview
pil_img = PIL.Image.fromarray(img.astype(np.uint8))
logger.update_absolute(turbo_steps - (frame_idx - tween_frame_idx) + 1, preview=("JPEG", pil_img, 512))
imgs.append(img)
x += 1
return imgs
#
# def standalone_cadence(img, prev_img, frame_idx, cadence_frames, args, anim_args, keys, raft_model=None, depth_model=None):
# if img is not None:
# if cadence_frames > 1:
# if isinstance(img, PIL.Image.Image):
# img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
# if isinstance(prev_img, PIL.Image.Image):
# prev_img = cv2.cvtColor(np.array(prev_img), cv2.COLOR_RGB2BGR)
#
# tween_frame_start_idx = max(0, frame_idx - cadence_frames)
# cadence_flow = None
# return_frames = []
# for tween_frame_idx in range(tween_frame_start_idx, frame_idx):
# tween = float(tween_frame_idx - tween_frame_start_idx + 1) / float(frame_idx - tween_frame_start_idx)
# advance_prev = prev_img is not None and tween_frame_idx > 0
# advance_next = tween_frame_idx > 0
#
# if anim_args.animation_mode in ['2D', '3D'] and anim_args.optical_flow_cadence != 'None':
# if keys.strength_schedule_series[tween_frame_start_idx] > 0:
# if cadence_flow is None and prev_img is not None and img is not None:
# cadence_flow = get_flow_from_images(prev_img, img, anim_args.optical_flow_cadence, raft_model) / 2
# img = image_transform_optical_flow(img, -cadence_flow, 1)
#
# if depth_model is not None:
# depth = depth_model.predict(img, anim_args.midas_weight, True)
# if advance_prev:
# prev_img, _, _ = anim_frame_warp(prev_img, args, anim_args, keys, tween_frame_idx, depth_model=depth_model, depth=self.depth, device='cuda', half_precision=True)
# if advance_next:
# img, _, _ = anim_frame_warp(img, args, anim_args, keys, tween_frame_idx, depth_model=depth_model, depth=self.depth, device='cuda', half_precision=True)
#
# if cadence_flow is not None:
# cadence_flow = abs_flow_to_rel_flow(cadence_flow, args.width, args.height)
# cadence_flow, _, _ = anim_frame_warp(cadence_flow, args, anim_args, keys, tween_frame_idx, depth_model=depth_model, depth=self.depth, device='cuda', half_precision=True)
# cadence_flow_inc = rel_flow_to_abs_flow(cadence_flow, args.width, args.height) * tween
# if advance_prev:
# prev_img = image_transform_optical_flow(prev_img, cadence_flow_inc, 1)
# if advance_next:
# img = image_transform_optical_flow(img, cadence_flow_inc, 1)
#
# if prev_img is not None and tween < 1.0:
# combined_img = prev_img * (1.0 - tween) + img * tween
# else:
# combined_img = img
#
# if anim_args.color_force_grayscale:
# combined_img = cv2.cvtColor(combined_img.astype(np.uint8), cv2.COLOR_BGR2GRAY)
# combined_img = cv2.cvtColor(combined_img, cv2.COLOR_GRAY2BGR)
# res = combined_img.astype(np.uint8)
# return_frames.append(res)
# return return_frames, prev_img, img
================================================
FILE: deforum_nodes/nodes/__init__.py
================================================
================================================
FILE: deforum_nodes/nodes/deforum_advnoise_node.py
================================================
import secrets
import torch
import numpy as np
import math
import pywt
import torch.nn.functional as F
from opensimplex import OpenSimplex
"""
Noise types to implement:
Dithering Noise: Introduces a monochrome or color dithering effect that simulates more colors in images with limited palettes. This can create a retro, pixelated look or be used to prevent banding in gradients.
JPEG Compression Noise: Simulates the artifacts introduced by JPEG compression, such as blockiness and loss of detail, which can be relevant for training models to be robust against compression artifacts.
Moire Pattern Noise: Simulates the moiré pattern, an interference pattern that can occur when an image with repetitive details is digitized. This is particularly relevant for document scanning and OCR applications.
Defocus Blur Noise: Simulates the effect of an out-of-focus lens, which can be used to generate data for depth estimation or autofocus algorithms.
Lens Distortion Noise: Introduces barrel or pincushion distortion to simulate lens imperfections, useful for calibration and correction algorithms.
Chromatic Aberration: Simulates the color fringing around high-contrast edges caused by lens dispersion, adding realism to synthetic images or for artistic effects.
Color Shift Noise: Randomly shifts the color balance of the image, simulating white balance inaccuracies or artistic color grading.
Speckle Noise in Color: While speckle noise is typically applied in a monochromatic context, it can be adapted to color images to simulate sensor noise in color imaging devices.
Atmospheric Turbulence Noise: Simulates the effect of atmospheric distortion, relevant for images taken over long distances in outdoor environments, like satellite or surveillance imagery.
Halftone Pattern Noise: Simulates the halftone dots used in printed media, which can be useful for preparing models that need to understand scanned printed documents or for creating retro effects.
Holographic Interference Noise: Simulates the pattern seen in holograms, which could be interesting for security feature recognition or artistic effects.
Color Quantization Noise: While quantization was mentioned, specifically focusing on reducing the color depth of an image can simulate older digital or printed media.
Film Grain: Simulates the random texture characteristic of film photography, adding a nostalgic or artistic quality to digital images.
Pattern Overlay Noise: Overlaying a fixed pattern, such as a grid or text watermark, to simulate watermarked paper or screen overlays.
"""
class AddAdvancedNoiseNode:
"""
A node to add various types of advanced noise to an image using PyTorch.
"""
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"images": ("IMAGE",),
"noise_type": (["wavelet", "value", "flow", "turbulence",
"ridged_multifractal", "reaction_diffusion", "voronoi", "simplex"],), # "gabor"
"amount": ("FLOAT", {
"default": 0.1,
"min": 0.0,
"max": 100.0,
"step": 0.01,
"display": "number"
}),
},
"optional": {
"seed": ("INT", {
"default": None,
"min": 0,
"max": 2 ** 32 - 1,
"step": 1,
"display": "number"
}),
# Voronoi specific parameters
"num_points": ("INT", {
"default": 100,
"min": 10,
"max": 1000,
"step": 10,
"display": "number"
}),
# Simplex Noise specific parameters
"scale": ("FLOAT", {
"default": 0.1,
"min": 0.01,
"max": 1.0,
"step": 0.01,
"display": "number"
}),
"octaves": ("INT", {
"default": 1,
"min": 1,
"max": 10,
"step": 1,
"display": "number"
}),
"persistence": ("FLOAT", {
"default": 0.5,
"min": 0.1,
"max": 1.0,
"step": 0.01,
"display": "number"
}),
"lacunarity": ("FLOAT", {
"default": 2.0,
"min": 1.0,
"max": 3.0,
"step": 0.01,
"display": "number"
}),
# Wavelet Noise specific parameters
"wavelet": (["haar", "db1", "sym2", "coif1", "bior1.3", "rbio1.3"],),
"mode": (["symmetric", "periodic", "reflect", "zero-padding"],),
# Additional parameters specific to Gabor noise
"frequency": ("FLOAT", {
"default": 0.1,
"min": 0.01,
"max": 1.0,
"step": 0.01,
"display": "number"
}),
"theta": ("FLOAT", {
"default": 0.0,
"min": 0.0,
"max": 2 * math.pi,
"step": 0.01,
"display": "slider"
}),
"sigma_x": ("FLOAT", {
"default": 10.0,
"min": 1.0,
"max": 50.0,
"step": 1.0,
"display": "number"
}),
"sigma_y": ("FLOAT", {
"default": 10.0,
"min": 1.0,
"max": 50.0,
"step": 1.0,
"display": "number"
}),
# Value Noise specific parameters
"res": ("INT", {
"default": 16,
"min": 4,
"max": 64,
"step": 1,
"display": "number"
}),
# Additional parameters for Flow Noise
"flow_scale": ("FLOAT", {
"default": 0.1,
"min": 0.01,
"max": 1.0,
"step": 0.01,
"display": "number"
}),
"flow_angle": ("FLOAT", {
"default": 0.0,
"min": 0.0,
"max": 2 * np.pi,
"step": 0.01,
"display": "number"
}),
# Reaction Diffusion Noise
"steps": ("INT", {
"default": 10,
"min": 1,
"max": 100,
"step": 1,
"display": "number"
}),
"Du": ("FLOAT", {
"default": 0.16,
"min": 0.01,
"max": 0.5,
"step": 0.01,
"display": "number"
}),
"Dv": ("FLOAT", {
"default": 0.08,
"min": 0.01,
"max": 0.5,
"step": 0.01,
"display": "number"
}),
"feed_rate": ("FLOAT", {
"default": 0.035,
"min": 0.01,
"max": 0.1,
"step": 0.001,
"display": "slider"
}),
"kill_rate": ("FLOAT", {
"default": 0.06,
"min": 0.01,
"max": 0.1,
"step": 0.001,
"display": "slider"
}),
# Additional parameters specific to each noise type can be defined here
}
}
RETURN_TYPES = ("IMAGE","IMAGE",)
RETURN_NAMES = ("NOISED_IMAGE","NOISE",)
FUNCTION = "add_advanced_noise"
CATEGORY = "deforum/noise"
display_name = "Add Advanced Noise"
def add_advanced_noise(self, images, noise_type, amount, seed=None, **kwargs):
B, H, W, C = images.shape
if seed is not None:
np.random.seed(seed)
torch.manual_seed(seed)
# Adjusted noise function dictionary
noise_function = {
"voronoi": generate_voronoi_noise,
"simplex": lambda H, W, **kw: generate_simplex_noise(H, W, **kw).unsqueeze(-1).repeat(1, 1, 1, C),
"wavelet": lambda H, W, **kw: generate_wavelet_noise(H, W, **kw).unsqueeze(-1).repeat(1, 1, 1, C),
"gabor": lambda H, W, **kw: generate_gabor_noise(H, W, **kw),
"value": lambda H, W, **kw: generate_value_noise(H, W, **kw).unsqueeze(-1).repeat(1, 1, 1, C),
"flow": lambda H, W, **kw: generate_flow_noise(H, W, **kw),
"turbulence": lambda H, W, **kw: generate_turbulence_noise(H, W, **kw),
"ridged_multifractal": lambda H, W, **kw: generate_ridged_multifractal_noise(H, W, **kw),
"reaction_diffusion": lambda H, W, **kw: generate_reaction_diffusion_noise(H, W, **kw),
# Additional noise functions adjusted similarly
}.get(noise_type, lambda H, W, **kw: torch.zeros((B, H, W, C)))
# Generate the noise pattern
noise = noise_function(H, W, seed=seed, **kwargs)
# Adjust for the case where noise is not returned with a batch dimension or channel dimension
if noise.dim() == 2: # Assuming noise is of shape (H, W)
noise = noise.unsqueeze(0).unsqueeze(-1) # Add batch and channel dimension
noise = noise.expand(B, -1, -1, C) # Expand to match (B, H, W, C)
elif noise.dim() == 3: # Assuming noise is of shape (B, H, W)
noise = noise.unsqueeze(-1) # Add channel dimension
noise = noise.expand(-1, -1, -1, C) # Expand to match (B, H, W, C)
# Ensure noise tensor device matches the images tensor device
noise = noise.to(images.device)
# from ..modules.better_resize.resize_right import resize
# belnd_img = resize(images, noise.shape, antialiasing=False)
# Apply the noise to the images
noisy_images = images.clone() + amount * noise.clone()
return (torch.clamp(noisy_images, 0, 1),noise,)
def generate_voronoi_noise(H, W, num_points=100, seed=None, *args, **kwargs):
"""
Generates a 2D Voronoi (Worley) Noise pattern.
Args:
H, W (int): The height and width of the output noise pattern.
num_points (int): Number of seed points to generate.
seed (int, optional): Random seed for reproducibility.
Returns:
torch.Tensor: A tensor containing the Voronoi noise pattern.
"""
if seed is not None:
torch.manual_seed(seed)
np.random.seed(seed)
# Generate random seed points
seeds_x = np.random.uniform(0, W, num_points)
seeds_y = np.random.uniform(0, H, num_points)
seeds = np.stack((seeds_x, seeds_y), axis=-1)
# Initialize the noise tensor
noise = torch.full((H, W), float('inf'))
# Compute distance from each pixel to the closest seed point
for y in range(H):
for x in range(W):
distances = torch.tensor([((x - sx) ** 2 + (y - sy) ** 2) for sx, sy in seeds])
noise[y, x] = torch.min(distances).sqrt()
# Normalize the noise pattern
noise = (noise - noise.min()) / (noise.max() - noise.min())
return noise
def generate_simplex_noise(H, W, scale=0.05, octaves=4, persistence=0.5, lacunarity=2.0, seed=None, **kwargs):
if seed is not None:
gen = OpenSimplex(seed)
else:
gen = OpenSimplex()
noise_pattern = np.zeros((H, W), dtype=np.float32)
for y in range(H):
for x in range(W):
nx = x / W - 0.5 # Normalize x coordinate to [-0.5, 0.5]
ny = y / H - 0.5 # Normalize y coordinate to [-0.5, 0.5]
noise_val = 0
amplitude = 1.0
frequency = 1.0
max_value = 0.0
for o in range(octaves):
noise_val += gen.noise2(x=nx * scale * frequency, y=ny * scale * frequency) * amplitude
max_value += amplitude
amplitude *= persistence
frequency *= lacunarity
# Normalize each point individually
noise_pattern[y, x] = (noise_val / max_value + 1) / 2 # Normalize to [0, 1]
return torch.from_numpy(noise_pattern)
def generate_wavelet_noise(H, W, octaves=3, wavelet='haar', mode='symmetric', **kwargs):
seed = kwargs.get('seed', secrets.randbelow(2 ** 32))
if seed is not None:
np.random.seed(seed)
# Generate initial random noise
noise = np.random.randn(H, W)
# Perform wavelet decomposition and reconstruction
coeffs = pywt.wavedec2(noise, wavelet=wavelet, mode=mode, level=octaves)
wavelet_noise = pywt.waverec2(coeffs, wavelet=wavelet, mode=mode)
# Normalize the wavelet noise to have values between 0 and 1
wavelet_noise = (wavelet_noise - np.min(wavelet_noise)) / (np.max(wavelet_noise) - np.min(wavelet_noise))
# Convert to a PyTorch tensor and add a channel dimension since input is assumed to be BxHxWxC
wavelet_noise_tensor = torch.from_numpy(wavelet_noise).float().unsqueeze(0).unsqueeze(0)
# Interpolate to match input size (H, W)
# Note: 'align_corners=False' is generally used for non-exact sizes but you might want to experiment
# with 'align_corners=True' or 'recompute_scale_factor=True' depending on your specific use case
wavelet_noise_tensor = F.interpolate(wavelet_noise_tensor, size=(H, W), mode='bilinear', align_corners=False)
# Remove the extra dimensions added for interpolation if necessary
wavelet_noise_tensor = wavelet_noise_tensor.squeeze()
return wavelet_noise_tensor
def generate_gabor_kernel(frequency, theta, sigma_x, sigma_y, n_stds=3, grid_size=28):
"""
Generates a Gabor kernel.
Args:
frequency (float): The frequency of the sinusoidal component.
theta (float): Orientation of the Gabor filter in radians.
sigma_x, sigma_y (float): Standard deviation of the Gaussian envelope along x and y axes.
n_stds (int): Number of standard deviations to consider for the kernel size.
grid_size (int): The size of the output tensor.
Returns:
torch.Tensor: A 2D tensor with the Gabor kernel.
"""
xmax = ymax = grid_size // 2
xmin = ymin = -xmax
(y, x) = torch.meshgrid(torch.arange(ymin, ymax + 1), torch.arange(xmin, xmax + 1))
x = x.float()
y = y.float()
# Convert theta to a tensor to use with torch.cos and torch.sin
theta_tensor = torch.tensor(theta)
rotx = x * torch.cos(theta_tensor) + y * torch.sin(theta_tensor)
roty = -x * torch.sin(theta_tensor) + y * torch.cos(theta_tensor)
g = torch.zeros(y.shape)
g = torch.exp(-0.5 * ((rotx ** 2 / sigma_x ** 2) + (roty ** 2 / sigma_y ** 2))) * torch.cos(
2 * np.pi * frequency * rotx)
return g
def generate_gabor_noise(H, W, frequency=0.1, theta=0.0, sigma_x=10.0, sigma_y=10.0, batch_size=1, channels=3, **kwargs):
"""
Generates a 2D Gabor Noise pattern and scales it to the size of the input images.
Args:
H, W (int): Height and width of the output noise pattern.
frequency (float): Frequency of the sinusoidal component.
theta (float): Orientation of the Gabor filter in radians.
sigma_x, sigma_y (float): Standard deviation of the Gaussian envelope.
batch_size (int): Number of images in the batch.
channels (int): Number of channels in the images.
Returns:
torch.Tensor: A tensor containing the Gabor noise pattern, resized to match input images dimensions.
"""
gabor_kernel = generate_gabor_kernel(frequency, theta, sigma_x, sigma_y, grid_size=min(H, W))
# Center the kernel in a larger tensor matching the image dimensions
pad_height = (H - gabor_kernel.size(0)) // 2
pad_width = (W - gabor_kernel.size(1)) // 2
pad_height_extra = H - gabor_kernel.size(0) - pad_height * 2
pad_width_extra = W - gabor_kernel.size(1) - pad_width * 2
# Pad the kernel to match the target size
gabor_padded = F.pad(gabor_kernel, [pad_width, pad_width + pad_width_extra, pad_height, pad_height_extra], "constant", 0)
# Ensure the pattern has the correct dimensions: batch_size x H x W x channels
gabor_padded = gabor_padded.unsqueeze(0) # Add a batch dimension
gabor_padded = gabor_padded.unsqueeze(-1) # Add a channel dimension
gabor_padded = gabor_padded.expand(batch_size, -1, -1, channels) # Expand to match the full dimensions
return gabor_padded
def generate_value_noise(H, W, res=16, seed=None, **kwargs):
"""
Generates a 2D Value Noise pattern with edge handling.
Args:
H, W (int): Height and width of the output noise pattern.
res (int): Resolution of the noise grid (smaller values mean smoother noise).
seed (int, optional): Seed for random number generator.
Returns:
torch.Tensor: A tensor containing the Value noise pattern.
"""
if seed is not None:
torch.manual_seed(seed)
# Create a grid of random values
grid_res_x, grid_res_y = W // res + 1, H // res + 1
random_values = torch.rand((grid_res_y, grid_res_x))
# Initialize output noise pattern
noise_pattern = torch.zeros((H, W))
# Generate the noise
for y in range(H):
for x in range(W):
# Grid cell coordinates in the random grid
cell_x, cell_y = x // res, y // res
# Ensure indices stay within bounds
next_cell_x = min(cell_x + 1, grid_res_x - 1)
next_cell_y = min(cell_y + 1, grid_res_y - 1)
# Local x and y in the grid cell
local_x, local_y = (x % res) / res, (y % res) / res
# Corners of the cell in the grid
c00 = random_values[cell_y, cell_x]
c10 = random_values[cell_y, next_cell_x]
c01 = random_values[next_cell_y, cell_x]
c11 = random_values[next_cell_y, next_cell_x]
# Interpolate between grid corner values
nx0 = lerp(c00, c10, fade(local_x))
nx1 = lerp(c01, c11, fade(local_x))
nxy = lerp(nx0, nx1, fade(local_y))
noise_pattern[y, x] = nxy
# Normalize the noise pattern
noise_pattern = (noise_pattern - noise_pattern.min()) / (noise_pattern.max() - noise_pattern.min())
return noise_pattern
def lerp(a, b, t):
"""Linear interpolation between a and b with t in [0, 1]."""
return a + t * (b - a)
def fade(t):
"""Fade function as defined by Ken Perlin; eases coordinate values."""
return t * t * t * (t * (t * 6 - 15) + 10)
def generate_flow_noise_pattern(H, W, scale=0.1, angle=0.0, seed=None):
"""
Generates a 2D Flow Noise pattern using a base noise to perturb the sampling space of another noise layer.
Args:
H, W (int): Height and width of the output noise pattern.
scale (float): Scale of the underlying noise.
angle (float): Base flow direction in radians.
seed (int, optional): Random seed for reproducibility.
Returns:
torch.Tensor: A tensor containing the Flow noise pattern.
"""
# Base noise for perturbation
base_noise = generate_simplex_noise(H, W, scale=scale, seed=seed)
# Generate gradient from angle for flow direction
flow_x = torch.cos(torch.tensor(angle))
flow_y = torch.sin(torch.tensor(angle))
# Apply perturbation based on the base noise and flow direction
perturb_x = base_noise * flow_x
perturb_y = base_noise * flow_y
# Another layer of noise for actual flow appearance
flow_noise = generate_simplex_noise(H, W, scale=scale / 2, seed=seed if seed is None else seed + 1)
# Final flow noise is a combination of perturbed coordinates and the flow noise layer
final_noise = flow_noise + perturb_x + perturb_y
# Normalize the noise pattern
final_noise = (final_noise - final_noise.min()) / (final_noise.max() - final_noise.min())
return final_noise
def generate_flow_noise(H, W, flow_scale=0.1, flow_angle=0.0, seed=None, batch_size=1, channels=3, **kwargs):
"""
Generates a 2D Flow Noise pattern and resizes it to match the input image size.
Args:
H, W (int): Height and width of the input images.
scale, angle: Parameters for the flow noise generation.
seed (int, optional): Seed for random number generator.
batch_size (int): The batch size of the input images.
channels (int): The channel count of the input images.
Returns:
torch.Tensor: A tensor of the Flow noise pattern, matched to input dimensions.
"""
# Generate the base noise pattern
base_noise = generate_flow_noise_pattern(H, W, flow_scale, flow_angle, seed)
# Ensure the noise tensor has the same dimensions as the input: (batch_size, H, W, channels)
# Interpolate noise to match the input size if needed
noise_resized = F.interpolate(base_noise.unsqueeze(0).unsqueeze(0), size=(H, W), mode='bilinear',
align_corners=False)
# Expand to match the batch and channel dimensions
noise_resized = noise_resized.repeat(batch_size, channels, 1, 1)
noise_resized = noise_resized.permute(0, 2, 3, 1) # Rearrange dimensions to match (B, H, W, C)
return noise_resized
def generate_turbulence_noise(H, W, scale=0.05, octaves=4, persistence=0.5, lacunarity=2.0, seed=None, **kwargs):
if seed is not None:
gen = OpenSimplex(seed)
else:
gen = OpenSimplex()
noise_pattern = np.zeros((H, W), dtype=np.float32)
for y in range(H):
for x in range(W):
nx = x / W - 0.5 # Normalize x coordinate to [-0.5, 0.5]
ny = y / H - 0.5 # Normalize y coordinate to [-0.5, 0.5]
noise_val = 0
amplitude = 1.0
frequency = 1.0
max_value = 0.0
for o in range(octaves):
# The key change for turbulence: using the absolute value of the noise
noise_val += abs(gen.noise2(x=nx * scale * frequency, y=ny * scale * frequency)) * amplitude
max_value += amplitude
amplitude *= persistence
frequency *= lacunarity
# Normalize each point individually
noise_pattern[y, x] = (noise_val / max_value)
# Normalize the whole pattern to [0, 1] range if desired
noise_pattern = (noise_pattern - noise_pattern.min()) / (noise_pattern.max() - noise_pattern.min())
return torch.from_numpy(noise_pattern)
def generate_ridged_multifractal_noise(H, W, scale=0.05, octaves=4, persistence=0.5, lacunarity=2.0, seed=None, **kwargs):
if seed is not None:
gen = OpenSimplex(seed)
else:
gen = OpenSimplex()
noise_pattern = np.zeros((H, W), dtype=np.float32)
for y in range(H):
for x in range(W):
nx = x / W - 0.5 # Normalize x coordinate to [-0.5, 0.5]
ny = y / H - 0.5 # Normalize y coordinate to [-0.5, 0.5]
noise_val = 0
amplitude = 1.0
frequency = 1.0
max_value = 0.0
for o in range(octaves):
signal = gen.noise2(x=nx * scale * frequency, y=ny * scale * frequency)
# Transformation for ridged noise: invert and square the signal
signal = 1.0 - abs(signal)
signal *= signal
noise_val += signal * amplitude
max_value += amplitude
amplitude *= persistence
frequency *= lacunarity
# Normalize each point individually
noise_pattern[y, x] = (noise_val / max_value)
# Normalize the whole pattern to [0, 1] range if desired
noise_pattern = (noise_pattern - noise_pattern.min()) / (noise_pattern.max() - noise_pattern.min())
return torch.from_numpy(noise_pattern)
def generate_reaction_diffusion_noise(H, W, steps=100, Du=0.16, Dv=0.08, feed_rate=0.035, kill_rate=0.06, seed=None, **kwargs):
"""
Generates a 2D Reaction-Diffusion pattern using a simplified Gray-Scott model.
Args:
H, W (int): Height and width of the output noise pattern.
steps (int): Number of simulation steps.
Du, Dv (float): Diffusion rates for substances A and B.
feed_rate, kill_rate (float): Rates for feed and kill reactions.
seed (int, optional): Seed for initializing the pattern.
Returns:
torch.Tensor: A tensor containing the Reaction-Diffusion noise pattern.
"""
if seed is not None:
torch.manual_seed(seed)
np.random.seed(seed)
# Initialize A and B concentrations with A being fully saturated and B having a noise pattern
A = torch.ones((H, W), dtype=torch.float32)
B = torch.rand((H, W), dtype=torch.float32) * 0.25 # Starting with a low concentration of B
# Laplacian kernel for diffusion
laplacian_kernel = torch.tensor([[0.05, 0.2, 0.05],
[0.2, -1.0, 0.2],
[0.05, 0.2, 0.05]], dtype=torch.float32)
for _ in range(steps):
LA = torch.nn.functional.conv2d(A.unsqueeze(0).unsqueeze(0), laplacian_kernel.unsqueeze(0).unsqueeze(0), padding='same').squeeze()
LB = torch.nn.functional.conv2d(B.unsqueeze(0).unsqueeze(0), laplacian_kernel.unsqueeze(0).unsqueeze(0), padding='same').squeeze()
# Reaction-diffusion equations
AB2 = A * B.pow(2)
dA = Du * LA - AB2 + feed_rate * (1 - A)
dB = Dv * LB + AB2 - (feed_rate + kill_rate) * B
A += dA
B += dB
# Normalize the B concentration pattern to be between 0 and 1
pattern = (B - B.min()) / (B.max() - B.min())
return pattern.unsqueeze(0) # Add a channel dimension
================================================
FILE: deforum_nodes/nodes/deforum_audiosync_nodes.py
================================================
import math
import pandas as pd
import scipy.signal
import scipy.ndimage
import numpy as np
from pydub import AudioSegment
from scipy.signal import butter, filtfilt, find_peaks
import librosa
schedule_types = [
"angle", "transform_center_x", "transform_center_y", "zoom", "translation_x", "translation_y", "translation_z",
"rotation_3d_x", "rotation_3d_y", "rotation_3d_z", "perspective_flip_theta", "perspective_flip_phi",
"perspective_flip_gamma", "perspective_flip_fv", "noise_schedule", "strength_schedule", "contrast_schedule",
"cfg_scale_schedule", "ddim_eta_schedule", "ancestral_eta_schedule", "pix2pix_img_cfg_scale", "subseed_schedule",
"subseed_strength_schedule", "checkpoint_schedule", "steps_schedule", "seed_schedule", "sampler_schedule",
"clipskip_schedule", "noise_multiplier_schedule", "mask_schedule", "noise_mask_schedule", "kernel_schedule",
"sigma_schedule", "amount_schedule", "threshold_schedule", "aspect_ratio", "fov", "near",
"cadence_flow_factor_schedule", "redo_flow_factor_schedule", "far", "hybrid_comp_alpha_schedule",
"hybrid_comp_mask_blend_alpha_schedule", "hybrid_comp_mask_contrast_schedule",
"hybrid_comp_mask_auto_contrast_cutoff_high_schedule", "hybrid_comp_mask_auto_contrast_cutoff_low_schedule",
"hybrid_flow_factor_schedule"
]
import numpy as np
from scipy.signal import savgol_filter
class ExtractDominantNoteAmplitude:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"audio_fft": ("AUDIO_FFT",),
},
"optional": {
"min_frequency": ("FLOAT", {"default": 20.0, "min": 0.0, "max": 20000.0, "step": 1.0}),
"max_frequency": ("FLOAT", {"default": 8000.0, "min": 0.0, "max": 20000.0, "step": 1.0}),
"magnitude_threshold": ("FLOAT", {"default": 0.01, "min": 0.0, "max": 1.0, "step": 0.01}),
"smoothing_window_size": ("INT", {"default": 5, "min": 1, "max": 101, "step": 1}),
# Odd number, savgol_filter constraint
},
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE","AMPLITUDE",)
RETURN_NAMES = ("dominant_note_amplitude","phase_amplitude",)
FUNCTION = "extract"
def extract(self, audio_fft, min_frequency, max_frequency, magnitude_threshold, smoothing_window_size):
dominant_frequencies_amplitude = []
dominant_frequencies_phase = []
for fft_data in audio_fft:
indices = fft_data.get_indices_for_frequency_bands(min_frequency, max_frequency)
magnitudes = np.abs(fft_data.fft)[indices]
phases = np.angle(fft_data.fft)[indices]
# Optional: Apply smoothing to magnitudes
if smoothing_window_size > 1 and len(magnitudes) > smoothing_window_size:
try:
magnitudes = savgol_filter(magnitudes, smoothing_window_size, 3) # window size, polynomial order
except ValueError:
# In case the smoothing_window_size is inappropriate for the data length
pass
# Apply magnitude threshold
magnitudes[magnitudes < magnitude_threshold] = 0
# Identify dominant frequency index, its amplitude, and phase
if magnitudes.size > 0: # Ensure there's data after filtering
dominant_index = np.argmax(magnitudes)
dominant_frequencies_amplitude.append(magnitudes[dominant_index])
dominant_frequencies_phase.append(phases[dominant_index])
else:
dominant_frequencies_amplitude.append(0)
dominant_frequencies_phase.append(0) # Use a default phase (e.g., 0) when no dominant frequency is found
# Convert to numpy array for consistency
return (np.array(dominant_frequencies_amplitude), np.array(dominant_frequencies_phase),)
class InverseFFTNode:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"magnitude": ("AMPLITUDE",),
"phase": ("AMPLITUDE",),
},
}
CATEGORY = "signal_processing"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("time_domain_signal",)
FUNCTION = "synthesize"
def synthesize(self, magnitude, phase):
# Ensure magnitude and phase arrays have the same length
if magnitude.shape != phase.shape:
raise ValueError("Magnitude and phase arrays must have the same length.")
# Combine magnitude and phase to form the complex spectrum
complex_spectrum = magnitude * np.exp(1j * phase)
# Perform the inverse FFT to convert the spectrum back to the time domain
time_domain_signal = np.fft.ifft(complex_spectrum)
# Since the input is real, the output should also be real. We take the real part to avoid numerical errors that might introduce imaginary parts.
return (np.real(time_domain_signal),)
class AmplitudeToAudio:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"amplitude": ("AMPLITUDE",),
"phase": ("AMPLITUDE",),
},
"optional": {
# You can define additional parameters such as sample rate if needed
"sample_rate": ("INT", {"default": 44100, "min": 1, "max": 192000, "step": 1}),
},
}
CATEGORY = "AudioProcessing"
RETURN_TYPES = ("AUDIO",)
RETURN_NAMES = ("audio_samples",)
FUNCTION = "convert"
def convert(self, amplitude, phase, sample_rate=44100):
if len(amplitude) != len(phase):
raise ValueError("Amplitude and phase sequences must have the same length.")
# Reconstruct the complex spectrum from amplitude and phase
complex_spectrum = amplitude * np.exp(1j * phase)
# Perform the inverse FFT
audio_samples = np.fft.ifft(complex_spectrum).real # Taking the real part as the output
# Convert the normalized audio samples to 16-bit integers
audio_samples_int16 = np.int16(audio_samples / np.max(np.abs(audio_samples)) * 32767)
# Creating an AudioSegment object from the NumPy array
audio_segment = AudioSegment(
audio_samples_int16.tobytes(),
frame_rate=sample_rate,
sample_width=audio_samples_int16.dtype.itemsize,
channels=1
)
# Create the AudioData object
audio_data = AudioData(audio_segment)
return (audio_data,)
class AudioData:
def __init__(self, audio_segment) -> None:
# Extract the sample rate
self.sample_rate = audio_segment.frame_rate
# Get the number of audio channels
self.num_channels = audio_segment.channels
# Extract the audio data as a NumPy array
samples = np.array(audio_segment.get_array_of_samples())
if audio_segment.channels == 2:
self.audio_data = np.reshape(samples, (len(samples)//2, 2))
else:
self.audio_data = samples
def get_channel_audio_data(self, channel: int):
if channel < 0 or channel >= self.num_channels:
raise IndexError(f"Channel '{channel}' out of range. total channels is '{self.num_channels}'.")
return self.audio_data[channel::self.num_channels]
def get_channel_fft(self, channel: int):
audio_data = self.get_channel_audio_data(channel)
return np.fft.fft(audio_data)
def xor(a, b):
return bool(a) ^ bool(b)
# class DeforumAmplitudeToKeyframeSeriesNode:
# @classmethod
# def INPUT_TYPES(s):
# return {"required":
# {"type_name": (schedule_types,),
# "amplitude": ("AMPLITUDE",),
# },
# "optional":
# {
# "max_frames": ("INT", {"default": 500, "min": 1, "max": 16500, "step": 1}),
# "math": ("STRING", {"default": "x/100"}),
# "filter_window": ("INT", {"default": 0, "min": 0, "max": 100, "step": 1}),
# "deforum_frame_data": ("DEFORUM_FRAME_DATA",)
# }
# }
#
# RETURN_TYPES = ("DEFORUM_FRAME_DATA", "AMPLITUDE", "STRING")
# FUNCTION = "convert"
# display_name = "Amplitude to Schedule"
# CATEGORY = "deforum/audio"
#
# def safe_eval(self, expr, t, x, max_f):
# allowed_locals = {
# "sin": math.sin, "cos": math.cos, "tan": math.tan,
# "asin": math.asin, "acos": math.acos, "atan": math.atan,
# "sinh": math.sinh, "cosh": math.cosh, "tanh": math.tanh,
# "asinh": math.asinh, "acosh": math.acosh, "atanh": math.atanh,
# "sqrt": math.sqrt, "exp": math.exp, "log": math.log,
# "abs": math.fabs, "pow": math.pow, "floor": math.floor, "ceil": math.ceil,
# "round": round, "min": min, "max": max, "pi": math.pi, "e": math.e,
# "factorial": math.factorial,
# "xor": xor, # Add custom xor function
# "t": t, "x": x, "max_f": max_f,
# # Adding boolean operations as lambda functions to simulate 'and', 'or', 'not'
# "and": lambda a, b: a and b,
# "or": lambda a, b: a or b,
# "not": lambda a: not a,
# }
# try:
# # Directly using 'if else' in expr is supported by Python syntax
# # Logical operations 'and', 'or', 'not', and 'xor' are supported via the above definitions
# return eval(expr, {"__builtins__": {}}, allowed_locals)
# except NameError as e:
# raise ValueError(f"Invalid expression: {e}")
# except Exception as e:
# raise ValueError(f"Error evaluating expression: {repr(e)}")
#
# @classmethod
# def IS_CHANGED(self, *args, **kwargs):
# return float("NaN")
#
# def convert(self, type_name, amplitude, max_frames=1500, math="x/100", filter_window=0, deforum_frame_data={}):
# max_f = int(max_frames)
#
# # Optionally apply a smoothing filter to the amplitude data
# if filter_window > 0:
# amplitude_smoothed = np.convolve(amplitude, np.ones(filter_window) / filter_window, mode='same')
# else:
# amplitude_smoothed = amplitude
#
# frame_index = 0
# modified_amplitude_list = []
# for x in amplitude_smoothed:
# modified_value = self.safe_eval(math, frame_index, x, max_f)
# modified_amplitude_list.append(modified_value)
# frame_index += 1
#
# # Ensure the modified amplitude list can still be used as an amplitude input
# modified_amplitude_series = pd.Series(modified_amplitude_list)
#
# # Format the series for visualization or further processing
# formatted_strings = [f"{idx}:({val})" for idx, val in modified_amplitude_series.items()]
# formatted_string = ", ".join(formatted_strings[:-1]) + " and " + formatted_strings[-1] if len(formatted_strings) > 1 else formatted_strings[0]
#
# # Update deforum_frame_data if necessary
# if "keys" in deforum_frame_data:
# deforum_frame_data["keys"][f"{type_name}_series"] = modified_amplitude_series.to_dict()
#
# return (deforum_frame_data, modified_amplitude_series.to_numpy(), formatted_string,)
import numpy as np
import pandas as pd
import math
class DeforumAmplitudeToKeyframeSeriesNode:
@classmethod
def INPUT_TYPES(cls):
return {"required":
{"type_name": (schedule_types,),
"amplitude": ("AMPLITUDE",),
},
"optional":
{
"max_frames": ("INT", {"default": 500, "min": 1, "max": 16500, "step": 1}),
"math": ("STRING", {"default": "x/100"}),
"filter_window": ("INT", {"default": 0, "min": 0, "max": 100, "step": 1}),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
"y": ("AMPLITUDE",),
"z": ("AMPLITUDE",),
}
}
RETURN_TYPES = ("DEFORUM_FRAME_DATA", "AMPLITUDE", "STRING")
FUNCTION = "convert"
display_name = "Amplitude to Schedule"
CATEGORY = "deforum/audio"
def safe_eval(self, expr, t, x, max_f, y=None, z=None):
allowed_locals = {
"sin": math.sin, "cos": math.cos, "tan": math.tan,
"asin": math.asin, "acos": math.acos, "atan": math.atan,
"sinh": math.sinh, "cosh": math.cosh, "tanh": math.tanh,
"asinh": math.asinh, "acosh": math.acosh, "atanh": math.atanh,
"sqrt": math.sqrt, "exp": math.exp, "log": math.log,
"abs": math.fabs, "pow": math.pow, "floor": math.floor, "ceil": math.ceil,
"round": round, "min": min, "max": max, "pi": math.pi, "e": math.e,
"factorial": math.factorial,
"xor": lambda a, b: a ^ b, # Python's `^` operator for XOR
"t": t, "x": x, "max_f": max_f, "y": y, "z": z,
# Boolean operations via lambda functions
"and": lambda a, b: a and b,
"or": lambda a, b: a or b,
"not": lambda a: not a,
}
try:
return eval(expr, {"__builtins__": {}}, allowed_locals)
except NameError as e:
raise ValueError(f"Invalid expression: {e}")
except Exception as e:
raise ValueError(f"Error evaluating expression: {repr(e)}")
def convert(self, type_name, amplitude, max_frames=500, math="x/100", filter_window=0, deforum_frame_data={}, y=None, z=None):
max_f = int(max_frames)
# Apply smoothing filter to the amplitude data if needed
amplitude_smoothed = np.convolve(amplitude, np.ones(filter_window) / filter_window, mode='same') if filter_window > 0 else amplitude
y_smoothed = np.convolve(y, np.ones(filter_window) / filter_window, mode='same') if y is not None and filter_window > 0 else y
z_smoothed = np.convolve(z, np.ones(filter_window) / filter_window, mode='same') if z is not None and filter_window > 0 else z
frame_index = 0
modified_amplitude_list = []
for idx, x in enumerate(amplitude_smoothed):
y_val = y_smoothed[idx] if y is not None else None
z_val = z_smoothed[idx] if z is not None else None
modified_value = self.safe_eval(math, frame_index, x, max_f, y=y_val, z=z_val)
modified_amplitude_list.append(modified_value)
frame_index += 1
modified_amplitude_series = pd.Series(modified_amplitude_list)
formatted_strings = [f"{idx}:({val})" for idx, val in modified_amplitude_series.items()]
formatted_string = ", ".join(formatted_strings[:-1]) + " and " + formatted_strings[-1] if len(formatted_strings) > 1 else formatted_strings[0]
if "keys" in deforum_frame_data:
deforum_frame_data["keys"][f"{type_name}_series"] = modified_amplitude_series.to_dict()
return (deforum_frame_data, modified_amplitude_series.to_numpy(), formatted_string,)
class DeforumAmplitudeToString:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"amplitude": ("AMPLITUDE",),
},
}
RETURN_TYPES = ("STRING",)
# RETURN_NAMES = ("POSITIVE", "NEGATIVE")
FUNCTION = "convert"
display_name = "Amplitude to String"
CATEGORY = "deforum/audio"
@classmethod
def IS_CHANGED(self, *args, **kwargs):
# Force re-evaluation of the node
return float("NaN")
def convert(self, amplitude):
return (str(amplitude),)
class DerivativeOfAmplitude:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"amplitude": ("AMPLITUDE",),
},}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("amplitude_derivative",)
FUNCTION = "derive"
display_name = "Derive Amplitude"
def derive(self, amplitude,):
derivative = np.diff(amplitude, prepend=amplitude[0]) # Use np.diff with prepend to maintain array length
return (derivative,)
class SpectralCentroid:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"audio_fft": ("AUDIO_FFT",),
},}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("FLOAT",)
RETURN_NAMES = ("spectral_centroid",)
FUNCTION = "calculate"
display_name = "Amplitude Spectral Centoid"
def calculate(self, audio_fft,):
magnitudes = np.abs(audio_fft) / len(audio_fft)
frequencies = np.linspace(0, audio_fft.sample_rate / 2, len(magnitudes))
centroid = np.sum(frequencies * magnitudes) / np.sum(magnitudes)
return (centroid,)
class TimeSmoothing:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"amplitude": ("AMPLITUDE",),
},
"optional": {
"window_size": ("INT", {"default": 5, "min": 1}),
}
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("smoothed_amplitude",)
FUNCTION = "smooth"
display_name = "Amplitude Time Smoothing"
def smooth(self, amplitude, window_size,):
smoothed_amplitude = np.convolve(amplitude, np.ones(window_size)/window_size, mode='same')
return (smoothed_amplitude,)
class BeatDetection:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"audio": ("AUDIO",),
},
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("beat_times",)
FUNCTION = "detect"
display_name = "Audio to Beat Amplitude"
def detect(self, audio):
beat_times = self.find_beat_times(audio)
# Assuming beat_times are indices, we convert these to time values
beat_times_in_seconds = beat_times / audio.sample_rate
beat_times_series = pd.Series(beat_times_in_seconds)
return (beat_times_series,)
def find_beat_times(self, audio):
if audio.num_channels > 1:
audio_data = audio.get_channel_audio_data(0)
else:
audio_data = audio.audio_data
envelope = self.extract_envelope(audio_data, audio.sample_rate)
smoothed_envelope = scipy.ndimage.gaussian_filter1d(envelope, sigma=5)
normalized_envelope = (smoothed_envelope - np.min(smoothed_envelope)) / (np.max(smoothed_envelope) - np.min(smoothed_envelope))
peaks, _ = scipy.signal.find_peaks(normalized_envelope, height=0.3) # The height threshold may need adjustment
return peaks
def extract_envelope(self, audio_data, sample_rate):
analytic_signal = scipy.signal.hilbert(audio_data)
envelope = np.abs(analytic_signal)
return envelope
class FrequencyRangeAmplitude:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"audio": ("AUDIO",),
},
"optional": {
"frequency_range": ("TUPLE", {"default": (20, 20000)}),
"window_size": ("INT", {"default": 1}),
"inverted": ("BOOLEAN", {"default": False}),
},
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("frequency_range_amplitude",)
FUNCTION = "analyze_frequency_range"
display_name = "Frequency Range Amplitude"
def analyze_frequency_range(self, audio, frequency_range=(20, 20000), window_size=1, inverted=False):
if audio.num_channels > 1:
audio_data = audio.get_channel_audio_data(0)
else:
audio_data = audio.audio_data
sample_rate = audio.sample_rate
# Apply bandpass filter
filtered_audio = self.bandpass_filter(audio_data, frequency_range[0], frequency_range[1], sample_rate)
# Calculate FFT
fft_result = np.fft.rfft(filtered_audio)
fft_freqs = np.fft.rfftfreq(len(filtered_audio), 1 / sample_rate)
# Extract amplitude
amplitudes = np.abs(fft_result)
# Normalize and smooth if necessary
normalized_amplitudes = self.normalize(amplitudes, window_size=window_size, inverted=inverted)
return (normalized_amplitudes,)
def bandpass_filter(self, data, lowcut, highcut, sample_rate, order=5):
nyquist = 0.5 * sample_rate
low = lowcut / nyquist
high = highcut / nyquist
b, a = butter(order, [low, high], btype='band')
y = filtfilt(b, a, data)
return y
def normalize(self, data, min_value=0, max_value=1, window_size=1, inverted=False):
# Normalization and smoothing logic as described in the notebook
# Return normalized data
pass # Implement normalization and optional smoothing as described
class BeatDetectionNode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"audio": ("AUDIO",),
"sample_rate": ("INT",{"default": 44100, "min": 8000, "max": 200000, "step":1}),
}
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("beat_times",)
FUNCTION = "detect_beats"
display_name = "Beat Detection v2"
def detect_beats(self, audio, sample_rate):
if audio.num_channels > 1:
audio_data = audio.get_channel_audio_data(0)
else:
audio_data = audio.audio_data
# Assuming audio_data is already pre-processed to be in the correct format
tempo, beats = librosa.beat.beat_track(y=audio_data, sr=sample_rate)
beat_times = librosa.frames_to_time(beats, sr=sample_rate)
return (beat_times,)
class TempoChangeDetectionNode:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {"audio": ("AUDIO",)},
"optional": {"threshold": ("FLOAT", {"default": 0.5, "min":0.0, "max":1250.0, "step":0.1})},
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("tempo_change_times",)
FUNCTION = "detect_tempo_changes"
display_name = "Tempo Change Detection"
def detect_tempo_changes(self, audio, threshold=0.5, fps=24.0):
audio_data = audio.get_channel_audio_data(0) if audio.num_channels > 1 else audio.audio_data
sample_rate = audio.sample_rate
# Calculate onset strength
onset_env = librosa.onset.onset_strength(y=audio_data.astype(np.float32), sr=sample_rate)
tempogram = librosa.feature.tempogram(onset_envelope=onset_env, sr=sample_rate)
dynamic_tempo = np.mean(tempogram, axis=1)
times = librosa.frames_to_time(np.arange(len(dynamic_tempo)), sr=sample_rate)
# Detect change points
change_points = np.abs(np.diff(dynamic_tempo)) > threshold
change_times = times[:-1][change_points]
# Convert change times to a sequence of amplitudes with the desired fps
# First, create a boolean array indicating change points
full_length = int(np.ceil(times[-1] * fps))
change_points_sequence = np.zeros(full_length, dtype=bool)
interpolated_times = np.arange(0, times[-1], step=1.0 / fps)
for time in interpolated_times:
index = int(time * fps)
if index < full_length:
change_points_sequence[index] = True
# Optionally, fill in missing values if needed, but here we map detected changes directly
# Return the sequence as desired
print(len(change_points_sequence)) # Debug print to check the length of the output sequence
return (change_points_sequence,)
class ConvertNormalizedAmplitude:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {"normalized_amp": ("NORMALIZED_AMPLITUDE",)},
}
CATEGORY = "deforum/audio"
RETURN_TYPES = ("AMPLITUDE",)
RETURN_NAMES = ("amplitude",)
FUNCTION = "convert_normalized_amplitude"
display_name = "Convert Normalized Amplitude"
def convert_normalized_amplitude(self, normalized_amp):
return (normalized_amp,)
================================================
FILE: deforum_nodes/nodes/deforum_cache_nodes.py
================================================
class DeforumCacheLatentNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"latent": ("LATENT",),
"cache_index": ("INT", {"default":0, "min": 0, "max": 16, "step": 1})
}
}
RETURN_TYPES = (("LATENT",))
FUNCTION = "cache_it"
CATEGORY = f"deforum/cache"
display_name = "Cache Latent"
OUTPUT_NODE = True
def cache_it(self, latent=None, cache_index=0):
from ..mapping import gs
from ..mapping import gs
if "latent" not in gs.deforum_cache:
gs.deforum_cache["latent"] = {}
gs.deforum_cache["latent"][cache_index] = latent
return (latent,)
class DeforumGetCachedLatentNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"cache_index": ("INT", {"default": 0, "min": 0, "max": 16, "step": 1})
}}
RETURN_TYPES = (("LATENT",))
FUNCTION = "get_cached_latent"
CATEGORY = f"deforum/cache"
OUTPUT_NODE = True
display_name = "Load Cached Latent"
def get_cached_latent(self, cache_index=0):
from ..mapping import gs
if gs.reset:
return (None,)
latent_dict = gs.deforum_cache.get("latent", {})
latent = latent_dict.get(cache_index)
return (latent,)
class DeforumCacheImageNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"cache_index": ("INT", {"default":0, "min": 0, "max": 16, "step": 1})
}
}
RETURN_TYPES = (("IMAGE",))
FUNCTION = "cache_it"
CATEGORY = f"deforum/cache"
display_name = "Cache Image"
OUTPUT_NODE = True
def cache_it(self, image=None, cache_index=0):
from ..mapping import gs
if "image" not in gs.deforum_cache:
gs.deforum_cache["image"] = {}
if image is not None:
gs.deforum_cache["image"][cache_index] = image.detach().clone()
return (image,)
class DeforumGetCachedImageNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"cache_index": ("INT", {"default": 0, "min": 0, "max": 16, "step": 1})
}}
RETURN_TYPES = (("IMAGE","MASK"))
RETURN_NAMES = ("IMAGE","MASK")
FUNCTION = "get_cached_latent"
CATEGORY = f"deforum/cache"
OUTPUT_NODE = True
display_name = "Load Cached Image"
def get_cached_latent(self, cache_index=0):
from ..mapping import gs
if gs.reset:
return (None, None)
img_dict = gs.deforum_cache.get("image", {})
image = img_dict.get(cache_index)
mask = None
if image is not None:
mask = image[:, :, :, 0]
image = image.detach().clone()
return (image ,mask,)
class DeforumCacheStringNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"input_string": ("STRING",{"default":""}),
"cache_index": ("INT", {"default":0, "min": 0, "max": 16, "step": 1})
}
}
RETURN_TYPES = (("STRING",))
FUNCTION = "cache_it"
CATEGORY = f"deforum/cache"
display_name = "Cache String"
OUTPUT_NODE = True
def cache_it(self, input_string=None, cache_index=0):
from ..mapping import gs
if "string" not in gs.deforum_cache:
gs.deforum_cache["string"] = {}
gs.deforum_cache["string"][cache_index] = input_string
return (input_string,)
class DeforumGetCachedStringNode:
@classmethod
def IS_CHANGED(cls, text, autorefresh):
# Force re-evaluation of the node
if autorefresh == "Yes":
return float("NaN")
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"cache_index": ("INT", {"default": 0, "min": 0, "max": 16, "step": 1})
}}
RETURN_TYPES = (("STRING",))
FUNCTION = "get_cached_string"
CATEGORY = f"deforum/cache"
OUTPUT_NODE = True
display_name = "Load Cached String"
def get_cached_string(self, cache_index=0):
from ..mapping import gs
img_dict = gs.deforum_cache.get("string", {})
string = img_dict.get(cache_index)
return (str(string),)
================================================
FILE: deforum_nodes/nodes/deforum_cnet_nodes.py
================================================
class DeforumControlNetApply:
@classmethod
def INPUT_TYPES(s):
return {"required": {"conditioning": ("CONDITIONING", ),
"control_net": ("CONTROL_NET", ),
"strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 10.0, "step": 0.01})
},
"optional": {"image": ("IMAGE",)}
}
RETURN_TYPES = ("CONDITIONING",)
FUNCTION = "apply_controlnet"
display_name = "Apply ControlNet [safe]"
CATEGORY = "deforum/controlnet"
def apply_controlnet(self, conditioning, control_net, strength, image=None):
if strength == 0 or image is None:
return (conditioning, )
c = []
control_hint = image.movedim(-1,1)
for t in conditioning:
n = [t[0], t[1].copy()]
c_net = control_net.copy().set_cond_hint(control_hint, strength)
if 'control' in t[1]:
c_net.set_previous_controlnet(t[1]['control'])
n[1]['control'] = c_net
n[1]['control_apply_to_uncond'] = True
c.append(n)
return (c, )
================================================
FILE: deforum_nodes/nodes/deforum_cond_nodes.py
================================================
import random
import torch
from ..modules.deforum_comfyui_helpers import blend_tensors, blend_methods
class DeforumConditioningBlendNode:
def __init__(self):
self.prompt = None
self.n_prompt = None
self.cond = None
self.n_cond = None
@classmethod
def INPUT_TYPES(s):
return {"required":
{"clip": ("CLIP",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
"blend_method": ([blend_methods]),
}
}
RETURN_TYPES = ("CONDITIONING", "CONDITIONING")
RETURN_NAMES = ("POSITIVE", "NEGATIVE")
FUNCTION = "fn"
display_name = "Blend Conditionings"
CATEGORY = "deforum/conditioning"
def fn(self, clip, deforum_frame_data, blend_method):
areas = deforum_frame_data.get("areas")
negative_prompt = deforum_frame_data.get("negative_prompt", "")
n_cond = self.get_conditioning(prompt=negative_prompt, clip=clip)
if not areas:
prompt = deforum_frame_data.get("prompt", "")
next_prompt = deforum_frame_data.get("next_prompt", None)
cond = self.get_conditioning(prompt=prompt, clip=clip)
# image = self.getInputData(2)
# controlnet = self.getInputData(3)
prompt_blend = deforum_frame_data.get("prompt_blend", 0.0)
#method = self.content.blend_method.currentText()
if blend_method != 'none':
if next_prompt != prompt and prompt_blend != 0.0 and next_prompt is not None:
next_cond = self.get_conditioning(prompt=next_prompt, clip=clip)
cond = blend_tensors(cond[0], next_cond[0], prompt_blend, blend_method)
print(f"[deforum] Blending next prompt: {next_prompt}, with alpha: {prompt_blend} ]")
else:
from nodes import ConditioningSetArea
area_setter = ConditioningSetArea()
cond = []
for area in areas:
prompt = area.get("prompt", None)
if prompt:
new_cond = self.get_conditioning(clip=clip, prompt=area["prompt"])
new_cond = area_setter.append(conditioning=new_cond, width=int(area["w"]), height=int(area["h"]), x=int(area["x"]),
y=int(area["y"]), strength=area["s"])[0]
cond += new_cond
return (cond, n_cond,)
def get_conditioning(self, prompt="", clip=None, progress_callback=None):
tokens = clip.tokenize(prompt)
cond, pooled = clip.encode_from_tokens(tokens, return_pooled=True)
return [[cond, {"pooled_output": pooled}]]
class DeforumInpaintModelConditioning:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"positive": ("CONDITIONING", ),
"negative": ("CONDITIONING", ),
"vae": ("VAE", ),},
"optional": {
"pixels": ("IMAGE",),
"mask": ("MASK",),
"latent": ("LATENT",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
}
}
RETURN_TYPES = ("CONDITIONING","CONDITIONING","LATENT")
RETURN_NAMES = ("positive", "negative", "latent")
FUNCTION = "encode"
display_name = "InpaintModelConditioning [safe]"
CATEGORY = "deforum/conditioning"
def encode(self, positive, negative, vae, pixels, mask, latent, deforum_frame_data={}):
reset = deforum_frame_data.get("reset", False)
if (pixels is not None and mask is not None) and not reset:
x = (pixels.shape[1] // 8) * 8
y = (pixels.shape[2] // 8) * 8
mask = torch.nn.functional.interpolate(mask.reshape((-1, 1, mask.shape[-2], mask.shape[-1])), size=(pixels.shape[1], pixels.shape[2]), mode="bilinear")
orig_pixels = pixels
pixels = orig_pixels.clone()
if pixels.shape[1] != x or pixels.shape[2] != y:
x_offset = (pixels.shape[1] % 8) // 2
y_offset = (pixels.shape[2] % 8) // 2
pixels = pixels[:,x_offset:x + x_offset, y_offset:y + y_offset,:]
mask = mask[:,:,x_offset:x + x_offset, y_offset:y + y_offset]
m = (1.0 - mask.round()).squeeze(1)
for i in range(3):
pixels[:,:,:,i] -= 0.5
pixels[:,:,:,i] *= m
pixels[:,:,:,i] += 0.5
concat_latent = vae.encode(pixels)
orig_latent = vae.encode(orig_pixels)
out_latent = {}
out_latent["samples"] = orig_latent
out_latent["noise_mask"] = mask
out = []
for conditioning in [positive, negative]:
c = []
for t in conditioning:
d = t[1].copy()
d["concat_latent_image"] = concat_latent
d["concat_mask"] = mask
n = [t[0], d]
c.append(n)
out.append(c)
return (out[0], out[1], out_latent)
else:
return (positive, negative, latent,)
class DeforumShuffleTokenizer:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"clip": ("CLIP",),
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
}
}
RETURN_TYPES = ("CLIP",)
FUNCTION = "fn"
display_name = "Shuffle Tokenizer"
CATEGORY = "deforum/conditioning"
def fn(self, clip, seed=42):
# Access the tokenizer from the clip object
tokenizer = clip.tokenizer
# Copy the original vocabulary for restoration if needed
original_vocab = tokenizer.vocab.copy()
# Seed the random number generator for reproducibility
seeded_random = random.Random(seed)
# Create a list of (key, value) pairs, shuffle it, then convert it back to a dictionary
items = list(original_vocab.items())
seeded_random.shuffle(items)
shuffled_vocab = dict(items)
# Update the tokenizer's vocabulary with the shuffled version
# This step is highly dependent on the tokenizer's implementation.
# If the tokenizer has a method to set its vocab, use it.
# Otherwise, you might need to directly set the attribute, if possible.
# tokenizer.set_vocab(shuffled_vocab) # Hypothetical method
tokenizer.vocab = shuffled_vocab # Direct attribute setting, if no method available
return (clip,)
================================================
FILE: deforum_nodes/nodes/deforum_data_nodes.py
================================================
from ..modules.deforum_ui_data import (deforum_base_params, deforum_anim_params, deforum_translation_params,
deforum_cadence_params, deforum_depth_params,
deforum_noise_params, deforum_color_coherence_params, deforum_diffusion_schedule_params,
deforum_hybrid_video_params, deforum_hybrid_video_schedules)
from ..modules.deforum_node_base import DeforumDataBase
from ..modules.deforum_comfyui_helpers import get_node_params
class DeforumBaseParamsNode(DeforumDataBase):
params = get_node_params(deforum_base_params)
display_name = "Base Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumAnimParamsNode(DeforumDataBase):
params = get_node_params(deforum_anim_params)
display_name = "Animation Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumTranslationParamsNode(DeforumDataBase):
params = get_node_params(deforum_translation_params)
display_name = "Translate Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumDepthParamsNode(DeforumDataBase):
params = get_node_params(deforum_depth_params)
display_name = "Depth Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumNoiseParamsNode(DeforumDataBase):
params = get_node_params(deforum_noise_params)
display_name = "Noise Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumColorParamsNode(DeforumDataBase):
params = get_node_params(deforum_color_coherence_params)
display_name = "ColorMatch Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumDiffusionParamsNode(DeforumDataBase):
params = get_node_params(deforum_diffusion_schedule_params)
display_name = "Diffusion Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumCadenceParamsNode(DeforumDataBase):
params = get_node_params(deforum_cadence_params)
display_name = "Cadence Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumHybridParamsNode(DeforumDataBase):
params = get_node_params(deforum_hybrid_video_params)
display_name = "Hybrid Parameters"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumHybridScheduleNode(DeforumDataBase):
params = get_node_params(deforum_hybrid_video_schedules)
display_name = "Hybrid Schedule"
def __init__(self):
super().__init__()
@classmethod
def INPUT_TYPES(s):
return s.params
class DeforumFrameDataExtract:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
}
}
RETURN_TYPES = ("INT","INT", "INT", "FLOAT", "STRING", "STRING", "FLOAT", "FLOAT", "BOOLEAN")
RETURN_NAMES = ("frame_idx", "seed", "steps", "cfg_scale", "sampler_name", "scheduler_name", "denoise", "subseed_strength", "first_run")
FUNCTION = "get_data"
display_name = "Frame Data Extract"
CATEGORY = "deforum/parameters"
def get_data(self, deforum_frame_data):
seed = deforum_frame_data.get("seed", 0)
steps = deforum_frame_data.get("steps", 10)
cfg = deforum_frame_data.get("cfg", 7.5)
sampler_name = deforum_frame_data.get("sampler_name", "euler_a")
scheduler = deforum_frame_data.get("scheduler", "normal")
denoise = deforum_frame_data.get("denoise", 1.0)
keys = deforum_frame_data.get("keys")
frame_idx = deforum_frame_data.get("frame_idx")
subseed_str = keys.subseed_strength_schedule_series[frame_idx]
first_run = deforum_frame_data.get("second_run", False)
return (frame_idx, seed, steps, cfg, sampler_name, scheduler, denoise, subseed_str, first_run, )
================================================
FILE: deforum_nodes/nodes/deforum_framewarp_node.py
================================================
import cv2
import numpy as np
import torch
from PIL import Image
from einops import rearrange, repeat
import comfy
from ..modules.deforum_comfyui_helpers import tensor2np, pil2tensor
class DeforumFrameWarpNode:
def __init__(self):
self.depth_model = None
self.depth = None
self.algo = ""
self.depth_min, self.depth_max = 1000, -1000
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
"warp_depth_image": ("BOOLEAN",{"default":False}),
},
"optional":
{
"depth_image":("IMAGE",),
}
}
RETURN_TYPES = ("IMAGE","IMAGE","IMAGE")
RETURN_NAMES = ("IMAGE","DEPTH", "WARPED_DEPTH")
FUNCTION = "fn"
display_name = "Frame Warp"
CATEGORY = "deforum/image"
def fn(self, image, deforum_frame_data, warp_depth_image, depth_image=None):
from deforum.models import DepthModel
from deforum.utils.deforum_framewarp_utils import anim_frame_warp
np_image = None
data = deforum_frame_data
if image is not None:
if image.shape[0] > 1:
for img in image:
np_image = tensor2np(img)
else:
np_image = tensor2np(image)
np_image = cv2.cvtColor(np_image, cv2.COLOR_RGB2BGR)
args = data.get("args")
anim_args = data.get("anim_args")
keys = data.get("keys")
frame_idx = data.get("frame_idx")
if frame_idx == 0:
self.depth = None
predict_depths = (
anim_args.animation_mode == '3D' and anim_args.use_depth_warping) or anim_args.save_depth_maps
predict_depths = predict_depths or (
anim_args.hybrid_composite and anim_args.hybrid_comp_mask_type in ['Depth', 'Video Depth'])
if depth_image is not None:
predict_depths = False
if self.depth_model == None or self.algo != anim_args.depth_algorithm:
self.vram_state = "high"
if self.depth_model is not None:
self.depth_model.to("cpu")
del self.depth_model
# torch_gc()
self.algo = anim_args.depth_algorithm
if predict_depths:
keep_in_vram = True if self.vram_state == 'high' else False
# device = ('cpu' if cmd_opts.lowvram or cmd_opts.medvram else self.root.device)
# TODO Set device in root in webui
device = 'cuda'
self.depth_model = DepthModel("models/other", device,
keep_in_vram=keep_in_vram,
depth_algorithm=anim_args.depth_algorithm, Width=args.width,
Height=args.height,
midas_weight=anim_args.midas_weight)
# depth-based hybrid composite mask requires saved depth maps
if anim_args.hybrid_composite != 'None' and anim_args.hybrid_comp_mask_type == 'Depth':
anim_args.save_depth_maps = True
else:
self.depth_model = None
anim_args.save_depth_maps = False
if self.depth_model != None and not predict_depths:
self.depth_model = None
if self.depth_model is not None:
self.depth_model.to('cuda')
if depth_image is not None:
depth_image = comfy.utils.common_upscale(depth_image.permute(0,3,1,2), args.width, args.height, upscale_method="bislerp", crop="disabled")
depth_image = depth_image.permute(0,2,3,1) * 255.0
if depth_image.dim() > 2:
depth_image = depth_image[0].mean(dim=-1) # Take the mean across the color channels
warped_np_img, depth, mask = anim_frame_warp(np_image, args, anim_args, keys, frame_idx,
depth_model=self.depth_model, depth=depth_image, device='cuda',
half_precision=True)
image = Image.fromarray(cv2.cvtColor(warped_np_img, cv2.COLOR_BGR2RGB))
tensor = pil2tensor(image)
if depth is not None:
num_channels = len(depth.shape)
if num_channels <= 3:
depth_image_pil = self.to_image(depth.detach().cpu())
else:
depth_image_pil = self.to_image(depth[0].detach().cpu())
ret_depth = pil2tensor(depth_image_pil).detach().cpu()
if warp_depth_image:
warped_depth, _, _ = anim_frame_warp(np.array(depth_image_pil), args, anim_args, keys, frame_idx,
depth_model=self.depth_model, depth=depth_image,
device='cuda',
half_precision=True)
warped_depth_image = Image.fromarray(warped_depth)
warped_ret = pil2tensor(warped_depth_image).detach().cpu()
else:
warped_ret = ret_depth
else:
ret_depth = tensor
warped_ret = tensor
self.depth = depth
# if gs.vram_state in ["low", "medium"] and self.depth_model is not None:
# self.depth_model.to('cpu')
# if mask is not None:
# mask = mask.detach().cpu()
# # mask = mask.reshape((-1, 1, mask.shape[-2], mask.shape[-1])).movedim(1, -1).expand(-1, -1, -1, 3)
# mask = mask.mean(dim=0, keepdim=False)
# mask[mask > 1e-05] = 1
# mask[mask < 1e-05] = 0
# mask = mask[0].unsqueeze(0)
# from ai_nodes.ainodes_engine_base_nodes.ainodes_backend.resizeRight import resizeright
# from ai_nodes.ainodes_engine_base_nodes.ainodes_backend.resizeRight import interp_methods
# mask = resizeright.resize(mask, scale_factors=None,
# out_shape=[mask.shape[0], int(mask.shape[1] // 8), int(mask.shape[2] // 8)
# ],
# interp_method=interp_methods.lanczos3, support_sz=None,
# antialiasing=True, by_convs=True, scale_tolerance=None,
# max_numerator=10, pad_mode='reflect')
return (tensor, ret_depth,warped_ret,)
# return [data, tensor, mask, ret_depth, self.depth_model]
else:
return (image, image,image,)
def to_image(self, depth: torch.Tensor):
depth = depth.cpu().numpy()
depth = np.expand_dims(depth, axis=0) if len(depth.shape) == 2 else depth
self.depth_min, self.depth_max = min(self.depth_min, depth.min()), max(self.depth_max, depth.max())
denom = max(1e-8, self.depth_max - self.depth_min)
temp = rearrange((depth - self.depth_min) / denom * 255, 'c h w -> h w c')
return Image.fromarray(repeat(temp, 'h w 1 -> h w c', c=3).astype(np.uint8))
================================================
FILE: deforum_nodes/nodes/deforum_hybrid_nodes.py
================================================
import copy
import cv2
import numpy as np
from PIL import Image
from deforum.generators.deforum_flow_generator import get_flow_from_images
from deforum.models import RAFT
from deforum.utils.image_utils import image_transform_optical_flow
from ..modules.deforum_comfyui_helpers import tensor2np, tensor2pil, pil2tensor
# from ..modules.deforum_constants import deforum_models, deforum_depth_algo
from ..mapping import gs
class DeforumApplyFlowNode:
methods = ['RAFT', 'DIS Medium', 'DIS Fine', 'Farneback']
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"flow_image": ("IMAGE",),
"flow_method": ([cls.methods]),
"flow_factor": ("FLOAT", {"default": 0.8, "min": 0, "max": 1.0}),
},
"optional":{
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
}
}
RETURN_TYPES = (("IMAGE",))
FUNCTION = "apply_flow"
CATEGORY = f"deforum/image"
OUTPUT_NODE = True
display_name = "Apply Flow"
def __init__(self):
self.image_cache = []
def apply_flow(self, image, flow_image, flow_method, flow_factor, deforum_frame_data={}):
# global deforum_models
if "raft_model" not in gs.deforum_models:
gs.deforum_models["raft_model"] = RAFT()
if deforum_frame_data.get("reset", None):
self.image_cache.clear()
if flow_image is not None:
temp_np = tensor2np(flow_image)
else:
temp_np = tensor2np(image)
self.image_cache.append(temp_np)
if len(self.image_cache) >= 2:
flow = get_flow_from_images(self.image_cache[0], self.image_cache[1], flow_method, gs.deforum_models["raft_model"])
img = image_transform_optical_flow(tensor2np(image), flow, flow_factor)
ret = pil2tensor(img)
self.image_cache = [self.image_cache[1]]
return (ret,)
else:
return (image,)
class DeforumHybridMotionNode:
raft_model = None
methods = ['RAFT', 'DIS Medium', 'DIS Fine', 'Farneback']
def __init__(self):
self.prev_image = None
self.flow = None
self.image_size = None
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"hybrid_image": ("IMAGE",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
"hybrid_method": ([s.methods]),
}
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "fn"
display_name = "Hybrid Motion"
CATEGORY = "deforum/image"
def fn(self, image, hybrid_image, deforum_frame_data, hybrid_method):
if self.raft_model is None:
self.raft_model = RAFT()
flow_factor = deforum_frame_data["keys"].hybrid_flow_factor_schedule_series[deforum_frame_data["frame_index"]]
p_img = tensor2pil(image)
size = p_img.size
pil_image = np.array(p_img).astype(np.uint8)
if self.image_size != size:
self.prev_image = None
self.flow = None
self.image_size = size
bgr_image = cv2.cvtColor(pil_image, cv2.COLOR_RGB2BGR)
if hybrid_image is None:
if self.prev_image is None:
self.prev_image = bgr_image
return (image,)
else:
self.flow = get_flow_from_images(self.prev_image, bgr_image, hybrid_method, self.raft_model, self.flow)
self.prev_image = copy.deepcopy(bgr_image)
bgr_image = image_transform_optical_flow(bgr_image, self.flow, flow_factor)
rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
return (pil2tensor(rgb_image),)
else:
pil_image_ref = np.array(tensor2pil(hybrid_image).resize((self.image_size), Image.Resampling.LANCZOS)).astype(np.uint8)
bgr_image_ref = cv2.cvtColor(pil_image_ref, cv2.COLOR_RGB2BGR)
bgr_image_ref = cv2.resize(bgr_image_ref, (bgr_image.shape[1], bgr_image.shape[0]))
if self.prev_image is None:
self.prev_image = bgr_image_ref
return (image,)
else:
self.flow = get_flow_from_images(self.prev_image, bgr_image_ref, hybrid_method, self.raft_model,
self.flow)
bgr_image = image_transform_optical_flow(bgr_image, self.flow, flow_factor)
rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
return (pil2tensor(rgb_image),)
================================================
FILE: deforum_nodes/nodes/deforum_image_nodes.py
================================================
import cv2
from PIL import Image
import numpy as np
from deforum.generators.deforum_noise_generator import add_noise
from deforum.utils.image_utils import maintain_colors, unsharp_mask, compose_mask_with_check
from ..modules.deforum_comfyui_helpers import tensor2pil, tensor2np, pil2tensor
class DeforumColorMatchNode:
def __init__(self):
self.depth_model = None
self.algo = ""
self.color_match_sample = None
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
"force_use_sample": ("BOOLEAN", {"default":False},)
},
"optional":
{"force_sample_image":("IMAGE",)}
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "fn"
display_name = "Color Match"
CATEGORY = "deforum/image"
def fn(self, image, deforum_frame_data, force_use_sample, force_sample_image=None):
if image is not None:
anim_args = deforum_frame_data.get("anim_args")
image = np.array(tensor2pil(image))
# image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
frame_idx = deforum_frame_data.get("frame_idx", 0)
if frame_idx == 0 and not force_use_sample:
self.color_match_sample = None
return (pil2tensor(image),)
if force_use_sample:
if force_sample_image is not None:
self.color_match_sample = np.array(tensor2pil(force_sample_image)).copy()
if anim_args.color_coherence != 'None' and self.color_match_sample is not None:
image = maintain_colors(image, self.color_match_sample, anim_args.color_coherence)
print(f"[deforum] ColorMatch: {anim_args.color_coherence}")
if self.color_match_sample is None:
self.color_match_sample = image.copy()
if anim_args.color_force_grayscale:
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.cvtColor(image, cv2.COLOR_GRAY2BGR)
# image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image = pil2tensor(image)
return (image,)
class DeforumAddNoiseNode:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
}
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "fn"
display_name = "Add Noise"
CATEGORY = "deforum/noise"
def fn(self, image, deforum_frame_data):
if image is not None:
keys = deforum_frame_data.get("keys")
args = deforum_frame_data.get("args")
anim_args = deforum_frame_data.get("anim_args")
root = deforum_frame_data.get("root")
frame_idx = deforum_frame_data.get("frame_idx")
noise = keys.noise_schedule_series[frame_idx]
kernel = int(keys.kernel_schedule_series[frame_idx])
sigma = keys.sigma_schedule_series[frame_idx]
amount = keys.amount_schedule_series[frame_idx]
threshold = keys.threshold_schedule_series[frame_idx]
contrast = keys.contrast_schedule_series[frame_idx]
if anim_args.use_noise_mask and keys.noise_mask_schedule_series[frame_idx] is not None:
noise_mask_seq = keys.noise_mask_schedule_series[frame_idx]
else:
noise_mask_seq = None
mask_vals = {}
noise_mask_vals = {}
mask_vals['everywhere'] = Image.new('1', (args.width, args.height), 1)
noise_mask_vals['everywhere'] = Image.new('1', (args.width, args.height), 1)
# from ainodes_frontend.nodes.deforum_nodes.deforum_framewarp_node import tensor2np
prev_img = tensor2np(image)
mask_image = None
# apply scaling
contrast_image = (prev_img * contrast).round().astype(np.uint8)
# anti-blur
if amount > 0:
contrast_image = unsharp_mask(contrast_image, (kernel, kernel), sigma, amount, threshold,
mask_image if args.use_mask else None)
# apply frame noising
if args.use_mask or anim_args.use_noise_mask:
root.noise_mask = compose_mask_with_check(root, args, noise_mask_seq,
noise_mask_vals,
Image.fromarray(
cv2.cvtColor(contrast_image, cv2.COLOR_BGR2RGB)))
noised_image = add_noise(contrast_image, noise, int(args.seed), anim_args.noise_type,
(anim_args.perlin_w, anim_args.perlin_h,
anim_args.perlin_octaves,
anim_args.perlin_persistence),
root.noise_mask, args.invert_mask)
# image = Image.fromarray(noised_image)
print(f"[deforum] Adding Noise {noise} {anim_args.noise_type}")
image = pil2tensor(noised_image).detach().cpu()
return (image,)
================================================
FILE: deforum_nodes/nodes/deforum_interpolation_nodes.py
================================================
import cv2
import torch
import numpy as np
from deforum import FilmModel
from deforum.models import DepthModel, RAFT
from comfy import model_management
from ..modules.standalone_cadence import CadenceInterpolator
from ..modules.deforum_comfyui_helpers import tensor2pil, pil2tensor
# from ..modules.deforum_constants import deforum_models, deforum_depth_algo
# from .deforum_cache_nodes import deforum_cache
# deforum_models = {}
# deforum_depth_algo = ""
from ..mapping import gs
class DeforumFILMInterpolationNode:
def __init__(self):
self.FILM_temp = []
self.model = None
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"inter_amount": ("INT", {"default": 2, "min": 1, "max": 10000},),
"skip_first": ("BOOLEAN", {"default":True}),
"skip_last": ("BOOLEAN", {"default":False}),
}
}
RETURN_TYPES = ("IMAGE",)
# RETURN_NAMES = ("POSITIVE", "NEGATIVE")
FUNCTION = "fn"
display_name = "FILM Interpolation"
CATEGORY = "deforum/interpolation"
@classmethod
def IS_CHANGED(self, *args, **kwargs):
# Force re-evaluation of the node
return float("NaN")
def interpolate(self, image, inter_frames, skip_first, skip_last):
if self.model is None:
self.model = FilmModel()
self.model.model.cuda()
return_frames = []
pil_image = tensor2pil(image.clone().detach())
np_image = np.array(pil_image.convert("RGB"))
self.FILM_temp.append(np_image)
if len(self.FILM_temp) == 2:
# with torch.inference_mode():
with torch.no_grad():
frames = self.model.inference(self.FILM_temp[0], self.FILM_temp[1], inter_frames=inter_frames)
# skip_first, skip_last = True, False
if skip_first:
frames.pop(0)
if skip_last:
frames.pop(-1)
for frame in frames:
tensor = pil2tensor(frame)[0]
return_frames.append(tensor.detach().cpu())
self.FILM_temp = [self.FILM_temp[1]]
print(f"[deforum] FILM: {len(return_frames)} frames")
if len(return_frames) > 0:
return_frames = torch.stack(return_frames, dim=0)
return return_frames
else:
return image.unsqueeze(0)
def fn(self, image, inter_amount, skip_first, skip_last):
result = []
if image.shape[0] > 1:
for img in image:
interpolated_frames = self.interpolate(img, inter_amount, skip_first, skip_last)
for f in interpolated_frames:
result.append(f)
ret = torch.stack(result, dim=0)
else:
ret = self.interpolate(image[0], inter_amount, skip_first, skip_last)
return (ret,)
class DeforumSimpleInterpolationNode:
def __init__(self):
self.FILM_temp = []
self.model = None
@classmethod
def INPUT_TYPES(s):
return {"required":
{"image": ("IMAGE",),
"method": (["DIS Medium", "DIS Fast", "DIS UltraFast", "Farneback Fine", "Normal"],), # "DenseRLOF", "SF",
"inter_amount": ("INT", {"default": 2, "min": 1, "max": 10000},),
"skip_first": ("BOOLEAN", {"default":False}),
"skip_last": ("BOOLEAN", {"default":False}),
},
"optional":{
"first_image": ("IMAGE",),
"deforum_frame_data": ("DEFORUM_FRAME_DATA",),
}
}
RETURN_TYPES = ("IMAGE", "IMAGE")
RETURN_NAMES = ("IMAGES", "LAST_IMAGE")
FUNCTION = "fn"
display_name = "Simple Interpolation"
CATEGORY = "deforum/interpolation"
@classmethod
def IS_CHANGED(self, *args, **kwargs):
# Force re-evaluation of the node
return float("NaN")
def interpolate(self, image, method, inter_frames, skip_first, skip_last):
return_frames = []
pil_image = tensor2pil(image.clone().detach())
np_image = np.array(pil_image.convert("RGB"))
self.FILM_temp.append(np_image)
print(len(self.FILM_temp))
if len(self.FILM_temp) == 2:
if inter_frames > 1:
if method != "Dyna":
from ..modules.interp import optical_flow_cadence
frames = optical_flow_cadence(self.FILM_temp[0], self.FILM_temp[1], inter_frames + 1, method)
# skip_first, skip_last = True, False
if skip_first:
frames.pop(0)
if skip_last:
frames.pop(-1)
for frame in frames:
tensor = pil2tensor(frame)[0]
gitextract_kwee4pen/
├── .gitattributes
├── .github/
│ └── workflows/
│ └── publish_action.yml
├── .gitignore
├── LICENSE
├── README.md
├── __init__.py
├── deforum_nodes/
│ ├── __init__.py
│ ├── exec_hijack.py
│ ├── mapping.py
│ ├── modules/
│ │ ├── __init__.py
│ │ ├── better_resize/
│ │ │ ├── __init__.py
│ │ │ ├── interp_methods.py
│ │ │ └── resize_right.py
│ │ ├── deforum_comfy_sampler.py
│ │ ├── deforum_comfyui_helpers.py
│ │ ├── deforum_constants.py
│ │ ├── deforum_node_base.py
│ │ ├── deforum_ui_data.py
│ │ ├── interp.py
│ │ └── standalone_cadence.py
│ └── nodes/
│ ├── __init__.py
│ ├── deforum_advnoise_node.py
│ ├── deforum_audiosync_nodes.py
│ ├── deforum_cache_nodes.py
│ ├── deforum_cnet_nodes.py
│ ├── deforum_cond_nodes.py
│ ├── deforum_data_nodes.py
│ ├── deforum_framewarp_node.py
│ ├── deforum_hybrid_nodes.py
│ ├── deforum_image_nodes.py
│ ├── deforum_interpolation_nodes.py
│ ├── deforum_iteration_nodes.py
│ ├── deforum_legacy_nodes.py
│ ├── deforum_logic_nodes.py
│ ├── deforum_noise_nodes.py
│ ├── deforum_prompt_nodes.py
│ ├── deforum_sampler_nodes.py
│ ├── deforum_schedule_visualizer.py
│ ├── deforum_video_nodes.py
│ └── redirect_console_node.py
├── examples/
│ ├── deforum_base.json
│ ├── deforum_cadence.json
│ ├── deforum_integrated.json
│ ├── deforum_ip_adapter.json
│ ├── deforum_simple.json
│ ├── deforum_stablecascade.json
│ └── deforum_stablecascade_legacy.json
├── install.py
├── node_list.json
└── web/
└── js/
└── deforumIterateNode.js
SYMBOL INDEX (413 symbols across 30 files)
FILE: deforum_nodes/exec_hijack.py
function map_node_over_list (line 4) | def map_node_over_list(obj, input_data_all, func, allow_interrupt=False,...
FILE: deforum_nodes/modules/better_resize/interp_methods.py
function set_framework_dependencies (line 17) | def set_framework_dependencies(x):
function support_sz (line 28) | def support_sz(sz):
function cubic (line 36) | def cubic(x):
function lanczos2 (line 47) | def lanczos2(x):
function lanczos3 (line 54) | def lanczos3(x):
function linear (line 61) | def linear(x):
function box (line 68) | def box(x):
FILE: deforum_nodes/modules/better_resize/resize_right.py
class NoneClass (line 8) | class NoneClass:
function resize (line 32) | def resize(input, scale_factors=None, out_shape=None,
function get_projected_grid (line 126) | def get_projected_grid(in_sz, out_sz, scale_factor, fw, by_convs, device...
function get_field_of_view (line 141) | def get_field_of_view(projected_grid, cur_support_sz, fw, eps, device):
function calc_pad_sz (line 153) | def calc_pad_sz(in_sz, out_sz, field_of_view, projected_grid, scale_factor,
function get_weights (line 204) | def get_weights(interp_method, projected_grid, field_of_view):
function apply_weights (line 217) | def apply_weights(input, field_of_view, weights, dim, n_dims, pad_sz, pa...
function apply_convs (line 251) | def apply_convs(input, scale_factor, in_sz, out_sz, weights, dim, pad_sz,
function set_scale_and_out_sz (line 280) | def set_scale_and_out_sz(in_shape, out_shape, scale_factors, by_convs,
function apply_antialiasing_if_needed (line 343) | def apply_antialiasing_if_needed(interp_method, support_sz, scale_factor,
function fw_ceil (line 358) | def fw_ceil(x, fw):
function fw_floor (line 365) | def fw_floor(x, fw):
function fw_cat (line 372) | def fw_cat(x, fw):
function fw_swapaxes (line 379) | def fw_swapaxes(x, ax_1, ax_2, fw):
function fw_pad (line 386) | def fw_pad(x, fw, pad_sz, pad_mode, dim=0):
function fw_conv (line 403) | def fw_conv(input, filter, stride):
function fw_arange (line 417) | def fw_arange(upper_bound, fw, device):
function fw_empty (line 424) | def fw_empty(shape, fw, device):
FILE: deforum_nodes/modules/deforum_comfy_sampler.py
function common_ksampler_with_custom_noise (line 10) | def common_ksampler_with_custom_noise(model, seed, steps, cfg, sampler_n...
function sample_deforum (line 44) | def sample_deforum(
function encode_latent (line 205) | def encode_latent(vae, latent, seed, subseed, subseed_strength, seed_res...
function generate_latent (line 224) | def generate_latent(width, height, seed, subseed, subseed_strength, seed...
function get_conds (line 236) | def get_conds(clip, prompt):
function apply_controlnet (line 247) | def apply_controlnet(conditioning, control_net, image, strength):
function decode_sample (line 264) | def decode_sample(vae, sample):
FILE: deforum_nodes/modules/deforum_comfyui_helpers.py
function pad_to_match (line 23) | def pad_to_match(tensor1, tensor2):
function parse_widget (line 46) | def parse_widget(widget_info: dict) -> tuple:
function get_node_params (line 62) | def get_node_params(input_params):
function get_current_keys (line 70) | def get_current_keys(anim_args, seed, root, parseq_args=None, video_args...
function tensor2pil (line 98) | def tensor2pil(image):
function pil2tensor (line 106) | def pil2tensor(image):
function tensor2np (line 110) | def tensor2np(img):
function get_latent_with_seed (line 114) | def get_latent_with_seed(seed):
function pil_image_to_base64 (line 117) | def pil_image_to_base64(pil_image):
function tensor_to_webp_base64 (line 123) | def tensor_to_webp_base64(tensor):
function generate_seed_list (line 141) | def generate_seed_list(max_frames, mode='fixed', start_seed=0, step=1):
function find_next_index (line 168) | def find_next_index(output_dir, filename_prefix, format):
function pyramid_blend (line 201) | def pyramid_blend(tensor1, tensor2, blend_value):
function gaussian_blend (line 212) | def gaussian_blend(tensor2, tensor1, blend_value):
function sigmoidal_blend (line 218) | def sigmoidal_blend(tensor1, tensor2, blend_value):
function blend_tensors (line 225) | def blend_tensors(obj1, obj2, blend_value, blend_method="linear"):
FILE: deforum_nodes/modules/deforum_constants.py
class DeforumStorage (line 1) | class DeforumStorage:
method __new__ (line 4) | def __new__(cls, *args, **kwargs):
method __init__ (line 14) | def __init__(self):
FILE: deforum_nodes/modules/deforum_node_base.py
class DeforumDataBase (line 1) | class DeforumDataBase:
method IS_CHANGED (line 12) | def IS_CHANGED(cls, text, autorefresh):
method get (line 16) | def get(self, deforum_data=None, *args, **kwargs):
FILE: deforum_nodes/modules/interp.py
function optical_flow_cadence (line 23) | def optical_flow_cadence(i1, i2, cadence, method="DIS Medium"):
function get_matrix_for_hybrid_motion (line 44) | def get_matrix_for_hybrid_motion(frame_idx, dimensions, inputfiles, hybr...
function get_matrix_for_hybrid_motion_prev (line 52) | def get_matrix_for_hybrid_motion_prev(frame_idx, dimensions, inputfiles,...
function get_flow_for_hybrid_motion (line 65) | def get_flow_for_hybrid_motion(frame_idx, dimensions, inputfiles, hybrid...
function get_flow_for_hybrid_motion_prev (line 76) | def get_flow_for_hybrid_motion_prev(frame_idx, dimensions, inputfiles, h...
function get_flow_for_hybrid_motion_prev_imgs (line 91) | def get_flow_for_hybrid_motion_prev_imgs(frame_idx, dimensions, current_...
function image_transform_ransac (line 107) | def image_transform_ransac(image_cv2, xform, hybrid_motion, border_mode=...
function image_transform_optical_flow (line 114) | def image_transform_optical_flow(img, flow, border_mode=cv2.BORDER_REPLI...
function image_transform_affine (line 123) | def image_transform_affine(image_cv2, xform, border_mode=cv2.BORDER_REPL...
function image_transform_perspective (line 132) | def image_transform_perspective(image_cv2, xform, border_mode=cv2.BORDER...
function get_hybrid_motion_default_matrix (line 141) | def get_hybrid_motion_default_matrix(hybrid_motion):
function get_hybrid_motion_default_flow (line 149) | def get_hybrid_motion_default_flow(dimensions):
function get_transformation_matrix_from_images (line 155) | def get_transformation_matrix_from_images(img1, img2, hybrid_motion, max...
function get_flow_from_images (line 186) | def get_flow_from_images(i1, i2, method):
function get_flow_from_images_DIS (line 204) | def get_flow_from_images_DIS(i1, i2, preset):
function get_flow_from_images_Dense_RLOF (line 211) | def get_flow_from_images_Dense_RLOF(i1, i2, last_flow=None):
function get_flow_from_images_SF (line 215) | def get_flow_from_images_SF(i1, i2, last_flow=None, layers=3, averaging_...
function get_flow_from_images_Farneback (line 219) | def get_flow_from_images_Farneback(i1, i2, preset="normal", last_flow=No...
function save_flow_visualization (line 243) | def save_flow_visualization(frame_idx, dimensions, flow, inputfiles, hyb...
function draw_flow_lines_in_grid_in_color (line 255) | def draw_flow_lines_in_grid_in_color(img, flow, step=8, magnitude_multip...
function draw_flow_lines_in_color (line 288) | def draw_flow_lines_in_color(img, flow, threshold=3, magnitude_multiplie...
function autocontrast_grayscale (line 327) | def autocontrast_grayscale(image, low_cutoff=0, high_cutoff=100):
function get_resized_image_from_filename (line 342) | def get_resized_image_from_filename(im, dimensions):
function remap (line 347) | def remap(img, flow, border_mode=cv2.BORDER_REFLECT_101):
function center_crop_image (line 362) | def center_crop_image(img, w, h):
function extend_flow (line 370) | def extend_flow(flow, w, h):
FILE: deforum_nodes/modules/standalone_cadence.py
class CadenceInterpolator (line 16) | class CadenceInterpolator:
method __init__ (line 17) | def __init__(self):
method new_standalone_cadence (line 25) | def new_standalone_cadence(self, args, anim_args, root, keys, frame_id...
FILE: deforum_nodes/nodes/deforum_advnoise_node.py
class AddAdvancedNoiseNode (line 44) | class AddAdvancedNoiseNode:
method IS_CHANGED (line 49) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 54) | def INPUT_TYPES(cls):
method add_advanced_noise (line 216) | def add_advanced_noise(self, images, noise_type, amount, seed=None, **...
function generate_voronoi_noise (line 256) | def generate_voronoi_noise(H, W, num_points=100, seed=None, *args, **kwa...
function generate_simplex_noise (line 291) | def generate_simplex_noise(H, W, scale=0.05, octaves=4, persistence=0.5,...
function generate_wavelet_noise (line 319) | def generate_wavelet_noise(H, W, octaves=3, wavelet='haar', mode='symmet...
function generate_gabor_kernel (line 348) | def generate_gabor_kernel(frequency, theta, sigma_x, sigma_y, n_stds=3, ...
function generate_gabor_noise (line 380) | def generate_gabor_noise(H, W, frequency=0.1, theta=0.0, sigma_x=10.0, s...
function generate_value_noise (line 414) | def generate_value_noise(H, W, res=16, seed=None, **kwargs):
function lerp (line 467) | def lerp(a, b, t):
function fade (line 471) | def fade(t):
function generate_flow_noise_pattern (line 476) | def generate_flow_noise_pattern(H, W, scale=0.1, angle=0.0, seed=None):
function generate_flow_noise (line 511) | def generate_flow_noise(H, W, flow_scale=0.1, flow_angle=0.0, seed=None,...
function generate_turbulence_noise (line 538) | def generate_turbulence_noise(H, W, scale=0.05, octaves=4, persistence=0...
function generate_ridged_multifractal_noise (line 569) | def generate_ridged_multifractal_noise(H, W, scale=0.05, octaves=4, pers...
function generate_reaction_diffusion_noise (line 604) | def generate_reaction_diffusion_noise(H, W, steps=100, Du=0.16, Dv=0.08,...
FILE: deforum_nodes/nodes/deforum_audiosync_nodes.py
class ExtractDominantNoteAmplitude (line 30) | class ExtractDominantNoteAmplitude:
method INPUT_TYPES (line 32) | def INPUT_TYPES(s):
method extract (line 51) | def extract(self, audio_fft, min_frequency, max_frequency, magnitude_t...
class InverseFFTNode (line 83) | class InverseFFTNode:
method INPUT_TYPES (line 85) | def INPUT_TYPES(cls):
method synthesize (line 99) | def synthesize(self, magnitude, phase):
class AmplitudeToAudio (line 113) | class AmplitudeToAudio:
method INPUT_TYPES (line 115) | def INPUT_TYPES(cls):
method convert (line 133) | def convert(self, amplitude, phase, sample_rate=44100):
class AudioData (line 159) | class AudioData:
method __init__ (line 160) | def __init__(self, audio_segment) -> None:
method get_channel_audio_data (line 174) | def get_channel_audio_data(self, channel: int):
method get_channel_fft (line 179) | def get_channel_fft(self, channel: int):
function xor (line 185) | def xor(a, b):
class DeforumAmplitudeToKeyframeSeriesNode (line 271) | class DeforumAmplitudeToKeyframeSeriesNode:
method INPUT_TYPES (line 273) | def INPUT_TYPES(cls):
method safe_eval (line 294) | def safe_eval(self, expr, t, x, max_f, y=None, z=None):
method convert (line 318) | def convert(self, type_name, amplitude, max_frames=500, math="x/100", ...
class DeforumAmplitudeToString (line 345) | class DeforumAmplitudeToString:
method INPUT_TYPES (line 347) | def INPUT_TYPES(s):
method IS_CHANGED (line 361) | def IS_CHANGED(self, *args, **kwargs):
method convert (line 365) | def convert(self, amplitude):
class DerivativeOfAmplitude (line 369) | class DerivativeOfAmplitude:
method INPUT_TYPES (line 371) | def INPUT_TYPES(s):
method derive (line 384) | def derive(self, amplitude,):
class SpectralCentroid (line 388) | class SpectralCentroid:
method INPUT_TYPES (line 390) | def INPUT_TYPES(s):
method calculate (line 403) | def calculate(self, audio_fft,):
class TimeSmoothing (line 409) | class TimeSmoothing:
method INPUT_TYPES (line 411) | def INPUT_TYPES(s):
method smooth (line 428) | def smooth(self, amplitude, window_size,):
class BeatDetection (line 432) | class BeatDetection:
method INPUT_TYPES (line 434) | def INPUT_TYPES(s):
method detect (line 448) | def detect(self, audio):
method find_beat_times (line 455) | def find_beat_times(self, audio):
method extract_envelope (line 468) | def extract_envelope(self, audio_data, sample_rate):
class FrequencyRangeAmplitude (line 475) | class FrequencyRangeAmplitude:
method INPUT_TYPES (line 477) | def INPUT_TYPES(s):
method analyze_frequency_range (line 496) | def analyze_frequency_range(self, audio, frequency_range=(20, 20000), ...
method bandpass_filter (line 513) | def bandpass_filter(self, data, lowcut, highcut, sample_rate, order=5):
method normalize (line 521) | def normalize(self, data, min_value=0, max_value=1, window_size=1, inv...
class BeatDetectionNode (line 526) | class BeatDetectionNode:
method INPUT_TYPES (line 528) | def INPUT_TYPES(s):
method detect_beats (line 543) | def detect_beats(self, audio, sample_rate):
class TempoChangeDetectionNode (line 553) | class TempoChangeDetectionNode:
method INPUT_TYPES (line 555) | def INPUT_TYPES(cls):
method detect_tempo_changes (line 567) | def detect_tempo_changes(self, audio, threshold=0.5, fps=24.0):
class ConvertNormalizedAmplitude (line 597) | class ConvertNormalizedAmplitude:
method INPUT_TYPES (line 599) | def INPUT_TYPES(cls):
method convert_normalized_amplitude (line 610) | def convert_normalized_amplitude(self, normalized_amp):
FILE: deforum_nodes/nodes/deforum_cache_nodes.py
class DeforumCacheLatentNode (line 3) | class DeforumCacheLatentNode:
method IS_CHANGED (line 6) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 12) | def INPUT_TYPES(cls):
method cache_it (line 26) | def cache_it(self, latent=None, cache_index=0):
class DeforumGetCachedLatentNode (line 38) | class DeforumGetCachedLatentNode:
method IS_CHANGED (line 41) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 47) | def INPUT_TYPES(cls):
method get_cached_latent (line 60) | def get_cached_latent(self, cache_index=0):
class DeforumCacheImageNode (line 70) | class DeforumCacheImageNode:
method IS_CHANGED (line 72) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 78) | def INPUT_TYPES(cls):
method cache_it (line 92) | def cache_it(self, image=None, cache_index=0):
class DeforumGetCachedImageNode (line 103) | class DeforumGetCachedImageNode:
method IS_CHANGED (line 106) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 112) | def INPUT_TYPES(cls):
method get_cached_latent (line 126) | def get_cached_latent(self, cache_index=0):
class DeforumCacheStringNode (line 142) | class DeforumCacheStringNode:
method IS_CHANGED (line 144) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 150) | def INPUT_TYPES(cls):
method cache_it (line 164) | def cache_it(self, input_string=None, cache_index=0):
class DeforumGetCachedStringNode (line 175) | class DeforumGetCachedStringNode:
method IS_CHANGED (line 178) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 184) | def INPUT_TYPES(cls):
method get_cached_string (line 197) | def get_cached_string(self, cache_index=0):
FILE: deforum_nodes/nodes/deforum_cnet_nodes.py
class DeforumControlNetApply (line 2) | class DeforumControlNetApply:
method INPUT_TYPES (line 4) | def INPUT_TYPES(s):
method apply_controlnet (line 16) | def apply_controlnet(self, conditioning, control_net, strength, image=...
FILE: deforum_nodes/nodes/deforum_cond_nodes.py
class DeforumConditioningBlendNode (line 7) | class DeforumConditioningBlendNode:
method __init__ (line 8) | def __init__(self):
method INPUT_TYPES (line 14) | def INPUT_TYPES(s):
method fn (line 27) | def fn(self, clip, deforum_frame_data, blend_method):
method get_conditioning (line 61) | def get_conditioning(self, prompt="", clip=None, progress_callback=None):
class DeforumInpaintModelConditioning (line 69) | class DeforumInpaintModelConditioning:
method INPUT_TYPES (line 71) | def INPUT_TYPES(s):
method encode (line 92) | def encode(self, positive, negative, vae, pixels, mask, latent, deforu...
class DeforumShuffleTokenizer (line 135) | class DeforumShuffleTokenizer:
method INPUT_TYPES (line 138) | def INPUT_TYPES(s):
method fn (line 150) | def fn(self, clip, seed=42):
FILE: deforum_nodes/nodes/deforum_data_nodes.py
class DeforumBaseParamsNode (line 8) | class DeforumBaseParamsNode(DeforumDataBase):
method __init__ (line 12) | def __init__(self):
method INPUT_TYPES (line 16) | def INPUT_TYPES(s):
class DeforumAnimParamsNode (line 20) | class DeforumAnimParamsNode(DeforumDataBase):
method __init__ (line 24) | def __init__(self):
method INPUT_TYPES (line 28) | def INPUT_TYPES(s):
class DeforumTranslationParamsNode (line 32) | class DeforumTranslationParamsNode(DeforumDataBase):
method __init__ (line 36) | def __init__(self):
method INPUT_TYPES (line 40) | def INPUT_TYPES(s):
class DeforumDepthParamsNode (line 44) | class DeforumDepthParamsNode(DeforumDataBase):
method __init__ (line 48) | def __init__(self):
method INPUT_TYPES (line 52) | def INPUT_TYPES(s):
class DeforumNoiseParamsNode (line 56) | class DeforumNoiseParamsNode(DeforumDataBase):
method __init__ (line 60) | def __init__(self):
method INPUT_TYPES (line 64) | def INPUT_TYPES(s):
class DeforumColorParamsNode (line 68) | class DeforumColorParamsNode(DeforumDataBase):
method __init__ (line 72) | def __init__(self):
method INPUT_TYPES (line 76) | def INPUT_TYPES(s):
class DeforumDiffusionParamsNode (line 80) | class DeforumDiffusionParamsNode(DeforumDataBase):
method __init__ (line 84) | def __init__(self):
method INPUT_TYPES (line 88) | def INPUT_TYPES(s):
class DeforumCadenceParamsNode (line 92) | class DeforumCadenceParamsNode(DeforumDataBase):
method __init__ (line 96) | def __init__(self):
method INPUT_TYPES (line 100) | def INPUT_TYPES(s):
class DeforumHybridParamsNode (line 103) | class DeforumHybridParamsNode(DeforumDataBase):
method __init__ (line 107) | def __init__(self):
method INPUT_TYPES (line 111) | def INPUT_TYPES(s):
class DeforumHybridScheduleNode (line 114) | class DeforumHybridScheduleNode(DeforumDataBase):
method __init__ (line 118) | def __init__(self):
method INPUT_TYPES (line 122) | def INPUT_TYPES(s):
class DeforumFrameDataExtract (line 125) | class DeforumFrameDataExtract:
method INPUT_TYPES (line 127) | def INPUT_TYPES(s):
method get_data (line 138) | def get_data(self, deforum_frame_data):
FILE: deforum_nodes/nodes/deforum_framewarp_node.py
class DeforumFrameWarpNode (line 12) | class DeforumFrameWarpNode:
method __init__ (line 13) | def __init__(self):
method INPUT_TYPES (line 19) | def INPUT_TYPES(s):
method fn (line 37) | def fn(self, image, deforum_frame_data, warp_depth_image, depth_image=...
method to_image (line 160) | def to_image(self, depth: torch.Tensor):
FILE: deforum_nodes/nodes/deforum_hybrid_nodes.py
class DeforumApplyFlowNode (line 16) | class DeforumApplyFlowNode:
method INPUT_TYPES (line 19) | def INPUT_TYPES(cls):
method __init__ (line 38) | def __init__(self):
method apply_flow (line 41) | def apply_flow(self, image, flow_image, flow_method, flow_factor, defo...
class DeforumHybridMotionNode (line 64) | class DeforumHybridMotionNode:
method __init__ (line 68) | def __init__(self):
method INPUT_TYPES (line 74) | def INPUT_TYPES(s):
method fn (line 88) | def fn(self, image, hybrid_image, deforum_frame_data, hybrid_method):
FILE: deforum_nodes/nodes/deforum_image_nodes.py
class DeforumColorMatchNode (line 8) | class DeforumColorMatchNode:
method __init__ (line 10) | def __init__(self):
method INPUT_TYPES (line 16) | def INPUT_TYPES(s):
method fn (line 33) | def fn(self, image, deforum_frame_data, force_use_sample, force_sample...
class DeforumAddNoiseNode (line 61) | class DeforumAddNoiseNode:
method INPUT_TYPES (line 63) | def INPUT_TYPES(s):
method fn (line 75) | def fn(self, image, deforum_frame_data):
FILE: deforum_nodes/nodes/deforum_interpolation_nodes.py
class DeforumFILMInterpolationNode (line 21) | class DeforumFILMInterpolationNode:
method __init__ (line 22) | def __init__(self):
method INPUT_TYPES (line 26) | def INPUT_TYPES(s):
method IS_CHANGED (line 42) | def IS_CHANGED(self, *args, **kwargs):
method interpolate (line 46) | def interpolate(self, image, inter_frames, skip_first, skip_last):
method fn (line 79) | def fn(self, image, inter_amount, skip_first, skip_last):
class DeforumSimpleInterpolationNode (line 94) | class DeforumSimpleInterpolationNode:
method __init__ (line 95) | def __init__(self):
method INPUT_TYPES (line 99) | def INPUT_TYPES(s):
method IS_CHANGED (line 120) | def IS_CHANGED(self, *args, **kwargs):
method interpolate (line 124) | def interpolate(self, image, method, inter_frames, skip_first, skip_la...
method fn (line 169) | def fn(self, image, method, inter_amount, skip_first, skip_last, first...
class DeforumCadenceNode (line 194) | class DeforumCadenceNode:
method __init__ (line 195) | def __init__(self):
method INPUT_TYPES (line 200) | def INPUT_TYPES(s):
method IS_CHANGED (line 219) | def IS_CHANGED(self, *args, **kwargs):
method interpolate (line 223) | def interpolate(self, image, first_image, deforum_frame_data, depth_st...
method fn (line 348) | def fn(self, image, first_image, deforum_frame_data, depth_strength, p...
FILE: deforum_nodes/nodes/deforum_iteration_nodes.py
class DeforumIteratorNode (line 15) | class DeforumIteratorNode:
method __init__ (line 16) | def __init__(self):
method IS_CHANGED (line 25) | def IS_CHANGED(cls, *args, **kwargs):
method INPUT_TYPES (line 31) | def INPUT_TYPES(cls):
method get (line 59) | def get(self, deforum_data, latent_type, latent=None, init_latent=None...
method get_current_frame (line 316) | def get_current_frame(self, args, anim_args, root, keys, frame_idx, ar...
class DeforumSeedNode (line 337) | class DeforumSeedNode:
method IS_CHANGED (line 339) | def IS_CHANGED(cls, *args, **kwargs):
method INPUT_TYPES (line 342) | def INPUT_TYPES(cls):
method get (line 355) | def get(self, seed, *args, **kwargs):
class DeforumBigBoneResetNode (line 359) | class DeforumBigBoneResetNode:
method IS_CHANGED (line 361) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 366) | def INPUT_TYPES(cls):
method get (line 377) | def get(self, reset_deforum, *args, **kwargs):
FILE: deforum_nodes/nodes/deforum_legacy_nodes.py
class DeforumSingleSampleNode (line 14) | class DeforumSingleSampleNode:
method INPUT_TYPES (line 16) | def INPUT_TYPES(cls):
method get (line 33) | def get(self, deforum_data, model, clip, vae, *args, **kwargs):
class DeforumSetVAEDownscaleRatioNode (line 154) | class DeforumSetVAEDownscaleRatioNode:
method INPUT_TYPES (line 156) | def INPUT_TYPES(s):
method fn (line 167) | def fn(self, vae, downscale_ratio):
FILE: deforum_nodes/nodes/deforum_logic_nodes.py
class DeforumImageSwitcherNode (line 3) | class DeforumImageSwitcherNode:
method INPUT_TYPES (line 5) | def INPUT_TYPES(cls):
method IS_CHANGED (line 20) | def IS_CHANGED(cls, text, autorefresh):
method compare (line 31) | def compare(self, option=True, image_true=None, image_false=None):
class DeforumComparatorNode (line 39) | class DeforumComparatorNode:
method INPUT_TYPES (line 41) | def INPUT_TYPES(cls):
method IS_CHANGED (line 51) | def IS_CHANGED(cls, text, autorefresh):
method compare (line 62) | def compare(self, int_1, int_2, condition):
class DeforumFloatComparatorNode (line 76) | class DeforumFloatComparatorNode:
method INPUT_TYPES (line 78) | def INPUT_TYPES(cls):
method IS_CHANGED (line 88) | def IS_CHANGED(cls, text, autorefresh):
method compare (line 99) | def compare(self, float_1, float_2, condition):
class DeforumAndNode (line 114) | class DeforumAndNode:
method INPUT_TYPES (line 116) | def INPUT_TYPES(cls):
method logical_and (line 130) | def logical_and(self, condition_1, condition_2, *additional_conditions):
class DeforumOrNode (line 133) | class DeforumOrNode:
method INPUT_TYPES (line 135) | def INPUT_TYPES(cls):
method logical_or (line 149) | def logical_or(self, condition_1, condition_2, *additional_conditions):
class DeforumNotNode (line 152) | class DeforumNotNode:
method INPUT_TYPES (line 154) | def INPUT_TYPES(cls):
method logical_not (line 166) | def logical_not(self, condition):
FILE: deforum_nodes/nodes/deforum_noise_nodes.py
class AddCustomNoiseNode (line 7) | class AddCustomNoiseNode:
method IS_CHANGED (line 12) | def IS_CHANGED(cls, text, autorefresh):
method INPUT_TYPES (line 17) | def INPUT_TYPES(cls):
method add_noise (line 141) | def add_noise(self, images, noise_type, amount, seed=None, temperature...
function add_noise_torch (line 148) | def add_noise_torch(images, noise_type='gaussian', seed=None, amount=0.1...
function generate_perlin_noise_2d (line 343) | def generate_perlin_noise_2d(shape, res, seed=None):
function generate_perlin_noise (line 396) | def generate_perlin_noise(B, C, H, W, res_x, res_y, seed=None, *args, **...
function generate_brownian_noise (line 414) | def generate_brownian_noise(shape, scale=1.0, seed=None):
function generate_approx_blue_noise (line 442) | def generate_approx_blue_noise(shape, seed=None, min_dist=1.0, sample_fr...
function generate_fractal_noise (line 481) | def generate_fractal_noise(B, C, H, W, res, octaves=5, persistence=0.5, ...
function generate_cellular_noise (line 514) | def generate_cellular_noise(H, W, num_points=100, *args, **kwargs):
FILE: deforum_nodes/nodes/deforum_prompt_nodes.py
class DeforumPromptNode (line 6) | class DeforumPromptNode(DeforumDataBase):
method __init__ (line 7) | def __init__(self):
method INPUT_TYPES (line 11) | def INPUT_TYPES(cls):
method get (line 28) | def get(self, prompts, deforum_data=None):
class DeforumAreaPromptNode (line 55) | class DeforumAreaPromptNode(DeforumDataBase):
method __init__ (line 60) | def __init__(self):
method INPUT_TYPES (line 64) | def INPUT_TYPES(cls):
method get (line 88) | def get(self, keyframe, mode, prompt, width, height, x, y, strength, d...
class DeforumUnformattedPromptNode (line 118) | class DeforumUnformattedPromptNode(DeforumDataBase):
method __init__ (line 119) | def __init__(self):
method INPUT_TYPES (line 123) | def INPUT_TYPES(cls):
method get (line 141) | def get(self, unformatted_prompts, keyframe_interval, deforum_data=None):
class DeforumAppendNode (line 160) | class DeforumAppendNode(DeforumDataBase):
method __init__ (line 161) | def __init__(self):
method INPUT_TYPES (line 165) | def INPUT_TYPES(cls):
method get (line 185) | def get(self, append_text, keyframe_interval, deforum_data=None, appen...
FILE: deforum_nodes/nodes/deforum_sampler_nodes.py
class DeforumKSampler (line 1) | class DeforumKSampler:
method INPUT_TYPES (line 3) | def INPUT_TYPES(s):
method sample (line 19) | def sample(self, model, latent, positive, negative, deforum_frame_data):
class DeforumVAEEncode (line 33) | class DeforumVAEEncode:
method INPUT_TYPES (line 35) | def INPUT_TYPES(s):
method encode (line 51) | def encode(self, vae, pixels, latent):
FILE: deforum_nodes/nodes/deforum_schedule_visualizer.py
function generate_complex_random_expression (line 178) | def generate_complex_random_expression(max_frames, seed=None, max_parts=3):
class DeforumScheduleTemplate (line 230) | class DeforumScheduleTemplate:
method INPUT_TYPES (line 232) | def INPUT_TYPES(cls):
method show (line 253) | def show(self, expression):
class DeforumAudioScheduleTemplate (line 256) | class DeforumAudioScheduleTemplate:
method INPUT_TYPES (line 258) | def INPUT_TYPES(cls):
method show (line 279) | def show(self, expression):
class DeforumScheduleTemplateRandomizer (line 283) | class DeforumScheduleTemplateRandomizer:
method INPUT_TYPES (line 285) | def INPUT_TYPES(cls):
method IS_CHANGED (line 297) | def IS_CHANGED(cls, text, autorefresh):
method show (line 308) | def show(self, seed, max_frames, max_parts):
class DeforumScheduleVisualizer (line 312) | class DeforumScheduleVisualizer:
method INPUT_TYPES (line 315) | def INPUT_TYPES(cls):
method show (line 338) | def show(self, schedule, max_frames, grid):
FILE: deforum_nodes/nodes/deforum_video_nodes.py
function save_to_file (line 27) | def save_to_file(data, filepath: str):
class DeforumLoadVideo (line 36) | class DeforumLoadVideo:
method __init__ (line 38) | def __init__(self):
method INPUT_TYPES (line 42) | def INPUT_TYPES(s):
method __init__ (line 66) | def __init__(self):
method load_video_frame (line 100) | def load_video_frame(self, video, reset, iterative, start_frame, retur...
method IS_CHANGED (line 147) | def IS_CHANGED(cls, text, autorefresh):
method VALIDATE_INPUTS (line 153) | def VALIDATE_INPUTS(cls, video):
function serve_temp_file (line 165) | async def serve_temp_file(request):
class DeforumVideoSaveNode (line 183) | class DeforumVideoSaveNode:
method __init__ (line 184) | def __init__(self):
method clear_cache_directory (line 193) | def clear_cache_directory(self):
method INPUT_TYPES (line 206) | def INPUT_TYPES(s):
method add_image (line 242) | def add_image(self, image):
method fn (line 254) | def fn(self,
method encode_audio_base64 (line 368) | def encode_audio_base64(self, audio_data, frame_count, fps, start_frame):
method save_video (line 409) | def save_video(self, full_output_folder, filename, counter, fps, audio...
method IS_CHANGED (line 445) | def IS_CHANGED(s, text, autorefresh):
function encode_audio_base64 (line 451) | def encode_audio_base64(audio_data, frame_count, fps):
function save_to_file (line 485) | def save_to_file(data, filepath: str):
FILE: deforum_nodes/nodes/redirect_console_node.py
class StreamToWebSocket (line 7) | class StreamToWebSocket:
method __init__ (line 8) | def __init__(self, original_stream, server, stream_type='stdout'):
method write (line 13) | def write(self, message):
method flush (line 24) | def flush(self):
method __getattr__ (line 27) | def __getattr__(self, attr):
class DeforumRedirectConsole (line 32) | class DeforumRedirectConsole:
method INPUT_TYPES (line 35) | def INPUT_TYPES(s):
method fn (line 47) | def fn(self, redirect_console):
FILE: install.py
function handle_stream (line 25) | def handle_stream(stream, is_stdout):
function process_wrap (line 35) | def process_wrap(cmd_str, cwd=None, handler=None):
function get_installed_packages (line 58) | def get_installed_packages():
function is_installed (line 72) | def is_installed(name):
function is_requirements_installed (line 84) | def is_requirements_installed(file_path):
function find_path (line 94) | def find_path(name: str, path: str = None) -> str:
function run_git_command (line 120) | def run_git_command(command: str, working_dir: str) -> None:
function clone_or_pull_repo (line 131) | def clone_or_pull_repo(repo_url: str, repo_dir: str) -> None:
function install (line 141) | def install():
function install_packages (line 173) | def install_packages():
function get_cuda_version (line 182) | def get_cuda_version():
function get_torch_version (line 192) | def get_torch_version():
function construct_wheel_name (line 200) | def construct_wheel_name(cuda_version, py_version, os_name):
function install_stable_fast (line 210) | def install_stable_fast():
function install_reqs (line 219) | def install_reqs():
FILE: web/js/deforumIterateNode.js
function chainCallback (line 31) | function chainCallback(object, property, callback) {
function uploadFile (line 49) | async function uploadFile(file) {
function fitHeight (line 78) | function fitHeight(node) {
function addVideoPreview (line 82) | function addVideoPreview(nodeType) {
function addUploadWidget (line 229) | function addUploadWidget(nodeType, nodeData, widgetName, type="video") {
function extendNodePrototypeWithFrameCaching (line 305) | function extendNodePrototypeWithFrameCaching(nodeType) {
method init (line 339) | init() {
method beforeRegisterNodeDef (line 347) | beforeRegisterNodeDef(nodeType, nodeData) {
class FloatingConsole (line 561) | class FloatingConsole {
method constructor (line 562) | constructor() {
method setupStyles (line 582) | setupStyles() {
method adjustContentContainerSize (line 606) | adjustContentContainerSize() {
method createTitleBar (line 615) | createTitleBar() {
method createContentContainer (line 629) | createContentContainer() {
method addEventListeners (line 634) | addEventListeners() {
method addMenuButton (line 674) | addMenuButton() {
method show (line 692) | show() {
method hide (line 696) | hide() {
method isVisible (line 700) | isVisible() {
method log (line 704) | log(message) {
method clear (line 711) | clear() {
method init (line 722) | init() {
Condensed preview — 50 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (625K chars).
[
{
"path": ".gitattributes",
"chars": 66,
"preview": "# Auto detect text files and perform LF normalization\n* text=auto\n"
},
{
"path": ".github/workflows/publish_action.yml",
"chars": 565,
"preview": "name: Publish to Comfy registry\non:\n workflow_dispatch:\n push:\n branches:\n - main\n paths:\n - \"pyprojec"
},
{
"path": ".gitignore",
"chars": 2763,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "LICENSE",
"chars": 1069,
"preview": "MIT License\n\nCopyright (c) 2024 Deforum LLC.\n\nPermission is hereby granted, free of charge, to any person obtaining a co"
},
{
"path": "README.md",
"chars": 2374,
"preview": "# Deforum for ComfyUI\n\nDeforum integration for ComfyUI.\n\n## Installation\n\nTo get started with Deforum Comfy Nodes, pleas"
},
{
"path": "__init__.py",
"chars": 188,
"preview": "from .deforum_nodes.mapping import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS\nWEB_DIRECTORY = \"./web\"\n\n__all__ = ['"
},
{
"path": "deforum_nodes/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deforum_nodes/exec_hijack.py",
"chars": 2773,
"preview": "import execution, nodes\norig_exec = execution.map_node_over_list\n\ndef map_node_over_list(obj, input_data_all, func, allo"
},
{
"path": "deforum_nodes/mapping.py",
"chars": 1962,
"preview": "import importlib\nimport inspect\nimport sys\n\"\"\"\nDEFORUM STORAGE IMPORT\n\"\"\"\nfrom .modules.deforum_constants import Deforum"
},
{
"path": "deforum_nodes/modules/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deforum_nodes/modules/better_resize/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deforum_nodes/modules/better_resize/interp_methods.py",
"chars": 1715,
"preview": "from math import pi\n\ntry:\n import torch\nexcept ImportError:\n torch = None\n\ntry:\n import numpy\nexcept ImportErro"
},
{
"path": "deforum_nodes/modules/better_resize/resize_right.py",
"chars": 19310,
"preview": "from typing import Tuple\nimport warnings\nfrom math import ceil\nfrom . import interp_methods\nfrom fractions import Fracti"
},
{
"path": "deforum_nodes/modules/deforum_comfy_sampler.py",
"chars": 10596,
"preview": "import secrets\nimport torch\nimport numpy as np\nfrom PIL import Image\n\nfrom deforum import ImageRNGNoise\nfrom deforum.uti"
},
{
"path": "deforum_nodes/modules/deforum_comfyui_helpers.py",
"chars": 10221,
"preview": "import base64\nimport math\nimport os\nimport random\nimport re\nfrom io import BytesIO\n\nimport cv2\nimport numexpr\nimport num"
},
{
"path": "deforum_nodes/modules/deforum_constants.py",
"chars": 714,
"preview": "class DeforumStorage:\n _instance = None # Class attribute that holds the singleton instance\n\n def __new__(cls, *a"
},
{
"path": "deforum_nodes/modules/deforum_node_base.py",
"chars": 585,
"preview": "class DeforumDataBase:\n\n # @classmethod\n # def INPUT_TYPES(s):\n # return s.params\n\n RETURN_TYPES = ((\"de"
},
{
"path": "deforum_nodes/modules/deforum_ui_data.py",
"chars": 17629,
"preview": "import comfy\ndeforum_base_params = {\n \"width\": {\n \"type\": \"spinbox\",\n \"default\": 512,\n \"min\": 64"
},
{
"path": "deforum_nodes/modules/interp.py",
"chars": 16986,
"preview": "import os\nimport random\nimport cv2\nimport numpy as np\n\n\n# def optical_flow_cadence(i1, i2, cadence, method=\"DIS Medium\")"
},
{
"path": "deforum_nodes/modules/standalone_cadence.py",
"chars": 15057,
"preview": "import os\n\nimport cv2\nimport numpy as np\nimport torch\n\nfrom deforum.generators.deforum_flow_generator import rel_flow_to"
},
{
"path": "deforum_nodes/nodes/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "deforum_nodes/nodes/deforum_advnoise_node.py",
"chars": 26449,
"preview": "import secrets\n\nimport torch\nimport numpy as np\nimport math\n\nimport pywt\nimport torch.nn.functional as F\nfrom opensimple"
},
{
"path": "deforum_nodes/nodes/deforum_audiosync_nodes.py",
"chars": 24388,
"preview": "import math\n\nimport pandas as pd\nimport scipy.signal\nimport scipy.ndimage\n\nimport numpy as np\nfrom pydub import AudioSeg"
},
{
"path": "deforum_nodes/nodes/deforum_cache_nodes.py",
"chars": 5164,
"preview": "\n\nclass DeforumCacheLatentNode:\n\n @classmethod\n def IS_CHANGED(cls, text, autorefresh):\n # Force re-evaluat"
},
{
"path": "deforum_nodes/nodes/deforum_cnet_nodes.py",
"chars": 1151,
"preview": "\nclass DeforumControlNetApply:\n @classmethod\n def INPUT_TYPES(s):\n return {\"required\": {\"conditioning\": (\"C"
},
{
"path": "deforum_nodes/nodes/deforum_cond_nodes.py",
"chars": 6680,
"preview": "import random\n\nimport torch\n\nfrom ..modules.deforum_comfyui_helpers import blend_tensors, blend_methods\n\nclass DeforumCo"
},
{
"path": "deforum_nodes/nodes/deforum_data_nodes.py",
"chars": 4458,
"preview": "from ..modules.deforum_ui_data import (deforum_base_params, deforum_anim_params, deforum_translation_params,\n "
},
{
"path": "deforum_nodes/nodes/deforum_framewarp_node.py",
"chars": 7725,
"preview": "\nimport cv2\nimport numpy as np\nimport torch\nfrom PIL import Image\nfrom einops import rearrange, repeat\n\nimport comfy\nfro"
},
{
"path": "deforum_nodes/nodes/deforum_hybrid_nodes.py",
"chars": 4690,
"preview": "import copy\n\nimport cv2\nimport numpy as np\nfrom PIL import Image\nfrom deforum.generators.deforum_flow_generator import g"
},
{
"path": "deforum_nodes/nodes/deforum_image_nodes.py",
"chars": 5371,
"preview": "import cv2\nfrom PIL import Image\nimport numpy as np\nfrom deforum.generators.deforum_noise_generator import add_noise\nfro"
},
{
"path": "deforum_nodes/nodes/deforum_interpolation_nodes.py",
"chars": 16695,
"preview": "import cv2\nimport torch\nimport numpy as np\n\nfrom deforum import FilmModel\nfrom deforum.models import DepthModel, RAFT\n\nf"
},
{
"path": "deforum_nodes/nodes/deforum_iteration_nodes.py",
"chars": 16144,
"preview": "import math\nimport secrets\nfrom types import SimpleNamespace\nimport torch\nfrom deforum import ImageRNGNoise\nfrom deforum"
},
{
"path": "deforum_nodes/nodes/deforum_legacy_nodes.py",
"chars": 6106,
"preview": "import os\nimport time\n\nfrom types import SimpleNamespace\nimport torch\nimport numpy as np\n\nfrom deforum import DeforumAni"
},
{
"path": "deforum_nodes/nodes/deforum_logic_nodes.py",
"chars": 4730,
"preview": "\n\nclass DeforumImageSwitcherNode:\n @classmethod\n def INPUT_TYPES(cls):\n return {\n \"required\": {\n"
},
{
"path": "deforum_nodes/nodes/deforum_noise_nodes.py",
"chars": 21301,
"preview": "import secrets\n\nimport torch\nimport numpy as np\n\n\nclass AddCustomNoiseNode:\n \"\"\"\n A node to add various types of n"
},
{
"path": "deforum_nodes/nodes/deforum_prompt_nodes.py",
"chars": 8235,
"preview": "import torch\n\nfrom nodes import MAX_RESOLUTION\nfrom ..modules.deforum_node_base import DeforumDataBase\n\nclass DeforumPro"
},
{
"path": "deforum_nodes/nodes/deforum_sampler_nodes.py",
"chars": 1830,
"preview": "class DeforumKSampler:\n @classmethod\n def INPUT_TYPES(s):\n return {\"required\":\n {\"model\""
},
{
"path": "deforum_nodes/nodes/deforum_schedule_visualizer.py",
"chars": 19333,
"preview": "import random\n\nfrom matplotlib.figure import Figure\nfrom matplotlib.backends.backend_agg import FigureCanvasAgg as Figur"
},
{
"path": "deforum_nodes/nodes/deforum_video_nodes.py",
"chars": 20610,
"preview": "import base64\nimport gc\nimport os\nimport shutil\nfrom io import BytesIO\nfrom aiohttp import web\nimport hashlib\n\nimport se"
},
{
"path": "deforum_nodes/nodes/redirect_console_node.py",
"chars": 2162,
"preview": "import sys\nimport asyncio\n\nconsole_redirected = None\nstdout_backup = sys.stdout\nstderr_backup = sys.stderr\nclass StreamT"
},
{
"path": "examples/deforum_base.json",
"chars": 28091,
"preview": "{\n \"last_node_id\": 186,\n \"last_link_id\": 579,\n \"nodes\": [\n {\n \"id\": 141,\n \"type\": \"Reroute\",\n \"pos\""
},
{
"path": "examples/deforum_cadence.json",
"chars": 31592,
"preview": "{\n \"last_node_id\": 191,\n \"last_link_id\": 595,\n \"nodes\": [\n {\n \"id\": 141,\n \"type\": \"Reroute\",\n \"pos\""
},
{
"path": "examples/deforum_integrated.json",
"chars": 5415,
"preview": "{\n \"last_node_id\": 6,\n \"last_link_id\": 7,\n \"nodes\": [\n {\n \"id\": 5,\n \"type\": \"DeforumPromptNode\",\n \""
},
{
"path": "examples/deforum_ip_adapter.json",
"chars": 58883,
"preview": "{\n \"last_node_id\": 176,\n \"last_link_id\": 558,\n \"nodes\": [\n {\n \"id\": 141,\n \"type\": \"Reroute\",\n \"pos\""
},
{
"path": "examples/deforum_simple.json",
"chars": 14898,
"preview": "{\n \"last_node_id\": 26,\n \"last_link_id\": 56,\n \"nodes\": [\n {\n \"id\": 7,\n \"type\": \"PreviewImage\",\n \"pos"
},
{
"path": "examples/deforum_stablecascade.json",
"chars": 31358,
"preview": "{\n \"last_node_id\": 198,\n \"last_link_id\": 598,\n \"nodes\": [\n {\n \"id\": 141,\n \"type\": \"Reroute\",\n \"pos\""
},
{
"path": "examples/deforum_stablecascade_legacy.json",
"chars": 62159,
"preview": "{\n \"last_node_id\": 195,\n \"last_link_id\": 591,\n \"nodes\": [\n {\n \"id\": 141,\n \"type\": \"Reroute\",\n \"pos\""
},
{
"path": "install.py",
"chars": 8843,
"preview": "import os\nimport shutil\nimport sys\nimport subprocess\nimport threading\nimport locale\nimport traceback\nimport re\nimport os"
},
{
"path": "node_list.json",
"chars": 999,
"preview": "{\n \"DeforumBaseParamsNode\": \"\",\n \"DeforumAnimParamsNode\": \"\",\n \"DeforumTranslationParamsNode\": \"\",\n \"Deforum"
},
{
"path": "web/js/deforumIterateNode.js",
"chars": 28054,
"preview": "import { app } from \"../../../scripts/app.js\";\nimport { api } from '../../../scripts/api.js'\nimport { ComfyWidgets } fro"
}
]
About this extraction
This page contains the full source code of the XmYx/deforum-comfy-nodes GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 50 files (564.5 KB), approximately 146.9k tokens, and a symbol index with 413 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.