Here is my workflow for someone to test, rebuild, improve and optimize!
It was inspired by:
downpink's workflow: https://civitai.com/models/559596
ipiv's image morphing: https://civitai.com/models/372584/
and Purz image masking: www.purz.xyz
TL;DR
Please build your own workflow using this as a basis, so that you can see the influence of each part.
You can start with a basic AnimateDiff workflow > Add the IPAdapter section starting with the video > add extra IPAdapters and test the influence > Add the ControlNets 1 by 1 to see their influence.
Many parts of this workflow are optional and sometimes not needed for instance, scaling input images/video.
Enjoy, have fun and poke around, never know what you may find.
ComfyUI
is a powerful, node-based interface for AI image generation that enables complex workflows for various tasks. In this tutorial, we'll explore an advanced video-to-video restyling workflow that combines multiple AI technologies, including ControlNet, Latent Consistency Models (LCM), and AnimateDiff. This process transforms an input video using reference images and multiple AI models to create a visually striking output that maintains the essence of the original while incorporating new artistic elements. Whether you're an AI enthusiast, visual artist, or content creator, this guide will help you harness ComfyUI's capabilities to produce captivating, AI-enhanced videos that push the boundaries of creative expression.
Understanding Workflow Components
To successfully implement this video restyling workflow, it's crucial to understand the key components:
a) Input Video: The original video you want to restyle. This serves as the base content for the transformation.
b) Reference Images: Two images that guide the style or composition of the output. These images influence the artistic direction of the restyled video.
c) ControlNet: A neural network architecture that allows precise control over the generation process. In this workflow, we use 2-3 ControlNet instances to guide various aspects of the video transformation, such as preserving structure, adding specific details, or controlling motion.
d) Latent Consistency Model (LCM): An AI model that enables faster and more efficient image generation. LCM helps maintain consistency between frames and speeds up the overall process.
e) AnimateDiff: A technology that enhances the smoothness and coherence of animations in AI-generated videos. It helps create fluid transitions between frames.
f) Upscaling: The process of increasing the resolution and enhancing the details of the generated frames, resulting in a higher-quality output video.
Tips for the process
Optimizing ControlNet Usage:
Start with subtle ControlNet influences and gradually increase strength as needed.
Use different ControlNet models for specific tasks (e.g., one for structure, another for details).
Experiment with the balance between different ControlNet inputs for optimal results.
Balancing Style Transfer and Original Content:
Adjust the strength of style transfer to maintain a good balance with the original video content.
Use preview frames to test settings before processing the entire video.
Performance Optimization:
Use a powerful GPU for faster processing.
Break long videos into smaller segments for easier management and faster iteration.
Utilize LCM to speed up the generation process without sacrificing quality.
Maintaining Temporal Consistency:
Leverage AnimateDiff to ensure smooth transitions between frames.
Use consistent settings across the entire video to avoid jarring changes.