Sign In

DreamyDiff, SD 1.5 LCM AnimateDiff VID2VID and TXT2VID

DreamyDiff, SD 1.5 LCM AnimateDiff VID2VID and TXT2VID


INTRO: This is a relatively simple workflow that provides AnimateDiff animation frame generation via VID2VID, TXT2VID & IPAdapter nodes, with added SEGS Detailer. My goal here is to create an incredibly compact yet robust, all-in-one personal template for the kind of animations I often enjoy creating using AnimateDiff. I often remote into my PC using a smaller laptop screen so this was built to fit everything into one screen (given a 16:9 resolution.) Efficiency Nodes help a lot with this endeavor, alongside the other incredible nodes, a huge thank you goes out to the authors/devs working hard on this all the time. This workflow assumes you know how to Install Missing Nodes using ComfyUI Manager, as well as having the ability to Install Models into correct directories.


Motion Module & Lora Downloads:

--V2.2 UPDATE:

New Tiled IPAdapter swapped in as part of the v2 Update for IPAdapter. Added Latent Switch node to simplify swapping from VID2VID/TXT2VID flows but then removed it because I couldn't get the boolean toggle to allow either one through? Is this real life?

The workflow is setup with great LCM sampler settings for speed as shared from guides by Inner-Reflections here:

Having said that, you can change to other samplers and run different motion modules for AnimateDiff, it's fun to experiment.

2x ControlNets here are DW Pose and Zoe Depth-Anything in VID2VID use, my personal favorites for the kind of video input I commonly use but feel free to change to yours for different results. This will work really well for dance videos and accurately tracking people turning around, spinning, jumping, ideal for action sports and fight scenes in my opinion. For best results, be careful of negative prompts that will conflict with what your video is trying to achieve.

1x Tiled IPAdapter this part uses most basic and simplified Tiled IPAdapter group and it is working out great, the new update is incredible. Thank you Matteo!

The Detailer group is pretty great for the most part, it makes for clearer details and sharpness in characters and faces, especially eyes. Some models do not require this kind of interference so feel free to test with/without for realistic or anime faces.

Interpolation node is here for smoothing/filling in skipped nth frames from Video In but can also be applied to TXT2VID animations, you can bypass this Node and match the frame rate of source video for "Video In" otherwise.

The Fast Groups RGTHREE Bypasser Node makes it easy to toggle Groups On/Off, necessary in order to easily swap between VID2VID and TXT2VID features.



Endless AnimateDiff loops here are also made with this workflow.

Feel free to comment with anything you've made with this workflow and have a nice day!