AnimateLCM Image to Video Workflow (I2V)

Introduction

This is a demonstration of what is possible with AnimateLCM Image to Video. After the AnimateLCM I2V pass for the initial video generation, I am doing a 2nd pass using AnimateLCM T2V with Cseti's general motion lora to clean up the details and get consistency across 32 frames. IPAdapter is used to keep the genration more faithful to the original image.

Please note that this workflow requires you to tweak the settings for each input image to get a decent video output. Also, the output video does not animate as much as the image to video services you get online. The video is limited to about 2 seconds at 30FPS.

CogVideoX (for 16GB VRAM)

Only use this workflow if you do not have the VRAM to run CogVideoX. If you have the VRAM (16GB), please install kijai's CogVideoX wrapper node (https://github.com/kijai/ComfyUI-CogVideoXWrapper/) and load the I2V workflow in examples folder. To lower VRAM usage: enable fp8 transformer in the (Down)load CogVideo Model node and enable_vae_tiling in the CogVideo Decode node.

Nodes

Red: Requires user to download/or select a model

Blue: Requires user input to change settings or provide an input image

Brown: Notes for the workflow, please read carefully while using the workflow

Custom nodes (install with manager)

• ComfyUI Frame Interpolation

• ComfyUI_IPAdapter_plus

• AnimateDiff Evolved

• ComfyUI-VideoHelperSuite

• ComfyUI Essentials

• KJNodes for ComfyUI

• ReActor Node for ComfyUI (optional)

Models Needed

• Cseti's general motion lora (models/animatediff_motion_lora): https://huggingface.co/Cseti/General_motionLoRA_32fr_sd15_ad2_v1

• https://huggingface.co/wangfuyun/AnimateLCM

  • AnimateLCM_sd15_t2v.ckpt (models\animatediff_models)

  • AnimateLCM_sd15_t2v_lora.safetensors (models\loras)

• https://huggingface.co/wangfuyun/AnimateLCM-I2V

  • AnimateLCM_sd15_i2v.ckpt (models\animatediff_models)

  • AnimateLCM_sd15_i2v_lora.safetensors (models\loras)

• https://huggingface.co/guoyww/animatediff/tree/cd71ae134a27ec6008b968d6419952b0c0494cf2

  • v3_sd15_adapter.ckpt (models\loras)

• Download with Manager

  • ip-adapter-faceid-plusv2_sd15_lora.safetensors

  • ip-adapter-plus_sd15.safetensors

  • CLIPVision model (IP-Adapter) CLIP-ViT-H-14-laion2B-s32B-b79K