Make Clouds Dance!

Introduction

This workflow makes use of AnimateDiff, IPAdapter and Controlnet to make clouds dance. The key innovation in this workflow is that I VAE encode the depth map and adjust the denoise such that the depth map gets denoised into clouds. This avoids the video morphing every 16 frames due to the context length of AnimateDiff.

Models

1. SD1.5 LCM Checkpoint. I use Dreamshaper 8 LCM: https://civitai.com/models/4384?modelVersionId=252914

2. AnimateLCM_sd15_t2v.ckpt: Download from https://huggingface.co/wangfuyun/AnimateLCM/tree/main and put in models/animatediff_models

3. depth_anything_v2_vitl.pth and control_v11f1p_sd15_depth_fp16.safetensors. Download using Manager.

4. IPAdapter PLUS: download ip-adapter-plus_sd15.safetensors using ComfyUI manager. Also download ViT-H and ViT-G clip_vision models using the ComfyUI manager.

Custom Nodes

Install with Manager:

1. ComfyUI-VideoHelperSuite

2. KJNodes for ComfyUI

3. ComfyUI's ControlNet Auxiliary Preprocessors

4. AnimateDiff Evolved

5. ComfyUI_IPAdapter_plus

6. ComfyUI Frame Interpolation