Introduction
This workflow makes use of AnimateDiff, IPAdapter and Controlnet to make clouds dance. The key innovation in this workflow is that I VAE encode the depth map and adjust the denoise such that the depth map gets denoised into clouds. This avoids the video morphing every 16 frames due to the context length of AnimateDiff.
Models
SD1.5 LCM Checkpoint. I use Dreamshaper 8 LCM: https://civitai.com/models/4384?modelVersionId=252914
AnimateLCM_sd15_t2v.ckpt: Download from https://huggingface.co/wangfuyun/AnimateLCM/tree/main and put in models/animatediff_models
depth_anything_v2_vitl.pth and control_v11f1p_sd15_depth_fp16.safetensors. Download using Manager.
IPAdapter PLUS: download ip-adapter-plus_sd15.safetensors using ComfyUI manager. Also download ViT-H and ViT-G clip_vision models using the ComfyUI manager.
Custom Nodes
Install with Manager:
ComfyUI-VideoHelperSuite
KJNodes for ComfyUI
ComfyUI's ControlNet Auxiliary Preprocessors
AnimateDiff Evolved
ComfyUI_IPAdapter_plus
ComfyUI Frame Interpolation