This workflow uses SDXL Lightning generated images as reference for SD 1.5 AnimateDiff LCM video generations, using SparseCtrl + IPAdapter to guide the video generation.
By combining AnimateLCM + SDXL Lightning we are able to generate txt2img -> img2vid animations lightning fast.
I suggest to disable/bypass upscaling process until you find a preview you are happy with, same with reference image. (or disable SDXL reference group entirely and load your own image)
You can also experiment with bypassing either SparseCTRL or IPAdapter. (One or both might result in better output, depending on reference image and animation)
Models used in this workflow:
1. AnimateLCM Lora & Motion module
2. Dreamshaper SDXL Lightning (or any SDXL checkpoint combined with Lightning Lora)
3. Photon (or any SD1.5 checkpoint)
4. SparseCtrl rgb (available in comfy manager)
5. IP-Adapter (available in comfy manager)
6. Clip vision for IP-Adapter (available in comfy manager - CLIP-ViT-H-14-laion2B-s32B-b79K)