Sign In
home
models
images
videos
posts
articles
bounties
tools
challenges
events
shop

Video Style Transfer (AnimateLCM, IPAdapter, Controlnet)

43
622
12
Updated: Aug 28, 2024
toolcontrolnetanimatediffipadapter
Download (6.74 KB)
Verified:
Other
Type
Workflows
Stats
622
0
Reviews
(43)
Published
Apr 25, 2024
Base Model
SD 1.5
Hash
AutoV2
CF682304AF
default creator card background decoration
PixelMuseAI's Avatar
PixelMuseAI

This is a vid2vid workflow to transfer the style from photos onto the video.

This workflow is based on the ideas from: https://civitai.com/models/372584/ipivs-morph-img2vid-animatediff-lcm

Models Needed

  1. Your favourite SD1.5 Checkpoint. Put in the models/checkpoints folder

  2. Controlnet. control_v11p_sd15_lineart_fp16.safetensors. Install using the manager, search for "controlnet"

  3. AnimateLCM. https://huggingface.co/wangfuyun/AnimateLCM Download AnimateLCM_sd15_t2v_lora.safetensors and place in the models/loras folder. Download AnimateLCM_sd15_t2v.ckpt and place in the models/animatediff_models folder

  4. IPAdapter Models. I am asking you to install more than what is needed so that if you change some of the options, it will still work. If you are getting errors, you will have to install the exact model that you are missing.

    • CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors and CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors. Install using the manager, search for "clip".

    • ip-adapter_sd15.safetensors, ip-adapter_sd15_light.safetensors, ip-adapter_sd15_vit-G.safetensors, ip-adapter-plus_sd15.safetensors. Install using the manager, search for "ipadapter"

How to use the Workflow

  1. Load the workflow and install the missing custom nodes using the manager. Restart ComfyUI

  2. Teal nodes are the models which you need to load. Blue nodes are the user inputs you need to provide.

  3. Load your video into the load video node. If your video is in landscape, set force_size to ?x512. Set frame_load_cap to 32 for testing out the workflow with the first 32 frames of your video. Set frame_load_cap to zero to use the whole video.

  4. Load the 4 images from which you want to transfer the style.

  5. Edit the frames in the CreateFadeMaskAdvance nodes. You must decide which frame you want to start applying the style and which frame you want to end applying the style. Set the mask to zero at the frame before you want to start and after you want to end. Set the start and end frame to 1.0. For example, if I want to apply the style from frame 90 - 135, you would have to do this: 89:(0.0), 90:(1.0), 135:(1.0), 136:(0.0)

  6. Enter your prompts as needed.

  7. Amend the controlnet as depending on your video. I use lineart because I wanted to transfer the style from artistic images.