ghost
Sign In
homemodelsimagesvideospostsarticlesbountieschallengeseventsupdatesshop

Video 2 video using Stable Diffusion guide - Part 1 (by ThinkDiffusion)

35

rocketguyishere's Avatar

rocketguyishere

Sep 5, 2023

(Updated: 2 years ago)

video generation guide
vid2vidmov2movstable diffusion videovideovideo2video
Video 2 video using Stable Diffusion guide - Part 1 (by ThinkDiffusion)

Introduction

In this guide, we'll be looking at creating animation videos from input videos, using Stable Diffusion and ControlNet. We will look at 3 workflows:

  • Mov2Mov  
    The simplest to use and gives ok results

  • SD-CN Animation
    Medium complexity but gives consistent results without too much flickering

  • Temporal Kit & Ebsynth
    The most complex but gives you the most control

Workflow 1: Mov2Mov

Firstly, you will need to add the Mov2Mov extension from the following url: https://github.com/Scholar01/sd-webui-mov2mov

  • Go into the extensions tab and paste the above url into the Install from URL tab within the extensions tab and click install

  • Go into the Installed tab within the Extensions tab and click Apply and restart UI

  • You should now see the Mov2Mov tab

  • Please note that if the Mov2Mov tab does not show up, then I'd suggest you close down your instance of Stable Diffusion and re-launch the webui-user.bat file

Creating our Video

  • (1) Select revAnimated_v122 as the Stable Diffusion checkpoint

  • (2) Positive Prompt: photorealistic, (photo of Sci fi cute girl wearing headphones:1.2), pink hair, pink background, Professional gouache painting of beautiful young (realistic) anime girl, beautiful portrait by Artgerm, Ross Tran, Loish and James Jean, trending on ArtStation, trending on Pixiv

  • (3) Negative Prompt: Watermark, Text, censored, deformed, bad anatomy, disfigured

  • (4) Upload your input video

0:00

/0:17

1×

  • (1) Select the sampling method as DPM++ 2M Karras

  • (2) Set the sampling steps to 20

  • (3) I have set the width to 1364 and height to 720 as this matches the resolution of the input video

  • (4) Click on the the MP4V format

  • (5) We can leave the Noise multiplier to 0 to reduce flickering

  • (6) Set the CFG scale to 11

  • (7) and the Denoising strength to 0.6

  • (1) Drop down ControlNet and click enable to ensure Control Net is activated

  • (2) and click Pixel Perfect ✅

  • (3) Select Lineart as the Control Type

  • (4) lineart_realistic as the pre-processor

  • (5) and control_v11p_sd15_lineart as the ControlNet model

  • (6) Set a control weight of 0.6

  • (7) a Control mode of balanced

  • (8) and a resize mode of Just resize

  • Click generate to see what we have created!

If you’re having issues with installation or slow hardware, you can try any of these workflows on a more powerful GPU in your browser with ThinkDiffusion.

Original article can be found here: https://learn.thinkdiffusion.com/video-2-video-using-stable-diffusion/

35