What is EbSynth?
a portrait of a beautiful blonde american woman in all blue NASA astronaut suit holding an (ice cream cone:1.1) <lora:add_detail:1>
Download the attachments for the source video and some finished examples with different blend rates.
Original (lowres gif, I could only fit part of the video in the size limit:)
EbSynth is a software package for "Fast Example-based Image Synthesis and Style Transfer". It is a technique dating to 2018. The original github page for EbSynth is available here: https://github.com/jamriska/ebsynth.
By combining EbSynth with modern diffusion models, the process of creating stylized video clips can be streamlined. With EbSynth, frames around each keyframe are interpolated, instead of each frame being a seperate diffusion image like mov2mov.
EbSynth works best for short video clips with less motion. The EbSynth pipeline works by masking and selecting keyframes from the original video, applying a new style to the keyframes in img2img, then interpolating nearby frames with EbSynth. The most challenging aspect of working with EbSynth is managing the transition between keyframes. Download the supplemental materials for the source video and some finished examples with different blend rates.
Selecting a video that requires fewer keyframes is better, because each key frame is likely to result in significant changes to the overall scene. Consider instead using mov2mov for scenes with high motion.
It's recommended to have completed the canny img2img guide before using EbSynth. You must have canny controlnet installed for this guide.
This guide is targeted at Windows, using the AUTOMATIC1111 webUI.
Installing:
Download and extract the EbSynth program. Make note of where you extract the files.
Install the EbSynth WebUi extension
Extensions tab -> install from URL -> https://github.com/s9roll7/ebsynth_utility
Install FFmpeg.
In Settings->ControlNet->Allow other script to control this extension
apply and reload the webUI. You should have a new tab for "Ebsynth Utility".
Getting ready:
Downscale the video
If the input video is too high resolution for your GPU, downscale the video. 720p works well if you have the VRAM and patience for it.
You can use FFmpeg to downscale a video with the following command:
ffmpeg -i input.mp4 -vf "scale=-1:720" output.mp4
In this example:
ffmpeg
is the command to start the FFmpeg tool.-i input.mp4
specifies the input file, in this caseinput.mp4
.-vf "scale=-1:720"
adjusts video's height to 720 pixels and will automatically adjust the width to maintain the aspect ratio. For vertical videos, use 1280 pixels for the height.output.mp4
is the output file.
Remember to replace
input.mp4
andoutput.mp4
with your actual file names
Select the EbSynth Utility tab.
"Project Directory" is where the output for each step of the pipeline is created.
Original movie path doesn't have to be in project directory
Project pipeline steps:
Every pipeline step has additional options under "configuration". Some of those are covered in this guide.
1. Extract and mask video frames
extract frames
After extracting frames, it uses a model to create masks for each image
Don't interrupt the masking process or it will restart from 0.
Default settings work fine for most videos.
--- GPU BOUND: TAKE A BREAK WHILE FRAMES ARE MASKED ---
2. Select keyframes for EbSynth
The utility will attempt to automatically select keyframes. Change the "Threshold of delta frame edge" to higher values to allow more varience between frames before it selects a new keyframe. Start with
Threshold of delta frame edge: 35
. If it outputs too many keyframes, try increasing the threshold and/or minimum keyframe gapMAKE SURE to select "STEP 2" radio button to avoid re-running step 1.
Minimum Keyframe Gap: 10
Maximum Keyframe Gap: 1000 (33 seconds at 30FPS)
You can select keyframes manually by copying files from
video_frame
tovideo_key
This step can be EXTREMELY TEDIOUS. For many videos, this process does not work very well regardless of what keyframes are selected. Your time is better spent selecting a video that needs only a few key frames, instead of selecting individual keyframes for EBsynth.
Approximately 0.3-2s per keyframe works well. Less than that causes too much flicker, and more than that causes too many video artifacts as EbSynth fails to interpolate over too many frames.
3. Generate img2img keyframes
Click "generate" to print instructions for img2img.
Instructions are also printed in the terminal for easy reference
I use the following settings:
For steps 5 and 6 I prefer to use the resolution of the input video
For step 7, follow my canny img2img guide to use higher denoising values https://civitai.com/articles/88/using-img2img-effectively-with-canny-controlnets
Select Checkpoint
Select sampler and # steps
Set canvas size
ControlNet settings (canny)
Denoising: 0.75
Set weight to 1 in both the controlnet and ebsynth script "controlnet option weight"
Put one of the keyframes in the img2img box
Check the "Generation TEST" box and generate until you're happy with the test keyframe
Note that the test generation does not use the mask from step 1, but it's still a good indication of how your prompt will perform over the set of keyframes.
Uncheck the "TEST" box and "Generate". If you have a lot of keyframes, this will take a while.
if you get
AttributeError: 'NoneType' object has no attribute 'convert'
, place any image in the img2img field and try again. It's a bug.
--- GPU BOUND: TAKE A BREAK WHILE KEYFRAMES GENERATE ---
Inspect the keyframe output. You want the keyframes to look pretty similar, without major changes in outfit, accessories, shapes, background, etc. This might take multiple tries, don't rush this step.
For best results, target minimal variation between generated keyframes
3.5 Optional: Perform color correction on the keyframes
Go to Ebsynth Utility -> configuration tab -> Stage 3.5
Choose the best img2img keyframe for color matching and drop it in the window
Generate. A backup of the img2img keys is created at
st3_5_backup_img2img_key
. The color corrected images are placed in theimg2img_key
folderSometimes this makes the result worse, so only do it if there's a lot of color variation in the img2img keyframes
If you have a good monitor please comment on the differences between the color correction algorithms
4. Rescale img2img output
Skip this step if your img2img keyframes are the same resolution as the input video. Otherwise, click "Generate" and follow the printed instructions
5. Generate .ebs files.
Just click generate
These are used with the Ebsynth program
Due to GUI limitation of the EBsynth program only ~15 keyframes are contained in each .ebs file
6. Run EbSynth
Double click the .ebs file and select "open with"... and choose the Ebsynth.exe application. It will open with the keyframe points prefilled.
Click "Run All".
If you generated multiple .ebs files in step 5, you will need run each file individually. It's possible to run multiple windows at a time, but beware that you can overload your system and invoke a BSOD this way, so don't go overboard.
--- CPU BOUND: TAKE A BREAK WHILE KEYFRAMES GENERATE ---
7. Final video render
Now pick the "blend rate" between each keyframe.
Blend rate of 1 is often too high and causes "smeary" transitions.
Blend rate of 0 will perform 0 blending between keyframes and there will be an abrupt transition between each keyframe
Find a blend rate that keeps the most detail while minimizing judder between keyframes
The supplemental materials includes the original video, and blend rates 0, 0.5, and 1.
Please share tips in the comments!