What is EbSynth?

a portrait of a beautiful blonde american woman in all blue NASA astronaut suit holding an (ice cream cone:1.1) <lora:add_detail:1>

Download the attachments for the source video and some finished examples with different blend rates.

Original (lowres gif, I could only fit part of the video in the size limit:)

EbSynth is a software package for "Fast Example-based Image Synthesis and Style Transfer". It is a technique dating to 2018. The original github page for EbSynth is available here: https://github.com/jamriska/ebsynth.

By combining EbSynth with modern diffusion models, the process of creating stylized video clips can be streamlined. With EbSynth, frames around each keyframe are interpolated, instead of each frame being a seperate diffusion image like mov2mov.

EbSynth works best for short video clips with less motion. The EbSynth pipeline works by masking and selecting keyframes from the original video, applying a new style to the keyframes in img2img, then interpolating nearby frames with EbSynth. The most challenging aspect of working with EbSynth is managing the transition between keyframes. Download the supplemental materials for the source video and some finished examples with different blend rates.

Selecting a video that requires fewer keyframes is better, because each key frame is likely to result in significant changes to the overall scene. Consider instead using mov2mov for scenes with high motion.

It's recommended to have completed the canny img2img guide before using EbSynth. You must have canny controlnet installed for this guide.

This guide is targeted at Windows, using the AUTOMATIC1111 webUI.

Installing:

1. Download and extract the EbSynth program. Make note of where you extract the files.

2. Install the EbSynth WebUi extension

   • Extensions tab -> install from URL -> https://github.com/s9roll7/ebsynth_utility

3. Install FFmpeg.

4. In Settings->ControlNet->Allow other script to control this extension

   • apply and reload the webUI. You should have a new tab for "Ebsynth Utility".

Getting ready:

1. Downscale the video

   • If the input video is too high resolution for your GPU, downscale the video. 720p works well if you have the VRAM and patience for it.

   • You can use FFmpeg to downscale a video with the following command:

     ffmpeg -i input.mp4 -vf "scale=-1:720" output.mp4

     In this example:

     • ffmpeg is the command to start the FFmpeg tool.

     • -i input.mp4 specifies the input file, in this case input.mp4.

     • -vf "scale=-1:720" adjusts video's height to 720 pixels and will automatically adjust the width to maintain the aspect ratio. For vertical videos, use 1280 pixels for the height.

     • output.mp4 is the output file.

     Remember to replace input.mp4 and output.mp4 with your actual file names

2. Select the EbSynth Utility tab.

   • "Project Directory" is where the output for each step of the pipeline is created.

   • Original movie path doesn't have to be in project directory

Project pipeline steps:

Every pipeline step has additional options under "configuration". Some of those are covered in this guide.

1. Extract and mask video frames

• extract frames

• After extracting frames, it uses a model to create masks for each image

• Don't interrupt the masking process or it will restart from 0.

• Default settings work fine for most videos.

  --- GPU BOUND: TAKE A BREAK WHILE FRAMES ARE MASKED ---

2. Select keyframes for EbSynth

• The utility will attempt to automatically select keyframes. Change the "Threshold of delta frame edge" to higher values to allow more varience between frames before it selects a new keyframe. Start with Threshold of delta frame edge: 35. If it outputs too many keyframes, try increasing the threshold and/or minimum keyframe gap

• MAKE SURE to select "STEP 2" radio button to avoid re-running step 1.

• Minimum Keyframe Gap: 10

• Maximum Keyframe Gap: 1000 (33 seconds at 30FPS)

• You can select keyframes manually by copying files from video_frame to video_key

• This step can be EXTREMELY TEDIOUS. For many videos, this process does not work very well regardless of what keyframes are selected. Your time is better spent selecting a video that needs only a few key frames, instead of selecting individual keyframes for EBsynth.

• Approximately 0.3-2s per keyframe works well. Less than that causes too much flicker, and more than that causes too many video artifacts as EbSynth fails to interpolate over too many frames.

3. Generate img2img keyframes

1. Click "generate" to print instructions for img2img.

   • Instructions are also printed in the terminal for easy reference

   • I use the following settings:

     • For steps 5 and 6 I prefer to use the resolution of the input video

     • For step 7, follow my canny img2img guide to use higher denoising values https://civitai.com/articles/88/using-img2img-effectively-with-canny-controlnets

     • Select Checkpoint

     • Select sampler and # steps

     • Set canvas size

     • ControlNet settings (canny)

     • Denoising: 0.75

     • Set weight to 1 in both the controlnet and ebsynth script "controlnet option weight"

2. Put one of the keyframes in the img2img box

3. Check the "Generation TEST" box and generate until you're happy with the test keyframe

   • Note that the test generation does not use the mask from step 1, but it's still a good indication of how your prompt will perform over the set of keyframes.

4. Uncheck the "TEST" box and "Generate". If you have a lot of keyframes, this will take a while.

   • if you get AttributeError: 'NoneType' object has no attribute 'convert', place any image in the img2img field and try again. It's a bug.

--- GPU BOUND: TAKE A BREAK WHILE KEYFRAMES GENERATE ---

• Inspect the keyframe output. You want the keyframes to look pretty similar, without major changes in outfit, accessories, shapes, background, etc. This might take multiple tries, don't rush this step.

• For best results, target minimal variation between generated keyframes

3.5 Optional: Perform color correction on the keyframes

• Go to Ebsynth Utility -> configuration tab -> Stage 3.5

• Choose the best img2img keyframe for color matching and drop it in the window

• Generate. A backup of the img2img keys is created at st3_5_backup_img2img_key. The color corrected images are placed in the img2img_key folder

• Sometimes this makes the result worse, so only do it if there's a lot of color variation in the img2img keyframes

• If you have a good monitor please comment on the differences between the color correction algorithms

4. Rescale img2img output

• Skip this step if your img2img keyframes are the same resolution as the input video. Otherwise, click "Generate" and follow the printed instructions

5. Generate .ebs files.

• Just click generate

• These are used with the Ebsynth program

• Due to GUI limitation of the EBsynth program only ~15 keyframes are contained in each .ebs file

6. Run EbSynth

1. Double click the .ebs file and select "open with"... and choose the Ebsynth.exe application. It will open with the keyframe points prefilled.

2. Click "Run All".

   

   • If you generated multiple .ebs files in step 5, you will need run each file individually. It's possible to run multiple windows at a time, but beware that you can overload your system and invoke a BSOD this way, so don't go overboard.

--- CPU BOUND: TAKE A BREAK WHILE KEYFRAMES GENERATE ---

7. Final video render

Now pick the "blend rate" between each keyframe.

• Blend rate of 1 is often too high and causes "smeary" transitions.

• Blend rate of 0 will perform 0 blending between keyframes and there will be an abrupt transition between each keyframe

• Find a blend rate that keeps the most detail while minimizing judder between keyframes

• The supplemental materials includes the original video, and blend rates 0, 0.5, and 1.

• Please share tips in the comments!