You can view the finished video with sound on my twitter
What can I make with this guide?
This article explains how to make morphed keyframe videos synced to an audio track using Deforum, an extension for the AUTOMATIC WebUi.
Deforum creates a video from multiple image prompts. You instruct deforum for how many video frames each prompt is applied. You can also provide different math functions to control how the camera moves during the video.
Installing
In AUTOMATIC1111 Install in the "Extensions" tab under "URL for extension's git repository"
Restart the WebUI
You should have a new tab for "Deforum" after reloading the web UI.
Royalty free music tracks
Here is a source for royalty free music to get started.
Deforum settings
the article attachments includes
article_settings.txt
which includes the complete settings used to generate the cover video. The audio track is also included.
Run tab:
Set sampler, steps, width and height
Keyframes Tab:
Set 3d, max keyframes
Determine max frames by (duration of audio * desired FPS)
In this case the audio track is 13 seconds * 24 desired FPS = 312 max frames
Set cadence to 1
I like to set CFG -> 6 and Seed -> random
Motion: The format is
<frame#>:<mathFunction>,<frame#2>:<mathFunction2>
The function is applied continuously until the next frame identifier, or until the end of the video.You want to use a waveform function- moving too far from the original focal point will cause a bad output, so you can't, for example, have the camera pan right for the entire video.
The math functions in the image below are used for the preview video in this article.
Parameters:
1*sin: Increase or decrease
1
to change the amplitude (maximum value) of the function. Increasing this will cause the camera to move more.(t+24)/74
24 frames of offset with a periodicity of 74 frames. Only the first keyframe (0) is offset in the example below. 74 Frames is approximately 1 measure of the music track.
You will notice that the preview starts by only moving in the X axis, and does not start moving in the Y axis until frame 56, and has a slight zoom at the end of the video starting at frame 277
Prompts tab:
Set the frame number where each prompt should be applied. You will need to calculate where to place the keyframes based on the BPM of your audio track and your desired FPS.
Things in "prompts positive" and "prompts negative" apply to every keyframe. To add negative prompts for one keyframe, add --neg blurry ugly old <other prompts>
at the end of the positive prompt.
To determine where to place the keyframes:
Use a tap tempo application
Tap the tempo of your audio track to get the Beats per Minute (BPM)
Calculate how many frames are in a measure
For an audio track with 80BPM:
Divide 80BPM by 60 to get the beats per second (BPS).
BPM/60 = BPS
Then, get Frames per beat (FPB) by dividing FPS by BPS
24FPS/1.33 BPS = ~18 FPB
18FPB * 4 beats per measure = 72 Frames per measure (FPM)
If the audio track does not start on the first beat of a measure, make sure to calculate the correct offset the first keyframe after 0, using (FPB * number of skipped beats)
The longer a video is, the more difficult it will be to keep in sync
Output Tab:
Set FPS
Set soundtrack path
Add soundtrack: File
auto-delete imgs when video is ready: Checked
Now is a good time to save your settings:
Generating
Click generate! This video took around 12 minutes to generate.
Don't be surprised if you get an error when you click generate. See troubleshooting below
Troubleshooting:
for: Error: 'invalid syntax (, line 0)'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli.
make sure your path to the audio file doesn't have
\
in it (change to/
)make sure you have the valid JSON (no trailing
,
on the last line for example)can validate JSON using a tool like JSONLint, paste your entire
Prompts
entry (including the{}
's) and click "Validate JSON" it should say:
for: Error: 'local variable 'value_is_number' referenced before assignment'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli.
double check your math functions. Remove them all and then re-add them one by one until you've identified the invalid function
Conclusion
Happy generating! Please share comments on what works, additional troubleshooting, and any other cool tips. This guide barely scratches the surface of what can be created with deforum.
Credits:
Checkpoint: https://civitai.com/models/64548/fcanimemix-2-fc-anime
Loras:
Royal Gowns: https://civitai.com/models/61135/royal-gowns
Fan BingBing Coture: https://civitai.com/models/16379/haute-couture-or-fan-bingbing-style-fashion
Queen of hearts: https://civitai.com/models/51014/photorealistic-queen-of-hearts
Traditional Maid Dress: https://civitai.com/models/22702/traditional-maid-dress
Jar Jar Binks: https://civitai.com/models/29405/jar-jar-binks-gungan-star-wars
Kaleidoscope Backgrounds: https://civitai.com/models/27549/kaleidoscopic-backgrounds