You can view the finished video with sound on my twitter

What can I make with this guide?

This article explains how to make morphed keyframe videos synced to an audio track using Deforum, an extension for the AUTOMATIC WebUi.

Deforum creates a video from multiple image prompts. You instruct deforum for how many video frames each prompt is applied. You can also provide different math functions to control how the camera moves during the video.

Installing

In AUTOMATIC1111 Install in the "Extensions" tab under "URL for extension's git repository"
- https://github.com/deforum-art/sd-webui-deforum
Restart the WebUI
You should have a new tab for "Deforum" after reloading the web UI.

Royalty free music tracks

Here is a source for royalty free music to get started.

Deforum settings

the article attachments includes article_settings.txt which includes the complete settings used to generate the cover video. The audio track is also included.

Run tab:

Set sampler, steps, width and height

Keyframes Tab:

Set 3d, max keyframes
Determine max frames by (duration of audio * desired FPS)
- In this case the audio track is 13 seconds * 24 desired FPS = 312 max frames
- Set cadence to 1
I like to set CFG -> 6 and Seed -> random

Motion: The format is <frame#>:<mathFunction>,<frame#2>:<mathFunction2> The function is applied continuously until the next frame identifier, or until the end of the video.
- You want to use a waveform function- moving too far from the original focal point will cause a bad output, so you can't, for example, have the camera pan right for the entire video.
- The math functions in the image below are used for the preview video in this article.
  - Parameters:
    - 1*sin: Increase or decrease 1 to change the amplitude (maximum value) of the function. Increasing this will cause the camera to move more.
    - (t+24)/74 24 frames of offset with a periodicity of 74 frames. Only the first keyframe (0) is offset in the example below. 74 Frames is approximately 1 measure of the music track.
- You will notice that the preview starts by only moving in the X axis, and does not start moving in the Y axis until frame 56, and has a slight zoom at the end of the video starting at frame 277

Prompts tab:

Set the frame number where each prompt should be applied. You will need to calculate where to place the keyframes based on the BPM of your audio track and your desired FPS.

Things in "prompts positive" and "prompts negative" apply to every keyframe. To add negative prompts for one keyframe, add --neg blurry ugly old <other prompts> at the end of the positive prompt.

To determine where to place the keyframes:

Use a tap tempo application
Tap the tempo of your audio track to get the Beats per Minute (BPM)
Calculate how many frames are in a measure
- For an audio track with 80BPM:
- Divide 80BPM by 60 to get the beats per second (BPS).
  - BPM/60 = BPS
- Then, get Frames per beat (FPB) by dividing FPS by BPS
  - 24FPS/1.33 BPS = ~18 FPB
  - 18FPB * 4 beats per measure = 72 Frames per measure (FPM)
- If the audio track does not start on the first beat of a measure, make sure to calculate the correct offset the first keyframe after 0, using (FPB * number of skipped beats)
- The longer a video is, the more difficult it will be to keep in sync

Output Tab:

Set FPS
Set soundtrack path
Add soundtrack: File
auto-delete imgs when video is ready: Checked

Now is a good time to save your settings:

Generating

Click generate! This video took around 12 minutes to generate.

Don't be surprised if you get an error when you click generate. See troubleshooting below

Troubleshooting:

for: Error: 'invalid syntax (, line 0)'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli.

make sure your path to the audio file doesn't have \ in it (change to /)
make sure you have the valid JSON (no trailing , on the last line for example)
can validate JSON using a tool like JSONLint, paste your entire Prompts entry (including the {}'s) and click "Validate JSON" it should say:

for: Error: 'local variable 'value_is_number' referenced before assignment'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli.

double check your math functions. Remove them all and then re-add them one by one until you've identified the invalid function

Conclusion

Happy generating! Please share comments on what works, additional troubleshooting, and any other cool tips. This guide barely scratches the surface of what can be created with deforum.