Prompt sliding: animatediff technique
Its pretty simple, if you want to make some high quality animatediff animations I found this technique to work well. I will be explaining how to use this technique in full for A1111 however the basic idea applies to all web-ui's
What you will need:
Two extensions:
stable-diffusion-webui-prompt-travel
A motion model:
AD stabilised motion (recomended but not required)
If you dont have a1111 you can get it here:
However personally I prefer to use A1111 through:
You can install the extensions in:
A1111 > Extensions > Install from url
or
A1111 > Extensions > Available > Load from (do not change url) > search for the extension name
Now to the fun part!
note: I have only tested this with an animation that has 2 states.
For the following example I am going to be walking you through on a set seed so you can follow along. I will be using rev animated with the following settings:
seed: 1208199985
Sampling steps: 15
Height/Width: 512x512
CFG scale: 7
Negative prompt: (worst quality, low quality:1.4), bad hands
Note: Rev animated is actually a poor model to use for Animatediff see notes at the bottom for an explanation.
Step 1: create a base prompt
Example: anime style, 1girl wearing a red dress, medium breasts,
Save this prompt somewhere in case you loose it.
Step 2: Modify prompt until you get the first pose.
Example: anime style, 1girl wearing a red dress, medium breasts, standing, (hand raised:1.2), fist,
Save this prompt as well
Step 3: Paste in original prompt and make the second pose.
Example: anime style, 1girl wearing a red dress, medium breasts, standing, peace sign, smile, blush
Save this as well
Step 4: Combining prompts.
We are going to take both pose prompts we made and combine them. Its not as simple as cutting and pasting because the order of the prompts matters. In general it is a good idea to keep similar prompts near each other.
Example: anime style, 1girl wearing a red dress, standing, medium breasts, standing, (hand raised:1.2), peace sign, fist, smile, blush
Take note that we have conflicting prompts (peace sign, fist) but we place them near each.
Step 5: Setting up animatediff
open animatediff (AD)
select your motion module - adstabilized or other (see above)
Select save format - Closed loop "A" (This will cause AD to aggressively cause the image to loop)
Note: R+P also causes AD to loop but in my testing it does not create as smooth transitions between loops.
Number of frames: 24
FPS: 10
Leave everything else on default.
Step 6: Setting up prompt traveling
Prompt traveling allows you to change the prompt mid-animation. This is done by prompting:
Mater prompt
(Frame #): prompt
(Frame #): prompt
Tail prompt
For this example we will not be using a tail prompt:
Simple background
0: anime style, 1girl wearing a red dress, standing, medium breasts, standing, (hand raised:1.2), peace sign, fist, smile, blush
15: anime style, 1girl wearing a red dress, standing, medium breasts, standing, (hand raised:1.2), peace sign, fist, smile, blush
Note: A more in depth explanation on how prompt traveling works is available on the prompt traveling gitHub page. I will not be explaining how it works as that is not the purpose of this tutorial.
Step 7: Prompt sliding
Look at the two prompts you made in steps 2&3. On the frame 0 prompt you want to raise the weight of the words unique to the first prompt. On frame 15 you want to raise the weight for the words on the second prompt. And vice versa lower the weight for the inapplicable words.
Example:
Simple background
0: anime style, 1girl wearing a red dress, standing, medium breasts, standing, (hand raised:1.2), (peace sign:0.3), (fist:1.7), (smile:0.3), (blush:0.3)
15: anime style, 1girl wearing a red dress, standing, medium breasts, standing, (hand raised:1.2), (peace sign:1.7), (fist:0.3), (smile:1.7), (blush:1.7)
Step 8(Final): Refining
Now that you have this all set up you can play with the weights, adjust the prompt and modify things until you have a animation you like. Keep in mind that if you add a word to one frame you need to add it to the others as well. This will generally keep the image consistent and the animation [not buggy].
Thank you for reading this tutorial! I hope this helps! Fell free to message me if you have questions or want to share your animations :).
My knowledge on this subject is not the end all be all. This is just a method I found that works for me. And who knows perhaps there is an even better way. Play around with it, break things go nuts! I want to see some mad science up in this! No wrong answers!
A couple things of note:
You can gen 10-12 images of the steps 2-3 prompts. if the resulting images are pretty consistent then the animations generated will be consistent.
If you have an issue where a particular part of the image is getting messed up and it is not part of the animation prompt you can add it to the master prompt and remove it from the frame prompts and that may help.
If your generated images are "jerky" or they don't flow well, consider using poses that are more similar/a model that has more consistent poses. Both of these issues can cause jerky animation. If you prompt "Hands raised" and on one image the subject has there hands above their head and in the next they have their arms lowered but hands raised (chest height), that's going to cause an issue.
If your base prompt does not produce a consistent subject (The thing you are trying to animate) using a set seed then you may have an issue with "flickering" where things will appear/disappear on every frame. Gen a few images of your prompt using a set see to see if your model has this issue. I was using another model for my AD testing and actually discovered this issue when I made this guide. Rev animated has inconsistent set seeds, who knew?
Consistency is key with AD so it is advisable to test your prompts before you gen a animation especially if it takes a while to generate a animation.