Intro
Here I present my recent results of research on video generation with SDXL lightning models.
Script used is available on Github. Simple explanation of generation loop is:
update variables in ComfyUI
generate image
wait until new image in output folder appears
copy file to input folder and rename to "processed.png"
in addition there are other stuff to initiate etc.. In comfy workflow "processed.png" is scaled up or outpainted (depending on cycle type), size is normalized and then next frame is generated with some value of denoise. Examples presented use also IPAdapter and ReActor for character consistency.
Burning problem
While playing with scripts I noticed that after couple iterations image is burned if no "movement" is applied, seed is constant or denoise values are too small. OFC some extreme cases of mentioned factors prevent occurrence of image burning but other effects may appear.
Movement
At first I started with zooming in. This was pretty successful in eliminating burning but other problem became apparent. Subject was changing its position in unpredictable way leading to artifacts like doubling or lack of subject at all. To fight that, I created another workflow that zooms out (outpaints). By alternating between both workflows, one can keep subject in place and make image relatively stable with little editing (eg. in Blender). Future plan include tests with sliding window and combination with ControlNets.
Seed
Second strategy tested is alternating seed. Although initial problem was solved another appeared. Animation became unstable in this way that small frequency elements were appearing and disappearing on frame-to-frame basis. My solution was changing seed in intervals. This led to better stability but then amination was unstable in low frequency elements. In other words, changing rapidly to completely different scene.
Denoise
I experimented a bit with modifying denoise values. If denoise is set to high then animation became high and low frequency unstable. In other hand, set denoise too low and you start to observe image burning, especially on areas unaffected by outpainting.
Experiment
To build better understanding of acceptable denoise ranges for my purpose I decided to run through full range of values, starting with 0.9 down to 0.05 in 0.05 steps. I use Dreamshaper lightning model. Here what I got after 44h rendering on RTX 2060 SUPER:
all at once:
red "seed++" marker indicates when seed is changed
whole uncut thing
Observations and conclusions
For nice consistent animation I would go for denoise in range of (0.45,0.55), preferably 0.45. Some people may find high frequency unstability suitable for their use though.
Another possibility is using low denoise for kind of glitchart. Downside is then low dynamics of generation.
Worth noting is that elements that are added after initial SD process are somehow immune to burning.
Also, denoise values and other observations may not be universal across all models. I made some tests with mskiller51's model and I found another, unique problems.
Anyway, hope you learned something. Cheers.