Classic dance animation;)

Now it's time for a classic dance animation. This is my second attempt at animating, I spent a lot of time to achieve the result that I liked. I was inspired by this post https://civitai.com/posts/631891 user kkleskk568. 

In a nutshell - it’s a working pipeline. But I ran into some problems:

First, it is important to choose the right source. Which I didn’t do :)

The dress blends into the background, which is not very cool, and the girl actively sings along, hence the generation is falling apart :)

I had to fix it. Put the video into AfterEffects, scale it down to 15 frames, and import it as a PNG sequence.


So, let's begin :).
A good solution is to do two consecutive runs for each frame. The first generation makes changes to the style of the entire image, but at the same time the degree of Denoising strength is only 0.3

We also scale the image using Tiled Diffusion 

And slightly blur the character in the frame while preserving the geometry using Control Net and the SoftEdge preprocessor, and adding OpenPose for better pose control. 

Fullscreen

Now the most interesting thing is that we have already made a good basis for detailing, all that remains is to actually detail it.

At this stage, I took a small piece of the sequence, ran it, and tested it for a very long time until I achieved a good outcome. As a result, the Denoising strength is set to 0, so as not to make changes to the picture itself and the background, which turned out to be stable.
To detail the character I used ADetailer and 2 models: one for the face and one for the body details. It is better to select the Denoising strength level settings for a specific picture. In my case, I decided not to make any major changes to the dress to make the video less screwed up, since the more changes and freedom you allow, the more wonky the video will be. I ended up setting the face Denoising strength level to 0.41.

As a result, the sequence came out with some problems.

Problems :) The girl actively sang along in the video. This caused the mouth to open incredibly wide in the finished shot, which ruined the video. I tried different solutions on the 2nd pass, but it didn’t help, sometimes the mouth completely fell apart. The solution was to take frames that were not suitable and run them through Image to Image with Denoising strength = 0 and ADetailer enabled, making the appropriate changes; in my case, I closed the mouth in all “broken” frames. We repeat the procedure until all the frames are fixed and fit perfectly. In some shots, I used Photoshop or SD Inpainting.

We return to AfterEffects and throw everything there as a PNG sequence of frames.

If desired, enable these items in the effects: Motion Blur and Frame Blending
This will help make the video smoother. If you wish, you can use Flowframes to increase the number of frames

or use DaVinci Resolve with the Deflicker effect.

Also, all manipulations with video can be carried out in a similar program instead of After Effects if this is more convenient for you.

It was an interesting experience overall, the only thing that remains is to understand how to use this in production :)