Making a video with lyrics and music with ChatGPT, Stable Diffusion and Riffusion.

Objective

Making a video with lyrics and music with ChatGPT, Stable Diffusion and Riffusion.

Environment

For this test I will use:

• Stable Diffusion with Automatic1111 (https://github.com/AUTOMATIC1111/stable-diffusion-webui)
  To install follow my article: https://civitai.com/articles/3725/stable-diffusion-with-automatic-a1111-how-to-install-and-run-on-your-computer

• Deforum Extension (https://github.com/deforum-art/deforum-for-automatic1111-webui.git )
  To install follow https://civitai.com/articles/4185/stable-diffusion-deforum-installing-and-generate-videos

• Animatediff ( https://github.com/continue-revolution/sd-webui-animatediff.git)
  To install follow https://civitai.com/articles/3736/stable-diffusion-extension-animatediff-to-generate-videos

• Riffusion : free tool to generate riffs(music with lyrics of 12 seconds) https://www.riffusion.com/

• ChatGPT : To generate the texts :https://chat.openai.com

• KdenLive : Video Tool to make the video with the sound (https://kdenlive.org/)

Checkpoint

• Truality Engine: https://civitai.com/models/158621/the-truality-engine

Model

• Antonia Vicious (https://civitai.com/models/234678/antonia-vicious )

Riffussion

Riffussion is a free tool, you can check it on https://www.riffusion.com/

Registration is free.

You can generate easily writing the Lyrincs and describing the sound.

Check out some of the example in the site:

You can share your creation by download the mp4, mp3 or with the share link

https://www.riffusion.com/riffs/b6234ac5-7ea4-48ca-99ab-9c49c706887f

A really cool option is to split your musin in vocals, drumps,etc.. normally It will create 4 different mp3.. it is great if you want to use in your favorite music tool like Ableton Live or LMMS,etc.

Generating the lyrics and music

First I to to ChatGPT:

My Prompt: need a song about Antonia a nice model that likes the beach.. max 25 words

Ouptut : Antonia, beach's grace, model's face, Sun-kissed sands, where dreams embrace. Her elegance shines, waves' whispering hymn, In her gaze, the ocean swims.

Now I go to riffusion and paste my text.

I want to do a pop song.. I write pop and after I press "Prompt genius"

Now I am ready to "Riff"..

Pressing Riff I get.

I get 3 results

https://www.riffusion.com/riffs/b50e93f2-4f26-4137-896a-ca07c3387e8e

https://www.riffusion.com/riffs/335f39f1-68a3-43b5-bb23-eb0e4179b164

https://www.riffusion.com/riffs/4020e8c8-b9b3-4a9c-8e57-774d779de523

I select the first and download the mp3 video.

Generating the video

Stable Diffusion prompt

DV_Antonia_Vicious beach's grace, model's face,Sun-kissed sands, where dreams embrace.Her elegance shines, waves' whispering hymn,In her gaze, the ocean swims.

Negative

blurry, blurry_image, Lowres, Low_resolution, Low_picture_quality, Low_picture_anime, extra_anatomy, extra_body, extra_navel, extra_face, extra_eyes, extra_chest, extra_nipples, extra_hips, extra_arms, extra_hands, extra_fingers, extra_legs, extra_feet, extra_toe, missing_anatomy, missing_body, missing_navel, missing_face, missing_eyes, missing_chest, missing_nipples, missing_hips, missing_arms, missing_hands, missing_fingers, missing_legs, missing_feet, missing_toe, sketch, sketching, (worst quality: 2), (low quality:2), ((monochrome)), ((grayscale)), inpainting, HDR, bad_prompt_version2, jpeg artifacts, signature, watermark, text, error, missing fingers

Other

Steps: 20

Sampler: Euler a

CFG scale: 7

Seed: 1499790331

Face restoration: GFPGAN

Size: 512x512

Model: theTrualityEngine_trualityENGINEPRO

I set ADdetailer to improve the face with a easy setup

I use the mediapipe_face_full with the name of the embedding DV_Antonia_Vicious

The aninateDiff settings for video are:

Motion module : mm_sd_v15_v2.ckpt

ouput : mp4

Frame per second : 24

Content batch size: 16

Close loop: R-P

Frame interpolation : FILM with Interp X of 6.

Video of this type will be 3 seconds... because the song is 12 seconds.. I need 4 videos.

Here is an example (wepb is not the best quality.. but you get the idea).

Final video

I got 4 mp4 video files and my mp3 sound file:

Drag and drop the files in kdenlive.

Select one video file and clic on Project >> Adjust profile to current clip.

Drag and drop the files in the timeline

In my case the song is less than 12 seconds, I will speed up a little the other videos.

To speed up, click on video with the right button and select "Change Speed"

I set the speed of my videos to 118%

I adjust the position leaving some space at beginning and end and set fade in and fade out with the green and red dots on the video.

We add the subtitles to the mp4 to show the text of the music

Click on the subtitle button and then double click to add subtitles.

Remember to save your project with Files>>Save

To generate the final video go to "Project >> Render"

Select the output filename

The output will be a Generic (HD for web,...) MP4 - H264/AAC

Then press Render to file.

And here is the output: https://civitai.com/posts/1506181

It is really cool to make quick riffs with videos.. I hope in the future I can make complete songs..

I hope you enjoy my article.