Announcing: Audio Reactive Video Animator - Bring Your Images and Videos to Life with Sound!

I'm excited to announce a new project I've been working on: the Audio Reactive Video Animator!

GitHub Repository: https://github.com/LamEmil/Audio-Reactive-Video-Animator

What is it?

The Audio Reactive Video Animator is a Python-based application that lets you create dynamic video animations that respond to audio input. You can take a static image or an existing video, feed it an audio file (like your favourite music track), and the tool will generate a new video where the visuals pulse, warp, and transform in sync with the sound.

Core Features:

Audio-Driven Effects:
- Peak-Based Optical Flow: This is one of the coolest effects! It creates warping and zooming visuals that are triggered by audio peaks – think drum hits, bass drops, or strong vocal transients. Your image will seem to punch and move with the beat.
- Breathing Effects: For more subtle or continuous reactions, the animator can analyze the overall energy (RMS) of the audio to create smooth "breathing" modulations. This includes:
  - Zoom
  - Brightness changes
  - Gaussian Blur
  - Saturation adjustments
  - Color (Hue) shifts
Highly Customizable: A user-friendly GUI (built with PyQt6) gives you extensive control over:
- Audio Analysis: Fine-tune sensitivity for peak detection, smoothing windows for RMS energy, and more.
- Effect Parameters: Control the strength, duration, attack/decay envelopes for peak effects, min/max ranges for breathing effects, and the style of visual transformations (e.g., zoom intensity, blur radius).
- Interpolation & Quality: Choose different interpolation methods for visual transformations to balance quality and performance.
Flexible Input/Output:
- Supports common image formats (PNG, JPG, etc.) and video formats (MP4, AVI, etc.) as the base visual input.
- Works with various audio file types (WAV, MP3, FLAC).
- Outputs MP4 video files. If you provide an audio file, it will be muxed into the final video (requires FFMPEG).
User-Friendly Interface: The GUI provides:
- Easy file loading for images/videos and audio.
- Clearly organized tabs for different effect categories (General Settings, Breathing Effects, Peak/Flow Effects).
- Real-time progress bar and status updates during video generation.
- A preview panel for your input image/video and a built-in video player to watch your creation right after it's done!
FFMPEG Integration: Uses FFMPEG for robust video encoding and audio muxing. You can specify your FFMPEG path if it's not in your system's PATH.

Why I Built This:

I wanted to create a tool that was both powerful and accessible for generating audio-reactive visuals without needing complex video editing software or extensive programming knowledge for basic use. The optical flow driven by audio peaks, in particular, can create some really unique and mesmerizing effects. I started out trying to get these effects using comfyui but for longer videos, comfy would freeze my pc as it tried to hand image frames to the next node. So I made my own ui for this.

Getting Started:

Follow the installation instructions in the README. https://github.com/LamEmil/Audio-Reactive-Video-Animator

The README on GitHub has more detailed installation and usage instructions, including troubleshooting tips.
The wiki has a user manual with full details on what all the settings do: https://github.com/LamEmil/Audio-Reactive-Video-Animator/wiki/User-Manual

Screenshots/Examples:

You can see example output videos here:

https://civitai.com/posts/16961117
https://civitai.com/posts/17020360
(second example links just uploaded so may take civit a few hours for all to show and get correctly tagged)

I will be posting more as I continue to make more videos with this.

Call for Feedback:

This is an ongoing project, and I'd love to hear your feedback!

What features would you like to see?
Did you encounter any issues?
Have you made something cool with it? Share it!

Feel free to open issues on GitHub for bug reports or feature requests. I always welcome feedback, I've tried testing as many combinations of use cases as possible to avoid errors and bugs but with a project like this its nearly impossible to catch everything so if you do encounter issues let me know. The code was made largely with the assistance of ai so the code is filled with comments and other stuff. I will likely be going through and cleaning that stuff up.

📽️ Announcing: Audio Reactive Video Animator 🎶