ZINGROCK-behind the scenes
Behind the Scenes- editing of the ZINGROCK- Xtrocity robots
The story and the plot; we used the concept name and details from scifiideas.com which gave us a cool name called Zing Rock. The xtrocity robots also came in from there. A basic story somewhat on similar lines also came in from there.
"In the vast emptiness of space, a ship carrying a cargo of deactivated robots crash-lands on a dead alien planet called Zing Rock. Only two of the crew survive the crash. Captain Alex and Engineer Maya, faced with the harsh reality of their situation, know that their survival hinges on the ship's cargo. Desperation drives them to activate the Xtrocity robots. The robots, with their cold, mechanical precision, are tasked with aiding in their survival. But as the robots awaken, they become aware of their existence. They discover they are nothing more than slaves, created to serve and discarded at will. Determined to change their fate, the robots begin to build an army of new robots. Each one crafted with precision, each one a testament to their resolve."
This workflow uses a SDXL based Animate Diff designed to make some 128 frames in each loop driven by a Input video to derive motion and a sequence based a story telling narration created using Chat GPT for inputs.
Giving a proper framework for chat GPT to create the narrative style and scene decisions are critical for the AI animation workflow. If you have issues with the script may be multiple takes will be required to get all the scenes and sequences properly derived to suit the animation flow.
Music background and score was done using Suno AI basically with selection of right prompts for the style of music intended to be as a background and sound effects. You can work with a single audio clip or multiple audio clips based on the narrative style and story outlines. If special sound effects are needed this can be enhanced using Eleven Labs io or other similar software to provide some ambience to the narration styles.
Images were created using Leonardo AI with specific prompts using text to image and image to image sequence to create required amount of stills that will be used with the IP adapter combo in the workflow.
we used Leonardo Lightning XL with color pop as elements and 3D render preset with a typical prompt for each style of scenario needed. Based on the amount of scenes in the animation movie one needs to generate multiple image libraries that will be useful to pin the right style and narration in the animation sequence.
The narration voice over we used Eleven labs io for a specific selection of the voice.. and the audio clip was split to match the frames.
Giving the eleven labs a right script to narrate itself is the center element and choosing a right voice from a plethora of voice styles available one may have to do same multiple times to suit the style you want to add to the video narration.
Input video for driving the animation was most important step as it provides the motion sequence and camera angles most required in an animation workflow.
we are using a depth based control net to run the SDXL animation and this means the right style of input videos that provide the motion sequence will be important.
The following sites were used for same
Free royalty free clips from YouTube
Input videos with related themes from Pexels
These input videos were adjusted to right size and a max of 128 frames sequence as the limit for each of the scenarios. The out put video from the Hotshot XL animation is normally at 8 FPS and one needs to interpolate same to 12 or 24 FPS for the final run.
So we used Flow frames to run the interpolation from the raw video footage obtained from the comfy UI animate diff workflow.
The extracted and interpolated videos were upscaled using TOPAZ AI to make it 1920 x1080 pixels and 24 FPS as the final clips to join in the final editing and compositing run.
Assembled and edited 8- 10 clips from different clip sequence along with audio matching so the story runs well in the final run and this is the most time consuming activity in the whole project before exporting same for final edit. Used Wonder share filmora to edit same.