CogVideoX Workflows

These are all IMG2VID workflows:

• CogVideoX_5b_fun_GGUF_Q4_0 - for lower VRAM - portrait and landscape aspect ratio

• Kijai/CogVideoX-Fun-5b - more "creative" results - portrait and landscape aspect ratio

• THUDM/CogVideoX-5bI2V - more demanding on system - more detailed and realistic movement - landscape aspect ratio only

11/04/24: I've just added a version that includes two workflows using different models that can also do portrait aspect ratio. I've made one specifically for you out there that have systems with less VRAM, the GGUF workflow. I noticed 20-30% of my normal VRAM usage compared to the THUDM/CogVideoX-5b-I2V version, hopefully that means those of you that couldn't get it to work before will be able to now.

I'm also pleased with this version's output, I've included two examples in both aspect ratios of the GGUF version, they're labeled if you hover over the box under 'ComfyUI', they are the one with the axolotls in brown outfits preparing deserts, and the one where one is stirring a stew.

Here is a custom GPT that can help with the prompts. You can read more about it here.

This is the result from using the GPT and this workflow.

I've included notes that I think might be helpful, and it's organized in a way that I prefer, with the commonly used inputs grouped together, it also has all of the things that I copy into the generation info near each other to make that easier.

When it comes to CogVideo, the prompting is really important. I've experimented a lot in terms of prompting, but almost all within CFG 8 and 50 steps

There are specific things you can do when going for certain motion, or when working with a certain art style. I figured that it would be helpful to create a collection of a variety of good generations in different art styles and ideas so you can see examples. All of the posts include the generation data and is linked here.

Sometimes the seed can mess with what you're trying to achieve, my approach is that if I can't get any movement with 2 different seeds, then there is an issue with the prompt. Rerolling the same same with a targeted negative prompt to get rid of an undesirable aspect is also effective.

Below are some general tips about the prompting that I hope will be useful:

1. Positive Prompt Setup: Building a Movement-First, Detailed Description

The positive prompt is where you define the scene, characters, and environment. In animation, leading with action-oriented, descriptive language keeps the scene fluid and engaging. Below are steps and examples to structure a strong positive prompt.

A. Define the Animation Style Clearly

• Use movement-focused language to specify the animation style without suggesting it’s a still image. Avoid terms like "painting" or "illustration," which may result in static outputs.

• Incorporate style cues that suggest action to reinforce a sense of movement and vibrancy.

Examples:

• Anime Scene: "vibrant anime-style animation of playful creatures in a magical landscape"

• Fantasy Action Scene: "high-intensity fantasy animation with dramatic lighting and dynamic movement"

• Surreal Setting: "whimsical animated sequence with surreal creatures in a colorful, candy-themed world"

B. Emphasize Character Movements and Interaction with the Environment

• Describe each character's movement to make the scene dynamic. Lead with action verbs and specific descriptors to bring the characters to life.

  • Examples of Action Verbs: “holding,” “jumping,” “walking,” “swinging,” “bouncing,” “twirling”

• Use modifiers to add personality to each action, creating more natural movements.

  • Examples of Descriptive Modifiers: “gripping tightly,” “bouncing excitedly,” “swaying gently,” “snarling menacingly”

Examples:

• Action Scene: "The knight shifts her stance, gripping her sword tightly, her cloak billowing with each movement."

• Playful Scene: "An axolotl bounces on a candy ball, laughing with delight, while another grabs a giant candy cane with a playful grin."

C. Add Environmental and Lighting Effects for Depth

• Describe environmental elements that are in motion, as this enhances immersion and prevents a static feel.

  • Examples: “trees swaying,” “fog drifting,” “sunlight casting shifting shadows,” “water rippling”

• Use lighting and shadow effects to add realism, especially for scenes with sunlight or complex lighting.

  • Examples of Lighting Terms: “reflections on water,” “shadows moving,” “sunlight filtering through the trees”

Examples:

• Forest Scene: "Sunlight filters through the trees, casting shifting shadows, while a light mist drifts along the forest floor."

• Candy Land Scene: "Candy canes sway, and the chocolate river flows gently, with sunlight casting cheerful reflections on the water."

D. Add Cinematic Camera Movements

• Describe the camera’s movement to maintain visual flow and prevent the model from interpreting the scene as a static, zoomed-in frame.

  • Examples: “camera panning across the scene,” “camera circling around the characters,” “camera following the character’s movements”

Examples:

• Dramatic Scene: "The camera slowly pans across the intense standoff, capturing the characters’ expressions and movements."

• Playful Scene: "The camera follows the bouncing axolotl to capture each joyful movement, adding energy to the scene."

2. Negative Prompt Setup: Filtering Out Static Elements and Artifacts

The negative prompt filters out static imagery, visual artifacts, and unintended details that can interrupt animation flow. This section is key to ensuring fluid motion and maintaining a high-quality aesthetic.

A. Combat Static and Still Imagery

• Avoid static frames by specifying terms that discourage still images, zoom-only frames, or frozen characters.

• Anti-Static Terms:

  • “still image,” “static shot,” “motionless figure,” “no movement,” “frozen scene,” “rigid pose,” “no action,” “zoom-in”

• Why It Works: These terms help prevent the model from interpreting the scene as a single frame, maintaining an expectation of motion from every part of the scene.

B. Anti-Stiffness for Characters and Objects

• Prevent characters and objects from appearing stiff by discouraging any rigid postures or frozen limbs.

• Anti-Stiffness Terms:

  • “stiff character,” “rigid figure,” “no leg movement,” “frozen posture,” “static pose,” “motionless limbs,” “no limb movement”

• Why It Works: By targeting body-specific terms like “no leg movement” or “motionless limbs,” you help keep characters’ bodies fluid and lifelike, especially for action-focused scenes.

C. Artifact and Quality Control Terms

• Remove visual artifacts like pixel noise, color bleeding, and jerky motion that could disrupt the clean aesthetic of the animation.

• Artifact Control Terms:

  • “jerky movements,” “pixel noise,” “color bleed,” “blurry edges,” “flickering light,” “excessive blur,” “overexposed lighting”

• Why It Works: Including terms like “pixel noise” and “color bleed” targets issues that often appear in lower-quality outputs, helping you achieve a professional, polished look.

D. Tips for Effective Negative Prompts

• Be specific about what you don’t want, especially with body parts and visual clarity.

• Combine static terms with artifact terms for balanced control.

  • Example Combination: “still image, no movement, jerky movements, pixel noise” filters both stillness and artifacts effectively.

Example Prompt for Quick Reference

To bring all these principles together, here’s a sample prompt setup using both positive and negative prompts:

Positive Prompt:

A vibrant anime-style animation of playful axolotl creatures in a colorful candy land. One axolotl creature holds a giant candy cane with a joyful grin, while another bounces excitedly on a candy ball, laughing with delight. Candy canes sway slightly, and the chocolate river flows gently, adding whimsy to the scene. Sunlight casts a cheerful glow across the candy land, with flowers bobbing in the grass and trees swaying gently in the background. The camera pans across the scene, capturing each joyful movement and the vibrant landscape.

Negative Prompt:

still image, static shot, motionless figure, rigid pose, no action, frozen scene, zoom-in, jerky movements, pixel noise, color bleed, blurry edges, flickering light, unnatural shadows, static background.

Final Tips for Success

1. Experiment with action verbs in the positive prompt. The more specific the action, the more likely you are to get a dynamic result.

2. Refine negative prompts based on results. If something still appears static or jerky, add more targeted anti-still terms or artifact filters.

3. Adjust settings as needed. 50 steps and CFG scale of 8 work well for most scenes, but you can adjust depending on complexity.