Generative AI as a signal processing problem - practical methods for improving winrates on quality

Link to the full video (all content on the Patreon is free and public):

Included negative expansion helper:

Summary of the video:
Generative AI as a Signal Processing Problem

(Organized Transcript)

1. Introduction: The Core Idea

Generative AI can be viewed through the lens of signal processing, not just as an artistic problem.
Artistic instincts can bias us to use familiar workflows (e.g., DAWs for musicians). This can be helpful but also limiting.
Node-based workflows offer superior flexibility and results compared to traditional workflows like DAWs.

Generative AI involves tuning a "signal," much like adjusting an antenna to improve TV/radio reception.
Analogy: Imagine shining a flashlight onto your hand or paper:
- Step count: How long the light shines (longer = clearer image, but too long can cause "burn-in").
- CFG (Classifier Free Guidance): The strength/wavelength of the light. Higher CFG = brighter, sharper image, but risks artifacting.
- Resolution: Distance between the flashlight and surface. Further = less clear, closer = sharper.

2. Managing Common Problems in Generative AI

Artifacting: When results appear distorted or "overexposed" (similar to overly bright spots from the flashlight analogy).
- Caused by:
  - High CFG
  - Too many sampling steps
  - Inappropriate resolution

Adjust CFG: Lower to reduce artifacting; increase carefully to sharpen.
Adjust Step Count: Often, reducing steps can actually improve outcomes and reduce compute.
Adjust Resolution: Sometimes increasing image size (resolution) can resolve CFG-related artifacting issues without adding steps.

High CFG and lower step count often yield better quality results efficiently.
Lowering CFG and increasing steps can increase stability but decreases efficiency significantly.

3. Sticky Negatives: A Powerful Method

Combines positive and negative prompts carefully.
Purpose: Reduces artifacting by helping AI precisely understand what to avoid (negative) and what to create (positive).
Anchors negatives specifically to the positives to ensure clarity in guidance.

Create detailed and expansive negative prompts, often larger than positive prompts.
Rotate around core concepts, refining negatives iteratively to increase precision.
Negative prompts need clear descriptors of what's undesirable (e.g., “too many legs,” “bad anatomy”).

Successfully showed clearer, more accurate images by increasing negative complexity and adjusting CFG/steps carefully.
Demonstrated improved image quality and reduced artifacts using minimal compute resources (few steps).

4. Advanced Prompting Techniques: Rolling Prompts

Rolling Prompts: Repeating prompt phrases multiple times to leverage the AI’s context window limitations.
Helps the AI achieve more nuanced, multi-perspective interpretations, reducing overfitting and increasing overall quality.

5. Extending Techniques to Music Generation (Sunno)

Hypothesis: There's a universal, underlying "human preference" for certain patterns or qualities.
Aim: To align generative models closer to this baseline human preference by carefully choosing and structuring negatives.

Uses broad descriptors and carefully selected years (based on objective measures like Billboard data accuracy).
Avoids prompts associated with biased data (e.g., religious, state-controlled music) to avoid distorted preference signals.
Result: Improved realism, subtlety, and clarity in generated audio outputs.

6. General Recommendations & Best Practices

Think of generative AI as tuning a clear signal:
- CFG = Signal Strength
- Steps = Exposure Time
- Resolution = Distance to Surface

Fewer sampling steps often yield better results if CFG and negative prompts are carefully managed.
Lower CFG, very high step counts often not efficient—there’s a sweet spot balance.
Quickly identify artifacting and adjust settings accordingly (CFG, steps, resolution, prompt expansion).

Do not underestimate the importance of negatives.
Expanding negatives precisely is often more impactful than expanding positives.
A comprehensive negative strategy significantly improves accuracy, reducing the need for corrective techniques (like inpainting).

7. Summary of Critical Takeaways:

Generative AI as signal processing: adjusting CFG, sampling steps, and resolution to optimize clarity.
Sticky negatives and rolling prompts help refine and stabilize AI output.
Comprehensive negative prompting can significantly improve quality and efficiency.
Applying these methods consistently yields superior results in both image and audio generation.