Image Links (view full post for text explanation):
Prompt Parser Strategies... | Civitai
Changelog for prompt_parser.py
Date: 1/30/2025
🔹 Key Changes:
✅ Updated Sequence Closures:
Changed sequence closures from
;and_ ;to:~for separating elements within a sequence.!for closing sequences.!!for closing top-level sequences.
✅ New Sequence Definition Rules:
Introduced top-level sequences (
top_level_sequence) and nested sequences (nested_sequence).This ensures better structuring and scoping of sequences.
Top Level sequences start with (
:::) and end with!!.Example:
character:::outfit::black_dress~accessory::silver_necklace!, hairstyle::ponytail~hair_color::blonde!, eye_color::green!!✅ Parent Requirement for Sequences:
Sequences (
::) must now have an explicit preceding prompt.Prevents orphan sequences without a defined parent.
✅ Refined plain Character Handling:
Removed certain characters from the plain text category that caused parsing issues.
Adjusted
plainto explicitly include necessary characters.
🔹 Other Fixes & Improvements:
✅ Optimized sequence resolution to enforce structure without affecting compatibility.
✅ Refactored sequence handling logic to better distinguish owners from attributes.
✅ Minor optimizations for parsing efficiency and error handling.
📌 Please update your prompts accordingly to ensure compatibility with the new parser changes.
Below is a link to show using the prompt parsing rules:
New Parser Update - Evolution of a Prompt | Civitai
To use this file, rename it to "prompt_parser.py" and place into your modules folder in A1111.
That leaves 3 files as attachements:
prompt_parser -A1111 = original file
prompt_parser - forge = original file with support for forge
prompt_parser = new file using the above new parsing techniques
Reasons to change from ; and _; to ~ and ! and !! and reasons to update to the new file for A1111
"_" is already defined in the and_rule and could cause parsing errors
"_" was defined in plain when it should have been removed
Other rules defined in plain should have been removed which conflict with current rules
Requiring a prompt to define a sequence wasn't included in the original file; in this file we explicitly define a prompt being the parent for child elements - and to do that you simply add ":::" instead of the standard two colon start - "::". To differentiate when a sequence and a top_level_sequence ends, we use two "!!" to denote an end to a top_level_sequence and only one "!" for a regular sequence.
I'll add a forge update soon.
Update 1/10/2025:
Added Forge Implementation:
Essentially - to work with forge you need to add in the SDConditioning for distilled_cfg_scale=None since this is now part of processing. For this code section:
class SdConditioning(list):
"""
A list with prompts for stable diffusion's conditioner model.
Can also specify width and height of created image - SDXL needs it.
"""
def __init__(self, prompts, is_negative_prompt=False, width=None, height=None, copy_from=None, distilled_cfg_scale=None):
super().__init__()
self.extend(prompts)
if copy_from is None:
copy_from = prompts
self.is_negative_prompt = is_negative_prompt or getattr(copy_from, 'is_negative_prompt', False)
self.width = width or getattr(copy_from, 'width', None)
self.height = height or getattr(copy_from, 'height', None)
self.distilled_cfg_scale = distilled_cfg_scale or getattr(copy_from, 'distilled_cfg_scale', None)I'm adding this file to my github as "prompt_parser - forge.py" to differentiate them. When you put it in your forge\modules folder, make a copy of the original prompt parser, then paste this file in. Rename this file to simply "prompt_parser.py" and then run a new instance of webui to use it.
I wanted to use python 3.9 and didn't feel like updating so I modified Forge to work with Python 3.9 --not getting into the details but here is a sample image using the prompt parser:
mountain::lake, Tree::oak, autumn_;, mountain:lake:0.7:15%-30%
Steps: 35, Sampler: DPM2, Schedule type: Karras Exponential, CFG scale: 7, Seed: 140850743528, Size: 960x640, Model hash: 81d4d52035, Model: Qasar_anireal, Denoising strength: 0.4, Hires Module 1: Use same choices, Hires CFG Scale: 7, Hires upscale: 2, Hires steps: 35, Hires upscaler: 4x-UltraSharp, Version: f2.0.1v1.10.1-previous-633-ge073e4ec
Adding in my scheduler was also very easy PM me if you need help with that.
UPDATE 12/10
And more updates planned
This update adds: reverse scheduling and scheduling with steps or %
How to use reverse scheduling
- after setting up the schedule, add "r" or "reverse":
[prompt]:0.7:r
or prompt:0.7:r
How to use scheduling with steps:
Note: it will clamp max steps if you specify in the prompt more than you actually use.
Note: 15-30 vs 15%-30% (example 100 steps) will not give you the same image. I thought it would and I tried to correct it and I guess I gave up, but I'm ok with it. It gives a slightly different image, and I'm ok with that. Let me show you an example of the differences between 15-30 and 15% to 30%:
mountain:lake:0.7:15%-30%
mountain:lake:0.7:15-30
Similar but different images.
And I'm ok with that, and hope you will be too. If not feel free to contribute on github.
So back to explaining step scheduling.
Essentiallly, you can instruct it to focus on steps 15-30 or 15% -30% of the generation (or pick your number/percent) and it will focus on those elements. You might not see much with a simple prompt with nothing else, but this is more of an example of how to use it versus the best prompt you can do.
Example prompt inputs for both -- please see the beach images here: Image post by KittensX | Civitai
or reference the below sample image. I thought the orange was a bit overstated but I kind of expected as much based on the prompt. I thought the sand and imprints in the sand were spot on for realisticness.
background::{white|blue|orange_sky}:50%-100%:r | {orange|white|blue_water}:10%-50%;, {sky, clouds}, {sand, beach::footprints in the sand;}
Steps: 100, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 118255258105, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.4, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 60, Hires upscaler: 8x_NMKD-Superscale_150000_G, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}
UPDATE 12/6/2024:
Added the following:
use | for OR functionality.
use & or AND for implicit AND functionality
use "::" to start a sequence, end it with ";" or if a nested sequence, must end each nested sequence with ";" with the last sequence with "_;".
use "_" to link concepts
for ungrouped requirements to join ideas like "red_hat" or "green_hair"
in progress ~ limited functionality in practice
use numbers joined with "_" for quantity. In practice, it might be easier for CLIP to detect the use of a number in a sentence rather than a parse rule. If I can't get it to function with parsing I may re-release without this function.
This is a work in progress. In theory, if the number is followed by an exclamation point, it should enforce at least that amount in the output. It might duplicate output without actually get the model to draw a quantity number of items. Like if you wanted to draw 2 moons 2_moons ...it might just be easier and the model might understand it if you just use natural language. Using 2_ or 4_ or whatever number inside a group is meant to simplify a prompt, but it might just not be implemented correctly. Hence why I've added some functionality but won't guarantee it's effectiveness.
Simple Examples to better understand these parsing rules:
| Function (allows or without using [])
red | blue | green
Only one option will be chosen during parsing or generation
Seems to either work great, or favor one output. Is not 100%.
Using & to enforce conditions
Simple example:
red & blue & green
Represents a combination where all conditions are required simultaneously.
In negative prompts, it would mean that none of these are allowed
Conjoining words with "_"
red|blue|green_eyes
Will return red_eyes or blue_eyes or green_eyes. Because the last sequence had the joiner "_", it allows you to do various things like blonde|brunette|red_hair
red_hair will group the first concept with the second concept
Sequences
Sequences are meant to enhance assignment of attributes to objects.
They can be simple, or they can include nested sequences. Example:
You could describe a person like this:
{female:: hair::blonde, natural color, long, curly;, eyes::beautiful, brown, large;, adult, wearing::gym shorts, midriff, socks, sneakers_;}, {male::brown eyes, green hair, red_shirt}
Seems to work fairly well, though I don't know everything and refinement is still needed, and though colors might be specified, sometimes the just end up on the wrong person. So I'm not 100% sure how to enforce this.
{2 people_{female:: hair::blonde, natural color, long, curly;, eyes::beautiful, brown, large;, adult, wearing::gym shorts, midriff, socks, sneakers_;}, {male::brown eyes, green hair, red_shirt}}
Negative prompt: {magazine, watermark}, nude
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758917, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
{2 people_{female:: blonde_hair:: natural color, long, curly;, brown_eyes::beautiful, large;, adult, blue_{gym shorts}, midriff, white_socks, pink_sneakers_;}, {male::brown_eyes, wearing::green_shirt, black_pants, striped_shorts;}}, park_background::bench, tree, brook, wooden_bridge::over the stream, concrete path_;,
Negative prompt: {magazine, watermark}, nude, hand:6_fingers AND 3_fingers AND wrong_fingercount;
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758915, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
Using sequence ("::")
In practice, assigning attributes to objects has been easier, whether you use a sequence or use conjoining "_"
female:: age-22, blonde, glasses, tall, blue eyes, pink shirt,
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 14870554953, Size: 960x640, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Denoising strength: 0.7, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 40, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
Using Sequence and Conjoiners
{2 females:: brunette|blonde hair:: long:: slightly wavy; , {blue|green_eyes}; , adult::age-18;, wearing::(red_panties) & (white_bra);, eyes::beautiful_eyes, (same_color);, height::same_size_;}, background::studio, deep black,solid black;, lighting: filtered, synthetic;,
Negative prompt: {magazine, watermark}, arms::3_arms, intersecting;, legs:3_legs;, {child::childlike features;, teen}, breasts::large;, hand::6_fingers, 3_fingers, no_thumb_;, fingers::trailing, unattached;, eyes:: plastic, bad, ugly;
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758916, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
{2 females:: brunette|blonde hair:: long:: slightly wavy; , {blue|green_eyes}; , adult::age-18;, wearing::(red_panties) & (white_bra);, eyes::beautiful_eyes, (same_color);, height::same_size_;}, background::studio, deep black, inside a room;, lighting: filtered, synthetic;,
Negative prompt: {magazine, watermark}, arms::3_arms, intersecting;, legs:3_legs;, {child::childlike features;, teen}, breasts::large;, hand::6_fingers, 3_fingers, no_thumb_;, fingers::trailing, unattached;, eyes:: plastic, bad, ugly;
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 84494253437, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
It doesn't seem to be 100% but these prompts are fresh and need refinement, but it looks promising.
More pictures
I have started to test out the functions. The OR and the AND function is iffy, but the use of the "_" to connect two objects like you would in a group using "{ }" seems to work fairly well. Using these may change a model to give a different style output, at least that's what I've noticed with mine. Here below are a few more sample pictures that I've been messing around with.
[mountain:lake:0.5], {stormclouds | sunny}
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 79500811399, Size: 960x640, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Denoising strength: 0.7, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 40, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
{female:: tall:: athletic; cheerful & kind_eyes; wearing tshirt & shorts_; | animal:: furry, small, playful; } & {red_hat, green_hair & blue_eyes};, photorealistic, a photo,
Negative prompt: {magazine, watermark}, nude
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758915, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
2_{female:: tall:: athletic, blonde; cheerful & kind_eyes_; | animal:: furry, small, playful; } & {red_hat | green_hair & blue_eyes};
Negative prompt: {magazine, watermark}, nude
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758915, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
2_{person:: tall:: athletic, muscular_; cheerful & kind_eyes_; | animal:: furry, small, playful_; } | 3_{red_hat | green_hair & blue_eyes};
Negative prompt: {magazine, watermark},
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758917, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
{2 females:: brunette|blonde hair:: long:: slightly wavy; , {blue|green_eyes}; , adult::age-18;, wearing::(red_panties) & (white_bra);, eyes::beautiful_eyes, (same_color);, height::same_size_;}, background::studio, deep black,solid black;, lighting: filtered, synthetic;,
Negative prompt: {magazine, watermark}, arms::3_arms, intersecting;, legs:3_legs;, {child::childlike features;, teen}, breasts::large;, hand::6_fingers, 3_fingers, no_thumb_;, fingers::trailing, unattached;, eyes:: plastic, bad, ugly;
Steps: 60, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 32477758914, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Clip skip: 2, Hypertile U-Net: True, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
==============================================
Also on github here Link updated on 12/6/2024
For updated pictures & Tests please view the post (click the view button) to see the entire post, which includes detailed comments and comparisons.
Another Test & Comparisons | Civitai
It shows how varying between grouping {}, transitions : , or scheduling, object1:object2:object3:weight example: mountain:lake:trees:0.5 can transform your image. It is a simple test meant to illustrate how you can use them in your pictures.
Update: added 'grouped' to the prompt section on line 17. Now it's properly parsing the grouped parse rules.
Update: commented out the sorted list return on res (~ line 142). Now it is returning res without sorting or removing duplicates.
New Features
So, I've added the following functionality:
1) Grouping using brackets {}.
2) Multiple scheduling using colons :.
3) Optional debugging (uncomment the print statements)
4) Apply weights to entire groups or individual transitions
Grouping
With phrases you can group with brackets to treat the phrase as one prompt. Gives more precision control over your image.
--Original Parser = No explicit support for grouping. Prompts are parsed individually or as scheduled transitions using :.
Benefit: Grouping improves control over scene composition by maintaining relationships between prompt elements.
All items are treated as a cohesive group. It can be simple or a complex prompt involving multiple phrases, groups, transitions, etc.
Weighted Prompts and Transitions
Brackets ([]) can include multiple colon-separated elements, followed by an optional weight. For example:
[mountain:lake:river:0.25] applies a 0.25 de-emphasis to the whole sequence.
Original Parser: Limited handling of weights and no clear support for multiple elements within brackets (beyond 2 elements).
Benefit: The improved parser provides fine-grained control over transitions and emphasis within a prompt.
Example use combining weighted prompts and grouping:
city:storm:lightning:tree:street:puddle:0.7, ({{closeup}, {green eyes}, {blonde hair}, {1 adult, woman, {peach orange dress}}}),
Negative prompt: watermark, {child, teen}, selfie, muscular, glow,
Steps: 45, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 6265453010, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Denoising strength: 0.55, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
If you can logically group items together, you can discover more efficient ways to craft what you want with possibly fewer images.
{puddle,lightning,storm clouds}:{sunset,city}:0.5, {woman, (brown)-eyes, white dress}:{man, (blond)-hair, green eyes}:0.9
Negative prompt: powerlines, watermark, {crowds of people}:1.1
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 121826728852, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires upscaler: Latent, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
{puddle,lightning,storm clouds}:{sunset,city}:0.7, {woman, {blue eyes}, {white dress}}:1.1, {{portrait}, {looking at camera}}:candid:1.1
Negative prompt: watermark, {child, teen}, selfie, muscular, separated, powerlines, {crowds of people}, cars, flooding,
Steps: 45, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 53043189590, Size: 640x960, Model hash: 31ca5c2b37, Model: ACR_fusion_v2_float16, Denoising strength: 0.55, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "31ca5c2b37"}
Conclusion
I've only scratched the surface what this can do, but my takeaways for what I like the most are:
(i.e. the two main features)
Groupings
Why? Obvious reasons - grouping descriptions together for objects and having the model do what I want it to do.
Multiple prompts scheduling
Why? While I have not really used this in the past, I find that being able to group a bunch of things together, and then assign it a group emphasis/deemphasis is a great way to create a new scene from scratch that involves less time crafting the background, the model, and any other special things like colors, material, etc. Can spend less time in development and more time creating images.
How to Install
(See Attachments)
Backup your current file in your modules folder for the same name, "prompt_parser.py"
Replace or move the prompt_parser.py file into your modules folder, which is located in your root (where you installed Stable Diffusion) folder. root\modules\prompt_parser.py.
Cover art
a majestic ((deer)):{standing on a moss-covered [rock] near the stream, looking over its shoulder}:0.95, a {lush rainforest, serene waterfall:0.6} with [misty clouds:clear skies:0.3] above and a [winding river:0.5]:rocky stream:0.4 flowing through, surrounded by {wildlife, vibrant flowers} and occasional [wildlife:birds:0.2] grazing near a [clearing:shaded grove:0.7]. At a distance, a majestic {mountain range, volcano:0.4} looms, casting a shadow over a {golden meadow:rolling hills:0.5}. Closer to the viewer, a foreground of {dense foliage, tall trees:0.7} with [sunlight streaming through:soft shade:0.4] adds contrast,
Negative prompt: cluttered background, harsh lighting, unnatural pose, blurry details
Steps: 65, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 102601704314, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}
Codeblock
'''
from __future__ import annotations
import re
from collections import namedtuple
import lark
# a prompt like this: "fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][: in background:0.25] [shoddy:masterful:0.5]"
# will be represented with prompt_schedule like this (assuming steps=100):
# [25, 'fantasy landscape with a mountain and an oak in foreground shoddy']
# [50, 'fantasy landscape with a lake and an oak in foreground in background shoddy']
# [60, 'fantasy landscape with a lake and an oak in foreground in background masterful']
# [75, 'fantasy landscape with a lake and an oak in background masterful']
# [100, 'fantasy landscape with a lake and a christmas tree in background masterful']
schedule_parser = lark.Lark(r"""
!start: (prompt | /[][():]/+)*
prompt: (emphasized | scheduled | grouped | alternate | plain | WHITESPACE)*
!emphasized: "(" prompt ")"
| "(" prompt ":" prompt ")"
| "[" prompt "]"
scheduled: "[" [prompt (":" prompt)+] "]" ":" [WHITESPACE] NUMBER [WHITESPACE] //allows use of optional brackets to apply weights to phrases
alternate: "[" prompt ("|" [prompt])+ "]"
grouped: "{" (prompt ",")+ [prompt] "}" // Group descriptions with commas
WHITESPACE: /\s+/
plain: /([^\\\[\]():|]|\\.)+/
%import common.SIGNED_NUMBER -> NUMBER
""")
def get_learned_conditioning_prompt_schedules(prompts, base_steps, hires_steps=None, use_old_scheduling=False):
"""
>>> g = lambda p: get_learned_conditioning_prompt_schedules([p], 10)[0]
>>> g("test")
[[10, 'test']]
>>> g("a [b:3]")
[[3, 'a '], [10, 'a b']]
>>> g("a [b: 3]")
[[3, 'a '], [10, 'a b']]
>>> g("a [[[b]]:2]")
[[2, 'a '], [10, 'a [[b]]']]
>>> g("[(a:2):3]")
[[3, ''], [10, '(a:2)']]
>>> g("a [b : c : 1] d")
[[1, 'a b d'], [10, 'a c d']]
>>> g("a[b:[c:d:2]:1]e")
[[1, 'abe'], [2, 'ace'], [10, 'ade']]
>>> g("a [unbalanced")
[[10, 'a [unbalanced']]
>>> g("a [b:.5] c")
[[5, 'a c'], [10, 'a b c']]
>>> g("a [{b|d{:.5] c") # not handling this right now
[[5, 'a c'], [10, 'a {b|d{ c']]
>>> g("((a][:b:c [d:3]")
[[3, '((a][:b:c '], [10, '((a][:b:c d']]
>>> g("[a|(b:1.1)]")
[[1, 'a'], [2, '(b:1.1)'], [3, 'a'], [4, '(b:1.1)'], [5, 'a'], [6, '(b:1.1)'], [7, 'a'], [8, '(b:1.1)'], [9, 'a'], [10, '(b:1.1)']]
>>> g("[fe|]male")
[[1, 'female'], [2, 'male'], [3, 'female'], [4, 'male'], [5, 'female'], [6, 'male'], [7, 'female'], [8, 'male'], [9, 'female'], [10, 'male']]
>>> g("[fe|||]male")
[[1, 'female'], [2, 'male'], [3, 'male'], [4, 'male'], [5, 'female'], [6, 'male'], [7, 'male'], [8, 'male'], [9, 'female'], [10, 'male']]
>>> g = lambda p: get_learned_conditioning_prompt_schedules([p], 10, 10)[0]
>>> g("a [b:.5] c")
[[10, 'a b c']]
>>> g("a [b:1.5] c")
[[5, 'a c'], [10, 'a b c']]
"""
if hires_steps is None or use_old_scheduling:
int_offset = 0
flt_offset = 0
steps = base_steps
else:
int_offset = base_steps
flt_offset = 1.0
steps = hires_steps
def collect_steps(steps, tree):
#if not tree or not hasattr(tree, 'children') or not tree.children: #debugs
#print("Invalid tree structure:", tree)
#return []
res = [steps] # Always include the final step
class CollectSteps(lark.Visitor):
def grouped(self,tree):
# Collect all descriptions within the group
group_descriptions = [
self._resolve_tree(child) if isinstance(child, lark.Tree) else str(child)
for child in tree.children]
#print(f"Group: {group_descriptions}") #debug
# Handle the group as a cohesive unit (e.g., append to results)
res.append(", ".join(group_descriptions))
def scheduled(self, tree):
#Validate tree.children #debug
if not hasattr(tree, "children") or not tree.children:
#print("Invalid tree or missing children:", tree) #debug
return
# Collect all prompts and the scheduling number
prompts = tree.children[:-2] # All but the last two children are options
number_node = tree.children[-2] # Second-to-last child is the scheduling number
# Debugging: Inspect tree structure
#print("Scheduled Node:", tree.pretty())
#print("Prompts:", prompts)
#print("Number Node:", number_node)
# Safeguard for missing or invalid children
if not prompts or not number_node:
#print("Invalid scheduled node structure:", tree) #debug
return
try:
# Convert number_node to a float
v = float(number_node) #keep as a float!
except ValueError:
#print(f"Invalid scheduling number: {number_node}") #debug
return
#Apply Weight (de-emphasis) to each prompt
weighted_prompts = [(prompt, v) for prompt in prompts]
# Divide steps equally for each transition between prompts
num_prompts = len(weighted_prompts)
#use when needed: If v can represent either a fraction or a raw number of steps (v < 1 or v >= 1
step_intervals = [
(i + 1) * (v / num_prompts) if v < 1 else (i + 1) * steps / num_prompts
for i in range(num_prompts - 1)
]
'''
#simpler calculation when you always use a fraction
step_intervals = [
int((i + 1) * (v * steps) / num_prompts) for i in range(num_prompts - 1)
]
'''
tree.children[-2] = step_intervals # Replace number_node with numeric step intervals
res.extend(step_intervals)
# Visit the tree and collect step intervals
CollectSteps().visit(tree)
#return sorted(set(res)) # Remove duplicates and sort
return res #does not remove duplicates or sort
def at_step(step, tree):
class AtStep(lark.Transformer):
def scheduled(self, args):
#print("Scheduled args:", args) #debug
# Ensure args is valid
if not args or len(args) < 2:
#print("Invalid scheduled args:", args) #debug
return
*prompts, when, _ , weight = args # Extract prompts and step boundaries
if not isinstance(when, list):
#print(f"Invalid step boundaries: {when}") #debug
return
# Select the appropriate prompt based on the step
for i, boundary in enumerate(when):
if step <= boundary:
yield f"({prompts[i]}:{weight})" # Apply weight (de-emphasis)
return
# Default to the last prompt with the weight if step exceeds boundaries
yield f"({prompts[-1]}:{weight})"
def alternate(self, args):
# Handle alternates with a cycle
args = ["" if not arg else arg for arg in args]
yield args[(step - 1) % len(args)]
def start(self, args):
#flatten nested structures into a single string
def flatten(x):
if isinstance(x, str):
yield x
else:
for gen in x:
yield from flatten(gen)
return ''.join(flatten(args))
def plain(self, args):
#handle plain text nodes
yield args[0].value
def grouped(self, args):
# Return the group as a cohesive string
return ", ".join(args)
def __default__(self, data, children, meta):
#handle all other nodes
for child in children:
yield child
return AtStep().transform(tree)
def get_schedule(prompt):
try:
tree = schedule_parser.parse(prompt)
#print(tree.pretty()) # Debugging: visualize the tree structure
except lark.exceptions.LarkError as e:
#print(f"Parsing error for prompt: {prompt}")
#if 0:
# import traceback
# traceback.print_exc()
return [[steps, prompt]]
return [[t, at_step(t, tree)] for t in collect_steps(steps, tree)]
promptdict = {prompt: get_schedule(prompt) for prompt in set(prompts)}
return [promptdict[prompt] for prompt in prompts]
ScheduledPromptConditioning = namedtuple("ScheduledPromptConditioning", ["end_at_step", "cond"])
class SdConditioning(list):
"""
A list with prompts for stable diffusion's conditioner model.
Can also specify width and height of created image - SDXL needs it.
"""
def __init__(self, prompts, is_negative_prompt=False, width=None, height=None, copy_from=None):
super().__init__()
self.extend(prompts)
if copy_from is None:
copy_from = prompts
self.is_negative_prompt = is_negative_prompt or getattr(copy_from, 'is_negative_prompt', False)
self.width = width or getattr(copy_from, 'width', None)
self.height = height or getattr(copy_from, 'height', None)
def get_learned_conditioning(model, prompts: SdConditioning | list[str], steps, hires_steps=None, use_old_scheduling=False):
"""converts a list of prompts into a list of prompt schedules - each schedule is a list of ScheduledPromptConditioning, specifying the comdition (cond),
and the sampling step at which this condition is to be replaced by the next one.
Input:
(model, ['a red crown', 'a [blue:green:5] jeweled crown'], 20)
Output:
[
[
ScheduledPromptConditioning(end_at_step=20, cond=tensor([[-0.3886, 0.0229, -0.0523, ..., -0.4901, -0.3066, 0.0674], ..., [ 0.3317, -0.5102, -0.4066, ..., 0.4119, -0.7647, -1.0160]], device='cuda:0'))
],
[
ScheduledPromptConditioning(end_at_step=5, cond=tensor([[-0.3886, 0.0229, -0.0522, ..., -0.4901, -0.3067, 0.0673], ..., [-0.0192, 0.3867, -0.4644, ..., 0.1135, -0.3696, -0.4625]], device='cuda:0')),
ScheduledPromptConditioning(end_at_step=20, cond=tensor([[-0.3886, 0.0229, -0.0522, ..., -0.4901, -0.3067, 0.0673], ..., [-0.7352, -0.4356, -0.7888, ..., 0.6994, -0.4312, -1.2593]], device='cuda:0'))
]
]
"""
res = []
prompt_schedules = get_learned_conditioning_prompt_schedules(prompts, steps, hires_steps, use_old_scheduling)
cache = {}
for prompt, prompt_schedule in zip(prompts, prompt_schedules):
cached = cache.get(prompt, None)
if cached is not None:
res.append(cached)
continue
texts = SdConditioning([x[1] for x in prompt_schedule], copy_from=prompts)
conds = model.get_learned_conditioning(texts)
cond_schedule = []
for i, (end_at_step, _) in enumerate(prompt_schedule):
if isinstance(conds, dict):
cond = {k: v[i] for k, v in conds.items()}
else:
cond = conds[i]
cond_schedule.append(ScheduledPromptConditioning(end_at_step, cond))
cache[prompt] = cond_schedule
res.append(cond_schedule)
return res
re_AND = re.compile(r"\bAND\b")
re_weight = re.compile(r"^((?:\s|.)*?)(?:\s*:\s*([-+]?(?:\d+\.?|\d*\.\d+)))?\s*$")
def get_multicond_prompt_list(prompts: SdConditioning | list[str]):
res_indexes = []
prompt_indexes = {}
prompt_flat_list = SdConditioning(prompts)
prompt_flat_list.clear()
for prompt in prompts:
subprompts = re_AND.split(prompt)
indexes = []
for subprompt in subprompts:
match = re_weight.search(subprompt)
text, weight = match.groups() if match is not None else (subprompt, 1.0)
weight = float(weight) if weight is not None else 1.0
index = prompt_indexes.get(text, None)
if index is None:
index = len(prompt_flat_list)
prompt_flat_list.append(text)
prompt_indexes[text] = index
indexes.append((index, weight))
res_indexes.append(indexes)
return res_indexes, prompt_flat_list, prompt_indexes
class ComposableScheduledPromptConditioning:
def __init__(self, schedules, weight=1.0):
self.schedules: list[ScheduledPromptConditioning] = schedules
self.weight: float = weight
class MulticondLearnedConditioning:
def __init__(self, shape, batch):
self.shape: tuple = shape # the shape field is needed to send this object to DDIM/PLMS
self.batch: list[list[ComposableScheduledPromptConditioning]] = batch
def get_multicond_learned_conditioning(model, prompts, steps, hires_steps=None, use_old_scheduling=False) -> MulticondLearnedConditioning:
"""same as get_learned_conditioning, but returns a list of ScheduledPromptConditioning along with the weight objects for each prompt.
For each prompt, the list is obtained by splitting the prompt using the AND separator.
https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/
"""
res_indexes, prompt_flat_list, prompt_indexes = get_multicond_prompt_list(prompts)
learned_conditioning = get_learned_conditioning(model, prompt_flat_list, steps, hires_steps, use_old_scheduling)
res = []
for indexes in res_indexes:
res.append([ComposableScheduledPromptConditioning(learned_conditioning[i], weight) for i, weight in indexes])
return MulticondLearnedConditioning(shape=(len(prompts),), batch=res)
class DictWithShape(dict):
def __init__(self, x, shape=None):
super().__init__()
self.update(x)
@property
def shape(self):
return self["crossattn"].shape
def reconstruct_cond_batch(c: list[list[ScheduledPromptConditioning]], current_step):
param = c[0][0].cond
is_dict = isinstance(param, dict)
if is_dict:
dict_cond = param
res = {k: torch.zeros((len(c),) + param.shape, device=param.device, dtype=param.dtype) for k, param in dict_cond.items()}
res = DictWithShape(res, (len(c),) + dict_cond['crossattn'].shape)
else:
res = torch.zeros((len(c),) + param.shape, device=param.device, dtype=param.dtype)
for i, cond_schedule in enumerate(c):
target_index = 0
for current, entry in enumerate(cond_schedule):
if current_step <= entry.end_at_step:
target_index = current
break
if is_dict:
for k, param in cond_schedule[target_index].cond.items():
res[k][i] = param
else:
res[i] = cond_schedule[target_index].cond
return res
def stack_conds(tensors):
# if prompts have wildly different lengths above the limit we'll get tensors of different shapes
# and won't be able to torch.stack them. So this fixes that.
token_count = max([x.shape[0] for x in tensors])
for i in range(len(tensors)):
if tensors[i].shape[0] != token_count:
last_vector = tensors[i][-1:]
last_vector_repeated = last_vector.repeat([token_count - tensors[i].shape[0], 1])
tensors[i] = torch.vstack([tensors[i], last_vector_repeated])
return torch.stack(tensors)
def reconstruct_multicond_batch(c: MulticondLearnedConditioning, current_step):
param = c.batch[0][0].schedules[0].cond
tensors = []
conds_list = []
for composable_prompts in c.batch:
conds_for_batch = []
for composable_prompt in composable_prompts:
target_index = 0
for current, entry in enumerate(composable_prompt.schedules):
if current_step <= entry.end_at_step:
target_index = current
break
conds_for_batch.append((len(tensors), composable_prompt.weight))
tensors.append(composable_prompt.schedules[target_index].cond)
conds_list.append(conds_for_batch)
if isinstance(tensors[0], dict):
keys = list(tensors[0].keys())
stacked = {k: stack_conds([x[k] for x in tensors]) for k in keys}
stacked = DictWithShape(stacked, stacked['crossattn'].shape)
else:
stacked = stack_conds(tensors).to(device=param.device, dtype=param.dtype)
return conds_list, stacked
re_attention = re.compile(r"""
\\\(|
\\\)|
\\\[|
\\]|
\\\\|
\\|
\(|
\[|
:\s*([+-]?[.\d]+)\s*\)|
\)|
]|
[^\\()\[\]:]+|
:
""", re.X)
re_break = re.compile(r"\s*\bBREAK\b\s*", re.S)
def parse_prompt_attention(text):
"""
Parses a string with attention tokens and returns a list of pairs: text and its associated weight.
Accepted tokens are:
(abc) - increases attention to abc by a multiplier of 1.1
(abc:3.12) - increases attention to abc by a multiplier of 3.12
[abc] - decreases attention to abc by a multiplier of 1.1
\( - literal character '('
\[ - literal character '['
\) - literal character ')'
\] - literal character ']'
\\ - literal character '\'
anything else - just text
>>> parse_prompt_attention('normal text')
[['normal text', 1.0]]
>>> parse_prompt_attention('an (important) word')
[['an ', 1.0], ['important', 1.1], [' word', 1.0]]
>>> parse_prompt_attention('(unbalanced')
[['unbalanced', 1.1]]
>>> parse_prompt_attention('\(literal\]')
[['(literal]', 1.0]]
>>> parse_prompt_attention('(unnecessary)(parens)')
[['unnecessaryparens', 1.1]]
>>> parse_prompt_attention('a (((house:1.3)) [on] a (hill:0.5), sun, (((sky))).')
[['a ', 1.0],
['house', 1.5730000000000004],
[' ', 1.1],
['on', 1.0],
[' a ', 1.1],
['hill', 0.55],
[', sun, ', 1.1],
['sky', 1.4641000000000006],
['.', 1.1]]
"""
res = []
round_brackets = []
square_brackets = []
round_bracket_multiplier = 1.1
square_bracket_multiplier = 1 / 1.1
def multiply_range(start_position, multiplier):
for p in range(start_position, len(res)):
res[p][1] *= multiplier
for m in re_attention.finditer(text):
text = m.group(0)
weight = m.group(1)
if text.startswith('\\'):
res.append([text[1:], 1.0])
elif text == '(':
round_brackets.append(len(res))
elif text == '[':
square_brackets.append(len(res))
elif weight is not None and round_brackets:
multiply_range(round_brackets.pop(), float(weight))
elif text == ')' and round_brackets:
multiply_range(round_brackets.pop(), round_bracket_multiplier)
elif text == ']' and square_brackets:
multiply_range(square_brackets.pop(), square_bracket_multiplier)
else:
parts = re.split(re_break, text)
for i, part in enumerate(parts):
if i > 0:
res.append(["BREAK", -1])
res.append([part, 1.0])
for pos in round_brackets:
multiply_range(pos, round_bracket_multiplier)
for pos in square_brackets:
multiply_range(pos, square_bracket_multiplier)
if len(res) == 0:
res = [["", 1.0]]
# merge runs of identical weights
i = 0
while i + 1 < len(res):
if res[i][1] == res[i + 1][1]:
res[i][0] += res[i + 1][0]
res.pop(i + 1)
else:
i += 1
return res
if __name__ == "__main__":
import doctest
doctest.testmod(optionflags=doctest.NORMALIZE_WHITESPACE)
else:
import torch # doctest faster
'''Some more experimentation Pictures
Experimenting a little...Definitely needs some tweaking!
{tree:1.5, sky}:ocean:island:a serene ocean:0.4, cloudy lighting conditions,
Negative prompt: {low-hanging clouds}:{crazy tree roots, crazy branches}:0.6
Steps: 65, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 126387303976, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}
Started here:
{tree, sky}:ocean:island:0.4, cloudy lighting conditions,
Steps: 65, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 119007798841, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}
then in the next iteration...a crazy tree with crazy roots. Howebeit, when I tried to remove said craziness, it reverted to a view without a crazy tree
{tree, sky}:ocean:island:0.4, cloudy lighting conditions,
Negative prompt: low-hanging clouds,
Steps: 65, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 9240202417, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}
....hence the need to refine this prompt.
{tree, sky}:ocean:island:a serene ocean:0.4, cloudy lighting conditions,
Negative prompt: {low-hanging clouds}:{crazy tree roots, crazy branches}:0.6
Steps: 65, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 7, Seed: 92633545050, Size: 960x540, Model hash: 7be8cfbcd2, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.7, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 45, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "7be8cfbcd2"}