Segmentation colors for controlnet
https://huggingface.co/datasets/huggingface/label-files/blob/main/ade20k-id2label.json
- (0) wall #787878
- (1) building #b47878
- (2) sky #6e6e6
- (3) floor #503232
- (4) tree #4c803
- (5) ceiling #787850
- (6) road #8c8c8c
- (7) bed #cc05ff
- (8) windowpane #e6e6e6
- (9) grass #4fa07
- (10) cabinet #e005ff
- (11) sidewalk #ebff07
- (12) person #96053d
- (13) earth #787846
- (14) door #8ff33
- (15) table #ff0652
- (16) mountain #8fff8c
- (17) plant #ccff04
- (18) curtain #ff3307
- (19) chair #cc4603
- (20) car #66c8
- (21) water #3de6fa
- (22) painting #ff0633
- (23) sofa #b66ff
- (24) shelf #ff0747
- (25) house #ff09e0
- (26) sea #907e6
- (27) mirror #dcdcdc
- (28) rug #ff095c
- (29) field #7009ff
- (30) armchair #8ffd6
- (31) seat #7ffe0
- (32) fence #ffb806
- (33) desk #aff47
- (34) rock #ff290a
- (35) wardrobe #7ffff
- (36) lamp #e0ff08
- (37) bathtub #6608ff
- (38) railing #ff3d06
- (39) cushion #ffc207
- (40) base #ff7a08
- (41) box #ff14
- (42) column #ff0829
- (43) signboard #ff0599
- (44) chest of drawers #633ff
- (45) counter #eb0cff
- (46) sand #a09614
- (47) sink #a3ff
- (48) skyscraper #8c8c8c
- (49) fireplace #fa0a0f
- (50) refrigerator #14ff00
- (51) grandstand #1fff00
- (52) path #ff1f00
- (53) stairs #ffe000
- (54) runway #99ff00
- (55) case #ff
- (56) pool table #ff4700
- (57) pillow #ebff
- (58) screen door #adff
- (59) stairway #1f00ff
- (60) river #bc8c8
- (61) bridge #ff5200
- (62) bookcase #fff5
- (63) blind #3dff
- (64) coffee table #ff70
- (65) toilet #ff85
- (66) flower #ff0000
- (67) book #ffa300
- (68) hill #ff6600
- (69) bench #c2ff00
- (70) countertop #8fff
- (71) stove #33ff00
- (72) palm #52ff
- (73) kitchen island #ff29
- (74) computer #ffad
- (75) swivel chair #a00ff
- (76) boat #adff00
- (77) bar #ff99
- (78) arcade machine #ff5c00
- (79) hovel #ff00ff
- (80) bus #ff00f5
- (81) towel #ff0066
- (82) light #ffad00
- (83) truck #ff0014
- (84) tower #ffb8b8
- (85) chandelier #1fff
- (86) awning #ff3d
- (87) streetlight #47ff
- (88) booth #ff00cc
- (89) television receiver #ffc2
- (90) airplane #ff52
- (91) dirt track #aff
- (92) apparel #70ff
- (93) pole #3300ff
- (94) land #c2ff
- (95) bannister #7aff
- (96) escalator #ffa3
- (97) ottoman #ff9900
- (98) bottle #ff0a
- (99) buffet #ff7000
- (100) poster #8fff00
- (101) stage #5200ff
- (102) van #a3ff00
- (103) ship #ffeb00
- (104) fountain #8b8aa
- (105) conveyer belt #8500ff
- (106) canopy #ff5c
- (107) washer #b800ff
- (108) plaything #ff001f
- (109) swimming pool #b8ff
- (110) stool #d6ff
- (111) barrel #ff0070
- (112) basket #5cff00
- (113) waterfall #e0ff
- (114) tent #70e0ff
- (115) bag #46b8a0
- (116) minibike #a300ff
- (117) cradle #9900ff
- (118) oven #47ff00
- (119) ball #ff00a3
- (120) food #ffcc00
- (121) step #ff008f
- (122) tank #ffeb
- (123) trade name #85ff00
- (124) microwave #ff00eb
- (125) pot #f500ff
- (126) animal #ff007a
- (127) bicycle #fff500
- (128) lake #abed4
- (129) dishwasher #d6ff00
- (130) screen #ccff
- (131) blanket #1400ff
- (132) sculpture #ffff00
- (133) hood #99ff
- (134) sconce #29ff
- (135) vase #ffcc
- (136) traffic light #2900ff
- (137) tray #29ff00
- (138) ashcan #ad00ff
- (139) fan #f5ff
- (140) pier #4700ff
- (141) crt screen #7a00ff
- (142) plate #ffb8
- (143) monitor #5cff
- (144) bulletin board #b8ff00
- (145) shower #85ff
- (146) radiator #ffd600
- (147) glass #19c2c2
- (148) clock #66ff00
- (149) flag #5c00ff(for controlnet1.1, COCO dataset colors are also supported, maybe this: https://github.com/SHI-Labs/OneFormer/blob/main/datasets/panoptic_coco_categories.json)
Example: hanafuda / 花札 / 고스톱
Make hanafuda/gostop cards like: https://www.behance.net/gallery/3110341/Hanafuda
colors used: tree, flower, poster, animal
Controlnet Enable X, no preprocessor, control_seg-fp16
Prompt: cherry blossom branches and flowers, a red poster with chinese characters and a bird
(mistake: it should be plum flower, not sakura flower, also the banner color is wrong, also wrong characters, also wrong bird ...)
Still need to divide into 4 cards.
Colors and controlnet-seg
As noted by seghierdesign, the colors where wrong in the original article. Here is the original control:
And the result:
As can be seen, the original colors were wrong (shift by one) but gave the correct result. So the original color table was literally 'not even wrong'.
Let's try some other colors:
Correct colors:
Incorrect colors 1 (color shift by 90 degrees)
Incorrect colors 2 (color shift by 180 degrees)
Very surprisingly, the first two give 'approximately correct' results (tree and bird at their designated places) while the third one cannot place the bird on the right. The only explanation that I can see is that the first two control images have green on the tree and flowers (and both ADE20k and COCO colors have green in flower/trees) and on the first two control images the bird is red/yellow (and both ADE20k and COCO have a red-ish segmentation color for animals/birds). Since this is not the case with the last one, a tree/flower is placed on the right instead of the bird because of the green color. This would also explain why the first control with incorrect colors worked (green flowers, purplish-kinda-red-ish bird).
So it seems that the blobs of colors give hints that there should be an object distinct from its background occupying all the designated space (compared to controlnet-scribble that just draws the outline). The color of the blob and the prompt give a hint to what is there (with stronger hints when ADE20K and COCO colors match or are similar). Since controlnet operates in latent space, it is hard to tell what 'similar colors' mean in this space, and you'd have to look at all COCO and ADE20k colors at once to determine what are possible color ambiguities.
Next level
Give one prompt for each part of image.
It gets boosted in the cross attention mapping.
https://github.com/cloneofsimo/paint-with-words-sd
(EDIT: the github for omost has a list of possible methods for putting certain objects in certain parts of the image https://github.com/lllyasviel/Omost )