Sign In

Lighting based picture control controlnet

1.5k
12k
86
Updated: Jun 5, 2023
tool
Download (1.35 GB)
Verified:
SafeTensor
Type
Controlnet
Stats
11,537
Reviews
(1,472)
Published
May 30, 2023
Base Model
SD 1.5
Hash
AutoV2
F0524D0CCA
default creator card background decoration
LatentCat's Avatar
LatentCat

ioclab/control_v1u_sd15_illumination_webui · Hugging Face

非常抱歉在过去的一周里没有添加详细描述，由于工作时间原因，现在我将他补充在下方：

这是一个类似depth的controlnet，同时包含了从0到1阶段的光影构图可控的目的的controlnet。

  • I am very sorry that I did not add the detailed description in the past week. Due to working time, now I will add it below:

  • This is a controlnet similar to depth, but also contains a controlnet for the purpose of controlling the light composition from 0 to 1.

从0-1的光影构图可控 -- text to image

  • Controllable composition of light and shadow from 0-1 - text to image

如果你是一个画师，或者你可以使用ps等软件进行图片的移动和拼合，你可以产出这样的一张图片去描述你想要的构图，如下面这张图，我希望左边是一个类似聚光灯一样的视觉焦点，同时空气中漂浮着闪烁的光点

  • If you are a painter, or if you can use Photoshop or other software to move and combine images, you can produce an image that describes your desired composition, such as the one below, where I want to have a visual focal point like a spotlight on the left, with twinkling points of light floating in the air

将这张图作为controlnet的输入后，调整权重为0.6，退出时间为0.65（末尾会进行解释）输入提示词masterpiece, best quality, High contrast, dim environment, bright lights, 1girl, standing in the center of the stage, spotlight hit her, ((floating stars))

你可以得到这样的结果

  • With this graph as input to controlnet, adjust the weight to 0.6, Enter the prompt word masterpiece, best quality, High contrast, dim environment, bright lights, 1girl, best quality, high contrast, dim environment, bright lights, 1girl, standing in the center of the stage, spotlight hit her, ((floating stars))

  • You can get something like this

你会发现，相比于传统的生成图片，这样的方式似乎可以更明确的让画面的明暗关系更可控，那现在我们继续修改。

我喜欢右下角的构图，但是我希望女孩的光影层次更丰富，比如衣服上有反光，并想调整他的比例关系，通过ps或其他图片软件去色，高斯模糊，抠图的步骤后，你可以得到这样的图片。或者你可以继续按自己的意愿去重新绘画

  • As you can see, compared to the traditional image generation, this method seems to make the relationship between light and shade more clearly controlled, so let's continue to modify.

  • I like the composition in the lower right corner, but I want the girl to have richer light and shadow layers, such as reflecting light on her clothes, and I want to adjust her proportion relationship. After the steps of decolorizing, Gaussian blur and matting through Photoshop or other picture software, you can get such pictures. Or you can go ahead and repaint as you wish

在提供图片的时候，你可以选择比较模糊的图片，这样可以让ai去补足更多的细节

现在调整prompt和controlnet的输入，输出图片。这个过程中你可以使用lora，其他controlnet来加强你对画面的控制力，该controlnet在训练的时候保留了对HDR对比度的修改，让他不会过曝/过暗而损失图片的细节，你可以通过常见的noiseoffest等提高对比度的光影lora来配合使用，在这个过程中如果你发现了过拟合等现象，可以适当的调低controlnet的权重以及退出时间。

下面可以得到这样的图片：

  • When providing images, you can choose a blurry image, which allows the ai to fill in more detail

  • Now adjust the prompt and controlnet input to output the picture. For more control, you can use lora, controlnet, which has been trained to modify the HDR contrast so that it doesn't overdark or overdark the image. You can use the noiseoffest and other light loras to improve contrast. If you find overfitting and other phenomena during this process, you can appropriately lower the weight and exit time of the controlnet.

  • You can get a picture like this:

可以看到，floating star 这样带有明亮语义的词，被引导到了画面中最亮的亮点部分，你可以用这个原理去制作诸如 刀光 星星 水花 等过去难以引导的素材，请见最后的例子

如果你发现你生成的图片是这样的，人物并没有按你想要的出现在目标位置：

  • As you can see, "floating star", a word with a bright meaning, has come to be seen in the brightest parts of the picture, and you can use this principle to make things that used to be difficult to navigate, such as knife light, star splash, and so on. See the last example

  • If you find yourself generating an image like this, where the person doesn't appear where you want them to:

这个时候你可以选择继续roll，或者调整构图。但是，我们不是孤军奋战！这是一个controlnet，并且可以和其他的controlnet完美配合，而在过去的controlnet里有这样一个seg可以使用:(有点丑，但是ai可以理解)

  • At this point you can choose to continue roll, or adjust the composition. But we are not alone! This is a controlnet and works perfectly with other ControlNets, whereas in the past controlnet there was a seg that you could use :(ugly, but ai understood)

当你开启第二个controlnet的时候，你会发现人物出现在了正确的位置，按照这个逻辑，还可以与其他的controlnet来配合

  • When you open the second controlnet, you will see that the character appears in the right place and, by this logic, works with the other controlnet

现在你可以使用tile或者img2img来高清，也可以调高分辨率开启hires，更换种子，看看ai在t2i里生成什么样不同的结果，但你可以看到被保留的星星，左边的焦点，被照亮的头发和脖子。。。：

  • Now you can use tile or img2img for HD, you can switch to higher resolution hires, change the seeds, see what different ai produces in t2i, but you can see retained stars, left focus, lit hair and neck... :

更换一个ckpt尝试一下，会保持这样的关系吗？

答案是会！

  • Try replacing a ckpt, will this relationship be maintained?

  • The answer is yes!

通过该Controlnet，可以让AI在生成的时候意识到哪里应该是阴影，哪里应该是明亮的部分。

  • With the Controlnet, it is possible for the AI to realize where the shadows should be and where the bright parts should be when generating.

从1-1.5的光影调整

  • Light and shadow adjustment from 1-1.5

在这张示例图中，我超级喜欢这样的头发边缘光和背后的烟花，并想在我的ckpt上去尝试复刻这样的构图。但是你可能知道，让sd重新去roll的话，很可能会失去我想保留的构图，于是我们可以用ps扣除烟花并制作这样的输入

  • In this example image, I really liked the hair edge light and the fireworks in the back and wanted to try to recreate this composition on my ckpt. But as you probably know, having sd roll back would probably lose the composition that I wanted to keep, so we could use Photoshop to subtract the fireworks and make this input

并且由于有原图，我制作了这样的seg

  • And because of the original artwork, I made this seg

  • The following please witness!

是的，还是相同的构图，相同的头发边缘光，但是更大的烟花出现了！通过controlnet和SD本身对prompt的理解，烟花作为明亮的关系出现在了我们预期出现的位置！

更换底膜，查看更多的效果，而一切我们想保留的，都有保留：

  • Yes, same composition, same hair edge light, but bigger fireworks appear! With controlnet's and SD's own understanding of the prompt, fireworks appear as bright relationships where we expected them to be!

  • Replace the bottom film, see more effect, and everything we want to keep, there are reservations:

同样的，我可以给一张图添加新的光源和漂浮物

  • Similarly, I can add new light sources and floats to an image

我也可以给刀身加金属发光，并引导水花/刀光在其附近出现，在身后添加一个灯，你不需要绘制的太清晰，只需要一个意向。

  • I can also add a metal glow to the blade and direct the splash/blade light to appear near it, and add a light behind it, you don't need to draw too clearly, just an intention.

在其他的场景中，该controlnet也保留了语义，你可以做出这样的图：

  • In other scenarios, where controlnet preserves semantics, you can make a diagram like this:

如何使用：

  • How to use？

你只需要下载并将其放进和其他的controlnet同一个模型文件夹中，刷新模型列表即可，目前还没有制作统一的预处理器，如果你有建议欢迎评论。

  • All you need to do is download and put it in the same model folder as the other controlnet models and refresh the list of models. No unified preprocessor has been made yet, please comment if you have suggestions.

QA：