Sign In

What's the deal with Illustrious?

8

Jan 5, 2025

(Updated: a day ago)

musing
What's the deal with Illustrious?

I see this question coming up over and over again: Illustrious seems to be new hotness, but how is it better than the status quo and Pony in particular? This is not meant to be a guide on how to make the pretty animu pix with Illustrious, nor a scientific or technical drilldown on the relative strengths and weaknesses, but a high level overview on how Illustrious feels to use and why it's worth it to check it out in the first place. So, here we go:

What Illustrious is

Illustrious is an SDXL-based model primarily geared towards generation of anime-style images. You can read the paper if you want but the tl;dr is that it's something like Pony + source_anime, but built from the ground (well, SDXL) up for that task. People liked what it does, so now Illustrious is more properly thought of as a whole family of models, just like Pony and its derivatives. Also, just like Pony, you should never be using the base model unless you are definitely sure you need to. The finetunes exist for a reason and are generally aesthetically superior. Because of this similarity, Illustrious can be trivially substituted for Pony in almost all workflows (Pony prompts will work with at most minor alterations, you just can't use all LoRAs and the ideal CFG is usually lower) - and its hardware requirements are also roughly the same, very unlike more technically advanced models.

What Illustrious is not

Illustrious isn't a next generation model like Flux or SD3 and still has all of the limitations of an SDXL model, especially in regard to prompting structure (tags). It still works a lot like Pony because it still is a lot like Pony. In particular, its results are not significantly superior to a properly pimped out Pony workflow, the difference is mostly in how much less of a hassle it is to get there. To put it bluntly: Don't expect more than a 1girl machine with benefits, more on that later.

The actual Benefits of Illustrious

  • Prompt adherence: This is the big one. Illustrious has a cleaner CLIP library and will just listen to your prompt better, and its vocabulary is larger in the field it is specialized in. This alone is the reason a lot of people would pick it over Pony - if nothing else, you will need fewer tries to get the image you're thinking of.

  • Better anatomy: It's better at hands, much better at feet and generally has an idea about how many limbs humans are supposed to have. Its signature mistake is to draw four instead of five toes, which is really mild if you know what kinds of body horror Pony can get up to.

  • Danbooru tags: Illustrious is trained on a more or less unfiltered 2023 Danbooru dataset, which means you can reliably use Danbooru's wiki as a dictionary on what to hand to Illustrious, while with Pony it's an unavoidable crapshoot. In particular, this includes characters, IPs and artists, all things that Pony will most likely need a LoRA for. Finding and using LoRAs isn't exactly hard but it's still something you don't have to bother with as much, never mind that stacking a dozen LoRAs can have a deleterious effect on image quality.

  • More creative: It's hard to say whether that's a benefit of base Illustrious or finetunes, but in my experience Illustrious is more likely to surprise me with an unusual or dynamic composition than Pony is. Same goes for backgrounds, Illustrious loves them so much it actually struggles to give you a single color sometimes.

  • Properly good at anime: Illustrious does not care about photos, ponies or furries, which is a very good thing if you don't want photos, ponies or furries. Base Illustrious will give you pretty anime pictures with a minimum amount of fuss, and finetunes can be exceptionally pretty even without quality tags or negatives. This also means that stylistically questionable concept LoRAs (e.g., ones trained on photos) have far less of a negative effect on the output because Illustrious will plaster over that with its own style much more than Pony does.

  • Clean interface: None of this score_ nonsense or magic negative prompts handed down from your grandparents. Tell it to make something pretty and tell it to avoid things you don't want to see. It's not that that stuff can't make a difference, but it's not strictly necessary at least.

Downsides of Illustrious

  • LoRA library: It doesn't have nearly as many LoRAs available as Pony has and that's a fact. As said before, it also needs them much less desperately than Pony does, but sometimes you'll still find stuff that Illustrious doesn't know and the relevant LoRAs are Pony-only. Illustrious has gained a lot of traction and its library is growing, but that won't always help you much if you want something out of the ordinary.

  • Stylistic preferences: Not only does it only do anime, it also has some pretty strong opinions on what anime is supposed to look like, which can be hard to overcome if you want something more unusual-looking. Same goes for composition, while it does seem to be capable of more creativity than Pony, more often than not it's also hard to break it out of its fundamental "one bright blob in the upper center of the image" portrait composition. And sometimes you do want a plain white background. In short, it's less of a blank slate than Pony is (and this is already a problem with Pony).

  • 1girl machine: It's still SDXL, don't expect to be able to show two characters and specifically assign them different hair colors, or to have one character wear glasses and not the other. Due to its generally stronger prompting, I found it somewhat less random and more predictable than Pony, but it's still cold comfort if it means predictably wrong a lot of the time. And there isn't much you can do about it short of making character LoRAs or inpainting.

A final word on LoRAs

Since a lot of the downsides come down to "but I need this LoRA", it should be said: Pony/SDXL LoRAs sometimes work with Illustrious just fine - unless you're on the CivitAI online generator, which won't allow this. In my experience, the image generation aspect works most of the time, while LoRAs that require prompting are likely to not work, probably because of the different CLIP schemes. If you can, it might be worth a try to just YOLO it. If Illustrious can generate a basic image element that the LoRA can then refine without being prompted to, it might just pay off.

8