THREADY’S TALK SHOW Hello, everyone! Today, I’m chatting with@theslysyl, a highly creative and interesting AI artist. As mentioned earlier, Sly runs AI locally on his machine using a sophisticated system. We’ll go into the details today and explore how it all works.Welcome to another episode of TTS!
Thank you for joining me today! I understand you have a busy schedule, so feel free to reply whenever it’s convenient for you. We can do this over the next few days until we finish our conversation. No pressure. I’ll start with the most basic question: Who is@theslysyloutside the Threads?
Syl: I’m many things, but mostly I’m a father and a bar manager at a local brewery who spends my little bit of free time reading or watching Sci-Fi/Fantasy and gaming. We also have 5 fish tanks totalling around 300 total gallons (1100 liters) of water and I dabble in other hobbies like hiking, gardening, music, tabletop RPGs, boardgames and quite a bit more.
I really like variety.
These days i’m mainly childcare during the day when my wife is working and then I work evenings.
Thready: Your background is incredibly interesting and versatile. Now, let me ask: do you have any programming skills or perhaps an interest in computer science? I wonder if a person needs education in that field to operate software like you do, or you simply followed steps from an online guide to set up your AI model?
Syl: I actually spent 6ish years in college bouncing around majors before deciding on computer science before I realized that coding all day wasn’t for me and dropping out to move cross country for a lady. My programming ability has mostly atrophied from those days in class but I can still whip up a few lines when I need too.
That’s partly why I like the Node interface of ComfyUI. It makes sense to me and I don’t need to do any coding to get everything working.
I custom built my workflow, but Other people did all the actual hard coding work. I just browse the ComfyUI Manager nodes and looked for things that intrigued me and integrated them into what I was previously using. I actually got a JSON of my workflow available in anticipation of this talk show and it’s available at My Model Merge + Lora Stack Workflow that upscales to 4k under 12GB RAM | Civitai
At least, I hope that link works!
Thready: At first glance, this appears to be a complex system for generating images. I believe you have full control over your output and put a lot of effort into it. But let’s discuss it in more detail later. In this segment, we’ll focus on you.I’d love to learn more about your interest in fish tanks. When did your passion for fishkeeping begin, and what types of fish do you currently have?
Syl: I’ve always loved nature, and growing up in Florida, nature always meant water to me. Not sure when I started watching nature documentaries but I’ve seen hundreds over the years and the weirdness and beauty of being under the water has always appealed to me.
I had a small fish tank when I was growing up. I actually kept turtles in it, part of where my icon came from. Now that I’m older and have space we acquired a bunch of fish tanks throughout the house
As to what fish? Dozens of types ranging from the classic neon tetras and goldfish (I have one goldy over a foot long), rainbowfish, plecos, otocinclus, gourami, tons of Corydoras subspecies, loaches, flying foxes, raspboras, freshwater shrimp… I could easily fill pages of information about my tanks.
Right now I exclusively keep fresh water as I don’t have time for the maintenance of a saltwater setup - and I also grow pothos out of most of my tanks, usually climbing up the walls behind them.
I find the tanks calming to watch.
Thready: Now, let’s discuss your AI art
How challenging was it to navigate the system you’re working with? The nodes thing remind me of Blender’s system where you assign textures to materials and perform similar tasks. I don’t know if you’re familiar with Blender, but I feel there are some similarities to their nodes system.How much control do you have over your outputs?
Syl: I’m sure if I dug into it straight now it would be pretty complex, but I started back with SD1.0 and a simpler GUI, then Automatic1111 when SD 1.5 came out and learned a bit more of the terminology and also what a wonderful resource CivitAI can be. I didn’t fully commit to ComfyUI until SDXL where I already knew how stuff like CFG, Steps, denoise patterns, alternate models and upscales etc. worked, and from there figuring out Model Merging, LORA stacking, Controlnet, inpainting and all the other bells and whistles wasn’t too much more to learn.
I personally take a “try everything and break stuff” approach so there was a LOT of experimentation as my node number grew. Especially once I saw how powerful Ultimate SD Upscaler could be. Though I rarely use them, with Controlnet and Inpainting my control could pretty much be absolute, to the degree that I could specify the exact location of object locations and image depth - but I kinda prefer the randomness.
When it comes to the actual subjects of the image, model merging And LORA stacking allow me to get a pretty well defined style and subject matter without even using a single prompt, so I can take an easy prompt and dramatically change the style simply by changing checkpoints and LORAs and how I weigh them.
This images prompt was a nonsensical couple of symbols, but because of models and LORAs I tuned the output towards a high contrast landscape with plants and water. The eclipse was a cool surprise, however.
Thready: I’ll be honest with you… I have no clue what you just said, but it sounds amazing!
From what I gather, your process isn’t just about prompting an image; you actually go through several steps to achieve the desired output.Could you explain it in simpler terms? How do you begin, and what adjustments do you make within your program to create such stunning, crisp images?
Syl: So, checkpoints, AKA models, are the main image resource. Different models are the difference between Dall-E, Midjourney, Stable Diffusion, etc. each one contains the data of billions of images, but even with the same prompt things turn out differently with different models.
What stable Diffusion, and running locally does, is allow community models. So there’s the base Stable Diffusion that has its billions of images and then people later added more images to its data in order to see the end result in a particular art style or subject.
These take time to create, but can be done locally with good enough hardware and if you spend the time curating the thousands+ images needed. Because of how training works, the end result is actually always the exact same size bit-wise, but the data inside and how it parses it is different.
These individual “tuned” models are already pretty big, and what I do is take two of them and combine them for every single image I generate. So if model A is really good for anime backgrounds and model B is really good for classical painting - I combine them and weigh the result towards 25% model A and 75% B, so I get anime subjects and boldness but also a classical painting style.
Then there are the LORAs.
LORA is “low rank adaption” and it it’s far more specialized. You use LORAs that can be tuned for a very distinct style, or even a single subject. They take as few as a few dozen pictures to train (though more is always better) and a certain prompt word or phrase and it makes that LORA show up in the final image pretty much 100% of the time. I have a LORA “stack” which involves choosing 3 LORAs (I’ve downloaded hundreds), weighing them like I do with the models, and then seeing where that goes.
So, if I want a steampunk castle in a forest at night, but anime style -
Model A: 60% Anime Style
Model B: 40% landscape / architecture
+
LORA A: steampunk 50%
LORA B: Ghibli Inspired style 30%
LORA C: high contrast night scene 20%
Then I make write the prompt, tweaking details and ratios as I go until I get to the output I want.
Once I get things looking just right and an output I’m happy with - I lean back and then queue up hundreds of images with different LORAs and models at random to see what comes out.
I let this run overnight and see what was generated in the morning.
I then do this again for the upscale. I take the base image, run it through a simple Upscaler to 3x the size (to 4k) and I have a secondary set of Model Merge + LORA stack and prompts - and then it automatically chops the image into a 3x3 grid, adds a tiny bit of noise and runs each 1/9th of the image through that secondary LORA+model merge adding new detail as though each 1/9th was full resolution. Then combines them back to the full 4k resolution and it’s done! Unless I want to fix stuff in post like the well known extra fingers.
…There’s actually still more going on. I trick the original generation into thinking it’s generating an image at 4k but is only generating at 1366x768 so it’ll generate finer details within my RAM limits, which I can then upscale later.
There’s some straight up magic with different samplers, the math behind the step method, how different CFG applies to different models. - Oh! CFG is basically the “strength of adherence” to the prompt - too low and none of the prompt shows up, too high and you get a mess. I have a node that does some behind the scenes programming magic to modify the CFG based on perceived adherence to the prompt and increases speed 3x by cutting out features in later steps. It’s a lot.
Thready: That’s really interesting and sounds like a lot of work. How much time do you need, approximately, to generate and render one image?
Syl: I use an RTX 3080 12GB and I can create a base image in as short as 10 seconds if I already have the models and LORAs in system memory. If it needs to re-merge things it takes about 30 seconds.
Upscaling to 4k is a process I’ve gotten down to around 90-120 seconds per image depending on various intricacies of the particular situation.
It takes some beefy hardware to switch models often, I’ve seen python using over 40GB system memory. I don’t have the VRAM to merge 3 models at once which is why I stick with 2.
Thready: I now have a better understanding of how you generate your art, and I find it quite impressive. Could you share how long you’ve been involved in AI art and when you discovered it? Also, which model did you start with?
Syl: I started with stable Diffusion 1.0 back in 2022.
It was rough and pretty ugly, I’ll be honest.I originally was generating stuff for the tabletop game that we were playing at the time, as our child wasn’t born yet and we still had the time for that sort of thing.
This is one of the earliest gens I can find as I’ve deleted most of my oldest by now. As I said, they weren’t very good.
Felt amazing at the time, however. Words into pictures! Even if vague and messed up, it was so cool!
Thready: How often does your output surpass your original idea? And how frequently do you encounter unexpected results, something beyond your initial vision, whether in a positive or negative way?
Syl: Sometimes. That example without a prompt I posted earlier definitely went beyond expectations, but sometimes when I’m trying to gen something heavily sci-fi or fantasy the models don’t quite understand it.
There’s a thing with every lantern looking the same that always haunts me and the want to create extra planets / moons in the sky of anything sci-fi. Also sometimes a LORA overpowers every other aspect and I get a beautifully composed image that has absolutely nothing to do with the prompt.
Thready: Could you share some of the works you’re most proud of? And what’s your favorite thing to work with: landscapes, portraits, abstraction… or do you create based on intuition and feeling?
Syl: Favorites is a really difficult one for me as it will constantly change.
I mainly work with landscapes because that’s just what I happen to enjoy aesthetically and I find generating people to really bring out the flaws in AI. I do generate them sometimes, but my story as of yet doesn’t have any actual characters as consistency is difficult.
It’s really about the vibes for me, and I go through an effort to avoid inconsistencies and photorealism.
Here’s a random selection that show off my styles.
Thready: This is just beautiful
Thready: let’s talk about Threads. Just like all of us, you’ve found a community here that supports your work and doesn’t judge you for creating AI art. Our community is incredibly friendly and supportive.However, let’s talk about some less pleasant aspects. Have you ever faced any negativity toward your art or engaged in arguments with the Anti-AI crowd?
Syl: A bit, not as much hate it seems as other members of our community but I make a point to try and be more educational than combative. I think the storytelling aspect of my gens helps with that as well. I’ve blocked a few haters though and the past few years online have taught me it’s usually best to avoid and block and move on than to engage.I don’t have the time or interest for arguments these days. Especially because this is a fun hobby for me and I’d like to keep it fun!
Thready: We should definitely maintain a healthy environment here. After all, we don’t truly require conflict. I believe individuals from that particular group lack education on the topic of AI initially, making it challenging to find common ground and engage in reasoned discussions with them. But let’s shift away from that topic for now.I’m curious, when did you join Threads, and what are your thoughts on its future? Do you think AI Threads will not only survive but also potentially grow in number?
Syl: I joined it pretty early on but It took me months to start posting. I was sharing my AI gens on Facebook and forums in bulk as I was generating them but only a few people found it interesting. So I decided to try my luck and start posting stuff here. I started following some other AI accounts and realized that I enjoyed telling stories to accompany my pictures and here we are.I think that Threads can only grow as Xitter, Blue sky, etc. fall.
I dread the day that ads come to threads.
Thready: We’re nearing the end of our conversation, and I have just a few remaining questions. Before we wrap up, I’d like you to mention some of your favorite AI creators within Threads who have inspired you and supported you on your AI journey on Threads, just to let them know you appreciate their work and support
Syl: Oh that’s a tough one. @artificialsweetener.ai @ai.missdeath @sipsigan @instag.amer @queenmosaik @marhalfwings @mary.lowd @digi_ai_creations @bhuddwha_studio @03_am_music @postmodern_iconoclast @nebulaasgard @sam.hologram @aylopop @fragments_of_noir @4e1st @sprinkleofai @fractus.barnsley @prime_generations @juliekarasik @karola.petersen
and I know there’s more but I think there’s a tag limit somewhere.
Thanks for this opportunity!