On March 11th, we launched one of the most ambitious updates in Civitai’s history: Full Checkpoint Coverage. We went from supporting 200 hand-picked models to opening the floodgates to over 25,000. This was a monumental step forward, made possible by months of preparation - scaling up our datacenter compute, overhauling our software and middleware, and implementing major infrastructure changes. We knew it wouldn’t be an easy transition, but we believed in the vision and did everything we could to ensure a smooth rollout.
Unfortunately, despite our best efforts, we’ve faced relentless service instability since launch. As of today, after exhausting every possible option to stabilize the system, we’ve made the difficult decision to roll back our changes. This means stepping away from Full Checkpoint Coverage - at least for now - and returning to a curated selection of the most popular models, chosen by Civitai staff.
What’s Rolling Back?
Rolling back Full Checkpoint Coverage means more than just limiting the number of supported models. We also have to roll back several other major features that were introduced alongside it:
Auction System for Model Featuring: This system, which allowed users to bid for featured, or "Boosted" model placement, will be temporarily removed. Note: Today's Auction winners will be refunded.
Popularity Temperature System: The dynamic temperature-based model popularity system will be reverted.
Prompt Parsing Changes: We will return to Automatic1111’s mean-normalized weight style for prompt parsing.
We will work to refine and reintroduce these features in a more stable and optimized way.
What Happened?
The challenge wasn’t a lack of preparation or foresight. Scaling to support tens of thousands of models is an incredibly complex problem, one that’s difficult to replicate in a testing environment. We anticipated growing pains, but the sheer volume of real-world traffic - tens of thousands of concurrent requests with unique model combinations - exceeded even our most cautious estimates.
On average, users were generating with 600 unique checkpoints at any given time, with peak moments reaching 1,000 unique models in active use. This was an incredible leap from the 200 we had before, and while it proved the demand is there, it also pushed our infrastructure beyond its limits.
Why Roll Back?
A partially working system isn’t good enough. We want Civitai to be a reliable platform where users can create without interruption. Right now, that means making the painful choice to scale back in order to provide stable, predictable service. While this decision stings, we believe it’s the best path forward.
The Future
Alongside these feature rollbacks, we will also be reverting some recent bug patches out of necessity. While this is not ideal, we want to assure our users that we are working hard to re-patch those issues as quickly as possible, ensuring a more stable and reliable platform moving forward. This is not the end of the road. The dream of full Checkpoint support is still alive, and we remain committed to finding a way to make it work. The insights we’ve gained from this rollout - what worked, what didn’t - will be invaluable as we rethink our approach.
We deeply appreciate your patience and support. We took a bold step, and while it didn’t pan out this time, we will return to this challenge with better data, better strategies, and a renewed commitment to making Full Checkpoint Coverage a reality in the future.
The rollback process has begun, and the Generator will become available when the update is complete and we have verified all systems are operational. There will be some disruption to site services while the rollback is in process.
The Generator will be online and offline intermittently during the rollback process, which may take some time!
Thank you for being part of this journey with us.