Tldr; Use the Lightning LoRA, not Lightning merged checkpoints.
In the ever-evolving realm of artificial intelligence, continuous advancements and refinements promise heightened efficacy and performance. Among these developments, the Lightning LoRA trend has recently attracted considerable attention. Some checkpoint finetuners have begun incorporating specific Lightning versions into their checkpoints. However, I've opted against doing so, much like my decision regarding Turbo and LCM. This article aims to elucidate my rationale for this choice and to illuminate potential drawbacks associated with this emerging trend within the broader ecosystem.
First and foremost, it's important to clarify that my reservations aren't directed at Lightning LoRA (and U-net) itself. Indeed, it can serve as an invaluable tool for swiftly conceptualizing ideas, with the intention of eventually refining them into finished works without the LoRA. However, the quick merging of Lightning LoRA with checkpoints warrants scrutiny. While superficially appealing, it's crucial to acknowledge that these merged versions lack further training. They rely solely on the existing augmentation of the LoRA, raising doubts regarding the necessity and effectiveness of such integrations.
Specifically, with such merged checkpoints, users forfeit the flexibility to adjust the LoRA's weight according to their specific needs. This limitation curtails the adaptability and versatility that standalone LoRA usage offers, hindering users' ability to refine outcomes effectively, and unable to smoothly go from concept to final artwork (lowering of weight progressively).
Moreover, the process of merging the Lightning LoRA with checkpoints and distributing both versions leads to unnecessary proliferation of checkpoint versions. This not only complicates versioning for CivitAI and its users but also contributes to unwarranted resource consumption for no added benefit. Furthermore, the abundance of version types confuses users, resulting in frequent errors as they inadvertently select incorrect settings on other checkpoints (and vice versa).
In essence, the integration of the Lightning LoRA into checkpoints warrants careful consideration, particularly regarding transparency. Unless the checkpoint is further trained specifically with the Lightning architecture in mind (a technical aspect beyond the scope of this discussion), merged Lightning checkpoints offer no real advantage over standalone LoRA usage. Given that none of the creators on CivitAI have undertaken this enhanced training, I somewhat have to question the purpose behind their released checkpoint versions.
Rather than simply merging the Lightning LoRA, creators should offer clear guidance on utilizing it separately with their checkpoint. This approach empowers users to make informed decisions, reduce confusion, and optimize the combination's potential without sacrificing flexibility.
In conclusion, while the allure of swiftly creating a Lightning version of checkpoints may be tempting for creators, it's imperative to critically evaluate the implications. Prioritizing transparency, efficiency, and user comprehension allows us to navigate the evolving landscape of AI model development responsibly and ethically. I hope that for future endeavors, particularly for SD3, creators will prioritize these considerations.