Sign In

ComfyUI - Multi-Model Merge and Gradient Merges

55
575
6
Updated: Oct 6, 2024
toolcomfyuiworkflownodesmerge
Type
Workflows
Stats
575
Reviews
Published
Sep 14, 2023
Base Model
SDXL 1.0
Hash
AutoV2
856D2957C2
default creator card background decoration
Holiday 2023: 5 lights
Akatsuzi's Avatar
Akatsuzi

These workflow templates are designed to help people get started with merging their own models.

The templates are intended for intermediate and advanced users of ComfyUI.

Multi-Model Merge and Gradient Merges

The model merging nodes and templates were designed by the Comfyroll Team with extensive testing and feedback by THM.

The templates have the following use cases:

  • Merging more than two models at the same time

  • Fine tuning model merges

  • Previewing graduated sets of images to evaluate which ratios or tuning settings produce the best results

  • Using model merges in normal workflows without saving a checkpoint model

Detailed notes on the merge methodology and operation of the nodes can be found at the bottom of the post.

List of Templates

  • Ultra-Simple Model Merge

    • for SDXL or SD1.5

  • Simple Model Merge

    • for SDXL or SD1.5

  • SDXL Model Merge

  • SDXL Gradient Model Merge

It is planned to add the following:

  • SD1.5 templates

  • More tuning options

  • More gradient scenarios

    • including gradient tuning

  • Load merge data from a file directly to the CR Apply Model Merge node

  • Simple GIF Maker

    • For previewing the results of gradient merges in an animation

Installation

Nodes:

CR Model Merge Stack

  • lists the models for merging, plus the model and clip ratios for each model

CR Apply Model Merge

  • a recursive loop applies each model in the stack in turn based on the above ratios

CR Gradient Float

  • linear interpolation between ratio values 0 and 1

Installation and Setup

Please see our CivitAI article

Installation and Setup Guide

Troubleshooting

Please see our CivitAI article

Troubleshooting Guide

On first use

  • Select the models and VAE

    • do not try mixing SD1.5 and SDXL models

  • Select an upscale model

  • Add LoRAs or set each LoRA to Off and None

  • Set the filename_prefix in Save Checkpoint

    • this will be the prefix for the output model

  • Set the filename_prefix in Save Image to your preferred sub-folder

  • Set control_after_generate in the Seed node to fixed

  • Do a test run

  • Save a copy to use as your template

Tips

  • You can add the merge model nodes at the start of any template replacing your normal load checkpoint node

  • Stacks can be extended by adding more stacked model nodes chained to the first stack node. In tests we have successfully merged 9 SDXL models using this method, but theoretically you could merge a lot more.

  • The merged models and txt files are saved to folders under \ComfyUI\output

Ultra-Simple Model Merge Template Features

  • ultra-simple design

  • SDXL or SD1.5

  • slots for up to 3 models

  • additional models can be added by adding more stack nodes

Simple Model Merge Template Features

  • simple design

  • SDXL or SD1.5

  • slots for up to 6 models and 6 LoRAs

  • outputs merge info to a file

  • additional models and LoRAs can be added by adding more stack nodes

See Stacked Model Methodology section below for methodology used

SDXL Model Merge Template

  • slots for up to 6 models and 6 LoRAs

  • image preview for the merged model

  • additional models and LoRAs can be added by adding more stack nodes

SDXL Gradient Model Merge Template

  • slots for up to 6 models and 6 LoRAs

  • gradient merging between the outputs of the two model stacks

  • image preview for the merged model

  • additional models and LoRAs can be added by adding more stack nodes

Multi-Model Methodology

The final ratios in the mix will depend on the merge_method selected.

It is important to note that actual ratios used in the final model will differ from your input ratios (see Maths section below). This is because recursive loops tend to bias the first model and then taper ratios of subsequent models.

Recursive Methods:

  • Recursive Method

    • The output of merging the first two models becomes the input for merging with the next model. This is then repeated in a loop until all models are merged.

  • Weighted Method

    • This is the same as the Recursive method but with additional adjustments applied to model and clip ratios

    • If the weight_factor is set to 1, it will produce the same results as the Recursive method

    • If the ratio is less than 1 then the ratios for each model will be adjusted according to a formula to reduce first model bias and reduce tapering on subsequent models

Additional options:

  • normalise_ratios

    • Default = 'Yes'

    • If the total of the ratios is greater or less 1 then the values will be automatically adjusted so that they add up to 1

  • weight_factor

    • Default = 1

    • This only works on the Weighted method

    • The ratio of each model is adjusted according to a formula to reduce first model bias and reduce tapering on subsequent models

Tuning

The weight factor allows you to fine tune merges without the need to adjust the ratios configured on each model. It is planned to add more tuning options.

An example of tuning using these nodes can be found in this article by THM
https://civitai.com/posts/595572

The method used in this article is not included in the current version of the nodes, but may be added as an option in the next release.

Maths

A recursive merge of three models using ratios of 0.33 / 0.33 / 0.33 will produce a final mix of 0.45 / 0.22 / 0.33.

A four model merge using ratios 0.25 / 0.25 / 0.25 / 0.25 will produce a final mix or 0.42 / 0.14 / 0.19 / 0.25.

This is because the recursive method biases the first model, and then reverse tapers subsequent models.

The weight_factor boosts the second model, and this has a cascade effect on subsequent models, excluding the last model. It also decreases the first model bias.

Credits

https://github.com/LucianoCirino - for the original stack concept

THM - for testing the nodes and for the cover picture