![Efficient, transparent and free file compression for your storage - [SSD/HDD, Windows]](https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/79f7cb2f-43b3-4394-a988-b3d195a1e182/width=1320/artium_v20_a_20241207193337_7_50_DPM++ 3M SDE_1194228643.jpeg)
If you, like me, download so many checkpoints and LoRAs that you'll probably see Flux 3 and Pony v31 before you've even had a chance to test your SD 1.5 files, this might interest you.
In this article, I explain how to free up space on your hard drive. In short, for those in a hurry:
Compression is transparent; your files are always available
You can save between 4% and 10% on a .safetensor file
This is not the "compress your file" option found in the "advanced" tab of a file's properties (don't use that !)
It requires (almost) no applications
Note: English is not my native language, so please take that into account (and point out any errors along the way).
Prerequisites:
Windows 10 minimum
Optional: Knowing how to run a shell script
How it works
This technique uses a command provided since Windows 10, but only accessible through the command line. It's the compact.exe command.
I won't explain how to use it here since Microsoft already does a great job of that: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/compact
Different compression modes are available:
XPRESS4K, very fast compression but not very efficient
XPRESS8K, the one recommended by default
XPRESS16K, the one I recommend
LZX, compresses better but is the slowest
When loading files, you don't notice a difference on a recent CPU because it decodes faster than the reader delivers the information. You can measure differences on very fast storage like NVMe SSDs (roughly speaking, at my place: 450GB/s instead of 560MB/s with a Ryzen 5700X... and a debatable measurement method).
On an SSD: you gain space
On an HDD: you can save time since the number of data to read is reduced (I didn't measure, so this is a theoretical remark)
Advantages:
Does not fragment the hard drive (on the contrary, compacting a fragmented file tends to defragment it)
Fast and efficient
Disadvantages:
Only works once: if you replace a file or download one into a directory that you have previously compacted, it will not be compressed itself.
Counterproductive with games using DirectStorage (for now, we're not concerned)
Does not work on already compressed files (.zip, .png, .jpg, .7z...) like any other compression application.
Note on the "compress file" option in the advanced properties table of a file
This method is part of the NTFS file system. I strongly advise against it, both on HDDs and SSDs, because it fragments files a lot. Moreover, it compresses poorly.
The CompactGUI Application
his application uses the compact.exe command, but packages it in a very easy-to-use interface. It therefore utilizes a system feature, which is more robust than an unknown algorithm developed in a back room.
It also offers to monitor your files to compress new arrivals (I don't use this functionality).
For its use, I let you read the author's page: https://github.com/IridiumIO/CompactGUI
Gain
I compressed my directories using the LZX method.
The gain on the checkpoint files is not phenomenal, but if you have a lot of checkpoints, you can save... to download others.
On the Loras, it's already better.
On the upscale models (which contain in addition to .onnx and .pth files), I gain 30%! (XPRESS16 method)
Compact works remarkably well on text files, but also on application folders (I recommend XPRESS8K for this type of folder).
Notes for owners of fast NVMe SSDs
If you have very fast storage, i.e., using the PCI-Express port, CompactGUI runs multiple threads and can make your PC unusable during compression, especially if the storage drive is also your system drive. Basically, the faster your storage and CPU are, the less usable it will be (!)
That's all I had to share.