[Guide] Backing your content up to HuggingFace via Google Colab & Vast/Runpod

Preface

Originally Published Middle of 2023, many of the images are from SD 1.5 Models.

(The generative pictures included were using models that were uploaded via this tool, and are upcoming updates to Epic Mix's old 2.5D version from months ago.)

Re-written with aid of Chat GPT, so it's clearer. Be aware the COLAB edition may not be as updated as the Jupyter edition.

INTRODUCTION:

Let's dive into exploring this fascinating tool I've pieced together with inspiration from various sources. Keep in mind, I'm not exactly a seasoned programmer, so what you're seeing here is essentially a collection of patched content that I've put to the test!

RESOURCES:

The ACTUAL GITHUB Page for this: https://github.com/duskfallcrew/HuggingFace_Backup

Nocrypt's notebook: https://colab.research.google.com/drive/1wEa-tS10h4LlDykd87TF5zzpXIIQoCmq

Markdown lang tricks for HF cards: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

Earth & Dusk Discord: https://discord.gg/5t2kYxt7An

DOWNLOADABLES : https://civitai.com/models/99314/huggingface-backup-jupyter-python-and-colab

COLAB EDITION TUTORIAL

THIS IS MISSING THE 3.1 SECTION OF THE ORIGINAL CODE AS THAT ONLY WORKS WITH A WEBUI INSTANCE - THIS IS SEPARATED ENTIRELY FROM WEBUI.

Now, let's take a look at the Colab edition. You might notice that my description in the Colab code isn't the fanciest – emojis can only do so much to beautify things, right? But hey, I'm still figuring out how to jazz up a Colab to make it my own. (Feel free to share a good chuckle in the comments!)

Before we delve further, ensure you've grabbed the latest file if you've made a copy. Personally, I don't work off the main file to avoid accidentally sharing any personal info.

A quick note: This process requires minimal resources – no web UI is necessary, and you won't need any remote UIs apart from the optional file browser included.

You might encounter a prompt to copy it to Drive and customize it to your preferences. This step ensures that all your settings from previous sessions are remembered – a handy feature, right?

Once you've made a copy, hit that CONNECT button. Just a heads up, it only requires an internet connection – Google Drive is totally optional, as is the Elfinder plugin.

You'll probably want to start by mounting your Drive, but hold on a sec! Let me walk you through what else this Colab has in store before we get into the nitty-gritty.

Meet ImJoy Elfinder, a popular remote UI for the filesystem. Whether you're using Google Drive options or not, this tool can help you visualize your actions and upload new content. Please note, though, that its compatibility with Colab might have changed. This is unsure due to recent changes in the last year to Google Colab's Acceptable Use Policy terms.

Remember Nocrypt's notebook? It's one of the best for Colab users, and I'm happy to direct you there for explanations on the HF hub. Even though the 3.1 section might not be available here, this Colab still proves useful, as I discovered while updating some models.

Why am I sharing this code? Well, because I didn't write it! You can easily find similar code samples from Google or Elfinder's Colab samples and patch them together.

Now, onto how to use this Colab:

Make sure you have a Huggingface token – you can create one here.

Run cells 1, 2, and 3.

You'll need to run cell 1 first to log in to your HF account and the details on how this works is here:

Optional Drive Mount:

When you go to mount your drive, you'll see a prompt asking if you trust this action. It's completely up to you to decide. This code is widely used in millions of notebooks, and I didn't create it. Just so you know, every SD Web UI notebook using Google Drive mount has this step.

After you click "Yes, I trust this" to authorize access, you'll see the option to mount your drive. Once you've got your token, you can proceed with mounting your drive. (For some reason, mine stays mounted, but hey, we all have our quirks!)

Why am I sharing this code? Well, because I didn't write it! You can easily find similar code samples from Google or Elfinder's Colab samples and patch them together.

Essentially, you'll paste your Huggingface token in section 1 and set up your repository name there. In the next section, I'll show you what the clean, usable notebook looks like.

When you view the copy I've shared with this tutorial (linking and attaching the Colab file in the zip, not just images), you'll notice that section 1 is clean. In section 2, there's an optional name that you should ideally replace. For example, ours says "RUNPOD BACKUP."

Remember, the Huggingface filesystem doesn't allow spaces in names, so use underscores like "hf_backup."

Keep in mind that our names might have changed since the original tutorial was written in mid-2023. Both the Colab and Jupyter versions may now say something different.

If you prefer, you can manually use section 3.2. However, if you're comfortable with section 3.3, it's worth noting that it works similarly to 3.2 but is evidently more stable.

To use 3.3, you'll need to add your GitHub email and username. If you haven't completed sections 1 and 2, you'll encounter an error that looks like this:

Now once you've uploaded it might look like:

(This actually shows you what I last uploaded to HF via this)

Just a reminder, my setup doesn't support the old multi-upload via folder because the code is designed for a web UI-based instance. You can either fork my notebook or share your wisdom to help me patch it.

Update: The Colab version may or may not have this feature now, but the Jupyter edition does, thanks to Victorchall's Python Widgets.

Now, onto the next fun part – what does your repo look like on Huggingface?

Here's what your empty repository will look like. I haven't created a repo card for this yet, and it uses Markdown lingo. I won't delve into how that works here, but I'll link some resources, including Nocrypt's notebook and some other cool tools to accompany this, at the end.

Now, let's see what it looks like with files:

You might notice that I've retained Nocrypt's original line in the code because I borrowed it from his notebook and credited him extensively, to the point where the fork of this project doesn't really feel like mine anymore. If it's uploaded via my notebook, it says "YOINKED BY EARTH AND DUSK". Interestingly, you can actually change this in your notebook settings. It's not necessary, but it's a fun personal touch. This way, I can easily identify which notebook I'm using.

Note for 2024: Please bear in mind that naming conventions may have changed as the code has been updated.

VAST/RUNPOD ETC TUTORIAL:

YATTA! We have LIFT OFF! Your setup might look a bit different from Colab, and I swear we forgot to do this. So, for this one, you'll need to go into your repository afterward and APPROVE all the merge requests/pull requests. We seem to have missed those screenshots entirely, but I think I can improvise a bit.

DISCLAIMER:

IF YOUR INSTANCE LOOKS LIKE THIS AND YOU ARE ON A SERVERLESS GPU:

THEN SADLY IT MAY NOT WORK. I HAVE NO CLUE HOW TO PROPERLY SET UP AN ENVIRONMENT INSTANCE TO FUNCTION WITHOUT A PROXY ON VAST, AND I NEARLY RUINED MY ENTIRE SD SETUP ONCE WHILE TRYING. BE FOREWARNED, WE'RE MORE OF PATCHERS THAN PROGRAMMERS.

IF, HOWEVER, YOUR INSTANCE LOOKS LIKE THIS (ROUGHLY):

Then you're good to go (mostly). I can't guarantee it'll work, but press on regardless.

INSTALLATION STAGE ONE:

When you get to this stage, you'll see the first section:

BEFORE YOU HIT THE "PLAY" button that shows up, please make sure to add EXCLAMATION points before all the PIP commands, or it WILL NOT WORK.

As you progress, you'll start to see more of the code appearing in what feels like an "EXPRESSION OF INSTALL" mode. This is when the verbose code output begins to display, providing detailed information about each step of the installation process.

IGNORE THE RED LINE: If you see the disclaimer above and follow it, the process should still work in theory. I don't understand Python; I simply patched code from several notebooks. That being said, most of the time, the red line just indicates minor conflicts. If it DOES not work after that, please file a bug report in the GitHub repository or mention me in Civitai DM system.

STAGE TWO/ LOGIN:

You've probably seen this before, but if you haven't, this is where you'll need to make sure you go to your SETTINGS. In there, you'll find a section on the side that says "ACCESS KEYS."

These ACCESS keys are somewhat akin to API and secret keys, allowing the user to access the Hugging Face Hub. I won't show you my ACTUAL one, but I have it hidden so I can show you what a created one looks like here:

If you've already CREATED one perviously copy yours to the clipboard.

IF YOU HAVE NOT then see this picture:

Once you're in the ACCESS KEYS section, select the WRITE token option and give it a unique name. Remember, this token is sensitive information, so keep it private just like your SSN or Tax ID. It acts as an API/Secret key for accessing the Hugging Face platform.

NOW THAT YOU HAVE THAT let's move on to the next part:

Unfortunately, I forgot to capture the specific login box in a screenshot. However, once you reach this stage in the notebook, you'll encounter a login prompt with instructions. Simply paste your WRITE access key into the designated field and click the LOGIN button. If the key is accepted, you'll see a confirmation message similar to the one shown above.

STAGE THREE - REPOSITORY AND USERNAME:

Username: Add your username. Repository: This is the location of your MODEL folder, so it should be in the format Username/nameofmodelhere.

Next: If you're uploading a different file TYPE, make sure you change "CKPT" to the specific file type you're uploading. We advise against using this method for MASS BACKUP of image files because, last time we tried that, we ended up with a backup repo containing 600+ pics that couldn't be merged LOL. We mainly use SAFETENSORS, and you'll need to MANUALLY CHANGE IT EACH time you want to upload files (i.e., when you change the file type).

Our typical situation looks like this (*Our username and sample repo are included; we now largely upload to the E&D Org):

As of 2024: It no longer defaults to ckpt, but the section that says "SAFETENSORS" can be changed to upload just about anything, it's just -- don't do 300+ image files on a friday night because i have a repo that's still dying from 300+ images being uploaded.

AGAIN:

UPLOADING (FINAL STAGE):

(DO NOT NAME YOUR FILES LIKE WE DO. When you explore our backup repositories, you might encounter peculiar names such as "Aeronscocainfuelednightmare" and others.)

This is what you'll encounter in the notebook interface. Once you've switched to safetensors, all your larger models and/or Loras will become visible. (Remember to place this notebook in any folder you're backing content up for; it will only detect the content within the folder it's added to.)

Keep in mind that your internet speed is determined by the server you're on or your local connection.

PULL REQUEST

Although I can't provide a visual of the initial pull request action, I can demonstrate what it looks like afterward:

Within the repository details, you'll find a file hash, date details, and the user who uploaded it.

Beyond that, it's fairly straightforward.

ADDITIONAL INFORMATION:

If someone proficient in programming sees opportunities for improvement, feel free to make adjustments.

UPDATE:

Included in THIS ARTICLE (We'll update the model version and add the update to GitHub) - in the ATTACHMENTS SECTION is a "DIFFUSERS UPLOAD" version for those of you performing SDXL conversions.

Screenshot provided here:

We've removed the original zip file attached to this article, as most of the images in this are out of date, and are only provided as a reference. All updates for the notebooks are pulled via Github and the Civitai model repo. If you'd like a PDF edition of this, we'll try and find an updated copy or make one ASAP.