Preface 1: I'm using Windows, so I wouldn't know how to troubleshoot issues with Linux or other systems.
Preface 2: The code was created for personal use, without any claims.
Preface 3: It was helped by my friend (Franci) and I used CHATGPT a lot.
Preface 4: I do my best for the calculation of convergence, Over Training, and Over Fitting, but I couldn't improve it with my limited knowled
Preface 5: The code is completely open source, so use it and modify it as you wish. If you make significant improvements, please share them as a response to this message to help everyone.
What the code does:
The code start to calculate everything starting from second epoch (or after the warmup steps)
The code creates a graph and provides a terminal response with the following values while the training is in progress (or finished) - The data and graph update automatically every 10 minutes:
-------------------------------------------------
Data collected inside your JSON files:
- Total Epochs: 300
- Warmup Steps: 500
- Repetitions: 1.0
- Batch Size: 1
-------------------------------------------------
Results:
- Best step: (99612.0, 0.10412197560071945)
- Worst step: (110063.0, 0.15593218803405762)
- Best epoch: 83 - Mean Loss: 0.12
- Worst epoch: 78 - Mean Loss: 0.14
- Convergence Epoch: 203 (Step: 241976.0)
- Overtraining Epoch: 214 (Step: 255088.0)
- Overfitting Epoch: 223 (Step: 265816.0)
-------------------------------------------------
And this kind of Graph:
How to make it work:
Save the PreCog-EN.py inside the OneTrainer path C:\OneTrainer\Workspace (my OneTrainer is installed on C:, but your path may differ).
Start OneTrainer and begin the training.
Once started, use the command line to navigate to this path C:\OneTrainer\Workspace\tensorboard\
You should find folders named similar to these: 2024-05-22_09-04-48
Copy the complete path somewhere. Example: C:\OneTrainer\Workspace\tensorboard\2024-05-22_09-04-48
Navigate to the venv\script\ folder (example: C:\OneTrainer\venv\Scripts)
In the window, where the path is shown, delete the path, type CMD, and the terminal window will open.
In the terminal window, type: tensorboard --logdir C:\OneTrainer\Workspace\tensorboard\2024-05-22_09-04-48 --host=127.0.0.1
Of course, you need to replace 2024-05-22_09-04-48 with the name of your folder. A window will open, leave it open and return to OneTrainer.Optional) In the bottom right, click on the Tensorboard button, from which you can retrieve the exact path of the CSV file with the SMOOTH LOSS data in case you need to change it within the code.
(If you have executed step 9, minimize the window) Open Explorer, and go to the folder where you saved the PreCog-EN.py file (which should be C:\OneTrainer\Workspace)
At the top in the path bar, type CMD and press enter.
In the command prompt, type the name of the PreCog-EN.py and press enter.
Now it will ask you to enter the number of images in your DATASET, do it and press enter.
A window will open again where you will need to select the JSON file related to TRAINING (it will ask you on the prompt). Usually, you can find the file here: C:\OneTrainer\training_presets\name_of_your_preset.json
Another window will open where you need to select the JSON file related to the CONCEPT DATASET (it will ask you on the prompt). Usually, you can find the file here: I:\OneTrainer\training_concepts\name_of_your_concept_datset.json
Now it will ask which smoothing method to use to calculate the best convergence, over training and over fitting. You can choose between:
Savitzky-Golay (I suggest this one)
Moving Average (not very good)
Gaussian Kernel (I really don't know)
Done.
If you want help in the development of this code, the github discussion is here:
https://github.com/Nerogar/OneTrainer/discussions/319
To Download PreCog-EN:
https://silverider76.iliadboxos.it:27440/share/rC8xVK_qv6BbRjH3/PreCog-EN.py