Preface
I wrote an article about how to upscale images using Python together with Stable Diffusion [2]. In this article I am focussing on upscaling of images using OpenCV and Machine Learning. The related tool can be found at [1].
Motivation
In order to be become more independent from the web user interface AUTOMATIC1111, I was thinking about how to upscale images for my purposes on my own. This article shows one approach I am following.
Presumptions
I assume that I have images that have a resolution of only 512 x 512 pixels. It could be shown, that this is a good resolution for the upscaling approach.
Prerequisites
The user needs to be download some pretrained models from the given links in my documentation. Afterwards the web UI can be used.
Python Core Code
The Python core code example of how to upscale an image with OpenCV is:
#!/usr/bin/python3
# Import the Python modules.
import cv2
from cv2 import dnn_superres
# Initialise the super resolution object.
sr = dnn_superres.DnnSuperResImpl_create()
# Read the pretrained model.
path = 'LapSRN_x8.pb'
sr.readModel(path)
# Set pretrained model and scale.
sr.setModel('lapsrn', 8)
# Load the image
image = cv2.imread('vulture_original.jpg')
# Upscale the image.
upscaled = sr.upsample(image)
# Save the upscaled image.
cv2.imwrite('vulture_upscaled.jpg', upscaled)Web UI
To simply test different pretrained models I have built a web UI with the Python upscaler integrated. I used for this purpose Gradio.
Figure 1: Feel and look of the Lazy Image Upscaler web UI
I added the possibility to compare the Machine Learning results with standard methods OpenCV offers. I am quite impressed how good they work. Up to 4096 x 4096 pixel the standard methods gives good results. One has to keep in mind, that the standard methods are much faster in calculation than the Machine Learning Super Resolution approach.
Special Features
I have built a few features into the web UI that I need. The image can be flipped horizontal and vertical. Inverting an image is possible as well as converting an image to grayscale.
I figured out, that the upscaled image are felt to be a little bit blurry. Therefor I added a button, which sharpens the image a little bit.
I developed a bunch of Kernel for sharpening the images. Kernel 0 works for the moment the best way.
Example
Here come an example of an original and upscaled image with sharpening.
Figure 2: Image with 512 x 512 pixel
Figure 3: Upscaled image with 4096 x 4096 pixel
Comparison of Second Example
I check a second time, how good or how bad my GUI works. Together with sharpening I got quite good results. I have to work out a little bit more the sharpening kernels, to get good or better results depending on the desired needs.
Figure 4: Enlarged comparison of the same image in 1024 x 1024 pixel and 4096 x 4096 pixel
The improvement achieved is easier and much better to see on the screen. On the screen the right photo looks a little bit better than the left one. It is sharper while all details are preserved.
Remarks
At the moment I am limited to a maximum resolution of 4096 x 4096 pixel in upscaling. 512 x 512 pixel and 1024 x 1024 pixel could be used as original for upscaling.
The upscaling is done by the CPU. If OpenCV is able to use a GPU, it is in principle possible, to do the upscaling on a GPU.
To-Do
Improvement of the documentation of the web UI at GitHub. The web UI has to checked that it is more fail safe.
For the future it is planned that I try to let OpenCV work on a GPU. This is not my most urgent task at the moment, so it has been postponed indefinitely.
The work from today was quick and dirty programming. I have to sanitize and optimize the code.
Open Issue
I intended to use my upscaling tool for CIVITAI as well. For this it is necessary that the Exif metadata of my images is preserved. It seems that with Gradio it is not possible at the moment to get the Exif metadata directly and then write it.
Last But Not Least
I work a lot with square images. Nevertheless my web user interface works also well with other formats like landscape or portrait.
Figure 5: Upscaled landscape image from 2048 x 1152 pixel to 4096 x 2304 pixel using INTER_LANCZOS4
Figure 6: Upscaled portrait image from 512 x 720 pixel to 4096 x 5760 pixel using INTER_LANCZOS4
Update
My new local version of my Upscaler preserves now the Exif metadata. This way I can quite fast use images of higher resolution here. See [3] for examples.
Final Words
Have a nice day. Have fun. Be inspired!
References
[1] https://github.com/zentrocdot/LazyUpscalerGui
[2] https://civitai.com/articles/8776/upscaling-using-stable-diffusion