Finding the best background removal tool isn’t as simple as it seems. My plan was to try out the major background removal models available on Hugging Face, download the one that gave the best results, and run it locally using Rembg. However, I was surprised to discover that the pre-converted ONNX models provided by Rembg lead to quality loss and doubled processing times. This made me realise that a thorough test on background removal quality and processing speed across different models and formats was necessary.
It turns out that even small things like the script can cause subtle differences. For example, local models with my own scripts tend to have more noticeable edge artifacts. After testing, the winner for this specific anime-focused comparison was the BirefNet model combined with Rembg 2.0. You may find the script files I wrote and instructions in my previous articles.
I’ve also seen people mention discrepancies in quality between the online demo and offline models from hugging face community. (I solve this issue with the method at the end of this article.)
Testing Results From Online BiRefNet Models
Testing Results From Other Online Models
Testing Results From Local Models With Customised Scripts
Testing Results From Local REMBG Tool With ONNX Models
The testing image was generated using my lora Pastelflat
About realistic photos
There is no single best background removal model for realistic photos. I constantly have to switch between BiRefNet HDSOD, BiRefNet Matting-HR, BiRefNet Lite 2K, and RMBG 2.0 (in order of tendency from over-cut to under-cut, with RMBG 2.0 being the weakest of all), occasionally combining them with post-editing.
Mystery solved
After thoroughly comparing most mainstream background removal tools online and open-source models running on my MacBook, I found that BiRefNet generates the best quality results. However, images processed by models running on my script always have edge artefacts. I've seen similar issues discussed by other users online. Fortunately, I reached out to the brilliant founder behind BiRefNet and solved the mystery.
If you're familiar with the Select and Mask tool in Photoshop, there’s an option that lets you control the smoothness of the mask. When it comes to calculation, it all boils down to mathematics. Adding blur to the mask requires complex calculations, which slow down the background removal process. This is especially critical in areas like video chats with virtual backgrounds (Zoom, Snapchat filters, etc.), where high-efficiency edge detection is essential. Marco Forte from PhotoRoom simplified the calculation using new equations that focus only on the bright pixels. The Hugging Face online demos for RMBG and BiRefNet have implemented this method in their running scripts, so I need to ask Claude to learn from the demo scripts and fix mine. I’d like to express my appreciation to Peng Zhang and Marco Forte for sharing their brilliant works with us.
Approximate Fast Foreground Colour Estimation (Blurfusion)
This code requires two dependencies: numpy for mathematical operations and opencv-python for implementing the blur. Since numpy is already installed for the models, we only need to add opencv-python this time.
Open Terminal and activate your BiRefNet environment. I use multiple environments for different dependencies, so I install opencv-python one by one.
source /Users/mbp/Documents/bir_env/bin/activateInstall opencv-python:
pip install opencv-python
Viola! The preparation part is easy. Now, I need to update all my quick action scripts. I'll upload them again once it's finished.
Here is the preset prompt I use for Claude to fix my script with Blurfusion at one shot, as I said don't expect AI can create proper script without rigorous prompt.
Add Fast Foreground Colour Estimation to background removal script and return me the full finished script.
1. Add this configuration at top of bash script:
```bash
# --- CONFIGURATION ---
# Refinement Radius - controls edge refinement intensity (higher = stronger effect)
REFINEMENT_RADIUS=90
# --- END CONFIGURATION ---
```
2. Add dependency check before running the Python script:
```bash
"$PYTHON_PATH" -c "import numpy; import cv2" &> /dev/null
if [ $? -ne 0 ]; then
echo "ERROR: Required Python dependencies missing (numpy or opencv-python)"
echo "Please install with: pip install numpy opencv-python"
exit 1
fi
```
3. Modify your Python script call to pass the refinement radius:
```bash
"$PYTHON_PATH" "$python_script" "$input_image_path" "$output_path" "$MODEL_PATH" "$REFINEMENT_RADIUS"
```
4. Add these universal refinement functions to your Python script:
```python
import numpy as np
import cv2
from PIL import Image
# Fast Foreground Colour Estimation Functions
def FB_blur_fusion_foreground_estimator(image, F, B, alpha, r=90):
"""
First-level foreground estimator using blur fusion
Parameters:
image: Input image (numpy array or PIL Image)
F: Foreground estimate
B: Background estimate
alpha: Alpha mask (0-1 range)
r: Blur radius
"""
if isinstance(image, Image.Image):
image = np.array(image) / 255.0
blurred_alpha = cv2.blur(alpha, (r, r))[:, :, None]
blurred_FA = cv2.blur(F * alpha, (r, r))
blurred_F = blurred_FA / (blurred_alpha + 1e-5)
blurred_B1A = cv2.blur(B * (1 - alpha), (r, r))
blurred_B = blurred_B1A / ((1 - blurred_alpha) + 1e-5)
F = blurred_F + alpha * (image - alpha * blurred_F - (1 - alpha) * blurred_B)
F = np.clip(F, 0, 1)
return F, blurred_B
def FB_blur_fusion_foreground_estimator_2(image, alpha, r=90):
"""
Two-stage foreground estimator for better quality
Parameters:
image: Input image (numpy array, values 0-1)
alpha: Alpha mask (numpy array, values 0-1)
r: Initial blur radius
"""
alpha = alpha[:, :, None]
F, blur_B = FB_blur_fusion_foreground_estimator(image, image, image, alpha, r)
return FB_blur_fusion_foreground_estimator(image, F, blur_B, alpha, r=6)[0]
def apply_refinement(original_image, mask, refinement_radius=90):
"""
Apply edge refinement to any segmentation mask
Parameters:
original_image: PIL Image or path to image
mask: PIL Image or numpy array with alpha values (0-255 or 0-1)
refinement_radius: Radius for refinement effect
Returns:
Refined image with alpha channel
"""
# Load image if path is provided
if isinstance(original_image, str):
original_image = Image.open(original_image).convert('RGB')
# Ensure images are PIL Images
if not isinstance(original_image, Image.Image):
original_image = Image.fromarray(np.uint8(original_image * 255) if original_image.max() <= 1.0 else np.uint8(original_image))
if not isinstance(mask, Image.Image):
if mask.max() <= 1.0:
mask = Image.fromarray(np.uint8(mask * 255))
else:
mask = Image.fromarray(np.uint8(mask))
# Convert to numpy arrays for refinement
img_np = np.array(original_image) / 255.0
mask_np = np.array(mask) / 255.0
# Apply refinement
refined_fg = FB_blur_fusion_foreground_estimator_2(img_np, mask_np, r=refinement_radius)
# Create final image with refined edges
refined_image = Image.fromarray((refined_fg * 255.0).astype(np.uint8))
refined_image.putalpha(mask)
return refined_image
```
5. Use the refinement in your model processing flow:
For BiRefNet:
```python
# Run the model to get the mask
with torch.no_grad():
preds = birefnet(input_images)[-1].sigmoid().cpu()
pred = preds[0].squeeze()
pred_pil = transforms.ToPILImage()(pred)
mask = pred_pil.resize(original_image.size)
# Apply refinement
refinement_radius = int(sys.argv[4]) # Get from script args
refined_image = apply_refinement(original_image, mask, refinement_radius)
refined_image.save(output_path)
```
For rmbg or other models:
```python
# Run the model to get the mask
mask = model.predict(image_path) # Your model's prediction method
# Get original image
original_image = Image.open(image_path).convert('RGB')
# Apply refinement
refinement_radius = int(sys.argv[4]) # Get from script args
refined_image = apply_refinement(original_image, mask, refinement_radius)
refined_image.save(output_path)
```