If Stable Diffusion mysteriously becomes slower when you minimize its launcher, switch to your browser, or cover its window with another application, your GPU may not be the problem at all.

I ran into exactly this on Windows 11 with an Intel hybrid CPU (P-cores + E-cores).

The strange part was that generation became significantly faster whenever Stability Matrix was the foreground application. If I minimized Stability Matrix or switched to another window, generation slowed down.

After a lot of testing, the cause turned out to be much simpler than expected:

Windows was moving the Stable Diffusion Python workload from Intel P-cores to E-cores when the application went into the background.

Forcing the Stable Diffusion python.exe process to use P-cores only completely removed the foreground/background performance difference.

TL;DR

If you are running Stable Diffusion on:

• Windows 11

• an Intel CPU with P-cores and E-cores

• AUTOMATIC1111, Forge / Forge Neo, ComfyUI, Stability Matrix, or something similar

and generation becomes slower when the application is in the background:

Check which CPU cores python.exe is using.

In my case:

Foreground

python.exe → P-cores
GPU stays well fed
Fast generation

Background

python.exe → E-cores
GPU utilization drops
Generation slows down

Fix

python.exe → P-core affinity only

After doing that, generation stayed fast even while Stability Matrix was in the background.

The symptom

I originally noticed this with Stability Matrix.

With Stability Matrix visible and in the foreground, Stable Diffusion generated images noticeably faster.

If I:

• minimized Stability Matrix

• switched to the browser

• covered Stability Matrix with another application

generation became slower.

The difference varied with the workload, but I was seeing roughly 15–30% lower throughput in some tests.

The same behavior happened with both:

• Stable Diffusion WebUI

• Stable Diffusion WebUI Forge Neo

That immediately made a WebUI-specific bug less likely.

My Forge Neo environment used an RTX 4080 SUPER with PyTorch 2.11.0 + CUDA 13.0 and SageAttention 2.

First suspicion: the NVIDIA GPU

The obvious first thought was GPU power management.

Maybe NVIDIA was changing:

• P-state

• GPU clock

• VRAM clock

• power limits

when the application entered the background.

So I monitored the GPU with nvidia-smi.

What I saw was unexpected.

The GPU remained in P2 in both conditions. Memory clocks also stayed constant. GPU core clocks were not meaningfully lower in the slow condition.

But GPU utilization was very different.

During the slower runs, utilization frequently dropped into roughly the 60–70% range.

During the fast runs, the GPU was much closer to 95–100%.

So the problem did not look like:

GPU is downclocking
→ Stable Diffusion becomes slower

It looked more like:

GPU is waiting for work
→ utilization falls
→ Stable Diffusion becomes slower

That points upstream, toward the CPU side of the CUDA workload.

Process priority did not fix it

I tried changing the Python process priority.

No meaningful improvement.

I also tested several Windows scheduling-related ideas:

• process priority

• Windows High QoS / disabling execution-speed throttling

• foreground process flags

• Windows "Background services" processor scheduling

• heterogeneous scheduling policy settings such as "Prefer performant processors"

None of those eliminated the slowdown.

Microsoft does expose separate scheduling policies for heterogeneous processors. SchedulingPolicy and ShortSchedulingPolicy, for example, can prefer "performant processors" or "efficient processors", and their default Automatic mode allows Windows to choose based partly on QoS.

But merely setting Prefer performant processors was not enough in my case.

I removed Stability Matrix from the equation

At this point I wanted to know whether Stability Matrix itself was causing the problem.

So I launched Forge Neo's Python environment directly, without Stability Matrix.

The behavior still existed.

For example, in one controlled test:

PowerShell foreground:
~6.6 it/s

PowerShell minimized/background:
~5.5 it/s

So Stability Matrix was clearly not the fundamental cause.

The parent application's foreground/background state was somehow influencing the Python workload.

I also tested console output

Another possibility was terminal rendering or stdout handling.

Stable Diffusion writes progress output constantly, and Stability Matrix also captures package output.

So I redirected Python's stdout and stderr directly to files.

The slowdown still followed foreground/background state.

That ruled out console rendering as the main cause.

Then Task Manager gave away the answer

I switched Task Manager to the logical processor view and watched CPU activity while generating.

This was the important observation:

Stability Matrix in foreground

P-cores were active.

Browser in foreground / Stability Matrix in background

The workload moved toward the E-cores.

That immediately explained the rest of the data.

Intel Thread Director is specifically designed to provide Windows with runtime information so that threads can be assigned between Performance-cores and Efficient-cores. Intel explicitly describes the system as dynamically placing workloads on different core types, including keeping background work from interfering with foreground applications.

For normal desktop workloads, this is usually exactly what you want.

For Stable Diffusion, however, the Python process is not just some harmless background task.

It is feeding a very fast GPU.

Why can the CPU matter so much for a GPU workload?

Stable Diffusion inference is obviously GPU-heavy, but the GPU does not operate independently.

At a simplified level:

Python / PyTorch
      ↓
CUDA runtime
      ↓
kernel launches / synchronization
      ↓
GPU work queue
      ↓
NVIDIA GPU

If the host-side thread cannot submit work quickly enough, the GPU starts waiting.

So you can end up with this:

Fast P-core
    ↓
CUDA work submitted quickly
    ↓
GPU utilization ~95–100%
    ↓
Higher it/s

versus:

Slower E-core
    ↓
host-side submission becomes the bottleneck
    ↓
GPU queue develops gaps
    ↓
GPU utilization falls
    ↓
Lower it/s

This also explains why GPU clock monitoring initially pointed in the wrong direction.

The GPU was capable of running fast.

It simply was not always receiving work fast enough.

The definitive test: force Python onto P-cores

I opened:

Task Manager → Details → python.exe → Set affinity

and disabled the E-core logical processors, leaving only the P-core CPUs enabled.

Then I repeated the background test.

The slowdown disappeared.

Stability Matrix could be behind the browser or otherwise in the background, and Stable Diffusion continued running at the higher speed.

That was the strongest test of the entire investigation.

Changing generic process priority did not solve it.

Changing power/QoS settings did not solve it.

Changing heterogeneous scheduling preference did not solve it.

Actually preventing Python from running on the E-cores solved it.

Windows' SetProcessAffinityMask API does exactly this: it defines which logical processors the process is allowed to execute on. Microsoft also notes that process affinity is inherited by child processes.

This is not just my machine

After finding the cause, I searched for similar reports.

There are several.

AUTOMATIC1111

An old AUTOMATIC1111 performance discussion contains a particularly interesting Raptor Lake test.

The user explicitly separated the two CPU core types and measured:

P-core side:
39 it/s

E-core side:
27.5 it/s

They were surprised that CPU performance could make such a large difference in what appears to be a mostly GPU-bound workload.

That is a huge difference.

ComfyUI: almost exactly the same bug

A ComfyUI user reported in 2025 that when the ComfyUI terminal was out of sight, Windows 11 started scheduling the Python workload on E-cores.

Their workaround was the same one I independently ended up using:

Task Manager → python.exe → affinity → P-cores only.

They also complained about having to do it manually every time, which is extremely relatable.

There is an even earlier ComfyUI report from 2024 describing performance dropping when the console was minimized or completely covered by another window. Multiple users reported the foreground window running significantly faster, and one eventually disabled E-cores to avoid the behavior.

So this is not specific to Stability Matrix.

It appears to be a broader interaction between:

Windows 11 + Intel hybrid scheduling + Python-based GPU inference workloads.

The manual fix

First identify the python.exe process actually running your Stable Diffusion backend.

For Stability Matrix this will usually be somewhere under a package directory such as:

StabilityMatrix\Data\Packages\...\venv\Scripts\python.exe

Then:

1. Open Task Manager

2. Go to Details

3. Find the correct python.exe

4. Right-click it

5. Select Set affinity

6. Leave only the logical processors corresponding to your P-cores enabled

Do not blindly copy another person's CPU numbers.

Logical processor numbering depends on the CPU and system configuration.

On some Intel CPUs the P-core threads appear first, followed by the E-cores, but you should verify this on your own machine.

Do not disable E-cores globally unless you actually want to

One workaround mentioned online is disabling E-cores entirely.

That certainly prevents Windows from putting Stable Diffusion on them, but I do not think that is the ideal solution.

E-cores are useful.

The problem is not that E-cores exist.

The problem is that this particular latency-sensitive Python/CUDA workload should not be moved onto them while generating.

Process-level affinity is therefore a much cleaner workaround.

Automating it

The annoying part of Task Manager affinity is that it resets when the process restarts.

The practical solution is a small watcher that:

1. waits for Stability Matrix package python.exe processes
2. detects the system's P-core logical processors
3. applies a P-core-only affinity mask

Windows exposes CPU topology information through SYSTEM_CPU_SET_INFORMATION.

On heterogeneous processors, Microsoft defines EfficiencyClass such that higher numerical values represent faster but less power-efficient processors. That gives applications a way to identify the performant CPU class programmatically.

Windows also provides APIs for setting process affinity or default CPU Sets.

So this can be automated without disabling E-cores system-wide.

One caveat: systems with more than 64 logical processors require processor-group-aware handling rather than a simple 64-bit affinity mask.

How to tell whether you have the same problem

This issue is worth checking if all of these are true:

• You are using Windows 11

• Your Intel CPU has both P-cores and E-cores

• Stable Diffusion gets slower when its launcher or console loses focus

• GPU clock does not significantly drop

• GPU utilization drops when performance drops

• Bringing the launcher back to the foreground immediately improves speed

Open Task Manager and switch the CPU graph to:

Change graph to → Logical processors

Then generate once with the application in front and once with another application in front.

If the active workload visibly moves from the P-core region to the E-core region when backgrounded, you probably found the problem.

Finally, force python.exe to the P-cores only.

If the foreground/background performance difference disappears, that is a very strong confirmation.

What did NOT fix it on my system

For reference, I tried quite a few things before finding the actual cause:

• raising Python process priority

• NVIDIA power-state investigation

• NVIDIA clock monitoring

• forcing High QoS

• Windows processor scheduling = Background services

• SCHEDPOLICY = Prefer performant processors

• SHORTSCHEDPOLICY = Prefer performant processors

• foreground-process-related Windows flags

• redirecting stdout/stderr away from the console

• bypassing Stability Matrix and launching Forge directly

None of those solved the underlying foreground/background difference.

P-core affinity did.

Conclusion

This was a surprisingly deep rabbit hole for something that initially looked like a GPU problem.

The actual chain on my system was:

Stable Diffusion launcher goes into background
                ↓
Windows schedules Python onto Intel E-cores
                ↓
host-side CUDA/PyTorch work becomes slower
                ↓
GPU is not fed quickly enough
                ↓
GPU utilization drops
                ↓
image generation slows down

And the workaround is:

Force Stable Diffusion's python.exe onto P-cores

If you have been wondering why ComfyUI, AUTOMATIC1111, Forge, Forge Neo, or Stability Matrix suddenly becomes faster whenever you bring its window to the front, check your P/E-core scheduling before spending hours tweaking CUDA, PyTorch, NVIDIA drivers, or attention implementations.

It might not be your GPU at all.