RTX 5070 Laptop Blackwell SM120: xFormers + MSLK 1.3 on Windows, Now Validated for Real Diffusers and Hires

Before You Read

This is a technical post for users trying to run xFormers on newer NVIDIA Blackwell GPUs under Windows.

You may find this useful if:

• you own an RTX 50-series GPU;

• xFormers does not install or run correctly;

• your frontend falls back to another attention implementation;

• xFormers reports that no compatible attention operator is available;

• MSLK Triton Split-K rejects head dimensions such as 40, 80, or 160;

• you want a documented Windows SM120 test stack rather than an unverified wheel;

• you use AUTOMATIC1111, Forge, ComfyUI, Diffusers, or another frontend that can use the xFormers FMHA API.

This remains an experimental community compatibility release.

It is not official upstream xFormers Windows support, and it is not official upstream MSLK Blackwell support.

The validated hardware is:

NVIDIA GeForce RTX 5070 Laptop GPU
Compute capability 12.0 / SM120
Windows x64

Other RTX 50-series GPUs may work, but they require independent testing before they should be treated as supported.

Short Summary

I previously shared an experimental xFormers + MSLK setup that proved one known FP16 attention workload could run on an RTX 5070 Laptop GPU under Windows.

That first result was useful, but it was limited.

The original validation centered on:

Q/K/V shape: (1, 9600, 1, 512)
dtype:       float16

The newer release goes much further.

MSLK 1.3 now supports the actual model-visible attention head dimensions required by the tested Diffusers UNet:

Logical 40  -> internal Triton tile 64
Logical 80  -> internal Triton tile 128
Logical 160 -> internal Triton tile 256

The stack has now been validated through:

• synthetic self-attention and cross-attention tests;

• numerical comparison against PyTorch SDPA;

• a real Diffusers UNet denoising step;

• normal 512×512 generation;

• repeated fixed-seed generation;

• rectangular 512×768 generation;

• 512×768 to 1024×1536 hires generation;

• provider-execution verification;

• startup and restart configuration checks.

The updated public releases are:

MSLK 1.3.0 — Windows Blackwell SM120

https://github.com/Kittensx/mslk-blackwell-windows/releases/tag/v1.3.0-blackwell-sm120

xFormers Blackwell v1.1 — SM120 FMHA Diagnostics and Execution Evidence

https://github.com/Kittensx/xformers-blackwell-windows/releases/tag/v1.1.0-blackwell-sm120

Install MSLK first, then xFormers.

What Changed Since the First Article

The first article documented a narrow but working path:

xFormers

+

MSLK

+

PyTorch 2.11.0+cu128

+

CUDA 12.8

+

RTX 5070 Laptop GPU

That earlier build successfully ran the K=512 FP16 validation shape and completed real AUTOMATIC1111 generation with --xformers.

The limitation was that a real Diffusers UNet did not use only head dimension 512.

The tested model required:

40
80
160

The earlier MSLK Triton Split-K operator rejected those dimensions.

Typical failure output looked like:

triton_splitKF is not supported because:
    Embed dim 40 not supported

The updated work solves that problem rather than merely bypassing the error.

Earlier release

- Known-good K=512 test

- FP16 validation

- A1111 startup and generation validation

- Basic Blackwell-compatible wheel packaging

- Limited visibility into which provider actually executed

Current release

- Real logical dimensions 40, 80, and 160

- Internal tile dimensions 64, 128, and 256

- Self-attention and cross-attention validation

- Exact provider selection

- Proof that the requested provider executed

- Real Diffusers UNet validation

- Normal and rectangular generation

- 1024×1536 hires generation

- Repeatability testing

- SDPA comparison

- Capability-based validation profiles

- Process-start configuration locking

- Detailed xFormers diagnostics and JSON reports

This is why I am treating this as a new article rather than quietly editing the original post.

What Problem This Solves

Installing xFormers is not the same thing as having a usable xFormers attention path.

A package can:

- install successfully;

- import successfully;

- report a version;

- expose an attention API;

and still fail when the model reaches a real attention layout.

On Blackwell hardware, the error may involve:

- an operator that was not built;

- a device capability guard that rejects SM120;

- unsupported dtype;

- unsupported head dimension;

- excessive shared-memory use;

- the dispatcher selecting no valid operator;

- a package being imported without the expected provider ever executing.

In this stack, the practical attention path is:

Application or frontend

-> xformers.ops.memory_efficient_attention

-> MSLK FMHA compatibility layer

-> MSLK Triton Split-K

-> CUDA FP16

-> NVIDIA Blackwell SM120

The newer releases focus on making that path observable, testable, and reproducible.

Image Evidence

xFormers/MSLK generation

The Technical Fix

Why 40, 80, and 160 Failed

The Triton Split-K kernel used the logical head dimension directly as an internal block shape.

Triton requires relevant block dimensions to use supported power-of-two tile sizes.

These model dimensions are not powers of two:

40

80

160

The direct implementation therefore failed during kernel compilation.

The solution was to distinguish:

logical head dimension
internal Triton tile dimension

The validated mappings are:

Logical 40  -> internal tile 64
Logical 80  -> internal tile 128
Logical 160 -> internal tile 256

Why This Does Not Change Model Semantics

The model still sees its original dimensions.

For a logical head dimension of 40:

- the model produces 40 Q lanes;

- the model produces 40 K lanes;

- the model produces 40 V lanes;

- the kernel internally uses a tile width of 64;

- lanes 40 through 63 are zero-filled;

- output lanes beyond 40 are masked;

- the returned output width remains 40.

The attention scale remains: 1 / sqrt(40)

It does not become: 1 / sqrt(64)

That distinction is important.

Using the tile width for scaling would change attention behavior.

Zero-filling padded Q and K lanes preserves the logical dot product. Zero-filling padded V lanes preserves the logical output lanes. Masking the padded reduction and output lanes prevents the internal tile width from leaking into model-visible results.

The same approach is used for:

80 -> 128

160 -> 256

Blackwell-Safe K=512 Support

The known-good K=512 workload remains supported:

Q/K/V shape: (1, 9600, 1, 512)

dtype: float16

operator: triton_splitKF

The earlier Windows Blackwell test exposed shared-memory pressure when the Split-K launch used:

BLOCK_N=64

The validated blackwell_safe policy uses a safer launch configuration for the SM120 environment, including:

BLOCK_N=32

This allowed the K=512 control to complete successfully while preserving the newer logical/tile support.

What MSLK 1.3 Adds

MSLK 1.3 is the release that contains the attention-kernel support.

Its major additions include:

- logical head dimensions 40, 80, and 160;

- internal Triton tile widths 64, 128, and 256;

- padded Q/K/V masking;

- padded Split-K reduction masking;

- logical attention-scale preservation;

- Blackwell-safe launch tuning;

- capability-based validation profiles;

- process-start configuration freezing;

- first-kernel execution evidence;

- startup and pending-setting diagnostics.

The public wheel is:

mslk-1.3.0-py3-none-any.whl

This is a Python-only wheel.

The validated attention implementation uses Triton JIT compilation. It does not require the unrelated native MSLK C++/CUDA extensions.

What xFormers Blackwell v1.1 Adds

The xFormers v1.1 release does not add another attention kernel.

The actual 40/80/160 kernel support is in MSLK.

The xFormers release adds visibility and control around the MSLK-backed FMHA compatibility layer.

New diagnostic helpers include:

get_fmha_provider_registry()

get_operator_support_report(inputs)

get_selected_operator(inputs)

resolve_forward_operator(operator)

execute_forward_with_evidence(...)

get_runtime_package_provenance()

These distinguish:

registered

supported

selected

executed

That distinction matters.

An operator should not be reported as used merely because:

- the package imported;

- the class registered;

- the dispatcher considered it;

- the processor attached successfully.

The execution-evidence path reports success only after the explicitly requested operator returns an output tensor.

The release also expands:

python -m xformers.info

and adds:

python -m xformers.info --json

The report can include:

- active xFormers module path;

- native _C.pyd path;

- xFormers build metadata;

- active MSLK module path;

- active Triton module path;

- GPU name;

- compute capability;

- registered FMHA providers;

- operator classes;

- supported dtypes;

- head-dimension limits;

- device capability guards;

- native-registration evidence.

Validated Environment

This is the environment used for the completed validation:

Operating system: Windows x64
GPU: NVIDIA GeForce RTX 5070 Laptop GPU
Compute capability: 12.0 / SM120
Python: 3.10.20
PyTorch: 2.11.0+cu128
CUDA runtime: 12.8
CUDA toolkit: 12.8
Triton-Windows: 3.7.1.post27
MSLK: 1.3.0
xFormers source: 0.0.35 / commit af84367e
Attention dtype: float16
Tuning policy: blackwell_safe

The xFormers public wheel is:

xformers-0.0.35+af84367e.sm120diag1-py39-none-win_amd64.whl

The xFormers 0.0.35 package identity is retained because the build is based on that upstream source line.

The GitHub release itself is called:

xFormers Blackwell v1.1

The package version and release title are intentionally separate.

Downloads

MSLK 1.3.0

Release page:

https://github.com/Kittensx/mslk-blackwell-windows/releases/tag/v1.3.0-blackwell-sm120

Wheel:

mslk-1.3.0-py3-none-any.whl

SHA256:

4402cdae1c2006d93179dc7a9341777c8521128459743242a894032edf8b4354

xFormers Blackwell v1.1

Release page:

https://github.com/Kittensx/xformers-blackwell-windows/releases/tag/v1.1.0-blackwell-sm120

Wheel:

xformers-0.0.35+af84367e.sm120diag1-py39-none-win_amd64.whl

SHA256:

0f029db4738d94a4cd7dd5d55ba527e8b24ada3b6efbdad70b0759244b6c7bba

Verify a downloaded wheel on Windows with:

certutil -hashfile mslk-1.3.0-py3-none-any.whl SHA256

certutil -hashfile xformers-0.0.35+af84367e.sm120diag1-py39-none-win_amd64.whl SHA256

Do not install a wheel when its checksum does not match the published release.

Recommended

requirements-blackwell.txt

--extra-index-url https://download.pytorch.org/whl/cu128
torch==2.11.0+cu128
triton-windows==3.7.1.post27

mslk @ https://github.com/Kittensx/mslk-blackwell-windows/releases/download/v1.3.0-blackwell-sm120/mslk-1.3.0-py3-none-any.whl#sha256=4402cdae1c2006d93179dc7a9341777c8521128459743242a894032edf8b4354

xformers @ https://github.com/Kittensx/xformers-blackwell-windows/releases/download/v1.1.0-blackwell-sm120/xformers-0.0.35%2Baf84367e.sm120diag1-py39-none-win_amd64.whl#sha256=0f029db4738d94a4cd7dd5d55ba527e8b24ada3b6efbdad70b0759244b6c7bba

The %2B in the xFormers URL represents the + character in the wheel filename.

Install with:

python -m pip install -r requirements-blackwell.txt

Installation

Step 1 — Open the Correct Environment

Install the wheels into the same Python environment used by your frontend.

For AUTOMATIC1111, navigate to the WebUI folder:

cd /d C:\A1111\stable-diffusion-webui

Activate its virtual environment:

venv\Scripts\activate.bat

For another application, activate that application's venv instead.

You should see the venv name at the beginning of the prompt.

Example:

(venv) C:\A1111\stable-diffusion-webui

If the venv is not active, you may install the packages into the wrong Python environment.

Step 2 — Confirm Python

python --version

where python

The validated environment used Python 3.10.20.

Step 3 — Install PyTorch and Triton-Windows

python -m pip install ^

torch==2.11.0+cu128 ^

torchvision==0.26.0+cu128 ^

torchaudio==2.11.0+cu128 ^

--index-url https://download.pytorch.org/whl/cu128

Then:

python -m pip install triton-windows==3.7.1.post27

Step 4 — Install MSLK First

python -m pip install --no-deps ^

https://github.com/Kittensx/mslk-blackwell-windows/releases/download/v1.3.0-blackwell-sm120/mslk-1.3.0-py3-none-any.whl

Step 5 — Install xFormers Second

python -m pip install --no-deps ^

https://github.com/Kittensx/xformers-blackwell-windows/releases/download/v1.1.0-blackwell-sm120/xformers-0.0.35%2Baf84367e.sm120diag1-py39-none-win_amd64.whl

The install order matters:

1. MSLK

2. xFormers

Step 6 — Confirm Installed Versions and Paths

python -c "import torch, mslk, xformers; print('Torch:', torch.__version__); print('CUDA:', torch.version.cuda); print('MSLK:', mslk.__version__, mslk.__file__); print('xFormers:', xformers.__version__, xformers.__file__)"

Expected key values:

Torch: 2.11.0+cu128

CUDA: 12.8

MSLK: 1.3.0

xFormers: 0.0.35+af84367e.sm120diag1

How To Validate the Installation

Step 1 — Run xformers.info

python -m xformers.info

Look for information similar to:

pytorch.version: 2.11.0+cu128

gpu.compute_capability: 12.0

gpu.name: NVIDIA GeForce RTX 5070 Laptop GPU

build.cuda_version: 1208

build.env.TORCH_CUDA_ARCH_LIST: 12.0

fmha.source_mode: mslk_facade

The exact formatting may differ, but the report should show:

- CUDA is available;

- the correct GPU is detected;

- the native xFormers extension loaded;

- the build targeted SM120;

- the FMHA implementation is visible;

- MSLK and Triton are available.

For a machine-readable report:

python -m xformers.info --json > xformers-info.json

Step 2 — Verify the Explicit K=512 Control

Create:

test_xformers_sm120.py

with:

import torch

from xformers.ops.fmha import execute_forward_with_evidence


print("Torch:", torch.__version__)
print("CUDA runtime:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())

if not torch.cuda.is_available():
    raise SystemExit("CUDA is not available.")

print("GPU:", torch.cuda.get_device_name(0))
print("Capability:", torch.cuda.get_device_capability(0))

query = torch.randn(
    (1, 9600, 1, 512),
    device="cuda",
    dtype=torch.float16,
)

key = torch.randn_like(query)
value = torch.randn_like(query)

output, evidence = execute_forward_with_evidence(
    query,
    key,
    value,
    operator="triton_splitKF",
)

print(evidence.to_dict())

if output is None or not evidence.executed:
    raise SystemExit("Explicit Triton Split-K execution failed.")

if not torch.isfinite(output).all():
    raise SystemExit("Output contains non-finite values.")

print("PASS")

Run:

python test_xformers_sm120.py

A successful result should include:

executed: True
provider: mslk_triton_splitk
PASS

This is stronger than merely confirming that xFormers imported.

Step 3 — Validate Real Generation

After the explicit test passes, start your frontend with its xFormers attention option.

For AUTOMATIC1111:

set COMMANDLINE_ARGS=--xformers

Then launch:

webui-user.bat

or:

webui.bat

Successful application-level validation means:

- the frontend starts;

- the model loads;

- image generation completes;

- no xFormers or MSLK operator error appears;

- the process does not silently fall back when explicit xFormers was requested.

Attention Settings Warning

For initial validation, use one attention backend at a time.

For AUTOMATIC1111, begin with:

--xformers

Avoid combining it with:

--opt-sdp-attention

--opt-sdp-no-mem-attention

--force-enable-xformers

Those options may alter which attention path is selected and make troubleshooting more difficult.

Also avoid:

--no-half

--precision full

The validated path uses FP16 attention.

The public release has not been validated for float32 attention.

Validation Results

Synthetic Attention Matrix

The required logical dimensions passed:

Head dimension 40: 16/16 cases passed

Head dimension 80: 16/16 cases passed

Head dimension 160: 16/16 cases passed

Total: 48/48 cases passed

Coverage included:

- self-attention;

- cross-attention;

- one head;

- eight heads;

- query lengths through 1024;

- repeated execution;

- finite-value checks;

- output-shape checks;

- comparison with PyTorch SDPA.

Maximum absolute errors against SDPA:

D=40: 0.00048828125

D=80: 0.00048828125

D=160: 0.0009765625

`

These are small FP16 numerical differences.

Known-Good K=512 Control

The K=512 control passed:

Shape: (1, 9600, 1, 512)

Dtype: float16

Provider: mslk_triton_splitk

Operator: triton_splitKF

BLOCK_N: 32

Output: finite

Execution: successful

Real Diffusers UNet Validation

The stack completed:

- the first real UNet denoising step;

- normal 512×512 generation;

- a repeated fixed-seed 512×512 generation;

- 512×768 generation;

- SDPA comparison;

- provider-execution metadata verification.

The repeated xFormers/MSLK result was pixel-exact.

That means the same fixed-seed xFormers/MSLK run produced an identical image on repeat.

## Normal Generation Comparison

The xFormers/MSLK and SDPA outputs were visually equivalent.

Measured mean pixel difference:

0.9158147 / 255

The repeated xFormers result had zero changed pixels.

Hires Validation

The tested hires workflow was:

Base: 512×768

Hires: 1024×1536

The hires pass executed through:

Provider: mslk_triton_splitk

Operator: triton_splitKF

The largest observed hires attention sequence length was:

24576

No failed custom-provider calls were recorded.

The repeated xFormers/MSLK hires output was pixel-exact.

Comparison against SDPA:

Mean absolute pixel difference: 1.216876 / 255

PSNR: 39.0819 dB

The images were manually reviewed and classified as visually equivalent.

## Hires Memory Measurements

The measured stage peaks were:

Base-pass peak allocated: 6.891 GiB

Hires second-pass peak allocated: 8.625 GiB

Final decode peak allocated: 3.738 GiB

The hires workload ran close to the practical memory boundary of the laptop GPU.

Users with different VRAM capacities should not assume the same hires resolution will fit.

## Startup and Restart Validation

The release also validated startup configuration behavior.

Confirmed:

- active startup settings are fingerprinted;

- environment changes after import are detected as pending;

- pending changes do not mutate the active process;

- first Triton execution records the active startup fingerprint;

- changing launch settings requires a full process restart.

The explicit K=40 startup test passed against SDPA with a maximum absolute error of:

0.0001220703125

Capability-Based Validation

The MSLK wheel includes:

mslk/attention/fmha/validated_profiles/sm120_triton_splitk_fp16.json

The profile records:

- compute capability;

- FP16 requirement;

- Torch CUDA runtime;

- Triton runtime;

- logical-to-tile mappings;

- operator identity;

- tuning policy;

- required execution capabilities;

- critical implementation hashes;

- validation results.

Package version strings are recorded for diagnostics, but they are not the sole compatibility gate.

That is intentional.

A wheel name or release suffix can change without changing attention behavior. Conversely, two packages can report the same version while containing different files.

The meaningful compatibility checks are:

• hardware capability

• runtime requirements

• kernel identity

• critical file hashes

• required xFormers API behavior

• successful explicit operator execution

Applications should reject the custom provider or fall back to PyTorch SDPA when those checks fail.

Process-Start Settings

The MSLK attention settings are frozen at process startup.

Available variables include:

• MSLK_FMHA_POLICY

• MSLK_FMHA_DEBUG

• MSLK_FMHA_BLOCK_M

• MSLK_FMHA_BLOCK_N

• MSLK_FMHA_NUM_WARPS

• MSLK_FMHA_NUM_STAGES

• MSLK_FMHA_EXPERIMENTAL_HEAD_DIMS

The validated policy is:

MSLK_FMHA_POLICY=blackwell_safe

Changing one of these settings after the process starts does not safely reconfigure already compiled Triton kernels.

Close and restart the Python process after changing them.

A browser refresh is not a process restart.

Observed Performance

The earlier A1111 validation showed a noticeable practical speed improvement compared with fallback attention implementations.

The newer work focused primarily on:

- correctness;

- real model compatibility;

- repeatability;

- provider verification;

- hires stability;

- memory measurement.

It was not designed as a broad performance benchmark across GPUs or frontends.

The measured 1024×1536 hires test completed successfully, but it operated close to the available memory boundary.

Performance will vary with:

- GPU;

- laptop power limit;

- cooling;

- model;

- resolution;

- batch size;

- sampler;

- denoising steps;

- VAE;

- frontend;

- memory-management policy.

Treat any performance observation as environment-specific unless independently reproduced.

Important Limitations

This remains an experimental compatibility package.

It has been validated on:

RTX 5070 Laptop GPU

SM120

Windows x64

FP16

PyTorch 2.11.0+cu128

CUDA 12.8

Triton-Windows 3.7.1.post27

It has not yet been validated across the entire RTX 50-series lineup.

I am not claiming confirmed support for:

RTX 5060

RTX 5070 Desktop

RTX 5080

RTX 5090

Those GPUs require real testing.

The release is also not claiming general support for:

- float32 attention;

- quantized Q/K/V;

- grouped Q/K/V;

- arbitrary training workloads;

- arbitrary Torch versions;

- arbitrary CUDA versions;

- arbitrary Triton versions;

- Linux;

- unsupported xFormers operators;

- every model architecture.

The logical/tile work was validated for the specific FP16 Split-K path described in this article.

Troubleshooting

No module 'xformers'

Confirm that you installed into the same venv used by the application:

where python
python -c "import xformers; print(xformers.__file__)"

xFormers imports but generation fails

Run:

python -m xformers.info --json > xformers-info.json

Then check:

- GPU name;

- compute capability;

- native extension path;

- MSLK path;

- Triton path;

- available operators;

- rejection reasons.

Embed dim 40 not supported

That usually indicates that the older MSLK package is still installed, or the new wheel was installed into another environment.

Check:

python -c "import mslk; print(mslk.__version__); print(mslk.__file__)"

Expected:

1.3.0

Shared-memory error on K=512

Confirm that the Blackwell-safe policy is active and restart the process:

set MSLK_FMHA_POLICY=blackwell_safe

Then close and relaunch the application.

Float32 operator rejection

That is expected for this validated route.

Use FP16 and avoid:

--no-half

--precision full

Wrong package path

It is possible to install the wheel into one Python environment and run the application from another.

Always check:

where python
python -c "import mslk, xformers; print(mslk.__file__); print(xformers.__file__)"

Reporting Results

If you test this on another Blackwell GPU, please include:

• GPU model

• Compute capability

• Driver version

• Windows version

• Python version

• PyTorch version

• CUDA runtime

• CUDA toolkit

• Triton-Windows version

• MSLK version

• xFormers version

• Frontend

• Model

• Launch arguments

• Resolution

• Whether python -m xformers.info completed

• Whether the explicit K=512 test passed

• Whether normal generation completed

• Whether hires generation completed

• Any xFormers, MSLK, Triton, or CUDA errors

The most useful reports include complete logs rather than only “works” or “does not work.”

For xFormers diagnostics:

python -m xformers.info --json > xformers-info.json

Attach that JSON report when possible.

Community Validation Wanted

I would especially like reports from:

RTX 5070 Desktop

RTX 5080

RTX 5090

Useful questions include:

- Does the explicit K=512 test pass?

- Do logical dimensions 40, 80, and 160 execute?

- Does your frontend complete real generation?

- Does hires generation work?

- Does another model require additional head dimensions?

- Does the Blackwell-safe policy remain necessary?

- Are there driver-specific differences?

- Does the stack remain stable over repeated generations?

Additional results will help determine which parts of this solution are broadly useful across Blackwell and which remain specific to the RTX 5070 Laptop GPU.

Why I’m Sharing This

The first release began as a narrow attempt to solve:

No module 'xformers'. Proceeding without it.

The real problem turned out to be deeper than package installation.

A usable attention path required:

- the correct Torch/CUDA environment;

- an SM120 xFormers build;

- a companion MSLK implementation;

- Blackwell-safe launch tuning;

- real model-dimension support;

- padded logical/tile handling;

- masked Split-K reduction;

- explicit operator selection;

- execution evidence;

- normal-generation testing;

- hires testing;

- SDPA comparison;

- restart-safe configuration.

That took enough investigation that it seemed worth publishing as a documented and checksummed release rather than leaving it as local patches.

The goal is not to claim universal RTX 50-series support.

The goal is to provide a reproducible starting point for other Windows Blackwell users who are stuck, and to make it easier for them to report useful results.

Credits

Credit belongs to the upstream maintainers and contributors of:

- xFormers;

- MSLK;

- Triton;

- Triton-Windows;

- PyTorch;

- CUDA;

- Diffusers;

- CUTLASS;

- the Stable Diffusion frontend ecosystem.

These releases package and document an experimental Windows Blackwell compatibility path for a specific validated environment.

They do not imply official upstream support.