FreeAttn: Unlocking Native E3M4 for Efficient and Accurate Attention

Zero-inner-loop-overhead (Dual-Free) block-scaled attention on NVIDIA Blackwell Tensor Cores — faster than SageAttention2++ in 8/8-bit with higher video PSNR and 100% 32K LongBench-v2 agreement, while unlocking native GQA 8/4 (8-bit QK⊤, 4-bit PV) and 4/4 NVFP4.

▶ Interactive Video 9-Scene Gallery 32K LLM Outputs Dual-Free Method Benchmarks
Scene
Right Side
Left: BF16 Ref
Right: FreeAttn E3M4 (+Free)
1.69s / 5.06s (f27)
TL;DR
Q

Why do existing quantized attention kernels on Blackwell either lose multi-step video and 32K LLM fidelity (3-bit mantissa FP8 E4M3 in SageAttention2++) or fail to use native FP8 E3M4 in streaming attention?

A

Hardware 32-element UE8M0 (unsigned 8-bit power-of-two) block scaling already keeps 99.93% of real Q/K/V energy inside [0.25, 30.0], leaving E4M3's 4th exponent bit idle. FreeAttn turns that bit into a 4th mantissa (E3M4, 3.82–4.62× lower MSE at 1.000× Tensor Core speed) and fuses 1-pass softmax-invariant outlier and row-sum cancellation, beating SageAttention2++ in speed, video PSNR, and 32K LongBench-v2 accuracy while extending to lossy 8/4 and 4/4 NVFP4.

Abstract

Low-precision FlashAttention must quantize Q, K, online softmax probabilities P, and V inside a streaming loop where channel outliers and row-sum drift quickly compound across video denoising steps and 32K-token contexts. Prior per-tensor FP8 studies favored E4M3 because unscaled spikes clip E3M4 (±30), while recent Blackwell ISA microbenchmarks probed static E3M4 matrix multiply without online P packing or outlier control (dropping Wan2.1-14B 720p video to 16.80 dB PSNR when applied raw).

We present FreeAttn, a Dual-Free attention design—meaning both representation refinement and algebraic error cancellation add zero extra matrix-multiply or rescale instructions inside the streaming attention loop. Across 7 models on a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB GDDR7), lossless-oriented FreeAttn 8/8 E3M4 (+Free) reaches 24.02 dB PSNR and 1.42× full-generation speedup on 50-step 720p Wan2.1-14B (+2.74 dB and 69s faster than SageAttention2++), achieves 32/32 (100%) LongBench-v2 agreement on 32K Qwen3-8B and Qwen3-32B at 1.95–1.99× attention speedup, and—with optional +Paid query/probability stabilization—unlocks native GQA 8/4 (8-bit QK⊤, 4-bit PV) and 4/4 NVFP4 where official SageAttention3 fails natively on GQA and suffers 80–143× higher 32K KL after 4× KV replication.

Free #1 · Representation

Idle Exponent → 4th Mantissa + Register F2FP

K = 32 UE8M0 scaling confines 99.93% of tensor energy to [0.25, 30.0], cutting quantization MSE 3.82–4.62× vs. E4M3 at 1.000× QMMA.SF speed, while 1-instruction dual-element F2FP packs online P in registers (+4.6%–17.4% faster complete call).

Free #2 · Softmax Invariance

1-Pass Outlier & Row-Sum Cancellation

Smooth-K/V, Scale-QK, and Hadamard-QK cancel algebraically across online softmax with zero inner-loop MMAs in one fused GDDR7 pass (1,467–1,559 GB/s vs. 420–445 GB/s unfused; 1.4%–4.1% of long-sequence latency).

01 · Multi-Step Video Diffusion

Multi-Step Video Generation (720p, 3D Orbit & Multi-Scene Grids)

All video comparisons use identical initial noise, prompt schedules, and VAE decoding on one RTX PRO 6000 Blackwell GPU, replacing only DiT self-attention. Drag the interactive 4-scene split-screen above to inspect 1:1 pixel fidelity against BF16, or switch across all nine synchronized 2×4 comparison videos below (spanning Wan2.1-14B 720p×81f, 60° 3D architectural orbit, 480p dynamics, and Wan2.1-1.3B).

Frame-27 Bus-Front Crops & Lossy 4-Bit NVFP4 Recovery

Click figure to enlarge
Paper Figure 1: Wan2.1-14B 720p Frame 27 comparison and 4-bit NVFP4 recovery
Figure 1 (Wan2.1-14B, 720p×81f, Frame 27): 8/8 E3M4 (+Free, 24.02 dB) preserves the two-window bus front (vs. four windows in SageAttention2++ and a rotated bus in SageAttention3). Bottom row shows lossy 4/4 NVFP4 recovery from 12.56 dB (raw) to 18.42 / 17.69 dB at 1.55× full-generation speedup, trading 6.33 dB PSNR vs. 8-bit E3M4 for faster 4-bit execution. [Click to zoom]
02 · Autoregressive GQA Language Models

32K GQA Language Models Across 8-Bit, 8/4 & 4-Bit NVFP4

Modern LLMs rely on Grouped-Query Attention (GQA, e.g., Hq = 32, Hkv = 8 in Qwen3-8B). Official Blackwell SageAttention3 supports only MHA at D = 128—crashing natively on GQA and collapsing into immediate <|im_end|> or ffff... gibberish even when K/V heads are replicated 4×. FreeAttn supports native GQA across D ∈ {64, 128, 256} from lossless-oriented 8/8 E3M4 (+Free) down to lossy 8/4 E3M4 and 4/4 NVFP4 (+Free+Paid).

Official SageAttention3 Native GQA Crash (H_q=32, H_kv=8): RuntimeError: The size of tensor a (32) must match the size of tensor b (8) at non-singleton dimension 1
Low-Precision GQA Spectrum (Qwen3-8B / 32B, 32-item 32K LongBench-v2 prefill & block-scaled decode KV).
Precision Tier & Method GQA 8B KL ↓ 8B / 32B Agree ↑ 8B Attn ↑ KV (8B KL ↓) Behavior
PyTorch SDPA BF16 (Ref) Yes 0.00000 100% / 100% 1.00× 16b (0.0000) Reference
FreeAttn 8/8 E3M4 (+Free) Yes 0.00325 100% / 100% 1.99× 8.25b (0.0087) 32/32 exact
FreeAttn 8/8 E4M3 (+Free) Yes 0.00794 87.5% / 93.8% 2.01× 8.25b (0.0313) 22.5K QA flip
SageAttention2++ (F16) Yes 0.00647 96.9% / 96.9% 1.76× 16b BF16 only 22.5K QA flip
FreeAttn 8/4 E3M4 (+Free+Paid) Yes 0.02026 90.6% / 96.9% 1.91× 6.25b (0.0556) Lossy; clean
FreeAttn 4/4 NVFP4 (+Free+Paid) Yes 0.02802 81.2% / 87.5% 1.94× 4.25b (0.2372) 69.5% 5-task; clean
SageAttention3 4/4 NVFP4* No 1.86–2.90† Fails / No GQA — — EOS & ffff...
32-item LongBench-v2 (S ≤ 32K) prefill means; KV = effective bits/elem (incl. K = 32 UE8M0) and 8-item 32K decode KL; 5-task = ARC-C, HellaSwag, MMLU, PIQA, WinoGrande (70.0% BF16). *†With 4× KV replication on 22.5K–31.9K QA, Sage3 KL is 1.86–2.90 vs. 0.0203–0.0234 for FreeAttn NVFP4 (80–143× worse).

Verbatim Greedy Outputs Across Precision Tiers (Qwen3-8B)

Switch tasks and precision views to compare exact decoded tokens
Task
View
03 · Dual-Free Attention Framework

Dual-Free Co-Design: Beyond Static ISA Probes & SageAttention2++

Simply invoking Blackwell's QMMA.SF E3M4 instruction inside FlashAttention fails because channel means inflate 32-element block scales and quantized probability error ΔP shifts row sums against uncentered V. FreeAttn co-designs Free #1 (4-bit-mantissa E3M4 + 1-instruction dual-element F2FP register packing of online P) with Free #2 (single-pass GDDR7 fusion of softmax-invariant outlier and row-sum cancellation).

Comparison Against Prior FP8 PTQ & Public ISA Probes.
Approach Scaling & Online P Outlier / Bias Control Streaming Attention Result
Per-Tensor FP8 PTQ (Kuzmin'22; Shen'24) Global / channel scale; no online P Static range calibration Spikes clip E3M4 → favors E4M3
Public E3M4 ISA Probe (blackwell-isa'26) Static QMMA.SF; no packed F2FP None (static GEMM probe) No attention kernel (Raw E3M4: 16.80 dB)
FreeAttn Dual-Free (Ours) K=32 UE8M0 + 1-inst register F2FP 1-pass Smooth-K/V + Scale/Had-QK 3.82–4.62× lower MSE; 24.02 dB; 32/32 LB-v2

Why FreeAttn Beats SageAttention2++ in Both Speed and Accuracy

SageAttention2++ mixes per-warp INT8 QK⊤ (requiring software tile dequantization and rescaling inside the loop) with 3-bit-mantissa FP8 E4M3 PV in FP16/FP22 accumulation without Smooth-V. FreeAttn 8/8 E3M4 (+Free) executes both QK⊤ and PV on unified hardware block-scaled QMMA.SF with full FP32 accumulation, 4-bit mantissas, and 1-instruction F2FP packing.

Paper Figure 5: Mechanism ablations
Figure 2 (Mechanism Ablations): (a, b) Free #1 E3M4 vs. E4M3 MSE under 32-element UE8M0 scaling; (c, d) register F2FP (+4.6%–17.4%) and Free #2 1-pass GDDR7 fusion; (e, f) stepwise Free/Paid gains (8/4 = 8-bit QK⊤ / 4-bit PV). [Click to zoom]
04 · Quantitative Benchmarks

Quantitative Benchmarks Across Operators, Video & 32K LLMs

All timings are measured on a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB GDDR7). Operator and captured-layer complete-call latencies include fresh-input preparation (smoothing, orthogonal scaling/Hadamard, and quantization) and output post-processing alongside kernel execution.

Table 1 · Arithmetic Formats and Architectural Support.
Implementation QK⊤ PV Head Dimensions & Attention Types
PyTorch SDPA / cuDNN BF16 BF16 D ∈ {64, 128, 256}, MHA & GQA
FlashAttention-4 BF16 BF16 D ∈ {64, 128}, MHA & GQA
SageAttention2 / 2++ (F16 or F32 Acc) INT8 FP8 E4M3 D ∈ {64, 128}, MHA & GQA (no D = 256)
SageAttention3 (Official Blackwell FP4) NVFP4 NVFP4 D = 128 only, MHA only (no GQA, no D ∈ {64, 256})
FreeAttn (Ours, Hardware QMMA.SF) FP8 E3M4 / E4M3 or NVFP4 FP8 E3M4 / E4M3 or NVFP4 D ∈ {64, 128, 256}, Native MHA & GQA
Paper Figure 4: Operator kernel and complete-call latency
Figure 3 (Operator Latency): Solid = kernel; framed = complete call. 8/8 E3M4 (+Free) matches E4M3 (1.000×) at 1.56–1.94× over PyTorch SDPA (1.05–1.53× over SageAttention2++; up to 3.4× on GQA / D ∈ {64, 256}). [Click to zoom]
End-to-end fidelity and speedup across 7 models
Figure 4 (Seven-Model Summary): 32K KL on four GQA LLMs, multi-scene PSNR on Wan2.1-1.3B / 14B, and 720p single-step Rel-L2 on Wan2.1-14B / Wan2.2-5B vs. speedup. [Click to zoom]
Table 2 · Multi-Step Video Diffusion Quality & Latency (Wan2.1-1.3B & Wan2.1-14B).
Model / Suite Configuration 1-Step Rel-L2 ↓ PSNR (dB) ↑ SSIM ↑ LPIPS ↓ Attn (s) ↓ Full Gen (s) ↓
Wan2.1-1.3B (480p, 20-Step, 6-Scene) PyTorch SDPA BF16 0.00000 Ref 1.0000 0.0000 4.71 (1.00×) 19.47 (1.00×)
Wan2.1-1.3B (480p, 20-Step, 6-Scene) SageAttention2++ (FP16) 0.00593 23.37 0.8444 0.0877 2.79 (1.69×) 17.53 (1.11×)
Wan2.1-1.3B (480p, 20-Step, 6-Scene) SageAttention3 (NVFP4) 0.01704 13.71 0.4636 0.4627 2.79 (1.69×) 17.52 (1.11×)
Wan2.1-1.3B (480p, 20-Step, 6-Scene) FreeAttn 8/8 E4M3 (+Free) 0.00567 21.01 0.7747 0.1461 2.44 (1.93×) 17.18 (1.13×)
Wan2.1-1.3B (480p, 20-Step, 6-Scene) FreeAttn 8/8 E3M4 (+Free) 0.00536 23.85 0.8605 0.0821 2.46 (1.92×) 17.20 (1.13×)
Wan2.1-14B (720p×81f, 50-Step Turntable) PyTorch SDPA BF16 0.00000 Ref 1.0000 0.0000 12.77 (1.00×) 2235.7 (1.00×)
Wan2.1-14B (720p×81f, 50-Step Turntable) SageAttention2++ (FP16) 0.00572 21.28 0.9164 0.0775 6.96 (1.83×) 1638.6 (1.36×)
Wan2.1-14B (720p×81f, 50-Step Turntable) SageAttention3 (NVFP4) 0.00947 13.78 0.8295 0.2852 6.03 (2.12×) 1517.2 (1.47×)
Wan2.1-14B (720p×81f, 50-Step Turntable) FreeAttn 8/8 E4M3 (+Free) 0.00545 20.17 0.9103 0.0767 6.45 (1.98×) 1561.9 (1.43×)
Wan2.1-14B (720p×81f, 50-Step Turntable) FreeAttn 8/8 E3M4 (+Free) 0.00517 24.02 0.9435 0.0481 6.53 (1.96×) 1569.7 (1.42×)
Wan2.1-14B (720p×81f, 50-Step Turntable) FreeAttn 4/4 NVFP4 (+Free+Paid) 0.00572 — — — 5.81 (2.20×) —
Wan2.1-14B (720p×81f, 50-Step Turntable) FreeAttn 4/4 NVFP4 (+Smooth-QKV+Had) — 17.69 0.8822 0.1494 — 1439.7 (1.55×)
Wan2.1-14B (720p×81f, 50-Step Turntable) FreeAttn 4/4 NVFP4 (Raw) 0.01263 12.56 0.8069 0.3580 3.64 (3.50×) 1430.0 (1.56×)
Wan2.1-14B (480p, 20-Step, 4-Scene Mean) SageAttention2++ (FP16) 0.00643 22.02 0.8314 0.0961 11.60 (1.63×) 81.61 (1.09×)
Wan2.1-14B (480p, 20-Step, 4-Scene Mean) SageAttention3 (NVFP4) 0.01069 16.24 0.6501 0.2665 13.05 (1.45×) 83.07 (1.07×)
Wan2.1-14B (480p, 20-Step, 4-Scene Mean) FreeAttn 8/8 E4M3 (+Free) 0.00730 20.25 0.7888 0.1247 10.81 (1.75×) 80.89 (1.10×)
Wan2.1-14B (480p, 20-Step, 4-Scene Mean) FreeAttn 8/8 E3M4 (+Free) 0.00595 22.41 0.8501 0.0811 10.92 (1.74×) 80.99 (1.10×)
Timing & row notes: On 20-step 480p suites, Attn (s) is cumulative 20-step self-attention time (Wan2.1-14B 480p BF16 reference: 18.96s Attn / 88.90s Full Gen); on 50-step 720p Turntable, Attn (s) is 1-step DiT self-attention latency (repeated across 100 CFG denoiser forwards, yielding the 666s Full Gen saving from 2235.7s to 1569.7s). On 720p 4-bit NVFP4, +Free+Paid reports the 1-step attention probe while +Smooth-QKV+Had reports the completed 50-step rollout. Across the 4-scene 50-step 720p Wan2.1-14B mean, 8/8 E3M4 (+Free) reaches 20.44 dB PSNR, 0.7939 SSIM (+0.0657 vs. E4M3), and 0.527× geometric-mean pixel MSE vs. matched 8/8 E4M3 (+Free).
Table 3 · 32K-Context GQA Language Prefill & LongBench-v2 (SageAttention3 unsupported natively on GQA).
Model & Suite Configuration Mean KL ↓ Agree ↑ Seq Match ↑ Attn (s) ↓ TTFT (s) ↓
Qwen3-8B · 32K Prefill (BF16: 0.889s / 2.583s) SageAttention2++ (FP16) 0.00179 99.02% — 0.512 (1.74×) 2.222 (1.16×)
Qwen3-8B · 32K Prefill FreeAttn 8/8 E4M3 (+Free) 0.00446 97.66% — 0.449 (1.98×) 2.181 (1.18×)
Qwen3-8B · 32K Prefill FreeAttn 8/8 E3M4 (+Free) 0.00187 98.63% — 0.451 (1.97×) 2.174 (1.19×)
Qwen3-32B · 32K Prefill (BF16: 3.141s / 10.073s) SageAttention2++ (FP16) 0.00152 98.05% — 1.774 (1.77×) 8.687 (1.16×)
Qwen3-32B · 32K Prefill FreeAttn 8/8 E4M3 (+Free) 0.00614 97.66% — 1.639 (1.92×) 8.520 (1.18×)
Qwen3-32B · 32K Prefill FreeAttn 8/8 E3M4 (+Free) 0.00144 98.24% — 1.659 (1.89×) 8.541 (1.18×)
Llama-3.1-8B · 32K Prefill (BF16: 0.792s / 2.470s) SageAttention2++ (FP16) 0.00137 98.97% — 0.456 (1.74×) 2.131 (1.16×)
Llama-3.1-8B · 32K Prefill FreeAttn 8/8 E4M3 (+Free) 0.00363 98.36% — 0.409 (1.94×) 2.081 (1.19×)
Llama-3.1-8B · 32K Prefill FreeAttn 8/8 E3M4 (+Free) 0.00146 98.92% — 0.414 (1.91×) 2.087 (1.18×)
Ministral-3-8B · 32K Prefill (BF16: 0.841s / 2.618s) SageAttention2++ (FP16) 0.00072 99.24% — 0.485 (1.73×) 2.260 (1.16×)
Ministral-3-8B · 32K Prefill FreeAttn 8/8 E4M3 (+Free) 0.00157 98.85% — 0.434 (1.94×) 2.200 (1.19×)
Ministral-3-8B · 32K Prefill FreeAttn 8/8 E3M4 (+Free) 0.00061 99.25% — 0.439 (1.92×) 2.210 (1.18×)
Qwen3-8B · LongBench-v2 (BF16: 0.488s / 1.70s) SageAttention2++ (FP16) 0.00647 96.9% 78.1% 0.278 (1.76×) 1.49 (1.13×)
Qwen3-8B · LongBench-v2 FreeAttn 8/8 E4M3 (+Free) 0.00794 87.5% 68.8% 0.242 (2.01×) 1.46 (1.16×)
Qwen3-8B · LongBench-v2 FreeAttn 8/8 E3M4 (+Free) 0.00325 100.0% (32/32) 93.8% 0.245 (1.99×) 1.46 (1.16×)
Qwen3-8B · LongBench-v2 FreeAttn 8/4 E3M4 (+Free+Paid) 0.02026 90.6% 78.1% 0.256 (1.91×) 1.46 (1.15×)
Qwen3-8B · LongBench-v2 FreeAttn 4/4 NVFP4 (+Free+Paid) 0.02802 81.2% 59.4% 0.252 (1.94×) 1.45 (1.16×)
Qwen3-32B · LongBench-v2 (BF16: 1.706s / 6.43s) SageAttention2++ (FP16) 0.00427 96.9% 75.0% 0.947 (1.80×) 5.68 (1.12×)
Qwen3-32B · LongBench-v2 FreeAttn 8/8 E4M3 (+Free) 0.00928 93.8% 75.0% 0.867 (1.97×) 5.59 (1.14×)
Qwen3-32B · LongBench-v2 FreeAttn 8/8 E3M4 (+Free) 0.00407 100.0% (32/32) 90.6% 0.876 (1.95×) 5.59 (1.14×)
Qwen3-32B · LongBench-v2 FreeAttn 8/4 E3M4 (+Free+Paid) 0.01730 96.9% 78.1% 0.908 (1.88×) 5.61 (1.14×)
Qwen3-32B · LongBench-v2 FreeAttn 4/4 NVFP4 (+Free+Paid) 0.02722 87.5% 81.2% 0.895 (1.91×) 5.56 (1.15×)
32K Prefill evaluates teacher-forced 32,768-token long-context passages (Agree = top-1 next-token match); LongBench-v2 evaluates 32 real 32K multi-choice QA items (Agree = answer match; Seq Match = 128-token greedy match). Adding +Paid lowers 8/8 E3M4 KL below Sage2++ on all four 32K LLMs; on 5,120 5-task items (8B / 32B), E3M4 (+Free) scores 70.2% / 76.4% (BF16: 70.0% / 76.3%) and NVFP4 (+Free+Paid) retains 69.5% / 75.9%.
Table 4 · Complete-Call Latency (ms) & Rel-L2 vs. FP32 on Captured Model Layers.
Implementation (ms / Rel-L2) Qwen3-8B L21 (2K GQA) Qwen3-32B L0 (32K GQA) Wan1.3B L18 (3.2K MHA) Wan14B L19 (75.6K MHA)
PyTorch SDPA (BF16) 0.193 / 0.00172 47.75 / 0.00168 0.274 / 0.00168 328.72 / 0.00167
FlashAttention-4 (BF16) 0.234 / 0.00172 47.95 / 0.00168 0.296 / 0.00168 327.19 / 0.00211
SageAttention2++ (FP16) 0.190 / 0.03325 26.09 / 0.04127 0.217 / 0.00901 181.07 / 0.00974
FreeAttn 8/8 E3M4 (+Free) 0.124 / 0.02026 24.83 / 0.02283 0.163 / 0.00929 169.54 / 0.01085
FreeAttn 8/8 E3M4 (+Free+Paid) 0.156 / 0.01636 31.08 / 0.01614 0.214 / 0.00802 225.49 / 0.00971
SageAttention3 (4/4 NVFP4) 0.309* / 0.86897 — (No GQA) 0.319 / 0.06123 149.62 / 0.07232
FreeAttn 8/4 E3M4 (+Free) 0.108 / 0.08559 20.23 / 0.08186 0.139 / 0.02490 140.76 / 0.02749
FreeAttn 4/4 NVFP4 (+Free+Paid) 0.109 / 0.10536 20.92 / 0.09092 0.138 / 0.05477 144.98 / 0.06396
*With 4× KV replication on Qwen3-8B L21, Sage3 runs at 0.309 ms with 0.86897 Rel-L2 (8.2× higher than FreeAttn NVFP4 +Free+Paid; 42.9× higher than E3M4 +Free).