MLRift

LLM GPU Inference — RX 7800 XT

Qwen3-0.6B

Greedy decode through the same compiler stack — no Python, no PyTorch runtime. Hardware: AMD Ryzen 9 7900X + AMD Radeon RX 7800 XT (gfx1100). All MLRift rows produce token-id output bit-identical to HuggingFace transformers.generate(do_sample=False). Peak VRAM read from /sys/class/drm/card1/device/mem_info_vram_used with 3-second cooldown between runs; idle baseline 174 MiB. Methodology in SLICE4_MEGAKERNEL_DESIGN.md.

mks16 = 4.23× PyTorch ROCm bf16, 6.57× ROCm fp32, 9.5× MLRift CPU. All MLRift mega paths use bf16 weights / fp32 GEMM — mantissa-truncated weights stream from VRAM, multiply happens in f32 lanes. Peak VRAM is slightly under PyTorch's native bf16 path and 42 % less than fp32. RSS is ~1.7-2.0 GiB across all MLRift configurations because the safetensors file is mmap'd; PyTorch carries an additional ~3 GiB of Python + transformers + tensor metadata. The mks-K kernels run as a single dispatch per layer (29 launches/token) with persistent work-groups and cross-WG barriers; spec_K = K is amortised through prompt-lookup-decode (PLD) speculative decoding with 99 % accept rate on the bench prompt. Number re-verified 2026-05-17 post-reboot; reproducer in docs/bench_2026-05-17.md.

Llama-3.2-1B-Instruct

Same hardware. The MLRift GPU path here is a per-layer mega-kernel emitted by MLRift's @kernel AST-walker (no hipcc, no LLVM, no ROCm runtime in the launcher — the --target=amdgpu-native KFD shim talks straight to /dev/kfd). Bit-exact vs llama.cpp greedy on 20 / 20 token ids for prompt "hello". Precision: bf16 weights, f32 activations, f32 accumulator — higher numerical fidelity than the PyTorch bf16 GPU baseline (which casts activations to bf16 too). Methodology in docs/bench_2026-05-13.md.

MLRift M=1 = 1.95× PyTorch bf16 GPU on Llama-3.2-1B, on any prompt (no PLD required). The per-layer mega-kernel fuses input-rmsnorm, QKV matmul, qkv-split + qknorm + RoPE, attention (decode-coop), o_proj + residual, post-norm, gate_up + silu + down + residual into a single dispatch with 9 cross-WG barriers — slashing per-layer launch overhead 9× and feeding the inner gemv loop with an s_clause-batched 12-load / 8-FMA pipeline (slice 4.25, 4× unroll). PyTorch fp32 VRAM (4749 MiB) leaves no headroom on a 16 GiB card for context expansion; MLRift's ~2.1 GiB envelope sits well below the bf16 baseline. +162% over PyTorch fp32 GPU at matching numerical fidelity.

Llama-3.2-3B-Instruct

Same mega-kernel emitter, scaled to a 3-B model with non-power-of-two GQA (32 Q heads, 8 KV heads — ratio 3:1 was the slice-8.1 libdivide fix). Greedy decode 20 tokens for prompt "hello", output byte-identical to PyTorch bf16+fp32 CPU: 11, 358, 2846, 8173, 304, 6975, 810, 922, 279, 7434, 315, 330, 23449, 5030, 1, 323, 1202, 25127, 369, 1057. Precision: bf16 weights, f32 activations, f32 accumulator. Slice-7.4 host-RSS dealloc (commit 447d220) saves 3.5 GiB of un-fused per-tensor bf16 originals after they're memcpy'd into the stacked qkv/gate-up layouts. Number re-verified 2026-05-17 post-reboot.

MLRift Llama-3.2-3B = 1.28× PyTorch bf16 GPU at -6% VRAM, byte-identical to PyTorch's greedy decode (3B was the model that established PT — not MLRift CPU — as the canonical GPU reference; see memory project_llama_3b_gpu.md). Slice 8.4 collapsed an earlier 7× cputree perf regression (5.88 → 40.5 tok/s) by removing a dead cputree dispatch path that was built on the inverted "CPU bf16 reference" framing. Higher host RSS than 1-B because the 28-layer Q8_0→bf16/fp32 dequant footprint scales with layer count; the slice-7.4 dealloc clawed back 3.5 GiB of those buffers (commit 447d220).

Mistral-7B-Instruct-v0.3

Same hardware, same mega-kernel emitter, scaled to a 7-B model. Prompt "hello", greedy decode, 20 tokens, output bit-identical to transformers.generate(do_sample=False): 1, 6312, 28709, 28725, 13, 13, 28710, 506, 264, 2996, .... PyTorch fp32 GPU OOMs on a 16 GiB card (the bf16 model alone is ~14 GiB) so the only PyTorch GPU baseline that fits is bf16. Precision: bf16 weights, f32 activations, f32 accumulator — same as Llama-1B and Qwen3-0.6B rows above.

MLRift Mistral-7B = 1.79-1.96× PyTorch bf16 GPU at parity VRAM, bit-identical greedy output through 22 of 23 tokens (the pos-6 divergence is exactly 1 bf16 ULP at the lm_head output and was bisected to reduction-tree topology — same family as Ollama/llama.cpp reduction trees, structurally sub-ULP to chase). The same 9-phase mega-kernel that runs Qwen3-0.6B (HIDDEN=1024), Llama-3.2-1B (HIDDEN=2048) and Llama-3.2-3B (HIDDEN=3072) extends to Mistral's HIDDEN=4096 with no kernel rewrite — only arch-globals retargeting. Slice 7.4 drops un-fused per-tensor host bf16 buffers right after fusing into the stacked qkv/gate-up layouts, trimming peak host RSS from 19.9 GiB to 14.0 GiB on the Q8_0 path. Slice 7.5 doubles the inner-loop unroll factor from 4× to 8× — each outer iter now issues 24 VMEM loads in one s_clause(23) batch and fires 16 v_fmac_f32 ops, widening the independent-FMA window from 8 to 16 to expose more ILP for the gfx11 issue pipeline. Slice 7.6 splits the single-accumulator FMA chain in the phase-3 inner gemv into 8 independent accumulators with a final 7-add reduction, then packs adjacent FMA pairs into gfx11 VOPD v_dual_fmac_f32 for RDNA3 dual-issue — Mistral Q8_0 21.7 → 22.9 tok/s (+5.5%). Tokens byte-identical to slice 7.5. Phase 9 (residual gemv / down_proj) retains the slice-7.5 pattern — applying multi-acc there wedges MES firmware at Mistral's K=14336 (bisected 2026-05-17 to the amdgpu: failed to remove hardware queue from MES fingerprint, where MES stops responding to host control messages). Slice 7.7 adds a 5-second time-based watchdog in std/kfd.mlr + std/hip_kfd.mlr that catches the wedge fingerprint, prints one [KFD-WEDGE] diagnostic, short-circuits subsequent dispatches, and skips the blocking DESTROY_QUEUE ioctl in teardown — launcher exits in ~13 s instead of the prior 60+ s opaque hang. Slice 7.9 structurally splits phase 9 (o_proj) and phase 17 (down_proj) into two pieces with a cross-WG barrier between them — gemv → intermediate scratch + residual-add. The gemv now matches phase 3's proven shape (multi-acc + VOPD + plain store) instead of the wedge-triggering dependent-VALU reduce → VMEM residual-load → store shape. Mistral 22.9 → 24.9 tok/s (+8.7%); Llama-1B / Llama-3B / Qwen3 keep the monolithic phase-9 emitter and stay byte-identical. The Q8_0 GGUF path (default, lighter RSS) and the bf16 safetensors path (heavier mmap+heap duplication, used for sub-ULP PT bisection) produce essentially identical tok/s.

LLM CPU Inference — Ryzen 9 7900X

The same compiler, AVX2 backend, no GPU. MLRift programs (examples/qwen3_generate.mlr, examples/llama3_generate.mlr, examples/qwen3_14b_q8_generate.mlr) load HF safetensors or GGUF weights via mmap, run AVX2 matmul / SiLU / RMSNorm / attention kernels through a raw clone() + futex thread pool, and emit greedy decode. Hardware: AMD Ryzen 9 7900X (12 c / 24 t, AVX2). MLRift output is bit-identical to llama.cpp greedy on all rows. The Llama-3.2-1B row uses MLRIFT_CPU_BF16=1 which dequantises Q8_0 weights to bf16 once at load and routes through MLRift's AVX2 2-wide bf16 matmul. Methodology in docs/bench_2026-05-13.md, BENCH_QWEN3.md, and BENCH_QWEN3_14B.md.

MLRift CPU beats PyTorch CPU bf16 on every model where a like-for-like baseline exists, and now uses less RAM doing it. 14B compares MLRift Q8_0 vs PyTorch BF16 — different storage dtypes, real-world end-to-end comparison rather than arithmetic-parity. PyTorch F32 at 14B requires ~56 GB RAM and does not fit on the 30 GB test box, so it is omitted. Llama-1B and Qwen3-VL-4B compare PyTorch CPU bf16 (MKL+oneDNN under the hood) against MLRift's AVX2 2-wide bf16 inner loop (mm_worker_bf16_f32_avx2_naive_2w). Slice 4.27 (2026-05-13 mem opt) drops the Llama-1B CPU bf16 peak RSS from 3.63 → 2.39 GB (-34 %) by issuing madvise(MADV_DONTNEED) on the Q8_0 GGUF pages immediately after each tensor's bf16 dequant, then re-using the tied lmhead bf16 buffer for per-token embedding lookup so the embed Q8_0 region can also be evicted. Across all rows MLRift's RSS sits below PyTorch's because the safetensors / GGUF file is mmap'd and Python+transformers metadata is absent. Llama-1B prompt: "hello", N=20.

60 M-Neuron Spiking-Net Benchmark

LIF spiking network, 60 million neurons, 240 million synapses (sparse CSR, K = 4), 2 000 timesteps, identical algorithm across every runtime. Hardware: AMD Ryzen 9 7900X (12 c / 24 t) + AMD Radeon RX 7800 XT. Spike counts are bit-identical across MLRift CPU, MLRift GPU, and cupy; PyTorch differs by ≈ 0.0003 % due to a HIP uint64 remainder fallback. Methodology in docs/bench_60m.md.

MLRift GPU runs the entire sim — CSR build, decay, LIF step, spike delivery — as four @kernel functions compiled to a single ROCm code object. The native-ISA row uses the AST-walking AMDGPU emitter (src/format_amdgpu.mlr) that emits all four kernels directly into a code object — no hipcc, no LLVM. Spike count is bit-identical to the HIP path over 120 billion neuron-step computations, including IEEE-correct f64 division and CAS-retry atomic_add_f64. Wall-time is essentially identical (+0.16 %); the win is the build pipeline — 0.92 ms .co build vs hipcc's 482 ms (524× faster) and zero toolchain dependency.