Expert offloading that stays byte-correct at a third of the VRAM — the design, the numbers, the wall we hit, and why we gave the whole thing away.
A modern MoE model activates a handful of experts per token but stores hundreds. On a 16 GB card that means most experts live in host RAM and get streamed in over PCIe — and the miss stream, not raw VRAM, becomes the thing that decides your speed. The obvious question: can you predict which experts fire next and prefetch them ahead of the router? We spent from April on it. It works better than we expected, and it still wasn't a business. Both of those are true; here's the honest version of both.
Start here, because it's the part everything else hangs on. The naive way to run a small expert cache is to mask the router to whatever you have resident — force it to pick from cached experts. That produces mode-collapsed garbage at small caches, every single time. We tried it. Don't.
The fix is a closed loop: never touch the gate. Let the router make its real top-K selection, then — inside the per-tensor callback, before the FFN op consumes the result — read that selection, compute which experts are missing, pick cache slots whose current occupant isn't in this batch's selection, copy the missing experts in on the compute stream, and update the map. Stream ordering does the synchronization. The predictor is allowed to be wrong; the swap makes the forward pass right anyway.
Tiers are sized by predictor confidence, not storage — a framing borrowed straight from manufacturing
statistical process control, where you size for variance under constraint. High-confidence band → VRAM (~25–35% of
experts); moderate → pinned host (~+50%); the rest cold. One hard floor falls out of it rather than being tuned:
n_slot ≥ 2 × top_k — half the slots must be safe-evict targets when the gate fires.
Measured on Qwen3-30B-A3B and Qwen3-235B-A22B, on the Quadro RTX 5000 + 125 GB host, early May 2026.
| Result | Number |
|---|---|
235B trace, L1+L2 hit rate vs static --cpu-moe | 0.786 → 0.896 (+11 pp) |
| 235B cold-tier misses | −51.5% |
| 30B-A3B Q8, output vs all-in-VRAM baseline at 64/128 slots | byte-identical |
| 30B-A3B still coherent at 40/128 slots (31%) | 95–99% L1+L2 hit |
| Predictor cost (AVX2 CPU) | 0.10 / 0.21 ms |
The finding I care about most is the third row: byte-identical output with half the experts resident, and still-coherent output at a third — and that comes from the closed loop, not from cache size. Prediction made it fast; the swap-before-FFN made it the same model.
We ran the harder question too: how predictable is expert routing, really? On a domain-mixed hostile corpus, top-8-of-128, 30B and 235B agreeing to within 0.005 everywhere:
Here's the part most writeups leave out. This is a working prototype — real numbers, real correctness — and it is not a product, and we are not the team to make it one. Taking it the last mile (clean integration behind a stable interface, breadth across model families, the grind of productionizing) is a different job than proving the idea, and it's a job we honestly didn't have the resources to finish. We poked at whether it was a paper or something protectable; it wasn't the pot of gold, and chasing that would have been us lying to ourselves.
So the better ending: hand it to the people already building the thing it belongs in. The full design, the traces, the predictor weights, and the hardware fixes went to the FreeToken team, free — take any of it that's useful. A working idea in a drawer is worth nothing; the same idea where a capable team can pick it up is worth something. That's not a consolation prize. That's the point of a lab that means the "give-back" it talks about.