A six-year-old Turing card running a modern mixture-of-experts runtime at laptop-4060 speed — the two fixes that got it there, and the numbers.
FreeToken is a mixture-of-experts inference runtime built to keep only the hot experts in VRAM and stream the rest. We wanted it on the hardware we actually have — not an H100 — so the target was a Quadro RTX 5000: 16 GB, Turing, launched 2019, paired with 125 GB of host RAM. Here's what it took, and what it did.
| Model | Format | Card | Decode |
|---|---|---|---|
| Qwen3.6-35B-A3B | NVFP4 | Quadro RTX 5000 (2019) | 36–40 tok/s |
That's in the neighbourhood of what a current RTX 4060 laptop posts on the same class of model — from a card two GPU generations older. The whole point of an offloading runtime is that VRAM stops being the wall; this is that claim, measured on old silicon.
With the open PRs #24 and #19 applied, two things still stood in the way. Both are
small; both are the kind of thing that silently wrecks output if you get them wrong.
Qwen3.5-122B-A10B-NVFP4
release among them) carry a quantization_config.ignore list that excludes the shared expert from
quantization — so it must be loaded as bf16, not as the block-quant format the rest of the tensors use.
Miss that and the shared expert loads as garbage; the model runs, and quietly produces worse tokens. The loader
now honours the ignore list per-tensor.These are offered upstream — happy to send the merge and loader fixes as a PR to FreeToken so the next person with a Turing card doesn't re-derive them. The runtime is theirs; we just wanted it to run where we live.