Performance & speculative decoding

Speed levers: quants, context, MoE experts, speculative decoding and MTP — plus how Bedrova benchmarks honestly.

How fast a model runs comes down to the model, its quant, the context length, and your Mac — roughly in that order. This page covers the levers you control and how Bedrova reports speed.

The basic levers

Quant. A more heavily quantized model (4-bit vs higher precision) is smaller and usually faster, at some quality cost. This is the biggest single dial.
Context length. Longer prompts slow the prefill and grow the KV cache. Trim context you don’t need.
Model size. A 7B model on a 16 GB Mac will out-pace a 70B model on a 64 GB Mac. Match the model to the work.
Runtime format. MLX is the preferred text path on Apple Silicon. GGUF broadens text compatibility through llama.cpp, but is labeled broad compatibility, usually slower when an MLX path is also viable.
Thermals. Sustained generation heats the Mac and it throttles. Watch Settings → Hardware on long runs.

Speculative decoding

Speculative decoding pairs your model with a small draft model that proposes several tokens at once; the main model verifies them in bulk, which raises throughput when the draft guesses well. Turn it on per model where a compatible draft exists (Chat controls and the model profile). Bedrova reports the acceptance rate so you can see whether the draft is helping; rejected drafts roll back cleanly.

MoE expert control

Mixture-of-Experts models activate only some expert sub-networks per token. For models where it’s been shown safe, Bedrova exposes expert-count control so you can trade quality for speed. This is gated to a safe range per model — pushing experts outside what’s been validated is exactly the kind of foot-gun Bedrova won’t expose silently.

Multi-token prediction (MTP)

MTP predicts several tokens per step and can speed up models that support it. It’s enabled only where the runtime proves it safe for that model; otherwise it’s gated and the UI says so. On frontier models in particular, MTP and multi-request concurrency are validated separately before they’re switched on.

How Bedrova benchmarks

Every speed number in the app comes from the same built-in benchmark: load time, time-to-first-token, prompt and generation tokens/sec, and peak unified memory. Published numbers on the benchmarks page follow a few rules:

Each run states the exact model, quant, runtime-pack version, runtime format (MLX/GGUF/Core ML/provider pack where relevant), context length, and Mac.
Numbers are single-machine measurements — yours vary with context, thermals, and what else is loaded.
Methodology and raw numbers are published together. No cherry-picking.

Public figures land with a formal, reproducible run at launch; until then the benchmarks page shows placeholders rather than preliminary internal numbers dressed up as final.