Findings.
Small models that run where your data lives. The code is open at github.com/inertialabsxyz/fornace.
-
A scheduler cuts the slowest first token of 8 agent requests to Qwen2 on Metal from 8.3 s to 0.9 s, and doubles the tokens per second at BF16
With continuous batching, requests join and leave one batch, and the typical first token comes after 180 to 240 ms. llama.cpp still gives 1.6 to 1.9× the tokens per second, because Candle's matmul kernels cost about 12 ms a step for 2 to 8 rows.
-
With 8 requests at once from one agent, llama.cpp gives 2.5× the tokens per second of Fornace on Metal; with one request, it is only 1.1× faster
Fornace, llama.cpp, and mistral.rs on one Mac, with the same weight bytes and one HTTP client. With one request, llama.cpp decodes 1.08 to 1.14× faster, and the difference is GPU time in each step. With an agent load, Fornace ran one request at a time.
-
A 0.5B draft makes Qwen2.5-1.5B decode 1.35× faster on code and 38 % slower on prose on Metal
On free prose, the draft makes decode 38 to 68 % slower at every setting, because checking K + 1 tokens in one pass costs 2.2 to 2.6 decode steps. On code that copies from its own context, a Q4 draft at K = 8 gives 1.35×, with the same text as the 1.5B alone.
-
Prefilling a shared system prompt once cuts the time to first token of 4 Qwen2 sequences 2.7× on Metal, and a paged KV cache is not needed for it
A contiguous cache that copies the shared prompt into each sequence gets the same gain. With Candle's Metal gather kernel, a paged cache makes each answer 47 to 61 % slower.
-
Q4 weights make Qwen2 decode on Metal 1.85× faster, at a 23 % perplexity cost
Q8 decodes 1.40× faster for a 0.4 % perplexity cost. The work also found two Candle bugs: BF16 RoPE tables that made perplexity 3.2× worse at 2,048 tokens, and a quantized Metal matmul that ignores its input offset.
-
Batch 4 gives 1.6× the total decode speed of batch 1 for Qwen2 on Metal, but each sequence decodes at 0.40×
Candle's Metal matmul uses a matrix-vector kernel only for one row. From two rows it uses the tiled GEMM, so batch 2 gives less total than batch 1, and batch 8 gives 2.7×.
-
Fused attention makes a 4,096-token Qwen2 prefill on Metal 4.5× faster and lowers its peak from 3,403 MiB to 1,470 MiB
Candle's fused Metal kernel,
candle_nn::ops::sdpa, never stores the score matrix. Prefill goes from 1,265 to 5,770 tokens per second, and decode is 75 % faster. -
A 1 MiB KV cache tensor holds a 512 MiB buffer in each layer, and makes a 4,096-token Qwen2 prefill on Metal peak at 14.5 GiB instead of 3.4 GiB
A reuse rule in Candle's Metal buffer pool lets a small tensor take a large free buffer. A cap on that rule brings the peak to 3.4 GiB, with no change in speed.
-
A non-contiguous KV cache cuts Qwen2 decode speed by 59 % at 2k context in Candle on Metal
One
.contiguous()call takes decode from 42 to 102 tokens per second on a 0.5B model at a 2,049-token context, on an Apple silicon Mac.