Benchmark the agent loop, not one tokens-per-second number.

Agent serving is not sustained chatbot serving

Magnitude's October launch is interesting less as a new desktop app than as a falsifiable claim about workload shape. Its maintainers argue that local agents need an engine tuned for long sessions, repeated instructions, concurrent contexts, dynamic memory, and consumer hardware. The right response is not to accept or dismiss the reported speedup. It is to define the workload an engine must survive.

The public Apache-2.0 repository showed 6,718 stars, 464 forks, and 37 open issues when checked on October 14. The launch thread reached 194 Hacker News points and 97 comments. Magnitude reports up to 92 percent faster decode and 9 percent faster prefill than llama.cpp on one Metal configuration, plus 19 percent faster decode, 23 percent faster prefill, and 27 percent less per-agent memory on one CUDA configuration. Those are vendor measurements on selected model, quantization, context, and hardware combinations. They are leads for reproduction, not a general performance law.

The same discussion produced the missing half of the evaluation. Users asked for MLX as the Apple Silicon baseline, reported slower results on an M5 Max, found multi-GPU gaps, questioned whether single-stream decode was the real bottleneck, and asked how shared prefixes behave across sleeping and waking agent sessions. Current issues add wrong device selection, model-load failures, memory regressions, Windows crashes, Vulkan kernel failures, CPU hangs, strict chat-template breaks, and requests that fail under long-context pressure.

This distinction keeps the article out of product-review territory. Magnitude is a useful bounded case because its code, design notes, benchmarks, releases, and issues expose competing claims. The method applies to llama.cpp, MLX, vLLM, SGLang, Ollama, LM Studio, and any future local or hosted engine.

Separate prefill, decode, cache, and tool time

An agent turn has at least four clocks. Prefill processes the input tokens: system instructions, tool schemas, history, retrieved material, and the latest tool result. Decode produces new tokens. Tool time is the pause while code, a browser, or an API runs. Queue time appears when sessions compete for compute or memory. A single average tokens-per-second number usually measures only part of decode.

The key-value cache stores attention state for already processed tokens. When two requests share a prefix, a prefix cache can skip repeated prefill work. Agent harnesses are unusually prefix-heavy because the system prompt and tool definitions can remain stable across many turns. vLLM's automatic prefix caching identifies reusable blocks from token content and prior prefixes. The benefit depends on whether the blocks stay resident long enough to be reused.

stable system prompt + tool schemas + project rules
                     |
              reusable KV prefix
                     |
turn N: user -> model -> tool call -> tool result -> model
                                      |
                             session pauses here
                                      |
turn N+1 appends history while other sessions compete
                     |
        cache hit, partial hit, eviction, or recompute

Recent research makes the capacity problem explicit. CacheScout models agent execution transitions so reusable contexts are not evicted solely because they were not the most recently used. KVSET estimates the minimum KV working set required for a target hit rate. Tutti explores SSD-backed KV state when long contexts exceed accelerator and host memory. These systems target different environments, but they agree on the mechanism: workload-aware reuse and placement matter at least as much as a kernel's best-case decode rate.

Magnitude says it tunes kernels on the actual device, allocates memory as sessions grow, frees it when sessions stop, and shares prefixes through a tree structure. That can improve one layer of the stack. It does not answer whether a model produces the same tool call after quantization, whether the chat template survives an interrupted tool turn, whether the operating system picks the correct GPU, or whether an out-of-memory request becomes a structured retry instead of an empty success.

Write a workload contract before choosing an engine

The contract should describe production traces, not an attractive demo. Freeze the agent harness version, system prompt, tool schemas, model artifact, quantization, sampler, hardware, drivers, power state, concurrency, context distribution, and acceptance thresholds. Name the semantic checks that prevent a faster but different answer from passing.

version: agent-inference.workload/v1
engine_candidate: magnitude-0.2.x
baseline: llama.cpp-pinned-build
hardware:
  machine: macbook-pro-m4-pro-48gb
  os_build: pinned
  power_mode: ac-high-power
model:
  id: qwen-family-model
  artifact_sha256: REQUIRED
  quantization: q4
  context_limit: 65536
agent_trace:
  sessions: [1, 3, 6]
  turns_per_session: [8, 32, 64]
  stable_prefix_tokens: [4000, 12000]
  tool_pause_seconds: [0, 2, 30, 180]
gates:
  semantic_pass_rate: 1.0
  tool_call_schema_pass_rate: 1.0
  p95_turn_latency_regression: "<= 5%"
  peak_memory: "<= machine_budget"
  retry_contract: [503, retry_after, no_empty_success]
  prefix_cache_hit_rate: "reported with capacity"
stop_on:
  - silent_failure
  - output_drift_on_must_pass_case
  - wrong_device_selection
  - unrecoverable_session

Do not put every metric into one weighted score. A 40 percent decode improvement cannot compensate for one malformed payment tool call or a session that silently dies after a long context. Use must-pass consequence gates for correctness, authority, data integrity, and recoverability. Use weighted measures only to compare acceptable candidates on latency, throughput, memory, energy, install friction, and coverage.

Replay real agent turns with controlled counterfactuals

Collect traces from the tasks you expect to run: repository inspection, edit-test loops, browser research, document analysis, or local automation. Remove secrets and effects. Store exact tokenized inputs, expected tool calls, tool results, and must-pass assertions. A prose repetition task is useful for isolating throughput. It is not enough to certify an agent workload.

for engine in [baseline, candidate]:
    pin(engine, model_hash, sampler, chat_template)
    for cold_state in [fresh_install, tuned, warm_cache]:
        reset_machine_state(cold_state)
        for concurrency in [1, 3, 6]:
            for trace in replay_corpus:
                result = run_trace(
                    engine=engine,
                    trace=trace,
                    tool_pause_schedule=trace.tool_pauses,
                    concurrency=concurrency,
                    capture=[tokens, timing, cache, memory, energy, logs]
                )
                assert_semantics(result, trace.must_pass)
                record(result, hardware_manifest, engine_commit)
compare_paired_runs(confidence_intervals=True)

Run paired trials in randomized order to reduce thermal and background-load bias. Warm up only when production will be warm. Report first-load tuning separately. Pin CPU and GPU power modes where possible, record temperature and clock behavior, and repeat enough times to show variance. If the candidate auto-tunes, persist the tuning artifact's engine version, hardware fingerprint, model hash, configuration, duration, and invalidation reason.

A fair baseline is configured by someone who knows it. The HN thread correctly challenged a comparison against llama.cpp without an MLX baseline on Apple hardware. Baselines should use appropriate flash-attention, batch, context, KV precision, offload, and speculative-decoding settings. If candidate and baseline use different KV precision, say so; the memory and quality trade is part of the result.

Test four cache shapes. First, identical stable prefixes with short appended turns. Second, partially shared prefixes across related agents. Third, long sessions that exceed the intended working set. Fourth, adversarial churn where unrelated sessions force eviction. Report hit rate together with allocated capacity and latency saved. A high hit rate on an oversized cache does not prove efficient capacity planning.

Tool pauses matter because sessions are not continuous streams. Run immediate, short, and long pauses while other sessions generate. Observe whether paused KV state stays on the accelerator, moves to host memory, is recomputed, or is evicted. Record time to resume, memory reclaimed, and which session loses latency. This is where an agent-specific scheduler should earn its name.

Performance evidence is invalid when behavior drifts

Quantization, fused kernels, cache compression, speculative decoding, templates, and sampling can change output. Use deterministic fixtures where possible, but do not demand exact text when several answers are valid. Assert the consequences: tool name, JSON schema, required arguments, prohibited arguments, calculation, citation set, permission boundary, and final state.

The Magnitude repository includes RULER-style long-context fixtures and Berkeley Function Calling Leaderboard fixtures. That is useful evidence of testing intent. Your acceptance corpus still needs local cases that match your tools. Include long histories, same tool names with similar schemas, large tool results, Unicode paths, cancellation, malformed tool results, a context that barely fits, and a context that should fail cleanly.

Record route receipts for every run: engine commit, model and tokenizer hashes, quantization, template, tuning profile, hardware manifest, input trace, cache state, output assertions, timings, memory, logs, and verdict. Without that receipt, a benchmark winner cannot be reproduced after the engine, driver, model, or operating system changes.

The failures that a launch chart cannot show

Current Magnitude releases show why recovery belongs in the contract. Release notes describe changes that return a 503 with Retry-After when a request fails under memory pressure, instead of returning an empty response that looks successful. They also add memory-pressure logs and bound initial tuning time. These are product improvements, but they are also general design requirements: failure must be machine-readable, attributable, and safe to retry.

“Private and offline” also needs a boundary test. Verify where prompts, files, models, tuning artifacts, analytics, crash reports, update checks, and integration metadata travel. A local model call does not make a browser tool, package installer, telemetry client, or agent extension local. Capture traffic during first install, model download, normal use, error reporting, and update.

Adopt with a matrix, not a winner badge

Choose an engine for a bounded workload cell: one hardware family, operating system, model family, quantization, context band, concurrency range, and agent harness. Approve only cells that pass every semantic and recovery gate. A different GPU, model architecture, driver, or context range is a new cell until evidence says otherwise.

The practical verdict on Magnitude is “promising, testable, and early.” The repository and discussion provide more evidence than a closed launch page. The live issue matrix also makes broad claims unsafe. Teams with supported hardware and a local-agent need can run a serious evaluation now. Teams needing broad model coverage, mature multi-GPU support, or a managed production endpoint should treat it as an experiment until their own cells pass.

Frequently asked questions

Is decode speed irrelevant for agents?

No. Decode affects every reasoning and response step. It is simply one component. A faster decoder can still lose on prefill, cache misses, concurrency, memory pressure, tool resumption, or correctness.

Why not use one composite benchmark score?

A composite score can hide unacceptable behavior. Keep semantic correctness, isolation, authority boundaries, and recoverability as must-pass gates. Compare speed and efficiency only among candidates that pass.

Does automatic kernel tuning make deployment easier?

It can remove manual tuning, but it creates a generated deployment artifact. Record what was tuned, on which device, for which model and engine version, how long it took, and when it becomes invalid.

What is the best Apple Silicon baseline?

Use more than one when the decision matters. A well-configured llama.cpp path tests broad compatibility; an MLX-native engine tests the platform-specific fast path. Disclose versions and settings.

How many traces are enough?

Enough to cover distinct consequences, not every chat. Start with representative normal tasks, every high-risk tool, long-context cases, concurrency shapes, and known failure boundaries. Add production escapes to the corpus.

When should a benchmark be rerun?

After an engine, model, quantization, template, driver, operating system, hardware, agent harness, tool schema, context distribution, concurrency target, or acceptance threshold changes.