Local Claude CLI finally makes sense on a DGX Spark cluster

A 98 GiB mixture-of-experts checkpoint does not fit on a 128 GB unified-memory box alongside its own KV cache. Two boxes and a tensor-parallel split do fit — if the memory is budgeted before launch, not after.

The workload that justifies it is a terminal coding agent. Claude CLI reads whole repositories, carries enormous context and runs for hours; quantise hard and it writes plausible nonsense, cut the context and it forgets the file it just edited. 262,144 tokens of context at 130 tok/s sustained — 242 at peak — across two nodes is where pointing it at your own hardware stops being a demo.

Every figure below is measured on one instrumented build.

Which Claude does it replace?

Sonnet 5. Not Opus 5. That is the evidence, not the price.

There is exactly one genuinely comparable measurement — a single harness running all three models itself, rather than splicing vendor-reported numbers from different scaffolds. On Artificial Analysis’s Intelligence Index v4.3:

LiveBench agrees on the ordering: 76.2 / 76.0 / 80.1.

Read the last row before celebrating. AA-Omniscience penalises confident wrongness, and only the open model is negative. It knows less and admits it less. That is the tax you pay, and it is why the frontier model still wins anything where being wrong is expensive.

What it costs to not pay for it

Two DGX Sparks are $9,398 at current MSRP — up from $3,999 a unit, on memory supply. Measured at the wall under load, the pair draws about 300 W, so electricity is roughly €1.90 a day: 1 % of the bill it replaces, and not worth modelling carefully.

The workload actually run through this gateway — 331,125,467 input tokens against 4,903,739 output, at an 89 % cache hit rate — priced at list:

The strongest cost argument is not the token price — it is the cache. Anthropic’s prompt cache is billed on write, expires after five minutes, and the documentation is explicit that parallel requests on a shared prefix will not all hit it: a swarm fanning out nineteen concurrent calls mostly pays full price. A local prefix cache is free, untimed, and has no such restriction. At a 67.5 : 1 input-to-output ratio, that asymmetry is where the money is.

And the honest counter-case is not Claude at all. Renting these same weights costs about $0.15 / $0.47 per million, and a hosted NVFP4 endpoint is measured at 138.7 tok/s against this cluster’s 59.8 single-seat. Against that, the hardware needs roughly 10.8 % sustained duty to break even — against Claude it needs 1.2–3.1 %.

At a glance

Topology

TP=2 for interactive work. It sets one hard requirement: a direct link, not a switch path — a per-layer collective runs thousands of times a second. Pin the collective library’s interface explicitly or it selects the management NIC.

How it fits together

Memory budget

Those rows sum to 64.46 GiB of model bytes per rank against a 98.1 GiB checkpoint, which only reconciles one way: TP=2 splits the expert and attention weights across both ranks, and replicates the embedding tables and the drafter on each. Splitting is not the same as halving.

Three consequences:

• Gate per role. 103.2 vs 97.7. A “100 GiB free” threshold admits a head that dies and refuses a worker that would run.
• Never gate on one sample. With a co-tenant loading and unloading, free memory swings 17.6 GiB between 5-second samples. Take the lowest of three.
• Under-provisioned launches die late. 77 GiB free against a 103.2 GiB need: no OOM kill, no swap storm — exit code 1 at 88 s.

KV and concurrency

16,392 B/token/rank × 262,144 tokens = 4.30 GB per seat. Pool 28 GB → 1,708,150 tokens → 6.52 seats → floor 6.

Only 12 of 48 layers are full attention; the hybrid’s recurrent state pads the page. Take the pool size from the engine’s own startup report, not from a formula you derived.

262,144 is not an arbitrary window — it is the last one that works. On this model’s own long-context evaluation, multi-needle retrieval scores 95.98 at 128K and 93.00 at 256K, then collapses to 40.53 at 512K and 26.44 at 1M. Simple single-needle retrieval survives to 1M; reasoning across several needles does not. Running at 262,144 sits on the last measured good point, and the “1M context” claim on models of this class should be read as retrieval-only.

Raising concurrency costs no KV. The pool is fixed; seats only decide occupancy, plus CUDA graphs. Past the pool the engine queues — it does not OOM and does not refuse to start. So the ceiling is advice: warn, then store what the operator asked for.

Measured over-subscription. This cluster runs 8 seats at the full window: 8 × 262,144 = 2,097,152 tokens against a 1,708,150-token pool — 23 % beyond what fits. Nothing fails. vLLM admits what the pool holds and queues the rest, sustaining a depth of 25–37, and aggregate throughput rises: median 129.6 tok/s and peaks of 242.5, against 106.3 at six seats. A gate that had clamped 8 down to 6 would have cost that throughput to protect against a failure that does not occur.

Sizing the pool: context × concurrency

KV per seat is almost exactly 1 GiB per 65,536 tokens at 16,392 B/token/rank. Non-KV cost per rank is fixed — 77.12 GiB on the head, 71.62 on the worker — so on a 121.7 GiB box the head can hold 44.58 GiB of KV. That, not the model, is what bounds concurrency.

Read it as a budget, not a limit. Two consequences worth acting on:

• This cluster is under-pinned. 26.1 GiB of KV is pinned where 44.6 fits. Raising --kv-cache-memory toward ~40 GiB would serve 9–10 seats at the full window with no queueing, leaving a few GiB of margin for the host.
• Halving the context roughly doubles the seats. 262,144 → 131,072 takes 6 seats to 13 at the same pool. For swarmed agents that rarely fill the window, that is the cheapest concurrency available.

Cold start: 499 s on this run

That is one start, instrumented end to end. Other starts of the same pair have taken 762–788 s: the phases are the same, their durations are not. Treat the shape as the finding and the total as a sample.

The shard set is walked twice — base weights, then the drafter’s. A single counter reaches 28/28 and restarts at 0, which reads as a crash loop. Two bars, not one.

Speculative decoding

95.0-second sample, both ranks, 6 seats busy:

Draft tokens run 1.30× above generated. Summing the wrong series inflates a dashboard by that factor; only generation_tokens is what a client receives.

Throughput

Generated, from 255 samples across ~20 minutes of sustained load at 8 seats:

The spread is the point. The same cluster, same load, same six minutes: 56.3 to 242.5 tok/s, a 4.3× range across six-second windows. Any figure quoted from one window is noise. Sample over a minute, and label cumulative totals with their window — a counter that restarts with the gateway must say so.

What to use it for

The honest boundary: when you need logic, not judgement. This is not the model to reach for when the output is design — a UI, a layout, copy that has to land — or when a single answer has to be exactly right the first time. Frontier hosted models are still better at taste and at one-shot precision, and pretending otherwise wastes everyone’s afternoon.

What it is very good at is volume over time.

One assumption worth dropping: that front-end work belongs on the hosted side. It is the intuitive call and the measurements do not support it — on WebDev Arena this model scores 1635, ahead of every Claude Opus 4.x (best 1559). The real divide is not logic-versus-design; it is single-shot versus sustained. Open MoE models hold their own on anything resolved in a few steps and degrade disproportionately across long horizons.

The workload that makes the case. This gateway’s cumulative counters since it started: 331,125,467 prompt tokens against 4,903,739 generated — a 67.5 : 1 input-to-output ratio, at an 89 % prompt cache hit rate. That is the signature of swarmed research agents: read enormous amounts, emit comparatively little. It is also the shape of workload that is most punishing to meter, because the bulk of it is input and the bulk of it is repetitive.

On a subscription, that pattern runs into quota long before it runs into capability. Here it runs into nothing — the marginal cost of the 331 millionth token is the electricity to produce it.

The gateway

Clients know one address. Everything else is the gateway’s problem.

Point Claude CLI’s base URL at the gateway and give it a model name the gateway recognises. That is the whole integration: an Anthropic-shaped API, streamed with read1 and never buffered, and nothing in the agent aware that two machines are behind it.

Handover, when a scheduler owns the boxes

• One reservation per node per generation, pinned, weighted as the whole box.
• Gate the claim, not the cleanup. A held node still allowed to accept work reloads a model into the memory you just freed.
• Ask before you let go. Acquire generation N+1 before releasing N; a 36-second gap was enough for a scheduler to put a model back on the card.
• Drain in flight under a deadline; never evict under a live stream.
• Name the blocker in the status surface: “co-tenant holds 17.6 GiB”, not “waiting”.

Five failures

Exposure

Relay inference. Answer everything else.

Closing the preflight while leaving simple GET open achieves nothing: no preflight is sent for a simple request. Fix the rule, not the reported instance.

Checklist

• collectives bound to the direct link, confirmed in the startup log
• both ranks: same context, same seats, same pinned KV
• engine’s KV report matches the predicted seat count
• per-role memory requirement cleared on three consecutive samples
• a captured real client request replays end to end
• gateway restarts without disturbing the pair
• in-flight depth returns to 0 after a forced backend timeout
• cross-origin GET to an API path answered by the gateway
• token totals reconcile as deltas against engine counters
• sustained rate measured over 60 s

Things that will bite you

A guard script you forgot. A per-minute cron doing docker start undoes a manual stop within 60 s; you will blame your own unload path.

The lucky sample. Any gate reading a 17.6 GiB-swinging number once eventually reads it at the wrong moment, and the failure lands minutes later somewhere else.

A green test that has never failed. Run each new check against the build immediately before its own fix. Two of mine passed against unfixed trees and would have certified fixes that were never written.

Prose is not evidence. A commit message describing a change is not the change. Read the diff first.

distributed systems