Together AI: a fintech runs coding agents on 56 B200s at 256K context

Together AI says a global fintech runs its internal coding assistant on GLM-5.2 across 56 B200s, and that the customer evaluated a 1M-context configuration and turned it down. The case study, published on 18 September 2026 and written by Together’s Ankit Goyal, doesn’t name the company. It describes a firm that ships financial products to millions of users across dozens of markets, so the coding workload sits on the critical path of how fast it ships.

The production endpoint runs at 256K context as 14 replicas of 4 B200s each, sized for concurrency rather than raw throughput. Together writes that doubling context to 1M would have cut the concurrency headroom this workload depends on, so the team stayed at 256K/512K. That’s a pointed decision, because the 1M window is what Z.ai leads with in its own documentation.

GLM-5.2’s model card advertises a solid 1M-token context under an MIT licence, and scores the model at 81.0 on Terminal Bench 2.1 under the Terminus-2 harness, against 85.0 for Claude Opus 4.8. Z.ai’s own developer documentation lists the context length as 1M and maximum output at 128K tokens. It’s a mixture-of-experts design, with 256 routed experts and 8 active per token in the published config.

But the fintech’s measured traffic sits well below that ceiling. Together reports input sequence lengths and request rates at three percentiles, and the pattern is few requests carrying very large prompts. That is why concurrency, rather than tokens per second, became the design priority.

Measure	p50	p90	p95
Input sequence length	81K tokens	163K tokens	178K tokens
Requests per second	3	6	7

What broke, and what fixed it

The post is unusually specific about a failure, which is rare in this format. A migration reshaped the endpoint toward fewer, larger replicas at the same total footprint. A few days later prefill capacity hit near 100%, requests queued for one to three minutes, and decode throughput collapsed to roughly 5 tokens per second.

Together API Support traced a single 192-second request through the metrics data and found it wasn’t compute-bound. Instead it had spent almost the entire span queued behind a 2.3M-token pending-prefill backlog from other requests, rather than behind its own 250K-token prompt. The fix was a live configuration change rather than a redeploy: restoring a tuned cache-session-aware routing policy in place of the default cache-aware-by-hash policy, and widening the max-inflight-per-worker threshold. Together says it went out the same day with zero downtime.

You can’t forecast a burst that’s driven by a hundred different engineering teams independently deciding to lean on their coding agent harder this week.

Ankit Goyal, Together AI

That line explains why the customer set its requirements around autonomy as well as raw performance. Under the earlier arrangement, every team that wanted coding agents filed a capacity request, and the platform team absorbed the coordination. Planning stopped working once adoption became unpredictable in both timing and size, because no forecast survives a hundred teams deciding independently.

The caveats, and the research alongside it

This is a vendor case study, and Together is selling the product it praises. The customer is anonymous, nobody outside the two companies has confirmed the numbers, and no price appears anywhere in the post. Still, the shape of the failure is specific enough to argue with. What’s checkable is the shape of the problem, which matches how inference bills actually behave.

Academic work is circling the same cost question from the other side. A paper submitted to arXiv on 7 July 2026 by Foivos Charalampakos, Md Ibrahim Ibne Alam, Iordanis Koutsopoulos and Koushik Kar models multi-model serving as inference networks. The authors prove the optimal activation policy has a threshold structure: query the lowest-cost model first, and invoke the expensive one only when confidence falls below a computed threshold. Their experiments with open source models report substantial cost reductions while meeting a specified performance budget.

Both point the same way, though from opposite directions. Serving cost is being decided by routing and scheduling policy, not by picking a bigger model or a bigger window. The GLM-5 technical report says the architecture cuts inference costs while holding long-context fidelity, but this deployment still shows a buyer capping the window to protect throughput under load.

Together says the customer is already scoping a dedicated GLM 5.1 node in a new region, for a chat-style customer-support workload rather than agentic coding. The thing to watch is whether other dedicated-inference buyers make the same trade, and cap context deliberately because concurrency is what their traffic actually needs.