AWS pairs OpenCode with Kimi K3 on Bedrock, but its benchmark link 404s
AWS published a walkthrough on 23 September for running an AI coding agent on open weight models, and it tells you twice to compare those models on the Artificial Analysis Coding Index. But that link returns a 404, and on the index that is live, not one of the three models appears in the coding agent AWS is recommending.
The post pairs OpenCode, an open source coding agent that runs in a terminal, with three open weight models on Amazon Bedrock. Those are Moonshot AI’s Kimi K3, OpenAI’s GPT-OSS 120B and NVIDIA’s Nemotron 3 Super 120B. A single opencode.json file sends planning work to one model and code generation to another.
AWS’s argument is that your code stays in your own account and you pay per token instead of per seat. The numbers behind that pitch hold up. Bedrock’s Flex tier costs 50 percent less than Standard, the global cross-Region profile runs about 10 percent below a geographic one, and default limits sit at 100M tokens per minute and 10K requests per minute.
The model specs check out too. AWS says Kimi K3 activates 104B of its 2.8T total parameters per token, and Moonshot’s own model card on Hugging Face gives exactly those figures, plus the 1M token context. The catch comes at the install step, because AWS pulls OpenCode from the sst/tap Homebrew tap, while OpenCode’s own install page uses anomalyco/tap and doesn’t mention the older one.
What the index actually measures
The live page sits at artificialanalysis.ai/agents/coding-agents, and it’s called the Coding Agent Index, currently v1.5. It’s a composite of three equally weighted benchmarks: DeepSWE v1.1 at 113 tasks, Terminal-Bench 4.0 at 66 tasks, and SWE-Atlas-QnA at 124 tasks. AWS instead describes the composite as SWE-Bench, Terminal-Bench and SWE-Atlas, but SWE-Bench isn’t in it.
The bigger gap is what the index ranks, which is agent and model pairs rather than models on their own. It carried 20 of those pairs when we read it on 26 September. OpenCode appears once, paired with Z.ai’s GLM-5.3, but the listed inference provider is Z.ai, not Bedrock. Kimi K3 appears only under Moonshot’s own Kimi Code CLI, and neither GPT-OSS 120B nor Nemotron 3 Super 120B is there in any harness.
| Agent and model | Index score | Cost per task | Time per task |
|---|---|---|---|
| Claude Code, Opus 5.5 (max) | 66.0 | $13.04 | 64.5 min |
| Codex, GPT-6 Sol (max) | 56.7 | $2.99 | 22.3 min |
| OpenCode, GLM-5.3 | 53.6 | $4.24 | 48.1 min |
| Kimi Code CLI, Kimi K3 | 51.9 | $5.05 | 61.5 min |
Read that table and the open weight case does survive, though not with the models AWS picked. OpenCode on GLM-5.3 scores 12.4 points below Claude Code on Opus 5.5 while costing roughly a third as much per task, and both of those are our own arithmetic on the index figures. GLM-5.3 is the model Z.ai launched in August while staging the release of its weights over cyber risk. The trade of some capability for a much smaller bill is the same one we worked through in open weight versus closed API costs.
The parity claim rests on a security benchmark
AWS’s performance parity bullet cites CrowdStrike, whose fine-tuned Nemotron hit 96% valid query accuracy against GPT-4o at 61% and Claude Sonnet 4.5 at 94%. CrowdStrike’s own post says the task was translating plain English into CrowdStrike Query Language, which analysts use to search security data. So that isn’t a coding benchmark, and the margin over Claude Sonnet 4.5 is only two points.
While we are evaluating multiple models, the model we are working to operationalize in the context of this blog is Llama-3.3-Nemotron-Super-49B-v1.5
CrowdStrike, footnote to its Nemotron customization post
That means the model behind the 96% is a 49B Llama Nemotron adapted with LoRA, not one of the Nemotron 3 models in AWS’s walkthrough. CrowdStrike says it is only beginning early evaluations of the Nemotron 3 family. Even so, none of that sinks the setup AWS describes, and the routing pattern is a sensible way to run a coding agent on your own account.
What’s missing is the evidence you’d use to pick between the three models, because the index AWS points at ranks only one of them, and under a rival agent. So watch whether Artificial Analysis adds OpenCode rows for the Bedrock models, and whether AWS fixes the link.
Get the daily rundown
One email each weekday with the AI news that matters, every claim linked to its primary source.
Free, one email each weekday, unsubscribe in one click. We never sell or share your address.
