AWS trains gpt-oss-20b to 84% on Wordle, or 37% by its own curve

AWS published a hands-on guide on 23 October for multi-turn reinforcement learning on Amazon SageMaker AI. The run in it took an open-weight model from a 61 to an 84 percent Wordle solve rate for about $339 in training tokens. The same adapter scores 37 percent on the training job’s own validation curve. That lower number isn’t a mistake, because the SageMaker AI hyperparameters reference is what recommends the setting that produces it.

Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI scores a whole episode rather than a single answer. SageMaker sends one task row to an agent you host on Amazon Bedrock AgentCore. The agent calls the policy model being trained, acts over as many turns as the task needs, then returns one reward at the end. Group Relative Policy Optimization, the PPO variant introduced with DeepSeekMath, then uses the spread of rewards across several rollouts of the same word to update a LoRA adapter.

That’s a different proposition from the usual prompting, retrieval or fine-tuning decision, because there are no target responses in the dataset at all. Each training row holds one secret word, and the agent gets six guesses with per-letter feedback after each one. The sample repository draws 640 training words and 128 validation words from the answer list with no overlap.

AWS picked gpt-oss-20b, a 21B-parameter Apache 2.0 model with 3.6B active parameters, because it has the lowest per-token price of the four supported base models and its reasoning effort is settable per request. The other three are Amazon Nova 2 Lite, Gemma 4 31B instruction-tuned and Qwen 3.6 27B. The repository logs the trained adapter against the base model at four decoding settings.

Setting	Base	Trained	Gain
Low effort, temperature 0	2%	37%	35 pts
Medium effort, temperature 0	16%	30%	14 pts
Low effort, temperature 1.0	15%	69%	54 pts
Medium effort, temperature 1.0	61%	84%	23 pts

Read down that table and the headline thins out. The 84 percent sits at medium reasoning effort and temperature 1.0, a combination the adapter never trained at. The repository records it as the average of two evaluation runs that returned 80 and 88 percent. The base model already solved 61 percent there without any training, so the lift is 23 points, the second smallest of the four cells.

The 37 percent comes from inside the training job, where the service fixes validation at temperature 0 and the low effort the run trained at. AWS says that curve went from 2 to 31 percent in the first ten steps, then moved between 35 and 48 percent, peaking at 48 percent at step 90 before ending at 37. With 128 words, one point is about 1.3 words, which AWS calls mostly noise after step 10.

For evaluation, use temperature = 0.0 (greedy decoding) so that eval curves are deterministic and comparable across runs.

Amazon SageMaker AI Developer Guide, hyperparameters reference

The guide argues the opposite case, that you should measure at the settings you serve at, and temperature 1.0 is also what training sampled at. Both readings are defensible, so the figure a buyer quotes depends on which one they picked. The cost is less ambiguous.

The 100-step run burned 491.6 million prefill, 159.7 million sample and 643.3 million train tokens across 105,486 rollouts in 12 hours 40 minutes. At $0.12, $0.30 and $0.36 per million in us-east-1, that came to $338.50, and the backward pass alone was $231.59 of it. The AgentCore runtime billed about $79 more for vCPU and memory, which puts the real total near $418 by our arithmetic, roughly 23 percent above the quoted figure.

Serverless per-million-token pricing is the part that travels to other tasks, since the train dimension scales with trajectory length. AWS reports the training reward stopped rising after about step 15 and validation stopped improving after about step 30, so more steps alone won’t help. Its untried levers are a curriculum that seeds games with prior guesses and a group size above the default of 8. The next number worth watching is what either does at temperature 0.