AWS Launches Multi-Turn RL in SageMaker: gpt-oss-20b Climbs From 15% to 69% for $339

AWS has added Multi-Turn Reinforcement Learning (MTRL) to SageMaker AI. It trains models on multi-step tasks where only the final outcome is known: the whole chain of actions either worked or it didn't. In a demo run, gpt-oss-20b rose from 61% to 84% of Wordle words solved on a held-out set at the default reasoning effort. At low effort, the setting used for training, the result rose from 15% to 69%. Training tokens cost about $339.

Why ordinary fine-tuning doesn't work for agents

Supervised fine-tuning teaches a model to produce the best answer to a single prompt. An agent that fills out an expense report or works through a support ticket behaves differently: each step depends on the previous ones, and there is usually no reference sequence to imitate. Single-turn RL can't assign credit to a decision made three steps before the outcome. MTRL works with the one signal that real workflows do provide: a reward for the completed episode. You don't describe the correct trajectory, only how to score the result.

The architecture splits into two halves. SageMaker AI owns the training job and the policy model. The agent, the environment and the reward function live in Amazon Bedrock AgentCore, or on your own infrastructure (EKS, EC2, Fargate) behind a Lambda forwarder. For each rollout, the service calls the AgentCore runtime with a single dataset row. A Strands-based agent reaches the policy model through an OpenAI-compatible endpoint, plays up to six turns through a guess_word tool, and returns a single number at the end. Group Relative Policy Optimization (GRPO) compares the rewards of several rollouts for the same word and updates the LoRA adapter.

The reward depends heavily on the outcome. A win gives between 1.05 and 1.55 depending on speed, a loss gives −0.5, and not a single valid guess gives −1.5. A small shaping term rewards good strategy within each level. The dataset contains only the target word. Word lists based on SCOWL live inside the agent and are used only to compute the reward. Validation uses 128 words that do not overlap with the 640 training words (five full batches of 128).

What 100 steps cost

MTRL doesn't make you pick instances. You pay per million tokens across three line items: prefill, sample and train. In multi-turn tasks, prefill grows because every turn rereads the history. The 100-step run processed 105 486 rollouts in 12 hours 40 minutes, at us-east-1 list prices:

• Prefill: 491 616 806 tokens at $0.12 per million, $58.99 • Sample: 159 737 098 tokens at $0.30, $47.92 • Train: 643 308 554 tokens at $0.36, $231.59 • Total: 1 294 662 458 tokens, $338.50 • AgentCore runtime: about $79 for vCPU and memory per CloudWatch metrics, or roughly $417 all in

The biggest line item, train, grows with trajectory length. That makes episode length and verbose reasoning the places to save. In the evaluation tasks, low reasoning effort spent 63–83% fewer sample tokens per game than medium effort.

Supported models and setup pitfalls

The service currently works with four base models, and each is trained through a LoRA adapter:

• Amazon Nova 2 Lite: us-east-1 and us-west-2 • GPT-OSS-20B: us-east-1 and us-west-2 • Gemma 4 31B (instruction-tuned): us-west-2 only • Qwen 3.6 27B: us-west-2 only

The authors chose GPT-OSS-20B because it has the lowest per-token price of the four and lets you set reasoning effort per request.

The guide covers several non-obvious pitfalls. The training job's role must trust job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. Both sts:AssumeRole and sts:TagSession are required, otherwise CreateJob fails with "Could not assume role". The principal is named only after several retries. The sagemaker_rft_handler decorator does not start the HTTP server itself. You need to wrap it in BedrockAgentCoreApp and @app.entrypoint, and the entrypoint must sit on the outside, otherwise every call hits a 30-second timeout. Launching requires botocore 1.43.37 or newer, the first version with the AgentRFT CreateJob operation. The job and the runtime must be in the same region, because CreateJob rejects a cross-region ARN. An MLflow app for metrics is also required.

Before spending GPU time, run a smoke test with the payload from the repository and substitute your own account into the ARN. The expected result is a 400 "Could not find job" and a reward of −1.5, because the agent never reached the model. This confirms that parsing, authentication and error reporting work. A 403 means the ARN contains a different account. A timeout means the server isn't starting.

What to do with this in practice

Wordle is only a teaching example here, and frontier models handle it fine. The practical value is elsewhere: take a small, cheap model and teach it a multi-step process that isn't in public training data, such as internal request approval or ticket routing. To do that, you change the task parsing, the tools and the reward function in main.py, and the rest of the pipeline stays the same. The dataset can be a string with a full message list, a tool configuration or a reward specification, because the service passes it to the agent without parsing it.

The data, the agent and the trained model stay in your AWS account and are not used to train the base models. The bottom line is simple: about $340 for tokens and $80 for the runtime buys a specialized 20B-class model. To keep the price from creeping up, design short episodes up front and a reward that weights the outcome above intermediate points.