S3AC brings the critic back to RL for LLMs: gains from 1.5B to 397B parameters

Researchers at Shanghai AI Laboratory have proposed S3AC (Stabilized, Selective, and Scalable Actor-Critic), a practical recipe for training a value critic in reinforcement learning with verifiable rewards on long chains of thought. The method was tested on dense and mixture-of-experts models ranging from 1.5B to 397B parameters. It outperformed a modern variant of GRPO called GRPO-X. At 35B parameters, S3AC sustained training with a single rollout per prompt, while critic-free baselines began degrading after their initial improvements.

Why the industry fled from the critic

In RLVR for long chains of thought, the critic has long been considered an expensive and finicky component. It needs to be trained separately, it consumes memory and compute, and it behaves unstably on long trajectories. That's why most teams use critic-free methods like GRPO: for each prompt, they generate a group of responses and compute a baseline from the group's average result.

This approach has two well-known weaknesses. First, there's no token-level credit assignment — the entire trajectory gets a single score, even though the error might have occurred at a few specific reasoning steps. Second, it requires generating groups of rollouts just to obtain a baseline for a prompt at all. This is expensive, and on tasks that are either too easy or impossibly hard, part of the generation is wasted.

A critic solves both problems: it produces per-token advantages and a per-prompt baseline. The S3AC authors explain why this hasn't worked for long chains of thought until now. First, training data for the critic poorly represents trajectories of different lengths. Second, early in training, gradients from a freshly initialized value head dominate and destabilize the rest of the network.

The S3AC recipe and selecting prompts before generation

The paper's main contribution is a set of techniques that make critic training stable on long reasoning chains. The trained critic provides both needed signals: token-specific advantages and prompt-specific baselines.

The second idea is Difficulty-Aware Prompt Sampling (DAPS). The critic's baseline shows how difficult a prompt currently is for the model. DAPS combines this estimate with the prompt's outcome history to decide which tasks are worth sending for generation. Selection happens before rollouts, so no generations are wasted. There's no need for offline difficulty initialization — no need to run the dataset through the model in advance and label it by difficulty.

• Method: S3AC, an actor-critic approach for LLM reasoning with verifiable rewards
• What the critic provides: token-specific advantages and prompt-specific baselines
• Data selection: DAPS, selecting prompts based on difficulty estimates and outcome history before generation
• Model range: dense and mixture-of-experts, from 1.5B to 397B parameters
• Comparison baseline: GRPO-X, a modern variant of GRPO
• Single-rollout mode: at 35B parameters, S3AC keeps improving, while tested critic-free methods degrade after initial gains
• Team: Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang
• Code: InternLM/xtuner repository on GitHub
• Publication: arXiv 2610.35518, Hugging Face Daily Papers

The most telling result is the single-rollout mode. Group-based methods lose their baseline without a group, and at 35B parameters, their training sags after an initial boost. The critic, by contrast, estimates the baseline on its own, so quality doesn't drop. And since the method scales up to 397B parameters, this isn't just a lab toy.

Who should pay attention to S3AC, and when

Teams fine-tuning reasoning models through RLVR run into the cost of rollouts. Selecting prompts by difficulty before generation saves exactly on that line item. If a group of answers to a task is bound to produce the same result anyway, it carries almost no training signal. DAPS tries to avoid spending compute on such tasks.

The second practical benefit is robustness with few rollouts per prompt. This will be useful wherever generating long chains is very expensive, such as on large MoE models. An implementation is available in xtuner, so testing the method on your own tasks doesn't require building a pipeline from scratch.

The limitations are also clear: the critic remains an extra component, and the advantage over GRPO-X has been shown on the authors' own test set of models and configurations. Independent reproductions will show how well the recipe transfers to other reward domains and other model families.