Stanford and Together AI Teach LLMs to Write Text Starting From an Outline: Generation Up to 31x Faster

A team from Stanford and Together AI has proposed a text generation method that fits neither autoregression nor diffusion. The model first sketches a "table of contents" for its answer, then fills it in with detail step by step. According to the authors, this approach delivers a speedup of up to 31x. The news was reported by the Chinese outlet Jiqizhixin.

Where the idea came from

The method builds on VAR (Visual Autoregressive Modeling), which won the best paper award at NeurIPS 2024. In image generation, VAR dropped the classic approach of traversing an image pixel by pixel or token by token. Instead, the model predicts the "next scale": first a very coarse version of the frame, then progressively more detailed ones. Each step refines the previous one, and within a single scale the tokens can be computed in parallel.

The Stanford researchers and Together AI engineers carried this principle over to text. In short, it is "top-down" generation: structure and key meanings first, then the specific words.

How this differs from the usual approaches

Today's LLMs write left to right, one token at a time. Each next word depends on all the previous ones, and this puts a hard cap on speed: one pass yields one token, no matter how many GPUs sit in the rack. Diffusion language models try to get around that limit. They start from "noise" or masked text and iteratively clarify the whole thing at once. But that route has its own problems with quality and with controlling response length.

The third option the authors describe takes different things from each approach. As in autoregression, the process has a clear order, except that it runs across levels of detail rather than across positions in a line. As in diffusion, many tokens are updated at every step. That is where the claimed speedup comes from.

Together AI's involvement is telling. The company makes money on inference for open models, so every model pass it saves translates into revenue. Generation speed today is limited not by the amount of FLOPS but by the sequential nature of decoding.

Key details

• Authors: Stanford University and Together AI
• Source idea: "next-scale prediction" from VAR (NeurIPS 2024, best paper)
• Principle: first a plan or rough sketch of the text, then refinement level by level
• Claimed speedup: up to 31x
• Positioning: a third paradigm alongside autoregression and diffusion

What the report leaves out

Based on the available excerpt of the publication, several things that determine the whole assessment remain unclear. The baseline against which the 31x gain was measured is not named. Comparing against plain token-by-token decoding with no optimizations is one thing; comparing against modern inference with speculative decoding and batching is another. There is no data on model sizes, text lengths, or quality on standard benchmarks. The "up to 31x" figure should be read as a best case, not a typical one.

There is also an architectural question. In images, scales arise naturally: a downscaled version of a picture is still a picture. Text has no "coarse resolution," and how the authors define the levels of the plan is the main thing to scrutinize.

What practitioners should check

For now this is a research project, not a product, and it is too early to build on it in production. But it is worth following. When the paper comes out with code, three things merit a look: quality on long-reasoning and coding tasks, where an error in the plan spoils everything after it; stability of instruction following; and the real speedup on the same GPUs at comparable quality. If the gain holds on responses thousands of tokens long, it would give engineers a new lever beyond quantization and speculative decoding. If not, it will remain an interesting experiment showing that ideas from computer vision transfer to language reasonably well.