Discrete Image Tokens Nearly Catch Up to Continuous Ones: FVQ Closes the Gap With a 59% Win Over Emu3

A team from Meituan LongCat, working alongside researchers from Huazhong University of Science and Technology, Shanghai Jiao Tong University, MBZUAI, and the Chinese University of Hong Kong, tackled a long-standing debate in multimodal AI: can an image be described with discrete tokens — roughly the way text is described with dictionary words — without losing meaning, or will continuous features (unrounded numerical vectors) always be more accurate?

The conventional view was simple: converting an image into discrete tokens inevitably introduces quantization error — meaning the image reconstructed from those tokens differs slightly from the original. Since there's an error, something must be lost, and discrete tokens can never be as "smart" as continuous features in principle. The authors tested this directly: they took an image representation that produces a noticeably larger reconstruction error and compared it with one that has a small error.

It turned out that the first one — with the larger error — helped the model understand the image better (answering questions about it, recognizing text, and so on) than the second one. In other words, reconstruction error and the actual "usefulness" of tokens for understanding are two different things, and one doesn't predict the other.

What mattered instead of reconstruction accuracy

The authors hypothesized that what matters is the "bit budget" — roughly speaking, how many genuinely distinguishable variants a set of tokens can encode. This budget grows linearly with the number of codebooks used (sets of tokens applied either sequentially or in parallel), but only very slowly — logarithmically — with the size of each individual codebook. So increasing the number of codebooks is far more efficient than making each codebook bigger.

The problem is that existing multi-codebook schemes hit a ceiling. For example, RVQ (residual vector quantization) works like this: the first codebook encodes the main part of the image, the second encodes what's left unencoded after the first (the residual), the third encodes the residual of the residual, and so on. Each successive codebook gets an increasingly "empty" slice of information. PQ (product quantization) has a similar issue: it splits the image feature into progressively narrower channel-wise chunks, and each codebook ends up with less and less content. In both cases, doubling the number of codebooks (say, from 8 to 16) barely improves quality — the codebooks simply stop contributing.

How FVQ works

The authors' solution is Factorized Vector Quantization (FVQ). Instead of breaking the image feature into increasingly small and impoverished pieces, FVQ creates several full, equally weighted "views" of the same complete feature — through specialized mathematical projections that are guaranteed to jointly preserve all the original information (a property known as a "tight frame"). Each view is encoded by its own separate codebook in parallel, with no "primary vs. residual" hierarchy. As a result, adding a new codebook doesn't weaken the information available to the others — and quality keeps improving even at 16 codebooks, a point where older methods already stall.

In practice, this means the more codebooks you add to FVQ, the more accurately the model recognizes things like small text in images — typically the first casualty of compressing an image into tokens.

Results on real-world data

The authors trained their final model — a Qwen3-8B equipped with a visual tokenizer built on a 16-codebook FVQ on top of Qwen3 ViT — on industrial-scale data and evaluated it on nine image-understanding benchmarks: MMMU, MathVista, MMStar, OCRBench, ChartQA, CharXiv (two variants), DocVQA, and InfoVQA.

• On average across these nine metrics, the model beat UniAR by 22.8% and Emu3-Chat by 59.2%
• This is the first known discrete-token model to match the quality of strong continuous systems at the level of Qwen3-VL, including on challenging image text-recognition tasks
• Code and model weights are available on GitHub (hustvl/FVQ), with a dedicated project page featuring demos

The practical takeaway for teams building multimodal models: chasing the lowest possible image reconstruction error isn't the best guide when choosing a tokenizer. It's more important to check how many genuinely useful bits a representation carries, and whether the quantization scheme hits a ceiling as new codebooks are added. If a team is already using RVQ or PQ and sees quality plateau as the number of codebooks grows, a parallel, factorized scheme like FVQ is a reasonable direction to experiment with — especially when precise text recognition and fine image detail matter for the product.