LLM as Decoder in ASR: Adapting Nvidia's SALM Architecture for Russian-Language Telephony

The speech recognition (ASR) industry has relied on a cascade approach for decades: first an acoustic model converts a sound spectrogram into letter probabilities, then a language model tries to assemble that into coherent text. The problem is that the first component irreversibly loses acoustic information at the moment of discretization, and the error compounds down the chain. The acoustic model has no context and often confuses homophones: words that sound identical but have different meanings can't be told apart without understanding the semantics.

The industry has tried several ways to solve this: integrated decoding (Beam Search + a small language model), autoregressive architectures (RNN-T, TDT) trained on sparse ASR datasets, and error correction via a large LLM applied on top of the text — but all of this stayed within the cascade paradigm. By late 2024, Nvidia proposed a fundamentally different architecture: SALM (Speech Augmented Language Model), which set a new record on the HuggingFace OpenASR leaderboard. This architecture became the basis for the Canary-Qwen-2.5B model.

How SALM Works: A Hybrid Approach Instead of a Cascade

Instead of a sequential pipeline, SALM merges the speech encoder and the LLM into a single system. FastConformer extracts deep acoustic features from raw audio. An adapter projects these embeddings into the LLM's token space. Qwen then treats this not as rigid letters but as a soft prompt — a continuous stream of meaning that bypasses its vocabulary. Crucially, the base LLM weights stay frozen: only the adapter and LoRA layers are trained, which prevents catastrophic forgetting and lets you strip out the LoRA layers afterward to use Qwen separately for pure text tasks.

Kontur decided to test how well this architecture holds up in their conditions: Russian-language telephony and video conferencing with a shortage of labeled data. The acoustic encoder was the open FastConformer model, fine-tuned on their own data. For the LLM, they tested two Qwen3 sizes — 0.6B and 1.7B — and picked the heavier one for quality despite the compute requirements. It was a classic story of porting SOTA technology into real-world conditions with a limited budget for data labeling.

Three Architectural Problems and Their Solutions

Hallucinations on padding. Initially, the team used the Whisper approach: fixed audio length, with short phrases padded out with silence at the raw waveform level. The model started generating complete garbage during these silent stretches, which broke the WER metric and slowed inference. The fix turned out to be simple: dropping audio-level padding in favor of classic token padding at the end of the sequence. The hallucinations disappeared.

The modality gap. A language model operates on a fixed vocabulary (say, 100,000 tokens), each mapped to a vector of fixed dimensionality. The acoustic encoder generates vectors of its own dimensionality describing phonetics and acoustics.

Even when the dimensions match, these are mathematically compatible but semantically foreign spaces: a vector from the encoder is just noise to the LLM until they're trained together. The adapter has to project the audio features so the LLM perceives them as native soft prompts. Kontur tested three adapter variants: a multilayer perceptron (2-layer MLP), a ConformerEncoder for additional 16x compression of audio representations (aimed at handling longer context), and a linear projection.

The Conformer turned out to be too heavy on parameters; the MLP gave slightly better accuracy, but the difference between MLP and the linear projection was minimal — so they went with the simplest option.

The result: a quality gain in the Russian-language telephony domain achieved without cutting corners on the acoustic encoder and without massive contrastive datasets. The architectural insights helped avoid typical pitfalls. Kontur isn't rushing to deploy this into production yet, but sees SALM as a promising direction for further improving WER in challenging conditions of spontaneous Russian speech.