NVIDIA Shrinks Nemotron from 120 to 74 Billion Parameters to Fit 24GB of VRAM

# NVIDIA Shrinks Nemotron from 120 to 74 Billion Parameters to Fit 24GB of VRAM

Companies building general-purpose AI models typically compete on size: more parameters, smarter model. But for a local AI PC that needs to run alongside other programs on limited VRAM, a different metric matters most — how much value the model delivers per gigabyte of memory. NVIDIA tackled this by taking its flagship multimodal model, Nemotron 3.5 Super VL (120 billion parameters, with 12 billion active per token), and shrinking it into Nemotron 3.5 Bolt 74B-A9B, a version that fits within 24GB of memory — exactly what's available on NVIDIA's compact RTX Spark and DGX Spark AI PCs.

How They Trimmed the Model Without Lobotomizing It

Rather than manually deciding which parts of the neural network to cut, NVIDIA used its own tool, Star Elastic, which automatically determines how many "experts" (separate subnetworks activated for specific tasks) to keep in each layer and which to prune. The process runs alongside training: a dedicated "router" explores different architecture configurations while knowledge distillation happens in parallel — the original 120-billion-parameter teacher model guides the compressed version toward correct answers so it doesn't lose capability. The result cut active experts per token from 22 to 9, and total model size from 120 to 74 billion parameters.

After compression, the model went through further training: first, image-understanding capability was restored by reattaching a freshly retrained visual module from the parent model to the compressed text backbone. Then came reinforcement learning on real agentic tasks — working with tools via the MCP protocol, handling terminal operations, and performing code reviews in the style of SWE benchmarks. This training is what pushed the MCP-Atlas benchmark score from 48 to 56 — meaning the model got noticeably better at multi-step tasks that require sequentially invoking different tools.

2-Bit Quantization — and How It Didn't Wreck Accuracy

The biggest memory-saving lever wasn't architectural trimming but aggressive weight quantization. Models typically store weights in 16-bit format (BF16), but NVIDIA compressed most of the "expert" layers down to just 2.0625 bits per parameter — meaning each number in the network now takes up roughly 8 times less space. This cut the model's footprint from 148GB (uncompressed BF16) to 22.1GB, an 85% reduction.

The team's key insight: it's better to start with a larger model and compress it more aggressively than to take a smaller model and compress it less — at the same final size, the former approach is more accurate. To prevent the quantization itself from tanking response quality, they used distillation during the compression process (Quantization-Aware Distillation): the uncompressed teacher model guides the compressed version in real time on how to preserve correct probability distributions over words. To get this right, the team tested several loss functions and settled on a Jaccard-based measure (which gauges the overlap between two probability sets) — it delivered the best results for this level of extreme compression.

Not every layer was compressed equally: some layers (for example, those handling long-context processing) were kept in NVFP4 format, while embeddings and the output layer stayed at 8-bit Q8_0 precision to avoid losing accuracy where it matters most.

How It Performs in Practice

On the RTX Spark, the model delivers 84 tokens per second thanks to an additional "draft" module called LiLiCorr — a small helper model that guesses several upcoming words ahead of time while the main model simply verifies them (a technique known as speculative decoding that speeds up generation without sacrificing quality). The speed is comparable to models with far fewer active parameters, yet Nemotron 3.5 Bolt outperforms them in accuracy on agentic-task benchmarks (GDPVal v2), web browsing (BrowseComp), and terminal operations (Terminal-Bench 2.1).

Across nine key benchmarks, the average score dropped from 57.58 for the full-size model to 54.74 for the compressed version — a loss of just 4.94%, despite the model becoming 6.7 times lighter in size (148GB → 22.1GB). For developers building local AI agents — those who want to run a powerful assistant on compact hardware without relying on the cloud — this is a concrete tool to work with: the weights are available in Hugging Face and GGUF formats, the model is supported in llama.cpp and vLLM, and NVIDIA has also released the LiLiCorr draft module itself to speed things up further.