NVIDIA Nemotron 3.5 Super VL: 120B Parameters, 12B Active, Running on AWS

NVIDIA has released Nemotron 3.5 Super VL, a multimodal model, and the AWS team has written up how to run it on your own Kubernetes clusters. The model has 120B parameters, but only 12B are active for each token (a word fragment).

It uses a mixture-of-experts scheme: the network contains groups of "experts," and a dedicated router picks just a few of them for each token. Compute cost ends up similar to a small model's, but GPU memory still has to hold all 120B weights. The model understands text, images, documents and video, and outputs text.

It is open: NVIDIA publishes the weights, the training datasets and fine-tuning recipes. The model was distilled from the larger Nemotron 3 Ultra, meaning it was trained to imitate the behavior of a bigger "teacher."

The Nemotron lineup on AWS already includes Nemotron 3 Nano, Nano Omni, Super, Ultra and Nemotron 3.5 Lightning. Super VL joins that list.

Hybrid Architecture and Claimed Results

The network alternates three types of layers: Mamba-2, standard attention and expert layers. Vision is handled by the RADIO encoder. The advantage of Mamba-2 is that it keeps a fixed-size state during generation. With standard attention, the KV-cache memory (which stores the context already read) grows with text length. That makes a context of up to 1M tokens noticeably cheaper than in a pure transformer. The model also has a built-in Multi-Token Prediction head, which enables speculative decoding: the model proposes several tokens ahead at once and then verifies them.

NVIDIA reports the following results (these are the company's own figures; the article includes no independent measurements):

• 63.44% on Hermes Bench, an agentic-task test;
• 87.05% on LangChain agent evaluations, i.e. checks of how well it works in a popular agent framework;
• 80.8% on CharXiv, a question set about charts from scientific papers.

The stated use cases are code review and debugging, cross-checking sources after a web search, reading tables and diagrams in documents, analyzing cybersecurity vulnerabilities and threats, and finding and analyzing incidents in video.

How Many GPUs You Need and How the Cluster Is Built

The minimum configurations for vLLM (an inference server) are:

• B200/GB200: 8 GPUs in BF16, 2 GPUs in NVFP4;
• B300/GB300: 4 GPUs in BF16, 1 GPU in NVFP4;
• H200: 8 GPUs in BF16, 4 GPUs in NVFP4;
• H100: 16 GPUs in BF16, 8 GPUs in NVFP4.

NVFP4 is a 4-bit number format. It compresses weights roughly four-fold relative to BF16, so it needs two to four times fewer cards. On H100, BF16 requires two nodes of 8 GPUs each.

AWS's reference architecture looks like this. The Kubernetes control plane handles the API. CPU nodes host service and test pods, while GPU nodes run the model. Container images live in Amazon ECR, weights are stored on a shared Amazon FSx for Lustre volume, and the GPU nodes are connected by Elastic Fabric Adapter (EFA) networking. The Kubernetes manifests are identical for three orchestration options: standard EKS, EKS Auto Mode (where AWS itself manages scaling, GPU drivers and EFA) and SageMaker HyperPod EKS. The setup was tested on p6-b200.48xlarge instances (8 B200 GPUs) and p5en.48xlarge instances (8 H200 GPUs). A step-by-step guide is available in the aws-do-eks project.

How to Plan Capacity Using Latency Curves

The model is served by NVIDIA Dynamo. It can run in two modes: entirely on a single node, or in a disaggregated setup where prefill (reading the prompt) and decode (generating the answer) run on different nodes. For vision models, the encoding of images and video can also be moved to a separate node.

First, the cluster is checked with an NCCL all-reduce microbenchmark. If all GPUs are loaded during the test and traffic flows over all EFA adapters, the network is configured correctly. Then the aiperf tool measures:

• TTFT, time to first token;
• ITL, the pause between tokens;
• TPS, total throughput in tokens per second.

The measurements are repeated at different numbers of simultaneous requests (a concurrency sweep). AWS gives a planning example. Set the limits: TTFT up to 600 ms, ITL up to 100 ms per token. Then find the rightmost point on the graph that stays under both limits. In the example, that is 32 simultaneous requests and about 320 tokens per second per replica, i.e. per prefill-decode pair. From there it is simple arithmetic: divide the target throughput by 320 to get the number of pairs.

On two nodes, the disaggregated mode gave a lower TTFT than the combined mode, while ITL stayed almost the same. A rising ITL means you need to add decode capacity; a rising TTFT means you need more prefill.

The practical takeaway for engineers: for the quantized model, a single modern B300 card is enough, and the choice between modes is best made on your own measurements, repeating the test after any change of precision, parallelism scheme, or Dynamo and vLLM version. The quality figures are so far claimed by NVIDIA itself, and the source material is part of AWS's partner documentation. Compare the model with competitors on your own tasks.