How to speed up transformer training with NVIDIA Apex: FusedAdam, FusedLayerNorm, and torch.amp

Training transformers is one of the most resource-intensive tasks in ML. Standard PyTorch works correctly but leaves significant performance gains untapped: most operations execute sequentially where they could be combined. NVIDIA Apex offers a set of optimized CUDA kernels, and native torch.amp adds mixed precision—together they enable achieving the same results faster and with lower VRAM consumption.

What is NVIDIA

Apex and why manual build from source is necessary

Apex is an open-source library from NVIDIA that extends PyTorch with optimized CUDA extensions. Its core mechanism consists of "fused" kernels: operations that combine multiple computational steps into a single GPU call. This reduces memory accesses and synchronization overhead—especially critical for small operations where kernel launch time exceeds actual computation time. Important caveat: standard `pip install apex` does not include CUDA extensions. Manual build from source with flags `--cuda_ext --cpp_ext` is required. This requires compatible versions of PyTorch, CUDA Toolkit, and C++ compiler. After installation, it's good practice to verify kernel availability programmatically through `apex.optimizers` and `apex.normalization`—to confirm native versions are used rather than slower Python fallbacks.

FusedAdam, FusedLayerNorm, and torch.amp

Three tools provide the primary performance boost when training transformers:

• FusedAdam—Adam optimizer rewritten in CUDA. Combines weight updates, first and second moment computation, norm clipping, and weight decay into a single kernel. Significantly fewer GPU memory accesses compared to standard implementation.
• FusedLayerNorm—Layer Normalization as a single CUDA kernel instead of a chain of sequential PyTorch operations (mean, variance, subtract, divide, scale, shift). For transformers with dozens of normalization layers, this delivers meaningful cumulative effect.
• torch.amp—built into PyTorch automatic mixed precision mechanism via `autocast()` and `GradScaler`. Switches computations to BF16 or FP16 where safe, preserving FP32 for gradient accumulation. Reduces VRAM consumption by roughly half and accelerates matrix multiplications on GPUs with tensor cores. The key advantage is minimal code changes: FusedAdam is a direct replacement for `torch.optim.Adam`, FusedLayerNorm replaces `nn.LayerNorm`, and torch.amp adds a wrapper around the forward pass without rewriting the rest of training logic.

Step-by-step benchmark

To isolate each component's contribution, optimizations are enabled incrementally, fixing iteration time and peak VRAM consumption at each step:

1. Baseline Adam + FP32—reference point 2. FusedAdam + FP32—gains from optimizer replacement 3. FusedAdam + FusedLayerNorm + FP32—add fused normalization 4. FusedAdam + FusedLayerNorm + torch.amp (BF16)—full configuration

This incremental approach avoids relying on "black box" aggregate results—each step's contribution becomes visible. Final configuration yields 1.5× to 2.5× speedup in iteration time relative to baseline; exact numbers depend on model size, batch size, and GPU architecture.

"Mixed precision combined with fused kernels is the de facto standard

for industrial transformer training."

What this means

For ML engineers this is a reproducible recipe with measurable results: build Apex from source, replace two components, and add an autocast context manager—without changes to data structures, metrics, or validation logic. Especially relevant under limited GPU-hour budgets: same experiments in less time means direct cost savings for teams training large models on their own hardware or in the cloud.