NetKet and JAX: Building a Transformer Model for Frustrated Spin Systems

Transformer architectures are starting to find a place in computational quantum physics. A new hands-on guide shows how to use NetKet and JAX to build a complete Neural Quantum States pipeline for a hard problem: the frustrated J1-J2 Heisenberg spin chain. This is not theory in a vacuum, but a reproducible setup in which the model, the sampler, the optimization and the accuracy check are combined in a single research loop.

The guide's central idea is that the Transformer architecture suits many-body quantum states, where long-range correlations between particles matter. Conventional numerical methods quickly run into the size of the state space, especially when the system is frustrated, meaning that competing interactions keep it from settling into a simple ordered energy minimum. Under these conditions, Neural Quantum States let you represent the wave function as a parameterized neural network and then optimize it with Variational Monte Carlo. NetKet serves as a ready-made environment for quantum computations, while JAX acts as the engine for high-precision, accelerated optimization.

The guide first sets up the basic physics of the problem. The author defines a one-dimensional chain of length L with periodic boundary conditions, in which nearest neighbors interact with coefficient J1 and next-nearest neighbors with J2. This combination creates the frustration that makes the problem interesting and non-trivial. The system is described with a NetKet graph, a spin-1/2 Hilbert space with fixed total projection and a Hamiltonian operator built through GraphOperator. JAX 64-bit precision is enabled in parallel, which matters for stable calculations in problems of this class.

Then comes the machine learning proper. The wave function is defined by a custom TransformerLogPsi model in Flax: spin configurations are encoded as tokens, receive embeddings and positional representations, and then pass through several blocks of self-attention and feed-forward layers. The example uses a hidden dimension of 96, four attention heads and six transformer layers. The model outputs the complex logarithm of the wave function amplitude. This is critical, because a quantum state cannot be adequately described by a real scalar alone. After information is aggregated across the whole chain by averaging, the network obtains a global representation of the configuration and can express more complex correlations than local ansätze.

The guide is also valuable because it does not stop at defining the model. For training, the author assembles a full VMC loop: a MetropolisExchange sampler, an MCState variational state, the Adam optimizer and Stochastic Reconfiguration, the analogue of natural gradient descent for quantum states. The example configuration uses 4096 samples, discards the initial states in the chains and runs about 250 optimization iterations. This stack is needed not only to reach low energy but also to monitor convergence. The code saves the trajectories of mean energy and variance, so you can see how steadily the model moves toward a good solution.

After training, the pipeline is used as a research tool. The author runs calculations for several values of J2 from 0 to 0.7 on a chain of 24 sites, records the final energies and estimates the peak of the structure factor. This goes beyond fitting neural network parameters: it shows how the system's physical behavior changes as frustration grows and where transitions between different phases of magnetic order may appear.

As an additional check, the model's quality is compared with exact diagonalization on a smaller system of 14 sites using the Lanczos method. Comparing energies gives a clear numerical reference for how close the variational transformer comes to the exact solution where an exact calculation is still possible.

The practical value of the material is that it bridges two worlds: modern deep learning architectures and real problems in computational physics. For ML engineers, it is a good example of using a transformer beyond text, images and standard tabular data. For physicists, it is a clear template for moving from the abstract idea of Neural Quantum States to a reproducible experiment with concrete metrics, benchmarks and observables. And for those working at the intersection of the two fields, the guide offers a base that can be extended: moving to larger lattices, adding symmetries, studying entanglement or building more complex time simulations.

What this means: the Transformer approach is gradually becoming a working tool not only for classic AI tasks but also for modeling quantum systems, where the cost of error is high and exact methods run out quickly. If NetKet and JAX are already in your stack, this material provides a nearly ready starting point for research-level experiments.