How Mixture-of-Experts Works: The Architecture Behind Modern Neural Networks

Apple researchers have found an inefficiency in the Mixture-of-Experts architecture, a routing method modern neural networks use to distribute data among specialized expert modules.

What is an expert path

In a sparse Mixture-of-Experts architecture, each token at each layer is independently routed to a small subset of N experts. A standard MoE makes this decision layer by layer, without taking into account the history of previous layers. With L layers, the number of theoretically possible unique routes is N^L: for a model with 8 experts and 32 layers, that is over 10^28 options.

Apple ML Research suggests looking at this differently: an "expert path" is the full sequence of expert choices a token makes across all layers of the model. This view turns the analysis of MoE from a set of independent per-layer decisions into an analysis of whole trajectories.

The authors' key observations:

• Despite N^L possible paths, tokens cluster into a small fraction of routes
• Popular paths are not random: they correspond to the linguistic functions of tokens (part of speech, syntactic role, semantic type)
• The vast majority of theoretically valid routes remain unexplored during training
• The authors call this "statistical inefficiency"

Why are unexplored paths a problem?

If most routes are not used, the experts on those paths receive very few gradient updates. The load is distributed unevenly: some experts are overloaded and see a disproportionately large number of tokens, while others sit idle and remain poorly trained.

On paper, the model has enormous capacity (N^L routes multiplied by the parameters of each expert), but in practice it uses only a small part of it. When scaling to hundreds of experts and dozens of layers, this gap between theoretical and practical potential becomes especially noticeable.

There is an additional complication: without explicit constraints, the router has to discover "good" paths on its own in a huge combinatorial space, which makes training less stable and less predictable.

How Path-Constrained MoE works

Apple ML Research proposes a family of architectures that explicitly narrow the effective space of paths. In Path-Constrained MoE, the router chooses the next expert based on the path the token has already taken, keeping it within an allowed subset of continuations.

The key principle: if useful paths make up a small fraction of N^L and correlate with the linguistic structure of tokens, explicitly constraining the space does not cost the model expressive power, but it radically changes training dynamics. Each expert within the allowed subset receives a more even flow of updates. "Dead zones," meaning paths that are never used in a standard design, shrink.

The link between active routes and the linguistic function of tokens gives such constraints a theoretical motivation: the architecture encodes what the model would discover empirically anyway.

What this means

The Apple ML Research work raises a systemic question about the design of sparse MoE architectures: standard independent routing produces a huge but practically empty combinatorial space. The Path-Constrained approach can make large language models more parameter-efficient, more stable in training and more predictable, especially when scaling to hundreds of experts and dozens of layers.

What is Mixture-of-Experts in neural networks?

It is an architecture in which each token (a unit of text) is routed to a subset of specialized experts instead of being processed by all of the model's parameters, which saves compute.

Why does the Apple research matter?

It shows that most theoretically possible routes in MoE architectures remain unused, which means the potential of such models is used inefficiently.