T-Tech Introduces LIMe, a Method That Trains Language Models With 15% Less Compute

Researchers at T-Tech presented LIMe at EMNLP 2026, a method that lets language models be trained with roughly 15% less compute. The figure comes from a comparison: a model with roughly a billion parameters using LIMe reached the same training loss as the baseline LLaMA, but its compute cost was 15% lower. Training loss shows how poorly a model predicts data during training. Benchmark quality did not drop. EMNLP is an A* conference, and this year it takes place in Hungary.

Why tokens start to look alike

In a transformer, each layer relies only on the hidden state of the previous layer. The hidden state is a set of numbers in which the model stores its current representation of each token. All of the context has to pass through it.

The authors show that this creates pressure toward representation collapse. Tokens become too similar to one another, the differences between them vanish, and quality suffers along with them.

How LIMe works

LIMe stands for Layer-Integrated Memory. It is a lightweight extension of the attention mechanism. For every layer, the model already stores key-value buffers: the keys and values that the attention heads work with. LIMe reuses these existing buffers and adds learnable routing weights on top of them. Each head and each layer has its own weights.

With these weights, an attention head can pull representations not only from the previous layer but also from earlier ones, and mix them. The size of the hidden state does not grow, and computations that have already been done are reused.

What the experiments showed

The testing ran on models with roughly a billion parameters. The results:

• the same training loss as the baseline LLaMA, at a 15% lower cost;
• LIMe outperformed other approaches on language modeling and multi-step reasoning benchmarks;
• the most noticeable gain came on synthetic arithmetic tasks and in architectures with many layers.

The authors also tested the underlying cause. LIMe has higher entropy of value vectors (that is, the value vectors carry more diverse information) and better token separability. According to the authors, the main problem, representation collapse, is indeed mitigated.

The authors also examined the learned routing weights. They show that the model systematically reuses both local and distant features.

Where to find it

The method description is published on arXiv (https://arxiv.org/abs/2502.09245). The authors have released the full implementation on GitHub: https://github.com/corl-team/lime.