Lyubomir Gorbatko Presents Sessa, an Alternative to Transformers and Mamba

Habr has published a breakdown of Sessa, an attempt to rethink decoder-only models and offer an alternative to the familiar Transformer/Mamba pairing. The author does not promise a ready replacement for the market leaders. Instead, he shows something more fundamental: different decoders can be described through a common foundation and then compared honestly on how they store and retrieve information from long context.

The article builds from simple to complex. First, the author re-derives the Transformer not as a set of familiar blocks but as an extension of ordinary convolution. The idea is that a fixed window and fixed coefficients quickly hit their limits: such a mixer sees only local context and adapts poorly to the task. If the weights depend on the input and are then normalized through softmax, attention emerges naturally. In this reading, the transformer's strength is its flexible comparison of the current token with earlier ones, but the cost is well known: compute grows with sequence length, and in the diffuse regime attention is worse at holding on to one specific distant element.

The article then moves on to S4D and Mamba. Here the author treats the problem as one of memory: instead of rereading the entire prefix each time, the model can accumulate the past in an internal state. This removes some of attention's problems and makes long sequences cheaper to process. But it has its own limit. In the author's account, Mamba works well when the selective state space mechanism can "freeze" the state in time and hold on to the needed signal. If this regime is poorly recognized, especially on noisy or very long sequences, the influence of old tokens decays exponentially, and pinpoint retrieval of the needed information becomes less reliable.

Against this background, Sessa is presented as a hybrid. The author proposes combining two ideas: keeping attention-like adaptivity while adding feedback, meaning controllable feedback through past states. Inside the layer there are two branches: forward, which gathers information from the prefix, and feedback, which reuses already accumulated states. The key point is that the coefficients of both branches depend on the current token and the sequence length. The model therefore gets a more flexible memory mechanism than the classic Transformer and more direct access to history than Mamba. In essence, this is an attempt to build attention into a recurrent loop rather than keeping the two approaches on opposite sides of the barricades.

The article's main emphasis is not the slogan "we beat transformers" but a comparison of memory regimes. The author considers a controlled scenario in which models struggle to focus precisely on one needed token. In this regime, the influence of distant tokens decays roughly as the inverse of distance in the Transformer, exponentially in Mamba, and more slowly in Sessa, whose tail fades more gradually. Theoretically, this gives more stable long-distance retrieval. In a multilayer configuration, Sessa can, according to the author, even sustain retrieval profiles without degradation over distance.

An arXiv paper and code have been published alongside the Habr post, and the study itself reports comparable long-context experiments. The author also states the current boundary of the result plainly: for now this is primarily theory and an architectural hypothesis, and the next important step is training at the scale of several billion parameters and testing outside carefully controlled regimes.

In short, the piece is interesting not only for Sessa itself but for how it explains things. It reduces Transformer, Mamba and the new architecture to a common scheme and shows exactly where their memory properties diverge. For anyone following the race in long-context models, it is an important signal: a notable alternative to transformers may come not from abandoning attention entirely, but from combining it with a more expressive recurrent memory.