Mamba

Mamba is a way of building AI language models that keeps memory use flat, however long the text. The transformer design behind most language models stores every earlier word and compares each new word with all of them, so long documents get slow and expensive. Mamba instead updates a fixed-size summary of what it has read. It is often combined with transformer layers, as in IBM's Granite 4.0 models.

How does it work?

Mamba is a state space model (SSM): it carries a hidden state of fixed size and updates it with every new token. Albert Gu and Tri Dao introduced it in December 2023. In earlier SSMs, the update worked the same way for every token. Mamba makes it depend on the token itself, so the model decides per token what to keep in its state and what to forget. The authors call this selection. A Mamba model has no attention layers: one repeated Mamba block does the work of both the attention and the MLP blocks of a transformer.

The Mamba paper describes the choice as a trade-off in compression. Attention does not compress the context at all: it keeps every earlier token in the KV cache, which makes it effective but slow on long texts. A model with a finite state needs no cache and generates each token in the same time, but it is only as good as what fits in that state.

How well does it work?

In the 2023 paper, Mamba generated text with five times the throughput of transformers of similar size, and a Mamba model of 3 billion parameters matched transformers twice its size. On real data its performance kept improving up to sequences of a million tokens. Mamba-2 (May 2024) reworked the core layer to be 2 to 8 times faster.

Mamba-3 followed in March 2026, from researchers at Carnegie Mellon, Princeton, Together AI and Cartesia AI. At 1.5 billion parameters, it averaged 56.4 on seven standard language tasks, against 55.4 for a transformer of the same size, and 57.6 in its MIMO variant.

Where does it fall short?

A fixed-size state has to compress the text, while attention can look everything up again. The Mamba-3 authors found their model competitive on question answering, but it struggled to extract information from semi-structured and unstructured documents. They predict that such layers will mainly be used in hybrid architectures that keep some attention layers; their own hybrid, with one attention layer for every five Mamba-3 layers, beat the pure transformer. IBM notes that transformers also keep an edge in in-context learning, such as few-shot prompting.

Where do you meet it?

• NVIDIA's Nemotron-H (April 2025) replaces most attention layers with Mamba layers and reports up to three times faster inference than similar open transformer models.
• IBM's Granite 4.0 (October 2025) includes hybrid models that combine Mamba-2 and transformer blocks in a 9:1 ratio. IBM reports that they need over 70% less memory for long inputs and several concurrent requests than conventional transformer models, and publishes the models under the Apache 2.0 licence.
• The code of Mamba, Mamba-2 and Mamba-3 is open source in the state-spaces/mamba repository.

Example

Suppose a customer service team wants a model on its own server that handles many long conversations at once. In a transformer, every conversation keeps a KV cache that grows with its length, so memory limits how many conversations fit. The Mamba layers of a hybrid model keep a fixed-size state per conversation, and only its few attention layers need a cache. IBM names this case for Granite 4.0: a customer service agent handling many detailed questions at the same time.