Can Mamba-3 replace Transformers for long contexts?

Not on its own, say its makers. In March 2026, researchers from Carnegie Mellon, Princeton, Together AI and Cartesia AI published Mamba-3, a state space model (SSM) that reads text with a fixed-size memory instead of looking back at every earlier token. In their tests, a Mamba-3 with 1.5 billion parameters beats a Transformer of the same size and stays fast on long inputs. It is weaker at pulling exact facts out of documents, so the authors expect it to be used in hybrids alongside attention layers.

Why do Transformers get expensive on long contexts?

A Transformer, the architecture behind most language models since 2017, lets every token look at all earlier tokens through the attention mechanism. The work grows with the square of the sequence length: at a million tokens, a single attention layer compares about half a trillion pairs of tokens. During generation the model also stores the keys and values of all earlier tokens in the KV cache, and that memory grows with every token. The Mamba-3 paper names exactly these two costs: "quadratic compute and linear memory".

FlashAttention (Dao et al., 2022) made exact attention faster by cutting reads and writes to GPU memory. The amount of computation still grows with the square of the length.

How does a state space model work?

SSMs come from control theory and signal processing: a system is described by a small hidden state that is updated with every new input. S4 (Gu, Goel and Ré, 2021) made the idea work for long sequences in deep learning. Simplified, one step looks like this:

1
h_t = A · h_(t-1) + B · x_t
2
y_t = C · h_t

• x_t is the input at step t, h_t the hidden state after that step and y_t the output.
• A sets how much of the old state is kept.
• B sets how much of the new input is written into the state.
• C sets what is read out of the state.

Because the state has a fixed size, every new token costs the same amount of work, however long the text already is. Compute grows linearly with the length of the text, and memory stays constant during generation. The price is that everything the model retains about earlier text has to fit in that state.

Mamba (Gu and Dao, December 2023) made these parameters depend on the current token, so the model decides per token what to keep and what to forget. Its paper reported five times the inference throughput of Transformers. Mamba-2 (Dao and Gu, May 2024) reworked the layer to run on the matrix units of a GPU, which made it 2 to 8 times faster than the Mamba layer. The dictionary entry on Mamba has more background.

What does Mamba-3 change?

Mamba-3 builds on Mamba-2 with three changes, each derived from the SSM view of the layer:

1. A more expressive recurrence. A new rule turns the continuous model into discrete steps; the authors call it exponential-trapezoidal discretization. Together with added bias terms, it lets Mamba-3 drop the short convolution that earlier recurrent layers relied on.
2. Complex-valued state updates. They work like a rotary position embedding (RoPE) that depends on the input. In the paper's tests this lets Mamba-3 solve parity (is the number of ones in a bit string even or odd?), a state-tracking task on which Mamba-2 scores at chance level.
3. Multi-input, multi-output (MIMO). The state update becomes a matrix multiplication, which keeps the GPU busier during generation: up to four times more computation per decoding step than Mamba-2 at the same state size, at about the same wall-clock latency.

The code and GPU kernels went open source under the Apache 2.0 licence in the state-spaces/mamba repository on 17 March 2026, a day after the paper. The paper was accepted at ICLR 2026.

How does Mamba-3 compare with a Transformer?

The authors trained every model with the same recipe on 100 billion tokens of the FineWeb-Edu dataset, at four sizes from 180 million to 1.5 billion parameters, with a context of 2,048 tokens. At 1.5 billion parameters, the average accuracy on seven standard language tasks was:

• Transformer: 55.4
• Mamba-2: 55.7
• Gated DeltaNet, another linear-time layer: 55.8
• Mamba-3: 56.4
• Mamba-3 with MIMO: 57.6

Mamba-3 with MIMO also matched the perplexity (how well a model predicts unseen text) of Mamba-2 with half the state size, which the authors translate into the same quality at half the latency.

The speed test ran on one H100 GPU with a batch of 128 requests, each reading a number of tokens and then generating as many. Mamba-3 (1.5B) was compared with Llama-3.2-1B, a Transformer, served by the inference engine vLLM:

• 512 tokens: 4.39 seconds for Mamba-3, 4.45 for Llama
• 4,096 tokens: 35.11 against 58.64 seconds
• 16,384 tokens: 140.61 against 976.50 seconds, about seven times faster

The authors estimated that last Llama figure from a run with batch size 16, because batch 128 did not fit in GPU memory.

Retrieval is the weak spot. A fixed-size state has to compress the context, while attention can look everything up again in its KV cache. The authors call pure Mamba-3 competitive on question answering, and they note that it struggles to extract information from semi-structured and unstructured documents. On SWDE, which asks for a value such as a release date from the HTML of a web page, it scored 28.5 against 48.9 for the Transformer. On FDA, key-value pairs from documents of the US Food and Drug Administration, it scored 23.4 against 58.4. A hybrid with one attention layer for every five Mamba-3 layers scored 58.5 and 65.9 on those tests, above the pure Transformer.

Will Mamba-3 replace Transformers?

Its authors expect a mix. Because a fixed-size state is weak at retrieval, they predict that linear-time layers such as Mamba-3 will be used mainly in "hybrid architectures" that keep some self-attention layers. That is how SSM layers already reach larger models. NVIDIA's Nemotron-H (April 2025) replaces most attention layers with Mamba layers and reports up to three times faster inference than open Transformer models of similar size. Tencent's Hunyuan-TurboS (May 2025) also combines Mamba and Transformer layers. Mamba-3 is a candidate for the linear part of such hybrids.

Transformers keep a head start in tooling: kernels such as FlashAttention and inference engines such as vLLM were built around attention.

What should developers do with Mamba-3 now?

• Read it as research. The largest model in the paper has 1.5 billion parameters, and the longest test measures perplexity at 32K tokens. Nothing in it shows how Mamba-3 behaves at a million tokens or at the size of commercial models.
• Don't treat it as a drop-in replacement. The paper evaluates pretrained models on language, retrieval and state-tracking tests. Fine-tuning and tool calling are outside its scope.
• Measure at your own context lengths. In the paper's speed test Mamba-3 had no advantage at 512 tokens and a sevenfold one at 16,384.
• Test retrieval on your own data. If your pipeline extracts fields from documents, look at hybrid models first: that is the task on which pure Mamba-3 fell furthest behind.