What problem are state-space models (like Mamba) trying to solve relative to standard attention, and why haven't they fully replaced transformers?

Standard self-attention has two related long-context costs: the attention computation itself scales quadratically with sequence length (every token attends to every other token), and the KV cache needed at inference grows linearly with how many tokens have been generated so far, becoming a serious memory bottleneck at long context lengths.

State-space models process sequences through a recurrent-style mechanism with a fixed-size hidden state, similar in spirit to older RNNs but designed to be trained efficiently in parallel. Their compute cost scales linearly with sequence length rather than quadratically, and their “cache” at inference is a constant-size state vector rather than something that grows with context — a genuinely attractive property for very long sequences.

They haven’t displaced transformers because that fixed-size state is also their weakness: a state-space model has to compress everything it needs to remember into a bounded-size representation, and tasks that need precise recall of a specific fact from far back in a long context have historically favored attention’s ability to look directly at any prior token. The most promising current direction is hybrid architectures that mix attention layers with state-space layers, rather than a full replacement in either direction.