Kimi K3 Architecture: Solving the KV Cache Memory Wall

Kimi K3, the 2.8-trillion-parameter frontier model released by Moonshot AI, is the clearest current answer to a problem that has quietly become the binding constraint on large language models. It is not compute. It is memory — specifically, the key-value cache that every transformer must carry as a conversation lengthens.

The Kimi K3 architecture is worth studying even if you never deploy it, because the design choices inside it are the ones every serious model team now has to make. This is a walkthrough of what the model does, why each decision was taken, and what it changes for anyone running long-context AI in production.

Why the KV cache became the bottleneck

Every modern transformer maintains a key-value (KV) cache. As the model reads a prompt and generates a reply, it stores the intermediate values it has already computed for each token so that it does not have to recompute the entire sequence from scratch on every step.

That cache is the reason generation feels fast. It is also the reason long context is expensive. The cache grows linearly with three things at once: the number of tokens, the number of layers, and the number of attention heads per layer.

At 2.8 trillion parameters, an unconstrained cache is not a rounding error. It is the dominant memory cost of serving the model, and it produces three concrete operational problems:

• Memory exhaustion — the cache competes with the weights themselves for high-bandwidth GPU memory.
• Fewer concurrent users — each active session holds its own cache, so cache size directly sets how many sessions fit on a GPU.
• Rising latency — the longer the conversation, the more memory traffic each new token requires.

Differences in hardware access have pushed laboratories towards divergent optimisation strategies. Teams facing tighter constraints on advanced accelerators have had to extract more capability from the memory budget they already have. Kimi K3 is the clearest demonstration so far of that pressure converted into engineering advantage.

Four ways to control cache growth

The field has converged on four complementary techniques for managing the memory bill without discarding useful long-context behaviour. Kimi K3’s designers drew on all four, so it helps to understand each on its own terms.

Share — Grouped-Query Attention (GQA)

An attention head is one of many parallel “views” a layer takes over the sequence. In standard attention, each head keeps its own set of keys and values.

Grouped-Query Attention lets multiple heads share a single set. It removes redundant copies without discarding any history, which makes it the cheapest possible win — and it is now the industry default rather than an optimisation.

Shrink — Multi-head Latent Attention (MLA)

Multi-head Latent Attention compresses each token’s key-value representation into a much smaller latent vector — a dense, low-dimensional summary — and reconstructs the full representation on demand.

DeepSeek-style MLA achieves roughly 57x compression per cache entry and delivers large throughput gains. The limitation is structural: compression shrinks each entry, but the number of entries still grows with every new token. It changes the slope of the line, not its direction.

Forget — Sliding-Window Attention

Sliding-window attention imposes a hard limit on retained history. Once the window fills, older tokens are discarded.

Memory then stays constant no matter how long the conversation runs. The cost is exact recall: information beyond the window is genuinely gone unless a complementary full-attention path is retained somewhere in the stack.

Replace — Linear and delta-style attention

The most radical option abandons the per-token notebook entirely. Each head instead maintains a fixed-size state that is updated in place as tokens arrive — closer to a running summary than an archive.

Memory footprint becomes fully independent of sequence length. The historical challenge has been quality: fixed-state methods have struggled to match full attention on exact recall of specific earlier facts.

Kimi Delta Attention and the 3:1 hybrid

Kimi K3 combines “replace” and “shrink” in a deliberately engineered hybrid rather than committing to either.

Three of every four attention layers use a fixed-size recurrent state called Kimi Delta Attention (KDA). The fourth layer retains a compressed full-recall path built on an enhanced form of Multi-head Latent Attention known as Gated MLA.

KDA operates as a recurrent state machine. A single fixed-size memory matrix is carried forward and edited by each successive token — the model does not accumulate a record, it revises one. Two refinements make that fixed state competitive with full attention rather than merely cheaper.

A delta-style update rule. Before writing a new value for the current key, the mechanism erases the previous association for that key. Without this, repeated writes to similar keys interfere with each other and the state degrades into noise. Removing the old association first keeps the memory clean as it is rewritten thousands of times.

128 independent learned decay rates, one per channel. Rather than forgetting everything at a single uniform rate, the model learns what to hold on to. Important facts persist across long horizons while transient detail fades. A useful side effect is that timing information becomes encoded in the decay pattern itself, reducing reliance on separate positional encodings.

The evidence behind the 3:1 ratio

Architectural claims at trillion-parameter scale are easy to make and expensive to verify. Moonshot’s approach was to validate the interleaving ratio first on a controlled 48-billion-parameter testbed known as Kimi Linear.

That testbed was trained under identical data and identical recipes, differing from its comparison model in one variable only: the attention mechanism. This is the right way to run the experiment, because it isolates the architecture from the confounding effect of data quality.

The hybrid configuration delivered three results against its full-attention twin:

• Up to 75% reduction in KV-cache memory
• Approximately 6x higher throughput at million-token contexts
• Superior long-context benchmark scores — not merely comparable ones

Critically, this came with no measurable quality degradation. That proven recipe was then scaled into the 2.8-trillion-parameter flagship, which is why the design reads as a considered engineering decision rather than a bet.

The supporting cast: sparsity, stability and 4-bit training

Memory efficiency is only one axis. Several complementary mechanisms address information flow, extreme sparsity, optimiser stability and quantisation-aware training.

• Attention Residuals — later layers can selectively attend back to earlier layer outputs rather than relying solely on a single residual stream, improving information survival across depth.
• Stable Latent MoE — a mixture-of-experts design with 896 experts, of which only 16 routed plus 2 shared are active per token, giving roughly 104 billion active parameters out of 2.8 trillion total. Latent projections keep inter-GPU communication compact, which matters because expert routing is a networking problem as much as a modelling one.
• Quantile Balancing — per-expert load thresholds derived dynamically from routing scores, preventing chronic overloading of popular experts without hand-tuned auxiliary losses.
• Per-Head Muon — an optimiser that updates each attention head independently, stopping a few high-magnitude heads from dominating the learning dynamics of an entire block.
• Native 4-bit post-training — the entire post-training phase runs under the 4-bit weight constraint, so the model adapts to low precision rather than suffering post-hoc compression artifacts. Compression focuses on the expert parameters that dominate storage.

Built for agents that run for hours

None of this is optimisation for its own sake. Kimi K3 is explicitly designed for long-horizon agentic workloads: multi-hour, multi-tool jobs such as large-codebase navigation and editing, GPU-kernel optimisation, CAD workflows and register-level chip-design demonstration tasks.

These workloads share a defining property. They return repeatedly to the same large context — the same repository, the same document set, the same tool schemas — over hours of operation.

That makes prefix-caching economics decisive. A fully cached million-token working set can be reused at a fraction of the cost of a cold prefill, so the cost curve of an agent session is set largely by how well the architecture holds a stable context in memory.

Moonshot pairs the model with custom kernels (FlashKDA), expert-parallel communication layers (MoonEP) and resumable agent sandboxes. Model, kernels and orchestration were co-designed for the same long-running, stateful workload profile — which is the only way theoretical memory savings become attainable in production.

The bottom line

The strategic read is straightforward. The next competitive advantage will not come from raw parameter count or peak FLOPS. It will come from systems that maintain useful, selectively retained context across million-token horizons while maximising concurrent sessions per GPU and minimising the cost of context reuse.

Memory architecture is now a primary design dimension alongside compute and interconnect. Kimi K3’s combination of fixed-state attention, hybrid full-recall paths, extreme MoE sparsity and quantisation-aware training is a coherent response to the dual pressure of scale and hardware constraint — and these patterns will shape both open-model development and the practical economics of large-context agent platforms through 2026 and 2027.

This is the same set of trade-offs we work through in building Yehia, our Arabic-first language model — where serving efficiency and long-context reliability matter as much to real users as leaderboard scores do. The most useful way to understand what these choices feel like in practice is to run a long, demanding session yourself: try Yehia in the Navid Playground and watch how the model behaves as the context grows.