Kimi K3 Open Model Surpasses Closed GPT and Anthropic Systems on Independent Benchmarks

Kimi K3 is an open-weight model that outperforms several closed-source leaders on independent benchmarks while remaining fully downloadable and deployable locally. Developed by Moonshot AI and analyzed in detail by Positive Technologies, the model reaches these results through simultaneous scaling in three directions: context length, layer depth, and expert count.

The first challenge was supporting a one-million-token context. Standard transformers maintain a KV cache that grows linearly with sequence length. Kimi K3 replaces this cache with a fixed-size state matrix S updated by the delta-rule. Two coefficients control the update: β for write strength and α for forgetting. To prevent underflow and overflow in BF16 arithmetic during long sequences, α is clamped between e⁻⁵ and 1.

Because state updates are inherently sequential, the team introduced the FlashKDA kernel, released alongside the model weights. FlashKDA parallelizes token processing inside each chunk on individual streaming multiprocessors and uses prefix-scan operations across GPUs to reconstruct correct sequential states after parallel segment processing.

Information propagation across many layers was addressed by replacing direct layer-to-layer handoff with learned pseudo-queries. Each layer retrieves a weighted combination of outputs from all preceding blocks rather than receiving only the immediate predecessor result. Layers are grouped into eight blocks of twelve layers each to keep memory requirements manageable.

The model employs 896 routed experts plus two shared experts. Tokens are first projected into a latent space half the original dimension. Only 16 routed experts are activated per token according to per-expert quantile thresholds that guarantee balanced load. A custom SiTU-GLU activation replaces SwiGLU to bound activation growth and prevent overflow during the four consecutive matrix multiplications inside each expert block.

These combined techniques allow Kimi K3 to maintain training stability and inference efficiency at unprecedented scale while remaining fully open. The architecture paper and FlashKDA kernel are publicly available, providing the community with concrete methods for scaling open models beyond current closed-source performance on selected tasks.