A linear attention model designed to bypass the limits of the KV cache

The Laplace 3B model, developed by the i14 lab, relies on a linear attention architecture rather than the classic softmax attention mechanism used by most large language models (LLMs). Unlike traditional transformer architectures, whose key-value (KV) cache grows with context length, Laplace 3B maintains a fixed recurrent state of 29.5 MB per sequence, regardless of the text size processed. This approach, based on Gated Linear Attention (GLA), compresses data history into a constant-size matrix, reducing attention computational complexity from O(T²) to O(T), according to explanations provided by the AWS Machine Learning blog.

This innovation addresses a major challenge of current models: managing long contexts. By eliminating dependence on an extensible KV cache, Laplace 3B paves the way for applications requiring persistent and predictable memory, such as long-term conversational agents or analysis of large documents, where serving requirements are stable and predictable.

Laplace 3B’s linear attention compresses history into a fixed recurrent state, eliminating dependence on an extensible KV cache and enabling applications requiring persistent and predictable memory.

A collaboration between i14 and AWS for a native implementation on Trainium2

To fully leverage the capabilities of the AWS Trainium2 processor, i14 and AWS collaborated on a complete rewrite of Laplace 3B’s implementation. The model, initially optimized for GPUs with specific Triton kernels, was reimplemented using standard PyTorch operators, compatible with Trainium2’s native PyTorch support via TorchNeuron Native. This adaptation allowed the team to take advantage of the processor’s hardware-specific features, including its software-managed on-chip memory (SBUF), composed of 128 partitions of 28 MiB each, as well as its 128×128 Tensor Engine.

Optimizations include a static unrolling of the GLA recurrence over 16 steps, necessary to meet Trainium2’s graph size and on-chip memory constraints. Additionally, the MuonFSDP2 optimizer was used to reduce inter-processor communication traffic by 24 times while increasing throughput by 1.63 times, as detailed in the performance metrics measured on a trn2.48xlarge instance (16 Trainium2 chips).

Record-breaking performance in training and serving on a single instance

The results obtained on the trn2.48xlarge instance are significant: Laplace 3B achieves a throughput of 120,000 tokens/second in training and a throughput of 11,544 output tokens/second in serving, while handling up to 1,280 concurrent sequences. These figures illustrate the efficiency of the optimized architecture for demanding workloads, where latency and scalability are critical.

The use of the Muon optimizer, combined with variable half-life geometric decay rates (from 1 to 2,048 tokens), enables dynamic management of attention on long sequences. This flexibility strengthens the model’s adaptability to varied use cases while maintaining controlled memory consumption.

What this means for businesses and administrations in West Africa

For Beninese or West African organizations working on projects requiring high-performance language models with long contexts, such as legal or administrative document analysis, or developing virtual assistants for customer service, this advancement could offer more cost-effective and scalable solutions. Independence from an extensible KV cache would reduce infrastructure costs, particularly for administrations handling large volumes of textual data without investing in high-end servers.

Local companies specializing in natural language processing or data science could also leverage architectures like Laplace 3B to deploy lightweight yet powerful models, tailored to the bandwidth and computational constraints often limited in West Africa. However, adoption would depend on the availability of AWS Trainium2 instances in the region, as well as the technical teams’ ability to integrate tools like PyTorch and TorchNeuron into their existing pipelines.

Finally, for public actors such as the Beninese government or regional institutions, this technology could inspire initiatives aimed at modernizing document management systems or improving access to digital services, provided that hosting costs on advanced cloud infrastructures remain affordable and aligned with local budgets.