AWS Trainium Gets SGLang: 17 Models, Kimi-K3 on 3 Nodes

Amazon says SGLang-Neuron is “publicly available”, letting developers serve models and generate reinforcement-learning rollouts with the open-source SGLang engine on AWS Trainium chips “using the same SGLang interface you use today”. The SGLang team’s release post calls v0.1.0 “the first public release of SGLang’s backend for AWS Trainium”.

The release post lists 12 language models and 5 diffusion models, 17 in all by our count. Its largest deployment, Kimi-K3, runs across three Trainium3 nodes, and GLM-5.2 serves a 1M-token context. On GPT-OSS-120B, the post says a host-memory cache delivers 72% to 75% more output throughput.

Business Pill · GPUs explained

An evergreen Business Engineer explainer on what GPUs do and why AI depends on them. It teaches the general idea only and does not discuss AWS, Trainium or this release.

What AWS and the SGLang Team Released

AWS says the software is accessed through an AWS Deep Learning Container and was “released by RadixArk in collaboration with Annapurna Labs”, Amazon’s chip unit. The SGLang post says it is “now open source and available in public preview”, co-developed by the SGLang team at RadixArk and Amazon Annapurna Labs, and that it “follows the closed beta” announced in October, about three weeks earlier by our arithmetic.

AWS describes it as an out-of-tree plugin: SGLang’s core keeps the API server, request handling and prefix caching, while the plugin implements “the Trainium-specific models, attention, mixture of experts (MoE), and the graph runner”. AWS’s walkthrough uses a Trainium3 instance. The code is on GitHub.

Which Models It Serves

The release post’s model table includes Kimi-K3.1, Kimi-K3, GLM-5.3, GLM-5.3-Flash, GLM-5.2, Qwen3.5, three Qwen3 sizes, GPT-OSS-120B, GLM-4-9B and DeepSeek-V2-Lite. Its diffusion table lists FLUX.1-dev, FLUX.3-Action, MiniMax-H3, Wan2.2 A14B and LingBot-World-V2, which it says all serve on one trn2.48xlarge.

The post says Kimi-K3 “is the largest deployment in this release”: “It runs on three Trainium3 nodes, one for prefill and two for decode”. It says GLM-5.2 serves 128K and 1M decode points, so the 1M-token point is about 7.8x the smaller one by our arithmetic, and that it runs on the public Neuron SDK 2.33.0.

The Throughput Figure

The one throughput number in the post is for its hierarchical cache, which keeps evicted cache pages in host memory and loads them back on a later hit. “On GPT-OSS-120B at TP8 and 32K context, HiCache L2 delivers 72% to 75% more output throughput”, the post says.

The post also says the reinforcement-learning loop was validated end to end on Qwen3-30B-A3B, with rollouts served by SGLang-Neuron and training on the TorchTitan backend. Our earlier pieces cover Amazon’s plan to sell Trainium outside AWS and inference passing half of AWS’s AI revenue.

The Structural Read

As we read it, the scale claim is the three-node Kimi-K3 deployment, which the post calls the largest in the release, with one prefill node and two decode nodes.

As we read the one throughput figure, it measures a cache setting on one model: the post says HiCache L2 delivers 72% to 75% more output throughput on GPT-OSS-120B at TP8 and 32K context.

Three Implications

FOR DEVELOPERS ON SGLANG AWS says SGLang-Neuron lets them serve models and generate RL rollouts on Trainium using the same SGLang interface, through an AWS Deep Learning Container.

FOR AWS AWS says the plugin was released by RadixArk in collaboration with Annapurna Labs, and its walkthrough uses a Trainium3 instance.

FOR OPEN-WEIGHT MODEL MAKERS The release post lists Kimi, GLM, Qwen, GPT-OSS and DeepSeek models among the 12 language models it serves on Trainium.

The Business Engineer Lens

This story maps onto the Business Engineer framework The State of The Inference Economy.

The framework says: “Inference is a revenue engine: continuous, distributed across millions of applications and agents, incurring marginal cost with every query.”

As we read it, SGLang-Neuron is about where that serving runs: on Trainium as well as GPUs; the framework does not discuss SGLang or Trainium.

What Is Not Established

Neither post compares Trainium serving speed or cost with Nvidia GPUs, and neither gives prices for running these models on Trainium. The 72% to 75% figure is for one model and one cache setting, measured by the developers, not independently.

The figures come from AWS and the SGLang team and were not independently verified. AWS was not contacted for this piece.

The Bottom Line

AWS says SGLang now runs on Trainium through a public plugin, and the release post lists 17 models, a three-node Kimi-K3 deployment and a cache that it says adds 72% to 75% output throughput on GPT-OSS-120B.