Inference Night: Key Takeaways

Why inference matters

• Inference can cost far more than training over a model’s lifetime. The opening speaker estimates that DeepSeek V3 used about six times more compute for inference in a year than for training; across a two-year model lifetime, inference could matter roughly 12 times more.
• Serving efficiency affects margins and product quality. Better inference can lower cost per token while improving latency and throughput.
• A request has two main compute stages:
• Prefill: Processes the input prompt; typically compute-bound.
• Decode: Generates output tokens one at a time; typically memory-bound and often the bottleneck.
• Important metrics include time to first token, time per output token, throughput, and goodput—the work delivered while meeting service-level targets.
• KV-cache reuse can greatly reduce repeated work. Its value depends on the workload: agents often reuse context, while unrelated chat sessions may not.

Three ways to make inference more efficient

The opening overview groups improvements into three areas:

1. Model changes
2. Mixture-of-experts architectures activate only some experts per token.
3. State-space and sliding-window designs can reduce attention work.

4. Latent attention, quantization, and distillation can lower compute or memory needs.

5. Kernels and runtimes

6. Specialized kernels, attention optimizations, and better memory management improve GPU utilization.

7. Techniques such as radix attention reuse shared prefixes; paging-style approaches manage KV-cache memory.

8. Serving harnesses

9. Route requests to suitable models rather than sending everything to the largest one.
10. Use speculative decoding, context compaction, and recursive approaches that let the model select relevant context.

Dan Fu: The Life Cycle of an Inference Request

How a request moves through a serving system

1. Routing: Choose a data center and GPU replica. Good routing balances load while favoring replicas that already hold useful KV-cache data.
2. Tokenization: Convert text and tool definitions into token IDs. Tool calling requires careful handling to ensure correct formats and behavior.
3. Scheduling and batching: Queue requests and decide how to batch them, balancing throughput against responsiveness.
4. Prefill: Process the prompt.
5. Decode: Generate tokens.
6. Response handling: Convert token IDs back to text and manage the lifetime of cached data.

Three optimization areas

Kernels

• GPU kernels perform the operations at the heart of inference, including matrix multiplication and attention.
• Small operations can have poor GPU utilization when run separately; kernel fusion and workload-specific optimization can help.
• Together AI is exploring agents that optimize kernels using real production traffic, hardware, and target models, then evaluate the results end to end.

Parallelism

• Large models must be split across GPUs using approaches such as tensor, pipeline, and data parallelism.
• Newer high-bandwidth GPU systems allow models—especially expert-based models—to be distributed across larger groups of GPUs.
• Larger deployments also require strong fault tolerance: a single failed GPU should not unnecessarily disable an entire model replica.

Quantization

• Quantization reduces the number of bits used to represent model weights, lowering memory requirements and potentially increasing throughput.
• It must be validated carefully because it can affect model quality and tool use.
• Together’s evaluation process combines public benchmarks, layer-level comparisons, agent tests, and internal applications such as a Flappy Bird coding test.
• Fu described a training-free quantization recipe that matched the original model’s performance in a MiniMax example, avoiding the risk of retraining without access to the original training data.

Jeff Morgan: Measuring Open-Model Serving Quality

Why provider differences matter

• Open-model providers can serve the same nominal model differently because of differences in quantization, hardware, model versions, sampling, kernels, cache precision, or tool-call parsing.
• The quality gap between providers serving the same open model can be substantial—sometimes larger than the gap between open and closed models.
• Small error rates compound across multi-step agents. A modest chance of deviation on each step can make a long coding task fail frequently.

Ways to monitor quality

• Run evaluations: Directly test model performance and tool use. This is informative but costly and slow.
• Use tool-call scoreboards: Standardized tests can reveal whether providers handle tools correctly.
• Compare output distributions: Compare a provider’s outputs with a trusted reference model. This can be relatively inexpensive, but may not explain the cause of a mismatch.
• Track log probabilities: Compare token probabilities over time to detect silent changes in a provider’s serving stack. This works best when the API exposes log probabilities and a reliable baseline exists.
• Monitor request patterns: Track tool-call errors, output lengths, and looping behavior without retaining all user content.

Improving consistency

• Model developers can publish more complete serving stacks, not just weights and sample code.
• Providers can use model-specific quantization recipes developed with the model teams.
• Applications can standardize tool handling and reasoning extraction in their own harnesses.
• Customer-specific, private evaluations can make it harder to optimize only for public benchmarks.
• Trusted execution environments may eventually help verify which weights and software are actually serving a model.

Morgan expects routing to increasingly combine cloud providers and local models, with quality testing needed in both settings.

Joe Fiati: Compiling Models Across Hardware

Luminal’s compiler approach

• Luminal targets GPUs and other accelerators with a shared compiler built around equality saturation.
• It reduces PyTorch models to a small set of primitive operations, then uses rewrite rules to represent equivalent ways of computing the same result.
• Instead of committing early to one rewrite, the compiler builds a search space of alternatives and selects implementations based on measured performance.
• It can target different hardware backends by describing their operations and how those operations relate to the compiler’s intermediate representation.

Search and trade-offs

• The compiler can measure candidate programs on real hardware rather than relying entirely on a predictive cost model.
• It prunes candidates using analytical bounds such as memory movement and roofline estimates, then searches a limited portion of the full space.
• Compilation can take longer in exchange for faster inference, since compiled programs may run many times in production.
• Luminal focuses mainly on compiling a model’s forward pass; higher-level serving features such as speculative decoding are handled by inference engines.
• Numerical accuracy matters: changes to operation order can alter results and potentially affect output quality.

Lang Shang: SGLang and Memory Management

Radix attention and prefix reuse

• SGLang’s early design targeted agent workloads, where multiple agents often share a long prompt and then branch into different tasks.
• Radix attention organizes cached prefixes in a radix tree so requests can reuse shared context instead of recomputing it.
• Cache-aware scheduling and eviction help preserve useful prefixes under limited GPU memory.

Scheduling and hierarchical caching

• As GPUs become faster, CPU-side scheduling can create idle gaps. SGLang’s overlap scheduling prepares future batches while the GPU is working, reducing blocking and improving throughput.
• SGLang also uses a hierarchy of cache levels across GPU memory, host memory, and storage to retain context and improve cache-hit rates.

Unified memory for newer architectures

• Models may combine full attention, sliding-window attention, and recurrent state. Each component has different requirements for cache matching and memory retention.
• Managing each memory pool independently can make prefix sharing difficult and waste memory.
• SGLang’s unified radix tree coordinates these components around token sequences, while a unified memory pool can flexibly allocate space according to workload needs.
• When memory becomes fragmented, compaction reorganizes it and updates page mappings without changing the virtual page IDs.
• The talk also described SGLang’s broader open-source work, including inference and systems for supporting reinforcement-learning training rollouts.

Overall

Efficient inference depends on the whole serving stack: routing, caching, scheduling, kernels, parallelism, quantization, compilation, and quality evaluation. As open models and agent workloads grow, providers must improve not only raw tokens per GPU, but also reliability, latency, and consistency across models and hardware.