SGLang, vLLM and the rise of the inference operating system

For the last two years, many enterprise AI conversations have focused on model choice: which foundation model, which context length, which benchmark, which price per token.

That discussion still matters. But the deeper infrastructure story is moving elsewhere.

The real bottleneck is becoming the inference operating system: the layer that decides how requests are batched, where KV cache lives, when prefill and decode are separated, how latency-critical traffic is protected, how structured outputs are constrained, and how GPU capacity is governed across tenants.

SGLang, vLLM and Kubernetes-native stacks such as llm-d are interesting because they make this shift visible. They are not merely “model servers”. They are early versions of the runtime control plane enterprises will need for agentic AI.

Why inference is different from normal application hosting

Traditional web infrastructure mostly scales stateless request handling. LLM inference is different.

Each request carries a growing state footprint: the KV cache. High throughput requires batching many requests, but those requests arrive at different times, have different prompt lengths, produce different output lengths, and compete for scarce accelerator memory. The vLLM paper framed this clearly: inefficient KV cache management wastes memory through fragmentation and duplication, limiting effective batch size; PagedAttention borrows the operating-system idea of paging to reduce that waste and share cache more flexibly.[9]

That is why inference infrastructure starts to resemble an operating system:

• memory management for KV cache,
• scheduling for prefill, decode and mixed workloads,
• admission control for overload,
• device abstraction across GPUs and accelerators,
• policy enforcement for tenants, models and data classes,
• observability for latency, throughput, queueing and cost.

The engineering question is no longer simply “can we run the model?” It is “can we operate the model as a reliable shared service?”

vLLM: the memory-and-throughput engine becomes a broader runtime

vLLM’s original breakthrough was PagedAttention: treating attention KV memory more like virtual memory, so serving can keep larger and more dynamic batches alive without excessive waste.[9]

The current vLLM direction is broader. Its README now presents vLLM as a fast serving library with PagedAttention, continuous batching, chunked prefill, prefix caching, CUDA/HIP graph execution, quantization, speculative decoding, distributed inference, structured outputs, tool-calling parsers and multiple serving APIs.[2]

The V1 architecture work is especially important. The vLLM team described a re-architecture of core components including the scheduler, KV cache manager, worker, sampler and API server; the V1 blog also calls out a simpler scheduler, near-zero-overhead prefix caching, cleaner tensor-parallel inference and default optimisations for higher-throughput serving.[7]

The strategic implication: vLLM is moving from “efficient serving engine” toward a runtime substrate where scheduling, cache policy, execution graphs and API compatibility are part of one operating model.

For enterprises, vLLM is attractive when the platform team wants:

• broad Hugging Face model compatibility,
• OpenAI-compatible serving as the default contract,
• strong throughput economics,
• many quantization and kernel back-end options,
• a large ecosystem around production serving.

But the architectural lesson is larger than vLLM itself: KV cache and scheduler policy are now first-class enterprise architecture concerns.

SGLang: inference for structured, multi-step language programs

SGLang approaches the problem from a slightly different angle. The SGLang paper starts from the observation that modern LLM applications increasingly require multiple generation calls, prompting control flow, parallelism, and structured inputs/outputs; SGLang combines a frontend language with a runtime optimized for those structured language-model programs.[10]

That runtime matters. SGLang’s README highlights RadixAttention for prefix/KV cache reuse, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, distributed parallelism, structured outputs, chunked prefill, quantization and multi-LoRA batching.[1]

The LMSYS team’s serving write-up is useful because it connects these features to real operating experience: SGLang Runtime has been used in Chatbot Arena-scale serving, combines optimised kernels with RadixAttention and compressed state machines for constrained decoding, and emphasises a highly efficient Python scheduler.[8]

SGLang is especially interesting for agentic and workflow-heavy systems because enterprise AI is not only “one prompt in, one answer out”. It is increasingly:

• retrieve, reason, call tools, verify;
• generate structured JSON, then validate it;
• branch based on model output;
• run parallel sub-prompts;
• reuse prefixes across similar tasks;
• route some paths through stronger or cheaper models.

In that world, the runtime must understand more than tokens. It must understand the shape of the program.

The new stack is disaggregated

Both SGLang and vLLM now point toward a more disaggregated inference architecture.

Instead of one monolithic server doing everything, teams increasingly separate:

1. gateway and routing — who is allowed in, which model or lane receives the request;
2. prefill — compute-heavy prompt ingestion;
3. decode — latency-sensitive token generation;
4. KV cache tiers — GPU memory, CPU memory, disk or external cache;
5. model execution workers — tensor, pipeline, data, expert or context parallelism;
6. observability and policy — SLOs, quota, cost attribution, audit trails.

vLLM’s recent release notes show this direction in the details: model-runner maturation, encode/prefill/decode disaggregation, KV offloading, connector plumbing and structured-output fixes appear beside kernel and model-support work.[6] SGLang’s latest release notes similarly include speculative decoding, data-parallel attention and performance-oriented scheduler changes alongside model and hardware support.[5]

This is what infrastructure maturity looks like: not one magic optimisation, but many control surfaces becoming explicit.

Kubernetes-native AI infra: from server to control plane

The next layer is the cluster operating model.

llm-d describes itself as a high-performance distributed inference serving stack for production deployments on Kubernetes, with guides and benchmarks for open-source models across accelerators and infrastructure providers.[4] Its README also emphasises intelligent routing and advanced KV-cache management, including tiered offloading to CPU or disk and global prefix-cache indexing.[4]

That is important because enterprise AI platforms cannot be operated one server at a time. They need a control plane that can answer:

• Which requests deserve premium latency?
• Which tenants can use which models?
• When should traffic fall back to a smaller model?
• How should batch jobs avoid starving interactive copilots?
• How much GPU cost belongs to each product domain?
• What happens when cache, queue or accelerator capacity saturates?

This is where AI infrastructure starts to look like a combination of API gateway, scheduler, memory manager, FinOps meter and risk-control system.

How to choose between SGLang and vLLM

The wrong enterprise question is: “Which one is the winner?”

The better question is: “Which workload contract are we trying to operate?”

A practical decision frame:

Workload pattern	Bias toward
Broad OpenAI-compatible model serving with large ecosystem support	vLLM
High-throughput generic inference with many model and quantization options	vLLM
Structured generation, constrained decoding and multi-step LLM programs	SGLang
Prefix-heavy agent workflows where cache reuse is central	SGLang
Kubernetes-native multi-tenant inference platform	vLLM/SGLang behind a gateway or llm-d-style control plane
Regulated enterprise workloads	whichever stack exposes the clearest policy, telemetry and isolation model

In practice, large organisations may run more than one engine. The architecture should allow that.

Do not hard-code the enterprise AI platform around one runtime as if it were a database. Treat inference engines as replaceable execution backends behind stable platform contracts: APIs, routing policy, observability, cost labels and security controls.

Enterprise architecture pattern: three inference lanes

A useful reference architecture is a three-lane inference platform:

1) Premium interactive lane

For customer-facing copilots, executive assistants, incident response and high-value workflows.

Design goals:

• strict p95/p99 latency SLOs,
• reserved capacity or priority scheduling,
• short queues and explicit backpressure,
• strong traceability and fallback rules.

2) Standard productivity lane

For internal assistants, knowledge search, document generation and team automation.

Design goals:

• high throughput,
• efficient batching,
• prefix caching,
• budget-aware model routing,
• standardised observability.

3) Batch and evaluation lane

For offline enrichment, synthetic data, red-team testing, evaluation and report generation.

Design goals:

• lower-cost capacity,
• relaxed latency,
• long-running job controls,
• clear cost attribution,
• separation from interactive traffic.

SGLang and vLLM can both sit behind this pattern. The strategic work is not only engine tuning. It is defining the lanes, policies and operational semantics.

What CTOs should ask now

If your organisation is moving beyond AI pilots, ask these questions before the next GPU purchase:

1. Where does KV cache policy live? If nobody owns it, nobody owns inference economics.
2. Can we separate prefill-heavy and decode-heavy workloads? If not, one workload class will quietly damage another.
3. Do we route by business criticality or only by endpoint? AI traffic needs intent-aware routing.
4. Can we explain cost per product, tenant and workload class? Without labels and metering, GPU cost becomes a shared mystery.
5. What is our overload behaviour? A mature platform returns controlled degradation, not random timeouts.
6. Can we swap inference engines without rewriting the business application? Stable contracts matter more than runtime fashion.

The takeaway

The next competitive advantage in enterprise AI will not come only from picking a better model.

It will come from operating inference as a governed platform: memory-aware, scheduler-aware, cache-aware, policy-aware and cost-aware.

vLLM shows how much performance can be unlocked when KV cache and batching are treated as core systems problems. SGLang shows why structured, multi-step language programs need a runtime that understands more than a single completion call. Kubernetes-native stacks such as llm-d show where the operating model is heading: a distributed control plane for inference, not a collection of ad-hoc model servers.

For enterprise architecture, the message is simple: build the inference operating system before agentic AI turns your GPU estate into an unmanaged production dependency.

Sources

[1] https://raw.githubusercontent.com/sgl-project/sglang/main/README.md — SGLang README [2] https://raw.githubusercontent.com/vllm-project/vllm/main/README.md — vLLM README [4] https://raw.githubusercontent.com/llm-d/llm-d/main/README.md — llm-d README [5] https://api.github.com/repos/sgl-project/sglang/releases/latest — SGLang latest GitHub release [6] https://api.github.com/repos/vllm-project/vllm/releases/latest — vLLM latest GitHub release [7] https://blog.vllm.ai/2025/01/27/v1-alpha-release.html — vLLM V1 alpha release blog [8] https://lmsys.org/blog/2024-07-25-sglang-llama3 — SGLang LMSYS Llama 3 blog [9] https://arxiv.org/abs/2309.06180 — vLLM paper [10] https://arxiv.org/abs/2312.07104 — SGLang paper