Serving at the Limit: LLM Inference with vLLM and Triton on Kubernetes

When an AI model leaves the training phase, the real challenge begins: productive inference operation. Serving a Large Language Model (LLM) in a standard container is inefficient. Latencies are too high, and GPU utilization is often poor because traditional web servers are not built for the sequential nature of token generation.

To operate LLMs economically and efficiently in the mid-market, we need to combine Kubernetes with specialized Inference Engines and intelligent Batching.

The Inefficiency of Naive Serving

An LLM generates token by token. When a user makes a request, the GPU is occupied for the entire duration of the response. In a naive setup, the second user would have to wait until the first response is finished—or we would have to provide a separate GPU for each user. Both are unacceptable in a production environment.

The Solution: PagedAttention and Continuous Batching

The current gold standard for solving this problem in Kubernetes is vLLM (or alternatively the NVIDIA Triton Inference Server).

1. vLLM and PagedAttention

vLLM revolutionizes memory management. Inspired by the virtual memory management of operating systems, vLLM uses PagedAttention.

• The Key: The KV-cache (key-value store of previous tokens) is no longer reserved in a large contiguous block in VRAM but in dynamic “pages.”
• The Result: We can process up to 24 times more requests on the same hardware than with traditional methods, as the fragmentation in GPU memory is almost reduced to zero.

2. Continuous Batching

Instead of waiting for a complete batch of requests to be fully generated, vLLM immediately funnels new requests as soon as a token from another user is ready. In your Kubernetes cluster, this leads to consistent, high GPU utilization and minimal wait times for end users.

Deployment Strategies: KServe and Knative

Inference workloads are extremely “spiky.” At night, you often need zero capacity, while during the day, requests skyrocket. This is where Serverless AI on K8s shines.

We rely on KServe:

• Scale-to-Zero: When no requests come in, the expensive GPU nodes are shut down via Knative.
• Canary Rollouts: You can test a new, fine-tuned model (e.g., Llama-3-70B v2) on just 5% of the traffic before switching the entire infrastructure.
• Transformer Architecture: KServe allows pre- and post-processing steps (such as tokenization or content filtering) to be separated from the actual inference container, simplifying scaling.

The Role of the Model Registry

In a professional K8s infrastructure, models are not “in the image.” This would inflate the container images to hundreds of gigabytes.

• The ayedo Way: Models are stored in an S3-compatible storage (MinIO or Cloud-Native Storage). When the pod starts, an Init-Container loads the model directly into the shared memory or onto the local NVMe disk of the node. This ensures fast startup times and clean versioning.

Conclusion: Inference is an Optimization Sport

To successfully deploy AI in the mid-market, one must reduce inference costs per request. Kubernetes, with vLLM and KServe, offers the perfect platform to maximize the utilization of expensive hardware. It is the step from “playing around” to a scalable digital product.

Technical FAQ: Inference & Serving

Which is better: vLLM or Hugging Face TGI (Text Generation Inference)? Both are excellent. vLLM currently often leads in throughput rate (tokens per second), especially due to PagedAttention. TGI, on the other hand, is very stable and deeply integrated into the Hugging Face ecosystem. We usually evaluate this based on the specific model type.

How large should the shared memory (/dev/shm) be for inference pods? AI frameworks massively use shared memory for inter-process communication. By default, this is very small in K8s pods (64MB). We often configure this via emptyDir with medium: Memory to several gigabytes to avoid crashes with large models.

Can I run inference on CPUs? For very small models (e.g., BERT for text classification) or with quantization (GGUF format via llama.cpp), this is possible. However, for modern LLMs from 7B parameters onwards, latency on CPUs is usually too high for a good UX.