GPU Slicing & Kubernetes: How to Efficiently Share Expensive AI Resources

In modern IT infrastructure, the GPU has become the new CPU. Whether it’s Large Language Models (LLMs), computer vision, or complex data analysis, the demand for computing power on graphics cards has massively increased in the mid-market. However, while CPUs have been efficiently virtualized and shared for decades, GPUs often present platform engineers with a dilemma: A high-end graphics card (like an NVIDIA H100 or A100) is often oversized for a single microservice, yet too expensive to leave idle.

The solution to this problem is GPU Slicing. In this post, you will learn how to partition your expensive hardware in Kubernetes clusters so that multiple workloads can benefit simultaneously without blocking each other.

The Challenge: The “All-or-Nothing” Principle

By default, Kubernetes treats a GPU as an indivisible resource. A pod requests nvidia.com/gpu: 1, and the system assigns the entire hardware exclusively to it. While this might make sense for intensive model training, for inference (running a model), where the GPU might only be 15% utilized, it leads to massive resource wastage.

To increase efficiency, three technical approaches have been established, which we use at ayedo in the context of sovereign infrastructures.

1. Multi-Instance GPU (MIG)

NVIDIA MIG is the most robust form of slicing at the hardware level (available from the Ampere architecture). Here, a physical GPU is divided into up to seven independent instances.

• Technical Depth: Each instance has its own isolated portion of memory (VRAM) and compute cores (SMs). This means hardware isolation: If a workload crashes or floods the memory in one instance, the other six instances remain completely unaffected.
• Advantage: Guaranteed Quality of Service (QoS) and security in multi-tenant environments.

2. Time-Slicing (Temporal Sharing)

If your hardware does not support MIG (e.g., older T4 or consumer cards), time-slicing is the way to go. Here, the GPU scheduler of Kubernetes is configured to allow multiple pods on the same GPU.

• Technical Depth: The GPU switches between different tasks in extremely fast cycles. However, there is no memory isolation. If a pod requests too much VRAM, all other pods on this GPU can crash with an Out-of-Memory (OOM) error.
• Advantage: Works on almost any hardware and is ideal for small inference services.

3. NVIDIA MPS (Multi-Process Service)

MPS is a software layer that sits between the application and the hardware. It allows you to allocate compute resources as a percentage.

• Technical Depth: You can assign a pod exactly 20% of the compute capacity, for example. The system bundles the kernel calls of different processes into a single task for the GPU, reducing overhead compared to time-slicing.

Implementation in Kubernetes Cluster

To utilize GPU slicing, we rely on the NVIDIA GPU Operator. This automates the loading of drivers, the configuration of the container runtime (nvidia-container-runtime), and the labeling of nodes.

The allocation is done as usual via YAML resource definitions. Instead of requesting a whole GPU, we use profiles:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

This approach allows you to consolidate development, staging, and production workloads on the same physical machine, drastically reducing operational costs.

Conclusion

By 2026, GPU slicing is no longer a gimmick but an economic necessity. By intelligently partitioning your hardware, you avoid “resource islands” and ensure that your AI initiatives remain scalable. Especially regarding digital sovereignty, this approach enables you to operate powerful AI services on your own hardware (on-premise or colocation) that can compete price-wise with major public cloud providers.

FAQ

What is GPU slicing in Kubernetes? GPU slicing refers to various techniques (such as MIG or time-slicing) to divide a physical graphics card into multiple smaller units. This allows multiple containers or Kubernetes pods to access the same GPU simultaneously, improving hardware utilization and reducing costs.

When should I use NVIDIA MIG? MIG (Multi-Instance GPU) is ideal when you need hard isolation between workloads. Since memory and compute cores are separated at the hardware level, it is the safest method for multi-tenant clusters or critical production environments, but it requires Ampere generation hardware (e.g., A100) or newer.

Can I share GPUs on older hardware? Yes, through “time-slicing”. Here, the driver divides the GPU time between the pods. However, since there is no true memory isolation, developers must ensure that applications do not overload the graphics memory (VRAM).

How does GPU slicing affect performance? With MIG, there is virtually no performance loss as resources are dedicated. With time-slicing, minimal latencies can occur due to context switching. In most inference scenarios, however, this effect is negligible compared to the cost savings.

Does ayedo support managed GPU infrastructures? Yes, ayedo integrates GPU support natively into managed Kubernetes environments. We configure the NVIDIA GPU Operator and assist companies in implementing slicing strategies that are both economically efficient and technically stable.