Storage in Kubernetes

Storage in Kubernetes is by no means trivial. Stateful workloads impose the highest demands on stability, performance, and availability—handling persistent data is thus one of the most complex tasks in the Cloud-Native environment. This article provides a comprehensive overview: from CSI, Cloud vs. On-Premise CSI, Longhorn, Ceph, and another solution, to Cloud-Controller-Manager, costs, scaling, redundancy, security, and the challenges of local storage landscapes.

What is CSI (Container Storage Interface)?

CSI, the Container Storage Interface, is a standard interface (API specification) that allows Kubernetes or other container orchestrations to work with any storage solutions (block or file) without changes to the Kubernetes core code. This enables external storage providers to develop CSI drivers and deploy them independently of the Kubernetes release cycle. The architecture typically includes two components: a Controller Plugin (for provisioning, attachment, etc.) and a Node Plugin (for mounting).

Advantages of CSI:

• Independent of the Kubernetes core (no more “In-Tree” plugins needed)
• Supports dynamic provisioning, snapshots, resize
• Drivers can be rolled out independently and individually

How does storage work “under the hood” in Kubernetes?

Kubernetes initially manages ephemeral storage via volumes that are tied to the lifecycle of pods. For persistence, we use:

• PersistentVolumes (PV) – abstraction of a storage space
• PersistentVolumeClaims (PVC) – user requests for storage
• StorageClasses – declare properties (e.g., SSD, replication)

Under the surface, Kubernetes performs the following for a PVC:

1. Recognize installed CSI driver
2. Dynamically provision volume (Controller Plugin)
3. Attach to node
4. Mount through Node Plugin
5. Pod receives persistent storage

In the background, block devices can be created, encrypted, buffered, or replicated—depending on the CSI driver.

Comparison: Cloud-CSI vs. On-Premise/Bare-Metal CSI

Cloud-CSI

• In cloud environments, providers like AWS, GCP, or Azure provide their own CSI drivers (e.g., AWS EBS CSI, GCE PD CSI)
• Highly integrable with cloud APIs: automatic provisioning, high availability, regionally distributable
• Convenient scalability as needed

On-Premise/Bare-Metal

• Local infrastructure: own hardware, often via iSCSI, RBD, NFS, etc.
• Requires own CSI drivers or storage stacks like Longhorn, Ceph, OpenEBS ZFS LocalPV, or StorPool
• More control, but higher responsibility: hardware, redundancy, configuration lie with the user

Comparison: Longhorn, Ceph, and another bare-metal capable solution

4.1 Longhorn

Longhorn is a Cloud-Native, distributed block storage developed by Rancher Labs. It runs lightweight, is easy to deploy via Helm, and offers replication, snapshots, restoration, high availability on standard hardware.

Advantages:

• Easy setup, user-friendly UI
• Ideal for small to medium-sized bare-metal clusters
• Low administrative effort

Disadvantages:

• Scalability lower than Ceph
• Performance overhead possible with intensive use, but improvable through optimization (uBlk, etc.)

Ceph (RookCeph)

Ceph is an established, distributed storage system that unites block (RBD), file (CephFS), and object storage (RGW) in one system. It offers high scalability, fault tolerance, self-healing, snapshotting, high data integrity, and is widely used in large production systems (e.g., CERN, OVH).

Advantages:

• Extremely scalable, versatile (object, file, block)
• High redundancy, data integrity, self-managing
• S3-compatible gateway (RGW)

Disadvantages:

• Configuration and operation complex, high maintenance effort
• Performance can be high in write-heavy and latency-sensitive scenarios

4.3 OpenEBS ZFS LocalPV (Bare-Metal Capable Solution)

OpenEBS ZFS LocalPV uses local ZFS drives per node, combined with Kubernetes StorageClasses via CSI. Many operators appreciate checksum protection (bitrot prevention), snapshots, CoW advantages—and combined with Longhorn, it results in a flexible, high-performance bare-metal solution.

Advantages:

• High local performance, ZFS features (snapshots, checksums, CoW, quotas)
• Flexible combinability (e.g., with Longhorn for replication)
• Particularly suitable for data-intensive workloads on-premise

Disadvantages:

• No native cloud scale, complex with consistent cluster replication
• Multi-node requires additional tools for replication

Comparison Table

Storage Handling in the Cloud (CSI)

In cloud environments, CSPs (Cloud Service Providers) take over many tasks:

• Install CSI drivers (e.g., via Helm)
• Define StorageClasses (e.g., type: gp3 for AWS EBS)
• Dynamic provisioning with PVC: Volume → Attachment → Mount
• Region/zone selection, automatic resize, snapshots, backups integrated depending on the cloud provider

The Cloud-Controller-Manager (CCM) takes on tasks such as:

• Node annotation based on cloud metadata (zones, network)
• Automatic management of LastClas, LoadBalancer services, LLDP, etc.
• CCM works closely with CSI drivers to orchestrate cloud-specific storage operations in Kubernetes (e.g., VolumeAttachment on AWS)—often in combination.

6. Role of the Cloud-Controller-Manager in Kubernetes

The Cloud-Controller-Manager decouples cloud-specific logic from the core Kubernetes. It takes on tasks such as:

• Node-Controller: synchronize nodes with cloud status
• Volume-Controller: orchestrate CSI drivers on cloud volumes
• Route/LoadBalancer-Controller: manage cloud load balancing, network routing

The CCM ensures that storage operations (e.g., EBS attachment) are orchestrated and respond to node failure.

Pros and Cons of Local vs. Cloud Storage

Local (On-Premise) Storage

Advantages:

• Control over hardware, data location, data protection
• Potentially cheaper with (long-term) stable utilization
• Performance (e.g., NVMe, ZFS) very good

Disadvantages:

• High maintenance effort (hardware, redundancy, network)
• Complex scaling—manual addition of capacity
• Implement security, backup & DR yourself
• More prone to errors in case of hardware failure

Cloud Storage

Advantages:

• High automation, scalability “on demand”
• Redundancy, security, and backup included
• Regional tiered storage options (e.g., SSD, HDD, archive)

Disadvantages:

• Ongoing costs (OPEX), potentially more expensive long-term
• Vendor lock-in (specific CSI drivers)
• Less control over physical security aspects

Challenges for On-Premise Storage

• Hardware failure: Disk/node failures must be compensated by replication or self-healing (e.g., Ceph or Longhorn).
• Network latency and bandwidth: Clusters across multiple racks/NICs require careful planning.
• Redundancy & backups: Multi-site or offsite backup requires own infrastructure (e.g., Velero).
• Complexity: Tools like Ceph require expert knowledge; ZFS also requires tuning.
• Performance tuning: ZFS/tuning vs. RADOS/Placement/CRUSH requires intimate knowledge.
• Fault tolerance: Fault tolerance in case of power failure, data corruption, bitrot—tools like ZFS (checksums) help but need to be set up.

Conclusion

Persistent storage in Kubernetes is no trivial task. The introduction of CSI has opened the path to modularity—both in the cloud and on-premise. Cloud-CSI offers convenience, while On-Premise CSI solutions like Longhorn, Ceph, and OpenEBS ZFS LocalPV provide control and performance.

The choice critically depends on use-case, expertise, budget, and operational focus:

• Longhorn: ideal for quickly deployable, uncomplicated bare-metal clusters
• Ceph/Rook: powerful and scalable—but with complexity costs
• OpenEBS ZFS LocalPV: for performance, data integrity, ZFS enthusiasts

Kubernetes demands a high level of strategic thinking in storage—from hardware to software, APIs, automation, cost control to compliance. In the end, it’s not just about technology, but also operational maturity.