From Ticket Infrastructure to On-Demand AI: How ayedo Built a Kubernetes-Based Data Engineering Platform for an Industrial Corporation

Data-driven innovation rarely fails due to a lack of ideas. It fails due to infrastructure.

Many industrial companies invest in Data Science, AI models, and Advanced Analytics—only to find that their platform doesn’t scale. GPU resources are scarce, development environments are rigid, and ETL processes are hard to scale. Every new use case becomes an infrastructure project.

In this post, we demonstrate through an anonymized project how ayedo built a Kubernetes-based data engineering platform for a global industrial corporation—for ETL pipelines, event streaming, analytical databases, and GPU-supported AI workloads.

The client remains anonymous. The approach is reproducible—especially for companies looking to combine on-prem stability with cloud flexibility.

Initial Situation: Functional Platform – But Structurally Hindered

The client is an international manufacturer of industrial raw materials with around 10,000 employees. About 30 specialists work in the field of Data Engineering and Advanced Analytics. The goal is to optimize production processes based on data, reduce energy consumption, and use AI models for quality control.

Technically, there was already an in-house orchestration platform based on HashiCorp Nomad and Docker. It worked—but it wasn’t designed to scale dynamically or give data teams autonomy.

Resources had to be requested through a central infrastructure team. GPU capacities were scarce, and on-prem procurement took months. Individual development environments—such as customized Jupyter setups or special R stacks—could only be realized with significant coordination effort.

The result was not only technical friction but also organizational. Data engineers waited for infrastructure instead of training models. ETL pipelines were delayed in going live. Innovation cycles were extended—not for technical reasons, but due to a lack of elasticity.

Especially with AI workloads, time is a critical factor. When training jobs wait weeks for GPU resources, data-driven optimization quickly becomes a strategic bottleneck.

The Core of the Problem: Infrastructure as Gatekeeper

The actual problem wasn’t Nomad or Docker. It was the underlying operating model.

Infrastructure was a centrally controlled bottleneck. Resources were manually planned. Development environments weren’t reproducibly containerized but partially individually configured. GPU usage was tied to fixed hardware.

This led to three structural weaknesses:

First, there was a lack of true self-service capability for data teams.
Second, scaling was linearly tied to physical resources.
Third, reproducibility across projects wasn’t systematically ensured.

To sustainably operate AI, ETL, and BI applications, a platform is needed that decouples compute, storage, and orchestration and makes them declaratively controllable.

ayedo’s Approach: Kubernetes as the Data Engineering Backbone

Together with the client, we migrated the entire data orchestration to Kubernetes—not as a mere infrastructure upgrade, but as a foundation for a modern, hybrid data platform.

The goal was to create an architecture that meets the following requirements:

• Self-service for data engineers
• On-demand scaling of CPU and GPU workloads
• Reproducible development environments
• Integration into existing corporate security policies
• Combination of on-prem stability and cloud elasticity

Kubernetes is particularly well-suited for these requirements because it standardizes workloads, dynamically allocates resources, and seamlessly integrates into hybrid cloud scenarios.

Coder: Reproducible Development Environments on Demand

A central element was the introduction of Coder on Kubernetes.

Data engineers can now start fully containerized development environments on-demand—via browser, RDP, or VS Code Extension. Each environment is versioned, reproducible, and isolated.

This solves several problems:

Development environments are no longer tied to individual workstations.
Configurations are defined as code.
Teams can share and standardize setups.

Instead of “It works on my machine,” there is now a consistent, containerized workspace.

Airflow, Kafka, and Analytical Databases: Scalable Data Pipelines

For the orchestration of ETL processes, Apache Airflow was established on Kubernetes. Airflow jobs run containerized and can be horizontally scaled. Compute-intensive transformations can be dynamically distributed to additional workers.

Apache Kafka serves as the event streaming backbone for production data from plants and sensors. Data streams are ingested almost in real-time and distributed to downstream systems.

For analytical workloads, TimescaleDB and ClickHouse were integrated—optimized for time-series and high-volume analyses. Both benefit from Kubernetes resource management and scalable storage.

CEPH as S3-Compatible Storage Backend

The platform requires not only compute but also scalable, highly available storage for training data, models, and artifacts.

CEPH was implemented as an S3-compatible backend. It combines high availability with horizontal scalability and allows the separation of performance and capacity requirements.

This enables efficient storage and processing of large data volumes—without proprietary dependencies.

GPU on Demand: Hybrid Cloud Without Vendor Lock-In

One of the biggest bottlenecks was GPU availability.

Through an integrated cloud-layer architecture, GPU resources can now be dynamically provisioned from European cloud providers. Training and simulation jobs are outsourced to the cloud as needed, without having to adjust the on-prem architecture.

The key here is: The workloads remain Kubernetes-native. There is no separate “cloud version.” Only the location of the cluster changes.

This creates true elasticity without dependency on a single hyperscaler.

Identity, Security, and Compliance

The entire platform is integrated into Azure Entra ID. Single sign-on and role-based access control ensure that the data platform remains compliant with the company’s security policies.

Harbor serves as a dedicated container registry with long-term artifact persistence. Models, ETL jobs, and container images are versioned and traceable.

This ensures reproducibility not only technically but also regulatorily.

Result: Data Engineering Without Infrastructure Bottleneck

After migrating to Kubernetes, the working method of the data engineering team has fundamentally changed.

GPU compute is available on-demand. Development environments can be started in minutes. New projects no longer require weeks of coordination with infrastructure teams.

ETL pipelines are containerized, orchestrated, and horizontally scalable. Training jobs can be flexibly shifted between on-prem and cloud.

Above all, the platform is reproducible. Models, pipelines, and environments are versioned. New teams can start immediately without historical legacy or setup issues.

What was previously an organizational bottleneck is now a strategic asset.

Strategic Value: Data Platform as an Innovation Engine

The new platform is more than a technical upgrade. It enables sustainable, economically scalable use of AI and big data workloads.

Innovations can be developed iteratively and put into production—without waiting for infrastructure. GPU bottlenecks no longer block roadmaps. New analysis projects can run in parallel without resource collisions.

Kubernetes acts as a universal orchestration layer for compute, storage, and network—independent of the physical location.

Why Kubernetes is the Right Choice for Data Engineering

Complex ETL pipelines, BI applications, and AI workloads are inherently resource-intensive and dynamic. Rigid infrastructure models are not designed for this.

Kubernetes enables:

• Elastic resource utilization
• GPU integration as a native resource
• Reproducible container workloads
• Hybrid cloud capability
• Clear separation of platform and application

Especially in corporate structures, this creates a balance between governance and speed.

Call to Action

If your data engineering team is being slowed down by infrastructure bottlenecks, GPU resources are scarce, or ETL workloads only scale with ticket processes, it’s time for a new platform model.

ayedo supports the development of Kubernetes-based data engineering platforms—with hybrid GPU usage, scalable storage, orchestrated ETL pipelines, and reproducible development environments.

This way, data is not just collected but strategically utilized.