Why Kubernetes Projects Become Complex Over Time – and How Companies Can Manage This Complexity

In the first part of this article, we explored why companies often systematically underestimate the effort surrounding Kubernetes. We saw that Kubernetes is primarily an orchestration system tasked with continuously enforcing the desired state of a container infrastructure. It also became clear that many tasks that enable stable platform operation are deliberately placed outside the core of Kubernetes.

This inevitably raises the next question:

Where does the actual effort arise in the daily operation of a Kubernetes platform?

The answer is not: in the cluster.

It arises everywhere technical decisions meet operational responsibility.

---

Observability: Why Functioning Applications Still Fail

Few terms have become as established in recent years as Observability. At the same time, few concepts are more frequently misunderstood.

Observability is often mistakenly used as a modern synonym for Monitoring. However, the two concepts differ fundamentally.

Those who do not understand this difference will sooner or later find that while a platform is excellently monitored, errors remain undetected for hours, or their actual cause cannot be traced.

Monitoring Answers Known Questions

Monitoring has been a fundamental part of professional IT operations for decades.

The idea is simple.

A system collects metrics.

CPU usage.

Memory.

Disk space.

Response times.

Network throughput.

Thresholds are then defined.

If CPU usage exceeds 90 percent, an alarm is triggered.

If an HTTP endpoint no longer responds, the on-call team receives a notification.

This approach works remarkably well—as long as the questions are already known.

This is precisely its greatest limitation.

Monitoring only answers questions that were asked beforehand.

No one configures an alarm for a problem whose existence is still unknown.

Distributed Systems Fundamentally Change Troubleshooting

This problem is particularly noticeable in Kubernetes environments.

In the past, a business application often consisted of a single server.

If an error occurred, an administrator logged into that specific system, analyzed log files, and usually identified the cause within a short time.

Cloud-native applications work entirely differently.

Today, a single user request often passes through a multitude of independent services.

An API gateway.

An authentication service.

Multiple microservices.

A message queue.

A database.

Perhaps additionally a search index or an external payment provider.

Each of these components has its own log files.

Its own metrics.

Its own error codes.

Its own timestamps.

Its own network connections.

The real challenge is no longer collecting data.

The challenge is recognizing the relationships between this data.

A Practical Example

Let’s assume the response time of a web application suddenly increases from 150 milliseconds to over three seconds.

The alarm goes off.

Monitoring reports increased latencies.

What now?

The CPU usage of all pods is below twenty percent.

There is enough memory available.

The network shows no abnormalities.

The Kubernetes cluster itself is working flawlessly.

This is where classic monitoring ends.

It only knows that a problem exists.

But not why.

The actual cause could look completely different.

A SQL query suddenly no longer uses an index.

A DNS resolver responds with delays.

A storage system takes unusually long for write operations.

An external authentication service delivers responses with high latency.

A library generates thousands of additional database queries due to an error.

A Kubernetes cluster can orchestrate all these scenarios without issue.

However, it recognizes none of them.

Not because Kubernetes is insufficient.

But because that is not its task.

Observability Answers Unknown Questions

This is where Observability comes in.

While Monitoring is based on predefined rules, Observability follows a fundamentally different approach.

The goal is not to generate as many alarms as possible.

The goal is to understand the internal state of a system based on its telemetry data.

The term originally comes from control theory.

A system is considered observable if its internal state can be reconstructed solely from its outputs.

Applied to software, this means:

An operator should be able to understand why a system behaves in a certain way without having to change the application or enable additional debug outputs.

This is why Observability encompasses much more than dashboards.

It combines three different data sources.

Metrics.

Logs.

And so-called Distributed Traces.

Why Logs Alone Are No Longer Enough

Logs are among the oldest tools in software development.

They document events.

A user logs in.

A database connection is opened.

A request fails.

A background process starts.

For a single application, this principle works excellently.

In a microservice architecture, however, a different picture quickly emerges.

Imagine an order process in an online shop.

A single order may generate entries in ten different applications.

In the API gateway.

In the authentication service.

In the shopping cart.

In the payment service.

In the warehouse management system.

In the invoicing service.

In the shipping service.

In the notification system.

Each application writes its own log file.

Without a common reference point, thousands of independent entries suddenly exist.

The actual story of the request is lost.

Distributed Tracing Makes Requests Visible

This is where the enormous value of Distributed Tracing lies.

Each incoming request receives a unique identity.

This so-called Trace ID accompanies the request throughout its entire lifecycle.

Each involved service adds information about its own processing.

This creates a complete timeline.

Which services were called?

How long did each processing step take?

Which database query caused the greatest delay?

Where did an error first occur?

Instead of manually evaluating hundreds of log files, the complete flow of a single user request can be understood within seconds.

Especially in Kubernetes environments, this transparency is invaluable.

Because Kubernetes continuously shifts workloads between nodes.

Pods are replaced.

Containers are restarted.

IP addresses change.

Static infrastructure practically no longer exists.

The actual constant is therefore not the server.

But the request itself.

OpenTelemetry Changes the Market

For a long time, Observability was closely tied to individual vendors.

Each provider used its own agents.

Own data formats.

Own APIs.

With the emergence of OpenTelemetry, this situation has fundamentally changed.

OpenTelemetry now defines an open standard for collecting metrics, logs, and traces.

Applications instrument themselves once against this standard.

The actual evaluation can then be done via different platforms.

Prometheus.

Grafana.

Jaeger.

Tempo.

Elastic.

Datadog.

Dynatrace.

New Relic.

This open approach fits perfectly with the philosophy of Kubernetes.

Not a single tool is the focus.

The key is standardized interfaces.

Why Observability Becomes a Platform Task

By this point, a pattern emerges that we have already encountered in previous chapters.

Kubernetes does not provide a complete solution.

It creates the conditions for specialized solutions to be integrated.

A powerful Observability platform is therefore never created by installing a single tool.

It begins with the architecture of an application.

Which metrics are collected?

Which events are logged?

How are logs structured?

Which trace contexts are transferred between services?

Which information can be stored at all for data protection reasons?

How long are telemetry data retained?

Who has access to production data?

How are alarm rules prioritized?

Who responds outside business hours?

These questions go far beyond installing a Prometheus Operator or a Grafana instance.

They affect architecture, operations, compliance, and organizational processes equally.

Good Observability Does Not Reduce Errors – It Reduces Their Impact

At this point, a common misunderstanding often arises.

Observability does not prevent outages.

It does not speed up applications.

It neither reduces memory usage nor automatically improves availability.

Its actual benefit lies elsewhere.

It shortens the time between the occurrence of a problem and its actual resolution.

In business, the metrics MTTD (Mean Time to Detect) and MTTR (Mean Time to Recover) are often mentioned.

The faster a problem is detected and the faster its cause can be clearly identified, the less economic damage, reputational loss, and operational interruptions occur.

Especially for business-critical applications, it is not just the probability of an error that determines the quality of a platform.

What is crucial is how quickly a company can become operational again after an unavoidable error.

And that is precisely why Observability is one of the central disciplines of professional platform operations today.

Not because modern applications fail more often.

But because their internal complexity has increased so much that classic monitoring methods can no longer reliably describe the actual state of distributed systems.

In the next chapter, we will look at an area whose importance has grown at least as much in recent years: the security of cloud-native platforms—and why Kubernetes provides numerous security mechanisms, but this does not automatically create a security concept.

Security in Kubernetes: Why a Secure Cluster Is Not Yet a Secure Platform

Few aspects are as controversially discussed when introducing Kubernetes as security.

Some view Kubernetes as particularly complex and therefore inevitably difficult to secure. Others argue the opposite, pointing to the numerous security mechanisms Kubernetes already provides by default. Both perspectives fall short.

Kubernetes is neither inherently secure nor insecure.

It offers a comprehensive toolbox with which very secure platforms can be realized. At the same time, the same tools can be used to create…