External Secrets Operator: The Reference Architecture for Hybrid Secrets Management

TL;DR

Secrets (API keys, database passwords) do not belong in Git code, but their runtime provisioning is often complex. Integrating AWS Secrets Manager directly into your application (via SDK) creates a hard vendor lock-in in the source code. The External Secrets Operator (ESO) solves this dilemma. It acts as a bridge, synchronizing secrets from external sources (AWS, Azure, Vault) and providing them as native Kubernetes Secrets. The result: The application remains cloud-agnostic and clean.

1. The Architectural Principle: Decoupling Code and Config

In Cloud-Native development, the “Twelve-Factor App” principle applies: Configuration (and thus secrets) should be strictly separated from the code and injected via environment variables.

Developers often violate this principle by embedding cloud SDKs (e.g., boto3 for AWS) directly into the application to retrieve secrets at runtime.

• The Problem: The code “knows” it is running on AWS. It is no longer portable.
• The Solution (ESO): The External Secrets Operator runs in the cluster and takes over the work. It polls the external API (e.g., AWS Secrets Manager), retrieves the secret, and stores it as a standard Kubernetes Secret. The application simply reads an environment variable—it does not know where the secret originally came from.

2. Core Feature: Multi-Provider Support and Templating

ESO is the “Swiss Army knife” for secrets. It supports almost every relevant backend provider: AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, Google Secret Manager, GitLab CI Variables, and many more.

The templating engine is particularly powerful:

Often, an external provider delivers a JSON blob (e.g., {“username”: “admin”, “password”: “xyz”}). However, the application expects two separate environment variables. ESO can parse, reformat, and bring the data into the exact format the application needs during synchronization. This eliminates the need for complex wrapper scripts during the Container startup process.

3. Security and RBAC

A common objection is: “Are Kubernetes Secrets secure?”

Yes, in modern clusters (like in the ayedo Stack), they are.

• Encryption at Rest: The secrets are stored encrypted in the etcd database.
• Granular RBAC: Thanks to Kubernetes RBAC, it can be precisely controlled which pod can read which secret.
• Minimal Permissions: Only the ESO controller needs access to the external AWS API. The hundreds of application pods no longer need any IAM permissions to call AWS Secrets Manager. This drastically reduces the attack surface.

4. Operating Models Compared: AWS SDK / CSI Driver vs. ayedo Managed ESO

Here it is decided whether your application developers need to be cloud experts or can focus on code.

Scenario A: Native Integration or CSI Driver (The Code Lock-in)

Many teams use the AWS Secrets and Configuration Provider (ASCP) or directly integrate the AWS SDK.

• Code Dependency: The developer must build AWS logic into the microservice. Switching to Azure means: rewriting code.
• Filesystem Constraint (CSI): The AWS CSI driver mounts secrets as files. However, many applications expect environment variables. This requires workarounds.
• No Cross-Cloud Logic: If you have part of the apps on-premise, the AWS method often does not work there or only through complex IAM user constructs.

Scenario B: External Secrets Operator with Managed Kubernetes from ayedo

In the ayedo App Catalog, ESO is the standard component for secrets management.

• Write Once, Run Anywhere: The application simply expects a secret named db-pass. Whether this secret comes from AWS, Azure, or a local vault in the background is defined solely by the platform engineer via CRD (ExternalSecret). The app code remains 100% portable.
• Central Governance: You can precisely control which namespaces are allowed to access which external stores (ClusterSecretStore vs. SecretStore).
• Automatic Rotation: If the secret changes in AWS, ESO automatically synchronizes it (configurable) into the cluster. A restart of the pods (via Reloader) ensures that the change goes live.

Technical Comparison of Operating Models

FAQ: ESO & Security Strategy

Are Kubernetes Secrets not insecure (Base64)?

This is an old misunderstanding. Base64 is just the encoding for display. What matters is how they are stored. In professional clusters (like with ayedo), Encryption at Rest is enabled. This means even if someone has physical access to the server’s hard drive, they cannot read the secrets. Additionally, RBAC prevents unauthorized users in the cluster from seeing the secrets.

What happens if the AWS API fails?

This is a major advantage of ESO over direct SDK integration. Since ESO synchronizes the secrets into Kubernetes secrets, they are still available even if the connection to AWS is temporarily interrupted. The application continues to start because it uses the local copy (the cache) in the cluster. With direct integration, the app would crash.

Can ESO also write secrets (Push)?

Yes, ESO has a feature called PushSecret. This allows you to create a secret in the Kubernetes cluster (e.g., generated by another tool) and write it back to the AWS Secrets Manager or Vault. This is useful for centrally securing certificates or generated passwords.

How does password rotation work?

ESO polls the external backend at a defined interval (e.g., every 10 minutes). If you rotate the database password in AWS, ESO updates the Kubernetes secret. In combination with a tool like Reloader (also often in the ayedo Stack), all pods can then be automatically restarted to load the new password—without any downtime.

Conclusion

Secrets are critical, but their management should not slow down development. Building AWS-specific logic into your applications ties your code to the platform. The External Secrets Operator breaks this dependency. It allows you to use powerful cloud backends like AWS Secrets Manager without sacrificing application portability. With the ayedo Managed Stack, teams receive a robust, provider-independent pipeline for secrets that combines security and developer experience.