Operator Deployment

How the Polycrate API tracks which clusters run the Polycrate Operator in which version and how rollouts of new operator versions are controlled via K8sApp, operator agents, SystemConfig keys and the Operator Rollout Dashboard.

Operator Deployment

The Polycrate API models which clusters have the Polycrate Operator installed, in which version, and how the rollout of new operator versions is controlled. The feature combines several existing building blocks — there is deliberately no dedicated OperatorDeployment model.

The interplay described here:

• K8sCluster + K8sApp — each cluster optionally has an app polycrate-operator whose installed_version describes the actual state on the cluster.
• Agent (kind=operator) — the operator registers as an agent with the API and sends heartbeats that keep it alive.
• SystemConfig — global keys such as OPERATOR_BLOCK_VERSION (target version) and OPERATOR_ROLLOUT_PERCENTAGE (canary percentage) control the desired state.
• Operator Rollout Dashboard — the UI that presents these sources in aggregated form.

Model view

┌──────────────┐  1..n  ┌──────────────────────┐
│  K8sCluster  │────────▶│ K8sApp               │
└──────┬───────┘         │  name=polycrate-op…  │
       │ optional FK     │  installed_version   │
       ▼                 └──────────────────────┘
┌──────────────┐
│ Agent        │ Operator runs on the cluster, pings API
│ kind=operator│
└──────────────┘

┌───────────────────────────┐
│ SystemConfig (global keys │
│  OPERATOR_BLOCK_VERSION   │
│  OPERATOR_ROLLOUT_…       │
└───────────────────────────┘

K8sApp = the concrete operator per cluster

The Polycrate Operator is a Polycrate block like any other. On every cluster where it runs, a K8sApp entry with name=polycrate-operator exists:

The comparison between installed_version and desired_version is the central information for the rollout dashboard.

Agent (kind=operator) = the operator at the other end

Every operator registers as an Agent with kind=operator at the API. From then on it sends:

• Heartbeats (updates last_seen)
• Probe results for endpoints assigned to it
• K8sApp status (incl. its own installed_version)

If the heartbeat stops, the agent becomes CRITICAL and creates a Downtime. This makes "operator on cluster X has been gone for 15 minutes" visible without additional monitoring configuration.

SystemConfig keys = desired state

Two keys control the platform-wide rollout:

Platform admins set these values in the UI under System → SystemConfig. A change to OPERATOR_BLOCK_VERSION affects all clusters by default — staged through OPERATOR_ROLLOUT_PERCENTAGE.

Rollout gating

The rollout of new versions follows a simple batching rule:

1. For each cluster, the API compares installed_version against desired_version.
2. If clusters diverge (new version available), only one batch is updated per rollout run — defined by OPERATOR_ROLLOUT_PERCENTAGE.
3. Clusters with an active Maintenance are updated preferentially (legitimate maintenance slot).
4. Clusters in state CRITICAL are deferred until their state settles.

An admin can force or pause an update per cluster on the dashboard.

Operator Rollout Dashboard

Path: System → Operator Rollout

The dashboard aggregates the data sources mentioned above and shows per cluster row:

Typical blockers:

• No agent token — the cluster has not yet set up the API credentials.
• OperatorConfig missing — no OperatorConfig CR pointing to the API exists in the cluster yet.
• Canary gate — this cluster is not yet part of the current rollout batch.

CRD synchronization

On release finalize, the Polycrate CLI exports the operator's CRDs and provides them as part of the block (see CLI release 0.34.0). The operator installs these CRDs in the cluster as soon as its new version is reconciled.

Consequence: An upgrade of the operator version can introduce new custom resources — this is an intended part of the rollout. Via the OpenAPI export, the API knows the label constants, so dashboards and filters can handle the new fields consistently (see Labels & Conventions).

Token provisioning

The operator needs an API token to register as an agent. The sequence:

1. Create an API token (UI → User → API Tokens) — scoped to the cluster's workspace.
2. Create a Kubernetes Secret — secret with key token in the operator's namespace.
3. OperatorConfig in the cluster → set credentials_ref.secret_name and token_key.
4. Restart the operator or wait for the next reconcile.

From then on the operator sends heartbeats and appears in the rollout dashboard.

Multi-version fleet

The dashboard also visualizes how many clusters run which version — e.g. "30 × 0.34.0, 12 × 0.35.0, 2 × 0.36.0". This is useful to:

• make compatibility decisions ("can I remove CRD field X while 30 clusters are still on the old version?"),
• adjust rollout pace,
• specifically address clusters with "stuck" versions.

API endpoints

• GET /api/v1/operator-rollout/ — aggregated dashboard data.
• GET /api/v1/k8sapps/?name=polycrate-operator — raw data per cluster.
• GET /api/v1/agents/?kind=operator — all operator agents.
• POST /api/v1/operator-rollout/{cluster_id}/force-update/ — force an update.
• POST /api/v1/operator-rollout/{cluster_id}/pause/ — pause the rollout for this cluster.

Access control

The operator dashboard is platform-admin-only, because the values are global and small changes have large effects. Individual cluster rows can be made accessible via RBAC so that customers can see the status of their own cluster without getting into the system settings.

Related topics

• Polycrate Operator — concept, installation, OperatorConfig.
• Endpoint Monitoring — how the operator runs probes as an agent.
• Action Runs — how the install / status actions of the operator block are executed.
• Labels & Conventions — OpenAPI-based label constants for rollout filters.
• Management Commands — CLI commands for rollout control.