Endpoint Monitoring

How the Polycrate API continuously probes endpoints via agents: the Endpoint, Agent and assignment models, state derivation, SLO/SLA, operator discovery, TLS monitoring, multi-region checks and API endpoints.

Endpoint Monitoring

Endpoint monitoring in the Polycrate API continuously checks whether external and internal services are reachable. It is the core of state detection for ManagedObjects: from a collection of regular probes against an endpoint, the aggregated state of an object (OK / WARNING / CRITICAL) is derived — and from that in turn Downtimes and notifications.

Polycrate separates what is checked (Endpoint) from who checks (Agent).

Model Overview

┌──────────────────┐         checks         ┌────────────┐
│      Agent       │────────────────────────▶│  Endpoint  │
│  (CLI / Operator)│    via Assignment       │            │
└──────────────────┘                         └─────┬──────┘
                                                   │ generic FK
                                                   ▼
                                          ┌──────────────────┐
                                          │   ManagedObject  │
                                          │ (Host, LB, PoP…) │
                                          └──────────────────┘

Endpoint — What Is Checked

An endpoint describes a single probe. Core fields:

Kinds at a glance:

• icmp — classic ping. Good for host reachability, requires CAP_NET_RAW on the executing agent.
• icmp — ping including latency values for reachability and edge comparisons (CAP_NET_RAW).
• http — HTTP(S) with path, method, expected status; the dashboard shows remote port and TLS status.
• tcp / udp — connect/probe checks (DB, broker, DNS, VPN, …).
• TLS inspection — expiry, issuer and chain from agent check results.

Agent Check Result History (API) provides the last N results per check for trends. Check result retention is typically 1 day (batched cleanup).

Agent — Who Checks

An agent is the executing actor that actually probes endpoints. It authenticates to the API with a token and pulls its work:

Every agent sends heartbeats and probe results to the API. Fields such as last_seen, health_status, effective_capacity and monitored_endpoints_count make its state transparent.

EndpointAgentAssignment — Who Checks What

The M2M assignment between endpoint and agent is explicit: an endpoint can be checked by several agents in parallel (for multi-region monitoring), and an agent typically checks many endpoints.

The assignment can be made

• manually (admin selects suitable agents per endpoint),
• automatically via scheduler logic that uses free agent capacity.

From Probe to Object State

A single probe has a result (up/down, latency, status code, TLS details). From this, per endpoint, the following is rolled up:

1. Hot path — status diffs from agent check results update the endpoint state/downtime immediately (not only at the next reconcile tick).
2. Reconcile — stale/recovery logic at a longer interval (typically 300s).
3. Metrics — VictoriaMetrics for uptime/latency tabs (status_blocks, latency, uptime).
4. ManagedObject state — attached endpoints feed into Conditions / object state: * OK — active probes successful * WARNING — partial failures / latency * CRITICAL — hard thresholds → Downtime + notifications

Agent health: unhealthy only after prolonged silence (SLA typically 300s).

Edge PoP Latencies

The PoP detail view and associated tabs show ICMP reachability from edge workspaces (latency per PoP). S3/region API endpoints can also be monitored. Workspace-less endpoints are delivered to the operator (including ICMP seed).

SLO / SLA

Every ManagedObject that carries endpoints can define SLO/SLA targets:

• target_availability_percent — e.g. 99.9 %
• actual_availability_percent — calculated from probe history
• error_budget_* — derived error budget

The values appear on the object's detail view and feed into reports.

Source Classification (API Side)

When endpoints come from multiple sources, the API classifies them so that ownership remains clear:

From the API's perspective, all three sources are equivalent: the same probe path (HTTP/TLS, agent assignment, conditions, downtime). Who created the endpoint does not change how it is checked.

Operator Discovery (Ingress and Gateway API)

The Polycrate Operator creates Endpoint CRs for public hostnames and syncs them to the API (source=discovery, typically kind=http). Annotations on the source control path, interval, timeout, expected status, TLS and port — see Operator Endpoint Discovery.

Same hostname in the same namespace: HTTPRoute wins the Endpoint CR (hostToName). Ingress does not overwrite it. Filters: ingress_classes or gateway_classes (empty = all). GRPCRoute/TLSRoute are not watched.

Mode annotation (default): without polycrate_endpoint_monitor=true no endpoint. Mode auto: all except polycrate_endpoint_monitor=false.

polycrate_k8sapp_name claims the endpoint for a K8sApp (discovery-source=ingress or httproute). Without a name: monitoring yes, no app downtime. polycrate_k8sapp_byoa=true sets API K8sApp.byoa only on these sourced CRs.

The BYOA block writes the same contract on Ingress and HTTPRoute.

TLS Monitoring

HTTP endpoints with HTTPS automatically receive TLS monitoring. The API generates alerts on:

• Certificate expired / not yet valid
• Certificate chain invalid
• Hostname mismatch
• TLS expiry < 30 days (threshold configurable per endpoint)

Per endpoint, ignore_tls_errors=true can optionally be set — required for internal services with self-signed certificates.

Metric Functions

Every endpoint provides the standard metric functions (see Metrics):

In addition, the agent exports Prometheus metrics such as polycrate_io_endpoint_up, polycrate_io_endpoint_response_time_ms, polycrate_io_endpoint_certificate_expiry_timestamp.

Multi-Region and Multi-Agent Monitoring

An endpoint can be assigned to several agents that check it in parallel from different regions. The result:

• Probe data per agent — the table shows per probe which agent responded.
• Matrix view — PoP-to-PoP latency as a metric (see Metrics → PoP-to-PoP Latency).
• Aggregated is_up value — "endpoint is online" means "at least n agents report OK" (threshold configurable per endpoint).

For platforms with distributed sites, it is sufficient to run one operator or CLI agent per site; the assignment to the endpoints is then done centrally.

Stale Checks and Missing Monitors

No evidence is not down. Reachability (ENDPOINT_UNREACHABLE) is only set after SYSTEM_DEGRADATION_THRESHOLD_SECONDS (default 300) and only from fresh checks (UP% below 50%).

Downtimes follow ownership: claimed including BYOA yes; unclaimed endpoint (Ingress or HTTPRoute without polycrate_k8sapp_name) no. Edge PoPs without an active reachability endpoint do not generate artificial unreachability.

Agent Lifecycle

An agent is itself a ManagedObject and has metric functions such as health_status and last_seen. If the heartbeat stops, the agent is set to state=CRITICAL and generates — like any other ManagedObject — a downtime. This way the platform itself detects when a monitoring site fails.

API Endpoints

• GET /api/v1/endpoints/ — list endpoints, filter by kind, content_type (i.e. which object is checked), is_up.
• POST /api/v1/endpoints/ — create an endpoint (including agent assignment).
• GET /api/v1/agents/ — list of registered agents.
• GET /api/v1/endpoint/{id}/metrics/?function=status_blocks&range=24h — metric per endpoint.

Typical Workflow

1. Create host / LoadBalancer / service — the object whose state you want to know.
2. Attach endpoint(s) — one or more probes per object.
3. Assign agent(s) — either the local operator, external CLI agents or both.
4. Define SLO — optional, for availability reports and error budgets.
5. Observability — metric tabs, downtimes, notifications run automatically.

Related Topics

• Polycrate Operator — discovery from Ingress and HTTPRoute, annotations, operator as agent.
• Downtime & Timeline — consequence of a CRITICAL state.
• PoPs & Providers — PoPs can carry their own endpoints (Pop.endpoint_monitor).
• Metrics — Prometheus metrics of probes and agents.
• User Management & RBAC — who may create/assign endpoints.