Polycrate API Metrics

The three metric layers of the Polycrate API: the Prometheus export at /api/v1/metrics, system metrics for the dashboard and per-object metric functions, all backed by VictoriaMetrics.

Polycrate API Metrics

The Polycrate API exposes metrics in three layers, unified by a shared contract pattern (MetricsMixin) and all queried against VictoriaMetrics.

Layer Endpoint Format Consumer
Prometheus export GET /api/v1/metrics text/plain VMAgent → VictoriaMetrics
System metrics GET /api/v1/system/metrics/ JSON Dashboard widget (system overview)
Object metrics GET /api/v1/{model}/{id}/metrics/ JSON Metric tabs on every detail page

System and object metrics use the same {data, meta, chart_spec} response shape and are rendered in the UI by the unified metricChart Alpine.js component.

Prometheus Export

Overview

At GET /api/v1/metrics the Polycrate API provides Prometheus-compatible metrics — without authentication, so that VMAgent/Prometheus can scrape without a token.

  • Endpoint: /api/v1/metrics
  • Caching: All metrics are precomputed every 60 seconds by a Celery task and cached in Redis with a TTL of 120 s. Scrapes under load respond in < 100 ms from the cache; a cache miss regenerates live.
  • Base labels: Every series carries polycrate_io_metrics_source="api" and polycrate_io_system_name="<instance>", so that API metrics can be clearly separated from agent, operator or Kubernetes metrics in VictoriaMetrics.

SLO/SLA

Metric Description
polycrate_io_api_breaches_active_count Active SLO/SLA breaches by type
polycrate_io_api_overall_availability_percent Weighted availability (SLO/SLA)
polycrate_io_api_error_budget_consumed_avg_percent Avg. consumed error budget

Endpoint Status

Metric Description
polycrate_io_api_endpoints_by_state Endpoints by state (OK, WARNING, CRITICAL)

Operations (Downtimes, Maintenance, Actions)

Metric Description
polycrate_io_api_action_runs_count Action runs by status
polycrate_io_api_downtimes_active_count Aggregated number of active downtimes per organization
polycrate_io_api_downtime_is_active Binary value 1 per active downtime — produces time series from which the downtime timeline is built. Absence means "over".
polycrate_io_api_downtime_affected_objects Number of currently affected objects per downtime (time series via VMAgent)
polycrate_io_api_maintenances_active_count Aggregated number of running maintenances per org/workspace
polycrate_io_api_maintenance_is_active Binary value 1 per running maintenance

→ Downtime & Timeline · Maintenances · Maintenance Windows

Resources (Counts)

Metric Description
polycrate_io_api_resource_count Resources per type (K8sCluster, K8sApp, LoadbalancerInstance, S3Bucket, DNSZone, Domain) per organization/workspace
polycrate_io_api_workspaces_total Total number of workspaces
polycrate_io_api_organizations_total Total number of organizations
polycrate_io_api_workspaces_count Workspaces per organization

Resource Details (Org Level, Persisted)

Values from model fields that are updated roughly every 5 minutes by reconciliation tasks — cheap to query, without the Prometheus export itself having to call S3/LB backends.

Metric Description
polycrate_io_api_s3_bucket_count Number of S3 buckets per organization
polycrate_io_api_s3_object_count Sum of S3 objects per organization
polycrate_io_api_s3_storage_bytes Used S3 storage per organization (bytes)
polycrate_io_api_loadbalancer_instance_count Number of LoadbalancerInstances per organization
polycrate_io_api_loadbalancer_traffic_bytes_per_second Sum of the 1h bandwidth averages per organization
polycrate_io_api_registry_storage_bytes Used artifact registry storage per organization

S3 Cluster Backend

Per S3Cluster, backend figures from the reconciliation against Rook-Ceph or MinIO are mirrored (for details see S3 Buckets & Object Storage).

Metric Backend Description
polycrate_io_api_s3cluster_ceph_osd_total_bytes rook-ceph Total OSD capacity
polycrate_io_api_s3cluster_radosgw_logical_used_bytes rook-ceph Logically used storage (radosgw)
polycrate_io_api_s3cluster_minio_capacity_usable_bytes minio Usable capacity
polycrate_io_api_s3cluster_minio_usage_bytes minio Used storage
polycrate_io_api_s3cluster_managed_buckets_usage_bytes both Sum across Polycrate-managed buckets
polycrate_io_api_s3cluster_managed_buckets_object_count both Sum of objects in Polycrate-managed buckets

Projects & Notes

Metric Description
polycrate_io_api_todos_count Todo notes by status (open, resolved) per org/workspace

→ Projects · Notes

Endpoint Monitoring (Agents & Coverage)

Metric Description
polycrate_io_api_agent_capacity Max. endpoint capacity per agent
polycrate_io_api_agent_assigned_endpoints Assigned endpoints per agent
polycrate_io_api_agent_utilization_percent Agent utilization
polycrate_io_api_agent_checks_per_minute Checks per minute per agent
polycrate_io_api_endpoint_coverage_total Endpoints by coverage (uncovered, single, multi)
polycrate_io_api_endpoint_coverage_rate_percent Share of endpoints with ≥ 1 agent
polycrate_io_api_endpoint_multi_coverage_rate_percent Share of endpoints with ≥ 2 agents
polycrate_io_api_endpoint_check_latency_avg_ms Avg. check latency
polycrate_io_api_endpoint_checks_total Check counter by status
polycrate_io_api_endpoint_check_success_rate_percent Success rate of the checks
polycrate_io_api_endpoint_monitoring_summary Summary (for alerting & dashboards)

→ Endpoint Monitoring

PoP-to-PoP Latency

Metric Description
polycrate_io_api_pop_latency_avg_ms Avg. latency between PoP pairs
polycrate_io_api_pop_latency_min_ms Minimum
polycrate_io_api_pop_latency_max_ms Maximum
polycrate_io_api_pop_check_count Number of checks between PoP pairs
polycrate_io_api_pop_assignment_count Number of active assignments per PoP pair

Activity Metrics (new since 0.15.0)

Time series over platform events — the basis for the system charts on the main dashboard. All values as 5-minute windows; increase(...[5m]) is suitable for trends.

Metric Description
polycrate_io_api_recent_activities_count All activity events (audit trail, lifecycle, deployments…)
polycrate_io_api_recent_state_changes_count Only kind="state_changed"
polycrate_io_api_recent_deployments_count ReplicaSet/StatefulSet deployments (from K8s discovery)
polycrate_io_api_recent_block_action_runs_count Block action runs (Polycrate CLI → API)
polycrate_io_api_recent_ssh_sessions_count New SSH sessions via bastions

→ Audit & Compliance

System Metrics (Dashboard Overview)

At GET /api/v1/system/metrics/ the API provides platform-wide time series in JSON format. This endpoint is used by the widgets on the main dashboard ("System Overview" tab), which show e.g. "Activities in the last 7 days".

Every system metric is a PromQL range query against the corresponding gauge from the Prometheus export:

System Metric Source Gauge Description
activities polycrate_io_api_recent_activities_count All activity events
state_changes polycrate_io_api_recent_state_changes_count State changes
deployments polycrate_io_api_recent_deployments_count Deployments
block_action_runs polycrate_io_api_recent_block_action_runs_count Block action runs
ssh_sessions polycrate_io_api_recent_ssh_sessions_count SSH sessions

The endpoint accepts the same parameters as object metrics:

  • metric=<name> — without parameters: list of available metrics (discovery)
  • time_from=<rfc3339>, time_to=<rfc3339> — time window

Object Metrics (Metric Functions)

On the detail page of every ManagedObject, Polycrate provides metric tabs that render time-series charts directly in the UI without a Grafana setup. The data comes from VictoriaMetrics; it is queried via the generic endpoint:

GET /api/v1/{model}/{id}/metrics/?metric=<name>&time_from=...&time_to=...

What Is a Metric Function?

A metric function is a query declared on the object type that:

  1. identifies the object in VictoriaMetrics via its standard labels (polycrate_cluster, polycrate_customer, polycrate_workspace, possibly polycrate_namespace),
  2. runs PromQL against metrics from agents/kube-state-metrics/kubelet/blackbox-exporter,
  3. renders the result as a chart (line, area, bar) with a unit (bytes, bytes/s, %, cores, count).

Available Functions per Object Type

Object Type Function(s) Description
Host cpu_usage, memory_usage CPU load & memory usage from node-exporter
K8sCluster cpu_usage, memory_usage Aggregated cluster resources (kube-state-metrics)
K8sApp pod_count, cpu_usage, memory_usage Metrics per app workload
K8sVolume volume_usage Used vs. total storage of the PVC (kubelet)
S3Cluster backend_total_capacity, backend_logical_used Backend capacity & logical usage (Ceph/MinIO)
S3Bucket bucket_usage, object_count Storage fill level & object count per bucket
LoadbalancerInstance bandwidth Inbound/outbound traffic per LB instance
Endpoint status_blocks, latency, uptime Blackbox check values (heatmap, latency curve, uptime percentage)
Agent checks_per_minute, utilization Workload of a monitoring agent

Time Range Control

Every widget has:

  • Time range dropdown (1h, 6h, 24h, 7d, 30d, custom)
  • "Reset to Initial Range" (returns the graph to the default time range)
  • Custom range from downtime events — on the downtime timeline, "Show in Metric" leads directly into the dashboard widget with a matching time window
  • Auto-step: automatic sample resolution depending on the time range (1 h → 15 s, 7 d → 5 min, 30 d → 30 min)

Usage & Integration

Scraping Configuration

# VictoriaMetrics / Prometheus scrape_config
scrape_configs:
  - job_name: 'polycrate-api'
    static_configs:
      - targets: ['api.polycrate.io']
    metrics_path: '/api/v1/metrics'

Since the API itself caches all values, a scrape interval of 30 – 60 s is recommended.

Typical Use Cases

  • Capacity planning — monitor resource counts, S3 storage and workspace growth
  • SLO dashboards — visualize availability and error budget in Grafana, together with agent metrics (same polycrate_* label schema)
  • Agent monitoring — check coverage, utilization and checks/min of the monitoring agents
  • Incident response — jump from the downtime timeline directly into the matching metric tab
  • Customer reporting — screenshots from the UI show resource trends without Grafana access

Interplay with Grafana

Metric functions and system widgets cover the operational quick look. For deeper dashboards (multi-object overlays, complex aggregations, alerting with VMAlert) Grafana remains the tool of choice — the shared polycrate_* labels ensure that the same queries work in the UI and in Grafana.

See Also