DataSources

DataSources pull external feeds (RSS, REST APIs, Polycrate Hub, OTC status, webhooks) into the Polycrate API and turn them into Notes, maintenances or block templates.

DataSources

DataSources are external feeds that pull content into the Polycrate API. They connect external systems (RSS feeds, REST APIs, Polycrate Hub) with the platform and generate objects in the system from them — typically Notes or block templates.

Each entry is evaluated periodically by a Celery task and creates new objects on first occurrence. Items that have already been imported are recognized idempotently and are not created again.

What DataSources Are For

Use case DataSource kind Creates
Status feeds from cloud providers (e.g. Hetzner Status) rss Notes kind=provider-status
Open Telekom Cloud status API otc-status Notes + structured maintenances
Zammad/webhook status mails (push) webhook Notes kind=provider-status
Security or product news rss Notes kind=news
Periodically query external REST APIs api Notes (generic)
Block templates from Polycrate Hub polycrate-hub Template blocks

Kinds at a Glance

rss — RSS/Atom Feeds

Polycrate fetches the feed at a configurable interval and creates one Note per new item. The Note links to the original item (source_url) and stores title, summary and timestamp.

Typically linked to a Provider: if the feed is, for example, the status page of an infrastructure provider, datasource.provider is set and the generated Notes automatically receive kind=provider-status plus a link to this provider. This way provider outages land directly in the timeline/dashboard context.

api — REST APIs

Like rss, but for arbitrary REST endpoints with a JSON response. A Credential can be attached (API token, Basic Auth) so that protected endpoints can be queried. The result is translated into Notes.

polycrate-hub — Block Templates from Poly Hub

Pulls block definitions from the central Polycrate Hub and creates them as installable template blocks (template=true). Replaces the former artifact discovery (see Artifacts).

otc-status — Open Telekom Cloud

Specialized pull against the OTC status API (fixed schema paths). Events are synchronized as Notes and, for maintenances, directly as Maintenances (including PoP matching via component regions). Content hashes allow updates on status changes.

webhook — Push Ingest (e.g. Zammad)

Push-only: no periodic sync.

POST /api/v1/datasources/{uuid}/ingest/

Optional HMAC (X-Hub-Signature) or bearer token. Typical flow: Zammad ticket mail → Note kind=provider-status → AI analysis → Maintenance and/or Incident including affected hosts/volumes/PoPs.

Manual re-scan: Note action rescan_provider_status or DataSource rescan_notes.

Sync Window

RSS and API sync consider items from the last 30 days (older items are ignored). otc-status uses a configurable sliding window (default 45 days).

Relationships

Provider ─────────────┐
                      │ optional FK
                      ▼
┌──────────────┐   ┌──────────────┐   creates     ┌─────────┐
│ Credential   │──▶│  DataSource  │──────────────▶│  Notes  │
│ (for api)    │   │  kind=rss    │               └─────────┘
└──────────────┘   │  kind=api    │
                   │  kind=…-hub  │   creates     ┌───────────────┐
                   └──────────────┘──────────────▶│ Block template│
                                                  └───────────────┘
  • Provider is a global catalog entry (manufacturer/vendor). A DataSource can point to a provider but does not have to — in that case an RSS feed produces, for example, "news" instead of "provider status".
  • Credential provides the authentication for the api and polycrate-hub kinds.
  • Notes are the primary output object. Every Note created by a DataSource receives a back-reference to the DataSource and can be displayed in the Notes tab of all affected objects.

Setting Up a DataSource

Steps in the UI:

  1. Open Admin → DataSources → New.
  2. Choose a Kind (rss, api, polycrate-hub, otc-status, webhook).
  3. Enter the URL or webhook config.
  4. Optional: link a Provider → Notes as provider status.
  5. Optional: Credential (for api / polycrate-hub).
  6. For pull kinds set the Interval; for webhook note the ingest URL + optional tokens.
  7. Save → first sync (pull) or wait for push.

Manual sync via the "Reconcile" action; re-scan for provider status Notes separately.

Control & Observability

  • Last run / last success: Every DataSource shows when it was last synced and whether errors occurred.
  • Activities: Every sync creates an activity entry (kind datasource_sync) with a counter "newly created Notes/blocks". The activities are visible via the metrics dashboard and in the detail view of the DataSource.
  • Error notifications: If a sync fails repeatedly, the DataSource is set to state=CRITICAL and a downtime entry is created — just like for any other ManagedObject.

Related Topics