Zurück zum Hub

cargo.ayedo.cloud/ayedo/k8s/powerdns

Registry-Block

polycrate block pull cargo.ayedo.cloud/ayedo/k8s/powerdns:0.1.6

PowerDNS Authoritative — Polycrate Block

Deploys PowerDNS Authoritative Server (pdns-auth) to Kubernetes. Supports single-node and multi-replica deployments with optional Lightning Stream LMDB replication via S3, and an optional DNSdist front-end proxy for load balancing, rate limiting, and DDoS protection.

Quick Reference

polycrate run powerdns install    # Deploy / update
polycrate run powerdns uninstall  # Remove all resources

Architecture

External DNS clients
        │
        ▼
  [ LoadBalancer ]
        │
        ▼
  [ DNSdist (optional) ]      ← rate limiting, caching, DDoS protection
    Deployment, N replicas
        │ per-pod backends via headless service
        ▼
  [ pdns-auth ]               ← authoritative DNS, LMDB or PostgreSQL
    StatefulSet / Deployment, N replicas
        │
        ├─── LMDB on PVC (ceph-block or similar)
        │
        └─── [ Lightning Stream (optional sidecar) ]
                    │
                    ▼
              [ S3 / Object Storage ]   ← cross-replica LMDB sync

Traffic flow with DNSdist enabled: - External DNS queries → LoadBalancer → DNSdist → pdns-auth (per-pod backends) - DNSdist applies packet cache, rate limits, and dynamic blocks before forwarding - pdns-auth is only reachable internally (ClusterIP); never directly exposed

Traffic flow without DNSdist: - External DNS queries → LoadBalancer → pdns-auth directly

Prerequisites

  • Polycrate CLI with a configured Kubernetes context (kubeconfig.from in the block config)
  • kubectl available in the execution environment
  • For Lightning Stream: an S3-compatible bucket (Ceph RGW, MinIO, AWS S3, etc.)
  • For metrics: VictoriaMetrics with VMServiceScrape CRD installed

Deployment Modes

deployment (default)

Standard Kubernetes Deployment. Suitable for PostgreSQL backend or ephemeral LMDB (no persistence). All pods share a single PVC is not possible; each pod would get its own in-memory LMDB.

config:
  pdns:
    mode: deployment
    replicas: 2
    backend: gpgsql

statefulset (recommended for LMDB + Lightning Stream)

Kubernetes StatefulSet with volumeClaimTemplates. Each pod gets its own persistent LMDB volume. Lightning Stream syncs LMDB state across all pods via S3.

Automatically promoted to StatefulSet when backend=lmdb and lightning_stream.storage.class is set — regardless of the mode field.

config:
  pdns:
    mode: statefulset
    replicas: 3
    backend: lmdb
    lightning_stream:
      enabled: true
      storage:
        class: ceph-block
        size: 10Gi

Important: Changing podManagementPolicy on an existing StatefulSet requires deleting it first: 

kubectl -n powerdns delete statefulset powerdns-auth
polycrate run powerdns install

daemonset

Runs one pod per node. Useful for infrastructure-level DNS where every node should serve DNS directly.

Configuration Reference

All configuration lives under blocks[name].config in workspace.poly.

Namespace

config:
  namespace: powerdns    # Kubernetes namespace (created automatically)

pdns — PowerDNS Authoritative

Image

config:
  pdns:
    image: powerdns/pdns-auth-49    # Docker image name
    image_tag: "4.9.13"             # Image tag

Version compatibility: Lightning Stream 0.5.x requires pdns-auth 4.9.x (LMDB schema version 5). Do not use pdns-auth 5.x with Lightning Stream 0.5.x.

Deployment

config:
  pdns:
    mode: deployment      # deployment | statefulset | daemonset
    replicas: 3           # Number of replicas (ignored for daemonset)

DNS Listener

config:
  pdns:
    dns:
      port: 53
      local_address: "0.0.0.0"

HTTP API

config:
  pdns:
    api:
      enabled: true
      port: 8081
      key: "your-api-key"           # Required when enabled
      allow_from: "0.0.0.0/0"       # Restrict in production

The API is accessible at http://powerdns-auth.<namespace>.svc.cluster.local:8081/api/v1.

For local access via port-forward: 

kubectl -n powerdns port-forward svc/powerdns-auth 8081:8081

# List zones
curl -H "X-API-Key: your-api-key" http://localhost:8081/api/v1/servers/localhost/zones

# Create a zone
curl -s -X POST \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"name":"example.com.","kind":"Native","nameservers":["ns1.example.com."]}' \
  http://localhost:8081/api/v1/servers/localhost/zones

# Add records (SOA required first, then A records)
curl -s -X PATCH \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "rrsets": [
      {
        "name": "www.example.com.",
        "type": "A",
        "ttl": 300,
        "changetype": "REPLACE",
        "records": [{"content": "192.168.1.10", "disabled": false}]
      }
    ]
  }' \
  http://localhost:8081/api/v1/servers/localhost/zones/example.com.

Backend: PostgreSQL (gpgsql)

config:
  pdns:
    backend: gpgsql
    postgresql:
      host: "postgres.default.svc.cluster.local"
      port: 5432
      database: powerdns
      user: powerdns
      password: "your-password"
      dnssec: false

The PostgreSQL schema must be initialized before the first run. See the pdns-auth gpgsql backend documentation.

Backend: LMDB

LMDB is an embedded key-value database stored on disk. Required for Lightning Stream multi-replica sync.

config:
  pdns:
    backend: lmdb
    lmdb:
      path: /var/lib/powerdns/pdns.lmdb    # Path inside the container / PVC
      map_size_mb: 65536                    # LMDB memory-map size in MB

The LMDB file is stored on a PVC per pod when lightning_stream.storage.class is set. Without a storage class, an emptyDir is used (data lost on pod restart).

Lightning Stream (LMDB multi-replica sync)

Lightning Stream runs as a sidecar in each pdns-auth pod. mode: sync uploads LMDB snapshots and downloads snapshots from other instances. mode: receive runs lightningstream receive (0.5.3): it downloads and merges snapshots and never writes them. Receive-only also disables storage cleanup, so a receiver does not delete another instance's snapshots. The block default is receive. The single writer sets mode: sync.

    lightning_stream:
      enabled: true
      mode: sync          # writer. Satellites use mode: receive
      image: powerdns/lightningstream
      image_tag: "0.5.3"

config:
  pdns:
    backend: lmdb                          # Required
    mode: statefulset                      # Recommended
    replicas: 3

    lightning_stream:
      enabled: true
      image: powerdns/lightningstream
      image_tag: "0.5.3"

      s3:
        endpoint: "https://s3.your-cluster.example.com"
        bucket: "powerdns-lightning-stream"
        prefix: "powerdns/"
        region: "us-east-1"
        access_key: "ACCESS_KEY"
        secret_key: "SECRET_KEY"

      sync_interval: 10                   # Seconds between S3 sync cycles

      storage:
        size: 10Gi
        class: ceph-block                 # StorageClass for PVCs

How it works: 1. Pod 0 writes a zone via the API → pdns-auth writes LMDB with Lightning Stream change markers 2. Lightning Stream detects the change, uploads a snapshot to S3 3. Lightning Stream on pods 1 and 2 downloads the snapshot and applies it to their local LMDB 4. After zone_cache_refresh_interval seconds, pods 1 and 2 reload their zone cache and serve the new zone

Requirements: - S3 bucket must be pre-created - Bucket must be accessible from all pods (credentials are stored in a Kubernetes Secret) - pdns.api.key must be set (Lightning Stream calls the pdns-auth API to trigger zone cache reload)

Data cleanup (required when changing schema version or for a fresh start): 

# Delete StatefulSet and PVCs
kubectl -n powerdns delete statefulset powerdns-auth
kubectl -n powerdns delete pvc -l app=powerdns-auth

# Clear S3 bucket prefix
mc rm --recursive --force <alias>/<bucket>/powerdns/

# Re-deploy
polycrate run powerdns install

Performance Tuning

config:
  pdns:
    server_id: "ns.example.com"
    loglevel: 4                        # 0=emergency ... 7=debug
    log_dns_details: false
    log_dns_queries: false             # Enable temporarily for troubleshooting
    any_to_tcp: true                   # Respond to ANY queries via TCP only
    version_string: anonymous          # Hide version from version.bind queries
    distributor_threads: 3             # Threads distributing queries to backends
    receiver_threads: 8                # Threads receiving queries from the network
    reuseport: true                    # SO_REUSEPORT for better multi-core throughput
    expand_alias: false                # ALIAS record expansion
    zone_cache_refresh_interval: 10   # Seconds between zone cache reloads
    extra_config: ""                   # Verbatim lines appended to pdns.conf

zone_cache_refresh_interval=10 is critical with Lightning Stream: after another pod creates a zone and syncs it via S3, this pod will serve REFUSED until it reloads its zone cache. Keep this low (5–30s).

Resources

config:
  pdns:
    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"

loadbalancer — External DNS Exposure

Creates a LoadBalancer Service. When DNSdist is enabled, the LoadBalancer targets DNSdist. When DNSdist is disabled, it targets pdns-auth directly.

config:
  loadbalancer:
    enabled: true
    annotations:
      lbipam.cilium.io/ips: "93.191.53.11"                    # Cilium IPAM: fixed IP
      # service.beta.kubernetes.io/aws-load-balancer-type: "nlb"  # AWS NLB
      # metallb.universe.tf/address-pool: "public"             # MetalLB pool

metrics — Prometheus / VictoriaMetrics

Creates VMServiceScrape resources for pdns-auth, DNSdist, and Lightning Stream.

config:
  metrics:
    enabled: true
    vmservicescrape:
      enabled: true

Scraped endpoints:

Component Port Path
pdns-auth 8081 /metrics
DNSdist (when enabled) 8083 /metrics
Lightning Stream (when enabled) 8500 /metrics

All metrics include a pod label (via relabeling) and inherit the workspace label from the scrape target.

security — Pod Security

config:
  security:
    enabled: true
    run_as_non_root: false       # Must be false — pdns-auth binds port 53 and needs root
    net_bind_service: true       # CAP_NET_BIND_SERVICE capability
    seccomp_profile: RuntimeDefault

Both powerdns-auth and lightning-stream containers run as runAsUser: 0 (root). This is required because: - pdns-auth binds port 53 (privileged port) - Both containers share a PVC for the LMDB files; running as root avoids file permission conflicts

dnsdist — DNS Proxy / Load Balancer

DNSdist is an optional front-end proxy. When enabled, it receives all external DNS traffic and distributes it across the pdns-auth pods.

Enable

config:
  dnsdist:
    enabled: true
    image: powerdns/dnsdist-19
    image_tag: "1.9.11"
    mode: deployment
    replicas: 2
    lb_policy: leastOutstanding    # leastOutstanding | roundrobin | firstAvailable | wrandom | whashed | chashed | random

ACL

config:
  dnsdist:
    acl:
      - "0.0.0.0/0"    # Allow all (default)
      - "::/0"

Queries from IPs not matching the ACL are immediately dropped.

Packet Cache

config:
  dnsdist:
    cache:
      enabled: true
      size: 1000000      # Max cached responses
      max_ttl: 86400     # Cap TTL for cached entries (seconds)
      min_ttl: 0         # Minimum TTL (0 = cache everything incl. TTL=0)
      stale_ttl: 60      # Serve stale cache entries while backend is down
      dont_age: false    # If true, TTLs are not decremented while cached

The packet cache is shared within each DNSdist pod. It absorbs repeat queries before they reach pdns-auth. A high cache hit ratio (>80%) significantly reduces load on the backend.

Rate Limiting

Per-IP query rate limiting. Queries exceeding the threshold are delayed or dropped.

config:
  dnsdist:
    rate_limiting:
      enabled: true
      delay_qps: 100      # QPS above which queries are delayed by delay_ms
      delay_ms: 100       # Milliseconds to delay queries above delay_qps
      drop_qps: 200       # QPS above which queries are dropped entirely
      ipv4_mask: 32       # Group by /32 = per individual IPv4 address
      ipv6_mask: 48       # Group by /48 for IPv6

Disable temporarily for load testing (rate_limiting.enabled: false) and re-enable afterwards.

Dynamic Blocks

Automatically and temporarily bans IPs that exceed query rate or SERVFAIL thresholds. Uses the dynBlockRulesGroup() API (DNSdist 1.9.x).

config:
  dnsdist:
    dyn_blocks:
      enabled: true
      exceed_qps: 300              # QPS threshold to trigger a block
      window_seconds: 10           # Measurement window
      block_duration: 60           # How long (seconds) the block lasts
      block_reason: "Exceeded query rate"

      # Also block IPs with high SERVFAIL rates (amplification attack mitigation)
      exceed_servfail_percent: 30
      exceed_servfail_window: 10
      exceed_servfail_duration: 120
      exceed_servfail_reason: "Excessive SERVFAIL rate"

Active blocks are visible in the Grafana dashboard under "Rate Limiting & Security".

TCP/UDP Tuning

config:
  dnsdist:
    tcp:
      max_client_threads: 10           # Parallel TCP connection threads
      recv_timeout: 2                   # TCP receive timeout (seconds)
      send_timeout: 2                   # TCP send timeout (seconds)
      max_queued_connections: 1000      # Max queued TCP connections
      max_queries_per_connection: 50    # Mitigates CVE-2025-30193 (stack exhaustion DoS)
    udp:
      max_outstanding: 65535            # Max outstanding UDP queries

DNSdist API

config:
  dnsdist:
    api:
      enabled: true
      port: 8083
      key: "your-dnsdist-api-key"          # Required when enabled
      allow_from: "127.0.0.1/8,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

The API also exposes /metrics (Prometheus format) on the same port.

For local access: 

kubectl -n powerdns port-forward svc/dnsdist 8083:8083

# Show server stats
curl -H "X-API-Key: your-dnsdist-api-key" http://localhost:8083/api/v1/servers/localhost

# Show backends
curl -H "X-API-Key: your-dnsdist-api-key" http://localhost:8083/api/v1/servers/localhost/pools

DNS over TLS (DoT)

config:
  dnsdist:
    dot:
      enabled: true
      port: 853
      cert_secret: "dnsdist-tls"    # Kubernetes Secret with tls.crt and tls.key

Create the TLS secret manually or via cert-manager: 

kubectl -n powerdns create secret tls dnsdist-tls \
  --cert=path/to/cert.pem \
  --key=path/to/key.pem

DNS over HTTPS (DoH)

config:
  dnsdist:
    doh:
      enabled: true
      port: 443
      path: "/dns-query"
      cert_secret: "dnsdist-tls"    # Kubernetes Secret with tls.crt and tls.key

Custom Lua Config

config:
  dnsdist:
    extra_config: |
      -- Custom Lua rules appended verbatim to dnsdist.conf
      addAction(QTypeRule(DNSQType.ANY), TCAction())

Resources

config:
  dnsdist:
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "2000m"
        memory: "1Gi"

Example Configurations

Minimal — single replica, PostgreSQL

- name: powerdns
  from: cargo.ayedo.cloud/ayedo/k8s/powerdns
  kubeconfig:
    from: k8s
  config:
    namespace: powerdns
    pdns:
      mode: deployment
      replicas: 1
      backend: gpgsql
      api:
        enabled: true
        key: "change-me"
      postgresql:
        host: "postgres.default.svc.cluster.local"
        database: powerdns
        user: powerdns
        password: "change-me"
    loadbalancer:
      enabled: true

Production — StatefulSet + Lightning Stream + DNSdist + Metrics

- name: powerdns
  from: cargo.ayedo.cloud/ayedo/k8s/powerdns
  kubeconfig:
    from: k8s
  config:
    namespace: powerdns

    pdns:
      mode: statefulset
      replicas: 3
      backend: lmdb
      api:
        enabled: true
        key: "your-pdns-api-key"
      lmdb:
        path: /var/lib/powerdns/pdns.lmdb
        map_size_mb: 65536
      lightning_stream:
        enabled: true
        s3:
          endpoint: "https://s3.your-cluster.example.com"
          bucket: "powerdns-lightning-stream"
          prefix: "powerdns/"
          region: "us-east-1"
          access_key: "ACCESS_KEY"
          secret_key: "SECRET_KEY"
        sync_interval: 10
        storage:
          size: 10Gi
          class: ceph-block
      zone_cache_refresh_interval: 10
      resources:
        requests:
          cpu: "200m"
          memory: "512Mi"
        limits:
          cpu: "2000m"
          memory: "2Gi"

    loadbalancer:
      enabled: true
      annotations:
        lbipam.cilium.io/ips: "93.191.53.11"

    metrics:
      enabled: true
      vmservicescrape:
        enabled: true

    dnsdist:
      enabled: true
      replicas: 2
      api:
        enabled: true
        key: "your-dnsdist-api-key"
      cache:
        enabled: true
        size: 1000000
      rate_limiting:
        enabled: true
        delay_qps: 100
        drop_qps: 200
      dyn_blocks:
        enabled: true
        exceed_qps: 300
      resources:
        requests:
          cpu: "200m"
          memory: "256Mi"
        limits:
          cpu: "2000m"
          memory: "1Gi"

Operations

Check pod status

kubectl -n powerdns get pods
kubectl -n powerdns get statefulset
kubectl -n powerdns get pvc

Check logs

# pdns-auth
kubectl -n powerdns logs statefulset/powerdns-auth -c powerdns-auth -f

# Lightning Stream sidecar
kubectl -n powerdns logs statefulset/powerdns-auth -c lightning-stream -f

# DNSdist
kubectl -n powerdns logs deployment/dnsdist -f

Test DNS resolution

# Via cluster-internal service
kubectl -n powerdns run dnstest --image=busybox --rm -it -- nslookup example.com powerdns-auth

# Via LoadBalancer IP
dig @<LB_IP> example.com A

# Via dnsdist (when enabled)
dig @<LB_IP> example.com A

# Force TCP (required when using kubectl port-forward)
dig @127.0.0.1 +tcp example.com A

Rolling update

polycrate run powerdns install

With podManagementPolicy: OrderedReady (default for StatefulSet), pods are updated one at a time. DNSdist's dynamic backend resolver detects IP changes within ~30 seconds and automatically replaces stale backends.

Verify Lightning Stream sync

# Check each pod sees the same zones
for i in 0 1 2; do
  echo "--- pod $i ---"
  kubectl -n powerdns exec powerdns-auth-$i -c powerdns-auth -- \
    curl -s -H "X-API-Key: your-api-key" \
    http://localhost:8081/api/v1/servers/localhost/zones | jq '[.[].name]'
done

# Check S3 sync status in Lightning Stream logs
kubectl -n powerdns logs statefulset/powerdns-auth -c lightning-stream | grep -E "upload|snapshot|error"

Load testing

# Install dnsperf
sudo apt install -y dnsperf   # Debian/Ubuntu
sudo dnf install -y dnsperf   # Fedora/RHEL

# Create query file
cat > /tmp/dns-queries.txt << EOF
example.com A
www.example.com A
EOF

# Run load test (5000 QPS, 60 seconds, 20 clients)
dnsperf -s <LB_IP> -d /tmp/dns-queries.txt -Q 5000 -l 60 -c 20 -T 4

Disable rate_limiting and dyn_blocks before load testing, then re-enable.

Troubleshooting

REFUSED for newly created zone

pdns-auth serves REFUSED if the zone is not in its in-memory zone cache. After a zone is created via the API (or synced via Lightning Stream from another pod), each pod must reload its zone cache.

  • Lower zone_cache_refresh_interval (default: 10s)
  • Verify the zone exists on the queried pod via the API

SERVFAIL with aa flag (pdns-auth returns it)

The zone exists but has no SOA record. PowerDNS requires a valid SOA to answer authoritatively.

curl -s -X PATCH \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "rrsets": [{
      "name": "example.com.",
      "type": "SOA",
      "ttl": 300,
      "changetype": "REPLACE",
      "records": [{"content": "ns1.example.com. hostmaster.example.com. 1 10800 3600 604800 300", "disabled": false}]
    }]
  }' \
  http://localhost:8081/api/v1/servers/localhost/zones/example.com.

Lightning Stream not uploading

  1. Verify lmdb-lightning-stream=yes is in pdns.conf (check ConfigMap): without this, pdns-auth does not write change markers and Lightning Stream sees no changes to upload
  2. Verify lmdb-shards=1 is set — Lightning Stream requires exactly 1 shard
  3. Check S3 credentials and bucket reachability from inside the pod: 
    kubectl -n powerdns exec powerdns-auth-0 -c lightning-stream -- \
  wget -qO- https://s3.example.com/<bucket>/?list-type=2
  4. Check Lightning Stream logs for upload errors

DNSdist backends all down after pdns-auth rolling restart

DNSdist uses a Lua maintenance() function that re-resolves per-pod hostnames every 30 seconds. After a rolling restart, stale backend IPs are replaced automatically within ~30 seconds. Monitor with:

kubectl -n powerdns logs deployment/dnsdist | grep -E "Marking downstream|Backend.*IP changed"

If backends stay down longer, check that the headless service powerdns-auth-headless exists and that pod DNS names resolve correctly:

kubectl -n powerdns exec deployment/dnsdist -- \
  getent hosts powerdns-auth-0.powerdns-auth-headless.powerdns.svc.cluster.local

Permission denied on LMDB files

Both powerdns-auth and lightning-stream containers must run as the same user (root, UID 0) to share a PVC. Verify the pod security context:

kubectl -n powerdns get pod powerdns-auth-0 -o jsonpath='{.spec.containers[*].securityContext}'

StatefulSet update blocked by immutable fields

Kubernetes does not allow changing podManagementPolicy, volumeClaimTemplates, or selector on an existing StatefulSet. Delete and recreate:

kubectl -n powerdns delete statefulset powerdns-auth
polycrate run powerdns install

Note: the PVCs are not deleted automatically. Data is preserved.

Kubernetes Resources Created

Resource Name Notes
Namespace <namespace> Created if not exists
Secret powerdns-secret API keys, S3 credentials
ConfigMap powerdns-config pdns.conf
ConfigMap lightningstream-config Lightning Stream YAML (when enabled)
ConfigMap dnsdist-config DNSdist Lua config (when enabled)
StatefulSet or Deployment powerdns-auth pdns-auth + Lightning Stream sidecar
Service (ClusterIP) powerdns-auth Internal access to DNS + API
Service (Headless) powerdns-auth-headless StatefulSet pod DNS names (when StatefulSet)
Service (LoadBalancer) powerdns-auth-lb External DNS (when LB enabled, DNSdist disabled)
Deployment dnsdist DNSdist (when enabled)
Service (ClusterIP) dnsdist Internal dnsdist access
Service (LoadBalancer) dnsdist-lb External DNS via dnsdist (when LB + dnsdist enabled)
VMServiceScrape powerdns-auth, dnsdist, lightningstream Metrics scraping (when enabled)
PersistentVolumeClaim lmdb-powerdns-auth-{0,1,2,...} Per-pod LMDB PVCs (StatefulSet only)