Production Example: Software Delivery Platform

Example of a production-ready software delivery platform with Polycrate on bare metal: HA Kubernetes with floating IP, hardening, Cilium, Longhorn, VictoriaMetrics, VictoriaLogs and cert-manager as blocks in one workspace.

Production Example: Software Delivery Platform

Overview

This example shows how to build a production-ready software delivery platform with Polycrate on bare-metal servers. The stack includes:

  • High-availability Kubernetes cluster (3 control planes, 6 workers)
  • Floating IP for control plane high availability (Keepalived)
  • Hardening of all nodes (SSH, auditd, Vector → VictoriaLogs)
  • Cilium as CNI with BGP, Gateway API and Hubble
  • Longhorn as distributed storage system
  • VictoriaMetrics for monitoring with Grafana
  • VictoriaLogs for centralized logging (container logs, syslog UDP/514)
  • cert-manager for automated TLS certificates (Let's Encrypt)
  • NGINX Ingress Controller with monitoring integration

Architecture:

graph TB
    subgraph "Controlplanes (HA)"
        CP1[Controlplane 1<br/>10.0.0.11]
        CP2[Controlplane 2<br/>10.0.0.12]
        CP3[Controlplane 3<br/>10.0.0.13]
        VIP[Floating IP<br/>10.0.0.27<br/>keepalived]

        CP1 --> VIP
        CP2 --> VIP
        CP3 --> VIP
    end

    subgraph "Workers"
        W1[Worker 1]
        W2[Worker 2]
        W3[Worker 3]
        W4[Worker 4]
        W5[Worker 5]
        W6[Worker 6]
    end

    VIP --> K8s[Kubernetes API<br/>:6443]

    subgraph "CNI & Networking"
        Cilium[Cilium CNI<br/>+ BGP + Gateway API]
        Hubble[Hubble Observability]
    end

    subgraph "Storage"
        Longhorn[Longhorn<br/>Replicated Storage]
    end

    subgraph "Ingress & TLS"
        Ingress[NGINX Ingress]
        CertManager[cert-manager<br/>Let's Encrypt]
    end

    subgraph "Observability"
        VM[VictoriaMetrics<br/>+ Grafana]
        VL[VictoriaLogs<br/>Syslog :514]
    end

    K8s --> Cilium
    K8s --> Longhorn
    K8s --> Ingress
    K8s --> CertManager
    K8s --> VM
    K8s --> VL

    Cilium --> Hubble

    style VIP fill:#e74c3c,color:#fff
    style K8s fill:#326ce5,color:#fff
    style Cilium fill:#f1c40f,color:#000

Workspace structure

polycrate-demo/
├── workspace.poly          # main configuration
├── inventory.yml           # Ansible inventory (hosts)
└── blocks/                 # blocks pulled from the PolyHub
    └── cargo.ayedo.cloud/
        └── ayedo/
            ├── k8s/
            │   ├── cilium/
            │   ├── cert-manager/
            │   ├── victoria-metrics-stack/
            │   ├── victoria-logs/
            │   ├── longhorn/
            │   ├── nginx/
            │   └── library/
            └── linux/
                ├── hardening/
                ├── keepalived/
                └── k8s-1.27/

Prerequisites

Hardware

  • Control planes: 3 servers (min. 4 vCPU, 8 GB RAM, 100 GB SSD)
  • Workers: 6 servers (min. 8 vCPU, 16 GB RAM, 500 GB SSD)
  • Network: all servers in the same layer-2 network (for the floating IP)

Software

  • Operating system: Ubuntu 22.04 LTS or Debian 12
  • SSH access: user with sudo rights on all servers
  • DNS: wildcard DNS for ingress (e.g. *.example.com)
  • Cloudflare: account + API key for the Let's Encrypt DNS01 challenge

Local installation

# Install Polycrate
curl -sSL https://get.polycrate.io | bash

# Create the workspace
mkdir -p ~/polycrate-workspaces/production
cd ~/polycrate-workspaces/production
polycrate workspace init --with-name production

Configuring the inventory

Create inventory.yml with your server IPs:

all:
  hosts:
    prod-controlplane-1:
      ansible_host: 10.0.0.11
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-controlplane-2:
      ansible_host: 10.0.0.12
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-controlplane-3:
      ansible_host: 10.0.0.13
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-1:
      ansible_host: 10.0.0.21
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-2:
      ansible_host: 10.0.0.22
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-3:
      ansible_host: 10.0.0.23
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-4:
      ansible_host: 10.0.0.24
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-5:
      ansible_host: 10.0.0.25
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

    prod-worker-6:
      ansible_host: 10.0.0.26
      ansible_user: "demo"
      ansible_ssh_port: 22
      ansible_python_interpreter: "/usr/bin/python3"
      ansible_become: "yes"

  children:
    k8s_controlplane:
      hosts:
        prod-controlplane-1:
        prod-controlplane-2:
        prod-controlplane-3:

    k8s_worker:
      hosts:
        prod-worker-1:
        prod-worker-2:
        prod-worker-3:
        prod-worker-4:
        prod-worker-5:
        prod-worker-6:

Workspace configuration

Create workspace.poly:

# Note: YAML anchors for illustration only; for production see the configuration docs.
project-name: &project-name production
k3s-token: &k3s-token "your-secure-k3s-token-here"
letsencrypt-email: &letsencrypt-email "admin@example.com"
cloudflare-api-email: &cloudflare-api-email "cloudflare@example.com"
cloudflare-api-key: &cloudflare-api-key "your-cloudflare-api-key"

ssh_port: &ssh_port 22
ssh_user: &ssh_user demo

name: *project-name
organization: &organization example-org

# OCI registry for private blocks (optional)
registry:
  endpoint: registry.example.com
  username: robot+production-user
  password: registry-password

blocks:
  # 1. Library (base block with helper functions)
  - name: library
    from: cargo.ayedo.cloud/ayedo/k8s/library:0.0.6

  # 2. Hardening (security + host collector for all nodes)
  - name: hardening
    from: cargo.ayedo.cloud/ayedo/linux/hardening:0.1.13
    config:
      hosts: all
      firewall:
        enabled: false  # dedicated router/firewall present
      ssh:
        port: *ssh_port
        user: *ssh_user
      auditd:
        enabled: true
      collectors:
        enabled: true
        vector:
          enabled: true
          cluster: *project-name
          logs:
            enabled: true
            path: /insert/elasticsearch
            endpoint: https://logs.example.com
            username: logs-user
            password: logs-password-change-me

  # 3. Keepalived (floating IP for control planes)
  - name: keepalived-controlplanes
    from: cargo.ayedo.cloud/ayedo/linux/keepalived:0.0.1
    config:
      hosts: k8s_controlplane
      app:
        virtual_ips:
          - 10.0.0.27/24  # floating IP
        password: vrrp-password-change-me
        master: prod-controlplane-1
        interface: eth0

  # 4. Kubernetes cluster (K3s)
  - name: k8s
    from: cargo.ayedo.cloud/ayedo/linux/k8s-1.27:0.3.6
    config:
      external_cloud_provider: false
      service_lb: false
      cluster_name: *project-name
      init:
        restart_nodes: false
      kube_proxy:
        disabled: true  # Cilium takes over kube-proxy
      oidc:
        enabled: true
        issuer_url: "https://id.example.com/"
        client_id: "kube-apiserver"
        username_claim: "email"
        groups_claim: "groups"
      apt:
        packages:
          - apparmor
          - apparmor-utils
          - unattended-upgrades
          - python3-pip
      flannel:
        disabled: true  # Cilium as CNI
      k3s:
        token: *k3s-token
        version: v1.34.1+k3s1
        tlssans:
          - "10.0.0.27"  # floating IP for the API server
      ansible_host_groups:
        controlplane: k8s_controlplane
        worker: "k8s_worker"

  # 5. Cilium CNI (Container Network Interface)
  - name: cilium
    from: cargo.ayedo.cloud/ayedo/k8s/cilium:0.2.12
    kubeconfig:
      from: k8s
    config:
      bandwidth_manager: true
      host_port: true
      kube_proxy_replacement: true
      node_port: true
      gateway_api: true
      k8s_service_host: 10.0.0.27  # floating IP
      k8s_service_port: 6443
      loadbalancer_algorithm: maglev
      hubble:
        enabled: true
      loadbalancer_acceleration: best-effort
      loadbalancer_mode: snat
      routing_mode: tunnel
      routing_cidr: "172.16.187.0/24"
      auto_direct_node_routes: false
      masquerade_enabled: true
      masquerade_disabled: false
      bpf_masquerade_disabled: false
      encryption:
        enabled: false
      external_ips: true
      bgp_controlplane: true
      l2announcements: false

  # 6. cert-manager (TLS certificates)
  - name: cert-manager
    from: cargo.ayedo.cloud/ayedo/k8s/cert-manager:0.4.0
    kubeconfig:
      from: k8s
    config:
      letsencrypt:
        email: *letsencrypt-email
        solver:
          type:
            - dns01
          provider: cloudflare
          create_secret: true
          cloudflare:
            api_email: *cloudflare-api-email
            api_key: *cloudflare-api-key

  # 7. VictoriaMetrics stack (monitoring + Grafana)
  - name: monitoring
    from: cargo.ayedo.cloud/ayedo/k8s/victoria-metrics-stack:0.12.5
    kubeconfig:
      from: k8s
    config:
      labels:
        cluster: *project-name
        customer: *organization
        workspace: *project-name
      grafana:
        enabled: true
        custom_dashboards: true
        persistence:
          enabled: true
          storageclassname: longhorn
          size: 30Gi
        admin_password: "grafana-admin-password-change-me"
        ingress:
          enabled: true
          host: grafana.example.com
          class: nginx
          tls:
            enabled: true
            issuer: letsencrypt-production
        oidc:
          auto_login: true
          enabled: true
          name: "SSO Login"
          client_id: "grafana-client-id"
          client_secret: "grafana-client-secret"
          auth_url: "https://id.example.com/application/o/authorize/"
          token_url: "https://id.example.com/application/o/token/"
          api_url: "https://id.example.com/application/o/userinfo/"
          allow_sign_up: true
          scopes: "openid profile email"
          admin_group: "Admins"
          editor_group: "Editors"
      vm:
        replicationfactor: 3
        storage:
          replicas: 2
          size: 100Gi
          storageclassname: "longhorn"
          memory_limit: 10Gi
          cpu_limit: 2
        select:
          replicas: 2
          size: 10Gi
          storageclassname: "longhorn"
          memory_limit: 8Gi
          cpu_limit: 2
        insert:
          replicas: 2
          memory_limit: 2Gi
          cpu_limit: 1
        agent:
          memory_limit: 2Gi
          cpu_limit: 2

  # 8. VictoriaLogs (centralized logging)
  - name: logs
    from: cargo.ayedo.cloud/ayedo/k8s/victoria-logs:0.2.6
    kubeconfig:
      from: k8s
    config:
      namespace: victoria-logs
      mode: single
      vlsingle:
        syslog:
          enabled: true
          loadbalancer:
            enabled: true
            ip: 10.0.0.28  # dedicated IP for syslog (UDP 514)
        pvc:
          class: longhorn
          size: 100Gi
        resources:
          limits:
            cpu: 2
            memory: 8Gi
          requests:
            cpu: 1
            memory: 4Gi
      monitoring:
        enabled: true
        vmservicescrape:
          enabled: true
      backup:
        enabled: false

  # 9. Longhorn (distributed block storage)
  - name: longhorn
    from: cargo.ayedo.cloud/ayedo/k8s/longhorn:0.2.3
    kubeconfig:
      from: k8s
    config:
      volume_replica_count: 3

  # 10. NGINX Ingress Controller
  - name: ingress
    from: cargo.ayedo.cloud/ayedo/k8s/nginx:0.2.2
    kubeconfig:
      from: k8s
    config:
      metrics:
        enabled: true
        dashboards:
          enabled: true
        vmservicescrape:
          enabled: true

Deployment process

Phase 1: Hardening & floating IP

# 1. Hardening of all nodes
polycrate run hardening install

# 2. Keepalived for the floating IP on the control planes
polycrate run keepalived-controlplanes install

# Verify: the floating IP should be reachable
ping 10.0.0.27

Phase 2: Kubernetes cluster

# 3. Provision the Kubernetes cluster (takes ~10-15 min)
polycrate run k8s install

# The kubeconfig is generated automatically to:
# artifacts/blocks/k8s/kubeconfig.yml

# Verify
export KUBECONFIG=artifacts/blocks/k8s/kubeconfig.yml
kubectl get nodes
# Should show 3 control planes + 6 workers (NotReady, since the CNI is missing)

Phase 3: Networking & storage

# 4. Deploy Cilium CNI
polycrate run cilium install

# Nodes should now be Ready
kubectl get nodes

# 5. Deploy Longhorn storage
polycrate run longhorn install

# Verify
kubectl get storageclass
# Should show "longhorn" StorageClass

Phase 4: Ingress & TLS

# 6. Deploy cert-manager
polycrate run cert-manager install

# 7. Deploy NGINX Ingress
polycrate run ingress install

# Verify
kubectl get pods -n ingress-nginx
kubectl get clusterissuer
# Should show: letsencrypt-production, letsencrypt-staging

Phase 5: Observability

# 8. Deploy VictoriaMetrics + Grafana
polycrate run monitoring install

# 9. Deploy VictoriaLogs
polycrate run logs install

# Verify
kubectl get pods -n victoria-metrics-stack
kubectl get pods -n victoria-logs

# Open Grafana
# https://grafana.example.com
# (Login with OIDC or admin / grafana-admin-password-change-me)

Workflows for complete deployments

You can define workflows to deploy several blocks one after another:

# Add to workspace.poly:
workflows:
  - name: deploy-complete-stack
    steps:
      - name: hardening
        block: hardening
        action: install

      - name: keepalived
        block: keepalived-controlplanes
        action: install

      - name: kubernetes
        block: k8s
        action: install

      - name: cilium
        block: cilium
        action: install

      - name: longhorn
        block: longhorn
        action: install

      - name: cert-manager
        block: cert-manager
        action: install

      - name: ingress
        block: ingress
        action: install

      - name: monitoring
        block: monitoring
        action: install

      - name: logs
        block: logs
        action: install

Run:

polycrate workflows run deploy-complete-stack

Verification & testing

Cluster status

# Nodes
kubectl get nodes -o wide

# Pods across all namespaces
kubectl get pods -A

# Storage
kubectl get pv,pvc -A

# Ingress
kubectl get ingress -A

Cilium status

# Cilium status
kubectl exec -n kube-system ds/cilium -- cilium status

# Hubble UI (optional)
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
# http://localhost:12000

Monitoring

# VictoriaMetrics targets
kubectl port-forward -n victoria-metrics-stack svc/vmsingle-vm 8429:8429
# http://localhost:8429/targets

# Grafana
# https://grafana.example.com
# Default dashboards:
# - Kubernetes / Compute Resources / Cluster
# - Kubernetes / Networking / Cluster
# - Cilium Metrics
# - NGINX Ingress Controller

Test deployment

Test the platform with a simple application:

# test-app.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: test-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: test-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: test-app
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: test-app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - test.example.com
    secretName: test-app-tls
  rules:
  - host: test.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

kubectl apply -f test-app.yaml

# Wait for the TLS certificate
kubectl get certificate -n test-app -w

# Test
curl https://test.example.com

Maintenance & updates

Kubernetes updates

# Update K3s to a new version
# In workspace.poly:
# k3s.version: v1.35.0+k3s1

polycrate run k8s upgrade

Block updates

# Pull the new block version
polycrate blocks pull cargo.ayedo.cloud/ayedo/k8s/cilium:0.3.0

# Adjust the version in workspace.poly
# from: cargo.ayedo.cloud/ayedo/k8s/cilium:0.3.0

# Run the upgrade
polycrate run cilium upgrade

Backup & disaster recovery

# Velero for cluster backups (optional)
polycrate blocks pull cargo.ayedo.cloud/ayedo/k8s/velero:latest

# Add to workspace.poly and configure
# Then deploy:
polycrate run velero install

Troubleshooting

Analyzing logs

# Polycrate logs
ls -la .logs/

# Latest transaction
cat .logs/latest.log

# Ansible output
cat .logs/<transaction-id>/ansible-output.log

SSH debugging

# Direct SSH access to a node
polycrate ssh prod-controlplane-1

# Test the Ansible inventory
polycrate inventory list
polycrate inventory ping

Kubernetes debugging

# Events
kubectl get events -A --sort-by='.lastTimestamp'

# Pod logs
kubectl logs -n <namespace> <pod-name>

# Describe for details
kubectl describe pod -n <namespace> <pod-name>

Best practices

1. Versioning

# Always pin specific versions
from: cargo.ayedo.cloud/ayedo/k8s/cilium:0.2.12
# Not:
from: cargo.ayedo.cloud/ayedo/k8s/cilium:latest

2. Secrets management

Important: secrets such as API keys, passwords and tokens do not belong in Git!

Recommended approaches:

  • Separate secrets file: create a secrets.poly file (not in Git) with sensitive values
  • Manual entry: enter secrets directly in workspace.poly before deployment, do not commit
  • External secrets management: use tools such as HashiCorp Vault, SOPS or age encryption

Example .gitignore: 

secrets.poly
*.key
*.pem
id_rsa*

3. Testing

  • Test updates in a staging environment first
  • Use separate workspaces for dev/staging/production
  • Document changes in Git commits

4. Monitoring

  • Set up alerting in Grafana
  • Monitor resource usage (CPU, RAM, disk)
  • Set PodDisruptionBudgets for critical apps

5. Backups

  • Regular backups with Velero
  • Test restore processes
  • Document recovery procedures

Resources

Next steps