k3s-server

Agent-less k3s Controlplane (Managed Controlplane / MCP) im Host-Cluster — Jinja-Manifeste + kubectl apply (wie polycrate-api).

Greenfield-Fork des früheren Blocks cargo.ayedo.cloud/ayedo/k8s/k3k (mship-partner-platform-2). Ein Pod stellt kube-apiserver, controller-manager, scheduler und (default) embedded etcd bereit. Worker joinen von außen über denselben Bootstrap-Token.

Beispiele: examples.poly — anzeigen mit polycrate block examples k3s-server.

Produkt-/Pricing-Kontext (Managed-CP):

• polycrate spec --block resource-based-pricing inspect 1 -w ~/.polycrate/workspaces/ayedo/data-workspace
• …/data-workspace/blocks/resource-based-pricing/MODELS.md §9
• aycloud Spec: polycrate spec inspect 1 -w ~/.polycrate/workspaces/ayedo/aycloud-platform-1

Architektur (Kurz)

Host-Cluster Namespace (<block.config.namespace>)
├── Deployment k3s-server (replicas=1, strategy=Recreate)
│   ├── container controlplane  → k3s server --disable-agent …
│   ├── sidecar pod-data-handler → sync kubeconfig + metrics-tls Secrets
│   └── container audit-log (default an) → tail audit NDJSON → stdout → VLogs
├── emptyDir audit-logs / ConfigMap <block.name>-audit-policy
├── PVC <block.name>  → /var/lib/rancher/k3s  (TLS, etcd, kubeconfig, creds)
├── Secret <block.name>-token / kubeconfig / metrics-tls
├── Service (+ optional LoadBalancer) und/oder Gateway+TLSRoute (ingress.type)
├── NetworkPolicy
└── VMPodScrape (optional, Host VictoriaMetrics)

• Gesamter data_dir (/var/lib/rancher/k3s) liegt auf der PVC — inkl. server/tls/, etcd, kubeconfig.
• Leaf-Certs: k3s Auto-Renew beim Start (<120 Tage Restlaufzeit). CA: 10 Jahre, manuell rotate-ca.
• Sidecar hält <name>-kubeconfig und <name>-metrics-tls nach kubeconfig-Änderungen aktuell (Polling 30s, falls kein inotify).
• block.labels werden auf Chart-Ressourcen und Playbook-Secrets/NetworkPolicy gesetzt (nicht in matchLabels).

Features

• agent-less k3s (Agent, Flannel, Traefik, ServiceLB, kube-proxy, CCM disabled; CoreDNS + metrics-server bleiben an)
• lokales Chart, Recreate-Strategy (RWO PVC)
• ein Token (k3s.token) → Secret <name>-token → K3S_TOKEN = K3S_AGENT_TOKEN
• Air-gap: image.*, pullSecrets, sidecar.image.*
• Placement: nodeSelector, tolerations, affinity
• Ready-Gates: rollout status, Deployment Available, kubeconfig Secret, /readyz
• NetworkPolicy (API + Metrics-Scrape)
• Probes (/readyz, /livez)
• Actions: install / uninstall / status / benchmark
• optionale Metrics + VMPodScrape im Block-Namespace
• optionaler kubeconfig-Export nach artifacts/secrets/<block.name>/
• API Audit Logging (Default an) + Sidecar audit-log → VictoriaLogs
• Edge exposure: ingress.type=nginx|gateway oder dedicated loadbalancer (mutual exclusive)
• Spec 676: k3s.cluster_domain → API host api.<cluster_domain>

Voraussetzungen

• Host-Cluster mit StorageClass (z.B. ceph-block) wenn PVC neu angelegt wird
• Optional: Cilium/MetalLB für loadbalancer.mode=cluster + feste IP
• Optional: VictoriaMetrics Operator im Host für metrics.enabled=true
• block.config.k3s.token ≥ 10 Zeichen (fail-loud, kein Default)

Quick Start (Envoy Gateway edge — recommended for new MCP)

Requires platform block envoy with default_gateway.enabled + shared VIP (see polycrate block examples envoy).

# workspace.poly
blocks:
  - name: k3s-server
    from: cargo.ayedo.cloud/ayedo/k8s/k3s-server:0.3.0
    kubeconfig:
      from: k8s
    config:
      namespace: k3s-cp
      persistence:
        storage_class: ceph-block
      loadbalancer:
        enabled: false
      ingress:
        enabled: true
        type: gateway
      k3s:
        # Spec 676: {workspace}.k8s.{zone} → API https://api.<cluster_domain>:443
        cluster_domain: "my-ws.k8s.example.com"
        token: "replace-me-token-min-10-chars"

polycrate run k3s-server install
polycrate run k3s-server status
polycrate block examples k3s-server

DNS: api.my-ws.k8s.example.com → Envoy shared VIP.

Token (fail-loud)

k3s:
  token: "replace-me-token-min-10"   # required, ≥10 chars — no defaults
  # agent_token: deprecated — if set, must equal token

--disable-agent knipst nur die Agent-Funktionalität im CP-Pod aus. Externe Worker joinen weiterhin mit demselben Token (K3S_AGENT_TOKEN = K3S_TOKEN).

Resources & PVC

resources:
  requests:
    cpu: "500m"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 8Gi
persistence:
  enabled: true
  storage_class: "ceph-block"   # required when creating PVC (no silent default)
  size: 20Gi
  existing_claim: ""            # optional; required if enabled: false

persistence.enabled=false verlangt existing_claim. Uninstall löscht die PVC (Cluster-Zustand weg).

Air-gap / Images

image:
  repository: registry.example.corp/rancher/k3s
  tag: ""                 # empty → v{{ app_version }}-k3s1
  pullPolicy: IfNotPresent
  pullSecrets:
    - name: regcred
sidecar:
  enabled: true
  image:
    repository: registry.example.corp/library/base
    tag: c87bdabe

Datastore

k3s:
  datastore:
    type: etcd          # etcd | sqlite | external
    existingSecret: ""
    secretKey: endpoint
    endpoint: ""
  etcd:
    auto_cluster_reset: true   # only for single-pod embedded etcd

API Exposure (mutual exclusive)

Exactly one path: edge (ingress.enabled) or dedicated LB (loadbalancer.enabled).

Cluster domain + API host (Spec 676)

k3s:
  cluster_domain: "my-ws.k8s.example.com"   # also k3s --cluster-domain

When ingress.hostname / loadbalancer.hostname are empty and cluster_domain ≠ cluster.local:

Advertise address (worker remotedialer)

k3s lists --advertise-address:--advertise-port as the kubernetes Service endpoints. Agents open wss://<addr>:<port>/v1-k3s/connect there. Default (unset) is the pod IP:6443, which workers cannot reach.

k3s:
  advertise_address: "203.0.113.20"   # Cluster API VIP, or a resolvable hostname
  advertise_port: 443                 # 0 = ingress.port (edge) or loadbalancer.port

Do not set --node-ip to the VIP — embedded etcd then binds that address and fails inside the pod.

k3s --advertise-address is IP-only; a hostname is resolved at install. Remotedialer therefore dials wss://<VIP>:<port>/…. Dedicated LB on :6443 has no SNI hostname match. If loadbalancer.ip is empty, install waits for the Service ingress IP (Cilium IPAM or hcloud-ccm) — no cloud-specific API.

Edge: Envoy Gateway (ingress.type=gateway)

loadbalancer:
  enabled: false
ingress:
  enabled: true
  type: gateway
  # Defaults attach TLSRoute to platform Gateway:
  #   parent_gateway_name: eg-shared
  #   parent_gateway_namespace: envoy-gateway-system
  #   parent_gateway_section_name: tls
  # Own Gateway instead: parent_gateway_name: "" + gateway_class_name: eg-shared|eg-dedicated

Host NetworkPolicy allows :6443 from envoy-gateway-system (or ingress.controller_namespace).

Edge: nginx Ingress (ingress.type=nginx)

ingress:
  enabled: true
  type: nginx
  class_name: nginx

Requires nginx with enable_ssl_passthrough. Whitelist: api_access.whitelist_cidrs → annotation.

Dedicated LoadBalancer

ingress:
  enabled: false
loadbalancer:
  enabled: true
  mode: cluster        # cluster | external
  ip: ""
  hostname: ""         # or rely on api.<cluster_domain>
  provider: cilium     # cilium | metallb | both
  port: 6443

• mode=cluster: Service LoadBalancer + Cilium/MetalLB annotations; IP/Hostname → --tls-san
• mode=external: no cluster LB Service; address only for SAN / kubeconfig
• Host NetworkPolicy: whitelist CIDRs on :6443 when set via api_access.whitelist_cidrs

NetworkPolicy

network_policy:
  enabled: true
  metrics_scrape_namespaces:
    - victoria-metrics-stack
# Prefer top-level api_access.whitelist_cidrs (applied to active exposure path).
api_access:
  whitelist_cidrs: []

Metrics (Host VictoriaMetrics)

metrics:
  enabled: false
  etcd: true                 # HTTP :2381
  apiserver: true            # HTTPS :6443/metrics — braucht Client-Cert
  controller_manager: true   # HTTPS :10257
  scheduler: true            # HTTPS :10259
  insecure_skip_verify: true # nur Server-Verify; Client-Auth bleibt
  vmpodscrape:
    enabled: true
    namespace: ""            # Default = Block-NS (für Sidecar-Sync nötig)

HTTPS-Endpoints brauchen das MCP-Admin-Client-Zertifikat. install legt Secret <block.name>-metrics-tls und die VMPodScrape im Block-Namespace an. Der Sidecar aktualisiert das Secret fortlaufend aus der kubeconfig — Leaf-Cert-Rotation bricht Scrapes nicht.

Voraussetzung für Sync: metrics.vmpodscrape.namespace = Block-Namespace (Default). Andere Namespaces → nur einmaliger Sync bei install (Sidecar-Role ist namespaced).

Agent-less + Scheduler-Metrics: k3s.disable.cloud_controller: true (Default).

Ohne Worker-Nodes: CoreDNS und metrics-server Deployments bleiben Pending — erwartet, bis Agents joinen. Die APIService metrics.k8s.io kann in diesem Fenster Namespace-Löschungen blockieren (NamespaceDeletionDiscoveryFailure). Mit Nodes (oder nach APIService-Recovery) ist das weg. kube-state-metrics ist unabhängig von metrics-server.

OIDC

k3s:
  oidc:
    enabled: true
    issuer_url: "https://id.example.com/realms/kubernetes"
    client_id: "k3s-server"
    username_claim: email
    groups_claim: groups

Bei export_kubeconfig.enabled schreibt install zusätzlich kubeconfig-oidc.yml (ohne Client-Cert, mit OIDC-User).

Zertifikate & Rotation

Manuell (Leaf):

kubectl -n <ns> exec deploy/<block> -c controlplane -- \
  /bin/k3s certificate check --data-dir=/var/lib/rancher/k3s --output table
kubectl -n <ns> exec deploy/<block> -c controlplane -- \
  /bin/k3s certificate rotate --data-dir=/var/lib/rancher/k3s
kubectl -n <ns> rollout restart deploy/<block>

Worker Join

curl -sfL https://get.k3s.io | K3S_URL=https://<api-host>:6443 K3S_TOKEN=<token> sh -s - \
  --kubelet-arg "cloud-provider=external" \
  --node-label "role=worker"

<token> = block.config.k3s.token.

Log level

k3s:
  log_level: "0"   # → k3s server -v=<n>
  debug: false

API Audit Logging

Default on (k3s.audit.enabled: true). kube-apiserver writes NDJSON; container audit-log tails to stdout. Host Vector → VictoriaLogs.

k3s:
  audit:
    enabled: true
    # Default Policy: Metadata-only. Optional Full-YAML Override:
    # policy: |
    #   apiVersion: audit.k8s.io/v1
    #   kind: Policy
    #   rules:
    #     - level: Metadata
audit_sidecar:
  enabled: true   # effective only when k3s.audit.enabled

• Policy-ConfigMap: <block.name>-audit-policy
• Log-Volume: emptyDir (not PVC); retention via VictoriaLogs
• Container name: audit-log

Query via CLI (workspace org LogsQL passthrough):

polycrate logs --block k3s-server --start 15m \
  --query 'kubernetes.container_name:"audit-log"'
polycrate logs --start 15m --format json \
  --query 'polycrate_block_name:"k3s-server" AND kubernetes.container_name:"audit-log"'

Example: polycrate block examples k3s-server → with-api-audit.

Actions

install

polycrate run k3s-server install

Validiert Token, storage_class, OIDC, … → Token-Secret → Jinja render → kubectl apply → rollout → kubeconfig Secret → /readyz → NetworkPolicy / Metrics / optional Export.

status

polycrate run k3s-server status

Ready = Namespace + Token-Secret + Deployment Available + kubeconfig Secret + /readyz (Exit 0/1/2).

uninstall

polycrate run k3s-server uninstall

Entfernt gerenderte Manifeste (plus Legacy-Helm-Release falls noch vorhanden), Token-/kubeconfig-/metrics-tls-Secrets, NetworkPolicy, VMPodScrape, PVC, Rest-Pods, Export-Artefakte.

Namespace wird nicht gelöscht (delete_namespace: false, Default). Opt-in nur für dedizierte Wegwerf-Namespaces:

config:
  delete_namespace: true

Managed Controlplane nutzt den Org-Slug als Namespace — dort darf delete_namespace nie true sein.

Secrets encryption

Default: --secrets-encryption mit Provider secretbox (Key auf der PVC unter server/cred/).

k3s:
  secrets_encryption:
    enabled: true
    provider: secretbox   # aescbc | secretbox

Lifecycle (idempotent): secrets-encryption-status | enable | disable | rotate.
Nach Enable/Disable k3s.secrets_encryption.enabled in der Workspace-Config anpassen, damit der nächste install nicht zurückkippt.

benchmark

polycrate run k3s-server benchmark

Roh-API Latency/QPS (ohne Worker). Reports unter artifacts/blocks/<name>/benchmarks/<run_id>/.

benchmark:
  profile: smoke   # smoke | stress
  iterations: 50
  concurrency: 4
  timeout_seconds: 30
  payload_bytes: 0

Beispiele (examples.poly)

polycrate block examples k3s-server
polycrate block examples k3s-server --no-pager

Troubleshooting

Changelog

Siehe CHANGELOG.poly.