cargo.ayedo.cloud/ayedo/k8s/rook

Polycrate block for deploying and managing Rook-Ceph on Kubernetes. Provides S3-compatible object storage (RGW), block storage (RBD via CSI), and shared filesystems (CephFS via CSI). Includes monitoring integration for VictoriaMetrics and an RGW metrics exporter.

Helm Charts: rook-ceph + ceph-csi-drivers + rook-ceph-cluster from https://charts.rook.io/release / https://ceph.github.io/ceph-csi-operator
Chart version: v1.20.9 / ceph-csi-drivers 1.0.4 (Ceph v20.2.4 / Tentacle default — pin cluster.ceph_image.tag in production; see "Upgrade Path")
Kubernetes minimum: v1.31
Ceph minimum: v19.2.0
Block version: 0.8.1

Architecture

The block is split into three independent instances in workspace.poly — operator, CSI drivers, cluster. This mirrors how Rook itself separates operator, CSI-driver and cluster lifecycle since chart 1.20.0:

rook-operator      →  installs rook-ceph Helm chart          (operator + ceph-csi-operator subchart)
rook-csi-drivers   →  installs ceph-csi-drivers Helm chart    (RBD/CephFS Driver CRs — REQUIRED since 1.20.0)
rook-cluster       →  installs rook-ceph-cluster Helm chart   (CephCluster CRD + pools + objectstore)

All three instances use the same block (cargo.ayedo.cloud/ayedo/k8s/rook) but activate different parts via operator.enabled, csi_drivers.enabled and cluster.enabled. Install order matters and is Rook-mandated: rook-operator → rook-csi-drivers → rook-cluster (declare them in this order in workspace.poly).

Helm release naming: the operator and cluster Helm releases are named after the instance (block.name, e.g. rook-operator, rook-cluster). The ceph-csi-drivers release is always named <instance-name>-csi-drivers instead — operator, csi_drivers and cluster all default to the same namespace (rook-ceph), so a bare block.name release for csi_drivers would collide with (and overwrite) the operator's Helm release if both were ever enabled on one instance instead of split as shown above.

Actions

polycrate run rook-operator install
polycrate run rook-csi-drivers install
polycrate run rook-cluster install
polycrate run rook-cluster status
polycrate run rook-cluster render

render action

Renders all non-Helm Kubernetes manifests (VMServiceScrape, VMPodScrape, Ingress, Exporter, Buckets) to:

<workspace>/<artifactsroot>/secrets/<block-name>/manifests/

The secrets/ path is intentional — rendered manifests may contain sensitive values. This directory must be excluded from git (add to .gitignore). The render action is primarily used for ArgoCD integration where manifests are generated ephemerally and applied directly.

Never commit the rendered manifests to git. Helm values and cluster config are managed via workspace.poly — the rendered output is ephemeral.

Minimal Setup (workspace.poly)

- name: rook-operator
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    operator:
      enabled: true

- name: rook-csi-drivers
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    csi_drivers:
      enabled: true

- name: rook-cluster
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    object_store:
      hosting:
        dns_names:
          - s3.example.com
    cluster:
      enabled: true
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: true

Configuration Reference

operator.*

csi_drivers.* (since 0.7.0 / chart 1.20.0)

Since Rook 1.20.0, CSI drivers (RBD/CephFS) are no longer deployed by the rook-ceph chart itself — a separate, required ceph-csi-drivers chart configures them via Driver/OperatorConfig CRs, reconciled by the ceph-csi-operator (bundled as a subchart of operator). Must be installed after operator, before cluster.

There is no single "disable all CSI drivers" toggle anymore (old operator.csi.csi_driver.disabled) — disable drivers.rbd.enabled and drivers.cephfs.enabled individually if S3-only (see Scenario H).

cluster.*

object_store.*

exporter.*

The RGW exporter collects per-bucket S3 metrics (request counts, bytes transferred, user quotas).

buckets[]

Pre-create S3 buckets via ObjectBucketClaims:

buckets:
  - name: my-bucket
    storage_class_name: ceph-bucket
    namespace: my-app
    additional_config: {}

OSD Device Selection

Rook needs raw, unformatted block devices for OSDs. Devices with existing partitions or filesystems are skipped automatically.

Option 1: All available devices on all nodes

cluster:
  spec:
    storage:
      use_all_nodes: true
      use_all_devices: true

Option 2: Regex filter (recommended for production)

Select specific devices by name pattern. The filter is a Go regex applied to the device name (without /dev/).

cluster:
  spec:
    storage:
      use_all_nodes: true
      use_all_devices: false
      device_filter: "^sd[b-z]"      # all sdb, sdc, ... sdz
      # device_filter: "^nvme[0-9]n1" # all NVMe drives
      # device_filter: "^sd[c-i]"     # sdc through sdi

Option 3: BlueStore DB/WAL on a separate device

BlueStore separates the block data (slow HDD) from metadata (DB) and write-ahead-log (WAL). Putting DB+WAL on an SSD or NVMe dramatically improves random write latency even in an HDD-only cluster.

If you have a partition available on a faster device (e.g. carved from an OS disk or a dedicated SSD), configure:

cluster:
  spec:
    storage:
      use_all_nodes: true
      use_all_devices: false
      device_filter: "^sd[c-z]"
      metadata_device: /dev/sdb1    # pre-partitioned, same path on ALL OSD nodes

Prerequisites: The partition must exist on every OSD node before running install. Rook does not create the partition — use your OS provisioning tooling (Ansible, cloud-init) to create it first.

Sizing: Target ~4GB per OSD for the DB partition. A single 128GB partition can serve ~30 OSDs comfortably.

Storage Backends

S3 Object Storage (RGW)

Always deployed. The block creates a single CephObjectStore named ceph-objectstore with: - Metadata pool: replicated 3x - Data pool: erasure coded 2+1 (2 data chunks + 1 coding chunk) - StorageClass: ceph-bucket (for ObjectBucketClaims)

Configure the public endpoint and gateway redundancy:

object_store:
  gateway:
    instances: 2       # ≥2 for HA
  hosting:
    dns_names:
      - s3.example.com
      - objects.example.com

To publish RGW without an Ingress, turn TLS and the LoadBalancer Service on and leave cluster.ingress.enabled off. Use external_ips where Keepalived already holds the address, or loadbalancer_ip where Cilium LB-IPAM (or MetalLB) assigns it. Do not set both. The public port stays 443.

With external_traffic_policy: Local, only nodes that run an RGW pod accept traffic. Pin those pods with placement.node_selector to every node that can own the address. The client TCP session then ends on the RGW pod, so the node sysctls apply: net.ipv4.tcp_wmem should be 4096 1048576 16777216 (1 MB default, 16 MB max) and net.ipv4.tcp_congestion_control should be bbr. The kernel default 4096 16384 4194304 caps a single stream at high RTT. Pods pick up sysctls only when they start. The block does not set them; the host tcp-tuning does. Confirm them inside a fresh RGW pod before shifting the address off the Ingress.

RBD Block Storage (Kubernetes PVCs)

Add one or more block pools. Each pool gets its own StorageClass.

cluster:
  block_pools:
    - name: ceph-blockpool
      failure_domain: host         # data replicated across different hosts
      replicated:
        size: 3                    # 3 replicas
      storage_class:
        enabled: true
        name: ceph-block           # StorageClass name
        is_default: false          # set true to make default StorageClass
        reclaim_policy: Delete
        allow_volume_expansion: true
        fstype: ext4               # ext4 or xfs

failure_domain values: host (default), rack, zone, osd

CephFS Shared Filesystem (RWX volumes)

CephFS supports ReadWriteMany — multiple pods can mount the same volume simultaneously.

cluster:
  file_systems:
    - name: ceph-filesystem
      failure_domain: host
      metadata_pool:
        replicated:
          size: 3
      data_pools:
        - name: data0
          failure_domain: host
          replicated:
            size: 3
      mds:
        active_count: 1      # active MDS daemons
        active_standby: true # keep a standby MDS for fast failover
      storage_class:
        enabled: true
        name: ceph-filesystem
        is_default: false
        reclaim_policy: Delete
        allow_volume_expansion: true

Setup Scenarios

Scenario A: 3-Node HDD Cluster (S3 + Block Storage)

Minimum viable production setup. All three nodes are both OSD hosts and control plane nodes.

Requirements: - 3 nodes, each with ≥1 dedicated data disk (e.g. /dev/sdb) - 10G network between nodes recommended

- name: rook-operator
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    operator:
      enabled: true
      discovery_daemon:
        enabled: true
      monitoring:
        vmpodscrape:
          enabled: true

- name: rook-csi-drivers
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    csi_drivers:
      enabled: true

- name: rook-cluster
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    object_store:
      gateway:
        instances: 2
      hosting:
        dns_names:
          - s3.example.com
    cluster:
      enabled: true
      toolbox:
        enabled: true
      monitoring:
        enabled: true
        vmservicescrape:
          enabled: true
        csi:
          vmpodscrape:
            enabled: true
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^sdb"
      ceph_config:
        global:
          osd_recovery_sleep_hdd: "0.1"
          osd_scrub_sleep: "0.1"
          osd_memory_target: "4294967296"
      block_pools:
        - name: ceph-blockpool
          failure_domain: host
          replicated:
            size: 3
          storage_class:
            name: ceph-block
            is_default: true
            fstype: ext4

Scenario B: 3-Node HDD Cluster with RGW Usage Metrics

Extends Scenario A with per-bucket S3 metrics (requires RGW usage logging enabled in Ceph).

    cluster:
      ceph_config:
        global:
          osd_recovery_sleep_hdd: "0.1"
          osd_scrub_sleep: "0.1"
          osd_memory_target: "4294967296"
        "client.rgw":
          rgw_enable_usage_log: "true"
          rgw_usage_log_tick_interval: "30"
          rgw_usage_log_flush_threshold: "1024"
    exporter:
      region: "fra1"
      pub_endpoint: "https://s3.example.com"
      vmservicescrape:
        enabled: true

Scenario C: 10-Node HDD Cluster (High Availability)

More OSD nodes allow using rack as failure domain (requires nodes spread across ≥3 racks). Increase replication and use erasure coding for data pools.

    cluster:
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^sd[b-h]"   # up to 7 OSDs per node
      block_pools:
        - name: ceph-blockpool
          failure_domain: rack         # tolerate full rack loss
          replicated:
            size: 3
          storage_class:
            name: ceph-block
            fstype: ext4
      ceph_config:
        global:
          osd_recovery_sleep_hdd: "0.1"
          osd_scrub_sleep: "0.05"
          osd_memory_target: "4294967296"

Scenario D: SSD/NVMe Cluster

For all-flash clusters, disable recovery sleep throttling and use xfs for better NVMe performance.

    cluster:
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^nvme[0-9]n1"
      block_pools:
        - name: ceph-blockpool
          failure_domain: host
          replicated:
            size: 3
          storage_class:
            name: ceph-block
            is_default: true
            fstype: xfs              # xfs performs better on NVMe
      ceph_config:
        global:
          osd_recovery_sleep_ssd: "0.0"   # no throttling on SSDs
          osd_scrub_sleep: "0.0"
          osd_memory_target: "4294967296"

Scenario E: Mixed HDD+SSD Cluster (BlueStore DB/WAL separation)

HDDs for data, SSD partition for BlueStore DB+WAL. This dramatically improves random write latency.

Pre-requisite: Create the DB partition on all OSD nodes before running install:

# On each OSD node: create a 128G partition on the SSD
parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary 0% 100%
# Result: /dev/sdb1 available as metadata device

    cluster:
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^sd[c-z]"   # data OSDs on HDDs
          metadata_device: /dev/sdb1  # SSD partition for DB+WAL
      ceph_config:
        global:
          osd_recovery_sleep_hdd: "0.1"
          osd_scrub_sleep: "0.1"
          osd_memory_target: "4294967296"

Scenario F: External Ceph Cluster (Consumer Mode)

Rook can operate as a consumer of an existing, externally managed Ceph cluster (e.g. managed by cephadm). In this mode, Rook provides the CSI drivers and Kubernetes integration but does not manage OSDs or Ceph daemons.

How it works

The block replaces the import-external-cluster.sh script entirely. You provide the credentials from create-external-cluster-resources.py directly in workspace.poly, and the block creates all required Kubernetes resources in the correct order.

Why this matters for CSI (important): The Rook CSI driver uses two different secrets depending on the operation: - Attach (ControllerPublishVolume): uses the secret referenced in the StorageClass parameters - Detach (ControllerUnpublishVolume): uses controllerPublishSecretRef from the ceph-csi-config ConfigMap — always the standard short names rook-csi-rbd-provisioner / rook-csi-cephfs-provisioner

If these standard-named secrets don't exist (e.g. because import-external-cluster.sh was run with --restricted-auth-permission, which produces long non-standard names), volume detach fails silently. The block always creates the standard-named secrets and a StorageClass that includes controller-publish-secret-name for correct fencing/detach behaviour.

Step 1: Generate credentials on the external Ceph cluster

Run create-external-cluster-resources.py on the provider Ceph cluster:

python3 create-external-cluster-resources.py \
  --format bash \
  --rbd-data-pool-name <your-rbd-pool> \
  --cephfs-filesystem-name <your-fs> \   # omit if no CephFS
  --skip-monitoring-endpoint

Note on --restricted-auth-permission: You may use this flag. The script-produced CSI_*_SECRET_NAME values (which are the CephX usernames) will be longer (e.g. csi-rbd-provisioner-<cluster>-<pool>). Use those exact values as user_id in the block config. The Kubernetes secret name is always the standard short form (rook-csi-rbd-provisioner) — the block enforces this regardless of user_id.

Step 2: Configure the block

- name: rook-operator
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    operator:
      enabled: true
      discovery_daemon:
        enabled: true
      monitoring:
        vmpodscrape:
          enabled: true

- name: rook-csi-drivers
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    csi_drivers:
      enabled: true

- name: rook-cluster
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    cluster:
      enabled: true
      spec:
        external:
          enabled: true       # consumer mode: no local OSDs
        crash_collector:
          enabled: false      # no local daemons to monitor
        storage:
          use_all_nodes: false
          use_all_devices: false
      external_cluster:
        fsid: "<ROOK_EXTERNAL_FSID>"
        mon_data: "<mon-a=ip1:6789,mon-b=ip2:6789,...>"   # ROOK_EXTERNAL_CEPH_MON_DATA
        external_username: "<ROOK_EXTERNAL_USERNAME>"
        user_secret: "<ROOK_EXTERNAL_USER_SECRET>"
        rbd_pool_name: "<RBD_POOL_NAME>"
        rgw_pool_prefix: "default"                         # RGW_POOL_PREFIX

        # StorageClass for the external RBD pool (replaces bash script StorageClass creation)
        storage_class:
          enabled: true
          name: ceph-block
          is_default: true
          reclaim_policy: Delete
          allow_volume_expansion: true
          fstype: ext4

        # Optional: RGW admin ops user (only if S3/RGW is used)
        # rgw_admin_ops_user:
        #   access_key: "<RGW_ADMIN_OPS_USER_ACCESS_KEY>"
        #   secret_key: "<RGW_ADMIN_OPS_USER_SECRET_KEY>"

        csi_secrets:
          rbd_provisioner:
            user_id: "<CSI_RBD_PROVISIONER_SECRET_NAME>"  # CephX username; same value as secret name suffix
            user_key: "<CSI_RBD_PROVISIONER_SECRET>"
          rbd_node:
            user_id: "<CSI_RBD_NODE_SECRET_NAME>"
            user_key: "<CSI_RBD_NODE_SECRET>"
          cephfs_provisioner:
            user_id: ""   # CSI_CEPHFS_PROVISIONER_SECRET_NAME — leave empty if no CephFS
            user_key: ""  # CSI_CEPHFS_PROVISIONER_SECRET
          cephfs_node:
            user_id: ""   # CSI_CEPHFS_NODE_SECRET_NAME — leave empty if no CephFS
            user_key: ""  # CSI_CEPHFS_NODE_SECRET

Credential mapping: script output → block config

user_id: The Python script sets CSI_*_SECRET_NAME as the CephX username and simultaneously uses it as the Kubernetes secret name suffix. The block stores this value as the userID field inside the K8s secret. The K8s secret name is always the standard short form (rook-csi-rbd-provisioner) — enforced by the block regardless of the user_id value.

What the block creates

Before the Helm chart:

1. rook-ceph-mon Secret — operator credentials (from external_username + user_secret)
2. rook-ceph-mon-endpoints ConfigMap — monitor addresses (from mon_data)
3. rgw-admin-ops-user Secret — optional, only if rgw_admin_ops_user.access_key is set

After the Helm chart:

4. rook-csi-rbd-provisioner Secret — required for RBD CSI attach and detach
5. rook-csi-rbd-node Secret — required for RBD CSI node stage operations
6. rook-csi-cephfs-provisioner Secret — optional, only if cephfs_provisioner.user_key is set
7. rook-csi-cephfs-node Secret — optional, only if cephfs_node.user_key is set
8. StorageClass (external_cluster.storage_class.name) — RBD StorageClass pointing to rbd_pool_name, with all CSI secret refs including controller-publish-secret-name for correct detach

No block_pools needed for external clusters. The StorageClass is created directly from external_cluster.rbd_pool_name + external_cluster.storage_class. block_pools is for internal clusters only.

Scenario F1: External cluster, RBD only (no CephFS)

      external_cluster:
        fsid: "<ROOK_EXTERNAL_FSID>"
        mon_data: "<ROOK_EXTERNAL_CEPH_MON_DATA>"
        external_username: "<ROOK_EXTERNAL_USERNAME>"
        user_secret: "<ROOK_EXTERNAL_USER_SECRET>"
        rbd_pool_name: "<RBD_POOL_NAME>"
        storage_class:
          enabled: true
          name: ceph-block
          is_default: true
          fstype: ext4
        csi_secrets:
          rbd_provisioner:
            user_id: "<CSI_RBD_PROVISIONER_SECRET_NAME>"
            user_key: "<CSI_RBD_PROVISIONER_SECRET>"
          rbd_node:
            user_id: "<CSI_RBD_NODE_SECRET_NAME>"
            user_key: "<CSI_RBD_NODE_SECRET>"
          cephfs_provisioner:
            user_id: ""
            user_key: ""
          cephfs_node:
            user_id: ""
            user_key: ""

Scenario F2: External cluster with RBD + CephFS + RGW

      external_cluster:
        fsid: "<ROOK_EXTERNAL_FSID>"
        mon_data: "<ROOK_EXTERNAL_CEPH_MON_DATA>"
        external_username: "<ROOK_EXTERNAL_USERNAME>"
        user_secret: "<ROOK_EXTERNAL_USER_SECRET>"
        rbd_pool_name: "<RBD_POOL_NAME>"
        rgw_pool_prefix: "default"
        storage_class:
          enabled: true
          name: ceph-block
          is_default: true
          fstype: ext4
        rgw_admin_ops_user:
          access_key: "<RGW_ADMIN_OPS_USER_ACCESS_KEY>"
          secret_key: "<RGW_ADMIN_OPS_USER_SECRET_KEY>"
        csi_secrets:
          rbd_provisioner:
            user_id: "<CSI_RBD_PROVISIONER_SECRET_NAME>"
            user_key: "<CSI_RBD_PROVISIONER_SECRET>"
          rbd_node:
            user_id: "<CSI_RBD_NODE_SECRET_NAME>"
            user_key: "<CSI_RBD_NODE_SECRET>"
          cephfs_provisioner:
            user_id: "<CSI_CEPHFS_PROVISIONER_SECRET_NAME>"
            user_key: "<CSI_CEPHFS_PROVISIONER_SECRET>"
          cephfs_node:
            user_id: "<CSI_CEPHFS_NODE_SECRET_NAME>"
            user_key: "<CSI_CEPHFS_NODE_SECRET>"

Scenario G: Single-Node Cluster

For development, edge, or lab setups where only one Kubernetes node is available. Several Rook defaults are tuned for multi-node HA and must be reduced for a single-node deployment:

Note: A single-node cluster has no redundancy at any level. Any node or daemon failure results in downtime. Suitable for non-production use only.

- name: rook-operator
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    operator:
      enabled: true

- name: rook-csi-drivers
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    csi_drivers:
      enabled: true

- name: rook-cluster
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    object_store:
      metadata_pool:
        failure_domain: osd
        replicated:
          size: 1
      data_pool:
        failure_domain: osd
        type: replicated
        replicated:
          size: 1
      gateway:
        instances: 1
      hosting:
        dns_names:
          - s3.example.com
    cluster:
      enabled: true
      pools_default:
        size: "1"        # sets osd_pool_default_size — affects .mgr, device_health_metrics, etc.
        min_size: "1"    # sets osd_pool_default_min_size
      spec:
        mon:
          count: 1
          allow_multiple_per_node: false
        mgr:
          count: 1
          allow_multiple_per_node: false
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^sdb"
      block_pools:
        - name: ceph-blockpool
          failure_domain: osd        # single node: osd, not host
          replicated:
            size: 1                  # no cross-host replication on single node
          storage_class:
            name: ceph-block
            is_default: true
            fstype: ext4

Scenario H: S3-Only Cluster (No Block/CephFS)

Minimal setup for backup storage or S3-compatible object storage only. No RBD or CephFS drivers needed.

- name: rook-operator
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    operator:
      enabled: true

- name: rook-csi-drivers
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    csi_drivers:
      enabled: true
      drivers:
        rbd:
          enabled: false        # not needed for S3-only
        cephfs:
          enabled: false        # not needed for S3-only

- name: rook-cluster
  from: cargo.ayedo.cloud/ayedo/k8s/rook:0.8.0
  kubeconfig:
    from: k8s
  config:
    object_store:
      gateway:
        instances: 2
      hosting:
        dns_names:
          - s3.example.com
    cluster:
      enabled: true
      spec:
        storage:
          use_all_nodes: true
          use_all_devices: false
          device_filter: "^sdb"
      # block_pools: []   (default, no RBD)
      # file_systems: []  (default, no CephFS)

ceph_config — Raw Ceph Configuration

cluster.ceph_config is passed directly to cephClusterSpec.cephConfig in the Rook CRD. This gives full control over all Ceph daemon configuration without any abstraction.

Format:

cluster:
  ceph_config:
    <section>:
      <key>: "<value>"   # ALL values must be strings

Sections map to Ceph config sections:

Important: Section names containing dots (like client.rgw) must be quoted in YAML.

All values must be strings — wrap numbers and booleans in quotes:

ceph_config:
  global:
    osd_memory_target: "4294967296"   # correct: string
    osd_memory_target: 4294967296     # wrong: integer

Useful ceph_config Keys

OSD performance (HDD)

OSD performance (SSD/NVMe)

RGW (S3) usage logging — required for per-bucket request/throughput metrics

Full list of available keys: https://docs.ceph.com/en/latest/rados/configuration/

Monitoring

The block integrates with VictoriaMetrics Operator and deploys VMServiceScrape and VMPodScrape resources. No Prometheus ServiceMonitor objects are created.

cluster.monitoring.enabled must be true for the MGR and Ceph Exporter VMServiceScrapes to work — this flag controls whether the chart creates the Kubernetes Service that exposes MGR metrics.

Full monitoring setup:

- name: rook-operator
  config:
    operator:
      monitoring:
        vmpodscrape:
          enabled: true

- name: rook-cluster
  config:
    cluster:
      monitoring:
        enabled: true
        vmservicescrape:
          enabled: true
        csi:
          vmpodscrape:
            enabled: true   # only useful if block_pools or file_systems are configured
    exporter:
      vmservicescrape:
        enabled: true

Ingress (RGW)

Expose the S3 endpoint via nginx Ingress:

- name: rook-cluster
  config:
    cluster:
      ingress:
        enabled: true
        class: nginx
        wildcard:
          enabled: true           # create wildcard DNS entry
        base_host: "s3.example.com"
        hosts: []
        tls:
          enabled: true
          issuer: letsencrypt-production

Upgrade Path

Rook-Ceph requires stepwise upgrades — never skip a minor version.

The install playbook automatically handles StorageClass immutability issues introduced in Rook v1.18: before upgrading, it detects and deletes any CSI StorageClasses that are missing required fencing parameters. These are recreated correctly by the Helm upgrade. Existing PVCs and PVs are not affected.

Upgrading an existing cluster from 0.6.x (chart 1.19.x) to 0.7.0 (chart 1.20.1)

Full analysis: .specs/2-rook-helm-chart-update-1-19-2-1-20-1-breaking-changes-migrationspfad.md.

1. Pre-flight: cluster must already be on Rook chart >= 1.19.5 (a critical Helm-ownership fix required for the 1.20 migration shipped there) and HEALTH_OK with no down/recovering/ undersized/backfilling PGs.
2. Decouple the Ceph major upgrade: unless you deliberately want to jump from Ceph Squid (v19.2.3) to Tentacle (v20.2.2, the new chart default) in the same step, set cluster.ceph_image.tag: v19.2.3 explicitly before upgrading the block. Ceph major upgrades end with an irreversible ceph osd require-osd-release tentacle — treat it as its own, separate, deliberate change.
3. Add the new rook-csi-drivers instance to workspace.poly (see "Architecture" above), positioned between the existing rook-operator and rook-cluster instances.
4. Migrate CSI toggles: if you previously set operator.csi.rdb_driver.disabled / .cephfs_driver.disabled / .csi_driver.disabled, move the (inverted) values to csi_drivers.drivers.rbd.enabled / .cephfs.enabled (see "Configuration Reference" above). If you set operator.csi.discovery_daemon.enabled, move it to operator.discovery_daemon.enabled (unrelated rename, not CSI-specific).
5. Run polycrate run rook-operator install, then polycrate run rook-csi-drivers install, then polycrate run rook-cluster install — in that exact order. Do not skip rook-csi-drivers install even if that Helm release is already deployed (operator-only catch-up). After the 1.20 CSI operator starts, Driver CRs without spec.*.serviceAccountName default to short SAs (rbd-nodeplugin-sa) that the ceph-csi-drivers chart never creates — ctrl/node plugins fail to schedule. Re-running the drivers chart writes the long names (rook-ceph-rbd-csi-ceph-com-*-sa). The operator playbook adopts rook-csi-operator-image-set-configmap when its Helm release-name is still the upstream default rook-ceph (this block releases the operator as rook-operator). Without that adopt, Helm 1.20.1 refuses the upgrade (rook#17696).
6. Verify CSI metrics did not silently break: the cluster.monitoring.csi.vmpodscrape selector changed (new ceph-csi-operator pod labels, <driver-name>-ctrlplugin/ -nodeplugin) — confirm kubectl -n rook-ceph get pods -l app=rook-ceph.rbd.csi.ceph.com-nodeplugin returns pods before trusting CSI dashboards again.

Patch upgrade 0.8.0 (chart 1.20.1) to 0.8.1 (chart v1.20.9)

Patch line inside 1.20.x. Reversible with Helm rollback as long as cluster.ceph_image.tag has not moved to Tentacle. Required before Ceph 20.2.4: Rook 1.20.1 cannot rotate daemon keys to aes256k (CVE-2025-30156).

1. Pull cargo.ayedo.cloud/ayedo/k8s/rook:0.8.1. Defaults: mgr.modules.rook.enabled: false, cephx.daemon.key_rotation_policy: Disabled, charts v1.20.9 / csi-drivers 1.0.4.
2. Keep the existing cluster.ceph_image.tag pin (e.g. v19.2.3).
3. polycrate run rook-operator install, then rook-csi-drivers install, then rook-cluster install. Operator + CSI first — the cluster chart then writes mgr.modules.rook.enabled: false.
4. Gate: every deployment rook-version=v1.20.9, ceph versions still on the pin, HEALTH_OK or WARN only, CSI PVs Bound, RGW reachable.

Ceph major upgrade with this block (Squid → Tentacle)

Irreversible after Rook sets require_osd_release tentacle. Do not combine with the Rook patch upgrade.

1. Rook already on >= 1.20.6 (1.20.9 via this block). rook mgr module off. Cluster HEALTH_OK/WARN only, all PGs active+clean. Optional: ceph osd set noautoscale.
2. Set cluster.ceph_image.tag to a Tentacle build (v20.2.4-20260818) and run polycrate run rook-cluster install. Rook rolls mon → mgr → osd → rgw and then pins require_osd_release tentacle.
3. After ceph versions shows only 20.2.4: set cluster.spec.security.cephx.daemon.key_rotation_policy: KeyGeneration and key_generation to current status.cephx.*.keyGeneration + 1. Install again. Rook rotates daemon keys to aes256k and rolls daemons once more. Restart the toolbox. Leave CSI keys on aes unless every node is kernel >= 7.0.
4. ceph osd unset noautoscale. Mute remaining AUTH_INSECURE_* warnings that refer to client/CSI keys.

Cluster Health Verification

Kubernetes resources

# Overall pod status — all should be Running or Completed
kubectl -n rook-ceph get pods

# CephCluster CR status — Phase should be "Ready", Health "HEALTH_OK"
kubectl -n rook-ceph get cephcluster

# Detailed CephCluster status (shows mon map, MGR active, OSD count)
kubectl -n rook-ceph get cephcluster -o yaml | grep -A 30 "^status:"

# CephObjectStore status — Phase should be "Ready"
kubectl -n rook-ceph get cephobjectstore

# OSD deployments — count should match number of configured drives
kubectl -n rook-ceph get deployment -l app=rook-ceph-osd

# MGR deployments — count should match cluster.spec.mgr.count
kubectl -n rook-ceph get deployment -l app=rook-ceph-mgr

# Mon pods — count should match cluster.spec.mon.count
kubectl -n rook-ceph get pods -l app=rook-ceph-mon

Ceph CLI (via toolbox)

Enable the toolbox with cluster.toolbox.enabled: true, then:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

# Overall cluster health — target: HEALTH_OK
ceph status

# OSD tree — shows all OSDs, their status (up/in) and placement
ceph osd tree

# OSD status summary
ceph osd stat

# Disk usage per pool
ceph df

# Pool list with replication/EC info
ceph osd pool ls detail

# Monitor quorum
ceph mon stat
ceph quorum_status -f json-pretty | jq '.quorum_names'

# MGR status — shows active and standby MGRs
ceph mgr stat

# Object store / RGW status
radosgw-admin zone get
radosgw-admin bucket list
radosgw-admin bucket stats

# PG (placement group) summary — all PGs should be active+clean
ceph pg stat

# Watch cluster events in real time
ceph -w

MGR replica scale-down

After decreasing cluster.spec.mgr.count from 2 to 1, the Rook operator removes the second MGR Deployment (e.g. mgr-b) asynchronously during the next reconcile cycle. Seeing both mgr-a and mgr-b immediately after the change is expected — mgr-a will be active and mgr-b standby. The operator removes mgr-b once the cluster is healthy.

To speed up the removal manually:

# Inside the toolbox: fail the standby MGR — Rook will not recreate it when count=1
ceph mgr fail mgr-b

Check operator logs if mgr-b persists:

kubectl -n rook-ceph logs -l app=rook-ceph-operator --tail=50 | grep -i mgr

Common issue checks

# OSD not starting — check for devices with existing data/partitions
kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare --tail=50

# Operator reconcile errors
kubectl -n rook-ceph logs -l app=rook-ceph-operator --tail=100 | grep -i error

# RGW not ready — credentials secret not yet created (OSD prepare can take 10+ min)
kubectl -n rook-ceph get secret | grep object-user

# CSI driver not functional — check controller-plugin pods (since chart 1.20.x /
# ceph-csi-operator; driver name defaults to csi_drivers.drivers.{rbd,cephfs}.name)
kubectl -n rook-ceph get pods -l app=rook-ceph.rbd.csi.ceph.com-ctrlplugin
kubectl -n rook-ceph get pods -l app=rook-ceph.cephfs.csi.ceph.com-ctrlplugin

# PVC stuck in Pending — check CSI node plugin
kubectl -n rook-ceph get pods -l app=rook-ceph.rbd.csi.ceph.com-nodeplugin
kubectl describe pvc <pvc-name> -n <namespace>

# ceph-csi-drivers chart not installed — Driver/OperatorConfig CRs missing
kubectl -n rook-ceph get drivers.csi.ceph.io,operatorconfigs.csi.ceph.io

Toolbox

The Ceph toolbox provides a shell with ceph CLI access for debugging and administration:

cluster:
  toolbox:
    enabled: true

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
ceph status
ceph osd tree
ceph df
radosgw-admin bucket list

RadosGW Admin API Credential (polycrate-api)

Polycrate-API kann S3 Buckets und User über die RadosGW Admin API verwalten. Dafür braucht der S3-Cluster ein Credential mit Admin-Berechtigungen.

1. CephObjectStoreUser mit Admin-Caps anlegen

Erstelle einen RGW-User mit users=* und buckets=* (Rook verwendet Singular user/bucket):

apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: rgw-admin-polycrate
  namespace: rook-ceph
spec:
  store: ceph-objectstore
  displayName: "Polycrate API Admin"
  capabilities:
    user: "*"
    bucket: "*"

Nach dem Apply erstellt Rook ein Secret: rook-ceph-object-user-ceph-objectstore-rgw-admin-polycrate.

Hinweis: Rook's capabilities-Feld deckt nur users/buckets ab. Die für polycrate-api ebenfalls zwingende metadata-Cap (siehe 2b) kann darüber nicht gesetzt werden und muss immer per radosgw-admin nachgezogen werden.

2. Access Key und Secret Key auslesen

kubectl -n rook-ceph get secret rook-ceph-object-user-ceph-objectstore-rgw-admin-polycrate -o jsonpath='{.data.AccessKey}' | base64 -d
kubectl -n rook-ceph get secret rook-ceph-object-user-ceph-objectstore-rgw-admin-polycrate -o jsonpath='{.data.SecretKey}' | base64 -d

2b. Caps nachträglich setzen (bei „lacks users=* capability“ oder „Authentication failed“ in polycrate-api)

Rook setzt beim CephObjectStoreUser ggf. nicht die Admin-API-Caps. Dann in rook-ceph-tools:

radosgw-admin caps add --uid=rgw-admin-polycrate --caps="users=*" --rgw-realm=ceph-objectstore
radosgw-admin caps add --uid=rgw-admin-polycrate --caps="buckets=*" --rgw-realm=ceph-objectstore
radosgw-admin caps add --uid=rgw-admin-polycrate --caps="metadata=*" --rgw-realm=ceph-objectstore

metadata=* ist PFLICHT: polycrate-api's Connectivity-Check (RadosGWAdminAPI.check_connectivity in src/s3/cluster_integrations/radosgw.py) ruft GET /admin/metadata/user auf. Ohne diese Cap antwortet RGW dort mit 403 AccessDenied und polycrate-api meldet fälschlich „Authentication failed for ... Check credential (api_user/api_key) and admin capabilities (users=, buckets=)“ — obwohl users=*/buckets=* bereits korrekt gesetzt sind. Mit metadata=* antwortet derselbe Request mit 404 NoSuchKey, was polycrate-api als „erreichbar“ wertet. Beobachtet auf aycloud-hel1-satellite-1 am 2026-07-06.

Realm-Pflicht: Rook nutzt Realm ceph-objectstore. user list und caps add benötigen --rgw-realm=ceph-objectstore, sonst user list = [].

3. In polycrate-api eintragen

• Credential (Kind: S3 Credential): api_user = Access Key, api_key = Secret Key
• S3 Cluster (Kind: Rook CEPH): admin_endpoint = https://<s3-host> (z.B. https://s3.example.com), credential zuweisen, namespace = rook-ceph, region = z.B. us-east-1

Die RadosGW Admin API nutzt denselben Host/Port wie die S3 API.