Keeping an Eye on Secure Containers: Detecting Container Drift with Admission Controllers

At Box, we use Kubernetes (K8s) to manage hundreds of microservices that enable us to stream data at petabyte scale. As part of our deployment process, we employ kube-applier as part of our GitOps workflows with declarative configuration and automated deployments. Developers declare their K8s applications in a Git repository, which requires code review and automated checks before changes can be merged and applied to our K8s clusters. However, with kubectl exec and similar commands, developers can directly interact with running containers and alter them from their deployed state. This interaction could undermine the change and code review processes enforced in our CI/CD pipelines. Moreover, it allows such affected containers to continue receiving traffic in the production environment over the long term.

To solve this problem, we developed a custom K8s component called kube-exec-controller, along with the associated kubectl plugin. These work together to detect and terminate potentially altered containers (caused by interactive kubectl commands) and reveal interaction events directly to the target pods for better visibility.

Admission Control for Interactive kubectl Commands

Once a request is sent to K8s, it must be authenticated and authorized by the API server to proceed. Additionally, K8s has a separate layer of protection called Admission Controllers that can intercept the request before an object is stored in etcd. There are various predefined admission controls compiled into the API server binary (e.g., ResourceQuota to enforce hard resource usage limits per namespace). Additionally, there are two dynamic admission controls called MutatingAdmissionWebhook and ValidatingAdmissionWebhook, which are used to either mutate or validate K8s requests. We adopted the latter to detect container drift at runtime caused by interactive kubectl commands. This entire process can be divided into three steps, which are detailed below.

1. Allowing Interactive kubectl Command Requests

First, we needed to enable a validating webhook that sends qualified requests to kube-exec-controller. To add the new validation mechanism specifically for interactive kubectl commands, we configured the webhook’s rules with resources as [pods/exec, pods/attach] and operations as CONNECT. These rules inform the cluster’s API server that all exec and attach requests should be subject to our admission control webhook. In the ValidatingAdmissionWebhook we configured, we specified a service reference (which can also be replaced by a url indicating the webhook’s location) and caBundle to enable validation of its X.509 certificate, both under the clientConfig stanza.

Here is a brief example of what our ValidatingWebhookConfiguration object looks like:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: example-validating-webhook-config
webhooks:
  - name: validate-pod-interaction.example.com
    sideEffects: None
    rules:
      - apiGroups: ["*"]
        apiVersions: ["*"]
        operations: ["CONNECT"]
        resources: ["pods/exec", "pods/attach"]
    failurePolicy: Fail
    clientConfig:
      service:

Source: Kubernetes Blog