The Art of Forensic Analysis of Kubernetes Containers

In my previous article, I introduced the topic of Checkpointing in Kubernetes and explained how it can be set up. This time, I want to show you how to analyze the created checkpoints. Checkpointing is currently still an alpha feature in Kubernetes, and this article provides a preview of what this feature might look like in the future.

Preparation

Details on configuring Kubernetes and the underlying CRI implementation to enable support for Checkpointing can be found in my previous article.

As an example, I have prepared a container image (quay.io/adrianreber/counter:blog) that I want to checkpoint and analyze below. With this container, I can create files within the container and store information in memory that I want to find later in the checkpoint.

To run this container, I need a Pod. For this example, I use the following Pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: counters
spec:
  containers:
  - name: counter
    image: quay.io/adrianreber/counter:blog

This results in a container named counter running in a Pod named counters.

Once the container is running, I perform the following actions with this container:

console $ kubectl get pod counters –template ‘{{.status.podIP}}’ 10.88.0.25 $ curl 10.88.0.25:8088/create?test-file $ curl 10.88.0.25:8088/secret?RANDOM_1432_KEY $ curl 10.88.0.25:8088

The first access creates a file named test-file with the content test-file in the container, and the second access stores my secret information (RANDOM_1432_KEY) somewhere in the container’s memory. The last access simply adds an additional line to the internal log file.

The last step before I can analyze the checkpoint is to tell Kubernetes to create the checkpoint. As described in the previous article, this requires accessing the checkpoint API endpoint of the kubelet.

For a container named counter in a Pod named counters in a Namespace named default, the kubelet API endpoint is accessible as follows:

# Run this on the node where the Pod is running
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"

For completeness, the following curl command line options are required to make curl accept the self-signed certificate of the kubelet and authorize the use of the kubelet checkpoint API:

--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key

Once checkpointing is complete, the checkpoint should be available at /var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar.

In the following steps of this article, I use the name checkpoint.tar when analyzing the checkpoint archive.

Source: Kubernetes Blog