Kubernetes 1.26: Efficient Job Tracking for Massive Batch Workloads!

The release of Kubernetes 1.26 introduces a stable implementation of the Job controller, which reliably tracks a large number of highly parallel jobs. The teams from SIG Apps and WG Batch have been working on this fundamental improvement since Kubernetes 1.22. After several iterations and scalability reviews, this is now the default implementation of the Job controller.

In combination with the indexed completion mode, the Job controller can manage massively parallel batch jobs and supports up to 100,000 concurrent Pods.

The new implementation has also enabled the development of the Pod failure policy, which is available as a beta feature in version 1.26.

What specifically changes for developers/DevOps teams?

To utilize job tracking with finalizers, you should upgrade to Kubernetes 1.25 or newer and create new jobs. This feature can also be used in v1.23 and v1.24 if you have the ability to enable the JobTrackingWithFinalizers Feature Gate.

If your cluster is running Kubernetes 1.26, job tracking with finalizers is a stable feature. For v1.25, it is behind this feature gate, and your cluster administrators may have explicitly disabled it—for example, if there is a policy to not use beta features.

Jobs created before the upgrade will continue to be tracked with the legacy behavior. This is to avoid retroactively adding finalizers to running Pods, which could lead to race conditions.

For maximum performance with large jobs, the Kubernetes project recommends using the indexed completion mode. In this mode, the control plane can track job progress with fewer API calls.

If you are a developer of operators for batch, HPC, AI, ML, or related workloads, we encourage you to use the Job API to delegate accurate progress tracking to Kubernetes. If there is something missing in the Job API that forces you to manage simple Pods, the Working Group Batch welcomes your feedback and contributions.

Notes on Deprecation

During the development of this feature, the control plane added the annotation batch.kubernetes.io/job-tracking to jobs created when the feature was enabled. This allowed for a safe transition for older jobs but was never intended to be permanent.

In version 1.26, we have deprecated the annotation batch.kubernetes.io/job-tracking, and the control plane will stop adding it in Kubernetes 1.27. With this change, the legacy job tracking implementation will also be removed. As a result, the Job controller will track all jobs with finalizers and ignore Pods that do not have the aforementioned finalizer.

Before upgrading your cluster to 1.27, we recommend checking that there are no running jobs without the annotation or waiting until these jobs are completed. Otherwise, you might observe the control plane recreating some Pods. We expect this to not affect users, as the feature has been enabled by default since Kubernetes 1.25, providing ample buffer for old jobs to complete.

ayedo GmbH is your partner when it comes to Kubernetes and modern container orchestration. Leverage our expertise to unlock the full potential of Kubernetes!

Source: Kubernetes Blog