Two Years of SIG Node CI: On the Path to More Stable Kubernetes Tests

Ensuring the reliability of SIG Node upstream code is a continuous effort that takes a lot of behind-the-scenes effort from many contributors. There are frequent releases of Kubernetes, base operating systems, Container runtimes, and test infrastructures, leading to a complex matrix that requires attention and continuous investment to “keep the lights on.”

In May 2020, the Kubernetes Node Special Interest Group (“SIG Node”) organized a new subproject for continuous integration (CI) for node-related code and tests. Since its inception, the SIG Node CI subproject has held weekly meetings, and even a full hour often isn’t enough to address all the bugs, test-related PRs, and issues, and to discuss all ongoing work within the group.

Over the past two years, we have fixed merge-blocking and release-blocking tests, reducing the time to merge pull requests from Kubernetes contributors, thanks to reduced test flakes. Initially, the node test jobs only passed 42% of the time, and through our efforts, we now ensure a consistent success rate of over 90%. We have closed 144 test failure issues and merged 176 pull requests in kubernetes/kubernetes alone. Additionally, we have helped subproject participants advance in the Kubernetes contributor hierarchy, with 3 new members, 6 new reviewers, and 2 new approvers.

The Node CI subproject is an accessible first step for new contributors to get started with SIG Node. There is a low barrier to entry for new contributors to tackle high-impact bugs and test fixes, although there is a long journey before contributors can climb the entire contributor hierarchy: it took over a year to establish two new approvers for the group. The complexity of all the different components driving Kubernetes nodes and its test infrastructure requires a long-term investment for developers to deeply understand the entire system at both high and low levels of detail.

We have several regular contributors in our meetings, yet our pool of reviewers and approvers is still small. Our goal is to further increase the number of contributors to ensure a sustainable distribution of work that doesn’t rely solely on a few key approvers.

It is not always obvious how subprojects within SIGs are formed, operated, and function. Each is unique to its supporting SIG and tailored to the projects the group is meant to support. As a group that has welcomed many first-time contributors to SIG Node, we want to share some details and successes from the past two years to demystify our internal operations and celebrate the hard work of all our dedicated contributors!

Timeline

May 2020. The SIG Node CI group was founded on May 11, 2020, with more than 30 volunteers signing up to improve the SIG Node CI signal and overall observability. Victor Pickard focused on getting Testgrid jobs to pass, as Ning Liao proposed forming a group around this effort and created the original group charter document. The SIG Node chairs supported the group’s formation with Victor as the subproject lead. Sergey Kanzhelev joined Victor shortly thereafter as co-lead.

In the kick-off meeting, we discussed which tests to focus on first and talked about merge-blocking and release-blocking tests, many of which failed due to infrastructure issues or faulty test code.

The subproject initiated weekly one-hour meetings to discuss and triage ongoing work.

June 2020. Morgan Bauer, Karan Goel, and Jorge Alarcon Ochoa were recognized as reviewers for the SIG Node CI group for their contributions, which significantly helped in the early phases of the subproject. David Porter and Roy Yang also joined the SIG test failure GitHub team.

Source: Kubernetes Blog