May 27, 2020·4 min read·engineeringkubernetesgovernanceinfrastructure

Kubernetes infrastructure needs community operating knowledge

The infrastructure working group's migration joins operational access with documentation and funding. Its progress shows why open source code alone cannot make a project's release systems community operated.

Kubernetes contributors could inspect the project's infrastructure tools without being able to administer the cloud project running them. That distinction is the starting problem in Kiran Oliver's introduction to the infrastructure working group. Google had built and maintained the supporting infrastructure, but its internal cloud project prevented community members from helping operate it. Publishing the tools did not provide the access or organizational arrangements needed to run them.

Today's article describes a migration already in progress. The working group became official in 2019, after community members worked with the Steering Committee on a charter. Its responsibilities cross Architecture, Contributor Experience, Release, and Testing. The subject is therefore an existing effort to transfer operations, not the creation of a new group this month.

I value the Kubernetes API machinery because it makes operational decisions expressible and inspectable. The infrastructure work raises a related question about the project itself. Contributors need to understand and change the systems that test and distribute Kubernetes if community ownership is to include its operation.

Funding began before the migration was complete

Google's August 29, 2018 funding announcement committed nine million dollars in cloud credits to CNCF over three years. The resources supported project testing, image downloads, and services such as DNS. Google described transferring day-to-day administration to Kubernetes project members, including Google contributors.

The grant supplied funding for a transition in operational responsibility. It did not mean that every service had already moved when the announcement appeared. Oliver's account explains the work still required to identify systems, move them, and make them sustainable. Reading the two announcements together distinguishes an organizational commitment from the engineering necessary to complete it.

The original cloud announcement also identifies scalability testing as a substantial use of the infrastructure. Distribution and testing both consume resources even though users do not see them as features in a release. The working group must make those costs visible to the community responsible for continuing the work.

An automated billing report is one of the concrete improvements listed today. The group begins meetings by reviewing it. That practice gives contributors a shared view of expenditure and makes billing part of routine operation. A grant is easier to govern when the people consuming it can see the consequences of infrastructure decisions.

Inventory comes before reliable transfer

The working group's questions begin with identifying dependencies. It needs to know which applications run, where they run, and where their source lives. It must distinguish custom software from existing products and record how services depend on one another. Administrative procedures are part of that inventory too.

This is more demanding than copying a deployment manifest. A service can keep running for a long time after its original author leaves. That continuity is useful for users, but it does not establish that another operator knows how to change or recover it. Oliver identifies precisely this problem in infrastructure that has not been updated for some time.

The migration also requires access policies, billing arrangements, and privacy and security rules. Those decisions affect who may perform operations and which responsibilities the community accepts. They cannot be inferred solely from the source code of an application. A technically reproducible deployment still needs an accountable administrative arrangement.

The article separates migration as it stands from subsequent improvement for sustainability. Moving a service first can reduce the amount of simultaneous change. Improving its automation later can reduce the knowledge required for routine administration. These are distinct tasks even when both belong to the same working group.

Documentation is part of operational capacity

Oliver describes documentation as a major constraint on progress. The group is volunteer led, and contributors can become overscheduled. Documentation and organization then receive less attention, which makes the migration slower. Bart Smykla is working on that gap, and the article asks for more contributors to help.

The consequence is practical. If operating knowledge stays with a small number of people, transferring an account does not make the service broadly maintainable. New administrators need enough information to identify dependencies and perform the required work. The article's inventory questions provide a useful definition of what that information must cover.

I regard that documentation as part of the infrastructure deliverable. A service whose configuration is visible but whose recovery procedure is unknown remains dependent on whoever remembers it. The working group's emphasis on operating knowledge addresses a different constraint from the availability of cloud credits.

Automation supports the same goal. The stated direction is to move away from people running scripts toward a more automated GitOps model. That can make changes and permissions easier to review through declared configuration. It still depends on contributors understanding what the automation manages and how its own failures are handled.

Progress includes the distribution path

The article reports community ownership of DNS for k8s.io and kubernetes.io, with a route for contributors to request record changes. It also reports community operation of the publishing bot that publishes staging repositories into their own top-level repositories. These are different services, but each affects how the project makes its work available.

The gcsweb.k8s.io service, which provides anonymous access to artifact buckets, runs on a community-owned cluster as well. Image promotion has an automated process with documented infrastructure and automated permission provisioning. These examples show that the migration concerns both user-facing artifact access and the internal work that produces published project outputs.

None of these reported improvements establishes that the full migration is finished. The working group explicitly includes documenting goals and indicators of completion among its responsibilities. That is necessary when many services and SIGs share an effort, because individual migrations do not provide an overall inventory of what remains.

The May account makes the dependency on contributors visible alongside the technical progress. Funding supports the cloud resources, access arrangements let community members operate them, and documentation lets additional people learn the work. The remaining inventory and migration tasks need that operating knowledge to become shared enough for services to remain maintainable when individual contributors move on.