JD.com moves application deployment into container images

JD.com's first container platform already provisioned resources in minutes and ran a substantial production workload. The reason to build its Kubernetes successor was therefore more specific than replacing slow infrastructure. In the Infrastructure Platform Department's February 10 report, the earlier platform still treated containers as virtual machines. Application installation remained a separate deployment operation. The new platform puts the application into the image and organizes the surrounding services around that artifact.

That distinction explains the migration better than the headline about leaving OpenStack. JDOS 2.0 retains the network and storage of JDOS 1.0, and its networking still uses OpenStack Neutron. Kubernetes changes how the platform represents and operates applications. It does not require the company to discard every infrastructure decision that preceded it.

I find this a useful account because it assigns the benefits to the changes that produced them. Container isolation and resource pooling had already delivered substantial improvements. The Kubernetes work addresses the remaining gap between allocating a container and maintaining a running application.

The first platform solved a different problem

Before 2014, JD.com's applications ran on physical machines. The team reports an average wait of a week between allocating capacity and bringing an application online. Applications lacked isolation, and moving them after a machine failure took hours. Existing compilation, deployment and monitoring tools compensated for some of those limitations without changing the allocation model.

JDOS 1.0 introduced Docker under OpenStack with the Novadocker driver. OpenStack managed the containers as virtual machines. This let the team pool computing resources and isolate application instances while retaining its existing application operations tools. According to the report, production resource requests fell to several minutes, and both deployment density and physical machine utilization increased threefold. These are the team's reported results for its first platform, not measurements of Kubernetes.

The distinction matters when reading the scale figures. The report attributes 150,000 running containers in November 2016 to the existing container platform. Its individual OpenStack container clusters ranged from 4,000 to 10,000 compute nodes. Those numbers do not describe Kubernetes cluster sizes. The Kubernetes platform described at the end of the article runs two clusters with about 20,000 Pods daily and has taken on about 20 percent of the company's applications.

The old images primarily contained an operating system and the application's runtime stack. Moving a workload into one of those containers still required the automatic deployment tools to install the application. An allocated container therefore did not completely describe what would execute inside it. The platform had improved resource delivery without yet making the image the complete application delivery artifact.

Building the application before scheduling it

JDOS 2.0 changes the image boundary. Its build process produces an image containing the complete software stack, including the application. Deployment then uses that image to create or update application Pods. The choice moves application assembly into the build pipeline, before the platform schedules the resulting workload.

The published pipeline starts when a developer pushes code to Git. That triggers a Jenkins master, which asks Kubernetes to create a Jenkins worker Pod. The worker checks out, compiles and packages the source. It sends the package and Dockerfile to a separate Docker image build node, which builds the image and pushes it to Harbor. Users then create or update the application's Pods in the required zones.

This is a specific division of execution responsibilities. Kubernetes runs the Jenkins workers, but the report does not say those workers perform the Docker build themselves. The separate image build node remains part of the design. Describing every build step as a Pod operation would conceal an actual dependency in the published architecture.

An image containing the application reduces the work left for deployment to perform inside a running container. The deployed artifact can represent the application's assembled software instead of only a prepared runtime environment. It also makes the registry part of application delivery. Harbor is consequently an explicit component of JDOS 2.0, alongside GitLab and Jenkins, rather than an incidental place to store base images.

Retaining Neutron behind Kubernetes objects

The networking design demonstrates how much of the old platform survives. JDOS 2.0 uses the VLAN model from OpenStack Neutron. Each Pod occupies a Neutron port and receives its own IP address. JD.com's Cane project integrates kubelet with Neutron through the Container Network Interface standard.

Cane also handles Kubernetes LoadBalancer Services. When such a Service is created, changed or deleted, Cane calls the corresponding Neutron load-balancing service interface. Kubernetes provides the application-facing object, while Neutron supplies the underlying network operation. Cane's Hades component provides internal DNS resolution for Pods. At publication, the team says it is preparing Cane's source for release, so this report should not be mistaken for an announcement that the implementation is already publicly available.

The architecture illustrates why an orchestration migration need not become a simultaneous network replacement. Existing infrastructure can remain behind a new application interface when an integration supplies the necessary behavior. That preserves previous engineering work, although it also leaves the platform responsible for the integration between the two systems.

The January description of SIG Cluster Lifecycle's responsibilities makes a related boundary explicit. Kubeadm handles Kubernetes bootstrap and deliberately leaves machine provisioning to other tools. JD.com's report concerns a different layer, but both accounts reject the assumption that adopting Kubernetes means one tool must own every infrastructure operation.

Application structure becomes part of the platform

JDOS 2.0 distinguishes systems from applications. A system contains several applications, and an application contains several Pods providing the same service. Departments can request one or more systems, with each system mapped to a Kubernetes namespace. This gives the platform an organizational structure above individual running containers.

The supporting tools also run largely on Kubernetes. The report lists GitLab, Jenkins, Harbor, Logstash, Elasticsearch and Prometheus among the containerized platform components. Kubernetes therefore operates both business workloads and much of the machinery used to deliver and observe them. The platform team still has to assemble that machinery into a coherent service for developers.

My preference for Kubernetes rests on its ability to express the intended running state and keep working toward it. The JD.com report connects that preference to a concrete platform requirement. Its SREs particularly value the replication controller, and the team reports application scaling in seconds. The substantive change is that an application now has a defined image, a grouping in the platform, and an orchestration mechanism, while the existing Neutron network continues to provide connectivity.