Docker Model Runner separates model distribution from inference execution

Docker Model Runner uses Docker's distribution conventions without starting a container for each model invocation. That distinction explains much of the initial design. Model files can travel through an OCI registry, while the inference engine runs as a native host process to use Apple silicon GPU acceleration.

Docker's April 9 introduction presents Model Runner as a beta in Docker Desktop 4.40. It targets local application development with an inference engine built on llama.cpp and an OpenAI-compatible API. The announcement combines familiar artifact handling with a deliberately different execution model.

I like Docker's ability to make an environment's inputs distributable. Applying the registry model to inference weights seems useful for the same reason. The benefit depends on keeping the artifact, the inference process, and the application's client configuration distinct enough to understand what each controls.

A model command sends an inference request

Docker's April 4 quickstart explains what happens behind the CLI. A model run command calls an inference server API hosted through Desktop. The server loads the requested model through llama.cpp and performs inference. It does not create a new container.

The guide demonstrates this with a small SmolLM2 variant. Its explicit tag records the model's parameter count and quantization choice.

docker model pull ai/smollm2:360M-Q4_K_M
docker model run ai/smollm2:360M-Q4_K_M "Explain what an image layer contains."

The prompt here is illustrative. The model reference and command syntax follow the quickstart. Docker describes this variant as having 360 million parameters and four-bit quantization, and documents its tag pattern as a model name followed by parameter and quantization information.

The first command makes the model available locally. The second makes a request through the server. Those are separate operations with separate effects, even though the naming resembles the image and container commands that Docker users already know.

The quickstart says the server loads models on demand. An application does not need to issue an interactive model run command before sending its own inference request. It does need the requested model to have been pulled and to be locally available.

That changes how application startup should be understood. A downloaded artifact is a prerequisite for serving the request, but an earlier interactive prompt is not. The model's presence in local storage and its presence in active memory describe different states.

The guide also describes the initial residency policy. A loaded model remains in memory until another model is requested or an inactivity timeout expires. It gives the current timeout as five minutes. A first request after unloading can therefore involve a load operation that a subsequent request does not require.

No performance number follows automatically from that behavior. It establishes that requests can exercise different execution paths depending on residency. A developer interpreting a local response time needs to know whether the server had already loaded the selected model.

Host execution provides the GPU path

The introduction explicitly connects native execution with Apple silicon acceleration. Docker runs the inference engine as a host process to avoid the performance limitations of placing it inside a virtual machine. The launch beta is for Macs with Apple silicon.

This is an architectural choice with consequences for how the system should be described. Packaging a model as an OCI artifact does not imply that the model runs in a container. The registry format determines how the files are distributed, while the runner determines where the inference process executes.

Dagger's April 9 integration article confirms the same distinction from an application's perspective. It describes models in GGUF format, packaged as OCI artifacts, and executed natively through Apple's Metal APIs. The article requires Apple silicon and Desktop 4.40 or later for the initial integration.

The shared Docker terminology can otherwise hide that difference. A model is an input to an inference engine. Calling its run command submits work to the engine that Desktop manages. The familiar CLI makes the operation easier to find, but the underlying lifecycle remains different from creating a container process.

For my preferred small service images, this division is appealing. An application image can contain its own executable and client code while the model remains an independently distributed dependency. That keeps the model's size and update schedule out of the service image's filesystem definition.

It also means the application's image alone cannot specify the complete inference environment. The selected model and runner arrangement remain relevant inputs. A reproducible application build and a reproducible model selection answer related questions, but each needs its own explicit definition.

The API has distinct host and container endpoints

The quickstart documents an OpenAI-compatible endpoint that applications can use through existing compatible clients. It gives containers the address http://model-runner.docker.internal/engines/v1. Host processes can use http://localhost:12434/engines/v1 after enabling TCP access on the host.

In the documented default arrangement, host access uses the Docker socket. Containers use the special internal endpoint. A host application expecting an ordinary TCP base URL therefore needs a configuration change before pointing its client at localhost.

The guide shows the Desktop command for that change. The port is explicit so that the resulting client base URL has a concrete target.

docker desktop enable model-runner --tcp 12434

The same guide supplies a Java client example with that base URL and the SmolLM2 model reference. The important mechanism is the separation between where the client sends requests and which model it asks the server to load. Changing one does not implicitly set the other.

OpenAI compatibility describes an interface used by client software. It does not establish that every model has the same capabilities or response behavior. An application still requests a specific locally available model, and the serving engine still imposes its own behavior.

The Dagger article supplies a concrete compatibility qualification. For its tool-using integration, it disables streaming because llama.cpp has limitations around streaming while using tools. That is a useful example of why adopting a compatible endpoint does not remove the need to understand the operation the client performs.

Dagger also documents an engine-specific endpoint under /engines/llama.cpp/v1/. Its integration selects the base URL through environment variables and can set a default model independently. The relevant point is that the application framework can redirect its requests to a local server without becoming responsible for the native inference engine itself.

OCI packaging supplies a distribution contract

The launch announcement frames OCI artifacts as a response to model files distributed through disconnected tools and custom download arrangements. A registry already has a role in versioning and retrieving artifacts. Docker applies that role to models available through its Hub catalog.

At launch, the announcement says developers can pull ready-to-use models from Docker Hub. Publishing custom models and broader registry integration appear in its future plans. Those plans should remain separate from the operations the beta currently provides.

That boundary is especially important because OCI compatibility can make a broad ecosystem possible without proving that a particular CLI already supports every part of it. The format and the current product commands are different evidence. The introduction establishes the packaging direction and a curated pull workflow.

Dagger makes the application-side consequence concrete. Its article connects a locally running workflow to the local model server through client settings. Model retrieval, native execution, and tool integration can therefore be assembled without requiring the application developer to implement model storage or manage llama.cpp directly.

I see the useful contribution as reducing the number of separately managed pieces in local inference development. The runner supplies a native server and a distribution convention, while the application uses an identifiable API endpoint. That is a specific reduction in setup work, even though the beta deliberately supports a limited hardware target.

The launch scope remains narrow

The introduction lists Windows GPU support, custom model publication, and integrations including Compose and Testcontainers as future work. None of those plans should be treated as part of the initial Apple silicon beta. A local development workflow can rely on what the release documents today while leaving those proposed extensions out of its dependency assumptions.

The beta's useful contract is already concrete. A developer pulls a named artifact, and the runner loads it for requests through its compatible API. Native execution gives the engine access to the supported host GPU, while the registry packaging gives the model an address that application configuration can name.

That contract also clarifies troubleshooting. A missing local model concerns retrieval and storage. An unreachable localhost endpoint can concern whether host TCP access was enabled. A request that triggers model loading has a different server path from one using a resident model. Each condition belongs to a documented part of the initial design.