May 11, 2022·6 min read·engineeringsqlitecloudflareserverless

Cloudflare D1 puts SQLite behind a Workers service

D1 proposes a managed relational database for Cloudflare Workers. SQLite supplies the query engine, while the service must define how remote execution, replication, and recovery fit together.

Running application code near a user does not establish where that application's durable state lives. A request can arrive at nearby compute and still need to cross the network to reach its database. Cloudflare's May 11 announcement of D1 proposes an integrated SQL database for Workers, built on SQLite. The company plans to manage placement and database operation as part of the platform. This is an announcement of the service, with beta access still ahead.

I find the choice of SQLite persuasive because the engine already separates relational querying from the need to operate a standalone database server. D1 introduces a service around that engine for a different deployment environment. Understanding the proposal requires keeping the engine's responsibilities distinct from the responsibilities of a distributed service. SQL compatibility says something useful about queries, while placement and remote execution determine a different part of application behavior.

SQLite supplies an engine with explicit interfaces

The SQLite public API in the April release describes a database connection as an object opened by sqlite3_open or related constructors. Prepared statements, stepping through execution, and finalization provide the underlying sequence for evaluating SQL. The convenience function sqlite3_exec wraps that sequence and can evaluate multiple statements, invoking a callback for result rows.

Those interfaces make embedding concrete. A host application controls the connection and calls the engine. The engine does not require an application to express every data operation as a separate remote service endpoint. A product catalog, for example, can express a count as an aggregate query rather than fetch all product records and count them in application code.

That division remains useful when a platform supplies the host. It can expose a narrower application interface while letting SQLite parse and evaluate the relational work behind it. Query semantics remain the SQL engine's responsibility. The surrounding service determines how requests arrive, which database they address, and how the result becomes available to the caller.

The distinction also limits what can be inferred from an engine choice. A successful local function call and a successful request to a remote service are different events. The latter introduces a caller that can lose its connection independently of database execution. The public API documents the engine that D1 builds around, but it does not establish the recovery contract of every service using that engine.

A binding identifies the database used by a Worker

Rita Kozlov and Glen Maddern's technical announcement illustrates access through a database binding on the Worker's env argument. Its proposed API includes a get method taking SQL and optional named parameters. The article expects beta invitations as early as June and previews batching, read replicas, and backups. These are the announced design and interface, not evidence of a generally available deployment today.

An abbreviated example using that announced interface makes the application boundary visible. It assumes a configured DB binding and a Product table.

async function countProducts(env) {
  return await env.DB.get(
    'SELECT count(*) AS num_products FROM Product;'
  );
}

The interesting property is the location of the work. The application asks for an aggregate result. It does not need to implement a product-list protocol merely to compute a count. Choosing SQL allows the operation to describe its result in terms of stored relations, even though the caller uses a platform-specific binding to submit it.

Binding configuration is a separate concern from SQL text. A query naming Product still needs an execution context identifying which database contains that table. Keeping the database selection outside the query can make the same application code usable with different configured environments. It also means that reviewing the query alone cannot establish which data a deployed application will modify.

This is an ordinary consequence of placing a query engine behind an application platform. An engineer assessing a deployment needs both the program and its configuration. Neither SQL familiarity nor a short client method replaces the need to know which database instance the method addresses.

Parameter values and query structure remain distinct

The SQLite API's parameter-binding interfaces establish a useful underlying model. A prepared statement has parameters whose values can be supplied separately from the SQL text. This lets the database interpret an input as a value at a defined location in a statement. The caller does not have to construct a new SQL expression by concatenating untrusted characters into the statement.

For example, the conceptual operation of fetching one product has a stable query structure and a changing identifier. The D1 announcement demonstrates named parameters for that case. The distinction matters regardless of whether the application submits the request in-process or through a service, because the intended operation is selection by a value, with the statement's structure remaining fixed.

The service wrapper introduces a further type boundary. JavaScript values have to become database parameter values, and database results have to become JavaScript results. The choice of SQLite does not by itself document every conversion a wrapper implements. The interface needs to define those conversions for the application that consumes it. A useful SQL engine can supply the relational operation while the surrounding API still determines how its inputs and outputs are represented.

This is why compatibility should be assessed at the operation an application actually performs. A query accepted by SQLite is one part of that operation. A parameter representation accepted by the platform and a result representation interpreted correctly by the application are also necessary. The announcement's short examples are useful because they expose these boundaries without requiring a large framework.

Batching addresses a cost introduced by remote execution

Several dependent database calls can require several sequential exchanges with the database service. If an application needs the first result before it can choose the second query, proximity of the initial Worker does not remove that dependency. A design that executes an entire request near the user still needs to account for where each data-dependent step runs.

The announced batching direction addresses operations that can be submitted together. Conceptually, two updates describing one purchase can be evaluated near the database without asking the application to send each update in a separate exchange. Whether a batch is atomic belongs in the service contract, alongside the behavior when one statement fails. Combining transport requests and defining a transaction are related design choices with different meanings.

SQLite's engine interface already distinguishes statements from the connection context in which they execute. A managed wrapper needs to preserve a coherent relationship between that context and the application's requested operation. An interface that accepts several statements must make clear whether they share a transaction and what a returned failure says about their effects. These are requirements implied by the proposed use, not claims about undocumented D1 internals.

I consider this more significant than a promise that every query runs close to every user. Data-dependent code needs an execution model that makes the number and meaning of remote exchanges understandable. An integrated service can help by providing operations whose scope matches the work the application needs to complete.

Placement and recovery belong to the service

Cloudflare's press release proposes locating data near users and integrating database creation with Workers. Those goals give the platform responsibility beyond SQL evaluation. A query engine evaluates operations against a database state. A placement system decides which host provides that state to a caller. The two layers need compatible contracts, particularly when a caller reads data after an update.

A read-only copy can answer queries without independently accepting changes. Its usefulness to a particular request depends on what state the request requires. An account page immediately following a change has a different requirement from a report that tolerates older information. Naming SQLite as the engine does not determine that application requirement, so a distributed database interface needs to make the relevant guarantees explicit.

Recovery creates a similar division. A backup preserves some state from a particular point. Returning to that state can restore database contents without reproducing every effect the application caused elsewhere. A purchase operation might involve communication outside the database. The database's recovery mechanism cannot automatically infer how that external work relates to a restored row. This is an application architecture concern for any managed database, including the service proposed here.

The appeal of D1 is an integrated place to express relational work for Workers applications. The announcement makes SQLite the query engine and Cloudflare the operator of the surrounding service. That allows the proposal to build on a small embedded engine while confronting network and placement questions at the platform level. The resulting developer contract needs to connect those responsibilities so that an application can tell what a completed database operation means.