SQLite database images become C API values
SQLite 3.23.0 adds optional APIs for moving a database between a connection and a memory buffer. Their ownership rules matter as much as the serialization itself.
An in-memory database is convenient until another component needs to receive it. SQL text can describe its schema and contents, but exporting statements means choosing an interchange representation and rebuilding the database through execution. SQLite already has a database file representation. Version 3.23.0 gives C applications a direct way to obtain that representation in memory and open a database from it.
The 2 April 2018 release notes introduce sqlite3_serialize() and sqlite3_deserialize(). Both require a build with SQLITE_ENABLE_DESERIALIZE. That qualification belongs in any description of the feature. An application cannot assume that finding this version of SQLite on a system means the optional entry points are available in its library.
I value SQLite's treatment of a database as an application artifact. These interfaces extend that model into memory without requiring an application to invent another database format. The interesting design work is in the agreement between the connection and the owner of the buffer.
Serialization produces a database image
The API declarations and documentation in the 3.23.0 header describe serialization in terms of a named database on a connection. The schema name can select main or another attached database. A size output reports the number of bytes in the image, while the return value points to the bytes.
For an ordinary disk database, the documentation describes the serialization as a copy of the disk file. For an in-memory or temporary database, it is the sequence of bytes that a backup to disk would produce. The result therefore retains database structure. It is not a stream of rows stripped of indexes and schema definitions.
The ordinary call allocates its result through sqlite3_malloc64(). The caller owns the returned allocation and must release it with the corresponding SQLite allocator interface. A failed allocation can return a null pointer, so a size variable alone is not proof that a usable buffer exists.
That ownership rule makes the result useful across an application boundary. The component receiving the image can keep it independently of subsequent work on the original connection, provided the application preserves the allocation until the receiver finishes. The cost is space for a complete serialized database. This API is appropriate when a complete image is the desired unit of transfer, not when the consumer wants a few selected records.
The schema parameter also prevents an easy conceptual mistake. A connection can refer to several databases, but the call serializes one named database. Code that treats a connection as a single undifferentiated object may otherwise assume that one buffer contains everything reachable through it.
Deserialization opens the buffer as a database
The reverse operation has a more involved contract. sqlite3_deserialize() disconnects the selected database on a connection and reopens it as an in-memory database based on the supplied buffer. Its arguments distinguish the length of the database image from the capacity of the allocation containing it.
Those sizes need not be equal. An application can supply spare capacity for later writes. Without permission to resize the allocation, the database cannot grow beyond that capacity. Deserialization is consequently more than an import operation that forgets its input after reading it. The supplied allocation becomes the memory on which the database operates.
The flags describe how that relationship ends. SQLITE_DESERIALIZE_FREEONCLOSE transfers responsibility for releasing the allocation to SQLite. The buffer must then come from SQLite's allocator. Without that flag, the caller retains responsibility for its memory and must preserve the buffer while SQLite uses it.
A separate SQLITE_DESERIALIZE_RESIZEABLE flag permits SQLite to grow the allocation using sqlite3_realloc64(). The spelling is part of the API. The header says to use it only together with FREEONCLOSE, which keeps resizing and final release under the same allocator and owner.
The SQLITE_DESERIALIZE_READONLY flag expresses a different choice. It tells SQLite to treat the resulting database as read-only. This can fit a lookup image supplied by an application, where SQL is a convenient query interface but mutation is outside the intended use.
A connection involved in a read transaction or a backup operation can make deserialization fail with SQLITE_BUSY. Replacing a named database therefore has a connection-state precondition as well as a buffer contract. A wrapper should expose that failure to its caller instead of presenting deserialization as an operation that always succeeds.
Avoiding a copy changes the contract
Serialization also offers SQLITE_SERIALIZE_NOCOPY. With that flag, SQLite does not allocate a fresh image. It returns a pointer to an existing contiguous in-memory representation when one exists. The header associates that representation with a preceding deserialization.
The flag is therefore a request with a possible null result. An ordinary database connection does not necessarily keep all of its pages in a single suitable allocation. A caller cannot infer that an in-memory database automatically satisfies the contiguous representation requirement.
The API still reports the database size when no-copy serialization finds no suitable representation. This is useful information, but it is separate from receiving a pointer. Code needs to keep those outcomes distinct when deciding whether to allocate its own destination or take another path.
I prefer an interface that makes this limitation explicit. An unconditional promise of zero-copy export would constrain the engine's internal storage choices. Returning null lets SQLite keep its normal representation while allowing applications that already supplied a contiguous image to benefit from it.
The copied and borrowed forms also deserve different wrapper types or documentation in a binding. The ordinary allocation has a clear release operation. A borrowed pointer remains tied to storage the connection uses. Treating both as an unqualified byte array would hide the condition that makes the optimization possible.
Small SQL changes have compatibility constraints too
The release also recognizes TRUE and FALSE constants and adds the IS TRUE and IS FALSE operator families. Its notes preserve compatibility with existing columns called true or false. When such columns exist, the identifiers refer to the columns.
That rule reflects the difficulty of extending a SQL vocabulary used by existing files and applications. A new constant cannot simply take priority over an identifier that an older schema already uses. The release adds the spelling while preserving the meaning of those existing references.
The serialization APIs face the same broad requirement at a different boundary. Applications gain a new way to handle a database while the result remains a database image. For me, that continuity is the useful part of the change. A program can choose a read-only image, reserve room for writes, or transfer allocation ownership, with each choice expressed in the C interface that opens the database.