SQLite gives extension pointers a separate representation

A native extension sometimes needs SQL execution to carry a reference to an object in the application process. Treating that reference as an ordinary value creates a dangerous ambiguity. SQL values come from queries and parameters, while a native pointer tells C code where to access memory. SQLite 3.20.0, released on 1 August, introduces pointer-passing interfaces and changes several extensions to require them.

The release explicitly accepts backward incompatibility for improved security. Finding the FTS5 API pointer now requires sqlite3_bind_pointer(). The carray and remember extensions also require pointer binding for their pointer arguments. An application using those interfaces must change the native code that supplies the pointer, even if its surrounding SQL remains familiar.

I consider that a justified compatibility break. An embedded engine shares an address space with its caller, so an extension that mistakes a SQL value for a trusted native reference can affect the entire application. A dedicated representation makes that boundary something the library can check.

A pointer has an SQL value of NULL

The 3.20.0 public header defines sqlite3_bind_pointer(S,I,P,T,D) as binding an SQL NULL to parameter I of statement S, while associating it with pointer P and type name T. The final argument supplies an optional destructor. SQLite calls that destructor when it has finished using the pointer.

This representation separates what SQL sees from what cooperating native code can retrieve. A pointer binding does not need to expose a native address as an integer SQL result. Its associated type string tells the consumer which pointer convention the producer intended to use. The header says that the string should have static storage, preferably as a string literal.

The released implementation in vdbeapi.c shows the checks made by sqlite3_value_pointer(). The value must have the pointer representation, including its subtype, and the requested type name must match through a string comparison. Otherwise the function returns a null pointer.

That check is more specific than asking whether a value can be converted to an integer of pointer width. A SQL integer does not acquire the internal pointer representation just because its bits resemble an address. The consumer also cannot retrieve a pointer under an unrelated type string. Those are useful boundaries for extensions that cooperate through SQL calls.

Producing and consuming native references

The interfaces cover parameters and function results. sqlite3_bind_pointer() associates a native object with a prepared-statement parameter. sqlite3_result_pointer() sets a function result to the pointer representation. A native consumer calls sqlite3_value_pointer() with the expected type name to retrieve the association.

The result implementation first sets its output to NULL and then attaches the pointer information. Its parameters include the type name and destructor, just as the binding interface does. This provides a common representation across two ways a value can enter an SQL expression, rather than leaving each extension to invent an encoding.

Object lifetime remains part of the native contract. A type name checks the convention used for retrieval, but it does not make an arbitrary object remain alive. The destructor interface gives SQLite a way to release an object it has finished using. When the caller retains ownership instead, that caller must still keep the object valid throughout the period of use.

I like that the mechanism is small enough to inspect. The consumer checks a representation and a type name, then returns the associated pointer or null. It does not require application code to infer trust from the contents of a BLOB or a numeric conversion.

An array exposed as a virtual table

The released carray extension supplies a concrete use. It exposes the contents of a C array as rows. Its SQL interface accepts a pointer argument and an element count, with an optional argument selecting the element type.

For a statement already prepared as SELECT * FROM carray(?1, 3), the native binding can use the new interface as follows. This fragment assumes that the extension has been registered and that stmt is the prepared statement. The static array remains alive for the statement execution.

static int values[] = {53, 9, 17};
int rc = sqlite3_bind_pointer(stmt, 1, values, "carray", 0);

The "carray" string is the pointer type expected by this extension. It is distinct from the optional SQL argument naming the array element type. The extension supports int32, int64, double and char*, with int32 as the default. Choosing how to interpret array elements and verifying how the pointer arrived are separate operations.

The count also remains meaningful. A native pointer does not by itself describe the length of the allocation it references. Passing the typed pointer therefore does not replace the caller's responsibility to supply the correct element count. This interface improves the origin and interpretation of a reference without turning C memory into a bounds-checked SQL container.

Changes around the native API

Version 3.20.0 also introduces sqlite3_prepare_v3() and its UTF-16 counterpart with preparation flags. The new SQLITE_PREPARE_PERSISTENT flag helps avoid misuse of lookaside memory by long-lived statements in the full-text and R-Tree extensions. This is another adjustment to how extensions cooperate with the core engine, though it addresses allocation behavior instead of pointer trust.

The release adds statement-status counters for repreparations, runs and memory use. It also adds a statement virtual-table extension. These additions make native statement behavior more observable. They should remain distinct from the pointer changes, which alter how a native reference travels through execution.

Other SQL changes have their own limits. Date and time functions may now appear in CHECK constraints, expression indexes and partial-index predicates when they do not use now, localtime or utc. secure_delete=FAST overwrites deleted content when doing so requires no additional I/O, but the notes explicitly allow deleted content to remain on free-list pages. Neither change warrants a broader guarantee than the release describes.

The pointer interface is the part that most directly affects extension authors upgrading this release. Existing integrations with FTS5 API discovery or pointer-taking extensions must use the new binding convention. That change lets native consumers require a matching pointer representation before they interpret a value as an address inside the application process.