rqlite changes its replicated command encoding

The cost of a replicated statement includes the representation that every node must receive and retain. rqlite 5.7.0 changes that representation from JSON to Protocol Buffers and adds conditional compression. The December 24 announcement describes the change as an improvement to write performance and log storage, while retaining the ability to read earlier JSON entries. The encoding decision now has an explicit schema and a compatibility path in the state-machine implementation.

This is a change to replicated commands, whose bytes carry operations through Raft. It does not require HTTP clients to start sending Protocol Buffer messages. The server still parses the public request before constructing the internal command. Distinguishing those interfaces is necessary to understand what the release changes and which programs need to understand the new representation.

I value rqlite's approach of replicating database operations around SQLite. That makes the cost of each represented operation relevant even when the SQL engine remains unchanged. A more compact command can reduce repeated work, provided the reader can still recover the operation's meaning from both new and existing log history.

The command schema separates the envelope and request

The schema in the 5.7.0 tag defines an outer Command with a command type, a subcommand byte string and a compressed flag. Command types distinguish queries, execution and metadata changes. The subcommand carries the message appropriate to that type, while the flag tells the decoder whether those bytes need decompression.

Database requests contain a transaction flag and repeated statements. A statement contains its SQL text and repeated parameters. The parameter message uses a oneof for the represented value, with integer, double, boolean, byte and string alternatives. These definitions make the data carried by a replicated operation visible independently of the Go code that uses it.

A short excerpt shows the statement and request relationship. These are internal protocol definitions, not a new HTTP body format.

message Statement {
    string sql = 1;
    repeated Parameter parameters = 2;
}

message Request {
    bool transaction = 1;
    repeated Statement statements = 2;
}

The SQL and parameter values remain separate. The new encoding does not turn parameterized SQL back into an interpolated string before replication. A repeated statement collection also remains distinct from the transaction flag that describes grouping. The protocol represents those concepts explicitly rather than inferring one from the presence of the other.

Query and execute requests add their own information around the common request. A query message includes consistency level and freshness fields, while both message types can carry timing preferences. This schema therefore represents the request behavior as well as the SQL. The state machine must decode the appropriate message type before passing its operation to the database layer.

Compression is attempted selectively

The request marshaler begins by inspecting the request's statements. Its default batch threshold is five statements. Otherwise it checks whether any SQL string reaches the default size threshold of 150 bytes. Meeting either condition makes the marshaler attempt compression after serializing the request with Protocol Buffers.

These thresholds select attempts, not guaranteed compressed output. The marshaler uses gzip, compares the compressed length with the original serialized length and ordinarily keeps compression only when the result is smaller. If the attempt does not reduce the size, it returns the uncompressed bytes and clears the compression choice. The internal marshaler also has a force option, which can retain compression despite the size comparison.

The SQL-size test is specifically a test of statement text length. It is not a threshold on the complete request including every parameter. A request's parameter data can therefore affect its encoded size without being the text examined by that particular eligibility test. Reading the condition matters when interpreting which workloads receive compression attempts.

The server startup code exposes -compression-size and -compression-batch with those defaults. An illustrative command can state them explicitly. It describes the eligibility policy rather than a target compressed-file size.

rqlited -compression-size 150 -compression-batch 5 ~/node-1

There is a computation tradeoff in attempting gzip. The encoder performs compression before it knows whether the result will be retained. Rejecting a larger result avoids storing that expansion, but does not eliminate the CPU work already spent producing it. The thresholds make the attempt policy a visible part of request encoding.

The store wraps the encoded bytes before Raft

The store's execute path first marshals the request and receives both the bytes and a compression result. It then constructs the outer command with the execute type, those subcommand bytes and the compressed flag. Finally it marshals that envelope and passes it to Raft's apply operation.

That sequence explains why compression metadata belongs in the envelope. The consumer needs to know how to interpret the subcommand before it can decode the request itself. The flag describes those inner bytes, which may be either a direct serialized message or its gzip representation. Command type and compression state together determine the decoding procedure.

On application, the state machine dispatches by command type, decodes the appropriate subcommand and invokes the corresponding database or metadata operation. The subcommand decoder decompresses when the flag is set, then unmarshals the recovered bytes as the expected Protocol Buffer message. Serialization and compression are separate steps on both sides.

I prefer this explicit envelope because it makes the decoding dependency inspectable. The consumer does not need to infer compression from a statement's apparent content or guess which message type the bytes describe. The command records those decisions, and the applying node follows them before executing the request.

Existing JSON history still has a reader

A log-encoding change cannot assume every entry was written by the new release. The store's apply callback first tries the legacy decoder. If that fails, it tries the new command decoder. Failure of both produces an error path rather than treating uninterpretable bytes as a valid database operation.

The legacy decoder recognizes the old JSON envelope and translates its operation into the new command model. For database commands it converts the SQL strings and optional parameter collections into statement messages. It preserves the transaction and timing fields. Metadata commands receive their corresponding typed messages too.

The conversion does not rewrite an old log entry on disk. The legacy implementation marshals the translated subcommand for use by the common apply path and marks that translated representation as uncompressed. Its comment explicitly says this conversion is not going back to disk. Compatibility happens while reading and applying the existing entry.

The decoder also handles history from before parameters existed. An absent parameter collection leaves the translated statements without parameter values. That is a second compatibility distinction within the legacy format. Old unparameterized commands and later parameterized JSON commands can both be represented for the new application path.

The announcement's claim is consequently supported by a specific mechanism. The new binary can read earlier JSON log entries through translation. That is a directional compatibility property. It should not be expanded into an unsupported claim that an older binary can interpret newly written Protocol Buffer commands or that every possible downgrade sequence is equivalent to an upgrade.

Log encoding and snapshots remain different representations

The store still captures SQLite database bytes and JSON metadata for its state snapshot in this version. A snapshot represents the state reached by applied commands, while log entries represent the commands themselves. The move to Protocol Buffers concerns the latter. It does not imply that every persistent byte in the data directory now has the same encoding.

This distinction also clarifies storage expectations. Conditional compression can reduce the bytes used by eligible command payloads, depending on their contents. It does not establish an equivalent percentage reduction for database pages or snapshots. Those represent different material and follow different code paths.

The release announcement reports about a ten percent write-performance improvement from basic testing and says storage savings depend on request compressibility. That is the project's reported observation, not a universal workload result. The implementation explains why results can vary, since batch shape and statement length affect whether compression is attempted and the actual encoded bytes determine whether it helps.

Version 5.7.0 therefore changes both how new commands are written and how old commands join the common application path. Protocol Buffer messages describe the operation, conditional gzip can reduce the subcommand bytes, and the envelope records how to decode them. Existing JSON history remains readable through the legacy translator, allowing the state machine to apply that history using the new typed request representation.