Dragonfly makes large replication objects resumable
Tagged chunks let Dragonfly resume partial objects after other records enter the stream. The loader retains object state and validates each chunk boundary.
Dragonfly's July 30 engineering article explains tagged chunks for large-collection replication. The format permits other records to enter the stream while a large object is incomplete. It preserves the requirement that a key's baseline data precede journaled mutations affecting it. The article reports doubled foreground throughput and a P99 reduction of about 87 percent in its full-sync test with concurrent writes. Those are results for the described scenario, not a general performance guarantee.
I consider this relevant to my preference for scaling one Dragonfly process before operating a cluster. Larger machines can hold larger objects, but recovery still has to move them while the primary handles requests. I would evaluate foreground service during replication separately from normal steady-state throughput. The serializer and loader changes make that evaluation more precise because they show where progress can pause and resume.
A chunk needs an object identity
The loader implementation recognizes RDB_OPCODE_TAGGED_CHUNK at the top level. It reads a stream identifier and a payload length, each represented as a 32-bit integer. It rejects identifier zero and a zero-sized payload. These checks establish that a chunk refers to a usable continuation and has bytes to consume.
A previously unseen stream identifier begins an object. Its payload enters the ordinary object-loading path, which reads the type and key. If that payload ends before the object is complete, the loader saves a StreamState. The structure contains the key, database index, type, object settings and pending read. Those fields are the context needed to interpret subsequent payload without repeating the initial object header.
When another chunk arrives with a known identifier, the loader retrieves that state and resumes ReadAndDispatchObject. A completed object removes its saved state. An incomplete object retains the updated pending read for another continuation. The implementation therefore distinguishes a temporary end to available payload from the end of the object being decoded.
The database index in that saved state matters. Interleaving allows the surrounding stream to change database context before an earlier object's continuation arrives. The patch changes buffered items to carry their own database index and constructs the database context for each item when loading. A resumed object cannot safely inherit whichever database the outer parser selected most recently.
Chunk boundaries become parser obligations
The loader validates that a chunk's declared payload has been consumed before clearing its active chunk state. It also rejects end-of-file while an active chunk or saved partial stream remains. This prevents a truncated serialization from becoming an apparently complete load simply because the outer parser reached its end marker.
Compression adds another boundary. The patch finishes the current chunk's accounting before switching to decompressed data. Its comment explains that the compressed payload bytes must already be consumed at that point. The decompressed buffer then has its own reading context. Counting decompressed bytes against the compressed envelope would confuse two different lengths.
I would include incomplete input and database-context changes in a format test suite. The implementation's own tests cover a hash split into two chunks and an interleaved database switch. Another test inserts a journal-offset record between chunks. Those cases verify parsing obligations that a simple save-and-load of small strings would never exercise.
The serializer preserves entry state across yielding
The serializer integration marks an entry as split when it flushes midway through serialization. Before calling the consuming function, it saves the buffer controller's state. After that call returns, it restores the entry identifier. This is necessary because the consume call is a point where other serialization work can run.
Tagging is selective in this patch. A nonzero active identifier denotes a data entry, and the controller adds an envelope only when that entry has split and the current bytes are nonempty. Journal items use the zero identifier in the controller's representation. The code also preserves previously finished data as a prefix when constructing an output blob, with an explicit ordering comment explaining that choice.
The patch handles compressed tagged output as complete blobs. Its legacy path avoids compressing the latter half of a split object as though that half were a new top-level record. The two paths therefore retain different requirements about where ordinary object parsing can begin. I would inspect the actual serialization configuration when comparing results, rather than assuming every transfer uses identical framing.
A list-loading correction illustrates another consequence of partial state. The loader previously considered converting a newly loaded single-node list into a compact listpack. The patch avoids that conversion when further chunks may arrive. An object that looks small after its first chunk is not necessarily small when complete, so early representation choices must account for continuation.
My acceptance test would combine large collections with ordinary foreground writes during a full sync, then check the final replica data. I would inspect latency during the transfer and confirm that interrupted or malformed chunk input fails explicitly. The format can permit more serialization work to proceed between chunks, while the loader still has to preserve object identity and reject unfinished data.