February 5, 2023·6 min read·engineeringdragonflyreplicationfailover

Dragonfly adds Redis Sentinel compatibility

Dragonfly 0.15 adds the protocol behaviour needed for Sentinel-managed failover. Its integration tests show how discovery and promotion fit together, and where the evidence remains limited.

A replacement for a Redis server has to satisfy more than application commands if the surrounding system manages failover. Monitoring software needs to discover replicas, identify the current primary, and change replication roles. Dragonfly's 0.15 release, published on February 5, adds compatibility with Redis Sentinel. This extends compatibility into the software that manages the server when its role changes.

The release includes a change to report replica IP addresses and listening ports in INFO REPLICATION. That detail explains part of the work behind the announcement. A monitor cannot promote a replica that it cannot discover or address. Exposing the right topology information belongs to the same operational contract as accepting a replication command.

I value Dragonfly's approach to using a larger machine because it can simplify cache deployment. A single process still needs an answer to losing that machine. Sentinel compatibility makes the existing Redis management ecosystem relevant to that question, while leaving replication and client reconnection as parts of the overall system that still have distinct responsibilities.

Discovery precedes promotion

The most concrete account of the intended interaction is the Sentinel integration test at the release tag. It starts Dragonfly instances separately from a Redis Sentinel container. The monitor is an external component. Dragonfly does not become a Sentinel process by enabling its own replication support.

The test initially tells Sentinel the host and port of the primary. A second Dragonfly instance becomes a replica through REPLICAOF. The test then waits until Sentinel reports one replica for that monitored deployment. This sequence makes discovery an explicit precondition of failover instead of assuming that creating a second process automatically makes it a candidate for promotion.

There are two different views of the deployment in this arrangement. The Dragonfly processes maintain their replication roles and data. Sentinel maintains information about which endpoint currently acts as primary. An application using Sentinel discovery needs the second view to find its write destination after a role change. An application connected directly to a fixed port has not acquired that behaviour merely because the server supports Sentinel.

The integration test asks Sentinel for the primary address with GET-MASTER-ADDR-BY-NAME. A corresponding inspection command, using the deployment name and port from the test fixture, is:

redis-cli -p 5555 SENTINEL GET-MASTER-ADDR-BY-NAME my_deployment

That command targets the monitor. Data commands target the Dragonfly endpoint that the monitor reports. The distinction matters when reading diagnostics because successful communication with Sentinel says nothing by itself about whether a particular application connection points at the current primary.

What the failover test establishes

The first test requests a failover through Sentinel and waits for the reported primary port to change to the replica's port. It then checks that Sentinel lists the original primary as a replica. These assertions cover the monitor's view of both sides of the role reversal. Promotion has to update more than the endpoint returned to clients.

The test also exercises data after the reversal. It writes a key to the newly promoted process, then waits until the original primary can read that value through replication. This is a stronger check than accepting a promotion command or observing a changed role string. It connects the topology change to a subsequent data operation across the new replication direction.

A second test stops the original primary process. It waits for Sentinel to report the remaining instance as primary and verifies that the promoted instance accepts a write. This covers a different trigger from an explicitly requested failover. One case changes roles while both processes exist, while the other removes the original write destination.

Both tests carry a skip marker in this release because they fail in CI. That is part of the evidence available today. The source describes intended scenarios in executable form, but the tagged configuration does not demonstrate that these scenarios run successfully in every CI execution. The release announcement and the state of its tests should be read together.

The fixture also uses a single Sentinel with quorum set to one and a short failure-detection interval. Those choices make a compact integration environment. They do not establish a deployment prescription for independent failures or a general bound on recovery time. The test waits for observations to change, and its timeout values belong to the test rather than to a service-level guarantee.

Replication has work beyond role changes

The release history includes support for blocking commands in replication and changes to automatic expiry. These details matter because a replica has to reflect data operations as well as its assigned role. A topology monitor can identify where writes should go without proving that every command path produces the required replicated state.

This is a useful boundary when interpreting compatibility. Sentinel discovery and promotion concern control operations. Replication concerns the contents of the database. An application library adds its own behaviour when it discovers that a connection is no longer suitable. Treating these as separate mechanisms makes an operational explanation more precise than calling the whole arrangement highly available.

For example, the test's post-promotion write checks the new replication direction with a single key. It does not claim that a write interrupted during failover has a particular outcome. Nor does it prove that an arbitrary client library will discard an existing connection and rediscover the primary. Those are different observations, requiring evidence from the paths that implement them.

I find the external-monitor model appealing because it preserves a familiar division of responsibility. Dragonfly supplies the datastore and replication behaviour, and Sentinel supplies the monitoring interaction demonstrated by the fixture. The value of compatibility is that the surrounding system can retain established interfaces while the datastore implementation changes.

The same release adds a native rate limiter

Another addition illustrates how compatibility reaches beyond the standard command set. Dragonfly implements CL.THROTTLE, the API associated with redis-cell. The string-command implementation documents its arguments and the five integers in its response. Applications can request admission and obtain retry information through one command.

The arguments identify a key, maximum burst, count per period, and period length. An optional quantity defaults to one. For example, the following command requests one unit with a count of thirty per sixty seconds and a maximum burst argument of fifteen:

redis-cli CL.THROTTLE api-limit:customer-42 15 30 60

The implementation calculates the reported total limit as maximum burst plus one. That arithmetic is worth stating because a caller that treats the burst argument as the returned limit will interpret capacity incorrectly. The response begins with a decision indicator, followed by total and remaining capacity. Its final fields describe retry and reset timing in seconds.

Internally, the command computes an emission interval in milliseconds and uses a stored theoretical arrival time. It compares the time implied by the requested quantity with the allowed delay tolerance. This representation lets the command account for rate consumption through time arithmetic rather than storing a separate record for every admitted request.

The parser rejects a zero count and guards arithmetic against overflow. It also rejects an emission interval that rounds down to zero. These checks mean that a syntactically numeric configuration can still be invalid for the implementation. The rate is expressed using integer milliseconds, so the relationship between period and count is part of the supported configuration.

The command runs through the transaction's single-hop scheduling path for its key. That places the decision and associated state change in the datastore command implementation. A caller does not need to implement its own read-modify-write sequence merely to reproduce this API. The returned retry information is available in the same response as the admission decision.

Compatibility includes packaging and application behaviour

The release also supplies Debian packages for both x86_64 and aarch64 and includes Lua fixes described as improving Laravel compatibility. These changes affect different adoption barriers. Packaging determines how the binary enters an environment, while Lua behaviour determines whether an existing application can execute its datastore logic correctly.

The Sentinel work belongs to that same practical definition of replacement. A datastore with the required data commands can still be difficult to substitute if topology inspection or role changes behave differently. Version 0.15 makes those management interactions explicit through replica reporting and the Sentinel test scenarios.

For a deployment that already uses Sentinel, the relevant new capability is the connection between reported topology and Dragonfly's replication roles. The tagged tests follow that connection through discovery, promotion, and a write after promotion. Their skipped status remains a concrete limit on what this release's automated evidence establishes.