Cassandra prepares for 4.0 through operational changes

Cassandra's January community update places release testing beside a discussion about how the database should run on Kubernetes. The connection interests me because Cassandra's appeal depends on its behavior across machines. Its attraction for active-active systems does not make it an automatic architectural choice. Operational complexity remains part of the decision, and the pending 4.0 release contains changes that make that complexity more explicit.

The January 19 project roundup reports that 4.0-beta4 shipped on December 30. Testing toward a release candidate is underway. It also reports that the Kubernetes special interest group is discussing a wider scope and work to merge operator efforts. Those are statements about work in progress. Neither establishes a finished operator agreement or a generally available 4.0 release.

The useful technical context is already available in the beta4 NEWS file. Several changes concern moments when a node appears ready to an external controller but still needs coordination with the database around it. Others concern moving existing data safely through an upgrade. Together they explain why testing a database release requires more than confirming that its process starts.

Starting a node includes connecting to its peers

The beta4 notes describe a new default startup behavior. Nodes establish intracluster connections and wait for all but one node in the local datacenter to connect and appear UP in gossip. The default wait is ten seconds. The stated purpose is to prevent a node from coordinating requests before it can connect to the cluster fast enough to handle them.

That distinction affects how to interpret startup. A running process and a coordinator able to reach its peers are different conditions. The change puts a bounded peer-connectivity wait inside Cassandra itself. It addresses a particular cause of failed coordination without claiming that elapsed time proves every operation will succeed.

The same notes name block_for_peers_timeout_in_secs as the setting that controls the wait, including whether it happens at all. They also describe block_for_peers_in_remote_dcs, which extends the requirement to all but one node in each remote datacenter. Enabling that option changes which connections can delay startup. The configuration therefore expresses a choice about the scope of readiness, especially for a deployment spanning datacenters.

I value this kind of change because it describes an operational dependency in terms the database understands. A controller cannot infer database readiness from process existence alone. The January operator discussion remains unfinished, but the startup change illustrates the database behavior that any such controller must accommodate.

Streaming has specific eligibility rules

Beta4 also contains whole-SSTable streaming through zero-copy APIs. The release notes identify faster transfers as the purpose. The tagged configuration file supplies qualifications that matter when interpreting that description.

The stream_entire_sstables setting permits eligible SSTables, including their components, to move between nodes through this path. The configuration describes reduced garbage-collection pressure on both sending and receiving nodes. It also says the setting defaults to enabled when omitted. An explicit configuration of that default is:

stream_entire_sstables: true

The same comments limit what this line promises. Internode encryption automatically disables the feature. At this beta, the comments identify Leveled Compaction as a supported use and refer to outstanding work for other compaction strategies. They also state that file transfers remain subject to the outbound streaming throughput setting. A faster transfer mechanism does not remove that configured bandwidth constraint.

Those details make the release's performance claim more useful. A deployment that requires encrypted internode traffic cannot assume it receives the whole-SSTable optimization merely because the setting is true. Similarly, preserving all components of an SSTable during transfer does not guarantee balanced disks. The configuration explicitly disclaims that guarantee. These are properties of the beta implementation, not predictions about what a later release might change.

Upgrade compatibility extends into stored metadata

The beta4 upgrade notes describe a less visible problem with frozen user-defined types. Cassandra 3.0 wrote SSTable serialization headers without the frozen information for these types. Later handling of non-frozen user-defined types makes that missing information consequential. The notes describe deserialization failures, including CorruptSSTableException, when the stored header and the required type interpretation disagree.

The mitigation rewrites the relevant serialization headers when Cassandra detects an upgrade from 3.0. It does not rewrite snapshots or backups. That exception matters because a successful migration of live files does not establish that every retained copy has received the same metadata correction. The notes also describe validation and repair options in sstablescrub for these headers.

Another compatibility requirement concerns added columns in repair-history and tracing tables. Queries against those tables can fail while a cluster contains mixed versions. The notes say 3.0.20 and 3.11.6 pre-add the columns and instruct users to reach those versions or higher before moving to 4.0. Query tracing belongs in this analysis because drivers and cqlsh can issue those queries automatically during the upgrade.

These changes give the January testing effort concrete subjects. Startup needs to account for peer connectivity. Streaming behavior depends on configuration and eligibility. Upgrades must reconcile old metadata and queries issued during mixed-version operation. I remain interested in Cassandra for systems that need its distributed behavior, and this is the level of operational detail that makes that interest useful. At beta4, the documented upgrade path includes intermediate-version requirements and retained files that the automatic metadata migration does not touch.