Ruby's OpenSSL wrapper could turn a positive comparison into equality

A comparison wrapper can change a security decision even when the underlying library returns the right answer. Ruby's OpenSSL extension normalized a native name-comparison result using an incorrect positive threshold. A result of one could fall through to the branch that returned zero.

The October 17 advisory for CVE-2018-16395 describes incorrect equality in OpenSSL::X509::Name. In the reported case, a right-hand name entry beginning with the left-hand entry's value could compare as equal when the full values differed. A malicious certificate could consequently receive an incorrect name-equality result.

The OpenSSL extension's fixed implementation makes the mechanism visible. It calls X509_NAME_cmp, then converts a negative result to minus one, a positive result to one and zero to zero. The previous implementation checked for a result greater than one in the positive branch.

I find Ruby's common object conventions helpful because operations such as equality should have a consistent meaning. A native extension has to preserve that meaning at its boundary with the underlying library. A one-character comparison error can invalidate the contract without producing an exception.

Normalization must preserve all three outcomes

The 2.1.1 implementation returns minus one for a negative native result and one only when the result exceeds one. Every remaining result returns zero. That incorrectly groups the positive value one with equality.

The corrected test is greater than zero. It preserves the three outcomes regardless of the magnitude of a positive native result. A wrapper normalizing an ordering result should classify its sign, not assume that positive results always exceed a particular value.

This explains why the failure can be silent. The native call returns an integer and the wrapper returns an allowed Ruby comparison value. The problem is that the wrapper assigns the result to the wrong semantic category. No malformed return type is needed to expose it.

The extension binds this normalized method as cmp and aliases <=> to it. It also includes Comparable. The comparison participates in Ruby's ordinary comparison model, so a wrong zero is consequential beyond a single explicitly named native function.

The reported prefix relationship supplies a concrete case where distinct values can reach the mistaken result. It should not be generalized into a claim that every pair of unequal names compares equal. The threshold defect requires the particular native result that the wrapper misclassifies.

Name comparison has a narrow subject

OpenSSL::X509::Name represents a structured name with entries such as common name and country. The advisory discusses equality between these name objects. It does not claim that comparing names establishes every condition needed to trust a certificate.

An application can use such a comparison as one input to a trust decision. If the comparison incorrectly returns equality, that application's policy may accept a relationship that the names do not actually satisfy. The consequence depends on where and how the application consumes the result.

That is different from asserting a universal bypass of all TLS verification. The advisory identifies the extension's name-equality operation and the possibility of an incorrect comparison with a malicious certificate. The source supports that specific mechanism.

The operand order also matters to the reported description. The advisory distinguishes the receiver on the left from the argument on the right and describes the right-hand entry starting with the left-hand value. A reproducer should retain that direction rather than assuming a symmetric implementation failure.

This is especially significant for equality, whose intended meaning should not depend on which operand is written first. An ordering-wrapper bug can violate expectations that application code ordinarily takes for granted. The corrected sign normalization restores the result classification at its source.

Similar-looking equality methods take different paths

The tagged implementation has a separate eql? method. It checks the argument's class and compares the direct X509_NAME_cmp result with zero. It does not use the normalized wrapper's greater-than-one test.

That distinction helps localize the defect. Two methods associated with equality can reach different code. A successful test using only eql? does not establish that cmp or <=> classifies all positive results correctly.

The extension also defines a hash method separately through the native name-hash operation. The advisory is about the flawed equality comparison, so it would be inaccurate to describe every name operation as broken. Understanding which method uses which helper avoids that overstatement.

My preference is for regression evidence to exercise the public operation implicated by the advisory. An implementation can preserve the direct equality helper while still returning zero through the ordering wrapper. Those paths need separate expectations.

The source documentation permits constructing a Name from a distinguished-name string or an array of entries. That makes it possible to isolate name comparison from certificate loading in a focused regression case. Such a case can preserve the differing entry values and check the comparison result without involving a network connection.

Remediation depends on the Ruby series

The advisory lists Ruby 2.3.7 and earlier, Ruby 2.4.4 and earlier, and Ruby 2.5.1 and earlier as affected within their respective series. It also includes Ruby 2.6 preview 2 and earlier. These are the bundled extension versions covered by the notice.

For Ruby 2.4 or later, the published workaround installs an OpenSSL gem containing the fix.

gem install openssl -v ">= 2.1.2"

The advisory treats Ruby 2.3 differently. It says that series cannot override its bundled OpenSSL through the gem, so users need to upgrade the Ruby installation. The workaround is therefore conditional on the interpreter series.

Updating a system's native OpenSSL library alone does not describe the source-level correction shown here. The mistaken test is in the Ruby extension's wrapper around X509_NAME_cmp. The relevant replacement must include that wrapper fix.

The fixed implementation classifies every positive result as greater, instead of allowing one to become zero. Applications depending on name comparison then receive the distinction the native comparison reported. The update instructions identify how to obtain that corrected extension through the supported path for each Ruby series.