RDoc replaces a file-opening API that could execute commands

Reading a project's files for documentation should not interpret a filename as a command. Ruby's May 2 advisory for CVE-2021-31799 describes an RDoc path where that distinction failed. A filename beginning with a pipe character and ending with tags could reach Kernel#open, causing command execution when a user ran RDoc over a malicious project.

The affected operation belongs to documentation input filtering. It does not require the user to intentionally run the project's application code. The advisory identifies RDoc releases 3.11 through 6.3.0 as affected and gives 6.3.1 as the corrected version. The attack depends on the naming and processing of a file, which makes the choice of opening API consequential.

I like Ruby's concise file-handling code, but an interface that accepts both paths and process commands demands care at a trust boundary. Documentation tools receive repository-controlled names. Those names should remain data when the tool decides whether a file is suitable to parse.

A preliminary read reached the dangerous behavior

The security fix changes one call inside remove_unparseable. RDoc filters files with several known extensions and examines names ending in tags. For those names, it opens the input in binary read mode and inspects the first hundred bytes for recognized tag-file signatures.

The preliminary inspection is the relevant operation. The tool is deciding that a file should not be treated as ordinary documentation input, yet the attempt to inspect it can already invoke behavior outside a file read. That puts the vulnerable action before the later parser choice.

The original call used unqualified open, reaching Kernel behavior. The corrected line explicitly uses File.open with the same filename, mode and block. This narrows the operation to the resource kind the code intended to access. The repair does not need to redesign the tag-file signature patterns because the command interpretation occurs when opening the input.

That is a useful distinction when reading the diff. The bytes being inspected are not what create the subprocess behavior. The input string passed to the opening method selects that behavior before the read. Limiting how many bytes the code reads cannot correct the choice of resource-opening API.

Kernel open deliberately supports subprocesses

The behavior is documented in Ruby 3.0.1's implementation and API comments. A path beginning with a pipe tells Kernel's open operation to create a subprocess. The resulting IO object connects the caller to the process. This is an intentional interface capability, not a new behavior introduced by the RDoc defect.

That capability can be useful when the caller intentionally wants to process command output through an IO interface. It becomes a vulnerability when a repository filename controls the same argument and the program assumes that the operation can only access a file. The mistake is an incorrect assumption about the API's accepted meanings.

Binary read mode does not remove that alternate interpretation. Nor does providing a block make the initial choice safer. A block can manage the lifetime of the returned resource while leaving the wrong kind of resource creation intact. The security property needs to apply before the resource is opened.

The corrected RDoc call preserves the ordinary file-reading structure. Its block still reads a bounded amount of input for classification. Explicitly selecting File identifies the intended operation where the externally controlled name enters the API.

The regression checks for unintended output files

The patch adds a test that creates a specially named file in a temporary directory. It then passes the filename through remove_unparseable and checks the returned list. More significantly, it checks the directory contents afterward to establish that no additional file was created by a command hidden in the name.

The returned classification alone would not prove that the operation had no side effect. A command could execute and still leave the surrounding filter with a plausible result. Looking for the unwanted artifact makes the security consequence observable. The test connects the API substitution to the behavior that must stop occurring.

The test also confirms that a literal file with the unusual name remains an input the method can handle. The fix does not depend on declaring every name beginning with a pipe invalid at the filesystem level. It keeps the filename literal by choosing an API whose purpose matches the intended operation.

I consider that a clearer repair than trying to maintain a growing list of filename characters to escape. The code needs a file object. Stating that requirement through the receiver removes the unintended process interpretation at the call itself.

A bundled documentation tool can need its own update

RDoc ships with Ruby, but the advisory provides a gem update route. It recommends installing a corrected RDoc and tells Bundler users to set a minimum version in their Gemfile. The relevant dependency declaration is straightforward.

gem "rdoc", ">= 6.3.1"

The declared requirement and the RDoc executable actually invoked need to agree. A project can use a bundled tool or a separately installed gem, and an application's dependency selection does not describe every command run outside that selection. The fix concerns the documentation generator processing project paths, so the version of that generator is the relevant one.

This also separates the documentation task from assumptions about source execution. Deciding to inspect a repository's comments does not mean deciding to execute a command encoded in one of its directory entries. The vulnerability crosses that boundary during a metadata-like filtering step. A review focused only on parsers that evaluate source would therefore miss the affected call. The literal filename reaches a general-purpose IO interface before the tool has decided whether the file belongs in the documentation set, which is the specific path corrected by the patch.

The advisory credits Alexandr Savca for reporting the issue. Its stated threat is command execution against a user attempting to generate documentation for a malicious Ruby project. The corrected call keeps that project's filename inside the file-access operation intended by the filter, before any documentation parsing occurs.