Perl 5.33.3 repairs recursive matches and exposes optimizer data

A successful regular expression match can still capture the wrong part of its input. The recursive-pattern bug repaired in Perl 5.33.3 demonstrates that distinction with an expression containing balanced parentheses. The engine accepted a match whose capture extended into a parenthesized term in the wrong way. October's snapshot repairs that behavior and separately adds a way to inspect facts discovered by the regex optimizer.

The 5.33.3 release notes identify both changes. The history file in the release records October 20 as its date. These changes matter to different audiences. Application authors need correct captures, while interpreter developers need inspectable compiler decisions that can support regression tests.

Recursive calls must retain the intended capture boundaries

The original bug report uses numbered recursive subpatterns to match part of an expression-like string. For the input a + b + (c + d), its author expects the first capture to contain a + b. The reported interpreter instead produces a capture extending into the last parenthesized term. That is an incorrect result, without requiring a crash or a failed compilation to make the problem visible.

The important distinction is between the whole match and a capture returned by it. A program can use a trailing operator to establish where a partial expression ends, then retrieve the preceding expression from a capture. If recursion and backtracking leave that capture with the wrong boundary, merely checking that the match succeeded will not reveal the failure. The returned substring is part of the interface the application depends on.

The release notes describe the repair narrowly as a fix for some regexes with recursive subpatterns matching incorrectly. They do not claim that every recursive-pattern problem has been removed. The report's specific input and expected capture provide a concrete regression case. Generalizing it into a statement about all parsers built with regexes would go beyond that evidence.

I value Perl's integrated matching syntax, including its ability to keep capture extraction close to a transformation. That preference makes capture correctness a first-class concern for me. A true match result is insufficient when the next expression consumes a substring the engine recorded along the way.

Compiler observations become available to tests

The updated re module adds optimization, which accepts a compiled regex and returns a hash reference containing optimization information discovered during compilation. Its documentation shipped with the snapshot states the purpose directly. The function makes it possible to write tests around that information. Passing something other than a compiled regex returns undef.

use re qw(optimization);

my $compiled = qr/abc/;
my $details = optimization($compiled);
print $details->{minlen}, "\n";

Here minlen is the least number of characters in any string that can match the compiled expression. The documented hash also includes minlenret, which describes the minimum number of characters in the matched substring. These quantities can differ because an assertion can require input characters without including them in the consumed match. The distinction gives tests more precise observations than one generic minimum-length value would provide.

The documented result also includes information about anchored or floating substrings that every match must contain. Byte and UTF-8 forms have separate entries and conditions. This allows a test to inspect the compiler's conclusions about a necessary substring, rather than infer them indirectly from how long a search takes. It does not make an optimizer observation a new rule that application authors should manually reproduce.

The function also reports facts such as whether a pattern is anchored to the start of the string. Its isall value describes whether optimizer information covers the whole expression sufficiently to avoid entering the regex runtime engine. These details expose decisions that otherwise remain behind an application-visible match. They help distinguish the expression's semantics from the particular route the implementation selects to enforce them.

Inspection data has an explicit stability limit

The documentation warns that the hash contents can change as optimizations develop, including between minor versions. That is a material interface constraint. A tool consuming the data cannot treat every key and interpretation as a stable long-term application protocol merely because the function returns an ordinary hash reference. The intended use is closely connected to testing the implementation that produces it.

My preference is for this kind of diagnostic interface to state its limits explicitly. Tests for compiler decisions and tests for returned captures answer different questions, and neither replaces the other. October's release supplies a correction to observable matching behavior alongside a more direct way to test optimization facts. A regression test can now express some compiler expectations without depending entirely on performance measurements or textual debug dumps, while application tests continue to check the captures that their own logic consumes.