Perl exposes regex programs before optimization

A dump of an optimized regular expression can show the compiler's final output while hiding the stage at which it went wrong. Perl 5.31.4 adds a view of the compiled pattern before optimization. Its release notes describe the facility primarily as a tool for finding bugs in the regex compiler. The September 20 release makes that view available through the re pragma as well as through debugging builds.

The distinction is useful because compilation and optimization perform different work. A wrong compiled form can already exist before optimization begins. Alternatively, the earlier form can represent the pattern correctly and a later transformation can introduce the defect. Seeing both gives a developer evidence about where to investigate.

I prefer Perl for short text transformations because its regex syntax expresses much of the work directly. That preference makes the implementation's inspectability valuable. An unexpected result can otherwise leave the author examining a source pattern without any visibility into how the interpreter has represented it.

The focused dump needs no debugging build

The release notes provide a pragma form that works on any Perl build carrying this release. It selects the new dump specifically.

use re qw(Debug DUMP_PRE_OPTIMIZE);
my $pattern = qr/foo|bar/;

The pattern is only an example to compile under the pragma. Its alternation is not a claimed reproduction of a compiler bug. The significant part is that the lexical scope includes the compilation of the expression whose earlier representation is needed.

The notes distinguish this focused form from two broader options. A Perl built with DEBUGGING can enable the output with the -Drv command-line switches. The pragma combination Debug COMPILE EXTRA also includes it on any build, but displays other information too. Selecting DUMP_PRE_OPTIMIZE is the route described for getting this particular additional view without the broader output.

The re module documentation at the release tag defines the new option as dumping the compiled pattern before the optimization phase. It also explains that re debugging directives are lexically scoped while having both compile-time and runtime effects. Placement therefore matters, especially for a pattern compiled in a different scope from the eventual matching code.

A view taken at the wrong point cannot answer the intended question. If the relevant expression was compiled outside the scope where the option applies, turning on matching diagnostics later does not describe that earlier compilation in the way this facility is intended to do. The source example keeps compilation beside the directive so that relationship is visible.

Broad debug shortcuts are not identical

The module documentation distinguishes several debugging shortcuts. ALL enables many options but explicitly excludes DUMP_PRE_OPTIMIZE, along with some other unusually verbose choices. Combining ALL and EXTRA is the documented route to every option. The spelling of a broad shortcut therefore cannot be taken as proof that this new view is present.

That matters when comparing diagnostic output from two reports. A report made with a familiar general debugging mode may lack the newly available pre-optimization stage. A report that selects the option explicitly states which representation was requested, reducing ambiguity about what the dump is supposed to show.

More output does not automatically provide a better comparison. The module describes some options as producing very large amounts of information, including capture-buffer and stack-related diagnostics. The new focused choice is useful because it answers a specific compiler question. It need not include all execution details to expose the earlier compiled pattern.

For my reading of a bug report, that explicit choice is preferable to an unspecified debug transcript. It identifies the stage being observed without requiring every reader to reconstruct a collection of default options. The benefit is a clearer relationship between the diagnostic evidence and the suspected compiler transformation.

Optimization already exposes derived information

The same re documentation describes regmust, which reports the optimizer's view of required fixed substrings. It distinguishes an anchored substring at a known offset from a floating substring that can occur within a range of positions. Its example uses here followed by a variable span and then there.

That interface explains why a pattern's optimized representation contains more than a direct copy of its source. The optimizer derives information that can guide matching. Such derived facts must remain consistent with the pattern's meaning. Inspecting the compiled form before those transformations gives compiler developers another point of comparison when that consistency fails.

The documentation qualifies regmust's result as what the current optimizer believes, rather than a definitive answer about the longest possible required string. That distinction is appropriate for diagnostic data. An inspection interface exposes implementation decisions so they can be examined, and it should not turn those decisions into a new language guarantee.

The new dump has the same practical audience. Most authors need the regex to match correctly without reading compiler internals. When that contract appears broken, the earlier representation helps the people investigating it determine which stage needs attention.

This release also repairs a matching leak

Perl 5.31.4 fixes a leak when a string without the SVf_UTF8 flag is matched against an expression containing Unicode literals. The release notes say each match attempt could leak a scalar value. They do not describe the new dump as the tool that found this bug, so the two changes should not be joined into an invented discovery story.

A separate correction reverts an unintended string-to-number conversion change from 5.30.0. Strings beginning with 0x or 0b no longer receive hexadecimal or binary interpretation during that implicit conversion. This is distinct from writing a numeric literal in Perl source and provides another reason to name the operation precisely when describing behavior.

The release states that it has no intentionally incompatible changes relative to 5.31.3. Its new regex output is an inspection capability, not a new matching rule. A developer can now observe the compiled program before optimization and compare it with later diagnostic information when investigating whether a transformation preserved the original expression.