Perl regex code blocks can now preserve engine optimizations
Perl 5.37.8 adds embedded regex code forms that leave engine optimizations enabled. Their execution frequency explains the tradeoff between observing a matcher and preserving its performance.
Putting a debugging callback inside a regular expression can change the algorithm being debugged. Perl's existing embedded-code forms disable some optimizations across the entire enclosing pattern. The release notes describe cases where that changes a linear operation into a quadratic one. A callback intended to explain matching can therefore make the matcher do substantially different work.
Perl 5.37.8, dated January 20 in the Perl release history, adds alternative forms that preserve those optimizations. The release notes shipped with this version call this optimistic evaluation. The optimism concerns permission to optimize the enclosing pattern. It does not promise a particular number of callback invocations, and it does not make embedded Perl code a pure operation.
I still prefer Perl for short text transformations because matching and substitution belong to the language itself. This change interests me because it exposes a performance consequence of that integration. Once a pattern contains executable Perl, the optimizer has to operate around effects that an ordinary literal match does not have.
Two forms with different return values
The existing (?{ ... }) construct executes a Perl block during matching. Its optimistic counterpart is (*{ ... }). The version's regex reference says the latter has the same behavior except that it leaves engine optimizations enabled. In an ordinary embedded-code assertion, the block's result goes into $^R. That assignment has localized behavior when the engine backtracks.
The other existing form, (??{ ... }), uses its block's return value as a subpattern. A returned string is compiled as a pattern, while a returned qr// object supplies an already constructed pattern. Perl 5.37.8 adds (**{ ... }) as the optimistic equivalent. The doubled punctuation retains the distinction between executing a block and obtaining a pattern from a block.
This distinction matters when reading unfamiliar code. Replacing a question mark with an asterisk changes the optimization contract. Adding a second question mark or asterisk changes what the engine does with the block's result. Those are separate decisions, even though the spellings differ by one character.
For the postponed-subpattern forms, the reference also explains capture scope. The inner subpattern has its own captures during its match, and those captures do not become the enclosing pattern's captures afterward. Embedded code can inspect enclosing captures while choosing the inner pattern. Keeping those two scopes separate prevents a dynamic subpattern from silently changing the meaning of the outer capture numbers.
Execution count is part of the contract
The regex reference has a section specifically about embedded code execution frequency. It leaves the exact invocation rules unspecified for the older forms too. Optimistic forms permit more variation because they preserve optimizations that the older forms suppress. A failed match may execute an optimistic block zero times.
That qualification rules out using a callback count as a portable measurement of logical candidates. The engine may reject an input through an optimization before reaching the embedded block. A counter then records executions on the engine's chosen path, not every position that a conceptual step-by-step matcher might examine. A different release can choose another path without changing whether the pattern matches.
Successful matches have more structure. The reference describes code executing in the appropriate order on the accepting path. Its quantified example executes the embedded block for each matched iteration. However, failed alternatives and unsuccessful searches do not acquire a stable callback schedule merely because a successful match has comprehensible ordering.
An external side effect therefore needs particular care. Writing a record from inside a pattern ties that write to regex execution. It does not tie it to the application's eventual decision to accept an input. If the requirement is one record for each accepted line, placing that operation after the successful match expresses the requirement directly. This follows from the documented execution model and does not depend on a particular optimizer implementation.
Backtracking restores some state
Perl provides specific tools for state that should follow backtracking. The reference explains that local changes made inside a code assertion accumulate along the match and unwind when the engine backtracks over them. They also unwind at the end of a successful match. $^R has corresponding localized behavior, so backtracking restores its earlier value.
These rules do not turn arbitrary effects into reversible operations. A lexical declaration introduces its normal scope, while localization has separate behavior inside the pattern. The documentation illustrates the distinction with a counter whose localized value follows the accepted repetition count after the engine backs out of earlier repetitions.
I regard that distinction as the main reading burden in embedded regex code. A reviewer needs to know whether a value belongs to the current matching attempt or represents an application action that should survive the match. The new syntax preserves this distinction while allowing the optimizer more freedom about whether an unsuccessful attempt reaches the block.
Where the performance claim stops
The release notes give an asymptotic example, not a universal speedup. Some patterns lose optimizations when they contain (?{ ... }), turning work that normally grows linearly with input length into quadratic work. Switching to the optimistic equivalent preserves the optimization for those patterns. The notes do not claim that every callback-bearing pattern has that problem or receives the same improvement.
That makes the change useful for code that can tolerate the documented invocation model. A block that supplies a subpattern must still supply the intended pattern whenever it executes. A block used to observe matching must accept that it observes actual engine execution, including paths skipped by optimization. Neither use gains a promise that execution counts remain stable between interpreter versions.
The regex reference also distinguishes literal embedded code from code text derived at runtime. Runtime-derived code requires use re 'eval' in scope because otherwise a supplied pattern could introduce executable snippets. The optimistic spelling changes optimization behavior within this embedded-code system. It does not establish a new trust boundary for externally supplied pattern text.
For ordinary one-off text work, I prefer to keep application actions outside the expression when the successful match already supplies the needed decision. Embedded code remains available for patterns that require it. Perl 5.37.8 gives those patterns an explicit way to retain optimizations, with the execution-frequency rules documented alongside the syntax.