Perl fixes Unicode folding across regex node boundaries

A case-insensitive pattern can stop matching correctly because its compiled representation needs another internal node. That is the failure repaired in Perl 5.29.7, released on January 20 according to the Perl release history. The release notes shipped with this version describe an off-by-one error in the mechanism that splits literal strings across regex nodes. The difficult part is preserving a Unicode match that can involve several characters.

This is a useful correction for a language whose regular expressions are part of everyday syntax. I still prefer Perl for one-off text transformations because matching and substitution fit directly into the expression doing the work. That convenience depends on the compiler preserving the meaning of the pattern when its representation changes. An internal limit should not become an accidental condition on which text the expression recognizes.

A folded character can occupy several positions

Case-insensitive matching needs more than changing ASCII capital letters to lowercase. The Unicode documentation in this release explicitly distinguishes full case folding from simple case folding. Its example maps U+1F88 to the sequence U+1F00 U+03B9 under full folding. Simple folding would use U+1F80 instead. A relationship between one character and several characters therefore belongs to the matching rules themselves.

That distinction changes what a regex compiler may safely split. Suppose the compiler has collected a run of literal text for a case-insensitive match. Splitting that run into pieces is an implementation decision. If a possible multi-character fold crosses the split, treating each piece independently can lose a valid match. The compiler has to preserve enough context around the boundary for the full folding relation to remain available.

The release notes identify the relevant representation as an EXACTFish regnode. Each such node can hold only a finite length of matching string. A longer string needs a following node, and sufficiently long patterns need several. This representation is not itself a language feature, so a Perl author should not need to know its capacity to write a correct expression.

The repair concerns the choice of boundary. When the proposed split could occur inside a multi-character fold, the compiler backs up by one or more characters and emits a shorter node. That adjustment leaves the affected text together for matching. The backing-up calculation had an off-by-one error. Correcting it restores the compiler's intended treatment of the boundary without requiring an application to rewrite the pattern.

Pattern length can expose a semantic error

The release notes do not describe every case-insensitive Unicode expression as broken. They describe a specific interaction between finite node storage and the adjustment for folds. A short example of Unicode equivalence explains the requirement, but does not establish that it reproduces this bug. The failing condition also involves the position at which the compiler divides a longer string.

This matters for understanding a regression report. An application can construct a larger pattern from ordinary pieces, with the troublesome boundary appearing only after those pieces are combined. The individual fragments can be valid and their small tests can pass. The compiled form of the combined expression can nevertheless reach the faulty calculation. Reducing the pattern without preserving that condition can make the failure disappear.

The important invariant is that the recognized text must not change because a literal run exceeds the storage available in one node. A compiler can choose different internal layouts for short and long expressions. Both layouts still have to implement the same folding rules. Here the boundary adjustment is part of making those representations equivalent, so its arithmetic directly affects observable matching behavior.

My preference for concise text-processing expressions makes this kind of fix more interesting than another syntax addition. A compact expression delegates substantial work to the runtime. Confidence in that expression includes confidence in the less visible cases, especially when Unicode makes the relationship between characters and positions less direct.

The surrounding release keeps compatibility in view

The same release notes report small optimizations involving character folding and character classes. They give no benchmark result for those changes, so this release does not justify a numerical performance claim. Correctness at a node boundary and faster handling of ordinary cases are separate properties. The former has a stated failure mechanism, while the latter is described only as a set of small optimizations.

There is also a relevant compatibility decision for XS authors. A deprecation affecting several UTF-8 macros had been scheduled to become fatal in 5.30. The notes move that deadline to 5.32 because problems appeared with some CPAN modules. That is a schedule stated by this development release, not a claim about what a future release has already shipped. It shows why tightening interpreter internals can require coordination with extension code.

Other corrections have similarly specific contracts. A bare eof call with no previous filehandle now returns true. A failure while compiling a format now aborts compilation, because continuing after that sub-parse could leave the parser in a state that might crash Perl. These are distinct repairs, but they share the need to keep an internal condition from producing an unintended result in user code.

Perl 5.29.7 is a development release, and its delta describes changes since 5.29.6. Its regex correction is therefore best understood at that scope. The documented result is a repaired boundary calculation for long folded strings. It does not promise a different Unicode model or a new matching operator. The existing model already requires full folding, and the compiler now handles this particular division of its internal representation correctly.