Perl 5.30 permits bounded variable-length lookbehind

A lookbehind assertion can express a condition on the text before a match without including that text in the result. Until this release, Perl required that condition to have a fixed width. Perl 5.30 accepts bounded variable-length lookbehind experimentally, allowing assertions such as an optional character or a bounded repetition. Its May 22 release brings a useful expansion to regex syntax, but the release notes set a maximum lookbehind length of 255 characters.

The interesting constraint is what the engine has to establish before the current match position. A fixed-width condition has one possible starting distance. A variable-width condition can have several, even when its source looks small. Perl's new facility accepts that variability within a bounded range and signals that its precise behavior remains experimental.

I value Perl most when a short expression describes a text transformation directly. A lookbehind can make the condition clearer by keeping preceding context out of the matched result. The new syntax is useful on those terms, provided its length calculation and experimental status remain part of how the expression is understood.

Context does not become part of the result

The regex manual shipped with 5.30 explains positive lookbehind with a word following a tab. The assertion checks the tab while the word constitutes the match. Negative lookbehind reverses the condition, accepting a position only when the preceding text does not satisfy its pattern.

The new release permits variability within those assertions. These patterns adapt the bounded examples in the release notes.

qr/(?<=foo?)bar/
qr/(?<!ba{1,9}r)foo/

The first expression permits the preceding context to end in fo or foo. Its optional o means the lookbehind is not fixed-width. The second expression rejects a match for foo when its preceding context matches b, followed by between one and nine a characters, followed by r. Both have a finite maximum length that falls within the new bound.

The assertion still does not consume text as part of the reported match. A caller interested in bar in the first example does not need to capture it separately merely to exclude its prefix from the result. That is the practical attraction of lookbehind when the requirement concerns the immediate context of a selected substring.

These examples are intentionally bounded. An upper limit on the amount of preceding text is part of the supported facility. The change should not be described as unrestricted backward matching or as permission to place arbitrary unbounded repetition inside an assertion. The compiler must be able to determine a maximum length no greater than the release's limit.

Acceptance comes with a warning

Variable-length lookbehind enables itself when the pattern uses it. It does not need a separate feature declaration, but compilation raises a warning in the experimental::vlb category. The manual says that behavior may change in response to experience, and even describes removal as possible if practical problems cannot be resolved.

That warning is a statement about the status of the construct, not evidence that a particular match failed. It appears at pattern compilation. Code that treats the category as fatal can retain behavior close to the earlier refusal to compile variable-width assertions. Code that accepts the feature is accepting the documented uncertainty about its precise semantics.

This is particularly relevant to captures inside an assertion. The manual warns that a non-greedy modifier inside a capturing group may not produce the expected result. A positive assertion establishing that some suitable context exists is a narrower requirement than using its captures to choose a particular interpretation of that context. The documentation gives a reason to keep those purposes separate.

For my use of small Perl expressions, the clearest early application is an assertion that answers a contextual condition without relying on elaborate captures inside it. That keeps the example aligned with the documented capability. It also avoids presenting experimental capture behavior as a settled interface merely because a simple pattern compiles.

Case folding changes the maximum width

The 255-character limit applies to the maximum length of a possible match, not simply the number of source characters in the assertion. The manual makes this concrete with case-insensitive matching of LATIN SMALL LETTER SHARP S. Under /i, that character can match the two-character sequence ss.

An assertion containing 127 such characters under case-insensitive matching can therefore require 254 characters of preceding text. Adding a 128th can require 256, which exceeds the limit and produces a compilation error. The source has grown by one character, but the relevant maximum match length has grown by two.

This example also explains why a pattern that looks fixed-width at first glance may become variable-width. Unicode full case folding permits some single characters to match sequences of two or three characters. Applying /i can change the set of possible match lengths even without an explicit question mark or repetition range in the source.

A reader therefore needs to consider both the written quantifiers and the matching rules attached to the pattern. Counting literal glyphs is insufficient. The limit concerns the characters the engine might match in the subject text, with Unicode folding included in that calculation.

This relationship is one reason the release's regex work belongs with its Unicode work. International text can affect the structure of matching itself. It is not only a matter of adding more entries to a table after an otherwise ASCII-only algorithm has been designed.

The keep operator solves a related problem

Perl already has \K, which the manual dates to 5.10.0. It excludes text matched before that point from the final matched result. That can express a variable-length prefix followed by the part that should be replaced, without using the new experimental lookbehind facility.

The manual illustrates the mechanism by replacing a capture-and-reinsert substitution with a keep expression.

s/(foo)bar/$1/g;
s/foo\Kbar//g;

Both examples retain foo while removing the following bar. In the first, the replacement explicitly puts the captured prefix back. In the second, \K keeps that prefix outside the part replaced. The manual describes the second form as more efficient for this case, but does not supply a timing figure.

The distinction from an assertion is still important. The keep expression first matches a prefix and then changes which portion belongs to the final result. A lookbehind asserts a condition at a position. They can provide similar results in a substitution like the documented example without becoming interchangeable descriptions of every regex problem.

For a transformation whose only requirement is to preserve the matched prefix, \K remains an established option. Variable-length lookbehind makes a different form of contextual condition available. The choice should follow the structure of the matching problem, including whether a positive or negative assertion is required.

The 5.30 manual also says that using \K inside another lookaround has behavior that is not well defined. That is a limit documented for this release, so a clear example should keep the operator outside nested assertions. The addition of variable-length lookbehind does not settle every combination of existing regex constructs.

Unicode selection becomes more expressive too

Perl 5.30 also introduces partial support for wildcard patterns in Unicode property values. The Unicode manual for this version describes an inner pattern that selects property values before the outer expression matches characters. A numeric-value property can select whole values from zero through five without listing each accepted value separately.

That facility is experimental in its own category, experimental::uniprop_wildcards. Its status is independent of the lookbehind warning. Accepting one experimental construct does not imply that the other has become stable. They also have different limits, because property-value matching supports only a subset of properties while lookbehind constrains maximum subject length.

The release updates Unicode support to 12.1. Its notes explain that the difference from Unicode 12.0 is the addition of the character for the new Japanese era name. The data update and regex syntax changes affect different aspects of an expression. The data determines available properties and characters, while the syntax determines how a program asks about them.

Byte operations have stricter boundaries

The same release makes previously deprecated sysread, syswrite, send and recv calls on :utf8 handles fatal. The release notes explain that those operations paid attention to the UTF-8 flag while ignoring other layers. A handle carrying an encoding layer such as UTF-16LE could therefore receive the wrong treatment. Reads also bypassed the layers' validation without performing equivalent validation themselves.

This is a separate change from lookbehind, but it matters to text-processing correctness. An expression's character-level rules depend on the program having a valid representation of the input. Permitting low-level operations to create invalidly encoded scalars undermines that assumption before matching starts.

Perl 5.30 also refuses pack operations that would return malformed UTF-8. The release describes this as both an incompatible change and a bug fix. Together, these changes make the boundary between byte handling and character operations more explicit. Within the regex engine, bounded lookbehind then operates on the documented possible character lengths, including the expansion allowed by Unicode case folding.