July 20, 2018·4 min read·engineeringperlregexunicode

Perl 5.29.1 refines the rules for regex braces

Perl 5.29.1 separates illegal brace syntax from literal uses that remain supported, and restricts delimiters to leave room for Unicode graphemes.

Reserving syntax for a language extension can break a program that only intended to match punctuation. Perl 5.29.1 addresses that tension in its treatment of unescaped left braces in regular expressions. The release notes say some formerly deprecated uses are now illegal, while most instances that previously produced a deprecation warning remain legal and receive a non-deprecation warning. That second change is part of the compatibility policy, not an incidental wording update.

The release history at the tag records July 20. This development release also tightens delimiter rules, with the documentation explaining the relationship between delimiters and Unicode graphemes. Both changes concern punctuation that can look unimportant in a compact pattern but affects how Perl can evolve its parser.

I still find Perl convenient for short text work. Explicitly escaping a literal brace is a reasonable cost when it makes a pattern unambiguous. The more difficult question is how much old code should stop compiling merely because its author left that literal intent implicit.

A warning need not promise removal

The deprecation documentation in this release gives the straightforward rule for a literal left brace. Escape it with a backslash or put it in a character class such as [{]. If braces also delimit the pattern, escape the corresponding literal right brace to avoid confusing the parser. The documented spelling makes the role of the punctuation explicit.

use strict;
use warnings;

my $pattern = qr{abc\{def\}ghi};

The reason for reserving unescaped braces is to permit future language extensions. However, the documentation says the restriction is not enforced in contexts where extensions are unlikely to conflict with the literal meaning. Some ambiguous-looking uses can therefore remain supported while still producing a warning that explains their interpretation.

That is a meaningful change from a deprecation warning. A deprecation warns of an intended withdrawal of support. A non-deprecation warning can instead identify a potentially confusing construction without making that same promise. This release changes the diagnostic for the retained cases to say that Perl is passing the unescaped brace through in the pattern.

The distinction avoids a blanket rule that every historical warning must become a fatal error. The release still makes some uses illegal, preserving room for syntax changes where the ambiguity matters. It leaves other cases available where rejecting them would cause breakage without a corresponding need for new syntax. The result requires context-sensitive rules, but those rules reflect different costs of preserving the old meaning.

The history of this work is already recorded in the tagged documentation. Literal brace uses were deprecated in Perl 5.20, with some cases becoming fatal in Perl 5.26. Other cases had not received warnings at the same time. That staggered coverage helps explain why one statement about all unescaped braces would misdescribe the transition.

Delimiters and visible characters differ

The delimiter change concerns a different layer of the parser. A grapheme can consist of several Unicode characters that together appear as one character to a reader. The documentation illustrates a base character combined with an accent. Rejecting non-standalone graphemes as delimiters reserves room for accepting complete multicharacter graphemes in delimiter syntax.

This is not a general prohibition on Unicode in patterns. It concerns the character used to delimit an operation. The contents of a pattern and the punctuation that identifies its boundary have different parsing roles. A rule about the boundary must remain intelligible even when the visible character is built from more than one code point.

The documentation also prohibits unassigned code points that might later receive assignments. Its reason is compatibility with a future Unicode assignment that could make a formerly accepted delimiter unsuitable as a standalone grapheme. A program should not gain a parsing failure merely because the interpretation of an unassigned delimiter changes under a newer character database.

The document distinguishes those potentially assignable positions from code points Unicode will never assign, including noncharacters and values above the Unicode maximum. It allows the latter as delimiters under its stated rule. That detail shows the actual criterion is future assignment and grapheme behaviour, rather than a simple division between ordinary ASCII and everything else.

I prefer familiar punctuation for delimiters in short programs, because it reduces the amount of parser knowledge a reader needs. The Unicode rule nevertheless deserves a precise explanation. It protects the ability to extend a character-oriented syntax without treating every displayed character as necessarily one code point.

Transliteration punctuation has its own semantics

The release also clarifies the documentation of tr when an apostrophe is its delimiter. In that form, hyphens are not special and \x{} does not interpolate. This is a documentation correction, distinct from the newly illegal brace cases. A punctuation choice can already affect how an operator interprets its contents, even without any release changing the implementation.

These changes concern related questions at different levels. Braces within a pattern can be literal text or reserved syntax. Delimiters tell the parser where the operation begins and ends. Transliteration delimiters can affect interpolation and range interpretation. Correct compatibility explanations need to identify which level a particular rule changes, so a maintainer can preserve the intended operation while adjusting its spelling.