Perl adds patterns inside Unicode property values

Selecting characters by a Unicode property can require a long enumeration of acceptable values. Perl 5.29.9 adds an experimental way to express that selection with a pattern inside the property specification. The release notes demonstrate a numeric-value property whose value is itself matched by a regular expression. The Perl history records this development release on March 20.

There are two different subjects of matching here. The outer expression consumes characters from the application string. The inner expression selects values from a Unicode property. Confusing those contexts produces a plausible-looking pattern with the wrong meaning, particularly when a numeric property value can contain more than one character.

I like the expressiveness of Perl's regular expressions for small transformations. This addition is useful where it states a character-selection rule more directly than an enumeration. It also adds a level of interpretation that deserves explanation before the expression becomes an unexplained part of a script.

Matching a property value

The version's Unicode manual gives this form for selecting characters whose numeric value is a whole number between zero and five.

qr!\p{numeric_value=/\A[0-5]\z/}!

The slash-delimited expression matches the property's value. Its anchors require that the complete value be a single digit in the specified range. The resulting character property matches Unicode code points with those numeric values. It can therefore include characters such as subscripts, as the manual explains. It does not mean that the outer expression recognizes only the ASCII characters used to write the inner pattern.

The shorter property name is nv. An expanded expression can also use an inline modifier such as (?x) inside the property-value pattern. That modifier controls the inner expression's interpretation of whitespace. The new construct remains a delimited pattern after the property name and equals sign.

Without the inner anchors, a value such as 1/2 could match because it contains characters selected by the range. The manual uses that example to explain why the anchors matter. Matching the spelling of a numeric value is different from comparing that value arithmetically. This syntax performs the former operation, even though the property being selected happens to describe numbers.

An independent regex context

The manual says the inner expression is independent of the main pattern and does not share its capturing groups. That separation prevents a reader from assuming the inner match is simply another parenthesized section of the outer match. Its inputs are the available Unicode property values, not the substring currently being examined in the application text.

Perl wraps the inner expression in (?iaa: ... ). The documented reason is that the property values available in this release use ASCII and should match without regard to case. A syntax error can expose that wrapper in a diagnostic even though the programmer did not write it. Understanding the wrapper therefore helps explain both ordinary matching and an otherwise surprising error message.

Modifiers cannot follow the inner expression's final delimiter. They must appear in an inline modifier construct. Delimiters must be ASCII punctuation, with restrictions caused by the enclosing property braces. A literal closing brace cannot appear inside the value pattern because it terminates the enclosing property specification. Choosing a different delimiter does not remove that outer syntactic boundary.

The manual recommends patterns that match either the full property-value name or its lowercase form with spaces and underscores removed. For example, its block-property examples match Old I names in either of those forms. The syntax therefore provides a pattern over documented property-value spellings, not unrestricted access to an arbitrary display label.

Partial support is a real limitation

The release describes support for most properties, with remaining work planned. That is not a statement that every Unicode property accepts the syntax today. The manual lists unsupported cases, including Name and Name Alias, along with mapping properties such as Case Folding and Decomposition Mapping. It also excludes forms with an Is_ prefix and certain discouraged property forms.

Using the feature raises the default-on experimental::uniprop_wildcards warning. The documentation explicitly reserves the possibility of changes as experience accumulates. A pattern that selects no legal property values also generates a warning. These diagnostics distinguish a deliberately experimental facility from an ordinary misspelling that silently produces an empty selection.

For my own reading of an expression, a short explicit list can remain preferable when the allowed values are few and fixed. The wildcard form becomes more persuasive when the selection has a recognizable rule over the property names. The choice should follow the intended character set, because shortening the source is only useful if the resulting expression remains understandable.

Unicode data also changes

Perl 5.29.9 supports Unicode 12.0 according to its release notes. That is separate from the new selection syntax. Updating the data can change the set of code points associated with a property, while the wildcard feature changes how a program specifies which property values it wants. A property-based expression depends on both decisions.

The same release accepts named-character escapes within single-quoted qr patterns, removing an earlier restriction associated with deferred evaluation. These changes extend different parts of regex handling. Property wildcards are the larger interpretive addition because the expression now contains a pattern that operates over metadata before the outer pattern applies to text. Reading those two contexts separately makes the numeric-value example precise and explains why its inner anchors are necessary.