Perl removes recursion from pack template parsing

A binary encoder has to interpret its format description before it can convert the supplied values. That description is itself a language, with directives and grouping rules. Perl 5.37.1 repairs a stack-overflow problem in parsing pack templates by avoiding recursion. The release notes identify this parser repair among the development release's selected fixes. The history included in the release records June 20 as its date.

The distinction between a format and its data is the useful story here. A small list of values does not by itself establish that the interpreter's work will be small. A template can ask for structured processing independently of the number of arguments. Removing recursive parsing addresses one way that processing could exhaust the interpreter's native stack.

A template describes conversion rules

The pack documentation in this release defines the operation as converting a list of values into a string according to a template. Converted values concatenate into the resulting string. The directives determine both representation and interpretation, including distinctions between native formats and explicit byte orders.

For example, C represents an unsigned octet and n represents an unsigned 16-bit value in network byte order. They let a program express a small binary record directly.

use strict;
use warnings;

my $record = pack "Cn", 7, 513;
print unpack("H*", $record), "\n";

The intended record contains the octet 07 followed by the two-byte big-endian representation 0201. This example uses a fixed template. It illustrates the separation between the description and the supplied values without reproducing a crashing input or depending on a machine's native integer byte order.

The manual documents separate directives for native integers and floating-point formats. Those are useful when an application specifically needs the local representation. A portable file or protocol must make that choice deliberately. Using an explicit network-order directive is part of describing the external record, not an optimization supplied automatically by pack.

I value Perl for transformations where the representation is explicit in a small piece of code. Binary templates provide that same directness for structured bytes. The interpreter still has to process that compact description safely, even when a template is much more complicated than this example.

Parsing consumes resources before conversion finishes

Templates support more than a sequence of individual letters. The manual includes repeat counts after directives, bracketed counts, and parenthesized groups. Byte-order modifiers can apply to a group and its subgroups. These rules require the template parser to track structure in addition to reading each value.

Recursion makes a parser's native call stack part of the resource cost of describing nested structure. The release note's stated repair avoids recursion and stack overflow during this parsing. That establishes the relevant improvement without requiring a claim that all possible template-driven resource use has disappeared.

An application can still request a large output. Repetition can consume many values or specify the length of a string field, depending on the directive. Therefore a parser fix and an application limit on output size solve different problems. The first concerns how the interpreter processes the format, while the second concerns how much work the application has authorized.

The distinction also helps interpret this release accurately. The notes do not describe a new binary format or a changed meaning for ordinary directives. They describe a repair to interpreter behavior while handling a template. Existing explicit templates remain the means of expressing layout. The repaired implementation changes how that description can be processed without recursive stack exhaustion.

Another fix concerns code embedded in a pattern

The same release repairs an interpreter panic when eval appears as the last statement inside a regular-expression code block. The notes show the affected shape as a pattern containing (?{ ...; eval { ... }; }). This concerns Perl code executed within a pattern, not every expression that happens to contain a regular expression.

That second repair is relevant to Perl's unusually close integration of pattern matching with the language. An embedded code block can combine interpreter operations that ordinary matching does not require. A panic in that combination is a different failure from a pattern simply failing to match. The release notes identify the concrete arrangement of constructs, which makes the scope of the repair clearer than a general claim about regular-expression reliability.

My preference for Perl in one-off text work depends on keeping those distinctions visible. A straightforward match and a pattern that executes code have different reasoning requirements. The June release addresses a failure in the latter combination while also repairing a parser for binary descriptions.

There is no need to treat either fix as a new feature available through a pragma. They are interpreter repairs in a development release. For the pack change in particular, the important mechanism is the removal of recursive parsing as a route to stack overflow, while the documented template language continues to specify which bytes the program asks Perl to construct.