November 20, 2024·4 min read·engineeringperlparsingdocumentation

Perl 5.41.6 stops mistaking POD directive prefixes for cut

The November snapshot fixes premature termination of embedded documentation and avoids a temporary list when indexed feeds a two-variable foreach.

Embedded documentation requires the interpreter to know precisely when ordinary program text resumes. Perl 5.41.6 fixes cases where a directive beginning with cut could incorrectly end a POD section. A prefix match was enough to treat certain longer identifiers as =cut, even though they named something else.

The release history records the development snapshot on 20 November. Its release notes identify three affected forms. A digit after cut, as in =cut2studio, or an underscore, as in =cut_grass, could end the section. In string eval, the problem extended to any identifier beginning with cut, such as =cute.

I appreciate Perl's convention of keeping documentation close to the code it explains. That convention depends on the parser respecting the separation between the two. A mistake in the terminator is more consequential than a formatting defect because it changes which text the interpreter treats as a program.

POD has a defined exit back to code

The POD reference shipped with this snapshot describes embedding documentation where Perl expects a new statement. During parsing, the interpreter ignores the POD text and resumes source processing at the terminating command. The reference illustrates the boundary using =cut and blank lines.

use strict;
use warnings;

=pod

This paragraph documents the program.

=cut

print "program text resumes\n";

The blank lines make the section boundary visible to both the interpreter and documentation processors. The reference notes that older processors have additional expectations about blank lines. A source file is therefore read by more than one implementation, each of which needs to identify the documentation boundaries correctly.

The November correction concerns the interpreter's treatment of directive names. A longer name should not become the terminating directive merely because its first letters match. The release notes identify the exact suffix cases that exposed the bug instead of claiming a general rewrite of POD parsing.

That scope also matters when describing compatibility. The fix does not make every unknown directive meaningful to a documentation formatter. It prevents the Perl parser from treating the specified longer names as the end of embedded POD. Documentation validation and interpreter parsing remain separate questions.

String eval had a broader failure case

The string eval case affected any identifier starting with cut, according to the notes. That makes it broader than the digit and underscore cases listed for the other parsing path. Code that constructs Perl source for evaluation can therefore encounter different behaviour from a file containing a similar section.

The useful regression distinction follows directly from those cases. A test that only checks =cut2studio in an ordinary file would not cover the reported =cute behaviour under string evaluation. Both paths need to agree about the actual terminator while preserving the different circumstances that previously triggered the error.

This is a parser-boundary problem. It does not require an assertion that the ignored documentation ought to compile as Perl. Quite the opposite, the point of embedding POD is that the interpreter does not interpret its paragraphs as code. Premature termination removes that protection before the author requested it.

Keeping the distinction explicit also explains why the failure can appear remote from the directive itself. Once the parser believes documentation has ended, subsequent prose becomes input to the source parser. The eventual error can concern text that was never intended as a statement, while the earlier prefix recognition caused the change of interpretation.

Indexed iteration no longer creates an intermediate list

The same snapshot includes a performance change with a similarly concrete scope. Passing builtin::indexed over an array into a two-variable foreach no longer creates a temporary list twice the array's size. The notes say the loop now walks the array directly in place, as an ordinary array foreach does.

The original expression first generated index/value pairs, then immediately unpacked those pairs into loop variables. The new implementation recognises that combination. It can supply the index and value while traversing the original array, avoiding materialisation of the intermediate sequence.

This does not imply that every call to indexed avoids a result list. The documented optimisation names a particular consumer and input shape. Code that needs to retain the generated pairs has a different requirement from a loop that immediately consumes them. Describing the optimisation narrowly preserves the reason it is possible.

For large arrays, the avoided allocation is easy to understand without attaching an invented benchmark. A temporary sequence containing an index and a value for every entry adds storage proportional to the input. The release removes that sequence for the specified loop form. It does not promise a particular wall-clock improvement for an unspecified program.

I am more interested in that mechanism than in a generic claim that the snapshot is faster. It identifies an allocation that the program no longer needs, while preserving a readable expression for paired iteration. A maintainer can recognise the affected code directly.

Build failures have their own platform scope

The release also fixes a crash early in builds using -msse and quadmath on 32-bit x86. The notes attribute it to a misaligned access. That is a specific build configuration, distinct from the source-parser correction and the iteration optimisation.

The snapshot therefore contains changes at several implementation levels without requiring a new headline language feature. For embedded documentation, its practical correction is exact. Names that merely start with cut no longer acquire the terminating meaning in the reported cases, allowing the interpreter to keep ignoring documentation until it reaches the intended boundary.