Perl 5.33.8 fixes a leak in regex compilation

The regex that exposed this leak did not need to match anything. During the investigation of Perl issue 18604, Hugo van der Sanden reduced the failure to compiling different pattern strings through qr{} in the same subroutine. That reduction matters because the original example included a substitution and a case-insensitive match. Neither operation was necessary to retain the unwanted objects.

The Perl 5.33.8 release notes identify the repair as a memory leak in regcomp.c. They list it under both performance improvements and selected fixes, referring to the same issue. Those two entries describe one repair. They do not establish a general regex speed improvement or a second independent memory problem.

I reach for Perl because matching and substitution fit directly into its grammar. That convenience also makes compiler correctness relevant to very small programs. An application can invoke substantial interpreter machinery through an expression that occupies only a few characters of source.

The report became smaller during diagnosis

The reporter initially described steadily increasing memory use with Perl 5.32.1 on Fedora 33. Follow-up comments reported that the code behaved as expected with earlier Perl versions on earlier Fedora releases. Those observations suggested a regression, but the initial comparisons changed both interpreter and operating-system versions. The subsequent interpreter-level investigation made the case more specific.

Van der Sanden used Devel::Leak to compare objects remaining after different iteration counts. His first diagnostic example performed a substitution on a pattern string and then used it in a match. He identified extra INVLIST objects with reference counts still set to one. A later reduction replaced the match with qr{} and discarded its result. Finally he removed the substitution too.

At that point the important ingredients were two different pattern strings that needed an inversion list, compiled through the same code location. The result was a smaller explanation of the failure than the original complete program provided. The compiler could leak even when application code did not retain the compiled regex. That makes the investigation relevant to programs whose memory growth might otherwise be blamed on a cache or a collection of results.

The issue contains a finite reproducer of this shape. This shorter illustration retains its two pattern inputs and deliberately throws away the compiled value. It is an illustration of the reported path, not a benchmark or a claim about a measured leak rate.

use strict;
use warnings;

sub compile_pattern {
    my ($pattern) = @_;
    qr{$pattern};
    return;
}

for (1 .. 10) {
    compile_pattern("a\\sb");
    compile_pattern("a\\sc");
}

Each input string contains a regex whitespace escape. Changing the final literal gives the compiler different patterns. No match against a large subject is needed in this example, and no array accumulates the returned regex values. The finite loop also avoids presenting an indefinitely running process as a routine diagnostic command.

The remaining objects identified the failing layer

The issue discussion calls the retained objects inversion lists. Their appearance in Devel::Leak output gave maintainers a concrete internal object to investigate. A process memory graph had shown the symptom, but an object-level comparison narrowed the explanation to regex compilation. These observations answer different questions. Resident memory says what the process occupies, while the remaining internal objects provide evidence about which allocation lifetime is wrong.

The debugging setup mattered as well. James E Keenan initially obtained addresses without the detailed object descriptions shown in van der Sanden’s output. The discussion established that a debugging-enabled Perl build was needed for the full diagnostic information. Keenan then confirmed that building with -DDEBUGGING produced the expected detail. That exchange explains why copying a reproducer alone might not reproduce the diagnostic output.

This was also independent of whether the original example declared all its variables for strict mode. Van der Sanden supplied the declarations needed to run it under strict and reported the same result. The later reduced example used lexical variables. Fixing undeclared names would therefore improve the example without resolving the interpreter allocation problem.

Error paths complicate allocation cleanup

Karl Williamson’s February comment identified a likely allocation involved in the leak and mentioned other paths that could leak when parsing encountered syntax errors or fatal warnings. He proposed making the objects mortal rather than manually decrementing their reference counts along every possible path. That comment describes a proposed approach during diagnosis. The release notes establish that a fix ships, but do not by themselves establish which broader cleanup strategy was adopted.

The distinction matters when explaining a small maintenance change. A successful compilation path and a parser error path can leave a function in different ways. The repair must account for the lifetime of temporary objects across those exits. A discussion about simplifying that ownership is more informative than assuming every memory leak is an omitted release at the end of a function.

I find the reduced reproducer particularly useful because it preserves the interpreter problem while removing the application story. It makes no claim about a production workload. It shows that a caller can discard a regex result correctly and still encounter retained compiler allocations.

The practical scope of 5.33.8 is that this identified compilation leak is repaired in the development release. The release does not quantify an application-wide memory reduction. Programs repeatedly compiling affected dynamic patterns are the relevant audience, and the issue supplies a concrete diagnostic history for understanding why the repair applies to them.