Perl fixes buffer sizing when transliteration changes encoding

Replacing one character with one character does not establish that a string will occupy the same number of bytes. Perl's transliteration implementation has to account for that distinction, including when the replacement changes the string's internal encoding. Today's Perl 5.40.2 fixes a heap buffer overflow in that operation, tracked as CVE-2024-56406.

The 5.40.2 release notes identify non-ASCII bytes in the search side of tr/// as the relevant condition. They locate the failure in S_do_trans_invmap(), where the destination pointer can overflow the allocated buffer. Nathan Mills discovered the vulnerability. The notes describe possible denial of service or arbitrary code execution on platforms without sufficient defenses.

The Perl history records date the release to April 13. Perl 5.38.4 supplies the corresponding maintenance correction on the older branch, as its release notes confirm. Both releases describe the same defect without intentional incompatibilities with their immediately preceding maintenance versions.

Perl's integrated text operations remain a reason I use it for one-off processing. They make concise programs possible, while placing responsibility for storage conversion inside the interpreter. This defect matters precisely because ordinary character replacement should not require an application author to reason about destination pointers in the runtime.

Transliteration maps characters through a compiled representation

The versioned operator documentation describes tr/// through search and replacement character lists. The security issue concerns the implementation of that mapping. Its behavior needs to remain correct even when the output character needs a different representation from the character it replaces.

The released runtime source in doop.c contains several transliteration helpers. S_do_trans_invmap() handles an inversion-map representation, including cases where the lists differ or relevant flags are present. That helper obtains the compiled search data and replacement mapping, then walks the input and writes translated output.

This representation allows an operation described by character ranges to become lookup data for runtime processing. Runtime code does not need to interpret the source spelling again for every character. It does, however, need information about how large the translated output can become. Precomputing a map does not remove that allocation requirement.

In the fixed source, the helper reads a maximum expansion value from the compiled representation. It also tracks whether the output is UTF-8 and whether an in-place edit is permissible. Those are related decisions. Editing in place is only safe when the existing storage can accommodate the operation under the assumptions the helper uses.

Input bytes and output bytes have different constraints

The release notes demonstrate the vulnerability with repeated character value \x{FF} translated to \x{100}. The example is useful without reproducing its large allocation or crash. It shows a replacement crossing the boundary between values representable in the original byte string and a value that forces UTF-8 output.

The fixed runtime checks for exactly this possibility. Its comment says that when a transliteration can force a non-UTF-8 input into UTF-8, the code cannot assume an in-place edit is safe. It disables that assumption before choosing the output buffer. The property belongs to the mapping and the input representation together.

This is why counting input characters is insufficient for allocation. The number of logical replacements can remain unchanged while the bytes written grow. A loop that advances one input character at a time can still advance its destination by more bytes than its source. The allocation has to cover that destination growth across the whole operation.

There is also a difference between malformed UTF-8 and a legitimate representation change. The advisory's example does not require a malformed encoded sequence to explain the failure. The implementation has to handle a valid character mapping whose output requires UTF-8 storage. Rejecting malformed input elsewhere would not establish that this mapping has enough destination space.

That distinction makes the bug more specific than a general warning about non-ASCII text. The relevant question is how the runtime calculates output space when a particular transliteration can change encoding. A program does not become safe merely because it has already validated some external text before invoking an affected interpreter operation.

Compilation supplies the expansion bound

The compiler source in op.c explains the maximum expansion calculation. It describes the ratio between the encoded sizes of replacement and source characters. A comment gives the example of a two-byte UTF-8 character replaced by a three-byte character, producing an expansion factor of one and a half.

The fixed calculation also accounts for the possibility that the original string is not UTF-8. When a mapping can force UTF-8 output and the source character is below 256, the calculation uses a source size of one byte for that case. That is the important relationship between the compiler's estimate and the runtime's actual input representation.

An estimate based only on the source character's UTF-8 size could miss that relationship. A character represented by one byte in the input may require more bytes once the entire output uses UTF-8. The allocation bound must cover the representations the runtime can actually receive, including the compact byte representation.

In the released runtime, non-in-place transliteration allocates from input length multiplied by the maximum expansion, with additional space for rounding and termination. The comment explains that compilation computed a worst-case bound because the runtime does not know how much this particular input will grow. The compiler's estimate therefore has a direct memory-safety consequence in the execution loop.

This division is a reasonable implementation strategy. An operation's mapping can provide an upper bound once, then runtime execution can allocate against that bound. Its correctness depends on agreement about units. Source character widths, input byte length, and output byte widths must describe compatible cases when combined in the calculation.

A representation change can restart the operation

The fixed runtime also exposes how encoding changes affect execution. When a replacement requires a value of at least 256 while the output is still in byte form, it changes the output mode to UTF-8 and restarts the transliteration. The source and destination pointers return to their initial positions, and the match count resets.

Restarting allows earlier characters to be emitted under the new output representation. A replacement late in the input can therefore affect the storage requirements for preceding output as well. The operation cannot treat the wider replacement as an isolated local write while leaving all earlier bytes interpreted under a different representation.

This is a useful detail when understanding the allocation bound. It has to support the output representation chosen for the completed string. It is not enough to reserve a small amount of extra space at the point where the wider character first appears. The loop may repeat earlier work with a different encoding decision.

The source also distinguishes unmapped characters from characters with explicit replacements. Even preserving a character can require encoding work when output has become UTF-8. The memory requirement follows the final representation of the string, including characters whose logical value does not change during transliteration.

The maintenance release keeps the correction localized

The release notes do not ask authors to replace transliteration with a different API. They identify a runtime vulnerability and provide a corrected maintenance interpreter. That is an appropriate scope for existing applications using the operation according to its documented meaning. The behavior being repaired concerns buffer ownership and sizing beneath that meaning. An application-level rewrite would also have to preserve the original mapping semantics, while the interpreter correction repairs the implementation shared by existing callers of the operation.

I regard this as a direct interpreter upgrade issue wherever an affected installation can reach the vulnerable operation. The advisory's possible effects are serious, while the fixes exist on both maintenance branches named in today's releases. The stated possibility of code execution should retain the advisory's qualification about platform defenses, rather than becoming a claim that every crash demonstrates successful exploitation.

Perl 5.40.2's allocation logic makes the source of the risk concrete. Transliteration can preserve a character-level mapping while changing byte storage for the resulting string. The compiled expansion bound and runtime representation decision must agree before the helper begins writing that result into its destination buffer.