Perl stops turning undefined values into empty strings during UTF8 upgrades

Changing how a string is stored should not silently decide that missing data exists. Perl 5.37.6 changes utf8::upgrade so an undefined argument remains undefined. Previously the operation changed it into a defined string of zero length. The development release notes list this under incompatible changes, and the history shipped with the release records November 20 as its date.

The behavioral difference is small to demonstrate and substantial for code that distinguishes a missing value from a present empty value. An application may assign those states different meanings. A utility call concerned with representation previously erased the distinction as a side effect.

Upgrade changes internal representation

The utf8 manual included in this version describes utf8::upgrade as an in-place conversion of a string's internal representation. It converts an octet sequence in the native encoding into Perl's UTF8 representation while leaving the logical character sequence unchanged. If the string is already upgraded, the operation has nothing to change.

This definition concerns the representation of characters Perl already has. It does not describe decoding an arbitrary external byte encoding. The manual explicitly directs readers to Encode for arbitrary encodings. Using upgrade on bytes whose source encoding has not been interpreted does not establish what text those bytes were meant to represent.

The new undefined-value behavior fits that narrower responsibility. There is no character sequence to upgrade in an undefined value, so the function now leaves it alone. Code can still choose to replace missing data with an empty string, but that becomes an application decision instead of an incidental consequence of this call.

use strict;
use warnings;

my $missing;
my $empty = "";

utf8::upgrade($missing);
utf8::upgrade($empty);

print defined($missing) ? "present\n" : "missing\n";
print defined($empty) ? "present\n" : "missing\n";

Under 5.37.6 the first value remains missing and the second remains present. The example checks definedness directly. It does not use truthiness as a substitute, because that would fail to demonstrate the distinction between these two states.

I value Perl's willingness to make text manipulation concise, but an implicit conversion should not settle a data-model question that the program has not asked. Preserving undef makes this particular utility easier to reason about when it appears in shared string-handling code.

Source encoding and value representation are separate

The utf8 name can obscure several different operations. The manual says use utf8 tells the parser that source text in the current lexical scope contains UTF-8. That affects such things as non-ASCII characters in literal strings and identifiers. It does not turn every input handle into a decoder or specify the encoding of a file opened at runtime.

The utility functions are available without use utf8. The example above contains only ASCII source characters and calls a function already supplied by the interpreter. Adding the pragma would describe the source file, not change the meaning of the undefined-value repair.

The manual also distinguishes utf8::encode, which replaces a character sequence with its corresponding octets and clears the internal UTF8 flag. That changes the logical sequence presented by the scalar. By comparison, upgrade changes the representation while retaining its character sequence. Confusing the two can produce incorrect reasoning about lengths and about what an external consumer receives.

A text boundary should therefore state what arrives and what leaves. Source text, decoded characters, and an encoded byte sequence have different roles even if Perl represents each using a scalar. The November change does not merge those roles. It removes a change in definedness from one internal representation operation.

Definedness becomes observable after the call

Compatibility concerns arise when existing code implicitly relied on the old side effect. A later defined check can now take a different branch. A caller that previously assumed the upgrade guaranteed a present string will need to account for the documented undefined case. This is why a repair that preserves information still belongs in the incompatible-change list.

The appropriate policy depends on the program. A missing optional field may need to remain absent. Another interface may explicitly require missing text to become an empty string. Neither policy follows from asking Perl to upgrade a string's internal representation, so the source should express whichever policy the interface requires.

The manual gives a related warning about utf8::is_utf8. Its flag reports an internal representation property and should not become a marker for whether data is text or binary. That decision belongs to the variable's intended use. The same discipline helps explain the new behavior of upgrade: internal representation operations should not be treated as declarations of application meaning.

My preference is to preserve distinctions until the application explicitly chooses to collapse them. With this release, a representation upgrade retains the difference between an undefined value and an empty string. A subsequent policy for missing input can now examine the same distinction that existed before the call.