Perl 5.33.9 separates environment keys from string storage

A string can have the same characters while occupying different bytes inside Perl. Passing its internal buffer directly to an operating-system API can therefore make externally visible behaviour depend on an implementation detail. The Perl 5.33.9 release notes describe exactly that problem with environment-variable names. Assigning through %ENV previously sent the key scalar's internal string buffer directly to the operating system.

The new behaviour first attempts to downgrade the string. If that cannot represent its characters, Perl warns and uses its UTF-8 form. Environment values have followed this approach since 5.18, according to the release notes. The change brings keys into the same conversion policy. It does not declare that every program receiving an environment understands Unicode names.

I value Perl's ability to express a text transformation in a small program. That preference makes implicit conversions worth understanding. A short script can cross an operating-system interface without any visible foreign-function code, and the interpreter still has to decide which bytes to send.

Character values do not specify the buffer

This release also expands the internal programming manual's discussion of passing strings to C. It describes a Perl string conceptually as a sequence of code points. A C library may instead expect an array of bytes with a terminating zero. The extension author must define the mapping between those representations.

The manual illustrates the ambiguity with a three-character string whose final character has value 0x8c. Perl can store those characters as three bytes, or store the same character sequence in UTF-8 using four bytes. An API that takes the raw internal buffer sees different input for the same Perl-level value. The example makes the environment-key bug easier to understand without assuming that the problem requires a character beyond the single-byte range.

A key containing only characters representable as bytes can still have an upgraded internal representation. Downgrading makes its outgoing representation independent of that upgrade. Conversely, a character above the byte range cannot be preserved through the same operation. That is the case for which the release specifies a warning followed by UTF-8 output.

These are distinct cases. Treating every upgraded string as necessarily containing wide characters would conflate its internal representation with its contents. The release note names upgraded keys because the old implementation exposed the representation. The new diagnostic concerns wide characters because those are what prevent the downgrade.

A warning describes the fallback

The new diagnostic is Wide character in setenv key (encoding to utf8). Its wording identifies the conversion point and the representation chosen after downgrading fails. A caller should not read a successful assignment as evidence that another process interprets those bytes with the same character model. Perl has selected a byte representation, but that cannot determine the receiving program's conventions.

The following illustrates two names with different conversion requirements. It is a demonstration of the documented assignment paths, not a recommendation to give portable environment variables non-ASCII names.

use strict;
use warnings;

my $byte_name = "TASK_" . chr(0xe9);
utf8::upgrade($byte_name);
$ENV{$byte_name} = "first";

my $wide_name = "TASK_" . chr(0x100);
$ENV{$wide_name} = "second";

Upgrading the first scalar changes its representation without making its final character exceed the byte range. Its characters permit the downgrade described by the release. The second key includes a character whose code point exceeds that range and therefore needs the warned UTF-8 fallback. The program changes its own process environment and deliberately leaves the warning enabled.

The important observation is the difference in representability. Merely checking whether a scalar uses UTF-8 internally cannot answer whether it can be converted to a byte string. The release's conversion order makes that distinction explicit. It also explains why an environment-key fix belongs with documentation about scalar extraction in extension code.

Extension code needs an encoding decision

The updated internal manual recommends SvPVbyte when a C interface expects bytes. It recommends SvPVutf8 for an interface expecting UTF-8 text. The choice follows the receiving API's contract. Extracting through SvPV can be valid when code also consults SvUTF8 and handles both representations, but the buffer alone is ambiguous.

The same section warns about embedded zero bytes. Where an API accepts a length, supplying that length can preserve the distinction between string data and a terminator. Where it does not, rejecting embedded zeros may be appropriate. Encoding and termination are separate obligations at the boundary, even though both become visible when a Perl scalar becomes a C string.

This documentation change is useful beyond the particular %ENV repair. It gives extension maintainers a model for reviewing other calls that expose scalar buffers. The question is what byte sequence the receiving function expects. A buffer extraction macro cannot answer that policy question by itself, and an internal UTF-8 flag is not a replacement for an API contract.

I prefer this correction because it removes a hidden dependency on string storage while preserving the distinction between bytes and characters. It also gives a specific warning when the established byte conversion cannot represent the name. In 5.33.9, the concrete result is that %ENV key assignment follows the conversion policy already documented for its values.