Perl recognizes Turkic casing in UTF-8 locales

UTF-8 does not settle how a program should change the case of a letter. Turkic languages distinguish the dotted and dotless forms of I in a way that the ordinary pairing of ASCII i and I does not express. Perl 5.29.8 begins using the alternate Unicode rules when it detects a Turkic UTF-8 locale. The release history dates the development release to February 20, and its versioned release notes describe the change explicitly.

This correction separates two questions that text-processing code can accidentally combine. An encoding determines how a character is represented. A locale can determine which case relationship an operation should use. Successfully decoding UTF-8 does not establish that an application's linguistic assumptions are correct.

Perl remains my choice for small text transformations, and this is one reason to keep their assumptions visible. A one-liner can express an operation concisely while still depending on a particular interpretation of text. Concision does not remove the distinction between a locale-sensitive transformation and one that should have the same result for every user.

The two case pairs

The release notes specify both mappings. The uppercase form of lowercase i is U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE. For capital I, the lowercase form is U+0131, LATIN SMALL LETTER DOTLESS I. Unicode supplies alternate casing rules for Turkic languages, and Perl previously ignored them in this situation.

The resulting change is more precise than adding support for UTF-8 in general. The locale manual included in this release says Perl has supported UTF-8 locales since 5.20, with qualifications around sorting and string comparison. This development work addresses the difference between Turkic UTF-8 locales and other UTF-8 locales. It does not replace the entire locale system.

The distinction also explains why treating ASCII as inherently independent of language can be wrong. Both input letters in these mappings belong to ASCII, but their appropriate partners under Turkic rules do not. An implementation that assumes every ASCII case conversion stays within ASCII would therefore lose information about the selected locale.

A useful way to describe the contract is to name the pairings directly. Under the relevant Turkic rules, the dot on lowercase i remains represented in its uppercase partner, and the uppercase undotted I has an undotted lowercase partner. The release implements those documented mappings when the appropriate locale applies. It does not infer a language from the contents of a string.

Selecting a locale remains explicit

The locale manual makes a distinction between the underlying process locale and the operations for which Perl uses it. Outside the POSIX module, Perl normally ignores the current locale unless the program requests locale behavior. The use locale pragma requests that behavior within its scope. The locale itself comes from the startup environment or an explicit call to setlocale.

That means installing this interpreter does not make all Perl text processing Turkic-sensitive. A relevant locale must be available and selected, and the operation must be subject to locale semantics. The operating system needs locale support, and the corresponding locale definitions must be installed. The manual emphasizes that names and installation methods vary across systems.

This separation matters for deployment. Selecting a locale that exists on one machine does not establish that the same locale is present elsewhere. Conversely, the presence of a UTF-8 locale does not mean that every locale-sensitive operation should follow the same casing table. The new detection addresses the latter distinction once the program is operating under a supported Turkic UTF-8 locale.

The manual also distinguishes locale categories. LC_CTYPE concerns character types, while LC_COLLATE concerns ordering and LC_NUMERIC concerns numeric presentation. A request to respect linguistic casing should be understood in that category structure. It should not become an unexplained assumption that all other aspects of number formatting or ordering now have the application's desired semantics.

My preference is to make those choices part of the program's stated input contract. Human text can require linguistic case behavior. A machine identifier can require a fixed interpretation. Applying one rule to both because the strings happen to use UTF-8 makes the representation decide something it cannot decide.

The same release changes another default

Perl 5.29.8 also updates JSON::PP to 4.00 and enables allow_nonref by default, following the policy described for JSON::XS 4.0. The release notes classify that as an incompatible change. It is unrelated to Turkic casing, but it provides another concrete example of a runtime update changing which inputs an operation accepts without changing the call written by its user.

For locale-sensitive text, the corresponding change is which character mapping applies under an already selected environment. For JSON::PP, it is whether non-reference values are allowed by default. Both deserve an explanation at the contract level, because the source expression alone may look unchanged after the update.

The release also adopts a faster inline function for converting a single byte to UTF-8. That is a separate implementation improvement. Its notes provide no timing result, and it should not be confused with the semantic work on casing. The byte conversion path concerns representation, while Turkic locale support concerns the rules applied to characters. Perl 5.29.8 changes both, but the dotted and dotless I mappings are what determine the observable linguistic result.