The missing UTF-8 character prefix in C

C can spell a UTF-8 string with a prefix that it cannot use for a character constant. Aaron Ballman's N2198 proposal addresses that inconsistency by adding u8 to the character-constant grammar. The WG14 document log lists the paper on January 2. The document itself is dated December 31, so this is a newly listed proposal with wording prepared at the end of last year.

The useful story is the distinction between a character's spelling and the number a program needs. Ballman gives a concrete motivation. A programmer targeting a narrow execution character set other than ASCII may still need ASCII values. A UTF-8 prefix can express that requirement because single-code-unit UTF-8 encodings agree with ASCII. Making the two kinds of literal look consistent is helpful, but the encoding guarantee is the more consequential part of the proposal.

C was my first language, and I value the precision that a small language can offer at an interface boundary. Character handling also shows how much meaning can sit behind a small piece of syntax. A prefix has to identify an encoding and a type with enough precision that code using the resulting value can rely on both.

A character constant does not name every possible encoding

N2198 starts with four string prefixes, u8, u, U and L, against three character prefixes, u, U and L. Its grammar change gives character constants an optional encoding prefix and puts all four spellings in the shared production. That is a compact grammatical change. The semantic work appears in the paragraphs that assign types and define the conversion from the source character to its execution representation.

The proposed spelling can be illustrated without claiming that an existing C implementation must accept it:

/* Proposed C syntax in N2198. */
u8'A'

The example contains a character that fits in the single-code-unit portion of UTF-8. It therefore shows the motivation without introducing the separate question of a character whose UTF-8 encoding occupies several code units. Calling the constant UTF-8 must not imply that a scalar value can contain an arbitrary UTF-8 sequence. The size and interpretation of the result still need their own rules.

That distinction matters for comparisons. If an interface specifies bytes in a particular encoding, a comparison constant needs to denote those bytes under the implementation's execution environment. Merely recognizing the source glyph does not establish the required numeric value. The paper's non-ASCII execution-set example makes the need visible without requiring a change to the representation of strings throughout a program.

The C++ precedent has a narrower contract

Ballman cites Richard Smith's N4267, the earlier C++ proposal for the same prefix. Smith's wording gives a UTF-8 character literal type char. Its value equals the ISO 10646 code point when that value fits in one UTF-8 code unit. A value that needs more than one code unit makes the program ill-formed. The wording also rejects a literal containing multiple characters.

Those restrictions explain the intended unit of representation. The literal denotes one code unit. It does not construct a sequence and then pack that sequence into an integer. For a reader accustomed to treating a visible character as the unit of text, this is an important limitation. The prefix identifies an encoding while the literal category limits how much of that encoding one expression can represent.

N4267 also distinguishes ordinary character literals from UTF-8 literals. Its ordinary single-character form takes its value from the execution character set. The UTF-8 form instead has an explicit code-point rule within the one-code-unit range. This gives the extra spelling a purpose even when both expressions contain the same letter in the source file.

The precedent helps explain the design, but C cannot inherit its wording without checking the surrounding definitions. Ballman explicitly describes a divergence between the languages. C uses environment macros and library conversion functions in its existing character rules. C++ can describe the correspondence by reference to ISO 10646. Similar tokens therefore do not establish identical semantics across the two languages.

The proposal still has unresolved wording

N2198 asks whether it should introduce an mbrtoc8 conversion function. Ballman says the existing multibyte facilities do not appear sufficient for the UTF-8 case. He also asks whether to follow the formulation used for UTF-8 string literals and whether an environment macro is needed. These are questions in the submitted paper, not facilities that the proposal has already specified.

There is also a visible type inconsistency. The added table row associates u8 with unsigned char, while the proposed prose says that a constant with that prefix has type char. Those are different declarations of the result type. An explanation of this paper cannot select one silently and present it as the settled C rule. The committee needs consistent wording before the prefix can provide a dependable type contract.

The conversion paragraph presents another reason to read the document as unfinished work. It continues to refer to the existing multibyte conversion functions while the rationale asks how UTF-8 conversion should be expressed. Adding a token to the grammar is therefore only one part of the change. The proposal must also specify which mapping determines the value and how that mapping relates to the chosen type.

I prefer this uncertainty to remain explicit in discussion of the feature. A programmer should be able to distinguish a useful proposal from a portable facility. That distinction is especially important in C, where an assumption about a small integer type can become part of a much larger interface contract.

What the additional spelling would communicate

The clearest benefit is local expression of an encoding requirement. The programmer can state that requirement beside the character itself. Readers then have a place to check the intended representation without inferring it from a build's default character set. Ballman's alignment argument also has merit because the same prefix would be available for both string literals and character constants.

The proposal's current limitations constrain what can be concluded today. N2198 supplies a rationale and a grammar change, but its open conversion questions and conflicting type wording remain part of the document. Smith's C++ paper supplies a precise precedent for a single-code-unit literal. The next step for the C wording is to establish an equally precise relationship among the prefix, the resulting type and the encoded value.