Go protobuf messages gain a behavioral reflection contract

A Go interface can be easy to satisfy while saying too little about the behavior its consumers require. The original protobuf message interface has this problem. A type can acquire the expected methods by embedding a generated message, yet fail when protobuf code treats the enclosing value as an ordinary generated message. The result can be a crash or unpredictable behavior in code that appeared to satisfy the interface.

The new Go protobuf API announced today changes that contract. A message exposes reflection over the protobuf type system, so generic operations can work through specified behavior. The new module is google.golang.org/protobuf. Its reflection design also supports messages whose schemas become known only at runtime.

I like interfaces that describe what their consumers actually need. This revision addresses an important gap between structural interface satisfaction in Go and the richer assumptions a serialization implementation makes about its inputs. The useful abstraction is the message behavior, including its schema, rather than a particular generated struct layout.

Go reflection does not describe the schema

Go reflection answers questions about Go types and values. A protobuf schema carries information that a Go struct alone does not express. Field options are one example. A schema author can annotate a field with application-specific policy, and a generic tool may need that policy without knowing the generated Go type in advance.

The announcement uses sensitive-data redaction to explain the difference. Its schema defines an extension of google.protobuf.FieldOptions that marks a field as non-sensitive. A redaction function should retain those annotated fields and clear others. Looking at whether a Go field is a string or an integer cannot establish that policy. The operation needs the protobuf field descriptor and the options attached to it.

Previously, requests for reflection and requests for custom message representations looked like separate extensions to the library. The redesign treats them as related requirements. If the interface fully describes a message, code that consumes the interface can examine and manipulate a generated message or another correct implementation through the same contract.

This is a stronger promise than recognizing a value as belonging to a generated family of types. It puts the behavior required by serialization and other tools into an explicit interface. Implementations still have obligations, but those obligations no longer depend solely on conventions about generated struct internals.

One entry method leads to the reflective message

The new message interface has one method, ProtoReflect, returning a protoreflect.Message. The single entry method avoids adding a large number of methods directly to generated message types. Operations on fields and descriptors live on the reflective value it returns.

The v1.20.0 reflection interfaces distinguish schema information from the particular Go representation. Descriptor returns the protobuf message descriptor. Type includes both Go and protobuf type information. Code that needs only the schema can use the descriptor without coupling itself to the representation.

Field access uses FieldDescriptor values. For an ordinary field, the descriptor must match the field known to that message. An extension descriptor has its own conditions, including the extended message name and extension range. This prevents a caller from treating an arbitrary field number or superficially similar descriptor as sufficient authorization to access a field.

The following small function illustrates the mutation contract without requiring an application-specific schema. It clears the populated known fields of a valid mutable message. It leaves unknown wire fields alone.

func ClearKnownFields(pb proto.Message) {
    m := pb.ProtoReflect()
    m.Range(func(fd protoreflect.FieldDescriptor, _ protoreflect.Value) bool {
        m.Clear(fd)
        return true
    })
}

Here proto refers to google.golang.org/protobuf/proto, and protoreflect refers to google.golang.org/protobuf/reflect/protoreflect. The callback returns true to continue. Range permits mutation of the current field, which makes clearing that field within the callback valid. Its iteration order is unspecified, so a tool must not infer schema order from callback order.

The example is intentionally narrower than complete redaction. Unknown fields have a separate representation and separate accessors. A policy tool must decide what to do with them. Clearing known fields does not establish that every byte of potentially sensitive input has disappeared.

Presence remains part of the data model

A reflective API must preserve distinctions between an absent field and a field with a default value where the schema supports them. The new Has method reports whether a field is populated according to protobuf rules. It is not simply a generic test of whether the corresponding Go value equals its zero value.

The tagged interface documentation describes nullable cases, including singular message fields, members of a oneof, and proto2 scalar fields. Such fields are populated when explicitly set. Other cases use different rules. A proto3 scalar field is populated when nonzero, and a repeated field is populated when nonempty.

These distinctions explain why schema-aware reflection needs its own field operations. A program that resets a Go struct member has not necessarily stated the same operation as clearing a protobuf field. Clear establishes that a subsequent Has reports false. Set also handles oneof membership by clearing another member that may already be set.

Composite values have their own access rules. Get can return an empty read-only view for an unpopulated message, list, or map. Mutable requests a mutable composite and may allocate it. This keeps reading an absent composite distinct from creating one, which matters for tools that inspect messages without intending to change their presence.

The API does not make arbitrary mutation safe. Incorrect value types can panic, and mutation methods are unsafe for concurrent use. A reflective tool therefore needs ownership of the message while it edits it. Reflection removes dependence on a generated layout, while retaining type and concurrency requirements.

Schema policy can drive generic operations

The announcement's redaction example obtains each field's options through its descriptor. It then reads the custom non-sensitive extension and clears fields whose option is false. An unset boolean extension supplies its default value, so the example treats fields as sensitive unless the schema explicitly marks them otherwise.

Extension access changes as part of the new API. The old proto.GetExtension returned a value and an error. The new function returns a value, with decoding errors reported during unmarshaling. That places malformed input handling at the decoding operation instead of requiring each later policy lookup to repeat it.

A recursive redactor still needs additional logic. Message-valued fields may contain annotated descendants, and repeated or map values require traversal of their contents. The published example explicitly stops short of that complete implementation. Its purpose is to show that policy comes from protobuf descriptors and options, not from conventions about Go field names.

I find this separation useful because it lets the schema state information that generic infrastructure needs. A consumer can implement one policy traversal without manufacturing knowledge of every generated message type. The policy still needs a precise definition for nested values and unknown data.

The reflective interface also returns the underlying message through Interface. That return path lets a tool inspect a value reflectively and then pass it to an operation expecting proto.Message. The entry and return methods connect schema-oriented manipulation with the ordinary message API without requiring the caller to name the generated Go type.

Compatibility uses one implementation underneath

An incompatible API does not require every dependency in a program to migrate together. The old module remains github.com/golang/protobuf. Its v1.4.0 API is implemented through the new API and includes conversion functions between the old and new message interfaces. That allows callers using different APIs to share the underlying implementation.

The new module begins at v1.20.0 and depends on the integrating v1.4.0 version of the old module. The announcement explains that the number makes the two API families easier to distinguish. APIv1 and APIv2 describe Go library APIs. Both support the proto2 and proto3 schema languages, so the library migration does not itself require changing a schema's syntax declaration.

The compatibility arrangement also matters for dependency selection. A program can remain on the earlier implementation until its maintainers update directly or select a dependency that requires the integrating version. Once the new API enters the program through the announced dependency arrangement, the old API uses the same implementation beneath its wrapper.

Other packages build on the new contract. dynamicpb supplies messages described at runtime, while protojson implements the canonical protobuf JSON mapping. protocmp supports comparisons with the Go comparison package, and protogen supports compiler plugins. These are different consumers of a message model that now exposes the schema and operations they need through defined interfaces.