Building libo11y around an OpenTelemetry registry

Every service and application I write needs observability, and setting it up is a chore that repeats in each one. Each codebase has to configure traces, metrics, and logs, establish the service's identity, export telemetry, propagate context to the services it calls, and shut down without losing buffered data. The work is nearly identical from project to project, and it is easy to get slightly wrong in any one of them. When every codebase carries its own version, small differences accumulate until telemetry from services built to the same policy no longer lines up.

That drift undermines the reason to adopt OpenTelemetry in the first place. OpenTelemetry defines one API across languages and one wire protocol, OTLP. Its semantic conventions add a shared vocabulary for attributes and metrics, so telemetry from any service means the same thing to any backend. The standard still leaves many decisions to each application, including which semantic conventions version it follows, how it names its environments, which propagators it installs, and which attributes it records. A team that makes those decisions differently in each service breaks the consistency the standard exists to provide. The HTTP conventions show how easily this happens. Services on older convention versions report http.status_code, while services on the current stable version report http.response.status_code, so an error-rate query written for one misses the other. Service maps, SLOs, dashboards, and alerts built on semantic conventions all depend on that consistency. No component reports an error when a service deviates, and its telemetry is simply missing from the views built on the shared names.

I wanted one standard implementation of that work, built carefully once and shared by every application I write. libo11y is that implementation. It owns the setup, and application code then uses the standard OpenTelemetry APIs and its language's usual logging facade, with no wrapper types for spans, metrics, or logs. The chore is not specific to my projects, so I am releasing libo11y as open source under the Apache 2.0 license for other developers who face it.

The wider plan

The Go module is the first implementation of a larger design. I plan libo11y for Go, Rust, Swift, C++, C, Node.js, Ruby, PHP, Erlang and Elixir, browser JavaScript, and Python, and after those for Java, Kotlin, Zig, and .NET with C#. I added Python begrudgingly. The last four come later because I know them least well, and I want to build fluency in each before I write a library other teams depend on. On the JVM, opinionated platforms such as Spring Boot already prescribe how an application produces telemetry, through the Micrometer Observation API and Micrometer Tracing, which can export to OpenTelemetry. A libo11y Java library will have to work alongside that model, because most Spring teams will keep using it. Browser JavaScript gets its own library, separate from Node.js, because client-side telemetry runs on devices I do not control and has no long-lived service identity. Each library implements the same design contract, which lives in docs/DESIGN.md, in the idiom of its own language. The contract fixes the behavior that has to match from one service to the next. Telemetry does not start without a usable service.name and a valid deployment.environment.name. Trace identity comes only from the active span context, and W3C Trace Context crosses every hop. Explicit options win over OTEL_* variables, and those win over library defaults.

Every language also shares one registry. The generator that produces go/spec today is meant to produce the typed keys, enums, and sensitivity tables for each of the other languages from the same YAML. A Python worker and a Rust service then validate against identical definitions. Where OpenTelemetry ships an official SDK for a language, the libo11y library builds on that SDK and its standard environment handling instead of reimplementing them. C and Zig have no official SDK, so those libraries will either wrap the C++ SDK or speak OTLP directly. Go is the only implementation released so far, and the other libraries have not shipped.

Polyglot systems

Most systems I work on cross language boundaries on the request path. A request can enter through a Node.js frontend, call a Go API, hand work to a Python worker, and finish in a Java billing service. When each team configures OpenTelemetry on its own, the telemetry diverges at every boundary. One service reports its environment as prod and another as production. Attribute names drift too, with user_id in one team's code and user.id in another's. A service that installs no propagator ends the trace at its door, and the next service starts a new one.

A shared contract removes those differences where the telemetry is created. Attribute keys and enum values come from the same registry in every language, so a query or alert that filters on deployment.environment.name="production" matches every service. W3C propagation is mandatory in each implementation, so one trace covers the whole request. The sensitivity table is shared as well, and a key classified as PII is dropped in the Python worker for the same reason it is dropped in the Go API. Each resource carries the same schema URL, which tells a backend which version of the conventions the service follows.

The design contract also describes how this will be checked across languages. Each implementation's example application will run against a shared OTLP capture harness, and a comparison of the normalized payloads will cover resource attributes, metric names, trace and log identity, and histogram buckets. That conformance suite is planned work.

Small and medium teams

Large organizations usually handle this with a platform team that maintains an internal telemetry SDK for each language. That team chooses the attribute names, writes the setup code, follows OpenTelemetry releases, and fixes the cases where a library leaks a secret into a span. A team of five or fifty engineers rarely has people to spare for that work. Each service then carries its own copy of the setup code, along with its own mistakes.

libo11y is meant to give those teams the result of that platform work without the headcount. Adopting it costs one setup call per service and agreement on the registry. In return a team gets required service identity, bounded shutdown, PII dropped by default, log and trace correlation, and metric exemplars. It also gets a test harness whose AssertConforms fails a unit test when emitted telemetry breaks the specification. A team can add its own environment names with WithEnvironmentValues. All telemetry leaves over OTLP, so changing observability backends does not touch application code.

Where it fits in an observability practice

Observability depends on joining signals. An engineer investigating a latency spike moves from a histogram bucket to an exemplar trace, from the trace to its spans in other services, and from a span to the log lines it produced. Each step relies on identifiers and attribute names that every service emitted the same way. A single service with a different environment value, a missing propagator, or logs without trace IDs breaks the path at that service.

The OpenTelemetry Collector handles a lot of fleet-level processing after the fact, such as sampling, enrichment, and attribute renaming. It cannot add a trace ID that a service never propagated, and it cannot recall a password that has already left the process. libo11y puts those guarantees in the SDK, where the telemetry is created, and leaves fleet-level processing to the collector.

Standardizing in the library also makes convention changes deliberate. The registry is versioned, and published schema files never change. Future renames will be recorded as OpenTelemetry schema transformations that a backend can apply. A team that changes a convention ships it as a new spec version with a recorded migration, and every service and dashboard can move to it on a known schedule.

One setup path

o11y.Setup constructs tracer, meter, and logger providers with one resource and propagator. It installs them as process globals, installs the OpenTelemetry error handler, and sets the default slog logger unless configured otherwise. Exporters use the standard OTEL_* variables, with explicit options taking precedence. A service can initialize the complete pipeline with the required identity.

sdk, err := o11y.Setup(ctx,
    o11y.WithServiceName("checkout"),
    o11y.WithEnvironment(spec.EnvironmentProduction),
)
if err != nil {
    return err
}
defer sdk.Shutdown(context.Background())

o11y.New constructs the same SDK without changing global state. I use it for tests, dependency injection, and processes that need isolated telemetry pipelines. Setup permits one active global SDK, while New has no process-wide guard.

The returned SDK owns shutdown. It flushes and stops logs, metrics, and traces concurrently. A context without a deadline receives a five-second deadline. Repeated calls return the first shutdown result, and errors identify the affected signals.

Strict identity requires a usable service.name and a valid deployment.environment.name. The service name may come from WithServiceName, OTEL_SERVICE_NAME, or service.name in OTEL_RESOURCE_ATTRIBUTES. The environment may come from WithEnvironment or deployment.environment.name in OTEL_RESOURCE_ATTRIBUTES. Options win over environment values.

The generated spec.Environment type defines production, staging, development, and test. WithEnvironmentValues adds organization-specific values. Missing values produce ErrNoServiceName or ErrNoEnvironment, while an unknown environment produces ErrInvalidEnvironment. WithoutStrictIdentity converts those failures into recorded resource violations for migration and tests.

The resource taxonomy also covers version and instance identity. service.version falls back to 0.0.0-unknown when an option, environment value, and build information provide no version. service.instance.id uses a UUIDv5 derived from Kubernetes pod identity when those attributes exist. Other processes receive a random UUIDv4.

The integration packages bind common servers and clients to the same providers and propagator. o11yhttp supplies net/http server middleware, an instrumented client transport, route-based span names, filters, and an optional request summary event. o11ygrpc supplies client and server stats.Handler implementations for traces, metrics, and context propagation. o11ygin supplies Gin middleware in a separate Go module so applications without Gin do not acquire its dependencies.

The registry is the telemetry contract

The telemetry contract resides in YAML under spec/registry. The manifest identifies the libo11y schema and the upstream semantic conventions release that the registry extends.

schema_url: https://schemas.o11y.ing/libo11y/1.0.0
semconv_version: 1.41.0

Resource tiers, forbidden attributes, metric instruments, span requirements, events, enums, sensitivity classes, and cardinality limits are registry data. go/internal/specgen decodes every registry file strictly and rejects unknown fields. Its policy checks cover naming, namespace ownership, sensitivity, cardinality, references, enums, event definitions, metric units, stability values, requirement levels, annotations, and generated Go name collisions.

Generation produces four sets of artifacts. go/spec contains typed keys, closed enums, event names, and version constants. go/internal/spectables contains the runtime and conformance lookup tables. docs/taxonomy contains the rendered resource documentation. spec/schemas/1.0.0 is the OpenTelemetry schema file for the registry version.

make spec-check validates the registry without writing generated files. Before writing generated files, make spec-generate runs the same validation and stops if it fails. Drift checking through make spec-drift regenerates the artifacts, checks the Git diff, and rejects untracked generated files. The CI gate therefore detects manual edits and generator drift.

Every resource built by libo11y carries https://schemas.o11y.ing/libo11y/1.0.0. Published schema versions are immutable. A specification change requires a new version and another schema file. Future renames must appear both as registry tombstones and OpenTelemetry schema transformations. The current generator refuses to produce a version after 1.0.0 until the registry can record that transformation history.

One sensitivity policy across the signals

The generator builds one key-based sensitivity table from the registry. Each SDK uses that table for logs, spans, its resource, and metric streams created through its meter provider.

Secret attributes are always dropped. PII attributes are dropped by default through PIIDrop. PIIAllow retains them after an explicit opt-in. PIIHash replaces values on logs, spans, and resources with hmac-sha256: followed by 32 hexadecimal digits. The token is a 128-bit truncation of HMAC-SHA256 over a canonical value rendering.

WithPIIMode selects the behavior. WithPIIHashKey supplies the key. New and Setup return ErrPIIHashKey if hash mode receives fewer than 32 key bytes. A stable key and the shared rendering produce the same token for the same value in a log attribute, span attribute, or resource attribute.

Metrics apply a stricter rule. SDK-owned metric streams filter data point attributes and exemplar filtered attributes. They drop PII in both PIIDrop and PIIHash modes because a separate token for each value would create additional series. Only PIIAllow retains those attributes.

The log policy covers console output, extra slog handlers, and the OTLP path used by the built-in slog bridge. A logger processor applies the same classification to records from other bridges using SDK.LoggerProvider. Grouped slog attributes use their dotted effective keys, and composite log values are walked with cycle, depth, and node limits.

Span classification covers span attributes and the attributes attached to span events and links. The exporter receives a redacted span view. A processor supplied through WithSpanProcessor also receives redacted data in OnEnd, although its OnStart callback still sees the live span.

Resource classification runs after detector, environment, and option values have been composed. OpenTelemetry Go merges its environment resource underneath the supplied resource. libo11y therefore shadows a dropped key from OTEL_RESOURCE_ATTRIBUTES with an empty value so the environment layer cannot restore the original value.

Drops and rewrites on logs, spans, and resources increment o11y.schema.violations. Its dimensions are o11y.signal and o11y.rule. Sensitivity failures use the rules secret_attribute and pii_attribute. Relaxed identity checks use identity_service_name, identity_environment, and identity_environment_invalid. Metric filtering does not increment the counter because it runs on the measurement path.

Key classification cannot protect a sensitive value stored under an unclassified key. The redact package provides capture-time protection for those cases. redact.Secret replaces a value across slog, formatting, and JSON output. redact.Masked preserves only the rune count. redact.Struct applies mask, redact, and omit tags recursively, with the strictest enclosing protection taking precedence.

An unknown o11y tag renders as REDACTED. Cyclic values and structures deeper than 32 levels also render as REDACTED. Map keys whose types may conceal protected data are redacted, as are errors whose reachable values may contain protected data. Panicking string and text-marshaling methods fail closed. Ordinary untagged scalar values still pass through, so callers must identify the values that need capture-time protection. The implementation documents these remaining limits:

• Values under unclassified keys receive no automatic protection.

• Log bodies and other free text remain unclassified. This includes span names, span status descriptions, and span event names.

• Instrumentation scope attributes are not classified.

• Metrics supplied through OpenTelemetry metric producers bypass SDK views and the sensitivity filter.

• Members of map-valued span and metric attributes are judged only by the outer attribute key.

• An injected span processor can inspect live attributes during OnStart.

• Baggage remains unclassified. The egress filter wraps every configured propagator, but its default predicate permits every key.

• Experimental advisory attributes from instruments are not accepted because every SDK-owned stream applies the policy attribute filter.

Conformance at test time and on the wire

o11ytest.Start constructs an in-memory SDK through o11y.New and WithoutOTELEnv. It changes no globals and opens no network connections. Ambient exporter, sampler, batching, log limit, resource, cardinality, and exemplar configuration cannot alter the harness. Two experimental OpenTelemetry Go switches have no overriding API, although the harness disables runtime metrics and therefore removes one of their effects.

OpenTelemetry providers still merge the environment resource internally. WithoutOTELEnv counters that behavior by shadowing every ambient resource key that the library did not define. Such a key appears on recorded telemetry with an empty value instead of its environment value.

Recorder.AssertConforms flushes before checking the result. It reports secret attributes detected through the violation metric, registered events missing required attributes, secret attributes that remain visible after processing, and missing tier-zero resource attributes. It also resolves grouped log attributes to their dotted keys. Secret metric attributes cannot appear in this report because the metric filter drops them without recording a violation.

The repository tests the exported representation as well as the in-memory SDK data. TestOTLPWirePayloads starts local gRPC and HTTP receivers for traces, metrics, and logs. The HTTP receiver decodes protobuf requests on the three OTLP paths. The test inspects decoded resource and signal payloads for schema URLs, removed sensitive values, correlation identifiers, and metric exemplars. make ci runs the repository's complete local gate set:

• lint runs golangci-lint for the main module and the Gin module.

• vet runs go vet for both modules.

• test runs both module suites with the race detector and bounded package parallelism.

• tidy-check requires go mod tidy -diff to produce no changes in either module.

• spec-check validates the registry.

• spec-drift checks every generated artifact.

• schema-published-test exercises the schema publication checker against a loopback server.

• release-notes-test tests scripts/release-notes.sh, which extracts a selected changelog section for GitHub Release notes.

• release-check validates the required module release sequence against a temporary local remote.

• vuln runs govulncheck for both modules after the other gates.

Current status

libo11y remains pre-1.0 under semantic versioning, so its API may change between minor versions. I released go/v0.1.0 and go/o11ygin/v0.1.0 on 2026-10-12. Both tags identify the same commit, which also carries spec/v1.0.0. The modules require Go 1.26.0 or later and are available through proxy.golang.org and pkg.go.dev.

go get github.com/jamesainslie/libo11y/go@v0.1.0
go get github.com/jamesainslie/libo11y/go/o11ygin@v0.1.0

The tag-triggered workflow in .github/workflows/release.yml validates the semantic version and requires a matching changelog section. It runs make vet test spec-check spec-drift, publishes the GitHub Release with notes extracted from the ## [0.1.0] section of go/CHANGELOG.md, and builds an external consumer against the module served by proxy.golang.org.

The current registry assigns one o11y.sensitivity value to each classified attribute. Its values are public, internal, pii, and secret. I plan to replace that single classification with two independent fields. One will express confidentiality, while the other will record whether the value contains personally identifiable information. That model is planned work and is not present in the generated tables or runtime policy.