April 7, 2022·4 min read·engineeringrustcoveragetestingcargo

Rust 1.60 maps execution counters back to source regions

Stable coverage instrumentation connects runtime counters to source regions embedded in the binary. Cargo's build timings provide separate evidence about compilation and dependency scheduling.

A coverage percentage is only interpretable when the tool explains what it counted. A function can run without every expression inside it executing, and one source line can contain several independently evaluated regions. Rust 1.60 stabilizes source-based coverage instrumentation that records execution against a compiler-generated map of those regions.

The 7 April announcement introduces the compiler flag alongside Cargo's build timing reports and new feature syntax. Coverage and timings measure different activities. One describes execution of an instrumented program, while the other describes the work of building it.

I find Rust's semantics compelling, but a useful development environment also needs evidence about what the program and its tools actually did. Source-based coverage interests me where it makes a missing test path concrete. The percentage is less useful than a report that identifies the expression the test never evaluated.

The binary contains the source mapping

The coverage documentation shipped with Rust 1.60 describes two parts of instrumentation. Rustc injects calls that increment execution counters and embeds information mapping those counters to source-code regions. The mapping records positions in the source, including region starts and ends.

Running the instrumented executable writes raw profile data when the program terminates. The counts alone do not supply the source report. Reporting tools need the matching binary too, because it contains the map that gives the counters their meaning.

That relationship matters for artifact handling. A profile from one executable and a map from another are not interchangeable merely because both came from the same source repository. The report describes an instrumented build and the runs associated with it. Keeping those artifacts together preserves the explanation of each count.

The basic build configuration can be supplied through Cargo's compiler flags. This command enables instrumentation for a project that builds a binary by default.

env RUSTFLAGS="-C instrument-coverage" cargo build -j 1

The explicit job count bounds this example's build concurrency. Instrumentation does not itself execute the program or select a meaningful workload. The subsequent run supplies the execution observations from which a report can be built.

Raw observations need processing

The versioned documentation describes llvm-profdata as the tool that merges and indexes raw profiles. Llvm-cov then combines that processed profile with the binary to produce summaries or source annotations. These are distinct steps, even when a wrapper automates them.

The default raw output filename is default.profraw. LLVM_PROFILE_FILE can choose another path and use documented substitutions such as a process identifier. That becomes useful when several program runs produce data and should not overwrite one another's output accidentally.

The release recommends the llvm-tools-preview component to obtain tools compatible with the compiler. Stabilizing the instrumentation flag does not freeze the raw profile format or every LLVM tool interface. The announcement specifically calls for matching the tool component and the compiler version.

This is an important compatibility boundary. An old coverage command copied from another environment can appear to use the right filenames while still invoking incompatible tools. The flag's stability concerns the supported compiler capability, while report production depends on the data and tool versions involved.

Functions, instantiations and regions differ

The coverage guide distinguishes function, instantiation, line and region coverage. Function coverage counts a function as executed when one of its instantiations executes. Instantiation coverage distinguishes versions of a generic function or code generated through macros. A generic function's name therefore does not describe every compiled instance.

Region coverage gives a more detailed view than whether an executable source line ran at least once. A boolean expression can stop after its first operand, leaving later operands unevaluated on that run. Several such regions can share a line. A line-oriented view can conceal that difference.

That affects how a test gap should be described. A report marking a line as covered does not establish that the test explored every evaluation path on that line. Conversely, an unexecuted generic instantiation can point to a particular use of a function rather than a wholly untested source definition.

My preference is to use coverage to identify an unanswered behavioral question. A counter can show that an error-handling region never ran. It cannot establish that an assertion would detect the wrong behavior if the region did run. Execution evidence and test sensitivity remain different properties.

Cargo timings describe build dependencies

The Cargo timing documentation for this release defines a unit as one compiler invocation. Its report displays each unit's duration and shows which other units become eligible to run when it completes. That provides a way to inspect the dependency chain affecting elapsed build time.

cargo build -j 1 --timings

The report is written under target/cargo-timings, with a timestamped copy as well as the current report. Its concurrency view distinguishes units waiting for an execution slot from units waiting for dependencies. Those are different reasons a compiler invocation has not started.

With one job, the example deliberately serializes Cargo's scheduled work. A report from a more concurrent build can reveal which dependency completions constrain available parallelism. The documentation cautions that this view does not show the compiler's internal concurrency, so active-unit counts should not be interpreted as a full count of execution threads.

That distinction makes the report useful for a specific diagnosis. If many units are blocked on one crate, adding execution slots cannot start them before that crate finishes. The dependency graph constrains the scheduling opportunity. If units are ready but waiting for slots, the limit being observed is different.

The timing report also identifies custom build-script units. Their duration can therefore be considered separately from the ordinary compilation units that depend on them.

I regard the two reporting capabilities as complementary evidence. Coverage explains where a particular program run executed source regions. Build timings explain how compiler invocations depended on one another during a build. Both become more useful when their units are preserved in the discussion, so a percentage or duration remains connected to the activity it actually describes.