Go's fuzzing beta separates discovery from regression testing

A test suite can replay a known failure reliably, but finding that failure may require searching through inputs nobody thought to write down. Native Go fuzzing connects those activities without giving them the same default behavior. Mutation is an explicit operation. The inputs that expose bugs can become ordinary regression cases, so repeating a failure does not require repeating the search that discovered it.

The June 3 beta announcement makes native fuzzing available for testing with the development toolchain. Katie Hockman and Jay Conrod describe semi-random input mutation that seeks new code coverage and exposes edge cases. This is a beta with incomplete functionality, and the announcement targets availability starting in Go 1.18. It is not a feature of the stable Go 1.16 toolchain.

I like the integration with Go's existing testing vocabulary. A package author can express a property near the rest of the tests, and the toolchain supplies the search mechanism. That leaves a useful division of responsibility, because the package author still has to define what counts as an incorrect result.

Coverage helps select inputs

The first-class fuzzing design draft explains the intended engine. Compiler instrumentation supplies coverage information, and a mutator generates candidate inputs. Inputs that reach new code contribute to a generated corpus that helps further exploration. This gives the search feedback about where execution went.

Coverage is useful evidence about exploration. It does not establish that the observed result was correct. A function can return the wrong value without panicking, and reaching the relevant branch will not automatically tell the engine that a domain rule was violated. The target needs assertions that express the property under examination.

This changes how a fuzz test should be read. Its arguments describe the space the engine can vary. Its body describes the observations that turn an input into a failure. The quality of those observations determines which semantic bugs the test can expose, even when the mutation engine explores a large number of cases.

The draft distinguishes a fuzz test from its target. A FuzzXxx function provides initial setup and seed values. The function supplied to Fuzz executes for each candidate. The outer function receives a testing.F, while the target receives a testing.T for reporting results. That preserves familiar test assertions inside the part that examines one input.

The beta example checks a round trip

The announcement's example uses net/url to parse a query string, encode the resulting query, and parse the encoded string again. It skips strings the first parse rejects. For accepted inputs, it requires the encoded result to parse successfully and yield the same query value. This is a property of the relationship between operations.

A shortened version of the beta example retains that property. This belongs in a test file when experimenting with the development toolchain described in the announcement. It uses a string input supported by the published beta example.

func FuzzQueryRoundTrip(f *testing.F) {
    f.Add("x=1&y=2")
    f.Fuzz(func(t *testing.T, input string) {
        query, err := url.ParseQuery(input)
        if err != nil {
            t.Skip()
        }
        encoded := query.Encode()
        again, err := url.ParseQuery(encoded)
        if err != nil {
            t.Fatalf("parse encoded query %q: %v", encoded, err)
        }
        if !reflect.DeepEqual(query, again) {
            t.Errorf("query changed after encoding: %v became %v", query, again)
        }
    })
}

The containing file needs imports for net/url, reflect, and testing. The example tests semantic equality of the parsed values. It does not require the encoded query to reproduce the original text byte for byte. Different textual spellings can represent the same query, so equality at the wrong level would make the test demand behavior the property does not need.

Skipping an initial parse error is also part of this particular property. The test asks what happens after a successful parse. It does not assert that every arbitrary string is a valid query. A separate property could examine rejection behavior, but this target deliberately chooses a smaller contract.

The seed gives the engine one valid example with two query parameters. Other meaningful seeds could exercise distinct forms accepted by the operation, but adding seed values does not replace the assertion. The assertion defines the failure, and seeds provide starting inputs for reaching it.

Search results have different storage roles

The draft gives seed inputs and generated inputs different purposes. Seeds belong to the test author and run with normal testing. The generated corpus lives outside the module in the cache and grows as the engine finds inputs that expand coverage. A generated input can be useful for continuing exploration without deserving a permanent place in a repository.

Failures need another treatment. The design describes saving a failing input so subsequent test runs can replay it. A package can retain that input as a regression case or turn the behavior into a conventional unit test. Either choice preserves the discovered problem without requiring another open-ended mutation run.

The distinction received explicit attention in proposal 44551's February discussion. Rob Pike raised concerns about CPU use and noisy test data entering repositories. Filippo Valsorda explained that the mutator-generated corpus would remain outside testdata. Only inputs producing test failures would be placed there for developers to decide whether to retain.

That discussion also addresses reproducibility. Without the fuzz flag, ordinary testing runs the seed and testdata inputs while leaving mutation disabled. The expensive search is not part of go build. It is not automatically part of every go test invocation either. This distinction is a design decision about how the testing tool fits into development.

I prefer that default because a regression suite and an exploratory search answer different questions. A regression run asks whether specified examples still satisfy their assertions. A fuzzing session asks whether another input can make the assertion fail. Giving those jobs separate invocation behavior makes their results easier to interpret.

Resource use is visible in the beta

The beta announcement warns that fuzzing can consume substantial memory and affect machine performance. By default it runs GOMAXPROCS worker processes in parallel. The parallel flag lowers the number of processes. This is a process count, so the resources consumed by one target execution can multiply across the workers.

The cache has a separate cost. Inputs that expand coverage are written beneath GOCACHE/fuzz. The announcement says there is no limit yet on the number of files or total bytes in that cache, and it may occupy several gigabytes. The development command's clean operation has a fuzzcache flag for removing it.

Those facts matter when judging the beta's readiness for routine use. They describe known operating behavior of the tooling. They do not invalidate the search model, but they make a fuzz run a resource-consuming development activity with explicit controls. Retaining only curated regressions in the repository also prevents a local search history from becoming mandatory input for every other developer.

The proposal discussion treats resource use as more than an implementation detail. Pike questions what happens if organizations turn a useful technique into a mandatory build ritual. Valsorda describes a development loop in which a code change prompts a fuzzing session, a discovered input supports debugging, and the resulting repair gains a regression test. That loop connects computation to an observable failure.

A shared format enables reuse

The draft's argument for native support is largely about integration. Existing fuzzing tools already find Go bugs, but separate commands and specialized conventions create additional work. A standard test function format and go command support give package authors a common way to expose targets. Infrastructure can then operate on that format without inventing a convention for each package.

This standardization also allows the target to survive improvements to the search engine. Its property belongs to the package. The mechanism that proposes inputs can change independently, provided it continues to run the target with its expected arguments. The draft presents a shared format as useful for future tooling and automated fuzzing infrastructure.

A target still needs to stay aligned with the package's intended behavior as the code changes. Josh Bleecher-Snyder raises that point in the February discussion, questioning whether a saved crash continues to test the intended regression after a target evolves. Preserving an input is useful evidence, but the assertion gives that input its meaning. A conventional unit test can state the specific repaired behavior when a general property no longer makes the regression clear. Both formats need review when their contract changes.

The beta is therefore useful for examining target design as well as engine behavior. A round-trip assertion can expose a relationship that ordinary example tests cover only narrowly. Saved failing inputs can make that discovery reproducible. The announced testing.F API supplies a way to express that relationship within the existing test structure while leaving mutation an explicit operation.