Docker Engine continues log reads after a damaged file
Engine 27.4 skips the remainder of a damaged log file and continues with subsequent files. The change preserves access to readable history without guessing where corrupted records resume.
A damaged record in an old log file can prevent access to newer records if the reader treats every file as one uninterrupted stream. The resulting failure is larger than the corruption itself. Docker Engine 27.4 changes that behavior by using a file boundary as the place where reading can resume safely.
The December 9 release notes describe the change for the JSON-file and local logging drivers. When reading or parsing an underlying log file fails, the reader skips its remainder and moves to the next file if one exists. The client stream no longer has to close at the first such failure.
I value Docker's repeatable packaging, but operational use also depends on being able to inspect what a container has done. Losing readable recent history because an older file is damaged makes that inspection harder. This change limits how much history a single parsing failure makes unavailable.
File boundaries become recovery boundaries
Brian Goff's implementation in PR #48842 explains why recovery stops at the next file. The commit message says the author investigated scanning ahead and correcting errors, but considered it too risky to parse content that should not become a record. The chosen behavior discards the remainder of the affected file.
That tradeoff is specific. The reader preserves access to subsequent files without claiming to reconstruct every valid record after the damaged region. There can be readable-looking bytes in the skipped portion, but accepting them would require another rule for determining where a trustworthy record begins.
The patch replaces the combined-reader approach with handling that resets the decoder for each selected file. A non-EOF decode error is logged, and the wrapper returns an end-of-file result to the forwarding code. This lets the outer loop advance to another file instead of terminating the entire read request.
Using EOF internally does not mean corruption is silently reclassified as an intact file. The implementation records the decode failure before ending that file's contribution. The release notes say these errors remain visible in daemon logs and enter traces when tracing is configured.
The client therefore receives the records the reader can continue to interpret, while the daemon retains evidence that part of the history was skipped. Those are separate outputs with different purposes. Continued delivery establishes availability of later records, not completeness of the original stored history.
An investigation using that output needs to retain the daemon's error information alongside the returned records. Otherwise, an omitted interval could be mistaken for a period in which the application wrote nothing.
Tail selection and decompression also change
The same work refactors handling of rotated files so compressed files are decompressed only when needed. Previously, starting a read could begin decompressing files that the requested tail did not require. A request for recent records should not necessarily have to process every older compressed file first.
The patch introduces file opener objects and separates selecting readers from forwarding decoded messages. Its compressed-file implementation can examine rotation metadata in the gzip header. Where the stored last timestamp precedes the requested start time, it can avoid reading that file's content for the request.
This improves the relationship between the user's requested window and the work needed to satisfy it. The change does not publish a universal speedup or a fixed resource saving. The supported improvement is avoiding decompression that the selected request does not need.
The reader also closes resources as it moves through files and handles early termination. Cancellation and a disappearing consumer are checked in the forwarding path. Continuing after corrupt input must still respect the caller ending the request, since resilience to one failure does not require reading indefinitely after the output is no longer wanted.
The recovery policy deliberately loses part of one file
Skipping a file's remainder is a conservative choice about parsing. It accepts a bounded loss of potentially recoverable records instead of inventing a resynchronization heuristic. The next file supplies a boundary the existing decoder can use without searching through arbitrary bytes for a plausible message.
That is the part of the change I find persuasive. The implementation states where recovery resumes and what it abandons. It gives an operator access to later history while preserving an explanation for missing records in the daemon's own diagnostics.
The release also corrects manifest ordering in image-list API output. With the manifests option enabled, GET /images/json preserves the order from the original image index. That change concerns faithful presentation of stored metadata, independent of the log reader's recovery behavior.
For logging, the final behavior is narrower than repairing damaged storage. The reader skips the rest of the affected file, reports the failure through daemon diagnostics, and continues with subsequent readable files. That preserves an explicit distinction between making surviving records available and claiming that the returned stream contains every record originally written.