Kafka 2.7.2 delays file-source buffer growth until line extraction fails

A full read buffer does not necessarily mean the next operation needs a larger one. It may already contain complete records that can be extracted to make space. Kafka 2.7.2, released November 15, fixes 26 issues, including a FileStreamSourceTask buffer-growth problem. The release notes identify that repair as KAFKA-10846.

The source task reads lines from a file or standard input and emits Connect source records. Its implementation manages line splitting itself because the ordinary blocking readLine operation would interfere with interruptible task behavior. That leaves buffer management and record extraction as explicit responsibilities in the task.

I value Kafka's retained record model, but the client turning external bytes into records also needs a precise account of progress. A buffer being full and a logical record being incomplete describe different conditions. This maintenance fix separates them before allocating more memory.

The old code grew before processing its input

The 2.7.1 task implementation reads into the unused portion of a character array. After a successful read, it increments the occupied offset. If that offset reaches the array length, it immediately allocates another array twice as large and copies the existing contents.

Only after that expansion does it attempt to extract complete lines. A buffer full of short, complete lines therefore reaches the same allocation branch as a buffer containing one unfinished long line. The first case has data ready to remove, but the capacity decision happens before removal can make room.

The new array replaces the old one and remains the task's buffer. Line extraction can reduce the occupied portion without reducing capacity. Repeated decisions to grow are consequently relevant to the task's retained allocation, rather than only to a temporary copy used during one call.

This is a sequencing problem. The code knows how to consume complete lines, but it asks whether the buffer needs to grow before using that knowledge. A check on physical occupancy alone cannot tell whether the application can make progress with the bytes it already has.

The repair extracts before expanding

The 2.7.2 implementation introduces a foundOneLine flag around line extraction. It attempts to extract records first and sets the flag whenever a complete line is found. Expansion moves after that loop and requires both no extracted line and a still-full buffer.

That preserves expansion for an input that cannot yet produce a complete record. It avoids using a full read as sufficient evidence to double the allocation. The change does not add a fixed maximum capacity, and it should not be described as limiting every possible input line to a bounded size.

The buffer still doubles when the condition holds. An arbitrarily long line without its terminator can therefore require additional space. The specific improvement is that complete lines get an opportunity to free occupied space before the code concludes more capacity is necessary.

This distinction matters when translating a release-note title into an operational claim. Repairing a path to unnecessary growth is useful. Promising that the task can never use increasing memory would require a size limit or another storage strategy that this implementation does not provide.

Line extraction also advances source position

The extractor scans for newline or carriage return. A carriage return at the end of currently available data waits for another character so it can distinguish the two-character line ending. Once it finds a complete line, it creates the string and shifts the remaining characters toward the beginning of the array.

It reduces the occupied offset by the consumed portion and advances the tracked stream position where that position exists. The task then builds a SourceRecord with the source filename and position alongside the emitted value. Capacity management sits next to this accounting, but those quantities serve different purposes.

The returned record batch also has a configured size limit. Reaching that count returns the records already assembled from the current poll. A batch limit describes the number of records returned, while buffer capacity describes the space available for parsing input. Neither should be inferred directly from the other.

I find this a useful example of why maintenance code deserves source-level reading. The visible fix is small, yet its effect depends on which state is being tested and when completed work is removed. The task now uses successful line extraction before deciding that its input representation needs to expand.

Kafka 2.7.2 supplies the repair within the existing release series. Complete records can be emitted without treating every full read as a reason to double the buffer, while genuinely incomplete lines retain the existing expansion behavior. The source offset and batch accounting continue to describe progress separately from the array's allocated capacity.