glibc 2.26 puts a small allocation cache in each thread

A thread that frees a small allocation may soon need another allocation of the same size. Repeating the full arena operation for that sequence spends work on coordination even when a recently freed chunk is already suitable. glibc 2.26 adds a per-thread cache that handles eligible allocations and frees without locking an arena. The August 2 announcement identifies DJ Delorie as its contributor, with help from Florian Weimer and Carlos O'Donell.

The release description makes the boundary explicit. Accessing the cache needs no lock, while refilling an empty cache requires the underlying arena. The optimization therefore changes the common route through the allocator without replacing the arena allocator itself. Requests that cannot use cached chunks still depend on the existing allocation machinery.

I like changes whose performance argument can be traced through the implementation. Here the relevant saving is specific, a successful cache lookup returns a chunk before the arena path. That makes the claim easier to assess than a general statement that memory allocation has become faster for every program.

The cache stores lists indexed by allocation size

The 2.26 malloc implementation defines a thread-local pointer to a cache structure. That structure contains an array of list heads and an array of counts. The source comment notes that the counts could be reconstructed by traversing the lists, but retaining them avoids that work.

Each cached entry contains a pointer to the next entry. The implementation overlays that entry on the freed chunk's user storage. The list therefore does not require a separately allocated node for each free operation. Once a chunk enters the cache, the allocator uses part of the storage formerly available to the application as bookkeeping.

The insertion operation links a chunk at the front of its size-class list and increments the count. Removal takes the first entry, advances the head to its next pointer and decrements the count. This is a last-in, first-out list within each class. Its simplicity is part of why a cache hit needs little work.

Those implementation details describe allocator-owned storage. An application has no right to keep using a chunk after freeing it merely because the same addresses remain mapped. The cache's use of that storage for links makes the separation visible in the source. A freed pointer is not an application handle to an idle object waiting for its next use.

A cache hit returns before arena acquisition

In __libc_malloc, the request first maps to the allocator's internal chunk size and then to a cache index. The implementation initializes the thread cache when necessary. It checks that the index belongs to the configured bins and that the corresponding list is nonempty. A successful check returns the entry through tcache_get.

Arena acquisition follows that return path in the source. This ordering is the mechanism behind avoiding the arena lock on a hit. It does not mean that the first allocation in a thread has the same cost as every later hit. Cache initialization allocates the per-thread structure through the arena machinery and can fail under memory pressure.

The initialization code handles that failure by leaving the cache unavailable and trying again on a later operation. The program still has the ordinary allocator path. This is an important property of the implementation, since a performance cache should not become a separate application-visible requirement for every small allocation to succeed.

Freeing a chunk has a corresponding eligibility decision. After initial validity checks, _int_free calculates the cache index and checks whether the thread's cache has room in that class. If so, it inserts the chunk and returns. If the class is unavailable or full, execution continues into the allocator's other free paths.

This also explains the meaning of per-thread ownership. The implementation uses the cache of the thread performing the operation. The arena remains part of the chunk's management, but the immediate cache route is selected from thread-local state. A program's allocation and free pattern therefore influences which thread accumulates reusable chunks.

Local reuse retains memory locally

The versioned tunables manual documents the cache's limits. glibc.malloc.tcache_max controls the largest request that can use it. The documented default and maximum are 1032 bytes on 64-bit systems and 516 bytes on 32-bit systems. A request beyond the supported size does not become a cache hit merely because another cached chunk is large enough in some informal sense.

glibc.malloc.tcache_count limits the number of chunks held in each size class. Its default is seven, and zero effectively disables the cache. These are per-class limits, so counting only the maximum request size understates the possible retained storage. Several lists can each hold several chunks at once.

The manual estimates the default maximum per-thread overhead at approximately 236 KB on 64-bit systems and 118 KB on 32-bit systems. These are approximate capacity figures, not a claim that every thread immediately allocates that much. Actual occupation depends on which classes the thread fills and how subsequent requests reuse them.

The per-thread scope creates the memory tradeoff. Cached chunks are immediately available for reuse through that thread's lists, while retaining them increases the allocator's local reserves. Across many threads, the capacity is repeated. A program with little reuse can pay retention costs without obtaining the same frequency of cache hits as a program that repeatedly allocates matching sizes.

The source includes thread cleanup that walks the cache lists and frees their entries. It clears the thread's cache pointer before doing so, preventing those frees from simply returning the same entries to that cache. It then releases the cache structure and records shutdown state. That code makes thread lifetime another relevant part of retention, alongside request sizes and counts.

There is also a distinction between the public allocation entry points in the source. __libc_calloc performs its own product-overflow check and clearing logic. Its implementation initializes the cache when appropriate but proceeds through an arena allocation path rather than duplicating the direct lookup shown in __libc_malloc. The existence of a malloc cache should therefore not be expanded into a claim that every allocation function takes an identical path. The entry point's obligations still matter, including whether newly returned storage must contain zeroes. Reading each path separately gives a more accurate account than treating all calls that return allocated memory as equivalent cache requests.

Refilling a cache has a different cost

The tunables documentation describes how a request that reaches an arena can also move additional chunks from arena lists into the cache. That prefill gives later requests a chance to take the shorter path. Work performed during one miss can therefore reduce work on subsequent hits.

Not every source list has the same scanning cost. The manual distinguishes bounded transfers from fastbins, smallbins and regular bins from work on the unsorted list. Chunks scanned from the latter need sorting, and the amount of scanning is not bounded by the size-class structure in the same way.

glibc.malloc.tcache_unsorted_limit lets a user bound the number of unsorted-list chunks scanned while looking for prefill candidates. Its default, also selected by zero, imposes no limit. That control concerns refill work. It should not be confused with tcache_count, which controls how many chunks a class can retain.

These controls are available through the tunables mechanism, which the release enables by default. For example, an ordinary shell invocation can disable tcache for a process using the documented zero-count setting. This command illustrates configuration, not a claim that disabling it improves a particular program.

env GLIBC_TUNABLES=glibc.malloc.tcache_count=0 ./application

The release announcement reports gains across workloads collected with an instrumented allocator and examined using a simulator. That is evidence supplied by the project, not a measurement of any application discussed here. The source provides a narrower prediction that can be stated without inventing numbers. Repeated eligible operations with populated lists can avoid arena locking, while misses and requests outside the cache's range continue to use arenas.

I consider the retention cost a reasonable part of that design, provided it remains visible. The versioned documentation supplies a size ceiling, a per-class count and a separate refill-scanning control. Those quantities explain how tcache trades local reserves for fewer arena operations. The implementation's early return on a hit and its cleanup at thread exit give that tradeoff a concrete beginning and end in the allocator's control flow.