The cache line

The note at the top of torch/_dynamo/cache_size.py describes the structure in one sentence: a linked list of entries, each a check function, an output code object, and a next pointer, hanging off the code object's co_extra scratch space. _debug_get_cache_entry_list hands you the head, and the walk is six lines.

Three captures of one function give three entries, each with a check_fn whose qualified name is ___make_guard_fn.<locals>.guard, a closure generated at compile time and holding the predicates lesson two printed. Every call walks this list from the head and runs check functions until one passes.

The recompile log therefore prints one failure line per entry rather than one per call. Three identical failure lines are not a repeated message; they are three cache entries each rejecting the same arguments for the same reason.

import torch
from torch._C._dynamo import eval_frame

def f(x, alpha):
    return torch.sin(x) * alpha

cf = torch.compile(f, backend="eager")
cf(torch.randn(4), 2.0)
cf(torch.randn(8), 2.0)
cf(torch.randn(4), 5.0)

entry, n = eval_frame._debug_get_cache_entry_list(f.__code__), 0
while entry is not None:
    n += 1
    print(n, entry.code.co_name, entry.check_fn.__qualname__)
    entry = entry.next
print("entries on f.__code__:", n)
# 1 f ___make_guard_fn.<locals>.guard
# 2 f ___make_guard_fn.<locals>.guard
# 3 f ___make_guard_fn.<locals>.guard
# entries on f.__code__: 3

cache_size_limit is 8 and accumulated_cache_size_limit is 64, and the note in cache_size.py explains why one number was not enough. Identity guards on nn.Module instances mean a normal model produces many entries on one code object through no fault of the author, so the small limit counts entries per identity bucket and the large one caps the code object overall.

Sixty-four is therefore the number that stops an ensemble. Ten instances of one module class compile ten graphs and nothing complains; the sixty-fifth instance trips the ceiling. Eight is the number that stops a function specializing on a Python value, which is the more common accident: nine distinct floats through one argument is enough.

Both cases print the same warning, and it names config.cache_size_limit (8) either way, because will_compilation_exceed checks the bucket limit and the total and the log line only ever formats the first. When you see that warning after more than eight compiles, read it as the accumulated limit and go look for identity guards.

The failure mode is worse than a stop. Watch the entry count across twelve calls with twelve distinct constants: it climbs to eight, and on the ninth call it goes to zero. The whole cache line is dropped, the code object is skipped from then on, the frame counter stops moving, and every later call runs eager. Eight compiles paid for and discarded in one call.

Dynamo does not stop working at that point, it moves down. In the variant below, f calls a helper; once f is skipped, the callback still sees the helper's frame and compiles that instead, so the helper's own code object grows an entry while f has none. Capture drops one level and quietly keeps a fraction of the benefit.

Compare the layer below, where the same cache-key discipline is paid a second time. The torch_xla dynamo bridge, in the lesson /pytorch/bridges/three-modes-one-machine, throws on a cache miss it cannot recover from, because it discarded the graph. Here, bursting the limit silently returns you to eager. One failure is loud and one is a slow benchmark, and neither is a recompile.

i  entries  stats                                        frames
0  1        {'calls_captured': 2,  'unique_graphs': 1}   {'total': 1, 'ok': 1}
3  4        {'calls_captured': 8,  'unique_graphs': 4}   {'total': 4, 'ok': 4}
7  8        {'calls_captured': 16, 'unique_graphs': 8}   {'total': 8, 'ok': 8}
8  0        {'calls_captured': 16, 'unique_graphs': 8}   {'total': 9, 'ok': 8}
9  0        {'calls_captured': 16, 'unique_graphs': 8}   {'total': 9, 'ok': 8}
11 0        {'calls_captured': 16, 'unique_graphs': 8}   {'total': 9, 'ok': 8}

WARNING torch._dynamo hit config.cache_size_limit (8)
WARNING    function: 'f' (burst.py:5)
WARNING    last reason: L['alpha'] == 0.0     # return torch.sin(x) * alpha

# and with a helper in the middle, capture moves one frame down:
# call 7   f: 8 entries   helper: 0 entries   frames {'total': 8,  'ok': 8}
# call 8   f: 0 entries   helper: 1 entry     frames {'total': 10, 'ok': 9}

The chapter's rule is that a new shape triggers a new capture. True once. automatic_dynamic_shapes is on by default, and the second distinct size for a dimension makes dynamo recompile that dimension as symbolic rather than as another constant. Shapes 4 then 8 cost two compiles. Shapes 16 and 32 after them cost nothing.

The guard is where the change is legible. The first entry checks size=[4]. The second checks size=[None] and adds a separate line, 2 <= L['x'].size()[0], because zero and one are specialized: a dimension of length 1 broadcasts, and code that is correct for 1 is not automatically correct for n. So a call at size 1 recompiles even against a dynamic entry, and that third graph checks size=[1].

mark_dynamic(x, 0) moves the same decision one compile earlier. Marked from the first call, sizes 4, 8, 16 and 32 share one graph and there is no static entry at all. It buys the first compile back, not correctness, and the size-1 call still compiles separately. dynamic=False goes the other way, turning automatic dynamic off and giving three shapes three graphs.

What dynamic shapes do not do is remove guards. Branch on x.shape[0] > 4 inside a marked-dynamic function and the equality check becomes an inequality: L['x'].size()[0] > 4, which holds for 8 and 16 and fails at 3. Symbolic shapes move the predicate from equality to a range; the range still comes from your Python.

import torch
from torch._dynamo.utils import counters

@torch.compile(backend="eager")
def f(x):
    return torch.sin(x) + 1

for n in (4, 8, 16, 32, 1):
    f(torch.randn(n))
    print("plain ", n, counters["stats"]["unique_graphs"])
# plain  4 1     [0/0] check_tensor(... size=[4], stride=[1])
# plain  8 2     [0/1] check_tensor(... size=[None], stride=[1])
# plain  16 2          [0/1] 2 <= L['x'].size()[0]
# plain  32 2
# plain  1 3     [0/2] check_tensor(... size=[1], stride=[1])

counters.clear(); torch._dynamo.reset()

@torch.compile(backend="eager")
def g(x):
    return torch.sin(x) + 1

for n in (4, 8, 16, 32, 1):
    y = torch.randn(n)
    torch._dynamo.mark_dynamic(y, 0)
    g(y)
    print("marked", n, counters["stats"]["unique_graphs"])
# marked 4 1 · marked 8 1 · marked 16 1 · marked 32 1 · marked 1 2