The frame it rewrites

The chapter above calls it attaching dynamo to a function, which is the right picture for reasoning about guards and the wrong one for reasoning about what runs. Open torch/_dynamo/eval_frame.py and the wrapper's body is a context. On entry it calls set_eval_frame(callback) and saves whatever callback was installed before; in the finally it puts the old one back. Between those two lines the interpreter asks dynamo about every Python frame it is about to execute, not only the one you decorated.

That single fact explains behaviour you would otherwise file as magic. A function dynamo has given up on can still have its callees compiled, because their frames are offered to the callback too while the wrapper is on the stack, which is where lesson four ends. And the C extension module holding the machinery is small enough to inventory in one line of Python: set_eval_frame, skip_code, reset_code, unsupported, set_guard_error_hook, _debug_get_cache_entry_list, and the two types _CacheEntry and _PyInterpreterFrame.

The unit of everything that follows is the code object, not the function object. Guards attach to a code object, the cache attaches to a code object, and the compile counter in every log line counts per code object. Build two closures from one def with equal captured values and they share a single cache entry and a single graph between them.

import torch
from torch._C._dynamo import eval_frame

print(sorted(n for n in dir(eval_frame) if not n.startswith("__")))
# ['_CacheEntry', '_PyInterpreterFrame', '_debug_get_cache_entry_list',
#  'reset_code', 'set_eval_frame', 'set_guard_error_hook', 'skip_code',
#  'unsupported']

Set TORCH_LOGS=bytecode and dynamo prints what it read and what it wrote. The function below is two tensor ops and a return. Its original code object is eleven instructions: a global load for torch, an attribute load for sin, a call, a store, then the add and the return.

The modified version is eight instructions and none of them mention torch. The entire body has been replaced by a call to __compiled_fn_0, a global dynamo installed on your module, followed by UNPACK_SEQUENCE because compiled functions return a tuple of outputs. Everything the tracer learned about your Python is gone from the bytecode; what remains is a call and an unpack.

Worth doing once by hand: the modified bytecode has no guard check in it. Nothing here tests a shape. The guard runs earlier, in C, before this code object is chosen at all, which is why lesson two can print guards and this listing cannot show them.

[0/0] ORIGINAL BYTECODE f simple.py line 3
[0/0]   3           0 RESUME                   0
[0/0]   5           2 LOAD_GLOBAL              1 (NULL + torch)
[0/0]              14 LOAD_ATTR                1 (sin)
[0/0]              24 LOAD_FAST                0 (x)
[0/0]              26 PRECALL                  1
[0/0]              30 CALL                     1
[0/0]              40 STORE_FAST               1 (y)
[0/0]   6          42 LOAD_FAST                1 (y)
[0/0]              44 LOAD_CONST               1 (1)
[0/0]              46 BINARY_OP                0 (+)
[0/0]              50 RETURN_VALUE

[0/0] MODIFIED BYTECODE f simple.py line 3
[0/0]   3           0 RESUME                   0
[0/0]               2 PUSH_NULL
[0/0]               4 LOAD_GLOBAL              4 (__compiled_fn_0)
[0/0]              16 LOAD_FAST                0 (x)
[0/0]              18 PRECALL                  1
[0/0]              22 CALL                     1
[0/0]              32 UNPACK_SEQUENCE          1
[0/0]              36 RETURN_VALUE

Put a print() between the two tensor ops and the same log shows what a graph break actually costs. The rewritten frame calls __compiled_fn_0 for the prefix, stores the result, runs the real print with the real argument, and then calls a global named __resume_at_70_1. The number 70 is the bytecode offset the break happened at.

That resume function is not in your source. ContinueExecutionCache.generate in torch/_dynamo/resume_execution.py synthesizes a code object whose first three instructions are a RESUME, a load of a synthetic local called ___stack0, and a JUMP_FORWARD over the part that already ran. Live variables come in as arguments. The interpreter enters it, jumps past the prefix, and lands on the continuation, which dynamo then compiles as a frame in its own right.

Read the two listings together and the cost model stops being a slogan about fusion. A break means a second code object, a second capture, a second set of guards, and a Python call on the boundary between them, on every single call after that.

[0/0_1] MODIFIED BYTECODE f break.py line 3
[0/0_1]               4 LOAD_GLOBAL              6 (__compiled_fn_0)
[0/0_1]              22 CALL                     1
[0/0_1]              32 STORE_FAST               3 (graph_out_0)
[0/0_1]              36 LOAD_GLOBAL              4 (print)
[0/0_1]              48 LOAD_CONST               1 ('mid')
[0/0_1]              70 CALL                     1
[0/0_1]              84 LOAD_GLOBAL              8 (__resume_at_70_1)
[0/0_1]              98 LOAD_FAST                1 (y)
[0/0_1]             104 CALL                     2
[0/0_1]             114 RETURN_VALUE

[1/0] ORIGINAL BYTECODE resume_in_f break.py line 6
[1/0]   6           0 RESUME                   0
[1/0]               2 LOAD_FAST                0 (___stack0)
[1/0]               4 JUMP_FORWARD            35 (to 76)
[1/0]         >>   76 POP_TOP
[1/0]   7          78 LOAD_FAST                1 (y)
[1/0]              82 BINARY_OP                0 (+)
[1/0]              86 RETURN_VALUE

The bracket in front of every dynamo log line is three numbers and it is the fastest diagnostic in the system. torch/_guards.py defines CompileId as a frame id and a per-frame compile count, printed as 0/1, and TraceId adds an attempt counter that appears as a trailing _1 when analysis had to restart.

So [0/0] is the first frame dynamo saw, compiled for the first time, captured on the first attempt. [0/1] is that same frame being recompiled: a guard missed. [1/0] is a different frame entirely, which for the listing above is the resume function. And [0/0_1] says the first attempt was abandoned, which is what a graph break does: convert_frame.py loops over attempts, catches RestartAnalysis, and gives up only past a hundred restarts.

Two glances at the same log now answer two different questions. Frames climbing means your program is being split. The second number climbing means a guard is missing and the same frame is being compiled again.

torch._dynamo.utils.counters is the instrument the chapter uses and it repays five minutes of reading. unique_graphs is incremented once per compiled graph, next to the call into your backend. calls_captured is incremented by count_calls(self.graph), the number of call nodes in the FX graph that was just built.

The measurement below pins both. A one-op function called five times reports calls_captured 1; a three-op function called five times reports 3. Neither number moves on a call that hits the cache, because both are incremented inside the compile path and a cache hit never enters it. What calls_captured counts is captured ops, summed over every compile.

Use unique_graphs when the question is how many times you paid, which is what the rest of this arc does. The lab LAB·P3 has the side-effect experiment that shows Python around the graph still running on every call, and it uses a plain global counter for it, which is the instrument that actually answers that question.

import torch
import torch._dynamo as dynamo
from torch._dynamo.utils import counters

def run(label, fn, n):
    dynamo.reset(); counters.clear()
    cf = torch.compile(fn, backend="eager")
    for _ in range(n):
        cf(torch.randn(4))
    print(label, dict(counters["stats"]))

run("1 op, 1 call  ", lambda x: x + 1, 1)
run("1 op, 5 calls ", lambda x: x + 1, 5)
run("3 ops, 1 call ", lambda x: torch.sin(x) * 2 + 1, 1)
run("3 ops, 5 calls", lambda x: torch.sin(x) * 2 + 1, 5)
# 1 op, 1 call   {'calls_captured': 1, 'unique_graphs': 1}
# 1 op, 5 calls  {'calls_captured': 1, 'unique_graphs': 1}
# 3 ops, 1 call  {'calls_captured': 3, 'unique_graphs': 1}
# 3 ops, 5 calls {'calls_captured': 3, 'unique_graphs': 1}