Who owns .grad

The chapter defines a leaf as a tensor you made rather than one an operation returned. The runtime definition is narrower and easier to check: is_leaf is true when grad_fn is None. Anything an op produced under a recording context carries a node, so it is not a leaf, so no accumulator was ever built for it, so there is nothing to write into.

Reading .grad on such a tensor returns None and prints a warning first, and the warning is unusually good. It says the attribute will not be populated, names retain_grad() as the way to change that, and then guesses out loud that you probably meant to read the leaf instead. Half of PyTorch's autograd errors read like this: a statement of what happened, followed by the two things you might have meant.

The warning fires on the read, not on the backward pass, so it can appear long before or long after the pass that would have filled the attribute. That timing is worth knowing when the warning shows up in a log with nothing around it.

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being
accessed. Its .grad attribute won't be populated during autograd.backward(). If
you indeed want the .grad field to be populated for a non-leaf Tensor, use
.retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by
mistake, make sure you access the leaf Tensor instead. See
github.com/pytorch/pytorch/pull/30531 for more informations.

Call retain_grad() on a non-leaf and the tensor keeps its node, keeps its place in the graph, and gains somewhere for a gradient to land on the way past. Nothing about the walk changes; one extra hook copies the gradient into .grad as it flows through.

The two numbers below are the point of doing it. With h = x * 2 and a loss of (h * h).sum(), the gradient at h is 4 and the gradient at x is 8, because the chain rule multiplies by the 2 that h was built with. Reading only x.grad you would have to divide to recover the middle; reading both, the factor is right there.

The cost is a live tensor per retained intermediate, held until you clear it. That makes retain_grad a debugging instrument rather than something to leave switched on, which is the same trade the next lesson's saved-tensor accounting makes explicit in bytes.

import torch

x = torch.ones(3, requires_grad=True)
h = x * 2
h.retain_grad()
(h * h).sum().backward()
print(h.is_leaf, h.grad_fn.name())   # False MulBackward0
print(h.grad)                        # tensor([4., 4., 4.])
print(x.grad)                        # tensor([8., 8., 8.])

Use one leaf twice in a forward pass and the natural picture is two gradients arriving at the accumulator and being added there, one after the other. That picture is wrong, and it is easy to falsify: hang a hook on the AccumulateGrad node and count how many times it runs.

It runs once. For ((x * 2) + (x * 3)).sum() the accumulator is called a single time and the tensor it receives is already 5, which is the sum of the 2 and the 3 that arrived down the two paths. The addition happened in a buffer the engine keeps per node while it waits for every incoming edge to report, not at the leaf.

So the word accumulate covers two different mechanisms with different scopes. Within one backward pass, contributions are summed in that buffer and delivered once. Across backward passes, the accumulator adds into whatever .grad already held, which is the behavior zero_grad exists to undo and the one the chapter's training loop is built around.

import torch

got = []
x = torch.ones(3, requires_grad=True)
y = ((x * 2) + (x * 3)).sum()
add = y.grad_fn.next_functions[0][0]
acc = add.next_functions[0][0].next_functions[0][0]
acc.register_hook(lambda grad_in, grad_out: got.append(grad_out[0].clone()))
y.backward()
print(len(got), got[0])   # 1 tensor([5., 5., 5.])
print(x.grad)             # tensor([5., 5., 5.])

Across steps, the accumulator adds in place into the tensor .grad already points at. Hold a reference to p.grad after one backward pass and it reads 4 after the next one, without you touching it, because it is the same object.

Then opt.zero_grad() runs and the reference stops tracking. set_to_none defaults to True here, and has since PyTorch 2.0, so the call assigns None to .grad rather than filling the existing tensor with zeros, and the tensor you were holding is left behind at its old value. Fewer kernel launches and one less live buffer per parameter, at the price of a name that describes what the method used to do.

Two consequences follow for anyone reading gradients between steps. A logger that captured p.grad once and reuses the handle is reading a tensor the optimizer abandoned. And p.grad is None rather than zero before the first backward pass of every step, so code that assumes a zero tensor is there needs set_to_none=False or a None check.

import torch

p = torch.ones(3, requires_grad=True)
(p * p).sum().backward()
held = p.grad
(p * p).sum().backward()
print(held, p.grad is held)       # tensor([4., 4., 4.]) True

opt = torch.optim.SGD([p], lr=0.0)
opt.zero_grad()                   # set_to_none=True is the default
print(p.grad, held)               # None tensor([4., 4., 4.])

torch.autograd.grad walks the same graph with the same engine and never reaches an AccumulateGrad node. It takes the outputs, the inputs you want gradients for, and returns a tuple. Nothing is written anywhere; .grad stays exactly as it was, None included.

That makes it the right call for anything that is not a training step. Second derivatives, influence functions, a gradient norm you want to inspect, a per-sample gradient you are about to reduce yourself: none of those want a side effect on the parameters, and all of them are one line with grad and three lines with backward plus a save and a restore.

It is also the call lesson four leans on, because gradcheck is built on it. A checker that accumulated into .grad while checking would corrupt the state of the model it was called on.

import torch

z = torch.ones(3, requires_grad=True)
print(torch.autograd.grad((z * z).sum(), z))   # (tensor([2., 2., 2.]),)
print(z.grad)                                  # None

(z * z).sum().backward()
print(z.grad)                                  # tensor([2., 2., 2.])