The address arithmetic

Every read a strided tensor performs is the same expression. Start at the storage offset, then for each axis add the index along that axis times that axis's stride. That position is an element index into the storage, and multiplying by the element size turns it into a byte address.

The chain below is one of the corpus rows, and the second printed line does the lookup twice. Once through the tensor's own indexing, and once by evaluating the expression against the flat arange the view was built from. Same number, because indexing is arithmetic and nothing more than arithmetic.

Nothing about that expression cares whether the strides are increasing, decreasing or repeated. It also does not care whether two different index tuples produce the same position, which is where the last section of this lesson goes.

import torch

base = torch.arange(64.)
y = base.reshape(2, 4, 8).transpose(0, 2)
print(y.shape, y.stride(), y.storage_offset())

i, j, k = 5, 2, 1
print(y[i, j, k].item(), base[0 + i * 1 + j * 8 + k * 32].item())

# torch.Size([8, 4, 2]) (1, 8, 32) 0
# 53.0 53.0

Transposing swaps two entries in both tuples, and the chapter above says so. The four operations below are the ones whose arithmetic is easy to get wrong, and each of them is one line of bookkeeping over shape, stride and offset.

Slicing rows is the only one that moves the offset. x[k:] keeps every stride, subtracts k from the first size, and pushes the offset forward by k times the first stride. A strided slice like x[:, 1::2] does both jobs at once: it multiplies the stepped axis's stride by the step, ceiling-divides that axis's size, and moves the offset by the start times the original stride.

unsqueeze is the one that looks like it should be free of arithmetic and is not. The inserted axis has size 1, so its stride is never stepped and could hold anything, but torch fills it with the stride of the axis it displaced times that axis's size. That is why a size-1 axis often carries a stride larger than any of its neighbours, and why a chain with an unsqueeze in it produces stride tuples that look wrong at first glance.

The six rows below are quoted from the corpus behind the stride oracle, which is 48 chains executed on torch 2.2.2 and stored with what torch printed. All 48 replayed identically on this machine on 14 August 2026, so these are measurements rather than derivations.

Work the third row by hand and the arithmetic from this lesson does all of it. Start at (2, 3, 4) with strides (12, 4, 1). The transpose swaps the outer two entries of both tuples, giving shape (4, 3, 2) and stride (1, 4, 12). Then [:, ::2] halves the middle size to 2 and doubles its stride to 8, which is exactly the (4, 2, 2) and (1, 8, 12) in the table.

The fifth row is worth working through twice. Two strided slices in a row leave a size-1 axis whose stride is 12, the same as the axis above it, because the second slice doubled a stride that no longer has anything to step over. A size-1 axis's stride is arbitrary in the sense that nothing reads it, and specific in the sense that torch still computes and stores a number there.

Set an axis's stride to zero and the multiply-add stops advancing along it. Every index along that axis lands on the same position, which is how broadcasting gets to be free: expand writes a zero and changes nothing else.

Two numbers in the run below refuse to fit together at first. The tensor claims 15 elements, and the storage under it holds 3. There is no rule saying a tensor's element count has to fit inside its storage, only that every position the map produces has to land inside it.

import torch

e = torch.arange(3.).unsqueeze(1).expand(3, 5)
print(e.stride(), e.numel(), e.untyped_storage().nbytes() // e.element_size())
print(torch._debug_has_internal_overlap(e))
try:
    e.add_(1)
except RuntimeError as err:
    print(err)

# (1, 0) 15 3
# 1
# unsupported operation: more than one element of the written-to tensor refers
# to a single memory location. Please clone() the tensor before performing the
# operation.

A tensor whose map sends two different index tuples to one position overlaps itself. Reading such a tensor is fine, because a read that returns the same number twice is still correct. Writing to it is not, because the result depends on the order the kernel happens to visit elements in.

torch has a check for this, and the check has three answers rather than two. MemOverlap is No, Yes, or TooHard, and the third one is the one to know about. An expanded tensor answers Yes and in-place ops on it raise. A tensor built by as_strided into a sliding window answers TooHard, so nothing raises, and the write goes ahead and produces a result that depends on the visiting order.

The four-by-three window below reads six values through a stride of one on both axes. Adding one in place should leave [1, 2, 3, 4, 5, 6] if every element were written once. What comes back is [1, 3, 5, 6, 6, 6], because the middle positions were visited three times each. No error, no warning, and a wrong answer that looks plausible.

No and Yes are the easy answers. TooHard means the check gave up and your write went through anyway.

// MemOverlap: Whether or not there is memory overlap
//
// No: Absolutely no memory overlap
// Yes: Absolutely yes memory overlap
// TooHard: There might be memory overlap, but it was too expensive to compute.
//
// NB: Please update the python test for these if you renumber them.
enum class MemOverlap { No, Yes, TooHard };

// >>> b = torch.arange(6.)
// >>> w = torch.as_strided(b, (4, 3), (1, 1))
// >>> torch._debug_has_internal_overlap(w)
// 2
// >>> w.add_(1)
// >>> b.tolist()
// [1.0, 3.0, 5.0, 6.0, 6.0, 6.0]

One shape of map torch will not build is a descending one. as_strided with a negative stride raises, and the message says Negative strides are not supported at the moment, which is the same reason torch.flip allocates instead of returning a view. Flip a tensor and check its stride and it reads like a fresh contiguous tensor, because it is one.

NumPy does allow it, and a reversed slice there is a view with a negative stride: np.arange(5)[::-1] reports strides of -8 and owns no data of its own, on numpy 2.2.6. So a habit carried over from NumPy costs a full copy here, every time you reverse.