Tensors

Take a tensor, call .t() on it, and nothing gets copied. What you're holding afterward still points at the same block of memory the original tensor pointed at; only the map describing how to walk that memory changed. A PyTorch tensor is that pair made explicit: a storage, one flat run of numbers, and metadata, a shape, a stride, and an offset, that says how to read positions out of it. Slicing, t(), permute, expand: none of them touch the storage. Each one just builds a new tensor object with new metadata pointed at the same bytes.

The stride is the part that does the actual reading. It says how many elements to skip in storage for each step along a given axis, so a (3, 4) tensor with strides (4, 1) means: move one whole row, four elements, to advance the first index, move one element to advance the second. Transpose it and the strides swap without a single number in storage moving; the shape says (4, 3) now, but you are reading the exact same numbers through a different map.

A tensor is a view of a storage, and the strides are the map.

import torch

x = torch.arange(12.).reshape(3, 4)
v = x.t()                    # a view: same storage, remapped
print(v.stride(), v.is_contiguous())   # (1, 4) False
c = v.contiguous()           # a copy that owns a row-major layout
print(c.stride())            # (4, 1)

row = x[0]                   # also a view
row += 100                   # in-place: x changes too
print(x[0])                  # tensor([100., 101., 102., 103.])

Call .contiguous() on that transposed view and something different happens: PyTorch allocates a fresh block of storage, walks the view in row-major order, and writes those values out linearly. The result is a new tensor with new storage, strides that read (4, 1) for this shape, the same layout you'd get building the tensor fresh. That's the boundary worth holding in your head: everything before .contiguous() reads the same bytes through different maps; .contiguous() is the one operation here that pays for an actual copy.

This is also why expand can be free while reshape sometimes can't be. expand works by setting a dimension's stride to zero, repeating one row or column without writing any new bytes, so it never needs the tensor to be contiguous first. reshape wants to reinterpret existing strides as a new shape, and when the current strides don't support that reinterpretation, there's no map that satisfies the request. reshape falls back to copying, quietly, exactly where a plain view would have refused outright.

That refusal is deliberate, and it's worth knowing which call you're making. .view() either finds a stride pattern that reinterprets the existing storage, or it raises. .reshape() promises the shape you asked for, however it has to get there, copy included.

Slice a row out of x and mutate it in place, and x changes too. row = x[0] is a view, so row += 100 isn't writing to some private copy of the first row; it's writing into the exact storage x already owns. Print x[0] afterward and the new values are sitting right there, because there was never a second copy to keep them separate.

That's a feature, not an accident. Views exist so that slicing, transposing, and reshaping stay cheap, and an in-place update through a view is how PyTorch lets you edit a tensor's data without allocating a new one. The cost shows up later: the next chapter's autograd tape assumes a tensor's value hasn't shifted under it once an operation has recorded it, and an in-place write through a view is exactly the kind of edit that can pull the floor out from under a gradient computation that hasn't run yet.

Every tensor carries its own dtype and its own device, and neither one is shared or inherited from context. Build a tensor from Python floats with no dtype specified and you get float32; that's the default floating type PyTorch reaches for, and it stays the working assumption unless you ask for float64 or one of the lower-precision types explicitly.

Try to add a CPU tensor to a CUDA tensor and PyTorch does not move one of them for you. It refuses, naming the mismatch, rather than guessing which device you meant and copying silently. An operation that insists you say .to(device) yourself is an operation that never lets a slow transfer over a real bus hide inside an innocent-looking +.

Everything in this chapter reduces to one skill: given a chain of view operations, read off the resulting shape, stride, and whether the result is contiguous, without running anything. The stride oracle at /gym/pytorch#drills drills exactly that, feeding you a chain of code and asking for the three numbers before you're allowed to check. It's a narrow skill, and it's the one that makes every stack trace involving contiguous() or a view() refusal legible on sight instead of mysterious.