Two objects, one tensor

Ask a tensor for its storage and what comes back has seven fields and no opinions. StorageImpl holds a data pointer, a size in bytes, three booleans, an allocator, and a slot for the Python object wrapping it. No shape. No stride. No dtype. A storage is a run of bytes with a length and a way to free itself.

The comment above the class states the invariant the rest of the library leans on: two non-null data pointers alias if and only if they came from the same storage. Aliasing is decidable because of that sentence. Instead of comparing address ranges operation by operation, torch compares storage identity.

The same comment names where mutation tracking lives, and it is not on the tensor. Version counting happens at the level of storages, which is why an in-place write through any view is visible to every other view of the same bytes. The museum's version-counter exhibit is where that turns into an error message; the fact to carry out of here is only which object owns the counter.

// c10/core/StorageImpl.h:11-21
// A storage represents the underlying backing data buffer for a
// tensor.  This concept was inherited from the original Torch7
// codebase; we'd kind of like to get rid of the concept
// (see https://github.com/pytorch/pytorch/issues/14797) but
// it's hard work and no one has gotten around to doing it.
//
// NB: storage is supposed to uniquely own a data pointer; e.g.,
// two non-null data pointers alias if and only if they are from
// the same storage.  Technically you can violate this invariant
// (e.g., you can create a non-owning StorageImpl with at::from_blob)
// but a lot of things won't work correctly, including:

// c10/core/StorageImpl.h:31-33, the third of those consequences
// - Version counts won't work correctly, because we do all VC tracking at the
//   level of storages (unless you explicitly disconnect the VC with detach);
//   mutation because data pointers are the same are totally untracked

// c10/core/StorageImpl.h:214-224, every field the class has
 private:
  DataPtr data_ptr_;
  SymInt size_bytes_;
  bool size_bytes_is_heap_allocated_;
  bool resizable_;
  // Identifies that Storage was received from another process and doesn't have
  // local to process cuda memory allocation
  bool received_cuda_;
  Allocator* allocator_;
  impl::PyObjectSlot pyobj_slot_;
};

TensorImpl is what a Python Tensor actually points at, and it owns one Storage by value. Everything the storage refused to know sits here: a packed sizes-and-strides container, an integer storage offset, a cached element count, a TypeMeta for the dtype, an optional device, and a dispatch key set that decides which kernel table a call lands in.

Two of those are caches rather than truth, and knowing that changes how you read a profile. numel_ is stored, not derived, so asking a tensor how many elements it has costs a load and not a product over the shape. The contiguity flags work the same way, and they get the last lesson of this arc to themselves.

One line in the header is worth reading twice. Above data_type_ the comment says the type meta must agree with the type meta in storage. Go back to the field list on StorageImpl and there is no type meta on it at all, in this version. The invariant outlived the field it was written about, which is a normal thing to find in a header this old and a good reason to check the fields rather than the comments.

In Python the typed view of a storage is on its way out too. Calling .storage() on torch 2.2.2 warns that TypedStorage is deprecated and points at untyped_storage(), which reports bytes and nothing else. Every storage number in this arc comes from that call.

// c10/core/TensorImpl.h:2803-2804
 protected:
  Storage storage_;

// c10/core/TensorImpl.h:2839-2850
  c10::impl::SizesAndStrides sizes_and_strides_;

  int64_t storage_offset_ = 0;
  // If sizes and strides are empty, the numel is 1!!  However, most of the
  // time, we will immediately set sizes to {0} and reset numel to 0.
  // (Can't do that in the default initializers, because there's no way to
  // spell "allocate a one-element array" for strides_).
  int64_t numel_ = 1;

  // INVARIANT: When storage is non-null, this type meta must
  // agree with the type meta in storage
  caffe2::TypeMeta data_type_;

Slice three corners off a tensor and the arithmetic connecting the two objects fits in one line. The address of a tensor's first element is the storage's data pointer plus the storage offset times the element size, and the slice below sits 17 elements into a 24-element storage, which on float32 is 68 bytes along.

The second printed line is the part people find surprising. The tensor reports 6 elements while its storage still reports 24, because slicing narrowed the map and freed nothing. A view keeps the entire allocation alive, so a small crop of a large tensor holds the large tensor's memory until both go away.

A view narrows the map. It never narrows the allocation.

import torch

base = torch.arange(24.)
x = base.reshape(2, 3, 4)
s = x[1:, 1:, 1:]

print(s.shape, s.stride(), s.storage_offset())
print(s.numel(), s.untyped_storage().nbytes() // s.element_size())
print(s.data_ptr() - x.data_ptr())
print(x.untyped_storage().data_ptr() == s.untyped_storage().data_ptr())

# torch.Size([1, 2, 3]) (12, 4, 1) 17
# 6 24
# 68
# True

Five is not an arbitrary number anywhere in this code. C10_SIZES_AND_STRIDES_MAX_INLINE_SIZE is 5, so a tensor of five axes or fewer carries its entire shape and stride tuple inside the TensorImpl object. Add a sixth axis and the container allocates out of line and keeps a pointer instead.

A comment near the bottom of the header does the arithmetic that explains why anyone cared. With 400 million live tensors in a training run, every 64-bit word added to a tensor costs another 3.2 gigabytes of RAM, and the comment records that they once ran out of memory at 160 bytes per TensorImpl.

The budget is enforced, not hoped for. A dummy class exists only to run static_asserts across the field sizes, and on a 64-bit build the whole object has to fit in 26 words, with the packed sizes and strides taking exactly 88 of those 208 bytes. Metadata is a memory cost, and somebody is watching it at compile time.

// Struct size matters.  In some production systems at Facebook, we have
// 400M live tensors during a training run.  Do the math: every 64-bit
// word you add to Tensor is an extra 3.2 gigabytes in RAM.
//
// If you are a Facebook employee, you can check if the run in question
// has tipped you over the point using the command here:
// https://fburl.com/q5enpv98
//
// For reference, we OOMed at 160 bytes (20 words) per TensorImpl.
// This is not counting overhead from strides out-of-line allocation and
// StorageImpl space and this is from before we inlined sizes and strides
// directly into TensorImpl as SmallVectors.

  // This is a 64-bit system
  static constexpr bool check_sizes() {
    constexpr size_t tsize = 26 * sizeof(int64_t);
    are_equal<sizeof(sizes_and_strides_), 88,  FieldNameEnum::sizes_and_strides_>();
    is_le<sizeof(TensorImpl),          tsize,  FieldNameEnum::TOTAL_SIZE>();

Since the dtype lives on the tensor and the storage is only bytes, you can point a second tensor at the same bytes and read them as something else. x.view(torch.int32) does exactly that, and the address does not move.

What prints is the float32 bit pattern shown as an integer. A 1.0 in single precision is 0x3F800000, which is 1065353216 in decimal, and there it is in the second slot. Nothing was converted and nothing was copied. One TypeMeta changed, and the same four bytes answered a different question.

This is also the cleanest way to see why the dtype could not have lived on the storage. Two tensors over one storage with two different dtypes is a legal thing to build, so the field has to sit on the object there can be many of.

import torch

x = torch.arange(12.).reshape(3, 4)
i = x.view(torch.int32)
print(i.dtype, i.data_ptr() == x.data_ptr())
print(i[0].tolist())

# torch.int32 True
# [0, 1065353216, 1073741824, 1077936128]

Chain three view operations and you might expect three links back up the chain. There is only one. _base on any view in the chain points at the root tensor that owns the storage, not at the immediate parent, so a slice of a transpose of a reshape reports the original 1-D arange as its base.

The last line is the detector this arc uses instead of comparing addresses. A tensor that owns fresh storage has _base set to None, so _base is None is a yes-or-no answer to whether a call copied. Lesson three leans on it hard, because reshape will not tell you which road it took.

import torch

base = torch.arange(12.)
x = base.reshape(3, 4)
v = x.t()
s = v[1:]
print(x._base is base, v._base is base, s._base is base)
print(s._base is v, base._base is None)
print(v.contiguous()._base is None)

# True True True
# False True
# True