Two layouts, one shape

Convert a four-axis tensor to channels-last and nothing about it changes logically. Same shape, same values at the same indices, and torch.equal says so. What changed is the stride tuple, from (60, 20, 5, 1) to (60, 1, 15, 3), and the bytes underneath got rewritten to match.

Read the raw storage in both and the difference is plain. The contiguous tensor stores 0, 1, 2, 3, 4, 5, which is one row of one channel. The channels-last tensor stores 0, 20, 40, 1, 21, 41, which is all three channels of one pixel, then all three channels of the next.

Which of those a kernel wants depends on what it loops over innermost. A convolution reading all channels at one pixel gets contiguous memory in the second layout and a strided gather in the first, which is what the format is for.

import torch

t = torch.arange(120.).reshape(2, 3, 4, 5)
tc = t.to(memory_format=torch.channels_last)

print(t.stride(), tc.stride())
print(torch.equal(t, tc), tc.data_ptr() == t.data_ptr())
print(tc.is_contiguous(), tc.is_contiguous(memory_format=torch.channels_last))
print(torch.as_strided(t, (6,), (1,)).tolist())
print(torch.as_strided(tc, (6,), (1,)).tolist())

# (60, 20, 5, 1) (60, 1, 15, 3)
# True False
# False True
# [0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
# [0.0, 20.0, 40.0, 1.0, 21.0, 41.0]

is_contiguous() does not compute anything. It reads a bit that was computed the last time the tensor's metadata changed, and there are five of them sitting in one bitfield: plain contiguous, channels-last 2d, channels-last 2d contiguous, and the two 3d equivalents. A sixth bit alongside them records whether the tensor is non-overlapping and dense.

The function behind the first bit is nineteen lines and worth knowing by heart. Walk the axes from the last to the first, carrying a running product that starts at 1. Skip any axis of size 1 entirely. For every other axis, the stride has to equal the running product, and then the product absorbs that axis's size.

The skip is the whole reason a shape with a 1 in it can carry a strange stride and still call itself contiguous. Nothing ever steps along an axis of size 1, so its stride is unconstrained, and the corpus row with shape (1, 1, 3, 4) and strides (24, 24, 4, 1) reports True for exactly that reason.

template <typename T>
bool _compute_contiguous(ArrayRef<T> sizes, ArrayRef<T> strides, T numel) {
  bool is_contiguous = true;
  if (numel == 0)
    return is_contiguous;
  T z = 1;
  // NB: make sure we do signed arithmetic
  for (int64_t d = int64_t(sizes.size()) - 1; d >= 0; d--) {
    const auto& size_d = sizes[d];
    if (size_d != 1) {
      if (strides[d] == z) {
        z *= size_d;
      } else {
        is_contiguous = false;
        break;
      }
    }
  }
  return is_contiguous;
}

The channels-last test is the contiguity test with the axis order hard-coded. Instead of walking axes 3, 2, 1, 0, it walks 1, 3, 2, 0: channels innermost, then width, then height, then batch. Same running product, same size-1 skip, same early exit.

The switch around it says something the docs do not. Only rank 4 is handled, with a TODO next to rank 3 saying it will be enabled once it is fully tested, and everything else returns false. So is_contiguous(memory_format=torch.channels_last) on a two-axis tensor is not an error and not a meaningful answer. It is False because the function had no case for it.

The 3d version is the same code with the order 1, 4, 3, 2, 0, and it handles rank 5 only. Two formats, two hard-coded orders, one algorithm.

template <typename T>
bool _compute_channels_last_contiguous_2d(
    ArrayRef<T> sizes,
    ArrayRef<T> strides) {
  // Please don't combine these code, constant array is used here to let
  // compiler fully unroll the loop to get better performance
  switch (sizes.size()) {
    case 4: {
      T expected = 1;
      for (auto& d : {1, 3, 2, 0}) {
        const auto& size_d = sizes[d];
        if (size_d != 1) {
          if (strides[d] != expected) {
            return false;
          }
          expected *= size_d;
        }
      }
      return true;
    }
      // NOLINTNEXTLINE(bugprone-branch-clone)
    case 3:
      // TODO dim == 3 case will be enabled once it is fully tested
      return false;
    default:
      return false;
  }
}

Give a tensor a single channel and both tests skip the same axis, which means both can pass. A (2, 1, 4, 5) tensor converted to channels-last comes back with strides (20, 1, 5, 1) and answers True to both is_contiguous() and is_contiguous(memory_format=torch.channels_last).

This is the ambiguity that makes layout bugs hard to reproduce. A test written with one channel, or a batch of one, or a one-by-one spatial size, will pass under either layout and prove nothing about which one your kernel receives. Give the test at least two of everything before you trust what it says about layout.

import torch

a = torch.arange(40.).reshape(2, 1, 4, 5).to(memory_format=torch.channels_last)
print(a.stride(), a.is_contiguous(), a.is_contiguous(memory_format=torch.channels_last))
print(torch.arange(12.).reshape(3, 4).is_contiguous(memory_format=torch.channels_last))

# (20, 1, 5, 1) True True
# False

The header opens by denying the thing its name suggests. A memory format is not a property of a tensor; it is a way to tell an operator how to organize its result. What a tensor carries is strides, and the five bits summarizing them. torch.channels_last is an argument you pass, and Preserve is the option that says to follow the inputs.

Follow that through an elementwise op and the consequence is a result whose layout depends on argument order. tc + t comes back with the channels-last strides and t + tc comes back row-major, with identical values either way, because the iterator picks a traversal order from its inputs, and the two calls hand it the same two tensors in the opposite order.

The header for that iterator says why in one line: reorder_dimensions() reorders dimensions to improve coalescing. It is picking the traversal order that makes the inner loop contiguous, and the output gets built to match. Which means a layout you established at the top of a model can survive a long way down it, and a single argument swap can drop it.

A layout is not something a tensor is. It is the arrangement an operator was asked to produce, remembered as strides.

// Memory format is not the property of a Tensor. It is the way to tell an
// operator how the result should be organized in memory and nothing more. That
// means memory format should never be used as return value for any tensor state
// interrogation functions (internally and externally).
//
// Possible options are:
//  Preserve:
//    If any of the input tensors is in channels_last format, operator output
//    should be in channels_last format
//
//  Contiguous:
//    Regardless of input tensors format, the output should be contiguous
//    Tensor.
//
//  ChannelsLast:
//    Regardless of input tensors format, the output should be in channels_last
//    format.

// >>> (tc + t).stride()
// (60, 1, 15, 3)
// >>> (t + tc).stride()
// (60, 20, 5, 1)
// >>> torch.equal(t + tc, tc + t)
// True