Single controller

Pathways inverts the shape chapter 12 built. Instead of one Python process per host, each blindly trusting the others to be in lockstep, there is exactly one Python client process, full stop. Underneath it sit a resource manager and a fleet of per-host workers that execute whatever programs the client hands them, gang-scheduled across however many pods the job spans.

The design is not a product improvement bolted onto McJAX; it is a different paper's answer to the same problem. The Pathways paper describes an asynchronous distributed dataflow system with centralized scheduling, amortized over large operations so that one controller issuing instructions to thousands of chips does not become the bottleneck it sounds like it should be.

The place chapter 11 spent an entire chapter building an abstraction is exactly the place Pathways plugs in. The client speaks ifrt_proxy, a client and server split of the IFRT interface, to a server that fronts the Pathways runtime on the other side. Your JAX code above that boundary does not change at all: the same jax.jit, the same arrays, the same sharding specs from chapter seven, now dispatching across a proxy instead of an in-process client.

One controller, thousands of chips, the same JAX.

That is the direct answer to a question worth asking plainly: how do you swap out the entire runtime underneath a model without touching the model. You do not touch the model. You touch the client IFRT talks to, and IFRT was built with exactly that seam in mind.

Centralizing control is not free, but what it buys is real. MPMD becomes possible: different programs running on different islands of the same job, the shape pipelining needs. The controller also outlives any individual worker, so a worker failing is a recoverable event instead of a gang-wide one, the elasticity chapter 12 explicitly could not offer. And the scale ceiling moves: a single gang-scheduled program is no longer the largest unit of work the system can express, because one controller can orchestrate many gangs across many pods at once.

Here is the part worth stating without hedging: the Pathways runtime itself is not open source. You can read the paper that describes its design, you can read the ifrt_proxy client and server code in the public XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → tree, since that boundary has to be public for anything to plug into it, and you can read the product documentation for the cloud surface built on top of it. What actually executes your program on the other side of that proxy is closed, the same way the kernel path marks libtpu closed rather than pretending otherwise. Treat every claim in this chapter about the runtime's internals as coming from the paper and the product surface, not from code you can read yourself.