TPU vs GPU: what a TPU is, how TPU pods work, and when to use one

What a TPU is

A tensor processing unit is an application-specific integrated circuit that Google built to run neural networks. Google put the first generation into its own data centers in 2015, published the design in 2017, and has rented TPUs through Google Cloud since 2018. Every Gemini model is trained and served on them.

The mechanism is the systolic array. A GPU has thousands of small programmable cores that fetch operands from registers and caches. A TPU's matrix multiply unit is a fixed grid of multiply-accumulate cells: weights are loaded once, activations flow through the grid like a wave, and each value is reused at every cell it passes without another trip to memory. That makes dense matrix multiplication, which is most of what a transformer does, very cheap in energy and silicon area. It also makes anything that is not dense matrix math comparatively awkward.

The chip is only half of the design. Each TPU has its own inter-chip interconnect (ICI) ports, so chips connect directly to their neighbours without going through a network card or a switch. That is what lets Google build pods of thousands of chips that behave like one machine.

The generations that matter in 2026

Google has shipped seven generations and announced an eighth. These are the ones you can rent or plan for, with Google Cloud on-demand list prices per chip-hour from the Cloud TPU pricing page in the cheapest US region as of October 2026.

Ironwood is the one most new workloads will land on. Per Google's TPU7x documentation, each chip delivers 4,614 FP8 TFLOPs (2,307 in BF16), with 192 GiB of HBM at about 7.4 TB/s and 1.2 TB/s of bidirectional ICI bandwidth. Google says it is more than four times faster per chip than Trillium, and made it available to Cloud customers in November 2025.

At Cloud Next in April 2026 Google split the line in two. TPU 8t is for training: a 9,600-chip superpod with 121 FP4 exaflops and 2 PB of HBM. TPU 8i is for inference and reinforcement learning: 1,152-chip pods, 288 GB of HBM per chip and 384 MB of on-chip SRAM to keep mixture-of-experts routing close to the compute, with an 80% better performance per dollar claim over Ironwood. Google says both reach general availability later in 2026, according to DCD's report. I would not plan a 2026 production launch on either until they show up in the pricing page.

Why the pod matters more than the chip

Compare spec sheets chip for chip and Ironwood lands in the same class as Nvidia's Blackwell B200: both carry 192 GB of HBM and a few petaflops of FP8. If that were the whole story, nobody would bother with the software migration. The difference is how many chips can share memory-speed bandwidth.

On the Nvidia side, the fast domain is NVLink. A GB200 NVL72 rack joins 72 GPUs; Vera Rubin NVL72 keeps that size in the second half of 2026, and Nvidia's 144-GPU Kyber rack is a 2027 product. Past the rack, GPUs talk over InfiniBand or Ethernet through network cards and switches, at a fraction of NVLink bandwidth.

On the TPU side, the fast domain is the whole pod. Ironwood arranges 64 chips as a 4x4x4 cube and links 144 cubes through optical circuit switches into a 3D torus of 9,216 chips with 1.77 PB of shared HBM. The optical switches can also reconfigure the topology around failed hardware without recabling. A training job that needs tensor and expert parallelism across hundreds of chips stays on the fast network instead of falling back to the data center fabric.

That is the actual reason Anthropic, which trains on TPUs, Trainium and Nvidia GPUs at the same time, agreed in October 2025 to use up to one million TPUs, and why it then contracted multiple gigawatts more from 2027, roughly 3.5 GW according to Broadcom's filing as reported by Data Center Knowledge.

From pod to cluster

A TPU cluster is several slices or pods working on one job. Inside a slice, chips use ICI. Between slices, Google's Multislice and Pathways software schedules work over the data center network. With TPU 8t, Google says its new Virgo network puts about 134,000 TPUs in a single data center fabric and connects more than a million TPUs across sites into one training cluster. Those numbers are for Google and its largest customers. On Google Cloud you can create Ironwood slices from 4 chips up to 2,048 chips yourself; anything larger goes through your account team.

TPU vs GPU, side by side

CPUs and NPUs sit at the ends of the same spectrum. A CPU is the general-purpose processor that orchestrates everything else. An NPU is a small neural accelerator inside a phone or laptop chip, built for on-device inference at a few watts. Neither competes with a TPU pod for training.

The software question decides it

TPUs do not run CUDA. Your model reaches the chip through the XLA compiler, from JAX, from PyTorch through PyTorch/XLA, or from vLLM for serving. Google announced native PyTorch support, TorchTPU with eager mode, at Cloud Next 2026, but it is still in preview with select customers.

In practice that splits teams into two groups. If you train or serve standard transformer architectures in JAX or plain PyTorch, the port is mostly configuration and sharding annotations. If your performance depends on hand-written CUDA kernels, FlashAttention variants you patched yourself, or a long tail of CUDA-only libraries, the port is an engineering project with its own timeline. That project, not the hourly rate, is usually the larger number in the business case.

Cost: compare per token, not per chip

Ironwood's $12.00 on-demand rate per chip-hour drops to $8.40 on a one-year commitment and $5.40 on three years. Those numbers mean little next to a GPU-hour price because the chips do different amounts of useful work on your model. The comparison that holds up is cost per training step at your target batch size, or cost per million output tokens at your latency target, measured on a short paid benchmark. For GPU rates outside the hyperscalers, see neoclouds compared.

The other custom chips

Google is not alone. AWS Trainium3, generally available since December 2025, is a 3 nm chip with 144 GB of HBM3e, sold in UltraServers of up to 144 chips, and AWS says Trainium4 will support Nvidia's NVLink Fusion so it can sit next to GPUs in the same rack. Anthropic's Project Rainier runs on about 500,000 Trainium2 chips. Microsoft Maia 200 and Meta MTIA serve their owners' internal models. Independent vendors such as Cerebras and Groq take a different approach entirely, keeping model weights in on-chip SRAM to sell very fast inference. The vendor-by-vendor analysis is on our partner site We The Flywheel, linked below.

When I would use TPUs

1. You are already on Google Cloud and your data, identity and compliance perimeter live there.
2. Your models are standard architectures in JAX or PyTorch, without a deep dependency on custom CUDA kernels.
3. The job is large and steady: a training run across hundreds of chips, or high-volume serving where a committed-use discount pays off.
4. You want a second supply line so a single GPU allocation decision cannot stall your roadmap.

I would stay on GPUs when the workload is small or experimental, when the team's speed depends on the CUDA ecosystem, or when the model has to run on premises or across several clouds. The middle path most companies end up on is the one Anthropic runs at much larger scale: keep GPUs as the default, and move the one or two workloads that dominate the compute bill to whichever custom chip benchmarks cheapest for them.

Questions to ask before you move a workload

1. Which TPU generation, in which region, and is the capacity on-demand or behind a reservation?
2. What slice size do I get without an account-team exception?
3. What does our model's cost per step or per million tokens look like on a one-week paid benchmark?
4. Which of our kernels and libraries have no XLA path, and who owns porting them?
5. If we commit for three years, what is the exit if TPU 8 or the next GPU generation changes the economics?

In this cluster

• AI data centers — the hub: power, cooling, equipment and capital
• Neoclouds compared — where to rent Nvidia GPUs outside the hyperscalers
• The largest AI data centers — including Anthropic's Trainium-based Project Rainier
• Liquid cooling for AI data centers — why Ironwood, Trainium3 and GB200 racks all need it

On our partner site

• AI chip companies 2026 — the vendor analysis of Google TPU, AWS Trainium, AMD, Cerebras, Groq and the rest, on our partner site We The Flywheel
• Google TPU vs AWS Trainium vs Nvidia GPU — the scored head-to-head, on We The Flywheel

Frequently asked questions

Sources: Google Cloud TPU7x documentation and Cloud TPU pricing (accessed 31 October 2026); Google blog, Ironwood (25 November 2025); Google Cloud, AI infrastructure at Next '26 (April 2026); DatacenterDynamics; AWS Trainium product page; CNBC (23 October 2025); Data Center Knowledge (8 April 2026). Vendor performance figures are as stated by the vendors.