AI technology explained: the six-layer stack from chips to agents

AI technology is not one product you can buy. It’s a stack of six layers, and each layer has its own vendors, its own unit of measurement and its own bottleneck. From the bottom up: the silicon, the data centre that houses it, and the training run. Then the serving system that answers a request, the retrieval layer that feeds a model your own data, and the agent layer that lets a model call tools and other agents. Every AI product you have heard of sits somewhere on that list, and most of the confusion in the category comes from people comparing things that live on different floors.

What follows walks the stack from chips to agents, using figures from pages we opened. Those are NVIDIA’s rack specification, Google’s TPU documentation, Epoch AI’s trend dashboard, two vendor price lists, three papers and two protocol specifications. Where a number is a vendor’s own benchmark rather than an independent one, we say so. If you want the words rather than the architecture, our guide to AI terminology covers the vocabulary and this piece covers the plumbing.

Layer	What it does	Unit you measure it in	Named example
Silicon	Matrix multiplication	FLOPs, HBM GB, GB/s	Blackwell GPU, TPU v6e
Interconnect and site	Makes many chips act as one	TB/s, megawatts	NVLink domain, gigawatt campus
Training	Turns compute and data into weights	Total FLOP, dollars	A frontier pre-training run
Serving	Answers requests at low latency	Tokens per second, $/MTok	vLLM, a hosted API
Context and retrieval	Puts your data in front of the model	Recall, tokens of context	pgvector, RAG pipelines
Agents	Lets a model act and delegate	Task success, time horizon	MCP servers, A2A agents

The unit of compute is a rack, not a chip

Ask what an AI chip is and the honest answer is that the question has moved up a level. NVIDIA’s GB200 NVL72 connects 36 Grace CPUs and 72 Blackwell GPUs in one liquid-cooled rack, and NVIDIA’s pitch is that the 72-GPU NVLink domain behaves as a single massive GPU. The NVLink Switch System carries 130 terabytes per second of GPU-to-GPU traffic inside that rack, with fifth-generation NVLink giving 1.8 TB/s of GPU-to-GPU interconnect.

Google sells the same idea in a different shape, which matters because the two designs are not interchangeable. Its documentation for TPU v6e, the generation it calls Trillium, gives each chip 918 bf16 TFLOPs, 32 GB of HBM and 1,638 GB/s of memory bandwidth. Those chips are wired into a pod of 256 on a 2D torus. That pod delivers 234.9 bf16 PFLOPs, which is roughly what you get multiplying 918 TFLOPs by 256, so the interconnect is doing its job rather than quietly losing you throughput.

The reason both vendors talk in racks and pods is that memory is the part that hasn’t kept up. Epoch AI’s trends dashboard, updated 5 February 2026, puts GPU memory bandwidth growth at 28% a year since 2008, doubling roughly every 2.8 years. Compute per dollar is moving far faster, at 49% a year since 2023. So the gap between what a chip can calculate and what it can feed itself widens every generation, and the fix is to bolt more chips together with very fat pipes.

The catch is that the headline speedups are vendor arithmetic. NVIDIA’s claim of 30x faster real-time inference against an H100 baseline is footnoted to one configuration. That footnote names a 50 millisecond time-to-last-token target, 32,768 input tokens, 1,024 output tokens, and HGX H100 systems scaled over InfiniBand. That’s the vendor’s own test, on a workload it chose. So it tells you the direction, not the number you should budget against.

Training compute grows 5x a year and efficiency claws back 3x

The training layer is where the money visibly burns. Epoch AI reports that training compute for frontier language models has grown 5x a year since 2020, doubling every 5.2 months, with a 90% confidence interval of 4x to 6x. Across the longer run since 2010, compute for notable models rose about 4.5x a year. The cost of those runs has climbed 3.5x a year since 2020, which is slower than the compute growth, because chips keep getting cheaper per unit of work.

But software pulls in the other direction. Epoch puts pre-training compute efficiency gains at roughly 3.0x a year, meaning the same capability needs about a third of the compute twelve months later. That also means any dollar figure attached to training a given capability decays fast, so quoting last year’s number as this year’s cost is one of the most common errors in AI coverage. Epoch’s own tracker puts the resulting lag at about eight months before frontier performance becomes reachable on consumer hardware.

Training compute rises about 5x a year while the compute needed for a fixed capability falls about 3x a year. Almost every argument about AI progress is really an argument about which of those two lines you are watching.

Growth rates from Epoch AI’s trends dashboard, 5 February 2026

That matters because both of those trends still land on concrete and copper. Epoch estimates the largest known AI data centre, SpaceXAI’s Colossus 2 in Memphis, at 1.1 million H100-equivalents of capacity, and expects Meta’s Hyperion to reach about 3.7 million H100-equivalents by January 2028. A site with one gigawatt of IT power costs roughly $38 billion in up-front capital, and takes about two years to build. Five hyperscalers own 71% of global AI compute, so the bottom two layers of this stack are the most concentrated part of it.

Serving is a memory problem wearing a compute costume

Once the weights exist, the expensive question is how many people you can answer at once, so the serving layer is where throughput is won. The bottleneck is the key-value cache, the running state a transformer keeps for every token in every active request. The PagedAttention paper presented at SOSP 2023 describes the problem plainly. That cache is huge and it grows and shrinks unpredictably. Naive allocation wastes it through fragmentation and duplication, which caps how many requests you can batch.

The authors borrowed virtual memory paging from operating systems and built vLLM around it. They reported 2x to 4x higher throughput than FasterTransformer and Orca at the same latency, with the gap widening on longer sequences and larger models. That single idea is why an open serving stack can now compete with a hyperscaler’s internal one, and it’s the reason your hosted API bill fell without any new chip arriving.

The result shows up on the price lists. Both major vendors now charge far less for input tokens they have already seen, because a cache hit skips the expensive prefill. On OpenAI’s pricing page, cached input is priced at a tenth of base input across the current generation. Anthropic’s pricing page uses the same 0.1x multiplier on most models, and a 0.025x multiplier on Claude Fable 5.1 and Mythos 5.1.

Model	Input per MTok	Cached input per MTok	Output per MTok
gpt-6-astra (short context)	$10.00	$1.00	$50.00
gpt-5.6-terra (short context)	$2.00	$0.20	$12.00
gpt-5-nano	$0.05	$0.005	$0.40
Claude Opus 5	$5.00	$0.50	$25.00
Claude Sonnet 5	$2.00	$0.20	$10.00
Claude Haiku 4.5	$1.00	$0.10	$5.00

Even so, two traps hide in that table. OpenAI’s batch tier halves every rate, so gpt-6-astra runs at $5.00 input and $25.00 output if you can wait. Above the short-context threshold, long-context input is billed at double the short-context rate and long-context output at 1.5 times it. Anthropic notes that Claude 4.7 and later use a newer tokenizer, one producing roughly 30% more tokens for the same text. So a lower per-token price does not automatically mean a lower bill. Per-token rates are only half of what you pay, which is the same lesson as our breakdown of what AI actually costs a business per month.

Retrieval exists because weights are frozen and context is not free

A trained model knows what was in its training data and nothing else, which is why the retrieval layer exists at all. The retrieval-augmented generation paper, published at NeurIPS 2020, framed the fix as pairing parametric memory, the weights, with non-parametric memory, a dense vector index the model can query at run time. Its authors reported state-of-the-art results on three open-domain question answering tasks, and generations that were more specific and more factual than a parametric-only baseline.

Six years on, that non-parametric memory is usually a database you already run. The pgvector extension adds exact and approximate nearest neighbour search to Postgres 13 and later, supporting single-precision, half-precision, binary and sparse vectors across L2, inner product, cosine, L1, Hamming and Jaccard distance. Because it lives inside Postgres, you keep joins, ACID transactions and point-in-time recovery, which is the sort of boring property that decides production architectures.

The counter-case is real, though, and it’s getting stronger. Epoch AI measures LLM context windows growing 30x a year since 2023, doubling every 2.4 months, with a wide confidence interval of 10x to 50x. If a model can read a million tokens, a lot of small corpora fit without any retrieval at all. What keeps retrieval alive is the pricing above: context you paste is context you pay for on every call, and a cache hit only helps when the prefix repeats.

Agents run on two protocols, and they do different jobs

The top layer is the one that changed most recently, and it’s the least settled. An agent is a model given tools, a loop and permission to act, and until 2024 every vendor wired those tools up their own way. The Model Context Protocol specification standardised the connection, using JSON-RPC 2.0 between three roles: hosts, which are the LLM applications, clients inside them, and servers that expose capabilities. Servers offer resources, prompts and tools, while clients can offer sampling, roots and elicitation.

The specification is unusually direct about the danger. It states that tools represent arbitrary code execution, and that hosts must obtain explicit user consent before invoking any tool. Descriptions of tool behaviour, it adds, should be treated as untrusted unless they come from a trusted server. That’s a protocol document telling you the threat model is prompt injection through tool metadata, which is worth reading before you connect a server you did not write.

Tools represent arbitrary code execution and must be treated with appropriate caution. In particular, descriptions of tool behavior such as annotations should be considered untrusted, unless obtained from a trusted server.

Model Context Protocol specification, version 2025-06-18

Sitting next to MCP, and often confused with it, is the Agent2Agent protocol, whose specification is published under Linux Foundation copyright. A2A handles the other half of the problem. Independent and often opaque agents use it to discover each other, and to negotiate whether they are exchanging text, files or structured data. They then manage a shared task without access to each other’s internal state, memory or tools. It reuses HTTP and JSON-RPC 2.0 rather than inventing transport. Its own Appendix B describes the split cleanly. An A2A client asks a peer agent to do a job, and that peer uses MCP to reach the tools it needs.

What the top of the stack still cannot do

Every layer below the agent layer has a clean benchmark, but the top one does not. The most honest attempt to build one is still worth knowing. METR’s time horizon paper, published at NeurIPS 2025, times human experts on software tasks and then asks how long a task can be before a model’s success rate drops to 50%. On their task suite, Claude 3.7 Sonnet, the frontier model when the paper first appeared in March 2025, sat at roughly 50 minutes of human-equivalent work.

That horizon has been doubling about every seven months since 2019, and the authors attribute the gains mainly to reliability and to adapting to mistakes, alongside better logical reasoning and tool use. They also flag the limits of their own result, including how far it generalises beyond their task mix. Read it as the best available measurement of agent reliability rather than as a settled fact, because a 50% success rate is not a threshold most businesses deploy against.

This is where the stack metaphor earns its keep. The lower layers compound predictably, since compute, bandwidth and price per token all move on measurable curves. The top layer depends on a model holding a plan together across many steps, and nobody has a curve for that which is as tight as the hardware ones. Our look at how the current model families differ covers which of them are tuned for that kind of long-running work.

What would change this picture

Three things would rearrange the layers, instead of just moving numbers inside them. The first is memory bandwidth breaking out of its 28% annual trend, because the entire rack-scale design exists to work around that constraint. The second is context pricing falling far enough that retrieval stops paying for itself, which the 30x annual growth in context windows makes plausible and the per-token price lists currently prevent.

The third is the agent time horizon breaking its seven-month doubling in either direction. If it accelerates, the orchestration layer absorbs work that currently sits with people. If it stalls, the protocol layer keeps maturing while the thing it orchestrates stays a fifty-minute worker. Most of the value in AI technology then stays where it is today, in the four layers below the agent. Watch the METR series and the memory bandwidth trend, because those two move first.