The self-hosted AI stack: what to run at each layer

You can run a complete AI stack on hardware you control. Whether you should depends almost entirely on how much of your time an hour of maintenance is worth.

Here’s what the pieces are, which ones are genuinely mature, and the cases where self-hosting is clearly the wrong call.

The five layers of a self hosted AI stack

Layer	Does what	Common choice
Inference engine	Runs the model	Ollama, llama.cpp, vLLM
Model weights	The model itself	Open-weight releases
Retrieval	Searches your documents	Postgres with pgvector
Gateway	Routing, keys, limits, logs	A proxy in front of everything
Interface	What people actually use	Open WebUI or your own app

You don’t need all five to start. One engine and one interface is a working system, and the rest gets added when something specific hurts.

Pick the engine for the workload

This is the decision that determines your operational experience, and the two options optimise for different things.

Ollama handles model download, quantisation and a local server behind two commands. It’s the right choice for a handful of users and it’s how most people should begin.

Serving one person and serving two hundred are different engineering problems. The engine that wins at one loses badly at the other.

For concurrency you want vLLM. Kwon and colleagues’ PagedAttention work reported improving throughput “by 2-4x” against prior systems at the same latency by managing attention memory the way an operating system manages pages.

Underneath the simpler tools sits llama.cpp, which runs inference across CPU and GPU and gives you direct control over quantisation and sampling when you need it.

Retrieval belongs in the database you already run

The instinct is to add a dedicated vector database. For a self-hosted stack that’s usually the wrong first move.

pgvector adds vector similarity search to Postgres, which means your embeddings live alongside your application data with the same backups, the same monitoring and the same restore procedure.

That matters more when you’re the operations team. A second datastore is a second thing to back up and a second thing to be woken by, and our guide to when you actually need a vector store covers where the threshold sits.

The gateway is the piece people skip

A proxy sitting between your applications and your model endpoints looks like unnecessary indirection until the first time you need it.

It gives you one place to enforce rate limits, rotate keys, log requests, and swap a model without touching application code. It also lets you fall back to a hosted provider when your own hardware is saturated.

Because most engines expose an OpenAI-compatible API, that fallback is a configuration change rather than a rewrite, which is the quiet reason this stack is worth assembling at all.

Sizing the hardware

Memory is the binding constraint. Weights at four bits need roughly half a byte per parameter, so a 13B model wants about 7GB before context.

Quantisation is what makes this affordable, and it holds up well. GPTQ reported quantising to 3 or 4 bits per weight “with negligible accuracy degradation relative to the uncompressed baseline”.

Budget for concurrency separately. Each simultaneous conversation holds its own context in memory, so ten users on a 13B model is a different machine from one user on the same model.

Our guide to what fits on a given machine works through the arithmetic in more detail.

Choosing weights

Open-weight does not mean open-licence, and the distinction has bitten people commercially.

Some releases carry restrictions on commercial use or on user counts. Others are genuinely permissive, and Apache 2.0 releases keep arriving at sizes that fit a single machine.

Read the licence before you build on a model, because migrating a stack because of a licence term is a worse afternoon than checking one.

What this genuinely gets you

Data residency is the strongest reason and the one that survives scrutiny. Nothing leaves your network, which answers a category of legal and contractual questions rather than mitigating them.

That has become more concrete as regulators engage. The erasure and accuracy obligations covered in our piece on GDPR and model weights are easier to satisfy when you control where every copy lives.

Cost predictability is the second. You’ve bought the hardware, so a spike in usage costs electricity rather than an invoice, which changes what you’re willing to build.

And stability. A hosted model can be updated or deprecated underneath you; a file on your disk behaves the same next year.

What it actually costs

The honest accounting is where most self-hosting arguments fall apart, so it’s worth doing properly.

You’re now responsible for uptime, updates, security patching, capacity planning and someone being available when it breaks. None of that appears in a hardware quote.

Capability is the other cost. Open models are good and frontier hosted models are better at hard reasoning and long documents, so some of your workload will still route outward.

Compare against real usage rather than a worst case. Our piece on what an AI feature costs shows how often the hosted bill is smaller than the intuition suggests.

When not to do this

If your usage is low, hosted wins on every axis including cost. A few hundred requests a day does not justify a machine and a maintenance rota.

If your team has no operations capacity, this becomes one person’s unpaid second job, and that arrangement fails quietly rather than loudly.

And if you need frontier capability specifically, self-hosting doesn’t provide it at any budget you’re likely to have.

A sensible order to build a self hosted AI stack

Start with Ollama and a 7B model on a machine you already own, plus Open WebUI so people can use it without writing code.

Add retrieval when the model needs your documents. Add the gateway when a second application appears. Move to vLLM when concurrency starts hurting, and not before.

Each step should be triggered by something that actually broke. A stack assembled in anticipation of problems you never have is the most expensive version of this you can build.