Vector databases explained: what they do and when you need one

Do you need a vector database? Probably not yet, and the honest version of this guide starts by talking you out of one.

Most teams reach for a dedicated vector store at the point where a table in the database they already run would do the job. Here’s what the thing actually does, and where the threshold sits.

What a vector actually is

An embedding model turns a piece of text into a list of numbers, typically a few hundred to a few thousand of them. Similar meanings land near each other in that space.

That’s the whole trick. Comparing two texts becomes comparing two lists of numbers, which a computer can do quickly and which doesn’t care whether the words match.

Keyword search finds documents containing your words. Vector search finds documents about your question. Those are different jobs and you often want both.

The technique got practical with sentence-level embedding models. Reimers and Gurevych’s Sentence-BERT reported reducing the effort of finding the most similar pair in a 10,000-sentence collection “from 65 hours with BERT / RoBERTa to about 5 seconds”, which is the difference between a research curiosity and a product.

Why anyone bothers with vector databases

Finding the nearest vectors by checking every one is simple and gets slow at scale, so the field uses approximate search instead.

The dominant method is HNSW. Malkov and Yashunin described it as “a fully graph based incremental K-ANNS structure” that “strongly outperforms” previous open-source approaches, and it’s what most vector stores run underneath.

Approximate means you trade a small amount of recall for a large amount of speed. For search that’s almost always the right trade, since the fifth-best match is usually fine.

At genuinely large scale the constraint moves to hardware. Johnson and colleagues built FAISS for billion-scale similarity search on GPUs, and if you’re operating there, this guide is not the one you need.

The threshold, stated plainly

Scale	What to use	Why
Under ~10,000 documents	An array in memory, brute force	Comparing 10,000 vectors takes milliseconds
10,000 to a few million	Postgres with pgvector, or similar	One system to operate, transactions included
Millions upward, low latency	A dedicated vector store	Index tuning and sharding start to matter
Billions	FAISS-class infrastructure	You are now running a search team

The second row is where most teams should stop and usually don’t. An extension in the database you already back up, monitor and know how to restore beats a second system with its own failure modes.

Adding a dedicated store means keeping two systems in sync. When a document is edited or deleted, both have to know, and stale vectors returning deleted content is a real and irritating class of bug.

Chunking decides your quality

You don’t embed whole documents. You split them, and how you split them affects results more than which database you picked.

Chunks that are too small lose the context needed to understand them. Chunks that are too large average several ideas into one vector, which matches everything weakly and nothing well.

Split on structure rather than character count where you can. A section, a heading, a paragraph group, anything the document itself treats as a unit will beat an arbitrary 500-character window.

And store the surrounding context alongside the chunk. Retrieving a paragraph plus its heading gives the model something it can actually use, which our piece on how RAG works goes into further.

Vector search alone is not enough

This is the mistake that costs teams the most time, so it’s worth stating early rather than discovering in production.

Semantic search is poor at exact matches. Product codes, error numbers, names, version strings and anything else where the literal characters matter will be reliably missed, because those carry little semantic signal.

The fix is hybrid retrieval: run keyword search and vector search together and merge the results. Nearly every serious system does this, and teams that skip it end up rebuilding it after the first support ticket about a missing part number.

Reranking helps further. Khattab and Zaharia’s ColBERT showed late-interaction scoring recovering much of the accuracy of full cross-attention at far lower cost, and a cheap first pass followed by a precise rerank is the standard shape.

Retrieving more is not retrieving better

A tempting response to poor results is to pass more chunks to the model. It usually makes things worse.

Stanford’s Lost in the Middle found that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts”.

So the correct chunk sitting eighth out of twenty may as well not be there. Five good chunks beat twenty mediocre ones, and they cost less per query too.

Things that bite later

Changing your embedding model invalidates every vector you have stored. There is no migration path; you re-embed the corpus, which is a real cost worth knowing before you pick a model.

Metadata filtering also interacts badly with approximate indexes. Filtering to a small subset after an approximate search can return very little, and doing it before means the index can’t help, so check how your chosen system handles that specific case.

And measure retrieval separately from generation. If you only evaluate final answers you can’t tell whether the model reasoned poorly or was handed the wrong documents, and those need different fixes.

Where this fits

The original RAG paper framed the point as combining “pre-trained parametric and non-parametric memory”, and the non-parametric half is what a vector store holds.

That framing is a useful check on your architecture. If you’re storing facts that change, they belong in retrieval rather than in a fine-tune, which our guide to how retrieval works works through.

If you are running the whole thing yourself, that ordering matters even more, and our guide to the self-hosted stack covers why retrieval usually belongs in the database you already operate.

Choosing between Postgres and a dedicated system is the decision most teams get wrong, and our comparison of where that threshold sits works through it with a sizing example.

Start brute force. Move to your existing database when brute force gets slow. Buy a dedicated system when you can name the specific limit you hit, and not before.