Multi-tenant RAG: keeping one tenant's documents out of another's answers

The first instinct when adding multi-tenancy to a RAG system is to reach for the same pattern you'd use anywhere else: a tenant_id column, a WHERE clause, done. That works for your relational tables. It does not automatically work for the vector store sitting next to them.

Where the leak actually happens

A retrieval-augmented system's failure mode isn't a SQL query missing a filter — it's a nearest-neighbor search returning the closest vectors regardless of whose documents they came from, unless you explicitly constrain it. If tenant A's onboarding PDF and tenant B's onboarding PDF are semantically similar (and onboarding PDFs usually are), an unconstrained retrieval can easily hand tenant A's support agent a chunk of tenant B's private document, because it's simply the nearest match in the embedding space.

This is invisible in almost every demo. One test tenant, one document set — there's nothing to leak into. It becomes visible exactly once, on the day a second real customer signs up and the first customer's support agent starts answering with details it shouldn't have.

The fix, and why it has to be structural

Every retrieval call needs a tenant filter applied at the vector store level, not as a post-filter on results. Filtering after retrieval means you already pulled the wrong tenant's data into memory and just chose not to show it — a much weaker guarantee than never fetching it in the first place. Most vector databases support metadata filtering at query time; the discipline is making sure literally every retrieval path in the codebase uses it, including the ones written in a hurry six months from now.

The second layer is per-tenant namespacing or collection separation where the vector store supports it, so a filter bug degrades to "no results" instead of "wrong tenant's results." Defense in depth matters more here than almost anywhere else in the stack, because the failure is silent — nothing crashes, nothing errors, the agent just quietly says something it shouldn't.

What I test for now

Two tenants, overlapping document topics, and an explicit assertion that querying as tenant A never returns a chunk owned by tenant B. It's a small test to write and it catches the exact class of bug that a single-tenant demo will never surface.