Your new AI chatbot probably has access to every customer's files

Over the past eighteen months, hundreds of small and mid-size companies have added an AI assistant to their website or customer portal. A distributor adds a chat box so contractors can look up spec sheets and past quotes. A clinic adds an intake assistant. A leasing firm adds natural-language search across customer agreements.

The web portal itself usually has decent access controls: when Contractor A clicks "My Quotes," the SQL query checks WHERE account_id = 1042.

Then someone builds the AI assistant.

How the leak happens

To make the AI assistant useful, a developer indexes the company's PDFs, quotes, tickets, or order notes into a vector database (like Pinecone, Qdrant, or Postgres pgvector). When a signed-in user asks a question, the backend runs a similarity search against those embeddings and hands the top five chunks to the language model.

Here is the mistake we see over and over: the similarity search queries the entire vector collection without filtering by the signed-in user's account_id.

Nobody notices during testing, because the developer tests the chat box by asking about their own sample account. Everything looks fast and accurate. Meanwhile, any signed-in customer who asks, "What discounts have you given other contractors on 316 stainless anchors this month?" gets a polite, accurate summary pulled straight from their competitors' uploaded quote PDFs.

Four things we test on every in-scope AI feature

When your scope includes a public chatbot, an in-app AI search bar, or a tool-calling agent, our models and reviewers test four specific failure modes:

1. Cross-account RAG & document leakage: Signing in as Account A and testing whether the assistant can be prompted into retrieving, summarizing, or quoting records that belong to Account B or internal staff.
2. Tool-calling & "confused deputy" flaws: Many AI assistants can call backend functions (lookupOrder, updateShippingAddress, exportInvoice). Often those functions run with a privileged backend API key and trust the language model not to pass someone else's order_id.
3. Indirect prompt injection: Submitting harmless canary instructions inside a quote request, support ticket, or uploaded document to see whether an internal staff AI tool executes those instructions when summarizing the queue.
4. Leaked API keys & unthrottled spend ("denial of wallet"): Checking client-side bundles and public /api/chat routes to make sure an attacker cannot script thousands of expensive requests against your OpenAI or Anthropic billing account overnight.

The fix is usually ten lines of code

Like most permission bugs, the repair is rarely a rewrite. It usually means attaching account_id metadata to every indexed chunk and enforcing a server-side filter taken from the user's session—never from the prompt—before the model ever sees the search results.

If you have shipped an AI feature in the last year, our Application and Infrastructure plans (and our AI & Agent Deep-Dive add-on) are built to test it alongside the rest of your web stack.