Retrieval-Augmented Generation has gone from a research paper term to something every B2B SaaS founder asks us about within the first ten minutes of a call. Some of that is genuine need. A fair amount of it is FOMO from watching a competitor announce an “AI-powered” feature. Worth separating the two before committing engineering time.
What RAG actually solves
A language model on its own only knows what it was trained on, which means it can’t answer questions about your product’s documentation, your customer’s specific account data, or anything that changed after its training cutoff. RAG fixes this by retrieving relevant information from your own data - a knowledge base, your database, your support tickets - and feeding it to the model alongside the user’s question, so the answer is grounded in your actual, current data instead of the model’s general knowledge.
That’s the whole idea: search first, then generate. It’s less “the AI knows everything” and more “the AI can read the right five documents out of your ten thousand and summarize them accurately.”
What the pipeline actually consists of
It helps to know what you are buying before scoping it, because “add RAG” describes five components, and the engineering effort is unevenly distributed between them.
- Chunking. Source documents are split into passages small enough to retrieve precisely. This sounds trivial and is the decision that most often determines whether the finished system works, because a chunk boundary drawn through the middle of an answer means the answer can never be retrieved whole.
- Embedding. Each chunk is converted into a vector that represents its meaning, so that questions can be matched against passages by similarity rather than by shared keywords.
- Storage and indexing. Those vectors go into a store that can search them quickly, alongside the original text and whatever metadata you will need to filter on later - source, customer, date, access level.
- Retrieval, and usually reranking. A question fetches the closest chunks, and a second pass reorders them by actual relevance to the question rather than by raw similarity. Skipping the second pass is a common shortcut and a common cause of plausible but off-target answers.
- Generation. The chosen passages are assembled into a prompt with the question and instructions about what to do when the passages do not contain an answer.
Notice that four of the five are search engineering, not model work. That distribution is the single most useful thing to understand about these projects, and it is why teams that treat RAG as a model problem tend to be disappointed by the result.
Where it’s actually earning its keep in B2B SaaS
- Support deflection. A support assistant that retrieves from your actual documentation and past resolved tickets answers real product questions instead of generic ones - this is the single highest-ROI use case we’ve built for clients, because it directly reduces support headcount pressure.
- Internal knowledge search. For SaaS companies with sprawling internal wikis, a RAG-powered search that actually understands the question (not just keyword-matches) saves real employee time, particularly at companies past 50-100 people where “ask in Slack” stops scaling.
- In-product data Q&A. Letting a customer ask “which of my invoices are overdue” in plain language, with the answer pulled from their actual account data, is a genuinely differentiating feature for analytics-heavy or data-heavy SaaS products.
The multi-tenant problem, which is where B2B differs from everything else
Most published guidance on RAG is written for single-tenant assistants over public or company-internal content. B2B SaaS is neither, and this is the requirement we most often find missing from a proposal written by somebody who has built a demo but not a product.
In a multi-tenant product, retrieval is an access-control decision. If one customer’s documents, tickets, or account records can be returned as context for another customer’s question, you have built a data leak with a natural language interface, and it will be discovered by the first person who asks a curious question rather than by your tests. The same applies inside a single tenant, where roles differ: a support agent and an end user asking the same question should not necessarily be served the same source passages.
The safe pattern is to treat tenant and role as a hard filter applied during retrieval rather than as an instruction in the prompt. Filtering at query time, against metadata stored with each chunk, means a passage the user is not entitled to is never a candidate. Telling the model to ignore documents the user should not see is not a control, because the passage has already been placed in the context window by the time the instruction is read. Getting this wrong is also the failure most likely to end up in a customer’s security review rather than in your bug tracker.
Where we push back on clients
Not every feature request needs RAG. If the answer to “add an AI chatbot” is really “our search is bad” or “our documentation is disorganized,” fixing search and documentation is usually faster, cheaper, and more reliable than layering a language model on top of the mess. We’ve talked more than one client out of a RAG project by first fixing their underlying data structure - and the resulting plain search often solved 80% of the actual user complaint.
The other place we push back: RAG systems are only as good as what they retrieve. If your knowledge base is outdated, contradictory, or poorly organized, RAG will confidently retrieve and summarize the wrong information just as fluently as the right information. Building the retrieval system before cleaning up the underlying content is building on a bad foundation.
When it goes wrong, look at retrieval before you look at the model
Almost every disappointing RAG system we have been asked to review had a retrieval problem being treated as a model problem, and teams lose weeks swapping models or rewriting prompts before checking whether the right passage was ever fetched. The diagnostic is simple: take the questions the system answers badly and look at what it actually retrieved. If the correct passage is not in that list, no amount of prompt engineering will produce a correct answer.
- The answer is split across a chunk boundary, so neither half is a strong match for the question and neither is retrieved.
- The index is stale. Documentation was updated, the index was not rebuilt, and the system is faithfully summarising last quarter’s policy.
- The question and the document use different vocabulary - customers say “bill”, the documentation says “invoice” - which similarity search handles better than keyword search but not perfectly, and which reranking or query expansion exists to address.
- Too much was retrieved. Passing twenty marginal passages instead of five good ones dilutes the context and reliably makes answers vaguer, not better.
How to tell whether it works before customers do
The most common gap in these projects is having no way to answer “is it good?” other than someone trying a few questions and forming an impression. That is not enough to ship on, and it is not enough to detect the regression that arrives when a model, a prompt, or the content changes.
What we ask for is modest: a set of real questions taken from actual support tickets or search logs, each with the source passage that genuinely answers it. That set gives you two separate measurements, and separating them is the point. First, how often the correct passage appears in what was retrieved, which tests the search half and is where most problems live. Second, how often the final answer is both correct and actually supported by the passages provided, which tests the generation half. A system can score well on the first and badly on the second, and the fix in each case is completely different.
It is also worth deciding, in advance, what the system should do when retrieval returns nothing useful. A product that says it cannot find an answer and offers a route to a human is behaving correctly. One that improvises is generating the exact liability the retrieval step was introduced to prevent.
What it costs to run, which is not what it costs to build
Budgets for these features tend to cover the build and stop there, but a RAG feature has a running cost with a different shape from the rest of your product. Every question spends money on retrieval and then on generation, and the generation cost rises with how much context you pass, which means the decision to retrieve ten passages instead of five is a permanent multiplier on your per-question bill rather than a one-time choice.
Content that changes has to be re-embedded and re-indexed, so a large, frequently edited knowledge base carries an ongoing cost that a static one does not. There is also a latency budget to plan: retrieval, reranking and generation happen in sequence, and a chain that is individually reasonable at each step can add up to a wait that feels slow in a support widget where the alternative is a search box that returns instantly.
None of this makes the feature a bad investment. It does mean the sensible way to size it is per question rather than per project, because that is the number that grows when the feature succeeds.
What “the new standard” actually means
We’d push back gently on calling RAG “the new standard” - it’s a standard for a specific class of problem: answering questions that require current, private, or specific data the model wasn’t trained on. For generating marketing copy, summarizing public information, or general reasoning tasks, plain language model calls are simpler and cheaper. The standard isn’t “use RAG everywhere,” it’s “know which problem you actually have.”
What we’d tell you before you scope this
Before building a RAG feature, we ask clients three things: what specific question are users actually asking that current search or documentation fails to answer, how current and clean is the underlying data it would retrieve from, and what happens when the system is confidently wrong - because it will be, sometimes. If you have solid answers to those three, RAG is usually the right call. If you don’t, that’s the actual project - fix the data and the failure handling first.
We’ve built this for clients across AI integration projects where it genuinely moved support and search metrics. Reach out if you want an honest read on whether this is the right investment for your product right now.