Quick Answer: A chatbot that knows your business answers from your documents, not from the model’s memory. The reliable way to build one is retrieval-augmented generation (RAG): index your policies and data, retrieve the relevant passages for each question, constrain the model to answer only from them, and cite the source. Fine-tuning teaches style and format; it does not teach facts that change. Prompting alone teaches almost nothing.
Retrieval cuts hallucination sharply but does not eliminate it: Stanford’s evaluation of leading RAG-based tools found they still hallucinated 17% to 33% of the time, against 43% for a general-purpose model. The difference between a demo and a deployable chatbot is the evaluation harness: golden questions, wrong-answer tests, freshness tests, and escalation rules that catch a confident wrong answer before a customer does.
In February 2024, a Canadian tribunal ordered Air Canada to honor a bereavement refund its chatbot had invented. The airline argued the chatbot was a separate legal entity responsible for its own words. The tribunal disagreed and awarded the customer $812. The amount was small. The precedent was not: your chatbot’s answers are your company’s answers.
Most businesses discover this the same way. Someone wires a large language model to a chat widget, adds a paragraph of instructions, and ships it. It sounds fluent and helpful for a week. Then it tells a customer about a refund policy you discontinued two years ago, cites a page that does not exist, and a support ticket becomes a legal email. The bot was never lying. It was doing exactly what a language model does when it has no access to your facts: producing the most plausible-sounding sentence.
This guide is the practical answer to what we think of as the parrot problem: how to build a chatbot that answers from your business instead of talking about it, what each approach actually costs, and the testing that separates a demo from something you can put in front of customers.
Why a Raw GPT Wrapper Fails: Hallucination Is the Architecture, Not a Bug
A language model is a very good predictor of the next word. It has read a large fraction of the public internet and none of your internal documents. When you ask it about your refund policy, it does not look anything up. It generates the refund policy a company like yours would plausibly have, in confident prose, because confident prose is what it was trained to produce. If your actual policy matches the plausible one, it looks like the bot knows your business. If it doesn’t, you get the Air Canada problem.
This is why “make it stop hallucinating” is the wrong request. Hallucination is not a defect you patch; it is what generation without grounding is. The fix is architectural: give the model your facts at the moment it answers, constrain it to use them, and make it show where each claim came from. There are three ways to do that, and they are not interchangeable.
The Three Approaches Compared: Prompting vs RAG vs Fine-Tuning
Every “custom chatbot” you have ever been pitched is one of these three, or a combination. Understanding what each one fixes, and what it cannot, is most of the decision.
| Approach | What it fixes | What it can’t fix | Cost profile |
|---|---|---|---|
| System prompt Instructions and a few facts pasted into every request | Tone, persona, a handful of rules, a small amount of stable information | Any knowledge larger than a few pages; anything that changes; verifiability | Near zero to build; pays for the same pasted text on every call |
| Retrieval (RAG) Fetch relevant passages from your documents, answer only from them | Factual grounding, citations, large and changing knowledge bases, auditability | Poor source documents; questions your docs don’t answer; retrieval misses | Moderate build; low ongoing; scales with document volume, not model size |
| Fine-tuning Retrain the model’s weights on your examples | Style, format, domain vocabulary, consistent structured output | Facts that change; citing sources; anything added after training day | High upfront (data prep, training runs); must be repeated as knowledge changes |
The most common and most expensive mistake is fine-tuning to teach facts. Fine-tuning changes how a model writes; it is a poor and unreliable way to change what it knows, and whatever it does absorb starts going stale the day training finishes. For a business whose facts change, retrieval is not one option among three. It is the one that works.
The RAG Walkthrough for Non-Engineers
Retrieval-augmented generation sounds complicated and is conceptually simple: open-book exam instead of closed-book. Here is the whole pipeline.
Step 1: your documents. Policies, help articles, contracts, product specs, pricing sheets, internal FAQs. The quality of everything downstream is capped by the quality of this pile, which is why the first week of a serious build is usually spent finding out which documents contradict each other. Step 2: chunk and index. Documents are split into passages of a few hundred words and stored in a way that lets the system find passages by meaning, not just keywords. Chunking is where a lot of quiet quality lives: split a refund policy mid-sentence and the bot retrieves half a rule. Step 3: retrieve. When a question arrives, the system pulls the handful of passages most relevant to it. Step 4: answer, grounded. The model receives the question and those passages, with instructions to answer only from them and to say so when they don’t contain the answer. Step 5: cite the source. Each answer links back to the passage it came from, so a customer, a support agent, or a lawyer can check it in one click.
Steps four and five are where hallucination is contained. The model is no longer free to generate a plausible policy; it is constrained to the actual one, and it has to show its work. That constraint is also what makes the system auditable, which matters more than most founders expect the first time a regulator or a customer’s counsel asks how an answer was produced.
The Freshness Advantage: Why Retrieval Beats Retraining
Business facts have a short half-life. Prices change quarterly, policies change with legal reviews, product names change with marketing. A fine-tuned model captures all of it as of one date and then drifts away from reality with every change, silently, because nothing in the model knows the world has moved on. Model decay is not a hypothetical: a study of 128 production models found 91% degrade over time without intervention, a pattern we mapped in detail in the half-life of an AI agent.
Retrieval sidesteps the problem. Update the document, re-index, and the next answer reflects the change, with no training run and no waiting. That is the practical reason RAG has become the default architecture for business chatbots: not because it is cleverer, but because your knowledge base is the only part of the system that has to be right today, and RAG is the only approach where updating it is cheap.
Building the Evaluation Harness: The Part That Separates Demos from Deployments
Here is the sobering data point. Stanford researchers evaluated the leading commercial legal research tools, all built on retrieval, across 202 expert-scored queries. The RAG tools hallucinated between 17% and 33% of the time, versus 43% for a general-purpose GPT-4 with no retrieval. Better, clearly. But the vendors had marketed these products as “hallucination-free,” and in the researchers’ words,
“The hallucination problem persists at significant levels.”
Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies (2025)
Architecture cuts the rate. Only measurement tells you what your rate actually is, and most chatbot projects never measure it, which is why they ship on the strength of a demo and fail in the field, the pattern behind why PoCs don’t move the efficiency needle. The harness is four test sets, run automatically on every change.
Golden questions: two hundred or more real customer questions with verified correct answers, scored automatically for factual match and source citation. This is your accuracy number, and it should be on a dashboard before launch. Wrong-answer tests: questions where the correct answer is “we don’t offer that,” designed to catch the bot’s instinct to please. A chatbot that invents a discontinued refund policy fails this set, and it is the set most teams forget. Freshness tests: questions whose answers changed last quarter, to verify the index was updated and the old passage isn’t still winning retrieval. Escalation rules: defined conditions under which the bot stops answering and hands off to a human: retrieval found nothing relevant, confidence is low, or the topic is on a high-stakes list (billing disputes, legal, safety). The escalation path is not a fallback; it is the feature that makes the whole system safe to deploy, and its trigger rate is a health metric worth watching weekly.
Have a chatbot that talks about your business instead of knowing it?
Techuz builds retrieval-grounded chatbots with citations on every answer, an evaluation harness that runs on every change, and escalation rules designed before launch, not after the first incident.
Cost Expectations by Approach and Scale
Costs vary with document volume, query volume, and the model tier, so treat these as shapes rather than quotes. A prompt-only bot is nearly free to build and surprisingly expensive to run at volume, because the same instructions and facts are sent on every call; it is also the cheapest way to end up in a tribunal. A retrieval system is a moderate build (weeks, not days, if the evaluation harness is included) and cheap to operate, because the model only sees a few relevant passages per question and the index is updated rather than retrained. Fine-tuning is the most expensive upfront (data preparation dominates, not compute) and recurs every time the knowledge changes, which for most businesses is the deciding argument against using it for facts.
The cost that dwarfs all three is the one that never appears in the build estimate: the cost of a confident wrong answer at scale. The evaluation harness is the cheapest line item in the project and the only one that bounds that risk. Unit economics of the model calls themselves are a separate topic, and one worth planning for early, since inference cost scales with usage in a way most software does not. The broader principle of where AI pays off, verification cost against task cost, is the one we laid out in where generative AI actually improves efficiency.
Decision Matrix: Use Case to Recommended Approach
| Use case | Recommended approach | Why |
|---|---|---|
| Customer support on policies, pricing, and product facts | RAG with citations and escalation | Facts change; answers must be verifiable; wrong answers carry liability |
| Internal knowledge assistant over docs, wikis, and tickets | RAG with access controls | Large, changing corpus; permissions must follow the source document |
| Brand-voice writing assistant or structured output generator | Fine-tuning, often plus a prompt | The goal is style and format, not facts |
| Domain-heavy assistant (legal, medical, technical) over your documents | RAG, optionally on a lightly fine-tuned model | Vocabulary from tuning, facts from retrieval, citations non-negotiable |
| Simple FAQ bot with under a dozen stable answers | System prompt | Small enough to paste; revisit the moment it grows |
The pattern is consistent: when the goal is knowing, retrieve; when the goal is sounding a certain way, tune; and when the answer can be wrong at someone’s expense, cite and escalate. Getting that architecture right is the core of building AI people can trust, and it is the difference between a chatbot that becomes an asset and one that becomes a legal exhibit.
The Pre-Launch Checklist
- Every answer cites the passage it came from, and the citation is clickable.
- The bot says “I don’t have that information” when retrieval finds nothing, and a test set proves it.
- Golden-question accuracy is measured, on a dashboard, before the first customer sees it.
- Freshness tests confirm last quarter’s changes are reflected in answers.
- Escalation rules exist for high-stakes topics, and a human is actually on the other end.
- The source documents have been reviewed for contradictions, because the bot will find them.
If any item is unchecked, you have a demo. Six checks and you have a chatbot that knows your business, and can prove it.
Build the version that can show its work
As an LLM development company and a generative AI development company, Techuz delivers custom LLM development with retrieval architecture, citations, and the evaluation harness as standard deliverables.
FAQs
What is the difference between RAG and fine-tuning for a business chatbot?
Retrieval-augmented generation looks up relevant passages from your documents at answer time and constrains the model to them, which is how a bot learns facts and can cite them. Fine-tuning retrains the model on examples, which changes style and format but is a poor and quickly stale way to teach facts. For knowledge that changes, retrieval wins.
Does RAG eliminate hallucinations?
No. It reduces them substantially: Stanford’s evaluation found leading retrieval-based tools hallucinated 17% to 33% of the time versus 43% for a general model without retrieval. The remaining risk is managed with citations, an evaluation harness, and escalation rules, not with architecture alone.
How much does it cost to build an AI chatbot that uses our own data?
It depends on document volume, query volume, and model tier, but the shape is consistent: prompt-only bots are cheap to build and costly to run at volume, retrieval systems are a moderate build with low ongoing cost, and fine-tuning is expensive upfront and recurs whenever facts change. The evaluation harness is the smallest line item and the one that bounds the cost of wrong answers.
Can we build a grounded chatbot on our existing help center and PDFs?
Yes, and that is the typical starting corpus. The first week usually goes to finding contradictions between documents, because the bot will surface every one of them. Clean, current source documents matter more than model choice.
What should happen when the chatbot doesn’t know the answer?
It should say so, cite nothing, and escalate to a human when the topic is high-stakes or retrieval found nothing relevant. That behavior has to be tested deliberately with wrong-answer test sets, because a model’s default instinct is to produce a plausible answer rather than admit a gap. An experienced AI development company will design the escalation path before the first customer conversation, not after it.
Sources
- Magesh, Surani, Dahl, Suzgun, Manning & Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies (2025), Stanford RegLab
- CBC News, Air Canada Found Liable for Chatbot’s Bad Advice on Bereavement Fares (2024)
- Vela et al., Temporal Quality Degradation in AI Models, Scientific Reports (2022)