Why a chatbot gives wrong answers from your documents

A chatbot "trained on your documents" can still give a wrong answer. It can quote an old price list, mix two policies into one, or invent a rule that is in none of your files. If you tried one and stopped trusting it, the cause is usually in how it was built and tested, not in your files alone.

Wrong answers cannot be removed completely. They can be made rarer, easier to spot, and less harmful. This post explains why they happen, what retrieval means in plain words, and how to test a bot with your own questions before customers or staff rely on it.

Why "trained on your files" does not mean always right

Most document chatbots do not learn your files the way a student learns a book. The underlying language model was trained earlier on general text. It writes fluent sentences by predicting likely words. Fluent is not the same as true.

Your documents are added afterwards, at question time. If that step goes wrong, the model falls back on its general habits and fills the gap with something plausible. AWS lists "presenting false information when it does not have the answer" as a known weakness of language models in its explainer on retrieval-augmented generation (opens in a new tab).

Common causes of wrong answers include the following.

• The right passage was never found, so the bot answered without it.
• The wrong passage was found, for example an old version of the policy.
• Two documents disagree and the bot blends them.
• The file is a scan or a table that was read badly.
• The question is not covered by any file, and the bot answers anyway.

How retrieval works, in plain words

Retrieval means finding the right pages first, then answering from them. The bot does not "remember" your files. It looks things up, like a staff member who opens the folder before replying.

The steps look like this.

1. Your documents are split into small pieces, often a few paragraphs each.
2. When someone asks a question, the system searches those pieces for the ones closest in meaning to the question.
3. It hands the best pieces to the language model along with the question.
4. The model writes an answer using those pieces, ideally with a note of where each one came from.

AWS describes the same idea: the model is directed to consult an outside knowledge source before it answers, which works without retraining the model. The same page presents this as "one approach to solving some of these challenges", not a cure.

Each step can fail. The split can cut a rule away from its exception. The search can pull the wrong piece. The model can ignore the piece it was given. That is why testing each answer matters more than trusting the method.

Clean the source documents first

Start before any technical work: give the bot a tidy library.

• Remove old versions. If a 2022 and a 2025 price list both sit in the folder, the bot may use either. Keep the current one, or label each file with its date and status.
• Resolve contradictions. If two documents disagree, decide which is right and fix or remove the other.
• Fix scans. A scanned PDF with no readable text may give the bot nothing. Run it through text recognition and check the result by eye.
• Check tables. Prices and eligibility rules often live in tables, which can read out of order. Test them with specific questions.
• Use clear headings and plain names. If staff call a product three different names, add the other names to the document.
• Leave out what should not be shared. Anything in the library can end up in an answer. Do not include confidential files, salaries or personal data unless the bot's users are allowed to see them.

Name one person as the owner of the library. When a policy changes, that person updates the file, and the bot is tested again.

Make the bot say "I don't know"

A good bot declines when your files do not answer the question. This does not happen by default. It has to be instructed and tested.

The instruction to the model can say, in plain terms, to answer only from the supplied passages, to say it does not know when they do not cover the question, and to point the user to a named person or contact. Wording alone does not guarantee compliance, so test it.

Run a "decline test". Ask five or more questions that your documents do not answer. Use things such as a product you do not sell, a policy you have never written, or a question about a competitor. A pass is a clear refusal with a route to a person. A fail is a confident answer.

Also test the opposite. A bot that declines everything is safe but useless, so ask questions your documents do answer, and check it replies.

Show the source with every answer

Every answer should name where it came from, such as the document title, the section and, ideally, a link that opens the page. AWS notes that this lets users check the source documents themselves.

This changes how people use the bot. A staff member who sees "Leave Policy 2025, section 4" can open it and confirm in ten seconds. An answer with no source forces blind trust.

Sources also help you debug. If a wrong answer cites the old price list, you know the fix is in the library. If it cites nothing, the model has gone beyond your files. Treat a missing source as a warning, and treat a source that does not support the claim as a failure.

Test with real past questions

The best test questions are the ones real people have already asked. Pull them from your email, support inbox, WhatsApp chats or a staff list of "questions I answer every week". Aim for a good spread. Include easy questions, tricky ones with exceptions, ones that need two documents, and ones with no answer in your files.

Write the correct answer yourself first, with the document and page that holds it. Then run each question through the bot and fill in a table like the one below. The rows are a template for you to fill in, not results from any real bot.

Run the full set before any change, then again after each change, such as cleaning the library, changing the instructions or switching the search settings. Keep both tables. The difference between the "before" and "after" runs shows what each change did on your own questions.

Mark a failure by type so you can fix the cause.

• Wrong source. Fix the library or the search.
• Right source, wrong answer. Fix the instructions or the way documents are split.
• No source, answer given. Tighten the decline rule.
• Correct but no source shown. Fix how the answer is displayed.

Keep the set and add to it. When a user reports a wrong answer, add that question to the table. Over time the set becomes a regression test that you re-run whenever the library or the bot changes.

Keep a person in the loop

For low-risk questions, such as office hours or where to find a form, a bot can answer alone. For higher-risk questions, such as legal terms, refunds, health, safety or money, a person should review the answer or take over the chat.

Practical steps help here.

• Give the bot a clear way to hand over to a named person.
• Let users flag an answer as wrong with one click, and read those flags weekly.
• Log the questions, answers and sources, so you can review mistakes.
• Tell users they are talking to an automated assistant, and that it can be wrong.
• Review the test set whenever a policy, price or product changes.

NIST's AI Risk Management Framework (opens in a new tab) is a voluntary guide for organisations that want to manage AI risks in a structured way. Small teams do not need all of it, but it is a useful reference.

Where to go from here

Collect 20 real questions from your own inbox, write the correct answers, and run them through the bot you have. That one table shows where the problems are. If you want a document bot built with source links, a decline rule and a test set, our AI automation service covers that work.