When the internal AI assistant has two answers to the same question
How an internal AI assistant should handle two conflicting policy documents: authority, effective dates, source owners and when to escalate.
The short answer
When two documents disagree, the assistant should not pick the one that sounds more confident or the one it found first. It should rank sources by authority and effective date, show which source it used, and say plainly when the conflict cannot be settled from the documents alone. In that case it should qualify the answer or route the question to the person who owns the policy. Guessing is the one thing it must not do.
Key points
- Contradictions are normal. Policies get updated, old copies stay in shared drives, and email clarifications never make it back into the document.
- Every source the assistant reads needs three labels: who owns it, what authority it carries, and from when it applies.
- The assistant should cite the source behind every policy answer, so staff can check it.
- When authority and dates do not settle a conflict, the right output is “these two sources disagree; here is who decides”.
- Fix the documents, not just the assistant. Every escalated conflict is a pointer to a document that needs an owner’s attention.
How two answers end up in one assistant
An internal assistant is usually connected to whatever the business already has: shared drives, SOP folders, policy PDFs, email archives, maybe a wiki. Nobody curated those for consistency. A policy revised last quarter sits next to the version it replaced. A regional manager’s email changed a rule for one branch. A sales deck paraphrased a policy loosely.
The assistant does not know any of this unless it is told. Given two documents that both look relevant, a plain retrieval system will pick whichever matches the question’s wording more closely. That is not the same as picking the one that is correct.
An illustrative example: free time on containers
Consider a hypothetical freight forwarder in Dubai with an internal assistant that sales and operations staff use to check policies.
A sales executive asks: “How many free days of container detention can I offer a regular customer on an import?”
The assistant finds two documents:
-
Document A: “Commercial Policy, version 3”, issued by the commercial director, effective 1 January. It says standard free time is set by the shipping line, and any extra free days require written approval from the commercial director.
-
Document B: “Sales Team Guide”, a slide deck prepared by a sales manager, last edited in March. It says sales executives may offer up to five additional free days to customers with a good payment history.
A naive assistant might answer “up to five additional days”, because Document B matches the phrasing of the question. That answer could cost the company real money if the shipping line charges detention that the forwarder has promised to absorb.
A well-designed assistant handles it differently:
-
Check authority. Document A is a formal policy owned by the commercial director. Document B is a guide owned by a sales manager. The policy outranks the guide.
-
Check dates. Document B is newer. That raises a real question: did the commercial director approve a change that Document A does not yet reflect?
-
Look for a link. There is no reference in Document B to an approval, and no later version of Document A.
-
Answer with a qualification. “The Commercial Policy (v3, effective 1 January, owned by the commercial director) says extra free days need the director’s written approval. The Sales Team Guide (March) mentions up to five days for good payers, but it is not a policy document and does not cite an approval. Until this is confirmed, treat extra free days as needing approval. This conflict has been flagged to the policy owner.”
-
Escalate. The assistant logs the conflict and notifies the commercial director, who either updates the policy or asks for the guide to be corrected.
The executive gets a usable, safe answer in seconds. The business gets a list of contradictions to resolve.
The labels every source needs
Before connecting documents to an internal assistant, label each one with:
-
Owner: the person accountable for its content, by role and name.
-
Authority level: for example, board or director policy, department SOP, team guide, reference or training material.
-
Effective date, and an expiry or review date if it has one.
-
Scope: which branches, countries, customer types or teams it applies to. A UAE rule and an India rule can both be correct.
-
Status: current, superseded, or draft. Superseded documents should either be removed or clearly marked so the assistant does not treat them as live.
A decision rule for conflicting answers
Give the assistant this order of reasoning:
-
If one source has higher authority and covers the same scope, use it, cite it, and mention the other only if it might mislead.
-
If both have the same authority, use the one with the later effective date, and cite both.
-
If a lower-authority source is newer and contradicts a higher one, give the higher-authority answer, state the conflict, and flag it to the owner.
-
If the question involves money, safety, a customer commitment or a legal term, and the conflict is not settled by steps 1 to 3, do not answer the substance. Route to the owner.
-
If the user does not have access to a source, do not reveal its contents to settle a conflict. Tell them who to ask.
Test the assistant against a set of deliberately conflicting documents before rollout, and check every answer against this rule.
Where SolvTree fits
This maps to our internal AI assistants: assistants that answer staff questions from your own documents, with source labels, citations, access boundaries and an escalation path when sources disagree. For more on how these work, see our post on the internal AI assistant. We start with one question area and its authoritative sources.
Next step: Bring one internal question your team asks often to a free 30-minute operational audit, and map with us where its authoritative answer lives. The report is yours to keep.
Frequently asked questions
- How should an internal AI assistant handle conflicting documents?
- Compare each source’s authority, scope, status and effective date. Cite the governing source and flag unresolved conflicts to its owner rather than choosing whichever document sounds more confident.
- Does a newer document always override an older policy?
- No. A newer team guide may lack authority to change a formal policy. State the conflict, use the applicable higher-authority source where appropriate, and ask the policy owner to resolve it.