AI pilot testing: a shadow-mode checklist before go-live

An AI pilot should prove that one defined workflow can be handled to an agreed quality standard with a measurable operational benefit. Before it acts live, run it in shadow mode, compare its proposals with independently reviewed cases and make a written release decision that specifies the permitted scope, human checks and stop conditions.

For a business evaluating an AI implementation partner, this evidence is more useful than a polished demonstration. It shows which work the system can handle, where a person is still needed and whether the remaining review effort makes the project worthwhile.

Define the pilot around one decision

“Help our accounts team” leaves too much room for disagreement about success. “Propose the ledger and party for each imported bank entry, or flag it for a client query” gives the team something it can test.

Write a short scope before building:

• Which inputs the system may read.
• What output it must produce.
• Which decisions it must defer to a person.
• Which systems and records are out of scope.
• Who currently performs and approves the work.
• What benefit would justify continuing.

Keep access narrow enough to support that scope. A classification test does not need permission to post vouchers. A support-draft test does not need permission to send emails. Check the actual tool permissions and downstream connections rather than relying only on an instruction that says “do not act”.

Use offline replay first, then live shadow mode

An offline replay tests historical cases. Give the AI only the information that would have been available when each decision was made, while keeping the agreed answer separate for evaluation. Later notes or outcomes can otherwise make the task artificially easy.

Live shadow mode tests incoming work alongside the current process. The established team continues independently, while the AI records its proposal using the information available at that point. Reviewers then compare the decisions. Its proposals cannot change live records, notify customers or trigger another automation. This stage can reveal timing, access and source-refresh problems that a clean historical dataset misses.

An assisted workflow comes later: a person reads the proposal, checks it and decides whether to use it in the real task. That may be a valuable destination in its own right. It needs a separate time-and-quality test because people behave differently when they start from an AI answer rather than a blank page.

Keep these modes distinct in the project plan. “A human is involved” does not explain whether they are evaluating an isolated test, approving a draft or checking work after an action has already happened.

Build a test set that reflects real work

Use recent cases from the actual workflow, with authorised access and no more personal or confidential information than the evaluation needs. Include ordinary cases, busy periods, exceptions, incomplete inputs and cases that should stop for a human decision.

Have the person responsible for the work agree the expected result before judging the AI output. Where two experienced reviewers disagree, resolve the policy or mark the case as genuinely ambiguous. Forcing one label onto an unresolved business question gives the test false certainty.

Record the source evidence for each label and preserve an untouched set for evaluation. Cases used to adjust prompts, mappings or rules become development examples. Rechecking only those cases shows whether a known mistake was fixed; fresh cases are needed to assess whether the change works more broadly.

Choose the sample with the workflow’s volume, variation and consequences in mind. There is no universal number of examples or pilot duration that proves every business process is ready.

Agree the release gate before looking at the scores

A release gate is the written decision rule for what happens next. It should cover:

1. Output quality: Which fields or decisions must be correct, and how are they checked?
2. Error severity: Which failures are minor, serious or release-blocking?
3. Deferral: Does the system recognise cases that require more information or human approval?
4. Review effort: How long do checking, correcting and handling exceptions take?
5. Access and actions: Can the system use only the permitted data and tools?
6. Recovery: Can the team stop new actions and reconcile anything already done?
7. Approval: Who is authorised to approve each operating mode and scope?

Use thresholds appropriate to the business risk. A single overall accuracy percentage can hide a small number of unacceptable errors. Likewise, a model’s stated confidence is not sufficient evidence that an output is correct.

Hypothetical example: bank-entry classification at a CA firm

Consider a hypothetical chartered accountancy firm in Mumbai that prepares monthly books for small traders. Staff classify bank entries by ledger and party, resolve client queries and post approved vouchers in Tally. The pilot proposes classifications only; it has no posting access.

The following figures are illustrative and are not SolvTree client results or performance benchmarks.

For an initial offline replay, the firm selects 400 historical entries across eight clients. Reviewers establish the expected treatment, including UPI transfers with unclear narrations, refunds, cash deposits and entries requiring a client query. Those labels are fixed before evaluating this run; later outcomes are withheld from the AI’s inputs.

The AI produces:

• 340 proposals that match the agreed treatment.
• 38 minor errors, such as a general expense head where a more specific one is required.
• 18 serious errors, including assigning a payment to the wrong supplier with a similar name.
• 4 classifications for entries that should have been flagged for a client query.

The match rate is 85%. That number alone does not answer whether the system should act. The four unsupported classifications breach the firm’s rule that unresolved entries must be escalated. Wrong-party errors also matter more than minor description edits.

The partners decide against automated posting. They keep the pilot proposal-only, improve the deferral rules and test the changes on a fresh sample. The missed cases remain in the regression set so the same failures can be checked in later versions. After the offline gate is met, a live shadow phase can test incoming entries and the actual data connections without posting anything.

Separately, the firm can test an assisted workflow in which staff review every proposal. That requires measuring the full task; the shadow-mode classification results do not establish a time saving on their own.

Measure the work that remains

Compare the current process with the assisted process on a comparable mix of cases. Record time spent reviewing correct proposals, correcting wrong ones, making client queries and completing the final approved action. Include time needed to find supporting records; a quick-looking answer may still take several minutes to verify.

Also record quality after review. A faster workflow is not an improvement if reviewers miss wrong-party assignments because the suggested answer looks plausible.

Separate recurring operational effort from one-time setup and evaluation work when assessing the business case. Both matter, but they answer different questions: whether the daily workflow improves and whether the project is worth the investment.

Make release a limited, reversible decision where possible

The pilot can lead to further testing, assisted use, a narrow live release or stopping the project. Record the evidence and reasons rather than treating a launch as the only successful outcome.

For any live release, name the workflow version, allowed case types, action limits, required reviews, monitoring owner and stop triggers. Start with the scope the evidence supports. Zero critical errors observed in a finite test does not guarantee that a rare failure cannot happen later.

Test how to stop new actions. Identify which completed actions can be reversed and which require a compensating correction or a person’s intervention. An already-sent message cannot simply be unsent by switching the system off. Recovery should include checking in-flight work and reconciling records so a restart does not duplicate actions.

Re-test when prompts, models, tools, source systems or business rules materially change. Monitor the released case mix as well; a workflow tested on one kind of customer or document may behave differently when the inputs expand.

What to ask an AI implementation partner

A narrow assistive task may suit an existing product with a clear review step. Custom delivery is more relevant when decisions depend on several business systems, unusual exception rules or approval paths that packaged tools cannot represent well.

Ask the provider to include the evaluation set, results by error type, measured review effort, permitted actions and release decision in the handover. Clarify who owns the test cases and who reruns them after changes.

SolvTree’s scoped delivery capabilities cover focused automations and multi-system builds. For an AI pilot, agree the test and release criteria as part of that scope before discussing wider autonomy.

If you are deciding whether to commission a pilot, discuss the decision you want to test with SolvTree. Describe the current process, the mistakes that would be unacceptable and the result that would justify proceeding. That creates a concrete basis for scoping the evaluation.