How to decide whether to keep paying for an AI pilot

The four numbers that matter

Cash. Money that actually leaves or enters the business differently because of the pilot: a recovered payment, a cancelled subscription, overtime no longer paid. If you cannot point to a ledger entry or an invoice, it is not cash.

Capacity. Hours of staff time no longer spent on the task. Useful, but only worth money if those hours go somewhere valuable or allow you to avoid a hire.

Review cost. Hours people spend checking the AI’s output. Almost every serious pilot needs some review, especially early. It must be counted.

Rework. Hours spent correcting outputs that were wrong, plus any downstream cleanup. This is the number most pilot reports leave out.

First calculate net capacity freed: baseline time minus all ongoing task time, including review and rework. To estimate economic benefit, add cash recovered or avoided to the value of net capacity you can genuinely redeploy, then subtract pilot fees and other costs not already counted. Do not subtract review and rework twice, or count the same benefit as both cash and capacity. For a fuller framework on return, see our post on AI automation ROI. This post is about turning that framework into a decision.

An illustrative scorecard: OTA payout reconciliation at a hotel

Consider a hypothetical 60-room hotel in Jaipur that sells through two online travel agencies (call them OTA A and OTA B) as well as direct. Each month, an accounts executive matches OTA payout statements against bookings in the property management system, checking commission, cancellations and taxes. The hotel ran an eight-week pilot in which an AI matched statement lines to bookings and flagged mismatches for review. All figures below are illustrative assumptions.

Baseline, measured before the pilot: about 12 hours a week on reconciliation, and the executive believed some commission overcharges were being missed.

Pilot cost: ₹55,000 a month, so ₹110,000 for the two months.

Results over eight weeks:

• OTA A: of roughly 400 statement lines, about 370 matched automatically and correctly. The flagged lines included several commission charges on cancelled bookings, which the hotel disputed and recovered: about ₹38,000.

• OTA B: its statements changed layout mid-pilot. Of roughly 250 lines, about 120 were matched wrongly and had to be fixed by hand.

• Review time: about 3 hours a week checking flags across both.

• Rework: about 4 hours a week, almost all on OTA B.

• Reconciliation time: down from 12 hours to about 7 hours a week including review and rework.

Reading the scorecard:

• Cash: ₹38,000 recovered, all from OTA A, against ₹110,000 of pilot cost. Not yet covered.

• Capacity: about 5 hours a week freed. The executive used part of it to chase older receivables, which has some value but no figure yet.

• Review cost: included in the remaining seven hours. Measure whether it falls on OTA A rather than assuming it will.

• Rework: high, and concentrated in one input.

A blended view says “partial success, unclear payback”. The separated view says something sharper: the pilot works well on OTA A and badly on OTA B.

Decision: test a narrower scope before committing to another paid period. OTA A is a candidate because matching was reliable in this sample and recoveries were found. Obtain the price for OTA A alone, estimate repeatable recoveries and redeployable net capacity, and compare them with all ongoing costs. Continue only if that forward-looking case or a bounded learning budget is acceptable. Pause OTA B until its input handling is corrected and re-tested. A further eight-week test is a proposal, not an automatic spending commitment.

Watch the confounders before celebrating. The recovered ₹38,000 may include a backlog that will not repeat every month. A seasonal dip in bookings would also make the hours look better than they are. Note both in the decision record.

A decision rule you can use

Score the pilot on each number, by input type where results differ, then apply this:

1. Keep: cash benefits plus the value of net redeployable capacity clearly exceed pilot fees and other costs not already counted, and quality is stable across inputs.

2. Narrow: results are good on some inputs or tasks and poor on others. Price the narrower scope and check its expected benefit before continuing it; drop or re-test the rest.

3. Pause: results depend on something outside the pilot that is about to change (a supplier format, a system migration, a policy). Stop paying until it settles.

4. Stop: after a fair run, review and rework cost as much as the time saved, and no cash benefit has appeared.

Before the next pilot, write down three things in advance: the baseline and how it was measured, the sample size you need before judging, and which result would trigger each of the four decisions.

Where SolvTree fits

This maps to our scoped engagements: pilots with a measured baseline and success criteria agreed before the build. We agree what success means before building: one workflow, one baseline and one decision rule. Commercial terms are set in the project agreement.

Next step: Bring one pilot, running or planned, to a free 30-minute operational audit and choose a baseline and a decision criterion with us. The report is yours to keep.