Key points
- Choose one number that matters to the business, and measure it against a baseline recorded before the pilot starts.
- Write the decision rules in advance: what result means go, what means stop, and who makes the call.
- Keep the pilot small, time-limited (30 to 60 days) and run on real data, so the result means something.
Why pilots drift
AI pilots rarely fail in the demo. They fail afterwards, when the team is asked whether to continue and nobody can answer clearly. The common causes are familiar: no baseline, so improvement cannot be shown; several goals, so every result looks like a partial success; no named owner; and a timeline that quietly stretches until the enthusiasm has gone.
A pilot is a decision tool. If it cannot end in a clear yes or no, it has not been designed properly.
Choose one number
Pick the single measure the business cares about most, and make sure it can be measured from data you already have. Examples by business type:
- Manufacturing: lost minutes per week on the pilot line, or OEE against its baseline.
- Distribution and supply: expedited orders per month, or on-time, in-full delivery from the pilot suppliers.
- Field service: first-time fix rate, or overtime hours per technician.
Other measures can be tracked as secondary checks, but one number decides the outcome.
Set the baseline before you start
Record at least four weeks of performance on the chosen measure before the tool is switched on. Use the same definitions you will use afterwards. A baseline recorded after the pilot starts is almost always contested.
An example (illustrative numbers)
A field service company records a first-time fix rate of 72% over the four weeks before a dispatch pilot. It agrees three things in writing:
| Rule | Agreed threshold |
|---|---|
| Go: continue and expand | First-time fix rate at or above 78% over the last 30 days of a 60-day pilot |
| Stop: end the pilot | Below 72% for two consecutive weeks, or any customer commitment missed because of a suggestion |
| Decision owner | Operations director, with the dispatch lead consulted |
With rules like these, the end of the pilot is a short meeting, not a debate.
Write the decision rules in advance
For each outcome, write down what it means and what happens next. Include a stop rule. A pilot where stopping is not an option is not a pilot, and a good vendor should accept a stop rule without difficulty.
Questions to ask any vendor
- How will you measure the number we agreed, and what data do you need access to?
- What is the baseline you would expect us to record, and who checks it?
- What happens to our data if we stop at day 30?
- What results have you seen before, and how were they measured? (Ask for the method, not only the headline.)
Our pilot process is built around this approach: a paid, fixed-scope pilot on your real data, judged against a number agreed up front. If you want to explore whether a pilot fits your business, tell us about the problem and we will reply within two business days.