Most AI pilots don't fail because the model isn't clever enough. They fail because nobody decided what would be different on a Tuesday afternoon once the pilot finished. The technology was demonstrated, the slide deck was approved, and then the work carried on exactly as before, with an extra tool nobody had a reason to open.
That's the pattern we keep meeting. There's no dramatic crash, just a slow drift back to the old way of doing things. The pilots that survive tend to be the ones that treated the pilot as a change to someone's job and judged it by whether that job got easier. Model choice and budget matter far less.
Why is the demo the wrong finish line?
A pilot usually starts with a question about capability. Can it summarise these documents? Can it draft a reply to this complaint? Can it turn last month's figures into a board note? Those questions are easy to answer and almost useless, because a demo shows the best case: a clean input, a cooperative example, an audience that wants to be impressed.
The work itself is the ordinary case. A messy input on a busy day, handled by someone who has eleven other things to do and no time to fight a tool that's nearly right.
So the pilot produces a convincing demonstration and no change. The demonstration wasn't faked. The tool really did summarise the documents. What nobody established was whose afternoon that summarising used to fill, what that person now does with the time, and what happens on the day the summary is wrong.
Signs a pilot is going nowhere
Nobody owns it. "The AI working group" isn't an owner. An owner is one named person who can say yes, can say no, and will be asked about it in three months. When responsibility is spread across a committee, nobody defends the pilot when the first awkward result appears.
There's no baseline. If you can't say what the task costs today, you can't show that anything improved. Teams measure after the pilot, find there's nothing to compare against, and the result becomes an opinion. The fix is dull but effective: count something before you start. Hours per week, revisions per proposal, days from enquiry to quote. Use records you already keep.
The tool sits outside the work. A separate tab, a separate login, a separate thing to remember. Every extra step is a reason to skip it on a busy day, and most days are busy. This is the most common way a technically successful pilot fades out.
Success is defined as usage. Logins, messages, active users. Those numbers rise during a pilot almost regardless of value, because people are curious. A pilot that reports usage has measured attention, and attention wears off.
What the pilots that work do differently
The successful ones are narrower than their sponsors wanted, and that's deliberate.
They pick one workflow. "AI for the sales team" is a platform ambition. "The first draft of the weekly pipeline note" is a pilot. It's narrow enough that you can tell whether it worked, and small enough that a failure doesn't embarrass anyone. If you're unsure how to scope it, getting the best from an agent sets out the routine.
They name a number before they start. Usually one the business already tracks, so nobody has to invent a measurement system. Time to first response. Edits per document. Hours spent on a recurring report.
They review failures every week. The useful artefact is a list of what came back wrong and why. That list turns into rules, examples and corrections, which is how output improves over a month. It's the same discipline as correcting your agent. A team that only reviews successes learns nothing and repeats the same mistakes politely.
They put the output where the work already happens. If the draft arrives somewhere someone already checks, it gets used. If it arrives somewhere new, it gets forgotten. That sounds trivial, and it decides more pilots than model choice does.
They accept a boring result. The winning pilot often automates a task nobody would put in a case study: reformatting supplier notes, turning a recorded handover into a checklist, drafting the same six paragraphs every month. Repeated and measurable is where the return comes from.
Most pilots should probably be stopped
The numbers aren't flattering. McKinsey's State of AI survey, published in November 2025, found 88 per cent of respondents saying their organisations use AI in at least one function, but only 39 per cent reporting any enterprise-level effect on earnings, and nearly two thirds not yet scaling it across the business (McKinsey).
In June 2025 Gartner predicted that over 40 per cent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls (Gartner).
We wouldn't treat either figure as precise. What they agree on is direction, and the direction is that most of this work isn't landing.
The conclusion we draw is unpopular with sponsors: stopping is a result. A pilot that ends with "this task isn't worth automating, and this is what we learned about why" has done its job. The failure is the pilot that never ends and never scales, absorbing a little attention every month and producing nothing anyone can point at.
Isn't experimenting worth something?
Yes, and the case for it is serious. Pilots are cheap options on an uncertain future. Demanding a hard business case before anyone understands the tool is how organisations kill useful experiments early. Capability has arrived in steps, and the firms that had already played with the previous step were the ones ready to move when the next one landed. On that view, a spread of scrappy pilots is rational, and the failure rate is the price of keeping options open.
Measurement is also harder than it looks. The headline figures come from surveys and forecasts, not audited results. In a 2025 randomised trial, METR found experienced developers were 19 per cent slower with the AI tools of the time while believing they were faster (METR). In February 2026 METR said developers are probably sped up by current tools, but that its newer data is too skewed by who agreed to take part to say by how much (METR update). That's a good warning about self-reported productivity, and about how fast this ground moves.
So we'd defend exploration. We'd just call it that. Exploration is timeboxed, promises no return and ends in a decision. A pilot with a sponsor, a budget and a slide deck is claiming more, and should be held to the claim.
Before you start the next one
- Name one workflow. Write it as a sentence: "the first draft of the monthly client report". If it takes more than a sentence, it's too big.
- Write down today's number. Time per run, cost per run or error rate. One number is enough.
- Name one owner. A person who'll be asked about it in ninety days.
- Decide what happens to a wrong output. Who catches it, how quickly, and what the fallback is. If the answer is "someone will notice", you're not ready.
- Set a stopping date and condition. "If the number hasn't moved by 12 December, we stop." This keeps the pilot from becoming permanent by default.
- Book fifteen minutes a week to review failures. Bring the list of what came back wrong and leave with one change to the instructions. A recurring prompt like this is what scheduled check-ins are for.
If you haven't chosen a workflow yet, start with how AI use differs from an AI agent before you commit budget to either.
Common questions
How long should an AI pilot run?
Usually six to ten weeks, long enough to see the task repeated several times. If the workflow happens weekly, eight weeks gives you eight data points: enough to spot a trend, and not so few that one good month fools you.
Do I need to calculate a return on investment?
Not a formal one. You need one number measured the same way before and after. A finance-grade model is often impossible for a task nobody currently times, and pretending otherwise produces fiction.
What if leadership wants a company-wide rollout?
Ask what would have to be true for the rollout to work, then test that in one team first. If you can't describe how one team's week changed, you've got nothing to roll out, only something to install.
Who should own a pilot?
Someone who does the work, or manages the people who do, and has the authority to change the process. IT can install the tool, but only an operational owner can change the job around it, which is what decides whether anything improves.