Why AI agent pilots stall before production

In June 2025 Gartner predicted that over 40 percent of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear business value and inadequate risk controls. That August, research from MIT's NANDA initiative reported that most enterprise generative AI pilots showed no measurable effect on profit and loss.

Our experience matches the pattern. The agents that stall are rarely limited by the model. They are limited by everything around it.

The four gaps

1. No one can show it works. A pilot is judged on a demo and a few anecdotes. Without a labelled test set drawn from real cases, every change is an argument, and nobody can sign off on a launch.

2. The agent can do too much. It holds a broad API key, writes directly to the system of record and decides consequential things inside the prompt. Security reviews, reasonably, say no.

3. No one can see what it did. When a customer complains, there is no trace of what the agent read, what it decided and why. Without that, there is no fixing it and no defending it.

4. No one can stop it. There is no console, no override and no clear escalation. People who are accountable for the outcome have no lever, so they do not let it run.

The proposal: models suggest, code decides

The single design choice that closes most of these gaps is a hard boundary between judgement and action.

We built two open-source agents around it. In an AI sales agent that qualifies leads over chat, email and voice, the language model produces only a score and a reason. Whether a lead passes, and whether a meeting is booked, is decided by plain TypeScript against a threshold, after an injection check. In an autonomous HR agent, the model reads a leave request written in plain English and extracts the dates; a policy engine of pure, tested functions checks balance, notice and blackout dates and makes the call. Anything outside policy goes to a person, and the agent says why.

From that boundary, the rest follows:

• Evaluation becomes possible. The model's job is narrow, a score or an extraction, so it can be tested against labelled examples. The decision logic is ordinary code with ordinary unit tests; the HR agent has 95 of them over the logic that decides things about people.
• Permissions shrink. The model needs no write access. Code holds the credentials and acts only through typed functions with limits.
• Every step is logged. Each run stores the input, the model's output and reasoning, the rule that fired and the action taken.
• People can intervene. A console shows every live case with the model's reasoning, and one click overturns any decision. Overrides become new test cases.

What this looks like in practice

1. 01Pick one decision that happens often and costs real time.
2. 02Collect a hundred or more real past cases with the right answer, and make that the evaluation set before writing a prompt.
3. 03Give the model the narrowest job that helps: classify, score, extract or draft.
4. 04Write the decision as code with explicit thresholds, and route everything uncertain to a person.
5. 05Ship to a slice of real traffic with the console open, and widen as the numbers hold.

The result is less impressive in a demo and much more likely to be running a year later.

What to measure

• Agreement with experts. How often the agent's decision matches what an experienced person would have done, on the evaluation set and on live samples.
• Override rate. How often people overturn the agent, and why. A falling rate with stable quality is the path to wider rollout.
• Escalation rate. The share of cases routed to people; too high wastes the agent, too low hides risk.
• Time and cost per case. Against the process it replaces, measured on the same cases.

How we would start

We begin every AI agent engagement by finding that one decision and building its evaluation set in the first week. For agents already in a pilot, our LLMOps and guardrails work adds the tracing, tests and console they are missing. The same boundary is also the first defence against prompt injection.