HAHayat Amin · Operator
Founder Q&A · Updated 2026-08-24

Why Do My AI Pilots Never Reach Production?

AI pilots never reach production because they are scoped as demos instead of as jobs. A demo proves the model can do something. A job has a named owner, real data, a written bar for promotion, and a defined path for what happens when the agent is wrong. Fix the scoping and the same technology graduates in about six weeks: one workflow inside one function, one accountable owner, shadow mode against live data, and the go-live numbers agreed before anybody writes a prompt.

Why companies get this wrong

The mistake is upstream of the technology. A pilot gets funded because someone senior wants to know if AI works here, which is a question with no answer. So the team builds the thing that answers it fastest: a demo on sample data, shown in a room, applauded, and then handed nowhere. Nobody wrote down what the agent would own. Nobody agreed what score meant go live. Nobody named the person who gets paged at 7am when it drafts nonsense. Six months and a few hundred thousand later, the sponsor has moved on and the second pilot inherits the same design.

The tell is in the data. Pilots built on exported sample files pass, and then meet production data and fall over inside a week, because the messy 15 percent that nobody exported is exactly where the work lives. Duplicate vendor records. Three spellings of the same customer. A supplier whose invoices arrive as photographs. The pilot never touched any of it, so the pilot proved nothing about the job.

Hayat Amin, fractional CFO, AI operator, and IP & patent strategist (London, United Kingdom). Hayat Amin builds and runs AI systems for companies whose ai pilots never reach production
Hayat Amin in London. He builds and runs AI agent systems inside companies across London, NYC, and Dubai.

The framework I use with clients

Five steps. Steps one and four are the ones companies skip, and they are the two that decide whether anything ships.

Step one: scope a job, not a capability. Write the pilot as a sentence a department head would recognise. Not "test AI in finance". Instead: "the agent codes and matches supplier invoices under 5,000 pounds, and the financial controller owns the output". One function, one workflow, one named owner with a salary. If you cannot name the owner in the first meeting, stop there. A pilot without an owner has no route into production, because there is nobody whose job improves when it works.

Step two: connect live data in week one, read only. Not a sample export. The real system, with read access and no write permissions. This feels slower and it is the single biggest predictor of whether the pilot survives, because every integration problem, permission fight, and data quality horror surfaces in week one when the budget still exists, rather than in month five when it does not.

Step three: run in shadow mode beside the human. For two to three weeks the agent does the work in parallel and its output goes nowhere. The human does the job as normal. Every case gets both answers logged. This gives you the only evidence that matters at promotion: on 200 real cases, how often did the agent and the human agree, and on the disagreements, who was right. That number is the business case. Nothing from a demo is.

Step four: write the promotion bar before you start. Three numbers, agreed by the owner and the sponsor in week one and never moved afterwards: the agreement rate that means go live, the rate that means kill it, and the maximum acceptable cost per case. My working defaults are 92 percent agreement over at least 200 real cases to promote, below 75 percent to kill, and a cost per case under a fifth of the loaded human cost of the same task. Set these after seeing results and you will move them, because everyone in the room wants the pilot to have worked.

Step five: design the failure path before the happy path. Every agent needs an escape hatch and a paper trail: confidence below a set threshold routes to a named human, every run logs what it saw and what it decided, and any case the agent refuses lands in a queue somebody checks daily. Build this in week two, not after the first bad week. The controls are what let a nervous CFO or a general counsel say yes, and their yes is the actual gate between pilot and production.

Two more things worth budgeting. Running an agent costs 20 to 40 percent of what it cost to build, every year, in monitoring, retuning, and model changes underneath you. And a serious rollout across a 100-person company takes 9 to 12 months of these six-week cycles, one function at a time. Companies that try to do six functions at once get six pilots that all stall at 80 percent.

From my operating seat

Inside one client I run, the rule is that no agent goes live until the person who will own it has personally disagreed with it in writing at least ten times during shadow mode. It sounds pedantic. It is the cheapest quality gate I have found, because it forces the owner to actually read the output rather than approve a dashboard, and by promotion day they know exactly where the thing is weak. Nothing has been rolled back since we started doing it.

Twenty years in the C-suite and three exits taught me the version of this lesson that predates agents. Every system that ever stuck had a person whose life got easier the day it went live, and every system that died had a sponsor whose life got easier the day it was announced. AI has not changed that. It has only made the announcement cheaper to produce, which is why there are so many more dead pilots than there used to be dead software projects.

How long should an AI agent pilot take?

Six weeks to a binary decision. Week one scopes the job and pulls the last 200 real cases. Weeks two and three build against live data in read only shadow mode. Weeks four and five run the agent beside the human on every new case. Week six compares and either promotes or kills. Past ten weeks with no go or no-go, it is not a pilot, it is a hobby with a budget code, and the sponsor stopped watching a month ago.

What does production ready mean for an AI agent?

Four things in writing: an accuracy bar met on at least 200 real historical cases, a named human who owns the output and gets paged when it breaks, a logged trail of what the agent saw and decided on every run, and a documented fallback that returns work to a person without losing it. An agent that is right 92 percent of the time and routes the rest to a named human is production ready. A demo that impresses nine times out of ten is not.

Who should own AI agents once they are in production?

The function that consumes the output. If the agent drafts month end reconciliations, finance owns it and the controller is accountable. IT owns plumbing, access, and security. An innovation team may build version one and must hand over the keys at promotion. Agents left with the team that built them stop improving the day that team starts the next pilot, and that handover gap kills more working systems than weak models do.

Where I come in

This is what I build and run inside companies: the scoping, the shadow runs, the promotion bars, the controls a CFO will sign, and then the agents themselves running in finance, pipeline, onboarding, and compliance as one system rather than a shelf of pilots. If you have spent a year proving AI works here and still have nothing in production, that is a design problem and it is fixable. See how I work as an AI agent operator, or start at meethayat.com.