Most AI projects that fail do not fail because the model was too weak. They fail because the team could not reach the data the model needed, nobody agreed on what a good answer looked like, and the impressive demo quietly fell apart the moment it met real users and real edge cases. The model is now the easy part. Everything around it is the hard part, and that is where budgets get burned.
The model became a commodity, the plumbing did not
A few years ago, picking and training a model was genuinely difficult work. Today you can reach a capable general model through an API in an afternoon, and open-weight models are strong enough that the choice between them rarely decides whether a project lives or dies. When I review a stalled AI initiative, the model is almost never the bottleneck. The bottleneck is the unglamorous work nobody scoped: getting clean, current, permissioned data into the system, and knowing whether the output is actually correct.
This is the reframe that saves projects. Stop treating the model as the centre of gravity. Treat it as one interchangeable component inside a data and evaluation problem. If you would not know how to answer "is this output good?" with evidence, no model will rescue you.
The real blockers
Data you cannot reach
The single most common killer is data access. The knowledge your AI needs lives somewhere painful: a legacy database nobody owns, a SaaS tool with a rate-limited API, PDFs scanned in 2011, a warehouse where the meaningful columns are undocumented, or a system where the data exists but you are not allowed to use it for this purpose. Teams routinely design a solution assuming the data is a clean, queryable feed, then discover it is none of those things.
Before you commit to an approach, answer these plainly:
- Where does the data physically live, and who controls access to it?
- Is it fresh enough? A model answering from a nightly export is useless for a question about something that changed an hour ago.
- Are you legally and contractually allowed to send it to the model you plan to use?
- How much cleaning, deduplication and structuring stands between the raw source and something usable?
If any of those answers is unknown, that uncertainty is your project risk, not the model.
Evaluation nobody defined
The second killer is that no one has defined what success means in measurable terms. "It should give good answers" is not a specification. When there is no agreed definition of a correct output, every review meeting becomes a matter of taste, the system cannot be improved deliberately, and you have no way to know if a change made things better or worse. You are flying blind, and blind projects drift until the budget runs out.
A usable definition of good is concrete: for this class of input, here is the expected behaviour, here is what counts as a failure, and here is roughly how often we can tolerate failure. If you cannot write that down, you are not ready to build. I go deeper on how to build this in how to evaluate an LLM system you cannot see inside.
Use cases that demo but do not survive production
The third killer is seductive. A demo is a controlled performance on inputs the builder already knows the system handles. Production is an adversarial environment of typos, ambiguous requests, out-of-scope questions, hostile users, and the long tail of cases you never imagined. A system that scores well on ten hand-picked examples can be embarrassing on the eleventh real one.
The gap between demo and production is not a polish problem you fix at the end. It is the actual work. If a proof of concept only proves the happy path, it has proven very little about whether the thing can ship.
A demo answers "can this ever work?" Production answers "does this work when I am not watching?" Those are different questions, and only the second one matters once money is on the line.
How to de-risk before you commit budget
You do not need a big programme to find out whether an AI project is viable. You need to attack the riskiest unknowns first, cheaply, before the expensive commitment.
- Prove data access before anything else. Spend the first days getting a real, representative sample of the actual data out of the actual source and into a form you can query. If this is hard, you have just learned the most important thing about your project, and you learned it for the price of a few days rather than a few months.
- Write the evaluation before you build the system. Assemble a set of real inputs with agreed correct outputs. This forces the definition-of-good conversation to happen early, while it is cheap to change direction, and it gives you a scorecard for every later decision.
- Test on adversarial and edge inputs, not curated ones. Deliberately feed the messy, ambiguous, out-of-scope cases. The failure rate there predicts production reality far better than any polished demo.
- Scope the smallest slice that delivers real value. A narrow use case that genuinely works beats a broad one that half-works everywhere. Narrow scope also shrinks the data and evaluation surface, which is exactly where the risk lives.
Done in this order, you spend a small amount to retire the two risks most likely to kill the project: can we reach the data, and can we tell if the output is any good. Only after those are answered does model choice, prompting and architecture deserve serious budget.
The uncomfortable summary
If your AI project is in trouble, the honest first question is almost never "did we pick the right model?" It is "can we actually reach the data this needs, and have we defined what a correct answer looks like?" Get those two right and the model becomes a straightforward, swappable choice. Get them wrong and no model on the market will save you.
If you are weighing up an AI initiative and want a clear-eyed read on where the real risk sits, that is exactly the conversation to have early. You can book a free consultation or look at how we approach AI development.