Most AI projects don’t fail because the model was wrong. They fail because the team built the right model for the wrong problem.
I see this every few months. A company spends three months building a document classifier. It hits 94% accuracy in testing. Then they ship it, and nobody uses it — because the actual bottleneck was never classification. It was getting the documents into the system in the first place.
The week-one decision
In the first week of any AI project, someone decides what the AI is going to do. That decision sounds obvious. It rarely is.
The mistake is starting from the technology instead of starting from the outcome. “We’ll use an LLM to summarize support tickets” is a technology-first decision. “We want first-response time under 10 minutes” is an outcome-first decision. Only the second one tells you whether the AI is actually solving anything.
A summarization model that saves 30 seconds per ticket does nothing for first-response time if the tickets are still sitting in a queue for four hours.
What actually gets in the way
Three patterns show up repeatedly:
Optimizing the wrong metric. Accuracy, F1, BLEU — these measure the model. They don’t measure whether the problem is solved. I’ve seen 98% accurate models that were useless and 78% accurate models that saved companies millions. The metric you train to isn’t always the metric that matters.
Skipping the process audit. Before you automate something, you need to understand it. What does the manual process actually look like? Who touches it? What are the edge cases that would never show up in a demo? The answer is almost always more complicated than the person who scoped the project said it was.
Building for the best case. AI in demos works on clean, well-formatted, representative data. Production data is none of those things. If you don’t know what happens when your model sees something it wasn’t trained on, you find out after launch.
The fix
The fix is boring: start with the outcome, not the tool. Before writing a line of code, be able to answer:
- What decision will this system help someone make?
- How will we know if it’s working?
- What does it do when it’s wrong?
If you can’t answer those three questions clearly before you start, you’re not ready to start.
The model is almost never the problem. The problem is almost always upstream.