How do you tell a good AI engineer from a confident one?
6 minute read. Updated 2026-08-08.
Confidence is not evidence of competence, and a hirer who cannot assess the technical work directly needs a different signal. The strongest one is whether a candidate can describe something that failed in production and exactly what they changed afterwards. A good engineer gives you a specific failure, a specific fix and what they'd still do differently. A confident one gives you a general story about success, or blames the data, the users, or the deadline.
Confidence measures how someone talks, not how good the work is
Most hirers judging an AI engineer end up judging how someone talks, not what they've built. That's the wrong filter, because confidence in an interview says little about the quality of the underlying work, and it's the signal most available to someone without the technical depth to check further.
The fix is not to distrust confidence outright. Some genuinely excellent engineers are also fluent talkers. The fix is to stop treating fluency as evidence and to ask questions that only someone who has actually done the work can answer well.
The single strongest question
Ask a candidate to describe something that failed in production, and what they changed afterwards. This one question does more work than any personality read, because it cannot be answered well from theory. It requires the candidate to have shipped something real, watched it break, and made a specific decision in response.
A strong answer names the failure precisely: a model that returned the wrong customer's record, a pipeline that silently dropped a data source for 3 weeks, an evaluation set that didn't cover the case that actually broke. It names the fix, and it usually names something the candidate would still do differently given the chance again.
A weak answer stays general. 'We had some data quality issues', with no named cause, no named fix, and no admission that a decision made by the candidate contributed to it, is the answer of someone who has either not shipped much, or has shipped it and is not willing to own the part that went wrong.
Other probes worth using
Ask how they would validate a model before it goes live, and listen for a specific method rather than a general claim of rigour. A concrete answer names a held-out test set, a defined threshold, a named failure mode they check for. A vague answer talks about 'making sure it's accurate' without saying how.
Ask about a time they said no to shipping something the business wanted. This tests judgement under pressure, which matters more in AI work than in most engineering, because the failure mode of a bad AI system is often confident and plausible rather than obviously broken. An engineer who has never refused a deployment has either always been right, which is unlikely, or has never been in a position to refuse.
Weaker signals worth noticing but not leading with include fluency with the newest tools and frameworks, which shifts too quickly to be a good hiring filter, and fluency in the vocabulary of the field, which correlates with reading rather than building.
What monitoring reveals
Ask what they monitor once a model is live, and what they check for drift. This question separates people who think about a model as a one-off build from people who think about it as something that degrades over time and needs watching. A good answer names specific metrics tracked after launch, not just accuracy at the point of testing.
If the answer is that monitoring happens but nobody has ever acted on an alert, that is worth probing further. Monitoring that never triggers a change is not really monitoring, it is a dashboard nobody reads.
Why the interview alone won't get you there
Even a well-run set of questions has a ceiling, because a fluent candidate can prepare good answers to predictable questions. Where the hire matters enough to justify it, pair the interview with a small, paid piece of representative work, time-boxed to a day or two, and look at what they actually produce.
The point of the paid task is not to get free work out of a candidate. It is to see whether the judgement described in the interview shows up in the artefact: does the code handle the edge case they said they'd check for, does the evaluation actually test what they claimed it would test.
What I would do
Lead every technical interview for an AI hire with the production failure question, before anything about tools, frameworks or model choice. The answer tells you more in 5 minutes than the rest of the interview combined, because it cannot be faked convincingly by someone who hasn't done the work.
Follow it with the validation and monitoring questions, and where the hire is significant, back all of it with a small paid task rather than relying on conversation alone. This applies whether the hire is permanent, fractional or contract; the failure question works the same way regardless of the commercial structure around it. Confidence will still be in the room either way. The job is to make sure it isn't the thing you're actually measuring.
What to do about it
- Lead the interview with a description of a production failure and what they changed.
- Do not treat fluent, confident answers as evidence of competence on their own.
- Ask for a specific validation method, not a general claim of rigour.
- Ask about a time they refused to ship something, to test judgement under pressure.
- Back a significant hire with a small paid task rather than conversation alone.
- Treat familiarity with the newest tools as a weak signal, not a strong one.
Questions people also ask
What if a candidate says they've never had anything fail in production?
Treat that as a warning sign rather than a good answer, unless they are very early in their career and say so plainly. Anyone who has shipped AI systems for several years has watched something go wrong: a model drifting, an evaluation missing a real-world case, a pipeline failing silently. An experienced candidate who claims a spotless record is either not being honest about the work, or hasn't shipped enough of it to have hit the failure modes that come with real production use.
How do I assess this if I'm not technical myself?
You don't need to judge the technical detail of the answer, only whether it is specific. A real failure has a name, a cause, and a fix that changed something concrete. A vague answer stays at the level of 'data quality' or 'stakeholder alignment' with nothing you could point to and check. That distinction is available to a non-specialist hirer without needing to evaluate the engineering itself.
Is a paid task fair to candidates?
Yes, provided it is scoped tightly, time-boxed, and either paid or built from work you would have needed doing anyway rather than being disguised free consultancy. A day or two of representative work, paid at a fair rate, gives a far better read on judgement than another round of conversation, and most experienced candidates will accept a well-structured version of this without objection.
Should I ask for references instead of running these probes myself?
References are worth taking up, but they answer a different question: whether past clients were satisfied, not whether the work was technically sound. A reference given by someone who liked working with the candidate will not tell you whether the evaluation approach they used was rigorous. Use references to check reliability and conduct, and use the probes in this piece to check the engineering judgement itself.
Where the figures come from
Every rate and salary quoted in this article is a median or percentile of figures advertised in UK job postings over the six months to 8 August 2026. They are not rates paid, and the gap widens at the top of a range.
- IT Jobs Watch, UK contract rates, 6 months to 8 August 2026, read 2026-08-08.
- IT Jobs Watch, UK permanent salaries, 6 months to 8 August 2026, read 2026-08-08.