How do you vet an AI engineer when you are not one?
Short answer
You do not need to understand the maths to vet an AI engineer well. Ask them to walk through a project that failed and explain why, then have someone technical review a piece of their actual code or a model they built. How well their explanation holds up under simple follow-up questions tells you more than any list of tools or certifications on their CV.
Nontechnical hirers often lean on weak signals because they feel objective. Job titles, tool lists, and phrases like "AI experience" are easy to check and easy to fake. None of them tell you whether the person makes good decisions under uncertainty, which is the actual job.
A better test is to ask about a project that did not go well. Listen for specific detail about what broke, why it broke, and what changed afterwards. Someone who can describe a real failure clearly, including their own part in it, is usually more trustworthy than someone who claims everything they have touched succeeded.
Bring in a second opinion where you can. Even thirty minutes with a technical reviewer, whether a freelance engineer, an advisor, or a developer already on your team, is enough for them to look at real code, a notebook, or a working demo and tell you whether it holds together.
Use a small paid test task rather than an unpaid take-home exercise that drags on for days. Pay for the time. It signals that you respect their time and it gives you a live sample of how they approach a defined problem rather than a polished portfolio piece.
Pay attention to how they talk about risk. Someone who can explain, in plain terms, where a model might be wrong, where data might leak, or where bias might creep in is usually more capable than someone who says the technology can do anything you ask of it.
Ask former clients or employers specific questions rather than general ones. Instead of asking whether the person was good, ask what went wrong on a project and how they responded. Specific answers are far harder to fabricate than a generic reference.
Watch for overclaiming built around model names and framework lists with no ability to explain a decision. If someone cannot say why they chose one approach over another, on a project they claim to have led, that is worth probing further.
No single check proves competence on its own. Combine a failure story, a piece of reviewed work, and a specific reference, and you will have a much clearer picture than any CV alone could give you.
Related questions
What if I do not know anyone technical enough to review the work?
Ask an existing developer on your team even if they work in a different area, most can spot basic quality issues in code. You can also hire a freelance reviewer for a short session, which is usually cheap relative to a bad full-time hire.
Should I ask candidates for a portfolio?
Yes, but ask how much of it was their own work versus a team effort, and what they would change if they built it again now. A confident answer to the second question is a good sign.
Are AI certifications worth anything in this process?
They show effort and some baseline knowledge, but not judgement under real conditions. Weight them lightly against an actual review of past work and a specific reference check.