Builds agentic workflows and retrieval systems that return the right thing.
What separates them
Retrieval quality is measurable; the practitioners worth hiring can produce the numbers, and almost nobody else can.
Ask these
01
What numbers do you look at to know whether your retrieval system is returning the right thing?
This is the central test of the discipline, since most people describe retrieval quality without ever measuring it.
A strong answer
Names specific metrics such as precision or recall against a labelled set, and describes concretely how that evaluation set was built.
The confident failure
Says they monitor quality closely, or cites user complaints as the main signal, without any labelled or measured basis.
02
Tell me about an agent workflow you built that you had to constrain or simplify because it was unreliable.
Tests real experience with agent failure modes rather than theoretical enthusiasm for autonomy.
A strong answer
Names a specific failure such as looping or wrong tool choice, describes the constraint added, and how they verified it actually helped.
The confident failure
Describes an impressively complex agent architecture with no mention of any failure or constraint ever being needed.
03
How do you decide when retrieval is the right approach versus fine-tuning or a larger context window?
Tests architectural judgement rather than a default reach for whichever tool is popular.
A strong answer
Gives concrete trade-offs such as data freshness, cost, latency and traceability, and describes a real decision made on that basis.
The confident failure
Says retrieval is generally better because it stays up to date, with no nuance and no real decision behind it.
04
How do you know when your system is confidently returning the wrong answer, and what do you do about it?
Tests whether they have built evaluation for silent failure, not just for errors that are obviously visible.
A strong answer
Describes a specific detection method such as grounding checks or a labelled test set, with a concrete example caught this way.
The confident failure
Says the system cites its sources so users can check, offered as the entire safeguard with nothing else behind it.
What we ask when assessing for the register
Harder, and answerable only by somebody who has done the work. Published because a question that stops working when it is known was never testing anything.
01
Walk me through how you built the evaluation set you used to measure retrieval quality. Where did the labels actually come from?
Separates people who have built real evaluation infrastructure from those who cite the idea in the abstract.
A strong answer
Describes a concrete labelling process, such as domain experts reviewing real query logs, and acknowledges its known limitations.
The confident failure
Says they used a model to judge relevance, offered as the whole method with no validation of that judge's own accuracy.
02
Tell me about an agent that took an action it should not have, and how you found out.
Tests whether they have actually operated agentic systems with real consequences, not just prototypes in a notebook.
A strong answer
Gives a specific incident, the detection mechanism, and a concrete change made to permissions or checks afterward.
The confident failure
Says they always have a human in the loop, offered as a blanket safeguard with no real incident or gap ever described.
Ask these whatever the discipline
Tell me about something you built that failed in production. What broke, how did you find out, and what did you change?
What would you refuse to do on this project, and what would you tell me instead?
How would you know, three months in, that this was not working?
What is the part of your own work that you are least confident about?
Relevant here, and none of them is evidence on its own. What each does and does not prove is set out in full on the certifications page.
Azure AI Engineer Associate, AI-102, Microsoft. That the holder can evaluate whether a retrieval system is returning the right thing. Assembly and evaluation are different skills, and the second one is the one that is scarce.