Designs the tests that say whether a system is good enough to ship.
What separates them
A strong evaluator designs tests that can fail and can tell you exactly what a pass does and doesn't prove; a weak one builds a suite that always comes back green and calls that reassurance.
Ask these
01
How would you design an evaluation to tell us whether a system is fit to ship, rather than one that mainly confirms it works on the cases we already expect?
Tests whether they design for failure modes rather than for confirmation.
A strong answer
They describe building adversarial or out of distribution tests rather than reusing familiar cases, and a fail threshold that was agreed before the evaluation ran, not chosen afterwards to match the result. They can say what actually happens if the system fails that threshold.
The confident failure
They describe a broad looking suite with many metrics but cannot say what a bad score would mean for the shipping decision, or how any threshold was set.
02
What would make you distrust an evaluation suite that always shows the system passing?
Tests understanding that evals can be gamed or made trivially easy, a core insight of the discipline.
A strong answer
They name a specific cause, for example the test set overlapping with data used to tune the system, or thresholds being set only after seeing the results, and give an example of catching this themselves.
The confident failure
They say a consistently passing suite is reassuring and a sign the system is solid, without ever questioning whether the test was hard enough to fail.
03
Tell me about an evaluation you ran that led to a system not shipping, or shipping with changes.
Real evidence of consequence rather than of process alone.
A strong answer
They give a specific finding and the decision that followed it, and can name who they had to convince and how that conversation went.
The confident failure
They describe running evaluations diligently but the finding never actually changed a decision, told as though that is normal, with no reflection on why not.
04
How do you evaluate something subjective, like whether a response is helpful or appropriate, without it just becoming your opinion?
Tests methodological rigour on the hardest category of eval: qualitative judgement.
A strong answer
They describe a written rubric applied by more than one rater, with a check on how much the raters agree with each other. They give an example of a disagreement that led to the rubric itself being revised.
The confident failure
They say they use their judgement and experience, or rely on a single reviewer, treating subjectivity as unavoidable rather than something that can be measured and controlled.
What we ask when assessing for the register
Harder, and answerable only by somebody who has done the work. Published because a question that stops working when it is known was never testing anything.
01
Describe an evaluation suite you built that later turned out to have a blind spot. How did you find out, and what did you change?
Only real practitioners have been caught out this way; tests honesty and depth.
A strong answer
They describe the specific blind spot and how it surfaced, whether through a red team finding or a real incident. They also describe the structural change they made afterwards, rather than simply adding one more test case.
The confident failure
They claim no major blind spot has occurred, or describe a trivial gap caught before any consequence, an implausibly clean record for this kind of work.
02
How do you set a pass or fail threshold for a metric before you have a track record of what safe actually looks like in production?
Tests statistical and judgement rigour under genuine uncertainty, as opposed to setting thresholds to match whatever the system already does.
A strong answer
They describe anchoring the threshold to a real world harm they are trying to avoid, and setting it conservatively at first with an explicit plan to revisit. They also describe getting stakeholder sign off on the risk tolerance behind that number.
The confident failure
They say the threshold was based on an industry standard without being able to say where that came from, or admit the threshold was set after seeing the first results.
Ask these whatever the discipline
Tell me about something you built that failed in production. What broke, how did you find out, and what did you change?
What would you refuse to do on this project, and what would you tell me instead?
How would you know, three months in, that this was not working?
What is the part of your own work that you are least confident about?
Relevant here, and none of them is evidence on its own. What each does and does not prove is set out in full on the certifications page.
Advanced in AI Audit, AAIA, ISACA. Technical depth. An auditor who can interrogate a model risk register is not the same as somebody who can tell you whether the evaluation set was any good.
AI Security Management, AAISM, ISACA. Hands-on adversarial capability. Managing a security programme and breaking a model are different jobs.