Builds the systems that put models into production and keeps them there.
What separates them
A working demo proves nothing; what matters is what they did the first time it broke under real load.
Ask these
01
Tell me about a system you built that broke in production. What actually went wrong, and what did you do?
Reveals whether they have operated systems at real scale or only built things that worked once, in a demo.
A strong answer
Names a specific failure mode such as latency, memory or drift, walks through the diagnosis, the fix, and what monitoring changed afterwards.
The confident failure
Describes the incident in vague terms, such as it not scaling, without any detail on how it was diagnosed or what specifically was changed.
02
How do you decide when to build something custom versus use an existing model or service?
Tests engineering judgement on build versus buy, rather than a default preference for building.
A strong answer
Gives concrete criteria such as latency requirement, cost per call, data sensitivity and control needs, and describes a real decision made on that basis.
The confident failure
Says it depends on the use case, without naming any concrete criterion that actually drove a past decision.
03
What does your process look like before you put a model change into production?
Reveals whether there is real rigour or a habit of shipping and watching what happens.
A strong answer
Describes offline evaluation, a staged rollout, a rollback plan, and names the specific metrics they watch during rollout.
The confident failure
Says they test thoroughly, without naming what is actually tested or how a rollback would work if the change went wrong.
04
Tell me about a time a model's behaviour degraded over time without any obvious code change.
Tests understanding of drift and production data quality, a common and often invisible failure mode.
A strong answer
Describes specific monitoring for distribution shift, the root cause analysis process used, and a concrete example of what was found.
The confident failure
Says they retrain regularly, offered as a blanket fix with no account of how the problem was diagnosed in the first place.
What we ask when assessing for the register
Harder, and answerable only by somebody who has done the work. Published because a question that stops working when it is known was never testing anything.
01
Walk me through the on-call incident that taught you the most about how your system actually fails, as opposed to how you designed it to fail.
Only genuine production experience produces an answer where the root cause differs from the original assumption.
A strong answer
Gives a specific timeline, a root cause that turned out to be different from the initial theory, and a concrete change made to prevent it happening again.
The confident failure
Describes an incident fluently but the resolution is generic, such as adding more monitoring, without naming a specific new check or process.
02
Tell me about a time you pushed back on a deadline because a system was not ready, and what happened next.
Tests judgement under delivery pressure, and whether they have ever actually been overruled, since never being overruled is itself a signal.
A strong answer
Names the specific technical risk, describes the real conversation with stakeholders, and is honest about the outcome including if it shipped anyway and went wrong.
The confident failure
Says they always insist on quality, without being able to describe a specific instance of friction or its real outcome.
Ask these whatever the discipline
Tell me about something you built that failed in production. What broke, how did you find out, and what did you change?
What would you refuse to do on this project, and what would you tell me instead?
How would you know, three months in, that this was not working?
What is the part of your own work that you are least confident about?
Relevant here, and none of them is evidence on its own. What each does and does not prove is set out in full on the certifications page.
AWS Certified Machine Learning Engineer, Associate, Amazon Web Services. That the holder has run anything at scale, or that they can tell a good architecture from a defensible one. It is scoped to one cloud, and the exam rewards recognising the AWS answer rather than the right answer.
Professional Machine Learning Engineer, Google Cloud. Anything about judgement on problems Google Cloud has no service for, which is most of the interesting ones.
Azure AI Engineer Associate, AI-102, Microsoft. That the holder can evaluate whether a retrieval system is returning the right thing. Assembly and evaluation are different skills, and the second one is the one that is scarce.
Deep Learning Institute certifications, NVIDIA. Independent judgement about whether the workload needed that stack in the first place.