Adapts existing models to a specific domain, and knows when not to.
What separates them
The hardest and least visible skill in this discipline is saying no to a fine-tune when prompting or retrieval would do the job better.
Ask these
01
Tell me about a project where you recommended against fine-tuning, even though the client or team wanted it. What did you suggest instead and why?
This is the discipline's hardest to fake skill: refusing the obvious approach.
A strong answer
They give a specific example and name the actual constraint, such as insufficient quality data or a problem better solved with retrieval or prompting. They also describe the alternative that was actually adopted.
The confident failure
They say they always assess the options first, in general terms, with no specific case where they said no to a fine-tune, or claim fine-tuning is usually the right call given enough data.
02
How do you know when a fine-tuned model has actually improved, rather than just looking different?
Tests evaluation discipline around adaptation specifically, rather than general model evaluation.
A strong answer
They describe a held out test set that wasn't used in tuning, and compare the tuned model against the base model and against a cheaper alternative, such as prompting, on the same task with the same metric.
The confident failure
They describe a qualitative look at a handful of outputs as sufficient evidence of improvement, with no held out comparison or baseline.
03
What can go wrong after you fine-tune a model that wasn't a problem with the base model?
Tests understanding of the actual risks of adaptation, such as catastrophic forgetting or overfitting to a narrow style.
A strong answer
They name a specific failure mode concretely, for example loss of instruction following outside the tuning set, and describe how they tested for that specifically rather than only for improvement on the target task.
The confident failure
They give a vague answer about overfitting without describing how they would detect it, or claim fine-tuning only ever improves the target behaviour with no side effects.
04
How do you decide how much data is enough before starting a fine-tuning project, and what do you do if there isn't enough?
Tests practical judgement about data sufficiency, a common project failure point.
A strong answer
They describe piloting on a small subset first to check whether there is a usable signal before committing to a full run. If the data is insufficient, they describe a credible fallback, such as prompting or retrieval, rather than pushing ahead regardless.
The confident failure
They give a specific number as a universal rule regardless of the task or model size, or say more data is always better without addressing what happens when data is scarce.
What we ask when assessing for the register
Harder, and answerable only by somebody who has done the work. Published because a question that stops working when it is known was never testing anything.
01
Describe a fine-tuning run that didn't work. What was the actual cause, and how did you diagnose it rather than just retrying with different settings?
Separates people who understand the mechanism from people who tune knobs until something works.
A strong answer
They give a specific diagnosis, for example noisy labels or a mismatch between the tuning task and the real task, and describe a diagnostic process such as inspecting failure cases by category rather than looking only at the aggregate metric.
The confident failure
They describe a failed run resolved by adjusting the learning rate or the number of epochs, without ever describing what the actual underlying problem was.
02
Tell me about a case where fine-tuning worked technically, met all your metrics, but you still advised against shipping it. What was the gap between the metric and the real requirement?
Tests whether they can distinguish a metric passing from a business or safety requirement actually being met.
A strong answer
They give a specific example and name the gap explicitly, for example that the metric didn't capture a rare but severe failure mode, or that the model degraded on inputs outside the tuning distribution that would be common in production.
The confident failure
They say this has never happened, that metrics and shipping decisions have always aligned, which is implausible given how narrow most fine-tuning metrics are.
Ask these whatever the discipline
Tell me about something you built that failed in production. What broke, how did you find out, and what did you change?
What would you refuse to do on this project, and what would you tell me instead?
How would you know, three months in, that this was not working?
What is the part of your own work that you are least confident about?
Relevant here, and none of them is evidence on its own. What each does and does not prove is set out in full on the certifications page.
AWS Certified Machine Learning Engineer, Associate, Amazon Web Services. That the holder has run anything at scale, or that they can tell a good architecture from a defensible one. It is scoped to one cloud, and the exam rewards recognising the AWS answer rather than the right answer.
Professional Machine Learning Engineer, Google Cloud. Anything about judgement on problems Google Cloud has no service for, which is most of the interesting ones.
Deep Learning Institute certifications, NVIDIA. Independent judgement about whether the workload needed that stack in the first place.