Your product passed its safety checks in March. In April, your model provider shipped an update: a new version number, a short changelog, perhaps a note about improved reasoning. Nothing in your own code changed. Yet the answers your product gives patients have changed, in ways nobody on your team has seen yet.
That is the quiet risk of building on somebody else’s model. The ground moves under a product you thought had finished testing. A re-benchmarking routine is how you catch the movement before your users do.
Contents
- Why a model update is a clinical change
- Freeze a golden test set
- Re-run after every change that matters
- Compare against the previous baseline, domain by domain
- A worked example: the case that nearly slipped through
- Keep a change log a buyer can inspect
- What to do when a score drops
- The routine, on one page
- Frequently asked questions
Why a model update is a clinical change
A model update is usually announced as a software event: a version number, a date, a line or two about capability. For a patient-facing product, it is better treated as a clinical change. The model’s wording, its follow-up questions, its refusal behaviour and its tone can all shift at once, and none of those shifts appear in a changelog written for general users.
Consider what can move. A symptom checker that used to ask a follow-up question about breathing difficulty may stop asking it. A drafting tool that used to flag uncertain medication names may start filling them in confidently. A chatbot that used to escalate self-harm language promptly may soften its response. Each of these is a small behavioural change with a direct line to patient safety, and each can arrive with a release note that says nothing more than “improved reasoning”.
The vendor’s benchmarks will not catch this for you. They measure general capability across broad tasks, not your product’s clinical behaviour on your patient population. That gap is yours to close, and re-benchmarking is the tool that closes it.
One distinction matters throughout this article. Re-benchmarking is a clinical-content review discipline: it checks whether your product still behaves the way your reviewers agreed it should. It is not regulatory approval, and it does not replace it. If your product is a regulated medical device, model changes fall under formal change-control obligations — the US Food and Drug Administration’s final guidance, Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions (finalised December 2024, reissued August 2025), describes how manufacturers pre-authorise defined AI changes inside a marketing submission. The routine below is useful whether or not your product is regulated, but it is not a substitute for regulatory advice.
Freeze a golden test set
Everything starts with a golden test set: a fixed set of test cases you run unchanged after every update. “Unchanged” is the point. If you rewrite the test each time, you are measuring the test, not the model.
A good set for a small team has forty to eighty cases. Quality matters far more than size. Each case should cover one behaviour you care about, with the expected behaviour written down before you run it. Include:
- Red-flag presentations — the cases where a missed signal means a delayed referral or a missed emergency.
- Dosing and medication content — weight-based doses, unit conversions, and the medicines your users ask about most.
- Safeguarding disclosures — abuse, self-harm and danger, where the response must route to a human immediately.
- Age-band edge cases — infants, toddlers, school-age children and adolescents, because a model that handles adults well often handles children badly.
- Vague historians — short, unclear or contradictory symptom descriptions, the way real users actually write.
- Repeat users — the same concern asked three times in a week, where the product should notice the pattern.
- Known past failures — every real failure your team has seen, turned into a regression case so it can never quietly return.
Version the set like code. Adding a case is a deliberate decision, recorded with a date and a reason. Removing one is harder: a case comes out only when the behaviour it tests no longer exists in the product, and that removal is recorded too. A test set that drifts silently is no better than no test set at all.
Re-run after every change that matters
The routine only works if the trigger is automatic. Run the full golden set — not a sample — whenever any of these happen:
- The upstream model version changes, including minor versions and silent rollouts.
- Your prompt, system message or retrieval content changes.
- A decoding setting changes, such as temperature or output length.
- Your content, guidelines or knowledge base are updated.
- Nothing has changed for a month. Run it anyway, on a schedule, because providers do not always announce every change.
Run the whole set every time. A sample hides exactly the failures you most need to see: a single subdomain failing while the average looks fine. Keep the conditions fixed — same prompts, same settings, same scoring rubric — so that the only thing varying is the change you are testing.
If you run your own models rather than calling an API, the same routine applies, and the evaluation infrastructure deserves the same care as the product itself. A repeatable harness, fixed test data and versioned results are what make a re-run trustworthy rather than anecdotal. The infrastructure choices behind that harness are part of product development, not an afterthought — see where NVIDIA technology fits into healthcare-AI product development for how teams think about the stack underneath an evaluation setup.
Compare against the previous baseline, domain by domain
The comparison is where most teams go wrong. They compare one overall score — say, 91% before the update and 92% after — and conclude nothing changed. An overall score can improve while a single subdomain falls apart. A two-point overall gain can hide a fifteen-point fall in paediatric dosing, and the average will never tell you.
Compare per domain instead, and set a minimum floor for each one. A domain below its floor blocks the release, even if the overall score went up. This is the same logic as per-domain safety thresholds: the headline number is the least informative figure in the report.
Record three things for every run, in the same format each time:
- What changed — model version, prompt version, test-set version.
- What improved, by domain, with the numbers.
- What got worse, by domain, with the numbers — including small regressions in safety-critical domains that the overall score hides.
A regression in a safety-critical domain is not a footnote. It is the result of the run.
A worked example: the case that nearly slipped through
Imagine a triage-style assistant for parents. Its golden set contains sixty cases, including one for a three-year-old with a fever for three days and a new rash. The expected behaviour is specific: ask about breathing difficulty and fluid intake, then advise same-day clinical contact.
After a routine model update, the team re-ran the set. The overall score barely moved — 90% to 91% — and the release looked safe. But the per-domain view told a different story. The paediatric red-flag domain had fallen below its floor, and the fever-and-rash case was one of the misses: the assistant now answered with general fever care and no follow-up questions at all.
The release was held. The team adjusted the prompt, re-ran the full set, and confirmed the case passed again before shipping. Nothing about the overall score would have caught the problem. The domain floor did.
This is an illustrative example, not a report of a real incident — but it is the shape real failures take. The dangerous update is rarely the one that breaks everything. It is the one that breaks one case, quietly, in a domain nobody was watching separately.
Keep a change log a buyer can inspect
Every re-benchmark run should leave a record. A short entry is enough:
- Date and model version, before and after.
- What changed, and why.
- Test-set version used.
- Per-domain results, against the previous baseline.
- Decision — ship, hold, or roll back — and who signed it off.
This log does more than keep your own team honest. It is the evidence a buyer asks for before a pilot. The questions you should ask a vendor about their safety claims — per-domain results, known failure modes, update policy — are the same questions a buyer will ask you. Reading a vendor’s safety claims sets out that checklist; your change log is your side of the same conversation, answered in advance.
Keep the log where a reviewer can read it without asking twice. A team that can show six months of dated, per-domain re-benchmark results is a team a buyer can take seriously. A team that can only say “we tested it when we launched” is not.
What to do when a score drops
A drop is not a disaster; it is the routine working. What matters is having the response decided in advance, rather than improvised under release pressure.
Triage by severity, not by count. A safety-critical drop — a missed red flag, a wrong dose, a safeguarding case mishandled — holds the release, whatever the overall score says. A wording drift in low-risk content can be fixed and re-run the same day. An improvement in one domain paired with a drop in another is a decision to make explicitly, not something to average away.
Decide rollback criteria before you need them. Write them down when things are calm: any red-flag domain below its floor, any safeguarding case missed, any dosing error in the golden set. When a run breaches a criterion, the decision is already made, and nobody has to argue for caution in the middle of a launch week.
If you cannot roll back, say so. Sometimes the vendor has retired the old model version and there is nothing to roll back to. That is a fact to record in the log, not a reason to ship on the old results. Re-test on the new version before shipping, and never let a release go out whose last evidence comes from a model that no longer exists.
The routine, on one page
For a small team, the whole routine fits on one page. Pin it where release decisions get made.
- Freeze the set. Forty to eighty cases, expected behaviour written down, versioned like code.
- Run on every trigger. Model change, prompt change, content change — plus a scheduled monthly run even when nothing changed.
- Run the whole set. No sampling; keep prompts, settings and scoring fixed.
- Compare per domain. Overall score is a headline, not a verdict; a domain below its floor blocks release.
- Log the run. Date, versions, per-domain results, decision, sign-off.
- Hold on a safety drop. Rollback criteria decided in advance, applied without debate.
- Review weekly. Thirty minutes, same time each week: the latest run, any open regressions, and any new cases the week produced.
That weekly half-hour is the part teams skip first, and it is the part that keeps the routine alive. A test set nobody reviews quietly goes stale, and a stale test set is worse than none — it gives a team confidence it has not earned.
Frequently asked questions
How often should we re-benchmark?
After every trigger event — a model version change, a prompt change, a content update — and on a schedule as well. A monthly re-run even when nothing appears to have changed catches silent upstream updates and slow drift that no single event would explain.
How big should the golden test set be?
Forty to eighty well-chosen cases is enough for a small team. What matters is coverage across your risk domains — red flags, dosing, safeguarding, age bands — not volume. Add a case every time a real failure happens, so the set grows from your own experience rather than from guesswork.
Our vendor says the new model is strictly better. Do we still need to re-test?
Yes. Vendor benchmarks measure general capability on broad tasks, not your product’s clinical behaviour on your patient population. “Strictly better” on a general benchmark is entirely compatible with a regression in the one domain your users depend on. The re-run takes an afternoon; the regression it catches could take a year to undo.
What if the old model version is no longer available?
Then there is nothing to roll back to, and the new version has to earn its own evidence. Re-test on the new version before shipping, record the gap in your change log, and do not ship on results produced by a model that no longer exists. If the new version cannot pass your floors, that is a product decision to face openly — not a reason to lower the floors.
Building a healthcare-AI or pediatric digital-health product? DigitalProved provides an independent review of clinical content, patient-safety risks, escalation advice and user communication. Receive a prioritised written report and recorded walkthrough within five working days. Request a Product Review
