Home » Founder Guides » Reading a Vendor’s Safety Claims: Questions to Ask Before Integrating a Health AI Tool

Founder Guides

Reading a Vendor’s Safety Claims: Questions to Ask Before Integrating a Health AI Tool

Elgaraali

October 30, 2026

3 Views

SaveSavedRemoved 0

• Introduction
• Why safety claims are so hard to read
• Decoding the common claims, line by line
• The evidence to request before you sign
• Red-flag phrases that should slow a deal down
• The 10-question checklist — send it as it stands
• What to do with the answers
• Frequently asked questions

Introduction

Sooner or later, a vendor sends you a deck with a slide titled “Safety”. It carries an accuracy figure, a compliance badge or two, a row of hospital logos, and a sentence about being clinically validated. It looks like evidence. Often, it is not — or not the kind of evidence that protects you when the tool meets your users.

That distinction matters because integration moves the risk onto you. Once a vendor’s model sits inside your product, your users experience its failures as your failures. A parent who receives a wrong escalation suggestion does not care which company built the model underneath. The regulator, the buyer, and the journalist will ask you the same question: what did you check before you integrated it?

This guide shows how to read a vendor’s safety claims line by line, what evidence to request before signing, and which phrases should slow a deal down. It ends with a ten-question checklist you can send to a vendor as it stands.

Why safety claims are so hard to read

Three things make vendor safety language slippery.

First, there is no single standard label for “safe” in health AI. A vendor can be accurate, compliant, cleared, and widely used — and each of those words can be true while saying something much narrower than it sounds. Accuracy on one dataset is not safety across your population. Compliance with a data-protection law is not proof that clinical advice is sound. Clearance for one intended use does not cover a different one.

Second, most safety claims are written by people whose job is to close deals. That does not make them dishonest, but it does mean the claim is framed in its most flattering light: the best dataset, the easiest case mix, the most generous definition of success. Your job is to ask for the version of the number that survives contact with your users.

Third, the evidence that matters most is often the evidence a vendor is least keen to share: where the tool fails, on whom, and how often. A vendor that answers those questions plainly is telling you something valuable. A vendor that will not answer them is also telling you something valuable.

Decoding the common claims, line by line

“95% accurate” — accurate at what, on whom?

A headline accuracy figure without context is close to meaningless. Before it impresses you, find out four things: what the model was predicting, whose data it was tested on, which cases were included, and how performance breaks down by subgroup. A model that is 95% accurate overall can still fail badly on one slice that matters to you — paediatric cases, rare presentations, or patients whose first language is not English. Ask for results broken down by clinical domain and by population, not just the headline number. If the vendor cannot provide a breakdown, treat the headline as marketing, not evidence.

“Clinically validated” — validated by whom, and where can you read it?

“Validated” is doing a lot of work in this sentence. It can mean anything from a peer-reviewed study in a real clinical setting to an internal test run by the vendor’s own team. Ask who did the validation, on what population, against what reference standard, and whether the results are published somewhere you can read them. An internal white paper is not worthless, but it is not the same as independent evaluation, and it should not be presented as though it were. Be especially careful when validation was done on adults and your users include children: performance does not transfer across age groups by default.

“FDA cleared” or “CE marked” — cleared for what, exactly?

Regulatory clearance is real evidence, but it is narrow evidence. In the United States, most AI-enabled devices reach the market through the 510(k) pathway, which requires showing the device is substantially equivalent to an already-marketed device for the same intended use — the FDA maintains a public list of authorised AI-enabled devices, and the large majority were cleared this way. That tells you something important about the device for its stated intended use. It does not tell you the tool is safe for your different use case, your population, or your integration. A clearance for triage support in adult emergency departments says nothing about a paediatric symptom checker built on top of the same model. Always ask for the exact intended use on the clearance, and check whether your planned use sits inside it. If it does not, you are not covered by that clearance at all, and you need specialist regulatory advice — this article is not that advice.

“HIPAA compliant” or “GDPR compliant” — compliant is not the same as safe

Data-protection compliance is about how personal data is handled: encryption, access control, retention, breach notification. All of it matters, and none of it says anything about whether the clinical advice the tool produces is correct. A tool can protect patient data impeccably and still give dangerous advice. Treat compliance claims as a data-governance check, and keep them in a separate column from clinical safety. Do not let a compliance badge answer a clinical question.

“Used by leading hospitals” — used how, and by whom?

Logos on a slide are not evidence of clinical performance. Ask whether those hospitals are paying customers or pilot sites, whether the tool is live in clinical workflows or sitting in an evaluation sandbox, and whether anyone at those organisations will speak to you as a reference. Anonymous logos and unnamed “leading hospitals” cost nothing to print. A vendor with genuine clinical traction can usually arrange a reference conversation. One that cannot should make you pause.

“Powered by advanced AI” — words that carry no claim at all

Some phrases sound technical and commit to nothing: “AI-powered”, “cutting-edge”, “next-generation”, “intelligent”. They describe no measurable property and cannot be verified. When a claim cannot be falsified, it is not evidence. Cross it out and move on to the claims that can be checked.

The evidence to request before you sign

When you move past the deck, ask for six things. A serious vendor will have most of them ready; a vendor that stalls on all six is giving you your answer.

1. Results by domain and population, not just overall. Per-domain performance, including paediatric subgroups if children are anywhere near your user base, and the size of the test set behind each figure.
2. A failure-mode catalogue. The known ways the tool gets things wrong, with examples, and what the vendor has done about each one. Every real product has failure modes. The absence of a list is itself a finding.
3. Data provenance. Where the training and test data came from, what populations they represent, and what is known about their gaps. A model trained on one population and sold into another is a foreseeable failure, not an accident.
4. A model-update policy in writing. What changes without notice, what triggers re-validation, and how you are told. Upstream model updates can quietly change clinical behaviour, so this is not paperwork — it is a safety control.
5. The human-oversight design. Who reviews flagged cases, how quickly, and what they can actually do about them. If the answer is “the clinician is in the loop” with no roster, no queue, and no escalation path, there is no loop. Our guide to human oversight in healthcare AI sets out what that chain should look like: https://digitalproved.com/human-oversight-healthcare-ai/
6. Incident history and disclosure practice. Has the tool caused or contributed to a safety incident, and how did the vendor tell customers? You are not looking for a perfect record. You are looking for honesty and a process.

Red-flag phrases that should slow a deal down

The 10-question checklist — send it as it stands

Copy these into an email to the vendor. Their answers, and how willingly they give them, are the real due diligence.

1. What is the tool’s performance broken down by clinical domain and by age group, including children where relevant, and how large was each test set?
2. What are the tool’s known failure modes, and what has changed in the product as a result of each one?
3. Where did the training and evaluation data come from, and which populations are under-represented in it?
4. For exactly which intended use is the tool cleared or certified, and does our planned use sit inside that intended use?
5. What changes to the model, prompts, or data trigger re-validation, and how and when will you notify us?
6. Who reviews flagged or low-confidence cases, within what time frame, and what action can they take?
7. What happens when the tool is uncertain — does it abstain, hedge, or answer anyway, and what does the user see?
8. Has the tool been involved in a safety incident, and what is your disclosure process when one occurs?
9. Can we run our own test set, including edge and paediatric cases, in a pilot before committing?
10. Can you put us in touch with two current customers using the tool in a comparable setting?

What to do with the answers

Score the answers, do not just collect them. A vendor that answers eight questions well and is honest about two gaps is a better bet than a vendor that answers all ten with slogans. Watch for evasiveness patterns: answers that restate the question, figures without denominators, and references to evidence that cannot be produced.

Then put the important answers in the contract: update notification windows, audit and testing rights, and an exit route if performance on your population falls below an agreed floor. A pilot in shadow mode — the tool runs, its outputs are logged, but users do not see them yet — lets you test on your real case mix before anyone’s care depends on it. Use your own test set for that pilot, including the awkward cases: vague symptom descriptions, anxious repeat users, and the presentations your population actually brings. Infrastructure choices deserve the same discipline: our guide to where NVIDIA technology fits into healthcare-AI product development makes the same point about platforms — a strong stack underneath does not validate the clinical layer on top: https://digitalproved.com/nvidia-healthcare-ai-product-development/

Finally, keep two kinds of review separate. An independent review of clinical content, patient-safety risks, escalation advice, and user communication can find weaknesses and prioritise fixes before launch. Regulatory assessment is a different process, done against a different standard, and neither substitutes for the other. A team that keeps those categories separate will read vendor claims more clearly too, because it will stop expecting one kind of evidence to do another kind’s job. Our guide to preparing a prototype for an independent clinical review sets out the evidence reviewers expect to see assembled: https://digitalproved.com/healthcare-ai-prototype-review-prep/

Frequently asked questions

Is a vendor’s accuracy figure enough to judge safety?

No. Accuracy is a single number averaged over a test set you did not choose. Safety depends on which cases fail, on whom, and what happens next. Ask for per-domain results, failure modes, and the oversight design around the tool — those three together say far more than any headline figure.

Does FDA clearance mean a tool is safe for my product?

Clearance covers a specific intended use. If your use differs — a different population, a different clinical task, a different setting — the clearance does not extend to you. Check the exact intended use on the clearance, and get specialist regulatory advice if your use sits outside it.

What is the single most revealing question to ask a vendor?

“Show me where it fails.” A vendor with a real failure-mode catalogue and a record of fixing what it found is demonstrating a safety culture. A vendor that claims not to fail is demonstrating something else.

Should I trust a vendor that will not share test data?

Some data genuinely cannot be shared, for privacy or commercial reasons. But safety evidence — aggregate results by domain, failure descriptions, validation summaries — can almost always be shared in some form. A blanket refusal to share any safety evidence is a reason to walk away, not a reason to trust harder.

How do I handle a vendor whose claims look strong but cannot be verified?

Run a pilot on your own terms: shadow mode, your own test set, your own case mix, and a written agreement about what happens if performance falls short. Unverifiable claims are a reason to test harder, not a reason to sign faster.

Building a healthcare-AI or pediatric digital-health product? DigitalProved provides an independent review of clinical content, patient-safety risks, escalation advice and user communication. Receive a prioritised written report and recorded walkthrough within five working days. Request a Product Review