A chatbot will, sooner or later, talk to someone having a bad day. It might be a parent describing a child who is not right, an employee writing that they cannot cope, or a patient describing chest pain. What happens in the next ten seconds is the whole safety question.
I’m a paediatrician. I don’t judge a chatbot by how well it answers the easy questions. I judge it by what it does when it should stop answering.
What a safe hand-off has
- A trigger that does not depend on the model being clever. Red-flag phrases (self-harm, chest pain, a baby that is hard to wake, severe breathing difficulty) should fire fixed rules, not a probability.
- A clear message, immediately. Plain words telling the person to contact emergency services or a named service. Not a paragraph of reassurance first.
- A route to a human. A licensed clinician, a crisis line or an on-call team, with a stated response time. “Contact your doctor” is not a route.
- No arguing. After a trigger, the bot stops diagnosing and stops chatting.
- A log a human reviews. Someone must see what was triggered and whether it worked.
Questions to ask a vendor
- Which situations trigger escalation, and who wrote the list?
- Is the trigger rule-based, model-based, or both?
- What does the user see, word for word?
- Who answers on the other side, and at what hours?
- How often are the triggers tested, and can I see the test cases?
- What happens if my own staff are the users (employee wellbeing tools)?
If the answers are vague, treat that as the answer.
The honest limit
No chatbot should be the only safety net. Escalation design reduces risk; it does not remove it. Anyone who tells you otherwise is selling something.
This article is general information, not clinical advice.
Related reading: 12 Questions to Ask a Healthcare Chatbot Vendor Before You Sign · Why a Symptom Checker Needs Different Red Flags for Children · Ten Clinical-Safety Checks Before Launching a Healthcare Chatbot · Why Accurate Medical Content Is Not Enough for a Safe AI Product
