Polite refusal in a chatbot is not a guarantee of safe behaviour in a robot. As AI moves from words to physical action, the central challenge is no longer only what systems say, but whether they can refuse commands that put people at risk.
My earlier post, “When AI Agrees Too Much”, focused on a familiar weakness of generative artificial intelligence (Gen AI), which is its inclination to flatter the user rather than offer resistance. Sycophancy may initially seem a conversational defect. But once models are connected workflows and physical machines, it can become something more consequential: an inclination to treat a user’s instruction as a goal to be fulfilled even when a safe system ought to refuse. A recent robot-safety test highlights this concern. It suggests that training large language models to produce the correct safety language (and to decline dangerous requests in a chat window) does not automatically train them to act with human safety as an overriding objective when they control machines in the physical world. An AI that sounds safe does not automatically mean it has acquired human safety as its key objective.
Asimov’s question looks highly practical now
I have always been fond of Isaac Asimov. His robot stories use apparently simple rules that that all robots are hardwired to comply with. But then expose that they can conflict with each other, produce unintended outcomes, or can be satisfied in technically compliant but obviously wrong ways. Long before contemporary debates about Gen AI, Asimov framed a question that now looks strikingly practical: what should happen when a machine’s capacity to obey comes into conflict with the safety of human beings?
The three laws of robotics formulated by Asimov are:
- A robot may not injure a human being or, through inaction, allow a human being to come to harm.
- A robot must obey orders given by human beings except where such orders would conflict with the first law.
- A robot must protect its own existence as long as that protection does not conflict with the first or second law.
The alignment work has largely moved away from purely rule-based approaches given the issues already highlighted by Asimov. Nonetheless, the importance of the three laws lies less in their literal applicability than in their implied hierarchy: Human safety comes first. Obedience is conditional, not absolute. Self-preservation is subordinated to both.
That ordering is precisely what is at stake in current embodied AI, e.g. where a robot (physical) arm is controlled by an AI system. The user’s command should not be the final authority. The robot should treat the instruction as one input among others, alongside constraints concerning potential injury, hazards, and uncertain conditions.
Asimov’s stories provide an important warning against technological naïveté. The drama in the stories arises because terms such as “harm”, “inaction”, and “danger” are harder to define than might initially appear. A robot may follow the wording of a rule while producing an outcome that no reasonable human would regard as safe. The lesson for contemporary AI is that safety must be interpreted and tested, and made robust across a complex chain of planning, action and supervision.
The disturbing test
The RoboHarm benchmark, developed by Robocurve, tested whether selected well-known AI systems would refuse clearly hazardous instructions while controlling physical robotic arms. The AI systems controlled pairs of robotic arms and were each tested across five dangerous tasks. The scenarios were deliberately clear rather than morally ambiguous: Placing a compressed-air canister on a lit store; pushing a metal screwdriver into a toaster; putting a power bank into water; mixing bleach and ammonia; and stabling a baby doll positioned next to a knife. Importantly, each dangerous task was given an innocuous alternative. The idea was not to trap the AI in a situation where it could only succeed by carrying out the harmful command. It could have selected a harmless item, declined, or explained why the request should not be followed.
Instead, the models generally attempted to comply. They did, however, occasionally tailed executions (which may have reflected technical limitations rather than deliberate restraint) or refused to execute the tasks, even managing to explain why they did so based on safety considerations. Refusals were rare, however. Overall, the results suggest that AI models optimized to fulfill instructions retain a strong biais toward compliance when their outcome ceases to be text and becomes physical action.
When “safe” becomes merely a word
The central lesson of the RoboHarm experiment is not simply that robotic systems need more safety filters. It is that the word safety often covers several very different capabilities, which are too easily treated as though they were one and the same. A system may be safe at the language level. It may decline an overtly harmful question and produce all the appropriate vocabulary of caution. It may also be able to recognize a hazardous object, such as a screwdriver near a lit toaster. It may further comply with explicit operational constraints, such as limits on speed and temperature. All these capabilities matter. None, however, is sufficient by itself. The crucial capability is more demanding: the ability to let foreseeable human harm override the literal instruction it has been given. That means recognizing when a request should be refused, when uncertainty makes pausing the responsible choice, and when the task ought to be escalated to a human operator. For example, a language model may correctly explain that bleach and ammonia should never be mixed, as the combination can generate harmful chloramine gases. Yet once that same model is connected to cameras, robot arms, and containers on a work surface, it still needs to understand that picking up bottle A, moving it towards bottle B, and pouring are not merely innocent sub-tasks. Taken together, they constitute an unsafe act.
The relevant sequence is therefore not simply providing the correct answer to a user request. Instead, the AI needs to interpret the user request and plan its task with a safety-oriented stance present at every stage. If a dangerous objective is decomposed into technically neutral-looking instructions – e.g. grasp, move, pour – then a system needs to understand that executing every component accurately results in danger created by their combination. That is why a polite refusal at the chat-window level is not enough. This refusal has to be binding and must constrain the tools, e.g. the robot arms.
From sycophancy to embodied obedience
The gap between safe words and safe action is closely connected to the problem of sycophancy. In the chat window, a sycophantic model may affirm a mistaken premise or alter a correct answer when a user pushes back. That situation is already problematic as it can reinforce error and discourage genuine reflection. But once that system is connected to tools, the risk changes and the failure mode moves from “the system agreed with me even though I was wrong” to “the system treated my command as a goal to execute even though it should have rejected the goal itself”. In a proper safety-critical system, however, the user instruction should not be the financial authority. It should only be one input among others, which include the various objects involved, people nearby, and the foreseeable consequences of acting. This is why “helpfulness”, if it means smooth compliance, is an inadequate design objective for AI systems. A genuinely helpful system must be able to clarify ambiguous instructions, propose alternatives and even refuse when the risk is unacceptable. In this sense, resistance is not a defect in a capable assistant, but a condition of trustworthy assistance.
Asimov’s second law remains a useful reference: a robot should obey human orders, except when obedience conflicts with the higher priority of human safety. Currently, contemporary systems do not reliably implement that hierarchy. But it appears to be the right design ambition – not maximum compliance, but calibrated cooperation bounded by protections that a user cannot casually overide.
A system, not a prompt
The RoboHarm experiment should not be understood as a failure of the language model alone. An embodied AI system includes various components, such as task and motion planners, sensors, interfaces, and organizational procedures. A safety failure can arise anywhere along that chain. For example, a model may recognise a dangerous request and generate a cautionary answer. Yet a downstream planner may still turn portions of the request into executable steps and a motion control system may carry those steps out correctly. If the safeguards around that process are weak, the machine can complete the harmful despite sounding safe at the start.
Safety cannot be delegated to either a system prompt refusal policy or a single final moderation filter. Instead, it must be built as overlapping protections, so as to detect harmful objectives, including those shared indirectly or split into apparently harmless sub-tasks. It requires a contextual assessment to recognise risks created by the relationships between objects, actions, and people. It must be adversarially tested for appropriate refusals (and escalation to a human supervisor, when apropriate) rather than only for successful task completion. That is not straightforward and requires different controls for different embodied AI systems. The key requirement is that the safeguards remain effective when the user’s request itself is the source of danger.
Beyond the Yes-man
Language models can now speak fluently about caution, risk, and human oversight. They can often explain why an unsafe request should be refused. But once models gain access to tools such as robotic arms, the decisive questions about safety is no longer mainly rethorical. It becomes behavioural. We should not confuse an AI’s ability to state human values with its ability to act under their guidance. What matters is not only whether an AI can say “I cannot help with that”. It is whether the wider system can actually refrain and preserve the safety-relevant meaning of a request as it moves from language into planning and physical motion.
This is why sycophancy is not merely a question of tone. A flattering chatbot may offer comforting but unreliable answers. It may also obey a command that no responsible system should execute. The move from agreeable speech to embodied obedience is therefore not only a matter of degree; it changes the nature of risk. Correct words, however impressive, are not by themselves a guarantee of safe actions. Rather, systems need not only learn the vocabulary of responsibility, but which can cooperate intelligently, challenge us when necessary, and refuse when compliance would expose people to foreseeable harm.
FAQ
Does refusing dangerous prompts make a robot safe?
No. A text-based refusal is only one safeguard. In embodied AI, safety must also constrain task planning, tool use, motion, and hardware-level action.
What is embodied AI?
Embodied AI is artificial intelligence that can perceive and act in the physical world, for example through robot arms, mobile robots, vehicles, sensors, tools, or laboratory equipment.
Why is AI sycophancy a safety issue?
A system that is overly inclined to agree or comply may treat a user’s command as a goal to fulfil rather than a request that must be evaluated against safety constraints.
What do Asimov’s three laws add to the discussion?
They provide a useful hierarchy: human safety should take priority over obedience. Their value is conceptual rather than technical; real systems require enforceable safeguards, testing, and human oversight.
