Engineering

Every robot demo shows what happens when the AI works. Nobody demos what happens when it doesn't.

3 min readMati Melchior
Every robot demo shows what happens when the AI works. Nobody demos what happens when it doesn't.

Every robot demo shows what happens when the AI works. The LiDAR returns clean data. The model infers correctly. The robot picks up the object, places it precisely, and the audience applauds.

Nobody demos what happens when the AI doesn't work.

On 19 July 2025, a Unitree H1 suspended from a crane during a private test began thrashing at full joint torque, collapsing the stand and sending technicians backwards. What makes this one useful is that the operator, Cix Liv of REK Robot, published the root cause himself: "we ran a full body policy while the feet weren't touching the ground." Not emergent behaviour, not a mystery. A learned whole-body control policy was activated in a physical state it was never trained for, and nothing in the stack checked that precondition before letting it command torque. The machine did exactly what it was told, at full power, in a world that did not match its assumptions.

The pattern repeats at much larger scale, and the regulatory record now documents it. In June 2026 Waymo filed a recall covering 3,871 vehicles because its driving system could enter freeway construction zones at speed — in the agency's words, by "inappropriately prioritizing the avoidance of other freeway hazards and/or failing to recognize the construction zone." In one set of events the vehicles drove past ramp-closure signs; in another, seven drove between cones marking a lane closure. In July 2026 Zoox recalled 105 robotaxis after one drove into an active fire scene it could not see through heavy smoke. Nothing in either system knew that it had stopped perceiving correctly.

The most instructive case is the one that was already fixed once. In late 2025, Waymo vehicles were observed passing stopped school buses with red lights flashing and stop-arms extended — twenty incidents logged by Austin ISD, six in Atlanta. Waymo ran live training sessions with actual school buses and shipped software updates. Five further violations occurred in November 2025, after those updates, and a recall followed in December covering 3,067 vehicles. Retraining the model did not close the failure mode. That is the clearest public argument there is for a deterministic layer that sits outside the learned policy and cannot be outvoted by it.

IEEE Spectrum recently published research on a related failure pattern: AI systems that fail catastrophically without crashing. No error codes. No alerts. No stack trace. The system keeps running, dashboards stay green, but behavior quietly drifts from what was intended. For safety-critical physical systems, this kind of silent degradation is the most dangerous failure mode — because nobody knows it's happening until someone gets hurt.

The test that matters for Physical AI isn't performance under ideal conditions. It's behavior under failure. What does the robot do when the LiDAR returns garbage data? When the force-torque sensor saturates? When the model crashes mid-motion?

IEC 61508 calls this the "safe state" — a defined, tested, documented response to detectable failures. If a vendor can't answer what their robot does when the AI stops working — in one sentence — the safety claim is aspirational, not engineered.

Share

Physical AI Safety Dispatch

Monthly analysis. No spam. One exclusive insight per issue.

One issue per month. Unsubscribe in one click from any email. Privacy policy.

We use cookies

This site uses essential cookies to function and, with your consent, analytics cookies (Google Analytics) to understand how the site is used. Learn more.