Why Software-Only Safety Has a Ceiling

Software safety has a ceiling. Let's talk about where it sits and why it matters.

Most of the Physical AI conversation right now assumes the safety problem is a software problem. Better detectors. Better model alignment. Better runtime monitors. Better formal methods. All of that work is useful. None of it breaks through the ceiling.

What software does well

Software is extraordinary at a specific set of tasks, and those tasks matter for safety. It monitors state and logs it. It compares inputs against expected ranges and raises alerts. It retries operations that failed. It fails over from one subsystem to another. It implements defensive behaviour like rate-limiting, bounds-checking, and policy enforcement across a changing system.

For the operational envelope — inside the conditions the software was written and tested for — these capabilities are exactly what you want. The failure modes inside the envelope tend to be the ones software can detect and respond to, because software was designed around them.

What software cannot do

What software cannot do is guarantee a physical outcome under worst-case conditions that weren't part of its design.

This sounds like a philosophical point. It is actually a very practical one, and it is the point that functional safety as a discipline has been organised around for forty years.

Consider what a safety guarantee actually requires. You have to show that under every combination of conditions the machine will encounter — not just the ones you tested — the dangerous outcome will not occur. For mechanical systems we've known how to do this for a century: you use materials with known strength, geometries with known tolerances, and you over-design by a factor that accounts for everything you haven't thought of. The guarantee is physical, and it remains true whether the computer is on, off, or confused.

For software, the equivalent is far harder. Software doesn't fail the way materials fail — it fails by doing something its author didn't imagine. Every large software system contains behaviour that no one predicted, because the combination space is larger than any test suite. That's not a criticism of any particular developer. It's a property of the medium. And it's why the functional-safety standards — IEC 61508, ISO 13849, IEC 62061 — all converge on the same architectural move for the highest integrity levels: independence. The safety guarantee has to come from somewhere the normal operational software can't reach.

Patches don't close the gap. They move it.

There's a response to this that goes roughly: "We can just patch the software. When we find a failure mode, we close it."

It's a reasonable response. It's also what every embedded software team has been doing for thirty years. The problem is that patches only close the failure modes you've already discovered. The failure modes you haven't discovered yet — which, for a learning system operating in an open world, is the interesting set — remain open. And for a machine that physically interacts with people, the cost of discovering a failure mode in the field is not another line item in a bug tracker.

Patches are part of the toolkit. They are not the guarantee. The older safety-critical industries worked this out the hard way.

The forty-year lesson

Aviation, nuclear, and rail did not arrive at their safety postures by theorising. They arrived at them after decades of accidents, investigations, and hard arguments about what would have prevented each one. The documents we now call IEC 61508, DO-178C, EN 50128 — these are the written-down versions of those arguments.

The lesson, stated as plainly as one can state it: for the parts of a system where failure has catastrophic consequences, the layer that guarantees against failure must be different from, and independent of, the layer that does the work. Different hardware where the risk justifies it. Different people designing each side. Different review tracks. At the highest integrity levels, diverse redundancy — two implementations that could both be wrong but are unlikely to be wrong in the same way at the same time.

In functional-safety terms this shows up as the Safety Integrity Level, or SIL. There are four SILs, one through four, corresponding to progressively stricter requirements on the probability of dangerous failure per hour of operation. Getting above SIL 2 effectively requires architectural independence — you can't get there by writing better code on the same box. At SIL 3 and SIL 4, independence is no longer a design choice. It's the thing being measured.

ISO 13849, the machinery-specific standard, uses a different but philosophically-aligned system based on Performance Levels and Category 3 and 4 architectures. IEC 62061 is the machinery-specific derivative of IEC 61508. The language differs; the architectural implication is the same. High-integrity safety comes from independence, not from better code on the same compute.

What comes next

Physical AI in 2030 will look different from Physical AI in 2020 — not because the models got smarter, although they did. The difference will be architectural. The machines that remain on the market, and specifically on the EU market after 20 January 2027, will be the machines whose safety case is a coherent functional-safety argument: independent safety logic, independent hardware where the integrity level demands it, deterministic failure modes, and documentation that an external assessor can actually evaluate.

The ones that don't get there won't necessarily fail loudly. They'll just quietly stop being placeable on the market, stop being insurable at reasonable rates, and stop being integrated by the serious buyers. The commercial force that will drive the change is not a sudden regulatory surprise. It's the steady, unglamorous work of procurement, certification, and insurance catching up to what the engineering has been doing.

This is the conversation the industry needs to have in public, and the reason I'm writing these posts. The software-only-safety position was a reasonable place to start twenty years ago. It is not a defensible place to land.