blog

kernels that change while running

Every serious robot I have worked on ships with a frozen kernel, and everyone involved knows it is the wrong trade.

The reasoning is sound in isolation. A kernel change on a fleet is the highest-blast-radius operation available to you: one bad build and a hundred machines are bricked in the field, some of them in places you cannot physically reach for a week. So the kernel gets frozen at whatever it was on the day the product shipped, and every improvement after that has to happen in userspace, where it cannot see or control the things that actually matter.

What gets given up is not convenience, it is observability and control. The interesting failures on a robot happen in the interval between a command leaving the planner and a current arriving at a motor, and that interval lives in the kernel. Freezing it means every diagnostic you add afterwards is an inference from the outside of the box.

The alternative I am building is not a friendlier update process. It is making behavior a first-class, loadable, verifiable object: programs that get checked before they run, priced in CPU time before they are admitted, and clamped by a monitor that owns the actuator interface. If the change is unsafe, the kernel refuses it. If it is expensive, the scheduler refuses it. Nothing is reflashed, and nothing gets to be trusted just because it compiled.

This is not novel as an idea. eBPF made exactly this trade in Linux, and the reason it was accepted is that the verifier arrived with it. What is missing for robotics is the timing half: memory safety is not enough when the failure mode you fear is a bounded loop that takes too long inside a timer interrupt. So the cost model matters as much as the verifier, and the honest claim is that both have to be paid for before anyone should let this near a machine that can hurt someone.

I do not think the frozen-kernel practice is stupid. It is a rational response to not having a mechanism. My argument is only that the mechanism is buildable, and that once it exists the trade stops being necessary.

what firmware taught me about trust

I spent a summer removing code from a hardware wallet, and it was the most useful engineering I have done.

The Cypherock X1 supported many coins. The bitcoin-only firmware supported one. Almost all of the work was deletion: every non-bitcoin path, every parser that would never see valid input again, every branch that existed for a currency the device would now refuse. What remained is running on ten thousand-odd devices holding more than half a billion dollars.

Deletion is a strange thing to be proud of until you think about what a wallet is for. Every line of code in that image is a line the user is implicitly trusting with money they cannot get back. A feature they do not use is not neutral, it is a liability they did not agree to. Removing it is a security improvement you can prove by reading a diff, which is more than most security work can claim.

The other half was making the build deterministic, so that the same source produces the same bytes and a user can check the thing on their desk against the thing in the repository. Reproducibility is what turns "trust me" into "check it yourself", and it costs almost nothing once the toolchain cooperates.

I now apply the same test to systems that have nothing to do with money. What does the user have to take on faith, and can I make that set smaller? On AxiomOS the answer was a verifier and a provenance gate. It is the same instinct, pointed at a different machine.

robots do not get to reboot

Most software gets to fail by stopping. A machine holding a propeller does not have that option.

In server work, crashing is a legitimate strategy. Fail fast, restart clean, let the supervisor sort it out, and the worst case is a retried request. The whole discipline of reliability engineering rests on the assumption that stopping is safe.

Put the same code in a drone and stopping is a fall. Put it in an arm and stopping is whatever position gravity prefers. The safe state has to be reached actively, which means something has to keep running precisely when the interesting thing has already gone wrong, and that something cannot be the component that failed.

This is why staged degradation keeps showing up in my work. Ten seconds without a heartbeat is a warning, thirty is return-to-home, sixty is an emergency landing. The ladder is not clever. Its value is that each rung is a decision made in advance, by someone calm, rather than in the moment by code that has lost its inputs.

The corresponding kernel-side requirement is a monitor that outlives the thing it guards: it owns the actuator interface, it clamps every command to an envelope, and it latches an emergency stop that no higher layer can talk it out of. If that component is correct, everything above it is allowed to be merely good.

hardware makes you honest

A logic analyzer has never once been impressed by my reasoning.

Software lets you argue with reality for a surprisingly long time. Logs are written by the same code whose correctness is in question, tests exercise the paths you thought to write down, and a plausible mental model can survive years without being contradicted, because nothing in the loop is independent of you.

Attach a probe to a pin and that stops. The waveform is produced by the hardware, not by your assumptions, and when it disagrees with your explanation the explanation is wrong. Three days of the Pi 5 PWM bring-up were spent discovering that my clock assumption was wrong, and the only reason it took three days rather than three weeks is that the capture would not let me be right.

The habit I took from that is to write down the expected waveform before running the test. It sounds like ceremony. In practice it converts a debugging session into an experiment, and it has caught mistakes I would otherwise have explained away as noise.

It also changed what I consider evidence. A benchmark on a machine I was editing while measuring is a story about my intentions. A capture from a clean image, tied to a commit and a hash, is a measurement. The distinction is the entire reason my projects have a provenance gate, and the reason it is allowed to demote my own results.