A launch room runs on one unglamorous promise: nothing fires until the sequence is committed, and anyone on console can call an abort. Picture the opposite world. An unattended server at two in the morning runs a cleanup script written by an engineer who left the company years ago. The script holds more permission than it needs, aims at a target list nobody has opened since, and answers to no one. On a bad night it does not call an abort. It simply deletes. The distance between those two worlds is what AutomationOps exists to close — the craft of operating routine automation as a production system, with the same pre-flight seriousness a spacecraft receives before it leaves the pad.
When a Routine Job Becomes the Emergency
Start with the diagnosis, because the pattern repeats everywhere. A task hurts, so somebody writes a script. The script works, so it gets scheduled. Months pass, the infrastructure around it drifts, its author moves on, and its permissions quietly grow. Nothing about the script changes, yet everything about the risk does. Operations teams discover the trap only when a rename turns a path into an empty string and the job, cheerfully obedient, treats that emptiness as a licence to act on everything it can see.
That failure has a famous name attached to it. Google engineers once built a removal tool called Diskerase, and a mistyped configuration let it conclude there was nothing to protect — so it started wiping data it should never have touched. The lesson they carried out of that room became a rule worth tattooing on every scheduler: empty never means all. If an automation resolves its target list to zero entries, the only correct behaviour is to stop and refuse.
Notice what actually failed in that story. Not hardware, not network, not code quality. Governing. Nobody had decided, in advance, how much damage that one job was allowed to do. This is precisely the question Astrology and Space readers will recognise from orbital mechanics: a satellite drifting a few metres is harmless; a satellite drifting with no station-keeping plan is a future collision. Same object, same speed — the plan is the safety.
AutomationOps, Explained in Flight Terms
Think of each automated job as a small vehicle. It has a mission (restart this service, rotate these keys, prune those logs), a propulsion system (the runner that executes it), and a blast radius (everything it could touch if it went wrong). Flying it well means accepting five checks before every mission, not just the first one. Just as the phases of a spacecraft flight each carry their own go/no-go gates, a mature automation passes its gates on every single run.
The five checks, in flight language:
- Rehearsal first. A dry run shows what the job would change without changing it. One caveat earns its keep here: in tools like Ansible, check mode can skip command and shell probes unless you explicitly flag them, so a green rehearsal is only as honest as the probes it actually exercised.
- Hard rule on empty lists. Zero resolved targets means refuse to run, never proceed on everything. The Diskerase story is the reason this rule exists.
- Capped blast radius. Act in small batches — a canary first, then the rest in waves. Runbook tools give you this natively: Ansible offers the serial keyword and max_fail_percentage, so a bad wave stops the flight instead of spreading across the fleet.
- A ceiling on tolerated failures. Decide in advance how many failures abort the mission. Three dead targets in a wave of a hundred is a signal, not a statistic to push through.
- A stop button everyone can find. Each job gets its own kill switch, plus a global one, and on-call engineers drill both twice a year for anything rated risky. A switch nobody has ever pulled is a decoration, not a control.
One more habit deserves importing from the flight deck: describe jobs by their destination, not their motion. A task phrased as “ensure the service is running” survives being rerun, while “start the service” explodes the second it fires twice. Idempotent design is why Google’s own repair automations could safely repeat every fifteen minutes — each pass nudged reality toward the same desired end, so an extra lap cost nothing and a missed lap healed itself. Write every job so that running it twice is boring, and half your failure modes vanish before you ever build a dashboard for them.
Matching Caution to the Mission: Risk Tiers
Not every job deserves the full pad. A read-only report that touches nothing cannot hurt anyone, so treating it like a fueling operation would waste everyone’s patience. The Atlas proposal sorts automations into four tiers. Tier 0 is read-only. Tier 1 changes one target but can be reversed. Tier 2 changes many targets, reversibly — this is where rehearsals, batches and health checks become mandatory. Tier 3 is the dangerous class: anything irreversible or destructive to data. Those jobs require a second person’s approval before they fire, deletion that lands in soft form first, and a staged rollout rather than one grand burn.
The security world already endorses this instinct. NIST’s dual-authorisation controls — AC-3(2) and CM-5(4) — formalise the idea that catastrophic actions need two humans, not one overconfident one. If your profession already treats certain acts as too weighty for a single signature, your cron jobs deserve the same respect.
What AutomationOps Asks of the Human on Console
There is an old truth from Lisanne Bainbridge’s 1983 paper on the ironies of automation: building a machine to remove human error does not remove error, it moves the errors earlier — into design time, where they are harder to see and more expensive to fix. The person who once typed each command now supervises the thing that types them, and supervision is a skill that decays when nothing ever goes wrong. That is why teams serious about this keep manual skills rehearsed rather than retired, and why the standing rule says anyone on call may pull any kill switch without asking permission first. Google’s SRE book calls automation a force multiplier rather than a cure-all — it multiplies whatever you already have, including your mistakes.
Ownership completes the picture. Every job in the estate should have a name attached: who owns it, when it was last reviewed, what would make it safe to retire. Automation is not a monument; it rots like any other system. The habit of scheduled review dates — the way station-keeping keeps a satellite parked over one spot with small, regular corrections — is what stops an estate from drifting into a museum of orphaned scripts.
Grounding the Analogy for Newcomers
If all of this sounds like engineering culture imported from another planet, start smaller than a full programme. Pick your three most dangerous scheduled jobs. Give each one an owner and a review date. Add the empty-list refusal. Wrap the riskiest in a rehearsal mode and split its targets into batches. Practise the stop button once, out loud, with the whole team watching. None of this requires new infrastructure — only the decision to stop treating automations as pets and start treating them as vehicles.
The reward is the same one that draws beginners to a first walk through low Earth orbit: a system that behaves predictably because somebody chose its constraints deliberately. The AutomationOps entry in The Ops Atlas lays out this whole discipline — controls, tiers, metrics, maturity stages — in the same spirit, and pairs well with an afternoon spent auditing your own scheduler. Machines that repeat our work will also repeat our oversights, at machine speed, until we give them boundaries. Flight controls are not a burden on automation. They are what make it trustworthy enough to keep.