A Mission Control Audit: Applying SRE Thinking to Spaceflight

Watch enough launch broadcasts and you notice that the dramatic moment, ignition and liftoff, is only a small part of a mission. The longer story is whether a spacecraft keeps doing its job for months or years afterwards. Ground teams that care about that long story borrow a habit from software operations called SRE, short for site reliability engineering, and the habit translates surprisingly well to a mission control room. This checklist walks you through an audit you can run on any space programme, hobby satellite project or even your own reading list of mission reports.

The discipline treats operations as an engineering problem rather than a series of heroic rescues. According to the summary in the Ops Atlas entry on SRE, the core move is to set targets based on what users actually experience and then let the gap between target and reality decide how fast a team may change things. For a spacecraft, the “user” might be a farmer waiting for a weather image or a scientist waiting for a downlink window. Keep that person in mind as you work down the list.

Step 1: Name the experience you are protecting

Start by writing one sentence describing what a user notices when the mission works. “A usable image of the cloud deck arrives within a few hours of capture” is a good example. Resist the urge to list subsystem temperatures here; those are internal readings, not the promise. The Atlas entry makes the same distinction, noting that standard telemetry such as the four classic golden signals is only a starting point and does not by itself tell you whether users are happy.

  • Is the promise written in the user’s language, not the engineer’s?
  • Can it be counted as good events divided by valid events?
  • Is it measured as close to the user as possible, such as at the ground station output rather than at the onboard recorder?

Step 2: Pick a target that is deliberately below perfect

The next audit question is uncomfortable for ambitious teams. What is the target, and is it 100 percent? If so, mark it red. The entry on SRE argues that perfect is the wrong goal for almost everything, because each extra increment of reliability may cost a hundred times more than the previous one. Anyone who has priced a redundant flight computer against a second one knows the feeling.

A sensible target leaves a small, known amount of failure on the table. Subtract the target from 100 percent and you get what the discipline calls an error budget. With a 99.9 percent goal over a rolling 28-day window, for example, the Atlas reports about 40 minutes of bad service as the allowance. That window length is recommended partly because four weeks contain the same number of each weekday, so a pass schedule that repeats weekly is compared fairly with itself.

Step 3: Check that the budget actually governs decisions

A budget that nobody consults is decoration. During your audit, look for a written policy that says what happens when the allowance runs out. One example in the source describes freezing all changes except emergency and security fixes once the trailing four-week allowance is spent. In spaceflight language, that is a rule like “no discretionary software uploads to the vehicle while we are still recovering from the last anomaly.”

Ask these questions of any programme you evaluate:

  1. Is there a signed policy, or only a slide deck?
  2. Does a single bad event that eats a large share of the budget, more than a fifth in the Atlas example, automatically trigger a written review?
  3. Is the budget ever nearly full month after month? That can mean the target was set too loosely, or the crew is over-delivering at the expense of useful change.

The last point surprises people. The Atlas describes teams that deliberately cause planned outages in one of Google’s internal lock services so that dependants do not forget that it can fail. Reliability, in other words, is something to be spent wisely, not hoarded.

Step 4: Audit the alerts for noise and for blind spots

Next, open the alert rules. Two failures are common. The first is alerting on every metric, which trains operators to ignore pages. The second is the opposite: thresholds that can never fire. The source gives a striking example, that at a 90 percent target an alert designed to catch a two percent hourly burn can never trigger at all, because the budget is too big for that burn to matter in time.

The recommended approach compares how fast the budget is being consumed against the pace that would exactly use it up. A burn rate of one means you finish the window with nothing left. In the published workbook pattern for a 99.9 percent target, a fast burn over one hour pages a human, while a slow bleed over days opens a ticket instead. Nobody is woken for a slow leak, and nobody ignores it either.

  • Do fast burns page and slow burns create tickets?
  • Has someone replayed a past incident to confirm the rules would have fired?
  • Are uninstrumented components silently excluded from the maths? The Atlas warns that a sloppy “not equal” filter can hide them.

Step 5: Count the toil on the ground crew

Mission control staff spend plenty of hours on repetitive work: manually reformatting telemetry, re-running the same console procedures, acknowledging the same nuisance alarms. The discipline calls this toil, work that is manual, repetitive, automatable, tactical and leaves nothing lasting behind. The Atlas reports a Google guideline of keeping it under half of an engineer’s time, and a 2026 industry survey cited there found a median near a third.

A tidy way to audit is to run a simple tally. If six engineers each have 80 working hours in a period, that is 480 hours; if 190 of them went to repetitive chores, toil is about 39.6 percent. The figure is below the cap, but a team would still want a plan to bring it down. The point of the exercise is that a cap without consequences is just a wish, so check that the organisation will redirect tickets or send developers back on call when the number slips.

Step 6: Look at the on-call burden

Spacecraft do not respect office hours. Check how often the duty team is actually woken. A figure to compare against: the source recommends no more than two incidents per twelve-hour shift, and about eight people in a rota for a single-site round-the-clock service. If your programme runs a thin rota covering a vehicle that can fault at any hour, flag it.

Step 7: Read the last three postmortems

Postmortems are where culture shows. Good ones are blameless in the sense that they look for conditions rather than culprits, yet they remain strict about actions and owners. The Atlas lists postmortems whose actions never close as a classic anti-pattern. Take the last three reports from the programme and ask: did each end with named owners and dates, and did previous items actually get finished?

A quick scoring sheet

Give one point for each step above that passes cleanly, and half a point for partial answers. Six or more suggests a healthy practice. Four or five means it is probably well-intentioned but under-governed. Below four, the programme is relying on luck and individual dedication. Either way, the value of the audit is the conversation it starts, not the number.

For gentler background, our walkthrough of how rockets get payloads into orbit sets the scene, and the beginner tour of low Earth orbit explains where many operational headaches begin. The software side of the same ideas appears in a post about building practical site reliability skills.

Keep the checklist handy the next time a mission makes headlines. The question worth asking is not only “did it launch?” but “what promise does it make, how much failure is it allowed, and who decides when to stop changing things?”