Picture a small volunteer team that runs a ground station for a handful of student-built satellites. Nobody is paid, everybody has a day job, and the antenna sits on the roof of a borrowed lab. This is a made-up crew, but the story they live through is the heart of DisasterRecoveryOps: the discipline of getting a service back inside an agreed time, and of practising that comeback until it is boring.
Let us call the team Orbit Desk. For two years their dashboard, their pass-prediction scripts and their archive of downlinked images lived on one rented server. They had backups, and they were proud of them. Missing was any firm reply to a simple question: if the server vanished at two in the morning, how long until the next pass could be received and stored again?
The night the rented server went dark
One Saturday the hosting provider lost a whole rack. Orbit Desk got an alert, the alert woke exactly one person, and that person spent a long stretch of the night doing something very human: waiting to be sure. Was it a blip? Should they wake the others? Who was even allowed to approve paying for a replacement machine in another region?
By the time anyone typed a recovery command, a good chunk of the night had gone. The technical rebuild itself, once started, was quick. The delay was in the deciding. That is the lesson the team would later find summed up in the Atlas page covering DisasterRecoveryOps, where the point is that the recovery clock starts when service stops, not when somebody finally chooses to act.
Counting the minutes honestly
The next weekend the crew sat down with a whiteboard and a stopwatch and ran a rehearsal. They wrote down what happened in two columns. In the people column went noticing the problem and agreeing to act. In the systems column went spinning up the standby and pointing traffic at it. When they added both columns, the total blew past the target they had casually promised the university club, which was one hour.
A worked example in the Atlas makes the same point with tidy numbers: a drill where the people part took thirty-one minutes and the machine part thirty-three, for a sixty-four-minute total against a sixty-minute goal. Orbit Desk laughed uncomfortably when they saw it, because their own sheet looked nearly identical, just with different digits.
If you have ever watched a launch countdown, you know the feeling. The count is not only about fuel and valves. It includes the go or no-go polls, the voice that says “hold”, the person who is authorised to say “resume”. Take those out of the timeline and you have a fantasy schedule.
Setting a decision deadline
Orbit Desk borrowed a simple rule. Take the recovery-time goal, subtract the slowest time any recent rehearsal needed for the technical part, and subtract a small safety margin. What remains is the latest moment at which somebody must say “we are failing over”. The Atlas calls this a decision deadline and illustrates it with sixty minus thirty-three minus five, giving twenty-two.
For the team, it changed the conversation completely. Instead of arguing about whether an alert was real, the on-call person only had to check the clock. If twenty-two minutes passed without a better explanation, the move was made. They also wrote down who could approve the spend, with a named backup for when the first person was asleep or on a flight.
Picking how much standby to pay for
Money was the next sticking point. A volunteer club cannot run a second full copy of everything around the clock. The crew learned that there is a menu of strategies, using names popularised by a big cloud provider: restore from backups, keep a pilot light burning, hold a scaled-down warm copy, or run a fully active second site. Each step up cuts the waiting time and raises the monthly bill.
Orbit Desk split their stuff into tiers. The command-and-schedule service, without which no pass could be used, got a warm copy. The picture archive, which could wait a day or two, stayed on restore-from-backup. That mirrors the tier matrix in the Atlas, where the tightest tier pairs short objectives with standby capacity and the loosest tier accepts a long wait and plain backups.
- Tier A: pass scheduling and the live status page, warm copy in a second region.
- Tier B: telemetry decoding jobs, a minimal pilot-light setup that can be scaled up.
- Tier C: the long-term archive, restored from backup when time allows.
Lost writes and the awkward return trip
The next surprise was about data. When Orbit Desk switched to the standby during a rehearsal, the last few seconds of telemetry that had been written on the old machine had not yet been copied across. Failing over can lose whatever was still in flight. A planned, orderly switchover is a different animal, because you pause writes first and nothing is lost.
The Atlas also warns that going back is a second migration, not an undo button. Orbit Desk therefore added a line to the plan: before touching the old primary, preserve it, copy it, and only then compare what is missing. They remembered a lesson from a beginner-friendly tour of low Earth orbit that they once shared with new members: a pass happens when it happens, so the data you miss is gone for good unless you captured it somewhere else. The sister piece, a plain-spoken geostationary orbit explainer, makes a similar point about fixed positions and fixed timing.
Practising until it is dull
The final change was cultural. Every quarter, Orbit Desk now runs a drill during daylight, announces it in the group chat, and times it. They log the numbers in a shared sheet: how long to notice, how long to decide, how long for the machines. Over a year they moved from missing their goal by a wide margin to finishing a few minutes early, and the most visible gain came from the decision column.
Further along the Atlas sketches five maturity steps, from objectives merely signed off up to switchovers performed routinely in production. Orbit Desk sits somewhere in the middle: the plan exists, it has been exercised twice inside the target, and the next goal is proving that the trip back works without anyone clicking around a console from memory.
What an astronomy-minded reader can take away
You do not need a satellite to borrow this. Anyone who keeps a website, a stargazing log or a small observatory feed alive can ask three questions. How long can this be down before people really suffer? Who decides to switch, and by when? And have we ever timed the whole thing from first alert to working service?
If the answers are vague, start with a stopwatch and a quiet afternoon. A sloppy rehearsal teaches more than a perfect document nobody has opened since it was written, and the habit of timing yourself will pay off on the night the real alert arrives.