When the Satellite Is Fine but the Uplink Is Not: CloudOps Diagnosis

A spacecraft can be perfectly healthy and still look broken from the ground. Telemetry stops, commands are not acknowledged, and the first instinct of a tired operator is to blame the vehicle. Sometimes the fault sits in the relay, the scheduling system or the ground station that carries the instructions. Cloud estates have the same split, and the discipline that watches the ground-side half is called CloudOps. This post treats it as a diagnosis problem: what the symptoms look like, which layer is really at fault, and what you change so the next failure is shorter.

The symptom: the vehicle is fine, the uplink is not

The Ops Atlas describes CloudOps as the work of operating a public-cloud estate through the provider’s control plane: the organisation hierarchy, the guardrails, the quotas, the choice of Regions, the provider’s own health signals and the changes the provider makes to managed services. The running workloads are somebody else’s concern. Think of it as mission control for the estate rather than the flight software.

The clearest illustration comes from the AWS us-east-1 incident of 19 and 20 October 2025. According to the Atlas entry, a DNS race condition around DynamoDB set the chain off. New instance launches failed for roughly eight hours, from about 2:25 AM to 10:36 AM, and the error most customers saw was “insufficient capacity”. That message sent many teams hunting for spare hardware, when the real problem was a failing control plane. Meanwhile, instances that were already running stayed healthy. In flight terms, the satellite kept orbiting; it was the commanding path that went dark.

If you enjoy orbital mechanics, our own primer on how a geostationary orbit holds its position makes a useful comparison: once a spacecraft is parked, it needs very little intervention. Cloud workloads that are already running behave similarly, and that is exactly why the launch path deserves separate attention.

Differential diagnosis: three errors that look alike

The Atlas draws a distinction that is easy to lose at three in the morning. A capacity error, a quota error and throttling can all surface as a refusal to start something, yet each has a different owner and a different fix.

  • Capacity shortage. The provider genuinely has nothing to give you in that location right now. The cure is flexibility: another zone, another instance family, or waiting.
  • Quota ceiling. The provider has plenty, but your account has been capped. This one is yours to forecast and to request in advance.
  • Throttling. You are asking too quickly. Slow the callers, add back-off, and spread requests out.

The October 2025 case adds a fourth possibility that wears the first one’s costume: a control-plane fault reported as lack of capacity. When all new launches in a Region fail at once while existing machines look fine, suspect the layer that creates machines before you suspect the machines.

Why the diagnosis keeps going wrong

Three habits make this fault hard to see. First, most monitoring is built around the workload, so a healthy dashboard hides a dead launch path. Second, the provider’s status page is a lagging witness. The Atlas notes that during Google’s global API outage on 12 June 2025, health updates trailed the event by about an hour, and customers whose monitoring ran on the same provider lost their view as well. Third, teams rarely rehearse recovery steps that depend on the control plane, so the first rehearsal is the real incident.

That Google event is a tidy example of a configuration fault travelling faster than anyone can react. A quota-policy change containing blank fields replicated worldwide within seconds, and recovery in us-central1 took roughly two hours and forty minutes. Nothing in a customer’s code had changed. A flight-controller would call this a bad command uplinked to the whole constellation at once.

The treatment: fewer control-plane steps in the recovery path

The Atlas recommends a property it calls static stability. The shorter your recovery-time objective, the fewer control-plane operations your recovery plan can afford to contain. For top-tier failover, the recommended number is zero: the plan should rely only on data-plane activity, which the Atlas attributes to AWS’s own guidance. Capacity is already provisioned, traffic shifts are pre-wired, and nobody needs to launch anything during the emergency.

Mission planners already think this way. A walkthrough of the phases of a spacecraft flight shows that the riskiest manoeuvres are scripted long before they happen, with autonomous fallbacks onboard for the moments when the link is down. Cloud recovery deserves the same preparation.

Practical steps that follow from the diagnosis:

  1. Write down every action your top-tier recovery plan requires, then mark each one as control plane or data plane. Anything in the first group is a risk to remove.
  2. Pre-provision the standby capacity instead of promising to launch it later.
  3. Host at least a minimal view of your own health outside the provider whose failure you are trying to detect.
  4. Keep a human-readable runbook for the awkward case where the console itself will not let you sign in. In the October event, the Atlas reports console sign-in problems in other Regions for roughly ninety minutes.

Prevention: quotas as a forecast problem

Not every refusal is an outage. Many are scheduled by our own inattention. The Atlas lists a few quota traps worth knowing. Reserved but unused capacity can still count against your on-demand allowance. Some GPU families start with a default of zero virtual CPUs, so a new project may be blocked before it begins. And at least one cloud vendor can adjust quotas on its own initiative, so a limit you checked last quarter may not be the limit today.

The Atlas also offers a simple runway calculation. Suppose a quota is 256 vCPUs, you currently use 184, and you add 2 vCPUs of net demand per day. Take 80 percent of the quota (204.8), subtract the current use, and divide by the growth: about 10.4 days until the warning line and around 36 until exhaustion. The number is illustrative, but the habit of measuring runway rather than waiting for a failed launch is the useful part.

Orbit planners call a similar idea a margin budget. A short beginner article on what makes a low Earth orbit so busy illustrates why: crowded lanes leave little room for improvisation, so slots are reserved well in advance. Cloud quotas work the same way.

Where CloudOps stops

It helps to be honest about boundaries. The Atlas treats spend management, security posture, policy writing, infrastructure-as-code authoring, networks, clusters, functions and host operating systems as neighbouring disciplines, not part of CloudOps itself. The discipline is narrower than it first sounds, which is exactly what makes it testable. The same entry labels the field “emerging”, noting that no neutral standards body defines it, so definitions from different vendors and publishers vary. Read their claims with that in mind.

A short field checklist

  • Can you tell, within five minutes, whether a launch failure is capacity, quota, throttling or a provider fault?
  • Do you watch your own estate from somewhere the provider’s outage cannot reach?
  • Does your top-tier failover plan contain zero control-plane steps?
  • Is quota runway measured, with an alarm well before the ceiling?
  • Has anyone rehearsed the plan with the console unavailable?

Should most of those answers come back positive, you have moved the failure from the surprise column to the rehearsed column. For the full definitions, the maturity ladder and the 30, 60 and 90 day plan, read the Ops Atlas entry on CloudOps. Then pick one workload, trace its recovery plan line by line, and see how many steps depend on a control plane that may be unavailable at the exact moment you need it.