Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
This week: the eight hours a provider explained in one sentence, and what that leaves in your evidence file.
Your provider's status page is their account of your outage. It arrives when they are ready, and for now it is the only account you have. |
The lead
Tuesday’s lesson was agreed before Tuesday’s facts were. Google Cloud lost a zone in its London region from early morning, Lloyds, HSBC, Royal Mail, Rightmove and DPD went with it, and by lunchtime the conclusion had settled into three things everybody already believed: multi-region, portability, tested failover. All three are reasonable. None of them is the part that hurts most afterwards.
What happened was explained in one sentence. “A power event caused an outage within europe-west2-b zone.” The Register’s timeline has the shape of the day: unplanned downtime from 06:08 UTC, a tentative recovery time of 13:00 given at 11:58, most customers back by 14:03. Dr Amro Al-Said Ahmad of Keele University put the count at around thirty-eight services down. Google says an analysis will follow its internal investigation, and that the status page gets no further updates. So the record of what happened to your service, this week at least, is one sentence written by somebody else.
Resilience advice answers the question of how you stay up. For most of those organisations the harder question arrives a fortnight later and sounds nothing like it. What degraded, when, for which customers, and on whose infrastructure. During the incident those two questions look like one. The week after, they come apart. Capacity in another region answers the first. Only telemetry you kept yourself answers the second, and if the thing that failed sat behind a boundary you rent, you may not have any.
Which is where March stops being an abstraction. PS26/2 from the FCA and PS7/26 from the PRA, both published on 18 March 2026, take effect on 18 March 2027 and require an annual register of material third-party arrangements. Some of Tuesday’s names will file one. A register entry is a claim about a dependency, and a claim wants evidence standing behind it. I wrote last week that the gap in my own worst outage, seventy hours with the regulator in the room, was never instrumentation but permission. Tuesday is that gap on a shorter clock. The question for your estate is not whether you would survive losing a zone. It is what you could produce, unaided, about what your service did during those eight hours.
Signal check
Two this week: one from the floor, one for whoever signs the contract.
Can you monitor a provider’s service status, so you are not hearing about their outage from your own customers first? Partly, and not the way most teams try it. A status page is editorial, not telemetry: IncidentHub’s record of the Cloud Storage outage of 8 October has the first update landing more than seven hours after the start. So do not subscribe to a status feed and file that as monitoring. Probe each rented dependency from outside, on a path a customer would take, from somewhere that is not inside the dependency, and record first customer impact against your own clock. The provider’s account then becomes a cross-check on yours instead of a substitute for it.
Who is running multi-cloud on purpose, and how are they surviving it? If a regulator or a customer contract requires two providers, I believe you, and the cost is the price of the clause. What I would push back on is buying it as insurance against days like Tuesday. The thing that fails over is rarely the thing that broke, you end up running two estates and reconciling two sets of evidence, and the second estate is the one nobody drills. The cheaper move is smaller and duller: keep your own copy of the telemetry for every rented component, and write down who is allowed to instrument it before the day you need to ask.
Latest on the blog
One, and deliberately not new. If the lead left you wanting your own copy of the signal and no budget to fund it, what I would spin up first has the component list.
What you missed
Two from the fortnight, and what each one is actually about.
Your proxy layer can now emit spans nobody wrote. Steef-Jan Wiggers for InfoQ, 10 October. Cloudflare has put Traces into open beta: security rules, cache decisions, routing and origin handling arrive as spans on one request timeline, with no instrumentation to write. It forwards W3C
traceparentheaders, so a trace that began upstream carries through, and spans export over OTLP to any backend. Read the pricing note beside it. From 1 December, billing moves to data ingested and retained, 50GB included on paid plans and $0.25 per GB after. Evidence from the boundary you rent is getting easier to collect, and it is also becoming something you pay to keep.Your vendors’ vendors went down too. Hrishikesh Barua for IncidentHub, 12 October, on the Cloud Storage outage in us-central1 on 8 October: 11:00 to 23:30 UTC by Google’s own record, the first status update more than seven hours in. IncidentHub logged four providers, Torq, Temporal, Elastic Cloud and Mixpanel, whose own notices named it as a cause. Its count puts Google Cloud at seven outages in the year to 12 October. The register asks about material third-party arrangements. This is the second-order version of that question. You know who you rent from. Do you know who they rent from?
Pick one rented dependency this week, the one you would have to write about if it failed on a Tuesday. With the status page closed, answer three things from your own data: when it started degrading for your customers, which of them it touched, and when it stopped. Whatever you cannot answer is the finding, and it is the first line of your register entry.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Tune your Signal. Which do you want more of? Technical deep-dives run like September’s piece on continuous profiling. Leadership takes run like this week’s lead. One click shapes what lands in your inbox.
Which do you want more of?
Allan
PS. New and free: The Outcome Question, a twenty-minute test for the number your CFO watches, and The Signal Audit, thirty minutes on your own last incident.
The chapter I give away is the one about who owns what when it breaks. Tuesday was eight hours of that argument, held in public.
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|