In partnership with
Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
Three weeks on, the London zone failure has its cause. A controller watched the power all night and reported that the power was fine.
|
Everybody instruments the system. Almost nobody instruments the route the evidence travels. |
The lead
In Issue 109 I wrote that Google's account of the London zone failure was one sentence written by somebody else, and that an analysis was promised. It landed on 4 November. A supplier's cable failed under the street on feeder A. Feeder B was already out of service, its breaker opened for repairs after problems turned up in a quarterly generator load test on the 14th. A generator bank sat on site and did not start. The batteries carried the load for their five designed minutes, then stopped.
The cause Google names is a voltage-sensing design flaw and a misconfigured controller. The logic read a healthy utility source while both distribution feeds sat dead, so no start command was ever issued. Notice what is missing. Nothing failed quietly. The controller was watching all night. It had a sensor, a signal and a rule, and all three did their jobs. The rule was pointed at a wire that was still live while the two that mattered were not.
Your estate has that shape in it somewhere. Everybody instruments the system. Almost nobody instruments the route the evidence travels. A collector that has quietly stopped exporting looks exactly like a slow Tuesday. An empty panel looks like a service with nothing to say. The OpenTelemetry project said it in one line on 10 November: "The application may be healthy while its telemetry path is not." Its fixes are cheap and dull. Reconcile the counts, so that accepted equals completed plus rejected plus still processing. Send a small synthetic event down the real path at a known interval, so the quiet is tested rather than trusted. The part worth keeping is the three-way check that tells a quiet system from a broken pipe, because it reads two signals together and names which one failed.
Watch the collector's own receiver and exporter counters too, because a running process is not a working signal. That matters more now than it did last year, because the thing reading your dashboard at four in the morning does not squint at an empty panel the way a tired engineer does. Hand an agent silence and it will report calm, with reasons, in decent English. Issue 110 asked how often yours is wrong while sounding certain. This is the case where it will be wrong and certain through no fault of its own. The full sequence as Google tells it, via The Register.
Signal check
One from the floor, one from the review meeting.
How do you tell that a provider is actually down, when your server trace says success and the user's browser shows a failure? Because your trace stops where your responsibility stops, and the user's experience does not. A 200 proves the call was accepted, not that what came back was usable. So record the outcome rather than the status code: did the payment clear, did the page render, did the asset load. Then run that check from where the user sits, not from inside your own network, because a provider degrading in one region is invisible to a probe that never leaves your datacentre. The gap between the two is the finding.
Why is the industry selling observability for AI agents rather than control, when we have already had incidents where watching was not enough? Because watching is sellable and control is accountable. A dashboard needs a budget line. A rule about what an agent may do without a person needs somebody's name against it, which is a harder conversation than a renewal. The London site was observable throughout, and the observing is the part that failed it. So the question to put to any proposal is not what it will show you. It is what it is allowed to do about what it sees, and who signs for that.
Latest on the blog
If you missed it on 3 November, Your AI-SRE Should Show Its Working walks through the four rungs of authority you can hand an agent, from read-only to acting alone, and how to test it before it touches production. And this Sunday: "99.97 per cent uptime, and customers still could not buy".
What you missed
Two worth your time.
OpenTelemetry's zero-code agent reached 1.0, and it now reaches the AI layer. Tyler Yahn announced eBPF Instrumentation v1.0.0 on 30 October. The headline is not a feature, it is a promise: configuration, deployment and stable telemetry now carry compatibility commitments, so an upgrade stops being a gamble. New coverage includes model providers, embeddings, vector retrieval and MCP calls. The caveats sit in the same post: runtime metrics are experimental, the detailed AI capture is opt-in, and config v1 is frozen. The full list of AI clients and gateways it now covers.
A test for whether you bought observability or a newer invoice. Rox Williams for Honeycomb, 10 November, writing up Charity Majors and Rick Clark on what they call o11ywashing: monitoring relabelled and resold. It is a vendor's podcast on a vendor's blog, so take the framing accordingly. Clark's own test travels regardless. Can you ask your tooling a question nobody anticipated when the dashboards were built? If you have to know what will break before you can investigate it, that is monitoring. Clark's five tests, one of them worked through.
Pick the quietest dashboard you own, the one nobody checks because it never says anything. Send one synthetic event through it this week and see whether it comes out the other end. If it does not, you have just found the panel your agent would read as calm.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Tune your Signal. Which do you want more of? Technical deep-dives run like the eBPF instrumentation read above. Leadership takes run like this week's lead. One click shapes what lands in your inbox.
Which do you want more of?
Allan
PS. New and free: The Outcome Question, a twenty-minute test for the number your CFO watches, and The Signal Audit, thirty minutes on your own last incident.
PPS. The podcast is about to get company. The new season of Metrics & Mayhem is already recording, with guests from every corner of our industry, and I'm excited about who's coming. No names yet. It starts in December. Follow the show and the first guest episode lands in your feed the day it drops.
|
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|
Find what’s slowing your GTM.
Most GTM problems aren’t one big failure. They’re one or two weak spots quietly dragging down the rest.
The free GTM Audit scores your go-to-market across four dimensions, then sequences a 90-day roadmap for your stage.
Drop your URL. Get your score in about 10 minutes.