Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
When Microsoft 365 came back on Saturday, two days of held instructions came back with it. What did the agents in your estate do with them?
Recovery restored the mail. It did not restore the order in which the decisions were made. |
The lead
Microsoft 365 started failing at about six in the evening UTC on Thursday 12 November. Microsoft's first public account was servers failing automated health checks. Then came a day and a half of fixes that did not hold: extra capacity that never took effect, routing changes, virtual machines scaled up and stuck in a bad state. Later updates blamed traffic management imbalances Microsoft was still trying to pin down, and by Friday evening it had no estimate for full recovery. The last customers came back on Saturday morning, around 36 hours in. The root cause is still not settled in public. Every failed fix, in the order Microsoft announced it.
One decision through that window was sound, and it is the one that matters here. Microsoft extended its email timeout window to 48 hours so delayed messages were held rather than dropped. Nothing was lost. So when mail flow resumed, two days of instructions arrived at once, in the order they were queued. The agents wired into those mailboxes had been failing throughout: Anthropic and OpenAI both reported their Microsoft 365 connections failing. They came back when the mail came back, and they read the backlog as the present.
Recovery restored the mail. It did not restore the order in which the decisions were made. Robert Szczerba, writing in Forbes on 13 November, sketches a hypothetical to show the shape of it. A customer emails on the Thursday to change where her new espresso machine is delivered. By the afternoon she has phoned instead, and a person has updated the order. On the Friday the held email lands, the agent reads it as a current request, switches the address back, and confirms it. Every step in that sequence was reasonable on its own. The gap sat between what the agent had been told and what a colleague had already done about it.
This is not a monitoring gap, which is why it will catch people. Issue 111 was about a quiet pipe reading as a calm system. This is the other half. An agent cannot tell a quiet queue from a stopped one, and once the queue drains it cannot tell old from now. The remedy is governance, not tooling: one named person who decides when an agent pauses and resumes, written down before the next outage rather than during it; a shared record of what has already been handled, which the agent reads before it changes anything; and a rule that a conflict goes to a human. Pausing the agent costs you the thing you bought it for, on the day you most want it. That is still cheaper than undoing a decision your own team already made for a customer. The lost-link rule he borrows from drone operations, and the two-order drill written out in full.
Signal check
Two from the week. One from the floor, one from the top of the house.
What do you actually put behind a /health endpoint, when it returns 200 while the app may not really be healthy? Less than you think, and that is the point. Liveness answers one question: should this instance keep taking traffic. Make it also check the database, the cache and three downstream APIs, and you have built a way for one slow dependency to empty the whole pool at once. Keep liveness thin and local. Dependency checks belong on a readiness probe, which controls traffic rather than restarts, and the real test belongs in a transaction run from outside. Microsoft's own first account last week was failed health checks. A 200 is a vote, not a verdict.
What evidence should an operations agent produce before you act on its root cause analysis? The same three things you would want from a contractor. What it looked at, with timestamps and the query, so somebody can run it again. What it ruled out and why, because a conclusion with no discarded alternatives is not an investigation, it is a preference. And what changed in the window, named: the deploy, the config, the flag, the feed. If it cannot give you the second one, you have bought a summariser. The test is whether somebody who disagrees with it can check it.
Latest on the blog
Sunday's piece is live. 99.97 Per Cent Uptime, and Customers Still Could Not Buy is about the gap between a green availability number and a checkout nobody could complete. The five-minute test, and why every outcome needs one named owner. This Sunday: three questions worth taking into your next quarterly review.
What you missed
Three worth your time.
The SRE book has been rewritten for a different decade. Google has put out a second edition of Site Reliability Engineering, ten years after the first, and Matt Saunders' write-up for InfoQ on 17 November shows what moved. Error budgets go from manually applied policy to burn-rate alerting and automated enforcement. Observability goes from static dashboards and manual triage to OpenTelemetry, tracing and automated correlation. What moved in each discipline, set side by side.
Netflix is building the shared model its agents are meant to argue over. Craig Risi for InfoQ, 13 November, on Netflix's observability ontology: a knowledge graph tying applications, deployments, incidents, alerts and dependencies together, so an investigation does not start by reconstructing how any of it relates. Netflix says a typical regression investigation runs to about four hours, mostly spent on exactly that, and that its correlation engine reads thirteen signal types to rank causal hypotheses. Netflix's own figures, unchecked by anyone else. The validation rules Netflix puts around the LLM-generated parts of the graph.
If you run collectors, you have a February date in the diary. The OpenTelemetry Collector's batch processor is deprecated, announced on 11 November by Bogdan Drutu, Dmitry Anoshin and Joshua MacDonald. Batching moves into the exporter's sending queue. Exporter-level batching becomes the default in v0.171.0 in February 2027, and the processor goes in v0.177.0 in May 2027. The trap is queue sizing: the queue now holds unbatched requests, so the default of 1000 carries far less than it did. The migration steps, and the queue-size trap underneath them.
One thing to do today. Find the agent in your estate that can change something a customer sees, and write down the name of the person who turns it off. Not the runbook. The name.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Allan
PS. New and free: The Outcome Question, a twenty-minute test for the number your CFO watches, and The Signal Audit, thirty minutes on your own last incident.
PPS. The new season of Metrics & Mayhem is recording now, with guests from every corner of our industry. No names yet. It starts in December. Follow the show and the first guest episode lands in your feed the day it drops.
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|
