Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
This week: the question your own estate cannot answer yet, and the date it stops being optional.
Also new this week, and free for subscribers with nothing to sign up for: The Outcome Question, a twenty-minute test for the number your CFO watches, and The Signal Audit, thirty minutes on your own last incident.
A control that works and a control nobody has ever exercised produce the same telemetry on a quiet day. Both look like nothing. |
The lead
Ask most teams to evidence a control and you get shown a dashboard. The panel is green. The service has held for ninety days. It has not been called out since the spring. Every one of those statements is true, checkable and easy to screenshot, and not one of them answers the question that was asked.
The difference between a working control and an untested one only surfaces on the day you need it, and that is the one day you do not get to choose. The obligation, meanwhile, is already written down. Article 12 of Commission Delegated Regulation (EU) 2024/1774, the technical standard sitting under DORA, has required financial entities to develop, document and implement logging procedures since July 2024. Read the verbs rather than the noun. Not hold logs. Build and operate the machinery that produces them. Two of its clauses are unusually well informed for a legal text: it wants measures to detect a failure of your logging systems, and it wants every clock synchronised to a documented reliable reference time source. Whoever drafted that second one has clearly spent an afternoon reconstructing an incident across three systems that disagreed about when things happened.
Your finance colleagues settled this argument decades ago and gave it a name. Information Produced by the Entity. Under the PCAOB’s AS 1105, a report your own systems generated becomes audit evidence only once its completeness and accuracy have been established. BDO’s guidance on the same doctrine carries the line that should stop an observability leader mid-sentence: conclusions built on incomplete or inaccurate information may be invalid even if the audit procedures themselves were well designed.
The UK has now put a date on it. PS26/2 from the FCA and PS7/26 from the PRA, both published on 18 March 2026, take effect on 18 March 2027: a register of material third-party arrangements, filed every year. Against that, Catchpoint’s SRE Report 2026, drawn from 418 responses, finds 17 per cent of organisations running chaos experiments regularly and 34 per cent who have never tested failure in production at all. I do not read that as negligence. Years ago, as head of monitoring, I worked an outage past seventy hours with the regulator in the room. The blind spot was a vendor-managed API gateway emitting nothing. Forty minutes after we were allowed to put an agent on it, we could see the problem. Seventy hours blind, forty minutes sighted. The gap was never instrumentation. It was permission, and permission tends to be withheld a long way from the desk that files the register. I have written up the test I would run instead of a maturity model, and it takes about a minute per control.
Signal check
Two worth answering: one from the thread under the last issue, one from the floor.
Once you have isolated the problem down to a process, a thread or a core, who actually owns it, and who can make the change? Noman L raised this under Issue 106, then added a step to his own sequence to make the point: utilisation, attribution, ownership, correlation, controlled change, validation. Ownership sits in the middle rather than at the end, and his point is that the ownership gap often outlasts the technical investigation. You can find the offending thread inside half an hour and then spend three days establishing who is allowed to restart it. No dashboard closes that gap. Write the owner down before the incident at the level of a name rather than a team, and treat the absence of a name as the finding.
What observability evidence should be mandatory before a service reaches production? Not which tool, the minimum bar. Three things, and none of them is a dashboard. One, the service emits an event when the thing it promises actually happens, not when its own process completes. Two, its clock agrees with everything it will ever be correlated against, because evidence that cannot be ordered will not carry an argument. Three, a competent reader who is not the author can look at its output and say what state the service is in. Fail the third and the control is really a person, and that person takes holidays.
What you missed
Two from the fortnight, and what each one is actually about.
OpenTelemetry’s Kubernetes attributes processor went stable, and it will break your dashboards. Craig Risi for InfoQ, 6 October. The processor reached v1.0.0 for logs, metrics and traces, and adopts the Kubernetes semantic conventions that stabilised in June. It is not fully backwards compatible:
container.image.tagbecomescontainer.image.tags, and any alert or query naming an old attribute stops matching without announcing itself. Feature gates can emit both during the migration, and the migration notes are where to start. Set that beside the lead. A rename is one of the ways evidence stops being produced while the panel stays green.A cloud provider now ships agent tracing as part of the platform. Ryan Daws for Developer Tech, 13 October. DigitalOcean has put Insights into public preview, built on OpenTelemetry, with an Action Gateway that traces an agent turn from the model call through each tool invocation: the Action Gateway detail. Every claim in the piece is DigitalOcean’s own and no independent voice is quoted, so skip the claims and keep the shape. A trace following one agent turn through every tool it called is close to the evidence a register will eventually ask for, and it is turning up in the platform rather than in something you buy separately.
Pick one control this week that would appear on a register. The failover you have never invoked will do. Try to evidence it cold, with no warning to the team, and write down two numbers: how long it took, and what you could not produce at all. The second number is the finding.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Allan
PS: The chapter I give away is the one about who owns what when it breaks. I wrote it about services, and it reads uncomfortably well against a register.
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|