Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
This week: the observability bill that never arrives as an invoice, and how to work out the size of yours.
Last Friday I said I wanted to take the second bill seriously. That turned out to be the wrong number. There are three.
Under-instrumentation never arrives as an invoice. It arrives as a redeploy. |
The lead
The bill you pay in redeploys. Three observability bills exist and most organisations only argue about one. The first arrives from your vendor, itemised, monthly, and gets its own meeting. The second is the one Brian Martin described in last week's issue: the runtime cost of instrumentation, paid in latency on every request. The third is larger than both and never arrives at all. You settle it in engineer hours, in builds, in the distance between somebody asking what happened and anybody being able to answer.
Last week three different people, across two subreddits, inside forty-eight hours, asked a version of the same question: how do I debug production without redeploying. The sharpest of them is running Rust microservices on Kubernetes, heavy async, tracing already in place, and is tired of shipping a build just to find out what a request did. Not an absence of telemetry. Telemetry that was there, and could not answer the question put to it.
If we're honest: most instrumentation is written to prove a system works, not to explain it when it does not. You instrument for the demo, for the SLO somebody signed, for the dashboard a director asked about in March. Every one of those answers a question you already knew was coming. The question that arrives at two in the morning is the one nobody anticipated, and nobody budgeted a span for it.
So price it, because almost nobody has. Take your last five incidents and count one number for each: how many times did somebody change code, add a log line, raise a sampling rate or redeploy, purely to answer a question the telemetry could not? That total is your under-instrumentation bill, denominated in deploys. Multiply it by your change lead time and you have it in hours. Done honestly, it will be the largest observability number in your organisation that has never appeared on a slide.
Where this breaks is the reflex fix. Counting the bill does not make paying it optional, and instrument everything just moves the money to the other two: more spans, more latency, a bigger invoice, and a point where the cure costs more than the illness. The discipline is narrower. Instrument the questions you actually got asked, and you now have that list, because you just counted it.
Signal check
Two this week, and underneath they are the same question. What can the system prove without you?
"How much forensic context do developers actually need to debug production failures?" Less than you are collecting, and not the same shape as it. The test is not volume, it is whether one engineer can reconstruct a single failed request end to end without opening a second tool. That needs four things: a trace id that survives every hop, the inputs that made this request unusual, the version of the code that served it, and the state of whatever it depended on at that moment. Most estates have the first and the third and are guessing at the other two. Start there rather than at retention. The teams drowning in data and the teams starved of it are very often the same teams.
"The guy who knew why is gone. What now?" You have just been handed an honest audit, and that is the only useful way to read it. Everything that person carried was undocumented reasoning: why the retry count is seven, why that queue drains differently on Mondays, which alert has been safe to ignore since 2021. None of it was ever in the telemetry, which is why it left with them. Do not start by writing the wiki nobody reads. Start with the next incident. Every question your team cannot answer without that person is a gap you can now name, and a name is the thing a budget attaches to.
Latest on the blog
Somebody turned it on. Nobody is selling you an unsupervised agent. They are selling you a dropdown, and somebody in your organisation has already changed it. This week's piece is about what a system has to prove before it earns each rung of autonomy, and who is signing when it moves. The four rungs, and what each one commits you to.
WebMCP breaks synthetic monitoring, and nobody has noticed yet. Last week's lead, and it belongs in this week's argument too. A synthetic check asserts a fixed route. An agent picks its own, so the check cannot fail in the way you would need it to. The one that turns a badly worded sentence into a production defect.
What you missed
Three worth your time, and why.
The reframe. Atlassian has described how it automates root cause analysis for large cloud-native incidents, written up by Craig Risi for InfoQ on 15 September. It treats RCA as a correlation problem rather than a collection one: anomalies found independently in metrics, traces and logs, normalised into one format and aligned on a single timeline, then walked through a service dependency graph built from real production traffic rather than from the architecture diagram. That last detail is a quiet admission that the diagram was wrong. What they look for in each of the three signals.
The one that sets your bill years in advance. Mike Goldsmith, a staff engineer at Honeycomb, has written up the adaptive tail sampling processor his team is donating to the OpenTelemetry Collector. Today you pick between head or probabilistic sampling, cheap and blunt, or tail sampling, which keeps what matters but needs rules you write and then maintain. The new one fingerprints similar traces and tunes each fingerprint toward a goal you set, a percentage or a throughput budget. It is working toward alpha upstream and runs today in Honeycomb's own distribution, so read the positioning as a vendor's and the mechanism as an operator's. Sampling is where you decide, months ahead, which questions you will not be able to answer. The worked config, and the trick that keeps quiet routes visible.
The market signal. Dynatrace is acquiring Arize AI, written up by Sam Weston for SiliconANGLE on 11 September from a theCUBE podcast with both companies' product chiefs, so weight it accordingly. It pulls AI evaluation and agent monitoring into a mainstream observability platform instead of leaving them in a tool of their own. If your LLM traces currently live somewhere other than your primary APM, and plenty do, this is the direction the platforms are betting on. It does not settle the question for your estate. It does mean the tool split you chose this year may get decided for you next year. Why both sides say their own customers asked for this.
That is the week. If you take one thing from it, count the redeploys in your last five incidents. You will know something about your estate by Friday that no dashboard was going to tell you.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Tune your Signal. Which do you want more of? Technical deep-dives are pieces like the WebMCP one above, taking a proposed standard apart before it reaches you. Leadership takes are the ownership pieces, like the autonomy ladder above. One click, and it changes what lands in your inbox.
Which do you want more of?
Allan
PS: The bill in this issue is an ownership problem before it is a technical one. Somebody owns the spans. Somebody owns the sampling rate. Almost nobody owns the list of questions the system cannot answer, which is why that list never gets shorter. Chapter 4 is about exactly that: three layers of ownership, and why most organisations build only the platform layer. It is the chapter I give away free, just below, and it ends on a move you can make on Monday. Take your most critical journey and name the person who owns the outcome, not the team.
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|