In partnership with
Observability you can act on, whether you build it or lead it, plus the few signals worth your time.
This week: the cycles on your cloud bill that nobody on your team wrote, and the signal that has started itemising them.
You already pay for every cycle on that host. Nothing on the invoice says which ones did useful work. |
The lead
The code you pay for and nobody wrote. Why is that host at eighty per cent? Most of us have put that question to a graph and had the same answer come back: it is busy. Confident about the number, silent about the cause. For years that silence was simply the deal, so we stopped asking it and went hunting by hand instead.
Two things happened to OpenTelemetry this year. It reached CNCF Graduated status in May, which is the safe, boring news, and boring is high praise for infrastructure. In March its profiles signal went to public alpha, and that is the one that changes your Tuesday. Continuous profiling becomes a first-class citizen of the pipeline you already run for metrics, traces and logs. It samples the running stack across every process on a host and names the functions actually on the CPU, in production, rather than in a lab you rebuilt to imitate production.
The interesting part is not the debugging. Israel Ogbole of the OpenTelemetry Profiling SIG described a case on OpenObservability Talks, written up by Dotan Horovits on 29 May: a process the cloud provider had injected into a customer's production environment was burning somewhere around 10 to 15 per cent of the CPU. Invisible in every per-service view, because nobody had instrumented it, and why would they. It surfaced only when somebody looked across the whole host at once and asked the blunt question. Their figure, from their episode, and it is the shape of the thing rather than a benchmark.
Reality check: this is a cost story wearing a debugging costume. You already pay for every cycle on that host. Nothing on the invoice says which ones did useful work. It does not separate your code from a third party's idle loop, a runaway sidecar, or a library doing far more than you ever asked of it. Whole-system profiling is the first signal that reads that bill line by line.
Two honest caveats before anyone gets carried away. Profiles is alpha, and the people building it say plainly that it should not carry anything critical yet, so this is evaluate rather than adopt. And it is not free: one more signal to store, and a flame graph nobody opens is storage with a prettier shape. So do the small version. One host, your busiest, a throwaway setup, one day. Then put two numbers side by side: which of the two is the outcome, and which is the early warning nobody could see before.
Signal check
One that came in this week.
"As operational context gets richer, deploy history, dependencies, incidents, traces, code changes, shouldn't that context itself tell us what telemetry is worth collecting?" It should, and the direction is right. Watch which way round you do it, though. Selection needs something to select against, and most estates today select against volume: this index is expensive, turn it down. That is a budget decision wearing an engineering hat, and it is how teams end up dropping the one field an investigation later needed. The version worth having selects against questions. Go back to your last five incidents, the same five I pointed you at last week, and write down what you could not answer, and what would have answered it. The list comes out short and uncomfortably specific, and it is the only test data you actually own. Once it exists, anything promising to shape your telemetry has something to be measured against, rather than being a slider that makes the bill smaller.
Latest on the blog
byte-size: What is OpenTelemetry profiling? Not new, and I will not pretend it is: published in July, refreshed on Monday so it sits properly under this week's lead. What a profile actually is, and why the old profilers belonged in the forensics drawer and this one does not.
What you missed
Two from the week, and what each one is actually about.
A vendor moved the cost decision upstream. Sumo Logic expanded its Data Pipelines so customers can cut and reroute telemetry before any of it is ingested, reported by Duncan Riley for SiliconANGLE on 21 September. The framing is familiar: ingest everything and analyse later has stopped being affordable, so choose earlier. The feature worth noticing is not the assistant that writes the rules. It is the step that comes before you switch a rule on, which is what decides whether any of this is safe.
Kubernetes 1.37 shipped, and the quiet item is the good one. Mostafa Radwan wrote it up for InfoQ on 21 September. The Metrics API reached general availability, which matters here because it is the API your autoscaler reads before it decides how many hosts you pay for; that decision now runs on a supported contract rather than per-team wiring. The change I would actually open the release notes for is elsewhere in the list: the control-plane fix that stops a restart making its own recovery worse.
The teams that get the fourth signal right will not be the ones who turn it on first. They will be the ones who already know which question they could not answer last time.
Before you go
Reply and tell me the observability problem on your desk this week. I read every reply. The best ones become a Signal check, and if it is a bigger piece of work, that is exactly the kind of thing I help teams with.
How useful was today's Signal?
Allan
PS: The chapter I give away is the one about who owns what when it breaks. A profile will tell you which function spent the money. It will not tell you whose line item it is, and that is the argument that actually costs you.
Everything in one place New here? Start with the free chapter. | ||||
More from Metrics & Mayhem
|
The Ultimate Claude Code Guide to ship like Anthropic engineers
AI will write 90% of code by the end of 2026, and only 10% of developers will stay relevant.
The engineers in that 10% aren't smarter. They just know how to use AI.
We put together the exact playbook Anthropic engineers use to make sure you're in the top 10%.
Sign up for The Code and get access to:
The Ultimate Claude Code Guide 2026 — 50+ tips and tricks to code 5x faster
The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day