The Fourth Signal Was Hiding In Your CPU Bill

You have almost certainly had this conversation with a graph. CPU on one of your busiest hosts has climbed to eighty per cent since the morning stand-up. The graph is confident about that number and completely silent about the reason. It tells you the machine is busy. It will not tell you what is making it busy.

For years, that silence was simply the deal. Metrics and traces were never built to answer the next question, so you learned not to ask it of them. You saw the symptom at high resolution and then went hunting by hand: reproduce it locally, or attach a profiler to a process that had usually calmed down by the time you got there. Then the deal changed.

Two things happened to OpenTelemetry this year. In March, its profiles signal entered public alpha. In May, the project reached Graduated status, the CNCF's highest maturity level. The graduation got the headlines. It is the less important of the two.

The graduation is a maturity badge, and it is a real one. It means the open standard for telemetry has grown up, and building on it is now the conservative choice rather than the brave one. Useful to know. It will not change your week. The profiles signal will. It makes continuous profiling a first-class citizen of the same pipeline you already run for metrics, traces and logs. That is the part that changes what you can see on a Tuesday morning when the graph is busy and unhelpful.

Metrics and traces are superb at what and when. High CPU. A latency spike at nine in the morning that clears by half past. What they cannot do is point at the specific function, on the specific host, that is spending the cycles. You get the symptom, beautifully rendered, and then you guess.

I have sat on the expensive version of this. A multi-bridge call at a tier-1 bank, split across three rooms: leadership in one, the incident itself in the second, engineering in the third. Every room could see the damage. None of them could see the cause, an API gateway whose agent had never been installed properly. The component everything depended on was the one nobody was watching, and the impact was massive.

That gap is where the expensive part of an incident lives. Not in noticing the problem, which the dashboard did for you, but in the slow, manual work of reproducing it somewhere you can attach a tool to it. Whole-system profiling is the signal that closes the gap. It samples the running stack across every process on the host and tells you which functions are on the CPU, continuously, in production, rather than in a lab you rebuilt to imitate production.

The reason profiling was never a standing signal is economics, not usefulness. Traditional profilers were heavy and language-specific. You instrumented one runtime, you paid a genuine performance tax for the privilege, so you ran it briefly, on demand, when you already suspected where the problem was. A signal you only switch on once you are already in trouble belongs in the forensics drawer.

What changed is the arrival of low-overhead, whole-system profiling built on eBPF, which Elastic donated into OpenTelemetry and which now rides in the Collector. It profiles the whole host, across language runtimes, without asking you to hand-instrument each service, and at an overhead low enough that leaving it running all the time is a defensible default rather than an indulgence. The exact overhead percentages are the kind of number each vendor quotes about its own agent, so I will not adopt one as fact here. The documented characteristic is the part that matters: low overhead, whole system, always on.

And a signal that is always on is a signal you can look back through after the fact. When the cost of leaving something running falls far enough, you stop treating it as a special occasion and just leave it on. That single shift, from on-demand to continuous, is the whole story.

The part we miss: this is a cost story as much as a debugging one. When you profile the whole system rather than one service you happen to have written, you see everything on the host that is spending CPU. Including the code that is not yours.

I heard a clean example of this on OpenObservability Talks, from Israel Ogbole of the OpenTelemetry Profiling SIG (Dotan Horovits wrote the episode up on 29 May). A process the cloud provider had injected into a customer's production environment turned out to be consuming around 10 to 15 per cent of the CPU. Invisible in any per-service view, because nobody had instrumented it, and why would they. It only surfaced when someone looked across the entire host at once and asked the blunt question: in total, what is burning these cycles?

You are already paying for every cycle on that host. Your cloud invoice does not itemise which of those cycles did useful work and which were spent by a third party's idle loop, a runaway sidecar, or a library doing far more than you ever asked of it. Whole-system profiling is the first signal that lets you read that invoice line by line.

I have also watched the bill go unread. At a tier-1 bank in Asia, a database cluster, three or four servers from memory, ran at peak CPU for more than three months. There was no alert on any of them, so nobody picked it up. Then one of them went down, and it caused absolute havoc.

Two caveats before anyone gets carried away.

First, the profiles signal is alpha. The people building it say plainly that you should not rely on it for anything critical yet. The format can still change, and the backends that ingest it are still being built. Alpha means evaluate, not adopt. If you need stability, the line to wait for is general availability, and no date has been announced for it.

Second, profiling costs something. It is one more signal to store and make sense of, and a flame graph nobody ever opens is just more storage cost with a prettier shape. The value sits in the one afternoon somebody spends reading it against a real cost question. Collect it and ignore it and you have bought yourself an expense, not an answer.

The graduation is the safe, boring news, and boring is high praise for infrastructure. The open standard for telemetry has matured, and betting on it is now the sensible default. Good. When I talk to senior leaders about standards, the line is the same every time: cut the noise where you can, and put a proper observability practice underneath everything. Without that foundation, everything else is useless. But the maturity badge is not the thing that changes how you work. The fourth signal is.

For years the honest answer to "why is this host so busy" was a shrug and a lab reproduction. That answer is changing, and it is changing inside the same pipeline you already run for everything else. What you get out of it is the ability to read a bill you have been paying in full, every month, without ever seeing the line items.

So here is the Monday version. Pick your single busiest host. Not the fleet, one host. Stand up whole-system profiling against it in a throwaway setup and leave it running for a day. Then put two numbers side by side: what that host costs a month, which whoever owns the cloud bill already has, and the share of its CPU going to code nobody on your team wrote, which the team that owns the host can now read off the profile. The first is the outcome. The second is the early warning, and until now nobody could see it. That is the moment the fourth signal stops being a press release and starts being useful.

There are over 3,000 of them in The Shift’s vault, already vetted, ready to explore.

Subscribe and get instant access to the tool vault, a 1000+ prompt library, and free AI courses built for people with actual work to do. 

Plus, the daily newsletter that keeps you sharp on everything moving in AI in under 5 minutes a day.

They’re also giving away a free 1-year Claude Pro subscription to 3 subscribers. 

Subscribe for free to enter. All free. All in one place.

Subscribe to The Shift

Continuous profiling samples the call stacks of running code at a steady rate, in production and all the time, then aggregates the samples so you can see which functions are spending CPU and, depending on the profiler, other resources such as memory. Instead of attaching a profiler once something has gone wrong, you already have the record. The OpenTelemetry documentation describes a profile as a collection of samples showing where applications consume resources during execution.

Not for anything critical. The profiles signal entered public alpha in March 2026, and the OpenTelemetry project says plainly that it should not be used for critical production workloads yet. No date has been announced for general availability. Evaluate it now, on a host you can afford to experiment with.

Metrics tell you how much and when. Logs tell you what the code chose to record. Traces tell you where in a request the time went. A profile tells you which function, on which host, was actually on the CPU. The first three see the symptom; the profile points at the code.

eBPF lets a profiler sample from inside the Linux kernel across every process on a host, without changing or instrumenting the services themselves. That is what makes whole-system, always-on profiling cheap enough to leave running. The OpenTelemetry profiler is the eBPF agent Elastic donated to the project, and with the alpha it runs as a Collector receiver. The eBPF byte-size covers the kernel side in plain English.

Graduation is the CNCF's top maturity level, and OpenTelemetry's was announced on 21 May 2026. It is a judgement on the project as a whole, on its governance and its adoption. It does not change the status of individual signals. Profiles is still alpha.

Not by itself. It shows you where the CPU goes, including code you did not write, which is the first step. The saving comes when someone reads the profile against a real cost question and acts on what they find.