This website uses cookies

Read our Privacy policy and Terms of use for more information.

Here is a quarterly business review. It is an illustration I use, not a client, and every figure in it is made up for the purpose.

Engineering presents first. Uptime for the quarter: 99.97 per cent. Every service level met, green across the board, and the VP of Engineering is pleased and a little relieved. Then Customer Success stands up. Checkout-failure tickets are up 12 per cent on the quarter. Satisfaction for the purchase flow is down. Three enterprise accounts raised it at their last contract review.

Both presentations are accurate. Neither can explain the other's number.

Nobody in that room lied, nobody was careless, and the dashboards were telling the truth about the thing they were built to measure. The trouble is that the thing they measure is not the thing the customer came for.

Both presentations are accurate. Neither can explain the other's number.

Two true numbers can still disagree about your quarter

Uptime answers a question about the system: was it running? A failed checkout answers a question about a person: could they finish what they came to do? Those sound like the same question until you notice they are owned by different people, held in different systems and reviewed in different meetings. Often the only thing connecting them is the hope that if one is healthy the other will be too.

That hope is the gap. A system can be up in every sense your tooling understands while a payment step times out for one customer in thirty, and nothing on the engineering dashboard moves, because the engineering dashboard was never asked about payment completion. Availability is an average over requests. A customer experiences a journey, and a journey is only as good as its weakest step.

We had complaints that some flows weren't working. The service wasn't used much. Its error rates were low because its user base was low. We were relying on dynamic baselining, where the tool learns what normal looks like, and on a service that quiet it wasn't kicking in. It was showing green on the dashboard, and it wasn't. We put it into hypercare and went through the response codes, the 200s, 400s and 500s, to look at the patterns, literally from the ground up. That was a big learning for us.

That is my experience, and it is one person's. I could be generalising from too few estates. But the mechanism does not depend on that one service, which is why I am prepared to state it as a position.

Uptime is a supplier's number, not a customer's

Here is the position. Uptime is the number a supplier reports about its own service. Keep it, because it is useful. It says nothing about whether your business worked.

If that sounds harsh to your engineering lead, it should not. Most were never asked to measure the customer's journey, and have no budget for it. They were asked for availability, and they delivered 99.97 per cent of it. The failure is upstream of them, in a leadership team that accepted a technical number as the proxy for a commercial one and never asked what sat between the two.

Charity Majors, in her Observability: A Manifesto for Honeycomb, defined observability as the power to ask new questions of your system without having to ship new code or gather new data. Monitoring watches for the failures you predicted. Observability lets you investigate the one you did not. For a leader, I would shorten it again. Monitoring answers "is the system healthy?" Observability answers "can the customer complete the transaction, and if not, why not?"

Chapter 2 of my book, Metrics & Mayhem, builds the case that the second question is the one a technology leader is actually accountable for. The logs, metrics and traces that most briefings call the three pillars describe inputs. None of them tells you whether the person on the other end finished.

One outcome, its signals, and a name

The fix is smaller than people expect, and it does not start with a tool.

Pick one outcome your CFO would recognise. Payment completion rate is the classic: the share of customers who start a checkout and finish it. Then list every technical signal that can move it. Checkout load time at the 99th percentile. Payment gateway latency. Errors from a third-party processor. Authentication availability. Cart session timeouts. Any of those can drop the completion rate, and unless they are tied to it, the person who gets the call starts guessing, moving between three or four dashboards owned by three or four teams.

Now mark which of those signals your team can actually see today. Expect the honest answer to leave some blank. A signal sits with another team in another tool. A signal is not measured at all. The blanks are the finding, and they are cheaper to fix than the quarter they cost you.

A stacked diagram headed "One outcome, its signals, and a name". At the top, a card labelled "One outcome" reads "Payment completion rate: the share of customers who start a checkout and finish it." An arrow points up into that card from a list headed "Signals that can move it", with a note above the status column that reads "Example estate", because the statuses are illustrative. Five signal cards follow, each with a status tag: checkout load time at the 99th percentile, can see; payment gateway latency, can see; errors from a third-party processor, another team's tool; authentication availability, can see; and cart session timeouts, not measured. The three signals the team can see carry outlined green tags, and the two blanks, another team's tool and not measured, carry solid amber tags so they stand out. Below the list, a card labelled "A name" asks "Who owns the join?" and answers "A person, not a team", then explains why: "Engineering owns the signals, product the experience, finance the revenue. In practice, the join falls to whoever is on call that night." A closing band reads "The blanks are the finding. Name one person who owns the join." The footer reads "Mastering Observability, masteringobservability.com".

One outcome, the five signals that can move it, and the person who owns the join. Statuses show an illustrative example estate, not a client.

Then comes the part most organisations skip. Write down who owns the connection between the outcome and those signals. A person, not a team. Engineering owns the signals, product owns the experience and finance owns the revenue, so the join between them belongs, in practice, to whoever happens to be on call that night. Call that what it is: an accident waiting for a Tuesday evening.

The blanks are the finding, and they are cheaper to fix than the quarter they cost you.

The five-minute test

You can find your own gap in twenty minutes, with a pen, and without buying anything.

Take your last major incident. Write down how long it took, from the moment the outcome moved to the moment someone could say why, in minutes. If the answer is more than five, or you had to ask someone to find out, you have located the gap.

Treat five minutes as a target and nothing grander. I chose it because it is short enough that a team has to have the pairing ready in advance. If your result is forty minutes because the cause sat in a service the responding team could not see, that is a normal result, and a useful one. Don't use it to embarrass a team. Its job is to turn a vague feeling that the dashboards and the business disagree into a number you can put in front of the people who fund the fix.

It also changes the quarterly review. The review at the top of this piece went wrong because each side brought an answer and nobody brought a question. If both sides had started from one outcome and asked how fast the business could explain a movement in it, the 12 per cent would have been a conversation about a known blind spot rather than a surprise.

Where this goes wrong

I would rather tell you the failure modes now than have you discover them in month three.

It becomes a data-integration project. Someone builds a pipeline that pushes infrastructure metrics into a business dashboard, ships it and declares victory. That is reporting. A view that refreshes every five minutes is a faster report, and it cannot help you investigate a failure you have never seen.

It sprawls. Pairing every outcome at once is expensive and exhausting, and it almost always stalls. Start with the one outcome that matters most to revenue, prove it on a real incident, then earn the right to add the next.

And nobody owns the join. If you take only one thing from this, take that one. When the pairing points at a third-party processor, someone has to decide what happens next, and that decision has to be made before the incident, not during it.

What to do before your next review

Before the next quarterly review, name one business outcome your stack should be able to explain and currently cannot. Then ask your engineering lead a single question: when this number last moved, how long did it take us to know why?

Whatever they say, you will have learned something no uptime figure was going to tell you. A strong uptime number tells you the supplier did its job. It cannot tell you whether the customer got through. Only the second question tells you that, and it is yours to ask.

If you want the whole test laid out, with a filled-in example and the questions to take into the room, The Outcome Question is free: one question, twenty minutes, and what to do on Monday.

 

Work with me

An honest read on what your observability is actually doing.

If you lead observability in a regulated enterprise, I run a fixed-scope Observability Assessment for senior IT and engineering leaders. It ends in a written roadmap and a readout, not a sales deck.

See how it works →

Not ready to talk? Start with a free chapter of Metrics & Mayhem.