Transcript 0:00 Right. You were told to instrument three pillars. Metrics, logs, traces. And it's some godforsaken time in the morning. 0:13 The question you always didn't predict that comes in ends up scattered across three tools that you know darn well are not talking to one another. 0:22 Today, I wanna show you the argument that says that those three pillars were never the right primitive in the first place. And then I wanna explain honestly why I think 0:35 the people who are making that gesticulation could be right. [upbeat music] So here's, here's where we are. It's early in the morning 0:53 and there's something hot. There's something wrong. Not catastrophically wrong, but wrong in the way that it's much harder to deal with than the usual issues. Your error rate is up, not everywhere, but it's just up. 1:10 It's really noticeable. And the question you need answered is a very specific one. It's not hitting every customer or a handful. It's only on the new checkout path. 1:27 It's only on mobile. It's only one region, and it's only on a specific set of accounts that have been gone live since the migration. And here's the one thing about that q- type of question. 1:43 You know darn well it's not on any dashboard at that particular point in time. No one can predict these multiphasic issues, multi-hop issues, because nobody could ever have had predicted it. 1:59 And that is the nature of these interesting failures. They always arrive in a combination nobody th- thought to map out in advance. So, so you do what we all do. 2:15 You go into logs, and you grep, and you find something looks suspicious. Then you go into your tracing, and you hunt for the slow span. It might be related. 2:24 Then you go back to the metrics to see whether that shape matches. It's the route we all do. Three tools, three query languages, generally three browser tabs. 2:44 And you are the integration layer. You're holding the customer ID in your head, copying it between windows, trying to keep the thread of the story that is scattered across three systems that 2:59 were never designed to talk to one another. Because that's not you... But, but that's not even the worst part of the situation. The worst part is that your metrics already threw away the answer. 3:18 Because somewhere along the line, somebody looked at the cost of your metrics and did the responsible thing. They stripped out some of the high cardinality fields, the, 3:31 the customer ID, the build version, the region, the feature flag, all the things that we do in a responsible way. This is perfectly sensible to a cost decision, and it means the, the detail that you now desperately need 3:49 was never in the data the moment it was collected and got deleted before it was even stored. You're not missing the answer because nobody looked. 4:01 You're missing it because it was aggregated away in a cost review eight months ago. So here's the sentence the whole argument rests on right now. So the primitive should be a wide row, 4:17 structured event per unit of work, and the pillars should be derived from it, not collected separately. Now, let me give you the picture because this is the one that makes it all click. 4:36 Imagine something goes badly wrong in a building where there were three witnesses. Stick with me. The first one only watches the doors. She can tell you exactly how many people went out, how fast. 4:54 That is your metrics. Counts and rates, very precise, no context whatsoever. The second one only heard what was shouted. He can tell you what was said, what time, but not who was there. 5:12 This is your logs. Rich in detail, but only the detail somebody chose to say it out loud. The third only tracked... The th... 5:30 The third only tracked how many people moved from room to room. She can tell you the path and where things slowed down. That's your traces. 5:42 Now, every one of those witnesses is honest, everyone is accurate, and everyone saw one fragment, just one. So to work out what actually happened, you interview all three. Of course you do. You write it down. 5:58 You try to stitch three partial accounts together into one story, and you can only ever ask each witness the questions From their vantage points and what it allows. 6:14 So now imagine instead you only had one witness who saw the whole thing start to finish. Who noticed who was there, what they were carrying, what time it was, what door they used, and how it all ended. 6:29 Would be nice, right? And crucially, a witness you can keep asking questions, not just one. Hear the answer and let that answer suggest the next question. The... 6:48 That's a wide event. One rich account of what happened you can interrogate afterwards. Not three fragments to, you know, to reconcile at 3:00 in the morning. One witness who saw the whole thing. 7:06 Now, just hold onto that because w- we're gonna come back to it. Right. Named parts. There are four of them in this order. Number one, the wide event. 7:22 A wide event is a single structured record of one unit of work, one request, one job, one transaction. That is the whole definition, and it's worth sitting with it before we go any further. 7:39 What makes it wide is how much you put on it. Not five fields. Hundreds. The customer ID, the region, the build, the build version, every feature flag that was on, 7:53 how long the database took, how long the external call took, the endpoint, the device, whether it errored and how. Everything you knew in that moment is captured in one row. Everything. 8:12 It's not a log line. It's not a metric. It's just the full context of one thing that happened at that particular point in time. Two, one we all know very well, cardinality. 8:29 This is the word that decides the entire argument, so let me really define it as accurately and as plainly as I can. Cardinality is simply how many distinct values a field can have. 8:44 Region is low cardinality. You can have a handful. Customer ID is very high cardinality because you can have millions. 8:54 And traditional metrics platforms struggle badly with high cardinality because of how they store things. 9:01 They pre-aggregate every u- unique combination of labels becomes its own series to keep, and the cost climbs deeply as you add more dimensions. We know this. We've seen the bills every month. 9:16 So teams do the rational thing. They strip out the high cardinality which they feel that they don't need, 9:25 which are, of course, exactly the fields that let you answer which customer, which build, and which region in this example. So hold both halves of this. 9:39 High cardinality is where the answer lives and is also where the cost lives. That's the tension of the whole story. Number three, observability 2.0, which really means to 9:57 derive rather than collect. So here is the actual claim, right? Stripped of all the marking you're here. This is it in a nutshell. 10:07 Instead of collecting three separate things, you store one thing, the wide events, and then your metrics and your traces become views computed over that store when you ask for them. You stop collecting the pillars. 10:22 You start deriving them. And let that sink in. That's it. That's the whole point of what people mean, in my opinion, of observability 2.0. Four, 10:38 query after the fact. And this is the payoff and the reason anyone bothers with this whole thing. 10:50 'Cause the raw detail is still sitting in your events, you can ask a question nobody designed for it at 3:00 in the morning 11:00 about a combination that happened that nobody ever predicted that's never gonna be on any dashboard. You do not need to have known the question in advance. 11:09 That's the capability you're now buying with these, with these wide events. Still with me? Good. Because now I have to be honest about the other side of this argument. 11:21 Because this is a live argument, not a settled fact, and, um, I can't pretend otherwise. So let me credit it properly first before we go any further. The clearest 11:37 advocates of this position are our friend Charity Majors from Honeycomb, and she has argued this publicly for years in her writing and her talks, and she's... W- we all know she's co-authored on this subject. 11:53 And credit where credit's due, she was making this case long before it was fashionable. A good deal of the vocabulary the rest of us now casually frame this work. 12:07 S- so when I push back, understand that I'm pushing back on the position, not the person. Right? To be very, very clear, I don't want this to be misconstrued Here is where I differ from this argument. 12:18 The framing that the three pillars are dead is, in my view, too strong for most estates that I've worked in. Mostly financial and a couple of governments, but you know, it's a bit strong. 12:34 Now, I have to correct my own advice here too, because for years, the rule I gave people straightforward, instrument the three pillars, instrument the three pillars, that's the standard, let's get on with it. 12:49 This was years ago. Obviously, there's profile and all that now, but years ago. 12:58 Now, that rule has a failure mode, and the failure mode is precisely the three in the morning I opened with, is you pre-aggregate for cost, you lose the high cardinality in detail, and you cannot answer the question that nobody predicted. 13:14 That's it. So, the rule, let's update it. Keep the pillars where they generally earn their place, but stop treating pre-aggregated metrics as the place where the detail lives, because it's not. It never was, guys. 13:30 You know that. Wide events for the questions you cannot predict, pillars for the ones you can. Two more honest things quickly. First is cost. Wide events are not free. 13:45 Storing each rich record for every single request is a real bill. And it's worth noting that vendors make this argument most loudly are also generally the people who are going to store it for you, all right? 13:59 So be careful. It does not make them wrong. It just means that you should do your own maths, guys. You know this. The second is maturity. 14:13 If your team knows Prometheus, Grafana, and nothing else, then moving to an events-first model is a retraining project, not a configuration change. Budget in the people. I always advocate this. You know I do. 14:29 Not just a platform. People first, always. So what do you actually do with this? So if you're in the pit, SRE, you're up against this daily struggle, then 14:48 do not sit there and rewrite all your instrumentation. That's a two-year product, and it will die in month four. So don't even try. Take one endpoint, the most critical one, and emit one wide event 15:06 request alongside one you already have. Everything you know about that request in one row, high cardinality fields included. Just one endpoint known. 15:20 The next time you have an incident on that path, just try to answer it from that one event alone. Obviously, you've got to remember it's there and everything else. 15:29 But if you have a troublesome journey, then you know you're in the right place. That one experiment will teach you more than any amount of arguing about data models in Reddit, in the internet. Trust me, it will. 15:48 Now, if you're at the table making the big bucks and the decisions, then this is a cost versus flexibility decision. And it's not a religion, right? 16:07 The pre-aggregated metrics are cheap, and they answer the questions. You already know what to ask. Wide events cost more because they answer the questions you had no idea were going to come down the pipe. 16:20 So think of it like that. So the question to put to your team is, what fraction of our three in the morning hours goes on questions nobody has designed a dashboard for? That's a great place to start. And 16:40 be interested in that fraction as well. Because if the honest answer they were going back to is most of them, you're already paying for that flexibility anyway. 16:51 You're just paying for it in engineer hours instead of the storage. And engineer hours at three in the morning are the most expensive you'll buy, right? 17:04 And if you took this seriously, here is what you would actually build. Not a migration. You build a default. You would decide that every new service emits one wide event per unit of work from the day it's written. 17:23 Not retrofitted later. Not as a special project. As the shape of how you instrument things by default. By default. 17:33 You would keep the high card fields on it deliberately because those are the fields that answer the questions that really matter. That you don't know are going to come. And you would put one rule around cost. 17:50 Because cost is what erodes this. You never strip a field to save money without writing down each question you have just made unanswerable. Do that and you stop having to choose sides in this argument at all. 18:06 You have the raw material for both at your disposal. Take the analogy with you. Because you're not really choosing between tools here. You're really not. 18:22 You're choosing how many weaknesses you get and how long you get to question them when something happens. Three weaknesses who saw each fragment or one that saw the whole thing and will still answer questions 18:41 at three in the morning. Still that framing for your next architectural review. You do not. You don't have to credit me, right? And that, my friends, is Tech Tuesday. 18:58 I'm Alan Mann. And if you like this, you know the book is out. Metrics and Mayhem, a CTO's Guide to Observability That Actually Works. It's out. It's all live. You can get your free chapter at masteringobservability.com. 19:13 And I look forward to talking to you again. Keep the topics coming in. I love doing these. And next is a signal drop next week. So much love to everyone. Much love. Be safe. Be safe.
