Transcript 0:00 [beep sound] Good morning from a very hot UK. Uh, a nice steady 35 degrees here. A very uncanny weather system here in the UK, especially for the end of June. I'm Alan Mann. 0:17 I lead IT ops focused on observability and people leadership under pressure. My book, Matrix to Mayhem, is out now, uh, and that deals with all of these topics. 0:26 So I was having a discussion with a client this week, and that's where this thing comes from. 0:31 And we were ta-- I was asked a question, "When you last had to prepare for a l- for a, for a launch of a product from an observability perspective, what did you do? How did it work?" 0:41 And all that, all that kind of good stuff. So after we got into those details, it sort of reminded me of, uh, one particular launch I, I'll never forget. I think... And of course, for the wrong reasons. 0:53 It was a new service. It was a big customer. But everyone was confident. We tested it. We'd load tested it, which was quite rare back in those days. 1:02 All the dashboards we had were green, and about 40 minutes after go live, it started to wobble. We started to see some degradation, some latency, but the team just fell apart. Arguing and sort of panic. 1:15 Not because the wobble was huge. It wasn't at first. They fell apart because no one had told them what to expect. Basically, no one had told them it was coming. So in their heads, a good launch was a quiet launch. 1:27 So the first sign of trouble didn't read as, "Here's the hard moment. This is normal." It read, "It's all going wrong." And the panic did more damage than the bug ever did. 1:38 So I was listening to a high-performance podcast this week, and one of the presenters told a story about his 10-year-old boy before a cup final. 1:49 The boy was really nervous, and instead of the usual, "Relax, son, have fun," the dad said something completely different. He said, "You're gonna have to suffer it today." So the kid kinda looks up at him and, "Yes. 2:02 What do you mean, Dad?" 2:04 And the dad said, "I promise you, there will be a moment in this game where you feel like you're not gonna win, where all the thoughts will come into your head, your mom, me, your dad, and your teammates, and what people will say at school. 2:16 That moment is coming. I just want you to know it's coming." And it did come. They went three, three-one up, then four-three down. The kid suffered exactly as promised. But here's the thing. 2:28 He'd been told, so when the hard moment arrived, it wasn't a catastrophe. It was the moment the dad mentioned. He played all the way through it, and the team that was winning suddenly wasn't. 2:41 And the ones who fell apart because no one did, you know. So kids who fell apart did because no one, b- because obviously no one prepared for it. 2:49 That's the whole game for us on a launch night or on an upgrade, or it could just be a, a change in patterns, Black Friday, whatever. You can't promise your team a quiet night or a quiet day. 3:02 Everyone who tells you the launch will be smooth is either lying or hasn't shipped fuck all. What can you do? The one genuinely useful thing is to tell them it's going to hurt them before it hurts. 3:15 So this leads me on to the discussion I had with the client was talking about pre-mortem. So that's basically what a pre-mortem is. Not pessimism, not bad vibes around the launch. 3:25 It's the dad on the way to the cup final, sitting down beforehand and saying, "Right, here's three... It's 3:00 in the morning, two days from now, and this has, this, this has gone wrong. What broke? Who's awake? 3:40 What are you staring at? And what can you reach for?" You name that hard moment in daylight where everybody's chilled or everyone's during their day. The light is on no one in particular. No one's panicking. 3:51 So when it does arrive at 2:00, 3:00 a.m., it gets recognized instead of feared. And the difference on the night is enormous. I cannot stress this enough. 4:00 Teams that rehearse the bad moment treats it, treats the first wobble as just information. "Oh, here it is. It's the thing we talked about. Here's the runbook we wrote exactly for this." 4:11 The team that assumed it would be fine treats the wobble as the sky is collapsing around them. Same event, completely different night. 4:19 The only variable is whether someone has had the nerve to stand up and say it out loud beforehand. "This is going to be a hard night at some point." Now, I didn't do that on the launch I told you about. 4:31 So the team, a quiet night because we wanted all that. That wasn't leadership. That was me managing my own nerves and calling it confidence. So I've stopped trying to reassure teams that things are gonna go smoothly. 4:45 It sounds kind, but it really isn't. It leaves them defenseless in the moment reality decrees them with the reassurance, the kinder thing. The harder thing to do is to say it out loud, "This will get rough. Where... 4:59 Here's roughly where, and here is what we're gonna have to do. We're gonna have to roll our sleeves up. We're gonna have to do it this way, that way." So, as always, here's the hard stop for this week. 5:10 A team that's rehearsed a bad night doesn't panic in it. A team that pretended it wouldn't come does. So here's the habit for next week. 5:20 Before your next launch or significant change or, or whatever it may be like, we've got a lot of Black Fridays coming up, um, and a lot of Amazon, let run a ten-minute pre-mortem. Just ten minutes. 5:30 Get the people who will be on those calls in a room, real or virtual, ask one question and write down the answers. At 3:00 in the morning, this change has gone wrong. Tell me the story. What's the first symptom? 5:42 Which dashboard shows it? Who's gonna g- who's gonna get called? What's the first thing they will try? What's the rollback? And who's, and who's allowed to call it? 5:53 You're not trying to predict or exact failure or extract failure or make people look bad. You'll, you'll be wrong about the specifics. That's fine. 6:01 You'll be-Try to make the shape of a bad night familiar so that when one comes, no one will. Your teams meet it with the, the recog- the recognition instead of fear. So they recognize it instead of running away from it. 6:16 But you, you, you can't give your team a promise that nothing will go wrong. Nobody can. Come on. We, we-- That's, that's why we do what we do. But you can give them something better than a promise. 6:27 You can give them the bad night in advance on your terms while everyone's calm, so the real one, when it hits them, finds them ready. 6:38 [outro Jingle] I've got two questions for today, and they both deal with this subject matter. So question number one. Uh, we run post-mortems, but they turn into a doc nobody opens or is bothered about on the night. 6:50 How do we stop that? Yeah. Uh, so pre-mor- pre-mortem that becomes a dead document. So yeah, I've done that. You, you sit everyone down, you fill in the lovely template. Everyone feels responsible, right? 7:01 And then, you know, 3 AM a few weeks later or on a weekend later, nobody's opened that document because the document was never pointed at, right? No one knows it was there or it was rehearsed. 7:12 So you actually want the team that goes, "This. So we talked about this," right? Instead of freezing. 7:17 So honestly, I'd almost, I'd almost bin the template and get people who actually, actually gets on the call in the room, like I said previously, in there for 10 minutes, and get them to say it out loud. 7:29 Get them to say what you've discussed out loud, more than just writing it down. I think we all kind of write things down, and they're easily forgotten. It's like a process, a tick box. 7:38 So you wanna try and sort of get away from that. And then, then take the two or three things that they reach for and put them where they'll find them half asleep, right? [laughs] And scared. 7:50 So, you know, your, your pre-mortem produces a, a beautiful artifact. And the team still panics because, you know, they find their own document, or they don't know where it is, or they weren't involved. 8:02 So yeah, so while it is good to have a document, document's not everything. I think just make sure you get the right people in the call beforehand, whether it's a few days before, like I said. 8:13 Make people say it out loud. Create that environment, that safe space where people can be honest about their fears about what's gonna happen. Next one. I'm on a small team with no time for game days or chaos drills. 8:25 Is this, is the pre-mortem still worth it? Honestly, the pre-mortem is the cheap version. That's the whole point of it. 8:33 So, so game days, chaos drills, you know, you know, killing a node every Monday, that's brilliant if you've got the time and the operational maturity. And most teams don't have that, let's be fair. 8:44 A pre-mortem is about 10 minutes and a conversation. No platform, no budget, no, no, no permission required. You just need the people who, who you'll get on the call when it breaks, right? 8:56 You know, being really open and honest for 10 minutes. Someone tells you that-- If someone tells you that, that that's negativity, then push back gently. No. 9:05 Nothing's negative about pilot checking and sort of, you know, gauging where we're gonna be. Naming, naming it where it hurts isn't pessimism. It's the cheapest assurance you can ever buy. You know, it's just... 9:17 So, so, so no. So you don't need the fancy stuff. You just need 10 minutes, uh, and that's it. Uh, to keep the questions coming in on all the themes. 9:24 Like you say, this, this, this whole pod this week come from, um, a question I received on Twitter. So, so thanks for that. So as a, as I close out, as always, thank you ever so much for your support. 9:37 You know, you know where, you know where to find me on all the socials. Please do me a favor. 9:42 As always, and I, I know you hear this from, from everybody, but if you really do like it, please share, please subscribe, and keep the conversations going with me. 9:51 There's so many options for you to do that, so many mediums to do that. I'm really enjoying the conversations lately. And before I finish, a big thank you to my kids for a great Father's Day present. 10:00 Literally a T-shirt with the book cover on it. So I'm very proud. So thanks, kids, for that. It's given me the idea, so for merch. 10:06 So, you know, like post in the bottom, would, would you buy some of these phrases on a T-shirt? Do you think it warrants it? Thanks, and remember, stay safe and more signal than noise. [outro jingle]