Transcript 0:00 Every observability vendor spent the last two years bolting an AI system into their dashboard, and every one of them hallucinates. Today, I'm gonna tell you about a standard that might quietly fix all that. 0:12 And if you already fought the wars before OpenTelemetry existed, you're gonna recognize this fight for sure. [electronic music] You're listening to Tech Tuesdays from Metrics and Mayhem. 0:30 Ten minutes to make sense of one thing that landed on my desk this week, whether you're in the pit or whether you sit at the table. So here's the world we're living in right now. 0:38 Every observability vendor you can name has bolted an AI system into their product in the last two years. Datadog has one, Dynatrace has one, Splunk has one, Grafana has one, New Relic has one. 0:51 Every one of them is a chatbot sitting on top of a dashboard powered by an LLM, and the vendor has plumbed in their own data using their own custom integration. 1:02 Because every vendor built their own bridge between LLMs and their, and their own data, every vendor's assistant is talking to a different set of tools in a different way with a different auth pattern. 1:14 If you're a practitioner, the AI in tool A cannot answer questions about tool B, even when you have both tools on the same screen. 1:24 If you're a CTO, you're paying for three overlapping AI assistants, and none of them can see the whole picture. So the assistant that was supposed to reduce the alert fatigue is one more thing to check. 1:35 That's the world MCP just landed straight into. So here's the thing, and I want you to understand before we go any further. LLMs on their own are useful for exactly one thing, guessing the next word. That's it. 1:51 Every action an LLM takes on your behalf, every dashboard it reads, every log it passes, every incident it drafts a runbook for is because someone plugged a tool into it. 2:04 And the way each vendor plugs their tools in is the reason no assistant knows your whole stack. Now, if that sentence made you think of another fight we've all had in observability, you're right. 2:17 Let me walk you through it. You see, we've been here before. Ten years ago, if you wanted your application to send telemetry to a monitoring backend, you picked the vendor's SDK. Datadog had one, 2:31 New Relic had one, AppDynamics had one. Every SDK was different. Every one had its own API, its own tags, its own idea of what a span was. If you wanted to switch backends, you rewrote your whole instrumentation. 2:47 If you wanted two backends at once, you ran two SDKs in parallel and hoped they didn't fight each other. 2:52 I've been in rooms where the reason a migration stalled for a whole year was that nobody wanted to touch the, the backend instrumentation. 3:01 Then OpenTelemetry landed, and OpenTelemetry quietly is one of the most boring and most important things that has ever happened to observability. 3:13 Because what OpenTelemetry actually is underneath all the marketing is a standard, a standard of how instrumentation talks to a backend. 3:23 One SDK on your application, one protocol on the wire, OTLP, one collector in the middle, and on the far side, any backend that speaks to the standard. The application doesn't care which backend. 3:39 The backend doesn't care which application. MCP is doing that. For LLMs and tools, an LLM on its own cannot do anything without your Datadog account. 3:51 An LLM with a Datadog integration can, but an integration written specific... in a, in a specific way by a specific team for a specific model. MCP is a standard released by Anthropic that says, 4:05 "This is how tools describe itself to an LLM, and this is how an LLM asks the tool to do something. This is what a response looks like." Standard on the wire. 4:17 And any LM that speaks MCP can talk to any tool that speaks MCP. That's it. That is the whole news. Right. Named parts. Learn them in this order, okay? 'Cause there's four. So one, MCP client. 4:34 That's the thing the LLM lives in. Claude, Claude Desktop is an MCP client. Uh, Cursor is one. VS Code with client is one. The client is what the human types into. Two, the MCP protocol. That's the wire format. 4:52 It is the OTLP of this world, and a spec that says how the message gets exchanged between client and server. You mostly don't need to think about it, and I'm, I'm not technical. You know that. I don't think about it. 5:05 But it's what makes any client talk to any server without custom integration. Three, the MCP server. Now, this is the interesting part. The server is built by the tool provider. Datadog will build a Datadog MCP server. 5:21 GitHub already ships one. Slack already ships one. Dynatrace already ships one. The server exposes what the tool can do in the way that an LLM can understand it. 5:33 So when you ask assistant to check your Dynatrace dashboard, the assistant talks to the Dynatrace MCP server, and the protocol gets the real answer. It's not a guess, and it's not a hallucination. 5:47 An answer that come from your actual data. Four, the underlying service. The thing the server sits in front of. 5:55 Datadog itself, Dynatrace itself, GitHub itself, AppDynamics itself, your database, whatever the tool actually is. Client, protocol, server, service. If you take one thing away from today 6:10 Take those four words in that order. Client, protocol, service, server, service. Okay, so are we good so far? Right. Now, I have to be honest with you. It's not all sunshine and rainbows. This is early. 6:23 The spec is still moving really aggressively. The clients we're using right now are inconsistent. The servers we're using are in their infancy and are inconsistent. 6:33 Setting one up on your own machine right now means editing JSON config, moving the binary, and hoping the version numbers line up, and all that good jazz. 6:41 If your first experience of MCP is trying to wire it up manually, you'll, you'll come away thinking it's overhyped, and you will be half right. Let's be fair. The second thing, the security model is not finished. 6:55 When you connect a client to a server, you're giving the LLM the ability to take real actions in your system. Okay? Just got to look at Open. Delete a database row if the server exposes that. Post in a Slack channel. 7:08 Approve a pull request. Nobody has fully worked out what the guardrails look like for that yet. Some are doing it well, some don't do it at all. And I'm not going to pretend that they have. 7:19 The third thing, and this is the one I want you to hold on to, because this might, might not be the winner. Anthropic proposed it. 7:27 Adoption has been faster than I expected, but history says the first standard is not always the one that lands. Open telemetry took years and two false starts before it stuck. We all know that. 7:40 Don't build your architecture assuming MCP is the final done deal. Build assuming it's the front runner. Those are different things. 7:48 So if it might land, and if it might quietly change the observability stack, what do you actually do about it? 7:57 If you're in the pit, meaning you're the one carrying the on full phone- the on-call phone calls, here's what I'd do this week. Pick one tool you use every day. Go and check whether it's got an MCP server yet. 8:10 Like I said, just GitHub already ships one. Datadog already ships one. Grafana, Dynatrace. Wire it up. Simple as that. Just wire it up. 8:22 Ask your assistant a question about that tool that you would normally have to click through a UI to answer. Just one. Just once. Just try it. 8:30 Feel what it's like when the assistant actually knows your data, and you can even compare the two, and you'll come away thinking either, "Oh, this is a game changer," or "It's not for us yet." 8:43 Both answers are worth having. If you're at the table, meaning you're the one signing the checks for observability stack, here's what I'd do next quarter. Ask your vendor in your renewal window one question. 8:56 What's your MCP story? If they don't have one, they don't have to have one just yet. But the direction in their answer tells you whether they've noticed the shift. A vendor may say, "We're, we're evaluating." 9:11 That means they've noticed it. If a vendor says, "MC what?" Is not going to be leading anything in the next two years. And for both of you, the thing to remember is this. 9:22 Every observability vendor built its own bridge between LLM and its data. Every one. 9:29 MCP is the first serious attempt to say, "Right, we're going to stop building bridges and start building the fucking road for, you know, for a change." It might work, it might not. 9:39 But the fight itself, the fight over the standard is where the value gets built. That's the part worth watching. And that, guys, is Tech Tuesday. I'm Alan Mann. 9:49 If you like this, the book is out, Metrics of Mayhem: A CTO's Guide to Observability That Actually Works. Kindle, paperback, hardback, it's all live. There's a free chapter at masteringobservability.com. 10:02 Just subscribe, and I look forward to seeing you on the next Tech Tuesday. Much love to you all. Take care. [outro jingle]