Your AI-SRE Should Show Its Working

It is three in the morning, and your on-call engineer is doing a job a machine could do faster. Reading the logs. Checking what changed in the release history. Lining up the alerts and forming a theory about what broke.

This year somebody, a vendor or one of your own engineers, has probably offered to take that work off you and called it AI-SRE. The trouble is that AI-SRE is two words stretched across four different things. At the bottom is something that reads your telemetry and writes you a tidy summary. At the top is something that can restart a database on its own while nobody is awake to stop it. If you cannot tell which one the demo is showing you, you cannot know what you are signing your operations up to.

This piece is the written companion to Tech Tuesday 5, What Is an AI-SRE? If you would rather hear it, listen to the episode or watch it on YouTube.

I am not here to talk you out of this. Think about where the first half hour of an incident goes: twenty or thirty minutes of gathering before a single moment of real judgement. Almost none of that is clever work, and it is exactly the sort of work a machine does better than we do. It does not tire, and unlike your engineer it does not get worse at four in the morning.

That work should not be done by an exhausted human in the dark. So if you keep one sentence from this piece, keep this one. AI-SRE is not a product you switch on. Where you sit on its ladder is a governance decision.

Think about how we teach people to drive. Two sets of controls. The instructor drives and explains, then the learner drives and talks through every decision, then the instructor steps in only when it matters. Eventually the learner drives alone, after a test that proves it was earned. Nobody hands over the keys on day one, and nobody keeps the instructor's foot over the brake for thirty years either.

The four rungs are the ones I set out in Somebody Turned It On. Read-only watches, writes the incident timeline and touches nothing. Advised says what broke, what it would do and who it would call, while a human does every part of it. Approved can act, but every action waits for a person to say yes: AI proposes, human approves, AI executes. Autonomous acts alone inside defined limits, and the honest version is narrow. A handful of well-understood actions, not your whole estate.

That piece was about who signs for each rung. This one is about what to look for under the bonnet.

A model on its own knows the internet. An AI-SRE has to know your estate: your logs, metrics and topology, your runbooks, the history of your past incidents. That is usually done by retrieval: feeding your own material to the model at the moment you ask, rather than hoping it picked something up along the way. In 2020 Patrick Lewis and eleven co-authors described retrieval-augmented generation and reported more specific and more factual output than a model working from its own memory alone.

In operations, that is the difference between an agent that knows how your payment service fails and one that knows how payment services fail in general. Only one of those is any use when an alert has just fired.

It also means the agent is only as good as what you feed it. Garbage in, garbage out still applies. Ground it on stale runbooks and it will tell you, with total certainty, how your estate used to work.

The serious products run several agents, not one. Specialists look at the logs, the metrics, the traces and the topology at the same time, and a supervisor takes what each of them found and turns it into a hypothesis. That parallelism is where the real speed comes from. A human investigates in sequence: one tab, then the next, then back to the first. The machine looks everywhere at once.

So here is what to look for when you are shown one. The hypothesis should arrive with its working. This deploy, this error rate, this dependency, the past incident that looked like it. If it hands you a conclusion with no evidence attached, you are taking a stranger's word for it at the worst possible moment.

Three words for the guardrails. Bounded means the agent can only act inside a defined blast radius: restart one service, not the cluster. Reversible means whatever it does can be undone: a rollback, yes; deleting, no. Approved means anything with real consequence waits for a person. Scaling out needs nobody's signature. Failing over a region, a person says yes first.

The OWASP Top 10 for LLM Applications lists excessive agency as a risk in its own right, caused by excessive functionality, excessive permissions or excessive autonomy, and one of its mitigations is the third guardrail by another name, a human approving high-impact actions before they are taken. If a vendor cannot explain clearly how all three guardrails work in their product, they have not built an AI-SRE. They have built a very confident chatbot with production-elevated credentials.

Now I have to be honest with you. Three honest things, and the third is the important one.

First, most of what is sold as autonomous is advised or approved wearing autonomous marketing. To be fair, approved is the right rung for almost every team today. The mismatch is the claim, not the product. But know which one you are buying, and get suspicious when a demo somehow never shows you the approval step.

Second, the demo fixes the easy incident. These systems handle recurring, seen-it-before failures well, and that is real value, because those are the larger share of your volume. The incident that takes your business down is usually the novel one, the combination nobody has seen, which is why nobody had an alert for it. That is exactly where a system reasoning from the past is weakest, and exactly where the demo will not go. When you trial one, bring it the incident that hurt you last year. Not the one in the demo.

Third, the rule I used to give people was simple: automate everything you can. I learned its failure mode the expensive way. We automated something and did not measure the blast radius properly. It cost us the client, and the reputational exposure was far too big. So the rule has changed. Stay away from anything with a large blast radius unless proper approval sits around it, because you can automate the action. You cannot automate the accountability.

Which is why I would say this to anyone about to sign. Do not buy the top rung for a team that has not earned the bottom one. If your telemetry is patchy and your runbooks are stale, an autonomous agent will not fix that. It just makes the mess move faster.

If you are in the pit, in an SRE team yourself, turn on one read-only agent. Point it at your estate and let it write the incident timeline for a month, without the authority to touch anything at all. A month to trial it, a quarter to earn the climb. Then compare its timeline with the one you wrote yourself, and keep score while you do it. Two numbers. How often it found the cause before you did. How often it was wrong but sounded certain. That second number is the one the ladder will ask for.

Where it agreed with you, you have evidence. Where it was wrong, you have learned which part of your estate it does not understand yet, and that is fine. It is the cheapest, safest evaluation you will ever run.

If you are at the table, take two questions into your next vendor meeting. Which rung is this on, actually? And who owns the gate when it fires? If nobody can give you a human name for the second, what you are being sold is an unknown risk, and an unknown risk always finds someone. Usually on the worst night of the quarter.

So before the next demo, start the scorecard. One read-only agent, one month, two numbers, and a human name for the gate. The instructor takes their foot off the brake when the record says so, not when the learner feels ready. Build the ladder before the agent.

An AI-SRE is software that uses AI agents to do site reliability engineering work during an incident: reading telemetry, correlating changes and alerts, proposing a cause and, on the higher rungs, acting on it. The label covers four levels of authority, from read-only summaries to autonomous remediation, so the first question is always which one you are looking at.

Read-only, which watches and explains and touches nothing. Advised, which recommends while a human acts. Approved, which acts only after a person says yes. Autonomous, which acts alone inside defined limits. Approved is the right rung for almost every team today.

Grounding means feeding the model your own logs, metrics, topology, runbooks and incident history at the moment it answers, usually through retrieval. Without it the agent reasons from general knowledge, and you get answers that are plausible rather than specific.

Only for a narrow set of actions that are bounded, reversible and well understood, where the record is already long and the undo has been tested. Anything with real blast radius should wait for a named person to approve it.

Run it read-only against your estate for a month, compare its incident timelines with your own, and track how often it was wrong while sounding certain. Then ask the vendor which rung the product is on and who owns the gate when it fires.