The Autonomous SRE: Using AgentOps to Hunt and Patch Live Cluster Failures

Static alerting tells you something is wrong. It has never once told me what to do about it. The last piece of the automation stack — I've been calling it AgentOps — exists to close that gap: a small system of AI agents that sit between "an alert fired" and "a human is now debugging a cluster at 3am."

Where the idea actually came from

Long before there was anything worth calling an agent, there was ordinary remediation automation — the kind everyone eventually writes. A watcher polls a health check, the check fails, a script runs: service is down, restart the service. It works, and it's also the ceiling of what that style of automation can ever do. It has exactly one move per failure, decided in advance, with no capacity to notice that this restart is the fourth one this week or that the real problem is a disk filling up two layers away. It responds to symptoms because that's all it was ever taught to recognize.

AgentOps is what happens when you stop trying to write a bigger if/else tree for every failure mode and instead ask a different question: what if the thing responding to the alert could actually investigate, the way a person would? The goal isn't smarter scripts. It's putting an actual SRE role on the roster — running as an AI agent across the lab, on call for the same class of problems a human on-call engineer would own, minus the human.

A catalog, not one big agent

Rather than one general-purpose watcher, there's a small catalog of narrowly scoped agents — on-call triage, deploy verification, dependency triage, cost monitoring, CI health — each with its own versioned runbook prompt. A dispatcher routes an incoming signal to the agent whose job it actually is. Scoping each agent to one responsibility does the same thing scoping a human role does: it makes "what is this agent allowed to reason about right now" an answerable question instead of an open-ended one.

Read-only by default

Every diagnostic agent runs in a plan-only mode: it can inspect state, read logs, query metrics, and reason about root cause, but it cannot execute anything against the cluster directly. Its output is a proposed remediation, not an action. Only a narrowly pre-approved playbook — the same kind of Ansible play a human would run by hand — actually touches production, and only for the specific, bounded class of fixes the agent is trusted to trigger unsupervised. Anything outside that scope stops at "here's what I found and what I'd do," and waits for a human.

There's also a hard spend ceiling: agent runs are metered against a monthly budget, and the system refuses to launch further runs once it's hit. An autonomous agent with an unbounded API bill is its own kind of incident.

Tracing the reasoning, not just the outcome

The useful property isn't that an agent fixed something — it's that the entire reasoning trail leading to the fix is logged and replayable: what it looked at, what it ruled out, what evidence it based the final call on. When the fix is right, that trail is a diagnosis write-up I didn't have to author. When it's wrong, it's the fastest possible path to finding out why an agent thought something it shouldn't have.

A case study: quorum loss, agent-diagnosed

A node dropped out of cluster quorum overnight. The watcher caught the membership flap and handed it to the on-call triage agent. Working entirely in read-only mode, the agent walked the stack from the link layer up, correlated interface error counters against the timing of the drops, and landed on a flaking network link — not a dead node — as the root cause. It proposed the same fix a human would reach for: re-run the node's configuration play to reassert known-good interface settings. That fix was inside the agent's pre-approved scope, so it ran automatically. The cluster was healthy again well before I looked at a screen, with a full diagnosis already written down.

The point was never that the agent is smarter than I am at reading a corosync log. It's that the reasoning trail is captured either way, and the fix is a version-controlled playbook run either way — which means the outcome is the same whether a human or an agent got there first.

Next post: taking this same instinct — model the system, then let something reason over the model — out of the server room and into the physical house.