A Read-Only SRE Agent for My Kubernetes Homelab

After I bought the RTX PRO 4000 for the lab, the first few days were mostly play: fit models into VRAM, benchmark them, break serving configs, and figure out what a single 24 GB GPU can really do. I wrote that part up in Day 2 With the RTX PRO 4000. At some point the obvious question showed up: fine, the models run, now what should they do?

The idea that kept pulling me back was GKE Autopilot. For me, GKE Autopilot is still the best Kubernetes experience I have used, and its product idea is useful even in a homelab: the cluster should watch itself more than I watch it.

The old sysadmin version of the same instinct was Logwatch: automate repetitive checks and let the report appear on its own. I used to work as a sysadmin, and that habit never really left. A good sysadmin is lazy in the old and useful sense: remove repetitive work and make drift visible. I wanted a Kubernetes-native version of that idea: something that could notice trouble, explain it, maybe write a short daily summary, and eventually propose a fix.

That idea stuck because observability is fun to build, but hard to keep healthy by hand. Wiring exporters, labels, dashboards, alert routes, network policies, and data paths is a good engineering puzzle. The long tail is different: watching alerts age, adding missing rules, deleting noisy ones, encoding dependencies between alerts, and tuning thresholds over months is important work, but I do not enjoy doing it manually.

AI made observability click for me in a new way. Incidents keep happening, new failure modes keep showing up, and the system always needs one more small adjustment. An agent that can follow incidents, investigate them, attach MCP-backed context from the tools I already use, and feed back clean summaries is a much better fit than a generic chatbot. I still like designing the security boundary and the wiring. I just want the long-term watching loop to have help.

That became the first useful workload for the GPU. I started with a few scattered GitHub repos, then found HolmesGPT, already a CNCF Sandbox project, read through the shape of the ecosystem, and decided to build a small read-only SRE agent around the pieces that fit my cluster.

The most dangerous version of an SRE agent is the one that is almost safe.

An assistant that can read the cluster and explain an incident is useful. An assistant that can mutate production because a model misread a log line is a different class of risk. I wanted the first version, not the second.

My homelab SRE agent is no longer just a small webhook proof of concept. It is a GitOps-managed service with Alertmanager input, Matrix output, Qdrant incident memory, local runbooks, HolmesGPT investigation, Grafana MCP, Prometheus, and Kubernetes context. All of it runs against one local model on one 24 GB GPU.

The names stack up quickly, so here is the short version. HolmesGPT is the open-source SRE investigation backend. MCP is the tool and context interface. Grafana MCP exposes Grafana data through that interface. LiteLLM is the OpenAI-compatible gateway in front of the local model. Qdrant stores incident recall vectors. llama.cpp serves the GGUF model.

The boundaries are conventional interfaces: an Alertmanager webhook, an internal investigator API, MCP, an OpenAI-compatible model endpoint, Qdrant, and the existing Matrix receiver. The specific tools can change without moving the write boundary into the model.

The boundary did not change: for cluster operations, the agent is read-only.

For now, that is exactly the right shape. It can investigate alerts, summarize what it found, and keep a small memory of resolved incidents. That history layer became more useful than I expected. The knowledge layer is still the hard part.

The Kubernetes API identities used by the agent are not authorized to patch a Deployment, delete a pod, exec into a container, read Secret objects, or apply remediation.

That boundary is the whole project.

To be precise: read-only means no mutation permissions in Kubernetes or the observability systems. The agent still has two intentional application-level side effects: notifications to humans and incident summaries written to its own Qdrant collection after redaction and structural admission checks.

Current architecture

#

In product maturity, this is a PoC moving toward an MVP. Operationally, it already sits in the real alert path of the homelab, so failures and noisy conclusions have real consequences for me. It has been live for about a month and used daily. This post is a snapshot of the system as it runs today, and the architecture is what experimentation converged on. The system is operational, but its semantic memory, MCP path, and GPU namespace isolation remain the least mature parts.

The current state in Git and in the live cluster is boring in the way I want:

• the GitOps state is stable, including the whole agent path and the observability stack
• the agent, HolmesGPT, Grafana MCP, and LiteLLM are running normally
• self-monitoring alerts are deployed and excluded from the agent route by label
• the proactive sweep runs every morning and exercises the same webhook path
• the live sre-agent ServiceAccount can read normal operational resources, but cannot read Secret objects or use write verbs

The RBAC sanity check, run against the live API server with kubectl auth can-i --as=:

get pods:              yes
list deployments:      yes
get secrets:           no
patch deployments:     no
create pods/exec:      no
delete pods:           no

That is the shape I want: enough visibility to investigate, not enough authority to become an unreviewed operator.

And the agent shell’s Kubernetes read-only boundary is not just a promise that some future refactor could quietly break. The chart ships a render-gate test, documented as a required validation step: render the chart, feed the manifest to the script, and it fails if the agent’s ClusterRole contains mutation verbs such as create, update, patch, delete, or deletecollection.

It also fails on sensitive privilege-granting and identity verbs: bind, escalate, impersonate, and approve. impersonate is not a typo here; in Kubernetes RBAC it means the subject can act as another user, group, or ServiceAccount. A read-only agent must not have a back door into a stronger identity. Separately, secrets is absent from the rendered resource list. The agent shell’s rendered Kubernetes RBAC contract is mechanically checkable rather than dependent on code review alone.

The incident loop

#

The useful part is the full path from alert to analysis. The agent can investigate and remember, but remediation still goes through a human and GitOps.

flowchart TD
AM[Alertmanager] -->|raw alert| MX[Matrix room]
AM -->|authenticated webhook| SA[sre-agent]
SA -->|policy, runbooks,
recalled incidents| HG[HolmesGPT]
HG -->|bounded read-only
investigation| OUT[analysis]
OUT -->|redacted note| MX
OUT -. admitted summary .-> MEM[(Qdrant memory)]
OUT -. draft fix .-> HUMAN[human GitOps path]

In practice, I do not start with a prompt box. Most of the time I do not talk to the agent first. It talks to me when the cluster has a reason.

The trigger is Alertmanager. The surface is Matrix. That is deliberate: I wanted the agent inside the incident loop I already use, not next to it as a second operations console.

When an alert fires, I get the regular Matrix notification and a second [SRE-Agent] message with the investigation. I read that message as a triage note, not as an instruction. The useful parts are the likely cause, the evidence it found, any matching runbook context, similar prior incidents, and a draft remediation if the fix belongs in Git.

The daily sweep is the other interaction mode. It is not a dashboard and it is not a whole-cluster report. It is a scheduled nudge that asks a narrow question: is anything unhealthy enough to mention? If it says nothing interesting, I ignore it. If it points at a pod, a failed Job, or a repeating readiness problem, I start there.

The write path is still mine:

• check the agent’s analysis against Grafana, logs, or the Argo CD Application
• edit the GitOps repo myself, or have a separate coding agent prepare the diff
• review the diff, let Argo CD apply it, and watch the health gates
• only after redaction and structural admission checks does the resolved summary become incident memory

That last separation matters. A generic chat endpoint would create a second operations interface with different habits, weaker routing, and worse auditability. The useful part is not conversation. It is a bounded incident loop that lands where I already work.

The split is intentional.

flowchart LR
subgraph shell["sre-agent owns"]
direction TB
S1[webhook auth] ~~~ S2[policy and queue] ~~~ S3[redaction] ~~~ S4[delivery and memory admission]
end

subgraph engine["HolmesGPT owns"]
direction TB
B1[bounded tool loop] ~~~ B2[read-only tools] ~~~ B3[model call via LiteLLM]
end

shell -->|one bounded request| engine
engine -->|analysis| shell

sre-agent is the security shell. It owns webhook authentication, policy, queue limits, redaction, notification delivery, Qdrant incident memory, and the decision that remediation is draft-only. It also keeps its own direct read-only tools - Loki queries included - so a HolmesGPT outage degrades the analysis instead of dropping the alert.

HolmesGPT is the investigation backend. It runs multi-step read-only investigations with Kubernetes, Prometheus, and - through Grafana MCP - dashboards and Loki. It does not own the incoming alert contract, the Matrix delivery path, or the memory write policy.

The queue between them is small and legible: two async workers with a 300-second deadline each. When the queue is full, new webhooks are dropped, a counter increments, and a SreAgentAnalysisQueueFull alert fires. Under overload, dropping an investigation is an acceptable degradation because the original human notification remains intact.

That separation means I can replace or tune the investigation engine without changing the safety boundary around the alert pipeline.

Operational results after one month

#

Numbers first, because “it works” is cheap. Prometheus retention here is fifteen days, so that is the observation window; the memory collection goes back to day one.

In the retained window the agent accepted a little over two hundred alert webhooks, ran about two hundred investigations, delivered roughly 430 Matrix notifications without a recorded delivery failure, and stored just under a hundred incident memories. During alert storms the two-worker queue dropped 8 analyses - by design. The notification metric is delivery volume, not unique incident count; the normal Alertmanager route still fans out separately.

The ok/degraded split tells the project’s story better than any changelog: a little over a hundred clean conclusions against just under a hundred degraded ones across the window. The degraded pile belongs almost entirely to the reliability war described below. In the current pod lifetime, over a hundred investigations completed with zero degraded results. On that build a typical investigation takes about a minute, a quarter run past ninety seconds, and runbook fast-paths answer in about three seconds with zero tool calls.

What the investigations actually look like, from the logs of the last day:

• The alert-rule typo. AIApiHighLatencyBurn fired, latency recovered on its own, and the agent concluded the alert expression references a metric that does not exist - ai_litm instead of ai_litellm. A machine that reviews my alert rules while triaging them is exactly what I wanted
• The alert/metric mismatch. Deep runs on AIRAGTEIEmbeddingErrors pulled pod logs, found only healthy responses, and noticed the request metric the alert watches only exists for the rerank container - the embedding component never emits it. The alert points at a metric its target does not produce
• The reboot math. MikrotikDeviceRebooted: fifteen tool calls across SNMP metrics, uptime counter at roughly seventy minutes, device already up. Real reboot, already recovered, nothing to do - and the notification says so instead of just repeating the alert
• The flapper. NextcloudOPcacheRestarts fired 36 times in a day. The agent decoded the three OPcache counters behind the rule and measured them correctly every single time - which is the polite way of saying the fix belongs in the alert rule, not in 36 investigations

Two more observations from living with it. Toolset usage is lopsided: Kubernetes and Prometheus tools do nearly all the work, over a hundred instant queries a day. And the model runs close to its ceiling: about seventy context compactions a day, with the prompt’s polite request for “at most six tool calls” routinely ignored in favor of fifteen or twenty - the worst run hit 21. Only the enforced caps hold.

How the current design emerged

#

The Git history tells the story better than any architecture diagram.

The audit. Before the agent existed, I ran a strict read-only audit of the whole cluster and wrote the report into the GitOps repo. The audit culture - gather evidence read-only, make prod writes explicit GO decisions - became the design DNA of everything that followed.

The staged birth. A couple of weeks later the agent materialized in one long day, but in a very specific order: the first commit staged the entire chart at replicas: 0 - RBAC, network policy, webhook auth, and the render-time read-only test all landed and were reviewable before a single pod ran. The first live sync was a separate, deliberate deploy-GO. Ten commits later it was live; five more brought Matrix delivery and incident memory online. The observability repo met it halfway the same day with an Alertmanager route and the matching cross-namespace egress policy.

The engine transplant. A few weeks in, the narrow built-in investigator gave way to HolmesGPT - vendored chart, staged at replicas: 0 again, read-only ServiceAccount, SaaS and telemetry off - and Alertmanager started routing warnings, not just criticals, into the agent.

The reliability war. The thirty-six hours after the transplant went to a wrong default temperature, a context-window lie, an unbounded investigation loop, and incident memory quietly poisoning itself - more on each below. The fixes shipped as an image tag that says what it took: v0.1.22-bounded-investigations.

The hardening pass. Right behind it, audit-driven: HolmesGPT’s RBAC narrowed to the CRDs this cluster actually runs, the sweep payload hardened against injection, and Grafana MCP added - with a same-day upgrade to its token handling (story below).

The release tags in between read like a changelog of the whole project: v0.1.0-webhook-auth, v0.1.3-redaction-escaped, v0.1.12-redactor-sha256-context, v0.1.18-recall-context, v0.1.22-bounded-investigations. Every capability arrived staged and read-only; every failure became a bounded config fix with the root cause written into the commit message.

The local model stack

#

Everything here runs on the same RTX PRO 4000 Blackwell SFF, 24 GB and 70 W, serving Qwen3.6-35B-A3B as a Q4 GGUF on llama.cpp with a 49,152-token context and a single serving slot - and that fact shapes every number in this post.

The timeout ladder, the step cap, the compaction thresholds, the alert grouping - they all exist because investigation tokens come out of one 24 GB budget that also serves everything else in the lab.

Two configuration details took real debugging:

• the agent model alias exists separately from the plain completion aliases because it tells the gateway the model really does support tool calls (otherwise the gateway strips tools from requests) and disables Qwen’s thinking mode - a thinking model burns its whole token budget on deliberation and returns empty tool calls with finish_reason=length
• the platform enforces a hard invariant that exactly one GPU-serving Deployment is active at a time, with a Prometheus alert (AIInferenceGPUReplicaInvariantBroken) watching the invariant itself

One design rejection belongs here too. In the LiteLLM version and proxy configuration I tested, enabling the gateway’s master key also required adding authentication to the existing /metrics scrape and /v1/models probe. I had not plumbed those clients yet, so the change broke the Prometheus scrape and produced false LiteLLMDown alerts capable of masking real outages. I reverted it. Access to the internal gateway is currently restricted at the network boundary through a strict ingress allowlist, with the client-side key plumbing kept ready for a future re-enable. Relying on network-level access control is a trade-off, but in this setup it was the documented outcome of an experiment rather than an accidental omission.

Alert routing and delivery

#

The first version only cared about critical alerts. That was too narrow.

Many real incidents start as warnings: latency climbs, one replica goes unhealthy, a route starts timing out. Waiting until every signal is critical means the agent misses the part where a concise explanation is most useful.

The current route sends critical and warning alerts to sre-agent, then continues to the normal Matrix routes. The agent is in the triage path, not the notification replacement path.

Warnings could easily flood a GPU-backed investigator; boring Alertmanager mechanics double as LLM cost control. The route groups by alert name, namespace, and severity, with a 30-second group wait, 5-minute group interval, and 12-hour repeat interval. A storm of fifty crashlooping pods collapses into one investigation, not fifty. And the noise never even reaches the agent: Watchdog, InfoInhibitor, known-flaky device alerts, and maintenance windows are all null-routed above it.

The ordering rules are load-bearing and documented in the repo:

• the sre-agent route must stay before the critical Matrix route
• continue: true must stay enabled
• the agent’s own self-monitoring alerts must never route back into it - enforced by a label match, not convention

That last point avoids a feedback loop where the agent investigates its own investigation failure.

The webhook itself refuses unauthenticated requests outright - Bearer token auth, with HMAC signing wired as a second mechanism. The credential deliberately rides an existing internal credential flow rather than minting a new one: one Vault path, one rotation procedure, one documented blast radius. The fewer credential flows I create, the fewer I have to rotate, audit, and explain later.

The agent does not talk to Matrix directly.

It posts an Alertmanager-shaped payload to the existing matrix-alertmanager-receiver service, prefixed [SRE-Agent] so humans can tell the analysis from the raw alert landing in the same room. That receiver already owns the Matrix token and room mapping; the agent only needs the receiver credential, not Matrix credentials.

Two properties fall out of this:

• Matrix token scope stays in exactly one component, and the receiver’s existing metrics and failure alerts keep applying
• notification delivery is bounded - four workers, five-second timeout - so a slow Matrix hop should not block investigation intake

I prefer reusing an operated delivery component over teaching every new automation service how to notify humans from scratch.

HolmesGPT as the investigator

#

The first agent design had narrow built-in investigation plans. Safe, but not rich enough once the cluster grew more telemetry and more runbooks.

The current agent delegates investigation to HolmesGPT, an open-source SRE investigation agent (0.36.0 in my cluster), through an internal API. HolmesGPT runs in the same namespace, uses the in-cluster LiteLLM gateway, and talks to the local agent model alias. Getting it predictable enough for my alert path was not “deploy a tool and hope” - it was a sequence of very specific fights.

The temperature fight. HolmesGPT’s requests went out at its client-side default of temperature 1.0, which made multi-tool investigations emit malformed tool-call JSON that the gateway could not parse. HolmesGPT’s model config now pins temperature 0.1 with three retries for its requests; the retries regenerate the rare malformed completion instead of failing the investigation.

The context-window lie. The local GGUF model is unknown to LiteLLM’s model metadata, so HolmesGPT fell back to assuming a 200,000-token window and reserved 64,000 tokens for output. Against the real 49,152-token slot, its compaction guard - input + 64,000 > window - was always true. Every broad investigation died in a logically impossible state. The first fix (declaring max_context_size) was reverted within hours because it made compaction so aggressive that even small targeted investigations failed. The durable fix is an explicit override pair: a 47,000-token content ceiling and an 8,192-token output budget, with compaction triggering at 85% instead of the default 95%. The early trigger keeps live input near 40,000 tokens, so input plus output actually fits the 49,152 slot instead of reproducing the original bug with smaller numbers.

The unbounded loop. In the HolmesGPT version and configuration I deployed, max_steps defaulted to 100, and my caller initially had no effective wall-clock bound around the full investigation. On a single GPU slot at roughly 18 seconds per turn, broad investigations walked the tool loop until the caller timed out - at the worst point almost every broad investigation ended “degraded”; the fix commit records ~97%. A step budget of eight (MAX_STEPS=8) turned it into a bounded loop that concludes in ~144-176 seconds, inside the agent’s 240-second investigation timeout, which itself sits inside the webhook worker’s 300-second deadline. In this deployment, HolmesGPT’s final step strips tools, so the model is forced to commit to a root-cause answer instead of asking for one more query.

The quiet poisoning. The nastiest part was second-order: those degraded placeholder results were being written into Qdrant incident memory, then recalled as top-scoring “similar incidents” into new investigations - the agent was teaching itself that every incident looks like a timeout. Bounding the investigations fixed the memory, not just the latency.

Around the engine itself, the guardrails are explicit:

• product telemetry and external SaaS integrations are disabled
• the vendored chart’s optional CRD read grants - Flux, Kafka, Keda, Crossplane, Istio, Velero, External Secrets, all on by default - are switched off, keeping only Argo CD and Gateway API. A “read-only” ClusterRole full of permissions for platforms I do not run is still unnecessary surface
• the internet toolset is off (SSRF and exfiltration surface); toolsets stay read-only and in-cluster
• one oddity worth naming: the bash toolset is enabled so HolmesGPT can spill oversized tool results to disk and read them back, but it runs behind a narrow command allowlist intended for read-only inspection. Shell remains one of the weaker parts of this boundary and is a candidate for replacement with a dedicated spill-to-disk tool
• images are pinned to versioned tags rather than digests because the pull-through registry may garbage-collect historical upstream digests. A committed digest that no longer exists bricks the pull. Every supply-chain checklist says “pin by digest”; every checklist assumes your registry keeps them

Grafana MCP for dashboard context

#

The newest integration is Grafana MCP.

HolmesGPT reaches a local Grafana MCP server (mcp-grafana 0.17.2) over streamable HTTP. It is wired to give investigations read-only access to dashboard metadata, datasource discovery, Prometheus queries, Loki queries, and Grafana navigation links. It is HolmesGPT’s only path to Loki. The outer sre-agent keeps a separate direct Loki client for degraded-mode investigations.

Status today: this is also the least exercised part of the stack - the model reaches for Kubernetes and Prometheus tools first, and dashboard context rarely wins the tool-selection race. Teaching investigations to use it well is toolset work still ahead of me.

The security shape is the interesting part:

• the pod mounts a ServiceAccount token so the Vault init flow can authenticate. The ServiceAccount has no application-specific Role or ClusterRole bindings, so that identity is not used as a Kubernetes inspection path
• writes are disabled at the flag level, and so is everything else non-essential: an explicit enabled-tools allowlist, an allowed-hosts pin to its own service DNS name (DNS-rebinding guard), a separate --disable-proxied for tools proxied from external MCP servers, and Loki responses capped at 500 log lines
• the Grafana token is delivered by Vault Agent injection and is not stored in the Kubernetes Secrets API

The version pin and the proxied-tools flag landed while I was finishing this post, and they are their own lesson. Reviewing this section, I found that the deployed 0.17.1 was already one security release behind: 0.17.2 binds environment-configured credentials to the configured Grafana URL, so a caller-supplied X-Grafana-URL header can no longer redirect the service-account token to another host. And re-reading the flags against the source showed that, in mcp-grafana 0.17.2, the enabled-tools allowlist quietly does not cover proxied tools - they register through hooks and need their own disable flag; the pod logs had been showing the proxied subsystem handshaking all along. Both fixes shipped through GitOps before publication. This corner of the ecosystem moves fast enough that a one-week-old review is already stale, which is an argument for agents that re-check their own stack, not against them.

That last point deserves one short story. Grafana MCP went live with a manually created token Secret - and the same day’s audit replaced it with Vault Agent injection: an init container renders a Viewer-scoped token into an in-memory file, authenticated by a Vault role bound to exactly that ServiceAccount in exactly that namespace. The injector webhook is explicitly set to failurePolicy: Fail for matching opt-in pods and excludes itself, avoiding that specific self-deadlock path during a restart. The first rollout hung at Init because default-deny networking also applies to Vault egress; that fix landed as its own commit. Default-deny is real, and I keep re-learning it.

And the failure domain stays clean. If Grafana MCP is down, HolmesGPT still investigates from Kubernetes and Prometheus, and the shell still has its own Loki, runbook, and memory context. If HolmesGPT is down, the shell degrades to a notification that says the investigation backend was unavailable instead of dropping the alert silently.

The same principle applies to the other dependencies: a slow memory lookup times out and triage continues without recall, and a model-path failure turns into degraded analysis while the raw alert keeps its normal Alertmanager route.

I did not want the Grafana token inside HolmesGPT. HolmesGPT talks to the MCP endpoint; the MCP pod holds the credential. That keeps the credential close to the one process that actually needs it.

Runbooks and incident memory

#

The agent mounts eight runbooks from a ConfigMap:

• argocd-app-degraded.md
• cilium-egress-deny.md
• litellm-readiness-degraded.md
• owu-degraded-after-env.md
• pod-crashloop-oom.md
• qdrant-empty-or-degraded.md
• qwen-gpu-llm-down.md
• tei-embed-permit-starvation.md

These are not generic incident tips. They encode the local operating model:

• Argo CD drift is fixed through Git, not by live edits
• default-deny networking needs both source egress and destination ingress
• LiteLLM readiness usually reflects an upstream model, embed, or rerank problem
• Qdrant can be healthy while an embedding pipeline produced zero vectors
• the GPU LLM lane runs exactly one serving Deployment, nothing shares the 24 GB card, and the model server’s restart count is a hard sentinel that must stay zero
• CPU embedding is intentionally slow and bounded until a better GPU lane exists

There is machinery behind the folder of markdown, and it is what makes this a system rather than vibes:

• each runbook carries frontmatter with an alert-matching glob and a priority, so the right runbook attaches to the right alert; pod-crashloop-oom.md sits at the lowest priority as the explicit generic fallback that loses to anything more specific
• runbook context injected into a prompt is byte-bounded (8 KiB), not dumped wholesale
• the ConfigMap is stamped with source and source-commit annotations, so every mounted runbook traces to a reviewed commit
• the chart hard-fails at render time if runbook mounting is half-configured or the runbook set is empty - a misconfigured mount cannot deploy

This is the kind of context I do want near an LLM: specific, operational, and reviewable in Git.

The memory side is deliberately narrow too. The agent writes resolved incident summaries into a dedicated Qdrant collection named sre-incident-memory.

That collection is the only durable state write the agent owns, and it is not a Kubernetes mutation. Notifications leave the agent too, but they are a controlled side effect through the existing receiver, not state.

The memory path is constrained at every step:

• incident memory is a separate collection from the OpenWebUI user-memory and RAG collections, and the separation is written down and checked, not just intended: the agent’s own runbook hard-forbids writing to any open-webui* collection, and a standing verification script watches the user-side collections’ contract
• embeddings go through the existing embed alias (a CPU lane; incident memory does not compete for GPU)
• text sent to embedding is capped at 2 KiB. Incident memory is not a raw transcript sink: the write candidate is a resolved-incident summary, capped, redacted, and checked against structural admission rules before it is stored. Those checks block degraded placeholders, but they do not yet guarantee that the summary contains a useful root cause
• recall is bounded twice: at most three similar prior incidents, fetched with an eight-second timeout - a slow memory lookup cannot stall triage

The Prometheus window recorded just under a hundred memory writes during the last fifteen days; the Qdrant collection contains 192 entries across the full month. Those entries cover 34 distinct alert types, with the first one written the same morning the agent went live. Each entry is a structured document: incident id, alert fingerprint, severity, an embedded summary, the evidence list, and a redaction flag. Retrieval works mechanically: firing alerts receive their three nearest prior incidents at similarity scores between 0.70 and 0.91. The harder problem is whether those retrieved entries contain useful conclusions.

Living with it for a month split the memory into two layers of very different quality. The history layer already earns its place: the collection is an honest record of what actually fires here and how often - the top entries are a flapping blackbox probe, the OPcache alert, and SNMP device blips, which is its own kind of triage insight. The knowledge layer is the current fight: even after the placeholder-guard build stopped the worst poisoning, too many entries still carry mid-investigation narration or a serialized tool-call artifact instead of a conclusion, chronic flappers mostly store “cleared, false positive”, and only a couple of the 192 hold a root cause worth recalling. The schema has a fingerprint field for deduplication that is not enforced yet, and entries are write-once with no expiry.

It is the part I care about most. Three prior incidents is enough to answer “have I seen this before?” without turning memory into a diary - but only if the entries are worth recalling. A memory that remembers garbage is worse than no memory at all.

I like this pattern because it keeps the memory job narrow. It remembers operational facts and prior failure modes. It does not absorb every note, chat, and personal document into one shared vector soup.

The security boundary

#

The live tool list is intentionally boring:

• Kubernetes read-only inspection
• Argo CD read-only context
• Prometheus queries
• Loki queries
• GPU/DCGM signals
• Qdrant health
• LiteLLM health
• OpenWebUI health
• Git read context

The service caps tool output at 32 KiB per call and redacts secret-like content before anything reaches the model or a notification. This is not optional. A read-only agent can still leak sensitive data if it may return unlimited logs, ConfigMap data, annotations, or error bodies unfiltered. The redactor has its own release history - three separate image tags iterate on it - because redaction is code, and code has bugs.

The same stance covers prompt injection. Alerts, logs, events, and Git text are treated as untrusted data, not instructions: tool access and memory admission are enforced by application policy and platform permissions, not by prompt wording. Read-only RBAC caps what a poisoned log line can do, but the residual risks - wasted context, a misleading analysis in Matrix, a bad memory write - are exactly why tool output is bounded, notifications are redacted, and memory admission is guarded by structural checks. Semantic quality remains an open problem.

The read-only guarantee is scoped to mutation permissions and the tool paths described here. It does not mean every readable source is free of sensitive data.

The boundary ends up checked three times, in three different places:

• the render gate: the chart’s test fails the rendered manifest if a write, privilege-granting, or identity verb ever appears in the agent’s ClusterRole
• Kubernetes: the agent shell has only get, list, and watch on the operational resources it needs. HolmesGPT has a broader read-only discovery role, but still no Secrets, no exec, and no mutation verbs. The API server refuses, whatever the application thinks
• the application: policy pins read-only mode, draft-only remediation, and refuses mutation lanes and unbounded outputs

Application policy is useful, but it should never be the only thing between a model and the API server.

Network policy is the other half of that boundary. The namespace is default-deny, so the agent needed explicit paths.

The sre-agent policy allows only the traffic it needs: DNS, the Kubernetes API, LiteLLM, Prometheus, Loki, Qdrant, the Matrix alert receiver, and HolmesGPT. Ingress to its webhook port is equally short: Alertmanager, Prometheus, Grafana, and the sweep job.

Three implementation details make this section more than YAML busywork:

• The flows are declared twice. Every agent, HolmesGPT, and Grafana MCP flow is rendered as both a native NetworkPolicy and a CiliumNetworkPolicy. In this cluster both are enforced by Cilium, so these are two declarations of intent rather than two independent enforcement planes. The native policy provides a portable baseline; the Cilium policy carries cluster-specific controls
• The chart declares both sides. Where the destination namespace is also default-deny, the agent’s chart renders the matching ingress-allow policies in the target namespaces itself. Source egress without destination ingress is a silent black hole in this cluster
• This k3s/Cilium path matches the translated backend port. The HolmesGPT service fronts port 80, but the egress rule must name the pod port 5050; the k3s API server needs both 443 and 6443. This one cost a three-commit debugging arc, with Hubble finally showing drops on a port none of the agent’s own egress manifests mentioned

The same pattern holds for HolmesGPT (egress only to DNS, the API server, LiteLLM, Prometheus, Grafana MCP) and for Grafana MCP (ingress only from HolmesGPT; egress only to DNS, Grafana, and Vault).

One footnote: the shared namespace enforces the privileged Pod Security level, because GPU workloads live there too. The restricted profile runs in audit mode as a tripwire, so all the non-root, read-only-rootfs, drop-ALL hardening on these pods is voluntary and monitored, not PSA-forced. This is a remaining weakness of the current layout. The agent components do not require privileged workloads, and moving them into a restricted namespace is a reasonable next hardening step. Adjacent non-GPU namespaces got PSA baseline enforcement in the same hardening pass.

The proactive sweep

#

The agent also has a daily proactive sweep.

It is a CronJob that posts a synthetic warning-level alert (ScheduledHealthSweep) through the same authenticated webhook path as every real alert, every morning at 06:00 UTC.

The sweep pod is deliberately tiny and deliberately powerless:

• no ServiceAccount token, no service links
• non-root, read-only root filesystem, no Linux capabilities, seccomp default profile
• bounded CPU and memory
• egress limited to DNS and the sre-agent webhook - the sweep cannot talk to anything else, including the Kubernetes API

The job itself is bounded like everything else: no concurrent runs, a hard deadline, one retry, and finished pods garbage-collected by TTL.

The sweep earned its keep twice already:

• the payload used to be assembled by printf in the container’s shell - an audit moved it to Helm-rendered JSON passed via an environment variable, so no punctuation in a summary string can ever break shell quoting at runtime (or worse)
• the first sweep prompt asked for a whole-cluster investigation, and promptly blew the entire time budget. It was re-scoped to “list only unhealthy things, never enumerate healthy resources”. Asking a bounded question turns out to be as important as bounding the loop that answers it

The point is not to build a second agent. The point is to exercise the same analysis path on a schedule so I notice when the investigation pipeline itself starts degrading - before a real incident does.

Log hygiene after AI

#

The observability repo now treats the agent as part of the alerting topology, not as an experiment next to it. The alert routing and the sender’s half of the cross-namespace egress pair live there, next to the AI inference recording rules, per-alias gateway telemetry, active-probe health, and curated AI dashboards that Prometheus and Grafana already carry.

The agent’s self-monitoring, meanwhile, ships in the agent’s own chart and travels with it: SreAgentDown fires on the absence of the build-info metric; SreAgentInvestigationsDegraded fires when a window contains degraded investigations and zero successful ones; queue saturation and notification failures have their own alerts. My favorite is HolmesGPTDown: it compares desired replicas against available replicas instead of alerting on zero pods - scaling to zero through Git is a valid operator decision, and the alert only fires when Git says one thing and the cluster does another. “Git is the intent channel” encoded in PromQL.

Log hygiene turned out to belong in this story too. Once an agent can query logs, whatever lands in Loki becomes model input, so Vector now scrubs search-query parameters (?q=) from my self-hosted search service’s logs before they reach Loki. Two details I keep because they generalize:

• the scrub regex deliberately stops only at & or a quote - not at whitespace - so multi-word queries embedded un-encoded in error URLs are fully redacted; over-redacting trailing text is the accepted worst case
• the fix took three commits, because Vector interpolates $-tokens in config files - including inside comments. The $1 backreference passed the standalone VRL unit test, then crash-looped in-cluster. Passed the unit test, failed the config loader

Redacting before ingestion beats asking every future log consumer to be careful.

What I still refuse to automate

#

The tempting version is an agent that sees an alert, edits YAML, syncs Argo CD, and declares victory.

I still do not want that as the default.

The production path today is deliberately boring: the alert fans out to humans and agent independently, the agent gathers bounded read-only evidence, recalls what it has seen before, and sends a structured analysis with a draft remediation for a human to apply through GitOps.

The config already names the gates before the gates exist: writes require an approval token and a lease, and neither mechanism exists yet. Direct apply can come later only if it brings approval tokens, leases, preflight checks, restart sentinels, Argo CD health gates, and a rollback story. Until then, a high-quality draft is the feature.

Roadmap and remaining gaps

#

The read-only stage did its job: the pipeline is useful and bounded enough to justify the next iteration. The next stage is an MVP, and the direction list is concrete:

• memory that deserves recall - enforced fingerprint deduplication, conclusions instead of closure notes, and an expiry story
• event correlation - 36 investigations of one flapping alert should become one investigation and one rule fix
• Git proposal automation - turn draft remediations into reviewable branches behind explicit human approval. Git remains the only write channel; direct Kubernetes mutation stays out of scope
• local update intelligence - Renovate against the local GitLab, a review bot for generated update branches, and an AI pass that checks release notes for security fixes or behavior changes before a dependency bump reaches the cluster
• more senses through MCP - Proxmox and MikroTik are the obvious next servers, and my own APIs deserve small custom MCPs; the toolset will keep changing
• read-only queries from Matrix - not a separate operations console, but an optional query surface inside the notification channel I already use, behind the same tool and security boundaries
• graph-based orchestration, if it earns its keep - evaluate whether explicit graphs improve boundedness, state handling, and testability enough to justify replacing the current shell-plus-engine split. LangGraph and LangChain are candidates, not predetermined choices

Lessons learned

#

The strongest design choice was deciding what the agent must not do, and then making that decision expensive to reverse: a render gate on RBAC verbs, an API server that refuses, an application that never grew a write path, and network policies that describe the real dependency graph twice.

The second strongest was staging everything at replicas: 0. Every component in this story - the agent, HolmesGPT, Grafana MCP - was fully reviewable in Git, RBAC and network policy included, before the first pod started. Going live was always a one-line diff with a paper trail.

And the quiet one: most of the real engineering was writing down budgets. Steps per investigation, tokens per context, bytes per tool result, incidents per recall, seconds per hop in a timeout ladder that actually nests. An agent without budgets is not autonomous; it is just unsupervised.

The Grafana MCP bump added one more lesson: update discovery is part of operations, not housekeeping. A one-patch release changed the security boundary of a component that was already running.

The agent is more capable than the first draft, but it is not more trusted. That is the important distinction.

It has better evidence, better local context, richer observability access, and a memory of prior incidents. It still cannot surprise me with a production mutation.

That is the line I want for AI in infrastructure: useful during triage, boring under pressure, and fenced off from the write path until the approval system is good enough to deserve it.