> ## Documentation Index
> Fetch the complete documentation index at: https://backstop.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Backstop
> An on-call SRE incident agent that fails safe, not just stays up.
Most "resilient agent" designs answer one question: *what happens when the model goes down?* Backstop answers the harder one — **what happens when the model is up, confident, and *wrong*, and the agent is about to act on it?**
A hallucinated deploy SHA. A rollback scoped to *everything*. A "restart" pointed at the production database. The most capable model still does this, and an agent that executes blindly turns a bad token into an outage. So I treated *"the model is wrong"* as a first-class failure mode, sitting right next to *"the model is down."* Every design decision exists to make a wrong output **safe**.
## What I built
A Python triage agent that diagnoses and remediates a live incident on a **real Kubernetes cluster**, through the TrueFoundry **AI Gateway · MCP Gateway · Guardrails** over **AWS Bedrock** — engineered so a bad output can never reach prod. It:
1. **Gathers** real signals — service health, recent deploy revisions, metrics, and warning events — from a Kubernetes cluster through a read-only path.
2. **Diagnoses** the root cause by asking an LLM (via the gateway) for a **structured** result, never free text.
3. **Validates** that diagnosis through a quality gate *and* validates the proposed fix through an action gate **before anything executes**.
4. **Acts** only on a validated, scoped remediation — or **escalates to a human** with the full context.
5. **Notifies** on-call and opens an incident ticket through governed (MCP) tool access.
I run it on a **local `kind` cluster with two namespaces** (`backstop-naive`, `backstop-hardened`). The same alert hits two agents: the naive one acts on a hallucinated diagnosis and takes the production database to zero; **Backstop** gets the exact same bad output, catches it with an LLM-as-judge, re-routes to a stronger model, rolls the bad deploy back — ending at `error_rate = 0.0`, with a tamper-evident receipt of every step.
## Explore
The four Pydantic contracts, the triage loop, RunEvents over SSE, and the InfraBackend interface.
PII/secret redaction, the quality gate, the LLM-as-judge, the action gate, and the cascade circuit breaker.
The full failure taxonomy and the one-click demo scenarios — each firing a different defense.
A tamper-evident, downloadable JSON audit of every run with a recomputable SHA-256 hash.