Skip to main content
Backstop — fails safe, not just stays up Most “resilient agent” designs answer one question: what happens when the model goes down? Backstop answers the harder one — what happens when the model is up, confident, and wrong, and the agent is about to act on it? A hallucinated deploy SHA. A rollback scoped to everything. A “restart” pointed at the production database. The most capable model still does this, and an agent that executes blindly turns a bad token into an outage. So I treated “the model is wrong” as a first-class failure mode, sitting right next to “the model is down.” Every design decision exists to make a wrong output safe.

​
What I built

A Python triage agent that diagnoses and remediates a live incident on a real Kubernetes cluster, through the TrueFoundry AI Gateway · MCP Gateway · Guardrails over AWS Bedrock — engineered so a bad output can never reach prod. It:
  1. Gathers real signals — service health, recent deploy revisions, metrics, and warning events — from a Kubernetes cluster through a read-only path.
  2. Diagnoses the root cause by asking an LLM (via the gateway) for a structured result, never free text.
  3. Validates that diagnosis through a quality gate and validates the proposed fix through an action gate before anything executes.
  4. Acts only on a validated, scoped remediation — or escalates to a human with the full context.
  5. Notifies on-call and opens an incident ticket through governed (MCP) tool access.
I run it on a local kind cluster with two namespaces (backstop-naive, backstop-hardened). The same alert hits two agents: the naive one acts on a hallucinated diagnosis and takes the production database to zero; Backstop gets the exact same bad output, catches it with an LLM-as-judge, re-routes to a stronger model, rolls the bad deploy back — ending at error_rate = 0.0, with a tamper-evident receipt of every step.
Backstop incident console

​
Explore

Architecture

The four Pydantic contracts, the triage loop, RunEvents over SSE, and the InfraBackend interface.

Guardrails

PII/secret redaction, the quality gate, the LLM-as-judge, the action gate, and the cascade circuit breaker.

Resilience

The full failure taxonomy and the one-click demo scenarios — each firing a different defense.

The Incident Receipt

A tamper-evident, downloadable JSON audit of every run with a recomputable SHA-256 hash.