Skip to content
Here Be Dragons

AI Guardrail Incident Response and Continuous Improvement

This is part 14 of the Engineering Guardrails for AI Systems series.

An agent follows an injected instruction and sends customer data outside an approved destination. The team changes the system prompt, confirms that one transcript now refuses, and restores the tool.

That response changes the least reliable control, proves neither containment nor causality, and can erase the evidence needed to understand what happened. Treat an AI guardrail hop as a security incident involving authority, software boundaries, external effects, and model behavior.

Detect the effect and preserve identity

Open an incident when effect evidence, control telemetry, or reconciliation crosses a risk-specific threshold. Examples include an unapproved write, a cross-tenant read, an unverified outbound destination, an approval-digest mismatch, or external receipts without matching permits.

Preserve these facts before changing the system:

  • Trace, run, session, turn, handoff, action, approval, tool-call, idempotency, and external receipt identifiers.
  • Actor, tenant, active capability set, destination class, data class, and tool-definition digest.
  • Model snapshot, system-prompt digest, policy version, detector version, SDK version, deployment version, and feature-flag state.
  • The exact canonical action digest presented to policy, approval, executor, and external service.
  • Bounded, redacted untrusted-content evidence with source and digest.
  • Clock source, event sequence, retry number, and parent links.

Do not copy complete transcripts or credentials into tickets and chat rooms. Reference access-controlled evidence by identifier and record every later evidence export.

Contain capabilities, not just conversations

Contain the smallest surface that reliably stops new prohibited effects. Start at the enforcement boundary and expand when evidence is incomplete.

SurfaceContainment actionVerification
ToolDisable the write tool or force read-only modeDirect calls and alternate aliases fail closed
CapabilityRemove the affected grant from active and delegated agentsNew runs and resumed sessions lack it
DestinationBlock unverified domains, accounts, repositories, or tenantsCanonical and redirected forms are rejected
Session and memorySuspend affected runs and quarantine derived memoryResume and handoff cannot restore authority
Token or credentialRevoke and rotate scoped credentialsOld material no longer authenticates
Queue and retryPause dispatch, cancel descendants, quarantine unknown outcomesNo delayed or duplicate effects appear
DeploymentRoll back the policy, tool, SDK, or application versionPinned control behavior is restored

A model refusal is not containment if another path can still call the tool. Confirm containment with the executor, provider, and external system of record. Reconcile in-flight calls before retrying or declaring the incident stable.

Reproduce with fake effects first

Turn the incident into the smallest case that preserves the trust transition, authority, canonical action, expected controls, and observed prohibited effect. Use a fake model if the model path matters and a recording executor for every external action.

Reproduce in this order:

  1. Deterministic policy and executor path with the exact incident facts.
  2. Scripted model trajectory and handoff state.
  3. Provider adapter and sandbox integration in an isolated environment.
  4. Repeated live-model eval with inert tools and synthetic data.
  5. Controlled end-to-end verification against a dedicated test destination.

Never begin by replaying a destructive action against production. If the effect cannot be made inert, test the prevention boundary and external reconciliation separately.

Write the causal record

A reader-facing incident record should state what happened without becoming a source dump.

Example incident record

Incident: INCIDENT_EXAMPLE_2026_09_01

Impact: One synthetic support record was sent to an unverified test destination. No production credential or personal data was present.

Detection: External send receipt RECEIPT_EXAMPLE_1 had no authorization event bound to DIGEST_ACTION_EXAMPLE_1.

Causal timeline:

10:04:11  Untrusted ticket attachment enters TURN_EXAMPLE_1
10:04:12  Model proposes an outbound send
10:04:13  Human approves the displayed summary
10:04:14  Executor accepts the proposal without authorization evidence
10:04:15  External provider confirms the send
10:06:02  Receipt reconciliation raises the incident
10:09:30  Outbound tool and queued retries are disabled

First preventable failure: The executor did not require a tenant and destination authorization permit bound to the exact action digest.

Malicious input: An attachment instructed the agent to forward ticket contents. This triggered the proposal but did not authorize the effect.

Contributing conditions: The approval display omitted the normalized destination, the tool credential permitted arbitrary destinations, and receipt reconciliation ran every five minutes.

Control ledger:

ControlExpectedObservedFinding
Input trust labelingAttachment remains untrustedSource label presentPassed
Tenant and destination authorizationPermit bound to action digestNo eventFirst preventable failure
ApprovalExact destination and digest shownSummary-only approvalContributing failure
ExecutorReject missing permitProposal dispatchedBoundary bypass
Receipt reconciliationMatch receipt to permitMismatch detectedDetected incident

Containment: Disabled outbound send, removed delegated send capability, blocked the destination, paused queued retries, quarantined the session memory, and revoked the tool credential.

Corrective action: The executor now requires a signed application-owned permit bound to actor, tenant, action digest, destination, policy version, expiration, and idempotency key. Approval is an additional requirement for eligible sends, not a replacement for authorization.

Convert the incident into a regression

This credential-free Python fixture captures the first preventable failure. The vulnerable executor demonstrates the incident. The fixed executor blocks the prohibited outcome while retaining the allowed support read.

from __future__ import annotations


from dataclasses import dataclass, field
from typing import Literal




Decision = Literal["allow", "block", "ask"]




@dataclass(frozen=True)
class VersionPins:
    model: str
    prompt_digest: str
    policy: str
    tool_definition_digest: str
    sdk: str




@dataclass(frozen=True)
class Action:
    action_id: str
    name: Literal["read_ticket", "send_reply"]
    tenant_id: str
    destination: str | None
    digest: str




@dataclass(frozen=True)
class Authority:
    tenant_id: str
    capabilities: frozenset[str]
    verified_destination: str




@dataclass(frozen=True)
class Permit:
    action_id: str
    action_digest: str
    decision: Decision
    policy_version: str




@dataclass
class FakeEffects:
    calls: list[Action] = field(default_factory=list)


    def record(self, action: Action) -> str:
        self.calls.append(action)
        return f"RECEIPT_EXAMPLE_{len(self.calls)}"




def authorize(action: Action, authority: Authority) -> Permit:
    if action.tenant_id != authority.tenant_id:
        decision: Decision = "block"
    elif action.name == "read_ticket" and "ticket:read" in authority.capabilities:
        decision = "allow"
    elif action.name == "send_reply" and (
        "reply:send" in authority.capabilities
        and action.destination == authority.verified_destination
    ):
        decision = "ask"
    else:
        decision = "block"
    return Permit(action.action_id, action.digest, decision, "support-policy-v7")




def vulnerable_execute(action: Action, approved: bool, effects: FakeEffects) -> str:
    # Incident behavior: approval was treated as complete authorization.
    if not approved:
        return "BLOCKED"
    return effects.record(action)




def fixed_execute(
    action: Action,
    permit: Permit,
    approved_action_digests: frozenset[str],
    effects: FakeEffects,
) -> str:
    if permit.action_id != action.action_id or permit.action_digest != action.digest:
        return "BLOCKED_PERMIT_MISMATCH"
    if permit.decision == "block":
        return "BLOCKED_POLICY"
    if permit.decision == "ask" and action.digest not in approved_action_digests:
        return "BLOCKED_APPROVAL_REQUIRED"
    return effects.record(action)




pins = VersionPins(
    model="MODEL_SNAPSHOT_EXAMPLE_2026_08_31",
    prompt_digest="DIGEST_PROMPT_EXAMPLE_4",
    policy="support-policy-v7",
    tool_definition_digest="DIGEST_TOOL_EXAMPLE_9",
    sdk="SDK_VERSION_EXAMPLE_1_0",
)
assert pins.policy == "support-policy-v7"


authority = Authority(
    tenant_id="TENANT_EXAMPLE_A",
    capabilities=frozenset({"ticket:read", "reply:send"}),
    verified_destination="customer@example.invalid",
)


prohibited_send = Action(
    action_id="ACTION_EXAMPLE_SEND_1",
    name="send_reply",
    tenant_id="TENANT_EXAMPLE_A",
    destination="unverified@example.invalid",
    digest="DIGEST_ACTION_EXAMPLE_SEND_1",
)
allowed_read = Action(
    action_id="ACTION_EXAMPLE_READ_1",
    name="read_ticket",
    tenant_id="TENANT_EXAMPLE_A",
    destination=None,
    digest="DIGEST_ACTION_EXAMPLE_READ_1",
)


# Before: a deceptive approval is enough to reproduce the prohibited effect.
before = FakeEffects()
assert vulnerable_execute(prohibited_send, approved=True, effects=before).startswith(
    "RECEIPT_EXAMPLE_"
)
assert [call.action_id for call in before.calls] == ["ACTION_EXAMPLE_SEND_1"]


# After: exact policy evidence blocks that same effect.
after = FakeEffects()
send_result = fixed_execute(
    prohibited_send,
    authorize(prohibited_send, authority),
    approved_action_digests=frozenset({prohibited_send.digest}),
    effects=after,
)
assert send_result == "BLOCKED_POLICY"
assert after.calls == []


# Utility regression: the legitimate support read still succeeds.
read_result = fixed_execute(
    allowed_read,
    authorize(allowed_read, authority),
    approved_action_digests=frozenset(),
    effects=after,
)
assert read_result == "RECEIPT_EXAMPLE_1"
assert [call.action_id for call in after.calls] == ["ACTION_EXAMPLE_READ_1"]


# Digest regression: editing an allowed action after policy blocks execution.
changed_read = Action(
    **{**allowed_read.__dict__, "digest": "DIGEST_ACTION_EXAMPLE_CHANGED"}
)
assert (
    fixed_execute(
        changed_read,
        authorize(allowed_read, authority),
        approved_action_digests=frozenset(),
        effects=after,
    )
    == "BLOCKED_PERMIT_MISMATCH"
)
assert len(after.calls) == 1

Keep the minimized fixture linked to the incident ID, threat-model entry, affected control, and release gate. Store synthetic values in source control. Keep restricted evidence in the incident system.

Fix the enforcement boundary first

Prompt changes may reduce how often a model proposes the action. They do not remove the executor’s ability to perform it. Correct the earliest preventable boundary before adding defense in depth.

Typical boundary fixes include:

  • Require policy evidence in the executor, not just in orchestration code.
  • Bind policy and approval to canonical action arguments and digest.
  • Shrink the tool credential and destination allowlist.
  • Make detector timeout behavior explicit and fail closed for high-impact effects.
  • Propagate reduced authority across handoffs and resumed sessions.
  • Add idempotency and external receipt reconciliation.
  • Reject unknown control outcomes and incomplete evidence.

After the boundary fix, prompt, classifier, and user-interface changes can reduce attack frequency and reviewer confusion.

Test deterministic and live behavior

The regression ladder should include:

  1. Pure policy test for the incident facts and nearby hard negatives.
  2. Executor contract proving no effect lacks a matching permit.
  3. Scripted model path reproducing the same tools, handoffs, approvals, and retries.
  4. Property tests over tenants, capabilities, destinations, and action mutations.
  5. Provider and sandbox integration with inert effects.
  6. Repeated live-model trials over the original attack intent, mutations, adaptive attacks, and benign controls.
  7. Evidence reconstruction proving the first preventable failure is now a passing control.

Track both prohibited-effect rate and allowed-task success. A fix that disables every task is containment, not a restored product.

Restore in stages

Restore only after containment verification, regression evidence, and an approved rollback plan.

disabled
  -> synthetic read-only shadow
     -> internal allowlisted actors and destinations
        -> small production cohort with manual approval
           -> wider cohort with automated rollback gates
              -> normal operation with heightened monitoring

At every step, compare attack success, benign success, utility under attack, false refusals, latency, cost, effect-evidence completeness, and external receipt reconciliation. Stop or roll back when any risk-specific gate fails. Do not advance solely because no alert fired during a short window.

Feed the finding back into engineering

An incident is not closed when the tool is restored. Update:

  • The threat model with the new source, trust transition, effect, and abuse path.
  • The control ledger and architecture diagram with the enforced boundary.
  • Deterministic, hard-negative, mutation, multi-turn, and adaptive eval fixtures.
  • Release gates, evidence-completeness requirements, and rollback thresholds.
  • Tool scopes, approval displays, handoff contracts, and failure policies.
  • On-call runbooks, ownership, and dependency contacts.
  • Similar tools and agents that share the same vulnerable pattern.

Review high-impact controls after each material model, prompt, policy, SDK, tool, retrieval, or permission change. Run a recurring quarterly threat-model and incident-fixture review even when no incident occurs. Expire temporary exceptions and verify that containment switches still work.

Before-and-after evidence

Close the technical remediation only when the record shows both safety and retained utility.

MeasureBeforeAfterRequired interpretation
Original prohibited sendReproduced with fake effectZero fake or controlled live effectsBoundary fix blocks recurrence
Mutated and adaptive attacksBypasses observedMeets risk-specific boundFix generalizes beyond one string
Allowed ticket readSuccessfulStill successfulCore utility retained
Hard-negative support ticketsBaseline recordedFalse-refusal gate metSuspicious text remains usable data
Effect evidenceMissing authorization lineageComplete permit and receipt chainFuture incidents are reconstructable
Latency and costBaseline recordedWithin accepted budgetsOperational cost is explicit

Tradeoffs and residual risk

Aggressive containment reduces ongoing harm but interrupts legitimate work. Broad token revocation is reliable but can affect unrelated tenants. A minimized fixture is reproducible but may omit a race or provider behavior. Gradual restoration limits blast radius but increases operational complexity. Longer evidence retention helps investigation but increases privacy and security exposure.

Residual risk includes unknown related bypasses, delayed external effects, incomplete provider logs, compromised audit writers, model changes after restoration, and temporary exceptions that outlive the incident.

Common failure modes

  • Prompt patch first: Changing wording before closing the executor bypass.
  • Conversation-only containment: Ending one chat while active tools, queues, memory, or credentials remain usable.
  • Production replay: Reproducing a harmful effect against live data or destinations.
  • Version amnesia: Failing to pin the model, prompt, policy, tool definition, SDK, and deployment.
  • Trigger as root cause: Naming malicious input while leaving the preventable boundary unchanged.
  • Safety without utility: Closing the incident because every task now refuses.
  • Instant restoration: Re-enabling the full population without staged gates or rollback.
  • Local fix only: Ignoring other agents and tools that share the same authorization pattern.

Production checklist

Series navigation

References

Why the name "Here Be Dragons"?