July 23, 2026 8 min read AI · SRE · Disaster Recovery

Your AI System Needs a Disaster Recovery Plan

Most teams have a disaster recovery plan for their database — backups, replication, failover. The AI layer gets a shrug: “we’ll deal with it if it happens.” Here’s the threat model, the three-layer DR framework, and the one-page runbook format I’ve used after three client outages.

You’ve built an AI system. It works great. It’s integrated into your business process. Everyone relies on it.

Then Anthropic has an outage. Or your Lambda functions start failing. Or a deployment goes wrong and your API starts returning garbage.

Do you have a plan?

Most teams don’t. They have a business continuity plan for their database: backups, replication, failover strategies. But the AI layer? “I guess we’ll deal with it if it happens.”

That’s a mistake. AI systems fail in ways that are hard to predict, and when they fail, you need to know immediately and have a playbook.

I’ve been through three AI system outages at clients. Each time, having a documented DR plan meant the difference between a 30-minute incident and a 6-hour crisis.

The Threat Model

What can go wrong with an AI system?

1. Upstream provider outage (Anthropic API goes down)

Duration: typically 30 minutes to a few hours. Impact: all Claude API calls fail. No fallback path unless you’ve architected one.

2. Data pipeline failure (your input processing breaks)

This is common. A Lambda function is misconfigured, or your data validation is too strict, and documents stop flowing to the processor. Claude never sees the data. Your system produces no output.

3. Model degradation or hallucination spike

Claude returns nonsensical or harmful output. Rare, but it happens. Your system propagates the bad data downstream.

4. Regional outage

Your Lambda functions are in us-east-1, and us-east-1 experiences a major failure. AWS guarantees an SLA, but that doesn’t mean zero downtime, and recovery takes time.

5. Cost runaway

A bug causes your system to make 100x more API calls than expected. Your monthly Claude bill goes from $500 to $50,000. This isn’t a service failure, but it’s still a disaster.

6. Integration failure

Your system successfully extracts data from a document but fails to post it downstream, to your database, your CRM, wherever it needs to go. The data is extracted, then lost.

The DR Framework

Think of disaster recovery in three layers: observability to detect the problem, fallback paths to keep operating in degraded mode, and recovery procedures to restore normal operation.

Layer 1: Observability

You can’t respond to a disaster you don’t know about.

Set up monitoring for:

API error rate

If the Claude API error rate exceeds 5%, alert immediately. This catches upstream outages.

import cloudwatch

def monitor_api_health():
    cw = boto3.client("cloudwatch")

    cw.put_metric_alarm(
        AlarmName="claude-api-error-rate-high",
        MetricName="APIErrorCount",
        Namespace="AIProcessing",
        Statistic="Sum",
        Period=60,
        EvaluationPeriods=2,
        Threshold=5,  # 5 errors in 2 minutes
        ComparisonOperator="GreaterThanThreshold",
        AlarmActions=["arn:aws:sns:us-east-1:123456789:alert-on-call"]
    )

Processing latency

If document processing is taking 10x longer than normal, something is wrong. Could be a slow API, could be network issues, could be Claude performance degradation.

Output quality

Validation failure rate should be stable, usually under 5%. If it suddenly spikes to 20%, Claude might be hallucinating more.

cw.put_metric_alarm(
    AlarmName="validation-failure-rate-high",
    MetricName="ValidationFailureRate",
    Namespace="AIProcessing",
    Statistic="Average",
    Period=300,
    Threshold=15,  # 15% failure rate
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:alert-on-call"]
)

Dead letter queue depth

If failed items are accumulating in your DLQ, something is broken.

cw.put_metric_alarm(
    AlarmName="dlq-depth-high",
    MetricName="ApproximateNumberOfMessagesVisible",
    Namespace="AWS/SQS",
    Dimensions=[{"Name": "QueueName", "Value": "document-processing-dlq"}],
    Statistic="Average",
    Period=60,
    Threshold=10,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:alert-on-call"]
)

These alarms should page your on-call engineer, not send an email. If you’re not awake, you can’t respond.

Layer 2: Fallback Paths

When things break, you need ways to keep operating.

Fallback 1: Buffering with SQS

Don’t call Claude synchronously. Put requests in a queue and process asynchronously.

If Claude is slow or erroring, requests back up in the queue. Once Claude recovers, you process the backlog. No requests are lost.

sqs = boto3.client("sqs")

def queue_document_for_processing(document_id: str, content: str):
    sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps({
            "document_id": document_id,
            "content": content,
            "timestamp": time.time()
        })
    )
    return {"queued": True}

# Lambda processes from queue
def process_from_queue(event, context):
    for record in event["Records"]:
        message = json.loads(record["body"])

        try:
            result = call_claude_api(message["content"])
            store_result(message["document_id"], result)
        except Exception as e:
            # Send to DLQ, let it retry later
            send_to_dlq(record)

Fallback 2: Synthetic responses for non-critical systems

Some AI uses are non-critical. If you’re generating summary emails and Claude is down, send a templated message instead: “Summary temporarily unavailable. We’ll email this later.”

Not ideal, but better than failing silently or timing out.

def generate_summary(document_text: str, fallback: bool = False):
    if fallback:
        return {
            "summary": "Summary temporarily unavailable. Please try again in 5 minutes.",
            "fallback": True
        }

    try:
        return call_claude_api(document_text)
    except APIError:
        # Use fallback
        return generate_summary(document_text, fallback=True)

Fallback 3: Multi-region deployment

If your Lambda functions are only in us-east-1, a regional outage takes you down.

Deploy to us-west-2 as well, and route traffic based on health checks.

resource "aws_lambda_function" "document_processor_east" {
  # ... configuration for us-east-1
}

resource "aws_lambda_function" "document_processor_west" {
  # ... configuration for us-west-2
}

resource "aws_route53_health_check" "east" {
  # ... health check for east region Lambda
}

# Route53 weighted routing policy
resource "aws_route53_record" "processor_api" {
  zone_id = aws_route53_zone.company.zone_id
  name    = "processor.company.com"
  type    = "A"

  weighted_routing_policy {
    weight = 100  # Primary
  }

  set_identifier = "east"
  alias {
    name                   = aws_api_gateway_stage.east.invoke_url
    zone_id                = aws_api_gateway_stage.east.zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "processor_api_west" {
  zone_id = aws_route53_zone.company.zone_id
  name    = "processor.company.com"
  type    = "A"

  weighted_routing_policy {
    weight = 0  # Secondary (activated if east fails)
  }

  set_identifier = "west"
  alias {
    name                   = aws_api_gateway_stage.west.invoke_url
    zone_id                = aws_api_gateway_stage.west.zone_id
    evaluate_target_health = true
  }
}

Fallback 4: Human review escalation

If the system is broken and you still need output, escalate to a human manually.

def process_document_with_escalation(document_id: str, content: str):
    try:
        return call_claude_api(content)
    except APIError as e:
        # Log the failure
        log_error(document_id, str(e))

        # Create a ticket for manual review
        jira = boto3.client("jira")
        jira.create_issue(
            project="OPS",
            summary=f"Manual review needed: {document_id}",
            description=f"AI processing failed for document {document_id}. Please review manually.",
            assignee="legal-ops-team@company.com"
        )

        return {"status": "escalated", "ticket_id": ticket_id}

Layer 3: Recovery Procedures

When a disaster happens, you need a runbook.

One-page runbook format:

INCIDENT: Claude API outage

DETECTION:
- CloudWatch alarm: claude-api-error-rate-high triggered
- Symptom: All document processing requests failing with 503

IMMEDIATE ACTIONS (first 5 minutes):
1. Page on-call engineer
2. Check Anthropic status page: https://status.anthropic.com
3. Verify AWS Lambda/API Gateway is healthy (CloudWatch dashboards)
4. Do NOT escalate to customers yet; wait for confirmation

MITIGATION (5-30 minutes):
1. Pause incoming document uploads (notify business owners via Slack)
2. Redirect queue processing to fallback region (if configured)
3. Monitor DLQ depth; it should stabilize
4. Post status update to #incidents Slack channel

RECOVERY (30 minutes+):
1. Monitor Anthropic status page for recovery
2. When API returns, process DLQ backlog (oldest documents first)
3. Validate output quality (sample 10 documents, check accuracy)
4. Resume normal operation once confidence is high
5. Post incident summary to #incidents

ESCALATION:
- 15 minutes: Notify VP Engineering
- 30 minutes: Notify CTO
- 1 hour: Notify CEO

COMMUNICATION:
- Update #incidents Slack every 15 minutes during outage
- Email stakeholders at 30 minutes if still ongoing
- Post mortem within 48 hours

Disaster Recovery Testing

You must test this. Untested DR plans don’t work.

Every quarter, run a disaster recovery drill:

Simulate Claude API being down (block API calls in code, return an error)
Verify your monitoring detects it within 2 minutes
Verify fallback paths work (DLQ, synthetic responses, etc.)
Measure time to recovery
Document what broke
Fix it

def test_dr_plan():
    # Simulate API failure
    original_call = anthropic.messages.create
    anthropic.messages.create = lambda **kwargs: raise APIError("Simulated outage")

    # Call your processing function
    result = process_document("test_doc", "test content")

    # Verify it failed gracefully
    assert result["status"] == "escalated" or result["fallback"] == True

    # Restore original
    anthropic.messages.create = original_call

Run this test monthly. If it fails, you have a gap in your DR plan.

RTO and RPO

Define your targets.

RTO (Recovery Time Objective): how long can you tolerate being down? For most systems, I recommend an RTO of 15 minutes for customer-facing critical paths and 1 hour for everything non-critical.

RPO (Recovery Point Objective): how much data can you afford to lose? With SQS buffering, you shouldn’t lose any documents: RPO = 0.

But if a document is in progress when things fail, can you retrace it? Document that too.

The Bottom Line

AI systems fail. When they do, you need monitoring that alerts you in minutes and not hours, fallback paths that let you keep operating in degraded mode, documented procedures so anyone can respond, and regular testing so your plan actually works.

Build this before you need it. Disaster recovery is cheap when it’s in code and expensive when it’s a crisis.

Get the free AI Readiness Checklist

15 questions to diagnose your team’s AI readiness, where you’ll see ROI fastest, and what to tackle first.

✓ Takes 5 minutes ✓ Actionable next steps ✓ No sales pitch

No spam. Unsubscribe anytime.

Ready to build AI that actually works?

Let’s talk about how SRE discipline transforms AI from a risky experiment into a reliable business system.

Book Your Free Discovery Call