October 8, 2026 8 min read SRE · AWS · Reliability

Building Retry Logic That Actually Works

A 3-minute Claude API outage turned into a 2-hour cascading failure, because of how my retry logic responded to it.

I built a system that retried failed requests with a simple exponential backoff strategy. On a Tuesday, the Claude API had a brief outage: three minutes. By the time it recovered, my Lambda was retrying requests every 30 seconds, generating 10x the normal traffic. The brief outage became a cascading failure that lasted two hours.

Retry logic is one of those things that feels simple and is actually dangerous if you get it wrong.

The retry strategies

Strategy 1: immediate retry (bad)

def call_claude_with_retry(prompt: str):
    for attempt in range(3):
        try:
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}]
            )
            return response
        except Exception as e:
            if attempt == 2:
                raise
            time.sleep(1)  # Wait 1 second and try again

This is bad because if the API is overwhelmed, you're hammering it immediately. Everyone retrying at once creates a thundering herd.

Strategy 2: linear backoff (better)

def call_claude_with_retry(prompt: str):
    for attempt in range(3):
        try:
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}]
            )
            return response
        except Exception as e:
            if attempt == 2:
                raise
            wait_time = (attempt + 1) * 2  # 2, 4, 6 seconds
            time.sleep(wait_time)

This spreads retries out over time: 2, 4, 6 seconds. Better than immediate retry, but still predictable. If 1,000 requests fail at the same time, they'll all retry at 2 seconds, then all retry again at 4 seconds.

Strategy 3: exponential backoff with jitter (good)

import random
import time

def call_claude_with_exponential_backoff(prompt: str, max_retries: int = 3):
    """Call Claude API with exponential backoff and jitter."""

    for attempt in range(max_retries):
        try:
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}]
            )
            return response
        except anthropic.APIError as e:
            if attempt == max_retries - 1:
                raise

            # Exponential backoff: 2^attempt
            base_wait = 2 ** attempt

            # Add jitter: +/-25% randomness
            jitter = base_wait * 0.25 * random.random()
            wait_time = base_wait + jitter

            print(f"Attempt {attempt + 1} failed. Retrying in {wait_time:.1f}s")
            time.sleep(wait_time)

With exponential backoff and jitter:

Attempt 1 fails at 0s: wait 2^0 = 1s + jitter (0-0.25s) = 1-1.25s
Attempt 2 fails at about 1.1s: wait 2^1 = 2s + jitter (0-0.5s) = 2-2.5s
Attempt 3 fails at about 3.2s: wait 2^2 = 4s + jitter (0-1s) = 4-5s

Total time: about 8-9 seconds across 3 attempts. Retries are spread out and not synchronized.

Distinguishing retryable vs. terminal errors

Not all errors should trigger a retry. If the API returns 400 (bad request), retrying won't help — you need to fix the input.

def is_retryable(exception: Exception) -> bool:
    """Determine if an exception should trigger a retry."""

    # Retryable: service is temporarily unavailable
    if isinstance(exception, anthropic.APIStatusError):
        if exception.status_code in [408, 429, 500, 502, 503, 504]:
            return True
        # Not retryable: bad request, auth error, etc
        return False

    # Retryable: network timeout or connection error
    if isinstance(exception, (TimeoutError, ConnectionError)):
        return True

    # Not retryable: everything else
    return False

def call_claude_with_smart_retry(prompt: str, max_retries: int = 3):
    """Call Claude with retry logic that distinguishes error types."""

    for attempt in range(max_retries):
        try:
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}]
            )
            return response
        except Exception as e:
            if not is_retryable(e):
                # Don't retry, raise immediately
                raise

            if attempt == max_retries - 1:
                # Out of retries
                raise

            # Exponential backoff
            wait_time = (2 ** attempt) + random.uniform(0, 2 ** attempt * 0.25)
            time.sleep(wait_time)

The key insight: 400 Bad Request will fail every time. Don't waste retries on it.

Idempotency: the critical requirement

Retry logic only works if your operation is idempotent: calling it multiple times produces the same result as calling it once.

Example: if your function reads a document, classifies it, and writes the result to a database, what happens if it retries?

Without idempotency:

Call 1: classify document, write to database, succeed. But the response is lost to a network timeout.
Retry: classify again, write again. Now you have two classification results in the database.

With idempotency:

Introduce an idempotency key, for example a hash of the document.
Before writing, check if this key already exists.
If it does, return the cached result instead of classifying again.

def classify_document_idempotent(document: str) -> str:
    """Classify document with idempotency."""

    # Create a stable idempotency key
    idempotency_key = hashlib.sha256(document.encode()).hexdigest()

    # Check if we've already classified this document
    cached = get_from_cache(idempotency_key)
    if cached:
        return cached

    # Call Claude
    classification = call_claude(f"Classify: {document}")

    # Cache the result
    store_in_cache(idempotency_key, classification)

    return classification

Now, even if the request retries 5 times, you get the same classification without calling Claude 5 times.

Lambda retry configuration

AWS Lambda has built-in retry behavior for asynchronous invocations, for example from SQS.

resource "aws_lambda_function" "process_document" {
  filename      = "lambda.zip"
  function_name = "process-document"
  role          = aws_iam_role.lambda_role.arn
  handler       = "index.handler"

  # Timeout must be less than SQS message visibility timeout
  timeout = 60
}

resource "aws_sqs_queue" "documents" {
  name                      = "documents"
  visibility_timeout_seconds = 300  # 5 minutes
  message_retention_seconds  = 86400  # 24 hours
}

resource "aws_sqs_queue" "dlq" {
  name = "documents-dlq"
}

resource "aws_lambda_event_source_mapping" "sqs_trigger" {
  event_source_arn  = aws_sqs_queue.documents.arn
  function_name     = aws_lambda_function.process_document.function_name
  batch_size        = 10

  # Retry configuration
  maximum_event_age_in_seconds = 3600  # Give up after 1 hour
  maximum_retry_attempts       = 2     # Retry up to 2 times
  bisect_batch_on_error        = true  # If a batch fails, split it in half and retry

  function_response_types = ["ReportBatchItemFailures"]

  dead_letter_config {
    target_arn = aws_sqs_queue.dlq.arn
  }
}

This configuration:

Retries failed messages up to 2 times.
If a batch of 10 messages has 1 failure, splits the batch and retries the failing message.
After 1 hour total, gives up and sends to the DLQ.
Only requires you to return which specific messages failed, not retry yourself.

Cost implications

Uncontrolled retries can explode your costs.

Example: a bug causes 1,000 requests a day to fail with a retryable error. Without proper retry logic:

Original 1,000 requests: 1,000 × $0.0135 = $13.50
3 automatic retries: 3,000 × $0.0135 = $40.50
Total: $54/day, or $1,620/month for a single bug.

With proper exponential backoff plus a circuit breaker:

Original 1,000 requests: $13.50
2 retries, not 3: about 2,000 × $0.0135 = $27
Circuit breaker stops after detecting a high failure rate: no further retries.
Total: about $40/day, or $1,200/month.

The circuit breaker pattern helps: if the error rate exceeds a threshold, stop retrying and fail fast.

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, reset_timeout: int = 60):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.failures = 0
        self.last_failure_time = None
        self.state = "CLOSED"  # CLOSED (working), OPEN (failing), HALF_OPEN (testing)

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.state = "OPEN"

    def should_attempt(self) -> bool:
        if self.state == "CLOSED":
            return True
        if self.state == "OPEN":
            # Check if reset timeout has passed
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = "HALF_OPEN"
                return True
            return False
        # HALF_OPEN: try one request
        return True

    def record_success(self):
        self.failures = 0
        self.state = "CLOSED"

When the circuit breaker trips, you fail fast instead of retrying, saving cost and reducing cascading failures.

Monitoring retries

Track retry behavior:

cloudwatch.put_metric_data(
    Namespace='APIMetrics',
    MetricData=[
        {
            'MetricName': 'RetryAttempts',
            'Value': attempt_number,
            'Unit': 'Count'
        },
        {
            'MetricName': 'SuccessAfterRetry',
            'Value': 1 if succeeded_after_retry else 0,
            'Unit': 'Count'
        }
    ]
)

If "SuccessAfterRetry" is high, more than 20% of requests needing retries, your system is fragile. If it's very low, close to zero, either your system is very reliable or your retries aren't working.

The checklist

Before shipping retry logic:

I've identified which errors are retryable vs. terminal.
My operation is idempotent.
I'm using exponential backoff and jitter, not linear or immediate retry.
Maximum retry time is reasonable, not days.
I have a circuit breaker to fail fast when the service is overwhelmed.
I'm monitoring retry rate and success-after-retry rate.
My DLQ has alerts configured.

Get this right and your system survives transient failures gracefully. Get it wrong, and you turn a 3-minute outage into a cascading disaster.

Want help hardening your error handling? Book a call and we'll walk through your retry logic together.

Get the free AI Readiness Checklist

15 questions to diagnose your team’s AI readiness, where you’ll see ROI fastest, and what to tackle first.

✓ Takes 5 minutes ✓ Actionable next steps ✓ No sales pitch

No spam. Unsubscribe anytime.

Ready to build AI that actually works?

Let’s talk about how SRE discipline transforms AI from a risky experiment into a reliable business system.

Start the Conversation

About the author

Charles Harvey is the founder of Three Moons Network and a site reliability engineer who builds production-grade AI automation for small businesses — monitoring, cost visibility, and documentation included. He writes about his hands-on AI experiments at floggingclaude.com. Connect on LinkedIn or see the code on GitHub.