August 4, 2026 8 min read AI · SRE · AWS

Dead Letter Queues: The Safety Net Your Pipeline Is Missing

If a document fails processing and you don’t have a dead letter queue, it just vanishes. Here’s how to build the safety net that catches it instead.

Your AI pipeline works great. Documents go in, Claude processes them, results come out.

Then one day, a document fails processing. Claude returns malformed data. Your validation rejects it. Where did it go?

If you don’t have a dead letter queue, it vanished. You have no idea what failed, why it failed, or whether customers are impacted. You’re just missing data and hoping nobody notices.

Every production pipeline needs a safety net for this.

Dead letter queues are boring infrastructure. But they’re the difference between handling a failure and losing it.

What’s a Dead Letter Queue?

A dead letter queue (DLQ) is where failed messages go.

You have a normal queue for documents to process. If a document fails — Lambda throws an error, validation rejects it — instead of disappearing or retrying forever, it goes to the DLQ.

The DLQ is a staging area. You can inspect failures, understand what went wrong, and decide whether to retry or escalate.

Without a DLQ, a failed message is lost forever. With one, it goes to the DLQ where you can investigate and retry.

Why You Need This

Scenario 1: Malformed input. A customer uploads a PDF that’s corrupted or in an unexpected format. Claude can’t process it. The Lambda function throws an error.

Without a DLQ, the error gets logged and the message disappears. The customer’s document vanishes, and they email support asking where their result is.

With a DLQ, the message lands there instead. You investigate, find the problem, email the customer, and ask for a corrected file. When they upload it, you reprocess.

Scenario 2: Downstream integration fails. Claude successfully extracts data from an invoice. But your integration with NetSuite is temporarily down. The Lambda can’t post the result.

Without a DLQ, the data is extracted but never reaches the system — you’re stuck manually hunting for it and retrying by hand. With a DLQ, the failed message waits there until NetSuite recovers, then you replay it. Data reaches the system with no manual work.

Scenario 3: Unexpected edge case. Your validation logic has a bug. It rejects valid data from a new customer because of an edge case you didn’t anticipate.

Without a DLQ, that valid data is just gone — missed customer service calls, missed revenue, no trace of what happened. With a DLQ, the failed messages pile up and the spike gets your attention. You investigate, find the bug, fix it, and replay everything that got rejected.

Scenario 4: Cost runaway. A bug causes your Lambda to make 100x more Claude API calls than it should. Your monthly bill goes from $500 to $50,000. You panic and kill the pipeline.

What happens to the messages that were in flight? Without a DLQ, they’re gone. With one, they landed there when the Lambda crashed. Fix the bug, then replay them with the corrected code.

SQS Dead Letter Queue Setup

SQS makes this easy:

# Main queue for documents to process
resource "aws_sqs_queue" "document_queue" {
  name                      = "document-processing"
  visibility_timeout_seconds = 300  # 5 minutes for Lambda to finish
  message_retention_seconds  = 86400  # Keep for 1 day
}

# Dead letter queue
resource "aws_sqs_queue" "document_dlq" {
  name                      = "document-processing-dlq"
  message_retention_seconds = 1209600  # Keep for 14 days (investigate later)
}

# Configure main queue to send failures to DLQ
resource "aws_sqs_queue" "document_queue" {
  name                      = "document-processing"
  visibility_timeout_seconds = 300

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.document_dlq.arn
    maxReceiveCount     = 3  # After 3 failed attempts, send to DLQ
  })
}

How it works:

Message arrives in main queue
Lambda processes it
If Lambda succeeds, message is deleted from queue
If Lambda fails (throws error, times out, crashes), message is made visible again
After 3 failed attempts, message is moved to DLQ

Lambda has a maximum retry count. After that, it’s the queue’s job to route failures.

Lambda and SQS Integration

In your Lambda:

import json
import boto3
import anthropic

sqs = boto3.client("sqs")

def lambda_handler(event, context):
    for record in event["Records"]:
        try:
            message = json.loads(record["body"])
            document_id = message["document_id"]
            content = message["content"]

            # Process the document
            client = anthropic.Anthropic()
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": content}]
            )

            extracted_data = response.content[0].text

            # Validate the response
            is_valid = validate_extracted_data(extracted_data)
            if not is_valid:
                raise ValueError("Extracted data failed validation")

            # Store the result
            store_result(document_id, extracted_data)

            # Delete from queue (success)
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=record["receiptHandle"]
            )

        except Exception as e:
            # Log the error
            print(f"Error processing {record['messageId']}: {str(e)}")

            # Don't delete the message; let SQS retry
            # After max retries, it goes to DLQ automatically
            raise

SQS handles the retry logic automatically. You just let the Lambda fail.

Investigating DLQ Messages

Messages pile up in the DLQ. Now what?

Step 1: Monitor DLQ Depth

Set up a CloudWatch alarm:

resource "aws_cloudwatch_metric_alarm" "dlq_depth" {
  alarm_name          = "document-processing-dlq-depth"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 300
  statistic           = "Average"
  threshold           = 10
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    QueueName = aws_sqs_queue.document_dlq.name
  }
}

When the alarm fires, you know something is wrong.

Step 2: Read and Analyze DLQ Messages

def inspect_dlq(queue_url: str, max_messages: int = 10):
    """Read messages from DLQ and inspect them."""

    sqs = boto3.client("sqs")
    response = sqs.receive_messages(
        QueueUrl=queue_url,
        MaxNumberOfMessages=max_messages,
        WaitTimeSeconds=10
    )

    failures = []

    for message in response["Messages"]:
        body = json.loads(message["Body"])
        failures.append({
            "message_id": message["MessageId"],
            "body": body,
            "receipt_handle": message["ReceiptHandle"],
            "approximate_receive_count": message.get("Attributes", {}).get("ApproximateReceiveCount")
        })

    return failures

# Example output
failures = inspect_dlq(dlq_url)
for failure in failures:
    print(f"Message {failure['message_id']}:")
    print(f"  Document ID: {failure['body']['document_id']}")
    print(f"  Received {failure['approximate_receive_count']} times")
    print(f"  Body: {failure['body']}")

Step 3: Categorize the Failure

Is it a bad input, like a customer uploading a corrupted PDF? A code bug, where your validation rejects data that’s actually fine? A service outage, where the Claude API itself was down? Or an integration failure, where a downstream system was unreachable? Each requires a different fix.

def categorize_failure(message: dict) -> str:
    document_id = message["document_id"]
    content = message.get("content", "")

    # Check if input is malformed
    if not content or len(content) > 100000:
        return "bad_input"

    # Check for PDF corruption indicators
    if "PDF" in str(content) and len(content) < 50:
        return "corrupted_pdf"

    # If we got here, likely a code or service issue
    return "code_or_service_issue"

Replaying Messages

Once you’ve fixed the root cause, replay the messages.

def replay_dlq_message(dlq_url: str, receipt_handle: str, message_body: dict):
    """Move a message from DLQ back to main queue."""

    sqs = boto3.client("sqs")

    # Send to main queue
    sqs.send_message(
        QueueUrl=main_queue_url,
        MessageBody=json.dumps(message_body)
    )

    # Delete from DLQ
    sqs.delete_message(
        QueueUrl=dlq_url,
        ReceiptHandle=receipt_handle
    )

    print(f"Replayed message {message_body['document_id']}")

# Replay all messages in DLQ
failures = inspect_dlq(dlq_url, max_messages=100)
for failure in failures:
    replay_dlq_message(dlq_url, failure["receipt_handle"], failure["body"])

Or replay selectively, fixing only specific failures:

# Replay messages for a specific document type
failures = inspect_dlq(dlq_url)
for failure in failures:
    if failure["body"]["document_type"] == "invoice":
        replay_dlq_message(dlq_url, failure["receipt_handle"], failure["body"])

Operational Pattern

Daily

Check DLQ depth. It should be near zero. If it’s not, investigate why.

When Failures Spike

Pull 10-20 messages from the DLQ and look for patterns — same document type, same time period, same error. Categorize the failure, fix the root cause, then replay the messages.

Weekly

Review DLQ metrics and look for trends. Are failures increasing? What types? Update your validation or error handling based on what you find.

Cleanup

Messages in the DLQ older than 14 days are automatically deleted. Log them for analysis before that happens, and pull out what you learned — “10% of failures are malformed PDFs, so add better input validation” is the kind of note worth keeping.

The Terraform Pattern

# variables.tf
variable "dlq_message_retention" {
  type    = number
  default = 1209600  # 14 days
}

variable "max_receive_count" {
  type    = number
  default = 3
}

# main.tf
resource "aws_sqs_queue" "dlq" {
  name                      = "${var.queue_name}-dlq"
  message_retention_seconds = var.dlq_message_retention

  tags = {
    Purpose = "Dead letter queue for ${var.queue_name}"
  }
}

resource "aws_sqs_queue" "main" {
  name                      = var.queue_name
  visibility_timeout_seconds = var.visibility_timeout

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = var.max_receive_count
  })

  tags = {
    Purpose = "Main processing queue"
  }
}

# CloudWatch alarm
resource "aws_cloudwatch_metric_alarm" "dlq_depth" {
  alarm_name          = "${var.queue_name}-dlq-depth"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 300
  statistic           = "Average"
  threshold           = 5
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    QueueName = aws_sqs_queue.dlq.name
  }
}

The Cost

Dead letter queues cost almost nothing. SQS charges per 64KB block of data — a DLQ holding 100 messages runs pennies a month.

The real cost of skipping one is hidden, and it shows up later as lost data and an emergency debugging session when a customer notices before you do.

A DLQ is cheap insurance. It sits there quietly until the day you need it.

The Pattern

Every queue needs a DLQ, and every async system needs somewhere for failures to land instead of vanishing.

Build it before you deploy, monitor it, and replay failures quickly once you understand them.

Your future self will thank you when the system breaks at 2am and you can investigate calmly instead of guessing.

Get the free AI Readiness Checklist

15 questions to diagnose your team’s AI readiness, where you’ll see ROI fastest, and what to tackle first.

✓ Takes 5 minutes ✓ Actionable next steps ✓ No sales pitch

No spam. Unsubscribe anytime.

Ready to build AI that actually works?

Let’s talk about how SRE discipline transforms AI from a risky experiment into a reliable business system.

Book Your Free Discovery Call

About the author

Charles Harvey is the founder of Three Moons Network and a site reliability engineer who builds production-grade AI automation for small businesses — monitoring, cost visibility, and documentation included. He writes about his hands-on AI experiments at floggingclaude.com. Connect on LinkedIn or see the code on GitHub.