October 6, 2026 7 min read AI · Monitoring · AWS

The Monday Morning Dashboard: What I Check First

Every Monday at 8:55 AM, before I check email or Slack, I open a single CloudWatch dashboard and spend 60 seconds scanning five panels. Those 60 seconds tell me whether a system is healthy or something’s silently degrading.

This is the dashboard I’ve built into every AI system I run. It’s not fancy, but it catches problems before they become pages.

The five panels

Panel 1: Invocation Success Rate (Last 7 Days)

This is a line graph showing successful Lambda invocations as a percentage. I want to see a flat line above 99%. If it dips below 98%, something’s wrong.

Metric: Successful Invocations / Total Invocations (rolling 1-hour window)
Threshold: RED if <98%, YELLOW if 98-99%, GREEN if >99%

Why this metric? It captures everything: Lambda errors, timeouts, code crashes, permission issues, downstream service failures. A 1% failure rate is still 8-10 failures in a million invocations. At 1,000 requests a day, that’s one failure a week — acceptable. Drop to 98% and you’re at 20 failures a week. Something’s wrong.

The math:

99% success = 0.1% error rate = 1,000 requests/day, 1 failure/day
98% success = 2% error rate = 1,000 requests/day, 20 failures/day
95% success = 5% error rate = 1,000 requests/day, 50 failures/day

I notice the difference between 99% and 98% immediately.

Panel 2: P99 Latency (Last 7 Days)

A line graph showing the 99th percentile of request latency. If your API calls take 5 seconds on average but P99 is 45 seconds, you’ve got a tail latency problem.

Metric: Duration (P99 percentile)
Threshold: RED if >30s, YELLOW if 15-30s, GREEN if <15s

For real-time AI systems, anything under 15 seconds is healthy. 15-30 seconds means something’s slow but probably not the main work, likely cold starts or verbose API responses. 30+ seconds means your Lambda might be timing out or a downstream service is hanging.

If P99 suddenly spikes from 8 seconds to 28 seconds, it usually means:

Claude API is slower than usual
Your Lambda cold start time increased, from a code change or a larger bundle
A downstream database is slow

Panel 3: Error Classification (Last 24 Hours)

A stacked bar chart showing error types. I categorize errors into:

4xx errors (user fault): invalid input, authentication failures, bad request formatting.
5xx errors (system fault): Lambda crashes, timeouts, downstream service errors.
Messages that failed every retry and landed in the dead-letter queue count separately as DLQ messages (unprocessable).

Metric: Count of errors by category
Threshold: RED if DLQ depth >5, YELLOW if >2

The DLQ depth is the number that matters most. If messages are piling up there, you’ve got a class of requests the system can’t handle: a new document type, an API rate limit, or something that regressed in the code.

I want that number at zero or close to it. Above 5, I go investigate.

Panel 4: Cost Trend (Last 30 Days)

A line graph showing daily AWS and Claude API spend. I plot two lines:

Projected spend: current daily average × days in month
Actual spend: cumulative spend so far

Metric: Estimated daily cost (AWS + Claude API)
Threshold: RED if projected exceeds budget by >15%, YELLOW if 5-15%, GREEN if within budget

For a system with a $500/month budget, I’d set GREEN at $400-500 projected, YELLOW at $500-575, and RED above $575.

If the cost jumps on day 5, you’ve got a runaway loop or unexpected traffic. This metric catches it immediately. The formula for daily cost:

# cost so far, divided by days elapsed, projected across the month
daily_aws = cloudwatch_cost_so_far / days_so_far
daily_claude = sum(input_tokens, output_tokens) * pricing_rate / days_so_far
projected = (daily_aws + daily_claude) * 30

Panel 5: DLQ Depth (Last 24 Hours)

A simple gauge showing how many messages are currently sitting in the dead-letter queue waiting to be handled.

Metric: SQS ApproximateNumberOfMessagesVisible (on DLQ)
Threshold: RED if >5, YELLOW if >2, GREEN if 0

A message in the DLQ means it failed normal processing three times, the default retry count, then failed Lambda error handling too. The system gave up and pushed it somewhere for a human to deal with.

A growing number means a leak. A static number usually means a batch of broken messages waiting on manual remediation.

The Terraform code

Here’s how I build this dashboard:

resource "aws_cloudwatch_dashboard" "ai_system" {
  dashboard_name = "ai-system-monday-health"

  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", { stat = "Sum" }],
            ["AWS/Lambda", "Errors", { stat = "Sum" }]
          ]
          period = 3600
          stat   = "Average"
          region = "us-east-1"
          title  = "Success Rate (%)"
          yAxis = {
            left = { min = 90, max = 100 }
          }
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Duration", { stat = "p99" }]
          ]
          period = 3600
          region = "us-east-1"
          title  = "P99 Latency (ms)"
          yAxis = {
            left = { min = 0, max = 60000 }
          }
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Errors", { stat = "Sum" }],
            ["AWS/ApiGateway", "4XXError", { stat = "Sum" }]
          ]
          period = 300
          region = "us-east-1"
          title  = "Error Classification (24h)"
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["Custom/Billing", "DailyEstimatedCost"]
          ]
          period = 86400
          region = "us-east-1"
          title  = "Cost Trend (30d)"
          yAxis = {
            left = { min = 0, max = 750 }
          }
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/SQS", "ApproximateNumberOfMessagesVisible", { dimensions = { QueueName = "dlq" } }]
          ]
          period = 300
          stat   = "Average"
          region = "us-east-1"
          title  = "DLQ Depth"
          yAxis = {
            left = { min = 0, max = 20 }
          }
        }
      }
    ]
  })
}

resource "aws_cloudwatch_metric_alarm" "success_rate_low" {
  alarm_name          = "ai-system-success-rate-low"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "SuccessRate"
  namespace           = "Custom/AISystem"
  period              = "300"
  statistic           = "Average"
  threshold           = "98"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  alarm_description   = "Success rate has dropped below 98%"
}

resource "aws_cloudwatch_metric_alarm" "dlq_depth_high" {
  alarm_name          = "ai-system-dlq-depth-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "1"
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = "300"
  statistic           = "Average"
  threshold           = "5"
  dimensions = {
    QueueName = "ai-system-dlq"
  }
  alarm_actions     = [aws_sns_topic.critical_alerts.arn]
  alarm_description = "DLQ has >5 messages"
}

What this catches

In the last 12 months, this dashboard has alerted me to:

A runaway Lambda that was retrying infinitely, pushing cost from $200/month to $1,800/month over a weekend. Caught by the cost trend spike on Monday morning.
A Claude API rate limit where a burst of traffic triggered a 10% failure rate. Caught by the success rate dropping to 90%, with latency spiking alongside it.
A misconfigured DLQ where messages were being written but never read, building a backlog nobody saw. Caught by DLQ depth climbing from 0 to 127 over a week.
A downstream database timeout where queries that normally took 200ms started taking 15+ seconds. Caught by P99 latency spiking from 5 seconds to 35 seconds.
A code regression where a new deployment broke input validation, failing nearly every request with a 400. Caught by success rate dropping from 99.8% to 60% within minutes.

Without this dashboard, I’d find out about all of these on Tuesday or Wednesday, after they’d already been bleeding money or users for 24+ hours.

Operational routine

Monday, 8:55 AM: open the dashboard.

Success rate: green? If not, check the error logs.
Latency: green? If not, check CloudWatch Logs Insights for slow invocations.
Errors: any DLQ messages? If yes, sample a few and see what’s wrong.
Cost: tracking to budget? If not, run a quick AWS Cost Explorer query.
DLQ depth: growing? If yes, start investigating why messages aren’t being processed.

Total time: 60 seconds when everything’s green, 10-15 minutes if something’s wrong.

That beats getting paged at 3 AM because nobody was watching.

Get the free AI Readiness Checklist

15 questions to diagnose your team’s AI readiness, where you’ll see ROI fastest, and what to tackle first.

✓ Takes 5 minutes ✓ Actionable next steps ✓ No sales pitch

No spam. Unsubscribe anytime.

Ready to build AI that actually works?

Let’s talk about how SRE discipline transforms AI from a risky experiment into a reliable business system.

Start the Conversation

About the author

Charles Harvey is the founder of Three Moons Network and a site reliability engineer who builds production-grade AI automation for small businesses — monitoring, cost visibility, and documentation included. He writes about his hands-on AI experiments at floggingclaude.com. Connect on LinkedIn or see the code on GitHub.