The Vendor Evaluation Checklist I Give Every Client

I’ve sat across the table from dozens of SMB owners who hired an AI consultant, paid $50k to $200k, and ended up with a system so tightly coupled to that consultant’s choices that they can’t touch it without breaking it. This doesn’t have to be your story.

Over the past three years working as an SRE, and now as an AI consultant, I’ve developed a checklist. It started as a spreadsheet. Now it’s the first thing I share with clients who ask me: “How do I evaluate whether this vendor is the right fit?”

I’m giving it to you.

The Five Critical Questions

Question 1: Who Owns the Infrastructure?

This is the first line. If your vendor builds the entire system on their AWS account, you’re renting a black box. You have no visibility, no control over cost optimization, and zero negotiating power if they raise prices.

The answer you want: “We’ll provision infrastructure in your AWS account. You own it. We’ll help you set it up and manage it, but it’s yours.”

Red flag: “We have a standard architecture that runs on our infrastructure.” This means you’re locked in.

Question 2: Can I See the Code?

Not necessarily all of it, but the key bits: the Lambda functions that process your data, the prompt templates, the validation logic. You should be able to audit it, understand it, and eventually maintain it yourself.

Red flag: “It’s proprietary.” Ask yourself: if this consultant disappears tomorrow, are you stuck?

Question 3: How Do You Monitor What’s Happening?

Your vendor should be showing you real dashboards: invocation counts, error rates, latency, cost per transaction. These should be visible to you in real time — not sent to you in a monthly report.

Red flag: “We send you a summary each month.” That’s not observability. That’s theatre.

Ask for specifics: are they using CloudWatch? Datadog? What metrics do they track? Can you access the dashboards, or do you have to ask for screenshots?

Question 4: What Happens at Handoff?

This is the one most vendors skip. Ask: “If we part ways in a year, what do I get? A runbook? Training? A month of on-call support?”

The answer matters. A professional vendor will have a documented handoff process. They’ll teach someone on your team to maintain the system.

Red flag: “We’ll be here forever, don’t worry about it.” That’s not reassuring. That’s a trap.

Question 5: How Do You Handle Failures?

Ask them to walk you through a failure scenario: “What happens if the Claude API is down for an hour? What happens if a Lambda function fails processing a document? How do you know it happened? How do I know it happened?”

Their answer reveals everything about their engineering maturity. A good vendor will describe retry logic, dead letter queues, alerts, and a documented runbook.

Red flag: “That won’t happen” or “we have redundancy.” Those aren’t answers. Press harder.

The Infrastructure Audit

Once you’ve cleared the five questions, audit the actual infrastructure. Ask for (or request read access to):

• Their Terraform modules (or CloudFormation templates)
• A diagram of the data flow
• Cost breakdown: what does it cost to process 1,000 documents? 10,000?
• Security architecture: how is your data encrypted? Where is it stored? Who can access it?
• Deployment process: how do updates roll out? Can you test in staging?

The goal isn’t to become an expert. It’s to verify that they’ve thought through the system. Vague answers are red flags.

The Red Flags Checklist

• Opaque pricing. “We’ll send you a proposal” instead of explaining how costs scale with usage.
• No staging environment. They deploy directly to production.
• No monitoring dashboard. You have to ask them what’s happening.
• Minimal documentation. If they can’t explain it clearly, they probably don’t understand it well.
• Resistance to your questions. Professionals answer. Amateurs deflect.
• No disaster recovery plan. They haven’t thought about what breaks.
• Vendor lock-in by design. Custom integrations that only they can maintain.

What Success Looks Like

A vendor you can trust will give you:

• Infrastructure in your account, with clear ownership
• Readable, auditable code
• Real-time observability (not monthly reports)
• A documented handoff plan
• Honest conversations about risks and constraints
• Costs that scale linearly with usage, not exponentially

Your Checklist

Use this when you’re evaluating. Score each vendor on a scale of 1–5 for each category above. Anything below a 3 on infrastructure ownership, code transparency, or monitoring? Pass.

You’re not looking for perfection. You’re looking for partners who’ve thought through the system, documented it, and aren’t trying to make you dependent on them forever.

That’s the consulting vendor worth paying for.