How to Evaluate an AI Consultant (Without Being Technical)

You need AI automation for your business, but you don't have engineers on staff. Most AI consultants are selling magic, not engineering. Here are seven questions that separate the professionals from the cowboys.

You're looking to hire a consultant. Here's what worries me: most AI consultants will promise custom models, AGI-powered workflows, and disruption. They'll charge hourly. They'll hand you code you don't understand. And when they leave, nothing works.

I work with SMBs specifically because I think this is backwards. You shouldn't have to be technical to hire someone smart. But you do need to ask the right questions.

Here are seven things every business owner should verify before signing.

Question 1: “How Will You Measure Success?”

A consultant who can't answer this is winging it.

You need specific metrics. Not “we'll build an AI system.” But: “We'll reduce invoice processing time from 16 hours per week to 2 hours per week, with 98% accuracy, and we'll track this every month.”

The metric should be:

• Measurable — you can count it.
• Baseline-aware — we know where you start.
• Tied to business outcome — hours saved, errors reduced, revenue per customer.

Red flag: “We'll implement cutting-edge AI optimization.” What does that mean? Does it matter?

Green flag: “We'll extract invoice data, store it in your system, and we'll measure accuracy against your current manual baseline.”

Question 2: “What’s Your Monitoring and Observability Strategy?”

If your consultant doesn't answer immediately, they don't have one.

Here's what matters: when your automation fails, you find out within minutes. Not when a customer complains.

Ask them:

• “How will we know if something breaks?”
• “How will we track costs month-to-month?”
• “Can I see a dashboard of what's running and how much it costs?”

For AWS-native work, this means CloudWatch logs, metrics, and dashboards. For other platforms, equivalent tools.

Red flag: “We'll set up some logging and you can check it if something goes wrong.” Reactive, not proactive.

Green flag: “We'll build dashboards in CloudWatch. You'll see invocation counts, error rates, token usage, and monthly costs. We'll email you a summary every month.”

Question 3: “Do You Use Infrastructure-as-Code?”

This question separates professionals from cowboys.

Infrastructure-as-Code (IaC) means your entire system — databases, compute, networking, permissions — is defined in code (usually Terraform). It lives in Git. Every change is tracked. If something breaks, you can revert it.

Red flag: “We'll click through the AWS console and set it up.” You're now dependent on their memory and notes. If they disappear, reproducing the system takes weeks.

Red flag: “We use a proprietary platform that handles infra for you.” You just locked yourself in again.

Green flag: “Your entire system — API, database, compute, permissions — will be in a Git repo. Terraform code. Reproducible. Auditable. You own it.”

Question 4: “How Do You Price This?”

This is where most consultants mislead.

Hourly billing is a trap. It misaligns incentives. The consultant gets paid more if they're slow and your project drags. There's no reason to finish efficiently or document well.

Red flag: “We charge $200/hour and estimate 120 hours, so it'll be ~$24K.” That's not a contract; that's a cap on your trust. Once they're 60 hours in, they'll discover they need another 60. Now you've spent $12K and you're halfway done.

Red flag: “We charge per API call or per token.” You're gambling with your budget.

Green flag: “This project is $12,500 fixed price. Includes infrastructure, code, documentation, and a 30-day warranty. If it needs tweaks in month 1, we fix it for free.”

Question 5: “What Happens When You Leave?”

This is the handoff question. And it matters more than the build.

Ask:

• “Will I have all the code in a Git repo?”
• “Will there be documentation for running this system?”
• “Will there be a runbook for when something breaks?”
• “Can you run a training session with my team?”
• “What's your warranty period?”

Red flag: “The code will be in our proprietary platform.” No. Walk away.

Red flag: “We'll be available for support.” That's not handoff; that's ongoing dependency.

Green flag: Code in GitHub. README documenting the system architecture. RUNBOOK with step-by-step troubleshooting. 30-day warranty. One training session with your ops team.

Question 6: “Will You Talk About ‘Custom Models’?”

Beware this phrase.

Most consultants will say: “We'll build a custom fine-tuned model for your business.” In 2026, this is usually unnecessary. Foundation models like Claude are so capable that fine-tuning helps only in very narrow, specific cases — and even then, prompt engineering often works better.

Red flag: “We'll fine-tune a model.” For 99% of SMB use cases, you don't need this. You need good prompt engineering, proper architecture, and better data quality.

Green flag: “We'll use Claude via API. We'll optimize the prompt for your specific data. If we hit edge cases, we'll add guardrails and validation logic.”

Simpler, cheaper, and actually better.

Question 7: “Can I See an Example?”

Ask for a portfolio. A real project. Code, architecture diagram, results.

Red flag: “We can't share client work due to NDAs.” Fair, but they should have at least one public or anonymized example.

Red flag: Case study that's vague: “We saved a company 40% on costs.” How? What was the baseline? What was the outcome?

Green flag: Specific examples with metrics, architecture diagrams, and code samples — even if anonymized.

The Checklist

Before you sign, verify all seven:

• Success metrics are specific and measurable.
• Monitoring plan exists and you can see it.
• Infrastructure is Terraform (or equivalent IaC).
• Pricing is fixed, not hourly or per-invocation.
• Handoff plan includes code, docs, training, warranty.
• No mention of custom models unless justified.
• Portfolio with real examples.

If any box is unchecked, push back or walk.

Bottom Line

You don't need to understand AI to hire an AI consultant. You need to understand whether they understand business engineering.

The right consultant will make this system simple, auditable, and yours. Not complicated, opaque, and dependent.