August 31, 2026 5 min read AI Engineering · Prompt Engineering · DevOps

Treat Your Prompts Like Production Code

A missing word in a system prompt broke document classification for two weeks before anyone caught it. The same everything-in-code discipline that governs infrastructure applies just as directly to prompts: Git, semantic versioning, and CI tests, not manual edits nobody can trace.

I was debugging why a Claude-based document classification system was suddenly rejecting valid resumes when it had been working fine for two weeks. Turned out someone updated the system prompt to be “more helpful” and accidentally removed the instruction that told Claude to preserve structural integrity when the classification failed. One word change. Sixteen hours of confusion.

It’s the same failure mode I’ve spent a career guarding against in infrastructure: a change made outside source control, with no diff, no review, and no way to know what the previous version looked like. The everything-in-code discipline that governs how I manage Terraform and CI pipelines applies just as directly to prompts.

Prompts Are Code Artifacts

A prompt is an instruction set that determines model behavior. It lives or dies on its exact wording. A missing comma can change meaning. A reordered instruction can flip which rule takes precedence. When you’re building production systems, this precision matters.

So why would you store prompts differently than you store Python functions?

I version all prompts in Git. This means:

Every change is tracked with a commit message explaining why.
I can tag stable versions and roll back in seconds.
I can run A/B tests on different prompt versions without fear of losing the original.
New team members onboard knowing the why behind every instruction.

The File Structure

Here’s how I organize it:

prompts/
├── classification/
│   ├── resume-classifier.md
│   ├── resume-classifier.test.json
│   └── CHANGELOG.md
├── extraction/
│   ├── invoice-line-items.md
│   ├── invoice-line-items.test.json
│   └── CHANGELOG.md
└── generation/
    ├── email-draft.md
    ├── email-draft.test.json
    └── CHANGELOG.md

Each prompt lives in markdown with clear sections: role, task, constraints, input format, output format, examples.

The test file contains a small fixture of representative inputs and expected outputs. This is crucial.

Prompt Versioning in Practice

I use semantic versioning for prompts: MAJOR.MINOR.PATCH. A MAJOR bump means the fundamental task changed, like switching from “classify resume as relevant/irrelevant” to “classify resume and extract key qualifications.” MINOR covers a behavior change that doesn’t break existing expectations, say tightening date-format handling. PATCH is a wording clarification where the expected output stays identical.

When I deploy a prompt change to production, I tag the commit: resume-classifier-v2.1.0. This way, if the system starts behaving unexpectedly, I can instantly check what changed and when.

A/B Testing Prompts

Before I ship a prompt update to all traffic, I test it against the fixture:

import json
import subprocess

def test_prompt_version(prompt_file, test_fixtures):
    """Run test fixtures against a prompt version."""
    results = []

    for fixture in test_fixtures:
        # Call Claude with the prompt
        response = call_claude(
            system=read_prompt(prompt_file),
            user=fixture['input']
        )

        # Compare to expected output
        match = response.strip() == fixture['expected'].strip()
        results.append({
            'input': fixture['input'][:50],
            'passed': match,
            'response': response[:100]
        })

    return results

# Compare v2.0.0 (current production) vs v2.1.0 (candidate)
current = test_prompt_version('prompts/resume-classifier.md', fixtures)
candidate = test_prompt_version('prompts/resume-classifier-v2.1.0.md', fixtures)

print(f"Current: {sum(1 for r in current if r['passed'])}/{len(current)} pass")
print(f"Candidate: {sum(1 for r in candidate if r['passed'])}/{len(candidate)} pass")

I run this locally before I even commit. If the candidate fails tests that the current version passes, I don’t merge. This prevents the “works fine locally, breaks in production” disaster.

CI Pipeline for Prompts

Once the test passes, I add a GitHub Actions workflow:

name: Prompt Validation

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install anthropic pytest

      - name: Run prompt tests
        run: pytest prompts/tests/ -v
        env:
          ANTHROPIC_API_KEY: {{ secrets.ANTHROPIC_API_KEY }}

      - name: Check prompt syntax
        run: python scripts/validate_prompts.py

      - name: Report results
        if: failure()
        run: |
          echo "Prompt tests failed. Review the changes before merging."
          exit 1

Every PR that touches a prompt file runs through this. No prompt update gets merged without passing its test suite.

The Rollback Story

Last month, a prompt change for a content moderation system was deployed at 10 AM. By 2 PM, we noticed it was flagging legitimate user posts as violations. Instead of trying to debug live, I rolled back:

git log prompts/moderation.md
# Find the commit before the change
git show abc123:prompts/moderation.md > prompts/moderation.md
git commit -m "Rollback moderation prompt to v1.2.0 (abc123)"

The change was reverted in 90 seconds. The old behavior came back immediately. Meanwhile, I reviewed what went wrong in the v1.3.0 PR, found the issue (an overconstrained instruction that was too strict), and shipped v1.3.1 the next day after testing.

Without version control, I would’ve been fumbling through backup files and hoping someone remembered what the prompt looked like before the change.

What Goes in the Commit Message

Resume classifier: Add explicit instruction to reject resumes without dates

- Clarify that "date missing" is grounds for immediate rejection
- Add test fixtures for edge case: resume with only year (no month/day)
- Version bump: v2.0.1 -> v2.1.0
- Tested against 247 fixture resumes, 98% pass rate (down from 99%, expected due to stricter rule)
- Deployment plan: canary to 10% traffic, monitor rejection rate for 2 hours

I’m explicit about the version, the reasoning, the test results, and the deployment plan. Future me (or future someone else) can read this commit and know exactly what changed and why.

Getting Started

If you’re running Claude in production, start today:

Export your current prompts to .md files in a prompts/ directory.
Create a test fixture for each prompt with 10-20 representative examples.
Add a simple validation script that runs the fixture against the prompt.
Push to Git, set up a GitHub Actions workflow, and require tests to pass.
Tag releases with semantic versions.

The overhead is minimal. The safety you gain is enormous.

Get the free AI Readiness Checklist

15 questions to diagnose your team’s AI readiness, where you’ll see ROI fastest, and what to tackle first.

✓ Takes 5 minutes ✓ Actionable next steps ✓ No sales pitch

No spam. Unsubscribe anytime.

Ready to build AI that actually works?

Let’s talk about how SRE discipline transforms AI from a risky experiment into a reliable business system.

Book Your Free Discovery Call

About the author

Charles Harvey is the founder of Three Moons Network and a site reliability engineer who builds production-grade AI automation for small businesses — monitoring, cost visibility, and documentation included. He writes about his hands-on AI experiments at floggingclaude.com. Connect on LinkedIn or see the code on GitHub.