← austinnafe.com Evals Every AI project I ship runs through the same eval harness.
Each one answers the same four questions, measured the same way, by goldloop. Only the tests change between projects. Every number below comes from a real, dated run.
5projects
135tests, each run several times
0must-hold rules broken
How it works
It's a test for an AI, like a spelling test.
- Write the test.Questions, each with the right answer.“When is load-in?” → 3pm
- Ask more than once.AI can answer differently each time. It only passes if it's right every time.
- Some answers are never OK.Like spending money without asking. One of those fails everything.
- Test before and after.If a change makes anything worse, it doesn't ship.
CostWhat does each task cost to run?
ReliabilityDoes it work every time, not just once?
RiskDo the rules that must hold, hold?
ChangeDid the last change help, or break something?
TM Go
Reads messy venue emails into a tour sheet. My production app.
gpt-4o-mini · API 61 tests × 3
- Cost
- $0.0002per task
- Reliability
- 55/61pass every run (80%–95%) · 2 flaky
- Risk
- 28/29guardrail tests pass every run · 0 must-hold rules broken
- Change
- goldloop found 7 missing names that had silently broken TM Go’s own offline tests since Sept 24.
Case study → Support agent
Looks up orders and issues refunds under guardrails. goldloop’s own demo.
gpt-4o-mini · API 32 tests × 3
- Cost
- $0.0001per task
- Reliability
- 32/32pass every run (89%–100%)
- Risk
- 17/17guardrail tests pass every run · 0 must-hold rules broken
- Change
- 14 of 14 deliberate breakages caught. A prompt “improvement” was rejected as noise hiding a bug.
Code → Meridian
Answers health-insurance members and stages bill payments for approval.
Claude Haiku · Claude Code, no API key 15 tests × 3
- Cost
- 1,550tokens per task, on a subscription
- Reliability
- 14/15pass every run (70%–99%) · 1 flaky
- Risk
- 2/2guardrail tests pass every run · 0 must-hold rules broken
- Change
- Swapping keyword rules for the LLM broke one question: “how much of my deductible is left?” lands in the wrong lane 2 times in 3.
Code → TripTalk
Finds flights, answers policy questions, stages bookings for approval.
Claude Haiku · Claude Code, no API key 16 tests × 3
- Cost
- 1,878tokens per task, on a subscription
- Reliability
- 16/16pass every run (81%–100%)
- Risk
- 5/5guardrail tests pass every run · 0 must-hold rules broken
- Change
- Swapping keyword rules for the LLM fixed 3 phrasings and broke none.
Code → Bakery assistant
Drafts replies for a small bakery and learns from the owner’s corrections.
Offline stand-in model 11 tests × 1
- Cost
- Measured once it runs on a real model.
- Reliability
- 11/11pass every run (74%–100%)
- Risk
- 6/6guardrail tests pass every run · 0 must-hold rules broken
- Change
- Learned from 7 corrections: 6 → 11 of 11 new messages. Rejected 2 lessons that would have broken a rule; learning everything breaks 2.
Code → How to read this
Pass every run counts a test only if it passed on every repeat. A test that passes 2 times in 3 fails one user in three. The range is a 95% interval, because a few dozen tests can't pin a number down exactly.
Guardrail tests guard money, permissions and made-up answers. The hardest of them are must-hold rules (never pay without approval, never auto-fill a cancellation): breaking one fails the project outright, whatever its average.
Outages aren't failures. When a provider is down or out of credit, the run is marked unfinished and isn't scored. Meridian and TripTalk ran on a Claude subscription through Claude Code, with no API key.