LangGraph
Bounded state-machine agent, not an open loop
Architecture Case Study Live in production
TM Go
An autonomous tour-management agent for independent bands — it advances shows over email and SMS, reads free-form venue replies, extracts the logistics, and keeps each tour's records in sync. This document covers the system, the seven engineering decisions that made it safe to write to a band's real data, and how it scales.
100%
Live field-extraction accuracy; CI gate at 80%
2
LLM providers with automatic failover
59
API routes; Pydantic-validated, containerized
01 System & stack
System overview
The real deployed system, with prototype shortcuts named as such rather than hidden.
FrontendReact + Vite, deployed on Vercel
APIStarlette → FastAPI — 59 routes; Pydantic-validated AI endpoints with OpenAPI docs
Agent coreLangGraph StateGraph — a router dispatching to explicit, deterministic pipelines
LLMLangChain
init_chat_model — OpenAI primary, Anthropic failoverSMSFastAPI + Twilio webhook
DataGoogle Sheets (per-band system-of-record) + Supabase (auth, storage, AI telemetry)
SchedulerRailway cron worker for reminders
Reliability
traced_llm observability + closed-loop evaluation + /api/ai-health dashboardDeliveryGitHub Actions CI (build · lint · eval gate) + Docker; Vercel/Render auto-deploy
Deliberately not included: billing (no Stripe yet). The API was Starlette, upgraded to FastAPI specifically for the AI endpoints. Naming what isn't there is part of the point.
Decision 01 Agent design
A bounded state-machine agent
The problemAdvancing a show is a multi-step, correctness-sensitive workflow. Writing an incorrect load-in time into a band's real tour data is worse than writing nothing at all — so the agent's behavior has to be predictable.
Naive approach
A ReAct agent that reasons and picks tools freely each turn.
It re-plans every turn, its tool order is non-deterministic, and it can skip steps. Effective in a demo; unacceptable for a workflow that writes to real data.
What I shipped
A hand-authored LangGraph StateGraph. A router node classifies intent once, then dispatches to explicit pipelines (e.g. advance: read venue → draft email).
The LLM only drafts prose and parses free text into JSON. Routing, sheet I/O, validation, and email transport are deterministic Python.
PrincipleDeterminism where correctness matters; the model only where language matters. The right move was not a more capable agent — it was a more constrained one.
Decision 02 State management
Deriving state rather than storing it
The problemThe interface must always show the correct "whose turn is it" across dozens of concurrent venue threads.
Naive approach
A stored status column on each thread.
It drifts out of sync: a reply arrives but the column still reads sent, so the interface reports the wrong state with full confidence.
What I shipped
"Whose turn" is computed as a pure function of (messages, sheet facts) — derived, never stored.
A derived value cannot drift from the source of truth. A stored copy eventually always does.
PrincipleIf a value can be recomputed from the source of truth, deriving it removes an entire class of consistency bugs.
Decision 03 Inbox modeling
The inbox as a workflow queue
The problemSend outreach, receive replies, convert replies into structured facts, and generate the next email — across dozens of venue threads at once.
Naive approach
Build an email client. That merges three representations of one conversation: live IMAP, logged sends, and optimistic UI state.
They disagree — messages reorder, duplicate, and deleted ones reappear.
What I shipped
Reframed as a queue keyed on the show, not the message. Each cycle reconstructs "what to ask" from (questions enabled) − (facts on sheet).
The system never relies on remembered conversation state — it recomputes what remains outstanding.
PrincipleCollapsing three sources of truth into one derived view eliminated the entire ordering-and-duplication bug class.
Decision 04 Reliability
Reliability as a design requirement
Each of these exists because a specific failure would place an incorrect fact in front of a band on tour. They are requirements, not polish.
Provider failover
OpenAI primary, Anthropic on failure. A single provider's outage is not the product's outage.
Tolerant parsing
JSON extraction survives code fences and stray prose, so a formatting quirk doesn't fail a valid extraction.
Human-in-the-loop
Extractions are reviewed before reaching the sheet. Gap-fill only — a confirmed value is never overwritten.
Validated inputs
A malformed request returns a clean 422, not a deep KeyError several calls into the stack.
Decision 05 Evaluation & observability
Measuring correctness, and learning from corrections
Online — closed loop
The human review step logs every accept, edit, and reject. Each correction is a labeled failure — a live dataset of the agent's weak points at zero labeling cost.
Offline — CI gate
A golden-set harness scores field extraction and fails the build below 80% accuracy (currently 100% live). A prompt or model change cannot merge if it regresses.
One tracing seam
traced_llm is the single point every model call passes through — recording endpoint, model, latency, tokens, estimated cost, and whether failover fired.
100%
Live extraction accuracy
p50 / p95
Latency, tracked live
Cost / call
Per-request spend
Failover %
Provider health
In one sentenceThe agent is instrumented end to end — per-call cost, latency, and failures are visible live, and every human correction becomes labeled evaluation data that gates deployment.
Decision 06 Data model
A prototype data model with a defined ceiling
Why it was the right call
Google Sheets is each band's system-of-record. Bands can see their own data with zero setup — ideal for proving the workflow.
Where it breaks
It is also what breaks first at scale: API quotas, no transactions, no concurrency guarantees.
I'm not presenting it as permanent — naming the ceiling is the point.
PrincipleSelecting the right tool for the current stage — and stating exactly where it will fail before it does — is the engineering judgment, not a gap in it.
Decision 07 Delivery
Evaluation-gated continuous delivery
CI on every push
GitHub Actions runs the frontend build, a real-error lint, and the evaluation gate. A regression in extraction accuracy blocks the merge.
Automatic deploy
A passing main deploys itself to Vercel and Render — no manual release step to forget.
Containerized backend
Docker packages the backend to run identically anywhere. This container is the unit a worker pool would replicate at scale.
Why it mattersBecause the backend is already containerized, moving to a worker pool is a deployment change, not a rewrite — the scaling path was designed in from the start.
Forward · 02 Scaling path
Behavior at scale, and the planned evolution
Presented as a roadmap, not a current claim. The system is not running at this scale today — but identifying precisely what fails first, and why, is the substance of the question.
Breaks first at 100×
- Sheets as database — quotas, no transactions
- IMAP polling and cold starts
- Synchronous LLM calls on the request path
- The three-source inbox merge
Target architecture
- Postgres as system-of-record; Sheets becomes an export
- Queue + worker pool for concurrent runs (the backend is already containerized for this)
- LLM gateway — caching, the existing failover, and per-tenant budgets, built on the
traced_llmseam - Event-driven email (Gmail API / webhooks) replacing polling
- Multi-tenant isolation, with evaluation as a release gate
Carry forward: deterministic-first design, provider failover, human-in-the-loop review, and the observability seam. The foundations are sound.
Rebuild: the storage model and the inbox's three-source flow — the prototype shortcuts that do not survive multi-tenant load, each for a reason I can state precisely.
Forward · 03 Summary
Summary
TM Go is an autonomous tour-management agent for independent bands — advancing shows over email and SMS, drafting venue outreach, parsing free-form replies into structured logistics, and keeping each tour's records in sync. The central engineering decision was constraining the agent: a hand-authored LangGraph StateGraph replaces an open ReAct loop, with the model confined to drafting and parsing and all state, routing, and I/O implemented as deterministic code. It ships with production-reliability plumbing — provider failover, tolerant parsing, validated inputs, human-in-the-loop review — and a traced_llm observability layer that doubles as a closed-loop evaluation harness: every human correction becomes labeled data, and a CI gate blocks any change that reduces extraction accuracy.
One-line version"I constrained an agent so it could be trusted with real data, then instrumented it so that every human correction became evaluation data gating deployment."
🤖 Generated with AI, this one — I'll own it. The day one of these is my own typing, you won't need the disclaimer to tell. Fewer em-dashes, more typos. Trust me, you'll know.