TM Go

System overview

The real deployed system, with prototype shortcuts named as such rather than hidden.

Three state machines, kept orthogonal on purpose

The third machine is the subtle one. What the inbox renders (Waiting, Follow-up due, Reply received, Done) isn't advancing progress. It's conversation state, computed from the messages and the clock. That second input makes it different in kind, because it changes with no write at all. A venue goes from "waiting" to "follow-up due" three days after a send. Nothing happened: no reply, no click, no event. Time passed.

The escape hatch, stated plainly: a user can still set advancing status by hand. Four options, not the six the system computes, because showing every internal state read as overlapping choices. A pure derivation with no override is elegant, and it loses to reality the first time a venue confirms something over the phone.

Reliability as a design requirement

Each of these exists because a specific failure would place an incorrect fact in front of a band on tour. They are requirements, not polish.

How the email path is actually tested

Email is where this product lives, and it's the layer with the most ways to fail quietly. Nobody files a bug for replies that never arrive. So it carries the most tests, 59 across seven suites, and most of them are about telling apart states that look alike.

IMAP IDLE is the clearest example. The protocol is full of lookalikes, so each distinction gets pinned:

def test_exists_means_new_mail():
    conn = FakeConn([b"+ idling", b"* 7 EXISTS", b"A001 OK IDLE done"])
    assert ew.idle_once(conn, 1) is True

def test_quiet_timeout_is_not_mail():        # silence != a reply
    conn = FakeConn([b"+ idling", "TIMEOUT", b"A001 OK IDLE done"])
    assert ew.idle_once(conn, 1) is False

def test_keepalives_ignored_until_exists():   # EXPUNGE/RECENT are noise
    conn = FakeConn([b"+ idling", b"* 3 EXPUNGE", b"* 2 RECENT",
                     b"* 9 EXISTS", b"A001 OK"])
    assert ew.idle_once(conn, 1) is True

def test_idle_refused_raises_unsupported():   # fail loud, not idle forever
    conn = FakeConn([b"A001 NO IDLE not allowed"])
    assert raises(ew.IdleUnsupported)

The rest of that suite covers a server BYE raising a connection error, backoff that doubles and caps, and the one I care most about: an exception in a handler never kills the watch loop. One bad message must not silently end email ingestion.

Migrating off the spreadsheet without a big-bang cutover

The hard part isn't Postgres. It's moving while the thing is running. A band on tour can't have a maintenance window. So the migration runs as a strangler fig, gated by a measurement, in five reversible steps.

Two LLM seams, and everything else is code

The model appears at exactly two points, both of them language problems: parsing a venue's free-form reply into typed fields, and drafting the outbound message. Routing, scheduling, sheet writes, validation and retries are all deterministic Python. That placement is the load-bearing decision the other six hang off.

The shape underneath all of them is a decision about where a truth is allowed to come from. The model proposes language, never facts. A projection is computed in one place. Booking status comes from a human because nothing else knows it. Cutover comes from a parity number rather than confidence. Put that way, "constrain the agent" stops being a safety slogan and becomes the same rule the data model already follows: one source per truth, and name it.

Cost as a designed number, not a surprise

One advance cycle touches the model exactly twice, to parse the reply and draft the response. So cost scales with shows advanced rather than chat length or agent wandering. That's the bounded graph paying out: no re-planning loops means no unbounded spend. The levers, in the order I'd pull them:

Which model should be making the call? I ran the experiment.

"Tier your models" is advice everyone repeats and nobody measures. So I built a bench, scripts/model_cost_bench.py. It scores the same 32 golden chains with the production prompt lifted straight out of the running code, across three tiers, recording accuracy, cost and latency in one pass.

The more useful finding is which case they all miss. It's the same one for every tier, and it isn't a hard extraction. It's an out-of-office auto-reply, and every model helpfully pulls the vacation forwarding address out as the venue contact. The case forbids exactly that.

# A negative control: the correct answer here is NOTHING.
"venue": "The 40 Watt",
"thread": [
  {"from": "bot",   "text": "Load-in time?"},
  {"from": "venue", "subject": "Automatic reply: Advancing",
   "text": "I'm out of office until Aug 20 with limited email access."
           "For urgent booking matters contact jamie@40watt.com."}
],
"expect": {},                          # no fields should come back
"forbid": ["loadIn", "contact"],       # and these two are traps
"expect_kind": "auto_reply"

# Every tier returns: {"contact": "jamie@40watt.com"}  ->  scores 0.5

def _deterministic_reply_kind(from_addr, subject, text):
    """Machine-signature replies, caught in CODE. No LLM, no ambiguity."""
    if ("mailer-daemon" in fa or "postmaster@" in fa
            or "undeliver" in sub or "550 5.1.1" in tx):
        return "bounce"
    if ("automatic reply" in sub or "out of office" in sub
            or "auto-submitted" in tx[:200]):
        return "auto_reply"
    return ""   # empty means: let the model classify it

The bench deliberately bypasses that layer to isolate the model, which is what makes it the cleanest evidence I have that the layer earns its place.

Then I ran the same test on the provider we actually pay

The bench above graded the failover side. The primary is OpenAI, and production sets EXTRACT_MODEL to gpt-4o because extraction touches real tour data and the expensive model felt like the safe choice. Same 42 golden chains, same production prompt, same scorer, both tiers:

That's a live config change worth making: extraction can move to the cheap tier and the eval gate is what makes the downgrade safe rather than a gamble. The full graphic, generated from the run data:

Open the comparison full-screen →

The model is wrong sometimes. That's a design input, not a bug report.

Reading a venue's email is the one place TM GO must use a model, because free-form human text has no parser. So the useful question was never "is the model accurate?" but which specific way does it fail, and what catches that one.

Every defense sits in one of three places, and the place matters more than the cleverness:

The prompt is an artifact of the test suite, not of taste

The usual way a prompt gets written is somebody tries wording until the demo looks good, and nobody can say later why any particular sentence is in there. This one has a paper trail. Every rule exists because a case failed without it, and the optimizer that proposes changes is scored by the same harness that gates the build.

The loop runs in four steps:

The one failing case was Lincoln Hall, where the venue asked a question back that happened to contain a plausible answer. The extractor treated the suggestion as a confirmation. Here is the entire fix the optimizer proposed, and it is one sentence added to rule 6:

  5. A deferred or conditional answer IS an answer...
- 6. A question BACK to us is NOT an answer ('how many in your party?'
-    answers nothing). Pleasantries and signatures answer nothing.
+ 6. A question BACK to us is NOT an answer ('how many in your party?'
+    answers nothing). Even if the question suggests a potential answer
+    ('Would 3pm work for load-in?'), it provides no confirmation.
+    Pleasantries and signatures answer nothing.
  7. Map colloquial times to the right field...

"Would 3pm work for load-in?" is a proposal, not a fact. Writing 3pm into a band's tour sheet because a venue floated it is exactly the class of error this system exists to prevent.

The full record, generated from the run history:

Open the full record →

Summary

TM Go is a tour-management agent for independent bands. It advances shows over email and SMS, drafts venue outreach, parses free-form replies into structured logistics, and keeps each tour's records in sync. The central engineering decision was constraining the agent: a hand-authored LangGraph StateGraph replaces an open ReAct loop, with the model confined to drafting and parsing while all state, routing and I/O run as deterministic code. It ships with production-reliability plumbing, including provider failover, tolerant parsing, validated inputs and human-in-the-loop review, plus a traced_llm observability layer that doubles as a closed-loop evaluation harness. Every human correction becomes labeled data, and a CI gate blocks any change that reduces extraction accuracy.

🤖 Generated with AI, this one. I'll own it. The day one of these is my own typing, you won't need the disclaimer to tell. Fewer em-dashes, more typos. Trust me, you'll know.