TM Go

System overview

The real deployed system, with prototype shortcuts named as such rather than hidden.

Three state machines, kept orthogonal on purpose

The third machine is the subtle one. What the inbox renders (Waiting, Follow-up due, Reply received, Done) isn't advancing progress. It's conversation state, computed from the messages and the clock. That second input makes it different in kind, because it changes with no write at all. A venue goes from "waiting" to "follow-up due" three days after a send. Nothing happened: no reply, no click, no event. Time passed.

The escape hatch, stated plainly: a user can still set advancing status by hand. Four options, not the six the system computes, because showing every internal state read as overlapping choices. A pure derivation with no override is elegant, and it loses to reality the first time a venue confirms something over the phone.

Reliability as a design requirement

Each of these exists because a specific failure would place an incorrect fact in front of a band on tour. They are requirements, not polish.

How the email path is actually tested

Email is where this product lives, and it's the layer with the most ways to fail quietly. Nobody files a bug for replies that never arrive. The repo runs 378 tests across 18 suites, and the conversational and document path holds the largest share — 77 on the chat brain alone, plus grounding, the watch loop, inbox tagging and the gateway. Most are about telling apart states that look alike.

IMAP IDLE is the clearest example. The protocol is full of lookalikes, so each distinction gets pinned:

def test_exists_means_new_mail():
    conn = FakeConn([b"+ idling", b"* 7 EXISTS", b"A001 OK IDLE done"])
    assert ew.idle_once(conn, 1) is True

def test_quiet_timeout_is_not_mail():        # silence != a reply
    conn = FakeConn([b"+ idling", "TIMEOUT", b"A001 OK IDLE done"])
    assert ew.idle_once(conn, 1) is False

def test_keepalives_ignored_until_exists():   # EXPUNGE/RECENT are noise
    conn = FakeConn([b"+ idling", b"* 3 EXPUNGE", b"* 2 RECENT",
                     b"* 9 EXISTS", b"A001 OK"])
    assert ew.idle_once(conn, 1) is True

def test_idle_refused_raises_unsupported():   # fail loud, not idle forever
    conn = FakeConn([b"A001 NO IDLE not allowed"])
    assert raises(ew.IdleUnsupported)

The rest of that suite covers a server BYE raising a connection error, backoff that doubles and caps, and the one I care most about: an exception in a handler never kills the watch loop. One bad message must not silently end email ingestion.

Migrating off the spreadsheet without a big-bang cutover

The hard part isn't Postgres. It's moving while the thing is running. A band on tour can't have a maintenance window. So the migration runs as a strangler fig, gated by a measurement, in five reversible steps.

The access boundary, and why signing in was never one

Every table in the app is keyed by a tour id, and until this week nothing anywhere recorded who owned it. Signing in proved who you were. It never proved the tour was yours.

Only the first gate existed — and for sheet-backed tours the id is the spreadsheet id, which travels in a URL. It was never a secret to begin with.

This was the last structural thing that had to land before real users: after the first paying customer, every migration here is a migration of somebody's live data. Three decisions in it matter more than the schema.

The invariant: you can't add a route that skips the boundary

Wrapping at registration only holds if something fails when one slips through. So the suite walks the real route list rather than trusting the convention.

def test_every_registered_route_is_wrapped():
    """A check you must remember to call WILL be missed on the next endpoint."""
    unwrapped = []
    for node in ast.walk(tree):              # the real Route(...) list
        if not _is_api_route(node):
            continue
        handler = node.args[1]
        if not _wrapped_in(handler, "guard_tour"):
            unwrapped.append(node.args[0].value)
    assert not unwrapped, (
        "these routes bypass the tour boundary — wrap them: " + ", ".join(unwrapped))

The other 20 are the bypass attempts and the things that must not break: the key hidden in a POST body, the full spreadsheet URL, the tour_key alias, an anonymous caller, a second member who should get through, a claimed tour locking everyone else out afterwards, and the body surviving the check.

Seven model seams, none of them able to write

This section used to say "exactly two." It hasn't been two since document intake and chat landed, and the honest count is more convincing than the tidy one — because the rule it was protecting never changed. The model appears wherever there is language to read or write, and nowhere near a fact.

Routing between lanes, scheduling, every write, validation, retries and the entire state engine remain deterministic Python. That placement is the load-bearing decision the other eight hang off, and it survived the seam count more than tripling — which is the actual test of whether a rule was structural or just a small system's convenience.

The shape underneath all of them is a decision about where a truth is allowed to come from. The model proposes language, never facts. A projection is computed in one place. Booking status comes from a human because nothing else knows it. Cutover comes from a parity number rather than confidence. Put that way, "constrain the agent" stops being a safety slogan and becomes the same rule the data model already follows: one source per truth, and name it.

Cost as a designed number, not a surprise

One advance cycle still touches the model exactly twice — parse the reply, draft the response — so that surface scales with shows advanced rather than agent wandering. That's the bounded graph paying out: no re-planning loops means no unbounded spend. But it is no longer the only surface, and saying so is the point of metering:

The levers, in the order I'd pull them:

Which model should be making the call? I ran the experiment.

"Tier your models" is advice everyone repeats and nobody measures. So I built a bench, scripts/model_cost_bench.py. It scores the golden chains with the production prompt lifted straight out of the running code, across three tiers, recording accuracy, cost and latency in one pass. Each bench below is a dated snapshot — this one ran at 32 chains and the OpenAI run further down at 42, because the set has kept growing. What matters is that within a run, the only variable is the model.

The more useful finding is which case they all miss. It's the same one for every tier, and it isn't a hard extraction. It's an out-of-office auto-reply, and every model helpfully pulls the vacation forwarding address out as the venue contact. The case forbids exactly that.

# A negative control: the correct answer here is NOTHING.
"venue": "The 40 Watt",
"thread": [
  {"from": "bot",   "text": "Load-in time?"},
  {"from": "venue", "subject": "Automatic reply: Advancing",
   "text": "I'm out of office until Aug 20 with limited email access."
           "For urgent booking matters contact jamie@40watt.com."}
],
"expect": {},                          # no fields should come back
"forbid": ["loadIn", "contact"],       # and these two are traps
"expect_kind": "auto_reply"

# Every tier returns: {"contact": "jamie@40watt.com"}  ->  scores 0.5

def _deterministic_reply_kind(from_addr, subject, text):
    """Machine-signature replies, caught in CODE. No LLM, no ambiguity."""
    if ("mailer-daemon" in fa or "postmaster@" in fa
            or "undeliver" in sub or "550 5.1.1" in tx):
        return "bounce"
    if ("automatic reply" in sub or "out of office" in sub
            or "auto-submitted" in tx[:200]):
        return "auto_reply"
    return ""   # empty means: let the model classify it

The bench deliberately bypasses that layer to isolate the model, which is what makes it the cleanest evidence I have that the layer earns its place.

Then I ran the same test on the provider we actually pay

The bench above graded the failover side. The primary is OpenAI, and production sets EXTRACT_MODEL to gpt-4o because extraction touches real tour data and the expensive model felt like the safe choice. Same 42 golden chains, same production prompt, same scorer, both tiers:

That's a live config change worth making: extraction can move to the cheap tier and the eval gate is what makes the downgrade safe rather than a gamble. The full graphic, generated from the run data:

Open the comparison full-screen →

The model is wrong sometimes. That's a design input, not a bug report.

Reading a venue's email is the one place TM GO must use a model, because free-form human text has no parser. So the useful question was never "is the model accurate?" but which specific way does it fail, and what catches that one.

Every defense sits in one of three places, and the place matters more than the cleverness:

The scaling path, no longer just a plan

This section used to say "presented as a roadmap, not a current claim." Most of it has since shipped, behind flags, fail-open, with the migration gated by measurement. The statuses below are per target and honestly marked.