Rendered from docs/backtest-plan.md at build time.
Agreed 2026-09-27. Status is tracked in the checkboxes; findings go at the bottom.
For every email that reached billing@alpha.school in the window, using only what a production system could have seen at the moment it arrived, how well could we have predicted what the AP team and QuickBooks eventually recorded — and which observable factors carry the signal?
[Gmail]/All Mail (not just INBOX),
~13k messages. Extend backwards once the pipeline works; the archive goes
to Jul 2024 and QuickBooks bills to Jul 2024 as well.mail_archive/ dataset
in scripts/data_sync.py), never git. The 100-message INBOX export
already in git stays as the site's demo slice.A feature is admissible only if it was observable when the message arrived. Everything else is an outcome.
| Admissible (features) | Outcome (ground truth) |
|---|---|
Headers: From, To/Cc (billing@ in To vs only Cc), Reply-To, Date, In-Reply-To, References, thread id |
Gmail labels the AP team applied later: Vendor/<name>, entity labels (Alpha 33155, …), state labels (Approved, Posted, …) |
| Subject, body, attachment names/types/sizes, attachment text (pdftotext / OCR) | The QuickBooks bill: company (= campus), class, account(s), vendor, amount, DocNumber, date entered |
Origin category (src/mail_origin.py), extracted invoice numbers, amounts, ZIPs, campus names |
Whether any bill was created (invoice vs. not-AP traffic) |
| QuickBooks history as of the arrival date: this vendor's prior bills (which companies, which accounts, typical amounts), whether the vendor exists in each company's vendor list, prior messages in the same thread | Time-to-post, rejections, cancellations (later) |
Two consequences worth stating plainly:
DocNumber lookup is only legitimate for follow-ups. For a
first-arrival invoice the bill does not exist yet. Every message gets a
flag, bill_existed_at_arrival, from the bill's MetaData.CreateTime.
Results are reported separately for new invoice (the hard, valuable
case) and follow-up on a known bill (mostly solvable by lookup).Tuition and other Fees Queries, Trilogy Query, Newsletter/*,
Account Access, Reports, … and anything never labelled AP.not_campus_specific. Truth: the
bill's company; the entity label as a cross-check.VendorRef and the
Vendor/<name> label.AccountRef.scripts/build_backtest.py)References /
X-GM-THRID, dated before this one) is a feature, not the unit.scripts/qbo_match_mail.py's signals (invoice number, attachment
hash/name, vendor+amount), pooled over all realms. Multi-bill messages
carry multiple labels; the docnum-only rule needs ≥5 characters or a
vendor token (the one false-positive pattern seen so far).TxnDate before the message.bill_existed_at_arrival, every truth column,
and the split. CSV + parquet, in S3 (datasets/backtest/).not_campus_specific; most common account).records/qbo_discovery/*/vendors.csv, restricted to bills
before the message).src/classifier.py, TF-IDF + logistic) on
(vendor, description, amount, entity) → account, trained on the
pre-split bills.The cascade reached 85% with three free signals; the remaining 15% is a feature-design problem. Three tiers, each admissible at arrival:
Tier 1 — structural (rules, exact, free). Sender domain and whether it is internal / an invoicing platform; origin category; forwarded-original sender; billing@ in To vs Cc; reply/forward; attachment count, types, PDF present; position in thread; weekday/hour.
Tier 2 — extracted from text (regex now, LLM later). Invoice numbers;
amounts; ZIPs and which company each maps to; campus / entity names and
their distinctive tokens; where in the text they occur — a ZIP in a
"Bill To" / "Ship To" block is the campus, a ZIP in a signature or
letterhead is the vendor (this is the "vendor address mistaken for
campus" failure); service-location phrases ("at the Miami campus", "for
Brownsville"); "PO", "quote", "estimate", "statement", "past due" cues
that separate invoices from other paper. An LLM extractor makes these
robust: one structured call per message returning {is_invoice,
vendor_name, invoice_number, invoice_date, amount, bill_to_name,
bill_to_address, bill_to_zip, service_location, service_description,
document_type}. Cached per message id, so it costs once. Needs an
Anthropic API key (the local gateway fronts the Claude Code subscription,
which is not for bulk API work).
Tier 3 — history as of arrival (the strongest, and only a production
system with QuickBooks access has it). For the sender domain, the
forwarded-original domain, and the extracted vendor name: distribution of
companies and accounts on earlier bills; whether the vendor exists in each
company's vendor list; typical amount; days since last bill. For the
thread: company already established. For the extracted invoice number:
whether a bill with that DocNumber already exists (follow-ups only).
Interactions worth a model rather than a cascade: a domain that serves several campuses plus a ZIP; a platform sender plus the vendor name in the subject; an internal forwarder plus the original vendor's history.
Jev returns a label and a confidence and nothing else. A traditional model on the same admissible features — TF-IDF over subject/body/attachment text, one-hot sender domain and origin category, the as-of vendor→company and vendor→account priors, amount — gives, per prediction, which features drove it (logistic-regression coefficients, or per-row SHAP on a gradient-boosted model). That is the only route to answering "what are the helpful predictors" directly rather than by ablation. Worth doing once the backtest table exists, because both models then score on exactly the same rows: same targets, same split, same metrics. Expected shape of the result: Jev wins on messy text (forwards, OCR'd invoices), the explainable model wins wherever the history prior decides it — and the explanation tells us which cases to route to which.
--since/--before, [Gmail]/All Mail, X-GM-LABELS,
X-GM-THRID, X-GM-MSGID, In-Reply-To, References, resume,
reconnect. Still BODY.PEEK + read-only select. (2026-09-27; also
fixed: mailbox names with spaces were never quoted, so
[Gmail]/All Mail never worked before.)mail_archive/: 13,247 messages,
5,010 threads, 5,874 with attachments, 2.6 GB (2026-09-27).docs/qbo-investigation.md.qbo_match_mail.py rewritten around indexes and run over
mail_archive/ × all realms: 1,199 messages ↔ 451 bills on strong
signals; 83% of bills dated in the window have a linked email;
592 new-invoice messages ↔ 300 bills; 99 messages touch more than one
company.build_backtest.py → records/datasets/backtest.csv: 13,247 rows,
8,139 train (Jul–Aug) / 5,108 test (Sep); 1,199 linked (592 new /
607 follow-up).backtest_baselines.py, no model calls) and Jev v3
(4) on the same 454 linked test messages — see Findings.backtest_ml.py: logistic regression + gradient boosting on the
admissible features; scored alongside the cascade and Jev (E9 in
docs/experiments.md).454 linked test messages (202 new-invoice, 252 follow-up).
| baseline | coverage | accuracy | new-invoice acc. | follow-up acc. |
|---|---|---|---|---|
| majority (Miami) | 100% | 42% | 24% | 56% |
| sender-domain history | 86% | 65% | 52% | 74% |
| thread already linked | 54% | 96% | 95% | 97% |
| ZIP / campus name in text | 18% | 80% | 90% | 72% |
| thread → rules → history | 95% | 85% | 80% | 89% |
By origin, the combined baseline: employee forward 94% (n=185, 100% coverage), vendor reply 97%, vendor platform 74%, vendor direct 73%, employee thread 100% at 53% coverage.
Reading: a thread that has already produced a bill almost never changes company, so the thread is the best single feature; the sender's domain is weak on its own because vendors serve several campuses; explicit ZIPs and campus names are precise but rare. The three together, with no model, route 95% of linked messages at 85% accuracy — that is the bar Jev has to clear on the same rows.
classify_locations_jev.py run over exactly the linked September messages
(records/datasets/backtest_jev/v3_test.csv; the run record moved out of
classification_runs/ so the site's "latest run" stays the INBOX slice).
Scored with the same rules: an answer only when confidence ≥ 0.7 and the
label is a specific company.
| coverage | accuracy | new-invoice | follow-up | employee fwd | vendor direct | vendor platform | |
|---|---|---|---|---|---|---|---|
| thread → rules → history | 95% | 85% | 80% | 89% | 94% | 73% | 74% |
| Jev v3 (confident, specific) | 52% | 45% | 66% | 28% | 37% | 67% | 43% |
| thread → rules → Jev → history | 97% | 80% | 81% | 80% | 94% | 67% | 62% |
Jev's confident errors, by shape:
not_campus_specific at ≥ 0.7: 133 of 368 confident answers. Every
one of those messages became a bill in a specific company. On an
invoice, "not campus specific" is never the operational answer — a bill
is always booked to exactly one company — so the label should be
removed from the choice set for anything that looks like a bill, or
treated as an abstention.Caveat: the truth set only contains messages that became bills in the seven authorized companies, so Jev's "not specific" answers can't be credited here even when a message genuinely had no campus — but for the target population (invoices), that is the right test.
Consequences for the next classifier version, in order of expected gain:
1. Thread and history cascade first; call the model only for what's left.
2. Choice set = companies with bills (plus a per-company vendor list as a
prior); drop not_campus_specific for bill-like messages.
3. Strip vendor letterhead/footer addresses before the model sees the
text, or pass the vendor's known HQ so it can discount it.
4. Then the explainable-model comparison on the same rows.