Rendered from docs/experiments.md at build time.
One entry per experiment, in the order they were run. Each records the question, the design, the data it ran on, the result, and what we decided because of it. Numbers are as measured at the time; the Datasets and Backtest pages show the current state.
Supporting documents: the backtest plan (the rules of the experiment), the QuickBooks investigation (what's in QuickBooks and how we reached it), and the campus classifier changelog.
Question. Given an email to billing@alpha.school, which campus does the expense belong to?
Design. A hosted Choice model (Jev, via TypeSafe) reads the subject,
sender, body and the text of PDF attachments (pdftotext, falling back to
OCR) and picks one label. v1 chose from the 47 campuses scraped from
alpha.school; v2 switched the taxonomy to the 73 QuickBooks companies —
the legal entities a bill can actually be posted to — enriched with street
addresses. Threshold 0.7 for "confident".
Data. 100 INBOX messages, 2026-09-23 → 25.
Result. v2: 36 confident specific campuses, 51 not_campus_specific,
19 uncertain; mean confidence 0.85. No ground truth yet, so accuracy was
unknown. One visible miss: a Sports Academy 75010 invoice labelled Alpha
Carrolton (shared ZIP).
Decision. Keep v2 as the taxonomy. Get ground truth before iterating on prompts.
Question. Can an email be tied to the bill it became, and does that give both labels (campus and account) at once?
Design. Read-only QuickBooks access to Alpha School 33155 (Miami). For
every bill since July, four signals against each message: attachment
hash, attachment filename, the vendor's invoice number (DocNumber) in
the text, vendor name plus exact amount.
Data. Same 100 messages × 95 Miami bills.
Result. 9 messages ↔ 11 bills, all on strong signals. Invoice number
is the workhorse; byte-identical attachments are rare (2). Every match
yields the bill's account and class. Of the 9 matched messages, v2 had
labelled 5 not_campus_specific — all forwards or reply threads. Only 5
of 95 bills were entered during the two-day mail window, so the low
bill coverage was timing, not method.
Decision. Invoice number → bill → company is a deterministic feature, not just evaluation data. Pull far more mail.
Question. Did the vendor send this to billing@, or did an employee forward it — and should a model decide that?
Design. Rules over headers and body (src/mail_origin.py): sender
domain internal / invoicing platform / vendor / system; forwarded block
present; the original sender inside the forward (first external
From:); billing@ in To or only Cc. Rules, not Jev, because the facts are
structural and the archive is ~80k messages.
Data. The 100 messages; later the 13k-message archive.
Result. On the 100: vendor direct 19, via platform 17, vendor reply 19, employee forward 18, employee thread 18, system 9; billing@ only Cc'd on 25. On the 1,199 archived messages that became bills: employee forward 42%, platform 22%, vendor direct 18%, employee thread 10%, vendor reply 7%.
Decision. Origin is a feature on every page and in every model input. Most invoices do not arrive from the vendor.
Question. Does telling the model how the message arrived (and who the original sender was) fix the forward/thread misses from E2?
Design. v2 plus an Origin block prepended to the input. Taxonomy, question, model, threshold unchanged, so the diff isolates one change.
Data. The 100 messages; scored on the 8 QuickBooks-linked ones.
Result. No improvement: 99/100 labels unchanged; confidence mostly fell (21 uncertain vs 19). Linked messages: v2 4 right / 2 wrong / 2 uncertain → v3 3 / 2 / 3. The two confident-wrong cases were identical — an employee's one-line reply to a reminder where the only campus evidence is the invoice number. The one new confident answer was the 75010 collision, now confidently wrong.
Decision. The misses are missing evidence, not misread evidence. Stop changing prompts; add deterministic pre-passes.
Question. Why weren't the vendor emails behind recent bills in INBOX?
Design. Read-only census of every Gmail label: count, unread, date
range (scripts/mailbox_census.py).
Data. The whole mailbox.
Result. 1,460 labels, 79,223 distinct messages back to Aug 2020. The
labels encode processing state (Awaiting Approval → Approved →
*For Payment → Posted), entity (Alpha 33155, …) and vendor
(Vendor/<name>, ~450 of them, 60k label-messages). INBOX is a slice;
processed invoices are filed.
Decision. Export the labelled archive, not a longer INBOX window. The labels are human ground truth for vendor, entity and became-a-bill.
Question. Do the 73 companies share a chart of accounts and class list, i.e. is Phase 1 one classifier or 73?
Design. Authorize six more companies (Brownsville, Orange County, Alpha High School, Sports Academy 78734 and 75010, Alpha Carrolton; Austin K-8 excluded as the atypical flagship); dump accounts, classes, vendors and bill summaries for each; diff.
Data. 7 companies, 2,130 bills / 2,606 lines.
Result. The chart of accounts is one template: 54 numbered accounts
identical across all seven; the rest are per-company bank/clearing
accounts and typos. Classes are per-company sub-campuses (Brownsville
K-8, Nova High School - Brownsville), not a global campus list. Every
bill line is account-based; 99.9% carry a class. Alpha Carrolton has no
bills at all.
Decision. One account classifier keyed on account number. Class is a per-company lookup. A company with no bills should not be a choice.
Question. Using only what a production system could see when an email arrived, how well could the bill's company have been predicted, and which observable factors carry the signal?
Design. The backtest plan. Features must be
observable at arrival (headers, origin, text, extracted numbers, thread
history, and QuickBooks history as of that date); Gmail labels and the
bill itself are outcomes. bill_existed_at_arrival splits new invoices
from follow-ups. Train Jul–Aug, test Sep. Predictors abstain when they
have nothing to say, so each is scored on coverage and accuracy.
Data. 13,247 archived messages (Jul → Sep) × 2,130 bills in 7 companies. 1,199 messages linked to 451 bills; 83% of in-window bills found their email; 592 first arrivals. Test: 454 linked September messages (202 new, 252 follow-up).
Result.
| predictor | coverage | accuracy | new invoice | follow-up |
|---|---|---|---|---|
| majority company | 100% | 42% | 24% | 56% |
| sender-domain history | 86% | 65% | 52% | 74% |
| thread already linked to a bill | 54% | 96% | 95% | 97% |
| ZIP / campus name in text | 18% | 80% | 90% | 72% |
| thread → rules → history | 95% | 85% | 80% | 89% |
By origin, the cascade scores 94% on employee forwards (the largest group), 97% on vendor replies, 73–74% on vendor-direct and platform mail.
Decision. Thread context is the best single feature; history alone is weak because vendors serve several campuses. The free cascade is the bar any model must clear on the same rows.
Question. Same rows, same rules: does the campus classifier beat the free cascade?
Design. v3 run over exactly the 454 linked test messages; an answer counts only when confidence ≥ 0.7 and the label is a specific company. Also a hybrid: thread → rules → Jev → history.
Result. Jev answers 52% of rows at 45% accuracy (66% on new
invoices, 28% on follow-ups). In the hybrid it lowers accuracy, 85% →
80%. Error shape: 133 of 368 confident answers were not_campus_specific
on messages that were bills in a specific company; 19 Sports Academy
75010 bills → Alpha Carrolton (no bills); 24 Miami bills → Santa Monica /
San Francisco (vendor letterheads).
Decision. Next classifier version constrains the choice set to
companies with bills, drops not_campus_specific for bill-like messages,
and strips vendor letterhead addresses — deterministic changes, then
re-score. Then the explainable-model comparison (E9).
Question. Can an explainable model on the admissible features beat the cascade, and which features does it use?
Design. Three feature tiers (see the plan): structural (origin, sender,
thread position, billing@ role), extracted (invoice numbers, amounts,
ZIPs — with Bill-To ZIPs separated from letterhead ZIPs — campus names,
document-type cues, a vendor-name guess from subject/sender; all now
read from the PDF text layer too), and as-of history (sender-domain,
forwarded-original-domain and vendor-name priors from earlier bills;
thread company). Two models on the same 745/454 split, abstaining below a
probability threshold: multinomial logistic regression on TF-IDF text +
one-hot categoricals + numerics (scripts/backtest_ml.py), and gradient
boosting on the tabular features only (no text).
Result. Adding PDF text to the rules lifted the cascade to 87% at 96.5% coverage first. Then:
| predictor | coverage | accuracy | new invoice | vendor platform | employee fwd |
|---|---|---|---|---|---|
| thread → rules → history | 96% | 87% | 84% | 75% | 96% |
| Jev v3 (≥0.7) | 52% | 45% | 66% | 43% | 37% |
| logistic regression (≥0.7) | 85% | 95% | 92% | 86% | 98% |
| gradient boosting, tabular (≥0.7) | 93% | 96% | 94% | 94% | 99% |
| gradient boosting (≥0.9) | 85% | 98% | |||
| thread → rules → GB → history | 99% | 91% | 89% | 87% | 98% |
The model fixes the cascade's weak spot: platform mail (QuickBooks /
Stripe notifications), where the sender domain carries no history.
Explanation: gradient boosting's top permutation importances are
f_thread_company_known_Alpha School 33155, LLC (0.0652), f_zip_companies_Alpha School 33155, LLC (0.0436), f_company_names_in_text_Alpha School 92610, LLC (Orange County) (0.0392), f_thread_company_known_Alpha School 78701, LLC (Alpha High School) (0.0348), f_zip_companies_Sports Academy 78734, LLC (TSA) (0.0335), f_thread_company_known_Sports Academy 75010, LLC (CSA) (0.0264); logistic
regression puts 87% of its weight on text, 11% on the
categorical (history / ZIP / thread) features.
Caveats. Six classes only (the companies with bills in the window); a production model must choose among all active companies and will need the per-company vendor lists as features. 745 training rows. The truth set is messages that became bills in seven companies.
Decision. The production shape is a cascade: thread → Bill-To ZIP → model → history, with the model abstaining below 0.9 and those rows going to a human. Jev is not in the path for campus. Next: the same for account; an LLM extractor for the Tier-2 fields (needs an API key); refit on more companies.