For every email in the archive, using only what was observable when it arrived, how well could the company (campus) of the resulting QuickBooks bill have been predicted? Rules of the experiment: docs/backtest-plan.md. Train = July–August, test = September; scores below are on the September messages that became bills.
Where the linked invoices came from: employee forward (509), vendor platform (266), vendor direct (216), employee thread (120), vendor reply (80), system (8)
Most invoices do not arrive from the vendor: an employee forwards them. "New invoice" = the bill did not yet exist in QuickBooks when the email arrived; "follow-up" = it did.
Each row is a predictor; each cell is accuracy on the rows it answered, with coverage (share of rows it answered) underneath. A predictor abstains when it has nothing to say — a router needs both numbers.
| predictor | all n=454 | new invoice n=202 | follow-up n=252 | employee forward n=185 | vendor direct n=70 | vendor platform n=127 | employee thread n=34 | vendor reply n=34 |
|---|---|---|---|---|---|---|---|---|
| majority | 42% 100% | 24% 100% | 56% 100% | 40% 100% | 33% 100% | 54% 100% | 35% 100% | 38% 100% |
| history | 65% 86% | 52% 80% | 74% 91% | 60% 91% | 74% 93% | 60% 100% | — | 96% 79% |
| vendor_history | 78% 25% | 78% 33% | 79% 19% | 82% 27% | 74% 39% | 75% 25% | 50% 6% | 100% 6% |
| thread | 96% 54% | 95% 48% | 97% 59% | 99% 93% | 44% 13% | 87% 18% | 100% 47% | 100% 79% |
| rules | 88% 36% | 92% 46% | 83% 27% | 87% 32% | 86% 40% | 100% 41% | 83% 18% | 54% 38% |
| thread+rules+history | 87% 96% | 84% 96% | 89% 97% | 96% 100% | 78% 97% | 75% 100% | 95% 59% | 97% 100% |
| jev_v3 | 45% 52% | 66% 52% | 28% 52% | 37% 51% | 67% 56% | 43% 66% | 43% 21% | 56% 26% |
| thread+rules+jev+history | 82% 98% | 83% 97% | 81% 98% | 96% 100% | 69% 100% | 62% 100% | 87% 68% | 97% 100% |
| ml_logreg@0.7 | 94% 85% | 92% 88% | 97% 82% | 98% 90% | 98% 84% | 86% 80% | 88% 74% | 100% 79% |
| ml_hgb@0.7 | 96% 93% | 94% 94% | 97% 93% | 99% 95% | 94% 94% | 94% 91% | 80% 88% | 100% 94% |
| thread+rules+hgb+history | 91% 99% | 89% 100% | 93% 99% | 98% 100% | 84% 100% | 87% 100% | 84% 91% | 97% 100% |
How to read it: thread = a bill was already linked earlier in the same email thread; rules = a Bill-To ZIP, any single ZIP, or a campus name in the text (including PDF text) names exactly one active company; history = the sender domain's most common company among bills entered before the message; vendor_history = the same keyed on a vendor-name guess; jev_v3 = the Jev campus classifier, counted only when confident (≥0.7) and specific; ml_logreg / ml_hgb = logistic regression (text + tabular) and gradient boosting (tabular only) trained on the July–August rows, counted when their probability is ≥ 0.7. Cascades try each in order and take the first answer.
Trained on the July–August linked messages, scored on September. Both abstain below a probability threshold.
| model | argmax accuracy | ≥0.5: accuracy / coverage | ≥0.7: accuracy / coverage | ≥0.9: accuracy / coverage |
|---|---|---|---|---|
| logreg | 89% | 92% / 95% | 95% / 85% | 99% / 63% |
| hgb | 92% | 94% / 99% | 96% / 93% | 98% / 85% |
Logistic regression, share of coefficient weight by feature group: text 87%, categorical 11%, numeric/bool 2%
Gradient boosting, permutation importance on the test rows (drop in accuracy when the feature is shuffled):
| feature | importance |
|---|---|
| f_thread_company_known_Alpha School 33155, LLC | 0.065 |
| f_zip_companies_Alpha School 33155, LLC | 0.044 |
| f_company_names_in_text_Alpha School 92610, LLC (Orange County) | 0.039 |
| f_thread_company_known_Alpha School 78701, LLC (Alpha High School) | 0.035 |
| f_zip_companies_Sports Academy 78734, LLC (TSA) | 0.034 |
| f_thread_company_known_Sports Academy 75010, LLC (CSA) | 0.026 |
| f_zip_companies_Alpha School 75010, LLC (Alpha Carrolton)|Sports Academy 75010, LLC (CSA) | 0.026 |
| f_n_amounts | 0.021 |
| f_subject_len | 0.015 |
| h_original_domain_top_company_Alpha School 33155, LLC | 0.012 |
| f_thread_company_known_Sports Academy 78734, LLC (TSA) | 0.009 |
| f_thread_company_known_Alpha School 92610, LLC (Orange County) | 0.009 |
| h_domain_prior_bills | 0.008 |
| f_zip_companies_Alpha School 78701, LLC (Alpha High School) | 0.007 |
| f_zip_companies_Alpha School 92610, LLC (Orange County) | 0.006 |
Alpha School 33155, LLC: f_zip_companies_Alpha School 33155, LLC +2.99, f_thread_company_known_Alpha School 33155, LLC +1.67, h_vendor_top_company_Alpha School 33155, LLC +1.26, h_original_domain_top_company_Alpha School 33155, LLC +0.92, f_original_domain_nmanagementgroup.com +0.84, f_company_names_in_text_ +0.81, f_vendor_key_daniel marquez +0.78, f_vendor_key_nicolas management +0.77, h_domain_top_company_Sports Academy 75010, LLC (CSA) +0.75, f_cue_quote +0.54
Alpha School 78521, LLC (Brownsville): f_company_names_in_text_Alpha School 78521, LLC (Brownsville) +1.65, f_zip_companies_Alpha School 78521, LLC (Brownsville) +1.65, f_thread_company_known_Alpha School 78521, LLC (Brownsville) +1.19, h_original_domain_top_company_Alpha School 78521, LLC (Brownsville) +0.94, f_original_domain_sbcglobal.net +0.93, f_original_domain_alpha.school +0.92, h_original_domain_top_company_Sports Academy 75010, LLC (CSA) +0.77, f_zip_companies_ +0.76, f_vendor_key_account servicing +0.71, f_sender_domain_billerpayments.com +0.71
Alpha School 78701, LLC (Alpha High School): f_thread_company_known_Alpha School 78701, LLC (Alpha High School) +2.17, f_zip_companies_Alpha School 78701, LLC (Alpha High School) +2.02, f_company_names_in_text_Alpha School 78701, LLC (Alpha High School) +1.50, f_original_domain_notification.intuit.com +1.38, h_original_domain_top_company_Alpha School 78701, LLC (Alpha High School) +1.18, f_vendor_key_ivy league roadmap +1.04, h_domain_top_company_Alpha School 78701, LLC (Alpha High School) +0.98, f_cue_po +0.84, f_vendor_key_languagebird +0.73, f_vendor_key_husch blackwell +0.71
Alpha School 92610, LLC (Orange County): f_company_names_in_text_Alpha School 92610, LLC (Orange County) +2.40, f_thread_company_known_Alpha School 92610, LLC (Orange County) +2.00, f_zip_companies_Alpha School 92610, LLC (Orange County) +1.84, f_original_domain_best-vip.com +1.63, h_original_domain_top_company_Alpha School 92610, LLC (Orange County) +0.85, f_sender_domain_best-vip.com +0.72, f_original_domain_gmail.com +0.67, f_vendor_key_best vip +0.63, f_vendor_key_school eatery +0.61, h_vendor_n_companies +0.57
Sports Academy 75010, LLC (CSA): f_zip_companies_Alpha School 75010, LLC (Alpha Carrolton)|Sports Academy 75010, LLC (CSA) +2.40, f_thread_company_known_Sports Academy 75010, LLC (CSA) +2.01, f_original_domain_yahoo.com +1.45, f_billto_zip_companies_Alpha School 75010, LLC (Alpha Carrolton)|Sports Academy 75010, LLC (CSA) +1.06, f_vendor_key_christina barron +0.74, f_vendor_key_ +0.73, h_original_domain_top_company_Sports Academy 75010, LLC (CSA) +0.71, f_original_domain_clearvisionroofing.com +0.70, f_company_names_in_text_Sports Academy 78734, LLC (TSA) +0.69, f_vendor_key_terry henry +0.66
Sports Academy 78734, LLC (TSA): f_zip_companies_Sports Academy 78734, LLC (TSA) +2.81, h_original_domain_top_company_Sports Academy 78734, LLC (TSA) +1.35, f_thread_company_known_Sports Academy 78734, LLC (TSA) +1.34, f_vendor_key_watchmen protective +1.20, f_vendor_key_accounts receivable +1.08, h_domain_top_company_Sports Academy 78734, LLC (TSA) +1.08, h_vendor_top_company_Alpha School 78701, LLC (Alpha High School) +0.96, f_original_domain_fastsigns.com +0.94, f_zip_companies_infrequent_sklearn +0.92, f_company_names_in_text_Sports Academy 78734, LLC (TSA) +0.81