Compare runs

4 runs of classify_locations_jev.py, joined per message. What changed in each version: classification_runs/CHANGELOG.md.

Runs

runmessageschoicesmean conf.< 0.7specific campusnot campus specific
v1 20260924T133942100490.871833 (33%)53 (53%)
v1 20260924T151708100490.861932 (32%)51 (51%)
v2 20260925T121304100750.851936 (36%)51 (51%)
v3 20260927T114857100750.842133 (33%)52 (52%)

Agreement between consecutive runs

from → tosame labelchangedbecame specificbecame not-specificuncertain → confidentconfident → uncertain
v1 → v192 (92%)84212
v1 → v251 (51%)49741010
v2 → v399 (99%)10146

v1 used a different label set (alpha.school campus names), so v1 → v2 changes are mostly renames.

Scorecard: messages tied to a QuickBooks company

19 messages were joined to bills in QuickBooks by invoice number / attachment (see Datasets). The bill's company is the ground truth for the campus label. A run is right if its label is that company, uncertain if below 0.7, otherwise wrong.

runrightuncertainwrongaccuracy (confident only)
v1 runs used alpha.school campus names as labels, not QuickBooks companies, so they score 0 here by construction.
v1 20260924T13394203160%
v1 20260924T15170804150%
v2 20260925T12130456838%
v3 20260927T11485737925%

Per message

messageoriginv1v1v2v3QuickBooks says
Re: Invoice #2177494employee forwardnot_campus_specific 98%not_campus_specific 98%not_campus_specific 92%not_campus_specific 94%Alpha School 92610, LLC (Orange County) (1 bill)
Open Invoices (Skyline Vision Security Alliance)vendor directnot_campus_specific 85%not_campus_specific 83%not_campus_specific 79%not_campus_specific 83%—
RE: Hamptons Summer Swag Invoices from Creative MC: 262547F+262538F+262663F+262670F+26268…vendor replyother_alpha_campus 40%not_campus_specific 44%not_campus_specific 77%not_campus_specific 50%—
Fwd: Open Invoices (Skyline Vision Security Alliance)vendor directnot_campus_specific 76%not_campus_specific 73%not_campus_specific 75%not_campus_specific 80%Alpha School 33155, LLC (1 bill)
Invoice #82121 for Sports Academy 75010 LLC - NCA from TechScapevendor directalpha_carrollton 91%alpha_carrollton 93%alpha_school_75010_llc_alpha_carrolton 54%alpha_school_75010_llc_alpha_carrolton 71%Sports Academy 75010, LLC (CSA) (1 bill)
This is your invoice 1419061 from Valley Alarmvendor directalpha_malibu 100%alpha_malibu 100%alpha_school_90265_llc_alpha_malibu 100%alpha_school_90265_llc_alpha_malibu 100%—
RE: VALLEY ALARM: Appt. On 7/20/26 at 11amvendor replynot_campus_specific 99%not_campus_specific 99%not_campus_specific 99%not_campus_specific 99%—
Re: Fwd: Alpha- 180 Maiden Lane Phase II - Invoices #2 & 3employee forwardalpha_new_york 84%alpha_new_york 87%alpha_early_center_10038_llc_alpha_early_center_fin_district 80%alpha_early_center_10038_llc_alpha_early_center_fin_district 84%—
Re: Payment status confirmation: Michael Kanari | Invoice $•••employee forwardother_alpha_campus 72%other_alpha_campus 71%other_alpha_campus 65%other_alpha_campus 78%—
Re: Payment status confirmation: Michael Kanari | Invoice $•••employee forwardother_alpha_campus 91%other_alpha_campus 83%other_alpha_campus 71%other_alpha_campus 82%—
Re: New payment request from Central Coast Contractor, Inc. - invoice 1064employee threadalpha_santa_barbara 100%alpha_santa_barbara 100%alpha_school_93101_llc_santa_barbara 100%alpha_school_93101_llc_santa_barbara 100%—
Re: XTIUM Overdue Invoices . . . Alpha School, LLCemployee forwardnot_campus_specific 100%not_campus_specific 100%not_campus_specific 69%not_campus_specific 82%—
New payment request from White Rock Air Conditioning and Heating LLC - invoice 9668vendor platformalpha_carrollton 100%alpha_carrollton 100%alpha_school_75010_llc_alpha_carrolton 59%alpha_school_75010_llc_alpha_carrolton 58%Sports Academy 75010, LLC (CSA) (1 bill)
Re: Invoice 10904 from Green Top Farms LLCemployee threadnot_campus_specific 99%not_campus_specific 99%not_campus_specific 98%not_campus_specific 99%—
Re: Zigco questionvendor replynot_campus_specific 81%not_campus_specific 81%sports_academy_78734_llc_tsa 45%not_campus_specific 45%—
Reminder: Invoice 12744 from Watchmen Protective Services, LLCvendor platformalpha_austin 100%alpha_austin 100%alpha_school_78746_llc_austin_k_8 100%alpha_school_78746_llc_austin_k_8 100%—
Reminder: Invoice 12745 from Watchmen Protective Services, LLCvendor platformalpha_austin 94%alpha_austin 91%alpha_school_78746_llc_austin_k_8 95%alpha_school_78746_llc_austin_k_8 95%Alpha School 78701, LLC (Alpha High School) (1 bill)
Reminder: Invoice 12746 from Watchmen Protective Services, LLCvendor platformalpha_austin 96%alpha_austin 97%alpha_school_78746_llc_austin_k_8 92%alpha_school_78746_llc_austin_k_8 97%Sports Academy 78734, LLC (TSA) (1 bill)
Reminder: Invoice 12713 from Watchmen Protective Services, LLCvendor platformalpha_austin 100%alpha_austin 100%alpha_school_78746_llc_austin_k_8 100%alpha_school_78746_llc_austin_k_8 100%—
Reminder: Invoice 12714 from Watchmen Protective Services, LLCvendor platformalpha_austin 94%alpha_austin 87%alpha_school_78746_llc_austin_k_8 61%alpha_school_78746_llc_austin_k_8 56%Alpha School 78701, LLC (Alpha High School) (1 bill)
Re: Alpha Billing 2FA Code Authenticator Issuesystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Re: Alpha Billing 2FA Code Authenticator Issuesystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Invoice - Reminder: Your payment to LanguageBird, Inc. is duevendor platformalpha_austin 61%alpha_austin 56%not_campus_specific 34%not_campus_specific 38%Alpha School 78701, LLC (Alpha High School) (1 bill)
Re: Alpha School – Billing Reconciliation, Ledger & Remaining Creditvendor replyalpha_miami 76%alpha_miami 78%alpha_school_33155_llc 77%alpha_school_33155_llc 66%Alpha School 33155, LLC (1 bill)
Reminder: Invoice 8366 is Due Soon.vendor platformother_alpha_campus 57%other_alpha_campus 58%nova_academy_78723_llc_nova_austin 89%nova_academy_78723_llc_nova_austin 80%—
Reminder: Invoice 8369 is Due Soon.vendor platformnot_campus_specific 99%not_campus_specific 98%not_campus_specific 48%not_campus_specific 39%—
Your Connect application is approvedvendor platformnot_campus_specific 60%not_campus_specific 63%alpha_school_78701_llc_alpha_high_school 90%alpha_school_78701_llc_alpha_high_school 72%—
Re: XTIUM Overdue Invoices . . . Alpha School, LLCvendor replynot_campus_specific 100%not_campus_specific 100%not_campus_specific 79%not_campus_specific 84%—
Your Order Is Completed 105-130390vendor directalpha_fort_worth 81%alpha_fort_worth 81%alpha_school_76107_llc_fort_worth_fletcher 93%alpha_school_76107_llc_fort_worth_fletcher 91%—
Re: Flourishing Families retreatsvendor replynot_campus_specific 99%not_campus_specific 99%not_campus_specific 97%not_campus_specific 96%—
Re: Alpha Billing 2FA Code Authenticator Issuesystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Bills paid (9/21/26) but not postedemployee threadnot_campus_specific 100%not_campus_specific 100%not_campus_specific 99%not_campus_specific 99%—
Fwd: This is your invoice 1419061 from Valley Alarmemployee forwardalpha_malibu 100%alpha_malibu 100%alpha_school_90265_llc_alpha_malibu 100%alpha_school_90265_llc_alpha_malibu 100%—
Automated Payment Reminder – Invoice #1029vendor platformalpha_miami 88%alpha_miami 88%alpha_school_33155_llc 93%alpha_school_33155_llc 93%Alpha School 33155, LLC (1 bill)
Re: Fwd: Open Invoices (Skyline Vision Security Alliance)employee forwardnot_campus_specific 99%not_campus_specific 99%not_campus_specific 98%not_campus_specific 97%Alpha School 33155, LLC (1 bill)
Automated Payment Reminder – Invoice #1035vendor platformalpha_miami 87%alpha_miami 89%alpha_school_33155_llc 90%alpha_school_33155_llc 90%Alpha School 33155, LLC (1 bill)
Automated Payment Reminder – Invoice #1036vendor platformalpha_miami 98%alpha_miami 98%alpha_school_33155_llc 98%alpha_school_33155_llc 98%Alpha School 33155, LLC (1 bill)
Re: Payment Needed: Harvard Faculty Club Retreat Depositsemployee threadalpha_orange_county 80%alpha_orange_county 81%alpha_school_92610_llc_orange_county 89%alpha_school_92610_llc_orange_county 68%—
Bills paid (9/22/26) but not postedemployee threadnot_campus_specific 69%not_campus_specific 64%not_campus_specific 89%not_campus_specific 65%—
2nd request Invoice 26223 from CM Security Consulting, Inc.vendor directnot_campus_specific 82%not_campus_specific 84%not_campus_specific 91%not_campus_specific 97%—
2nd request -Invoice 26222 from CM Security Consulting, Inc.vendor directnot_campus_specific 52%not_campus_specific 49%not_campus_specific 84%not_campus_specific 78%—
Fwd: Invoice - Reminder: Your payment to LanguageBird, Inc. is dueemployee forwardalpha_high_austin 71%alpha_high 61%alpha_school_78701_llc_alpha_high_school 85%alpha_school_78701_llc_alpha_high_school 64%Alpha School 78701, LLC (Alpha High School) (1 bill)
F-24317: Payment due soon – don’t forget!vendor directnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
RE: VALLEY ALARM: Appt. On 7/20/26 at 11amvendor replynot_campus_specific 100%not_campus_specific 99%not_campus_specific 99%not_campus_specific 99%—
RE: VALLEY ALARM: Appt. On 7/20/26 at 11amvendor replynot_campus_specific 99%not_campus_specific 99%not_campus_specific 99%not_campus_specific 99%—
Re: Invoices for S5 - btx + atxvendor replynot_campus_specific 45%alpha_brownsville 44%not_campus_specific 55%not_campus_specific 69%—
Re: Alpha School Invoicesvendor replynot_campus_specific 38%alpha_brownsville 45%alpha_school_78521_llc_brownsville 52%alpha_school_78521_llc_brownsville 51%—
New payment request from Headway Circle LTD - invoice 1154vendor platformnot_campus_specific 67%not_campus_specific 73%nova_academy_78723_llc_nova_austin 91%nova_academy_78723_llc_nova_austin 93%—
Re: Payment Request – CP Building Group Pay Apps 003 & 004 – Houston 777 ($•••)employee threadalpha_houston_heights 100%alpha_houston_heights 100%alpha_school_77008_llc_alpha_houston_heights 100%alpha_school_77008_llc_alpha_houston_heights 100%—
Re: Payment status confirmation: Michael Kanari | Invoice $•••employee forwardother_alpha_campus 75%other_alpha_campus 78%other_alpha_campus 62%other_alpha_campus 80%—
Payment remindervendor directnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
Following Up – Additional Resourcesvendor directnot_campus_specific 99%not_campus_specific 99%not_campus_specific 100%not_campus_specific 100%—
New Vendor Setupvendor directnot_campus_specific 100%not_campus_specific 98%not_campus_specific 97%not_campus_specific 99%—
SETTLEMENT PAYMENT REQUEST: Alpha Holdings and Good Guardvendor directnot_campus_specific 100%not_campus_specific 100%not_campus_specific 98%not_campus_specific 97%—
Re: 2026.09.07- ALPHA 00692- Invoicevendor replyalpha_dorado 99%alpha_dorado 99%alpha_school_00692_llc_dorado 99%alpha_school_00692_llc_dorado 99%—
Re: Automated Payment Reminder – Invoice #1029employee threadnot_campus_specific 100%not_campus_specific 100%not_campus_specific 99%not_campus_specific 99%Alpha School 33155, LLC (1 bill)
Re: Automated Payment Reminder – Invoice #1036employee threadnot_campus_specific 100%not_campus_specific 100%not_campus_specific 99%not_campus_specific 99%Alpha School 33155, LLC (1 bill)
Re: Flourishing Families retreatsvendor replyalpha_orange_county 73%alpha_orange_county 75%alpha_school_92610_llc_orange_county 80%alpha_school_92610_llc_orange_county 78%—
Re: XTIUM Overdue Invoices . . . Alpha School, LLCemployee forwardnot_campus_specific 100%not_campus_specific 100%not_campus_specific 82%not_campus_specific 90%—
Re: 2205 East Hebron Pkwy Invoicevendor replyalpha_carrollton 99%alpha_carrollton 99%alpha_school_75010_llc_alpha_carrolton 100%alpha_school_75010_llc_alpha_carrolton 99%—
XTIUM Invoice Number 10013443vendor platformother_alpha_campus 70%other_alpha_campus 71%other_alpha_campus 50%other_alpha_campus 49%—
Re: Alpha School Invoicesemployee threadnot_campus_specific 99%not_campus_specific 99%not_campus_specific 96%not_campus_specific 97%—
Put Students in the Spotlightvendor directnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
Re: Fwd: 2nd request Invoice 26223 from CM Security Consulting, Inc.employee forwardnot_campus_specific 93%not_campus_specific 94%not_campus_specific 95%not_campus_specific 84%—
Re: Flourishing Families retreatsemployee threadalpha_orange_county 61%alpha_orange_county 60%alpha_school_92610_llc_orange_county 82%alpha_school_92610_llc_orange_county 70%—
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 97%alpha_brownsville 95%alpha_school_78521_llc_brownsville 94%alpha_school_78521_llc_brownsville 91%—
Re: Nellie's Catering Weekly Vendorvendor replynot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
FW: PAST DUE INVOICES vendor directalpha_miami 49%alpha_miami 45%not_campus_specific 46%not_campus_specific 50%Alpha School 33155, LLC (1 bill)
Fw: Nellie's Catering Weekly Vendorvendor directnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
Fwd: PAST DUE INVOICESemployee forwardnot_campus_specific 49%alpha_miami 45%not_campus_specific 48%not_campus_specific 51%Alpha School 33155, LLC (1 bill)
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 94%alpha_brownsville 93%alpha_school_78521_llc_brownsville 96%alpha_school_78521_llc_brownsville 89%—
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 95%alpha_brownsville 94%alpha_school_78521_llc_brownsville 93%alpha_school_78521_llc_brownsville 90%—
Re: 2026.09.07- ALPHA 00692- Invoiceemployee threadnot_campus_specific 54%not_campus_specific 55%alpha_school_00692_llc_dorado 48%alpha_school_00692_llc_dorado 52%—
Fwd: 2026.09.07- ALPHA 00692- Invoice [URGENT]employee forwardalpha_dorado 91%alpha_dorado 89%alpha_school_00692_llc_dorado 99%alpha_school_00692_llc_dorado 99%—
RE: Outstanding Invoices Events Collectivevendor replynot_campus_specific 53%alpha_orange_county 51%alpha_school_92610_llc_orange_county 72%alpha_school_92610_llc_orange_county 66%—
RE: Flourishing Families retreatsvendor replyalpha_orange_county 84%alpha_orange_county 82%alpha_school_92610_llc_orange_county 81%alpha_school_92610_llc_orange_county 81%—
Fwd: Routing help needed: unmatched edu.ops thread - Your bill is readyemployee forwardnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
Fwd: Routing help needed: unmatched edu.ops thread - Your bill is readyemployee forwardalpha_nashville 99%alpha_nashville 99%alpha_school_37212_llc_alpha_nashville 98%alpha_school_37212_llc_alpha_nashville 100%—
Current Statementvendor directalpha_new_york 55%not_campus_specific 44%alpha_early_center_10038_llc_alpha_early_center_fin_district 57%alpha_early_center_10038_llc_alpha_early_center_fin_district 54%—
RE: XTIUM Overdue Invoices . . . Alpha School, LLCvendor replynot_campus_specific 100%not_campus_specific 100%not_campus_specific 83%not_campus_specific 84%—
Invoicesystemother_alpha_campus 72%other_alpha_campus 69%nextgen_academy_78717_llc 79%nextgen_academy_78717_llc 79%—
RE: XTIUM Overdue Invoices . . . Alpha School, LLCemployee forwardnot_campus_specific 100%not_campus_specific 100%not_campus_specific 81%not_campus_specific 87%—
Re: ALPHA ANYWHERE NYC 27838672 REFUNDvendor replyother_alpha_campus 48%other_alpha_campus 40%other_alpha_campus 39%other_alpha_campus 50%—
Your exclusive look at emcee's event spacesvendor directnot_campus_specific 99%not_campus_specific 99%not_campus_specific 99%not_campus_specific 99%—
Your United Site Services receipt [#1657-9557]vendor platformnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%—
Your United Site Services receipt [#1214-0650]vendor platformnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 99%Sports Academy 78734, LLC (TSA) (1 bill)
ALPHA Invoicevendor directalpha_austin 99%alpha_austin 99%alpha_school_78746_llc_austin_k_8 96%alpha_school_78746_llc_austin_k_8 91%—
RE: XTIUM Overdue Invoices . . . Alpha School, LLCvendor replynot_campus_specific 100%not_campus_specific 100%not_campus_specific 84%not_campus_specific 90%—
New sign-in using a backup codesystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Security alertsystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
New sign-in using a backup codesystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Security alertsystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
Security alertsystemnot_campus_specific 100%not_campus_specific 100%not_campus_specific 100%not_campus_specific 100%—
156 William Street - Alpha School - October 2026 Rent Statementvendor directalpha_new_york 39%other_alpha_campus 34%not_campus_specific 59%not_campus_specific 59%—
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 95%alpha_brownsville 96%alpha_school_78521_llc_brownsville 94%alpha_school_78521_llc_brownsville 88%—
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 92%alpha_brownsville 91%alpha_school_78521_llc_brownsville 94%alpha_school_78521_llc_brownsville 86%—
Re: Nova High School - Brownsvilleemployee threadalpha_brownsville 95%alpha_brownsville 92%alpha_school_78521_llc_brownsville 96%alpha_school_78521_llc_brownsville 87%—
Re: 2026.09.07- ALPHA 00692- Invoice [URGENT]employee forwardalpha_san_juan 86%alpha_san_juan 85%alpha_school_00901_llc_alpha_san_juan 40%alpha_school_00901_llc_alpha_san_juan 37%—
Re: Outstanding Invoices Events Collectiveemployee threadalpha_orange_county 56%alpha_orange_county 53%alpha_school_92610_llc_orange_county 71%alpha_school_92610_llc_orange_county 76%—
Re: XTIUM Overdue Invoices . . . Alpha School, LLCemployee forwardnot_campus_specific 100%not_campus_specific 100%not_campus_specific 87%not_campus_specific 92%—