Prove the model is better before you switch anything on.
Upload your labelled history and tell us what good means. Model Form fine-tunes a model on your data, holds back examples you have never seen, and scores it against what you run today on accuracy, latency and cost.
The scorecard comes before the deployment decision, not after it.
Where it loses
Deploy on 8 of 11 queues. Route the other 3 to your current system.
The held-out set
Scored on 5,240 tickets the model has never seen.
52,400 records
Train
41,920
Validation
5,240
Held out
5,240
sealed until scoring
The split is stratified by queue, so a rare category is not accidentally absent from the test set.
The held-out set is scored once, at the end. It is not used to choose the base model, the hyperparameters or the stopping point.
Your incumbent is scored on exactly the same 5,240 tickets, on the same day, through its production configuration.
Most internal pilots fail this test, because the thing that looked promising was measured on data it had already seen.
Where it loses
Three of eleven queues, and why.
Only 410 of the 52,400 tickets were billing disputes, so there was not enough history to learn the pattern.
The honest fix is more labelled history, not more training.
The training set is 97% English.
Either a multilingual base model, which costs more to serve, or routing these to the incumbent, which costs nothing to implement.
Every evaluation finds something like this. A platform that reports a clean sweep has either overfitted or is not looking.
The recommendation
A routing rule, not a switch.
You deploy where it wins and keep what you have where it does not. The next evaluation in ninety days moves the line, or it does not.
routing:
custom_model:
- "Shipment tracking" # 94% vs 86%
- "Booking changes" # 93% vs 85%
- "Invoicing questions" # 92% vs 83%
- "Integrations and API" # 90% vs 82%
- "Carrier onboarding" # 92% vs 85%
- "Account access" # 94% vs 87%
- "Rate quotes" # 89% vs 81%
- "Damage claims" # 89% vs 76%
incumbent:
- "Customs documentation" # 84% vs 88%, behind
- "Contract renewals" # 80% vs 85%, behind
- "Billing disputes" # 79% vs 88%, behind
override:
- when: language != "en"
route: incumbent
blended:
accuracy: 92.4%
cost_per_1k: $0.74
revisit: 90 daysThe weights are the easy part.
Owning a model is not owning a file. It is owning the benchmark that decides whether it is any good, the training process that produced it, and the system that serves it. Model Form hands you all four, and the evaluation harness stays yours whether or not you keep us.
- The trained weightsyours, exportable
- The held-out evaluation setyours, exportable
- The training configurationyours, exportable
- The serving containeryours, exportable
It stays sharp.
01
Managed hosting
The model is served on infrastructure we run, at the latency you were scored on.
02
Drift monitoring
Recent live traffic is scored against your chosen metric every week.
03
Periodic retraining
New labelled tickets are folded in on a schedule you set.
04
Re-evaluation
Each retrained model faces a fresh held-out set before it replaces the last.
Drift is detected by scoring recent live traffic against the same metric you chose at the start, which is the only version of drift monitoring that means anything.
Pricing
Fixed fee for the evaluation. Usage after that.
A fixed fee covers data preparation, training and the scorecard, whatever the result. If the custom model does not beat what you run today, you get that finding and you owe nothing further. Deployment is priced on tokens served, and you can take the weights and host them yourself at any point.
Measured on 5,240 tickets neither system had seen.
Run one evaluation.
Bring a labelled dataset and the metric you care about. You get a scorecard either way.