Machine learning model evaluation is measuring whether a trained model is any good on data it has not seen — the right metric for the job, a split that does not leak, and a number you can defend. It is not training. It is not deployment. It is not device verification and validation.
The same job used to live here as “machine learning model validation” and “what is model validation.” Those were keyword wraps of this page. How you train the net is a different article. How you ship it is a different article. Clinical definitions of sensitivity and specificity as stats sit on what is sensitivity and specificity. Hospital device V&V / ISO is medical device verification and validation — a different job, not a 301 target.
Evaluation versus validation
People use the words as if they were twins. On this site they land on one URL because the job is the same: is the model good? The split of labor inside that job is short:
- Training set — the net learns here.
- Validation set — you tune learning rate, architecture, early stopping. You may look at this many times.
- Test set — you score once, at the end. If you tune on it, it is not a test set.
- Evaluation — the whole measurement job: which metric, which split, which curve, whether the number leaked.
A paper AUC is not a site number. Vendor, protocol, and population move the score. Re-evaluate on your scanners.
Classification: more than accuracy
Accuracy is the fraction of correct calls. On a rare finding it lies. A net that always says “no tumor” on a set that is 99% negative is 99% accurate and useless. Work from the four counts:
- True positive / true negative — the model was right.
- False positive — it flagged a healthy study. Extra read, extra anxiety, sometimes an extra biopsy.
- False negative — it missed a finding. That is the one that delays care.
| Metric | What it asks | When it matters |
|---|---|---|
| Accuracy | Overall correct / all calls | Lies when the finding is rare |
| Precision | Of the positives you called, how many were real? | False alarms are expensive |
| Recall (sensitivity) | Of the real positives, how many did you find? | Missing the finding is the failure mode |
| Specificity | Of the real negatives, how many did you clear? | Screening; too many false alarms bury the list |
| F1 | Harmonic mean of precision and recall | You need both, and one number |
The clinical trade is the same every time: a false alarm versus a miss. Screening usually wants recall. A confirmation step before an invasive act wants precision. The longer clinical-stats version is the sensitivity / specificity page.
Regression and segmentation
A volume in millilitres, a count, a blood-pressure forecast — that is regression. MAE is the average absolute miss, in the original units. RMSE squares first, so one large miss dominates. R² is the fraction of variance the model explains, not an error in millilitres.
A mask — tumor bed, organ, vessel — is overlap, not a class label.
- IoU (Jaccard) — intersection divided by union. 1 is a perfect outline; 0 is no overlap.
- Dice — twice the intersection over the sum of the two areas. The usual medical-imaging overlap score; it is kinder to small structures than IoU.
Dice as a loss during training lives on the training page. Dice as a metric lives here.
How you split so the number is honest
A single 80/20 cut is fast and noisy: one lucky test slice and the paper looks good. K-fold (usually 5 or 10) rotates the hold-out so every study is tested once; you report the mean. Stratified k-fold keeps the class mix in every fold — required when the finding is 5% of the set, or one fold can have zero positives.
Medical volumes add a split the generic posts skip: Group k-fold, by patient. Slices or series from the same person must stay in one fold. A “test” slice that shares a patient with train is leakage, even if the filename looks new. Leave-one-out is k-fold with k = N; named more often than it is worth the compute.
Fit scalers and augment stats on the training fold only. Normalize-then-split is the classic leak: the test mean walked into training.
ROC and PR
Most nets emit a score, not a hard yes. The threshold is a choice. A ROC curve plots true-positive rate against false-positive rate as that threshold moves. AUC is the area under that curve: 1.0 separates perfectly, 0.5 is a coin flip. ROC is optimistic when the positive class is rare.
A precision–recall curve is the honest plot on an imbalanced set. It asks: as you catch more of the real findings (recall), how many of the flags are still real (precision)? That is the screening question.
What breaks the number
- Leakage. Same patient in train and test. Preprocess fit on the whole set. A report date that encodes the label. The test set is a vault.
- The wrong metric. Optimize accuracy on a 1% finding and you ship a model that never fires.
- Lab versus clinic. One vendor, one protocol, one hospital. Evaluate on the scanners you will use. Shift is not a surprise; it is the default.
- Stale models. New machine, new population, new protocol — the score moves. Evaluation does not end at launch. Watching drift after ship is the deployment page; deciding the metric is this one.
What this page is not
- Not training. Data, annotation, ResNet / U-Net, and the loss sit on machine learning model training.
- Not deployment. Docker, API, and live monitoring sit on machine learning model deployment.
- Not device V&V. ISO / FDA process for a medical device is medical device verification and validation.
- Not a sklearn tutorial. There is no listing and no reliability diagram on this page. K-fold is the method in prose, not a second URL.
PYCAD builds the imaging side of this — pipelines and viewers when the score has to hold up on a clinic’s scanners. Case studies.