Explainable AI in healthcare is the set of methods that show why a model flagged a case — which pixels, which labs, which features — instead of returning a bare score. It is not a second “what is AI in healthcare” explainer. That page is here.
A convolutional net can mark a nodule and not tell you which voxels drove the mark. A clinician who cannot check the mark will not use it. That is the trust problem. XAI is the attempt to put the evidence next to the score.
Transparency vs interpretability
- Transparency is how the model is built: the architecture, the loss, the training data, the code path. You can read the blueprint.
- Interpretability is why this case got this output: which features or pixels moved the score. You can check the mark against the image or the chart.
Both matter. A transparent linear model still needs the coefficients explained in clinical language. An opaque net can still be locally interpreted after the fact. Neither is a substitute for a signed report.
Post-hoc vs inherent
Post-hoc methods leave the trained model alone and explain one prediction (or the global feature ranking) afterwards. Two names you will actually see on imaging and tabular work:
- LIME (Local Interpretable Model-agnostic Explanations; Ribeiro, Singh & Guestrin, KDD 2016). Perturb the input around one case, watch the black-box output change, fit a tiny local surrogate (usually a sparse linear model). The surrogate is the explanation for that case only. Paper: arXiv:1602.04938.
- SHAP (SHapley Additive exPlanations; Lundberg & Lee, NeurIPS 2017). Assign each feature a credit for moving the prediction from a baseline, using Shapley values from cooperative game theory. Local and global. More expensive than LIME; more consistent across cases. Paper: arXiv:1705.07874.
Inherent models are readable without a detective layer. A decision tree is a yes/no path. A linear (or logistic) model is a weighted sum. You trade some peak accuracy on messy pixels for a path a reviewer can follow. That trade-off is real on some tasks and oversold on others — post-hoc methods exist so you can keep the accurate net and still ask “why this case.”
| Technique | When | What you get | Limit |
|---|---|---|---|
| LIME | One case, any black box | A local, sparse “these features mattered” | Unstable if you re-run; local only |
| SHAP | Local or global feature credit | A signed contribution per feature | Slow on large models; still an approximation |
| Decision tree | You need a path a reviewer can walk | The splits themselves | Shallow trees underfit pixels |
| Linear / logistic | Tabular risk scores, simple relationships | A coefficient per variable | Misses interactions unless you add them |
Heat maps on images
On a scan the explanation is usually a saliency or class-activation map (Grad-CAM and cousins) overlaid on the MRI, CT, or X-ray. Warm pixels are the region that moved the score. The radiologist sees the region, not just a probability. That is the clinical form of post-hoc XAI — the same job as LIME/SHAP, drawn on anatomy.
A heat map can be wrong while the classification is right (and the reverse). It highlights correlation in the trained net, not a proven lesion. The reader still has to look at the original series. Detection is not diagnosis; an overlay is not a report. The generic diagnosis page is AI for medical diagnosis.
What it does not do
- It does not automatically make the model fair. An explanation can show that zip code or scanner vendor drove the score. Fixing that is a dataset and training problem.
- It does not always cost accuracy. LIME and SHAP sit on top of the existing net. Switching to a shallow tree to “be explainable” can cost accuracy. That is a model choice, not a law of XAI.
- It is not a legal checkbox. The EU AI Act treats many medical systems as high-risk and asks for documentation and human oversight. FDA SaMD review asks for a file, not a LIME plot. An overlay helps a reviewer; it is not the clearance.
- It is not a market forecast. Vendor CAGRs and “63% of hospitals” slides are not a reason to buy an explanation layer.
If the question is what AI in healthcare is at all — the stack, the imaging vs notes vs ops split, how a hospital actually starts — that is what is artificial intelligence in healthcare.
PYCAD builds the imaging side of this — custom pipelines and web DICOM viewers when the study, and the overlay, have to live in a clinic app. Case studies.