Hang a CT, MRI, or PET volume into a viewer or clinic AI rail and you usually want one reusable 3D backbone that can fine-tune into segmentation, detection, classification, retrieval, or report tooling without training each head from scratch on a tiny local set. That is the clinic ask behind arXiv:2609.26924, nnFoundation: 3D Foundation Models for Radiology, from Harsy, Wald, Gotkowski, Kirchhoff, Knopp, Rokuss, and co-authors under shared last authorship of Vollmuth, Isensee, and Maier-Hein (DKFZ / Human Radiome Project). The paper trains a complementary CNN and ViT pair on 2.1 million CT, MRI, and PET volumes from 125 institutional and public datasets, then scores them on 108 downstream tasks, including domain shift, external partners, and low-data or low-compute regimes.
Most radiology AI still ships as one-task models trained fully supervised on expert labels. That is brittle when scanners, sites, or protocols change, and expensive when you only have a handful of annotated cases for a new organ or finding. Prior 3D foundation efforts helped, but stayed narrower in scale or evaluation. nnFoundation treats transferable volumetric features as a product you can drop into established nnU-Net and nnDetection workflows, not a one-architecture claim that one backbone fits every radiological task.
What hangs on the viewer
Upstream is a 3D cross-sectional series (CT, MRI, or PET). Downstream, after fine-tuning or frozen-feature transfer, are task heads your viewer already understands: dense segmentation masks, detection boxes, classification scores, retrieval neighbors, or report-generation features. The hangable story for a DICOM shop is not a new overlay format. It is pretrained 3D encoders that plug into nnU-Net planning for segmentation and nnDetection for detection, so you keep the same crop, spacing, and inference path you already ship.
Practically, CNN weights are the first pick when the viewer must draw precise organ or lesion boundaries. ViT weights are the first pick when the job is global semantics: frozen features, retrieval, or report-style language heads. Dataset-aware post-hoc topology adaptation during weight transfer is part of the release story, so a single checkpoint can remap into the topology nnU-Net or nnDetection would plan for that dataset instead of freezing one fixed architecture for every series.
How it works in plain words
The authors compare pretraining recipes under matched compute and land on masked autoencoding (MAE) for both backbones. They then train two complementary models on the full 2.1M-volume Human Radiome Project corpus: nnFoundationCNN and nnFoundationViT. Self-supervised MAE hides 3D patches and asks the network to reconstruct them, so the encoder learns volumetric structure without pixel-level labels. Fig. A1 in the paper shows that reconstruction story across Abdomen CT, Knee MRI, Brain MRI, and Chest CT (Original, Masked, CNN, ViT).
Transfer is not fixed-topology only. Building on nnU-Net and nnDetection planning, they add heuristic dynamic weight adaptation that maps pretrained kernels and stages onto the dataset-specific topology at initialization. That matters for anisotropic series (their MSD Pancreas example is the clearest: fixed-topology pretrained transfer still lagged a dataset-adapted from-scratch nnU-Net until topology alignment produced a further gain). Release path: models integrated into official nnU-Net and nnDetection repositories, plus dedicated fine-tuning paths for classification, retrieval, and vision-language tasks. The paper points to the nnSSL pretraining framework and downstream repos on GitHub.
What the numbers say
Across the 108-task suite, both nnFoundation models beat training from scratch and prior 3D foundation baselines on average, with a clear task split. On segmentation, nnFoundationCNN led and beat the next-best foundation baseline CTFM by about 3.0 Dice points and nnU-Net trained from scratch by about 1.9 Dice points. Under that broad protocol, no competing foundation model surpassed nnU-Net; nnFoundationCNN was the one that did. Those Dice deltas are ranking evidence across many datasets, not a device claim for any single organ overlay.
On classification, nnFoundationViT took the best mean rank and improved over training from scratch by about 8.3 AUROC points (mean AUROC near 0.75 among several close foundation baselines). ViT also led mean rank on image-to-image retrieval and overall report generation (for example CT-RATE F1 43.7 vs 42.8 for the next-best method in the paper). Under domain shift across seven paired OOD settings, nnFoundationCNN reached about 74.9 OOD Dice while retaining about 88.8% of its in-distribution score. Dynamic topology adaptation added about 1.2 Dice on 34 segmentation sets and about 1.2 mAP on 9 detection sets versus fixed-topology transfer. Low-data fine-tuning showed the largest relative gains versus scratch training when labeled cases were scarce.
Where it fails and what not to trust
One backbone is not enough. CNN wins spatial localization; ViT wins global semantic and frozen-feature jobs. If you ship only one encoder into every viewer rail, you will mis-match the task. Fixed-topology transfer without dataset-aware remapping underperforms on anisotropic or extreme-aspect volumes. The pretraining corpus itself is modality-imbalanced (MRI heavy, PET thin at about 1.7%), so PET transfer needs extra local scrutiny. Detection comparisons against other foundation models were limited because baseline fine-tuning integrations were missing, so treat detection numbers as nnFoundation-within-nnDetection evidence.
nnFoundation is research foundation weights for fine-tuning inside established workflows, not a medical device and not a cleared overlay product. Validate on your scanners, protocols, and label definitions before you hang auto masks or auto reports. Prefer keeping CNN and ViT as selectable backends behind the same viewer job types rather than advertising a universal 3D AI.
For a viewer or clinic AI shop
Wire CT/MRI/PET volumes in through your existing nnU-Net or nnDetection train and infer path; hang task-ready masks, boxes, or feature rails out, with an explicit choice of CNN vs ViT by job type (localization vs global semantics). Turn on dataset-aware topology adaptation at weight transfer so anisotropic series do not inherit a mismatched encoder shape. Gate low-data organ projects on a small local fine-tune rather than freezing a random from-scratch U-Net when labels are scarce.
If you already ship a custom DICOM viewer or clinic AI platform, start by swapping the segmentation or detection backbone to the released nnFoundation checkpoints inside nnU-Net / nnDetection, keep your overlay and QA UX, and A/B against your current from-scratch baseline on one external site before you widen the hang. Rebuild from arXiv:2609.26924. PDF: https://arxiv.org/pdf/2609.26924.
Sources
- Harsy, C. U., Wald, T., Gotkowski, K., Kirchhoff, Y., Knopp, M., Rokuss, M., et al. nnFoundation: 3D Foundation Models for Radiology. arXiv:2609.26924, 2026. https://arxiv.org/abs/2609.26924. PDF: https://arxiv.org/pdf/2609.26924.
- Pretraining: 2.1 million CT/MRI/PET volumes from 125 datasets (Human Radiome Project / THRP); complementary MAE-pretrained CNN and ViT; released into nnU-Net and nnDetection.
- Supporting numbers from the paper (research ranking evidence, not a device claim): segmentation CNN vs CTFM ~+3.0 Dice, vs nnU-Net-from-scratch ~+1.9 Dice; classification ViT ~+8.3 AUROC vs scratch; dynamic topology adaptation ~+1.2 Dice / +1.2 mAP; OOD CNN ~74.9 Dice with ~88.8% ID retention across seven paired shifts.