Nature Communications published SAT3D on 11 August 2026 (accepted 31 July 2026). The paper is from Himashi Peiris, Sizhe Wang, Gary Egan, Mehrtash Harandi, Meng Law, and Zhaolin Chen at Monash University and Alfred Health. SAT3D is Segment Any Tumour 3D: a promptable volumetric model trained on public tumour sets, plus a 3D Slicer plugin you can load a volume into and click.
Mean in-distribution Dice is 0.672 against nnU-Net at 0.676. That is on par, not a blowout. The gap that does show up is against other 3D SAM-style models, and on a few hard, low-contrast targets. Code is at github.com/himashi92/SAT3D. Weights sit on Figshare 10.6084/m9.figshare.30155497.
Swin encoder, then a voxel-wise critic as a dense prompt
The backbone follows the SAM split: 3D image encoder, 3D prompt encoder, 3D mask decoder. The encoder is a tiny Swin (embedding dimension 48) with 3D patch partition and patch merging. Sparse prompts are points. Dense prompts are the previous mask plus a confidence map from a CNN critic.
The critic does voxel-wise real/fake classification on the predicted mask and writes a confidence volume. High-uncertainty voxels sit on the boundary, which is where tumours usually fail. That map is thresholded (T = 0.3 in the paper) and concatenated with the previous mask as the next dense prompt. Training also uses Dice plus cross-entropy, a small adversarial term (weight 0.01), and a masked uncertainty loss (weight 0.1). First step has an empty previous mask. Later steps reuse both the mask and the critic map. During training they generate five successive predictions per volume when the prompt budget is set to m = 5.
SAT3D was trained from scratch on this tumour pile. SAM-Med3D was pretrained on about 150,000 3D volumes from 80-plus public sets. The Turbo variant used about 500,000 scans from 120-plus sets. FastSAM3D sat around 45,000 scans and trades capacity for speed. The SAT3D bet is a smaller Swin plus the critic loop, not a bigger pretraining corpus.
SLIP is about click wait-time and undo cache; SAT3D is tumour-specific uncertainty as a dense prompt.
17,075 volume-mask pairs from 11 public sets
They train on 17,075 three-dimensional volume-mask pairs drawn from 11 public collections: BraTS 2021, LiTS, KiTS 2023, AutoPET 2024, HNTSMRG 2024, TDSC-ABUS 2023, KiPA 2022, and the Medical Segmentation Decathlon lung, colon, pancreas, and hepatic-vessel tasks. That is 3,884 tumour and cancer cases for training and 694 for in-distribution test (about an 85/15 split of the 11 sets). Modalities are CT, CTA, MRI (T1-weighted, T2-weighted, FLAIR, contrast-enhanced T1), FDG-PET, and ultrasound.
Targets include brain subregions (edema / SNFH, enhancing tumour, necrotic and non-enhancing core), head-and-neck GTVp and GTVn, breast, lung, liver, hepatic, renal, kidney, pancreas, colon, and whole-body FDG-avid lesions. Volumes are cropped or padded to 128^3 and z-score normalised. Inference uses a sliding window of the same patch size. Prompt-based models are scored with 5, 10, 15, and 20 foreground points. No background points, boxes, or scribbles. The headline table uses 20 points. If a method returns several candidate masks, they keep the highest Dice, so those VFM numbers are an upper bound for that click budget. nnU-Net is trained from scratch on the same splits, with no prompts. Dice rose for all prompt models as the click count went from 5 to 20. SAT3D moved the steadiest on liver, pancreas, and renal. FastSAM3D swung the most between prompt settings.
On BraTS and AutoPET, nnU-Net sees all modalities as multi-channel input. SAT3D and the other vision foundation models use a single-modality loader. For the main comparison the authors report the best mean Dice across modalities for the VFMs. SAT3D can still run on one contrast when the rest of the protocol is missing. nnU-Net trained as a four-channel BraTS model cannot do that without a workaround.
In-distribution Dice sits next to nnU-Net
Mean Dice on the 694-case in-distribution test, 20-point setting for the prompt models:
- SAT3D 0.672
- nnU-Net 0.676
- SAM-Med3D Turbo 0.550
- SAM-Med3D 0.503
- FastSAM3D 0.457
0.672 is overlap on the test set. If you still mix the coefficient with the training loss, read the note on Dice versus Dice loss.
Per-class Dice from Table 1 is more useful than the mean. SAT3D leads pancreas (0.704 versus nnU-Net 0.427) and head-and-neck GTVp (0.795 versus 0.684). Breast is 0.773 versus 0.494. Colon primaries are 0.616 versus 0.532. nnU-Net stays ahead on large, high-contrast liver tumour (0.647 versus 0.545) and lung (0.761 versus 0.728), and on several brain subregions and GTVn (0.739 versus 0.580). Renal Dice is SAM-Med3D Turbo 0.904, nnU-Net 0.880, SAT3D 0.860. SAT3D does have the lowest HD-95 on renal (4.14 mm versus 9.35 mm for nnU-Net). The paper’s own line is that SAT3D matched or exceeded nnU-Net on several hard targets when you also look at HD-95 and relative volume error, not that it won every abdominal Dice cell.
Across DSC, IoU, HD-95, ASSD, and RVE, a Friedman test ranks SAT3D first (average rank 1.73) and nnU-Net second (2.46), with chi-square 89.54 (p = 1.65e-18). Wilcoxon tests on Dice find SAT3D higher than SAM-Med3D and FastSAM3D on most solid tumours, and comparable or slightly lower than nnU-Net on liver and lung. AutoPET whole-body lesion Dice stays low for every VFM, including SAT3D (0.326 versus nnU-Net 0.518 on the generic “Tumour” row). The authors point at one PET set and one whole-body set in training.
Zero-shot HECKTOR, Prostate158, CrossMoDA
Held-out eval is HECKTOR 2022 CT (GTVp and GTVn), Prostate158 T2-weighted MRI, and CrossMoDA 2022 contrast-enhanced T1 vestibular schwannoma. SAT3D saw HNTSMRG 2024 MRI for head-and-neck labels, so HECKTOR is cross-modality, not a new class. Prostate and vestibular schwannoma were not in training.
Against SAM-Med3D Turbo:
- HECKTOR GTVp Dice 0.630 versus 0.573
- HECKTOR GTVn 0.578 versus 0.422
- Prostate158 0.507 versus 0.265 (HD-95 6.52 mm versus 15.40 mm)
- CrossMoDA vestibular schwannoma Dice 0.758, ASSD 0.68 mm
HECKTOR HD-95 dropped by about 30% versus Turbo. That is transfer off the 11-set pile, not a claim that SAT3D replaces a site-tuned nnU-Net on those tasks.
67.2 G FLOPs and 11.1 M parameters at 128^3
At a 128x128x128 input, SAT3D is 67.2 G FLOPs and 11.1 M trainable parameters. SAM-Med3D is 180 G and 101 M. FastSAM3D is cheaper in FLOPs (47.1 G) and heavier in parameters (53.2 M). Most of the cost is the image encoder. The prompt encoder and mask decoder are under 2% of the FLOPs. The critic is listed at about 11.5 G FLOPs and runs at lower resolution for the uncertainty map.
Training used two NVIDIA A6000 48 GB GPUs, PyTorch 2.4.1, AdamW (learning rate 8e-4), 500 epochs, mixed precision, and a per-GPU batch of 3 with gradient accumulation of 20. Ablations also ran on Pawsey Setonix (AMD). The Slicer plugin was tested on 3D Slicer 5.6.4 on Windows, with MONAI sliding-window inference.
What the 3D Slicer plugin actually does
The plugin loads a volume in Slicer, runs the paper’s preprocess, accepts point prompts, and overlays the SAT3D mask. You can threshold the overlay and edit a region of interest. There is no command-line step in that path. Install notes and plugin source live in the GitHub repo. The authors call this a preliminary design, not a PACS client. They show iterative refinement: add points, watch the mask update, keep the overlay in the 3D view.
If you already hang AI in a DICOM viewer, treat SAT3D as a tumour-click layer on top of the organ engine you already trust. RadYOLO is still the faster first-pass box. AMPLIFAI ships feature masks you can hang before a category head. SAT3D is the promptable tumour mask, with the critic map marking where the boundary is guessing.
Prompt quality still moves the score. Poorly placed points hurt, especially on diffuse margins. Ultrasound and uncommon sites are thin in the pile. Memory and time stay higher than a 2D slice model because the path is volumetric. The Nature paper is CC BY-NC-ND 4.0. The code is MIT. The software is a research plugin, not a cleared device.
Sources
- Peiris, H., Wang, S., Egan, G., Harandi, M., Law, M., Chen, Z. Segment any tumour: an uncertainty-aware vision foundation model for whole-body analysis. Nat Commun (2026). Published 11 August 2026. DOI: 10.1038/s41467-026-76531-2. https://www.nature.com/articles/s41467-026-76531-2
- Preprint: arXiv:2511.09592 (HTML: https://arxiv.org/html/2511.09592)
- Code (MIT): https://github.com/himashi92/SAT3D
- Weights: https://doi.org/10.6084/m9.figshare.30155497