Whole-body PET/CT lesion segmentation has a labelling problem that anyone who has tried to build a training set knows well. Hospitals have thousands of PET/CT exams in the archive, but only a few hundred of them come with lesion contours, and those contours take a nuclear medicine physician a long time per patient. In arXiv:2610.40284, Amartya Bhattacharya, Bashirul Azam Biswas, Biratal Raj Wagle, Michael S. Leapman, Matthew E. Maeder, James B. Yu and Indrani Bhattacharya (Dartmouth Geisel School of Medicine, Yale University and Dartmouth Hitchcock Medical Center) put the unlabelled part of that archive to work. They pretrain a 3D segmentation network on 5,910 whole-body PET/CT scans without a single label, then fine-tune it on AutoPET-III and compare it with the same network trained from scratch.
The method is called LoRe-MIM, short for Lost-or-Replaced Masked Image Modeling. We covered a general 3D feature model in nnFoundation. This paper is narrower: one pretext task built for the PET and CT pair, and a close look at whether it helps lesion segmentation where readers care most, in the liver and in bone.
What the method does
The pretraining is masked image modeling, the same family as masked autoencoders. You hide parts of the input and train the network to rebuild them. The twist is in how the parts are hidden.
- Paired patches. The network sees an aligned PET patch and CT patch of 96 x 128 x 128 voxels, split into small blocks of 12 x 16 x 16 voxels.
- Pick blocks in each modality. A set of blocks is chosen in the PET and, independently, in the CT.
- Lost or Replaced. Each chosen block is either zeroed out (“Lost”) or overwritten with the block from the same location in the other modality (“Replaced”), with a coin flip deciding which. A replaced CT block now holds PET intensities, and the other way round.
- Rebuild both. A SwinUNETR (the MONAI version) learns to reconstruct the original PET and CT. Errors on replaced blocks get the full weight in the loss, and the rest get a lower weight.
- Fine-tune. The pretrained network is then trained for lesion segmentation on the AutoPET-III training and validation sets inside the AutoPET-tuned nnU-Net framework of Rokuss et al. The authors try three versions: everything trainable, the encoder and decoder trainable with what the paper calls the backbone frozen, and the decoder alone.
The replaced blocks are what I find interesting. Zeroed blocks teach the network to fill holes from the surrounding anatomy, which any masked model does. A block of PET values sitting inside a CT forces the network to notice that something is wrong and decide what tissue belongs there, which only works if it has learned how uptake and anatomy relate. The ablations back this up. Mixing both perturbations reconstructed the images better than the other schemes the authors tried, including replacement alone and zeroing PET and CT at the same spots.
The pretraining data is mixed on purpose: AutoPET-III, SPADE and ViMED (FDG), plus 1,599 unlabelled PSMA PET/CT exams from Dartmouth Hitchcock. That covers melanoma, lung cancer, lymphoma and prostate cancer, two tracers and several scanners. A fifth set, Deep-PSMA (200 scans: 100 patients imaged with both FDG and PSMA before lutetium therapy), was kept out of both pretraining and fine-tuning and used only as an external test.
What it would hang on a scan
The viewer gets nothing new. The output is the same lesion mask any PET/CT segmentation model gives you, one label map over the whole body, ready for SUV statistics, total lesion volume or a per-lesion list. What changes is how many of those overlay blobs are wrong. The featured image shows the case the authors picked for Figure 8: the scratch model marks a spot of uptake low in the abdomen as a lesion, and the pretrained model leaves it alone while keeping the real lesion. In the zoomed patches of Figure 7, a liver lesion the scratch model missed entirely shows up after pretraining.
For a product team, the more useful part may be the recipe. If you already train lesion models on public data, you probably also have a pile of unlabelled exams from partner sites. The paper suggests that pretraining on a mixed pile like that, with this pretext task, helps the fine-tuned model on scans from a centre it never saw. The authors also check how the features hold up with noise, PET/CT misregistration and a missing modality, on a simpler patch classification task.
Results that matter
On the AutoPET-III held-out test set, LoRe-MIM pretraining raised the SwinUNETR’s Dice from 0.526 to 0.581 and cut the false positive volume by about 40%. That moved the SwinUNETR from behind the AutoPET-tuned nnU-Net trained from scratch to slightly ahead of it. The external Deep-PSMA set moved in the same direction, and that result carries more weight, since the network saw none of those scans in pretraining or fine-tuning, and the patients are a narrower group: men with advanced prostate cancer about to start lutetium therapy.
The region breakdown is where I would look as a reader. The gains were largest in the liver, where FDG background uptake hides metastases, and in high-risk bone regions on Deep-PSMA, where specificity rose clearly. The label-efficiency curves show what you would expect from pretraining: the gap is widest with 1% to 5% of the labels and shrinks as you add labelled scans.
The pretrained decoder features also beat DINOv3 and a 3D ResNet-50 when used for a simple cancer versus non-cancer patch classifier, including under added noise and PET/CT misalignment.
Where it fails
- Not every lesion. Figure 7, row 4 is a Deep-PSMA liver lesion that the decoder-only version misses and the encoder and decoder version only partly covers. Full fine-tuning and the scratch nnU-Net get it. Which fine-tuning variant you pick changes which lesions you lose.
- nnU-Net still wins on precision per lesion. The scratch nnU-Net kept the best lesion-level positive predictive value on both test sets. If your users are annoyed by false lesion calls one by one, the pretrained SwinUNETR does not fix that outright.
- Missing PET hurts badly. In the robustness tests, removing PET cost every model a large share of its classification accuracy, LoRe-MIM included, though it held up better than the baselines. Missing CT hurt much less. Do not expect this to run on CT alone.
- Modest absolute numbers. A Dice around 0.58 on whole-body lesions is a research baseline, not something to measure tumour burden with unattended.
- Some tricks did not help. PET/CT-specific augmentation and variable masking ratios during pretraining did not improve results, so the gains come from the data mix and the masking scheme, not from tuning.
- One architecture, one fine-tuning set. Everything is SwinUNETR fine-tuned on AutoPET-III. Whether the pretraining helps an nnU-Net, or a model fine-tuned on PSMA labels, is not tested.
Code, data and licence
The preprint does not link code or pretrained weights, and it does not say whether they will be released. AutoPET-III, SPADE, ViMED and Deep-PSMA are public datasets under their own terms. The Dartmouth Hitchcock PSMA cohort is institutional and not shared. The paper was submitted to arXiv on 29 October 2026 under CC BY 4.0, and the work was funded by an NIH IDeA grant. This is a research method, not a cleared product.
My take
The Lost-or-Replaced trick is small and easy to add to any masked pretraining loop for paired modalities, and the external Deep-PSMA result is the reason I would take it seriously. Without released weights, though, the 5,910-scan pretraining is the expensive part you would have to redo yourself. If I were building a PET/CT lesion model today, I would try this pretext on our own unlabelled exams, fine-tune the whole network, and keep a scratch nnU-Net running beside it to compare lesion by lesion.