Anyone who has shipped an organ segmentation model to a second hospital knows the drill. The CT model that looked great on the training site stumbles on a new scanner, and on MRI it is close to useless. Derong Yu and Guoyan Zheng from the Institute of Medical Robotics at Shanghai Jiao Tong University posted arXiv:2610.29090 on 21 October 2026 with ReSeg, a model for the liver, spleen and both kidneys that never sees a real patient image during training. It learns from synthetic volumes painted onto organ masks, and the same weights are then run on CT, MRI and PET without any fine-tuning.
Why this matters for a viewer or a clinical platform
Most of the segmentation we deploy is tied to one modality and often one protocol. A site that sends abdominal MRI next to its CT usually ends up with two models, two validation runs and two sets of labelled data, and the MRI labels are the expensive ones. In the paper, an nnU-Net trained on CT and tested on the AMOS MRI set reached an average Dice of 33.49%. ReSeg, trained on no real images at all, reached 90.38% on the same set. A model that holds up across modalities with one set of weights is attractive for anyone who runs pre-segmentation inside a viewer, because it means one container and one validation story instead of a matrix of them.
What goes in and what comes out
The input is an abdominal volume (CT, MRI or PET) as NIfTI. The repository’s demo script takes one image file and a checkpoint and writes a label map. The output has four labels, for the liver, the spleen, the right kidney and the left kidney, and no others. Training patches are 128 x 128 x 128 voxels at 3 mm isotropic spacing, so the masks are coarse by design and would need resampling back to the native grid before they are drawn on the original slices.
How it works
The training data are not images. They are segmentation masks from the public TotalSegmentator CT dataset, used as shape templates. A generator the authors call the Generalizable Appearance model assigns every label a random intensity from a Gaussian mixture, adds Perlin noise, and applies random resolution and contrast changes, so the same anatomy shows up looking like CT one time, like some odd MRI sequence the next, and like nothing a scanner would produce the time after that. Unlike SynthSeg-style generators, it also draws the lower bound of those mean intensities at random, which spreads the contrasts further apart. The idea is that real CT, MRI and PET end up somewhere inside the range the network has already seen.
Training happens in two stages. In the first, a network called GSRNet learns features from pairs of synthetic volumes that share one label map but look different. The label maps here are shuffled, with organ shapes drawn from different patients, so the network has to tell structures apart by appearance and local context alone. A contrastive loss compares overlapping patches as well as whole volumes, which the authors say helps when the field of view changes between scans. In the second stage, the encoder from GSRNet starts a segmentation network, OESNet, trained on synthetic images built from one real patient’s full set of masks, so the organs stay where they belong. To fake the abdominal cavity around them, a series of morphological closings with shrinking spheres fills the space between organs, ribs get merged into one bone label and vertebrae into one spine label. The decoder also reads features from the first-stage network, which stays frozen.
What the evaluation shows
The authors tested on seven public datasets: BTCV, AMOS-CT and WORD for CT, the Learn2Reg abdomen MRI set, AMOS-MR and MRISegmentator for MRI, and an organ-annotated subset of AutoPET for PET. Against SynthSeg retrained on the same TotalSegmentator masks, two SAM-based zero-shot methods and several few-shot methods that do get some labelled real scans, ReSeg had the best average Dice on all four core sets, and it beat the other zero-shot methods on the three extra ones. On PET the other zero-shot methods produced nothing usable, while ReSeg’s average was 72.40%, with the liver at 82.81% and the other three organs near 69%. The qualitative figures match. In Fig. 9 of the paper, on an AMOS-MR slice, SynthSeg barely marks the liver and ZeroShotSAM paints the liver as kidney, while ReSeg’s mask is close to the reference. The paper also adapts the method to whole-heart segmentation on the MMWHS CT and MRI data, but that required changing the label processing and retraining the segmentation network.
Where it breaks
Four organs is a narrow scope. If you need the pancreas, the gallbladder, vessels or lesions, ReSeg does not cover them, and adding a new structure means generating new synthetic data and retraining stage two. PET is the weak spot: a standard deviation of 27 Dice points on the spleen means some cases fail badly, and the authors do not show which. The 3 mm working resolution will blur small kidneys and thin spleens, and nothing in the paper looks at organs with large tumours, after surgery, or with transplants, since the templates come from TotalSegmentator’s mix of routine CT. All the test sets are public research data, already curated, so nothing here tells you how it does on a cold PACS feed with odd orientations and cropped fields of view. There is also no reader study and no timing data.
Code, weights and license
The code is public at github.com/Kendra9999/ReSeg under the MIT license, with scripts for the synthetic data pipeline, both training stages and a demo for inference. As of 22 October 2026 there is no released checkpoint. The demo expects a local .pth file and the repository has no releases or download link, so you have to train it yourself. The stack is PyTorch 2.1 with MONAI and the older mmcv-full 1.7.2 and mmdet 2.28, which can be awkward to install on recent CUDA versions. Stage one reuses the data generator from Anatomix (MIT). The shape templates come from the TotalSegmentator dataset, which is CC BY 4.0, so commercial use is allowed with attribution.
How we would use it
We build DICOM viewers and imaging platforms, and the first place we would try this is as a default organ pre-segmentation that does not care which modality arrives. In practice that means converting the DICOM series to NIfTI, running ReSeg in a container, resampling the mask to the source grid and returning it as a DICOM SEG object linked to the original series, so the user can toggle and edit it as an overlay. We would keep a modality-specific nnU-Net where one already exists and is validated, and use ReSeg for the long tail: MRI protocols nobody labelled, PET-only studies and new scanners. The synthetic recipe may be more useful than the four-organ model itself, since it is a way to build a starting model for a new structure from masks alone. For background, see our posts on nnU-Net for medical image segmentation, on NV-Generate-Body synthetic CT label maps, and on what a DICOM viewer does.
Sources
- Yu, D., Zheng, G. ReSeg: Retraining-free segmentation of abdominal organs via generalizable semantic representation learning. arXiv:2610.29090, 2026. https://arxiv.org/abs/2610.29090. PDF: https://arxiv.org/pdf/2610.29090.
- ReSeg code (MIT): https://github.com/Kendra9999/ReSeg.
- Anatomix (MIT): https://github.com/neel-dey/anatomix.
- TotalSegmentator dataset v2 (CC BY 4.0): https://zenodo.org/records/10047292.
- Featured image: Figs. 8 and 9 of arXiv:2610.29090 (AMOS-CT and AMOS-MR rows, input, reference and ReSeg columns), real paper pixels, cropped and resized only.