If you ship a DICOM viewer or a segmentation platform, you already know the bottleneck. Dense 3D segmentation is the default answer in papers. In a product, the first thing a radiologist needs is usually not a perfect organ contour. It is a box, a class, a score, and a region they can click.
That is the gap RadYOLO is built for. It is a 3D extension of YOLO11 from Fraunhofer MEVIS (Geissler, Müller-Groh, Meine) that does joint detection, classification, and instance segmentation on CT and MRI. Most volumes go through in one pass. The paper is on arXiv and is set for the 2nd MICCAI Workshop on Efficient Medical AI (EMA4MICCAI) in Strasbourg on 27 September 2026. Code is public under AGPL-3.0.
If you already run nnU-Net for organs, this is not a drop-in replacement. It is a different output shape, and it is fast enough that a “run on this study” button in a viewer becomes realistic.
Detection is what a viewer can hang immediately
Most medical AI stacks still train a dense segmenter, then try to recover objects after the fact. Connected components on a mask, organ volume as a fake confidence score, maybe a heuristic to split touching lesions. That works for a liver. It is a weak way to find nodules or a primary breast lesion.
RadYOLO is built as a detector first. Anchor-free 3D boxes, a class, a score, and a prototype instance mask at half resolution per axis, then a trilinear upsample. You get instances without a separate clustering step. The authors also changed the box math so the network predicts extent and center offset relative to the grid cell, which is kinder to small objects than the original YOLO11 face-offset setup.
That is the same “where, not just whether” pressure we wrote about for PatchChestCT. A whole-volume mask, or a study-level finding, is a badge. A box in the viewport is a tool.
CT-native foundation models such as Merlin make the language side cheaper. They do not replace a localizer you can overlay. If Merlin can talk about a lesion, you still need something that points at it in under a second.
What they actually trained
Five public datasets, each split 60/20/20 for train, validation, and test:
- AMOS22: CT and MRI, 15 abdominal organs
- Liver Lesions: CT lesion masks
- LUNA16: lung nodules
- MAMA-MIA: DCE breast MRI, primary tumor
- VerSe: vertebrae, 26 classes
Like YOLO11, RadYOLO ships five widths: nano, small, medium, large, extra-large. The paper only reports nano (~10M parameters) and small (~38M; 37M on LUNA16). nnU-Net in this comparison is the ResEnc M preset at ~192M. nnDetection sits around 19-21M, but the authors keep a four-checkpoint ensemble plus test-time flips (eight combinations). They tried turning that off. Detection dropped hard, so the speed comparison is against that expensive setup, not a single nnDetection forward pass.
Times include image loading, preprocessing, inference, and writing results. They do not include process startup or weight loading. Hardware is an Nvidia A100 and an Intel Xeon Gold 5320. That is a cluster, not a laptop, but it is also not a “forward pass only” number.
Most datasets are resampled to a fixed voxel extent and predicted in one shot. LUNA16 is the exception: fixed spacing of 0.7 x 0.7 x 1.3 mm and patch-based inference, same spacing as the baselines. That is why LUNA is the slow case in their table, not a failure of the architecture.
The numbers that change a product plan
On GPU, RadYOLO is 8-46x faster than nnU-Net, and 36-236x faster than nnDetection. On CPU it finishes a volume in 1.0-33 s. That CPU range is still faster than nnU-Net on a GPU in this table.
MAMA-MIA is the clean example. RadYOLO nano is 0.5 s GPU / 1.0 s CPU. Small is 0.4 s GPU / 1.2 s CPU. nnU-Net is about 10 s GPU / 18 min CPU. If your viewer has a “find the lesion” action, those two worlds are different products.
LUNA16 is the expensive end because of the patches. Small still lands at 5.8 s GPU / 33 s CPU, against 48 s / 94 min for nnU-Net. AMOS22, the mixed CT/MR organ set, is 1.8 s GPU / about 2 s CPU against 29 s / 8.9 min. Liver lesions on GPU go from 23 s (nnU-Net) to 0.5 s (RadYOLO small). That is the 46x end of the range. The 8x end is LUNA16.
Inference memory with ONNX Runtime stays around 0.6-1.1 GB. Training is a different story. The instance head stores a mask per assigned object, so datasets with many objects per volume spike. Liver Lesions nano hit 43 GB in training. If you only need boxes, the paper says you can drop the segmentation head.
Where it wins, and where you should keep nnU-Net
On detection, RadYOLO beats nnDetection on four of five datasets and is on par on LUNA16. The honest LUNA16 line is nnDetection mAP@0.1 of 0.71 versus 0.69 for RadYOLO small. That is the one “comparable” result in the abstract, not a hidden win.
Versus nnU-Net, RadYOLO is better on the lesion tasks (Liver Lesions, LUNA16, MAMA-MIA) across the detection metrics they report. On LUNA16, small mAP@0.1 is 0.69 versus 0.52. On MAMA-MIA it is 0.96 versus 0.85. When they only ask for rough localization (mAP@0.1), RadYOLO matches or beats nnU-Net on all five datasets.
Tighten the IoU (mAP@0.1-0.95) and nnU-Net pulls ahead on large anatomy: 0.84 versus 0.65 on AMOS22, 0.81 versus 0.67 on VerSe. That is more precise box placement on organs and vertebrae, not a better find rate at a loose gate.
Segmentation Dice is the same split. nnU-Net wins organs and vertebrae (AMOS22 0.89 versus 0.78, VerSe 0.83 versus 0.66). On lesions the gap shrinks (Liver Lesions 0.75 versus 0.71). RadYOLO actually wins Dice on LUNA16 (0.49 versus 0.34) and Hausdorff-95 on Liver Lesions (26 versus 38 mm) and MAMA-MIA (13 versus 43 mm). If you still quote Dice without saying what it is, read the note on the Dice coefficient versus Dice loss. 0.78 on AMOS22 is overlap on the test set, not a training-objective trick.
The authors are clear about why. Prototype masks at half resolution are less expressive than voxel-wise nnU-Net. For compact lesions that is enough, and it produces fewer catastrophic outliers on Hausdorff. For organ boundaries you still want a dense segmenter.
They also chose an IoU floor of 0.1 when they average mAP. In 2D YOLO papers the floor is usually 0.5. In 3D, a small lesion with a fuzzy edge can be “found” at 0.1 and still be a useful jump-to in a viewer. A 0.5 gate will punish that case. Design your own threshold around the object size you ship, not around a COCO habit.
How this lands in a viewer or segmentation stack
Treat RadYOLO as a first-pass object layer, not as a replacement contour engine.
On the imaging side, training and eval are volume-native. If your clinical stack is DICOM, you still have the usual job of turning a NIfTI file into a DICOM series before you hang overlays. Keep spacing and geometry honest or the boxes will drift relative to the hanging protocol.
On the model side, start with nano unless a LUNA-style small-object task forces patches. One trained checkpoint, no ensemble, no test-time flips. That is the point. You are not paying nnDetection’s four-checkpoint, eight-flip bill to get a comparable box.
On the viewer side, a scored 3D box is cheap to render:
- jump-to-finding on a study, without waiting on a GPU queue
- a lesion worklist you can refresh on CPU in seconds
- a coarse instance mask you refine with a foundation segmenter instead of drawing every voxel
- QA: “the model said nodule. Show me the box.”
That last one is the difference between a badge and a tool. If the box is late, or it sits on the table, you will see it before you care about mAP@0.1-0.95.
License and what you cannot ship
The repo disclaimer is explicit. The software is not qualified as a medical product, or as part of one. It is provided as is, without a validation claim.
The license is AGPL-3.0. If you modify RadYOLO and offer it as a network service, you owe users the corresponding source. That is a real block for a proprietary SaaS viewer that just wraps the weights. Research, internal eval, and open products are the easy path. A closed commercial fork is a legal review, not a pip install.
The arXiv text is still a preprint. The authors say it has not gone through peer review or post-submission corrections, and the workshop version-of-record DOI is not filled in yet. Treat the tables as the current public numbers, not as a cleared clinical claim.
What to do next
Do not start by ripping nnU-Net out of organ segmentation. Start with one lesion task you already show in a viewer and time the current path, I/O included. Then install from the Fraunhofer MEVIS repo:
uv build .
# or: pip install .
uv run -m radyolo train path/to/config.yamlDataset format, splits, optional preprocess, and inference are in the repo docs. If the overlay is late, or the boxes miss, you will see it in the viewport before a leaderboard number helps you.
If the box is useful and the organ contour is not, keep RadYOLO for detection and leave nnU-Net on the organs. That split is the paper’s actual conclusion, and it is the one you can ship.
Sources
- Geissler, K., Müller-Groh, L., Meine, H. RadYOLO: Computationally Efficient 3D Object Detection and Segmentation in CT and MRI. arXiv:2608.00508. https://arxiv.org/abs/2608.00508 (HTML: https://arxiv.org/html/2608.00508)
- Code (AGPL-3.0): https://github.com/FraunhoferMEVIS/RadYOLO
- 2nd MICCAI Workshop on Efficient Medical AI (EMA4MICCAI), Strasbourg, 27 September 2026