Schäfer, Geissler, Nicke, and colleagues at Fraunhofer MEVIS, with Charité, University Hospital Frankfurt, University Hospital Erlangen, Tübingen, Hannover, and Siemens Healthineers, submitted arXiv:2608.16268 on 17 August 2026. The paper is CoM³eT: A foundation model for medical image analysis through federated, multidimensional context integration. CoM³eT is Co-representation Multidimensional Multitask Medical Transformer. A Swin Transformer V2-Base vision backbone (86.9 M parameters) turns each 2D slice or tile into a patch token and a feature pyramid. An image transformer (4.21 M) mixes those tokens across slices. A pyramid transformer (2.11 M) writes that context back into the dense maps. A decoder then writes hyperpixel tokens. The same stack is used for CT, MRI, X-ray, ultrasound, and whole-slide pathology. Pretraining is supervised multitask on more than 60 tasks and more than 100,000 patients on 10 NVIDIA A100 80 GB GPUs.
On a multicenter contrast-enhanced breast MRI set of 3,936 patients, 3D Dice without the pyramid transformer is 25.28% (95% CI 22.86 to 27.73). With it, 58.62% (55.92 to 61.31). nnU-Net is 33.13% (30.71 to 35.51) on the same full set. Restrict the set to scans that already contain a lesion and the two methods sit at 53.3% and 53.0%. Mean volume error drops from 4.46 ml without the pyramid to 2.15 ml with it; nnU-Net is 4.23 ml. Frozen CoM³eT ranked first on UNICORN at 0.442 ± 0.022 against a theoretical best-per-task ensemble at 0.357 ± 0.014, first in radiology (0.458 ± 0.027) and pathology (0.482 ± 0.018). Public code sits at github.com/FraunhoferMEVIS/MedicalMultitaskModeling. The listed UNICORN_ENCODER checkpoint is the 2D vision backbone only. The paper does not list a downloadable CoM³eT-Base with the image and pyramid transformers. Fraunhofer-Gesellschaft has a patent pending on the training algorithm and architecture (EP23209015.9).
Frozen Swin-B, then two transformers
The vision backbone is Swin Transformer V2-Base. It accepts RGB images of variable size and returns hierarchical feature maps. Global average pooling plus a linear projection collapses each map into one 512-dimensional patch token. Groups of those tokens represent a CT or MRI volume, a multiparametric series, or a whole-slide image. For dense work the same backbone also keeps the spatial pyramids. The image transformer is an encoder-only BERT-style stack over the patch tokens. It keeps one contextualized token per slice. Relative position uses a symmetric ALiBi bias measured in token steps, not mm, so adjacent slices sit at distance 1 and multiparametric volumes can share a spatial index.
The pyramid transformer is the piece a DICOM viewer hangs on a mask. It takes the image-transformer tokens V and a feature map F of shape (C, H, W) for S slices, builds an S x S sigmoid attention matrix from V, and mixes F at every spatial location with that matrix. LayerNorm, GELU, a learnable scale, and a residual write the result back onto F. Softmax is not used. The decoder turns those maps into hyperpixel tokens (dimension 64 on CoM³eT-Base, 32 on the UNICORN submission because of RAM on Grand Challenge). Classification reads patch tokens. Segmentation reads hyperpixel tokens. Task heads are a linear layer or a single convolution. The MedSAM2-Anatomy post is a frozen TotalSegmentator prior with three MedSAM2 tracks and a union. Ten-case MedSAM3 LoRA is a rank-16 adapter on a promptable SAM. This paper is a frozen Swin plus two small transformers.
UNICORN is 48 shots on a frozen encoder
UNICORN (Stegeman et al., arXiv:2603.02790) is a private Grand Challenge benchmark: 12 tasks, 5 radiology and 7 pathology, 48 labeled training examples per task, frozen foundation-model weights, lightweight adapters on extracted features. Eight of the twelve tasks are multidimensional (five 3D, three whole-slide). CoM³eT was the only submitted model that ran on every vision and vision-language task, and it was best on 6 of 12. Radiology average 0.458 ± 0.027 versus 0.341 ± 0.012 for an ensemble of two adapted nnU-Net models pretrained on CT and MRI. Pathology average 0.482 ± 0.018 versus 0.356 ± 0.015 for TITAN plus CONCH. An ensemble restricted to 3D patch features rather than voxel-level maps scored 0.077 ± 0.006. The paper states that only CoM³eT scored above random-guessing on the multiparametric MRI segmentation task. Those UNICORN numbers are rescaled task metrics from random to perfect, not Dice. Do not mix them with the 3D Dice cells below.
Breast MRI is the ugly cell without the pyramid
The authors fine-tune on three 3D segmentation settings that differ in cohort size and how much they need slice context. If you still mix the coefficient with the training loss, read the note on Dice versus Dice loss.
- Breast lesion MRI, 3,936 patients, DCE protocols on 1.5 T and 3 T GE, Siemens, and Philips scanners, mean lesion volume 6.1 ± 20.3 ml. Full-set 3D Dice: nnU-Net 33.13%, CoM³eT without the pyramid transformer 25.28%, with it 58.62%. Lesions-only: nnU-Net 53.0% (50.5 to 55.4), CoM³eT 53.3% (50.7 to 55.9). Mean volume error: nnU-Net 4.23 ml (3.57 to 4.96), CoM³eT 4.46 ml without the pyramid, 2.15 ml (1.65 to 2.75) with it. The 3,936-patient set is private Siemens Healthineers data; access is listed as corresponding-author plus Siemens permission.
- Aorta and branches on CTA. Train 100 volumes (AortaSeg24), test 56 (AVT), 1 mm isotropic, 578 to 801 slices. Without the pyramid: 80.45% (79.02 to 81.47) versus nnU-Net 80.38% (78.33 to 81.36). With it: 82.11% (80.70 to 83.17). The paper notes that several small vessels on the same slice still leak, and that the pyramid attends across slices rather than voxels. Task heads were not tuned; a 3D convolution head is listed as untested.
- Uterus MRI (myometrium, junctional zone, endometrium), 13 subjects from RACOON FADEN, T2 BLADE, 0.643 x 0.643 x 3.0 mm. Without the pyramid: 75.49% ± 0.5 versus a 3D U-Net at 71.0%. With it: 79.4% ± 0.3. nnU-Net did not produce useful results on this anisotropic 13-case set. Partial fine-tuning on the same task is 76.3% ± 0.4.
Breast classification on a separate 1,330-study Siemens collection (1,299 patients, 1,796 lesions) reports AUC rather than Dice. CoM³eT 0.859 (0.818 to 0.897) versus a video Swin Transformer at 0.840 (0.796 to 0.879). Drop the image transformer and AUC falls to 0.782 (0.732 to 0.829).
Partial fine-tuning freezes 97.5% of the model
Partial fine-tuning freezes the Swin backbone and trains the image transformer, pyramid transformer, and decoder. That cuts trainable size by 97.5%. An update at 32 patch tokens on an A100 drops from 2330 ms to 712 ms. Cache the frozen tokens and the same step is 1.5 ms against 2.3 s for full fine-tuning. Across the reported tasks, mean score is 0.833 full versus 0.839 partial. A TOST equivalence test at a 0.05 margin holds (p_lower < 0.001, p_upper = 0.008). Ultrasound lesion segmentation, a modality absent from pretraining, is 75.5% ± 2.6 full and 71.1% ± 2.3 partial.
Federated fine-tuning on a T4 and an A5000
Federated fine-tuning uses the same freeze. Sites cache patch tokens locally and sync only the image-transformer weights, every 4 local steps, through Flower over gRPC/HTTP2. The run is three sites: Charité (Tesla T4, four T4s for the cache step), UKFFM (RTX A5000, four A5000s for cache), and a public biochemical-recurrence pool. Starting checkpoint is CoM³eT-Base without recurrence-specific pretraining. CoM³eT-FL C-index on Cancer Recurrence (Whole Slide) is 0.743 ± 0.012 against pooled 0.754 ± 0.012. Surgery Outcome AUC is 0.820 ± 0.125 against pooled 0.795 ± 0.170. TOST at a 0.05 margin: p_lower = 0.019, p_upper = 0.032. Local validation used for model selection: UKFFM 0.754, Charité 0.738, Public 0.727. The paper records timeouts, access policy, and security work as operational cost.
How this lands in a viewer
If you already hang AI in a DICOM viewer, treat this as a frozen-encoder recipe for a 3D overlay that has to see more than one slice. Encode each axial (or each mpMRI series) once with Swin-B, cache the tokens, then train or swap a pyramid transformer plus a 1-layer head per task. That is the 97.5% freeze and the 1.5 ms cached step in the paper. Keep nnU-Net or TotalSegmentator as the first-pass organ mask on in-distribution CT. Use this stack when the job is detect-and-segment on a large lesion set, breast DCE MRI being the measured example, or when the volume is anisotropic and a 3D U-Net is the specialist that runs, as on the 13-case uterus set. RadYOLO is still the faster first-pass box. SAT3D is tumour plus uncertainty in Slicer. SLIP is click latency and undo. Ten-case MedSAM3 LoRA is a site adapter on a promptable SAM.
Pretraining used 10 A100s. Downstream partial fine-tuning and the federated run used consumer cards. The 3,936-patient breast MRI set and the 13-case uterus set are access-restricted. Rebuild from the protocol in arXiv:2608.16268 and the public M3 encoder. Do not wait on a Base checkpoint with the pyramid transformer. None is listed. The pending patent is EP23209015.9. Retrospective overlap is not a device claim. The preprint is the source of record until a venue version exists.
Sources
- Schäfer, J. R., Geissler, K., Nicke, T., Tappermann, C., Heber, K., Petersen, E., et al. CoM³eT: A foundation model for medical image analysis through federated, multidimensional context integration. arXiv:2608.16268. Submitted 17 August 2026. https://arxiv.org/abs/2608.16268 (HTML: https://arxiv.org/html/2608.16268)
- Code (live when checked): https://github.com/FraunhoferMEVIS/MedicalMultitaskModeling. Public checkpoint named in the README:
UNICORN_ENCODER(2D vision backbone). Companion SDK: https://github.com/FraunhoferMEVIS/m3-sdk - Stegeman, M., et al. Designing UNICORN: A unified benchmark for imaging in computational pathology, radiology, and natural language. arXiv:2603.02790. https://arxiv.org/abs/2603.02790
- Isensee, F., et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 2021. DOI: 10.1038/s41592-020-01008-z
- Schäfer, R., et al. Overcoming data scarcity in biomedical imaging with a foundational multi-task model. Nat. Comput. Sci. 2024. DOI: 10.1038/s43588-024-00662-z (UMedPT predecessor)
- Liu, Z., et al. Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR 2022. DOI: 10.1109/CVPR52688.2022.01170
- Radl, L., et al. AVT: Multicenter aortic vessel tree CTA dataset. Data in Brief 2022. DOI: 10.1016/j.dib.2022.107801
- Patent pending: EP23209015.9 (Fraunhofer-Gesellschaft; inventors J.R.S., T.N., J.L., F.K.)