Earlier MRI foundation models, including GE’s own Decipher-MR, mostly give you one embedding per series or study. That is useful for search and triage, but a DICOM viewer needs to know where the liver is, which box belongs to the left kidney, and whether the report called that organ abnormal. A GE HealthCare team in Seattle and Budapest (Noel D’souza, Zhijian Yang, Erhan Bas and 14 co-authors) posted arXiv:2610.13056 on 9 October 2026 with A4-MR. It is a 3D MRI vision-language model that finds 125 anatomy classes as 3D boxes without any prompt, gives each box its own embedding aligned with report text, and keeps healthy and abnormal versions of the same organ apart in that embedding space. No code or weights are public.
What goes in and what comes out
The input is a 3D MRI series. Before the encoder sees it, the volume is cropped to the foreground with Otsu’s method, scaled to the range -1 to 1 using the 1st and 99th intensity percentiles, and resized to 384 x 384 x 48 voxels with the aspect ratio kept. The encoder is a 3D ViT-Base with 8 x 8 x 8 patches.
Out of one forward pass you get a whole-image embedding, a 3D bounding box and class for every anatomy the DETR decoder detects (200 object queries, 125 anatomy classes plus “no object”), and one embedding per detected anatomy that lives in the same space as the text encoder. The class list covers abdomen, pelvis, spine, head and neck, and brain structures down to the fornix and mammillary bodies. Dense masks need a separate decoder trained on top of the frozen encoder.
The featured image comes from Fig. 4 of the paper: one external knee MRI case that none of the models saw in training, shown on three sagittal slices with femoral, tibial and patellar cartilage and the meniscus overlaid. From left to right are VNet, nnU-Net and MedNeXt (all trained end to end), then A4-MR with a frozen encoder and a trained decoder, then the reference standard. The cyan arrows are the authors’ marks for the larger misses.
How it is trained
The pretraining set is about 50,000 MRI studies with 491,000 series from 16 body regions, several scanner vendors and many sequence types. Reports exist for 25,500 of those studies. The team kept 85% of studies for pretraining and held out 15% for downstream validation.
Nobody hand-drew boxes for 50,000 studies. The anatomy boxes come from TotalSegmentator, GE’s own segmentation models, and, for sequences with no segmentation model, mask coordinates carried across protocols with DICOM header geometry. Each mask is then reduced to a box. On the text side, Claude 3.7 Sonnet rewrote every free-form report into one entry per anatomy with its findings and an abnormal yes/no flag. The authors manually reviewed a subset of those outputs for hallucinations.
Training runs in two stages. Stage one is self-supervised: DINOv2-style student-teacher learning on 3D crops for the image encoder, and masked language modeling on the reports for a PubMedBERT text encoder. Stage two adds the DETR localization head and two contrastive losses, one between the whole-image token and the report, and one between each detected anatomy and its own sentence of findings.
The abnormality trick sits in that second contrastive loss. Two normal thalami from different patients count as a positive pair. An abnormal organ is a positive pair only with its own finding, so an inflamed chiasma and a chiasma with a lesion are pushed apart instead of being pulled into one “chiasma” cluster.
What the numbers say
Localization is the result I would look at first. On 20 AMOS22 MR validation volumes (247 organ boxes after the authors dropped 12 pairs with unreliable ground truth), A4-MR reaches a mean 3D box IoU of 66.3 with no prompt at all (Table 7). MedSAM2 given a ground-truth box on one slice gets 65.5. MedSAM2 with boxes on three slices (74.8) and an nnUNet3D trained on AMOS (82.4) are still better.
Per organ (Supplementary Table 23), the picture splits by shape. For compact organs such as liver, spleen and kidneys, one prompted box already pins MedSAM2 at 90 to 97 IoU against 75 to 81 for A4-MR. For long or hollow organs, slice-wise propagation struggles and A4-MR wins: esophagus 61.2 vs 38.4, stomach 76.0 vs 51.4, inferior vena cava 70.7 vs 47.9, duodenum 60.9 vs 41.1. The adrenal glands and the gallbladder stay at 42 to 52 IoU, which the authors tie to the 8-voxel patch.
For segmentation, the encoder stays frozen and a SegResNet-style decoder of about 8 million parameters is trained on each dataset (Table 5). Mean Dice is 85.3 on AMOS against 85.7 for Decipher-MR and 85.6 for nnUNet3D. On a proprietary pelvis set whose 11 organs were not in the pretraining labels, it is 87.6 against 88.7 for nnUNet3D. On a proprietary knee cartilage set with a larger 58M decoder, it averages 87.3 Dice against 85.9 for both MedNeXt and VNet (Table 6). The label-efficiency curve matters more for a small team with few annotations. With 10 labeled knee volumes A4-MR reaches 68.2 validation Dice while VNet stays at 0.7 (Fig. 5).
On CirrMRI600+ cirrhosis detection with an MLP probe on frozen embeddings, AUROC is 98.9 with all training data and 97.2 with 25% of it, the same as MedSigLIP with 100% (Table 3). Removing the abnormality-aware loss drops the full-data score to 96.1 (Table 4).
Where it falls short
The authors list the gaps themselves. Pretraining labels are weak and noisy: boxes from other models, abnormal flags from an LLM, and only part of that reviewed by people. Masks were cut down to boxes, so the encoder never learned shape from voxel labels. Anything smaller than one 8 x 8 x 8 patch along an axis is hard to resolve, which is why small organs lag.
Abnormality is one binary flag per organ per report. It carries no severity, no lesion count and no location inside the organ, and the abnormality evaluation is mostly one organ and one disease (liver cirrhosis). The detector also has a fixed list of 125 anatomies. You can query that list with free text, for example “anatomies required for vision” pulls up eyes, lenses and optic nerves, but an anatomy outside the list cannot be localized. The knee is not on the list, so the knee result comes from the trained decoder rather than the detector.
Most of the strongest downstream numbers come from proprietary GE data (the pelvis, prostate and knee sets), so they cannot be reproduced outside the company. The prostate structured-report experiment is small (50 validation studies, no separate test set). There it helps prostate volume error most (16.43 cc vs 20.32 cc for Decipher-MR) and loses some lesion recall, which the authors attribute to mean-pooling the three series.
Code, weights, and data
The paper gives no repository link. On 12 October 2026 I found no GitHub repository and no Hugging Face model card for A4-MR, and no public GE HealthCare model release under that name. I found no public code or weights for the earlier GE model it builds on, Decipher-MR, either. The public evaluation sets are AMOS22 (MR) and CirrMRI600+. The retrieval set is a held-out slice of the private pretraining corpus.
How we would wire it into a viewer
If GE ships this inside its own products or releases the encoder, the prompt-free boxes are the piece I would plug in first. A box per organ, computed once when a series arrives, can drive “jump to pancreas” navigation, auto-set the field of view for a hanging protocol, crop the input for a downstream organ model, or pre-fill the location field of a structured report. We would store the boxes as machine-generated annotations tied to the series, never burned into the pixels, and show the class name next to each one so a reader can reject a wrong box in one click.
The per-anatomy embeddings open a second use: search a PACS archive for “abnormal liver on T2” at the organ level instead of the study level. Before trusting that we would test it on our own sites’ sequences, check how it handles organs smaller than a patch, and compare the frozen-encoder masks against an nnU-Net trained on the same local labels. For two related pieces on this site, see our posts on MedSAM2 for hip MRI anatomy and AI knee cartilage pre-segmentation.
Sources
- D’souza, N., Yang, Z., Singla, N., Doshi, P., Koos, K., Tan, F., Nunes, B., Rusko, L., Kolozsvari, B., Sabo, E., Valeriano, E., Gupta, V., Zare Estakhraji, S. I., Lebel, M., Haddadpour, F., Kass-Hout, T., Bas, E. A4-MR: Abnormality-aware Anatomical Alignment for MR. arXiv:2610.13056, 2026. https://arxiv.org/abs/2610.13056. PDF: https://arxiv.org/pdf/2610.13056.
- Code and weights: none found on GitHub or Hugging Face, checked 12 October 2026.
- Yang, Z. et al. Decipher-MR: A Vision-Language Foundation Model for 3D MRI Representations. arXiv:2509.21249.
- Ji, Y. et al. AMOS: a large-scale abdominal multi-organ benchmark (AMOS22). CirrMRI600+ liver cirrhosis MRI dataset, as cited in the paper.
- Featured image: Fig. 4 of arXiv:2610.13056 (external knee MRI case, sagittal slices C, D and E), as published.