GRACE: Fixing Segmentation Masks With Retrieved Verified Cases

Fig. 5 of arXiv:2611.23189 by Marcianò et al.: liver, pancreas and spleen CT and prostate peripheral zone MRI slices showing the ground-truth mask, a synthetically damaged mask and its GRACE correction, and a MedSAM2 mask and its GRACE correction.

Most segmentation pipelines in a clinic end with a person fixing masks. The model leaks into a neighbouring organ, drops a few slices or leaves a hole, and someone in the viewer repaints it before the volume or the contour can be used. Automatic correctors exist, but they are usually built for one organ, so a platform with forty structures would need forty of them. GRACE, from EURECOM, Sorbonne Université and King’s College London (arXiv:2611.23189, published in Transactions on Machine Learning Research, 10/2026), trains one refiner for every structure. For each case it looks up a verified mask of the same organ in a library of past cases and uses it as a guide.

What it does

GRACE sits after your segmentation model. It takes the image, the mask the model produced and the name of the target structure. It searches a knowledge base of verified cases for masks of that same structure that look similar, picks the closest one and hands it to the refiner as a shape prior. The refiner returns a corrected mask. It never sees the upstream model’s weights, only its output, so it can sit behind a vendor model, an in-house nnU-Net or a promptable tool such as MedSAM2.

The organ-specific knowledge lives in the library. To support a new structure, you add verified cases of it to the index. The refiner is not retrained.

How it works

  • The library. Each verified case is drawn as red contours over its scan, paired with a short text prompt naming the structure, and embedded with the frozen MedGemma 4B encoder. A small projection head, trained so that similarity in its space tracks how well two mask shapes overlap, turns that embedding into the search key, and a FAISS index holds the keys. In the reported setup the library holds 3,913 per-structure cases from the training splits.
  • Retrieval. The search is restricted to cases of the same structure, so a liver query only ever gets a liver back. The top match becomes the exemplar.
  • The refiner. Three variants share that interface. GRACE-U is a 3D U-Net with three input channels: the mask to fix, the image and the retrieved mask. GRACE-M does the same in the latent space of the frozen MAISI autoencoder with a mask-tuned decoder. GRACE-M-D adds flow matching, which gives several plausible corrections per case and so a measure of uncertainty.
  • Training without real failures. Pairs of broken and correct masks are rare, so the authors damage ground-truth masks on purpose: erosion, dilation, warped boundaries, boundary noise, false-positive blobs and dropped slices, at several severities. Training on corrupted ground truth rather than on one model’s mistakes keeps the refiner from learning a single segmentor’s habits. The exemplar channel is blanked for 10% of training samples so the refiner still works when retrieval finds nothing.

The training corpus covers 15 public CT, MRI and ultrasound datasets, 3,866 volumes and 38 structures across abdomen, thorax, pelvis, brain and heart.

What they found

The authors test GRACE on the output of seven real segmentors, grouped by how good they already are. On a deliberately weak 3D U-Net, with an input Dice of 0.386, GRACE-U adds 0.353. On TotalSegmentator, whose CT organ masks start at 0.932, the average change is close to zero. MedSAM2 sits in between. Simple morphological clean-up barely moves any of them, and a single-anatomy autoencoder prior (PostDAE) makes every tier worse.

The featured image is Fig. 5 of the paper. In each row the left pair shows a mask broken by the training corruptions and its correction; the right pair shows a real MedSAM2 output and its correction. In the pancreas row, a partial MedSAM2 mask comes back as a coherent organ.

Three held-out datasets that never enter training or the library (ProstateX prostate MRI, CHAOS abdominal CT and LUMIERE glioblastoma follow-up MRI) improve for every segmentor tested. When a whole organ is removed from training and then supplied only as 118 to 302 verified cases in the library, the frozen refiner still improves it. With the exemplar channel switched off, the authors measure that the retrieved mask accounts for 40% of the improvement at the most severe corruption.

Why a viewer or clinic team should care

If you run segmentation inside a DICOM viewer or a review platform, the expensive part is the correction step, and it is the same chore for every organ. GRACE fits that slot as a single service: model output in, corrected mask out, with the retrieved case available to show the reviewer why the mask changed. Every mask a reviewer approves can go back into the library, so the corrector gets better references over time without a training run. The authors name four settings where refinement pays off: structures without a strong off-the-shelf segmentor, black-box vendor models, new structures that need support without retraining, and promptable tools fed with imperfect prompts.

It is also cheap at inference. GRACE-U takes 0.31 seconds per case on one NVIDIA A100, so it can run on every study rather than on a sample. The public package includes an input quality gate that only refines cases whose estimated quality is below a threshold you choose, which is how we would deploy it: leave confident masks alone and send the doubtful ones through the refiner and then to a human.

Limits and failure modes

  • Good masks can get worse. Among cases whose input Dice is above 0.8, 3.8% lose more than 0.05 after refinement. On a strong CT segmentor the authors claim no improvement, which is why a gate in front of it matters.
  • It does not cross modalities on its own. A refiner trained without MRI, with a library of CT livers, improves MRI livers far less than one trained with MRI. A refiner trained without ultrasound makes cardiac ultrasound masks worse, even though retrieval still picks the right structure.
  • New organs need examples. The leave-one-organ-out test needed 118 to 302 verified masks per organ, and all four were solid organs or organ zones. This is index extension, not zero-shot.
  • The wrong label hurts. In a fully automatic pipeline, a wrong structure name pulls the wrong exemplar; for the pancreas a wrong-class exemplar is slightly worse than none.
  • The MedSAM2 and ScribblePrompt results use an oracle bounding box from the ground truth, so real prompts would give worse starting masks.
  • The flow-matching variant needs about seven times the compute of GRACE-M and its output depends on the random seed. No reader study or clinical endpoint was measured.

Code and licence

The code is public at github.com/robustml-eurecom/grace under the MIT licence: the refiner, the corruption schedule, the retrieval index, the evaluation protocol with a harm analysis, and the quality gate, with a CPU quickstart on a synthetic corpus. When we checked on 18 November 2026 the repository listed no pretrained checkpoint or prebuilt knowledge base to download, although the paper says these are released, so expect to train the refiner (the README asks for a GPU with 16 GB or more) and build your own library from your verified cases. The packaged retrieval ranks neighbours by a coarse 16-voxel shape embedding rather than the MedGemma pipeline described in the paper, and you can plug in your own vectors. The paper was peer reviewed at TMLR; the arXiv copy is released under arXiv’s non-exclusive distribution licence. All 18 datasets used are public.

On the blog, see also Taurus, an LLM reviewer for aortic CT annotation, MedSAM2-Anatomy on hip MRI without retraining and quality control in radiology.

Source

Paper: Vincenzo Marcianò, Xiaoming Zhang, Michela Antonelli, Sébastien Ourselin and Maria A. Zuluaga, “Retrieval-Augmented Correction for Generalist Medical Image Segmentation”, Transactions on Machine Learning Research (10/2026), OpenReview, arXiv:2611.23189 (cs.CV, 17 November 2026). Affiliations: EURECOM, Sorbonne Université, King’s College London. Featured image: Fig. 5 of the paper, four of five rows and five of nine columns, panels cropped and resized only, with the paper’s Dice and HD95 overlays trimmed.

We build custom medical imaging platforms — advanced DICOM viewers, AI segmentation, and the clinical systems around them.

Copyright © 2026 PYCAD. All Rights Reserved.