Hang a colonoscopy frame or a prostate MRI slice and look at the structure you care about. GazeRefine turns those sparse, duration-weighted fixations into a dense overlay without segmentation labels, fine-tuning, adapters, or gradient updates. Mohammed Oussama Benyahia and colleagues (Université Sorbonne Paris Nord, Northwestern University, VSB-Technical University of Ostrava) posted arXiv:2609.01310 on 1 September 2026. The paper is GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation. Code is public at github.com/MohammedOussamaBEN/GazeRefine (HTTP 200 when checked on 2 September 2026).
Inference is a frozen DINOv3 ViT-L/16 pass plus a short prototype loop. There is no SAM prompt encoder and no task head to retrain when the organ or modality changes. Gaze is the prompt. The overlay grows past the pixels the eyes hit, then stops when prototypes converge or after a fixed iteration budget.
Fixations become foreground and background anchors
The image is mapped to patch embeddings from a frozen DINOv3 backbone at 592×592 (37×37 patches, 1024-D). In parallel, each fixation (x, y, duration) builds a duration-weighted Gaussian gaze map. That map is normalized onto the patch grid and split into complementary foreground and background weights. Soft pooling of the patch features under those weights yields the initial foreground and background prototypes.
Those two vectors are the starting anchors. Everything after that is refinement in the same frozen feature space. No mask is ever shown to a trainable network.
Five refinement steps, then a 0.5 threshold
At each iteration the method scores every patch by cosine similarity to the current foreground prototype minus similarity to the background prototype, then floors negative scores at zero. Scores propagate to each patch’s k nearest neighbors in feature space (k = 5, τ = 0.1). Soft foreground weights update the prototypes. To limit semantic drift, the updated prototypes are blended back toward the original gaze anchors with λ = 0.5.
Refinement stops at T = 5 or when both prototypes move less than ε = 10⁻⁶. The final score map is upsampled to image resolution, min-max normalized, and thresholded at 0.5. Hyperparameters are shared across both benchmarks except Gaussian spread: σ = 2.0 on colonoscopy, σ = 0.5 on prostate MRI.
Where the overlays held, and where they did not
Evaluation follows GazeMedSeg gaze annotations on Kvasir-SEG (1,000 colonoscopy images, 5 to 99 fixations after dropping sub-50 ms and out-of-bounds points) and NCI-ISBI (789 prostate MRI slices, 6 to 54 fixations). GazeRefine is scored only on the held-out test splits used by the trainable baselines.
On Kvasir-SEG, mean Dice is 88.10% ± 0.14, close to a fully supervised nnU-Net trained on 90% of the labeled data (88.41% ± 0.47) and above the zero-shot gaze SAM baseline GazeMedSAMv2 (85.19% ± 0.13). On NCI-ISBI, GazeRefine lands at 80.28% ± 0.11, behind GazeMedSAMv2 at 85.62% ± 0.10. The authors point to low-contrast prostate boundaries that sit less cleanly in a general-purpose feature space, while polyps often carry stronger texture cues. Ablation is blunt: drop background cleaning and Dice collapses to 28.70% on Kvasir and 13.50% on NCI-ISBI. Affinity propagation matters more on prostate MRI than on polyps.
How this lands in a viewer
If you already hang endoscopy or prostate MRI in a DICOM viewer, treat GazeRefine as an interactive overlay rail driven by gaze, not by box or scribble prompts. Load the frame, capture fixations from an eye tracker (or a calibrated proxy), run frozen DINOv3, refine prototypes, hang the binary mask. Fail closed when gaze hardware is missing, when fixations fall outside the FOV, when the structure is low-contrast relative to surrounding tissue in DINOv3 space, or when you need multi-class 3D volumes rather than a single 2D binary mask. Do not treat the Kvasir number as a device claim for prostate MRI, and do not skip background cleaning in a production loop.
SLIP is interactive 3D segmentation with latency and undo. SAT3D is tumour plus uncertainty in Slicer. MedPixel hangs overlays from a phrase or a loose box. This paper is clinician gaze in, training-free overlay out, on a frozen vision transformer.
Rebuild from arXiv:2609.01310 and the public GitHub repo. As of 2 September 2026 the repository responds. The preprint is the source of record until a camera-ready version exists.
Sources
- Benyahia, M. O., Tliba, M., Kerkouri, M. A., Yousra, T., Wang, B., Bengtsson, M., Durak, G., Keles, E., Ming, Z., Penhaker, M., Beghdadi, A., Bagci, U., Chetouani, A. GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation. arXiv:2609.01310, posted 1 September 2026. https://arxiv.org/abs/2609.01310. Code: github.com/MohammedOussamaBEN/GazeRefine (HTTP 200 on 2 September 2026).
- Wang, B., et al. GazeMedSeg. Gaze-annotated medical segmentation benchmarks used for Kvasir-SEG and NCI-ISBI fixations (cited as [36] in the paper).
- Siméoni, O., et al. DINOv3. arXiv:2508.10104, 2025. https://arxiv.org/abs/2508.10104
- Jha, D., et al. Kvasir-SEG: A Segmented Polyp Dataset. Multimedia Modeling 2020. DOI: 10.1007/978-3-030-37734-2_37
- NCI-ISBI 2013 Challenge. Automated Segmentation of Prostate Structures. TCIA NCI-ISBI 2013.
- Isensee, F., et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 2021. DOI: 10.1038/s41592-020-01008-z
- Kirillov, A., et al. Segment Anything. ICCV 2023. arXiv:2304.13844