Hang a polyp endoscopy frame or a dermoscopic lesion and type a short instruction such as “Segment the polyp region using red.” InstEditSeg runs latent diffusion editing on a Stable Diffusion backbone with DINOv3 multi-scale feature guidance, then returns the same frame with a color-coded overlay painted on the original background. Ziquan Liu, Zhewei Zhu, and Xuyang Shi (Southwest University of Science and Technology) posted arXiv:2609.02004 around 2 September 2026. The paper is InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation. Code: https://github.com/wincharm001/InstEditSeg (HTTP 200 on 6 September 2026).
Most segmentation rails emit a standalone binary mask. InstEditSeg treats segmentation as instruction-driven image editing instead. The edited output stays close to the natural-image prior of latent diffusion models, which the authors argue shrinks the natural-to-medical domain gap and keeps secondary lesions that discriminative decoders often drop.
Instruction plus frame, color overlay on the same pixels
Training builds image-text samples from segmentation annotations: a textual instruction and a randomized overlay color. At inference the model takes the medical frame and the instruction, denoises with a DDIM scheduler, and writes the color region onto the original image. A DINOv3 visual encoder feeds a DINO Feature Guidance Block that builds a multi-scale pyramid. The pyramid is fused into the diffusion U-Net by channel concatenation and zero-initialized convolution so hierarchical discriminative priors enter without wiping pretrained weights. Dual-branch classifier-free guidance needs only two forward passes per denoising step.
Evaluation covers polyp sets (Kvasir-SEG, ETIS, CVC-ClinicDB, CVC-ColonDB) plus unseen PolypGen, and skin sets (ISIC2016 in-domain, ISIC2017 held out). Overlay-on-original is compared against binary masks and color masks on a black background; the overlay formulation wins those ablations by a wide margin.
Where the numbers landed
On unseen PolypGen, InstEditSeg reports the best Dice of 83.92% and IoU of 77.50%. On unseen ISIC2017 it reports the best Dice of 83.14% (IoU 75.62%). On the 119 PolypGen test images that contain multiple polyps, it reaches 75.93 Dice versus EMCAD at 63.18 (+12.75). In-domain it stays competitive: 92.10 Dice on Kvasir-SEG and 92.83 on CVC-ClinicDB, while EMCAD remains ahead on some in-domain sets (for example 93.74 Dice on Kvasir-SEG). Five DDIM steps reach about 91.78 Dice at 711 ms per image on Kvasir-SEG (A100); 25 steps reach 92.10 Dice at 3.58 s. EMCAD sits at 218 ms for 93.74 Dice, so the generative path is slower when latency is the hard constraint.
How this lands in a viewer
If you already hang colonoscopy or dermoscopy frames in a DICOM viewer, treat InstEditSeg as an instruction-to-overlay rail. Pass the frame plus a short color instruction, run the diffusion edit with DINOv3 guidance, and hang the returned color overlay on the same pixels for review. Fail closed when the instruction asks for unsupported attribute or spatial modifiers (the paper flags “largest only” and “rightmost”), when an unseen color request binds the wrong region, or on latency-critical paths where a discriminative pass such as EMCAD is required. Color sensitivity is a known limit of the paradigm.
FreNet quiets cluttered lesion backgrounds before encoding and returns a cleaner contour. This paper is text instruction plus frame in, color-coded overlay on the original pixels out.
Rebuild from arXiv:2609.02004. As of 6 September 2026 the abstract, PDF, and GitHub repo respond (HTTP 200). The preprint is the source of record until a camera-ready version exists.
Sources
- Liu, Z., Zhu, Z., Shi, X. InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation. arXiv:2609.02004, posted ~2 September 2026. https://arxiv.org/abs/2609.02004 (HTTP 200 on 6 September 2026). PDF: https://arxiv.org/pdf/2609.02004 (HTTP 200 on 6 September 2026). Code: https://github.com/wincharm001/InstEditSeg (HTTP 200 on 6 September 2026).