MedPixel takes a hung slice plus a phrase, a click, or a loose box and returns a mask overlay and a short sentence. One reserved token in the language model is projected into a SAM2 prompt embedding, and that is the hang path. Yang, Shi, Chen, Sun, Leng, Shi, Cai, and Jiang at Zhejiang University, Fudan University, Kuaishou, East China Normal University, and Shanghai AI Laboratory submitted arXiv:2608.09818 on 10 August 2026. The paper is MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation. The same backbone covers five interfaces: name the organ, describe it without naming it, click or box it, ask for a grounded explanation, or answer a medical question with no mask.
The GitHub page exists at github.com/yhy-whu/Medpixel. The README still says code and checkpoints will be released soon. Checked 27 August 2026: no training or inference code, no weights. The backbone is 2D or slice-based. The authors say volumetric and multi-turn use are untested.
Qwen2.5-VL, SAM2, one reserved token
The language side is Qwen2.5-VL at 3B or 7B. The mask side is a SAM2 decoder. LoRA (rank 128, alpha 256) sits on the q, k, v, and o projections of the language model and the Qwen vision tower. Base weights stay frozen. When the response needs a mask, the model emits a reserved token. Its hidden state is mapped by a learned projector into the SAM2 prompt space. SAM2 then writes the overlay from that embedding plus the image features.
A click or a box enters the same sequence through a second reserved token. A small prompt encoder turns the geometry into a sparse vector, a second projector lifts it into the language hidden size, and that vector replaces the token embedding before the language model runs. The phrase and the sloppy box share one sequence, so language can still name the target when the box is off.
Stage 1 is joint supervised fine-tuning: next-token loss plus a mask term (focal, Dice, IoU, objectness). Samples without a mask drop the mask term. Stage 2 is Pixel-Level Preference Optimization (PLPO). The Stage 1 model writes eight candidate replies on reasoning prompts. Each reply is decoded to a mask and scored against the ground-truth mask. Pairs are kept only if the winner is at least 0.30 Dice and beats the loser by 0.10. That produced 1,786 preference pairs. Direct Preference Optimization then updates the language adapters. The SAM2 branch and the segmentation projector stay frozen. Training hardware is an NVIDIA RTX PRO 6000. Stage 1 is one epoch, AdamW, cosine schedule, 3% warmup, sequence length 4096, bf16. Stage 2 is three epochs at 1 x 10-6.
444,297 records from existing masks, no extra LLM
MedPLG-440K is built from the BiomedParse training annotations. The authors do not hire an external language model to write the text. For each image-mask pair they measure centroid, relative area, aspect ratio, fill, connected components, contour regularity, foreground-to-background intensity, and intensity variance. Those numbers become seven descriptors: location, size, shape, components, boundary, contrast, and uniformity. Contrast wording follows the modality (hyperdense on CT, hyperintense on MRI, hyperechoic on ultrasound). Templates then fill four task types.
- T1 referring: 94,261 records. The prompt names the target ("Segment the liver.").
- T2 reasoning: 21,367 records. The prompt hides the name and describes function, anatomy, or appearance.
- T3 interactive: 318,670 records. A point or a tight box is sampled from the mask.
- T4 explanatory: 9,999 records. The reply mixes a visual description with a short clinical sentence and the mask token.
Unique records total 444,297. Stage 1 mixes those with public medical VQA and distilled clinical reasoning for 1,842,802 effective instances (64% VQA, 32% grounding after replication, 4% reasoning). T1 and T4 are sampled twice, T2 three times. The BiomedParse test split is held out. T1 uses 24,391 clips (one frame each). T2 and T4 use 2,500 samples each. T3 uses 2,000 (1,000 point, 1,000 box). T5 is 10,267 multiple-choice items (1,647 image, 8,620 text). Dice and NSD in the paper are percent, 5-pixel NSD tolerance. If you still mix the coefficient with the training loss, read the note on Dice versus Dice loss.
Five interfaces, one hang path
Table 1 is the in-domain table. Numbers below are paper units (percent) converted to the 0-1 overlap used on this site.
On T1 referring, MedPixel-7B is 0.850 Dice and 0.617 NSD against BiomedParse 0.800 and 0.550. On T2 reasoning, 0.667 against Citrus-V 0.375, the strongest pixel-language row that reports a T2 Dice. On T4 explanatory, 0.765 against UniBiomed 0.358. On T3 interactive (point and box averaged), MedPixel-3B is 0.760 and MedPixel-7B is 0.759. MedSAM2 on the same T3 column is 0.757. A click beats a box on this split: MedPixel-7B is 0.8016 on points and 0.7165 on boxes.
PLPO is the Stage 2 gain on the language-heavy tasks. T2 7B goes from 0.629 after SFT to 0.667 after PLPO. T4 goes from 0.756 to 0.765. T1 and T3 barely move. METEOR on T2 stays near 51. Target grounding accuracy on T2 is 70.8 after PLPO (72.5 after SFT). The paper treats that drop as small relative to the mask gain. On T5, MedPixel-7B is 47.5 image-MCQ and 58.1 text-MCQ against HuatuoGPT-Vision 42.0 and 46.7.
Zero-shot transfer uses no extra training. MeCoVQA-G+ (2,719 samples, eight modalities) is 0.411 slice-level and 0.474 modality-level Dice for MedPixel-7B against BiomedParse 0.358 and 0.440. HD95 falls from 224.3 / 177.1 to 152.3 / 123.0. MedPLIB-7B is 0.472 slice-level, but that model was trained on this benchmark. U-MRG-14K (2,480 implicit queries, ten modalities) is 0.3791 against MedReasoner-7B 0.3778. The MedReasoner row uses a language model to write a prompt for a fixed MedSAM2. MedPixel writes the mask in one model.
A loose box still hangs a mask
SAM-family models in this paper peak on a tight ground-truth box and fall when the box is scaled or shifted. MedPixel is flatter. On the 1,000-box T3 subset, MedPixel-7B is 0.7165 at scale 1.00 (tight) and 0.7532 at scale 2.00 (double width and height, center fixed). MedSAM at those two scales is 0.8397 then 0.4459. Joint shift-and-scale at r = 1.00 (center moved by up to one box width or height, each side scaled independently) leaves MedPixel-7B at 0.6234, a 9.31-point drop from the tight box. MedSAM drops 71.83 points to 0.1214. Severe contraction is the paper's own weak cell: if the box no longer covers the organ, the overlay can miss a lobe. A tight box used as the mask itself is 0.6618 Dice on that subset, so the model is not just painting the rectangle.
The authors note that MedPLG language is template text, not dictation. PLPO needs a ground-truth mask to rank replies. Explanations were not read by a clinician. Research overlap is not a device claim.
How this lands in a viewer
If you already hang AI in a DICOM viewer, treat this as a slice tool: a text field or a sloppy ROI, one reserved token, a SAM2 overlay, and a one-line caption. Keep a 3D organ model (nnU-Net, TotalSegmentator) on the native volume. Use MedPixel when the reader wants to name the structure in words, or when the box they drew is larger than the finding. The 2D / slice backbone does not write a native 3D mesh. Sagittal and coronal hang of a filled stack is a post-step. Public-set overlap is not a scanner-vendor claim.
RadYOLO is still the faster first-pass box. SAT3D is tumour plus uncertainty in Slicer. SLIP is click latency and undo. Ten-case MedSAM3 LoRA is a site adapter on a promptable SAM. CoM3eT is a frozen Swin plus a pyramid transformer. 3D-CurvSegFlow is three Euler steps on a vessel tree. CoInS-Net is two endpoints and an in-between plane. This paper is one language-mask token on a hung slice.
Rebuild from the protocol in arXiv:2608.09818. Do not wait on study weights. The GitHub README still says they are coming. Retrospective overlap on public slices is not a device claim. The preprint is the source of record until a venue version exists.
Sources
- Yang, H., Shi, M., Chen, Z., Sun, H., Leng, H., Shi, X., Cai, Y., Jiang, Y. MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation. arXiv:2608.09818. Submitted 10 August 2026. https://arxiv.org/abs/2608.09818 (HTML: https://arxiv.org/html/2608.09818). GitHub: https://github.com/yhy-whu/Medpixel. Repo exists; README states code and checkpoints will be released soon. No weights or training scripts found when checked 27 August 2026.
- Bai, S., et al. Qwen2.5-VL Technical Report. arXiv:2502.13923. arXiv:2502.13923
- Ravi, N., et al. SAM 2: Segment Anything in Images and Videos. ICLR 2025.
- Zhao, T., et al. BiomedParse: A Biomedical Foundation Model for Image Parsing. arXiv:2405.12971. arXiv:2405.12971
- Ma, J., et al. MedSAM2: Segment Anything in 3D Medical Images and Videos. arXiv:2504.03600. arXiv:2504.03600
- Yan, Z., et al. MedReasoner: Reinforcement Learning Drives Reasoning Grounding. AAAI 2026. (U-MRG-14K comparison row)
- Rafailov, R., et al. Direct Preference Optimization. NeurIPS 2023.
- Isensee, F., et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 2021. DOI: 10.1038/s41592-020-01008-z