Hang a PET volume, a CT volume, and the clinical report beside the same head-and-neck case, and most fusion stacks still wait until after patch embedding or pooling before the modalities talk. CoMLP keeps the conversation at high spatial resolution with cooperatively gated MLPs instead of dense cross-attention, then hangs primary-tumor and lymph-node overlays. Mingyuan Meng, Shuchang Ye, Mingjian Li, Zhenyu Zhao, Jinman Kim, and Lei Bi (Shanghai Jiao Tong University / University of Sydney / China Unicom) posted arXiv:2609.04781 around 4 September 2026. The paper is CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation. No public code repository is linked in the preprint (checked 8 September 2026).
Viewer shops already know the PET/CT hang. The hard part is letting anatomy and metabolism (and the report) meet before downsampling erases the boundary you care about, without blowing GPU memory on pairwise attention over full-resolution tokens.
Same gate for PET/CT and for the report
CoMLP is a dual-path MLP block with cooperative cross-gating. The mainstream feature is split into two streams. The spatial gate for one stream is built from the other stream concatenated with the complementary modality, so PET can gate CT (and the reverse) without building an attention matrix. A regional branch handles local neighborhoods. A dilated branch links corresponding locations across regions for longer-range context. Region size R is 8 on the 3D stacks and 12 on the 2D ones.
That same block sits in two places. An inter-image fusion encoder runs CoMLP from the first, full-resolution feature level through coarser scales so PET and CT exchange detail before pooling. A vision-language fusion decoder encodes the report with BiomedBERT and re-injects it through Text-infused Semantic Guidance blocks that call CoMLP again at each upsample step. Training uses Dice plus binary cross-entropy on an NVIDIA A100 40 GB.
Where it was measured
OPC is 524 HECKTOR 2022 oropharynx cases with paired PET/CT, expert-written reports, and tumor/node labels (415/109 train/test by clinical center). NPC is 866 nasopharyngeal carcinoma cases with the same package (632/234 by center). Volumes are resampled to 1 mm isotropic and cropped to 160 cubed. PET becomes SUV; CT is clipped to [-1024, 1024] HU. Three public 2D sets test report-guided fusion alone: QaTa-COV19 chest X-rays (9,258), MosMedData+ infection CT slices (2,729), and Kvasir-SEG polyps (1,000), each with established text descriptions.
On OPC and NPC with PET, CT, and the report, CoMLP reaches 81.24% and 82.95% Dice (average 82.10%). Dropping the report but keeping PET/CT fusion still averages 80.51%, ahead of SwinCross, C2MAOT, and CIPA. On the three 2D language-guided sets the average Dice is 88.24%. Metrics sit in the body as supporting numbers; the practical win is that fusion stays usable at full resolution.
At a matched coarser resolution, multi-axis cross-attention needs about 38 GB train / 7.5 GB infer memory versus CoMLP’s 24 / 5.3 GB, with 0.98 versus 0.90 TFLOPs. That gap is what lets CoMLP keep interacting at the full-resolution level where the attention baseline becomes infeasible.
How this lands in a viewer
If you already hang PET/CT in a DICOM viewer, treat CoMLP as a fusion recipe for the overlay path, not a new worklist. Wire PET and CT as separate series, keep the report text as an optional prompt channel, and hang primary-tumor and node masks as editable overlays. Fail closed when you only have one modality, when the report is a free-text EHR dump that was never written against the PET/CT you are viewing, or when GPU memory forces fusion after aggressive downsampling and the boundary you need is already gone. The OPC/NPC reports in this paper were written by experts from the images without seeing the ground-truth masks; that is cleaner than many EHR notes and still not the same as your local dictation template.
MetaStructAtlas hangs whole-body PET/CT organ masks with grounded answers. This paper is paired head-and-neck PET/CT plus report in, tumor and node overlay out without waiting for dense attention after the stack has already been reduced.
Rebuild from arXiv:2609.04781. As of 8 September 2026 the abstract and PDF respond (HTTP 200). The preprint is the source of record until a camera-ready version exists. No public weights or repo are listed yet.
Sources
- Mingyuan Meng, Shuchang Ye, Mingjian Li, Zhenyu Zhao, Jinman Kim, Lei Bi. CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation. arXiv:2609.04781, posted ~4 September 2026. https://arxiv.org/abs/2609.04781 (HTTP 200 on 8 September 2026). PDF: https://arxiv.org/pdf/2609.04781 (HTTP 200 on 8 September 2026). No public code linked in the preprint as of 8 September 2026.