Whole-body PET/CT is joint reading: metabolic uptake on PET, anatomy on CT, and the hard part is tying both to the same organs across the scan. Most 3D medical vision-language models still train on regional CT. They miss the systemic metabolic story and the organ-level grounding a clinic needs when a report mentions FDG-avid nodes beside a liver mass. MetaStructAtlas hangs co-registered whole-body PET and CT volumes with organ masks and radiology text that points back to those regions. MetaStructVQA then asks grounded questions against that evidence. Chenguang Zheng, Le Xue, Yichi Zhang, Wenbo Zhang, Zehui Ling, Gang Feng, Xin Gao, Yuan Qi, Yuan Cheng, Zixin Hu, and Mei Tian (Fudan / Shanghai Academy of Artificial Intelligence for Science / Shanghai Universal Medical Imaging Diagnostic Center) posted arXiv:2609.03690 around 3 September 2026. Dataset page: Hugging Face withtst/MetaStructAtlas (HTTP 200 on 5 September 2026). Code: github.com/withtst/MetaStructVQA (HTTP 200 on 5 September 2026).
If you ship a custom DICOM viewer for nuclear medicine, this is a research resource for PET+CT overlay rails with organ grounding and multimodal Q&A, not a CT-only regional VLM demo.
What you hang on the scan
Each case is a co-registered 3D PET volume and 3D CT volume from whole-body 18F-FDG PET/CT, plus organ-level masks and a structured radiology report. Masks start from TotalSegmentator on CT (103 standardized anatomical structures per subject), then denser 3D parcellation into 305 fine-grained sub-structures. Report phrases are anchored to those 3D regions so morphological CT findings and FDG uptake statements sit on the same organs rather than floating as ungrounded captions.
MetaStructVQA turns that atlas into 100,565 QA pairs across three tiers. Tier 1 is foundational grounding: organ ID given a highlighted mask, plus CT-versus-PET modality ID. Tier 2 is mask-guided characterization: morphological descriptors on CT and FDG metabolism descriptions on PET/CT. Tier 3 strips the mask for harder localization and adds report-level synthesis (attributes, SUVmax-style quantities, spatial relations, negative findings). Hard-negative distractors keep the choices anatomically close or clinically plausible so models cannot coast on language priors alone.
Dataset scale in plain numbers
MetaStructAtlas holds 490 retrospectively collected, co-registered whole-body PET/CT volumes (219 female, 271 male; mean age about 61 years) from Shanghai Universal Medical Imaging Diagnostic Center, 2021 to 2024, imaged on a Siemens Biograph 64. The abstract cites 50,470 organ-level segmentation masks and grounded radiology reports. The authors also report 7,265 explicit anatomical entities from expert reports pinned to 3D coordinates. MetaStructVQA averages about 205 QA pairs per subject. Roughly three-quarters of Tier 2/3 samples describe normal physiology; about one-quarter carry abnormal descriptors such as nodules, masses, or hypermetabolic uptake, which matches an oncologic screening mix more than a lesion-only corpus.
On MetaStructVQA, current 3D medical VLMs stay well below clinical thresholds. Organ identification with a mask (Type-1a) sits near random for the Hulu series (about 25 to 26 percent versus a 25 percent chance baseline). Hulu-32B reaches 67.76 percent on comprehensive Type-3 questions and leads several metabolic tasks, which is useful as a ceiling check, not a green light for deployment.
Where it fails / what to check before you wire it in
This is a research dataset and benchmark. It is not a cleared clinical product. Annotations are organ-level, not lesion-level, so fine tumor borders, sub-centimeter nodules, and lesion-burden scores are out of scope. Masks are derived on CT and transferred to PET by co-registration; PET-native segmentation would be tighter where resolution and functional-anatomical shift matter. The tracer is 18F-FDG only. PSMA, DOTATATE, and other pathways are not covered. Reports were authored and senior-reviewed in a nuclear medicine workflow, then structured with LLM-assisted extraction under human audit; still treat labels as research-grade. Confirm license and access terms on the Hugging Face dataset card before any commercial or clinical use.
How this lands in a viewer
Treat MetaStructAtlas as a PET+CT fusion and grounding sandbox for a custom viewer stack. Load the paired NIfTI (or your DICOM-derived volumes), hang CT anatomy with the organ mask overlays, fuse PET uptake, and surface MetaStructVQA-style questions whose answers must cite the highlighted region or fail closed. Use Tier 1 for organ/modality UI checks, Tier 2 for mask-guided morphology and FDG readout panels, and Tier 3 when you want the model to localize without a pre-drawn mask. Fail closed when PET and CT are not co-registered, when the organ mask is missing or wrong laterality, when the tracer is not FDG, or when the question asks for lesion contours the atlas does not provide.
MedPixel hangs phrase or loose-box prompts into overlays and sentences on 2D medical images. This paper is whole-body PET and CT in, organ masks plus grounded answers out for 3D nuclear medicine reading.
Rebuild from arXiv:2609.03690. As of 5 September 2026 the abstract and PDF respond (HTTP 200). Hugging Face dataset and GitHub code links above also respond (HTTP 200). The preprint is the source of record until a camera-ready version exists.
Sources
- Zheng, C., Xue, L., Zhang, Y., Zhang, W., Ling, Z., Feng, G., Gao, X., Qi, Y., Cheng, Y., Hu, Z., Tian, M. MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT. arXiv:2609.03690, posted ~3 September 2026. https://arxiv.org/abs/2609.03690 (HTTP 200 on 5 September 2026). PDF: https://arxiv.org/pdf/2609.03690 (HTTP 200 on 5 September 2026). Dataset: https://huggingface.co/datasets/withtst/MetaStructAtlas (HTTP 200 on 5 September 2026). Code: https://github.com/withtst/MetaStructVQA (HTTP 200 on 5 September 2026).