Most CT AI we wire into viewers comes in two separate boxes. A segmentation network draws organs and lesions but has nothing to say about them. A vision-language model writes “small pericardial effusion” or “no thyroid lesion” but gives the reader no way to check where it looked. In arXiv:2611.06471, Yulu Xie, Fan Wang, Hao Luo and 14 co-authors from Alibaba DAMO Academy, Tsinghua University and five Chinese hospitals present HoloCT, a single 3D model that does both. You give it a CT volume and a text prompt, and it answers in text and, when asked, attaches a 3D mask to each finding it names.
The code and both checkpoints are released under MIT, so a product team can try it on its own scans without a licensing negotiation.
How it works
HoloCT reads each scan twice. A global encoder (adapted from the Med3DVLM DCFormer) sees the whole volume resized to 128 × 256 × 256 voxels for context. A detailed encoder sees the scan resampled to 1.5 mm isotropic spacing, cropped to the body, so small structures keep their shape. Tokens from both branches go straight into a Qwen3-VL language model, in a 4B and an 8B version.
When the prompt asks for grounding, the language model emits a special segmentation token next to the finding it is describing. That token drives a 3D mask decoder (initialized from VISTA3D) that also takes skip features from the detailed encoder. The result is text like “Consolidations in both lungs” with a mask file tied to that phrase.
One model covers four kinds of work:
- Segmentation from text. “Segment the liver” or a free-text finding like “peripheral ground-glass opacities in both upper lobes”.
- Lesion screening. Is there a lesion of type X, and if so, where. It can screen for several lesion types in one pass.
- Question answering. Multiple-choice and open questions about the scan, optionally with the supporting mask.
- Report drafting. Narrative or structured reports, with a mask per anatomical region or, on chest CT, per finding.
The training corpus, HoloDATA, holds 195,921 CT scans from 73 public datasets and six hospital cohorts, converted into 6.2 million instruction dialogues and 2.8 million CT and mask pairs over 246 structures and 50 lesion types. Training took about 3,500 H100 GPU hours for the 4B model and 4,500 for the 8B.
What the masks add
Asking for the mask also improved the text. When the authors removed the grounding request from screening prompts, mean macro F1 across 18 lesion types fell from 0.562 to 0.505 for HoloCT-8B. In the paper’s report examples, the grounded version corrected a middle-lobe atelectasis placed on the wrong side and picked up a sliding hiatal hernia the text-only report missed. Blocking the language model from attending to the fine-grained tokens dropped screening F1 from 0.678 to 0.061, so the text side leans heavily on the same spatial features the mask decoder uses.
The featured image comes from Fig. 6a of the paper: zero-shot lesion masks on clinical CT from three hospitals the model never trained on, HoloCT-8B against VoxTell, a text-prompted 3D segmenter. On laryngeal lesions, mean Dice went from 0.107 for VoxTell to 0.488 for HoloCT-8B. In a fine-tuned reader study on 141 thyroid scans with histopathology, HoloCT-8B reached 81.6% accuracy against 72.3% to 78.0% for four radiologists, though its specificity was only 65.6%.
Why a viewer or clinic team should care
For a team building a DICOM viewer or a clinical AI platform, the output format matters most. Every claim the model makes can come with a binary 3D mask, which is exactly what a viewer knows how to display. That supports a review workflow that free-text AI cannot offer:
- Click a sentence, see the region. Link each line of a draft report to its mask overlay, so the reader confirms or rejects the statement on the slices instead of trusting prose.
- One model behind several tools. Organ contouring, lesion screening and report drafting can share one deployment and one prompt interface, instead of a zoo of task-specific networks.
- Open-vocabulary contours. Because segmentation is driven by text, a user can ask for a structure the product never shipped a model for. Expect weaker results on those, as shown below.
You still have to build the plumbing. The demos take a NIfTI volume in Hounsfield units and write response.txt, a result.json that maps each mask to its label, and one mask_000.nii.gz per segmentation token. To hang the output in a PACS-connected viewer you need a DICOM to NIfTI step on the way in and a conversion to DICOM SEG or RTSTRUCT on the way out, as we described for NLST CT overlays. The README asks for an NVIDIA GPU with bfloat16 support and does not list memory needs or run times.
Where it falls short
- Finding-level masks are chest only. Grounded reports with a mask per finding exist only for chest CT. Abdominal reports ground to anatomical regions, not findings.
- Diffuse findings and bone lesions are weak. On free-text chest findings, mean Dice was 0.291, and only 0.252 for diffuse, non-focal findings. On unseen bone lesions it scored 0.154, below VISTA3D. Finding-grounded report F1 was 0.278.
- One volume at a time. No prior-study comparison, no multiphase reading, no link to the EHR, MRI or PET.
- Retrospective, one country. All clinical cohorts are Chinese, scored retrospectively. The reader study used a fine-tuned model on those cohorts, and colon cases had no reference masks.
- No uncertainty output. The authors say safe deployment still needs calibrated uncertainty and automated quality control, which the model does not provide.
Code, weights and license
Code is at github.com/alibaba-damo-academy/HoloCT under the MIT license, with six demo scripts (screening, VQA, two kinds of grounded reporting, target segmentation and finding-level segmentation) and single-case evaluation scripts. Weights are on Hugging Face as HoloCT-4B and HoloCT-8B, also MIT, not gated. The README credits Qwen3-VL, Med3DVLM and MONAI VISTA and points to a third-party notices file for their licenses. The README does not offer a HoloDATA download, and the six hospital cohorts are not public. The preprint is under arXiv’s standard non-exclusive license. Credit: Yulu Xie, Zilong Li, Shen Xu, Yifan Bai, Liuhong Shi, Jiao Chen, Xianglei Kong, Wenchao Zhu, Na Shen, Xianghua Ye, Yang Yan, Jiaju Zheng, Yunsong Peng, Yanzhou Su, Chaohui Yu, Hao Luo and Fan Wang. For related reading on PYCAD, see a CT report model that keeps 3D tokens intact and Merlin and CT-native foundation models.
My take
In HoloCT the mask is part of the answer, and the MIT license on the weights makes it easy to test. I would treat it as a research engine for a review UI: run it on a local batch, show each sentence with its overlay, and measure how often readers reject the mask. The weak spots on diffuse disease and bone, and the lack of uncertainty, mean it should draft text and point to regions while a radiologist makes the call.