Hang a structured chest or abdominal CT report, abnormality answers, and reviewable radiologist-style reasoning beside the volume, then keep asking follow-ups without losing 3D spatial context. That is the clinic ask Myronenko, Yang, Tang, and colleagues take on in arXiv:2609.27511, posted around late September 2026. The paper is NV-Reason-CT: 3D Visual Language Model for CT Analysis, from NVIDIA with NIH and other clinical collaborators. The stack couples a native 3D vision transformer (COLIPRI-initialized Primus) with Qwen3.5-4B, and it passes every visual token plus explicit depth, height, and width coordinates into the language model without aggressive spatial merging.
CT interpretation still asks a viewer or clinic AI rail to connect findings across hundreds of slices, name laterality and extent, and write a coherent impression. Many CT VLMs compress the volume into a short token sequence before the LLM sees it. NV-Reason-CT keeps a 24 x 24 x 24 grid (13,824 tokens) indexed in 3D through multimodal rotary positional encoding, then trains with radiologist-guided reasoning plus Group Relative Policy Optimization (GRPO) on verifiable abnormality rewards.
What hangs on the viewer
Upstream is a regional chest or abdominal crop from a volumetric CT. Anatomy-aware cropping uses a detected lung base to place a 192 x 192 x 192 voxel cube at 2-mm isotropic spacing (384 mm per side). One crop is processed at a time. Downstream outputs are a structured Findings and Impression report, Yes/No answers to abnormality and anatomy questions, multi-turn follow-ups, and a reviewable reasoning span that lists observations, differentials, and uncertainty before the final answer.
For a DICOM viewer or CT AI shop, the hangable pieces are the regional crop provenance on the study, the structured report series or side panel, abnormality badges derived from prompted answers or report-derived labels, and a reasoning drawer the reader can audit. Interactive questions stay on the same volume tokens, so a follow-up about a lesion should still see the same 3D coordinates.
How it works in plain words
The vision encoder maps the crop to a full 24 x 24 x 24 token grid. A learned projector sends every token to Qwen3.5-4B without further spatial pooling. Grid dimensions travel with the tokens so the decoder can assign (d, h, w) indices. Training uses a curated corpus of about 550,000 multimodal instruction examples from 70,111 unique CT image inputs drawn from CT-RATE, CancerVerse, and an internal NIH set. Tasks mix standardized reports, abnormality and anatomy questions, multi-turn dialogue, refusal examples, and radiologist-authored narrations that also guide report-grounded synthetic reasoning.
Stage one is end-to-end supervised fine-tuning (SFT) of encoder, projector, and language model together. Stage two is GRPO: the policy samples structured reports, scores finding-set agreement and report structure with region-specific rewards, and updates from relative advantages across samples. Classification at inference does not need a separate head; abnormality presence is read from Yes/No generation (and AUROC from Yes/No token probabilities when ranking is needed).
What the numbers say
On the corrected CT-RATE validation cohort (3,002 chest reconstructions, 18 original abnormalities), NV-Reason-CT reaches macro-F1 0.614 and macro-AUROC 0.871 without a task-specific classifier. Report generation on the same split yields report-derived macro-F1 0.592 from Findings and Impression text alone. Figure 6 of the paper also places the model at macro-F1 0.766 on Merlin abdomen and 0.531 on the 16-label RAD-ChestCT mapping under their stated protocols.
A preliminary within-subject reader study with two US board-certified radiologists (10 cases: 5 chest, 5 abdomen; 2 normal, 8 abnormal) compared unaided review to review with the model reasoning and structured report. Mean reported interpretation-plus-reporting time fell from 26.25 min without AI to 13.13 min with AI, a 50% cut. Readers could copy and edit the AI report. The authors frame this as perceived quality and efficiency, not a diagnostic accuracy trial.
Where it fails and what not to trust
Like other generative VLMs, the model can invent findings or explanations that the image does not support. Reviewable reasoning is not a guarantee that the text matches the model’s internal computation. The crop heuristic can pick a bad field of view when lung coverage is limited or unusual; findings outside the selected 384-mm cube never reach the language model. Training does not cover the full mix of scanners, protocols, institutions, and rare disease, and direct expert narrations are a limited seed for a much larger synthetic expansion.
Clinical indications, labs, history, and priors are not systematically conditioned in the reported evals. The reader study is small (n=2 readers, 10 cases) and measures perception and self-reported time. Automated scores depend on label ontologies and report extractors that inherit report ambiguity. This is research software with released weights and code, not a cleared medical device.
For a viewer or clinic AI shop
Wire regional chest or abdomen CT in; hang structured report text, abnormality answers, and a reasoning panel out, with crop bounds shown so readers know what the model actually saw. Prefer keeping interactive follow-ups on the same 3D token grid rather than re-pooling the volume for each question. Surface refusal behavior for non-CT or mismatched inputs instead of forcing a reading.
If you already ship a CT viewer, natural hang points are a side-by-side report draft with provenance naming NV-Reason-CT, clickable abnormality chips tied to prompted answers, and a gate that forces human edit before chart export. Rebuild thresholds and crop QA on your own scanner mix. Audit hallucinations against your readers before you advertise hand-off. Model and training code are public: GitHub NVIDIA-Medtech/NV-Reason-CT, weights on Hugging Face nvidia/NV-Reason-CT.
Rebuild from arXiv:2609.27511. PDF: https://arxiv.org/pdf/2609.27511.
Sources
- Myronenko, A., Yang, D., Tang, Y., Turkbey, B., Simon, B., Harmon, S., Makwana, R., Aboian, M., Azamat, S., Hamamci, I. E., Er, S., Menze, B., Edgar, M., He, Y., Guo, P., Xu, D. NV-Reason-CT: 3D Visual Language Model for CT Analysis. arXiv:2609.27511, 2026. https://arxiv.org/abs/2609.27511. PDF: https://arxiv.org/pdf/2609.27511. Code: https://github.com/NVIDIA-Medtech/NV-Reason-CT. Model: https://huggingface.co/nvidia/NV-Reason-CT.
- Public and related benchmarks cited in the paper include CT-RATE, Merlin Abdominal CT, RAD-ChestCT, and COLIPRI (vision encoder initialization). Training recipe: end-to-end SFT then GRPO with region-specific abnormality and report-structure rewards.
- Preliminary reader study details from the paper: 2 radiologists, 10 cases, mean time 26.25 min unaided vs 13.13 min AI-assisted (~50% reduction). Treat as perception and efficiency signal only.