For years, much of radiology AI has centered on chest X-rays. That made sense: X-ray datasets were easier to release, benchmarks were more mature, and early multimodal workflows fit naturally around 2D imaging.
But CT is where much of the clinical complexity actually lives. That is why Merlin, a computed tomography vision-language foundation model published in Nature, is worth a close look. The team released code, model weights, and a documented dataset access path — something the community can inspect, install, and test, not only cite. The generic explainer for what radiology AI does in a clinic is artificial intelligence in radiology; this page is the Merlin release.
Why CT-native models matter
CT is fundamentally different from the 2D setups that shaped the first wave of medical vision-language models. Instead of a single projection, CT gives a volumetric view of anatomy and pathology: richer spatial structure, more organs and tissue types, heavier reporting, and much heavier demands on architecture and data infrastructure. Stretching a 2D radiology net into this setting is usually a limit. CT needs models that are native to 3D reasoning and that connect imaging with language in a structured way. Merlin is presented as a CT-native multimodal foundation model, not a 2D system stretched into a volume.
What Merlin is
According to the paper and project materials, Merlin is a 3D vision-language foundation model for computed tomography trained on CT scans, radiology reports, and structured electronic health record data. The released materials reference support for image-text embeddings, phenotype classification, five-year disease prediction, and radiology report generation. That framing is infrastructure for CT-centered workflows, not a one-off demo.
What has actually been released
The project is not just a paper. It ships with several tangible assets:
| Asset | Where |
|---|---|
| Code, setup, demos | GitHub StanfordMIMI/Merlin |
| Installable package | merlin-vlm on PyPI |
| Model weights | Hugging Face stanfordmimi/Merlin |
| Merlin Abdominal CT Dataset | Stanford AIMI (data-use agreement) |
People can inspect the codebase, review the inference setup, test the package, and evaluate whether the released resources are actually usable. That is the kind of release the medical imaging community can engage with in a practical way — including teams working on machine learning in medical imaging.
The dataset
The Merlin Abdominal CT Dataset is described as 25,494 CT scans from 18,317 unique patients, with paired radiology reports plus metadata and task files. Project docs also reference splits, report findings, zero-shot disease classification labels, five-year disease prediction labels, and demographic or acquisition metadata. That is a model-and-dataset stack, which is what serious progress in medical imaging AI usually requires.
The dataset is not instant-download open. Access goes through a Stanford AIMI data use agreement. Researchers still get a real, documented path — more actionable than a paper that cites private training data with no release plan.
Why this matters now
Medical AI is moving past the first wave of 2D radiology benchmarks toward richer multimodal infrastructure. Related efforts such as CT-RATE and RadGenome-Chest CT invest in 3D imaging benchmarks, paired image-language resources, and modality-specific dataset ecosystems. Merlin fits that trend: a foundation-model-style resource built for CT, not a narrow single-task net.
Progress here is not only model design. It also depends on large multimodal datasets, reproducible evaluation, released code and checkpoints, and enough transparency for others to test what was actually built. Merlin contributes across those dimensions, including for teams thinking about radiology AI workflow automation and how CT-native systems fit a product. Building or deploying a medical imaging AI product? The Medical AI Deployment Checklist covers validation, infrastructure, workflow integration, and launch readiness.
What to actually test
A lot of medical AI announcements are hard to act on. Merlin is different because it gives concrete checks: can the package be installed? Can the inference demos run? How accessible are the weights? How usable is the dataset request flow? What downstream tasks can a team realistically reproduce? That is infrastructure, not marketing.
A note of caution
Merlin is promising, but “open” in medical AI comes with constraints. The dataset is governed by a data-use agreement. Clinical data access is restricted. Practical usability depends on how smooth setup and inference really are. Benchmark strength is not clinical readiness. CT foundation modeling is not solved. The field now has a more serious, shareable, testable CT-native foundation-model release than it had before. That is already a step.
What Merlin signals
It points toward radiology foundation models that are modality-native instead of loosely adapted from 2D, multimodal by design, backed by paired imaging, reports, and metadata, and released with enough infrastructure for other people to do real work. If that continues, the next wave is less isolated single-task models and more general-purpose imaging backbones for classification, report generation, retrieval, forecasting, and decision support. It also sits next to the shift toward radiology AI agents inside real DICOM workflows.
Sources
- Merlin: a computed tomography vision-language foundation model and dataset — Nature
- StanfordMIMI/Merlin — GitHub
- stanfordmimi/Merlin — Hugging Face
- Merlin Abdominal CT Dataset — Stanford AIMI
- CT-RATE: Generalist Foundation Models from a Multimodal Dataset for 3D Medical Imaging
- RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT
PYCAD builds the imaging side of this — custom pipelines and web DICOM viewers when the study has to live in a clinic app. Case studies.