VoxForge: One Frozen 3D Encoder for MRI, CT and PET

Fig. 6 of arXiv:2611.25309 by Anirudh and Thiagarajan: reference circle of Willis vessels, the SPECTRE result (Dice 49.2) and the VoxForge result (Dice 64.8) segmented from non-contrast CT, with correct, missed and extra vessel segments colour coded.

Most teams that build on a 3D medical foundation model end up fine-tuning it for every task, which means a separate copy of the weights per organ, per modality and per client. The promise of a foundation model is the opposite: one frozen encoder whose features you compute once and reuse for segmentation, classification and registration. VoxForge, from Rushil Anirudh and Jayaraman J. Thiagarajan at Amazon AGI (arXiv:2611.25309, preprint, 19 November 2026), tests that promise strictly. Every encoder in the paper stays frozen and only a small task head is trained on top.

The result is a single 388M-parameter vision transformer that, across MRI, CT and a modality it never saw in pretraining (FDG PET), has the best or tied-best average rank in every task group the authors scored. For anyone running a viewer or an analysis platform, one backbone with many light heads is the setup that holds up in production.

What VoxForge is

VoxForge is a plain ViT that cuts a volume into small 8 x 8 x 8 voxel tokens. A 160³ crop becomes 8,000 patch tokens plus one class token. The authors start from Primus-L, a ViT designed for 3D segmentation, and change three things: they add a class token, widen the encoder from 1056 to 1152 channels, and let the learned position table grow with the crop size. The patch embedding is the only convolution, so every head reads transformer tokens directly.

Training follows the DINOv3 self-distillation recipe. A student network sees two global and four local crops of a volume and learns to match an exponential-moving-average teacher. Some patch tokens are masked and the student predicts the teacher’s tokens there (iBOT). A Gram anchoring term keeps patch-to-patch similarities close to an earlier teacher snapshot, which stops the dense features from drifting. The authors credit it with keeping the dense structure stable, which is what a frozen segmentation readout depends on.

MRI first, then CT

Earlier work (MedCoSS) found that MRI and CT conflict when you pretrain on both at once, and solved it by training them in stages. VoxForge keeps the staging but puts MRI first, because MRI shows the same anatomy under many intensities across sequences, scanners and sites, and starting there should teach the encoder to separate anatomy from contrast before it ever sees Hounsfield units. The three stages are MRI at 128³, MRI at 128³ and 160³, then MRI plus CT moving to CT at 160³. Both modalities are rescaled to [0, 1] between per-volume percentiles, with CT clamped at -1024 HU first.

The data is cleared for commercial use

This is the detail I would flag to anyone building a product. The pretraining corpus is 234,919 public adult volumes from about 71K subjects: 63% MRI (21 TCIA collections and 895 OpenNeuro datasets, mostly brain, breast and prostate) and 37% CT (49 TCIA collections, mostly NLST chest CT). There is no PET at all. A collection was included only if its licence permits commercial use and derived works, which leaves CC0, PDDL, public domain and CC BY. The authors read the licence in the archive and in each dataset’s own files, took the most restrictive statement, and dropped 10 OpenNeuro datasets and 3 CT collections whose metadata said permissive but whose files did not. They also audited the corpus against every evaluation set and report that none of their test sets are in it.

Pretraining took 216 hours on 64 H200 GPUs. Nobody is retraining this in a clinic, which is the point of releasing frozen weights.

What the frozen features can do

The comparison set is wide: nnFoundation’s CNN and ViT as generalists, CT specialists (SPECTRE, VoxelFM, Merlin, CT-Instruct), MRI specialists (the Triad models) and registration methods (anatomix, uniGradICON, ANTs SyN, MIND-SSC). All encoders share the same heads and seeds, and every margin carries a paired bootstrap interval. These are the results I would look at first:

  • Small vessels from non-contrast CT. On ISLES’24, the head is trained to find the circle of Willis on non-contrast CT, where the vessels are barely visible. VoxForge reaches 54.0% Dice, about 7 points above the next encoder (VoxelFM at 46.9%). This is the fine CT anatomy that the SPECTRE authors warned a multi-modality model would lose.
  • PET it never saw. On all 1,014 autoPET studies in five-fold cross-validation, segmenting FDG-avid lesions from the PET channel alone, VoxForge has the best lesion Dice and F1 in every fold. It beats both nnFoundation encoders, which were pretrained on PET.
  • Few labels. With 10% of the TotalSegmentator CT training cases (29 of 293), it keeps 97% of its full-label Dice and matches SPECTRE trained on all 293. A quarter of the labels brings it within half a Dice point of nnFoundation’s ViT trained on everything.
  • No head at all. On MRI, a k-NN vote over the nearest training patients ranks it first on all three classification targets, and scoring each token by its distance to a bank of lesion-free tissue localises stroke and glioma lesions at about four times the AUPRC of the next encoder.
  • MRI to CT registration. Added to the standard MIND-SSC descriptor, its tokens register MRI to CT about as well as anatomix, a network built for that task.

Where it fits in a product

If you maintain a viewer or a reading platform with several AI features, a frozen backbone like this changes the cost structure. You run the encoder once per series, cache the tokens, and attach small heads for organ segmentation, a classifier, lesion candidates or a registration step. Adding a new structure becomes a head trained on tens of cases, not a new 3D network. The label-efficiency numbers are the strongest argument for that workflow, because small annotated sets are what most clinical projects have.

Two practical notes from the protocol. Each encoder was fed data at spacings inside its own pretraining range, so you will need to match VoxForge’s spacing and intensity contract rather than feed native DICOM grids blindly. And the 160³ crop means whole-body or large field-of-view work still goes through tiling.

Limits

  • No weights or code yet. As of 20 November 2026 the paper and arXiv page link no repository or model card. The licence audit is written so the corpus “can be released with the model”, but nothing is out, so you cannot test it today.
  • No ablation. The authors did not isolate the effect of the objective, the data mix or the token size, because each arm costs thousands of GPU-hours. The stage-by-stage checkpoints change data and training length together.
  • Smaller corpus. 235K volumes is about 10% of nnFoundation’s corpus, and the authors did not measure how frozen 3D features scale with data or model size.
  • Coverage. No ultrasound or radiography in pretraining, adult data only, and crops of at most 160³ voxels.
  • Frozen only. Every result is a frozen encoder with a trained head, compared against other frozen encoders. An earlier study the authors cite (Curia-MAE) found that nnU-Net trained from scratch still beat frozen encoders, so for a single task with plenty of labels a dedicated network may still win.
  • Numbers across papers don’t line up. The authors point out that published 3D foundation models use different heads, splits and exposure checks, so compare within one protocol. Two CT specialists in their tables had test-set images in their own pretraining, which the paper flags.
  • Preprint. No peer review stated. The paper includes an LLM usage statement saying the authors used LLMs to help with experimentation, data curation and leakage audits.

For related reading on our side, see our posts on MedImgAnnotator for batch CT and MRI annotation and GRACE for correcting segmentation masks.

Source

Paper: Rushil Anirudh and Jayaraman J. Thiagarajan (equal contribution), “VoxForge: A Unified Foundation Model for Dense 3D Features Across MRI, CT and PET”, arXiv:2611.25309 (19 November 2026), preprint. Affiliation: Amazon AGI. Featured image: the contrast panel of Fig. 6 (circle-of-Willis segmentation from non-contrast CT on ISLES’24: reference, SPECTRE and VoxForge), cropped from the paper and resized, with labels added.

We build custom medical imaging platforms — advanced DICOM viewers, AI segmentation, and the clinical systems around them.

Copyright © 2026 PYCAD. All Rights Reserved.