MedVTok: One Open Tokenizer for 2D Images and 3D Scans

Most imaging AI stacks carry one encoder per modality. A chest X-ray model, a pathology model, a CT model that only understands volumes, and a 2D model that has to be fed CT one slice at a time. Each one has its own preprocessing, its own embedding space and its own failure modes. MedVTok, from Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu and colleagues at Fudan University, the Shanghai Innovation Institute, Shanghai AI Laboratory, Stanford School of Medicine, Monash University, Shanghai Jiao Tong University and Ant Group, replaces that pile with one tokenizer. It is described in arXiv:2610.35842 (NeurIPS 2026), and the code and weights are already public.

What a visual tokenizer is, and why one shared tokenizer matters

A visual tokenizer turns an image into a sequence of compact vectors (tokens) and can turn those tokens back into an image. Generative models and multimodal language models work on the tokens, not on pixels. If the tokens keep fine detail, you can reconstruct and generate from them. If they also carry meaning, you can classify, retrieve, segment and answer questions from them.

MedVTok maps 2D images and 3D volumes from nine modalities into one token space: CT, MRI, X-ray, ultrasound, histopathology, fundus photography, OCT, endoscopy and dermoscopy. It was trained on 28,769,502 images and 70,144 volumes (about 35.3K CT and 34.9K MRI) drawn from public datasets.

Why a viewer or AI pipeline team should care

If you build a DICOM viewer with search, a report assistant, or an AI pipeline that touches more than one modality, the encoder layer is where the maintenance cost piles up. A single frozen encoder that accepts a fundus photo, a pathology tile and a full abdominal CT volume, and returns embeddings in the same space, means one feature store, one similarity search and one adapter into a language model. The paper tests the tokenizer frozen, with only small heads or a language model on top, which matches how a product team would deploy it.

For a CT or MRI viewer, the 3D side matters most. Many “medical foundation” encoders are 2D and treat a CT as a stack of independent pictures. MedVTok patches volumes directly in 3D, so a 32-slice 256×256 CT becomes 8 x 16 x 16 = 2,048 patches with real depth context.

What the method does

1. One transformer for every shape. Encoder and decoder each have 27 transformer blocks with width 1152, and the encoder starts from SigLIP2-SO400M. Every input is cut into 4x16x16 patches. A 2D image is zero-padded to four frames along depth, a volume is patched along all three axes. Rotary position encoding works on (z, x, y) coordinates, and patches from different images are concatenated into one flat sequence, so a single batch can mix 256 and 512 pixel images, 16 and 64 slice volumes, and 2D with 3D, without padding.
2. A continuous latent. The bottleneck is a KL-regularized Gaussian latent with 48 channels per token, trained with L1, L2, perceptual (LPIPS) and adversarial losses so that reconstructions stay sharp.
3. Slice Differential Regularization (SliDR). Per-voxel losses can be satisfied while neighboring slices drift apart, which shows up as banding or flicker when you scroll. SliDR adds an L1 penalty on the first- and second-order differences between consecutive slices, comparing the reconstruction with the input. Unlike total variation, it does not push the volume toward smoothness; it asks the reconstruction to change from slice to slice the way the real scan does.
4. Multi-Expert Representation Alignment (MERA). To give tokens clinical meaning, MedVTok aligns its features with frozen specialist encoders: DermLIP for dermoscopy, RET-CLIP for fundus, CONCH for pathology, PeskaVLP for endoscopy, MedImageInsight for 2D CT, MRI, ultrasound, X-ray and OCT, SPECTRE for 3D CT and HLIP for 3D MRI. One loss pulls everything into a shared space with contrastive learning; a second matches each expert in its own space. Expert features are computed offline once, so the experts never run during training.
5. Two stages. Stage 1 trains on 2D images only. Stage 2 adds volumes and switches SliDR on. Each stage ran about 100K steps on 16 H200 GPUs, roughly 16 and 44 hours.

One small design choice mattered: averaging the patch tokens (mean pooling) for the global embedding instead of an attention-pooling head gave substantially better segmentation in the ablations, because attention pooling squeezes out the spatial detail that dense tasks need.

What they found

All results use the tokenizer frozen. The comparison set includes medical tokenizers such as MedITok, PUMIT and MedVAE, general ones, and semantic encoders such as BiomedCLIP and MedImageInsight.

• Reconstruction. Averaged over the nine 2D modalities, MedVTok reached 40.18 dB PSNR against 32.91 dB for MedITok. On 3D CT from AMOS-MM it reached 36.50 dB against 31.87 dB for PUMIT and 31.30 dB for MedVAE-3D. 3D MRI was harder at 31.77 dB.
• Classification and retrieval. With k-nearest-neighbor classification on frozen features, average accuracy across nine datasets was 80.23%, ahead of MedImageInsight at 76.21%. MedImageInsight still won two of the nine datasets, and CONCH stays stronger on pathology.
• Segmentation. A single linear layer on frozen tokens gave an average Dice of 76.07 across eight datasets. The CT entry (MosMedData+ lung lesions) was only 42.91, so do not read this as a CT segmentation model.
• Generation, VQA and reports. Used as the latent space for a diffusion model, it gave the best average FID but was behind MedITok on CT and endoscopy. As the vision encoder inside LLaVA-Med it gave the best average VQA accuracy, and plugged into a 3D report generator on AMOS-MM CT it raised the GREEN score from 0.386 to 0.413.
• SliDR in isolation. Starting from the same checkpoint and 10K volumes, SliDR beat direct gradient matching, total variation and Laplacian penalties on both CT and MRI reconstruction. Raising its weight from 0.1 to 0.5 bought little extra 3D quality and cost 2.6 points of classification accuracy.

The featured image pairs two figures from the paper. On the left, Fig. 7 shows volume renderings of a pelvic CT from AMOS-MM and an abdominal MRI: the ground truth, MedVAE-3D, and MedVTok, each reconstructed from the frozen tokens. On the right, Fig. 3 shows per-pixel error maps on a brain MRI slice, where brighter means larger error; the MedVAE map is visibly brighter than the MedVTok one.

What this means for your stack

1. Try it as a shared embedding layer first. Similarity search across a mixed archive (find studies that look like this one) and quick linear probes are the cheapest way to see whether the tokens work on your data.
2. Use it as the visual front end of a multimodal model. The VQA and report results are early, but they show that the tokens can feed a language model for both 2D images and CT volumes through one interface.
3. Get the intensity handling right. Inputs are 3-channel tensors normalized to [-1, 1], shaped [3, H, W] for images or [3, D, H, W] for volumes. In the paper, CT for reconstruction was clipped to [-1000, 1000] HU and MRI to the 0.5th and 99.5th percentiles per volume; the report task used a [-160, 240] HU window. Your DICOM-to-tensor code must follow the same choices or the tokens will drift.
4. Budget GPU memory for 3D. The paper does not give inference memory figures, so measure on your own volume sizes. The repository notes that Stage 2 training batches can exceed 48 GiB per GPU, and the full training run used 16 H200s.

Where it falls short

• Not every task wins. CONCH is still better on pathology, MedImageInsight wins some classification sets, MedITok gives better CT and endoscopy synthesis, and the CT segmentation probe is weak.
• Only nine common modalities. The authors list other modalities, clinical photos and figures from publications as unexplored. 3D support covers CT and MRI only.
• Experts are frozen. The semantic quality is bounded by the seven expert encoders. If an expert is weak for your sub-domain, the tokens inherit that.
• Research benchmarks, not clinical validation. All evaluations are on public datasets with a frozen model. Anything that reaches a clinician needs testing on your own scanners and populations.

Code, weights and license

The code is on GitHub at Masaaki-75/medvtok with a demo notebook, feature extraction scripts and the two-stage training recipe. The checkpoint (about 3.3 GB) is public and not gated on Hugging Face at massaki75/medvtok. Both the repository and the model card state the MIT license, although the paper text says the release would use Apache 2.0, so check the license file at the commit you adopt. Retraining needs the seven expert encoders, some of which (DermLIP, CONCH) require access requests and carry their own terms, plus the public training datasets under their own licenses. The paper itself is released under CC BY 4.0. The project page is at masaaki-75.github.io/medvtok.

Related reading on PYCAD

• Merlin: CT-native foundation models for radiology AI
• nnFoundation: task-ready 3D features from CT, MRI and PET
• CT reports and finding answers with 3D tokens intact

Sources

• Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, et al. MedVTok: A General-Purpose Medical Visual Tokenizer. arXiv:2610.35842, 2026, NeurIPS 2026. Abstract and PDF.
• Code: github.com/Masaaki-75/medvtok (MIT). Weights: huggingface.co/massaki75/medvtok (MIT).
• Featured image: Fig. 7 and Fig. 3 of the paper, tiles cropped and resized only.