CNNs or ViT for medical imaging is an architecture comparison: a convolutional net (local filters, shared weights) versus a vision transformer (image patches + self-attention). It is not a definition of machine learning for imaging. It is not a “what is a CNN” explainer.
If you meant what imaging ML actually returns → machine learning for medical imaging. If you meant convolution, pooling, LeNet / AlexNet / ResNet → convolutional neural networks explained. If you meant the product job (second reader, who signs) → AI in radiology.
This is the 2023 PYCAD Team note. The code is public: amine0110/CNNs_vs_ViT. It is one run on two public PNG X-ray sets, not a general law.
How the two architectures look at a scan
| CNN | ViT | |
|---|---|---|
| Unit | A small kernel slides across the image | The image is cut into patches; each patch is a token |
| Inductive bias | Locality and translation equivariance are built in | Almost none. Position is an embedding you add |
| Context | Grows with depth (receptive field). Early layers see edges | Self-attention can mix any two patches from layer 1 |
| Data hunger | Works with smaller labeled sets because of the bias | Wants a lot of data, or a heavy pretrain, or it underfits |
| Typical medical use | 2D X-ray / fundus; 3D U-Nets on CT / MRI | When you already have a large pretrain or a global-context task |
A CNN is the default in medical imaging because labeled studies are expensive and lesions are often local. A ViT wins when the cue is long-range (a small finding that only makes sense against the whole film) and you can afford the pretrain. Hybrids (a CNN stem, a transformer neck) exist for the same reason. None of that is “transformers replaced convolution.”
The 2023 run (what we actually trained)
Two public Kaggle sets, both PNG X-rays, not DICOM:
- Bone fracture detection — fractured / healthy.
- RSNA bone age — hand films with an age label.
CNNs: Xception, VGG16, InceptionV3 in Keras, ImageNet weights, head frozen at first, trained 40 epochs. ViT: ViT-Base, 16×16 patches, PyTorch, 10 epochs. The loops are on GitHub, not here.
model = return_model(input_dim, len(classes), head=head, freeze=True)
model.compile(loss='categorical_crossentropy', optimizer=SGD(learning_rate=lr),
metrics=['accuracy'])
model.fit(train_ds, epochs=epochs, validation_data=val_ds, callbacks=[save_weights])That run: the CNN heads stayed around 44% accuracy after 40 epochs. The ViT reached about 94% after 10. That is a result on those PNGs with that recipe. It is not “ViT is 2× better on medical imaging.”
Why the gap is not a law: different frameworks, a frozen ImageNet CNN versus a trained ViT, PNG not a 12-bit DICOM, no site shift, no external test hospital. A CNN that is actually trained for the task — or a 3D U-Net on CT — is a different experiment. Read the repo before you paste the percentages into a deck.
When to pick which
- Small labeled set, local finding, 2D or 3D U-Net already works. Start with a CNN. Most clinic segmentation still looks like this.
- You have a large pretrain or the cue is global. Try a ViT or a hybrid. Budget the compute.
- You need a number a paper will believe. Re-validate on your scanners. A Kaggle accuracy is not a site number. That rule is the same on the ML imaging page.
What this page is not
- Not “what is machine learning for medical imaging.” Method layer, three outputs, pipeline → 5756.
- Not “what is a CNN.” Convolution / pooling / AlexNet / ResNet → 7712.
- Not a leaderboard. Two public PNG sets, one 2023 recipe.
If the model has to land in a viewer a clinician already uses, that is the imaging piece. Case studies.