Medical image segmentation is labeling every pixel — or every voxel — in a scan so an organ, a lesion, or a vessel is a mask you can measure, not a blob you squint at. It is not classification (“this is a brain MRI”). It is not a box around a finding. It is not how you train nnU-Net, click in Fiji, or prompt SAM. Those are other pages.
The same job used to live here as “3D image segmentation,” “deep learning segmentation,” “medical image segmentation techniques,” and “brain segmentation.” Those URLs were this explainer with a different keyword. Brain as an organ how-to (FreeSurfer, hippocampus, a BraTS training loop) is not this page. Heart / lung / liver / mandible pages stay where they are.
What the mask actually is
A CT or MRI is a stack of slices. Each slice is a 2D grid of pixels. The stack is a volume. The unit in that volume is a voxel — a 3D pixel with a spacing in millimetres. Segmentation assigns a label to every voxel: background, liver, tumor, aorta. The output is a mask the same shape as the scan. From that mask you get volume, diameter, surface, a 3D model, or a radiation target.
2D segmentation labels one slice. 3D segmentation sees the volume, so a vessel that leaves the slice is still the same object. Slice-wise 2D is cheaper. Volume 3D is what you want when the structure is not a flat disk.
Semantic, instance, panoptic
Three names, three questions. Medical examples, not street cars.
| Type | Question | Medical example |
|---|---|---|
| Semantic | Which class is this voxel? | Every liver voxel is “liver.” Two metastases share one “tumor” label. |
| Instance | Which object is this voxel? | Metastasis 1 and metastasis 2 get separate masks. You can count them. |
| Panoptic | Class for stuff, instance for things | Fat / air as class; each lesion as its own ID. |
Most clinic organ models are semantic. Instance shows up when you need a count (nodules, mets). Panoptic is the research name for doing both in one pass.
The methods people actually mean
The old “techniques” URL was a listicle of eight names. They still matter as a map. Deep learning did not delete thresholding; it replaced it when the boundary is not an intensity cliff.
| Method | How it decides | Use when | Breaks when |
|---|---|---|---|
| Threshold | Voxel intensity vs a cutoff (global, local, Otsu) | Bone on CT, high-contrast lung air | Soft-tissue boundaries, uneven bias |
| Region growing | Seed voxel, absorb similar neighbors | A connected, homogeneous organ | Leaky edges, bad seed |
| Watershed | Treat intensity as terrain; split at ridges | Touching cells / particles | Oversegmentation without markers |
| Active contours / snakes | A curve that shrinks to an edge | Smooth organ walls | Bad init, weak gradients |
| Level set | Implicit surface that can split and merge | Vessels, changing topology | Heavy compute, twitchy parameters |
| Graph cut | Min-cut on a voxel graph (region + boundary cost) | A few objects with a decent prior | Huge 3D volumes, memory |
| Atlas | Register a labeled template, fuse labels | Normal anatomy (brain structures) | Tumors, resections, odd anatomy |
| CNN / U-Net | Learns the mask from labeled scans | Most modern organ and lesion work | No labels, domain shift, no GPU |
U-Net is the workhorse
A convolutional net looks at local neighborhoods and builds up to “this is a kidney.” U-Net (2015) is the architecture that stuck in biomedical imaging: an encoder that compresses context, a decoder that writes the mask back to full resolution, and skip connections so the edges are not mush. 3D U-Net does the same on the volume. Ronneberger’s page is the original.
Classification vs segmentation: classification tags the study or the crop. Segmentation tags every voxel. That is the whole difference. Accuracy is a Dice or IoU against a human mask, not a vibe. A paper Dice of 0.90 on one hospital’s scanner is not a number you can paste onto the next vendor.
Where it shows up in care
Oncology. Tumor and organ-at-risk outlines for radiation. Volume change between visits. The mask is a target, not a signed response.
Cardiology. Chamber volumes, ejection fraction, an aorta you can measure. The number is useful when it is re-checked on your machines.
Neurology. Lesion load, a tumor bed, atrophy maps. Brain MRI is not one image: T1 shows anatomy, T2 / FLAIR show fluid and many lesions. BraTS-style labels (edema, enhancing, core) are a dataset convention, not a protocol you run from this page. Anatomy-specific how-tos (heart, lung, liver, mandible) live on their own URLs — this page stays the methods explainer.
Then you actually draw the mask
- Train a net on your own dataset: nnUNet for medical image segmentation (
plan_and_preprocess/train/predict). - Click in Fiji: image segmentation in ImageJ (threshold, watershed, SAMJ, Analyze Particles).
- Prompt SAM on a DICOM: SAM for medical imaging.
Desktop Slicer + a trained spine model is a different job (3D Slicer nnU-Net spine). MONAI preprocess / augment for tumor volumes is that series, not this page.
Why clinic installs stall
Labels are expensive. A model trained on one scanner’s contrast dies on the next (domain shift). 3D training wants a GPU. A tool that does not land in PACS does not get used. Regulatory review is a process, not a footer. None of that is a reason to keep a second “what is segmentation” URL.
What this page is not
- Not a train loop. Dataset folders and
nnUNetv2_trainsit on the nnU-Net how-to. - Not Fiji. Menu paths sit on the ImageJ page.
- Not SAM on DICOM. That is SAM for medical imaging.
- Not an anatomy atlas. Brain / heart / lung / liver how-tos are other clusters.
- Not the service page. MONAI / nnU-Net deployment is an engagement, not this explainer.
If you are scoping a first clinic model, use the medical AI deployment checklist. PYCAD builds the imaging side of this. Case studies.