Find duplicated scans

Find duplicated scans is a 2024 PYCAD Team how-to: hash the pixel array of a DICOM / NIfTI / NRRD file, then compare hashes. Two files with the same hash are the same pixels, even if the names differ. It is not a quality-assurance program. It is not report peer review. It is not a phantom / QC schedule. It is not a PYCAD product.

If you meant the imaging-QA program (equipment / protocols / people) → medical imaging quality assurance. If you meant peer review / report QA → quality assurance in radiology. If you meant phantoms / daily-weekly tests / reject analysis → quality control in radiology.

Duplicates sneak into a training folder when the same study is exported twice, saved under a new name, or copied as both .nii and .nrrd. A model that sees the same volume in train and val is a leak, not a better AUC. This page is the hash, not a department QA stack.

What the hash is doing

1. Read the pixel array (pydicom / nibabel / pynrrd). Tags are ignored.
2. Cast to float32 and min–max scale to 0–1 so a linear rescale of the same pixels still matches.
3. SHA-256 the bytes.
4. Same digest → duplicate. Different digest → not this test’s problem.

Supported extensions in the snippet: .dcm, .nii, .nii.gz, .nrrd, .nhdr. A DICOM series is many .dcm files; this hashes one file at a time. Two series that are the same study under different UIDs will match only if you compare the same instance, not the folder as a whole.

What it will not catch: a re-windowed JPEG, a resampled grid, a crop, or a series that was anonymized and re-exported with different pixels. Hash matching is exact (after that min–max), not perceptual.

The checker

import os
import hashlib
import numpy as np
import pydicom
import nibabel as nib
import nrrd

class ScanDuplicateChecker:
    def __init__(self, folder_path=None):
        self.folder_path = folder_path
        self.supported_formats = [".dcm", ".nii", ".nii.gz", ".nrrd", ".nhdr"]

    def _ext(self, file_path):
        name = file_path.lower()
        if name.endswith(".nii.gz"):
            return ".nii.gz"
        return os.path.splitext(file_path)[1].lower()

    def get_image_data(self, file_path):
        ext = self._ext(file_path)
        if ext == ".dcm":
            return pydicom.dcmread(file_path).pixel_array
        if ext in (".nii", ".nii.gz"):
            return nib.load(file_path).get_fdata()
        if ext in (".nrrd", ".nhdr"):
            data, _header = nrrd.read(file_path)
            return data
        raise ValueError(f"Unsupported file format: {ext}")

    def preprocess_image(self, image_data):
        image_data = np.asarray(image_data, dtype=np.float32)
        lo = np.min(image_data)
        hi = np.max(image_data)
        if hi == lo:
            return np.zeros_like(image_data, dtype=np.float32)
        return (image_data - lo) / (hi - lo)

    def compute_hash(self, image_data):
        return hashlib.sha256(self.preprocess_image(image_data).tobytes()).hexdigest()

    def check_duplicate(self, file1, file2):
        return self.compute_hash(self.get_image_data(file1)) == self.compute_hash(self.get_image_data(file2))

    def check_folder_for_duplicates(self):
        if not self.folder_path:
            raise ValueError("No folder path provided")
        file_hashes = {}
        duplicates = []
        for root, _, files in os.walk(self.folder_path):
            for name in files:
                path = os.path.join(root, name)
                if self._ext(path) not in self.supported_formats:
                    continue
                try:
                    digest = self.compute_hash(self.get_image_data(path))
                except Exception as exc:
                    print(f"Error processing {path}: {exc}")
                    continue
                if digest in file_hashes:
                    duplicates.append((path, file_hashes[digest]))
                else:
                    file_hashes[digest] = path
        return duplicates if duplicates else "No duplicates found"

# two files
print(ScanDuplicateChecker().check_duplicate("a.nii", "b.nii"))
# a folder
print(ScanDuplicateChecker("path/to/scans").check_folder_for_duplicates())

Install is pip install numpy pydicom nibabel pynrrd. The original 2024 note shipped a Windows .bat that created a venv and launched a small compare-two / scan-folder UI. The method is the class above; the .bat is not a product.

What this page is not

• Not 6118 / 5872 / 6016. Dedup of a research folder is not a QA program, not report peer review, and not a phantom schedule. 705 left this URL on purpose. Do not flatten it into those keepers.
• Not a PYCAD “duplicate checker” SKU or a clinic de-dup appliance.
• Not Calendly, not a custom-solution footer. Dropped.

If the same study has to be flagged inside a clinic viewer, that is the imaging piece. Case studies.