Most papers on oral cancer photographs score one image at a time. A screening program does not work that way. A dentist or health worker photographs a patient’s mouth several times (a median of four photos per patient in this paper’s data), notes whether they smoke or chew betel quid, and has to decide whether to send that person to a specialist. What the program watches most closely is how many sick patients it misses. In arXiv:2611.08360, Rupam Mukherjee, Mehakdeep Singh, Shirin Dasgupta and Subhamoy Mandal (Department of Medical Science and Technology, IIT Kharagpur, and Dr. B. C. Roy Multi-Speciality Medical Research Centre) build the model around that decision. They call it OraRefer.
What OraRefer does
OraRefer takes all the smartphone photographs of one patient plus five risk factors (age, sex, smoking, betel quid and alcohol) and returns a referral decision. The decision comes from a threshold with a stated guarantee: with 95% confidence, sensitivity on new patients is at least the target you set, 90% in the paper. A second mode, Mondrian conformal triage, sorts patients into refer, clinician review or routine care.
A patient counts as referable if any of their photographs shows an oral potentially malignant disorder (OPMD) or cancer. OPMDs are visible mucosal changes such as leukoplakia that carry a raised cancer risk, so catching them is the point of screening.
How it works
There are three layers.
The image model. A DINOv2 ViT-B/14 encoder, fine-tuned end to end, feeds a DPT decoder that segments background, mouth and lesion. Besides the class token and the mean token, the classifier head gets a lesion-weighted average of the tokens and the predicted lesion mass, so the class decision depends on where the network thinks the lesion is. Removing that segmentation guidance costs 0.023 in four-class macro AUROC, and the drop shows up in all five folds.
Self-localization. Training mixes full frames with mouth crops. At test time a first pass on the full photo predicts the mouth, the bounding box is padded by 8% and cropped, and a second pass segments and classifies the crop, so no one has to draw a box or a polygon. The predicted box has a median IoU of 0.981 with the annotated box, and swapping in the annotated box changes macro AUROC by only 0.001.
Patient fusion and referral. Each photo yields class probabilities (corrected for the class sampling used in training and temperature-scaled), lesion area, component count and boundary complexity, and uncertainty from 30 Monte Carlo dropout samples. Per patient these are summarized (maximum, mean, top-two mean, fraction above 0.5, noisy-OR), joined with the risk factors, and fed to a calibrated L2 logistic regression. The authors cross-fit the image model at two levels so the fusion model only ever sees out-of-sample image outputs, because a patient-level bound needs out-of-sample scores at both stages. Without it, the calibration scores would be too optimistic.
The threshold itself is an order-statistic rule. Sort the scores of referable patients in a calibration set and pick the rank given by a binomial bound. For exchangeable patients this guarantees sensitivity of at least the target with probability 1 minus delta. It needs at least 29 referable calibration patients for a 90% target at 95% confidence. No retraining is involved.
What it hangs on a case
Per photograph, the released inference script writes a row with the four class probabilities, the probability of OPMD or cancer, predicted lesion area and component count, boundary complexity, three uncertainty values, and the predicted mouth box. Per patient you get one calibrated score and either a refer flag or a triage action. In a viewer or a teledentistry app that maps to a lesion overlay on each photo, a mouth box, an uncertainty badge, and a single patient-level recommendation with the operating point it was calibrated for.
The numbers worth knowing
The data are 3,000 photographs of 714 patients, taken by dental surgeons at a teaching hospital in Sri Lanka and published by Piyarathne et al. 381 patients are referable, 61 of them with cancer. All results are patient-level five-fold cross-validation.
Patient-level AUROC is 0.919, against 0.890 for attention-based multiple-instance learning. At the 90% target, the guaranteed threshold detects 95% of referable patients, including 58 of 61 with cancer, at a specificity of 0.610. That means 69% of patients get referred. The conformal threshold, which only guarantees the target on average, lands at 0.906 sensitivity and 0.793 specificity. Triage refers 29% directly, sends 39% to review and 32% to routine care, and routine care receives 5.1% of referable patients.
Why a clinical AI team should care
- The referral layer is model-agnostic.
orarefer.referraltakes calibration labels and scores from any classifier. If you ship a triage model for chest X-ray, mammography or dermatology, the same few lines give you a threshold with a written sensitivity guarantee, and a flag that tells you when the calibration set is too small for it. - Each site can set its own operating point. Because calibration needs no model fit, a new clinic can collect a local sample of confirmed cases and set a threshold for its own population. That is a cleaner story for a deployment review than a single cut-off picked on the development data.
- One decision per patient. The paper shows that learned fusion beats taking the maximum or mean of photo scores. Any product that reads several images per patient, whether views, series or follow-ups, faces the same aggregation question.
- Photographs are imaging too. DICOM has a visible-light photography object, and teledentistry and dermatology platforms already store these images. A viewer that hangs a lesion mask and a calibrated referral next to them is a small step from what PACS-style tools already do.
Where it falls short
- One dataset, one hospital, one country, photographs taken by dental surgeons. Community health workers with older phones and worse lighting are the real screening setting, and the paper has no external test set.
- Hitting the guarantee meant referring 69% of patients, which would swamp most referral pathways. The triage mode helps, but 39% still go to clinician review.
- The guarantee assumes the new patients are exchangeable with the calibration set. A clinic must calibrate on its own confirmed cases, at least 29 referable patients, and labels for those cases need to be reliable. In this dataset histopathology confirmed all cancers but only 56.4% of OPMD diagnoses.
- The three missed cancers were localized and segmented well (best-photo lesion Dice of at least 0.825) but scored low, with no photo above 0.380. Missed OPMD cases had small lesions, a median 0.04 of the predicted mouth against 0.32 for detected ones.
- Lesion segmentation is rough, with a Dice of 0.580 on photos that have a lesion. Treat the overlay as a pointer to the lesion.
- All results are retrospective. The authors call for prospective evaluation in community screening.
Code, weights and license
Code is on GitHub at ZenAesthete/OraRefer under Apache-2.0, with training, annotation-free inference, fusion, cross-fitting and the referral and triage tools. The README is upfront that settings the paper does not state use standard working values, so your results will differ from the paper’s run, and that baselines and analysis code are not included. Trained weights are not distributed. They are available on request for non-commercial research and stay subject to the dataset terms. The dataset (Zenodo, doi:10.5281/zenodo.10664056) is restricted access, granted to accounts linked to an institutional email, under CC BY-NC-ND 4.0. So the referral toolkit is usable today in a commercial product, but the trained oral model and the data are research-only. The code is research software and not a medical device.
Paper: arXiv:2611.08360 · Code: github.com/ZenAesthete/OraRefer (Apache-2.0) · Data: Piyarathne et al., Zenodo (restricted, CC BY-NC-ND 4.0)