Anyone who has annotated a full-spine X-ray knows the problem. Vertebral corners are easy to see and hard to name. The corner you clicked could belong to T7 or to T8, and a detector that looks at a small patch has the same doubt. Augustin Daridon and Sofiane Sifaoui (shared first authors), with Ayman Assi, Mohamad Karam, Wafa Skalli, Elsa Angelini and Loïc Le Folgoc, attack it in two stages in arXiv:2610.36951. The team spans LTCI at Télécom Paris (Institut Polytechnique de Paris), the Clinical Biomechanics of the Spine Research Fund in Paris and the Laboratory of Biomechanics and Medical Imaging at Saint-Joseph University in Beirut. Pix2Point finds candidate landmarks from dense pixel embeddings. SpineSAM, a shape and appearance model, picks the configuration that looks like a real spine.
Why spine landmarks matter
Corner points on each vertebra are the raw material for the measurements a spine surgeon or orthopaedist reads off a standing X-ray: Cobb angle, sagittal vertical axis, pelvic tilt, sacral slope, lumbar lordosis, thoracic kyphosis. Placing those points by hand on every vertebra from C3 to L5 is slow, and small labelling errors move the angles. Automatic detection has been studied for years, mostly by regressing a Gaussian blob per landmark. Those methods do well on clear cases. On ambiguous ones they can put a corner on the wrong vertebra, and every angle that depends on it moves.
Stage 1: Pix2Point, landmarks as pixel embeddings
Pix2Point treats detection as a representation learning problem. A generic 2D encoder-decoder (a 2D nnU-Net, per the README) maps each pixel of the X-ray to a 512-dimensional embedding. Each landmark has a prototype vector in the same space. The heatmap for a landmark is a softmax over the image of the cosine similarity between each pixel’s embedding and that landmark’s prototype, and a contrastive loss pulls the right pixel towards its prototype and pushes the rest away.
Two design choices matter for the second stage.
- Heatmaps are free-form. There is no Gaussian target, so the heatmap can keep several peaks when the image is ambiguous. SpineSAM uses those extra peaks.
- The backbone is generic, so it can be pretrained. Because nothing in the network is detection-specific, the authors initialise it with Pix2Rep-v2, a dense self-supervised pretraining recipe from the same group. Prototypes can be trained as parameters, or computed at inference as the spherical mean of training embeddings for each landmark.
Stage 2: SpineSAM, a shape prior you can differentiate
SpineSAM fits a statistical spine shape to the heatmaps in three steps.
- Shape model. Landmark coordinates are normalised per subject so the infero-posterior corner of T1 sits at +1 and that of L5 at −1 along the spine axis. A PCA on the training set gives a Gaussian over plausible spine configurations.
- Discrete initialisation. The ten strongest peaks of each heatmap become candidates. Landmarks are grouped by anatomy (22 vertebrae from C3 to L5 with four corners each, plus the odontoid, the sacrum and the acetabula) and chained in a directed graph from top to bottom. Node weights come from heatmap log-probabilities, edge weights from the PCA prior, and Dijkstra’s shortest path picks a joint configuration in about a second per subject.
- Continuous refinement. From that start, LBFGS in PyTorch minimises a MAP energy that combines heatmap likelihood and shape prior. Heatmaps are sampled with bilinear interpolation, so the final points land at sub-pixel positions.
The graph step is what answers the T7-or-T8 question. Adjacent vertebrae look alike, so the energy has many local minima, and the authors note that LBFGS alone does not always reach the global one. The shortest path scores the full chain before any gradient step.
What the numbers say
The method was tested on three sets. The public AASCE challenge data provides a coronal test split with 68 corners from T1 to L5. Two private sets, one coronal and one sagittal from dual-view acquisitions, have 134 test images each and 93 landmarks per view. On AASCE, the full method reaches a mean radial error of 2.18% of spine length, against 2.55% for nnLandmark. On the private sets the error is about 3 mm in both views. In the ablation, the contrastive loss, the pretraining, the graph step and the LBFGS refinement each lower the error.
It does not win everything. On the private coronal set nnLandmark ranks better on the five clinical parameters, and on the sagittal set another method has a higher detection rate within 2 mm. Clinical parameters were only scored on the private data, so the public benchmark gives no Cobb angle comparison.
What it would look like in a viewer
On a full-spine radiograph in a DICOM viewer, the output is a labelled point set: four corners per vertebra from C3 to L5, the odontoid tip, two sacral points and the two acetabula. From there the viewer draws the endplate lines, the Cobb angle, the pelvic parameters and the sagittal balance lines, and the reader drags any point that looks wrong. Because SpineSAM keeps the whole configuration plausible, a fix on one vertebra could in principle be propagated through the shape model rather than leaving neighbours misaligned. The paper does not test an interactive loop, though. For a scoliosis clinic or a pre-operative planning tool, the realistic use today is a first pass that a clinician reviews, with the angles recomputed after every correction.
What you need to run it
- Code. BioMedTP/Pix2Point on GitHub covers the Pix2Point stage on AASCE only. SpineSAM is not in the repository, so the full-method numbers cannot be reproduced from it yet. The repository has no licence file at the time of writing, which by default means no reuse rights.
- Weights. The self-supervised pretrained backbone is on Hugging Face at sifaoso/Pix2Point, with the model card marked Apache 2.0. It is a starting point for training, not a landmark detector. No trained landmark model is released.
- Data. AASCE has to be requested from the challenge organisers. The private dual-view sets are not shared.
- Hardware. The README asks for a GPU with at least 40 GB for training, Python 3.12 and PyTorch 2.13. Inference on the AASCE test set runs on CPU in about three minutes.
- Preprocessing. Images are fitted into a 752×256 frame with the aspect ratio kept, contrast-enhanced with CLAHE and saved as 8-bit NIfTI.
Where it can fail
- Unusual anatomy. The shape model assumes 22 vertebrae from C3 to L5. Transitional or extra vertebrae break that assumption, and the authors list sub- and supernumerary vertebrae as future work.
- Severe deformity and hardware. A PCA prior learned from a training population pulls towards its average shape. Rare severe curves and instrumented spines may be under-represented, and the paper does not report results on them separately.
- Scale. AASCE has no pixel size, so errors there are in percent of spine length. Millimetre accuracy comes only from the private data.
- Reproducibility. Without SpineSAM code or trained weights, anyone starting today has to retrain Pix2Point and reimplement the shape model. The README also notes that a small fraction of landmarks can move between PyTorch versions.
Licence and disclosure
The paper is on arXiv under the standard non-exclusive distribution licence. The code repository has no licence file, and the pretrained backbone on Hugging Face is tagged Apache 2.0. One author, Wafa Skalli, is declared as co-founder of Skairos, a company that develops patient-specific biomechanical simulation software for spine surgery. This is research code, not a cleared device.
My take
The useful idea is the split of labour. The network is allowed to be uncertain, and anatomy settles the question. For anyone building spine measurement into a viewer, the graph-plus-PCA step is the part worth copying even with a different detector, because it targets the error clinicians notice first: a corner on the wrong vertebra. I would wait for SpineSAM code and a trained model before planning around it, and I would test it on post-operative and transitional spines first.