The hybrid spatial agent takes one 2D axial CT slice and a binary spatial question in plain language, for example “Is the liver left of the spleen?”, and returns a 1 or 0 plus an audit trail. That trail keeps the original prompt, the extracted query, the parsed tuple (organ, relation, organ), ontology-matched class names, the selected YOLO boxes, the geometric comparison, and the final answer. Simon Vincent Abel, Heiko Hillenhagen, Michael Götz, Timo Ropinski, Ayhan Can Erdur, and Daniel Santak Wolf (Ulm University, Ulm University Hospital, TUM) submitted arXiv:2608.21140 on 21 August 2026 for the MICCAI 2026 Agentic AI for Medicine Workshop. Shared last authorship is Erdur and Wolf. Code is public at github.com/DeveloperNomis/MICCAI-Medical-Agent.
If you hang CT in a DICOM viewer, this is a left/right or above/below check you can inspect on the current slice. YOLO draws the two organs and centroids order them, so the language model is out of the truth-value path. Direct VLMs sit near chance on this task. The hybrid is a spatial yes/no building block on a hung axial slice.
Parse the question, then order the centroids
The path is parse, ontology match, YOLO localize, then deterministic geometry from object centroids. Horizontal and vertical ordering only. LangChain is orchestration. No neural model predicts the final truth value.
The parser is a lightweight extractor with a VLM fallback. The repo installs spaCy en_core_web_sm for that first pass. A question such as the liver/spleen example becomes a tuple, the two names are mapped onto detector classes, and only those two boxes go to geometry. If ontology matching fails, the agent returns invalid and writes an audit entry instead of guessing.
61-class YOLO, one box per organ
The detector is YOLO. The README training example is YOLOv8m, imgsz=512, 500 epochs, batch 128, freeze=2. It covers 61 anatomical classes. Boxes for training come from the MIRP training-cohort segmentations, plus AMOS and BTCV, for 421,023 instances. Patient-level isolation holds versus the test set. Inference does not use segmentation masks. The highest-confidence box per class is the one that goes to the centroid test.
Checked 29 August 2026: checkpoints/yolo_detector_best.pt is in the GitHub tree. A GET on the raw file returned HTTP 200 (about 45 MB). The .gitignore excludes *.pt in general, then un-ignores that one checkpoint. Large VLM checkpoints are not in the repo. A demo MIRP subset sits in demo-data. You still set local data paths, and the full MIRP set is on Hugging Face and github.com/Wolfda95/MIRP_Benchmark.
938 held-out pairs, 55 misses
Evaluation is the held-out MIRP RQ1 test set, 938 image-question pairs. The hybrid agent receives the same user prompt as the direct VLMs, which must answer exactly 1 or 0.
Qwen2-VL plus hybrid reaches 94.1% accuracy and 94.2% F1. Direct Qwen2-VL is 51.6% accuracy and 56.8% F1, a gain of 42.5 points. MedGemma plus hybrid is 91.6% / 91.3%. Direct MedGemma is 51.8% / 67.2%. No invalid binary outputs. Prior MIRP leaderboard VLMs (GPT-4o, Gemma 3, Pixtral, MedGemma) sit near 50% on a different split.
The best hybrid still misses 55 cases. Imprecise localization is 28 of them (50.9%), parsing or query extraction is 15 (27.3%), and missing detection is 12 (21.8%). Routing, ontology, geometry ambiguity, and formatting contributed zero. A sloppy box can move the centroid enough to flip left/right. The method is pairwise 2D axial only. It does not do volumetric reasoning, distance, containment, overlap, or multi-structure consistency.
How this lands in a viewer
Hang the two boxes and the 1/0 on the current axial. Keep the audit panel open so a reader can see which centroids were compared. If an organ is missing, fail closed. Stay on pairwise axial.
MedPixel is phrase or loose box on a hung slice. CoInS-Net is two endpoints and an in-between plane. SAT3D is tumour plus uncertainty in Slicer. This paper is one axial CT slice and an auditable left/right call.
Rebuild from arXiv:2608.21140 and the public repo. YOLO weights are in checkpoints/. VLM weights are not. Retrospective binary accuracy on 938 pairs is not a device claim. The preprint is the source of record until a venue version exists.
Sources
- Abel, S.V., Hillenhagen, H., Götz, M., Ropinski, T., Erdur, A.C., Wolf, D.S. A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans. arXiv:2608.21140. Submitted 21 August 2026. MICCAI 2026 Agentic AI for Medicine Workshop. https://arxiv.org/abs/2608.21140 (HTML: https://arxiv.org/html/2608.21140). Hugging Face: https://huggingface.co/papers/2608.21140. GitHub: https://github.com/DeveloperNomis/MICCAI-Medical-Agent. Contact: daniel.wolf@uni-ulm.de. Checked 29 August 2026: YOLO weights at
checkpoints/yolo_detector_best.pt(HTTP 200); large VLM checkpoints not in the repo. - Wolf, D., Hillenhagen, H., Taskin, B., Bäuerle, A., Beer, M., Götz, M., Ropinski, T. Your other left! Vision-language models fail to identify relative positions in medical images. MICCAI 2025. MIRP dataset: https://huggingface.co/datasets/Wolfda95/MIRP_Your_Other_Left, https://github.com/Wolfda95/MIRP_Benchmark.
- Ji, Y., et al. AMOS: A large-scale abdominal multi-organ benchmark. NeurIPS 2022.
- Landman, B., et al. Multi-Atlas Labeling Beyond the Cranial Vault (BTCV). MICCAI Workshop, 2015.
- Jocher, G., Chaurasia, A., Qiu, J. Ultralytics YOLOv8. 2023. github.com/ultralytics/ultralytics
- Wang, P., et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv:2409.12191. 2024.
- Sellergren, A., et al. MedGemma Technical Report. arXiv:2507.05201. 2025.