RepConTUR: Report-Prompted 3D Tumor Contours for Abdominal CT

Radiotherapy target contouring routinely uses the diagnostic CT together with its radiology report, and the report usually says where the tumor is, down to the liver segment or the part of the pancreas. Most auto-contouring models never see that text. They segment every tumor-like region in the organ, and the user deletes the ones that are not the target. In arXiv:2610.40094, Seunghwan Lee, Sangjoon Park, Jongmin Lee, Yoon Jo Kim, Wonyoung Cho, Jin Sung Kim and Han Joo Chae (Oncosoft Inc., Seoul, and the Department of Radiation Oncology, Yonsei University College of Medicine) present RepConTUR, a 3D model that takes the CT plus a suborgan phrase, and optionally the whole narrative report, and returns a volumetric mask of the tumor the text points to.

We covered a related idea earlier, LocusCT, which learns finding masks from PACS arrows and one report sentence. RepConTUR is aimed at a narrower job: gross tumor volume contours in the liver, kidney and pancreas, conditioned on suborgan location, with the full report allowed as context.

What the model is made of

RepConTUR is a multimodal language model with a 3D segmentation head. It has three parts.

1. A 3D CT encoder. The CT-ViT from CT-CLIP, which attends within each axial plane and then across planes. The authors pretrained it with a CLIP objective on chest CT and report pairs from CT-RATE plus abdominal pairs from Merlin, then froze it. It reads the volume resampled to 1.5 x 1.5 x 3.0 mm on a 240 x 240 x 140 grid, clipped to [-1000, 1000] HU. An attentional pooler turns its output into 256 visual tokens.
2. A small language model. InternVL2.5-1B, initialised from the medically pretrained UniBioMed weights, frozen and adapted with LoRA. The visual tokens and the text enter together from the first layer, so the image and the report are processed jointly rather than merged at the end.
3. A 3D mask decoder. The model emits a [SEG] token. Its hidden state is projected to a query for the VoxTell decoder, which sits on a frozen nnU-Net residual encoder and runs on 1 mm isotropic 192³ patches. The decoder turns the query into a full-resolution volumetric mask.

The whole model is about 1.41 billion parameters.

Where the training labels came from

The data is AbdomenAtlas 3.0: 9,262 abdominal CT volumes with tumor masks for liver, kidney and pancreas, organ and suborgan masks such as the Couinaud segments, and a report for every volume. Those reports are synthetic. The RadGPT pipeline derives a structured report from the masks (count, size, location and attenuation per suborgan), and a language model rewrites it as narrative text. The authors kept the 3,798 volumes with at least one tumor, since radiotherapy planning only happens for tumor-positive patients.

The organ-level tumor masks were then split into suborgan targets. Each connected tumor component is matched to the suborgans it overlaps, using the fraction of the component inside each suborgan, and the report’s stated count per suborgan decides which components belong where. From the same data they built four tasks and trained them together:

• Referring segmentation (RES): “Please segment the tumor in hepatic segment 7.”
• Report-grounded segmentation (GSEG): the same request with the full narrative report in front of it.
• Report generation from the CT.
• Visual question answering, 71,783 checked question and answer pairs about tumor presence, count, location, size band and enhancement.

Training ran 20 epochs on four H100 80 GB GPUs.

The workflow on one patient

1. Take the planning or diagnostic CT and its report.
2. A separate language model reads the report and lists the tumor-bearing suborgans, for example “segment 4 of the liver” and “pancreas body”. If the report gives only an organ-level finding, the prompt becomes a whole-organ phrase such as “all tumors in the kidney”.
3. RepConTUR encodes the CT once and caches it. With GSEG, the report is also cached in the language model.
4. Each suborgan prompt produces its own mask. The masks are combined by union into one organ prediction.

On an RTX A5000, for the most tumor-heavy test case with seven suborgan targets, the authors timed about 24 seconds, against about 6 seconds for an nnU-Net specialist. The cost is about 5 seconds of encoding per volume and roughly 2.7 seconds per target, so a single-target case is closer to 8 seconds.

What a viewer would hang on the scan

The output is one binary 3D mask per prompt, on the CT grid. In a contouring tool I would store each one as its own structure named after the prompt, so “tumor in segment 4” stays attached to the voxels it produced, and export them as RTSTRUCT or DICOM SEG. When the queried region has no tumor the model returns an empty mask, and the viewer should show that as an answer (“no tumor found for this prompt”) rather than as a silent blank. Figure 3 of the paper shows this: switching the prompt from segment 5 to segment 6 moves the mask, and asking for segment 3 or a tumor-free kidney gives an empty output on that slice.

I would keep the generated report text and VQA answers out of the clinical view. The paper’s own report example gives the largest kidney lesion as 1.0 x 0.8 cm where the reference says 3.6 x 3.3 cm.

Results that matter for a clinic

The main gain is fewer contours on tumors the user did not ask for. Figure 2 makes the point: with “tumor in segment 4 of the liver”, VoxTell, UniBioMed and BiomedParse v2 also mark other real liver tumors on the same slice, and with “tumor in the kidney” some of them mark the tumor-free kidney on the other side. In the test split, averaged over the three organs, instance-level precision was 0.397 for RepConTUR with the phrase alone and 0.433 with the full report, against 0.071 or lower for every text-promptable baseline. BiomedParse v2 still had the highest recall in every organ.

Against vision-only specialists (nnU-Net, and R-Super, which uses reports only during training), the authors tested four public cohorts outside AbdomenAtlas: HCC-TACE-Seg, Colorectal-Liver-Metastases, CPTAC-PDA and CPTAC-CCRCC. With the report, mean detection F1 was 0.544 against 0.415 for nnU-Net, again mostly through precision. In domain, nnU-Net stayed ahead on kidney tumors, which are already easy to see on contrast-enhanced CT.

Where it breaks

• Every report is synthetic. The training reports come from the masks. The four external cohorts had no reports, so the authors generated them with the same pipeline from the ground-truth tumor masks and automatic organ masks. The report given to the model therefore already knows the correct suborgans. Real reports vary in style and are not always consistent, and the authors state that the gain still has to be confirmed on real report and image pairs.
• Small tumors. Below 1 cm, precision stays low for every model, this one included. On small pancreatic tumors it is close to zero.
• Organ-level findings. When the report only says “multiple liver lesions”, there is no suborgan to prompt, and the model falls back to a whole-organ query that behaves more like ordinary segmentation.
• Three organs only. Liver, kidney and pancreas. Other abdominal organs and body regions are untested.
• Not all report content is used. In the authors’ relevancy analysis, location words carried the most weight and the words describing tumor contact with vessels carried little, so I would not expect vessel-relation text to change the contour.
• No editing-time study. The clinical value of auto-contouring is the time it saves, and the authors did not measure it.
• An extra language model in the loop. The suborgan parsing step used an OpenAI model in the paper. A hospital deployment would need a local parser and a check on its errors.

Code, weights and licence

The code is public at github.com/Oncosoft-Research/RepConTUR under Apache-2.0, with training, fine-tuning and a Gradio demo. The weights are on Hugging Face at Oncosoft-Research/RepConTUR and the suborgan annotations and question pairs at the matching dataset repository, both under CC BY-NC-SA 4.0, which does not allow commercial use. The CT volumes and masks come from AbdomenAtlas 3.0 and are downloaded separately. The preprint is CC BY-NC-SA 4.0 and was submitted to CVPR 2027. I checked the repository and model card on 31 October 2026. This is a research model, not a cleared contouring product.

My take

I like the idea because it matches how contouring is already done. The report chooses which tumor to contour, at suborgan level, and the model draws far fewer of the extras a person would otherwise delete. Code, weights and annotations are all public, so it can be tested on local data now. The open question is the report. Everything here was tested with reports written from the answer masks, so before trusting it I would run it on a set of local cases with their real reports and count how often the parsed suborgan is wrong, and how many tumors the parser misses.