Shap-E for medical imaging

Shap-E for medical imaging is a 2023 PYCAD Team how-to: run OpenAI’s Shap-E (Jun et al., arXiv:2305.02463) to sample a 3D implicit function from a text prompt (or an image), then dump a mesh. A Colab-shaped fork with a PLY→STL step lives at amine0110/medical-shap-e. It is not a CT-to-mesh segmenter. It is not a PYCAD product.

If you meant 2D GAN samples from a catalogue → medigan. Those can both stay: a mammography PNG is not a text-to-3D implicit. If you meant a patient-specific lung mesh from CT → 3D lung model. If you meant the 3D imaging field → 3D medical imaging.

PYCAD builds custom web DICOM viewers and imaging models. It does not ship Shap-E, a text-to-organ SKU, or a print bureau.

What Shap-E actually is

Shap-E is a conditional generative model for 3D assets. It does not output one mesh format and stop. It outputs the parameters of an implicit function that you can decode as a NeRF (density + color along a ray) or as an STF (signed distance + texture) and then extract a mesh. The 2023 paper trains in two stages: an encoder that maps an existing 3D asset into those implicit parameters, then a diffusion model on the encoder’s outputs, conditioned on text or a view.

That is text-to-3D (or image-to-3D). It is not “read this DICOM and give me the patient’s liver.” A prompt like “a human heart” gives you a generic heart-shaped asset. A segmentation from a CT gives you that patient’s liver. Do not mix the two in a deck.

What it is useful for (and not)

• A teaching mesh. A prompt-built organ for a slide or a viewer demo when you do not have a clearance to show a study.
• A starting shape. Something to drop into a scene or a print-prep tool. You still have to look at it. Hearts with the apex on the wrong side are not “creative.”
• Not surgical planning. Planning on a generated prior is not planning on the patient. Patient-specific models stay on the 3D-lung / reconstruction URLs.
• Not a 2D GAN. medigan writes PNG/array samples. This writes a 3D implicit. Different file, different lie you can tell.

The call

Upstream is pip install -e . from openai/shap-e, plus CLIP. The official notebooks are sample_text_to_3d.ipynb and sample_image_to_3d.ipynb. The 2023 Colab fork (amine0110/medical-shap-e) was a packaging of that notebook so a local CUDA fight did not block a first run. Weights are large; a GPU is the difference between a minute and a day.

A text sample, in the shape of the official API:

from shap_e.diffusion.sample import sample_latents
from shap_e.diffusion.gaussian_diffusion import diffusion_from_config
from shap_e.models.download import load_model, load_config
from shap_e.util.notebooks import decode_latent_mesh

device = "cuda"
xm = load_model("transmitter", device=device)
model = load_model("text300M", device=device)
diffusion = diffusion_from_config(load_config("diffusion"))

latents = sample_latents(
    batch_size=1,
    model=model,
    diffusion=diffusion,
    guidance_scale=15.0,
    model_kwargs=dict(texts=["a human liver"]),
    progress=True,
    clip_denoised=True,
    use_fp16=True,
    use_karras=True,
    karras_steps=64,
    sigma_min=1e-3,
    sigma_max=160,
    s_churn=0,
)
mesh = decode_latent_mesh(xm, latents[0]).tri_mesh()
with open("liver.ply", "wb") as f:
    mesh.write_ply(f)

Confirm argument names against the repo if a release moves them. guidance_scale and karras_steps are the knobs that change “looks like a liver” vs “a red blob.” This is a smoke test, not a dataset.

Shap-E’s mesh dump is usually PLY. Print pipelines and a lot of surgical-planning tools want STL:

import trimesh

mesh = trimesh.load_mesh("liver.ply")
mesh.export("liver.stl", file_type="stl")

Install for that line is pip install trimesh. PLY can carry color; STL will not. That is expected.

What this page is not

• Not a PYCAD Shap-E product, Colab service, or organ marketplace. The model is OpenAI’s. The fork is a 2023 packaging.
• Not 386. medigan is 2D GANs. This is 3D implicits. Both stay.
• Not 246 / 664. Patient-specific meshes from a scan are a different job.

If the mesh is only there to stand up a viewer you will later point at a real study, that is the imaging piece. Case studies.

Paper: Jun et al., 2023, arXiv:2305.02463. Repos as linked above.