AIM-Research-Lab/Medical-SAM3: Universal Prompt-Driven Medical Segmentation

Medical imaging demands pixel-perfect accuracy, yet traditional segmentation pipelines require extensive manual annotation or domain-specific model retraining for every new organ, lesion, or imaging modality. The rise of prompt-driven foundation models has changed how developers approach computer vision tasks in general domains, but medical applications—with their 3D volumes, specialized vocabularies, and strict evaluation protocols—present unique challenges that generic segmentation tools fail to address. AIM-Research-Lab/Medical-SAM3 enters this gap as a purpose-built foundation model for universal prompt-driven medical image segmentation, supporting both 2D benchmark evaluation and 3D volume training through text and box prompts.

What is AIM-Research-Lab/Medical-SAM3?

Medical-SAM3 is an open-source foundation model for medical image segmentation maintained by the AIM Research Lab. Built on the SAM3 architecture, it extends prompt-driven segmentation into the medical domain with unified 2D and 3D capabilities. The repository (189 stars, 30 forks, primarily Python^{↗ Bright Coding Blog}) provides not only inference tooling but also complete training and evaluation pipelines—an uncommon combination in medical AI tooling, where pretrained weights often arrive without reproducible training code.

The project emerged from research published in January 2026 (arXiv:2601.10880) and has seen active development through mid-2026, with the V2 3D training and evaluation code released in June 2026. Pretrained weights are distributed via Hugging Face under the Chongcong/Medical-SAM3 model hub entry, reducing friction for practitioners who need immediate experimentation without training from scratch.

Medical-SAM3 sits at the intersection of foundation models, computer vision, and medical AI. Its relevance stems from a specific technical bet: that medical segmentation can be unified under a single promptable model rather than fragmented into dozens of task-specific networks. The repository bundles the SAM3 implementation directly (sam3/), eliminating external dependency version conflicts that plague research reproduction.

Key Features

Unified 2D and 3D segmentation pipelines. Medical-SAM3 provides distinct but integrated pathways: the inference/ directory handles 2D benchmark evaluation across datasets like CHASE_DB1, STARE, and CVC-ClinicDB with box and text prompts, while medical/ contains full 3D training infrastructure and JSON-aligned evaluation protocols. This dual structure acknowledges that clinical deployment often requires both slice-by-slice analysis and volumetric consistency.

Prompt flexibility via text and bounding boxes. The model accepts both text prompts (using a BPE vocabulary in assets/) and box prompts, enabling zero-shot adaptation to new anatomical structures without retraining. This matters for developers building interactive annotation tools or integrating with radiology reporting workflows where natural language descriptions are already generated.

Self-contained SAM3 implementation. The sam3/ directory includes the model architecture, trainer, and Hydra configuration system. No separate SAM3 repository clone is required, which simplifies environment management and ensures version compatibility between the base model and medical extensions.

Hugging Face model distribution. Pretrained weights are available through the Hugging Face Hub, enabling direct loading via standard transformers workflows rather than custom download scripts.

Reproducible training configurations. The configs/medsam3_stage1_train_all_unified Hydra config provides a documented starting point for fine-tuning on proprietary 3D annotations, with explicit path preparation via prepare_config_paths.sh for environment-specific directory structures.

Use Cases

Interactive radiology annotation workstations. Developers building tools for radiologists can integrate Medical-SAM3's text-prompted segmentation to accelerate contouring of organs-at-risk or tumor volumes. A radiologist typing "left ventricle myocardium" or drawing a rough bounding box receives a segmentation mask without switching between organ-specific models.

Benchmark evaluation and method comparison. Researchers validating novel architectures against established medical segmentation datasets can use the inference/ pipeline as a reproducible baseline. The built-in support for CHASE_DB1, STARE, and CVC-ClinicDB provides immediate entry points without dataset conversion overhead.

3D volume fine-tuning for institutional data. Hospitals and research groups with proprietary 3D annotated volumes (CT, MRI) can fine-tune the pretrained SAM3 weights using the medical/ training code. The Hydra configuration system supports experiment tracking and hyperparameter sweeps appropriate for research groups iterating on model design.

Foundation model behavior analysis. ML researchers studying prompt engineering, zero-shot generalization, or foundation model robustness in medical domains gain a concrete implementation with documented evaluation protocols. The 3d_val.py script's --text-only and --use-train-prompts flags enable controlled ablation studies.

Educational reproduction of medical AI papers. Students and independent researchers can reproduce the paper's results using the provided training code, addressing a common failure mode in medical AI where published methods lack executable implementations.

Installation & Setup

Medical-SAM3 requires Python and assumes CUDA availability for GPU training. The setup process involves repository cloning, environment variable configuration, and editable installation with training dependencies.

First, clone the repository and navigate into it:

git clone https://github.com/AIM-Research-Lab/Medical-SAM3.git
cd Medical-SAM3

Set the required environment variables. These paths must reflect your local directory structure:

export MEDSAM3_ROOT=$(pwd)
export MEDSAM3_DATA_ROOT=/path/to/project
export MEDSAM3_CVPR_ROOT=$MEDSAM3_DATA_ROOT/converted_cvpr_biomedsegfm

Install in editable mode with training dependencies:

pip install -e ".[train]"

If your YAML configuration paths differ from defaults, run the preparation script once:

bash medical/training/prepare_config_paths.sh

For 2D inference only, navigate to the inference directory and execute evaluation scripts directly. The SAM3 model code is already present in sam3/—no additional clone needed.

Real Code Examples

2D Benchmark Evaluation

The simplest entry point runs pretrained weights against standard 2D medical benchmarks:

cd inference
python run_medsam3_evaluation.py \
  --checkpoint /path/to/checkpoint.pt \
  --model-name medsam3

This script evaluates Medical-SAM3 on datasets like CHASE_DB1 (retinal vessel segmentation), STARE, and CVC-ClinicDB (polyp detection). The --model-name medsam3 flag ensures correct architecture loading. Results include quantitative metrics and visualization outputs for baseline comparison against prior methods. Developers should replace /path/to/checkpoint.pt with their actual checkpoint location or downloaded pretrained weights.

3D Training from Config

Fine-tuning on 3D medical annotations uses the Hydra-configured training script:

python sam3/train/train.py \
  -c configs/medsam3_stage1_train_all_unified \
  --num-gpus 4

The -c flag specifies the unified training configuration that combines multiple datasets. The --num-gpus 4 parameter enables data-parallel training across four GPUs—adjust based on available hardware. This configuration represents the stage 1 training regime described in the paper, training on all available data before potential task-specific refinement.

3D Evaluation with Text Prompts

Held-out evaluation uses a separate script with explicit prompt control:

python medical/evaluation/3d_val.py \
  --ckpt experiments/medsam3_stage1_train_all_unified/checkpoints/checkpoint.pt \
  --out-dir experiments/results/3d_val/unified \
  --eval-all-volumes \
  --text-only \
  --use-train-prompts

The --eval-all-volumes flag ensures complete volume processing rather than slice sampling. --text-only restricts prompts to text descriptions, testing the model's language understanding without spatial guidance. --use-train-prompts evaluates with the same prompt distribution seen during training, measuring in-distribution performance before zero-shot evaluation.

Advanced Usage & Best Practices

Environment isolation is essential. Medical-SAM3's editable installation (pip install -e ".[train]") modifies the local Python environment. Use virtual environments or containers to prevent conflicts with other SAM variants or medical imaging toolkits.

Path configuration before first run. The prepare_config_paths.sh script resolves a common failure mode where absolute paths in YAML configs mismatch deployment environments. Run this proactively rather than debugging path errors during training initialization.

GPU memory scaling. The 3D training configuration assumes multi-GPU setups. For single-GPU experimentation, reduce batch size or use gradient accumulation—though this may require config modification beyond the documented defaults.

Prompt engineering for clinical vocabularies. The BPE vocabulary in assets/ was trained on medical text. Domain-specific synonyms or abbreviations common in your institution's reports may require prompt variation to achieve optimal segmentation quality. Consider systematic prompt variation studies for production deployment.

Checkpoint versioning. The experiments/ directory structure supports multiple training runs. Establish naming conventions early—medsam3_stage1_train_all_unified is descriptive but lengthy; symlink or wrapper scripts can simplify repeated evaluation calls.

Comparison with Alternatives

SAM (Meta's original) provides the architectural foundation but lacks medical-specific pretraining and text prompting. It serves general segmentation well but requires substantial medical fine-tuning to achieve comparable performance.

nnU-Net remains the practical standard for many medical segmentation competitions with robust auto-configuration, but it trains task-specific models rather than offering unified prompt-driven inference. For workflows requiring rapid adaptation across anatomical structures without retraining, Medical-SAM3's prompt interface offers distinct advantages.

The trade-off: Medical-SAM3's foundation model approach may underperform nnU-Net on narrowly defined tasks with abundant training data, while excelling in low-data or multi-task scenarios where prompt flexibility reduces model proliferation.

FAQ

What license covers Medical-SAM3? The repository lists "Other" as its license. Review the repository's LICENSE file directly before commercial use or redistribution.

Can I run inference without training? Yes. Download pretrained weights from Hugging Face (Chongcong/Medical-SAM3) and use run_medsam3_evaluation.py in the inference/ directory.

What hardware is required for 3D training? The documented config uses 4 GPUs. Reduced configurations are possible but untested in the provided documentation.

Does it support DICOM natively? The README specifies "3D / JSON eval" and "converted_cvpr_biomedsegfm" paths. DICOM support likely requires preprocessing to the expected format—verify against dataset conversion utilities in your environment.

Is the online demo available? As of the last commit (June 2026), the demo is marked "🚧 Doing" in the todo list. Monitor the repository for release announcements.

How does text prompting compare to box prompting in accuracy? The evaluation script's --text-only flag suggests text-only performance is measurable and potentially variable. Run comparative evaluation on your target anatomy to determine suitability.

Can I contribute or report issues? Standard GitHub workflows apply. The repository accepts issues and pull requests through its GitHub page.

Conclusion

AIM-Research-Lab/Medical-SAM3 delivers a technically concrete foundation model for medical image segmentation, distinguishing itself through unified 2D/3D pipelines, prompt flexibility, and—critically—released training code alongside pretrained weights. For developers building radiology tools, researchers benchmarking segmentation methods, or ML engineers exploring foundation model behavior in medical domains, it provides reproducible infrastructure rather than black-box APIs.

The project is best suited for teams comfortable with PyTorch, Hydra configurations, and GPU cluster management. Those seeking plug-and-play commercial solutions may find the setup requirements substantial; those needing customizable, research-grade medical segmentation will find the transparency valuable.

Explore the repository, download pretrained weights from Hugging Face, and evaluate on your target datasets: https://github.com/AIM-Research-Lab/Medical-SAM3

For related foundation model developments in medical imaging, see [INTERNAL_LINK: medical-foundation-models-comparison].