An enhanced multi-view extension for SAM 3D Objects, featuring adaptive fusion strategies for improved 3D reconstruction quality from multiple viewpoints.

🔗 Basic Version: For a simpler averaging-based approach, check out our basic multi-view fork.

• Adaptive Multi-View Fusion: Unlike simple averaging, we employ a confidence-aware fusion mechanism that automatically weighs contributions from different views based on their reliability.

• Multiple Weighting Strategies:

  • Entropy-based: Uses attention entropy as uncertainty measure
  • Visibility-based: Uses self-occlusion detection via DDA ray tracing
  • Mixed: Combines both strategies for robust weighting

• Per-Latent Weighting: Our method operates at the latent level, enabling fine-grained control over how information from different views is combined for each spatial location.

• Improved Reconstruction Quality: Better handling of occluded regions and view-dependent artifacts through intelligent fusion.

This is an active research project. We are continuously exploring new fusion strategies and will release updates as our research progresses.

Single-View (View 3)	Single-View (View 6)	Multi-View (Adaptive Fusion)
Input Image	Input Image	Input Images
↓ 3D Reconstruction ↓
3D Result	3D Result	3D Result

Please follow the installation instructions in the basic multi-view version or in SAM 3D Objects.

python run_inference_weighted.py \
    --input_path ./data/example \
    --mask_prompt stuffed_toy \
    --image_names 0,1,2,3,4,5,6,7

python run_inference_weighted.py \
    --input_path ./data/example \
    --mask_prompt stuffed_toy \
    --image_names 0,1,2,3,4,5,6,7 \
    --no_stage1_weighting --no_stage2_weighting

To use visibility-based weighting for Stage 2, you need to first run Depth Anything 3 (DA3) to obtain camera poses:

Step 1: Install Depth Anything 3

Please follow the installation instructions at Depth Anything 3.

Step 2: Run DA3 to get camera poses

python scripts/run_da3.py \
    --image_dir ./data/example/images \
    --output_dir ./da3_outputs/example

Step 3: Run weighted inference with visibility

python run_inference_weighted.py \
    --input_path ./data/example \
    --mask_prompt stuffed_toy \
    --image_names 0,1,2,3,4,5,6,7 \
    --da3_output ./da3_outputs/example/da3_output.npz \
    --stage2_weight_source visibility

📖 Full Parameters: See README_PARAMETERS.md for detailed parameter documentation.

input_path/
├── images/
│   ├── 0.png
│   ├── 1.png
│   └── ...
└── object_name/  # mask folder
    ├── 0.png
    ├── 1.png
    └── ...

Mask Format: RGBA format where alpha channel stores mask (alpha=255 for object, alpha=0 for background).

This project builds upon SAM 3D Objects by Meta. We thank the original authors for their excellent work.

This project inherits the SAM License from the original SAM 3D Objects project.