dust3r pytorch inference minimal

Open In Github

Minimal code for running the Dust3r model with PyTorch.

The goal of this repository is to provide the minimal code to test and understand how the model works.

The official model contains a lot of code that is not necessary for inference.
The model is relatively simple, however handling more than 2 images requires global alignment.
To make the model easier to be exported to ONNX.

Assume image size is constant for all images
Precalculate rope2d’s sines and cosines since the image size is constant
Divide the model into encoder, decoder and head
Replace expm1 in the depth estimation head with exp -1 to make it possible to export to ONNX
Add visualization with rerun-sdk
Remove unnecessary code for the minimum of running the model with 2 images

Not all features are supported, such as global alignment, more than 2 images, etc. For those cases, the following repository is recommended:

mini-duster: A miniature version of dust3r only for performing inference. This makes it much easier to use without needing the training/data/eval code. Tested on Linux, Apple Silicon Macs, and Windows

  
git clone https://github.com/ibaiGorordo/dust3r-pytorch-inference-minimal.git
cd dust3r-pytorch-inference-minimal
pip install -r requirements.txt

Recommended but requires uv to be installed (https://docs.astral.sh/uv/getting-started/installation/)

git clone https://github.com/ibaiGorordo/dust3r-pytorch-inference-minimal.git
cd dust3r-pytorch-inference-minimal
uv sync

Additionally, to activate the uv environment: source .venv/bin/activate in macOS and Linux or .venv\Scripts\activate in Windows

You can additionally set visualize_depth=True when using visualize_output() function to visualize the depth map.

python example_image_pair.py

Example to visualize the raw output of the model: the pointclouds from both cameras aligned with the first camera.

python example_raw_output.py

The model is divided into the encoder (dynamic batch size) and decoder-head when exporting into ONNX.

python export_onnx.py

The code is taken from the official Dust3r repository which is distributed under the CC BY-NC-SA 4.0 License. See LICENSE for more information.

# Copyright (C) 2024-present Naver Corporation. All rights reserved.
# Licensed under CC BY-NC-SA 4.0 (non-commercial use only).