yyfz/Pi4
Machine LearningReconstructs 4D scene geometry and camera motion from images or video in a single forward pass, for graphics and vision researchers.
What Pi4 does
Pi4 is a compact (~0.5B-parameter) foundation model that reconstructs scene geometry and estimates motion together in one forward pass. It accepts a single image, an unordered image collection, or a video without camera calibration, and outputs local point maps, camera poses, dense 2D/3D tracks, dynamic masks, and confidence values. The project targets research in 3D vision, scene reconstruction, and video tracking.
The model is trained on at most 64 views, but the README reports joint reconstruction and tracking from thousands of input images. It ships with model weights on Hugging Face, a browser viewer for exported results, and a Python API that handles normalization, inference precision, and gradient disabling internally.
Quickstart
Prepare a Linux x86-64 machine with Python 3.11, an NVIDIA GPU with BF16 support, and a CUDA 12.6-compatible driver. From the repository root:
conda create -n pi4 python=3.11 -y
conda activate pi4
python -m pip install --upgrade pip
python -m pip install -r requirements-cu126.txt
python -m pip check
python -c "import torch; assert torch.cuda.is_available(), 'CUDA is not available'; print(torch.__version__, torch.version.cuda)"
This installs PyTorch 2.12.1 with CUDA 12.6, Triton 3.7.1, and the pinned FlexGEMM backend; Git and a C compiler are required. See the installation notes for other CUDA builds and checks.
Run reconstruction and tracking on the bundled figure-skating clip:
python example.py examples/figure_skating.mp4 \
--output outputs/quickstart \
--query-frame 0 --stride 1 --max-frames 150 --viewer
The example downloads and caches the checkpoint from Hugging Face automatically, or you can pass a local file with --weights. Then view the exported results in a browser:
python viewer/serve.py outputs/quickstart --port 8080
Open the printed address to orbit the reconstruction, inspect camera motion and point tracks, or play the sequence alongside the RGB input. Viewing requires no GPU or model weights. For the Python API, load frames with load_images, normalize to floating-point RGB in [0, 1] shaped [B, T, 3, H, W] with height and width divisible by 16, and call the model with query_frame. Omit query_frame for geometry-only inference. The inference guide documents tensor shapes, coordinate conventions, and export formats.
Constraints
- Inference needs an NVIDIA GPU with BF16 support; the 150-frame configuration measured 12.03 GiB peak GPU memory on an A800 80 GB. Smaller GPUs require reducing frame count, and the full 750-frame demo needs substantially more memory.
- Tracking requires at least two images.
- The bundled clips are 512 × 288 or 288 × 512;
--stridesamples frames at load time. - Outputs use the model's native scale for point maps and camera translations, with 3D tracks in predicted metric world coordinates.
- Original Pi4 code and model weights are licensed CC BY-NC 4.0, which does not permit commercial use. Third-party components keep their upstream licenses.
HotRepo's take
In its first day on the new-project list, Pi4 recorded 117 stars and 3 forks, gaining 12 stars in the roughly one-hour window between its first and last star observations on 2026-11-18. That is a strong opening for a research release with no recorded discussion threads and no earlier ranking history. With a 0.5B model, weights on Hugging Face, and a viewer that runs without a GPU, the project lowers the barrier to trying 4D reconstruction while its non-commercial license limits production use.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- zju3dv/SplatAnythingFeed-forward 3D Gaussian splatting model that reconstructs objects, rooms, streets, rigs and panoramas from images.
- IfcOpenShell/IfcOpenShellAn open-source IFC library and geometry engine for software developers and AEC professionals working with building models.
- shiyu-coder/KronosA pretrained foundation model that forecasts K-line price and volume sequences for quantitative researchers and traders.