RapidPoseTriangulation/README.md

RapidPoseTriangulation

Fast multi-view multi-person pose reconstruction, packaged as a Python-first C++ library.

This repository started from the original upstream RapidPoseTriangulation repository:

• Percipiote/RapidPoseTriangulation

The current fork keeps the triangulation core, exposes it through nanobind, and adds an RGB-D reconstruction path ported from the original SimpleDepthPose repository:

• Percipiote/SimpleDepthPose

What This Repository Is Now

• A packaged library named rapid-pose-triangulation with Python bindings under rpt
• A C++ core built with scikit-build-core and nanobind
• A triangulation library for calibrated multi-view 2D detections
• An RGB-D reconstruction helper layer that samples aligned depth, applies joint offsets, lifts poses into world coordinates, and merges per-view proposals

Current package status:

• Python >=3.10
• Runtime dependencies: NumPy, jaxtyping
• Current version: 0.2.0

Current Capabilities

The public Python API exposed by rpt currently includes:

• Camera/config helpers: make_camera, convert_cameras, make_triangulation_config
• Input preparation: pack_poses_2d
• Triangulation: triangulate_poses, triangulate_debug, triangulate_with_report
• Tracking/debug helpers: filter_pairs_with_previous_poses
• RGB-D helpers: sample_depth_for_poses, apply_depth_offsets, lift_depth_poses_to_world, merge_rgbd_views, reconstruct_rgbd

At a high level there are now two supported reconstruction paths:

1. Multi-view RGB triangulation from calibrated 2D detections
2. Multi-view RGB-D reconstruction from calibrated 2D detections plus aligned depth images

Installation And Development Workflow

Clone the repo and use uv for local development:

git clone https://git.weihua-iot.cn/crosstyan/RapidPoseTriangulation.git
cd RapidPoseTriangulation
uv sync --group dev

Run the test suite:

uv run pytest -q

Run static typing checks against the Python package:

uv run basedpyright

Build source and wheel artifacts:

uv build

run_container.sh is still present in the repo, but it is a leftover helper script rather than the primary or best-supported development workflow.

Typing Workflow

The Python package ships a typed facade in src/rpt plus a checked-in stub for the compiled nanobind module at src/rpt/_core.pyi.

Refresh the extension stub after changing the bindings:

cmake --build build --target rpt_core_stub
cp build/bindings/rpt/_core.pyi src/rpt/_core.pyi
uv run basedpyright

tests/test_typing_artifacts.py checks that the checked-in _core.pyi matches the generated nanobind stub whenever the build artifact is available.

Python API Overview

Typical triangulation flow:

import numpy as np
import rpt

cameras = rpt.convert_cameras(raw_cameras)
config = rpt.make_triangulation_config(
    cameras,
    roomparams=np.asarray([[5.6, 6.4, 2.4], [0.0, -0.5, 1.2]], dtype=np.float32),
    joint_names=joint_names,
)

poses_2d, person_counts = rpt.pack_poses_2d(views, joint_count=len(joint_names))
poses_3d = rpt.triangulate_poses(poses_2d, person_counts, config)

Typical RGB-D flow:

poses_2d, person_counts = rpt.pack_poses_2d(views, joint_count=len(joint_names))
poses_3d = rpt.reconstruct_rgbd(
    poses_2d,
    person_counts,
    depth_images,
    config,
    use_depth_offsets=True,
)

The lower-level RGB-D helpers are also available if you want to inspect or customize the intermediate steps:

• sample_depth_for_poses: sample aligned depth around visible 2D joints
• apply_depth_offsets: add per-joint offsets derived from SimpleDepthPose
• lift_depth_poses_to_world: convert [u, v, d, score] joints into world-space [x, y, z, score]
• merge_rgbd_views: merge per-view world-space pose proposals into final poses

Ported From SimpleDepthPose

This fork ports the RGB-D fusion path from SimpleDepthPose into rpt.

Original upstream repository:

• Percipiote/SimpleDepthPose

The ported pieces are:

• Depth sampling around each visible 2D joint, based on the add_depth preprocessing flow
• Per-joint depth offsets, matching the SimpleDepthPose body-surface correction idea
• UVD-to-world lifting using the calibrated camera intrinsics/extrinsics
• Multi-view RGB-D pose fusion logic adapted from PoseFuser

Compared with the original SimpleDepthPose implementation, the port here has been changed to fit a reusable library:

• The workflow is exposed as stateless functions instead of script-driven pipelines
• The fusion logic lives in the rpt core instead of a separate wrapper class
• Camera and scene configuration are routed through TriangulationConfig
• The RGB-D path is covered by repo tests and packaged with the same Python API as the triangulation path

This repo does not attempt to port the full SimpleDepthPose project. It only ports the RGB-D reconstruction pieces that fit the current library scope.

Changed Vs Upstream RapidPoseTriangulation

Compared with the original upstream repository below, this fork has materially changed structure and scope:

• Percipiote/RapidPoseTriangulation
• SWIG bindings were replaced with nanobind
• The repo was converted into a Python package under src/rpt
• The triangulation interface was simplified around immutable cameras and config structs
• The core was reshaped into a more library-oriented, zero-copy style API
• Debug tracing and tracked association reports were added
• Upstream integration layers and extra tooling were removed, including the old extras/ stack and related deployment/inference wrappers
• An RGB-D reconstruction pipeline was added by porting and adapting parts of SimpleDepthPose

In practice, upstream is closer to a larger project tree with integrations and historical tooling, while this fork is closer to a compact reconstruction library.

Testing

The repo currently ships Python-facing tests for both triangulation and RGB-D reconstruction:

uv run pytest tests/test_interface.py
uv run pytest tests/test_rgbd.py

Or run everything:

uv run pytest -q

The checked-in sample data under data/ is used by the triangulation tests.

Upstream References

Original upstream repositories referenced by this fork:

• RapidPoseTriangulation: Percipiote/RapidPoseTriangulation
• SimpleDepthPose: Percipiote/SimpleDepthPose