crosstyan
f1a2372b3c
Set up pose_tracking_exp as a uv-managed Python package for offline multiview body tracking experiments. This initial commit includes: - the typed package scaffold, CLI entrypoints, and repo-local uv configuration - scene and replay loaders for generic JSON replays and ActualTest parquet inputs - ParaJumping payload conversion and RTMPose-to-body20 normalization - a custom articulated tracker with tentative, active, and lost lifecycle handling - RPT-backed proposal generation, camera convention handling, and multiview reprojection updates - regression tests for normalization, camera conventions, ActualTest ingestion, seeding, and tracker smoke flows - project documentation covering extrinsic formats and the ActualTest calibration caveat
pose_tracking_exp
Offline multiview body tracking experiments built around:
RapidPoseTriangulationfor geometric birth proposals- a typed replay format for recorded per-camera detections
- a recursive active/lost tracker with fixed bone lengths
Install
uv sync --extra dev
Run
uv run pose-tracking-exp run data/scene.json data/replay.jsonl
scene.json may declare camera extrinsics in either format:
opencv_world_to_camera: OpenCVsolvePnP/cv2.projectPointsconvention. This is the default.rpt_camera_pose: camera pose in world coordinates, which is whatRapidPoseTriangulationexpects internally.
The loader normalizes both to OpenCV extrinsics for reprojection and converts to RPT pose only when building the triangulation config.
If you already have an older hand-authored scene file that stored RPT camera pose directly, set extrinsic_format explicitly to rpt_camera_pose.
Convert ParaJumping Payload Records
uv run pose-tracking-exp convert-parajumping input.jsonl output.jsonl
ActualTest Calibration Caveat
ActualTest_WeiHua/camera_params.parquet appears to store raw OpenCV extrinsics from the ChArUco pipeline, not camera poses. The tracker now converts those values before calling RapidPoseTriangulation, because RPT expects camera centers and camera-to-world rotation.
In repo terms:
- OpenCV reprojection keeps
R,T, andrvecas world-to-camera extrinsics. - RPT export uses the derived camera pose
pose_R = R^Tandpose_T = -R^T t.
There is still one upstream caveat: the ParaJumping calibration notebook averages rvec samples component-wise before writing the parquet. That is a rough approximation for rotations and can introduce some bias even when the convention is handled correctly.