feat: initialize offline multiview pose tracking experiment
Set up pose_tracking_exp as a uv-managed Python package for offline multiview body tracking experiments. This initial commit includes: - the typed package scaffold, CLI entrypoints, and repo-local uv configuration - scene and replay loaders for generic JSON replays and ActualTest parquet inputs - ParaJumping payload conversion and RTMPose-to-body20 normalization - a custom articulated tracker with tentative, active, and lost lifecycle handling - RPT-backed proposal generation, camera convention handling, and multiview reprojection updates - regression tests for normalization, camera conventions, ActualTest ingestion, seeding, and tracker smoke flows - project documentation covering extrinsic formats and the ActualTest calibration caveat
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
# pose_tracking_exp
|
||||
|
||||
Offline multiview body tracking experiments built around:
|
||||
|
||||
- `RapidPoseTriangulation` for geometric birth proposals
|
||||
- a typed replay format for recorded per-camera detections
|
||||
- a recursive active/lost tracker with fixed bone lengths
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
uv run pose-tracking-exp run data/scene.json data/replay.jsonl
|
||||
```
|
||||
|
||||
`scene.json` may declare camera extrinsics in either format:
|
||||
|
||||
- `opencv_world_to_camera`: OpenCV `solvePnP` / `cv2.projectPoints` convention. This is the default.
|
||||
- `rpt_camera_pose`: camera pose in world coordinates, which is what `RapidPoseTriangulation` expects internally.
|
||||
|
||||
The loader normalizes both to OpenCV extrinsics for reprojection and converts to RPT pose only when building the triangulation config.
|
||||
If you already have an older hand-authored scene file that stored RPT camera pose directly, set `extrinsic_format` explicitly to `rpt_camera_pose`.
|
||||
|
||||
## Convert ParaJumping Payload Records
|
||||
|
||||
```bash
|
||||
uv run pose-tracking-exp convert-parajumping input.jsonl output.jsonl
|
||||
```
|
||||
|
||||
## ActualTest Calibration Caveat
|
||||
|
||||
`ActualTest_WeiHua/camera_params.parquet` appears to store raw OpenCV extrinsics from the ChArUco pipeline, not camera poses. The tracker now converts those values before calling `RapidPoseTriangulation`, because RPT expects camera centers and camera-to-world rotation.
|
||||
|
||||
In repo terms:
|
||||
|
||||
- OpenCV reprojection keeps `R`, `T`, and `rvec` as world-to-camera extrinsics.
|
||||
- RPT export uses the derived camera pose `pose_R = R^T` and `pose_T = -R^T t`.
|
||||
|
||||
There is still one upstream caveat: the ParaJumping calibration notebook averages `rvec` samples component-wise before writing the parquet. That is a rough approximation for rotations and can introduce some bias even when the convention is handled correctly.
|
||||
Reference in New Issue
Block a user