chore: checkpoint ground-plane calibration refinement work
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# Draft: Ground Plane Refinement & Depth Map Persistence
|
||||
|
||||
## Requirements (confirmed)
|
||||
- **Core problem**: Camera disagreement — different cameras don't agree on where the ground is (floor at different heights/angles)
|
||||
- **Depth saving**: Save BOTH pooled depth maps AND raw best-scored frames per camera, so pooling parameters can be re-tuned without re-reading SVOs
|
||||
- **Integration**: Post-processing step — a new standalone CLI tool that loads existing extrinsics + saved depth data and refines
|
||||
- **Library**: TBD — user wants to understand trade-offs before committing
|
||||
|
||||
## Technical Decisions
|
||||
- Post-processing approach: non-invasive, loads existing calibration JSON + depth data
|
||||
- Depth saving happens inside calibrate_extrinsics.py (or triggered by flag)
|
||||
- Ground refinement tool is a NEW script (e.g., `refine_ground_plane.py`)
|
||||
|
||||
## Research Findings
|
||||
- **Current alignment.py**: Aligns world frame based on marker face normals, NOT actual floor geometry
|
||||
- **Current depth_pool.py**: Per-pixel median pooling exists, but result is discarded after use (never saved)
|
||||
- **Current depth_refine.py**: Optimizes 6-DOF per camera using depth at marker corners only (sparse)
|
||||
- **compare_pose_sets.py**: Has Kabsch `rigid_transform_3d()` for point-set alignment
|
||||
- **Available deps**: numpy, scipy, opencv — sufficient for RANSAC plane fitting
|
||||
- **Open3D**: Provides ICP, RANSAC, visualization but is ~500MB heavy dep
|
||||
|
||||
## Open Questions (Resolved)
|
||||
- **Camera count**: 2-4 cameras (small setup, likely some floor overlap)
|
||||
- **Observation method**: Point clouds don't align when overlayed in world coords
|
||||
- **Error magnitude**: Small — 1-3° tilt, <2cm offset (fine-tuning level)
|
||||
- **Floor type**: TBD (assumed flat for now)
|
||||
- **Library choice**: TBD — recommendation below
|
||||
|
||||
## Library Recommendation Analysis
|
||||
Given: 2-4 cameras, small errors, flat floor assumption, post-processing tool
|
||||
|
||||
**numpy/scipy approach**:
|
||||
- RANSAC plane fitting: trivial with numpy (random sample 3 points, fit plane, count inliers)
|
||||
- Plane-to-plane alignment: rotation_align_vectors already exists in alignment.py
|
||||
- Point cloud generation from depth+intrinsics: simple numpy vectorized operation
|
||||
- Kabsch alignment: already exists in compare_pose_sets.py
|
||||
- Verdict: **SUFFICIENT for this use case**. No ICP needed since we're fitting to a known target (Y=0 plane).
|
||||
|
||||
**Open3D approach**:
|
||||
- Overkill for plane fitting + rotation correction
|
||||
- Would be useful if we needed dense ICP between overlapping point clouds
|
||||
- 500MB dep for what amounts to ~50 lines of numpy code
|
||||
- Verdict: **Not needed for the initial version**
|
||||
|
||||
**Decision**: Use Open3D for point cloud operations (user wants it available for future work).
|
||||
Also add h5py for HDF5 depth map persistence.
|
||||
|
||||
## Confirmed Technical Choices
|
||||
- **Library**: Open3D (RANSAC plane segmentation, ICP if needed, point cloud ops)
|
||||
- **Depth save format**: HDF5 via h5py (structured, metadata-rich, one file per camera)
|
||||
- **Visualization**: Plotly HTML (interactive 3D — floor points per camera, consensus plane, before/after)
|
||||
- **Integration**: Standalone post-processing CLI tool (click-based, like existing tools)
|
||||
- **Error handling**: numpy/scipy for math, Open3D for geometry, existing alignment.py patterns
|
||||
|
||||
## Algorithm (confirmed via research + codebase analysis)
|
||||
1. Load existing extrinsics JSON + saved depth maps (HDF5)
|
||||
2. Per camera: unproject depth → world-coord point cloud using extrinsics
|
||||
3. Per camera: Open3D RANSAC plane segmentation → extract floor points
|
||||
4. Consensus: fit a single plane to ALL floor points from all cameras
|
||||
5. Compute correction rotation: align consensus plane normal to [0, -1, 0]
|
||||
6. Apply correction to all extrinsics (global rotation, like current alignment.py)
|
||||
7. Optionally: per-camera ICP refinement on overlapping floor regions
|
||||
8. Save corrected extrinsics JSON + generate diagnostic Plotly visualization
|
||||
|
||||
## Final Decisions (all confirmed)
|
||||
- **Depth save trigger**: `--save-depth <dir>` flag in calibrate_extrinsics.py
|
||||
- **Refinement granularity**: Per-camera refinement (each camera corrected based on its floor obs)
|
||||
- **Test strategy**: TDD — write tests first, following existing test patterns in tests/
|
||||
|
||||
## Scope Boundaries
|
||||
- INCLUDE: Depth map saving (HDF5), ground plane detection per camera, consensus plane fitting, per-camera extrinsic correction
|
||||
- INCLUDE: Standalone post-processing CLI tool (`refine_ground_plane.py`)
|
||||
- INCLUDE: Plotly diagnostic visualization
|
||||
- INCLUDE: TDD with pytest
|
||||
- INCLUDE: New deps: open3d, h5py
|
||||
- EXCLUDE: Modifying the core ArUco detection or PnP pipeline
|
||||
- EXCLUDE: Real-time / streaming refinement
|
||||
- EXCLUDE: Non-flat floor handling (ramps, stairs)
|
||||
- EXCLUDE: Dense multi-view reconstruction beyond floor plane
|
||||
Reference in New Issue
Block a user