diff --git a/py_workspace/.sisyphus/drafts/depth-extrinsic-verify.md b/py_workspace/.sisyphus/drafts/depth-extrinsic-verify.md new file mode 100644 index 0000000..d85f196 --- /dev/null +++ b/py_workspace/.sisyphus/drafts/depth-extrinsic-verify.md @@ -0,0 +1,55 @@ +# Draft: Depth-Based Extrinsic Verification/Fusion + +## Requirements (confirmed) + +- **Primary Goal**: Both verify AND refine extrinsics using depth data +- **Integration**: Add to existing `calibrate_extrinsics.py` CLI (new flags) +- **Depth Mode**: CLI argument with default to NEURAL_PLUS (or NEURAL) +- **Target Geometry**: Any markers (from parquet file), not just ArUco box + +## Technical Decisions + +- Use ZED SDK `retrieve_measure(MEASURE.DEPTH)` for depth maps +- Extend `SVOReader` to optionally enable depth mode +- Compute depth residuals at detected marker corner positions +- Use residual statistics for verification metrics +- ICP or optimization for refinement (if requested) + +## Research Findings + +### Depth Residual Formula +For 3D point P_world with camera extrinsics (R, t): +``` +P_cam = R @ P_world + t +z_predicted = P_cam[2] +(u, v) = project(P_cam, K) +z_measured = depth_map[v, u] +residual = z_measured - z_predicted +``` + +### Verification Metrics +- Mean absolute residual +- RMSE +- Depth-normalized error: |r| / z_pred +- Spatial bias detection (residual vs pixel position) + +### Refinement Approach +- ICP (Iterative Closest Point) on depth points near markers +- Point-to-plane ICP for better convergence +- Initialize with ArUco pose, refine with depth + +## User Decisions (Round 2) + +- **Refinement Method**: Direct optimization (minimize depth residuals to adjust extrinsics) +- **Verification Output**: Full reporting (console + JSON + optional CSV) +- **Depth Filtering**: Confidence-based (use ZED confidence threshold + range limits) + +## Open Questions + +- Test strategy: TDD or tests after? +- Minimum markers/frames for reliable depth verification? + +## Scope Boundaries + +- INCLUDE: Depth retrieval, residual computation, verification metrics, optional ICP refinement +- EXCLUDE: Bundle adjustment, SLAM, right camera processing