zed-playground/py_workspace/.sisyphus/notepads/depth-extrinsic-verify/learnings.md

Task 4 Complete: CLI Flags Added

Successfully added all required CLI flags:

• --verify-depth / --no-verify-depth
• --refine-depth / --no-refine-depth
• --depth-mode [NEURAL|ULTRA|PERFORMANCE|NONE]
• --depth-confidence-threshold
• --report-csv

All flags appear in --help output.

Next Steps

Tasks 5 and 6 require integrating depth verification and refinement into the main workflow:

• Task 5: Add depth verification logic to main() function
• Task 6: Add depth refinement logic to main() function
• Task 7: Create unit tests for depth modules

The integration needs to:

1. Parse depth_mode string to sl.DEPTH_MODE enum
2. Pass depth_mode to SVOReader when verify_depth or refine_depth is enabled
3. After computing extrinsics, run depth verification
4. If refine_depth is enabled, run optimization and update results
5. Add verification stats to output JSON
6. Optionally write CSV report

Task 5 Complete: Depth Verification Integration

• Refactored depth_verify.py:

  • Implemented compute_marker_corner_residuals to return detailed (marker_id, corner_idx, residual) tuples.
  • Added confidence map filtering logic: filters out points where ZED confidence > threshold (lower is better in ZED SDK).
  • Fixed compute_depth_residual to handle projection errors and bounds checking robustly.
  • Return type DepthVerificationResult now includes list of all residuals for CSV reporting.

• Updated svo_sync.py:

  • Added confidence_map to FrameData dataclass.
  • Updated SVOReader to retrieve both depth and confidence measures when depth mode is enabled.

• Integrated into calibrate_extrinsics.py:

  • Stores a "verification sample" (last valid frame with depth) during the processing loop.
  • Post-process verification: After T_mean is computed, runs verification using the stored sample and the final consensus pose.
  • Adds depth_verify block to the output JSON with RMSE, mean absolute error, and validity counts.
  • Writes detailed CSV report if --report-csv is provided.

• Verification:

  • uv run pytest -q passes (10 tests).
  • LSP checks pass for modified files.

Type Fixes in aruco/svo_sync.py

• Updated type annotations to use modern Python 3.12 syntax:
  • Replaced List, Dict, Optional with list, dict, and | None.
  • Added missing type arguments to generic classes (e.g., dict[str, Any]).
  • Added explicit class attribute annotations for SVOReader.
• Verified that lsp_diagnostics no longer reports reportMissingTypeArgument or reportDeprecated for this file.
• Confirmed that uv run pytest passes after changes.

Depth Refinement Integration

• Pattern: Post-process refinement is superior to per-frame refinement for extrinsic calibration.
  • Why: Per-frame refinement is noisy and computationally expensive. Refining the robust mean pose against a high-quality frame (or average of frames) yields more stable results.
  • Implementation: Store a representative frame (e.g., last valid frame with depth) during the loop, then run refinement once on the final aggregated pose.
• Verification: Always verify before and after refinement to quantify improvement.
  • Metrics: RMSE of depth residuals, delta rotation/translation.
  • Guardrails: Skip refinement if insufficient valid depth points are available (e.g., < 4 points).

Task 6 Complete: Depth Refinement Integration

• Refinement Logic:

  • Implemented post-process refinement: verifies initial pose, refines using L-BFGS-B optimization, then verifies refined pose.
  • Updates JSON output with refine_depth stats and depth_verify_post metrics.
  • Calculates and reports RMSE improvement.
  • Uses refined pose for final CSV report if enabled.

• Code Quality:

  • Cleaned up loop logic to remove per-frame refinement attempts (inefficient).
  • Added proper type hints (List, Dict, Any, Optional, Tuple) to satisfy LSP.
  • Removed agent memo comments.
  • Verified with pytest (18 passed) and calibrate_extrinsics.py --help.

• Key Implementation Details:

  • Refinement only runs if sufficient valid depth points (>4) exist.
  • Refined pose (T_refined) replaces the original RANSAC mean pose in the output JSON if refinement is successful.
  • Original RANSAC stats remain in stats field, while refinement deltas are in refine_depth.