zed-playground/py_workspace/.sisyphus/notepads/multi-frame-depth-pooling/learnings.md

Depth Pooling Implementation

• Implemented pool_depth_maps in aruco/depth_pool.py.
• Uses np.nanmedian for robust per-pixel depth pooling.
• Supports confidence gating (lower is better) and min_valid_count threshold.
• Handles N=1 case by returning a masked copy.
• Vectorized implementation using np.stack and boolean masking for performance.

2026-02-07: Depth Pooling Test Implementation

• Implemented comprehensive unit tests for pool_depth_maps in tests/test_depth_pool.py.
• Verified handling of:
  • Empty input and shape mismatches (ValueError).
  • Single map behavior (masked copy, min_valid_count check).
  • Median pooling logic with multiple maps.
  • Invalid depth values (<=0, non-finite).
  • Confidence gating (ZED semantics: lower is better).
  • min_valid_count enforcement across multiple frames.
• Type checking with basedpyright confirmed clean (after fixing unused call results and Optional handling in tests).

Task 4: CLI Option Wiring

• Added --depth-pool-size (1-10, default 1) to calibrate_extrinsics.py.
• Wired the option through main to apply_depth_verify_refine_postprocess.
• Maintained backward compatibility by defaulting to 1.
• Extended verification_frames to store a list of top-N frames per camera, sorted by score descending.
• Maintained backward compatibility by using the first frame in the list for current verification and benchmark logic.
• Added depth_pool_size parameter to main and passed it to apply_depth_verify_refine_postprocess.

2026-02-07: Multi-Frame Depth Pooling Integration

• Integrated pool_depth_maps into calibrate_extrinsics.py.
• Added --depth-pool-size CLI option (default 1).
• Implemented fallback logic: if pooled depth has < 50% valid points compared to best single frame, fallback to single frame.
• Added depth_pool metadata to JSON output.
• Verified N=1 equivalence with regression test tests/test_depth_pool_integration.py.
• Verified E2E smoke test:
  • Pool=1 vs Pool=5 showed mixed results on small sample (20 frames):
    • Camera 41831756: -0.0004m (Improved)
    • Camera 44289123: +0.0004m (Worse)
    • Camera 44435674: -0.0003m (Improved)
    • Camera 46195029: +0.0036m (Worse)
  • This variance is expected on small samples; pooling is intended for stability over larger datasets.
  • Runtime warning All-NaN slice encountered observed in nanmedian when some pixels are invalid in all frames; this is handled by nanmedian returning NaN, which is correct behavior for us.