zed-playground/py_workspace/docs/icp-depth-bias-diagnosis.md

ICP Depth Bias Diagnosis Report

Executive Summary

Recent diagnostics indicate that while geometric overlap between camera pairs is high (~71%–80%), significant cross-camera depth bias is the primary blocker for ICP convergence. The overlap regions exhibit acceptable planarity, suggesting that the issue is not a degenerate scene geometry but rather a systematic inconsistency in depth measurement between specific camera units.

Measured Diagnostics

1. Shared FOV Overlap Proxies

Geometric overlap is sufficient across the cluster.

• Range: 0.707 – 0.799
• Pair Minima: ~0.71
• Caution: Geometric overlap \neq usable overlap. High overlap values do not guarantee ICP success if depth noise or bias is high.

2. Valid Depth Ratios

Percentage of pixels with valid depth data per camera:

• 41831756: 0.871
• 44289123: 0.870
• 44435674: 0.789
• 46195029: 0.805

3. Overlap Region Planarity

Measured via \lambda_3 / \sum \lambda_i (ratio of smallest eigenvalue to sum of eigenvalues):

• Mean Range: 0.136 – 0.170
• Interpretation: The overlap regions are not grossly degenerate (not perfectly planar, but sufficiently structured for ICP).

4. Signed Residual Symmetric Bias Ranking

Median absolute signed residuals between camera pairs (sorted by worst inconsistency):

1. 44289123 - 44435674: 0.137m
2. 41831756 - 44435674: 0.115m
3. 44435674 - 46195029: 0.098m
4. 41831756 - 46195029: 0.095m
5. 44289123 - 46195029: 0.082m
6. 41831756 - 44289123: 0.038m

Oracle Interpretation

The strongest evidence points to cross-camera depth bias.

• Camera 44435674 is involved in the top 3 most biased pairs, suggesting it is the primary outlier in depth scale or offset.
• The bias (up to 13.7cm) is significantly larger than the expected noise floor for ZED cameras at typical ranges, which prevents ICP from finding a stable global minimum.

What this implies for ICP behavior

• ICP will likely "drift" or fail to converge because the point clouds from different cameras do not represent the same physical surface at the same coordinates.
• Residuals will remain high even at the "optimal" alignment.
• Point cloud "ghosting" or double-surfaces will be visible in fused outputs.

Remediation Plan (3-Step)

1. Step 1: Per-Camera Depth Validation (Go/No-Go)

   • Measure a known flat target at a fixed distance (e.g., 2m) with each camera.
   • Go Criteria: Absolute depth error < 2% of distance.
   • No-Go: If error > 5%, recalibrate internal camera parameters or apply a linear scale correction.

2. Step 2: Pairwise Scale Alignment

   • Use the least-biased pair (41831756-44289123) as the "golden" reference.
   • Calculate a scale/offset correction factor for 44435674 to minimize the 13.7cm median residual.

3. Step 3: Constrained ICP with Bias Compensation

   • Re-run ICP using the corrected depth maps.
   • Success Criteria: Median residuals drop below 0.05m across all pairs.

Recommended Immediate Next Diagnostic

Perform a Static Target Depth Sweep. Place a planar target in the center of the shared FOV and record 100 frames from all cameras. Calculate the per-camera mean distance to the plane to isolate the absolute offset per unit.

Remediation Applied (2026-02-11)

Automatic depth bias estimation and pre-correction has been implemented and integrated into the refine_ground_plane.py pipeline.

Outcome:

• Bias-enabled run: Successfully optimized 1 non-reference camera that previously failed to converge due to depth inconsistency.
• No-bias run: Optimized 0 cameras (baseline), confirming that depth bias was indeed the primary blocker for this dataset.

The system now estimates per-camera offsets relative to a reference unit and applies them to the point clouds before ICP registration, significantly improving robustness to unit-to-unit depth variations.