zed-playground/py_workspace/docs/calibrate-extrinsics-workflow.md

Calibrate Extrinsics Workflow

This document explains the workflow for calibrate_extrinsics.py, focusing on ground plane alignment (--auto-align) and depth-based refinement (--verify-depth, --refine-depth).

CLI Overview

The script calibrates camera extrinsics using ArUco markers detected in SVO recordings.

Key Options:

• --svo: Path to SVO file(s) or directory containing them.
• --markers: Path to the marker configuration parquet file.
• --auto-align: Enables automatic ground plane alignment (opt-in).
• --verify-depth: Enables depth-based verification of computed poses.
• --refine-depth: Enables optimization of poses using depth data (requires --verify-depth).
• --max-samples: Limits the number of processed samples for fast iteration.
• --debug: Enables verbose debug logging (default is INFO).

Ground Plane Alignment (--auto-align)

When --auto-align is enabled, the script attempts to align the global coordinate system such that a specific face of the marker object becomes the ground plane (XZ plane, normal pointing +Y).

Prerequisites:

• The marker parquet file MUST contain name and ids columns defining which markers belong to which face (e.g., "top", "bottom", "front").
• If this metadata is missing, alignment is skipped with a warning.

Decision Flow: The script selects the ground face using the following precedence:

1. Explicit Face (--ground-face):

   • If you provide --ground-face="bottom", the script looks up the markers for "bottom" in the loaded map.
   • It computes the average normal of those markers and aligns it to the global up vector.

2. Marker ID Mapping (--ground-marker-id):

   • If you provide --ground-marker-id=21, the script finds which face contains marker 21 (e.g., "bottom").
   • It then proceeds as if --ground-face="bottom" was specified.

3. Heuristic Detection (Fallback):

   • If neither option is provided, the script analyzes all visible markers.
   • It computes the normal for every defined face.
   • It selects the face whose normal is most aligned with the camera's "down" direction (assuming the camera is roughly upright).

Logging: The script logs the selected decision path for debugging:

• Mapped ground-marker-id 21 to face 'bottom' (markers=[21])
• Using explicit ground face 'bottom' (markers=[21])
• Heuristically detected ground face 'bottom' (markers=[21])

Depth Verification & Refinement

This workflow uses the ZED camera's depth map to verify and improve the ArUco-based pose estimation.

1. Verification (--verify-depth)

• Input: The computed extrinsic pose (T_{world\_from\_cam}) and the known 3D world coordinates of the marker corners.
• Process:
  1. Projects marker corners into the camera frame using the computed pose.
  2. Samples the ZED depth map at these projected 2D locations (using a 5x5 median filter for robustness).
  3. Compares the measured depth (ZED) with the computed depth (distance from camera center to projected corner).
• Output:
  • RMSE (Root Mean Square Error) of the depth residuals.
  • Number of valid points (where depth was available and finite).
  • Added to JSON output under depth_verify.

2. Refinement (--refine-depth)

• Trigger: Runs only if verification is enabled and enough valid depth points (>4) are found.
• Process:
  • Uses scipy.optimize.minimize (L-BFGS-B) to adjust the 6-DOF pose parameters (rotation vector + translation vector).
  • Objective Function: Minimizes the squared difference between computed depth and measured depth for all visible marker corners.
  • Constraints: Bounded optimization to prevent drifting too far from the initial ArUco pose (default: ±5 degrees, ±5cm).
• Output:
  • Refined pose replaces the original pose in the JSON output.
  • Improvement stats (delta rotation, delta translation, RMSE reduction) added under refine_depth.

Fast Iteration (--max-samples)

For development or quick checks, processing thousands of frames is unnecessary.

• Use --max-samples N to stop after N valid samples (frames where markers were detected).
• Example: --max-samples 1 will process the first valid frame, run alignment/refinement, save the result, and exit.

Example Workflow

Full Run with Alignment and Refinement:

uv run calibrate_extrinsics.py \
  --svo output/recording.svo \
  --markers aruco/markers/box.parquet \
  --aruco-dictionary DICT_APRILTAG_36h11 \
  --auto-align \
  --ground-marker-id 21 \
  --verify-depth \
  --refine-depth \
  --output output/calibrated.json

Fast Debug Run:

uv run calibrate_extrinsics.py \
  --svo output/ \
  --markers aruco/markers/box.parquet \
  --auto-align \
  --max-samples 1 \
  --debug \
  --no-preview