zed-playground/py_workspace/.sisyphus/notepads/aruco-svo-calibration/learnings.md

2026-02-04 Init

• Repo is script-first, but aruco/ imports work via Python namespace package even without __init__.py.
• No existing test suite. pytest is not installed; will need to be added (likely as dev dependency) before tests can run.
• Existing CLI patterns use click (e.g., streaming_receiver.py).

Pose Math Utilities

• Created aruco/pose_math.py for common SE(3) operations.

• Standardized on 4x4 homogeneous matrices for composition and inversion.

• Inversion uses the efficient property: [R | t; 0 | 1]^-1 = [R^T | -R^T * t; 0 | 1].

• Reprojection error calculation uses cv2.projectPoints and mean Euclidean distance.

• ArUco marker geometry loading and validation logic implemented in aruco/marker_geometry.py.

• Use awkward and pyarrow for efficient parquet loading of multi-dimensional arrays (corners).

• Reshaping ak.to_numpy output is necessary when dealing with nested structures like corners (4x3).

SVO Sync

• Added aruco/svo_sync.py with SVOReader and FrameData to open multiple SVO2 files, align starts by timestamp, and grab frames.
• Verified with real local SVO2 files: able to open, sync, and grab frames for 2 cameras.

ArUco Detector

• Added aruco/detector.py implementing:
  • ArUcoDetector creation (DICT_4X4_50)
  • marker detection (BGR or grayscale input)
  • ZED intrinsics -> K matrix extraction
  • multi-marker solvePnP pose estimation + reprojection error
• Verified pose estimation with synthetic projected points and with a real SVO-opened camera for intrinsics.
• Implemented PoseAccumulator in aruco/pose_averaging.py for robust SE(3) pose averaging.
• Added RANSAC-based filtering for rotation and translation consensus.
• Implemented quaternion eigen-mean fallback for rotation averaging when scipy is unavailable.
• Used median for robust translation averaging.

Task 6: ArUco Preview Helpers

• Implemented aruco/preview.py with draw_detected_markers, draw_pose_axes, and show_preview.

• Ensured grayscale images are converted to BGR before drawing to support colored overlays.

• Used cv2.drawFrameAxes for pose visualization.

• show_preview handles multiple windows based on camera serial numbers.

• Removed *.parquet from .gitignore to allow versioning of marker geometry data.

Unit Testing

• Added pytest as a dev dependency.
• Implemented synthetic tests for pose math and averaging.
• Discovered that OpenCV's projectPoints is strict about tvec being floating-point; ensured tests use dtype=np.float64.
• Verified that PoseAccumulator correctly filters outliers using RANSAC and computes robust means.

Pytest sys.path Pitfall

When running pytest via a console script (e.g., uv run pytest), the current working directory is not always automatically added to sys.path. This can lead to ModuleNotFoundError for local packages like aruco. Fix: Create a tests/conftest.py file that explicitly inserts the project root into sys.path:

import sys
import os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\")))

This ensures deterministic import behavior regardless of how pytest is invoked.