refactor: things

workspaces/py_workspace/.sisyphus/notepads/depth-extrinsic-verify/issues.md

@@ -0,0 +1,6 @@
+## 2026-02-05
+
+### Verification dataset missing / no detections
+- Current bundled SVOs (ZED_SN*.svo2) appear to produce 0 ArUco detections in `calibrate_extrinsics.py`.
+- This blocks end-to-end verification that output JSON includes `depth_verify`, `depth_verify_post`, and `refine_depth`, and that `--report-csv` emits rows.
+- Unit tests cover depth verification/refinement math and bounds, but integration output fields need a dataset with visible markers.

workspaces/py_workspace/.sisyphus/notepads/depth-extrinsic-verify/learnings.md

@@ -0,0 +1,82 @@
+## Task 4 Complete: CLI Flags Added
+
+Successfully added all required CLI flags:
+- --verify-depth / --no-verify-depth
+- --refine-depth / --no-refine-depth  
+- --depth-mode [NEURAL|ULTRA|PERFORMANCE|NONE]
+- --depth-confidence-threshold
+- --report-csv
+
+All flags appear in --help output.
+
+## Next Steps
+
+Tasks 5 and 6 require integrating depth verification and refinement into the main workflow:
+- Task 5: Add depth verification logic to main() function
+- Task 6: Add depth refinement logic to main() function
+- Task 7: Create unit tests for depth modules
+
+The integration needs to:
+1. Parse depth_mode string to sl.DEPTH_MODE enum
+2. Pass depth_mode to SVOReader when verify_depth or refine_depth is enabled
+3. After computing extrinsics, run depth verification
+4. If refine_depth is enabled, run optimization and update results
+5. Add verification stats to output JSON
+6. Optionally write CSV report
+
+## Task 5 Complete: Depth Verification Integration
+
+- **Refactored `depth_verify.py`**:
+  - Implemented `compute_marker_corner_residuals` to return detailed (marker_id, corner_idx, residual) tuples.
+  - Added confidence map filtering logic: filters out points where ZED confidence > threshold (lower is better in ZED SDK).
+  - Fixed `compute_depth_residual` to handle projection errors and bounds checking robustly.
+  - Return type `DepthVerificationResult` now includes list of all residuals for CSV reporting.
+
+- **Updated `svo_sync.py`**:
+  - Added `confidence_map` to `FrameData` dataclass.
+  - Updated `SVOReader` to retrieve both depth and confidence measures when depth mode is enabled.
+
+- **Integrated into `calibrate_extrinsics.py`**:
+  - Stores a "verification sample" (last valid frame with depth) during the processing loop.
+  - Post-process verification: After `T_mean` is computed, runs verification using the stored sample and the final consensus pose.
+  - Adds `depth_verify` block to the output JSON with RMSE, mean absolute error, and validity counts.
+  - Writes detailed CSV report if `--report-csv` is provided.
+
+- **Verification**:
+  - `uv run pytest -q` passes (10 tests).
+  - LSP checks pass for modified files.
+### Type Fixes in aruco/svo_sync.py
+- Updated type annotations to use modern Python 3.12 syntax:
+  - Replaced `List`, `Dict`, `Optional` with `list`, `dict`, and `| None`.
+  - Added missing type arguments to generic classes (e.g., `dict[str, Any]`).
+  - Added explicit class attribute annotations for `SVOReader`.
+- Verified that `lsp_diagnostics` no longer reports `reportMissingTypeArgument` or `reportDeprecated` for this file.
+- Confirmed that `uv run pytest` passes after changes.
+
+## Depth Refinement Integration
+- **Pattern**: Post-process refinement is superior to per-frame refinement for extrinsic calibration.
+  - **Why**: Per-frame refinement is noisy and computationally expensive. Refining the robust mean pose against a high-quality frame (or average of frames) yields more stable results.
+  - **Implementation**: Store a representative frame (e.g., last valid frame with depth) during the loop, then run refinement once on the final aggregated pose.
+- **Verification**: Always verify *before* and *after* refinement to quantify improvement.
+  - **Metrics**: RMSE of depth residuals, delta rotation/translation.
+  - **Guardrails**: Skip refinement if insufficient valid depth points are available (e.g., < 4 points).
+
+## Task 6 Complete: Depth Refinement Integration
+
+- **Refinement Logic**:
+  - Implemented post-process refinement: verifies initial pose, refines using L-BFGS-B optimization, then verifies refined pose.
+  - Updates JSON output with `refine_depth` stats and `depth_verify_post` metrics.
+  - Calculates and reports RMSE improvement.
+  - Uses refined pose for final CSV report if enabled.
+
+- **Code Quality**:
+  - Cleaned up loop logic to remove per-frame refinement attempts (inefficient).
+  - Added proper type hints (`List`, `Dict`, `Any`, `Optional`, `Tuple`) to satisfy LSP.
+  - Removed agent memo comments.
+  - Verified with `pytest` (18 passed) and `calibrate_extrinsics.py --help`.
+
+- **Key Implementation Details**:
+  - Refinement only runs if sufficient valid depth points (>4) exist.
+  - Refined pose (`T_refined`) replaces the original RANSAC mean pose in the output JSON if refinement is successful.
+  - Original RANSAC stats remain in `stats` field, while refinement deltas are in `refine_depth`.
+- Opt-in depth behavior: Depth computation is now explicitly disabled in `calibrate_extrinsics.py` unless `--verify-depth` or `--refine-depth` is set. This prevents unnecessary GPU/CPU overhead during standard extrinsic calibration.

workspaces/py_workspace/.sisyphus/notepads/depth-extrinsic-verify/problems.md

@@ -0,0 +1,10 @@
+## 2026-02-05
+
+### Blocker: integration output validation needs a marker-visible SVO
+
+To complete remaining plan checkboxes:
+- "--verify-depth produces verification metrics in output JSON"
+- "--refine-depth optimizes extrinsics and reports pre/post metrics"
+- "--report-csv outputs per-frame residuals to CSV file"
+
+Need at least one SVO set where ArUco markers are detected and poses accumulate.