feat(calibration): robust depth refinement pipeline with diagnostics and benchmarking
This commit is contained in:
@@ -1,113 +1,178 @@
|
||||
# Agent Context & Reminders
|
||||
# AGENTS.md — Repository Guide for Coding Agents
|
||||
|
||||
## ZED SDK Architecture
|
||||
This file is for autonomous/agentic coding in `/workspaces/zed-playground`.
|
||||
Primary active Python workspace: `/workspaces/zed-playground/py_workspace`.
|
||||
|
||||
### Streaming API vs Fusion API
|
||||
---
|
||||
|
||||
The ZED SDK provides two distinct network APIs that are often confused:
|
||||
## 1) Environment & Scope
|
||||
|
||||
| Feature | Streaming API | Fusion API |
|
||||
|---------|---------------|------------|
|
||||
| **Data Transmitted** | Compressed video (H264/H265) | Metadata only (bodies, objects, poses) |
|
||||
| **Bandwidth** | 10-40 Mbps | <100 Kbps |
|
||||
| **Edge Compute** | Video encoding only | Full depth NN + tracking + detection |
|
||||
| **Host Compute** | Full depth + tracking + detection | Lightweight fusion only |
|
||||
| **API Methods** | `enableStreaming()` / `setFromStream()` | `startPublishing()` / `subscribe()` |
|
||||
- Python package manager: **uv**
|
||||
- Python version: **3.12+**
|
||||
- Core deps (py_workspace): `pyzed`, `opencv-python`, `click`, `numpy`, `scipy`, `loguru`, `awkward`, `jaxtyping`
|
||||
- Dev deps: `pytest`, `basedpyright`
|
||||
- Treat `py_workspace/loguru/` and `py_workspace/tmp/` as non-primary project areas unless explicitly asked.
|
||||
|
||||
### Key Insight
|
||||
---
|
||||
|
||||
**There is NO built-in mode for streaming computed depth maps or point clouds.** The architecture forces a choice:
|
||||
## 2) Build / Run / Lint / Test Commands
|
||||
|
||||
1. **Streaming API**: Edge sends video → Host computes everything (depth, tracking, detection)
|
||||
2. **Fusion API**: Edge computes everything → Sends only metadata (bodies/poses)
|
||||
### Python (py_workspace)
|
||||
|
||||
### Code Patterns
|
||||
Run from: `/workspaces/zed-playground/py_workspace`
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run python -V
|
||||
```
|
||||
|
||||
Run scripts:
|
||||
```bash
|
||||
uv run streaming_receiver.py --help
|
||||
uv run recording_multi.py
|
||||
uv run calibrate_extrinsics.py --help
|
||||
```
|
||||
|
||||
Type checking / lint-equivalent:
|
||||
```bash
|
||||
uv run basedpyright
|
||||
```
|
||||
|
||||
Tests:
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
Run a single test file:
|
||||
```bash
|
||||
uv run pytest tests/test_depth_refine.py
|
||||
```
|
||||
|
||||
Run a single test function:
|
||||
```bash
|
||||
uv run pytest tests/test_depth_refine.py::test_refine_extrinsics_with_depth_with_offset
|
||||
```
|
||||
|
||||
Run by keyword:
|
||||
```bash
|
||||
uv run pytest -k "depth and refine"
|
||||
```
|
||||
|
||||
Useful verbosity / fail-fast options:
|
||||
```bash
|
||||
uv run pytest -x -vv
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `pyproject.toml` sets `testpaths = ["tests"]`
|
||||
- `norecursedirs = ["loguru", "tmp", "libs"]`
|
||||
|
||||
### C++ sample project (body tracking)
|
||||
|
||||
Run from:
|
||||
`/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build`
|
||||
|
||||
#### Streaming Sender (Edge)
|
||||
```cpp
|
||||
sl::StreamingParameters stream_params;
|
||||
stream_params.codec = sl::STREAMING_CODEC::H265;
|
||||
stream_params.port = 30000;
|
||||
stream_params.bitrate = 12000;
|
||||
zed.enableStreaming(stream_params);
|
||||
```
|
||||
|
||||
#### Streaming Receiver (Host)
|
||||
```cpp
|
||||
sl::InitParameters init_params;
|
||||
init_params.input.setFromStream("192.168.1.100", 30000);
|
||||
zed.open(init_params);
|
||||
// Full ZED SDK available - depth, tracking, etc.
|
||||
```
|
||||
|
||||
#### Fusion Publisher (Edge or Host)
|
||||
```cpp
|
||||
sl::CommunicationParameters comm_params;
|
||||
comm_params.setForLocalNetwork(30000);
|
||||
// or comm_params.setForIntraProcess(); for same-machine
|
||||
zed.startPublishing(comm_params);
|
||||
```
|
||||
|
||||
#### Fusion Subscriber (Host)
|
||||
```cpp
|
||||
sl::Fusion fusion;
|
||||
fusion.init(init_params);
|
||||
sl::CameraIdentifier cam(serial_number);
|
||||
fusion.subscribe(cam, comm_params, pose);
|
||||
```
|
||||
|
||||
## Project: Multi-Camera Body Tracking
|
||||
|
||||
### Location
|
||||
`/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/`
|
||||
|
||||
### Architecture
|
||||
- **ClientPublisher**: Receives camera streams, runs body tracking, publishes to Fusion
|
||||
- **Fusion**: Subscribes to multiple ClientPublishers, fuses body data from all cameras
|
||||
- **GLViewer**: 3D visualization of fused bodies
|
||||
|
||||
### Camera Configuration (Hard-coded)
|
||||
From `inside_network.json`:
|
||||
|
||||
| Serial | IP | Streaming Port |
|
||||
|--------|-----|----------------|
|
||||
| 44289123 | 192.168.128.2 | 30000 |
|
||||
| 44435674 | 192.168.128.2 | 30002 |
|
||||
| 41831756 | 192.168.128.2 | 30004 |
|
||||
| 46195029 | 192.168.128.2 | 30006 |
|
||||
|
||||
### Data Flow
|
||||
```
|
||||
Edge Camera (enableStreaming) → Network Stream
|
||||
↓
|
||||
ClientPublisher (setFromStream) → Body Tracking (host)
|
||||
↓
|
||||
startPublishing() → Fusion (INTRA_PROCESS)
|
||||
↓
|
||||
Fused Bodies → GLViewer
|
||||
```
|
||||
|
||||
### Build
|
||||
```bash
|
||||
cd "/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build"
|
||||
cmake ..
|
||||
make -j4
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
./ZED_BodyFusion <config_file.json>
|
||||
```
|
||||
|
||||
## Related Samples
|
||||
---
|
||||
|
||||
### Camera Streaming Receiver
|
||||
`/workspaces/zed-playground/playground/camera streaming/receiver/cpp/`
|
||||
- Simple streaming receiver sample
|
||||
- Shows basic `setFromStream()` usage with OpenCV display
|
||||
## 3) Rules Files Scan (Cursor / Copilot)
|
||||
|
||||
## ZED SDK Headers
|
||||
Located at: `/usr/local/zed/include/sl/`
|
||||
- `Camera.hpp` - Main camera API
|
||||
- `Fusion.hpp` - Fusion module API
|
||||
- `CameraOne.hpp` - Single camera utilities
|
||||
As of latest scan:
|
||||
- No `.cursorrules` found
|
||||
- No `.cursor/rules/` found
|
||||
- No `.github/copilot-instructions.md` found
|
||||
|
||||
If these files are later added, treat them as higher-priority local policy and update this guide.
|
||||
|
||||
---
|
||||
|
||||
## 4) Code Style Conventions (Python)
|
||||
|
||||
### Imports
|
||||
- Prefer grouping: stdlib → third-party → local modules.
|
||||
- ZED imports use `import pyzed.sl as sl`.
|
||||
- In package modules (`aruco/*`), use relative imports (`from .pose_math import ...`).
|
||||
- In top-level scripts, absolute package imports are common (`from aruco... import ...`).
|
||||
|
||||
### Formatting & structure
|
||||
- 4-space indentation, PEP8-style layout.
|
||||
- Keep functions focused; isolate heavy logic into helper functions.
|
||||
- Favor explicit CLI options via `click.option` rather than positional ambiguity.
|
||||
|
||||
### Typing
|
||||
- Type hints are expected on public and most internal functions.
|
||||
- Existing code uses both `typing.Optional/List/Dict` and modern `|` syntax; stay consistent with surrounding file.
|
||||
- Use `jaxtyping` shape hints when already used in module (`TYPE_CHECKING` guard pattern).
|
||||
- Avoid `Any` unless unavoidable (OpenCV / pyzed boundaries).
|
||||
|
||||
### Naming
|
||||
- `snake_case`: functions, variables, modules.
|
||||
- `PascalCase`: classes.
|
||||
- `UPPER_SNAKE_CASE`: constants (e.g., dictionary maps).
|
||||
|
||||
### Docstrings
|
||||
- Include concise purpose + `Args` / `Returns` where helpful.
|
||||
- For matrix/array-heavy functions, document expected shape and units.
|
||||
|
||||
### Logging & user output
|
||||
- CLI-facing messaging: use `click.echo`.
|
||||
- Diagnostic/internal logs: use `loguru` (`logger.debug/info/warning`).
|
||||
- Keep debug noise behind `--debug` style flag where possible.
|
||||
|
||||
### Error handling
|
||||
- Raise specific exceptions (`ValueError`, `FileNotFoundError`, etc.) with actionable messages.
|
||||
- For CLI fatal paths, use `click.UsageError` / `SystemExit(1)` patterns found in project.
|
||||
- Validate early (shape/range/None checks) before expensive operations.
|
||||
|
||||
---
|
||||
|
||||
## 5) Testing Conventions
|
||||
|
||||
- Framework: `pytest`
|
||||
- Numeric assertions: prefer `numpy.testing.assert_allclose` where appropriate.
|
||||
- Exception checks: `pytest.raises(..., match=...)`.
|
||||
- Add tests under `py_workspace/tests/`.
|
||||
- If adding behavior in `aruco/*`, add or update corresponding tests (`test_depth_*`, `test_alignment`, etc.).
|
||||
|
||||
---
|
||||
|
||||
## 6) ZED-Specific Project Guidance
|
||||
|
||||
### Architecture reminder: Streaming vs Fusion
|
||||
- Streaming API: send compressed video, compute depth/tracking on host.
|
||||
- Fusion API: publish metadata (bodies/poses), lightweight host fusion.
|
||||
- There is no built-in “stream depth map” mode in the same way as metadata fusion.
|
||||
|
||||
### Depth units
|
||||
- Be explicit with coordinate units and keep units consistent end-to-end.
|
||||
- Marker geometry/parquet conventions in this repo are meter-based; do not mix mm/m silently.
|
||||
|
||||
### Threading
|
||||
- OpenCV GUI (`cv2.imshow`, `cv2.waitKey`) belongs on main thread.
|
||||
- Capture/grab work in worker threads with queue handoff.
|
||||
|
||||
### Network config
|
||||
- Use `zed_network_utils.py` and `zed_settings/inside_network.json` conventions.
|
||||
|
||||
---
|
||||
|
||||
## 7) Agent Execution Checklist
|
||||
|
||||
Before editing:
|
||||
1. Identify target workspace (`py_workspace` vs playground C++).
|
||||
2. Confirm commands from this file and nearby module docs.
|
||||
3. Search for existing tests covering the area.
|
||||
|
||||
After editing:
|
||||
1. Run focused test(s) first, then broader test run as needed.
|
||||
2. Run `uv run basedpyright` for type regressions.
|
||||
3. Keep diffs minimal and avoid unrelated file churn.
|
||||
|
||||
If uncertain:
|
||||
- Prefer small, verifiable changes.
|
||||
- Document assumptions in commit/PR notes.
|
||||
|
||||
Reference in New Issue
Block a user