114 lines
3.2 KiB
Markdown
114 lines
3.2 KiB
Markdown
# Agent Context & Reminders
|
|
|
|
## ZED SDK Architecture
|
|
|
|
### Streaming API vs Fusion API
|
|
|
|
The ZED SDK provides two distinct network APIs that are often confused:
|
|
|
|
| Feature | Streaming API | Fusion API |
|
|
|---------|---------------|------------|
|
|
| **Data Transmitted** | Compressed video (H264/H265) | Metadata only (bodies, objects, poses) |
|
|
| **Bandwidth** | 10-40 Mbps | <100 Kbps |
|
|
| **Edge Compute** | Video encoding only | Full depth NN + tracking + detection |
|
|
| **Host Compute** | Full depth + tracking + detection | Lightweight fusion only |
|
|
| **API Methods** | `enableStreaming()` / `setFromStream()` | `startPublishing()` / `subscribe()` |
|
|
|
|
### Key Insight
|
|
|
|
**There is NO built-in mode for streaming computed depth maps or point clouds.** The architecture forces a choice:
|
|
|
|
1. **Streaming API**: Edge sends video → Host computes everything (depth, tracking, detection)
|
|
2. **Fusion API**: Edge computes everything → Sends only metadata (bodies/poses)
|
|
|
|
### Code Patterns
|
|
|
|
#### Streaming Sender (Edge)
|
|
```cpp
|
|
sl::StreamingParameters stream_params;
|
|
stream_params.codec = sl::STREAMING_CODEC::H265;
|
|
stream_params.port = 30000;
|
|
stream_params.bitrate = 12000;
|
|
zed.enableStreaming(stream_params);
|
|
```
|
|
|
|
#### Streaming Receiver (Host)
|
|
```cpp
|
|
sl::InitParameters init_params;
|
|
init_params.input.setFromStream("192.168.1.100", 30000);
|
|
zed.open(init_params);
|
|
// Full ZED SDK available - depth, tracking, etc.
|
|
```
|
|
|
|
#### Fusion Publisher (Edge or Host)
|
|
```cpp
|
|
sl::CommunicationParameters comm_params;
|
|
comm_params.setForLocalNetwork(30000);
|
|
// or comm_params.setForIntraProcess(); for same-machine
|
|
zed.startPublishing(comm_params);
|
|
```
|
|
|
|
#### Fusion Subscriber (Host)
|
|
```cpp
|
|
sl::Fusion fusion;
|
|
fusion.init(init_params);
|
|
sl::CameraIdentifier cam(serial_number);
|
|
fusion.subscribe(cam, comm_params, pose);
|
|
```
|
|
|
|
## Project: Multi-Camera Body Tracking
|
|
|
|
### Location
|
|
`/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/`
|
|
|
|
### Architecture
|
|
- **ClientPublisher**: Receives camera streams, runs body tracking, publishes to Fusion
|
|
- **Fusion**: Subscribes to multiple ClientPublishers, fuses body data from all cameras
|
|
- **GLViewer**: 3D visualization of fused bodies
|
|
|
|
### Camera Configuration (Hard-coded)
|
|
From `inside_network.json`:
|
|
|
|
| Serial | IP | Streaming Port |
|
|
|--------|-----|----------------|
|
|
| 44289123 | 192.168.128.2 | 30000 |
|
|
| 44435674 | 192.168.128.2 | 30002 |
|
|
| 41831756 | 192.168.128.2 | 30004 |
|
|
| 46195029 | 192.168.128.2 | 30006 |
|
|
|
|
### Data Flow
|
|
```
|
|
Edge Camera (enableStreaming) → Network Stream
|
|
↓
|
|
ClientPublisher (setFromStream) → Body Tracking (host)
|
|
↓
|
|
startPublishing() → Fusion (INTRA_PROCESS)
|
|
↓
|
|
Fused Bodies → GLViewer
|
|
```
|
|
|
|
### Build
|
|
```bash
|
|
cd "/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build"
|
|
cmake ..
|
|
make -j4
|
|
```
|
|
|
|
### Run
|
|
```bash
|
|
./ZED_BodyFusion <config_file.json>
|
|
```
|
|
|
|
## Related Samples
|
|
|
|
### Camera Streaming Receiver
|
|
`/workspaces/zed-playground/playground/camera streaming/receiver/cpp/`
|
|
- Simple streaming receiver sample
|
|
- Shows basic `setFromStream()` usage with OpenCV display
|
|
|
|
## ZED SDK Headers
|
|
Located at: `/usr/local/zed/include/sl/`
|
|
- `Camera.hpp` - Main camera API
|
|
- `Fusion.hpp` - Fusion module API
|
|
- `CameraOne.hpp` - Single camera utilities
|