crosstyan
435ea09451
BREAKING CHANGE: ClientPublisher::open() signature changed from open(sl::InputType input, Trigger* ref, int sdk_gpu_id) to open(const std::string& ip, int port, Trigger* ref, int sdk_gpu_id) The ClientPublisher now receives camera streams over network using setFromStream() instead of opening local USB cameras. This allows the host to receive video from edge devices running enableStreaming(). Changes: - Use init_parameters.input.setFromStream(ip, port) for stream input - Keep body tracking and Fusion publishing (INTRA_PROCESS) unchanged - Add getSerialNumber() getter for logging/debugging - Improve error messages with context (ip:port) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
139 lines
4.1 KiB
Markdown
139 lines
4.1 KiB
Markdown
# Agent Context & Reminders
|
|
|
|
## ZED SDK Architecture
|
|
|
|
### Streaming API vs Fusion API
|
|
|
|
The ZED SDK provides two distinct network APIs that are often confused:
|
|
|
|
| Feature | Streaming API | Fusion API |
|
|
|---------|---------------|------------|
|
|
| **Data Transmitted** | Compressed video (H264/H265) | Metadata only (bodies, objects, poses) |
|
|
| **Bandwidth** | 10-40 Mbps | <100 Kbps |
|
|
| **Edge Compute** | Video encoding only | Full depth NN + tracking + detection |
|
|
| **Host Compute** | Full depth + tracking + detection | Lightweight fusion only |
|
|
| **API Methods** | `enableStreaming()` / `setFromStream()` | `startPublishing()` / `subscribe()` |
|
|
|
|
### Key Insight
|
|
|
|
**There is NO built-in mode for streaming computed depth maps or point clouds.** The architecture forces a choice:
|
|
|
|
1. **Streaming API**: Edge sends video → Host computes everything (depth, tracking, detection)
|
|
2. **Fusion API**: Edge computes everything → Sends only metadata (bodies/poses)
|
|
|
|
### Code Patterns
|
|
|
|
#### Streaming Sender (Edge)
|
|
```cpp
|
|
sl::StreamingParameters stream_params;
|
|
stream_params.codec = sl::STREAMING_CODEC::H265;
|
|
stream_params.port = 30000;
|
|
stream_params.bitrate = 12000;
|
|
zed.enableStreaming(stream_params);
|
|
```
|
|
|
|
#### Streaming Receiver (Host)
|
|
```cpp
|
|
sl::InitParameters init_params;
|
|
init_params.input.setFromStream("192.168.1.100", 30000);
|
|
zed.open(init_params);
|
|
// Full ZED SDK available - depth, tracking, etc.
|
|
```
|
|
|
|
#### Fusion Publisher (Edge or Host)
|
|
```cpp
|
|
sl::CommunicationParameters comm_params;
|
|
comm_params.setForLocalNetwork(30000);
|
|
// or comm_params.setForIntraProcess(); for same-machine
|
|
zed.startPublishing(comm_params);
|
|
```
|
|
|
|
#### Fusion Subscriber (Host)
|
|
```cpp
|
|
sl::Fusion fusion;
|
|
fusion.init(init_params);
|
|
sl::CameraIdentifier cam(serial_number);
|
|
fusion.subscribe(cam, comm_params, pose);
|
|
```
|
|
|
|
## Project: Multi-Camera Body Tracking
|
|
|
|
### Location
|
|
`/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/`
|
|
|
|
### Architecture
|
|
- **ClientPublisher**: Receives camera streams, runs body tracking, publishes to Fusion
|
|
- **Fusion**: Subscribes to multiple ClientPublishers, fuses body data from all cameras
|
|
- **GLViewer**: 3D visualization of fused bodies
|
|
|
|
### Camera Configuration (Hard-coded)
|
|
From `inside_network.json`:
|
|
|
|
| Serial | IP | Streaming Port |
|
|
|--------|-----|----------------|
|
|
| 44289123 | 192.168.128.2 | 30000 |
|
|
| 44435674 | 192.168.128.2 | 30002 |
|
|
| 41831756 | 192.168.128.2 | 30004 |
|
|
| 46195029 | 192.168.128.2 | 30006 |
|
|
|
|
### Data Flow
|
|
```
|
|
Edge Camera (enableStreaming) → Network Stream
|
|
↓
|
|
ClientPublisher (setFromStream) → Body Tracking (host)
|
|
↓
|
|
startPublishing() → Fusion (INTRA_PROCESS)
|
|
↓
|
|
Fused Bodies → GLViewer
|
|
```
|
|
|
|
### Build
|
|
```bash
|
|
cd "/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build"
|
|
cmake ..
|
|
make -j4
|
|
```
|
|
|
|
### Run
|
|
```bash
|
|
./ZED_BodyFusion <config_file.json>
|
|
```
|
|
|
|
## Related Samples
|
|
|
|
### Camera Streaming Receiver
|
|
`/workspaces/zed-playground/playground/camera streaming/receiver/cpp/`
|
|
- Simple streaming receiver sample
|
|
- Shows basic `setFromStream()` usage with OpenCV display
|
|
|
|
## Important Paths
|
|
|
|
### ZED SDK Installation
|
|
- **Root**: `/usr/local/zed/`
|
|
- **Headers**: `/usr/local/zed/include/sl/`
|
|
- `Camera.hpp` - Main camera API
|
|
- `Fusion.hpp` - Fusion module API
|
|
- `CameraOne.hpp` - Single camera utilities
|
|
- **Libraries**: `/usr/local/zed/lib/`
|
|
- **CMake config**: `/usr/local/zed/zed-config.cmake`
|
|
- **Tools**: `/usr/local/zed/tools/`
|
|
- **Settings**: `/usr/local/zed/settings/`
|
|
|
|
### Project Structure
|
|
```
|
|
/workspaces/zed-playground/
|
|
├── playground/
|
|
│ ├── body tracking/multi-camera/cpp/ # This project
|
|
│ │ ├── src/ClientPublisher.cpp
|
|
│ │ ├── src/main.cpp
|
|
│ │ ├── include/ClientPublisher.hpp
|
|
│ │ └── build/
|
|
│ └── camera streaming/receiver/cpp/ # Streaming receiver sample
|
|
└── zed_settings/
|
|
└── inside_network.json # Camera network config
|
|
```
|
|
|
|
### Configuration Files
|
|
- **Camera network config**: `/workspaces/zed-playground/zed_settings/inside_network.json`
|
|
- **Fusion config** (for ZED360 tool): Generated JSON with camera poses and communication params
|