Files

crosstyan 435ea09451 refactor: change ClientPublisher to accept camera streams instead of local USB

BREAKING CHANGE: ClientPublisher::open() signature changed from
  open(sl::InputType input, Trigger* ref, int sdk_gpu_id)
to
  open(const std::string& ip, int port, Trigger* ref, int sdk_gpu_id)

The ClientPublisher now receives camera streams over network using
setFromStream() instead of opening local USB cameras. This allows the
host to receive video from edge devices running enableStreaming().

Changes:
- Use init_parameters.input.setFromStream(ip, port) for stream input
- Keep body tracking and Fusion publishing (INTRA_PROCESS) unchanged
- Add getSerialNumber() getter for logging/debugging
- Improve error messages with context (ip:port)

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

2026-02-04 03:25:21 +00:00

4.1 KiB

Raw Blame History

Agent Context & Reminders

ZED SDK Architecture

Streaming API vs Fusion API

The ZED SDK provides two distinct network APIs that are often confused:

Feature	Streaming API	Fusion API
Data Transmitted	Compressed video (H264/H265)	Metadata only (bodies, objects, poses)
Bandwidth	10-40 Mbps	<100 Kbps
Edge Compute	Video encoding only	Full depth NN + tracking + detection
Host Compute	Full depth + tracking + detection	Lightweight fusion only
API Methods	`enableStreaming()` / `setFromStream()`	`startPublishing()` / `subscribe()`

Key Insight

There is NO built-in mode for streaming computed depth maps or point clouds. The architecture forces a choice:

Streaming API: Edge sends video → Host computes everything (depth, tracking, detection)
Fusion API: Edge computes everything → Sends only metadata (bodies/poses)

Code Patterns

Streaming Sender (Edge)

sl::StreamingParameters stream_params;
stream_params.codec = sl::STREAMING_CODEC::H265;
stream_params.port = 30000;
stream_params.bitrate = 12000;
zed.enableStreaming(stream_params);

Streaming Receiver (Host)

sl::InitParameters init_params;
init_params.input.setFromStream("192.168.1.100", 30000);
zed.open(init_params);
// Full ZED SDK available - depth, tracking, etc.

Fusion Publisher (Edge or Host)

sl::CommunicationParameters comm_params;
comm_params.setForLocalNetwork(30000);
// or comm_params.setForIntraProcess(); for same-machine
zed.startPublishing(comm_params);

Fusion Subscriber (Host)

sl::Fusion fusion;
fusion.init(init_params);
sl::CameraIdentifier cam(serial_number);
fusion.subscribe(cam, comm_params, pose);

Project: Multi-Camera Body Tracking

Location

/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/

Architecture

ClientPublisher: Receives camera streams, runs body tracking, publishes to Fusion
Fusion: Subscribes to multiple ClientPublishers, fuses body data from all cameras
GLViewer: 3D visualization of fused bodies

Camera Configuration (Hard-coded)

From inside_network.json:

Serial	IP	Streaming Port
44289123	192.168.128.2	30000
44435674	192.168.128.2	30002
41831756	192.168.128.2	30004
46195029	192.168.128.2	30006

Data Flow

Edge Camera (enableStreaming) → Network Stream
    ↓
ClientPublisher (setFromStream) → Body Tracking (host)
    ↓
startPublishing() → Fusion (INTRA_PROCESS)
    ↓
Fused Bodies → GLViewer

Build

cd "/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build"
cmake ..
make -j4

Run

./ZED_BodyFusion <config_file.json>

Camera Streaming Receiver

/workspaces/zed-playground/playground/camera streaming/receiver/cpp/

Simple streaming receiver sample
Shows basic setFromStream() usage with OpenCV display

Important Paths

ZED SDK Installation

Root: /usr/local/zed/
Headers: /usr/local/zed/include/sl/
- Camera.hpp - Main camera API
- Fusion.hpp - Fusion module API
- CameraOne.hpp - Single camera utilities
Libraries: /usr/local/zed/lib/
CMake config: /usr/local/zed/zed-config.cmake
Tools: /usr/local/zed/tools/
Settings: /usr/local/zed/settings/

Project Structure

/workspaces/zed-playground/
├── playground/
│   ├── body tracking/multi-camera/cpp/    # This project
│   │   ├── src/ClientPublisher.cpp
│   │   ├── src/main.cpp
│   │   ├── include/ClientPublisher.hpp
│   │   └── build/
│   └── camera streaming/receiver/cpp/     # Streaming receiver sample
└── zed_settings/
    └── inside_network.json                # Camera network config

Configuration Files

Camera network config: /workspaces/zed-playground/zed_settings/inside_network.json
Fusion config (for ZED360 tool): Generated JSON with camera poses and communication params

4.1 KiB Raw Blame History