zed-playground/AGENTS.md

Agent Context & Reminders

ZED SDK Architecture

Streaming API vs Fusion API

The ZED SDK provides two distinct network APIs that are often confused:

Feature	Streaming API	Fusion API
Data Transmitted	Compressed video (H264/H265)	Metadata only (bodies, objects, poses)
Bandwidth	10-40 Mbps	<100 Kbps
Edge Compute	Video encoding only	Full depth NN + tracking + detection
Host Compute	Full depth + tracking + detection	Lightweight fusion only
API Methods	enableStreaming() / setFromStream()	startPublishing() / subscribe()

Key Insight

There is NO built-in mode for streaming computed depth maps or point clouds. The architecture forces a choice:

1. Streaming API: Edge sends video → Host computes everything (depth, tracking, detection)
2. Fusion API: Edge computes everything → Sends only metadata (bodies/poses)

Code Patterns

Streaming Sender (Edge)

sl::StreamingParameters stream_params;
stream_params.codec = sl::STREAMING_CODEC::H265;
stream_params.port = 30000;
stream_params.bitrate = 12000;
zed.enableStreaming(stream_params);

Streaming Receiver (Host)

sl::InitParameters init_params;
init_params.input.setFromStream("192.168.1.100", 30000);
zed.open(init_params);
// Full ZED SDK available - depth, tracking, etc.

Fusion Publisher (Edge or Host)

sl::CommunicationParameters comm_params;
comm_params.setForLocalNetwork(30000);
// or comm_params.setForIntraProcess(); for same-machine
zed.startPublishing(comm_params);

Fusion Subscriber (Host)

sl::Fusion fusion;
fusion.init(init_params);
sl::CameraIdentifier cam(serial_number);
fusion.subscribe(cam, comm_params, pose);

Project: Multi-Camera Body Tracking

Location

/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/

Architecture

• ClientPublisher: Receives camera streams, runs body tracking, publishes to Fusion
• Fusion: Subscribes to multiple ClientPublishers, fuses body data from all cameras
• GLViewer: 3D visualization of fused bodies

Camera Configuration (Hard-coded)

From inside_network.json:

Serial	IP	Streaming Port
44289123	192.168.128.2	30000
44435674	192.168.128.2	30002
41831756	192.168.128.2	30004
46195029	192.168.128.2	30006

Data Flow

Edge Camera (enableStreaming) → Network Stream
    ↓
ClientPublisher (setFromStream) → Body Tracking (host)
    ↓
startPublishing() → Fusion (INTRA_PROCESS)
    ↓
Fused Bodies → GLViewer

Build

cd "/workspaces/zed-playground/playground/body tracking/multi-camera/cpp/build"
cmake ..
make -j4

Run

./ZED_BodyFusion <config_file.json>

Related Samples

Camera Streaming Receiver

/workspaces/zed-playground/playground/camera streaming/receiver/cpp/

• Simple streaming receiver sample
• Shows basic setFromStream() usage with OpenCV display

ZED SDK Headers

Located at: /usr/local/zed/include/sl/

• Camera.hpp - Main camera API
• Fusion.hpp - Fusion module API
• CameraOne.hpp - Single camera utilities