Files

T

crosstyan 213adee887 feat: add streamer-owned recording control service

Introduce a dedicated streamer-side recording control plane instead of sharing the producer recorder API.

- register streamer-owned recorder endpoints as a NATS micro service
- add explicit MP4 and MCAP recorder control protobufs and subject helpers
- wire recorder lifecycle handling into the pipeline runtime
- add MP4 writer and depth-alignment support files used by the new recording flow

2026-04-12 20:22:28 +08:00

2.9 KiB

Raw Blame History

Depth Alignment

Exported ZED MCAP files can carry RGB video and depth at different raster sizes.

For the current kindergarten zed4 exports, the common pair is:

video: 1920x1200
depth: 960x512

That means RGB and depth do not share aspect ratio. The files stay alignable because the exporter writes two separate calibration topics:

/{label}/calibration for video
/{label}/depth_calibration for depth

See mcap_layout.md for the topic contract.

What The Mapping Means

The correct way to align depth onto RGB is to use the two calibration matrices, not to assume matching pixel grids.

For the same camera, with zero distortion and identity rectification, the mapping reduces to a 2D affine transform:

u_rgb = (fx_rgb / fx_depth) * u_depth + (cx_rgb - (fx_rgb / fx_depth) * cx_depth)
v_rgb = (fy_rgb / fy_depth) * v_depth + (cy_rgb - (fy_rgb / fy_depth) * cy_depth)

and the inverse:

u_depth = (fx_depth / fx_rgb) * u_rgb + (cx_depth - (fx_depth / fx_rgb) * cx_rgb)
v_depth = (fy_depth / fy_rgb) * v_rgb + (cy_depth - (fy_depth / fy_rgb) * cy_rgb)

For the sampled kindergarten zed4 files, those offsets are effectively zero, so the mapping becomes an anisotropic resize:

u_rgb ~= 2.0 * u_depth
v_rgb ~= 2.34375 * v_depth

This is why the practical overlay behavior is a stretch, not a crop.

It is still better to derive the mapping from the two calibration topics than to hardcode 2.0 and 2.34375, because the exact calibration can vary by camera and export settings.

Helper Script

Use the alignment helper to inspect the calibration pair and optionally export an example overlay:

uv run --extra viewer python scripts/mcap_depth_alignment.py \
  /workspaces/data/kindergarten/bar/2026-03-18T11-59-41/2026-03-18T11-59-41_zed4.mcap \
  --camera-label zed4

To export example images:

uv run --extra viewer python scripts/mcap_depth_alignment.py \
  /workspaces/data/kindergarten/bar/2026-03-18T11-59-41/2026-03-18T11-59-41_zed4.mcap \
  --camera-label zed4 \
  --frame-index 400 \
  --output-dir /tmp/zed4_alignment_demo

That command writes:

rgb_frame.png
depth_native_colorized.png
depth_aligned_to_rgb_colorized.png
depth_overlay_on_rgb.png
rgb_aligned_to_depth.png

What The Helper Actually Does

The script:

reads /{label}/calibration and /{label}/depth_calibration
computes the affine mapping implied by the two intrinsic matrices
decodes one RGB frame and one depth frame from the MCAP
warps depth into RGB space with cv2.warpAffine
optionally warps RGB into depth space with the inverse mapping

For the current exported ZED MCAP contract, that is the right simple alignment path.

If a future export starts carrying non-zero distortion or non-identity rectification, consumers should switch from this affine shortcut to a full camera-model reprojection path.

2.9 KiB Raw Blame History

Depth Alignment

What The Mapping Means

Helper Script

What The Helper Actually Does

2.9 KiB

Raw Blame History