Files

T

crosstyan 51d03d4279 feat(encode): add Jetson Multimedia API encoder backend

Integrate a native Jetson Multimedia API encoder path and keep the existing encoded access-unit contract for RTMP, RTP, and recording consumers.

This adds conditional Jetson MMAPI detection in CMake, builds the required NVIDIA sample common classes into a dedicated support library, and compiles the new backend only when the platform dependencies are present.

The runtime selector now lets encoder.backend=auto prefer Jetson MM for NVIDIA hardware requests while keeping encoder.backend=ffmpeg as an explicit FFmpeg path. Device selection semantics are updated so nvidia requires the Jetson backend, auto can fall back to FFmpeg software, and software remains FFmpeg-only.

The Jetson backend converts supported raw inputs through swscale, feeds NvVideoEncoder in YUV420M, emits Annex-B access units, and harvests decoder configuration from a warmup keyframe so downstream packetizers keep their existing contract.

This also splits FFmpeg encoder option handling into a shared header, updates runtime config/help text and tester defaults, and refreshes compatibility/caveat documentation to reflect the new selection behavior.

Build-tree runtime RPATH handling is tightened so GCC 15 builds keep the matching libstdc++ visible locally. Verification covered GCC 15 builds, RTP H.264/H.265 tester runs, RTMP H.264 stub publish, local live SRS smoke with cvmmap://zed1, and remote execution on 192.168.2.155 using the deployed bundle lib directory for the GCC 15 runtime.

2026-04-15 18:37:44 +08:00

3.8 KiB

Raw Blame History

Operational Caveats

This document captures the current runtime constraints for cvmmap-streamer after the FFmpeg-only cleanup.

Input Model

cvmmap URI Only

The runtime ingests from a cvmmap URI, for example:

./build/cvmmap_streamer --run-mode pipeline --input-uri 'cvmmap://zed@/tmp/cvmmap'

Legacy flags such as --shm-name, --zmq-endpoint, --input-mode, and the dummy-input options are no longer supported.

Encoder Path

Jetson Hardware Encode Uses The Jetson Multimedia API Backend

The public backend surface is still:

--encoder-backend auto
--encoder-backend ffmpeg

--encoder-backend ffmpeg keeps the existing FFmpeg encoder path. --encoder-backend auto may now select the in-repo Jetson Multimedia API backend on Jetson builds when hardware encoding is requested.

The removed GStreamer backend is still unavailable.

NVIDIA Device Mode On Jetson No Longer Depends On FFmpeg Hardware Encoders

On Jetson builds with CVMMAP_STREAMER_ENABLE_JETSON_MM enabled, --encoder-device nvidia uses the Jetson Multimedia API encoder directly rather than desktop *_nvenc or Jetson FFmpeg *_v4l2m2m encoder exposure.

Selection behavior is now:

--encoder-device software -> FFmpeg software encode (libx264/libx265)
--encoder-device nvidia -> Jetson Multimedia API backend only; fail if unavailable
--encoder-device auto -> try Jetson Multimedia API first on Jetson builds, then fall back to FFmpeg software

Useful local checks on Jetson:

test -e /dev/v4l2-nvenc
gst-inspect-1.0 nvv4l2h264enc nvv4l2h265enc

FFmpeg encoder enumeration is no longer the authoritative Jetson hardware-encode check for this repo.

Low-Latency Defaults

The current low-latency defaults are:

gop=30
b_frames=0
encoder-family-specific low-latency options where FFmpeg exposes them

This favors immediacy over compression efficiency.

Transport Caveats

RTMP Is Enhanced RTMP Only

The repo now supports:

libavformat RTMP output
ffmpeg_process RTMP output

The removed custom RTMP packetizer and domestic mode are no longer available. RTMP is always Enhanced RTMP.

RTP SDP Is Optional

The RTP publisher can write an SDP sidecar when --rtp-sdp is set. This is only for generic receivers that need an out-of-band description. If the consumer is preconfigured, skip the SDP output.

No Direct RTSP/WebRTC Publishing

This repo does not contain a direct RTSP publisher or a direct WebRTC/WHEP publisher.

If you want browser playback or WHEP, publish RTMP to a media server such as SRS and let that server expose WHEP/WebRTC.

Media Scope

No Audio Support

This project is video-only. It publishes or records encoded video access units only.

If you need audio transport, muxing, or A/V sync, that is outside the current repo scope.

Recording Caveats

MCAP Stores Encoded Video Frames

MCAP recording stores one foxglove.CompressedVideo message per encoded access unit. Replay depends on encoded keyframes carrying decoder configuration; this is handled by the current writer on keyframes.

MCAP Body Recording Is Raw Packet Capture

When the producer exposes ZED body tracking, MCAP recording stores one raw cvmmap.body_tracking.v1 message per body PUB packet on /camera/body by default. These payload bytes are not normalized by cvmmap-streamer; downstream consumers should parse them with the cv-mmap body-tracking v1 contract.

External Server Caveats

Local SRS Defaults Can Hit `ulimit`

On this machine, stock SRS configs can fail at startup if max_connections exceeds the local ulimit -n. The checked-in smoke profile uses max_connections 64 to avoid that local failure mode.

Server Latency Still Matters

Low encoder latency does not guarantee low end-to-end latency. SRS, ZLMediaKit, HTTP-FLV players, and browser buffering can all add delay on top of the encoder path.

3.8 KiB Raw Blame History