cvmmap-streamer/docs/encoded_passthrough.md

Encoded Passthrough From cv-mmap ABI v2.1

Summary

cvmmap-streamer can now consume encoded access units directly from a cv-mmap shared-memory snapshot when the producer exposes the ABI v2.1 encoded plane.

This is intended for producers such as cv-mmap with video.backend = "udp_rtp":

• cv-mmap receives RTP H.265
• cv-mmap parses once and publishes raw BGR plus encoded AU in the same snapshot
• cvmmap-streamer can forward or mux the encoded AU directly

The video no longer needs a decode -> encode round trip inside the streamer.

Input selection

input.video_source controls how the pipeline uses the snapshot:

[input]
uri = "cvmmap://default"
nats_url = "nats://localhost:4222"
video_source = "auto"

Valid values:

• auto: prefer encoded passthrough when the encoded AU plane is present, otherwise use raw encoding
• raw: always use the raw left plane and the local encoder backend
• encoded: require the encoded AU plane; fail if it is missing

CLI equivalent:

--input-video-source auto|raw|encoded

Encoded passthrough behavior

When video_source resolves to encoded mode:

• no encoder backend is created
• the streamer reads codec, bitstream format, keyframe flag, frame rate, and stream PTS from the SHM metadata
• width and height still come from the raw left-plane frame metadata
• the encoded AU is forwarded as-is

Current assumptions:

• codec is H.265 or H.264
• bitstream format is Annex B
• one SHM encoded plane contains one AU
• time base is treated as 1/1e9

Outputs

RTP

RTP output accepts passthrough access units directly.

• in encoded mode, SDP signaling and RTP packetization follow the codec from SHM metadata
• config.encoder.codec is ignored for RTP while passthrough is active
• raw mode still uses the configured local encoder and its codec

RTMP

RTMP output accepts passthrough access units directly.

• If decoder config is present, it is used.
• If decoder config is absent, the output relies on in-band parameter sets from keyframes.

For cv-mmap udp_rtp, this works because the producer republishes keyframes with VPS/SPS/PPS in-band.

MCAP

MCAP recording writes the compressed video directly.

• no local encoder is used
• keyframes are written exactly from the encoded AU payload
• depth and body-tracking recording continue to use the same raw/depth/NATS paths as before

Fallback to raw mode

If the producer does not expose the encoded plane, auto mode falls back to the existing raw pipeline:

• read raw BGR from SHM
• push frames through the configured encoder backend
• publish or record the encoder output

This keeps existing producers compatible.