crosstyan/zed-playground
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d6c7829b1e8891c521c8595e695660e035757d73
zed-playground/py_workspace/docs/visualization-conventions.md
T

17 KiB
Raw Blame History

Visualization Conventions & Coordinate Frame Reference

Status: Canonical reference as of 2026-02-08. Applies to: visualize_extrinsics.py, calibrate_extrinsics.py, and inside_network.json.

Executive Summary

The visualize_extrinsics.py script went through multiple iterations of coordinate-frame switching (OpenCV ↔ OpenGL), Plotly camera/view hacks, and partial basis transforms that created compounding confusion about whether the visualization was correct. The root cause was conflating Plotly's scene camera settings with actual data-frame transforms: adjusting camera.up, autorange: "reversed", or eye position changes how you look at the data but does not change the coordinate frame the data lives in.

After several rounds of adding and removing --world-basis, --render-space, and --pose-convention flags, the visualizer was simplified to a single convention:

  • All data is in OpenCV convention (+X right, +Y down, +Z forward).
  • No basis switching. The --world-basis flag was removed.
  • Plotly's scene camera is configured with up = {x:0, y:-1, z:0} so that the OpenCV +Y-down axis renders as "down" on screen.

The confusion was never a bug in the calibration math — it was a visualization-layer problem caused by trying to make Plotly (which defaults to Y-up) display OpenCV data (which is Y-down) without a clear separation between "data frame" and "view frame."

Ground Truth Conventions

1. Calibration Output: world_from_cam

calibrate_extrinsics.py stores poses as T_world_from_cam (4×4 homogeneous):

T_world_from_cam = invert_transform(T_cam_from_world)
  • solvePnP returns T_cam_from_world (maps world points into camera frame).
  • The script inverts this before saving to JSON.
  • The translation column T[:3, 3] is the camera center in world coordinates.
  • The rotation columns T[:3, :3] are the camera's local axes expressed in world frame.

JSON format (16 floats, row-major 4×4):

{
  "44289123": {
    "pose": "0.878804 -0.039482 0.475548 -2.155006 0.070301 0.996409 ..."
  }
}

2. Camera-Local Axes (OpenCV)

Every camera's local frame follows the OpenCV pinhole convention:

Axis Direction Color in visualizer
+X Right Red
+Y Down Green
+Z Forward (into scene) Blue

The frustum is drawn along the camera's local +Z axis. The four corners of the frustum's far plane are at (±w, ±h, frustum_scale) in camera-local coordinates.

3. Plotly Scene/Camera Interpretation Pitfalls

Plotly's 3D scene has its own camera model that controls how you view the data:

Plotly setting What it does What it does NOT do
camera.up Sets which direction is "up" on screen Does not transform data coordinates
camera.eye Sets the viewpoint position Does not change axis orientation
yaxis.autorange = "reversed" Flips the Y axis tick direction Does not negate Y data values
aspectmode = "data" Preserves metric proportions Does not imply any convention

Critical insight: Changing camera.up from {y:1} to {y:-1} makes the plot look like Y-down is rendered correctly, but the underlying Plotly axis still runs bottom-to-top by default. This is purely a view transform — the data coordinates are unchanged.

Historical Confusion Timeline

This section documents the sequence of changes that led to confusion, for future reference. All commits are on visualize_extrinsics.py.

Phase 1: Initial Plotly Rewrite (7b9782a)

  • Rewrote the visualizer from matplotlib to Plotly with a --diagnose mode.
  • Used Plotly defaults (Y-up). OpenCV data (Y-down) appeared "upside down."
  • Frustums pointed in the correct direction in data space but looked inverted.

Phase 2: Y-Up Enforcement (a8d3751)

  • Attempted to fix by setting camera.up = {y:1} and using autorange: "reversed".
  • This made the view look correct for some angles but introduced axis-label confusion.
  • The Y axis ticks ran in the opposite direction from the data, misleading users.

Phase 3: Render-Space Option (ab88a24)

  • Added --render-space flag to switch between "cv" and "opengl" rendering.
  • The OpenGL path applied a basis-change matrix diag(1, -1, -1) to all data.
  • This actually transformed the data, not just the view — a correct approach but introduced a second code path that was hard to validate.

Phase 4: Ground Plane & Origin Triad (18e8142, 57f0dff)

  • Added ground plane overlay and world-origin axis triad.
  • These were drawn in the data frame, so they were correct in CV mode but appeared wrong in OpenGL mode (the basis transform was applied inconsistently to some elements but not others).

Phase 5: --world-basis with Global Transform (79f2ab0)

  • Renamed --render-space to --world-basis with cv and opengl options.
  • Introduced world_to_plot() as a central transform function.
  • In opengl mode: world_to_plot applied diag(1, -1, -1) to all points.
  • Problem: The Plotly camera.up and axis labels were not always updated consistently with the basis choice, leading to "it looks right from one angle but wrong from another" reports.

Phase 6: Restore After Removal (6330e0e)

  • --world-basis was briefly removed, then restored due to user request.
  • This back-and-forth left the README with stale documentation referencing both the old and new interfaces.

Phase 7: Final Cleanup — CV Only (d07c244)

  • Removed --world-basis entirely.
  • world_to_plot() became a no-op (identity function).
  • Plotly camera set to up = {x:0, y:-1, z:0} to render Y-down correctly.
  • Axis labels explicitly set to X (Right), Y (Down), Z (Forward).
  • Added --origin-axes-scale for independent control of the origin triad size.
  • Removed --diagnose, --pose-convention, and --render-space flags.

This is the current state.

Peculiar Behaviors Catalog

# Symptom Root Cause Fix / Explanation
1 Frustum appears to point in "-Z" direction Plotly default camera has Y-up; OpenCV frustum points +Z which looks "backward" when viewed from a Y-up perspective Set camera.up = {y:-1} (done in current code). The frustum is correct in data space.
2 Switching to --world-basis opengl makes some elements flip but not others The world_to_plot() transform was applied to camera traces but not consistently to ground plane or origin triad Removed --world-basis. Single convention eliminates partial-transform bugs.
3 yaxis.autorange = "reversed" makes ticks confusing Plotly reverses the tick labels but the data coordinates stay the same. Users see "0 at top, -2 at bottom" which contradicts Y-down intuition. Removed autorange: reversed. Use camera.up = {y:-1} instead, which rotates the view without mangling tick labels.
4 Camera positions don't match inside_network.json inside_network.json stores poses in the ZED Fusion coordinate frame (gravity-aligned, Y-up). calibrate_extrinsics.py stores poses in the ArUco marker object's frame (Y-down if the marker board is horizontal). These are different world frames. Not a bug. The two systems use different world origins and orientations. To compare, you must apply the alignment transform between the two frames. See FAQ below.
5 Origin triad too small or too large relative to cameras Origin triad defaulted to --scale (camera axis size), which is often much smaller than the camera spread Use --origin-axes-scale 0.6 (or similar) independently of --scale.
6 Bird-eye view shows unexpected orientation --birdseye uses orthographic projection looking down the Y axis. In CV convention, Y is "down" so this is looking from below the scene upward. Expected behavior. The bird-eye view shows the X-Z plane as seen from the -Y direction (below the cameras).

Canonical Rules Going Forward

  1. Single convention: All visualization data is in OpenCV frame. No basis switching.
  2. world_to_plot() is identity: It exists as a hook but performs no transform. If a future need arises for basis conversion, it should be the only place it happens.
  3. Plotly camera settings are view-only: Never use autorange: reversed or axis negation to simulate a coordinate change. Use camera.up and camera.eye only.
  4. Poses are world_from_cam: The 4×4 matrix maps camera-local points to world. Translation = camera position in world. Rotation columns = camera axes in world.
  5. Colors are RGB = XYZ: Red = X (right), Green = Y (down), Blue = Z (forward). This applies to both per-camera axis triads and the world-origin triad.
  6. Units are meters: Consistent with marker parquet geometry and calibration output.

Current CLI Behavior

Available Flags

visualize_extrinsics.py
  -i, --input TEXT          [required] Path to JSON extrinsics file
  -o, --output TEXT         Output path (.html or .png)
  --show                    Open interactive Plotly viewer
  --scale FLOAT             Camera axis length (default: 0.2)
  --frustum-scale FLOAT     Frustum depth (default: 0.5)
  --fov FLOAT               Horizontal FOV degrees (default: 60.0)
  --birdseye                Top-down orthographic view
  --show-ground/--no-show-ground    Ground plane toggle
  --ground-y FLOAT          Ground plane Y position (default: 0.0)
  --ground-size FLOAT       Ground plane side length (default: 8.0)
  --show-origin-axes/--no-show-origin-axes  Origin triad toggle (default: on)
  --origin-axes-scale FLOAT Origin triad size (defaults to --scale)
  --zed-configs TEXT         ZED calibration file(s) for accurate frustums
  --resolution [FHD1200|FHD|2K|HD|SVGA|VGA]
  --eye [left|right]

Removed Flags (Historical Only)

Flag Removed In Reason
--world-basis d07c244 Caused partial/inconsistent transforms. Single CV convention is simpler.
--pose-convention d07c244 Only world_from_cam is supported. No need for a flag.
--diagnose d07c244 Diagnostic checks moved out of the visualizer.
--render-space 79f2ab0 Renamed to --world-basis, then removed.

Note

: The README.md still contains stale references to --world-basis, --pose-convention, and --diagnose in the Troubleshooting section. These should be cleaned up to match the current CLI.

Verification Playbook

Quick Sanity Check

# Render with origin triad at 0.6m scale, save as PNG
uv run visualize_extrinsics.py \
  --input output/e2e_refine_depth_smoke_rerun.json \
  --output output/_final_opencv_origin_axes_scaled.png \
  --origin-axes-scale 0.6

Expected result:

  • Origin triad at (0,0,0) with Red→+X (right), Green→+Y (down), Blue→+Z (forward).
  • Camera frustums pointing along each camera's local +Z (blue axis).
  • Camera positions spread out in world space (not bunched at origin).
  • Y values for cameras should be negative (cameras are above the marker board, which is at Y≈0; "above" in CV convention means negative Y).

Interactive Validation

# Open interactive HTML for rotation/inspection
uv run visualize_extrinsics.py \
  --input output/e2e_refine_depth_smoke_rerun.json \
  --show \
  --origin-axes-scale 0.6

What to check:

  1. Rotate the view: The origin triad should remain consistent — Red/Green/Blue always point in the same data-space directions regardless of view angle.
  2. Hover over camera centers: Tooltip shows the camera serial number.
  3. Frustum orientation: Each frustum's open end faces away from the camera center along the camera's blue (Z) axis.

Bird-Eye Sanity Check

uv run visualize_extrinsics.py \
  --input output/e2e_refine_depth_smoke_rerun.json \
  --birdseye --show \
  --origin-axes-scale 0.6

Expected: Top-down view of the X-Z plane. Cameras should form a recognizable spatial layout matching the physical installation. The Red (X) axis points right, Blue (Z) axis points "up" on screen (forward in world).

FAQ

"Why does an OpenGL-like view look strange?"

Because the data is in OpenCV convention (Y-down, Z-forward) and Plotly defaults to Y-up. When you try to make Plotly act like an OpenGL viewer (Y-up, Z-backward), you need to either:

  1. Transform all data by applying diag(1, -1, -1) — correct but doubles the code paths and creates consistency risks.
  2. Adjust the Plotly camera — only changes the view, not the data. Axis labels and hover values still show CV coordinates.

We chose option (2) with camera.up = {y:-1}: minimal code, no data transformation, axis labels match the actual coordinate values. The trade-off is that the default Plotly orbit feels "inverted" compared to a Y-up 3D viewer. This is expected.

"Does flipping axes in the view equal changing the world frame?"

No. Plotly's camera.up, camera.eye, and autorange: reversed are purely view transforms. They change how the data is displayed but not what the coordinates mean. The data always lives in the frame it was computed in (OpenCV/ArUco world frame).

If you set camera.up = {y:1} (Plotly default), the plot will render Y-up on screen, but the data values are still Y-down. This creates a visual inversion that looks like "the cameras are upside down" — they're not; the view is just flipped.

"How do I compare with the C++ viewer and inside_network.json?"

The C++ ZED Fusion viewer and inside_network.json use a different world frame than calibrate_extrinsics.py:

Property calibrate_extrinsics.py ZED Fusion / inside_network.json
World origin ArUco marker object center Gravity-aligned, first camera or user-defined
Y direction Down (OpenCV) Up (gravity-aligned)
Pose meaning T_world_from_cam T_world_from_cam (same semantics, different world)
Units Meters Meters

To compare numerically:

  1. The relative poses between cameras should match (up to the alignment transform).
  2. The absolute positions will differ because the world origins are different.
  3. To convert: apply the alignment rotation that maps the ArUco world frame to the Fusion world frame. If --auto-align was used with a ground face, the ArUco frame is partially aligned (ground = XZ plane), but the origin and yaw may still differ.

Quick visual comparison: Look at the shape of the camera arrangement (distances and angles between cameras), not the absolute positions. If the shape matches, the calibration is consistent.

"Why are camera Y-positions negative?"

In OpenCV convention, +Y is down. Cameras mounted above the marker board (which defines Y≈0) have negative Y values. This is correct. A camera at Y = -1.3 is 1.3 meters above the board.

"What does inside_network.json camera 41831756's pose mean?"

Translation: [0.0, -1.175, 0.0]
Rotation: Identity

This camera is the reference frame origin (identity rotation) positioned 1.175m in the -Y direction. In the Fusion frame (Y-up), this means 1.175m below the world origin. In practice, this is the height offset of the camera relative to the Fusion coordinate system's origin.

Methodology: Comparing Different World Frames

Since inside_network.json (Fusion) and calibrate_extrinsics.py (ArUco) use different world origins, raw coordinate comparison is meaningless. We validated consistency using rigid SE(3) alignment:

  1. Match Serials: Identify cameras present in both JSON files.
  2. Extract Centers: Extract the translation column t from T_world_from_cam for each camera.
    • Crucial: Both systems use T_world_from_cam. It is not cam_from_world.
  3. Compute Alignment: Solve for the rigid transform (R_align, t_align) that minimizes the distance between the two point sets (Kabsch algorithm).
    • Scale is fixed at 1.0 (both systems use meters).
  4. Apply & Compare:
    • Transform Fusion points: P_aligned = R_align * P_fusion + t_align.
    • Position Residual: || P_aruco - P_aligned ||.
    • Orientation Check: Apply R_align to Fusion rotation matrices and compare column vectors (Right/Down/Forward) with ArUco rotations.
  5. Up-Vector Verification:
    • Fusion uses Y-Up (gravity). ArUco uses Y-Down (image).
    • After alignment, the transformed Fusion Y-axis should be approximately parallel to the ArUco -Y axis (or +Y depending on the specific alignment solution found, but they must be collinear with gravity).

Result: The overlay images in output/ were generated using this aligned frame. The low residuals (<2cm) confirm that the internal calibration is consistent, even though the absolute world coordinates differ.

Appendix: Stale README References

The following lines in py_workspace/README.md reference removed flags and should be updated:

  • Line ~104: References --pose-convention (removed).
  • Line ~105: References --world-basis opengl (removed).
  • Line ~116: References --diagnose (removed).

These were left from earlier iterations and do not reflect the current CLI.

Reference in New Issue View Git Blame Copy Permalink