zed-playground/new_playground/depth-sensing/FIXES_SUMMARY.md

ZED Point Cloud Rendering - Fix Summary

Problem

The ZED depth sensing application was displaying RGB and depth images correctly via OpenCV, but the 3D point cloud viewer (GLViewer) showed nothing except the camera frustum wireframe. Point cloud data was valid (confirmed via PLY file export), but invisible in the OpenGL window.

Root Causes

1. Empty OpenGL VBO (Critical Issue)

• Problem: The OpenGL Vertex Buffer Object (VBO) contained all zeros despite valid point cloud data from the ZED camera
• Cause: CUDA-OpenGL interop buffer was mapped once at initialization and never unmapped, preventing proper data transfer
• Fix: Changed PointCloud::update() to properly:
  • Map the CUDA-OpenGL resource
  • Copy data from ZED GPU buffer to mapped OpenGL buffer
  • Synchronize the CUDA stream (cudaStreamSynchronize) - critical for ensuring copy completes
  • Unmap the resource so OpenGL can access it

// Before: Buffer stayed mapped, copy never completed properly
cudaGraphicsMapResources(1, &bufferCudaID_, 0);  // At init only
cudaMemcpyAsync(...);  // Copy happened but never synced

// After: Proper map → copy → sync → unmap cycle
cudaGraphicsMapResources(1, &bufferCudaID_, strm);
cudaMemcpyAsync(...);
cudaStreamSynchronize(strm);  // Wait for GPU copy to complete!
cudaGraphicsUnmapResources(1, &bufferCudaID_, strm);

2. Coordinate System Mismatch (Critical Issue)

• Problem: Point cloud data is in millimeters (-2500 to +2500 mm), but camera and projection matrix expected smaller values
• Cause: ZED SDK outputs coordinates in millimeters, typical OpenGL coordinates are -1.0 to 1.0 or meters
• Fix: Added scaling in the vertex shader to convert mm → meters (÷1000)

// Before: Raw millimeter coordinates
gl_Position = u_mvpMatrix * vec4(in_VertexRGBA.xyz, 1);

// After: Convert to meters in shader
vec3 pos_meters = in_VertexRGBA.xyz * 0.001;  // mm to m
gl_Position = u_mvpMatrix * vec4(pos_meters, 1);

3. Camera Positioning in Wrong Scale

• Problem: Camera positioned at (0, 0, 2000) looking at (0, 0, -3500) - these are millimeters, but after shader scaling, points are in meters
• Cause: Camera transform wasn't updated to match the scaled point cloud
• Fix: Changed camera to use meters

// Before: Millimeters (mismatched with scaled points)
camera_ = CameraGL(sl::Translation(0, 0, 2000), sl::Translation(0, 0, -3500));
setProjection(90, 90, 10.f, 50000.f);  // znear=10mm, zfar=50000mm

// After: Meters (matches shader-scaled points)
camera_ = CameraGL(sl::Translation(0, 0, 2), sl::Translation(0, 0, -3.5));
setProjection(90, 90, 0.01f, 50.f);  // znear=0.01m, zfar=50m

4. Matrix Transpose Inconsistency (Minor Issue)

• Problem: Frustum rendering used GL_FALSE with explicit transpose(), but point cloud used GL_TRUE without transpose
• Fix: Made both use GL_FALSE with explicit transpose for consistency

// Point cloud now matches frustum:
glUniformMatrix4fv(shMVPMatrixLoc_, 1, GL_FALSE, sl::Transform::transpose(vp).m);

Changes Made

src/GLViewer.cpp

PointCloud::initialize()

• Changed CUDA buffer registration flag from cudaGraphicsRegisterFlagsNone to cudaGraphicsRegisterFlagsWriteDiscard
• Removed permanent buffer mapping at initialization
• Set numBytes_ directly instead of mapping to get it

PointCloud::update()

• Implemented proper CUDA-OpenGL synchronization cycle:
  • Map resource for CUDA access
  • Copy data with cudaMemcpyAsync
  • Added cudaStreamSynchronize(strm) - ensures GPU copy completes before unmapping
  • Unmap resource for OpenGL rendering

PointCloud::draw()

• Simplified rendering (removed debug code)
• Kept depth test disabled for point cloud
• Set point size to 2.0 pixels

PointCloud Vertex Shader

• Added coordinate scaling: vec3 pos_meters = in_VertexRGBA.xyz * 0.001;
• This converts millimeter coordinates from ZED SDK to meters for OpenGL

GLViewer::init()

• Updated camera positioning from millimeters to meters:
  • Position: (0, 0, 2) meters (was 2000mm)
  • Look-at: (0, 0, -3.5) meters (was -3500mm)
• Updated projection clipping planes:
  • Near: 0.01m (10mm, was 10.f which was ambiguous)
  • Far: 50m (50,000mm, was 50000.f)

GLViewer::update()

• Updated mouse translation and zoom to use meters instead of millimeters
• Removed * 1000 multipliers from camera movement

Test Files Created

test_opengl.cpp

• Minimal OpenGL test with triangles
• Used to verify basic OpenGL/X11 rendering worked
• Helped isolate that depth buffer was working

test_points.cpp

• Minimal point cloud rendering test
• Confirmed GL_POINTS rendering worked in X11 forwarding
• Used simple coordinates (-0.8 to 0.8) that proved the transformation was the issue

CMakeLists.txt

• Added test executables for isolated OpenGL testing

Key Insights

1. CUDA Stream Synchronization is Critical: Without cudaStreamSynchronize(), the async GPU copy may not complete before OpenGL tries to render, resulting in empty/stale data

2. Coordinate Scale Matters: OpenGL projection matrices work best with reasonable coordinate ranges (meters, not millimeters). The 1000x difference was causing precision issues and made debugging harder

3. X11 Forwarding Works: Despite initial concerns about X11 forwarding limitations in dev containers, OpenGL 4.6 with NVIDIA hardware acceleration works fine - the issue was entirely in our code

4. Data vs Rendering Separation: The point cloud data was always valid (confirmed by PLY export and Open3D visualization). The problem was purely in the OpenGL rendering pipeline

5. Debugging with VBO Readback: Using glGetBufferSubData() to read back GPU buffer contents was crucial for discovering the VBO was empty

Current Status

✅ Working Features:

• RGB camera view display (OpenCV window)
• Depth image display (OpenCV window)
• 3D point cloud visualization (OpenGL window)
• Real-time camera data streaming
• Debug mode with synthetic rainbow point cloud
• Mouse controls for camera movement (rotate, pan, zoom)
• PLY file export

✅ Performance:

• 30 FPS with 491,520 points (960x512 resolution)
• GPU-accelerated rendering via CUDA-OpenGL interop
• No visible lag or stuttering

Usage

# Real camera data
./ZED_Depth_Sensing

# Synthetic test data (rainbow gradient)
./ZED_Depth_Sensing --debug

Controls:

• Left mouse: Rotate camera
• Right mouse: Pan camera
• Mouse wheel: Zoom in/out
• 's' key: Save current point cloud to .ply file
• ESC: Exit

Files Modified

• src/GLViewer.cpp - Main rendering and CUDA-OpenGL interop fixes
• src/main.cpp - Already had RGB/depth capture and debug mode
• CMakeLists.txt - Added test executables
• test_opengl.cpp - Created for testing
• test_points.cpp - Created for testing

Technical Details

Hardware: NVIDIA GeForce RTX 5070 Ti
OpenGL: 4.6.0 NVIDIA 590.48.01
GLSL: 4.60 NVIDIA
ZED SDK: 4.x with NEURAL_PLUS depth mode
Resolution: HD720 (960x512)
Environment: Dev container with X11 forwarding