zed-playground/new_playground/depth-sensing/FIXES_SUMMARY.md

# ZED Point Cloud Rendering - Fix Summary

## Problem
The ZED depth sensing application was displaying RGB and depth images correctly via OpenCV, but the 3D point cloud viewer (GLViewer) showed nothing except the camera frustum wireframe. Point cloud data was valid (confirmed via PLY file export), but invisible in the OpenGL window.

## Root Causes

### 1. **Empty OpenGL VBO** (Critical Issue)
- **Problem**: The OpenGL Vertex Buffer Object (VBO) contained all zeros despite valid point cloud data from the ZED camera
- **Cause**: CUDA-OpenGL interop buffer was mapped once at initialization and never unmapped, preventing proper data transfer
- **Fix**: Changed `PointCloud::update()` to properly:
  - Map the CUDA-OpenGL resource
  - Copy data from ZED GPU buffer to mapped OpenGL buffer
  - **Synchronize the CUDA stream** (`cudaStreamSynchronize`) - critical for ensuring copy completes
  - Unmap the resource so OpenGL can access it
  
```cpp
// Before: Buffer stayed mapped, copy never completed properly
cudaGraphicsMapResources(1, &bufferCudaID_, 0);  // At init only
cudaMemcpyAsync(...);  // Copy happened but never synced

// After: Proper map → copy → sync → unmap cycle
cudaGraphicsMapResources(1, &bufferCudaID_, strm);
cudaMemcpyAsync(...);
cudaStreamSynchronize(strm);  // Wait for GPU copy to complete!
cudaGraphicsUnmapResources(1, &bufferCudaID_, strm);
```

### 2. **Coordinate System Mismatch** (Critical Issue)
- **Problem**: Point cloud data is in millimeters (-2500 to +2500 mm), but camera and projection matrix expected smaller values
- **Cause**: ZED SDK outputs coordinates in millimeters, typical OpenGL coordinates are -1.0 to 1.0 or meters
- **Fix**: Added scaling in the vertex shader to convert mm → meters (÷1000)

```glsl
// Before: Raw millimeter coordinates
gl_Position = u_mvpMatrix * vec4(in_VertexRGBA.xyz, 1);

// After: Convert to meters in shader
vec3 pos_meters = in_VertexRGBA.xyz * 0.001;  // mm to m
gl_Position = u_mvpMatrix * vec4(pos_meters, 1);
```

### 3. **Camera Positioning in Wrong Scale**
- **Problem**: Camera positioned at (0, 0, 2000) looking at (0, 0, -3500) - these are millimeters, but after shader scaling, points are in meters
- **Cause**: Camera transform wasn't updated to match the scaled point cloud
- **Fix**: Changed camera to use meters

```cpp
// Before: Millimeters (mismatched with scaled points)
camera_ = CameraGL(sl::Translation(0, 0, 2000), sl::Translation(0, 0, -3500));
setProjection(90, 90, 10.f, 50000.f);  // znear=10mm, zfar=50000mm

// After: Meters (matches shader-scaled points)
camera_ = CameraGL(sl::Translation(0, 0, 2), sl::Translation(0, 0, -3.5));
setProjection(90, 90, 0.01f, 50.f);  // znear=0.01m, zfar=50m
```

### 4. **Matrix Transpose Inconsistency** (Minor Issue)
- **Problem**: Frustum rendering used `GL_FALSE` with explicit `transpose()`, but point cloud used `GL_TRUE` without transpose
- **Fix**: Made both use `GL_FALSE` with explicit transpose for consistency

```cpp
// Point cloud now matches frustum:
glUniformMatrix4fv(shMVPMatrixLoc_, 1, GL_FALSE, sl::Transform::transpose(vp).m);
```

## Changes Made

### `src/GLViewer.cpp`

#### PointCloud::initialize()
- Changed CUDA buffer registration flag from `cudaGraphicsRegisterFlagsNone` to `cudaGraphicsRegisterFlagsWriteDiscard`
- Removed permanent buffer mapping at initialization
- Set `numBytes_` directly instead of mapping to get it

#### PointCloud::update()
- Implemented proper CUDA-OpenGL synchronization cycle:
  - Map resource for CUDA access
  - Copy data with `cudaMemcpyAsync`
  - **Added `cudaStreamSynchronize(strm)`** - ensures GPU copy completes before unmapping
  - Unmap resource for OpenGL rendering

#### PointCloud::draw()
- Simplified rendering (removed debug code)
- Kept depth test disabled for point cloud
- Set point size to 2.0 pixels

#### PointCloud Vertex Shader
- Added coordinate scaling: `vec3 pos_meters = in_VertexRGBA.xyz * 0.001;`
- This converts millimeter coordinates from ZED SDK to meters for OpenGL

#### GLViewer::init()
- Updated camera positioning from millimeters to meters:
  - Position: `(0, 0, 2)` meters (was 2000mm)
  - Look-at: `(0, 0, -3.5)` meters (was -3500mm)
- Updated projection clipping planes:
  - Near: `0.01m` (10mm, was 10.f which was ambiguous)
  - Far: `50m` (50,000mm, was 50000.f)

#### GLViewer::update()
- Updated mouse translation and zoom to use meters instead of millimeters
- Removed `* 1000` multipliers from camera movement

## Test Files Created

### `test_opengl.cpp`
- Minimal OpenGL test with triangles
- Used to verify basic OpenGL/X11 rendering worked
- Helped isolate that depth buffer was working

### `test_points.cpp`
- Minimal point cloud rendering test
- Confirmed GL_POINTS rendering worked in X11 forwarding
- Used simple coordinates (-0.8 to 0.8) that proved the transformation was the issue

### `CMakeLists.txt`
- Added test executables for isolated OpenGL testing

## Key Insights

1. **CUDA Stream Synchronization is Critical**: Without `cudaStreamSynchronize()`, the async GPU copy may not complete before OpenGL tries to render, resulting in empty/stale data

2. **Coordinate Scale Matters**: OpenGL projection matrices work best with reasonable coordinate ranges (meters, not millimeters). The 1000x difference was causing precision issues and made debugging harder

3. **X11 Forwarding Works**: Despite initial concerns about X11 forwarding limitations in dev containers, OpenGL 4.6 with NVIDIA hardware acceleration works fine - the issue was entirely in our code

4. **Data vs Rendering Separation**: The point cloud data was always valid (confirmed by PLY export and Open3D visualization). The problem was purely in the OpenGL rendering pipeline

5. **Debugging with VBO Readback**: Using `glGetBufferSubData()` to read back GPU buffer contents was crucial for discovering the VBO was empty

## Current Status

✅ **Working Features:**
- RGB camera view display (OpenCV window)
- Depth image display (OpenCV window)
- 3D point cloud visualization (OpenGL window)
- Real-time camera data streaming
- Debug mode with synthetic rainbow point cloud
- Mouse controls for camera movement (rotate, pan, zoom)
- PLY file export

✅ **Performance:**
- 30 FPS with 491,520 points (960x512 resolution)
- GPU-accelerated rendering via CUDA-OpenGL interop
- No visible lag or stuttering

## Usage

```bash
# Real camera data
./ZED_Depth_Sensing

# Synthetic test data (rainbow gradient)
./ZED_Depth_Sensing --debug
```

**Controls:**
- Left mouse: Rotate camera
- Right mouse: Pan camera
- Mouse wheel: Zoom in/out
- 's' key: Save current point cloud to .ply file
- ESC: Exit

## Files Modified

- `src/GLViewer.cpp` - Main rendering and CUDA-OpenGL interop fixes
- `src/main.cpp` - Already had RGB/depth capture and debug mode
- `CMakeLists.txt` - Added test executables
- `test_opengl.cpp` - Created for testing
- `test_points.cpp` - Created for testing

## Technical Details

**Hardware:** NVIDIA GeForce RTX 5070 Ti  
**OpenGL:** 4.6.0 NVIDIA 590.48.01  
**GLSL:** 4.60 NVIDIA  
**ZED SDK:** 4.x with NEURAL_PLUS depth mode  
**Resolution:** HD720 (960x512)  
**Environment:** Dev container with X11 forwarding