OpenGait/.sisyphus/plans/demo-visualizer.md

Work Plan: Real-time OpenCV Visualizer for Demo

Objective

Add a real-time OpenCV-based visualizer to the opengait/demo pipeline. It will display the live stream with bounding boxes, tracking IDs, FPS, label, and confidence, alongside a window for the segmentation result. The user can toggle between the raw high-res segmentation mask, the normalized 64x44 silhouette, or both side-by-side using keyboard controls.

Scope

• IN: Create visualizer.py, update pipeline.py to calculate rolling FPS and extract visualization payloads, update __main__.py with a --visualize CLI flag.
• OUT: Modifying underlying ScoNet, data loaders, or publishers.

Guardrails & Assumptions

• Graceful degradation: Handle frames where no person is detected (bbox/mask is None) without crashing.
• Cleanup: Ensure cv2.destroyAllWindows() is called when the pipeline stops or crashes.
• Non-blocking: cv2.waitKey(1) must be used to keep the pipeline moving in real-time.

Tasks

Task 1: Add CLI Flag in __main__.py

• Modify opengait/demo/__main__.py.
• Add --visualize boolean argument to the argparse configuration (action="store_true").
• Pass visualize=args.visualize to the ScoliosisPipeline constructor.

Task 2: Create visualizer.py

• Create opengait/demo/visualizer.py and implement class OpenCVVisualizer.
• Add an internal state variable self.mask_mode = 0 (0: Both, 1: Raw, 2: Normalized).
• Add method update(frame, bbox, track_id, mask_raw, silhouette, label, confidence, fps).
• Main Window: Draw bbox (if exists), put text for track_id, fps, label, and confidence. Use cv2.imshow('OpenGait Stream', frame).
• Segmented Window: Display mask_raw, silhouette (upscaled), or a side-by-side combination based on self.mask_mode. Use cv2.imshow('Segmented Result', segmented_display).
• Include key = cv2.waitKey(1).
  • If key == ord('m'), increment self.mask_mode (modulo 3) to toggle the mask view.
  • If key == ord('q'), return False to signal pipeline exit.
• Add a close() method calling cv2.destroyAllWindows().

Task 3: Update pipeline.py for Data Extraction & FPS

• Modify ScoliosisPipeline.__init__ to accept visualize: bool = False and conditionally instantiate self.visualizer = OpenCVVisualizer().
• Update FPS logic in run(): Calculate an EMA (Exponential Moving Average) FPS per frame for real-time overlay.
• Modify _select_silhouette to return (silhouette, mask_raw, bbox) so the raw mask and bbox are preserved for visualization.
• In process_frame, gather and return mask_raw, bbox, silhouette, track_id, label, confidence to the main loop (either in the result dict or as a separate return value).
• In the run() loop, after processing, extract these values and call self.visualizer.update(...). Break loop if it returns False.
• Add a try/finally block in run() to ensure self.visualizer.close() is called.

Final Verification Wave

• Run the demo with --visualize on a sample video.
• Verify both windows appear and update in real-time.
• Verify FPS, tracking ID, label, and confidence are rendered correctly on the main window.
• Verify pressing 'm' toggles the segmented window between Both, Raw, and Normalized views.
• Verify pressing 'q' closes the windows and exits the pipeline cleanly.