crosstyan
06a6cd1ccf
Wire cvmmap-client to the local development path and record ongoing orchestration artifacts for reproducible local workflow context.
47 lines
3.3 KiB
Markdown
47 lines
3.3 KiB
Markdown
# Work Plan: Real-time OpenCV Visualizer for Demo
|
|
|
|
## Objective
|
|
Add a real-time OpenCV-based visualizer to the `opengait/demo` pipeline. It will display the live stream with bounding boxes, tracking IDs, FPS, label, and confidence, alongside a window for the segmentation result. The user can toggle between the raw high-res segmentation mask, the normalized 64x44 silhouette, or both side-by-side using keyboard controls.
|
|
|
|
## Scope
|
|
- **IN**: Create `visualizer.py`, update `pipeline.py` to calculate rolling FPS and extract visualization payloads, update `__main__.py` with a `--visualize` CLI flag.
|
|
- **OUT**: Modifying underlying ScoNet, data loaders, or publishers.
|
|
|
|
## Guardrails & Assumptions
|
|
- **Graceful degradation**: Handle frames where no person is detected (bbox/mask is None) without crashing.
|
|
- **Cleanup**: Ensure `cv2.destroyAllWindows()` is called when the pipeline stops or crashes.
|
|
- **Non-blocking**: `cv2.waitKey(1)` must be used to keep the pipeline moving in real-time.
|
|
|
|
## Tasks
|
|
|
|
### Task 1: Add CLI Flag in `__main__.py`
|
|
- Modify `opengait/demo/__main__.py`.
|
|
- Add `--visualize` boolean argument to the `argparse` configuration (`action="store_true"`).
|
|
- Pass `visualize=args.visualize` to the `ScoliosisPipeline` constructor.
|
|
|
|
### Task 2: Create `visualizer.py`
|
|
- Create `opengait/demo/visualizer.py` and implement `class OpenCVVisualizer`.
|
|
- Add an internal state variable `self.mask_mode = 0` (0: Both, 1: Raw, 2: Normalized).
|
|
- Add method `update(frame, bbox, track_id, mask_raw, silhouette, label, confidence, fps)`.
|
|
- **Main Window**: Draw `bbox` (if exists), put text for `track_id`, `fps`, `label`, and `confidence`. Use `cv2.imshow('OpenGait Stream', frame)`.
|
|
- **Segmented Window**: Display `mask_raw`, `silhouette` (upscaled), or a side-by-side combination based on `self.mask_mode`. Use `cv2.imshow('Segmented Result', segmented_display)`.
|
|
- Include `key = cv2.waitKey(1)`.
|
|
- If `key == ord('m')`, increment `self.mask_mode` (modulo 3) to toggle the mask view.
|
|
- If `key == ord('q')`, return `False` to signal pipeline exit.
|
|
- Add a `close()` method calling `cv2.destroyAllWindows()`.
|
|
|
|
### Task 3: Update `pipeline.py` for Data Extraction & FPS
|
|
- Modify `ScoliosisPipeline.__init__` to accept `visualize: bool = False` and conditionally instantiate `self.visualizer = OpenCVVisualizer()`.
|
|
- Update FPS logic in `run()`: Calculate an EMA (Exponential Moving Average) FPS per frame for real-time overlay.
|
|
- Modify `_select_silhouette` to return `(silhouette, mask_raw, bbox)` so the raw mask and bbox are preserved for visualization.
|
|
- In `process_frame`, gather and return `mask_raw`, `bbox`, `silhouette`, `track_id`, `label`, `confidence` to the main loop (either in the result dict or as a separate return value).
|
|
- In the `run()` loop, after processing, extract these values and call `self.visualizer.update(...)`. Break loop if it returns `False`.
|
|
- Add a `try/finally` block in `run()` to ensure `self.visualizer.close()` is called.
|
|
|
|
## Final Verification Wave
|
|
- Run the demo with `--visualize` on a sample video.
|
|
- Verify both windows appear and update in real-time.
|
|
- Verify FPS, tracking ID, label, and confidence are rendered correctly on the main window.
|
|
- Verify pressing 'm' toggles the segmented window between Both, Raw, and Normalized views.
|
|
- Verify pressing 'q' closes the windows and exits the pipeline cleanly.
|