# RapidPoseTriangulation

Fast triangulation of multiple persons from multiple camera views. \
A general overview can be found in the paper [RapidPoseTriangulation: Multi-view Multi-person Whole-body Human Pose Triangulation in a Millisecond](https://arxiv.org/pdf/2503.21692).

<div align="center">
    <img src="media/2d-k.jpg" alt="2D detections"" width="65%"/>
    <b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</b>
    <img src="media/3d-p.jpg" alt="3D detections" width="30%"/>
    <br>
    <br>
    <img src="media/2d-p.jpg" alt="3D to 2D projection" width="95%"/>
</div>

<br>

## Build

- Clone this project with submodules:

    ```bash
    git clone --recurse-submodules https://gitlab.com/Percipiote/RapidPoseTriangulation.git
    cd RapidPoseTriangulation/
    ```

- Enable GPU-access for docker building:

  - Install _nvidia_ container tools: [Link](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)

  - Run `sudo nano /etc/docker/daemon.json` and add:

    ```json
    {
      "runtimes": {
        "nvidia": {
          "args": [],
          "path": "nvidia-container-runtime"
        }
      },
      "default-runtime": "nvidia"
    }
    ```

  - Restart docker: `sudo systemctl restart docker`

- Build docker container:

  ```bash
  docker build --progress=plain -t rapidposetriangulation .
  ./run_container.sh
  ```

- Build triangulator:

  ```bash
  cd /RapidPoseTriangulation/swig/ && make all && cd ../tests/ && python3 test_interface.py && cd ..

  cd /RapidPoseTriangulation/scripts/ && \
  g++ -std=c++2a -fPIC -O3 -march=native -Wall -Werror -flto=auto \
    -I /RapidPoseTriangulation/rpt/ \
    -isystem /usr/include/opencv4/ \
    -isystem /onnxruntime/include/ \
    -isystem /onnxruntime/include/onnxruntime/core/session/ \
    -isystem /onnxruntime/include/onnxruntime/core/providers/tensorrt/ \
    -L /onnxruntime/build/Linux/Release/ \
    test_skelda_dataset.cpp \
    /RapidPoseTriangulation/rpt/*.cpp \
    -o test_skelda_dataset.bin \
    -Wl,--start-group \
      -lonnxruntime_providers_tensorrt \
      -lonnxruntime_providers_shared \
      -lonnxruntime_providers_cuda \
      -lonnxruntime \
    -Wl,--end-group \
    $(pkg-config --libs opencv4) \
    -Wl,-rpath,/onnxruntime/build/Linux/Release/ \
  && cd ..
  ```

- Download _ONNX_ models from [model registry](https://gitlab.com/Percipiote/RapidPoseTriangulation/-/ml/models) and save them to `mmdeploy/extras/exports/`.
  Upon the first usage, they will be converted to _TensorRT_ models, which will take a few minutes. \
  (Note that this conversion is not deterministic and will each time result in slightly different models and therefore also slightly different benchmark results.)

- Test with samples:

  ```bash
  python3 /RapidPoseTriangulation/scripts/test_triangulate.py
  ```

- Test with [skelda](https://gitlab.com/Percipiote/skelda/) dataset:

  ```bash
  export CUDA_VISIBLE_DEVICES=0
  python3 /RapidPoseTriangulation/scripts/test_skelda_dataset.py
  ```

<br>

## Extras

- Exporting tools for 2D models are at [mmdeploy](extras/mmdeploy/README.md) directory.

- For usage in combination with ROS2 see [ros](extras/ros/README.md) directory.

- Running on a Nvidia Jetson is also possible following [jetson](extras/jetson/README.md) directory.

<br>

## Citation

Please cite [RapidPoseTriangulation](https://arxiv.org/pdf/2503.21692) if you found it helpful for your research or business.

```bibtex
@article{
  rapidtriang,
  title={{RapidPoseTriangulation: Multi-view Multi-person Whole-body Human Pose Triangulation in a Millisecond}},
  author={Bermuth, Daniel and Poeppel, Alexander and Reif, Wolfgang},
  journal={arXiv preprint arXiv:2503.21692},
  year={2025}
}
```