HoloLens 2 & Detecting Infrared markers for Navigation on OST AR headsets

by: Hisham Iqbal (hiqb217)

Overview

This is the C++ DLL used for the image-processing and tool-pose estimation used by the system reported in our publication: Semi-Automatic Infrared Calibration for Augmented Reality Systems in Surgery^[1] by Hisham Iqbal & Ferdinando Rodriguez y Baena.

This project is a Windows Runtime Component (C++/WinRT)^[2] DLL designed to enable the HoloLens 2 to detect the presence of tools equipped with IR-reflective markers. The resulting .dll and .winmd files can be copied and easily consumed on a C# Unity app (see HL2-DINO-Unity for example).

The classes in this project are designed to carry out the following tasks:

Grabbing sensor data from the HoloLens 2's AHAT depth sensor
Image processing to locate the presence of IR-reflective markers visible to the headset
Providing an interface for the C# Unity application to receive sensor images, tool pose data etc.

3rd Party Dependencies

Libraries/Headers	Note
`ResearchModeAPI.h`	From the HoloLens2ForCV project on GitHub
`OpenCV 4.5.3`	Compiled for compatibility with UWP applications^[3]. Static libraries included within this project
`Eigen`	Header-only matrix library with no extra compilation required
`Shiny-UWP`	Used for gathering profiler data^[4]. Static libraries included.

‍Note: Shiny-UWP is optional and can be removed altogether if you are not interested in this info.

Getting Started

git clone https://github.com/HL2-DINO/DINO-DLL.git
Open HL2DinoPlugin.sln, set project configs to Release and ARM64
[Optional] Make any changes to the source code as you wish
From Visual Studio menus, [Build] -> [Build Solution]
Copy generated .dll and .winmd from the ARM64/Release/HL2DinoPlugin/ folder into your Unity project inside the folder Assets/Plugins/WSA

To see how to set up a sample Unity project, check out the DINO-Unity repositories.

You can use this project as is to build a .dll and .winmd in your Unity project. If you wish to make alterations to the source code to insert your own algorithms, then make any necessary adjustments, and compile for Release and ARM64 architecture.

Two outputs of this process (HL2DinoPlugin.dll & HL2DinoPlugin.winmd) will be contained in ARM64/Release/HL2DinoPlugin. These need to be copied over to your Unity project structure inside the folder Assets/Plugins/WSA.

Acknowledgements

If this project is useful for your research or work, please considering citing the following publication:
@inproceedings{Iqbal2022,

author = {Hisham Iqbal and Ferdinando Rodriguez y Baena},

journal = {2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022)},

title = {Semi-Automatic Infrared Calibration for Augmented Reality Systems in Surgery},

year = {2022},

}
Thanks go out to:
- Prof Ferdinando Rodriguez y Baena - For project supervision, endless advice, and for guidance with the point-matching algorithms used in the project.
- Andreas Keller - For providing crucial help with proper compilation and linking of the OpenCV libraries used in this project. See Andreas's excellent HoloLens2-IRTracking project for an alternative setup which achieves similar goals.
- Arjun Muhunthan - For the many impromptu code-reviews I requested and for being a great source of C++ wisdom.
- Dr Stefano Galvan - For his patience and time to provide best practices for software and algorithmic design as well as valuable insights into open-sourcing software.
The design of the DLL in this project was based on some very useful open-source projects which are really worth exploring for your own research:
1. HoloLens2-ResearchMode-Unity originally made available by petergu684.
2. HoloLens2ForCV originally made available by Microsoft.

‍Note: Gear Icon in repository logo created by Freepik - Flaticon

Appendix

Additional Notes

There are some additional features which have not been implemented yet to expedite the process of open-sourcing.

TO-DO's on the DLL side:

🔲 Move away from detecting contours every single frame

In this version, blob/contour detection runs every frame, which is the most time consuming part of the image processing loop. By using tracking algorithms to lower the frequency of detection, we could potentially get some performance gain.

🔲 Integrate filtering into the processing of the noisy sensor data.

Noise in the sensor data (infrared response / computed depth) results in noisy estimates for tool pose. This could be tackled with simple low-pass filters added to each TrackedTool struct, or an implementation of Kalman filters incorporating data from other streams of the HoloLens (visual light environmental cameras, IMU data).

🔲 Adding support to process data in millimetres and metres.

The internal processing and exposed outputs of pose estimation is done entirely with metres. With some small scaffolding in the code, we can support doing this in millimetres. At present, you can pass in tool config data as millimetres, but it's converted into metres before it reaches the DLL.

License

This project is licensed under the BSD License.

Other Build Information

The project was originally built to target Windows 10 SDK, version 2004 (10.0.19041.0).
This project has been tested and built with VS 2019 & VS 2022, with Universal Windows Platform development tools installed.

References

[1]: Iqbal H., Rodriguez y Baena, F. (2022) Semi‑Automatic Calibration for Augmented Reality Systems in Surgery. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems https://dx.doi.org/10.1109/IROS47612.2022.9982215

[2]: The project was built using a Windows Runtime Component (C++/WinRT) project template on Visual Studio 2019. See this tutorial for a reference on how to set up your own project.

[3]: If you want to re-compile OpenCV with different flags or for a different major version, it is highly recommended you follow the instructions found in this project README written by Andreas Keller. Some more reading on the topic can be done here and here

[4]: A forked repository of the original Shiny Profiler.