Background

This was initially a team project of the course '3D Vision' at ETHZ, proposed by AugmedIT. We were supervised by Jonas Hein (PhD student at ETHZ), Jene Meulstee (product developer of AugmedIT), and Tristan van Doormaal (one of founders of AugmedIT). After the course was finished, AugmedIT extends this project into an internship (ongoing at this moment).

Introduction

With augmented reality (AR), virtual 3D models can be superimposed into the real world of the user’s view. AugmedIT has created a workflow that can automatically transform MRI scans of patients with brain tumours [1], into 3D holograms for the HoloLens2. These holograms can be used by doctors to prepare or plan their surgical procedure. However, it remains challenging how these 3D holograms can be fused with the real patients. In the existing method, a pointer is used to indicate manually 5 points on both hologram and patient to register the hologram.

In this project, a semi-automatic process is proposed to register existing medical 3D holograms to patients. We first utilize the research mode of Hololens2 to obtain RGB-Depth data of a human head. The data is then transferred to a remote server via a TCP connection for processing. Next, we utilize a multiway-registration-based method to align the existing patient data, such as MRI data, with the processed point cloud. Visualizations and experiments demonstrate the success of our registration method. Our method outperforms the previous method by reducing the Fiducial Registration Error from 8.5 mm to 1.51 mm.

Overview of our registration pipeline.

Details

Unfortunately, we can not release our documentations and code. If you are interested, please refer to:

[1] T. Fick, J. van Doormaal, E. Hoving, L. Regli, T van Doormaal - Holographic patient tracking after bed movement for augmented reality neuronavigation using a head-mounted display. Acta Neurochir (Wien). 2021 Jan 29. doi: 10.1007/s00701-021-04707-4.

Acknowledgements

The website template was borrowed from Michaël Gharbi.