Real-time High-Fidelity Augmented Reality in Laparoscopic Liver Resection

Liver disease and liver cancer together cause ~2.5% of deaths annually in England, with a projected rise of liver cancer by 6% from 2025 to 2040 in the UK.

Long-term damage to the liver (e.g. Cirrhosis and liver cancer) is treated by surgical removal of parts of the liver or transplant. Laparoscopic (keyhole) and robotic liver resections are pivotal in modern surgery, offering minimally invasive options for removing unhealthy liver parts while preserving function with faster healing.

Due to a lack of tactile feedback, liver opacity, and the limitation in the accessibility of interpretable pre-operative 3D MRI/CT scans during the surgery, surgeons need to memorise and interpret the complex liver structures and associated abnormalities throughout, making the surgical procedure longer and technically difficult.

I propose developing a real-time augmented reality (AR) system using an advanced data-driven method that will automatically overlap liver segments, functional structures, blood vessels and optimal liver margins from pre-operative scans onto the laparoscopic scene.

I will address current unmet clinical needs by enhancing the visualisation, enabling faster and optimal functional organ-preserving liver surgery, leading to improved patient safety and recovery. I will address current gaps in existing algorithms, such as the lack of real-time performance and algorithmic robustness. I have co-designed the project with surgeons, patients, and industry.

I will develop a novel framework comprising the following: 1) Extraction of 3D models with eight liver segments and other information such as vessel map and tumour. 2) Intraoperative ultrasound segmentation of major liver landmarks. 3) Mapping of the pre-operative anatomy and key landmarks overlaid onto the liver during surgery. 4) Real-time dynamic navigation using vision and sensor-based system design for improved robustness.