In the corridors of the Peter MacCallum Cancer Centre in Victoria, a world-first Australian/Israeli medical technology project is reshaping the way surgeons plan for and execute intricate surgeries.

Traditionally, surgeons rely on 2D scans to visualise the structure of a tumour and its relationship to surrounding organs and tissue. The challenge has always been mentally translating these 2D images into a 3D structure.

Now, with the power of augmented reality (AR), this gap is being bridged.

Surgeons can don a headset, see 3D models of tumours in AR, rotate them, zoom in, and virtually “walk around” them, providing a holistic understanding of their position, size, and relation to other critical structures.

The initiative is being led by Sheba Medical Centre Fellows Dr Eyal Mor, Dr Danielle Hadar and Dr Michal Eifer, in conjunction with surgeons at Peter Mac including Associate Professor David Gyorki.

At Peter Mac, surgeons have trialled the technology and applied it to historic scans of retroperitoneal sarcoma, a rare cancer that can develop at the back of the abdomen in the body’s connective tissue contacting and often involving a number of important structures and organs.

The technology enabled the surgeons to see, with a new level of detail, where these tumours were located and how they were entwined with healthy tissue or organs.

Mor said surgeons were quick to grasp its vast potential.

“We’ve had random surgeons just passing the office and asking what we were doing waving our hands in the air, and then they put the glasses on and knew it was amazing. Everyone was saying how amazing this technology is,” he said.

This pioneering initiative is part of the ongoing collaboration between Peter Mac and Sheba Medical Centre’s ARC Centre for Digital Innovation and its AR hub.

The technology was developed at the ARC Centre for Digital Innovation, based at Sheba, and a team of experts flew from Israel to Melbourne to help stage the trial.

Gyorki said, “You want to have a very clear idea exactly how the surgery is going to go, and this provides even more information and allows us to better prepare, … so there are as few surprises as possible or no surprises intraoperatively.”

Idan Goldberger, CEO of the Australian Friends of Sheba Medical Centre, said, “The collaboration between Sheba and Peter MacCallum is more than a partnership; it’s a harmonious fusion of minds and visions.

This technology is in its infancy and its use at Peter Mac, to date, has been limited to mock scenarios and seeking hands-on feedback from surgeons.

Gyorki emphasised the need for further refinement but believes the 3D AR technology can be used in real time surgical planning for patients within two years.

Nevertheless, the promise of AR in surgical oncology is evident.

As the technology evolves and integrates deeper into clinical practice, it could make operations more efficient, reducing post-operative complications, and potentially transforming patient outcomes.