Augmented Reality Systems for Surgical Procedures and for Tele-collaboration:  Experiences and Future Prospects

Henry Fuchs
Department of Computer Science
Sitterson Hall
University of North Carolina at Chapel Hill, USA.

Augmented reality systems (systems which augment the user's "real world" with synthetic virtual objects) may soon become useful for a variety of applications. Two projects, with different goals and application characteristics, demonstrate a contrast of technological possibilities, problems, and solutions.  In the first project, we seek to improve the physician's constrained views of the surgical site imposed by minimally invasive procedures such as laparoscopy or ultrasound-guided needle biopsy. In both these surgical situations we seek to deliver views similar to those afforded by the more-invasive open procedure.  We present these views to the physician with a video see-through head-mounted display. The synthetic imagery may come from whatever auxiliary inputs are available: the real-time ultrasound images, laparoscope images (a miniature camera inserted into the patient), preoperative CT scans, etc.  We expect that the more natural hand-eye coordination enabled by such a system will ease the difficulty of performing the delicate manipulation tasks in these surgical procedures.  Major research challenges remain:1) in real-time depth determination of the "real scene" to allow correct visibility compositing among the real and the synthetic parts of the scene, 2) in real-time range determination of laparoscopic images, and 3) determination of useful visual presentations of these scenes for which an overload of useful visual
information is often available.

The second project, on tele-collaboration, involves research groups within the US National Tele-immersion Initiative and multiple sites within the US NSF Science and Technology Center in Computer Graphics and Scientific Visualization.  For this project, we are attempting to augment a user's local work environment with realistic visual presentation of one or more distant collaborators.  We hope to provide a collaboration experience closer to that of talking around one's desk with someone than of a conventional tele-video conference that provides a "talking head" image of a distant collaborator on a small window on one's computer screen.  The images of the distant collaborators appear life-sized and projected in the appropriate locations, providing a surround experience similar to that of sitting at a conference table with participants sitting at one's left, right and across the table.  Remaining challenges are to make such experiences 3D rather than just 360-degree 2D surround.  We are encouraged
with recent results from our collaborators at the University of Pennsylvania and others with their rapid depth extraction from multiple cameras. Display of such scenes from a distant site can be achieved in a variety of ways, currently most reasonable to us being polarized-image stereo projection combined with head tracking.  Although similar systems have been developed in the past, new capabilities (e.g., 3D scene acquisition, higher resolution through multiple displays, larger area displays, setup in standard office environment) should make the new systems both more visually compelling and easy to use.

An ever-increasing variety of acquisition and display methods are potentially available for implementing augmented reality.  With increasing power of affordable components, we may see increases in the number of users comparable to the increase in "desktop" user environments within the past twenty years -- from the domain of a few specialists to nearly universal adoption.