JP7367041B2 - UI for head-mounted display systems - Google Patents

Description

TECHNICAL FIELD The disclosed technology relates generally to head-mounted displays and, more particularly, to methods and systems for controlling multiple system modes for head-mounted displays.

A head-mounted display is a wearable electronic display that projects an image into the wearer's field of vision. In virtual reality applications, the projected image typically blocks the wearer's field of view. In augmented reality applications, the projected image is typically superimposed on a portion of the wearer's field of view, allowing the wearer to see real and virtual features simultaneously.

U.S. Pat. No. 5,781,165 to Tabata, entitled "Image Display Apparatus of Head Mounted Type," discloses a user-controlled head mounted display. While wearing the head-mounted display, a virtual image is projected on the LCD screen in stereo, allowing the wearer to view a three-dimensional virtual image. The wearer controls the display of the virtual image by pressing buttons on the electronic controller. When the head-mounted display detects rotational and positional movement, for example when the wearer moves his or her head, the display of the virtual image is adjusted to appear stationary on the virtual image plane.

Yasukawa's U.S. Pat. No. 5,977,935, entitled "Head-Mounted Image Display Device and Data Processing Apparatus Including the same," discloses a head-mounted device for displaying augmented reality. I will. The wearer's field of vision is divided into sections. The wearer sees the real world in one section of his or her field of vision. One or more virtual screens are projected in other sections of the wearer's field of vision. By moving their gaze by moving their head, the wearer controls which virtual screen is displayed, similar to moving a mouse to control a display. A foot pedal acts as an input device, similar to a mouse click.

Yasukawa's U.S. Pat. No. 6,320,559, entitled "Head-Mounted Image Display Device and Data Processing Apparatus Including the same," discloses a head-mounted device for displaying virtual content. I will. The wearer switches the content on the display by moving his/her line of sight. The wearer also controls display attributes through voice input. The wearer may scroll through the displayed images by moving his/her line of sight.

US Pat. No. 6,396,497 to Reicheln, entitled "Computer User Interface with Head Motion Input," discloses a wearable display system with a 360 degree viewing space. Each point in view space is identified by a particular yaw and pitch position. By moving their head, the wearer changes the display in the viewing window, which occupies only a (25° x 20°) wedge in the 360° viewing space. A scroll bar displayed within the view window helps the wearer keep track of their current position within the view space.

Lemelson's U.S. Pat. No. 6,847,336, entitled "Selectively Controllable Heads-Up Display System," describes a surgeon's wear that has multiple user interfaces that allow the wearer to control the display. Disclose type display. This system incorporates eye tracking to move a cursor on the display screen, and either a foot pedal or voice command to perform "mouse click" selections.

Miller's U.S. Pat. No. 7,127,401, entitled "Remote Control of a Medical Device Using Speech Recognition and Foot Controls," provides a voice-controlled medical imaging device for use by surgeons. Disclose. Controlling medical imaging devices by voice frees up the surgeon's hands to perform medical procedures.

US Pat. No. 8,941,559 to Bar-Zeev, entitled "Opacity Filter for Display Device," discloses a wearable augmented reality display. Each pixel of the display is individually adjustable in level from maximum opacity to transparency, allowing light from the real-world scene to be blocked or transmitted accordingly. The controller controls the opacity of designated pixels for displaying the virtual image superimposed on the real world scene.

Raffle's U.S. Pat. No. 9,285,872, entitled “Using Head Gesture and Eye Position to Wake a Head Mounted Device,” uses gesture recognition and eye tracking to switch modes of operation for a wearable display unit. Disclose what you will incorporate.

US Pat. No. 9,286,730 to Bar-Zeev, entitled "Opacity Filter for Display Device," discloses a wearable augmented reality display. Each pixel of the display is individually adjustable in level from maximum opacity to transparency, allowing light from the real-world scene to be blocked or transmitted accordingly. The position of the virtual image superimposed on the real-world scene is controlled by eye tracking.

Abdullahi's U.S. Pat. No. 9,292,084, entitled "Control Systems and Methods for Head-Mounted Information System," uses gesture control and preconfigured gesture profiles to control the display of a wearable device. Disclose what you will do. Gestures may be hand or head gestures. Buttons are provided to turn gesture control features on and off.

US Pat. No. 9,383,816 to Hennelly, entitled "Text Selection Using HMD Head-Tracker and Voice-Command," discloses hands-free text selection for head-mounted displays. A wearer of a head-mounted display selects displayed text by a combination of head and hand gestures.

The project titled “Head-Mounted Display, Program for Controlling Head-Mounted Display, and Method of Controlling Head-Mounted Display” Liya's U.S. Patent No. 9,523,854 is a head-mounted display (HMD). Discloses a technology that improves the usefulness of. An image to be displayed on the HMD is selected based on detection that the direction of the HMD is outside the wearer's line of sight.

Moravetz's U.S. Pat. No. 9,588,343, entitled “Menu Navigation in a Head-Mounted Display,” navigates virtual menus by tracking the position and orientation (P&O) of an HMD relative to its focal point. Disclose how to navigate.

US Pat. No. 9,696,797 to Abdullahi, entitled "Control Systems and Methods for Head-Mounted Information System," discloses a method for navigating virtual menus by gesture control.

Fateh's U.S. Patent No. 9,804,669, entitled "High Resolution Perception of Content in a Wide Field of View of a Head-Mounted Display," Virtual image resolution based on Discloses a method for controlling. The resolution of the display area that is currently within the user's focal point is then increased, along with the display area that is predicted to be within the wearer's focal point. The resolution of the display area between these two focus areas is reduced.

Stafford's U.S. Patent No. 9,897,805, entitled "Image Rendering Responsive to User Actions in Head Mounted Display," uses eye tracking to predict upcoming movements of the wearer's gaze. A unit-mounted display is disclosed. The quality of the rendered image is adjusted based on the predicted motion.

Chuang's US Patent Publication No. US20120027373 A1, entitled “Head-Mounted Display Device Having Interactive Function and Method Thereof,” proposes a method based on detected movement of a display device. Head-mounted display that adjusts the displayed image Disclose.

Parkinson's United States Patent Publication No. US2015022 entitled “Head-Tracking Based Technique for Moving On-Screen Objects on Head Mounted Display (HMD)” 0142 A1 uses a head tracking controller to move a cursor on the display of an HMD device. Disclose a method for controlling.

US Patent Publication No. US20150346813 A1 to Vargas, entitled “Hands Free Image Viewing on Head Mounted Display,” discloses a head gesture-based user interface for controlling the display of an HMD device. Display features controllable via head gestures include zooming in and out of the displayed image, panning the displayed image, scrolling through a series of displayed images, and switching between various modes of operation.

Andersso entitled “Optical Head Mounted Display, Television Portal Module and Methods for Controlling Graphical User Interface” US Patent Publication No. US20170031538 A1 of Mr. Discloses entering the mode for controlling.

Zhang's US Patent Publication No. US20170242495 A1, entitled "Method and Device of Controlling Virtual Mouse and Head-Mounted Displaying Device," Head movement to control virtual mouse for type display Disclose incorporating detection.

Nazareth's U.S. Patent Publication No. US20170273549 A1, entitled “Portable Surgical Methods, System, and Apparatus,” describes a camera for acquiring live images of a surgical procedure, and a camera for displaying a live feed of the acquired images. A kit is disclosed for housing surgical equipment including a wireless headset, voice-activated controls for the system, and a computer for storing data related to the surgical procedure.

United States Patent Publication No. US20180113507 A1 to Abdullahi, entitled “Control Systems and Methods for Head-Mounted Information System,” discloses gesture control modes for head-mounted display (HMD) devices. I will. Gesture control mode is enabled upon detection of a gesture control enable signal and allows the user to navigate menus via gestures.

An objective of the disclosed technology is to provide new methods and systems for user interfaces for head-mounted display systems.

According to the disclosed technology, a head-mounted display configured to be worn by a surgeon, a tracker configured to track head gesture input by the surgeon, and a tracker configured to detect foot motion input by the surgeon. a computer coupled to a head-mounted display, a tracker, and a footswitch; a user interface configured to provide head gesture input received from the tracker in conjunction with foot motion input received from the switch and display images related to the surgical procedure on a head mounted display; , when the head-mounted display system is in the first system mode, perform the first action in the head-mounted display system, and when the head-mounted display system is in the second system mode, perform the first action on the head-mounted display system; The head gesture input is configured to apply the received head gesture input in conjunction with the foot motion input to perform a second action at the type display system.

In some embodiments, the tracker is at least partially integrated with and configured to track the head mounted display.

In some embodiments, the tracker is a camera external to the head mounted display.

In some embodiments, the system further comprises a shutter coupled to the head mounted display, wherein when the shutter is open the head mounted display is at least partially transparent and when the shutter is closed. In this case, the head-mounted display is substantially opaque.

In some embodiments, one of the first action and the second action controls a shutter.

In some embodiments, the user interface further comprises a microphone configured to detect voice commands, and the input further comprises voice commands.

In some embodiments, the user interface further comprises an eye tracker configured to detect eye movements by the surgeon, and the input further comprises eye movements.

In some embodiments, the system is selected from the group consisting of a camera system configured to capture images, an illumination system configured to operate with the camera system, a camera head positioner, and a robotic arm. and a positioning mechanism.

In some embodiments, the first action and the second action include controlling characteristics of an image displayed via the HMD, controlling a camera system, controlling a lighting system, and controlling a positioning mechanism. selected from the group consisting of:

In some embodiments, controlling the characteristics of the image displayed via the HMD includes selecting the content of the image, zooming in and out, scrolling between at least two virtual screens, From displaying pictures in picture (PIP), displaying overlays on top of raw images, centering images, displaying menus, navigating menus, and controlling regions of interest in images. Actions selected from a group consisting of:

In some embodiments, controlling the camera system comprises performing an action selected from the group consisting of controlling optical and electrical camera characteristics, and controlling position and orientation of the camera system.

In some embodiments, controlling the lighting system includes selecting at least one of a plurality of illuminators, selecting an intensity setting, selecting a filter, and turning the lighting on or off. comprising performing an action selected from a group consisting of:

In some embodiments, an image of the surgical field is displayed in a display-fixed state via the head-mounted display when the head-mounted display system is in the first system mode, and a video of the surgical field is When the head-mounted display system is in the second system mode, a world-fixed state is displayed on one of the plurality of virtual screens via the head-mounted display.

In some embodiments, the computer is further configured to apply a head gesture input to switch from the first system mode to the second system mode.

In some embodiments, the first region of the range of head positions and orientations of the wearer of the head-mounted display corresponds to a first system mode, and the first region of the range of head positions and orientations of the wearer of the head-mounted display The second region of the position and orientation range corresponds to a second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is a head-mounted display head-mounted display. The purpose is to change the position and orientation of the wearer's head from the first area to the second area.

In some embodiments, the computer is further configured to display a menu overlaid on the image on the head-mounted display, the menu displaying a first menu item for a first system mode; A second menu item is displayed for a second system mode.

According to other aspects of the disclosed technology, a method for interacting with a head-mounted display system is provided, the method comprising: displaying to a surgeon images related to a surgical procedure on the head-mounted display; receiving head gesture input by the surgeon from the head tracker in conjunction with foot motion input by the surgeon from the footswitch; and when the head mounted display system is in a first system mode, the head mounted display; received in conjunction with a foot motion input to perform a first action at the system and perform a second action at the head mounted display system when the head mounted display system is in a second system mode; applying the head gesture input.

In some embodiments, one of the first action and the second action comprises controlling a shutter coupled to the head-mounted display, and when the shutter is open, the head-mounted display is transparent, and when the shutter is closed, the head-mounted display is opaque.

In some embodiments, the method further comprises detecting voice commands by the surgeon and applying the voice commands to control the head mounted display system.

In some embodiments, the method further comprises detecting eye movements by the surgeon and applying the eye movements to control the head mounted display system.

In some embodiments, the method includes: illuminating a surgical field with an illumination system configured with a head-mounted display system; and using a camera system configured with the head-mounted display system. The method further comprises acquiring an image of the surgical field.

In some embodiments, the method further comprises controlling the position of the camera system to acquire images via a positioning mechanism selected from the group consisting of a camera head positioner and a robotic arm.

In some embodiments, the first action and the second action are controlling characteristics of an image displayed via a head mounted display, controlling a camera system, controlling a lighting system. , and controlling the positioning mechanism.

In some embodiments, controlling the characteristics of the image displayed via the head-mounted display includes selecting the content of the image, zooming in and out, scrolling between at least two virtual screens. and displaying an overlay over the raw image.

In some embodiments, controlling the camera system includes performing an action selected from the group consisting of selecting a camera to acquire an image, and controlling the position and orientation of the camera system. Be prepared.

In some embodiments, controlling the lighting system includes selecting at least one of a plurality of illuminators, selecting an intensity setting, selecting a filter, and turning the lighting on or off. comprising performing an action selected from a group consisting of:

In some embodiments, the image is displayed in a display-fixed state via the head-mounted display when the head-mounted display system is in the first system mode, and the image is displayed in a display-fixed state via the head-mounted display system. When in the second system mode, it is displayed in a world-locked state on one of a plurality of virtual screens via a head-mounted display.

In some embodiments, the method further comprises applying the received head gesture input in conjunction with the foot motion input to switch from the first system mode to the second system mode.

In some embodiments, the first region of the range of head positions and orientations of the wearer of the head-mounted display corresponds to a first system mode, and the first region of the range of head positions and orientations of the wearer of the head-mounted display The second region of the position and orientation range corresponds to a second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is a head-mounted display head-mounted display. The wearer's head position and orientation are changed from the first region toward the second region.

In some embodiments, the method further comprises displaying a menu overlaid on the image on the head-mounted display, the menu displaying a first menu item for a first system mode; displaying a second menu item for the second system mode; head gesture input and foot motion input are transferred from the first system mode to the second system mode by selecting and activating the second menu item; Applied to switch.

The disclosed technology will be better understood and appreciated from the following detailed description taken in conjunction with the drawings.

1 is a schematic diagram of a system for controlling multiple modes of operation for a head-mounted display (HMD) coupled to a camera head, constructed and operable in accordance with embodiments of the disclosed technology; FIG. FIG. 2 is a schematic block diagram of a computer constructed and operable in accordance with other embodiments of the disclosed technology. 1B is a schematic block diagram of the HMD of FIG. 1A constructed and operable in accordance with further embodiments of the disclosed technology; FIG. 1B is a schematic block diagram of the camera head of FIG. 1A constructed and operable in accordance with other embodiments of the disclosed technology; FIG. 1B is a schematic block diagram of the camera system of FIG. 1A constructed and operable in accordance with further embodiments of the disclosed technology; FIG. 1B is a schematic block diagram of the illumination system of FIG. 1A constructed and operable in accordance with other embodiments of the disclosed technology; FIG. 1A is a schematic block diagram of the footswitch of FIG. 1A constructed and operative in accordance with further embodiments of the disclosed technology; FIG. 2 is a conceptual diagram of a user interface for system 100 configured and operable in accordance with other embodiments of the disclosed technology. FIG. 2A is a conceptual diagram of the flow of information and control for the system of FIG. 1A through the user interface of FIG. 2A, configured and operable in accordance with further embodiments of the disclosed technology; FIG. FIG. 7 illustrates an exemplary layout for multiple P&O areas of an HMD, each mapped to various system modes, configured and operable in accordance with other embodiments of the disclosed technology. 3A illustrates a display of a raw image overlaid with hints indicating which head gestures invoke system modes associated with the zones described with respect to FIG. 3A, configured and operable in accordance with embodiments of the disclosed technology; FIG. 3 illustrates a menu for switching to various system modes overlaid on a video displayed via an HMD, generally configured and operable in accordance with further embodiments of the disclosed technology; FIG. 7 illustrates an implementation for displaying one or more status bars indicative of system settings, constructed and operable in accordance with other embodiments of the disclosed technology. FIG. 7 illustrates a projected image as perceived by a surgeon when a display module is in a display-fixed state, constructed and operable in accordance with other embodiments of the disclosed technology; FIG. Overall, two views are shown, each displayed in a world-fixed state on a virtual screen via an HMD, configured and operable in accordance with further embodiments of the disclosed technology, each view representing a variety of views of the surgeon's head. corresponds to different positions and orientations. FIG. 7 illustrates an exemplary implementation of a surgeon-visible virtual screen state constructed and operable in accordance with other embodiments of the disclosed technology; FIG. 12 illustrates a menu overlaid on a live image feed that allows a user to interact with the system and switch to various system modes, configured and operable in accordance with further embodiments of the disclosed technology. 6A illustrates a system mode menu overlaid on a raw image feed after selecting a system mode menu item from the menu of FIG. 6A, constructed and operable in accordance with other embodiments of the disclosed technology; FIG. 10 illustrates a pre-operative image displayed in a PIP overlaid on a raw image feed, constructed and operable in accordance with further embodiments of the disclosed technology. 6B illustrates the menu of FIG. 6A overlaid on a raw image feed with a PIP, configured and operable in accordance with other embodiments of the disclosed technology; FIG. 3 illustrates a series of raw images with corresponding content displayed within a PIP, configured and operable in accordance with other embodiments of the disclosed technology as a whole; 3 illustrates a series of raw images with corresponding content displayed within a PIP, configured and operable in accordance with further embodiments of the disclosed technology; 6L-6P with corresponding content displayed in two PIPs configured and operable in accordance with further embodiments of the disclosed technology; FIG. 1 illustrates an implementation for drawing symbols on a display of live images, generally constructed and operable in accordance with embodiments of the disclosed technology. 2 illustrates a menu displayed overlaid on a raw image, constructed and operable in accordance with embodiments of the disclosed technology. 3 illustrates an exemplary implementation of a user interface screen that enables designation of a selected viewing area for focus, generally configured and operable in accordance with further embodiments of the disclosed technology. 7B illustrates another view of the menu of FIG. 7A overlaid on a raw image, constructed and operable in accordance with embodiments of the disclosed technology; FIG. 3 illustrates another exemplary implementation of a menu-driven user interface for focusing on a selected viewing area, constructed and operable in accordance with other embodiments of the disclosed technology; Generally, a technique for correcting 3D image distortion caused by a mismatch between an actual working distance and a camera's design working distance, constructed and operable in accordance with the disclosed techniques, is illustrated. 8A generally illustrates the optical system of FIG. 8A at various working distances configured and operable in accordance with embodiments of the disclosed technology; FIG. 8A illustrates the optical system of FIG. 8A configured and operable in accordance with embodiments of the disclosed technology to focus on two different focal planes; FIG. 8A-8B illustrate images displayed via an HMD after application of the shifting technique described according to FIGS. 8A-8B, generally constructed and operative in accordance with embodiments of the disclosed technology; FIG. FIG. 3 is a schematic diagram of a method for controlling a head-up display system constructed and operable in accordance with other embodiments of the disclosed technology.

The disclosed technology overcomes the shortcomings of the prior art by providing a multimodal user interface (UI) for a head mounted display (HMD) for use in surgical applications. An example of such a surgical application is microsurgical procedures performed under the magnification of a surgical microscope. Microsurgical techniques are commonly used in ophthalmic and neurosurgery. Generally, in microsurgical procedures, a surgeon's hands are occupied by surgical tools, and the surgeon requires a wide range of hands-free methods to interface with the surgical microscope and control system parameters. The more robust a system is and the wider the range of features it provides, the more complex the interface for controlling these features. An added challenge in providing an interface using such hands-free methods is transferring complexity to other input means, such as footswitches. In fact, surgeons are often required to use complex footswitches to control various system parameters, taking off their shoes and using their own toes to fumble for buttons and pedals while keeping a close eye on the surgical field. must be used.

The UI of the disclosed technology allows the surgeon to use a relatively simple footswitch while providing a wide range of functionality through various system modes, each supporting a different set of actions. In one implementation, the footswitch has only three pedals, allowing the surgeon to easily locate the appropriate pedal while wearing the shoe and without lifting the heel off the floor. A given hands-free gesture may cause multiple different actions depending on the system mode. This technique provides the surgeon with a wide range of functionality using a relatively small number of hand-free gestures. Hand-free gestures are designed to be intuitive and allow surgeons to naturally interface with the system and control various features, allowing surgeons to use most of their attention as well as their own hands for surgical procedures. enable you to dedicate. The surgeon may switch system modes through one or more hands-free inputs, such as motion sensing, foot or voice control, eye tracking, and the like. In some embodiments, the surgeon switches system modes by footswitch-enabled head gestures.

The systems described herein include multiple components, such as, for example, a camera system and illumination system for acquiring images of the surgical field, and an HMD for viewing these images. Various system modes allow the surgeon to control various aspects of the system. Some system modes allow the surgeon to use gestures to control system parameters such as magnification settings (i.e. zoom in and zoom out), focus and illumination settings for a camera head unit located above the surgical field. enable. Other system modes allow surgeons to use gestures to control the display via the HMD, for example by selecting content to view and manipulating displayed images and preoperative data. enable.

The methods described here may also be applied to surgical procedures other than microsurgery, i.e., where magnified images of the surgical field are available, such as visor-guided surgery (herein "VGS") based on tracking and augmented reality capabilities. It can be done even if there is no. In these procedures, HMDs extend the surgeon's view of the patient, allowing the surgeon to see anatomical features and surgical tools as if the patient's body were partially transparent. These procedures may optionally be performed without any magnified images of the surgical field and therefore without the use of a camera head unit.

Reference is now made to FIG. 1A, which is a schematic diagram of a system for controlling multiple modes of operation for an HMD, designated generally at 100, constructed and operable in accordance with embodiments of the disclosed technology. . System 100 includes an HMD 102, a footswitch 104, a mechanical arm 106, a touch screen 108, a cart 116 that houses and supports a computer 118 (not shown), a camera head 110, and a camera head positioner 111. include. Camera head 110 houses a camera system 112 and an illumination system 114. Camera system 112 includes at least two high resolution cameras (not shown). Mechanical arm 106 connects computer 118 to camera head positioner 111 and camera head 110. Computer 118 communicates with camera head positioner 111 , camera head 110 , camera system 112 , and lighting system 114 via one or more wires and/or cables (not shown) integrated within cart 116 and mechanical arm 106 . electrically coupled to.

Computer 118 controls the position and orientation of camera head 110 via camera head positioner 111 . Camera head positioner 111 includes multiple motors to control the position of camera head 100 in x, y, z coordinates. Camera head positioner 111 further includes a motor for controlling the tilt of camera head 100. Computer 118 also controls operating parameters for lighting system 114 and camera system 112, details of which are discussed below.

HMD 102, footswitch 104, and computer 118 are provided with one or more transceivers (not shown). Transceivers can include, for example, electrical or fiber optic cables (not shown), Wifi, Bluetooth, Zigbee, short range, medium range, long range, and microwave RF, wireless optical means (e.g. laser, lidar, infrared), acoustic means, ultra Any suitable wired and/or wireless communication means and protocols are supported, such as sonic means. Computer 118 is coupled to HMD 102 and footswitch 104 via a transceiver. Computer 118 streams the images to HMD 102 via transceiver, allowing surgeon 120 to view the images on HMD 102. Images may be acquired by the camera system 112 and processed in real time by the GPU to view live video of the surgery. Alternatively, images may be acquired pre-operatively, stored in memory, and rendered in real time by the GPU. In some implementations, images may be streamed or downloaded from a remote server, such as a cloud-based server. For example, in a VGS procedure, a model of a body part generated by segmentation from a CT or MRI image may be stored in memory and rendered in real time by a GPU to the HMD 102 from the perspective of the surgeon 120. In other embodiments, the images sent to HMD 102 are obtained from an external device, such as an endoscope. Computer 118 determines which images to stream to surgeon 120 based on the current system mode and one or more inputs received from the surgeon via the user interface.

In one embodiment, a single surgeon 120 is wearing the HMD 102 to view a magnified video of the surgical field 124 while performing a surgical procedure on a patient 122. Camera system 112 acquires a stream of images of surgical field 124 corresponding to a surgical procedure being performed on patient 122. Computer 118 receives and processes the stream of images and transmits the processed images to HMD 102 via a transceiver. Surgeon 120 views images via HMD 102 and controls one or more settings of system 100 through system modes. Each system mode may control a different set of system parameters. The surgeon 120 may, for example, perform a handsless gesture, such as a head gesture while pressing the foot switch 104, make an eye movement that is tracked by an eye tracker 136, or use acoustically driven means such as, for example, voice control. , may switch system modes to control a selected set of system parameters. Details of these and other user interface methods are discussed later in this specification.

Alternatively, the surgeon 120 may directly control one or more system parameters using a handsless method, such as by depressing the footswitch 104 while making a head gesture. The same gesture may control various features of system 100 depending on the system mode. For example, depressing the footswitch 104 pedal while performing a head gesture may cause one action when the system 100 is in one system mode, but when the system 100 is in another system mode, the footswitch Pressing the same pedal 104 and performing the same head gesture may cause different actions. By providing multiple system modes, a relatively small number of pedals and head gestures may control a wide range of features.

For example, in one system mode, the surgeon 120 may zoom in by pressing the first pedal of the footswitch 104 while moving his head upward. As a result, the surgeon 120 sees a smaller sized surgical field 124 magnified over a larger portion of his field of vision. By pressing the same pedal of the footswitch 104 while performing the same upward head gesture in different system modes, the surgeon 120 can scroll upward in the multiple preoperative images displayed via the HMD 102. . The gestures for each system mode are intended to be intuitive to allow surgeon 120 to naturally interface with system 100. Thus, although system 100 provides a wide range of controllable features, controlling these features with the user interface disclosed herein does not require a lengthy training period. Rather, the surgeon 120 makes gestures that are intuitive to the features he wishes to control. For example, when viewing multiple images displayed across multiple virtual screens, the surgeon 120 may scroll to the left and view the screen displayed to the left by turning his head to the left. Control via head gestures may provide superior precision and resolution. For example, as some surgeons find, controlling focus via head gestures is less effective than using a footswitch because head gestures allow for more precise one-shot control and eliminate overshoot or undershoot. It turns out to be easy compared to In some embodiments, the trajectory of the head gesture determines what is done and the velocity of the head gesture determines the speed at which the action is performed. For example, a surgeon may zoom in quickly by moving his head quickly and zoom in slowly by moving his head slowly.

In some embodiments, the footswitch 104 is a button, similar to how pressing the <Shift>, <Cntl>, and <Caps Lock> buttons on a standard keyboard activates individual feature sets on a keyboard. Equipped to activate other feature sets via press-down. In some embodiments, the second set of features is enabled for all buttons and features of footswitch 104 simultaneously in a single action by surgeon 120. For example, briefly pressing one of the buttons on footswitch 104 (e.g., a button corresponding to a menu) and then releasing that button activates a second set of functions for all buttons on footswitch 104. . An indication that the second feature set has been enabled is displayed to the surgeon 120 via the HMD 102. Alternatively, the second set of features may be activated by voice command or by other means.

In other embodiments, the second set of features is enabled individually for each button on footswitch 104. Accordingly, pressing one of the buttons of footswitch 104 once briefly and immediately releasing it followed by a long press of the same button may activate the second set of head gestures for that button only. For example, a button that enables XY motor control with a head gesture without "shifting" may enable up/down and left/right panning of the ROI when "shifted." In some embodiments, one or more buttons of footswitch 104 are equipped to differentiate between full and half presses to enable various feature sets. For example, one of the buttons on the footswitch 104 may control lighting for the system 100, with a full press of the button controlling one aspect of the lighting, e.g. coaxial lighting, and a half press of the button controlling one aspect of the lighting, e.g. Control other aspects of lighting, such as floodlights (e.g., a full press activates a head gesture that controls coaxial lighting; a half press activates a head gesture that controls floodlights).

In other embodiments, system 100 supports several HMDs simultaneously. One of the users, e.g., surgeon 120, is designated the "principal" surgeon who can control all system settings. In some embodiments, other users wearing HMDs similar to HMD 102 control only the settings of images displayed through their respective HMDs, but not the HMDs they are wearing. They do not control external parameters, for example they do not control the focus of camera system 112 and the illumination by illumination system 114. In other embodiments, other users wearing HMDs similar to HMD 102 may control one or more settings for system 100.

The description of system 100 that follows is described from two perspectives. A description of a typical hardware layout for the components of system 100 is provided below with respect to FIGS. 1B-1G. 2A-2B illustrate exemplary logical layouts for the user interface of system 100. Note that these figures are intended to illustrate only one exemplary implementation of possible implementations for system 100. These figures are not intended to limit the invention to any particular implementation.

Reference is now made to FIG. 1B, which is a schematic block diagram of computer 118 of FIG. 1A, constructed and operable in accordance with other embodiments of the disclosed technology. It should be noted that this diagram is only a typical implementation, and where applicable, software modules may be used in place of hardware modules and vice versa. Computer 118 includes at least one processor 118A, at least one transceiver 118B, a power supply 118C, at least one memory 118D, at least one analog-to-digital A/D converter 118E, at least one digital-to-analog D/A converter 118F, and , a mechanical arm controller 118G, a camera system controller 118H, a lighting system controller 118I, a head tracker controller 118J, an image processor 118K, an eye tracker controller 118L, an acoustic analyzer 118M, and at least one bus 118N. Processor 118A, transceiver 118B, power supply 118C, memory 118D, A/D converter 118E, D/A converter 118F, mechanical arm controller 118G, camera system controller 118H, lighting system controller 118I, head tracker controller 118J, image processor 118K, Eye tracker controller 118L and acoustic analysis device 118M are coupled via at least one bus 118N.

Although computer 118 is shown as a single unit, this is for conceptual purposes only. Computer 118 may be implemented as multiple distributed units that act as a single control unit for system 100. For example, at least one processor 118A, transceiver 118B, power supply 118C, and memory 118D each operate in a distributed and coordinated manner across components of system 100 (e.g., camera head 110, HMD 102, footswitch 104, remote server, etc.). may include multiple processors, memory, power supplies, and transceivers.

Mechanical arm controller 118G controls a motor included within camera head positioner 111 to adjust the x, y, z axes or tilt of camera head 110. Camera system controller 118H controls the operation of the cameras included within camera system 112, such as selecting the camera or cameras for image acquisition, adjusting F-number, focus, etc. Lighting system controller 118I controls operation of lighting system 114, such as selection of illuminators, intensities, one or more filters, and the like. Head tracker controller 118J controls the operation of one or more head tracker components provided in system 100 for tracking the head of surgeon 120. The image processor 118K performs functions such as zooming in, zooming out, digital filtering, sharpening, smoothing, color correction, fusing several image sources, embedding one image source into another in a picture-in-picture form, etc. Depending on the application, images received from either camera system 112, memory 118D, or GPU are processed. The eye tracker controller 118L controls the eye tracker configured by the HMD 102. Acoustic analysis device 118M applies sound recognition to sounds received from microphones provided to system 100, such as by applying a speech recognition algorithm to recognize voice commands.

Reference is now made to FIG. 1C, which is a schematic block diagram of the HMD 102 of FIG. 1A, constructed and operative in accordance with further embodiments of the disclosed technology. The components of HMD 102 may be incorporated into an adjustable frame worn by surgeon 120, as shown in FIG. 1A. HMD 102 includes driver 102A, transceiver 102B, memory (eg, buffer) 102C, and display module 130. In some embodiments, HMD 102 further includes a liquid crystal (LC) shutter 132, a tracking component 134, an eye tracking component 136, a microphone 138, a speaker 139, a vibration motor 141, and at least one bus 102D. Driver 102A, transceiver 102B, memory 102C, display module 130, shutter 132, tracking component 134, eye tracking component 136, microphone 138, speaker 139, and vibration motor 141 are coupled via at least one bus 102D.

Display module 130 may be a binocular display that includes two display modules 130A and 130B to project images for each eye of surgeon 120. In some embodiments, HMD 102 is monocular. Shutter 132 is mechanically coupled to display module 130 and is operable to open and close. When shutter 132 is closed, display module 130 is opaque and enhances the image displayed via HMD 102. When shutter 132 is fully or partially open, display module 130 is transparent in the area where shutter 132 is open, allowing surgeon 120 to see an image overlaid on top of the real world view. do. A partially or fully open shutter 132 may be suitable for augmented reality or real world view configurations. Alternatively, open shutter 132 allows surgeon 120 to interact and communicate with other surgeons in the operating room, such as other surgeons or nurses, as desired. Tracking component 134 obtains position and/or orientation data regarding HMD 102 and transmits the position and/or orientation data to head tracker controller 118J via respective transceivers 102B and 118B. Microphone 138 records voice commands or other sounds made by surgeon 120 and transmits the sounds to acoustic analysis device 118M via respective transceivers 102B and 118B. Eye tracking component 136 acquires eye movement data for surgeon 120 and transmits the eye movement data to eye tracker controller 118L via respective transceivers 102B and 118B.

The display module 130 is placed on the HMD 102 so as to be located in the line of sight of the surgeon 120, as shown in FIG. 1A. Shutter 132 is configured by display module 130 and can be switched between open, closed, and partially closed states. HMD 102 is always transparent when shutter 132 is open, and always opaque when shutter 132 is closed. When shutter 132 is fully open, surgeon 120 views the real world through the optics of display module 130, which is transparent. Shutter 132 may be partially closed to hide the background behind a feature displayed in a picture-in-picture (PIP) overlaid on display module 130, for example, and may be partially closed to hide the background behind a feature displayed in a picture-in-picture (PIP) overlaid on display module 130; Enhances the contrast with the real world view seen through the part. This is useful in displaying one or more features within the PIP during a VGS procedure in augmented reality applications. Display module 130 may be implemented using any suitable technology as known in the art. The display module 130 may include an image source, eg, a small OLED monitor, as well as image projection optics, eg, waveguide optics, prismatic optics, projection onto a visor, and the like. Alternatively, the display module may scan images directly to the surgeon's 120 retina without a monitor. HMD 102 receives an image source from computer 118.

Tracking component 134 includes components for tracking head movement of surgeon 120. Tracking components 134 may include MEMs based on accelerometers 134A and gyroscopes 134B, one or more light emitters 134C, optical sensors 134D, electromagnetic sensors 134E, and the like. In some embodiments, tracking component 134 further includes a compass (not shown). Light emitter 134C may be one or more active light emitting devices, such as light emitting diodes (LEDs), or one or more light reflectors. Tracking component 134 generates translational and rotational motion data for HMD 102 to determine updated orientation and/or position coordinates for HMD 102.

In some embodiments, head tracking is performed external to the HMD 102, for example by using a camera (not shown). Optionally, tracking components 134 include components internal to HMD 102 as well as components external to HMD 102. In some embodiments, the HMD 102 includes an eye tracker component 136 for tracking the gaze of one or both eyes of the surgeon wearing the HMD 102. In some embodiments, microphone 138 is integrated into HMD 102 to record voice commands of surgeon 120. Alternatively, microphone 138 may be external to HMD 102 and integrated into camera head 110, for example. It should be noted that the embodiments described herein are intended as exemplary only, and any suitable combination and arrangement of components may be used to achieve the functionality disclosed herein. obtain. Computer 118 controls tracking component 134 via tracker controller 118I. Computer 118 communicates with HMD 102 via transceivers 118B and 102B to display the image feed on HMD display module 130 according to the selected system mode and additional settings and parameters defined by surgeon 120, described in more detail below. control.

Reference is made to FIG. 1D, which shows a schematic block diagram of camera head 110 (FIG. 1A), constructed and operable in accordance with other embodiments of the disclosed technology. Camera head 110 houses a camera system 112, a lighting system 114, and optionally a microphone 138. In some embodiments, camera head 110 further houses a tracking subsystem (not shown) that includes sideways and/or downwards optical tracking components, sensors and/or light emitters, and possibly other tracking components. Camera head 110 is mechanically coupled to cart 116 via arm 106 and camera head positioner 111. Camera head 110, camera system 112, lighting system 114, and microphone 138 are electrically coupled to computer 118 (FIG. 1B) by one or more cables and/or wires integrated within arm 106. Computer 118 controls the xyz position and tilt of camera head 110 relative to surgical field 124 by controlling motors integrated within camera head positioner 111 .

In some embodiments, other positioning mechanisms, such as, for example, a robotic arm, such as a 6-DOF (degrees of freedom) robotic arm commonly used for neurosurgical applications, are used to obtain images of the surgical field 124. can be used in place of camera head positioner 111 and mechanical arm 106 to control the position of camera system 112. Unlike ophthalmic surgery applications where changes in camera position and orientation are minute, in neurosurgical applications the changes in camera position and orientation are relatively large. For such applications, the surgeon 120 activates the ability to fix the camera control to his or her head and uses head gestures to control the camera position and orientation of the camera head 110 via the robotic arm. It may be possible to do so. Surgeon 120 enables this feature, such as through a user interface such as footswitch 104, or by issuing a voice command. The head tracking functionality of HMD 102, described in detail below, senses rotational and translational movements of HMD 102 and converts them into corresponding rotational and translational movements for a robotic arm that controls camera head 110. Optionally, the user interface allows the surgeon 120 to memorize or "bookmark" the current position and orientation of the camera head 110 with respect to the camera, whereby this position and orientation can be updated via the HMD 102, for example. It can be restored later by selecting this feature from the menu displayed.

Computer 118 controls the operation of camera system 112 and illumination system 114 according to system modes and inputs received from surgeon 120 via the user interface. Computer 118 further controls system 100 according to voice commands detected by microphone 138.

Computer 118 may digitally or optically/mechanically control display features for HMD 102. For example, computer 118 may digitally perform zoom-in/zoom-out operations by reducing the area of pixels from the overall video captured by camera system 112 and stream the reduced area to HMD 102. Alternatively, computer 118 may implement the zoom in/zoom out functionality optically by adjusting the optics of high resolution cameras 140A and 140B. Similarly, computer 118 may digitally implement a scrolling function by shifting regions of pixels taken from the overall video captured by camera system 112 and stream those pixels to HMD 102. Alternatively, the computer 118 adjusts the position of the camera head 110 in the lateral direction along the Scrolling may be performed mechanically.

Reference is now made to FIG. 1E, which is a schematic block diagram of the camera system 112 of FIGS. 1A and ID, constructed and operative in accordance with further embodiments of the disclosed technology. Camera system 112 is housed in camera head 110 and coupled to computer 118 via one or more wires and cables integrated within mechanical arm 106 (FIG. 1A). Computer 118 further controls each of the components within camera system 112, such as by selecting them to receive image feeds.

In one embodiment, camera system 112 includes at least two high resolution cameras 140A and 140B. Cameras 140A and 140B are provided with high resolution optics for capturing high resolution magnified stereo images of surgical field 124. In some embodiments, camera head 110 further includes either an iOCT scanner 142, an IR camera 148, or other camera for multispectral imaging. Camera system 112 is configured to stream images captured by cameras 140A and 140B to computer 118, which processes the images and optionally sends them to HMD 102. Images from cameras 140A and 140B are streamed to surgeon 120's eyes via HMD display modules 130A and 130B. The surgeon's brain generates 3D images from the streamed video. Computer 118 controls the display of image feeds via the HMD according to the selected system mode and any additional settings and parameters defined by surgeon 120. Computer 118 is used to control aspects of system 100 and controls additional cameras that capture images that are not displayed to surgeon 120. For example, computer 118 controls an IR camera for detecting movement within the operating room. Image processor 118L processes images acquired by camera system 112 according to settings defined by surgeon 120, for example, to perform color correction, sharpening, filtering, zooming, etc. Computer 118 may adjust the illumination, focus, FOV, and magnification for the image feed according to one or more settings instructed by surgeon 120 via a user interface described below, and stream the processed image feed to HMD 102. .

System 100 allows surgeon 120 to select a magnification level for the video acquired by camera system 112 at the beginning of a surgical procedure and dynamically adjust the magnification as needed throughout the surgical procedure. Make it. System 100 allows for dynamically updating region of interest (ROI), zoom, and focus configurations using mechanical, optical, electrical, or digital means. Mechanical and optical updates to the XY positioning and zoom optics for cameras 140A, 140B are performed via camera system controller 118H (FIG. 1B). Digital zoom functionality and XY positioning are implemented by changing the size and XY position of the ROI displayed via HMD 102.

In some embodiments, the surgeon 120 uses head gestures to dynamically update the ROI and zoom settings. In some embodiments, the surgeon 120 enables the system 100 to automatically center the camera system 112. In this case, system 100 automatically updates the XY position dynamically. Some examples of dynamic updates include auto-centering, auto-zooming, auto-focusing, and auto-illumination.

System 100 provides automatic centering functionality to dynamically update the raw image feed. One type of automatic centering follows the illuminated area during retinal procedures when fiber illumination is used and only a portion of the retina may be illuminated. Other types of automatic centering follow the center of the patient's eye during anterior segment procedures, such as cataract procedures (eg, when the surgeon moves the patient's eye). Other types of automatic centering follow tooltips. It should be noted that the examples given herein are intended only as typical implementations and are not intended to limit the invention. Automatic centering may be performed on any patient anatomy, any medical device, or generally on any element or information appearing within the image.

Typically, retinal procedures are performed using an additional lens (not shown) suspended from the camera head 110 directly above the center of the patient's 122 eye. Therefore, during these procedures, camera head 110 must not be moved unless the patient's eyes are moving, and movement of camera head 100 will move this additional lens. This limitation means that adjusting the position of camera system 112 via XY motor control is not a suitable technique for tracking the illuminated area during retinal procedures. As a result, adjustments to the displayed ROI for these procedures are made digitally as part of the auto-centering function to keep the physical position of the camera system 112 fixed. The resolution of cameras 140A and 140B is generally higher than the resolution of the display monitor of HMD 102. Therefore, digital adjustment of the ROI is performed by selecting a subset of the entire pixel frame acquired by camera system 112. When the surgeon 120 views a small ROI as a zoom in the overall image, the ROI may be dynamically selected from the entire pixel frame. Centering and especially automatic centering functions can be performed either by controlling the XY motors and/or by adjusting the ROI, depending on the needs and circumstances. For example, depending on the type of treatment, one technique may be more suitable than another.

In other embodiments, automatic centering to the illumination region, center of the patient's eyes, tool tips, etc. is performed at discrete time intervals. For example, if the distance between the pupil and the edge of the image is smaller than a predefined threshold, centering is performed either by changing the ROI or by controlling the XY motors so that the pupil appears in the center of the image. be done. As another example, centering is performed during a posterior segment procedure by automatically updating the ROI so that the illumination spot is centered in the image if the illumination spot on the retina exceeds a predefined margin. This is in contrast to continuous automatic centering, where the image is centered frame by frame. Continuous automatic centering can pose a risk if the patient's eyes are moving but appear stationary to the surgeon.

In a further embodiment, the centering itself is performed in a continuous spatial manner, as opposed to a discrete spatial manner. For example, the image is updated in a stepwise and continuous manner until centered (eg, the XY motors and/or the ROI are affected in a stepwise and continuous manner) to avoid sudden changes in the image.

In other embodiments, prior to centering, system 100 alerts the user of pending changes. The warning may be via an alarm displayed on the HMD, a sound, a vibration in the HMD, or the like. The alert notifies the surgeon to expect changes in the image and/or changes in the XY position of the camera head unit. Optionally, the user may cancel pending centering via UI 160. For example, when a warning is given that the system is about to center the image, the user has a time window (eg, 3 seconds) to cancel the centering. In a further embodiment, when centering is canceled, system 100 disables subsequent automatic centering. This may be for a predetermined period of time (eg, 5 minutes), until the system mode is changed (eg, by switching from a previous mode to a later mode), or for the duration of a surgical procedure. In other embodiments, once the occurrence of automatic centering is canceled, the system disables automatic centering until a noticeable change occurs in the image, such as can be detected by an image processing algorithm or deep learning algorithm.

System 100 provides automatic zooming functionality to dynamically update the live image feed. The automatic zoom function may be implemented mechanically by controlling zoom optics within camera head 110, or digitally on the live image feed via image processor 118K. This configuration allows zoom settings to be automatically changed based on surgeon-specific preferences. One type of automatic zooming remembers the last zoom factor used in each system mode (eg, the previous mode) and automatically sets the zoom when the system is turned on and the mode is switched. Another type of automatic zooming identifies various stages in a procedure and remembers the surgeon's preferences for each stage. Yet another type of automatic zooming dynamically changes the zoom according to the size of the light spot produced by the fiber illumination, which changes its size by being held by the surgeon at various distances from the retina. The zoom magnification corresponds to the spot size. As the surgeon moves the fiber tip farther and the spot becomes larger, the displayed image zooms out. As the surgeon moves the fiber tip closer and the spot becomes smaller, the displayed image zooms in.

Similarly, system 100 allows for manual and automatic focusing. For automatic focusing, the surgeon 120 may specify a point within the surgical field that should be kept in focus. Alternatively, the surgeon 120 may have the system automatically focus on the tip of the tool that the surgeon is holding and manipulating. Alternatively, the surgeon may have the system automatically focus on the center of the illuminated area in the retinal procedure. The system maintains focus on the specified location, fixture, or spotlight. In some embodiments, system 100 provides one-time focus on a specified point. Surgeon 120 may specify a point on the raw image using a combination of head/foot gestures and/or surgical tools to focus on that point as desired.

Reference is now made to FIG. 1F, which is a schematic block diagram of a lighting system 114 (FIG. 1A) constructed and operable in accordance with other embodiments of the disclosed technology. Illumination system 114 is housed within camera head 110 and coupled to computer 118 via one or more wires and cables integrated within mechanical arm 106. Illumination system 114 includes one or more of a white floodlight 150, an IR floodlight 152, a coaxial light 154A, and a coaxial light 154B. Coaxial lights 154A and 154B operate with cameras 140A and 140B, respectively. Generally, illumination system 114 includes at least one visible light emitter for illuminating surgical field 124. The light emitter may be combined with one or more filters (not shown). For example, the lighting may be based on white-emitting LEDs with a filter that blocks blue light due to phototoxicity considerations so that the emitted light is not itself white. Illumination system 114 may illuminate with non-visible illumination, such as infrared (IR) light or ultraviolet (UV) light. In some embodiments, system 100 includes IR illumination combined with a low resolution/wide field of view camera to detect motion within surgical field 124. Coaxial illumination is used in anterior segment ophthalmological procedures to create the red-eye effect caused by the reflection of light from a patient's retina. Computer 118 controls lighting system 114 with lighting controller 118H, such as by selecting light sources, intensities, and the like.

In some embodiments, system 100 allows automatic lighting. Image processor 118K analyzes the raw image feed for under- or over-illuminated areas and notifies computer 118 to manipulate position or intensity by illumination system 114. Optionally, the illumination may be optimized on elements selected by the surgeon, such as, for example, surgical instruments. In some embodiments, the system 100 identifies that the surgeon 120 has completed one stage and is beginning a new stage, and automatically optimizes the lighting for the new stage. For example, if an external phaco system streams data, system 100 will know that the mode is currently set to phaco mode. In such a case, the floodlights are automatically switched off and the red-eye lighting is switched on.

Reference is made to FIG. 1G, which shows a schematic block diagram of footswitch 104 (FIG. 1A), constructed and operable in accordance with further embodiments of the disclosed technology. A footswitch is shown with three sensors 104A, 104B, and 104C, a transceiver 104D, and a vibration motor 104E. The foot switch is located on the ground adjacent to the surgeon's 120 foot. Surgeon 120 controls the operational characteristics of system 100 by making foot movements detected by footswitch 104. Sensors 104B, 104C, 104D are configured to sense foot movement and send notifications to computer 118 (FIG. 1B) via respective transceivers 104A and 118A. Sensors 104B, 104C, 104D may be pedals, buttons, touch pads, joysticks, pressure-based, image-based, etc. Note that this implementation is intended to be merely exemplary, and footswitch 104 may have more or fewer sensors, or multiple buttons, each with various options. In some embodiments, footswitch 104 is a wearable device configured with an inertial sensor to track foot gestures. Optionally, the wearable footswitch is additionally or alternatively configured with a light sensor and a light emitter (not shown) for detecting one or more foot gestures. For example, a light emitter configured with a wearable footswitch projects a pattern onto a surface, such as a floor. One or more optical sensors detect changes in the projected pattern to infer foot gestures. The optical sensor may be configured with a wearable footswitch, but this is not required. Vibration motor 104E provides feedback to surgeon 120 regarding settings and system parameters for system 100 via footswitch 104. Reference is now made to FIG. 2A, which illustrates a schematic block diagram of an exemplary user interface (UI) 160 for system 100 (FIG. 1A), constructed and operable in accordance with other embodiments of the disclosed technology. UI 160 is a logical view of the components of system 100 described above with respect to FIGS. 1B-1G that interface with surgeon 120. It should be understood that the components included in UI 160 are intended to be merely exemplary, and components may be added to or removed from UI 160 according to various implementations. FIG. 2A shows components in various system units logically grouped according to their functionality. UI 160 includes two modules: UI Input 160A, which includes components of system 100 that receive input from surgeon 120, and UI Output 160B, which includes components of system 100 that serve as outputs in response to received inputs. including. Computer 118 receives user input via UI input 160A, processes the input, and issues one or more system responses via UI output 160B.

UI input 160A includes head gesture tracker 162. Head gesture tracker 162 includes a tracking component 134 (FIG. 1C) that is integrated with and internal to HMD 102, and optionally a head gesture tracker component 164 that is external to HMD 102. For example, head gesture tracker component 164 may be a camera located within the operating room that captures images of surgeon 120. UI input 160A further includes a footswitch 104 (FIG. 1A) for receiving foot-based input, an eye tracking system 166, an audio system 168, and a touch screen 108 (FIG. 1A) for receiving input from other surgeons. )including. Eye tracking system 166 includes eye tracking component 136 (FIG. 1C) coupled to eye tracker controller 118L (FIG. 1B), both of which are configured to sense and track the direction of the surgeon's gaze. Acoustic system 168 includes a microphone 138 (FIG. 1C) coupled to a voice recognition device 118M (FIG. 1B), which together receive, detect, and process acoustic signals and voice commands issued by the surgeon. It is composed of In some embodiments, microphone 138 is external to HMD 102.

UI output 160B includes display module 130 (FIG. 1C), shutter 132, touch screen 108 (FIG. 1A), camera head 110 (FIG. 1D), camera head positioner 111, camera system 112 (FIG. 1E), and lighting system 114. (FIG. 1F). Based on the received inputs, computer 118 controls the state of shutter 132, the display of content on display module 130, and the display of content on touch screen 108, as well as camera head 110, camera head positioner 111, camera system 112. , and control settings for the lighting system 114. In some embodiments, the UI output 160B is connected to one or more systems external to the system 100, such as a phacovitrectomy machine, and provides the surgeon 120 with direct access to the system external to the system 100 via the UI 160. Give control. In some embodiments, system 100 further supports connection to a 2D or 3D external monitor (not shown) for displaying any or all of the available video streams as well as supporting multiple HMDs simultaneously. do.

In some embodiments, UI output 160B further includes a speaker, such as speaker 139 (FIG. 1C), for emitting sound output, such as, for example, a tone or an audio message to notify surgeon 120. In some embodiments, UI output 160B further includes one or more vibration motors 161, such as vibration motor 141 (FIG. 1C) or vibration motor 104E (FIG. 1G). For example, the vibration motor may be an eccentric rotating mass vibration motor (ERM) using a DC vibration motor, or a linear resonant actuator (LRA) to display system parameters to the surgeon 120. The vibration motor may be integrated into either the HMD 102, the footswitch 104, or any other component of the system 100. For example, vibrations may indicate that the x-motor has reached its limit of motion, or that the system 100 has automatically identified a tool that is located extremely close to the retina. In some embodiments, the system 100 monitors the surgeon's 120 head to ensure that the surgeon's 120 posture is ergonomically appropriate, and the system 100 monitors the surgeon's 120 head so that the surgeon's 120 Alerts you if placed in an ergonomically inappropriate position that could result in pain or injury. The alarm may be emitted, for example, via a message displayed by the HMD, a vibration, or the issuance of an audible alarm, for example a tone.

Surgeon 120 interacts with system 100 via user interface 160 while wearing HMD 102 to perform a surgical procedure on patient 122 . Thus, using hands-free gestures, the surgeon 120 can control the operating parameters of the system 100, such as the image processing parameters of the computer 118 (e.g., zoom, color correction), the content displayed via the HMD 102 (e.g., the camera system 112, preoperative data retrieved from memory, overlaid virtual features, virtual screens and menu openings), the state of the shutter 132 (e.g., open, partially open, closed). control settings for the illumination system 114 (e.g., light emitters, filters, intensity, wavelength, angle); control settings for the camera system 112 (e.g., camera, zoom, focus, shutter); control settings for the camera head 110 (e.g., orientation and alignment by adjusting xyz motors and tilt), selection of menu items to control any of the above, control of system modes, etc.

In some embodiments, UI 160 allows storing snapshots and video and recording audio for later use. Audio recordings can be converted to text for use as notes, surgical summaries, naming and tagging filters. Keyword tags may then be applied by deep learning algorithms. In some embodiments, the system 100 supports automatic generation of a surgical report based on a predetermined template with placeholders for preoperative data, snapshots from raw images, and audio text of recorded notes/summaries. do. Automatic report generation supports additional identification of data such as, for example, patient 122 name, surgeon 120 name, date, time and duration of the surgical procedure. Where appropriate, machine learning and image processing may be applied to obtain treatment-related data. For example, the type of surgical instrument used and an image feed of the surgical procedure may be obtained to populate the report. UI 160 provides a "Send to Report" menu item that allows the surgeon to upload selected snapshots and pre-operative images to a report. In some embodiments, UI 160 presents an automatically generated claim summary that may be configured according to a predefined template. The claim summary may include a list of surgical procedures performed, etc.

If video and audio are recorded separately, the audio recording is saved with a timestamp that allows the audio to be synchronized with the corresponding video file. Similarly, when audio notes are converted to text, these are saved with synchronized timestamps that allow the notes to be displayed as subtitles on the corresponding video feed.

In some embodiments, the UI input 160A may be used to respond to multiple simultaneous hand movements performed by the surgeon 120, such as head gestures detected via the head gesture tracker 162 simultaneously with foot movements detected via the footswitch 104. The device is configured to detect input as a free gesture. In these embodiments, the enable input from footswitch 104 serves to prevent inadvertent head movements by surgeon 120 from affecting the operation of system 100 by enabling head gestures. Upon receiving input from UI input 160A, computer 118 causes a corresponding action depending on the system mode.

In some embodiments, the surgeon 120 may temporarily set the system 100 to an activation-free state in which default actions are activated by various head gestures without activation input from the footswitch 104. For example, the surgeon 120 may set the default actions to be focus control for head up and head down gestures, zoom in and out for left and right head gestures, or enable free head gestures to switch system modes. It's fine.

A description of some exemplary user inputs available to the surgeon via UI input 160A to control aspects of system 100 now follows. This list is not intended to be limiting.

- Head Gestures A common user interface method for surgical HMDs based on the use of head gestures. Head gestures are detected by head tracker 162 of UI input 160A (FIG. 2A). Surgeon 120 may set system parameters such that some head gestures are free to enable, allowing him to control system 100 using only head gestures. For example, for certain stages of a medical procedure, the surgeon 120 may prefer to work in an activation-free mode, allowing himself to operate the system 100 using only head gestures without activation via the footswitch 104. It can be possible. Surgeon 120 may issue voice commands to turn off activation via footswitch 104 for this effect, and simply use head gestures to control features such as zoom or focus, for example. The surgeon 120 may, for example: zoom, focus, and illumination intensity using the camera head 110;
For example, scrolling within a data item such as a preoperative image displayed on the display module 130,
o To control continuous and discrete system characteristics such as lighting on/off, image enhancement options, enabling/disabling directly via head gestures or indirectly via menu controls using head gestures Use head gestures.

- Head gestures activated by footswitch 104.
Footswitch 104 (FIG. 1G) is provided with one or more pedals, such as pedals 104A, 104B, 104C, for example. Each of the pedals 104A, 104B, 104C may enable a group of head gestures for controlling various sets of system parameters. Each group consists of all the gestures used, such as up-and-down movements, side-to-side movements, gaze, small nods, and back-and-forth movements. The use of footswitch 104 with head gestures allows surgeon 120 to control more system features than are available with head gestures alone. Enabling head gestures via additional user input also improves system safety by eliminating inadvertent changes, e.g. accidental switching to a different system mode due to natural head movements. let

For example, a footswitch with three pedals may enable the following groups of five head gestures in normal system mode.
○Press left pedal (104A)
○Up and down head gestures: Focus on the enlarged image.
○ Subtle up-and-down nod: Toggle between enabling and disabling automatic focusing. For example, when enabling one-time autofocusing at a specified point, a crosshair may appear as an overlay on the image, and the user must rotate their head to specify a point within the surgical field to autofocus. You can move the cross. Releasing the pedal selects the specified point. Surgeon 120 can determine which type of automatic focusing (e.g., one-time automatic focusing on a designated point, continuous automatic focusing on a designated point, continuous automatic focusing on a designated point, You may choose whether automatic focusing on the tool tip or automatic focusing on the center of the light spot is activated.
○Left and right head gestures: Zoom the enlarged image.
○Slight left/right swing: Switch between enabling and disabling automatic zooming.
○Press left + center pedal (104A and 104B)
○ Up/down and left/right head gestures: Scrolling in the y and x directions in the enlarged image when the ROI is displayed.
○ Subtle up-and-down nod: Switching between enabling and disabling automatic centering based on ROI changes.
○Press down the center pedal (104B)
○ Up/down and left/right head gestures: menu navigation.
○Long gaze: Open a submenu or select a menu that appears in the foreground.
o Release of center pedal 104B: Activates the selected menu item.
○Press right + center pedal (104C and 104B)
○ Up/down and left/right head gestures: move the y and x motors.
○Slight up/down nod: Switching between enabling and disabling automatic centering based on XY changes.
○Press the right pedal (104C)
○Up and down head gesture: floodlight intensity.
○ Subtle up and down nod: Toggle floodlight on and off status.
○Left and right head gestures: red eye (coaxial) illumination intensity.
○Slight left/right swing: Switch coaxial lighting on and off state.

・Acoustic input.
System 100 is configured to respond to acoustic signals detected via one or more microphones 138. Surgeon 120 may issue one or more voice commands and/or sounds to enable or disable features, switch to a different system mode, modify or enhance displayed images, etc. Acoustic input is detected by microphone 138 and processed by computer 118. For example, the surgeon 120 may use one click sound to enable/disable one feature and a second sound to enable/disable a second feature. The sounds for activating features of system 100 may be user-specific and customized by surgeon 120. Surgeon 120 may teach system 100 to recognize a preferred set of sounds. A microphone (not shown) physically placed near the surgeon's 120 mouth (eg, part of the HMD 102) may pick up very quiet sounds that are outside the patient's 122 hearing range. This can be advantageous in procedures performed without anesthesia, such as many eye surgical procedures. In these situations, it may be preferable for the patient 122 not to hear voice commands issued by the surgeon 120 to control the system 100.

・Hand gesture.
In some embodiments, system 100 is operated by a handheld controller (not shown) or by an external tracker (not shown) that senses hand gestures made by surgeon 120 and sends them to computer 118 for processing. Provide one or more captured hand gesture inputs. For example, a handheld controller may be designed similarly to an electronic mouse, a remote control, a joystick, or it may be a wearable device, such as a bracelet or a ring. Alternatively, hand gestures may be captured optically by cameras and/or other sensors integrated into the HMD or elsewhere. Hand gestures may be used in conjunction with any of the UI techniques disclosed herein. For example, hand gestures may be used in conjunction with menus or virtual 3D buttons that are "locked" to positions within the operating room or to the patient's eyes. In some embodiments, system 100 detects hand gestures by tracking the user's fingers. In other embodiments, the system 100 detects hand gestures by tracking a surgical tool held by the surgeon 120 such that the surgeon 120 "touches" and "touches" a menu item or virtual 3D button with the surgical tool. "Press".

- Physical control via camera head 110.
Camera head 110 may be provided with one or more manually operated buttons, handles, and switches, and a control panel. These may be controlled by the surgeon 120 or an attending nurse.

- User eye tracking combined with virtually displayed menus or buttons.
Surgeon 120 may navigate the virtually displayed menu options via eye tracker 136 (FIG. 1C). For example, the surgeon may stare at a virtual menu item for a predetermined period of time, or blink according to a pattern, or perform other such "eye gestures." This UI may be used in conjunction with any of the other UIs described herein. In addition to display-fixed virtual menus, menus may be world-fixed or anatomy-fixed. For example, 3D virtual buttons may be displayed "locked" to the patient's 122 eye, such as around the treatment area. The surgeon 120 may "press" the displayed button by gazing at the virtual button, and the eye tracking component 136 detects the gaze.

·touch screen.
Control and operating parameters for system 100 may be set, monitored, and modified via touchscreen 108. Touch screen 108 may be covered by sterile nylon for use by an operating room nurse or surgeon 120. In addition to controlling operational settings by the surgeon 120 wearing the HMD 102, the touch screen 108 allows for controlling modes, settings, and preferences that cannot be controlled by the surgeon 120 wearing the HMD 102. Make it. Touchscreen 108 may display any of the video feeds available to system 100, either in raw form or in a format visible to the wearer of HMD 102, eg, using overlays, PIP, etc.

System modes determine how system 100 interprets and reacts to user input made by surgeon 120 to modify the user interface configuration. For example, in preoperative mode, the surgeon 120 scrolls through multiple images displayed via the HMD 102 by simultaneously pressing the pedal of the footswitch 104 while turning his or her head. In normal mode, the surgeon 120 zooms in or out the image displayed via the HMD 102 by simultaneously pressing the pedal of the footswitch 104 while moving his or her head to the left (zoom in) or right (zoom out). Zoom out. In some embodiments, computer 118 only enables execution of the action while the appropriate pedal of footswitch 104 is pressed.

Below are some typical system modes provided by system 100 and the characteristics they control. The system mode defines which parameters are manipulated for the system 100, such as display settings for the HMD 102 or operational settings for the camera system 112. Surgeon 120 selects and initiates the mode through a combination of UI inputs, including head gestures, eye movements, voice commands, and foot movements, detected by UI 160A. Computer 118 processes the UI input and causes a system response via one or more system output components of UI 160B. The system mode determines responses such as one or more of the content displayed on the HMD 102, the screen state, the state of the shutter 132, UI functionality, and operational settings for the camera head 110. The exemplary mode list below is not intended to be exhaustive and does not include all of the characteristics controllable by each system mode.

・Normal mode (default mode). In this system mode, default values are generally valid, but may be customized by the surgeon 120. An image feed captured by camera system 112 is displayed via HMD 102. Surgeon 120 primarily uses UI 160 to control display attributes such as, for example, zoom, focus, XY position for camera system 112, and lighting settings for illumination system 114.

・Transparent mode. In this system mode, display module 130 is turned off, LC shutter 132 is open, and surgeon 120 can see the real world view.

- Posterior segment non-contact lens mode. This system mode allows the surgeon 120 to view the surgical field 124 in an inverted or posterior segment mode. The settings for this mode are as follows.
○Eye reversal Reversal.
○Floodlights off. This may be user configurable.
○Red-eye lighting off.
○Color scheme Fiber illumination (color correction is material specific).
○Image enhancement Posterior segment eigenvalue.
○Motor status Limited speed and range. These parameters can be limited by the surgeon by controlling the progression or speed of the head gesture.

・External mode. These system modes correspond to additional external devices (not shown) that may be connected to system 100, such as, for example, an endoscope. One of the external system modes has the following settings.
○Red-eye lighting on. Intensities for red-eye and floodlights are predefined.
○Overlay Additional data is overlaid from an external system.
o Functions of the UI 160 can be customized. For example, the definition for footswitch 104 may be different in this mode than in other modes. This is to allow the surgeon 120 to control an external machine using the footswitch 104. The menu that displays the options will be modified accordingly to reflect the different functionality.

- Preoperative data mode (“preoperative” mode). In this mode, the user views pre-planning data, such as preoperative images or notes and/or patient data that he has prepared. After viewing the data, the user returns to the previous mode to continue the surgical procedure. The user may also select a set of images visible within the PIP from the image library and scroll within the selected images after returning to the previous mode, as described in more detail below in FIGS. 6C-6J. . The settings for this mode are as follows.
○Display content. In place of the live magnified image acquired at the surgical field 124, the HMD displays a GUI that allows the user to access, view and scroll through any data desired.
○UI function. This mode allows viewing and scrolling through preoperative data. For example, gestures in normal mode may control zoom or focus of an enlarged image. In pre-operative data mode, no enlarged image is displayed and the same gesture allows scrolling etc. of the displayed pre-operative data.
○Lighting. Illumination by the illumination system 114 may be set to a minimum value in the preoperative data mode or turned off, dimmed, and switched to other light emitters included in the illumination system, such as IR lighting with IR floodlights 152. . The IR lighting allows the surgeon 120 to detect the movements of the attending nurse while viewing the pre-operative data and may be turned on automatically. The illumination may be returned to the previous non-IR setting when the surgeon 120 switches back to normal system mode.

- PIP (Picture in Picture) system mode. Picture-in-picture (PIP) displays display data such as preoperative images, live video feeds, previously captured video snapshots, GIFs, and other data displayed via the HMD 102. Enables display in a window overlaid on the current background image. Surgeon 120 may predefine several different PIPs, their locations, and what content to display within each PIP. For example, PIP displays preoperative OCT images when working in posterior segment non-contact lens mode, or when in endoscopic mode and the background image is the live video feed from the endoscopic camera. Enlarged images may be displayed. Surgeon 120 may turn each default PIP on and off. Surgeon 120 may control the position and size and on/off status of the PIP via UI 160. The position of the PIP in the field of view of the HMD 102 may be dynamically changed according to the position of the automatically detected region of interest within the main image, and may be set by the surgeon 120 via the UI 160. The position of the PIP may also be locked to the real world (a "world-locked PIP") or fixed to the patient's anatomy, for example in a VGS procedure. When in PIP mode (ie, after "selecting" a PIP), activating a snapshot may only save the PIP image. This may be useful for storing iOCT images, for example.

When a PIP is being displayed, an overlayed menu will include a PIP-specific functions may be added. In some embodiments, while performing an augmented reality (VGS) procedure, the driver 102A of the HMD 102 closes the shutter 132 within the area of the display module 130 where the PIP is displayed. This allows the surgeon 120 to view the real world through a partially transparent screen while viewing preoperative, intraoperative, or other data displayed within one or more PIPs while blocking ambient light behind the PIP. It is possible to see. When the PIP is locked to the patient's anatomy, the computer 118 determines the area where the shutter is closed so that the background is always occluded behind the PIP while the surgeon 120 moves his head. May be controlled dynamically.

In some implementations, switching to PIP system mode is only possible when at least one PIP is displayed via the HMD. Switching to PIP system mode may be performed, for example, by pressing a footswitch button that brings up a menu and selecting PIP as if it were a menu item. This changes the UI functionality to the default UI functionality according to the content displayed within the PIP. For example, if the PIP displays preoperative data, the UI is carried over from the preoperative data mode (e.g., allowing the surgeon 120 to scroll through the set of preoperative images displayed within the PIP) and the PIP When displaying live magnified images, the UI can be configured to display normal mode (e.g., allowing the surgeon 120 to control magnification) or posterior segment non-contact lens mode, or generally (some other not described here). (as such modes may exist) will be inherited from the last mode selected for viewing the raw image feed. Special UI features may be added to the UI carried over in PIP mode, and others may be tweaked for PIP mode. For example, when a pre-operative image is displayed in the PIP, the surgeon 120 may use head gestures to control a virtual pointer to specify corresponding points in the displayed pre-operative image and the live magnified image (this , which may be performed to manually align the preoperative data to the raw image).

・iOCT (intraoperative OCT) mode. This system mode allows the iOCT scanner 142 to acquire OCT data. Settings for this mode may include the following:
○iOCT on.
o PIP On, images acquired via the iOCT scanner 142 are displayed in the PIP.
o UI Options This feature allows the surgeon 120 to control operating parameters for the iOCT scanner 142, such as scan position and angle, using head gestures or by navigating menus, etc. I will provide a. For example, a user may use a menu to switch between iOCT scanning modalities: B-scan and volumetric scan.
o B-scan indicator overlay This feature overlays a symbol indicating the location of the live iOCT B-scan on the live magnified image.

Other treatment-specific and stage-specific modes: Generally, various treatment-specific and stage-specific modes may be predefined for displaying treatment- and stage-specific data and overlays.
o Data overlay, for example data from a phacovitrectomy machine is enabled.
o Aligned overlays are enabled. This view displays image registration symbols and/or images and/or 3D models overlaid on a real-world enlarged view of the surgical field to assist the surgeon 120 during the surgical procedure.
o UI UI functionality may be varied to allow the user to control the position, depth, and orientation of the overlay or to control other devices connected to the system. For example, the angle of the line overlaying the display may be manipulated using head gestures and/or footswitch 104.
○PIP (Picture in Picture). The PIP may be enabled to show relevant pre-operative or intra-operative video and/or data via the PIP.

・External video feed mode:
o In some modes, the HMD 102 is mounted outside the operating room, for example, to allow the surgeon 102 to monitor patients waiting in a waiting room or to monitor the progress of different surgical procedures within different operating rooms. Displaying an external video feed such as may be provided by one or more cameras located. These videos may be displayed within the PIP or on the side screen of the HMD 102. They can be displayed one video at a time or as a collection of videos on multiple screens. This mode allows the surgeon to view all available video feeds on multiple screens. Surgeons often watch video feeds from the waiting room, other operating rooms, or search databases of instructional videos that explain how to perform rare procedures, such as those that can occur in unpredictable circumstances. It's fine. The surgeon may predetermine when and how to view selected videos, such as within a PIP, and when and how to return to a previous mode.

・Teaching mode:
In some embodiments, system 100 may be guided simultaneously by an assistant surgeon (not shown), such as a senior surgeon, working with surgeon 120. The assistant surgeon wears a second HMD that is coupled to computer 118 and camera head 110 and operates similarly to HMD 102. The second HMD also displays a view of the surgical field 124, and the assistant surgeon may control a cursor on a display that is synchronized with the display in module 130 of HMD 102, and the surgeon 120 may control a cursor on a display that is synchronized with the display in module 130 of HMD 102. allows you to see. In this way, the assistant surgeon can guide the surgeon 120 during the surgical procedure. Similarly, the assistant surgeon may invoke one or more PIPs that indicate preoperative or other data, etc. to the surgeon 120.

In this mode, a senior surgeon may supervise surgeon 120 performing a surgical procedure, such as when surgeon 120 is a resident. In some embodiments, eye tracking component 136 (FIG. 1C) tracks the gaze of surgeon 120 and determines the display area of HMD 102 that surgeon 120 is fixating. The system 100 displays a symbol indicating the area that the surgeon 120 is gazing at to the assistant surgeon via the second HMD. In this way, the assistant surgeon can monitor the focus of the surgeon 120.

In some embodiments, the senior surgeon may overlay features and symbols to guide the surgeon 120 while performing the procedure. The surgeon 120 sees the end effects overlaid on the raw image and does not see the menus and drawing options that the supervising surgeon sees in this mode. Features in teach mode include:
o As described in more detail below in Figures 6K-6O, a "Paint"-like menu for selecting drawing options, virtual tools/markers/pointers to display, marking color, erasing markings, etc. Show to the Chief Surgeon.
o Enabling senior surgeons to control the pointer or draw using head gestures or with a dedicated wand.
o When drawing on real-time images, line drawings (lines, symbols, etc.) may be locked to the patient's anatomy, i.e., overlay symbols (e.g., as the surgeon moves the patient's eyes) move the patient's anatomy. Automatically/dynamically change its position so that it appears locked to a scientific structure. The line drawing may be 3D.
o Senior surgeons may use the touch screen for instructions and drawings if they are not wearing an HMD. The senior surgeon may guide the operating surgeon remotely, for example in his or her office or home, using a regular screen and mouse, and possibly using a tablet or smartphone to direct and draw on real-time images. .
o A symbol may be displayed in the corner of the displayed image to indicate to the operating surgeon that someone has connected to the system 100 system 100 in "draw" mode and is enabled to draw on the live image feed.
○ The operating surgeon will be provided with the option to delete. When features are drawn on the display, a menu item is added to the current menu (i.e., when the operating surgeon invokes the menu, the menu is displayed to him), and the operating surgeon clears one or more of the drawn features. enable. The menu may allow all drawing features to be cleared by selecting a single "clear all" menu item. Alternatively, the menu presents a submenu that allows the surgeon 120 to select which drawing features to erase and how many.
o A freeze option may be provided to freeze the raw image feed to facilitate drawing of features.
○3D symbol/marker:
- 3D symbols are symbols that are displayed to both eyes to appear at a given depth relative to the anatomy. A 3D marker generates a 3D image fixed symbol (eg, a 3D straight line).
- The user may select a 3D pointer or marker that is fully controlled by the user, i.e. the user controls both its XY position and depth (the user can, for example, depth may be controlled manually using head gestures).
- Alternatively, the user may select a 3D pointer or marker that automatically locks to the depth of the currently focused anatomical structure, so that as the user moves the symbol, the user can Get the sensation of following a surface (e.g. moving up and down).
- If the latter is used, the height is automatically calculated and updated when focus is changed by the user. For example, if the first focus point is on a transparent surface, such as the cornea, and the second focus point is behind (below) the first focus point, such as a point on the anterior capsule; The symbol positions in the two images (displayed to the HMD user's eyes) change in different ways to reflect the new depth.
○ Wand:
- The wand is a controller, such as a mouse or a remote control device, optionally having buttons of various types, handheld, wireless or wired, optionally within a sterile cover. with optionally integrated inertial tracking. This can be used both for pointing/drawing as described above and for controlling virtual tools.
- The supervising senior surgeon may use a handheld controller to move the virtual tools in a natural way to show the trainee how to perform a surgical procedure. The virtual tool appears in the display as a 3D tool that the user views as an overlay on the live image via the HMD. This is based on a 3D model of the tools available for selection by the user via the menu/GUI in this mode.

Surgeon 120 selects and switches system modes for system 100 using one or more hands-free gestures detected via UI input 160A. In some embodiments, the surgeon 120 switches system modes by making head gestures detected by the head tracker 162 while simultaneously making foot movements detected by the footswitch 104. In other embodiments, the surgeon 120 switches the system through voice commands detected by the microphone 138. Computer 119 determines how to interpret subsequent user input according to the system mode.

For example, in one system mode, upward rotation of the head performed by surgeon 120 while depressing pedal 104A of footswitch 104 increases the intensity of illumination by illumination system 114. In different system modes, the same head gesture performed while pressing pedal 104A of footswitch 104 scrolls the image displayed via HMD 102. A lateral head gesture performed by the surgeon 120 while depressing the pedal 104B of the footswitch 104 may switch between these two system modes. By allowing multiple system responses to the same gesture, system mode provides a broader range of functionality for system 100, allowing surgeon 120 to control features of system 100 without using his or her hands and improve surgical procedures. It allows you to keep both hands free for doing things.

Reference is now made to FIG. 2B, which illustrates an exemplary control flow diagram for the user interface of FIG. 2A, constructed and operative in accordance with further embodiments of the disclosed technology. Computer 118 (FIG. 1B) receives user input via UI input 160A, such as, for example, a head gesture from head tracker 162 with a simultaneous indication of a foot press of a pedal on footswitch 104 (FIG. 1G); Determines the system response depending on the system mode, ie, the response is mode-specific. In one system mode, user input may result in a first action in system 100, and in a different system mode, the same user input may result in different actions in system 100. The flow diagram of FIG. 2B is intended to illustrate the flexibility and range of functionality that may be controlled via user interface 160 in conjunction with multiple system modes. This diagram is intended to be exemplary only and is not intended to limit the invention to any particular implementation or example disclosed herein.

Computer 118 receives user input from head tracker 162 and footswitch 104 via UI input 160A. Case 1. The user input is an indication that the surgeon 120 has performed a heads-up gesture while depressing the pedal 104A of the footswitch 104. Since the current system mode is normal, computer 118 performs a zoom out on the image displayed on HMD 102, shown as action 240. Case 2. The user input is the same as described above, eg, an indication that the surgeon 120 performed a heads-up gesture while depressing the pedal 104A of the footswitch 104. However, because the current system mode is set to preoperative OCT ("preoperative" mode), computer 118 performs scrolling between preoperative OCT images, shown as action 242. Case 3. The user input is an indication that the surgeon 120 has performed a heads-up gesture while depressing the pedal 104B of the footswitch 104. Since the current system mode is normal, computer 118 switches the system mode from normal to vital, as shown as action 244. Mode switching is discussed in more detail below with respect to FIG. 3A.

By responding to multiple user inputs obtained from multiple user interfaces as described herein, system 100 provides a wide range of operational features that allow surgeon 120 to keep his or her hands free to perform surgical procedures. allows you to control. Inputs may cause one system response in one system mode and different system responses in different system modes, providing a wide range of functionality with a relatively small set of inputs, eg, head gestures, foot presses, etc. Footswitch 104 operates to control the response of system 100 only to intentional gestures by surgeon 120. An inadvertent gesture by the surgeon 120 will not result in a response by the system 100 unless the footswitch 104 is pressed simultaneously.

According to one embodiment, surgeon 120 may select the system mode by controlling the position and orientation of HMD 102. The range of possible positions and orientations for HMD 102 is divided into zones, each zone corresponding to a different system mode. By moving his head to align the position and orientation of HMD 102 with a zone, surgeon 120 may select the system mode that corresponds to that zone. System 100 may require surgeon 120 to press a footswitch while manipulating his or her head to prevent accidental switching to a new mode.

3A depicting an exemplary layout 300 for various positions and orientations for a multi-zoned HMD 102 (FIG. 1A) configured and operable in accordance with other embodiments of the disclosed technology. Referenced. Layout 300 shows multiple zones, each zone being multiple orientation areas for HMD 102, each area corresponding to a different system mode. System 100 allows surgeon 120 to switch from one system mode to another by rotating HMD 102 by rotating his or her head. The surgeon 120 may customize the functionality of the HMD 102 by defining multiple angular and positional zones, each of which is a range of his or her head motion, and determine which system mode will be implemented for each of each zone. .

Layout 300 shows zones 302, 304, 306, 308, and 310, respectively, for HMD 102 of FIG. 1A. Dashed lines 312, 314, 316, and 318 define the respective zones by indicating angular thresholds between zones 302, 304, 306, 308, and 310. Straight line 312 defines the threshold between vital zone 308 and normal zone 302, straight line 314 defines the threshold between normal zone 302 and surgical field zone 306, and straight line 314 defines the threshold between normal zone 302 and preoperative zone. 306 , and straight line 318 defines a threshold between normal zone 302 and nurse zone 304 . By depressing the footswitch 104 pedal while crossing one of the thresholds 312, 314, 316, and 318, the system mode changes the orientation of the HMD 102 to match the new zone. Switched to correspond to .

Zone 302 labeled "Normal" is associated with a forward horizontal orientation of HMD 102 having a normal system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 302, the normal system mode is initiated. Shutter 132 is closed and a live video feed of images captured by camera head 110 is rendered via HMD 102.

The zone 304 labeled "Nurse" is associated with the right horizontal orientation of the HMD 102 having the nurse system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 304, the nurse system mode is initiated. The shutter 132 opens and the video feed is turned off, allowing the surgeon 120 to directly see and communicate with the nurse while wearing the HMD 102.

Zone 306 labeled "Preoperative" is associated with a left horizontal orientation of the surgeon's 120 head having a preoperative data system mode. When the surgeon 120 presses the footswitch 104 while moving his head to align the HMD 102 with the zone 306, the preoperative data system mode is initiated. The shutter 132 is maintained in a closed state, and preoperative data is displayed via the HMD 102. In some embodiments, UI functionality may change when switching to preoperative system mode. For example, menu items may be different and head gestures may cause different actions to occur in system 100 than in other system modes to allow the surgeon to view and scroll through pre-operative data.

Zone 308 labeled "Vitals" is associated with a forward and upward orientation of HMD 102 having a vital signs system mode. When the surgeon 120 presses the footswitch 104 while moving his or her head to align the HMD 102 with the zone 308, the vital system mode is initiated and data indicative of one or more vital signs of the patient 122 is transmitted via the HMD 102. will be displayed.

The zone 310 labeled "operative field" is associated with the forward and downward orientation of the HMD 102 having the surgical field system mode. When the surgeon 120 presses the footswitch 104 while moving his head to align the HMD 102 with the zone 310, the shutter 132 opens, the display is turned off, and the surgeon 120 sees a real world view of the surgical field 124. Is possible.

In some embodiments, the system mode changes the HMD 102's current Determined according to P&O. Surgeon 120 switches the system mode of system 100 by moving his head across one or more of angle thresholds 312, 314, 316, and 318 while holding down footswitch 104. When the surgeon's 120 head orientation exceeds the predetermined angle and the HMD 102 aligns with the new zone, the system mode switches to the system mode corresponding to the new zone. For example, to remain in pre-operative mode, the surgeon 120 holds down the footswitch 104 while maintaining his line of sight (to the left) above the angle threshold 316. When the foot switch 104 is first pressed, if the current HMD P&O exceeds the threshold of the current mode, the system mode is switched instantly. In this embodiment, when the surgeon releases footswitch 104, the system mode switches back to the original mode. This mode of operation is useful, for example, if the user likes to quickly glance at pre-operative data or nurses. In some embodiments, when the surgeon releases footswitch 104, the system mode remains in the current mode, allowing the user to move the head freely again without affecting the system mode. This mode of operation is useful, for example, if the user prefers to comfortably review preoperative data over an extended period of time.

Angle thresholds 312, 314, 316, and 318 may further enable hysteresis-enabled control. For example, when crossing the threshold from normal zone 302 to nurse zone 304, a larger threshold may be in effect, and when crossing again from nurse zone 304 to normal zone 302, a smaller angular threshold may be in effect. . This is useful to avoid unwanted mode switching when the head orientation is close to the angle threshold. The hysteresis state (enabled/disabled) and hysteresis size may be configured by the surgeon 120. System 100 may automatically generate two hysteresis-generating thresholds from a single threshold.

In other embodiments, zones 302, 304, 306, 308, and 310 are not defined by precise thresholds. Rather, activation of modes associated with zones 302, 304, 306, 308, and 310 is based on relative head gestures by surgeon 120. For example, by depressing pedal 104C of foot switch 104 while turning one's head more than 10 degrees in either direction, one can rotate in the direction of relative rotation required to switch from the current system mode to another system mode. The system mode switches accordingly.

In further embodiments, input from footswitch 104 is not required to switch modes. Angle thresholds 312, 314, 316, and 318 are used as triggers to change system modes. The system mode corresponding to each of the areas is determined according to the current P&O of the HMD 102. For example, to remain in preoperative mode, the surgeon 120 must maintain his line of sight (to the left) above the angle threshold 416.

In yet other embodiments, "touching" the angle threshold by slightly turning the head switches the system mode corresponding to the respective zone. When the surgeon 120 "touches" the angle threshold 316 while depressing the footswitch 104, the system switches from the normal mode to the preoperative mode corresponding to the preoperative region 306. During this mode, the surgeon 120 comfortably looks forward and views preoperative data without distorting his or her neck for long periods of time. By touching the threshold value 316 a second time while pressing the foot switch 104, the system mode switches again to "Normal" corresponding to the normal region 302, and the surgeon views the live enlarged image of the surgical field 124 via the HMD 102. make it possible. The footswitch 104 is configured to enable switching between system modes such that only pressing a particular pedal of the footswitch 104 and simultaneously touching the threshold by a head gesture results in switching to a new system mode. used. This is to avoid accidental mode switching.

Here, hints are provided indicating to the user which head gestures are required to invoke various system modes associated with the zones described with respect to FIG. 3A, configured and operable in accordance with embodiments of the disclosed technology. Reference is made to FIG. 3B, which shows a display of an overlaid raw image 320. The image 320 displayed on the field of view of the HMD 102 corresponds to the normal mode associated with the zone 302. In some embodiments, the surgeon 120 elects to display features 324, 326, 328, and 330 regarding the system 100 that indicate to the surgeon which head gestures invoke which system modes. Features 324, 326, 328, and 330 are displayed transparently or translucently around the periphery of the field of view so as not to obscure the display of image 320. Feature 324 with the symbol "Nurse" suggests to the surgeon that by turning his head to the right in the direction of feature 324, the nurse mode corresponding to zone 304 of FIG. 3A is invoked. Feature 326 with the symbol "pre-operative" suggests to the surgeon that by turning the head to the left in the direction of feature 326, the pre-operative mode corresponding to zone 306 of FIG. 3A is invoked. Feature 328 with the symbol "Vital" suggests to the surgeon that by turning the head upwards towards feature 328, the vitals mode corresponding to zone 308 of FIG. 3A is invoked. The feature 330 with the symbol "field" indicates to the surgeon that by pointing the head down in the direction of the feature 330, the field mode corresponding to zone 310 of FIG. 3A is invoked. This feature is useful if the surgeon 120 is still new to the system and can be turned on or off as needed. Similarly, the surgeon 120 may freely choose to enable mode switching via head gestures only, without the footswitch 104, or via the footswitch 104, depending on the situation and need. You may return.

In one embodiment, features 324, 326, 328, and 330 are displayed only when surgeon 120 presses footswitch 104 to enable head gesture-based mode switching. Optionally, the display of features 324, 326, 328, and 330 may be default settings. It should be noted that the display of specific features 324, 326, 328, and 330 is merely exemplary. Generally, features are displayed in a manner that corresponds to the current mode. Therefore, different system modes cause different features to be displayed. For example, when system 100 is in pre-operative mode, only one feature is displayed superimposed on the image in the field of view, instructing surgeon 120 to turn the head to the right to return to normal mode. In such a case, feature 324, which currently indicates "nurse," would indicate normal and features 326, 328, and 330 would not be displayed.

The surgeon 120 may define and configure system modes by scratching or copying an existing mode and modifying it using the touch screen 108. Once at least two system modes have been defined, the surgeon 120 wears the HMD 102 and gazes in the general direction in which he or she wishes a mode switch to occur next, and selects the mode associated with each of the zones defined by the thresholds. A system mode may be associated with an orientation region by selecting . Once the mode is additionally defined by location within the operating room, the surgeon 120 may physically move to the desired location to cause the system mode switch. This is done in a dedicated system mode with special menus and gestures that support various system settings and configurations. This is optionally done by each new user as an initial setup or configuration step. The user may invoke and manipulate the mode designation alone via the HMD menu or with the aid of a second person operating the GUI on the touch screen.

The default display state is a full screen state. In this state, HMD 102 displays a full screen display fixed image that appears fixed to surgeon 120. Content displayed via the HMD 102 appears as a screen firmly suspended from the HMD 102. As the surgeon 120 moves his head, the screen moves with him and appears fixed relative to him. In addition to full screen states, system 100 provides virtual screen states. Surgeon 120 may invoke the virtual screen state from the full screen state. The virtual screen state may be invoked using any combination of the user interface methods described above, such as by pressing pedal 104B of footswitch 104, using head gestures or voice commands. Virtual screen state changes only the screen state, leaving all other system characteristics of the current mode unchanged (eg, displaying the live image feed).

According to further embodiments of the disclosed technology, switching between system modes is achieved by virtual screen states. In the virtual screen state, content is displayed in one or more world-fixed virtual screens that appear fixed in the operating room. As the surgeon 120 moves his head, the virtual screen moves in and out of his field of view, allowing the surgeon 120 to scroll through the content displayed thereon. Depending on the distance between the surgeon 120 and the virtual location of the virtual screen, the surgeon 120 may view multiple virtual screens at one time. This is discussed in more detail below with respect to FIG. 5C.

To switch between system modes using virtual screen states, each system mode has a virtual screen that displays the content that the surgeon 120 would see in full screen if that system mode is selected, and that acts as a type of thumbnail. It can be shown by This allows the surgeon 120 to simultaneously preview content available in various system modes without switching modes. The details and resolution of the content displayed on the virtual screen may differ from that displayed in the corresponding full screen state due to screen size, technical issues, and limitations on bandwidth. For example, an overlay symbol that appears in a full screen state may not appear in a thumbnail image displayed in a virtual screen state. However, the images are substantially similar in both virtual screen and full screen conditions. The surgeon 120 may switch system modes in the virtual screen state using any suitable UI input, such as, for example, staring at the virtual screen for a predetermined period of time or a head gesture with a press of the footswitch 104. Not all system modes are necessarily represented by a virtual screen, such as a system mode with the display turned off.

When the system mode is selected, the screen state automatically switches to full screen state in the selected mode. Surgeon 120 may configure system 100 to remain in the virtual screen state after switching modes. In this case, all system characteristics except the screen state change. In one embodiment, the virtual screen state is itself a system mode that allows the surgeon 120 to view several virtual screens simultaneously. In this case, all other system properties, including user interface properties, are uniquely defined for this system mode.

In some embodiments, system modes may be switched via touch screen 108. In some applications, it is desirable to allow a third party, such as a nurse, to switch to different system modes. In this case, the nurse is provided with a touch screen coupled to the computer 118. The touch screen 108 may be coated with sterile nylon for use within the operating room by any of the attending sterile surgeons.

In some embodiments, system modes may be switched via voice commands or other acoustic commands, such as a tongue click. Microphone 138 records one or more acoustic commands issued by surgeon 120. Computer 118 analyzes the acoustic commands to determine various system functions, such as switching system modes. To avoid false activation, system 100 may be configured to respond only to acoustic commands that begin with an identifying tone, such as by speaking the system's 100 identifying name. For example, surgeon 120 may switch to normal mode by reading the identification name of system 100 and commanding system 100 to switch to normal mode.

According to other embodiments of the disclosed technology, a menu is displayed via the HMD 102. The menu may be invoked by pressing a pedal of footswitch 104, such as pedal 104B, or by other UI means, such as voice control or dedicated head gestures. Menu invocation causes a menu to overlay the currently displayed content via HMD 102 and allows items within the overlay menu to be navigated and selected via head gestures. The menu may be displayed, such as as a translucent display fixed overlay onto the current image displayed via the HMD 102, or within a separate virtual screen.

For example, one position and orientation may correspond to a menu display, according to the concept shown in FIG. 3A, where different positions and orientations of the HMD 102 correspond to different functions. When the surgeon wearing the HMD 102 looks in a predetermined direction, for example, downward, a menu is displayed. In some embodiments, the menu may initially be displayed in a compact or collapsed format so that it occupies less display area. In other embodiments, the menu is displayed in a direction away from the front of the HMD 102 (in a folded or normal state), for example, perpendicular to the front direction of the HMD 102. The menu may fully open after the surgeon continues to look in a predetermined direction for a predetermined period of time. In some embodiments, the menu is not invoked if the surgeon turns or orients his or her head as part of a head gesture configured to activate other functions. The surgeon 120 navigates and controls menus displayed as HMD fixed overlays via head gestures. In some embodiments, pressing the footswitch 104 brings up a menu and allows the user to navigate and highlight various menu items using head gestures, and releasing the footswitch brings up the menu items. Activate. For example, to select a menu item, surgeon 120 depresses an available pedal on footswitch 104 and manipulates his head to select the menu item. Release of footswitch 104 activates the action corresponding to the selected menu item.

The menu may list any number of parameters, such as, but not limited to, various system modes available for system 100. Each system mode is represented by a symbol or a menu item that includes text such as a textual representation of the system mode, eg, "Normal," "Nurse," "Pre-op," etc. Other menu options that are added to the system mode may be included in the menu. Additional menu options may be mode-specific regarding controllable characteristics in that system mode. For example, in normal mode, the menu may include items that allow turning floodlights on and off and red-eye lighting on and off. In preoperative mode, these options do not appear; instead, the preoperative menu enables the opening of folders containing available preoperative datasets, and a subsection for selecting which datasets to display. You can display the menu. The preoperative menu may further include items for controlling various attributes for displaying preoperative data. Menu options may be mode-specific, procedure-specific, and stage-specific, ie, the menu appearance may change depending on the system mode and stage during the procedure. In addition to switching between system modes, menus allow switching between states.

In some embodiments, the menu is displayed in the margin of the HMD 102 display so as not to block the live image displayed in the center of the display area provided by the HMD 102. As the surgeon 120 navigates through each menu item in the menu displayed in the margin, the menu item is displayed to indicate to the surgeon 120 where he is on the menu and what the effect of selecting that menu item is. The functionality may be enabled temporarily. For example, if the surgeon 120 navigates over a menu item that emphasizes the color green to allow viewing of membranes that have been jetted with green dye, that feature will be temporarily enabled and the surgeon 120 will be able to view the raw The surgeon 120 may be shown which menu item he is currently navigating to and what the effect of selecting that menu item is, while allowing him to maintain his gaze on the menu item. However, some menu items may be associated with functionality that is not immediately apparent while staring at the live image displayed via the HMD 102. A non-visual indicator, such as a sound or vibration, may be triggered when a new menu item is highlighted to allow the surgeon 120 to keep his gaze on the live image while tracking his navigation of the menu. . This allows the surgeon 120 to track his or her navigation of the menu without looking directly at it, so the surgeon 120 can remain focused on the live image.

In some embodiments, the menu in the right margin of the display is application-specific. Examples of such applications include preoperative OCT applications, teaching applications, preplanning applications, phacovitrectomy settings and metrics applications, intraoperative OCT applications, and the like. For example, when you press a footswitch menu button, the menu in the left margin of the display is the main menu, the menu in the right margin of the display is dedicated to the application, and you can turn on the application by activating the corresponding menu item. enable. Switching between the two branches of the menu (ie the menus on the left and right margins of the display) is done by turning the head from side to side.

In other embodiments, when an application is turned on and the footswitch menu button is then pressed, the menu that appears in the right margin is a dedicated menu for the activated application. For example, when a preoperative OCT application is activated, a PIP with a B-scan appears in the display and a straight line indicating the location of the B-scan is overlaid on the raw image. When the footswitch menu button is pressed, the menu that appears in the right margin is a dedicated menu for preoperative OCT applications rather than the general application menu in the right margin of the display. The surgeon 120 can view the preoperative OCT, for example, to turn on and off a linear overlay indicating the B-scan position on the raw image, to switch to other screens where different OCT scan sets can be selected for display within the PIP, etc. Menu items may be navigated and selected from dedicated menus to turn off applications, etc. When the preoperative OCT application is closed and the footswitch menu button is next pressed, a general application menu will appear in the right margin of the display. Note that these examples are provided merely to illustrate the range of features of system 100 that may be controlled via UI 160.

In a further embodiment, the same footswitch button used to invoke and operate the menu may also enable screen switching by simultaneously pressing the footswitch button and rotating the head. For example, turning the head slightly to the right while holding down the footswitch menu button will highlight the first menu item in the menu branch that appears in the right margin of the display. Turning the head further to the right may cause the screen to switch.

The following examples are given for treatment-specific and stage-specific menu options. During cataract treatment, an external machine (not shown) may be connected to system 100. The external machine introduces data for display as an overlay on the rendered live enlarged video on display module 130. When the system 100 identifies that an external machine is connected, the display module 130 displays menu options for toggling between "enabled" and "disabled" states of "external data overlay" and "rupture guidance." When the system 100 identifies that a treatment step, such as a phacoemulsification step, is complete based on the captured data, the display module 130 displays an "enabled" flag for "IOL integrity guidance" instead of "rupture guidance." ” and “Disabled.”

The menus may be used to control features and aspects of system 100, including selecting and switching to new system modes and returning to previous system modes. Accessing system modes according to P&O zones only may limit the number of available system modes to the number of zones in view, so the menu allows to increase the number of available system modes. obtain. Thus, one system mode may be a menu mode presenting menu items for selecting other system modes, as well as items for controlling additional system features and parameters. While in menu mode, a menu for system mode may be UI function mode specific and displayed via HMD 102, i.e. certain gestures and movements by surgeon 120 result in one function while in menu mode; Provides different functionality in different system modes. Menu mode may be configured to perform any suitable UI interface, such as by performing one or more of the following: pressing pedal 104C on footswitch 104, issuing a voice command recorded by microphone 138, performing a head gesture, making eye movements, etc. can be called using

Reference is now made generally to FIGS. 4A-4C, which illustrate exemplary system mode menus overlaid on a display of an enlarged image 414, constructed and operable in accordance with further embodiments of the disclosed technology. The precise implementation details described below are intended as exemplary and not as limitations of the invention. System mode menu 400 allows surgeon 120 to switch system modes while viewing live image stream 414. The system mode menu 400 appears similar to the zone layout of FIG. 3A, although this is not a requirement. Surgeon 120 views live magnified image stream 414 of surgical field 124 via HMD 102 (FIG. 1A) configured in full screen mode. Surgeon 120 performs a predetermined foot gesture, such as, for example, depressing pedal 104B of footswitch 104. In response, computer 118 displays system mode menu 400 overlaid on enlarged image stream 414. The menu items of system mode menu 400 are transparently overlaid on enlarged image stream 414, for example, to avoid obscuring the displayed image.

In the exemplary implementation shown, the system mode menu 400 includes menu items 402 (Normal), 404 (Vitals), 406 (Nurse), 408 (Surgical Field), 410 (Preoperative), and 412 (Cancel). including. Since the current system mode is normal, element 402 is highlighted (FIG. 4A). In some embodiments, menu items are spatially arranged according to their function. For example, in an operating room where a nurse is on the right side of the surgical field 124, menu item 406 (nurse mode) is displayed on the right side of the field of view of HMD 102, and menu item 402 (normal mode) is displayed on the center of the field of view. Is displayed. To navigate system mode menu 400, surgeon 120 rotates his or her head while depressing pedal 104B of footswitch 104.

Referring to FIG. 4B, surgeon 120 turns his head to the left to intersect the angle threshold separating menu items 402 and 410. The emphasis switches from menu item 402 to menu item 410 (pre-operative mode). In one embodiment, only the emphasis is switched to preoperative mode menu item 410. The system mode has not been switched yet. When the surgeon 120 lifts his foot off the footswitch 104 to release the pedal 104B, the system mode switches to the mode corresponding to the selected menu item 410, such as preoperative mode (FIG. 4B). In other embodiments, when a highlighted menu item is toggled, the background image also changes, as if the system mode were toggled, allowing the user to see a preview of the image associated with that menu item.

Referring to FIG. 4C, the surgeon 120 holds down the pedal 104B and turns his head toward the lower right corner. When the angle of his head intersects the angle threshold set for menu item 412 corresponding to the cancel selection, the emphasis switches to menu item 412 (Cancel). Surgeon 120 selects menu item 412 by releasing footswitch pedal 104B to hide menu 400 without changing system modes, and cancels the action that brought about menu 400 by pressing pedal 104B. Thus, surgeon 120 cancels system mode menu 400 and system 100 remains in the current system mode.

In this embodiment, the screen state is display fixed. As the surgeon moves his head, image 414 remains fixed within his field of view, regardless of his head movement. However, the emphasis on the selected menu item of menu 400 overlaid on fixed image 414 shifts with his head movement.

A selected menu item can be marked, for example, by displaying the menu item in a different font, background color, displaying a cursor pointing to the menu item, or distinguishing the menu item from other unselected menu items. may be enhanced in a variety of ways by any other suitable technique. The emphasis moves from one menu item to the next as HMD 102 rotates toward the second menu item while footswitch 104 is pressed. This is shown in Figure 4B. When the surgeon turns his or her head to the left, the menu item 410 corresponding to preoperative mode is selected and highlighted, and the menu item 410 corresponding to normal is deselected and displayed normally. . Surgeon 120 may personalize system 100's response to his or her head gestures. For example, head motion may be scaled such that a small head tilt translates into a correspondingly large angular shift, or vice versa. Alternatively, the orientation of HMD 102 controls a cursor (not shown) displayed on menu 400 that selects menu items. Manipulating the orientation of the HMD 102 manipulates the cursor; for example, head movements up and down and left and right move the cursor up and down and left and right accordingly. Releasing footswitch 104 activates the system mode corresponding to the currently selected menu item, thereby toggling the system mode.

In other embodiments, eye tracking may be implemented to control menu 400. Eye movements play the role of the head movements described above. Staring at a menu item may highlight the menu item, and staring continuously for longer than a predetermined period of time may activate the menu item.

Additional menus may be displayed via HMD 102 to select functions other than system mode. Some menu items may be expanded to display one or more submenus when selected. The selection method may include pressing or releasing one of the pedals of footswitch 104, keeping a menu item highlighted for longer than a predetermined period of time, using a voice command detected by microphone 138, or other methods. This can be implemented by any suitable UI method. When expanded into a submenu, the submenu may replace or be displayed with the original menu. The menu may also include standard menu operations, such as back/return and exit/cancel.

Some menu items may be turned on and off via the UI 160. A toggle item is a menu option that toggles between two states, with the currently displayed menu option representing the currently inactive state. For example, if shutter 132 is closed, the menu option for controlling the shutter appears as "Open Shutter" and vice versa. Examples of switching items in the menu are described below.
○ Basic HMD system operations controllable via menu -Display (on/off). This shuts the display on or off. When the display is turned off, no images are displayed, but a menu can be invoked (ie, displayed) to allow the user to turn the display back on. Controlling the display state is useful in VGS treatments where the HMD is transparent (ie, if the HMD is equipped with a shutter, the shutter is open at this stage). In these procedures, the user may desire to turn off the display during treatment steps that do not require HMD guidance so that the images do not interfere with viewing the surgical field.
-Full screen/virtual screen. In the full screen state, only one screen fills the FOV, but in the virtual screen state, multiple virtual screens are displayed. Screen states are discussed further below with respect to FIGS. 5A-5C.
-Shutter (open/close). This controls the state of shutter 132. The shutter 132 may be partially opened or closed, such as to display features overlaid on the real world view when a large portion of the screen is transparent.
- Red eye lighting (on/off).
- Recording (on/off). This controls a feature of system 100 that allows surgeon 120 to record portions of the surgical procedure that can be recalled at a later stage.
○Advanced HMD operation (enable/disable), one or more of the following.
・Automatic focus,
・Automatic centering,
・Automatic zooming.
○HMD display parameters (on/off).
・PIP on/off switching for each default PIP displayed,
- External data overlay for displaying data acquired externally via the HMD 102.
- Various procedure-specific and stage-specific overlay symbols, images, and models.
-Specified pointer/cursor.
○iOCT (on/off). This controls the OCT images.

In some embodiments, in addition to mode-specific menus, UI 160 displays dedicated specific menus for controlling various functions of system 100. For example, UI 160 displays different dedicated menus to control each of zoom, focus, lighting, and other functions. Each dedicated menu allows the surgeon 120 to access options related only to a particular function. For example, a menu dedicated to the focus function allows the surgeon 120 to select various autofocus settings.

In some embodiments, the surgeon 120 accesses menus dedicated to specific functions via the main menu. In other embodiments, the surgeon 120 directly accesses a menu dedicated to specific functions, such as by pressing a button on the footswitch 104 twice to activate the menu. For example, the surgeon 120 may open a lighting function-specific menu by performing two short presses followed by a long press on a button on the footswitch 104 that activates a head gesture to control the lighting. The surgeon 120 may use head gestures or eye tracking to navigate a dedicated menu as described herein and activate the highlighted menu option by releasing a button on the footswitch 104. .

In some embodiments, the surgeon 120 may cause the footswitch 104 by only partially depressing the associated button (e.g., halfway down) or by issuing a voice command while depressing the footswitch 104; or A dedicated menu is opened by performing a head gesture such as a nod while pressing the foot switch 104. For example, a "yes" head gesture may control an enabled function (eg, focus, zoom, lighting, etc.), and a "no" head gesture may open a dedicated menu.

For example, if the surgeon 120 turns his or her head from side to side while pressing the footswitch button to activate a focus feature, a dedicated menu for controlling the focus feature, such as an autofocus feature, may appear on the display. Appears in Via this dedicated menu, the surgeon 120 may select whether autofocus acts on a specified point, surgical tool, etc., and whether autofocusing is one-time or continuous.

In some embodiments, upon activating the action via the footswitch 104, the surgeon (by moving his or her head up and down) " Performing a "Yes" motion but facing left or right activates or toggles two separate actions that are associated with the activated action. Alternatively, a head gesture to the left may toggle an individual action, and a head gesture to the right may open a dedicated menu (or vice versa).

For example, when activating "lighting", the up and down head gestures change the lighting intensity, and the "turn left" head gesture switches the lighting between two discrete levels. For example, one level may be "high" and the other "low." Optionally, these levels can be predetermined and configured by the user. Optionally, after the surgeon adjusts the illumination level during the surgery, the system 100 remembers the illumination level and the next time the illumination action is switched (from high to low or vice versa), the same illumination Restore levels. Continuing with this example, the "right" gesture switches the lighting between "on" and "off" states.

As another example, when enabling "Focus", the left head gesture may toggle the autofocus feature "on" or "off", and the right gesture opens a dedicated menu for controlling the focus feature. It's fine. Reference is now made to FIGS. 4D-4G, which illustrate implementations for displaying one or more status bars indicative of system settings, constructed and operable in accordance with embodiments of the disclosed technology. The illustrations of FIGS. 4D-4G are intended to be exemplary only; other system indicators may be added or replaced with the indicators shown in FIG. 4D. Referring to FIG. 4D, status bar 420 includes a plurality of various symbols indicating the current configuration of various settings of system 100. Status bar 420 is displayed via HMD 102 in response to input by surgeon 120 via UI 160, for example. Status bar 420 includes a clock 422, an ROI indicator 424, a footswitch indicator 426, lighting indicators 428 and 430, a system mode indicator 432, and a color correction indicator 434. Clock 422 displays the current time. ROI indicator 424 shows the displayed ROI for the entire ROI. In FIG. 4D, ROI indicator 424 shows a smaller rectangle representing the displayed ROI located within a larger rectangle representing the entire ROI, which may occur, for example, when surgeon 120 uses a zoom-in feature to view a magnified portion of the raw image. It was displayed. A number indicating the current magnification will be displayed. Footswitch indicator 426 indicates the available functions of footswitch 104. In the example shown, the available functionality is indicated by four icons. Moving clockwise, the icon in the upper left corner indicates XY motor scrolling, the icon in the upper right corner indicates illumination, the icon in the lower right corner indicates focus, and the icon in the lower left corner indicates zoom. In some embodiments, an icon indicating the currently active function of footswitch 104 is highlighted. Lighting indicator 428 indicates that red-eye lighting is set to "on" and the level is 62. Lighting indicator 430 indicates that the floodlight is set to "on" and the level is 65. System mode indicator 432 displays the current system mode, which in this example is posterior segment mode. Color correction indicator 434 displays the type of color correction applied to the image, and in this example, no color correction has been made. Status bar 420 may be displayed at any suitable location via HMD 102, such as at the top of the display or any other location set by surgeon 120, and at any depth when displayed to both eyes. , the depth can be set. In some embodiments, status bar 420 is displayed only when parameters are changed. The status bar 420 displays additional system characteristics such as depth of field, digital gain, and patient data including, for example, name, age, gender, name of the surgeon or surgeons, date, location, hospital name or clinic name, etc. may include other suitable parameters.

Reference is now made to FIG. 4E, which shows a live image of a patient's eye undergoing a surgical procedure overlaid with a status bar 420 indicating the current system configuration to the surgeon 120.

Reference is now made to FIG. 4F, which shows the raw image of FIG. 4E with a second status bar 442 overlaid in addition to the status bar 420. In this exemplary implementation, status bar 442 is an arc that indicates the latest changes to the lighting settings. Status bar 442 is displayed to update surgeon 120 about the current settings of system 100 in response to changes made to the lighting.

Reference is now made to FIG. 4G, which shows another raw image 444 of a patient's eye undergoing a surgical procedure. A status bar 446 and a status bar 448 are displayed on the raw image 444. Status bar 446 is configured differently than status bar 420 of FIG. 4E. Status bar 448 appears when an external device, such as a phacovitrectomy machine, is connected to system 100 and displays system settings corresponding to the external device. In some embodiments, a user may configure system 100 to continuously display status bar 420 or to hide status bar 420 in response to user input or mode changes. In some embodiments, the default setting is to display status bar 420 continuously. In some embodiments, system 100 provides multiple different status bars (not shown) in addition to status bar 420. Each status bar may be configured differently, the one or more status bars configured to be displayed sequentially, and the one or more status bars configured to be displayed by the user in response to any of the parameter changes. Appears in response to input, in response to external device detection, etc.

In some embodiments, system 100 may issue a reminder or alarm when the phaco probe is not centered in the lens capsule. In some embodiments, system 100 may issue a reminder or alert, depending on the surgical technique selected, for example, if the chopper is not positioned below the phaco probe. System 100 may automatically stop suctioning the surgical instrument if the posterior capsule is suctioned toward the phaco, as may be detected via image processing, for example. The system 100 is configured by the surgeon 120 to calculate the distance between the patient 120 and the tool using either processing of surgical procedure images, analysis of iOCT data, analysis of preoperative or intraoperative models of the eye, etc. Surgical tools used may be tracked.

In some embodiments, system 100 tracks one or more surgical tools, such as by applying image processing techniques to raw images, applying image processing techniques to intraoperative OCT data (iOCT), etc. It's fine. System 100 may include a dedicated tool tracking subsystem based on optical or electromagnetic tracking, or the like. Optionally, the tool tracking subsystem is also based on a preoperative or intraoperative model of the eye.

Tool tracking functionality may be used for a variety of features. For example, tool tracking allows a user to use a surgical tool used as an electronic pen during a surgical procedure. This may be useful in a teaching mode where the supervising surgeon may place markings on the raw image to guide the surgeon. UI 160 allows surgeon 120 to turn drawing features on or off, eg, via voice commands, footswitch 104, etc. Alternatively, a physical tool may be paired with the system 100 to press a virtual button or select a menu item displayed via the HMD 102, such as around the retina in a posterior segment procedure, or around the retina in an anterior segment procedure. may be used to select items from a menu displayed around the eye or elsewhere in the operating room.

Tool tracking functionality may be used to detect and indicate the distance between the tip of the tool and the patient, such as the retina in the case of eye surgery. The tool tracking feature may be used to initiate automatic safety measures, such as automatically stopping phacoaspiration. Tool tracking functionality may be used to guide the surgeon 120 during the procedure. These examples are intended as exemplary only and do not limit this functionality to the particular examples given.

A description of system characteristics that can be controlled via user interface 160 is now provided below. Note that some system characteristics are indirectly controlled by the user when switching modes. For example, switching to a different mode may change the positioning of the camera head 110, or the activation of the camera, or the illuminator. A user may not be able to adjust these settings directly, but only indirectly by switching system modes and other settings inherent in system 100. This list is not intended to be limiting. Each system mode is characterized by two or more of the following system characteristics:
-Display status (on/off),
-Screen status (display/world fixed),
-Shutter status (open/closed),
- Selection of display content (raw image/preoperative/other),
- lighting conditions;
-color correction scheme,
-Image enhancement scheme,
- eye inversion,
- motor speed status,
- motor boundary conditions,
- Picture-in-picture (PIP) display (on/off)
- User interface settings,
-Display overlay, and -iOCT settings.

The display state for HMD 102 may be set to "on" or "off," and the default display state for module 130 is "on." When the display is turned off, computer 118 stops the image source for display modules 130A and 130B. Alternatively, computer 118 streams a black image to HMD 102.

Screen states for HMD 102 define how images and content are displayed to surgeon 120. There are two screen states for the HMD 102: a full screen display fixed state and a virtual screen world fixed state. In the full screen display fixed state, the image appears to be displayed on a screen that is rigidly suspended from the HMD 102, and the displayed image remains fixed and moves with the surgeon 120. This is the default state. In the world fixation state, one or more virtual screens are displayed as if they were fixed in the operating room. As the surgeon 120 moves his head, the displayed content changes such that portions of the displayed content move in and out of the FOV of the HMD 102.

Reference is now made to FIG. 5A, which illustrates a projected image as perceived by surgeon 120 when display module 130 is in a display-locked state, constructed and operable in accordance with another embodiment of the disclosed technology. FIG. 5A shows images 500 and 502 displayed in a fixed full-screen display. The field of view of HMD 102 is filled with live magnified images acquired via camera system 112 and displayed as if fixed to the surgeon's 120 head. Elements 512 and 514 indicate the orientation of the surgeon's 120 head. The dashed arrows on elements 512 and 514 indicate the front facing orientation. The solid arrow in element 512 with respect to image 500 indicates that the orientation of surgeon 120 is 5° to the right of the front facing orientation. The solid arrow in element 514 with respect to image 502 indicates that the orientation of surgeon 120 is 25° to the right from the front facing orientation. By moving your head to change orientation, the image that appears in your field of vision does not change as if the screen were fixed to your head, but instead moves with you as you make the head gesture. .

wherein the virtual screen world is displayed via the HMD 102, each corresponding to a different position and orientation of the head of the surgeon 120, generally configured and operable in accordance with further embodiments of the disclosed technology. Reference is made to FIGS. 5B-5C, which show two views 504 and 506. In the views shown in FIGS. 5B-5C, two adjacent virtual screens 508 and 510 are displayed from the same distance from the surgeon 120 and having the same (virtual) size. In other embodiments, multiple virtual screens with varying (virtual) sizes, positions, and distances may be used. Surgeon 120 simultaneously views images displayed through both world-fixed virtual screens 508 and 510. Virtual screen 508 displays live video of the surgical field, and virtual screen displays preoperative data.

Each of views 504 and 506 includes a raw image feed displayed via virtual screen 508 and corresponding preoperative data displayed via virtual screen 510. Virtual screens 504 and 506 appear as if they were monitors placed at fixed locations within the operating room (OR). As shown in FIG. 5A, elements 512 and 514 of FIGS. 5B-5C each indicate the orientation of the surgeon's 120 head. The dashed arrows on elements 512 and 514 indicate the front facing orientation. The solid arrow in element 512 with respect to view 504 (FIG. 5B) indicates that the orientation of surgeon 120 is 5 degrees to the right from the front facing orientation. The solid arrow in element 514 with respect to view 506 (FIG. 5C) indicates that the orientation of surgeon 120 is 25° to the right from the front facing orientation.

As shown, view 506 of FIG. 5C appears as if shifted to the left relative to view 594 of FIG. 5B, which corresponds to rightward head movement of surgeon 120. Virtual screen 508 in view 506 (FIG. 5B) shows a smaller portion of the raw video than virtual screen 508 in view 504 (FIG. 5C), and virtual screen 510 in view 506 (FIG. 5C) shows a smaller portion of the raw video than virtual screen 508 in view 504 (FIG. 5C). 5B shows more pre-operative data than the virtual screen 510 in FIG. 5B). By moving his head from side to side, surgeon 120 can scroll through the data displayed simultaneously on virtual screens 508 and 510 and see a larger or smaller portion of each, as desired. . A precision world-fixed screen implementation may be realized using a six degrees of freedom (DOF) head gesture tracker with tracking component 134 to perform global position and orientation tracking. In some embodiments, virtual screen mode may be implemented with a 3-DOF tracker that only supports head orientation, eg, if absolute screen position is not required.

Reference is now made to FIG. 5D, which illustrates an exemplary implementation of a virtual screen state, constructed and operable in accordance with other embodiments of the disclosed technology. Surgeon 120 of FIG. 1A wears HMD 102 and observes two virtual screens 508 and 510 of FIGS. 5B-5C while performing surgery on patient 122. A virtual screen 508 displays the raw image feed acquired by cameras 140A and 140B (FIG. 1E), and a virtual screen 510 displays the preoperative image feed that appears as if it were being displayed on other monitors located within the operating room. Project data. Content displayed via HMD 102 may be determined by the system mode. Options for displayed content include: raw image streams acquired via camera system 112 (default), preoperative data (e.g., medical images and/or data about the patient), intraoperative data (i.e., e.g., phacovitrectomy system data from other devices such as), raw image streams provided from an endoscopic camera (not shown). In some embodiments, the option further includes viewing live image streams obtained from a similar system in an adjacent operating room, allowing the surgeon 120 to follow the progress of a parallel surgery, or in the waiting room. It offers live image streams provided from installed Wi-Fi cameras, viewing a cloud-based library of images and videos for intra-surgical reference, and more. The video captured by cameras 140A and 140B and projected through HMD 102 is in 3D. Each eye of the surgeon 120 sees a video feed from a corresponding camera. Other video and image displays via HMD 102 may be either 2D or 3D. For example, videos from a cloud-based library may be 3D and pre-operative images may be 2D.

The shutter state of the shutter 132 of the HMD 102 (FIG. 1C) may be open or closed, and the default state in microsurgery is the closed state. When shutter 132 is closed, display module 130 is opaque, allowing surgeon 120 to view the displayed image via HMD 102 without seeing ambient light behind the displayed image, which reduces the contrast of the displayed image. make it possible. When shutter 132 is open, display module 130 is transparent, allowing surgeon 120 to see the real world view. When the shutter 132 is partially open, the display module displays the background behind the picture-in-picture (PIP), such as when displaying one or more features within the PIP during a VGS procedure in an augmented reality application. The portions where the shutter 132 is open are transparent and the portions where the shutter 132 is closed are opaque to hide and enhance the contrast of the image within the PIP.

Individual lighting states for the lighting system 114 include selecting floodlights on, off, or low, selecting coaxial lighting on, off, or low, and selecting automatic lighting states. Lighting system 114 further includes continuous lighting conditions. Once automatic illumination is enabled, system 100 automatically adjusts the illumination as a function of image histogram analysis, other image processing analyses, HMD 102 position and orientation, surgical procedure progress, and additional parameters. Settings regarding automatic lighting include enable and disable (default).

When the auto-centering state is enabled, the system 100 automatically centers the magnified image according to a default setting that depends on the system mode. For example, in anterior segment mode, centering may be performed using an This can be done by using XY motors when moving. Centering is based on image analysis according to predefined settings that define centering characteristics. Settings related to automatic centering are
○Valid,
○Disabled (default)
including.

When autofocusing is enabled, the system 100 automatically focuses the magnified image according to user preferences. The user may specify the point within the surgical field that the system will next lock onto and maintain in focus (if the camera head or patient moves). Alternatively, the user may choose to automatically focus on the tip of the tool. In a posterior segment procedure, the user may choose to automatically focus on the center of the illuminated area. Settings related to automatic focusing are
○Lock,
○Tooltip,
○center,
○Disabled (default)
may include.

Various color correction schemes may be enabled via system mode. For example, the options are
○ Correction for floodlights (default),
○ Correction for various fiber illumination sources,
Includes o corrections for enhancement when using various dyes, and o user-configurable color schemes.

The eye inversion state is controlled indirectly through the system mode. In the normal state (default), the video stream from left camera 140B is streamed to the left eye of surgeon 120, and the video stream from right camera 140A is streamed to the right eye. The inverted state is commonly used for posterior segment ophthalmological procedures using non-contact lenses. In the inverted state, the video stream from left camera 140B is streamed to the right eye of surgeon 120, and the video stream from right camera 140A is streamed to the left eye. The image is also rotated 180 degrees.

Motor speed states define parameters for various motors of system 100 that control camera focus for camera system 112, X, Y, and Z position for camera head 110, and so on. Options for this state include fast/slow.

Motor boundary conditions limit the degrees of freedom for each of the motors integrated within camera head positioner 111. Options include full motion and limited motion.

Picture-in-picture (PIP) display is controlled by surgeon 120. Options for each PIP include off (default), on, and other. Surgeon 120 defines some default options for the number of PIPs and PIP positions, and determines which options are valid in each system mode and what content is displayed within each PIP in each system mode. You may decide.

User interface settings that define which actions correspond to which user inputs may be customized by the surgeon 120 or by other operators. In some embodiments, various settings regarding the user interface are controlled via touch screen 108. Options for user interface settings include:
○Settings related to foot switch 104. These settings define which commands belong to each button or pedal 104A, 104B, and 104C of footswitch 104.
○Head gesture definition. These settings define which commands belong to each head gesture by the surgeon 120.
○Menu definition. These settings define which options appear when the menu is invoked by the surgeon 120.
○Voice command settings. These settings toggle the voice command functionality state (on/off) and allow the surgeon 120 to turn the voice control feature of the system 100 on or off as desired. The voice command settings may be personalized using known techniques, such as machine learning, to familiarize the computer with commands preferred by the surgeon 120.

It should be understood that this list is merely representative and is not intended to be exhaustive.

Computer 118 controls the display of various overlay symbols to guide surgeon 120 during the surgical procedure. Display overlay settings define which symbols are overlaid on images displayed via HMD 102 and may be customized by surgeon 120.

Additional settings may control other system characteristics, such as attaching additional cameras or sensors (not shown) to camera head 110. For example, via iOCT settings, the surgeon controls iOCT image acquisition. Settings include off (default), on, iOCT scanner operation, mode specific settings such as anterior versus posterior segment mode, and the like.

System 100 allows surgeon 120 to view anatomy-based data overlaying the raw image feed displayed via HMD 102. Surgeon 120 can customize the overlay of data onto the live video feed in a non-obtrusive manner. Settings allow customization, such as turning overlaid data on and off, showing only a portion of the overlay, and manually moving overlay borders.

The anatomy-based data is displayed in alignment with the raw magnified images, where "registration" preserves the spatial relationship between the anatomy-based data and the raw magnified stereoscopic images acquired by cameras 140A and 140B. It is understood as doing. The use of the term "registration" herein is intended to capture a broader meaning than is understood in the traditional context with respect to image registration. Image registration involves the transformation or alignment of at least a portion of one image to at least a portion of another image. Registration as performed herein further includes pinpointing corresponding locations on the two images using any suitable technique, such as identifying similar features, but not necessarily the It does not involve matching at least a portion of the images.

Registration may be based on pre-calibration and matching features in the anatomy-based data to features detected in images acquired in real time by cameras 140A and 140B. The two cameras 140A and 140B are pre-calibrated so that the direction and depth (i.e. position) of each feature imaged by cameras 140A and 140B can be calculated, and vice versa, The symbol may be overlaid on each of the raw images to appear at the desired location relative to cameras 140A and 140B. Registration as defined herein is not to be confused with visor-guided surgery (VGS) or navigation procedures in which a model of an anatomical structure is registered to a tracker coordinate system. In some embodiments, correspondence between anatomical data and raw images is established using position-related data. Correspondence between anatomical data and displayed images may be preserved using techniques other than tracking, for example using position data mapping pre-operative data to real-time data.

Anatomy-based features are generated by OCT images (both preoperatively and intraoperatively), 2D maps of the retina such as retinal thickness maps, 3D models of the epiretinal membrane derived from volumetric OCT scans, and corneal topographers. the desired orientation of a toric intraocular lens (IOL) relative to the eye, pre-planning information marked on a 2D image of the patient's retina by the surgeon, intraoperative markings made by the supervising surgeon, etc.

In some embodiments, a snapshot of the retina taken at one magnification during one stage of surgery is used as a mini-map for a later stage of surgery performed at a higher magnification. During retinal surgery, surgeons commonly use fiber illumination, which illuminates only a small portion of the retina when held near the retina. Minimap allows for better spatial awareness when working at higher magnification. A minimap is displayed within the PIP, and a circle (or other) symbol overlaid on the minimap indicates the location of the illuminated area on the snapshot. Real-time alignment algorithms allow the live video feed to remain aligned to the snapshot even if the eye or camera head is moved. This technique is also useful for endoscopic procedures where the endoscopic video is presented full screen while symbols representing the footprint of the video are overlaid on the mini-map displayed within the PIP. Registration of the endoscopic video to the snapshot may be based on either feature matching or tracking of the endoscope.

In some embodiments, symbols are overlaid on the raw video to indicate data locations of images displayed within the PIP. System 100 allows surgeon 120 to customize this feature, such as by controlling PIP transparency, and customize the position of the PIP in the FOV of HMD 102. For example, one such symbol may be a straight line that is overlaid on the raw magnified image stream and indicates the position in the raw image of the patient's anatomy that corresponds to the preoperative image displayed within the PIP. . This straight line represents the position and angle of the preoperative OCT B-scan displayed in the PIP relative to the raw image.

In some embodiments, symbols are overlaid on the image displayed within the PIP to indicate the location of features appearing within the live video. One example is a symbol overlaid on a snapshot used as a minimap, as described above. As another example, a straight line representing the angle of the toric IOL appearing in the live magnified image may be overlaid on the pre-operative image of the eye displayed in the PIP. The angle of the straight line is corrected in real time with respect to both the rotation of the eye and the instantaneous angle of the IOL inside the eye.

User interface 160 provides for viewing images displayed within a PIP or PIPs overlaid on top of a display of the live image feed. For example, such images may relate to preoperative data, OCT images, notes, and intraoperative raw data such as B-scan images from iOCT, videos acquired via an endoscope, and the like. Through the menu, the surgeon 120 controls the PIP position at a location of his choice, such as the upper left corner of the field of view. This feature allows the surgeon 120 to view preoperative or raw data with the raw image feed occupying the central area of his or her field of view. The size and position of the PIP, along with the content displayed therein, is set by the surgeon 120 via the UI 160.

The figures described below allow the surgeon 120 to enter and exit preoperative system mode by manipulating menu items in the displayed menu and view preoperative data within the PIP with symbols aligned to the raw image. It is intended to show an exemplary implementation for a user interface implemented as a menu overlaid on the field of view. Some menu items represent actions that are taken when selected, while other menu items represent additional menus that result in additional sets of actions. Surgeon 120 navigates the user interface with footswitch 104 and head gestures, as described above.

Here, a menu 630 overlaid on the raw image feed 614, configured and operable in accordance with further embodiments of the disclosed technology, allows a user to interact with the system and switch to various system modes. Reference is made to FIG. 6A. Menu 630 displays the following menus as system settings configurable by surgeon 120: overlay 618, autofocus 620, color correction 622, autocentering 624, PIP 626, system mode 628, and cancel 612.

The overlay setting is currently turned off, as indicated by "Off" displayed in menu item 618, indicating to surgeon 120 that the overlay can be turned on by selecting item 618. The automatic focus setting is currently turned off, as indicated by the "Off" displayed in menu item 620, presenting the surgeon 120 with switching the overlay on by selecting item 620. Similarly, the color correction is currently set to red, as indicated by the "red" displayed in color correction menu item 622, and the color correction is currently set to red, prompting the surgeon 120 to switch to a different color correction by selecting this menu item. presents the option. The automatic centering setting is currently set to "off," as indicated by "off" displayed in menu item 624, and the surgeon 120 is given the option of toggling this setting on by selecting this menu item. present. The PIP setting is currently set to "Off," as indicated by "Off" displayed in menu item 626, and provides surgeon 120 with the option to toggle this setting on by selecting this menu item. present. Cancel 612 and system mode 628 are provided for the surgeon 120 to exit the menu and switch to a different system mode, respectively. System mode menu item 628 is currently highlighted, indicating that surgeon 120 may currently select this menu item.

In one embodiment, the "Color Correction" menu item is highlighted and a submenu containing various color correction options is opened, one of which allows the color correction feature to be toggled off. In some embodiments, when a menu item in the color correction submenu is highlighted, the effect of the color correction scheme corresponding to the highlighted menu item is displayed before the surgeon selects it by releasing footswitch 104. Performed and displayed on the raw image. This allows the surgeon 120 to see the effects of one of the various color correction schemes before selecting one. This feature may be enabled for additional image processing functions, such as image sharpening, for example.

In some embodiments, the opened menu is displayed around the image rather than in the center, allowing the surgeon 120 to evaluate the effects of various image processing and color corrections without the menu covering or occluding portions of the image. make it possible to Menu items may be displayed vertically around the perimeter. However, navigating the menu does not require the surgeon 120 to shift his or her gaze away from the live image. To scroll and highlight various menu items, the surgeon 120 moves his head up and down while keeping his gaze on the central area of the live image.

Surgeon 120 navigates between various menu items by pressing foot switch 104 and moving his head to direct his gaze between menu items. For example, to select system mode menu item 628, the surgeon holds down footswitch 104 while maintaining menu item 628 highlighted (eg, "long gaze") for a predetermined period of time. This results in the system mode menu shown below in FIG. 6B, collapsing the currently presented advanced menu into an item 616 around the field of view, allowing the surgeon 120 to direct his/her line of sight while pressing the footswitch 104 as appropriate. Allows you to return to this menu by attaching it. Thus, in one implementation, to navigate between various menus and open submenus, the surgeon 120 presses and holds the footswitch 104 for a predetermined period of time to highlight various menu items representing the collapsed menu. do. Releasing footswitch 104 is not necessary to navigate between the various menus. However, to select a menu item to activate or deactivate a system setting associated with the selected menu item, the surgeon 120 releases the footswitch 104 while looking at the menu item. Therefore, activation of footswitch 104 is necessary to perform an action in system 100, rather than to navigate between menu items representing possible actions that may be taken.

Referring to FIG. 6B, a system mode menu overlaid on a raw image feed after system mode menu item 628 of menu 630 of FIG. 6A is selected, configured and operable in accordance with other embodiments of the disclosed technology. 600 is shown. Selecting menu item 628 of FIG. 6A by depressing footswitch 104 and gazing at menu item 628 for a period of time causes system mode menu 600 corresponding to system mode menu 400 of FIGS. 4A-4D to change to menu 630 of FIG. 6A. will be displayed instead of. The previously displayed "Advanced" menu 630, shown in FIG. 6A, collapses into menu items 616 displayed around the perimeter of the field of view, such as in the upper left corner, as shown in FIG. 6B. System mode menu 600 includes contactless lens menu item 602, vitals menu item 604, nurse menu item 606, surgical field item 606, preoperative menu item 610, and cancel menu item 612, which are overlaid on raw image 614. indicate. Preoperative menu item 610 is highlighted, indicating that this menu item can now be selected by surgeon 120.

Referring to FIG. 6C, the preoperative image is displayed in a PIP overlaid on the raw image feed, configured and operable in accordance with further embodiments of the disclosed technology. Selecting menu item 610 (“Preoperative”) of FIG. 6B by releasing footswitch 104 switches the system mode to preoperative mode. In this mode, the user selects a folder of preoperative OCT images to be displayed within the PIP. In some embodiments, this mode allows the surgeon 120 to adjust and define the PIP according to user-defined settings, such as size and position. After selecting which preoperative images to view within the PIP, the surgeon 120 returns to non-contact lens mode using the footswitch 104 and the menu navigation techniques described above. Returning to the non-contact lens mode, the pre-operative image 632 is displayed within the PIP at the top left hand corner of the display of the live image 614, as shown in FIG. 6C. The preoperative OCT image is of the same tissue displayed in raw image 614. Note that the location and size of the PIP are merely exemplary and may be user-defined by the surgeon 120.

Referring to FIG. 6D, menu 630 of FIG. 6A is shown overlaid on a raw image feed with a PIP, configured and operable in accordance with other embodiments of the disclosed technology. Surgeon 120 causes menu 630 to reappear, for example, by depressing footswitch 104. Menu 630 is displayed overlaid on raw image 614 with PIP 632 displayed in the top left hand corner. Overlay menu item 618 is highlighted, indicating that surgeon 120 has selected this menu by depressing footswitch 104 and making a head gesture. The state of the overlay setting is currently off, as indicated by "Off" displayed in menu item 618, and prompts surgeon 120 to toggle this setting on by selecting menu item 618. .

Reference is now made to FIGS. 6E-6J, which illustrate a series of raw images 614 with corresponding content displayed within a PIP 632, generally constructed and operable in accordance with other embodiments of the disclosed technology. 6E-6J show an overlay 634 superimposed on the raw image 614. Overlay 634 is a straight line that indicates the position on raw image 614 that corresponds to the preoperative OCT image displayed within PIP 632 . The raw image 614 is a 3D image, i.e. each eye of the surgeon 120 sees a slightly different image that together form the 3D image 614, so the straight line 634 represents the two images projected to each eye of the surgeon 120. are individually overlaid on each of them. As a result, the surgeon 120 sees an overlaid straight line 634 with the correct depth perception on the surface of the 3D image 614 showing the retina.

In some embodiments, computer 118 stores a plurality of pre-operative images in a library. As each image is displayed within PIP 632, the position on raw image 614 that corresponds to the image is indicated by an overlay, such as a straight line 634. The tissue location indicated by each overlay 634 corresponds to a preoperative OCT image simultaneously displayed within PIP 632. The user scrolls within the set of preoperative images by depressing the pedal of footswitch 104 and simultaneously performing head gestures (e.g., head down rotation to scroll down, and head up rotation to scroll up). . As the preoperative OCT image displayed in PIP 632 changes, the corresponding position on the real-time image is calculated and displayed by overlay straight line 634. In the image shown, the position on the real-time image is correspondingly shifted downward. Calculations for corresponding locations on real-time images are performed frame-by-frame when image 614 is raw and the overlay requests dynamic updates, even if the preoperative OCT image displayed within PIP 632 has not been modified by surgeon 120. can be carried out. This calculation is based on data generated by the preoperative OCT imager, comprising a preoperative image of the retina and image positions corresponding to all preoperative OCT images. Using this pre-operative image of the retina, the corresponding location on the real-time image 614 can be calculated. This feature allows the surgeon 120 to use handless gestures to scroll in the pre-operative data within the PIP 632 with a corresponding overlay on the live image feed 614. It may be noted that the display of OCT images within PIP 632 is intended to be illustrative only and not limiting of the invention. PIP 632 may display other related content corresponding to the location on raw image 614 indicated by straight line 634.

In some embodiments, the surgeon 120 may freeze the raw images while viewing the preoperative data within the PIP. This may be useful for drawing features or symbols on the raw image while the PIP displays the pre-operative data along with the raw image feed. Freezing the image during drawing is useful when the raw video displays movements due to involuntary saccadic eye movements, breathing, etc., for example. The drawn symbols may be used to guide the surgeon 120 in subsequent actions, such as cutting an incision, after unfreezing the live image feed.

In some embodiments, the surgeon 120, via the UI 160, may specify a point within the image displayed via the HMD 102. For example, a head gesture activated by footswitch 104 may move a cursor to specify a point on the image. Alternatively, points may be designated on the image by tracking the eye movements of the surgeon 120. The designation may serve a variety of purposes, such as:
- Specify points for manual alignment. Corresponding points in both the PIP and the live video may be specified to manually align images within the PIP.
- Specifying a point for locating a point in the aligned image. After the images (or models) are registered, the surgeon 120 can specify points in the live video and the corresponding points in the registered images will be highlighted, and vice versa.
- Specify the point for calling PIP. The small PIP may be invoked in the vicinity of a point specified by the surgeon 120, and the content of the PIP may be associated with the specified point, such that the surgeon 120 focuses on a point within the surgical field 124 and Enables you to view related data near.
- Specifying points to control the region of interest (ROI) of the iOCT scanner 142; One or more 2D B-scan or 3D volumetric iOCT images are displayed along with the raw video, and the surgeon can determine the exact location of the iOCT scanner 142 by designating a symbol, such as a cross, in the video acquired by the camera head 112. Control ROI.
- Specify a point for automatic focus. Autofocus may be activated based on the ROI around the specified point.

In some embodiments, the surgeon 120 controls the symbol overlaid on the live image feed via head gestures, and the preoperative image corresponding to the location of the overlaid symbol is displayed within the PIP. As the position of the symbol changes, the corresponding image displayed within the PIP changes. In some embodiments, if there is no preoperative image corresponding to the selected location on the raw image feed, computer 118 selects the preoperative image closest to the selected location and calculates that distance to surgeon 120. Shown below.

Reference is now made to FIGS. 6K-6P, which illustrate a series of raw images 642 with corresponding content displayed within a PIP 644, generally configured and operable in accordance with other embodiments of the disclosed technology. The embodiment of FIGS. 6K-6P is substantially similar to that described above with respect to FIGS. 6E-6J, with the notable difference that a boundary line 646 of the PIP 644 is used in place of the straight line 634 of FIGS. 6K-6P. , indicating the correspondence between the content shown in PIP 644 and the organization displayed in raw image 642. In the example of FIGS. 6K-6P, PIP 644 displays an OCT cross-section of the tissue shown in raw image 642, but this is intended to be illustrative only and not limiting. Other content may be displayed within the PIP 644 if appropriate. The OCT images shown in PIP 644 were acquired during the preoperative or intraoperative phase and stored in memory 118D (FIG. 1B) for later use. During the treatment, the HMD 102 displays the surface of the living tissue as a raw image 642. The raw image 642 may be a real-time video representation of the tissue of the patient 122 or, as described below with respect to FIGS. 6R-6V, for example, to allow the surgeon to draw features for further use. , may be a frozen portion of real-time video.

When superimposed on the raw image 642, the HMD 102 displays in the PIP 644 a previously acquired OCT cross-sectional view of the raw tissue at the position indicated by the boundary line 646. 6K-6P are overlaid onto the raw image 642 at a number of different positions starting from the top left position of the raw image 642 (FIG. 6K) and moving down and to the right in the raw image 642 (FIGS. 6L-6P). PIP644 is shown. For each location of PIP 644 on raw image 642, the OCT image displayed within PIP 644 at that location corresponds to a cross-section of tissue indicated by border line 646. Thus, by manipulating the position of PIP 644 on raw image 642, surgeon 120 may view various cross-sectional slices of tissue.

Reference is now made to FIG. 6Q, which illustrates the raw image 642 of FIGS. 6L-6P with corresponding content displayed within two PIPs 644 and 648, configured and operable in accordance with further embodiments of the disclosed technology. . The embodiment of FIG. 6Q is substantially similar to that described above with respect to FIGS. 6L-6P, except that two PIPs 644 and 648 are superimposed on raw image 642, and each PIP Content corresponding to the organization indicated by boundaries 646 and 650 is shown. PIP 644 shows an OCT cross-sectional slice of living tissue at the location indicated by border line 646, and PIP 648 shows an OCT cross-sectional slice of tissue at the location indicated by border line 650. As discussed above, the surgeon may manipulate the position of PIPs 644 and 648 on raw image 642 to display cross-sectional slices of tissue at various locations.

Reference is now made to FIGS. 6R-6V, which generally illustrate implementations of drawing functionality, constructed and operative in accordance with further embodiments of the disclosed technology. 6R to 6V show the results of drawing symbols using the drawing function displayed via the HMD 102. In one embodiment, the drawing is performed while wearing an HMD, such as HMD 102, that displays live images of patient 122. The drawing function is activated via one or more menus displayed via the HMD, and the HMD wearer selects a drawing tool (not shown) using techniques described herein. enable. The drawing tool is displayed on top of the raw image, and the HMD wearer manipulates the drawing tool using one or more of head gestures, foot switch 104, wand operation, eye movement tracking, etc. Allows to draw symbols overlaid on the raw image. Alternatively, drawing may be performed via a user interface (not shown) provided in system 100, such as a mouse or wand.

In one mode of operation, the surgeon 120 draws features during a preoperative planning stage, for example, for later use as a guide during a medical procedure. In this case, the surgeon 120 views the drawing menu and tools displayed over the raw image via the HMD 102, along with any preoperative data. Surgeon 120 draws one or more preliminary symbols on the raw image. In one embodiment, the raw image feed is frozen during rendering to increase accuracy. Once drawn, the drawn symbol remains visible with the raw image after the raw image feed is unfrozen and no longer displays preoperative data, allowing the surgeon 120 to view the raw image feed while performing a medical procedure. It is possible to see drawing symbols overlaid on the

In other modes of operation, such as a teach mode, a supervising surgeon (not shown) draws symbols to guide the surgeon 120 who subsequently performs the procedure. In the teach mode, the supervising surgeon may be in the operating room with the surgeon 120 and draw symbols on the fly. Alternatively, the supervising surgeon may be located elsewhere and draw the symbols remotely. The supervising surgeon may draw symbols either through another HMD, through a smartphone, tablet, or other touch screen, or through a mouse or wand with a standard monitor, etc.

Referring to FIG. 6R, a raw image 660 of patient's 122 eye is shown prior to virtual rendering of the symbol. 6S-6T, after selecting a virtual drawing tool from a paint menu (not shown), the surgeon virtually draws a symbol 662 on the raw image 660, for example to indicate where to make an incision. Symbol 662 is shown partially drawn in FIG. 6S and fully drawn in FIG. 6T. As mentioned above, the symbol 662 may be drawn by the senior surgeon wearing the second HMD during the teach mode or by the surgeon 120 wearing the HMD 102. Symbol 662 may be rendered using any suitable UI technology, such as those referenced herein. For example, to draw symbol 662, surgeon 120 depresses footswitch 104 while moving the virtual drawing tool using a head gesture that follows a trajectory corresponding to the shape of symbol 662, and once drawing symbol 662 is completed ( (not shown) and releases the foot switch 104. Alternatively, the surgeon may use a mouse or his or her finger to draw on the live image displayed via the touch screen. The surgeon drawing on the live image may optionally not be present in the operating room and may use various UI methods for drawing from a remote location (such as an HMD and footswitch to display the 3D live image within his or her office). (e.g., using a 2D raw image, drawing on a tablet displaying the 2D raw image, etc.). Referring to FIG. 6U, surgeon 120 uses symbols 662 drawn and overlaid on raw image 660 as a guide for making an incision in patient's 122 eye. Surgeon 120 places surgical instrument 664 at a physical location in patient's 122 eye that corresponds to the location indicated by symbol 662 on raw image 660. Referring to FIG. 6V, surgeon 120 moves surgical instrument 664 along a path corresponding to symbol 662 to make an incision in patient's 122 eye. In this manner, surgeon 120, or a senior surgeon, may draw a virtual path on raw image 660 before making an incision to ensure that subsequent incisions are made correctly.

After the surgeon 120 draws one or more symbols or markings to highlight particular features, the surgeon 120 displays which markings while viewing the live image in order to proceed with the procedure using the markings for guidance. You may choose to do so. Surgeon 120 may scroll through any number of pre-drawn markings during the procedure according to need and validity. The system 100 may be configured to erase markings after a predetermined period of time to prevent them from obscuring the display of the live image once the markings are no longer in use. In one embodiment, the symbol automatically disappears according to default settings (eg, after 10 seconds). In other embodiments, a dedicated pre-planning application menu allows the surgeon 120 to disable (ie, turn off) each of the markings.

Reference is now made generally to FIGS. 7A-7E, which illustrate exemplary implementations of menu-driven user interfaces for focusing on selected viewing areas, constructed and operable in accordance with further embodiments of the disclosed technology. be done. In the figures described below, the user interface is implemented as a series of menus overlaid on the raw image stream. Surgeon 120 uses footswitch 104 in combination with head gestures to navigate the menus, as described above.

Referring to FIG. 7A, a menu 730 is displayed overlaid on the raw image feed 714 for focusing on a selected viewing area configured and operable in accordance with embodiments of the disclosed technology. . Menu 730 corresponds to menu 630 shown in FIG. 6A described above. Similar to menu 630, menu 730 includes the following menu items: overlay 718, autofocus 720, color correction 722, autocentering 724, PIP 726, system mode 728, and cancel 712. Menu item 720 for the autofocus feature is highlighted, indicating that this item may currently be selected by surgeon 120. The autofocus feature is currently off, as indicated by the "off" setting displayed in menu item 720, and the surgeon 720 turns on the autofocus feature by "pressing" and activating menu item 720. I'm asking you to switch to . In some embodiments, the surgeon 120 may press the footswitch 104 without releasing the footswitch 104 while staring at the menu item 720 (FIG. 7A) for a predetermined period of time, such as 0.5 seconds. Autofocus is turned on by highlighting menu item 720 for a predetermined period. In other embodiments, the surgeon turns on autofocus after highlighting menu item 720 and releasing footswitch 104.

7B-7D, an exemplary implementation of a user interface screen that enables designation of a selected viewing area for focus is constructed and operable in accordance with further embodiments of the disclosed technology. shown. Activating autofocus 720 to turn on this feature causes UI screen 740 to overlay the display of image 714 and replace menu 730 shown in FIG. 7A. UI screen 740 includes a prompt 742 and designation symbols 744, shown in this implementation as a set of brackets. In some embodiments, the surgeon 120 turns on autofocus by releasing the footswitch 104. Prompt 742 requests surgeon 120 to manually specify an area within raw image 714 to perform autofocus. The surgeon 120 specifies the area by operating the designation symbol 744 on the display of the image 714. The position of designation symbol 744 may be manipulated using any of the techniques described above, such as by performing a head gesture in combination with foot switch 104, for example. Optionally, the surgeon 120 designates the area of focus by staring at the desired area for a predetermined period of time, such as 0.5 seconds. Eye tracker component 136 (FIG. 1C) tracks the line of sight of surgeon 120 to detect designated and selected areas. In some embodiments, system 100 continuously provides automatic autofocus to designated areas based on tracking of surgeon's 120 eye movements. It should be noted that other symbols may be used as designation symbols according to user preferences and circumstances. For example, the designation symbol may be an arrow, a cursor symbol, etc. to precisely designate a particular point.

Referring to FIG. 7C, the surgeon 120 moves the designation symbol 744 to an area 746 to the left of the image 714 that is not fully in focus. Region 746 is indicated by a dashed circle. The surgeon 120 moves the designation symbol 744 using any suitable UI technique, such as, for example, by pressing the foot switch 104 and rotating his or her head to the left. Surgeon 120 designates areas 746 to improve focus. If autofocus is activated by highlighting menu item 720 for a predetermined period of time without releasing footswitch 104, the surgeon places designation symbol 744 in the region by gazing at region 746 and releasing footswitch 104. 746, the area 746 for automatic focus is designated. When autofocus is activated by releasing the footswitch 104, the surgeon 120 sets the area for autofocus by placing a designation symbol 744 in the area 746 for a predetermined period of time, such as 0.5 seconds. Specify 746. System 100 may tolerate slight movement while designation symbol 744 is located in region 746.

Referring to FIG. 7D, computer 118 digitally focuses on area 746 designated by designation symbol 744. Comparing the region 746 of FIG. 7C before focus and the region 746 of FIG. 7D after focus may show the result of digital focus.

Autofocus can be implemented in various modes. In one mode, when autofocus is activated and a focus point (i.e., area 746) is selected, autofocus locks on the selected area and remains in focus until the autofocus feature is deactivated. do. This may be useful in applications where the position and orientation of the camera head changes to capture different parts of the patient's anatomy. As long as the selected region is within the raw image feed, this region will remain in focus.

In other modes, "autofocus" is a one-time action. Each time the autofocus feature is activated by selecting menu item 720, a designation symbol 744 is displayed to prompt the surgeon 120 to designate the area to focus on. Rather than locking onto a specified area, the system 100 focuses on the specified area as a one-time action. In this implementation, menu item 720 says "Auto Focus" without specifying "off" or "on."

Reference is now made to FIG. 7E, which is another view of the menu of FIG. 7A overlaid on a raw image feed, constructed and operable in accordance with embodiments of the disclosed technology. Menu 730 was invoked and redisplayed using the techniques described herein. However, at this time, the menu item 720 indicating the autofocus feature displays "on", indicating that autofocus is currently on. To turn off autofocus, surgeon 120 selects menu item 720 using the techniques described above.

7F-7F illustrating other exemplary implementations of menu-driven user interfaces for focusing on selected areas of the field of view, generally configured and operable in accordance with other embodiments of the disclosed technology; Reference is made to 7I. 7F-7I are substantially similar to FIGS. 7A-7D described above. Referring to FIG. 7F, a menu 750 is shown overlaid on a raw image 752. Menu 750 corresponds to menu 630 of FIG. 6A and includes the following menu items: overlay 754, autofocus 756, color correction 758, autocentering 760, PIP 762, system mode 764, and cancel 768. Surgeon 120 turns on autofocus via UI 160, as described above. Referring to FIG. 7G, a UI screen 768 is overlaid on the raw image 752. UI screen 768 displays a prompt 770 and a symbol 772 (eg, a pair of parentheses). Prompt 770 requests surgeon 120 to specify a focus point on raw image 752 by manipulating symbol 772. Referring to FIG. 7H, the surgeon 120 moves symbol 772 to designate an out-of-focus feature 774 (eg, a blurred black circle on a white background). Referring to FIG. 7I, features 774 in raw image 752 are displayed in sharper focus than in FIG. 7H. Surgeon 120 can continue the surgical procedure while viewing features 774 with better resolution than before.

System 100 further provides a pre-planning phase that allows surgeon 120 to customize the display of data via HMD 102 to include one or more overlays. Before performing a surgical procedure, surgeon 120 may add one or more symbols and notes to the preoperative image. During the surgical procedure, added symbols and notes are displayed in an overlay aligned with the live video captured by camera system 112. For example, marks indicating the optimal location for photocoagulation are displayed, overlaid, and aligned to the raw image feed. As another example, the surgeon 120 colors areas of the preoperative image to create a color map, such as blue for attached retina and red for detached retina. The color map is then overlaid on the live video during the surgical procedure.

System 100 further allows surgeon 120 to control the capture of any live image snapshots captured by camera head 110 throughout the surgical procedure. The snapshot is stored on computer 118. The stored snapshots may be recalled from memory and viewed by the surgeon on a separate screen or PIP on display module 130 at a later stage of the surgical procedure to monitor progress or assist in guidance. Snapshots may be stored in 2D or 3D format. Snapshots may also be stored in short video clips, or “GIF” format, that are repeatedly rendered on display module 130 within the PIP. This GIF format is advantageous for viewing membranes. In PIP mode (ie, after "selecting" a PIP), activating a snapshot may save only the PIP image. This may be useful for storing iOCT or endoscopic images displayed within the PIP.

For example, a snapshot may be taken after dye is applied to enhance the membrane. Even after the dye is absorbed and disappears, the surgeon 120 can still see the stained area within the PIP displayed with the live video feed. Since the snapshot is aligned to the live video, the surgeon 120 may specify a point in the PIP as described above, and the corresponding point in the live video will be automatically highlighted, colored, or marked with a symbol. Ru.

The disclosed technology further allows the surgeon 120 to create GIFs. Surgeon 120 initiates recording of images acquired by camera head 110, such as by using pedal 104 in conjunction with a head gesture, and stops recording using a similar set of UI inputs. For example, system 100 displays a GIF menu item (not shown) that, when selected, causes system 100 to record an image for a GIF. To select a GIF menu item, the surgeon 120 holds down the footswitch 104 and stares at the GIF menu item for a predetermined period of time, such as two seconds. To finish recording images for the GIF, surgeon 120 releases footswitch 104. The surgeon 120 assigns the name by head gesture or via voice control to a menu item that describes identifying details, such as a timestamp with the identity of the patient 122, for example. In some embodiments, system 100 automatically assigns names to snapshots or GIFs using known techniques. The GIF may be stored in memory 118D (FIG. 1B) and later recalled and displayed on a separate screen or within the PIP as appropriate by selecting a menu item.

When the system 100 is activated, the surgeon 120 moves the camera head 110 closer to the design position, utilizing coaxial/red-eye illumination beams that overlap around the patient's eyes only when the camera head 110 is correctly positioned. May be placed manually. The surgeon 120 may then instruct the system to automatically fine-tune the position of the camera head 110 to the correct working distance. Computer 118 automatically raises high-resolution cameras 140A and 140B to an appropriate height above the area of interest in surgical field 124; mechanical arm controller 118G operates the z-axis motor; operates the XY motors to center the patient's eyes and sets the lenses of cameras 140A and 140B to nominal shutter, focus, and zoom values, and the illumination system controller 118I directs the illumination system 114 to the anterior segment setting. Control.

System 100 may be configured to save system settings for later use. After the surgical procedure is completed, surgeon 120 may edit system settings via touch screen 108 and name the settings for future use. The settings may override the "default settings" or may later appear as a menu item displayed via a menu on the HMD 102. Touch screen 108 allows surgeon 120 to select any saved settings that are later displayed via HMD 102.

In some embodiments, system 100 identifies surgeon 120 when wearing HMD 102. A user, such as a surgeon 120 or a nurse, selects a surgeon 120 from an existing list of surgeons, each surgeon in the list being associated with a set of preconfigured settings, as described above. The system 100 may be configured using any suitable technique, such as voice recognition, or an eye tracker camera integrated with the HMD 120, for example, to compare acquired images of the surgeon's 120 eyes with previously acquired images. The biometric data is used to automatically identify the surgeon 120, such as using a biometric data. Once the surgeon 120 is correctly identified, the system 100 is configured with preconfigured settings associated with the surgeon 120.

Similarly, in some embodiments, the system 100 includes real-time images of the eyes of the patient 122 and (perhaps automatically compares the preoperative images. This is, for example, to eliminate errors and to ensure that the preoperative data the surgeon is using belongs to the patient 122 and that the appropriate surgical procedure is being performed on the appropriate eye. System 100 alerts surgeon 120 of any discrepancies between the preoperative data being used and the raw image stream acquired for patient 122.

In some embodiments, system 100 displays a checklist to surgeon 120 either at the beginning, during, or at the end of a surgical procedure. Surgeon 120 may manually check items from the checklist that are actions performed, or system 100 may detect the performance of these actions and automatically check items from the list. The checklist may be customized for each surgeon and may be displayed upon identifying the surgeon 120, as described above. The checklist may be displayed on the side screen or as a PIP in the real-time image, and the surgeon may choose to turn it on or off. In some embodiments, the computer 118 (FIG. 1B) sends a centering symbol via the HMD 102 to assist the surgeon 120 in correctly positioning the HMD 102 to allow the surgeon 120 to see the entire FOV. Display. For example, the centering symbol may be a frame or other suitable symbol denoting the boundaries of the FOV. The symbol is displayed when the surgeon 120 first dons the HMD 102, when the HMD is moved after being stationary, as detected by the HMD tracker, or upon user command (via any UI method). If the HMD 102 moves or is misaligned, it may be automatically displayed. If the HMD 102 includes the ability to change the interpupillary distance (IPD) of the display, this feature may assist in verifying that the IPD was set correctly and may be adjusted as necessary.

In some embodiments, system 100 detects and corrects for effects related to non-overlapping images in a 3D digital microscope. This problem arises when the camera captures an object that is not at the designed working distance and the surgeon uses the lens focus motor to set the focus. When this occurs, the images produced by the two cameras 140A and 140B (FIG. 1E) will not completely overlap. If these images were streamed to both eyes of surgeon 120 via displays 130A and 130B (FIG. 1C), they could result in eye strain. Overlapping parts of the images are not displayed as if they came from the same direction in space. The surgeon 120 must squint so that his brain merges the two images into a single 3D image.

This situation can be solved automatically, for example by using image processing to calculate the boundaries of the overlap between two 2D images, or by using calibration data to derive the actual working distance from the state of the focus motor. can be detected. Each camera lens focus motor state has a corresponding pre-calibrated working distance. By comparing the actual working distance and the design working distance, this discrepancy can be calculated and taken into account.

One solution is to shift the two images and fill in the missing part of the image with black bars. In this way, both eyes of the surgeon 120 see the overlapping portions of the images as coming from the same direction, and the ends of the non-overlapping FOV are visible to only one eye of each (unless the displayed image does not include black areas. (in which case the display unit does not emit any light in the corresponding area of the display).

8A-8B, generally constructed and operable in accordance with the disclosed techniques, illustrate techniques for correction effects with respect to non-overlapping images caused by a mismatch between the actual working distance and the design working distance of a camera. Referenced. FIG. 8A shows an optical setup 800 showing a discrepancy between the actual working distance 802 and the designed working distance 804. As a result of the mismatch, the images acquired by cameras 140A and 140B (FIG. 1E), represented by straight lines 28 and 11, respectively, are misaligned and do not completely overlap, so that the images appear to be coming from different directions. looks like.

Referring to FIG. 8B, the results of the shift and correction of images captured by cameras 140A and 140B are shown. Grid 810 shows a numerical depiction of the image captured by camera 140A as displayed in the left eye of surgeon 120, and grid 812 shows a numerical depiction of the image captured by camera 140B as displayed in the right eye of surgeon 120. show. Due to the discrepancy between the actual working distance and the design distance for cameras 140A and 140B, the two rightmost columns of grid 810 are not included in grid 812 and the two leftmost columns of grid 812 are not included in grid 810. Returning to FIG. 8A, endpoints of the straight line corresponding to the image acquired by camera 140A, labeled 28, that do not overlap with the straight line corresponding to the image acquired by camera 140B are not included in grid 812. Corresponds to the upper right corner of grid 810. The end point of the straight line corresponding to the image acquired by camera 140B that does not overlap with the straight line corresponding to the image acquired by camera 140A, indicated as 11, is the upper left corner of grid 812 that is not included in grid 810. corresponds to

To correct for this mismatch, grid 810 is shifted one column to the right, shown as grid 814, and grid 812 is shifted one column to the left, shown as grid 816. The leftmost column 814A of grid 814 is filled with black because this portion of the image is not displayed to surgeon 120, and the rightmost column 816A of grid 816 is filled with black because this portion of the image is not displayed to surgeon 120. Grid 818 shows in the center (white background) the overlapping features of both grids 810 and 812 that are visible to both eyes of surgeon 120. The leftmost column 818A, shown with a gray background, is visible only to the surgeon's 120 right eye, and the rightmost column 818B, shown with a gray background, is visible only to the surgeon's 120 left eye.

The shifting techniques described above may be performed continuously while the focus of cameras 140A and 140B is adjusted. Alternatively, the shifting technique may be performed only upon completion of the focus adjustment, for example when the enabling pedal of footswitch 104 is released. In this case, the shifting technique is performed gradually over a short period of time, such as a few seconds, so that the surgeon 120 does not experience sudden changes in the displayed image. The surgeon 120 may turn the shifting technique on or off and set its duration via the UI 160. Note that in the example of FIG. 8B, the combined image has the same magnification as each of the independent images. However, this is not required and the magnification may be reduced to some extent so that the combined image represented by grid 818 comprises the entire imaging FOV. For example, the numerical representation of the combined image includes all columns of each acquired image, where the leftmost column corresponds to the leftmost column of grid 812 (with number 11 as the top leftmost grid entry) and the rightmost column , corresponds to the rightmost column of grid 810 (having number 28 as the top grid entry). In effect, when this option is applied, when the user changes focus, the zoom will also change slightly.

In some scenarios, non-overlapping images may occur. One such scenario that requires correction using a shifting technique occurs at the beginning of a procedure when the surgeon 120 positions the camera head 110 approximately at the design working distance. Surgeon 120 manually adjusts the height of cameras 140A and 140B housed within camera head 110 above the surgical site until the two spots of light produced by the two floodlights of illumination system 114 overlap. obtain. These lights may be designed to overlap at the design working distance. If camera head 110 is not located at the full design working distance, the images generated by cameras 140A and 140B and projected to both eyes of surgeon 120 via displays 130A and 130B will be completely overlapping, as described above in FIG. 8B. Do not fit.

Reference is now made to FIGS. 8C-8E, which illustrate the optical system 800 of FIG. 8A at various working distances, generally configured and operable in accordance with embodiments of the disclosed technology. The top line 820 of FIGS. 8C-8E shows the image captured by the left camera 140A (FIG. 1E) and displayed by the display 130A (FIG. 1C), and the bottom line 822 of FIGS. ) and displayed by display 130B (FIG. 1C). Dashed line 824 in FIGS. 8C-8E indicates the object plane. Errors in the positioning of cameras 140A and 140B are exaggerated for illustrative purposes. FIG. 8C shows an optical system 800 with a camera located at a design working distance. Therefore, there is no discrepancy between the actual working distance and the design working distance. The images obtained with this configuration do not require any shifting to fully overlap as shown by overlap lines 820 and 822.

Referring to FIG. 8D, an optical system 800 is shown having cameras 140A and 140B (FIG. 1E) located below the design working distance, resulting in a discrepancy between the actual working distance and the design working distance. As a result of this mismatch, the images captured by each of cameras 140A and 140B and displayed via displays 130A and 130B, respectively, do not completely overlap. Straight line 820 representing the image captured by left camera 140A and displayed by display 130A does not completely overlap straight line 822 representing the image captured by right camera 140B and displayed via display 130B. As a result, overlapping parts in the two images appear to come from different directions, and the user's left and right eyes are squinted in an attempt to generate a combined 3D image in the brain. This can lead to eye strain unless the image is corrected, such as by using the shifting techniques described above.

Referring to FIG. 8E, an optical system 800 is shown having cameras 140A and 140B located above the design working distance, resulting in a discrepancy between the actual working distance and the design working distance. As a result of this mismatch, the images captured by each of cameras 140A and 140B do not completely overlap. Straight line 820 representing the image captured by left camera 140A and displayed by display 130A does not overlap straight line 822 representing the image captured by right camera 140B and displayed by display 130B. As mentioned above, if the shifting technique is not applied, the left and right eyes will squint in an attempt to generate a combined 3D image in the brain, resulting in eye strain.

In some embodiments, for example, if system 100 includes a z-axis motor to control the height of camera head 110, UI 160 may be configured to automatically adjust the height of camera head 110 to the design working distance. , presents an initialization option, i.e. an "Initial" or "Home" menu item. In embodiments where system 100 is not configured with a Z motor to control the height of camera head 110, the computer may Image processing techniques are applied and if the discrepancy is greater than an acceptable threshold, the surgeon 120 is notified to manually adjust the distance. For example, UI 160 displays a message guiding surgeon 120 how much, ie, how many centimeters or inches, to adjust the height either up or down. However, the surgeon 120 may choose to continue working at a different distance than the design working distance, for example to increase the imaging FOV (lower side is 3D with only the portion of the image overlap rather than all of the FOV). visible).

Another scenario that requires correction using a shifting technique is when the surgeon 120 uses a lens focus integrated into the cameras 140A and 140B to change the focal plane when the cameras 140A and 140B are properly positioned at the design working distance. This is the case when a motor is used. For example, when surgeon 120 switches between anterior and posterior segment modes, or focuses on different planes within the same mode, the discrepancy between the images acquired by cameras 140A and 140B can be significant enough to result in eye strain. It can be big. In such cases, the shifting techniques described above may be applied to match the images acquired by cameras 140A and 140B and reduce eye strain experienced by surgeon 120.

Reference is now made to FIG. 8F, which depicts the optical system 800 of FIG. 8A configured to focus on two different focal planes 830 and 832, constructed and operable in accordance with embodiments of the disclosed technology. The design working distance is indicated by dashed line 834. Focal plane 830 is located above design working distance 834 and focal plane 832 is located below design working distance 834. Top line 836 represents the image captured by left camera 140A and displayed via display 130A, and bottom line 838 represents the image captured by right camera 140B and displayed via display 130B. The top line 836 is shown shifted to the left relative to the bottom line 838, indicating a mismatch between the two acquired images that can be corrected using the shifting techniques described herein. Similar to focal plane 832, top line 840 represents the image captured by left camera 140A and displayed via display 130A, and bottom line 842 represents the image captured by right camera 140B and displayed via display 130B. represents. The top line 840 is shown shifted to the right relative to the bottom line 842 to indicate the mismatch between the two acquired images when cameras 140A and 140B are aligned to the focal plane 832. This mismatch may also be corrected using the shifting techniques described herein.

In the embodiments described above, the shift was made to allow comfortable stereoscopic viewing when the surgeon's eyes are looking forward. In some embodiments, system 100 may be configured to intentionally create a mismatch between two displayed images. The amount of shift may be uniquely tailored to each user of system 100. The displayed image may be shifted horizontally, vertically, or both. The shift may be unique to each of the surgeon's eyes. This shifting feature may be useful for reducing eye strain for surgeons who experience tropia (ie, misalignment of both eyes). The amount and direction of the shift may be configured to vary as a function of head orientation (as indicated by the tracker) for head orientation dependent oblique conditions.

In some embodiments, user-specific shifts may be configured during a one-time calibration process and saved for each user to be automatically implemented for subsequent use of system 100. Optionally, the surgeon may adjust the shift characteristics during the surgical procedure. Optionally, the head orientation-based shift is performed in stages and only after a predetermined period of time when the head orientation of the wearer of the HMD 102 has not changed. This is to prevent shifts in the displayed image in response to performing head gestures to activate other functions, which may disturb some users. In some cases, the displayed image may be non-uniformly shifted to compensate for possible oblique dependence in the viewing direction (eg, vertical viewing direction). For example, if tropia depends on the vertical viewing direction, the shift may vary continuously (not necessarily linearly) from the top of the image to the bottom. Optionally, if the HMD 102 enables a change in focus of the HMD optics, the amount and direction of the shift may depend on the focus. However, focus-dependent shifts may also be implemented for users who do not exhibit dysphoria. The shifts described above may also result in binocular competition effects that may require correction, as discussed below. Reference is now made to FIGS. 8G-8I, which illustrate images displayed via the HMD 102 after application of the shifting techniques described herein. Shift effects are exaggerated for clarity. FIG. 8G shows a 2D raw image 850 captured by camera 140A. The 2D raw image 850 has been shifted to the right according to the shifting techniques described herein, as indicated by the black border 852 displayed at the left edge of the raw image 850. FIG. 8H shows a 2D raw image 854 captured by camera 140B. The 2D raw image 854 has been shifted to the left according to the shifting techniques described herein, as indicated by the black border 856 displayed at the right edge of the raw image 854.

If the amount of shift performed by the shift technique is significant, binocular competition effects can occur. Referring to FIG. 8I, a representation of a 3D raw image 858 is shown. Image 858 is a combination of 2D raw image 850 of FIG. 8G and raw image 854 of FIG. 8H. Range 864 shows the portion of image 858 created from both images 850 and 854 and displayed by both displays 130A and 130B, respectively. Range 866 of image 858 shows the portion of image 858 that was created from image 854 only and displayed only by display 130B, and range 868 of image 858 shows the portion of image 858 that was created from image 854 only and displayed only by display 130A. 858 part is shown. The brain corrects for black borders 852 and 856 in images 850 and 854, respectively, causing surgeon 120 to see a significant contrast in the brightness of the areas delineated by dashed lines 860 and 862, and the brightness of image 858 plummets by half. do. The middle area of image 858 (range 864) is displayed by both displays 130A and 130B, whereas the left edge of image 858 (range 866) is displayed only by display 130B, and the right edge of image 858 (range 868) is displayed only by display 130A. displayed by. One solution is to gradually reduce the brightness of both images 850 and 854 in the areas near the "missing edges", i.e. 852 and 856, respectively, and reduce the brightness of both images 850 and 854 in the areas delineated by straight lines 866 and 868. The goal is to eliminate significant contrast. Since a linear gradient may be detectable by the brain, the reduction in brightness may follow a non-linear gradient.

Reference is now made to FIG. 9, which is a schematic illustration of a method for controlling a head-up display system, constructed and operative in accordance with other embodiments of the disclosed technology.

In procedure 900, a surgical field is illuminated and at least one image of the illuminated surgical procedure is acquired. Referring to FIG. 1A, the illumination system 114 of the head-mounted display system 100 illuminates the surgical field 124. Camera system 112 captures at least one image of illuminated surgical field 124.

At step 902, at least one image related to a surgical procedure is displayed to a surgeon via a head-mounted display. The image may be either at least one image acquired via camera system 112 and displayed in real time, a pre-operative image acquired in advance, or an image related to UI 160, such as a menu, overlay, etc. Referring to FIG. 1A, at least one image is displayed via head-mounted display 102.

At step 904, head gesture input is received in conjunction with foot motion input. Referring to FIG. 2A, head tracker 162 receives head gesture input by surgeon 120 (FIG. 1A) and foot switch 104 receives foot motion input from surgeon 120. Computer 118 receives head gesture input in conjunction with foot movement input.

In step 906, performing a first action at the head mounted display system when the head mounted display system is in a first system mode, and when the head mounted display system is in a second system mode; The head gesture input received in conjunction with the foot motion input is applied to perform a second action at the head mounted display system. Referring to FIG. 2B, the computer 118 uses foot motion inputs to perform a zoom out action 240 when the system mode is normal and to perform a scroll action 242 when the system mode is set to preoperative. Applying the associated received head gesture input.

In some embodiments, the first and second actions comprise either controlling a shutter coupled to a display module of the head-mounted display, and when the shutter is open, the display module at least When partially transparent and the shutter is closed, the display module is substantially opaque. Referring to FIG. 1C, shutter 132 is configured with display module 130 of head-mounted display 102.

In other embodiments, the first and second actions include controlling an image displayed via a head mounted display, controlling a camera system, controlling a lighting system, controlling a camera head positioner. It comprises either of the following: Referring to FIG. 2A, computer 118 (FIG. 1B) applies inputs received via UI input 160A to control components of UI output 160B; The system 114 controls the images displayed on the display module 130 via the HMD 102 and one of the camera head positioner 111 or the robotic arm.

In some embodiments, controlling the images displayed via the head-mounted display includes selecting images from the group consisting of live feed video, VGS video, and preoperative iOCT video, zooming in, and zooming in and out. scrolling between at least two virtual screens; displaying a picture-in-picture (PIP); displaying an overlay on top of the raw image; centering the image; displaying a menu; and controlling a region of interest in an image. The characteristics of images displayed via a head-mounted display may be controlled according to any suitable technique, such as digitally or optically, for example.

In some embodiments, controlling the camera system comprises controlling one or more optical or electrical characteristics of the camera system. For example, a camera may be selected to capture images, and the position and orientation of the camera system may be controlled. Referring to FIG. 1E, computer 118 (FIG. 1B) selects one of high resolution cameras 140A and 140B, IR camera 148, and iOCT scanner 142. Referring to FIG. 1A, computer 118 controls the position and orientation of camera system 112 via camera head positioner 111.

In some embodiments, controlling the lighting system includes selecting at least one of a plurality of illuminators, selecting an intensity setting, selecting a filter, and controlling the position and orientation of the lighting system. Comprising any of the following: Referring to FIG. 1F, computer 118 (FIG. 1B) selects between white floodlight 150, IR floodlight 152, and coaxial lights 154A and 154B. Referring to FIG. 1A, computer 118 controls the position and orientation of illumination system 114 via camera head positioner 111.

At step 908, a voice command by the surgeon is detected. Voice commands are applied to control the head-mounted display system. Referring to FIG. 1D, computer 118 (FIG. 1B) receives voice commands by surgeon 120 (FIG. 1A) via microphone 138. Computer 118 applies voice commands to control head mounted display system 100.

At step 910, eye movements by the surgeon are detected and the eye movements are applied to control a head-mounted display system. Referring to FIG. 1C, eye tracking component 136 tracks eye movements of surgeon 120 and provides the eye movements to computer 118 (FIG. 1B) via transceivers 102B and 118B.

At step 912, a head gesture input is applied to switch from a first system mode to a second system mode. Referring to FIGS. 2A-2B, computer 118 applies head gesture input received via UI input 160A to switch to vital system mode 244. In some embodiments, computer 118 applies head gesture input received in conjunction with foot motion input to switch modes.

In some embodiments, the image is displayed in a display-fixed state via the head-mounted display when the head-mounted display system is in a first system mode. The image is displayed in a world-fixed state via the head-mounted display on one of the plurality of virtual screens when the head-mounted display system is in a second system mode, and the second action Scroll the virtual screen. Referring to FIG. 5A, a display fixation state is shown. The video displayed to surgeon 120 does not change when he moves his gaze from forward (500) to 30 degrees to the left (502). Referring to FIG. 5B, a world fixation state is shown. The video displayed to surgeon 120 changes as he moves his gaze from 5 degrees to the right (504) to 25 degrees to the right (506). The content filling the field of view shifts as if the virtual screen were a fixed object in the operating room.

In some embodiments, a first region of the head-mounted display's field of view corresponds to a first system mode, and a second region of the head-mounted display's field of view corresponds to a second system mode. handle. Head gesture input applied to switch from the first system mode to the second system mode aligns the surgeon's line of sight with the second region. Referring to FIG. 3A, region 302 at the center of the field of view corresponds to normal system mode, and region 306 to the left of region 302 corresponds to preoperative system mode. Surgeon 120 may switch system modes by moving his line of sight from region 302 to region 306.

In some embodiments, a menu is displayed overlaid on an image displayed on a head-mounted display. The menu displays a first menu item relating to a first system mode in a first area of the field of view of the head mounted display and a second menu item overlaid in a second area of the field of view of the head mounted display. Displaying a second menu item regarding system mode. A head gesture input applied to switch from the first system mode to the second system mode aligns the surgeon's line of sight with the second menu item. Referring to FIGS. 4A-4C, a menu 400 is shown overlaid on a video of a surgical procedure. Menu 400 includes multiple menu items 402, 404, 406, 408, 410, and 412. Surgeon 120 may switch the system mode, for example, from highlighted menu item 402 in FIG. 4A to highlighted menu item 410 in FIG. 4B, by moving his or her gaze between menu items.

Those skilled in the art will understand that the disclosed technology is not limited to what has been specifically shown and described above. Rather, the scope of the disclosed technology is defined solely by the claims set forth below.

Claims (16)

A head-mounted display system (100) for use in surgical applications that allows different system responses to the same user input in different system modes , the head-mounted display system (100) comprising:
a head mounted display (102) configured to be worn by a surgeon;
a tracker (162) configured to track head gesture input by the surgeon;
a footswitch (104) configured to detect foot motion input by the surgeon;
a computer (118) coupled to the head-mounted display (102), the tracker (162), and the footswitch (104), the computer comprising:
displaying images related to a surgical procedure via the head-mounted display (102) ;
receiving head gesture input from the tracker (162);
receiving foot motion input from the footswitch (104) in conjunction with the head gesture input;
performing a first action in the head mounted display system (100) when the head mounted display system (100) is in a first system mode; 2, the head mounted display system (100) receives a head motion input in conjunction with the foot motion input to perform a second action different from the first action; applying a gesture input, wherein in both the first system mode and the second system mode, the head gesture input controls the head-mounted display system (100); and the foot motion input is the same in the first system mode and the second system mode ;
a computer configured to:
A head-mounted display system equipped with The head mounted display system of claim 1, wherein the tracker (162) is configured to track the head gesture input by tracking the head mounted display (102). The head-mounted display system (100) further comprises a shutter (132) coupled to the head-mounted display (102), and when the shutter (132) is open, the head-mounted display The head-mounted display system of claim 1, wherein (102) is at least partially transparent, and when the shutter (132) is closed, the head-mounted display (102) is opaque. The head mounted display system of claim 3, wherein one of the first action and the second action controls the shutter (132). a camera system (112) configured to capture said images; an illumination system (114) configured to operate with said camera system (112); a camera head positioner (111); and a robotic arm. The head-mounted display system of claim 1, further comprising a positioning mechanism selected from the group. At least one of the first action and the second action,
controlling characteristics of the image displayed via the head-mounted display (102);
controlling the camera system (112);
6. The head-mounted display system of claim 5, selected from the group consisting of: controlling the lighting system (114); and controlling the positioning mechanism. Controlling the characteristics of the image displayed via the head-mounted display (100) includes selecting content of the image, zooming in and out, scrolling between at least two virtual screens. displaying a picture-in-picture (PIP); displaying an overlay over a raw image; centering the image; displaying a menu; navigating a menu; 7. The head-mounted display system of claim 6, comprising an action selected from the group consisting of controlling a region. The computer (118) is further configured to display a menu overlaid on the image on the head-mounted display (102), and the computer (118) is configured to display a menu in the first system mode. The head-mounted display system according to claim 1, wherein the head-mounted display system displays a second menu in the second system mode. A method for interacting with a head-mounted display system (100) for use in a surgical system that allows for different system responses to the same user input in different system modes , the method comprising:
displaying images related to the surgical procedure via the head-mounted display (102) to a surgeon wearing the head-mounted display (102);
receiving head gesture input by the surgeon from a head tracker (162) in conjunction with foot motion input by the surgeon from a foot switch (104) ;
performing a first action in the head mounted display system (100) when the head mounted display system (100) is in a first system mode; applying head gesture input received in conjunction with the foot motion input to perform a second action at the head-mounted display system (100) when in the head-mounted display system (100); The first action is different from the second action, and in both the first system mode and the second system mode, the head gesture input controls the head-mounted display system (100) and controls the head-mounted display system (100). the gesture input and the foot movement input are the same in the first system mode and the second system mode ;
A method of providing. One of the first action and the second action comprises controlling a shutter (132) coupled to the head-mounted display (102), when the shutter (132) is open. 10. The method of claim 9, wherein the head mounted display (102) is transparent and when the shutter (132) is closed the head mounted display (102) is opaque. illuminating a surgical field (124) with a lighting system (114) configured with the head-mounted display system (100); and a camera system (112) configured with the head-mounted display system (100). 10. The method of claim 9, further comprising: obtaining an image of the surgical field (124) illuminated using a operative field (124). 12. The method of claim 11, further comprising controlling the position of the camera system (112) to acquire the images via a positioning mechanism selected from the group consisting of a camera head positioner (111) and a robotic arm. Method described. At least one of the first action and the second action,
controlling characteristics of the image displayed via the head-mounted display (102);
controlling the camera system (112);
13. The method of claim 12, selected from the group consisting of: controlling the illumination system (114); and controlling the positioning mechanism. Controlling the characteristics of the image displayed via the head-mounted display (102) includes selecting content of the image, zooming in and out, scrolling between at least two virtual screens. 14. The method of claim 13, comprising an action selected from the group consisting of: displaying a picture-in-picture (PIP); and displaying an overlay over the raw image. Controlling said camera system (112) is selected from the group consisting of: selecting a camera (140A, 140B) for acquiring said image; and controlling a position and orientation of said camera system (112). 14. The method of claim 13, comprising performing an action that has been performed. The method further comprises displaying a menu overlaid on the image on the head-mounted display, the displaying comprising: displaying a first menu in the first system mode; and displaying a second menu in the second system mode.

Images