JP2022509460A - UI for head-mounted display system - Google Patents

Abstract

HMDs configured to be worn by the surgeon, trackers configured to track the surgeon's head gestures, footswitches configured to detect foot movement inputs by the surgeon, and HMDs, trackers, It has a computer coupled to the footswitch and a user interface with an HMD, tracker, and footswitch to provide the computer with head gestures associated with foot movement inputs and display images of the surgical procedure on the HMD. In the configured HMD system, the computer performs the first action in the HMD system when the HMD system is in the first system mode, and the computer is in the HMD system when the HMD system is in the second system mode. A UI for an HMD system configured to apply a received head gesture in connection with a foot movement input to perform the action of 2.
[Selection diagram] FIG. 1A

Description

The disclosed techniques generally relate to head-mounted displays, in particular to methods and systems for controlling multiple system modes for head-mounted displays.

The head-mounted display is a wearable electronic display that projects an image into the wearer's field of view. In virtual reality applications, projected images generally block the wearer's field of view. In augmented reality applications, projected images are generally superimposed on a portion of the wearer's field of view, allowing the wearer to see real and virtual features at the same time.

Mr. Tabata's US Pat. No. 5,781,165, entitled "Image Display Apratatus of Head Mounted Type," discloses a user-controlled head-mounted display. While the head-mounted display is worn, the virtual image is projected in stereo on the LCD screen, and the wearer can see the three-dimensional virtual image. The wearer controls the display of the virtual image by pressing a button on the electronic controller. When the head-mounted display detects rotation and positional movement, for example when the wearer moves his or her head, the display of the virtual image is adjusted to appear stationary on the virtual image plane.

Mr. Yaskawa's US Pat. No. 5,977,935, entitled "Head-Mounted Image Display Device and Data Processing Apratatus Including the same," discloses a head-mounted device for displaying augmented reality. The wearer's field of view is divided into compartments. The wearer sees the real world in one section of his field of view. One or more virtual screens are projected in other sections of the wearer's field of view. By moving his gaze by moving his head, the wearer controls which virtual screen is displayed, much like moving a mouse to control a display. The foot pedal acts as an input device, much like a mouse click.

Mr. Yaskawa's US Pat. No. 6,320,559, entitled "Head-Mounted Image Display Device and Data Processing Applatus Including the same," discloses a head-mounted device for displaying virtual content. The wearer switches the content of the display by moving his / her line of sight. The wearer also controls the attributes of the display by voice input. The wearer can scroll the displayed image by moving his or her line of sight.

Reichern's US Pat. No. 6,396,497, entitled "Computer User Interface with Head Motion Input," discloses a wearable display system with a 360-degree view space. Each point in view space is identified by a particular yaw and pitch position. By moving his head, the wearer changes the display in the view window, which occupies only the (25 ° x 20 °) wedge in the 360 degree view space. Scrollbars displayed in the view window help the wearer keep track of their current position in view space.

Lemelson's US Pat. No. 6,847,336, entitled "Selectively Controlled Heads-Up Display System," is a surgeon's wear with multiple user interfaces that allow the wearer to control the display. Disclose the type display. The system incorporates either eye tracking to move the cursor over the display screen and either a foot pedal or voice command to perform a "mouse click" selection.

Miller's US Pat. No. 7,127,401, entitled "Remote Control of a Medical Device Using Speech Recognition and Foot Controls," discloses a voice-controlled medical imaging device for use by surgeons. Controlling the medical imaging device by voice frees the surgeon's hands to perform the medical procedure.

Bar-Zev's US Pat. No. 8,941,559, entitled "Opacity Filter for Display Device," discloses a wearable augmented reality display. Each pixel of the display can be individually adjusted for levels from maximum opacity to transparency, allowing light from the real world scene to be blocked or transmitted accordingly. The controller controls the opacity of the pixels specified to display the virtual image superimposed on the real world scene.

Raffle's U.S. Pat. No. 9,285,872, entitled "Using Head Gesture and Eye Position to Work a Head Mounted Device," provides gesture recognition and eye tracking to switch operating modes for wearable display units. Disclose to incorporate.

Bar-Zev's US Pat. No. 9,286,730, entitled "Opacity Filter for Display Device," discloses a wearable augmented reality display. Each pixel of the display can be individually adjusted for levels from maximum opacity to transparency, allowing light from the real world scene to be blocked or transmitted accordingly. The position of the virtual image superimposed on the real world scene is controlled by eye tracking.

Abdollahi's US Pat. No. 9,292,084, entitled "Control Systems and Methods for Head-Mount Information System," uses gesture control and preconfigured gesture profiles to control the display of wearable devices. Disclose what to do. The gesture may be a hand or head gesture. Buttons are provided to turn gesture control features on and off.

Henelly's US Pat. No. 9,383,816, entitled "Text Selection Using HMD Head-Tracker and Voice-Command," discloses a hand-free text selection for head-mounted displays. The wearer of the head-mounted display selects the displayed text by the combination of the head and the hand gesture.

"Head-Mounted Display, Project for Controlling Head-Mounted Display, and Method of Control Head-Mounted Display", Mr. Kriya's U.S. Patent No. 9, 523, 528 Disclose the technology to improve the usefulness of. The image to be displayed on the HMD is selected based on the detection that the direction of the HMD is outside the wearer's line of sight.

Mr. Moravetz's US Pat. No. 9,588,343, entitled "Meu Navigation in a Head-Mounted Display," provides a virtual menu by tracking the position and orientation (P & O) of the HMD with respect to the focal point of the HMD. Disclose how to navigate.

Abdollahi's US Pat. No. 9,696,797, entitled "Control Systems and Methods for Head-Mounted Information System," discloses a method for navigating virtual menus through gesture control.

Fateh's U.S. Patent No. 9,804,669, entitled "High Resolution of Concept in a Wide field of View of a Head-Mounted Display," is a virtual image based on the wearer's focal point. Disclose a method for controlling. The resolution of the display area currently within the user's focal point is then increased along with the display area that is expected to be within the wearer's focal point. The resolution of the display area between these two focus areas is reduced.

Staffold's US Pat. No. 9,897,805, entitled "Image Rendering Responsive to User Actions in Head Mounted Display," incorporates eye tracking to predict the next wearer's gaze movement. Disclose a part-mounted display. The quality of the rendered image is adjusted based on the predicted movement.

Chung's U.S. Patent Publication No. US20120027373 A1, entitled "Head-Mounted Display Device Having Interactive Function and Method Thereof," is a head-mounted display based on the detected movement of a display device. To disclose.

Parkinson's U.S. Patent Publication No. US20150220142 A1 using a cursor on a head-tracking device, entitled "Head-Tracking Based Technology for Moving On-Screen Objects on Head Mounted Display (HMD)" Disclose how to control.

Mr. Vargas, US Patent Publication No. US20150346813 A1, entitled "Hands Free Image Viewing on Head Mounted Display," discloses a head gesture-based user interface for controlling the display of HMD devices. Display features that can be controlled via head gestures include zooming in and out of the display image, panning the display image, scrolling along a series of display images, and switching between different modes of operation.

"Optical Head Mounted Display, Television Portable Module and Methods for Controlling Graphical User Interface" Discloses entering a mode for controlling.

Zhang's U.S. Patent Publication No. US201702424995 A1 is a virtual mouse for head-mounted display, entitled "Method and Device of Controlling Visual Mouse and Head-Mounted Displaying Device". Disclose that the detection is incorporated.

Nazareth's US Patent Publication No. US20170273549 A1, entitled "Portable Surgical Methods, System, and Apparatus," is a camera for acquiring raw images of surgical procedures, for displaying a raw feed of the acquired images. Disclosed is a kit for accommodating surgical equipment, including a wireless headset, voice-operated control of the system, and a computer for storing data about the surgical procedure.

Abdollahi's US Patent Publication No. US201801113507 A1, entitled "Control Systems for Head-Mounted Information System," discloses a gesture control mode for a head-mounted display (HMD) device. Gesture control mode is enabled when the gesture control enable signal is detected and allows the user to navigate the menu through the gesture.

The object of the disclosed technology is to provide new methods and systems for user interfaces for head-mounted display systems.

According to the disclosed technology, a head-mounted display configured to be worn by the surgeon, a tracker configured to track the surgeon's head gesture input, and a surgeon's foot movement input to be detected. A user interface that includes at least a foot switch configured in, a computer coupled to a head-mounted display, a tracker, and a foot switch, and a head-mounted display, a tracker, and a foot switch. A computer provides a head gesture input received from the tracker in connection with the foot movement input received from the switch, and a user interface configured to display images related to the surgical procedure on a head-mounted display. When the head-mounted display system is in the first system mode, the first action is taken in the head-mounted display system, and when the head-mounted display system is in the second system mode, the head-mounted display system is mounted on the head. The type display system is configured to apply the received head gesture input in connection with the foot movement input to perform a second action.

In some embodiments, the tracker is at least partially integrated with the head-mounted display and is configured to track the head-mounted display.

In some embodiments, the tracker is an external camera for a head-mounted display.

In some embodiments, the system further comprises a shutter coupled to the head-mounted display, where the head-mounted display is at least partially transparent and the shutter is closed when the shutter is open. In the case, the head-mounted display is substantially opaque.

In some embodiments, one of the first and second actions controls the shutter.

In some embodiments, the user interface further comprises a microphone configured to detect voice commands, and the input further comprises voice commands.

In some embodiments, the user interface further comprises an eye tracker configured to detect eye movements by the surgeon, and the input further comprises eye movements.

In some embodiments, the system is selected from a group consisting of a camera system configured to capture images, a lighting system configured to work with the camera system, and a camera head positioner and robot arm. It is further equipped with a positioning mechanism.

In some embodiments, the first action and the second action control the characteristics of the image displayed via the HMD, control the camera system, control the lighting system, and the positioning mechanism. Is selected from a group consisting of controlling.

In some embodiments, controlling the characteristics of an image displayed via the HMD is to select the content of the image, zoom in and out, scroll between at least two virtual screens, and picture. From displaying in-pictures (PIPs), displaying overlays on top of raw images, centering images, displaying menus, navigating menus, and controlling areas of interest in images. It has an action selected from the group consisting of.

In some embodiments, controlling a camera system comprises performing an action selected from a group consisting of controlling optical and electric camera characteristics and controlling the position and orientation of the camera system.

In some embodiments, controlling a lighting system is to select at least one of a plurality of illuminators, select an intensity setting, select a filter, and turn lighting on or off. Be prepared to perform an action selected from a group of.

In some embodiments, the image of the surgical field is displayed in a fixed state via the head-mounted display when the head-mounted display system is in the first system mode, and the video of the surgical field is displayed. When the head-mounted display system is in the second system mode, it is displayed in a world-fixed state on one of a plurality of virtual screens via the head-mounted display.

In some embodiments, the computer is further configured to apply a head gesture input to switch from the first system mode to the second system mode.

In some embodiments, the first region of the head position and orientation range of the wearer of the head-mounted display corresponds to the first system mode and the head of the wearer of the head-mounted display. The second area of the position and orientation range corresponds to the second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is the head-mounted display. The position and orientation of the wearer's head is changed from the first region to the second region.

In some embodiments, the computer is further configured to display an overlaid menu on the image on a head-mounted display, the menu displaying the first menu item with respect to the first system mode. Display a second menu item for the second system mode.

According to another aspect of the disclosed technique, a method for interacting with a head-mounted display system is provided in which the surgeon is presented with an image of the surgical procedure on the head-mounted display. Receiving the surgeon's head gesture input from the head tracker in connection with the surgeon's foot movement input from the footswitch and the head-mounted display when the head-mounted display system is in the first system mode. Received in connection with foot movement input to perform a first action in the system and to perform a second action in the head-mounted display system when the head-mounted display system is in the second system mode. It is provided with applying the said head gesture input.

In some embodiments, either the first action and the second action comprises controlling a shutter coupled to a head-mounted display, if the shutter is open, the head-mounted display. Is transparent and the head-mounted display is opaque when the shutter is closed.

In some embodiments, the method further comprises detecting a voice command by the surgeon and applying the voice command to control a head-worn display system.

In some embodiments, the method further comprises detecting eye movements by the surgeon and applying eye movements to control a head-worn display system.

In some embodiments, the method was illuminated using a lighting system configured with a head-mounted display system to illuminate the surgical field and a camera system configured with a head-mounted display system. Further prepare for acquiring an image of the surgical field.

In some embodiments, the method further comprises controlling the position of the camera system to acquire an image via a positioning mechanism selected from the group consisting of a camera head positioner and a robot arm.

In some embodiments, the first action and the second action control the characteristics of the image displayed via the head-worn display, control the camera system, control the lighting system. , And a group consisting of controlling the positioning mechanism.

In some embodiments, controlling the characteristics of an image displayed through a head-mounted display is to select the content of the image, zoom in and out, and scroll between at least two virtual screens. It comprises an action selected from a group consisting of doing, displaying a picture-in-picture (PIP), and displaying an overlay on top of the raw image.

In some embodiments, controlling a camera system performs an action selected from a group consisting of selecting a camera for acquiring an image and controlling the position and orientation of the camera system. Be prepared.

In some embodiments, controlling a lighting system is to select at least one of a plurality of illuminators, select an intensity setting, select a filter, and turn lighting on or off. Be prepared to perform an action selected from a group of.

In some embodiments, the image is displayed in a fixed state via the head-mounted display when the head-mounted display system is in the first system mode, and the image is displayed in a head-mounted display system. Is displayed in a world-fixed state on one of a plurality of virtual screens via a head-mounted display when is in the second system mode.

In some embodiments, the method further comprises applying a head gesture input received in connection with a foot movement input to switch from a first system mode to a second system mode.

In some embodiments, the first region of the head position and orientation range of the wearer of the head-mounted display corresponds to the first system mode and the head of the wearer of the head-mounted display. The second area of the position and orientation range corresponds to the second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is the head-mounted display. The position and orientation of the wearer's head is changed from the first region to the second region.

In some embodiments, the method further comprises displaying an overlaid menu on an image on a head-mounted display, wherein the menu displays a first menu item with respect to a first system mode, the first. Display a second menu item for the second system mode, and head gesture input and foot movement input from the first system mode to the second system mode by selecting and activating the second menu item. Applies to switch.

The disclosed technology is understood and recognized in more detail from the following detailed description taken up in connection with the drawings.

FIG. 3 is a schematic diagram of a system for controlling multiple modes of operation for a head-mounted display (HMD) coupled to a camera head, configured and operable according to embodiments of the disclosed technique. FIG. 3 is a schematic block diagram of a computer configured and operational according to other embodiments of the disclosed technology. FIG. 3 is a schematic block diagram of the HMD of FIG. 1A, configured and operable according to a further embodiment of the disclosed technique. FIG. 3 is a schematic block diagram of the camera head of FIG. 1A, configured and operable according to other embodiments of the disclosed technology. FIG. 3 is a schematic block diagram of the camera system of FIG. 1A, configured and operable according to a further embodiment of the disclosed technology. FIG. 3 is a schematic block diagram of the lighting system of FIG. 1A, configured and operable according to other embodiments of the disclosed technology. FIG. 3 is a schematic block diagram of the footswitch of FIG. 1A, configured and operable according to a further embodiment of the disclosed technique. FIG. 3 is a conceptual diagram of a user interface for system 100 that is configured and operational according to other embodiments of the disclosed technology. It is a conceptual diagram of the flow of information and control about the system of FIG. 1A via the user interface of FIG. 2A, which is configured and operational according to a further embodiment of the disclosed technology. Shown is a typical layout for multiple P & O regions of the HMD, each mapped to different system modes, configured and operational according to other embodiments of the disclosed technique. Shown is an overlaid raw image display with hints indicating which head gestures configured and operational according to embodiments of the disclosed technique call the zone and associated system modes described with respect to FIG. 3A. Overall, it presents a menu for switching to various system modes overlaid on the video displayed via the HMD, configured and operational according to further embodiments of the disclosed technology. Shown is an implementation for displaying one or more status bars indicating system settings that are configured and operational according to other embodiments of the disclosed technology. Shows a projected image as perceived by the surgeon when the display module is in a fixed display state, configured and operable according to other embodiments of the disclosed technique. Overall, it presents two views, each configured and operational according to a further embodiment of the disclosed technique, each displayed in a world-fixed state on a virtual screen via an HMD, each view being a variety of surgeon's heads. Corresponds to various positions and orientations. Demonstrates a typical implementation of a surgeon-visible virtual screen state that is configured and operational according to other embodiments of the disclosed technology. Shown is an overlaid menu on a raw image feed that is configured and operational according to a further embodiment of the disclosed technology, allowing the user to interact with the system and switch to various system modes. An overlayed system mode menu is shown on the raw image feed after selecting a system mode menu item from the menu of FIG. 6A, which is configured and operational according to other embodiments of the disclosed technique. Shown is a preoperative image displayed within an overlaid PIP on a raw image feed that is configured and operational according to a further embodiment of the disclosed technique. Shown is a menu of FIG. 6A overlaid on a raw image feed with PIP, configured and operational according to other embodiments of the disclosed technique. Overall, it shows a series of raw images with the corresponding content displayed within PIP, configured and operational according to other embodiments of the disclosed technology. Overall, it shows a series of raw images with the corresponding content displayed within PIP, configured and operational according to a further embodiment of the disclosed technique. FIG. 6 shows raw images of FIGS. 6L-6P with corresponding content displayed within two PIPs that are configured and operational according to a further embodiment of the disclosed technique. Overall, an implementation for drawing symbols on a raw image display that is configured and operational according to embodiments of the disclosed technique is shown. Shown is an overlayed menu displayed on a raw image that is configured and operational according to an embodiment of the disclosed technique. Overall, we show a typical implementation of a user interface screen that allows you to specify a field of view area selected for focus that is configured and operational according to a further embodiment of the disclosed technology. Shown is another view of the menu of FIG. 7A overlaid on a raw image that is configured and operational according to embodiments of the disclosed technique. Overall, we show another typical implementation of a menu-driven user interface for focusing on a selected field of view area that is configured and operational according to other embodiments of the disclosed technology. As a whole, techniques for correcting 3D image distortion caused by a discrepancy between the actual working distance and the design working distance of the camera, which is configured and operable according to the disclosed techniques, are shown. Overall, it shows the optical system of FIG. 8A at various working distances that is configured and operable according to embodiments of the disclosed technique. 8A shows an optical system of FIG. 8A configured to focus on two different focal planes configured and operable according to embodiments of the disclosed technique. Overall, an image displayed via the HMD is shown after application of the shift technique described according to FIGS. 8A-8B, which is configured and operational according to embodiments of the disclosed technique. FIG. 3 is a schematic representation of a method for controlling a heads-up display system that is configured and operational according to other embodiments of the disclosed technology.

The disclosed technique overcomes the shortcomings of prior art by providing a multimodal user interface (UI) for a head-mounted display (HMD) for use in surgical applications. An example of such a surgical application is a microsurgical procedure performed under the magnification of a surgical microscope. Microscopic surgery is commonly used in ophthalmic surgery and brain surgery. In general, in microsurgery procedures, the surgeon's hands are occupied by surgical tools, so the surgeon needs a wide range of hand-free methods to interface with the surgical microscope and control system parameters. The more robust the system and the wider the range of features the system offers, the more complex the interface for controlling these features. An additional challenge in providing an interface using such a hands-free method shifts complexity to other input means, such as footswitches. In fact, surgeons often need to use complex footswitches to control various system parameters, taking off their shoes and their toes to fumble for buttons and pedals while gazing at the surgical field. Must be used.

The UI of the disclosed technology allows surgeons to use relatively simple footswitches while providing a wide range of functions via different system modes, each supporting different action sets. In one implementation, the footswitch has only three pedals, allowing the surgeon to easily find the appropriate pedal while wearing shoes and without lifting the heel off the floor. A given hands-free gesture can trigger a number of different actions, depending on the system mode. This technique provides surgeons with a wide range of functions with a relatively small number of hands-free gestures. Hands-free gestures are designed to be intuitive, allowing the surgeon to naturally interface with the system and control various features, allowing the surgeon to perform most of his attention as well as his hands in the surgical procedure. Makes it possible to offer. The surgeon may switch the system mode with one or more hands-free inputs such as motion sensing, foot or voice control, eye tracking, and the like. In some embodiments, the surgeon switches the system mode with a footswitch-enabled head gesture.

The system described herein includes a plurality of components such as a camera system and a lighting system for acquiring images of the surgical field, and an HMD for viewing these images. The different system modes allow the surgeon to control different aspects of the system. Some system modes allow the surgeon to use gestures to control system parameters such as magnification settings (ie zoom in and zoom out), focus and lighting settings for the camera head unit located above the surgical field. enable. Other system modes allow the surgeon to use gestures to control the display via the HMD, for example by selecting content for viewing and manipulating displayed images and preoperative data. enable.

The methods described herein may also be applied to surgical procedures other than microsurgery, i.e., magnified images of the surgical field, such as visor-guided surgery based on follow-up and augmented reality capabilities (here "VGS"). It can be done even if it is not. In these procedures, the HMD expands the surgeon's view of the patient, allowing the surgeon to see the anatomical features and surgical tools as if the patient's body were partially transparent. These procedures can optionally be performed without any magnified image of the surgical field and thus without the camera head unit.

Reference is made here to FIG. 1A, which is a schematic diagram of a system for controlling a plurality of operating modes for an HMD, which is configured and operational according to embodiments of the disclosed technique and is shown as 100 as a whole. .. The system 100 includes a cart 116, a camera head 110, and a camera head positioner 111 that accommodate the HMD 102, foot switch 104, mechanical arm 106, touch screen 108, computer 118 (not shown) and support the touch screen 108. include. The camera head 110 houses the camera system 112 and the lighting system 114. The camera system 112 includes at least two high resolution cameras (not shown). The mechanical arm 106 connects the computer 118 to the camera head positioner 111 and the camera head 110. The computer 118 includes a camera head positioner 111, a camera head 110, a camera system 112, and a lighting system 114 via one or more wires and / or cables (not shown) integrated within the cart 116 and the machine arm 106. Is electrically coupled to.

The computer 118 controls the position and orientation of the camera head 110 via the camera head positioner 111. The camera head positioner 111 includes a plurality of motors for controlling the position of the camera head 100 in the x, y, and z coordinates. The camera head positioner 111 further includes a motor for controlling the tilt of the camera head 100. The computer 118 also controls operating parameters for the lighting system 114 and the camera system 112, the details of which will be described later.

The HMD 102, footswitch 104, and computer 118 are provided with one or more transceivers (not shown). Transceivers include, for example, electrical or fiber optic cables (not shown), Wifi, Bluetooth, Zigbee, short-range, medium-range, long-range, and microwave RF, wireless optical means (eg laser, lidar, infrared), acoustic means, ultra. Corresponds to any suitable wired and / or wireless communication means and protocols, such as sonic means. The computer 118 is coupled to the HMD 102 and the foot switch 104 via a transceiver. The computer 118 streams the image to the HMD 102 via a transceiver, allowing the surgeon 120 to view the image on the HMD 102. The images are acquired by the camera system 112 and can be processed in real time by the GPU to watch live video of the surgery. Alternatively, the image can be captured prior to surgery, stored in memory and rendered in real time by the GPU. In some implementations, images can be streamed or downloaded from remote servers, such as cloud-based servers. For example, in a VGS procedure, a model of the body part generated by division from a CT or MRI image can be stored in memory and rendered in real time by the GPU to the HMD 102 from the perspective of the surgeon 120. In another embodiment, the image transmitted to the HMD 102 is obtained from an external device, such as an endoscope. The computer 118 determines which image to stream to the surgeon 120 based on the current system mode as well as one or more inputs received from the surgeon via the user interface.

In one embodiment, one surgeon 120 wears the HMD 102 to watch a magnified video of the surgical field 124 while performing a surgical procedure on the patient 122. The camera system 112 acquires a stream of images of the surgical field 124 corresponding to the surgical procedure performed on the patient 122. Computer 118 receives and processes a stream of images and sends the processed images to the HMD 102 via a transceiver. The surgeon 120 sees the image through the HMD 102 and controls one or more settings of the system 100 depending on the system mode. Each system mode may control different sets of system parameters. The surgeon 120 performs handless gestures such as head gestures while pressing the foot switch 104, eye movements tracked by the eye tracker 136, or by using acoustically driven means such as voice control. , The system mode may be switched to control the set of selected system parameters. Details of these user interface methods and other user interface methods will be described later herein.

Alternatively, the surgeon 120 may directly control one or more system parameters using a handless method, such as performing a head gesture while pressing the footswitch 104. The same gesture may control various features of the system 100 depending on the system mode. For example, pressing the pedal of the footswitch 104 while performing a head gesture can cause one action when the system 100 is in one system mode, but the footswitch when the system 100 is in another system mode. Pressing the same pedal on 104 and performing the same head gesture can cause different movements. By providing multiple system modes, a relatively small number of pedals and head gestures can control a wide range of features.

For example, in one system mode, the surgeon 120 may zoom in by pressing the first pedal of the footswitch 104 while moving his head upwards. As a result, the surgeon 120 sees a small size surgical field 124 that is magnified over a larger portion of his field of view. By pressing the same pedal on the footswitch 104 while performing the same upward head gesture in different system modes, the surgeon 120 can scroll upward in multiple preoperative images displayed via the HMD 102. .. Gestures for each system mode are intended to be intuitive to allow the surgeon 120 to naturally interface with the system 100. Thus, the system 100 provides a wide range of controllable features, but controlling these features by the user interface disclosed herein does not require a redundant training period. Rather, the surgeon 120 makes gestures that are intuitive to the features he desires to control. For example, when viewing a plurality of images displayed across a plurality of virtual screens, the surgeon 120 can scroll to the left and see the screen displayed to the left by turning his head to the left. Control through head gestures can provide excellent accuracy and resolution. For example, as some surgeons will notice, head gestures allow for more accurate one-shot control and eliminate overshoots or undershoots, so controlling focus through head gestures is possible with a footswitch. Turns out to be easier compared to. In some embodiments, the trajectory of the head gesture determines what is done, and the speed of the head gesture determines the speed at which the movement is performed. For example, a surgeon may zoom in quickly by moving his head quickly and zoom in slowly by moving his head slowly.

In some embodiments, the footswitch 104 is a button as well as a method of activating a separate feature set for the keyboard by pressing the <Shift>, <Cntl>, <Caps Lock> buttons on a standard keyboard. Equipped to activate other feature sets via press. In some embodiments, the second feature set is activated simultaneously for all buttons and features of the footswitch 104 in a single action by the surgeon 120. For example, pressing one of the buttons on the footswitch 104 (eg, the button corresponding to the menu) once briefly and then releasing that button activates a second feature set for all the buttons on the footswitch 104. .. The indication that the second feature set has been activated is displayed to the surgeon 120 via the HMD 102. Alternatively, the second feature set may be activated by voice command or by other means.

In another embodiment, the second feature set is individually activated for each button on the footswitch 104. Therefore, pressing one of the buttons of the footswitch 104 briefly once, releasing it immediately, and then pressing and holding the same button may activate the second head gesture set for that button only. For example, a button that enables XY motor control by a head gesture without a "shift" can enable up / down and left / right panning of the ROI when "shifted". In some embodiments, one or more buttons on the footswitch 104 are equipped to distinguish between full press and half press to enable various feature sets. For example, one of the buttons on the footswitch 104 may control the lighting with respect to the system 100, a full press of the button may control one aspect of lighting, eg coaxial lighting, and a half press of the button, for example. Control other aspects of lighting, such as floodlights (for example, a full press activates a head gesture that controls coaxial lighting, and a half press activates a head gesture that controls floodlights).

In another embodiment, the system 100 supports several HMDs simultaneously. One of the users, eg the surgeon 120, is designated as the "representative" surgeon who can control all system settings. In some embodiments, other users wearing an HMD similar to the HMD 102 control only the settings of the image displayed via their respective HMDs, but of the HMD they are wearing. They do not control external parameters, for example they do not control the focus of the camera system 112 and the lighting by the lighting system 114. In another embodiment, another user wearing an HMD similar to the HMD 102 may control one or more settings with respect to the system 100.

The description of the system 100 shown below is described from two perspectives. A description of a typical hardware layout for the components of the system 100 will be described below with respect to FIGS. 1B-1G. 2A-2B illustrate a typical logical layout for the user interface of the system 100. It should be noted that these figures are intended to illustrate only a typical implementation of one of a plurality of possible implementations for the system 100. These figures are not intended to limit the invention to any particular implementation.

Here, reference is made to FIG. 1B, which is a schematic block diagram of computer 118 of FIG. 1A, which is configured and operational according to other embodiments of the disclosed technology. It should be noted that this figure is just a typical implementation, where software modules may be used in place of hardware modules and vice versa. The computer 118 includes at least one processor 118A, at least one transceiver 118B, power supply 118C, at least one memory 118D, at least one analog digital A / D converter 118E, at least one digital analog D / A converter 118F, and , Mechanical arm controller 118G, camera system controller 118H, lighting system controller 118I, head tracker controller 118J, image processor 118K, eye tracker controller 118L, acoustic analyzer 118M, and one or more of at least one bus 118N. Processor 118A, Transceiver 118B, Power supply 118C, Memory 118D, A / D converter 118E, D / A converter 118F, Mechanical arm controller 118G, Camera system controller 118H, Lighting system controller 118I, Head tracker controller 118J, Image processor 118K, The eye tracker controller 118L and the acoustic analyzer 118M are coupled via at least one bus 118N.

Computer 118 is shown as a single unit, but this is for conceptual purposes only. Computer 118 may be implemented as a plurality of distributed units operating as a single control unit for system 100. For example, each of at least one processor 118A, transceiver 118B, power supply 118C, and memory 118D is distributed and cooperates with the components of the system 100 (eg, camera head 110, HMD 102, foot switch 104, remote server, etc.). May include multiple processors, memory, power supplies, and transceivers.

The mechanical arm controller 118G controls a motor contained within the camera head positioner 111 to adjust the x, y, z axis or tilt of the camera head 110. The camera system controller 118H controls the operation of the cameras included in the camera system 112, such as selecting a camera or a plurality of cameras for acquiring an image, adjusting an F value, a focus, and the like. The lighting system controller 118I controls the operation of the lighting system 114, such as selection of illuminator, intensity, one or more filters, and the like. The head tracker controller 118J controls the operation of one or more head tracker components provided to the system 100 to track the head of the surgeon 120. The image processor 118K can use, for example, zoom in, zoom out, digital filtering, sharpening, smoothing, color correction, fusing several image sources, embedding one image source in another image source with a picture-in-picture form, etc. Depending on the application, the image received from either the camera system 112, the memory 118D, or the GPU is processed. The eye tracker controller 118L controls the eye tracker configured by the HMD 102. The acoustic analyzer 118M applies sound recognition to the sound received from the microphone provided to the system 100, for example by applying a voice recognition algorithm to recognize voice commands.

Here, reference is made to FIG. 1C, which is a schematic block diagram of HMD 102 of FIG. 1A, which is configured and operational according to a further embodiment of the disclosed technique. The components of the HMD 102 may be incorporated within an adjustable frame fitted by the surgeon 120, as shown in FIG. 1A. The HMD 102 includes a driver 102A, a transceiver 102B, a memory (eg, a buffer) 102C, and a display module 130. In some embodiments, the HMD 102 further includes a liquid crystal (LC) shutter 132, a tracking component 134, an eye tracking component 136, a microphone 138, a speaker 139, a vibration motor 141, and at least one bus 102D. The driver 102A, transceiver 102B, memory 102C, display module 130, shutter 132, tracking component 134, eye tracking component 136, microphone 138, speaker 139, and vibration motor 141 are coupled via at least one bus 102D.

The display module 130 may be a binocular display comprising two display modules 130A and 130B for projecting an image for each eye of the surgeon 120. In some embodiments, the HMD 102 is monocular. The shutter 132 is mechanically coupled to the display module 130 and can operate to open and close. When the shutter 132 is closed, the display module 130 is opaque and enhances the image displayed via the HMD 102. When the shutter 132 is fully or partially open, the display module 130 is transparent in the area where the shutter 132 is open, allowing the surgeon 120 to see the overlaid image on top of the real world view. do. The partially or fully open shutter 132 may be suitable for augmented reality or real world view configurations. Alternatively, the open shutter 132 allows the surgeon 120 to interact and communicate with other surgeons in the operating room, for example, other surgeons or nurses, as needed. The tracking component 134 acquires position and / or orientation data with respect to the HMD 102 and transmits the location and / or orientation data to the head tracker controller 118J via the transceivers 102B and 118B, respectively. The microphone 138 records a voice command or other sound issued by the surgeon 120 and transmits the sound to the acoustic analyzer 118M via the transceivers 102B and 118B, respectively. The eye tracking component 136 acquires eye movement data of the surgeon 120 and transmits the eye movement data to the eye tracker controller 118L via the respective transceivers 102B and 118B.

The display module 130 is arranged in the HMD 102 so as to be located in the line of sight of the surgeon 120, as shown in FIG. 1A. The shutter 132 is configured by the display module 130 and can be switched between an open state, a closed state, and a partially closed state. The HMD 102 is always transparent when the shutter 132 is open and opaque when the shutter 132 is closed. When the shutter 132 is fully open, the surgeon 120 sees the real world through the optical system of the transparent display module 130. The shutter 132 may be partially closed to hide the background behind the features displayed in the picture-in-picture (PIP) overlaid on the display module 130, for example, the features and other features of the display module 130. Increases the contrast with the real-world view seen through the part. This is useful for displaying one or more features in PIP during VGS treatment in augmented reality applications. The display module 130 may be implemented using any suitable technique as known in the art. The display module 130 may include, for example, an image projection optical system such as a waveguide optical system, a prism optical system, and projection onto a visor, as well as an image source such as a small OLED monitor. Alternatively, the display module may scan the image directly into the retina of the surgeon 120 without a monitor. The HMD 102 receives an image source from the computer 118.

The tracking component 134 includes components for tracking the head movements of the surgeon 120. The tracking component 134 may include an MEM based on an accelerometer 134A and a gyroscope 134B, one or more light emitters 134C, an optical sensor 134D, an electromagnetic sensor 134E, and the like. In some embodiments, the tracking component 134 further includes a compass (not shown). The light emitter 134C may be one or more active light emitting elements such as light emitting diodes (LEDs), or one or more light reflectors. Tracking component 134 generates translational and rotational motion data for the HMD 102 to determine updated orientation and / or position coordinates for the HMD 102.

In some embodiments, head tracking is performed outside the HMD 102, for example by using a camera (not shown). Optionally, the tracking component 134 includes components outside the HMD 102 as well as components inside the HMD 102. In some embodiments, the HMD 102 comprises an eye tracker component 136 for tracking the line of sight of one or both eyes of the surgeon wearing the HMD 102. In some embodiments, the microphone 138 is integrated into the HMD 102 to record the voice commands of the surgeon 120. Alternatively, the microphone 138 may be external to the HMD 102 and integrated into, for example, a camera head 110. It should be noted that the embodiments described herein are intended as merely exemplary and any suitable combination and configuration components are used to achieve the functions disclosed herein. obtain. The computer 118 controls the tracking component 134 via the tracker controller 118I. The computer 118 communicates with the HMD 102 via the transceivers 118B and 102B and displays the image feed on the HMD display module 130 according to the selected system mode and additional settings and parameters defined by the surgeon 120, which will be described in more detail below. To control.

Reference is made to FIG. 1D showing a schematic block diagram of the camera head 110 (FIG. 1A) configured and operable according to other embodiments of the disclosed technique. The camera head 110 houses the camera system 112, the lighting system 114, and optionally the microphone 138. In some embodiments, the camera head 110 further houses a tracking subsystem (not shown) that includes sideways and / or downward optical tracking components, sensors and / or illuminants, and optionally other tracking components. The camera head 110 is mechanically coupled to the cart 116 via the arm 106 and the camera head positioner 111. The camera head 110, camera system 112, lighting system 114, and microphone 138 are electrically coupled to computer 118 (FIG. 1B) by one or more cables and / or wires integrated within the arm 106. The computer 118 controls the xyz position and tilt of the camera head 110 with respect to the surgical field 124 by controlling a motor integrated in the camera head positioner 111.

In some embodiments, for example a robotic arm, eg, another positioning mechanism such as a 6-DOF (degree of freedom) robotic arm commonly used for neurosurgical applications, to obtain an image of the surgical field 124. Can be used in place of the camera head positioner 111 and the mechanical arm 106 to control the position of the camera system 112. Unlike ophthalmic surgery applications, where camera position and orientation changes are subtle, camera position and orientation changes are relatively large in neurosurgery applications. For such applications, the surgeon 120 activates the ability to secure control of the camera to his head and he uses a head gesture to control the position and orientation of the camera on the camera head 110 via the robot arm. It may be possible to do. The surgeon 120 activates this feature, for example, through a user interface such as the footswitch 104, or by issuing voice commands. The head tracking function of the HMD 102, which will be described in detail later, senses the rotational and translational movements of the HMD 102 and converts them into the corresponding rotational and translational movements for the robot arm controlling the camera head 110. Optionally, the user interface allows the surgeon 120 to memorize or "bookmark" the current position and orientation of the camera head 110 with respect to the camera, whereby this position and orientation is, for example, via the HMD 102. It can be restored later by selecting this feature from the menu displayed.

The computer 118 controls the operation of the camera system 112 and the lighting system 114 according to the input received from the surgeon 120 via the system mode and user interface. Computer 118 also controls system 100 according to voice commands detected by microphone 138.

The computer 118 may digitally or optically / mechanically control the display features relating to the HMD 102. For example, the computer 118 may digitally perform a zoom-in / zoom-out operation by reducing the pixel area from the entire video captured by the camera system 112 and stream the reduced area to the HMD 102. Alternatively, the computer 118 may optically implement the zoom in / zoom out function by adjusting the optical system of the high resolution cameras 140A and 140B. Similarly, the computer 118 may digitally implement the scrolling function by shifting the area of pixels extracted from the entire video acquired by the camera system 112 and stream those pixels to the HMD 102. Alternatively, the computer 118 may optically / adjust the position of the camera head 110 laterally along the XY axes via the camera head positioner 111 to shift a portion of the surgical field displayed on the HMD 102. Scrolling may be performed mechanically.

Here, reference is made to FIG. 1E, which is a schematic block diagram of the camera system 112 of FIGS. 1A and 1D, which is configured and operational according to a further embodiment of the disclosed technique. The camera system 112 is housed in the camera head 110 and coupled to the computer 118 via one or more wires and cables integrated within the mechanical arm 106 (FIG. 1A). The computer 118 also controls each of the components within the camera system 112, for example by selecting them to receive an image feed.

In one embodiment, the camera system 112 comprises at least two high resolution cameras 140A and 140B. The cameras 140A and 140B are provided with a high resolution optical system for capturing a high resolution magnified stereo image of the surgical field 124. In some embodiments, the camera head 110 further comprises either an iOCT scanner 142, an IR camera 148, or another camera for multispectral imaging. The camera system 112 is configured to stream the images acquired by the cameras 140A and 140B to the computer 118, which processes the images and optionally sends them to the HMD 102. Images from cameras 140A and 140B are streamed to the eyes of surgeon 120 via the HMD display modules 130A and 130B. The surgeon's brain produces a 3D image from the streamed video. The computer 118 controls the display of the image feed via the HMD according to the selected system mode and any additional settings and parameters defined by the surgeon 120. The computer 118 is used to control aspects of the system 100 and controls an additional camera that captures images that are not visible to the surgeon 120. For example, the computer 118 controls an IR camera or the like for detecting movement in the operating room. The image processor 118L processes the image acquired by the camera system 112 according to the settings defined by the surgeon 120, for example to perform color correction, sharpening, filtering, zooming, and the like. The computer 118 may adjust the illumination, focus, FOV, and magnification for the image feed according to one or more settings instructed by the surgeon 120 via the user interface described below, and stream the processed image feed to the HMD 102. ..

The system 100 allows the surgeon 120 to select a magnification level for the video acquired by the camera system 112 at the beginning of the surgical procedure and dynamically adjust the magnification throughout the surgical procedure as needed. To. System 100 allows the region of interest (ROI), zoom, and focus configuration to be dynamically updated using mechanical, optical, electrical, or digital means. Mechanical and optical updates to the XY positioning and zoom optics for the cameras 140A, 140B are performed via the camera system controller 118H (FIG. 1B). The digital zoom function and XY positioning are performed by changing the size and XY position of the ROI displayed via the HMD 102.

In some embodiments, the surgeon 120 uses a head gesture to dynamically update the ROI and zoom settings. In some embodiments, the surgeon 120 activates the system 100 to automatically center the camera system 112. In this case, the system 100 automatically updates the XY position dynamically. Some examples of dynamic updates include automatic centering, automatic zooming, automatic focusing, and automatic lighting.

The system 100 provides an automatic centering function for dynamically updating the raw image feed. One type of automatic centering follows the illuminated area during a retinal procedure in which fiber illumination is used and only part of the retina can be illuminated. Other types of automatic centering follow the center of the patient's eye during anterior ocular procedures, such as cataract procedures (eg, when the surgeon moves the patient's eyes). Other types of automatic centering follow tooltips. It should be noted that the examples given herein are intended as merely typical implementations and are not intended to limit the invention. Automatic centering can be performed on any patient's anatomy, any medical device, or any element or information that generally appears in the image.

Generally, the retinal procedure is performed with an additional lens (not shown) suspended from the camera head 110 just above the center of the eye of patient 122. Therefore, during these procedures, the camera head 110 must not be moved unless the patient's eyes move, and the movement of the camera head 100 will move this additional lens. This constraint means that adjusting the position of the camera system 112 via XY motor control is not a suitable technique for tracking the illuminated area during retinal procedures. As a result, adjustments to the ROI displayed for these actions are digitally made as part of the automatic centering function to keep the physical position of the camera system 112 fixed. The resolutions of the cameras 140A and 140B are generally higher than the resolution of the display monitor of the HMD 102. Therefore, the digital adjustment of the ROI is performed by selecting a subset of the entire pixel frame acquired by the camera system 112. When the surgeon 120 sees a small ROI as a zoom in the whole image, the ROI can be dynamically selected from the whole pixel frame. Centering and especially automatic centering functions can be performed using either controlling the XY motor and / or adjusting the ROI, depending on needs and circumstances. For example, depending on the type of treatment, one technique may be more suitable than the other.

In other embodiments, automatic centering on the illuminated area, the center of the patient's eye, tooltips, etc. is performed at discrete time intervals. For example, if the distance between the pupil and the edge of the image is less than the default threshold, centering is performed either by changing the ROI so that the pupil appears in the center of the image or by controlling the XY motor. Will be done. As another example, if the illuminated spot on the retina exceeds a predetermined margin, centering is performed during the posterior ocular procedure by automatically updating the ROI so that the illuminated spot is centered on the image. This is in contrast to continuous automatic centering, where the image is centered frame by frame. Continuous automatic centering can pose a risk if the patient's eyes are moving but appear stationary to the surgeon.

In a further embodiment, the centering itself is performed in a continuous space system, as opposed to a discrete space system. For example, the image is updated stepwise and continuously until centered (eg, the XY motor and / or ROI are affected stepwise and continuously) so as not to cause sudden changes in the image.

In another embodiment, prior to centering, system 100 warns the user of pending changes. The warning may be via an alarm displayed on the HMD, a sound, vibration in the HMD, or the like. The warning notifies the surgeon to anticipate changes in the image and / or changes in the XY position of the camera head unit. Optionally, the user may undo pending centering via UI160. For example, when given a warning that the system is trying to center an image, the user has a time window (eg, 3 seconds) to cancel the centering. In a further embodiment, when the centering is canceled, the system 100 disables subsequent automatic centering. This may be for a predetermined period of time (eg, 5 minutes), until the system mode is changed (eg, by switching from the previous mode to the later mode), or during surgery. In another embodiment, when the occurrence of automatic centering is canceled, the system disables automatic centering until there is a noticeable change in the image that can be detected, for example, by an image processing algorithm or a deep learning algorithm.

The system 100 provides an automatic zooming function for dynamically updating the raw image feed. The automatic zooming function may be mechanically implemented by controlling the zoom optical system in the camera head 110, or may be digitally implemented in the raw image feed via the image processor 118K. This configuration allows the zoom settings to be automatically changed based on the surgeon's specific preferences. One type of automatic zooming remembers the last zoom factor used in each system mode (eg, the previous mode) and automatically sets the zoom when the system is turned on and the mode is switched. Other types of automated zooming identify different stages of the procedure and remember the surgeon's preferences for each stage. Other types of automatic zooming dynamically change the zoom according to the size of the spot of light produced by the fiber illumination, which changes its size by being held by the surgeon at various distances from the retina. The zoom factor corresponds to the size of the spot. As the surgeon moves the fiber chip farther and the spot grows larger, the displayed image is zoomed out. As the surgeon moves the fiber chip closer and the spot becomes smaller, the displayed image is zoomed in.

Similarly, the system 100 enables manual and automatic focusing. For automatic focusing, the surgeon 120 may specify a point in the surgical field that should be kept in focus. Alternatively, the surgeon 120 may have the system automatically focus on the tip of the tool held and operated by the surgeon. Alternatively, the surgeon may have the system automatically focus on the center of the illuminated area in the retinal procedure. The system remains focused on the specified position, fixture, or spotlight. In some embodiments, the system 100 provides one-time focus on a designated point. The surgeon 120 may specify a point on the raw image using a combination of head / foot gestures and / or surgical tools to focus on that point as needed.

Here, reference is made to FIG. 1F, which is a schematic block diagram of a lighting system 114 (FIG. 1A) configured and operable according to other embodiments of the disclosed technology. The lighting system 114 is housed within the camera head 110 and coupled to the computer 118 via one or more wires and cables integrated within the mechanical arm 106. The lighting system 114 includes one or more of white floodlight 150, IR floodlight 152, coaxial light 154A, and coaxial light 154B. Coaxial light 154A and 154B operate with cameras 140A and 140B, respectively. Generally, the lighting system 114 includes at least one visible light emitter for illuminating the surgical field 124. The illuminant can be combined with one or more filters (not shown). For example, the illumination may be based on a white light emitting LED having a filter that blocks blue light in consideration of phototoxicity so that the emitted light itself is not white. The illumination system 114 may be illuminated with invisible illumination, such as infrared (IR) light or ultraviolet (UV) light. In some embodiments, the system 100 includes IR illumination combined with a low resolution / wide field camera to detect movement within the surgical field 124. Coaxial illumination causes the red-eye phenomenon caused by the reflection of light from the patient's retina, which is used in anterior ophthalmic procedures. The computer 118 controls the lighting system 114 with the lighting controller 118H by selecting, for example, a light source, an intensity, and the like.

In some embodiments, the system 100 enables automatic lighting. The image processor 118K analyzes the raw image feed with respect to the under-illuminated or over-illuminated area and informs the computer 118 to manipulate the position or intensity by the illumination system 114. Optionally, the lighting can be optimized for elements selected by the surgeon, such as surgical instruments. In some embodiments, the system 100 identifies that the surgeon 120 has completed one step and is initiating a new step, and automatically optimizes the lighting for the new step. For example, if an external faco system is streaming data, system 100 knows that the mode is currently set to faco mode. In such cases, the floodlight is automatically turned off and the red-eye lighting is turned on.

Reference is made to FIG. 1G showing a schematic block diagram of the foot switch 104 (FIG. 1A) configured and operable according to a further embodiment of the disclosed technique. The footswitch is shown with three sensors 104A, 104B, and 104C, a transceiver 104D, and a vibration motor 104E. The foot switch is located on the ground adjacent to the surgeon 120's foot. The surgeon 120 controls the motion features of the system 100 by performing foot movements detected by the footswitch 104. Sensors 104B, 104C, 104D are configured to sense foot movement and transmit notifications to computer 118 (FIG. 1B) via transceivers 104A and 118A, respectively. The sensors 104B, 104C, 104D may be any of pedals, buttons, touchpads, joysticks, pressure bases, image bases and the like. It should be noted that this implementation is intended to be merely exemplary and the footswitch 104 may have more or fewer sensors, or multiple buttons, each with different options. In some embodiments, the footswitch 104 is a wearable device configured with an inertial sensor for tracking foot gestures. Optionally, the wearable footswitch is additionally or as an alternative configured with an optical sensor and a light emitter (not shown) for detecting one or more foot gestures. For example, a illuminant configured with a wearable footswitch projects a pattern onto a surface, such as the floor. One or more optical sensors detect changes in the projected pattern to infer foot gestures. The optical sensor can be configured with a wearable footswitch, but this is not required. The vibration motor 104E provides the surgeon 120 with feedback on the settings and system parameters for the system 100 via the foot switch 104. Reference is made here to FIG. 2A, which shows a schematic block diagram for a typical user interface (UI) 160 for system 100 (FIG. 1A), which is configured and operational according to other embodiments of the disclosed technology. The UI 160 is a logical view of the components of the system 100 described above with respect to FIGS. 1B-1G that interface with the surgeon 120. It should be understood that the components included in the UI 160 are intended to be merely exemplary and that components can be added or removed from the UI 160 according to various implementations. FIG. 2A shows the components in various system units logically grouped according to their function. The UI 160 consists of two modules: a UI input 160A that includes components of the system 100 that receive input from the surgeon 120, and a UI output 160B that includes components of the system 100 that serve as outputs in response to the received input. including. Computer 118 receives user input via UI input 160A, processes the input, and issues one or more system responses via UI output 160B.

The UI input 160A includes a head gesture tracker 162. The head gesture tracker 162 includes a tracking component 134 (FIG. 1C) integrated into the HMD 102 and inside the HMD 102, and optionally a head gesture tracker component 164 outside the HMD 102. For example, the head gesture tracker component 164 may be a camera placed in the operating room that acquires an image of the surgeon 120. The UI input 160A further includes a foot switch 104 for receiving foot-based inputs (FIG. 1A), an eye tracking system 166, an acoustic system 168, and a touch screen 108 for receiving inputs from other surgeons (FIG. 1A). )including. The eye tracking system 166 includes an eye tracking component 136 (FIG. 1C) coupled to an eye tracker controller 118L (FIG. 1B), both configured to sense and track the direction of the surgeon's line of sight. The acoustic system 168 includes a microphone 138 (FIG. 1C) coupled to a speech recognition device 118M (FIG. 1B), both for receiving, detecting, and processing acoustic signals and voice commands issued by the surgeon. It is composed of. In some embodiments, the microphone 138 is external to the HMD 102.

The UI output 160B includes a display module 130 (FIG. 1C), a shutter 132, a touch screen 108 (FIG. 1A), a camera head 110 (FIG. 1D), a camera head positioner 111, a camera system 112 (FIG. 1E), and a lighting system 114. Includes content displayed via (FIG. 1F). Based on the input received, the computer 118 controls the state of the shutter 132, the display of the content of the display module 130, and the display of the content of the touch screen 108, as well as the camera head 110, the camera head positioner 111, and the camera system 112. , And the settings for the lighting system 114. In some embodiments, the UI output 160B is connected to one or more systems external to the system 100, such as a facobitrectomy machine, to the surgeon 120 directly to the external system of the system 100 via the UI 160. Give control. In some embodiments, the system 100 further displays any or all of the available video streams and also supports connection to a 2D or 3D external monitor (not shown) to support multiple HMDs simultaneously. do.

In some embodiments, the UI output 160B further includes a speaker, such as a speaker 139 (FIG. 1C), for emitting a sound output, such as a dial tone or a voice message to notify the surgeon 120. In some embodiments, the UI output 160B further includes one or more vibration motors 161 such as, for example, vibration motor 141 (FIG. 1C) or vibration motor 104E (FIG. 1G). For example, the vibration motor may be an eccentric rotary mass vibration motor (ERM) using a DC vibration motor, or a linear resonant actuator (LRA) for displaying system parameters to the surgeon 120. The vibration motor may be integrated into any of the HMD 102, the foot switch 104, or any other component of the system 100. For example, vibration may indicate that the xmotor has reached its motion limit, or that System 100 has automatically identified a tool that is extremely close to the retina. In some embodiments, the system 100 monitors the surgeon 120's head to ensure that the surgeon 120's posture is ergonomically appropriate, with the surgeon 120 necking his head. If placed in an ergonomically inappropriate position that can cause pain or injury, it will alert you. The alarm can be issued, for example, through the issuance of an acoustic alarm such as a message, vibration, or a dial tone displayed by the HMD.

The surgeon 120 interacts with the system 100 via the user interface 160 while wearing the HMD 102 to perform a surgical procedure on the patient 122. Thus, using the hand-free gesture, the surgeon 120 can see the operating parameters of the system 100, such as the image processing parameters of the computer 118 (eg, zoom, color correction), the content displayed via the HMD 102 (eg, the camera system). Images acquired by 112, preoperative data acquired from memory, overlaid virtual features, opening and closing of virtual screens and menus, state of shutter 132 (eg open, partially open, closed). Control settings for the lighting system 114 (eg, light emitter, filter, intensity, wavelength, angle), control settings for the camera system 112 (eg, camera, zoom, focus, shutter), control settings for the camera head 110 (eg, for example). Orientation and consistency by adjusting the xyz motor and tilt), selection of menu items to control any of the above, control of system mode, etc. may be controlled.

In some embodiments, the UI 160 makes it possible to store snapshots and videos, and record audio for later use. Audio recordings can be converted to text for use as notes, surgical summaries, naming and tagging filters. The keyword tag can then be applied by a deep learning algorithm. In some embodiments, the system 100 assists in the automatic generation of surgical reports based on a predetermined template with placeholders for preoperative data, snapshots from raw images, recorded notes / summary audio text. do. Automatic report generation assists in the identification of additional data such as, for example, the name of patient 122, the name of surgeon 120, date, time and duration of surgical procedure. Where appropriate, machine learning and image processing may be applied to obtain treatment-related data. For example, the type of surgical instrument used and an image feed of the surgical procedure may be obtained to add data to the report. The UI 160 provides a "Send to Report" menu item that allows the surgeon to upload selected snapshots and preoperative images to the report. In some embodiments, the UI 160 presents an automatically generated billing summary that can be configured according to a default template. The claim summary may include a list of surgical procedures performed, and the like.

If the video and audio are recorded separately, the audio recording is saved with a time stamp that allows the audio to be synchronized with the corresponding video file. Similarly, when audio notes are converted to text, they are saved with a sync timestamp that allows the note to appear as a subtitle on the corresponding video feed.

In some embodiments, the UI input 160A is a plurality of simultaneous hands performed by the surgeon 120, for example, a foot movement detected via the footswitch 104 and a head gesture detected via the head gesture tracker 162. It is configured to detect input as a free gesture. In these embodiments, the enable input from the footswitch 104 serves by activating the head gesture to prevent accidental head movements by the surgeon 120 from affecting the operation of the system 100. Upon receiving the input from the UI input 160A, the computer 118 triggers the corresponding action, depending on the system mode.

In some embodiments, the surgeon 120 may temporarily set the system 100 to an activation-free state in which the default action is activated by various head gestures without an activation input from the footswitch 104. For example, Surgeon 120 sets the default actions to be focus control for head-up and head-down gestures, zoom-in and out for left and right head gestures, or enables free head gestures to switch between system modes. You can do it.

Here is a description of some typical user inputs available to the surgeon via the UI input 160A to control aspects of the system 100. This list is not intended to be limited.

Head Gesture A common user interface method for surgical HMDs based on the use of head gestures. The head gesture is detected by the head tracker 162 of the UI input 160A (FIG. 2A). The surgeon 120 may set the system parameters so that some head gestures are enable-free and allow him to control the system 100 using only the head gestures. For example, for a particular stage of medical procedure, the surgeon 120 prefers to work in activation free mode and prefers to operate the system 100 himself using only the head gesture without activation via the footswitch 104. It can be possible. The surgeon 120 may issue voice commands for this effect to turn off activation via the footswitch 104 and use head gestures to control features such as zoom or focus, for example. The surgeon 120, for example, has the ability to zoom, focus, and illuminate with the camera head 110.
〇 For example, scrolling within data items such as preoperative images displayed on the display module 130,
〇 To control continuous and discrete system characteristics such as lighting on / off, image enhancement options, enable / disable, etc. via menu control, either directly or indirectly through head gestures. Use head gestures.

-Head gestures enabled by the footswitch 104.
The foot switch 104 (FIG. 1G) is provided with one or more pedals such as, for example, pedals 104A, 104B, 104C. Each of the pedals 104A, 104B, 104C may enable a group of head gestures to control various system parameter sets. Each group consists of all the gestures used, such as up-and-down movement, left-right movement, gaze, small nod, and back-and-forth movement. The use of the footswitch 104 with head gestures allows the surgeon 120 to control more system features than are available with head gestures alone. Also, enabling head gestures via additional user input improves system safety by eliminating inadvertent changes, such as accidental switching to different system modes due to the natural movement of the head. Let me.

For example, a footswitch with three pedals may enable the following five groups of head gestures in normal system mode.
○ Pressing the left pedal (104A)
○ Upper and lower head gestures: Focus of enlarged image.
○ Slight up and down nods: Switching between enabled and disabled states of automatic focusing. For example, if you enable one-time autofocusing at a given point, a cross may appear as an overlay on the image, and the user rotates his head to specify a point in the surgical field to autofocus. You may move the cross. Release of the pedal selects the specified point. The surgeon 120 uses a menu or touch screen to nod his head to determine what kind of autofocus (eg, one-time autofocus to a specified point, continuous autofocusing to a specified point, You may choose whether autofocusing to the tooltip, autofocusing to the center of the light spot) is activated.
○ Left and right head gestures: Zoom of enlarged image.
○ Slight left / right swing: Switching between enabled and disabled states of automatic zooming.
○ Pressing the left + center pedal (104A and 104B)
○ Up / down and left / right head gestures: Scrolling in the y and x directions in the enlarged image when the ROI is displayed.
○ Slight up and down nods: Switching between enabled and disabled states of automatic centering based on ROI changes.
○ Pressing the center pedal (104B)
○ Up / down and left / right head gestures: Menu navigation.
○ Long gaze: Open a submenu or select a menu that appears in the foreground.
○ Release of center pedal 104B: Activates the selected menu item.
○ Pressing the right + center pedal (104C and 104B)
○ Up / down and left / right head gestures: Move y and x motors.
○ Slight up and down nods: Switching between enabled and disabled states of automatic centering based on XY changes.
○ Pressing the right pedal (104C)
○ Upper and lower head gestures: Floodlight intensity.
○ Slight up and down nods: Switching the floodlight on and off.
○ Left and right head gestures: Red-eye (coaxial) lighting intensity.
○ Slight left / right swing: Switching the coaxial lighting on and off.

-Sound input.
The system 100 is configured to respond to acoustic signals detected via one or more microphones 138. The surgeon 120 may issue one or more voice commands and / or sounds to enable or disable features, or to switch to a different system mode, modify or enhance the displayed image, and so on. The acoustic input is detected by the microphone 138 and processed by the computer 118. For example, the surgeon 120 may use a single tongue tap to enable / disable one feature and a second sound to enable / disable a second feature. The sound for activating the features of the system 100 may be user specific and may be customized by the surgeon 120. The surgeon 120 may teach the system 100 to recognize a set of sounds of his choice. A microphone (not shown) physically placed near the mouth of the surgeon 120 (eg, part of the HMD 102) can pick up a very quiet sound that is out of the audible range of the patient 122. This can be advantageous in procedures performed without anesthesia, for example in many eye surgery procedures. In these situations, it may be preferable for the patient 122 not to hear the voice commands issued by the surgeon 120 to control the system 100.

・ Hand gesture.
In some embodiments, the system 100 is by a handheld controller (not shown) or by an external tracker (not shown) that senses hand gestures performed by the surgeon 120 and sends them to computer 118 for processing. Provides one or more acquired hand gesture inputs. For example, the handheld controller may be designed similar to an electronic mouse, remote controller, joystick, or may be a wearable device such as a bracelet or ring. Alternatively, the hand gesture can be obtained optically by an HMD or other integrated camera and / or other sensor. Hand gestures can be used in connection with any of the UI techniques disclosed herein. For example, hand gestures can be used in operating room locations or in connection with menus or virtual 3D buttons that are "locked" to the patient's eye. In some embodiments, the system 100 detects hand gestures by tracking the user's finger. In another embodiment, the system 100 detects a hand gesture by tracking a surgical tool held by the surgeon 120, who "touches" and "touches" a menu item or virtual 3D button with the surgical tool. It is possible to "press".

-Physical control via the camera head 110.
The camera head 110 may be provided with one or more manually operated buttons, handles, and switches, and a control panel. These may be controlled by the surgeon 120 or an attendant nurse.

-User eye tracking in combination with virtually displayed menus or buttons.
The surgeon 120 may navigate the virtually displayed menu options via the eye tracker 136 (FIG. 1C). For example, the surgeon may stare at a virtual menu item for a predetermined period of time, or blink according to a pattern, or perform other such "eye gestures." This UI may be used in connection with any of the other UIs described herein. In addition to the display-fixed virtual menu, the menu may be world-fixed or anatomical-fixed. For example, a 3D virtual button may appear "locked" in the eyes of patient 122, for example around a treatment area. The surgeon 120 may "press" the button displayed by staring at the virtual button, and the eye tracking component 136 detects the stare.

·touch screen.
Control and operating parameters for the system 100 can be set, monitored, and modified via the touch screen 108. The touch screen 108 may be covered with sterile nylon for use by an operating room nurse or surgeon 120. In addition to controlling motion settings by the surgeon 120 wearing the HMD 102, the touch screen 108 can control modes, settings, and preferences that cannot be controlled by the surgeon 120 wearing the HMD 102. To. The touch screen 108 may display any of the video feeds available to the system 100 in raw format or in a format visible to the wearer of the HMD 102, using, for example, overlays, PIPs, and the like.

The system mode modifies the user interface configuration by determining how the system 100 interprets and reacts to the user input made by the surgeon 120. For example, in preoperative mode, the surgeon 120 scrolls through a plurality of images displayed via the HMD 102 by simultaneously pressing the pedal of the foot switch 104 while turning his head. In normal mode, the surgeon 120 zooms in or zooms in on the image displayed via the HMD 102 by simultaneously pressing the pedal of the foot switch 104 while moving his head to the left (zoom in) or to the right (zoom out). Zoom out. In some embodiments, only the computer 118 enables the execution of the action while the appropriate pedal of the footswitch 104 is pressed.

The following are some typical system modes provided by system 100 and the characteristics they control. The system mode defines which parameters are manipulated for the system 100, for example display settings for the HMD 102 or operation settings for the camera system 112. The surgeon 120 selects and initiates a mode by a combination of UI inputs detected by the UI 160A, including head gestures, eye movements, voice commands, and foot movements. The computer 118 processes the UI input and provokes a system response via one or more system output components of the UI 160B. The system mode determines the response, such as one or more of the content displayed on the HMD 102, the screen state, the state of the shutter 132, the UI function, and the operation settings for the camera head 110. The following typical mode list is not intended to be exhaustive and does not include all of the characteristics controllable by each system mode.

-Normal mode (default mode). In this system mode, the default values are generally valid, but can be customized by the surgeon 120. The image feed acquired by the camera system 112 is displayed via the HMD 102. The surgeon 120 primarily uses the UI 160 to control display attributes such as zoom, focus, XY position for the camera system 112, and lighting settings for the lighting system 114.

-Transparent mode. In this system mode, the display module 130 is turned off, the LC shutter 132 is open, and the surgeon 120 can see the real world view.

-Rear eye non-contact lens mode. This system mode allows the surgeon 120 to see the surgical field 124 in inverted or posterior ocular mode. The settings related to this mode are as follows.
○ Eye inversion Inversion.
○ Floodlight off. This may be user configurable.
○ Red-eye lighting is off.
○ Color scheme Fiber lighting (color correction is material-specific).
○ Image enhancement Eigenvalue of the posterior eye.
○ Motor state Limited speed and range. These parameters can be limited by the surgeon by controlling the progress or speed of the head gesture.

-External mode. These system modes correspond to additional external devices (not shown) that may be connected to the system 100, such as an endoscope. One of the external system modes has the following settings.
○ Red-eye lighting on. Intensities for red-eye and floodlight are predefined.
○ Overlay Additional data is overlaid from the external system.
○ The functions of UI160 can be customized. For example, the definition for the footswitch 104 may be different in this mode than in other modes. This is to allow the surgeon 120 to control the external machine with the foot switch 104. Accordingly, the menus that display options will be modified to reflect different features.

-Preoperative data mode ("preoperative" mode). In this mode, the user sees pre-planned data such as preoperative images or notes and / or patient data prepared by the user. After viewing the data, the user returns to the previous mode to continue the surgical procedure. The user may also select an image set that can be viewed in PIP from the image library, return to the previous mode, and then scroll within the selected image, as will be described in detail in FIGS. 6C-6J. .. The settings related to this mode are as follows.
○ Display content. Instead of the raw magnified image acquired in the surgical field 124, the HMD displays a GUI that allows the user to access any data desired and view and scroll the data.
○ UI function. This mode allows you to view and scroll preoperative data. For example, a gesture in normal mode can control the zoom or focus of a magnified image. In the preoperative data mode, the magnified image is not displayed, and the same gesture allows scrolling of the displayed preoperative data, etc.
○ Lighting. Illumination by the illumination system 114 can be set to a minimum in the preoperative data mode, or turned off and dimmed, and switched to other emitters included in the illumination system, such as IR illumination with IR floodlight 152. .. IR lighting allows the surgeon 120 to detect the movement of the nurse present while viewing the preoperative data and can be turned on automatically. Lighting can be restored to the previous non-IR settings when the surgeon 120 switches back to normal system mode.

-PIP (picture in picture) system mode. The picture-in-picture (PIP) display displayed data such as preoperative images, raw video feeds, previously captured video snapshots, GIFs, and, for example, vital signs and other data via the HMD 102. Allows you to display in an overlaid window on top of the current background image. Surgeon 120 may predefine several different PIPs, their location, and what content to display within each PIP. For example, the PIP displays a preoperative OCT image when working in the posterior non-contact lens mode, or when in the endoscopic mode and the background image is a live video feed from the endoscopic camera. An enlarged image may be displayed. Surgeon 120 may turn each default PIP on and off. The surgeon 120 may control the position and size of the PIP and the on / off state via the UI 160. The position of the PIP in the field of view of the HMD 102 may be dynamically changed according to the position of the automatically detected region of interest in the main image and may be set by the surgeon 120 via the UI 160. The location of the PIP may also be locked to the real world (“world fixed PIP”) or fixed to the patient's anatomy, for example in a VGS procedure. When in PIP mode (ie, after "selection" of PIP), snapshot activation may only store PIP images. This can be useful, for example, for storing iOCT images.

For example, in order to enable or disable the overlay symbol indicating the position on the raw magnified image corresponding to the preoperative image currently displayed in PIP, when PIP is displayed, the overlaid menu will be PIP-specific. Features can be added. In some embodiments, the driver 102A of the HMD 102 closes the shutter 132 within the area of the display module 130 where the PIP is displayed while performing the augmented reality (VGS) procedure. This allows the surgeon 120 to see the real world through a partially transparent screen, while simultaneously blocking the ambient light behind the PIP and displaying preoperative, intraoperative, or other data within one or more PIPs. It is possible to see. When the PIP is locked to the patient's anatomy, the computer 118 covers the area behind the PIP where the shutter is closed so that the background is always blocked while the surgeon 120 is moving his head. It may be controlled dynamically.

In some implementations, switching to PIP system mode is possible only when at least one PIP is displayed via the HMD. Switching to PIP system mode can be done, for example, by pressing a footswitch button that calls up a menu, and by selecting PIP as if it were a menu item. As a result, the UI function is changed to the default UI function according to the content displayed in the PIP. For example, if the PIP displays preoperative data, the UI is taken over from the preoperative data mode (for example, allowing the surgeon 120 to scroll through the set of preoperative images displayed within the PIP) and the PIP. When displaying a live magnified image, the UI may be in normal mode (eg, allowing the surgeon 120 to control the magnification) or posterior non-contact lens mode, or in general (some other not listed here). (Because such a mode can exist) it is inherited from the last selected mode to view the raw image feed. Special UI features may be added to the UI inherited in PIP mode, others may be fine-tuned for PIP mode. For example, if the PIP displays a preoperative image, the surgeon 120 may use a head gesture to control the virtual pointer to specify a corresponding point in the displayed preoperative and raw magnified image (which is). , Can be done to manually align preoperative data to raw images).

-IOCT (intraoperative OCT) mode. This system mode causes the iOCT scanner 142 to acquire OCT data. Settings for this mode may include:
○ iOCT is on.
○ PIP on, the image acquired via the iOCT scanner 142 is displayed in the PIP.
○ UI option This feature allows the surgeon 120 to control operating parameters for the iOCT scanner 142, such as scan position and angle, using head gestures or by navigating menus. I will provide a. For example, the user may use a menu to switch between the iOCT scan modes of B scan and volumetric scan.
○ B scan indicator overlay This feature overlays a symbol indicating the position of the raw iOCT B scan on the raw magnified image.

• Other treatment-specific and stage-specific modes: In general, various treatment-specific and stage-specific modes may be predefined for displaying treatment-specific and stage-specific data and overlays.
○ Data overlays, such as data from Facobitrectomy machines, are enabled.
○ Aligned overlays are enabled. This view displays an overlayed image alignment symbol and / or image and / or 3D model on a real-world magnified view of the surgical field to assist the surgeon 120 during the surgical procedure.
○ UI UI features may vary to allow the user to control the position, depth, and orientation of the overlay, or to allow other devices connected to the system to be controlled. For example, the angle of the straight line that overlays the display can be manipulated using head gestures and / or footswitch 104.
○ PIP (picture in picture). PIP can be enabled to show relevant preoperative or intraoperative video and / or data via PIP.

-External video feed mode:
○ In some modes, the HMD 102 may be used, for example, outside the operating room to allow the surgeon 102 to monitor patients waiting in the waiting room or to monitor the progress of different surgical procedures in different operating rooms. Display an external video feed as may be provided by one or more cameras deployed. These videos may be displayed in the PIP or on the side screen of the HMD 102. They can be viewed as a jumble of videos, one video at a time, or on multiple screens. This mode allows the surgeon to see all available video feeds on multiple screens. Surgeons may look at video feeds from the waiting room, other operating rooms, or search a database of teaching material videos that explain how to perform rare procedures that can occur, for example, in unpredictable situations. It's okay. The surgeon may predetermine when and how to watch the selected video, for example in PIP, and when and how to return to the previous mode.

・ Teaching mode:
In some embodiments, the system 100 may be simultaneously guided by an assistant surgeon (not shown), such as an older surgeon, working with the surgeon 120. The assistant surgeon wears a second HMD that is coupled to the computer 118 and camera head 110 and operates similarly to the HMD 102. The second HMD also displays a view of the surgical field 124, the assistant surgeon may control a cursor on the display synchronized with the display in module 130 of the HMD 102, and the surgeon 120 may control the cursor controlled by the assistant surgeon. Allows you to see. In this way, the assistant surgeon can guide the surgeon 120 during the surgical procedure. Similarly, the assistant surgeon may call one or more PIPs that indicate preoperative or other data to the surgeon 120.

In this mode, the senior surgeon may instruct the surgeon 120 performing the surgical procedure, for example if the surgeon 120 is a trainee. In some embodiments, the eye tracking component 136 (FIG. 1C) tracks the gaze of the surgeon 120 and determines the display area of the HMD 102 that the surgeon 120 is gazing at. The system 100 displays a symbol indicating the area that the surgeon 120 is gazing at to the assistant surgeon via the second HMD. In this way, the assistant surgeon can monitor the focus of the surgeon 120.

In some embodiments, the senior surgeon may overlay features and symbols to guide the surgeon 120 while performing the procedure. Surgeon 120 sees the overlaid end effect on the raw image and does not see the menus and drawing options that the supervisor sees in this mode. Features in the teaching mode include:
○ As will be described in more detail in Figures 6K to 6O, a menu such as "Paint" for selecting drawing options and selecting virtual tools / markers / pointers to display, marking colors, marking erasure, etc. will be displayed. Show to the chief surgeon.
○ Allowing older surgeons to control or draw pointers using head gestures or with dedicated wands.
○ When drawing on a real-time image, the line drawing (lines, symbols, etc.) may be locked to the patient's anatomy, ie the overlay symbol is the patient's anatomy (eg, when the surgeon moves the patient's eyes). Automatically / dynamically change its position so that it appears locked to the anatomy. The line drawing may be 3D.
○ Senior surgeons may use a touch screen to direct and draw if they are not wearing an HMD. Senior surgeons may remotely guide the surgeon, for example, using a regular screen and mouse in their office or home, and possibly using a tablet or smartphone to direct and draw on real-time images. ..
O A symbol indicating to the surgeon that someone has connected to System 100 System 100 in "drawing" mode and enabled drawing to the raw image feed may be displayed at the corners of the displayed image.
○ The surgeon will be provided with an erasure option. When a feature is drawn on the display, a menu item is added to the current menu (ie, when the surgeon calls the menu, he sees the menu) and the surgeon erases one or more drawing features. Enables. The menu may allow you to erase all drawing features by selecting a single "Erase All" menu item. Alternatively, the menu presents a submenu that allows the surgeon 120 to select which drawing features to erase and the number thereof.
○ A freeze option may be provided to freeze the raw image feed to facilitate the drawing of features.
○ 3D symbol / marker:
A 3D symbol is a symbol that appears in both eyes to appear at a given depth to the anatomy. The 3D marker produces a 3D image fixation symbol (eg, a 3D straight line).
The user may select a 3D pointer or marker that is fully controlled by the user, i.e. the user controls both its XY position and depth (user is enabled, for example, by a depth control pedal). You may manually control the depth using a head gesture).
Alternatively, the user may select a 3D pointer or marker that automatically locks to the depth of the currently focused anatomy, and when the user moves the symbol, the user can see that the symbol is of the anatomy. Get the feeling of following the surface (eg moving up and down).
-When the latter is used, the height is automatically calculated and updated when the focus is changed by the user. For example, if the first focus point is on a transparent surface such as the cornea and the second focus point is posterior (lower) to the first focus point such as a point on the anterior capsule. The symbolic positions in the two images (displayed in both eyes of the HMD user) change in different ways to reflect the new depth.
○ Wand:
The wand is a mouse-like or remote control-like controller, optionally with various types of buttons, handheld, wireless or wired, and optionally within a sterile cover. Has optional integrated inertial tracking. It can be used both for instruction / drawing as described above and for controlling virtual tools.
• The supervising senior surgeon may use a handheld controller to move the virtual tool in a natural way to show the trainee how to perform the surgical procedure. The virtual tool appears in the display as a 3D tool that the user sees as an overlay on the raw image via the HMD. It is based on a 3D model of the tools available for the user to select via the menu / GUI in this mode.

The surgeon 120 selects and switches the system mode for the system 100 using one or more hands-free gestures detected via the UI input 160A. In some embodiments, the surgeon 120 switches the system mode by performing the head gesture detected by the head tracker 162 while simultaneously performing the foot movement detected by the foot switch 104. In another embodiment, the surgeon 120 switches the system with voice commands detected by the microphone 138. Computer 119 determines how to interpret subsequent user input according to the system mode.

For example, in one system mode, the upward rotation of the head performed by the surgeon 120 while pressing the pedal 104A of the foot switch 104 increases the intensity of illumination by the illumination system 114. In different system modes, the same head gesture performed while pressing pedal 104A of footswitch 104 scrolls the image displayed via HMD 102. The lateral head gesture performed by the surgeon 120 while pressing the pedal 104B of the foot switch 104 may switch between these two system modes. By allowing multiple system responses to the same gesture, the system mode provides a wider range of functions for the system 100, allowing the surgeon 120 to control the features of the system 100 without the use of his own hands and perform surgical procedures. Allows you to keep both hands free to do.

Here, reference is made to FIG. 2B, which shows a typical control flow diagram for the user interface of FIG. 2A, which is configured and operational according to a further embodiment of the disclosed technique. The computer 118 (FIG. 1B) receives user input via the UI input 160A, such as a head gesture from the head tracker 162 that is simultaneously indicated to be pressed by the foot of the pedal of the foot switch 104 (FIG. 1G). The system response is determined depending on the system mode, that is, the response is mode-specific. In one system mode, user input may result in a first action in system 100, and in different system modes, the same user input may result in different actions in system 100. The flow diagram of FIG. 2B is intended to show the flexibility and scope of functionality that can be controlled via the user interface 160 in relation to multiple system modes. This figure is intended to be merely exemplary and is not intended to limit the invention to any particular implementation or example disclosed herein.

The computer 118 receives user input from the head tracker 162 and the foot switch 104 via the UI input 160A. Case 1. The user input is an indication that the surgeon 120 performs a head-up gesture while pressing the pedal 104A of the foot switch 104. Since the current system mode is normal, the computer 118 performs a zoom out on the image displayed on the HMD 102, which is shown as action 240. Case 2. The user input is the same as described above, for example, an indication that the surgeon 120 performs a head-up gesture while pressing the pedal 104A of the foot switch 104. However, because the current system mode is set to preoperative OCT (“preoperative” mode), the computer 118 performs scrolling between preoperative OCT images, shown as action 242. Case 3. The user input is an indication that the surgeon 120 performs a head-up gesture while pressing the pedal 104B of the foot switch 104. Since the current system mode is normal, the computer 118 switches the system mode from normal to vital, as shown as action 244. Mode switching will be described in more detail with respect to FIG. 3A.

By responding to a plurality of user inputs obtained from the plurality of user interfaces described herein, the system 100 has a wide range of operating features with the surgeon 120 leaving his hands free to perform the surgical procedure. Allows you to control. Inputs may provoke one system response in one system mode and different system responses in different system modes, providing a wide range of functions with a relatively small set of inputs, such as head gestures, foot presses, and so on. The foot switch 104 operates to control the response of the system 100 only to intentional gestures by the surgeon 120. An inadvertent gesture by the surgeon 120 does not result in a response by the system 100 if the footswitch 104 is not pressed at the same time.

According to one embodiment, the surgeon 120 may select the system mode by controlling the position and orientation of the HMD 102. The range of possible positions and orientations with respect to the HMD 102 is divided into multiple zones, each zone corresponding to a different system mode. By moving its head to match the position and orientation of the HMD 102 with the zone, the surgeon 120 may select the system mode corresponding to that zone. The system 100 may require the surgeon 120 to press the footswitch while manipulating his head to prevent accidental switching to the new mode.

Here, FIG. 3A shows a typical layout 300 for various positions and orientations with respect to a plurality of zoned HMD 102 (FIG. 1A) configured and operational according to other embodiments of the disclosed technique. Referenced. The layout 300 shows a plurality of zones, each of which is a plurality of orientation regions with respect to the HMD 102, and each region corresponds to a different system mode. The system 100 allows the surgeon 120 to switch from one system mode to another by rotating the HMD 102 by rotating his head. The surgeon 120 may customize the functionality of the HMD 102 by defining multiple angle and position zones, each of which is within the range of his or her head movement, and determine which system mode is performed for each of the zones. ..

Layout 300 shows zones 302, 304, 306, 308, and 310 for the HMD 102 of FIG. 1A, respectively. Dashed lines 312, 314, 316, and 318 define their respective zones by indicating angular thresholds between zones 302, 304, 306, 308, and 310. The straight line 312 sets the threshold between the vital zone 308 and the normal zone 302, the straight line 314 sets the threshold between the normal zone 302 and the surgical field zone 306, and the straight line 314 sets the normal zone 302 and the preoperative zone. A threshold between the normal zone 302 and the nurse zone 304 is set by the straight line 318. By pressing the pedal of the footswitch 104 while traversing one of the thresholds 312, 314, 316, and 318 by reorienting the HMD 102 to match the new zone, the system mode is set to the new zone. Switch to correspond to.

The zone 302 labeled "normal" is associated with the forward horizontal orientation of the HMD 102 having the normal system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 302, the normal system mode is initiated. The shutter 132 is closed and the raw video feed of the image acquired by the camera head 110 is rendered via the HMD 102.

Zone 304 labeled "nurse" is associated with a rightward horizontal orientation of the HMD 102 having the nurse system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 304, the nurse system mode is initiated. The shutter 132 is opened, the video feed is turned off, and the surgeon 120 is able to see and communicate directly with the nurse while wearing the HMD 102.

The zone 306 labeled "preoperative" is associated with the left horizontal orientation of the head of the surgeon 120 having the preoperative data system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 306, the preoperative data system mode is initiated. The shutter 132 is kept closed and preoperative data is displayed via the HMD 102. In some embodiments, UI functionality may change when switching to preoperative system mode. For example, menu items may be different and the head gesture may cause the system 100 to perform different actions than other system modes to allow the surgeon to view and scroll the preoperative data.

Zone 308, labeled "vital", is associated with a forward-upward orientation of the HMD 102 having the vital signs system mode. When the surgeon 120 presses the foot switch 104 while moving his head to align the HMD 102 with the zone 308, the vital system mode is initiated and data indicating one or more vital signs of the patient 122 is via the HMD 102. Is displayed.

The zone 310 labeled "surgical field" is associated with the anterior-downward orientation of the HMD 102 having the surgical field system mode. When the surgeon 120 presses the footswitch 104 while moving his head to align the HMD 102 with the zone 310, the shutter 132 opens, the display is turned off, and the surgeon 120 sees a real-world view of the surgical field 124. It is possible.

In some embodiments, the system mode is the current HMD 102 only while pressing the foot switch 104 (eg, a button in the default foot switch 104 to trigger a mode switch based on the HMD's P & O). Determined according to P & O. The surgeon 120 switches the system mode of the system 100 by moving his head across one or more of the angle thresholds 312, 314, 316, and 318 while pressing the foot switch 104. When the surgeon 120's head orientation exceeds a predetermined angle and the HMD 102 coincides with the new zone, the system mode switches to the system mode corresponding to the new zone. For example, to stay in preoperative mode, the surgeon 120 keeps his gaze above the angle threshold 316 (to the left) while pressing the footswitch 104. When the foot switch 104 is pressed for the first time, if the P & O of the current HMD exceeds the threshold value of the current mode, the system mode is switched instantly. In this embodiment, when the surgeon releases the footswitch 104, the system mode switches back to the original mode. This mode of operation is useful, for example, if the user prefers a quick glance at preoperative data or nurses. In some embodiments, when the surgeon releases the footswitch 104, the system mode remains in the current mode, allowing the user to move his head freely again without affecting the system mode. This mode of operation is useful, for example, if the user prefers to comfortably review preoperative data over a long period of time.

Angle thresholds 312, 314, 316, and 318 may further enable hysteresis enable control. For example, a larger threshold may be valid when crossing the threshold from the normal zone 302 to the nurse zone 304, and a smaller angle threshold may be valid when crossing the nurse zone 304 to the normal zone 302 again. .. This is useful for avoiding undesired mode switching when the head orientation is close to the angle threshold. The hysteresis state (enabled / disabled) and the hysteresis size may be configured by the surgeon 120. The system 100 may automatically generate two hysteresis generation thresholds from a single threshold.

In other embodiments, zones 302, 304, 306, 308, and 310 are not defined with accurate thresholds. Rather, the activation of modes associated with zones 302, 304, 306, 308, and 310 is based on relative head gestures by surgeon 120. For example, by pressing pedal 104C of the footswitch 104 while pointing its head in any direction more than 10 degrees, the relative rotation direction required to switch from the current system mode to another system mode. The system mode is switched to correspond.

In a further embodiment, the input from the footswitch 104 is not required to switch modes. The angle thresholds 312, 314, 316, and 318 are used as triggers to change the system mode. The system mode corresponding to each of the regions is determined according to the current P & O of the HMD 102. For example, in order to remain in preoperative mode, the surgeon 120 needs to keep his gaze above the angle threshold 416 (to the left).

In another embodiment, "touching" the angle threshold by turning the head slightly switches the system mode corresponding to each zone. When the surgeon 120 "touches" the angle threshold 316 while pressing the foot switch 104, the system switches from the normal mode to the preoperative mode corresponding to the preoperative region 306. During this mode, the surgeon 120 comfortably looks forward to view the preoperative data without distorting his neck for extended periods of time. By touching the threshold 316 a second time while pressing the foot switch 104, the system mode switches again to "normal" corresponding to the normal region 302, and the surgeon sees a live magnified image of the surgical field 124 via the HMD 102. Make it possible. The footswitch 104 is used to enable switching between system modes such that pressing a particular pedal of the footswitch 104 while simultaneously touching the threshold with a head gesture results in switching to a new system mode. Used. This is to avoid accidental mode switching.

Here is a hint to the user which head gesture is needed to invoke the various system modes associated with the zones described with respect to FIG. 3A, configured and operational according to embodiments of the disclosed technique. See FIG. 3B, which shows the display of the overlaid raw image 320. The image 320 displayed on the field of view of the HMD 102 corresponds to the normal mode associated with the zone 302. In some embodiments, the surgeon 120 chooses to display features 324, 326, 328, and 330 with respect to the system 100, suggesting to the surgeon which head gesture calls which system mode. Features 324, 326, 328, and 330 are displayed transparently or translucently around the field of view so as not to obscure the display of the image 320. Feature 324 with the symbol "nurse" suggests to the surgeon that the nurse mode corresponding to zone 304 in FIG. 3A is invoked by turning the head to the right in the direction of feature 324. Feature 326 with the symbol "preoperative" suggests to the surgeon that the preoperative mode corresponding to zone 306 in FIG. 3A is invoked by turning the head to the left in the direction of feature 326. Feature 328 with the symbol "vital" suggests to the surgeon that by pointing the head upwards in the direction of feature 328, the vital mode corresponding to zone 308 in FIG. 3A is invoked. Feature 330 with the symbol "field" suggests to the surgeon that the surgical field mode corresponding to zone 310 in FIG. 3A is invoked by pointing the head downwards in the direction of feature 330. This feature is useful if the surgeon 120 is still unfamiliar with the system and can be switched on or off as needed. Similarly, the surgeon 120 may choose to freely enable mode switching only by head gesture without using the foot switch 104, or to enable via the foot switch 104, depending on the circumstances and needs. You may return.

In one embodiment, features 324, 326, 328, and 330 are displayed only when the surgeon 120 presses the footswitch 104 to enable mode switching based on head gestures. Optionally, the display of features 324, 326, 328, and 330 may be the default setting. It should be noted that the display of specific features 324, 326, 328, and 330 is merely exemplary. In general, features are displayed in a manner that corresponds to the current mode. Therefore, different system modes display different features. For example, if the system 100 is in preoperative mode, only one feature instructing the surgeon 120 to turn his head to the right to return to normal mode is overlaid on the image in the field of view. In such cases, the feature 324 currently indicating "nurse" indicates normal and the features 326, 328, and 330 are not displayed.

Surgeon 120 may define and configure a system mode by copying a scratch, or an existing mode, and modifying it with a touch screen 108. Once at least two system modes have been defined, the surgeon 120 wears the HMD 102, stares at the approximate direction in which the mode switch is desired to be triggered next, and the mode associated with each of the zones defined by the threshold. The system mode may be associated with the orientation area by selecting. If the mode is additionally defined by the position in the operating room, the surgeon 120 may physically move to the desired position that causes the system mode switch. This is done in a dedicated system mode with special menus and gestures that support various system settings and configurations. This is optionally done by each new user as an initial setup or configuration step. The user may call and operate the mode designation alone via the HMD menu or with the help of a second person manipulating the GUI on the touch screen.

The default display state is the full screen state. In this state, the HMD 102 displays a fixed image on the full screen display that appears fixed to the surgeon 120. The content displayed via the HMD 102 appears as a screen firmly suspended from the HMD 102. When the surgeon 120 moves his head, the screen moves with him and appears fixed to him. In addition to the full screen state, the system 100 provides a virtual screen state. Surgeon 120 may recall the virtual screen state from the full screen state. The virtual screen state can be invoked using any combination of the user interface methods described above, such as pressing pedal 104B of footswitch 104, using head gestures or voice commands. The virtual screen state changes only the screen state and keeps all other system characteristics of the current mode (eg, displaying the raw image feed) unchanged.

According to a further embodiment of the disclosed technology, switching between system modes is achieved by a virtual screen state. In the virtual screen state, the content is displayed within one or more world-fixed virtual screens that appear fixed in the operating room. As the surgeon 120 moves his head, the virtual screen moves in and out of his field of view, allowing the surgeon 120 to scroll through the content displayed there. Depending on the distance between the surgeon 120 and the virtual position of the virtual screen, the surgeon 120 may see multiple virtual screens at once. This will be described in more detail with respect to FIG. 5C.

To switch between system modes using a virtual screen state, each system mode displays the content that the surgeon 120 will see in full screen when that system mode is selected, and acts as a type of thumbnail. Can be indicated by. This allows the surgeon 120 to simultaneously preview the content available in the various system modes without switching modes. The details and resolution of the content displayed on the virtual screen may differ from what is displayed in the corresponding full screen state due to screen size, technical issues, and bandwidth limitations. For example, an overlay symbol that appears in the full screen state may not appear in the thumbnail image displayed in the virtual screen state. However, the images are substantially similar in both virtual and full screen states. The surgeon 120 may switch the system mode in the virtual screen state using any suitable UI input, such as staring at the virtual screen for a predetermined period of time, or head gesture with pressing the foot switch 104. Not all system modes are represented by a virtual screen, for example system modes with the display turned off.

When the system mode is selected, the screen state automatically switches to the full screen state in the selected mode. The surgeon 120 may configure the system 100 to remain in the virtual screen state after switching modes. In this case, all system characteristics except the screen state change. In one embodiment, the virtual screen state is itself a system mode that allows the surgeon 120 to view several virtual screens at the same time. In this case, all other system characteristics, including user interface characteristics, are uniquely defined for this system mode.

In some embodiments, the system mode may be switched via the touch screen 108. In some applications it is desirable to allow a third party, such as a nurse, to switch to a different system mode. In this case, the nurse is provided with a touch screen coupled to the computer 118. The touch screen 108 may be coated with sterile nylon for use in the operating room by any of the attending sterile surgeons.

In some embodiments, the system mode may be switched via voice commands or other acoustic commands such as tongue tapping. The microphone 138 records one or more acoustic commands issued by the surgeon 120. Computer 118 analyzes acoustic commands to determine various system functions, such as switching system modes. To avoid erroneous activation, the system 100 may be configured to respond only to acoustic commands that begin with the identification sound, for example by uttering the identification name of the system 100. For example, the surgeon 120 may switch to normal mode by reading the distinguished name of the system 100 and instructing the system 100 to switch to normal mode.

According to other embodiments of the disclosed technique, the menu is displayed via the HMD 102. The menu may be invoked by pressing the pedal of the footswitch 104, for example pedal 104B, or by other UI means such as voice control or dedicated head gesture. Invoking the menu allows the currently displayed content to overlay the menu via the HMD 102 and to navigate and select items in the overlay menu via the head gesture. The menu may be displayed as a semi-transparent display fixed overlay to the current image displayed via the HMD 102, or within a separate virtual screen.

For example, one position and orientation may correspond to a menu display according to the concept shown in FIG. 3A corresponding to different positions and orientations of the HMD 102. When the surgeon wearing the HMD 102 looks in a predetermined direction, for example downward, a menu is displayed. In some embodiments, the menu may initially be displayed in a small or collapsible manner so as to occupy less display area. In another embodiment, the menu is displayed away from the front of the HMD 102 (in the folded or normal state), eg, orthogonal to the front of the HMD 102. The menu can be fully opened after the surgeon has been looking in a predetermined direction for a given period of time. In some embodiments, the menu is not recalled if the surgeon points or orients his or her head as part of a head gesture configured to enable other functions. The surgeon 120 navigates and controls the menu displayed as the HMD fixed overlay through the head gesture. In some embodiments, pressing the footswitch 104 brings up a menu, allowing the user to navigate and make various menu items stand out using head gestures, and releasing the footswitch makes the menu item stand out. Activate. For example, to select a menu item, the surgeon 120 presses on an available pedal on the footswitch 104 and manipulates his head to select a menu item. Release of the footswitch 104 activates the action corresponding to the selected menu item.

The menu may list any number of parameters, such as, but not limited to, various system modes available for system 100. Each system mode is represented by a symbol or text, such as a textual representation of the system mode, such as a menu item that includes "normal," "nurse," "preoperative," and so on. Other menu options added to system mode may be included in the menu. Additional menu options may be mode-specific with respect to the controllable characteristics of that system mode. For example, in normal mode, the menu may include items that allow the floodlight to be turned on and off and the red-eye light to be turned on and off. In preoperative mode, these options do not appear, instead the preoperative menu is a sub to enable opening of folders containing available preoperative datasets and select which dataset to display. You may display the menu. The preoperative menu may further include items for controlling various attributes for displaying preoperative data. Menu options may be mode-specific, treatment-specific, and stage-specific, i.e., the menu appearance may vary depending on the system mode and stage during treatment. In addition to switching between system modes, menus allow switching between states.

In some embodiments, the menu is displayed in the margin of the display of the HMD 102 so as not to block the raw image displayed in the center of the display area provided by the HMD 102. When the surgeon 120 navigates through each menu item in the menu displayed in the margin, the menu item is shown to the surgeon 120 where he is on the menu and what the effect of selecting that menu item is. Features can be temporarily enabled. For example, when the surgeon 120 navigates over a menu item that emphasizes green to allow him to see the film sprayed with the green dye, the feature is temporarily enabled and the surgeon 120 takes a raw image. The surgeon 120 may be shown which menu item he is currently navigating and what the effect of selecting that menu item is, while allowing him to keep an eye on. However, some menu items may be associated with features that are not instantly apparent while staring at the raw image displayed via the HMD 102. Non-visual indicators such as sound or vibration can be triggered when new menu items are highlighted to allow the surgeon 120 to keep an eye on the raw image while tracking his menu navigation. .. This allows the surgeon 120 to keep track of the menu navigation by himself without having to look directly at it, so that the surgeon 120 can continue to gaze at the raw image.

In some embodiments, the menu in the right margin of the display is application-specific. Examples of such applications include preoperative OCT applications, teaching applications, preplanning applications, facobitrectomy settings and metrics applications, intraoperative OCT applications, and the like. For example, when you press the footswitch menu button, the menu in the left margin of the display is the main menu, the menu in the right margin of the display is application-only, and you turn on the application by activating the corresponding menu item. Enables. Switching between two branches of a menu (ie, the menu in the left and right margins of the display) is done by turning the head left or right.

In another embodiment, when the application is turned on and then the footswitch menu button is pressed, the menu that appears in the right margin is a dedicated menu for the activated application. For example, when the preoperative OCT application is activated, a PIP with a B scan appears in the display and a straight line indicating the position of the B scan is overlaid on the raw image. When the footswitch menu button is pressed, the menu that appears in the right margin is not a general application menu in the right margin of the display, but a dedicated menu for preoperative OCT applications. Surgeon 120 preoperative OCT to switch to another screen where different OCT scan sets may be selected for display within the PIP, for example to turn on and off the overlay of the straight line indicating the B scan position on the raw image. You may navigate and select menu items from the dedicated menu, such as to turn off the application. When the preoperative OCT application is closed and then the footswitch menu button is pressed, the general application menu appears in the right margin of the display. It should be noted that these examples are provided only to illustrate the range of features of the system 100 that can be controlled via the UI 160.

In a further embodiment, the same footswitch button used to call and operate the menu may also enable screen switching by simultaneously turning the head while pressing the footswitch button. For example, pressing the footswitch menu button and turning the head slightly to the right highlights the first menu item in the menu branch that appears in the right margin of the display. Turning the head further to the right can cause screen switching.

The following examples are given for treatment-specific and stage-specific menu options. During cataract treatment, an external machine (not shown) may be connected to system 100. The external machine introduces data to be displayed as an overlay on the rendered raw magnified video in the display module 130. When the system 100 identifies that an external machine has been connected, the display module 130 displays a menu option for switching between the "enabled" and "disabled" states of the "external data overlay" and "burst guidance". When the system 100 identifies that a treatment step, such as an ultrasonic lens emulsification suction step, has been completed based on the captured data, the display module 130 is "effective" in the "IOL integrity guidance" rather than the "rupture guidance". Shows menu options for switching between "and".

Menus can be used to control features and aspects of the system 100, including selecting and switching between new system modes and returning to the previous system mode. Accessing system modes according to P & O zones only can limit the number of available system modes to the number of zones in the field of view, so the menu allows to increase the number of available system modes. obtain. Thus, one system mode may be a menu mode that presents menu items for selecting another system mode, as well as items for controlling additional system features and parameters. While in menu mode, menus relating to system mode are specific to UI function mode and may be displayed via the HMD 102, i.e. certain gestures and movements by surgeon 120 provide one function while in menu mode. Brings different features in different system modes. The menu mode can be any suitable UI interface, such as pressing pedal 104C on footswitch 104, issuing voice commands recorded by microphone 138, performing head gestures, performing one or more eye movements, etc. Can be called using.

Here, as a whole, is referred to FIGS. 4A-4C showing a typical system mode menu overlaid on the display of the magnified image 414, which is configured and operational according to a further embodiment of the disclosed technique. The exact implementation details described below are intended as typical examples and are not intended to limit the invention. The system mode menu 400 allows the surgeon 120 to switch between system modes while viewing the raw image stream 414. The system mode menu 400 appears similar to the zone layout of FIG. 3A, but this is not a requirement. The surgeon 120 sees the live magnified image stream 414 of the surgical field 124 via the HMD 102 (FIG. 1A) configured in the full screen state. The surgeon 120 performs a default foot gesture, such as pressing the pedal 104B of the foot switch 104. In response, the computer 118 displays an overlaid system mode menu 400 on the magnified image stream 414. Menu items in the system mode menu 400 are transparently overlaid on the magnified image stream 414, for example to avoid obscuring the displayed image.

In the typical implementation shown, the system mode menu 400 has menu items 402 (normal), 404 (vital), 406 (nurse), 408 (surgical field), 410 (preoperative), and 412 (cancelled). including. Since the current system mode is normal, element 402 is emphasized (FIG. 4A). In some embodiments, menu items are spatially arranged according to their function. For example, in an operating room where the nurse is on the right side of the surgical field 124, the menu item 406 (nurse mode) is displayed on the right side of the field of view of the HMD 102 and the menu item 402 (normal mode) is in the center of the field of view. Is displayed. To navigate the system mode menu 400, the surgeon 120 turns his head while pressing the pedal 104B of the foot switch 104.

Referring to FIG. 4B, the surgeon 120 turns his head to the left and intersects the angle threshold that separates the menu items 402 and 410. Emphasis switches from menu item 402 to menu item 410 (preoperative mode). In one embodiment, only emphasis is switched to preoperative mode menu item 410. The system mode has not been switched yet. When the surgeon 120 lifts his foot from the footswitch 104 to release the pedal 104B, the system mode switches to a mode corresponding to the selected menu item 410, eg, preoperative mode (FIG. 4B). In another embodiment, when the highlighted menu item is toggled, the background image also changes as if the system mode was switched, allowing the user to see a preview of the image associated with that menu item.

Referring to FIG. 4C, the surgeon 120 turns his head toward the lower right corner while holding down the pedal 104B. When the angle of his head intersects the angle threshold set for the menu item 412 corresponding to the cancel selection, the emphasis switches to the menu item 412 (cancel). The surgeon 120 selects menu item 412 by releasing pedal 104B of the footswitch and cancels the action resulting in menu 400 by pressing pedal 104B in order to hide menu 400 without changing the system mode. Thus, the surgeon 120 cancels the system mode menu 400 and the system 100 remains in the current system mode.

In this embodiment, the screen state is fixed to the display. When the surgeon moves his head, image 414 remains fixed within his field of view, regardless of his head movement. However, the emphasis on the selected menu item of the menu 400 overlaid on the fixed image 414 shifts with his head movement.

The selected menu item, for example, displays the menu item in various fonts, background colors, displays a cursor pointing to the menu item, or distinguishes the menu item from other unselected menu items. It can be emphasized in various ways by any other suitable technique for. The emphasis is that when the HMD 102 rotates toward the second menu item while the foot switch 104 is pressed, it moves from one menu item to the next menu item. This is shown in FIG. 4B. When the surgeon turns his head to the left, the menu item 410 corresponding to the preoperative mode is selected and highlighted, and the menu item 410 corresponding to the normal is deselected and displayed normally. .. The surgeon 120 may personalize the system 100's response to his head gesture. For example, head movement can be scaled so that a small head tilt turns to a correspondingly large angular shift, or vice versa. Alternatively, the orientation of the HMD 102 controls a cursor (not shown) displayed on the menu 400 that selects a menu item. Manipulating the orientation of the HMD 102 manipulates the cursor, for example, up / down and left / right head movements move the cursor up / down and left / right accordingly. Release of the footswitch 104 activates the system mode corresponding to the currently selected menu item, thereby switching the system mode.

In other embodiments, eye tracking may be performed to control menu 400. The eye movement plays the role of the above-mentioned head movement. Gaze at a menu item may emphasize the menu item, and staring continuously for longer than a predetermined period may activate the menu item.

Additional menus may be displayed via the HMD 102 to select functions other than system mode. Some menu items may be expanded to display one or more submenus when selected. The selection method is to press or release one of the pedals of the footswitch 104, keep the menu item highlighted for longer than the default period, use voice commands detected by the microphone 138, or otherwise. It can be carried out by the appropriate UI method of. When expanded to a submenu, the submenu may replace the original menu or be displayed with the original menu. Menus may also include standard menu operation items such as back / return and exit / cancel.

Some menu items can be turned on and off via UI160. The switch item is a menu option that switches between the two states, and the currently displayed menu option represents the currently inactive state. For example, if the shutter 132 is closed, the menu option for controlling the shutter will appear as "Open Shutter" and vice versa. Examples of switching items in the menu are described below.
○ Basic HMD system operation that can be controlled via the menu ・ Display (on / off). This shuts on or off the display. When the display is turned off, the image is not displayed, but the menu can be invoked (ie, displayed) to allow the user to turn the display back on. Controlling the display state is useful in VGS procedures where the HMD is transparent (ie, if the HMD comprises a shutter, the shutter opens at this stage). In these procedures, the user may wish to turn off the display during the procedure phase, which does not require guidance by the HMD, so that the image does not interfere with viewing the surgical field.
-Full screen / virtual screen. In the full screen state, only one screen fills the FOV, but in the virtual screen state, multiple virtual screens are displayed. The screen state will be further described later with respect to FIGS. 5A-5C.
-Shutter (open / closed). This controls the state of the shutter 132. The shutter 132 may be partially opened and closed, for example to display overlay features in a real world view when most of the screen is transparent.
-Red-eye lighting (on / off).
-Recording (on / off). This controls the features of the system 100 that allow the surgeon 120 to record some of the surgical procedures that may be recalled at a later stage.
○ Advanced HMD operation (valid / invalid), one or more of the following.
・ Autofocus,
・ Automatic centering,
-Automatic zooming.
○ HMD display parameters (on / off).
-PIP on / off switching for each default PIP displayed,
An external data overlay for displaying data acquired externally via the HMD 102.
-Various treatment-specific and stage-specific overlay symbols, images, and models.
-Specified pointer / cursor.
○ iOCT (on / off). This controls the OCT image.

In some embodiments, in addition to the mode-specific menu, the UI 160 displays a dedicated menu for controlling various functions of the system 100. For example, the UI 160 displays different dedicated menus to control each of zoom, focus, lighting, and other functions. Each dedicated menu allows the surgeon 120 to access options related only to specific functions. For example, a menu dedicated to the focus function allows the surgeon 120 to select various autofocus settings.

In some embodiments, the surgeon 120 accesses a menu dedicated to the unique function via the main menu. In another embodiment, the surgeon 120 directly accesses the menu dedicated to the unique function, for example by pressing the button of the foot switch 104 twice to activate the menu. For example, the surgeon 120 may open a menu dedicated to lighting functions by performing two short presses followed by a long press on the button of the footswitch 104 that activates a head gesture to control lighting. Surgeon 120 may use head gestures or eye tracking to navigate dedicated menus as described herein and activate highlighted menu options by releasing the button on footswitch 104. ..

In some embodiments, the surgeon 120 issues a voice command by pressing the relevant button on the footswitch 104 only partially (eg, half-pressing), or by pressing the footswitch 104 while issuing a voice command. The dedicated menu is opened by performing a head gesture such as nodding while pressing the foot switch 104. For example, a "yes" head gesture may control enabled features (eg focus, zoom, lighting, etc.), and a "no" head gesture may open a dedicated menu.

For example, if the surgeon 120 turns his head to the left or right while pressing the footswitch button to enable the focus function, a dedicated menu for controlling the focus function, such as the autofocus feature, will be displayed on the display. Appears in. Through this dedicated menu, the surgeon 120 may choose whether autofocus acts on a specified point, surgical tool, etc., and whether autofocusing is one-time or continuous.

In some embodiments, when enabling an action via the footswitch 104, the surgeon "by moving his head up and down" to control a continuous (ie, smooth) action. Making a "yes" move, but making a left-right move, activates or switches between two individual actions associated with the activated action. Alternatively, a left-pointing head gesture may switch between individual actions, and a right-pointing head gesture may open a dedicated menu (or vice versa).

For example, when enabling "lighting", the upper and lower head gestures change the lighting intensity, and the "leftward" head gesture switches the lighting between two individual levels. For example, one level may be "high" and the other may be "low". Optionally, these levels are pre-determined and may be configured by the user. Optionally, after the surgeon adjusts the light level during surgery, the system 100 remembers the light level and then the same light when the light action is switched (high to low and vice versa). Restore the level. Continuing this example, the "right" gesture switches the lighting between the "on" and "off" states.

As another example, when enabling Focus, the left head gesture may toggle the autofocus feature on or off, and the right gesture opens a dedicated menu to control the focus feature. You can do it. See here are FIGS. 4D-4G showing implementations for displaying one or more status bars indicating system settings that are configured and operational according to embodiments of the disclosed technology. The figures in FIGS. 4D-4G are intended to be merely exemplary, and other system indicators may be added or replaced with the indicators shown in FIG. 4D. Referring to FIG. 4D, the status bar 420 contains a plurality of different symbols indicating the current configuration for different settings of the system 100. The status bar 420 is displayed via the HMD 102, for example, in response to input by the surgeon 120 via the UI 160. The status bar 420 includes a clock 422, a ROI indicator 424, a footswitch indicator 426, lighting indicators 428 and 430, a system mode indicator 432, and a color correction indicator 434. The clock 422 displays the current time. The ROI indicator 424 indicates a display ROI for the entire ROI. In FIG. 4D, the ROI indicator 424 has a small rectangle indicating the display ROI, which is located within a large rectangle indicating the entire ROI, which can occur, for example, if the surgeon 120 uses the zoom-in feature to see a magnified portion of the raw image. It was displayed. A number indicating the current magnification is displayed. The footswitch indicator 426 indicates the available functionality of the footswitch 104. In the example shown, the available features are indicated by four icons. When moving clockwise, the icon in the upper left corner indicates scrolling of the XY motor, the icon in the upper right corner indicates lighting, the icon in the lower right corner indicates focus, and the icon in the lower left corner indicates zoom. In some embodiments, the icon indicating the currently active function of the footswitch 104 is highlighted. Illumination indicator 428 indicates that red-eye illumination is set to "on" and the level is 62. The illumination indicator 430 indicates that the floodlight is set to "on" and the level is 65. The system mode indicator 432 indicates the current system mode, in this example it is the posterior eye mode. The color correction indicator 434 indicates the type of color correction applied to the image, and in this example, no color correction is performed. The status bar 420 may be displayed at any depth, if visible to both eyes, at any suitable position via the HMD 102, for example at the top of the display or any other position set by the surgeon 120. , Depth can be set. In some embodiments, the status bar 420 is displayed only when the parameters are changed. The status bar 420 includes additional system features such as field depth, digital gain, and patient data including name, age, gender, name of surgeon or multiple surgeons, date, location, hospital name or clinic name, etc. Other suitable parameters may be included.

See FIG. 4E, which shows a raw image of the eyes of a patient undergoing surgery, with the status bar 420 overlaying the current system configuration to surgeon 120.

Here, in addition to the status bar 420, FIG. 4F showing the raw image of FIG. 4E overlaid with the second status bar 442 is referenced. In this typical implementation, the status bar 442 is a bow indicating the latest changes to the lighting settings. The status bar 442 is displayed to keep the surgeon 120 up to date on the current settings of the system 100 in response to changes made to the lights.

Here, reference is made to FIG. 4G showing another raw image 444 of the eye of a patient undergoing surgery. A status bar 446 and a status bar 448 are displayed on the raw image 444. The status bar 446 is configured differently from the status bar 420 of FIG. 4E. The status bar 448 appears when an external device, such as a Facobitrectomy machine, is connected to the system 100 and displays the system settings corresponding to the external device. In some embodiments, the user may configure the system 100 to continuously display the status bar 420 or to hide the status bar 420 in response to user input or mode changes. In some embodiments, the default setting is to display the status bar 420 continuously. In some embodiments, the system 100 provides a plurality of different status bars (not shown) in addition to the status bar 420. Each status bar may be configured differently, with one or more status bars configured to display continuously, with one or more status bars responding to any of the parameter changes to the user. Displayed in response to input, in response to detection of an external device, and so on.

In some embodiments, the system 100 may issue a reminder or alarm when the facoprobe is not in the center of the capsular bag. In some embodiments, the system 100 may issue a reminder or alarm, depending on the surgical technique selected, for example if the chopper is not located under the facoprobe. The system 100 may automatically stop aspiration of the surgical instrument if the posterior capsule is aspirated towards the faco, as can be detected, for example via image processing. The system 100 is used by the surgeon 120 to calculate the distance between the patient 120 and the tool using any of the processing of images of the surgical procedure, the analysis of iOCT data, the analysis of preoperative or intraoperative models of the eye, and the like. The surgical tools used may be tracked.

In some embodiments, the system 100 tracks one or more surgical tools, for example by applying image processing techniques to raw images, applying image processing techniques to intraoperative OCT data (iOCT), and the like. It's okay. The system 100 may have a dedicated tool tracking subsystem based on optical tracking, electromagnetic tracking, etc. Optionally, the tool tracking subsystem is also based on preoperative or intraoperative models of the eye.

The tool tracking function can be used for various features. For example, tool tracking allows the user to use a surgical tool that is used as an electronic pen during a surgical procedure. This can be useful in a teaching mode where the supervising surgeon can mark the raw image to guide the surgeon. The UI 160 allows the surgeon 120 to switch drawing features on or off, for example via voice commands, footswitch 104, and the like. Alternatively, a physical tool is paired with the system 100 to press a virtual button or to select a menu item displayed via the HMD 102, eg, around the retina in posterior ocular surgery, or anterior ocular treatment. Can be used to select an item from a menu displayed around the eye or displayed anywhere in the operating room.

The tool tracking feature can be used to detect and indicate the distance between the tip of the tool and the patient, eg, the retina in the case of eye surgery. The tool tracking function can be used to initiate automatic safety measures, such as automatically stopping lens aspiration. The tool tracking function can be used to guide the surgeon 120 during the procedure. These examples are intended as merely exemplary and are not limited to a particular example given this function.

Here, a description of the system characteristics that can be controlled via the user interface 160 is shown below. It should be noted that some system characteristics are indirectly controlled by the user when switching modes. For example, switching to a different mode may position the camera head 110, activate the camera, or change the illuminator. The user may not be able to adjust these settings directly, but only indirectly by switching between the system mode and other settings inherent in the system 100. This list is not intended to be limited. Each system mode is characterized by two or more of the following system characteristics:
-Display status (on / off),
-Screen status (display / world fixed),
-Shutter state (open / closed),
-Selection of display content (raw image / preoperative / other),
-Lighting conditions,
-Color correction scheme,
-Image enhancement scheme,
-Eye inversion,
-Motor speed state,
-Motor boundary state,
-Display (on / off) picture-in-picture (PIP)
-User interface settings,
-Display overlay, and-iOCT settings.

The display state for the HMD 102 may be set to "on" or "off" and the default display state for the module 130 is "on". When the display is turned off, the computer 118 shuts down the image source for the display modules 130A and 130B. Alternatively, the computer 118 streams the black image to the HMD 102.

The screen condition with respect to the HMD 102 defines how the image and content are displayed to the surgeon 120. With respect to the HMD 102, there are two screen states, a full screen display fixed state and a virtual screen world fixed state. In the full-screen display fixed state, the image appears as displayed on a screen firmly suspended from the HMD 102, and the displayed image moves with the surgeon 120 while remaining fixed. This is the default state. In the world fixed state, one or more virtual screens are displayed as if they were fixed in the operating room. When the surgeon 120 moves his head, the display content changes such that a portion of the display content moves in and out of the FOV of the HMD 102.

Here, reference is made to FIG. 5A showing a projected image as perceived by the surgeon 120 when the display module 130 is in a fixed display state, configured and operable according to other embodiments of the disclosed technique. FIG. 5A shows images 500 and 502 displayed in a fixed state on a full screen display. The field of view of the HMD 102 is filled with a live magnified image acquired via the camera system 112 and is displayed as if it were fixed to the head of the surgeon 120. Elements 512 and 514 indicate the orientation of the surgeon 120's head. Dashed arrows on elements 512 and 514 indicate frontal orientation. The solid arrow of element 512 with respect to image 500 indicates that the surgeon 120 is oriented 5 ° to the right of the front facing orientation. The solid arrow of element 514 with respect to image 502 indicates that the surgeon 120 is oriented 25 ° to the right of the frontal orientation. By moving one's head to turn, the image that appears in the field of view does not change as if the screen is fixed to one's head, but moves with oneself at the same time as performing a head gesture. ..

Here, as a whole, displayed in a virtual screen world fixed state via the HMD 102, each configured and operable according to a further embodiment of the disclosed technique, each corresponding to a different position and orientation of the surgeon 120's head. See FIGS. 5B-5C showing the two views 504 and 506 made. In the views shown in FIGS. 5B-5C, the two adjacent virtual screens 508 and 510 are displayed with the same (virtual) size from the same distance from the surgeon 120. In other embodiments, multiple virtual screens with different (virtual) sizes, positions and distances may be used. Surgeon 120 simultaneously views images displayed through both world fixed virtual screens 508 and 510. The virtual screen 508 displays a live video of the surgical field, and the virtual screen displays preoperative data.

Each of views 504 and 506 contains a raw image feed displayed via virtual screen 508 and corresponding preoperative data displayed via virtual screen 510. The virtual screens 504 and 506 appear as if they were monitors installed in a fixed position within the operating room (OR). As shown in FIG. 5A, the elements 512 and 514 of FIGS. 5B-5C indicate the orientation of the head of the surgeon 120, respectively. Dashed arrows on elements 512 and 514 indicate frontal orientation. The solid arrow of element 512 with respect to view 504 (FIG. 5B) indicates that the surgeon 120 is oriented 5 ° to the right of the frontal orientation. The solid arrow of element 514 for view 506 (FIG. 5C) indicates that the surgeon 120 is oriented 25 ° to the right of the frontal orientation.

As shown, view 506 of FIG. 5C appears as if shifted to the left with respect to view 594 of FIG. 5B, which corresponds to the surgeon 120's rightward head movement. The virtual screen 508 in view 506 (FIG. 5B) shows a smaller portion of the live video than the virtual screen 508 in view 504 (FIG. 5C), and the virtual screen 510 in view 506 (FIG. 5C) is view 504 (FIG. 5C). Shows most of the preoperative data than the virtual screen 510 in FIG. 5B). By moving his head from side to side, the surgeon 120 can scroll through the data displayed simultaneously on the virtual screens 508 and 510 to see more or less of each as needed. .. Implementation of a precise world fixed screen can be achieved using a 6 degrees of freedom (DOF) head gesture tracker with tracking components 134 for performing global position and orientation tracking. In some embodiments, the virtual screen mode may be implemented with a 3-DOF tracker that only supports head orientation, for example if the absolute position of the screen is not required.

Here, reference is made to FIG. 5D showing a typical implementation of a virtual screen state configured and operational according to other embodiments of the disclosed technology. Surgeon 120 of FIG. 1A wears the HMD 102 and observes the two virtual screens 508 and 510 of FIGS. 5B-5C while operating on patient 122. The virtual screen 508 displays the raw image feed acquired by the cameras 140A and 140B (FIG. 1E), and the virtual screen 510 appears preoperatively as if it were displayed on another monitor located in the operating room. Project the data. The content displayed via the HMD 102 may be determined by the system mode. Options for displayed content include raw image streams (default) acquired via the camera system 112, preoperative data (eg medical images and / or data about the patient), intraoperative data (ie, eg, Facobitrectomy system). Data from other devices such as), raw image streams provided by an endoscopic camera (not shown) may be included. In some embodiments, the option further allows the surgeon 120 to view the raw image stream taken from a similar system in the adjacent operating room, allowing the surgeon 120 to track the progress of parallel surgery, in the waiting room. It provides a live image stream provided by an installed Wi-Fi camera, browsing a cloud-based library of images and videos for reference during surgery, and more. The video acquired by the cameras 140A and 140B and projected via the HMD 102 is 3D. Each eye of the surgeon 120 sees a video feed from the corresponding camera. Other video and image displays via the HMD 102 may be either 2D or 3D. For example, video from a cloud-based library may be 3D and preoperative images may be 2D.

The shutter state of the shutter 132 of the HMD 102 (FIG. 1C) may be open or closed, and the default state in microsurgery is the closed state. When the shutter 132 is closed, the display module 130 is opaque and the surgeon 120 sees the image displayed through the HMD 102 without seeing the ambient light behind the display image which reduces the contrast of the display image. Make it possible. When the shutter 132 is open, the display module 130 is transparent, allowing the surgeon 120 to see the real world view. When the shutter 132 is partially open, the display module displays the background behind the picture-in-picture (PIP) when displaying one or more features within the PIP, for example in an augmented reality application during a VGS procedure. The portion where the shutter 132 is open is transparent and the portion where the shutter 132 is closed is opaque in order to conceal and enhance the contrast of the image in the PIP.

Individual lighting states for the lighting system 114 include selecting floodlights on, off, or low, selecting coaxial lighting on, off, or low, and selecting automatic lighting conditions. The lighting system 114 further includes a continuous lighting condition. When automatic lighting is enabled, the system 100 automatically adjusts the lighting as a function of image histogram analysis, other image processing analysis, HMD102 position and orientation, surgical procedure progress, and additional parameters. Settings for automatic lighting include enable and disable (default).

When the automatic centering state is enabled, the system 100 will automatically center the magnified image according to the default settings that depend on the system mode. For example, in anterior eye mode, centering may be performed using an XY motor, and in posterior eye mode, centering may be performed digitally by changing the display ROI obtained from the entire imaging frame, or by the patient's eyeball. This can be done by using an XY motor when in motion. Centering is based on image analysis according to the default settings that define the centering characteristics. Settings related to automatic centering are
○ Valid,
○ Disabled (default)
including.

When auto-focusing is enabled, the system 100 will automatically focus on the magnified image according to user preference. The user may specify a point in the surgical field that the system then locks and remains in focus (if the camera head or patient moves). Alternatively, the user may choose to automatically focus on the tip of the tool. In the posterior eye procedure, the user may choose to automatically focus on the center of the illuminated area. Settings related to automatic focusing are available.
○ Lock,
○ Tooltip,
○ Center,
○ Disabled (default)
May include.

Various color correction schemes can be enabled via system mode. For example, the option is
○ Correction for floodlight (default),
○ Correction for various fiber lighting sources,
○ Includes corrections for enhancement when using various dyes, and ○ user-configurable color schemes.

The eye inversion state is indirectly controlled via the system mode. In the normal state (default), the video stream from the left camera 140B is streamed to the left eye of the surgeon 120, and the video stream from the right camera 140A is streamed to the right eye. The inverted state is commonly used for posterior ophthalmic procedures with non-contact lenses. In the inverted state, the video stream from the left camera 140B is streamed to the surgeon 120's right eye and the video stream from the right camera 140A is streamed to the left eye. The image is also rotated 180 degrees.

The motor speed state defines parameters for various motors of the system 100 that control camera focus for the camera system 112, X, Y, and Z positions for the camera head 110 and the like. Options for this state include high speed / low speed.

The motor boundary state limits the degree of freedom for each of the motors integrated within the camera head positioner 111. Options include full motion and limited motion.

The display of the picture-in-picture (PIP) is controlled by the surgeon 120. Options for each PIP include off (default), on, and more. Surgeon 120 defines some default options for the number of PIPs and PIP location, which options are valid in each system mode, and what content is displayed in each PIP in each system mode. You may decide.

User interface settings that define which actions correspond to which user inputs can be customized by the surgeon 120 or by other operators. In some embodiments, various settings for the user interface are controlled via the touch screen 108. Options for user interface settings include:
○ Settings related to the foot switch 104. These settings define which command belongs to each button or pedal 104A, 104B, and 104C of the footswitch 104.
○ Head gesture definition. These settings define which command belongs to each head gesture by the surgeon 120.
○ Menu definition. These settings define which options appear when the menu is called by the surgeon 120.
○ Voice command settings. These settings toggle the voice command function state (on / off) and allow the surgeon 120 to switch the voice control feature of the system 100 on or off as needed. Voice command settings can be personalized using known techniques such as machine learning to familiarize the computer with the commands preferred by the surgeon 120.

It should be understood that this list is just a classic and is not intended to be exhaustive.

The computer 118 controls the display of various overlay symbols to guide the surgeon 120 during the surgical procedure. The display overlay settings define which symbols are overlaid on the image displayed via the HMD 102 and can be customized by the surgeon 120.

Additional settings may control other system characteristics, such as attaching an additional camera or sensor (not shown) to the camera head 110. For example, through the iOCT setting, the surgeon controls the iOCT image acquisition. Settings include off (default), on, iOCT scanner operation, such as mode-specific settings such as anterior-eye vs. posterior-eye mode.

The system 100 allows the surgeon 120 to see the anatomy-based data overlaid on the raw image feed displayed via the HMD 102. Surgeon 120 can customize the overlay of data to the live video feed out of the way. The settings allow customization, such as turning overlay data on and off, showing only part of the overlay, and manually moving the overlay border.

The anatomy-based data is displayed along with the raw magnified image, where "alignment" preserves the spatial relationship between the anatomical-based data and the raw magnified stereoscopic images acquired by the cameras 140A and 140B. It is understood as to do. The use of the term "alignment" herein is intended to capture a broader meaning than is understood in the traditional context with respect to image alignment. Image alignment relates to conversion or alignment of at least a portion of one image with respect to at least a portion of another image. The alignment performed herein further comprises accurately identifying the corresponding positions on the two images using any suitable technique, such as identifying similar features, and not necessarily two. It does not involve matching at least part of the image.

Alignment may be based on pre-calibration and matching of features in anatomy-based data to features detected in images acquired in real time by cameras 140A and 140B. The two cameras 140A and 140B are precalibrated so that the orientation and depth (ie position) of each feature imaged by the cameras 140A and 140B can be calculated and vice versa, the feature or The symbol can be overlaid on each of the raw images to appear in the desired position relative to the cameras 140A and 140B. Alignment as defined herein should not be confused with visor-guided surgery (VGS) or navigation procedures in which a model of anatomical structure is aligned to the tracker coordinate system. In some embodiments, the correspondence between anatomical data and raw images is established using position-related data. Correspondence between anatomical data and displayed images can be stored using techniques other than tracking, for example with position data that maps preoperative and real-time data.

Anatomy-based features are generated by OCT images (both preoperative and intraoperative), 2D maps of the retina, such as reticulated thickness maps, 3D models of the epiretinal membrane derived from volumetric OCT scans, and corneal topographers. It may include a 3D model of the cornea, the desired orientation of the toric intraocular lens (IOL) with respect to the eye, pre-planning information marked on the 2D image of the patient's retina by the surgeon, intraoperative markings made by the supervising surgeon, and the like.

In some embodiments, snapshots of the retina taken at one magnification in one stage of surgery are used as a minimap for the post-stage of surgery performed at higher magnification. During retinal surgery, surgeons generally use fiber illumination that illuminates only a small portion of the retina when held near the retina. Minimaps allow for better spatial cognition when working at higher magnifications. The minimap is displayed within the PIP, and the overlaid circular (or other) symbol on the minimap indicates the location of the illuminated area on the snapshot. The real-time alignment algorithm makes it possible to maintain the alignment of the raw video feed to the snapshot even if the eye or camera head is moved. This technique is also useful for endoscopic procedures in which the endoscopic video is presented in full screen, while the symbols representing the video footprint are overlaid on the minimap displayed within the PIP. Alignment of endoscopic video to snapshots may be based on either feature matching or endoscopic tracking.

In some embodiments, symbols are overlaid on the live video to indicate the data position of the image displayed in the PIP. The system 100 allows the surgeon 120 to customize this feature, for example by controlling the PIP transparency, and to customize the position of the PIP in the FOV of the HMD 102. For example, one such symbol may be a straight line in the raw image of the patient's anatomy that is overlaid on the raw magnified image stream and corresponds to the preoperative image displayed in the PIP. .. This straight line represents the position and angle of the preoperative OCT B scan displayed in the PIP with respect to the raw image.

In some embodiments, symbols are overlaid on the image displayed in the PIP to indicate the location of the features that appear in the live video. One example is the overlaid symbol on the snapshot used as a minimap, as described above. As another example, a straight line representing the angle of the toric IOL appearing in the raw magnified image can be overlaid on the preoperative image of the eye displayed in the PIP. The angle of the straight line is corrected in real time with respect to both the rotation of the eye and the momentary angle of the IOL inside the eye.

The user interface 160 provides to view a plurality of images displayed within the overlayed PIP or the plurality of PIPs on top of the display of the raw image feed. For example, such images may relate to preoperative data, OCT images, notes, and intraoperative raw data such as B-scanned images from iOCT, video acquired via an endoscope. Through the menu, the surgeon 120 controls the PIP position at a position of his choice, for example, the upper left corner of the field of view. This feature allows the surgeon 120 to view preoperative or raw data with a raw image feed that occupies a central area of his field of view. The size and location of the PIP, along with the content displayed therein, is set by the surgeon 120 via the UI 160.

The figure below allows the surgeon 120 to enter and exit the preoperative system mode by manipulating menu items in the displayed menu and view preoperative data within the PIP with symbols aligned to the raw image. It is intended to show a typical implementation of a user interface implemented as an overlaid menu in the field of view. Some menu items represent the actions that take place when selected, while others represent additional menus that result in an additional set of actions. The surgeon 120 navigates the user interface with the footswitch 104 and the head gesture as described above.

Here, a menu 630 overlaid on the raw image feed 614, configured and operational according to a further embodiment of the disclosed technology, which allows the user to interact with the system and switch to various system modes. FIG. 6A, which is shown, is referenced. Menu 630 displays the following menus, overlay 618, autofocus 620, color correction 622, autocentering 624, PIP626, system mode 628, and cancel 612 as system settings configurable by the surgeon 120.

The overlay setting is currently turned off, as indicated by the "off" displayed in menu item 618, indicating to the surgeon 120 that the overlay can be turned on by selecting item 618. The autofocus setting is currently turned off, as indicated by the "off" displayed in menu item 620, and presents the surgeon 120 to switch the overlay on by selecting item 620. Similarly, the color correction is currently set to red, as indicated by the "red" displayed in the color correction menu item 622, and the surgeon 120 switches to a different color correction by selecting this menu item. The option is presented. The auto-centering setting is currently set to "off", as indicated by "off" displayed in menu item 624, and the surgeon 120 has the option to turn this setting on by selecting this menu item. To present. The PIP setting is currently set to "off", as indicated by "off" displayed in menu item 626, and the surgeon 120 has the option to turn this setting on by selecting this menu item. Present. Cancel 612 and system mode 628 are each provided for surgeon 120 to exit the menu and switch to a different system mode. The system mode menu item 628 is currently highlighted, indicating that the surgeon 120 can now select this menu item.

In one embodiment, the "Color Correction" menu item is highlighted and a submenu containing various color correction options is opened, one of which allows the color correction feature to be switched off. In some embodiments, when a menu item in a color correction submenu is highlighted, the effect of the color correction scheme corresponding to the highlighted menu item is before the surgeon is selected by releasing the footswitch 104. It is implemented and displayed on the raw image. This allows the surgeon 120 to see its effect before choosing one of the various color correction schemes. This feature can be enabled for additional image processing features such as image sharpening.

In some embodiments, the opened menu is displayed around the image rather than in the center, and the surgeon 120 evaluates the effects of various image processing and color corrections without the menu covering and blocking a portion of the image. Allows you to. Menu items can be displayed vertically around them. However, navigating the menu does not require the surgeon 120 to shift his gaze from the raw image. To scroll and highlight various menu items, the surgeon 120 moves his head up and down while keeping an eye on the central area of the raw image.

The surgeon 120 navigates between various menu items while pressing the foot switch 104 and moving his head to direct his gaze between the menu items. For example, to select the system mode menu item 628, the surgeon keeps the menu item 628 highlighted (eg, "long-term gaze") for a predetermined period of time while pressing the footswitch 104. This results in the system mode menu shown in FIG. 6B below, which folds the currently presented advanced menu into item 616 around the field of view, and the surgeon 120, as appropriate, directs his gaze while pressing the footswitch 104. By attaching it, it is possible to return to this menu. Therefore, in one implementation, in order to navigate between different menus and open submenus, the surgeon 120 highlights the various menu items that represent the collapsed menu for a given period of time while pressing the footswitch 104. do. Release of the footswitch 104 is not necessary to navigate between the various menus. However, in order to select a menu item for activating or deactivating system settings associated with the selected menu item, the surgeon 120 releases the foot switch 104 while looking at the menu item. Therefore, enabling the footswitch 104 is necessary to perform actions in the system 100, not to navigate between menu items that represent possible actions that can be performed.

Referring to FIG. 6B, a system mode menu overlaid on the raw image feed after the system mode menu item 628 of menu 630 of FIG. 6A, which is configured and operational according to other embodiments of the disclosed technique, is selected. 600 is shown. When the menu item 628 of FIG. 6A is selected by staring at the menu item 628 for a certain period while pressing the foot switch 104, the system mode menu 600 corresponding to the system mode menu 400 of FIGS. 4A to 4D becomes the menu item 630 of FIG. 6A. Is displayed instead of. The previously displayed "advanced" menu 630 shown in FIG. 6A is collapsed into a menu item 616 displayed around the field of view, for example, in the upper left corner, as shown in FIG. 6B. The system mode menu 600 includes a non-contact lens menu item 602, a vital menu item 604, a nurse menu item 606, a surgical field item 606, a preoperative menu item 610, and a cancel menu item 612, which are overlaid on the raw image 614. indicate. Preoperative menu item 610 is highlighted to indicate that this menu item can now be selected by surgeon 120.

Referring to FIG. 6C, the preoperative image is displayed in an overlaid PIP on a raw image feed that is configured and operational according to a further embodiment of the disclosed technique. Selecting menu item 610 (“preoperative”) in FIG. 6B by releasing the footswitch 104 switches the system mode to preoperative mode. In this mode, the user chooses to display the folder of preoperative OCT images in the PIP. In some embodiments, this mode allows the surgeon 120 to adjust and define the PIP according to user-defined settings such as size and position. After selecting which preoperative image to view within the PIP, the surgeon 120 returns to non-contact lens mode using the footswitch 104 and the menu navigation technique described above. Returning to the non-contact lens mode, the preoperative image 632 is displayed in the PIP at the upper left hand corner of the display of the raw image 614, as shown in FIG. 6C. The preoperative OCT image is of the same tissue as displayed in the raw image 614. It should be noted that the location and size of the PIP is merely exemplary and may be user-defined by the surgeon 120.

Referring to FIG. 6D, a menu 630 of FIG. 6A, superimposed on a raw image feed with PIP, configured and operational according to other embodiments of the disclosed technique is shown. The surgeon 120 reappears the menu 630, for example by pressing the foot switch 104. The menu 630 is displayed overlaid on the raw image 614 together with the PIP 632 displayed in the upper left hand corner. The overlay menu item 618 is highlighted to indicate that the surgeon 120 has selected this menu by performing a head gesture while pressing the footswitch 104. The overlay setting state is currently off, as indicated by the "off" displayed in menu item 618, and requires the surgeon 120 to switch this setting on by selecting menu item 618. ..

Here, as a whole, is referred to FIGS. 6E-6J showing a series of raw images 614 with corresponding content displayed within PIP632, configured and operational according to other embodiments of the disclosed technology. 6E-6J show an overlay 634 superimposed on the raw image 614. The overlay 634 is a straight line indicating a position on the raw image 614 corresponding to the preoperative OCT image displayed in the PIP 632. The raw image 614 is a 3D image, i.e., because each eye of the surgeon 120 sees a slightly different image forming the 3D image 614 together, the straight line 634 is the two images projected on both eyes of the surgeon 120. It is individually overlaid for each of. As a result, the surgeon 120 sees a straight line 634 overlaid with the correct sense of depth on the surface of the 3D image 614 showing the retina.

In some embodiments, the computer 118 stores a plurality of preoperative images in a library. When each image is displayed within PIP632, the position on the raw image 614 corresponding to the image is indicated by an overlay, such as a straight line 634. The tissue location indicated by each overlay 634 corresponds to a preoperative OCT image simultaneously displayed within PIP632. The user scrolls within the set of preoperative images by simultaneously performing head gestures (eg, head-down rotation for scroll-down and head-up rotation for scroll-up) while pressing the pedal of the footswitch 104. .. When the preoperative OCT image displayed in PIP632 changes, the corresponding position on the real-time image is calculated and displayed by the overlay straight line 634. In the image shown, the position on the real-time image shifts correspondingly downward. The calculation for the corresponding position on the real-time image is per frame when the image 614 is raw and the overlay requests a dynamic update, even if the preoperative OCT image displayed in PIP632 has not been modified by the surgeon 120. Can be done in. This calculation is based on data generated by a preoperative OCT imager with image positions corresponding to the preoperative image of the retina and all preoperative OCT images. Using this preoperative image of the retina, the corresponding position on the real-time image 614 can be calculated. This feature allows the surgeon 120 to use a handless gesture to scroll through the preoperative data within the PIP 632 with the corresponding overlay on the raw image feed 614. It should be noted that the display of the OCT image in the PIP 632 is intended to be merely exemplary and is not intended to limit the invention. The PIP 632 may display other related content corresponding to the position on the raw image 614 indicated by the straight line 634.

In some embodiments, the surgeon 120 may freeze the raw image while looking at the preoperative data in the PIP. This can be useful for drawing features or symbols on the raw image while the PIP is displaying the preoperative data along with the raw image feed. Freezing an image during drawing is useful if the live video shows movements resulting from, for example, unconscious Sakadick eye movements, breathing, and so on. The drawn symbols can be used to guide the surgeon 120 to subsequent actions, such as an incision in the incision, after unfreezing the raw image feed.

In some embodiments, the surgeon 120 may specify a point in the image displayed via the HMD 102 via the UI 160. For example, the head gesture enabled by the footswitch 104 may move the cursor to specify a point on the image. Alternatively, points may be designated on the image by tracking the eye movements of the surgeon 120. The designation may serve various purposes, for example:
-Specify a point for manual alignment. Corresponding points in both PIP and live video can be specified for manually aligning the images in PIP.
-Specify a point to locate a point in the aligned image. After the image (or model) has been aligned, the surgeon 120 can specify a point in the live video, highlighting the corresponding point in the aligned image and vice versa.
-Specify a point to call PIP. The small PIP may be called near a point designated by the surgeon 120, and the content of the PIP may be associated with the designated point, where the surgeon 120 focuses on a point within the surgical field 124 and that point. Allows you to see related data in the vicinity of.
-Specify a point to control the region of interest (ROI) of the iOCT scanner 142. One or more 2D scans or 3D volumetric iOCT images are displayed with the live video, and the surgeon can specify a symbol, such as a cross, in the video acquired by the camera head 112 to ensure that the iOCT scanner 142 is accurate. Control the ROI.
-Specify a point for autofocus. Autofocus can be activated based on the ROI around the specified point.

In some embodiments, the surgeon 120 controls the overlaid symbol on the raw image feed via a head gesture, and the preoperative image corresponding to the position of the overlaid symbol is displayed in the PIP. As the position of the symbol changes, the corresponding image displayed in the PIP changes. In some embodiments, if there is no preoperative image corresponding to the selected position on the raw image feed, the computer 118 selects the preoperative image closest to the selected position and sets the distance to the surgeon 120. Shown in.

Here, as a whole, is referred to FIGS. 6K-6P showing a series of raw images 642 with corresponding content displayed within PIP644, configured and operational according to other embodiments of the disclosed technology. The embodiments of FIGS. 6K-6P are substantially similar to those described above with respect to FIGS. 6E-6J, except that the boundary line 646 of PIP644 is used instead of the straight line 634 of FIGS. 6K-6P. , The correspondence between the content shown in PIP644 and the tissue displayed in the raw image 642 is shown. In the examples of FIGS. 6K-6P, PIP644 displays an OCT cross-sectional view of the tissue shown in the raw image 642, which is intended to be merely exemplary and is not limiting the invention. Other content may be displayed within PIP644 if it may be valid. The OCT images shown in PIP644 were acquired preoperatively or intraoperatively and stored in memory 118D (FIG. 1B) for later use. During the procedure, the HMD 102 displays the surface of the raw tissue as a raw image 642. The raw image 642 may be a real-time video display of the tissue of patient 122, or as described below with respect to FIGS. 6R-6V, to allow the surgeon to draw features, for example for further use. , May be a frozen part of a real-time video.

When superposed on the raw image 642, the HMD 102 displays, within the PIP 644, a previously acquired OCT cross section of the raw tissue at the position indicated by the boundary line 646. FIGS. 6K-6P are overlaid on the raw image 642 at a plurality of various positions starting from the upper left position of the raw image 642 (FIG. 6K) and moving downward and to the right in the raw image 642 (FIGS. 6L-6P). PIP644 is shown. For each position of PIP644 on the raw image 642, the OCT image displayed within PIP644 at that position corresponds to the cross section of the tissue indicated by the boundary line 646. Thus, by manipulating the position of the PIP 644 on the raw image 642, the surgeon 120 may see various cross-sectional slices of the tissue.

Reference is now made to FIG. 6Q showing the raw image 642 of FIGS. 6L-6P having the corresponding content displayed within two PIP 644s and 648 that are configured and operational according to a further embodiment of the disclosed technique. .. The embodiment of FIG. 6Q is substantially similar to that described above with respect to FIGS. 6L-6P, but it should be noted that two PIPs 644 and 648 are superposed on the raw image 642 and each PIP is respectively. The content corresponding to the organization indicated by the boundaries 646 and 650 is shown. PIP644 shows an OCT cross-section slice of raw tissue at the position indicated by the boundary line 646, and PIP648 indicates an OCT cross-section slice of the tissue at the position indicated by the boundary line 650. As mentioned above, the surgeon may manipulate the positions of PIP 644 and 648 on the raw image 642 to display cross-sectional slices of tissue at various locations.

Here, as a whole, reference is made to FIGS. 6R-6V showing implementation of a drawing function that is configured and operational according to a further embodiment of the disclosed technique. 6R-6V show the result of drawing the symbol using the drawing function displayed via the HMD 102. In one embodiment, the drawing is performed while wearing an HMD that displays a raw image of the patient 122, such as the HMD 102. The drawing function is activated via one or more menus displayed via the HMD, allowing the wearer of the HMD to select a drawing tool (not shown) using the techniques described herein. Enables. The drawing tool is displayed on top of the raw image, and the HMD wearer can operate the drawing tool using one or more of the head gesture, foot switch 104, wand operation, eye movement tracking, and so on. Allows you to draw overlaid symbols on the raw image. Alternatively, drawing may be performed via a user interface (not shown), mouse or wand, etc. provided to the system 100.

In one mode of motion, the surgeon 120 draws features during the preoperative planning phase, for example for later use as a guide during a medical procedure. In this case, the surgeon 120 sees the drawing menu and tools displayed on the raw image via the HMD 102, along with any preoperative data. The surgeon 120 draws one or more preliminary symbols on the raw image. In one embodiment, the raw image feed is frozen during drawing to improve accuracy. The drawn symbol remains displayed with the raw image after the raw image feed is unfrozen and the preoperative data is no longer displayed when drawing is complete, and the surgeon 120 is on the raw image feed while performing medical procedures. It is possible to see the drawing symbols overlaid on.

In other modes of operation, such as the teaching mode, the supervising surgeon (not shown) draws a symbol to guide the surgeon 120 to perform subsequent procedures. In the teaching mode, the supervising surgeon may be in the operating room with the surgeon 120 and draw the symbol on the spot. Alternatively, the supervising surgeon may be located elsewhere and draws the symbol remotely. The supervising surgeon may draw the symbol either via another HMD, via a smartphone, tablet, or other touch screen, or via a mouse or wand with a standard monitor.

Referring to FIG. 6R, a raw image 660 of the eyes of patient 122 is shown before the symbol is virtually drawn. Referring to FIGS. 6S-6T, after selecting the virtual drawing tool from the paint menu (not shown), the surgeon virtually draws the symbol 662 on the raw image 660, for example to indicate the position of the incision. Symbol 662 is shown in the partially drawn state in FIG. 6S and in the fully drawn state in FIG. 6T. As mentioned above, the symbol 662 may be drawn by the senior surgeon wearing the second HMD during the teaching mode, or by the surgeon 120 wearing the HMD 102. Symbol 662 is drawn using any suitable UI technique, eg, as referenced herein. For example, in order to draw the symbol 662, the surgeon 120 presses the footswitch 104 while moving the virtual drawing tool with a head gesture that follows the trajectory corresponding to the shape of the symbol 662, and when the drawing of the symbol 662 is completed ( (Not shown), the foot switch 104 is released. Alternatively, the surgeon may use the mouse or his or her finger to draw on the raw image displayed via the touch screen. The surgeon who draws on the raw image may optionally not be in the operating room and has various UI methods for drawing from a remote location (HMD and footswitch to display the 3D raw image in his office). , Or drawing on a tablet displaying a 2D raw image, etc.). Referring to FIG. 6U, the surgeon 120 uses the symbol 662 drawn and overlaid on the raw image 660 as a guide for incising the eye of patient 122. The surgeon 120 places the surgical instrument 664 in a physical position in the eye of the patient 122 that corresponds to the position indicated by the symbol 662 on the raw image 660. Referring to FIG. 6V, the surgeon 120 moves the surgical instrument 664 along the path corresponding to the symbol 662 to make an incision in the eye of patient 122. Thus, the surgeon 120, or senior surgeon, draws a virtual path on the raw image 660 before making the incision to ensure that the subsequent incision is done correctly.

After the surgeon 120 draws one or more symbols or markings to highlight a particular feature, the surgeon 120 displays which markings while looking at the raw image to proceed with the procedure with the markings for guidance. You may choose whether to do it. Surgeon 120 may scroll through any number of pre-painted markings during the procedure, depending on need and validity. When the markings are no longer used, the system 100 may be configured to erase the markings after a predetermined period of time to prevent them from obscuring the display of the raw image. In one embodiment, according to the default setting (eg after 10 seconds), the symbol disappears automatically. In other embodiments, a dedicated pre-planning application menu allows the surgeon 120 to disable (ie, turn off) each of the markings.

See here, as a whole, FIGS. 7A-7E showing a typical implementation of a menu-driven user interface for focusing on a selected field of view area, configured and operable according to a further embodiment of the disclosed technique. Will be done. In the figures described below, the user interface is implemented as a series of menus that are overlaid on the raw image stream. The surgeon 120 navigates the menu using the footswitch 104 in combination with the head gesture, as described above.

Referring to FIG. 7A, the menu 730 displays an overlaid menu 730 on the raw image feed 714 to focus on a selected field of view area that is configured and operational according to embodiments of the disclosed technique. .. The menu 730 corresponds to the menu 630 shown in FIG. 6A described above. Like menu 630, menu 730 includes menu items for overlay 718, autofocus 720, color correction 722, autocentering 724, PIP726, system mode 728, and cancel 712. Menu item 720 for autofocus features is highlighted to indicate that this item may currently be selected by the surgeon 120. The autofocus feature is currently off, as indicated by the "off" setting displayed in menu item 720, and the surgeon 720 turns on the autofocus feature by "pressing" and activating menu item 720. I want to switch to. In some embodiments, the surgeon 120 is by pressing the foot switch 104 without releasing the foot switch 104 while staring at menu item 720 (FIG. 7A) for a predetermined period of time, for example 0.5 seconds. Turn on autofocus by highlighting the default time period menu item 720. In another embodiment, the surgeon highlights menu item 720 and releases the footswitch 104 before turning on autofocus.

Referring to FIGS. 7B-7D, a typical implementation of a user interface screen that allows to specify a field of view area selected for focus, configured and operational according to a further embodiment of the disclosed technique. Shown. When autofocus 720 is activated to turn on this feature, the UI screen 740 is overlaid with the display of image 714, replacing the menu 730 shown in FIG. 7A. The UI screen 740 includes a prompt 742 and a designation symbol 744, which is shown as a set of parentheses in this implementation. In some embodiments, the surgeon 120 turns on autofocus by releasing the footswitch 104. Prompt 742 requires the surgeon 120 to manually specify an area within the raw image 714 to perform autofocus. The surgeon 120 designates an area by manipulating the designation symbol 744 on the display of the image 714. The position of designation symbol 744 can be manipulated using any of the techniques described above, such as performing head gestures in combination with the foot switch 104. Optionally, the surgeon 120 specifies an area of focus by staring at the desired area for a predetermined period of time, for example 0.5 seconds. The eye tracker component 136 (FIG. 1C) tracks the line of sight of the surgeon 120 to detect designated and selected areas. In some embodiments, the system 100 continuously performs automatic autofocus to the designated area based on the tracking of eye movements of the surgeon 120. It should be noted that other symbols may be used as designated symbols depending on the user's preference and circumstances. For example, the designation symbol may be an arrow, a cursor symbol, or the like in order to specify a particular point accurately.

Referring to FIG. 7C, the surgeon 120 moves the designation symbol 744 to the area 746 on the left side of the image 714 which is not fully in focus. Region 746 is indicated by a dashed circular line. The surgeon 120 moves the designation symbol 744 using any suitable UI technique, such as rotating his head to the left while pressing the foot switch 104. Surgeon 120 designates an area 746 that improves focus. If autofocus is activated by highlighting menu item 720 for a given period without releasing footswitch 104, the surgeon gazes at area 746 and releases footswitch 104 to area designation symbol 744. By arranging it in 746, the area 746 for automatic focus is designated. When autofocus is activated by releasing the footswitch 104, the surgeon 120 places the designation symbol 744 in the area 746 for a predetermined period of time, for example 0.5 seconds, to create an area for autofocus. Specify 746. System 100 may tolerate slight movements while designation symbol 744 is located in region 746.

Referring to FIG. 7D, the computer 118 digitally focuses on the area 746 designated by the designation symbol 744. Comparing the area 746 of FIG. 7C before focus with the area 746 of FIG. 7D after focus may show the result of digital focus.

Autofocus can be implemented in various modes. In one mode, when autofocus is activated and a focus point (ie, region 746) is selected, autofocus stays in focus until the autofocus feature is deactivated. do. This can be useful in applications where the position and orientation of the camera head changes to capture different parts of the patient's anatomy. As long as the selected area is in the raw image feed, this area will remain in focus.

In other modes, "autofocus" is a one-time action. Each time the autofocus feature is activated by selecting menu item 720, the designation symbol 744 is displayed and the surgeon 120 is prompted to specify the area of focus. The system 100 does not lock to the designated area, but focuses on the designated area as a one-time action. In this implementation, menu item 720 is written as "autofocus" without specifying "off" or "on".

Here, reference is made to FIG. 7E, another diagram of the menu of FIG. 7A, which is configured and operational according to an embodiment of the disclosed technique and is overlaid on a raw image feed. Menu 730 is displayed again after being called using the techniques described herein. However, at this time, the menu item 720 indicating the autofocus feature displays "on", indicating that the autofocus is currently on. To switch the autofocus off, the surgeon 120 selects menu item 720 using the techniques described above.

Here, as a whole, FIG. 7F-FIG. 7F showing another typical implementation of a menu-driven user interface for focusing on a selected area of the field of view, configured and operable according to other embodiments of the disclosed technology. 7I is referenced. 7F-7I are substantially similar to FIGS. 7A-7D described above. Referring to FIG. 7F, an overlaid menu 750 is shown on the raw image 752. Menu 750 corresponds to menu 630 of FIG. 6A and includes menu items for overlay 754, autofocus 756, color correction 758, autocentering 760, PIP762, system mode 764, and cancel 768. The surgeon 120 turns on autofocus via the UI 160, as described above. Referring to FIG. 7G, the UI screen 768 is overlaid on the raw image 752. The UI screen 768 displays prompt 770 and the symbol 772 (eg, a pair of parentheses). Prompt 770 requests the surgeon 120 to specify a focus point on the raw image 752 by manipulating the symbol 772. Referring to FIG. 7H, the surgeon 120 moves the symbol 772 to specify an out-of-focus feature 774 (eg, a blurred black circle on a white background). Referring to FIG. 7I, feature 774 of the raw image 752 is displayed with a sharper focus than in FIG. 7H. Surgeon 120 can continue the surgical procedure while viewing feature 774 with better resolution than before.

The system 100 further provides a pre-planning step that allows the surgeon 120 to customize the display of data via the HMD 102 to include one or more overlays. Prior to performing the surgical procedure, the surgeon 120 may add one or more symbols and notes to the preoperative image. During the surgical procedure, the added symbols and notes are displayed in an overlay aligned with the live video captured by the camera system 112. For example, a mark indicating the optimal position for photocoagulation is displayed, overlaid, and aligned with the raw image feed. As another example, the surgeon 120 colors the area of the preoperative image to create a color map, such as blue for the attached retina and red for the detached retina. The color map is then overlaid on the live video during the surgical procedure.

The system 100 further allows the surgeon 120 to control the acquisition of snapshots of any raw image taken by the camera head 110 throughout the surgical procedure. The snapshot is stored on computer 118. The stored snapshots are recalled from memory and may be seen by the surgeon on a separate screen or PIP on the display module 130 at a later stage of the surgical procedure to monitor progress or assist guidance. Snapshots can be stored in 2D or 3D format. The snapshot may also be stored as a short video clip that is repeatedly rendered on the display module 130 within PIP, or as a "GIF" format. This GIF format is advantageous for viewing the membrane. In PIP mode (ie, after "selecting" PIP), snapshot activation may only store PIP images. This can be useful for storing iOCT images or endoscopic images displayed within PIP.

For example, snapshots can be taken after the dye has been applied to enhance the film. Even after the dye has been absorbed and disappeared, the surgeon 120 can still see the stained area within the PIP displayed with the live video feed. Since the snapshot is aligned to the live video, the surgeon 120 may specify a point in the PIP as described above, and the corresponding point in the live video is automatically highlighted, colored, or marked with a symbol. To.

The disclosed technique further allows the surgeon 120 to create a GIF. The surgeon 120 starts recording the image acquired by the camera head 110, for example by using the pedal 104 in connection with the head gesture, and stops recording using a similar UI input set. For example, the system 100, when selected, displays a GIF menu item (not shown) that causes the system 100 to record an image about the GIF. To select a GIF menu item, the surgeon 120 gazes at the GIF menu item for a predetermined period of time, for example, 2 seconds, while pressing the footswitch 104. To finish recording the image for the GIF, the surgeon 120 releases the foot switch 104. Surgeon 120 assigns a name, for example, by head gesture to a menu item that describes an identification detail, such as a time stamp with the identity of patient 122, or via voice control. In some embodiments, the system 100 automatically assigns a name to the snapshot or GIF using known techniques. The GIF is stored in memory 118D (FIG. 1B) and can be recalled later and displayed in the PIP as appropriate on a separate screen or by selecting a menu item.

When the system 100 is activated, the surgeon 120 uses coaxial / red-eye illumination rays that overlap around the patient's eye only when the camera head 110 is properly positioned to bring the camera head 110 closer to the design position. It may be placed manually. The surgeon 120 may then instruct the system to automatically fine-tune the position of the camera head 110 to the correct working distance. The computer 118 automatically lifts the high resolution cameras 140A and 140B to a suitable height above the area of interest in the surgical field 124, ie the mechanical arm controller 118G operates the z-axis motor and the controller 118H is in the image. Operate the XY motor so that the patient's eye is centered, set the lenses of the cameras 140A and 140B to the nominal shutter, focus, and zoom values, and the lighting system controller 118I sets the lighting system 114 to the anterior eye setting. Control.

The system 100 may be configured to store system settings for later use. After the surgical procedure is complete, the surgeon 120 may edit the system settings via the touch screen 108 and name the settings for future use. The setting may take precedence over the "default setting" or may appear later as a menu item displayed via a menu on the HMD 102. The touch screen 108 allows the surgeon 120 to select any storage settings that will be displayed later via the HMD 102.

In some embodiments, the system 100 identifies the surgeon 120 when wearing the HMD 102. A user, such as a surgeon 120 or a nurse, selects a surgeon 120 from an existing list of surgeons, and each surgeon in the list is associated with a preconfigured set of settings, as described above. The system 100 uses any suitable technique, such as voice recognition, or an eye tracker camera integrated with the HMD 120, eg, to compare the acquired images of the surgeon 120's eyes with previously acquired images. The surgeon 120 is automatically identified using biometric data, such as by use. If the surgeon 120 is correctly identified, the system 100 is configured with preconfigured settings associated with the surgeon 120.

Similarly, in some embodiments, the system 100 is acquired with respect to a real-time image of the patient 122's eyes and (possibly) patient 122 to verify that patient 122 corresponds to the scheduled surgical procedure. Automatically compare with the preoperative image. This is, for example, to eliminate errors and ensure that the preoperative data used by the surgeon belongs to patient 122 and that the appropriate surgical procedure is performed for the appropriate eye. System 100 warns surgeon 120 of any discrepancies between the preoperative data used and the raw image stream acquired for patient 122.

In some embodiments, the system 100 displays a checklist to the surgeon 120 either at the beginning, during, or at the end of the surgical procedure. The surgeon 120 may manually check the items that are the actions performed from the checklist, or the system 100 may detect the execution of these actions and automatically check the items from the list. The checklist may be customized for each surgeon and may be displayed when identifying the surgeon 120, as described above. The checklist may be displayed on the side screen or as a PIP in a real-time image, and the surgeon may choose to turn it on or off. In some embodiments, the computer 118 (FIG. 1B) has a centering symbol via the HMD 102 to assist the surgeon 120 in properly placing the HMD 102 to allow the surgeon 120 to see the entire FOV. Is displayed. For example, the centering symbol may be a frame, or any other suitable symbol indicating the boundaries of the FOV. The symbol is when the surgeon 120 first wears the HMD 102, when it is detected by the HMD tracker when the HMD is moved after a resting state, or per user command (via any UI method). If the HMD 102 moves or is misaligned, it may be displayed automatically. If the HMD 102 has the ability to change the interpupillary distance (IPD) of the display, this feature may assist in verifying that the IPD has been set correctly and may be adjusted as needed.

In some embodiments, the system 100 detects and corrects effects on non-overlapping images in a 3D digital microscope. This problem arises when the camera captures an object that is not within the designed working distance and the surgeon uses a lens focus motor to set the focus. When this happens, the images produced by the two cameras 140A and 140B (FIG. 1E) do not completely overlap. When these images are streamed to both eyes of the surgeon 120 via the displays 130A and 130B (FIG. 1C), they can result in eye strain. Overlapping parts of the image do not appear as if they came from the same direction in space. The surgeon 120 must squint so that his brain merges the two images into a single 3D image.

This situation is automatic, for example, by using image processing to calculate the boundaries of the overlap of two 2D images, or by using calibration data to derive the actual working distance from the state of the focus motor. Can be detected in. The focus motor state of each camera lens has a corresponding pre-calibrated working distance. This discrepancy can be calculated and taken into account by comparing the actual working distance with the designed working distance.

One solution is to shift the two images to fill in the lost portion of the image with the black bar. In this way, the surgeon 120's eyes see the overlapping parts of the images as originating from the same direction, and the non-overlapping FOV ends are each visible to only one eye (although the displayed image has a black area). If provided, the display unit does not emit any light in the corresponding area of the display).

Here, FIGS. 8A-8B show techniques for the correction effect on non-overlapping images caused by the discrepancy between the actual working distance of the camera and the design working distance, which can be configured and operated according to the disclosed techniques as a whole. Referenced. FIG. 8A shows an optical setup 800 showing a discrepancy between the actual working distance 802 and the design working distance 804. As a result of the discrepancy, the images acquired by the cameras 140A and 140B (FIG. 1E), represented by straight lines 28 and 11, respectively, are misaligned and do not completely overlap, so the images are coming from different directions. looks like.

Referring to FIG. 8B, the results of shift and correction of the images acquired by the cameras 140A and 140B are shown. The grid 810 shows the numerical depiction of the image acquired by the camera 140A displayed in the left eye of the surgeon 120, and the grid 812 shows the numerical depiction of the image acquired by the camera 140B displayed in the right eye of the surgeon 120. show. Due to the discrepancy between the actual working distance and the design distance for the cameras 140A and 140B, the two rightmost rows of the grid 810 are not included in the grid 812 and the two leftmost rows of the grid 812 are not included in the grid 810. Returning to FIG. 8A, the endpoint of the straight line corresponding to the image captured by the camera 140A, which does not overlap with the straight line corresponding to the image captured by the camera 140B, shown 28, is not included in the grid 812. Corresponds to the upper right corner of the grid 810. The endpoint of the straight line corresponding to the image captured by the camera 140B, which does not overlap with the straight line corresponding to the image captured by the camera 140A, indicated as 11, is the upper left corner of the grid 812 that is not included in the grid 810. Corresponds to.

To correct for this discrepancy, the grid 810 is shifted one column to the right as shown as the grid 814, and the grid 812 is shifted one column to the left as shown as the grid 816. The leftmost column 814A of the grid 814 is blacked out because this part of the image is not visible to the surgeon 120, and the rightmost column 816A of the grid 816 is blacked out because this part of the image is not visible to the surgeon 120. The grid 818 shows the overlapping features of both grids 810 and 812 visible to the surgeon 120 in the center (white background). The leftmost column 818A shown on a gray background is visible only to the right eye of the surgeon 120, and the rightmost column 818B shown on a gray background is visible only to the left eye of the surgeon 120.

The shift technique described above may be performed continuously while the cameras 140A and 140B are in focus. Alternatively, the shift technique may only be performed when the focus adjustment is complete, eg, when the activation pedal of the footswitch 104 is released. In this case, the shift technique is performed gradually over a short period of time, eg, a few seconds, so that the surgeon 120 does not experience abrupt changes in the displayed image. The surgeon 120 may switch the shift technique on or off and set the duration via the UI 160. Note that in the example of FIG. 8B, the combined image has the same magnification as each of the independent images. However, this is not essential and the magnification may be reduced to some extent so that the combined image represented by the grid 818 comprises the entire imaging FOV. For example, the numerical representation of the combined image contains the entire column of each acquired image, the leftmost column corresponds to the leftmost column of grid 812 (which has number 11 as the upper left grid entry) and the rightmost column corresponds to. , Corresponds to the rightmost column of grid 810 (having number 28 as the top grid entry). In practice, when this option is applied, the zoom will change slightly as the user changes focus.

In some scenarios, non-overlapping images can occur. One such scenario that requires correction using shift techniques occurs at the beginning of the procedure in which the surgeon 120 places the camera head 110 approximately at the design working distance. The surgeon 120 manually adjusts the heights of the cameras 140A and 140B housed in the camera head 110 above the surgical site until the two spots of light generated by the two floodlights of the lighting system 114 overlap. obtain. These lights can be designed to overlap at the design working distance. If the camera head 110 is not perfectly located at the design working distance, the images produced by the cameras 140A and 140B and projected into the surgeon 120's eyes through the displays 130A and 130B are fully overlapped, as described above in FIG. 8B. Do not fit.

Here, as a whole, reference is made to FIGS. 8C-8E showing the optical system 800 of FIG. 8A at various working distances, configured and operable according to embodiments of the disclosed technique. The top line 820 of FIGS. 8C-8E shows an image captured by the left camera 140A (FIG. 1E) and displayed by the display 130A (FIG. 1C), and the bottom line 822 of FIGS. 8C-8E is the right camera 140B (FIG. 1E). ) Is captured and displayed by the display 130B (FIG. 1C). Dashed lines 824 in FIGS. 8C-8E indicate an object plane. Errors in the positioning of cameras 140A and 140B are exaggerated for illustration purposes. FIG. 8C shows an optical system 800 with a camera located at the design working distance. Therefore, there is no discrepancy between the actual working distance and the designed working distance. The images obtained by this configuration do not require a shift to completely overlap as indicated by the overlapping lines 820 and 822.

Referring to FIG. 8D, an optical system 800 is shown that is located below the design working distance and has cameras 140A and 140B (FIG. 1E) that result in a discrepancy between the actual working distance and the design working distance. As a result of this discrepancy, the images acquired by each of the cameras 140A and 140B and displayed through the displays 130A and 130B, respectively, do not completely overlap. The straight line 820 that shows the image acquired by the left camera 140A and displayed by the display 130A does not completely overlap with the straight line 822 that shows the image taken by the right camera 140B and displayed through the display 130B. As a result, the overlapping parts of the two images appear to originate from different directions, and the user's left and right eyes are narrowed in an attempt to generate a combined 3D image in the brain. This can result in eye strain unless the image is corrected, for example by using the shift technique described above.

Referring to FIG. 8E, an optical system 800 with cameras 140A and 140B located above the design working distance and resulting in a discrepancy between the actual working distance and the design working distance is shown. As a result of this discrepancy, the images acquired by each of the cameras 140A and 140B do not completely overlap. The straight line 820 showing the image acquired by the left camera 140A and displayed by the display 130A does not overlap with the straight line 822 showing the image taken by the right camera 140B and displayed by the display 130B. As mentioned above, when the shift technique is not applied, the left and right eyes are narrowed in an attempt to generate a combined 3D image in the brain, resulting in eye strain.

In some embodiments, for example, if the system 100 includes a z-axis motor for controlling the height of the camera head 110, the UI 160 is for automatically adjusting the height of the camera head 110 to the design working distance. , Presents initialization options, ie "Initial" or "Home" menu items. In embodiments where the system 100 is not configured with a Z motor to control the height of the camera head 110, the computer determines a discrepancy between the actual working distance and the designed working distance. Image processing techniques are applied and if the discrepancy is greater than the permissible threshold, the surgeon 120 is informed to manually adjust the distance. For example, the UI 160 displays a message, either up or down, that guides the surgeon 120 how much, ie, how many centimeters or inches, how many heights to adjust. However, the surgeon 120 may choose to continue working at a distance different from the design working distance, for example to increase the imaging FOV (the lower side is 3D, not all of the FOV, but only where the images overlap). I can see it).

Another scenario that requires correction using shift technology is the lens focus integrated into the cameras 140A and 140B by the surgeon 120 to change the focal plane when the cameras 140A and 140B are properly positioned at the design working distance. This is the case when a motor is used. For example, when the surgeon 120 switches between anterior and posterior eye modes, or focuses on different planes within the same mode, the discrepancy between the images acquired by the cameras 140A and 140B is enough to cause eye strain. Can grow. In such cases, the shift techniques described above may be applied to match the images acquired by the cameras 140A and 140B and reduce the eye strain experienced by the surgeon 120.

Here, reference is made to FIG. 8F showing an optical system 800 of FIG. 8A configured to focus on two different focal planes 830 and 832 that are configured and operable according to embodiments of the disclosed technique. The design working distance is indicated by the dashed line 834. The focal plane 830 is located above the design working distance 834 and the focal plane 832 is located below the design working distance 834. The top line 836 represents an image captured by the left camera 140A and displayed through the display 130A, and the bottom line 838 represents an image captured by the right camera 140B and displayed through the display 130B. Top line 836 is shown shifted to the left with respect to bottom line 838 and indicates a discrepancy between the two acquired images that can be corrected using the shift technique described herein. Similar to the focal plane 832, the top line 840 represents an image captured by the left camera 140A and displayed through the display 130A, and the bottom line 842 is an image captured by the right camera 140B and displayed through the display 130B. Represents. The top line 840 is shown shifted to the right with respect to the bottom line 842, indicating a discrepancy between the two acquired images when the cameras 140A and 140B are adjusted to the focal plane 832. This discrepancy can also be corrected using the shift technique described herein.

In the embodiments described above, the shift was performed to allow comfortable stereoscopic viewing when the surgeon's eyes are looking forward. In some embodiments, the system 100 may be configured to intentionally generate a discrepancy between the two display images. The amount of shift can be uniquely tailored to each user of system 100. The displayed image can be shifted horizontally, vertically, or both. The shift may be unique to each of the surgeon's eyes. This shift feature can be useful for reducing eye strain in surgeons experiencing obliqueness (ie, misalignment of both eyes). The amount and direction of the shift can be configured to vary as a function of head orientation (as indicated by the tracker) in the case of an oblique state that depends on head orientation.

In some embodiments, the user-specific shift may be configured during a single calibration process and may be stored per user to be automatically performed for later use of the system 100. Optionally, the surgeon may adjust the shift characteristics during the surgical procedure. Optionally, the head orientation-based shift is performed stepwise and only after the head orientation of the HMD 102 wearer has not changed for a predetermined period of time. This is to prevent the display image from shifting in response to performing a head gesture to enable other features that could interfere with some users. In some cases, the displayed image may be non-uniformly shifted to compensate for possible tilt dependence in the line-of-sight direction (eg, vertical line-of-sight direction). For example, if the tilt depends on the direction of the vertical line of sight, the shift can change continuously (not necessarily linearly) from the top to the bottom of the image. Optionally, if the HMD 102 enables the change of focus in the HMD optics, the amount and direction of shift may depend on the focus. However, focus-dependent shifts can also be performed for users who do not exhibit tilt. The shifts described above may also result in binocular competitive effects that may require correction, as described below. Here, FIGS. 8G-8I showing images displayed via the HMD 102 after application of the shift technique described herein are referenced. The effect of the shift is exaggerated for clarity. FIG. 8G shows a 2D raw image 850 acquired by the camera 140A. The 2D raw image 850 is shifted to the right according to the shift technique described herein, as indicated by the black border 852 displayed at the left edge of the raw image 850. FIG. 8H shows a 2D raw image 854 acquired by the camera 140B. The 2D raw image 854 is shifted to the left according to the shift technique described herein, as indicated by the black border 856 displayed at the right edge of the raw image 854.

A binocular competitive effect can occur if the amount of shift performed by the shift technique is significant. Referring to FIG. 8I, a representation of the 3D raw image 858 is shown. Image 858 is a combination of the 2D raw image 850 of FIG. 8G and the raw image 854 of FIG. 8H. Range 864 shows a portion of image 858 created from both images 850 and 854 and displayed by both displays 130A and 130B, respectively. The range 866 of the image 858 shows the portion of the image 858 created only from the image 854 and displayed only by the display 130B, and the range 868 of the image 858 is an image created only from the image 850 and displayed only by the display 130A. The part of 858 is shown. The brain corrects the black borders 852 and 856 of images 850 and 854, respectively, to allow the surgeon 120 to see a significant contrast in the brightness of the area depicted by the dashed lines 860 and 862, and the brightness of image 858 plummets in half. do. The middle area (range 864) of image 858 is displayed by both displays 130A and 130B, while the left edge of image 858 (range 866) is displayed only by display 130B and the right edge of image 858 (range 868) is display 130A only. Displayed by. One solution is to gradually reduce the brightness of both images 850 and 854 in the "missing edge", i.e., in the area near 852 and 856, respectively, in the brightness of the area drawn by the straight lines 866 and 868. It is to eliminate the remarkable contrast. Since the linear gradient can be detectable by the brain, the reduction in brightness can follow a non-linear gradient.

Reference is made here to FIG. 9, which is a schematic representation of a method for controlling a heads-up display system that is configured and operational according to other embodiments of the disclosed technology.

In step 900, the surgical field is illuminated and at least one image of the illuminated surgical procedure is acquired. Referring to FIG. 1A, the lighting system 114 of the head-mounted display system 100 illuminates the surgical field 124. The camera system 112 acquires at least one image of the illuminated surgical field 124.

In step 902, at least one image of the surgical procedure is displayed to the surgeon via the head-mounted display. The image may be at least one image acquired via the camera system 112 and displayed in real time, a preoperative image acquired in advance, or an image relating to the UI 160, such as a menu, overlay, or the like. Referring to FIG. 1A, at least one image is displayed via the head-mounted display 102.

In step 904, the head gesture input is received in association with the foot movement input. Referring to FIG. 2A, the head tracker 162 receives the head gesture input by the surgeon 120 (FIG. 1A) and the foot switch 104 receives the foot movement input from the surgeon 120. The computer 118 receives the head gesture input in association with the foot movement input.

In step 906, when the head-mounted display system is in the first system mode, the first action is taken in the head-mounted display system, and the head-mounted display system is in the second system mode. In order to perform the second action in the head-mounted display system, the head gesture input received in connection with the foot movement input is applied. Referring to FIG. 2B, the computer 118 performs the zoom-out action 240 when the system mode is normal, and the foot movement input to perform the scroll action 242 when the system mode is set preoperatively. Apply the associated received head gesture input.

In some embodiments, the first and second actions include controlling the shutter coupled to the display module of the head-mounted display, and if the shutter is open, the display module is at least When partially transparent and the shutter is closed, the display module is substantially opaque. Referring to FIG. 1C, the shutter 132 is configured with the display module 130 of the head-mounted display 102.

In another embodiment, the first and second actions are controlling an image displayed via a head-mounted display, controlling a camera system, controlling a lighting system, a camera head positioner. Equipped with any of the controls. Referring to FIG. 2A, the computer 118 (FIG. 1B) applies the input received via the UI input 160A to control the components of the UI output 160B, i.e. the computer 118 is the camera system 112, illumination. It controls the image displayed on the display module 130 via the system 114, the HMD 102, and one of the camera head positioners 111 or the robot arm.

In some embodiments, controlling an image displayed through a head-mounted display involves selecting an image from a group consisting of live feed video, VGS video, and preoperative iOCT video, zooming in and zooming. Out, scrolling between at least two virtual screens, displaying picture-in-pictures (PIPs), displaying overlays on top of raw images, centering images, displaying menus, menus It comprises either navigating and controlling the area of interest of the image. The characteristics of the image displayed via the head-mounted display can be controlled according to any suitable technique, for example digitally or optically.

In some embodiments, controlling a camera system comprises controlling one or more optical or electrical properties of the camera system. For example, a camera for acquiring images may be selected and the position and orientation of the camera system may be controlled. Referring to FIG. 1E, the computer 118 (FIG. 1B) selects one of the high resolution cameras 140A and 140B, the IR cameras 148, and the iOCT scanner 142. Referring to FIG. 1A, the computer 118 controls the position and orientation of the camera system 112 via the camera head positioner 111.

In some embodiments, controlling a luminaire is to select at least one of a plurality of illuminators, to select an intensity setting, to select a filter, and to position and orient the illuminating system. Equipped with any of the controls. Referring to FIG. 1F, the computer 118 (FIG. 1B) selects one of the white floodlight 150, the IR floodlight 152, and the coaxial lights 154A and 154B. Referring to FIG. 1A, the computer 118 controls the position and orientation of the lighting system 114 via the camera head positioner 111.

In step 908, a voice command by the surgeon is detected. Voice commands are applied to control a head-mounted display system. Referring to FIG. 1D, the computer 118 (FIG. 1B) receives a voice command from the surgeon 120 (FIG. 1A) via the microphone 138. The computer 118 applies voice commands to control the head-worn display system 100.

In step 910, eye movements by the surgeon are detected and the eye movements are applied to control the head-worn display system. Referring to FIG. 1C, eye tracking component 136 tracks eye movements of surgeon 120 and provides eye movements to computer 118 (FIG. 1B) via transceivers 102B and 118B.

In step 912, a head gesture input is applied to switch from the first system mode to the second system mode. Referring to FIGS. 2A-2B, the computer 118 applies the head gesture input received via the UI input 160A in order to switch to the vital system mode 244. In some embodiments, the computer 118 applies a head gesture input received in association with the foot movement input to switch modes.

In some embodiments, the image is displayed in a fixed display state via the head-mounted display when the head-mounted display system is in the first system mode. The image is displayed in a world-fixed state on one of multiple virtual screens via the head-mounted display when the head-mounted display system is in the second system mode, and the second action is multiple. Scroll the virtual screen. Referring to FIG. 5A, a fixed display state is shown. The video displayed to Surgeon 120 does not change when he moves his gaze from anterior (500) to 30 degrees to the left (502). With reference to FIG. 5B, the world fixed state is shown. The video displayed to Surgeon 120 changes as he moves his gaze from 5 degrees (504) to the right to 25 degrees (506) to the right. The content that fills the field of view shifts as if the virtual screen were a fixed object in the operating room.

In some embodiments, the first area of the field of view of the head-mounted display corresponds to the first system mode and the second area of the field of view of the head-mounted display corresponds to the second system mode. handle. The head gesture input applied to switch from the first system mode to the second system mode aligns the surgeon's line of sight with the second area. Referring to FIG. 3A, the region 302 in the center of the field of view corresponds to the normal system mode and the region 306 to the left of the region 302 corresponds to the preoperative system mode. The surgeon 120 may switch system modes by moving his line of sight from region 302 to region 306.

In some embodiments, the menu is displayed as an overlay on the image displayed on the head-mounted display. The menu displays the first menu item for the first system mode in the first area of the field of view of the head-mounted display and the second overlaid in the second area of the field of view of the head-mounted display. Display a second menu item for the system mode. The head gesture input applied to switch from the first system mode to the second system mode aligns the surgeon's line of sight with the second menu item above. With reference to FIGS. 4A-4C, the menu 400 is shown overlaid on the surgical procedure video. Menu 400 includes a plurality of menu items 402, 404, 406, 408, 410, and 412. The surgeon 120 may switch the system mode from, for example, menu item 402 highlighted in FIG. 4A to menu item 410 highlighted in FIG. 4B by moving his gaze between menu items.

Those skilled in the art will appreciate that the techniques disclosed are not limited to those specifically indicated and described above. Correctly, the scope of the disclosed technology is defined only by the claims set forth below.

Claims (30)

A head-mounted display configured to be worn by the surgeon,
With a tracker configured to track head gesture input by the surgeon,
A foot switch configured to detect the foot movement input by the surgeon,
With the head-mounted display, the tracker, and the computer coupled to the footswitch,
A user interface comprising the head-mounted display, the tracker, and the footswitch.
The computer is provided with the head gesture input received from the tracker in connection with the foot motion input received from the foot switch.
A head-mounted display system comprising a user interface configured to display images relating to a surgical procedure in the head-mounted display.
The computer
When the head-mounted display system is in the first system mode, the first action is performed in the head-mounted display system, and when the head-mounted display system is in the second system mode, the above-mentioned For the head-mounted display system configured to apply the head gesture input received in connection with the foot movement input to perform a second action in the head-mounted display system. User interface. The system of claim 1, wherein the tracker is at least partially integrated with the head-mounted display and is configured to track the head-mounted display. The system according to claim 1, wherein the tracker is an external camera of the head-mounted display. The head-mounted display further comprises a shutter coupled to the head-mounted display, the head-mounted display is at least partially transparent when the shutter is open, and the head-mounted display when the shutter is closed. The system of claim 1, wherein the type display is substantially opaque. The system of claim 4, wherein the first action and one of the second actions control the shutter. The system of claim 1, wherein the user interface further comprises a microphone configured to detect a voice command, and the input further comprises said voice command. The system of claim 1, further comprising an eye tracker configured to detect eye movements by the surgeon, wherein the input further comprises said eye movements. It further comprises a camera system configured to acquire the image, a lighting system configured to operate with the camera system, and a positioning mechanism selected from the group consisting of a camera head positioner and a robot arm. The system according to claim 1. The first action and the second action are
Controlling the characteristics of the image displayed via the head-mounted display,
Controlling the camera system,
8. The system of claim 8, selected from the group consisting of controlling the lighting system and controlling the positioning mechanism. Controlling the properties of the image displayed through the head-mounted display is to select the content of the image, zoom in and out, scroll between at least two virtual screens, and so on. Displaying a picture-in-picture (PIP), displaying an overlay on top of a raw image, centering the image, displaying a menu, navigating the menu, and controlling the area of interest in the image. 9. The system of claim 9, comprising an action selected from a group consisting of. 9. The system of claim 9, wherein controlling the camera system comprises controlling one or more optical and electrical properties of the camera system. Controlling the lighting system is selected from a group consisting of selecting at least one of a plurality of illuminators, selecting an intensity setting, selecting a filter, and turning the illumination on or off. 9. The system of claim 9, comprising performing the performed action. The image is displayed in a fixed display state via the head-mounted display when the head-mounted display system is in the first system mode, and the video is displayed by the head-mounted display system. The system according to claim 1, wherein in the second system mode, the system is displayed in a world fixed state on one of a plurality of virtual screens via the head-mounted display. The system of claim 1, wherein the computer is further configured to apply the head gesture input to switch from the first system mode to the second system mode. The first region of the head position and orientation range of the wearer of the head-mounted display corresponds to the first system mode, and the head position and orientation of the wearer of the head-mounted display. The second area of the range corresponds to the second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is the head-mounted type. 14. The system of claim 14, wherein the wearer's head position and orientation of the display is changed from the first region to the second region. The computer is further configured to display an overlaid menu on the image on the head-mounted display, wherein the menu displays a first menu item with respect to the first system mode, said first. The second menu item is displayed with respect to the second system mode, and the head gesture input and the foot movement input are the second from the first system mode by selecting and activating the second menu item. The system according to claim 1, which is applied to switch to the system mode of 2. A way to interact with a head-mounted display system,
Showing the surgeon an image of the surgical procedure on a head-mounted display,
Receiving the surgeon's head gesture input from the head tracker in connection with the foot movement input by the surgeon from the footswitch.
When the head-mounted display system is in the first system mode, the first action is performed in the head-mounted display system, and when the head-mounted display system is in the second system mode, the above-mentioned A method comprising applying the head gesture input received in connection with the foot movement input to perform a second action in a head-mounted display system. One of the first action and the second action comprises controlling a shutter coupled to the head-mounted display, and when the shutter is open, the head-mounted display is transparent. 17. The method of claim 17, wherein the head-mounted display is opaque when the shutter is closed. 17. The method of claim 17, further comprising detecting a voice command by the surgeon and applying the voice command to control the head-worn display system. 17. The method of claim 17, further comprising detecting eye movements by the surgeon and applying the eye movements to control the head-worn display system. The surgical field is illuminated by a lighting system configured with the head-mounted display system, and the image of the illuminated surgical field is acquired using the camera system configured with the head-mounted display system. 17. The method of claim 17, further comprising doing so. 21. The method of claim 21, further comprising controlling the position of the camera system to obtain the image via a positioning mechanism selected from the group consisting of a camera head positioner and a robot arm. The first action and the second action are
Controlling the characteristics of the image displayed via the head-mounted display,
Controlling the camera system,
22. The method of claim 22, selected from the group comprising controlling the lighting system and controlling the positioning mechanism. Controlling the properties of the image displayed through the head-mounted display is to select the content of the image, zoom in and out, scroll between at least two virtual screens, and so on. 23. The method of claim 23, comprising an action selected from a group consisting of displaying a picture-in-picture (PIP) and displaying an overlay over a raw image. Claiming that controlling the camera system comprises selecting a camera for acquiring the image and performing an action selected from a group consisting of controlling the position and orientation of the camera system. 23. Controlling the luminaire consists of a group consisting of selecting at least one of a plurality of illuminators, selecting an intensity setting, selecting a filter, and turning the illuminating system on or off. 23. The method of claim 23, comprising performing the selected action. The image is displayed in a fixed state via the head-mounted display when the head-mounted display system is in the first system mode, and the image is displayed by the head-mounted display system. 17. The method of claim 17, wherein when in the second system mode, the head-mounted display is displayed on one of a plurality of virtual screens in a world-fixed state. 17. The method of claim 17, further comprising applying the head gesture input received in connection with the foot movement input in order to switch from the first system mode to the second system mode. The first region of the head position and orientation range of the wearer of the head-mounted display corresponds to the first system mode, the head position of the wearer of the head-mounted display and the head position. The second area of the orientation range corresponds to the second system mode, and the head gesture input applied to switch from the first system mode to the second system mode is head-mounted. 28. The method of claim 28, wherein the wearer's head position and orientation of the type display is changed from the first region to the direction of the second region. The head-mounted display further comprises displaying an overlaid menu on the image, wherein the menu displays a first menu item with respect to the first system mode and with respect to the second system mode. The second menu item is displayed, and the head gesture input and the foot movement input are changed from the first system mode to the second system mode by selecting and activating the second menu item. 28. The method of claim 28, which is applied to switch.

Images