JP2018180840A - Head-mount display control device, operation method and operation program thereof, and image display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mount display control device, operation method and operation program thereof, and an image display system capable of effectively avoiding a false recognition of a gesture operation, and of improving a usability.SOLUTION: In a recognizing unit of a control device and a processing unit thereof, a selection processing unit 85 executes a selection process of selecting, when a double-tap operation is recognized by an operation recognizing unit 80, one option among multiple options relating to a 3D image, and of changing the option to be selected in a cyclic manner every time the double-tap operation is recognized. A change processing unit 86 executes a change process of, when a tap-and-hold operation is recognized by the operation recognizing unit 80, changing the display scheme of the 3D image associated with the one option selected by the selection processing unit 85.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a control device for a head mounted display, an operation method and an operation program thereof, and an image display system.

  A virtual object using computer graphics is displayed on a head mounted display (hereinafter referred to as HMD (Head Mounted Display)) mounted on the head of the user, and the virtual object is displayed to the user like a real object existing in real space There is known a technology for making it recognize.

  Patent Document 1 describes an HMD control device that recognizes various gesture operations using a user's hand on a virtual object. The control device performs a change process of changing the display mode of the virtual object, such as the display position and the display size of the virtual object, according to the recognized gesture operation. For gesture operation, slide operation to move the fingertip of the index finger sideways, and tap and hold to move the hand in a state in which the index finger and the fingertip of the thumb are in contact (the virtual object is grasped by the index finger and the fingertip of the thumb) The operation is illustrated.

  As described in Patent Document 1, in the technique using an HMD, a gesture operation is often adopted as an operation input method. In order to perform the change process of changing the display mode of the virtual object by such gesture operation, a gesture operation for selecting one of a plurality of options such as the type of change process or the target of the change process ( Hereinafter, two types of a selection operation) and a gesture operation (hereinafter, change operation) for actually performing the change process associated with one option selected in the selection operation are required.

  Although patent document 2 is not a technique using HMD, it has adopted gesture operation as an operation input method to a three-dimensional stereoscopic image displayed on a display facing the user. In Patent Document 2, an operation of moving the hand forward and backward quickly (hereinafter, click operation, and in Patent Document 2, click operation) is a selection operation. Then, each time the click operation is recognized, the option to be selected is cyclically switched. The options include a plurality of three-dimensional stereoscopic images to be changed, and a display position, a display size, and a display direction of a three-dimensional stereoscopic image which is a type of the changing process.

  Further, in Patent Document 2, a change operation is a slide operation in which the hand is moved forward and backward at a moving speed slower than the specified speed of the click operation, and a slide operation in which the hand is moved along the display surface of the display. The user performs the slide operation after selecting the desired three-dimensional stereoscopic image and the type of change processing by the above-described click operation. Thereby, processing for changing the display position, display size, or display direction of the three-dimensional stereoscopic image is performed.

JP, 2014-071812, A Unexamined-Japanese-Patent No. 2005-322071

  Here, it is necessary to distinguish between the selection operation and the change operation. For example, when one of the selection operation and the change operation is erroneously recognized as the other, processing unintended by the user is performed, for example, the change processing is performed even though the selection operation has been performed. , Because it becomes inconvenient.

  In patent document 2, the click operation which is selection operation, and the slide operation which differs only in moving speed are included in change operation. Therefore, the click operation is erroneously recognized as a slide operation, and processing for changing the display position, display size, or display direction of the three-dimensional stereoscopic image is performed, or conversely, the slide operation is erroneously recognized as a click operation. The type of change processing or the target is likely to be switched, and there is a possibility that the usability may deteriorate.

  An object of the present invention is to provide a control device of a head mounted display capable of effectively preventing erroneous recognition of a gesture operation and improving usability, an operation method and an operation program thereof, and an image display system. .

  In order to solve the above problems, a control device for a head mounted display according to the present invention is a control device for a head mounted display mounted on the head of a user and controlling operation of the head mounted display for making the user recognize a virtual space. A display control unit for displaying a three-dimensional virtual object in a virtual space, an operation recognition unit for recognizing a gesture operation using the user's hand on the virtual object, and pressing at least one finger other than the thumb toward the thumb When double-tapping operation in which an operation to push up from 2 is performed twice consecutively is recognized as a gesture operation, one of a plurality of options relating to the virtual object is selected, and is selected each time the double-tapping operation is recognized A selection processing unit that switches options cyclically, wrists, elbows, and shoulders When a slide operation for moving a hand with one of the support points is recognized as a gesture operation, a change processing unit that performs change processing for changing the display mode of the virtual object accompanying the one option selected by the selection processing unit Equipped with

  It is preferable that the display control unit displays a menu bar in which a plurality of options are arranged in order in the virtual space, and the menu bar focus-displays one option selected by the selection processing unit. In this case, the display control unit preferably displays the menu bar only when the double tap operation is recognized as a gesture operation. Preferably, the display control unit displays a menu bar at the viewpoint of the user.

  The options are preferably the display position, display size, and display orientation of the virtual object, which is the type of change processing.

  The virtual object preferably has a plurality of structures, and the option is preferably a plurality of structures to be subjected to the change process. In this case, it is preferable that the change processing unit performs, as the change processing, a transmittance change process of changing the transmittance of one of the plurality of structures selected by the selection processing unit.

  The virtual object is preferably a three-dimensional volume rendered image of a human body. In addition, the slide operation is preferably a tap-and-hold operation in which the hand is moved in a state in which the fingertips of two fingers are in contact with each other.

  The method of operating the control device of the head mounted display according to the present invention is a method of operating the control device of the head mounted display mounted on the head of the user and controlling the operation of the head mounted display for making the user recognize the virtual space. Control step for displaying the virtual object in the virtual space, an operation recognition step for recognizing a gesture operation using the user's hand on the virtual object, and pushing at least one finger other than the thumb toward the thumb and pushing it up When a double tap operation in which the operation is performed twice consecutively is recognized as a gesture operation, one option is selected from the plurality of options related to the virtual object, and the option to be selected each time the double tap operation is recognized Selection processing steps to switch to cyclic, Change processing to change the display mode of the virtual object attached to one option selected in the selection processing step when the slide operation to move the hand with any of neck, elbow or shoulder as a fulcrum is recognized as a gesture operation And a change processing step of

  The operation program of the control device of the head mounted display according to the present invention is three-dimensional in the operation program of the control device of the head mounted display mounted on the head of the user and controlling the operation of the head mounted display for making the user recognize the virtual space. Control function to display the virtual object in the virtual space, an operation recognition function to recognize the gesture operation using the user's hand to the virtual object, and push at least one finger other than the thumb toward the thumb and push up When a double tap operation in which the operation is performed twice consecutively is recognized as a gesture operation, one option is selected from the plurality of options related to the virtual object, and the option to be selected each time the double tap operation is recognized With selection processing function to switch to cyclic, Change processing to change the display mode of the virtual object attached to one of the options selected by the selection processing function when a slide operation for moving the hand with one of the neck, elbow or shoulder as a fulcrum is recognized as a gesture operation And have the computer execute the change processing function to

  The image display system according to the present invention further includes a head mounted display mounted on the head of the user, and a control device for controlling the operation of the head mounted display, and displaying an image that makes the user recognize the virtual space through the head mounted display. In the system, a display control unit for displaying a three-dimensional virtual object in a virtual space, an operation recognition unit for recognizing a gesture operation using the user's hand on the virtual object, and at least one finger other than the thumb directed to the thumb If a double tap operation in which an operation after pressing and pushing up twice is performed twice consecutively is recognized as a gesture operation, one of a plurality of options regarding the virtual object is selected, and each time the double tap operation is recognized , To switch the choice to cyclic selection When a slide operation for moving the hand with one of the wrist, elbow, or shoulder as a fulcrum is recognized as a gesture operation, the display mode of the virtual object associated with one of the options selected by the selection processing unit is displayed. And a change processing unit that performs change processing to be changed.

  According to the present invention, a selection operation, which is a gesture operation for selecting one of a plurality of options related to a virtual object, is pressed twice at least after pushing at least one finger other than the thumb toward the thumb. A slide operation for moving a hand with one of a wrist, an elbow, and a shoulder as a fulcrum as a change operation which is a gesture operation for performing a change process of changing the display mode of the virtual object as a double tap operation performed continuously Therefore, it is possible to provide a control device for a head mounted display, an operation method and an operation program thereof, and an image display system capable of effectively preventing erroneous recognition of a gesture operation and improving usability.

FIG. 2 is a diagram showing an image display system and an image storage server. 1 is an external perspective view of an image display system. It is a figure which shows a mode that a surgical procedure of a target patient is examined using an image display system. It is a figure for demonstrating the formation of augmented reality space. It is a figure which shows the CT scan image which comprises 3D image. It is a flow chart which shows a rough flow which considers a subject patient's operation plan using an image display system. It is a block diagram of a computer which constitutes a control device. It is a block diagram which shows each function part of CPU of a control apparatus. It is a block diagram which shows the detail of a recognition part and a process part. It is a figure which shows 1st corresponding information. It is a figure which shows 1st corresponding information. It is a figure which shows 1st corresponding information. It is a figure which shows 1st corresponding information. It is a figure which shows 1st corresponding information. It is a figure which shows 1st corresponding information. It is a figure which shows 2nd corresponding information. It is a figure which shows 2nd corresponding information. It is a figure which shows the selection process in adjustment mode. It is a figure which shows the selection process in observation mode. It is a figure which shows the change process which moves the display position of 3D image. It is a figure which shows the change process which reduces the display size of 3D image. It is a figure which shows the process which changes the display direction of a 3D image. FIG. 7 illustrates the process of changing the transmission of selected structures of a 3D image. It is a figure which shows the detail of the virtual image in adjustment mode. It is a figure which shows the detail of the virtual image in observation mode. It is a flowchart which shows the process sequence of a control apparatus. It is a figure which shows 2nd Embodiment which displays an adjustment mode display bar only, when double tap operation is recognized. It is a figure which shows 3rd Embodiment which displays an adjustment mode display bar on a user's viewpoint position. It is a figure which shows another example of an image display system. It is a figure which shows another example of an image display system.

First Embodiment
In FIG. 1, the image display system 10 is an integrated HMD 11 and control device 12, and is used, for example, in the conference room 14 of the medical facility 13. The image display system 10 is mounted on the head of the user 15. The user 15 is a medical staff such as a doctor or a nurse who belongs to the medical facility 13. The head means a portion located above the neck of the human body in a standing position, and is a portion including a face and the like.

  The image display system 10 is communicably connected to the image storage server 19 via a network 17 such as a LAN (Local Area Network). The image storage server 19 is installed, for example, in the computer room 18 of the medical facility 13.

  The image storage server 19 stores various medical images of the patient acquired at the medical facility 13. The medical image includes a three-dimensional volume rendering image (hereinafter, 3D image) 40 (see FIG. 3) obtained by reconstructing a CT (Computed Tomography) scan image 45 (see FIG. 5) by image processing. The image storage server 19 uses an image ID for identifying each medical image, a patient ID (Identification Data) for identifying a patient who has taken a medical image, or a type of modality for taking a medical image, shooting date, etc. It is possible to search for a medical image as a search key.

  The image storage server 19 searches for a medical image corresponding to the search key in response to a distribution request including the search key from the image display system 10, and distributes the searched medical image to the image display system 10. Although only one image display system 10 is connected to the image storage server 19 in FIG. 1, a plurality of image display systems 10 are actually provided in the medical facility 13 and a plurality of image display systems 10 are provided in the image storage server 19. An image display system 10 is connected.

  In FIG. 2, the image display system 10 includes a main body 25 and a mounting unit 26. When the image display system 10 is attached, the main body 25 is located in front of the user 15, and the attachment 26 is located in the upper half of the user 15 's head.

  The main body unit 25 includes a protective frame 27, a screen 28, a sensor unit 29, and a control unit 30. The protective frame 27 is a single transparent plate that is curved so as to cover the entire eyes of the user 15, and is formed of, for example, a transparent colored plastic. The screen 28 is disposed inside the protective frame 27. The screen 28 is assigned to each of the eyes of the user 15, and a nose pad 31 is provided at the center. The screen 28 is formed of a transparent material like the protective frame 27. The user 15 visually recognizes the real space RS (see FIG. 4) through the screen 28 and the protective frame 27. That is, the HMD 11 of the image display system 10 is transmissive.

  In addition, a virtual image using computer graphics is projected and displayed from the projection unit (not shown) on the inner surface of the screen 28 facing the eye of the user 15. As well known, the projection unit is configured of a display element such as liquid crystal that displays a virtual image, and a projection optical system that projects the virtual image displayed on the display element toward the inner surface of the screen 28. In this case, the screen 28 is configured of a half mirror that reflects the virtual image and transmits light in the real space RS. The projection unit projects and displays a virtual image on the inner surface of the screen 28 at a predetermined frame rate (for example, 60 frames / second). The virtual image is reflected on the inner surface of the screen 28 and enters the eye of the user 15. Thereby, the user 15 recognizes the virtual image as a virtual image on the virtual space VS (see FIG. 4). The screen 28 itself may be a display element such as liquid crystal.

  The virtual image includes a virtual object that the user 15 recognizes in the augmented reality space ARS (see FIG. 3) in the same manner as a real object existing in the real space RS. The virtual object is here a 3D image 40 (see FIG. 3) of the upper body of the patient. The user 15 performs a gesture operation on the 3D image 40 using his / her hand 37 (see FIG. 3).

  The sensor unit 29 is located at the top of the screen 28 and is covered with a protective frame 27 as with the screen 28. The sensor unit 29 belongs to the HMD 11 and includes various sensors for grasping the use state of the image display system 10. The sensor is a space recognition unit configured of a plurality of infrared emitters and a plurality of space recognition cameras for spatially recognizing the use peripheral environment of the image display system 10, a video camera having a color imaging device, an inertial measurement unit ( IMU (inertial measurement unit) etc. The IMU includes an acceleration sensor, an angular velocity sensor (gyroscope), and an orientation sensor (magnetometer), and cooperates with a video camera to perform so-called head tracking that tracks the movement of the head of the user 15.

  The sensor unit 29 is also provided with a gesture operation recognition unit configured of an infrared emitter and a depth sensor for recognizing a gesture operation using the hand 37 of the user 15 on the 3D image 40. The depth sensor is a sensor that detects infrared light emitted from an infrared emitter and reflected from a real object including the hand 37 to obtain information of the real space RS in the depth direction of the viewpoint of the user 14. The depth sensor images a field of view substantially the same as the augmented reality space ARS that the user 15 recognizes through the HMD 11 at a predetermined frame rate (for example, 60 frames / second). The user 15 performs a gesture operation within the field of view of the depth sensor.

  The sensor unit 29 is also provided with a microphone for recording the sound of the use peripheral environment of the image display system 10. In addition to gesture operation, this microphone also enables voice operation. Here, although the transmissive HMD 11 is illustrated, a non-transmissive HMD that projects a superimposed image on the inner surface of the screen 28 by superimposing a virtual image on the captured image of the physical space RS captured by a video camera You may use.

  The control unit 30 is attached to the back side of the crescent-shaped top plate of the main body unit 25. The control unit 30 includes drivers of various sensors disposed in the sensor unit 29, a power supply circuit for supplying power from the built-in battery to each unit, a wireless communication circuit for wirelessly communicating with external devices such as the image storage server 19 and the like A device 55, a memory 56, and a central processing unit (CPU) 57 (see FIG. 7) for overall control of these devices are provided. In the image display system 10 of the present embodiment in which the HMD 11 and the control device 12 are integrated, the control unit 30 configures the control device 12.

  The mounting portion 26 is in the shape of a band having a width of about several centimeters, and is composed of the inner ring 32 and the outer ring 33. The inner ring 32 has a continuous O-shape and is fixed in a state of being fitted to the upper half of the head of the user 15. On the inner surface of the inner ring 32 in contact with the head of the user 15, a shock absorbing pad such as urethane foam is attached. The inner ring 32 is adjustable in size to the head of the user 15. In addition, the inner ring 32 is movable forward and backward with respect to the outer ring 33 within a predetermined movement range, and with respect to the outer ring 33 within a predetermined angle range with an axis passing through both ears of the user 15 as a rotation center. Is rotatable.

  The outer ring 33 is C-shaped with the back of the head cut away. The outer ring 33 bends outward from the initial position or shrinks inward from the bent state toward the initial position according to the size adjustment of the inner ring 32.

  At the portion of the outer ring 33 facing the user's 15 ears, a speaker 34 for outputting sound is provided (only the left speaker 34 is shown in FIG. 2). Although not shown, at the top of the speaker 34, there are provided a volume adjustment button for the speaker 34 and a brightness adjustment button for the backlight of the screen 28. Further, although not shown, a power button and a USB (Universal Serial Bus) terminal are provided in the cutout on the back of the outer ring 33 on the back of the head.

  The image display system 10 operates with the power of the built-in battery via the power supply circuit, and performs wireless communication with an external device such as the image storage server 19 with a wireless communication circuit. Thus, the image display system 10 can be used wirelessly. By connecting a USB cable to the USB terminal of the outer ring 33 and connecting it to an external device by wire, power supply from the external device and wired communication with the external device become possible.

  FIG. 3 shows a state in which the user 15 in the conference room 14 of the medical facility 13 is using the image display system 10 to consider the surgery plan of a patient who is scheduled for surgery (hereinafter referred to as a target patient). The user 15 recognizes the augmented reality space ARS through the HMD 11.

  A table 36 is installed in the conference room 14. At the center of the table 36, a 3D image 40 of the upper surface of the upper body of the target patient is displayed as a virtual object. In the 3D image 40, the display position in the real space RS is fixed substantially at the center of the table 36, the body axis is along the long side of the table 36, the neck is the left side of the user 15, and the waist is the right side of the user 15. It is arranged as.

  The user 15 stands approximately at the center of one long side of the table 36. The user 15 points a hand (right hand) 37 and a finger (index finger) 38 to the table 36. The 3D image 40 is only a virtual object, and the 3D image 40 does not exist in the table 36 in the real space RS. For this reason, the 3D image 40 is indicated by a broken line on the table 36 of FIG.

  Since the display position of the 3D image 40 in the real space RS is fixed substantially at the center of the table 36, the appearance of the 3D image 40 in the augmented reality space ARS recognized by the user 15 differs according to the standing position of the user 15 . Specifically, in the standing position shown in FIG. 3, the user 15 sees the right half of the 3D image 40 on the near side and the neck on the left. When the user 15 stands on the other long side of the table 36 opposite to the standing position shown in FIG. 3, the left half of the 3D image 40 is seen on the near side and the neck is on the right.

  Also, for example, when the user 15 approaches the table 36, the 3D image 40 is enlarged and displayed as the user 15 approaches. Conversely, when the user 15 moves away from the table 36, the 3D image 40 is displayed in a reduced size as the user 15 moves away.

  Thus, the display of the 3D image 40 is changed according to the three-dimensional positional relationship between the image display system 10 (user 15) and the table 36. The three-dimensional positional relationship between the image display system 10 and the table 36 performs space recognition of the entire conference room 14 including the table 36 by the space recognition unit of the sensor unit 29, and performs head tracking by the video camera and IMU. Can be recognized. As a matter of course, the appearance in the augmented reality space ARS also differs depending on the standing position of the user 15, such as the table 36 etc., which is a real object of the real space RS.

  FIG. 4 shows the formation of the augmented reality space ARS that the user 15 recognizes. The user 15 visually recognizes the real space RS in which the table 36 and his / her hand 37 exist through the screen 28 and the protective frame 27. In addition, the user 15 views the virtual space VS in which the 3D image 40 exists through the inner surface of the screen 28. Thereby, the user 15 recognizes an augmented reality space ARS in which the real space RS and the virtual space VS are fused.

  In the virtual space VS, a cursor 41 indicating the viewpoint position of the user 15 is superimposed and displayed. The cursor 41 has a double circle shape. The display position of the cursor 41 is determined from the result of head tracking by cooperation of the video camera of the sensor unit 29 and the IMU. That is, the cursor 41 moves in conjunction with the movement of the head of the user 15. For example, when the user 15 turns the head upward, the cursor 41 also moves upward.

  FIG. 5 shows one of the plurality of CT scan images 45 constituting the 3D image 40. As shown in FIG. Here, the front-rear direction of the human body along the line connecting the abdominal back parallel to the plane (slice plane) of the CT scan image 45 is taken as the depth direction. The CT scan image 45 includes skin 46, subcutaneous tissue 47 such as fat, muscle tissue 48 such as rectus abdominis muscle, bone 49 such as rib, sternum, liver 50A, stomach 50B, spleen 50C etc. And a plurality of structures of human tissue such as a lesion 51 such as a tumor. The 3D image 40 is a reconstruction of a CT scan image 45 in which a plurality of structures of human tissue are shown. Therefore, the 3D image 40 is a virtual object in which a plurality of structures overlap in the depth direction. In the 3D image 40, for example, the skin 46 has a skin color, the bone 49 has a gray color, the viscera 50 has a pink color, and the lesion 51 has a red color.

  Among the plurality of structures, the skin 46, the subcutaneous tissue 47, the muscle tissue 48, and the bone 49 exist in this order from the outside regardless of the position on the 3D image 40. On the other hand, in the 3D image 40, the internal organs 50 such as the liver 50A change according to the position in the body axis direction. Here, the liver 50A, stomach 50B, and spleen 50C are exemplified, but since it is the 3D image 40 of the upper body, other internal organs 50 such as the kidney, the pancreas, the small intestine, and the large intestine are also included in the 3D image 40.

  FIG. 6 is a flowchart showing a rough flow when the user 15 uses the image display system 10 to examine the surgical procedure of the target patient. First, a list of file icons of the 3D image 40 is displayed on the HMD 11, and a 3D image 40 of a target patient to be displayed as a virtual object is selected by the user 15 from the list (step ST100). After the 3D image 40 is selected, the image display system 10 shifts to setting of the initial display position of the 3D image 40 (step ST110). As the initial display position, a rough position, such as on the table 36, is set.

  After setting the initial display position, the image display system 10 shifts to the adjustment mode (step ST120). In the adjustment mode, it is possible to select any of the display position, display size, and display orientation of the 3D image 40, and change these from the initial display position to the preference of the user 15.

  That is, in the adjustment mode, the display position, display size, and display direction of the 3D image 40 in the real space RS correspond to a plurality of options regarding the virtual object. Further, in the adjust mode, the process of changing the display position, the display size, and the display direction of the 3D image 40 corresponds to a process of changing the display mode of the virtual object accompanying the selected one option.

  When the display position, display size, and display orientation of the 3D image 40 become the preference of the user 15 in the adjust mode, and the user 15 instructs the finish of the adjust mode, the image display system 10 shifts to the observe mode (step ST130). . In the observation mode, it is possible to observe the 3D image 40 at the display position, display size, and display orientation set in the adjustment mode.

  In the observation mode, it is also possible to select a structure such as the skin 46 and change the transmittance of the selected structure. For example, each structure such as the skin 46, the subcutaneous tissue 47, the muscle tissue 48, the bone 49, etc. is selected, the transmittance of each structure is changed to 100% (transparent), and the lesion 51 to be operated is directly viewed To be possible.

  That is, in the observation mode, a plurality of structures correspond to a plurality of options for virtual objects. In the observation mode, the transmittance changing process for changing the transmittance of the selected structure corresponds to the changing process for changing the display mode of the virtual object accompanying the one selected option. In the observation mode, a virtual scalpel may be applied to the 3D image 40 using a virtual scalpel.

  In FIG. 7, the control device 12 of the image display system 10 includes the storage device 55, the memory 56, the CPU 57, and the like described above. These are interconnected via a data bus 58. In addition to these, the control device 12 includes drivers of various sensors, a power supply circuit, and a wireless communication circuit.

  The storage device 55 is, for example, a hard disk drive built in the control device 12. The storage device 55 stores control programs such as an operating system, various application programs, and various data associated with these programs.

  The memory 56 is a work memory for the CPU 57 to execute processing. The CPU 57 loads the program stored in the storage device 55 into the memory 56, and executes the processing according to the program, to control each part of the control device 12 in an integrated manner.

  In FIG. 8, the storage device 55 stores an operation program 65. The operation program 65 is an application program for causing a computer constituting the control device 12 to function as a control device of the HMD 11. In the storage device 55, in addition to the operation program 65, first correspondence information 66 (see FIGS. 10 to 15) and second correspondence information 67 (see FIGS. 16 and 17) are stored.

  When the operation program 65 is activated, the CPU 57 functions as the sensor information acquisition unit 70, the 3D image acquisition unit 71, the recognition unit 72, the processing unit 73, and the display control unit 74 in cooperation with the memory 56 and the like.

  The sensor information acquisition unit 70 acquires sensor information from the sensor unit 29 of the HMD 11. The sensor information includes a captured image of a space recognition camera, a captured image of a video camera, detection signals of various sensors of the IMU, a captured image of a depth sensor, and sound recorded by a microphone. The sensor information acquisition unit 70 outputs the acquired sensor information to the recognition unit 72.

  The 3D image acquisition unit 71 issues a distribution request for the 3D image 40 to the image storage server 19. Further, the 3D image acquisition unit 71 acquires the 3D image 40 transmitted from the image storage server 19 in response to the distribution request. The 3D image acquisition unit 71 outputs the acquired 3D image 40 to the display control unit 74.

  The recognition unit 72 performs various recognition based on the sensor information from the sensor information acquisition unit 70. The recognition unit 72 outputs the recognition result to the processing unit 73 and the display control unit 74. The processing unit 73 performs various processes based on the recognition result from the recognition unit 72. The processing unit 73 outputs the processing result to the display control unit 74. The display control unit 74 has a display control function of controlling display of a virtual image including the 3D image 40 on the HMD 11.

  In FIG. 9, the recognition unit 72 includes an operation recognition unit 80. Further, the processing unit 73 includes a selection processing unit 85 and a change processing unit 86.

  The captured image of the depth sensor in the sensor information from the sensor information acquisition unit 70 is input to the operation recognition unit 80. The operation recognition unit 80 has an operation recognition function of recognizing a gesture operation based on the image captured by the depth sensor and the first correspondence information 66. The operation recognition unit 80 outputs information of the recognized gesture operation (hereinafter, gesture information) to the processing unit 73 as a recognition result.

  In addition to the operation recognition unit 80, the recognition unit 72 is provided with various recognition units such as a voice recognition unit, a motion recognition unit, and a space recognition unit. The voice recognition unit recognizes a specific voice from among the voices recorded in the microphone of the sensor information. The specific voice is a voice uttered by the user 15, and predetermined words and phrases are registered. The motion recognition unit recognizes the motion of the head of the user 15 based on the captured image of the video camera among the sensor information and detection signals of various sensors of the IMU. The space recognition unit spatially recognizes the use peripheral environment of the image display system 10 based on an image captured by the space recognition camera.

  Based on the second correspondence information 67, the processing unit 73 recognizes the gesture operation represented by the gesture information from the operation recognition unit 80 as various operation commands according to the type. Then, various processes are performed according to the recognized operation command. More specifically, the selection processing unit 85 of the processing unit 73 selects one of a plurality of options regarding the 3D image 40 according to the selection command (see FIGS. 16 and 17) of the operation commands, It is responsible for the selection processing function to switch to cyclic. The plurality of options are the types of change processing such as the display position, display size, and display orientation of the 3D image 40 in the adjust mode, as described with reference to FIG. It is a structure. The selection processing unit 85 outputs the information of one selected option (hereinafter, option information) to the display control unit 74 as a processing result.

  The change processing unit 86 changes the display mode of the 3D image 40 accompanying the one option selected by the selection processing unit 85 in accordance with the change command (see FIGS. 16 and 17) of the operation commands. It is responsible for the change processing function that performs processing. The change process is also the change process of the display position, the display size, and the display direction of the 3D image 40 in the adjust mode as described with reference to FIG. 6, and the process of changing the transmittance of the structure in the observe mode It is. The change processing unit 86 outputs information indicating the result of the change processing (hereinafter, change information) to the display control unit 74 as a processing result.

  The processing unit 73 includes various processing units in addition to the selection processing unit 85 and the change processing unit 86. For example, an image selection processing unit that causes the 3D image acquisition unit 71 to issue a distribution request for the 3D image 40 according to an image selection command (see FIG. 16) of the operation commands, a mode switching command There is a mode switching processing unit or the like that switches between the adjustment mode and the observation mode according to (17). Further, the processing unit 73 recognizes the operation command as well as the gesture information on the specific voice recognized by the voice recognition unit, and performs various processes according to the recognized operation command.

  As shown in FIGS. 10 to 15, in the first correspondence information 66, the transition of the movement of the hand 37 of the user 15 and the corresponding gesture operation are registered. First, in the first correspondence information 66A shown in FIG. 10, from the state of GA1 in which the fingers of two fingers 38 (here, the index finger and the thumb) are separated, one finger 38 (here) is shown as GA2. After depressing the forefinger) toward the thumb, a series of movements are registered which push up one finger 38 and release the finger tips of the two fingers 38 again without leaving an interval as shown in GA 3. For this movement, a single tap operation is registered as a gesture operation.

  The first correspondence information 66B shown in FIG. 11 is the same as GA1 to GA3 of the single tap operation in FIG. 10 up to GB1 to GB3, but after releasing the fingertips of two fingers 38 in GB3, it continues further As shown in GB4, press down one finger 38 (here the index finger) toward the thumb again and push up one finger 38 again as shown in GB5 to release the finger tips of two fingers 38 It is registered. In other words, this movement is a movement in which an operation of pushing at least one finger 38 other than the thumb toward the thumb and pushing it up twice in a row is performed. In the first correspondence information 66B, a double tap operation is registered as a gesture operation for this movement.

  In addition, in single tap operation and double tap operation, when at least one finger 38 other than the thumb is depressed toward the thumb, it is necessary to bring the respective fingers of the thumb and the at least one finger 38 other than the thumb into contact. There is no.

  The movement of the first correspondence information 66C shown in FIG. 12 is the same as that of GA1 and GA2 in the single tap operation of FIG. 10 up to GC1 and GC2. However, while in FIG. 10 the finger tips of two fingers 38 are separated by GA 3, in FIG. 12, as shown by GC 3 R, the hand 37 is left when the finger tips of two fingers 38 are in contact. I am moving from the right. In the first correspondence information 66C, the right slide operation of the tap-and-hold operation is registered as the gesture operation for this movement.

  The hand 37 is moved from right to left in a state where the fingertips of the two fingers 38 are in contact with the first correspondence information 66D shown in FIG. 13 as shown in GC3L contrary to GC3R in FIG. Movement is registered. For this movement, the left slide operation of tap-and-hold operation is registered as the gesture operation. The motions of GC1 and GC2 are the same as in FIG. 12, and thus are not shown in FIG. The same applies to FIGS. 14 and 15 that follow.

  In the first correspondence information 66E shown in FIG. 14, as shown in GC3U, the movement of moving the hand 37 from the bottom to the top is registered in a state where the fingertips of the two fingers 38 are in contact. For this movement, the upper slide operation of the tap-and-hold operation is registered as the gesture operation. On the other hand, as indicated by GC3D in the first correspondence information 66F shown in FIG. 15 as opposed to GC3U in FIG. 14, with the fingertips of the two fingers 38 in contact, the hand 37 is from top to bottom The movement to be moved is registered. For this movement, the lower slide operation of the tap-and-hold operation is registered as the gesture operation.

  Although illustration is omitted, in addition to the right and left slide operation and the up and down slide operation shown in FIGS. 12 to 15 as the tap-and-hold operation, the first correspondence information 66 is touched with the fingertips of two fingers 38. In the state, an oblique slide operation is also registered, in which the hand 37 is moved, for example, from the lower left to the upper right or vice versa, and further from the lower right to the upper left or vice versa.

  The single-tap operation shown in FIG. 10 and the double-tap operation shown in FIG. 11 are gesture operations with the metacarpal-phalangeal joint which is a joint of the base of the finger 38 as a fulcrum. On the other hand, the tap-and-hold operation including the right / left slide operation, the up / down slide operation, or the oblique slide operation (not shown) shown in FIGS. This is a gesture operation in which a slide operation for moving the hand 37 with one of them as a fulcrum is added.

  The operation recognition unit 80 recognizes the fingertips of the two fingers 38 from the hand 37 of the user 15 captured in the image captured by the depth sensor using a known image recognition technology. Then, it is determined which of the movements of the hand 37 registered in the first correspondence information 66 the movement locus of the recognized fingertip corresponds to.

  For example, when the fingertips of two fingers 38 move to almost the same position from a state where they are separated from each other and are separated again without putting an interval, the operation recognition unit 80 is registered in the first correspondence information 66A of FIG. It judges that it corresponds to movement and recognizes gesture operation as single tap operation. In addition, when the fingertips of two fingers 38 move away from each other to almost the same position, and the position of the fingertips of two fingers 38 moves from left to right in the same condition, the operation recognition unit 80 It judges that it corresponds to the movement registered into the 1st corresponding information 66C of Drawing 12, and recognizes gesture operation as right slide operation of tap and hold operation. The at least one finger 38 other than the thumb is not limited to the index finger, and may be the middle finger, the ring finger, or even the little finger. In addition, the index finger and the middle finger may be used.

  As shown in FIG. 16 and FIG. 17, the second correspondence information 67 is information in which a gesture operation and an operation command for each phase corresponding to the gesture operation are registered. In the second correspondence information 67, the presence or absence of viewpoint alignment is also registered.

  First, when the phase is the selection of the 3D image 40 shown in step ST100 of FIG. 6, an image selection command is assigned to a single tap operation involving viewpoint alignment. Viewpoint alignment is used to select the desired 3D image 40 from the list of file icons of the 3D image 40. That is, if the viewpoint position (cursor 41) is aligned with the desired 3D image 40 in the list of file icons of the 3D image 40 and a single tap operation is performed in that state, a distribution request for the 3D image 40 with the viewpoint position aligned is It is distributed from the 3D image acquisition unit 71 to the image storage server 19.

  Subsequently, when the phase is setting of the initial display position of the 3D image 40 shown in step ST110 of FIG. 6, a display position setting command is assigned to a single tap operation involving viewpoint alignment. The viewpoint alignment in this case is used to determine the initial display position of the 3D image 40. That is, after the selection of the 3D image 40 is completed, the viewpoint position is matched with the real object in the real space RS, and a single tap operation is performed in that state, the 3D image 40 is displayed on the real object with the viewpoint position matched.

  In the adjust mode shown in step ST120 of FIG. 6, a selection command for cyclically switching the selection of the type of change processing such as the display position, display size, and display orientation of the 3D image 40 is assigned to the double tap operation. On the other hand, in the observation mode shown in step ST130 of FIG. 6, a selection command for switching the selection of a plurality of structures to be changed is cyclically assigned to the double tap operation.

  When the display position is selected in the adjust mode (phase = adjust mode (display position)), a change command for moving the display position of the 3D image 40 to the right is assigned to the right slide operation of the tap and hold operation It is done. Further, a change command for moving the display position of the 3D image 40 to the left is assigned to the left slide operation of the tap and hold operation. Although not shown, a change command for moving the display position of the 3D image 40 up and down is assigned to the vertical slide operation of the tap and hold operation, and the diagonal slide operation of the tap and hold operation is allocated. , A change command for moving the display position of the 3D image 40 diagonally forward or diagonally back is assigned.

  When the display size is selected in the adjust mode (phase = adjust mode (display size)), a change command for enlarging the display size of the 3D image 40 is assigned to the right slide operation of the tap-and-hold operation. . Further, a change command for reducing the display size of the 3D image 40 is assigned to the left slide operation of the tap and hold operation.

  When the display orientation is selected in the adjust mode (phase = adjust mode (display direction)), the change command to rotate the 3D image 40 clockwise about the vertical axis is tapped in response to the right slide operation of the tap and hold operation. Change commands to rotate the 3D image 40 counterclockwise in the vertical axis are assigned to the left slide operation of the and hold operation. In addition, the change command for rotating the 3D image 40 clockwise about the horizontal axis substantially parallel to the direction of the right-left slide operation of the tap-and-hold operation is the tap-and-hold operation under the tap-and-hold operation. Change commands for rotating the 3D image 40 counterclockwise in the horizontal axis are assigned to slide operations.

  In the observation mode, a change command for increasing the transmittance of the selected structure is assigned to the right slide operation of the tap-and-hold operation. In addition, a change command to decrease the transmittance of the selected structure is assigned to the left slide operation of the tap-and-hold operation.

  Further, as an operation command common to all phases, an end command for ending the operation program 65 is assigned to a single tap operation accompanied with viewpoint alignment. The viewpoint alignment in this case is used to select the end button 103 shown in FIGS. 24 and 25. In addition, in the second correspondence information 67, setting of the initial display position of the display of the 3D image 40 in response to a single tap operation involving viewpoint alignment to the reset button 105 (see FIG. 24 and FIG. 25). A reset command etc. to be returned immediately after is registered.

  The viewpoint alignment is used when selecting the 3D image 40, setting an initial display position of the 3D image 40, inputting an end command, inputting a reset command, or the like. The selection command and the change command do not need to be aligned with the viewpoint, and can be input only by a double tap operation or a tap and hold operation.

  As shown in FIG. 16 and FIG. 17, the operation commands differ depending on the phase even with the same gesture operation. Therefore, the processing unit 73 always recognizes the current phase, and recognizes the gesture operation represented by the gesture information from the operation recognition unit 80 as an operation command adapted to each phase.

  Further, in this example, a double tap operation is assigned to a selection operation which is a gesture operation for selecting one of a plurality of options, and on the other hand, a change process for changing the display mode of the 3D image 40 is performed. The tap-and-hold operation is assigned to the change operation which is a gesture operation.

  In the adjustment mode, as shown in FIG. 18, the selection processing unit 85 cyclically switches the type of change processing in the order of display position, display size, and display orientation each time a double tap operation is recognized. When the double tap operation is recognized in a state where the last display orientation is selected, the selection processing unit 85 returns to the first display position and selects. At the initial stage of transition to the adjustment mode, the first display position is selected. The option information in this case is any of the display position, the display size, and the display direction.

  On the other hand, in the observation mode, as shown in FIG. 19, the selection processing unit 85 selects the structure to be subjected to the modification process as the skin 46, the subcutaneous tissue 47, and the muscle tissue 48 each time the double tap operation is recognized. , Bone 49, viscera 50, and lesion 51 are cyclically switched in this order. When there are multiple types of internal organs 50 such as liver 50A, stomach 50B, etc., each internal organ is treated as one option. Also in this case, as in the case of FIG. 18, when the double tap operation is recognized in a state where the lesion 51 whose order is the last is selected, the selection processing unit 85 returns to the first skin 46 and selects it. At the initial stage of transition to the observation mode, the first skin 46 is selected. The option information in this case is any of the plurality of structures 46 to 51. The skin 46, the subcutaneous tissue 47, and the muscle tissue 48 may be collectively treated as one option, for example, a body surface.

  The names of the respective structures are also registered in advance as specific voices recognized by the voice recognition unit. Therefore, in the observation mode, each structure can be directly selected by the user 15 uttering the name of each structure instead of the double tap operation.

  FIG. 20 is a diagram showing a process of changing the display position of the 3D image 40 in the adjustment mode (display position). The upper part of the arrow indicates that the user 15 is trying to bring the fingertips of the two fingers 38 of the hand 37 into contact with each other and to start the tap-and-hold operation. From this state, as shown in the lower part of the arrow, when the user 15 moves the hand 37 from right to left, the operation recognition unit 80 recognizes the left slide operation of the tap-and-hold operation.

  According to the second correspondence information 67 shown in FIG. 16, a change command for moving the display position of the 3D image 40 to the left is assigned to the left slide operation of the tap and hold operation in the adjust mode (display position). There is. Therefore, the change processing unit 86 performs a change process of moving the display position of the 3D image 40 to the left. The change information in this case is the amount of displacement of the display position of the 3D image 40 to the left. The displacement amount changes according to the movement distance of the hand 37, and the displacement amount also increases as the movement distance of the hand 37 increases.

  FIG. 21 is a diagram showing a process of changing the display size of the 3D image 40 in the adjustment mode (display size). The upper part of the arrow is a state where the user 15 is about to start the tap-and-hold operation, as in the upper part of the arrow in FIG. The lower part of the arrow is also in a state in which the user 15 moves the hand 37 from the right to the left, as in the lower part of the arrow in FIG. Also in this case, the operation recognition unit 80 recognizes the left slide operation of the tap-and-hold operation.

  However, according to FIG. 16, since the change command for reducing the display size of the 3D image 40 is assigned to the left slide operation of the tap-and-hold operation in the adjustment mode (display size), the change processing unit 86 Performs a change process to reduce the display size of the 3D image 40. The change information in this case is a reduction rate of the display size of the 3D image 40. The reduction ratio also increases as the movement distance of the hand 37 increases, as in the displacement amount of FIG.

  FIG. 22 is a diagram showing a process of changing the display orientation of the 3D image 40 in the adjustment mode (display orientation). Also in this case, as in the case of FIG. 20 and FIG. 21, the case where the left slide operation of the tap-and-hold operation is recognized by the operation recognition unit 80 is illustrated.

  However, according to FIG. 17, a change command for rotating the display direction of the 3D image 40 counterclockwise is assigned to the left slide operation of the tap-and-hold operation in the adjustment mode (display direction). The change processing unit 86 performs a change process of rotating the display direction of the 3D image 40 counterclockwise to the vertical axis. The change information in this case is the amount of rotation of the 3D image 40. The amount of rotation also increases as the movement distance of the hand 37 increases.

  FIG. 23 is a diagram showing the process of changing the transmittance of the 3D image 40 in the observation mode. In this case, contrary to FIGS. 20 to 22, a case where the right slide operation of the tap-and-hold operation is recognized by the operation recognition unit 80 is illustrated. Moreover, the case where the skin 46 is selected as a structure is illustrated.

  According to FIG. 17, the change command which increases the transmittance of the selected structure is assigned to the right slide operation of the tap and hold operation in the case of the observation mode. For this reason, the change processing unit 86 performs a transmittance changing process to increase the transmittance of the selected structure of the 3D image 40, in this case, the skin 46. The change information in this case is the transmittance, and as the transmittance of the hand 37 increases, the displacement width also increases.

  FIG. 23 illustrates an example in which the transmittance of the target area 87 set in advance in the 3D image 40 is changed as indicated by hatching. The target area 87 has a size that covers most of the abdomen, and is set in a square pole shape along the depth direction. In FIG. 23, the transmittance of the skin 46 in the target area 87 is increased, and the subcutaneous tissue 47, which is a structure under the skin 46, is seen through. The target area 87 may have a cylindrical or elliptical cylindrical shape. Further, the entire 3D image 40 may be set as the target area 87.

  The display control unit 74 determines the 3D image 40 from the 3D image acquisition unit 71 based on the space recognition result of the use peripheral environment of the image display system 10 from the recognition unit 72 and the recognition result of the movement of the head of the user 15 To edit. More specifically, the display control unit 74 performs enlargement / reduction processing and rotation processing on the 3D image 40 so as to have the size and the orientation as viewed from the position of the user 15 indicated by the recognition result of the head movement. These enlargement / reduction processing and rotation processing are performed independently of the display size and display orientation change processing of the change processing unit 86.

  Further, the display control unit 74 edits the 3D image 40 based on the change information from the change processing unit 86. For example, when the reduction ratio of the display size of the 3D image 40 is given as the change information from the change processing unit 86, as shown in the lower part of the arrow in FIG. To reduce

  24 and 25 show details of the virtual image displayed in the virtual space VS. FIG. 24 shows the case of the adjustment mode, and FIG. 25 shows the case of the observation mode. The display control unit 74 displays the adjustment mode display bar 90 and the observation mode display bar 91 as a virtual image in the virtual space VS in addition to the 3D image 40 and the cursor 41.

  The display control unit 74 displays the display bars 90 and 91 at symmetrical positions in the lower center of the virtual space VS. Further, the display control unit 74 makes the display bars 90 and 91 translucent, and allows the user 15 to visually recognize the 3D image 40 and the real object through the display bars 90 and 91.

  On the adjustment mode display bar 90, icons 90A, 90B, and 90C, which represent the display position, the display size, and the display direction, which are options in the adjustment mode, are arranged in order. That is, the adjustment mode display bar 90 corresponds to a menu bar in which a plurality of options are arranged in order. The display position icon 90A is a cross arrow shape, the display size icon 90B is a magnifying glass shape, and the display orientation icon 90C is a shape in which two oval arrows cross at right angles.

  In FIG. 24, in the adjust mode, in order to notify the user 15 that the mode is the adjust mode, the adjust mode display bar 90 is focused and displayed as hatched. Conversely, in the case of the observation mode of FIG. 25, the observation mode display bar 91 is displayed in focus.

  Focus display refers to displaying a selected option among a plurality of options in distinction from other options. In the focus display, for example, the selected option and the other option are displayed in different colors, such as displaying the selected option in red and the other options in gray (gray out), or the selected option A display method is adopted in which only the frame is surrounded or the display size of the selected option is made to be large and noticeable. In FIGS. 24 and 25, the display bar of one mode currently selected is displayed, for example, in amber, and the display bar of the other mode is displayed, for example, in gray.

  In FIG. 24, the display control unit 74 focuses and displays an icon of one option selected by the selection processing unit 85 among the icons 90A to 90C. FIG. 24 shows that the display position is selected as one option by the selection processing unit 85, and the icon 90A at the display position is focused and displayed as indicated by the hatched and dashed frame 92. In this case, for example, the icon of the selected option is displayed in white, the other icons are grayed out, or the icon of the selected option is surrounded by a red frame.

  In the adjust mode, the display control unit 74 changes the cursor 41 to the same shape as the icon of the selected option. In FIG. 24, since the display position is selected as one option in the selection processing unit 85, the cursor 41 is changed to the same cross arrow shape as the icon 90A of the display position.

  In FIG. 25, in the observation mode, the display control unit 74 displays the structure list 95 on the right side of the virtual space VS. In the structure list 95, names of a plurality of structures such as the skin 46 and the liver 50A are arranged in order, together with the color bar 96 indicating the display color in the virtual space VS. That is, similarly to the adjustment mode display bar 90, the structure list 95 also corresponds to a menu bar in which a plurality of options are arranged in order.

  The display control unit 74 focuses and displays the name of one option selected by the selection processing unit 85 among the names of the structures in the structure list 95. In this case, for example, the name of one option selected by the selection processing unit 85 is white, the other name is gray, and the display size of the name of one option selected by the selection processing unit 85 is , By making it larger than other names. FIG. 25 shows that the skin 46 is selected as one option by the selection processing unit 85, and the text "skin" is displayed in focus.

  Further, in the observation mode, the display control unit 74 displays the annotation 97 on the 3D image 40. The annotation 97 indicates the name of the currently selected structure (here, “skin”), a line indicating the position of the structure on the 3D image 40, and a transmittance indicator bar 98 indicating the transmittance of the structure. And consists of When the transmittance is 0% (the state where the structure is completely visible), the display control unit 74 fills the entire transmittance display bar 98 with, for example, light blue. Then, when the transmittance increases, the filled portion is reduced and the filled portion is eliminated with the transmittance of 100%.

  The display control unit 74 displays the end button 103, the read button 104, and the reset button 105 in the upper right portion of the virtual space VS. The end button 103 is a button for ending the operation program 65. By setting the viewpoint position (cursor 41) to the end button 103 and performing single-tap operation, an end command is input to the processing unit 73, and the operation program 65 is ended.

  The read button 104 is a button for causing the HMD 11 to display a list of file icons of the 3D image 40 when selecting the 3D image 40 to be displayed as a virtual object. The reset button 105 is a button for returning the display of the 3D image 40 immediately after setting of the initial display position.

  The operation of the above configuration will be described below with reference to the flowchart of FIG. First, in the conference room 14, the user 15 wears the image display system 10 on the head, turns on the power, and starts the operating program 65 on the operating system. As a result, each functional unit such as the recognition unit 72, the processing unit 73, and the display control unit 74 is built in the CPU 57 of the control device 12, and the control device 12 functions as a control device of the HMD 11.

  Then, as shown in step ST100 of FIG. 6, the 3D image 40 of the target patient to be displayed as a virtual object is selected. As a result, a distribution request for the 3D image 40 is transmitted from the 3D image acquisition unit 71 to the image storage server 19, and the 3D image 40 is distributed from the image storage server 19 to the 3D image acquisition unit 71.

  In step ST500, the 3D image 40 from the image storage server 19 is acquired by the 3D image acquisition unit 71. The user 15 sets the initial display position of the 3D image 40 as shown in step ST110 of FIG. Thus, under the control of the display control unit 74, the 3D image 40 is displayed at the set initial display position (step ST510, display control step).

  The user 15 approaches, for example, the 3D image 40 to grasp the lesion 51 in detail, or moves away from the 3D image 40 to grasp the whole image. Alternatively, the user 15 changes the face orientation and the standing position in order to observe the lesion 51 from different angles. Following the movement of the user 15, the display control unit 74 updates the display of the 3D image 40.

  The user 15 performs various gesture operations on the 3D image 40 (YES in step ST520). In the control device 12, in the operation recognition unit 80, the gesture operation is recognized based on the captured image of the depth sensor acquired by the sensor information acquisition unit 70 and the first correspondence information 66 (step ST530, operation recognition step).

  If the gesture operation recognized in step ST530 is a double tap operation (YES in step ST540), selection processing is performed in selection processing unit 85 (step ST550, selection processing step). Specifically, in the case of the adjustment mode, as shown in FIG. 18, one of the display position, the display size, and the display orientation, which is the type of change processing, is selected. Then, whenever the double tap operation is recognized, the type of change processing is cyclically switched in the order of the display position, the display size, and the display direction. On the other hand, in the case of the observation mode, as shown in FIG. 19, one of the plurality of structures 46 to 51 to be changed is selected. Then, whenever the double tap operation is recognized, the target of the change process is cyclically switched in the order of the plurality of structures 46 to 51.

  If the gesture operation recognized in step ST530 is a tap and hold operation (NO in step ST540, YES in step ST560), the change processing unit 86 performs a change process (step ST570, change process step). Specifically, in the case of the adjustment mode, as shown in FIGS. 20 to 22, any one of the display position, the display size, and the display orientation of the 3D image 40 is changed. In the case of the observation mode, as shown in FIG. 23, the transmittance changing process is performed on the selected structure. When any one of the display position, display size, and display orientation is changed, any one of the displacement amount, the scaling ratio, and the rotation amount is output from the change processing unit 86 to the display control unit 74 as the change information. Ru. When the transmittance changing process is performed, the transmittance is output from the change processing unit 86 to the display control unit 74 as the change information.

  If the gesture operation recognized in step ST530 is neither a double tap operation nor a tap and hold operation (both in step ST540 and ST560 NO), that is, if it is a single tap operation, the processing unit 73 performs other processing. It is performed (step ST580). For example, when a single tap operation is performed in the adjustment mode, a mode switching process of shifting to the observation mode is performed. Also, when a single tap operation involving viewpoint position alignment to the end button 103 is performed, the end process of the operation program 65 is performed (YES in step ST590). If the termination process of the operation program 65 is not performed (NO in step ST590), the 3D image 40 is edited by the display control unit 74 as necessary, and then the process returns to step ST510.

  The selection operation for selecting one of a plurality of options is a double-tap operation, and the change operation for changing the display mode of the 3D image 40 is a tap-and-hold operation. The change operation can be clearly distinguished, and there is no possibility that one of the selection operation and the change operation is mistakenly recognized as the other. For this reason, processing that is not intended by the user is not performed due to misrecognition. Therefore, false recognition of the gesture operation can be effectively prevented, and usability can be improved.

  The display control unit 74 has an adjust mode display bar 90 in which icons 90A to 90C of display position, display size, and display orientation, which are options in the adjust mode, are arranged in order, and a plurality of structures which are options in the observe mode. The structure list 95 in which the names are arranged in order is displayed in the virtual space VS, and the selected one option is displayed in focus in the adjustment mode display bar 90 and the structure list 95, so that the user 15 can You can see at a glance which option is selected. Especially when the option is cyclically switched only by double tap operation without the viewpoint alignment, the user 15 tends to lose sight of the currently selected option. According to such a focus display, the selected option is selected Losing is definitely prevented.

  There are three types of change processing, the display position of the 3D image 40, the display size, and the display orientation. For this reason, even if the type of change process is selected and cyclic switching is performed, the option makes one round with only three double tap operations. Therefore, it can be said that the stress of the gesture operation applied to the user 15 when selecting a desired option is relatively low.

  On the other hand, the number of structures to be subjected to the change process is relatively large, and accordingly, the stress of the gesture operation when selecting the desired option affects the user 15 more than the case of the change process type. Therefore, in the first embodiment, the user 15 can select each structure by uttering the name of each structure.

  Here, in order to reduce the stress of the gesture operation applied to the user 15, a method may be considered in which the structure is selected not by cyclically switching the structure by the double tap operation, but by a selection operation involving viewpoint alignment. Specifically, the operation of selecting a structure accompanied by viewpoint alignment is a method of performing some kind of gesture operation in a state where the cursor 41 is aligned with a desired structure. However, when a method of selecting a structure by a selection operation involving such viewpoint alignment is adopted, viewpoints are made at places such as the viscera 50 which are not visible before changing the transmittance, and the lesion 51 which is a minute part of the visceral 50. Since it is difficult to align the position (cursor 41), the viewpoint alignment is more stressful for the user 15. Therefore, the stress applied to the user 15 can be reduced as a result by selecting a plurality of structures to be changed as options and cyclically switching the structures by double tap operation.

  The change operation is an operation to increase / decrease numerical values such as a displacement amount, a scaling ratio, a rotation amount, and a transmittance, such as a display position, a display size, a display direction, or a transmittance. For this reason, as a change operation, not a gesture operation such as a single tap operation or a double tap operation which strikes the finger 38, but a slide operation in which the user 15 can image increase and decrease of numerical values such as right and left and up and down like tap and hold operation. Is reasonable.

  Here, as shown in FIG. 12, the movement of the hands 37 of GC1 and GC2 in the tap-and-hold operation is the same as the movement of the hands 37 of GA1 and GA2 in the single-tap operation shown in FIG. Therefore, if the user 15 moves the hand 37 like GC1 or GC2 in order to perform tap-and-hold operation, if the user changes his mind and stops the tap-and-hold operation, the movement of the hand 37 is exactly as illustrated. It is the same as the single tap operation shown at 10. Therefore, in this case, there is a possibility that the operation recognition unit 80 erroneously recognizes a single tap operation.

  For this reason, if a single tap operation, not a double tap operation, is adopted as a selection operation, there is a risk that the tap and hold operation, which is a change operation, is mistakenly recognized as a single tap operation, which is a selection operation. Therefore, instead of single tap operation, double tap operation is adopted as selection operation. As a result, it is easy to misinterpret as a single tap operation, but even when a proper tap-and-hold operation is adopted as the change operation, it is possible to effectively prevent the erroneous recognition of the selection operation and the change operation.

  The slide operation is not limited to the tap-and-hold operation. For example, in a state where the finger of one finger 38 such as a forefinger stands, the hand 37 is moved with any of the wrist, elbow, or shoulder as a fulcrum Good.

Second Embodiment
In the first embodiment, the adjust mode display bar 90 as a menu bar and the structure list 95 are displayed in the virtual space VS regardless of the presence or absence of the double tap operation. However, in the second embodiment shown in FIG. , Display the menu bar only when double-tap operation is recognized.

  FIG. 27 shows an example of the adjustment mode display bar 90. The upper part of the arrow indicates that the display position of the type of change processing is selected. In this state, the display control unit 74 does not display the adjustment mode display bar 90 in the virtual space VS. When the user 15 performs a double-tap operation to switch the type of change processing from the display position to the display size from the upper state of the arrow, and the double-tap operation is recognized by the operation recognition unit 80, the display control unit 74 shows the adjustment mode display bar 90 in the virtual space VS as shown in the lower part of the arrow. Then, after a predetermined time, for example, 5 seconds, the adjustment mode display bar 90 is erased.

  As described above, since the adjust mode display bar 90 is displayed only when the double tap operation is recognized, the time in which the view of the user 15 is restricted by the adjust mode display bar 90 can be shortened as much as possible. Although the adjust mode display bar 90 is illustrated in FIG. 27, the structure list 95 may also be displayed only when a double tap operation is recognized.

Third Embodiment
In the third embodiment shown in FIG. 28, in addition to displaying the menu bar only when the double tap operation is recognized, the menu bar is displayed at the viewpoint position of the user 15.

  FIG. 28 shows an example of the adjustment mode display bar 90 as in the case of FIG. 27. The upper part of the arrow indicates the state in which the display position of the type of change processing is selected. From this state, when the double tap operation is performed by the user 15 and this double tap operation is recognized by the operation recognition unit 80, the display control unit 74 displays the viewpoint of the user 15 of the virtual space VS as shown in the lower part of the arrow. The adjustment mode display bar 90 is displayed at the position, that is, the position of the cursor 41. Then, after a predetermined time, for example, 5 seconds, the adjustment mode display bar 90 is erased.

  As described above, since the adjust mode display bar 90 is displayed at the viewpoint position of the user 15, the viewpoint movement of the user 15 at the time of confirming the option being selected can be minimized. It is considered that the viewpoint of the user 15 is concentrated on the 3D image 40 in the examination of the surgical procedure of the target patient. Therefore, if the viewpoint movement of the user 15 at the time of confirming the option currently selected is minimized, the user 15 can consider the surgical procedure of the target patient without diverting the viewpoint from the 3D image 40. Also in this case, as in the second embodiment, the structure list 95 may be displayed at the viewpoint of the user 15.

  In addition, as the change process for the structure, instead of or in addition to the transmittance change process of the first embodiment, the display color, the brightness, or the sharpness of the selected structure may be changed.

  Furthermore, instead of performing the change process on the selected structure, the change process may be uniformly performed on all other structures other than the selected structure. For example, when the selected structure is the liver 50A, the permeability of the skin 46, the subcutaneous tissue 47, the muscle tissue 48, the bone 49, and the other viscera 50 of other structures other than the liver 50A is collectively changed. . In this way, when it is desired to view only the selected structure, the transmittance of the other structure can be rapidly changed to 100%, which is preferable. It may be configured that the user 15 can switch whether to perform the change process on the selected structure or to perform the change process on another structure other than the selected structure.

  A gesture operation recognition unit such as a depth sensor may not be mounted on the image display system 10. As long as at least the hand 37 of the user 15 can be imaged, the installation position of the gesture operation recognition unit does not matter. For this reason, the gesture operation recognition unit does not have to image the same field of view as the augmented reality space ARS that the user 15 recognizes. For example, the gesture operation recognition unit may be installed on the wall or ceiling of the conference room 14 where the hand 37 of the user 15 is reflected.

  In the above embodiments, the image display system 10 in which the HMD 11 and the control device 12 are integrated is illustrated, but the present invention is not limited to this. As shown in FIG. 29, the image display system 110 in which the HMD 111 and the control device 112 are separated may be used. Although FIG. 29 illustrates a desktop personal computer having a display 113 and an input device 114 configured of a keyboard and a mouse, the control device 112 may be a notebook personal computer, or a tablet. It may be a computer.

  Alternatively, as shown in FIG. 30, the image display system 120 may have the portable computer 122 that can be carried by the user 15 with his / her waist or the like carried by the portable computer 122 to perform the function of the control device of the HMD 121. In this case, the portable computer may be a dedicated product specialized for the image display system 120 instead of a commercially available personal computer.

  Also, the image storage server 19 may be responsible for all or part of the functions of the control device of the HMD. For example, the image storage server 19 may be responsible for the function of the display control unit 74. Alternatively, a server other than the image storage server 19 may be responsible for the control device of the HMD. Furthermore, a network server installed not in the computer room 18 of the medical facility 13 but in an external facility may serve as the control device of the HMD. The image storage server 19 may also be installed as a network server in an external facility.

  As described above, the hardware configuration of the computer constituting the control device of the HMD of the present invention can be variously modified. It is also possible to configure the HMD control device with a plurality of separated computers as hardware for the purpose of improving processing performance and reliability. For example, the functions of the sensor information acquisition unit 70 and the 3D image acquisition unit 71, the functions of the recognition unit 72 and the processing unit 73, and the function of the display control unit 74 are distributed to three computers. In this case, the HMD control device is configured of three computers.

  As described above, the hardware configuration of the computer can be appropriately changed in accordance with the required performance such as processing capacity, safety, and reliability. Furthermore, not only hardware but also application programs such as the operation program 65 can of course be duplicated or distributed and stored in a plurality of storage devices for the purpose of securing safety and reliability. is there.

  The image display system or the HMD is not limited to a type in which the entire user's 15 eyes are covered with the protective frame 27 and the mounting unit 26 is fixed to the head of the user 15 as in the above embodiments. An eyeglass-type image display system may be used that has a temple worn on the user's 15 ear, a nose pad that hits the lower part of the eyes, a rim that holds the screen, and the like.

  The HMD is not limited to the one that causes the user 15 to recognize the augmented reality space ARS in which the real space RS and the virtual space VS are fused. Only the virtual space VS may be recognized by the user 15.

  The 3D image 40 is not limited to the supine image of the upper body of the target patient. It may be a supine image of the whole body or an image of another part such as the head. Further, as the structure, in addition to the skin 46, the subcutaneous tissue 47 and the like exemplified in each of the above embodiments, a vascular tissue such as portal vein and inferior vena cava or a nerve tissue such as spinal cord may be included.

  In each of the above embodiments, the examination of the surgical procedure of the target patient performed in the medical facility 13 is illustrated as the application of the image display system, and the 3D image 40 is illustrated as the virtual object. However, the application of the image display system is the target patient It is not limited to the examination of the surgical procedure, and hence the virtual object is not limited to the 3D image 40. For example, the application of the image display system may be a game application, and the game character may be a virtual object.

  In each of the above-described embodiments, for example, various types of sensor information acquiring unit 70, 3D image acquiring unit 71, recognition unit 72, processing unit 73, display control unit 74, operation recognition unit 80, selection processing unit 85, change processing unit 86 and the like The hardware-like structure of a processing unit (processing unit) that executes processing is various processors as shown below.

  The various processors include a CPU, a programmable logic device (PLD), a dedicated electric circuit, and the like. The CPU is a general-purpose processor that executes software (program) and functions as various processing units, as is well known. The PLD is a processor that can change the circuit configuration after manufacture, such as an FPGA (Field Programmable Gate Array). The dedicated electric circuit is a processor having a circuit configuration specially designed to execute a specific process such as an application specific integrated circuit (ASIC).

  One processing unit may be configured of one of these various processors, or configured of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA) It may be done. In addition, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units by one processor, first, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. . Second, as typified by a system on chip (SoC) or the like, there is a mode using a processor that realizes functions of the entire system including a plurality of processing units by one IC chip. Thus, the various processing units are configured using one or more of the above-described various processors as a hardware structure.

  Furthermore, the hardware-like structure of these various processors is more specifically an electric circuit (circuitry) combining circuit elements such as semiconductor elements.

From the above description, it is possible to grasp the control device of the head mounted display according to the following supplementary item 1.
[Appendix 1]
In a control device for a head mounted display mounted on a user's head and controlling an operation of a head mounted display that causes the user to recognize a virtual space,
A display control processor for displaying a three-dimensional virtual object in the virtual space;
An operation recognition processor that recognizes a gesture operation using the user's hand on the virtual object;
When a double-tap operation in which an operation of pressing at least one finger other than the thumb toward the thumb and then pushing it up twice is recognized as the gesture operation, one of the plurality of options for the virtual object is selected. A selection processing processor that selects an option and switches the option to be selected cyclically each time the double tap operation is recognized;
The display mode of the virtual object associated with the one option selected by the selection processing processor when a slide operation for moving a hand around any of a wrist, an elbow, or a shoulder is recognized as the gesture operation. A control device for a head mounted display, comprising: a change processing processor that performs change processing to be changed.

  The present invention is not limited to the above embodiments, and it goes without saying that various configurations can be adopted without departing from the scope of the present invention. In addition, it is also possible to appropriately combine the various embodiments and the various modifications described above. The present invention also extends to storage media for storing programs in addition to programs.

10, 110, 120 Image Display System 11, 111, 121 Head Mounted Display (HMD)
12, 112, 122 Control Device 13 Medical Facility 14 Conference Room 15 User 17 Network 18 Computer Room 19 Image Storage Server 25 Main Unit 26 Mounting Unit 27 Protective Frame 28 Screen 29 Sensor Unit 30 Control Unit 31 Nose Pad 32 Inner Ring 33 Outer Ring 34 speaker 36 table 37 hand 38 finger 40 three-dimensional volume rendering image (3D image)
41 cursor 45 CT scan image 46 skin 47 subcutaneous tissue 48 muscle tissue 49 bone 50 viscera 50A liver 50B stomach 50C spleen 51 lesions 55 storage device 56 memory 57 CPU
58 data bus 65 operation program 66, 66A to 66F first correspondence information 67 second correspondence information 70 sensor information acquisition unit 71 3D image acquisition unit 72 recognition unit 73 processing unit 74 display control unit 80 operation recognition unit 85 selection processing unit 86 change Processing section 87 Target area 90 Adjust mode display bar (menu bar)
90A to 90C Icon 91 Observe mode display bar 92 Frame 95 Structure list (menu bar)
96 color bar 97 annotation 98 transmittance display bar 103 end button 104 read button 105 reset button ARS augmented reality space RS reality space VS virtual space ST100 to ST130, ST500 to ST590 steps

Claims (12)

In a control device for a head mounted display mounted on a user's head and controlling an operation of a head mounted display that causes the user to recognize a virtual space,
A display control unit for displaying a three-dimensional virtual object in the virtual space;
An operation recognition unit that recognizes a gesture operation using the user's hand on the virtual object;
When a double-tap operation in which an operation of pressing at least one finger other than the thumb toward the thumb and then pushing it up twice is recognized as the gesture operation, one of the plurality of options for the virtual object is selected. A selection processing unit that selects an option and switches the option to be selected cyclically each time the double tap operation is recognized;
The display mode of the virtual object associated with the one option selected by the selection processing unit when a slide operation for moving the hand with any one of a wrist, elbow, and shoulder as a fulcrum is recognized as the gesture operation. A control device for a head mounted display, comprising: a change processing unit that performs change processing to be changed. The display control unit displays, in the virtual space, a menu bar in which the plurality of options are arranged in order.
The control device for a head mounted display according to claim 1, wherein the one option selected by the selection processing unit is displayed in focus on the menu bar.   The control device for a head mounted display according to claim 2, wherein the display control unit displays the menu bar only when the double tap operation is recognized as the gesture operation.   The control device of the head mounted display according to claim 3, wherein the display control unit displays the menu bar at a viewpoint position of the user.   The control device for a head mounted display according to any one of claims 1 to 4, wherein the option is a display position, a display size, and a display direction of the virtual object, which is a type of the change process. The virtual object has a plurality of structures,
The control device for a head mounted display according to any one of claims 1 to 5, wherein the option is the plurality of structures to be subjected to the change process.   The head mount according to claim 6, wherein the change processing unit performs, as the change processing, a transmittance change process of changing a transmittance of one of the plurality of structures selected by the selection processing unit. Display control device.   The control device of the head mounted display according to any one of claims 1 to 7, wherein the virtual object is a three-dimensional volume rendering image of a human body.   The control device for a head mounted display according to any one of claims 1 to 8, wherein the slide operation is a tap-and-hold operation of moving a hand in a state in which the fingertips of two fingers are in contact. In a method of operating a control device of a head mounted display mounted on a user's head and controlling an operation of a head mounted display that causes the user to recognize a virtual space,
A display control step of displaying a three-dimensional virtual object in the virtual space;
An operation recognition step of recognizing a gesture operation using the user's hand on the virtual object;
When a double-tap operation in which an operation of pressing at least one finger other than the thumb toward the thumb and then pushing it up twice is recognized as the gesture operation, one of the plurality of options for the virtual object is selected. A selection processing step of selecting an option and cyclically switching the option to be selected each time the double tap operation is recognized;
The display mode of the virtual object associated with the one option selected in the selection processing step is displayed when a slide operation for moving the hand with any one of a wrist, elbow, and shoulder as a fulcrum is recognized as the gesture operation. A method of operating a control device of a head mounted display, comprising: a change processing step of changing the change processing. In an operating program of a control device of a head mounted display mounted on a user's head and controlling an operation of a head mounted display that causes the user to recognize a virtual space,
A display control function for displaying a three-dimensional virtual object in the virtual space;
An operation recognition function of recognizing a gesture operation using the user's hand on the virtual object;
When a double-tap operation in which an operation of pressing at least one finger other than the thumb toward the thumb and then pushing it up twice is recognized as the gesture operation, one of the plurality of options for the virtual object is selected. A selection processing function of selecting an option and cyclically switching the option to be selected each time the double tap operation is recognized;
The display mode of the virtual object associated with the one option selected by the selection processing function when a slide operation for moving the hand with any one of a wrist, an elbow and a shoulder as a fulcrum is recognized as the gesture operation An operating program of a control device of a head mounted display that causes a computer to execute a change processing function that performs change processing to change An image display system comprising: a head mounted display mounted on the head of a user; and a control device controlling an operation of the head mounted display, wherein the user is made to recognize the virtual space through the head mounted display.
A display control unit for displaying a three-dimensional virtual object in the virtual space;
An operation recognition unit that recognizes a gesture operation using the user's hand on the virtual object;
When a double-tap operation in which an operation of pressing at least one finger other than the thumb toward the thumb and then pushing it up twice is recognized as the gesture operation, one of the plurality of options for the virtual object is selected. A selection processing unit that selects an option and switches the option to be selected cyclically each time the double tap operation is recognized;
The display mode of the virtual object associated with the one option selected by the selection processing unit when a slide operation for moving the hand with any one of a wrist, elbow, and shoulder as a fulcrum is recognized as the gesture operation. An image display system comprising: a change processing unit that performs change processing to be changed.

Images