JP7017385B2 - Head-mounted display device, display system and display method - Google Patents

Description

The present invention relates to a head-mounted display device, a display system, and a display method suitable for medical use.

Conventionally, endoscopic devices have been widely used in the medical field. The endoscope device is a medical device having an insertion portion having an elongated flexibility, and the operator can insert the insertion portion into the subject and observe the inside of the subject. The endoscopic image in the subject captured by the endoscope can be displayed on the monitor. In recent years, as a monitor for displaying an endoscopic image, a head-mounted display device which is mounted on the head and displays an image on a display unit arranged in the user's visual sense may be adopted.

The head-mounted display device is configured to arrange a virtual display area in the field of view of the wearer and display a virtual image based on a video signal on the virtual display area. A device that employs a see-through optical system has also been developed as a head-mounted display device, and by using this display device, it is possible to display various information without obstructing the view of the outside world.

For example, when a head-mounted display device that employs such a see-through optical system is used in surgery, the patient is directly observed through the see-through optical system, and at the same time, the endoscope is displayed in the virtual display area. You can check the image.

However, the medical staff does not always see the endoscopic image, but may also see other staff and equipment in the operating room. In this case, the endoscopic image displayed on the head-mounted display device may obstruct the view. Some have the function of turning off the display of the virtual display area by touching the housing surface of the head-mounted display device, but since the housing surface is a non-sterile area, doctors and scrub nurses can display it. Cannot be turned on and off.

Therefore, Patent Document 1 discloses a technique of detecting the line-of-sight direction of a user and switching between an outer shape priority display that prioritizes an outer shape and a virtual image priority display that prioritizes a virtual image according to the detected line-of-sight direction.

Japanese Unexamined Patent Publication No. 2017-138609

However, in the proposal of Patent Document 1, the use in a medical field where a plurality of people cooperate to treat a patient, for example, is not considered, and the display is given priority according to the work mode of only one user. Switching is done. Therefore, there is a problem that effective display control cannot always be performed in the medical field or the like.

INDUSTRIAL APPLICABILITY The present invention holds a plurality of settings, reads out the optimum settings for each user, and controls the display, so that effective display can be performed when a plurality of people work in cooperation with each other. It is an object of the present invention to provide a type display device, a display system, and a display method.

The head-mounted display device according to one aspect of the present invention is attached to an image receiving unit for receiving an endoscopic image by endoscopic surgery and a housing mounted on the wearer's head, and is attached to the wearer's head. Analysis results of a display unit that forms a virtual image based on the endoscopic image in front of the eyes, a storage unit that holds setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, and an image analysis unit. The setting unit for setting which wearer the housing is mounted on, the angle detection unit for detecting the tilt angle of the housing, the tilt angle detected by the angle detection unit, and the setting. Based on the comparison with the threshold value of the angle included in the setting information about the wearer set by the unit, the control unit that determines whether the display of the virtual image is on or off, and the determination result of the control unit. A head-mounted display including a display control unit that controls the on / off of the display of the virtual image and sets the display size of the virtual image whose display is on based on the wearer set by the setting unit. In the device , when the wearer is a doctor who performs a surgical operation, the control unit directs the wearer's line of sight below a predetermined angle below the horizontal direction at the time of performing the surgery. In the case of a nurse who turns on the display of the virtual image, turns off the display of the virtual image when facing upward from the predetermined angle, and selects the necessary treatment tool by the wearer when performing a surgical operation. In some cases, the virtual image display is turned off when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction, and the virtual image is turned off when the line of sight is directed upward from a predetermined angle. Turn on the display of virtual images .

The display system according to one aspect of the present invention is a display system including a plurality of head-mounted display devices and a system controller for transmitting information on the wearer of the plurality of head-mounted display devices. The unit-mounted display device is attached to a transmission / reception unit for transmitting / receiving control information, an image receiving unit for receiving an image, and a housing mounted on the wearer's head, and is attached to the image in front of the wearer's eyes. A display unit that forms a virtual image based on this, a storage unit that holds setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, and which wearer the housing is mounted on. The setting unit to be set, the angle detection unit that detects the tilt angle of the housing, the tilt angle detected by the angle detection unit, and the angle included in the setting information about the wearer set by the setting unit. The control unit that determines on / off of the display of the virtual image based on the comparison with the threshold value of, and the setting unit that controls the display of the virtual image on / off based on the determination result of the control unit. A display control unit that sets the display size of the virtual image whose display is on based on the wearer set by, and a position detection unit that acquires position information of the housing and outputs it to the system controller by the transmission / reception unit. The system controller determines the wearer of each of the plurality of head-mounted display devices based on the position information received from the plurality of head-mounted display devices, and determines the determination result. When the wearer is a doctor who performs a surgical operation, the control unit is given to the setting unit of each head-mounted display device, and the line of sight of the wearer is lower than the horizontal direction at the time of performing the surgical operation. Turn on the display of the virtual image when facing downward from the predetermined angle, turn off the display of the virtual image when facing upward from the predetermined angle, and wear the virtual image when performing a surgical operation. When the person is a nurse who selects the necessary treatment tool, the display of the virtual image is turned off when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction, and the predetermined display is turned off. Turn on the display of the virtual image when facing upward from the angle .

The display method according to one aspect of the present invention includes an image receiving step of receiving an image and a display step of being attached to a housing mounted on the wearer's head and forming a virtual image based on the image in front of the wearer's eyes. , A storage step for holding setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, a setting step for setting which wearer the housing is mounted on, and the housing. Based on the comparison between the angle detection step for detecting the tilt angle of the body, the tilt angle detected in the angle detection step, and the angle threshold included in the setting information for the wearer set by the setting step. , The control step for determining the on / off of the display of the virtual image, the on / off control of the display of the virtual image based on the determination result of the control step, and based on the wearer set by the setting step. The control step comprises a display control step for setting the display size of the virtual image whose display is on, and the control step is such that if the wearer is a doctor performing a surgical operation, the wearer performs the surgical operation. The display of the virtual image is turned on when the line of sight is directed downward from a predetermined angle below the horizontal direction, and the display of the virtual image is turned off when the line of sight is directed upward from the predetermined angle. In the case of a nurse who selects the necessary treatment tool when performing a surgical operation, the virtual image of the wearer when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction. The display is turned off, and the display of the virtual image is turned on when the object is facing upward from a predetermined angle .

According to the present invention, by holding a plurality of settings and reading out the optimum settings for each user to control the display, it is possible to perform effective display when a plurality of people work in cooperation with each other. Has an effect.

The block diagram which shows the display system which includes the head-mounted display device which concerns on 1st Embodiment. It is explanatory drawing for demonstrating the state which the user wears the head-mounted display device of FIG. An explanatory diagram for explaining the contents of the setting information. An explanatory diagram for explaining the display control of the control unit 11 with respect to the setting contents of FIG. An explanatory diagram for explaining the display control of the control unit 11 with respect to the setting contents of FIG. The flowchart for demonstrating the operation of 1st Embodiment. The flowchart which shows the operation flow adopted in the 2nd Embodiment of this invention. An explanatory diagram for explaining the content of the setting information stored in the storage unit 12 in the second embodiment. An explanatory diagram for explaining a display area of a virtual image on a virtual display area that is visible to the nurse's eyes. The block diagram which shows the head-mounted display device which concerns on 3rd Embodiment of this invention. An explanatory diagram for explaining the content of the setting information stored in the storage unit 12 in the third embodiment. The flowchart which shows the operation flow adopted in 3rd Embodiment. The block diagram which shows the display system including the head-mounted display device which concerns on 4th Embodiment of this invention. The explanatory view for demonstrating the determination method of the wearer in 4th Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a display system including a head-mounted display device according to the first embodiment of the present invention. Further, FIG. 2 is an explanatory diagram for explaining a state in which the user wears the head-mounted display device of FIG. 1.

In this embodiment, a see-through type head-mounted display device is adopted, and it is possible to optimally control the display of a virtual image for each wearer of the head-mounted display device that works in cooperation with each other. .. In addition, in this embodiment, an example of an operation in which a plurality of surgeons cooperate to perform treatment on an affected part in the abdominal cavity of a patient under endoscopic observation will be described. The device is not limited to the device used for such surgery.

In general surgery, surgery may be performed by two or more doctors who are the patient's surgeon and assistant, and three or more scrub nurses (hereinafter, simply referred to as nurses) who hand over the instruments to the doctors. Although there are many, the present embodiment is not limited to this, and supports the collaborative work of two or more workers who wear the head-mounted display device.

In the display system of the present embodiment, as shown in FIG. 1, n head-mounted display devices D1 to Dn (hereinafter, when it is not necessary to distinguish each of them), n (n is an integer of 2 or more) are used. It has a head-mounted display device D).

First, with reference to FIG. 2, a state in which a wearer 50 such as a doctor or a nurse wears the head-mounted display device D will be described. The head-mounted display device D shown in FIG. 2 shows an example configured by a spectacles type, but devices having various shapes such as a hat type and goggles type can be adopted. The head-mounted display device D has a support portion for attaching the housing containing the circuit of FIG. 1 to the head 50a. For example, when the head-mounted display device D is configured as a spectacle-shaped display device, as shown in FIG. 2, the vine 61a and the nose pad 61b of the spectacle frame 61 are main support portions, and the head-mounted display device D is a head-mounted display device. D is supported by the head 50a of the wearer 50 with a sufficient fixing force. As a result, the head-mounted display device D moves according to the movement of the head 50a of the user 50, and changes the direction according to the direction of the face.

In FIG. 2, a part of the spectacle frame 61 is provided with a circuit accommodating portion 61c in which each circuit in the head-mounted display device D of FIG. 1 is accommodated. Further, on the side portion of the spectacle frame 61, for example, an image pickup unit accommodating portion 20a for accommodating an image pickup unit 20 (see FIG. 1) for imaging a visual field range similar to the field of view of the wearer is arranged. The image pickup unit 20 gives the image captured image obtained by taking an image to the recording unit 21. The recording unit 21 is designed to record the supplied captured image.

The left and right lenses 61d fitted to the left and right rims of the spectacle frame 61 constitute a see-through optical system. The lens 61d allows light from an observation object to be transmitted and incident on the wearer's eyes, thereby making it possible to form an image (hereinafter referred to as a real image) perceived by the transmitted light.

Further, a display unit 13 and a display control unit 14, which will be described later, are provided in the circuit accommodating unit 61c, and the lens 61d constitutes a part of the display unit 13. The display control unit 14 emits video light based on the video signal from the display unit 13. The lens 61d constituting the display unit 13 guides the image light and directs its emission direction toward the wearer's eyes. As a result, a virtual display area perceived at a predetermined distance from the eyes is formed in a part of the field of view of the wearer 50, and an image (virtual image) based on the video signal is visually recognized in this virtual display area. ..

In the above description, the lens 61d has been described as constituting the display unit 13, but the lens 61d has only a function of transmitting light from an observation object, and is provided with a light guide member separate from the lens 61d. The image light based on the image signal may be guided to the eyes. Further, the lens 61d may be omitted and the see-through optical system may be configured by air. The see-through head-mounted display device according to the present embodiment may have any optical system configuration as long as the wearer can perceive both a real image and a virtual image. good.

In the surgical field, for example, the endoscope 25 of FIG. 1, the processor 30, the system controller 40, and a plurality of head-mounted display devices D are arranged. The endoscope 25 has an elongated and flexible insertion portion inserted into the body cavity of a subject and an operation portion (not shown) connected to the base end of the insertion portion and provided with various controls. Have. A cable (not shown) extends from the operation unit, and the endoscope 25 and the processor 30 are electrically connected via this cable.

The endoscope 25 irradiates the subject with illumination light from the tip of the insertion portion. An image sensor (not shown) is arranged at the tip of the insertion portion, and the reflected light (return light) from the subject reflected by the subject is imaged as a subject optical image on the light receiving surface of the image sensor. It has become. The image sensor obtains an image pickup signal by photoelectrically converting this optical image. The endoscope 25 outputs an image pickup signal acquired by the image pickup element to the processor 30.

The processor 30 has a control unit 31. The control unit 31 may be configured by a processor using a CPU or the like and operate according to a program stored in a memory (not shown) to control each unit, or may be a part of a function in an electronic circuit of hardware. Or it may be the one that realizes all.

The processor 30 has a signal processing unit 32, and the signal processing unit 32 is controlled by the control unit 31 to receive an image pickup signal from the endoscope 25 and perform signal processing to perform an endoscope image (moving image). And still images). The processor 30 gives the transmission unit 33 an endoscopic image after signal processing by the signal processing unit 32. The transmission unit 33 is controlled by the control unit 31 to transmit the input endoscopic image to the system controller 40.

The system controller 40 has a control unit 41. The control unit 41 may be configured by a processor using a CPU or the like and operate according to a program stored in a memory (not shown) to control each unit, or may be a part of a function in an electronic circuit of hardware. Or it may be the one that realizes all.

The signal receiving unit 42 of the system controller 40 is controlled by the control unit 41 to receive the endoscopic image transmitted from the transmitting unit 33 of the processor 30. The system controller 40 has image transmission units 43B1, 43B2, ..., 43Bn corresponding to the head-mounted display devices D1, D2, ..., Dn (hereinafter, when it is not necessary to distinguish between them, the image transmission unit 43B). ) Is provided. The image transmitting units 43B1, 43B2, ... 43Bn can be controlled by the control unit 41 to supply the endoscopic images received by the signal receiving unit 42 to the head-mounted display devices D1, D2, ... Dn, respectively. It has become like.

The system controller 40 has control information transmission / reception units 43A1, 43A2, ..., 43An corresponding to the head-mounted display devices D1, D2, ..., Dn (hereinafter, when it is not necessary to distinguish between them, control information transmission / reception). Part 43A) is provided. The control information transmission / reception units 43A1, 43A2, ... 43An transmit various control information from the control unit 41 to the head-mounted display devices D1, D2, ... Dn, respectively, and the head-mounted display devices D1, D2, ... Various control information from Dn can be received and supplied to the control unit 41. The control unit 41 can individually control the transmission of the endoscopic image to each head-mounted display device D by transmitting and receiving control information to and from each head-mounted display device D. It has become.

The head-mounted display devices D1, D2, ... Dn have the same configuration as each other. The head-mounted display device D includes a control unit 11, which is composed of a processor using a CPU or the like and operates according to a program stored in the storage unit 12 to control each unit. It may be present, or it may be a hardware electronic circuit that realizes a part or all of the functions.

The head-mounted display device D includes a control information transmission / reception unit 15. The control information transmission / reception unit 15 can transmit various control information from the control unit 11 to the processor 40, and can receive various control information from the processor 40 and supply the control information to the control unit 11. Further, the head-mounted display device D includes an image receiving unit 16. The image receiving unit 16 is controlled by the control unit 11 to receive the endoscopic image from the processor 40 and supply it to the display control unit 14.

The display control unit 14 gives the input endoscope image (video signal) to the display unit 13. The display unit 13 emits video light based on this video signal, and causes the video light to enter the eyes of the wearer 50 by a light guide member (not shown) of the lens 61d. In this way, the endoscopic image based on the video signal is visually recognized as a virtual image in the virtual display area perceived at a predetermined distance from the eyes, which is a part of the field of view by the eyes of the wearer 50.

In the present embodiment, the display control unit 14 outputs the endoscope image to the display unit 13 to display a virtual image (hereinafter, also referred to as an on state), and the display control unit 14 displays the endoscope image. The state in which the output is stopped and the virtual image is hidden (hereinafter, also referred to as an off state) is controlled by the control unit 11. The control unit 11 controls the display of the on / off state of the display control unit 14 based on the setting information stored in the storage unit 12.

In the present embodiment, the storage unit 12 stores setting information regarding a plurality of control targets (wearers) for each control target. FIG. 3 is an explanatory diagram for explaining the contents of this setting information. In the example of FIG. 3, it is shown that the wearer has individually registered the setting contents for the doctor, the doctor assistant, and the scrub nurse. In the example of FIG. 3, for the doctor and the doctor's assistant, the virtual image is displayed when the operator's line-of-sight direction is less than -10 degrees with respect to the horizontal direction, and the virtual image is displayed when the degree is -10 degrees or more. Indicates to be set to the off state to stop. As for the line-of-sight direction, the horizontal direction is 0 degree, the upper angle is positive, and the lower angle is negative.

Further, in the example of FIG. 3, for the scrub nurse, the virtual image is displayed when the operator's line-of-sight direction is -10 degrees or more with respect to the horizontal direction, and the virtual image is displayed when the degree is less than -10 degrees. Indicates to be set to the off state to stop. As described above, the setting information of the storage unit 12 is individually set for each wearer.

The setting information stored in the storage unit 12 can be updated by a user operation.

4A and 4B are explanatory views for explaining the display control of the control unit 11 with respect to the setting contents of FIG. The control unit 11 controls the on / off of the virtual image by comparing the line-of-sight direction of the wearer 50 with the threshold value (angle) in the setting information stored in the storage unit 12. For this display control, the head-mounted display device D is provided with an angle detection unit 17. The angle detection unit 17 can be configured by an acceleration sensor, obtains the tilt angle of the head-mounted display device D, and outputs angle information to the control unit 11. The control unit 11 determines the line-of-sight direction of the wearer of the head-mounted display device D based on the detection result of the angle detection unit 17.

The head-mounted display device D is configured to be substantially fixed to the wearer 50's head 50a mainly by a vine 61a and a nose pad 61b, and the inclination angle of the wearer 50 with respect to the line-of-sight direction is substantially defined. ing. Therefore, the control unit 11 can determine the line-of-sight direction of the wearer 50 based on the tilt angle of the head-mounted display device D detected by the angle detection unit 17.

The control unit 11 may improve the determination accuracy in the line-of-sight direction by calibration. For example, by operating the operation unit 19 with the wearer 50 facing the line of sight in the horizontal direction, the control unit 11 may determine the tilt angle at the time of the operation as the horizontal direction. The operation unit 19 is composed of switches, keys, buttons and the like (not shown), and is adapted to receive user operations and supply operation signals to the control unit 11.

For example, FIG. 4A shows settings for a doctor and a doctor assistant (hereinafter, both are simply referred to as doctors). The control unit 11 controls the display control unit 14 so as to turn it on when the doctor determines that the line of sight is directed below -10 degrees (less than -10 degrees) with respect to the horizontal direction. .. On the contrary, when the doctor determines that the line of sight is directed upward by 10 degrees or more downward with respect to the horizontal direction (-10 degrees or more), the display control unit 14 is controlled so as to turn it off.

On the other hand, FIG. 4B shows the setting for the scrub nurse. The control unit 11 controls the display control unit 14 so as to turn it off when the nurse determines that the line of sight is directed below -10 degrees (less than -10 degrees) with respect to the horizontal direction. .. On the contrary, when the nurse determines that the line of sight is directed upward by 10 degrees or more downward with respect to the horizontal direction (-10 degrees or more), the display control unit 14 is controlled so as to be in the ON state.

The doctor will perform the operation on the patient. Generally, in this surgical operation, the doctor looks at the patient lying on the bed (not shown). That is, the line-of-sight direction of the doctor is often less than -10 degrees with respect to the horizontal direction. In the surgical operation under endoscopic observation, it is preferable that the doctor's line of sight is directed toward the patient and at the same time, the endoscopic image of the patient can be observed. On the contrary, when the doctor turns his / her line of sight above the horizontal, for example, it is preferable that the display of the virtual image is turned off so that only the real image can be visually recognized for the purpose of handing over the treatment tool. Therefore, for the doctor, the on state is set at less than -10 degrees with respect to the horizontal, and the off state is set at -10 degrees or more.

On the other hand, the scrub nurse should be able to confirm the endoscopic image basically similar to that of the doctor. However, in order to hand the necessary treatment tool to the doctor, it is necessary to select the treatment tool from the treatment tool table on which the treatment tool is placed, and the line of sight is generally on the table located below the horizontal direction. When pointing, it is better to turn off the virtual image for more accurate selection work. Therefore, the nurse is controlled differently from the doctor by setting the off state at less than -10 degrees and setting the on state at -10 degrees or more with respect to the horizontal.

In the above description, an example is shown in which the threshold angle for on / off control is set to a predetermined -10 degree, but for example, the line of sight used for determining the on / off control of the virtual image for each wearer. It is also possible to store the threshold setting information desired by each wearer in the storage unit 12 by operating the operation unit 19 with the line of sight directed in the direction to obtain the inclination angle.

In the present embodiment, a setting unit 18 is provided for setting which setting information in the storage unit 12 is used by the control unit 11. The setting unit 18 is designed to set which target (wearer) in FIG. 3 the wearer is. The setting unit 18 can designate the wearer to be the target of display control to the control unit 11, for example, based on the operation of a setting button (not shown) of the operation unit 19. The control unit 11 reads out the setting information recorded in the storage unit 12 for the wearer designated by the setting unit 18. The control unit 11 controls the display of a virtual image for each head-mounted display device D based on the tilt angle detected by the angle detection unit 17 and the setting contents for the designated wearer stored in the storage unit 12. I do.

Next, the operation of the embodiment configured in this way will be described with reference to FIG. FIG. 5 is a flowchart for explaining the operation of the head-mounted display device D.

For example, a case where two doctors and a scrub nurse participate in endoscopic surgery will be described. A patient lies on an operating table (not shown), for example, two doctors are located on both sides of the patient, and a scrub nurse is waiting to hand over the treatment tools to the doctor who is the surgeon. do. It is assumed that the two doctors and the scrub nurse are wearing the head-mounted display devices D1 to D3, respectively.

In the endoscope 25, the insertion portion is inserted into, for example, the body cavity of a patient, the inside of the body cavity is imaged, and an image pickup signal is output to the processor 30. The processor 30 performs signal processing on the image pickup signal by the signal processing unit 32 to generate an endoscopic image. The transmission unit 33 transmits the generated endoscopic image to the system controller 40. The control unit 41 of the system controller 40 causes the signal receiving unit 42 to receive the endoscopic image. On the other hand, the control units 11 of the head-mounted display devices D1 to D3 worn by the doctor and the nurse request the system controller 40 to transmit the endoscopic image by the control information transmission / reception unit 15. The control unit 41 causes the head-mounted display devices D1 to D3 to transmit the endoscopic image from the image transmission units 43B1 to 43B3 based on the control information received by the control information transmission / reception unit 43A.

In this way, the endoscopic images are received by the image receiving units 16 of the head-mounted display devices D1 to D3 and supplied to the display control unit 14. When the control unit 11 controls the on state, the display control unit 14 is controlled by the control unit 11 to supply a video signal based on the endoscopic image to the display unit 13. As a result, the endoscope image is visually recognized as a virtual image by the display unit 13 in the eyes of each wearer. When the control unit 11 controls the off state, the display control unit 14 is controlled by the control unit 11 to stop the output of the endoscopic image to the display unit 13 and turn off the display of the virtual image. .. In this case, the wearer does not see the virtual image, and the wearer can easily see the real image.

For such on / off control of the virtual image, first, the setting unit 18 sets the control target. That is, the setting unit 18 of the head-mounted display devices D1 and D2 sets that the wearer is a doctor, for example, by operating the operation unit 19. Further, the setting unit 18 of the head-mounted display device D3 sets that the wearer is a scrub nurse, for example, by operating the operation unit 19.

In step S1 of FIG. 5, the control unit 11 determines the display control target according to the setting of the setting unit 18. That is, the control unit 11 of the head-mounted display devices D1 and D2 sets the target as a doctor, and the control unit 11 of the head-mounted display device D3 sets the target as a scrub nurse.

In step S2, the control unit 11 of the head-mounted display devices D1 and D2 reads the setting information regarding the doctor from the storage unit 12, and the control unit 11 of the head-mounted display device D3 sets the nurse from the storage unit 12. Read the information. The control unit 11 of each head-mounted display device D reads the angle detection result of the angle detection unit 17 in step S3, and determines the line-of-sight direction of the wearer in step S4.

In the next step S5, the control unit 11 compares the angle in the line-of-sight direction based on the angle detection result with the threshold angle included in the setting information read from the storage unit 12 (step S5). As a result of this comparison, the control unit 11 controls the virtual image on or off.

Now, for example, assume that the doctor is in the middle of an operation. In this case, the doctor looks at the patient. In this case, the line-of-sight direction is often less than -10 degrees with respect to the horizontal direction, and the control units 11 of the head-mounted display devices D1 and D2 refer to the display control unit 14 from the setting information of FIG. Instruct the ON state (step S6). This allows the doctor to confirm the patient and at the same time confirm the endoscopic image (virtual image) displayed in the virtual display area. The doctor can check the endoscopic image on the hand side while keeping the line of sight in the direction of the object to be operated on, that is, the direction of the hand, and hand eye coordination is possible.

At the time of such a doctor's operation, the nurse shall look at the treatment tool table to select the treatment tool. Then, the line-of-sight direction in this case is often less than -10 degrees with respect to the horizontal direction, and the control unit 11 of the head-mounted display device D3 refers to the display control unit 14 from the setting information of FIG. Instruct the off state (step S7). As a result, in the head-mounted display device D3 worn by the nurse, the virtual image by the display unit 13 is not displayed, the nurse can easily see on the treatment tool table, and the treatment tool can be reliably selected.

As described above, in the present embodiment, the display control on the head-mounted display devices D1 and D2 worn by the doctor and the display control on the head-mounted display device D3 worn by the nurse can be individually performed. It is possible.

When the doctor directs his / her line of sight above the angle of -10 degrees with respect to the horizontal direction for delivery of the treatment tool, the control unit 11 sets the off state and the virtual image by the display unit 13. Is hidden. This makes it possible to prevent the endoscopic image from being unnecessarily displayed and improve workability.

In addition, the nurse can confirm the endoscopic image similar to that of the doctor when the line of sight is not dropped on the treatment tool table, and it is possible to deepen the understanding of the doctor's instructions and the like.

As described above, in the present embodiment, since the setting information for each display control target can be stored and the display control target can be set respectively, the work is performed for each wearer who works in cooperation with each other. Individual display control is possible according to the contents of. This makes it possible to improve work efficiency when a plurality of people work together.

(Second embodiment)
FIG. 6 is a flowchart showing an operation flow adopted in the second embodiment of the present invention. In FIG. 6, the same procedure as in FIG. 5 is designated by the same reference numeral and description thereof will be omitted. The hardware configuration in this embodiment is the same as in FIG.

In the first embodiment, it has been described that the on or off control of the virtual image is performed by setting only the angle to be compared with the line-of-sight direction which is the threshold value of the on state and the off state as the setting information. In this embodiment, not only the on / off control of the virtual image but also the display mode in the on state is controlled.

That is, as shown in FIG. 6, when the control unit 11 makes an on determination in step S5, in the next step S8, it determines whether or not the target of display control is a nurse. When the target of the display control is a doctor, the control unit 11 performs a normal full-screen display in step S9, and when the target of the display control is a nurse, a sub-screen display is performed in step S10.

Next, the operation of the embodiment configured in this way will be described with reference to FIGS. 7 and 8. FIG. 7 is an explanatory diagram for explaining the contents of the setting information stored in the storage unit 12 in the present embodiment. Further, FIG. 8 is an explanatory diagram for explaining a display area of a virtual image on a virtual display area visible to the eyes of a nurse.

In FIG. 7, the setting information regarding the doctor and the doctor assistant is the same as in FIG. In the example of FIG. 7, the setting information regarding the nurse is different from that of FIG. 3, and for the scrub nurse, not only the off state is set at less than -10 degrees and the on state is set at -10 degrees or more with respect to the horizontal direction, but also on. It shows that the display is performed on the sub screen in the state.

Now, for example, for the selection of treatment tools to be handed over to the doctor, it is assumed that the nurse directs the line of sight below the angle of -10 degrees with respect to the horizontal direction. In this case, the control unit 11 of the head-mounted display device D worn by the nurse shifts the process from step S5 to step S7 to set the off state, and hides the virtual image by the display unit 13. This prevents the endoscopic image from being unnecessarily displayed and enables the nurse to perform reliable selection work.

The nurse shall then direct his gaze above a -10 degree angle relative to the horizontal to pass the selected treatment tool to the doctor. Then, the control unit 11 of the head-mounted display device D worn by the nurse makes an on determination in step S5, then determines in step S8 that the target is the nurse, and shifts the process to step S10. .. In step S10, the control unit 11 controls the display control unit 14 to cause the display unit 13 to display the endoscope image on a sub-screen.

FIG. 8 shows the display area of the virtual image in this case, and the display area 72 of the endoscope image (virtual image) to be visually recognized by the nurse is sufficiently small with respect to the virtual display area DV that can be formed in the display unit 13. It is a screen. For nurses, it may be better to improve the visibility of the doctor's real image than to improve the visibility of the endoscopic image for the delivery of the treatment tool. For the nurse, by visually recognizing the virtual image in the display area 72 which is a sub screen, it is possible to prevent the visibility of the doctor's real image from being deteriorated by the endoscopic image when the treatment tool is handed over.

Regarding the doctor, the control unit 11 of the head-mounted display device D worn by the doctor shifts the process from step S8 to step S9, and displays a virtual image in FIG. 8 as in the first embodiment. The full screen is displayed in the display area 71.

As described above, in the present embodiment, as in the first embodiment, not only the effect that the on / off control of the display of the virtual image can be enabled for each wearer, but also the display mode of the virtual image Also has the advantage that it can be set individually for each wearer. For example, for a nurse, a virtual image can be displayed on a sub-screen, the field of view of the nurse can be prevented from being obstructed by the endoscopic image, and the workability of the nurse can be improved.

(Third embodiment)
FIG. 9 is a block diagram showing a head-mounted display device according to a third embodiment of the present invention. In FIG. 9, the same components as those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted.

In the first and second embodiments, as setting information, an example in which only the angle to be compared with the line-of-sight direction is set as the threshold value of the on state and the off state has been described. In this embodiment, on / off control is performed based not only on the angle to be compared with the line-of-sight direction but also on the received image and the image obtained by imaging.

The head-mounted display device DG differs from the head-mounted display device D in FIG. 1 in that the feature recording unit 22 is provided. A display system using a head-mounted display device DG instead of the head-mounted display device D in FIG. 1 may be constructed, and the head-mounted display device D and the head-mounted display device DG may be constructed. A display system in which is mixed may be adopted.

The feature recording unit 22 records image features related to an image for determining display control. For example, the feature recording unit 22 records information on image features peculiar to endoscopic images, for example, image features of organ images. Further, the feature recording unit 22 records information on image features peculiar to the case where the subject is a doctor.

The image pickup unit 20 captures a field of view similar to the field of view of the wearer wearing the head-mounted display device DG, supplies the captured image to the recording unit 21, and also outputs the captured image to the control unit 11. Further, the image receiving unit 16 outputs the received endoscopic image to the control unit 11.

The control unit 11 compares the image features of the image captured by the image pickup unit 20 with the image features recorded in the feature recording unit 22 by a known image recognition process, and the image captured by the image pickup unit 20 is obtained by the doctor. It is designed to determine whether or not an image is included. Further, the control unit 11 compares the image features of the image received by the image receiving unit 16 with the image features recorded in the feature recording unit 22 by the image recognition process, and the received image is an effective endoscope. Determine if it is an image. For example, when the insertion portion of the endoscope 25 is not inserted into the body, the image received by the image receiving portion 16 is not an effective endoscope image obtained by imaging the inside of the body cavity.

FIG. 10 is an explanatory diagram for explaining the contents of the setting information stored in the storage unit 12 in the present embodiment. In FIG. 10, the setting information regarding the angle to be compared with the line-of-sight direction regarding the doctor and the nurse is the same as in FIG. In the example of FIG. 10, it is shown that the on / off control of the virtual image is performed according to the image recognition result.

That is, when the control unit 11 of the head-mounted display device DG when the wearer is a doctor determines that a valid endoscopic image has not been received based on the image recognition result, the line-of-sight direction of the wearer. The display control unit 14 is controlled so as to turn off the display of the virtual image regardless of the angle of.

Further, when the control unit 11 of the head-mounted display device DG when the wearer is a nurse determines that the subject of the image pickup unit 20 is a doctor based on the image recognition result, the control unit 11 is in the line-of-sight direction of the wearer. The display control unit 14 is controlled so as to turn off the display of the virtual image regardless of the angle.

Although the control unit 11 has described an example of performing display control based on the image recognition result of both the received image and the captured image, the control unit 11 is to perform the display control based on the image recognition result of either one. It may be.

Next, the operation of the embodiment configured in this way will be described with reference to FIG. FIG. 11 is a flowchart showing an operation flow adopted in the third embodiment. In FIG. 11, the same procedure as in FIG. 5 is designated by the same reference numeral and description thereof will be omitted. The flow of FIG. 11 is different from the flow of FIG. 5 in that the procedure of steps S11 to S15 is added.

That is, as shown in FIG. 11, the control unit 11 determines the line-of-sight direction in step S4, and then determines whether the wearer of the own machine is a doctor or a nurse in step S11. In the next step S12, the control unit 11 of the head-mounted display device DG when the wearer is a doctor executes an image recognition process for the image received by the image receiving unit 16 to perform effective endoscopy. Determine if a mirror image has been input. The control unit 11 controls the display control unit 14 so as to turn off the display of the virtual image regardless of the angle in the line-of-sight direction of the wearer when a valid endoscopic image is not input (step). S7). As a result, it is possible to prevent the visibility of the real image from deteriorating by displaying the image that does not need to be observed as a virtual image. When the control unit 11 determines that a valid endoscopic image has been input, the control unit 11 makes a determination based on the comparison between the angle detection result and the threshold value in step S5.

On the other hand, when the wearer is a nurse, the control unit 11 of the head-mounted display device DG shifts the processing from step S11 to step S14, and image recognition processing for the captured image obtained by imaging in the imaging unit 20. Is executed, and it is determined whether or not the image of the doctor is included in the captured image, that is, whether or not the nurse turns his face toward the doctor (step S15). When the nurse determines that the face is facing the doctor, the control unit 11 controls the display control unit 14 so as to turn off the display of the virtual image regardless of the angle of the wearer's line of sight (the control unit 11). Step S7). When the control unit 11 determines that the nurse does not turn his face toward the doctor, the control unit 11 makes a determination based on the comparison between the angle detection result and the threshold value in step S5.

As a result, when the nurse turns his face to the doctor to hand over the treatment tools, etc., the display of the virtual image can be turned off, improving the visibility of the real image and performing reliable handing work, etc. enable.

As described above, in the present embodiment, the same effect as that of the first embodiment can be obtained, and the case where the display of the virtual image is not necessary can be determined by the image recognition, and the visibility of the real image can be improved. .. This makes it possible to improve workability.

(Fourth Embodiment)
FIG. 12 is a block diagram showing a display system including a head-mounted display device according to a fourth embodiment of the present invention. In FIG. 12, the same components as those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted.

In each of the above embodiments, it has been described that the wearer to be the target of the display control is set by the setting unit 18 based on the user operation of the operation unit 19. On the other hand, the present embodiment shows an example in which the setting unit 18 automatically determines the wearer and sets the wearer. As for the display control in the present embodiment, any of the control methods according to the first to third embodiments may be adopted.

In the display system of FIG. 12, the system controller 80 is adopted instead of the system controller 40, and the head-mounted display device DH (DH1, DH2, ... DHn) is adopted instead of the head-mounted display device D. Is. The head-mounted display device DH employs a position detection unit 24, and the setting unit 18 sets the wearer based on the information from the system controller 80 received by the control information transmission / reception unit 15. Different from the embodiment.

The position detection unit 24 can detect the position of the head-mounted display device DH by using, for example, GPS (global positioning system) and supply the detection result to the control unit 11. The control unit 11 transfers the position detection result to the system controller 80 via the control information transmission / reception unit 15.

The control unit 41 of the system controller 80 is given a position detection result from each head-mounted display device DH via the control information transmission / reception unit 43A. The control unit 41 determines whether the wearer of each head-mounted display device DH is a doctor or a scrub nurse based on the received position detection results, and determines the determination result via the control information transmission / reception unit 43A. It is designed to be transmitted to each head-mounted display device DH. The setting unit 18 sets the determination result received via the control information transmission / reception unit 15 in the control unit 11.

For example, the control unit 41 may obtain the distance relationship between the head-mounted display devices DH based on the position detection result, and determine whether the user is a doctor or a nurse based on the obtained distance relationship.

FIG. 13 is an explanatory diagram for explaining the determination method in this case. FIG. 13 shows the planar shape of the operating room 91 having the doorway 92, and shows the positional relationship between the doctor and the nurse in the operating room 91. In FIG. 13, the circle 95 surrounding “Dr” indicates the position of the doctor, the circles 96 and 97 surrounding “assistant” indicate the position of the doctor assistant, and the circle 98 surrounding “view” indicates the nurse. Indicates the position of. Normally, during surgery, the doctor and the doctor's assistant are located near the bed 93 on which the patient lies, as shown by the circles 95-97 in FIG. 13, and the nurse is the treatment instrument table 94, as shown by the circle 98. Often located in the vicinity of.

The control unit 41 uses the position detection result of each head-mounted display device DH worn by the doctor and the nurse to reach each other head-mounted display device DH based on each head-mounted display device DH. Find the distance. The control unit 41 determines that the wearer is a nurse for the head-mounted display device DH having the largest sum of the distances to the other head-mounted display devices DH obtained for each head-mounted display device DH. do. As shown in FIG. 13, the doctors are located relatively close to each other, while the nurse is located relatively far from the doctor and the doctor's assistant. Therefore, it can be determined that the head-mounted display device DH having the largest sum of the distances to the other head-mounted display device DH is worn by the nurse. Further, it is considered that the head-mounted display device DH other than the head-mounted display device DH worn by the nurse is worn by a doctor or a doctor's assistant.

The control unit 41 causes each head-mounted display device DH to transmit the wearer's information about each head-mounted display device DH. In this way, the setting unit 18 of each head-mounted display device DH can automatically set the wearer of the own machine based on the information from the system controller 80.

The system controller 80 is different from the system controller 40 in that the image analysis unit 81 is provided. Further, the display system of FIG. 12 includes a camera 90. When the determination method using the position detection result described above is adopted, the image analysis unit 81 and the camera 90 can be omitted.

The camera 90 is arranged, for example, on the ceiling of the operating room, and can image a doctor, a doctor's assistant, and a nurse in the operating room. The captured image obtained by the camera 90 is supplied to the image analysis unit 81. The image analysis unit 81 extracts image features by image analysis processing on the input captured image, identifies a person in the captured image based on comparison with the image feature information recorded in a memory (not shown), and images. The analysis result is output to the control unit 41. The control unit 41 can determine the doctor and the nurse based on the image analysis result of the person, for example, by the same determination method as in FIG. 13, and acquire the detection result of the position.

Further, the image analysis unit 81 may be able to determine the doctor and the nurse in the captured image by using the image features of the doctor and the nurse. In this case, the control unit 41 can obtain the detection result of the position of the doctor and the position of the nurse based on the analysis result of the image analysis unit 81.

The control unit 41 is based on a comparison between the position detection results for the doctor and the nurse obtained by these methods and the position detection results received from each head-mounted display device DH via the control information transmission / reception unit 43A. , It is determined whether the wearer of each head-mounted display device DH is a doctor or a nurse, and the determination result is transmitted to each head-mounted display device DH.

In this way, the setting unit 18 can automatically set the wearer of the own machine based on the information from the system controller 80.

In the above description, the position detection result from each head-mounted display device DH is given to the system controller 80, the wearer who wears each head-mounted display device DH is determined by the system controller 80, and the determination result is determined. An example of transmitting to each head-mounted display device DH has been described, but the control unit 41 of the system controller 80 is controlled by the control unit 11 of any one or more head-mounted display devices DH. It may be. In this case, the system controller 80 for controlling the display of the virtual image can be omitted.

Further, in the head-mounted display device DG of FIG. 9, it is possible to determine by image recognition whether the doctor is wearing it or the nurse is wearing it alone. For example, the feature recording unit 22 records image features such as forceps, a scope, and an instrument such as gauze that the scrub nurse delivers to the doctor. The control unit 11 of the head-mounted display device DG is a device that the scrub nurse hands over to the doctor by comparing the image features of the captured image obtained by the image pickup unit 20 with the image features read from the feature recording unit 22. Is determined whether or not is imaged. For example, when the control unit 11 determines that the device has been imaged for a period of a predetermined ratio or longer, the wearer of the head-mounted display device DG is a nurse, and the other head-mounted display is displayed. The wearer of the device DG may be determined to be a doctor.

Further, when image features of the patient's surgical site, for example, image features such as drapes and tracals are recorded in the feature recording unit 22, the control unit 11 of the head-mounted display device DG captures images by the image pickup unit 20. By comparing the image features of the captured image obtained above with the image features read from the feature recording unit 22, it can be determined whether or not the patient's surgical unit is imaged. For example, when it is determined that the patient's surgical site has been imaged for a period of a predetermined ratio or longer, the wearer of the head-mounted display device DG is a doctor, and the other head-mounted display device DG. The wearer may determine that he is a nurse.

In these cases, it is determined whether or not the image corresponding to the image feature read from the feature recording unit 22 is detected depending on whether or not the size of the feature portion on the image is larger than a predetermined threshold value. You may.

Other actions such as display control are the same as those in the above embodiments.

As described above, in the present embodiment, the wearer who is the target of the display control can be automatically specified. As a result, it is possible to automatically control the display of a virtual image suitable for the wearer's work for each wearer without requiring the user's setting work, and the workability can be significantly improved.

The present invention is not limited to each of the above embodiments as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in each of the above embodiments. For example, some components of all the components shown in the embodiment may be deleted. In addition, components across different embodiments may be combined as appropriate.

For example, in each of the above embodiments, an example in which -10 degrees with respect to the horizontal direction is adopted as the threshold value for on / off control of the virtual image is shown, but the present invention is not limited to this and can be appropriately set. It is clear.

11 ... Control unit, 12 ... Storage unit, 13 ... Display unit, 14 ... Display control unit, 15 ... Control information transmission / reception unit, 16 ... Image reception unit, 17 ... Angle detection unit, 18 ... Setting unit, 19 ... Operation unit, 20 ... Image pickup unit, D ... Head-mounted display device.

Claims (11)

An image receiver that receives endoscopic images from endoscopic surgery ,
A display unit that is attached to a housing mounted on the wearer's head and forms a virtual image based on the endoscopic image in front of the wearer's eyes.
A storage unit that holds setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, and a storage unit.
A setting unit that sets which wearer the housing is attached to based on the analysis results of the image analysis unit, and a setting unit.
An angle detection unit that detects the tilt angle of the housing, and
Based on the comparison between the tilt angle detected by the angle detection unit and the threshold value of the angle included in the setting information for the wearer set by the setting unit, it is determined whether the display of the virtual image is on or off. Control unit and
A display control that controls the display of the virtual image on and off based on the determination result of the control unit, and sets the display size of the virtual image whose display is on based on the wearer set by the setting unit. In a head-mounted display device provided with a unit ,
When the wearer is a doctor who performs a surgical operation, the control unit is described when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction at the time of performing the surgery. When the display of the virtual image is turned on, the display of the virtual image is turned off when facing upward from the predetermined angle, and the wearer selects the necessary treatment tool when performing surgery. Turns off the display of the virtual image when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction, and displays the virtual image when the line of sight of the wearer is directed upward from a predetermined angle. Head-mounted display device to turn on . The head-mounted display device according to claim 1, wherein the storage unit holds the setting information when the wearer is a doctor or a nurse. The head-mounted display device according to claim 1, wherein the setting unit sets the wearer based on a user operation. The setting unit is the housing obtained based on the analysis result of the image analysis unit that analyzes the captured image captured by the imaging unit that is provided in the operating room and captures the person in the operating room and identifies the person in the captured image. The head-mounted display device according to claim 1, wherein the wearer is set according to the determination result of which wearer the wearer is wearing. The head-mounted display device according to claim 1, wherein the control unit changes the setting information based on a user operation. As the setting information, an angle indicating the line-of-sight direction for each wearer is set as a threshold value.
The control unit is characterized in that it determines on / off of the display of the virtual image based on the comparison between the angle in the line-of-sight direction of the wearer obtained from the inclination angle and the threshold value of the angle in the setting information. The head-mounted display device according to claim 1. The setting information includes information on display control of a virtual image according to the type of the image.
The control unit is characterized in that, depending on the type of image received by the image receiving unit, the control unit determines to turn off the display of the virtual image regardless of the tilt angle detected by the angle detecting unit. The head-mounted display device according to 1. Further equipped with an image pickup unit that captures an image of the surrounding area and obtains an image captured image,
The setting information includes information on display control of a virtual image according to the type of the captured image.
The head according to claim 1, wherein the control unit determines to turn off the display of the virtual image regardless of the tilt angle detected by the angle detection unit, depending on the type of the captured image. Part-mounted display device. Further equipped with an image pickup unit that captures an image of the surrounding area and obtains an image captured image,
The head-mounted display device according to claim 1, wherein the setting unit determines and sets the wearer based on image analysis of the captured image. In a display system including a plurality of head-mounted display devices and a system controller for transmitting information on the wearer of the plurality of head-mounted display devices.
The plurality of head-mounted display devices have a transmission / reception unit for transmitting / receiving control information and a transmission / reception unit, respectively.
An image receiver that receives images and
A display unit that is attached to a housing mounted on the wearer's head and forms a virtual image based on the image in front of the wearer's eyes.
A storage unit that holds setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, and a storage unit.
A setting unit that sets which wearer the housing is attached to, and
An angle detection unit that detects the tilt angle of the housing, and
Based on the comparison between the tilt angle detected by the angle detection unit and the threshold value of the angle included in the setting information for the wearer set by the setting unit, it is determined whether the display of the virtual image is on or off. Control unit and
A display control that controls the display of the virtual image on and off based on the determination result of the control unit, and sets the display size of the virtual image whose display is on based on the wearer set by the setting unit. Department and
It is provided with a position detection unit that acquires the position information of the housing and outputs the position information to the system controller by the transmission / reception unit.
The system controller is
Based on each position information received from the plurality of head-mounted display devices, the wearer of each of the plurality of head-mounted display devices is determined, and the determination result is determined by the setting of each head-mounted display device. Give to the club ,
When the wearer is a doctor who performs a surgical operation, the control unit is described when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction at the time of performing the surgery. When the display of the virtual image is turned on, the display of the virtual image is turned off when facing upward from the predetermined angle, and the wearer selects the necessary treatment tool when performing surgery. Turns off the display of the virtual image when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction, and displays the virtual image when the line of sight of the wearer is directed upward from a predetermined angle. A display system characterized by turning on . The image reception step to receive the image and
A display step that is attached to a housing mounted on the wearer's head and forms a virtual image based on the image in front of the wearer's eyes.
A storage step for holding setting information for controlling the display of the virtual image for each of a plurality of wearers of the housing, and a storage step.
A setting step for setting which wearer the housing is attached to, and
An angle detection step for detecting the inclination angle of the housing and
Based on the comparison between the tilt angle detected in the angle detection step and the threshold value of the angle included in the setting information for the wearer set by the setting step, it is determined whether the display of the virtual image is on or off. Control step,
Based on the determination result of the control step, the display of the virtual image is controlled to be turned on and off, and the display size of the virtual image whose display is turned on is set based on the wearer set by the setting step. Equipped with steps ,
The control step is described when the wearer is a doctor who performs a surgical operation, and when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction at the time of performing the surgery. When the display of the virtual image is turned on, the display of the virtual image is turned off when facing upward from the predetermined angle, and the wearer selects the necessary treatment tool when performing surgery. Turns off the display of the virtual image when the wearer's line of sight is directed downward from a predetermined angle below the horizontal direction, and displays the virtual image when the line of sight of the wearer is directed upward from a predetermined angle. A display method characterized by turning on .

Images