JP7141209B2 - Endoscope display device and endoscope system - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to an endoscope display device and an endoscope system, and more particularly to an endoscope display device and an endoscope system used for procedures using wearable devices such as smart glasses.

2. Description of the Related Art Conventionally, in the medical field, endoscope apparatuses have been widely used for observation of organs in body cavities, therapeutic treatment using treatment instruments, surgical operations under endoscopic observation, and the like. In general, an endoscope apparatus transmits an imaging signal of an object obtained by an electronic endoscope equipped with an imaging device such as a charge-coupled device (CCD) at the tip of an insertion section to a processor for image processing. Apply. An endoscopic image obtained by image processing is output from the processor to an endoscopic monitor and displayed.

In recent years, in surgical operations under endoscopic observation, a style has been proposed in which endoscopic images taken by a scopist are displayed on a spectacle-type wearable device (hereinafter referred to as wearable glasses) worn by the operator while performing the procedure. , is being put to practical use. By displaying endoscopic images on the wearable glasses, the surgeon can check the endoscopic images without moving the line of sight from the operative field to the monitor, which is advantageous in that the procedure can be performed smoothly. be.

In addition, some wearable glasses are equipped with various sensors, and by applying imagers, distance sensors, etc., wearable glasses have been developed that can superimpose and display virtual objects on endoscopic images. . Among them, wearable glasses that have an annotation function (a function to generate a drawn image according to the user's gesture) and can display the drawn image generated by the annotation superimposed on the endoscopic image, It is very convenient for use for education and information sharing, and has been attracting attention in recent years.

By the way, in an operation using wearable glasses, when the operator and the scopist stand at positions sandwiching the patient, the orientation of the endoscopic image captured by the scopist matches the orientation of the surgical field seen by the operator. Instead, it goes in the opposite direction.

Therefore, according to the positional relationship between the operator and the scopist, the direction of the endoscopic image taken by the scopist is reversed to There has been proposed an endoscope display device that matches the orientation of the endoscope (see, for example, Patent Document 1).

JP-A-2001-145640

However, in the above proposal, although the method of inverting the endoscopic image displayed on the wearable glasses of the operator is mentioned, the drawing image generated by the annotation is not mentioned at all. For example, when marking a predetermined part of an endoscopic image using annotations and sharing the marking position information, if only the endoscopic image is reversed and the drawn image is displayed as it is, the endoscopic image displayed on the wearable glasses There is a problem that the position of the marking on the image is displayed at a position different from the intended position, and the information is not conveyed correctly.

Therefore, the present invention provides an endoscopic display device and an endoscopic system that can display an endoscopic image including annotations on wearable glasses in an appropriate orientation according to the direction of the field of view of the operator. intended to provide

An endoscope display device according to one aspect of the present invention uses an image signal receiving unit capable of receiving an image signal generated by an imaging signal captured by a scopist, and a function of generating a virtual object according to a user's gesture. a drawing generating/transmitting unit that generates and transmits a drawn image, a drawing receiving unit that can receive the drawn image, and a position detecting unit that detects first position information based on the scopist. Also, it determines whether to display the endoscopic image based on the image signal in normal rotation or in reverse display according to the first position information, and rotates the drawn image received by the drawing reception unit in normal rotation. a determination unit for determining whether to display or reverse display; and a display direction of the endoscopic image and a display direction of the drawn image are switched according to the determination result of the determination unit to switch the endoscopic image and the drawn image. and a display unit for displaying the superimposed image superimposed by the image generation unit on the wearable glasses.

An endoscope system according to one aspect of the present invention includes an endoscope that outputs an imaging signal imaged by a scopist, an endoscopic image based on the imaging signal generated from the imaging signal, and a user's gesture. a superimposed image generating device that receives a drawn image generated by a function of generating a virtual object and generates a superimposed image by superimposing the endoscopic image and the drawn image; and a display device for displaying. The superimposed image generation device reverses or reverses the drawn image in accordance with second position information between the scopist and the transmission source of the drawn image, and superimposes the drawn image on the endoscopic image. It has an overlapping portion. The display device includes a drawing generation/sending unit that generates and transmits the drawn image using a function of generating a virtual object according to the user's gesture, and detects third position information based on the scopist. a position detection unit, a determination unit that determines whether to display the superimposed image in normal rotation or reverse display according to the third position information, and an image that switches the display orientation of the superimposed image according to the determination result of the determination unit. and a display unit configured to display the superimposed image switched by the image generation unit on the wearable glasses.

According to the endoscopic display device and the endoscopic system of the present invention, an endoscopic image including annotations can be displayed on the wearable glasses in an appropriate orientation according to the direction of the field of view of the operator. .

The figure explaining an example of the whole structure of the procedure using the endoscope system concerning embodiment of this invention. 1 is a block diagram for explaining an example of the overall configuration of an endoscope system according to a first embodiment; FIG. 4 is a flowchart for explaining a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the first embodiment; 4 is a flowchart for explaining an example of a procedure for determining the positional relationship between the user of the display device and the scopist; FIG. 4 is a diagram for explaining the positional relationship between a user and a scopist according to angles; FIG. 10 is a flow chart explaining another example of the procedure for determining the positional relationship between the user of the display device and the scopist; FIG. FIG. 4 is a diagram for explaining the positional relationship between a user and a scopist according to distance; The figure explaining an example of the endoscope image displayed on each display apparatus. The figure explaining an example of the endoscope image displayed on each display apparatus. The block diagram explaining an example of the whole structure of the endoscope system concerning 2nd Embodiment. 9 is a flowchart for explaining a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the second embodiment; 13 is a flowchart for explaining a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the third embodiment; The figure explaining an example of the endoscope image displayed on each display apparatus. The figure explaining an example of the endoscope image displayed on each display apparatus. Flowchart for explaining another procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the third embodiment

Embodiments will be described below with reference to the drawings.
(First embodiment)

FIG. 1 is a diagram illustrating an example of the overall configuration of a procedure using an endoscope system according to an embodiment of the present invention. The endoscope system of the present embodiment is used for surgery or the like in which a plurality of operators treat an affected area in the abdominal cavity of a patient using treatment tools such as electric scalpels under endoscopic observation.

For example, as shown in FIG. 1, an endoscopic surgery involving two operators 10a and 10b and a scopist 200 imaging an affected area 101 with an endoscope 2 will be described below. When performing endoscopic surgery on the left abdomen of a patient 100 lying in a right lateral decubitus position on an operating table, an operator 10a as a surgeon is positioned on the right side (back side) of the patient 100, An operator 10b who is an assistant is positioned on the left side (abdominal side). That is, the operator 10a and the operator 10b are positioned to face each other with the patient 100 interposed therebetween. A scopist 200 is positioned next to the operator 10 a , holds an endoscope 2 connected to the processor 3 , and photographs the affected area 101 .

Note that the number of operators is not limited to two, and three or more operators may perform the procedure, such as having a second assistant wearing separate wearable glasses participate. Also, the scopist 200 may wear wearable glasses and participate in the procedure.

Operators 10a and 10b wear display devices 1a and 1b, which are wearable glasses, respectively. The display devices 1a and 1b superimpose and display an endoscopic image captured by the scopist 200 and a drawn image added to the endoscopic image by annotation by the operators 10a and 10b.

FIG. 2 is a block diagram illustrating an example of the overall configuration of the endoscope system according to the first embodiment; The endoscope system of this embodiment includes a plurality of display devices 1 a and 1 b, an endoscope 2 and a processor 3 . In FIG. 2, solid lines indicate the flow of image signals, and dotted lines indicate the flow of communication signals.

The endoscope 2 has an elongated insertion section that can be inserted into a patient's body cavity or the like, and an imaging device such as a CCD is arranged at the distal end of the insertion section. The insertion section may be flexible or rigid (rigid endoscope used in surgical operations). The endoscope 2 is also provided with a light guide that guides illumination light to the distal end of the insertion section.

The processor 3 performs various types of processing on the video signal output from the imaging device to generate an endoscopic image to be displayed on the display devices 1a and 1b. The processor 3 includes a signal processing section 31 , a transmission section 32 and a control section 33 .

The signal processing unit 31 performs predetermined processing such as AGC processing (auto gain control processing) and CDS processing (correlated double sampling processing) on the analog image signal output from the image pickup device, and then converts it into a digital image signal. Convert to image signal. Then, the digital image signal is subjected to white balance processing, color correction processing, distortion correction processing, enhancement processing, and the like, and is output to the transmission unit 32 .

The transmission unit 32 transmits the image-processed endoscopic image input from the signal processing unit 31 to the display devices 1 a and 1 b connected to the processor 3 .

The control unit 33 transmits and receives communication signals to and from the display devices 1a and 1b. When a request is input from the display devices 1a and 1b, the content of the request is input to the signal processing unit 31 and fed back to the image processing.

The display devices 1a and 1b as endoscope display devices have a function of generating drawn images using an annotation function, and receive endoscopic images and drawn images generated by other display devices and display them appropriately. It has a function of superimposing and displaying. Since the display device 1a and the display device 1b have the same configuration, the detailed configuration will be described here using the display device 1a as an example.

The display device 1a includes a communication signal transmission/reception section 11a, an image signal reception section 12a, an image generation section 13a, a control section 14a, a position detection section 15a, and a user registration section 16a. The display device 1a also has a display section 17a, a drawing generating section 18a, a drawing transmitting section 19a, and a drawing receiving section 20a.

The communication signal transmission/reception unit 11 a transmits/receives communication signals to/from the processor 3 . A communication signal received from the processor 3 is output to the control unit 14a.

The image signal receiving unit 12 a receives an image signal (endoscopic image) that has been subjected to predetermined image processing by the processor 3 . The received endoscopic image is output to the image generator 13a.

The position detection unit 15a detects the positional relationship between the operator 10a wearing the display device 1a and the scopist 200, and determines whether or not the viewing direction is the same. The determination result is input to the control section 14a.

The user registration unit 16a registers the attributes of the user who wears the display device 1a. Attributes to be registered include, for example, whether the user is a scopist or an operator, whether or not a drawn image can be generated using the annotation function, whether or not a drawn image can be displayed, etc. For example, an authority category related to drawn images can be mentioned. The registration contents of the user registration section 16a are output to the control section 14a as necessary.

The drawing generation unit 18a uses annotations to generate a drawing image to be added to the endoscopic image. Conventionally proposed techniques can be used as a method of creating a rendered image using annotations. For example, when a user wearing the display devices 1a and 1b draws a desired figure in the air using a finger or hand, the movement of the finger or hand is detected by a camera or the like attached to the display device 1a or 1b. A method of generating a virtual object according to a gesture can be used. Examples of drawn images include a mark surrounding a specific location in an endoscopic image, an arrow pointing to a specific location, and an annotation using characters.

The drawing image created by the drawing generation unit 18a is output to the drawing transmission unit 19a. The drawing transmission unit 19a outputs the input drawing image to the drawing reception units 20a and 20b of all the connected display devices 1a and 1b.

The drawing reception unit 20a outputs the input drawing image to the image generation unit 13a. In addition, it outputs the information of the display device from which the drawn image is transmitted to the control unit 14a.

The control unit 14a as a determination unit transmits and receives necessary information and instructions from each of the communication signal transmission/reception unit 11a, the position detection unit 15a, the user registration unit 16a, and the drawing reception unit 20a, and sends them to the image generation unit 13a. In response, an instruction regarding superimposition of the endoscopic image and the rendered image is input.

FIG. 3 is a flowchart for explaining a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the first embodiment. The control unit 14a determines how to superimpose the endoscopic image and the drawing image according to the procedure shown in FIG. 3, and outputs the result to the image generation unit 13a.

First, the positional relationship between the user (operator 10a) of the display device 1a and the scopist 200 who is capturing an endoscopic image is determined (S1). FIG. 4 is a flow chart illustrating an example of a procedure for determining the positional relationship between the user of the display device and the scopist. A specific determination method of S1 will be described with reference to FIG.

First, it is determined whether or not the user wearing the display device is a scopist (S11). For the determination, for example, user attribute information registered in the user registration unit 16a is used. If it is determined to be a scopist (S11, Yes), the process proceeds to S17, it is determined that the viewing directions of the scopist and the user wearing the display device are the same, and the positional relationship determination processing ends.

On the other hand, if it is determined that the user is not a scopist (S11, No), the positional relationship between the user and the scopist is detected (S12). FIG. 5 is a diagram for explaining the positional relationship between the user and the scopist according to angles. The positional relationship between the scopist and the user of the display device is roughly classified into three patterns shown in FIGS. FIG. 5A shows a case where the scopist and the user face each other with the patient in between, but the scopist is not positioned in front of the user. For example, the positional relationship between the scopist 200 and the operator 10b in FIG. 1 corresponds to this relationship. FIG. 5(b) shows a case where the scopist and the user are facing each other across the patient, and the scopist is positioned in front of the user. For example, in FIG. 1, the positional relationship between the operator 10a and the operator 10b corresponds to this relationship. FIG. 5(c) shows the case where the scopist is positioned at the patient's feet and is 90 degrees out of alignment with the user.

In any pattern, in S12, as the positional relationship between the user and the scopist, a line extending forward from the center of the scopist's face is defined as a center line (a line indicated by a dashed dotted line in FIG. 5). A line connecting the center of the user's face and the center of the user's face and the angle θu formed by the center line are detected.

Subsequently, the angle θu detected in S12 is compared with a predetermined angle θ1 (S13). The predetermined angle θ1 is an angle set for determining whether or not the scopist and the user are in opposing positions (positional relationship in which the viewing directions are not the same). In FIG. 5, when the user is positioned within an angle θ1 from the scopist 200, it is determined that the scopist and the user are facing each other. The predetermined angle θ1 can be calculated from the width of the operating table on which the patient 100 is lying, the distance between the operators (or the operator and the scopist) standing side by side on one side of the operating table, or the like, or can be calculated from the display device. users can select arbitrary values and register in advance.

If the angle θu detected in S12 is larger than the preset angle θ1 (S13, No), it is determined that the viewing directions of the scopist and the user wearing the display device are the same (S17), and the positional relationship is determined. end the determination process. For example, when the positional relationship between the scopist 200 and the user (operator 10a or operator 10b) is the pattern shown in FIG. ) (b) corresponds to the pattern of the positional relationship between the scopist and the user.

On the other hand, if the angle θu detected in S12 is smaller than the preset angle θ1 (S13, Yes), the time during which the angle θu remains smaller than the angle θ1 is counted (S14).

If the angle θu becomes larger than the angle θ1 within a preset time (S15, No), the scopist or the user is moving or has temporarily changed the direction of the face, so the angle θu is temporarily smaller than the angle .theta.1, and the angle .theta.u is determined to be larger than the angle .theta.1 in the normal state. In this case as well, it is determined that the viewing directions of the scopist and the user wearing the display device are the same (S17), and the positional relationship determination processing ends.

On the other hand, if the angle θu continues to be smaller than the angle θ1 even after reaching the preset time (S15, Yes), it is determined that the viewing directions of the scopist and the user wearing the display device are different. (S16), and the positional relationship determination process ends. For example, this corresponds to the case where the positional relationship between the scopist 200 and the operator 10b is the pattern shown in FIGS. 5(a) and 5(b).

Note that determination of the positional relationship between the user (operator 10a) of the display device 1a and the scopist 200 who is capturing the endoscopic image is not limited to the determination method using the angle as described above, and other methods may be used. method. FIG. 6 is a flowchart illustrating another example of the procedure for determining the positional relationship between the user of the display device and the scopist. In the procedure shown in FIG. 6, the distance between the scopist and the user is used to determine the positional relationship.

First, it is determined whether or not the user wearing the display device is a scopist (S11). If it is determined to be a scopist (S11, Yes), the process proceeds to S17, it is determined that the viewing directions of the scopist and the user wearing the display device are the same, and the positional relationship determination processing ends.

On the other hand, if it is determined that the user is not a scopist (S11, No), the positional relationship between the user and the scopist is detected (S121). FIG. 7 is a diagram for explaining the positional relationship between the user and the scopist depending on the distance. The positional relationship between the scopist and the user of the display device is roughly divided into three patterns shown in FIGS. 7(a), (b) and (c). FIG. 7A shows a case where the scopist and the user face each other across the patient, but the scopist is not positioned in front of the user. For example, the positional relationship between the scopist 200 and the operator 10b in FIG. 1 corresponds to this relationship. FIG. 7B shows a case where the scopist and the user face each other across the patient, and the scopist is positioned in front of the user. For example, in FIG. 1, the positional relationship between the operator 10a and the operator 10b corresponds to this relationship. FIG. 7(c) shows the case where the scopist is positioned at the patient's feet and is 90 degrees out of alignment with the user.

In any pattern, in S121, as the positional relationship between the user and the scopist, the length of the line connecting the center of the scopist's head and the center of the user's head (=the distance between the scopist and the user ru) is detected.

Subsequently, the distance ru detected in S121 is compared with predetermined distances r1 and r2 set in advance (S131). The predetermined distances r1 and r2 are distances set to determine whether or not the scopist and the user are in opposing positions (positional relationship in which the viewing directions are not the same). In FIG. 7, when the distance ru from the scopist 200 is within the range of distances r1 and r2, it is determined that the scopist and the user are facing each other. The predetermined distances r1 and r2 are calculated from the width of the operating table on which the patient 100 is lying, the distance between the operators (or the operator and the scopist) standing side by side on one side of the operating table, or the like. A user of the display device selects an arbitrary value and registers it in advance.

If the distance ru detected in S121 is not a value between the preset distances r1 and r2 (S131, No), it is determined that the viewing directions of the scopist and the user wearing the display device are the same ( S17), the positional relationship determination process is terminated. For example, when the positional relationship between the scopist 200 and the user (operator 10a or operator 10b) is the pattern shown in FIG. ) (b) corresponds to the pattern of the positional relationship between the scopist and the user.

On the other hand, when the distance ru detected in S121 is a value between the preset distances r1 and r2 (S131, Yes), the state where the distance ru is a value between the distances r1 and r2 continues. The time spent is counted (S14).

If the distance ru changes within a preset time and is no longer a value between the distance r1 and the distance r2 (S15, No), the value of the distance ru is changed because the scopist or the user is moving. It is assumed that the value is temporarily between the distance r1 and the distance r2. In this case as well, it is determined that the viewing directions of the scopist and the user wearing the display device are the same (S17), and the positional relationship determination processing ends.

On the other hand, if the value between the preset distance r1 and the distance r2 continues even after the preset time is reached (S15, Yes), the scopist and the user wearing the display device are different from each other (S16), and the positional relationship determination process ends. For example, this corresponds to the case where the positional relationship between the scopist 200 and the operator 10b is the pattern shown in FIGS. 7(a) and 7(b).

As described above, the positional relationship between the user and the scopist 200 capturing the endoscopic image is determined (FIG. 3, S1). When it is determined that the visual field directions of the scopist and the user are the same (S1, Yes), the controller 14b outputs an instruction to the image generator 13a to erectly display the endoscopic image.

Subsequently, the control unit 14b determines whether or not the transmission source of the drawn image based on the annotation and the viewing direction of the user are the same (S3). If it is determined that the visual field directions are the same (S3, Yes), the control unit 14b outputs an instruction to the image generation unit 13a to display the drawn image upright and superimpose it on the endoscopic image ( S4). On the other hand, if it is determined that the viewing directions are not the same (S3, No), the control unit 14b outputs an instruction to the image generation unit 13a to reverse display the drawn image and superimpose it on the endoscopic image ( S5).

In S1, when it is determined that the visual field directions of the scopist and the user are not the same (S1, No), the control unit 14b outputs an instruction to the image generation unit 13a to reversely display the endoscopic image.

Subsequently, the control unit 14b determines whether or not the transmission source of the drawn image based on the annotation and the viewing direction of the user are the same (S7). If it is determined that the visual field directions are the same (S7, Yes), the control unit 14b outputs an instruction to the image generation unit 13a to display the drawn image in an upright position and superimpose it on the endoscopic image ( S8). On the other hand, if it is determined that the viewing directions are not the same (S7, No), the control unit 14b outputs an instruction to the image generation unit 13a to reverse display the drawn image and superimpose it on the endoscopic image ( S7).

The image generation unit 13a superimposes the image signal (endoscopic image) input from the image signal reception unit 12a and the drawing image input from the drawing reception unit 20a to generate a superimposed image to be displayed on the display unit 17a. do. Superimposition of the endoscopic image and the drawing image is executed according to an instruction from the control unit 14a.

The display unit 17a displays an image (a superimposed image obtained by superimposing an endoscopic image and a drawn image) generated by the image generating unit 13a.

8 and 9 are diagrams for explaining an example of an endoscopic image (superimposed image) displayed on each display device. FIG. 8A shows an example of a superimposed image when both an endoscopic image and a drawn image are displayed in an upright position. For example, in FIG. 1, when the sender of the drawn image is the operator 10a and the user of the display device is also the operator 10a, an image as shown in FIG. 8A is displayed on the user's display device 1a. be. That is, both the endoscopic image 102 captured by the scopist 200 and the drawing image (graphic surrounding the vicinity of the distal end of the treatment instrument) 103a generated on the endoscopic image 102 by the operator 10a are displayed upright.

FIG. 8(b) shows a case in which both the endoscopic image and the drawn image are displayed upright when the viewing direction of the scopist and the user are the same, but the viewing direction of the drawing image transmission source and the user are different. 4 shows an example of a superimposed image of . For example, in FIG. is the operator 10a.

Since the viewing direction of the operator 10b is not the same as that of the scopist 200, the endoscopic image is reversely displayed on the display device 1b of the operator 10b. That is, an endoscopic image 102' as shown in FIG. 9(a) is displayed. FIG. 9(a) shows that the endoscopic image 102' is reversely displayed and the drawn image 103b is displayed in the normal direction when the viewing directions of the scopist and the user are different, but the drawing image transmission source and the user's viewing direction are the same. An example of a superimposed image in the case of standing display is shown.

When the operator 10b generates the drawn image 103b on the endoscopic image 102' so as to surround the vicinity of the distal end of the treatment instrument, the drawn image 103b is arranged in the upper right part of the endoscopic image. When the rendered image 103b is displayed erectly and superimposed on the endoscopic image 102 displayed erectly, the rendered image 103b is displayed at a portion different from the vicinity of the distal end of the treatment instrument, as shown in FIG. 8(b). It will be done. That is, the drawn image 103b is displayed to the operator 10a in a part different from the part intended by the operator 10b.

Therefore, by inverting the drawn image 103b and superimposing it on the endoscopic image 102, a superimposed image as shown in FIG. 8(c) is displayed on the user's display device 1a. That is, the drawn image 103b' is displayed at the lower left of the endoscopic image 102 (=the position surrounding the vicinity of the distal end of the treatment instrument, which is the position intended by the operator 10b). FIG. 8(c) shows that the endoscopic image is displayed in an upright position and the drawn image is reversely displayed when the visual field directions of the scopist and the user are the same, but the visual field directions of the sender of the drawn image and the user are different. An example of a superimposed image is shown.

FIG. 9(b) shows a case in which the endoscopic image is reversed and the drawn image is displayed upright when the viewing direction of the scopist and the user and the viewing direction of the drawn image transmission source and the user are both different. 4 shows an example of a superimposed image of . For example, in FIG. 1, the sender of the drawn image is the operator 10a, and the user of the display device is the operator 10b. This is the case when

Since the viewing direction of the operator 10a is the same as that of the scopist 200, the endoscopic image is displayed upright on the display device 1a of the operator 10b. That is, an endoscopic image 102 as shown in FIG. 8(a) is displayed.

When the operator 10a generates the drawn image 103a on the endoscopic image 102 so as to surround the vicinity of the distal end of the treatment instrument, the drawn image 103a is arranged in the lower left portion of the endoscopic image. When the drawn image 103a is displayed in an upright position and superimposed on the reversed endoscope image 102', the drawn image 103a is displayed at a portion different from the vicinity of the distal end of the treatment instrument, as shown in FIG. 9B. It will be done. That is, for the operator 10b, the drawn image 103a is displayed in a part different from the part intended by the operator 10a.

Therefore, by inverting the drawn image 103a and superimposing it on the endoscopic image 102', a superimposed image as shown in FIG. 9C is displayed on the user's display device 1b. That is, the drawn image 103a' is displayed at the upper right of the endoscopic image 102' (=the position surrounding the vicinity of the distal end of the treatment instrument, which is the position intended by the operator 10a). FIG. 9(c) shows a superimposed image when both the endoscopic image and the drawn image are reversely displayed when the viewing directions of the scopist and the user are different, and when the drawing image transmission source and the user's viewing direction are different. An example is shown.

In this manner, the display device of the present embodiment determines the positional relationship between the scopist and the user, and displays the endoscopic image in an appropriate orientation according to the viewing direction. Also, the positional relationship between the sender of the drawn image and the user is determined, and the drawn image is displayed in an appropriate orientation according to the viewing direction. Therefore, the position of the drawn image to be superimposed and displayed on the endoscopic image is displayed at the intended position of the transmission source of the drawn image, so that information can be correctly transmitted. That is, an endoscopic image including annotations can be displayed on the wearable glasses in an appropriate orientation according to the direction of the operator's field of view.

In the above description, the endoscopic image and the drawn image are superimposed and displayed on all display devices. The superimposed image may be displayed only on the display device of the selected user, and only the endoscopic image may be displayed on the other users.
(Second embodiment)

In the above-described first embodiment, in each user's display device, the direction in which the input endoscope image is displayed and the direction in which the input drawn image is displayed are individually determined, and was superimposed and displayed in the image generation unit. In contrast, in the present embodiment, the endoscopic image and the rendered image are superimposed in advance by a device outside the display device, and each display device displays the input superimposed image in an appropriate orientation. ing.

The configuration of the endoscope system of this embodiment is shown in FIG. It is similar to the endoscope system of the first embodiment described above. Hereinafter, portions different from the endoscope system of the first embodiment will be described, and the same components will be denoted by the same reference numerals, and description thereof will be omitted.

FIG. 10 is a block diagram illustrating an example of the overall configuration of an endoscope system according to the second embodiment; The relay device 4 receives an endoscopic image from the processor 3, and also receives drawn images from the connected display devices 1a and 1b, and superimposes them on the endoscopic image in an appropriate orientation. Then, the superimposed image is output to each of the display devices 1a and 1b.

The relay device 4 as a superimposed image generating device includes a receiving section 41 , a control section 42 , drawing receiving sections 43 a and 43 b , an image superimposing section 44 and a transmitting section 45 .

The receiving unit 41 receives an image signal (endoscopic image) that has been subjected to predetermined image processing by the processor 3 . The received endoscopic image is output to the image superimposing unit 44 .

The control unit 42 transmits and receives communication signals to and from the processor 3 and the display devices 1a and 1b.

The drawing reception unit 43 a receives the drawing image generated in the display device 1 a from the drawing transmission unit 19 a and outputs the drawing image to the image superimposing unit 44 . Similarly, the drawing reception unit 43 b receives the drawing image generated in the display device 1 b from the drawing transmission unit 19 b and outputs it to the image superimposing unit 44 . When three or more display devices are connected to the relay device 4, a drawing reception unit is further provided for receiving a drawing image from each display device. The drawn images input to the respective drawn receiving units are output to the image superimposing unit 44 .

The image superimposing unit 44 displays the drawn images input from the drawing receiving units 43a and 43b in an orientation corresponding to the viewing directions of the scopist and the user who created the drawn images. superimpose on the image. A superimposed image is output to the transmission unit 45 .

The transmitter 45 outputs the superimposed image to the image signal receivers 12a and 12b of the connected display devices 1a and 1b.

The image signal receivers 12a and 12b of the display devices 1a and 1b output the received superimposed images to the image generators 13a and 13b. The image generators 13a and 13b respectively reverse the superimposed images and output them to the display units 17a and 17b when reverse display instructions are input from the controllers 14a and 14b. When an instruction for erect display is input, the superimposed image is directly output to the display units 17a and 17b.

The control units 14a and 14b determine whether or not the viewing directions of the scopist and the user wearing the display devices 1a and 1b are the same. If the visual field directions are the same, an instruction is output to the image generators 13a and 13b so as to erectly display the superimposed image. If the directions are different, an instruction is output to the image generators 13a and 13b to reversely display the superimposed image.

FIG. 11 is a flowchart illustrating a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the second embodiment. First, in S21, an image signal (endoscopic image) subjected to predetermined image processing by the processor 3 and a drawn image generated by the display device 1a or the display device 1b are input to the relay device 4. . Next, the image superimposing unit 44 of the relay device 4 determines whether or not the visual field direction of the scopist 200 who captured the endoscopic image is the same as that of the user of the display device 1a or the display device 1b which is the transmission source of the drawing image. judge. Whether or not the viewing directions are the same may be determined based on whether or not the user is positioned within a predetermined angle range from the scopist 200 as described in the first embodiment, or may be performed within a predetermined distance range. may be performed depending on whether the user is located at or not. Also, the determination is not limited to these methods, and another method may be used for determination.

For example, a case where the endoscope system of this embodiment is used in an endoscopic surgery shown in FIG. 1 will be described below. When the source of the drawn image is the display device 1a, it is determined that the viewing directions of the user 10a of the display device 1a and the scopist 200 are the same (S22, Yes). In this case, the input drawing image is directly superimposed on the endoscopic image (S23). The superimposed image generated in S23 is, for example, an image obtained by superimposing a drawn image 103a, which is an erect display of the image transmitted from the display device 1a, on the erect endoscopic image 102 shown in FIG. 8(a). .

On the other hand, when the source of the drawn image is the display device 1b, it is determined that the viewing directions of the user 10b of the display device 1b and the scopist 200 are different (S22, No). In this case, the input drawing image is reversed and superimposed on the endoscopic image (S24). For example, as shown in FIG. 8C, a drawing image 103b' obtained by reversely displaying the image transmitted from the display device 1b is superimposed on the erect endoscopic image 102. FIG.

The superimposed image generated in S23 or S24 is output from relay device 4 to display devices 1a and 1b (S25). When the superimposed image is input, the controllers 14a and 14b of the respective display devices 1a and 1b determine whether or not the viewing directions of the scopist 200 and the user of the display devices 1a and 1b are the same (S26). . The determination in S26 may be made based on whether or not the user is positioned within a predetermined angular range from the scopist 200 as described in the first embodiment, or whether or not the user is positioned within a predetermined distance range. can be done by Also, the determination is not limited to these methods, and another method may be used for determination.

Since it is determined that the viewing direction of the user 10a of the display device 1a is the same as that of the scopist 200 (S26, Yes), the control section 14a instructs the image generating section 13a to display the superimposed image in an upright position. The image generation unit 13a causes the display unit 17a to erectly display the superimposed image according to the instruction from the control unit 14a (S27). For example, the superimposed image shown in FIG. 8(a) or FIG. 8(c) is displayed on the display unit 17a of the display device 1a.

On the other hand, since it is determined that the viewing direction of the user 10b of the display device 1b is different from that of the scopist 200 (S26, No), the control unit 14b instructs the image generation unit 13b to display the superimposed image in reverse. . The image generation unit 13b causes the display unit 17a to reversely display the superimposed image according to the instruction from the control unit 14b (S28). For example, the superimposed image obtained by inverting FIG. 8(a) (the superimposed image shown in FIG. 9(c)), or the superimposed image obtained by inverting FIG. 8(c) (shown in FIG. 9(a) superimposed image) is displayed on the display unit 17b.

As described above, the endoscope system of the present embodiment creates a superimposed image by superimposing an endoscopic image and a drawn image in an appropriate orientation in advance in the relay device, and transmits the superimposed image to each display device. . Each display device determines the direction in which to display the superimposed image (upright display or reverse display) according to the visual field directions of the scopist and the user wearing the display device, and displays the superimposed image. Therefore, the display device does not need to determine the positional relationship between the source of the drawn image and the user, or superimpose the drawn image and the endoscopic image, thereby simplifying the structure of the device.
(Third embodiment)

In the first and second embodiments described above, the drawn image is reversed as it is according to the positional relationship between the source of the drawn image and the user, and superimposed on the endoscopic image. When annotations are used for the purpose of educating and supporting junior doctors, drawn images may include not only markings but also character information. For example, it is conceivable to draw notes (for example, "beware of bleeding") in characters when treating the marked site, or to draw the organ name of the marked site.

As described above, when a drawn image includes characters, if the drawn image is reversed as it is and superimposed on the endoscopic image, the characters are displayed upside down, making it difficult to read. Therefore, in the present embodiment, when characters are included in the drawn image, the characters are always displayed upright to improve visibility.

The configuration of the endoscope system of this embodiment is the endoscope system of the first embodiment described using FIG. 2 or the endoscope system of the second embodiment described using FIG. Since the configuration is the same as that of , a description thereof will be omitted. A method for displaying characters in an upright position when characters are included in a drawn image according to the present embodiment will be described below.

First, the case of using the endoscope system of the first embodiment described with reference to FIG. 2 will be described. The endoscope system of the first embodiment superimposes an endoscope image and a drawn image on each of the display devices 1a and 1b. FIG. 12 is a flowchart illustrating a procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the third embodiment. 13 and 14 are diagrams for explaining an example of an endoscopic image to be displayed on each display device.

The procedure for generating the superimposed image shown in FIG. 12 determines whether or not the drawn image contains characters when the drawn image is reversed and superimposed on the endoscopic image. is cut out and further inverted to display the character in an upright position. Therefore, detailed description of the same processing procedure is omitted.

First, the positional relationship between the user of the display devices 1a and 1b and the scopist 200 is determined, and if the viewing directions are the same (S1, Yes), the endoscopic image is displayed upright (S2). Next, the positional relationship between the source of the drawn image and the user of the display device is determined. Superimposed to generate a superimposed image (S4).

For example, when the endoscope system of the present embodiment is used in the endoscopic surgery shown in FIG. 1, when a drawn image is generated by the user 10a of the display device 1a, the display device 1a displays the image shown in FIG. 2, a superimposed image is generated by superimposing the rendered image 103a on the endoscopic image 102 displayed erect. Since the drawn image 103a is not reversed, even if the drawn image 103a contains characters, they are displayed as they are, so that visibility is not impaired.

In S3, when it is determined that the viewing direction of the display device is different from that of the user (S3, No), the drawn image is reversed (S31). Next, it is determined whether or not the drawn image includes characters (S32). It should be noted that the determination as to whether or not a character is included in the drawn image is performed using a conventional character recognition technique in an image or the like. If no characters are included (S32, No), the drawing image reversed in S31 is superimposed on the endoscopic image to generate a superimposed image (S5). If characters are included in the drawn image (S32, Yes), a region in which characters are drawn is cut out from the drawn image reversed in S31, and only the region is reversed. Then, the drawn image in which the character area is inverted is superimposed on the endoscopic image to generate a superimposed image (S33).

For example, this pattern corresponds to a case where a drawn image is generated by the user 10b of the display device 1b and a superimposed image is displayed on the display device 1a. When the drawn image does not include characters, as shown in FIG. 8C, a superimposed image is generated by superimposing a drawn image 103b′ obtained by reversing the image drawn by the operator 10b on the endoscopic image 102. be done. Although the drawn image 103b is reversed, since the drawn image 103b does not contain characters, the visibility is not impaired even if the drawn image 103b is directly superimposed on the endoscopic image.

On the other hand, when the drawn image includes characters, if a drawn image 103b' obtained by reversing the image drawn by the operator 10b is superimposed on the endoscopic image 102 as it is, the image shown in FIG. turn into. That is, since the characters of "Bleeding caution" are displayed in reverse video, the characters are difficult to read. Therefore, in S33, by cutting out and inverting only the character area, a superimposed image as shown in FIG. 13(c) can be generated, thereby improving the visibility.

The positional relationship between the users of the display devices 1a and 1b and the scopist 200 is determined, and if the viewing directions are different (S1, No), the endoscopic image is displayed in reverse (S6). Next, the positional relationship between the source of the drawn image and the user of the display device is determined. Superimposed to generate a superimposed image (S8).

For example, this pattern corresponds to a case where a drawn image is generated by the user 10b of the display device 1b and a superimposed image is displayed on the display device 1b. On the display device 1b, as shown in FIG. 14(a), a superimposed image is generated by superimposing a drawn image 103b on a reversed endoscopic image 102' in an upright position. Since the rendered image 103b is not reversed, even if the rendered image 103b contains characters, they are displayed as they are, so that visibility is not impaired.

In S7, when it is determined that the viewing direction of the display device is different from that of the user (S7, No), the drawn image is reversed (S34). Next, it is determined whether or not the drawn image includes characters (S35). If no characters are included (S35, No), the drawing image reversed in S34 is superimposed on the endoscopic image to generate a superimposed image (S9). When characters are included in the drawn image (S35, Yes), a region in which characters are drawn is cut out from the drawn image reversed in S34, and only the region is reversed. Then, the rendered image in which the character area is inverted is superimposed on the endoscopic image to generate a superimposed image (S36).

For example, this pattern corresponds to a case where a drawn image is generated by the user 10a of the display device 1a and a superimposed image is displayed on the display device 1b. When the drawn image does not contain characters, as shown in FIG. 9C, a drawn image 103a' obtained by reversing the image drawn by the operator 10a is superimposed on the reversed endoscopic image 102'. An image is generated. Although the drawn image 103a is reversed, since the drawn image 103a does not contain characters, the visibility is not impaired even if the drawn image 103a is directly superimposed on the endoscopic image.

On the other hand, in the case where the drawn image includes characters, if the drawn image 103a' obtained by reversing the image drawn by the operator 10a is superimposed on the reversed endoscopic image 102' as it is, as shown in FIG. image. That is, since the characters of "Bleeding caution" are displayed in reverse video, the characters are difficult to read. Therefore, by cutting out and inverting only the character area in S36, a superimposed image as shown in FIG. 14(c) can be generated, thereby improving the visibility.

In this manner, the display device of the present embodiment determines the positional relationship between the scopist and the user, and displays the endoscopic image in an appropriate orientation according to the viewing direction. Also, the positional relationship between the sender of the drawn image and the user is determined, and the drawn image is displayed in an appropriate orientation according to the viewing direction. Therefore, the position of the drawn image to be superimposed and displayed on the endoscopic image is displayed at the intended position of the transmission source of the drawn image, so that information can be correctly transmitted. That is, an endoscopic image including annotations can be displayed on the wearable glasses in an appropriate orientation according to the direction of the operator's field of view. Furthermore, when the drawn image is displayed in reverse, if characters are included in the drawn image, the characters are displayed upright by cutting out and inverting the character area. Therefore, regardless of the display orientation of the drawn image, the drawn characters are always displayed upright, thereby improving the visibility.

Next, the case of using the endoscope system of the second embodiment described with reference to FIG. 10 will be described. In the endoscope system of the second embodiment, the relay device 4 superimposes the endoscopic image and the drawing image. FIG. 15 is a flowchart illustrating another procedure for generating an endoscopic image (superimposed image) to be displayed on each display device according to the third embodiment.

The procedure for generating a superimposed image shown in FIG. 15 determines whether or not characters are included in the drawn image when the drawn image is reversed and superimposed on the endoscopic image. 11 is the same as the procedure for generating the superimposed image shown in FIG. Therefore, detailed description of the same processing procedure is omitted.

First, in S21, an image signal (endoscopic image) subjected to predetermined image processing by the processor 3 and a drawn image generated by the display device 1a or the display device 1b are input to the relay device 4. . Next, the image superimposing unit 44 of the relay device 4 determines whether or not the viewing direction of the scopist 200 who captured the endoscopic image is the same as that of the user of the display device 1a or the display device 1b, which is the transmission source of the drawing image. (S22). If it is determined that the visual field directions are the same (S22, Yes), the input drawing image is directly superimposed on the endoscopic image (S23) and output to each display device (S25). The superimposed image generated in S23 is, for example, as shown in FIG. 13A, an image obtained by superimposing a drawn image 103a, which is an erect image transmitted from the display device 1a, on an erect endoscopic image 102. becomes.

On the other hand, if it is determined that the visual field directions are different (S22, No), the input drawing image is reversed and superimposed on the endoscopic image (S24). Next, it is determined whether or not the drawn image includes characters (S41). If no characters are included (S41, No), the superimposed image is not subjected to any particular processing and is output to each display device as it is (S25).

When characters are included in the drawn image (S41, Yes), a region in which the characters are drawn is cut out from the reversed drawn image, and only the region is reversed (S42). Then, the superimposed image in which the character area is inverted is output to each display device (S25).

When the drawn image does not contain characters, the superimposed image generated in S24 is obtained by inverting the image transmitted from the display device 1b to the erect endoscopic image 102, for example, as shown in FIG. 8(c). It becomes an image in which the displayed drawing image 103b' is superimposed. If the drawn image contains characters, the image will be as shown in FIG. 13(b), for example. That is, since the characters of "Bleeding caution" are displayed in reverse video, it is difficult to read the characters if they are displayed as they are. Therefore, in S42, by cutting out and inverting only the character area, a superimposed image as shown in FIG. 13(c) can be generated, thereby improving the visibility.

When the superimposed image is input, the controllers 14a and 14b of the respective display devices 1a and 1b determine whether or not the viewing directions of the scopist 200 and the user of the display devices 1a and 1b are the same (S26). . When it is determined that the viewing direction is the same as that of the scopist 200 (S26, Yes), the control section 14a instructs the image generating section 13a to display the superimposed image in an upright position. The image generation unit 13a causes the display unit 17a to erectly display the superimposed image according to the instruction from the control unit 14a (S27). For example, the superimposed image shown in FIG. 8(a) or FIG. 8(c) is displayed on the display unit 17a.

On the other hand, when it is determined that the viewing direction is different from that of the scopist 200 (S26, No), the control section 14a instructs the image generating section 13a to reversely display the superimposed image (S28). Next, it is determined whether or not the superimposed image includes characters (S43). If no characters are included (S43, No), the superimposed image is displayed on the display unit 17a without being subjected to any special processing. For example, the superimposed image shown in FIG. 9(a) or FIG. 9(c) is displayed on the display unit 17a.

When characters are included in the superimposed image (S43, Yes), a region where the characters are drawn is cut out from the reversed superimposed image, and only the region is reversed (S44). Then, a superimposed image in which the character area is inverted is displayed on the display section 17a. If the characters are not reversed, for example, an image as shown in FIG. 14B will be displayed. That is, since the characters of "Bleeding caution" are displayed in reverse video, the characters are difficult to read. Therefore, in S44, by cutting out and inverting only the character area, a superimposed image as shown in FIG. 14(c) can be generated, thereby improving the visibility.

As described above, the endoscope system of the present embodiment creates a superimposed image by superimposing an endoscopic image and a drawn image in an appropriate orientation in advance in the relay device, and transmits the superimposed image to each display device. . Each display device determines the direction in which to display the superimposed image (upright display or reverse display) according to the visual field directions of the scopist and the user wearing the display device, and displays the superimposed image. Therefore, the display device does not need to determine the positional relationship between the source of the drawn image and the user, or superimpose the drawn image and the endoscopic image, thereby simplifying the structure of the device.

Furthermore, when the drawn image is displayed in reverse, if characters are included in the drawn image, the characters are displayed upright by cutting out and inverting the character area. Therefore, regardless of the display orientation of the drawn image, the drawn characters are always displayed upright, thereby improving the visibility.

Each "unit" in this specification is a conceptual representation corresponding to each function of the embodiments, and does not necessarily correspond one-to-one to specific hardware or software routines. Therefore, in this specification, the embodiment has been described assuming a virtual circuit block (part) having each function of the embodiment. Also, each step of each procedure in this embodiment may be executed in a different order, multiple steps may be executed at the same time, or a different order may be executed for each execution, as long as it does not contradict its nature. Furthermore, all or part of each step of each procedure in this embodiment may be realized by hardware.

While several embodiments of the invention have been described, these embodiments have been illustrated by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 1a, 1b... Display device 2... Endoscope 3... Processor 4... Relay device 10a, 10b... Operator 11a, 11b... Communication signal transmitter/receiver 12a, 12b... Image signal receiver 13a, 13b Image generation units 14a, 14b Control units 15a, 15b Position detection units 16a, 16b User registration units 17a, 17b Display units 18a, 18b Drawing generation units 19a, 19b Drawing transmission units , 20a, 20b... Drawing receiving unit 31... Signal processing unit 32... Sending unit 33... Control unit 41... Receiving unit 42... Control unit 43a, 43b... Drawing receiving unit 44... Image superimposing unit 45 Transmitter 100 Patient 101 Affected area 102 Endoscopic image 103a, 103b Drawn image 200 Scopist

Claims (11)

an image signal receiving unit capable of receiving an image signal generated from an imaging signal imaged by a scopist;
a drawing generation/sending unit that generates and transmits a drawn image using a function that generates a virtual object in accordance with a user's gesture ;
a drawing receiving unit capable of receiving the drawn image;
a position detection unit that detects first position information based on the scopist;
determining whether to display the endoscopic image based on the image signal in normal rotation or in reverse according to the first position information, and displaying the drawn image received by the drawing reception unit in normal rotation; a determination unit that determines whether to display in reverse;
an image generating unit that switches the display orientation of the endoscopic image and the display orientation of the drawn image according to the determination result of the determination unit, and superimposes the endoscopic image and the drawn image;
a display unit that displays the superimposed image superimposed by the image generation unit on the wearable glasses;
An endoscope display device comprising: The first position information is a relative angle between the scopist and the endoscope display device, and the determination unit determines that the relative angle is continuously included within a preset reference angle for a certain period of time. 2. The endoscopic display device according to claim 1, wherein a determination is made to the effect that the endoscopic image is to be reversely displayed. The first position information is the distance between the scopist and the endoscope display device, and the determination unit continuously keeps the distance within a preset reference distance range for a certain period of time. 2. The endoscopic display device according to claim 1, wherein the endoscopic image is determined to be displayed in reverse when the endoscopic image is displayed. The image generation unit reverses the drawn image when an instruction to reverse display the drawn image is input from the determination unit, and then reverses the characters if the drawn image includes characters. 4. The endoscope display device according to any one of claims 1 to 3, wherein the endoscope display device according to any one of claims 1 to 3, characterized in that the display is performed in normal rotation. an endoscope that outputs an imaging signal imaged by a scopist;
receiving an endoscopic image based on the imaging signal generated from the imaging signal and a drawn image generated by a function of generating a virtual object in accordance with a user's gesture, and generating the endoscopic image and the drawn image; a superimposed image generating device that generates a superimposed image by superimposing the
a display device for displaying the superimposed image on wearable glasses;
An endoscope system comprising
The superimposed image generation device reverses or reverses the drawn image in accordance with second position information between the scopist and the transmission source of the drawn image, and superimposes the drawn image on the endoscopic image. having an overlapping portion,
The display device includes a drawing generation/sending unit that generates and transmits the drawn image using a function of generating a virtual object according to the user's gesture, and detects third position information based on the scopist. a position detection unit, a determination unit that determines whether to display the superimposed image in normal rotation or reverse display according to the third position information, and an image that switches the display orientation of the superimposed image according to the determination result of the determination unit. An endoscope system, comprising: a generation unit; and a display unit that displays the superimposed image switched by the image generation unit on wearable glasses. The second position information is a relative angle between the scopist and the transmission source of the drawn image, and the superimposed image generating device keeps the relative angle within a preset reference angle for a certain period of time. 6. The endoscope system according to claim 5, wherein the drawn image is reversed and superimposed on the endoscopic image when the drawn image is reversed. The second position information is the distance between the scopist and the transmission source of the drawn image, and the superimposed image generating device keeps the distance within a preset reference distance range for a certain period of time. 6. The endoscopic system according to claim 5, wherein in some cases the rendered image is inverted and superimposed on the endoscopic image. The third position information is a relative angle between the scopist and the display device, and the determining unit determines that the superimposed 8. The endoscope system according to claim 6, wherein determination is made to reverse display the image. When the third position information is the distance between the scopist and the endoscope display device, and the determination unit determines that the distance is continuously included within a preset reference distance range for a certain period of time 8. The endoscope system according to claim 6 or 7, wherein a determination to the effect that said superimposed image is to be displayed in a reversed state is made at the beginning. When the drawn image is reversed and superimposed on the endoscopic image, the superimposed image generation device reverses the drawn image and, if the drawn image includes characters, reverses the characters, and reverses the characters. 10. The endoscope system according to any one of claims 5 to 9, wherein the display is reversed. 11. The endoscope according to claim 10, wherein when the superimposed image is reversed, the image generation unit reverses the characters when the superimposed image includes characters, and displays the characters in normal rotation. system.

Images