JP2017086688A - Image combining device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image combining device and method for optimizing a display style pf a trocar image in a composite image of an endoscopic image and a trocar image.SOLUTION: A composite image is generated by combining an endoscopic image 91 imaged by using an endoscope and a trocar images 92, 93 imaged by using a trocar with camera. A displacement detection part detects a displacement of a treatment tool 22 in the endoscopic image 91, and outputs displacement information to an image combining part. The image combining part changes a display style of trocar images 92, 93 by moving an insertion position of the trocar images 92, 93 in accordance with a displacement of the treatment tool 22.SELECTED DRAWING: Figure 20

Description

  The present invention relates to an image synthesizing apparatus and method for synthesizing endoscopic images.

  In the medical field, medical imaging devices that capture various medical images, such as endoscopes, ultrasonic apparatuses, X-ray imaging apparatuses, and surgical field cameras, are known. In medical treatment of patients, medical treatment is often performed while comparing and contrasting a plurality of types of medical images taken with these various medical devices. In particular, in surgery, an image synthesizer that synthesizes multiple types of medical images in real time so that multiple types of medical images can be observed in real time is known (see, for example, Patent Document 1).

  The image synthesizing apparatus of Patent Literature 1 has a plurality of input interfaces (input I / Fs) for inputting a plurality of types of medical images from a plurality of types of medical imaging devices, and a plurality of images input from each input I / F. The types of medical images are combined to create a combined image (paragraph 0019 to paragraph 0026, FIG. 1). The composite image is, for example, a picture-in-picture (PinP) in which another image is inserted in the image, or a picture-out picture (PoutP) in which a plurality of images are arranged so as not to overlap in one screen. (Paragraph 0027).

  Further, the image synthesizing apparatus of Patent Literature 1 automatically determines the type of the input medical image based on, for example, the color, and selects a preset synthesis pattern according to the determined type of the medical image. It has a pattern selection function. A composite image is created according to the selected composite pattern (paragraphs 0054-0070). In Patent Document 1, as one example of use of such an image synthesizing apparatus, a case where a rigid endoscope and a surgical field camera are combined for laparoscopic surgery for the purpose of educating an operator is exemplified.

  Since the subject of a rigid endoscope is an organ in the abdominal cavity, the endoscopic image is based on red, whereas the surgical field camera is a medical person working in the surgical field such as an operator or assistant Since the staff is photographed from a bird's-eye view, the subject is the appearance and procedure of the medical staff. Therefore, the operating field image by the operating camera is based on green and blue, which are the colors of clothes worn by medical staff. The difference in color distinguishes the endoscopic image and the surgical field image (paragraph 0059).

  In the case of such an educational use, as a composite pattern, a composite pattern by the PinP method in which an important endoscopic image is inserted into an endoscopic image as a parent image and an operation field image as a child image is selected (paragraph) 0088).

Japanese Patent No. 5347089

  As is well known, in endoscopic surgery such as laparoscopic surgery and thoracoscopic surgery, in addition to rigid endoscopes for securing the visual field in the body cavity such as the abdominal cavity and chest cavity, forceps, electric knife, stapler, etc. A trocar is used as a treatment tool and an insertion assisting tool for inserting the treatment tool into a body cavity. Compared with laparotomy where the site to be treated can be observed with the naked eye, the field of view is significantly limited in the endoscopic surgery where no macroscopic observation is possible. In order to perform an appropriate procedure, it is preferable that the number of blind spots is as small as possible. Therefore, securing a wide field of view is an issue. In particular, for a treatment instrument that performs a procedure in direct contact with a region to be treated, a visual field that can accurately confirm the position and posture is required.

  In order to meet these demands, the inventors are working on the development of a trocar with a camera in which a camera is provided at the tip of the trocar. Since the trocar is an insertion aid for the treatment instrument, the position and posture of the treatment instrument can be photographed by the trocar camera at the distal end portion of the trocar.

  It is preferable that an image captured by such a trocar camera (referred to as a trocar image) can be viewed in real time at the same time as an endoscopic image. In view of this, the inventors are studying to display a generated composite image by combining a trocar image and an endoscopic image by the PinP method or the PoutP method.

  However, when synthesizing a trocar image and an endoscopic image, it has been found that a problem occurs only by creating a synthesized image based on a synthesized pattern set in advance as in Patent Document 1.

  For example, in laparoscopic surgery, the field of view of the rigid endoscope is, as a rule, fixed at a position where the entire site to be treated can be seen from within the abdominal cavity. In this case, when an endoscopic image is a parent image and a trocar image is a child image and a trocar image is inserted into the endoscopic image to create a composite image, if the insertion position of the trocar image is fixed, the parent image The insertion position of the trocar image is always a blind spot in the image. When the distal end portion of the treatment instrument is located at a position that becomes a blind spot in the endoscopic image, the distal end portion of the treatment instrument in the endoscopic image is hidden by the trocar image. The problem that the position of cannot be confirmed arises.

  Further, in order to use a plurality of treatment tools, a case where a plurality of camera-equipped trocars for each treatment tool are used is conceivable. The plurality of camera-equipped trocars are disposed on the left and right, for example, with respect to the rigid endoscope. The fields of view of the left and right trocar cameras arranged in this way are reversed in the left and right directions in the endoscopic image depending on whether or not the respective photographing optical axes intersect. For example, when the photographing optical axes do not intersect, the field of view of the right-side trocar camera is targeted on the right side in the endoscopic image, and the field of view of the left-side trocar camera is on the left side in the endoscopic image. Is targeted. On the other hand, when the photographing optical axes intersect, the field of view of the right-side trocar camera is the left side in the endoscopic image, and the field of view of the left-side trocar camera is the endoscopic image The right side is the target. For this reason, the right and left of the insertion position of the trocar image do not coincide with the left and right of the field of view, depending on whether or not the photographing optical axes intersect.

  When the display mode is determined in advance as in Patent Document 1, the insertion positions of the trocar images of the trocar cameras arranged on the left and right in the endoscopic image serving as the parent image are either left or right. It is fixed to crab. Therefore, even if the left and right fields of view are reversed due to the presence or absence of the crossing of the optical axis, the insertion position of the left and right trocar images in the endoscopic image is fixed, so the relationship between the insertion position and the left and right fields of view is intuitive. There is a problem that it is difficult to grasp.

  As described above, when the endoscopic image and the trocar image are synthesized, there is a problem to be solved regarding the display mode of the synthesized image only by using the technique of the prior art, and improvement has been demanded.

  An object of the present invention is to provide an image synthesizing apparatus and method for appropriately displaying a trocar image in a synthesized image when synthesizing an endoscopic image by an endoscope and a trocar image by a trocar camera. .

  The image composition apparatus of the present invention includes an endoscope image acquisition unit, a trocar image acquisition unit, a displacement information acquisition unit, and an image composition unit. The endoscopic image acquisition unit acquires an endoscopic image in the body cavity imaged by the endoscope. The trocar image acquisition unit acquires a trocar image in the body cavity imaged by a trocar camera provided in the trocar for inserting the treatment instrument into the body cavity. The displacement information acquisition unit acquires displacement information representing the displacement of the treatment instrument and / or trocar. The image composition unit is an image composition unit that composes an endoscopic image and a trocar image to generate a composite image, and changes the display mode of the trocar image in the composite image based on the displacement information.

  As the display mode, the image composition unit preferably changes the insertion position of the trocar image in the composite image.

  There are a plurality of trocars, the trocar image acquisition unit acquires a plurality of trocar images from a trocar camera provided in each of the plurality of trocars, and the image composition unit synthesizes one endoscopic image and a plurality of trocar images. It is preferable.

  The displacement information includes information indicating the presence / absence of crossing of the photographing optical axes of the plurality of trocar cameras in the endoscopic image, and the image composition unit synthesizes according to the presence / absence of the intersection of the photographing optical axes. It is preferable to change the insertion position of a plurality of trocar images in the image.

  The image composition unit preferably inserts the trocar image into the endoscopic image by a picture-in-picture method using the endoscopic image as a parent image and the trocar image as a child image.

  When the treatment instrument is displaced in the endoscopic image, the image composition unit preferably moves the insertion position of the trocar image to a position that does not overlap the treatment instrument.

  The image composition unit determines the insertion position of the trocar image in the endoscopic image to be a position closer to the proximal end than the distal end portion of the trocar or treatment instrument on the long axis of the trocar or treatment instrument. When the trocar or the treatment instrument is displaced in the image, it is preferable to make the insertion position follow the displacement.

  It is preferable that the image composition unit has a function of hiding the trocar image in the composite image when the treatment tool moves out of the field of view of the endoscopic image.

  In the case where the trocar and the treatment tool protruding from the trocar are displayed in the endoscopic image, the image composition unit is configured to display the treatment tool when the treatment tool is retracted into the trocar. It is preferable to have a function of determining that the trocar image has moved out of the field of view and hiding the trocar image.

  It is preferable that the image composition unit has a function of continuing display without hiding the trocar image when the trocar moves together with the treatment tool out of the field of view.

  The image composition unit preferably has a function of changing at least one of a display size and a display magnification of the trocar image.

  The image composition unit preferably has a function of changing a display size of the trocar image in accordance with an approaching state of the treatment instrument and the trocar image in the composite image.

  When the trocar camera is rotated by the rotation of the trocar around the long axis, the image synthesizing unit preferably performs posture correction so that the display posture of the trocar image in the composite image matches the display posture of the endoscopic image.

  The initial position of the insertion position of the plurality of trocar images is preferably one of the four corners of the composite image.

  Preferably, the image composition unit displays port identification information for identifying each port into which a plurality of trocars are respectively inserted in a patient's body on each trocar image in the composite image.

  It is preferable to have a displacement detector that analyzes the endoscopic image, detects the displacement of the trocar and / or the treatment instrument, and outputs displacement information.

  The image composition method of the present invention includes an endoscope image acquisition step, a trocar image acquisition step, a displacement information acquisition step, and an image composition step. The endoscopic image acquisition step acquires an endoscopic image in the body cavity imaged by the endoscope. In the trocar image acquisition step, a trocar image in the body cavity is acquired by a trocar camera provided in the trocar for inserting the treatment tool into the body cavity. In the displacement information acquisition step, displacement information representing the displacement of the treatment instrument and / or the trocar is acquired. The image combining step is an image combining unit that combines the endoscope image and the trocar image to generate a combined image, and changes the display mode of the trocar image in the combined image based on the displacement information.

  According to the present invention, it is possible to provide an image composition apparatus and an operation method thereof that make a display mode of a trocar image in a composite image suitable when an endoscope image by an endoscope and a trocar image by a trocar camera are combined. Can do.

It is a schematic diagram of a laparoscopic system. It is sectional drawing in the abdominal cavity of the state which inserted the trocar. It is explanatory drawing of the positional relationship of the port hole for endoscopes, and the port hole for treatment tools. It is an external appearance perspective view of the trocar with a camera of the state which expand | deployed the camera. It is an external appearance perspective view of the trocar with a camera of the state which stored the camera. It is explanatory drawing of the storage mechanism of a retractable camera. It is a block diagram which shows the outline of the electric constitution of a laparoscope system. It is explanatory drawing of setting information. It is explanatory drawing of a synthetic | combination condition setting screen. It is explanatory drawing of a trocar left-right setting screen. It is explanatory drawing of an endoscopic image and a trocar image. It is explanatory drawing of a composite image generation process. It is explanatory drawing of a synthesized image display screen. It is explanatory drawing of the synthesized image of the state which the trocar image and the treatment tool overlapped. It is explanatory drawing of the synthesized image which saved the trocar image from the position of the treatment tool. It is explanatory drawing of the process which changes a display mode in response to the displacement of a treatment tool. It is explanatory drawing of a displacement information matching process. It is a flowchart of a display mode change process. It is explanatory drawing of the synthesized image in 2nd Embodiment. It is explanatory drawing of the synthesized image after the display mode change in 2nd Embodiment. It is explanatory drawing of the synthesized image which displayed port identification information. It is explanatory drawing of the synthesized image of the state which the imaging | photography optical axis crossed. It is explanatory drawing of the synthesized image after swapping the right and left of the trocar image from the state of FIG. It is explanatory drawing of the right-and-left exchange process of a trocar image. It is explanatory drawing of the attitude | position correction | amendment which rotates a trocar image 180 degrees. It is explanatory drawing of attitude | position correction | amendment which rotates a trocar image 90 degrees. It is explanatory drawing of the synthesized image of Example 1 of 6th Embodiment. It is explanatory drawing of a synthesized image when a treatment tool moves out of the visual field. It is a flowchart which shows the process sequence of 6th Embodiment. It is explanatory drawing of the synthesized image of Example 2 of 6th Embodiment. It is explanatory drawing of a synthesized image when a treatment tool retracts in a trocar. It is explanatory drawing of a synthesized image when a treatment tool and a trocar move out of a visual field. It is explanatory drawing of the display size change function of the trocar image of 7th Embodiment. It is explanatory drawing of the display magnification change function of a trocar image. It is explanatory drawing of the example which changes the display size of a trocar image according to the approach state with a treatment tool. It is explanatory drawing which shows the state after the change of the synthesized image of FIG. It is explanatory drawing of the flowchart of the process of FIG. It is explanatory drawing of the synthesized image of a picture out picture.

[First Embodiment]
As shown in FIG. 1, a laparoscopic system 10 that is an example of a body cavity observation system is used by a medical staff ST including a doctor to observe the inside of an abdominal cavity of a patient P during laparoscopic surgery. The laparoscopic system 10 includes an endoscope system and a trocar camera system. The endoscope system includes an endoscope 11, a processor 18, a trocar with a first camera (hereinafter referred to as a first trocar) 16, a trocar with a second camera (hereinafter referred to as a second trocar) 17, a processor 18, and a monitor 19. And a console 20.

  The first trocar 16 and the second trocar 17 are obtained by adding a camera function to an insertion instrument used as an insertion port for inserting a treatment tool 22 such as forceps into the abdominal cavity. In the following, the first trocar 16 and the second trocar 17 may be simply referred to as the trocars 16 and 17 by omitting the first and second modifications if it is not necessary to distinguish between them. is there.

  The processor 18 performs image processing for each of the endoscopic image in the abdominal cavity photographed by the endoscope 11 and the trocar images in the abdominal cavity photographed by the cameras of the first trocar 16 and the second trocar 17. The processor 18 has an image composition function for compositing an endoscopic image and each trocar image, and functions as an image composition device of the present invention. As shown in FIG. 1, a composite image of an endoscopic image and a trocar image is displayed on the monitor 19 of the processor 18. The visual field in the abdominal cavity is provided to the medical staff ST by such a composite image.

  As shown in FIGS. 2 and 3, the endoscope 11 is inserted into the abdominal cavity of the patient P through the trocar 21. Unlike the first trocar 16 and the second trocar 17, the trocar 21 for the endoscope 11 is a normal trocar without a camera. The trocar 21 is an insertion instrument used as an insertion port for inserting the endoscope 11 into the abdominal cavity. The trocar 21 has a substantially cylindrical pipe portion and a handle portion which is provided on the proximal end side of the pipe portion and has a diameter larger than that of the pipe portion. The trocar 21 is provided with an insertion hole penetrating the inside in the axial direction, and the endoscope 11 is inserted into the insertion hole.

  Further, as will be described later, since the abdominal cavity is infused by injecting carbon dioxide in the abdominal cavity, the handle portion is provided with a valve for preventing gas leakage through the insertion hole. Since the endoscope 11 is inserted into the valve, the valve is required to have a function of hermetically sealing the insertion hole while passing the endoscope 11 through the insertion hole. Therefore, for example, a duckbill valve is used as the valve.

  In laparoscopic surgery, a plurality of port holes 26, 27 for inserting the trocars 21, 16, 17 are opened in the abdominal wall 23 (see FIG. 2) of the patient P by incision. As shown in FIG. 3, in this example, the number of the port holes 26, 27 is, for example, a total of three port holes 26 for the endoscope 11 and two port holes 27 for the treatment instrument 22. It is. As for the positions of the three port holes 26, 27, for example, the port hole 26 for the endoscope 11 is provided in the center, and the port holes 27 for the treatment instrument 22 are provided on the left and right sides thereof. In addition, the number and position of the port holes 26 and 27 in this example are merely examples, and can be determined as appropriate depending on the target site for surgery, the number of treatment tools to be used, and the like.

  The trocar 21 is inserted into the port hole 26 and fixed to the abdominal wall 23. The first trocar 16 and the second trocar 17 are inserted into the two port holes 27 and fixed to the abdominal wall 23. Thereby, the trocar 21 can be used as an insertion port for the endoscope 11, and the first trocar 16 and the second trocar 17 can be used as an insertion port for the treatment instrument 22.

  As shown on the screen of the monitor 19 in FIG. 1, the endoscope 11 inserted into the central insertion port (trocar 21) provides a field of view of the entire treatment target site in the abdominal cavity. The first and second trocars 16 and 17 located on both sides of the treatment tool provide a peripheral visual field around the distal end portion 22A of the treatment instrument 22.

  As shown in FIG. 2, during laparoscopic surgery, a pneumoperitoneum treatment is performed to expand the abdominal cavity by injecting carbon dioxide gas. The first trocar 16 and the second trocar 17 are provided with a connection port 49 (see FIGS. 4 and 5) to which an air supply pipe of a gas supply device (not shown) is connected. Carbon dioxide gas supplied from the gas supply device is injected into the abdominal cavity through the first trocar 16 and the second trocar 17.

  The endoscope 11 is a rigid endoscope in which the insertion portion 11A is formed of a hard member such as metal. At the distal end of the insertion portion 11A, an illumination window that irradiates illumination light to a subject (internal organs, etc.) within the abdominal cavity, and a camera unit 28 that receives the reflected light reflected by the subject and photographs the subject (FIGS. 2 and 7). For example). The camera unit 28 includes an image pickup device (not shown) that photoelectrically converts received light, such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary metal-oxide-semiconductor) image sensor, and an image pickup surface of the image pickup device. A photographing lens (not shown) for forming an optical image of the subject.

  The imaging device is, for example, a color imaging device in which R (red), G (green), and B (blue) micro color filters are provided on the imaging surface. The three-color micro color filter is, for example, in a Bayer array, and any one color is assigned to each pixel. The camera unit 28 outputs the electrical signal photoelectrically converted by the image sensor as an RGB three-color image signal. The image sensor can shoot a moving image and outputs an image signal at a predetermined frame rate. Image signals are sequentially output to the processor 18 via signal lines.

  A signal line, a light guide, and the like are provided in the insertion portion 11A. The light guide guides illumination light supplied from a light source device (not shown) to the illumination window. At the proximal end portion of the endoscope 11, one end of a universal cable (not shown) in which a signal line and a light guide are disposed is provided. At the other end of the universal cable, a connector for connecting the light guide to the light source device and a connector for connecting the signal line to the processor 18 are provided. The endoscope 11 is connected to the light source device and the processor 18 via a universal cable.

  As shown in FIGS. 4 to 6, each of the first and second trocars 16 and 17 includes a pipe portion 31 and a handle portion 32 provided on the proximal end side of the pipe portion 31, as in the case of the normal trocar 21. It has. Each trocar 16, 17 is provided with an insertion hole 33 that passes through the pipe portion 31 and the handle portion 32 in the axial direction, and the treatment instrument 22 is inserted into the insertion hole 33. As shown in FIGS. 4 and 5, the distal end portion 22 </ b> A of the treatment instrument 22 protrudes from the outlet of the insertion hole 33 formed at the distal end portion 31 </ b> A of the pipe portion 31.

  A retractable camera unit 36 is provided at the tip 31 </ b> A of the pipe unit 31. As shown in FIG. 5, the camera unit 36 is movable between a storage position stored inside the pipe unit 31 and a deployment position where the camera unit 36 projects in a radial direction from the pipe unit 31 as shown in FIG. Is provided.

  The camera unit 36 includes a camera unit 37 (see FIG. 7) and a housing 38 that houses the camera unit 37. The camera unit 37 corresponds to a trocar camera. The camera unit 37 includes an image sensor (not shown) and a photographic lens 37A, like the camera unit 37 of the endoscope 11. Further, on both sides of the photographing lens 37A of the camera unit 37, light emitting elements 37B such as LEDs (Light Emitting Diodes) are provided as a light source of illumination light irradiating the subject.

  As shown in FIG. 6, the pipe portion 31 has a double structure of an outer cylinder 41 and an inner cylinder 42. The inner cylinder 42 is provided to be slidable in the axial direction with respect to the outer cylinder 41. The inner cylinder 42 extends to the proximal end side of the handle portion 32, and a flange-shaped operation portion 42 </ b> A for sliding the inner cylinder 42 is provided on the proximal end side of the inner cylinder 42. When the operation portion 42A is pulled to the proximal end side, the inner cylinder 42 moves backward to the proximal end side, and when the operation portion 42A is pushed to the distal end side, the inner cylinder 42 moves forward to the distal end side. The storage and deployment of the camera unit 36 are performed by the sliding operation of the inner cylinder 42 by the operation unit 42A.

  Boss 43 protruding in the horizontal direction orthogonal to the axial direction is provided on both side surfaces of the camera unit 36. The inner cylinder 42 is provided with an engagement plate 46 in which an engagement groove 46 </ b> A that engages with the boss 43 is formed. The engagement plate 46 slides in the axial direction together with the inner cylinder 42. The engaging groove 46A is inclined with respect to the axial direction, and in cooperation with the boss 43, the sliding movement of the engaging plate 46 that slides in the axial direction moves the camera unit 36 up and down perpendicular to the axial direction. It functions as a cam mechanism that converts it into a lifting motion.

  When the engagement plate 46 is retracted to the proximal end by pulling the operation portion 42A to the proximal end side with the camera portion 36 in the deployed position, the camera portion 36 is lowered by the engagement of the engagement groove 46A and the boss 43. Then, it moves from the development position to the storage position. On the contrary, when the engagement plate 46 is advanced to the distal end side by pushing the operation portion 42A toward the distal end side with the camera portion 36 in the retracted position, the camera portion 36 is moved by the engagement between the engagement groove 46A and the boss 43. Move from the storage position to the deployment position.

  The pipe part 31 is inserted into and removed from the port hole 27 in a state where the camera part 36 is stored. After the pipe part 31 is inserted into the abdominal cavity, the camera part 36 is deployed.

  In the state where the camera unit 36 is in the retracted position, a part of the housing 38 of the camera unit 36 constitutes a part of the outer peripheral surface of the distal end portion 31A of the pipe unit 31, and a step is generated from the other part of the pipe unit 31. Not formed. This prevents the pipe portion 31 from being caught when being inserted into and removed from the port hole 27.

  Moreover, in the pipe part 31, the anti-slip | skid 31B is formed behind 31 A of front-end | tip parts. The non-slip 31B is for securely fixing the pipe portion 31 to the abdominal wall 23 at a desired insertion position. The anti-slip 31B includes a plurality of irregularities formed in a circumferential direction orthogonal to the axial direction, and has a higher coefficient of friction than other portions. When the positions of the slip stopper 31B and the abdominal wall 23 are matched, the pipe portion 31 is fixed to the abdominal wall 23 by the action of the slip stopper 31B.

  As with the trocar 21, the handle portion 32 is provided with a duckbill valve (not shown) for preventing gas leakage from the insertion hole. The handle portion 32 is provided with a connector portion 48 to which a cable (not shown) for connecting to the processor 18 is detachably connected. The connector 48 is provided with a terminal that is connected to a signal line or a power supply line extending from the camera unit 37 and to which the connected wiring is connected.

  Further, the handle portion 32 is provided with a connection port 49 to which an air supply pipe of a gas supply device is connected. A carbon dioxide gas supply path that is supplied through the signal line, the power supply line, and the connection port 49 is disposed between the outer cylinder 41 and the inner cylinder 42.

  As shown in FIG. 7, the processor 18 of the laparoscopic system 10 includes a control unit 71, an image processing unit 72, a video signal processing unit 73, first to third input I / Fs 74A, 74B, 74C, and a memory 76. ing. The control unit 71 comprehensively controls each unit of the processor 18. The console 20 is an operation unit such as a keyboard, a mouse, or a foot switch, and inputs an operation signal to the control unit 71.

  The control unit 71 comprehensively controls each unit of the processor 18. The control unit 71 receives an operation of the console 20 and performs processing and various settings according to the operation content. Further, the control unit 71 outputs a drive signal to the camera unit 28 of the endoscope 11 via a driver (not shown) to control the drive of the camera unit 28. Further, the control unit 71 controls the drive of the camera unit 37 by outputting a drive signal to the camera units 37 of the first and second trocars 16 and 17 via a driver (not shown).

  The third input I / F 74C is an interface that is connected to the cable of the endoscope 11 and acquires an image signal output from the endoscope 11. The first and second input I / Fs 74A and 74B are connected to the cables of the first and second trocars 16 and 17, respectively, and acquire image signals output from the camera units 37 of the trocars 16 and 17. Interface.

  The processor 18 synthesizes the endoscopic image and the trocar image that are sequentially input from the input I / Fs 74A to 74C in real time. Therefore, the control unit 71 performs synchronous control that synchronizes the drive timings of the camera unit 37 and the camera unit 28. Thereby, the input timing of the endoscopic image to be synthesized and the trocar image can be synchronized.

  The endoscope 11 is provided with an AFE (Analog Front End) circuit 68 that processes an analog image signal output from the camera unit 28. The AFE circuit 68 performs CDS (correlated double sampling) to remove the reset noise of the image sensor, amplifies the gain of the output signal from the image sensor, and corrects so that a constant output level is maintained. An AGC (Auto Gain Control) circuit that performs analog conversion, an ADC (Analog Digital Converter) conversion circuit that converts an analog signal into a digital signal, and the like. The image signal of the endoscopic image converted into a digital signal is input to the third input I / F 74C.

  An AFE circuit that performs signal processing on an image signal output from the camera unit 37 is provided in the connector section 48 of each trocar 16, 17. The configuration and function of the AFE circuit are the same as those of the AFE circuit 68 of the endoscope 11. An image signal of a trocar image converted into a digital signal is input to each of the first and second input I / Fs 74A and 74B.

  The RGB image signal of the endoscopic image is input to the image processing unit 72 through the third input I / F 74C. In addition, the RGB image signal of the trocar image is input to the image processing unit 72 through the first input I / Fs 74A and 74B. The image processing unit 72 performs image processing such as demosaicing (synchronization) processing, white balance processing, gamma processing, and YC processing on the RGB image signal of the endoscopic image and the RGB image signal of the trocar image.

  In demosaic processing, an image signal having only one color signal of RGB for each pixel is converted into an image signal having three color signals of RGB for each pixel by interpolation processing. In the white balance process, the balance of RGB color signals with respect to white is adjusted, and in the gamma process, input / output characteristics corresponding to the camera units 28 and 37 and the monitor 19 are corrected. In the YC process, RGB image signals are converted into luminance signals (Y signals) and color difference signals (Cb signals, Cr signals).

  The image processing unit 72 has an image composition function in addition to such a general image processing function. The image processing unit 72 includes an image composition unit 77 and a displacement detection unit 78. The image composition unit 77 synthesizes one endoscopic image photographed by the endoscope 11 and the two trocar images photographed by the trocars 16 and 17 to generate a composite image.

  The image composition unit 77 receives the first and second trocar images that have undergone image processing from the first and second input I / Fs 74A and 74B, and the endoscopic image that has undergone image processing from the third input I / F 74C. Get each. In this example, the image composition unit 77 functions as an endoscopic image acquisition unit that acquires an endoscopic image and a trocar image acquisition unit that acquires a trocar image that acquires each trocar image.

  The image composition unit 77 generates a composite image based on preset composition conditions. In the memory 76, setting information 79 including a synthesis condition is recorded. In addition, the image composition unit 77 has a function of changing the display mode of the composite image once generated according to the displacement of the treatment tool 22 in the endoscopic image.

  As shown in FIG. 8, the synthesis condition includes, for example, display mode setting and trocar left / right setting. The display mode setting is a setting related to the display mode of the trocar image in the composite image. The display mode setting includes, for example, an initial setting of the insertion position and a change condition.

  In this example, the composite image is a trocar image inserted into the endoscopic image by the PinP method using the endoscopic image as a parent image and the trocar image as a child image (see FIG. 13). The initial setting of the insertion position is an initial setting relating to the insertion position indicating where the trocar image is to be inserted in the endoscopic image. In this example, the “lower corner of the screen” is set for both the first and second input I / Fs 74A and 74B to which the trocar images are input, and in this initial setting, the lower corner of the composite image has four lower corners. The corner is the initial insertion position.

  The initial setting of the insertion position may not be the four corners of the composite image. However, since the main subject of the endoscopic image is often located at the center of the screen, the initial setting is preferably four corners that are less likely to overlap with the main subject.

  The change condition is a condition for changing the display mode of the trocar image. In this example, it is a condition for changing the insertion position of the trocar image in the composite image. The change condition includes, for example, “treatment instrument interlock”. “Treatment instrument interlocking” is to change the insertion position of the trocar image in the composite image in conjunction with the displacement of the treatment instrument in the endoscopic image.

  The trocar left and right setting is a setting for making the left and right of the first and second trocars 16 and 17 correspond to the left and right of the two trocar images input from the first and second input I / Fs 74A and 74B. . In this example, as shown in FIG. 3, the first trocar 16 is inserted into the right port hole 27 of the port hole 26 of the endoscope 11, and the second trocar 17 is inserted into the left port hole 27. . For this reason, the first trocar 16 is disposed on the right side and the second trocar 17 is disposed on the left side with respect to the endoscope 11.

  In the endoscopic image, the treatment instrument 22 inserted into the first trocar 16 located on the right side of the endoscope 11 is depicted on the right side, and the treatment instrument 22 inserted into the second trocar 17 located on the left side is depicted on the left side. It is drawn to. Therefore, if the left and right of the trocars 16 and 17 (left and right of the port hole 27) do not correspond to the left and right of the insertion position of the trocar image in the composite image, it is difficult to see the composite image. It is the trocar left / right setting that sets such a left / right correspondence.

  In this example, since the first trocar 16 located in the port hole 27 on the right side of the endoscope 11 is set to the first input I / F 74A, the first input I / F 74A is set to the right (R). The Since the second trocar 17 located in the left port hole 27 of the endoscope 11 is set to the second input I / F 74B, the second input I / F 74B is set to the left (L).

  In the trocar left / right setting, the port numbers (ports No. 1 and No. 2) are port identification information arbitrarily assigned to each of the plurality of port holes 27 provided in the body of the patient P. For example, in order to identify the positions of the two port holes 27 shown in FIG. 3, an arbitrary port number may be given to each port hole 27 for each hospital, and each port hole 27 may be referred to by the port number. The port number may be set arbitrarily by the user because the provision rule may differ for each hospital. In this example, the right port hole 27 is set as port No1, and the left port hole 27 is set as port No2.

  The setting of the synthesis condition is performed on the synthesis condition setting screen 81 shown in FIG. The synthesis condition setting screen 81 is an operation screen that is displayed on the monitor 19 and operated by the keyboard or mouse of the console 20. The composition condition setting screen 81 is provided with a display mode setting area 82 and a trocar left / right setting button 83.

  The display mode setting area 82 is provided with an input box 82A for initial setting of the insertion position and an input box 82B for setting change conditions. Each input box 82A, 82B displays a pull-down menu that displays a plurality of items (upper corner of the screen, lower corner of the screen, fixed initial position, interlocked treatment tools, etc.) when a click operation is performed with the mouse designated by the pointer 84. To do. An item selected from this menu is set in each input box 82A, 82B.

  When the trocar left / right setting button 83 is operated, a trocar setting screen 86 shown in FIG. 10 is displayed. The trocar setting screen 86 has a first area 86A for displaying the endoscopic image 91 and a second area for displaying the first and second trocar images 92 and 93 input from the input I / Fs 74A and 74B. A region 86B is provided. In the following, as with the first and second trocars 16 and 17, the first trocar image 92 and the second trocar image 93 are the first and second trocar images 93 when there is no need to distinguish them clearly. In some cases, the trocar images 92 and 93 are simply omitted.

  In the endoscopic image 91 in the first area 86A, an insertion frame 94 indicating the insertion position for inserting each trocar image 92, 93 is displayed. The insertion positions of the two insertion frames 94 are set according to the initial setting of the insertion positions. This example shows an example in which “screen lower corner” is set as the initial insertion position, and two insertion frames 94 are displayed at the screen lower corner of the endoscopic image 91.

  In each insertion frame 94, letters L and R indicating the left and right in the endoscopic image 91 and the port numbers (No 1 and No 2) set in the left and right port holes 27 in the hospital are displayed. . Further, in the first region 86A, a number guide diagram 96 showing the positions of the port holes 26 and 27 shown in FIG. 3 and the respective port numbers is displayed in the schematic diagram of the abdomen that is the operation target site. From the number guide diagram 96, the port numbers assigned to the port holes 26 and 27 and the positional relationship between the two port holes 27 into which the trocars 16 and 17 are inserted can be confirmed. As shown in this example, the left and right (L, R) positions in the two insertion frames 94 correspond to the display of the port numbers (No2, No1).

  In accordance with the display in the insertion frame 94 and the guidance display such as the number guide diagram 96, the left and right trocars are set. For example, in this example, as shown in FIG. 3, the first trocar 16 is inserted into the right port hole 27 (No 1) as viewed from the medical staff ST facing the front of the patient P. Since the first trocar 16 is connected to the first input I / F 74A, the insertion position of the first trocar image 92 input from the first input I / F 74A is set on the right side of the endoscopic image 91. Is done.

  Specifically, for example, on the trocar setting screen 86, the first trocar image 92 input from the first input I / F 74A is designated by the pointer 84, and the first trocar image 92 is moved to the right side by a drag operation. Move to the insertion frame 94. By this operation, the first input I / F 74A and the right insertion frame 94 are associated with each other, and the insertion position of the first trocar image 92 is set.

  On the other hand, as shown in FIG. 3, the second trocar 17 is inserted into the left port hole 27 (No. 2) as viewed from the medical staff ST. Since the second trocar 17 is connected to the second input I / F 74B, the insertion position of the second trocar image 93 input from the second input I / F 74B is set on the left side of the endoscopic image 91. Is done. The second trocar image 93 is moved to the left insertion frame 94 by the drag operation of the pointer 84 described above. By this operation, the second input I / F 74B and the left insertion frame 94 are associated with each other, and the insertion position of the second trocar image 93 is set. The contents of such trocar left / right settings are recorded in setting information 79 shown in FIG.

  As shown in FIG. 11, the field of view of the endoscope 11 is set at a position where the surgical field can be looked down around a target site 97 (shown by hatching) that is a treatment target such as a lesion. In the endoscopic image 91, each treatment tool 22 that protrudes from the left and right trocars 16 and 17 and accesses the target portion 97 at the center of the screen is located on both sides of the screen.

  On the other hand, the field of view 98 of each trocar 16, 17 is set so that the distal end portion 22A of the treatment instrument 22 protruding from the distal end of each trocar 16, 17 is included in the imaging range of each trocar image 92, 93. The This is because the main purpose of capturing the trocar image is to secure the visual field around the distal end portion 22A of the treatment instrument 22 that accesses the target portion 97 and its periphery.

  The image composition unit 77 generates a composite image 99 according to the procedure shown in FIG. At the time of image composition, the image composition unit 77 reads the initial setting contents of the insertion positions of the trocar images 92 and 93 and the contents of the trocar left and right settings from the setting information 79, and determines the display mode of the trocar images 92 and 93. To do. In this example (example shown in FIG. 8), the image composition unit 77 displays the insertion position of the first trocar image 92 input from the first input I / F 74A according to the read setting content, the screen of the endoscopic image 91. The right side of the lower corner is determined, and the insertion position of the second trocar image 93 input from the second input I / F 74B is determined to be the left side of the lower corner of the screen.

  The image composition unit 77, in accordance with the determined display mode, the endoscope image 91 input from the third input I / F 74C and the trocar images 92, 93 input from the first and second input I / Fs 74A, 74B. And a composite image 99 is generated. The image composition unit 77 outputs the generated composite image 99 to the video signal processing unit 73.

  The video signal processing unit 73 converts the image signal of the synthesized image obtained by the image synthesis into a video signal to be output to the monitor 19 such as a composite signal or a component signal. Such image synthesis and video signal processing are sequentially performed in synchronization with the frame rates of the camera units 28 and 37.

  FIG. 13 is an example of a composite image display screen 101 that displays the composite image 99. In addition to outputting the composite image 99 to the monitor 19, the processor 18 can output the composite image 99 to the memory for storage. Further, on the composite image display screen 101, in addition to the output of the composite image 99, enlarged images of the trocar images 92 and 93 may be displayed. Further, a mode may be provided in which the endoscope image 91 and the trocar images 92 and 93 are independently output to the monitor 19 without outputting the composite image 99.

  As shown in FIGS. 14 to 17, the image composition unit 77 has a function of changing the display mode of the trocar images 92 and 93 in conjunction with the displacement of the treatment instrument 22 in the endoscopic image 91. ing. As shown in FIGS. 14 to 17, in the endoscopic image 91, when two treatment tools 22 are depicted on the left and right of the screen, the image composition unit 77 determines the left and right of the treatment tool 22. The displacement of each treatment tool 22 is detected. Here, in the endoscopic image 91 and the trocar images 92 and 93 of FIGS. 14 to 17, the symbols (R) and (L) that are additionally given to the symbols of the treatment tools 22 are the numbers of the treatment tools 22. In the screen of the endoscopic image 91, it is a subdivision code that represents the right and left relative rendering positions, which are depicted as right (R) or left (L). In the following, left and right subdivision codes are attached to the treatment instrument 22 and the distal end portion 22A as necessary.

  With regard to the function of changing the display mode of the image composition unit 77, for example, as shown in FIG. 14, in the endoscopic image 91, the right treatment tool 22 (R) is displayed on the screen as indicated by an arrow pointing downward on the screen. Consider a case where the center part on the right side moves to the lower right corner of the screen. Since the lower right corner of the screen, which is the movement destination of the treatment tool 22, is set to the initial insertion position of the first trocar image 92, the moved treatment tool 22 (R) overlaps the first trocar image 92, Hidden in the shadow.

  In such a case, as shown in FIG. 15, the image composition unit 77 has the first trocar image 92 at a position where the treatment instrument 22 (R) and the first trocar image 92 do not overlap in the endoscopic image 91. To move. When the left treatment tool 22 (L) moves in the endoscopic image 91, the image composition unit 77 moves the second trocar image 93 as in the right case. The image composition unit 77 changes the display mode of the trocar images 92 and 93 in conjunction with the displacement of each of the treatment instruments 22 (R) and (L). Such a display mode change is performed based on the displacement information output by the displacement detection unit 78. Note that such a display mode change function can be turned on / off by setting, and is effective when the change condition is set to interlock with the treatment instrument as shown in the setting information 79 shown in FIGS. 7 and 8. .

  As shown in FIG. 7, the endoscopic image 91 input from the third input I / F 74 </ b> C is also input to the displacement detection unit 78 in addition to the image composition unit 77. The displacement detector 78 detects the displacement of the treatment instrument 22 by analyzing the endoscope image 91.

  As shown in FIG. 16, specifically, the displacement detection unit 78 first performs an extraction process for extracting the treatment tool 22 from the endoscopic image 91. The extraction of the treatment tool 22 is performed by, for example, a known pattern matching technique. Extraction by pattern matching is performed by collating the feature amount representing the shape of the treatment instrument 22 set in advance with the feature amount of each part in the endoscopic image 91, and a feature set in the endoscope image 91 in advance. This is performed by determining whether or not there is a treatment tool 22 having an amount.

  In extracting the treatment tool 22, a method of identifying the treatment tool 22 by color may be used. Since the treatment tool 22 is made of a metal such as stainless steel or a resin, the color of the treatment tool 22 is clearly different from the color of the biological tissue based on red, which is the color of the blood vessel. In the endoscopic image 91, the treatment tool 22 can be identified based on such a color difference. Of course, a pattern matching method or a color identification method may be combined.

  Next, the displacement detector 78 determines the position of the extracted treatment instrument 22. The position information is output as coordinate information where the extracted treatment instrument 22 is located in the endoscopic image 91. When there are two extracted treatment instruments 22 as in this example, the displacement detection unit 78 is based on the coordinate information, and the extracted right (R) and left (L) of each treatment instrument 22 is relative. Determine the position.

  The displacement detection unit 78 determines the position of the treatment tool 22 for each frame of the endoscopic image 91 that is output from the camera unit 28 at a predetermined frame rate and sequentially input through the processor 18, and the treatment tool 22 is between frames. When the position changes, the displacement of the treatment tool 22 is detected. When the displacement detector 78 detects the displacement of the treatment instrument 22, the displacement detector 78 outputs displacement information to the image composition unit 77. The displacement information includes position information of the treatment instrument 22 after the displacement. Here, the image composition unit 77 corresponds to a displacement information acquisition unit. Such displacement detection of the treatment instrument 22 is performed individually for each of the extracted left and right treatment instruments 22 (R) and (L).

  For example, as shown in the endoscopic image 91 shown in FIG. 16, when the position of the right treatment instrument 22 (R) is moved from the position indicated by the solid line before displacement to the position indicated by the wavy line at the bottom of the screen. The displacement detector 78 outputs displacement information including position information of the treatment instrument 22 (R) indicated by the wavy line after displacement. Since this displacement information is related to the right treatment instrument 22 (R), the displacement information includes right and left identification information indicating that the information is information on the right treatment instrument 22 (R). Such displacement information is generated for each treatment instrument 22 (R), (L), and is output from the displacement detection unit 78 to the image composition unit 77.

  When the treatment instrument interlock is set as the change condition of the setting information 79, the image composition unit 77 changes the display mode of the composite image 99 based on the input displacement information. At the time of this change, the image composition unit 77 first executes a displacement information associating process for associating the input displacement information with the trocar images 92 and 93 whose display mode is to be changed based on the displacement information. To do.

  In the displacement information association processing, the image composition unit 77 determines whether the input displacement information is the left or right treatment tool 22 (R) in the endoscopic image 91 based on the left / right identification information included in the displacement information. ), (L). Then, the image composition unit 77 associates the input displacement information with each of the trocar images 92 and 93 based on the left / right discrimination result of the displacement information.

  As shown in this example, when the right treatment instrument 22 (R) moves in the endoscopic image 91, the displacement information output by the displacement detector 78 includes right (R) as shown in FIG. ) Representing left and right identification information. On the other hand, referring to the trocar left / right setting of the setting information 79, the initial insertion position in the endoscopic image 91 is set to the right (R) because the first trocar image 92 input from the first input I / F 74A. It turns out that it is. Therefore, the image composition unit 77 associates the input displacement information with the first trocar image 92. Accordingly, the display mode change target based on the input displacement information is specified as the first trocar image 92.

  Similarly, when the input displacement information includes left / right identification information representing the left (L), the image composition unit 77 associates the displacement information with the second trocar image 93. The image composition unit 77 changes the display mode of the trocar images 92 and 93 based on the left and right displacement information after performing the left-right association.

  In FIG. 16, after the displacement information association processing, the image composition unit 77 determines whether or not it is necessary to change the display mode based on the displacement information. The reason for determining whether or not the treatment tool 22 is displaced is that the position of the treatment tool 22 after the displacement specified by the displacement information and the insertion position of the trocar images 92 and 93 in the composite image 99 do not overlap. This is because there is no need to change the display mode.

  In the necessity determination, the image composition unit 77 determines whether or not the position of the treatment instrument 22 after the displacement specified by the displacement information and the insertion position of any of the trocar images 92 and 93 in the composite image 99 overlap. To do. In this example, since the position after displacement of the right treatment tool 22 (R) overlaps with the right trocar image 92, the image composition unit 77 determines that the display mode needs to be changed. In that case, as shown in the composite image 99, the insertion position of the trocar image 92 is moved to the upper right so that the trocar image 92 does not overlap the treatment instrument 22 (R).

  On the other hand, when the position of the treatment instrument 22 after the displacement specified by the displacement information and the insertion position of the trocar images 92 and 93 in the composite image 99 do not overlap, the image composition unit 77 changes the display mode. Judge as unnecessary. In this case, the image composition unit 77 does not move the trocar images 92 and 93 and does not change the display mode of the composite image 99. In this case, the composite image 99 is generated according to the display mode determined by the initial setting.

  The effect | action by the structure demonstrated above is demonstrated referring the flowchart shown in FIG. When performing laparoscopic surgery, first, for example, three port holes 26 and 27 are opened in the abdomen of the patient P as shown in FIG.

  A normal trocar 21 having no camera function is inserted into the central port hole 26. First and second trocars 16 and 17 having a camera function are inserted into the two port holes 27 on both sides of the port hole 26. In this example, the first trocar 16 is inserted into the right port hole 27 and the second trocar 17 is inserted into the left port hole 27 when viewed from the medical staff ST facing the abdomen of the patient P.

  Thereafter, the endoscope 11 is connected to the third input I / F 74C of the processor 18. The first trocar 16 is connected to the first input I / F 74A of the processor 18, and the second trocar 17 is connected to the second input I / F 74B.

  After the connection is completed, the processor 18 is activated. As a result, driving of the camera unit 28 of the endoscope 11 and the camera units 37 of the first and second trocars 16 and 17 is started, and photographing is started. The camera unit 28 outputs an image signal at a predetermined frame rate, and the image signal processed by the AFE circuit 68 is input to the processor 18. The camera unit 37 also outputs an image signal at a predetermined frame rate, and the image signal that has been subjected to signal processing by the AFE circuit in the connector unit 48 is input to the processor 18.

  In the processor 18, the image signal of the endoscopic image 91 input from the third input I / F 74 </ b> C is input to the image processing unit 72 and subjected to image processing. Similarly, the image signals of the first and second trocar images 92 and 93 input from the first and second input I / Fs 74A and 74B are input to the image processing unit 72 and subjected to image processing. Thereby, the acquisition of the endoscope image 91 and the trocar images 92 and 93 is started in the image processing unit 72 of the processor 18 (S101, S102).

  When the endoscope image 91 and the trocar images 92 and 93 are input, the image composition unit 77 starts an image composition process for generating a composite image 99 (S103). At this stage, there is a case where the user does not set the synthesis condition, so the condition preset at the time of manufacture is used as the synthesis condition, or the condition set at the previous activation is inherited. The composite image 99 generated by the image composition unit 77 is output to the monitor 19 via the video signal processing unit 73.

  The medical staff member ST inserts the endoscope 11 and the trocars 16 and 17 into the abdominal cavity while confirming the composite image 99 displayed on the monitor 19. First, the endoscope 11 is inserted into the abdominal cavity using the trocar 21 as an insertion port. Then, the treatment instrument 22 is inserted into the abdominal cavity using the trocars 16 and 17 as insertion ports.

  Thereafter, in the processor 18, the synthesis condition setting screen 81 shown in FIG. In the display mode setting, for example, “lower corner of the screen” is set as the initial insertion position, and “treatment instrument interlocking” is selected as the change condition.

  Further, the trocar left / right setting is performed through the trocar setting screen 86. In this example, since the first trocar 16 inserted into the right port hole 27 (port No. 1) is connected to the first input I / F 74A, the first trocar image 92 is set on the right side. On the other hand, since the second trocar 17 inserted into the left port hole 27 (port No. 2) is connected to the second input I / F 74B, the second trocar image 93 is set on the left side.

  When the composition condition is set, the image composition unit 77 performs image composition according to the set composition condition. The endoscopic image 91 is also input to the displacement detection unit 78 in addition to the image composition unit 77. When the endoscope image 91 is input, the displacement detector 78 starts detecting the displacement of the treatment instrument 22 based on the image analysis of the endoscope image 91 (S104). When the displacement detection unit 78 detects the displacement of the treatment tool 22, the displacement detection unit 78 outputs displacement information including the position information of the treatment tool 22 after the displacement to the image composition unit 77 (Y in S105).

  When two treatment tools 22 are depicted in the endoscopic image 91, the displacement detection unit 78 detects the right (R) of each treatment tool 22 extracted from the endoscopic image 91 based on the coordinate information. ), The relative position of the left (L) is determined. Then, the displacement detector 78 outputs displacement information for each extracted treatment instrument 22. As shown in FIG. 17, the displacement information includes right (R) and left (L) left and right identification information.

  After associating the input displacement information as shown in FIG. 17, the image composition unit 77 determines whether or not the display mode needs to be changed based on the displacement information (S106). Based on the positional information of the treatment tool 22 after displacement included in the displacement information, the image composition unit 77 includes the position of the treatment tool 22 after displacement and the initial positions of the trocar images 92 and 93 in the endoscopic image 91. It is determined whether or not the insertion position overlaps.

  As shown in FIG. 14, in the endoscopic image 91, when the position of the treatment instrument 22 (R) after displacement overlaps the initial insertion position of the first trocar image 92, the image composition unit 77 displays the display mode. It is determined that a change is necessary (Y in S106).

  In this case, the image composition unit 77 changes the display mode of the composite image 99 by moving the insertion position of the first trocar image 92 to a position that does not overlap the first trocar image 92, as shown in FIG. To do.

  On the other hand, even if the treatment instrument 22 is displaced, if it does not overlap any of the initial insertion positions of the trocar images 91 and 92, it is determined that it is not necessary to change the display mode (N in S106). The image composition unit 77 repeats such processing until the display of the composite image 99 is completed (S108).

  The medical staff ST performs the procedure by operating the treatment tool 22 while observing the abdominal cavity through the composite image 99 displayed on the monitor 19. The medical staff ST can secure an overall visual field centering on the target region 97 by the endoscopic image 91 in the composite image 99.

  In addition, the trocar images 92 and 93 in the composite image 99 can secure a visual field centering on the distal end portion 22A of the treatment instrument 22. As a result, it is possible to supplement the field of view of the part that becomes a blind spot only by the endoscopic image 91 with the trocar images 92 and 93. In particular, it is possible to accurately confirm the state of the distal end portion 22A accessing the target site 97, such as the position and posture of the distal end portion 22A of the treatment instrument 22, from the trocar images 92 and 93. This makes it possible to perform an appropriate procedure.

  The composite image 99 allows both the endoscopic image 91 and the trocar images 92 and 93 to be confirmed on one monitor 19 without switching the screen. In addition, when the composite image 99 is displayed, the movement of the line of sight can be reduced as compared to the case where the endoscopic image 91 and the trocar images 92 and 93 are displayed on separate monitors without being combined. For this reason, visibility improves compared with the case where the synthesized image 99 is not displayed.

  Further, in the processor 18, when the treatment tool 22 is displaced in the endoscopic image 91 and the displaced treatment tool 22 overlaps the initial insertion position of any of the trocar images 92 and 93, The display mode is changed by moving the insertion position of at least one of the trocar images 92 and 93 to a position that does not overlap. Therefore, in the synthesized image 99, the position of the distal end portion 22A of the treatment instrument 22 does not become a blind spot.

  As described above, according to the processor 18, when the composite image 99 is generated, the trocar image 92 in the composite image 99 is obtained in the sense that it is possible to secure as much as possible the field of view of the distal end portion 22A of the treatment instrument 22 that is important in the procedure. , 93 can be made appropriate.

[Second Embodiment]
The second embodiment shown in FIGS. 19 and 20 is an example in which the display mode of the trocar images 92 and 93 is changed in a mode different from the first embodiment. The second embodiment is the same as the first embodiment in that the display mode is changed in conjunction with the treatment tool.

  In the first embodiment, the trocar images 92 and 93 are retracted so that the treatment instrument 22 and the trocar images 92 and 93 do not overlap. In contrast, in the second embodiment, as shown in FIG. 19, the insertion position of the trocar images 91 and 92 in the endoscopic image 91 is set on the long axis of the treatment instrument 22 more than the distal end portion 22A. Determine the position on the proximal side. Then, as shown in FIG. 20, when the treatment instrument 22 is displaced in the endoscopic image 91, each trocar image 91 is maintained while maintaining the relative positional relationship between the treatment instrument 22 and each trocar image 91, 92. , 92 is made to follow the treatment instrument 22.

  In FIG. 19, in the endoscopic image 91, the left and right treatment tools 22 are displayed in a lateral orientation in which the respective distal end portions 22 </ b> A face the central target site 97 from both sides of the screen. In this state, since the proximal end side of the distal end portion 22A of each treatment instrument 22 is located at the left and right ends of the screen, the trocar images 91 and 92 are inserted at the positions.

  In FIG. 20, when the left and right treatment tools 22 change from the horizontal posture indicated by the dotted line to the posture where the distal end portion 22A faces upward as shown by the solid line, the proximal end of the distal end portion 22A of the treatment tool 22 The side will be located below the screen. Following this displacement, the insertion positions of the trocar images 92 and 93 also move downward in the screen.

  In the endoscopic image 91, when the insertion position of the trocar images 91 and 92 is set to the proximal end side with respect to the distal end portion 22A, the distal end portion 22A accessing the target site 97 becomes a shadow of the trocar images 91 and 92. There is no. The trocar images 91 and 92 follow the displacement of the treatment tool 22 while maintaining the relative positional relationship between the trocar images 91 and 92 and the treatment tool 22. Therefore, even if the treatment instrument 22 is displaced, the trocar images 91 and 92 are positioned on the proximal end side of the displaced distal end portion 22A, so that the distal end portion 22A is always displayed.

  As shown in FIG. 11, the camera portions 36 of the trocars 16 and 17 are located on the proximal end side of the distal end portion 22 </ b> A on the long axis of the treatment instrument 22. Since the actual positional relationship between the treatment tool 22 and the camera unit 36 and the positional relationship in the endoscopic image 91 of the display position of the treatment tool 22 and the insertion position of the trocar images 92 and 93 substantially match, There is also a merit that there is little discomfort. Moreover, since the trocar images 92 and 93 follow the same direction as the distal end portion 22A as compared with the first embodiment, the state of the distal end portion 22A in the endoscopic image 91 and the trocar images 92 and 93 Even when the state of the tip 22A is alternately checked, there is an advantage that the line of sight movement is small.

  When changing the display mode as in the second embodiment, for example, the following processing is performed in the image processing unit 72. First, as in the first embodiment, the position information of the treatment instrument 22 in the endoscopic image 91 is acquired by image analysis by the displacement detection unit 78. In the second embodiment, the displacement detection unit 78 acquires posture information of the treatment tool 22 in addition to the position information of the treatment tool 22.

  The posture information is information indicating in which direction the top and bottom, left and right of the endoscopic image 91 the distal end portion 22A of the treatment instrument 22 is facing. For example, the distal end portion 22A faces upward in the endoscopic image 91. Is the reference angle (0 °), the current angle is output so that the value increases as it rotates clockwise.

  In the example of FIG. 19, the distal end portion 22A of the right treatment instrument 22 faces the left direction of the endoscopic image 91, and thus a value of about 270 ° is output as the current angle. Since the distal end portion 22A of the left treatment tool 22 faces the right direction of the endoscopic image 91, a value of about 90 ° is output. Similarly, the treatment tool 22 indicated by the solid line in FIG. 20 is output at a value of about 180 ° because the distal end portion 22A of the treatment tool 22 faces upward in the endoscopic image 91. If such posture information is known, the orientation of the distal end portion 22A of the treatment instrument 22 can be known.

  The posture information is input to the image composition unit 77 as displacement information together with the position information. The image composition unit 77 grasps the position of the treatment tool 22 from the position information. Then, by grasping the orientation of the distal end portion 22A from the posture information, the position on the proximal end side of the distal end portion 22A that is the insertion position of the trocar images 92 and 93 is specified.

[Third Embodiment]
The third embodiment shown in FIG. 21 is an example in which port numbers (port Nos. 1 and 2), which are port identification information of a plurality of port holes 27, are displayed on the trocar images 92 and 93 in the composite image 99. As a result, it is possible to easily confirm the correspondence between the trocar images 92 and 93 and the port numbers simply by looking at the composite image 99.

  In the example shown in FIG. 21, the right trocar image 92 is an image acquired from the first trocar 16 inserted into the right port hole 27 to which the port No1 is assigned. Is displayed. Further, since the left trocar image 93 is an image acquired from the second trocar 17 inserted into the left port hole 27 to which the port No. 2 is assigned, the port No. 2 is displayed in the vicinity of the trocar image 93. In this example, in addition to the port number, symbols “R” and “L” indicating left and right are also displayed.

  The display position of the port number may not be in the vicinity of the trocar images 92 and 93 but in each of the trocar images 92 and 93. Further, the port identification information may be other than a number, and may be a character, a symbol, or a combination of a character, a symbol, and a number. Further, color information may be used as the port identification information. When color information is used, for example, a method of surrounding each trocar image 92, 93 with a colored frame is conceivable.

  Such display of port identification information is more effective as the number of first and second trocars 16 and 17 increases. The reason is as follows. In other words, when there are three or more trocars 16 and 17, there are also three or more trocar images. In that case, unlike the case of two trocar images, it is difficult to distinguish the trocar images only from the left and right. This is because if the port identification information is displayed, it is possible to identify which port hole 27 the image corresponds to even if there are three or more trocar images.

[Fourth Embodiment]
In the fourth embodiment shown in FIGS. 22 to 24, the insertion positions of the trocar images 92 and 93 are determined depending on whether or not the photographing optical axes LA of the camera units 36 of the plurality of trocars 16 and 17 intersect in the composite image 99. It is a form to change.

  As shown in FIG. 22, when two trocars 16, 17 are arranged on the left and right, depending on the position and posture of each trocar 16, 17 and treatment tool 22, photographing by the camera unit 36 of the right first trocar 16 is performed. The optical axis LA and the imaging optical axis LA of the camera unit 36 of the second trocar 17 on the left side may intersect. A point XP indicates an intersection of the two photographing optical axes LA.

  When the imaging optical axes LA intersect, the field of view 98R of the camera unit 36 of the first trocar 16 arranged in the right is directed to a relatively left region in the endoscopic image 91, and the camera of the second trocar 17 on the left side. The field of view 98L of the part 36 is intended for a relatively right region in the endoscopic image 91. On the other hand, the insertion positions of the trocar images 92 and 93 are in accordance with the arrangement of the left and right trocars 16 and 17, the insertion position of the trocar image 92 corresponding to the right first trocar 16 is on the right side, and the left trocar 17 is inserted. The insertion position of the trocar image 93 is on the left side.

  Therefore, when the photographing optical axes LA intersect, the left and right sides of the visual fields 98R and 98L of the camera unit 36 of the trocars 16 and 17 and the insertion positions of the trocar images 92 and 93 in the endoscopic image 91 The left and right are reversed.

  This becomes clearer when compared with the modes of FIGS. 11, 19, and 20 in which the photographing optical axes LA do not intersect. In FIGS. 11, 19, and 20, the field of view of the camera unit 36 of the first trocar 16 in the right arrangement is a relatively right area in the endoscopic image 91, and the second trocar 17 in the left arrangement is used. The field of view of the camera unit 36 is relatively in the left region. The insertion position of the first trocar image 92 corresponding to the first trocar 16 arranged on the right side is the right side, and the insertion position of the second trocar image 93 corresponding to the second trocar 17 arranged on the left side is the left side.

  Therefore, in the endoscopic image 91, the left and right sides of the visual fields 98R and 98L of the camera unit 36 of the trocars 16 and 17 and the right and left of the insertion positions of the trocar images 92 and 93 coincide.

  As shown in FIG. 22, if the insertion position of each trocar image 92, 93 and the left and right of the visual fields 98R, 98L are reversed due to the intersection of the photographing optical axes LA, it is difficult to intuitively grasp the relationship between the insertion position and the left and right of the visual field. Therefore, when the photographic optical axes LA intersect, the image composition unit 77 switches the left and right of the insertion positions of the trocar images 92 and 93 as shown in FIG. 23 to insert the insertion positions and the visual fields 98R and 98L. The left / right reverse state of is eliminated.

  When changing the display mode as in this example, as shown in FIG. 24, for example, the following processing is performed in the image processing unit 72. First, the displacement detector 78 detects the displacement of the treatment instrument by image analysis of the endoscopic image 91, as in the first and second embodiments. In the fourth embodiment, the displacement detection unit 78 further determines the intersecting state of the imaging optical axis LA by image analysis.

  The determination of the crossing state of the imaging optical axis LA is performed as follows, for example. As shown in FIG. 22, in the endoscopic image 91, the treatment instrument 22 is displayed, but the trocars 16 and 17 used as insertion ports of the treatment instrument 22 are often not displayed. Therefore, the displacement detection unit 78 estimates the position and direction of the imaging optical axis LA of the camera unit 36 of each trocar 16, 17 corresponding to the treatment tool 22 from the position and posture of the treatment tool 22.

  Since the treatment tool 22 protrudes from the distal end portion of each trocar 16, 17, if the position and posture of the treatment tool 22 are known, the approximate position and orientation of the camera unit 36 can be estimated based on that. For example, as shown in FIG. 22, if it is found that the right treatment tool 22 is located at the upper right side of the screen and the distal end portion 22A faces the lower left side of the screen, any area on the extension line in the major axis direction of the treatment tool 22 It can be roughly estimated whether or not the camera unit 36 is located. Since the direction in which the camera unit 36 is fixed at the distal ends of the trocars 16 and 17 is determined, if the position of the camera unit 36 is known, the direction can also be estimated.

  And if the position and direction of the camera part 36 are known, the position and direction of the imaging optical axis LA can be estimated. Such a process can be performed for the photographic optical axes LA of the left and right trocars 16 and 17, respectively, to determine a crossing state indicating the presence or absence of the intersection of the photographic optical axes LA.

  The displacement detection unit 78 outputs displacement information including the intersection state of the imaging optical axis LA to the image composition unit 77 in addition to the position information and posture information of the treatment instrument 22. The image composition unit 77 determines the crossing state of the photographing optical axis LA based on the displacement information. If the photographing optical axis LA intersects as shown in FIG. 22, the trocar images 92 and 93 are inserted. Swap left and right for position.

  As a result, as shown in FIG. 23, the insertion positions of the trocar images 92 and 93 and the left and right reversed states of the visual fields 98R and 98L are eliminated. It becomes easy to grasp. Thereby, the display mode of the trocar images 92 and 93 in the composite image 99 becomes appropriate.

[Fifth Embodiment]
In the fifth embodiment shown in FIGS. 25 and 26, when the camera unit 36 is rotated by the rotation of the trocar 16 around the long axis, the display posture of the trocar images 91 and 92 in the composite image 99 is changed to the endoscopic image 91. This is a mode for performing posture correction that matches the display posture.

  As shown in FIG. 25, since the camera unit 36 is fixed to the distal end portion of the trocar 16, when the trocar 16 rotates around the major axis AX of the trocar 16, the left and right sides of the camera unit 36 (T (Top), B (Bottom), L (Left), and R (Right)) also change.

  FIG. 25 is an example in which the camera unit 36 positioned above the screen of the endoscopic image 91 (field of view of the endoscope 11) rotates 180 ° around the major axis AX from the initial position. At the initial position, the top and bottom left and right (TA, BA, LA, RA) of the trocar image 92A taken by the camera unit 36 substantially coincides with the top and bottom left and right of the endoscopic image 91. When the trocar 16 is rotated 180 ° from this state, the top and bottom (TB, BB, LA, RB) of the trocar image 92B photographed by the camera unit 36 is rotated 180 ° with respect to the trocar image 92A.

  Since the top and right (TA, BA, LA, RA) of the trocar image 92A and the endoscopic image 91 are substantially the same, if the trocar image 92B is inserted into the endoscopic image 91 in the display posture as it is, the endoscope Since the top and bottom of the image 91 and the top and bottom of the trocar image 92B do not match, it is difficult to intuitively grasp the relationship between the top and bottom of the image 91B.

  Therefore, the image composition unit 77 rotates the display posture of the input trocar image 92B by 180 ° and performs posture correction to match the display posture of the endoscopic image 91.

  In this case, in order to acquire the posture information of the trocar 16, for example, a position measurement system 106 is provided. The position measurement system 106 is a known three-dimensional position measurement system using a magnetic sensor, for example. The position measurement system 106 includes a transmitter that generates a magnetic field and a receiver that receives a change in the magnetic field. The transmitter and receiver are each composed of three orthogonal coils. An electromotive force is generated in the receiver by exciting the transmitter, and the three-dimensional position information is measured by measuring the electromotive force.

  The position measurement system 106 constitutes a displacement detector that outputs trocar posture information representing how much the trocar 16 has rotated from the initial position as displacement information representing the displacement of the trocar 16. The initial position of the trocar 16 is set by the following procedure, for example. First, by manually rotating the trocar 16, the camera unit 36 of the trocar 16 is adjusted to a posture for outputting a trocar image 92 </ b> A that substantially coincides with the left and right sides of the endoscope image 91. In this state, an operation unit (such as a setting button) of the position measurement system 106 is operated to store in the position measurement system 106 that the current state is the initial position. Thereafter, the position measurement system 106 outputs the rotation angle based on the initial position as trocar posture information.

  FIG. 26 shows an example in which the trocar 16 is rotated 90 ° from the initial position. Posture correction is performed on the trocar image 92C photographed by the camera unit 36 in a state where the trocar 16 is rotated by 90 ° from the initial position. The top and bottom (TC, BC, LC, RC) of the trocar image 92C is shifted by 90 ° with respect to the trocar image 92A taken at the initial position.

  The image composition unit 77 rotates the display posture of the trocar image 92C by 90 ° based on the trocar posture information from the position measurement system 106, thereby matching the display posture of the trocar image 92C with the display posture of the endoscope image 91. To correct the posture.

  Note that the trocar image 92C has, for example, a horizontally long aspect ratio such as 4: 3 or 16: 9 in the normal posture, and thus becomes vertically long when rotated 90 °. Therefore, the image composition unit 77 rotates the display orientation of the trocar image 92C by 90 °, and then corrects the aspect ratio to landscape by trimming processing. Since the trimmed image is blank on both sides, enlargement processing is performed by electronic zoom. After such correction, the trocar image 92C and the endoscope image 91 are combined.

  Even when the trocar 16 is rotated and displaced in this way, the display postures of the endoscopic image 91, the trocar image 92B (FIG. 25), and the trocar image 92C are matched by changing the display mode by posture correction with respect to the trocar image. . For this reason, it becomes easy to intuitively grasp the top-to-bottom relationship between the endoscope image 91 and the trocar image. Thereby, the display mode of the trocar image in the composite image 99 becomes appropriate.

  In this example, a three-dimensional position measurement system using a magnetic sensor has been described as an example of the position measurement system 106 that functions as a trocar displacement detection unit. The position measurement system 106 only needs to be capable of detecting the rotation angle of the trocar, and may be, for example, a three-dimensional position measurement system using an ultrasonic wave or an acceleration sensor.

  In addition, an operating field camera capable of photographing the hand of the medical staff ST, the treatment tool 22, and the trocars 16 and 17 from the overhead position is provided on the ceiling of the operating room, and the trocars 16 and 17 rotation angles may be detected. In this case, the surgical field camera functions as a displacement detector that detects the rotation angle of the trocar.

  In each of the above-described embodiments, the displacement detection unit 78 that detects the displacement of the treatment tool 22 by image analysis has been described as an example of the displacement detection unit that detects the displacement of the treatment tool 22, but instead of the displacement detection unit 78 by image analysis. In addition, the position measurement system 106 shown in the fifth embodiment may be used as a displacement detection unit that detects the displacement of the treatment instrument 22.

  When the trocar is reflected in the endoscopic image, the rotation angle of the trocar may be detected by image analysis using the displacement detection unit 78 based on image analysis. By using the displacement detection unit 78 based on image analysis, a simple configuration can be achieved as compared with the case of using a three-dimensional position measurement system.

[Sixth Embodiment]
In the sixth embodiment shown in FIGS. 27 to 32, the display mode of the trocar images 92 and 93 in the composite image 99 is changed depending on whether the treatment instrument 22 is in the field of view of the endoscopic image 91 or out of the field of view. The aspect to change is shown.

Example 1
In the first embodiment shown in FIGS. 27 to 29, first, as shown in FIG. 27, the treatment instrument 22 is displayed on the endoscopic image 91 and the trocar images 92 and 93 are displayed on the composite image 99. Think. In this state, as shown in FIG. 28, when the treatment instrument 22 (R) disappears from the endoscopic image 91 by moving the right treatment instrument 22 (R) out of the field of view of the endoscopic image 91. In addition, the trocar image 92 corresponding to the treatment tool 22 (R) outside the visual field is not displayed.

  The display mode change in this case is executed according to the procedure shown in the flowchart of FIG. In FIG. 29, the displacement detector 78 monitors whether or not the treatment instrument 22 has moved from the inside of the field of view of the endoscopic image 91 to the outside of the field of view (S201). When the treatment instrument 22 moves outside the field of view, the displacement detection unit 78 outputs displacement information including information indicating that to the image composition unit 77. The image composition unit 77 associates the input displacement information with the trocar image, and specifies the trocar image corresponding to the treatment instrument 22 moved out of the field of view. Then, the identified trocar image is hidden in the composite image 99 (S202). In the example of FIG. 28, the right treatment tool 22 (R) has moved out of the field of view of the endoscopic image 91, so the image composition unit 77 hides the first trocar image 92 on the right side.

  Further, the displacement detection unit 78 monitors whether or not the treatment instrument 22 that has moved out of the visual field has entered the visual field of the endoscopic image 91 again (S203). When entering again within the field of view, displacement information to that effect is output to the image composition unit 77. The image composition unit 77 redisplays the trocar image corresponding to the treatment instrument that has entered in the composite image 99 (S204). Such a process is repeated until the display is completed (S205).

  By performing the processing of the first embodiment, the display of the trocar images 92 and 93 in the composite image 99 can be kept to the minimum necessary. In the case of the composite image 99 by the PinP method, the insertion position of the trocar images 92 and 93 is a blind spot for the field of view of the endoscopic image 91. From the viewpoint of reducing the blind spot of the endoscopic image 91, the endoscope It is preferable that there are no trocar images 92 and 93 that overlap a part of the field of view of the mirror image 91. Further, when the treatment tool 22 is not displayed in the endoscopic image 91, there is little need to display the trocar images 92 and 93 that display the periphery of the distal end portion 22A of the treatment tool 22. Therefore, by performing the processing of the first embodiment, the display of the trocar images 92 and 93 can be minimized, and the blind spot of the endoscopic image can be reduced.

(Example 2)
As in the second embodiment shown in FIGS. 30 and 31, the distal end portions of the trocars 16 and 17 may be displayed in the endoscopic image 91 in addition to the treatment tool 22. In such a case, when the treatment tool 22 protrudes from the trocars 16 and 17, it is determined that the treatment tool 22 is within the field of view of the endoscopic image 91, and the treatment tool 22 is retracted into the trocars 16 and 17. May be determined to be out of the field of view. Even in the determination of the second embodiment, the display and non-display of the trocar images 92 and 93 can be switched as in the first embodiment.

  As shown in FIG. 30, when the right treatment instrument 22 (R) protrudes from the first trocar 16, the displacement detector 78 detects that the treatment instrument 22 (R) is within the field of view of the endoscopic image 91. Judge that there is. In this case, the image composition unit 77 displays the first trocar image 92 in the composite image 99.

  From such a state, for example, when the treatment instrument 22 (R) is exchanged, the treatment instrument 22 (R) is pulled out from the first trocar 16. In this case, the treatment instrument 22 (R) protruding from the first trocar 16 as shown in FIG. 30 is retracted into the first trocar 16 as shown in FIG. 31. When the treatment instrument 22 (R) is retracted into the first trocar 16, the displacement detector 78 outputs displacement information indicating that the treatment tool 22 (R) has moved out of the field of view of the endoscopic image 91. The image composition unit 77 determines that the treatment tool 22 (R) has moved out of the field of view based on the input displacement information, and hides the first trocar image 92 corresponding to the treatment tool 22 (R). .

  When another treatment tool 22 (R) is inserted into the first trocar 16 and protrudes from the first trocar 16, the inserted treatment tool 22 (R) enters the endoscope image 91 again. In this case, the first trocar image 92 is displayed again.

(Example 3)
The third embodiment shown in FIG. 32 is based on the premise that both the treatment instrument 22 and the trocar are displayed in the endoscopic image 91 as in the second embodiment. In the second embodiment, the display and non-display of the trocar image 92 (or 93) are changed depending on whether the treatment instrument 22 is within the field of view of the endoscopic image 91 or outside the field of view. On the other hand, in the third embodiment, in addition to the treatment instrument 22, in consideration of whether the trocar 16 (or 17) is in the field of view or out of the field of view, the display of the trocar image 92 (or 93) is performed. Change the hide.

  Specifically, as shown in FIGS. 30 and 31, for example, when the trocars 16 and 17 are in the field of view of the endoscopic image 91, the treatment tool 22 is in the field of view as in the second embodiment. In some cases, the trocar images 92 and 93 corresponding to the treatment tool 22 in the visual field are displayed. For example, when the treatment tool 22 (R) moves out of the visual field, the treatment tool 22 (R outside the visual field). The trocar image 92 corresponding to) is not displayed.

  On the other hand, for example, when the first trocar 16 is moved out of the field of the endoscopic image 91 together with the right treatment instrument 22 (R) as shown in FIG. No. 77 continues the display without hiding the first trocar image 92 corresponding to the treatment tool 22 (R) outside the visual field. When the first trocar 16 corresponding to the treatment tool 22 (R) moves out of the field of view together with the treatment tool 22 (R), the treatment tool 22 (R) and the trocar 16 are removed from the endoscopic image 91. I can't figure out where it is. Knowing where the treatment tool 22 (R) and the trocar 16 are in the body cavity leads to an improvement in the safety of the procedure.

  Therefore, as shown in FIG. 32, when both the right treatment instrument 22 (R) and the right trocar 16 corresponding to the right treatment instrument 22 (R) move out of the field of view of the endoscopic image 91, the display of the first trocar image 92 is displayed. continue. Accordingly, it is possible to grasp the approximate position of the treatment instrument 22 (R) and the trocar 16 in the body cavity through the field of view of the first trocar image 92, and the safety of the procedure is improved. FIG. 32 illustrates the right treatment tool 22 (R), but the same applies when both the left treatment tool 22 (L) and the corresponding left trocar 17 are moved out of the field of view. The display is continued without the second trocar image 93 being hidden.

[Seventh Embodiment]
The seventh embodiment shown in FIG. 33 to FIG. 37 is a form having a function of changing the display size and display magnification of the trocar image in the composite image 99.

  As shown in FIG. 33, the designation of the display size of the trocar image 92 is performed through the console 20, for example. The display size designation information is recorded in the setting information 79 through the control unit 71. The image composition unit 77 refers to the display size designation information of the setting information 79 and changes the display size of the trocar image 92 in the composite image 99. Thus, for example, as shown in the composite image 99 in FIG. 33, the display size is changed as in the trocar images 92S0 (normal size), 92S1 (large size), and 92S2 (small size). The display size may be specified in stages such as large, medium, and small, or may be specified steplessly.

  Since the display size of the trocar image can be changed, an appropriate display size can be selected as necessary. Therefore, the blind spot of the endoscopic image 91 generated by the trocar image can be minimized in the composite image 99. it can.

  As shown in FIG. 34, the display magnification of the trocar image 92 is also changed through the console 20, for example. To change the display magnification, without changing the display size of the trocar image 92, the display magnification is enlarged (zoomed in) as shown in the trocar image 92Z1, or the display magnification is reduced (zoomed out) as shown in the trocar image 92Z2. Is done.

  For example, when blood vessel resection or the like is performed using an ultrasonic coagulation / cutting device or the like as the treatment tool 22, extreme care is taken so as not to damage other portions than the treatment target portion. In such a case, if the display magnification of the trocar image 92 can be changed, it is possible to enlarge and observe the treatment target portion, so the display magnification changing function is very effective.

  Also, as shown in FIGS. 35 to 37, the display size of the trocar image 92 may be changed according to the approach state of the treatment instrument 22 and the trocar image 92 in the composite image 99. For example, as shown in FIG. 35, the treatment tool 22 is displayed above the trocar image 92 in the composite image 99. In this state, as shown in FIG. 36, when the treatment instrument 22 moves downward and approaches the insertion position of the trocar image 92, the display size of the trocar image 92 is reduced.

  As shown in FIG. 37, the image composition unit 77 first determines the approaching state between the treatment instrument 22 and the trocar image 92 in the composite image 99 based on the displacement information from the displacement detection unit 78. The displacement information includes coordinate information of the treatment instrument 22 in the endoscopic image 91. As described above, the image composition unit 77 knows the initial insertion position of the trocar image 92 and the subsequent movement position. Based on these pieces of information, the image composition unit 77 grasps the approach state such as the distance between the treatment tool 22 and the trocar image 92 and from which direction the treatment tool 22 is approaching the trocar image 92. The display size of the trocar image is changed according to the grasped approach state.

  If the display size of the trocar image 92 is changed in accordance with the approach state of the treatment tool 22 and the trocar image 92, the treatment tool 22 in the endoscopic image 91 can be prevented from entering the blind spot of the trocar image 92. it can.

  In the first embodiment, the trocar image 92 is moved according to the movement of the treatment tool 22 so that the treatment tool 22 and the trocar image 92 do not overlap. Such a function of moving the insertion position of the trocar image and a function of changing the display size may be combined. For example, a combination of moving the insertion position of the trocar image 92 after reducing the display size of the trocar image 92 to a predetermined size is conceivable.

  In each of the above-described embodiments, an example in which an image synthesis function of an endoscopic image and a trocar image is built in one processor 18 and the processor 18 functions as an image synthesis device is described. A form independent of the processor 18 may be adopted. In the case of the independent form, there is a case where there is a merit that the modification of the existing processor (incorporation of the image composition function, etc.) can be reduced.

  Further, the example in which the image processing of the endoscopic image and the trocar image is executed by one processor 18 has been described. For example, the endoscope processor that executes the image processing of the endoscopic image, and the image processing of the trocar image It may be divided into two trocar camera processors that execute In this case, an image synthesizing function may be incorporated into the endoscope processor or trocar camera processor, so that the processor incorporating the image synthesizing function may function as an image synthesizing apparatus. If the image synthesis function is built in the trocar camera processor, there is a case where it is possible to obtain a merit that the modification of the existing endoscope processor can be reduced as in the case where the above-described image synthesis apparatus is completely independent.

  In each of the embodiments described above, the laparoscopic system 10 used for laparoscopic surgery has been described as an example of an intracorporeal observation system to which the image processing apparatus of the present invention is applied. However, a thoracoscopic system used for thoracoscopic surgery may be used. . Further, the endoscope 11 may be a flexible endoscope instead of a rigid endoscope.

  In each of the above embodiments, the present invention has been described with reference to an example in which the composite image 99 is generated by the PinP technique in which the trocar images 92 and 93 are inserted into the endoscopic image 91. However, as shown in FIG. The present invention can also be applied to the case where the synthesized image 108 is generated by the PoutP method for synthesizing the trocar images 92 and 93 around the endoscopic image 91. In this case, since the trocar images 92 and 93 are located outside the endoscopic image 91, the trocar images 92 and 93 are moved in the endoscopic image 91 shown in the first and second embodiments. It is not possible to apply this mode. However, the left and right trocar images 92 and 93 corresponding to the crossing state of the photographing optical axis LA shown in the fourth embodiment and the posture correction of the trocar image according to the rotation of the trocar shown in the fifth embodiment are applied. It is possible.

  In each of the above-described embodiments, an example in which two trocar images are combined with an endoscopic image has been described. However, the number of trocar images to be combined may be one, or may be three or more. The number of trocar images can be changed as appropriate according to the number of trocars with a camera to be used.

  Further, the present invention is not limited to the above-described embodiments, and it goes without saying that the embodiments are appropriately modified within a range not departing from the present invention, such as a combination of the above-described embodiments. In addition to the image composition apparatus and method, the present invention extends to an image composition program that causes a computer to function as an image composition apparatus and a storage medium that stores the image composition program.

10 Laparoscopic system 16, 17 Trocar with camera 18 Processor 19 Monitor 20 Console 36 Camera unit 37 Camera unit 72 Image processing unit 74A, 74B, 74C Input I / F
77 Image composition unit 78 Displacement detection unit 91 Endoscopic images 92 and 93 Trocar image

The present invention is a patent application subject to the application of the 19th Japan Medical Research and Development Organization, “Medical-Engineering Collaboration Promotion Project Demonstration Project” commissioned research, Article 19 of the Industrial Technology Strengthening Act, The present invention relates to an image synthesizing apparatus and method for synthesizing endoscopic images.

Claims (17)

An endoscope image acquisition unit for acquiring an endoscopic image in a body cavity imaged by an endoscope;
A trocar image acquisition unit for acquiring a trocar image in the body cavity imaged by a trocar camera provided in the trocar for inserting a treatment instrument into the body cavity;
A displacement information acquisition unit for acquiring displacement information representing displacement of the treatment instrument and / or the trocar;
An image composition unit that synthesizes the endoscopic image and the trocar image to generate a composite image, and an image composition unit that changes a display mode of the trocar image in the composite image based on the displacement information; An image composition apparatus.   The image composition device according to claim 1, wherein the image composition unit changes the insertion position of the trocar image in the composite image as the display mode. There are multiple trocars,
The trocar image acquisition unit acquires a plurality of trocar images from the trocar camera provided in each of the plurality of trocars,
The image synthesizing apparatus according to claim 2, wherein the image synthesizing unit synthesizes one endoscopic image and a plurality of trocar images. The displacement information includes information indicating the presence or absence of crossing of the photographing optical axes of the plurality of trocar cameras in the endoscopic image,
The image synthesizing apparatus according to claim 3, wherein the image synthesizing unit changes an insertion position of the plurality of trocar images in the synthesized image according to presence / absence of crossing of the photographing optical axes.   The image composition unit inserts the trocar image into the endoscopic image by a picture-in-picture method using the endoscopic image as a parent image and the trocar image as a child image. The image synthesizing device according to item.   The image synthesizing apparatus according to claim 5, wherein when the treatment tool is displaced in the endoscopic image, the image synthesis unit moves the insertion position of the trocar image to a position that does not overlap the treatment tool. The image synthesizing unit determines the insertion position of the trocar image in the endoscopic image as a position closer to the proximal end than the distal end portion of the trocar or the treatment instrument on the long axis of the trocar or the treatment instrument. And
Furthermore, when the said trocar or the said treatment tool is displaced in the said endoscopic image, the image composition apparatus of Claim 5 which makes the said insertion position track the said displacement.   The said image synthetic | combination part has a function which hides the said trocar image in the said synthesized image, when the said treatment tool moves out of the visual field of the said endoscopic image. Image composition device.   In the case where the trocar and the treatment tool protruding from the trocar are displayed in the endoscopic image, the image composition unit is configured to display the treatment tool when the treatment tool is retracted into the trocar. The image synthesizing apparatus according to claim 8, wherein the image synthesizing apparatus has a function of determining that the trocar image has moved out of the field of view and not displaying the trocar image.   The said image synthetic | combination part has a function which continues a display, without making the said trocar image non-display, when the said trocar with the said treatment tool moves out of the said visual field. Image composition device.   The image composition device according to claim 5, wherein the image composition unit has a function of changing at least one of a display size and a display magnification of the trocar image.   The image synthesizing apparatus according to claim 11, wherein the image synthesizing unit has a function of changing a display size of the trocar image in accordance with an approaching state of the treatment tool and the trocar image in the synthesized image.   The image composition unit corrects the posture of the trocar image in the composite image to match the display posture of the endoscopic image when the trocar camera is rotated by the rotation of the trocar around the long axis. The image composition device according to claim 1, wherein   The image synthesizing apparatus according to claim 4, wherein an initial position of the insertion positions of the plurality of trocar images is one of four corners of the synthesized image.   15. The image synthesizing unit displays port identification information for identifying each port into which a plurality of trocars are respectively inserted in a patient's body on each trocar image in the synthesized image. The image composition device according to any one of the above.   The displacement detection part which analyzes the said endoscopic image, detects the displacement of the said trocar and / or a treatment tool, and outputs the said displacement information is given in any one of Claim 1 thru | or 15. Image synthesizer. An endoscopic image acquisition step of acquiring an endoscopic image in a body cavity imaged by an endoscope;
A trocar image acquisition step of acquiring a trocar image in the body cavity imaged by a trocar camera provided in a trocar for inserting a treatment instrument into the body cavity;
A displacement information obtaining step for obtaining displacement information representing displacement of the treatment instrument and / or the trocar;
An image composition unit that composes the endoscopic image and the trocar image to generate a composite image, and an image composition step for changing a display mode of the trocar image in the composite image based on the displacement information. An image composition method.

Images