JP4916114B2 - Endoscope device - Google Patents

Description

  The present invention relates to an endoscope apparatus, and more particularly to a technique for obtaining a three-dimensional relationship between an insertion tool or treatment tool of an endoscope apparatus and a subject and notifying a user.

  In an endoscopic device in which an endoscope image is displayed on a screen and a user performs a treatment using a treatment tool while watching the image, the image of the treatment tool operated with the subject displayed by the endoscope is displayed. Take action while watching.

19 and 20 show a state of treatment by an endoscope through which a treatment tool can be inserted and an endoscope image at that time.
FIG. 19 shows a turbine blade 301 that is a subject, a distal end portion 303 of an endoscope, and a treatment tool 304, and FIG. 20 is an image taken by an endoscope displayed on a display screen 300 of the endoscope apparatus. A video is shown.

  In this example, the crack 302 in the turbine blade 301 is marked with the treatment tool 303. However, since the three-dimensional positional relationship cannot be grasped even when the display screen 300 is viewed, the treatment tool 303 is attached. It is difficult to determine which part of the turbine blade 301 the treatment tool 303 will contact when extended. Therefore, in such a procedure, since it is difficult to grasp the three-dimensional positional relationship between the subject and the treatment tool only with the two-dimensional image of the endoscope, the part that depends on experience and intuition is large.

  Patent Document 1 proposed as one of the techniques for dealing with this point provides a mechanism for detecting a three-dimensional position using a magnetic sensor or the like for an insertion tool such as an endoscope or a treatment tool. When the insertion tool is inserted into the treatment target position, the magnetic sensor outputs the magnetic field generated by the magnetic field generator as a signal, and the position of each insertion tool is calculated from this signal, and the insertion tool is inserted into the endoscope image. The insertion position is synthesized and displayed.

  Patent Document 2 relates to a medical endoscope. Biological image information is stored in advance in the apparatus, and position information, distance information, and biological image information of a treatment instrument are measured by a sensor, A configuration is disclosed in which it is combined with the recorded biological image information and displayed.

By using an apparatus to which the methods disclosed in these patent documents and the like are applied, it is possible to grasp the three-dimensional positional relationship between the treatment object and the treatment tool and perform the treatment.
JP 2002-253480 A (see FIG. 1, paragraphs [0017] and [0020]) JP 2001-204738 A (see FIG. 4, paragraphs [0033] to [0039])

  However, depending on the subject, it is not always possible to provide a position detection mechanism such as a sensor in the insertion portion or the treatment instrument as in the configuration disclosed in Patent Document 1 described above. Further, the image information of the subject may not be acquired in advance as in the configuration disclosed in Patent Document 2. In particular, in an industrial endoscope apparatus that uses an aircraft engine or heat exchanger for inspection or the like, it is difficult to apply these methods because of the material and shape of the subject, inspection conditions, and the like.

  In this case, the treatment tool is operated only with a two-dimensional image photographed by the endoscope, but it is difficult to obtain perspective with only the two-dimensional image as shown in FIG. The distance of the tool is difficult to understand. Therefore, when inserting the treatment tool, it is not known which position on the subject the treatment tool will contact even if the photographed image is viewed. Therefore, the treatment tool can be appropriately brought close to the treatment target. It is difficult and takes time and effort.

  The present invention has been made in view of the above problems, and it is not necessary to provide a special position detection mechanism for the insertion tool, and it is not necessary to store image information of a subject in advance, It is an object of the present invention to provide an endoscope apparatus that can grasp a distance from a subject, and as a result, can easily bring a treatment tool closer to a treatment target and can perform an appropriate treatment.

In order to achieve the above object, an endoscope apparatus according to the present invention is an endoscope apparatus that performs a treatment with a treatment tool, an imaging unit that captures an endoscopic image of a subject, and the internal that is captured by the imaging unit. A three-dimensional information acquisition unit that performs three-dimensional measurement using an endoscopic image and acquires three-dimensional information of the subject; a contact position where the treatment tool is expected to contact the subject; the three-dimensional information; and And a contact position calculation unit that calculates based on the position information of the treatment instrument.

  According to the present invention, it is not necessary to provide a special position detection mechanism for the insertion tool, and it is not necessary to store the image information of the subject in advance, and the contact position and distance between the insertion tool and the subject can be determined at the time of treatment. I can grasp. For this reason, the user can easily bring the treatment tool close to the treatment target, and can perform appropriate treatment quickly, easily and accurately.

An embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating an external configuration of an endoscope apparatus 1 according to the present embodiment.
As shown in FIG. 1, an endoscope apparatus 1 includes a scope unit 4 in which an optical adapter including one capable of stereo measurement is detachable, a control unit 2 that houses the scope unit 4, and a system of the endoscope apparatus 1. A remote controller 6 that performs operations necessary to execute overall various operation controls, and a liquid crystal monitor (hereinafter referred to as an LCD) that displays endoscope images and operation control contents (such as processing menus and notifications to users). 3. A normal endoscopic video, or a face-mounted display (hereinafter referred to as FMD) 7 that can stereoscopically view the endoscopic video as a pseudo stereo video, and an FMD that outputs video data to the FMD 7 FIG. 2 is a circuit block diagram mainly illustrating a portion that performs control processing of the endoscope apparatus 1 of FIG. 1.

  As shown in FIG. 2, the endoscope unit 11 to which the scope unit 4 is connected is configured to be mounted in the control unit 2 shown in FIG. 1, for example. The endoscope unit 11 includes a light source device for obtaining illumination light necessary for photographing and an electric bending device for bending the endoscope insertion portion 4 freely and electrically. The endoscope unit 11 is connected to a camera control unit (hereinafter referred to as CCU) and a connection interface (hereinafter referred to as connection I / F) 29 in the processing control unit 20 provided in the control unit 2. Has been.

  An imaging signal input from a solid-state imaging device 43 (see FIG. 4) described later at the distal end of the endoscope insertion unit 4 is input to the CCU 12 via the endoscope unit 11. The CCU 12 converts the input imaging signal into a video signal such as an NTSC signal and outputs the video signal to the connection I / F 29 of the processing control unit 20 in the control unit 2.

In FIG. 2, the processing control unit 20 mounted in the control unit 2 includes a CPU 26, a ROM 27 and a RAM 28 that control operations for realizing various functions of the endoscope apparatus based on a main program, a connection interface ( Hereinafter, a connection interface 29 is described, a card interface (hereinafter referred to as a card interface) 22, a video signal processing circuit 21 and an audio signal processing circuit 23, and these are connected to the bus 24. It has become.

  The connection I / F 29 is a connection interface such as RS-232C that connects devices outside the processing control unit 20 such as the remote controller 6, the endoscope unit 11, and the CCU 12 to the internal bus 24 of the processing control unit 20.

  The remote controller 6 is used when the user inputs an operation instruction to the endoscope apparatus when operating the endoscope unit 11 or the CCU 12. The operation instruction input from the remote controller 6 is directly notified to the CPU 26 or the endoscope unit 11 or the CCU 12 through the connection I / F 29, and each unit that receives the operation performs an operation for realizing the operation instruction. Do.

The card I / F 22 is a compact flash (registered trademark) memory card 13 or PCMC.
This is a card slot to which an IC card such as the IA memory card 14 is detachably connected. When an IC card is inserted into the card I / F 22, the type of the IC card is checked. If the IC card is a memory card, the CPU 2 is connected via the card I / F 22.
6 is used to reproduce data such as stored control processing information and video information, to import data stored in the memory card into the control unit 2, or to control processing information and video information. Etc. can be written and recorded in the memory card.

  The video signal processing circuit 21 displays an operation menu generated by the processing of the endoscope video and the CPU 26 based on the video signal from the CCU 12 in order to display a composite image obtained by synthesizing the captured video from the CCU 12 and the graphic of the operation menu. The image is combined with the image, and further necessary processing for displaying on the screen of the LCD 3 is performed and the image is output to the LCD 3. Thereby, an image in which the endoscope image and the operation menu are combined is displayed on the LCD 3. Note that the video signal processing circuit 21 can simply perform processing for displaying an image of an endoscope video or an image such as an operation menu alone.

  The audio signal processing circuit 23 converts the sound collected by the microphone 15 into an audio signal to be recorded on a recording medium such as a memory card, or an audio signal obtained by reproducing the recording medium such as a memory card or the CPU 26. The audio signals generated by the above are input, and processing necessary for reproduction, such as amplification processing, is performed on these audio signals and output to the speaker 16. Thereby, the audio signal is reproduced by the speaker 16.

  The CPU 26 executes a program stored in the ROM 27 and controls various circuit units so as to perform processing according to the purpose, thereby controlling the operation of the entire system. The ROM 27 is a memory storing a basic program and firmware for controlling the endoscope apparatus 1, and the RAM 28 is a memory used as a work memory for the CPU 26.

FIG. 3 is a perspective view showing a main configuration of the endoscope apparatus with the scope unit 4 as the center. This figure shows a state in which the optical adapter is removed from the distal end portion of the insertion portion.
The scope portion 4 includes at least a flexible elongated insertion portion 4a that is inserted into the inspection target space, an intermediate connection portion 4b, and a universal cable 4c. Of these, the insertion portion 4a incorporates an observation optical system and illumination optical system for observation, and has a head portion 4a1 and an elongated flexible tube portion 4a3 arranged at the foremost position, and this head portion 4a1. And a flexible tube portion 4a3 are provided with a bending portion 4a2 that can be freely bent remotely by a user's operation instruction.

  The distal end portion of the intermediate connecting portion 4b is connected to the base end portion 49 of the flexible tube portion 4a3 of the insertion portion 4a. The intermediate connecting portion 4b is provided with a grip portion 4b1 that can be grasped by one user with one hand. ing. A connecting portion between the channel port portion 4b2 and the tip end portion 4c1 of the universal cable 4c is provided in parallel at the rear end portion of the grip portion 4b1. The channel port portion 4b2 is provided with a forceps port (proximal opening end) 48 of an internal channel 47 extending along the axial direction inside the insertion portion 4a. Further, the connecting portion of the universal cable 4c is disposed in a state where it is inclined obliquely with respect to the axial direction of the insertion portion 4a.

In FIG. 3, a power button 18 for instructing at least power-on / off of the endoscope apparatus 1 is provided to the remote controller 6 for a user to input various operation control instructions to the endoscope apparatus 1. And a joystick 19 for remotely inputting an operation instruction in the bending direction of the bending portion 4a2 of the scope portion 4 in the vertical and horizontal directions.

  Further, as shown in FIG. 3, a substantially semicircular protruding portion 41 having an observation optical system 106 is formed on the tip surface of the head portion 4a1. The distal end face 101 of the internal channel 33 is disposed on the distal end surface of the head portion 4a1 along the axial direction inside the insertion portion 4a. Further, illumination windows 104 of an illumination optical system are formed on both sides of the distal end opening 101.

  In addition, an optical adapter mounting portion 44 having a mount portion having an appropriate structure such as a bayonet type or a screw type is provided at the distal end portion of the head portion 4a1 of the endoscope apparatus 1, and the optical adapter mounting portion 44 is It is possible to attach an optical adapter to the distal end portion of the head portion 4a1 of the endoscope apparatus 1 by using it.

  There are a plurality of types of optical adapters, and they can be used depending on the application. For example, FIG. 3 shows that the direct-view binocular adapter 100a1 and the side-view binocular adapter (binocular observation optical system) 100a2 are attached in a detachable state. In the endoscope apparatus 1 according to the present embodiment, it is possible to select from among a plurality of types of optical adapters as necessary and attach them to the distal end portion of the head portion 4a1.

  In FIG. 3, the optical adapters 100a1 and 100a2 are respectively provided with an adapter observation optical system 107 that performs photographing, an adapter illumination window 105 that irradiates illumination light, and an adapter entrance 103 that allows a treatment tool to enter and exit. As shown in FIG. 3, each adapter observation optical system 107 of the direct-viewing binocular adapter 100a1 and the side-viewing binocular adapter 100a2 for stereo measurement is provided with two left and right observation windows (observation optical systems) 108L and 108R, respectively. .

  FIG. 4 shows a state in which the direct-viewing binocular adapter 100a1 for stereo measurement is attached to the head portion 4a1. The following description is basically the same when other optical adapters such as the optical adapter 100a2 are attached.

  When the optical adapter 100a1 is attached to the optical adapter attachment portion 44 of the head portion 4a1, the observation optical system 106 of the head portion 4a1 and the adapter observation optical system 107 of the optical adapter 100a1 are connected, and similarly, the two of the head portion 4a1 are connected. The illumination window 104 and the adapter illumination window 105 are connected to the front end opening 101 of the head portion 4a1 and the adapter opening 103. The observation optical system 106 of the projecting portion 41 is provided with an optical lens group 42 and a CCD 43 that is a solid-state imaging device that images an observation image formed by the optical lens group 42. Observation images from two observation windows 108L and 108R 107 are formed on one CCD 43 through two optical paths via the observation optical system 106 of the head portion 4a1.

The CCD 43 may be configured such that the two CCDs 43 are provided in the head portion 4a1 so as to correspond to the two observation windows 108L and 108R of the adapter observation optical system 107, respectively.
FIG. 5 is a configuration diagram of functional blocks realized by software of the endoscope apparatus 1.

  Each function shown in FIG. 5 is realized by the CPU 26 shown in FIG. 2 executing a program in a memory card mounted on the ROM 27, the RAM 28, or the card I / F 22 using the RAM 28 as a work memory. In this example, each functional block in the figure is realized by a software method, but dedicated hardware that implements the function of each functional block is installed in the endoscope apparatus 1 to implement the hardware. It is good also as a structure implement | achieved by.

  In FIG. 5, the system control unit 201 controls the entire endoscope apparatus 1, and by depositing data with other connected functional blocks, the other functional blocks and the endoscope apparatus 1. The hardware that constitutes the system is controlled. In this figure, the system control unit 201 includes a recording / playback instruction unit 202, an operation detection unit 203, a graphic display instruction unit 204, an image input unit 205, a stereo measurement unit 206, and a treatment instrument contact position calculation control unit 207. Each functional block is connected.

The recording / playback instruction unit 202 is connected to the card I / F 22, and the system control unit 20
1, data in a memory card attached to the card I / F 22 is read out and sent to the system control unit 201, or video signal data is sent to the card I / F 22 via the system control unit 201. Write to the memory card attached to the. Operation detection unit 20
3 is connected to a unit to which an operation instruction is given by a user such as the remote controller 6. When the user operates the remote controller 6 or the like, the operation instruction is accepted and the content is notified to the system control unit 201. When the notified instruction content is an image recording or reproduction command, the system control unit 201 notifies the recording / reproduction instruction unit 202 of the recording / reproduction instruction unit 202 and the card I / F 22 to the recording / reproduction instruction unit 202. Is instructed to read or write video data.

  The graphic display instruction unit 204 is responsible for generating image data of an image to be displayed graphically on the LCD 3. Based on an instruction from the system control unit 201, graphic display image data necessary for an endoscope operation such as an operation menu is obtained. The image is output to the LCD 3 and the FMD 7 via the video signal processing circuit 21. The image input unit 205 transfers the endoscope video input from the CCU 12 to the system control unit 201 via the video signal processing circuit 21. A video processing signal circuit 31 is connected to the graphic display instruction unit 204 and the image input unit 205. The stereo measurement instruction unit 206 performs stereo measurement on the image transferred from the image input unit 205 and notifies the system control unit 201 of the result. When the user gives an operation instruction using the remote controller 6, the system control unit 201 instructs the stereo measurement instruction unit 206 to execute stereo measurement, and causes the graphic display instruction unit 204 to display a graphic on the LCD 3 or the FMD 7. .

Next, stereo measurement performed by the stereo measurement instruction unit 206 will be described.
FIG. 6 is a diagram for explaining how to obtain the three-dimensional coordinates of measurement points by stereo measurement based on the principle of triangulation.

  From the binocular adapter 100a1 or the binocular adapter 100a2, the image information with parallax can be acquired through the two left and right adapter observation windows 108R and 108L. Based on this image information, the distance to the subject can be measured from the principle of triangulation.

In the figure, points corresponding to the optical centers on the left (adapter observation window 108L) and right (adapter observation window 108R) images are taken as origins OL and OR, respectively, and measurement points 52L, The coordinates of 52R are (X _L , Y _L ) and (X _R , Y _R ), respectively. Further, the optical center 51L of the adapter pig observation window 108L (left) is set as the origin O, a straight line passing through the origin O and the optical center 51R of the adapter pig observation window 108R (right) is the X axis, a direction parallel to the optical axis is the Z axis, In the space coordinates with the direction perpendicular to the axis and the Z axis as the Y axis, the three-dimensional coordinate of the measurement point 53 is (X, Y, Z).

When the user operates the remote controller 6 and designates the left measurement point 52L (coordinates (X _L , Y _L )) while viewing the display screen on the LCD 3 on which the captured image is displayed, the stereo measurement unit 206 is The coordinates (X _R , Y _R ) of the right measurement point 52R corresponding to this point are determined by an existing matching method. At this time, if the distance between the left and right optical centers is D and the focal length is F, the values D and F are already known, so that the three-dimensional coordinates (X, Y, Z) of the measurement point 53 are obtained by the triangulation method.
X = t × X _L
Y = t × Y _L
Z = t × F
Where t = D / (X _R −X _L + D)
It is obtained.

  In the matching method, when the measurement point 52R is designated from the image of the left observation optical system by the operation of the remote controller 6, it is similar to the vicinity of the designated measurement point 52R in the left image by the image recognition process. This is a method of searching for the shape from the image of the right observation optical system and obtaining the right measurement point 52L.

In the following description, 3 of the measurement points 53 obtained from the coordinates (X _L , Y _L ) of the left measurement point 52L and the coordinates (X _R , Y _R ) of the right measurement point 52R obtained by the processing by the matching method. The dimensional coordinate is expressed as P ((X _L , Y _L )) = (X, Y, Z).

  Thus, when the coordinates of the measurement point on the image are determined, the three-dimensional coordinates (X, Y, Z) of the measurement point 53 are obtained using the known parameter values D and F. Then, by obtaining the three-dimensional coordinates (X, Y, Z) of the plurality of measurement points 53, various distances such as the distance between the observation points, the distance between the line connecting the two points and one point, the area, the depth, the surface shape, etc. Measurement is possible.

  The treatment tool contact position calculation unit 207 calculates a position where the treatment tool and the subject contact from the treatment tool path and the three-dimensional information of the subject acquired by the stereo measurement unit 206, and based on the result, the system control unit 201 performs a graphic. The display instruction unit 213 displays a contact position where the treatment tool is expected to contact on the LCD 3. Details of this point will be described later.

Next, a treatment example using a treatment tool using the endoscope apparatus 1 of the present embodiment will be described.
In the endoscope apparatus 1, various optical adapters can be detachably attached to the head part 4 a 1 of the scope part 4 when necessary, but in this example, as shown in FIG. 4, the optical adapter of the head part 4 a 1 is attached. The treatment example by the treatment tool when the direct-viewing binocular adapter 100a1 is attached to the unit 44 is shown.

First, with the direct-viewing binocular adapter 100a1 attached to the head part 4a1, the insertion part 4a of the scope part is inserted into the examination target space of the subject.
FIG. 7 shows a state in which the insertion portion 4a is inserted up to a position near the turbine blade 301 to be inspected. The turbine blade 301 has cracks 302. In the state shown in FIG. 7, the cracks 302 are observed from the endoscopic image. In this example, an operation of marking the turbine blade 301 containing cracks 302 with a paint attached to the tip of the treatment tool will be described in order to facilitate identification at the time of disassembly and maintenance.

  FIG. 7 shows that the forceps 114 is inserted from the forceps port 48 of the channel port portion 4b2 for marking. The forceps 114 includes a forceps including paint at the tip of an elongated flexible forceps insertion portion 114a. A tip portion 114b is provided. Then, as shown in FIG. 8, the forceps tip 114b of the forceps 114 is protruded from the adapter opening 103, and the images from the observation windows 108L and 108R are displayed on the LCD 3 or the FMD 7, and the forceps 114 and the remote The joystick 19 of the controller 6 is operated to bring the forceps tip 114b into contact with the turbine blade 301 for marking.

  At this time, on the display screen of the LCD 3 or the FMD 7, a position where the forceps tip 114 b is expected to contact the turbine blade 301 when the forceps 114 are projected is displayed and notified to the user.

Information notification processing regarding the contact position between the forceps tip 114b and the turbine blade 301 will be described.
FIG. 9 is a flowchart showing processing performed by the system control unit 201 during this notification processing. This processing is realized by the CPU 26 shown in FIG. 2 executing a program on a memory such as the ROM 27 and the RAM 28 using the RAM 28 as a work memory.

  First, in step S01, the system control unit 201 causes the stereo measurement unit 206 to perform stereo measurement using an image captured by the endoscope received from the image input unit 206, and acquires three-dimensional information of the subject.

  Next, in step S02, the system control unit 201 causes the treatment instrument contact position calculation control unit 207 to calculate a contact position based on the three-dimensional information of the subject obtained in step S01 and the treatment instrument path.

In step S03, the system control unit 201 notifies the contact position calculated in step S02. In step S04, the system control unit 201 causes the calculated distance L ₁ between the contact position and the endoscope insertion portion from the coordinates of the contact position to the treatment instrument contact position calculation controller 207, Ru is notified of the result in step S05 . As next step S 06, Ru is detected the treatment instrument distal end position based on the object 3-dimensional information on the treatment instrument contact position calculation controller 207. As next step S 07 to its, to calculate the distance L ₂ between the contact position and the treatment instrument distal end position in the treatment tool contact position calculation control unit 207, to notify the result in step S 08.

Above manner the obtained contact position, the distance L ₁ of the contact position and the endoscope insertion portion, the distance L ₂ of the contact position between the treatment instrument distal end position, the system control unit 201, the graphic display instruction section 209 LCD 3 And instructing to display on the display screen of FMD7, and the contents thereof are notified to the user.

For this display method, for example, in the left and right endoscopic images, x marks are displayed at the positions of the points W _L (Q _Z ) and W _R (Q _Z ) on the endoscopic image corresponding to the contact position. The person in charge. Further, the distance L ₁ between the endoscope insertion portion and the contact position, and when the treatment tool is detected on the endoscope image, the distance L ₂ between the treatment tool and the contact position is displayed numerically.

The endoscope apparatus of the present embodiment has a function of a warning contact of the treatment instrument to the user, in step S09, the system control unit 201, a distance L ₂ to the contact position with the treatment tool predetermined If it is smaller (step S09, Yes), as a step S10, a message is displayed on the display screen of the LCD 3 or FMD 7, or the speaker 16 is connected via the audio signal processing circuit 23. To notify the user by voice. By repeating the processes from step S01 to step S10 described above (step S09, No), the user is notified of information regarding the contact position according to the operation of the endoscope and the treatment tool.

It will be described in detail how to calculate the contact position performed by step S 0 2 in the treatment instrument contact position calculation controller 207 of FIG.
FIG. 10 is a perspective view showing a state in which the insertion portion 4a is inserted to a position near the turbine blade 301 to be inspected and the treatment instrument 114 is protruded.

  In FIG. 10, the optical center of the left objective lens is the origin O, the straight line passing through the left optical center and the right optical center is the X axis, the Z axis is parallel to the optical axis, and the Y axis is perpendicular to the X axis and Z axis. A coordinate system is set as an axis.

  When the treatment tool extends straight with respect to the turbine blade 301 when the treatment tool is projected, the treatment tool takes a path indicated by a broken line 311 in FIG. That is, if the treatment instrument is delivered straight from the adapter entry port 103 (coordinates (b / 2, a, 0)) that is the treatment instrument delivery port, the three-dimensional treatment instrument path W (Z) is

It becomes.
Next, a treatment instrument path in the endoscope image will be considered.
FIG. 11 is a diagram illustrating a treatment target, a treatment instrument path, and a contact position on an endoscopic image corresponding to FIG.

On the endoscopic image, in FIG. 11, points corresponding to the optical centers of the left (the optical system of the observation window 108L) and the right (the optical system of the observation window 108R) are the origins OL and OR, and the right and left side endoscopes are viewed. The coordinate systems of the X _L axis, the Y _L axis, the X _R axis, and the Y _R axis are set in the horizontal and vertical directions on the mirror images 314L and 313R, respectively. At this time, the treatment instrument paths W _L (Z) and W _R (Z) on the left and right endoscope images on the endoscope image are respectively set to the origins OL and OR.

It becomes. In the above formula, f is the focal length. The treatment instrument paths W _L (Z) and W _R (Z) on the endoscopic image are indicated by dotted lines 312L and 312R in FIG.
The contact position on the left endoscope image 313L is determined by the three-dimensional treatment instrument path W (Z) represented by Expression (1) and the treatment instrument path on the left endoscope image represented by Expression (2). Calculation is made based on W _L (Z) and the three-dimensional information of the subject. In the following description, the calculation of the contact position on the left endoscopic image will be described, but the contact position on the right endoscopic image is also calculated on the right side shown by the equations (1) and (3). The same can be calculated based on the treatment instrument path W _R (Z) on the endoscopic image and the three-dimensional information of the subject.

First, stereo measurement is performed along the treatment instrument path W _L (Z) on the endoscopic image. Thereby, the three-dimensional coordinates of the object on the treatment instrument path among the subjects are obtained. A point where the obtained three-dimensional coordinate P (W _L (Z)) and the treatment instrument path W (Z) coincide is a contact position Q = W _L (Q _Z ) = (Q _X , Q _Y , Q _Z ). Become.

  In the above description, the treatment instrument 114 has been described on the premise that the treatment instrument 114 is delivered straight from the treatment instrument delivery port 103 (coordinates (b / 2, a, 0)) in the Z-axis direction (direction parallel to the optical axis). The cable for delivering the treatment instrument 114 may bend due to its own weight. Therefore, in this case, it is necessary to correct the treatment instrument path and calculate the contact position.

As an example of the case where the cable bends, FIG. 12 shows the X coordinate and the Y coordinate of the position of the treatment tool 114 with respect to the length sent by the treatment tool 114.
In the case of the figure, it is shown that the cable that sends out the treatment instrument 114 bends in both the X-axis direction and the Y-axis direction by its own weight when the sent length increases.

FIG. 13 shows a treatment instrument path on the endoscope image when the cable is bent.
In the left endoscopic image 320L and the right endoscopic image 320R, when the cable for delivering the treatment instrument 114 bends, the route of the treatment instrument follows a different path from the case where it does not bend at all. .

  The three-dimensional treatment instrument path when the cable bends is not constant as in the case where the X and Y coordinates do not bend, and changes depending on the value of the Z coordinate relative to the length of the cable sent. When the three-dimensional treatment instrument path is WT (Z), for example, it is expressed by the following equation.

WT _x (Z) and WT _y (Z), which are the values of the X and Y coordinates in Equation (4), include two unknowns p1, p2, q1, and q2, respectively. It is obtained by solving simultaneous equations by detecting three or more points as measurement points from the upper cable and calculating ternary coordinates. Thus, the three-dimensional treatment instrument path WT (Z) when the cable is bent can be calculated, and the contact position Q can be calculated according to the above flow.

In the equation (4), the values of the X and Y coordinates are expressed by a quadratic expression of Z, but the three-dimensional treatment instrument path WT (Z) may be expressed by using an expression of a third or higher order depending on the characteristics of the cable. good. In this case, by detecting the point on the cable by the number of unknowns included in the formula as a measurement point, likewise derivable values of unknowns can be calculated treatment tool path.

As described above, in the endoscope apparatus according to the present embodiment, the contact position can be accurately calculated by correcting the treatment tool path even when the treatment tool bends due to its own weight.
FIG. 14 is a block diagram of a portion that performs processing for obtaining a contact position centered on the treatment instrument contact position calculation control unit 207.

  14, the treatment instrument contact position calculation control unit 207 includes a treatment instrument information storage unit 231, a treatment instrument path calculation unit 232, a subject three-dimensional information acquisition unit 233, a contact position calculation unit 234, and a contact position-endoscope insertion unit. A distance calculation unit 236, a treatment instrument tip position detection unit 237, and a contact position-treatment tool tip distance calculation unit 238 are provided.

The treatment instrument information storage unit 231 stores various information about the treatment instrument such as three-dimensional information of the treatment instrument delivery port position and information on the delivery direction of the treatment instrument. The treatment instrument path calculation unit 232 stores the information on the treatment instrument. When calculating the route, the information is read from the treatment instrument information storage unit 231. The treatment instrument path calculation unit 232 calculates the three-dimensional treatment instrument path W (Z) and the treatment instrument paths W _L (Z) and W _R (Z) on the left and right endoscope images. The subject three-dimensional information acquisition unit 233 outputs the treatment tool path W _L (Z) on the endoscopic image obtained by the treatment tool path calculation unit 232 to the stereo measurement unit 206 via the system control unit 201, and as a result. The subject three-dimensional information P (W _L (Z)) obtained by the stereo measurement unit 206 is received from the system control unit 201 and sent to the contact position calculation unit 234 and the treatment instrument tip position detection unit 237. The contact position calculation unit 234 determines the treatment tool from the three-dimensional treatment tool path W (Z) obtained by the treatment tool path calculation unit 232 and the subject three-dimensional information P (W _L (Z)) from the subject three-dimensional information acquisition unit 233. The contact position Q is obtained and output to the system control unit 201 and the contact position-endoscope insertion unit distance calculation unit 236.

The contact position-endoscope insertion unit distance calculation unit 236 calculates a distance L ₁ between the contact position Q and the endoscope insertion unit 4 a from the coordinate information of the contact position Q, and outputs this to the system control unit 201. The treatment instrument tip position detection unit 237 obtains the tip position of the treatment tool from the subject three-dimensional information P (W _L (Z)), and outputs it to the contact position-treatment tool tip distance calculation unit 238. The contact position-treatment tool tip distance calculation unit 238 calculates a distance L ₂ between the contact position and the treatment tool tip from the coordinate information of the contact position Q and the treatment tool tip, and outputs the result to the system control unit 201.

  Details of the contact position calculation processing by the treatment instrument contact position calculation control unit 207 will be described using the flowchart of FIG. 15 in such a configuration. The process shown in the figure corresponds to the process that the system control unit 201 causes the treatment instrument contact position calculation control unit to perform in step S02 of the flowchart of FIG.

In the following description, only the process for the left endoscopic image is described for the sake of simplification, but the treatment instrument path calculation unit 232 also applies the right endoscopic image to the right endoscopic image. The treatment instrument path W _R (Z) is calculated, and the same processing as described below is performed using this value.

First, in step S101, the treatment instrument path calculation unit 232 reads the treatment instrument delivery position (b / 2, a, 0) and the treatment instrument delivery direction stored in the treatment instrument information storage unit 231, and from these information, The three-dimensional treatment instrument path W (Z) and the treatment instrument path W _L (Z) on the endoscope image are calculated.

In step S102, a predetermined initial value Z0 is set as an initialization process for the variable Z for setting a value in the Z-axis direction.
Then, in step S103, the subject three-dimensional information acquisition unit 233 obtains the variable Z value by substituting the variable Z value into the treatment tool path W _L (Z) on the left endoscopic image obtained by the treatment tool path calculation unit 232. This point is notified to the system control unit 201, and the three-dimensional coordinate P (W _L (Z)) of this point is obtained by stereo measurement by the stereo measurement unit 206.

In response to this, the system control unit 201 causes the stereo measurement unit 206 to obtain the three-dimensional coordinates of the point notified from the subject three-dimensional information acquisition unit 233 and returns the result to the subject three-dimensional information acquisition unit 233. Note that an endoscopic image input to the stereo measurement unit 206 when performing this stereo measurement is input from the image input unit 214 via the system control unit 201. The stereo measurement unit 206 performs stereo measurement on the input endoscopic image with W _L (Z) notified from the system control unit from the subject three-dimensional information acquisition unit 233, and performs three-dimensional coordinates P (W _L (Z)) is obtained.

When the three-dimensional coordinate P (W _L (Z)) is obtained, the contact position calculation unit 234 obtains the obtained P (W _L (Z)) and the treatment tool path calculation unit 232 in step S104. The paths W (Z) are compared. As a result, if the distance between P (W _L (Z)) and W (Z) (for example, Euclidean distance) is equal to or greater than a predetermined threshold value e (No in step S104), it is assumed that the two points do not coincide with each other in step S105. If the distance is smaller than the threshold value e (step S104, Yes), it is considered that the two points coincide with each other, and in step S107, the contact position calculation unit 234 determines W (Z) as the contact position Q. = (Q _X , Q _Y , Q _Z ), this three-dimensional coordinate Q is output to the system control unit 201 as a contact position where the treatment tool contacts the subject, and the processing of FIG. 15 is terminated.

  In step S105, the value of the variable Z is updated by being increased by a predetermined numerical value Z1 that is the resolution in the Z-axis direction, and in step S106, it is determined whether the value of the variable Z exceeds a predetermined calculation limit distance Z2. As a result, if the value of the variable Z exceeds the calculation limit distance Z2 (step S106, Yes), after notifying the system control unit 201 that the contact position is unknown as step S108, the process of FIG. .

If the value of the variable Z does not exceed the calculation limit distance Z2 in step S106 (No in step S106), the process returns to step S103, and the processes of steps S103 to S106 are repeated with the updated variable Z value. Thereafter, in step S104, the distance between the point P (W _L (Z)) and the point W (Z) is less than or equal to the threshold value e, and the contact point Q is found, or in step S106, the value of the variable Z is calculated as the calculation limit distance Z2. Steps S103 to S106 are repeated until the contact point Q is obtained.

When the contact point Q is obtained, the contact position-endoscope insertion unit distance calculation unit 236 then calculates the distance L ₁ between the contact point Q and the endoscope insertion unit from the calculated coordinates of the contact point Q, and the system Output to the control unit 201.

The treatment instrument tip position detection unit 237 detects the treatment tool tip position based on the subject three-dimensional information P (W _L (Z)), and the contact position-treatment tool tip distance calculation unit 238 detects the treatment tool tip position. The distance L _{2 of the} contact position is calculated and output to the system control unit 201.

As described above, the contact position, which is the information output to the system control unit 201, the distance L ₁ between the contact position and the endoscope insertion unit, the distance L ₂ between the treatment tool distal end position and the contact position, It is converted into display data for graphic display using the graphic display instruction unit 213 and displayed on the display screen of the LD 3 or FMD 7 to notify the user.

An example of the display is shown in FIG.
In the figure, X marks 331L and 331R are displayed on the points W _L (Q _Z ) and W _R (Q _Z ) corresponding to the contact position of the treatment tool on the endoscope images 330L and 330R of the left and right optical systems. , Notify the user. Further, the distance L ₁ between the endoscope insertion portion and the contact position is displayed on the display portion 332 in mm units. Furthermore, when the treatment tool is detected on the endoscopic image, the distance L ₂ between the treatment tool and the contact position is also displayed in mm.

  Thus, since the contact position is synthesized and displayed on the endoscope image, it is possible to easily grasp the contact position on the endoscope image. Further, since the distance between the contact position and the endoscope insertion portion and the distance between the contact position and the treatment instrument tip are also notified, the length for inserting the treatment instrument can be known from this distance, and an accurate operation can be performed.

  Note that the endoscope tip or the treatment tool may be shaken during operation of the insertion portion. At this time, the calculated contact position Q is not stable, and the display of the contact position becomes unclear. In such a case, in the endoscope apparatus of the present embodiment, the display of the contact position is clarified by calculating and displaying the center of vibration from the values of the plurality of contact positions Q obtained by the shake.

Accordingly, even when the treatment instrument or the endoscope insertion portion is shaken and vibration occurs at the contact position, it is possible to accurately notify the contact position by calculating the center of vibration.
As a contact position notification method, a three-dimensional shape of a subject may be configured by stereo measurement for a predetermined region of an image including the contact position, and the contact position may be synthesized and displayed on the three-dimensional shape of the subject. A display example in this case is shown in FIG.

  In the case of the figure, based on the three-dimensional information obtained by stereo measurement, the subject is displayed in a three-dimensional display 340 using a wire frame model, and the contact position is notified to the user by an X mark on the wire frame.

In this way, by displaying the subject three-dimensionally and combining and displaying the contact position during the display, it is possible to perform the treatment with the treatment tool while grasping the three-dimensional shape of the treatment object.
As described above, in the endoscope apparatus according to the present embodiment, the user can easily grasp the three-dimensional relationship between the insertion portion of the endoscope apparatus, the treatment tool, and the subject. For example, the cracked turbine frame When marking a card, it is possible to easily identify the part to be marked, appropriately approach the treatment tool to the place to be marked, and perform the work reliably.

Moreover, the endoscope apparatus of this embodiment can use not only markings but also forceps as a treatment tool used in the endoscope apparatus as shown in FIG.
In the case of FIG. 18, it is possible to perform an operation of grasping and collecting the article 351 or the like in the inspection target space with the forceps 114b. Such an operation cannot be performed well unless the contact position between the treatment tool and the article is known. However, in the endoscope apparatus of the present embodiment, the contact position is notified, so that the position to be grasped by the forceps becomes clear. The work of grabbing becomes easier.

Furthermore, the endoscope apparatus according to the present embodiment is not limited to marking with paint or collection work with forceps, and various treatments can be performed using treatment tools other than forceps.
In the above description, the case where the endoscope apparatus is configured as an industrial endoscope apparatus has been described as an example. However, the endoscope apparatus according to the present embodiment is not limited to an industrial apparatus, and may be used in other fields such as medical applications. Even when configured as an endoscope apparatus, similar actions and effects can be obtained.

  Furthermore, in the above description, the user is notified of the obtained contact position, and the user performs the treatment. However, the endoscope apparatus may automatically perform the treatment from the obtained contact position information. good.

It is a figure which shows the external structure of the endoscope apparatus of this embodiment. It is the circuit block diagram described centering on the part which manages the control processing of an endoscope apparatus. It is a perspective view which shows the principal part structure of the endoscope apparatus centering on a scope part. It is a figure which shows the state with which the direct vision binocular adapter for stereo measurement was mounted | worn with the head part. It is a block diagram of the functional block implement | achieved by the software of an endoscope apparatus. It is explanatory drawing of how to obtain | require the three-dimensional coordinate of the measurement point by the stereo measurement based on the principle of a triangulation. It is a figure (the 1) which shows the state which inserted the insertion part to the vicinity position of the test object. It is a figure (the 2) which shows the state which inserted the insertion part to the vicinity position of the test object. It is a flowchart which shows the process which a system control part performs at the time of a notification process. It is a figure for demonstrating the method of calculation of a contact position, and is a perspective view which shows the state which inserted the insertion part to the vicinity position of the test object, and protruded the treatment tool. It is a figure which shows the treatment target, treatment tool path, and contact position on the endoscopic image corresponding to FIG. It is a figure which shows the example of the bending method of the cable with respect to the length which the treatment tool sent out. It is a figure which shows the treatment tool path | route on an endoscope image when a cable bends. It is a block diagram of the part which performs the process which calculates | requires the contact position centering on a treatment tool contact position calculation control part. It is a flowchart which shows the calculation process of the contact position by a treatment tool contact position calculation control part. It is a figure which shows the example of the display screen which performs notification of a contact position. It is a figure which shows the example of the display screen which synthesize | combines and displays a contact position on the three-dimensional shape of a to-be-photographed object, and notifies a contact position. It is a figure which shows the example at the time of using a treatment tool as forceps with an endoscope apparatus. It is a figure which shows the mode of the treatment by the endoscope which can insert the treatment tool in the conventional endoscope apparatus. It is a figure which shows the endoscopic image | video displayed on the treatment character by the endoscope which can insert the treatment tool in the conventional endoscope apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Endoscope apparatus 2 Control unit 3 Liquid crystal monitor 4 Scope part 6 Remote controller 7 Face mount display 8 FMD adapter 11 Endoscope unit 12 Camera control unit 13 Compact flash (registered trademark) memory card 14 PCMCIA memory card 15 Microphone 16 Speaker 20 Processing Control Unit 21 Video Signal Processing Circuit 22 Card Interface 23 Audio Signal Processing Circuit 24 Bus 26 CPU
27 ROM
28 RAM
DESCRIPTION OF SYMBOLS 29 Connection interface 41 Projection part 42 Optical lens group 43 Solid-state image sensor 44 Optical adapter mounting part 47 Internal channel 48 Forceps opening 100a1 Direct-sighted binocular adapter 100a2 Side-viewing binocular adapter 201 System control part 202 Recording / reproduction | regeneration instruction | indication part 203 Operation detection part 204 Graphic display instruction unit 205 Image input unit 206 Stereo measurement unit 207 Treatment instrument contact position calculation control unit

Claims (6)

In an endoscope apparatus that performs treatment with a treatment tool,
An imaging unit that captures an endoscopic image of a subject with parallax;
A three-dimensional information acquisition unit that performs stereo measurement using the endoscope video imaged by the imaging unit and acquires the three-dimensional information of the subject;
The path of the treatment tool, a treatment instrument route calculating unit for calculating, based on the moving direction of the treatment tool and the position of the treatment instrument outlet,
The contact position where the treatment tool is expected to come into contact with the subject is calculated from the path of the treatment tool obtained by the treatment tool path calculation unit and the three-dimensional information obtained by the three-dimensional information acquisition unit. A contact position calculator,
An endoscope apparatus comprising: The internal contact according to claim 1, wherein the contact position is a position where the three-dimensional information of the subject calculated based on the path of the treatment tool matches the path of the treatment tool. Mirror device. A contact position notifying unit for notifying the contact position;
The endoscope apparatus according to claim 1, wherein the contact position notifying unit notifies the contact position by combining and displaying information on the contact position on the endoscopic image. A distance between the contact position and the distal end position of the treatment instrument is calculated from the three-dimensional coordinate information of the contact position and the three-dimensional coordinate information of the distal end position of the treatment instrument obtained based on the three-dimensional information. A contact position-treatment instrument tip position distance calculation unit;
The endoscope apparatus according to claim 3, wherein the contact position notifying unit notifies a distance between the contact position and a distal end position of the treatment instrument.   The endoscope apparatus according to claim 4, further comprising a notification unit that notifies a user when a distance between the contact position and the distal end position of the treatment instrument becomes smaller than a predetermined value. The treatment instrument path calculation unit corrects the movement direction of the treatment instrument when the movement direction of the treatment instrument changes depending on the length of the cable that sends out the treatment instrument, and the path of the treatment instrument To calculate
The internal contact according to claim 1, wherein the contact position calculation unit calculates the contact position using a path of the treatment tool calculated using the corrected movement direction of the treatment tool. Mirror device.