JP2015107268A - Endoscope observation support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an endoscope observation support device which effectively supports observation with an ultrasonic probe.SOLUTION: An endoscope observation support device includes: an ultrasonic probe 32 which can change a projection amount from an endoscope tip part; a sensor device 18 which detects the tip positions of an endoscope 26 and the ultrasonic probe 32; a camera/ultrasonic image observation position estimation part 56 which performs association between the observation positions of the endoscope 26 and the ultrasonic probe 32 in a three-dimensional image of luminal organs constructed based on three-dimensional image data; an ultrasonic image acquisition part 48 which consecutively acquires a plurality of ultrasonic images of the observation positions while changing the projection amount of the ultrasonic probe 32; and an ultrasonic image three-dimensional reconstruction part 72 which reconstructs a three-dimensional image of the observation position by the ultrasonic probe 32 on the basis of the plurality of acquired ultrasonic images, suitably grasps correspondence between the observation position of the ultrasonic probe 32 and the three-dimensional image of luminal organs and reconstructs a three-dimensional image of a target site.

Description

  The present invention relates to an endoscope observation support apparatus that supports observation with an endoscope inserted into a hollow organ, and in particular, to effectively support observation with an ultrasonic probe provided in the endoscope. Regarding improvements.

  At present, as an examination of a luminal organ of a subject, an endoscopic examination is performed in which an endoscope is inserted into the luminal organ to acquire an internal image. For example, bronchoscopy in which an endoscope (bronchoscope) is inserted from the mouth or nose of a subject is performed as an inspection of the bronchial lumen. In this bronchoscopy, the surface properties of the bronchial lumen can be observed with an endoscope. However, in observation with an endoscope, it is difficult to observe a tissue outside the bronchial lumen (on the other side of the bronchial wall). For this reason, an ultrasonic probe is inserted into, for example, a forceps opening of an endoscope in order to investigate to what extent the lesion has infiltrated into a region suspected of having a lesion such as lung cancer. Then, based on the ultrasonic image acquired by the ultrasonic probe, the observation of the target portion and the diagnosis are performed.

  With respect to the endoscopy as described above, a technique for supporting (navigating) observation with the endoscope has been proposed. For example, the techniques described in Patent Literature 1 and Non-Patent Literature 1 are examples. The technique described in Patent Document 1 guides an endoscope to a target site by indicating which branch should be inserted when the endoscope is inserted. The technique described in Non-Patent Document 1 is a virtual that is generated from information obtained by an endoscope insertion length sensor, an image obtained from an endoscope camera, and a CT image taken in advance before endoscopy. The position of the current endoscope is estimated using the modified endoscope image.

Japanese Patent No. 4822142

Xiongbiao Luo, Takayuki Kitasaka, Kensaku Mori, "Externally Navigated bronchoscopy using 2-D motion sensors: Dynamic phantom validation," IEEE Transactions on Medical Imaging, DOI (identifier) 10.1109 / TMI.2013.2263152 (2013/05/08)

  By the way, in an endoscope provided with an ultrasonic probe capable of changing the amount of protrusion from the base end, the observation position by the ultrasonic probe and the observation position by the endoscope are changed by changing the protrusion amount of the ultrasonic probe. And the corresponding relationship between the observation position of the ultrasonic probe and the three-dimensional image of the hollow organ constructed based on the three-dimensional image data also changes. For this reason, the conventional technique cannot always efficiently support the observation with the ultrasonic probe provided in the endoscope. Such a problem has been newly found in the process of continual research by the present inventors with the intention of improving the performance of an endoscope equipped with an ultrasonic probe.

  The present invention has been made in the background of the above circumstances, and an object of the present invention is to provide an endoscope observation support apparatus that effectively supports observation with an ultrasonic probe.

  In order to achieve such an object, the gist of the first aspect of the present invention is that observation based on a three-dimensional image data of a hollow organ obtained in advance by an endoscope inserted into the hollow organ is provided. An ultrasonic observation support device that supports the ultrasound probe, which can change the amount of protrusion from the proximal end portion, and the distal end position detection unit that detects the distal end position of the endoscope and the distal end position of the ultrasonic probe And an observation position by the endoscope and an observation position by the ultrasonic probe in the three-dimensional image of the hollow organ constructed based on the three-dimensional image data based on the detection result by the tip position detection unit And an ultrasonic wave that continuously acquires a plurality of ultrasonic images of the observation position by the ultrasonic probe while changing the amount of protrusion from the base end part of the ultrasonic probe. An image acquisition unit, and a three-dimensional image reconstruction unit that reconstructs a three-dimensional image of the observation position by the ultrasonic probe based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit. It is characterized by this.

  As described above, according to the first aspect of the present invention, the ultrasonic probe that can change the amount of protrusion from the proximal end portion, and the distal end position detection unit that detects the distal end position of the endoscope and the distal end position of the ultrasonic probe, Based on the detection result by the tip position detection unit, the observation position by the endoscope and the observation position by the ultrasonic probe in the three-dimensional image of the hollow organ constructed based on the three-dimensional image data Ultrasound image acquisition for continuously acquiring a plurality of ultrasonic images of observation positions by the ultrasonic probe while changing the projection amount from the base end part of the ultrasonic probe, and an observation position association unit that performs association And a three-dimensional image reconstruction unit that reconstructs a three-dimensional image of the observation position by the ultrasonic probe based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit. Since the correspondence relationship between the observation position by the ultrasonic probe and the observation position by the endoscope changes, it is constructed based on the observation position of the ultrasonic probe and the three-dimensional image data. Even when the correspondence relationship with the three-dimensional image of the luminal organ changes, the correspondence between the observation position by the ultrasonic probe and the observation position by the endoscope, and further, the three-dimensional image of the luminal organ It is possible to appropriately grasp the relationship and reconstruct a three-dimensional image of the target part. That is, it is possible to provide an endoscope observation support apparatus that effectively supports observation with an ultrasonic probe.

  The gist of the second invention subordinate to the first invention is that a three-dimensional portion of a specific part at an observation position by the ultrasonic probe is based on a plurality of ultrasonic images acquired by the ultrasonic image acquiring unit. A three-dimensional image of the specific part extracted by the specific part image extraction unit based on a result of association by the specific part image extraction unit for extracting an image and the observation position association unit is converted into the three-dimensional image data. And a specific part image composition display unit that composes and displays a three-dimensional image of the luminal organ constructed based on the above. In this way, based on the ultrasonic image acquired by the ultrasonic probe, a three-dimensional image of a specific part including the lesioned part is suitably displayed on the three-dimensional image of the luminal organ to be observed. Composite display is possible.

  The gist of the third invention subordinate to the first invention or the second invention is that the three-dimensional image of the luminal organ constructed based on the three-dimensional image data is acquired by the endoscope. An image display control unit that synchronously displays an endoscopic image and an ultrasonic image acquired by the ultrasonic probe. In this way, practical navigation for displaying a three-dimensional image, an endoscopic image, and an ultrasonic image of the luminal organ to be observed can be realized.

  The gist of the fourth invention according to any one of the first to third inventions is constructed based on the three-dimensional image data based on the result of association by the observation position association unit. An observation position display control unit for displaying an observation position by the ultrasonic probe on a three-dimensional image of the luminal organ. In this way, practical navigation for displaying the observation position by the ultrasonic probe on the three-dimensional image of the luminal organ to be observed can be realized.

  The gist of the fifth invention according to any one of the first to fourth inventions is that the tip position detector is provided at the tip of the endoscope and the tip of the ultrasonic probe, respectively. The tip position of the endoscope and the tip position of the ultrasonic probe are detected based on the detection result of the magnetic position sensor. In this way, the tip positions of the endoscope and the ultrasonic probe can be detected in a practical manner.

  The gist of the sixth invention according to any one of the first to fourth inventions is that the tip position detecting unit is a detection result of a magnetic position sensor provided at the tip of the endoscope. The tip of the ultrasonic probe is detected based on the detection result of the ultrasonic probe insertion length sensor that detects the tip position of the endoscope and the insertion length of the ultrasonic probe. The position is detected. In this way, the tip positions of the endoscope and the ultrasonic probe can be detected in a practical manner.

  The gist of the seventh invention according to any one of the first to fourth inventions is that the distal end position detector detects an insertion length sensor of the endoscope that detects the insertion length of the endoscope. Based on the result and the detection result of the ultrasonic probe insertion length sensor for detecting the insertion length of the ultrasonic probe, the tip position of the endoscope and the tip position of the ultrasonic probe are detected. In this way, the tip positions of the endoscope and the ultrasonic probe can be detected in a practical manner.

It is a figure which illustrates the structure of the endoscope observation assistance apparatus which is an Example of this invention. It is a figure which illustrates roughly the structure of the endoscope main body with which the endoscope observation assistance apparatus shown in FIG. 1 was equipped. It is a figure which shows schematically the structure of the edge part of the tubular part in the endoscope main body shown in FIG. It is a figure which shows typically the structure of the endoscope main body shown in FIG. 2 in the aspect which detects the front-end | tip position of an ultrasonic probe using an ultrasonic probe insertion length sensor. It is a figure explaining extraction of the various information which concerns on the lungs of a subject by the endoscope observation assistance apparatus shown in FIG. It is a figure which shows typically an example of the image of the bronchus produced | generated by the endoscope observation assistance apparatus shown in FIG. It is a figure explaining the lesioned part detected corresponding to the image of the bronchus shown in FIG. It is a figure explaining a mode that the ultrasonic image of a lesioned part is acquired with the ultrasonic probe of the endoscope observation assistance apparatus shown in FIG. It is a figure explaining reconstruction of the three-dimensional image of the lesion part based on the several ultrasonic image acquired continuously by the ultrasonic probe of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows an example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows another example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows another example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows another example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows another example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. It is a figure which shows another example of the observation screen displayed on a video display apparatus at the time of observation of the endoscope observation assistance apparatus shown in FIG. 7 is a flowchart for explaining a main part of an example of transformation matrix determination control between a coordinate system corresponding to an endoscope and a coordinate system corresponding to a magnetic position sensor by the control device of the endoscope observation support apparatus shown in FIG. 1. 6 is a flowchart for explaining a main part of an example of transformation matrix determination control between a coordinate system corresponding to an ultrasonic probe and a coordinate system corresponding to a magnetic position sensor by the control device of the endoscope observation support apparatus shown in FIG. 1. 3 is a flowchart for explaining a main part of an example of transformation matrix determination control between a coordinate system corresponding to a three-dimensional virtual image and a coordinate system corresponding to a magnetic position sensor by the control device of the endoscope observation support apparatus shown in FIG. 1. . It is a flowchart explaining the principal part of an example of the endoscopic observation assistance control by the control apparatus of the endoscopic observation assistance apparatus shown in FIG.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating the configuration of an endoscope observation support apparatus 10 (hereinafter simply referred to as support apparatus 10) that is an embodiment of the present invention. As shown in FIG. 1, the support device 10 according to the present embodiment includes a control device 12, a video display device 14, an endoscope device 16, a sensor device 18, and an endoscope body 20. It is configured. The control device 12 is, for example, a so-called computer that processes and controls electronic information based on a predetermined program stored in advance in a ROM while using a temporary storage function of a RAM by a CPU. The video display device 14 is preferably a known display device (display) such as a TFT (Thin Film Transistor Liquid Crystal), and is connected to the control device 12 and the like via an output interface.

  FIG. 2 is a diagram schematically illustrating the configuration of the endoscope body 20. As shown in FIG. 2, the endoscope body 20 includes a bendable (flexible) tubular portion 22 formed in a long tubular shape, and an operation provided at one end of the tubular portion 22. The unit 24 is provided. The tubular portion 22 is directed to the distal end portion of the tubular portion 22, that is, the end portion 22 a opposite to the operation portion 24, in response to an operation by an angle (knob) (not shown) provided in the operation portion 24. It is configured so that the direction can be freely changed. Preferably, the tubular portion 22 is a forceps port (instrument insertion channel) that is a configuration for taking in and out a treatment tool such as a biopsy forceps, a pipe for sending or sucking air or water, and the like. Is incorporated. Such a configuration will be described later with reference to FIG.

  The tubular portion 22 is provided with an endoscope 26. The endoscope 26 is a known electronic scope or fiberscope used for observing a luminal organ, and is preferably a bronchoscope mainly used for observing the trachea or bronchus. The endoscope 26 is connected to the observation portion 28 provided at an end portion (tip portion) 22a opposite to the operation portion 24 in the tubular portion 22, and the tubular portion 22 while being connected to the observation portion 28. The cable part 30 provided in the inside is provided and comprised. The endoscope 26 is connected to the endoscope apparatus 16 via the cable part 30. That is, the endoscope image acquired by the observation unit 28 is configured to be supplied to the endoscope device 16 and the control device 12 via the cable unit 30.

  The observation unit 28 corresponds to an image (subject image) acquisition unit in the endoscope 26 and is provided at an end 22 a corresponding to the distal end of the tubular portion 22. When the endoscope 26 is an electronic scope, the observation unit 28 corresponds to an image sensor such as a CCD (Charge Coupled Device). When the endoscope 26 is a fiberscope, the observation unit 28 corresponds to a lens unit for taking the image into a glass fiber (optical fiber) provided in the tubular unit 22. A light source 22b (see FIG. 3) for illuminating an observation site by the observation unit 28 in a hollow organ is provided near the observation unit 28 in the tubular portion 22. In the case where the endoscope 26 is a fiberscope, another optical fiber for guiding illumination light into the hollow organ is incorporated in the tubular portion 22.

  The endoscope body 20 is provided with an ultrasonic probe 32. The ultrasonic probe 32 emits an ultrasonic wave to a part to be observed and receives a reflected wave of the ultrasonic wave, thereby acquiring an ultrasonic image of the part to be observed. It is. The ultrasonic probe 32 includes a transmission / reception unit 34 provided at an end (tip portion) 22a of the tubular portion 22 opposite to the operation unit 24, and the transmission / reception unit 34 and the endoscope device 16. A cable portion 36 to be connected is provided. The transmission / reception unit 34 is a device that emits an ultrasonic wave to the site to be observed and receives a reflected wave of the ultrasonic wave, and corresponds to the main body of the ultrasonic probe 32. The transmission / reception unit 34 is connected to the endoscope device 16 via the cable unit 36. That is, the ultrasonic image acquired by the transmission / reception unit 34 is configured to be supplied to the endoscope device 16 and then to the control device 12 via the cable unit 36.

  The ultrasonic probe 32 is preferably provided by being inserted through a channel 22c (see FIG. 3) corresponding to a forceps opening provided in the tubular portion 22. That is, the cable portion 36 is provided by being inserted through the channel 22 c provided in the tubular portion 22, and the transmitting / receiving portion 34 is provided so as to protrude from the end portion 22 a of the tubular portion 22. The transmission / reception unit 34 is configured to change the amount of protrusion of the tubular portion 22 from the end 22a. That is, in this embodiment, the end portion (end portion on the side opposite to the operation portion 24) 22a of the tubular portion 22 corresponds to the base end portion. As shown in FIG. 3, the transmission / reception unit 34 is preferably formed in a longitudinal shape in the protruding direction from the end 22a. More preferably, it is configured in a columnar shape having an axial center in the protruding direction from the end 22a. The ultrasonic probe 32 preferably includes an actuator that controls the protruding amount of the transmitting / receiving unit 34 from the end 22a. This actuator may automatically change the protruding amount of the transmission / reception unit 34 according to a command from the control device 12 or the like, or according to an operation of a knob or the like provided in the operation unit 24. The protrusion amount of the transmission / reception unit 34 may be changed. The actuator preferably controls the amount of protrusion from the end 22a while continuously rotating the transmitting / receiving unit 34. Alternatively, the ultrasonic probe 32 is configured to be able to manually change the protruding amount of the transmitting / receiving unit 34 from the end 22a by manually changing the insertion length with respect to the tubular portion 22 (hand). It may be what was done.

  FIG. 3 is a diagram schematically showing the configuration of the end portion 22 a of the tubular portion 22. As shown in FIG. 2, the endoscope body 20 preferably includes magnetic position sensors 38 and 40 at the distal end portion of the endoscope 26 and the distal end portion of the ultrasonic probe 32, respectively. . That is, the magnetic position sensor 38 is near the observation unit 28 at the distal end of the endoscope 26, and the magnetic position sensor 40 is near the transmission / reception unit 34 at the distal end of the ultrasonic probe 32. Each is provided. FIG. 3 shows a mode in which a magnetic position sensor 38 provided corresponding to the distal end portion of the endoscope 26 is fixed (externally attached) to the side surface of the tubular portion 22. The position sensor 38 may be incorporated in the channel of the tubular portion 22 or the like. The position of the magnetic position sensor provided corresponding to the tip of the ultrasonic probe 32 is also limited to the mode shown in FIG. 3 as long as the tip of the ultrasonic probe 32 can be suitably detected. Not.

  The magnetic position sensors 38 and 40 are preferably known magnetic position sensors that detect the position of each sensor by detecting a magnetic field (magnetic signal) generated by a magnetic field generator, and the sensor device 18 is connected. The sensor device 18 includes a magnetic field generator (not shown). In the sensor device 18, for example, when each magnetic position sensor 38, 40 receives a magnetic signal transmitted from the magnetic field generation device, for example, five-dimensional information, that is, three-dimensional information from each magnetic position sensor 38, 40. Two-dimensional position information and two-dimensional direction information are acquired and supplied to the control device 12 and the like via the sensor device 18. Thus, the sensor device 18 is preferably based on the detection results of the magnetic position sensors 38 and 40 provided at the distal end portion of the endoscope 26 and the distal end portion of the ultrasonic probe 32, respectively. The tip position of the endoscope 26 and the tip position of the ultrasonic probe 32 are detected. That is, in the present embodiment, the sensor device 18 corresponds to a tip position detection unit that detects the tip position of the endoscope 26 and the tip position of the ultrasonic probe 32. The sensor device 18 may directly detect the tip position of the endoscope 26 and the tip position of the ultrasonic probe 32 by the magnetic position sensors 38 and 40, or the magnetic position sensor 38. , 40 may be used to calculate (estimate) the distal end position of the endoscope 26 and the distal end position of the ultrasonic probe 32 based on the detection results.

  The support apparatus 10 includes an ultrasonic probe insertion length sensor 90 that detects the insertion length of the ultrasonic probe 32 with respect to the tubular portion 22 as an alternative to the magnetic position sensor 40 corresponding to the ultrasonic probe 32. It may be a thing. FIG. 4 is a diagram schematically showing the configuration of the endoscope main body 20 in a mode in which the tip position of the ultrasonic probe 32 is detected using the ultrasonic probe insertion length sensor 90. The ultrasonic probe insertion length sensor 90 is a known non-contact insertion length sensor such as an optical movement distance sensor or a rotary encoder. As shown in FIG. 4, the ultrasonic probe insertion length sensor 90 is preferably provided in an opening on the operation unit 24 side in a channel 22c such as a forceps port provided in the tubular portion 22. The insertion length (insertion length dimension) of the ultrasonic probe 32 with respect to the tubular portion 22 is detected from the opening.

  In the embodiment shown in FIG. 4, the sensor device 18 preferably detects the tip position of the endoscope 26 based on the detection result of the magnetic position sensor 38, and the tip position of the endoscope 26 and Based on the detection result of the ultrasonic probe insertion length sensor 90, the tip position of the ultrasonic probe 32 is detected. Since the length dimension of the channel 22c provided in the tubular portion 22 and the length dimension of the ultrasonic probe 32 are obtained in advance, the protruding amount of the ultrasonic probe 32 from the end portion 22a corresponding to the base end portion. Corresponds to the insertion length of the ultrasonic probe 32 into the tubular portion 22. In the present embodiment, the distal end position of the endoscope 26 corresponds to the end portion 22 a of the tubular portion 22. Therefore, the distal end position of the ultrasonic probe 32 as a relative position with respect to the distal end position of the endoscope 26 based on the distal end position of the endoscope 26 and the protruding amount of the ultrasonic probe 32 from the end portion 22a. Can be detected.

  As indicated by a broken line in FIG. 4, the support device 10 further includes the endoscope for the subject (a luminal organ to be observed) as an alternative to the magnetic position sensor 38 corresponding to the endoscope 26. An endoscope insertion length sensor 92 that detects the insertion length of the mirror 26 (tubular portion 22) may be provided. The endoscope insertion length sensor 92 is, for example, a known bronchoscope insertion length sensor attached to an endoscope insertion assisting tool (mouthpiece) 94 that holds a subject on the lip during an endoscopic examination. In this embodiment, the sensor device 18 determines the position of the distal end of the endoscope 26 and the ultrasonography based on the detection result of the endoscope insertion length sensor 92 and the detection result of the ultrasonic probe insertion length sensor 90. The tip position of the acoustic probe 32 is detected.

  As yet another aspect, the sensor device 18 may detect the amount of protrusion of the ultrasonic probe 32 from the end 22a corresponding to the base end. For example, a pattern such as a two-dimensional code is marked at a predetermined interval in the longitudinal direction (projecting direction from the end 22a) of the transmission / reception unit 34, and the pattern from the end 22a is detected by detecting the pattern. The amount of protrusion of the ultrasonic probe 32 (transmission / reception unit 34) may be detected. Alternatively, by analyzing an image of the ultrasonic probe 32 (transmission / reception unit 34) protruding from the end 22a included in the endoscopic image acquired by the endoscope 26, the image from the end 22a is analyzed. A mode of calculating (estimating) the protruding amount of the ultrasonic probe 32 is also conceivable.

  As shown in FIG. 1, the control device 12 includes a storage unit 42. The storage unit 42 is preferably a known storage device (storage medium) such as a hard disk drive capable of storing information. It may be a removable storage medium such as a small memory card or an optical disk. The storage unit 42 is provided with various databases such as an image database 44 and an organ database 46. The image database 44 stores three-dimensional image data of a subject obtained in advance by a known X-ray CT (Computed Tomography) apparatus, magnetic resonance (MRI) apparatus, or the like. In the case where the three-dimensional image data is obtained by X-ray CT, X-rays are irradiated to the subject from multiple directions, the X-rays transmitted through the subject are detected by a detector, and transmitted X The dose information is processed by a computer and reconstructed as a three-dimensional image. The image database 44 stores, for example, a three-dimensional grayscale image of a subject obtained by X-ray CT, a magnetic resonance apparatus, or the like, in other words, a pixel value (density value) for each voxel that is a unit volume of the subject. Is done. The image database 44 stores three-dimensional image data of a luminal organ obtained by X-ray CT, a magnetic resonance apparatus, or the like. Preferably, three-dimensional image data of bronchi obtained by X-ray CT, a magnetic resonance apparatus, or the like is stored.

  The organ database 46 stores various types of information related to the luminal organ of the subject to be observed. Preferably, various types of information regarding the bronchus of the subject are stored as an example of an organ having a tree structure. For example, the organ database 46 includes, as information on the bronchi of the subject, corresponding to each of a plurality of parts identified by a predetermined anatomical name in the bronchus, Stores characteristic information such as traveling direction, length, and cross-sectional area.

  The support apparatus 10 of the present embodiment is configured so that the endoscope inserted into the hollow organ based on the three-dimensional image data obtained in advance by an X-ray CT, a magnetic resonance apparatus, or the like regarding the hollow organ of the subject. Control for supporting observation by the mirror 26 and the ultrasonic probe 32 is performed. Preferably, the three-dimensional virtual image corresponding to an organ having a tree structure such as a bronchus is generated, and observation of the three-dimensional virtual image is supported. Preferably, based on the three-dimensional image data of the bronchi, control is performed to support observation with the endoscope 26 and the ultrasonic probe 32 inserted into the bronchus. That is, when observing a predetermined bronchus with the endoscope 26 and the ultrasonic probe 32, the three-dimensional image data obtained in advance corresponding to the bronchus and stored in the image database 44 is read out. By performing the control to be described later based on the above, observation by the endoscope 26 and the ultrasonic probe 32 inserted into the bronchi is supported. Or you may observe and support the three-dimensional virtual image corresponding to luminal organs (hollow organ), such as large intestine, small intestine, duodenum, stomach, bile duct, pancreatic duct, lymphatic duct.

  As shown in FIG. 1, the support apparatus 10 includes an ultrasonic image acquisition unit 48, a CT image data capture unit 50, an information processing unit 52, an information extraction unit 60, an information fusion processing unit 62, and an anatomical name information display. The control part, such as the part 80 and the information visualization part 82, is provided. These control units are preferably all functionally provided in the control device 12, but each control unit is individually provided in a plurality of control devices and computers and can communicate with each other. It may be configured. A part of each control unit may be provided in a control device different from the support device 10. For example, the information fusion processing unit 62, the information visualization unit 82, and the like may be provided in a device different from the support device 10 (for example, a display-only device).

  The ultrasonic image acquisition unit 48 acquires an ultrasonic image of the observation position by the ultrasonic probe 32. That is, an ultrasonic image of the observation position is acquired by emitting an ultrasonic wave from the transmission / reception unit 34 of the ultrasonic probe 32 to the site to be observed and receiving a reflected wave of the ultrasonic wave. .

  The ultrasonic image acquisition unit 48 continuously acquires a plurality of ultrasonic images of the observation position by the ultrasonic probe 32 while changing the protruding amount of the tubular portion 22 from the end 22a of the ultrasonic probe 32. To do. Preferably, the transmitting / receiving unit 34 continuously rotates the transmitting / receiving unit 34 and changes the amount of protrusion of the transmitting / receiving unit 34 from the end 22a by an actuator provided in the ultrasonic probe 32. A plurality of ultrasonic images are acquired continuously. In other words, while the transmitter / receiver 34 is continuously rotated with respect to the site to be observed in the subject and a minute amount is taken in and out, a plurality of ultrasonic images of the site are continuously acquired. That is, continuous scanning is performed on the three-dimensional region of the observation target region in the subject.

  The CT image data capturing unit 50 captures, from the image database 44, three-dimensional image data obtained in advance corresponding to the luminal organ to be observed and stored in the image database 44. That is, the three-dimensional image data is read and developed in a RAM or the like in the control device 12.

  The information processing unit 52 processes information supplied from the endoscope device 16 and the sensor device 18. That is, various types of information related to observation of the subject by the endoscope device 16 and the sensor device 18 are processed. Therefore, the information processing unit 52 includes a camera / sensor / probe calibration unit 54, a camera / ultrasound image observation position estimation unit 56, and a luminal organ image creation unit 58. Hereinafter, processing of each control unit will be described.

  The camera / sensor / probe calibration unit 54 calibrates (corrects) information supplied from the endoscope device 16 and the sensor device 18 based on a predetermined algorithm. For example, distortion correction in an endoscopic image acquired by the endoscope 26 is performed. Calibration of distortion in the ultrasonic image acquired by the ultrasonic probe 32 is performed. Calibration of position blur detected by the magnetic position sensors 38 and 40, calibration of magnetic field distortion, and the like are performed.

  The luminal organ image creating unit 58 creates an image of the luminal organ to be observed based on the three-dimensional image data captured by the CT image data capturing unit 50. For example, a three-dimensional virtual image of the luminal organ is generated based on the three-dimensional image data captured by the CT image data capturing unit 50. The three-dimensional image data stored in the image database 44 is obtained by continuously feeding the subject in the body axis direction while continuously rotating the X-ray irradiator / detector using, for example, a CT apparatus. This is three-dimensional image data obtained by continuously scanning a three-dimensional region of a subject. The luminal organ image creation unit 58 reads out such 3D image data and creates a 3D virtual image of the organ by stacking tomographic images of successive slices of the 3D region. In the present embodiment, display control of a three-dimensional virtual image showing a virtual outer shape of an organ to be observed (for example, an image representing an appearance viewed from the front side) will be described. The present invention is also suitable for display control in which a three-dimensional virtual endoscopic image corresponding to an image obtained by inserting an endoscope into a lumen is displayed based on the three-dimensional image data. Applied.

  The information extraction unit 60 extracts various types of information related to the luminal organ to be observed with respect to the three-dimensional virtual image created by the luminal organ image creation unit 58. For example, each of the parts in the three-dimensional virtual image generated by the luminal organ image creation unit 58 is associated with a plurality of organs (organ parts) identified by predetermined anatomical names. Identify which organ (part of the organ) corresponds to. Preferably, features such as the running direction, length, and cross-sectional area of each part in the three-dimensional virtual image generated by the luminal organ image creation unit 58 are extracted, and the feature information stored in the organ database 46 is extracted. By referencing, it is identified which organ the target region corresponds to. For example, regarding the feature information stored in the organ database 46, the organ having the closest feature such as the traveling direction, length, cross-sectional area, etc. is identified as the organ corresponding to the target region. The information extraction unit 60 uses the VOI (Volume Of Interest: hereinafter referred to simply as VOI) as a volume region of the three-dimensional virtual image of the organ generated by the luminal organ image creation unit 58. The organ to be observed may be identified by dividing and extracting organ region information and structure information in the VOI.

  FIG. 5 is a diagram for explaining extraction of various types of information related to the lungs of the subject by the information extraction unit 60. As shown in FIG. 5, the information extraction unit 60 corresponds to a portion corresponding to the trachea and bronchus and a lymph node of the lung in the three-dimensional virtual image of the organ generated by the luminal organ image creation unit 58. Segmentation (segmentation) of a portion corresponding to a blood vessel (artery, vein) of a lung, a portion corresponding to a lung field and a lung lobe. A plurality of organs identified by an anatomical name included in each of a portion corresponding to the divided trachea and bronchus, a portion corresponding to a lymph node, a portion corresponding to a blood vessel, and a portion corresponding to a lung field and a lung lobe The classification is performed. As described above, each part in the three-dimensional virtual image generated by the luminal organ image creation unit 58 is associated with the lung structure in the subject.

  The camera / ultrasound image observation position estimation unit 56 estimates the observation position by the endoscope 26 and the observation position by the ultrasonic probe 32 based on the detection result by the sensor device 18. The observation position by the endoscope 26 corresponds to the lens position of the observation unit 28, for example. The observation position by the ultrasonic probe 32 corresponds to, for example, a portion (for example, a front end portion of the transmission / reception unit 34) that transmits and receives ultrasonic waves in the transmission / reception unit 34. The camera / ultrasonic image observation position estimation unit 56 uses the observation position by the endoscope 26 and the ultrasonic probe 32 as relative positions in the three-dimensional virtual image created by the luminal organ image creation unit 58. Estimate the observation position. In other words, in the present embodiment, the camera / ultrasound image observation position estimation unit 56 is constructed based on the detection result by the sensor device 16 and is constructed based on the three-dimensional image data. This corresponds to an observation position correspondence unit that associates an observation position by the endoscope 26 and an observation position by the ultrasonic probe 32 in the image.

  The camera / ultrasound image observation position estimation unit 56 includes, for example, (1) a coordinate system corresponding to the endoscope 26, (2) a coordinate system corresponding to the ultrasonic probe 32, and (3) the magnetic position. Considering a plurality of coordinate systems (for example, a three-dimensional coordinate system, respectively) such as a coordinate system corresponding to the sensors 38 and 40, and (4) a coordinate system corresponding to the three-dimensional virtual image, a transformation matrix between the coordinate systems By deriving, the observation position by the endoscope 26 and the observation position by the ultrasonic probe 32 are associated with each other. For example, (a) a transformation matrix between a coordinate system corresponding to the endoscope 26 and a coordinate system corresponding to the magnetic position sensor 38, and (b) a coordinate system corresponding to the ultrasonic probe 32 and the magnetic position. Deriving a transformation matrix between the coordinate system corresponding to the sensor 40, and (c) a transformation matrix between the coordinate system corresponding to the three-dimensional virtual image and the coordinate system corresponding to the magnetic position sensors 38 and 40, The observation position by the endoscope 26 and the observation position by the ultrasonic probe 32 are associated with each other. Preferably, characteristics of the endoscopic image obtained by analyzing the endoscopic image acquired by the endoscope 26, and ultrasonic waves obtained by analyzing the ultrasonic image acquired by the ultrasonic probe 32 The coordinate systems are associated based on the characteristics of the image and the three-dimensional virtual image (for example, the bronchus core line) obtained by analyzing the three-dimensional virtual image. For the correspondence of this coordinate system, see “Tatsuya Hisato, Takayuki Kitasaka, Yugo Ra and Kensaku Mori,“ Ultrasonic Bronchoscope Navigation System Based on Image Information Integration, ”IEICE Technical Report, Vol.111, No. .47 (IE2011 9-39), Page.7-12 (2011.05.12) ”.

  The information fusion processing unit 62 controls the information processed by the information processing unit 52 to fuse (synthesize) various information and display the information on the video display device 14 via the information visualization unit 82. Preferably, as described in detail below, a three-dimensional virtual image of the luminal organ created by the luminal organ image creation unit 58, an endoscope image acquired by the endoscope 26, The ultrasonic image acquired by the ultrasonic probe 32 is displayed in synchronization with the video display device 14. That is, in this embodiment, the information fusion processing unit 62 functions as an image display control unit. Therefore, the information fusion processing unit 62 includes an insertion path generation unit 64, a camera / probe position presentation unit 66, a CT image / ultrasound image fusion unit 68, a path name display unit 70, and an ultrasound image three-dimensional reconstruction unit 72. , A specific part detection unit 74, a specific part image extraction unit 76, and a biopsy target site display unit 78. Hereinafter, processing of each control unit will be described.

  The insertion path generation unit 64 generates path information corresponding to the path through which the endoscope body 20 (tubular part 22) is inserted in the three-dimensional virtual image created by the luminal organ image creation unit 58. To do. For example, based on the result of association by the camera / ultrasound image observation position estimation unit 56, the observation position by the endoscope 26 from the starting point at which the endoscope body 20 is inserted in the three-dimensional virtual image. Route information corresponding to the route to (or the observation position by the ultrasonic probe 32) is generated. For example, in the three-dimensional virtual image corresponding to the bronchi, the observation position (or the ultrasonic probe 32) by the endoscope 26 from the throat side end portion of the trachea corresponding to the starting point where the endoscope body 20 is inserted. The route information corresponding to the route to the observation position) is generated. The insertion path generation unit 64 preferably generates path information corresponding to the path to the specific part detected by the specific part detection unit 74 described later. For example, route information corresponding to the route from the starting point at which the endoscope main body 20 is inserted to the specific part detected by the specific part detection unit 74 is generated.

  The route name display unit 70 displays name information related to the route information generated by the insertion route generation unit 64. For example, based on the result of association by the camera / ultrasound image observation position estimation unit 56, the anatomical name of the organ on the path corresponding to the path information extracted by the information extraction unit 60 is displayed. . For example, regarding the route information corresponding to the bronchi, bronchial names such as “Left Main Bronchus” and “Right Main Bronchus” on the route corresponding to the route information are displayed in the route information generated by the insertion route generation unit 64. Display at the corresponding position of the organ.

  The anatomical name information display unit 80 generates and displays anatomical name information of the organ to be observed. For example, text data (character image data) corresponding to the anatomical name of the organ to be observed is generated and displayed. Preferably, the generated text data is synthesized and displayed on the three-dimensional virtual image of the hollow organ created by the hollow organ image creation unit 58. Preferably, in the three-dimensional virtual image of the luminal organ created by the luminal organ image creation unit 58, the region corresponding to the organ that is the observation target extracted (identified) by the information extraction unit 60, Text data corresponding to the anatomical name of the organ is synthesized and displayed.

  The camera / probe position presentation unit 66 presents the observation position by the endoscope 26 and the observation position by the ultrasonic probe 32 on the three-dimensional virtual image created by the luminal organ image creation unit 58. . Preferably, based on the estimation result by the camera / ultrasound image observation position estimation unit 56, that is, the result of the correspondence between the observation position by the endoscope 26 and the observation position by the ultrasonic probe 32, the luminal organ The observation position by the endoscope 26 and the observation position by the ultrasonic probe 32 are displayed on the three-dimensional virtual image created by the image creation unit 58. That is, in the present embodiment, the camera / probe position presentation unit 66 corresponds to an observation position display control unit.

  The CT image / ultrasound image fusion unit 68 synthesizes and displays the three-dimensional virtual image created by the luminal organ image creation unit 58 and the ultrasound image acquired by the ultrasound probe 32. Preferably, the luminal organ image creation unit 58 creates the luminal organ image creation unit 58 based on the estimation result by the camera / ultrasonic image observation position estimation unit 56, that is, the result of the observation position association by the ultrasonic probe 32. The ultrasonic image acquired by the ultrasonic probe 32 is synthesized and displayed at the observation position by the ultrasonic probe 32 on the three-dimensional virtual image.

  The ultrasonic image three-dimensional reconstruction unit 72 uses the ultrasonic probe 32 based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit 48 while changing the protruding amount of the ultrasonic probe 32. A three-dimensional image of the observation position is reconstructed. For example, based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit 48, a three-dimensional virtual image of the observation site is created by stacking tomographic images of consecutive slices of a three-dimensional region. That is, in this embodiment, the ultrasonic image three-dimensional reconstruction unit 72 corresponds to a three-dimensional image reconstruction unit.

  The specific part detection unit 74 detects a specific part in the three-dimensional image of the hollow organ constructed based on the three-dimensional image data. Preferably, as the specific part, a lesioned part in the three-dimensional image of the hollow organ, that is, a part suspected of having a lesion is detected. Preferably, based on the three-dimensional image data, the presence of a lesion in the luminal organ is detected by matching a predetermined condition such as the condition of a voxel having a predetermined CT value. For example, the presence of a lesion such as lung cancer is detected outside the bronchial lumen of the subject (on the other side of the bronchial wall).

  The specific part image extraction unit 76 observes the specific part detected by the specific part detection unit 74 using the ultrasonic probe 32 based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit 48. A three-dimensional image of the specific part at the position is extracted. For example, the three-dimensional corresponding to the specific part detected by the specific part detection unit 74 from the three-dimensional virtual image of the observation part by the ultrasonic probe 32 reconstructed by the ultrasonic image three-dimensional reconstruction part 72 Extract a virtual image. Hereinafter, detection of a specific part (lesion) and extraction of a three-dimensional virtual image corresponding to the specific part will be described with reference to FIGS.

  FIG. 6 is a diagram schematically illustrating an example of a bronchial image generated by the support apparatus 10. The image 100 shown in FIG. 6 is an object to be observed by the endoscope 26 created by the luminal organ image creation unit 58 based on the three-dimensional image data captured by the CT image data capture unit 50. This corresponds to a three-dimensional virtual image (outline image) of a certain hollow organ. In the example shown in FIG. 6, the path 102 of the endoscope 26 inserted into the bronchus is synthesized on the bronchial image 100 based on the path information generated by the insertion path generation unit 64. Yes. Further, the endoscope image 104 acquired by the endoscope 26 is synthesized and displayed at the observation position of the endoscope 26 on the bronchial image 100 and is generated by the insertion path generation unit 64. In the route information, the corresponding organ information is displayed by the route name display unit 70.

  FIG. 7 is a diagram for explaining a lesion portion detected in correspondence with the bronchial image 100. In the example shown in FIG. 7, in the bronchial image 100 that is a three-dimensional virtual image created by the luminal organ image creation unit 58, the specific part detection unit 74 detects the lesioned part 106 as the specific part. It shows how it is done. When the lesioned part 106 is detected in this way, the insertion path generation unit 64 generates path information corresponding to the path from the starting point where the endoscope body 20 is inserted to the lesioned part 106. . In the example shown in FIG. 7, the path from the starting point at which the endoscope body 20 is inserted to the lesioned part 106 on the bronchial image 100 based on the path information generated by the insertion path generating unit 64. 108 is synthesized.

  FIG. 8 and FIG. 9 are diagrams for explaining how an ultrasonic image of the lesioned part 106 is acquired by the ultrasonic probe 32. By inserting (advancing) the endoscope 26 into the bronchus according to the path 108 synthesized with the bronchial image 100, as shown in FIG. The transmitting / receiving unit 34) of the acoustic probe 32 can be easily guided to the lesioned part 106. Here, the endoscope 26 cannot observe the tissue outside the bronchial lumen (on the other side of the bronchial wall). Therefore, an ultrasonic image is acquired from the lesioned portion 106 by the ultrasonic probe 32. FIG. 8 shows a state in which the ultrasonic image 108 acquired by the ultrasonic probe 32 is synthesized and displayed at the observation position of the ultrasonic probe 32 on the bronchial image 100. As shown in FIG. 8, the ultrasonic image 108 is an image corresponding to a plane perpendicular to the longitudinal direction of the transmitter / receiver 34 in the ultrasonic probe 32, for example. The ultrasound image 108 acquired by the ultrasound probe 32 includes an image of a tissue outside the bronchial lumen. For example, the ultrasound image 108 includes an image 110 corresponding to the bronchi, an image 112 corresponding to a blood vessel near the bronchus, and an image 114 corresponding to the lesioned part 106.

  In the support device 10, as shown in FIG. 9, the amount of protrusion of the ultrasonic probe 32 from the end 22 a of the tubular portion 22 is changed by the ultrasonic image acquisition unit 48, while the lesioned portion 106 is changed. A plurality of ultrasonic images 108a, 108b, 108c,... (Hereinafter simply referred to as ultrasonic images 108 unless otherwise distinguished) are acquired continuously. For example, a plurality of ultrasonic images 108 of the part are continuously acquired while the ultrasonic probe 32 is inserted and removed minutely with respect to a thin tube near the lesioned part 106 in the bronchus. Preferably, the transmission / reception unit 34 is continuously rotated, and a plurality of ultrasonic images 108 of the part are continuously acquired while the ultrasonic probe 32 is slightly inserted into and extracted from the tube. A three-dimensional virtual image of the observation site of the ultrasonic probe 32 is reconstructed by stacking tomographic images of successive slices of a three-dimensional region based on the plurality of ultrasonic images 108 thus acquired. Then, as shown in FIG. 9, a three-dimensional image 116 corresponding to the lesioned part 106 is extracted from the three-dimensional virtual image.

  The CT image / ultrasonic image fusion unit 68 is extracted by the specific part image extraction unit 76 based on the observation position of the ultrasonic probe 32 estimated by the camera / ultrasonic observation position estimation unit 56. A three-dimensional image of the specific part is synthesized and displayed on the three-dimensional image of the luminal organ created by the luminal organ image creation unit 58. For example, the three-dimensional image 116 corresponding to the lesioned part 106 extracted as described above is synthesized and displayed at a position corresponding to the lesioned part 106 in the bronchial image 100. That is, in the present embodiment, the CT image / ultrasound image fusion unit 68 corresponds to a specific part image composition display unit. Here, when the ultrasonic image 108 is acquired, the protruding amount of the ultrasonic probe 32 is changed. Therefore, the relative positional relationship of the observation position by the ultrasonic probe 32 with respect to the three-dimensional image of the hollow organ created by the hollow organ image creation unit 58 also changes. Since the observation position of the ultrasonic probe 32 is associated with the three-dimensional image of the luminal organ by the camera / ultrasonic image observation position estimation unit 56, the reconstructed three-dimensional image of the specific part is displayed as the whole organ. Can be synthesized and displayed at a more accurate position in the three-dimensional image.

  The biopsy target site display unit 78 displays the specific site detected by the specific site detection unit 74 in the three-dimensional image of the hollow organ created by the luminal organ image creation unit 58. Display as. Preferably, the CT image / ultrasound image fusion unit 68 displays the three-dimensional image 116 corresponding to the lesioned portion 106 synthesized and displayed on the three-dimensional image of the luminal organ as a biopsy target site. The biopsy target site is, for example, a site to be subjected to biopsy with a treatment tool such as a needle inserted into a forceps opening provided in the tubular portion 22 of the endoscope body 20 or a biopsy forceps. Equivalent to. That is, it corresponds to a site where the needle is stabbed for tissue collection or a site grasped by a biopsy forceps. For example, when the subject is breathing in endoscopic examination of bronchi, the shape of the bronchus is slightly deformed according to the breathing. For this reason, when a biopsy target site is determined based on a three-dimensional image created based on three-dimensional image data of a luminal organ obtained in advance, the biopsy target site is a site where tissue should be collected (for example, , There is a risk of slight deviation from the lesioned part 106). In the present embodiment, a three-dimensional image 116 corresponding to the lesioned part 106 generated based on a plurality of ultrasonic images 108 continuously acquired by the ultrasonic probe 32 is displayed as a biopsy target site. Therefore, it is possible to present a more accurate position as a part where the needle is inserted for tissue collection or a part grasped by the biopsy forceps.

  As described above, the image generated and synthesized by the information processing unit 52 and the information fusion processing unit 54 is displayed on the video display device 14 via the information visualization unit 82. 10 to 15 illustrate an observation screen displayed on the video display device 14 via the information visualization unit 82. In the observation screen 118 shown in FIG. 10, the endoscope image acquired by the endoscope 26 and the image 120 including the path name of the endoscope 26, and the position and direction of the endoscope 26 are created. A three-dimensional virtual image 122 inside the luminal organ is displayed synchronously in one screen. In the observation screen 124 shown in FIG. 11, in addition to the images 120 and 122, an image 126 including the movement and path of the endoscope 26, the specific site (lesioned portion 106) as a target site, peripheral information, and the like is 1 Are displayed in sync on one screen. In the observation screen 128 shown in FIG. 12, in addition to the images 120 and 122, an image 130 including the movement and path of the endoscope 26, an ultrasonic image acquired by the ultrasonic probe 32, and the ultrasonic probe The ultrasonic image 132 acquired by 32 is displayed synchronously within one screen. In the observation screen 134 shown in FIG. 13, in addition to the images 120, 122, and 132, the internal view on a three-dimensional virtual image (CT image) constructed based on the three-dimensional image data of a hollow organ obtained in advance. An image 136 showing an observation position by the mirror 26 and the ultrasonic probe 32, a movement and a route of the endoscope 26, the specific part as a target part and peripheral information, an ultrasonic image acquired by the ultrasonic probe 32, and the like. The included image 138 is displayed in synchronization within one screen. In the observation screen 140 shown in FIG. 14, the images 120, 122, 132, 136, and 138 are displayed in a single screen in a different arrangement from the observation screen 134 shown in FIG. 13. In the observation screen 142 shown in FIG. 15, in addition to the images 120, 122, 132, 136, and 138, the specific region as a target region reconstructed from a plurality of ultrasonic images acquired by the ultrasonic probe 32 is displayed. A three-dimensional image 144 and an image 146 presenting a biopsy target site for the three-dimensional image reconstructed from the plurality of ultrasonic images are displayed in synchronization within one screen.

  FIG. 16 is a flowchart for explaining a main part of an example of conversion matrix determination control between the coordinate system corresponding to the endoscope 26 and the coordinate system corresponding to the magnetic position sensor 38 by the control device 12 of the support device 10. And is repeatedly executed at a predetermined cycle.

  First, in step (hereinafter, step is omitted) SA1, an endoscopic image acquired by the endoscope 26 and a detection result of the magnetic position sensor 38 are captured. Next, in SA2, the endoscopic image captured in SA1 is analyzed, and features in the endoscopic image are extracted. Next, in SA3, the initial value of the transformation matrix is set based on the detection result of the magnetic position sensor 38 captured in SA1 and the features in the endoscopic image extracted in SA2. Next, in SA4, a conversion error due to the conversion matrix set in SA3 is calculated. Next, in SA5, it is determined whether or not the conversion error calculated in SA4 is within a predetermined allowable error range. If the determination at SA5 is negative, the process after SA4 is executed again after the conversion matrix is changed at SA6. If the determination at SA5 is affirmative, the determination at SA7 is SA3. The transformation matrix set in step S6 or the transformation matrix changed in SA6 is determined as a transformation matrix between the coordinate system corresponding to the endoscope 26 and the coordinate system corresponding to the magnetic position sensor 38, This routine is terminated.

  FIG. 17 illustrates a main part of another example of conversion matrix determination control between the coordinate system corresponding to the ultrasonic probe 32 and the coordinate system corresponding to the magnetic position sensor 40 by the control device 12 of the support device 10. This flowchart is repeatedly executed at a predetermined cycle.

  First, in SB1, an endoscopic image acquired by the ultrasonic probe 32 and a detection result of the magnetic position sensor 40 are captured. Next, in SB2, the ultrasonic image captured in SB1 is analyzed, and features in the ultrasonic image are extracted. Next, in SB3, the initial value of the transformation matrix is set based on the detection result of the magnetic position sensor 40 captured in SB1 and the features in the ultrasonic image extracted in SB2. Next, in SB4, a conversion error due to the conversion matrix set in SB3 is calculated. Next, in SB5, it is determined whether or not the conversion error calculated in SB4 is within a predetermined allowable error range. If the determination at SB5 is negative, the process after SB4 is executed again after the conversion matrix is changed at SB6. However, if the determination at SB5 is affirmative, SB7 is changed to SB3. After the transformation matrix set in step SB6 or the transformation matrix changed in SB6 is determined as a transformation matrix between the coordinate system corresponding to the ultrasonic probe 32 and the coordinate system corresponding to the magnetic position sensor 40, This routine is terminated.

  FIG. 18 shows another example of conversion matrix determination control between the coordinate system corresponding to the three-dimensional virtual image and the coordinate system corresponding to the magnetic position sensors 38 and 40 by the control device 12 of the support device 10. FIG. 6 is a flowchart for explaining a section, which is repeatedly executed at a predetermined cycle.

  First, in SC1, a three-dimensional virtual image (CT image) generated based on previously obtained three-dimensional image data of a hollow organ and the detection results of the magnetic position sensors 38 and 40 are captured. Next, in SC2, the three-dimensional virtual image captured in SC1 is analyzed, and, for example, a bronchial core line (center line along the longitudinal direction of the bronchus) is extracted as a feature in the three-dimensional virtual image. Next, in SC3, the initial value of the transformation matrix is set based on the detection results of the magnetic position sensors 38 and 40 taken in in SC1 and the bronchial core line extracted in SC2. Next, in SC4, a conversion error based on the conversion matrix set in SC3 is calculated. Next, in SC5, it is determined whether or not the conversion error calculated in SC4 is within a predetermined allowable error range. If the determination at SC5 is negative, the process after SC4 is executed again after the conversion matrix is changed at SC6. If the determination at SC5 is affirmative, the process proceeds to SC3 at SC7. After the transformation matrix set in step S6 or the transformation matrix changed in SC6 is determined as a transformation matrix between the coordinate system corresponding to the three-dimensional virtual image and the coordinate system corresponding to the magnetic position sensors 38 and 40. This routine is then terminated.

  FIG. 19 is a flowchart for explaining a main part of an example of the endoscope observation support control by the control device 12 of the support device 10 and is repeatedly executed at a predetermined cycle.

  First, in S1, three-dimensional image data of a luminal organ obtained in advance is read from the image database 44, and a three-dimensional virtual image of the luminal organ is generated based on the three-dimensional image data. Next, in S <b> 2, an endoscopic image is acquired by the endoscope 26 and an ultrasonic image is acquired by the ultrasonic probe 32. Next, in S3, based on the transformation matrix determined in SA1 to SA7, SB1 to SB7, and SC1 to SC7, the endoscope is applied to the three-dimensional virtual image of the luminal organ generated in S1. 26 observation positions and the observation position of the ultrasonic probe 32 are associated with each other. Next, in S4, the endoscopic image acquired by the endoscope 26 on the three-dimensional virtual image of the luminal organ generated in S1 based on the result of the association in S3, the super image The ultrasonic image acquired by the sonic probe 32 is synthesized, and information such as the positions of the endoscope 26 and the tip of the ultrasonic probe 32 is presented and displayed on the video display device 14. Next, in S5, it is determined whether or not the presence of a lesion is detected as a specific site in the three-dimensional virtual image of the luminal organ generated in S1. When the determination at S5 is negative, the processing after S1 is executed again. However, when the determination at S5 is affirmative, the route information to the lesion part detected at S5 is obtained at S6. Created and presented on the three-dimensional virtual image of the luminal organ generated in S1. Next, in S7, it is determined whether or not the distal end position of the endoscope 26 (ultrasound probe 32) has reached the lesioned part detected in S5. While the determination of S7 is denied, the process waits by repeating the determination of S7. However, when the determination of S7 is affirmed, in S8, the ultrasonic probe 32 from the end 22a is While the protrusion amount is changed, a plurality of ultrasonic images at the observation position by the ultrasonic probe 32 are continuously acquired. Next, in S9, a three-dimensional image of the observation position by the ultrasonic probe 32 is reconstructed based on the plurality of ultrasonic images acquired in S8. Next, in S10, the three-dimensional image of the lesion detected in S5 is extracted from the three-dimensional image reconstructed in S9. Next, in S11, the three-dimensional image of the lesion extracted in S10 is synthesized at a position corresponding to the detected lesion on the three-dimensional virtual image of the luminal organ generated in S1. After being displayed on the video display device 14, this routine is terminated.

  In the above control, S2 and S8 are the operation of the ultrasonic image acquisition unit 48, S3 is the operation of the camera / ultrasonic image observation position estimation unit 56 as the observation position association unit, and S1 is the lumen organ. S4 is the operation of the image creation unit 58, S4 is the operation of the information fusion processing unit 62 as the image display control unit and the camera / probe position presentation unit 66 as the observation position display control unit, and S11 is the specific part image composition display unit. The operation of the CT image / ultrasound image fusion unit 68 as S9, the operation of the ultrasound image 3D reconstruction unit 72 as a 3D image reconstruction unit, and S10 of the specific part image extraction unit 76 It corresponds to each operation.

  Thus, according to the present embodiment, the ultrasonic probe 32 that can change the amount of protrusion from the end 22a as the base end, the distal end position of the endoscope 26, and the distal end position of the ultrasonic probe 32 are detected. Observation by the endoscope 26 in a three-dimensional image of the luminal organ constructed based on the three-dimensional image data based on the detection result by the sensor device 18 as the tip position detecting unit and the detection result by the sensor device 18 The camera / ultrasonic image observation position estimation unit 56 (S3) as an observation position correspondence unit for associating the position and the observation position by the ultrasonic probe 32, and the end 22a of the ultrasonic probe 32 from the end 22a. An ultrasonic image acquisition unit 48 (S8) for continuously acquiring a plurality of ultrasonic images at the observation position by the ultrasonic probe 32 while changing the protrusion amount, and the ultrasonic image An ultrasonic image three-dimensional reconstruction unit 72 (a three-dimensional image reconstruction unit that reconstructs a three-dimensional image of the observation position by the ultrasonic probe 32 based on a plurality of ultrasonic images acquired by the acquisition unit 48 ( S9), the correspondence relationship between the observation position by the ultrasonic probe 32 and the observation position by the endoscope 26 is changed, and further, the observation position of the ultrasonic probe 32 and the observation position are changed. Even when the correspondence relationship with the three-dimensional image of the hollow organ constructed based on the three-dimensional image data changes, the observation position by the ultrasonic probe 32, the observation position by the endoscope 26, and further, It is possible to appropriately grasp the correspondence relationship with the three-dimensional image of the luminal organ and reconstruct the three-dimensional image of the target part. That is, it is possible to provide the support apparatus 10 that effectively supports observation with the ultrasonic probe 32.

  A specific part image extracting unit 76 (S10) for extracting a three-dimensional image of a specific part at an observation position by the ultrasonic probe 32 based on a plurality of ultrasonic images acquired by the ultrasonic image acquiring unit 48; Based on the result of association by the camera / ultrasound image observation position estimation unit 56, a three-dimensional image of the specific part extracted by the specific part image extraction unit 76 is constructed based on the three-dimensional image data. Since it is provided with the CT image / ultrasound image fusion unit 68 (S11) as a specific part image synthesis display unit that is synthesized and displayed on the three-dimensional image of the luminal organ, it is acquired by the ultrasound probe 32. Based on the ultrasonic image, a three-dimensional image of a specific part including the lesioned part can be suitably synthesized and displayed on the three-dimensional image of the luminal organ to be observed.

  A three-dimensional image of the hollow organ constructed based on the three-dimensional image data, an endoscopic image acquired by the endoscope 26, and an ultrasonic image acquired by the ultrasonic probe 32 Since the information fusion processing unit 62 (S4) is provided as an image display control unit for synchronous display, a three-dimensional image, endoscopic image, and ultrasonic image of the luminal organ to be observed Practical navigation that displays can be realized.

  Based on the result of association by the camera / ultrasonic image observation position estimation unit 56, the observation position by the ultrasonic probe 32 on the three-dimensional image of the hollow organ constructed based on the three-dimensional image data. Since the camera / probe position presenting unit 66 (S4) is provided as an observation position display control unit for displaying the image, the observation position by the ultrasonic probe 32 is displayed on the three-dimensional image of the luminal organ to be observed. Practical navigation can be realized.

  The sensor device 18 detects the position of the distal end of the endoscope 26 based on the detection results of the magnetic position sensors 38 and 40 provided at the distal end of the endoscope 26 and the distal end of the ultrasonic probe 32, respectively. Since the tip position of the ultrasonic probe 32 is detected, the tip positions of the endoscope 26 and the ultrasonic probe 32 can be detected in a practical manner.

  The sensor device 18 detects the distal end position of the endoscope 26 based on the detection result of the magnetic position sensor 38 provided at the distal end portion of the endoscope 26, and the distal end position of the endoscope 26 and Since the tip position of the ultrasonic probe 32 is detected based on the detection result of the ultrasonic probe insertion length sensor 90 that detects the insertion length of the ultrasonic probe 32, the endoscope 26 is used in a practical manner. In addition, the tip position of the ultrasonic probe 32 can be detected.

  The sensor device 18 includes a detection result of an endoscope insertion length sensor 92 that detects the insertion length of the endoscope 26 and a detection result of an ultrasonic probe insertion length sensor 90 that detects the insertion length of the ultrasonic probe 32. Based on the above, the tip position of the endoscope 26 and the ultrasonic probe 32 are detected in a practical manner because the tip position of the endoscope 26 and the tip position of the ultrasonic probe 32 are detected. It can be detected.

  The preferred embodiments of the present invention have been described in detail with reference to the drawings. However, the present invention is not limited to these embodiments, and may be implemented in other modes.

  For example, in the above-described embodiment, the aspect in which the support device 10 supports the observation of the bronchus as a luminal organ has been described, but the present invention is not limited to this, and the support device 10 includes, for example, It is also preferably used for observing a three-dimensional virtual image corresponding to a luminal organ (hollow organ) such as the large intestine, small intestine, duodenum, stomach, bile duct, pancreatic duct, lymphatic duct and the like.

  In the above-described embodiment, the ultrasonic probe 32 includes the transmission / reception unit 34 at the tip thereof, and acquires an ultrasonic image corresponding to a plane perpendicular to the longitudinal direction of the transmission / reception unit 34. The ultrasonic probe provided in the present invention is not limited to such an embodiment. For example, the present invention is also suitably applied to an endoscope observation support apparatus that includes an ultrasonic probe that emits ultrasonic waves from a side (side) in the longitudinal direction and receives reflected waves at the side. Is done. The endoscope 26 is provided with a single observation part 28 at its distal end, but in addition to the end 22a of the tubular part 22 or a single or a plurality of observations in the middle except for the end 22a. The part 28 may be provided.

  In the above-described embodiment, the display mode in which the images 120, 122, etc. are displayed in synchronization with one screen of the video display device 14 on the observation screen 118 has been described, but a plurality of video display devices 14 are provided. Of course, the above-mentioned 120, 122, etc. may be displayed in synchronism with different screens (different video display devices).

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

  DESCRIPTION OF SYMBOLS 10: Endoscope observation assistance apparatus, 18: Sensor apparatus (front-end | tip position detection part), 22a: End part (base end part), 26: Endoscope, 32: Ultrasonic probe, 38, 40: Magnetic position sensor 48: Ultrasonic image acquisition unit 56: Camera / ultrasonic image observation position estimation unit (observation position correspondence unit) 62: Information fusion processing unit (image display control unit) 66: Camera / probe position presentation unit ( (Observation position display control unit), 68: CT image / ultrasound image fusion unit (specific site image synthesis display unit), 72: ultrasound image 3D reconstruction unit (3D image reconstruction unit), 76: specific site image Extraction unit, 90: ultrasonic probe insertion length sensor, 92: endoscope insertion length sensor, 100: image (three-dimensional image), 104: endoscopic image, 106: lesion (specific site), 108: ultrasound image

Claims (7)

An endoscopic observation support device that supports observation with an endoscope inserted into a hollow organ based on three-dimensional image data of the hollow organ obtained in advance,
An ultrasonic probe that can change the amount of protrusion from the base end, and
A tip position detector for detecting a tip position of the endoscope and a tip position of the ultrasonic probe;
Correspondence between the observation position by the endoscope and the observation position by the ultrasonic probe in the three-dimensional image of the hollow organ constructed based on the three-dimensional image data based on the detection result by the tip position detection unit An observation position corresponding part for attaching,
An ultrasonic image acquisition unit for continuously acquiring a plurality of ultrasonic images at the observation position by the ultrasonic probe while changing the amount of protrusion from the base end in the ultrasonic probe;
A three-dimensional image reconstruction unit that reconstructs a three-dimensional image of the observation position by the ultrasonic probe based on a plurality of ultrasonic images acquired by the ultrasonic image acquisition unit. Endoscope observation support device. A specific part image extracting unit that extracts a three-dimensional image of a specific part at an observation position by the ultrasonic probe based on a plurality of ultrasonic images acquired by the ultrasonic image acquiring unit;
Based on the result of the association by the observation position association unit, the three-dimensional image of the specific part extracted by the specific part image extraction unit is created based on the three-dimensional image data. The endoscopic observation support apparatus according to claim 1, further comprising: a specific part image composition display unit that performs composition display on a three-dimensional image.   A three-dimensional image of the hollow organ constructed based on the three-dimensional image data, an endoscopic image acquired by the endoscope, and an ultrasonic image acquired by the ultrasonic probe are synchronized. The endoscopic observation support apparatus according to claim 1, further comprising an image display control unit to be displayed.   Observation position display for displaying the observation position by the ultrasonic probe on the three-dimensional image of the hollow organ constructed based on the three-dimensional image data based on the result of the association by the observation position association unit The endoscope observation support apparatus according to any one of claims 1 to 3, further comprising a control unit.   The tip position detection unit is configured to detect the tip position of the endoscope and the ultrasound probe based on detection results of a magnetic position sensor provided at the tip of the endoscope and the tip of the ultrasound probe, respectively. The endoscope observation support apparatus according to any one of claims 1 to 4, wherein the tip position of the endoscope is detected.   The tip position detection unit detects a tip position of the endoscope based on a detection result of a magnetic position sensor provided at a tip portion of the endoscope, and the tip position of the endoscope and the ultrasonic wave The endoscope observation support according to any one of claims 1 to 4, wherein a tip position of the ultrasonic probe is detected based on a detection result of an ultrasonic probe insertion length sensor for detecting an insertion length of the probe. apparatus.   The tip position detection unit is based on a detection result of an endoscope insertion length sensor that detects an insertion length of the endoscope and a detection result of an ultrasonic probe insertion length sensor that detects an insertion length of the ultrasonic probe. The endoscope observation support apparatus according to any one of claims 1 to 4, which detects a tip position of the endoscope and a tip position of the ultrasonic probe.