JP6206869B2 - Endoscopic observation support device - Google Patents

Description

  The present invention relates to an endoscopic observation support apparatus that supports observation using an endoscope inserted into a hollow organ based on three-dimensional image data of the hollow organ, and more particularly to a hollow organ such as a large intestine. An object of the present invention is to provide an endoscope observation support device that supports observation.

  In recent years, in the medical field, diagnosis and examination using an image of a subject obtained by an X-ray CT (Computed Tomography) apparatus, a magnetic resonance (MRI) apparatus, or the like has been widely performed. For example, CT colonography (Computed Tomography Colonography), which is a colorectal polyp inspection method using three-dimensional CT images, is becoming popular. In this CT colonography, by detecting the protrusion shape from the CT image of the large intestine, a site where a lesion such as a large intestine polyp is suspected is found. When a suspicious site is found, a colonoscopy is performed in which an endoscope is inserted through the anus.

  However, it is difficult to identify a site suspected of having a lesion such as a polyp discovered in an examination based on three-dimensional image data such as CT colonography in a subsequent endoscopic examination. For this reason, in endoscopy, endoscopic navigation for guiding the endoscope to a site where a lesion found in the examination based on the three-dimensional image data is suspected is required. In conventional endoscope navigation, position information indicating which part of a CT image corresponding to the tip position of the endoscope corresponds to position information is obtained, and navigation information is generated based on the obtained position information. . For example, this is the technique described in Non-Patent Document 1 and the like. In this technique, a position sensor is provided at a predetermined site in an endoscope, and in an examination based on three-dimensional image data based on information obtained from the position sensor in an endoscopic examination using the endoscope. Endoscopic navigation is performed to guide the endoscope to a site where a detected lesion is suspected.

Shingo Tsuji, Yukitaka Fumimura, Takayuki Kitasaka, Yuichiro Hayashi, Eiji Ito, Masaharu Fujii, Tetsuya Nagatani, Taiichi Wadabayashi, Toshihiko Wakabayashi, Kensaku Mori, ?? A simple correction method for endoscope axis rotation error, “JSCAS, Vol.12, No.2, pp.65-77 (2010/06) Daisuke Deguchi, Kenta Akiyama, Kensaku Mori, Takayuki Kitasaka, Yasuhito Suenaga, Calvin R Maurer Jr, Hirotsugu Takabatake, Masaki Mori, Hiroshi Natori,? A method for bronchoscope tracking by combining a position sensor and image registration, ″ Computer Aided Surgery, Vol .11, No.3, pp.109-117 (2006/05)

  However, although the above-mentioned conventional technique can perform guidance of an endoscope well with respect to organs with relatively little deformation such as the brain, otonasal cavity, and bronchi, it relates to easily deformable luminal organs such as the large intestine. Had a problem that it was difficult to guide the endoscope. That is, in an endoscopic examination related to a deformable luminal organ such as the large intestine, the shape or length of the luminal organ changes variously during the examination. For this reason, it is difficult to guide the endoscope to a site where a lesion found in the examination based on the three-dimensional image data is suspected by simply using the conventional position sensor. In other words, endoscope observation support devices that support observation of easily deformable luminal organs such as the large intestine have not yet been developed, and for endoscopic navigation of such easily deformable luminal organs. There have been no studies on suitable image display methods.

  The present invention has been made against the background of the above circumstances, and an object of the present invention is to provide an endoscope observation support apparatus that supports observation of a luminal organ such as the large intestine.

In order to achieve such an object, the gist of the first aspect of the present invention is that observation based on a three-dimensional image data of a hollow organ obtained in advance by an endoscope inserted into the hollow organ is provided. A virtual core calculation unit that calculates a virtual core along the longitudinal direction of the luminal organ based on the three-dimensional image data, and is inserted into the luminal organ An endoscope line calculation unit for calculating an endoscope line indicating the shape of the endoscope; the virtual core line calculated by the virtual core line calculation unit; and the endoscope calculated by the endoscope line calculation unit. A matching processing unit for deriving a correspondence relationship with an endoscope line, an observation position estimation unit for estimating an observation position of the endoscope in the lumen organ based on the correspondence relationship derived by the matching processing unit , The tube based on the three-dimensional image data A virtual developed image display control unit that displays a virtual developed image of an organ and displays the observation position of the endoscope estimated by the observation position estimating unit on the virtual developed image of the luminal organ, and The virtual developed image display control unit includes a predetermined range including the observation position of the endoscope in the virtual developed image of the luminal organ based on an error of the observation position of the endoscope estimated by the observation position estimation unit. The display of is different from the other parts .

As described above, according to the first aspect of the present invention, the virtual core calculation unit that calculates the virtual core along the longitudinal direction of the hollow organ based on the three-dimensional image data, and the virtual core calculation unit inserted into the hollow organ An endoscope line calculation unit that calculates an endoscope line indicating the shape of the endoscope, the virtual core line calculated by the virtual core line calculation unit, and the endoscope calculated by the endoscope line calculation unit A matching processing unit for deriving a correspondence relationship with a mirror line, and an observation position estimation unit for estimating the observation position of the endoscope in the lumen organ based on the correspondence relationship derived by the matching processing unit, Even if it is a easily deformable luminal organ such as the large intestine, the endoscope is matched by matching the virtual core wire of the luminal organ and the endoscope line indicating the shape of the endoscope. Observation support by . That is, it is possible to provide an endoscope observation support apparatus that supports observation of a luminal organ such as the large intestine. Further, a virtual development image of the luminal organ is displayed based on the three-dimensional image data, and the observation position of the endoscope estimated by the observation position estimation unit is displayed on the virtual development image of the luminal organ. Therefore, practical endoscope navigation for displaying a virtual developed image of a luminal organ corresponding to an observation position by the endoscope can be realized. Further, the virtual developed image display control unit determines the observation position of the endoscope in the virtual developed image of the luminal organ based on an error in the observation position of the endoscope estimated by the observation position estimation unit. Since the display of the predetermined range including the display is different from the other parts, it is possible to realize endoscope navigation that makes it possible to display a virtual developed image of the luminal organ corresponding to the observation position by the endoscope more easily. .

  The gist of the second invention subordinate to the first invention is that the coordinate system corresponding to the virtual core wire calculated by the virtual core wire calculating unit and the inner part calculated by the endoscope line calculating unit are used. A coordinate system registration processing unit that performs association with a coordinate system corresponding to an endoscope line, and the matching processing unit, based on a result of association by the coordinate system registration processing unit, The correspondence relationship with the endoscope line is derived. In this way, it is possible to derive the correspondence between the virtual core line of the luminal organ and the endoscope line indicating the shape of the endoscope in a preferable and practical manner.

  The gist of the third invention subordinate to the first invention or the second invention is that the matching processing unit detects a plurality of landmarks for the virtual core wire calculated by the virtual core wire calculation unit, and detects them. The correspondence relationship between the virtual core line and the endoscope line is derived based on a plurality of landmarks. In this way, it is possible to derive the correspondence between the virtual core line of the luminal organ and the endoscope line indicating the shape of the endoscope in a preferable and practical manner.

  The gist of the fourth invention according to any one of the first to third inventions is that the image of the luminal organ is displayed based on the three-dimensional image data and is estimated by the observation position estimating unit. And an organ image display control unit for displaying the observation position of the endoscope on the image of the luminal organ. In this way, it is possible to realize practical endoscope navigation in which the observation position by the endoscope is displayed on the image of the luminal organ to be observed.

  The gist of the fifth invention according to any one of the first to fourth inventions is that the observation position of the endoscope estimated by the observation position estimation unit based on the three-dimensional image data. And a virtual endoscopic image display control unit for displaying a virtual endoscopic image of the luminal organ corresponding to. In this way, practical endoscope navigation that displays a virtual endoscopic image of a luminal organ corresponding to an observation position by an endoscope can be realized.

  The gist of the sixth invention that is dependent on the fifth invention is that the virtual endoscopic image display control unit is estimated by the observation position estimating unit based on the three-dimensional image data. A global virtual endoscopic image of the luminal organ corresponding to the observation position of the mirror is displayed. In this way, it is possible to realize practical endoscope navigation for displaying a global virtual endoscopic image of a luminal organ corresponding to an observation position by an endoscope.

The subject matter of the seventh invention, which is dependent on any one of the first to sixth inventions, is a lesion presence detector that detects the presence of a lesion in the luminal organ based on the three-dimensional image data, A lesion detection result display control unit for displaying a detection result by the lesion presence detection unit. In this way, it is possible to realize practical endoscope navigation that displays the presence or absence of a lesion in a luminal organ that is an object to be observed by an endoscope.

The gist of the eighth invention subordinate to the seventh invention is that the observation position of the endoscope estimated by the observation position estimation unit has reached the vicinity of the lesion detected by the lesion presence detection unit. In such a case, a lesion notification control unit that performs notification to that effect is provided. In this way, when the observation position by the endoscope reaches the vicinity of the lesion, it is possible to realize practical endoscope navigation that notifies that effect.

The gist of the ninth invention is that the image of the luminal organ displayed by the organ image display control unit of the fourth invention, the virtual endoscopic image display control unit of the fifth invention or the sixth invention. Displaying at least one of the displayed virtual endoscopic image of the luminal organ and the virtual development image of the luminal organ displayed by the virtual development image display control unit of the first invention; lesion detection result display control unit of the seventh invention is the presence of lesions detected by the lesion-existence detection portion of the seventh invention, the luminal organ iconography, virtual endoscopic image of the luminal organ, and the tube It is displayed on a site corresponding to the detected lesion in at least one of the virtual developed images of the hollow organ. In this way, the image of the luminal organ corresponding to the observation position of the endoscope, the virtual endoscopic image, the virtual developed image, and the like are displayed, and the detected lesion site is displayed on each image. Thus, practical and easy-to-understand endoscope navigation can be realized.

The gist of the tenth aspect of the present invention dependent on the third aspect is that the matching processing unit detects a plurality of landmarks in the virtual core line based on the shape of the virtual core line, and detects the detected virtual core line. The correspondence relationship between the virtual core line and the endoscope line is derived based on the distance from the plurality of landmarks to the endoscope line. In this way, it is possible to derive the correspondence between the virtual core line of the luminal organ and the endoscope line indicating the shape of the endoscope in a preferable and practical manner.

The gist of the eleventh invention subordinate to the third invention or the tenth invention is that the matching processing unit is configured such that each point on the virtual core line and each point on the endoscope line between the landmarks. Is derived by linear interpolation. In this way, it is possible to derive the correspondence between the virtual core line of the luminal organ and the endoscope line indicating the shape of the endoscope in a preferable and practical manner.

The gist of the twelfth invention dependent on any of the third invention, the tenth invention, and the eleventh invention is that the matching processing unit is included in the inner portion included between the adjacent landmarks. Each time the length of the endoscope line is changed, the correspondence relationship between the virtual core line and the endoscope line is updated. In this way, each time the endoscope is pulled out or inserted from the luminal organ, the correspondence between the virtual core line of the luminal organ and the endoscope line indicating the shape of the endoscope is changed. Derived in real time.

It is a figure which illustrates the structure of the endoscope observation assistance apparatus which is a suitable Example of this invention. It is a figure which illustrates roughly the structure of the endoscope with which the endoscope observation assistance apparatus of FIG. 1 was equipped. It is a functional block diagram explaining the principal part of an example of the control function with which the endoscope observation assistance apparatus of FIG. 1 was equipped. FIG. 3 is a front view illustrating a large intestine phantom that is a model of a large intestine that is a suitable observation target by the endoscope of FIG. 2, and shows a state before insertion of the endoscope. FIG. 3 is a front view illustrating a large intestine phantom which is a model of a large intestine that is a suitable observation target by the endoscope of FIG. 2, and shows a state in which the endoscope is inserted. It is an example of the three-dimensional image obtained by X-ray CT regarding the large intestine phantom of the state shown in FIG. The virtual core wire calculated corresponding to the three-dimensional image of FIG. 6 is illustrated. It is a figure which shows the large intestine CT image which is an example of the three-dimensional image of the large intestine which is a luminal organ. The line figure created by applying a well-known thinning algorithm to the large intestine CT image shown in FIG. 8 is illustrated. The line figure from which the false branch was deleted from the line figure shown in FIG. 9 is illustrated. The mode that several points were acquired at equal intervals on the line figure shown in FIG. 10 is illustrated. 11 illustrates a virtual core line obtained by a known cubic spline interpolation with respect to a plurality of points shown in FIG. It is a figure explaining the specific example of calculation of the endoscope line by CPU of the endoscope observation assistance apparatus of FIG. It is a figure explaining the linearization of the predetermined location in the virtual core wire computed by CPU of the endoscope observation assistance apparatus of FIG. It is a figure explaining the specific example of ICP registration by CPU of the endoscope observation assistance apparatus of FIG. 1, and has illustrated before matching of a coordinate system is performed. It is a figure explaining the specific example of ICP registration by CPU of the endoscope observation assistance apparatus of FIG. 1, and has shown the state by which matching of the coordinate system was performed. It is a figure explaining the detection of the landmark about the virtual core wire by the CPU of the endoscope observation assistance apparatus of FIG. 1, and derivation | leading-out of the corresponding point in an endoscope line. It is a figure explaining the process which derives | leads-out the correspondence relationship between each point on a virtual core line and each point on an endoscope line by linear interpolation by CPU of the endoscope observation assistance apparatus of FIG. It is a figure which illustrates the endoscope navigation screen displayed on an image | video display apparatus in the case of observation assistance control by the endoscope observation assistance apparatus of FIG. It is a figure which illustrates the global virtual endoscopic image of a luminal organ displayed by CPU of the endoscope observation assistance apparatus of FIG. It is a figure which shows another example of the endoscope navigation screen displayed on an image | video display apparatus in the case of the observation assistance control by the endoscope observation assistance apparatus of FIG. 3 is a flowchart for explaining a main part of an example of endoscope navigation control by a CPU of the endoscope observation support apparatus of FIG. 1. It is a flowchart explaining the principal part of an example of the image display process in control of FIG.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating the configuration of an endoscope observation support apparatus 10 (hereinafter simply referred to as support apparatus 10) that is an embodiment of the present invention. As shown in FIG. 1, the support device 10 of this embodiment includes a CPU 12 that is a central processing unit, a ROM 14 that is a read-only memory, a RAM 16 that is a write / read memory as needed, a storage device 18, and an image. A display device 20, an input device 22, and an endoscope 24 are provided. The CPU 12 is a so-called microcomputer that processes and controls electronic information based on a predetermined program stored in advance in the ROM 14 while using the temporary storage function of the RAM 16.

  The storage device 18 is preferably a known storage device (storage medium) such as a hard disk drive capable of storing information. It may be a removable storage medium such as a small memory card or an optical disk. The video display device 20 is preferably a known display device (display) such as a TFT (Thin Film Transistor Liquid Crystal), and is connected to the CPU 12 or the like via an output interface 26. The input device 22 is a known input device such as a keyboard or a mouse, and is connected to the CPU 12 or the like via an input interface 28.

  The endoscope 24 is a known electronic scope or fiberscope used for observing a hollow organ, and preferably a colonoscope mainly used for observing the large intestine. The endoscope 24 is connected to the CPU 12 and the like via an interface 30. FIG. 2 is a diagram schematically illustrating the configuration of the endoscope 24. As shown in FIG. 2, the endoscope 24 is provided at a bendable (flexible) tubular portion 32 formed in a long tubular shape and at one end portion (tip portion) of the tubular portion 32. Provided with an observation part 34, an operation part 36 provided at the end of the tubular part 32 opposite to the observation part 34, and a connection cable 38 provided extending from the operation part 36. Configured.

  The tubular portion 32 is formed in a long tubular shape having a length dimension of about 1.2 to 1.3 m and a diameter dimension of about 8 to 13 mm, for example, by an angle (knob) (not shown) provided in the operation section 36. According to the operation, the direction in which the distal end portion provided with the observation unit 34 is directed can be freely changed. In the vicinity of the observation part 34 in the tubular part 32, a light source for illuminating an observation part by the observation part 34 in a hollow organ is provided. When the endoscope 24 is a fiberscope, another optical fiber for guiding illumination light into the luminal organ is incorporated in the tubular portion 32. The tubular portion 32 preferably incorporates a mechanism for taking in and out the biopsy forceps, a pipe for sending and sucking air and water, and the like.

  The observation unit 34 corresponds to an image (subject image) acquisition unit in the endoscope 24 and is provided at the distal end of the tubular portion 32. When the endoscope 24 is an electronic scope, the observation unit 34 corresponds to an imaging element such as a CCD (Charge Coupled Device). When the endoscope 24 is a fiberscope, the observation unit 34 corresponds to a lens unit for taking the image into a glass fiber (optical fiber) provided in the tubular unit 32.

  The connection cable 38 is a cable for connecting the endoscope 24 to the main body of the endoscope support apparatus 10, and a connection terminal is provided at an end of the connection cable 38 opposite to the operation unit 36. 40 is provided. By connecting the connection terminal 40 to a corresponding connection terminal provided in the main body of the endoscope support apparatus 10, the endoscope 24 is connected to the CPU 12 or the like via the interface 30.

  In the endoscope 24 of the present embodiment, preferably, a plurality (four in the present embodiment) of position sensors 42a, 42b, 42c, 42d (hereinafter simply referred to as the position sensor 42 unless otherwise distinguished) are provided. The tubular portion 32 is provided. In FIG. 2, the position where each position sensor 42 is provided is indicated by a white arrow. As shown in FIG. 2, the plurality of position sensors 42 are preferably arranged at a predetermined interval (for example, an interval of 20 cm) from the distal end portion (the side where the observation portion 34 is provided) of the tubular portion 32. ing. The position sensor 42 a provided at the distal end portion of the tubular portion 32 is preferably attached in the instrument insertion channel of the endoscope 24. The plurality of position sensors 42 are preferably built in the tubular portion 32, but may be fixed (externally attached) to the side surface of the tubular portion 32 or the like. The plurality of position sensors 42 are preferably known magnetic position sensors. For example, a transmitter (not shown) is provided separately from each position sensor 42, and a magnetic signal transmitted from the transmitter is transmitted to each position. By receiving the sensor 42, five-dimensional information, that is, three-dimensional position information and two-dimensional direction information, is acquired from each position sensor 42 and supplied to the CPU 12 or the like.

  As shown in FIG. 1, an image database 44 is provided in the storage device 18 provided in the support device 10. The image database 44 stores three-dimensional image data of a subject obtained in advance by a known X-ray CT (Computed Tomography) apparatus, magnetic resonance (MRI) apparatus, or the like. In the case where the three-dimensional image data is obtained by X-ray CT, X-rays are irradiated to the subject from multiple directions, the X-rays transmitted through the subject are detected by a detector, and transmitted X The dose information is processed by a computer and reconstructed as a three-dimensional image. The image database 44 stores, for example, a three-dimensional grayscale image of a subject obtained by X-ray CT, a magnetic resonance apparatus, or the like, in other words, a pixel value (density value) for each voxel that is a unit volume of the subject. Is done. The image database 44 stores three-dimensional image data of a luminal organ obtained by X-ray CT, a magnetic resonance apparatus, or the like. Preferably, three-dimensional image data of the large intestine obtained by X-ray CT, a magnetic resonance apparatus or the like is stored.

  The support apparatus 10 of the present embodiment is configured so that the endoscope inserted into the hollow organ based on the three-dimensional image data obtained in advance by an X-ray CT, a magnetic resonance apparatus, or the like regarding the hollow organ of the subject. Control to support observation by the mirror 24 is performed. Preferably, based on the three-dimensional image data of the large intestine, control is performed to support observation with the endoscope 24 inserted into the large intestine. That is, when observing a predetermined large intestine with the endoscope 24, three-dimensional image data obtained in advance corresponding to the large intestine and stored in the image database 44 is read out, and described later based on the three-dimensional image data. By performing the control, observation by the endoscope 24 inserted into the large intestine is supported.

  4 and 5 are front photographs illustrating a large intestine phantom 80, which is a model of the large intestine that is a suitable observation target by the endoscope 24, and FIG. 4 shows the endoscope 24 inserted therein. FIG. 5 shows a state where the endoscope 24 is inserted. In FIG. 5, the endoscope 24 (tubular portion 32) inserted into the large intestine phantom 80 is indicated by a broken line. FIG. 6 shows an example of a three-dimensional image 82 obtained by X-ray CT for the large intestine phantom 80 in the state shown in FIG. As shown in comparison with FIGS. 4 and 5, in an endoscopic examination for a deformable luminal organ such as the large intestine, the shape and length of the luminal organ changes variously during the examination. To do. That is, by inserting the endoscope 24 into the large intestine as shown in FIG. 4, the curved portion of the sigmoid colon or the transverse colon is deformed along the bend of the tubular portion 32 of the endoscope 24. The wrinkles at that portion become dense and the shape in front view changes greatly as shown in FIG. Therefore, even when a site suspected of having a large intestine lesion is found based on image data corresponding to the three-dimensional image 82 as shown in FIG. It is difficult to display an endoscopic image in association with an actual endoscopic image or to guide the endoscope 24 to a site where the lesion is suspected. The support device 10 according to the present embodiment is based on the above problems, and will be described in detail below so that the observation can be favorably supported even in an endoscopic examination for a deformable luminal organ such as the large intestine. Control to do.

  FIG. 3 is a functional block diagram illustrating a main part of an example of a control function provided in the support device 10. The virtual core calculation unit 50, the endoscope line calculation unit 52, the matching processing unit 54, the observation position estimation unit 60, the lesion presence detection unit 62, the lesion notification control unit 64, and the image display control unit 66 shown in FIG. Preferably, all of them are functionally provided in the CPU 12, but each control unit may be individually provided in a plurality of CPUs or computers and configured to be able to communicate with each other. . Further, a part of each control unit may be provided in a control device different from the support device 10. For example, the lesion notification control unit 64 and the image display control unit 66 may be provided in a device different from the support device 10 (for example, a display-dedicated device).

  The virtual core calculation unit 50 calculates a virtual core along the longitudinal direction of the luminal organ corresponding to the three-dimensional image data based on the three-dimensional image data stored in the image database 44. This virtual core line corresponds to a rough center line of the hollow organ corresponding to the three-dimensional image data. The virtual core calculation unit 50 extracts, for example, a target luminal organ region from the three-dimensional image data, and the virtual skeleton representing a rough center line of the luminal organ from the extracted luminal organ region. Is calculated.

FIG. 8 is a view showing a large intestine CT image 84 which is an example of a three-dimensional image of the large intestine which is a luminal organ. A specific example of the derivation of the virtual core line by the virtual core line calculation unit 50 for the three-dimensional image data corresponding to the large intestine CT image 84 shown in FIG. 8 will be described with reference to FIGS. First, based on the three-dimensional image data corresponding to the large intestine CT image 84 shown in FIG. 8, the large intestine region (large intestine region) corresponding to the three-dimensional image data is extracted by a known region expansion method. For example, a portion having a CT value of −600 HU or less in the three-dimensional image data is extracted as a large intestine region. Next, a line figure passing near the center of the extracted large intestine region is created. For example, by applying a known thinning algorithm to the extracted large intestine region, a line figure passing through the vicinity of the center is created. FIG. 9 exemplifies a line figure 86 created by such processing. Next, the false branch in the created line figure 86 is deleted. That is, the relatively short line figure branched from the main line figure (curve) that continuously extends the longest is deleted. FIG. 10 illustrates a line figure 88 from which false branches have been deleted by such processing. Next, a plurality of points are acquired at equal intervals on the line figure 88 from which the false branches have been deleted. FIG. 11 illustrates a plurality of points 90 obtained by such processing. Next, a smooth virtual core line is calculated for the obtained plurality of points 90 by known cubic spline interpolation. That is, a virtual core 92 as shown in FIG. 12 is calculated using the acquired plurality of points 90 as control points for cubic spline interpolation. The virtual core wire 92 calculated as described above is expressed as, for example, a point sequence p ^C _iC (i ^C = 1,..., I ^C ).

  The endoscope line calculation unit 52 calculates an endoscope line indicating the shape of the endoscope 24 inserted into the lumen organ. This endoscope line preferably corresponds to a rough center line of the tubular portion 32 of the endoscope 24. That is, the endoscope line calculation unit 52 calculates an endoscope line indicating the shape of the tubular portion 32 of the endoscope 24 inserted into the luminal organ. Preferably, the endoscope line is calculated by known Hermite spline interpolation based on position information and direction information detected by each of the plurality of position sensors 42. According to Hermite spline interpolation, an interpolation curve reflecting the position and direction at the control point can be created. Preferably, the endoscope line calculation unit 52 performs Hermite spline interpolation based on position information and direction information detected by each position sensor 42, using detection results of the plurality of position sensors 42 as control points. . More preferably, the position of the anus of the subject detected from the three-dimensional image data is added to the control point in the Hermitian spline interpolation. More preferably, the parameters for the Hermite spline interpolation are determined in consideration of the properties (e.g., ease of bending) of the tubular portion 32 obtained experimentally in advance.

FIG. 13 is a diagram for explaining a specific example of calculation of an endoscope line by the endoscope line calculation unit 52. First, position information and direction information detected by each of the plurality of position sensors 42 are acquired, and a plurality of control points 94a, 94b, 94c, 94d (hereinafter, unless otherwise distinguished) as shown in FIG. Is simply referred to as a control point 94). In the example shown in FIG. 13, the control point 94a is the detection result of the position sensor 42a, the control point 94b is the detection result of the position sensor 42b, and the control point 94c is the detection result of the position sensor 42c. The control points 94d correspond to the detection results of the position sensor 42d, respectively. Next, Hermite spline interpolation is performed on the plurality of control points 94a to 94d. In this Hermite spline interpolation, preferably, the properties of the tubular portion 32 obtained experimentally in advance are taken into consideration. FIG. 13B illustrates a spline interpolation curve 96 obtained by such processing. Next, a control point 94e indicating the position of the anus of the subject detected from the three-dimensional image data is added, and Hermite spline interpolation is performed on the plurality of control points 94a to 94e. By such processing, an endoscope line 98 as shown in FIG. 13C representing the shape of the endoscope 24 is calculated. The endoscope line 98 calculated as described above is expressed as, for example, a point sequence p ^S _iS (i ^S = 1,..., I ^S ).

  The matching processing unit 54 derives a correspondence relationship between the virtual core line calculated by the virtual core line calculation unit 50 and the endoscope line calculated by the endoscope line calculation unit 52. In other words, matching is performed between a virtual core line along the longitudinal direction of the hollow organ and an endoscope line indicating the shape of the endoscope inserted into the hollow organ. Preferably, in the processing relating to the large intestine of the subject, the endoscope line and the virtual core line are targeted for an endoscope line in a state where the distal end portion (observation unit 34) of the endoscope 24 is inserted up to the cecum. The correspondence relationship is derived. In order to perform such processing, as shown in FIG. 3, the matching processing unit 54 includes a coordinate system registration processing unit 56 and a line registration processing unit 58. FIG. 7 illustrates a virtual core 100 calculated by the virtual core calculator 50 corresponding to the three-dimensional image 82 of FIG. Hereinafter, specific examples of processing of the coordinate system registration processing unit 56 and the line registration processing unit 58 for deriving the correspondence relationship between the virtual core line 100 and the endoscope line 98 illustrated in FIG. explain.

  The coordinate system registration processing unit 56 corresponds to the coordinate system corresponding to the virtual core line calculated by the virtual core line calculation unit 50 and the endoscope line calculated by the endoscope line calculation unit 52. Performs association with the coordinate system. Since the three-dimensional image and the position sensor 42 have different coordinate systems, the virtual core line calculated by the virtual core line calculation unit 50 and the endoscope line calculated by the endoscope line calculation unit 52 are used. Each has a different coordinate system. Accordingly, in deriving the correspondence between the virtual core line and the endoscope line, it is necessary to associate the coordinate systems thereof. The coordinate system registration processing unit 56 preferably associates the coordinate system corresponding to the virtual core line with the coordinate system corresponding to the endoscope line by ICP registration using a known ICP algorithm. . This ICP registration is detailed in “Paul J. Besl and Neil D. McKay,“ A method for registration of 3-D shapes, ”IEEE Trans. PAMI, vol. 14, no. 2, 1992.”.

In the ICP registration, preferably, the virtual core line 100 and the endoscope line 98 are each represented by a point sequence. For example, the virtual core line 100 is represented as a point sequence p ^C _iC (i ^C = 1,..., I ^C ), and the endoscope line 98 is represented by a point sequence p ^S _iS (i ^S = 1 _,. , I ^S ). Next, rotation and translation are applied to the point sequence corresponding to the endoscope line 98 to bring it close to the point sequence corresponding to the virtual core line 100. Then, a transformation matrix related to the point sequence of the endoscope line 98 is calculated from the rotation and translation when the distance between the two point sequences is the smallest.

  The coordinate system registration processing unit 56 preferably performs the ICP registration after linearizing a core wire at a predetermined location in the virtual core wire 100. FIG. 14 is a diagram for explaining linearization of a predetermined portion in the virtual core wire 100. The coordinate system registration processing unit 56 preferably linearizes a predetermined core line that is largely bent in the virtual core line 100 as pre-processing for the ICP registration. Preferably, two core lines corresponding to the transverse colon and the sigmoid colon in the virtual core line 100 are linearized. For example, in the example shown in FIG. 14, the core wire 102 corresponding to the transverse colon and the core wire 104 corresponding to the sigmoid colon surrounded by a broken line are respectively linearized. The drawing on the right side of the drawing shows a virtual core wire 100 in which the core wires 102 and 104 at the two locations are straightened (the core wire before straightening is indicated by a one-dot chain line). When the endoscope 24 is inserted into the large intestine of the subject, the transverse colon and the sigmoid colon are greatly bent, and the endoscope 24 is inserted as shown in comparison with FIGS. Then, the bending is extended and the shape is changed to a shape close to a straight line when viewed from the front. On the other hand, the remaining part of the large intestine, that is, the descending colon, the ascending colon, and the like originally have a shape that is almost straight, and even when the endoscope 24 is inserted, the shape does not change greatly. Therefore, based on the characteristics of large intestine deformation in such colonoscopy, the core wire 102 corresponding to the transverse colon and the core wire 104 corresponding to the sigmoid colon are respectively linearized, thereby making correspondence by the ICP registration. More favorable results are obtained.

  The coordinate system registration processing unit 56 preferably performs the ICP registration by associating end points (points at both ends) of the virtual core line 100 and the endoscope line 98 with each other. Here, when the large intestine is an observation target, the points at both ends of the virtual core wire 100 preferably correspond to the cecum and anus. In a state where the endoscope 24 is inserted up to the cecum in the large intestine, preferably, the points at both ends of the endoscope line 98 correspond to the cecum and anus as well. Therefore, the coordinate system registration processing unit 56 preferably performs the ICP registration by associating points corresponding to the cecum and anus at both ends of the virtual core line 100 and the endoscope line 98 with each other.

  15 and 16 are diagrams for explaining a specific example of ICP registration by the coordinate system registration processing unit 56. FIG. 15 shows the coordinate system of the virtual core line 100 and the coordinate system of the endoscope line 98. FIG. 16 shows a state before the coordinate system is associated, and FIG. 16 shows a state where the coordinate system is associated. In FIG. 15 and FIG. 16, the virtual core wire 100 is indicated by a solid line and the endoscope line 98 is indicated by a broken line for distinction. In the state shown in FIG. 15, the coordinate system of the virtual core line 100 is not associated with the coordinate system of the endoscope line 98. From the state shown in FIG. 15, when the endoscope line 98 is rotated and translated with respect to the virtual core line 100, the point sequence corresponding to the endoscopic line 98 corresponds to the virtual core line 100. To be close to. Here, preferably, as shown in FIG. 16, end points (points at both ends) of the virtual core wire 100 and the endoscope line 98 are associated with each other. That is, points corresponding to the cecum and anus in the virtual core line 100 and the endoscope line 98 are associated with each other. Then, from the rotation and translation of the endoscope line 98 when the distance between the point sequence corresponding to the virtual core line 100 and the point sequence corresponding to the endoscope line 98 is the smallest, the endoscope line A transformation matrix related to 98 point sequences is calculated. The transformation matrix derived in this way corresponds to the relationship for associating the coordinate system of the virtual core line 100 with the coordinate system of the endoscope line 98. With the above processing, registration between the coordinate system of the virtual core line 100 and the coordinate system of the endoscope line 98 is automatically realized without requiring manual labor (manual operation) to obtain corresponding points. The This registration of the coordinate system may be performed at least once for the observation support by the support apparatus 10 for a predetermined luminal organ, and may not be performed a plurality of times.

  The line registration processing unit 58 uses the virtual core line 100 calculated by the virtual core calculation unit 50 and the endoscope line calculation unit 52 based on the result of association by the coordinate system registration processing unit 56. A correspondence relationship with the calculated endoscope line 98 is derived. In other words, regarding the endoscope line 98 and the virtual core line 100 in which the coordinate system is associated with the virtual core line 100, each point on the virtual core line 100 and the endoscope line 98. Create a correspondence between each point above. Preferably, a plurality of landmarks are detected for the virtual core wire 100 calculated by the virtual core wire calculation unit 50, and the correspondence relationship between the virtual core wire 100 and the endoscope line 98 based on the plurality of landmarks. Is derived.

  FIG. 17 is a diagram for explaining detection of landmarks for the virtual core wire 100 and derivation of corresponding points in the endoscope line 98 by the line registration processing unit 58. The line registration processing unit 58 preferably detects a plurality of landmarks in the virtual core wire 100 based on the shape of the virtual core wire 100. For example, a plurality of landmarks in the virtual core wire 100 are detected based on the position on the virtual core wire 100 and the curvature of the core wire at each position. In the process for the large intestine, preferably, as shown in FIG. 17, (a) a landmark 106 a corresponding to the cecum in the virtual core 100, and (b) a landmark corresponding to liver curvature in the virtual core 100. 106b, (c) a landmark 106c corresponding to the splenic curvature in the virtual core wire 100, (d) a landmark 106d corresponding to a connecting portion between the sigmoid colon and the descending colon in the virtual core wire 100, and (e) the virtual. The five landmarks 106a to 106e of the landmark 106e corresponding to the anus in the core wire 100 (hereinafter simply referred to as the landmark 106 unless otherwise distinguished) are detected.

  In FIG. 17, the landmarks 106a and 106e are indicated by circles, and the landmarks 106b, 106c and 106d are indicated by squares. A landmark 106 a corresponding to the cecum corresponds to an end point (end point on the cecum side) in the virtual core wire 100. The landmark 106b corresponding to the liver curvature is, for example, when the vertical direction of the paper is the z-axis direction (the upper direction of the paper is the positive direction of the z-axis, the same in the following description), the ascending colon and the transverse colon Corresponds to the point at which the z coordinate becomes maximal. The landmark 106c corresponding to the splenic curve corresponds to, for example, a point where the z coordinate is maximum between the transverse colon and the descending colon. The landmark 106d corresponding to the connecting portion between the sigmoid colon and the descending colon corresponds to, for example, a point where the z coordinate is minimized in the vicinity of the connecting portion between the sigmoid colon and the descending colon. The landmark 106e corresponding to the anus corresponds to an end point (an anus end point) in the virtual core wire 100.

  The line registration processing unit 58 preferably uses the virtual core line 100 and the virtual core line 100 based on the distance from the plurality of landmarks 106 to the endoscope line 98 in the virtual core line 100 detected as described above. A correspondence relationship with the endoscope line 98 is derived. For example, with respect to the endoscope line 98 in which the coordinate system is associated with the virtual core line 100, the distance from each of the plurality of landmarks 106 on the virtual core line 100 to a point on the endoscope line 98. Based on the above, corresponding points 108a to 108e corresponding to the landmarks 106a to 106e in the endoscope line 98 (hereinafter simply referred to as corresponding points 108 unless otherwise distinguished) are derived. Preferably, a point on the endoscope line 98 having the smallest distance from each landmark 106 is derived as a corresponding point 108 corresponding to the landmark 106. In FIG. 17, the corresponding points 108a to 108e of the endoscope line 98 corresponding to the plurality of landmarks 106a to 106e in the virtual core line 100 are indicated by asterisks. That is, the corresponding point 108a is the landmark 106a, the corresponding point 108b is the landmark 106b, the corresponding point 108c is the landmark 106c, the corresponding point 108d is the landmark 106d, and the corresponding point 108e is the landmark. 106e respectively.

  The line registration processing unit 58 preferably corresponds to each point on the virtual core wire 100 between the landmarks 106 detected as described above and each point on the endoscope line 98. The relationship is derived by known linear interpolation. FIG. 18 is a diagram illustrating a process for deriving a correspondence relationship between each point on the virtual core line 100 and each point on the endoscope line 98 by linear interpolation. As shown in FIG. 18, the line registration processing unit 58 is preferably configured such that each point between the landmarks 106 on the virtual core line 100 and a corresponding point 108 (target) on the endoscope line 98 (Corresponding to the landmark 106)) Processing for associating points between each other at equal intervals is performed.

  That is, in the example shown in FIG. 18, each point of the section from the landmark 106a to the landmark 106b on the virtual core line 100, and the corresponding point 108a to the corresponding point 108b on the endoscope line 98. A process for associating each point in the previous section with an equal interval is performed. Each point of the section from the landmark 106b to the landmark 106c on the virtual core line 100, each point of the section from the corresponding point 108b to the corresponding point 108c on the endoscope line 98, etc. Perform the process of associating at intervals. Each point of the section from the landmark 106c to the landmark 106d on the virtual core line 100, each point of the section from the corresponding point 108c to the corresponding point 108d on the endoscope line 98, etc. Perform the process of associating at intervals. Each point in the section from the landmark 106d to the landmark 106e on the virtual core line 100, each point in the section from the corresponding point 108d to the corresponding point 108e on the endoscope line 98, etc. Perform the process of associating at intervals.

  The line registration processing unit 58 preferably performs the virtual core line every time the length of the endoscope line 98 included between the landmarks 106 adjacent to each other in the virtual core line 100 is changed. The correspondence relationship between 100 and the endoscope line 98 is updated. In other words, the correspondence between the virtual core wire 100 and the endoscope line 98 is updated as needed as the length dimension of the endoscope 24 inserted into the hollow organ changes. For example, in an endoscopic examination of the large intestine, after the distal end portion (observation unit 34) of the endoscope 24 is inserted up to the cecum, the endoscope 24 is pulled out of the large intestine and observed by the endoscope 24. Is done. That is, in the colonoscopy with the endoscope 24, the length dimension of the endoscope 24 (tubular portion 32) inserted into the large intestine of the subject changes as needed. Therefore, every time the length of the endoscope line 98 included between the landmarks 106 is changed, a process of updating the correspondence between the virtual core line 100 and the endoscope line 98 is performed. In addition, the correspondence between the virtual core wire 100 and the endoscope line 98 can be derived more strictly.

For example, as shown in FIG. 18, each point in the section from the landmark 106a to the landmark 106b on the virtual core line 100, and from the corresponding point 108a to the corresponding point 108b on the endoscope line 98. In the process of associating with each point in the section, the distance from the initial corresponding point 108a to 108b is l ₁ and the position where the distal end of the endoscope 24 corresponds to the cecum (corresponding to the initial corresponding point 108a) Let us consider a case in which only the length dimension l ₁ -l ₂ (from the original corresponding point 108b to the position of the distance l ₂ ) is drawn out from the position where it is to be performed. It is assumed that the length dimension of the section from the landmark 106a to the landmark 106b on the virtual core wire 100 is L. In this case, each point of the section from the landmark 106b on the virtual core line 100 to a length L·l ₂ / l ₁ in the direction toward the landmark 106a and the endoscope line 98 A process of deriving each point of the section from the corresponding point 108b to the length l ₂ in the direction toward the corresponding point 108a by the linear interpolation is performed. Also preferably, when the distal end portion of the endoscope line 98 is within a specified distance range from any landmark 106 on the virtual core wire 100, the distal end portion of the endoscope line 98 is However, the linear interpolation may be performed in association with such a short-range landmark 106.

  As described above, when the correspondence relationship between the endoscope line 98 and the virtual core line 100 is derived, the observation unit 34 of the endoscope 24 corresponding to the endoscope line 98 is moved to the virtual core line 100. It can be identified at which position in the luminal organ corresponding to. That is, the observation position estimation unit 60 is configured to perform the internal determination in the luminal organ that is the observation target of the endoscope 24 based on the correspondence derived by the matching processing unit 54 (line registration processing unit 58). The observation position of the endoscope 24 is estimated. In the endoscope 24 of the present embodiment, the observation unit 34 is provided at one end (tip) of the tubular portion 32. Therefore, the observation position estimation unit 60 preferably has a position corresponding to a point on the virtual core 100 that is associated with an end point on the distal end side (observation unit 34 side) of the endoscope line 98. Estimated as the observation position of the endoscope 24.

  The lesion presence detection unit 62 detects the presence of a lesion in the hollow organ based on the three-dimensional image data. Preferably, based on the three-dimensional image data, the presence of a lesion in the luminal organ is detected by matching a predetermined condition such as a condition of a voxel having a predetermined CT value. For example, whether or not there is a protruding part suspected of having a lesion such as a polyp or cancer on the surface of the luminal organ in a three-dimensional image corresponding to the three-dimensional image data by known CT colonography (Computed Tomography Colonography) or the like. Determine whether. When such a protrusion is present, the presence of a lesion in the luminal organ is detected. That is, the lesion presence detection unit 62 preferably detects a position where a projection-like portion suspected of having a lesion in the hollow organ exists based on the three-dimensional image data. In addition, a part suspected of having a lesion such as colitis, arteriovenous malformation, and colon line type may be detected based on the surface property of the luminal organ in the three-dimensional image corresponding to the three-dimensional image data.

  When the observation position of the endoscope 24 estimated by the observation position estimation unit 60 reaches the vicinity of the lesion detected by the lesion presence detection unit 62, the lesion notification control unit 64 notifies that fact. Notification (warning) is performed. In other words, the observation position of the endoscope 24 in the luminal organ to be observed, which is estimated by the observation position estimation unit 60, is detected based on the three-dimensional image data corresponding to the luminal organ. When the distance from the lesion site detected by the unit 62 falls within a predetermined distance range, a notification to that effect is given. Preferably, in the video display device 20, a warning display indicating that the observation position of the endoscope 24 estimated by the observation position estimation unit 60 has reached the vicinity of the lesion detected by the lesion presence detection unit 62. I do. That is, the lesion notification control unit 64 may be included in a lesion detection result display control unit 76 described later. Alternatively, the lesion notification control unit 64 may perform the notification with a warning sound or the like via a speaker or the like (not shown) provided in the support apparatus 10.

  The image display control unit 66 causes the video display device 20 to display various images (videos) related to the control by the support device 10 via a graphics processor or the like (not shown). That is, various images supporting the observation by the endoscope 24 inserted into the hollow organ are displayed on the video display device 20. In order to perform such processing, as shown in FIG. 3, the image display control unit 66 includes a real endoscopic image display control unit 68, an organ graphic image display control unit 70, a virtual endoscopic image display control unit 72, a virtual A developed image display control unit 74 and a lesion detection result display control unit 76 are provided. Hereinafter, processing by each display control unit will be described.

  The organ image display control unit 70 displays the image of the luminal organ based on the three-dimensional image data. This pictorial image of the luminal organ is, for example, an image showing the shape of the whole luminal organ in a front view, and is preferably an outer shape image of the luminal organ. That is, the organ pictorial image display control unit 70 preferably displays a pictorial image showing the outer shape of the luminal organ (for example, an image showing an appearance viewed from the front side) based on the three-dimensional image data. That is, in the observation support control by the support apparatus 10, the three-dimensional image data obtained in advance corresponding to the luminal organ to be observed by the endoscope 24 and stored in the image database 44 is read out, A contour image of the luminal organ is generated based on the three-dimensional image data by a technique and displayed on the video display device 20.

  The organ image display control unit 70 displays the observation position of the endoscope 24 estimated by the observation position estimation unit 60 on the image of the lumen organ in the image display control of the hollow organ. FIG. 19 is a diagram illustrating an endoscope navigation screen 110 displayed on the video display device 20 by the image display control unit 66 when the support support control is performed by the support device 10. As shown in FIG. 19, the organ image display control unit 70 preferably displays an outline image 112 of the luminal organ on a part of the endoscope navigation screen 110 and also displays the outline image 112 on the outline image 112. The observation position 114 of the endoscope 24 estimated by the observation position estimation unit 60 is displayed. Preferably, every time the observation position 114 of the endoscope 24 estimated by the observation position estimation unit 60 changes, the display of the observation position 114 on the outline image 112 is updated.

  The virtual endoscopic image display control unit 72 displays a virtual endoscopic image of the luminal organ based on the three-dimensional image data. That is, in the observation support control by the support apparatus 10, the three-dimensional image data obtained in advance corresponding to the luminal organ to be observed by the endoscope 24 and stored in the image database 44 is read out, A virtual endoscopic image of the luminal organ is generated based on the three-dimensional image data by a technique and displayed on the video display device 20.

  The virtual endoscopic image display control unit 72 sets the observation position of the endoscope 24 estimated by the observation position estimation unit 60 based on the three-dimensional image data corresponding to the luminal organ that is the observation target. A virtual endoscopic image of the corresponding luminal organ is displayed. Preferably, from the observation position of the endoscope 24 estimated by the observation position estimation unit 60, based on the viewpoint position and viewpoint direction of the endoscope 24 (observation unit 34), a preset viewing angle, and the like. Then, a virtual endoscopic image of the luminal organ is generated and displayed on the video display device 20. As shown in FIG. 19, the virtual endoscopic image display control unit 72 preferably displays a virtual endoscopic image 116 corresponding to the observation position on a part of the endoscopic navigation screen 110. Preferably, every time the observation position of the endoscope 24 estimated by the observation position estimation unit 60 changes, the display of the virtual endoscopic image 116 (viewpoint position, viewpoint direction, etc.) is updated.

  The virtual endoscopic image display control unit 72 preferably displays a global virtual endoscopic image of the luminal organ based on the three-dimensional image data. This global virtual endoscopic image is a virtual endoscopic image having a maximum viewing angle of 90 ° or more, for example, a virtual endoscopic image having a visual field that surrounds the viewpoint position in a spherical shape. The virtual endoscopic image display control unit 72 preferably generates the global virtual endoscopic image using a known volume rendering method by generating a ray from the viewpoint position to the visual field. That is, in the observation support control by the support apparatus 10, the three-dimensional image data obtained in advance corresponding to the luminal organ to be observed by the endoscope 24 and stored in the image database 44 is read out, A global virtual endoscopic image of the luminal organ is generated based on the three-dimensional image data by a technique and displayed on the video display device 20.

  The virtual endoscopic image display control unit 72 is preferably configured such that the endoscope 24 estimated by the observation position estimation unit 60 based on three-dimensional image data corresponding to the luminal organ to be observed. A global virtual endoscopic image of the hollow organ corresponding to the observation position is displayed. Preferably, from the observation position of the endoscope 24 estimated by the observation position estimation unit 60, based on the viewpoint position and viewpoint direction of the endoscope 24 (observation unit 34), a preset viewing angle, and the like. Then, a global virtual endoscopic image of the luminal organ is generated and displayed on the video display device 20. FIG. 20 is a diagram illustrating a global virtual endoscopic image 118 of the luminal organ displayed by the virtual endoscopic image display control unit 72. A global virtual endoscopic image 118 shown in FIG. 20 corresponds to a maximum viewing angle of 360 °. The global virtual endoscope image 118 shown in FIG. 20 may be displayed on the endoscope navigation screen 110 together with the outline image 112, the virtual endoscope image 116, and the like.

  The virtual developed image display control unit 74 displays a virtual developed image of the luminal organ based on the three-dimensional image data. That is, in the observation support control by the support apparatus 10, the three-dimensional image data obtained in advance corresponding to the luminal organ to be observed by the endoscope 24 and stored in the image database 44 is read out, A virtual developed image of the luminal organ is generated based on the three-dimensional image data by a technique and displayed on the video display device 20.

  The virtual developed image display control unit 74 sets the observation position of the endoscope 24 estimated by the observation position estimation unit 60 on the virtual developed image of the lumen organ in the virtual developed image display control of the hollow organ. To display. As shown in FIG. 19, the virtual developed image display control unit 74 preferably displays the virtual developed image 120 of the luminal organ on a part of the endoscope navigation screen 110 and estimates the observation position. Based on the error of the observation position of the endoscope 24 estimated by the unit 60, the brightness of the predetermined range 120a including the observation position of the endoscope 24 in the virtual developed image 120 and the remaining portions 120b and 120c Display with different brightness. Preferably, the display of the virtual developed image 120 is controlled so that the lightness of the predetermined range 120a is relatively brighter than the lightness of the remaining portions 120b and 120c. Alternatively, the predetermined range 120a in the virtual developed image 120 is not particularly filtered, and the other portions 120b and 120c are displayed with a filter such as smoke (an effect that makes it difficult to visually recognize like a wrinkle) or a so-called mosaic. Control may be performed. Alternatively, display control is performed such that the color of the predetermined range 120a in the virtual developed image 120 is different from that of the remaining portions 120b and 120c, and the method of drawing shadows (shadows) is different. May be. Preferably, an observation position error is predetermined for each section of the luminal organ to be observed, and the virtual developed image display control unit 74 is configured to estimate the internal position estimated by the observation position estimation unit 60. It is determined in which section of the luminal organ to be observed the observation position of the endoscope 24 is included, and an error in the observation position is determined. The virtual developed image display control unit 74 preferably updates the display of the virtual developed image 120 each time the observation position of the endoscope 24 estimated by the observation position estimation unit 60 changes. That is, the predetermined range 120a to be displayed brighter than the remaining portion in the virtual developed image 120 is updated.

  The lesion detection result display control unit 76 displays the detection result by the lesion presence detection unit 62. Preferably, the presence of a lesion detected by the lesion presence detection unit 62 is displayed by the outline image 112 displayed by the organ iconic image display control unit 70 and the virtual endoscope image display control unit 72. A region corresponding to the detected lesion in at least one of the virtual endoscopic image 116, the global virtual endoscopic image 118, and the virtual developed image 120 displayed by the virtual developed image display control unit 74. Display. In the example shown in FIGS. 19 and 20, the lesion position in each image is shown surrounded by a broken line in order to make the drawing easy to see. However, in actual control, the lesion detection result display control unit 76 is preferably The part corresponding to the detected lesion in each image is displayed in a color-changed manner with a marker or the like of a color (for example, purple) different from the remaining part. That is, in the example shown in FIG. 19, a lesion site 116 d corresponding to the position of the lesion detected on the virtual endoscopic image 116 is replaced with a lesion corresponding to the position of the lesion detected on the virtual developed image 120. Each part 120d is displayed. Regarding the outline image 112, the observation position 114 corresponds to the detected lesion position. In the example shown in FIG. 20, a lesion site 118d corresponding to the position of the detected lesion is displayed on the global virtual endoscopic image 118. Preferably, the lesion detection result display control unit 76 changes the outline image 112 and the virtual endoscopic image 116 each time the observation position of the endoscope 24 estimated by the observation position estimation unit 60 changes. The display of the lesion site in the global virtual endoscopic image 118, the virtual developed image 120, etc. is updated.

  The actual endoscope image display control unit 68 causes the video display device 20 to display an image (actual image) acquired by the observation unit 34 of the endoscope 24. That is, an image acquired by the observation unit 34 and input via the connection cable 38 and the interface 30 is displayed on the video display device 20. Preferably, an image acquired by the observation unit 34 of the endoscope 24 is displayed by the outline image 112 displayed by the organ iconographic display control unit 70 and the virtual endoscopic image display control unit 72. The image is displayed on the video display device 20 together with at least one of the virtual endoscopic image 116, the global virtual endoscopic image 118, and the virtual developed image 120 displayed by the virtual developed image display control unit 74. Alternatively, the external image 112, the virtual endoscopic image 116, the global virtual endoscopic image 118, the virtual developed image 120, etc. are displayed on a video display device different from the video display device 20 on which the image display device 20 is displayed. An image acquired by the observation unit 34 of the endoscope 24 may be displayed. Further, the actual endoscope image display control unit 68 may be provided in a control device separate from the support device 10.

  FIG. 21 illustrates another endoscope navigation screen 130 displayed on the video display device 20 by the image display control unit 66 during observation support control by the support device 10 in observing the large intestine by the endoscope 24. It is a figure to do. As shown in FIG. 21, the image display control unit 66 preferably includes an actual observation image (live colonoscopy image) 132 of the endoscope 24 displayed by the real endoscope image display control unit 68. The outline image 134 of the large intestine displayed by the organ image display control unit 70 and the virtual endoscopic image (virtualized endoscopic image) 136 of the large intestine displayed by the virtual endoscopic image display control unit 72. And the supine virtual developed image 138 and prone virtual developed image 140 of the large intestine displayed by the virtual developed image display control unit 74, and the presence of a lesion corresponding to the detection result by the lesion presence detecting unit 62 (polyp navigation) 142), a warning 144 that the observation position of the endoscope 24 estimated by the observation position estimation unit 60 has reached the vicinity of the lesion detected by the lesion presence detection unit 62, and the lesion site Correspondingly, an endoscope navigation screen 130 including a virtual endoscopic image (corresponding virtualized colonoscopic image) 146 displayed by the virtual endoscopic image display control unit 72 is displayed on one screen of the video display device 20. Display. Further, a lesion site 136 d corresponding to the position of the lesion detected by the lesion presence detection unit 62 is displayed on the virtual endoscopic image 136.

  FIG. 22 is a flowchart for explaining a main part of an example of endoscope navigation control by the CPU 12 of the support apparatus 10, and is repeatedly executed at a predetermined cycle.

  First, in step (hereinafter, step is omitted) S1, it is determined whether or not observation support (navigation) control of the endoscope 24 by the support apparatus 10 is started. Preferably, the distal end portion (observation unit 34) of the endoscope 24 is inserted up to the cecum, and it is determined whether or not the observation of the endoscope 24 is started. If the determination at S1 is negative, the routine is terminated. If the determination at S1 is affirmative, at S2, the routine proceeds to the large intestine to be observed by the endoscope 24. The three-dimensional image data acquired in advance and stored in the image database 44 is read out. Next, in S3, the virtual core line 100 along the longitudinal direction of the large intestine is calculated based on the three-dimensional image data read in S2.

  Next, in S4, an endoscope line 98 indicating the shape of the endoscope 24 inserted into the large intestine is calculated based on position information and direction information detected by each of the plurality of position sensors 42. . Next, in S5, a coordinate system registration for associating the coordinate system corresponding to the virtual core line 100 calculated in S3 and the coordinate system corresponding to the endoscope line 98 calculated in S4. Processing is executed. Next, in S6, based on the result of the coordinate system registration process in S5, the correspondence between the virtual core line 100 calculated in S3 and the endoscope line 98 calculated in S4 is calculated. A line registration process is executed. Next, in S7, the observation position (tip position) of the endoscope 24 in the large intestine is estimated based on the correspondence relationship derived in S6.

  Next, in S8, it is determined whether or not the length of the endoscope line 98 included between the landmarks 106 adjacent to each other in the virtual core line 100 has been changed. If the determination in S8 is affirmed, the processes in and after S6 are executed again. However, if the determination in S8 is negative, after the image display process shown in FIG. 23 is executed in SS, In S9, it is determined whether or not the observation support control of the endoscope 24 by the support device 10 is terminated. If the determination in S9 is negative, the processing from S8 is executed again, but if the determination in S9 is affirmative, the routine is terminated accordingly.

  In the image display control shown in FIG. 23, first, an image (actual image) acquired by the observation unit 34 of the endoscope 24 is displayed on the video display device 20 in SS1. This SS1 process need not necessarily be executed. Next, in SS2, it is determined whether or not it is an external image display mode in which the external image of the large intestine is displayed on the video display device 20. When the determination of SS2 is negative, the processing after SS4 is executed. However, when the determination of SS2 is positive, in SS3, based on the three-dimensional image data read in S2. An outline image showing the outline of the large intestine is generated and displayed on the video display device 20, and the observation position of the endoscope 24 estimated in S7 is displayed on the outline image of the large intestine.

  Next, in SS4, it is determined whether or not the video display device 20 is in a virtual endoscopic image display mode for displaying the virtual endoscopic image of the large intestine. If the determination at SS4 is negative, the processing below SS6 is executed, but if the determination at SS4 is affirmative, at SS5, based on the three-dimensional image data read at S2. , A virtual endoscopic image of the large intestine corresponding to the observation position of the endoscope 24 estimated in S7 is generated and displayed on the video display device 20. Next, in SS6, it is determined whether or not it is a virtual developed image display mode in which the video display device 20 displays the virtual developed image of the large intestine. If the determination at SS6 is negative, the processing from SS8 is executed. If the determination at SS6 is affirmative, based on the three-dimensional image data read at S2 at SS7. A virtual unfolded image of the large intestine is generated and displayed on the video display device 20, and the internal image in the virtual unfolded image of the large intestine is based on the error of the observation position of the endoscope 24 estimated in S7. The brightness of a predetermined range including the observation position of the endoscope 24 is displayed brighter than the remaining part.

  Next, in SS8, based on the three-dimensional image data read in S2, it is determined whether or not there is a suspected lesion in the large intestine. If the determination at SS8 is negative, the processing from SS10 onward is executed. If the determination at SS8 is positive, the presence of the detected lesion is displayed on the video display device 20 at SS9. Is done. Preferably, at least one of the outline image displayed at SS3, the virtual endoscopic image displayed at SS5, and the virtual developed image displayed at SS7, at a site corresponding to the detected lesion. Display its presence. Next, in SS10, it is determined whether or not the observation position of the endoscope 24 estimated in S7 has reached the vicinity of the lesion detected in SS8. When the determination of SS10 is negative, the control is returned to the control shown in FIG. 22. However, when the determination of SS10 is positive, the observation position of the endoscope 24 is After a warning approaching a region suspected of having a lesion in the large intestine is notified in the video display device 20 or the like, the control returns to the control shown in FIG.

  In the above control, S3 is the operation of the virtual core calculation unit 50, S4 is the operation of the endoscope calculation unit 52, S5 and S6 are the operation of the matching processing unit 54, and S5 is the coordinate system registration. S6 is the operation of the line registration processing unit 58, S7 is the operation of the observation position estimation unit 60, SS8 is the operation of the lesion presence detection unit 62, and SS11 is the lesion notification. For the operation of the control unit 64, SS for the operation of the image display control unit 66, SS1 for the operation of the real endoscopic image display control unit 68, SS3 for the operation of the organ image display control unit 70, and SS5 for SS7 corresponds to the operation of the virtual endoscopic image display control unit 72, SS7 corresponds to the operation of the virtual developed image display control unit 74, and SS9 corresponds to the operation of the lesion detection result display control unit 76, respectively.

  Thus, according to the present embodiment, the virtual core calculation unit 50 (S3) that calculates the virtual core 100 along the longitudinal direction of the luminal organ based on the three-dimensional image data, and the inside of the luminal organ An endoscope line calculation unit 52 (S4) for calculating an endoscope line 98 indicating the shape of the endoscope 24 inserted into the endoscope 24, the virtual core line 100 calculated by the virtual core line calculation unit 50, and Based on the matching processing unit 54 (S5 and S6) for deriving the correspondence with the endoscope line 98 calculated by the endoscope line calculation unit 52, and the correspondence derived by the matching processing unit 54, Since it includes the observation position estimation unit 56 (S7) that estimates the observation position of the endoscope 24 in the hollow organ, even if it is a deformable lumen organ such as the large intestine, Virtual core 100 of the luminal organ The matching of the endoscope line 98 indicating the shape of the endoscope, the support of the observation by the endoscope 24 can be suitably implemented. That is, the support device 10 that supports observation of a luminal organ such as the large intestine can be provided.

  The coordinate system corresponding to the virtual core line 100 calculated by the virtual core line calculation unit 50 is associated with the coordinate system corresponding to the endoscope line 98 calculated by the endoscope line calculation unit 52. A coordinate system registration processing unit 56 (S5) is provided, and the matching processing unit 54 determines whether the virtual core wire 100 and the endoscope line 98 are based on the result of association by the coordinate system registration processing unit 56. Since the correspondence relationship is derived, the correspondence relationship between the virtual core wire 100 of the luminal organ and the endoscope line 98 indicating the shape of the endoscope 24 can be derived in a preferable and practical manner.

  The matching processing unit 54 detects a plurality of landmarks 106 with respect to the virtual core wire 100 calculated by the virtual core wire calculation unit 50, and the virtual core wire 100 and the endoscope line based on the plurality of landmarks 106. 98, the correspondence between the virtual core wire 100 of the luminal organ and the endoscope line 98 indicating the shape of the endoscope 24 can be derived in a suitable and practical manner. .

  Based on the three-dimensional image data, an outline image 112 that is a graphic image of the luminal organ is displayed, and an observation position of the endoscope 24 estimated by the observation position estimation unit 56 is displayed on the outline image 112. Since the organ image display control unit 70 (SS3) is provided, practical endoscope navigation for displaying the observation position by the endoscope 24 on the outline image 112 of the luminal organ to be observed is realized. it can.

  A virtual endoscopic image display for displaying a virtual endoscopic image 116 of the luminal organ corresponding to the observation position of the endoscope 24 estimated by the observation position estimation unit 56 based on the three-dimensional image data. Since the control unit 72 (SS5) is provided, practical endoscope navigation for displaying the virtual endoscopic image 116 of the luminal organ corresponding to the observation position by the endoscope 24 can be realized.

  The virtual endoscopic image display control unit 72 is a global virtual type of the luminal organ corresponding to the observation position of the endoscope 24 estimated by the observation position estimation unit 56 based on the three-dimensional image data. Since the endoscopic image 118 is displayed, practical endoscopic navigation for displaying the global virtual endoscopic image 118 of the luminal organ corresponding to the observation position by the endoscope 24 can be realized.

  Based on the three-dimensional image data, the virtual development image 120 of the luminal organ is displayed, and the observation position of the endoscope 24 estimated by the observation position estimation unit 56 is displayed on the virtual development image 120 of the luminal organ. Since the virtual developed image display control unit 74 (SS7) to be displayed is provided, practical endoscope navigation for displaying the virtual developed image 120 of the luminal organ corresponding to the observation position by the endoscope 24 is performed. realizable.

  The virtual developed image display control unit 74 is configured to detect the endoscope 24 in the virtual developed image 120 of the luminal organ based on the error in the observation position of the endoscope 24 estimated by the observation position estimating unit 56. Since the display of the predetermined range 120a including the observation position is displayed differently from the other portions 120b and 120c, the virtual development image 120 of the luminal organ corresponding to the observation position by the endoscope 24 is further easily understood. Endoscopic navigation to be displayed can be realized.

  A lesion presence detection unit 62 (SS8) that detects the presence of a lesion in the hollow organ based on the three-dimensional image data, and a lesion detection result display control unit 76 (SS9) that displays a detection result by the lesion presence detection unit 62. Therefore, practical endoscope navigation for displaying the presence or absence of a lesion in a hollow organ that is an object to be observed by the endoscope 24 can be realized.

  When the observation position of the endoscope 24 estimated by the observation position estimation unit 56 reaches the vicinity of a lesion detected by the lesion presence detection unit 62, a lesion notification control unit 64 that performs notification to that effect. Since (SS11) is provided, when the observation position by the endoscope 24 reaches the vicinity of the lesion, practical endoscope navigation for notifying that effect can be realized.

  Outline image 112 of the luminal organ displayed by the organ image display control unit 70, virtual endoscopic image 116 of the luminal organ displayed by the virtual endoscopic image display control unit 72, global virtual interior At least one of the endoscopic image 118 and the virtual developed image 120 of the luminal organ displayed by the virtual developed image display control unit 74 is displayed, and the lesion detection result display control unit 76 displays the lesion presence. The presence of a lesion detected by the detection unit 62 is determined based on the outline image 112 of the luminal organ, the virtual endoscopic image 116 of the luminal organ, the global virtual endoscopic image 118, and the virtual development of the luminal organ. Since it is displayed on a portion corresponding to the detected lesion in at least one of the images 120, a practical and easy-to-understand endoscope is displayed by displaying the detected lesion on each image. It can be realized navigation.

  The matching processing unit 54 detects a plurality of landmarks 106 in the virtual core wire 100 based on the shape of the virtual core wire 100, and the endoscope line 98 from the detected plurality of landmarks 106 in the virtual core wire 100. Since the correspondence relationship between the virtual core line 100 and the endoscope line 98 is derived based on the distance up to, the virtual core line 100 of the luminal organ and the endoscope 24 in a suitable and practical manner. It is possible to derive the correspondence with the endoscope line 98 indicating the shape of

  The matching processing unit 54 derives a correspondence relationship between each point on the virtual core line 100 and each point on the endoscope line 98 between the landmarks 106 by linear interpolation. The correspondence relationship between the virtual core wire 100 of the hollow organ and the endoscope line 98 indicating the shape of the endoscope 24 can be derived in a specific manner.

  Whenever the length of the endoscope line 98 included between the landmarks 106 adjacent to each other is changed, the matching processing unit 54 associates the virtual core line 100 with the endoscope line 98. Since the relationship is updated, each time the endoscope 24 is pulled out or inserted from the hollow organ, the virtual core 100 of the hollow organ and the endoscope showing the shape of the endoscope 24 are displayed. The correspondence with the mirror line 98 can be derived in real time.

  The preferred embodiments of the present invention have been described in detail with reference to the drawings. However, the present invention is not limited to these embodiments, and may be implemented in other modes.

  For example, in the above-described embodiment, the support device 10 supports the observation of the large intestine as a luminal organ by the endoscope 24, but the present invention is not limited to this. The endoscope 24 may support observation of a luminal organ such as a small intestine.

  In the above-described embodiment, the endoscope 24 is provided with the single observation part 34 at the distal end thereof. However, the endoscope 24 is singular in addition to the distal end of the tubular part 32 or in the middle part excluding the distal end. Thru | or the thing provided with the some observation part 34 may be sufficient. In such observation by the endoscope 24, the observation position estimation unit 60 in the support device 10 preferably estimates observation positions corresponding to the plurality of observation units 34 in the endoscope 24, respectively.

  In the above-described embodiment, the display mode in which the outline image 112, the virtual endoscopic image 116, and the virtual developed image 120 are displayed on one screen in the video display device 20 has been described. Of course, the outer shape image 112, the virtual endoscopic image 116, and the virtual developed image 120 may be displayed on different screens (different video display devices).

  In the above-described embodiment, the tubular portion 32 of the endoscope 24 is provided with four position sensors 42 at a predetermined interval. However, the number of position sensors is three or less, or five or more position sensors. May be provided. The greater the number of position sensors, the better the relative position of the endoscope 24 (tubular portion 32) relative to the luminal organ can be detected.

  In the above-described embodiment, the matching processing unit 54 detects five landmarks 106 in the virtual core wire 100 corresponding to the large intestine. However, the matching processing unit 54 detects four or less landmarks or six or more landmarks. It may be detected. The greater the number of landmarks detected, the better the correspondence between the virtual core line 100 and the endoscope line 98 can be derived.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

  10: Endoscopic observation support device, 24: Endoscope, 50: Virtual core calculation unit, 52: Endoscope calculation unit, 54: Matching processing unit, 56: Coordinate system registration processing unit, 60: Observation position Estimator 62: Lesion presence detector 64: Lesion notification controller 70: Organ image display controller 72: Virtual endoscopic image display controller 74: Virtual developed image display controller 76: Lesion detection result Display control unit, 98: Endoscopic line, 100: Virtual core line, 106: Landmark, 112: Outline image (graphic image), 114: Observation position, 116, 136, 146: Virtual endoscopic image, 116d, 118d, 120d, 136d: lesion site, 118: global virtual endoscopic image, 120: virtual developed image, 120a: predetermined range, 120b, 120c: extra portion, 138: supine virtual developed image, 140: prone virtual developed image, 1 2: The presence of lesions, 144: warning

Claims (12)

An endoscopic observation support device that supports observation with an endoscope inserted into a hollow organ based on three-dimensional image data of the hollow organ obtained in advance,
A virtual core calculation unit that calculates a virtual core along the longitudinal direction of the luminal organ based on the three-dimensional image data;
An endoscope line calculation unit for calculating an endoscope line indicating the shape of the endoscope inserted into the lumen organ;
A matching processing unit for deriving a correspondence relationship between the virtual core line calculated by the virtual core line calculation unit and the endoscope line calculated by the endoscope line calculation unit;
An observation position estimation unit that estimates the observation position of the endoscope in the luminal organ based on the correspondence relationship derived by the matching processing unit ;
A virtual development image of the luminal organ is displayed based on the three-dimensional image data, and a virtual display image of the observation position of the endoscope estimated by the observation position estimation unit is displayed on the virtual development image of the luminal organ. A developed image display control unit,
The virtual developed image display control unit is a predetermined unit including an observation position of the endoscope in a virtual developed image of the luminal organ based on an error in the observation position of the endoscope estimated by the observation position estimation unit. An endoscope observation support apparatus characterized in that the display of the range is a display different from that of the remaining part . Coordinate system registration for associating a coordinate system corresponding to the virtual core calculated by the virtual core calculator with a coordinate system corresponding to the endoscope calculated by the endoscope line calculator With a processing unit,
The endoscope according to claim 1, wherein the matching processing unit derives a correspondence relationship between the virtual core line and the endoscope line based on a result of association by the coordinate system registration processing unit. Observation support device.   The matching processing unit detects a plurality of landmarks for the virtual core line calculated by the virtual core line calculation unit, and derives a correspondence relationship between the virtual core line and the endoscope line based on the plurality of landmarks. The endoscopic observation support device according to claim 1 or 2.   An organ image display control unit that displays a pictorial image of the luminal organ based on the three-dimensional image data, and displays an observation position of the endoscope estimated by the observation position estimation unit on a graphic image of the luminal organ. The endoscopic observation support apparatus according to any one of claims 1 to 3, further comprising:   A virtual endoscopic image display control unit configured to display a virtual endoscopic image of the luminal organ corresponding to the observation position of the endoscope estimated by the observation position estimation unit based on the three-dimensional image data; The endoscope observation support apparatus according to any one of claims 1 to 4, wherein the endoscope observation support apparatus is provided.   The virtual endoscopic image display control unit is a global virtual endoscope of the luminal organ corresponding to the observation position of the endoscope estimated by the observation position estimation unit based on the three-dimensional image data. The endoscope observation support apparatus according to claim 5, which displays an image. A lesion presence detector that detects the presence of a lesion in the hollow organ based on the three-dimensional image data;
The endoscopic observation support apparatus according to any one of claims 1 to 6 , further comprising a lesion detection result display control unit configured to display a detection result by the lesion presence detection unit. When the observation position of the endoscope estimated by the observation position estimation unit reaches the vicinity of a lesion detected by the lesion presence detection unit, a lesion notification control unit that notifies that fact is provided The endoscope observation support apparatus according to claim 7 . An image of the luminal organ displayed by the organ image display control unit according to claim 4,
A virtual endoscopic image of the luminal organ displayed by the virtual endoscopic image display control unit according to claim 5 or 6,
And displaying at least one of the virtual developed images of the luminal organ displayed by the virtual developed image display control unit according to claim 1 ,
Lesion detection result display control unit according to claim 7, for the presence of lesions detected by the lesion-existence detection portion according to claim 7, wherein the luminal organ iconography, virtual endoscopic image of the luminal organ And at least one of the virtual developed images of the luminal organ displayed on a site corresponding to the detected lesion.   The matching processing unit detects a plurality of landmarks in the virtual core line based on the shape of the virtual core line, and based on distances from the plurality of landmarks in the detected virtual core line to the endoscope line The endoscope observation support apparatus according to claim 3, wherein a correspondence relationship between a virtual core line and the endoscope line is derived. The endoscope according to claim 3 or 10 , wherein the matching processing unit derives a correspondence relationship between each point on the virtual core line and each point on the endoscope line between the landmarks by linear interpolation. Mirror observation support device. The matching processing unit updates the correspondence between the virtual core line and the endoscope line every time the length of the endoscope line included between the landmarks adjacent to each other is changed. The endoscopic observation support device according to any one of claims 3, 10 , and 11 .