JP2009056238A - Endoscope apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an endoscope apparatus surely and quickly inserting an endoscope to a target site. <P>SOLUTION: This endoscope apparatus is provided with: a virtual image generation means for generating a virtual image of a body cavity pathway inside a subject based on image data of a three-dimensional region of the subject; a guide image generation means for generating the guide image of a branch part of the body cavity pathway based on the virtual image; and an image composition means superimposedly displaying the endoscopic image of the body cavity pathway and the guide image of a next branch part of the body cavity pathway in the periphery side of an imaging position of the endoscopic image, wherein the guide image is one or more images selected from the virtual endoscopic image 6d, a virtual endoscopic schema image, a moving-directional display image and a template image 6T. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an endoscope apparatus that inserts an endoscope into a body cavity path in a subject, and in particular, an endoscope that performs insertion navigation using a virtual image of a body cavity path based on image data of a three-dimensional region of the subject. Relates to the device.

  In recent years, diagnosis using a three-dimensional image has been widely performed. For example, by capturing a tomographic image of a subject with an X-ray CT (Computed Tomography) apparatus, three-dimensional image data in the subject is obtained, and a target site is diagnosed using the three-dimensional image data. It has become.

  In the CT apparatus, the subject is continuously moved while continuously rotating the X-ray irradiation position and the detection position, thereby subjecting the subject to a helical continuous scan (helical scan). Then, a three-dimensional image is formed from a large number of tomographic two-dimensional images of the subject.

  One of the three-dimensional images that are widely diagnosed is a three-dimensional image of a lung bronchus. The three-dimensional image of the bronchus is used to three-dimensionally grasp the position of an abnormal part suspected of lung cancer, for example. In order to confirm an abnormal part by biopsy, a bronchoscope is inserted, a biopsy needle, a biopsy forceps, or the like is taken out from the distal end, and a tissue sample is collected.

In a body duct having a multi-stage branch, such as the bronchi, when the location of the abnormal part is close to the periphery of the branch, it is difficult to correctly reach the target site in a short time in the endoscope. For this reason, for example, in Japanese Patent Application Laid-Open No. 2004-180940 or Japanese Patent Application Laid-Open No. 2005-131042, a three-dimensional image of a duct in the subject is formed based on image data of a three-dimensional region of the subject, It is disclosed that a route to a target point along a pipeline is obtained on a three-dimensional image, and a virtual endoscopic image of the pipeline along the route is formed based on the image data. Then, as shown in FIG. 32, an endoscope apparatus is proposed in which a virtual endoscopic image 6d and an endoscopic image 6f are displayed side by side to navigate the bronchoscope to a target site. ing. Further, in the endoscope apparatus, as shown in FIG. 32, the marker 6G is displayed in the path hole to be advanced, and the similarity between the virtual endoscopic image and the endoscopic image is obtained using image processing. It is disclosed that when the degree of similarity is equal to or higher than a predetermined value, a virtual endoscopic image is rotated and displayed in the same direction as the endoscopic image.
JP 2004-180940 A JP 2005-131042 A

  However, even if the endoscope apparatus disclosed in Japanese Patent Laid-Open No. 2004-180940 or Japanese Patent Laid-Open No. 2005-131042 is used, a virtual endoscopic image and an endoscopic image are processed using image processing. When the similarity is compared, if the similarity is equal to or less than a predetermined value, it is determined that the insertion has misidentified the branch. Therefore, the surgeon obtains an endoscopic image 6d similar to the virtual endoscopic image 6f while comparing the virtual endoscopic image 6d displayed on the left and right with the endoscopic image 6f. Was operating the endoscope to be able to. However, it may not be easy for the surgeon to compare the two images displayed on the left and right. Also, it is not clear to the surgeon how much the virtual endoscopic image 6d and the endoscopic image 6f are properly aligned by image processing.

  An object of this invention is to provide the endoscope apparatus which can insert an endoscope in a target site | part reliably and rapidly.

  In order to achieve the above object, an endoscope apparatus according to the present invention includes a virtual image generation unit that generates a virtual image of a body cavity in the subject based on image data of a three-dimensional region of the subject, and the virtual image A guide image generating means for generating a guide image of a branch portion of the body cavity path, an endoscopic image of the body cavity path, and a next branch on the distal side from the imaging position of the endoscopic image in the body cavity path An image compositing means that superimposes and displays the guide image of the unit, wherein the guide image is one or more selected from the virtual endoscope image, a virtual endoscope schematic diagram image, a moving direction display image, and a template image It is an image.

  The present invention provides an endoscope apparatus that can insert an endoscope into a target site reliably and quickly.

  Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment>
<Description of device configuration>
FIG. 1 is a configuration diagram showing a configuration of an endoscope apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 1, an endoscope apparatus 1 according to the present embodiment is an endoscope that inserts into a bronchus that is a body cavity of a subject patient, images the inside of the bronchus, and biopsyes a region of interest at the end of the bronchus. A bronchoscope device 2 having 2A, and a virtual bronchoscope image (Virtual Bronscope Scope: hereinafter referred to as a virtual endoscopic image or a VBS image) based on CT image data, and from the imaging unit 2a A real-time endoscopic image 6f (hereinafter referred to as an endoscopic image or a live image) obtained by the bronchoscope device 2 based on the imaging signal and a guide image shown in FIG. 4 or the like serving as an insertion guide for the endoscope 2A. 6Z is superimposed on the monitor 6 and displayed on the monitor 6 for insertion navigation into the bronchus of the endoscope 2A, and an operator's instruction is input to the insertion support device 5. 1 input unit 8 and a second input unit 19 for inputting a route setting instruction of the insertion support apparatus 5.

  The bronchoscope device 2 includes an endoscope 2A that is inserted into a body cavity of a subject, and an endoscope control unit 2B that controls the endoscope 2A. An imaging means 2a (hereinafter referred to as “camera”) such as a CCD is disposed at the distal end portion 2b of the endoscope 2A.

  The insertion support device 5 can convert three-dimensional image data generated by a known CT device (not shown) that captures an X-ray tomographic image of a patient, such as an MO (Magneto Optical disk) device or a DVD (Digital Versatile Disk) device, etc. A CT image data capturing unit 11 that is captured via a portable storage medium, a CT image data storage unit 12 that stores three-dimensional image data captured by the CT image data capturing unit 11, and a CT image data storage unit 12. An MPR image generation unit 13 that generates an MPR (MultiPlanar Reconstruction) image based on the three-dimensional image data being inserted, and an insertion path (hereinafter referred to as insertion) from the MPR image generated by the MPR image generation unit to the bronchial region of interest of the endoscope 2A Called route or route) A route setting unit 14 having a route setting unit and a re-route setting unit to set, and a VBS image having a continuous route set by the route setting unit 14 based on the three-dimensional image data stored in the CT image data storage unit 12. A VBS image generation unit 15 having a virtual endoscope image generation unit that generates a frame unit, a VBS image storage unit 16 that stores a VBS image generated by the VBS image generation unit 15, and an image composition that generates a display screen to be described later An image processing unit 17 that is an image processing unit having means, a guide image generation unit, and a matching processing unit, an insertion path generated by the route setting unit 14, a navigation screen generated by the image processing unit 17, and the like. And a display control unit 18.

  Here, image matching will be described. Image matching is image processing performed by the matching processing means of the image processing unit 17. The similarity between the VBS image and the endoscopic image 6 f is compared. If the similarity is equal to or greater than a predetermined value, both images are the same. It is the process which judges that it is the image which showed the branch part of. The matching is performed by known image processing, and either matching at the pixel data level or matching at the feature level extracted from the image may be used. Further, “matchable or successful matching” means that the image processing unit 17 determines that the two images subjected to the matching process are images showing the same branching unit.

  At the time of matching, the image processing unit 17 may erroneously recognize the VBS image and the endoscopic image 6f even though they are images indicating the same branch portion. This misrecognition of the image processing unit 17 can be solved to some extent by a more advanced matching process, but there is also a problem such as a decrease in processing speed. For this reason, the insertion support apparatus 5 of the endoscope apparatus 1 according to the present embodiment also performs navigation for capturing the endoscopic image 6f suitable for matching in addition to navigation to the correct insertion path.

  Since the matching between the live image and the virtual endoscopic image 6d is performed in units of frames of the live image, the actual matching process is a matching between the static endoscopic image and the virtual endoscopic image 6d. It is not necessary to perform matching processing for all frames of the live image, and it is performed at an appropriate interval.

  The matching process is also used when the image correction unit performs a rotation process of rotating the virtual endoscopic image 6d in order to match the display directions of the guide image including the virtual endoscopic image 6d and the endoscopic image 6f. Is done. In this case, when the image processing unit 17 determines that the virtual endoscopic image 6d and the endoscopic image 6f are similar images by the matching process, the rotation processing can be performed automatically or manually.

  The CT image data storage unit 12 and the VBS image storage unit 16 may be configured by a single hard disk device, and the MPR image generation unit 13, route setting unit 14, VBS image generation unit 15, and image processing unit 17. Can be composed of one arithmetic processing circuit. The CT image data capturing unit 11 captures CT image data via a portable storage medium such as an MO or a DVD. However, a CT apparatus or a hospital server storing the CT image data is connected to the hospital LAN. In such a case, the CT image data capturing unit 11 may be configured by an interface circuit that can be connected to the in-hospital LAN, and CT image data may be captured through the in-hospital LAN.

  Next, based on the image data of the three-dimensional region of the subject, the route setting for setting the insertion route of the body cavity path for inserting the distal end portion 2b of the endoscope 2A from the outside of the body cavity to the site of interest in the subject. A route setting unit 14 having means and a VBS image generating unit 15 having virtual endoscope image generating means will be described.

  Prior to the insertion of the endoscope 2A, the insertion support device 5 captures the 3D image data of the patient generated by the CT device by the CT image data capturing unit 11, and the CT image data storage unit 12 stores the captured 3D image data. To store. Then, an MPR image including, for example, three different cross-sectional images of the patient is generated by the MPR image generation unit, and the route setting unit 14 includes an insertion start point A1 of the MPR image in the body cavity close to the outside of the body cavity shown in FIG. When the insertion end point E1 in the vicinity of the region of interest 100B in the subject is designated by the second input unit 19, a route setting process for setting a body cavity path connecting the insertion start point A1 and the insertion end point E1 is performed, and the endoscope A 2A insertion path R1 is set. When the insertion path R1 is set, the VBS image generation unit 15 generates a continuous two-dimensional virtual endoscopic image 6d of the set path for each frame, and stores the generated virtual endoscopic image 6d in a VBS image. Store in unit 16.

  The virtual endoscopic image 6d is generated by the VBS image generation unit 15, but the virtual endoscopic image 6d of the branching unit is also described later in the guide image generation unit of the image processing unit 17 as a guide image of the branching unit. In this way, the image is generated while changing the imaging condition of the virtual endoscope V2A. Further, the guide image generation means of the image processing unit 17 generates not only the virtual endoscope image 6d but also the virtual endoscope schematic diagram image 6J and the moving direction display image 6H shown in FIG. . Further, a template image 6T, which is a guide image of a branching unit described later, is also generated by the guide image generating unit of the image processing unit 17. The image processing unit 17 also includes an image correction unit that performs rotation processing of a guide image including a virtual endoscopic image, and an image synthesis unit that superimposes and displays the endoscopic image and the guide image.

  When the insertion support apparatus 5 sets an insertion path and generates and stores a VBS image, the insertion support apparatus 5 displays the insertion path and the like on the monitor 6. FIG. 3 is a diagram showing an example of the display screen 6a displaying the insertion path and the like. As shown in FIG. 3, the display screen 6a displays patient information, branch information 6A, an image 6B displaying the insertion path R1 of the endoscope 2A, and a VBS image 6C not shown in detail. Is done. In the image 6B, the insertion of the endoscope 2A in the vicinity of the region of interest 100B in the subject from the insertion start point A1 of the endoscope 2A in the body cavity is superimposed on the bronchial image 100A of the patient generated from the three-dimensional image. The end point E1 and the insertion path R1 of the endoscope 2A set by the route setting unit 14 are displayed.

<Description of display screen>
Next, FIG. 4 is a diagram showing a display form at the time of insertion navigation of the display screen 6 a of the monitor 6. Although only one monitor 6 is shown in FIG. 1, it is also possible to simultaneously display display screens having different display forms using a plurality of monitors.

  As shown in FIG. 4, on the left side of the display screen 6a, a live image 6f and a guide image are superimposed and displayed on a live image display area 6F of a bronchial bifurcation J on the insertion path R1. Here, since the display screen 6a is a display screen in the fourth branching section, there are two guide images: a virtual endoscope schematic diagram image 6J4 and an arrow image 6H which is a moving direction display image. The guide image to be superimposed is a guide image of the next branching portion on the distal side from the current camera position, that is, a branching portion that is reached next in the insertion operation of the endoscope, in other words, an endoscope image in the body cavity path Is a guide image of the next branch portion in the body cavity path, in other words, the branch portion that performs the matching process next.

  In addition, the superimposed display is to display the images in the same display area, and in general, the images can be superimposed and displayed by simultaneously displaying images of different layers. Note that these images are processed by the image synthesizing unit of the image processing unit 17 and displayed in a superimposed manner. Further, displaying in each of the plurality of image display areas of the same display screen 6a does not correspond to the superimposed display.

  Since the display screen 6a is a display screen at the fourth branch section, the right VBS image display area 6D has a branch that is the next branch section on the distal side from the position of the camera 2a of the endoscope 2A in the body cavity. A marker 6G is superimposed on a path hole for proceeding along the insertion path R1 and displayed as a VBS image 6d4 of the part J4, a virtual endoscope schematic diagram image (hereinafter referred to as a VBS schematic diagram image) 6J4 of the branch part J4 is doing. The position and size of the VBS schematic diagram image 6J4 displayed in the VBS image display area 6D in the VBS image display area 6D, and the live image of the virtual endoscope schematic diagram image 6J4 displayed in the live image display area 6F The position and size in the display area 6F are the same. That is, only the virtual endoscope schematic diagram image 6J4 in the VBS image display area 6D is copied and superimposed on the live image display area 6F. Then, the guide image displayed in the live image display area 6F and the guide image displayed in the live image display area 6F move in conjunction with each other. For this reason, the surgeon can confirm the deviation between the VBS image 6d and the live image 6f even when paying attention only to the live image display area 6F.

  Among the branch thumbnail VBS images displayed in the VBS image display area 6E on the lower side of the display screen 6a, the frame 6E4 corresponding to the VBS image 6d4 is displayed in a thick frame or in color, and is distinguished from other branch thumbnail VBS images. The surgeon can easily recognize which branching part the VBS image displayed in the VBS image display area 6D is.

  Next, each image displayed on the display screen 6a will be described. As described above, the VBS image is a two-dimensional image formed from the three-dimensional bronchial image data of the bronchus 100A. In other words, the VBS image is a virtual image obtained by imaging the lumen of the virtual bronchus 100A with a virtual endoscope. Although the VBS image can be formed for all lumen portions of the bronchus 100A, the insertion support apparatus 5 displays the VBS image of the branch portion J on the display screen 6a during the image matching process.

  Next, the branch thumbnail VBS image is an image obtained by reducing and simplifying the VBS image. Note that the branch thumbnail VBS image display area 6E also has a function of a VBS image selection icon, and the VBS image displayed in the VBS image display area can be changed by selecting a desired branch thumbnail VBS image.

  The VBS schematic diagram image 6J is a figure showing contour lines of respective inner walls of a plurality of holes on the peripheral side of each branch portion formed based on the VBS image 6d, the branch thumbnail VBS image, or the three-dimensional bronchial image data, It is the image comprised from several outline figure. In addition, the endoscope schematic diagram image not displayed in FIG. 4 is a figure showing the contour lines of the inner walls of the plurality of holes on the distal side of each branch portion formed based on the endoscope image, that is, It is an image composed of a plurality of contour figures.

  The arrow image 6H, which is a moving direction display image, indicates the moving direction in which the endoscope 2A should be operated in order to match the center position of the branch portion of the endoscopic image 6f with the center position of the branch portion of the VBS image 6d. It is an arrow shown to the person. In the present embodiment, since the arrow image 6H connects the center position of the branch portion of the VBS image 6d from the center position of the branch portion of the endoscope image 6f, the arrow image 6H operates the endoscope 2A. Not only the direction of power movement but also the movement distance is shown.

  As a modification of the display screen 6a of this embodiment, FIG. 5 shows an endoscope image 6f, a virtual endoscope image 6d, a virtual endoscope schematic diagram image 6J, and an arrow image 6H on one screen. The superimposed state is shown. In FIG. 5, the virtual endoscopic image 6d is not displayed for easy understanding of the display. The virtual endoscopic image 6d is preferably displayed so as to be thinner and less conspicuous than the endoscopic image 6f so as not to hinder the visibility of the endoscopic image 6f.

  Next, FIG. 6 is a configuration diagram for explaining the configuration of the image processing unit 17. The image processing unit 17 extracts the branch line contour figure from the endoscopic image 6f, the VBS image, and the like, compares the similarity, etc., and the guide image such as the schematic diagram image from the contour line figure. An image generation unit 17b to generate, an image storage unit 17d for storing images, an image superimposition unit 17c which is an image composition unit for superimposing and displaying the endoscope image 6f and a guide image, and the like A navigation screen generation unit 17e that generates each display screen is provided.

  Next, a method for generating each image by the image processing unit 17 will be described. The VBS schematic diagram image 6J is formed based on the VBS image 6d, the branch thumbnail VBS image 6E, or the three-dimensional bronchial image data. As a forming method, there are a method by binarization using the VBS image 6d or the branch thumbnail VBS image 6E and a method of forming from a virtual cross-sectional image using three-dimensional bronchial image data.

<Method for Forming VBS Schematic Image from VBS Image by Binarization Process>
First, a method of forming a VBS schematic diagram image 6J by binarization processing will be described with reference to FIGS. FIG. 7 is a diagram for explaining the process of the binarization process, and FIG. 8 is a diagram for explaining the flow of the process of forming the VBS schematic diagram image 6J by the binarization process based on the branch thumbnail VBS image 6E. It is a flowchart of. The binarization process is a method of converting a gray scale image into a black and white image and extracting a contour line from the boundary of the black and white image. FIG. 8 is a flowchart showing the operation of the image processing unit 17 for forming the VBS schematic diagram image 6J of a plurality of branch portions on the incoming path.

  The image processing unit 17 extracts the contour line, that is, extracts the contour line of each hole from the gray-scale branch thumbnail VBS image 6E, and starts the formation of the VBS schematic diagram image 6J composed of a plurality of contour figures.

<Step S11>
The image processing unit 17 inputs 1 as the initial value of the number n of the branch thumbnail VBS image 6E to be selected in order to form the VBS schematic diagram images 6J of all the branch units J in the insertion path.

<Step S12>
As shown in FIG. 7A, the image processing unit 17 reads the nth branch thumbnail VBS image data.

<Step S13>
As shown in FIG. 7B, the image processing unit 17 extracts only pixels that are darker than a predetermined threshold from the pixel data of the branch thumbnail VBS image data, sets the pixel to black, A binarization process is performed for white pixels. Since the region at the back of the branching portion to be inserted, that is, the far region is captured as a relatively dark image, a contour image of the lumen cross section on the distal side of the branching portion ahead of the insertion direction can be obtained.

<Step S14>
As shown in FIG. 7C, the image processing unit 17 extracts the outermost contour line of each hole of the branch portion indicated by the binary branch thumbnail VBS image data. In FIG. 7 (C), two contour line figures composed of closed contour lines are extracted. A schematic diagram composed of the contour figure is a VBS schematic diagram image 6J.

<Steps S15 and S16>
The image processing unit 17 increments the number n by 1 and performs the same processing as above to form the VBS schematic diagram image 6J of each branch unit, and VBS schematic diagram images of all the branch units J in the insertion path When 6J is formed, the process is completed.

<Method 1 for forming VBS schematic diagram image from virtual cross-sectional image>
Next, two methods of forming the VBS schematic diagram image 6J from the virtual cross-sectional image using the three-dimensional bronchial image data will be described in order using FIG. 9 and FIG. 9 and 10 show views of the virtual three-dimensional image 100V observed from a viewpoint perpendicular to the lumen direction based on the three-dimensional bronchial image data in the vicinity of the branch portion J. This virtual image includes information on the inner wall 100VI of the bronchus and information on the core line 100V1 indicating the center of the bronchus. VY1 indicates a branch point on the core wire. The virtual camera V2a at the virtual endoscope distal end V2b is on the core line at a position of a predetermined distance L2 from the branch point VY1.

  In the method of forming the VBS schematic diagram image 6J from the first virtual cross-sectional image shown in FIG. 9, a predetermined value from the branch point VY1 on the line obtained by extending the core line 100V1 connecting the position of the virtual camera V2a and the branch point VY1 is predetermined. A cross-sectional image of the body cavity path including the points c1 and c2 where the plane perpendicular to the extension line including the point VL at the distance L1 intersects the body cavity path core line is used. That is, in FIG. 9A, a virtual cross-sectional image of the lumen inner wall including the point c1 displayed by the CI-CI line and a virtual lumen wall including the point c2 displayed by the CII-CII line. A virtual image obtained by capturing the cross-sectional image with the virtual camera V2a is a VBS schematic diagram image 6J including the contour line figure 6JI and the outline figure 6JII shown in FIG. 9B. Note that the distance L1 and the distance L2 are obtained as distances that can be reliably matched, and are set in advance.

<Method 2 for forming VBS schematic diagram image from virtual cross-sectional image>
In the method of forming the VBS schematic diagram image 6J from the second virtual cross-sectional image shown in FIG. 10, it is perpendicular to the body cavity path including the points c3 and c4 whose core line distance from the predetermined branch point VY1 is at the position of L3. Use cross-sectional images of directions. That is, in FIG. 10A, a virtual sectional image of the lumen inner wall including the point c3 displayed by the CIII-CIII line, and a virtual sectional image of the lumen inner wall including c4 displayed by the CIV-CIV line. Is a VBS schematic diagram image 6J composed of the contour line figure 6JIII and the outline figure 6JVI shown in FIG. 10B. The distance L1 and the distance L3 are obtained as distances that can be surely matched and are set in advance.

  As the contour line figure, the figure 6JI or the like obtained by the above processing may be further ellipsoidally approximated to be a simpler figure.

<First moving direction display image>
Next, formation of a moving display image will be described with reference to FIGS. FIGS. 11 to 13 are diagrams for explaining a method of forming a moving direction display image using a circumscribed circle of each contour line figure of each schematic diagram image, and FIGS. 14 and 15 illustrate the moving direction display image. FIG. 16 is a flowchart for explaining an operation flow of the image processing unit 17 for forming, and FIGS. 16 to 20 are diagrams illustrating an example of a display screen 6a displaying a moving display image.

  FIG. 11A is a diagram for explaining the first moving direction display image, and shows a live image display area 6F. The live image display area 6F includes a live image 6f and an endoscope model. The graphic image 6K and the circumscribed circle 6KC of the contour line figure constituting the endoscope schematic diagram image 6K are displayed in a superimposed manner.

  On the other hand, FIG. 11B shows a VBS image display area 6D. In this case, the VBS image display area 6D includes a VBS image 6d, a virtual endoscope schematic diagram image 6J, and a virtual endoscope model. A circumscribed circle 6JC of a contour line figure constituting the figure image 6J is superimposed and displayed. That is, it corresponds to the display layer P4 shown in FIG. Since the layer P4 and the layer P2 in FIG. 12 are linked, if any layer is rotated, the other layers also perform the same rotation.

  FIG. 11C shows a live image display area 6F. The live image display area 6F includes a live image 6f, an endoscope schematic diagram image 6K, a circumscribed circle 6KC, and a virtual endoscope schematic diagram. The image 6J and the circumscribed circle 6JC are superimposed and displayed, but the live image 6f is not displayed for the sake of explanation. That is, as shown in FIG. 12, a display layer P1 that displays the circumscribed circle 6KC, a display layer P2 that displays the circumscribed circle 6J, and a display layer P3 that displays the live image 6f and the endoscope schematic diagram image 6K. It is an image displayed in a superimposed manner. The arrow image 6h shows the direction and distance from the center of the circumscribed circle 6KC to the center of the circumscribed circle 6JC.

  FIG. 16 shows a display screen 6a in which two circles 6KC and 6JC, which are moving direction display images, are superimposed on the live image 6f. As shown in FIG. 16, in the VBS image display area 6D, a marker 6G indicating a path hole for proceeding along the insertion path is superimposed on the VBS image 6d. As long as the marker 6G can clearly indicate the path hole to be advanced, various marker displays such as emphasizing the outline of the hole or displaying it in a color that easily attracts the attention of a specific operator can be used. .

  Next, the operation flow of the image processing unit 17 will be described with reference to the flowchart of FIG.

  When the display of the live image 6f starts, the operation of the image processing unit 17 starts.

<Step S31>
The image processing unit 17 binarizes the VBS image selected by the icon or the like in the branch thumbnail VBS image display area 6E, extracts a contour line figure, and forms a virtual endoscope schematic diagram image 6J.

<Step S32>
The image processing unit 17 obtains a circumscribed circle 6JC of the extracted outline figure. Here, the circumscribed circle refers to a circle with the smallest diameter that includes all contour line figures.

<Step S33>
The image processing unit 17 copies the circumscribed circle 6JC to the layer P2, and superimposes it on the live image display layer P3.

<Step S34>
The image processing unit 17 binarizes the endoscope image, extracts the contour line figure, and forms the endoscope schematic diagram image 6K. Here, the endoscopic image is a stationary endoscopic image.

<Step S35>
The image processing unit 17 obtains a circumscribed circle 6KC of the extracted contour figure.

<Step S36>
The image processing unit 17 copies the circumscribed circle 6KC to the layer P1 and superimposes it on the live image display layer P3.

<Step S37>
The image processing unit 17 generates a static endoscope image from the live image until another VBS image is selected, performs the process again, and always sets the position of the circumscribed circle 6KC in a new state as the live image display layer P3. Superimpose.

<Step S38>
When the image processing unit 17 reaches another branching unit and a different VBS image is selected, the image processing unit 17 continues the processing based on the virtual endoscopic image of the branching unit. The image processing unit 17 continues to operate until there is an end signal.

  As shown in FIG. 16, the surgeon determines the endoscope 2A so that the center of the circumscribed circle 6KC of the branch portion of the endoscopic image coincides with the center of the circumscribed circle 6JC of the branch portion of the virtual endoscopic image. To operate. Then, as shown in FIG. 17, the operator can easily match the center positions of the branch points of the live image 6f and the virtual endoscopic image 6d. That is, these two circles 6KC and 6JC are the first moving direction display images. When the center positions of the two circles coincide with each other, the image processing unit 17 performs matching processing without malfunction and performs rotation processing for rotating the virtual endoscopic image 6d. After the rotation processing, as shown in FIG. 18, since the display directions of the virtual endoscopic image 6d and the live image 6f coincide with each other, the operator 6G indicates a path hole for proceeding along the insertion path. And the live image 6f can be easily recognized.

<Second moving direction display image>
Next, an arrow image that connects the center points of two circumscribed circles that are the second movement direction display images will be described with reference to FIGS. 11, 13, 15, and 19.

  FIG. 11D shows a live image display area 6F. In the live image display area 6F, an arrow image 6H is superimposed on the live image 6f, but the live image 6f is displayed for the sake of explanation. Not done. In the image of FIG. 11D, as shown in FIG. 13, three display layers P1, P2, and P3 are superimposed and displayed. FIG. 19 shows the display screen 6a, and an arrow image 6H superimposed on the live image 6f is an arrow connecting the center point 6Kc of the circumscribed circle 6KC and the center point 6Jc of the circumscribed circle 6JC. On the other hand, in the VBS image display area 6D, a marker 6G or the like indicating a path hole for proceeding along the insertion path is superimposed and displayed on the VBS image 6d.

  Next, the operation flow of the image processing unit 17 will be described with reference to the flowchart of FIG. Steps 51 to 55 are the same as steps 31 to 35 in the flowchart of FIG.

<Step 56>
An arrow image 6H is generated from the center point 6Kc of the circle 6KC to the center point 6Jc of the circle 6JC.

<Step 57>
The arrow image 6H is displayed on the layer P2, and is displayed superimposed on the live image display layer P3.

13 and 15, the circle 6KC and the circle 6JC are also superimposed on the live image display layer P3 together with the arrow image 6H, but only the arrow image 6H is displayed as a live image as shown in FIG. It may be displayed superimposed on the layer P3. Alternatively, as shown in FIG. 4, the arrow image 6H and the virtual endoscope schematic diagram image 6J may be superimposed and displayed on the live image display layer P3, or the endoscope schematic diagram image 6K may be displayed. The circle 6KC is a figure generated from the endoscopic image 6f, and is not generated from the virtual endoscopic image 6d. However, one guide image is formed by combining with the circle 6JC generated from the virtual endoscopic image 6d.

<Third moving direction display image>
Next, a method for forming a moving display image using the barycentric point of the branch part as the third moving display image will be described. FIG. 20 illustrates a method of forming a moving display image using the barycentric point of the branching part. FIG. FIG. 20A shows a live image display area 6F. In this case, the live image display area 6F is divided into a live image 6f and an endoscope schematic diagram image 6K and an endoscope schematic diagram image 6K. A mark 6M indicating the position of the center of gravity of the part is superimposed and displayed. Here, the endoscope schematic diagram image 6K is formed by a method similar to the method of forming the VBS schematic diagram image 6J from the various virtual images described above.

  On the other hand, FIG. 20B shows a VBS image display area 6D. In this case, the VBS image display area 6D includes a VBS image 6d, a virtual endoscope schematic diagram image 6J, and a virtual endoscope model. A mark 6N indicating the position of the center of gravity of the branching portion of the graphic image 6J is superimposed and displayed. Here, the barycentric position is a position determined based on the shading pattern of the live image or the VBS image. The binarization process for converting the gray scale image into a black and white image is performed, and the total number of black pixels in the entire image. It can be obtained from the average value of the coordinates of each black pixel when the number falls below a certain ratio.

Hereinafter, the operation of the determination method of the center of gravity position of the branching portion by the image processing unit 17 will be described in detail with reference to FIG.
<Step 61>
The image processing unit 17 sets a predetermined initial value t ₀ as a threshold value when performing binary processing for converting a live image or VBS image into a monochrome image.

<Step 62>
The image processing unit 17, a threshold value as an initial value t _0, performs a binary process, calculates the total number n of black pixels.

<Step 63, Step 64>
The image processing unit 17 determines whether the total number n of black pixels calculated in step 62 is larger or smaller than a predetermined ratio K (for example, 20%) with respect to the number of pixels of the entire screen. After the threshold value is decreased by the minute value Δt, the process returns to step 62 again to calculate the total number n of black pixels.

  That is, in the state of passing through step 63, the threshold value is set so that the total number n of black pixels is smaller than a predetermined ratio (for example, 20%) with respect to the number of pixels of the entire screen. Note that a predetermined ratio K is selected such that when a normal live image or VBS image is binarized, the shape of the bronchial branch is partially extracted.

<Step 65, Step 66>
In step 63, the image processing unit 17 determines whether the total number n of black pixels is larger or smaller than a predetermined ratio K (for example, 20%) with respect to the number of pixels of the entire screen. Calculates the total number n of black pixels after increasing the threshold value by a minute value Δt.

<Step 67, Step 68>
The image processing unit 17 determines whether the total number n of black pixels is larger or smaller than a predetermined ratio K (for example, 20%) with respect to the total number of pixels on the screen, and until the ratio of black pixels exceeds K. repeat.

<Step 69>
The image processing unit 17 obtains the coordinate values a1 (x1, y1) and a2 (x2, y2) to an (xn, yn) on the respective screens of the black pixels a1, a2 to an, An average value A ((x1 + x2 +, + xn) / n, (y1 + y2 +, + yn) / n) is calculated and set as the gravity center position N or M of the branch portion.

  FIG. 20C shows a live image display area 6F. In the live image display area 6F, the live image 6f and the arrow image 6H are superimposed and displayed. Is not displayed.

  Similarly to the moving direction display image formed from the circumscribed circle, the surgeon operates the endoscope 2A to move the endoscope distal end portion 2b in the moving direction indicated by the arrow image 6H. When the movement amount indicated by 6H is moved, the center positions of the branch points of the live image 6f and the virtual endoscopic image 6d can be easily matched.

  In addition to the display of the moving direction display image such as an arrow image, information on the amount of movement of the endoscope tip that is actually necessary may be displayed. For example, it is information that the center positions of the live image 6f and the virtual endoscopic image 6d coincide with each other by how many mm in each of the X direction and the Y direction.

<Navigation by template image 6T>
As described above, the center position of the branch portion of the endoscopic image 6f is determined by the operation of the endoscope according to the navigation of the guide image such as the moving direction display image, at the branch portion of the virtual endoscopic image 6f. If they coincide with the center position, both images are easily matched by the image processing unit 17. For this reason, the image processing unit 17 performs matching without malfunction, and as a result, the virtual endoscopic image 6d can be automatically rotated. However, matching may not be possible even after the above operation.

  On the other hand, even if the rotation process is not performed, even when the directions of the endoscope image 6f and the virtual endoscope image 6d are matched to some extent from the beginning, the image processing unit 17 may fail to match. .

  This is because the matching processing by the image processing unit 17 is restricted from the viewpoint of the matching processing speed, the cost of the insertion support apparatus, and the like. Specifically, the position in the bronchus of the camera 2a of the endoscope 2A is shifted in the vertical and horizontal directions (XY directions) with respect to the virtual bronchus position of the virtual camera V2a that captured the virtual endoscopic image. There are also restrictions on the range that can be matched. Further, the position of the endoscope 2A in the bronchus of the camera 2a is shifted in the depth direction (Z direction) of the bronchial lumen with respect to the virtual bronchus position of the virtual camera V2a that captured the virtual endoscopic image 6d. Even in this case, there is a restriction on the range that can be matched. Furthermore, as described above, the range in which the camera 2a of the endoscope 2A can be matched is also limited with respect to the imaging condition of which direction in the space is upward.

  FIG. 22 is a diagram for explaining the relationship between the position in the bronchus of the camera 2a of the endoscope 2A and the endoscopic image 6f. 22A shows the position of the endoscope 2A in the bronchus 100A, FIG. 22A shows the virtual endoscopic image 6d, and FIG. 22C shows the endoscopic image 6f. . The camera 2a of the endoscope 2A is located at a position 100I corresponding to a location where the virtual camera V2a of the virtual endoscope V2A images the virtual endoscopic image 6d. With the endoscope 2A in the state of FIG. 22A, the virtual endoscopic image 6d and the endoscopic image 6f can be matched only by rotating the virtual endoscopic image 6d.

  On the other hand, in the endoscope 2A in the state shown in FIG. 22B, there is a shift in the depth direction and the vertical direction. For this reason, 6K which shows the hole of the branch part in an endoscopic image is smaller than 6L which shows the hole of the branch part in a virtual endoscopic image, and the center of a branch part has shifted | deviated a little more. In the endoscope 2A in the state shown in FIG. 22C, the central axis is deviated. For this reason, the center of the branching portion is shifted in 6K indicating the hole of the branching portion in the endoscopic image. Further, in the endoscope 2A in the state shown in FIG. 22D, the direction of the endoscope 2A is shifted. For this reason, the center of the branching portion is shifted in 6K indicating the hole of the branching portion in the endoscopic image.

  When the position of the endoscope 2A is deviated from the virtual endoscope image capturing position of the virtual endoscope V2A, the surgeon moves the endoscope 2A from the state 2A1 in FIG. It was necessary to operate to become a state. However, the surgeon had to make trial and error because it was unclear how far the matching could be performed.

  The template image 6T in the endoscope apparatus 1 according to the present embodiment has a specific range of the above-described restrictions, in other words, a range that can be matched within the range. It is a guide image shown in FIG.

  Hereinafter, template figures will be described with reference to FIGS. FIG. 23 is a flowchart showing an operation flow of the image processing unit 17 for generating a template graphic, and FIGS. 24 and 25 are diagrams for explaining a method of generating the template image 6T.

  First, the flow of the template generation operation by the image processing means of the image processing unit 17 will be described with reference to the flowchart of FIG.

<Step S71>
The image processing unit 17 varies within a range in which the position and direction of the virtual camera V2a of the virtual endoscope are set, and generates a plurality of virtual endoscope images. For example, as illustrated in FIG. 24, the image processing unit 17 changes the camera direction to the vertical direction and the horizontal direction with respect to the image 6dS of the center condition, and generates eight virtual endoscope images 6d10 to 6d17. . The image processing unit 17 further performs a matching process on the eight virtual endoscopic images 6d10 to 6d17 with the image 6dS of the central condition. When matching is possible, the camera direction is further changed. When matching is not possible, the change in camera direction is reduced. By repeating this process, it is possible to obtain a virtual endoscopic image when the maximum vertical and horizontal shifts in the camera direction that can be matched have occurred.

<Step S72>
The image processing unit 17 generates each virtual endoscope schematic diagram image 6J from each virtual endoscope image 6d.

<Step S73>
The image processing unit 17 sets an area that includes all the contour line figures by superimposing all the virtual endoscope schematic diagram images 6J.

<Step S74>
The image processing unit 17 records the outer edge of the region including all the contour line figures as the template image 6TA. As shown in FIG. 24, the template image 6TA is a template image that is formed of a polygon, includes all contour lines in the schematic diagram of the virtual endoscope, and shows the outer edge of the matching range.

  In the above description, the camera direction is changed. Similarly, the template image 6T indicating the matching range can be generated by performing the same process for the camera position and the rotation direction. That is, the template image 6T of the present embodiment is a schematic diagram of a plurality of virtual endoscopes generated from virtual endoscopic images at different positions and / or directions of virtual imaging means based on the virtual images. The image processing means recognizes that the standard virtual endoscope schematic diagram generated from the virtual endoscopic image captured under the standard conditions of the position and direction of the imaging means and the virtual endoscope schematic diagram of the same branching portion It is an image including contour lines constituting a plurality of virtual endoscope schematic diagrams that can be matched, that is, matched. Here, the standard condition of the position and direction of the virtual imaging means is a position on the core line of the lumen where the shape of the branching portion can be easily recognized and is parallel to the core line.

  The surgeon can quickly perform the matching process by operating the endoscope 2A so that the template image 6T displayed superimposed on the live image 6f includes a hole in the branching portion. .

<First Modification of Template Image 6T>
Next, a first modification of the template image 6T will be described with reference to FIG. FIG. 25 is a diagram for explaining a method of generating the template image 6T when the camera position is shifted.

  As shown in FIG. 25, the image processing unit 17 changes the camera position, that is, the position in the depth direction of the lumen path with respect to the image 6dS of the central condition, Then, a virtual endoscopic image 6d19 in the case of being far from the branching unit is generated. The image processing unit 17 further matches the virtual endoscopic image 6d18 or the virtual endoscopic image 6d19 with the image 6dS of the central condition, and when the matching can be performed, the camera position is further changed. Reduce changes in camera position. By repeating this process, a virtual endoscopic image 6d at the closest and farthest positions of the camera positions that can be matched is obtained.

  A range surrounded by the outer edge 6Tb1 of the virtual endoscopic image 6d at the closest position and the outer edge 6TB2 of the virtual endoscopic image 6d at the farthest position is a template figure 6TB indicating a matching range. The template figure 6TB is a constituent element of the template image 6T showing the outer edge and inner edge of the range that can be matched, and is constituted by a quadrangle that includes all the outline figure of the virtual endoscope schematic diagram.

<Second Modification of Template Image 6T>
Next, a second modification of the template image 6T will be described using FIG. The operator may repeatedly puncture the endoscope 2A several times when inserting the endoscope. That is, the endoscope is inserted into the same branch portion a plurality of times. The three images in FIG. 26A show three virtual endoscope schematic diagram images 6J20, 6J21, and 6J22 that the image processing unit 17 was able to match with the endoscope image 6f in the past. FIG. 26B shows a state in which individual contour line figures indicating two holes in the three virtual endoscope schematic diagram images are superimposed. FIG. 26 (C) shows a state in which the outer edge of a range including each of the contour lines indicating the two holes is approximated by a curve, and FIG. 26 (D) is configured by the curve 2. A template image 6T showing a range in which matching is possible is shown by two contour lines. In the present modification, the image shown in FIG. 26D is used as a template image 6T that is superimposed on the live image 6f.

  That is, FIG. 26 (D) is configured by a curve including individual outline figures of a plurality of virtual endoscope schematic diagram images 6J that can be matched by the image processing unit 17 in the same branch portion. This is a template image 6T.

  Since the template image 6T of this modification is generated based on the virtual endoscope schematic diagram image 6J that has been actually matched, the accuracy of the matching range is high. For this reason, the insertion operation of the endoscope 2A becomes quick.

  Note that the image shown in FIG. 26B can also be used as a template image.

That is, it is an image obtained by superimposing individual contour lines indicating two holes in three virtual endoscope schematic diagram images that have been successfully matched in the past. The shape of the part does not change that much. For this reason, it is also possible to generate a template image of each branch part from matching data of a plurality of subjects performed in the past by the same method.

<Third Modification of Template Image 6T>
Next, a third modification of the template image 6T will be described using FIG. As described in the above modification, the operator may repeatedly puncture the endoscope 2A several times when inserting the endoscope. The three images in FIG. 27A show three virtual endoscope schematic diagram images 6J30, 6J31, and 6J32 that the image processing unit 17 has been able to match with the endoscope image 6f in the past. Among the three images, the virtual endoscope schematic diagram image 6J32 is an image in the previous matching process. In this modification, the virtual endoscope schematic diagram image 6J32, which is the template image 6T matched by the image processing unit 17 last time, is used as the template image 6T to be superimposed on the live image 6f.

  Note that the template image 6T of this modification is not a template image strictly because it does not indicate a matching range, but is a guide image that exhibits the same effects as the template image, and is referred to as a template image here. Since the template image 6T of the present modification is a virtual endoscope schematic diagram image immediately before the actual matching, the possibility of matching in the vicinity thereof is very high. For this reason, by using this template image as a guide image, the insertion operation of the endoscope 2A becomes quick.

<Different use of guide images>
The surgeon can use different guide images depending on the insertion state. For example, when the relationship between the endoscopic image 6f and the virtual endoscopic image 6d is deviated so as not to be recognized at all when the branch portion is reached, the surgeon first displays the moving direction display image as a guide image. It is good to choose. Then, when the surgeon operates the endoscope 2A according to the navigation of the moving direction display image and the center position of the branch portion is reached, the live image 6f and the virtual endoscopic image 6d, more precisely, the frame image of the live image and When each of the schematic diagram images of the virtual endoscopic image is matched by the hole shape and the matching is successful, the image correcting unit performs the virtual endoscopic image 6d so that the directions in which the plurality of holes in both the images are aligned match each other. Rotate the

  Even after the rotation processing, if the matching of the same branch portion cannot be performed because the distance to the branch portion is long, the operator may select a template image as the guide image. At this stage, at least the center positions of the branching portions of both images coincide with each other, so that the moving direction display image is not necessary and is not selected as the guide image. Then, when the surgeon operates the endoscope 2A according to the navigation of the template image and introduces a branching portion in the endoscopic image within the matching range indicated by the template figure, the matching process is performed. It is confirmed that the insertion is performed.

  The two matching processes in the above case are matching processes with different matching accuracy, the matching process for the first rotation process is a simple matching process, and the subsequent matching of whether or not the same branching part is complicated Matching process. In simple matching processing, images can be automatically sampled from live images in frame units and used as matching endoscopic images for matching processing. However, in complex matching processing, the operator stops the endoscope and presses the button. It is preferable to take a still image by operating.

<Fixing means>
Next, in the endoscope apparatus 1 according to the embodiment of the present invention, the endoscope apparatus 1 having fixing means capable of fixing the distal end portion 2b and the body cavity path to the distal end portion 2b of the endoscope 2A will be described. FIG. 28 is a view for explaining the distal end portion 2b of the endoscope 2A inserted into the lumen of the bronchus 100, FIG. 29 is a block diagram showing the configuration of the fixing means, and FIG. It is a flowchart for demonstrating the flow of operation | movement of the fixing device 2c.

  As shown in FIG. 28, the distal end portion 2b of the endoscope 2A inserted up to the bronchial bifurcation 100D is supplied with fluid to a balloon 2C which is a fixing means capable of fixing the bronchial body cavity and expanded. Endoscope tip 2b is fixed to the bronchial tube wall by balloon 2C. The balloon 2C is made of silicone rubber or the like that is a material rich in stretchability, and carbon dioxide or air can be used as the fluid.

  As shown in FIG. 29, the fixing device 2c of the endoscope apparatus 1 according to the embodiment of the present invention includes a balloon 2C that is a fixing tool, a fluid supply unit 2D that supplies a fluid to the balloon 2C, and a fixing device. Control unit 2E, and three operation buttons (start button 2F1, stop button 2F2, release button 2F3) for the surgeon to instruct the operation to the control unit 2E. The control unit 2E of the fixing device is connected to the control unit 5B of the insertion support device 5 and the control unit 2L of the bronchoscope 2 by a signal cable.

  Next, the operation flow of the fixing means will be described with reference to FIG.

  When the endoscope distal end portion 2b is inserted to a predetermined position of the branching portion 100D, the fixing means starts operating.

<Step S82>
The fixing device 2c stands by until there is a signal input from the start button 2F1.

<Step S83>
When there is a signal input from the start button 2F1, the fluid control unit 2D starts injecting fluid into the balloon 2C.

<Steps S84, S85, S86>
When there is a signal from the stop button 2F2 (S84) or when the fluid pressure is equal to or higher than a predetermined value (S85), the fluid control unit 2D stops fluid injection into the balloon 2C (S86).

<Step S87>
Since the fixing device 2c is in a state in which the balloon 2C is inflated and the endoscope distal end portion 2b is fixed to the inner wall of the bronchus, the fixing completion signal is sent to the control unit 5B of the insertion support device 5 and the bronchial endoscope device control. Transmit to unit 2L. In response to this fixing completion signal, the bronchoscope control unit 2L captures an endoscope image 6f by the camera 2a and transmits it to the insertion support device 5. The insertion support apparatus 5 compares the endoscopic image 6f with a virtual endoscopic image or the like and performs matching processing.

<Step S88>
When the matching process is completed, the insertion support device 5 transmits a matching end signal to the fixing device 2c.

<Step S89>
When receiving the matching end signal, the fixing device 2c releases the fluid to the atmosphere. Then, the balloon 2C is deflated, and the fixation of the endoscope distal end portion 2b to the inner wall of the bronchus is released. The fixation of the endoscope distal end portion 2b is also released by a signal input from the release button 2F3.

  The endoscope distal end portion 2b is in a freely moving state, and the insertion of the endoscope is continued. When the endoscope distal end portion 2b is inserted to the predetermined position of the next branching portion, the operation from step S81 is performed again.

  In the endoscope apparatus 1 according to the embodiment of the present invention, when the image of the branching unit 100D is captured by the camera 2a of the endoscope 2A, a clear image can be captured without camera shake. Further, there is no time lag between the time when the surgeon operates the imaging switch of the camera and the time when the actual imaging is performed, that is, a time lag of the imaging target does not occur. For this reason, the accuracy in image processing is improved, and the endoscope can be more reliably navigated to the target site.

  Further, while the image processing or the like by the insertion support device 5 is being performed, or while the operator is interrupting the insertion, the distal end portion of the endoscope that is not intended by the operator due to the patient's cough or the like The position 2b does not move. For this reason, it is possible to more quickly navigate the endoscope to the target site.

  The balloon 2C2b may be incorporated in an endoscope accessory such as an overtube and used in combination with the endoscope 2A, instead of being arranged at the distal end portion of the endoscope.

<Distance measuring means>
Next, in the endoscope apparatus 1 according to the embodiment of the present invention, the distance measurement for guiding the camera 2a that captures the endoscopic image 6f of the branching portion of the bronchus to a predetermined distance from the branching point of the branching portion 100D. Means 2G, 2H, 2I, 2J may be provided. FIG. 31A is a diagram for explaining a state in which the endoscope 2A provided with the distance measuring means 2G is inserted into the branching section 100D, and FIG. 31B is a distance measuring means 2G having a different shape. FIG. 31C is a diagram for explaining a state in which the endoscope 2A including the non-contact distance measuring unit 2J is inserted into the branching unit 100D. FIG. 31D is a diagram showing an example of an endoscopic image 6f when the distance to the branch portion is measured by the non-contact distance measuring means 2J.

  As described above, in order to efficiently match the virtual endoscopic image 6d and the endoscopic image 6f, the position of the camera 2a that captures the endoscopic image 6d is at a predetermined distance from the branch point. This is very important. If the position of the camera 2a is closer than a predetermined distance from the branch point, the hole shape of the branch portion is too large, and if the position of the camera 2a is farther than the predetermined distance from the branch point, the hole shape of the branch portion is too small. However, the insertion support apparatus 5 may not be able to perform matching well.

  As shown in FIG. 31A, the endoscope apparatus 1 can measure the distance from the camera 2a that captures the endoscopic image 6f of the bronchial bifurcation 100D to the inner wall of the bifurcation point of the bifurcation 100D. It has measuring means 2G. As shown in FIG. 31B, the distance measuring instrument 2G is an elongated instrument in which the tip 2G2 is rounded and can be output from the endoscope tip through the forceps channel of the endoscope 2A. The distance measuring instrument 2G can output from the endoscope tip by a predetermined length by the positioning fastener 2G1, and can also measure the output length. For this reason, in the endoscope apparatus 1 of the present embodiment, the endoscope distal end portion 2b can be accurately guided to a predetermined distance from the branch point. For this reason, it is possible to easily and quickly capture the endoscopic image 6f that is easily matched with the virtual endoscopic image 6d.

  As the distance measuring means, an attachment type distance measuring instrument 2H shown in FIG. 31B or a built-in type distance measuring instrument 2I can also be used. Alternatively, it can be used as a distance measuring instrument by a method such as calibrating a known treatment instrument having other current applications.

  The distance measuring means is preferably a means that does not obstruct the field of view of the camera 2a of the endoscope 2A, and the non-contact distance measuring means 2J is particularly preferable. FIG. 31C shows an endoscope 2A having non-contact distance measuring means 2J. The non-contact distance measuring means 2J is a means for measuring the distance by transmitting infrared rays and measuring the reflected wave. Since the non-contact distance measurement 2J uses waves outside the visible light region such as infrared rays and ultrasonic waves, it may not be easy to determine which distance is actually measured. For this reason, it is preferable to display a measurement location with visible light as shown in FIG. FIG. 31D is an example of an endoscopic image 6f, and a measurement point that displays a distance measurement location with visible light on a branch point inner wall of a branch portion 100D that branches into a pipeline 6J1 and a pipeline 6J2. The mark 2K is displayed.

  In addition, the insertion support apparatus 5 performs control so that a still endoscope image can be captured only when the camera 2a and the branch point are at an appropriate distance from the measurement result of the distance measuring instrument, or at an appropriate distance. It is also possible to display on the display screen 6a that the range has been entered. Conversely, the insertion support device 5 can also alert the operator by issuing a warning if it is outside the appropriate distance range.

<Additional notes>
An endoscope insertion support method, a virtual image generation step of generating a virtual image of a body cavity in the subject based on image data of a three-dimensional region of the subject, and the body cavity path based on the virtual image A guide image generation step of generating a guide image of the branch part, an endoscope image of the body cavity path, and the guide image of the next branch part distal to the imaging position of the endoscope image in the body cavity path And an image processing step, wherein the guide image is one or more images selected from the virtual endoscope image, the virtual endoscope schematic diagram image, the moving direction display image, and the template image. The image processing step includes an image correction step of matching the display direction of the endoscopic image of the branching unit with the display direction of the virtual endoscopic image and the guide image, and the image correction step Is at least said The endoscope insertion aid method for rotating the one or more guide image including a virtual endoscopic image.

  The present invention is not limited to the above-described embodiments and modifications, and various changes and modifications can be made without departing from the scope of the present invention.

It is a lineblock diagram showing the composition of the endoscope apparatus of a 1st embodiment of the present invention. It is a figure which shows the example of the display screen which displayed the insertion path | route. It is a figure which shows the example of the display screen which displayed the insertion path | route. It is a figure which shows the display form at the time of the insertion navigation of the display screen of a monitor. It is a figure which shows the modification of the display form at the time of the insertion navigation of the display screen of a monitor. It is a block diagram for demonstrating the structure of an image process part. It is a figure for demonstrating the process of a binarization process. It is a flowchart for demonstrating the flow of the formation process of the VBS schematic diagram image by the binarization process based on the branch thumbnail VBS image. The figure which observed the virtual three-dimensional image from the viewpoint perpendicular | vertical to a luminal direction based on the three-dimensional bronchial image data near a bifurcation part is shown. The figure which observed the virtual three-dimensional image from the viewpoint perpendicular | vertical to a luminal direction based on the three-dimensional bronchial image data near a bifurcation part is shown. It is a figure for demonstrating the method of forming the moving direction display image using the circumscribed circle of the outline figure of a schematic diagram image. It is a figure for demonstrating the method of forming the moving direction display image using the circumscribed circle of the outline figure of a schematic diagram image. It is a figure for demonstrating the method of forming the moving direction display image using the circumscribed circle of the outline figure of a schematic diagram image. It is a flowchart for demonstrating the flow of operation | movement of the image process part for forming a moving direction display image. It is a flowchart for demonstrating the flow of operation | movement of the image process part for forming a moving direction display image. It is a figure which shows the example of the display screen which displayed the movement display image. It is a figure which shows the example of the display screen which displayed the movement display image. It is a figure which shows the example of the display screen which displayed the movement display image. It is a figure which shows the example of the display screen which displayed the movement display image. It is a figure which shows the example of the display screen which displayed the movement display image. It is a flowchart which shows the flow of operation | movement of the image process part for center-of-gravity position determination. It is a figure for demonstrating the relationship between the position in the bronchi of the camera of an endoscope, and an endoscopic image. It is a flowchart which shows the flow of operation | movement of the image process part for producing | generating a template figure. It is a figure for demonstrating the production | generation method of the template image 6T. It is a figure for demonstrating the production | generation method of the template image 6T. It is a figure for demonstrating the production | generation method of the template image 6T. It is a figure for demonstrating the production | generation method of the template image 6T. It is a figure for demonstrating the front-end | tip part of the endoscope inserted in the lumen of a bronchus. It is a block diagram which shows the structure etc. of a fixing means. It is a flowchart for demonstrating the flow of operation | movement of a fixing device. It is a figure for demonstrating a distance measurement means. It is a figure which shows the display screen of the conventional endoscope apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Endoscope apparatus, 2 ... Bronchoscope apparatus, 2A ... Endoscope, 2C ... Balloon, 2b ... End-of-endoscope part, 5 ... Insertion support apparatus, 6 ... Monitor, 6F ... Live image display area, 6G ... Marker, 6H ... arrow image, 6J ... virtual endoscope schematic diagram image, 6JC, 6KC ... circumscribed circle, 6Jc, 6Kc ... center point, 6a ... display screen, 6d ... virtual endoscopic image, 6f ... endoscopic image 100, bronchi, P1, P2, P3, P4 ... layer, R ... insertion path, V2a ... virtual camera, V2b ... virtual endoscope tip, VBS image ... virtual endoscopic image

Claims (22)

Virtual image generating means for generating a virtual image of a body cavity in the subject based on image data of a three-dimensional region of the subject;
Based on the virtual image, a guide image generating means for generating a guide image of a branch portion of the body cavity path;
An image synthesis means for superimposing and displaying the endoscopic image of the body cavity path and the guide image of the next branching portion on the distal side from the imaging position of the endoscopic image in the body cavity path;
The endoscope apparatus, wherein the guide image is one or more images selected from the virtual endoscopic image, a virtual endoscope schematic diagram image, a moving direction display image, and a template image. Comprising image processing means,
The image processing means includes image correction means for matching the display direction of the endoscopic image of the branching unit with the display direction of the virtual endoscopic image and the guide image, and the image correction means includes at least the image correction means The endoscope apparatus according to claim 1, wherein the endoscope apparatus rotates one or more guide images including a virtual endoscopic image.   The endoscope apparatus according to claim 1, wherein the image synthesizing unit superimposes and displays the endoscope image and an endoscope schematic diagram image. Virtual endoscopic image generation means for generating a virtual endoscopic image in a body cavity in the subject based on image data of a three-dimensional region of the subject;
One or more images selected from a virtual endoscope schematic diagram image, a moving direction display image, and a template image based on the virtual endoscopic image of the next branching portion in the body cavity path with respect to the endoscope image in the body cavity path Guide image generation means for generating
Image composition means for superimposing and displaying the endoscopic image and the guide image;
Comprising image processing means,
The image processing means includes image processing means for rotating the virtual endoscopic image based on a comparison between the endoscopic image in the branching unit and the virtual endoscopic image. Mirror device.   The said endoscope schematic diagram image is comprised by the outline figure which shows the outline extracted by the binarization process of the said endoscope image, The Claim 3 or Claim 4 characterized by the above-mentioned. Endoscopic device.   The virtual endoscope schematic diagram image is an outline obtained by capturing virtual cross-sectional images of a plurality of holes on the distal side of the branching portion from the position of the virtual imaging means on the body cavity core line located outside the body cavity from the branching portion. The endoscope apparatus according to any one of claims 1 to 5, wherein the endoscope apparatus includes an outline figure indicating   A contour showing a contour of a cross section including a point where a plane perpendicular to the straight line intersects the body cavity core line on an extension of a straight line connecting the imaging means position and the branch point of the branch portion, the virtual cross-sectional image The endoscope apparatus according to claim 6, wherein the endoscope apparatus is configured by a line figure.   The endoscope apparatus according to claim 6, wherein the virtual cross-sectional image is configured by a contour figure indicating a contour of a cross section including a point having the same core distance from the branch point.   6. The virtual endoscope schematic diagram image is configured by a contour line figure showing a contour line extracted by binary processing based on the virtual endoscopic image. The endoscope apparatus described in 1.   2. The moving direction display image is an arrow from a center position of the branch portion of the endoscope schematic diagram image to a center position of the branch portion of the virtual endoscope schematic diagram image. The endoscope apparatus according to any one of claims 9 to 9.   The endoscope apparatus according to claim 10, wherein the center position of each of the endoscope schematic diagram image and the virtual endoscope schematic diagram image is a gravity center position of each of the branch portions.   11. The center position of each of the endoscope schematic diagram image and the virtual endoscope schematic diagram image is a center point of a circumscribed circle including each of the contour line figures. Endoscope device.   10. The moving direction display image is a circumscribed circle including the contour line figures of the endoscope schematic diagram image and the virtual endoscope schematic diagram image, respectively. The endoscope apparatus according to any one of the above.   The template image is the virtual endoscopic image based on the virtual image at a position of different virtual imaging means and / or a direction of the virtual imaging means, and a plurality of virtual endoscopes that can be matched by the image processing means The endoscope apparatus according to any one of claims 1 to 13, wherein the endoscope apparatus includes an image including each contour line figure in the mirror schematic diagram.   The template image is an image including individual contour lines of the plurality of virtual endoscope schematic diagram images that can be matched by the image processing means in the same branch portion. The endoscope apparatus according to any one of claims 2 to 13.   The endoscope apparatus according to claim 15, wherein the template image is an image including all contour lines of the virtual endoscope schematic diagram.   The endoscopic device according to any one of claims 14 to 16, wherein the template image shows an outer edge of the matching range.   The endoscope apparatus according to any one of claims 14 to 16, wherein the template image shows an outer edge and an inner edge of the matching range.   The endoscope apparatus according to any one of claims 1 to 18, wherein the guide image is the guide image previously matched by the image processing unit in the same branch portion. Comprising imaging means for imaging the body cavity path;
The endoscope apparatus according to any one of claims 1 to 19, wherein the endoscope includes a fixing unit capable of fixing the distal end portion and the body cavity path to a distal end portion.   21. The endoscope apparatus according to claim 20, wherein when the imaging unit captures an endoscopic image, the distal end portion is fixed to the body cavity by the fixing unit.   The guide image selected first is one or more images including the moving direction display image, and the guide image selected after the rotation processing of the guide image by the image correction unit includes the template image. The endoscope apparatus according to any one of claims 2 to 21, wherein the endoscope apparatus is one or more images not including the moving direction display image.