JP7071240B2 - Inspection support equipment, methods and programs - Google Patents

Description

The present disclosure relates to an inspection support device, method and program for supporting an inspection using an endoscope or the like on a tubular structure such as a bronchus.

In recent years, attention has been focused on techniques for observing and treating tubular structures such as the large intestine and bronchi of patients using an endoscope. However, as an endoscopic image, an image in which the color and texture inside the tubular structure are clearly expressed by an image sensor such as a CCD (Charge Coupled Device) can be obtained, while a two-dimensional image of the inside of the tubular structure is obtained. It is represented by. For this reason, it is difficult to know which position in the tubular structure the endoscopic image represents. In particular, since the bronchial endoscope has a small diameter and a narrow field of view, it is difficult to reach the target position at the tip of the endoscope.

Therefore, using a three-dimensional image acquired by tomography using a modality such as a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device, a route to a target point in the tubular structure is obtained in advance. A method is proposed to generate a virtual endoscopic image similar to the image actually taken by the endoscope from the 3D image and to navigate the path of the endoscope to the target point using the virtual endoscopic image. Has been done. For example, in Patent Document 1, path information representing a path of a tubular structure is acquired from a three-dimensional image, a large number of virtual endoscopic images are generated along the path, and images are taken by the virtual endoscopic image and the endoscope. By matching with the actual endoscopic image, which is the actual endoscopic image obtained by performing, and specifying the virtual endoscopic image at the current position of the endoscope, the tip position of the endoscope can be determined. Specific methods have been proposed.

However, even if a navigation image is used, in the case of a structure having a multi-step branching path such as a bronchus, it requires a skillful technique to reach the target position of the tip of the endoscope in a short time.

Therefore, the tip position of the endoscope is specified by converting the image intensity of the virtual endoscope image to gray scale and matching the real endoscope image with the virtual endoscope image converted to gray scale. (See Patent Document 2).

Japanese Patent No. 5718537 Japanese Patent Application Laid-Open No. 2003-265408

On the other hand, since the endoscope is inserted into a tubular structure, body fluid may adhere to the lens at the tip of the endoscope or the lens may become cloudy due to the influence of mucus or the like in the body. In addition, an object that cannot be captured by tomography may be included in the actual endoscopic image. In this way, when the real endoscope image contains objects such as cloudiness of the lens and objects that cannot be captured by tomography as noise, even if the virtual endoscope image is converted to gray scale, the real endoscope image And the virtual endoscopic image cannot be matched accurately. As a result, the position of the tip of the endoscope cannot be accurately specified.

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to enable accurate matching between a real endoscopic image and a virtual endoscopic image.

The inspection support device according to the present disclosure includes an image acquisition unit that acquires a first medical image in a first expression format and a second medical image in a second expression format different from the first expression format.
By converting the first expression format of the first medical image and the second expression format of the second medical image into the third expression format, respectively, the converted first medical image and the converted second expression format are obtained. A conversion unit that acquires medical images, and
A similarity calculation unit for calculating the similarity between the converted first medical image and the converted second medical image is provided.

The "expression format" means, so to speak, the style of an image that affects the impression of the image on the viewer when the image is displayed. For example, if the medical image is acquired by actually taking a picture, the expression format of the medical image is a live-action style. Further, if the medical image is generated by CG (computer graphics), the expression format of the medical image is CG style. Further, if the medical image is generated by projecting a three-dimensional image using a projection method such as volume rendering, the expression format of the medical image is CG style and volume rendering image style. Further, if the medical image is a real endoscopic image, the medical image is a live-action style and is a real endoscopic image style. Even in the actual endoscopic image, the medical image taken by narrow-band light observation is a narrow-band light breeze, and the medical image taken by white light observation is a white light breeze. If the medical image is a virtual endoscopic image, the medical image is CG-like and virtual endoscopic image-like. Each expression format is determined by characteristics such as a range of parameters related to image gradation and color expression, texture expression (image roughness, fineness, noise amount and noise characteristics), surface reflectance, and transparency. However, it is not limited to these.

In the inspection support device according to the present disclosure, the first medical image is an actual image acquired by photographing the subject, and the second medical image captures the subject by a method different from the actual image. It may be an image generated from the acquired image.

Further, in the examination support device according to the present disclosure, the first medical image is a real endoscopic image showing the inner wall of the tubular structure generated by the endoscope inserted into the tubular structure in the subject. can be,
The second medical image is a virtual endoscopic image that simulates the inner wall of the tubular structure, which is generated from a three-dimensional image including the tubular structure of the subject.
The converted first medical image is a first depth image showing the distance from the viewpoint of the first medical image to the inner wall of the tubular structure.
The converted second medical image may be a second depth image representing the distance from the viewpoint of the second medical image to the inner wall of the tubular structure.

Further, in the inspection support device according to the present disclosure, a position for estimating the position of the endoscope in the tubular structure by using a virtual endoscope image in which the calculated similarity is equal to or higher than a predetermined threshold value. It may further include an estimation unit.

Further, in the inspection support device according to the present disclosure, the position estimation unit estimates the position of the endoscope in the tubular structure based on the difference in the pixel values between the first depth image and the second depth image. It may be something to do.

Further, the inspection support device according to the present disclosure further includes a display control unit that displays a second depth image and a first depth image corresponding to the estimated position of the endoscope on the display unit. May be good.

Further, the inspection support device according to the present disclosure may further include a virtual endoscope image generation unit that generates a virtual endoscope image from a three-dimensional image.

The examination support method according to the present disclosure acquires a first medical image of the first expression format and a second medical image of a second expression format different from the first expression format.
By converting the first expression format of the first medical image and the second expression format of the second medical image into the third expression format, respectively, the converted first medical image and the converted second expression format are obtained. Get a medical image,
The degree of similarity between the converted first medical image and the converted second medical image is calculated.

It should be noted that the inspection support method according to the present disclosure may be provided as a program for causing a computer to execute the inspection support method.

Other inspection support devices according to the present disclosure include a memory for storing an instruction to be executed by a computer and a memory.
The processor comprises a processor configured to execute a stored instruction.
The first medical image of the first expression format and the second medical image of the second expression format different from the first expression format are acquired.
By converting the first expression format of the first medical image and the second expression format of the second medical image into the third expression format, respectively, the converted first medical image and the converted second expression format are obtained. Get a medical image,
The degree of similarity between the converted first medical image and the converted second medical image is calculated.

According to the present disclosure, a first medical image having a first expression format and a second medical image having a second expression format different from the first expression format are acquired. Then, the expression formats of the first medical image and the second medical image are converted into the third expression format, and the converted first medical image and the converted second medical image are acquired. Further, the degree of similarity between the converted first medical image and the converted second medical image is calculated. Therefore, even if the first medical image and the second medical image contain noise, the noise can be canceled when the expression format is converted to the third expression format. Therefore, according to the present disclosure, the degree of similarity between the first medical image and the second medical image can be calculated accurately.

A hardware configuration diagram showing an outline of a diagnostic support system to which the inspection support device according to the embodiment of the present disclosure is applied. The figure which shows the schematic structure of the inspection support device realized by installing the inspection support program in a computer. Figure showing bronchial image A diagram showing a bronchial image with an endoscopic path set Diagram to illustrate the conversion of virtual endoscopic images Diagram for explaining the range of virtual endoscopic images for calculating similarity Diagram to explain the estimation of the position of the tip of the endoscope Figure to explain the correction of the position of the viewpoint Figure showing display image Flow chart showing processing performed in this embodiment A hardware configuration diagram showing an outline of a diagnostic support system to which an inspection support device according to another embodiment of the present disclosure is applied.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. FIG. 1 is a hardware configuration diagram showing an outline of a diagnosis support system to which the inspection support device according to the present embodiment is applied. As shown in FIG. 1, in the system of the present embodiment, the endoscope device 3, the three-dimensional image capturing device 4, the image storage server 5, and the inspection support device 6 are connected in a communicable state via the network 8. Has been done.

The endoscope device 3 includes an endoscope scope 31 that photographs the inside of a tubular structure of a subject, a processor device 32 that generates an image of the inside of the tubular structure based on a signal obtained by imaging, and the like. ..

The endoscope scope 31 has an insertion portion inserted into the tubular structure of the subject continuously attached to the operation portion 3A. The endoscope scope 31 is connected to the processor device 32 via a universal cord detachably connected to the processor device 32. The operation unit 3A commands an operation so that the tip 3B of the insertion portion bends in the vertical direction and the horizontal direction within a predetermined angle range, or operates the puncture needle attached to the tip of the endoscope scope 31. Includes various buttons for collecting tissue samples. In this embodiment, the endoscope scope 31 is a flexible scope for the bronchus and is inserted into the bronchus of the subject. Then, light guided by an optical fiber from a light source device (not shown) provided in the processor device 32 is irradiated from the tip 3B of the insertion portion of the endoscope scope 31, and the subject is irradiated with the image pickup optical system of the endoscope scope 31. An image of the bronchus is acquired. In addition, in order to facilitate the explanation, the tip 3B of the insertion portion of the endoscope scope 31 will be referred to as the endoscope tip 3B in the following description.

The processor device 32 converts the imaging signal captured by the endoscope scope 31 into a digital image signal, corrects the image quality by digital signal processing such as white balance adjustment and shading correction, and generates an endoscope image T0. .. The generated image is a color moving image represented by a predetermined sampling rate such as 30 fps, and one frame of the moving image is the endoscope image T0. The endoscopic image T0 is sequentially transmitted to the image storage server 5 or the inspection support device 6. Here, in the following description, the endoscope image T0 taken by the endoscope device 3 will be referred to as a real endoscope image T0 in order to distinguish it from a virtual endoscope image described later.

The three-dimensional imaging apparatus 4 is an apparatus that generates a three-dimensional image V0 representing the portion to be inspected of the subject by photographing the portion to be inspected. Specifically, the endoscopic scope 31 is formed into a tubular structure. CT devices, MRI devices, PET (Positron Emission Tomography), ultrasonic diagnostic devices, and the like, which are inserted and imaged by a method other than the method of photographing a tubular structure. The 3D image V0 generated by the 3D image capturing device 4 is transmitted to and stored in the image storage server 5. In the present embodiment, the three-dimensional imaging apparatus 4 generates a three-dimensional image V0 in which the chest including the bronchi is photographed. In the present embodiment, the three-dimensional image capturing device 4 is assumed to be a CT device, but the present invention is not limited to this.

The image storage server 5 is a computer that stores and manages various types of data, and includes a large-capacity external storage device and database management software. The image storage server 5 communicates with other devices via the network 8 to send and receive image data and the like. Specifically, image data such as the actual endoscope image T0 acquired by the endoscope device 3 and the 3D image V0 generated by the 3D image capturing device 4 are acquired via a network and are stored in a large capacity external storage device. It is saved and managed in a recording medium such as. The real endoscope image T0 is a moving image. Therefore, it is preferable that the actual endoscopic image T0 is transmitted to the inspection support device 6 without going through the image storage server 5. The storage format of the image data and the communication between the devices via the network 8 are based on a protocol such as DICOM (Digital Imaging and Communication in Medicine).

The inspection support device 6 has the inspection support program of the present disclosure installed on one computer. The computer may be a workstation or personal computer directly operated by the diagnosing doctor, or it may be a server computer connected to them via a network. The inspection support program is recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disk Read Only Memory), and is installed in a computer from the recording medium. Alternatively, it is stored in the storage device of the server computer connected to the network or in the network storage in a state accessible from the outside, and is downloaded to the computer used by the doctor who is the operator of the inspection support device 6 as requested. Will be installed.

FIG. 2 is a diagram showing a schematic configuration of an inspection support device realized by installing an inspection support program on a computer. As shown in FIG. 2, the inspection support device 6 includes a CPU (Central Processing Unit) 11, a memory 12, and a storage 13 as a standard workstation configuration. Further, the inspection support device 6 is connected to a display unit 14 and an input unit 15 such as a mouse. The display unit 14 is composed of a liquid crystal display or the like.

The storage 13 is composed of a hard disk drive or the like, and has a real endoscope image T0, a three-dimensional image V0, and an inspection acquired from the endoscope device 3, the three-dimensional image capturing device 4, the image storage server 5, and the like via the network 8. An image generated by the processing by the support device 6 (for example, a virtual endoscopic image described later), various information necessary for the processing, and the like are stored.

Further, the inspection support program is stored in the memory 12. As a process to be executed by the CPU 11, the inspection support program includes a real endoscope image T0 generated by the processor device 32, a three-dimensional image V0 generated by the three-dimensional image capturing device 4, and a virtual image generated as described later. Image acquisition process for acquiring image data such as an endoscope image, bronchial image generation process for generating a three-dimensional bronchial image B0 representing a bronchial graph structure from a three-dimensional image V0, and a virtual endoscopic image from a three-dimensional image V0. The first corresponding to the real endoscopic image by converting the generated virtual endoscopic image generation process, the expression format of the real endoscopic image, and the expression format of the virtual endoscopic image into the expression format of the depth image, respectively. Conversion process to acquire the second depth image corresponding to the depth image and the virtual endoscopic image, similarity calculation process to calculate the similarity between the first depth image and the second depth image, and the calculated similarity. Position estimation processing that estimates the position of the endoscope tip 3B in the bronchus using a virtual endoscopic image whose degree is equal to or higher than a predetermined threshold, and real endoscopic image T0 and virtual endoscope. A display control process for displaying an image, a bronchial image B0, a position of a specified endoscope tip 3B, and the like on a display unit 14 is defined.

Then, when the CPU 11 executes these processes according to the program, the computer, which is the inspection support device 6, has the image acquisition unit 21, the bronchial image generation unit 22, the virtual endoscope image generation unit 23, the conversion unit 24, and the similarity. It functions as a calculation unit 25, a position estimation unit 26, and a display control unit 27.

The image acquisition unit 21 acquires the actual endoscope image T0 obtained by capturing the inside of the bronchus at a predetermined viewpoint position by the three-dimensional image V0 and the endoscope device 3. When the three-dimensional image V0 and the real endoscope image T0 are already stored in the storage 13, the image acquisition unit 21 acquires the three-dimensional image V0 and the real endoscope image T0 from the storage 13. May be good. The real endoscopic image T0 is an image showing the inner surface of the bronchus, that is, the inner wall of the bronchus. The actual endoscope image T0 is output to the display control unit 27 and displayed on the display unit 14. Further, the image acquisition unit 21 acquires the virtual endoscope image K0 generated by the virtual endoscope image generation unit 23 and stored in the storage 13 from the storage 13, as will be described later.

The bronchial image generation unit 22 generates a three-dimensional bronchial image B0 by extracting the bronchial structure from the three-dimensional image V0. Specifically, the bronchial image generation unit 22 uses a method described in, for example, Japanese Patent Application Laid-Open No. 2010-220742 to display a three-dimensional graph structure of the bronchial region included in the input three-dimensional image V0. It is extracted as a bronchial image B0. Hereinafter, an example of the extraction method of this graph structure will be described.

In the three-dimensional image V0, the pixels inside the bronchus are represented as regions showing low pixel values because they correspond to the air region, but the bronchial walls are represented as cylinders or linear structures showing relatively high pixel values. Will be done. Therefore, the bronchi are extracted by performing structural analysis of the shape based on the distribution of pixel values for each pixel.

The bronchus branches in multiple stages, and the diameter of the bronchi becomes smaller as it approaches the end. The bronchial image generation unit 22 performs multiple-resolution conversion of the three-dimensional image V0 to generate a plurality of three-dimensional images having different resolutions so that bronchi of different sizes can be detected, and for each three-dimensional image of each resolution. Apply the detection algorithm.

First, at each resolution, the Hessian matrix of each pixel of the three-dimensional image is calculated, and it is determined whether or not the pixel is in the tubular structure from the magnitude relation of the eigenvalues of the Hessian matrix. The Hessian matrix is a matrix whose elements are the second-order partial differential coefficients of the density values in the directions of each axis (x-axis, y-axis, z-axis of the three-dimensional image), and is a 3 × 3 matrix as shown in the following equation. ..

When the eigenvalues of the Hessian matrix in an arbitrary pixel are λ1, λ2, and λ3, when two eigenvalues of the eigenvalues are large and one eigenvalue is close to 0, for example, when λ3, λ2 >> λ1, λ1≈0 is satisfied. , The pixel is known to be a tubular structure. Further, the eigenvector corresponding to the minimum eigenvalue (λ1≈0) of the Hessian matrix coincides with the main axis direction of the tubular structure.

Although the bronchi can be represented by a graph structure, the tubular structure extracted in this way is not always detected as one graph structure in which all the tubular structures are connected due to the influence of a tumor or the like. Therefore, after the detection of the tubular structure from the entire three-dimensional image V0 is completed, each of the extracted tubular structures is within a certain distance and connects arbitrary points on the two extracted tubular structures. By evaluating whether the angle formed by the direction of the basic line and the main axis direction of each tubular structure is within a certain angle, it is determined whether or not a plurality of tubular structures are connected and extracted. Reconstruct the connection relationship of the tubular structure. This reconstruction completes the extraction of the bronchial graph structure.

Then, the bronchial image generation unit 22 classifies the extracted graph structure into a start point, an end point, a branch point and an edge, and connects the start point, the end point and the branch point by the edge to represent a three-dimensional graph. The structure can be obtained as bronchial image B0. FIG. 3 is a diagram showing a bronchial image B0. The graph structure generation method is not limited to the above-mentioned method, and other methods may be adopted.

The virtual endoscope image generation unit 23 is a virtual endoscope depicting the inner wall of the bronchus as seen from the viewpoint in the three-dimensional image V0 corresponding to the viewpoint of the real endoscope image T0 in the target path in the bronchus. Generate image K0. Hereinafter, the generation of the virtual endoscope image K0 will be described.

First, the virtual endoscope image generation unit 23 acquires information on the path for inserting the endoscope tip 3B, which is preset in the bronchus. The route information is, for example, a bronchial image B0 displayed on the display unit 14, and the displayed bronchial image B0 used by the operator using the input unit 15, but is not limited thereto. .. FIG. 4 shows the path 40 of the endoscope set in the bronchial image B0.

The virtual endoscopic image generation unit 23 generates a virtual endoscopic image K0 representing the inner wall of the bronchus at a predetermined interval along the path 40 in the bronchus represented by the acquired path information. The predetermined interval may be, for example, a 10-pixel interval on the path 40, but is not limited to this.

The virtual endoscope image generation unit 23 sets each position on the path 40 as a viewpoint. Then, three-dimensional images V0 on a plurality of lines of sight radially extended in the insertion direction of the endoscope tip 3B (that is, the direction toward the end of the bronchus) from each set viewpoint are projected onto a predetermined projection surface. A projected image is generated by performing central projection. This projected image is a virtual endoscope image K0 that is virtually generated assuming that the image was taken at the tip position of the endoscope. As a specific method of central projection, for example, a known volume rendering method or the like can be used. Further, it is assumed that the angle of view (range of line of sight) and the center of the field of view (center of the projection direction) of the virtual endoscope image K0 are set in advance by input by the operator or the like. The generated plurality of virtual endoscope images K0 are stored in the storage 13. Since the three-dimensional image V0 is a CT image in the present embodiment, the virtual endoscopic image K0 is a monochrome image represented by the CT value.

The conversion unit 24 converts the representation format of the real endoscope image T0 into the representation format of the depth image. Further, the expression format of the virtual endoscope image K0 generated by the virtual endoscope image generation unit 23 is converted into the expression format of the depth image. To this end, the converter 24, for example, "Atapour-Abarghouei, A. and Breckon, T.P. (2018)'Real-time monocular depth estimation using synthetic data with domain adaptation.', 2018 IEEE Conference on Computer Vision and Pattern Recognition (2018) CVPR). Salt Lake City, Utah, USA, 18-22 June 2018. ”(hereinafter referred to as Non-Patent Document 1) is used. The method described in Non-Patent Document 1 is a method of converting an image into a depth image by using a model such as a neural network that outputs a depth image when the image is input. The model is generated by performing machine learning using various images and information representing the depth (that is, depth) of the images as teacher data. In the depth image, the pixel value represents the distance from the viewpoint position. Specifically, the pixel value of the depth image becomes lower as the distance from the viewpoint position increases, and as a result, the image becomes darker as the distance from the viewpoint position increases. Then, by setting the relationship between the pixel value of the depth image and the distance in advance, the distance from the viewpoint position can be calculated based on the pixel value of the depth image.

The conversion unit 24 in the present embodiment includes, as a converter, a model composed of a neural network or the like described in Non-Patent Document 1 in which machine learning is performed so as to convert an image expression format into a depth image expression format. As a machine learning method, a known method can be used. For example, a support vector machine, a deep neural network, a convolutional neural network, a recurrent neural network, and the like can be used. In the present embodiment, when the real endoscope image T0 contains cloudiness of the lens and an object or the like that cannot be captured by tomographic imaging as noise, the noise is removed and the real endoscope image T0 is obtained. Machine learning is done to convert to a depth image. Further, a converter that converts the representation format of the real endoscope image T0 and the virtual endoscope image K0 into the representation format of the depth image by logical operation may be used. With such a converter, as shown in FIG. 5, the representation formats of the real endoscope image T0 and the virtual endoscope image K0 are converted into the representation format of the depth image, and as a result, the representation format of the real endoscope image T0 is obtained. The corresponding first depth image D1 and the second depth image D2 corresponding to the virtual endoscopic image are acquired. In addition, noise contained in the actual endoscopic image T0, such as an object that cannot be captured by cloudiness of the lens and tomography, is canceled by the conversion. The first depth image D1 corresponds to the first converted medical image, and the second depth image D2 corresponds to the converted second medical image.

The actual endoscopic image T0 constitutes one frame of a moving image. Therefore, the conversion unit 24 converts the representation format of the real endoscope image T0 acquired sequentially into the representation format of the depth image. Further, the conversion unit 24 may convert all the virtual endoscope images K0 generated by the virtual endoscope image generation unit 23 into depth images in advance. Further, as will be described later, the virtual endoscope image K0 around the estimated position of the endoscope tip 3B may be converted into a depth image every time the position estimation process is performed. In the present embodiment, all virtual endoscope images K0 generated by the virtual endoscope image generation unit 23 are converted into depth images in advance. The second depth image D2 acquired by this is stored in the storage 13.

The similarity calculation unit 25 is similar to the first depth image D1 and the second depth image D2 for the real endoscope image T0 sequentially acquired in order to estimate the position of the endoscope tip 3B described later. Calculate the degree. As the degree of similarity, the reciprocal of the sum of the absolute values of the differences in pixel values between the corresponding pixels of the first depth image D1 and the second depth image D2, the reciprocal of the sum of squares of the differences, and the like are used. Can be done.

The position estimation unit 26 estimates the position of the endoscope tip 3B in the bronchus using a virtual endoscope image in which the similarity calculated by the similarity calculation unit 25 is equal to or higher than a predetermined threshold value Th1. ..

As described above, the real endoscopic image T0 constitutes one frame of the moving image. Therefore, the similarity calculation unit 25 includes a first depth image D1 and a second depth image D2 for the latest real endoscope image T0 among the real endoscope images T0 sequentially acquired in the moving image. Calculate the similarity of. Here, in order to estimate the position of the endoscope tip 3B, the similarity calculation unit 25 generates all the virtual endoscope images K0 stored in the first depth image D1 and the storage 13. The degree of similarity with the second depth image D2 may be calculated, but the amount of calculation is large. Therefore, in this embodiment, the degree of similarity is calculated as follows.

First, the similarity calculation unit 25 specifies a second depth image D2, which is a reference generated at a reference position in the bronchus, in the reference depth image DB. Then, the degree of similarity between the latest first depth image D1 of the sequentially acquired first depth images D1 and the reference depth image DB is calculated. As the reference depth image DB, a second depth image D2 generated from a virtual endoscopic image whose viewpoint position is the position closest to the entrance of the bronchus in the bronchial path 40 may be used. When the first depth image D1 having a similarity with the reference depth image DB of the threshold value Th2 or more is acquired, the position estimation unit 26 estimates that the position of the endoscope tip 3B is at the reference position. The threshold value Th2 may be the same as or different from the threshold value Th1.

In this way, when the position estimation unit 26 estimates that the position of the endoscope tip 3B is at the reference position, the similarity calculation unit 25 determines the predetermined range A0 from the reference position Pb as shown in FIG. The degree of similarity between the second depth image D2 of one or more in the inside and the latest first depth image D1 is calculated. When the endoscope tip 3B moves inside the bronchus and a certain real endoscope image T0 is acquired, it is estimated that the endoscope tip 3B is at the position P1 as shown in FIG. , The similarity calculation unit 25 relates to the first depth image D1 for the next acquired real endoscope image T0, and has one or more second depth images within a predetermined range A0 including the position P1. The degree of similarity between D2 and the actual endoscopic image T0 is calculated. That is, since the real endoscope image T0 constitutes one frame of the moving image, each time the real endoscope image T0 is acquired, the similarity calculation unit 25 performs the latest first depth image D1 and the second. The degree of similarity with the depth image D2 of is calculated.

The position estimation unit 26 has a second degree of similarity between the second depth image D2 and the latest first depth image D1 in the range A0, which has a threshold value Th1 or more and the highest degree of similarity. The position of the viewpoint that generated the virtual endoscope image K0 corresponding to the depth image D2 is estimated to be the position of the endoscope tip 3B.

Since the first depth image D1 and the second depth image D2 are used in the present embodiment, it can be estimated that the position of the endoscope tip 3B in the bronchus in the traveling direction is more accurate. can. FIG. 7 is a diagram for explaining the estimation of the position of the endoscope tip 3B. As described above, in the depth image, the pixel value represents the distance in the depth direction. Therefore, the position estimation unit 26 sets the corresponding regions R1 and R2 for the second depth image D2max having the largest similarity with the first depth image D1 and the first depth image D1, respectively. Then, a representative value (for example, an average value, an intermediate value, a maximum value, etc.) of the pixel values in the region R1 is calculated, and the representative value is converted into the distance L1 from the viewpoint position. Similarly, a representative value (for example, an average value, an intermediate value, a maximum value, etc.) of a pixel value in the region R2 is calculated, and the representative value is converted into a distance L2 from the viewpoint position.

The difference ΔL between the distance L1 and the distance L2 calculated in this way is the position of the viewpoint that generated the virtual endoscope image K0 corresponding to the second depth image D2max having the maximum similarity, and the tip of the endoscope. It represents the amount of misalignment with the position of 3B. Hereinafter, ΔL is used as a reference code for the amount of misalignment. For example, when the calculated distance L1 = 11 mm and the distance L2 = 20 mm, the misalignment amount ΔL is L2-L1 = 9 mm. Based on the misalignment amount ΔL calculated in this way, the position estimation unit 26 corrects and corrects the position of the viewpoint that generated the virtual endoscope image K0 corresponding to the second depth image D2 having the largest similarity. The position is estimated as the position of the endoscope tip 3B. For example, when the misalignment amount ΔL is 9 mm, the position of the viewpoint is corrected 9 mm forward (that is, the insertion direction of the endoscope tip 3B). When the amount of misalignment becomes a negative value, the position estimation unit 26 corrects the position of the viewpoint to the rear (that is, the direction in which the tip 3B of the endoscope returns).

FIG. 8 is a diagram for explaining the correction of the position of the viewpoint. As shown in FIG. 8, it is assumed that the position of the viewpoint that generated the virtual endoscopic image K0 corresponding to the second depth image D2 having the highest similarity is the position P10. Further, it is assumed that the misalignment amount ΔL is a positive value. The position estimation unit 26 shifts the position P10 in the traveling direction of the endoscope tip 3B by the amount of misalignment ΔL, and estimates the position of the endoscope tip 3B as the position P11.

The display control unit 27 displays the second depth image D2 corresponding to the virtual endoscope image K0 at the position of the first depth image D1 and the endoscope tip 3B on the display unit 14. The display control unit 27 may display the bronchial image B0 on the display unit 14 and display the displayed bronchial image B0 to specify the endoscope tip 3B. Further, the real endoscopic image T0 may be displayed in place of or in addition to the first depth image D1. Further, the virtual endoscope image K0 may be displayed in place of or in addition to the second depth image D2.

FIG. 9 is a diagram showing a display image of the display unit 14. As shown in FIG. 9, the display image 50 displays a first depth image D1, a second depth image D2 at the position of the endoscope tip 3B, and a bronchial image B0. In the bronchial image B0, the movement locus 51 of the endoscope and the position Pt of the endoscope tip 3B are displayed.

Next, the processing performed in this embodiment will be described. FIG. 10 is a flowchart showing the processing performed in the present embodiment. The bronchial image B0 is generated by the bronchial image generation unit 22, the virtual endoscopic image K0 is generated by the virtual endoscopic image generation unit 23, and the second depth image D2 for the virtual endoscopic image K0. Is generated by the conversion unit 24 and is stored in the storage 13 respectively. When the endoscope tip 3B is inserted into the subject and an operation start instruction is input from the input unit 15, the image acquisition unit 21 acquires the actual endoscope image T0 (step ST1), and the conversion unit 24 determines. The representation format of the real endoscope image T0 is converted into the representation format of the depth image, and the first depth image D1 is acquired (step ST2). Then, the similarity calculation unit 25 calculates the similarity between the first depth image D1 and the reference depth image DB (step ST3). Further, the position estimation unit 26 determines whether or not the similarity calculated by the similarity calculation unit 25 is equal to or higher than the threshold value Th2 (step ST4). If step ST4 is denied, the process returns to step ST1 and the processes of steps ST1 to ST4 are repeated.

When step ST4 is affirmed, the position estimation unit 26 estimates that the endoscope tip 3B is at the reference position Pb (step ST5). Subsequently, the image acquisition unit 21 acquires the real endoscope image T0 (step ST6), and the conversion unit 24 converts the expression format of the real endoscope image T0 into the expression format of the depth image, and the first depth image. Acquire D1 (step ST7). Subsequently, the similarity calculation unit 25 corresponds to the first depth image D1 acquired in step ST7 and the virtual endoscope image K0 within a predetermined range from the current position of the endoscope tip 3B. The degree of similarity with the depth image D2 is calculated (step ST8). Then, the position where the position estimation unit 26 acquires the virtual endoscope image K0 corresponding to the second depth image D2 having the similarity of the threshold value Th1 or more and the largest similarity is set to the endoscope tip 3B. It is estimated to be the position of (step ST9). Subsequently, the display control unit 27 displays the first depth image D1 and the second depth image D2 (step ST10). Then, the processes of steps ST6 to ST10 are repeated until the end instruction is given (step ST11: affirmative).

In the processing of the above embodiment, there may be a case where the second depth image D2 whose similarity is equal to or higher than the threshold value Th1 does not exist. In such a case, the position of the endoscope tip 3B may be estimated by expanding the range in which the second depth image D2 for calculating the similarity is acquired.

As described above, in the present embodiment, the representation formats of the real endoscope image T0 and the virtual endoscope image K0 are converted into the representation formats of the depth image, and the first depth image D1 and the second depth image D2 are obtained. It was acquired and the degree of similarity between the first depth image D1 and the second depth image D2 was calculated. Therefore, even if the real endoscope image T0 contains an object or the like that cannot be captured by cloudiness of the lens or tomography as noise, when the real endoscope image T0 is converted into the expression format of the depth image, , Noise can be canceled. Therefore, according to the present embodiment, the degree of similarity between the real endoscope image T0 and the virtual endoscope image K0 can be calculated accurately. Further, by using the similarity calculated in this way, the position of the endoscope tip 3B can be estimated accurately.

In the above embodiment, the position estimation unit 26 estimates the reference position of the endoscope tip 3B, but the present invention is not limited to this. For example, as shown in FIG. 11, it is assumed that the endoscope device 3 has a position detection device 34, and the position detection device 34 detects the position of the endoscope tip 3B while detecting the first depth images D1 and the second. The degree of similarity with the depth image D2 of the endoscope may be calculated to estimate the position of the endoscope tip 3B.

In this case, the position detecting device 34 detects the position and orientation of the endoscope tip 3B in the body of the subject. Specifically, the position detection device 34 has an echo device having a detection area of a three-dimensional coordinate system based on the position of a specific part of the subject, and the echo device is characteristic of the endoscope tip 3B. By detecting the shape, the relative position and orientation of the endoscope tip 3B in the body of the subject is detected. Then, the detected position and orientation information of the endoscope tip 3B is output to the inspection support device 6 as position information Q0 (see, for example, Japanese Patent Application Laid-Open No. 2006-61274). The detected positions and orientations of the endoscope tip 3B correspond to the viewpoint and the line-of-sight direction of the actual endoscope image T0 obtained by photographing. Here, the position of the endoscope tip 3B is represented by the three-dimensional coordinates with respect to the position of the specific portion of the subject described above. Further, the position information Q0 is output to the inspection support device 6 at the same sampling rate as the actual endoscopic image T0.

As described above, when the position detection device 34 is used, the similarity calculation unit 25 has one or more second depth images D2 within a predetermined range A0 from the position in the bronchi represented by the position information Q0. And the first depth image D1 are calculated. Similar to the above embodiment, the position estimation unit 26 has a threshold value Th1 or more and the highest degree of similarity among all the similarities between the second depth image D2 and the first depth image D1 in the range A0. It is estimated that the position where the virtual endoscope image K0 corresponding to the second depth image D2 having a large value is generated is the position of the endoscope tip 3B. Also in this case, as in the above embodiment, the position shift amount may be obtained based on the pixel values of the first depth image D1 and the second depth image D2, and the estimated position may be corrected.

Further, in the above embodiment, the similarity calculation unit 25 calculates the similarity between the first depth image D1 and the second depth image D2 each time the actual endoscopic image T0 is acquired. However, it is not limited to this. For example, using the method described in Japanese Patent Application Laid-Open No. 2016-163609, it is determined whether or not the bronchial bifurcation structure is included in the real endoscopic image T0, and the bronchial bifurcation structure is included in the real endoscopic image T0. The first depth image D1 may be acquired and the degree of similarity between the first depth image D1 and the second depth image D2 may be calculated only when is included.

Further, in the above embodiment, the case where the inspection support device of the present disclosure is applied to the observation of the bronchus has been described, but the present invention is not limited to this, and tubular structures such as stomach, large intestine and blood vessels are endoscopeed. The present disclosure can also be applied when observing by.

Further, in the above embodiment, the position shift amount is calculated from the pixel values of the first depth image D1 and the second depth image D2 to correct the position of the endoscope tip 3B, but the present invention is limited to this. It is not something that will be done. The position estimation unit 26 may estimate the viewpoint position of the virtual endoscope image K0 estimated based on the similarity as the position of the endoscope tip 3B as it is.

Further, in the above embodiment, the inspection support device 6 includes a bronchial image generation unit 22 and a virtual endoscopic image generation unit 23, but the present invention is not limited thereto. The bronchial image generation unit 22 and the virtual endoscopic image generation unit 23 may be provided as devices separate from the inspection support device 6. In this case, the bronchial image B0 and the virtual endoscopic image K0 are input to the inspection support device 6 from separate devices and acquired by the image acquisition unit 21.

Further, in the above embodiment, the representation formats of the real endoscope image T0 and the virtual endoscope image K0 are converted into the representation format of the depth image, but the present invention is not limited to this. For example, the real endoscope image T0 and the virtual endoscope image K0 may be converted into an expression format other than the expression format of the depth image such as a CG-like expression format.

Further, in the above embodiment, the real endoscopic image is used as the first medical image and the virtual endoscopic image is used as the second medical image, but the present invention is not limited thereto. Any medical image can be used as long as the expression formats of the first medical image and the second medical image are different, such as the first medical image being a CT image and the second medical image being an MRI image. ..

Further, in the above embodiment, the inspection support device according to the present disclosure is applied to a device that supports endoscopy, but is not limited thereto. The present disclosure may be applied to any device that assists in an examination to obtain a first medical image in a first representation and a second medical image in a second representation that is different from the first representation. can.

Further, in the above embodiment, for example, an image acquisition unit 21, a bronchial image generation unit 22, a virtual endoscope image generation unit 23, a conversion unit 24, a similarity calculation unit 25, a position estimation unit 26, and a display control unit 27 are used. As the hardware structure of the processing unit that executes various processes, the following various processors can be used. As described above, the various processors include CPUs, which are general-purpose processors that execute software (programs) and function as various processing units, as well as circuits after manufacturing FPGAs (Field Programmable Gate Arrays) and the like. Dedicated electricity, which is a processor with a circuit configuration specially designed to execute specific processing such as Programmable Logic Device (PLD), which is a processor whose configuration can be changed, and ASIC (Application Specific Integrated Circuit). Circuits etc. are included.

One processing unit may be composed of one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). ) May be configured. Further, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client and a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units. Second, as typified by System On Chip (SoC), there is a form that uses a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined can be used.

3 Endoscope device 3A Operation unit 3B Endoscope tip 4 3D imaging device 5 Image storage server 6 Inspection support device 8 Network 11 CPU
12 Memory 13 Storage 14 Display 15 Input 21 Image acquisition 22 Bronchial image generator 23 Virtual endoscope image generator 24 Converter 25 Similarity calculation unit 26 Position estimation unit 27 Display control unit 31 Endoscope scope 32 Processor Device 40 Path 50 Display image 51 Movement locus A0 Range B0 Bronchial image D1 First depth image D2 Second depth image K0 Virtual endoscope image ΔL Positional deviation P1 Position Pb Reference position Pt Position R1, R2 Area T0 Real Mirror image

Claims (7)

A real endoscopic image having a live-like representation of the inner wall of the tubular structure, generated by an endoscope inserted into the tubular structure in the subject, and the tubular structure of the subject. An image acquisition unit that acquires a virtual endoscopic image having a CG-like expression format that pseudo-represents the inner wall of the tubular structure, which is generated from a three-dimensional image including the image.
The inner wall of the tubular structure from the viewpoint of the real endoscope image by converting the expression form of the live-viewing style of the real endoscopic image into the expression format of the depth image-like using the trained neural network. By acquiring a first depth image representing the distance to and converting the CG-like expression format of the virtual endoscope image into the depth image-like expression format, the virtual endoscope image is viewed from the viewpoint. A conversion unit that acquires a second depth image showing the distance to the inner wall of the tubular structure, and
An inspection support device including a similarity calculation unit for calculating the similarity between the first depth image and the second depth image. A claim further comprising a position estimation unit for estimating the position of the endoscope in the tubular structure using a virtual endoscope image in which the calculated similarity is equal to or higher than a predetermined threshold value. The inspection support device according to 1. The second aspect of the present invention, wherein the position estimation unit estimates the position of the endoscope in the tubular structure based on the difference in pixel values between the first depth image and the second depth image. Inspection support device. The inspection support device according to claim 2 or 3, further comprising a display control unit for displaying the second depth image and the first depth image corresponding to the estimated position of the endoscope on the display unit. .. The inspection support device according to any one of claims 1 to 4, further comprising a virtual endoscope image generation unit for generating the virtual endoscope image from the three-dimensional image. A real endoscopic image having a live-like representation of the inner wall of the tubular structure generated by a computer inserted into the tubular structure in the subject, and the tubular of the subject. A virtual endoscopic image having a CG-like expression format that pseudo-represents the inner wall of the tubular structure, which is generated from a three-dimensional image including the structure, is acquired.
The inner wall of the tubular structure from the viewpoint of the real endoscopic image by converting the real-life representation of the real endoscopic image into a depth image-like representation using the trained neural network. By acquiring a first depth image representing the distance to and converting the CG-like expression format of the virtual endoscope image into the depth image-like expression format, the virtual endoscope image is viewed from the viewpoint. A second depth image showing the distance to the inner wall of the tubular structure was obtained.
An inspection support method for calculating the degree of similarity between the first depth image and the second depth image. A real endoscopic image having a live-like representation of the inner wall of the tubular structure, generated by an endoscope inserted into the tubular structure in the subject, and the tubular structure of the subject. A procedure for acquiring a virtual endoscopic image having a CG-like representation format that pseudo-represents the inner wall of the tubular structure, which is generated from the including three-dimensional image.
The inner wall of the tubular structure from the viewpoint of the real endoscopic image by converting the real-life representation of the real endoscopic image into a depth image-like representation using the trained neural network. By acquiring a first depth image representing the distance to and converting the CG-like expression format of the virtual endoscope image into the depth image-like expression format, the virtual endoscope image is viewed from the viewpoint. A procedure for acquiring a second depth image showing the distance to the inner wall of the tubular structure, and
An inspection support program that causes a computer to execute a procedure for calculating the degree of similarity between the first depth image and the second depth image.

Images