JPWO2019088008A1 - Image processing equipment, image processing methods, programs, and endoscopic systems - Google Patents

Abstract

Provided are an image processing device, an image processing method, a program, and an endoscopy system that enable efficient information collection of lesions and the like that virtual endoscopy is not good at. A first image input unit for inputting a virtual endoscopic image, a second image input unit for inputting a real endoscopic image, and an associating unit (57) for associating a virtual endoscopic image with a real endoscopic image. , The first feature region extraction unit (51) that extracts the first feature region from the virtual endoscopic image, the second feature region extraction unit (54) that extracts the second feature region from the real endoscopic image, the second A storage unit (61) that stores at least one of the information of the non-extracted area associated with the feature area and not extracted as the first feature area and the information of the second feature area associated with the non-extracted area. ) Is provided.

Description

The present invention relates to an image processing apparatus, an image processing method, a program, and an endoscopic system, and particularly to an analysis of a virtual endoscopic image.

In recent years, attention has been paid to a technique for observing or treating a tubular structure such as a patient's large intestine using an endoscope. The endoscopic image is an image taken by using an image sensor such as a CCD (Charge Coupled Device). The endoscopic image is an image in which the color and texture inside the tubular structure are clearly expressed. On the other hand, the endoscopic image is a two-dimensional image showing the inside of a tubular structure. For this reason, it is difficult to grasp which position in the tubular structure the endoscopic image represents.

Therefore, a virtual endoscopic image similar to the image actually taken by the endoscope is generated by using the three-dimensional inspection image obtained by tomography using a modality such as a CT device or an MRI device. The method to do is proposed.

The virtual endoscopic image may be used as a navigation image to guide the endoscope to a target position in a tubular structure. CT is an abbreviation for Computed Tomography. MRI is an abbreviation for Magnetic Resonance Imaging.

Therefore, an image of the tubular structure is extracted from the three-dimensional inspection image, and the image of the tubular structure is photographed using an endoscope to correspond to the actual endoscopic image obtained. A method has been proposed in which a virtual endoscopic image at the current position of the endoscope is generated from a three-dimensional inspection image of a tubular structure and displayed.

Patent Document 1 describes a medical support system that forms a three-dimensional organ model in a virtual space by inputting an endoscopic image obtained from an endoscopic device and provides a more realistic image.

Patent Document 2 generates a virtual endoscopic image so that the composition representing the relative positional relationship between the color endoscopic real image and the virtual endoscopic image matches, and the affected portion can be easily seen from the endoscopic image. An endoscopic system to be displayed in an aspect is described.

Patent Document 3 describes a medical image observation device that stores coordinate data of a specific portion of medical image data in association with the medical image data. The medical image observing apparatus described in Patent Document 3 maintains the image quality of the findings portion for which the image quality is not desired to be deteriorated, and reduces the amount of image data used for image display to improve the efficiency of image display.

Patent Document 4 describes an image processing method for detecting an object from a three-dimensional medical image. Learning is applied to the image processing method described in Patent Document 4 when detecting an object from a three-dimensional medical image.

Japanese Unexamined Patent Publication No. 11-104072 Japanese Unexamined Patent Publication No. 2006-61274 JP-A-2009-254690 Japanese Unexamined Patent Publication No. 2011-177517

Virtual colonoscopy based on virtual colonoscopy is good at detecting protrusions such as convex polyps, but is not good at detecting flat lesions. In order to improve the performance of virtual colonoscopy, it is desired to collect information on lesions that virtual colonoscopy is not good at. The above-mentioned problems in the virtual colonoscopy are the same for the virtual colonoscopy applied to the observed site other than the large intestine such as the trachea.

None of the inventions described in Patent Documents 1 to 4 focus on the technical problem of collecting information on lesions, which is difficult for virtual endoscopy, and do not solve the above-mentioned technical problem.

The present invention has been made in view of such circumstances, and is an image processing device, an image processing method, a program, and an endoscope that enable efficient information collection of lesions, etc., which virtual endoscopy is not good at. The purpose is to provide a system.

In order to achieve the above object, the following aspects of the invention are provided.

The image processing apparatus according to the first aspect uses a first image input unit for inputting a virtual endoscopic image generated from a three-dimensional inspection image of the subject and an endoscope to image an observation target of the subject. A second image input unit for inputting the obtained real endoscopic image, an associating unit for associating the virtual endoscopic image with the real endoscopic image, and the virtual endoscopic image as the first condition. A first feature region extraction unit that extracts a matching first feature region, and a second feature region extraction unit that extracts a second feature region that matches the second condition corresponding to the first condition from a real endoscopic image. , Information on the non-extracted region that is associated with the second feature region of the real endoscopic image and is not extracted as the first feature region from the virtual endoscopic image, and the second feature associated with the non-extracted region. It is an image processing apparatus including a storage unit for storing at least one of the information of the area.

According to the first aspect, the information of the non-extracted region of the virtual endoscopic image associated with the second feature region of the real endoscopic image and the information of the second feature region associated with the non-extracted region At least one is preserved. This makes it possible to efficiently collect information on feature regions that are extracted from the real endoscopic image but not extracted from the virtual endoscopic image.

An example of a three-dimensional inspection image is a three-dimensional inspection image obtained by tomography of a subject using a CT device. An example of a virtual colonoscope is a virtual colonoscope whose subject is the large intestine.

It is preferable to include a first condition setting unit for setting the first condition applied to the extraction of the first feature region. Further, it is preferable to include a second condition setting unit for setting a second condition applied to the extraction of the second feature region.

In the second image input unit, moving images may be sequentially input as real endoscopic images, or moving images may be collectively input as real endoscopic images in a file format.

The second aspect is the configuration in which the image processing apparatus of the first aspect includes an extraction result giving unit that gives an extraction result of extracting the first feature region from the virtual endoscopic image to the real endoscopic image. May be good.

According to the second aspect, whether or not the second feature region of the virtual endoscopic image is associated with the non-extracted region by using the extraction result of the virtual endoscopic image given to the real endoscopic image. It is possible to specify whether or not the image is a real endoscopic image including a second feature region associated with a non-target region of the real endoscopic image.

As an example of the extraction result of extracting the first feature region from the virtual endoscopic image, there is information for identifying a non-extracted region that has not been extracted as the first feature region. Examples of the information for identifying the non-extracted region include information on the position of the non-extracted region and an image of the non-extracted region.

The third aspect is the extraction result of extracting the first feature region from the virtual endoscopic image and the extraction result of extracting the second feature region from the real endoscopic image in the image processing apparatus of the first aspect or the second aspect. It may be configured to include a comparison unit for comparing with.

According to the third aspect, it is possible to efficiently associate the first feature region of the virtual endoscopic image with the second feature region of the real endoscopic image based on the comparison using the comparison unit.

The fourth aspect may be the image processing apparatus of the third aspect, in which the comparison unit may be configured to compare the corresponding positions of the virtual endoscopic image and the real endoscopic image.

According to the fourth aspect, it is possible to compare the virtual endoscopic image and the real endoscopic image at the corresponding positions.

In the fifth aspect, in the image processing apparatus of any one of the first to fourth aspects, the second characteristic region of the real endoscopic image is associated with the non-extracted region of the virtual endoscopic image. The configuration may include a determination result input unit for inputting a determination result for determining whether or not.

According to the fifth aspect, it is possible to acquire the determination result of the manual determination of the user.

The sixth aspect is a configuration in which the image processing apparatus according to any one of the first to fifth aspects includes a display unit for displaying the extraction result of the virtual endoscopic image and the extraction result of the real endoscopic image. May be.

According to the sixth aspect, the user can confirm the extraction result of the virtual endoscopic image and the extraction result of the real endoscopic image.

A seventh aspect is a configuration in which the image processing apparatus according to any one of the first to sixth aspects includes an extraction result input unit for inputting an extraction result obtained by extracting a second feature region from an actual endoscopic image. May be good.

According to the seventh aspect, it is possible to acquire the extraction result of the manual extraction by the user.

The eighth aspect is the image processing apparatus according to any one of the first to sixth aspects, wherein the second feature region extraction unit automatically extracts the second feature region from the actual endoscopic image. Good.

According to the eighth aspect, the second feature region can be automatically extracted from the actual endoscopic image.

A ninth aspect is the image processing apparatus according to any one of the first to eighth aspects, wherein the second feature region extraction unit may be configured to extract a lesion as a second feature region from a real endoscopic image. ..

According to the ninth aspect, it is possible to identify the non-extracted region of the virtual endoscopic image corresponding to the lesion.

A tenth aspect is the image processing apparatus according to any one of the first to ninth aspects, wherein the storage unit corresponds to a three-dimensional coordinate value in a virtual endoscopic image of a non-extracted area and a non-extracted area. The configuration may be such that at least one of the actual endoscopic images including the two feature regions is stored.

According to the tenth aspect, the storage unit can store the coordinate values in the virtual endoscopic image of the non-extracted region and the real endoscopic image including the second feature region corresponding to the non-extracted region.

The storage unit may store the coordinate value of the representative position of the non-extracted area as the coordinate value for specifying the non-extracted area. An example of the representative position of the non-extracted region is the position of the center of gravity of the non-extracted region.

The storage unit may store a plurality of coordinate values included in the non-extracted area as coordinate values for specifying the non-extracted area. An example of a plurality of coordinate values included in the non-extracted region is a plurality of coordinate values that specify the edge of the non-extracted region.

The storage unit may store the coordinate values in the virtual endoscopic image of the non-extracted region in association with the real endoscopic image including the second feature region corresponding to the non-extracted region.

The eleventh aspect may be the image processing apparatus of any one of the first to tenth aspects, wherein the first feature region extraction unit may be configured to extract a lesion as a first feature region from a virtual endoscopic image. ..

According to the eleventh aspect, it is possible to efficiently collect information on lesions extracted from a real endoscopic image without being extracted from a virtual endoscopic image.

Examples of lesions include lesions having two-dimensional features that can be visually recognized using color, texture, and the like.

A twelfth aspect is the image processing apparatus according to any one of the first to eleventh aspects, wherein the first feature area extraction unit includes information on the second feature area stored in the storage unit and a non-extraction area. The extraction rule for extracting the first feature region from the virtual endoscopic image may be updated by using the correspondence.

According to the twelfth aspect, it is possible to update the extraction rule for extracting the first feature region by using the information of the feature region not extracted from the virtual endoscopic image. This can improve the performance of virtual endoscopy.

A thirteenth aspect is an image processing apparatus according to any one of the first to twelfth aspects, wherein the first feature region extraction unit applies an extraction rule generated by using machine learning to a virtual endoscope. The first feature region may be extracted from the image.

According to the thirteenth aspect, it is possible to improve the performance of efficient virtual endoscopy using machine learning.

The image processing method according to the fourteenth aspect includes a first image input step of inputting a virtual endoscope image generated from a three-dimensional inspection image of a subject, and an image of an observation target of the subject using an endoscope. A second image input step for inputting the obtained real endoscope image, a matching step for associating the virtual endoscope image generated from the virtual endoscope image with the real endoscope image, and a virtual The first feature region extraction step of extracting the first feature region that matches the first condition from the endoscopic image, and the second feature region that matches the second condition corresponding to the first condition from the actual endoscopic image. Information on the second feature region to be extracted and the non-extracted region that is associated with the second feature region of the real endoscope image and is not extracted as the first feature region from the virtual endoscope image, and non-extraction This is an image processing method including a storage step of storing at least one of the information of the second feature area associated with the area.

According to the fourteenth aspect, the same effect as that of the first aspect can be obtained.

In the 14th aspect, the same items as those specified in the 2nd to 13th aspects can be appropriately combined. In that case, the component responsible for the processing or function specified in the image processing apparatus can be grasped as the component of the image processing method responsible for the corresponding processing or function.

The program according to the fifteenth aspect is a first image input function for inputting a virtual endoscope image generated from a three-dimensional examination image of a subject into a computer, and an image of an observation target of the subject is imaged using an endoscope. Second image input function for inputting the obtained real endoscope image, matching function for associating the virtual endoscope image generated from the virtual endoscope image with the real endoscope image, virtual endoscopy The first feature area extraction function that extracts the first feature region that matches the first condition from the mirror image, and the second feature region that matches the second condition corresponding to the first condition are extracted from the actual endoscopic image. 2 Feature area extraction function, information on non-extracted area that is associated with the second feature area of the real endoscope image and is not extracted as the first feature area from the virtual endoscope image, and corresponds to the non-extracted area There is a program that realizes a storage function that stores at least one of the information in the attached second feature area.

According to the fifteenth aspect, the same effect as that of the first aspect can be obtained.

In the fifteenth aspect, the same items as those specified in the second to thirteenth aspects can be appropriately combined. In that case, the component responsible for the processing or function specified in the image processing apparatus can be grasped as the component of the program responsible for the corresponding processing or function.

A fifteenth aspect is a system having at least one or more processors and at least one or more memories, and a first image input for inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject. Function, second image input function to input the real endoscope image obtained by imaging the observation target of the subject using the endoscope, the virtual endoscope image generated from the virtual endoscope image and the actual A matching function that associates with an endoscopic image, a first feature area extraction function that extracts a first feature area that matches the first condition from a virtual endoscopic image, and a first condition from a real endoscopic image. The second feature area extraction function that extracts the second feature area that matches the corresponding second condition, and the second feature area of the real endoscope image are associated with the virtual endoscopy image as the first feature area. It can be configured as a system that realizes a storage function for storing at least one of the information of the non-extracted region and the information of the second feature region associated with the non-extracted region.

The endoscope system according to the 16th aspect uses an endoscope, a first image input unit for inputting a virtual endoscope image generated from a three-dimensional inspection image of the subject, and an endoscope. The second image input unit for inputting the real endoscopic image obtained by imaging the observation target of the above is associated with the virtual endoscopic image generated from the virtual endoscopic image and the real endoscopic image. The association unit, the first feature area extraction unit that extracts the first feature area that matches the first condition from the virtual endoscope image, and the second condition that matches the first condition from the real endoscope image. A non-extracted region that is associated with the second feature region of the real endoscope image and is not extracted as the first feature region from the virtual endoscope image and the second feature region extraction unit that extracts the second feature region. This is an endoscope system including a storage unit for storing at least one of the information of the second feature region and the information of the second feature region associated with the non-extracted region.

According to the 16th aspect, the same effect as that of the 1st aspect can be obtained.

In the 16th aspect, the same items as those specified in the 2nd to 13th aspects can be appropriately combined. In that case, the component responsible for the process or function specified in the image processing apparatus can be grasped as the component of the endoscope system responsible for the corresponding process or function.

The endoscope system according to the seventeenth aspect is an endoscope system including an endoscope, an image processing device, and a storage device, and the image processing device is a virtual one generated from a three-dimensional inspection image of a subject. A first image input unit for inputting an endoscopic image, a second image input unit for inputting a real endoscopic image obtained by imaging an observation target of a subject using an endoscope, and virtual endoscopy. A matching unit that associates a virtual endoscope image generated from a mirror image with a real endoscope image, and a first feature region that extracts a first feature region that matches the first condition from the virtual endoscope image. The storage device includes an extraction unit and a second feature region extraction unit that extracts a second feature region that matches the second condition corresponding to the first condition from the real endoscope image, and the storage device is a real endoscope image. At least one of the information of the non-extracted area associated with the second feature area of the above and not extracted as the first feature area from the virtual endoscopic image and the information of the second feature area associated with the non-extracted area. It is an endoscope system equipped with a storage unit that stores one of them.

According to the 17th aspect, the same effect as that of the 1st aspect can be obtained.

In the 17th aspect, the same items as those specified in the 2nd to 13th aspects can be appropriately combined. In that case, the component responsible for the processing or function specified in the image processing apparatus can be grasped as the component of the endoscope system responsible for the corresponding processing or function.

The eighteenth aspect may be a configuration in which the storage device is communicably connected to the image processing device via a network in the endoscope system of the seventeenth aspect.

According to the eighteenth aspect, it is possible to efficiently collect information on a feature region that is extracted from a real endoscopic image but not extracted from a virtual endoscopic image by using a storage device connected to a network.

According to the present invention, at least the information of the non-extracted region of the virtual endoscopic image associated with the second feature region of the real endoscopic image and the information of the second feature region associated with the non-extracted region Either one is saved. This makes it possible to efficiently collect information on feature regions that are extracted from the real endoscopic image but not extracted from the virtual endoscopic image.

FIG. 1 is a schematic view showing the overall configuration of an endoscopic system. FIG. 2 is a functional block diagram showing the functions of the medical image processing apparatus. FIG. 3 is a functional block diagram showing the functions of the medical image analysis processing unit. FIG. 4 is a schematic diagram of a CTC image. FIG. 5 is a schematic view of an endoscopic image. FIG. 6 is an explanatory diagram of the first feature region extraction. FIG. 7 is an explanatory diagram of the second feature region extraction. FIG. 8 is an explanatory diagram of another example of the second feature region extraction. FIG. 9 is a schematic diagram showing an example of associating lesions. FIG. 10 is a schematic diagram showing an example of associating folds. FIG. 11 is a schematic diagram showing an example of associating folds using fold numbers. FIG. 12 is an explanatory diagram of an example of a comparison process between a CTC image using a lesion and an endoscopic image. FIG. 13 is a flowchart showing the procedure of the image processing method.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification, the same components are designated by the same reference numerals, and duplicate description is omitted.

[Overall configuration of the endoscope system]
FIG. 1 is a schematic view showing the overall configuration of an endoscopic system. The endoscope system 9 shown in FIG. 1 includes an endoscope 10, a light source device 11, a processor 12, a display device 13, a medical image processing device 14, an operating device 15, and a monitoring device 16. Be prepared. The endoscope system 9 is communicably connected to the image storage device 18 via the network 17.

The endoscope 10 is an electronic endoscope. Further, the endoscope 10 is a flexible endoscope. The endoscope 10 includes an insertion unit 20, an operation unit 21, and a universal cord 22. The insertion portion 20 includes a tip end and a base end. The insertion portion 20 is inserted into the subject. The operating unit 21 is gripped by the operator to perform various operations. The operation unit 21 is continuously provided on the base end side of the insertion unit 20. The entire insertion portion 20 has a small diameter and is formed in a long shape.

The insertion portion 20 includes a soft portion 25, a curved portion 26, and a tip portion 27. The insertion portion 20 is configured by connecting the soft portion 25, the curved portion 26, and the tip portion 27 in series. The soft portion 25 has flexibility in order from the proximal end side to the distal end side of the insertion portion 20. The curved portion 26 has a structure that can be curved when the operating portion 21 is operated. The tip 27 includes a photographing optical system (not shown), an image sensor 28, and the like.

The image sensor 28 is a CMOS image sensor or a CCD image sensor. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor, which is an English notation for complementary metal oxide semiconductors. CCD is an abbreviation for Charge Coupled Device, which is an English notation for a charge-coupled device.

An observation window (not shown) is arranged on the tip surface 27a of the tip portion 27. The observation window is an opening formed on the tip surface 27a of the tip portion 27. A photographing optical system (not shown) is arranged behind the observation window. The image light of the observed portion is incident on the image pickup surface of the image pickup element 28 through the observation window, the photographing optical system, and the like. The image sensor 28 captures the image light of the observed portion incident on the image pickup surface of the image sensor 28 and outputs an image pickup signal. Imaging here includes the meaning of converting image light into an electrical signal.

The operation unit 21 includes various operation members. Various operating members are operated by the operator. Specifically, the operation unit 21 includes two types of bending operation knobs 29. The bending operation knob 29 is used when the bending operation of the bending portion 26 is performed.

The operation unit 21 includes an air supply / water supply button 30 and a suction button 31. The air supply / water supply button 30 is used in the air supply / water supply operation. The suction button 31 is used during a suction operation.

The operation unit 21 includes a still image capturing instruction unit 32 and a treatment tool introduction port 33. The still image shooting instruction unit 32 is used when giving a shooting instruction for the still image 39 of the observed portion. The treatment tool introduction port 33 is an opening for inserting the treatment tool into the inside of the treatment tool insertion passage through which the inside of the insertion portion 20 is inserted. The treatment tool insertion passage and the treatment tool are not shown.

The universal cord 22 is a connection cord for connecting the endoscope 10 to the light source device 11. The universal cord 22 includes a light guide 35, a signal cable 36, and a fluid tube (not shown) that are inserted through the inside of the insertion portion 20.

Further, the end of the universal cord 22 includes a connector 37a connected to the light source device 11 and a connector 37b branched from the connector 37a and connected to the processor 12.

When the connector 37a is connected to the light source device 11, a light guide 35 and a fluid tube (not shown) are inserted into the light source device 11. As a result, necessary illumination light, water, and gas are supplied from the light source device 11 to the endoscope 10 via the light guide 35 and a fluid tube (not shown).

As a result, the illumination light is emitted from the illumination window (not shown) of the tip surface 27a of the tip portion 27 toward the observed portion. Further, in response to the pressing operation of the air supply / water supply button 30, gas or water is injected from the air supply / water supply nozzle (not shown) on the tip surface 27a of the tip portion 27 toward the observation window (not shown) on the tip surface 27a.

When the connector 37b is connected to the processor 12, the signal cable 36 and the processor 12 are electrically connected. As a result, the image pickup element 28 of the endoscope 10 outputs the image pickup signal of the observed portion to the processor 12 and the control signal is output from the processor 12 to the endoscope 10 via the signal cable 36.

In the present embodiment, a flexible endoscope has been described as an example of the endoscope 10, but as the endoscope 10, various types of electrons capable of capturing moving images of an observed portion such as a rigid endoscope are used. An endoscope may be used.

The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. As the illumination light, white light or light in a specific wavelength band can be applied. The illumination light may be a combination of white light and light in a specific wavelength band. The light source device 11 is configured so that light in a wavelength band according to the purpose of observation can be appropriately selected as illumination light.

The white light may be either light in a white wavelength band or light in a plurality of wavelength bands. The specific wavelength band is narrower than the white wavelength band. As the light of a specific wavelength band, light of one kind of wavelength band may be applied, or light of a plurality of wavelength bands may be applied. A particular wavelength band is sometimes referred to as special light.

The processor 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. Further, the processor 12 acquires an image pickup signal from the image pickup element 28 of the endoscope 10 via the connector 37b and the signal cable 36. The processor 12 applies a predetermined frame rate to acquire an image pickup signal output from the endoscope 10.

The processor 12 generates a moving image 38 of the observed portion based on the imaging signal acquired from the endoscope 10. Further, when the still image shooting instruction unit 32 is operated by the operation unit 21 of the endoscope 10, the processor 12 is observed based on the image pickup signal acquired from the image pickup element 28 in parallel with the generation of the moving image 38. A still image 39 of the part is generated. The still image 39 may be generated at a higher resolution than the resolution of the moving image 38.

When generating the moving image 38 and the still image 39, the processor 12 corrects the image quality by applying digital signal processing such as white balance adjustment and shading correction. The processor 12 may add incidental information defined by the DICOM (Digital Imaging and Communications in Medicine) standard to the moving image 38 and the still image 39.

The moving image 38 and the still image 39 are in-vivo images taken in the subject, that is, in the living body. Further, when the moving image 38 and the still image 39 are images obtained by imaging using light in a specific wavelength band, both are special optical images. Then, the processor 12 outputs the generated moving image 38 and the still image 39 to the display device 13 and the medical image processing device 14, respectively. The processor 12 may output the moving image 38 and the still image 39 to the image storage device 18 via the network 17 according to the communication protocol conforming to the DICOM standard.

The display device 13 is connected to the processor 12. The display device 13 displays the moving image 38 and the still image 39 input from the processor 12. A user such as a doctor performs an advance / retreat operation of the insertion unit 20 while checking the moving image 38 displayed on the display device 13, and when a lesion or the like is detected at the observed site, the still image photographing instruction unit 32 is used. It can be operated to take a still image of the observed part.

A computer is used as the medical image processing device 14. As the operation device 15, a keyboard, a mouse, or the like that can be connected to a computer is used. The connection between the operating device 15 and the computer may be either a wired connection or a wireless connection. As the monitor device 16, various monitors that can be connected to a computer are used.

As the medical image processing device 14, a diagnostic support device such as a workstation or a server device may be used. In this case, the operating device 15 and the monitoring device 16 are provided for each of a plurality of terminals connected to a workstation or the like. Further, as the medical image processing device 14, a medical service support device that supports the creation of a medical report or the like may be used.

The medical image processing device 14 acquires the moving image 38 and stores the moving image 38. The medical image processing device 14 acquires the still image 39 and stores the still image 39. The medical image processing device 14 controls the reproduction of the moving image 38 and the reproduction of the still image 39.

The operation device 15 is used for inputting an operation instruction to the medical image processing device 14. The monitoring device 16 displays the moving image 38 and the still image 39 under the control of the medical image processing device 14. The monitor device 16 functions as a display unit for various information in the medical image processing device 14. The monitor device 16 is an example of a display unit that displays the extraction result of the virtual endoscopic image. The monitor device 16 is an example of a display unit that displays the extraction result of the actual endoscopic image.

The image storage device 18 connected to the medical image processing device 14 via the network 17 stores the CTC image 19. The CTC image 19 is generated using a CTC image generator (not shown). CTC is an abbreviation for CT colonography, which represents a three-dimensional CT examination of the large intestine.

A CTC image generator (not shown) generates a CTC image from a three-dimensional inspection image. The three-dimensional inspection image is generated from an imaging signal obtained by imaging the inspection target site using a three-dimensional image imaging device. Examples of the three-dimensional image imaging device include a CT device, an MRI device, a PET (Positron Emission Tomography), an ultrasonic diagnostic device, and the like. In this embodiment, an example is shown in which a CTC image 19 is generated from a three-dimensional inspection image obtained by photographing the large intestine.

The endoscope system 9 may be communicably connected to the server device via the network 17. The server device can be applied to a computer that stores and manages various data. The information stored in the image storage device 18 shown in FIG. 1 may be managed by using a server device. Note that the DICOM standard, a protocol compliant with the DICOM standard, and the like can be applied to the storage format of the image data and the communication between the devices via the network 17.

[Functions of medical image processing equipment]
FIG. 2 is a functional block diagram showing the functions of the medical image processing apparatus. The medical image processing apparatus 14 shown in FIG. 2 includes a computer (not shown). The computer functions as an image acquisition unit 41, an information acquisition unit 42, a medical image analysis processing unit 43, and a display control unit 44 based on the execution of the program. The medical image processing device 14 includes a storage unit 47 that stores information used for various controls of the medical image processing device 14.

[Image acquisition section]
The image acquisition unit 41 includes a CTC image acquisition unit 41a and an endoscopic image acquisition unit 41b. The CTC image acquisition unit 41a acquires a CTC image via an image input / output interface (not shown). The endoscope image acquisition unit 41b acquires the endoscope image 37 via an image input / output interface (not shown). The connection form of the image input / output interface may be wired or wireless. The CTC image acquisition unit 41a and the endoscopic image acquisition unit 41b will be described in detail below.

<< CTC image acquisition unit >>
The CTC image acquisition unit 41a acquires the CTC image 19 stored in the image storage device 18 shown in FIG. The CTC image 19 acquired by using the CTC image acquisition unit 41a shown in FIG. 2 is stored in the image storage unit 48. The CTC image acquisition unit 41a can be applied with the same configuration as the endoscopic image acquisition unit 41b described later. Reference numeral 19b represents a viewpoint image. The viewpoint image 19b is an image of the field of view at the viewpoint set in the CTC image 19. The viewpoint is illustrated with reference numeral P in FIG. The viewpoint image and the details of the viewpoint will be described later.

Here, the term image in the present embodiment includes the concept of data representing an image or the concept of a signal. The CTC image is an example of a virtual endoscopic image. The CTC image corresponds to a virtual colonoscopic image. The CTC image acquisition unit 41a is an example of a first image input unit for inputting a virtual endoscopic image.

《Endoscopic image acquisition section》
The endoscopic image acquisition unit 41b acquires the endoscopic image 37 generated by using the processor 12 shown in FIG. The endoscopic image 37 includes a moving image 38 and a still image 39 shown in FIG. In the present embodiment, the endoscope image 37 generated by using the processor 12 shown in FIG. 1 is acquired, but the endoscope image 37 stored in the external storage device may be acquired. The endoscopic image acquisition unit 41b shown in FIG. 2 may acquire the endoscopic image 37 via various information storage media such as a memory card.

When the still image 39 is taken during the shooting of the moving image 38, the endoscopic image acquisition unit 41b acquires the moving image 38 and the still image 39 from the processor 12 shown in FIG. The medical image processing device 14 stores the moving image 38 and the still image 39 acquired by using the endoscopic image acquisition unit 41b in the image storage unit 48. Reference numeral 38a represents a plurality of frame images constituting the moving image 38.

The medical image processing device 14 does not need to store all the moving images 38 of the endoscopic image 37 input from the processor 12 or the like in the image storage unit 48, and the still image capturing instruction unit 3 shown in FIG.
When a still image of the observed portion is taken in response to the operation of 2, the moving image 38 for 1 minute before and after that may be stored in the image storage unit 48 shown in FIG. The 1 minute before and after the shooting represents a period from 1 minute before shooting to 1 minute after shooting.

The endoscopic image acquisition unit 41b is an example of a second image input unit that inputs an actual endoscopic image. The endoscopic image 37 corresponds to a real endoscopic image.

[Information acquisition department]
The information acquisition unit 42 acquires information input from the outside via the operation device 15 or the like. For example, when the determination result and the extraction result determined by the user are input using the operation device 15, the information acquisition unit 42 acquires the determination information and the extraction information of the user.

The information acquisition unit 42 is an example of a determination result input unit. The information acquisition unit 42 is an example of an extraction result input unit.

[Medical image analysis processing department]
The medical image analysis processing unit 43 collects information on lesions that are detected in the examination using the endoscopic image 37 and not detected in the examination using the CTC image 19. Details of collecting lesion information will be described later. The details of the medical image analysis processing unit 43 will be described later.

[Deep learning algorithm]
The medical image analysis processing unit 43 performs image analysis processing using deep learning based on the deep learning algorithm 65. The deep learning algorithm 65 is an algorithm including a known convolution neural network method, a fully connected layer, and an output layer.

Deep learning is sometimes called deep learning. The convolution neural network is an iterative process of the convolution layer and the pooling layer. Convolutional neural networks are sometimes referred to as convolutional neural networks. Since the image analysis process using deep learning is a known technique, a specific description thereof will be omitted. Deep learning is an example of machine learning.

[Display control unit]
The display control unit 44 controls the image display of the monitor device 16. The display control unit 44 functions as a reproduction control unit 44a and an information display control unit 44b.

<< Playback control unit >>
The reproduction control unit 44a controls the reproduction of the CTC image 19 acquired by using the CTC image acquisition unit 41a and the endoscopic image 37 acquired by using the endoscopic image acquisition unit 41b. When the operation device 15 is used to reproduce an image, the reproduction control unit 44a executes a display control program to control the monitor device 16. The display control program is included in the program stored in the program storage unit 49.

As an example of display control of the CTC image 19 and the endoscopic image 37, an example of displaying either the CTC image 19 or the endoscopic image 37 in full screen can be mentioned. As another display control example of the CTC image 19 and the endoscope image 37, an example of displaying the CTC image 19 and the endoscope image 37 in parallel on one screen can be mentioned. The reproduction control unit 44a may switch between the two displays described above.

As an example of display control of the endoscopic image 37, there is a form in which either the moving image 38 or the still image 39 is displayed in full screen. As another display example of the endoscopic image 37, there is a form in which the moving image 38 and the still image 39 are displayed in parallel on one screen. The reproduction control unit 44a may switch between the two displays described above.

<< Information display control unit >>
The information display control unit 44b controls the display of the incidental information of the CTC image 19 and the display control of the incidental information of the endoscopic image 37. As an example of the incidental information of the CTC image 19, the coordinate values of each region constituting the CTC image 19 can be mentioned. As another example of the incidental information of the CTC image 19, information representing a feature region can be mentioned. Details of the feature area will be described later.

Examples of incidental information of the endoscopic image 37 include the frame number of the frame image 38a constituting the moving image 38 and the imaging time of the moving image 38. Examples of incidental information of the endoscopic image 37 include the imaging conditions of the still image 39, the imaging time of the still image 39, and the thumbnail image of the still image 39.

The information display control unit 44b controls the display of information necessary for various processes in the medical image analysis processing unit 43. Examples of various processes in the medical image analysis processing unit 43 include a correspondence process between the CTC image 19 and the endoscope image 37, a feature area extraction process of the CTC image 19, and a feature area extraction process of the endoscope image 37. Be done.

As another example of various processes in the medical image analysis processing unit 43, a comparison process between the CTC image 19 and the endoscope image 37, a process of imparting the extraction result of the CTC image 19 to the characteristic region of the endoscope image 37, and a CTC Examples include storage processing of a characteristic area of the endoscopic image 37 based on the extraction result of the image 19. Details of various processes in the medical image analysis processing unit 43 listed here will be described later.

[Memory]
The storage unit 47 includes an image storage unit 48. The image storage unit 48 stores the CTC image 19 acquired by the medical image processing device 14 and the endoscopic image 37. Although FIG. 2 illustrates an embodiment in which the medical image processing device 14 includes a storage unit 47, a storage device or the like that is communicably connected to the medical image processing device 14 via a network may include the storage unit 47. .. An example of the above-mentioned storage device is an image storage device 18 which is communicably connected via the network 17 shown in FIG.

The storage unit 47 includes a program storage unit 49. The program stored using the program storage unit 49 includes an application program for causing the medical image processing device 14 to execute the reproduction control of the moving image 38. Further, the program stored using the program storage unit 49 includes a program for causing the medical image processing device 14 to execute the processing of the medical image analysis processing unit 43.

The medical image processing device 14 may be configured by using a plurality of computers or the like. A plurality of computers and the like may be connected so as to be able to communicate with each other via a network. The plurality of computers referred to here may be separated in terms of hardware, or may be integrally configured in terms of hardware and may be functionally separated.

[Hardware configuration of various control units]
The hardware-like structure of the control unit that executes various controls of the medical image processing apparatus 14 shown in FIG. 2 is various processors as shown below. The same applies to the medical image analysis processing unit 43 shown in FIG.

For various processors, CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software and functions as various control units, is a processor whose circuit configuration can be changed after manufacturing. A dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing a specific process such as a certain PLD (Programmable Logic Device) or ASIC (Application Specific Integrated Circuit), is included. The software here is synonymous with a program.

One processing unit may be configured by one of these various processors, or may be configured by using two or more processors of the same type or different types. Examples of two or more processors include a plurality of FPGAs or a combination of a CPU and an FPGA. Further, a plurality of control units may be configured by one processor. As an example in which a plurality of control units are configured by one processor, first, as represented by a computer such as a client device and a server device, one combination of one or more CPUs and software is used. There is a form in which a processor is configured and this processor functions as a plurality of control units. Secondly, as typified by SoC (System On Chip), there is a form in which a processor that realizes the functions of the entire system including a plurality of control units with one IC chip is used. As described above, the various control units are configured by using one or more of the various processors described above as a hardware structure. IC is an abbreviation for Integrated Circuit, which is an English notation for integrated circuits.

[Detailed explanation of medical image analysis processing unit]
FIG. 3 is a functional block diagram showing the functions of the medical image analysis processing unit. The endoscope 10 in the following description is illustrated in FIG. The CTC image 19, the viewpoint image 19b, the endoscope image 37, and the frame image 38a are shown in FIG.

The medical image analysis processing unit 43 shown in FIG. 3 includes a first feature area extraction unit 51, a first condition setting unit 52, a second feature area extraction unit 54, a second condition setting unit 56, and an association unit. It includes 57, a comparison unit 58, a comparison result giving unit 60, a storage unit 61, and an extraction rule updating unit 62.

<< First feature area extraction unit >>
The first feature region extraction unit 51 extracts the first feature region, which is a feature region that meets the specified first condition, from the CTC image 19. Examples of first feature regions of CTC image 19 include lesions, folds, points of change between colons, and blood vessels. The blood vessel includes a running pattern of the blood vessel.

<< First condition setting unit >>
The first condition setting unit 52 sets the first condition. The first condition is an extraction condition applied to the extraction process using the first feature region extraction unit 51. The first condition setting unit 52 can set the information input by using the operation device 15 shown in FIG. 2 as the first condition. The above-mentioned example of the first feature region is grasped as an example of the first condition.

<< Second feature area extraction unit >>
The second feature region extraction unit 54 extracts a second feature region, which is a feature region that meets the specified second condition, from the endoscopic image 37 shown in FIG. Examples of the second feature region of the endoscopic image 37 include lesions, folds, change points between colons, and blood vessels, as in the first feature region of the CTC image 19.

The second feature region extraction unit 54 may automatically extract the second feature region that matches the second condition from the endoscopic image 37. The second feature region extraction unit 54 may acquire an extraction result in which the user manually extracts the second feature region that matches the second condition from the endoscopic image 37. The user may input the extraction result manually extracted using the information acquisition unit 42 shown in FIG.

<< Second condition setting unit >>
The second condition setting unit 56 sets a second condition corresponding to the first condition as an extraction condition of the second feature region of the endoscopic image 37. The second condition corresponding to the first condition includes the same second condition as the first condition. For example, when a lesion is set as the first condition, the lesion may be set as the second condition.

In the first condition and the second condition, specific lesions such as polyps and inflammation may be set instead of the comprehensive concept of lesions. Further, the first condition and the second condition may be a combination of a plurality of conditions.

When a lesion is set as the second condition and the lesion is extracted from the endoscopic image 37 as the second characteristic region, it corresponds to the detection of the lesion in the endoscopy. The above-mentioned example of the second characteristic region is grasped as an example of the second condition.

<< Correspondence part >>
The association unit 57 associates the CTC image 19 shown in FIG. 2 with the endoscopic image 37. An example of the association between the CTC image 19 and the endoscopic image 37 is the association between the first feature region of the CTC image 19 and the second feature region of the endoscopic image 37. For example, when a lesion is detected as the second feature region of the endoscopic image 37, the first feature region of the CTC image 19 corresponding to the detected lesion is associated with the second feature region of the endoscopic image 37. ..

The association between the CTC image 19 and the endoscopic image 37 includes the association between the non-extracted region of the CTC image 19 and the second feature region of the endoscopic image 37.

《Comparison part》
The comparison unit 58 compares the CTC image 19 with the endoscopic image 37. Then, among the second feature regions of the endoscopic image 37, the second feature region associated with the non-extracted region of the CTC image 19 is specified.

In other words, the comparison unit 58 determines whether or not the second feature region of the endoscopic image 37 is associated with the non-extracted region of the CTC image 19. The comparison unit 58 may execute the automatic determination. The comparison unit 58 may include a determination result input unit for inputting a user's determination result. The comparison result of the comparison unit 58 is stored using the storage unit 61. Details of the comparison results will be described later.

《Comparison result giving section》
The comparison result giving unit 60 shown in FIG. 3 gives the comparison result of the comparison unit 58 to the endoscopic image 37. It is also possible that the comparison unit 58 takes on the function of the comparison result giving unit 60. The endoscopic image 37 to which the comparison result of the comparison unit 58 is given is stored by using the storage unit 61.

The comparison result of the comparison unit 58 can be grasped as the extraction result of the first feature region. The comparison result giving unit 60 is an example of the extraction result giving unit.

《Preservation part》
The storage unit 61 stores the comparison result of the comparison unit 58. The storage unit 61 may also be used as another storage unit. The comparison result stored using the storage unit 61 is used for updating the extraction rule applied to the first feature region extraction unit 51. Note that updating the extraction rule is a concept that includes changing the extraction rule. The storage of the comparison result here is synonymous with the memory of the comparison result. In the present specification, preservation and memory are treated as synonymous without distinction.

<< Extraction rule update section >>
The extraction rule update unit 62 updates the extraction rule applied to the first feature area extraction unit 51 by using the coordinate values of the non-extraction area associated with the second feature area of the endoscopic image 37. The update of the extraction rule applied to the first feature region extraction unit 51 is executed by using the deep learning algorithm 65. The extraction rule after the update is stored using an update rule storage unit (not shown).

[Explanation of medical image analysis method]
Next, a medical image analysis method in a large intestine examination will be described. The large intestine examination is an example, and the medical image analysis method described below can be applied to an examination of other parts such as the bronchi.

The medical image analysis method described below is a lesion or the like that is detected in the endoscopy using the endoscopic image 37 shown in FIG. 2 and is not detected in the virtual colonoscopy using the CTC image 19. This is an example of an image processing method for collecting information on the above.

[CTC image]
FIG. 4 is a schematic diagram of a CTC image. The overall image 19a shown in FIG. 4 is a form of the CTC image 19 showing the entire large intestine, which is the observed site. The observed site is synonymous with the subject and the observation target of the subject.

Entire image 19a is placed one or more viewpoints P on the path 19c that is set from the start point P _0, while changing the sequentially viewpoint P to the end point, not shown, inside of the lumen from the viewpoint P It is an image assuming that you have seen it. The path 19c can be generated by thinning the entire image 19a. A known thinning method can be applied to the thinning process. Although a plurality of viewpoints P are shown in FIG. 4, the arrangement and quantity of the viewpoints P can be appropriately determined according to inspection conditions and the like.

When each viewpoint P of the entire image 19a is specified, it is possible to display a viewpoint image representing an image of the field of view at the designated viewpoint P. The viewpoint image at each viewpoint P is shown in FIG. 6 with reference numerals 19b ₁ and 19b ₂ .

At each viewpoint P, a viewpoint image reflecting the imaging direction of the endoscope 10 may be generated. The viewpoint image reflecting the imaging direction of the endoscope 10 may be generated for each of the plurality of imaging directions. The overall image 19a shown in FIG. 4 and the viewpoint image (not shown in FIG. 4) are included in the concept of the CTC image 19 shown in FIG.

The endoscopic image 37 whose overall image 19a is shown in FIG. 4 is set with three-dimensional coordinates (not shown). As the three-dimensional coordinates set in the endoscope image 37, the three-dimensional coordinates having an arbitrary reference position of the endoscope image 37 as the origin can be applied. As the three-dimensional coordinates, any three-dimensional coordinates such as Cartesian coordinates, polar coordinates, and cylindrical coordinates can be applied.

[Virtual colonoscopy]
In virtual colonoscopy, a CT device is used to image the large intestine, a CT image of the large intestine is acquired, and a CTC image 19 generated by performing image processing on the CT image of the large intestine is used to detect a lesion or the like. .. In the virtual colonoscopy, the pointer 19d, which looks like an endoscope 10, is moved along the path 19c in conjunction with the movement of the endoscope 10. The position of the pointer 19d is derived from the movement condition of the endoscope 10.

For example, to set the start point of view corresponding pointer 19d to the start point P ₀ is the moving start point of the endoscope 10. Start point P ₀ of the endoscope 10 is applicable to insertion start position of the endoscope 10 shown in FIG. Start point P ₀ of the endoscope 10, may be applied to the movement start position of the endoscope 10 (not shown). An example of the insertion start position of the endoscope 10 is a position corresponding to the anus. An example of the movement start position of the endoscope 10 is a position corresponding to the cecum.

The endoscope 10 can grasp the position inside the observed portion by using a sensor (not shown). Further, the endoscope 10 can derive a movement vector representing the movement speed and the movement direction of the endoscope 10 by using a sensor (not shown).

The pointer 19d can be moved in conjunction with the movement of the endoscope 10 by using the position inside the observed portion of the endoscope 10 and the movement vector of the endoscope 10. The position of the pointer 19d in the whole image 19a can be specified by using the coordinate values.

Since the CTC image 19 has three-dimensional information, the virtual colonoscopy is strong in detecting convex shapes such as polyps. It is also strong in detecting polyps hidden behind folds. On the other hand, since the CTC image 19 does not have color information and texture information, virtual colonoscopy is not good at detecting flat lesions and differences in surface conditions.

[Endoscopic image]
In endoscopy, lesions and the like are detected using the endoscopic image 37. That is, in the endoscopy, the position, shape, and the like of the lesion are specified by observing the moving image 38 generated in real time using the endoscope 10. For the endoscopy, a reproduced image of the endoscopic image 37 may be used.

FIG. 5 is a schematic view of an endoscopic image. As an example of the endoscopic image 37, an arbitrary frame image 38a constituting the moving image 38 is shown in FIG. The frame image 38a shown in FIG. 5 is a two-dimensional image. The frame image 38a has color information and texture information.

Since the endoscopic image 37 has color information and texture information, endoscopy is strong in detecting flat lesions, differences in surface conditions, and the like. Endoscopy can detect lesions that could not be detected by virtual colonoscopy. Therefore, it is possible to generate information that interpolates the virtual colonoscopy using the test results of the endoscopy.

The processing that constitutes the medical image analysis method used for collecting information that interpolates the virtual colonoscopy using the examination results of the endoscopy will be described below. In this embodiment, an example of collecting information on lesions that virtual colonoscopy is not good at is shown.

In the present embodiment, the position of the endoscopic image 37 is specified by using the frame image 38a. The position of the endoscopic image 37 may be specified without using the frame image 38a.

[Feature area extraction process]
FIG. 6 is an explanatory diagram of the first feature region extraction. FIG. 6 illustrates the viewpoint image 19b ₁ and the viewpoint image 19b ₂ at an arbitrary viewpoint P among the CTC images 19. The concept that includes the viewpoint image 19b ₁ and the viewpoint image 19b ₂ shown in FIG. 6 is the viewpoint image 19b.

In the first feature region extraction process, the first feature region 80 is extracted from the CTC image 19 shown in FIG. 6 by using the first feature region extraction unit 51 shown in FIG. A known feature region extraction technique can be applied to the process of extracting the first feature region 80 from the CTC image 19. The same applies to the second feature region extraction described later.

In the viewpoint image 19b ₁ shown in FIG. 6, a convex polyp is extracted as the first feature region 80. On the other hand, although not shown in FIG. 6, the CTC image 19 has a region in which the first feature region is not extracted. In the CTC image 19, a region in which the first feature region is not extracted is defined as a non-extracted region.

In both the first feature region 80 of the CTC image 19 and the non-extracted region (not shown), the coordinate values in the three-dimensional coordinates set in the CTC image 19 can be specified.

FIG. 7 is an explanatory diagram of the second feature region extraction. FIG. 7 shows an arbitrary frame image 38a ₁ of the endoscopic images 37. The extraction result of the second characteristic region can be treated as the result of endoscopy.

In the second feature region extraction process, the second feature region 70 is extracted from the endoscopic image 37 using the second feature region extraction unit 54 shown in FIG. In the frame image 38a ₁ shown in FIG. 6, a convex polyp is extracted as the second feature region 70. The convex polyp, which is the second feature region 70 shown in FIG. 7, can be extracted from the CTC image 19.

FIG. 8 is an explanatory diagram of another example of the second feature region extraction. In the frame image 38a ₃₁ shown in FIG. 8, an inflamed lesion is extracted as the second characteristic region 76. The second feature region 76 of the frame image 38a ₃₁ shown in FIG. 8 is an example of a feature region that can be detected by endoscopy, but is difficult to detect by virtual colonoscopy.

The information of the first feature region 80 shown in FIG. 6 and the information of the non-extracted region (not shown) are stored in the storage unit 61 shown in FIG. 2 as the extraction result of the first feature region. Further, the information of the second feature region 70 shown in FIG. 7 and the information of the second feature region 76 shown in FIG. 8 are shown in FIG. 2 as the extraction result of the second feature region or the endoscopy result. It is stored in the storage unit 61.

[Association processing]
The association processing will be described with reference to FIGS. 9 to 11. FIG. 9 is a schematic diagram showing an example of associating lesions. FIG. 9 shows an example in which the second feature region 70, which is a convex polyp, is detected in the frame image 38a ₁ of the endoscopic image 37.

Note that the viewpoint image 19b ₁ shown in FIG. 9 is a viewpoint image 19b ₁ shown in FIG. Viewpoint image 19b ₂ as shown in FIG. 9 is a viewpoint image 19b ₂ shown in FIG. The first feature region 80 shown in FIG. 9 is the first feature region 80 shown in FIG.

The frame image 38a ₁ shown in FIG. 9 is a frame image 38a ₁ shown in FIG. The second feature region 70 shown in FIG. 9 is the second feature region 70 shown in FIG. 7.

Hereinafter, a case where endoscopy is performed while moving the pointer 19d indicating the position of the endoscope 10 along the path 19c of the entire image 19a corresponding to the movement of the endoscope 10 will be described. The position of the endoscope 10 is known by using the sensor included in the endoscope 10. Further, the orientation of the endoscope 10 at each viewpoint P is the tangential direction at each viewpoint P of the path 19c shown in FIG.

When the second feature region 70 is extracted from the endoscopic image 37, the association unit 57 shown in FIG. 3 searches the CTC image 19 for the first feature region 80 corresponding to the second feature region 70. When the first feature region 80 of the CTC image 19 corresponding to the second feature region 70 of the endoscopic image 37 is detected, the first feature region 80 of the CTC image 19 and the second feature region 80 of the endoscopic image 37 Corresponds to 70.

In other words, the concept of associating the CTC image 19 with the endoscopic image 37 includes the concept of forming a pair of the components of the CTC image 19 and the components of the endoscopic image 37. The concept of associating the CTC image 19 with the endoscope image 37 may include the concept of searching for and specifying the component of the CTC image 19 corresponding to the component of the endoscope image 37. ..

FIG. 10 is a schematic diagram showing an example of associating folds. The frame image 38a ₁₁ shown in FIG. 10 has folds extracted as the second feature region 72. The viewpoint image 19b ₁₁ has folds extracted as the first feature region 82. Figure 10 is a perspective image _{19b 12} at the viewpoint P successive relative view point P of the viewpoint image _{19b 11} and the viewpoint image _{19b 13,} is illustrated. The association unit 57 shown in FIG. 3 associates the first feature area 82 shown in FIG. 10 with the second feature area 72.

FIG. 11 is a schematic diagram showing an example of associating folds using fold numbers. The number of folds does not change in either the CTC image 19 or the endoscopic image 37. Therefore, it is possible to determine the reference folds and associate the CTC image 19 with the endoscopic image 37 using the fold numbers.

The frame image 38a ₂₁ shown in FIG. 11 has folds extracted as the second feature region 74. The viewpoint image 19b ₂₁ has folds extracted as the first feature region 84. The viewpoint image 19b ₂₂ and the viewpoint image 19b ₂₃ shown in FIG. 11 are also pleated as the second feature region. The illustration of the second feature region of the viewpoint image 19b ₂₂ and the viewpoint image 19b ₂₃ is omitted.

N ₁ attached to the viewpoint image 19b ₂₁ is an integer representing the number of folds. The same applies to n ₂ attached to the viewpoint image 19b ₂₂ , n ₃ attached to the viewpoint image 19b ₂₃ , and n ₁ attached to the frame image 38a ₂₁ .

The number _{n 1} of folds in the frame image _{38a 21,} if the number _{n 1} of folds in the viewpoint image _{19b 21} matches, mapping unit 57 shown in FIG. 3, a second characteristic region 74 shown in FIG. 11 It is associated with the first feature area 84.

Although not shown, the conversion points between the colons and the blood vessels can be associated with each other in the same manner as in the examples described with reference to FIGS. 9 to 11.

[Comparison processing]
Next, the comparison process will be described with reference to FIG. FIG. 12 is an explanatory diagram of an example of a comparison process between a CTC image using a lesion and an endoscopic image. The comparison process is executed using the comparison unit 58 shown in FIG.

In the comparison process, the CTC image 19 from which the first feature region such as the first feature region 80 is extracted and the endoscopic image from which the second feature region such as the second feature region 70 and the second feature region 76 are extracted are extracted. Compare with 37. The comparison process compares the corresponding positions of the CTC image 19 and the endoscopic image 37 with each other.

Specifically, among the second feature regions of the endoscope image 37, the second feature region 76 of the endoscope image 37 that is not associated with the first feature region of the CTC image 19 is specified. In other words, the second feature region 76 of the endoscopic image 37 associated with the non-extracted region 86 of the CTC image 19 is specified. The comparison result of the comparison process is stored in the storage unit 61 shown in FIG.

In the frame image 38a ₁ shown in FIG. 12, a convex polyp is extracted as the second feature region 70. The second feature region 70 of the frame image 38a ₁ is associated with the first feature region 80 of the viewpoint image 19b ₁ . Therefore, the frame image 38a ₁ is not a specific target.

On the other hand, in the frame image 38a ₃₁ shown in FIG. 12, an inflamed lesion is extracted as the second characteristic region 76. However, the second feature region 76 of the frame image 38a ₃₁ is not associated with the first feature region 80 or the like of the CTC image 19.

Since the virtual colonoscopy is not good at detecting flat lesions and differences in surface conditions, the first feature region corresponding to the second feature region 76 is not extracted from the CTC image 19. Therefore, the frame image 38a ₃₁ is a specific target.

When the frame image 38a ₃₁ is specified, the coordinate value of the non-extracted area 86 of the CTC image 19 corresponding to the second feature area 76 of the frame image 38a ₃₁ is specified.

The viewpoint image 19b ₃₁ shown in FIG. 12 is a viewpoint image at the viewpoint P corresponding to the imaging position of the frame image 38a ₃₁ . The position of the non-extracted region 86 of the viewpoint image 19b ₃₁ corresponds to the position of the second feature region 76 of the frame image 38a ₃₁ .

[Comparison result granting process]
The comparison result giving process is executed by using the comparison result giving unit 60 shown in FIG. In the comparison result giving process, the comparison result of the comparison process is given to the frame image 38a from which the second feature area associated with the non-extracted area is extracted.

For example, the coordinate values of the non-extracted region 86 in the viewpoint image 19b ₃₁ are given to the frame image 38a ₃₁ from which the second feature region 76 shown in FIG. 12 is extracted as a comparison result of the comparison process.

[Preservation process]
The storage process is executed using the storage unit 61 shown in FIG. In the storage process, the endoscopic image 37 to which the comparison result is given is stored. Specifically, the storage unit 61 stores the endoscopic image 37 to which the coordinate values of the non-extracted region 86 are assigned.

The information stored using the storage unit 61 is information on lesions extracted from the endoscopic image 37 that are not extracted from the CTC image 19. Therefore, among the lesions extracted from the endoscopic image 37, information on the lesions not extracted from the CTC image 19 can be collected.

[Extraction rule update process]
The extraction rule update process is executed using the extraction rule update unit 62 shown in FIG. Specifically, the extraction rule updating unit 62 executes the deep learning algorithm 65 on the information in the non-extracted region of the CTC image 19 corresponding to the lesion extracted from the endoscopic image 37, and the CTC image 19 to the first. The extraction rule for extracting the first feature region that matches one condition is updated.

In the extraction process of the first feature region using the extraction rule after the update, the first feature region can be extracted at the position of the CTC image 19 corresponding to the non-extraction region when the extraction rule before the update is used.

[Procedure of image processing method]
FIG. 13 is a flowchart showing the procedure of the image processing method. When the image processing method is started, first, the CTC image input step S10 is executed. In the CTC image input step S10, the CTC image 19 is input from the image storage device 18 shown in FIG. 1 via the network 17 and the CTC image acquisition unit 41a shown in FIG. The CTC image 19 is stored using the image storage unit 48 shown in FIG. The CTC image input step S10 is an example of the first image input step.

After the CTC image 19 is input in the CTC image input step S10 shown in FIG. 13, the first feature region extraction step S12 is executed. In the first feature region extraction step S12, the first feature region is extracted from the CTC image 19 by using the first feature region extraction unit 51 shown in FIG. An example of the first feature region is illustrated with reference numeral 80 in FIG.

In the endoscope image input step S14 shown in FIG. 13, the endoscope image 37 is input via the endoscope image acquisition unit 41b shown in FIG. In the endoscope image input step S14 shown in FIG. 13, the moving image 38 captured by the endoscope 10 shown in FIG. 1 is acquired in real time. The moving image 38 is shown in FIG. The moving image 38 is stored using the image storage unit 48. In the following description, the moving image 38 can be read as an endoscopic image 37. The endoscopic image input step S14 is an example of the second image input step.

The moving image 38 input in the endoscopic image input step S14 is displayed on the monitor device 16 shown in FIG. Further, the monitoring device 16 displays the whole image 19a shown in FIG. 4, and causes the pointer corresponding to the position of the endoscope 10 to be displayed on the whole image 19a. Further, the pointer of the whole image 19a moves on the path 19c in conjunction with the endoscope 10.

When the moving image 38 is input in the endoscopic image input step S14, the second feature region extraction step S16 is executed. In the second feature region extraction step S16, the second feature region is extracted from the moving image 38 shown in FIG. 2 by using the second feature region extraction unit 54 shown in FIG. An example of the second feature region is shown in FIG. 12 with reference numerals 70 and 76.

The second feature region extraction step S16 shown in FIG. 13 is executed by applying a predetermined sampling cycle during the period in which the moving image 38 is input. The second feature region extraction step S16 corresponds to lesion detection in endoscopy.

In the second feature region extraction step S16, the association step S18 is executed each time the second feature region is created from the moving image 38. The association step S18 is extracted in the first feature region of the CTC image 19 extracted in the first feature region extraction step S12 and in the second feature region extraction step S16 by using the association unit 57 shown in FIG. The association with the second feature area of the moving image 38 is executed. The result of the association is stored in the image storage unit 48 shown in FIG.

In the associating step S18 shown in FIG. 13, the comparison step S20 is executed after the result of associating the first feature area of the CTC image 19 with the second feature area of the moving image 38 is stored. In the comparison step S20, the CTC image 19 and the moving image 38 are compared, and the non-extracted region of the CTC image 19 associated with the second feature region of the moving image 38 is specified.

That is, in the comparison step S20, a non-extracted region that should be originally extracted as the first feature region is searched from the CTC image 19. An example of the non-extracted region identified in the comparative step S20 is illustrated with reference numeral 86 in FIG.

Further, in the comparison step S20 shown in FIG. 13, the coordinate values of the specified non-extracted region are specified. As the coordinate value of the non-extracted region, the coordinate value of the representative position of the non-extracted region such as the position of the center of gravity of the non-extracted region can be applied. As the coordinate values of the non-extracted region, a plurality of coordinate values representing the edges of the non-extracted region may be applied. The coordinate values of the non-extracted region identified in the comparison step S20 are stored in the storage unit 61 shown in FIG. 3 as a comparison result between the moving image 38 and the CTC image 19.

In the comparison step S20, after the comparison result between the moving image 38 and the CTC image 19 is derived, the process proceeds to the comparison result giving step S22. In the comparison result giving step S22, the comparison result derived in the comparison step S20 is given to the second feature region of the moving image 38 corresponding to the non-extracted region of the CTC image 19.

In the storage step S24, the moving image 38 to which the comparison result is given is stored in the storage unit 61 shown in FIG. The image processing method is terminated after the storage step S24. The information stored in the storage unit 61 can be used to update the extraction rules applied when extracting the first feature region from the CTC image 19.

[Action effect]
[1]
According to the endoscope system 9 and the image processing method configured as described above, among the second feature regions of the endoscope image 37, the non-extracted regions are not associated with the first feature region of the CTC image 19. A second feature region 76 corresponding to 86 is identified. When the second feature region 76 corresponding to the non-extracted region 86 is specified, the coordinate value of the non-extracted region 86 corresponding to the specified second feature region 76 is specified.

This makes it possible to efficiently collect information on the first feature region, which is specified in the endoscopic image 37 but not specified in the CTC image 19.

[2]
The extraction rule of the first feature region can be updated by using the information of the first feature region specified in the endoscopic image 37 but not specified in the CTC image 19.

[3]
Information on the coordinate values of the non-extracted region is added to the endoscopic image 37 from which the second feature region corresponding to the non-extracted region is extracted. As a result, it is possible to associate the endoscopic image 37 from which the second feature region has been extracted with the coordinate values of the non-extracted region.

Further, the extraction rule of the first feature region can be updated by using the correspondence between the second feature region and the non-extraction region of the endoscopic image 37.

[Modification example]
[Modification example of CTC image]
<< First example >>
The medical image processing apparatus 14 shown in FIG. 2 may include a CTC image generation unit that generates a CTC image 19 from a three-dimensional inspection image such as a CT image. The medical image processing apparatus 14 may acquire a three-dimensional inspection image via the CTC image acquisition unit 41a and generate a CTC image 19 using the CTC image generation unit.

<< Second example >>
The viewpoint P shown in FIG. 4 is not limited to the path 19c. The viewpoint P can be set at any position. The visual field direction of the viewpoint image 19b can be arbitrarily set according to the imaging direction of the endoscope 10.

<< Third example >>
The viewpoint image 19b may be a two-dimensional inspection image obtained by converting a three-dimensional inspection image in an arbitrary cross section of the entire image 19a into a two-dimensional image.

[Modified example of first feature region extraction]
<< First example >>
The three-dimensional inspection image used for generating the CTC image may be used for the extraction of the first feature region.

<< Second example >>
The extraction of the first feature region may be linked with the extraction of the second feature region of the endoscopic image 37. For example, when the second feature region is extracted from the endoscope image 37, the position of the CTC image 19 corresponding to the frame image 38a of the endoscope image 37 from which the second feature region is extracted is specified and specified. The first feature region may be extracted at the position of the CTC image 19 obtained. When the first feature region is not extracted, the coordinate value of the position of the specified CTC image 19 may be stored in the storage unit 61 shown in FIG.

<< Third example >>
The relationship between the sitting table values of the first feature area and the first feature area and the relationship between the coordinate values of the non-extracted area and the non-extracted area may be stored in a database. The above-mentioned database may be used when specifying the first feature area 80 corresponding to the second feature area 70 or the non-extracted area 86 corresponding to the second feature area 76. The database may be stored using the storage unit 61 shown in FIG. 3, or may be stored using a storage device external to the medical image processing device 14.

[Modification of second feature region extraction]
<< First example >>
The second condition for extracting the second characteristic region is not limited to the lesion. It may be a second condition that enables extraction of a second characteristic region that can improve the accuracy of virtual colonoscopy.

<< Second example >>
The extraction of the second feature region may be performed by reproducing the moving image 38.

[Modification example of information stored using the storage unit]
The storage unit 61 shown in FIG. 3 may include the first condition when the first feature region is extracted as the information of the non-extracted region corresponding to the second feature region of the endoscopic image 37.

[Variation example of comparison processing]
<< First example >>
In the comparison process, the user may specify the non-extracted region of the CTC image 19 corresponding to the second feature region of the endoscopic image 37. For example, the monitor device 16 shown in FIG. 1 displays the second feature region of the endoscopic image 37 and the non-extracted region of the CTC image 19.

Based on the second feature region of the endoscope image 37 displayed on the monitor device 16 and the non-extracted region of the CTC image 19, the user can use the non-extracted region of the CTC image 19 corresponding to the second feature region of the endoscope image 37. Identify the extraction area.

A specific information input unit for inputting specific information of the user is provided, and a non-extracted area of the CTC image 19 corresponding to the second feature area can be specified based on the specific information input via the specific information input unit. .. As the specific information input unit, the information acquisition unit 42 shown in FIG. 2 can be used.

<< Second example >>
The process of associating the first feature region of the CTC image 19 with the second feature region of the endoscopic image 37 may be omitted. That is, when the second characteristic region is extracted as a lesion from the endoscopic image 37, the CTC image 19 may be searched to identify a characteristic region similar to the second characteristic region.

For example, from the first feature region of the CTC image 19, the difference value from the feature quantity evaluation value of the second feature region of the endoscopic image 37 has a feature quantity evaluation value equal to or less than a specified threshold value. May be specified. The information on the position of the pointer 19d shown in FIG. 4 may be used for searching the CTC image 19.

[Modification example of storage unit]
The storage unit 61 shown in FIG. 3 may apply a storage device that is communicably connected to the medical image processing device 14 shown in FIG. 2 via a network.

[Modification example of information stored using the storage unit]
<< First example >>
The storage unit 61 shown in FIG. 3 may store the endoscopic image 37 associated with the non-extracted region 86 of the CTC image 19. The endoscopic image 37 referred to here shows the frame image 38a ₃₁ shown in FIG.

<< Second example >>
The storage unit 61 shown in FIG. 3 may store the coordinate values of the non-extracted area 86 associated with the second feature area 76 of the endoscopic image 37. When a plurality of coordinate values are used when specifying the non-extracted area 86, a plurality of coordinate values may be stored as the coordinate values of the non-extracted area 86.

That is, the storage unit 61 stores the frame image 38a ₃₁ to which the coordinate values of the non-extracted area 86 are added, the frame image 38a ₃₁ , and at least one of the coordinate values of the non-extracted area 86.

[Modification of illumination light]
The following modifications can be applied to a specific wavelength region.

<< First example >>
A first example of a particular wavelength band is the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nanometers or more and 450 nanometers or less, or 530 nanometers or more and 550 nanometers or less, and the light of the first example is 390 nanometers or more and 450 nanometers or less, or It has a peak wavelength in the wavelength band of 530 nanometers or more and 550 nanometers or less.

<< Second example >>
A second example of a particular wavelength band is the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nanometers or more and 615 nanometers or less, or 610 nanometers or more and 730 nanometers or less, and the light of the second example is 585 nanometers or more and 615 nanometers or less, or It has a peak wavelength in the wavelength band of 610 nanometers or more and 730 nanometers or less.

<< Third example >>
The third example of a specific wavelength band includes a wavelength band in which the absorption coefficient differs between oxidized hemoglobin and reduced hemoglobin, and the light in the third example has a peak wavelength in a wavelength band in which the absorption coefficient differs between oxidized hemoglobin and reduced hemoglobin. Has. The wavelength band of this third example includes a wavelength band of 400 ± 10 nanometers, 440 ± 10 nanometers, 470 ± 10 nanometers, or 600 nanometers or more and 750 nanometers or less, and the light of the third example is It has a peak wavelength in the wavelength band of 400 ± 10 nanometers, 440 ± 10 nanometers, 470 ± 10 nanometers, or 600 nanometers or more and 750 nanometers or less.

<< 4th example >>
The fourth example of the specific wavelength band is the wavelength band of the excitation light used for observing the fluorescence emitted by the fluorescent substance in the living body and exciting the fluorescent substance. For example, it is a wavelength band of 390 nanometers or more and 470 nanometers or less. The observation of fluorescence may be referred to as fluorescence observation.

<< Example 5 >>
A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nanometers or more and 820 nanometers or less, or 905 nanometers or more and 970 nanometers or less, and the light of the fifth example is 790 nanometers or more and 820 nanometers or less. Alternatively, it has a peak wavelength in a wavelength band of 905 nanometers or more and 970 nanometers or less.

[Example of generating a special light image]
The processor 12 may generate a special optical image having information in a specific wavelength band based on a normal optical image obtained by imaging with white light. Note that the generation here includes acquisition. In this case, the processor 12 functions as a special optical image acquisition unit. Then, the processor 12 obtains a signal in a specific wavelength band by performing an operation based on color information of red, green, and blue, or cyan, magenta, and yellow contained in a normal optical image.

In addition, red, green, and blue may be expressed as RGB (Red, Green, Blue). In addition, cyan, magenta, and yellow may be expressed as CMY (Cyan, Magenta, Yellow).

[Example of generating a feature image]
The processor 12 may generate a feature image such as a known oxygen saturation image based on at least one of a normal optical image and a special optical image.

[Example of application to image processing equipment]
An image processing apparatus can be configured using some of the configurations of the endoscopic system described above. For example, the medical image processing device 14 shown in FIG. 2 can function as an image processing device.

[Example of application to a program that makes a computer function as an image processing device]
The above-mentioned image processing method can be configured as a program that realizes a function corresponding to each step in the image processing method by using a computer. For example, a computer may be configured to implement a CTC image input function, a first feature area extraction function, an endoscopic image input function, a second feature area extraction function, a mapping function, a comparison function, and a storage function.

The CTC image input function corresponds to the first image input function. The endoscopic image input function corresponds to the second image input function.

It is possible to store the program that realizes the above-mentioned image processing function on a computer in a computer-readable information storage medium, which is a tangible non-temporary information storage medium, and provide the program through the information storage medium. is there.

Further, instead of the mode in which the program is stored and provided in the non-temporary information storage medium, the mode in which the program signal is provided via the network is also possible.

[Combination of Embodiments and Modifications]
The components described in the above-described embodiment and the components described in the modified examples can be used in combination as appropriate, or some components can be replaced.

In the embodiment of the present invention described above, the constituent requirements can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by a person having ordinary knowledge in the art within the technical idea of the present invention.

9 Endoscope system 10 Endoscope 11 Light source device 12 Processor 13 Display device 14 Medical image processing device 15 Input operation unit 16 Display device 17 Network 18 Image storage device 19 CTC image 19a Overall image 19b, 19b ₁ , 19b ₂ , 19b ₃ , 19b ₁₁ , 19b ₁₂ , 19b ₁₃ , 19b ₂₁ , 19b ₂₂ , 19b ₂₃ , 19b ₃₁ Viewpoint image 19c Path 19d Pointer 20 Insertion 21 Operation unit 22 Universal cord 25 Flexible part 26 Curved part 27 Tip part 28 Imaging element 29 Curved operation knob 30 Air supply / water supply button 31 Suction button 32 Still image shooting instruction unit 33 Treatment tool introduction port 35 Light guide 36 Signal cable 37 Endoscopic image 37a, 37b Connector 38 Moving image 38a, 38a ₁ , 38a ₁₁ , 38a ₂₁ , 38a ₃₁ Frame image 39 Still image 41 Image acquisition unit 41a CTC image acquisition unit 41b Endoscopic image acquisition unit 42 Information acquisition unit 43 Medical image analysis processing unit 44 Display control unit 44a Playback control unit 44b Information display control unit 47 Storage unit 48 Image storage unit 49 Program storage unit 51 First feature area extraction unit 52 First condition setting unit 54 Second feature area extraction unit 56 Second condition setting unit 57 Correspondence unit 58 Comparison unit 60 Comparison result giving unit 61 Storage unit 62 Extraction rule updater 65 Deep learning algorithm 70, 72, 76 Second feature area 80, 82, 84, 86 First feature area 86 Non-extraction area P Viewpoint P ₀ Start point n ₁ , n ₂ , n ₃ Fold number S10 From S22 Each step of the image processing method

Claims (18)

A first image input unit for inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject, and
A second image input unit that inputs an actual endoscopic image obtained by imaging the observation target of the subject using an endoscope, and
An associating unit that associates the virtual endoscopic image with the real endoscopic image,
A first feature region extraction unit that extracts a first feature region that matches the first condition from the virtual endoscopic image, and a first feature region extraction unit.
A second feature region extraction unit that extracts a second feature region that meets the second condition corresponding to the first condition from the actual endoscopic image, and a second feature region extraction unit.
Information on a non-extracted region that is associated with the second feature region of the real endoscopic image and is not extracted as the first feature region from the virtual endoscopic image, and is associated with the non-extracted region. A storage unit that stores at least one of the information in the second feature area, and
Image processing device equipped with. The image processing apparatus according to claim 1, further comprising an extraction result imparting unit that imparts an extraction result obtained by extracting the first characteristic region from the virtual endoscopic image to the real endoscopic image. Claim 1 or 2 including a comparison unit for comparing the extraction result of extracting the first feature region from the virtual endoscopic image with the extraction result of extracting the second feature region from the real endoscopic image. The image processing apparatus according to. The image processing apparatus according to claim 3, wherein the comparison unit compares the corresponding positions of the virtual endoscopic image and the real endoscopic image with each other. A determination result input unit for inputting a determination result for determining whether or not the second feature region of the real endoscopic image is a region associated with the non-extracted region of the virtual endoscopic image is provided. The image processing apparatus according to any one of claims 1 to 4. The image processing apparatus according to any one of claims 1 to 5, further comprising a display unit for displaying the extraction result of the virtual endoscopic image and the extraction result of the real endoscopic image. The image processing apparatus according to any one of claims 1 to 6, further comprising an extraction result input unit for inputting an extraction result obtained by extracting the second characteristic region from the actual endoscopic image. The image processing apparatus according to any one of claims 1 to 6, wherein the second feature region extraction unit automatically extracts the second feature region from the actual endoscopic image. The image processing apparatus according to any one of claims 1 to 8, wherein the second feature region extraction unit extracts a lesion as the second feature region from the actual endoscopic image. The storage unit stores at least one of the three-dimensional coordinate values of the non-extracted region in the virtual endoscopic image and the real endoscopic image including the second feature region corresponding to the non-extracted region. The image processing apparatus according to any one of claims 1 to 9. The image processing apparatus according to any one of claims 1 to 10, wherein the first feature region extraction unit extracts a lesion as the first feature region from the virtual endoscopic image. The first feature region extraction unit uses the correspondence between the information of the second feature region stored in the storage unit and the non-extracted region to obtain the first feature region from the virtual endoscopic image. The image processing apparatus according to any one of claims 1 to 11, wherein the extraction rule to be extracted is updated. The first feature region extraction unit applies an extraction rule generated by using machine learning to extract the first feature region from the virtual endoscopic image according to any one of claims 1 to 12. The image processing apparatus described. The first image input step of inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject, and
A second image input step of inputting an actual endoscopic image obtained by imaging an observation target of a subject using an endoscope, and
A matching step of associating a virtual endoscopic image generated from the virtual endoscopic image with the real endoscopic image,
A first feature region extraction step of extracting a first feature region that matches the first condition from the virtual endoscopic image, and a first feature region extraction step.
A second feature region extraction step of extracting a second feature region that meets the second condition corresponding to the first condition from the actual endoscopic image, and a second feature region extraction step.
Information on a non-extracted region that is associated with the second feature region of the real endoscopic image and is not extracted as the first feature region from the virtual endoscopic image, and is associated with the non-extracted region. A storage step of storing at least one of the information in the second characteristic region, and
Image processing method including. On the computer
First image input function for inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject,
A second image input function that inputs an actual endoscopic image obtained by imaging the observation target of the subject using an endoscope,
An associating function for associating a virtual endoscopic image generated from the virtual endoscopic image with the real endoscopic image.
A first feature area extraction function that extracts a first feature area that matches the first condition from the virtual endoscopic image.
Corresponds to the second feature region extraction function that extracts the second feature region that matches the second condition corresponding to the first condition from the real endoscopic image, and the second feature region of the real endoscopic image. At least one of the information of the non-extracted region attached and not extracted as the first feature region from the virtual endoscopic image and the information of the second feature region associated with the non-extracted region A program that realizes the save function. With an endoscope
A first image input unit for inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject, and
A second image input unit for inputting an actual endoscopic image obtained by imaging an observation target of a subject using the endoscope, and
An association unit that associates the virtual endoscope image generated from the virtual endoscope image with the real endoscope image, and
A first feature region extraction unit that extracts a first feature region that matches the first condition from the virtual endoscopic image, and a first feature region extraction unit.
A second feature region extraction unit that extracts a second feature region that meets the second condition corresponding to the first condition from the actual endoscopic image, and a second feature region extraction unit.
Information on a non-extracted region that is associated with the second feature region of the real endoscopic image and is not extracted as the first feature region from the virtual endoscopic image, and is associated with the non-extracted region. A storage unit that stores at least one of the information in the second feature area, and
Endoscope system equipped with. An endoscope system equipped with an endoscope, an image processing device, and a storage device.
The image processing device is
A first image input unit for inputting a virtual endoscopic image generated from a three-dimensional examination image of a subject, and
A second image input unit for inputting an actual endoscopic image obtained by imaging an observation target of a subject using the endoscope, and
An association unit that associates the virtual endoscope image generated from the virtual endoscope image with the real endoscope image, and
A first feature region extraction unit that extracts a first feature region that matches the first condition from the virtual endoscopic image, and a first feature region extraction unit.
A second feature region extraction unit that extracts a second feature region that meets the second condition corresponding to the first condition from the actual endoscopic image, and a second feature region extraction unit.
With
The storage device is
Information on a non-extracted region that is associated with the second feature region of the real endoscopic image and is not extracted as the first feature region from the virtual endoscopic image, and is associated with the non-extracted region. An endoscope system including a storage unit that stores at least one of the information in the second feature region. The endoscope system according to claim 17, wherein the storage device is communicably connected to the image processing device via a network.

Images