JP2021065293A - Image processing method, image processing device, image processing program, teacher data generation method, teacher data generation device, teacher data generation program, learned model generation method, learned model generation device, diagnosis support method, diagnosis support device, diagnosis support program, and recording medium that records the program - Google Patents

Abstract

To facilitate observation by a medical image.SOLUTION: A control device 3 includes: a white light acquisition unit 63 for acquiring a white light image, which is an observation object of a living body; a fluorescent image acquisition unit 64 for acquiring a fluorescent image of the observation object; a lesion information generation unit 65 for generating lesion information on a lesion in the observation object on the basis of the fluorescent image; a lesion information adding unit 66 for adding the lesion information to the white light image; and a display unit 67 for displaying the white light image to which the lesion information is added, in a monitor 4.SELECTED DRAWING: Figure 1

Description

The present invention relates to a diagnosis using a medical image, and more particularly to a diagnosis using an endoscope.

In recent years, the H. pylori infection rate, which is one of the causes of gastric cancer, has decreased, but the number of gastric cancer cases has not decreased due to the super-aging society. With the widespread use of endoscopic submucosal dissection, the importance of endoscopic diagnosis of early gastric cancer is increasing, but it may be difficult to diagnose lesion deterioration or presence. In addition, multiple cancers are often overlooked, and it is desired to develop an objective diagnostic method for picking up lesions.

As such a diagnostic method, a photodynamic diagnostic method (PDD: Photodynamic Diagonosis) can be mentioned. PDD is a diagnostic method that combines a photosensitizer with tumor affinity and excitation light. When a specific photosensitizer such as 5-aminolevulinic acid (5-ALA) is administered to the human body, the photosensitizer accumulates in tumor tissue and new blood vessels. The presence or absence of a lesion can be diagnosed by irradiating a substance accumulated in a tissue with excitation light and detecting fluorescence generated from the excited substance.

In the fluorescent image in PDD, the boundary contrast between the lesion site and the non-lesion site is improved as compared with the normal white light image. However, due to its characteristics, fluorescence images are not suitable for detailed observation of non-lesioned areas.

On the other hand, Patent Document 1 discloses that a fluorescence image and a white light image are simultaneously acquired and displayed in parallel on a TV monitor. With these two images, both lesions and non-lesions can be observed.

Japanese Unexamined Patent Publication No. 2006-94907

However, in Patent Document 1, since it is necessary to constantly compare the fluorescent image and the white light image alternately, the burden on the doctor is heavy and it is difficult to shorten the time required for the medical examination.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a technique for facilitating observation with a medical image.

In order to solve the above problems, the present invention includes the following aspects.
Item 1.
The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A lesion information addition step for adding the lesion information to the medical image,
A display step of displaying the medical image to which the lesion information is added on a monitor, and
Image processing method with.
Item 2.
Item 2. The image processing method according to Item 1, wherein the lesion information is position information indicating the position of the lesion.
Item 3.
The medical image is a white light image taken when the observation target is irradiated with white light from an endoscope.
Item 1 or 2, wherein the medical information is a fluorescence image taken when the observation target is irradiated with excitation light for exciting a photosensitive substance from the endoscope for photodynamic diagnosis. Image processing method.
Item 4.
The first acquisition unit that acquires the medical image of the observation target of the living body,
A second acquisition unit that acquires medical information other than the medical image of the observation target, and
A lesion information generation unit that generates lesion information related to a lesion in the observation target based on the medical information.
A lesion information addition part that adds the lesion information to the medical image,
A display unit that displays the medical image to which the lesion information is added on a monitor, and
Image processing device equipped with.
Item 5.
The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A lesion information addition step for adding the lesion information to the medical image,
A display step of displaying the medical image to which the lesion information is added on a monitor, and
An image processing program that allows a computer to execute.
Item 6.
A computer-readable non-temporary recording medium on which the image processing program according to Item 5 is recorded.
Item 7.
The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A teacher data generation step of adding the lesion information to the medical image to generate teacher data for machine learning, and
Teacher data generation method with.
Item 8.
The medical image is a white light image taken when the observation target is irradiated with white light from an endoscope.
Item 7. The teacher data according to Item 7, wherein the medical information is a fluorescence image taken when the observation target is irradiated with excitation light for exciting a photosensitive substance from the endoscope for photodynamic diagnosis. Generation method.
Item 9.
The first acquisition unit that acquires the medical image of the observation target of the living body,
A second acquisition unit that acquires medical information other than the medical image of the observation target, and
A lesion information generation unit that generates lesion information related to a lesion in the observation target based on the medical information.
A teacher data generation unit that adds the lesion information to the medical image to generate teacher data for machine learning, and a teacher data generation unit.
Teacher data generator equipped with.
Item 10.
The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A teacher data generation step of adding the lesion information to the medical image to generate teacher data for machine learning, and
A teacher data generator that lets a computer run.
Item 11.
A computer-readable non-temporary recording medium on which the teacher data generation program according to Item 10 is recorded.
Item 12.
Machine learning is performed using the teacher data generated by the teacher data generation method according to item 7 or 8.
A trained model generation method for generating a trained model in which the medical image of an unknown observation target is input and lesion information related to a lesion in the unknown observation target is output.
Item 13.
Machine learning is performed using the teacher data generated by the teacher data generation method according to item 7 or 8.
A learning unit that generates a trained model that inputs the medical image of an unknown observation target and outputs lesion information related to a lesion in the unknown observation target.
Trained model generator with.
Item 14.
The third acquisition step of acquiring the medical image of the subject to be observed, and
The medical image acquired in the third acquisition step is input to the trained model generated by the trained model generation method according to Item 12, and lesion information related to the lesion in the observation target of the subject is acquired. Lesion information acquisition step and
Diagnosis support method equipped with.
Item 15.
The third acquisition department that acquires the medical image of the subject to be observed, and
The medical image acquired by the third acquisition unit is input to the trained model generated by the trained model generator according to Item 13 to acquire lesion information regarding the lesion in the observation target of the subject. Lesion information acquisition department and
Diagnostic support device equipped with.
Item 16.
The third acquisition step of acquiring the medical image of the subject to be observed, and
The medical image acquired in the third acquisition step is input to the trained model generated by the trained model generation method according to Item 12, and lesion information related to the lesion in the observation target of the subject is acquired. Lesion information acquisition step and
A diagnostic support program that lets a computer execute.
Item 17.
A computer-readable non-temporary recording medium on which the diagnostic support program according to item 16 is recorded.

According to the present invention, observation by a medical image can be facilitated.

It is a block diagram which shows the structure of the endoscope system which concerns on one Embodiment of this invention. This is an example of a white light image. This is an example of a fluorescence image to be observed, which is the same as the white light image shown in FIG. It is an image in which a frame surrounding a region identified as a lesion is marked with respect to the fluorescence image shown in FIG. This is an example of a white light image to which lesion information is added. (A) is an example of a fluorescence image, and (b) is an image in which a region identified as a lesion is marked on the fluorescence image shown in FIG. 6 (a). This is an example of a white light image to which lesion information is added. This is an example of a white light image and a fluorescence image displayed in parallel. It is a flowchart which shows the processing procedure of an image processing method. It is a flowchart which shows the processing procedure of the teacher data generation method. It is a block diagram which shows the structure of the trained model generator which concerns on one Embodiment of this invention. It is a flowchart which shows the processing procedure of the trained model generation method. It is a block diagram which shows the structure of the endoscope system which concerns on one Embodiment of this invention. It is a flowchart which shows the processing procedure of the diagnosis support method.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The present invention is not limited to the following embodiments.

(System configuration)
FIG. 1 is a block diagram showing a configuration of an endoscope system 1 according to an embodiment of the present invention. The endoscope system 1 includes an endoscope 2, a control device 3, and a monitor 4.

The endoscope 2 is a gastrointestinal endoscope used for observing the upper gastrointestinal tract by photodynamic diagnosis (PDD). Inserted into the upper digestive tract. The observation target is not limited to the upper gastrointestinal tract.

The endoscope 2 includes optical fibers F1 and F2 inside. A phosphor 21, a light deflection diffusing member 22, and an image pickup device 23 are arranged at the tip of the endoscope 2, and two exit windows and one observation window are provided on the tip surface of the endoscope 2. Has been done. The phosphor 21 faces one of the exit windows, the light deflection diffuser 22 faces the other of the exit windows, and the image sensor 23 faces the observation window.

As will be described later, blue light (center emission wavelength 445 nm) is emitted from the tip of the optical fiber F1, and purple excitation light (center emission wavelength 410 nm) for PDD is emitted from the tip of the optical fiber F2. .. The blue light from the optical fiber F1 is converted into white light by the phosphor 21 and emitted from one of the exit windows. The purple excitation light from the optical fiber F2 is emitted from the other side of the exit window via the light deflection diffusion member 22. Further, the incident light to the observation window is imaged on the light receiving surface of the image pickup device 23 by a lens (not shown). The image sensor 23 photoelectrically converts the incident light to generate an analog image signal.

The structure of the endoscope 2 is not limited to this. For example, an endoscope that does not have an optical fiber and has a light source attached to the tip thereof, such as a capsule endoscope described later, may be used.

The control device 3 is connected to the endoscope 2 and mainly controls light emission by the endoscope 2 and processes an image signal from the image sensor 23. The control device 3 includes a light source device 5 and a processor 6. The control device 3 corresponds to the image processing device and the teacher data generation device described in the claims.

The light source device 5 includes a light source 51 that generates blue light, a light source 52 that generates purple excitation light, and a light source driving unit 53 that drives the light sources 51 and 52. In the present embodiment, the light sources 51 and 52 are LED light sources, but a fluorescent light source such as a laser light source or a xenon lamp can also be used.

The processor 6 includes a timing controller 61, an AD conversion unit 62, a white light image acquisition unit (first acquisition unit) 63, a fluorescence image acquisition unit (second acquisition unit) 64, a lesion information generation unit 65, and a lesion. It includes an information addition unit 66, a display unit 67, and a storage unit 68.

The timing controller 61 outputs a signal for controlling the timing at which the light source driving unit 53 drives the light sources 51 and 52. Specifically, the timing controller 61 controls the light source driving unit 53 so that the light sources 51 and 52 are alternately driven at a predetermined cycle (for example, 1/60 ms). Further, the timing controller 61 outputs a signal for controlling the timing at which the white light image acquisition unit 63 and the fluorescence image acquisition unit 64 acquire the white light image and the fluorescence image, which will be described later, respectively.

The AD conversion unit 62 AD-converts the image signal from the image sensor 23 and outputs the digital image signal to the white light image acquisition unit 63 and the fluorescence image acquisition unit 64.

The white light image acquisition unit 63 has a function of acquiring a white light image captured when the observation target is irradiated with white light from the endoscope 2. Specifically, the timing controller 61 outputs a signal permitting image acquisition to the white light image acquisition unit 63 at the timing when the light source 51 is being driven. In response to this, the white light image acquisition unit 63 captures the image signal output from the AD conversion unit 62. FIG. 2 is an example of a white light image.

The fluorescence image acquisition unit 64 has a function of acquiring a fluorescence image (PDD image) captured when the observation target is irradiated with purple excitation light for exciting a photosensitive substance from the endoscope 2. Specifically, the timing controller 61 outputs a signal permitting image acquisition to the fluorescence image acquisition unit 64 at the timing when the light source 52 is being driven. In response to this, the fluorescence image acquisition unit 64 captures the image signal output from the AD conversion unit 62. FIG. 3 is an example of a fluorescence image to be observed, which is the same as the white light image shown in FIG.

The lesion information generation unit 65 has a function of generating lesion information regarding a lesion in an observation target based on a fluorescence image. When a lesion such as a tumor cell is present in the observation target, a photosensitive substance is accumulated in the tumor cell, so that fluorescence (wavelength 635 nm) is emitted from the lesion portion when irradiated with purple excitation light. As a result, in the fluorescent image, the contrast between the lesioned portion and the non-lesioned portion (in the present embodiment, which means the site where the tumor does not exist) becomes large. The lesion information generation unit 65 identifies a site where fluorescence is detected as a lesion area, and generates position information indicating the position of the lesion as lesion information.

In addition, the lesion information generation unit 65 may use an algorithm generated by performing machine learning using the fluorescence image and the definitely diagnosed lesion information as teacher data in order to generate the lesion information from the fluorescence image.

FIG. 4 is an image in which a frame H1 surrounding a region identified as a lesion is marked on the fluorescence image shown in FIG. The position (coordinates) of the area surrounded by the frame H1 corresponds to the lesion information. The lesion information may be generated automatically by analyzing the fluorescence image, or manually by the user (doctor) of the endoscope system 1 by operating an input device such as a mouse or a touch panel. May be good.

The lesion information addition unit 66 has a function of adding (annotating) lesion information to a white light image. In the present embodiment, the lesion information addition unit 66 adds the lesion information (position of the lesion portion) generated by the lesion information generation unit 65 based on the fluorescence image to the white light image acquired substantially at the same time as the fluorescence image. , The region of the corresponding coordinates in the white light image is marked so as to be distinguishable from other regions.

FIG. 5 shows an image in which lesion information is added to the white light image shown in FIG. In FIG. 5, the frame H2 is overlaid on the area having the same coordinates as the area surrounded by the frame H1 shown in FIG.

The mode in which the lesion information is added to the white light image is not particularly limited. Other examples of fluorescent and white light images are shown in FIGS. 6 and 7. FIG. 6A is an example of a fluorescence image acquired by the fluorescence image acquisition unit 64, and FIG. 6B shows a region identified as a lesion in the fluorescence image shown in FIG. 6A. It is an image in which the frame H3 is marked.

The lesion information generation unit 66 generates the position information of the region identified as the lesion portion in the fluorescence image as the lesion information, and the lesion information addition unit 66 adds the lesion information to the white light image. FIG. 7 shows an example of a white light image to which lesion information is added, and a focus frame H4 surrounding a region corresponding to the marked region in FIG. 6B is overlaid.

The display unit 67 has a function of displaying a white light image to which lesion information is added on the monitor 4. As a result, the monitor 4 displays a white light image in which the lesion portion is identifiable as shown in FIGS. 5 and 7, so that the user can recognize the lesion portion and the non-lesion portion at the same time. Can be done. Therefore, as compared with the conventional technique of simply displaying the white light image and the fluorescence image in parallel, the observation by the white light image becomes easier and the time required for the examination can be shortened.

As shown in FIG. 8, the display unit 67 may display a fluorescence image on the monitor 4 together with the white light image to which the lesion information is added.

The storage unit 68 stores various programs used for arithmetic processing by the processor 6 and various data generated by the arithmetic processing, and is composed of, for example, a flash memory, an HDD, an SSD, and the like.

In the teacher data generation mode described later, the lesion information addition unit 66 stores the white light image to which the lesion information is added in the storage unit 68 as teacher data D1 for machine learning. In this case, the lesion information addition unit 66 corresponds to the teacher data generation unit described in the claims. The storage process may be performed according to an operation by the user, or may be performed at a predetermined cycle.

(Medical examination mode)
The endoscope system 1 shown in FIG. 1 can operate in two modes, a medical examination mode for examining a subject and a teacher data generation mode for generating teacher data. FIG. 9 is a flowchart showing a processing procedure of an image processing method implemented by the endoscope system 1 in the examination mode. In this case, the control device 3 functions as the image processing device described in the claims.

In step S1, the subject is asked to take a photosensitizer (for example, 5-ALA), and after a lapse of a predetermined time, the endoscope 2 is inserted into the upper gastrointestinal tract. The type of the photosensitive substance is not particularly limited, and examples thereof include L-glucosamine, photofurin, and rezaphyrin in addition to 5-ALA.

After that, white light and purple excitation light are alternately emitted from the endoscope 2. In step S2 (first acquisition step), when the observation target is irradiated with white light, the white light image acquisition unit 63 acquires a white light image. Subsequently, when the irradiation light from the endoscope 2 is switched to the purple excitation light, the fluorescence image acquisition unit 64 acquires a fluorescence image in step S3 (second acquisition step).

After executing step S3, step S2 may be executed.

Subsequently, in step S4 (lesion information generation step), the lesion information generation unit 65 generates lesion information regarding the lesion in the observation target based on the fluorescence image.

Subsequently, in step S5 (lesion information addition step), the lesion information addition unit 66 adds lesion information to the white light image.

Subsequently, in step S6 (display step), the display unit 67 displays a white light image to which the lesion information is added on the monitor 4.

Steps S3 to S6 are repeatedly executed each time the white light and the purple excitation light are irradiated until the examination is completed (YES in step S7). For example, when the cycle of switching between the white light and the purple excitation light is 1/60 ms, steps S3 to S6 are executed in a cycle of 1/30 ms. As a result, the white light image to which the lesion information is added is displayed on the monitor 4 as a moving image.

The lesion information is not limited to the position information indicating the position of the lesion, and may be, for example, information indicating only the presence or absence of the lesion. In that case, in step S6, when displaying the white light image in which the lesion is present, an alarm indicating the presence of the lesion may be displayed on the white light image, or a warning indicating the presence of the lesion may be notified by voice.

(Teacher data generation mode)
FIG. 10 is a flowchart showing a processing procedure of the teacher data generation method implemented by the endoscope system 1 in the teacher data generation mode. In this case, the control device 3 functions as the teacher data generation device described in the claims.

Steps S1 to S6 shown in FIG. 10 are the same as steps S1 to S6 shown in FIG. The user can give an instruction to save the white light image at an arbitrary timing (for example, the timing of moving to a different observation site) while viewing the white light image to which the lesion information is added.

When the instruction to save the white light image is given (YES in step S8), in step S9, the lesion information addition unit 66 stores the white light image to which the lesion information is added as teacher data D1 for machine learning. Store at 68.

Steps S2 to S9 are repeated until the examination of one subject is completed (YES in step S10). As a result, a set of teacher data D1 (learning data set) is accumulated in the storage unit 68. In the present embodiment, the storage unit 68 for storing the teacher data D1 is built in the control device 3, but the storage unit 68 is not limited to this, and may be an external storage or a storage on the cloud. There may be.

If the amount of accumulated teacher data D1 is not sufficient at the end of the examination (NO in step S11), steps S1 to S10 are executed again for other subjects. Steps S1 to S10 are repeated until the teacher data D1 is sufficiently accumulated (YES in step S11). In the teacher data creation mode, the examination of the subject was also performed, but it is not always necessary to carry out the examination in the clinical setting.

As described above, in the teacher data creation mode, the lesion information generation unit 65 generates lesion information based on the fluorescent image (step S4), and the lesion information addition unit 66 adds the lesion information to the white light image. Teacher data D1 is created. As will be described later, by performing machine learning using the teacher data D1, it is possible to generate a trained model for predicting the presence or absence of a lesion from an unknown white light image.

Conventionally, in order to create teacher data similar to that of the present embodiment, a human has identified a lesion in a white light image based on a definitive diagnosis result and marked the identified lesion. Since it is necessary to perform this work for each of the white light images, the burden of the teacher data generation work is heavy, and it takes a lot of time to accumulate the teacher data necessary for machine learning.

On the other hand, in the present embodiment, since the lesion portion in the white light image is specified by using the lesion information generated from the fluorescence image, the generation of teacher data can be automated. Therefore, a large amount of teacher data can be accumulated in a short time, and the difficulty of creating a learned model and a diagnostic support program, which will be described later, can be reduced.

The detection accuracy of lesions by PDD is extremely high, and in clinical cases in the Department of Functional Pathology, Tottori University School of Medicine, of the 26 lesions diagnosed as positive based on fluorescent images, 23 are malignant tumors and 3 are benign. It was a tumor. Therefore, the teacher data D1 has a quality close to that of the conventional teacher data generated based on the definitive diagnosis. Therefore, it is possible to increase the sensitivity specificity in the diagnostic support by AI, which will be described later.

Further, at least the white light image acquisition unit 63, the fluorescence image acquisition unit 64, the lesion information generation unit 65, the lesion information addition unit 66, and the display unit 67 of the processor 6 shown in FIG. 1 are realized by hardware by an integrated circuit or the like. However, a CPU or GPU (not shown) of the processor 6 reads an image processing program or a teacher data generation program stored in the storage unit 68 into a main storage device (memory) (not shown) and executes the program in terms of hardware. It can also be realized.

In this case, the program may be downloaded to the control device 3 via a communication network, or a computer-readable non-temporary recording medium such as an SD card or CD-ROM on which the program code of the program is recorded may be downloaded. The above program may be supplied to the control device 3 via the control device 3.

(Generation of trained model)
Subsequently, machine learning using the teacher data D1 will be described.

FIG. 11 is a block diagram showing a configuration of the trained model generation device 7 according to the present embodiment. The trained model generator 7 can be configured by, for example, a general-purpose computer, and includes a storage unit 71 and a learning unit 72.

The storage unit 71 stores various programs used for arithmetic processing by the trained model generation device 7, and various data generated by the arithmetic processing, and is composed of, for example, an HDD and an SSD. The storage unit 71 stores the teacher data D1 generated by the processor 6 described above (that is, generated by the teacher data generation method according to the present embodiment).

The learning unit 72 is a functional block realized by the GPU or CPU of the learned model generation device 7 executing a predetermined learning program, and performs machine learning using the teacher data D1. As a result, the learning unit 72 generates a trained model D2 that inputs a white light image of an unknown observation target and outputs lesion information related to a lesion in the unknown observation target. The learning method is not particularly limited, but for example, deep learning, a support vector machine, a random forest, or the like can be used.

FIG. 12 is a flowchart showing a processing procedure of the trained model generation method implemented by the trained model generation device 7.

In step S20, the teacher data D1 generated by the processor 6 is transferred to the trained model generation device 7 and stored in the storage unit 71.

In step S21, the learning unit 72 executes machine learning by inputting the teacher data D1 into a learning model such as a neural network. When the machine learning is completed, the trained model D2 is created (step S22). The created trained model D2 is stored in the storage unit 71 (step S23).

(Diagnosis support by AI)
Next, diagnostic support using the trained model D2 will be described.

FIG. 13 is a block diagram showing the configuration of the endoscope system 11 according to the present embodiment. The endoscope system 11 includes an endoscope 12, a control device 13, and a monitor 14.

The endoscope 12 is used for observing the upper gastrointestinal tract with a white light image. That is, unlike the endoscope 2 shown in FIG. 1, the endoscope 12 does not have a function of capturing a fluorescent image. The observation target is not limited to the upper gastrointestinal tract.

The endoscope 12 includes an optical fiber F3 inside. A phosphor 121 and an image pickup device 122 are arranged at the tip of the endoscope 12, and an exit window and an observation window are provided on the tip surface of the endoscope 2. The phosphor 121 faces the exit window, and the image sensor 122 faces the observation window.

Blue light (center emission wavelength 445 nm) is emitted from the tip of the optical fiber F3. The blue light from the optical fiber F3 is converted into white light by the phosphor 121 and emitted from the exit window. The incident light on the observation window is imaged on the light receiving surface of the image sensor 122 by a lens (not shown). The image sensor 122 photoelectrically converts the incident light to generate an analog image signal.

The control device 13 is connected to the endoscope 12, and mainly controls the light emission by the endoscope 12 and processes the image signal from the image sensor 122. The control device 13 includes a light source device 15 and a processor 16. The control device 13 corresponds to the diagnostic support device described in the claims.

The light source device 15 includes a light source 151 that generates blue light and a light source driving unit 152 that drives the light source 151. As the light source 151, the same light source as the light source 51 shown in FIG. 1 can be used.

The processor 16 includes a timing controller 161, an AD conversion unit 162, a white light image acquisition unit (third acquisition unit) 163, a storage unit 164, a lesion information acquisition unit 165, and a display unit 166. ..

The timing controller 161 outputs a signal for controlling the timing at which the light source driving unit 152 drives the light source 151, and outputs a signal for controlling the timing at which the white light image acquisition unit 163 acquires the white light image. ..

The AD conversion unit 162 AD-converts the image signal from the image sensor 122 and outputs the digital image signal to the white light image acquisition unit 163.

The white light image acquisition unit 163 has a function of acquiring a white light image captured when the observation target is irradiated with white light from the endoscope 12. Specifically, the timing controller 61 outputs a signal permitting image acquisition to the white light image acquisition unit 163 while the light source 151 is being driven. In response to this, the white light image acquisition unit 163 captures the image signal output from the AD conversion unit 162. The white light image acquisition unit 163 may be configured to constantly acquire a white light image regardless of whether or not the light source 151 is driven.

The storage unit 164 stores various programs used for arithmetic processing by the processor 16 and various data generated by the arithmetic processing, and is composed of, for example, a flash memory, an HDD, an SSD, and the like. The storage unit 164 stores in advance the trained model D2 generated by the trained model generation device 7 described above (that is, generated by the trained model generation method according to the present embodiment).

The lesion information acquisition unit 165 reads the learned model D2 into a main storage device (memory) (not shown), inputs the white light image acquired by the white light image acquisition unit 163 into the learned model D2, and inputs the learned model D2 to the subject. The lesion information regarding the lesion in the observation target is acquired from the trained model D2. In the present embodiment, the lesion information is the position information of the lesion portion in the white light image, but is not limited to this, and may be, for example, information indicating only the presence or absence of the lesion.

Further, in the present embodiment, the lesion information acquisition unit 165 adds the acquired lesion information to the white light image, and the white light image to which the lesion information is added (for example, a white light image in which a frame is superimposed on the lesion portion). To generate. The display unit 166 displays a white light image to which the lesion information is added on the monitor 14. This shows the user where the lesion is suspected.

As described above, in the endoscope system 11, the endoscope 12 does not have a function of capturing a fluorescent image, but the lesion information acquisition unit 165 inputs a white light image into the trained model D2. We are acquiring lesion information. That is, the lesion information acquisition unit 165 can obtain information equivalent to the lesion information obtained by PDD from the white light image. Therefore, the user can detect the lesion with the same accuracy as PDD only by the white light image.

FIG. 14 is a flowchart showing a processing procedure of the diagnostic support method implemented by the endoscope system 11.

In step S30, the endoscope 12 is inserted into the upper gastrointestinal tract of the subject. The photosensitizer is not taken orally before the insertion of the endoscope 12.

In step S31 (third acquisition step), the white light image acquisition unit 163 acquires a white light image captured when the observation target is irradiated with white light from the endoscope 12.

In step S32 (lesion information acquisition step), the lesion information acquisition unit 165 inputs the white light image acquired in step S30 into the learned model D2 to acquire lesion information related to the lesion in the observation target of the subject. .. Further, the lesion information acquisition unit 165 adds the acquired lesion information to the white light image to generate a white light image to which the lesion information is added (step S33).

In step S34, the display unit 166 displays a white light image to which the lesion information is added on the monitor 14.

When the lesion information is information indicating only the presence or absence of a lesion, the user may be warned that the lesion may exist by, for example, a warning display or voice instead of steps S33 and S34. ..

At least the white light image acquisition unit 163, the lesion information acquisition unit 165, and the display unit 166 of the processor 16 shown in FIG. 13 may be realized by hardware by an integrated circuit or the like, but a CPU (not shown) of the processor 16 or It can also be realized by the GPU by reading the diagnostic support program stored in the storage unit 164 into a main storage device (memory) (not shown) and executing it.

In this case, the program may be downloaded to the control device 13 via a communication network, or a computer-readable non-temporary recording medium such as an SD card or CD-ROM on which the program code of the program is recorded may be downloaded. The program may be supplied to the control device 13 via the control device 13.

(Additional notes)
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention.

For example, in the above embodiment, the endoscope 2 is a gastrointestinal endoscope which is a kind of flexible endoscope, but the present invention is not limited to this. Table 1 shows examples of endoscopes applicable to the present invention.

The endoscope includes those used for NOTES (natural orifice transluminal endoscopic surgery). NOTES is a procedure in which a flexible endoscope is inserted through a natural hole such as the mouth, anus, or vagina, and reaches the abdominal cavity through a luminal wall such as the stomach wall for diagnosis and treatment.

Further, in the above embodiment, the medical information used for generating the lesion information is a fluorescent image (PDD image), and the medical image to which the generated lesion information is added is a white light image. And the combination of medical information is not limited to this. For example, the medical image to which the lesion information is added may be another endoscopic image such as an image-enhanced endoscopic image. Table 2 shows an example of a combination of medical images and medical information applicable in the present invention.

Image-enhanced endoscopic images include NBI (Narrow Band Imaging), BLI (Blue LASER Imaging), LCI (Linked Color Imaging), AFI (Autofluorescence Imaging) and the like. For example, as the medical image and medical information, a combination of a white light image and NBI, a fluorescence image and NBI can be considered.

In addition, the dye endoscopy image is an image obtained by the dye endoscopy method in which various pigment agents are sprayed during endoscopy and the reaction is observed, and the recognition of the lesion and the determination of the lesion range are determined. -Useful for evaluating depth of invasion. Table 3 shows examples of dye endoscopy methods, dyes, and principles.

Further, in the above embodiment, the medical information for generating the lesion information to be added to the medical image is a single piece of information, but may be a plurality of pieces of information. That is, lesion information may be generated based on a plurality of pieces of information, and the generated lesion information may be added to the medical image. For example, a fluorescence image and NBI may be used as medical information, and lesion information generated from these may be added to the white light image. In this case, the pathological tissue may be different between the site that is positive in the fluorescence image and dark in NBI and the site that is positive in the fluorescence image and is light in NBI. In this way, by using a plurality of medical information, more detailed lesion information can be obtained, and the quality of teacher data can be improved.

In addition, the complex teacher data acquired at the same time using a plurality of medical information as described above not only indicates the boundary region between the lesion site and the non-lesion site, but also learns to identify the site that may become a lesion in the future. It can be used to create a model. In addition, the learning model created in this way can output an approximate value as a prediction of the possibility that a certain region becomes a lesion site at a specific future time point. Furthermore, the lesion information obtained by the endoscopic system of the present invention can be used as auxiliary information during treatment. For example, the boundary information of the lesion can be used as information for designating the position of the treatment target in an automatic treatment device (automatic endoscope or the like).

The present invention can be applied to diagnosis using medical images not only in gastroenterology but also in all clinical departments such as surgery, neurosurgery, urology, otolaryngology, and respiratory medicine.

1 Endoscope system 2 Endoscope 3 Control device 4 Monitor 5 Light source device 6 Processor 7 Learned model generator 11 Endoscope system 12 Endoscope 13 Control device 14 Monitor 15 Light source device 16 Processor 21 Phosphor 22 Light deflection Diffusing member 23 Image pickup element 51 Light source 52 Light source 53 Light source drive unit 61 Timing controller 62 AD conversion unit 63 White light image acquisition unit 64 Fluorescent image acquisition unit 65 Disease information generation unit 66 Disease information addition unit 67 Display unit 68 Storage unit 71 Storage unit 72 Learning unit 121 Phosphorus 122 Imaging element 151 Light source 152 Light source driving unit 161 Timing controller 162 AD conversion unit 163 White light image acquisition unit 164 Storage unit 165 Disease information acquisition unit 166 Display unit D1 Teacher data D2 Learned model F1 Optical fiber F2 Optical fiber F3 Optical fiber

Claims (17)

The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A lesion information addition step for adding the lesion information to the medical image,
A display step of displaying the medical image to which the lesion information is added on a monitor, and
Image processing method with. The image processing method according to claim 1, wherein the lesion information is position information indicating the position of the lesion. The medical image is a white light image taken when the observation target is irradiated with white light from an endoscope.
The medical information is a fluorescent image taken when the observation target is irradiated with excitation light for exciting a photosensitive substance for photodynamic diagnosis, according to claim 1 or 2. The image processing method described. The first acquisition unit that acquires the medical image of the observation target of the living body,
A second acquisition unit that acquires medical information other than the medical image of the observation target, and
A lesion information generation unit that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation unit.
A lesion information addition part that adds the lesion information to the medical image,
A display unit that displays the medical image to which the lesion information is added on a monitor, and
Image processing device equipped with. The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A lesion information addition step for adding the lesion information to the medical image,
A display step of displaying the medical image to which the lesion information is added on a monitor, and
An image processing program that allows a computer to execute. A computer-readable non-temporary recording medium on which the image processing program according to claim 5 is recorded. The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A teacher data generation step of adding the lesion information to the medical image to generate teacher data for machine learning, and
Teacher data generation method with. The medical image is a white light image taken when the observation target is irradiated with white light from an endoscope.
The teacher according to claim 7, wherein the medical information is a fluorescence image taken when the observation target is irradiated with excitation light for exciting a photosensitive substance from the endoscope for photodynamic diagnosis. Data generation method. The first acquisition unit that acquires the medical image of the observation target of the living body,
A second acquisition unit that acquires medical information other than the medical image of the observation target, and
A lesion information generation unit that generates lesion information related to a lesion in the observation target based on the medical information.
A teacher data generation unit that adds the lesion information to the medical image to generate teacher data for machine learning, and a teacher data generation unit.
Teacher data generator equipped with. The first acquisition step of acquiring a medical image of a living body to be observed, and
The second acquisition step of acquiring medical information other than the medical image of the observation target, and
A lesion information generation step that generates lesion information related to a lesion in the observation target based on the medical information, and a lesion information generation step.
A teacher data generation step of adding the lesion information to the medical image to generate teacher data for machine learning, and
A teacher data generator that lets a computer run. A computer-readable non-temporary recording medium on which the teacher data generation program according to claim 10 is recorded. Machine learning is performed using the teacher data generated by the teacher data generation method according to claim 7 or 8.
A trained model generation method for generating a trained model in which the medical image of an unknown observation target is input and lesion information related to a lesion in the unknown observation target is output. Machine learning is performed using the teacher data generated by the teacher data generation method according to claim 7 or 8.
A learning unit that generates a trained model that inputs the medical image of an unknown observation target and outputs lesion information related to a lesion in the unknown observation target.
Trained model generator with. The third acquisition step of acquiring the medical image of the subject to be observed, and
The medical image acquired in the third acquisition step is input to the trained model generated by the trained model generation method according to claim 12, and lesion information related to the lesion in the observation target of the subject is acquired. Lesion information acquisition step and
Diagnosis support method equipped with. The third acquisition department that acquires the medical image of the subject to be observed, and
The medical image acquired by the third acquisition unit is input to the trained model generated by the trained model generator according to claim 13 to acquire lesion information related to the lesion in the observation target of the subject. Lesion information acquisition department and
Diagnostic support device equipped with. The third acquisition step of acquiring the medical image of the subject to be observed, and
The medical image acquired in the third acquisition step is input to the trained model generated by the trained model generation method according to claim 12, and lesion information related to the lesion in the observation target of the subject is acquired. Lesion information acquisition step and
A diagnostic support program that lets a computer execute. A computer-readable non-temporary recording medium on which the diagnostic support program according to claim 16 is recorded.

Images