WO2020017211A1 - Medical image learning device, medical image learning method, and program - Google Patents

Abstract

Provided are a medical image learning device, a medical image learning method, and a program by which learning for handling a plurality of images having different spectral distributions can be performed without requiring many correct images to be collected for each of the images having different spectral distributions. The medical image learning device is provided with: image acquisition units (30, 40) which acquire a first image and a second image having different spectral distributions; an image processing unit (40) which generates a third image by performing image processing on the first image; and a learning unit (30) that is caused to learn a recognizer to be adopted for automatic recognition by using a first image group, a second image group, a third image group, and correct data for the respective images. The image processing unit generates the third image from the first image by suppressing, in the first image, signals in a bandwidth having different characteristics from those in the second image and/or by enhancing, in the first image, signals in a bandwidth having characteristics identical or similar to those in the second image.

Description

Medical image learning device, medical image learning method, and program

The present invention relates to a medical image learning device, a medical image learning method, and a program, and particularly to learning of a special light observation image.

In the medical field, AI technology applying deep learning called deep learning is expected. As an example of the AI technology, there is an automatic detection of a lesion and an automatic identification of a lesion. AI is an abbreviation for Artificial Intelligence.

Patent Literature 1 describes an electronic endoscope apparatus that captures a wavelength region of visible light and generates a normal image that is a color image representing the visible light region and a diagnostic image that is a narrow-band spectral image. . The document describes that a diagnostic image is generated by applying an arithmetic process to a normal image to obtain an image equivalent to a narrowband image obtained using a narrowband bandpass filter. .

Patent Document 2 describes a machine learning support device that supports machine learning of an artificial intelligence device and verification of the operation of the artificial intelligence device for multi-sensor information. The document describes that virtual data for learning is generated by randomly changing other waveform portions while retaining the characteristic spectrum portion of the spectral data. Further, the document describes that an artificial intelligence device is learned in advance using a virtual data set.

JP 2010-75368 A JP-A-2017-102755

However, developing a learning model requires a large amount of correct data sets. Here, the correct answer data set means an image and a set of image recognition results. In the medical field, collecting correct answer data sets is a barrier to the development of learning models. The observation using the endoscope apparatus includes not only observation using normal light but also observation using special light such as BLI (Blue LAZER Imaging). The development of an AI that can handle multiple modes also requires the collection of a large number of correct data sets.

白色 One example of the normal light here is white light. White light may include light in multiple wavelength bands. An example of the special light is light in a band narrower than the wavelength band of white light. The special light may include light of a plurality of wavelength bands.

Patent Document 1 has no description about learning. Further, although Patent Literature 1 describes the adjustment of luminance and the like, it does not describe switching of a process that focuses on a difference between band components in a spectral distribution.

Patent Document 2 describes a learning virtual data set generated by randomly changing other wavelength components while retaining the spectral components of the spectral data. On the other hand, Patent Literature 2 does not disclose that a band component in a corresponding band is used for learning in a plurality of image groups having different spectral distributions.

The present invention has been made in view of such circumstances, and can realize learning corresponding to a plurality of images having different spectral distributions without collecting a large amount of correct images for images having different spectral distributions. It is an object to provide an image learning device, a medical image learning method, and a program.

た め The following aspects of the invention are provided to achieve the above object.

The medical image learning device according to the first aspect includes an image acquisition unit that acquires a first image and a second image having different spectral distributions from each other, and an image processing unit that performs image processing on the first image to generate a third image. , A first image group including a plurality of first images, a second image group including a plurality of second images, and a third image group including a plurality of third images, and a first image group, a second image group, and A learning unit that learns a recognizer to be applied to the automatic recognition using the correct data of each of the third image group, and an image processing unit, wherein the second image among the bands included in the spectral distribution of the first image Image processing for suppressing a signal in a band having a characteristic different from that of the corresponding band, and enhancing a signal of a band having the same characteristic as the corresponding band of the second image or a signal of a similar band among the bands included in the first image. To perform at least one of the image processing A medical image learning device for generating a third image.

According to the first aspect, for the first image and the second image having different spectral distributions, among the bands included in the spectral distribution of the first image, the signal or the signal of the band having the same characteristic as the corresponding band of the second image is used. A third image is generated by enhancing a signal in a similar band or suppressing a signal in a band having different characteristics from a corresponding band of the second image. Automatic recognition using a first image group including a plurality of first images, a second image group including a plurality of second images, and a third image group including a plurality of third images, and correct data of each image group Train a classifier to apply to. Accordingly, learning for each image group having a different spectral distribution can be realized without collecting a large amount of correct images for each image group having a different spectral distribution.

画像 An example of an image having a different spectral distribution is an image to which a different observation mode is applied. As an example of the observation mode, there is a mode in which the illumination light is different.

According to a second aspect, in the medical image learning device according to the first aspect, the image processing unit may perform a process of converting a luminance variance on the first image.

According to the second aspect, the third image can be generated from the first image by applying the process of converting the luminance variance to the first image.

In a third aspect, in the medical image learning device according to the first aspect, the image processing unit may be configured to perform a process of changing the brightness of the first image to a specified brightness value.

According to the third aspect, it is possible to generate a third image from the first image by applying the process of changing the luminance to the specified luminance value to the first image.

According to a fourth aspect, in the medical image learning device according to the first aspect, the image processing unit may perform a process of changing a luminance to a random luminance value on the first image.

According to the fourth aspect, it is possible to generate a third image from the first image by applying the process of changing the luminance to a random luminance value to the first image.

A fifth aspect is the medical image learning device according to the first aspect, wherein the first image is a normal light image captured using normal light, and the second image is captured using special light having a narrower band than the normal light. It may be configured to be a special light image.

According to the fifth aspect, the first image can be generated by applying the normal light observation mode. In addition, the second image can be generated by applying the special light observation mode.

According to a sixth aspect, in the medical image learning device according to the fifth aspect, the image processing unit may perform a process of suppressing a B channel component on the normal light image.

According to the sixth aspect, the third image can be generated by suppressing the B channel component of the normal light image.

According to a seventh aspect, in the medical image learning device according to the fifth aspect, the image processing unit may perform a process of enhancing at least one of the R channel component and the G channel component on the normal light image.

According to the seventh aspect, at least one of the R channel component and the G channel component of the normal light image can be enhanced to generate the third image.

In an eighth aspect, in the medical image learning device according to the fifth aspect, the image processing unit may be configured to perform processing for suppressing the R channel component on the normal light image.

According to the eighth aspect, the third image can be generated by suppressing the R channel component of the normal light image.

特殊 As an example of the special light image in the eighth mode, there is an image in which the R channel component of the normal light image is missing.

According to a ninth aspect, in the medical image learning apparatus according to the fifth aspect, the image processing unit may perform a process of enhancing a B-channel component and a G-channel component on the normal light image.

According to the ninth aspect, the third image can be generated by enhancing the B channel component and the G channel component of the normal light image.

特殊 As an example of the special light image in the ninth aspect, there is an image in which the R channel component of the normal light image is missing.

A tenth aspect is the medical image learning device according to the first aspect, wherein the image processing unit is configured to suppress a specified frequency component of a spatial frequency in a processing target band of the first image, or to perform spatial processing in a processing target band of the first image. A configuration may be adopted in which processing for emphasizing a prescribed frequency component of the frequency is performed.

According to the tenth aspect, of the spatial frequencies of the first image, frequency components different from those of the second image are suppressed, or frequency components that are the same as or similar to those of the second image are emphasized. Images can be generated.

An eleventh aspect is the medical image learning device according to the tenth aspect, wherein the first image is a normal light image captured using normal light, and the second image is captured using special light having a narrower band than the normal light. It is a special light image, and the image processing unit may be configured to perform processing for enhancing the high frequency component of the B channel component of the normal light image.

According to the eleventh aspect, the high frequency component among the spatial frequencies in the B channel component of the normal light image can be emphasized to generate the third image from the normal light image.

According to a twelfth aspect, in the medical image learning device according to any one of the first to eleventh aspects, the learning unit may switch a learning method between learning of the first image group and learning of the second image group. Good.

According to the twelfth aspect, learning accuracy can be improved.

A thirteenth aspect is a medical image learning device according to any one of the first to eleventh aspects, wherein the learning unit switches a learning method between learning of the second image group and learning of the third image group. It may be.

According to the thirteenth aspect, learning accuracy can be improved.

In a fourteenth aspect, in the medical image learning device according to the twelfth aspect or the thirteenth aspect, the learning unit may be configured to apply a learning method for suppressing or enhancing the effect of the third image on the learning.

According to the fourteenth aspect, the effect of the third image on learning can be suppressed or emphasized. Thereby, learning accuracy can be improved.

A fifteenth aspect is the medical image learning device according to the twelfth aspect or the thirteenth aspect, wherein the learning unit is configured to convert the first image group, the third image group, the correct data of the first image group, and the correct data of the third image group. The learning using the correct data of the second image group and the second image group is performed on the learned recognizer that has been used or the learning device that has learned using the correct data of the third image group and the third image group. It may be configured to be implemented.

According to the fifteenth aspect, the correctness of the second image group and the second image group, which are true data having higher reliability than the third image group, for at least the learned recognizer using the third image group. Re-learn using data. Thereby, the accuracy of learning can be improved, and the accuracy of the recognizer can be improved.

The medical image learning method according to the sixteenth aspect includes an image acquisition step of acquiring a first image and a second image having different spectral distributions from each other, and an image processing step of performing image processing on the first image to generate a third image. , A first image group including a plurality of first images, a second image group including a plurality of second images, and a third image group including a plurality of third images, and a first image group, a second image group, and Using the correct data of each of the third image group, a learning step of learning a recognizer to be applied to automatic recognition, and the image processing step includes the second image of the band included in the spectral distribution of the first image Image processing for suppressing a signal in a band having a characteristic different from that of the corresponding band, and enhancing a signal of a band having the same characteristic as the corresponding band of the second image or a signal of a similar band among the bands included in the first image. To perform at least one of the image processing A medical image learning method for generating a third image from the first image.

According to the sixteenth aspect, the same effect as in the first aspect can be obtained.

１６In the sixteenth aspect, matters similar to the matters specified in the second to fifteenth aspects can be appropriately combined. In this case, the component that performs the process or function specified in the medical image learning device can be grasped as the component of the medical image learning method that performs the corresponding process or function.

A program according to a seventeenth aspect provides a computer with an image acquisition function of acquiring a first image and a second image having different spectral distributions, an image processing function of performing image processing on a first image to generate a third image, and A first image group including a plurality of first images, a second image group including a plurality of second images, and a third image group including a plurality of third images, and a first image group, a second image group, and a second image group. A program for realizing a learning function of learning a recognizer to be applied to automatic recognition using the correct data of each of the three images, wherein the image processing function includes a first image among the bands included in the spectral distribution of the first image. Image processing for suppressing a signal in a band having a characteristic different from the corresponding band of the two images, and a signal of a band having the same characteristic as the corresponding band of the second image or a signal of a similar band among the bands included in the first image Image processing that emphasizes At least subjected to one, it is a program that generates a third image from the first image.

According to the seventeenth aspect, the same effect as in the first aspect can be obtained.

１７ In the seventeenth aspect, matters similar to the matters specified in the second to fifteenth aspects can be appropriately combined. In this case, the component that performs the process or function specified in the medical image learning device can be grasped as the component of the program that performs the corresponding process or function.

According to the present invention, for the first image and the second image having different spectral distributions, among the bands included in the spectral distribution of the first image, signals or similar signals having the same characteristic as the corresponding band of the second image are used. The third image is generated by emphasizing the signal of the band to be processed or suppressing the signal of the band having the characteristic different from the corresponding band of the second image. Automatic recognition using a first image group including a plurality of first images, a second image group including a plurality of second images, and a third image group including a plurality of third images, and correct data of each image group Learn the classifier applied to. Accordingly, learning for each image group having a different spectral distribution can be realized without collecting a large amount of correct images for each image group having a different spectral distribution.

FIG. 1 is a block diagram illustrating an example of a hardware configuration of the endoscope image learning device according to the embodiment. FIG. 2 is a functional block diagram of the endoscope image learning device according to the embodiment. FIG. 3 is a functional block diagram of the learning unit shown in FIG. FIG. 4 is a flowchart illustrating a procedure of the endoscopic image learning method according to the embodiment. FIG. 5 is an explanatory diagram of the spectral sensitivity of the image sensor. FIG. 6 is an explanatory diagram showing the relationship between the intensity distribution of the illumination light and the spectral distribution of the endoscope image. FIG. 7 is an explanatory diagram showing the relationship between the spectral distribution of the normal light image and the spectral distribution of the special light image in the NBI. FIG. 8 is an overall configuration diagram of the endoscope system. FIG. 9 is a functional block diagram of the endoscope system. FIG. 10 is a graph showing the light intensity distribution.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same components are denoted by the same reference characters, and redundant description will be omitted as appropriate.

[Endoscope image learning device]
The endoscopic image learning device according to the present embodiment extends the number of learning data by combining images acquired in different observation modes. By using the images acquired in different observation modes for learning, highly accurate learning can be realized even when the number of images acquired in a specific observation mode is not sufficient. This makes it possible to realize highly accurate AI technology. The endoscope image learning device according to the embodiment is an example of a medical image learning device.

In the present embodiment, an example of learning applied to an attention area automatic recognition system using LCI (Linked Color Imaging) will be described. LCI means a special light color enhancement function. In the following description, the term “learning” and the term “machine learning” will be treated as synonyms.

[Hardware configuration of endoscope image learning device]
FIG. 1 is a block diagram illustrating an example of a hardware configuration of the endoscope image learning device according to the embodiment. The endoscope image learning device 10 can apply a personal computer or a workstation.

The endoscope image learning device 10 includes a communication unit 12, a first storage device 14, a second storage device 16, a third storage device 17, an operation unit 18, a CPU 20, a RAM 22, a ROM 24, and a display unit 26. Note that CPU is an abbreviation of Central Processing Unit. RAM is an abbreviation for Random @ Access @ Memory. ROM is an abbreviation for Read \ Only \ Memory.

The communication unit 12 is an interface that processes communication with an external device. The communication unit 12 can apply a general-purpose communication interface. The communication unit 12 can apply either wired or wireless communication.

The first storage device 14 has a first data set including a normal light image group including a plurality of normal light images captured by applying the normal light observation mode and a correct data set indicating correct image recognition results of each normal light image. It is memorized.

The second storage device 16 stores a special light image group including a plurality of special light images by applying the special light observation mode and a second data set including correct data indicating a correct image recognition result of each special light image. .

The third storage device 17 is a third data set including a processed image group including a plurality of processed images obtained by performing image processing on the normal light image or the special light image, and correct data indicating a correct image recognition result of each processed image group. Is stored.

In this embodiment, an example will be described in which a processed image is generated from a normal light image and used for learning. Correct data representing a correct recognition result of the processed image may be extracted from the first data set. Note that a processed image may be generated from the special light image, and correct data of the processed image may be extracted from the second data set.

The first storage device 14, the second storage device 16, and the third storage device 17 can apply a large-capacity storage device. As the first storage device 14, the second storage device 16, and the third storage device 17, a storage device arranged outside the endoscope image learning device 10 can be applied. The first storage device 14, the second storage device 16, and the third storage device 17 may be communicably connected to the endoscope image learning device 10 via a network.

The normal light image shown in the embodiment is an example of the first image. The normal light image group described in the embodiment is an example of a first image group. The special light image shown in the embodiment is an example of the second image. The special light image group described in the embodiment is an example of a second image group.

The normal light image is a color image obtained by irradiating normal light and imaging in observation of the inside of the body cavity using the endoscope apparatus. The special light image is a color image obtained by irradiating special light and capturing an image in observation of a body cavity using an endoscope apparatus.

Normal light is light in which light in all wavelength bands of visible light is mixed almost evenly. The normal light image is used for normal observation. One example of the ordinary light is white light. A group of ordinary light images can be collected in a relatively large number.

Special light is light in various wavelength bands according to the purpose of observation, which is light in one specific wavelength band or light in a plurality of specific wavelength bands. The special light has a band narrower than the white wavelength band, and is used for NBI (Narrow Band Imaging), LCI, FICE (Flexible Spectral Imaging Color), and the like. NBI means narrowband observation.

The first example of the specific wavelength band is, for example, a blue band or a green band in a visible region. The wavelength band of the first example includes a wavelength band of 390 nm to 450 nm, or 530 nm to 550 nm. The light of the first example has a peak wavelength in a wavelength band of 390 nm to 450 nm, or 530 nm to 550 nm.

第二 The second example of the specific wavelength band is, for example, a red band in a visible region. The wavelength band of the second example includes a wavelength band of 585 nm to 615 nm, or a wavelength band of 610 nm to 730 nm. The light of the second example has a peak wavelength in a wavelength band from 585 nm to 615 nm or from 610 nm to 730 nm.

The third example of the specific wavelength band includes a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin, and the light of the third example has a peak wavelength at a wavelength band having a different absorption coefficient between oxyhemoglobin and reduced hemoglobin. Having. The third example wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less. The light of the third example has a peak wavelength in a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less.

The fourth example of the specific wavelength band is used for fluorescence observation which is observation of fluorescence emitted from a fluorescent substance in a living body. The wavelength band of the fourth example is a wavelength band of the excitation light for exciting the fluorescent substance, for example, a wavelength band from 390 nm to 470 nm.

五 The fifth example of the specific wavelength band is a wavelength band of infrared light. The fifth example wavelength band includes a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm. The light of the fifth example has a peak wavelength in a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm.

The special light image captured under the special light having the specific wavelength band is for obtaining an easy-to-view image according to the observation purpose of the lesion, for example, an observation purpose such as wanting to observe a surface structure. Is used only when the number of data items is small.

In the present embodiment, the first data set of the normal light image group stored in the first storage device 14 is prepared more than the second data set of the special light image group stored in the second storage device 16. It is assumed that A number is prepared for the third data set to compensate for the shortage of the second data set.

In addition, in the first storage device 14, the second storage device 16, and the third storage device 17, as an example of the correct answer data stored in association with each normal light image, each special light image, and each processed image, Examples include the type of lesion shown in the light image and the special light image, data indicating the position of the lesion, and identification information unique to the case. As an example of the classification of the lesion, there are two classifications, neoplastic or non-neoplastic, or NICE classification. The data indicating the position of the lesion may be rectangular information surrounding the lesion or mask data that covers the lesion. Note that NICE is an abbreviation of NBI International Colorectal Endoscopic Classification.

Although the first storage device 14, the second storage device 16, and the third storage device 17 are provided in the endoscope image learning device 10, the first storage device 14, the second storage device 16, and the third storage device are provided. 17 may be arranged outside the endoscope image learning device 10. In this case, a data set for learning can be acquired from a database external to the endoscope image learning device 10 via the communication unit 12.

The operation unit 18 uses a keyboard, a mouse, and the like that are connected to the computer applied to the endoscope image learning apparatus 10 by wire or wirelessly. The operation unit 18 receives various operation inputs for learning. That is, the operation unit 18 is used as a part of the user interface.

The CPU 20 reads various programs stored in the ROM 24 or a hard disk device (not shown) and executes various processes. An example of the program to be read is an endoscope image learning program.

The RAM 22 is used as a work area of the CPU 20. The RAM 22 is used as a storage unit that temporarily stores the read programs and various data.

As the display unit 26, various monitors such as a liquid crystal monitor that can be connected to a computer applied to the endoscope image learning device 10 are used. The display unit 26 is used as a part of a user interface. The display unit 26 and the operation unit 18 may be integrally formed by applying a monitor device of a touch panel type to the display unit 26.

ＣＰＵ In the endoscope image learning device 10, the CPU 20 reads the endoscope image learning program stored in the ROM 24, the hard disk device, or the like based on the instruction signal transmitted from the operation unit 18. The CPU 20 executes an endoscope image learning program.

[Functions of the endoscope image learning device]
<Overview>
FIG. 2 is a functional block diagram of the endoscope image learning device according to the embodiment. FIG. 2 is a functional block diagram showing main functions of the endoscope image learning device 10 shown in FIG. The endoscope image learning device 10 includes a learning unit 30 and an image generation unit 40.

The learning unit 30 uses the first data set stored in the first storage device 14, the second data set stored in the second storage device 16, and the third data set stored in the third storage device 17. Learn and generate a learning model for image recognition. In the present embodiment, a convolutional neural network, which is one of the learning models, is constructed. Hereinafter, CNN (Convolution Neural Network) represents a convolutional neural network.

The image generation unit 40 generates a processed image from the normal light image. Specifically, for a corresponding wavelength band between the spectral distribution of the normal light image and the spectral distribution of the special light image, a process of emphasizing the band component when the band components are the same or similar, or When the band components are different, a process of suppressing the band components is performed. Details of the image processing in the image generation unit 40 will be described later. The learning unit 30 and the image generation unit 40 may have a function of an image acquisition unit that acquires a normal light image and a special light image.

<Learning Department>
FIG. 3 is a functional block diagram of the learning unit shown in FIG. The learning unit 30 includes a CNN 32, an error calculating unit 34, and a parameter updating unit 36.

The CNN 32 is, for example, a part corresponding to a recognizer that recognizes the type of a lesion appearing in an endoscope image. The CNN 32 has a plurality of layer structures and holds a plurality of weight parameters. The CNN 32 may change from an unlearned model to a learned model by updating the weight parameter from an initial value to an optimal value.

The CNN 32 includes an input layer 32a, an intermediate layer 32b, and an output layer 32c. The intermediate layer 32b includes a set 32f composed of a convolutional layer 32d and a pooling layer 32e, and a fully connected layer 32g. Each layer of the learning unit 30 has a structure in which a plurality of nodes are connected using edges.

The normal light image 14a, the special light image 16a, and the processed image 17a to be learned are input to the input layer 32a. The normal light image 14a, the special light image 16a, and the processed image 17a are transmitted to the CNN 32.

The intermediate layer 32b is a part that extracts features from an image input using the input layer 32a. The convolutional layer 32d performs a filtering process on a nearby node in the previous layer, and acquires a feature map. For the filter processing, a convolution operation using a filter is applied.

The pooling layer 32e reduces the feature map output from the convolutional layer 32d to a new feature map. The convolution layer 32d plays a role of feature extraction such as edge extraction from an image. The pooling layer 32e plays a role in providing robustness so that the extracted features are not affected by translation or the like.

The intermediate layer 32b is not limited to the case where the convolutional layer 32d and the pooling layer 32e constitute one set. For example, the convolutional layer 32d may be continuous or may include a normalization layer (not shown).

The output layer 32c outputs a recognition result for classifying the type of lesion shown in the endoscopic image based on the features extracted using the intermediate layer 32b. The trained CNN 32 may, for example, classify endoscopic images into three categories: neoplastic, non-neoplastic, and other. The learned CNN 32 may output a score corresponding to neoplasm, a score corresponding to non-tumor, and a score corresponding to others as a recognition result. Note that the sum of the three scores is 100%.

(4) Arbitrary initial values are set for the filter coefficient applied to each convolutional layer 32d of the CNN 32 before learning, the offset value, and the connection weight with the next layer in a fully connected layer (not shown).

The error calculator 34 acquires the recognition result output from the output layer 32c of the CNN 32, the correct answer data 14b for the normal light image 14a, and the correct answer data 16b for the special light image 16a. Note that the correct data corresponding to the processed image 17a may be the correct data 14b for the normal light image 14a.

The error calculator 34 calculates an error between the recognition result and the correct answer data. Examples of a method for calculating the error include soft max cross entropy and sigmoid.

The parameter updating unit 36 adjusts the weight parameter of the CNN 32 by applying the error back propagation method based on the error calculated by using the error calculating unit 34. The adjustment process of the weight parameter of the CNN 32 is repeatedly performed, and the learning is repeatedly performed until the difference between the output of the CNN 32 and the correct data becomes small.

The learning unit 30 performs learning to optimize each parameter of the CNN 32 by using all data sets of the normal light image group, all data sets of the special light image group, and all data sets of the processed image group. Generate a trained model. Note that the endoscope image shown in the embodiment is an example of a medical image.

[Procedure of endoscope image learning method]
FIG. 4 is a flowchart illustrating a procedure of the endoscopic image learning method according to the embodiment. The endoscope image learning method shown in FIG. 4 includes a normal light image acquisition step S10, a special light image acquisition step S12, an image processing step S14, a processed image storage step S16, a learning step S18, and a recognizer update step S20. . The endoscope image learning method described in the embodiment is an example of a medical image learning method.

In the normal light image acquisition step S10, the learning unit 30 and the image generation unit 40 illustrated in FIG. 2 read the normal light image 14a from the first storage device 14. In the special light image acquisition step S12, the learning unit 30 reads the special light image 16a from the second storage device 16. The special light image acquisition step S12 may be performed before the learning step S18.

In the image processing step S14, the image generator 40 performs image processing on the normal light image 14a to generate a processed image. In the processed image storage step S16, the image generation unit 40 stores the processed image 17a in the third storage device 17.

In the learning step S18, the learning unit 30 reads the processed image 17a from the third storage device 17, and performs learning using the normal light image 14a, the special light image 16a, and the processed image 17a. In the recognizer updating step S20, the learning unit 30 updates the recognizer.

[Specific Example of Image Processing Using Image Generation Unit]
Next, image processing applied to the image generation unit 40 shown in FIG. 2 will be described in detail. The image generation unit 40 generates a processed image 17a by performing a process of enhancing or suppressing a signal in a wavelength band in the spectral distribution of the normal light image 14a.

If the band components in the corresponding wavelength band between the normal light image 14a and the special light image 16a are different between the normal light image 14a and the special light image 16a, the image generation unit 40 suppresses the signal of the corresponding wavelength band in the normal light image 14a. I do.

On the other hand, if the band components in the corresponding wavelength band between the normal light image 14a and the special light image 16a are the same or similar between the normal light image 14a and the special light image 16a, the corresponding wavelength band in the normal light image 14a May be emphasized. This makes it possible to increase the number of pieces of learning data applicable to learning of the special light image 16a.

(4) The image generation unit 40 can perform post-processing such as correction to which inverse processing is applied to an image difference that is different from the wavelength dependence and is caused by a difference in processing between observation modes. The learning unit 30 can apply the corrected image to learning. As an example of the processing between the observation modes, a linear matrix, a lookup table, and the like are given.

<First embodiment>
FIG. 5 is an explanatory diagram of the spectral sensitivity of the image sensor. The imaging device is provided in an endoscope that acquires an endoscope image. The imaging device is shown in FIG. The endoscope is shown in FIG.

横 The horizontal axis of the spectral sensitivity characteristic 100 shown in FIG. 5 represents the wavelength. The vertical axis of the spectral sensitivity characteristic 100 represents the sensitivity. The spectral sensitivity characteristic 100 has a B-channel component 102 in the B-channel wavelength band. The spectral sensitivity characteristic 100 has a G channel component 104 in a G channel wavelength band. The spectral sensitivity characteristic 100 has an R channel component 106 in an R channel wavelength band. B represents blue. G represents green. R represents red.

FIG. 6 is an explanatory diagram showing the relationship between the intensity distribution of the illumination light and the spectral distribution of the endoscope image. Reference numeral 120 shown in FIG. 6 represents the intensity distribution of normal light. Reference numeral 160 represents the intensity distribution of the illumination light in LCI. The horizontal axis of the normal light intensity distribution 120 and the illumination light intensity distribution 160 in the LCI represent wavelength. The vertical axis represents the intensity of the illumination light. Note that the illumination light in LCI is an example of special light.

The symbol 140 indicates the spectral distribution of the normal light image 14a. The horizontal axis of the spectral distribution 140 of the normal light image 14a represents a wavelength. The vertical axis represents signal strength.

分光 The spectral distribution 140 of the normal light image 14a has a B channel component 142 in the B channel wavelength band. The spectral distribution 140 of the normal light image 14a has a G channel component 144 in the G channel wavelength band. The spectral distribution 140 of the normal light image 14a has an R channel component 146 in the wavelength band of the R channel.

The symbol 180 represents the spectral distribution of the special light image 16a in LCI. The horizontal axis of the spectral distribution 180 of the special light image 16a in LCI represents a wavelength. The vertical axis represents signal strength. The spectral distribution 180 of the special light image 16a in LCI has a B channel component 182 in the B channel wavelength band. The spectral distribution 180 of the special light image 16a in LCI has a G channel component 184 in a G channel wavelength band. The spectral distribution 180 of the special light image 16a in LCI has an R channel component 186 in the wavelength band of the R channel.

分光 The spectral distribution 140 of the normal light image 14a and the spectral distribution 180 of the special light image 16a in LCI are different in the characteristics of the B channel component. That is, the B channel component of the normal light image 14a and the B channel component of the special light image 16a in LCI each have two peaks, but have different sensitivities at the two peaks. In other words, the normal light image 14a and the special light image 16a in the LCI have different B channel components.

On the other hand, the G channel component and the R channel component in both are equal except for the difference in absolute value. That is, the difference between the normal light image 14a and the special light image 16a in the LCI is smaller in the image including only the G channel component and the R channel component than in the image including only the B channel component. In other words, the characteristics of the G channel component and the R channel component are similar between the normal light image 14a and the special light image 16a in LCI.

From such a property, the image information of the G channel component in the normal light image 14a and the image information of the R channel component in the normal light image 14a are also useful in learning the special light image 16a in LCI. On the other hand, the image information of the B channel component in the normal light image 14a is considered to have unnecessary or adverse effects in learning the special light image 16a in LCI.

Therefore, the image generation unit 40 performs processing for suppressing the image information of the B channel component in the normal light image 14a to generate the processed image 17a. The learning unit 30 learns the special light image 16a in LCI by adding the processed image 17a to the normal light image 14a and the special light image 16a.

処理 As an example of the process of suppressing the image information of the B channel component in the normal light image 14a, there is a process of multiplying the luminance value in the normal light image 14a by 1 / α. Here, α is an arbitrary constant exceeding 1. As another example of the process of suppressing the image information of the B channel component in the normal light image 14a, there is a process of setting the luminance value to a specified value. An example of the prescribed value is a luminance value that minimizes luminance. The process of suppressing the image information of the B channel component may be a process of randomly giving the luminance value of each pixel.

(4) The luminance value in the normal light image 14a may be shifted together with the process of multiplying the luminance value in the normal light image 14a by 1 / α. Note that the process of multiplying the luminance value by 1 / α described in the embodiment is an example of a process of converting the luminance variance. The process of converting the luminance variance may include a process of shifting the luminance value in the normal light image 14a.

Instead of the process of suppressing the image information of the B channel component in the normal light image 14a, at least one of the image information of the G channel component and the image information of the R channel component in the normal light image 14a is emphasized, and the B channel The influence of the image information of the component may be reduced.

As an example of the process of enhancing the image information of the G channel component and the image information of the R channel component in the normal light image 14a, there is a process of increasing the luminance value of the normal light image 14a by β times. Here, β is an arbitrary constant exceeding 1.

(4) The luminance value in the normal light image 14a may be shifted together with the process of multiplying the luminance value in the normal light image 14a by β. Note that the process of multiplying the luminance value by β shown in the embodiment is an example of a process of converting the luminance variance. The process of converting the luminance variance may include a process of shifting the luminance value in the normal light image 14a.

<Effects>
According to the image processing according to the first embodiment, the normal light image 14a is subjected to the process of suppressing the B channel component in the spectral distribution or the process of enhancing at least one of the G channel component and the R channel component. This makes it possible to expand learning data applicable to learning of the special light image 16a in LCI.

<Second embodiment>
The image processing according to the second embodiment is applied to the case where the normal light image 14a is used for learning the special light image 16a in LCI, similarly to the image processing according to the first embodiment. In the second embodiment, image processing is performed focusing on the difference between the B channel components in the normal light image 14a and the special light image 16a.

The B-channel component of the special light image 16a in the ＩLCI includes a larger number of short wavelength components than the B-channel component of the normal light image 14a. Thus, the special light image 16a in the LCI has a property of representing a minute blood vessel structure or the like with higher contrast than the normal light image 14a.

The minute blood vessel structure corresponds to the high frequency component of the spatial frequency of the special light image 16a in LCI. Therefore, the image generation unit 40 generates a processed image 17a obtained by performing a process of emphasizing a high frequency component of a spatial frequency on the image of the B channel component of the normal light image 14a. This makes it possible to bring the B-channel component image of the normal light image 14a closer to the B-channel component image of the special light image 16a. Therefore, it is possible to expand the learning data of the special light image 16a in the LCI.

As an example of defining a high-frequency component, there is an example in which the frequency of a special light image 16a containing many high-frequency components is analyzed and a frequency or a frequency band to be emphasized is defined based on a frequency distribution. For example, a reference frequency is calculated, and a frequency component equal to or higher than the reference frequency among the spatial frequency components of the processing target image may be set as a high frequency component. The reference frequency can be calculated as the lower limit of a range in which a certain percentage or more of the frequency components of the frequency components included in the processing target image are included.

マ ス ク As an example of the process of enhancing the high frequency component of the spatial frequency, there is a mask process using an unsharp mask. As an example of the process of suppressing the high-frequency component of the spatial frequency, there is a filter process using a low-pass filter. The high-frequency component of the spatial frequency shown in the embodiment is an example of a prescribed frequency component of the spatial frequency. The image of the B channel component shown in the embodiment is an example of the processing target band.

<Effects>
According to the image processing according to the second embodiment, the process of emphasizing the high frequency component of the spatial frequency is performed on the image of the B channel component of the normal light image 14a. The processed image 17a shows a minute blood vessel structure or the like with a higher contrast than the normal light image 14a. This makes it possible to expand learning data applicable to learning of the special light image 16a in LCI.

<Third embodiment>
The image processing according to the third embodiment is applied to a case where a special light image 16a in NBI is used in learning applied to an automatic recognition system for a region of interest using the normal light image 14a.

FIG. 7 is an explanatory diagram showing the relationship between the spectral distribution of a normal light image and the spectral distribution of a special light image in NBI. Reference numeral 200 indicates the spectral distribution of the normal light image 14a. The horizontal axis of the spectral distribution 200 of the normal light image 14a represents a wavelength. The vertical axis represents signal strength.

分光 The spectral distribution 200 of the normal light image 14a has a B channel component 202 in the B channel wavelength band. The spectral distribution 200 of the normal light image 14a has a G channel component 204 in a G channel wavelength band. The spectral distribution 200 of the normal light image 14a has an R channel component 206 in an R channel wavelength band.

Reference numeral 220 indicates a spectral distribution characteristic of the special light image 16a in NBI. The horizontal axis of the spectral distribution 220 of the special light image 16a in NBI represents a wavelength. The vertical axis represents signal strength. The special light image 16a in the NBI has a B channel component 222 in the wavelength band of the B channel. The special light image 16a in the NBI has a G channel component 224 in the G channel wavelength band.

On the other hand, in the special light image 16a in the NBI, the R channel component is lost when receiving light. The spectral distribution 220 of the special light image 16a in the NBI shown in FIG. 7 indicates the missing R channel component using a dashed line. That is, in the normal light image 14a and the special light image 16a in the NBI, the R channel components are different.

On the other hand, the B channel component 222 and the G channel component 224 of the special light image 16a in the NBI have different bandwidths from the B channel component 202 and the G channel component 204 of the normal light image 14a, but have overlapping bands.

に お い て In the normal light image 14a and the special light image 16a in the NBI, the B channel component and the G channel component have properties closer to each other than the R channel components. That is, in the normal light image 14a and the special light image 16a in the BI, the B channel components and the G channel components are similar.

Therefore, the image generation unit 40 performs a process of suppressing the R channel component 206 in the normal light image 14a to generate the processed image 17a. The learning unit 30 can apply the processed image to the learning of the special light image 16a in the NBI.

The image generation unit 40 may perform processing for enhancing at least one of the B channel component 202 and the G channel component 204 of the normal light image 14a to generate the processed image 17a. The learning unit 30 can apply the processed image 17a to learning of the special light image 16a in NBI. The processing of suppressing the processing target band component of the normal light image 14a and the processing of emphasizing the processing target band component are the same as in the first embodiment. The description here is omitted.

<Effects>
According to the image processing according to the third embodiment, the normal light image 14a is subjected to the suppression processing of the R channel component or the enhancement processing of at least one of the G channel component and the B channel component. This makes it possible to expand the learning data applicable to learning the special light image 16a in the NBI.

<Application Example of Third Embodiment>
The B channel component 222 or the G channel component 224 of the special light image 16a in the NBI has a large band component in which the hemoglobin absorption coefficient is large compared to the B channel component 222 or the G channel component 224 of the normal light image 14a. The special light image 16a has a property of representing a blood vessel structure or the like with a higher contrast than the normal light image 14a. Therefore, similarly to the image processing according to the second embodiment, the image generation unit 40 performs a process of enhancing the high-frequency component in the spatial frequency of the image of the B-channel component 222 or the image of the G-channel component 224 of the normal light image 14a. To generate the processed image 17a. The processed image 17a shows a minute blood vessel structure or the like with a higher contrast than the normal light image 14a. This makes it possible to expand the learning data applicable to learning the special light image 16a in the NBI.

<Fourth embodiment>
The image processing according to the fourth embodiment is applied when learning is performed by applying a plurality of image groups acquired by applying different observation modes. The image processing shown in the first embodiment to the third embodiment is an endoscope image obtained by applying an observation mode different from the original, an endoscope image acquired by applying the original observation mode in a pseudo manner. This is a process for approaching.

Then, if the influence of another observation mode becomes too large, the improvement of the learning accuracy may be less than expected. Therefore, the learning method applied to the learning unit 30 is devised to adjust the balance of the plurality of image groups. Specifically, the learning method for learning the normal light image group and the learning method for the special light image group is switched. Switching between the learning method for learning the normal light image group and the learning method for the special light image group includes switching between the learning method for learning the processed image 17a generated from the normal light image 14a and the learning method for learning the special light image 16a. . Hereinafter, a specific example of switching the learning method will be described.

《Introduction of weight coefficient》
Many of the learning methods including CNN implement learning that minimizes or maximizes a specific objective function. Therefore, an objective function calculated only from the image group to which the original observation mode is applied is set to L_maim, and an objective function calculated only from the image group to which the observation mode different from the original is applied is set to L_sub. Learning for minimizing or maximizing the objective function L expressed by using Expression 1 is performed.

L = L_maim + λ × L_sub Equation 1
Note that λ in Equation 1 is a design parameter. An arbitrary constant less than 1 is applied to the design parameter λ. When the value of the design parameter λ is made smaller, the effect of the second term representing the influence of an observation mode different from the original term on the objective function L can be further suppressed with respect to the first term representing the influence of the original observation mode.

The case where learning is performed using the image group of the normal light image 14a and the image group of the special light image 16a in LCI is as follows. An objective function calculated from only the image group of the special light image 16a is defined as L_maim. The objective function calculated from only the image group of the normal light image 14a is L_sub. By appropriately setting the design parameter λ, learning for minimizing or maximizing the objective function L of Equation 1 can be performed.

内 The endoscope image learning device 10 shown in FIG. 2 may include a design parameter setting unit for setting the design parameter λ. In addition, the endoscope image learning device 10 may include an objective function storage unit that stores the objective function L. The learning unit 30 can read out the objective function L and the design parameter λ and perform learning.

《Transfer learning》
First, a CNN is learned using a plurality of image groups having different observation modes. Thereafter, the CNN is re-learned using only the image group to which the original observation mode is applied, using the learned parameters as initial values.

Specifically, relearning is performed using the special light image 16a for the CNN that has been learned using the normal light image 14a and the processed image 17a or the CNN that has been learned using the processed image 17a.

This makes it possible to use the information of the image group to which the observation mode different from the original is applied, and to bring the image group closer to the image group to which the original observation mode is applied.

The learning unit 30 includes a first learning unit that learns the CNN using a plurality of image groups having different observation modes, and a second learning unit that re-learns the CNN using only the image group to which the original observation mode is applied. Can be provided.

<Effects>
According to the image processing according to the fourth embodiment, the influence of the image group to which the observation mode different from the original is applied is suppressed while using the information of the image group to which the observation mode different from the original is applied. Learning can be performed.

[Overall Configuration of Endoscope System for Obtaining Endoscope Image]
FIG. 8 is an overall configuration diagram of the endoscope system. The endoscope system 300 shown in FIG. 8 includes an endoscope 302, a light source device 311, a processor device 312, a display device 313, an image processing device 314, an input device 315, and a monitor device 316.

The endoscope 302 is an electronic endoscope. The endoscope 302 is a flexible endoscope. The endoscope 302 includes an insertion section 320, an operation section 321 and a universal cord 322. The insertion section 320 is inserted into the subject. The insertion section 320 is formed in a small diameter and a long shape as a whole.

The insertion section 320 includes a flexible section 325, a curved section 326, and a tip section 327. The insertion section 320 is configured by connecting a flexible section 325, a bending section 326, and a tip section 327 in a row. The flexible portion 325 has flexibility in order from the proximal end to the distal end of the insertion portion 320. The bending section 326 has a structure that can be bent when the operation section 321 is operated. The distal end portion 327 has a built-in image pickup optical system and image pickup element 328 (not shown).

(4) As the image sensor 328, a CMOS image sensor or a CCD image sensor is applied. CMOS is an abbreviation for Complementary \ Metal \ Oxide \ Semiconductor. CCD is an abbreviation for Charge \ Coupled \ Device.

観 察 An observation window (not shown) is arranged on the distal end surface 327a of the distal end portion 327. The observation window is an opening formed on the distal end surface 327a of the distal end portion 327. A cover (not shown) is attached to the observation window. An imaging optical system (not shown) is arranged behind the observation window. The image plane of the observation site enters the imaging surface of the imaging element 328 via an observation window, an imaging optical system, and the like. The image sensor 328 captures the image light of the observed region that has entered the image capturing surface of the image sensor 328 and outputs an image signal. The term “imaging” as used herein means that reflected light from the region to be observed is converted into an electric signal.

The operation unit 321 is continuously provided on the base end side of the insertion unit 320. The operation unit 321 includes various operation members operated by an operator. Specifically, the operation unit 321 includes two types of bending operation knobs 329. The bending operation knob 329 is used when performing a bending operation of the bending portion 326. Note that the surgeon may be called a doctor, an operator, an observer, a user, or the like.

The operation unit 321 includes an air / water supply button 330 and a suction button 331. The air / water button 330 is used when the operator performs an air / water operation. The suction button 331 is used when an operator performs a suction operation.

The operation unit 321 includes a still image capturing instruction unit 332 and a treatment instrument introduction port 333. The still image capturing instruction unit 332 is operated by the operator when capturing a still image of the observed region. The treatment instrument introduction port 333 is an opening for inserting the treatment instrument into the treatment instrument insertion passage that passes through the inside of the insertion section 320. The illustration of the treatment tool insertion passage and the treatment tool is omitted.

The universal cord 322 is a connection cord for connecting the endoscope 302 to the light source device 311. The universal cord 322 includes a light guide 335, a signal cable 336, and a fluid tube (not shown) that pass through the inside of the insertion section 320.

The distal end of the universal cord 322 includes a connector 337a connected to the light source device 311 and a connector 337b branched from the connector 337a and connected to the processor device 312.

When the connector 337 a is connected to the light source device 311, the light guide 335 and a fluid tube (not shown) are inserted into the light source device 311. Thereby, necessary illumination light, water, and gas are supplied from the light source device 311 to the endoscope 302 via the light guide 335 and a fluid tube (not shown).

As a result, the illumination light is emitted from the illumination window (not shown) of the distal end surface 327a of the distal end portion 327 toward the observation target site. Further, in response to the pressing operation of the air / water supply button 330, gas or water is jetted from an air / water supply nozzle (not shown) on the distal end surface 327a of the distal end portion 327 toward an observation window (not shown) on the distal end surface 327a.

When the connector 337b is connected to the processor device 312, the signal cable 336 and the processor device 312 are electrically connected. As a result, an imaging signal of the observed region is output from the imaging element 328 of the endoscope 302 to the processor device 312 via the signal cable 336, and a control signal is output from the processor device 312 to the endoscope 302.

In the present embodiment, a flexible endoscope has been described as an example of the endoscope 302. However, as the endoscope 302, various types of electronic endoscopes capable of capturing moving images of a site to be observed, such as a hard endoscope, are described. An endoscope may be used.

The light source device 311 supplies illumination light to the light guide 335 of the endoscope 302 via the connector 337a. As the illumination light, white light or light in a specific wavelength band can be applied. The illumination light may be a combination of white light and light of a specific wavelength band. The light source device 311 is configured to be able to appropriately select light in a wavelength band according to an observation purpose as illumination light.

The white light may be light in a white wavelength band or light in a plurality of wavelength bands. The specific wavelength band is a band narrower than the white wavelength band. As light of a specific wavelength band, light of one type of wavelength band may be applied, or light of a plurality of wavelength bands may be applied. A specific wavelength band may be called special light.

The processor device 312 controls the operation of the endoscope 302 via the connector 337b and the signal cable 336. In addition, the processor device 312 acquires an image signal from the image sensor 328 of the endoscope 302 via the connector 337b and the signal cable 336. The processor device 312 acquires an image signal output from the endoscope 302 by applying a specified frame rate.

The processor device 312 generates an endoscope image which is an observation image of the observed part based on the imaging signal acquired from the endoscope 302. The endoscope image 338 here includes a moving image. The endoscope image 338 may include a still image 339.

When the still image imaging instruction unit 332 of the operation unit 321 is operated, the processor device 312 generates a still image 339 of the observed part based on the imaging signal acquired from the imaging element 328 in parallel with the generation of the moving image. . The still image 339 may be generated at a higher resolution than the resolution of the moving image.

(4) When the endoscope image 338 is generated, the processor device 312 performs image quality correction using digital signal processing such as white balance adjustment and shading correction. The processor device 312 may add additional information specified by the DICOM standard to the endoscope image 338. Note that DICOM is an abbreviation for Digital Imaging and Communications in Medicine.

The endoscope image 338 is an in-vivo image of the inside of the subject, that is, the inside of the living body. When the endoscope image 338 is an image obtained by imaging using light in a specific wavelength band, both are special light images. Then, the processor device 312 outputs the generated endoscope image 338 to each of the display device 313 and the image processing device 314. The processor device 312 may output the endoscope image 338 to a storage device (not shown) via a network (not shown) according to a communication protocol conforming to the DICOM standard.

The display device 313 is connected to the processor device 312. The display device 313 displays the endoscope image 338 transmitted from the processor device 312. The operator can perform the operation of moving the insertion section 320 forward and backward while checking the endoscope image 338 displayed on the display device 313. When the surgeon detects a lesion or the like in the observed region, the surgeon can operate the still image imaging instruction unit 332 to capture a still image of the observed region.

コ ン ピ ュ ー タ A computer is used for the image processing device 314. As the input device 315, a keyboard, a mouse, and the like connectable to a computer are used. The connection between the input device 315 and the computer may be either a wired connection or a wireless connection. As the monitor device 316, various monitors that can be connected to a computer are used.

As the image processing device 314, a diagnosis support device such as a workstation and a server device may be used. In this case, the input device 315 and the monitor device 316 are provided for each of a plurality of terminals connected to a workstation or the like. Further, as the image processing device 314, a medical service support device that supports creation of a medical report or the like may be used.

The image processing device 314 acquires the endoscope image 338 and stores the endoscope image 338. The image processing device 314 controls the reproduction of the monitor device 316. The image processing device 314 illustrated in FIG. 8 can have the function of the endoscope image learning device 10 described with reference to FIGS. 1 to 7. The input device 315 illustrated in FIG. 8 corresponds to the operation unit 18 illustrated in FIG. The monitor device 316 shown in FIG. 8 corresponds to the display unit 26 shown in FIG.

Note that the term image in this specification includes the meaning of a still image 339 such as an electric signal representing an image and information representing an image. The term image in this specification means at least one of the image itself and image data.

用語 In addition, the term image storage can be read as image storage. Image storage here means non-temporary storage of an image. The image processing device 314 may include a temporary storage memory for temporarily storing an image.

The input device 315 is used to input an operation instruction to the image processing device 314. The monitor device 316 displays the endoscope image 338 under the control of the image processing device 314. The monitor device 316 may function as a display unit of various information in the image processing device 314.

The image processing device 314 can be connected to a storage device (not shown) via a network (not shown). For communication between the devices via the image storage format and the network, the DICOM standard, a protocol based on the DICOM standard, and the like can be applied.

記憶 A storage device (not shown) to which data is temporarily stored can be applied. The storage device may be managed using a server device (not shown). A computer that stores and manages various data can be applied to the server device.

[Observation mode of endoscope system]
FIG. 9 is a functional block diagram of the endoscope system. The endoscope system 300 is configured to be switchable between a normal light observation mode and a special light observation mode. The operator can switch between a normal light observation mode and a special light observation mode by operating an observation mode switching button (not shown).

The light source device 311 includes a first laser light source 400, a second laser light source 402, and a light source control unit 404. The first laser light source 400 is a blue laser light source having a center wavelength of 445 nm. The second laser light source 402 is a violet laser light source having a center wavelength of 405 nm. As the first laser light source 400 and the second laser light source 402, laser diodes can be used. Light emission of the first laser light source 400 and the second laser light source 402 is individually controlled using the light source control unit 404. The emission intensity ratio between the first laser light source 400 and the second laser light source 402 can be changed freely.

The endoscope 302 includes a first optical fiber 410, a second optical fiber 412, a phosphor 414, a diffusion member 416, an imaging lens 418, an imaging device 328, and an analog-to-digital converter 420.

照射 An irradiation unit is configured using the first laser light source 400, the second laser light source 402, the first optical fiber 410, the second optical fiber 412, the phosphor 414, and the diffusion member 416.

Laser light emitted from the first laser light source 400 is applied to the phosphor 414 disposed at the distal end 327 of the endoscope 302 via the first optical fiber 410. The phosphor 414 is configured to include a plurality of types of phosphors that absorb part of the blue laser light from the first laser light source 400 and excite and emit light in a range from green to yellow. Accordingly, the light emitted from the phosphor 414 is blue which transmits the blue laser light from green to excitation light not absorbed by the excitation light L ₁₁ and the phosphor 414 in the range up to the yellow from the first laser light source 400 the laser beam L ₁₂ is combined, the light L ₁ of the white or pseudo white.

白色 Incidentally, the white light mentioned here is not limited to a light that strictly includes all wavelength components of visible light. For example, any light containing a specific wavelength band, such as R, G, and B, may be used. Light including a wavelength component from green to red, or light including a wavelength component from blue to green may be broadly defined. Shall be included.

On the other hand, the laser light emitted from the second laser light source 402 is applied to the diffusion member 416 disposed at the distal end 327 of the endoscope 302 via the second optical fiber 412. For the diffusion member 416, a resin material or the like having a light-transmitting property can be used. Light emitted from the diffusion member 416, a light L ₂ of the narrow band wavelength light amount is uniform in the exposure area.

FIG. 10 is a graph showing the light intensity distribution. The light source control unit 404 changes the light amount ratio between the first laser light source 400 and the second laser light source 402. Thus, the light quantity ratio between the light L ₁ and the light L ₂ is changed, the changed wavelength pattern of the light L ₁ and the irradiation light L ₀ is a composite light of the light L ₂ is, wavelengths different patterns depending on the observation mode it can be irradiated with the irradiation light L ₀ of.

内 The endoscope system 300 shown in FIG. 9 includes an imaging unit including an imaging lens 418, an imaging element 328, and an analog-to-digital conversion unit 420. As shown in FIG. 8, the imaging unit is disposed at the distal end 327 of the endoscope 302.

撮 像 The imaging lens 418 shown in FIG. 9 focuses the incident light on the imaging element 328. The image sensor 328 generates an analog signal corresponding to the received light. An analog signal output from the image sensor 328 is converted to a digital signal using the analog-to-digital converter 420 and input to the processor device 312.

The processor device 312 includes an imaging control unit 440, an image processing unit 442, an image acquisition unit 444, and an image recognition unit 446.

The imaging control unit 440 controls the light source control unit 404 of the light source device 311, the imaging element 328 and the analog-to-digital conversion unit 420 of the endoscope 302, and the image processing unit 442 of the processor device 312, and uses the endoscope system 300. It controls the imaging of the moving image and the still image.

The image processing unit 442 performs image processing on the digital signal input from the analog-to-digital conversion unit 420 of the endoscope 302 to generate an image. The image processing unit 442 performs image processing according to the wavelength pattern of irradiation light at the time of imaging.

The image acquisition unit 444 acquires the image generated by the image processing unit 442. That is, the image acquisition unit 444 sequentially acquires a plurality of images that are taken in chronological order by applying a fixed frame rate to the inside of the body cavity of the subject. Note that the image acquisition unit 444 may acquire an image input from the input unit 447 or an image stored in the storage unit 468. Further, an image may be obtained from an external device such as a server connected to a network (not shown). The images in these cases are also preferably a plurality of images taken in time series.

The image recognition unit 446 performs image recognition of the image acquired by the image acquisition unit 444 using the learning model learned by the endoscope image learning device 10. In the present embodiment, a lesion is recognized from the image acquired by the image acquisition unit 444. The lesion is not limited to the one caused by the disease but includes an area having a state different from a normal state in appearance.

Examples of lesions include polyps, cancer, colonic diverticulum, inflammation, treatment marks, clipping points, bleeding points, perforations, and vascular atypia. Examples of the treatment scar include an EMR scar and an ESD scar. EMR is an abbreviation for Endoscopic \ Mucosal \ Resection. ESD is an abbreviation for Endoscopic \ Submucosal \ Dissection.

The display control unit 450 causes the display device 313 to display the image generated using the image processing unit 442. The display control unit 450 may superimpose the lesion recognized using the image recognition unit 446 on the image so as to be recognizable.

The storage control unit 452 causes the storage unit 468 to store the image generated using the image processing unit 442. For example, and stores the information of the wavelength pattern of the irradiation light L ₀ at the time of capturing a captured image and the image according to the acquired instruction of a still image in the storage unit 468.

(4) An example of the storage unit 468 is a storage device such as a hard disk. Note that the storage unit 468 is not limited to the one built in the processor device 312. For example, an external storage device (not shown) connected to the processor device 312 may be used. The external storage device may be connected via a network (not shown).

The endoscope system 300 configured as described above normally captures a moving image at a fixed frame rate, and displays the captured image on the display device 313. Further, a lesion is detected from the captured moving image, and the detected lesion is superimposed on the moving image so as to be recognizable and displayed on the display device 313.

According to the endoscope system 300, the image recognition unit 446 using the learning model trained by the endoscope image learning device 10 is applied to automatically recognize the endoscope image and perform image recognition of the special light image. Can be done properly.

内 The endoscope system 300 described with reference to FIGS. 8 to 10 can acquire the normal light image 14a and the special light image 16a.

[Modification of Endoscope System]
[Modification of Processor Device]
The processor device 312 may have the function of the image processing device 314. That is, the processor device 312 may be configured integrally with the image processing device 314. In such an embodiment, the display device 313 can also serve as the monitor device 316. The processor device 312 may include a connection terminal to which the input device 315 is connected.

[Example of generating feature image]
As a medical image, based on at least one of a white band light, a normal image obtained by irradiating light of a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light of a specific wavelength band The calculation may be used to generate a feature image.

[Example of application to a program that causes a computer to function as an image processing device]
The above-described endoscopic image learning method can be configured as a program that realizes a function corresponding to each step in the endoscopic image learning method using a computer. For example, a computer has an image acquisition function of acquiring first and second images having different spectral distributions from each other, an image processing function of performing image processing on the first image to generate a third image, and a plurality of first images. Each of the first image group, the second image group including the plurality of second images, and the third image group including the plurality of third images, and the first image group, the second image group, and the third image group A program for realizing a learning function of learning a recognizer to be applied to automatic recognition using correct answer data that is a correct recognition result, wherein the image processing function is configured to perform a second function of a band included in the spectral distribution of the first image. Image processing to suppress the signal of the band having a different characteristic from the corresponding band of the image, and among the bands included in the first image, the corresponding band of the second image and the characteristic of the signal of the same band or a signal of a similar band. Less image processing to emphasize Also subjected to either a can realize a program for generating a third image from the first image.

A program for causing a computer to realize the above-described endoscopic image learning function is stored in a computer-readable information storage medium, which is a non-transitory information storage medium that is a tangible entity, and the program is provided through the information storage medium. Is possible.

態 様 Alternatively, instead of storing and providing the program in the non-transitory information storage medium, a mode in which the program signal is provided via a network is also possible.

[Combinations of Embodiments and Modifications]
The components described in the above-described embodiment and the components described in the modified examples can be appropriately combined and used, and some components can be replaced.

In the embodiment of the present invention described above, constituent elements can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by those having ordinary knowledge in the art within the technical spirit of the present invention.

Reference Signs List 10 Endoscope image learning device 12 Communication unit 14 First storage device 14a Normal light image 16 Second storage device 16a Special light image 17 Third storage device 17a Processed image 18 Operation unit 20 CPU
22 RAM
24 ROM
26 display unit 30 learning unit 32 CNN
32a Input layer 32b Intermediate layer 32c Output layer 32d Convolutional layer 32e Pooling layer 32f Set 32g Fully connected layer 34 Error calculator 36 Parameter updater 40 Image generator 100 Spectral sensitivity distribution 102 B channel component 104 G channel component 106 R channel component 120 Normal light intensity distribution 140 Normal light image spectral distribution characteristic 142 B channel component 144 G channel component 146 R channel component 160 Illumination light intensity distribution in LCI 180 Spectral distribution characteristic of special light image 182 B channel component 184 G channel component 186 R channel component 200 Spectral distribution characteristic of normal light image 202 B channel component 204 G channel component 206 R channel component 220 Spectral distribution characteristic of special light image 222 B channel component 224 G channel component 300 Endoscope system 30 Endoscope 311 Light source device 312 Processor device 313 Display device 314 Image processing device 315 Input device 316 Monitor device 320 Insertion section 321 Operation section 322 Universal cord 325 Flexible section 326 Curved section 327 Tip section 327a Tip face 328 Image sensor 329 Curve operation knob 330 Air supply / water supply button 331 Suction button 332 Still image imaging instruction section 333 Treatment tool introduction port 335 Light guide 336 Signal cable 337a Connector 337b Connector 338 Endoscope image 339 Still image 400 First laser light source 402 Second laser light source 404 Light source control Unit 410 first optical fiber 412 second optical fiber 414 phosphor 416 diffusion member 418 imaging lens 420 analog-to-digital conversion unit 442 image processing unit 444 image acquisition unit 446 image recognition unit 447 input unit 450 Display control unit 452 Storage control unit 468 Storage units S10 to S21 Each step of the endoscope image learning method

Claims (18)

1. An image acquisition unit that acquires a first image and a second image having different spectral distributions from each other,
   An image processing unit that performs image processing on the first image to generate a third image,
   A first image group including a plurality of the first images, a second image group including a plurality of the second images, and a third image group including a plurality of the third images, and the first image group, the second A learning unit that learns a recognizer to be applied to automatic recognition, using the correct answer data of the image group and the third image group,
   With
   The image processing unit is an image processing for suppressing a signal of a band having a characteristic different from a corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. A medical image that generates the third image from the first image by performing at least one of image processing that emphasizes a signal in a same band or a signal in a similar band, and a characteristic corresponding to the band of the second image. Learning device.
2. The medical image learning device according to claim 1, wherein the image processing unit performs a process of converting a luminance variance on the first image.
3. The medical image learning device according to claim 1, wherein the image processing unit performs a process of changing a luminance to a prescribed luminance value on the first image.
4. The medical image learning device according to claim 1, wherein the image processing unit performs a process of changing luminance to a random luminance value on the first image.
5. The first image is a normal light image captured using normal light,
   The medical image learning device according to claim 1, wherein the second image is a special light image captured using special light having a band narrower than normal light.
6. The medical image learning device according to claim 5, wherein the image processing unit performs a process of suppressing a B channel component on the normal light image.
7. 6. The medical image learning device according to claim 5, wherein the image processing unit performs a process of enhancing at least one of an R channel component and a G channel component on the normal light image.
8. The medical image learning device according to claim 5, wherein the image processing unit performs a process of suppressing an R channel component on the normal light image.
9. The medical image learning device according to claim 5, wherein the image processing unit performs a process of enhancing a B channel component and a G channel component on the normal light image.
10. The image processing unit performs a process of suppressing a specified frequency component of a spatial frequency in a processing target band of the first image or a process of emphasizing a specified frequency component of a spatial frequency in a processing target band of the first image. The medical image learning device according to claim 1.
11. The first image is a normal light image captured using normal light,
    The second image is a special light image captured using special light of a narrower band than normal light,
    The medical image learning device according to claim 10, wherein the image processing unit performs a process of enhancing a high frequency component of a B channel component of the normal light image.
12. The medical image learning device according to any one of claims 1 to 11, wherein the learning unit switches a learning method between learning of the first image group and learning of the second image group.
13. The medical image learning device according to any one of claims 1 to 11, wherein the learning unit switches a learning method between learning of the second image group and learning of the third image group.
14. The medical image learning device according to claim 12 or 13, wherein the learning unit applies a learning method that suppresses or emphasizes the influence of the third image on learning.
15. The learning unit, the first image group, the third image group, the correct data of the first image group, and a learned recognizer using the correct data of the third image group, or the third image group 14. The learning device according to claim 12, wherein learning is performed using correct data of the second image group and the second image group for a recognizer that has learned using correct data of the third image group. Medical image learning device.
16. An image acquisition step of acquiring a first image and a second image having different spectral distributions from each other,
    An image processing step of performing image processing on the first image to generate a third image,
    A first image group including a plurality of the first images, a second image group including a plurality of the second images, and a third image group including a plurality of the third images, and the first image group, the second A learning step of learning a recognizer to be applied to automatic recognition, using the correct answer data of the image group and the third image group,
    Including
    The image processing step is an image processing for suppressing a signal of a band having a characteristic different from a corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. A medical image that generates the third image from the first image by performing at least one of image processing that emphasizes a signal in a same band or a signal in a similar band, and a characteristic corresponding to the band of the second image. Learning method.
17. On the computer,
    An image acquisition function for acquiring a first image and a second image having different spectral distributions from each other,
    An image processing function of performing image processing on the first image to generate a third image, and a first image group including a plurality of first images, a second image group including a plurality of the second images, and a plurality of Learning to learn a recognizer to be applied to automatic recognition using a third image group including the third image, and correct data of each of the first image group, the second image group, and the third image group. A program that realizes a function,
    The image processing function is an image processing for suppressing a signal of a band having a characteristic different from a corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. A program that performs at least one of image processing for enhancing a signal in a band having the same characteristic as a band corresponding to the second image or a signal in a similar band, and generates the third image from the first image.
18. A non-transitory and computer-readable storage medium, wherein the instructions stored in the storage medium are read by a computer,
    An image acquisition function of acquiring a first image and a second image having different spectral distributions from each other,
    An image processing function of performing image processing on the first image to generate a third image,
    A first image group including a plurality of the first images, a second image group including a plurality of the second images, and a third image group including a plurality of the third images, and the first image group, the second Using a correct function of each of the image group and the third image group, a learning function of learning a recognizer to be applied to automatic recognition,
    The image processing function is an image processing that suppresses a signal of a band having a characteristic different from a corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. The corresponding band of the second image and the characteristics are subjected to at least one of image processing for enhancing a signal of the same band or a signal of a similar band, and generating the third image from the first image. A storage medium that allows a computer to execute a learning function.

Images