JPWO2020003992A1 - Learning device and learning method, and medical image processing device - Google Patents

Abstract

Provided are a learning device and a learning method capable of efficiently generating a model for performing image recognition on a medical image having a specific image quality, and a medical image processing device. The learning device (1) uses a first model (M1) that performs image recognition on a first-quality medical image by learning using a first-quality medical image group composed of first-quality medical images. Based on the generated first learning unit (10) and the first model (M1) generated by the first learning unit (10), a second medical image group composed of second-quality medical images is used. It includes a second learning unit (20) that generates a second model (M2) that performs image recognition on a medical image having a second image quality by learning. The second medical image group is composed of medical images having the same image quality as the image quality of the medical image to be recognized, and the first medical image group is composed of medical images having an image quality different from that of the second image quality. Since the second learning unit (20) learns based on the learning result of the first learning unit (10), it is possible to generate a highly accurate model even when the number of the second medical image group is small.

Description

The present invention relates to a learning device and a learning method, and a medical image processing device, and in particular, a learning device and a learning method for generating a model for performing image recognition on a medical image, and medical treatment using the model. Regarding image processing equipment.

There are known techniques for automatically detecting lesions from medical images and classifying lesions by type using an image recognition model generated by machine learning. In machine learning, image recognition such as detection and classification becomes possible by learning a large amount of images according to a problem.

According to Patent Document 1, even when the number of images for learning is small, as a method of generating a highly accurate image recognition model, pre-learning is performed using an image group similar to the characteristics of the image group to be learned, and then. , A method of main learning with a group of images to be learned has been proposed. Specifically, pre-learning is performed by an image group in which the shape of the subject is similar, an image group in which a biological phantom is imaged, an image group in which the tissue structure of the subject is similar, an image group in which a mimic organ is imaged by the same imaging system, and the like. After that, a method of main learning with the image group to be learned has been proposed.

Further, in Patent Document 2, the first learning is performed by the first image group imaged at the first frame rate, and then by the second image group imaged at a second frame rate lower than the first frame rate. A method of performing the second learning has been proposed.

International Publication No. 2017/22 14212 International Publication No. 2017/175282

By the way, in machine learning, if the image quality of the image prepared for learning is biased, the bias in the image quality is also learned. Therefore, there is a problem that the accuracy of recognition is lowered when the image of the image quality deviating from the bias of the image quality of the learned image group is recognized. Since the learning methods of Patent Documents 1 and 2 do not perform learning in consideration of image quality, there is a drawback that the recognition accuracy is lowered when an image having an image quality different from the image quality of the learned image group is recognized.

On the other hand, in a device such as an endoscope that captures a medical image, there may be a difference in image quality depending on the model. Therefore, in order to generate a highly accurate image recognition model, it is necessary to optimize learning for each model.

However, generating a zero-based image recognition model for each model has a problem that a great deal of labor and cost are required.

The present invention has been made in view of such circumstances, and provides a learning device and a learning method capable of efficiently generating a model for performing image recognition on a medical image having a specific image quality, and a medical image processing device. The purpose is to do.

The means for solving the above problems are as follows.

(1) With a first learning unit that generates a first model that performs image recognition on a first-quality medical image by learning using a first medical image group composed of a first-quality medical image. , Based on the first model, image recognition is performed on the second image quality medical image by learning using the second medical image group composed of the second image quality medical image different from the first image quality. A learning device including a second learning unit that generates two models.

According to this aspect, first, the first model is generated by performing the first learning in the first medical image group of the first image quality. Then, based on the first model, the second model is generated by performing the second learning in the second medical image group of the second image quality. Since the second learning is performed based on the first learning result, it is possible to generate an accurate model even when the number of images for training is small. Therefore, when generating a model for performing image recognition on a medical image having a specific image quality, after performing the first learning on the first medical image group having abundant images for learning, medical treatment with the desired image quality is performed. By performing the second learning in the image group (second medical image group), it is possible to efficiently generate a model that performs image recognition for a medical image having a desired image quality.

The images constituting the medical image include both moving images and still images. A moving image can be regarded as a time-series image group including a plurality of frames. Further, the image quality here means the image quality of the images constituting one frame in the moving image.

(2) The learning device according to (1) above, wherein the first medical image group is composed of medical images having a first resolution, and the second medical image group is composed of medical images having a second resolution different from the first resolution.

According to this aspect, the first medical image group is composed of the first resolution medical image, and the second medical image group is composed of the second resolution medical image different from the first resolution. As a result, when a model for image recognition is generated for a medical image having a specific resolution, an accurate model can be efficiently generated even when the number of learning images having the corresponding resolution is small. ..

(3) The learning device according to (2) above, wherein the second resolution is lower than the first resolution.

According to this aspect, the second medical image group is composed of medical images having a resolution (second resolution) lower than the resolution (first resolution) of the medical images constituting the first medical image group. As a result, when a model for image recognition is generated for a medical image having a specific resolution, an accurate model can be efficiently generated even when the number of learning images having the corresponding resolution is small.

(4) The learning device according to (3) above, wherein the first medical image group is composed of medical images having a resolution of 4K or more, and the second medical image group is composed of medical images having a resolution of less than 4K.

According to this aspect, the first medical image group is composed of medical images having a resolution of 4K or more, and the second medical image group is composed of medical images having a resolution of less than 4K. As a result, when generating a model that performs image recognition for a medical image having a specific resolution of less than 4K, an accurate model can be efficiently produced even when the number of learning images having a corresponding resolution is small. Can be generated.

The 4K image is a high-definition image having about 4000 pixels on the long side. In particular, it refers to an image having a horizontal × vertical number of pixels of about 4000 × 2000. The commonly known "4K UHDTV (Ultra High Definition Television)" and "DCI 4K" are included in "4K" herein. "4K UHDTV" is 4K defined by the International Telecommunication Union (ITU), and is 4K with 3840 horizontal pixels and 2160 vertical pixels. "DCI 4K" is 4K defined by Digital Cinema Initiatives (DCI), to which movie companies and the like are affiliated, and is an image of 4096 horizontal x 2160 vertical pixels.

(5) The learning device according to (3) above, wherein the first medical image group is composed of medical images having a resolution of 8K or more, and the second medical image group is composed of medical images having a resolution of less than 8K.

According to this aspect, the first medical image group is composed of medical images having a resolution of 8K or more, and the second medical image group is composed of medical images having a resolution of less than 8K. As a result, when generating a model that performs image recognition for a medical image having a specific resolution of less than 8K, an accurate model can be efficiently produced even when the number of learning images having a corresponding resolution is small. Can be generated.

The 8K image is a high-definition image having about 8000 pixels on the long side. In particular, it refers to an image having a horizontal × vertical number of pixels of about 8000 × 4000. The commonly known "8K UHDTV" (also referred to as 8K Ultra-high-definition television, 8K Ultra HDTV, 8K UHDTV, 8K UHD, Super Hi-Vision 8K, etc.) is included in "8K" herein. "8K UHDTV" is 8K defined by the International Telecommunication Union, and is an image of 7680 horizontal × 4320 vertical pixels.

(6) The learning device according to (2) above, wherein the second resolution is higher than the first resolution.

According to this aspect, the second medical image group is composed of medical images having a resolution (second resolution) higher than the resolution (first resolution) of the medical images constituting the first medical image group. As a result, when a model for image recognition is generated for a medical image having a specific resolution, an accurate model can be efficiently generated even when the number of learning images having the corresponding resolution is small.

(7) The learning device according to (6) above, wherein the first medical image group is composed of medical images having a resolution of less than 4K, and the second medical image group is composed of medical images having a resolution of 4K or more.

According to this aspect, the first medical image group is composed of medical images having a resolution of less than 4K, and the second medical image group is composed of medical images having a resolution of 4K or more. As a result, when generating a model that performs image recognition for a medical image having a specific resolution of 4K or higher, an accurate model can be efficiently produced even when the number of learning images having the corresponding resolution is small. Can be generated.

(8) The learning apparatus according to (6) above, wherein the first medical image group is composed of medical images having a resolution of less than 8K, and the second medical image group is composed of medical images having a resolution of 8K or more.

According to this aspect, the first medical image group is composed of medical images having a resolution of less than 8K, and the second medical image group is composed of medical images having a resolution of 8K or more. As a result, when generating a model that performs image recognition for a medical image having a specific resolution of 8K or higher, an accurate model can be efficiently produced even when the number of learning images having the corresponding resolution is small. Can be generated.

(9) The learning device according to (1) above, wherein the first medical image group is composed of medical images having a smaller amount of noise than the medical images constituting the second medical image group.

According to this aspect, the first medical image group is composed of medical images having a smaller amount of noise than the medical images constituting the second medical image group. As a result, when generating a model that performs image recognition for a medical image with a specific noise amount, an accurate model can be efficiently generated even when the number of learning images having a corresponding noise amount is small. it can.

(10) The learning device according to (1) above, wherein the first medical image group is composed of medical images having a larger amount of noise than the medical images constituting the second medical image group.

According to this aspect, the first medical image group is composed of medical images having a larger amount of noise than the medical images constituting the second medical image group. As a result, when generating a model that performs image recognition for a medical image with a specific noise amount, an accurate model can be efficiently generated even when the number of learning images having a corresponding noise amount is small. it can.

(11) The learning device according to (1) above, wherein the first medical image group is composed of medical images having a wider angle of view than the medical images constituting the second medical image group.

According to this aspect, the first medical image group is composed of medical images having a wider angle of view than the medical images constituting the second medical image group. This makes it possible to generate a model that performs image recognition on a medical image having a specific angle of view without learning from the image that has been cut out or the like.

(12) The first medical image group is composed of medical images taken by an endoscope, and the second medical image group is a medical image taken by an endoscope different from the endoscope that took the first medical image group. The learning device of (1) above, which is composed of.

According to this aspect, the first medical image group is composed of medical images taken by an endoscope, and the second medical image group is an endoscope different from the endoscope that took the first medical image group. It consists of medical images taken. As a result, when generating a model that performs image recognition on a medical image taken by a specific endoscope, the accuracy is high even when the number of learning images taken by the endoscope is small. A good model can be generated efficiently.

(13) The learning device according to (12) above, wherein the second medical image group is composed of medical images taken by an endoscope having specifications different from those of the endoscope that captured the medical images constituting the first medical image group. ..

According to this aspect, the second medical image group is composed of medical images taken by an endoscope having specifications different from those of the endoscope that captured the medical images constituting the first medical image group. For example, medical examinations taken with endoscopes with different sizes of mounted image sensors, number of pixels, etc., endoscopes with different focal lengths of the mounted optical systems, endoscopes with different amounts of generated noise, etc. Consists of images.

(14) The learning device according to any one of (1) to (13) above, wherein the first model and the second model are composed of a convolutional neural network.

According to this aspect, the first model and the second model are composed of a convolutional neural network.

(15) A model that is composed of a medical image acquisition unit that acquires a medical image and a second model generated by any one of the learning devices (1) to (14) above, and performs image recognition on the medical image. And, a medical image processing device equipped with.

According to this aspect, the model for performing image recognition on the medical image is composed of the second model generated by the learning device according to any one of (1) to (14) above. As a result, image recognition can be performed accurately on a medical image having a specific image quality.

(16) The medical image processing apparatus according to (15) above, further comprising a plurality of models and a model switching unit for switching a model to be used.

According to this aspect, a plurality of models for performing image recognition are provided and used by switching. As a result, image recognition can be performed using an appropriate model according to the image quality.

(17) An endoscope information acquisition unit for acquiring information on an endoscope that has taken a medical image is further provided, and a plurality of models use a second medical image group taken by different endoscopes for a second. The medical image processing apparatus according to (16) above, which is generated by learning in the learning unit, and the model switching unit switches the model to be used based on the endoscopic information acquired by the endoscope information acquisition unit.

According to this aspect, a plurality of models for performing image recognition are provided. Each model is generated by learning in the second learning unit using a second medical image group taken by different endoscopes, and is used depending on the endoscope in which the medical image to be recognized is taken. The model to be used is automatically switched.

(18) The medical image processing apparatus according to (17) above, wherein the plurality of models are generated by learning in the second learning unit using a second medical image group photographed by endoscopes having different specifications.

According to this aspect, a plurality of models are generated by learning in the second learning unit using a second medical image group photographed by endoscopes having different specifications from each other.

(19) The medical image according to (18) above, wherein the plurality of models are generated by learning in the second learning unit using a second medical image group taken by endoscopes having different resolutions or noise amounts from each other. Processing equipment.

According to this aspect, a plurality of models are generated by learning in the second learning unit using a second medical image group captured by endoscopes having different resolutions or noise amounts from each other.

(20) A step of generating a first model for performing image recognition on a first-quality medical image by learning using a first medical image group composed of a first-quality medical image, and a first step. A second model that recognizes a second image quality medical image by learning using a second medical image group composed of a second image quality medical image different from the first image quality based on the model. A learning method with steps to generate.

According to this aspect, first, the first model is generated by performing the first learning in the first medical image group of the first image quality. Then, based on the first model, the second model is generated by performing the second learning in the second medical image group of the second image quality. Since the second learning is performed based on the first learning result, it is possible to generate an accurate model even when the number of images for training is small. Therefore, when generating a model for performing image recognition on a medical image having a specific image quality, after performing the first learning on the first medical image group having abundant images for learning, medical treatment with the desired image quality is performed. By performing the second learning in the image group (second medical image group), it is possible to efficiently generate a model that performs image recognition for a medical image having a desired image quality.

According to the present invention, it is possible to efficiently generate a model that performs image recognition on a medical image having a specific image quality.

A block diagram showing an embodiment of a configuration of a learning device Schematic diagram showing an example of the configuration of CNN Conceptual diagram of CNN settings constituting the first model and the second model Diagram showing an example of the hardware configuration of the learning device Flowchart showing the learning procedure performed by the learning device A flowchart showing a learning procedure based on the learning result of a high-resolution learning image group. A flowchart showing a learning procedure based on the learning result of a low-resolution learning image group. Flowchart showing the learning procedure based on the learning result by the low noise learning image group A flowchart showing a learning procedure based on the learning result of a group of high-noise learning images. A flowchart showing a learning procedure based on the learning result of a wide-angle learning image group. A flowchart showing a learning procedure based on the learning results of learning images taken with different endoscopes. A block diagram showing an embodiment of the configuration of an endoscopic image processing device. The figure which shows an example of the hardware composition of an endoscopic image processing apparatus. Block diagram showing a modified example of an endoscopic image processing device Block diagram showing other modifications of the endoscopic image processing device

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Configuration of learning device]
FIG. 1 is a block diagram showing an embodiment of the configuration of the learning device.

The learning device 1 of the present embodiment is configured as a device that generates a model for performing image recognition on an endoscopic image obtained by endoscopy by machine learning. The endoscopic image is an example of a medical image. In particular, the learning device 1 of the present embodiment is configured as a device that generates a model for performing image recognition on an endoscopic image having a specific image quality by machine learning. The image recognition performed here is, for example, detection of a lesion included in an image, classification by type of lesion, and the like.

As shown in FIG. 1, the learning device 1 of the present embodiment has a first learning unit 10 and a first learning unit 10 that generate a first model M1 that performs image recognition on an endoscopic image of the first image quality by machine learning. Based on the first model M1 generated by the first learning unit 10, the second learning unit 20 that generates the second model M2 that performs image recognition on the endoscopic image of the second image quality, and the entire learning device 1 It is provided with a learning control unit 30 that comprehensively controls the operation of the above. Further, a first learning data set 12 for learning by the first learning unit 10 and a second learning data set 22 for learning by the second learning unit 20 are provided.

The first learning unit 10 generates a first model M1 that performs image recognition on an endoscope image of the first image quality by learning using the first learning data set 12. The first model M1 is composed of, for example, a convolutional neural network (CNN). The first learning unit 10 optimizes the weight parameter of each layer of the CNN constituting the first model M1 by learning.

The first learning data set 12 is composed of a first endoscopic image group which is a learning image group. The first endoscopic image group is an example of the first medical image group, and is composed of an endoscopic image of the first image quality.

The images constituting the endoscopic image include both still images and moving images. A moving image can be regarded as a time-series image group including a plurality of frames. The data constituting the image is data having each intensity value (luminance value) of red (Red, R), green (Green, G), and blue (Blue, B) in pixel units. Further, the image quality means the image quality of the images constituting one frame in the case of moving images.

The second learning unit 20 learns using the second learning data set 22 based on the first model M1 generated by the first learning unit 10 to obtain an endoscopic image of the second image quality. A second model M2 for image recognition is generated. The second model M2 is composed of, for example, CNN. The second model M2 generated by the second learning unit 20 is a model that performs image recognition on an endoscopic image having a specific image quality. That is, the image recognition process is performed using the second model M2 as the trained model. The second learning unit 20 optimizes the weight parameters of each layer of the CNN constituting the second model M2 by learning.

The second learning data set 22 is composed of a second endoscopic image group which is a learning image group. The second endoscopic image group is an example of the second medical image group, and is composed of an endoscopic image having a second image quality different from the first image quality. The image quality (second image quality) of the endoscope images constituting the second endoscope image group is the same as the image quality (including the same image quality) as the object to be image-recognized.

On the other hand, the first learning data set 12 is composed of an endoscopic image having an image quality different from that of the second image quality (first image quality). Therefore, the image quality of the endoscopic image for which image recognition is performed is composed of an endoscopic image having a different image quality. Specifically, it is composed of a higher quality or lower quality endoscopic image.

The learning control unit 30 controls the operations of the first learning unit 10 and the second learning unit 20, and controls the overall operation of the learning device 1. Further, the learning control unit 30 sets the CNNs constituting the first model M1 and the second model M2.

FIG. 2 is a schematic diagram showing an example of the configuration of CNN.

As shown in the figure, the CNN is composed of a multi-layer neural network composed of a convolutional layer, a normalization layer, a pooling layer, and the like.

FIG. 3 is a conceptual diagram of CNN settings constituting the first model and the second model. In the figure, (A) is a diagram showing an example of a CNN constituting the first model M1, and (B) is a diagram showing an example of a CNN constituting the second model M2.

The learning control unit 30 sets the CNN that constitutes the first model M1, and sets the CNN that constitutes the second model M2 based on the learned first model M1. In the present embodiment, the CNN of the second model M2 is set by resetting the weight parameters of some layers of the CNN constituting the trained first model M1. The layers that reset the weight parameters are some layers that are close to the output. In the example shown in FIG. 3, the weight parameters of the final three layers (fully connected layer, fully connected layer and Softmax layer) surrounded by the broken line BL are reset to set the CNN of the second model M2. In this case, the weight parameter of the trained first model M1 is set as an initial value for the remaining layers surrounded by the solid line SL.

[Hardware configuration of learning device]
FIG. 4 is a diagram showing an example of the hardware configuration of the learning device.

The learning device 1 is composed of computers such as a server computer and a client computer, and includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, a RAM (Random Access Memory) 53, an HDD (Hard Disk Drive) 54, and communication. It includes an interface 55, an input / output interface 56, and the like. Further, the learning device 1 includes an input device 57, a display device 58, and the like.

By executing the program, the CPU 51 controls each part of the learning device 1 and realizes each function of the learning device 1. The ROM 52 stores various programs executed by the CPU 51, various data, and the like. The RAM 53 provides the CPU 51 with a work area. The HDD 54 stores various programs and various data executed by the CPU 51. The communication interface 55 is an interface (interface; I / F) for connecting the learning device 1 to a network 59 such as a LAN (Local Area Network). The learning device 1 communicates with an external device via the communication interface 55. The input / output interface 56 is an interface for connecting an external device such as an input device 57 and a display device 58 to the learning device 1. The input device 57 inputs information according to the operation by the user to the learning device 1. The input device 57 is composed of, for example, a keyboard, a mouse, and the like. The display device 58 displays various information. The display device 58 is composed of, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or the like.

Each function of the first learning unit 10, the second learning unit 20, and the learning control unit 30 constituting the learning device 1 is realized by the CPU 51 executing a predetermined program. Further, the first learning data set 12 and the second learning data set 22 are stored in the HDD 54.

[Learning method]
<< Basic procedure of learning >>
FIG. 5 is a flowchart showing a learning procedure performed by the learning device.

First, the CNN constituting the first model M1 is set (step S1).

Next, the first learning is performed on the set CNN using the first learning data set 12 (step S2). That is, learning is performed with the first endoscopic image group composed of the endoscopic images of the first image quality. As a result, the first model M1 that performs image recognition on the endoscopic image of the first image quality is generated.

Next, based on the learned first model M1, the CNN constituting the second model M2 is set (step S3). As described above, in the present embodiment, the weight parameters of some layers close to the output of the trained first model M1 are reset to set the CNN of the second model M2 (see FIG. 3).

Next, the second training is performed on the set CNN using the second training data set 22 (step S4). That is, learning is performed with the second endoscope image group composed of the endoscope images of the second image quality. As a result, the second model M2 that performs image recognition on the endoscopic image of the second image quality is generated.

Here, the second endoscopic image group used in the second learning has the same image quality (similar level) as the image quality of the endoscopic image (endoscopic image having a specific image quality) to be image-recognized. Consists of endoscopic images (including image quality). As a result, the second model M2 generated in the second learning becomes a model capable of image recognition for an endoscope image having the same image quality as the endoscope image to be image-recognized.

As described above, in the learning device 1 of the present embodiment, after performing the first learning with the endoscope image group of the first image quality, the endoscope of the second image quality is based on the first learning result. The second learning is performed on the image group. Since the second learning is performed based on the first learning result, it is possible to generate an accurate model even when the number of images for training is small. Therefore, for example, in the case of generating a model that performs image recognition on a medical image having a specific image quality, after performing the first learning with abundant image quality of the image for learning, the medical image group of the desired image quality The second learning is done at. As a result, the target model can be efficiently generated.

"Example"
<Learning with learning images with different resolutions>
In the case of generating a model that performs image recognition on an endoscopic image having a specific resolution, the first endoscopic image group having a resolution (first resolution) different from that of the endoscopic image that performs image recognition is used. Based on the result, the second learning is performed on the endoscopic image group having the same resolution (second resolution) as the endoscopic image for image recognition. In this case, (1) the first learning is performed using an endoscopic image group having a higher resolution (first resolution) than the endoscopic image for which image recognition is performed, and image recognition is performed based on the result. A method of performing the second learning with an endoscopic image group having the same resolution as the endoscopic image (second resolution), and (2) endoscopy having a lower resolution (first resolution) than the endoscopic image performing image recognition. A method of performing the first learning using the mirror image group, and based on the result, performing the second learning with the endoscopic image group having the same resolution (second resolution) as the endoscope image for image recognition. , There is. Hereinafter, the cases (1) and (2) will be described separately.

(1) Learning based on the learning result of a high-resolution learning image group When generating a model that performs image recognition on an endoscopic image having a specific resolution, the endoscopic image that performs image recognition Within the same resolution (including similar resolution) as the resolution of the endoscope image for which image recognition is performed based on the result of the first learning performed using the endoscopic image group having a resolution higher than the resolution. The second learning is performed on the spectroscopic image group.

FIG. 6 is a flowchart showing a learning procedure based on the learning result of the high-resolution learning image group.

First, the CNN constituting the first model M1 is set (step S11).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S12). The first learning data set 12 is composed of an endoscopic image having a resolution relatively higher than the resolution of the endoscopic image to be image-recognized. By this first learning, a first model M1 that performs image recognition on an endoscopic image having a relatively high resolution is generated.

Next, based on the learned first model M1, the CNNs constituting the second model M2 are set (step S13).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S14). The second learning data set 22 is composed of an endoscope image having the same resolution (including a similar resolution) as the resolution of the endoscope image to be image-recognized. By this second learning, a model (second model M2) capable of image recognition for an endoscopic image having a target resolution is generated.

For example, when generating a model that performs image recognition for an endoscopic image having a resolution of less than 4K, an endoscopic image group having a resolution of 4K or more (for example, an endoscopic image group having a 4K resolution or an 8K resolution) After that, based on the first learning result, it is conceivable to perform the second learning with the endoscopic image group of the target resolution (for example, 2K resolution, etc.). For example, when generating a model that performs image recognition on an endoscopic image having a 2K resolution, the first learning is performed on the endoscopic image group having a 4K resolution, and then 2K is performed based on the first learning result. The second learning is performed on the endoscopic image group of the resolution. Thereby, for example, even when the endoscopic image for learning at 2K resolution is insufficient, the target model can be efficiently generated at no cost.

The 2K image is a high-definition image having about 2000 pixels on the long side. In particular, it refers to an image having a horizontal × vertical number of pixels of about 2000 × 1000. Therefore, general full high-definition (width 1920 x height 1080) and the like are included in 2K here.

Further, for example, when generating a model for image recognition for an endoscopic image having a resolution of less than 8K, an endoscopic image group having a resolution of 8K or more (for example, an endoscopic image group having an 8K resolution) is used. It is conceivable to perform the first learning and then perform the second learning on the endoscopic image group of the target resolution (for example, 2K resolution or 4K resolution) based on the first learning result. For example, when generating a model that performs image recognition on an endoscopic image having a 4K resolution, the first learning is performed on the endoscopic image group having an 8K resolution, and then 4K is performed based on the first learning result. The second learning is performed on the endoscopic image group of the resolution. As a result, for example, even when there is a shortage of endoscopic images for learning at 4K resolution, it is possible to efficiently generate a target model at no cost.

In general, large hospitals such as university hospitals use relatively high-resolution endoscopes (for example, oral endoscopes), and small hospitals such as clinics use relatively low-resolution endoscopes (for example, oral endoscopes). For example, a nasal endoscope) is used. In addition, since the number of examinations is smaller in a small hospital than in a large hospital, there is a problem that it is difficult to collect images for learning. Therefore, the learning image of the target resolution may be insufficient.

According to the learning method of this aspect, even when generating a model for image recognition for an endoscope image taken by a low-resolution endoscope used in a small hospital or the like, there are few cases. The model can be optimized at cost.

(2) Learning based on the learning result of a low-resolution learning image group When generating a model that performs image recognition on an endoscopic image having a specific resolution, the endoscopic image that performs image recognition Within the same resolution (including similar resolution) as the resolution of the endoscopic image for image recognition based on the result of the first learning using the endoscopic image group with a resolution lower than the resolution. The second learning is performed on the spectroscopic image group.

FIG. 7 is a flowchart showing a learning procedure based on the learning result of the low-resolution learning image group.

First, the CNN constituting the first model M1 is set (step S21).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S22). The first learning data set 12 is composed of an endoscopic image having a resolution relatively lower than the resolution of the endoscopic image to be image-recognized. By this first learning, a first model M1 that performs image recognition on an endoscopic image having a relatively low resolution is generated.

Next, based on the learned first model M1, the CNNs constituting the second model M2 are set (step S23).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S24). The second learning data set 22 is composed of an endoscope image having the same resolution (including a similar resolution) as the resolution of the endoscope image to be image-recognized. By this second learning, a model (second model M2) capable of image recognition for an endoscopic image having a target resolution is generated.

For example, when generating a model that performs image recognition for an endoscopic image having a resolution of 4K or higher, the first endoscopic image group having a resolution of less than 4K (for example, an endoscopic image group having a 2K resolution) is the first. After that, based on the first learning result, it is conceivable to perform the second learning with an endoscopic image group having a target resolution (for example, 4K resolution). For example, when generating a model that performs image recognition on an endoscopic image having a 4K resolution, the first learning is performed on the endoscopic image group having a 2K resolution, and then 4K is performed based on the first learning result. The second learning is performed on the endoscopic image group of the resolution. As a result, for example, even when there is a shortage of endoscopic images for learning at 4K resolution, it is possible to efficiently generate a target model at no cost.

Further, for example, when generating a model that performs image recognition for an endoscopic image having a resolution of 8K or higher, an endoscopic image group having a resolution of less than 8K (for example, an endoscopic image group having a 4K resolution or a 2K resolution). ), And then, based on the first learning result, it is conceivable to perform the second learning with an endoscopic image group having a target resolution (for example, 8K resolution). For example, when generating a model that performs image recognition on an endoscopic image having an 8K resolution, the first learning is performed on the endoscopic image group having a 4K resolution, and then 8K is performed based on the first learning result. The second learning is performed on the endoscopic image group of the resolution. Thereby, for example, even when the endoscopic image for learning at 8K resolution is insufficient, the target model can be efficiently generated at no cost.

With the development of endoscopes, it is expected that the resolution of images will be further increased in the future. In that case, it is conceivable that there will be a shortage of high-resolution images to be used for learning.

According to the learning method of this aspect, even when the learning image of the target resolution is insufficient, the model of the target resolution can be efficiently generated by using the existing learning image group.

<Learning with learning image groups with different amounts of noise>
When generating a model for image recognition for an endoscopic image with a specific amount of noise, the first learning is performed using an endoscopic image group with a different amount of noise from the endoscopic image for image recognition. Based on the result, the second learning is performed on the endoscope image group having the same amount of noise (including the same amount of noise) as the noise amount of the endoscope image for which image recognition is performed. In this case, (1) the first learning is performed using an endoscopic image group having a smaller amount of noise than the endoscopic image for which image recognition is performed, and based on the result, the endoscopic image for which image recognition is performed is performed. A method of performing the second learning with an endoscopic image group having the same amount of noise as the amount of noise (including a similar amount of noise), and (2) endoscopy having a larger amount of noise than the endoscopic image performing image recognition. The first learning is performed using the mirror image group, and based on the result, the endoscopic image group has the same amount of noise (including the same amount of noise) as the noise amount of the endoscope image for which image recognition is performed. There is a method of performing the second learning. Hereinafter, the cases (1) and (2) will be described separately.

(1) Learning based on the learning result by the low noise learning image group When generating a model that performs image recognition on an endoscopic image with a specific amount of noise, from the endoscopic image that performs image recognition The first learning is performed using the low-noise endoscopic image group, and based on the result, the noise amount is the same as the noise amount of the endoscopic image for image recognition (including the same amount of noise). The second learning is performed on the endoscopic image group.

FIG. 8 is a flowchart showing a learning procedure based on the learning result of the low noise learning image group.

First, the CNN constituting the first model M1 is set (step S31).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S32). The first learning data set 12 is composed of an endoscope image having a noise amount relatively smaller than the noise amount of the endoscope image to be image-recognized. For example, when generating a model that performs image recognition on an endoscope image taken by a low-end endoscope, the endoscope image taken by a low-end high-end endoscope is used for the first learning. Configure the data set 12. By this first learning, a first model M1 that performs image recognition on an endoscopic image having relatively low noise is generated.

Next, based on the learned first model M1, the CNNs constituting the second model M2 are set (step S33).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S34). The second learning data set 22 is composed of an endoscope image having the same amount of noise (including the same amount of noise) as the amount of noise of the endoscope image to be image-recognized. For example, when generating a model that performs image recognition on an endoscope image taken by a low-end endoscope, the second learning data set 22 is obtained from the endoscope image taken by the low-end endoscope. To configure. By this second learning, a model (second model M2) capable of image recognition for an endoscopic image having a target noise amount is generated.

In general, large hospitals such as university hospitals use endoscopes with a relatively low amount of noise (so-called high-end endoscopes), and small hospitals such as clinics have a larger amount of noise than that. An endoscope is used. In addition, since the number of examinations is smaller in a small hospital than in a large hospital, there is a problem that it is difficult to collect images for learning. Therefore, the learning image of the target noise amount may be insufficient.

According to the learning method of this aspect, a model for image recognition is generated for an endoscope image taken by an endoscope (endoscope with a relatively large amount of noise) used in a small hospital or the like. Even if you do, you can optimize your model at low cost.

(2) Learning based on the learning result of the high-noise learning image group When generating a model that performs image recognition on an endoscopic image with a specific amount of noise, from the endoscopic image that performs image recognition The first learning is performed using the high-noise endoscopic image group, and based on the result, the noise amount is the same as the noise amount of the endoscopic image for image recognition (including the same amount of noise). The second learning is performed on the endoscopic image group.

FIG. 9 is a flowchart showing a learning procedure based on the learning result of the high noise learning image group.

First, the CNN constituting the first model M1 is set (step S41).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S42). The first learning data set 12 is composed of an endoscope image having a noise amount relatively larger than the noise amount of the endoscope image to be image-recognized. By this first learning, a first model M1 that performs image recognition on an endoscopic image having relatively high noise is generated.

Next, based on the learned first model M1, the CNNs constituting the second model M2 are set (step S43).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S44). The second learning data set 22 is composed of an endoscopic image having the same amount of noise (including a similar amount of noise) as the amount of noise of the endoscopic image to be image-recognized. By this second learning, a model (second model M2) capable of image recognition for an endoscopic image having a target noise amount is generated.

With the development of endoscopes, it is expected that the noise of images will be further reduced in the future. In that case, it is conceivable that there will be a shortage of low-noise images to be used for learning.

According to the learning method of this aspect, even when the learning image of the target noise amount is insufficient, the model of the target noise amount is efficiently generated by using the existing learning image group. it can.

<Learning based on learning results using wide-angle learning image groups>
When generating a model for image recognition for an endoscopic image with a specific angle of view, the first learning is performed using an endoscopic image group with a wider angle of view than the endoscopic image for image recognition. Based on the result, the second learning is performed with an endoscopic image group having the same angle of view (including substantially the same angle of view) as the angle of view of the endoscopic image for which image recognition is performed.

FIG. 10 is a flowchart showing a learning procedure based on the learning result of the wide-angle learning image group.

First, the CNN constituting the first model M1 is set (step S51).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S52). The first learning data set 12 is composed of an endoscope image having an angle of view relatively wider than the angle of view of the endoscope image to be image-recognized. By this first learning, the first model M1 that performs image recognition on an endoscopic image having a wider angle than the target endoscopic image is generated.

Next, based on the learned first model M1, the CNNs constituting the second model M2 are set (step S53).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S54). The second learning data set 22 is composed of an endoscope image having the same angle of view (including substantially the same angle of view) as the angle of view of the endoscope image to be image-recognized. This second learning produces an image recognition model (second model M2) optimized for use in a particular endoscope.

In this way, by learning with the learning image group of the target angle of view based on the learning result with the wide-angle learning image group, it is not necessary to perform re-learning with the cropped image, and more. Learning can be done at low cost.

<Learning based on the learning results of learning images taken with other endoscopes>
In the case of generating a model that performs image recognition on an endoscope image taken by a specific endoscope, the first learning is performed using an endoscope image group taken by another endoscope. Based on the result, the second learning is performed with the endoscope image group taken by the endoscope that performs image recognition.

FIG. 11 is a flowchart showing a learning procedure based on the learning results of the learning image groups taken by different endoscopes.

First, the CNN constituting the first model M1 is set (step S61).

Next, the set CNN is subjected to the first learning using the first learning data set 12 (step S62). The first learning data set 12 is composed of an endoscope image taken by an endoscope (another endoscope) different from the endoscope that performs image recognition. For example, an endoscope taken with an endoscope having different specifications (image sensor size, image sensor resolution, image sensor type, imaging optical system configuration, light source type, etc.) from an endoscope that performs image recognition. Consists of. By this first learning, the first model M1 that performs image recognition on the endoscope image taken by the other endoscope is generated.

Next, based on the learned first model M1, the CNN constituting the second model M2 is set (step S63).

Next, the set CNN is subjected to the second learning using the second learning data set 22 (step S64). The second learning data set 22 is composed of an endoscope image taken by the same endoscope (including an endoscope of the same model and the same specifications) as the endoscope performing image recognition. By this second learning, a model (second model M2) capable of image recognition for a target endoscopic image is generated.

According to the learning method of this aspect, even when the learning image of the endoscope to be image-recognized is insufficient, the learning image group of other endoscopes which is abundant can be used. It is possible to efficiently generate a model that recognizes an endoscopic image taken by a specific endoscope.

In addition, there may be individual differences in the endoscopes even if they are of the same model. According to the learning method of this aspect, an image recognition model of a specific endoscope can be efficiently generated even when there are individual differences.

[Modification of learning device]
<< Modification example of hardware configuration of learning device >>
In the above embodiment, the functions of the first learning unit 10 and the second learning unit 20 are realized by the same computer, but the functions may be realized by a plurality of computers. For example, the functions of the first learning unit 10 and the second learning unit 20 can be realized by separate computers.

<< Configuration of 1st model and 2nd model >>
In the above embodiment, the model for image recognition is configured by CNN, but the configuration of the model for image recognition is not limited to this. Any model generated by machine learning will do.

<< Setting of the second model >>
In the above embodiment, the CNN of the second model M2 is set by resetting the weight parameters of some layers of the CNN constituting the trained first model M1, but the second model M2 is set. The method is not limited to this. For example, a method of re-learning the entire CNN with the weight parameter of the trained first model M1 as an initial value, and a method of replacing the input layer and the output layer of the trained first model M1 to perform the second learning. , Various methods such as a method of fixing the weight parameter of a part of the trained first model M1 (for example, a layer for feature extraction) and learning only another layer (for example, a layer for recognition). Can be adopted.

Further, when the weight parameters of some layers of the trained first model M1 are reset and the second learning is performed as in the above embodiment, the learning coefficient may be changed in each layer. For example, in the layer in which the weight parameter is reset, the learning coefficient may be set larger than that in the other layers so that the learning proceeds faster, and the second learning may be performed.

A so-called transfer learning (also referred to as fine tuning) method can be adopted for the second learning including the setting of the second model.

[Endoscopic image processing device]
FIG. 12 is a block diagram showing an embodiment of the configuration of the endoscopic image processing apparatus.

The endoscopic image processing device 100 is an example of a medical image processing device. The endoscopic image processing device 100 acquires an endoscopic image having a specific image quality and recognizes the acquired endoscopic image (detection of lesions contained in the image, classification by type of lesion, etc.). And output the result. For image recognition, the image recognition model generated by the learning device 1 is used.

As shown in FIG. 12, the endoscope image processing device 100 includes an endoscope image acquisition unit 110 that acquires an endoscope image to be recognized, and an image recognition unit that performs image recognition on the acquired endoscope image. It includes 112, a recognition result output unit 114 that outputs a recognition result, and an image processing control unit 116 that controls the whole.

The endoscopic image acquisition unit 110 is an example of a medical image acquisition unit, and acquires an endoscopic image (medical image) to be recognized. This endoscopic image is an endoscopic image having a specific image quality.

The image recognition unit 112 performs image recognition (detection of lesions included in the image, classification by type of lesion, etc.) on the endoscopic image acquired by the endoscopic image acquisition unit 110. The image recognition unit 112 is composed of an image recognition model (learned model) generated by the learning device 1. Therefore, the first learning is performed with the learning image group (first endoscopic image group) having an image quality different from the target image quality, and based on the learning result, the learning image group (second) with the target image quality is performed. It is composed of a model (second model) generated by learning with an endoscopic image group).

The recognition result output unit 114 outputs the recognition result by the image recognition unit 112 in a predetermined format. For example, it outputs to a monitor in a predetermined display format.

The image processing control unit 116 controls the operation of each unit in an integrated manner.

[Hardware configuration of endoscopic image processing device]
FIG. 13 is a diagram showing an example of the hardware configuration of the endoscopic image processing device.

The endoscopic image processing device 100 is composed of computers such as a server computer and a client computer, and includes a CPU 121, a ROM 122, a RAM 123, an HDD 124, a communication interface 125, an input / output interface 126, and the like. Further, the learning device 1 includes an input device 127, a display device 128, and the like.

By executing the program, the CPU 121 controls each part of the endoscopic image processing device 100 and realizes each function of the endoscopic image processing device 100. The ROM 122 stores various programs executed by the CPU 121, various data, and the like. The RAM 123 provides a work area for the CPU 121. The HDD 124 stores various programs and various data executed by the CPU 121. The communication interface 125 is an interface for connecting the endoscopic image processing device 100 to a network 59 such as a LAN. The endoscopic image processing device 100 communicates with an external device via the communication interface 125. The input / output interface 126 is an interface for connecting an external device such as an input device 127 and a display device 128 to the endoscopic image processing device 100. The input device 127 inputs information according to the operation by the user to the endoscopic image processing device 100. The input device 127 includes, for example, a keyboard, a mouse, and the like. The display device 128 displays various information. The display device 128 is composed of, for example, a liquid crystal display, an organic EL display, or the like.

Each function of the endoscope image acquisition unit 110, the image recognition unit 112, and the recognition result output unit 114 is realized by the CPU 121 executing a predetermined program.

The endoscopic image to be recognized is stored in, for example, HDD 124 and acquired from HDD 124. Alternatively, it is stored in an external storage device connected via the network 59, and is acquired from the external storage device via the network 59. Alternatively, it is obtained from an endoscope device connected via the network 59 via the network 59. The endoscope image acquisition unit 110 acquires an endoscope image to be recognized from a designated acquisition destination under the control of the image processing control unit 116.

The recognition result is displayed on the display device 128, for example, in a predetermined display format. The recognition result output unit 114 outputs the recognition result of the image recognition unit 112 to the display device 128 in a predetermined format under the control of the image processing control unit 116.

[Image processing method]
First, the endoscopic image acquisition unit 110 acquires an endoscopic image to be recognized. This endoscopic image is an endoscopic image of a specific image quality. Next, the image recognition unit 112 performs image recognition on the acquired endoscopic image. Next, the recognition result output unit 114 outputs the recognition result.

In the endoscopic image processing device 100 of the present embodiment, since image recognition is performed by a model optimized for a specific image quality, highly accurate image recognition can be performed.

[Modification example of endoscopic image processing device]
<< Modification 1 of the endoscopic image processing device >>
FIG. 14 is a block diagram showing a modified example of the endoscopic image processing device.

As shown in the figure, the endoscope image processing device 100A of this example is different from the endoscope image processing device 100 of the above embodiment in that it further includes a model switching unit 130 for switching a model used for image recognition. To do.

The image recognition unit 112 is provided with a plurality of models that perform image recognition on the endoscopic image, and the model to be used is switched by the model switching unit 130. This model is an optimized model by performing the second learning. The plurality of prepared models are stored in, for example, the ROM 122 or the HDD 124.

The model switching unit 130 switches the model to be used in response to an instruction from the image processing control unit 116. The image processing control unit 116 switches the model to be used in response to an instruction from the user.

For example, when a plurality of endoscopes having different specifications are used for inspection, a model optimized for each endoscope is prepared. Then, the model used for image recognition is switched according to the endoscope used for the examination. This enables highly accurate image recognition.

In addition, since there may be individual differences in the same model of endoscope, prepare a model optimized for each endoscope and use it for image recognition according to the endoscope used for the examination. Switch models. This enables more accurate image recognition.

Further, in general, even if the endoscope has a different model, a processor device (a device that processes an imaging signal output from the endoscope and generates image data) is often shared. By preparing a model optimized for each model and switching the model used for image recognition according to the endoscope used for the examination, more accurate image recognition becomes possible.

<< Modification 2 of the endoscopic image processing device >>
FIG. 15 is a block diagram showing another modification of the endoscopic image processing apparatus.

The endoscope image processing device 100B of this example further includes an endoscope information acquisition unit 140 that acquires information of the endoscope that has captured the endoscope image to be recognized. It is different from the image processing device 100A. As the model used by the image recognition unit 112 for image recognition, a plurality of models optimized for each endoscope used for the inspection are prepared.

The endoscope information acquisition unit 140 acquires information on the endoscope that has captured the endoscope image to be recognized, and outputs the information to the image processing control unit 116. The image processing control unit 116 instructs the model switching unit 130 to switch so that the corresponding model is used based on the acquired endoscopic information. The model switching unit 130 switches the model to be used in response to an instruction from the image processing control unit 116. For example, a table in which the type (model) of the endoscope and the corresponding model are associated with each other is prepared, and the model is switched by referring to the table.

According to the endoscopic image processing device 100B of this example, a model suitable for image recognition is automatically switched, so that highly accurate image recognition is always possible.

As the plurality of models to be used by switching, in addition to the mode in which a plurality of models are prepared according to the specifications of the endoscope, a model generated by performing the second learning with learning image groups having different resolutions from each other. , A model generated by performing the second learning on learning image groups having different noise amounts, a model generated by performing the second learning on learning image groups having different combinations, and the like are prepared. .. Then, an appropriate model is selected according to the application.

[Other embodiments]
《Medical image》
In the above embodiment, the case of recognizing an endoscopic image as a medical image as an example has been described as an example, but the medical image to which the present invention can be applied is not limited to this.

"Medical images" to which the present invention can be applied include CT (Computerized Tomography) images, X-ray images, ultrasonic diagnostic images, MRI (Magnetic Resonance Imaging) images, and PET (Positron Emission Tomography) in addition to endoscopic images. Various types of images such as images, SPECT (Single Photon Emission Computed Tomography) images, and fundus images are included.

The medical image processing device of the present disclosure can be used as a diagnostic support device that supports medical examination, treatment, diagnosis, etc. by a doctor or the like. The term "diagnostic support" includes the concept of medical examination support and / or treatment support.

<< About hardware configuration >>
The hardware that realizes the learning device and the medical image processing device can be configured by various processors as shown below.

Various processors include CPUs (Central Processing Units) and FPGAs (Field Programmable Gate Arrays), which are general-purpose processors that execute programs and function as various processing units. A programmable logic device (PLD), a dedicated electric circuit which is a processor having a circuit configuration specially designed for executing a specific process such as an ASIC (Application Specific Integrated Circuit), and the like are included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs or a combination of a CPU and an FPGA. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which the processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. is there. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

"Endoscope"
The endoscope is not limited to a flexible endoscope, and may be a rigid endoscope or a capsule endoscope.

<< About the observation light of the endoscope >>
As the observation light (illumination light) of the endoscope, white light, light of one or a plurality of specific wavelength bands, or light of various wavelength bands according to the observation purpose such as a combination thereof is selected. White light is light in a white wavelength band or light in a plurality of wavelength bands. The "specific wavelength band" is a band narrower than the white wavelength band. Specific examples regarding a specific wavelength band are shown below.

<First example>
A first example of a particular wavelength band is, for example, the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example is in the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. It has a peak wavelength within the wavelength band.

<Second example>
A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example is within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Has a peak wavelength within the wavelength band of.

<Third example>
The third example of a specific wavelength band includes a wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin, and the light in the third example peaks in the wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin. Has a wavelength. The wavelength band of the third example includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, a wavelength band of 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is the above 400 ±. It has a peak wavelength in the wavelength band of 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less.

<4th example>
The fourth example of the specific wavelength band is used for observing the fluorescence emitted by the fluorescent substance in the living body (fluorescence observation), and the wavelength band of the excitation light that excites the fluorescent substance, for example, 390 nm to 470 nm.

<Example 5>
A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example is in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. It has a peak wavelength within the wavelength band.

<< Switching the observation light of the endoscope >>
As the type of light source, a laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof can be adopted. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject, the purpose of observation, etc., and during observation, the wavelength of the illumination light is determined according to the type of subject, the purpose of observation, etc. It is preferable to combine and / or switch. When switching the wavelength, for example, the wavelength of the emitted light is switched by rotating a disk-shaped filter (rotary color filter) arranged in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength. You may.

The image sensor used for the endoscope is not limited to a color image sensor in which a color filter is provided for each pixel, and may be a monochrome image sensor. When a monochrome image sensor is used, the wavelength of the illumination light can be sequentially switched to perform surface-sequential (color-sequential) imaging. For example, the wavelength of the emitted illumination light may be sequentially switched between purple, blue, green, and red, or a wide band light (white light) is irradiated and a rotary color filter (red, green, blue, etc.) is used. The wavelength of the emitted illumination light may be switched. Further, the wavelength of the illumination light emitted by the rotary color filter by irradiating one or a plurality of narrow band lights may be switched. The narrow band light may be infrared light having two or more wavelengths having different wavelengths.

<< Example of generating a special light image >>
The processor device that processes the image of the endoscope may generate a special optical image having information in a specific wavelength band based on a normal optical image obtained by imaging with white light. The generation referred to here includes the concept of "acquisition". The processor device 16 transmits a signal in a specific wavelength band to red (R), green (G), and blue (B), or cyan (Cyan, C), magenta (Magenta, M), which are usually contained in an optical image. ), It can be obtained by performing an operation based on the color information of Yellow (Y).

<< About the program that makes the computer realize the functions of the learning device and the medical image processing device >>
A program for realizing the functions of the learning device and the medical image processing device described in the above-described embodiment on a computer is recorded on an optical disk, a magnetic disk, or a computer-readable medium such as a semiconductor memory or other tangible non-temporary information storage medium. However, it is possible to provide a program through this information storage medium. Further, instead of the mode in which the program is stored and provided in such a tangible non-temporary information storage medium, it is also possible to provide the program signal as a download service by using a telecommunication line such as the Internet.

It is also possible to provide a part or all of the functions of the learning device and the medical image processing device described in the above-described embodiment as an application server, and provide a service that provides the processing function through a telecommunication line.

1 Learning device 10 1st learning unit 12 1st learning data set 16 Processor device 20 2nd learning unit 22 2nd learning data set 30 Learning control unit 51 CPU
52 ROM
53 RAM
54 HDD
55 Communication interface 56 Input / output interface 57 Input device 58 Display device 59 Network 100 Endoscopic image processing device 100A Endoscopic image processing device 100B Endoscopic image processing device 110 Endoscopic image acquisition unit 112 Image recognition unit 114 Recognition result Output unit 116 Image processing control unit 121 CPU
122 ROM
123 RAM
124 HDD
125 Communication interface 126 Input / output interface 127 Input device 128 Display device 130 Model switching unit 140 Endoscopic information acquisition unit M1 First model M2 Second model S1 to S4 Learning procedure S11 to S14 Learning with high-resolution learning image group Result-based learning procedure S21 to S24 Learning procedure based on learning result by low-resolution learning image group S31 to S34 Learning procedure based on learning result by low-noise learning image group From S41 S44 Learning procedure based on the learning result of the high-noise learning image group S51 to S54 Learning procedure based on the learning result of the wide-angle learning image group S61 to S64 For learning taken with different endoscopes Learning procedure based on the learning result of the image group

Claims (20)

A first learning unit that generates a first model that performs image recognition on the first image quality medical image by learning using the first medical image group composed of the first image quality medical image.
Image recognition is performed on the second image quality medical image by learning using the second medical image group composed of the second image quality medical image different from the first image quality based on the first model. The second learning part that generates the second model to be performed, and
A learning device equipped with. The first medical image group is composed of a first resolution medical image, and the second medical image group is composed of a second resolution medical image different from the first resolution.
The learning device according to claim 1. The second resolution is lower than the first resolution.
The learning device according to claim 2. The first medical image group is composed of medical images having a resolution of 4K or more, and the second medical image group is composed of medical images having a resolution of less than 4K.
The learning device according to claim 3. The first medical image group is composed of medical images having a resolution of 8K or more, and the second medical image group is composed of medical images having a resolution of less than 8K.
The learning device according to claim 3. The second resolution is a resolution higher than the first resolution.
The learning device according to claim 2. The first medical image group is composed of medical images having a resolution of less than 4K, and the second medical image group is composed of medical images having a resolution of 4K or more.
The learning device according to claim 6. The first medical image group is composed of medical images having a resolution of less than 8K, and the second medical image group is composed of medical images having a resolution of 8K or more.
The learning device according to claim 6. The first medical image group is composed of medical images having a smaller amount of noise than the medical images constituting the second medical image group.
The learning device according to claim 1. The first medical image group is composed of medical images having a larger amount of noise than the medical images constituting the second medical image group.
The learning device according to claim 1. The first medical image group is composed of medical images having a wider angle of view than the medical images constituting the second medical image group.
The learning device according to claim 1. The first medical image group is composed of medical images taken by an endoscope, and the second medical image group is a medical image taken by an endoscope different from the endoscope that took the first medical image group. Consists of
The learning device according to claim 1. The second medical image group is composed of medical images taken by an endoscope having specifications different from those of the endoscope that captured the medical images constituting the first medical image group.
The learning device according to claim 12. The first model and the second model are composed of a convolutional neural network.
The learning device according to any one of claims 1 to 13. The medical image acquisition department that acquires medical images and
A model composed of the second model generated by the learning device according to any one of claims 1 to 14 and performing image recognition on the medical image, and a model.
Medical image processing device equipped with. With multiple of the above models
A model switching unit that switches the model to be used,
The medical image processing apparatus according to claim 15, further comprising. An endoscope information acquisition unit for acquiring information on the endoscope that captured the medical image is further provided.
The plurality of the models are generated by learning in the second learning unit using the second medical image group photographed by different endoscopes.
The medical image processing apparatus according to claim 16, wherein the model switching unit switches the model to be used based on the information of the endoscope acquired by the endoscope information acquisition unit. The plurality of the models are generated by learning in the second learning unit using the second medical image group photographed by endoscopes having different specifications.
The medical image processing apparatus according to claim 17. The plurality of models are generated by learning in the second learning unit using the second medical image group captured by endoscopes having different resolutions or noise amounts from each other.
The medical image processing apparatus according to claim 18. A step of generating a first model for image recognition on the first-quality medical image by learning using a first medical image group composed of a first-quality medical image, and a step of generating the first model.
Image recognition is performed on the second image quality medical image by learning using the second medical image group composed of the second image quality medical image different from the first image quality based on the first model. Steps to generate the second model to be done,
Learning method with.

Images