WO2020003991A1 - Medical image learning device, method, and program - Google Patents

Medical image learning device, method, and program Download PDF

Info

Publication number
WO2020003991A1
WO2020003991A1 PCT/JP2019/022909 JP2019022909W WO2020003991A1 WO 2020003991 A1 WO2020003991 A1 WO 2020003991A1 JP 2019022909 W JP2019022909 W JP 2019022909W WO 2020003991 A1 WO2020003991 A1 WO 2020003991A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
learning
medical image
medical
model
Prior art date
Application number
PCT/JP2019/022909
Other languages
French (fr)
Japanese (ja)
Inventor
正明 大酒
Original Assignee
富士フイルム株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 富士フイルム株式会社 filed Critical 富士フイルム株式会社
Priority to JP2020527358A priority Critical patent/JP7187557B2/en
Publication of WO2020003991A1 publication Critical patent/WO2020003991A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B23/00—Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
    • G02B23/24—Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes

Definitions

  • the present invention relates to a medical image learning apparatus, a method, and a program, and particularly relates to a technique for performing learning for accurate image recognition with a relatively small number of data.
  • Non-Patent Document 1 In image analysis for recognition, machine learning of images such as deep learning (Deep Learning) is widely used (for example, Non-Patent Document 1).
  • Medical images include a special light image captured by a light source called special light, in addition to a normal light image captured by a light source called normal light.
  • a normal light image is mainly used, so that a relatively large number of normal light images can be collected.
  • the special light image is selectively used in accordance with the observation purpose, so that the number of data is not so large as compared with the normal light image. Therefore, learning using the special light image cannot be performed sufficiently, and it is difficult to increase the recognition accuracy of the special light image by the learned recognizer.
  • Patent Document 1 at least one characteristic of a shape of a subject shown in a target image group to be learned, a tissue structure of the subject shown in the target image group, and at least one characteristic of an imaging system of a device that captures the target image group are similar.
  • An image processing device including a unit has been proposed. Thereby, even if the number of data of the target image group used for the main learning is small, it is possible to perform high-precision learning.
  • the image group used for the pre-learning described in Patent Literature 1 is a similar image group in which at least one characteristic of the shape of the subject, the tissue structure of the subject, and the imaging system of the imaging device is similar to the target image group to be learned. . That is, in Patent Literature 1, the target image group to be learned and the similar image group are a special light image group captured by the special light observation light and a normal light image group captured by the normal light observation light. That is, it does not describe that the images are images captured by different observation lights.
  • the present invention has been made in view of such circumstances, and provides a medical image learning apparatus, a method, and a program that can generate a model that performs high-precision image recognition on a medical image captured with special light.
  • the purpose is to do.
  • a medical image learning apparatus provides a medical image learning apparatus for learning by using at least a first image group including a plurality of first medical images captured by ordinary light.
  • a first learning unit that generates a first model of the second medical image based on the first model, and learning using a second image group including a plurality of second medical images captured using special light.
  • a second learning unit that generates a second model that performs image recognition on.
  • the first learning unit performs learning (preliminary learning) using a first image group including first medical images captured by ordinary light that can be collected in a relatively large amount.
  • the first model for image recognition can be satisfactorily generated.
  • the second learning unit performs learning (also referred to as “fine tuning”) using a second image group including a plurality of second medical images based on the first model.
  • a second model that performs appropriate image recognition on the second medical image can be generated from the two image groups.
  • the first learning unit generates the first model using the second medical image. Thereby, the number of medical images used for generating the first model can be increased.
  • a medical image learning apparatus includes an image processing unit that converts a first medical image having a plurality of color channels into a first medical image having one color channel, and a first learning unit. Is preferably learned using a first image group having one converted color channel. By converting the first medical image into the first medical image having one color channel, the feature amount of the first medical image can be made closer to the feature amount of the second medical image, and the generation of the first model can be improved. Can be done properly.
  • the image processing unit converts a first medical image having a plurality of color channels into a first medical image having only a luminance signal, thereby forming one color channel. It is preferable that the first medical image has the following. Thereby, the first learning unit can extract the basic feature amount such as an edge which is not affected by the color information including a plurality of color channels and the identification rule, and can generate an appropriate first model. it can.
  • the first medical image and the second medical image having a plurality of color channels are converted into the first medical image and the second medical image each having one color channel. It is preferable that the first learning unit includes a first image group and a second image group having one converted color channel.
  • the first image group and the second image group are used for learning in the first learning unit
  • the first image group and the second image group including a plurality of color channels are converted into image groups each having one color channel.
  • the first model can be more appropriately generated by using the obtained model.
  • the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having only the luminance signal. Is converted into a first medical image and a second medical image having one color channel. Accordingly, the first learning unit can extract a basic feature amount such as an edge that is not affected by color information including a plurality of color channels, and can generate an appropriate first model.
  • the medical image learning apparatus further includes an extraction unit that extracts a first medical image having one color channel from a first medical image having a plurality of color channels, wherein the first learning unit includes It is preferable to perform learning using the first image group having one extracted color channel.
  • a first medical image having one color channel is extracted from a first image group having a plurality of color channels, and learning is performed using the first image group having the extracted one color channel. It is possible to extract basic features such as edges which are not affected by the color information, and to generate an appropriate first model.
  • a first medical image and a second medical image having one color channel are respectively extracted from a first medical image and a second medical image having a plurality of color channels. It is preferable that the first learning unit learns using the first image group and the second image group having one extracted color channel.
  • the first medical image having one color channel each from the first image group and the second image group including a plurality of color channels
  • the first model can be more appropriately generated.
  • the plurality of color channels are preferably three channels of three primary colors or three channels of a luminance signal and two color difference signals.
  • a third model for performing image recognition on the first medical image by learning using the first image group based on the first model is generated. And a third learning unit.
  • the first model can also be used when generating a third model for performing image recognition on the first medical image.
  • the first medical image and the second medical image are each preferably an image captured by an endoscope device.
  • the first model and the second model are configured by a convolutional neural network.
  • a medical image learning method includes a first image group including a plurality of first medical images captured with normal light and a second image group including a plurality of second medical images captured with special light.
  • An image group preparing step a first learning section generating a first model for image recognition by learning using at least the first image group, and a second learning section generating a first model based on the first model. And generating a second model for performing image recognition on the second medical image by learning using the second image group.
  • the image processing unit includes a step of converting a first medical image having a plurality of color channels into a first medical image having one color channel, In the step of generating one model, it is preferable that learning is performed using a first image group having one converted color channel.
  • the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having one color channel.
  • the step of generating the first model includes the step of converting into a medical image, and the step of generating the first model learns using the first image group and the second image group having one converted color channel.
  • the extracting unit includes a step of extracting a first medical image having one color channel from a first medical image having a plurality of color channels, In the step of generating the model, it is preferable that learning is performed using the first image group having one extracted color channel.
  • the extraction unit extracts the first medical image and the second medical image having one color channel from the first medical image and the second medical image having a plurality of color channels. It is preferable that the step of generating the first model includes the step of extracting the images, and the step of generating the first model is performed using the first image group and the second image group having one extracted color channel.
  • a medical image learning program includes a first image group including a plurality of first medical images captured by normal light and a second image including a plurality of second medical images captured by special light.
  • a function of acquiring an image group, a function of generating a first model for image recognition by learning using at least the first image group, and learning using a second image group based on the first model Thereby, the function of generating the second model for performing image recognition on the second medical image is realized by the computer.
  • a good first model is generated by performing learning using at least a first image group including first medical images captured by ordinary light, which can be collected in a relatively large amount. Based on one model, learning is performed using a second image group composed of a plurality of second medical images captured by special light, so that the second medical image can be obtained from the second image group having a relatively small number of data.
  • a second model that performs appropriate image recognition can be generated.
  • FIG. 1 is a block diagram showing an example of a hardware configuration of a medical image learning device according to the present invention.
  • FIG. 2 is a block diagram showing a first embodiment of the medical image learning device 10-1 according to the present invention.
  • FIG. 3 is a functional block diagram illustrating an embodiment of the first learning unit 30.
  • FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention.
  • FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention.
  • FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention.
  • FIG. 7 is a flowchart showing an embodiment of the medical image learning method according to the present invention.
  • FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG.
  • FIG. 9 is a view showing step S14-2 showing a second modification of step S14 shown in FIG.
  • FIG. 1 is a block diagram showing an example of a hardware configuration of a medical image learning device according to the present invention.
  • a personal computer or a workstation can be used as the medical image learning device 10 shown in FIG. 1.
  • the medical image learning device 10 of the present example mainly includes a communication unit 12, a large-capacity storage or a first database 14, It comprises a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display unit 26.
  • a communication unit 12 mainly includes a communication unit 12, a large-capacity storage or a first database 14, It comprises a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display unit 26.
  • a communication unit 12 mainly includes a communication unit 12, a large-capacity storage or a first database 14, It comprises a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and
  • the communication unit 12 is a unit that performs communication processing with an external device by wire or wirelessly and exchanges information with the external device.
  • the first database 14 includes a first image group (normal light image group) including a plurality of first medical images (normal light images) captured by normal light, and correct data indicating a correct image recognition result of each normal light image.
  • the second database 16 stores a second image group (special light image) including a plurality of second medical images (special light images) captured with special light.
  • the normal light image (first medical image) and the special light image (second medical image) are color images captured by the endoscope apparatus under different light sources.
  • the normal light is light (white light) in which light in all wavelength bands of visible light is almost uniformly mixed, and the normal light image is used for normal observation. Therefore, a relatively large number of ordinary light image groups can be collected.
  • the special light is light of various wavelength bands according to the observation purpose, which is a combination of light of one specific wavelength band or light of a plurality of specific wavelength bands, and a band narrower than the white wavelength band. And is used for narrow band observation (NBI (Narrow band imaging), FICE (Flexible spectrum imaging color enhancement)).
  • NBI Near band imaging
  • FICE Flexible spectrum imaging color enhancement
  • a first example of the specific wavelength band is, for example, a blue band or a green band in a visible region.
  • the wavelength band of the first example includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the first example has a peak wavelength in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. .
  • the second example of the specific wavelength band is, for example, a red band in a visible region.
  • the wavelength band of the second example includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the second example has a peak wavelength in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm. .
  • the third example of the specific wavelength band includes a wavelength band having different absorption coefficients between oxyhemoglobin and reduced hemoglobin, and the light of the third example has a peak wavelength at a wavelength band having different absorption coefficients between oxyhemoglobin and reduced hemoglobin.
  • the wavelength band of the third example includes 400 ⁇ 10 nm, 440 ⁇ 10 nm, 470 ⁇ 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example includes the above 400 ⁇ 10 nm, 440 ⁇ 10 nm, 470 nm. It has a peak wavelength in a wavelength band of ⁇ 10 nm, or 600 nm to 750 nm.
  • the fourth example of the specific wavelength band is a wavelength band (390 nm to 470 nm) of excitation light used for observation of fluorescence emitted from a fluorescent substance in a living body (fluorescence observation) and for exciting this fluorescent substance.
  • the fifth example of the specific wavelength band is a wavelength band of infrared light.
  • the wavelength band of the fifth example includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.
  • the first data set of the normal light image group stored in the first database 14 is prepared more than the second data set of the special light image group stored in the second database 16.
  • the correct answer data stored in association with each normal light image and each special light image includes, for example, the type of a lesion shown in the normal light image and the special light image, Data indicating the position of the lesion, identification information unique to the case, and the like can be considered.
  • the classification of lesions includes two classifications, neoplastic and non-neoplastic, and NICE classification.
  • the data indicating the position of the lesion may be rectangular information surrounding the lesion or mask data that covers the lesion.
  • the first database 14 and the second database 16 are provided in the medical image learning device 10, but may be provided externally.
  • a data set for learning can be obtained from an external database via the communication unit 12.
  • the operation unit 18 uses a keyboard, a mouse, and the like that are connected to the computer by wire or wirelessly, and receives various operation inputs during machine learning.
  • the CPU 20 reads various programs (including the medical image learning program according to the present invention) stored in the ROM 24 or a hard disk device (not shown) and executes various processes.
  • the RAM 22 is used as a work area of the CPU 20, and is used as a storage unit for temporarily storing read programs and various data.
  • the display unit 26 includes various monitors such as a liquid crystal monitor that can be connected to a computer, and is used together with the operation unit 18 as part of a user interface.
  • the CPU 20 reads a medical image learning program stored in the ROM 24, a hard disk device, or the like in response to an instruction input from the operation unit 18 and executes the medical image learning program, as described later. Function as a medical image learning device.
  • FIG. 2 is a block diagram showing a first embodiment of the medical image learning device 10-1 according to the present invention, and is a functional block diagram showing main functions of the medical image learning device 10 shown in FIG.
  • the medical image learning device 10-1 shown in FIG. 2 includes a first learning unit 30 and a second learning unit 40.
  • the first learning unit 30 performs learning using the data set of the normal light image stored in the first database 14 and the data set of the special light image stored in the second database 16, thereby performing learning for image recognition.
  • Generate a model (first model).
  • a convolutional neural network (CNN: Convolution Neural Network), which is one of the learning models, is constructed.
  • FIG. 3 is a functional block diagram showing an embodiment of the first learning unit 30.
  • the first learning unit 30 shown in FIG. 3 mainly includes a CNN 32, an error calculation unit 34, and a parameter update unit 36.
  • the CNN 32 is, for example, a part corresponding to a recognizer that recognizes the type of a lesion appearing in a medical image, has a plurality of layer structures, and holds a plurality of weight parameters.
  • the CNN 32 can change from an unlearned model to a learned model by updating the weight parameter from an initial value to an optimal value.
  • the CNN 32 includes an input layer 32A, a plurality of sets including a convolutional layer and a pooling layer, an intermediate layer 32B having a fully connected layer, and an output layer 32C. Each layer has a plurality of “nodes” represented by “edges”. The structure is connected by
  • the normal light image 14A to be learned is input to the input layer 32A.
  • the intermediate layer 32B has a plurality of sets each including a convolutional layer and a pooling layer and a fully connected layer, and is a portion for extracting features from an image input from the input layer.
  • the convolutional layer filters nearby nodes in the previous layer (performs a convolution operation using a filter) to obtain a “feature map”.
  • the pooling layer reduces the feature map output from the convolutional layer to a new feature map.
  • the “convolution layer” has a role of extracting features such as edge extraction from an image, and the “pooling layer” has a role of providing robustness so that the extracted features are not affected by translation or the like. Note that the intermediate layer 32B is not limited to the case where the convolutional layer and the pooling layer are set as one set, but also includes the case where the convolutional layer is continuous and the normalization layer.
  • the output layer 32C is a part that outputs a recognition result for classifying the type of lesion appearing in the medical image based on the features extracted by the intermediate layer 32B.
  • the medical image is classified into three categories of “neoplastic”, “non-neoplastic”, and “other”, and the recognition result is “neoplastic”, “non-neoplastic”, and “other”.
  • the recognition result is “neoplastic”, “non-neoplastic”, and “other”.
  • An arbitrary initial value is set for the filter coefficient and offset value applied to each convolutional layer of the CNN 32 before learning, and the connection weight between the fully connected layer and the next layer.
  • the error calculator 34 acquires the recognition result output from the output layer 32C of the CNN 32 and the correct data for the normal light image 14A, and calculates an error between the two.
  • a method of calculating the error for example, softmax cross entropy, sigmoid, or the like can be considered.
  • the parameter update unit 36 adjusts the weight parameter of the CNN 30 by the error back propagation method based on the error calculated by the error calculation unit 34.
  • the parameter adjustment process is repeatedly performed, and learning is repeatedly performed until the difference between the output of the CNN 32 and the correct answer data becomes small.
  • the first learning unit 30 uses all data sets of the normal light image group stored in the first database 14 to perform learning for optimizing each parameter of the CNN 32, thereby obtaining a learned model (first model). Generate
  • the second learning unit 40 uses the learned model learned by the first learning unit 30 (using the parameters of the learned CNN 32 as initial values) and stores the special model stored in the second database 16.
  • a learning model (second model) for performing image recognition on the special light image is generated by learning again using only the light image data set.
  • the second learning unit 40 has the same configuration as the first learning unit 30 shown in FIG. 3, and thus a detailed description thereof will be omitted. Note that the second learning unit 40 may be configured by the same learning unit as the first learning unit 30.
  • learning is performed using a data set of normal light images that can be collected by the first learning unit 30 in a relatively large amount. Can be satisfactorily generated. Since the second learning unit 40 performs learning (also referred to as “fine tuning”) using only the special light image group based on the learning model generated by the first learning unit 30, the second learning unit 40 has a relatively small number of data. A learning model for performing appropriate image recognition on a special light image can be generated from a small group of special light images.
  • the first learning unit 30 of the medical image learning apparatus 10-1 learns using the data set of the normal light image and the data set of the special light image. Learning may be performed by using.
  • FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention.
  • the medical image learning device 10-2 of the second embodiment shown in FIG. 4 differs from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an image processing unit 50 is added. I do.
  • the normal light image stored in the first database 14 and the special light image stored in the second database 16 are respectively composed of three color channels of red (R), green (G), and blue (B). Images.
  • the image processing unit 50 converts the normal light image having three color channels of RGB stored in the first database 14 and the special light image having three color channels of RGB stored in the second database 16 into Each of them is converted into a normal light image and a special light image having one color channel, and the converted normal light image and special light image having one color channel are output to the first learning unit 30 as a learning data set. .
  • the image processing unit 50 converts the RGB color image into a normal light image and a special light image having one color channel by performing monochrome processing.
  • a process of generating a luminance signal (Y signal) from the RGB color signals (R signal, G signal, B signal) by the following equation can be considered.
  • the first learning unit 30 acquires a normal light image and a special light image (monochrome image) having one color channel converted by the image processing unit 50 as a learning data set, and performs image recognition using the acquired data set.
  • a learning model (first model) is generated.
  • the second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB special light image data set stored in the second database 16. Thus, a learning model for performing image recognition on the special light image is generated.
  • the filter used in at least the first convolutional layer of the CNN also has one channel.
  • the data set used for learning in the second learning unit 40 is a special light image of a plurality of color channels (RGB)
  • the filter used in at least the first convolution layer of the CNN also has a plurality of channels (three channels). become. Therefore, the same parameters are used as initial values for the three-channel filters used in the first convolutional layer of the CNN of the second learning unit 40, respectively.
  • the medical image learning device 10-2 of the second embodiment it is possible to perform pre-learning by making the feature amount of the normal light image close to the feature amount of the special light image. Re-learning using only the data set can be performed more appropriately.
  • the image processing unit 50 of the medical image learning apparatus 10-2 of the second embodiment performs monochrome processing on the normal light image and the special light image, respectively.
  • the learning 30 may be performed using only the data set of the normal light image subjected to the monochrome processing.
  • the image processing unit 50 is not limited to the case of generating a “black-and-white image” including a luminance signal, but may be any unit that generates a single color (one channel) image. Preferably, an image is generated.
  • FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention.
  • the medical image learning device 10-3 of the third embodiment shown in FIG. 5 is different from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an extraction unit 60 is added. .
  • the normal light image stored in the first database 14 and the special light image stored in the second database 16 are color images each including three color channels of RGB.
  • the extraction unit 60 extracts a normal light image having three color channels of RGB stored in the first database 14 and a special light image having three color channels of RGB stored in the second database 16, respectively. A normal light image and a special light image having one color channel are extracted.
  • the extraction unit 60 performs one channel of B It is preferable to extract only the normal light image and the special light image.
  • the extraction unit 60 determines whether or not R is 1 It is preferable to extract the normal light image and the special light image of only the channel. That is, it is preferable that the extraction unit 60 extracts an image of one color channel close to the tint of the special light image from the normal light image and the special light image having three color channels of RGB.
  • the first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a learning data set, and uses the acquired data set to perform a learning model for image recognition ( (First model).
  • the second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB special light image data set stored in the second database 16. Thus, a learning model for performing image recognition on the special light image is generated.
  • the feature amount of the normal light image can be made closer to the feature amount of the special light image to perform pre-learning. Re-learning using only the data set can be performed more appropriately.
  • the extraction unit 60 of the medical image learning device 10-3 extracts a normal light image and a special light image having one color channel from the RGB normal light image and the special light image. Even if only the normal light image having one color channel is extracted from the RGB normal light image, the first learning unit 30 may perform learning using only the data set of the extracted normal light image having one color channel. Good.
  • FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention.
  • the medical image learning device 10-4 of the fourth embodiment shown in FIG. 6 differs from the medical image learning device 10-2 of the second embodiment shown in FIG. 4 mainly in that a third learning unit 70 is added. Different.
  • the third learning unit 70 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB normal light image data set stored in the first database 14. Thus, a learning model (third model) for performing image recognition on the normal light image is generated.
  • the parameters of the learned CNN learned by the first learning unit 30 are used as the initial values.
  • the learning time can be reduced as compared with the case where learning is performed using only the data set of the optical image.
  • the third learning unit 70 may use the parameter of the learned CNN learned by the first learning unit 30 of the medical image learning device 10-3 of the third embodiment shown in FIG. 5 as an initial value. Good.
  • the normal light image and the special light image stored in the first database 14 and the second database 16 of the present embodiment are color images of RGB “three primary colors”, respectively, but are not limited thereto.
  • a color image of “the three primary colors” of cyan (C), magenta (M), and yellow (Y) which have a complementary color relationship may be used.
  • the normal light image and the special light image stored in the first database 14 and the second database 16 include a luminance signal (Y) generated from RGB color signals and two color difference signals (Cr, Cb). It may be a three-channel color image.
  • RGB, CMY, and YCrCb can be mutually converted.
  • FIG. 7 is a flowchart showing an embodiment of the medical image learning method according to the present invention, and shows a processing procedure of each unit of the medical image learning apparatus 10-1 of the first embodiment shown in FIG.
  • the first database 14 and the second database 16 previously store a data set of a normal light image group for learning and a data set of a special light image group (step of preparing a data set).
  • the first learning unit 30 acquires a data set of the normal light image group and a data set of the special light image group from the first database 14 and the second database 16 (Steps S10 and S12).
  • the first learning unit 30 may acquire a data set in units of mini-batch of about 10 to 100, or may acquire data sets one by one.
  • the first learning unit 30 generates a learning model (first model) for image recognition by performing learning using the acquired data sets of the normal light image and the special light image (step S14).
  • a learned CNN which is one of the learning models, is constructed.
  • the second learning unit 40 uses the learned model learned by the first learning unit 30 (using the parameter of the learned CNN as an initial value) to generate the special light image stored in the second database 16.
  • a learning model (second model) for performing image recognition on the special light image is generated (step S16).
  • the medical image learning method of this embodiment since learning (prior learning) is performed using a data set of normal light images that can be collected by the first learning unit 30 in a relatively large amount, a learning model for image recognition is used. It can be produced well. Then, the second learning unit 40 performs re-learning using only the special light image group based on the learning model generated by the first learning unit 30, so that even the special light image group having a relatively small number of data is used. A learning model for performing appropriate image recognition on a special light image can be generated.
  • step S14 the first learning unit 30 generates a learning model for image recognition by learning using the data set of the normal light image and the special light image, but uses only the data set of the normal light image. You may learn.
  • FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG.
  • the image processing unit 50 (FIG. 4) includes a normal light image having three color channels of RGB stored in the first database 14 and three colors of RGB stored in the second database 16.
  • the special light image having the channel is converted into a normal light image and a special light image (monochrome image) each having one color channel (step S20).
  • the first learning unit 30 acquires the monochrome image converted by the image processing unit 50 as a data set for learning, and uses the acquired data set as a learning model of CNN (one of the learning models for image recognition). (1 model) is generated (step S22). Note that the parameter of the learned CNN learned in step S22 is used as an initial value of the CNN of the second learning unit 40 in step S16 shown in FIG.
  • FIG. 9 is a view showing step S14-2 showing a second modification of step S14 shown in FIG.
  • the extraction unit 60 (FIG. 5) includes a normal light image having three color channels of RGB stored in the first database 14, and three color channels of RGB stored in the second database 16.
  • the normal light image and the special light image each having one color channel are extracted from the special light image having (Step S30). It is preferable that the extraction unit 60 extracts an image of one color channel close to the tint of the special light image from the normal light image and the special light image having three color channels of RGB.
  • the first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a data set for learning, and generates a learning model for image recognition using the obtained data set.
  • a learning model (first model) of one CNN is generated (step S32).
  • the parameter of the learned CNN learned in step S32 is used as an initial value of the CNN of the second learning unit 40 in step S16 shown in FIG.
  • the CNN 32 shown in FIG. 3 is a learning model for recognizing the type of lesion shown in the medical image, but may be a learning model for performing segmentation for recognizing the position (lesion area) of the lesion shown in the medical image. .
  • the CNN apply a full-layer convolution network (FCN: Fully Convolution Network), which is a type of the CNN, and can grasp the position of a lesion appearing in a medical image at a pixel level.
  • FCN Fully Convolution Network
  • the present invention is also applicable to machine learning models other than CNN, such as DBN (Deep Belief Network) and SVM (Support Vector Machine).
  • DBN Deep Belief Network
  • SVM Small Vector Machine
  • the hardware structure for executing various controls of the medical image learning apparatus 10 of the present embodiment is various processors as described below.
  • the circuit configuration can be changed after manufacturing such as CPU (Central Processing Unit) and FPGA (Field Programmable Gate Array), which are general-purpose processors that execute software (programs) and function as various control units.
  • Special-purpose circuits such as a programmable logic device (Programmable Logic Device: PLD), an ASIC (Application Specific Integrated Circuit), and a dedicated electric circuit having a circuit configuration specifically designed to execute a specific process are included. It is.
  • One processing unit may be configured with one of these various processors, or configured with two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which a processor functions as a plurality of control units.
  • SoC system-on-chip
  • a processor that realizes the functions of the entire system including a plurality of control units by one IC (Integrated Circuit) chip is used.
  • the various control units have a hardware structure using one or more of the above-described various processors.
  • the hardware structure of these various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.
  • the present invention also includes a medical image learning program that is installed in a computer to function as the medical image learning device according to the present invention, and a recording medium on which the medical image learning program is recorded.

Abstract

Provided are a medical image learning device, method, and program which are capable of creating a model that performs highly accurate image recognition on a medical image imaged under special light. This medical image learning device (10-1) includes a first learning part (30) that creates a first model for image recognition by learning using a normal light image group comprising a plurality of normal light images imaged under normal light, and a second learning part (40) that creates a second model, which performs image recognition on a special light image, by learning using a special light image group comprising a plurality of special light images imaged under special light on the basis of the first model. The first learning part (30) performs prior learning using the normal image group which can be collected in relatively large numbers, and is thus capable of favorably creating the first model. The second learning part (40) relearns using the special light image group on the basis of the first model, and is thus capable of creating the second model that appropriately performs image recognition on the special light image which is relatively small in number.

Description

医療画像学習装置、方法及びプログラムMedical image learning device, method and program
　本発明は医療画像学習装置、方法及びプログラムに係り、特に比較的少ないデータ数で精度のよい画像認識のための学習を行う技術に関する。 The present invention relates to a medical image learning apparatus, a method, and a program, and particularly relates to a technique for performing learning for accurate image recognition with a relatively small number of data.
　医療分野においては、内視鏡装置を用いた検査が行われている。近年においては、画像解析によって画像に含まれる病変を検出する、病変を種別ごとに分類する等の認識を行い、報知することで検査を支援することが知られている。 検 査 In the medical field, examinations using endoscope devices are performed. In recent years, it has been known to support examination by performing recognition such as detecting a lesion included in an image by image analysis and classifying the lesion by type, and reporting the recognition.
　認識のための画像解析においては、深層学習（Deep Learning）をはじめとする画像の機械学習が広く使用されている（例えば、非特許文献１）。 In image analysis for recognition, machine learning of images such as deep learning (Deep Learning) is widely used (for example, Non-Patent Document 1).
　深層学習では問題に応じた画像を大量に学習させることで分類、検出といった画像認識が可能となる。 In deep learning, image recognition such as classification and detection becomes possible by learning a large number of images according to the problem.
　ところで、医療画像には、通常光と呼ばれる光源により撮像される通常光画像の他に、特殊光と呼ばれる光源により撮像される特殊光画像がある。 Medical images include a special light image captured by a light source called special light, in addition to a normal light image captured by a light source called normal light.
　目視による検査では、主に通常光画像が使用されるため、通常光画像は比較的多く集めることができる。 (4) In a visual inspection, a normal light image is mainly used, so that a relatively large number of normal light images can be collected.
　一方、特殊光画像は、観察目的に合わせて選択的に使用されるため、通常光画像に比べて、あまりデータ数が多くない。そのため、特殊光画像による学習を十分に行うことができず、学習済み認識器による特殊光画像に対する認識精度を上げることが困難であった。 On the other hand, the special light image is selectively used in accordance with the observation purpose, so that the number of data is not so large as compared with the normal light image. Therefore, learning using the special light image cannot be performed sufficiently, and it is difficult to increase the recognition accuracy of the special light image by the learned recognizer.
　これに対し、特許文献１には、学習対象の対象画像群に写る被写体の形状、対象画像群に写る被写体の組織構造及び対象画像群を撮像した機器の撮像系の少なくとも１つの特性が類似する類似画像群に基づいて事前学習が行われ、事前学習の結果及び対象画像群に基づいて本学習が行われた本学習結果に基づいて、識別対象の画像群を識別した識別結果を出力する識別部を備える画像処理装置が提案されている。これにより、本学習に使用する対象画像群のデータ数が少量であっても、高精度な学習を行うことができるようにしている。 On the other hand, in Patent Document 1, at least one characteristic of a shape of a subject shown in a target image group to be learned, a tissue structure of the subject shown in the target image group, and at least one characteristic of an imaging system of a device that captures the target image group are similar. Classification in which pre-learning is performed based on a group of similar images, and based on a result of the pre-learning and main learning performed based on the target image group, a classification result in which a group of images to be classified is output is output An image processing device including a unit has been proposed. Thereby, even if the number of data of the target image group used for the main learning is small, it is possible to perform high-precision learning.
国際公開第２０１７／２２１４１２号WO 2017/221412
　特許文献１に記載の事前学習に使用する画像群は、学習対象の対象画像群と被写体の形状、被写体の組織構造及び撮像した機器の撮像系の少なくとも１つの特性が類似する類似画像群である。即ち、特許文献１には、学習対象の対象画像群と類似画像群とが、特殊光の観察光で撮像された特殊光画像群と通常光の観察光で撮像された通常光画像群であること（異なる観察光で撮像された画像群同士であること）が記載されていない。 The image group used for the pre-learning described in Patent Literature 1 is a similar image group in which at least one characteristic of the shape of the subject, the tissue structure of the subject, and the imaging system of the imaging device is similar to the target image group to be learned. . That is, in Patent Literature 1, the target image group to be learned and the similar image group are a special light image group captured by the special light observation light and a normal light image group captured by the normal light observation light. That is, it does not describe that the images are images captured by different observation lights.
　本発明はこのような事情に鑑みてなされたもので、特殊光で撮像された医療画像に対して高精度な画像認識を行うモデルを生成することができる医療画像学習装置、方法及びプログラムを提供することを目的とする。 The present invention has been made in view of such circumstances, and provides a medical image learning apparatus, a method, and a program that can generate a model that performs high-precision image recognition on a medical image captured with special light. The purpose is to do.
　上記目的を達成するために本発明の一の態様に係る医療画像学習装置は、通常光で撮像された複数の第１医療画像からなる第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成する第１学習部と、第１モデルを元に、特殊光で撮像された複数の第２医療画像からなる第２画像群を用いて学習することにより、第２医療画像に対する画像認識を行う第２モデルを生成する第２学習部と、を備える。 In order to achieve the above object, a medical image learning apparatus according to one aspect of the present invention provides a medical image learning apparatus for learning by using at least a first image group including a plurality of first medical images captured by ordinary light. A first learning unit that generates a first model of the second medical image based on the first model, and learning using a second image group including a plurality of second medical images captured using special light. And a second learning unit that generates a second model that performs image recognition on.
　本発明の一の態様によれば、第１学習部は、比較的多く集めることができる通常光で撮像された第１医療画像からなる第１画像群を用いて学習（事前学習）を行うため、画像認識用の第１モデルを良好に生成することができる。そして、第２学習部は、第１モデルを元に、複数の第２医療画像からなる第２画像群を用いて学習（「ファインチューニング」ともいう）を行うため、比較的データ数の少ない第２画像群からでも第２医療画像に対して適切な画像認識を行う第２モデルを生成することができる。 According to one aspect of the present invention, the first learning unit performs learning (preliminary learning) using a first image group including first medical images captured by ordinary light that can be collected in a relatively large amount. , The first model for image recognition can be satisfactorily generated. The second learning unit performs learning (also referred to as “fine tuning”) using a second image group including a plurality of second medical images based on the first model. A second model that performs appropriate image recognition on the second medical image can be generated from the two image groups.
　本発明の他の態様に係る医療画像学習装置において、第１学習部は、第２医療画像も用いて第１モデルを生成することが好ましい。これにより、第１モデルの生成に使用する医療画像をより多くすることができる。 In the medical image learning device according to another aspect of the present invention, it is preferable that the first learning unit generates the first model using the second medical image. Thereby, the number of medical images used for generating the first model can be increased.
　本発明の更に他の態様に係る医療画像学習装置において、複数の色チャンネルを有する第１医療画像を、１つの色チャンネルを有する第１医療画像に変換する画像処理部を備え、第１学習部は、変換された１つの色チャンネルを有する第１画像群を用いて学習することが好ましい。第１医療画像を、１つの色チャンネルを有する第１医療画像に変換することで、第１医療画像の特徴量を第２医療画像の特徴量に近づけることができ、第１モデルの生成をより適切に行うことができる。 A medical image learning apparatus according to still another aspect of the present invention includes an image processing unit that converts a first medical image having a plurality of color channels into a first medical image having one color channel, and a first learning unit. Is preferably learned using a first image group having one converted color channel. By converting the first medical image into the first medical image having one color channel, the feature amount of the first medical image can be made closer to the feature amount of the second medical image, and the generation of the first model can be improved. Can be done properly.
　本発明の更に他の態様に係る医療画像学習装置において、画像処理部は、複数の色チャンネルを有する第１医療画像を、輝度信号のみの第１医療画像に変換することで、１つの色チャンネルを有する第１医療画像とすることが好ましい。これにより、第１学習部は、複数の色チャンネルからなる色情報の影響を受けないエッジ等の基本的な特徴量、識別規則を抽出することができ、適切な第１モデルを生成することができる。 In the medical image learning device according to still another aspect of the present invention, the image processing unit converts a first medical image having a plurality of color channels into a first medical image having only a luminance signal, thereby forming one color channel. It is preferable that the first medical image has the following. Thereby, the first learning unit can extract the basic feature amount such as an edge which is not affected by the color information including a plurality of color channels and the identification rule, and can generate an appropriate first model. it can.
　本発明の更に他の態様に係る医療画像学習装置において、複数の色チャンネルを有する第１医療画像及び第２医療画像を、それぞれ１つの色チャンネルを有する第１医療画像及び第２医療画像に変換する画像処理部を備え、第１学習部は、変換された１つの色チャンネルを有する第１画像群及び第２画像群を用いて学習することが好ましい。 In the medical image learning device according to still another aspect of the present invention, the first medical image and the second medical image having a plurality of color channels are converted into the first medical image and the second medical image each having one color channel. It is preferable that the first learning unit includes a first image group and a second image group having one converted color channel.
　第１学習部での学習に第１画像群及び第２画像群を使用する場合、複数の色チャンネルからなる第１画像群及び第２画像群を、それぞれ１つの色チャンネルを有する画像群に変換されたものを使用することで、第１モデルをより適正に生成できるようにしている。 When the first image group and the second image group are used for learning in the first learning unit, the first image group and the second image group including a plurality of color channels are converted into image groups each having one color channel. The first model can be more appropriately generated by using the obtained model.
　本発明の更に他の態様に係る医療画像学習装置において、画像処理部は、複数の色チャンネルを有する第１医療画像及び第２医療画像を、輝度信号のみの第１医療画像及び第２医療画像に変換することで、１つの色チャンネルを有する第１医療画像及び第２医療画像とすることが好ましい。これにより、第１学習部は、複数の色チャンネルからなる色情報の影響を受けないエッジ等の基本的な特徴量を抽出することができ、適切な第１モデルを生成することができる。 In the medical image learning device according to still another aspect of the present invention, the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having only the luminance signal. Is converted into a first medical image and a second medical image having one color channel. Accordingly, the first learning unit can extract a basic feature amount such as an edge that is not affected by color information including a plurality of color channels, and can generate an appropriate first model.
　本発明の更に他の態様に係る医療画像学習装置において、複数の色チャンネルを有する第１医療画像から、１つの色チャンネルを有する第１医療画像を抽出する抽出部を備え、第１学習部は、抽出された１つの色チャンネルを有する第１画像群を用いて学習することが好ましい。複数の色チャンネルからなる第１画像群から１つの色チャンネルを有する第１医療画像を抽出し、抽出した１つの色チャンネルを有する第１画像群を用いて学習することで、複数の色チャンネルからなる色情報の影響を受けないエッジ等の基本的な特徴量を抽出することができ、適切な第１モデルを生成することができる。 In a medical image learning apparatus according to still another aspect of the present invention, the medical image learning apparatus further includes an extraction unit that extracts a first medical image having one color channel from a first medical image having a plurality of color channels, wherein the first learning unit includes It is preferable to perform learning using the first image group having one extracted color channel. A first medical image having one color channel is extracted from a first image group having a plurality of color channels, and learning is performed using the first image group having the extracted one color channel. It is possible to extract basic features such as edges which are not affected by the color information, and to generate an appropriate first model.
　本発明の更に他の態様に係る医療画像学習装置において、複数の色チャンネルを有する第１医療画像及び第２医療画像から、１つの色チャンネルを有する第１医療画像及び第２医療画像をそれぞれ抽出する抽出部を備え、第１学習部は、抽出された１つの色チャンネルを有する第１画像群及び第２画像群を用いて学習することが好ましい。 In a medical image learning apparatus according to still another aspect of the present invention, a first medical image and a second medical image having one color channel are respectively extracted from a first medical image and a second medical image having a plurality of color channels. It is preferable that the first learning unit learns using the first image group and the second image group having one extracted color channel.
　第１学習部での学習に第１画像群及び第２画像群を使用する場合、複数の色チャンネルからなる第１画像群及び第２画像群から、それぞれ１つの色チャンネルを有する第１医療画像及び第２医療画像を抽出し、抽出した１つの色チャンネルの画像群を学習に使用することで、第１モデルをより適正に生成できるようにしている。 When the first image group and the second image group are used for learning in the first learning unit, the first medical image having one color channel each from the first image group and the second image group including a plurality of color channels By extracting the second medical image and the second medical image and using the extracted image group of one color channel for learning, the first model can be more appropriately generated.
　本発明の更に他の態様に係る医療画像学習装置において、複数の色チャンネルは、３原色の３チャンネル、又は輝度信号及び２つの色差信号の３チャンネルであることが好ましい。 In the medical image learning apparatus according to still another aspect of the present invention, the plurality of color channels are preferably three channels of three primary colors or three channels of a luminance signal and two color difference signals.
　本発明の更に他の態様に係る医療画像学習装置において、第１モデルを元に、第１画像群を用いて学習することにより、第１医療画像に対する画像認識を行う第３モデルを生成する第３学習部と、を備えることが好ましい。第１モデルは、第１医療画像に対する画像認識を行う第３モデルを生成する場合にも使用することができる。 In the medical image learning device according to still another aspect of the present invention, a third model for performing image recognition on the first medical image by learning using the first image group based on the first model is generated. And a third learning unit. The first model can also be used when generating a third model for performing image recognition on the first medical image.
　本発明の更に他の態様に係る医療画像学習装置において、第１医療画像及び第２医療画像は、それぞれ内視鏡装置により撮像された画像であることが好ましい。 In the medical image learning device according to still another aspect of the present invention, the first medical image and the second medical image are each preferably an image captured by an endoscope device.
　本発明の更に他の態様に係る医療画像学習装置において、第１モデル及び第２モデルが、畳み込みニューラルネットワークで構成されることが好ましい。 に お い て In the medical image learning device according to still another aspect of the present invention, it is preferable that the first model and the second model are configured by a convolutional neural network.
　本発明の更に他の態様に係る医療画像学習方法は、通常光で撮像された複数の第１医療画像からなる第１画像群及び特殊光で撮像された複数の第２医療画像からなる第２画像群を準備するステップと、第１学習部が、第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成するステップと、第２学習部が、第１モデルを元に第２画像群を用いて学習することにより、第２医療画像に対する画像認識を行う第２モデルを生成するステップと、を含む。 A medical image learning method according to still another aspect of the present invention includes a first image group including a plurality of first medical images captured with normal light and a second image group including a plurality of second medical images captured with special light. An image group preparing step, a first learning section generating a first model for image recognition by learning using at least the first image group, and a second learning section generating a first model based on the first model. And generating a second model for performing image recognition on the second medical image by learning using the second image group.
　本発明の更に他の態様に係る医療画像学習方法において、画像処理部が、複数の色チャンネルを有する第１医療画像を、１つの色チャンネルを有する第１医療画像に変換するステップを含み、第１モデルを生成するステップは、変換された１つの色チャンネルを有する第１画像群を用いて学習することが好ましい。 In a medical image learning method according to still another aspect of the present invention, the image processing unit includes a step of converting a first medical image having a plurality of color channels into a first medical image having one color channel, In the step of generating one model, it is preferable that learning is performed using a first image group having one converted color channel.
　本発明の更に他の態様に係る医療画像学習方法において、画像処理部が、複数の色チャンネルを有する第１医療画像及び第２医療画像を、１つの色チャンネルを有する第１医療画像及び第２医療画像に変換するステップを含み、第１モデルを生成するステップは、変換された１つの色チャンネルを有する第１画像群及び第２画像群を用いて学習することが好ましい。 In the medical image learning method according to still another aspect of the present invention, the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having one color channel. Preferably, the step of generating the first model includes the step of converting into a medical image, and the step of generating the first model learns using the first image group and the second image group having one converted color channel.
　本発明の更に他の態様に係る医療画像学習方法において、抽出部が、複数の色チャンネルを有する第１医療画像から、１つの色チャンネルを有する第１医療画像を抽出するステップを含み、第１モデルを生成するステップは、抽出された１つの色チャンネルを有する第１画像群を用いて学習することが好ましい。 In the medical image learning method according to still another aspect of the present invention, the extracting unit includes a step of extracting a first medical image having one color channel from a first medical image having a plurality of color channels, In the step of generating the model, it is preferable that learning is performed using the first image group having one extracted color channel.
　本発明の更に他の態様に係る医療画像学習方法において、抽出部が、複数の色チャンネルを有する第１医療画像及び第２医療画像から、１つの色チャンネルを有する第１医療画像及び第２医療画像をそれぞれ抽出するステップを含み、第１モデルを生成するステップは、抽出された１つの色チャンネルを有する第１画像群及び第２画像群を用いて学習することが好ましい。 In the medical image learning method according to still another aspect of the present invention, the extraction unit extracts the first medical image and the second medical image having one color channel from the first medical image and the second medical image having a plurality of color channels. It is preferable that the step of generating the first model includes the step of extracting the images, and the step of generating the first model is performed using the first image group and the second image group having one extracted color channel.
　本発明の更に他の態様に係る医療画像学習プログラムは、通常光で撮像された複数の第１医療画像からなる第１画像群及び特殊光で撮像された複数の第２医療画像からなる第２画像群をそれぞれ取得する機能と、第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成する機能と、第１モデルを元に第２画像群を用いて学習することにより、第２医療画像に対する画像認識を行う第２モデルを生成する機能と、をコンピュータに実現させる。 A medical image learning program according to still another aspect of the present invention includes a first image group including a plurality of first medical images captured by normal light and a second image including a plurality of second medical images captured by special light. A function of acquiring an image group, a function of generating a first model for image recognition by learning using at least the first image group, and learning using a second image group based on the first model Thereby, the function of generating the second model for performing image recognition on the second medical image is realized by the computer.
　本発明によれば、比較的多く集めることができる通常光で撮像された第１医療画像からなる第１画像群を少なくとも用いて学習を行うことで、良好な第１モデルを生成し、この第１モデルを元に、特殊光で撮像された複数の第２医療画像からなる第２画像群を用いて学習を行うため、比較的データ数の少ない第２画像群からでも第２医療画像に対して適切な画像認識を行う第２モデルを生成することができる。 According to the present invention, a good first model is generated by performing learning using at least a first image group including first medical images captured by ordinary light, which can be collected in a relatively large amount. Based on one model, learning is performed using a second image group composed of a plurality of second medical images captured by special light, so that the second medical image can be obtained from the second image group having a relatively small number of data. Thus, a second model that performs appropriate image recognition can be generated.
図１は、本発明に係る医療画像学習装置のハードウエア構成の一例を示すブロック図である。FIG. 1 is a block diagram showing an example of a hardware configuration of a medical image learning device according to the present invention. 図２は、本発明に係る医療画像学習装置１０－１の第１実施形態を示すブロック図である。FIG. 2 is a block diagram showing a first embodiment of the medical image learning device 10-1 according to the present invention. 図３は、第１学習部３０の実施形態を示す機能ブロック図である。FIG. 3 is a functional block diagram illustrating an embodiment of the first learning unit 30. 図４は、本発明に係る医療画像学習装置１０－２の第２実施形態を示すブロック図である。FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention. 図５は、本発明に係る医療画像学習装置１０－３の第３実施形態を示すブロック図である。FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention. 図６は、本発明に係る医療画像学習装置１０－４の第４実施形態を示すブロック図である。FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention. 図７は、本発明に係る医療画像学習方法の実施形態を示すフローチャートである。FIG. 7 is a flowchart showing an embodiment of the medical image learning method according to the present invention. 図８は、図７に示したステップＳ１４の第１変形例を示すステップＳ１４－１を示す図である。FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG. 図９は、図７に示したステップＳ１４の第２変形例を示すステップＳ１４－２を示す図である。FIG. 9 is a view showing step S14-2 showing a second modification of step S14 shown in FIG.
　以下、添付図面に従って本発明に係る医療画像学習装置及び方法の好ましい実施形態について説明する。 Hereinafter, preferred embodiments of the medical image learning apparatus and method according to the present invention will be described with reference to the accompanying drawings.
　［医療画像学習装置のハードウエア構成］
　図１は、本発明に係る医療画像学習装置のハードウエア構成の一例を示すブロック図である。 [Hardware configuration of medical image learning device]
FIG. 1 is a block diagram showing an example of a hardware configuration of a medical image learning device according to the present invention.
　図１に示す医療画像学習装置１０としては、パーソナルコンピュータ又はワークステーションを使用することができ、本例の医療画像学習装置１０は、主として通信部１２と、大容量のストレージもしくは第１データベース１４、第２データベース１６と、操作部１８と、ＣＰＵ（Central Processing Unit）２０と、ＲＡＭ(Random Access Memory)２２と、ＲＯＭ(Read Only Memory)２４と、表示部２６とから構成されている。 A personal computer or a workstation can be used as the medical image learning device 10 shown in FIG. 1. The medical image learning device 10 of the present example mainly includes a communication unit 12, a large-capacity storage or a first database 14, It comprises a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display unit 26.
　通信部１２は、有線又は無線により外部装置との通信処理を行い、外部装置との間で情報のやり取りを行う部分である。 The communication unit 12 is a unit that performs communication processing with an external device by wire or wirelessly and exchanges information with the external device.
　第１データベース１４は、通常光で撮像された複数の第１医療画像（通常光画像）からなる第１画像群（通常光画像群）と、各通常光画像の正しい画像認識結果を示す正解データとからなる画像認識用（学習用の）第１データセットを保存し、第２データベース１６は、特殊光で撮像された複数の第２医療画像（特殊光画像）からなる第２画像群（特殊光画像群）と、各特殊光画像の正しい画像認識結果を示す正解データとからなる画像認識用（学習用の）第２データセットを保存している。 The first database 14 includes a first image group (normal light image group) including a plurality of first medical images (normal light images) captured by normal light, and correct data indicating a correct image recognition result of each normal light image. The second database 16 stores a second image group (special light image) including a plurality of second medical images (special light images) captured with special light. A second data set for image recognition (for learning), which includes a light image group) and correct data indicating a correct image recognition result of each special light image, is stored.
　ここで、通常光画像（第１医療画像）及び特殊光画像（第２医療画像）とは、内視鏡装置によりそれぞれ異なる光源下で撮像されたカラー画像である。 Here, the normal light image (first medical image) and the special light image (second medical image) are color images captured by the endoscope apparatus under different light sources.
　通常光は、可視光の全ての波長帯域の光がほぼ均等に混ざった光（白色光）であり、通常光画像は、通常観察に使用される。したがって、通常光画像群は、比較的多く集めることができる。 (4) The normal light is light (white light) in which light in all wavelength bands of visible light is almost uniformly mixed, and the normal light image is used for normal observation. Therefore, a relatively large number of ordinary light image groups can be collected.
　一方、特殊光は、１つの特定の波長帯域の光、又は複数の特定の波長帯域の光の組み合わせた、観察目的に応じた各種の波長帯域の光であり、白色の波長帯域よりも狭い帯域を有し、狭帯域観察（NBI(Narrow band imaging)、FICE(Flexible spectral imaging color enhancement)）に使用される。 On the other hand, the special light is light of various wavelength bands according to the observation purpose, which is a combination of light of one specific wavelength band or light of a plurality of specific wavelength bands, and a band narrower than the white wavelength band. And is used for narrow band observation (NBI (Narrow band imaging), FICE (Flexible spectrum imaging color enhancement)).
　特定の波長帯域の第１例は、例えば可視域の青色帯域又は緑色帯域である。この第１例の波長帯域は、３９０ｎｍ以上４５０ｎｍ以下又は５３０ｎｍ以上５５０ｎｍ以下の波長帯域を含み、且つ第１例の光は、３９０ｎｍ以上４５０ｎｍ以下又は５３０ｎｍ以上５５０ｎｍ以下の波長帯域内にピーク波長を有する。 第 A first example of the specific wavelength band is, for example, a blue band or a green band in a visible region. The wavelength band of the first example includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the first example has a peak wavelength in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. .
　特定の波長帯域の第２例は、例えば可視域の赤色帯域である。この第２例の波長帯域は、５８５ｎｍ以上６１５ｎｍ以下又は６１０ｎｍ以上７３０ｎｍ以下の波長帯域を含み、且つ第２例の光は、５８５ｎｍ以上６１５ｎｍ以下又は６１０ｎｍ以上７３０ｎｍ以下の波長帯域内にピーク波長を有する。 第 The second example of the specific wavelength band is, for example, a red band in a visible region. The wavelength band of the second example includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the second example has a peak wavelength in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm. .
　特定の波長帯域の第３例は、酸化ヘモグロビンと還元ヘモグロビンとで吸光係数が異なる波長帯域を含み、且つ第３例の光は、酸化ヘモグロビンと還元ヘモグロビンとで吸光係数が異なる波長帯域にピーク波長を有する。この第３例の波長帯域は、４００±１０ｎｍ、４４０±１０ｎｍ、４７０±１０ｎｍ、又は６００ｎｍ以上７５０ｎｍ以下の波長帯域を含み、且つ第３例の光は、上記４００±１０ｎｍ、４４０±１０ｎｍ、４７０±１０ｎｍ、又は６００ｎｍ以上７５０ｎｍ以下の波長帯域にピーク波長を有する。 The third example of the specific wavelength band includes a wavelength band having different absorption coefficients between oxyhemoglobin and reduced hemoglobin, and the light of the third example has a peak wavelength at a wavelength band having different absorption coefficients between oxyhemoglobin and reduced hemoglobin. Having. The wavelength band of the third example includes 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example includes the above 400 ± 10 nm, 440 ± 10 nm, 470 nm. It has a peak wavelength in a wavelength band of ± 10 nm, or 600 nm to 750 nm.
　特定の波長帯域の第４例は、生体内の蛍光物質が発する蛍光の観察（蛍光観察）に用いられ且つこの蛍光物質を励起させる励起光の波長帯域（３９０ｎｍから４７０ｎｍ）である。 The fourth example of the specific wavelength band is a wavelength band (390 nm to 470 nm) of excitation light used for observation of fluorescence emitted from a fluorescent substance in a living body (fluorescence observation) and for exciting this fluorescent substance.
　特定の波長帯域の第５例は、赤外光の波長帯域である。この第５例の波長帯域は、７９０ｎｍ以上８２０ｎｍ以下又は９０５ｎｍ以上９７０ｎｍ以下の波長帯域を含み、且つ第５例の光は、７９０ｎｍ以上８２０ｎｍ以下又は９０５ｎｍ以上９７０ｎｍ以下の波長帯域にピーク波長を有する。 The fifth example of the specific wavelength band is a wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.
　このような特定の波長帯域を有する特殊光下で撮像された特殊光画像は病変が見にくいため、表面構造を観察したい等の観察目的に応じた場合にしか使用されず、データ数が多くない。 特殊 The special light image captured under such special light having a specific wavelength band is difficult to see the lesion, so it is used only for an observation purpose such as observation of a surface structure and the number of data is not large.
　本例では、第１データベース１４に保存されている通常光画像群の第１データセットは、第２データベース１６に保存されている特殊光画像群の第２データセットよりも多く準備されているものとする。 In this example, the first data set of the normal light image group stored in the first database 14 is prepared more than the second data set of the special light image group stored in the second database 16. And
　また、第１データベース１４及び第２データベース１６において、各通常光画像及び各特殊光画像に関連付けて保存されている正解データは、例えば通常光画像及び特殊光画像内に写っている病変の種類、病変の位置を示したデータ、症例固有の識別情報などが考えられる。病変の分類においては、腫瘍性、非腫瘍性の２分類、NICE分類などが挙げられる。病変の位置を示すデータは、病変を囲む矩形の情報や、病変を覆い隠すようなマスクデータなどが考えられる。 In the first database 14 and the second database 16, the correct answer data stored in association with each normal light image and each special light image includes, for example, the type of a lesion shown in the normal light image and the special light image, Data indicating the position of the lesion, identification information unique to the case, and the like can be considered. The classification of lesions includes two classifications, neoplastic and non-neoplastic, and NICE classification. The data indicating the position of the lesion may be rectangular information surrounding the lesion or mask data that covers the lesion.
　本例では、第１データベース１４、第２データベース１６は、医療画像学習装置１０が備えているが、外部に設けられたものでもよい。この場合、通信部１２を介して外部のデータベースから学習用のデータセットを取得することができる。 In the present example, the first database 14 and the second database 16 are provided in the medical image learning device 10, but may be provided externally. In this case, a data set for learning can be obtained from an external database via the communication unit 12.
　操作部１８は、コンピュータに有線接続又は無線接続されるキーボード及びマウス等が用いられ、機械学習に当たって各種の操作入力を受け付ける。 The operation unit 18 uses a keyboard, a mouse, and the like that are connected to the computer by wire or wirelessly, and receives various operation inputs during machine learning.
　ＣＰＵ２０は、ＲＯＭ２４又は図示しないハードディスク装置等に記憶された各種のプログラム（本発明に係る医療画像学習プログラムを含む）を読み出し、各種の処理を実行する。ＲＡＭ２２は、ＣＰＵ２０の作業領域として使用され、読み出されたプログラムや各種のデータを一時的に記憶する記憶部として用いられる。 The CPU 20 reads various programs (including the medical image learning program according to the present invention) stored in the ROM 24 or a hard disk device (not shown) and executes various processes. The RAM 22 is used as a work area of the CPU 20, and is used as a storage unit for temporarily storing read programs and various data.
　表示部２６は、コンピュータに接続可能な液晶モニタ等の各種モニタが用いられ、操作部１８とともに、ユーザインターフェースの一部として使用される。 The display unit 26 includes various monitors such as a liquid crystal monitor that can be connected to a computer, and is used together with the operation unit 18 as part of a user interface.
　上記構成の医療画像学習装置１０は、操作部１８により指示入力によりＣＰＵ２０が、ＲＯＭ２４やハードディスク装置等に記憶されている医療画像学習プログラムを読み出し、医療画像学習プログラムを実行することにより、後述するように医療画像学習装置として機能する。 In the medical image learning apparatus 10 having the above configuration, the CPU 20 reads a medical image learning program stored in the ROM 24, a hard disk device, or the like in response to an instruction input from the operation unit 18 and executes the medical image learning program, as described later. Function as a medical image learning device.
　［医療画像学習装置の第１実施形態］
　図２は、本発明に係る医療画像学習装置１０－１の第１実施形態を示すブロック図であり、図１に示した医療画像学習装置１０の主要な機能を示す機能ブロック図である。 [First Embodiment of Medical Image Learning Apparatus]
FIG. 2 is a block diagram showing a first embodiment of the medical image learning device 10-1 according to the present invention, and is a functional block diagram showing main functions of the medical image learning device 10 shown in FIG.
　図２に示す医療画像学習装置１０－１は、第１学習部３０と第２学習部４０とを備えている。 The medical image learning device 10-1 shown in FIG. 2 includes a first learning unit 30 and a second learning unit 40.
　第１学習部３０は、第１データベース１４に保存された通常光画像のデータセットと、第２データベース１６に保存された特殊光画像のデータセットとを用いて学習することにより画像認識用の学習モデル（第１モデル）を生成する。本例では、学習モデルの一つである畳み込みニューラルネットワーク（ＣＮＮ：Convolution Neural Network）を構築する。 The first learning unit 30 performs learning using the data set of the normal light image stored in the first database 14 and the data set of the special light image stored in the second database 16, thereby performing learning for image recognition. Generate a model (first model). In this example, a convolutional neural network (CNN: Convolution Neural Network), which is one of the learning models, is constructed.
　図３は、第１学習部３０の実施形態を示す機能ブロック図である。 FIG. 3 is a functional block diagram showing an embodiment of the first learning unit 30.
　図３に示す第１学習部３０は、主としてＣＮＮ３２と、誤差算出部３４と、パラメータ更新部３６とから構成される。 3 The first learning unit 30 shown in FIG. 3 mainly includes a CNN 32, an error calculation unit 34, and a parameter update unit 36.
　ＣＮＮ３２は、例えば、医療画像に写っている病変の種類を画像認識する認識器に対応する部分であり、複数のレイヤー構造を有し、複数の重みパラメータを保持している。ＣＮＮ３２は、重みパラメータが初期値から最適値に更新されることで、未学習モデルから学習済みモデルに変化しうる。 The CNN 32 is, for example, a part corresponding to a recognizer that recognizes the type of a lesion appearing in a medical image, has a plurality of layer structures, and holds a plurality of weight parameters. The CNN 32 can change from an unlearned model to a learned model by updating the weight parameter from an initial value to an optimal value.
　このＣＮＮ３２は、入力層３２Ａと、畳み込み層とプーリング層から構成された複数セット、及び全結合層を有する中間層３２Ｂと、出力層３２Ｃとを備え、各層は複数の「ノード」が「エッジ」で結ばれる構造となっている。 The CNN 32 includes an input layer 32A, a plurality of sets including a convolutional layer and a pooling layer, an intermediate layer 32B having a fully connected layer, and an output layer 32C. Each layer has a plurality of “nodes” represented by “edges”. The structure is connected by
　入力層３２Ａには、学習対象である通常光画像１４Ａが入力される。 The normal light image 14A to be learned is input to the input layer 32A.
　中間層３２Ｂは、畳み込み層とプーリング層とを１セットとする複数セットと、全結合層とを有し、入力層から入力した画像から特徴を抽出する部分である。畳み込み層は、前の層で近くにあるノードにフィルタ処理し（フィルタを使用した畳み込み演算を行い）、「特徴マップ」を取得する。プーリング層は、畳み込み層から出力された特徴マップを縮小して新たな特徴マップとする。「畳み込み層」は、画像からのエッジ抽出等の特徴抽出の役割を担い、「プーリング層」は抽出された特徴が、平行移動などによる影響を受けないようにロバスト性を与える役割を担う。尚、中間層３２Ｂには、畳み込み層とプーリング層とを１セットとする場合に限らず、畳み込み層が連続する場合や正規化層も含まれる。 The intermediate layer 32B has a plurality of sets each including a convolutional layer and a pooling layer and a fully connected layer, and is a portion for extracting features from an image input from the input layer. The convolutional layer filters nearby nodes in the previous layer (performs a convolution operation using a filter) to obtain a “feature map”. The pooling layer reduces the feature map output from the convolutional layer to a new feature map. The “convolution layer” has a role of extracting features such as edge extraction from an image, and the “pooling layer” has a role of providing robustness so that the extracted features are not affected by translation or the like. Note that the intermediate layer 32B is not limited to the case where the convolutional layer and the pooling layer are set as one set, but also includes the case where the convolutional layer is continuous and the normalization layer.
　出力層３２Ｃは、中間層３２Ｂにより抽出された特徴に基づき医療画像に写っている病変の種類を分類する認識結果を出力する部分である。学習済みＣＮＮ３２では、例えば、医療画像を、「腫瘍性」、「非腫瘍性」、「その他」の３つのカテゴリに分類し、認識結果は、「腫瘍性」、「非腫瘍性」及び「その他」に対応する３つのスコア（３つのスコアの合計は１００％）として出力する。 The output layer 32C is a part that outputs a recognition result for classifying the type of lesion appearing in the medical image based on the features extracted by the intermediate layer 32B. In the learned CNN 32, for example, the medical image is classified into three categories of “neoplastic”, “non-neoplastic”, and “other”, and the recognition result is “neoplastic”, “non-neoplastic”, and “other”. Are output as the three scores corresponding to "" (the sum of the three scores is 100%).
　学習前のＣＮＮ３２の各畳み込み層に適用されるフィルタの係数やオフセット値、及び全結合層における次の層との接続の重みは、任意の初期値がセットされる。 初期 An arbitrary initial value is set for the filter coefficient and offset value applied to each convolutional layer of the CNN 32 before learning, and the connection weight between the fully connected layer and the next layer.
　誤差算出部３４は、ＣＮＮ３２の出力層３２Ｃから出力される認識結果と、通常光画像１４Ａに対する正解データとを取得し、両者間の誤差を算出する。誤差の算出方法は、例えばソフトマックスクロスエントロピー、シグモイドなどが考えられる。 The error calculator 34 acquires the recognition result output from the output layer 32C of the CNN 32 and the correct data for the normal light image 14A, and calculates an error between the two. As a method of calculating the error, for example, softmax cross entropy, sigmoid, or the like can be considered.
　パラメータ更新部３６は、誤差算出部３４により算出された誤差を元に、誤差逆伝播法によりＣＮＮ３０の重みパラメータを調整する。 The parameter update unit 36 adjusts the weight parameter of the CNN 30 by the error back propagation method based on the error calculated by the error calculation unit 34.
　このパラメータの調整処理を繰り返し行い、ＣＮＮ３２の出力と正解データとの差が小さくなるまで繰り返し学習を行う。 調整 The parameter adjustment process is repeatedly performed, and learning is repeatedly performed until the difference between the output of the CNN 32 and the correct answer data becomes small.
　第１学習部３０は、第１データベース１４に保存された通常光画像群の全てのデータセットを使用し、ＣＮＮ３２の各パラメータを最適化する学習を行うことで、学習済みモデル（第１モデル）を生成する。 The first learning unit 30 uses all data sets of the normal light image group stored in the first database 14 to perform learning for optimizing each parameter of the CNN 32, thereby obtaining a learned model (first model). Generate
　図２に戻って、第２学習部４０は、第１学習部３０により学習した学習済みモデルを元に（学習済みＣＮＮ３２のパラメータを初期値として使用し）、第２データベース１６に保存された特殊光画像のデータセットのみを用いて再度学習することにより、特殊光画像に対する画像認識を行う学習モデル（第２モデル）を生成する。 Returning to FIG. 2, the second learning unit 40 uses the learned model learned by the first learning unit 30 (using the parameters of the learned CNN 32 as initial values) and stores the special model stored in the second database 16. A learning model (second model) for performing image recognition on the special light image is generated by learning again using only the light image data set.
　第２学習部４０は、図３に示した第１学習部３０と同様に構成されるため、その詳細な説明は省略する。尚、第２学習部４０は、第１学習部３０と同一の学習部により構成されていてもよい。 The second learning unit 40 has the same configuration as the first learning unit 30 shown in FIG. 3, and thus a detailed description thereof will be omitted. Note that the second learning unit 40 may be configured by the same learning unit as the first learning unit 30.
　第１実施形態の医療画像学習装置１０－１によれば、第１学習部３０により比較的多く集めることができる通常光画像のデータセットを用いて学習（事前学習）を行うため、画像認識用の学習モデルを良好に生成することができる。そして、第２学習部４０は、第１学習部３０により生成された学習モデルを元に、特殊光画像群のみを用いて学習（「ファインチューニング」ともいう）を行うため、比較的データ数の少ない特殊光画像群からでも特殊光画像に対して適切な画像認識を行う学習モデルを生成することができる。 According to the medical image learning apparatus 10-1 of the first embodiment, learning (pre-learning) is performed using a data set of normal light images that can be collected by the first learning unit 30 in a relatively large amount. Can be satisfactorily generated. Since the second learning unit 40 performs learning (also referred to as “fine tuning”) using only the special light image group based on the learning model generated by the first learning unit 30, the second learning unit 40 has a relatively small number of data. A learning model for performing appropriate image recognition on a special light image can be generated from a small group of special light images.
　尚、第１実施形態の医療画像学習装置１０－１の第１学習部３０は、通常光画像のデータセットと特殊光画像のデータセットとを用いて学習するが、通常光画像のデータセットのみを用いて学習してもよい。 Note that the first learning unit 30 of the medical image learning apparatus 10-1 according to the first embodiment learns using the data set of the normal light image and the data set of the special light image. Learning may be performed by using.
　［医療画像学習装置の第２実施形態］
　図４は、本発明に係る医療画像学習装置１０－２の第２実施形態を示すブロック図である。尚、図４に示す医療画像学習装置１０－２において、図２に示した第１実施形態の医療画像学習装置１０－１と共通する部分には同一の符号を付し、その詳細な説明は省略する。 [Second Embodiment of Medical Image Learning Apparatus]
FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention. In the medical image learning device 10-2 shown in FIG. 4, the same parts as those of the medical image learning device 10-1 of the first embodiment shown in FIG. Omitted.
　図４に示す第２実施形態の医療画像学習装置１０－２は、主として画像処理部５０が追加されている点で、図２に示した第１実施形態の医療画像学習装置１０－１と相違する。 The medical image learning device 10-2 of the second embodiment shown in FIG. 4 differs from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an image processing unit 50 is added. I do.
　第１データベース１４に保存されている通常光画像及び第２データベース１６に保存されている特殊光画像は、それぞれ赤（Ｒ）、緑（Ｇ）、青（Ｂ）の３つの色チャンネルからなるカラー画像とする。 The normal light image stored in the first database 14 and the special light image stored in the second database 16 are respectively composed of three color channels of red (R), green (G), and blue (B). Images.
　画像処理部５０は、第１データベース１４に保存されているＲＧＢの３つの色チャンネルを有する通常光画像、及び第２データベース１６に保存されているＲＧＢの３つの色チャンネルを有する特殊光画像を、それぞれ１つの色チャンネルを有する通常光画像及び特殊光画像に変換し、変換後の１つの色チャンネルを有する通常光画像及び特殊光画像を、学習用のデータセットとして第１学習部３０に出力する。 The image processing unit 50 converts the normal light image having three color channels of RGB stored in the first database 14 and the special light image having three color channels of RGB stored in the second database 16 into Each of them is converted into a normal light image and a special light image having one color channel, and the converted normal light image and special light image having one color channel are output to the first learning unit 30 as a learning data set. .
　画像処理部５０は、ＲＧＢのカラー画像をモノクロ処理することで、１つの色チャンネルを有する通常光画像及び特殊光画像に変換する。 (4) The image processing unit 50 converts the RGB color image into a normal light image and a special light image having one color channel by performing monochrome processing.
　ここで、モノクロ処理の一例としては、例えば、ＲＧＢの色信号（Ｒ信号、Ｇ信号、Ｂ信号）から、次式により輝度信号（Ｙ信号）を生成する処理が考えられる。 Here, as an example of the monochrome process, for example, a process of generating a luminance signal (Y signal) from the RGB color signals (R signal, G signal, B signal) by the following equation can be considered.
　［数１］
　Ｙ＝０．３Ｒ＋０．５９Ｇ＋０．１１Ｂ
　第１学習部３０は、画像処理部５０により変換された１つの色チャンネルを有する通常光画像及び特殊光画像（モノクロ画像）を、学習用のデータセットとして取得し、取得したデータセットにより画像認識用の学習モデル（第１モデル）を生成する。 [Equation 1]
Y = 0.3R + 0.59G + 0.11B
The first learning unit 30 acquires a normal light image and a special light image (monochrome image) having one color channel converted by the image processing unit 50 as a learning data set, and performs image recognition using the acquired data set. A learning model (first model) is generated.
　第２学習部４０は、第１学習部３０により学習した学習済みＣＮＮのパラメータを初期値として使用し、第２データベース１６に保存されたＲＧＢの特殊光画像のデータセットのみを用いて再度学習することにより、特殊光画像に対する画像認識を行う学習モデルを生成する。 The second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB special light image data set stored in the second database 16. Thus, a learning model for performing image recognition on the special light image is generated.
　ここで、第１学習部３０での学習に用いるデータセットは、１チャンネルの画像（モノクロ画像）であるため、少なくともＣＮＮの最初の畳み込み層で使用されるフィルタも１チャンネルになる。一方、第２学習部４０での学習に用いるデータセットは、複数の色チャンネル（ＲＧＢ）の特殊光画像であるため、少なくともＣＮＮの最初の畳み込み層で使用されるフィルタも複数チャンネル（３チャンネル）になる。したがって、第２学習部４０のＣＮＮの最初の畳み込み層で使用される３チャンネルのフィルタには、それぞれ同じパラメータを初期値として使用する。 Here, since the data set used for learning in the first learning unit 30 is a one-channel image (monochrome image), the filter used in at least the first convolutional layer of the CNN also has one channel. On the other hand, since the data set used for learning in the second learning unit 40 is a special light image of a plurality of color channels (RGB), the filter used in at least the first convolution layer of the CNN also has a plurality of channels (three channels). become. Therefore, the same parameters are used as initial values for the three-channel filters used in the first convolutional layer of the CNN of the second learning unit 40, respectively.
　第２実施形態の医療画像学習装置１０－２によれば、通常光画像の特徴量を特殊光画像の特徴量に近づけて事前学習することができ、第２学習部４０での特殊光画像のデータセットのみによる再学習をより適切に行うことができる。 According to the medical image learning device 10-2 of the second embodiment, it is possible to perform pre-learning by making the feature amount of the normal light image close to the feature amount of the special light image. Re-learning using only the data set can be performed more appropriately.
　尚、第２実施形態の医療画像学習装置１０－２の画像処理部５０は、通常光画像及び特殊光画像をそれぞれモノクロ処理しているが、通常光画像のみをモノクロ処理し、第１学習部３０は、モノクロ処理された通常光画像のデータセットのみを用いて学習してもよい。 The image processing unit 50 of the medical image learning apparatus 10-2 of the second embodiment performs monochrome processing on the normal light image and the special light image, respectively. The learning 30 may be performed using only the data set of the normal light image subjected to the monochrome processing.
　また、画像処理部５０は、輝度信号からなる「白黒画像」を生成する場合に限らず、単一色（１チャンネル）の画像を生成するものであればよく、特殊光画像の色相に近似したモノクロ画像を生成することが好ましい。 Further, the image processing unit 50 is not limited to the case of generating a “black-and-white image” including a luminance signal, but may be any unit that generates a single color (one channel) image. Preferably, an image is generated.
　［医療画像学習装置の第３実施形態］
　図５は、本発明に係る医療画像学習装置１０－３の第３実施形態を示すブロック図である。尚、図５に示す医療画像学習装置１０－３において、図２に示した第１実施形態の医療画像学習装置１０－１と共通する部分には同一の符号を付し、その詳細な説明は省略する。 [Third Embodiment of Medical Image Learning Apparatus]
FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention. In the medical image learning device 10-3 shown in FIG. 5, the same parts as those of the medical image learning device 10-1 of the first embodiment shown in FIG. Omitted.
　図５に示す第３実施形態の医療画像学習装置１０－３は、主として抽出部６０が追加されている点で、図２に示した第１実施形態の医療画像学習装置１０－１と相違する。 The medical image learning device 10-3 of the third embodiment shown in FIG. 5 is different from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an extraction unit 60 is added. .
　第１データベース１４に保存されている通常光画像及び第２データベース１６に保存されている特殊光画像は、それぞれＲＧＢの３つの色チャンネルからなるカラー画像とする。 The normal light image stored in the first database 14 and the special light image stored in the second database 16 are color images each including three color channels of RGB.
　抽出部６０は、第１データベース１４に保存されているＲＧＢの３つの色チャンネルを有する通常光画像、及び第２データベース１６に保存されているＲＧＢの３つの色チャンネルを有する特殊光画像から、それぞれ１つの色チャンネルを有する通常光画像及び特殊光画像を抽出する。 The extraction unit 60 extracts a normal light image having three color channels of RGB stored in the first database 14 and a special light image having three color channels of RGB stored in the second database 16, respectively. A normal light image and a special light image having one color channel are extracted.
　ここで、第２データベース１６に保存されている特殊光画像が、青色又は青紫の波長帯域にピーク波長を有する特殊光により撮像された内視鏡画像の場合、抽出部６０は、Ｂの１チャンネルのみの通常光画像及び特殊光画像を抽出することが好ましい。また、第２データベース１６に保存されている特殊光画像が、赤色又は近赤外の波長帯域にピーク波長を有する特殊光により撮像された内視鏡画像の場合、抽出部６０は、Ｒの１チャンネルのみの通常光画像及び特殊光画像を抽出することが好ましい。即ち、抽出部６０は、ＲＧＢの３つの色チャンネルを有する通常光画像及び特殊光画像から、特殊光画像の色味に近い１つの色チャンネルの画像を抽出することが好ましい。 Here, when the special light image stored in the second database 16 is an endoscope image captured by special light having a peak wavelength in a blue or blue-violet wavelength band, the extraction unit 60 performs one channel of B It is preferable to extract only the normal light image and the special light image. When the special light image stored in the second database 16 is an endoscope image captured by special light having a peak wavelength in a red or near-infrared wavelength band, the extraction unit 60 determines whether or not R is 1 It is preferable to extract the normal light image and the special light image of only the channel. That is, it is preferable that the extraction unit 60 extracts an image of one color channel close to the tint of the special light image from the normal light image and the special light image having three color channels of RGB.
　第１学習部３０は、抽出部６０により抽出された１つの色チャンネルを有する通常光画像及び特殊光画像を、学習用のデータセットとして取得し、取得したデータセットにより画像認識用の学習モデル（第１モデル）を生成する。 The first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a learning data set, and uses the acquired data set to perform a learning model for image recognition ( (First model).
　第２学習部４０は、第１学習部３０により学習した学習済みＣＮＮのパラメータを初期値として使用し、第２データベース１６に保存されたＲＧＢの特殊光画像のデータセットのみを用いて再度学習することにより、特殊光画像に対する画像認識を行う学習モデルを生成する。 The second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB special light image data set stored in the second database 16. Thus, a learning model for performing image recognition on the special light image is generated.
　第３実施形態の医療画像学習装置１０－３によれば、通常光画像の特徴量を特殊光画像の特徴量に近づけて事前学習することができ、第２学習部４０での特殊光画像のデータセットのみによる再学習をより適切に行うことができる。 According to the medical image learning apparatus 10-3 of the third embodiment, the feature amount of the normal light image can be made closer to the feature amount of the special light image to perform pre-learning. Re-learning using only the data set can be performed more appropriately.
　尚、第３実施形態の医療画像学習装置１０－３の抽出部６０は、ＲＧＢの通常光画像及び特殊光画像から１つの色チャンネルを有する通常光画像及び特殊光画像を抽出しているが、ＲＧＢの通常光画像から１つの色チャンネルを有する通常光画像のみを抽出し、第１学習部３０は、抽出された１つの色チャンネルを有する通常光画像のデータセットのみを用いて学習してもよい。 The extraction unit 60 of the medical image learning device 10-3 according to the third embodiment extracts a normal light image and a special light image having one color channel from the RGB normal light image and the special light image. Even if only the normal light image having one color channel is extracted from the RGB normal light image, the first learning unit 30 may perform learning using only the data set of the extracted normal light image having one color channel. Good.
　［医療画像学習装置の第４実施形態］
　図６は、本発明に係る医療画像学習装置１０－４の第４実施形態を示すブロック図である。尚、図６に示す医療画像学習装置１０－４において、図４に示した第２実施形態の医療画像学習装置１０－２と共通する部分には同一の符号を付し、その詳細な説明は省略する。 [Fourth Embodiment of Medical Image Learning Apparatus]
FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention. In the medical image learning device 10-4 shown in FIG. 6, the same parts as those in the medical image learning device 10-2 of the second embodiment shown in FIG. Omitted.
　図６に示す第４実施形態の医療画像学習装置１０－４は、主として第３学習部７０が追加されている点で、図４に示した第２実施形態の医療画像学習装置１０－２と相違する。 The medical image learning device 10-4 of the fourth embodiment shown in FIG. 6 differs from the medical image learning device 10-2 of the second embodiment shown in FIG. 4 mainly in that a third learning unit 70 is added. Different.
　第３学習部７０は、第１学習部３０により学習した学習済みＣＮＮのパラメータを初期値として使用し、第１データベース１４に保存されたＲＧＢの通常光画像のデータセットのみを用いて再度学習することにより、通常光画像に対する画像認識を行う学習モデル（第３モデル）を生成する。 The third learning unit 70 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and performs learning again using only the RGB normal light image data set stored in the first database 14. Thus, a learning model (third model) for performing image recognition on the normal light image is generated.
　第４実施形態の医療画像学習装置１０－４によれば、第１学習部３０により学習した学習済みＣＮＮのパラメータを初期値として使用するため、第３学習部７０は、任意の初期値から通常光画像のデータセットのみを用いて学習する場合に比べて学習時間の短縮化を図ることができる。 According to the medical image learning device 10-4 of the fourth embodiment, the parameters of the learned CNN learned by the first learning unit 30 are used as the initial values. The learning time can be reduced as compared with the case where learning is performed using only the data set of the optical image.
　尚、第３学習部７０は、図５に示した第３実施形態の医療画像学習装置１０－３の第１学習部３０により学習した学習済みＣＮＮのパラメータを初期値として使用するようにしてもよい。 Note that the third learning unit 70 may use the parameter of the learned CNN learned by the first learning unit 30 of the medical image learning device 10-3 of the third embodiment shown in FIG. 5 as an initial value. Good.
　本実施形態の第１データベース１４及び第２データベース１６に保存されている通常光画像及び特殊光画像は、それぞれＲＧＢの「光の３原色」のカラー画像としたが、これに限らず、ＲＧＢの補色の関係にあるシアン（Ｃ）、マゼンタ（Ｍ）、イエロー（Ｙ）の「色の３原色」のカラー画像でもよい。 The normal light image and the special light image stored in the first database 14 and the second database 16 of the present embodiment are color images of RGB “three primary colors”, respectively, but are not limited thereto. A color image of “the three primary colors” of cyan (C), magenta (M), and yellow (Y) which have a complementary color relationship may be used.
　また、第１データベース１４及び第２データベース１６に保存されている通常光画像及び特殊光画像は、ＲＧＢの色信号から生成される輝度信号（Ｙ）と２つの色差信号（Ｃｒ，Ｃｂ）からなる３チャンネルのカラー画像でもよい。 The normal light image and the special light image stored in the first database 14 and the second database 16 include a luminance signal (Y) generated from RGB color signals and two color difference signals (Cr, Cb). It may be a three-channel color image.
　尚、ＲＧＢ、ＣＭＹ、ＹＣｒＣｂのカラー画像は、相互に変換できることは言うまでもない。 も な い Needless to say, color images of RGB, CMY, and YCrCb can be mutually converted.
　［医療画像学習方法］
　図７は、本発明に係る医療画像学習方法の実施形態を示すフローチャートであり、図２に示した第１実施形態の医療画像学習装置１０－１の各部の処理手順に関して示している。 [Medical image learning method]
FIG. 7 is a flowchart showing an embodiment of the medical image learning method according to the present invention, and shows a processing procedure of each unit of the medical image learning apparatus 10-1 of the first embodiment shown in FIG.
　第１データベース１４及び第２データベース１６には、学習用の通常光画像群のデータセット及び特殊光画像群のデータセットが事前に格納される（データセットを準備するステップ）。 デ ー タ The first database 14 and the second database 16 previously store a data set of a normal light image group for learning and a data set of a special light image group (step of preparing a data set).
　第１学習部３０は、第１データベース１４及び第２データベース１６から通常光画像群のデータセット及び特殊光画像群のデータセットを取得する（ステップＳ１０、Ｓ１２）。第１学習部３０は、１０～１００前後のミニバッチ単位でデータセットを取得してもよいし、１枚ずつ取得するようにしてもよい。 The first learning unit 30 acquires a data set of the normal light image group and a data set of the special light image group from the first database 14 and the second database 16 (Steps S10 and S12). The first learning unit 30 may acquire a data set in units of mini-batch of about 10 to 100, or may acquire data sets one by one.
　第１学習部３０は、取得した通常光画像及び特殊光画像のデータセットを用いて学習することにより画像認識用の学習モデル（第１モデル）を生成する（ステップＳ１４）。本例では、学習モデルの一つである学習済みＣＮＮを構築する。 The first learning unit 30 generates a learning model (first model) for image recognition by performing learning using the acquired data sets of the normal light image and the special light image (step S14). In this example, a learned CNN, which is one of the learning models, is constructed.
　続いて、第２学習部４０は、第１学習部３０により学習した学習済みモデルを元に（学習済みＣＮＮのパラメータを初期値として使用し）、第２データベース１６に保存された特殊光画像のデータセットのみを用いて再度学習することにより、特殊光画像に対する画像認識を行う学習モデル（第２モデル）を生成する（ステップＳ１６）。 Subsequently, the second learning unit 40 uses the learned model learned by the first learning unit 30 (using the parameter of the learned CNN as an initial value) to generate the special light image stored in the second database 16. By learning again using only the data set, a learning model (second model) for performing image recognition on the special light image is generated (step S16).
　この実施形態の医療画像学習方法によれば、第１学習部３０により比較的多く集めることができる通常光画像のデータセットを用いて学習（事前学習）を行うため、画像認識用の学習モデルを良好に生成することができる。そして、第２学習部４０は、第１学習部３０により生成された学習モデルを元に、特殊光画像群のみを用いて再学習を行うため、比較的データ数の少ない特殊光画像群からでも特殊光画像に対して適切な画像認識を行う学習モデルを生成することができる。 According to the medical image learning method of this embodiment, since learning (prior learning) is performed using a data set of normal light images that can be collected by the first learning unit 30 in a relatively large amount, a learning model for image recognition is used. It can be produced well. Then, the second learning unit 40 performs re-learning using only the special light image group based on the learning model generated by the first learning unit 30, so that even the special light image group having a relatively small number of data is used. A learning model for performing appropriate image recognition on a special light image can be generated.
　尚、ステップＳ１４では、第１学習部３０は、通常光画像及び特殊光画像のデータセットを用いて学習することにより画像認識用の学習モデルを生成するが、通常光画像のデータセットのみを用いて学習してもよい。 In step S14, the first learning unit 30 generates a learning model for image recognition by learning using the data set of the normal light image and the special light image, but uses only the data set of the normal light image. You may learn.
　＜第１変形例＞
図８は、図７に示したステップＳ１４の第１変形例を示すステップＳ１４－１を示す図である。 <First Modification>
FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG.
　図８において、画像処理部５０（図４）は、第１データベース１４に保存されているＲＧＢの３つの色チャンネルを有する通常光画像、及び第２データベース１６に保存されているＲＧＢの３つの色チャンネルを有する特殊光画像を、それぞれ１つの色チャンネルを有する通常光画像及び特殊光画像（モノクロ画像）に変換する（ステップＳ２０）。 8, the image processing unit 50 (FIG. 4) includes a normal light image having three color channels of RGB stored in the first database 14 and three colors of RGB stored in the second database 16. The special light image having the channel is converted into a normal light image and a special light image (monochrome image) each having one color channel (step S20).
　第１学習部３０は、画像処理部５０により変換されたモノクロ画像を、学習用のデータセットとして取得し、取得したデータセットにより画像認識用の学習モデルの一つであるＣＮＮの学習モデル（第１モデル）を生成する（ステップＳ２２）。尚、ステップＳ２２により学習された学習済みＣＮＮのパラメータは、図７に示すステップＳ１６において、第２学習部４０のＣＮＮの初期値として使用される。 The first learning unit 30 acquires the monochrome image converted by the image processing unit 50 as a data set for learning, and uses the acquired data set as a learning model of CNN (one of the learning models for image recognition). (1 model) is generated (step S22). Note that the parameter of the learned CNN learned in step S22 is used as an initial value of the CNN of the second learning unit 40 in step S16 shown in FIG.
　＜第２変形例＞
図９は、図７に示したステップＳ１４の第２変形例を示すステップＳ１４－２を示す図である。 <Second modification>
FIG. 9 is a view showing step S14-2 showing a second modification of step S14 shown in FIG.
　図９において、抽出部６０（図５）は、第１データベース１４に保存されているＲＧＢの３つの色チャンネルを有する通常光画像、及び第２データベース１６に保存されているＲＧＢの３つの色チャンネルを有する特殊光画像から、それぞれ１つの色チャンネルを有する通常光画像及び特殊光画像を抽出する（ステップＳ３０）。抽出部６０は、ＲＧＢの３つの色チャンネルを有する通常光画像及び特殊光画像から、特殊光画像の色味に近い１つの色チャンネルの画像を抽出することが好ましい。 In FIG. 9, the extraction unit 60 (FIG. 5) includes a normal light image having three color channels of RGB stored in the first database 14, and three color channels of RGB stored in the second database 16. The normal light image and the special light image each having one color channel are extracted from the special light image having (Step S30). It is preferable that the extraction unit 60 extracts an image of one color channel close to the tint of the special light image from the normal light image and the special light image having three color channels of RGB.
　第１学習部３０は、抽出部６０により抽出された１つの色チャンネルを有する通常光画像及び特殊光画像を、学習用のデータセットとして取得し、取得したデータセットにより画像認識用の学習モデルの一つであるＣＮＮの学習モデル（第１モデル）を生成する（ステップＳ３２）。尚、ステップＳ３２により学習された学習済みＣＮＮのパラメータは、図７に示すステップＳ１６において、第２学習部４０のＣＮＮの初期値として使用される。 The first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a data set for learning, and generates a learning model for image recognition using the obtained data set. A learning model (first model) of one CNN is generated (step S32). The parameter of the learned CNN learned in step S32 is used as an initial value of the CNN of the second learning unit 40 in step S16 shown in FIG.
　［その他］
　図３に示したＣＮＮ３２は、医療画像に写っている病変の種類を画像認識する学習モデルであるが、医療画像に写っている病変の位置（病変領域）を認識するセグメンテーションを行う学習モデルでもよい。この場合のＣＮＮは、ＣＮＮの一種である全層畳み込みネットワーク（ＦＣＮ：Fully Convolution Network）を適用し、医療画像に写っている病変の位置を画素レベルで把握できるものが好ましい。 [Others]
The CNN 32 shown in FIG. 3 is a learning model for recognizing the type of lesion shown in the medical image, but may be a learning model for performing segmentation for recognizing the position (lesion area) of the lesion shown in the medical image. . In this case, it is preferable that the CNN apply a full-layer convolution network (FCN: Fully Convolution Network), which is a type of the CNN, and can grasp the position of a lesion appearing in a medical image at a pixel level.
　また、本発明は、例えばＤＢＮ（Deep Belief Network）、ＳＶＭ（Support Vector Machine）などのＣＮＮ以外の機械学習のモデルにも適用できる。 The present invention is also applicable to machine learning models other than CNN, such as DBN (Deep Belief Network) and SVM (Support Vector Machine).
　更に本実施形態の医療画像学習装置１０の各種制御を実行するハードウエア的な構造は、次に示すような各種のプロセッサ（processor）である。各種のプロセッサには、ソフトウェア（プログラム）を実行して各種の制御部として機能する汎用的なプロセッサであるＣＰＵ（Central Processing Unit）、ＦＰＧＡ（Field Programmable Gate Array）などの製造後に回路構成を変更可能なプロセッサであるプログラマブルロジックデバイス（Programmable Logic Device：ＰＬＤ）、ＡＳＩＣ（Application Specific Integrated Circuit）などの特定の処理を実行させるために専用に設計された回路構成を有するプロセッサである専用電気回路などが含まれる。 The hardware structure for executing various controls of the medical image learning apparatus 10 of the present embodiment is various processors as described below. For various processors, the circuit configuration can be changed after manufacturing such as CPU (Central Processing Unit) and FPGA (Field Programmable Gate Array), which are general-purpose processors that execute software (programs) and function as various control units. Special-purpose circuits such as a programmable logic device (Programmable Logic Device: PLD), an ASIC (Application Specific Integrated Circuit), and a dedicated electric circuit having a circuit configuration specifically designed to execute a specific process are included. It is.
　１つの処理部は、これら各種のプロセッサのうちの１つで構成されていてもよいし、同種又は異種の２つ以上のプロセッサ（例えば、複数のＦＰＧＡ、あるいはＣＰＵとＦＰＧＡの組み合わせ）で構成されてもよい。また、複数の制御部を１つのプロセッサで構成してもよい。複数の制御部を１つのプロセッサで構成する例としては、第１に、クライアントやサーバなどのコンピュータに代表されるように、１つ以上のＣＰＵとソフトウェアの組合せで１つのプロセッサを構成し、このプロセッサが複数の制御部として機能する形態がある。第２に、システムオンチップ（System On Chip：ＳｏＣ）などに代表されるように、複数の制御部を含むシステム全体の機能を１つのＩＣ（Integrated Circuit）チップで実現するプロセッサを使用する形態がある。このように、各種の制御部は、ハードウエア的な構造として、上記各種のプロセッサを１つ以上用いて構成される。 One processing unit may be configured with one of these various processors, or configured with two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which a processor functions as a plurality of control units. Secondly, as represented by a system-on-chip (SoC), a form in which a processor that realizes the functions of the entire system including a plurality of control units by one IC (Integrated Circuit) chip is used. is there. As described above, the various control units have a hardware structure using one or more of the above-described various processors.
　更にまた、これらの各種のプロセッサのハードウエア的な構造は、より具体的には、半導体素子などの回路素子を組み合わせた電気回路（circuitry）である。 Furthermore, the hardware structure of these various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.
　また、本発明は、コンピュータにインストールされることにより、本発明に係る医療画像学習装置として機能させる医療画像学習プログラム、及びこの医療画像学習プログラムが記録された記録媒体を含む。 The present invention also includes a medical image learning program that is installed in a computer to function as the medical image learning device according to the present invention, and a recording medium on which the medical image learning program is recorded.
　更に、本発明は上述した実施形態に限定されず、本発明の精神を逸脱しない範囲で種々の変形が可能であることは言うまでもない。 Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.
１０、１０－１、１０－２、１０－３、１０－４　医療画像学習装置
１２　通信部
１４　第１データベース
１４Ａ　通常光画像
１６　第２データベース
１８　操作部
２０　ＣＰＵ
２２　ＲＡＭ
２４　ＲＯＭ
２６　表示部
３０　第１学習部
３２Ａ　入力層
３２Ｂ　中間層
３２Ｃ　出力層
３４　誤差算出部
３６　パラメータ更新部
４０　第２学習部
５０　画像処理部
６０　抽出部
７０　第３学習部
Ｓ１０、Ｓ１２、Ｓ１４、Ｓ１４－１、Ｓ１４－２、Ｓ１６、Ｓ２０、Ｓ２２、Ｓ３０、Ｓ３２　ステップ 10, 10-1, 10-2, 10-3, 10-4 Medical image learning device 12 Communication unit 14 First database 14A Normal light image 16 Second database 18 Operation unit 20 CPU
22 RAM
24 ROM
26 display unit 30 first learning unit 32A input layer 32B intermediate layer 32C output layer 34 error calculation unit 36 parameter update unit 40 second learning unit 50 image processing unit 60 extraction unit 70 third learning unit S10, S12, S14, S14- 1, S14-2, S16, S20, S22, S30, S32 Step

Claims (19)

  1. 　通常光で撮像された複数の第１医療画像からなる第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成する第１学習部と、
    　前記第１モデルを元に、特殊光で撮像された複数の第２医療画像からなる第２画像群を用いて学習することにより、前記第２医療画像に対する画像認識を行う第２モデルを生成する第２学習部と、
    　を備えた医療画像学習装置。     A first learning unit that generates a first model for image recognition by learning using at least a first image group including a plurality of first medical images captured by normal light;
    Based on the first model, a second model for performing image recognition on the second medical image is generated by learning using a second image group including a plurality of second medical images captured by special light. A second learning unit;
    Medical image learning device equipped with
  2. 　前記第１学習部は、前記第２医療画像も用いて前記第１モデルを生成する請求項１に記載の医療画像学習装置。 The medical image learning device according to claim 1, wherein the first learning unit generates the first model using the second medical image.
  3. 　複数の色チャンネルを有する前記第１医療画像を、１つの色チャンネルを有する前記第１医療画像に変換する画像処理部を備え、
    　前記第１学習部は、前記変換された１つの色チャンネルを有する前記第１画像群を用いて学習する請求項１に記載の医療画像学習装置。     An image processing unit that converts the first medical image having a plurality of color channels into the first medical image having one color channel;
    The medical image learning device according to claim 1, wherein the first learning unit learns using the first image group having the converted one color channel.
  4. 　前記画像処理部は、前記複数の色チャンネルを有する前記第１医療画像を、輝度信号のみの前記第１医療画像に変換することで、前記１つの色チャンネルを有する前記第１医療画像とする請求項３に記載の医療画像学習装置。 The image processing unit may convert the first medical image having the plurality of color channels into the first medical image having only a luminance signal to be the first medical image having the one color channel. Item 4. The medical image learning device according to Item 3.
  5. 　複数の色チャンネルを有する前記第１医療画像及び前記第２医療画像を、それぞれ１つの色チャンネルを有する前記第１医療画像及び前記第２医療画像に変換する画像処理部を備え、
    　前記第１学習部は、前記変換された１つの色チャンネルを有する前記第１画像群及び前記第２画像群を用いて学習する請求項２に記載の医療画像学習装置。     An image processing unit that converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image each having one color channel,
    The medical image learning device according to claim 2, wherein the first learning unit performs learning using the first image group and the second image group having the converted one color channel.
  6. 　前記画像処理部は、前記複数の色チャンネルを有する前記第１医療画像及び前記第２医療画像を、輝度信号のみの前記第１医療画像及び前記第２医療画像に変換することで、前記１つの色チャンネルを有する前記第１医療画像及び前記第２医療画像とする請求項５に記載の医療画像学習装置。 The image processing unit converts the first medical image and the second medical image having the plurality of color channels into the first medical image and the second medical image including only a luminance signal, thereby converting the one medical image and the second medical image. The medical image learning device according to claim 5, wherein the first medical image and the second medical image having a color channel are used.
  7. 　複数の色チャンネルを有する前記第１医療画像から、１つの色チャンネルを有する前記第１医療画像を抽出する抽出部を備え、
    　前記第１学習部は、前記抽出された１つの色チャンネルを有する前記第１画像群を用いて学習する請求項１に記載の医療画像学習装置。     An extraction unit that extracts the first medical image having one color channel from the first medical image having a plurality of color channels,
    The medical image learning device according to claim 1, wherein the first learning unit performs learning using the first image group having the extracted one color channel.
  8. 　複数の色チャンネルを有する前記第１医療画像及び前記第２医療画像から、１つの色チャンネルを有する前記第１医療画像及び前記第２医療画像をそれぞれ抽出する抽出部を備え、
    　前記第１学習部は、前記抽出された１つの色チャンネルを有する前記第１画像群及び前記第２画像群を用いて学習する請求項２に記載の医療画像学習装置。     An extraction unit that extracts the first medical image and the second medical image having one color channel from the first medical image and the second medical image having a plurality of color channels, respectively.
    The medical image learning device according to claim 2, wherein the first learning unit learns using the first image group and the second image group having the extracted one color channel.
  9. 　前記複数の色チャンネルは、３原色の３チャンネル、又は輝度信号及び２つの色差信号の３チャンネルである請求項３から８のいずれか１項に記載の医療画像学習装置。 The medical image learning device according to any one of claims 3 to 8, wherein the plurality of color channels are three channels of three primary colors or three channels of a luminance signal and two color difference signals.
  10. 　前記第１モデルを元に、前記第１画像群を用いて学習することにより、前記第１医療画像に対する画像認識を行う第３モデルを生成する第３学習部と、
    　を備えた請求項３から９のいずれか１項に記載の医療画像学習装置。     A third learning unit configured to generate a third model for performing image recognition on the first medical image by learning using the first image group based on the first model;
    The medical image learning device according to any one of claims 3 to 9, comprising:
  11. 　前記第１医療画像及び前記第２医療画像は、それぞれ内視鏡装置により撮像された画像である請求項１から１０のいずれか１項に記載の医療画像学習装置。 The medical image learning device according to any one of claims 1 to 10, wherein the first medical image and the second medical image are images captured by an endoscope device.
  12. 　前記第１モデル及び前記第２モデルが、畳み込みニューラルネットワークで構成される請求項１から１１のいずれか１項に記載の医療画像学習装置。 The medical image learning device according to any one of claims 1 to 11, wherein the first model and the second model are configured by a convolutional neural network.
  13. 　通常光で撮像された複数の第１医療画像からなる第１画像群及び特殊光で撮像された複数の第２医療画像からなる第２画像群を準備するステップと、
    　第１学習部が、前記第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成するステップと、
    　第２学習部が、前記第１モデルを元に前記第２画像群を用いて学習することにより、前記第２医療画像に対する画像認識を行う第２モデルを生成するステップと、
    　を含む医療画像学習方法。     Preparing a first image group consisting of a plurality of first medical images taken with normal light and a second image group consisting of a plurality of second medical images taken with special light;
    A step in which a first learning unit generates a first model for image recognition by learning using at least the first image group;
    A second learning unit for learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
    Medical image learning method including:
  14. 　画像処理部が、複数の色チャンネルを有する前記第１医療画像を、１つの色チャンネルを有する前記第１医療画像に変換するステップを含み、
    　前記第１モデルを生成するステップは、前記変換された１つの色チャンネルを有する前記第１画像群を用いて学習する請求項１３に記載の医療画像学習方法。     An image processing unit including converting the first medical image having a plurality of color channels into the first medical image having one color channel;
    14. The medical image learning method according to claim 13, wherein the step of generating the first model learns using the first image group having the converted one color channel.
  15. 　画像処理部が、複数の色チャンネルを有する前記第１医療画像及び前記第２医療画像を、１つの色チャンネルを有する前記第１医療画像及び前記第２医療画像に変換するステップを含み、
    　前記第１モデルを生成するステップは、前記変換された１つの色チャンネルを有する前記第１画像群及び前記第２画像群を用いて学習する請求項１３に記載の医療画像学習方法。     An image processing unit including converting the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having one color channel,
    The medical image learning method according to claim 13, wherein the step of generating the first model learns using the first image group and the second image group having the converted one color channel.
  16. 　抽出部が、複数の色チャンネルを有する前記第１医療画像から、１つの色チャンネルを有する前記第１医療画像を抽出するステップを含み、
    　前記第１モデルを生成するステップは、前記抽出された１つの色チャンネルを有する前記第１画像群を用いて学習する請求項１３に記載の医療画像学習方法。     An extracting unit for extracting the first medical image having one color channel from the first medical image having a plurality of color channels,
    14. The medical image learning method according to claim 13, wherein the step of generating the first model learns using the first image group having the extracted one color channel.
  17. 　抽出部が、複数の色チャンネルを有する前記第１医療画像及び前記第２医療画像から、１つの色チャンネルを有する前記第１医療画像及び前記第２医療画像をそれぞれ抽出するステップを含み、
    　前記第１モデルを生成するステップは、前記抽出された１つの色チャンネルを有する前記第１画像群及び前記第２画像群を用いて学習する請求項１３に記載の医療画像学習方法。     An extracting unit that extracts the first medical image and the second medical image each having one color channel from the first medical image and the second medical image having a plurality of color channels,
    14. The medical image learning method according to claim 13, wherein the step of generating the first model learns using the first image group and the second image group having the extracted one color channel.
  18. 　通常光で撮像された複数の第１医療画像からなる第１画像群及び特殊光で撮像された複数の第２医療画像からなる第２画像群をそれぞれ取得する機能と、
    　前記第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成する機能と、
    　前記第１モデルを元に、前記第２画像群を用いて学習することにより、前記第２医療画像に対する画像認識を行う第２モデルを生成する機能と、
    　をコンピュータに実現させる医療画像学習プログラム。     A function of acquiring a first image group composed of a plurality of first medical images captured with normal light and a second image group composed of a plurality of second medical images captured with special light, respectively;
    A function of generating a first model for image recognition by learning using at least the first image group;
    A function of generating a second model for performing image recognition on the second medical image by learning using the second image group based on the first model;
    Medical image learning program to make a computer realize
  19. 　非一時的かつコンピュータ読取可能な記録媒体であって、前記記録媒体に格納された指令がコンピュータによって読み取られた場合に、
    　通常光で撮像された複数の第１医療画像からなる第１画像群及び特殊光で撮像された複数の第２医療画像からなる第２画像群をそれぞれ取得する機能と、
    　前記第１画像群を少なくとも用いて学習することにより画像認識用の第１モデルを生成する機能と、
    　前記第１モデルを元に、前記第２画像群を用いて学習することにより、前記第２医療画像に対する画像認識を行う第２モデルを生成する機能と、
    　を含む医療画像学習機能をコンピュータに実行させる記録媒体。     A non-transitory and computer-readable recording medium, wherein the instructions stored in the recording medium are read by a computer,
    A function of acquiring a first image group composed of a plurality of first medical images captured with normal light and a second image group composed of a plurality of second medical images captured with special light, respectively;
    A function of generating a first model for image recognition by learning using at least the first image group;
    A function of generating a second model for performing image recognition on the second medical image by learning using the second image group based on the first model;
    A recording medium for causing a computer to execute a medical image learning function including:
PCT/JP2019/022909 2018-06-28 2019-06-10 Medical image learning device, method, and program WO2020003991A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2020527358A JP7187557B2 (en) 2018-06-28 2019-06-10 MEDICAL IMAGE LEARNING APPARATUS, METHOD AND PROGRAM

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018123442 2018-06-28
JP2018-123442 2018-06-28

Publications (1)

Publication Number Publication Date
WO2020003991A1 true WO2020003991A1 (en) 2020-01-02

Family

ID=68986566

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/022909 WO2020003991A1 (en) 2018-06-28 2019-06-10 Medical image learning device, method, and program

Country Status (2)

Country Link
JP (1) JP7187557B2 (en)
WO (1) WO2020003991A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021140600A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, endoscope system, and image processing method
WO2021140602A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, learning device and learning method
WO2021140601A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, endoscope system, and image processing method
WO2021166749A1 (en) * 2020-02-18 2021-08-26 ソニー・オリンパスメディカルソリューションズ株式会社 Learning device and medical image processing device
WO2022071264A1 (en) * 2020-09-29 2022-04-07 テルモ株式会社 Program, model generation method, information processing device, and information processing method
JP2022059598A (en) * 2020-02-07 2022-04-13 カシオ計算機株式会社 Identification apparatus, identification device learning method, and program
WO2024009631A1 (en) * 2022-07-06 2024-01-11 富士フイルム株式会社 Image processing device, and method for operating image processing device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003033324A (en) * 1991-03-11 2003-02-04 Olympus Optical Co Ltd Endscope device
JP2005065976A (en) * 2003-08-22 2005-03-17 Olympus Corp Endoscopic apparatus
WO2017175282A1 (en) * 2016-04-04 2017-10-12 オリンパス株式会社 Learning method, image recognition device, and program
JP2017221486A (en) * 2016-06-16 2017-12-21 ソニー株式会社 Information processing device, information processing method, program, and medical observation system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003033324A (en) * 1991-03-11 2003-02-04 Olympus Optical Co Ltd Endscope device
JP2005065976A (en) * 2003-08-22 2005-03-17 Olympus Corp Endoscopic apparatus
WO2017175282A1 (en) * 2016-04-04 2017-10-12 オリンパス株式会社 Learning method, image recognition device, and program
JP2017221486A (en) * 2016-06-16 2017-12-21 ソニー株式会社 Information processing device, information processing method, program, and medical observation system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PENG, XISHUAI ET AL.: "Multi-modal scene categorization using multi-tasks learning", 2014 12TH INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING (ICSP),, 6 November 2016 (2016-11-06), pages 1106 - 1111, XP033076126, ISSN: 2164-5221, DOI: 10.1109/ICSP.2016.7878000 *
SONOYAMA, SHOJI ET AL.: "Transfer Learning for Bag-of-Visual words approach to NBI endoscopic image classification", 37TH ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY, 25 August 2015 (2015-08-25), pages 785 - 788, XP032810308, DOI: 10.1109/EMBC.2015.7318479 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021140600A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, endoscope system, and image processing method
JPWO2021140600A1 (en) * 2020-01-09 2021-07-15
WO2021140602A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, learning device and learning method
WO2021140601A1 (en) * 2020-01-09 2021-07-15 オリンパス株式会社 Image processing system, endoscope system, and image processing method
JP7346596B2 (en) 2020-01-09 2023-09-19 オリンパス株式会社 Image processing system and program
JP7429715B2 (en) 2020-01-09 2024-02-08 オリンパス株式会社 Image processing system, endoscope system, image processing system operating method and program
JP2022059598A (en) * 2020-02-07 2022-04-13 カシオ計算機株式会社 Identification apparatus, identification device learning method, and program
JP7331913B2 (en) 2020-02-07 2023-08-23 カシオ計算機株式会社 Discrimination device, discriminator learning method and program
WO2021166749A1 (en) * 2020-02-18 2021-08-26 ソニー・オリンパスメディカルソリューションズ株式会社 Learning device and medical image processing device
WO2022071264A1 (en) * 2020-09-29 2022-04-07 テルモ株式会社 Program, model generation method, information processing device, and information processing method
WO2024009631A1 (en) * 2022-07-06 2024-01-11 富士フイルム株式会社 Image processing device, and method for operating image processing device

Also Published As

Publication number Publication date
JP7187557B2 (en) 2022-12-12
JPWO2020003991A1 (en) 2021-06-24

Similar Documents

Publication Publication Date Title
WO2020003991A1 (en) Medical image learning device, method, and program
Anubala et al. Performance Analysis of Hookworm Detection using Deep Convolutional Neural Network
AU2018282455B2 (en) Diagnosis assisting device, and image processing method in diagnosis assisting device
US8345936B2 (en) Multispectral iris fusion for enhancement and interoperability
Sinthanayothin Image analysis for automatic diagnosis of diabetic retinopathy
JP7005767B2 (en) Endoscopic image recognition device, endoscopic image learning device, endoscopic image learning method and program
JP7019815B2 (en) Learning device
US11948080B2 (en) Image processing method and image processing apparatus
CA3153067C (en) Picture-detecting method and apparatus
Zhang et al. Computerized facial diagnosis using both color and texture features
Wang et al. Facial image medical analysis system using quantitative chromatic feature
CN114145012A (en) Image processing apparatus, image pickup apparatus, method, and program
JP7087390B2 (en) Diagnostic support device, image processing method and program
CN116012895A (en) Palm feature recognition equipment, recognition method thereof and storage medium
Reza et al. Automatic detection of optic disc in fundus images by curve operator
JP7291389B2 (en) Object identification method, information processing device, information processing program, and lighting device
TWI803223B (en) Method for detecting object of esophageal cancer in hyperspectral imaging
CN110807739A (en) Image color feature processing method, system and device for target detection and storage medium
Hashim et al. A study on RGB color extraction of psoriasis lesion using principle component analysis (PCA)
JP6585623B2 (en) Biological information measuring device, biological information measuring method, and biological information measuring program
WO2022044425A1 (en) Learning device, learning method, program, learned model, and endoscope system
CN114511567B (en) Tongue body and tongue coating image identification and separation method
Constantinou et al. A comparison of color correction algorithms for endoscopic cameras
JP7449004B2 (en) Hyperspectral object image detection method using frequency bands
US20230215003A1 (en) Learning apparatus, learning method, image processing apparatus, endoscope system, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19825651

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020527358

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19825651

Country of ref document: EP

Kind code of ref document: A1