JP2022187833A - Machine learning apparatus, control method of the same, and program - Google Patents

Abstract

To provide a machine learning apparatus capable of performing accurate diagnosis and pathological quantification even when images captured under various environmental lights or by various imaging apparatus are used for machine learning.SOLUTION: A disclosed machine learning apparatus comprises obtaining means for obtaining an affected area image obtained by photographing an affected area of a patient and chromaticity information indicating chromaticity when the affected area image is obtained, and inference means for inferring a diagnosis result for the affected area using the affected area image and chromaticity information by a trained machine learning model.SELECTED DRAWING: Figure 1

Description

The present invention relates to a machine learning device, its control method and program.

In recent years, there has been known a technique using a machine learning model such as deep learning in order to perform diagnosis and pathological quantification using medical images obtained by photographing a subject. In general, learning a machine learning model such as deep learning requires a large amount of learning data, and a large amount of images collected as learning data can be captured under various environmental lights or by various imaging devices. In other words, the large number of images used as training data are affected by various ambient light and the characteristics of the photographic equipment. There is Furthermore, in some cases, the colors close to the primary colors are processed or corrected to improve the appearance of the image.

Regarding the generation of such image data, Patent Literature 1 discloses a technique of performing white balance correction on an image to keep the white color of the image captured under different ambient light or under an imaging device at a constant chromaticity. ing.

JP 2020-96219 A

By the way, when making a diagnosis or pathological quantification using medical images, for example, when the affected area is skin, there are cases where redness is used as an indicator to determine how severe the disease is or how much blood flow increases. be. Also, in a stained microscopic image, blue may be used as an indicator.

Under different ambient light or imaging equipment, the shades of images of affected areas with colors close to the primary colors are different. There are challenges that cannot be effectively improved. In this regard, Japanese Patent Application Laid-Open No. 2002-200000 discloses a technique for matching the white color of a photographed image generated by applying white balance correction, but it does not consider how to handle images with variations as learning data. I didn't.

The present invention has been made in view of the above problems, and an object of the present invention is to achieve accurate diagnosis and pathological quantification even when images captured under various environmental lights or by various imaging devices are used for machine learning. It is to realize a technology that can make it possible.

In order to solve this problem, for example, the machine learning device of the present invention has the following configuration. That is, an acquisition means for acquiring an affected area image obtained by photographing an affected area of a patient and chromaticity information indicating the chromaticity when the affected area image is obtained, and a learned machine learning model obtains the affected area image and the chromaticity. and an inference means for inferring a diagnosis result for the affected area using the information.

According to the present invention, accurate diagnosis and pathological quantification can be performed even when images captured under various environmental lights or by various imaging devices are used for machine learning.

1 is a diagram showing an outline of an information processing system according to an embodiment of the present invention; FIG. FIG. 2 is a block diagram showing a hardware configuration example of each device in the information processing system according to the first embodiment; FIG. 2 is a diagram showing a functional configuration example of each device in the information processing system according to the first embodiment; FIG. 4 is a diagram for conceptually explaining a learning model according to the first embodiment; FIG. 4 is a diagram for explaining an overview of the operation of the information processing system according to the first embodiment; 4 is a flowchart showing a series of operations related to acquisition of learning data in Embodiment 1; Flowchart showing a series of operations related to inference and re-learning in Embodiment 1 Diagram for explaining the chromaticity coordinate diagram A diagram for explaining changes in chromaticity due to ambient light FIG. 4 is a diagram showing a configuration example of learning data according to the first embodiment;

(Embodiment 1)
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(Overview of information processing system)
An outline of an information processing system according to the present embodiment will be described with reference to FIG.

The clinic 10 is, for example, a hospital that diagnoses patients. For example, in the clinic 10, a digital camera 101, a client terminal 103, a light source 151, a chromaticity meter 102, and a reference color chart 152 are arranged. A digital camera 101 is an example of an imaging device capable of capturing an image. The client terminal 103 is, for example, a personal computer, and is an example of an information processing apparatus capable of transmitting images and measurement data received from the digital camera 101 and colorimeter 102 to an external server. The reference color chart 152 is composed of, for example, white, primary color red, primary color blue, and primary color green charts. The reference color chart 152 is placed near the patient or the affected part when the digital camera 101 photographs the affected part of the patient 153 . Further, when the object to be photographed is a transmission image of a sample on a glass plate, a transmission chart may be used. The chromaticity meter 102 is a measuring instrument for measuring the chromaticity of the reference color chart 152 .

In the clinic 10, the digital camera 101 generates an image of the affected area of the patient 153 under the illumination of the light source 151, and the chromaticity meter 102 measures the chromaticity of the reference color chart 152 under the same illumination. to measure. The client terminal 103 receives the input of the diagnosis results of the patient 153 examined by the doctor 154 . In this embodiment, the diagnosis result includes, for example, the dosage of the medicine to be provided to the affected area, but if the content of the diagnosis is made by a doctor based on the condition of the affected area (to be imaged), other information is included. may be The client terminal 103 communicates with the digital camera 101 and the chromaticity meter 102 by wire or wirelessly, and obtains the measured chromaticity information (also simply referred to as chromaticity information) of the reference color of each panel and the photographed affected area image. receive. The client terminal 103 then transmits the photographed affected area image, chromaticity information, and diagnosis results (also referred to as an affected area data set) to the data server 108 via the Internet 106 and the local area network 107 .

The data server 108 is an example of a server device that holds learning data used in machine learning and input data to be inferred in inference processing. Data (affected area data set) transmitted from the client terminal 103 of the clinic 10 is stored in a database, for example, and the stored data is provided in response to a request from the inference server 109 . That is, the reasoning server 109 can access the lesion data set on the data server 108 via the local area network 107 . The inference server 109 is an example of a machine learning device that performs inference processing and learning processing using machine learning. The inference server 109 performs inference processing for estimating a diagnosis result based on the accessed affected area image and chromaticity information, and processing for learning a machine learning model based on the affected area data set. There may be a plurality of clinics that are similar in configuration to clinic 10, for example, another clinic is indicated by clinic 11. FIG.

(Hardware configuration example of each device)
Next, with reference to FIG. 2, a hardware configuration example of each device constituting the information processing system 10 will be described. The digital camera 101 includes a CPU 201 , a ROM 202 , a RAM 203 , a shooting operation section 204 , a system bus 205 , a communication I/F 206 , an imaging section 207 , a display section 208 , a camera engine 209 and a storage section 210 . The CPU 201 may include one or more processors, and controls operations of the digital camera 101 by reading and executing programs stored in the ROM 202 . The CPU 201 receives a user's operation via the shooting operation unit 204 and instructs the imaging unit 207 to shoot. An image captured by the imaging unit 207 is temporarily stored in the RAM 203 , then subjected to image processing by the camera engine 209 and stored in the storage unit 210 . In this embodiment, the storage unit 210 stores, for example, affected area image data obtained by photographing an affected area of a patient. The stored image data is transmitted to the client terminal 103 via the communication I/F unit 206 . Each block in the digital camera 101 is interconnected through the system bus 205 .

Colorimeter 102 includes CPU 221 , ROM 222 , RAM 223 , operation unit 224 , system bus 230 , communication I/F 226 , light receiving unit 227 , display unit 208 , and calculation unit 229 . The CPU 221 may include one or more processors, and controls the operation of the colorimeter 102 by reading and executing programs stored in the ROM 222 . In the chromaticity meter 102, the CPU 221 receives a user's operation via the operation unit 224 and instructs the light receiving unit 227 to receive light. The light receiving section 227 includes a light receiving element for acquiring color information. The color information obtained by the light receiving unit 227 is temporarily stored in the RAM 223 , then subjected to arithmetic processing by the arithmetic unit 229 and stored in the RAM 223 . The stored chromaticity information includes, for example, x-coordinate values and y-coordinate values in a color space (chromaticity coordinate system) defined by CIE1931. The stored chromaticity information is transmitted to client terminal 103 via communication I/F unit 226 . Each block in colorimeter 102 is interconnected through system bus 230 .

The client terminal 103 includes a CPU 212 , HDD 213 , RAM 214 , NIC 215 , input section 217 , display section 218 and system bus 219 . The CPU 221 may include one or more processors, and controls operations of the client terminal 103 by reading and executing programs stored in the HDD 213 . The CPU 212 receives the diseased part image data from the digital camera 101 via the communication I/F unit 216 and stores it in the RAM 214 . Also, the CPU 212 receives chromaticity information from the chromaticity meter 102 via the communication I/F 216 and stores it in the RAM 214 . The input unit 217 receives diagnosis results from the doctor 154 who diagnosed the affected area of the patient 153 . A diagnosis result is stored in the RAM 214 . Diagnosis results will be described later. The affected area image data, chromaticity information, and diagnostic results (that is, affected area data set) collected by the client terminal 103 are transmitted to the data server 108 via the network interface card (NIC) 216 . Each block in the client terminal 103 is interconnected through the system bus 219 .

Data server 108 includes CPU 251 , HDD 252 , RAM 253 , NIC 254 and system bus 255 . The data server 108 is configured to be able to communicate with client terminals of various clinics located over a wide area via the Internet 106 . Data server 108 and reasoning server 109 are connected via local area network 107 to exchange large amounts of data quickly. The data server 108 receives data from one or more client terminals 103 via the NIC 254 and stores the received data (affected area image data, chromaticity information, and diagnosis results) in the HDD 252 . CPU 251 may include one or more processors, and controls operations of data server 108 by reading and executing programs stored in HDD 252 . Each block in data server 108 is interconnected through system bus 255 .

The inference server 109 includes a CPU 261 , HDD 262 , RAM 263 , NIC 264 , system bus 265 , GPU 266 , input section 267 and display section 268 . The CPU 261 may include one or more processors, and controls operations of the inference server 109 by reading and executing programs stored in the HDD 262 . In addition, the CPU 261 executes inference and re-learning processes independently or in cooperation with the GPU 266 using programs stored in the HDD 262 . The CPU 261 and GPU 266 access the diseased part image data, chromaticity information and diagnostic results stored in the data server 108 via the NIC 264 and local area network 107 . The GPU 266 is a processor that can perform efficient operations by processing data in parallel. When learning is performed multiple times using a machine learning model for deep learning, it is effective to perform processing on the GPU 266 . For example, the processing by the learning unit 327 may use the GPU 266 in addition to the CPU 261 . When executing a learning program including a learning model, the CPU 261 and the GPU 266 may learn by performing calculations in cooperation. In addition, the processing of the learning unit 327 may be performed only by the CPU 261 or the GPU 266 . Also, the estimation unit 325 may use the GPU 266 in the same manner as the learning unit 327 . A device specialized for machine learning may be used other than the GPU.

(Example of software configuration of each device)
Next, a software configuration example of each device will be described with reference to FIG. Each of the software configurations shown in FIG. 3 is implemented by hardware resources such as the CPU in the hardware configuration example shown in FIG. 2 executing programs. One or more of the functional blocks shown in FIG. 3 may be implemented by hardware such as an ASIC or programmable logic array (PLA), or by a combination of software and hardware resources.

A configuration example of the digital camera 101 will be described. The digital camera 101 includes a camera control section 301 , an image processing section 302 , an image file generation section 303 and a data transmission/reception section 304 . A camera control unit 301 is realized by, for example, a program stored in the CPU 201 and the ROM 202, and issues instructions to each functional block for controlling the behavior of the digital camera 101. FIG. The image processing unit 302 processes the image captured by the imaging unit 207 into a color image. The image file generation unit 303 generates the color image processed by the image processing unit 302 as an image data file according to the image file format. The image processing unit 302 and the image file generation unit 303 are implemented by, for example, a hardware processing circuit of the camera engine 209 and programs stored in the ROM 202 . The data transmission/reception unit 304 transmits the image data file to the client terminal 103 .

A configuration example of the chromaticity meter 102 will be described. The chromaticity meter 102 includes a chromaticity meter controller 305 , a light receiver 306 , a chromaticity calculator 307 , and a data transmitter/receiver 308 . The chromaticity meter control unit 305 is realized by the program stored in the ROM 222 and the CPU 221 and issues instructions to each functional block for controlling the behavior of the chromaticity meter 102 . The color received by the light receiving unit 227 is converted into chromaticity information by the chromaticity calculation unit 307 . Chromaticity calculator 307 is implemented by a program stored in ROM 222 and CPU 221 . A data transmission/reception unit 308 transmits the chromaticity information to the client terminal 103 .

A configuration example of the client terminal 103 will be described. The client terminal control unit 309 is implemented by the programs stored in the HDD 213 and the CPU 212 . The input unit 310 is implemented by an input unit 217 such as a keyboard and a program stored in the HDD 213, and receives the input of the diagnosis results of the doctor 154 examining the affected part of the patient 153. FIG. The client terminal control unit 309 sends the image data file received from the digital camera 101, the chromaticity information received from the chromaticity meter 102, and the diagnosis result as an affected area data set to the data server 108 via the data transmission/reception unit 311. Send.

A configuration example of the data server 108 will be described. The data server control unit 323 is realized by programs stored in the CPU 251 and the HDD 252 . Data collecting/providing unit 321 is implemented by programs stored in NIC 254 , CPU 251 and HDD 252 . The data collecting/providing unit 321 receives data in response to a request from the client terminal 103 or another server, stores the data in the data storage unit 322, retrieves the data stored in the data storage unit 322, and requests the data. Send to the original device. The data collecting/providing unit 321 also holds, as a set of data, associations among image data files, chromaticity information, and diagnosis results. The data storage unit 322 is realized by the programs stored in the CPU 251 and the HDD 252 and the HDD 252 , and holds image data files, chromaticity information, and diagnosis results received from the client terminal 103 .

A configuration example of the inference server 109 will be described. The inference server control unit 328 is implemented by the programs stored in the HDD 262 and the CPU 261 . The inference server controller 328 periodically monitors whether new data has arrived at the data server 108 . When new data arrives at the data server 108, the estimator 325 is instructed to estimate the new data. It should be noted that the estimation instruction may be given by the user via the input unit 267 and the result may be displayed on the display unit 268 . The data transmission/reception unit 324 has a data acquisition function of acquiring from the data server 108 a file of affected area image data, chromaticity information associated with the affected area image data, and diagnostic results associated with the affected area image data. The inference server control unit 328 refers to the affected area image data file, the chromaticity information, and the diagnosis result stored in the data server 108 . The estimation unit 325 is realized by the program stored in the HDD 262 , the CPU 261 and the GPU 266 . The estimation unit 325 performs estimation processing based on the affected area image data file and the chromaticity information stored in the data server 108, and obtains an estimation result. The estimation result is a result of estimating a doctor's diagnosis result based on the affected area image, and is, for example, a numerical value such as a dosage estimated from the affected area image, but may be another quantitative estimation result. The difference calculation unit 326 calculates the difference between the estimation result by the estimation unit 325 and the diagnosis result. If the inference server control unit 328 determines that the difference is greater than the threshold, it instructs the learning unit 327 to relearn the inference model 401 .

FIG. 4 schematically shows the relationship among input data, an inference model, and output data when using the machine learning model according to this embodiment. The inference model 401 is, for example, a neural network having an input layer, one or more intermediate layers (also called hidden layers), and an output layer. The affected area image data 402 and the chromaticity information 403 form input data and are input to the input layer of the inference model 401 . Dosage 405 is output data and is output from the output layer of inference model 401 . The doctor's diagnosis result 404 is used as teaching data. Note that the neural network of the inference model shown in FIG. 4 is shown schematically, and the number of neurons shown in FIG. 4 does not necessarily correspond to the input data and output data.

The inference model 401 uses a trained machine learning model to estimate the dosage from the affected area image data (that is, the inference stage). Outputs the dose 405 to be administered. Also, in order to make the inference model 401 learn, the doctor's diagnosis result 404 corresponding to the teacher data is referred to and compared with the output of the inference model. That is, the doctor's diagnosis result 404 is compared with the dosage output by the inference model 401 based on the affected area image data 402 and the chromaticity information 403 . The result of comparison between the doctor's diagnosis result and the dosage is used, for example, to calculate the inference error of the inference model. At the stage of learning the inference model, the internal parameters (for example, weights of the neural network) of the inference model 401 under learning are gradually changed and optimized based on the inference error.

(Outline of information processing system operation)
Next, an overview of the operation of the information processing system 10 according to this embodiment will be described with reference to FIG. First, in the clinic 10 , the affected part of the patient 153 is photographed under the light source 151 . In S<b>501 , the chromaticity meter 102 acquires chromaticity information of the reference color chart 152 . The reference color chart 152 has, for example, white, a primary color of red, a primary color of blue, and a primary color of green, and the chromaticity meter 102 obtains the respective chromaticity values of each chart. Also, the chromaticity meter 102 transmits chromaticity information to the client terminal 103 . In S<b>502 , the digital camera 101 captures an image of the affected area of the patient 153 . Also, the digital camera 101 transmits the photographed affected area image to the client terminal 103 . In S<b>503 , the client terminal 103 acquires the diagnosis result of the doctor 154 examining the patient 153 . In S<b>504 , the client terminal 103 uploads the chromaticity information of the reference color chart, the affected area image, and the diagnosis result to the data server 108 . In S505, when new data is uploaded to the data server 108, the inference server 109 performs inference processing on the affected area image and outputs the diagnosis result. In S506, the inference server 109 calculates the difference between the diagnosis output in S505 and the diagnosis by the doctor obtained in S503. In S507, if the difference exceeds the threshold, the inference server 109 inputs the chromaticity information of the reference color chart, the affected area image, and the diagnosis result to the inference model to perform re-learning.

(A series of operations related to acquisition of learning data in a clinic)
Next, a series of operations related to learning data acquisition will be described with reference to FIG. 6A. Note that this operation is started, for example, when the doctor 154 issues an imaging instruction to the digital camera 101 and an acquisition instruction of chromaticity information to the chromaticity meter via the client terminal 103 . Unless otherwise specified, each operation of the digital camera 101, the colorimeter 102, and the client terminal 103 is realized by each CPU (CPU 201, CPU 221, and CPU 212) executing each program and controlling each part. be.

In S601, the chromaticity meter 102 acquires the chromaticity of the reference color chart 152 placed near the patient. For example, if the basic color chart is a four-color chart, the chromaticity meter 102 acquires chromaticity information for four colors. In S<b>602 , the chromaticity meter 102 transfers the acquired chromaticity information to the client terminal 103 . The client terminal 103 stores the chromaticity information acquired from the chromaticity meter 102 in the internal HDD 213 .

In S603, the digital camera 101 captures an affected part image including the affected part of the patient 153 in accordance with the imaging instruction. In S604, the digital camera 101 transfers the image to the client terminal and saves it.

In S605, the client terminal 103 determines whether shooting has ended. For example, if the instruction from the doctor 154 is an instruction to photograph a plurality of affected areas, and the specified number of photographs have not been taken, the client terminal 103 determines that the photographing has not been completed, and proceeds to step S609. proceed to If the client apparatus 103 determines that the shooting has ended, the process advances to step S606.

In S<b>606 , the client terminal 103 receives the diagnosis result of the diagnosis performed by the doctor 154 with reference to the photographed affected area image. The diagnosis result includes, for example, the optimal dosage of medicine to be provided to the imaged affected area, as will be described later with reference to FIG. The dosage is entered, for example, "36" mg for prescribing medicine M001.

In S<b>607 , the doctor 154 saves the diagnosis result in the client terminal 103 . At this time, the client terminal 103 stores the diagnosis result in association with the corresponding affected area image data and chromaticity information. The client terminal 103 saves the diagnosis result input in S606 to the HDD 213, for example, in response to receiving an instruction to complete the input of the diagnosis result from the doctor 154, or in response to receiving an instruction to save the diagnosis result. Save to

In S608, the client terminal 103 uploads the chromaticity information of the reference color chart 152, the affected area image data, and the diagnosis result (affected area data set) to the data server 108 in response to an upload instruction from the doctor 154, for example. This series of operations ends after the client terminal 103 uploads the affected area data set to the data server 108 .

In S609, the client terminal 103 determines whether there is a change in the imaging device or ambient light. For example, the client terminal 103 returns the process to S601 to photograph and colorize the affected area again if there is a change in the affected area imaging equipment or ambient light compared to when the affected area was imaged in S603. The client terminal 103 advances to step S603 to continue photographing if there is no change in the photographing equipment or ambient light.

In the present embodiment, the case where the client terminal 103 transmits the affected area data set to the data server 108 each time the client terminal 103 receives the doctor's diagnosis result has been described as an example. However, the client terminal 103 may transmit to the data server 108 a plurality of affected area data sets stored within a predetermined period of time.

(Series of operations related to inference and re-learning)
Next, a series of operations related to inference and re-learning will be described with reference to FIG. 6B. Note that the machine learning model of the inference server 109 has already been learned using a predetermined affected area data set (a data set with the number of samples necessary for optimizing the inference model) stored in the data server 108 . Also, this operation is started, for example, when a new affected area data set is stored in the data server 108 from the client terminal 103 . Further, unless otherwise specified, each operation of the data server 108 and the inference server 109 is realized by each CPU (CPU 251, CPU 261) executing each program and controlling each part.

In S<b>651 , the inference server 109 reads the affected area image newly stored in the data server 108 . For example, in response to receiving from the data server 108 the notification information notifying of the arrival of the affected area data set, the inference server 109 transmits a request to the data server 108 to read out the affected area data set. Receive image data.

In S652, the chromaticity information of the reference color associated with the affected area image stored in the data server 108 is read. For example, when the inference server 109 receives from the data server 108 the notification information for notifying the arrival of the affected area data set, the inference server 109 transmits the chromaticity information included in the affected area data set together with the affected area image data in addition to the affected area image data. , from the data server 108 .

In S653, the inference server 109 further causes the GPU 266 to perform image diagnosis based on the affected area image using the inference model of the machine learning model. That is, the inference server 109 infers a diagnosis result (for example, dosage of medicine to be prescribed in this embodiment) based on the image of the affected area.

In S654, the inference server 109 reads from the data server 108 the diagnosis result associated with the affected area image read in S651. Specifically, for example, the inference server 109 requests the data server 108 for the diagnosis result associated with the affected area image read in S<b>651 and receives the corresponding diagnosis result from the data server 108 .

In S655, the inference server 109 calculates the difference between the inference result obtained in S653 and the diagnosis result obtained in S654. For example, if the inference result is the dosage of medicine, the difference between the dosage inferred in S653 and the dosage stored in the data server 108 (correct dosage entered by the doctor 154) is calculate.

In S656, the inference server 109 determines whether the difference calculated in S655 is greater than or equal to the threshold. If the difference calculated in S655 is equal to or greater than the threshold, the inference server 109 causes the difference between the inference result by the machine learning model and the diagnosis result by the doctor to be large. proceed. If the difference calculated in S655 does not exceed the threshold, the inference server 109 terminates this series of operations.

In S657, the inference server 109 causes the GPU 266 to re-learn the machine learning model using the affected area data set of the data server 108. FIG. For example, the inference server 109 generates a predetermined affected area data set (data set with the number of samples required for optimizing the inference model) stored in the data server 108, which includes the affected area data set related to the affected area image acquired in S651. re-learn using As described above, when the inference result differs from the doctor's diagnosis result by a predetermined amount or more, the inference model can be re-learned to obtain a highly accurate inference model.

(Example of learning data)
Next, an example of learning data according to this embodiment will be described with reference to FIG. In the data structure shown in FIG. 7, among the file of the affected area image, the chromaticity information, and the diagnostic result held by the data server 108 as a set of data (a set of affected area data sets), the data of the chromaticity information and the diagnostic result are It shows an example of the structure to be stored. In this example, the substantive data of the image data is referenced by the image ID. The chromaticity information may store chromaticity format information indicating the chromaticity format, and chromaticity values of the reference colors red, green, blue, and white.

For example, in cases of epidermis, lesions that collect blood near the skin are observed, and symptoms are diagnosed based on the reddish color, shape, and distribution. In burns, the depth of the affected area and the state of bleeding are diagnosed based on the redness of the affected area. Therefore, it is possible to estimate the diagnosis result (for example, the dosage of medicine) based on the condition of the affected part by learning the image of the affected part and the results of the actual diagnosis of the affected part by the doctor. be. However, in this embodiment, by further inputting the chromaticity information of the reference color to the inference model, the inference model can estimate the diagnosis result based on the image of the affected area while considering the chromaticity information. Realize highly accurate diagnostic results.

An image output from an imaging device such as a digital camera may have different colors depending on ambient light and imaging equipment. For example, there are differences in the characteristics of color filters for color separation of image sensors and differences in image processing. In addition, ambient light also causes a difference in visible color, and a difference may occur depending on a light source such as an incandescent light bulb or daylight.

Also, it is known that the color gamut left in the image file by the imaging device is narrower than the color gamut that humans can perceive. 2. Description of the Related Art In order to improve the appearance of an image due to the narrow color gamut, there are cases where the imaging apparatus corrects the colors of the captured image and then saves the corrected image. At this time, the color correction method varies depending on the imaging device, and the color information used for the correction (color information includes, for example, white color information) is discarded after the development process and recorded as data in the image file. It is not what is being done. That is, when a large amount of learning data is formed by collecting a large amount of affected area images taken at various clinics, etc., images taken with various imaging devices under various environmental lights are included. It is difficult to obtain accurate learning data.

In this embodiment, the color information of the reference color near the end of the color gamut that is perceivable by humans under the shooting environment is acquired, and the color information is retained even after image processing and utilized as learning data. Furthermore, by holding the chromaticity information of the basic colors in a wider area than the color gamut that can be handled by the imaging device, it becomes an index for estimating the color tone of the affected area image before undergoing the image processing of the imaging device, and can be used as an inference model. Accuracy can be improved.

In addition, when outputting a diagnosis result based on an image of an affected area, information on colors near the primary colors is important. As mentioned above, in epidermal cases, lesions that collect blood near the skin are observed and diagnosed based on the color, shape, and distribution of redness. The condition is diagnosed by the redness of the affected area. Alternatively, in the example of a biological sample, in a microscopic specimen stained so that the target tissue is blue, the bluish color in the microscopic observation image indicates a quantitative numerical value such as the number or density of the target tissue as a diagnostic result. There is also

FIG. 8 shows the visible region represented by the chromaticity coordinate system and the displayable region of each device. As a method of expressing color information numerically, for example, the CIE1931 chromaticity coordinate system can be used. In this coordinate system, the farther away from the white point, the wider the color gamut, which corresponds to human perception. The colors of the images captured by each imaging device are represented by the ratio of the three primary colors, ie, red, green, and blue (RGB). ) as vertices.

Specifically, in the example of FIG. 8, 801 indicates the visible region of the color gamut perceivable by humans. Reference numerals 802 to 804 indicate the reference color primary color R point, reference color primary color G point, and reference color primary color B point, respectively, and 805 indicates the white point. 806 shows an example of the affected area chromaticity, which is close to the primary color R point.

811 indicates an example of the color gamut when saving the image, for example, the sRGB color gamut. 812 to 814 respectively indicate the primary color R point, the primary color G point, and the primary color B point in the sRGB color gamut. 821 shows an example of the color gamut that can be displayed on the display device. 822 to 824 indicate the primary color R point, the primary color G point, and the primary color B point, respectively, in the color gamut that can be displayed by the display device.

For example, when an object is photographed, even if the color of the object is at the edge of the visible range 801, the color gamut 811 narrows to the color gamut 811 when the imaging device takes the image, performs image processing, and saves the image in a file. Furthermore, when displaying an image file, the color gamut is narrowed to 821 that can be displayed by the display device.

FIG. 9 shows an example of a case where images of the same affected area captured by an imaging device under different environmental lights are output as sRGB images. As shown in this example, the chromaticity 806 of the affected area image and the chromaticity of the reference color primary color R point 802 are clipped to the coordinate values of the sRGB primary color R on the image, and color difference information is lost.

Thus, in this embodiment, in addition to the diseased part image and the diagnosis result by the doctor, the learning data obtained by combining the chromaticity information obtained by measuring the chromaticity of the reference color is used. By doing so, the inference model can be made to perform inference with the chromaticity information taken into consideration, so that a highly accurate inference model can be obtained.

In the above-described embodiment, a case where deep learning, in which a neural network is used to generate feature amounts and connection weighting coefficients for learning by itself, is used as an inference model has been described as an example. However, other algorithms may be used as specific algorithms for machine learning the inference model. Other algorithms may include, for example, decision trees, support vector machines, and like algorithms.

Also, the learning unit 327 described above may include an error detection unit and an update unit. The error detection unit obtains an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the teacher data. The error detector may use a loss function to calculate the error between the output data from the neural network and the teacher data. Based on the error obtained by the error detection unit, the update unit updates the weighting coefficients for coupling between nodes of the neural network so that the error is reduced. This updating unit updates the connection weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients and the like between nodes of each neural network so as to reduce the above error.

(Embodiment 2)
In the first embodiment, the case where the data of the affected area image and the chromaticity information are separate has been described as an example. In the second embodiment, an example will be described in which chromaticity information is added to the meta area of the diseased part image data. Although the present embodiment differs from the first embodiment in the method of holding the affected area image and the chromaticity information, the system configuration and the configuration example of the devices constituting the system, such as the digital camera 101, can be the same as those in the first embodiment. can. Therefore, the same reference numerals are given to the same configurations as in the first embodiment, and overlapping descriptions are omitted.

In S608, the client terminal control unit 309 transmits the image data file received from the digital camera 101, the chromaticity information received from the chromaticity meter 102, and the received diagnosis result as a set of data to the data server 108. do. At this time, the client terminal control unit 309 processes the image file to embed the data structure shown in FIG. 7 as metadata in the diseased part image data. That is, metadata includes various information such as chromaticity information. By doing so, the data server 108 simplifies the trouble of storing the affected area image data, the chromaticity information, and the diagnosis result in association with each other. Therefore, the data server 108 can manage one set of data as one file. In other words, the data server 108 does not need to have an RDB function that associates a plurality of files, so the database server can be realized at low cost.

(Embodiment 3)
In the above-described embodiment, the case where the digital camera 101 and the colorimeter 102 are separate devices has been described as an example. However, the digital camera 101 and the chromaticity meter 102 may be configured as an integrated device. A tristimulus value type chromaticity meter calculates chromaticity from outputs of three sensors, X, Y, and Z, which simulate human spectral luminosity. On the other hand, the imaging device of a digital camera produces an image from an array of three R, G, and B sensors. Therefore, the image pickup device of the digital camera 101 includes, for example, six sensors of X, Y, Z, R, G, and B in the light receiving section, and includes a light receiving section having the output of the colorimeter and the output of the imaging device. good. For example, X sensor and R sensor, Y sensor and G sensor, and Z sensor and B sensor are correlated. By doing so, the light-receiving unit having three sensors of R, G, and B can output the XYZ output value as the chromaticity meter and the RGB output value of the imaging device.

As described above, in the present embodiment, an affected area image obtained by photographing an affected area of a patient and chromaticity information indicating the chromaticity when the affected area image is obtained are acquired, and the affected area image and the Chromaticity information is used to infer the diagnosis result for the affected area. By doing so, accurate diagnosis and pathological quantification can be performed even when images captured under various environmental lights or by various imaging devices are used for machine learning.

Also, the inferred diagnosis result is compared with the diagnosis result provided by the doctor who diagnosed the condition of the affected area. Then, if the difference between the inferred diagnosis result and the diagnosis result provided by the doctor is equal to or greater than a threshold, the learning data including the affected area image, the chromaticity information, and the diagnosis result is used to re-learn the learning model. made it By doing so, when the estimation result contains an error of a predetermined value or more, the learning data containing the sample with the error is used for re-learning, and the inference accuracy can be further improved.

Note that, in the above-described embodiment, the inference model 401 has been described as an example of using a machine-learned model. However, the inference algorithm may also be a rule-based process such as a lookup table (LUT). In that case, for example, the relationship between the input data and the output data is created in advance as an LUT. Then, it is preferable to store this created LUT in the memory of the RAM 263 . When processing the inference model 401, output data can be obtained by referring to the stored LUT. In other words, the LUT is a program for performing processing equivalent to that of the processing unit, and can perform inference processing by operating in cooperation with the CPU, GPU, or the like.

A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA). Also, the processor or circuitry may include a digital signal processor (DSP), data flow processor (DFP), neural processing unit (NPU), or neural engine.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 101...Digital camera, 102...Color meter, 103...Client terminal, 108...Data server, 109...Inference server, 324...Data transmission/reception part, 325...Estimation part, 327...Learning part

Claims (10)

Acquisition means for acquiring an affected area image obtained by photographing an affected area of a patient and chromaticity information indicating chromaticity when the affected area image is obtained;
and an inference means for inferring a diagnosis result for the affected area using the affected area image and the chromaticity information according to a learned machine learning model. The inference means uses the affected area image and the chromaticity information according to the machine learning model to infer the dosage of the medicine to be provided to the affected area as the diagnosis result. The machine learning device according to claim 1. comparison means for comparing the diagnosis result inferred by the inference means with the diagnosis result provided by the doctor who diagnosed the condition of the affected area;
learning including the affected area image, the chromaticity information, and the diagnosed area associated with the affected area image when a difference between the inferred diagnosis and the diagnostic provided by the doctor is greater than or equal to a threshold; 3. The machine learning device according to claim 1, further comprising learning means for re-learning said machine learning model using data. 4. The machine learning device according to any one of claims 1 to 3, wherein the acquiring means acquires the affected area image and the chromaticity information from the same file. the chromaticity information is included in metadata of the file;
5. The machine learning apparatus according to claim 4, wherein said acquisition means acquires said chromaticity information from metadata of said file. 6. The machine according to any one of claims 1 to 5, wherein the chromaticity information is information obtained by measuring the chromaticity of a reference color panel placed near the patient or the affected area. learning device. 7. The machine learning device according to any one of claims 1 to 6, wherein the chromaticity information is information measured by a chromaticity meter. 8. The machine learning device according to claim 1, wherein said chromaticity information includes coordinate values of a color space defined by CIE1931. A control method for a machine learning device,
an acquisition step of acquiring an affected area image obtained by photographing an affected area of a patient and chromaticity information indicating chromaticity when the affected area image is obtained;
and an inference step of inferring a diagnosis result for the affected area using the affected area image and the chromaticity information according to a learned machine learning model. A program for causing a computer to function as each means of the machine learning device according to any one of claims 1 to 8.

Images